American Association of Variable Star Observers (AAVSO)
Fri, 12/17/2010 - 04:05
How does one go about programatically extracting light curve data from the AAVSO? Can someone provide a pointer to an API or some sample code snippets? Thanks!
For access to the AAVSO International Database (AID), Java apps like Zapper and VStar use direct SQL via JDBC. As far as I am aware, the LCG and other web based tools also use straight SQL to the AID. Unlike tools such as VStar and Zapper, they are server-side already of course.
Awhile back I raised the possibility of a Web Service (WSDL-based or RESTful) to:
a. Keep all SQL access server-side (security, performance, ...).
b. Broaden the class of apps that could get access to the AID. I was thinking specifically of devices like the iPhone and iPad and before seeing your iPhone app, had been considering working on a light curve reader for the iPhone. But now I can let someone else worry about that. :)
I'm happy to contribute to further discussion and investigation regarding an AID Web Service however, assuming AAVSO is supportive of such a thing.
This VStar SourceForge wiki page summarises the input methods VStar uses:
In addition to being able to read AAVSO download format files, I've wondered about taking an http stream from selection of a dataset via a page like this http://www.aavso.org/data-download, but modified to allow parameters such as AUID and date range to be passed in the URL rather than having to use a form, making it amenable to programmatic access. This is presumably the approach you take to VSX access in your iPhone app, right?
There's also the XML VOTable format, but I'll make that the subject of another post since I have a question about it.
There may well be other methods I'm not aware of. Chris, Aaron, Sara, others?
Multiple people have expressed interest in having access to the LCG and other tools via an API. At some point in the future I'd like to provide a RESTful API to those tools.
Will's plan to develop RESTful APIs is terrific and opens up many interesting possibilities for creating third-party apps and for disseminating AAVSO data. AND for ensuring the stability of applications in lieu of inevitable schema design changes. Does vstar have a dedicated mysql account with various access privileges defined? Or is it a dedicated account for all third-party software? Two more quick questions:
1. Who has a db schema? Will?
2. Where do I sign up for such access? :-)
Agreed re: Will's plan for a RESTful API. Yes, immunity against schema changes is a point I forgot to add to my post. I would personally feel a lot happier not doing direct database access from a non-server-side app for this reason and the others I mentioned.
VStar uses an account that enforces read-only access to AID and VSX and permits insertion into a discrepant reporting table. This account was set up at the start of the project last year. There's nothing special about it other than that. I also had a testing database set up early on so I can run regression tests against a database that doesn't change.
You would have to ask Will or Doc Kinne for access.
describes the AID schema. The last two sections are no longer relevant since VSX is used for object identification now (it wasn't to begin with). Here's the class VStar currently uses to access it via SQL:
Agreed re: Will's plan for a RESTful API. Yes, immunity against schema changes is a point I forgot to add to my post. I would personally feel a lot happier not doing direct database access from a non-server-side app for this reason and the others I mentioned.
VStar uses an account that enforces read-only access to AID and VSX and permits insertion into a discrepant reporting table. This account was set up at the start of the project last year. There's nothing special about it other than that. I also had a testing database set up early on so I can run regression tests against a database that doesn't change.
You would have to ask Will or Doc Kinne for access.
describes the AID schema. The last two sections are no longer relevant since VSX is used for object identification now (it wasn't to begin with). Here's the class VStar currently uses to access it via SQL:
Hi John
For access to the AAVSO International Database (AID), Java apps like Zapper and VStar use direct SQL via JDBC. As far as I am aware, the LCG and other web based tools also use straight SQL to the AID. Unlike tools such as VStar and Zapper, they are server-side already of course.
Awhile back I raised the possibility of a Web Service (WSDL-based or RESTful) to:
a. Keep all SQL access server-side (security, performance, ...).
b. Broaden the class of apps that could get access to the AID. I was thinking specifically of devices like the iPhone and iPad and before seeing your iPhone app, had been considering working on a light curve reader for the iPhone. But now I can let someone else worry about that. :)
I'm happy to contribute to further discussion and investigation regarding an AID Web Service however, assuming AAVSO is supportive of such a thing.
This VStar SourceForge wiki page summarises the input methods VStar uses:
https://sourceforge.net/apps/mediawiki/vstar/index.php?title=Input
although since the web page upgrade, the download format page link has broken.
More recently, we've added support for arbitrary data sources via VStar's plug-in capability. See http://www.aavso.org/vstar-plugin-library for examples.
In addition to being able to read AAVSO download format files, I've wondered about taking an http stream from selection of a dataset via a page like this http://www.aavso.org/data-download, but modified to allow parameters such as AUID and date range to be passed in the URL rather than having to use a form, making it amenable to programmatic access. This is presumably the approach you take to VSX access in your iPhone app, right?
There's also the XML VOTable format, but I'll make that the subject of another post since I have a question about it.
There may well be other methods I'm not aware of. Chris, Aaron, Sara, others?
You asked for code snippets. Here is a VStar class that shows SQL to access the AID via JDBC:
http://vstar.svn.sourceforge.net/viewvc/vstar/trunk/src/org/aavso/tools…
See also http://vstar.svn.sourceforge.net/viewvc/vstar/trunk/src/org/aavso/tools… for other data access classes.
This fairly ugly data access layer code is undergoing further refactoring as I get time.
As this class suggests, I convert datasets irrespective of source to a canonical format:
http://vstar.svn.sourceforge.net/viewvc/vstar/trunk/src/org/aavso/tools…
Regards,
David Benn
Multiple people have expressed interest in having access to the LCG and other tools via an API. At some point in the future I'd like to provide a RESTful API to those tools.
Will's plan to develop RESTful APIs is terrific and opens up many interesting possibilities for creating third-party apps and for disseminating AAVSO data. AND for ensuring the stability of applications in lieu of inevitable schema design changes. Does vstar have a dedicated mysql account with various access privileges defined? Or is it a dedicated account for all third-party software? Two more quick questions:
1. Who has a db schema? Will?
2. Where do I sign up for such access? :-)
John / RJB
Agreed re: Will's plan for a RESTful API. Yes, immunity against schema changes is a point I forgot to add to my post. I would personally feel a lot happier not doing direct database access from a non-server-side app for this reason and the others I mentioned.
VStar uses an account that enforces read-only access to AID and VSX and permits insertion into a discrepant reporting table. This account was set up at the start of the project last year. There's nothing special about it other than that. I also had a testing database set up early on so I can run regression tests against a database that doesn't change.
You would have to ask Will or Doc Kinne for access.
The first part of this page:
https://sourceforge.net/apps/mediawiki/vstar/index.php?title=AAVSO_Inte…
describes the AID schema. The last two sections are no longer relevant since VSX is used for object identification now (it wasn't to begin with). Here's the class VStar currently uses to access it via SQL:
http://vstar.svn.sourceforge.net/viewvc/vstar/trunk/src/org/aavso/tools…
As I mentioned in another post, this can be done better than it is currently, as per this SourceForge tracker that I have not had time to get to yet:
https://sourceforge.net/tracker/?func=detail&aid=3070124&group_id=26330…
That page provides the SQL I'll move to, so that's probably better to look at.
Regards,
David
Agreed re: Will's plan for a RESTful API. Yes, immunity against schema changes is a point I forgot to add to my post. I would personally feel a lot happier not doing direct database access from a non-server-side app for this reason and the others I mentioned.
VStar uses an account that enforces read-only access to AID and VSX and permits insertion into a discrepant reporting table. This account was set up at the start of the project last year. There's nothing special about it other than that. I also had a testing database set up early on so I can run regression tests against a database that doesn't change.
You would have to ask Will or Doc Kinne for access.
The first part of this page:
https://sourceforge.net/apps/mediawiki/vstar/index.php?title=AAVSO_Inte…
describes the AID schema. The last two sections are no longer relevant since VSX is used for object identification now (it wasn't to begin with). Here's the class VStar currently uses to access it via SQL:
http://vstar.svn.sourceforge.net/viewvc/vstar/trunk/src/org/aavso/tools…
As I mentioned in another post, this can be done better than it is currently, as per this SourceForge tracker that I have not had time to get to yet:
https://sourceforge.net/tracker/?func=detail&aid=3070124&group_id=26330…
That page provides the SQL I'll move to, so that's probably better to look at.
Regards,
David