Hi all,
Using VStar to analyze light curves for newly found variables I often needed load data from simple text files and join them with surveys’ data. One of the features I missing was lack of band (filter) specification in VStar simple text format. Well, it is possible to create text files in AAVSO Download format, which supports filters however that format requires too many extra columns to be created.
Also, there is no way to tell VStar what kind of Julian Day is in a file of a simple format, JD or HJD (for short-period variables it is important).
I decided to create my own plugin which supports (a) filter specification and (b) HJD. The plugin should be as compatible as possible with existing Simple Text Format.
I did it, probably the plugin might be interesting for other members of the community (some other people were interested in such a feature as filer specification in the input file: https://www.aavso.org/vstar-self-generated-file)
The plugin can be downloaded here (the folder contains compiled plugin along with sources): https://drive.google.com/open?id=1qgq5GWfcF9SHQ87LsoA__beu9GTDwa6r
Download “pmak.vstar.plugin.ExtendedTextFileFormatObservationSource.jar” and put it into vstar_plugins folder, then restart VStar. Now you should see an additional item in the File menu “New Star from Extended Text Format File...” (see attached picture).
The plugin can load comma-separated VStar Simple Text Format (see VStar Manual) without any modifications. Unlike VStar, it does not validate (intentionally) ObsCode value so you may use non-standard observer codes, like ASAS-SN, for example. It could be useful for filtering.
The plugin does not recognize value separator automatically (like VStar does), so you should specify a separator to load tab-separated VStar Simple Text Format using a directive #DELIM at the beginning of the file:
#DELIM=tab
(#DELIM directive has the same meaning as for AAVSO Upload Format)
To specify filter (band) you can use global directive #FILTER. For example, to tell the plugin that all observations in a file were made using Johnson V filter, put the following line at the top of a file:
#FILTER=V
You can also specify a filter for each observation separately in the 6th column of a data file (doing so you will make the file incompartible with VStar Simple Text Format Reader):
2457007.701,13.143,0.014,PMAK,,V
2457007.702,13.116,0.014,PMAK,,V
2457008.7,13.098,0.015,PMAK,,V
2457009.695,13.104,0.015,PMAK,,V
The plugin supports #NAME directive, as VStar reader does:
#NAME=Some Star
2457007.701,13.143
2457007.702,13.116
2457008.7,13.098
To specify that time values are HJD, use #DATE directive:
#Date=HJD
(Directive names are case-insensitive, #Date is the same as #DATE)
Full list of directives:
#NAME=(object name). May appear in any line.
#DATE=(date column type: JD or HJD, default: JD). May appear before observations only.
#DELIM=(delimiter, default: comma). May appear in any line (i.e. to change delimiter)
#FILTER=(default value, if not specified in FILTER column). May appear in any line (i.e. to change the filter)
#OBSCODE=(default Observer Code value, if not specified in ObsCode column). May appear in any line (i.e. to change ObsCode)
#MAGSHIFT=(value to be added to magnitudes, default = 0.0). May appear in any line (i.e. to change magnitude shift)
#DATEADD=(value to be added to date values, default = 0.0). May appear in any line (i.e. to change the value to be added to dates)
#ERROR=(N to ignore all validation errors but time and mag; Y: do not ignore errors; any other values are ignored). May appear in any line.
#FIELDS=(list of fields, see explanation beneath)
#any comment
Using the plugin you can read files of different formats. Existence and position of data fields can be set with #FIELDS directive. The following #FIELDS directive corresponds to a default set of fields:
#FIELDS=Time,Mag,MagErr,ObsCode,Flag,Filter
To read AAVSO download format, put the following line at the top of a file:
#Fields=time,mag,magerr,,filter,obscode,,,,,,,,flag
Using this directive I was able to read some example data files from ‘data’ VStar folder, which cannot be read by VStar itself! (Those files include ch_cyg_data_2454600_to_2455000.csv, eps_aur_2445000.csv, ss_cyg_nov_5_2007_to_2009.csv, etc.)
Another example: the following directives allow us to read ASAS-SN Sky Patrol files:
#Date=hjd
#Fields=time,,,,,mag,magerr,,,filter
#ObsCode=ASAS-SN
#HJD,UT Date,Camera,FWHM,Limit,mag,mag_err,flux(mJy),flux_err,Filter
2457007.70083,2014-12-16.2012822,bb,1.67,16.107,13.063,0.014,22.844,0.299,V
2457007.70229,2014-12-16.2027437,bb,1.78,16.108,13.036,0.014,23.416,0.301,V
2457008.69975,2014-12-17.2002721,bb,2.49,16.031,13.018,0.015,23.812,0.311,V
2457009.69530,2014-12-18.1958917,bb,1.76,15.976,13.024,0.015,23.683,0.328,V
2457009.69670,2014-12-18.1972825,bb,1.62,16.058,13.046,0.015,23.187,0.306,V
<...>
You can find other examples in “samples” folder. “TestForErrors” subfolder contains sample files which generate different errors (for test purposes).
If you are interested in this plugin and find any bugs or errors, please let me know about them.
Best regards,
Max
P.S. Please note that currently, the plugin may not properly recognize quoted fields with delimiters inside quotes that may appear in a data file (i.e. comments in AAVSO Download Format File). "Comment" field appears between "ObsCode" and "Flag" field. To overcome this, use #FIELDS directive without "flag" field when reading AAVSO Download Format:
#Fields=time,mag,magerr,,filter,obscode
I'll give it a try, Max, thanks very much.
Brian
Hi Max,
Quick question: Are you using Windows, Linux, or Mac? I am on Windows 7. Java Version 8 Update 211 (build 1.8.0_211-b12).
I created a local folder for plug-ins and pointed VStar to it (see attachment) using the Preferences dialog. In the attachment I am showing where VStar stores the settings in the Windows registry.
Unfortunately when I restart VStar your plug-in does not show in the File menu. I also tried rebooting. No luck. Also nothing in the VStar log file.
Any ideas?
Thanks,
Brian
Hi Brian,
I'm using Windows 10 and Windows 7 (on different machines).
VStar looks for its plugins in C:\Users\(user-name)\vstar_plugins folder. If you have downloaded any plugin (see https://www.aavso.org/vstar-plugin-library) this folder should exist (with downloaded plugins in it).
If you do not have "vstar_plugins" folder in your home folder, create it and put the plugin in it. VStar should load new plugins after restart.
Best regards,
Max
P.S. I tried to change Plugin Location Base URL using Preferences (see attached pictures). Nothing changed, VStar still looks for plugins in C:\Users\(user-name)\vstar_plugins. It seems those settings are related to the download location (from where to download), not to the destination.
Hi Max,
The good news is that your plug-in works although I barely scratched the surface of all the options you offer. So far I just tested the "#NAME" directive and adding the "V" filter to my observations.
With regards to the plug-in file location, the magic was like you said creating the folder "vstar_plugins" in my User directory though it appears as if VStar looks there regardless of the setting in Preferences. In fact I returned the setting to its default location at "https://www.aavso.org/sites/default/files/vstar-plugins/vstar-plugins-2…" but it still picks up your plug-in in my local folder!
Thanks again,
Brian
Thank you for info, Brian!
If you have any suggestions/comments please let me know!
Best regards,
Max
Hi Max,
Is there a way to describe a Check Star (name, magnitude, error, filter) and if so does it adversely affect Phase Plot?
Brian
Hi Brian,
You could make a separate file for check star (if checkstar data is in the same file as your target, simply make a copy and define required fields with #FIELDS directive). Then you can load both files in VStar to see both curves.
Se attached example: file1_target.txt for target, file2_check.txt for check star (actually the only difference between the files is content of #FIELDS directive (and also #NAME). The result is in a picture attached: green points are for the target, red (filtered by NAME) are for checkstar.
Max
P.S. There are no problems with the phase plot, however, for period analysis (DFT, AoV) you should exclude checkstar data first.
Hi Max
This is really good. Thanks.
Some time ago I decided that when complex fields (e.g. quoted fields containing quotes) are present, it's best delegate CSV parsing in VStar to a library. Handling all aspects of CSV parsing properly can take some effort.
You may find the ObservationFieldSplitter class I wrote useful for this purpose and should be able to replace your call to line.split(...) with a call to ObservationFieldSplitter.getFields().
I agree that what you've done is heading in the direction of what I was thinking here:
https://www.aavso.org/comment/53790#comment-53790
Re: what you were wondering about with VStar plugins directory, yes, the plugin Prefs setting only affects where plugin files can be loaded via the Plugin Manager. The simplest thing is to take your jar and add it to vstar_plugins, as you mention to Brian.
If I have further feedback on the plugin or code I'll let you know.
David
Thank you, David, I will try CSV reader.
Max
Hi all,
I've made some changes in the Extended Text Format Plugin https://drive.google.com/drive/folders/1qgq5GWfcF9SHQ87LsoA__beu9GTDwa6r
1) Now it handles quoted fields (instead of using CSV reader I wrote my own short method (based on my old code come from Delphi years), I'm not sure this was the best way yet it allows me better control of what the method does).
2) A new directive #ESCAPINGQUOTES= has been added. It can be #ESCAPINGQUOTES=N (default value), which implies that internal double quotes in quoted fields should be represented as "" (double double-quotes) or #ESCAPINGQUOTES=Y, in such case double quotes should be represented as \". Using this directive (along with #FIELDS directive) I can read "SN2011FE.csv" example file without errors, while the standard reader cannot read several lines. The "SN2011FE.csv" example file with additional directives is here https://drive.google.com/open?id=1us1CzwOOLHezsRc9EddR-kh6sPGiWQ-f Please note that the directives do not interfere with the standard plugin so the file can be read by the standard plugin too.
3) #FIELDS directive now recognizes 'name' and 'comments' fields. See "SN2011FE.csv" example above.
4) The plugin is compatible with both stable 2.20 and unstable VStar 2.21 Rev. 1630, I've removed DateInfo.setJulianDay() call.
Well, probably it's time to write some documentation. Now a short description of directives is in the source code available by the link at the beginning of the message.
David, what do you think about including this plugin into the standard bundle (this will need renaming package), if you find it useful? Besides its main functionality (filter specification in simple files etc.) It could be used to read new data sources if plugins for them not yet created (using appropriate directives).
Sincerely,
Max
Hi Max
I think including it will be great! I was planning to suggest it. I can imagine using it.
When the new release is made, there will be a few new plugins.
I like to get independent testing of my plugins so I can point a tester at this if you like.
Thanks for thinking about writing doco too. That's useful and important.
David
Hi David,
Yes, independent testing would be great.
Max
Max,
David Benn asked me to test your plugin. I tested all of the files you had in your sample files download. Everything works great!! Nice job. Is there anything else that you need to have tested?
Dave
HDHA
Hi Dave,
Thank you for your job! Now I have no additional things to be tested.
Best regards,
Max
Hi Max
The code is in SourceForge now.
Let us know when you have a user document you'd like feedback on or want help with.
David
Thank you very much, David!
I will prepare a doc, probably slightly later. Sorry for the delay,.
Best regards,
Max
No worries Max. There's still plenty to do before the next VStar release.
David
Hi Max
I have made a candidate release available for testing which includes the plugin jar.
Do you want any assistance with writing a doc or feedback?
Any changes you want to make to the plugin?
I should also check that you want the plugin to be available in this release.
One other thing that occurred to me was that the name "Extended Text Format" may confuse some people when they see that, thinking about AAVSO Upload Format which has Visual and Extended modes. Thoughts?
Thanks.
David
Hi David,
1) I'll try to make a draft of the plugin documentation this week. It would be nice if you help me with subsequent style and spelling corrections: English is not my mother tongue...
2) Currently, I do not intend to make any changes
3) Yes, I want :)
4) Well, the proposed name (AAVSO Upload Format which has Visual and Extended modes) is good.
Best regards,
Max
Hi Max
Yep, no worries. Function over form. :) Churn out a document with the content you want and we'll go from there.
The name I was mentioning (AAVSO Upload Format) is used by another format and plugin. Still thinking about what makes sense here. Dave and Cliff may also want to make suggestions here.
Thanks Max!
David
The plugin is renamed to "Flexible Text Format Plugin". Thank Cliff Kotnik for the idea!
I've updated "Flexible Text Format" plugin. First, I've introduced a new directive, #VELAFILTER. Also some minor fixes have been made. The source, compiled jar and a documentation are available by the link: https://drive.google.com/open?id=1hCWKb4_rI5HfdADmFhb7imC39cIiRbJQ
About #VELAFILTER directive:
#VELAFILTER=<VeLa expression>
VeLa expression can be specified to filter data.
In the following example (SuperWASP CSV file) data to be read are restricted by #VELAFILTER directive. Only observations having uncertainty (MagErr) less than 0.03 and magnitude less than 11.5 will be read from file. Please note that in the example #MAGSHIFT is applied so VeLa filter sees modified “magnitude” values.
#FIELDS=TIME,,MAG,MAGERR
#FILTER=SuperWASP
#DATE=HJD
#MAGSHIFT=-0.1054
## VeLaFilter: "magnitude" after applying #MAGSHIFT and "time" after applying #DATEADD!
#VELAFILTER= uncertainty <0.03 and magnitude < 11.5
#HJD,camera,magnitude,"magnitude error"
2453901.661042,145,10.9648,0.0050
2453901.686169,145,10.9604,0.0054
2453901.702905,145,10.9807,0.0067
2453902.654005,145,10.9801,0.0046
2453902.674919,145,10.9838,0.0046
2453902.685532,145,10.9785,0.0047
...
A new directive #DEFINESERIES has been introduced.
The source, compiled jar, and documentation are available by the link: https://drive.google.com/open?id=1hCWKb4_rI5HfdADmFhb7imC39cIiRbJQ
#DEFINESERIES=SeriesDescription, ShortName, ColorHexCode
Example (new series with name "TESS +0.73"):
#DEFINESERIES= TESS +0.73, TESS +0.73, #660099
#FILTER= TESS +0.73
Fixed #VELAFILTER directive (before the fix VelaFilter did not work if contains '=').
With this fix, star name can contain '=' too (this was impossible before the fix)
Hi Max, all
I like Max's idea of optionally applying a VeLa filter at observation load time so much that a day or so ago I finished generalising this so that all observation source plugins will get this essentially for free.
For those plugins that currently load observations from a file or specified URL, this will Just Work with no additional coding by the plugin developer.
For load from AID, I've added VeLa filter text area to the dialog. The same for Gaia DR2, APASS & AAVSOnet epoch photometry, and B-V plugins. So there are a variety of examples now.
In all cases there is now core logic to accept and apply a filter.
This will be in the next release.
Max, your existing plugin code will still work here and when ready, in conjunction with the next release, we can minimally modify your code to use the common code instead.
David
Hi David,
Ok, thank you!
Best regards,
Max
I just realized more of my confusion: I don't know the function or purpose of a VeLa filter and have no idea why we need it. I think it might be a text filter of some kind and am thinking that #FILTER has nothing to do with the optical filter. AAVSO uses "band", how does VSTAR pick the filter band out of a file? As in UVBRI.
Another way to say it is; What needs to be in the data file so that VSTAR recognizes the filter type/band?
Ray
Hi Ray,
Here is FlexibleTextFormatPlugin documentation (with description of directives): https://drive.google.com/open?id=1jdL9NPrbZ7UO7nrOM4azm8FklgvU48sNDnzYx…
#VELAFILTER defines "data filter". It is optional.
#FILTER defines BAND
'ASAS-SN' band is available only if ASAS-SN plugin is installed. If not, you may specify V band instead:
#FILTER=V
Best regards,
Max
Thanks Max
That nails it down a bit better
Ray