Regular VPhot Users:
You may have noticed that we are occasionally getting "out of storage memory" errors recently. In the last few months, my daily report indicates that the number of daily users has increased from 20 to 30. The number of stored images has increased significantly. That is a "good" thing, but it also has negative consequences.
Our AWS storage capacity is a finite resource that we are pushing up against. There are a few fixes as indicated below:
1. Users could/should delete any of their images that are already processed and reported, as soon as possible.
2. We could reduce the storage time limit for images from 120 days to a shorter period (e.g., 30 days)? There is also a database of analysis reports that we could reduce.
3. We could institute some sort of a "fee for use" or "fee for storage" scenario? This could help pay for extra on-line storage.
I request that you provide some comments/recommendations with respect to these options. Be honest and creative! I suspect we are all annoyed by these memory errors and would like them to go away as painlessly as possible! ;-)
Thanks, Ken
Only being a member for a short time, though options 2 and 3 'fee for storage' sounds to me acceptable.
Jari
I would rather eliminate observations I've already sent so I prefer Options 1 or 2. Don't really want to pay for it.
I don't use VPhot as much as I once did, but: you probably want a user's storage space to be at least roughly commensurate with the user's contributions, that is, don't set up a disincentive to submit lots of data (from lots of images). That supports option #2 over #3. (Whereas option #1 is not a policy but simply a wish.)
I'm ok with Option 2, not Option 3.
Question: Are images compressed/archived? If not, how about a script that runs periodically that compresses data older than 30 days and deletes after 120 days. The older than 30 day data is still available but would require it to be unarchived.
Another idea would be to have a box to click that would select an "extended storage" request. Make default 30 days, but if for some reason the user needs to archive it longer, they could check the box.
Both are more work for the software coders, so these ideas may not be practical...
I agree with Eric that option 1 will not work, and has not worked. I think option 2 (shorter storage time) would work the best, at least to deal with the current situation.
I think 30 days is too short for starters. How about starting the new policy with a 90 day limit. If that turns out to still be too long, then go to a 60 day limit, then even shorter if necessary. I think doing this should be considered just as a way of providing breathing room to devise a long term solution.
Long term, I think VPhot users should have an option to pay for image storage, perhaps an annual fee for an X number of GB's. I don't think there should be a charge for members to use VPhot.
Phil
Option 2 is probably best. I agree 30 days is probably too short, but 60 or 90 would like help a lot.
Raising awareness of the issue (like this) will hopefully remind people about deleting processed images that don't need to be in VPHOT anymore, but since that is unmanagable, I think option 2 is the way it will have to be.
Speaking of which, I'll go delete a bunch of my images I don't need anymore right now!
Why do people need to store images on VPhot for extended periods? Upload what you intend to reduce, reduce it, then delete it. Storage is what your home computer is for.
Dennis
Dennis I totally agree, that's what I do. All stored on my home computer.
This may be impractical for any number of reasons, but as an Amazon Prime member I am allowed unlimited no-cost space on AWS for images, but their list does not include FITS images. So, after some period of time could images be converted to and saved as JPGs or some other supported format?? That assumes AAVSO gets an Amazon Prime membership, but that's cheap!
Peter
VPhot is meant to be a pipeline, not an archive. This is fundamental.to the design and management of VPhot.
The retention period of the images is meant to give you enough time to do your analysis. Then you should delete the images to make room for other users.
This gets trickier regarding the analysis reports. These are quite nice and are not deleted. But they need to be culled too. As soon as I figure out how!
If you have strong opinions about the analysis reports, tell me. Maybe we can work out a way to download/upload them for review.
George
What about allowing the user to download everything from VPHOT (report, image with stars identified, estimates, apertures, etc) to their computer? Once downloaded, the info will be atomatically deleted from VPHOT. Would also be good if the user can upload this info back into VPHOT if later on they want to do additional work. The info will go into a temp file with a short expiration period.
I always thought it wold be great to look at a VPHOT image and other info without having to log into VPHOT. Maybe this solution would allow it.
Gary
I agree with George and Dennis. I like storing images for less than 30 days. That should be long enough for even the heavy workload produced by the AAVSOnet telescopes. Seems like that if I have not processed the data in less than 5 days, I could just reload the images to process them when I get time. Mine are usually gone in less than 1 to 5 days. The Analysis logs are kept weeded out. I do like to keep my sequences for couple years for consistancy. I use VPHOT for a stacker and a pipeline. I store multiple copies of my million images locally. My through-put is usually less than 800 images per clear night.
Ray
I favor option 2. The current allowance of storage time of image files up to 120 days is excessive. A 30 day limit is preferable though that may be a bit generous also. How about a limit of 20 days or so? And culling the storage of analysis logs is reasonable IMHO.
IMHO, it is generally agreed by most, that VPhot does not need to nor should store a user's images forever! It obviously does not now.. However, there are circumstances that we consider in making a decision about how long that storage period should be. First and foremost is that we have a finite modest storage capacity now (1 TB).
Many members analyze their images the next day after collection. I normally do that. However, when I am facilitating the VPhot course or I leave on a 2-week vacation before I get to them, I need some extra time. The VPhot Choice course is 5 weeks long. Should I/we expect that I might lose them within a few days and need to upload them again? Perhaps? I have a relatively fast internet connection, so it is not that big a problem but still an extra step. A friend has a poor DSL connection and really groans when he is uploading his images.
Some users are students working on a class project and analyzing data over a semester or so? Should we take their academic schedule into account? Some of our members are conducting research over a few months. For both of these groups, a longer storage time is more efficient.
I'm sure each of us can think of circumstances under which they would like images to remain on the system for a "while", without worrying about them disappearing / require uploading. It is simply a question of ease of use.
Since several comments (expressed or implied) above indicate a general unwillingness to support the VPhot tool in any financial way beyond their dues and others' donations, it is quite clear to me that the only choice is to reduce the storage time limit to a level that is consistent with the number and size of images uploaded to VPhot? Most will be happy, A few will not. In most cases (not all?), it is actually simply a condition of expectation and modification of practice.
This is reportedly NOT a problem for most users AND can be accepted by most others through regular attention to their analysis process. From what I heard, 30 days may be a lower limit to consider. The reason I phrase it that way is that in fact, we have already previously reduced the 120 days to 100 days. We reduced it because otherwise, we had to deal with more frequent errors, manual purging, rebooting and dealing with complaints.
We have observed more use, more images, and bigger images. As long as that trend continues, the time limit will shrink at some uncertain rate to achieve compliance with our 1 TB storage capacity. Despite being a "hope" not a "requirement", is "asking" users to delete their images effective enough?My observation / excuse is that I (and others) keep forgetting to do that! ;-( If it happens automatically, I don't have to worry/remember.
Additional comments / alternatives that do not involve re-writing VPhot code?
Ken
There are some cases were keeping the images more than necessary can save a lot of stress: one of the comps that I was using for a project was later discovered to be a variable so I had to delete all my measurements and do it all over again. I had already ~300 images uploaded to VPhot in the course of two months so my images were still in the system, thank god... I think that reducing the storage time up to 30 days is ok.
Individual limited storage could help manage the reduced space we have, but this involves re-writing Vphot code and for every new member we would get a smaller piece of the cake (1TB). I agree that we should have a free storage and add a fee for those who need more.
Juan Pablo
(1) If the max storage time per image were reduced to 30 or 60 days, wouldn't that solve the overall storage problem for now?
(2) If so, and if each image has the observer name/code attached, shouldn't it be possible to run a pretty simple daily cron job to send an automated form e-mail, say, 5 days before each user's images are to be deleted, so that they have fair warning to do something with them in anticipation of losing them? That way, VPhot and its storage system wouldn't have to be changed at all.
I am a new and use VPhot not for a long time but option 2 and 3 are fine for me,
BR Horst
Inspired by Dennis and George words:
Coding this can be from difficult to a nightmare, but how about transferring images older than a certain amount of time (e. g. 30 days, even shorter) to some user-defined storage, or deleted if no such storage is defined?
Nowadays everybody seems to have a Google Drive, Microsoft One Drive, AWS S3, etc... with plenty of room available, if one wants to keep his/her archive.
Regards.