Work Flow for DSLR Photometry ?

First the easy part , about color depth:

The DSLR raw image (NEF, CR2, ORF...whatever the manufacturer calls it) contains the image information at the native bitwidth of the sensor's analog to digital converter, usually 14bit, 12 bit for older models. This is a more or less raw , digitized readout of the sensor, without (hopefully) any postprocessing done. At fixed exposure length, the ISO setting is pretty much the only other setting affecting the RAW image. All the other fancy settings like whitebalance, noise reduction, contrast modifications etc etc influence how the camara's on-board image processor converts this 12 or 14 bit RAW image into a JPEG image if you configure your camera tio generate one as well. And the JPEG images will be 8 bit per color channel. But we don't care about teh JPEGs at all in photometry, of course.

The workflow ... that's a more complicated matter and deserves a post of it's own.

Cheers

HB   

This is an excellent question. I have figured out a workflow for myself involving several software packages, and I somehow convince myself it's kind of good enough .. and then every two years the b Per eclipse campaign comes along and I find myself doing a lot more manual work than I'd prefer to do... so no recommendation from my side, I'm still optimizing myself.

Some thoughts tho from my own experience so far:

1) for some time I was frustrated with some tools not getting along with the RAW format of my Canon EOS 1100D . Solution: I gave up. I will now convert all of my images up-front to DNG  format using an Adobe tool. I think this is a good approach overall.

2) A main problem is how to get the correct meta-data (at the very least observation time and exposure length, possibly also sensor temperature , GPS location and the correct filter code (TR,TG,TB)) from the camera RAW files all the way into the headers of the FITS files that you will want to work with in the end. For example I use IRIS to split the DNG files (see above) into 4 color channel fit files, but it seems to completely mess up the meta-data. Even exposure time is somehow wrong (e.g. if true exposure time is 5 seconds, generated fits header will state 50 seconds ... why???) . The observation time is another headache. Is the FITS header set to the correct time or does it have some offset (end of exposure vs. start of exposure vs. midpoint, time zone issues, daylight saving time ..... ) ? So I have scripts that read the timestamps from the EXIF data of JPG files that the camera generates together with the raws, and fumble them into the corresponding FITS headers with another script....not an ideal solution. There must be a better way.

3) A complication is the fact that most DSLRs have 2 green pixels for every red and blue pixel, so where in the process do you combine the 2 green channels (right at the front of the pipeline when converting RAWs? If your DSLR has a high pixel count (20+ M), you probably don't want to split your images into 3 "full-size" R G B de-bayered & interpolated images because that is very wasteful in terms of storage and processing time. You probably want to split into images that are 1/4th the original size each, 1 pixel in the FITs image for every green blue and red sub-pixel in the RAW.

4) bit depth: DSLRs often have just 14 bit , older models even just 12 bit resolution. Some tools working with FITS image data seem to assume 16 bit depth for tasks like bad pixel identification or star identification for registering images for stacking .... another minor complication but still a complication to be aware of.

So what I'd like to see is a tool that splits a DNG file into 4 or 3 color channels (R,G1,G2,B or R, G1+G2, B ) without increasing the total size of the data and correctly carrying over the meta-data, optionally scaling the data to 16 bit. Asking too much :-) ?

Cheers

HB

I use AIP4Win for the final photometry but I feed it FITS images batch-converted from the RAWs. AIP4win has AFAIK no means of dealing with dozens, hundreds or (as in the case of my b Per observations) 1000s of RAW images efficiently in a batch way (but it can extract photomety from a converted FITs image at a rate of 1 FITS image per second for my camera). 

The Fitsworks software has some pretty flexible batch processing options, and I think I'll look into it again soon.

> Because of some technical issues with the spreadsheet, I am now looking at automating the reduction

>and reporting processes with a Python script.  But that still won't relieve me of the tedium of target

>selection in the FITS images.

Lesve Photometry

http://www.dppobservatory.net/AstroPrograms/LesvePhotometryDownloadPage…

is automating pretty much everything as it does plate solving on each image so it can find specified target, comparison and check stars all by itself. It needs FITS images tho as well. 

CS

HBE

I used to use AIP4Win for image preprocessing and aperture photometry all the time, until a missing system file caused me grief.

AIP4Win allows batch importing of (in my case) Canon EOS CR2 RAW files, and processes those, without any need for any serparte conversion to the FITS format prior to processing. There is a very long list of cameras from which image files can be imported. All operations can be perfomed in batch mode. Aperture photometry on my (admittedly fast) computer runs throught a batch of images at, I think from memory, just a little slower than 1 frame per second.

The user selects the format of the photometry output. I chose the raw measurements, and imported them into a spreadsheet for the calculation of instrumental and transformed V and B magnitudes, and the plotting of light curves from time series photometry. I used a template workbook for each variable, and thus only the data need be copied and pasted into relevant columns. Formulae existed as part of each template.

I would seriously recommend getting hold of AIP4Win if you are starting in DSLR photometry. Mark Blackford wrote a user guide (MS Word document) a few years ago, which is very useful.

Roy

I should have explained that the plotting of light curves in the spreadsheet after importing AIP4Win data required some manual intervention - that part was not fully automated. A template for the charts helps.

Roy

Interesting. AIP4win doesn't correctly parse my EOS 1100D raws , but works on DNGs, so it's still an option for me.

But how exactly do you perform batch photometry on (say) the green channel of the RAWs? Loading the raws (including the ones for the calibration ) and setting the "whitebalance coefficients" in the DSLR conversion settings ?

Strange, because that is kind of slow on my system. My Camera has 12 M pixels, and the extraction of the green channel via the Debayer/DSLR conversion settings (settings : grayscale = 0* Red + 0 * Blue + 1 * green ) will result in again a 12 M pixel interpolated grayscale image from just the green channel .. which runs significantly slower  than working on 3 M pixel images if working on grayscale FITS from just a green channel (more than five times slower on my PC  with the added overhead from conversion and larger images).

CS

HB

You process one channel at a time, as shown in the attached screenshot.

See the red ovoid around the green channel near the bottom.

Roy

As well as AIP4Win,  Mark Blackford’s guide also gives guidance for using IRIS and Muniwin for DSLR photometry. I think my preference when I started would have been to use AIP4Win but I was put off at the time by stories of problems running it with Windows 10 and the user group didn’t seem to be very active. I'm not very technical and I didn’t want to buy software and then find I couldn’t get it working. Richard Berry has now set up a new user group which seems very active and there are obviously a good number of people using the software without major problems so I think I would be much more reassured.

Instead, I went for IRIS. Mark Blackford’s guidance goes all the way through from de-coding DSLR  raw files to photometry using the dialogue boxes in IRIS. From the IRIS output file I was able to prepare a file to load the data into the BAAVSS website which satisfied the minimum layout requirements but with no reporting of errors.

Andy Wilson of the BAA has developed an excellent spreadsheet which takes output files from AIP4Win, gives some very helpful sense checks and graphs along the way and prepares detailed files, complete with error reporting ready to load both the BAAVSS and AAVSO databases. More recently, Richard Lee has added the facility to read AstroimageJ output files as well. So now, instead of using the IRIS photometry, I take a sequence of registered, FIT files from IRIS and load them into AstroimageJ and do the photometry there. The AiJ output text file can be loaded into the BAA spreadsheet from which the input files for both databases are produced.

 I needed to find the gain (which changes with ISO setting for a DSLR), readout noise and dark noise for my camera so that the SNR and error reporting would work and I found Berry & Burnell’s book The Handbook of Astronomical Image Processing helpful for this. I have now written IRIS batch files to automate the part of the process which is performed in IRIS

.  If anyone wants to take a look at the BAA spreadsheet it can be downloaded together with guidance, from the BAAVSS website. Click on "Database", then "Notes on Submitting Observations".

Thanks.

Yes this will create interpolated, full size grayscale images, bigger than strictly necessary, but again, as Stephen pointed out, this really matters only if you are working with huge datasets.

So coming back at Brad's original question, I guess we can converge on an full recommendation to get AIP4win. AFAIK the only way to get the software is to buy the book "The Handbook of Astronomical Image Processing".

HB

I use AIP4WIN, and do everything one image at a time, without automation or batch processing. I use the bracketing tool while shooting to get a range of exposures of each target, normally 0.7 stops apart, at two different aperture settings. This gives me a workable range of exposures of each target field.

AIP4WIN won't work with Foveon raw files,so I use Sigma Photo Pro to convert to 16 bit tif images.

I create my working images in AIP4WIN by using the split colors tool, calibrating and saving as FITS.

I make my measurements using the single star photometry tool, and use a calculator for computation.

Computations are written with pencil on looseleaf paper, and are submitted using WebObs.

I know this won't appeal to most of you, but I wasted so much time trying to get my laptop to do all the work, I wasn't submitting any observations. All my programming experience was limited to COBOL back in the days of punchcards (1984-88.) I was much more productive extracting data from standard reports by hand than writing retrieval programs of dubious accuracy.

I spend about four hours a day producing 12 observations, or one star per hour, 

I also feel much closer to the work, and the stars measured as a result.

Hello everyone and forgive my English.

In my case, I am using Astroart 7.0, the workflow is quite comfortable, the program has a very intuitive and complete preprocessing. You can directly load the calibration images (linght, darks, bias, flats) in RAW format (Nef, cr2, etc.), the result is a single image stacked and aligned with the possibility of extracting the individual data from the stacked images (Adu's) .

My workflow:

1. I check the images are not saturated and within the linearity of my DSLR, always with the image without demosaic, the interpretation (interpolation) of the color can produce saturated pixels.

2. Pre-treatment of images (Images, darks, bias, flats), average or add, hot-pixel removal.
3. Channel separation, I only usually work with the G channel which is the one that most closely matches the V Jhonson standard.
4. Checking the FWHM or growth curve to determine the aperture photometry.
5. Select calibration stars and target with VSP.
6. Apply Linear Photometry.
7. Data collection, magnitudes and mag error of the different comparison stars.

In my opinion it is one of the most practical programs that I have used, not only for photometry, astrometry, etc.

Greetings and clear skies