SWIRC DATA REDUCTION PIPELINE

Last update: 2013-02-04 by M Conroy

This is the initial release of the Smithsonian Wide-field InfraRed Camera (SWIRC) data reduction pipeline. The model data flow is to reduce up to an entire run at once, if so commanded. In addition, break points can be set to stop the execution after a given phase if so desired. The pipeline can be rerun from the beginning and any steps already completed will be skipped.

The pipeline programs are mostly Korn Shell 93 scripts (ksh) which, which appropriate supporting software, will run on both Linux and Solaris systems. There is one C program, which is recompiled for the appropriate architecture.

Installation Overview Usage
Options

sw_pipline sw_flatproc sw_pinwheel

sw_setup sw_objproc sw_imwcs

sw_darkproc sw_skysub sw_swstack

sw_darkcheck sw_chanfix

Email Comments & Bug reports

INSTALLATION Back to top

Installation involves running a script to check your environment and copying the various programs to a directory you specify, plus making the pinwheel correction program, written in C, for the appropriate CPU architecture. Most of the environment checks examine your PATH environment variable to make sure the right support ksh (ksh93), IRAF, Starbase and WCStools programs are available. To run the installation, execute:

/data/tdc/src/SwircPipe/Setup <setup-dir name>

If dirname is blank, the installation copies to the current directory.

If Setup detects errors, it reports them and exits before proceeding.

For example, in the CF domain (= cfyp.cfa.harvard.edu ), the leading elements of your path should be:

/data/oiropt/starbase /data/oiropt/bin /data/oir/bin

Note that /data/oiropt/ mounts different directories according to the computer architecture, either Solaris or Redhat Linux.
Note: As of September 2009, Solaris support is discontinued.

Also, the following environment variables must be set:

IRAFARCH

iraf

FPATH

daterange

Five of the files copied over are parameter files, not programs, and are replaced only if the user so specifies, via a dialog for each one. Four of these are files for the Sextractor program ( defswirc.sex, defswirc.nnw, defswirc.param, and defswirc.conv ) and one file, swirc_bad.reg, a ds9 region file used to screen out sources found in the bad regions of the chip.

Note that the software is copied from file in a CVS repository tagged 'stable'. This will generally not be the very latest version until some testing has been done.

Finally, the last preparation for running the reductions is to set the sw_HOME environment variable to the directory containing the five Sextractor-related files, put the directory containing the pipeline software into the PATH, and set the UPARM environment variable to the location of your IRAF parameters.

OVERVIEW Back to top

sw_pipeline: The master script which calls each of the other stages in the pipeline.
sw_setup: Copies files from the raw archive area to a user-supplied scratch storage area. Does some consistency checks ("dark" files should have IMAGETYP = 'dark', etc.), and interactively allows the user to delete 'test' files, 'focus' files, etc.

Makes a starbase table of all the remaining files 'files.db', and uses support script sw_filesumm to determine which files do not have dark frames of the same exposure time or flats with the same filter.
sw_darkproc: Determines which darks to combine and does so, creating one average exposure file for each exposure time.
sw_darkcheck: If a file (as recorded in objtimes.db) does not have the corresponding dark file, this checks the nearby nights to find the correct one, linking it into the Darks subdirectory if found.
sw_flatproc: Determines which sky or dome flats to combine, then does so, creating one normalization file for each filter.
sw_objproc: Dark-subracts and normalizes each object file.
sw_skysub: Sky-subtracts each object file, using various heuristics, with supporting scripts sw_skysetproc and sw_skysubproc.
sw_chanfix: Checks the chip channel with variable bias and fixes it if necessary.
sw_pinwheel: Fixes the 'pinwheel' bias variation in quadrants 2, 3 and 4. Quadrant 1 is so noisy this correction is not attempted.
sw_imwcs: Call Sextractor and imwcs on each image to get a sub-arc-sec world coordinate system into each header, using support routine sw_imsex and summarizing with sw_wcssumm.
sw_swstack: Uses IRAF imcombine to stack all exposures of each object (by filter, of course) into a single result image.

USAGE Back to top

FIRST, make sure the environment variables mentioned in the installation section, above, are set up.

If you are invoking this from a csh or tcsh shell, use setenv, e.g.:

 
  setenv UPARM ~/iraf/uparm/   note the trailing slash!
  setenv sw_HOME <setup-dir>/swprogs    
  setenv PATH ~/swprogs:${PATH}

If you use a Bourne or Korn shell, set the variable and export it globally:

  UPARM=/home/me/iraf/uparm/;  export UPARM
  sw_HOME=/home/me/swprogs;   export sw_HOME
  PATH=/home/me/swprogs:${PATH}; export PATH

THEN, cd to your scratch area and start up the pipeline. I find piping the output to a log file is a good idea, and you probably want to expand the terminal window size to all or most of the screen. The example following is for csh syntax:

  sw_pipeline [options] date-range |& tee pipe.log

  sw_pipeline [options] date-range >& pipe.log & sleep 1; tail -f pipe.log

where date-range is a dash- or comma-separated list of dates, of the form YYYY.MMDD. When two dates are separated by dashes, the script expands them to include all dates between. Thus, the daterange:

   2005.1130-2005.1202,2006.0112-2006.0114

expands to the set

   2005.1130 2005.1201 2005.1202 2006.0112 2006.0113 2006.0114

The default is for the pipeline to copy the raw data for these dates from the SWIRC archive to the scratch area. This is handled by sw_setup, as detailed below.

Note that if you have separately copied the data to a directory or are rerunning the pipeline from inside the directory, a simple "." (== current directory) instead of a daterange is valid also.

Since the dark count levels are not linear with exposure time, there must be darks at the same exposure time as the objects being processed. If these darks are not found, the sw_darkcheck program looks at neighboring nights to see if the appropriate darks can be found.

If all the necessary darks are not known to be available in a night's dataset, on the initial run of the pipeline, several nights should be processed through the ``dark processing'' phase. Use the -BD option to exit after this stage, then restart with only the night(s) of your data. The pipeline will skip all the previously completed processing.

Options Back to top

SW_PIPELINE options

NOTE 1: The sw_pipeline script and all the other sw_* scripts have available the -x option to dump a Usage description and exit. This is useful anytime you need a reminder of a script's options.

NOTE 2: All the sw_* scripts except sw_pipeline, sw_setup, and sw_darkcheck can take, as the last arguments on the command line, a list of FITS files to process. If given, this list overrides any directory given with the -p option.

The option list below is extensive, as many of the options are simply passed to the other programs of the pipeline. Generally, you only need to specify a verbosity level (-v, -vv, etc.) and the date range. There are also a number of options to skip steps or break out of the processing after a certain stage.

Option list:

-d 3: - drop first N darks of each set
-D 300: - Max separation time (sec) in an object set, from end of one exposure to beginning of the next.
-n 15: - Number of sky frames to use in skysub
-g 31.0: - Min separation angle from the object used to choose sky frames. If there are not enough (default 15), the separation is halved and the routine tries again.
-o 10.0: - Min object exposure time (seconds) to be considered valid. Objects with shorter exposure times are flagged for possible deletion in the sw_setup routine via an interactive user prompt.
-p <dir>: - reduction directory. Using the current directory to start is the default.
-j <num>: - number of parallel jobs to fork. If not given, the pipeline chooses a number based on the number of CPUs.
-M 16: - background mesh size for J filter skysub
-v: - increase verbose level (can be used more than once). Using -vv at first will generate lots of output which may be useful to understanding the processing, but usually zero or one -v is preferred.
-q: - silent - no output except bad errors. Use cautiously.
-x: - print Usage summary and exit
-a /home/swirc/Archive/rawdata/data/SWIRC: - Raw archive directory. This is very unlikely to change.
-m /data/wdocs/swirc/Calibration/mask.pl: - bad pixel mask, copied to each reduction directory in case the user wants to fiddle with it. It is applied just before sky subtraction.
-w 0.5: - WCS fit residual minimum in arc-sec. If sw_imwcs finds a value larger than this after trying various tolerances and catalogs, the image cannot be used for stacking.
-S <filter>: - Add filter to those flagged for special treatment. Normally filter J is the only one.
+S <filter>: - only this filter gets special treatment. Give this as none to turn off special treatment for the J filter.
-s [400:1600,400:1600: - statistics image section. The cosmetics are bad at the top of the chip, so only the center is used for many operations.
-H [50:2000,50:1600]: - H filter statistics section, used in sky subtraction.
-J [380:780,525:775]: - J filter statistics section, used in sky subtraction. This is carefully chosen to sample the peak scattered-light area.

-R $sw_HOME/swirc_bad.reg: - bad-region file for sw_imwcs. Can be viewed or regenerated with DS9. Used before imwcs to exclude sources from Sextractor located in the bad regions.
-c: - copy datadir, even if files already there. Normally the sw_setup routine tries to not copy data files twice.
+A or -A: - skip sw_setup copies and checks (use caution!). This can save time when rerunning the pipeline, as there are some time-consuming checks that need not be repeated.
+C: - skip bad channel correction. Available just in case.
+P: - skip pinwheel correction. Available just in case.
-B <breakpoint> where "breakpoint" is a letter: I = Initial setup D = Dark processing F = Sky Flat processing O = Object dark & flat correction S = Sky background correction C = Bad channel correction P = Pinwheel correction W = WCS calculation - Break out of pipeline at the indicated stage. This might be used, for example, to do an entire run's dark processing so that dark exposures of the correct length can be made available to a night otherwise missing them. Then, the pipeline can be restarted for a particular night of interest.
-x: - Print usage message and quit

SW_SETUP options Back to top

The sw_setup script copies data from the archive area to the user's scratch reduction areas. It skips copying file names containing "test" or "focus", and checks for some simple problems such as a file of image type object having a filter of "D1" or "D2" (i.e., a dark).

Several operating files, including the bad pixel mask file, the bad region file and the Sextractor control files are copied from sw_HOME to thereduction directory. This is allows the user to alter these if it is desired to rerun some processing later.

After copying files, it presents to the user a table of files with the header keyword OBJECT value containing "test" or non-dark or non-flat files with exposures less than 10 seconds. The user is queried if to delete these files from further processing. If the answer is "no", the user if further queried if he or she wants to iterate through the list and delete or retain files individually.

Next, sw_setup makes a starbase table of a useful subset of keywords from all the remaining files, outputting it to files.db, and uses support script sw_filesumm to create some informational files about the number of darks, flats, objects and so on. It also determines which files do not have dark frames of the same exposure time or flats with the same filter.

Finally, sw_setup checks each header for the BPM (bad pixel mask) keyword. Any files without it have the BPM=<path>/mask.pl keyword set, and the ORIGBPM keyword as to the original source (e.g. $sw_HOME).

Since there is a lot of file copying and accessing going on in this script, once the file set is determined to be correct and the headers in order, any reruns via sw_pipeline ought to have the -A option set to skip this routine.

Option list:

-a /home/swirc/Archive/rawdata/data/SWIRC: - archive directory (raw data file location)
-m /home /data/wdocs/swirc/Calibration/mask.pl: - bad pixel mask, copied to current directory
-p <dir>: - reduction directory. Default is "."
-c: - copy datadir, even if files already there
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit

SW_DARKPROC options Back to top

Determines which darks to combine and does so, creating one average exposure file for each exposure time.

The file-choosing algorithm is to look for one or more series of dark exposures, with less than 15 seconds from the end of one to the beginning of the next, and save these in a list. Since it is empirically observed that initial darks in a sequence are anomalous, the first three (controlled by the -d option) are dropped from each series found. Thus, this algorithm thus will reject any single test dark, but will combine two sets from the afternoon and the following morning. The combine is done using IRAF's imcombine, with "average" combination, "avsigclip" rejection, plus the bad pixel mask.

At the end, the Darks subdirectory contains the combined darks where each file is of the form, e.g., Dark_10.0.fits, and a summary table of all of them in darkstats.db.

Option list:

-d 3: - drop first N darks of each set. Default is 3.
-l files.db: - DB of FITS relevant values
-f darktimes.db: - table with counts of files at each exposure time
-p <dir>: - reduction directory. Defaults to ".".
-s [*,*]: - statistics image section
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit
file-list: - Process only the files in filelist. Overrides the -p option.

SW_DARKCHECK options Back to top

If a file (as recorded in objtimes.db) does not have the corresponding dark file, this script checks the nearby nights to find the correct one, linking it into the Darks subdirectory if found. It first looks for the correct dark in the night after the current one, then the preceding, then two nights later, then two nights earlier, and so on. When a dark correction file is found, it is linked into the Darks subdirectory.

Option list:

-f darktimes.db: - DB list of darks
-o objtimes.db: - DB list of object times
-p <dir>: - reduction directory. Defaults to ".".
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit

SW_FLATPROC options Back to top

Determines which sky or dome flats to combine and does so, creating one normalization file for each filter. Each file is in the Flats subdirectory, of the form, e.g. Flat_H.fits. The combination logs and some summary informational files are also in the Flats subdirectory.

Option list:

-l files.db: - DB of FITS relevant values
-h 7000: - Use only flats w/ higher than this counts
-p <dir>: - reduction directory. Defaults to ".".
-s [400:1600,400:1600]: - statistics image section
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit
file-list: - Process only the files in filelist. Overrides the -p option.

SW_OBJPROC options Back to top

Dark-subtracts and normalizes each object file, using the appropriate dark and flat file, with the output being "(input - dark) / flat".

Option list:

-l files.db: - DB of FITS relevant values
-j <num>: - Max number of multiprocessing jobs
-m mask.pl: - pixel mask file
-p <dir>: - reduction directory. Defaults to ".".
-s [400:1600,400:1600]: - statistics image section
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit
file-list: - Process only the files in filelist. Overrides the -p option.

SW_SKYSUB options Back to top

Sky-subtracts each object file, using various heuristics, with supporting scripts sw_skysetproc and sw_skysubproc.

N.B.: J-filter stray light is (attempted to be) suppressed by extra steps. These steps may compromise photometry of extended objects.

The exposures to be used for skys are combined as normal, using a median filter.
The backgrounds of the combined exposures and of the object exposure to be corrected is measured from the lower-left corner where there is empirically observer to be no contaminating light. This constant background is subtracted off of both the combined skys and the object.
The "peak" of the contamination in both is then measured via a small statistics box centered on the observed peak contamination area. The object is then sky-subtracted, using the relative heights of the peak contamination box as a scaling factor.
The constant background of the object exposure is added back in.
Since this eliminates some but not all of the contamination, the Sextractor program is called with a smaller than normal (default 16 instead of 64) background mesh size to output the background-subtracted image.

Option list:

-g 31.0: - Min separation angle (arc-sec)
-j <num>: - Max number of parallel processes
-l files.db: - DB of FITS relevant values
-p <dir>: - reduction directory. Default is ".".
-M 16: - Mesh size for J sky background subtraction (if special processing is performed).
-n 15: - Number of images to use for sky determination
-t 300: - Max time (sec) between images in a set (end of one to beginning of the next)
-F <filter>: - Do only sets of filter >filter
-S <filter>: - Add filter to those flagged for special treatment. Normally filter J is the only one.
+S <filter>: - only this filter gets special treatment. Give this as none to turn off special treatment for the J filter.
-s <statsect>: - statistics image section both filters, overrides options -J and -H.
-J [380:780,525:775]: - statistics image section for J filter. This is empirically determined to be the region of the peak of the scattered light.
-H [50:2000,50:1600]: - statistics image section for H filter
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit
file-list: - Process only the files in filelist. Overrides the -p option.

SW_CHANFIX Back to top

Checks the chip channel with variable bias and fixes it if necessary.

Option list:

-d 15.0: - bad channel difference limit
-j <num>: - Max number of parallel processes to use
-l: - DB of FITS relevant values
-p <dir>: - reduction directory. Defaults to ".".
-s [1024:2048,384:512]: - bad channel image section
-u [1024:2048,513:768]: - above bad channel image section
-b [1024:2048,128:383]: - below bad channel image section
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit
file-list: - Process only the files in filelist. Overrides the -p option.

SW_PINWHEEL Back to top

Fixes the 'pinwheel' bias variation in quadrants 2, 3 and 4. Quadrant 1 is so noisy this correction is not attempted.

Option list:

-j <num>: - Max number of parallel processes to use
-l files.db: - DB of FITS relevant values
-p <dir>: - reduction directory. Defaults to ".".
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit
file-list: - Process only the files in filelist. Overrides the -p option.

SW_IMWCS Back to top

Call Sextractor and imwcs on each image to get a sub-arc-sec world coordinate system into each header, using support routine sw_imsex and summarizing with sw_wcssumm. The initial WCS attempt is with the 2MASS catalog ("tmc") but if that fails the USNO-B1.0 catalog ("ub1") is used. There must be at least eight matches with a position separation less than 0.5 arc-sec for the WCS to be valid.

Option list:

-c tmc: - Initial catalog
-a ub1: - Alternate catalog
-n 400: - number of stars to keep in pre-cleanup Sextrator catalog
-R ./swirc_bad.reg: - Bad region file
-j <num>: - Max number of parallel processes to use
-l files.db: - DB of FITS relevant values
-p <dir>: - reduction directory. Defaults to "."
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit
file-list: - Process only the files in filelist. Overrides the -p option.

SW_SWSTACK Back to top

Uses IRAF imcombine to stack all exposures of each object (sorted by filter, of course) with a valid WCS into a single result image. The combination uses "average", reject="sigclip", zero="median", scale="median", weight="!INVMODE" (inverse of the mode * 1000.).

Ouput images and the logs are in the subdirectory Stack, with names of the form, e.g., NGC1893_H.stk.fits.

At this point all fits files should be in one of the subdirectories, either Raw or Unstack or some intermediate one. Any that are left had some processing error, presumably an invalid WCS so they couldn't be stacked.

Option list:

-j <num>: - Max number of parallel processes to use
-l files.db: - DB of FITS relevant values
-s [50:2000,50:1600]: - statistics image section
-p <dir>: - reduction directory. Defaults to ".".
-v: - increase verbose level
-q: - silent - no output
-x: - Print usage message and quit
file-list: - Process only the files in filelist. Overrides the -p option.

sw_pipline	sw_flatproc	sw_pinwheel
sw_setup	sw_objproc	sw_imwcs
sw_darkproc	sw_skysub	sw_swstack
sw_darkcheck	sw_chanfix

I	= Initial setup
D	= Dark processing
F	= Sky Flat processing
O	= Object dark & flat correction
S	= Sky background correction
C	= Bad channel correction
P	= Pinwheel correction
W	= WCS calculation
	- Break out of pipeline at the indicated stage. This might be used, for example, to do an entire run's dark processing so that dark exposures of the correct length can be made available to a night otherwise missing them. Then, the pipeline can be restarted for a particular night of interest.