#-----------------------------------------------------------------------------
# IRACproc: IRAC Post-BCD Processing v4.3 PROCESSING
#-----------------------------------------------------------------------------
(c) 2005 Mike Schuster, Massimo Marengo (mmarengo@cfa.harvard.edu)
(c) 2005 SAO - Smithsonian Astrophysical Observatory
Harvard-Smithsonian Center for Astrophysics
Smithsonian Astrophysical Observatory
30 JUL 2008
* It is assumed that your IRAC BCD products were taken from the archive June
4, 2004, or later. If you have data that is older than this, you can get the
latest pipeline calibration by downloading the data again.
* The SSC has improved the WCS pointing information in the headers of the BCD
frames beginning with the S13.2 pipeline. We recommend that you always use
the latest pipeline BCD data. We have therefore set as the default for
IRACproc 'USE_REFINED_POINTING = 0' when running MOPEX (i.e. off), as this
uses the so called 'super-boresight' pointing from the SSC rather than the
less accurate refined pointing for each IRAC band. When processing data from
previous versions of the pipeline (S12 and earlier) make sure to switch on
the 'USE_REFINED_POINTING = 1' option when running MOPEX (i.e. on), as there
was no super-boresight pointing for these pipelines. This logical switch can
be found in three configuration files located in the Lists/ subdirectory
created after running the fixup routine in Processing #1 below, and should be
changed prior to proceeding with Processing #4. The files are fif_tmpl.nl,
mosaic_tmpl.nl, and overlap_tmpl.nl.
Creating mosaics of your IRAC data with IRACproc is a 3-step process:
step 1: Initialize SSC delivered AOR data and create customized
configuration files.
step 2: Create master image lists for processing (one per channel).
step 3: Create mosaics.
For additional information on the IRACproc scripts:
Documentation for IRACproc scripts can be found in
IRACproc/docs/scriptname.html
To view options for perlDL scripts type 'pdlscriptname()' at the perldl>
prompt.
PerlDL script options are specified with the following syntax:
OPTION_NAME=>'option_value'. Strings must be specified within quotes ''.
To view information files for perlDL scripts type 'help pdlscriptname' at
the perldl> prompt.
To see options for perl scripts type 'scriptname -h' at the command
prompt.
Each new run of IRACproc will clobber all intermediary data products for that
channel.
#-----------------------------------------------------------------------------
# Processing Step #1 - Data Initialization
#-----------------------------------------------------------------------------
Run the script fixup to create the proc_dir directory structure and prepare
files for IRACproc. "proc_dir" is where you will process your data. Your
working directory will be: /<data_path>/proc_dir/reduce/.
Go to the location where all of the processing work will be done.
user_name][ cd /<data_path>/
Form 1: Make a list (aor.lis) of related AOR directories (i.e. for the same
field) to be processed with a single run (relative as well as absolute paths
are OK).
user_name][ fixup -pmasks mmmyy -aorlist aor.lis proc_dir
Form 2: You can also list each AOR (w/ path) on the command line.
user_name][ fixup -pmasks mmmyy proc_dir aor1 aor2 ... aorN
The script fixup takes several options that should aid with setting up your
data for processing. This is a short list of parameters that are likely to
change for individual programs. See IRACproc/docs/fixup.html for additional
information. See IRACproc/pmask/pmasks.README for details on the SSC Pmasks.
USAGE: perl fixup.pl [--help][--ver][--aorlist file]
[--link][--bcdpath string][--prefix string]
[--pmasks mmmyy][--subarray]
proc_dir [aor1 aor2 ... aorN]
To reset the configuration files in an existing IRACproc processing directory:
------------------------------------------------------------------------------
user_name][ cd /<data_path>/
user_name][ refixup -pmasks mmmyy proc_dir
USAGE: perl refixup.pl [--help][--ver][--pmasks mmmyy][--subarray] proc_dir
For programs that have many AORs - with one AOR per field:
----------------------------------------------------------
Use the script multi_fixup to run fixup in batch mode. The file aor.lis
should be a list of unrelated AOR directories (one AOR per field). The
proc_dir/ directory for each AOR will be given the name "Target Name" as
listed in any one of the README files in <rAOR>/ch<CH>/bcd/. See
IRACproc/docs/multi_fixup.html for additional information.
user_name][ multi_fixup -pmasks mmmyy -aorlist aor.lis
USAGE: perl multi_fixup.pl [--help][--ver][--aorlist file][--link]
[--pmasks mmmyy][--subarray]
#-----------------------------------------------------------------------------
# Additional Processing Step For IRAC Sub-Array Data ONLY
#-----------------------------------------------------------------------------
If you are processing Sub-Array data, you will need to perform one of the
following additional steps before continuing.
Method 1: Preferred. Coadds each image pack. To view options type
'subcoadd_bcd()' at the perlDL prompt. To view information file type
'help subcoadd_bcd' at the perlDL prompt, or see
IRACproc/docs/subcoadd_bcd.html for additional information.
user_name][ cd proc_dir/reduce/
user_name][ perldl
perldl> subcoadd_bcd({CH=>1})
perldl> subcoadd_bcd({CH=>2})
perldl> subcoadd_bcd({CH=>3})
perldl> subcoadd_bcd({CH=>4})
Some typical parameters for coadding sub-array image packs:
CH IRAC channel: '1', '2', '3', '4'
PMASKS Set of Pmasks, as distributed by the SSC, specified by
month and year, available options: jun06, dec05, jun05,
feb05, jan05, oct04, jun04, mar04
SIGMA Number of sigmas for outlier rejection (default 5)
Note: This routine also places the original _bcd.fits, _bdmsk.fits, and
_bunc.fits image packs in .bcd_pack_tmp/.
Method 2: Very slow. This method should only be used if you really need each
layer of a sub-array image pack separately. To view options type
'subsplit_bcd()' at the perlDL prompt. To view information file type
'help subsplit_bcd' at the perlDL prompt, or see
IRACproc/docs/subsplit_bcd.html for additional information. To slice the
image packs into individual images:
user_name][ cd proc_dir/reduce/
user_name][ perldl
perldl> subsplit_bcd({CH=>1})
perldl> subsplit_bcd({CH=>2})
perldl> subsplit_bcd({CH=>3})
perldl> subsplit_bcd({CH=>4})
Note: This routine also places the original _bcd.fits, _bdmsk.fits, and
_bunc.fits image packs in .reduce_tmp/.
#-----------------------------------------------------------------------------
# Wrapper/Log For Batch Processing Steps 2 and 3
#-----------------------------------------------------------------------------
After running 'fixup' in step 1, there will be a perlDL wrapper file called
'my_iracproc.pdl' in the reduce/ sub-directory. This is useful for batch
processing steps 2 and 3 for all 4 IRAC channels at once, and will remain a
log of your data processing. Simply use a text editor to modify the file
accordingly for each data set. Then, 'cd' into reduce/, start perlDL and type
'my_iracproc()' at the prompt. A duplicate of the original, un-modified file
is also placed in the Lists/ sub-directory.
#-----------------------------------------------------------------------------
# Processing Step #2 - Make Master Image Lists
#-----------------------------------------------------------------------------
Change dir to the target that you wish to process and start PDL. To view
options type 'mkmopex_bcd()' at the perlDL prompt. To view information file
type 'help mkmopex_bcd' at the perlDL prompt, or see
IRACproc/docs/mkmopex_bcd.html for additional information.
user_name][ cd proc_dir/reduce/
user_name][ perldl
Make the initial list of images to be processed:
perldl> mkmopex_bcd('../Lists/list.par',{FOV=>'all',FT=>30,CH=>1})
perldl> mkmopex_bcd('../Lists/list.par',{FOV=>'all',FT=>30,CH=>2})
perldl> mkmopex_bcd('../Lists/list.par',{FOV=>'all',FT=>30,CH=>3})
perldl> mkmopex_bcd('../Lists/list.par',{FOV=>'all',FT=>30,CH=>4})
Some typical parameters for selecting images (matching is exact):
FOV field of view: 'on', 'off', 'sub', or 'all' (default)
CH IRAC channel: '1', '2', '3', '4'
FT frame time (default 'all')
XT exposure time (default 'all')
AO AORID (default 'all')
WCS_COORDS WCS coordinates (deg) where images are centered - [RA,DEC]
WCS_SIZE WCS box size (angular width in deg) where images are located,
automatically pads region to include all images that may
have pixels within selected region - [RA_SIZE,DEC_SIZE]
Each command produces the file Lists/bcd_I<CH>.lis. You may remove images
from the processing by removing them from the image list: Lists/bcd_I<CH>.lis
(this can be automated by following the instructions in HOWTO_INSPECT). You
may alter the BCD frames with additional preprocessing, however, two
guidelines must be followed: 1) the image names must be identical to those
originally placed in Lists/bcd_I<CH>.lis, and 2) the image list bcd_I<CH>.lis
must retain its name and location.
#-----------------------------------------------------------------------------
# Processing Step #3 - Make Mosaic Images w/ Outlier Rejection
#-----------------------------------------------------------------------------
Create a mosaic of the images in Lists/bcd_I<CH>.lis. To view options type
'mkmosaic_bcd()' at the perlDL prompt. To view information file type
'help mkmosaic_bcd' at the perlDL prompt, or see
IRACproc/docs/mkmosaic_bcd.html for additional information.
perldl> mkmosaic_bcd({NAME=>'',MASTER_FIF=>0,STEPS=>[1,1,1,1,1,1],CH=>1})
perldl> mkmosaic_bcd({NAME=>'',MASTER_FIF=>0,STEPS=>[1,1,1,1,1,1],CH=>2})
perldl> mkmosaic_bcd({NAME=>'',MASTER_FIF=>0,STEPS=>[1,1,1,1,1,1],CH=>3})
perldl> mkmosaic_bcd({NAME=>'',MASTER_FIF=>0,STEPS=>[1,1,1,1,1,1],CH=>4})
Quit PDL - you will need to restart for the next target.
perldl> quit
By default, copies of the final mosaics and coverage maps are placed in
analysis/ with the filenames: I<CH>_<NAME>_mosaic.fits and
I<CH>_<NAME>_mosaic_cov.fits.
----------------
Some typical options for making mosaics:
NAME The string placed in the default output image filenames
I<CH>_<NAME>_mosaic.fits and I<CH>_<NAME>_mosaic_cov.fits
- default 'IRACproc'
CH IRAC channel: '1' (default), '2', '3', '4'
STEPS Set of six logical flags to turn on/off stages of processing
ALL SIX STEPS MUST BE RUN AT LEAST ONCE
- [STEP1,STEP2,STEP3,STEP4,STEP5,STEP6]
- defaults are 1 (yes)
STEP1 = run overlap correction
STEP2 = create median mosaic
STEP3 = run 1st layer outlier rejection
STEP4 = create metric images
STEP5 = create final mosaic (w/ 2nd layer outlier rejection)
STEP6 = copy final mosaic to analysis/ subdirectory
# FIF options:
MASTER_FIF Set this logical flag to use a master Fiducial Image Frame
no = 0 (default), yes = 1 (by default created from bcd_I1.lis,
bcd_I2.lis, bcd_I3.lis, and bcd_I4.lis)
FIF_FILE Set this to use an existing master Fiducial Image Frame
(the default location/file is ../Lists/FIF.tbl)
FIF_IMG_LIST Set this to use an existing image list for creating the
master Fiducial Image Frame
(the default location/file is ../Lists/bcd_ok.lis)
CROTA2 Position angle of N in final mosaic, CCW from +y-axis
'A' (average, default), '0.' for North up
PIXEL_SIZE Mosaic pixel size in arcseconds (default is 0.86267016)
# First layer outlier rejection options:
CR_FLAG Set this logical switch to flag outliers in BCD Dmask images
(_bdmsk2.fits), otherwise routine only clears bit 9 in Dmask
- default = 1 (yes)
To remove the unnecessary files in an IRACproc processing directory:
--------------------------------------------------------------------
user_name][ cd /<data_path>/
user_name][ cleanup dir
USAGE: perl cleanup.pl [--help][--ver][--dirlist file][dir1 dir2 ... dirN]
----------------
Higher Coverage Mosaics:
------------------------
If your data set is such that the depth of coverage is at least 5 frames at
every sky location, you may choose to skip the intermediary steps that
perform a first pass at outlier rejection in order to expedite processing by
setting CR_FLAG=>0. If however your data set has two epochs of observation
(e.g. at different orientations), then you will want to run the the first
layer of outlier rejection since this layer is designed to work with bimodal
distributions in the pixel stack (CR_FLAG=>1, default).
Low Coverage Mosaics:
---------------------
The ability of the first layer of outlier rejection (crflag_bcd) to
effectively remove transients is dependent on the pointing refinement
available for each channel. This is typically much worse for channels 3+4,
and can negatively affect coverage on-source in shallow coverage mosaics if
the pointing refinement is not used (or not very good). The S13.2 pipeline
has significantly improved the pointing refinement for channels 3+4, and
IRAC users should download a new copy of their AORs processed with the latest
version of the pipeline.
Maps (w/o dithering) & Latents:
-------------------------------
If your data set is a map that was produced by stepping the array by a large
fraction of a frame without dithering the telescope in between steps, then
you may want to 'turn off' the masking of latents (bit 5 in the Dmask). The
short term latents of bright stars are picked up by the SSC pipeline
processing and flagged in the Dmask (bit 5) of subsequent frames. The
repetition of the mapping causes these short term latents to overlap in sky
coordinates before they have time to fully decay, and so they will literally
punch holes in the final mosaic. This effect is caused by the absence of
dithering, and is not a consequence of shallow coverage depth; that is,
repeating each map position many times will not by itself prevent these
overlapping latents. This happens irregardless of any outlier rejection, in
other words, adjusting the outlier rejection thresholds will have no effect.
This can be avoided by modifying the following parameter files, in
Lists/: mosaic.par, overlap.par, median.par, and minabs.par. You would need
to change the DCE_Status_Mask_Fatal_BitPattern by subtracting 32 (=2^5).
Background Gradients In Large IRAC Mosaics:
-------------------------------------------
Most, and possibly all, large IRAC mosaics suffer from significant,
artificial background gradients, affecting the measured fluxes and colors for
extended emission in all bands. This effect most likely arises from small
zodiacal background gradients over the IRAC array, or imperfect flat-fields
and geometrical distortion corrections. Although automated background
matching is performed as part of the IRACproc processing, it only corrects
for constant pedestal offsets for each frame. As a result, small gradients
build up over large mosaics manifesting as a position dependent zero-point.
To remove the background gradients a surface can be subtracted from each
mosaic. The surfaces can be constructed by trend-fitting the pedestal offsets
for each frame, as output by the MOPEX overlap correction module. The notes
located in IRACproc/docs/BKGFIX_GRAD_NOTES describe one method for finding
and removing the gradients in a way that does not add or subtract any net
flux from the mosaic.
In addition, the notes located in IRACproc/docs/BKGFIX_NOTES describe one method
for calculating and subtracting a dark frame from an IRAC mosaic data set.
There is also a method for fixing erroneous offset corrections from MOPEX's
overlap correction module around extremely bright stars.
Bright Stars:
-------------
If you have a very bright star (e.g. 1st mag, or brighter) and have only a
single IRAC field-of-view, then set BACKGRND=>0.
No Dithering/Mapping:
---------------------
If your data is neither dithered nor mapped, then DERIV_SCALE should be set
to a low value, for example <0.1.
Adjusting Outlier Rejection:
----------------------------
If you find that outlier rejection has not been aggressive enough, then
reduce the options SCALE_THRESH and/or SCALE_CR_THRESH and rerun
mkmosaic_bcd. Be mindful of the coverage of the final mosaic (in the sky
background, as well as on top of point sources)
- check analysis/I<CH>_<NAME>_mosaic_cov.fits.
If you find that outlier rejection has been too aggressive, then increase the
options SCALE_THRESH and/or SCALE_CR_THRESH and rerun mkmosaic_bcd.
Persistent Transients:
----------------------
The mosaic.fits, mosaic_cov.fits, median.fits, and
absolute_minimum_mosaic.fits (if created with CR_FLAG=>1) files can be
checked and compared across channels for suspected transient events that
persist into the final mosaic image.
The final mosaic and coverage images are located in: mosaic_I<CH>/Combine/
The median image is located in: median_I<CH>/Combine/
The absolute minimum mosaic image is located in: minabs_I<CH>/Combine/
----------------------
You may notice the following warning scroll on the screen:
mkmopex_bcd: WARNING cannot find file I<CH>_########_####_0_1_bmet.fits
This is normal when running IRACproc for the first time on a set of images.