CHELLEPROC

Last Update: 2013-02-04 by M Conroy
Email Comments & Bug reports

The HectoChelle pipeline is intended to process data (i.e. "science") files and calibration files to the point where individual spectra from the fibers are extracted, normalized and wavelength calibrated. If appropriate, a few standard velocity correlations are also run. Thus, the spectra are ready for for analysis by the PI.

The pipeline also performs summaries of the data to allow the user to assess the quality of the data. At the moment, the pipeline processes data files only. The processing of skyflats and biases is still under construction.

The pipeline can be run completely automatically, i.e. without user intervention. However, to selectively process observations is probably the more usual use of the pipeline.

The code makes use of a HectoChelle reference directory, normally the same for all users on a given domain. This is referenced via the HPATH variable, and is set before the pipeline code is started. The reference area contains the various Thorium-Argon line lists, wavelength templates, velocity templates, etc., and the pipeline code itself.

The pipeline code consists of a series of bash shell scripts that call Starbase and IRAF programs. The IRAF tasks are accessed through IRAF's host CL scripting capability. The IRAF parameters used in each routine are explicitly set from the dparm subdirectory in the reference area and are saved in the reduction subdirectory as documentation of their values.

The data are normally initially selectively copied from the HectoChelle raw data archive, /home/hecto/CHELLE/rawdata/data/ARCHCHELLE, but can be copied from somewhere else or run on data in an existing subdirectory.

CHELLEPREP - Prepare to run the pipeline

Set HPATH to the location of the Hectochelle reference directory:

If your shell is csh or tcsh, type:

    
        setenv HPATH /home/hecto/ChellePipe

If your shelle is sh, bash or ksh, type:

        HPATH=/home/hecto/ChellePipe
        export HPATH

cd to your reduction area, e.g. /data/heproc/hproc or anywhere you can create and write into a subdirectory.

Use chelleprep to check files from the Hectochelle archive you may want to reduce. It is a good idea to make your terminal window longer and wider, perhaps 60 by 132 or so.

    $HPATH/chelleprep [options] [path] 2006.0407

Example:

You are in your reduction area, in subdirectory. To run the processing on the data from April 6, 2006, type

      $HPATH/chelleprep -a 2006.0407

The -a option means take data only from the archive, in case there are any subdirectories named 2006.0407 in the current directory. The above command outputs a file summary, in UT time order (file names have been shortened to make it fit here):

       105 FITS files in /home/hecto/CHELLE/rawdata/data/ARCHCHELLE/2006.0406
         FILENAME       CfgCksum   PI      FILT BIN Exp     UT-OBS
         -------------- ---------- ------- ---- --- ------  --------
         domeflat.0979  3339178378 Mathieu RV31 1 1   40.0  02:44:59
         domeflat.0980  3339178378 Mathieu RV31 1 1   40.0  02:47:54
         comp.0981      3339178378 Mathieu RV31 1 1  240.0  02:52:17
         ngc2506_1.0982 3339178378 Mathieu RV31 1 1 1200.0  03:30:18
         ngc2506_1.0983 3339178378 Mathieu RV31 1 1 1200.0  03:52:23
         ngc2506_1.0984 3339178378 Mathieu RV31 1 1 1200.0  04:14:09
         ngc2506_1.0985 3339178378 Mathieu RV31 1 1 1200.0  04:36:58
         comp.0986      3339178378 Mathieu RV31 1 1  240.0  04:59:16
         domeflat.0987  2012819680 Carney  RV31 1 1   40.0  05:30:09
         comp.0988      2012819680 Carney  RV31 1 1  600.0  05:34:09
         m5_m_1.0989    2012819680 Carney  RV31 1 1  761.0  06:05:12

The chelleprep program by default does not show any files with test, focus, sky or sflat in the filename. The full count of FITS files is given; 105 in the above example. Calling chellprep with the the -l argument lists all FITS files, used with the calibration reduction processing [still under construction].

The chelleprep program then asks if you want to preceed to divide up the files into processing set, i.e. sets with the same filter, binning, and fiber positioner configuration. Unless you've chosen the wrong date or see some other problem, normally the answer is yes.

The next phase identifies file sets whose configuration, filter and binning are identical. The user is asked if to process each set. If the answer is no, the program proceeds to the next set. If the answer is yes, the files in that set are copied to a subdirectory in the user's reduction area.
For example, if the date is 2006.0407, and the data set is:
```
    FILENAME                PI(*)         FILT   BIN   Exp     UT-OBS
    ---------------------   -----------   ----   ---   ------  --------
    domeflat.1086           Szentgyorgi   RV31   2 3      4.0  04:19:20
    comp.1087               Szentgyorgi   RV31   2 3     90.0  04:20:35
    NGC2419_merged_1.1088   Szentgyorgi   RV31   2 3   2200.0  04:27:26
    NGC2419_merged_1.1089   Szentgyorgi   RV31   2 3   2200.0  05:04:30
    NGC2419_merged_1.1090   Szentgyorgi   RV31   2 3   2200.0  05:41:34
    comp.1091               Szentgyorgi   RV31   2 3     90.0  06:20:15
```
the data files (*.fits, *.cfg, *.cat, *_map) are copied to a local directory whose path is made up of the date, filter plus binning, and object name. In the above example, this is ./2006.0407/RV31_2x3/NGC2419_merged_1.
Also copied into the directory are copies of the default pipeline control parameters, chelleproc.par and the pipeline script itself, chelleproc.
(*) Andy's name is mispelled that way in the actual file headers.

CHELLEPROC - run the data reduction

When all sets have been either copied into subdirectories or skipped, the program asks if you want to proceed to the chelleproc program to start reducing each of the data sets. For the present, this will only do one set at a time, but a multiprocessing version will be available when the chelleproc program is better verified to be correct.

If you want to manually run the chelleproc program, cd into the data directory, e.g.

  
      cd 2006.0407/NGC2419_merged_1

then execute:

  
       ./chelleproc

I think it's a very good idea to pipe the standard output to the tee program to both see it and save it to a file:

  
        ./chelleproc |& tee chelleproc.log       (csh, tcsh)
   or
        ./chelleproc 2>&1 | tee chelleproc.log   (sh, bash, ksh)

If the proper flat and wavelength templates are available, you may not have anything more to do other than watch the output. At the finish, the results are transferred into a RESULT_{objectname} subdirectory. In there, the "split" files, that is the individual spectra, are in a directory with a name of the form 1d.{objectname}.
Files from intermediate stages of processing are saved in directories of the form Un..., e.g. Undisp has files before dispersion correction, Unnorm has the files from before normalization, etc. These can be useful to check on the processing, but take up a 3/4 of the final space used, and should be deleted once the reduction is accepted.