1. Using pyuvdata   (ALPHA VERSION)
2. Using MIR mir2ms   (BETA VERSION)
3. Using MIR autofits   (RECOMMENDED)
4. Using sma2casa.py   (NO LONGER SUPPORTED)

1.   Using pyuvdata

IMPORTANT An alpha version of pyuvdata is available on the RTDC. We encourage users to make use of this and report any issues to garrett.keating@cfa.harvard.edu.

Pyuvdata is a python interface to interferometric datasets. It allows the conversion of datasets from one format to another with multiple data formats supported. Here we give the example of converting SMA data from raw MIRIAD format to UVFITS and CASA measurement sets.

• Retrieval
  The package can be downloaded using pip or conda. If you are working on the RTDC this is already installed.

  $ conda install -c conda-forge pyuvdata

• Status
  [Oct 2021] In testing. Currently pyuvdata only supports a single receiver, as a result polariaztion data is not yet supported. Additionally the data are not being Tsys corrected at this time.

• Instructions
  
  1. If running on the RTDC you must activate the conda pyuvdata environment.

     $ conda activate pyuvdata

     Note that you can see a list of available environments using

     $ conda env list

  2. Follow one of the templates below to create an executable script. Omit the cwd elements if you want to define an explcit path for the output file/directory. The irec value sets the receiver to be used, where 0 = 230, 1 = 345, 2 = 400 and 3 = 240.

     To UVFITS

     import os
from pyuvdata import UVData


     cwd = os.getcwd()
UV = UVData()
UV.read_mir("/sma/data/flux/mir_data/210808_13:37:36", isource=[1], irec=2, isb=[1], corrchunk=[1])
UV.write_uvfits(cwd+"/210808_133736_rx400.uvfits", spoof_nonessential=True)

     To measurement set

     import os
from pyuvdata import UVData


     cwd = os.getcwd()
UV = UVData()
UV.read_mir("/sma/data/flux/mir_data/210808_13:37:36", isource=[1], irec=2, isb=[1], corrchunk=[1])
UV.write_ms("cwd+"/210808_133736_rx400")

  3. Read the output file/directory into CASA.

     UVFITS file

     CASA: importuvfits(fitsfile='210808_133736_rx400.uvfits', vis='210808_133736.vis')
CASA: ms.open(vis)

     Measurement set directory

     CASA: vis='210808_133736_rx400'
CASA: ms.open(vis)

2.   Using MIR mir2ms

IMPORTANT A beta version of mir2ms is available with MIR. mir2ms is a MIR task that converts a raw (uncalibrated) SMA dataset directly to CASA measurment set in IDL. We encourage users to report any issues to Charlie Qi (cqi@cfa.harvard.edu).

• Retrieval
  The script comes packaged with the June 2021 version of MIR. This is available on the RTDC or you can find it on the MIR github page sma-mir.

• Status
  [Oct 2021] Header informaion on the gunnLO is currently incompatible with mir2ms; there is a work-around described below. Currently only a single receiver is supported.

• Instructions
  
  1. Create a .pro script in your current working directory; in this example it has been named mymir.pro (remember that the filename must match the name given to the 'pro' in the first line of the file). There are two required tasks in this file - applying the Tsys correction and fixing the problem data header. You can optionally provide instructions to perform some basic data cleaning (e.g. flagging pointing scans & spikes). The mymir.pro script should follow this template.

     As part of the routine, mir2ms utilizes autofits (see 'Using MIR autofits' below) to generate UVFITS files per source, sideband, and chunk. Unlike autofits, mir2ms opens your local version of CASA then imports and concetenates the UVFITS files automatically.

     WARNING  This routine creates over 200 temporary files and directories in your cwd which will require ~ 5x the disk space of the input data directory.

  2. Run mir2ms in MIR.

     IDL> mir2ms, dir='210808_13:37:36', rx=230, /mymir, outname='210808_133736'

     Then wait a long time. When the script has finished, exit IDL and you will find the temporary files have been deleted. You will be left with an outname_rx.ms directory in your cwd along with IDL and CASA log files.

3.   Using MIR autofits

This option converts data already calibrated in MIR to CASA measurement set format. The data are written out in UVFITS format from MIR, then the provided script must used to import it to CASA.

• Retrieval
  You can find the CASA import script at MIRFITStoCASA.py This script is used in place of CASA's importuvfits procedure which does not propagate the weights correctly from MIR.

• Status
  [July 2019] A bug in fits_out (found inside the MIR autofits routine) has been fixed. This caused each chunk and sideband to have slightly different uv coordinates (meters) in casa. This problem meant users saw a narrower u-v coverage per baseline, and got u-v coordinates wrong by a few percent.

  [Mar 2018] MIR's fits_out routine (called by autofits) now outputs correct sky frequency header information for continuum UVFITS files. Spectral data are unaffected.

• Instructions
  
  1. Create the UVFITS file in MIR by using the autofits routine. This will loop over all chunks, sidebands and receivers on a source-by-source basis. In this example our source is named orion.

     IDL> select,/p,/re
IDL> autofits, source='orion'

     after providing your source name, autofits creates a separate file for each SWARM chunk (s1-s4), sideband, and receiver. The files are named with the following convention SOURCE_SB_CHUNK_RX.UVFITS   (e.g. ORION_L_S1_RX345.UVFITS, ORION_U_S4_RX230.UVFITS).

     For a typical SWARM data set you will get 16 spectral files, along with 4 extra files for the pseudo-continuum chunks, C1, which can be ignored.

  2. Next switch to CASA and use the provided script MIRFITStoCASA.py to import your data. This script loops over all the .UVFITS files and converts each of them to the measurement set (.ms) format.

     CASA: import sys 
     CASA: sys.path.append('/path/to/MIRFITStoCASA.py/') 
     CASA: import MIRFITStoCASA 
     CASA: fullvis='allorion.ms'
     CASA: allNames = []
     CASA: for sou in ['ORION']:
     CASA:    for rx in ['345','240']:
     CASA:       for sb in ['L','U']:
     CASA:          for i in ['1','2','3','4']:
     CASA:             name = sou+"_"+sb+"_S"+i+"_RX"+rx
     CASA:             print("------converting "+name+" ....")
     CASA:             MIRFITStoCASA.MIRFITStoCASA(UVFITSname=name+'.UVFITS', MSname=name+'.ms')  
     CASA:             allNames.append(name+'.ms')
     

  3. Next concatenate all the newly created measurement sets into a single file. The input here is allNames which is the python list of .ms files created in step 2. The name of the concatenated output file (fullvis) was also defined in step 2.

     CASA: concat(vis=allNames,concatvis=fullvis,timesort=True)

  4. Check the content of the final, concatenated measurement set:

     CASA: listobs(fullvis)

  5. Flag the noisy edge channels.

     CASA: flagdata(vis=fullvis, mode='manualflag', spw='*:0~nflagedge;(ntotal-nflagedge)~(ntotal-1)')

     where nflagedge should be substituted with the integer number of channels you want to flag from the edges, and ntotal should be substituted with the integer number of channels per chunk (or spw in CASA). ntotal can be found from the previous listobs command.

     The choice of how many edge channels to flag can be made by looking at the amplitude behavior as a function of frequency in each chunk, e.g. by using CASA's plotms function. Note that this can be slow if a lot of channels are present and/or a lot of tracks have previously been combined:

     CASA: plotms(vis=fullvis, xaxis='channel', yaxis='amp', avgtime='1e20', avgscan=True, iteraxis='spw')

     This will show increased noise in the edge channels. We advise a conservative trim of about 8% (so that nflagedge~0.08ntotal).

  6. Repeat the loop to create a new measurement set for another source. Alternatively, add a second source name to the for sou loop in step 2.

3. Using sma2casa.py

BE AWARE: This routine converts raw, uncalibrated SMA only. It is untested with newer versions of CASA and should not be considered reliable. A new routine for converting raw SMA data is under development.

Two python scripts are used for the conversion - sma2casa.py and smaImportfix.py. This section describes the most common use case, but a more detailed description, including a full list of options, can be found at sma2casa Details.

• Retrieval
  You can retrieve the python scripts from the github software repository.

  $ git clone https://github.com/kenyoung/sma2casa.git

• Status
  The system temperature correction is applied by sma2casa.py. However, it is unable to create a Tsys table that is recognized as such by CASA. In CASA, you can still view the system temperature data and check the values.

  Warning: SMAImportFix.py only works with versions of CASA older than 4.7.

• Instructions
  This sequence of instructions is tailored to RTDC machines running tcsh. The details will need to be modified to match your system.
  1. Check your current version of python.

     $ python --version
     

  2. If it isn't anaconda2, add the location to your path.

     $ setenv PATH /usr/local/anaconda2/bin/:$PATH

  3. Run sma2casa.py. The makevis.c amd makevis.so files will need to be in the same location as sma2casa.py. See sma2casa Details for a full description of the options.

     $ sma2casa.py 090202_07:19:01 -l -t

     sma2casa will produce a directory for each chunk and each sideband (e.g. Lower_s00, Lower_s01 ... Lower_s48), along with a file called sourceTable.

  4. Copy smaImportFix.py to your current working directory.

     $ cp /usr/bin/sma/smaImportFix.py.
     

  5. Run CASA (4.2).

     $ /opt/casapy-42.0.28322-021-1-64b/casapy
     

  6. Start to follow the steps outlined in this example script. The relevant line for calling smaImportFix.py is:

     CASA: execfile('smaImportFix.py')

     smaImportFix.py recognizes the presence of the Lower_sXX files created by sma2casa.py and creates a directory called MyDataLower (or MyDataUpper if dealing with the upper sideband).

     You may encounter FitsIDItoMS() warnings, or ones reporting "No valid pointing tables present", but these are benign and wont prevent the data being processed through to mapping.

  7. Read MyDataLower (for this example) into CASA and continue processing.

     CASA: vis ='MyDataLower'
CASA: ms.open(vis)
CASA: listobs()