Try out the SWARM2CASA pipeline for SMA users

• 2018-08-06 at 10:00am HST and 4:00pm EDT


• Agenda:
• 1. Status and early tests
Software architecture (v1.0.0)
• 2. Version 1.0.0 and setup on HILO servers
• Called for internal test (July 6, 2018)
• Mirror the installation on Hilo system -
Copied over from RTDC in Cambridge
• Location -
/sma/local/opt/sma/SWARM2CASA/
• CASA -
CASA 5.3.0 copied over from RTDC installation, located outside the pipeline package: /sma/reduction/jzhao/SMA2CASA_TESTING/casa-release-5.3.0-143.el6
changed the symbolic link under /sma/local/opt/sma/SWARM2CASA/casabin
• 3. Demo & Testing script
cp /sma/local/opt/sma/SWARM2CASA/script/HILO/swarm2casa.csh180205 ~/swarm2casa.csh
10 GB data of 180205_04:06:54,
hilodr1: realtime 1.5hr, not available since test on Saturday (8/3)
hilodr2: realtime 1.5hr, available since the test on Sunday (8/5)
hilodr3: server seems to be not on the network, no tests can be done for the new CASA pipeline setup since (8/3)

• Issues to test -
• Public servers to test -
hilodr1, hilodr2, hilodr3
• Performance issues -
• i/o for swarmRead (C): good (100% CPU)
• i/o for FITS stream out (Fortran): poor in (10~??% CPU)
• CASA importfits: OK (100% CPU when running) but hiccup with 1.5min pause in CPU ??
• CASA concat: poor with hiccup pattern (running 30s -> pause 60s -> ...) ??
• CASA listobs: improved, the report instantly sent over
• CASA plotms: improved, GUI performance like local
• Test (August 3) report on 180506_03:52:53, about 30 GB data
Taking 4hr to get the all data into CASA measurementSet on the server hilodr1, about 4 times longer than rtdc9 on RTDC/RG.
• Tests for hilod1 (August 3) and hilod2 (August 5) reported on 180205_04:06:54, about 10 GB data
Taking 1.5 hr to get the all data into CASA measurementSet on the Hilo servers, about 5 times longer than rtdc9 on RTDC/RG.
The performance on Hilo systems has been improved but the issue on being a factor of 4~5 slow needs to be resolved.
• 4. Tested items
• Optimizing lower level I/O (C)
• FITS standard data streams adaptable to CASA (Fortran)
• Python module to execute CASA routines (C++)
• CASA
• Listobs for headers¹
• GUI plotms for variables & interactive editting
editing (flagging) spectral spikes -
spectra -
uv-coverage for calibrators and targets -
fringe amplitude vs uvwave -
fringe phase vs uvwave -

• Issues
SMA antenna coordinates (right-handed geocentric system) -
4.7.0 seems to interpret it correctly -
The new versions 5.0.0, 5.1.0, 5.3.0 cann't -

Requesting CASA/NRAO for helping out the SMA antenna coordinate issue²
• 5. Further test & to do
• CASA
• Calibration and imaging
Need to try out actual data from SMA science observation programs
Reqeust internal scientists for end-to-end reduction of SMA data with the pipeline and CASA³.
Geoff: Can you provide a data set from recent SMA observation (PI: Bower)? Use the data to test calibration/imaging with CASA.
• SMA Pipeline
• swarmChecker.c
• to classify the online data issues, providing more intelligent software to patch the data
  went through all data files taken in January 2018, the readable rate seems to be improved as compared to the rate of 50%; with implemented data patches, January data (2018) should be all readable, excluding very small volume data files.
• as looking for recent data file prepared for this meeting (8/6) and a report from RTDC Holly Thomas (8/1), we noticed that the data files since May/June (?) are not able to read with the new pipeline. We need to work with the online programmer Attila Kovacs (?) to find out what has been changed on the online data acquisition program, in particular, the format in the header tables.
• March/April had observations for some specific purposes. The hid ac patches are needed to repare these files.
• collection of testing information and calibrator models
to correct and calibration via the pipeline
_______________________
¹Tested versions: 4.7.0, 5.0.0, 5.1.0, 5.3.0
²The issue of antenna header variables does not affect on calibrations and imaging, but correct coordinates are needed in baseline correction for re-calculating uv coordinates. We checked antenna header variables written in the FITS streams with AIPS. AIPS interprets them correctly.
³We may need CASA/NRAO to adapt SMA table for Tsys and implement/adapt SMA specific frequency-weighting parameters or mechanisms in various calibration/imaging program routines of CASA.