from:	Jun-Hui Zhao 
to:	Holly Sarah 
date:	Jul 16, 2018, 5:58 PM
subject:	Re: smamiriad account home directory in Hilo exceeds 10G limit

HI Holly,
      Thanks for collecting the questions. As we briefly discussed on last Friday, the swarm2casa is
new software to pipeline the wideband data produced from the upgraded SWARM correlator to CASA
environment, the first step of building next generation data-reduction software to handle SMA wideband
data by integrating astro-statistical/calibration/imaging algorithms as well as astro-information collecting from
staff testing and data/users' source models as well as more intelligent algorithms/methods. The current version
is a serial code in C, Fortran, and C++ of CASA executed with python modules using a single CPU unit.

After building workable software architecture populated with the detailed coding , much of my efforts so far in the
past a few months have been made in the following aspects:

1) optimizing the i/o routines of extracting from the SMA archived SWARM wideband data  and streaming them out.
2) utilizing the FITS standards to build data stream adaptable for CASA and converting them into measurementSet
3) gone through a whole month of data sets produced in January of 2018 to classify the online data issues and developing
   a more intelligent software to automatically correct the data errors. The on-line data readable rate in 2018 appears to
   have been improved as compared to that of the 2017 data but still having a significant amount of data files are subject to data issues. 
4) optimizing setup for inspection and examination of the header information and details of visibilities as function of
   various variables with the routines and GUI of CASA based on the limit bandwidth across network.
5) testing server/system setup for the configuration to execute/operate swarm2casa software.
    The RTDC system appears to be excellent for the new software. The efficiency of using CPU with the server rtdc9 appears
    to be nearly 100% for the processes coded in C, Fortran and C++. With next multiple-threads upgrade, we will improve the efficiency
    of using the CPU units available in the system, speeding up the data reduction. We still have serious issues to run this new software
    with the public servers on the Hilo computer system.
  
    These are all lower level tests. Except for examining head information and general inspecting the fringe visibility and uv distribution,
I haven't got chance to go through the high level issues on calibration and imaging. We are looking forward to having feedback
from SMA science staff concerning these issues. 

    Actually, I have setup Users/testers Feedback link on the bottom of the web page :  https://www.cfa.harvard.edu/~jzhao/SMA-CASA/swarm2casaRelease.html
I put my email address there for the purposes of collecting questions, bug reports and suggestions for this first version. Please
encourage users to use this path to feedback their opinions. As future upgrading, RTDC may create information archive and web-link
to collecting information from users and SMA staff.

     Regards,
     Jun-Hui