from:	Jun-Hui Zhao 
to:	Geoffrey Bower 
cc:	Thomas Mac Cooper 
bcc:	"Moran, Jim" 
date:	Aug 6, 2018, 11:41 AM
subject:	Re: swarm2casa conversion software

Hi Geoff,
     Here is a link for the agenda that we may go through in today's meeting:
https://www.cfa.harvard.edu/~jzhao/SMA-CASA/testmtg2018/swarm2casaTmtg080618.html
What is your number that I can call you at 10am HST? or you can call me office at 617 496 7895.
We can start our two-party meeting to discuss the agenda listed. A few notes below that I would let you know
before the meeting:

1) the new pipeline software are copied over from RTDC along with latest publicly released CASA 5.3.0.   
    The setup has been made on Hilo computing system and I tested the three public servers there,
    hilodr1, hilodr2 and hilodr3. hilodr3 seems to be not on line; hilodr1 was running through my test
    on Friday (8/3) but not available since Saturday (8/4). Only hilodr2 seems to work. I did
    a test last night and it successfully went through my test.
    The installation has been on /sma/local/opt/sma/SMA2CASA, the disk hosts SMA offline software.

2) I noticed that the network transaction via my account (jzhao on hilo system) to my desktops at both my office
    and home has been improved. But the issues on the i/o performance on the servers remain; specifically
    for the C and C++ programs from both the new pipeline and CASA, the efficiency of using CPU units
    appears to be normal (nearly 100%) but involving Fortran programs, the efficiency appears to be still
    poor (~10%). As a result, the process of reading SMA online data and converting all into CASA measurementSet
    takes much longer time on the Hilo servers, a factor of 4~5 longer than the RTDC server rtdc9.
    The typical time of running through the new pipeline on rtdc9 is ~1 hr for 30 GB data sets, ~15min for 10 GB files.

3) Lately, the online programmer may have made some changes on the format of header tables. Most of files
    appear to need patches. I found a small volume data set 180205_04:06:54, 10 GB. For hilodr2, which is readable.
    It takes 1.5 hr to go through the pipeline processes. The default setup for hilo is using the big data 30GB taken on
    May 6, 2018. We will this 10 GB data as a demo for today. The PI of the data is Jan Forbrich; we use it only for
    the purpose of our demo for today. The setup, swarm2casa.csh180205,  is also available.  You may copy over the C-shell script:
         cp /sma/local/opt/sma/SWARM2CASA/script/HILO/swarm2casa.csh180205 ./swarm2casa.csh
   
    and start running on hilodr2 before our meeting:
        ./swarm2casa.csh

4) The purpose of this meeting is to start next step test: calibration/imaging SMA wideband (SWARM) data
    on CASA. The slow i/o on Hilo servers as well as the online data issue will make our next-step test harsh
    on the Hilo system.
    Please try out this new pipeline and calibrating/imaging your SMA wideband data on CASA.

    Regards,

     Jun-Hui
Geoffrey Bower
	
	Aug 6, 2018, 2:09 PM
Hi Jun-Hui, My office phone is 808 961 2945. Talk to you at 10. I think Mac will be able to join. Cheers, Geoff
Jun-Hui Zhao
	
	Aug 6, 2018, 3:18 PM
Great! I will call this number at 10am HST. Jun-Hui
Geoffrey Bower
	
	Aug 6, 2018, 4:20 PM
Mon Aug 6 10:12:22 HST 2018 Start swarm2casa.csh script - SWARM2CASA pipeline version: SMA-SWARM2CASA 1.0.0 (Smithsonian Astrophysical Observatory) loaded from
Jun-Hui Zhao 
	
Aug 6, 2018, 4:45 PM
	
to Geoffrey
Hi Geoff,
     Do you source .cshrc while you login? If so, would you please send me a copy
of this file? Or any file sourced for the  environment setup while you login. I need it to generate the same error as you encountered.
     Thanks,
     Jun-Hui
Geoffrey Bower 
	
AttachmentsAug 6, 2018, 4:44 PM
	
to me
Attachments area
Preview attachment .cshrc
[Binary File]
Jun-Hui Zhao
	
	Aug 6, 2018, 4:55 PM
Thanks, I got it. Jun-Hui
Jun-Hui Zhao
	
	Aug 6, 2018, 5:12 PM
I repeated the error with your .cshrc. I will back to you as soon as I have a solution. Jun-Hui
Jun-Hui Zhao
	
	Aug 7, 2018, 2:50 PM
Hi Geoff: Thanks for your .cshrc. The library issue that you found yesterday is due to my end. I develop the new software with my desktop that was used for the
Thomas Mac Cooper
	
	Aug 7, 2018, 3:56 PM
I have had a look at the virtual machines hilodr1,2,3 and noticed that all three are only using the public network for access to the data. I'll start working to
Geoffrey Bower
	
	Aug 7, 2018, 4:02 PM
Hi Jun-Hui, I launched your new version of the script and it appears to be running properly, if slowly. As a benchmark, here is a snippet of the output: inhset=
Jun-Hui Zhao
	
	Aug 7, 2018, 6:01 PM
Great, Geoff. You have made a critical step forward! Can you try one of your recent data? The data taken in the period of the first two months in 2018 appear to
Jun-Hui Zhao
	
	Aug 7, 2018, 6:03 PM
Thanks, Mac. We are looking forward to trying out the 10G network. Jun-Hui
Geoffrey Bower
	
	Aug 7, 2018, 9:31 PM
Hi Jun-Hui, I was able to load this data set set INPUT_FILE_NAME = 180517_05:16:39 It looks ok in listobs but most of the data appears to be flagged for no reas
Geoffrey Bower
	
	Aug 7, 2018, 10:27 PM
One other attempt on 180613_03:41:04 crashed after the following messages: [gbower@hilodr2 SMA]$ ./swarm2casa.csh Tue Aug 7 15:54:55 HST 2018 OUTPUT_FILE_PREFIX
Jun-Hui Zhao
	
	Aug 8, 2018, 12:45 PM
Hi Geoff, The default assumes that the online flagging is good and the pipeline applies the flagging states to the data while converting to CASA measurementSet.
Geoffrey Bower
	
	Aug 8, 2018, 2:34 PM
Thanks, Jun-Hui. I will explore whether there are other good data sets that we can test. Do you know if there is a way in CASA to reomve the online flags that a
Jun-Hui Zhao
	
	Aug 13, 2018, 9:42 AM
Hi Geoff, I was busy on other things. Just read this message alerted by google reminder. You may try unflagging function in either plotms or flagdata. If not wo
Geoffrey Bower
	
	Aug 13, 2018, 1:22 PM
I tried unflagging with flagdata and that seemed to work. Jennie is now starting calibration of the data with CASA. She’s new to this. Her only experience is wi
Jun-Hui Zhao 
	
AttachmentsAug 17, 2018, 5:14 PM
	
to Jennie, Holly, Thomas, Glen, Ryan, Geoffrey, bcc: Jim
Thanks, Geoff. That is a good start. I also implemented options of resetting the online flag state to unflagging in the new pipeline and made a test  on the server rtdc9 with your data set (the screen prompted log attached). The test script
swarm2casa_180517.csh is added to /opt/sma/SWARM2CASA/script/RTDC
The new version SMA-SWARM2CASA 1.0.2 will be left on the RTDC system for a week before exporting it to
HILO.

Holly: I tested the feedback link (to my email) on my personal web page. It appears to work for my self-test
but it may be broken with outside world. It would be great if you can relay a feedback link to my email: jzhao@cfa.harvard.edu at RTDC concerning the new SWARM2CASA pipeline.

Regards,    Jun-Hui
Attachments area
Preview attachment v1.0.2test.log
[Binary File]
Zhao, Jun-Hui 
	
Aug 30, 2018, 6:16 PM
	
to Geoffrey, Jennie, Holly, Thomas, Glen, Ryan, bcc: Jim
Hi,
     So far, I have heard no complaints concerning the new version 1.0.2 on the RTDC system. The version is
now exported to HILO. The information on my webpage for using/testing the trial version has been updated:
https://www.cfa.harvard.edu/~jzhao/SMA-CASA/swarm2casaRelease.html
    Regards,
    Jun-Hui