Meetings for the new SWARM data pipeline and CASA for SMA data reduction
 

Try out the SWARM2CASA pipeline for SMA users

  • 2018-11-29 (Thursday) at 11:00am HST and 4:00pm EST
    •
  • Room M240, 160 Concord Ave., Cambridge, MA
    •
  • Agenda:
    •

    A series of special meetings concerning SMA SWARM data.

    • 1. Review of updated data format and software limits
    • 2. SMA observing modes
      • modeInfo
        • For example:
        rtdc9.cfa.harvard.edu: more /sma/data/science/mir_data/181103_02:55:10/modeInfo
        2 8
        The 1st integer is number of receivers used.
        The 2nd integer is number of swarm chunks produced.
      • How many modes that SMA operation normally supports?
        • The standard mode for the current SMA operation -
          • The mode code of 2 8 is used now when NC=4 (four SWARM chunks per rx)
          with a number of polarization states 1 (non pol) or 4 (full pol).
        • The standard mode in the near future -
          • The mode code will be 2 12 when NC=6 comes online.
        • Dual rx operation in general -
          • The mode code will be 2 NC.
          ___________________________________
          NC is a number of SWARM correlator chunks used in SMA operation.
      • Extraordinary modes found in the SMA data
        • One rx and one pol but number of baselines > 28, for examples:
          • 180407_03:02:50 ............ mode code: 2 2 ??
          181013_02:48:00 ............ mode code: 2 8
        • Two rx and one parallel pol (LL) and one cross pol (LR), for examples:
          • 180803_03:12:39 ............ mode code: 2 8
        • Varying in frequency configuration from source to source, for examples:
          • 180131_02:22:07 ............ mode code: 2 2 ??
          Peculiar modes appear to be often used in the 2018 SMA operations.
        • Where to find further infomation concerning these modes, SMA operation log?
        • The detailed information on the peculiar modes is needed to interpret the data.
    • 3. Syetem temperature measurements (Attila)
      • tsys_read , a header table stored Tsys in baseline-based order.
        • Antenna-based Tsys data can be extracted from the baseline-based storage.
          For examples:
        • 170923_04:19:52
        • 181002_03:55:28
          • Issues -
        • Two independent measurements in Tsys rather than one stored in each of the baseline entries according to the entry tsysString.nMeasurements = 2 provided by tsys_read .
        • The document of the online storage format appears to provide inadequate information to interpret the data entries in the case of dual rx operation modes involved -
            • On each baseline corresponding to rx1, there are two sets of four values: lower IF freq, upper IF freq, LSB Tsys, USB Tsys. It is known that LSB Tsys = USB Tsys. Is first set for rx 1 and the second set for rx 2?
            Then, on the baselines corresponding to rx2, the data strings repeat those for rx1.
        • In the dual rx operation mode, tsys_read appears to supply redundant information(?).
        ________________________
        Notes from the meeting:
      • eng_read, antenna-based Tsys stored for both rx1 and rx2 with no ambiguous separation between the entries (Karto).
        • The entries in eng_read table appear to be useful but be careful to use them since some of the elements padding with random numbers. The elements needed for extracting antenna-based Tsys are good (Karto): 
              int antennaNumber;
      ...
      int inhid    ;      /*  integration id #                                                       */
      int ints     ;      /*  integration #                                                          */
      double dhrs  ;      /*  hrs from ref_time                                                      */
      double ha    ;      /*  hour angle                                                             */
      double lst   ;      /*  lst                                                                    */
      ...
      double tsys ;
      double tsys_rx2 ;
      double ambient_load_temperature ;
    • 4. Error alert system & online data errors
      • Do we have online error alert system?
          • Yes.
      • Does the online data storage record the error information caused by hardware and online software systems as well as non-standard observing operations?
          • Yes.
          The online errors are given in the integer parameter of flags in the header structure table sp_read. The definitions of the flag states given in we_read.
      • Data flag status and detailed error information needed for the purposes of:
        • Speeding up error identification and fixing data file errors with data patches by swarmCheck, a computing program of the pipeline software swarm2casa.
        • The specific data patches being coded in swarmRead, one of the key computing program routines in swarm2casa, to interpret online storage formatted SMA data.
        • Both swarm2casa & CASA require antenna-based information on the error flag status in details.
    • 5. Flag table: error information on antenna-based flag status (Attila)
      • sp_read
          • the entry int flags provide baseline-based flag status using the flag status given in we_read. We could solve for the antenna-based flags from baseline-based storage if having adequate infomation on the definition of flag status in the antenna-pair combinations for baselines. The current online storage data format appears to have not defined the baseline-based flag status in details (see email discussion).
      • we_read
          • This header table stores the antenna-based information for flags along with the weather information. The antenna-based information on system errors and weather conditions stored in we_read appears to be valuable data needed for both swarm2casa & CASA.
          Issues -
          How to extract the information from this table?
        • Can anyone who has tested we_read provide us a handy routine to read this table?
        • May Attila provide the code lines related to we_read from his online storage program?
      • Definition of antenna-based error & online flag status
          • The corresponding integers are stored in we_read as int flags[11]?
          Elements 9 and 10 are no use?
          The name of variable for most items is clear for the type of errors.
        • A few entries need to be clarified:
          • Item 8: SFLAG_IRIG_TIME ?
          • Item 13: SFLAG_WACKY_OFFSETS? pointing offset or LO/IF offset or waveplate offset or any other offset?
          • Item 15: SFLAG_AVE_TRACKING?
          • Item 16: SFLAG_PEAK_TRACKING? These two items for pointing tracking?
          • Item 19: SFLAG_OPERATOR? Operator's non-standard operation actions, such as ?
      • Interpretation of baseline-based flag status for handling visibilities in general:
        • Bad when the value of flags in sp_read is 1 or larger.
        • Otherwise good.
    • 6. Wish list and to do
      • Antenna-based Tsys
          • Clarify the confusion in lower/upper IF entries and the rx entries in the baseline-based storage table tsys_read
        • pipeline correction for Tsys
        • accommondate antenna-based Tsys table to SMA measurementSet for CASA
      • Antenna-based errors | flag status
          • Sort out the definition on baseline-based flag status.
          Or, find a way to read the table we_read
        • Using the information to create a routine to fix data issues in a intelligent way.
      • Patches for long source names of mosaic fields (large number)
        • Taking a common prefix as Mosaic fields' name with position offsets w.r.t. the reference position.
          • Meanwhile, the observing operation may advise users to use a common short name (15 or less characters) along with position offsets for moasic fields.
        ......
      ______________________________

      Reference links -