Verification/pipeline analysis of correlated data in AIPS

The purposes of data verification are:

1. to make sure the correlated data are OK, and no recorrelation is required.
2. to provide feedback to PIs, and telescopes, on how their observations went, and any problems with the data.

We also want to create off-source flagging (UVFLG) and calibration (ANTAB) files for each experiment, to provide to the PIs.

Pipeline verification is now done immediately after correlation as described in the Espresso documentation.

An AIPS data area is currently set up on cuppa01 and cuppa06. Since 10 August 2010, cuppa06 is used for the output data, and the pipeline area is /data/corr/pipe

Choose a vacant AIPS ID number for a new experiment. To see which AIPS ID numbers are currently in use:

aipsid.pl .

The correlator output data is stored in /data/corr/corrdat/<expcode>, where <expcode> should be replaced with the desired experiment code. (Note: some old experiments have been removed from this data area to create space, but have been copied to cuppa03:/arch/corr/corrdat).
Correlator input files are kept in /home/corr/LBA/<year>/<session>/<expcode>.

After pipelining, the output file names and descriptions should be listed in the “Correlation Notes” for that particular experiment on the wiki. This page can be found by clicking on the experiment name in the correlation summary table: http://cira.ivec.org/dokuwiki/doku.php/correlator/records/<year> (Note the wiki page is now created using lba_feedback.py after running the pipeline)

For each station, a file formatted for task UVFLG should be extracted from the telescope observing log.

For Hobart, Ceduna and Tidbinbilla, the files should have been placed in the experiment ftp area at ATNF, which is linked from the experiment setup page on the VLBI Observers wiki. Usually the files are named <expcode><st>.uvflg where <expcode> is the experiment code and <st> is the 2-letter station code.

Off-source flagging files for ATNF stations are generated by scripts also linked from experiment page on the Observers wiki.

In case the flag file for a particular station is not available, the a priori flag file generated by SCHED (which can also be found in the experiment ftp area, <expcode>.flag) may be used for that station, but it is always better to use the off-source data extracted from the observing logs if possible, as this reflects when the telescopes were actually off-source.

The UVFLG files from each station can be concatenated into one file <expcode>.uvflg

Some editing may be required - see example below.

Get UVFLG file for Tid from the ftp area:

wget ftp://ftp.atnf.csiro.au/pub/people/vlbi/v212/v212a/v212ati.flag

Get UVFLG files for ATNF stations using Chris Phillip's script linked from http://www.atnf.csiro.au/vlbi/wiki/index.php?n=LBAJun2008.V212A (under “Observing logs” at the bottom of the page). I do this by right-clicking on the link, select “Copy Link Location” and paste into the terminal where I'm running wget. But then I have to edit the line to escape '&' characters in the query (by putting a '\' before them).

For Parkes:

wget -O v212apa.uvflg http://www.atnf.csiro.au/cgi-bin/vlbi/parkes-onsource.pl?session=Jun2008\&start=159/09:00:00\&end=159/20:58:00

For ATCA (need to add year for August 2008 and earlier experiments):

wget -O v212aat.uvflg http://www.atnf.csiro.au/cgi-bin/vlbi/atca-onsource.pl?start=159/09:00:00\&end=159/20:58:00\&year=2008

For Mopra:

wget -O v212amp.uvflg http://www.atnf.csiro.au/cgi-bin/vlbi/mopra_onsource.pl?exper=v212a

Concatenate:

cat v212ati.flag v212a??.uvflg » v212a.uvflg

Edit station names in the uvflg file if necessary. Run prtan to see list of antenna names in AIPS. For v212a, it was necessary to replace 'TID' with 'DSS43' and replace all 'ATCA' with 'CATW109'. Also edited to flag Tid data before start/after stop time.

See below for a description of how to load data and produce diagnostic plots interactively in AIPS.

An alternative, quicker and efficient way of processing the LBA data and producing some useful plots is to use the LBA pipeline.

This is a python/ParselTongue based pipeline, which is available on the cuppa cluster.

How it Works

The LBA pipeline takes a text file as an input, which contains the relevant details of an experiment you wish to process the data from.

An example of the LBA pipeline input file is supplied below. This can serve as a template for the LBA experiments, in which just the required change of parameters can make it an input file for a certain experiment:

LBA pipe input file

The input file template can be picked from here, edited for a certain experiment and then saved as <expcode>.inp

Pipeline Setup:

The first step is to create a <expcode> directory in the standard area - currently this is cuppa06:/data/corr/pipe. Create two sub-directories 'in' and 'out' in the <expcode> directory. The 'in' should contain the pipeline input file and the .uvflg file and 'out' is the directory which the plot files will be written to.

Put the <expcode>.inp and <expcode>.uvflg in the /data/corr/pipe/<expcode>/in

The parameters to in the input file to run the pipeline can be set for a certain experiment as follows:

tmask = 1                         $ tmask is a range of steps to be carried out. Generally it is advisable to set tmask=1 as it will run the whole pipeline
experiment = <expcode>            $ the ident of the experiment e.g. v252j
t_int = 2                         $ integration time. Pick it up from the experiment page on this wiki
userno = aipsid                   $ the AIPS ID you wish to use for the processing
refant = CATW104, PKS, MOPRA      $ a priotised list of reference antennas. Generally the compact array can be specified as the first in the list
plotref = CATW104, PKS, MOPRA     $ plotref is a list of antennas, baselines to which will be plotted. Specify all antennas used in an experiment if you wish
                                    to plot all baselines
phaseref = <calibrator>           $ the phase calibrator and the respective source can be specified if phase-referencing is to be done
target =   <source>

solint = -1                       $ solution interval. '-1' set the interval to the typical scan length
fit_file =                        $ path to the data file. The correlated data normally resides in /data/corr/corrdat/<expcode> The required .rpf files (multiple 
                                    separated with a comma) can be specified here.
indir = /data/pipe/<expcode>/in   $ the directory containing the input file and the .uvflg file
outdir = /data/pipe/<expcode>/out $ the directory to write out the plot files to

                                    The optional parameters can be left as they are

Running the Pipeline

After completing the pipeline setup, cd to the directory /data/pipe/<expcode> and run the pipeline as follows:

pipe@cuppa06:/data/corr/pipe/<expcode>$ ~corrtest/progs/Pypeline/LBA.py <expcode>.inp

Where <expcode>.inp is the input file created previously.

The pipeline will automatically run various AIPS tasks such as ATLOD, DTSUM, LISTR, POSSM, VPLOT, FRING etc and will produce useful diagnosis and verification plots in pdf files written in the directory /data/pipe/<expcode>/out.

The plots can be viewed in any pdf viewer and uploaded to the CIRA ftp as described below.

Look in the calibrator database

Note that there is now an account 'pipe' (same password as corr) for analysis. Set an environment variable to point to the data area, e.g. export DATA=/data/corr/corrdat/<expcode>. If you start aips from e.g. /data/corr/pipe/<expcode> you can use PWD to write plots out to that area.

Change to the directory that is to contain the output plots etc. for the experiment to be verified. Otherwise set an environment variable with the path to this directory, e.g. export V212A=/data/pipe/v212a.

Start AIPS with

aips tv=local:0

Enter AIPS ID number at the prompt.

Three windows should appear on your local desktop: the message server, the TV screen server, and TEK server (generally not used). If the message server dies, AIPS messages will appear in the main AIPS terminal instead.

Note that the adverbs for tasks listed below are just a guide, not a complete list, i.e. for a specific task you may need to reset some other input parameters in order to get sensible output. Feel free to add other specific adverb settings to these instructions if you think this will help you and others in the future! Use help or explain (task) to get help on AIPS tasks and their adverbs.

Since January 2010, output data are FITS-IDI format created with difx2fits. These can be loaded with task FITLD. Specify DATAIN.

Prior to this, the output data was normally in RPFITS format, which must be loaded with task ATLOD.

task 'atlod'
infile 'PWD:<expcode>.rpf $ if required, replace PWD with previously set environment variable for data file directory
outname '<expcode>        $ TIP: open quote in AIPS can be used to read/write lower case, closed quotes ALWAYS refer to upper case
aparm -1,0,1           $ aparm(3) retains autocorrelations. aparm(1)=-1 "kludge" polarizations. 
inp                    $ shows inputs to task
go                     $ executes task

It seems aparm(1) has to be set to -1 to properly handle single polarization LBA data. For dual polarizations aparm(1) can be set to zero. Defaults should be OK for all other parameters.

Check on progress in the message window. Example messages:

CUPPA0> ATLOD1: Task ATLOD  (release of 31DEC08) begins
CUPPA0> ATLOD1: Using compressed uv data format.
CUPPA0> ATLOD1: Dropping all flagged visibilities.
CUPPA0> ATLOD1: Retaining autocorrelations.
CUPPA0> ATLOD1: Vis weights set to integration time in units of 15 sec
CUPPA0> ATLOD1: Averaging all pulsar bins
CUPPA0> ATLOD1:
CUPPA0> ATLOD1: Processing file:  1
CUPPA0> ATLOD1: Input polarizations: RR,LL
CUPPA0> ATLOD1:
CUPPA0> ATLOD1: Create v212a       .UVDATA.   1 (UV)  on disk  1  cno    1
CUPPA0> ATLOD1: Recomputing UVWs for pre-20/02/1998 LBA data
CUPPA0> ATLOD1:
CUPPA0> ATLOD1: File:  1,  Scan:   1
CUPPA0> ATLOD1: Date: 2008-06-07    UT start:  0 09:00:01
CUPPA0> ATLOD1:
CUPPA0> ATLOD1: Visibilities accepted/rejected for each source/frequency:
CUPPA0> ATLOD1:    Sideband              +1          +1          +1          +1
CUPPA0> ATLOD1:    AIPS IF axis           1           2           3           4
CUPPA0> ATLOD1:    Source/Freq(GHz)   2.226000   2.242000   2.258000   2.274000
CUPPA0> ATLOD1:    1424-418           6000/0      6000/0      6000/0      6000/0
CUPPA0> ATLOD1:    SN2004IP          97200/0     97200/0     97200/0     97200/0
CUPPA0> ATLOD1:    J1826-3650        97200/0     97200/0     97200/0     97200/0
CUPPA0> ATLOD1:    1921-293          15000/0     15000/0     15000/0     15000/0
CUPPA0> ATLOD1: Category summaries of visibilities for this scan:
CUPPA0> ATLOD1:
CUPPA0> ATLOD1: Finished reading file number  1.
CUPPA0> ATLOD1:
CUPPA0> ATLOD1: The average visibility weight was  0.133
CUPPA0> ATLOD1: Appears to have ended successfully
CUPPA0> ATLOD1: cuppa01 31DEC08 TST: Cpu=     13.3  Real=     24  IO=       443

To view the new catalog entry and check the header information of the loaded file:

ucat
getn 1
imh     $ imheader (all AIPS commands use minimum match)

Example:

>imh
AIPS 1: Image=1424-418  (UV)         Filename=v212a       .UVDATA.   1
AIPS 1: Telescope=ATLBA              Receiver=
AIPS 1: Observer=                    User #= 1975
AIPS 1: Observ. date=07-JUN-2008     Map date=26-MAR-2009
AIPS 1: # visibilities    215400     Sort order  TB
AIPS 1: Rand axes: UU-L  VV-L  WW-L  BASELINE  TIME1  SOURCE  FREQSEL
AIPS 1:            WEIGHT  SCALE
AIPS 1: ----------------------------------------------------------------
AIPS 1: Type    Pixels   Coord value     at Pixel     Coord incr   Rotat
AIPS 1: COMPLEX      1   1.0000000E+00       1.00  1.0000000E+00    0.00
AIPS 1: STOKES       2  -1.0000000E+00       1.00 -1.0000000E+00    0.00
AIPS 1: FREQ        65   2.2260000E+09       1.00  2.5000000E+05    0.00
AIPS 1: IF           4   1.0000000E+00       1.00  1.0000000E+00    0.00
AIPS 1: RA           1    14 27 56.297       1.00       3600.000    0.00
AIPS 1: DEC          1   -42 06 19.427       1.00       3600.000    0.00
AIPS 1: ----------------------------------------------------------------
AIPS 1: Coordinate equinox 2000.00
AIPS 1: Maximum version number of extension files of type HI is   1
AIPS 1: Maximum version number of extension files of type FQ is   1
AIPS 1: Maximum version number of extension files of type SU is   1
AIPS 1: Maximum version number of extension files of type AN is   1
AIPS 1: Maximum version number of extension files of type NX is   1
AIPS 1: Maximum version number of extension files of type CL is   1

In general the data should appear with sort order 'TB'. (In case not, it may be necessary to run UVSRT or MSORT, followed by INDXR, but why are the data out of order?)

Sometimes there will be more than one output data file for a given experiment. It may make sense to load them into a single file (e.g. for different time-ranges correlated with the same parameters), or if different correlation parameters were used (e.g. a spectral line and a continuum pass), load each pass separately.

task 'uvflg'
getn 1
intext 'PWD:<expcode>.uvflg   $file with flagging commands
inp                           $check inputs
go

Delete edge channels?

Summary of (u,v) data points per baseline for entire experiment:

task 'dtsum'
docrt -1                          $ print to file (prints to screen if docrt >1)
aparm 0                           $ set all aparm values to 0
outprint 'PWD:<expcode>.DTSUM.txt $ specify filename
inp                               $ check inputs
go

This can be useful for checking antenna numbers.

Note that if flag table has not been applied, visibilities will be present for all stations for the whole experiment, whether the station is off-source/recording or not.

Scan listing:

task 'listr'
optype 'scan'
outprint 'PWD:<expcode>.SCAN.txt  $ specify filename
inp                               $ check inputs
go

These will indicate e.g. RFI at particular stations.

task 'possm'
stokes 'half' $ =RR,LL pols
aparm 0
aparm(7) 1    $ Label x-axis with frequency
aparm(8) 1    $ to plot total power
aparm(9) 1    $ plot IFs together
solint -1     $ average over each scan
nplots 9      $ 9 plots per page (the maximum)
dotv -1       $ plot to file
inp           $ check inputs
go

Note: this can result in a lot of output, especially for experiments with a large number of scans!

imh will show the number of 'PL' (plot) type extension files produced.

Print the plots to a postscript file:

task 'lwpla'
plver 1
inver 9999 $ >= number of plots
outfile 'PWD:<expcode>_POSSM_AUTOCORR.ps
inp        $ check inputs
go 

Check the output file: in a terminal window outside of AIPS, 'gv <expcode>_POSSM_AUTOCORR.ps'

You can then use the verb EXTDEST (inext 'pl';inver -1;extdest) to delete the PL extension files.

Alternatively, you will have to keep track of the maximum PL version number after each run, in order to set 'plver' each time you run LWPLA to produce output plot files (otherwise you will end up with one really huge postscript file containing all plots). Also, if outfile already exists, LWPLA will append to it, so use a unique file name if you don't want this.

task 'vplot'
optype ''         $ cross-correlation
bchan 10;echan 55 $ discard edge channels (example, for 64-channel data - CHECK how many channels in your data (FREQ in command 'imh'))
avgchan 1         $ average channels
stokes 'half'     $ for dual polarization (RR,LL)
solint 0.5        $ averaging time (mins)
aparm 0
bparm 0,-1        $ plot amp & phase together
nplots 6
inp               $ check inputs
go

task 'lwpla'
plver 1
inver 9999 $ >= number of plots (Can check again after vplot completes, using imh to check PL number - should be small)
outfile 'PWD:<expcode>_VPLOT_UNCAL.ps
inp        $ check inputs
go 

Note: If flagging has not been applied, this plot will contain off-source data! Also, where a station has NO data, the amplitudes appear as zero but the phase does not - i.e. the phases at such times may look meaningful, but are not! Be warned…

Preferably choose calibrator sources (adverb sources); these plots will not be very useful for faint targets. Picking which sources are calibrators and which are targets might require some experience. The target source may be obvious from the experiment title. Otherwise it may help to check the scan listing produced with LISTR (observed sources are listed near the end). A typical phase-referencing observation alternates between target and phase-reference calibrator scans. To check which sources are likely to be calibrators you can try looking for them in the ATCA calibrators catalogue.

Sometimes the target sources will be many and bright (e.g. for experiments such as v252); in that case just leave the source selection blank.

You may wish to limit the output by setting antennas to plot baselines to a chosen reference antenna (or two).

The output of DTSUM shows the antenna numbers (or use task PRTAN).

task 'possm'
sources '1424-418','J1826-3650','1921-293' $ example source selection
stokes 'half' $ May wish to plot cross-hand pols 'RL,LR' (if present) to a separate file.
bchan 0;echan 0
aparm 1,1,0,0,-180,180,1,0,1 $ plot phase with fixed range -180,180 (see help)
solint -1     $ average over each scan
nplots 9      $ 9 plots per page (the maximum)
dotv -1       $ plot to file
inp           $ check inputs
go

task 'lwpla'
plver 1
inver 9999 $ >= number of plots
outfile 'PWD:<expcode>_POSSM_UNCAL.ps
go 

Example inputs. These will need to be specified carefully for each experiment.

Note: Old LBA data in RPFits format, read in with ATLOD, requires the integration time to be set (dparm(4)). The “correlation notes” on the wiki should include what integration time was used. This does NOT need to be set for data in FITS-IDI format read in with FITLD (default for experiments correlated since January 2010).

task 'fring'
calsour '1424-418','J1826-3650','1921-293' $ select calibrators
bchan 10;echan 55                          $ discard edge channels (in this case there are 64 channels per IF)
refant 1                                   $ reference antenna should be sensitive, and on source most of the time
search 3,2,4                               $ prioritised list for exhaustive search (not usually needed) 
solint 2                                   $ solution interval (minutes) - typical length of the phase-reference calibrator scan.
aparm 0
aparm(6)=2                                 $ prints a lot of information including antenna S/N ratio
aparm(9)=1                                 $ do exhaustive baseline search
dparm(4)=2                                 $ the integration time in seconds - MUST BE SET for LBA data
inp
go

Note: If you realise you've messed something up, and you need to run abort, be careful - it doesn't always abort nicely. If as a consequence you get errors and your task starts dying of “unnatural causes”, see this thread. Moral of the story: Run getn 1 to go back to your original file and clrs to clear any problem statuses. If like Claire, you aborted because you'd forgotten to reset the pl or sn files, best option is to let it run, it seems, but DO remember to clear them before you run your task again :)

Note 2: ucat will show the READ/WRITE status flag. If you see this and there's no task running, then it's probably a spurious flag from an aborted task (do clrstat)

task 'snplt'
inex 'sn'            $ plot the 'SN' (solution) table
inver 0
sources ''
nplots 8
optype 'dela'        
$pixrange -5e-8,5e-8  $ example to limit displayed y-axis range to +/-50 nanosec (time in seconds)
dotv -1
inp
go

Delay solutions should be typically within tens of nanoseconds. No unexpected gaps?

Note there may be a few outlying points (due to e.g. 'bad' solution with marginal S/N)

Print delay solutions to a file:

tget lwpla
outfile 'PWD:<expcode>_FRING_DELAY.ps
inp
go

inext 'pl';inver -1;extdest

tget snplt
optype 'rate'        
pixrange 0
inp
go

Typically less than a few milliHz?

tget lwpla
outfile 'PWD:<expcode>_FRING_RATE.ps
inp
go

inext 'pl';inver -1;extdest

tget snplt
optype 'phas'
inp
go

Nicely connected on phase-reference calibrator? (May not appear well-connected for observations of many sources spread over the sky)

tget lwpla
outfile 'PWD:<expcode>_FRING_PHASE.ps
inp
go

inext 'pl';inver -1;extdest

Apply fringe-fit solutions from calibrator sources used in FRING (calsour) to all sources (sources), creating a new 'CL' table.

task 'clcal'
interpol 'ambg'     $ uses rates to resolve phase ambiguities (where phase changes by large amount between cal scans)
bparm 0             $ for no smoothing
inver 1             $ specify SN table to apply
gainver 0;gainuse 0 $ apply to highest numbered CL table, write new CL table
$ NB - keep account of SN & CL tables - be careful not to apply solutions twice. 
inp
go

tget 'vplot'
docal 1      $ apply the CL table (gainuse 0 = highest)
inp
go

tget lwpla
outfile 'PWD:v212a_VPLOT_FRING.ps
inp
go

inext 'pl';inver -1;extdest

tget possm
sources '1424-418','J1826-3650','1921-293' $ example source selection
stokes 'half'
bchan 0;echan 0
aparm 1,0     $ autoscale phase plots this time to see small deviations
aparm(7)=1    $ label x-axis with frequency
aparm(9)=1    $ plots IFs together in a row
solint -1     $ average over each scan
nplots 9      $ 9 plots per page (the maximum)
dotv -1       $ plot to file
inp           $ check inputs
docal 1       $ apply FRING solutions
inp
go

tget lwpla
outfile 'PWD:<expcode>_POSSM_FRING.ps
inp
go

inext 'pl';inver -1;extdest

For the fringe-fitted data, the phases should be close to zero and flat across the band (in the parallel-hand polarizations RR,LL, and where there is sufficient S/N).

Example:

task 'uvplt'
sour 'SN2004IP''    $ 1 source at a time
bchan 32;echan 32   $ plot only 1 channel from each IF to speed things up
aparm 0
bparm 6,7,0         $ plot u,v coverage
inp                 $ check inputs
go                    

tget lwpla
outfile 'PWD:v212a_<srcnam>_UV.ps
inp
go

inext 'pl';inver -1;extdest

To do: bandpass calibration, amplitude calibration (ANTAB, APCAL), imaging…

When you've finished with AIPS, exit with the command kleenex to close all servers.

(If needed, convert plots to PDF with ps2pdf. NB the pipeline output plots are already in PDF)

Ftp to cira.ivec.org as astronomy (password required).

passive to use passive mode
cd to anonymous/<expcode>
mput <expcode>*.pdf
mput <expcode>*.txt

Edit the “Analysis notes and links to plots” section on http://cira.ivec.org/dokuwiki/doku.php/correlator/records/<expcode>

Insert links to the plots in the ftp area (replace all “expcode” with the actual experiment code) and comments as required.

Template for analysis notes

There is now a script ~corrtest/progs/Pypeline/lba_feedback.py to produce the wiki page, using the templates which the pipeline outputs:

The verb ALLDEST can be used to delete all AIPS files for a user number, once the data have been finished with, e.g.

inname ''
inclass ''
inseq 0
intype ''
alldest

When all data have been deleted, ALLDEST will also ask to destroy all SAVE/GET and TGET files.

FREESPAC tells how much space is available on the AIPS disk.