vex2difx is a program that takes a vex files (such as one produced by sched with various tables based on observe-time data appended) and a configuration file (described below) and generates one or more .input files for use with difx. Each .input file is accompanied by a .calc file which is used by calcif2 to generate the .delay and .uvw files needed at correlation time. vex2difx along with calcif2 supercedes the functionality of vex2config and vex2model.

Users and future developers of vex2difx should be aware of the approach used in designing vex2difx which can be summarized as follows:

1. The output files should never need to be hand edited
2. Simple experiments should not require complicated configuration
3. All features implemented by mpifxcorr should be accessible
4. All experiments expressible by vex should be supported
5. The configuration file should be human and machine friendly
6. Command line arguments should not influence the processing of the vex file

Note that not all of these ideals have been completely reached as of now. It is not the intention of the developer to guess all possible future needs of this program. Most new features will be easy to implement so send a message to the difx-users mailing list for requests.

The VLBI scheduling programs sched and sked both produce vex files that are used to control antennas for observations. Certain information that is not available prior to an observation needs to be provided to vex2difx in some way. One way is to append this data to the vex file. The alternative is to provide it in the .v2d file (as shown further down). This information includes:

1. The Earth orientation parameters ($EOP block in the vex file, or EOP blocks in the .v2d file)
2. The antenna clock offsets ($CLOCK block in the vex file, or clock values in the ANTENNA blocks of the .v2d file)
3. The volume serial numbers for the recording media ($TAPELOG_OBS block, or file lists in the ANTENNA blocks of the .v2d file)

Population of these three tables is necessarily a correlator/array specific operation and is the responsibility of the vex2difx user to arrange.

Note, only formal vex files are supported as input to vex2difx. Similar looking ovex files used at some/all Mark4 correlators are not acceptable, however, with a small amount of work an ovex file can be hand converted to a valid vex file. It would not be hard to write a conversion script to do this automatically.

The configuration file consists of a number of global parameters that affect the way that jobs are created and several sections that can customize correlation on a per-source, per mode, or per scan basis. All parameters (those that are global and those that reside inside sections) are specified by a parameter name, the equal sign, and one value, or a comma-separated list of values, that cannot contain whitespace. Whitespace is not required except to keep parameter names, values, and section names separate. All parameter names and values are case sensitive except for source names and antenna names. The # is a comment character; any text after this on a line is ignored.

• bool → A boolean value that can be True or False. Any value starting with 0, f, F, or - will be considered False and otherwise True.
• float → A floating point number. Can be of the forms: 1.23, 1.2e-4, -12.6, 4
• int → An integer.
• string → Any sequence of printable(non-whitespace) characters. Certain fields require strings of a maximum length or certain form.
• date → A number or string representing Universal Time. Several formats are supported:
  • Modified Julian Day : 54345.341944
  • Vex time format : 2009y245d08h12m24s
  • VLBA-like format : 2009SEP02-08:12:24 (Note - between date and time!)
  • ISO 8601 format : 2009-09-02T08:12:24

DiFX2.0 adds support for features like multiple simultaneous phase centres. The corresponding functionality in vex2difx is only available in DiFX2.0, obviously. Any parameters which are only available in DiFX2.0 are italicised like this.

Global parameters can be specified one or many per line such as:

maxGap = 2000 # seconds

or

mjdStart = 52342.522 mjdStop=52342.532 

Note that the baselines parameter supports the following syntaxes: A1-A2 A1+A2+A3-A4+A5 A1-* A1+A2-* and so on. For each list member, all baselines consistant with an antenna match on both sides will be kept.

A source section can be used to change the properties of an individual source, such as its position or name. In the future this is where multiple correlation centers for a given source will be specified. A source section is enclosed in a pair of curly braces after the keyword SOURCE followed by the name of a source, e.g.:

SOURCE 3C273
{
  source parameters go here
}

or equivalently

SOURCE 3c273 { source parameters go here }

An antenna section allows properties of an individual antenna, such as position, name, or clock/LO offsets, to be adjusted.

The optional arguments for addZoomFreq control spectral averaging (currently constrained to be same as the parent band) and whether or not the parent band is still correlated - default is that it is *not* correlated. These are more for potential future compatibility.

Please note that vex uses as a clock sign convention that is positive for a formatter with its clock running fast (i.e., the second tick happens too early). The clockOffset and clockRate in this ANTENNA section, as well as FITS files, have the opposite sign convention.

Setup sections are enclosed in braces after the word SETUP and a name given to this setup section. The setup name is referenced by a RULE section (see below). A setup with the special name default will be applied to any scans not otherwise assigned to setups by rule sections. If no setup sections are defined, a setup called default, with all default parameters, will be implicitly created and applied to all scans. The order of setup sections is immaterial.

It is possible to specify the Earth Orientation Parameters (EOPs) through the .v2d file. Normally these values will be appended to the vex file, but there may be cases where a completely unmodified vex file is desired (eVLBI maybe?). Like ANTENNA and SOURCE sections, each EOP section has a name. The name must be in a form that can be converted directly to a date (see above for legal date formats). Conventional use suggests that these dates should correspond to 0 hours UT; deviation from this practice is at the users risk. It is not advised to mix EOP values stored in the vex and .v2d files.

Example section

EOP 55005 { tai_utc=34 ut1_utc=0.236958 xPole=0.10597 yPole=0.53906 }

A rule section is used to assign a setup to a particular source name, calibration code (currently not supported), scan name, or vex mode. The order of rule sections does matter as the order determines the priority of the rules. The first rule that matches a scan is applied to that scan. The correlator setup used for scans that match a rule is determined by the parameter called setup. A special setup name SKIP causes matching scans not to be correlated. Any parameters not specified are interpreted as fully inclusive. Note that multiple rule sections can reference the same setup section. Multiple values may be applied to any of the parameters except for setup. This is accomplished by comma separation of the values in a single assignment or with repeated assignments. Thus

RUlE rule1
{
  source = 3C84,3C273
  setup = BrightSourceSetup
}

is equivalent to

RULE rule2
{
  source = 3C84 3C273
  setup = BrightSourceSetup
}

is equivalent to

RULE rule3
{
  source = 3C84
  source = 3C273
  setup = BrightSourceSetup
}

The names given to rules (e.g., rule1, rule2 and rule3 above) are not used anywhere but are required to be unique.

Note that source names and calibration codes reassigned by source sections are not used. Only the names and calibration codes in the vex file are compared.

Currently vex2difx operates in one of two modes:

• normal The output of vex2difx is a set of files useful for correlating the data. This is, as the name suggests, the normal mode of operation for vex2difx.

• profile This mode specializes in making files useful for forming pulse profiles in preparation for pulsar gating. The difference compared to normal mode is that the standard autocorrelations are turned off and instead are computed as if they are cross correlations. This allows multiple pulsar bins to be stored. No formal cross correlations are performed. To be useful, one must create and specify a binconfig file and select only the pulsar(s) from the experiment.

vex2difx is executed on the command line with:

vex2difx [options] inputFile

Although no command line options can change the way that vex2difx processes a file, there are some options that the user may find useful:

• -h or –help Print usage information to the screen. This is the same as if no arguments were supplied to vex2difx.
• -o or –output Writes a configuration file called inputFile.params which is a valid configuration file identical to inputFile, but with all assumed defaults populated. This is useful to see what was actually assumed.
• -v or –verbose Prints much more information to the screen. Use this option twice for even more information.
• -d or –delete-old Deletes all output from previous runs of vex2difx with same prefix. This is most useful when rerunning and a smaller number of jobs are created.
• -s or –strict Treat some warnings as errors and quit.

If you have a problem with vex2difx, please email the difx users email group. Be sure to include the following in the email:

1. A description of the problem
2. The v2d file supplied to vex2difx
3. The vex file pointed to from the v2d file
4. the captured output when running vex2difx with extra verbosity (use options -v -v)

The following example demonstrates the simplest case where all defaults are assumed

vex=trivial.vex

The following is a more realistic case for a simple experiment

vex=simple.vex

SETUP default
{
  nChan=64
  tInt =3.0 
}

This shows how to change the coordinates of two sources in a file

vex=coords.vex

SOURCE J1232+131 { ra=12h32m15.12s dec=13d07'12.5" }
SOURCE PLANETX   { ra=11h59m59.999s dec=-12d59'59.88" }

SETUP default
{
  nChan=128
}

This is a more complicated file showing how to apply different correlator setups to different sources

vex=twosetups.vex
maxGap=1000  # don't split the jobs at every source change,
             # instead, make just 2 interleaved jobs
antennas=BR,FD,HN,MK  # select only these four antennas for now  

SETUP target
{
  nchan=1024
  tInt =1.2
}

SETUP calibrator
{
  nchan=32
  tInt =4
}

RULE calRule
{
  source=J1234+1231,3C84,3C273
  setup =calibrator
}

RULE targetRule
{
  # note: not specifying any restrictions so all sources that don't 
  # match above will match here
  setup = target   
}

The above could have used a default setup rather than a catch-all rule and resulted in the same output.

vex2difx allows .input file generation for two types of media. A single .input file can have different media types for different stations. Ensuring specification of media is important as antennas with no media will be dropped from correlation. The default media choice is Mark5 modules. The TAPELOG_OBS table in the input vex file should list the time ranges valid for each module. Jobs will be split at Mark5 module boundaries; that is, a single job can only support a single Mark5 unit per station. All stations using Mark5 modules will have DATA SOURCE set to MODULE in .input files. If file-based correlation is to be performed, the TAPELOG_OBS table is not needed and the burden of specifying media is moved to the .v2d file. The files to correlate are specified separately for each antenna in an ANTENNA block. Note when specifying filenames, it is up to the user to ensure that full and proper paths to each file are provided and that the computer running the datastream for each antenna can “see” that file. Two keywords are used to specify data files. They are not mutually exclusive but it is not recommended to use both for the same antenna. The first is “file”. The value assigned to “file” is one or more (comma separated) files. It is OK to have multiple “file” keywords per antenna; all files supplied will be stored in the same order internally. The second keyword is “filelist” which takes a single argument, which is a file containing the list of files to read. This “filelist” file only needs to be visible to vex2difx. This file contains a list of filenames and optionally start and stop dates (in one of the formats listed above). Comments can be started with a # and are ended by the end-of-line character. Like for the “file” keyword, the filenames listed must be in time order, even if start and stop dates are supplied. An example “filelist” file is below:

# This is a comment.  File list for MK for project BX123
/data/mk/bx123.001.m5a  54322.452112 54322.511304
/data/mk/bx123.002.m5a  54322.512012 54322.514121 # a short scan
/data/mk/bx123.003.m5a  54322.766323 54322.812311 

If times for a file are supplied, the file will be included in the .input file DATA TABLE only if the file time range overlaps with the .input file time range. If not supplied, the file will be included regardless of the .input file time range, which could incur a large performance problem.

A few sample ANTENNA blocks are shown below:

ANTENNA MK 
{
  filelist=bx123.filelist.mk
}

ANTENNA OV { file=/data/ov/bx123.001.m5a, 
                  /data/ov/bx123.002.m5a,
                  /data/ov/bx123.003.m5a }

ANTENNA PT { file=/data/pt/bx123.003.m5a } # recording started late here

ANTENNA default { networkPort = 320 }  # all antennas without ANTENNA setups will get this

Certain events cause a forced job break that could, in some cases, end up requiring many individual software correlations to complete processing of a project. Effort has been made to minimize the number of these cases. The following situations will cause a job break: change in clock model at a station, change of a Mark5 module, change in number of channels or sub-bands, multiple simultaneous subarrays, and leap seconds. Future versions of vex2difx and DiFX may relax some of these circumstances. Some parameters have defaults that may cause more job splitting than is desired (such as maxLength) and can be tuned.

The Mark5B format, including its 2048 Mbps extension, is now supported by vex2difx. The “track assignments” for Mark5B format has never been formally documented. Vex2difx has adopted the track assignment convention used by Haystack. Formally speaking, Mark5B has no “tracks”. Instead it stores up to 32 bitstreams in 32 bit words. The concept of “fanout” is no longer used with Mark5B. Instead, the equivalent operation of spreading one bitstream among 1 or more bits in each 32 bit word is performed automatically. Thus to specify a Mark5B mode, only three numbers are needed: Total data bit rate (excluding frame headers), number of channels, and number of bits per sample (1 or 2). The number of bitstreams is the product of channels and bits.

The $TRACKS section of the vex file is used to convey the bitstream assignments. Individually, the sign and magnitude bits for each channel are specified with fanout_def statements. In unfortunate correspondence with existing practice, 2 is the first numbered bitstream and 33 is the highest. In 2-bit mode, all sign bits must be assigned to even numbered bitstreams and the corresponding magnitude bit must be assigned to the next highest bitstream. To indicate that the data is in Mark5B format, one must either ensure that a statement of the form

track_frame_format = MARK5B;

must be present in the appropriate $TRACKS section or

format = MARK5B

must be present in each appropriate ANTENNA section of the .v2d file. As a concrete example, a complete $TRACKS section may resemble:

$TRACKS;
def Mk34112-XX01_full;
  fanout_def = A : &Ch01 : sign : 1 : 02;
  fanout_def = A : &Ch01 : mag  : 1 : 03;
  fanout_def = A : &Ch02 : sign : 1 : 04;
  fanout_def = A : &Ch02 : mag  : 1 : 05;
  fanout_def = A : &Ch03 : sign : 1 : 06;
  fanout_def = A : &Ch03 : mag  : 1 : 07;
  fanout_def = A : &Ch04 : sign : 1 : 08;
  fanout_def = A : &Ch04 : mag  : 1 : 09;
  fanout_def = A : &Ch05 : sign : 1 : 10;
  fanout_def = A : &Ch05 : mag  : 1 : 11;
  fanout_def = A : &Ch06 : sign : 1 : 12;
  fanout_def = A : &Ch06 : mag  : 1 : 13;
  fanout_def = A : &Ch07 : sign : 1 : 14;
  fanout_def = A : &Ch07 : mag  : 1 : 15;
  fanout_def = A : &Ch08 : sign : 1 : 16;
  fanout_def = A : &Ch08 : mag  : 1 : 17;
  fanout_def = A : &Ch09 : sign : 1 : 18;
  fanout_def = A : &Ch09 : mag  : 1 : 19;
  fanout_def = A : &Ch10 : sign : 1 : 20;
  fanout_def = A : &Ch10 : mag  : 1 : 21;
  fanout_def = A : &Ch11 : sign : 1 : 22;
  fanout_def = A : &Ch11 : mag  : 1 : 23;
  fanout_def = A : &Ch12 : sign : 1 : 24;
  fanout_def = A : &Ch12 : mag  : 1 : 25;
  fanout_def = A : &Ch13 : sign : 1 : 26;
  fanout_def = A : &Ch13 : mag  : 1 : 27;
  fanout_def = A : &Ch14 : sign : 1 : 28;
  fanout_def = A : &Ch14 : mag  : 1 : 29;
  fanout_def = A : &Ch15 : sign : 1 : 30;
  fanout_def = A : &Ch15 : mag  : 1 : 31;
  fanout_def = A : &Ch16 : sign : 1 : 32;
  fanout_def = A : &Ch16 : mag  : 1 : 33;
  track_frame_format = MARK5B;
enddef;

vex2difx is written in c++ and makes heavy use of the standard template library. This makes applying standard algorithms (sorting, traversing, associating) container members simple and error-free. An object-oriented approach is used. The base class for many of the classes is VexInterval, which is simply a pair of modified Julian days specifying a time interval. From this many other classes, such as scan, job, experiment, flag, … are derived. These classes are defined and implemented in vextables.h and vextables.cpp. This makes simple operations on VexInterval objects (such as sorting and determining overlap) apply automatically to the higher level objects. The vex parsing library from the Field System was borrowed from Goddard Space Flight Center. This code is duplicated with little change within the vex/ subdirectory of the vex2difx source tree. Source file vexload.cpp contains the code that calls the vex parser routines to populate the VexData structure which is then used as the model from which to make jobs. vex2difx uses the difxio library for writing DiFX .input and .calc files. Currently the .flag files are written natively within vex2difx, however this may change.

To aid diagnosis of an experiment and forming jobs, vex2difx keeps an internal list of events. An event could be the experiment starting or stopping, recording at a station starting or stopping, a leap second, an antenna joining or leaving a scan, and others. Event types are enumerated in the vextables.h source file.

Splitting of an experiment into one or more jobs is one of the main functions of vex2difx. The first step in this process is to divide the experiment into JobGroups. A JobGroup is a collection of scans that can be combined into one FITS file. Examples of cases where a JobGroup boundary must be made include changing number of spectral channels or polarizations. The JobGroup boundaries happen at exacting times, dictated entirely by the scheduled scans. The second layer of splitting considers media changes. Often there is a gap between the end of recording on one Mark5 module and beginning recording on the next. vex2difx aims to be smart about choosing when to split jobs to minimize the total number of jobs created.

List of remaining issues

• Add a “default” option for the antenna table
• Improve handling of case mismatches (e.g., Ny vs. NY)
• VDIF support
• Better handle modes/setups that don't use all provided antennas
• Extensive testing of many modes
• Support pulsar polyco with gate open/close support for simple gating
• Support nAntenna != nDatastream
• Don't require $DIFX_VERSION (see bugs below)
• Warn if source, scan, antenna, or mode name referenced from .v2d file is not in vex file
• eVLBI support
• Support time formats other than decimal MJD
• Mark5B support
• Set up correlation off disk files rather than modules
• Ability to select which baselines are retained or dumped (akin to antennas= )
• Support IF selection
• VLBA hardware correlator time alignment option
• Improve blocks per send calculation
• Support for polarization swapping
• Support for ANTENNA sections
• Improved ra, dec parsing
• Write .flag file indicating baselines/antennas to turf after correlation
• Allow setting DataStream buffer as a total size in MB or seconds (e.g., 256 MB or 10sec)
• Use links from GLOBAL and STATION tables to CLOCK, EOP and TAPELOG_OBS as appropriate (example file: gk022.vix)

• Core dumps if DIFX_VERSION is not set (fixed in DiFX-1.5 branch and trunk, 2010Mar04)
• In eVLBI (NETWORK) mode, warns about missing media specification and does not add the Network Table to input file (fixed in trunk, 2010July)
• Default AvChans is not 1

• By default, the “CORE CONF FILENAME” does not need to include the experiment name. “threads” is good enough (CJP) (set with threadsFile= 2010July)
• If only a single job is to be run, the output filename does not need the _# prefix. For the way ATNF eVLBI run this is particularly messy (as each time DiFX is run the wall clock start time is added to the file name)(CJP). (enabled by setting startSeries=0 2010July03)