[IPTA-DBPL] Description of EPTA DB/Pipeline

[Patrick Lazarus]

Hi All,

Here's a description of TOASTER, the EPTA's unified data archive, pipeline,
and TOA database. The data files input to TOASTER are multi-channel,
multi-subint, RFI-cleaned, calibrated folded archives. The steps for
removing RFI and calibrating data differ between telescopes:

Data preparation

*Nancay* (from Greg Desvignes):

   - NUPPI data are flux and pol calibrated using pac. Bad channels are
   identified and removed using a psrsh script. No pcm calibration is done as
   it should have little effect for Nancay data (the parallatic angles are
   very small).

*Jodrell* (from Cees Bassa):

   - ROACH data are passed through the spectral kurtosis algorithm in
   real-time, bad channels are identified and removed using a psrsh script.
   Data are not yet calibrated.
   - DFB data are prepared similarly to ROACH, but no spectral kurtosis.
   Instead they are cleaned by hand.

*Effelsberg*:

   - Asterix data are cleaned by identifying bad profiles (i.e. chan/subint
   pair) using a script using the PSRCHIVE-python bindings. This is similar to
   median zapping with paz, but operates on each subint separately and uses
   multiple statistics to identify RFI. We have started polarization
   calibrating the data using pac, but there are still some kinks to work out.

*Westerbork* (from Gemma Janssen):

   -
   - The WSRT array is calibrated as an array regularly (phase and
   polarisation) before each timing run. Apart from this, no other calibration
   is done. For the higher frequencies (1380 and 2273MHz) in most cases there
   is no need for RFI zapping, however at 92cm there are usually a few narrow
   channels that need to be zapped which is done by hand on a case-by-case
   basis.

*Sardinia* is still being commissioned.

TOASTER

TOASTER (“*TOAs* *T*rack*er*”) is an integrated data archive, timing data
reduction pipeline, and TOA database developed by the EPTA to store its
next-generation data and TOAs with sufficient meta data and processing
information to complete reproduce all analysis performed through the
package.

The TOASTER package contains a series of command-line driven programs
designed to interact with a database. The code is compatible with multiple
SQL variants. It has been tested with MySQL and SQLite. All interactions
with the database (inserts, updates, deletions, and querying) are mediated
by TOASTER programs. These programs include substantial error-checking to
prevent the database ending up in an inconsistent state. Additionally, the
database itself requires corresponding entries to exist, if they are
referred to by another entry. This is accomplished with “foreign key
constraints”.

The TOASTER includes programs, and database tables for storing information
about pulsars, telescopes, observing systems, folded archives, templates,
and .par files. For each archive loaded into TOASTER a series of diagnostic
plots and values are automatically computed and also stored in the
database. These diagnostics can also be computed and added to the database
at any time. It is also possible to get overview plots of folded archives
stored in the data archive.

TOASTER has an integrated pipeline that can be used to reduce folded
archives. The resulting TOAs are stored in the database with all
information required (including software version numbers) to reproduce the
analysis, as well as all meta data as requested by the (extended) IPTA .tim
file format. At the core of the pipeline are “manipulator” plugins. Each
manipulator defines a compute-function that takes as input a single folded
archive and reduces it into an output archive. At present, within the EPTA,
the pipeline is used to simply scrunch input archives to a user-defined
(using command-line arguments) number of channels, subints and bins. The
input data files are assumed to be RFI-cleaned and calibrated before being
reduced. In principle this could be expanded to include RFI-excision and
calibration.

The integrated pipeline computes and stores diagnostics (values and plots)
about the reduced archive that TOAs are computed from. Also, pat-produced
3-panel diagnostic plots (profile, template, and difference) are
automatically produced and stored in the database.

TOAs not produced by the integrated pipeline can also be loaded into
TOASTER. This is used, for example, to load EPTA legacy TOAs into the EPTA
database, or for loading IPTA TOAs. Obviously, much of the ancillary data
is missing for these TOAs, making it impossible to fully reproduced these
results. However, the advantage of loading orphaned TOAs into TOASTER is
that they are stored in the same place as TOASTER-produced TOAs, and
accessible from the same interface.

TOAs are stored in a format-independent way, allowing them to be written
out in multiple formats. The “Princeton” and “TEMPO2” formats are
implemented. Future formats (e.g. XML, binary) could be added. The
TEMPO2-format allows flags consisting of TOA meta data to be added to each
TOA line. A single command-line argument allows TOAs to be written out with
all flags required by the (extended) IPTA format.

Overview plots (histograms, observing cadence) of all (or a subset) of TOAs
stored in the database can be easily computed.

Notes

   - TOASTER programs provide some basic functionality for pipelining
   components together.
   - TOASTER is configurable with a config file.
   - No intermediate data products are stored (other than diagnostics).
   - TOASTER is written entirely in python.
   - TOASTER code is commented and documented.
   - The beginnings of a web-interface to TOASTER has been implemented.
   (This is done in collaboration with the McGill/PALFA group.)
   - TOAs can be marked as “bad” in the database. Also, users can store
   comments about TOAs in the database.
   - Data files and plots are not stored directly in the database. Instead,
   basic information, and pointers to the file system are stored. This keeps
   the database comparatively light.
   - PSRCHIVE is required.

What's Good

   - Supports multiple telescopes/observing systems.
   - Easily configurable.
   - Stores versioning information.
   - Unified data archive, pipeline, database.
   - TOAs produced with integrated pipeline are fully-reproducible.
   - Documented and commented.
   - Extensive error checking.
   - Automatically generated diagnostics.
   - Components can be combined in a command-line pipeline.
   - Manipulator plugins allow multiple pipelines to be added (i.e. no need
   to re-invent the logistical framework).
   - Pulsar aliases, and telescope aliases are stored in the database (i.e.
   pulsar J-names, B-names, and other variants can all be used transparently).

What's Bad

   - No automatic RFI-excision or calibration (may be added with new
   manipulator plugins).
   - Manipulators can only take a single folded archive as input.
   - Only one manipulator can be applied to each input file.
   - Extra TOA information (e.g. time-offsets) are not stored in the
   database.
   - Pipeline runs sequentially.
   - No support for paas-produced ”.m” files.
   - Folded archives and templates must be PSRCHIVE-compatible. (vap is
   used to fetch header information to populate database.)
   - Archive headers must contain a base-set of information.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pulsarastronomy.net/pipermail/ipta-dbpipeline_lists.pulsarastronomy.net/attachments/20130715/9841d0b8/attachment-0002.html>