[IPTA-DBPL] NANOGrav pipeline and database description

[Robert Ferdman]

Hello everyone,

The following is a short description of the NANOGrav processing pipeline and database description.

In addition, there is some interest from the timing and database groups here (and especially from those who were not present at the Krabi meeting) that we basically recap the discussion that was had there, and that everyone can get a look at the TOASTER demo.

We would be interested in having a telecon at some point in the future for doing the above, and in particular so that we can all discuss the needs and wants for an IPTA database (apart from just TOAs, which everyone wants :).

Thanks, and talk soon,

Rob (on behalf of the NANOGrav database working group).


Pipeline:

NANOGrav has typically run multiple data processing (calibration through TOA generation) pipelines in parallel in order to cross-check the results, compare different methods, etc.  The current psrchive-based processing scripts handle both ASP/GASP and GUPPI/PUPPI data (and in principle any psrchive-readable format), and do the following steps:
Initial data combination and partial scrunching (if necessary)
Polarization and flux calibration: both full-mueller-matrix ("pac -S") and "ideal feed assumption" (pac -x) calibration versions are computed, if an appropriate mueller matrix solution is input (not yet done for all receivers).
RFI removal: Blank a static set of bad frequencies per telescope/receiver, then follow up with paz's median zapper.
Additional scrunching to desired time/freq resolution for TOAs.
Template profile generation: full-sum then wavelet denoising (psrsmooth)
TOA generation
The processing uses a combination of python scripts and make (for dependency tracking, running parallel processing jobs, etc).


Database:

The NANOGrav Data System stores or collates Raw profiles, intermediate profiles, templates, TOAs, .tim files and ephemeris. The system as a whole loads raw data and data products into the database and makes them available, based on user permissions, to the community.

Details:
The system loads, store and retrieves raw data and the products of the timing analysis group.  This includes, but is not limited to:
TOAs
ephemerides
template profiles
intermediate profile data products
ISM-related/cyclic spectroscopy-derived data prodcuts
perhaps will at some point store GW analysis-related products
The timing and GW analysis is left to specialists; the system is designed to store and retrieve associations between data and data products
Web interface is written in python (which means that retrieval scripts in it are portable), and is used to perform queries and retrieve data, and eventually will allow the user to upload certain data products
Data files themselves not kept in database; instead file location is detailed.  When a user wants to retrieve a data file (either raw or intermediate/processed), they:
perform a query using the web interface
request those data files that result from the query
data file are then copied to a separate server for ftp downloads, where they remain temporarily
user is notified once data is ready to be retrieved, with the location of the data
user downloads data to local machine
As stated above, the system is designed to store any intermediate profiles deemed necessary by the timing analysis group (but doesn't require it)
We have made it a requirement that detail of processing steps (i.e. pipeline info) are associated with data products and are also stored in the database
The aim was to impose minimum constraints on the timing analysis group workflow, so that they can judge what they want to store and how to describe it
The loading of data products is automated, and will typically be done after each observing session is completed.  Once uploaded to the Cornell server (and to UBC for backup purposes), the data is then stored and recorded in the database, and just requires a defined file which describes the association between products

Good:

Stores location information for raw data with associated observation metadata
Stores intermediate data products for any use
Matches working practices of our scientists
Flexible; stores associations between files and products without assumptions
Tracks full provenance (and custom recursive query returns user-specified generations of predecessors)
Designed to allow multiple (experimental, competing, etc) pipelines
Clean separation of storage, loading, processing and interface layers, allowing separate development of each
Loads asp/gasp and puppi/guppi files but extensible to any desired raw data file format.  This will be especially useful for the eventual addition of historical data sets such as those from mark 2, 3, and 4 backends, ABPP, and/or PSPM
Requirements on timing group are light
Interface produces some basic diagnostic plots and will be extended to include more

Bad:

Relies on the scientists to upload some information about how products were made for full record-keeping; this is the flip side of the flexibility (but within a pipeline it is generally automated)
General flaw with relational databases, but *direct* expert retrieval requires the user to have some proficiency with SQL (typical retrieval is easily made to be automataic)


-----------------------------------------------------------
Dr. Robert Ferdman
Research Associate

Department of Physics
McGill University
Ernest Rutherford Physics Building, room 315 
3600 rue University
Montréal, QC
Canada H3A 2T8   



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pulsarastronomy.net/pipermail/ipta-dbpipeline_lists.pulsarastronomy.net/attachments/20130715/289e9694/attachment-0002.html>