[IPTA-DBPL] PPTA: Data processing/ToA database

Mon Jul 15 19:51:13 EDT 2013

Hi all,

 This is the message that I sent around before to a small email distribution list about the PPTA pipeline.

Cheers,
 George

Hi Patrick et al.,

 I have put together a document describing how the PPTA pipeline works and, at the end, given some notes on what I think is "good" and what is "bad" about our scripts (the document is attached to this email).  Hopefully this will help if we try and use a system similar to TOASTER for our pipelines and/or ToA access.  Please let me know if anything isn't clear or if you'd like more information.

Cheers,
 George

George Hobbs
Research Scientist
Astronomy and Space Science
CSIRO

Phone: +61 2 9372 4652
george.hobbs at csiro.au<mailto:george.hobbs at csiro.au> | www.csiro.au<http://www.csiro.au/>
Address: PO Box 76, Epping NSW 1710
Australia (Cnr Vimiera & Pembroke Roads, Marsfield NSW, 2122)

On 04/07/2013, at 7:33 AM, Patrick Lazarus <plazarus at mpifr-bonn.mpg.de<mailto:plazarus at mpifr-bonn.mpg.de>> wrote:

Hi All,

On a few occasions during the IPTA Meeting last week, the question of what data products and information (in addition to the current IPTA timfile format) are required/useful for creating combined IPTA data sets, and for data analysis projects. NANOGrav, the PPTA and the EPTA are each developing different solutions to this problem. Ultimately, the data produced by each of these consortia will be combined into a single IPTA data set. Thus it makes sense for database, data reduction, and data combination groups across the IPTA to discuss requirements and possible solutions.

It is likely too late to collect all the information we'd like for the legacy data sets. However, now is probably the right time to start thinking about what is required for the next-generation data sets.

Following a brief discussion, it was mentioned that at the very least, the pipeline/databases of each of the three PTA projects should produce/track some common set of basic meta-data and diagnostics. Also, having compatible database schemas would likely be beneficial.Seeing as most of us are likely more interested in doing science, and not donkey-work, it may make sense to jointly develop tools, where possible.

With this in mind the following plan was proposed to stimulate discussion and collaboration:

0. (By July 8th) - Get a list of people involved in DB/Pipeline from NANOGrav, EPTA, PPTA. Create an exploder/mailing list.
1. (By July 15th) - Send details of current pipelines and what output products are made and how they are stored. Highlight details on what aspects of our databases/pipelines we think are good and which are bad.
2. (By July 22nd) - Comment on the information sent around to try and come up with a set of user-requirements for an IPTA pipeline/database.
3. (Around July 22nd) - Connect with IPTA data combination group to find out what would have found useful for the current IPTA data set.
4. (At some point) - Connect with the IPTA steering committee to let them know of our plans and ideas and to find out who else should be part of the process.

Please share your comments, objections, concerns, etc.

Looking forward to work together,
Patrick

On 2 July 2013 04:47, Robert Ferdman <rferdman at physics.mcgill.ca<mailto:rferdman at physics.mcgill.ca>> wrote:
Hi Scott et al.,

I'm not sure whetehr Ryan or Glenn are on the nano-time list, so I've cc'd them here as well.

For future reference, the NANOGrav DB group's exploder is nano-db at nanograv.org<mailto:nano-db at nanograv.org>.

Cheers,

Rob.

-----------------------------------------------------------
Dr. Robert Ferdman
Research Associate

Department of Physics
McGill University
Ernest Rutherford Physics Building, room 315
3600 rue University
Montréal, QC
Canada H3A 2T8

On 1 Jul 2013, at 09:50, Scott Ransom <sransom at nrao.edu<mailto:sransom at nrao.edu>> wrote:

Hi Patrick,

The relevant people are all (I think) on the nano-time mailing list. Plus Nate and Adam, who I've cc'd.

Scott

On 07/01/2013 09:36 AM, Patrick Lazarus wrote:
Hi George,

I agree with your recap.

I suggest we add

0. Get a list of people involved in DB/Pipeline from NANOGrav, EPTA,
PPTA. I'll create an exploder/mailing list.

Scott, can you let us know who from NANOGrav we should be in touch with.
It would be good to get input from the DB group, as well as some of the
people responsible for the data reduction.

Thanks,
Patrick

On 29 June 2013 09:22, <George.Hobbs at csiro.au<mailto:George.Hobbs at csiro.au>
<mailto:George.Hobbs at csiro.au>> wrote:

   Hi Patrick,

     Thanks for the organising the meeting with us on TOASTER.  It does
   look great and I'm sure that we can move forward with that (or
   similar) system and really improve the data sharing within the IPTA.

   My understanding is that we were going to do something like the
   following.  Did I remember correctly?

   1. We'd all send you details of our current pipelines and what
   output products are made and how they are stored
   2. We'll include details on what aspects of our databases/pipelines
   we think are good and which are bad
   3. We'll all comment on the information sent around to try and come
   up with a set of user-requirements for an IPTA pipeline/database
   4. We'll then see whether TOASTER meets this requirements, whether
   it could with some changes or whether we should start again and
   design a system from scrath
   5. At some point you'll connect with Joris and his group to find out
   what the "data putting together group" would have found useful for
   the current IPTA data set
   6. At some point we'll all connect with the IPTA steering committee
   to let them know of our plans and ideas and to find out who else
   should be part of the process

   Is that okay?  To keep pushing this forward should we agree on
   getting you some comments on the current pipelines within 2 weeks?

   Cheers,
     George

   *George Hobbs*
   Research Scientist
   Astronomy and Space Science
   CSIRO

   Phone: +61 2 9372 4652<tel:%2B61%202%209372%204652> <tel:%2B61%202%209372%204652>
   george.hobbs at csiro.au<mailto:george.hobbs at csiro.au> <mailto:george.hobbs at csiro.au> *|*
   www.csiro.au<http://www.csiro.au/> <http://www.csiro.au/>
   Address: PO Box 76, Epping NSW 1710
   Australia (Cnr Vimiera & Pembroke Roads, Marsfield NSW, 2122)

--
Scott M. Ransom            Address:  NRAO
Phone:  (434) 296-0320<tel:%28434%29%20296-0320>               520 Edgemont Rd.
email:  sransom at nrao.edu<mailto:sransom at nrao.edu>             Charlottesville, VA 22903 USA
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989

--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pulsarastronomy.net/pipermail/ipta-dbpipeline_lists.pulsarastronomy.net/attachments/20130715/0060112a/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pipeline.pdf
Type: application/pdf
Size: 660166 bytes
Desc: pipeline.pdf
URL: <http://lists.pulsarastronomy.net/pipermail/ipta-dbpipeline_lists.pulsarastronomy.net/attachments/20130715/0060112a/attachment.pdf>