SSC SI Data Processing Pipeline Plans Tom Stephens USRA Information Systems Development Manager SSSC Meeting – Sept 29, 2009

Going Forward There’s more work to do –Increased scope of existing pipelines –Additional pipelines for non FSI instruments –NASA is increasing the project management requirements for the pipeline development The goal is for the Science Center to shoulder the vast majority of the burden of this new work –We will deal with the increased software development requirements –We will take on more user support functions in regards to general investigators –Allow the instrument teams to focus on the instrument and their science However, some increased involvement from the instrument teams will be required.

Centralizing the Pipeline Development Final production pipelines will be implemented at the Science Support Center (SSC) –Goal is to develop a unified system that is easier and more efficient to develop, use, maintain and manage Benefits –Standardized code base in a controlled set of languages and standards Initially C/C++, IDL, Python and JAVA Eases maintenance Reduces learning curve for new development team members –Unified code base Maximize reuse of pipeline components –Simplify development –Ease maintenance –Unified interface Probable that calibration pipelines cannot be fully automated Reduces friction when operators need to run different pipelines –Improved project management Maximize efficient use of resources Ability to focus resources on problematic/challenging areas

Roles and Responsibilities – Instrument Teams Provide algorithms for data reduction –Can be in any “reasonable” format Teams have software they are using for their own data reduction to do their science, this can be provided “as is” to help build the official pipelines Science Center staff will translate what is provided into pipeline code to run in the DCS –Includes documentation on software and algorithms –Also includes sample test data Work with SSC staff to validate implemented pipelines Work with SSC staff to support guest investigators using instrument for observing –This will proceed along the currently established plans The goal is for the instrument teams to not need to provide substantive additional support over what they are currently contracted to do

Roles and Responsibilities – Science Center Responsible for final implementation of pipeline code –Coding –Integration into the DCS –Documentation –Testing/Validation –Maintenance Responsible for ensuring suitable calibration data is taken for pipeline reduction Operates pipelines and validates output science data

SSC-SI Team Interface Points Each instrument will have a primary and backup Instrument Scientist at the SSC –Understands the instrument –Works with instrument teams to develop pipeline algorithms –Assists in validation of developed pipeline –Help support guest investigators with observation planning and archive data –Helps to define calibration observations –Provide SSC institutional knowledge about instrument One or more software developers will also work with the instrument teams, through the instrument scientist, to implement and test the data pipeline code. Implemented software will be provided back to instrument teams for their own use if desired.

Initial Impact – FSI teams Initial pipeline development will proceed as currently planned –Pipelines will be developed and delivered to SSC –SSC will take ownership and be responsible for future maintenance and changes We will want to get preliminary deliveries of your pipeline software sooner rather than later –We need to work out the mechanism for delivery –Integrate software with the DCS early and often –SSC staff are willing to assist with development if desired or needed

Initial Impact – Other Instruments Increased interaction between SSC staff and SI team –Improve SSC understanding of instrument –Improve SSC understanding of data analysis Work with SSC staff to develop algorithms/sample code for data processing –Use “whatever” is comfortable to instrument teams –Will take already existing tools/software –SSC will do the work of fitting the algorithms into the DCS pipelining system

Longer Term Impacts Focuses on maintenance aspects of instrument and pipeline development –Should not be major additional effort over current plans Instrument teams provide input to SSC during operation/maintenance period after initial pipelines developed –Improved algorithms/techniques for data reduction/calibration –Pipeline modification due to changes in instrument response or capabilities –Work with SSC staff to validate implementation and data produced by processing pipelines SSC continues to be responsible for actual implementation, operation and maintenance

Staffing This is an increase in the scope of the SSC’s pipeline development work –More and expanded pipelines –Increased software development process overhead –Additional staff will be required to accomplish this work SSC is working on a detailed staffing plan to address the development effort Details are still being worked with NASA Final development timeline and impact to SI teams will depend on the final details

Timeline Goal is to have a fully functioning pipeline implemented for each instrument as they start their official commissioning flights after the 2011 down time –However we’d like to get the existing FORCAST pipeline installed in the DCS for early science as a test for the SSC Not an Early Science Requirement Help us identify and address issues with integration Timing of commissioning will drive ramp-up of pipeline development effort and staff over the next two years –Start with re-integration of FORCAST & FLITECAM pipelines –Integrate FIFI LS pipeline –Using “lessons learned”, expand work to implementation of expanded pipelines and develop pipelines for other instruments –For a given instrument we plan to start implementation ~2 years before commissioning flights are scheduled to begin

Summary There is a lot of work to be done to implement the SMOR-II recommended pipelines The majority of this effort will fall on the SOFIA Science Center We do not want to place any additional burdens on the instrument teams than are absolutely necessary We are developing detailed plans of how to implement this work

Backup slides

High Level Development Process Pipeline/algorithm delivery from SI Team SSC Pipeline Implementation Error and Issue Reports Pipeline Validation Improvements and updates developed by SI teams and SSC Pipelines deployed to production