1. Software Development Plan GBT e2e Software Review August 30, 2004 Nicole Radziwill (nradziwi@nrao.edu)nradziwi@nrao.edu

2. Development Team 1 Deputy Division Head (also Software Engineer) Amy Shelton (75%) 2 Astronomers Jim Braatz (50%), Bob Garwood (75%) 2 Software Engineer I Mark Clark, Joe Brandt 5 Software Engineer II Ray Creager, David Fleming, Paul Marganian, Melinda Mello, Eric Sessoms (1 Co-op Student) Ron DuPlain  However, only 7 FTE available for maintenance/development in 2004 (Given measured science time, other duties, and operational support – intend to raise this to 8 FTE in 2005 by reducing overhead and op support)

3. Scientific Support All GB staff scientists have a research allocations, and other duties which take the major fraction of their time. All GB staff scientists support visiting observers. –Since GBT is still not mature, this can be a large and variable workload (I.e. tracking down new problems which have not been previously identified). –Nature of work (on call, night-time) can be extremely disruptive. Main scientific support: –Ron Maddalena (also visiting observer support, Q-band commissioning) => 25% FTE primarily for data handling –Frank Ghigo (also VLBI, visiting observer support, Ka-band commissioning) => 25% FTE primarily for ease-of-use Less than ~ 1 FTE scientific support for all software-related projects combined!

4. Constituents Visiting Observers –Provide Operational Support and Real-Time Response to Faults Staff Scientists –Respond to problems encountered during receiver and program checkouts Electronics –Upgrading Managers –Responding to Configuration Changes Operations –Safety Issues –Antenna Stow/Movement for Maintenance Computing SDD –Still must maintain infrastructure, which requires effort Development Work –New projects –Continued upgrades to existing systems This chart does not show a distribution of effort, because different requests require different levels of effort Approximate Distribution of Worked Requests (Q1 to Q3 2004) * This chart shows worked requests only, and does not include operational support or overhead, including testing, or total requests. Everyone wants more effort allocated to their concerns!

5. Strategic Software Goals Strategic GoalProject Enable high-frequency operation of the GBT Antenna. PTCS Provide capabilities to generate new science from the GBT Caltech Continuum Backend, Penn Array Rx, Ka Band Rx, continuous stream of new instrumentation….. Increase scientific productivity by making GBT easier to use (also aids support scientists) Ease of Use Conserve vital high-frequency weather by supporting automated queue-based dynamic scheduling, and remote observing Ease of Use Provide reliable data analysis for all standard GBT observing modes Data Handling PRIORITY LOWER HIGHER Site strategic goals: New scientific capability trumps ease of use/ease of data analysis!

6. Strategic Software Goals - continued GBT is still a developing telescope in early operations. We consider robust, routine execution of desired scientific capabilities (e.g. high- frequency operation, Ka-band receiver, cross-correlation modes of spectrometer) as the highest priority. Pipelines and archive will be in greater focus when uniformly robust data are being collected, and especially when imaging capability is fully realized. Data analysis pipelines and archive have been lower priority compared to other project work, given the scientific and software effort available.

7. ObsProject ObsUnitSet Science Products consists of Scan Subscan Proposal Submission Tool Scheduling Block Execution Block ObsProposal Obs Prep Tool APIs Quick Look Results Calibration Results Quick Look Display (GFM) Tel Cal Tool Data Processing (*) Data Capture * Science products may be produced as part of the pipeline or in an offline mode Goal for the Logical Structure of a GBT Observing Project (software subsystem boundaries noted by dashed boxes)

8. Review of CY 2004 Major Goals & Accomplishments GBT Site Software Goals: [goals of non-sdd projects not listed] Significantly improve ease of use: Production release of new configuration and observing APIs with new data display Beta release of new data reduction package by the end of the year Produce design specifications for remote observing Improve systems reliability by completing the Linux Migration, and reduce telescope test time needed by completing simulators Production release of pulsar spigot modes for general use Results: Beta release of Observation Management System end August 2004; remainder of year to be used training staff astronomers and working out bugs. Standard Observing Modes and Data Reduction Cases documented, initial release of IDL modules in Q3 04  Insufficient scientific support to complete design specifications for remote observing Linux Migration on track for completion for all devices except Spectral Processor (which requires full software reengineering) by end of December; Simulators on track for October Spigot ease of use work to be conducted in September and October will make pulsar spigot modes general use

9. Goals for CY 2005-2006 Support engineering and commissioning of Caltech Continuum Backend and Penn Array Receiver PTCS - complete design for upgrade of telescope control system, to include device-to-device communication where necessary Support ongoing instrument development programs (e.g. 3mm receiver) Complete design study for pipeline processing solutions for Penn Array & OTF Imaging and implement a prototype pipeline at GB Complete data capture and calibration subsystems Eliminate the need for visiting observers to use engineering applications for all GBT standard observing modes, which requires implementing observing modes for near-earth objects, creating the spectral line data display, and (most critically) improving balancing Move to completely Scheduling-Block based observing by the end of 2005; move to dynamically scheduled SB-based observing by end of 2006 Implement data quality diagnostics system Reengineer IF system and Spectral Processor for reliability

10. Projects Indicates substantial software support required

11. Resource Utilization Projects do not occupy all of our development resources: * With respect to approximate hours spent on various tasks summed over development cycles in 2004

12. Desired Tasks for 2005 Project/Task (Includes Maintenance and New Development)Hrs Required/Yr One Plan… PTCS: Antenna, Active Surface, Auxiliary Sensor Work for critical path items5040 Instrumentation: Caltech Continuum Backend Commiss. (incl. data analysis)12606300 Instrumentation: K-band upgrade and Wideband Spectrometer Support840  Data: Complete Design Study on Pipeline Alternatives & SD Support in AIPS++16807980 Data: Prototype Pipeline Development for Penn Array & OTF Imaging at GBT16809660 Data: Complete Telescope Calibration Subsystem1260  Data: Complete Data Capture Module (Preprocessing API & Online Filler)1260  Ease of Use: Implement Near-Earth Observations84010500 Ease of Use: Implement Full Balancing Solution (acc to memo by Maddalena)126011760 Ease of Use: Spectral Line Quick Look Data Display84012600 Ease of Use: Source Catalogs Integration420  Ease of Use: Implement “Program” Level of Queue-Based Management1260  Software CM&E: IF Manager Root Cause & Improvements1680  Software CM&E: Spectral Processor RC & Improvements2100  Software CM&E: Bug Fixes126013860 13440 work hours (or 8 FTE) available in 2005+ * From target levels of 10% overhead and 10% OS in 2005 22680 work hours needed in 2005 – 68.8% oversubscribed * Uses a basis of 210 work days/year/FTE = 26.25 days per dev cycle = 210 hours per cycle. Divide Hrs Required/Yr by 210 to get # of FTE-cycles. 1680 hrs = 1 year of work.

13. Tradeoffs are Omnipresent We are consistently about 2-3 FTE understaffed for the projects that we have already committed to. Every 6 weeks the GBT Project Planning Committee meets to select between competing activities for the upcoming six weeks due to the understaffing. Each cycle, a lack of software and/or scientific support limits throughput of accomplishments. The tradeoffs which are considered at EVERY review include: Maintenance vs. New Development (e.g. Reengineering the Spectral Processor, which is highly unreliable, vs. performing enhancements to the active surface to support high frequency observations) Ease of Use vs. Data Handling (we recover more scientific staff by executing on ease of use tasks, but satisfy greater organizational pressure by working on data handling)

14. 1H 042H 041H 052H 051H 062H 061H 072H 07 OBS API PROGRAM LEVEL INTEG STOPGAPS 3MM ANALYSIS/SDM1 PTCS UPGRADES TO MONITOR & CONTROL DESIGN STUDY FUTURE INSTRUMENTATION PENN ARRAY EASE OF USE DATA PROJECT OTHER CCB GBT Software Timeline IDL CFG API SBs KA/MM FULL FXNALITY PROTO-PL/NEW DISH PIPELINE DATA CAPTURE NEW OBS MODES

15. Conclusions GB SDD understaffed in comparison to planned work by ~2-3 FTEs GB strategic priorities are not identical to those of ALMA, EVLA –Pipeline processing has been less of a driver in the past; just coming into focus with Penn Array on the horizon Available resources are: –~ 2 FTEs SDD + 0.25 Scientific Staff for all “observing” related software development. –~ 2 FTEs SDD + 0.25 Scientific Staff for all “data analysis” related software development. Lack of SDD effort causes software-only projects to slip, but effects are contained within that project. Lack of SDD effort for other projects disrupts the efforts of all other disciplines. => multi-disciplinary needs often trump EoU/Data Handling