October 16, 2013 presentation to the NAOJ Workshop

Funds every field of science – mostly as grants to individual researchers at US universities and non-profit organizations ~ $7B annual budget: ~$1.3B for Mathematical and Physical Sciences (MPS) About $260M for Astronomy More than $130M annually to operate existing astronomical observatories Fraction spent on infrastructure increasing due to Atacama Large Millimeter Array (ALMA) operations and expenditures for mid-scale instrumentation and infrastructure. Future budget outlook: ~flat Physics and Astronomy outlook: declining multi-disciplinary initiatives and new applied research programs compete with core disciplinary research, reducing core programs in recent years 2

Some have been spectacular successes: 3 Cornell Electron Storage Ring (CESR) -- high-luminosity 6+6 GeV electron- positron collider at Cornell University > 30 years operation And some were not: Project Mohole - an ambitious attempt to drill through the Earth's crust into the Mohorovicic discontinuity Many lessons learned!

Six large facilities under construction: Advanced LIGO, Advanced Technology Solar Telescope, Alaska Region Research Vessel, Atacama Large Millimeter Array (with Japan, and nearly complete!), National Ecological Observatory Network, Ocean Observatories Initiative Two recently completed smaller scale facilities: National Center for Atmospheric Research’s Wyoming Supercomputer Center, ALVIN Replacement Human Occupied Vehicle Large Synoptic Survey Telescope One may start construction in FY 2014 Depends on Congressional funding Planned complete in 2022? 4

NSF doesn’t have a long-term “facility roadmap” of future plans NSF doesn’t manage big projects or operate big facilities (except in Antarctica) NSF oversees planning, construction, and operation, but the skilled project managers and technical staff work for a university or non-profit organization (the “awardee’), who is responsible for project accomplishment Construction funded through a separate Congressional appropriation The awardee contracts with major industrial firms to design and build the project. NSF expects awardee to be capable of managing very large (>$100M) subcontracts to industry Funding to Awardee via “Cooperative Agreement,” not a contract. This means NSF has substantial project involvement. For example: NSF shares project risk with Awardee, necessitates risk management plan NSF can make management changes at Awardee NSF has latitude to change scope and budget, if necessary NSF approves major expenditures and significant changes to plan NSF requires monthly progress reporting of financial and technical status 5

Project planning A new project must have a credible budget estimate, a resource-loaded schedule, a sound management plan, and a core team capable of accomplishing the project Assessment and evaluation NSF asks independent experts to evaluate the merits of the proposed science, and the soundness of the project plan, including the budget estimate NSF prioritization and budget planning The most important factor for NSF is whether it can afford to operate the new facility after construction is complete In a flat budget environment, this means that there is a competition between new opportunities and areas of ongoing support – what is the most important science? The next few slides provide more details about these 3 topics 6

NSF expects projects to develop planning using sequentially increasing maturity stages: Horizon: Define Quantitative Science Goals and Objectives; derive functional requirements and top level technical requirements Develop parametric “top down” cost and schedule estimates, based on scaling historical data  Conceptual Design Report Develop “bottom up” cost and schedule estimates, based on detailed engineering design, site specific factors, input from industry, interconnection of functional elements, estimates of risk and uncertainty, Project Execution Plan  Preliminary Design Report Develop “ready to build” plan, based substantially on supplier estimates, recruit key staff, secure site permits  Final Design Report Conceptual Design is like the outline of a book – with all the structure and chapter headings. The Preliminary Design and Final Design activities fill in the details. Final Design Report = instruction manual for building facility. Generally similar processes at US Department of Energy and NASA 7 Preliminary Design Final Design Construction Operations Horizon planning and Conceptual Design

Horizon Assessment: NSF internally prioritizes among candidate projects. NSF Director authorizes NSF funding of activities leading to Conceptual Design Report Conceptual Design Review: Results in rough definition of project scope and coarse estimate (>50% uncertainty) of construction budget. National Science Board (NSB) authorizes significant funds for planning to develop Preliminary Design Report Preliminary Design Review: Sets “reference baseline” definition of scope, budget, schedule, and risk. If review committee satisfied, and NSB continues to place high priority on the project, request Presidential concurrence (Office of Management and Budget) followed by Congressional appropriation of $ Final Design Review: Assesses whether project is ready to start construction. If review committee satisfied, NSB authorizes project team to start spending construction money 8

National Science Board takes on the very challenging task of prioritizing new facility projects across disciplines Extremely hard: telescopes vs ships comparison OMB/Congress expectation that annual construction budget < $200M, so resource loaded schedule and annual construction budget projections are critical factors  Need high confidence in Total Project Cost estimate, but also well supported annual budget estimates for each project year! NSF and NSB are reactive: strong, well organized and continuous community advocacy is very important influence on prioritizing Broader societal impacts are a big factor in decision: number of users, education and training opportunities, open data access NSB can and does re-prioritize 9

Cost estimates are inherently biased too low: They tend to omit or overlook necessary scope They are usually over-optimistic regarding technical difficulty of planned work NSF has a “no-overrun policy” – budget shortfalls in construction arising from known risks must be mitigated within budget, and by de-scoping if necessary. Strong reluctance to go back to Congress and ask for more $ But NSF is not crazy – ATST budget augmented by $34M this year to recognize cost increases due to prolonged legal challenges to Hawaiian site use Message to communities not to bring an immature project plan and overoptimistic budget request to NSF 10

In a flat budget environment, the capacity to sustain operation is the principal factor in deciding whether to undertake new construction But operating cost projections are almost always biased low because they overlook essential scope and underestimate technical difficulty and risk Perform full life-cycle cost estimation early, and update regularly: Operating cost projections Budgeting for science use Termination and contingent liabilities (partnership withdrawal, for example) 12

Construction activity requires a big pre-construction investment (5- 25% of total capital cost, and studies have shown that greater preconstruction spending results in more accurate predictions of actual project cost) Project management is ~10% of Total Project Cost (TPC) From idea to groundbreaking requires ~10-20 years Earned Value Management (EVM) Reporting is one of many important diagnostic indicators: By the time a project is 10% complete, EVM cost/schedule variances should be meaningful. Good managers use EVM as a management tool, not just a reporting tool Early cost/schedule problems are leading indicators of bigger problems later – pay attention to early warnings! Costs to operate industrial strength project management controls software ~1-2% of total construction cost. It’s worth it! 13

Risk adjusted cost is for “known unknowns” Known unknowns = risks known to the cost estimators This includes provisions for factors such as commodity price volatility, foreign exchange uncertainty, and uncertain start dates due to delays, including those caused by Federal Gov. Uncertainty in Federal appropriation process and schedule are part of the landscape for Project Management and budgeting, include these factors in the risk-adjusted budget 14

Awardee’s Project Execution Plans are expected to follow the Project Management Body of Knowledge (PMBOK) to develop a robust Risk Management Plan (5 step process) Budget and schedule estimates are similar to physical measurements; they have values and uncertainties. Each uncertainty can have random and systematic components NSF develops its own “Internal Management Plan” to manage its own risk in overseeing the project: Identify risks to agency, especially early indicators of future trouble, possible remediation “Unknown unknowns” (=risks unknown to the cost estimator) are funding agency’s responsibility, and are dealt with at program policy level.  Maintain flexibility and reserve capacity to adopt alternate courses of action in future planning 15

International collaboration may make it possible to undertake new projects too expensive for any single country alone Coordination with project planning and assessment becomes even more challenging with multiple funding partners! Partnership agreements must recognize the unique roles and responsibilities of each partner, and NOT treat partners symmetrically For example, NSF funds and oversees large projects, but does not manage or directly operate them US strongly endorses open researcher access to facilities based on merit review. Opposition to European “Pay to play” model. Experimental groups do not contribute to operating costs of US accelerators (ICFA guidelines), telescopes, etc. 16

US open data access policies may create partnership disincentives in some cases: why pay for operations when you can get the data for free? Uncertain annual appropriation processes are true for many countries, and complicate construction and operation budgeting Partnership agreements need to promote long term stability International partnership agreements must enable individual partners to exercise their expected stewardship roles as overseers of sovereign funds Big projects are inherently part of political dialogue because of the size of projected budgets Projects have foundered when political influence has resulted in premature project start with incomplete plans (ITER, SSC, DUSEL) and there has been painful re-scoping with others (ALMA, …) 17

Partnership agreements are founded on good will – structure must be carefully planned to align interests to avoid destabilizing collaboration. Include means for prompt dispute resolution! International partnerships have intrinsic overhead costs that must be recognized, and different partners have different costs. In-kind contributions may have a cost to the other partners. Robust systems engineering, spanning partner boundaries, is essential Conduct and manage a single project rather than parallel activities Need to synchronize development process when there are other partners or potential partners Define the appropriate governance models for construction and operation Extraordinary projects are successful when led by extraordinary people. Detailed policies and agreements don’t compensate for this. 18

Annual NSF facility plan – see on-line meeting materials for NSB February board meetings NSF Large Facilities Manual – available at GAO Cost Assessment Guide – see especially the discussions of cost and contingency estimation, and the many helpful case studies PMBOK – see especially its discussion of risk management AACE-I – international organization that describes best practices for cost and risk estimation 19