Presentation on theme: "The High Cost of Regulation A View of DO-254 Dr. Steve Suddarth Director (505) 803-2684 1."— Presentation transcript:
The High Cost of Regulation A View of DO-254 Dr. Steve Suddarth Director (505)
Outline About Standards About Development Philosophy What DO-254 Means for Us Steve’s Personal Opinion
Why Do We Have Standards (good reasons) Set expectations –Safety (DOT regulations) –Reliability (MIL-Spec) –Quality (five star hotel) Ensure compatibility –Interfaces (CDMA) –Form factors (Shipping Containers) –Portability (C++) Determine Style –Management (ISO9000)
Why Do We Have Standards (bad reasons) To settle an open argument –PI = 22/7 ? –1861 Springfield Rifle As a means of protectionism –Large contractors – big process As a means of “other” political control –Bureaucracy instead of representation As a means of delay or denial –Egypt and Coptic Church licenses
Most Standards are Enforced (many in law) U.S. Constitution 14 CFR AC Common Law / Precedent DO-254 DO-178B 10 U.S.C. AFR, etc. Fed. Acq. Reg (FAR) Fed. Av. Reg (FAR) Contract
Regulatory Burdens An Example Year – 1998 –241 Acts of Congress –4,899 regulatory rules! –68,591 pages of regulation in Fed. Register! The decisions it makes, have as much sense as this picture
A Classic Acquisition Story What: Henry Repeater Rifle Available To: Entire Union Army When Available: Start of Civil War Status: Refused by Quartermaster-General Why: –No clear requirement –Not military rugged (later proved effective anyway) –Increased demand for ammo –Not good classic soldiering Eventual outcome: Soldiers bought their own repeaters – used with great effect Other outcome: Countless needless casualties Confederate Assessment: “That damned Yankee rifle you can load on Sunday and fire all week!” Source: Azriel Lorber, Misguided Weapons
Some Safety Stories Commerce Sec’y Ron Brown + 36 others – 1996 –Mountain crash on instrument approach –ADF approach (1930’s technology) –No GPS! (Not yet approved / certified) Vs.
Safety Stories (cont.) Mel Carnahan (Missouri Gov) & Son – 2000 –Vacuum pump failure (1930’s technology) Serious, acknowledged issue! –Preventable with electronic flight systems No practical solutions certified in 2000 for light aircraft Vs.
The Great Systems Management Debate Agile / Spiral Development –Development begins as early as possible –Goal is prototype ASAP – improvement later –“Spiral”, repeated or continuous development is encouraged Waterfall / Formal Methods –Process starts with rigorous requirements –Top-down traceability –Goes all the way to pseudo-code or actual code –Coding does not begin until requirements are set –Some linkages and concepts may be provably correct
Agile vs. Waterfall Methods Arguments can be made in favor of either extreme Developers/Managers often mix methods among most successful companies/efforts DO-178B and DO-254 do not allow for such flexibility –They come down hard on the side of formal methods only
My Summary of DO-254 Developed by standards body – RTCA Formally enforced by FAA through AC Covers all “complex electronic H/W”: –FPGA, ASIC, PLD, … Openly stresses: –Requirements capture –Requirements tracking –Process assurance –Config Mgt. –V&V Requires certification that the entire design process was done since the beginning DOES LITTLE TO ACTUALLY TEST OR REQUIRE RELIABILITY!
Key DO-254 Philosophical Points Commonly shared between –DO-254 –DO-178B (Software standard) Object-Oriented Methods are Suspect –Dislike of dynamic memory / heap operations –Data hiding should be avoided (e.g. “private” variable declarations) –Fear of “dead” code on object creation (source CAST paper 4) Methods (for development / validation) –Automated = suspicious –Manual = safer –History is generally not an acceptable guide for reliability Life cycle process flow seems more important than tested reliability –Required now by FAA regulation –“DO-178B is much more than just a certification, it is a process that starts the same day as the development project itself. It is nearly impossible to take an existing application that was not developed in cooperation with the FAA or European equivalent and get it certified for any DO-178B levels, and definitely impossible for Level A applications.” Negative affects of these standards on cost/schedule/risk are not treated as relevant
The Concept of “Orphans” DO-178B and DO-254 require –Formal requirements at project start –Traceability during the development process –Test verification goes beyond bugs Must trace back to original requirements A Project can be “Orphaned” –If it does not have the proper beginning –It is essentially treated as “born bad” Excludes many great developments from certification –Nearly all open-source! –VTK, ITK, FLTK, QT, … –Java … –Linux and most RTOS kernels
What’s the Payoff from Extensive Up-Front Planning? Pervasive, long-term experiment in up-front planning and analysis for decision making: PPBS –Government policy since 1965 –No conclusive evidence of its success in reducing cost/schedule –Avg. schedule growth by USAF from 5 to 8 years –Considerable evidence that it cost agility “PPBS has failed everywhere and at all times” Aaron Wildavsky - The Politics of the Budgetary Process, 1974 no substantial improvement has occurred in average cost growth over the last 30 years, despite the implementation of several initiatives intended to mitigate cost growth Drezner et Al., Rand Report, 1993
Balance Sheet Costs: –Barrier to entry for new designs / small firms –Delay –Cost –Orphans –Difficulty incorporating COTS –Obsolescence Benefits –Some work directly related to reliability –Dubious benefits from formal life-cycle requirements process
Bottom Line – Everyone has an Opinion … Author’s opinion, not an organizational position Opinion –We should use some form of guidance to: Minimize risk Minimize cost Maximize payoff –We should count on decision makers to make accountable decisions Program managers Leaders Investors Chief Engineers –Standards organizations not accountable - should not be final authorities
Steve’s Suggestion Our business is not “one size fits all” –Life critical spacecraft –Nationally critical spacecraft –Expensive craft –Less expensive craft We need to tailor reliability requirements and methods Decisions should be left to leaders/managers We should publish guidelines to make it easier Reliability is not directly related to life-cycle requirements tracking We should not use DO-254 as our guide