What does new technology mean in terms of development? Are you encouraged to apply new technology in a project? If a powerful tool is able to speed up the development, will you prefer it?

Anatomy of 4GL Disaster CS470 - Software Engineering I

Generation Languages  1GL: low-level languages that are machine language.  Do things in machine’s way (her way or high way)  2GL: also low-level languages that generally consist of assembly languages.  Do things in some way (M-H common ground)  3GL: high-level languages such as C/C++.  Do things in procedural way (What-to-do / How to-do)  4GL: languages that consist of statements similar to statements in a human language, commonly used in database programming and scripts.  Do things in query way (just tell “what-to-do”)  5GL: languages that contain visual tools to help develop a program. A good example of a 5GL is Visual Basic.  Do things in graphical way (GUI code generation / Event-driven programming)  1GL: low-level languages that are machine language.  Do things in machine’s way (her way or high way)  2GL: also low-level languages that generally consist of assembly languages.  Do things in some way (M-H common ground)  3GL: high-level languages such as C/C++.  Do things in procedural way (What-to-do / How to-do)  4GL: languages that consist of statements similar to statements in a human language, commonly used in database programming and scripts.  Do things in query way (just tell “what-to-do”)  5GL: languages that contain visual tools to help develop a program. A good example of a 5GL is Visual Basic.  Do things in graphical way (GUI code generation / Event-driven programming)

Introduction  In 1985, an upgrade to the New Jersey DMV database system was activated.  It was written using “Ideal”, a new 4GL.  Hundreds of thousands of errors resulted, along with that many dollars spent in fixing them.  The only viable solution was to rewrite some of the core components using Cobol.  In 1985, an upgrade to the New Jersey DMV database system was activated.  It was written using “Ideal”, a new 4GL.  Hundreds of thousands of errors resulted, along with that many dollars spent in fixing them.  The only viable solution was to rewrite some of the core components using Cobol.

The Real Question  Anatomy of a 4GL disaster?  Anatomy of a human disaster?  Anatomy of a 4GL disaster?  Anatomy of a human disaster?

What Happened?

The DMV Needs an Upgrade  In 1983, the New Jersey DMV knew their DB systems needed upgrading.  Two options for implementation:  Price Waterhouse (consulting firm)  DMV’s Division of Systems and Communications (SAC)  In 1983, the New Jersey DMV knew their DB systems needed upgrading.  Two options for implementation:  Price Waterhouse (consulting firm)  DMV’s Division of Systems and Communications (SAC)

Implementation Option: Price Waterhouse  Completely new system.  $6.5 million, plus $8.5 million for hardware.  DMV Director and deputy (Snedeker and Kline) support Price Waterhouse.  “Price Waterhouse’s greatest strength is in project management and control.”  Price Waterhouse would be involved in the design anyway, so why not let them implement it?  Completely new system.  $6.5 million, plus $8.5 million for hardware.  DMV Director and deputy (Snedeker and Kline) support Price Waterhouse.  “Price Waterhouse’s greatest strength is in project management and control.”  Price Waterhouse would be involved in the design anyway, so why not let them implement it?

Implementation Option : Price Waterhouse  Can get things done faster than SAC.  SAC thought not to have enough qualified personnel.  “[SAC’s] greatest weakness is project management and control.”  Can get things done faster than SAC.  SAC thought not to have enough qualified personnel.  “[SAC’s] greatest weakness is project management and control.”

Implementation Option: DMV’s SAC  Update current systems to work more effectively, and add new modules.  $2 million, plus $3 million for hardware.  Price Waterhouse could help design.  SAC employees are already familiar with the systems.  Price Waterhouse does not have implementation experience.  Update current systems to work more effectively, and add new modules.  $2 million, plus $3 million for hardware.  Price Waterhouse could help design.  SAC employees are already familiar with the systems.  Price Waterhouse does not have implementation experience.

An External Opinion  The governor’s office asked for advice from the Science Management Corp. (SMC), another consulting firm with knowledge of the DMV’s systems.  They recommended against Price Waterhouse taking charge of the whole project.  Price Waterhouse would only be involved in the design phase.  (The federal government actually requires that the design firm not be involved in implementation.)  The governor’s office asked for advice from the Science Management Corp. (SMC), another consulting firm with knowledge of the DMV’s systems.  They recommended against Price Waterhouse taking charge of the whole project.  Price Waterhouse would only be involved in the design phase.  (The federal government actually requires that the design firm not be involved in implementation.)

The Verdict  Price Waterhouse chosen.  Why?  Nobody is exactly sure.  They wanted it done “quickly”.  Price Waterhouse made a few generous political contributions.  Project manager: A lawyer.  Price Waterhouse chosen.  Why?  Nobody is exactly sure.  They wanted it done “quickly”.  Price Waterhouse made a few generous political contributions.  Project manager: A lawyer.

Implementation: Plan  “Ideal” to be the primary programming language.  4GL (Ideal)  Introduced eight months earlier by ADR (Applied Data Research).  Price Waterhouse’s contract stated that they had evaluated Ideal to determine that it was suited for the job.  “Ideal” to be the primary programming language.  4GL (Ideal)  Introduced eight months earlier by ADR (Applied Data Research).  Price Waterhouse’s contract stated that they had evaluated Ideal to determine that it was suited for the job.

Conceptual Problems: Speed  Ideal is a 4GL.  Simplifies coding at the expense of processing speed.  More than three times slower than Cobol.  Fine under the right circumstances, but not good for heavy multi-user data processing.  Ideal lacks DB indexing.  Ideal is a 4GL.  Simplifies coding at the expense of processing speed.  More than three times slower than Cobol.  Fine under the right circumstances, but not good for heavy multi-user data processing.  Ideal lacks DB indexing.

Conceptual Problems: Interfacing  Ideal does not allow computer-to- computer interfacing.  The DMV project required communication between over 50 computers.  “Price Waterhouse never seemed to grasp the issue. It didn’t seem to realize the system would not run on a dedicated DMV machine.”  Ideal does not allow computer-to- computer interfacing.  The DMV project required communication between over 50 computers.  “Price Waterhouse never seemed to grasp the issue. It didn’t seem to realize the system would not run on a dedicated DMV machine.”

Implementation: Problems  16 months from deadline - Price Waterhouse: “The uncertainties associated with the use of Ideal represent an acceptable risk.”  15 months - “The support necessary from ADR is slow and difficult to obtain.”  14 months - ADR gets copy of DB design, says nothing about potential problems.  16 months from deadline - Price Waterhouse: “The uncertainties associated with the use of Ideal represent an acceptable risk.”  15 months - “The support necessary from ADR is slow and difficult to obtain.”  14 months - ADR gets copy of DB design, says nothing about potential problems.

Implementation: Problems  12 months - “The satisfactory resolution of ADR-related technical problems would remove a major issue threatening timely completion of the project.”  9 months - ADR informs Price Waterhouse and SAC of two potential problems:  Slow response times  Limitation on number of online users  8 months - System tests show transactions taking more than twice as long as acceptable.  12 months - “The satisfactory resolution of ADR-related technical problems would remove a major issue threatening timely completion of the project.”  9 months - ADR informs Price Waterhouse and SAC of two potential problems:  Slow response times  Limitation on number of online users  8 months - System tests show transactions taking more than twice as long as acceptable.

Implementation: Problems  7 months  Price Waterhouse outlines plan to limit number of online terminals to 200, rather than 1,000.  A few modules will be delayed, and some of the code will be written in Cobol.  7 months  Price Waterhouse outlines plan to limit number of online terminals to 200, rather than 1,000.  A few modules will be delayed, and some of the code will be written in Cobol.

ADR and IDEAL 4GL  Applied Data Research (ADR). ADR is often credited as being the first independent software company, having introduced products in the late 1960s and prevailed in antitrust struggles against IBM to allow the business to survive. Basically, it sold programmer productivity tools. This led it to acquire Datacom/DB, an inverted-list DBMS developed in the Dallas area. In the early 1980s, Datacom/DB began to boom, and was on a track to surpass both IDMS and ADABAS in market share until DB2 showed up and blew them all away. ADR was particularly aided by its fourth- generation language (4GL) IDEAL, which was an excellent product notwithstanding the famous State of New Jersey fiasco.  (As John Landry said to me about that one, “4GLs are powerful tools. In particular, they allow you to write bad programs really quickly.”)  Applied Data Research (ADR). ADR is often credited as being the first independent software company, having introduced products in the late 1960s and prevailed in antitrust struggles against IBM to allow the business to survive. Basically, it sold programmer productivity tools. This led it to acquire Datacom/DB, an inverted-list DBMS developed in the Dallas area. In the early 1980s, Datacom/DB began to boom, and was on a track to surpass both IDMS and ADABAS in market share until DB2 showed up and blew them all away. ADR was particularly aided by its fourth- generation language (4GL) IDEAL, which was an excellent product notwithstanding the famous State of New Jersey fiasco.  (As John Landry said to me about that one, “4GLs are powerful tools. In particular, they allow you to write bad programs really quickly.”)

Launch!  All old systems turned off. (PW did not plan for enough processor power to run in parallel.)  New systems turned on.  Log on could take one hour.  Response times could take three minutes.  Nightly batch updates could take days.  All old systems turned off. (PW did not plan for enough processor power to run in parallel.)  New systems turned on.  Log on could take one hour.  Response times could take three minutes.  Nightly batch updates could take days.

Epilogue  After a lawsuit, Price Waterhouse agreed to convert as much of the system as necessary to Cobol at their expense.  The update was still not complete as of six months later.  After a lawsuit, Price Waterhouse agreed to convert as much of the system as necessary to Cobol at their expense.  The update was still not complete as of six months later.

Summary  What went wrong?  Wrong tool for the job (4GL for heavy data processing)  Overly-optimistic management  Ignoring warning signs  Lack of quality assurance reviews or early feasibility testing.  What went wrong?  Wrong tool for the job (4GL for heavy data processing)  Overly-optimistic management  Ignoring warning signs  Lack of quality assurance reviews or early feasibility testing.

Summary  “It’s a story of an organization and individuals abandoning established management and data processing practices and, it appears, common sense, in a rush to collect on the promise of the new generation of programming languages.”