1. Introduction to Software Engineering
“Software is hard.”
---Donald Knuth

2. Context and motivation for course topics
Motivation for topics covered
I want to start with a short story that will put the topics to be discussed into context. In short, demand for quality software is strong and growing. Average industry performance is not pretty. Especially worrisome are spectacular failures—those that have occurred in the past and prospects for even greater ones in the future. With this as a background we explore some of the root causes of software failure. It doesn’t have to be this way though. There are organizations that routinely deliver reliable software according to predictable schedules and budgets. The question is how?
[Provide context and rational for the material that follows in subsequent chapters]
This opening chapter provides context and motivation for learning about better methods of software development. In short, it follows the story outlined in figure 1. The demand for reliable software is strong and growing at a rapid pace. At the same time, average industry performance isn't keeping up with expectations. Especially worrisome are spectacular failures-- those that have occurred in the past and the prospects for even more alarming ones in the future. Projects fail for many reasons but there are patterns of failure that suggest where attention should be focused. Despite the high average failure rates for the industry as a whole, there are organizations that routinely deliver reliable software according to predictable schedules and budgets. The question is how?

3. Software is pervasive throughout our economy and culture
Modern civilization runs on software.
Nearly all of the products, services and innovations that power the industrialized world depend on software.
If you aren’t amazed by the extent to which our economy and culture depend on software, I’ll forgive you because software is largely invisible. You can’t see it. You can only experience it indirectly. Let’s make it tangible.
The extent to which software is present in everyday products and services often goes unnoticed because software is invisible.
We are not only dependent on software, but also our ability to develop and maintain it efficiently and effectively.
Our civilization runs on software. We are deeply dependent on software.
Software plays a central role in our daily lives.
The magnitude and extent to which we depend on software often goes unrecognized
[Software is Critical and ubiquitous nature of software is important]
Software is pervasive throughout our economy and culture
All digital devices are dependent on software.
Software plays a bigger role in our everyday lives than most people realize. The central role of software in our daily lives often goes unrecognized.
deeply dependent we have become on software.
Nearly all products and services in one way or another depend on software
Software makes commerce more efficient
Computers are indispensable.
Software is important to our economy and culture. Responsible for major progress/advances in entertainment, health, well-being, etc.
The economics of software is truly amazing and unprecedented. There is zero cost to copying and distributing a program. Tools are accessible. You can’t afford to create your own car assembly line but the tools needed to create the next Google, YouTube or Facebook are available to anyone for little or no cash investment.
– Packaged software is $90Billion industry. World wide it’s about 200 billion.

4. For Example
Did you know there is a ton of software in the average luxury car?
No, I mean that literally. The average luxury car contains almost 100 million lines of code.
Printed and bound, 100 million lines of code weighs over 3,000 lbs or 1.5 tons!
This car runs on code
The intangibility of software is one of it’s great benefits.
Take for example the auto industry and their products. The average luxury car today contains close to 100 million lines of code [This car runs on code]. One hundred million of anything is hard to imagine, but software is especially hard to imagine because it's intangible. You can't see or touch software directly. You can only experience its effects indirectly through a machine's behavior.
Thankfully software doesn't have to be kept in a tangle form when in use. Printed out, the software contained in a premium automobile is enough to fully cover a two-lane highway for 13.5 miles. Just image what it would be like walking this highway looking for a defect somewhere in the code. Because it's invisible, most people don't realize just how much their everyday experiences depend on the effective operation of software.

5. Printed out, the software contained in a premium automobile is enough to fully cover a two-lane highway for 13.5 miles
If that doesn’t impress you, consider that…
Just image what it would be like walking this highway looking for a defect somewhere in the code.

6. The Demand for Software is Strong and Growing
Moore’s Law
Software replacing hardware
Product differentiation with software
Open Source
Growing interest in mobile devices (smart phones, tablet computers, etc.) (Post-PC era)
Autonomous Vehicles
The US Occupational Outlook Handbook has some interesting info and data on computer-related job prospects. That also list SE as a discipline.
“Wage-and-salary employment is expected to grow 38 percent by the year 2016, compared with only 11 percent growth projected for the entire economy. In addition, this industry will add more than 489,000 jobs over the decade, placing it among the 10 industries with the largest job growth. An increasing reliance on information technology, combined with the falling prices of computers and related hardware, will spur demand for computer systems design and related services. “
[Consider adding data on job prospects]

7. Average Industry Performance Isn’t Flattering
Ref: The Future of Software Projects: Innovations & Emerging Trends in Software Quality Assurance [
In the latest survey from the Standish Group:
37% of projects were successful where success is defined as completing the project on time and on budget and delivering the features and functions that were initially specified.
21% of projects were canceled before completion.
42% of projects were challenged. The Standish group defines a challenged project as one that finishes with something usable, but is simultaneously finishes late, is over budget and offers fewer features than originally specified.
The January/February 2010 issue of IEEE Software features an article entitled The Rise and Fall of the Chaos Report Figures.
TBD: Perhaps a more scientific study is the one from Scott Ambler:
Results from the 2011 Standish Survey

8. Project performance by methodology
More interesting is performance as a function of development methodology. [
Again, because Standish doesn’t publish underlying data or survey methodology, it is hard to know how to interpret the results.

9. Industry Progress
2011 report breaks out agile projects:
Average industry performance over the years (Standish Survey)
As alarming as these numbers are, they represent an improvement from earlier years.

10. Larger projects are much more prone to failure than smaller ones

11. Defects by application size
Ref: Economics of Software Quality, Jones
Here is more data showing the diseconomy of scale associated with software projects.
The table shows the number of defects per function point (about 50 LOC) for projects of different sizes. Looking at the total row at the bottom of the table you can see that larger applications have more defects. Defects lead to rework so this is more evidence that large projects take more time and money per LOC or function point. What is particularly interesting about the data in the table is how requirements, architecture and design defects increase at a greater rate than coding or implementation defects. This data shows why spending more time on requirements and design is justified as projects grow larger.

12. Spectacular Failures are Particularly Worrisome
Ariane-5 -
Denver International Airport baggage handling system -
Patriot Missile -
Therac-25 –
Add to this list the terrible tragedy of _____ in 20__.
More worrisome is pivotal dramatic software failure coming in the future that is likely to lead to licensing and regulation of software engineers. Name one occupation where the work of the practitioners directly effects the health and safety of the public where there is no licensing and/or regulation.
TBD – More examples of dramatic failures: A Direct Path to Dependable Software, ACM
“the consequences of a
software failure in all its aspects are
becoming increasingly serious. Particularly
alarming is the seemingly unavoidable
fallibility of large software, since a
malfunction in an advanced hardware-software
system can be a matter of life and death,
not only for individuals, but also for
vehicles carrying hundreds of people and
ultimately for nations as well.”

13. Why do projects fail?
Estimation error. Projects that are estimated poorly are doomed from the start.
Performance failure
Poor planning (bad methodology choice)
Incorrect and optimistic status reporting
Unrealistic schedule pressure
New and changing requirements during development
Inadequate quality control
Poor communication
Optimistic status could be a symptom of using wrong/poor methodology.
According to Standish survey methodology matters as well.
Good summary of why projects fail: whyprojectsfail.pdf

14. Why do projects fail? [Cont]
Perceived failure is the sum of estimation failure and performance failure.
Software projects fail for many reasons. There are many more ways for things to go wrong than there are for things to go right.

15. The difficulties in software development are rooted in the fundamental nature of software
Intangibility – software is not far removed from pure thought stuff [Brooks].
Complexity
Software is dynamic
Software automates complex processes (lower bound for complexity)
Software has a greater number of discrete states than any other human artifact and transitioning between these states is not a smooth and continuous process [tbd: find Fred Brooks ref]
Software complexity and project complexity grows more than linearly with size of product and size of team (see next slide)
Changeability – software is soft which leads to change requests
Scalability – projects don’t scale up easily.
(Should I introduce here the idea that software had diseconomies of scale? There is a nice discussion of this in Making Software.)
It’s more difficult to reason about intangible products than it is with physical products. For example, an oil leak in a drilling rig is easy to find (just follow the flow of oil back to its source). A memory leak in a program is much more difficult to find. The physical expression of the program offers few hints to the source of the leak.
Intangible nature of software also leads to bad decisions. Management may push for more changes when the system is hopelessly disorganized. Because software is intangible, managers might not get an accurate assessment of stability.
Software is tangible in the sense you can see the lines of code (tangible expression of program). But lines of code don’t represent the dynamic runtime structure of the program. It’s the static and dynamic nature of the program that you need to understand.
That software is complex is an easy/obvious observation. Why is software complex?
“dynamic aspects of software applications: When software is executing, it has more moving parts and they move more rapidly than any other known product. ”
“Law” of conservation of complexity. Can organize and manage it better to limit how much you are exposed to at once, but can’t completely eliminate complexity.
Engineering and management challenges stem from the intrinsic nature of software.
As a consequent of software’s intangibility, customers don’t always know what they are buying. A customer might contract for a software system and think he is getting a product well-suited to the task only later to find it impossible to modify or extend.
Software is intangible. Any interaction with software will be indirect. Yes, you can browse a tangible listing and see the results on the screen as it executes but these are all indirect ways of interacting with it. They are all abstractions from a certain perspective.
Intangibility works both ways. Because the work is intangible it’s hard for senior management to understand the level of effort to do something. Developers can rush changes and leave the system in a state of disorder. Management tolerates it because it’s intangible and hard to detect.

16. Software complexity and project complexity grows more than linearly with size of product and size of team
There is diseconomies of scale in software development. (Distribution has incredible economies of scale.)
The M’s above could represent units of product (lines of code or modules) or people being added to a project. Both need to be organized in such a way that as the number of M’s increase the interactions with existing units grows at a rate closer to n^1 than n^2.
Hello Professor Burris, …
I also want to say I found a real life dis-economy of scale in software where the company I work for is trying to ramp up after a funding round by adding a bunch more engineers and the (non technical) managers are getting frustrated because the productivity is not going up anywhere near linearly.
Thanks,
Each person paints a different side of the building (north, south, east, west).
You sand, you hold can, you hold brush, you clean brushes.
You two prepare the surface, you two paint, you clean the brushes.

17. Software is not constrained by physical laws or the properties of materials. The software equivalent of the following is just the starting point for many applications.
This photo does a great job of illustrating a very subtle concept in s/w dev.
The following image went viral on the Internet in It's fascinating because it pushes the limits of what's possible in the real world. Many considered it a little too improbable and just assumed it was photoshopped--a product of the virtual world. It is in fact an actual structure created by a Dutch theater group.
As improbably as this design seems, the software equivalent is just the starting point for many software systems. Software systems are abstract and intangible. They are not constrained by physical laws or the properties of materials. Anything is possible. Consequently, absent self-imposed constraints, software systems tend towards extreme complexity. (dangerous and fragile states) In the physical world there are laws of nature and limits on available resources. Not many job sites have an endless supply of lumber and cement. In software development there is an endless supply of lines of code within easy reach. When there are no natural limits on what can be done, complexity spirals out of control.
Because software is intangible, even when it gets to this point, those that don’t recognize the true state of the software will encourage more building.
[Trying to understand why software development is difficult and how and why it gets into a desperate state.]
Software systems “are not constrained by the properties of materials, governed by physical laws or by manufacturing processes.”

18. No Silver Bullet
In 1987 Fred Brooks wrote an influential paper entitled “No Silver Bullet—Essence and Accidents of Software Engineering”.
In this paper he argues, “there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity”
Transitioning to finding a solution.
Don’t propose a simple answer to the problem, otherwise you look silly.

19. Silver Bullet Syndrome
Silver Bullet Syndrome = expecting exceptional productivity gains from one tool or technique.
Switch from batch programming to online terminals
3Rd generation languages
Case Tools AI
OOP
Ruby on Rails
Process improvement initiatives and methodologies (CMMI, Agile Methods, TQM, etc.)
Software Craftsmanship
Frameworks!
First step to better software development performance is learning to recognize and resist these seductive but ineffective practices.
There are no silver bullets, but there is a proven arsenal of weapons that can be deployed.

20. Essence and Accidents
To make his point, Fred Brooks, makes the distinction between two types of difficulties associated with software development:
Essential Difficulties
Accidental Difficulties
Most of what software engineers now do is devoted to the essential.
Essential complexities aren’t easily resolved. For example, you can address accidental complexity of memory leaks by switching to a language that supports garbage collection. There is no instant resolution to the problem of eliciting requirements or creating a flexible design.
Subtitle of paper

21. Accidental Complexities
The accidental complexities of software development are consequential to the tangible form of the solution, what you might call self-inflicted problems. If you choose Java as your implementation language, you will have to deal with the complications—large and small—that go along with programming in Java (e.g. how to convert a string to an int, how to read data from a file, etc.).
Examples of accidental complexities:
Expressing conceptual constructs with a programming language
Learning to use IDE’s such as Visual Studio and Eclipse
Learning to use configuration management tools such as SVN Git and ant gradle.
Testing the fidelity of the representation
Special tricks need to get C preprocessor macros to work correctly. For example, needing to wrap do { … }while(0) around statements in a macro definition in order to preserve the semantics when a semicolon is added after the macro use.
Testing the written code is an accidental complexity. Testing is a product of how we develop software. Some day there may be a method of development that doesn’t require testing. (Some product manufactures have improved their manufacturing process to such a degree they no longer need to test the product at the end of the assembly line.)
With advances in programming tools and techniques, many accidental complexities have disappeared over the years. Today if you don’t want to hassle with memory management issues, there are several languages with automatic garbage collection at your disposal (pun intended!). Thanks to high level languages, the intellectual distance between the problem concept and solution expression is much shorter than years past.
Can instantly get rid of accidental complexities (but might replace one accidental complexity with another at the same time).
“You could also think of accidental properties as incidental, discretionary, optional, and happenstance.”
Accidental complexities are problems we create for ourselves, the consequences of how we choose to solve the problem.

22. Essential Complexities
Essential complexities are inherent in the problem. They can’t be avoided.
Examples:
Defining precisely what to build
Coming up with the conceptual construct of the solution
Validating the problem definition and solution construct

23. History of Software Engineering
The conference title, ‘Software Engineering’, was deliberately chosen to be provocative, suggesting the opportunity for the discipline of software development to become a new branch of engineering with sound theoretical foundations on pare with other established branches of engineering.
The term software engineering came into popular use after a 1968 NATO conference on the subject*. Technological advances in integrated circuits during the decade leading up to the conference made it possible to use computers for ever more complicated tasks. This rapid growth of computing power created a demand for larger more sophisticated software systems. Some of these systems were beyond the current capabilities in software design and production. The conference was organized in part to bring attention to the growing problem of delivering large software systems according to predictable schedules and budgets.
* Contrary to some reports, the conference might have popularized then term ‘software engineering’ but it didn’t coin the term. There is evidence to suggest that Douglas Ross was teaching seminars on Software Engineering at MIT prior to the 1968 conference [Douglas Ross Talks About Structured Analysis, IEEE Computer, July 1985].
We no longer use manual type writers and bulky real-to-real analog recording devices (see picture above), but many of the comments on software made at the conference are still relevant today. “”
TBD:
Images are from: [Brian Randell, ]
1968 NATO Conference on Software Engineering

24. Software Engineering
Software engineering is “the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, that is, the application of engineering to software” [IEEE]
is “a systematic approach to the analysis, design, assessment, implementation, test, maintenance and reengineering of software, that is, the application of engineering to software.” [Laplante 2001, Comprehensive Dictionary of Computer Science, ..]
“"the disciplined application of engineering, scientific, and mathematical principles and methods to the economical production of quality software.”
Software engineering is not just programming. Programming accounts for only about 15% of the total effort spent on the typical software project [Boehm, 1987]. The rest of the time is spent establishing requirements, designing a solution, verifying results, planning, coordinating, communicating and in general managing the development process and associated products. Software engineering is an area of study concerned with this broader view of software development.
Software engineering follows a tradition of engineering in other disciplines. There are established branches of engineering for constructing bridges (civil engineering), engines (mechanical engineering), and various electrical devices (electrical engineering). Engineering is the application of scientific knowledge and practical experience to the production of useful objects. Software engineering aspires to apply the same systematic, disciplined and quantitative approach to the production of software.

25. What is Engineering?
Engineers apply science and technology to develop cost-effective solutions to practical problems.
The Accreditation Board for Engineering and Technology (ABET) defines Engineering as:
“The profession in which a knowledge of the mathematical and natural sciences gained by study, experience, and practice is applied with judgment to develop ways to utilize, economically, the materials and forces of nature for the benefit of mankind”
Engineers “combine knowledge of science, mathematics, and economics to solve technical problems that confront society.”
Engineering is the application of science to the needs of humanity. Scientists create knowledge and engineers put it to work solving practical problems.
Grounded in science and fundamental principles
Solve practical problems using the principles of science
Thomas Tredgold’s classic definition of engineering: “Engineering is the art of directing the great sources of power in nature of the use and convenience of man.” [1828] or “for the benefit of the human race” or “for the general benefit of mankind” or “for the good of humanity” “applying … to the needs of mankind”
Engineering has made life “not only longer, but richer and better worth living”
There were “miraculous advances” between 1850 and 1950 due to application of technology.
Many of the comforts and pleasures in life are made possible by engineering achievements. Thinks we take for granted: clean water, electricity, air conditioning, refrigeration. Transportation: jet airplanes, cars, ocean liners. Entertainment: home theaters, portable audio players. Communication: satellites, cell phones, the Internet. Construction: bridges, roads, dams, skyscrapers. Green energy sources: solar cells, wind turbines, biofuels. Space: Hubble Space telescope, Space shuttle, rovers exploring distant planets.
Engineering miracles would free man from drudger and burdensome labor
Engineers believe in the service to humanity.
For nearly a century starting in 1850, engineers were the rock stars of their day. The public greatly admired the each new engineering feet: railways joining cities, opening of bridges, etc. technology was idolized. Engineers were “generally admired and occasionally lionized.
Technological progress was met with great enthusiasm. It made life easier and more comfortable for most people. The accomplishments seemed miraculous.
engineering is “the art or science of making practical application of the knowledge of pure sciences.”
Engineering disciplines have a core body of knowledge or underlying science that can be used to solve practical problems. For example, chemical engineering has chemistry and electrical engineering has math and physics.

26. Simple Quiz for better understanding accidental and essential complexities.
For each of the following 4 quotes, indicate whether it was a statement made during the 1968 conference on software engineering or the 2011 conference on software engineering.
43 years later
Here is a simple quiz I think will help you understand how the nature of software engineering has changed over the years. Next to each of the following quotes, indicate whether it was a statement made at the 1968 conference on Software Engineering in Garmisch, Germany or the 2011 conference on Software Engineering in Honoulu, Hawaii.

27. “The discussions were recorded by several reporters and most were also recorded on magnetic tape.”
“There is probably no single ‘correct’ order in which to take a series of design decisions, though some orderings can usually be agreed to be better than others. Almost invariably some early decisions, thought at the time to have been clearly correct, will turn out to have been premature.”
“Few large systems have been written in high-level languages. This is not surprising since there are inevitable penalties in size and performance of compiled code, and these factors have been paramount in people’s minds.”
“Users are interested in systems requirements and buy systems in that way. But that implies that they are able to say what they want. Most of the users aren’t able to.”
B shows experience.
Scroll down for the answer…
All 4 quotes are from the 1968 conference on Software Engineering. A and C are accidental complexities. The problems seem quaint by today’s standards. B and D are essential complexities. The statements were made almost 50 years ago and are just as true today as they were in There is no deterministic process for designing software. Defining software requirements is difficult because it is hard for users to say what they want. These are problems that aren’t going to be made obsolete by any new invention anytime soon.
Some area quaint some are just as true today as they were 40+ years ago. B and D will probably be true 40 years from today.
Compilers can write more efficient machine code than most people.

28. Success with Software is Possible
There are organizations that are succeeding with their software development efforts. Being successful with software means:
Having the ability to give accurate estimates that get more precise over time
Delivering on schedule and within budget
Being able to control software quality
Satisfied customers
High morale among staff members
High productivity
No silver bullets but there is a proven arsenal of weapons that can be deployed.
Controlling quality = delivering software at a predictable level of quality.
If the project team can answer “yes” to the following question, there is high morale. “Would you be willing to work on a similar project under similar conditions?”
Without high productivity you can’t be competitive.

29. No Universal Solutions
No one solution is appropriate for all projects

30. Important Factors in Project Success
Management experience
Technical staff experience
Appropriate quality control measures
Stable requirements
Component reuse
Use of specialists
Use of appropriate methodologies
Use of appropriate estimating techniques and tools
Use of appropriate planning techniques and tools
Formal progress tracking
Experienced clients
Adequate tools
If all 12 success factors above are present, the chances of a successful outcome exceed 95% for all but the largest projects [Jones 1996, Software Systems: Failure and Success,  p 50]. If none are present the chances of success drops below 50%.

31. Software Engineering

32. Software
“Computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a computer system.”

33. Stop here

34. Characteristics of successful practice looks a lot like engineering
Disciplined
Systematic
Quantitative
Growing body of knowledge that can be applied in a systematic way
Applying well-understood knowledge to solve practical problems

35. Economic value of software to companies
Companies invest in software in order to:
Reduce cost, and/or
Increase revenue
Companies invest in software to reduce cost and/or increase revenue. Costs can be reduced by automating tasks currently performed manually. (Example: automated record keeping at insurance companies and banks.) Revenue can be increased by scaling the business, that is, making improvements in manufacturing and marketing that permit more products and services to be sold. (Example: Wal-Mart’s inventory management system helps to ensure stores never run out of stock or have an excess of product.) Another opportunity for increasing revenue is through innovation—new products and services are made possible by advances in software and information technology. Examples: computer gaming, search and online auctions are billion dollar industries that wouldn’t exist without software.
Reference: The Economics of Software Quality, Chapter 1.
Ref: Application Software for Insurance in the 1960s and 1970s. JoAnne Yates.
(and new business models)

36. Software Categories
It is important to be aware of the different types of software. A technique or methodology that works well for the development of one type of software could be a disaster when used in the production of another type of software. The categories are:
Shrink-wrapped or packaged software
Custom or bespoke software
Consultingware
Embedded software
Throwaway or one-off programs
Ref:
Shrink-wrapped or packaged software. Shrink-wrapped software is software that has a potentially large, usually diverse, user base. Obvious examples include Microsoft Word and Adobe Photoshop. Not-so-obvious examples include Gimp and the Twitter client for iPhone. Usability is a priority because customers usually have other options. Quality is important for the same reason. There isn’t one customer you can turn to for requirements. An important development decision is deciding what features to include in the application. Software as a service (SaaS) is also included in this category.
Custom or bespoke software. Custom software is software built to solve a very specific problem, usually one for which there is no shrink-wrapped solution. Custom software may be developed using in house staff or commissioned from outside developers. Requirements are more certain with custom development. There is usually one person you can go to for detailed requirements. Usability and other non-functional quality attributes may not be as important as they are with shrink-wrapped software because there are no ready alternatives to the software being developed.
Consultingware. Consultingware has elements of shrink-wrapped and custom software. The vendor of consultingware supplies the customer with software that does about 80% of what is needed and high priced consultants to adapt the software for the other 20% of what is needed. Consultants may write new code and/or modify configuration files. They need to be experts in the application domain as well as the technical details of the base system.
Embedded software. Embedded software controls everyday devices and machines most people don’t think of as computers. For example: cars, audio equipment and appliances. Quality is much more important with embedded software because often there is no mechanism for updates. Embedded software runs on dedicated hardware often with limited resources (memory, processing power and libraries). This puts a premium on efficiency over other non functional attributes such as readability and maintainability.
Throwaway or one-off programs. A throwaway program is one you write quickly for a one-time task. For example: data conversion, test data creation, some admin task such as creating tables in a database. A throwaway program doesn’t need to have a good design or be readable. It just has to function properly and not make any subtle mistakes.