Leiden Institute of Advanced Computer Science Software Engineering Leiden Institute of Advanced Computer Science Who made software before? Lecture Series for BSc. “Computer Science” year 2 (Fall semester 2013) 1

Outline Introduction / Course logistics Introductory lecture Software Engineering What is Software Engineering? What does a Software Engineer do? What does a Software Engineering Process look like? These slides are based on slides by Drs. Werner Heijstek, Dr. Natallia Kokash, Dr. Michel Chaudron and Professor Dr. Hans van Vliet 2

Software Engineering “Course Logistics” Leiden Institute of Advanced Computer Science Who made software before? Lecture Series for BSc. “Computer Science” year 2 (Fall semester 2013) 3

Who am I? Tim Cocx http://www.timcocx.nl Room 124 (Thursdays), 071-5275777, tcocx@liacs.nl Teach this course for the second time Also: statistics Main job: Haagse Hogeschool I hold M.Sc. (2004) & Ph.D. (2009) degrees from Leiden University My current main focus points are programming and software design Married with two cats

What you will learn in this course? Engineering = skill + knowledge This course 30% knowledge and 70% skills Basic concepts, vocabulary of Software Engineering Main activities in Software Engineering projects Main methods and techniques excluding: programming (Maybe) Guest Lectures by professionals Software Engineering as an academic research area 5

The Only Really Important Info http://www.liacs.nl/~tcocx/SE2013

Lectures Schedule There is a iCal file on the website for phone / tablet usage Week 1 Morning: Course & Subject Introduction Afternoon: Nothing / Free / Self-Study Week 2 All day: Reverse Engineering Week 3 All day: Modelling Week 4-14: Morning: Class Afternoon: Lab Week 15 Monday & Tursday: All Day Assessments 20-12-2013 10.00 Exam NB: 3rd October: Only party NB: 24th October: No class, exam in afternoon 7

Grading Week 2: Reverse Engineering Assignment (20%) Deadline Week 4 Week 8: Modelling Exam (20%) Week 15: Assessments (40%) Week 16: Theoretical Exam (20%) All grades must be sufficient (>=5.5)

Lab You will do a Software Engineering Project In teams of 6-10 people Focus on a proper development process Results: Documentation, process “Log’s”, Software Programming skills are required. Assignment: You’ll see when the lab starts (surprise!)

Software Engineering “An Introduction” Leiden Institute of Advanced Computer Science Who made software before? Lecture Series for BSc. “Computer Science” year 2 (Fall semester 2013) 10

The Software Crisis “The major cause of the software crisis is that the machines have become several orders of magnitude more powerful! To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem.” Edsger Dijkstra The Humble Programmer, Communications of the ACM (1972) 11

Division Computer costs 100 Hardware Development 60 Percent of total cost Software 20 Maintenance 1955 1970 1985 Year © SE, Introduction, Hans van Vliet 12

The Crisis Manifest Projects running over-budget Projects running over-time Software was very inefficient Software was of low quality Software often did not meet requirements Projects were unmanageable and code difficult to maintain Software was never delivered

The Birth of Software Engineering The term Software Engineering was introduced at 1968/69 NATO conferences Idea: software development is not an art, or a bag of tricks We should build software like we build bridges

What is Software Engineering? The process of solving customers’ needs by the systematic development and evolution of large, high- quality software systems within cost, time and other constraints Solving customers’ problems This is the goal of software engineering Sometimes the solution is to buy, not build Adding unnecessary features does not help solve the problem Software engineers must communicate effectively to identify and understand the problem 15

Definition (IEEE) “Software Engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software” © SE, Introduction, Hans van Vliet 16

Is Software Engineering, Engineering? Software is logical, rather than physical Progress is hard to see Software is not continuous Further reading: Henry Petroski, Design Paradigms: Case Histories of Error and Judgement in Engineering A. Spector & D. Gifford, A Computer Science Perspective of Bridge Design, Communication of the ACM 29, 4 (1986) p 267-283

Types of Software Custom For a specific customer Generic Sold on open market Often called COTS (Commercial Off The Shelf) Shrink-wrapped Embedded Built into hardware Hard to change 18

Types of Software Real time software E.g. control and monitoring systems Must react immediately Safety often a concern Business Information Systems (Data processing) Used to run businesses Accuracy and security of data are key Some software has both aspects! 19

The Nature of Software Software is intangible Hard to understand development effort Software is easy to reproduce Cost is in its development in other engineering products, manufacturing is the costly stage The industry is labor-intensive Hard to automate 20

The Nature of Software (Untrained) people can hack something together Quality problems are hard to notice Software is easy to modify People make changes without fully understanding it Software does not ‘wear out’ It deteriorates by having its design changed: erroneously, or in ways that were not anticipated, thus making it complex 21

The CHAOS Report 1994 1996 1998 2000 2002 2004 2006 2009 Successful 16% 27% 26% 28% 34% 29% 35% 32% Challenged 53% 33% 46% 49% 51% 44% Failed 31% 40% 23% 15% 18% 19% 24% 1994 1996 1998 2000 2002 2004 2006 2009 Successful 16% 27% 26% 28% 34% 29% 35% 32% Challenged 53% 33% 46% 49% 51% 44% Failed 31% 40% 23% 15% 18% 19% 24% 1994 1996 1998 2000 2002 2004 2006 2009 Successful 16% 27% 26% 28% 34% 29% 35% 32% Challenged 53% 33% 46% 49% 51% 44% Failed 31% 40% 23% 15% 18% 19% 24% http://www.projectsmart.co.uk/docs/chaos-report.pdf

Central Themes in Software Engineering Software Engineering is concerned with large programs These programs are complex Software evolves Development must be efficient You’re doing it together Software must effectively support users Involves different disciplines Software Engineering is a balancing act - Space Shuttle software: 40 million lines of code - complexity often is not “mathematical” complexity, but caused by the myriad of details to be handled - reality changes, so the software has to change as well (e.g. salary administration and changing tax regulations) - software costs a tremendous amount of money (it is very labor intensive) - OS 360 cost 5000 person years (so you need rules and regulations - hence process models) - so user involvement - software engineers tend to have little knowledge of the application domain © SE, Introduction, Hans van Vliet 23

What does a Software Engineer do? in a team interacting with clients individually Microsoft 1978 programming listening Specializing in different roles - designing, programming, testing … documenting explaining planning feedback presenting selling brainstorming discussing planning reviewing reporting 24

Simple “Life Cycle” Model Typical distribution of effort (excluding maintenance) problem requirements engineering 15% reqs specification design 20% design implementation 20% system testing 45% working system maintenance © SE, Introduction, Hans van Vliet 25

Requirements Engineering Includes Domain analysis Defining the problem Requirements gathering Obtaining input from as many (relevant) sources as possible Requirements analysis Organizing the information Requirements specification Writing detailed instructions about how the software should behave Results in a description of the DESIRED system: which functions possible extensions required documentation performance requirements Includes a feasibility study Resulting artifact: requirements specification - the requirements document often has a legal status too. - different stakeholders want different documents (the designer/impementer wants a precise, formal document, while the user is best served by a document in her natural language) - it is important the the requirements document is very precise; no TBD’s (To Be Determined). If you do not know what is being asked, chances are you will not come up with the right answer. 26

Design Includes Deciding how the requirements should be transferred to software, using the available technology Includes: Systems engineering: Deciding what should be in hardware and what in software Software architecture: Dividing the system into subsystems and deciding how the subsystems will interact Detailed design of the internals of a subsystem User interface design Design of databases Earliest design decisions captured in software architecture Decomposition into parts/components; what are the functions of, and interfaces between, those components? Emphasis on what rather than how Resulting artifact: specification 27

Implementation Focus on individual components Goal: a working, flexible, robust, … piece of software Not a bag of tricks Present-day languages have a module and/or class concept 28

Testing Does the software do what it is supposed to do? Are we building the right system? (validation) Are we building the system right? (verification) Start testing activities in phase 1, on day 1 Testing should not start after the implementation phase. Cleanroom development: developers are allowed to read the programs they’ve written, they are not allowed to execute them. Testing in Cleanroom is done by others. Question to students: do you dare to give a warranty on the software you wrote?

Maintenance Correcting errors found after the software has been delivered Adapting the software to changing requirements, changing environments, ...

Effort Distribution Rule of thumb: 40-20-40 distribution of effort Trend: enlarge requirements specification/design slots; reduce test slot Beware: maintenance alone consumes 50-75% of total effort

Actual Effort Distribution

Software Engineering in a Nutshell

Codex Hamurabi “If a builder build a house for someone, and does not construct it properly, and the house which he built falls in and kills its owner, then that builder shall be put to death.” Article 229 of the Code of Hammurabi (1780 BC) - first king of the Babylonian Empire - Conquered most of Mesopotamia - Codex Hammurabi is a well-preserved Babylonian law code. - It is one of the oldest deciphered writings of significant length in the world. - 6th Babylonian king, Hammurabi, enacted the code - The Code consists of 282 laws, with scaled punishments, adjusting "an eye for an eye, a tooth for a tooth" (lex talionis) Also: if a house falls down and it kills the son on the owner, then the son of the builder shall be put to death als een huisvrouw niet bevalt, gooi haar dan in de rivier http://www.wsu.edu/~dee/MESO/CODE.HTM 34

Software Engineering Ethics Act consistently with the public interest Act in a manner that is in the best interest of the client and employer Ensure that products meet the highest professional standards possible Maintain integrity in professional judgment Managers shall promote an ethical approach Advance the integrity and reputation of the profession Be fair to and supportive of colleagues Participate in lifelong learning and promote an ethical approach

Difficulties and Risks in Software Engineering Complexity and large numbers of details Uncertainty about technology Uncertainty about requirements Uncertainty about software engineering skills Constant change Deterioration of software design Political risks 36

Software Development Methods Software Projects are large and complex A phased approach to control it is necessary Traditional models are document-driven: There is a new pile of paper after each phase is completed Evolutionary models recognize that much of what is called maintenance is inevitable

Software Development Process Models Waterfall Iterative 38

Software Development Process Models Waterfall Model (Mid 70ies) Time Requ. Eng. & Architecting milestone 1 milestone 2 Specification milestone 3 Design milestone 4 Implementation milestone 5 Increments may also be internal Different technologies may have different lifecycle models Test No iterations Big bang scenario  First-time right 39

The waterfall model Feasibility study Analysis System Design User Requirements Analysis System Design Program Design Coding Testing Operation Decomission 40 40

Pro's and Cons of the Waterfall Model Imposes structure on complex projects Every stage needs to be checked and signed off: Elimination of midstream changes Good when quality requirements dominate cost and schedule requirements Cons: Limited scope for flexibility / iterations Full requirements specification at the beginning: User specifications No tangible product until the end 41 41

Problems of the Waterfall Process (2) Different phases are handled by different people Business Modeling Consultants Architect(s) IT-Specialists IT-Engineers Requirements & Architecting Specification & Design Implementation Testing Communication becomes highly critical 42

Software Development Process Models Waterfall Model (Mid 70ies) Test Specification Design Implementation Requ. Eng. & Architecting Evolutionary Models (80ies) Scope Time The size of the different phases is not an indication about their length Increments (Spiral cycles) Iteration 43

Rational Unified Process (RUP) Phases Disciplines Iterations 44

Problems of Evolutionary Models Inflexible point solutions The initial release is optimized for demonstration, consequently the architecture is difficult to extend High-risk downstream capabilities The initial release often defers quality attributes (dependability, scalability, etc.) in favor of early functionality Evolutionary development Incremental development E.g. Barry Boem’s spiral model Prototyping Construction of a durable prototype that can be extended in increments [check] 45

Incremental delivery Each component delivered must give some system increment 1 2 3 design build install evaluate first incremental delivery design build install evaluate second incremental delivery design build install evaluate This approach involves breaking the application down into small components which are then implemented and delivered in sequence; each component delivered must give some benefit to the user Time-boxing is often associated with an incremental approach; here the scope of the deliverable for an increment is rigidly constrained by an agreed deadline, even at the cost of dropping some functionality third incremental delivery Each component delivered must give some benefit to the stakeholders 46

The Plan 47

Reality 48

The Agile Manifesto “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.”

12 Agile Principles Satisfy the customer through early and continuous delivery Welcome changing requirements, even late in development Deliver working software frequently Business people and developers work together daily Build projects around motivated individuals Convey information via face- to-face conversation Working software is the primary measure of progress Maintain a constant pace indefinitely Give continuous attention to technical excellence Simplify: maximize the amount of work not done Teams self-organize Teams retrospect and tune their behaviors

Prototyping Requirements elicitation is difficult software is developed because the present situation is unsatisfactory however, the desirable new situation is as yet unknown Prototyping is used to obtain the requirements of some aspects of the system Prototyping should be a relatively cheap process use rapid prototyping languages and tools not all functionality needs to be implemented production quality is not required

Prototyping Approaches Throwaway prototyping the n-th prototype is followed by a waterfall-like process Evolutionary prototyping the n-th prototype is delivered

Pro's and Cons of Prototyping The resulting system is easier to use User needs are better accommodated The resulting system has fewer features Problems are detected earlier The design is of higher quality The resulting system is easier to maintain The development incurs less effort Cons The resulting system has more features The performance of the resulting system is worse The design is of less quality The resulting system is harder to maintain The prototyping approach requires more experienced team members

Popular Agile Methods Rapid Application Development (RAD) & Dynamic System Development Method (DSDM) Extreme Programming (XP) Feature Driven Development (FDD) Unified Processes: Rational Unified Process Agile Unified Process (AUP) Open Unified Process (OpenUP)/Basic Essential Unified Process (EssUP) Scrum

(R)UP (Rational) Unified Process. Philosophy: Four phases in Waterfall Inception: Determine the plan and starting REQ Elaboration: Elaborate on REQ and Design Construction: Building & Testing Transition: Delivery, Installation, Instruction Within the phases iteration takes place until a predetermined ‘Milestone’ has been reached Disciplines are part of the team for the project duration There are, however, peaks in usage Artifacts (documents): Change during the project. There are however peaks in work on them.

Rapid Application Development (RAD) Evolutionary development, with time boxes fixed time frames within which activities are done; Time frame is decided upon first, then one tries to realize as much as possible within that time frame; Other elements: Joint Requirements Planning (JRD) and Joint Application Design (JAD), workshops in which users participate; Requirements prioritization through a triage; Development in a SWAT team: Skilled Workers with Advanced Tools

Dynamic System Development Method Is a RAD framework Popular in the UK Fundamental idea: fix time and resources (timebox), adjust functionality accordingly One needs to be a member of the DSDM consortium

Extreme Programming (XP) Everything is done in small steps The system always compiles, always runs Client as the center of development team Developers have same responsibility w.r.t. software and methodology Pair Programming Manifesto: If … is good then [fill in extreme form]

Agile versus Traditional Development Lightweight (Agile) Heavyweight Developers Knowledgeable, collocated, collaborative. Plan-driven, adequate skills, access to external knowledge. Customers Dedicated, knowledgeable, collocated, collaborative, representative, empowered Access to knowledgeable, collaborative, representative, empowered customers Requirements Largely emergent, rapid change Knowable early, largely stable Architecture Designed for current requirements Designed for current and foreseeable requirements Size Smaller teams and products Larger teams and products Primary objective Rapid value High assurance

Producing Software Means Using Software Builders build pieces, integrators integrate them Component-Based Development (CBSD) Software Product Lines (SPL) Commercial Off-The-Shelves (COTS) Service Orientation (SOA)

Software Quality... Usability Users can learn it and fast and get their job done easily Efficiency It doesn’t waste resources such as CPU time and memory Reliability It does what it is required to do without failing Maintainability It can be easily changed Reusability Its parts can be used in other projects, so reprogramming is not needed 61

Software Quality... Customer: User: solves problems at easy to learn; an acceptable cost in efficient to use; terms of money paid and helps get work done resources used QUALITY SOFTWARE Development manager: Developer: sells more and easy to design; pleases customers easy to maintain; while costing less easy to reuse its parts to develop and maintain 62

Software Quality The different qualities can conflict Increasing efficiency can reduce maintainability or reusability Increasing usability can reduce efficiency Setting objectives for quality is a key engineering activity You then design to meet the objectives Avoids ‘over-engineering’ which wastes money Optimizing is also sometimes necessary E.g. obtain the highest possible reliability using a fixed budget 63

Homework ‘Teach yourselves’ UML diagrams This way you can start the assignment with a head- start.