Presentation on theme: "Effectiveness of Systems Engineering Techniques on New Product Development: Results from Interview Research at Corning Incorporated Presentation to INCOSE."— Presentation transcript:
1 Effectiveness of Systems Engineering Techniques on New Product Development: Results from Interview Research at Corning IncorporatedPresentation to INCOSE International SymposiumChicago, ILJuly 13, 2010Francis Vanek, Rich Grzybowski, Peter Jackson, andMatt Whiting
2 Energy Systems Engineering: Evaluation and Implementation Features:Systems approach to energyMajor energy sources:FossilNuclearRenewableTechnical and financial feasibilityFlyers available afterward
3 Opening Thought: Does the ‘Systems Approach’ Matter? From the field of energy systems: the ‘plug-in hybrid’ + ‘vehicle-to-grid’ future vision:Stakeholder 2:Liquid FuelProvidersStakeholder 1:Electric UtilitiesStart by thinking expansively about systems engineering and the systems approachVision of future: advanced plug-in hybrid cars run on either electricity from a variety of sources or liquid fuels, and can charge and discharge at home, at work, anywhere.Systems approach says we need to work with stakeholders to understand the market or the missionTwo GIANT stakeholders: electric utilities and liquid fuel providers. If we don’t start by understanding these stakeholders, this grand vision is likely to fail.
4 Outline of this talk Project Background Development of a Methodology Application within Corning IncorporatedObservations and ConclusionsNon-disclosure agreement: Corning projects are identified bynumber only, and any identifying information has been masked.
5 ThemesHeart of the research: 19 on-site interviews in Corning, NY, April 2008 to March 2009Two hypotheses for today’s talk:Interview and analysis methodology is effective for learning about SE practiceThe methodology specifically applied to Corning shows correlation between use of SE and project performance
6 Genesis of Project Discussion in 2005: Aware of measurement of SE effectiveness in mil/aero sector“What about commercial world?”Launch Corning-Cornell project in 2006
7 SE Effectiveness Research: Challenge Posed and Previous Work Sarah Sheard:Limitations on quantifying ROIEric Honour:Previous work comparing multiple projectsNational Defense Industry Association (NDIA):2007 study of projects in member organizations(Keep this snappy)Sarah has written persuasively on the difficulty of quantifying what is the percent returnWe’ve gone after a simpler question: can you empirically justify more SE input to avoid product development pitfalls?NDIA: not only for the results but their documentation of their methodology, useful for designing our own study
8 Background about Corning Incorporated World leader in specialty glass and ceramics.Create and make keystone components to enable high-technology systemsConsumer electronics, mobile emissions control, telecommunications and life sciences.Sustained investment in R&D at 10%150+ years of materials science and process engineering knowledgeDistinctive collaborative culture.Founded1851HeadquartersCorning, New YorkEmployeesApproximately 24,000 worldwide2009 Sales$5.4 BillionFortune 500 Rank (2009)414R&D based in Corning, NYCorning: Fortune 500 company.Specialty is providing keystone components that make possible advanced technologies ranging from consumer electronics to mobile emissions controls.Headquarters in Corning, is also site of the research park where the interviews took placeCLICK
9 Corning’s market segments LCD Glass SubstratesGlass Substrates for OLED and LTPS-LCDOptical Fiber & CableHardware & EquipmentFiber optic connectivity productsOptical connectivity productsEmissions Control ProductsLight-duty gasoline vehiclesLight-duty and heavy-duty diesel vehiclesStationaryCell Culture & BioprocessAssay & High-Throughput ScreeningGenomics & ProteomicsGeneral Laboratory ProductsDisplay Optics & ComponentsSemiconductor Optics & ComponentsAerospace & DefenseAstronomyOptical MetrologyOphthalmicTelecom ComponentsSpecialty GlassDisplay FuturesNew Business DevelopmentDrug Discovery TechnologyEquity CompaniesCormetech, Inc.Dow Corning Corp.Eurokera, S.N.C.Samsung Corning Precision Glass Company, LTD (SCP)SpecialtyMaterialsOtherProducts& ServicesLifeSciencesEnvironmentalTechnologiesTelecomDisplayTechnologyFive many market segments as you see from display technologies to life sciences. The 19 Project that we interviewed were spread across these 5 market segments
10 The “home-run hitting” Importance of Flexibility in Studying Use of SE Process across Different FirmsThe “base hitting”enterpriseVersusIf we look at two different types of firms, Corning is a “home-run hitter” that looks to make a business out of critical breakthroughs that make new commercial products possible.Different from the “base hitter” that launches new products incrementally by making relatively small changes to existing know-how.There is no one right or wrong answer, but it is important to be able to adapt how you study the use of the SE Process to reflect the nature of the firmThe “home-run hitting”enterpriseImages source: Google images
11 Different from findings in related fields: Background from Literature Review: Search for Evidence of SE EffectivenessMain findings:Isolated instances of connection between SE and project successNo systematic studies of SE effectiveness in commercial worldDifferent from findings in related fields:Total Quality Management (TQM) and Six SigmaAdd reference to literature review
12 Outline Project Background Development of a Methodology Application within Corning IncorporatedObservations and Conclusions12
13 Developing a methodology Ontology (Framework for understanding SE)Specific case of Corning Incorporated2. General approach to studying projects
14 1. Ontology: Commonalities among SE standards Source: Honour & Valerdi (2006), “Advancing an ontology for systems engineering to allows consistent measurement”.Conference on Systems Engineering Research, Los Angeles, 2006.
15 Eight major areas of SE Input Market analysisRenaming of “mission definition”Requirements engineeringSystems architectingSystems implementationTechnical analysisTechnical managementScope managementVerification & validation15
16 NDIA ‘Mosaic Diagram’: “Better SE Capability = 2. General Approach: NDIA Study as Example46 projects from 22 NDIA membersFound discernable correlationSelf-reported limitationsUse of ‘proxies’ within organization“Most problematic” part of studyUneven response rateOverall response below expectation46 projects grouped into 3 groups, based on whether they had low, medium, or high capability. The three groups are then evaluated in terms of performance, ranging from red for less well performing, to green for high performing.As we move from left to right, performance in general improves, showing correlationLow SE Med. SE High SENDIA ‘Mosaic Diagram’:“Better SE Capability =Better Performance”
17 Nomenclature Used during Project “SE Director”:From enterprise side (Corning in this case)Hosted project, lined up interviews“Interviewer”:From outside (Cornell in this case)Carried out interviews“Interviewee”:PMs and SEs from specific projects
18 Designing and Launching Interview Process SE Director and interviewer design interview before startingFlexibility in methodology:Choose SE techniques based on characteristics of firmLimited time for interviewsFocus on key areasCustomize questions for firmScoring questions: 0, ½, or 1 pt
19 Characteristics of Interviews Techniques for improving qualityNon-disclosure agreementDirectly interview project managers / systems engineersRequire interview to show evidence (reports, powerpoints, etc)NOT ALLOWED to take data away from interviewEvaluate results AFTER interviewRetain key statements of interviewees for affinity grouping analysis
20 3. Specific Approach: Interviews at Corning Incorporated SE input componentChose four SE areas for CorningMarket analysisRequirements analysisVerification & validationTechnical analysis (renamed “tradeoff analysis” for Corning study)14 questions total on SE inputProject performance componentSchedule & budget adherenceProgress toward goals
21 Outline Project Background Development of a Methodology Application within Corning IncorporatedObservations and Conclusions21
22 Presentation of Results from within Corning Three questions:Q1. What is the level of SE Input?Q2. How is the project performing?Q3. Was there correlation?22
23 Background on Projects Interviewed Corning use of “Stage Gate” processIdentify project’s level of maturity“Stage Gate Review” determines whether or not project passes to next stageAll projects were either early- or mid-stageEither in progress or shelvedShelved projects: check for archiving of findings
24 Questions Asked About Market Analysis (SE Area 1) “To what extent did you analyze:“The overall market (including total size, segmentation, or target share)?”“Response of specific prospective customers to the product?”“Position of competitors regarding product?”“What is the current value proposition of the product?”
25 Example of Response: a Project with strong Market Segmentation Analysis… “Global market for product X by annual gross valueTotal value: $X million/year”(SimulatedData)
26 …Detailed Comparison of Offering… “C.O.O. for PROPOSED product X compared toexisting products from Competitors A, B, and C”
27 …and Clear Value Proposition… Value to CustomerValue to Corning…will score well in “Market Analysis” SE area
28 Highlights from SE Areas 2-4 Won’t present remaining 10 questions (for brevity)Highlight #1: Trace from value proposition to requirements to testing to tradeoff analysesHighlight #2: Scheduling of testing, progress toward meeting requirements, etc.
29 Results from Evaluation of Level of SE Input Projects could earn maximum of 3 or 4 points in each areaScoring reported in terms of percent of maximumBoth for each of 4 SE areas and overall scoreEx: 7 out of 14 possible points = 50% overallRating of score based on mean & S.D.One SD above mean: threshold for high inputOne SD below mean: threshold for low input
30 Result for Overall Project Input: Ranking by Percent of Possible Points μ + = 71%μ = 58% = 13%μ - = 45%Code for Level of SE Input: Green = High, Blue = Mid, Red = Low. Breaks at +/- 1 SD from mean
31 Contribution to Overall Percent Score by SE Areas 1 thru 4: HighestAverage:MarketLowest:TradeoffVariability in SE input observed. What about performance?
32 Characterizing Project Performance Original goals: use project dataAdherence to schedule, budget, staffingProjection of likely product success based on development to dateAvailable data did not supportLack of differentiation for point #1Difficult to make projections for point #2Substitute: subjective evaluation. Possible ratings:“Satisfactory”“Struggling”“Superior”Corroborated with performance data where possiblePreliminary assessment:Agreement to revisit with Corning in early 201132
33 Project performance: A “satisfactory” project Steady progress toward maturityOccasional delays, small cost overruns, etcLooking at documentation reveals small shortfallsExpected product requirements not met on scheduleMemoranda or Powerpoint shows reveal minor “glitches” in everyday project life13 out of 19 projects
34 Project performance: A “struggling” project Characteristics:Schedule delays or cost overruns beyond the “noise” levelMajor intervention and reworkChronic difficulties in communicating with customer or upper managementFailure to progress through stage gates over extended period3 out of 19 projects
35 Project performance: A “superior” project Characteristics:Faster-than-expected progress through Stage Gate processAccelerated growth in customer interestDocumentation suggesting strong ROI potential once completeAwards won in design competitions3 out of 19 projects
36 So How Does SE Input Match Up Against Project Performance?
37 Comparison: Focus on Superior and struggling Projects Satisfactory Projects: Usually near mean SE scoreSuperior Projects: Consistently high scores across boardAverage Overall SE Input for 3 projects = 83%All other projects (16) = 53%Struggling Projects: Fall short on one or more SE Inputs
38 Comparison: Superior vs. Struggling Proj:1917SatisfactoryProjects7811Mkt:Reqs:V&V:Trade:PROJPERF:Graphical presentation: Choose 1 project from green and red and explain
39 Mosaic Diagram of Overall SE Input: Some Degree of Correlation Interpretation: correlation is lumpy but perceptible.
40 Could Project Maturity Explain Degree of SE Input and Project Performance? Struggling projects might also be early stageAs projects pass stage gates, maturity leads to better performanceComparison of struggling and superior projects2 out of 3 struggling projects in Stage 3Other one had advanced to S3, then returned to S2Superior projects: 2 in Stage 3, 1 in Stage 4Conclusion: does not explain in this case
41 Interpretation of Findings Medium or high SE input does not guarantee high project performanceBut it CAN help project to progress smoothlyStruggling projects can have overall SE Input score at or near averageBUT, low SE Input scores in 1 or 2 areas can be correlated with struggling performanceMatches anecdotal evidence from project history
42 Outline Project Background Development of a Methodology Application within Corning IncorporatedObservations and Conclusions42
43 Observations from the Interview Research Process SE Director should not “telegraph” performance of project to interviewerInterviewer should not “telegraph” underlying SE Input under study to intervieweePre-interview phone call to establish rapportReview of data useful to interviewee also“Where did that value proposition go anyway?”
44 Evaluation of Hypothesis 1: Effectiveness of Methodology Yes, the methodology is effective:With NDA in place, interviewees spoke candidlyRequirement to show documentation improves accuracy and is manageableTime commitment manageable for interviewer: 20 interviews over ~1 year, 1 or 2 interviews per visitFor interviewer at a distance, challenge of schedulingInterviewer should come from outside firmApproach is repeatable at other firmsChoice of SE techniques and questions to ask is flexible
45 Evaluation of Hypothesis 2: Correlation within Corning Within the limits of our number of surveys & preliminary assessment of performance, we uncovered evidence thatSuperior projects had higher average SE input than satisfactory or strugglingStruggling projects had low SE input in at least 1 of 4 SE Input areasFindings support greater emphasis on SE input“Precautionary principle”: if lack of SE leads to failure some of the time, investing in SE is worth it
46 Possible Next Steps Repeat methodology within other firms Help SE Directors to create firm-specific case for SEFurther test adaptability of interview techniqueDevelop meta-analysis of SE effectivenessUse results from multiple firmsMulti-project, multi-firm data setDevelop materials on metrics and measurement in INCOSE systems engineering handbook
47 What is the Value of SE? Parting Quote Gathered from Interview Affinity grouping of retained quotes from intervieweesQuotes as well as scores are integral part of data gathering“Gist” of quote instead of verbatim transcriptFrom affinity group “quotes in support of SE”:
48 Thank You for Attending! Corning Museum of GlassCorning, NYCornell UniversityIthaca, NYQ&A, DiscussionSource: ;
50 Additional Resources This Powerpoint show: Link on DropBoxFull final report: “Corning-Cornell SEROI Study”Link:Conference paper for IS 2010 Chicago:Link to full report on DropBox
51 Comparison of UHFs at Boeing: Compared to UHF1, UHF 2-3 achieved: More product complexity for similar costShorter integration test cycleShorter overall project duration
52 Questions Asked About Requirements Analysis (SE Area 2) “What are the Technical Performance Measures (TPMs) for evaluating product?”“What is the ‘trace’ of the value proposition to the TPMs?”“What is the anticipated schedule for achieving TPMs, if one exists?”
53 Questions Asked About Verification & Validation (SE Area 3) “Do you have a test plan for product development?”“What is the trace from the TPMs to the testing procedures?”“What is the schedule, if any, for carrying out required tests designated in the test plan?”
54 Questions Asked About Tradeoff Analysis (SE Area 4) “What are the criteria and alternatives used in tradeoff analyses?”“What is the trace from the product TPMs to the tradeoff analysis?”“What documentation did you produce of research before and selection rationale after the analysis?”“What evidence can you provide of stakeholder involvement in the tradeoff analysis?”
55 Review of Individual Area and Overall SE Input as Basis for Comparison Mean scores by areaMarket Analysis = 75%Requirements = 59%Verification & Validation = 53%Tradeoff Analysis = 49%Overall SE Input71% or more: high SE input45% to 71%: medium SE input45% or less: low SE input
56 Comparison: Focus on Superior and struggling Projects Satisfactory Projects: Usually near mean SE scoreSuperior Projects: Consistently high scores across boardAll 3 projects: HI input for market analysisProjects 1 & 9: MED input on Reqs/V&V/TradeoffProject 17: HI on all but RequirementsStruggling Projects: Fall short on one or more SE InputsProject 7: LO input on V&VProject 8: LO input on requirementsProject 11: LO input on market + requirements56