We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byMikaela Thrift
Modified about 1 year ago
© Microsoft Corporation Analyze This! 145 Questions for Data Scientists in Software Engineering Andrew Begel, Thomas Zimmermann Microsoft Research, USA June 4, 2014, ICSE 2014
© Microsoft Corporation Meet Greg Wilson (Mozilla) The plural of anecdote is not evidence.
© Microsoft Corporation It will never work… in theory. Blog post on August 22, 2012 Ten Questions for Researchers Greg asked a software developer friend at Mozilla, “Give me a list of 10 questions that you’d really like software engineering researchers to answer.”
© Microsoft Corporation Find me some evidence! emacs vs. vi vs. IDEs: which one makes me more productive? Is it really twice as hard to debug as it is to write the code in the first place? When does it make sense to reinvent the wheel vs. use an existing library? Are conferences worth the money? How much do they help junior, intermediate, or senior programmers? (and 6 more…)
© Microsoft Corporation Tom and Andrew: Let’s ask Microsoft engineers what they would like to know!
© Microsoft Corporation http://aka.ms/145Questions
© Microsoft Corporation ❶
categories Analyze This! 145 Questions for Data Scientists in Software Engineering. Andrew Begel, Thomas Zimmermann.
© Microsoft Corporation raw questions (provided by the respondents) “How does the quality of software change over time – does software age? I would use this to plan the replacement of components.”
© Microsoft Corporation raw questions (provided by the respondents) “How does the quality of software change over time – does software age? I would use this to plan the replacement of components.” “How do security vulnerabilities correlate to age / complexity / code churn / etc. of a code base? Identify areas to focus on for in-depth security review or re-architecting.”
© Microsoft Corporation raw questions (provided by the respondents) “How does the quality of software change over time – does software age? I would use this to plan the replacement of components.” “How do security vulnerabilities correlate to age / complexity / code churn / etc. of a code base? Identify areas to focus on for in-depth security review or re-architecting.” “What will the cost of maintaining a body of code or particular solution be? Software is rarely a fire and forget proposition but usually has a fairly predictable lifecycle. We rarely examine the long term cost of projects and the burden we place on ourselves and SE as we move forward.”
© Microsoft Corporation raw questions (provided by the respondents) “How does the quality of software change over time – does software age? I would use this to plan the replacement of components.” “How do security vulnerabilities correlate to age / complexity / code churn / etc. of a code base? Identify areas to focus on for in-depth security review or re-architecting.” “What will the cost of maintaining a body of code or particular solution be? Software is rarely a fire and forget proposition but usually has a fairly predictable lifecycle. We rarely examine the long term cost of projects and the burden we place on ourselves and SE as we move forward.” descriptive question (which we distilled) How does the age of code affect its quality, complexity, maintainability, and security?
© Microsoft Corporation ❷ Discipline: Development, Testing, Program Management Region: Asia, Europe, North America, Other Number of Full-Time Employees Current Role: Manager, Individual Contributor Years as Manager Has Management Experience: yes, no. Years at Microsoft
© Microsoft Corporation Microsoft’s Top 10 Questions Essential Essential + Worthwhile 1. How do users typically use my application?80.0%99.2% 2. What parts of a software product are most used and/or loved by customers? 72.0%98.5% 3. How effective are the quality gates we run at checkin?62.4%96.6% 4. How can we improve collaboration and sharing between teams? 54.5%96.4% 5. What are the best key performance indicators (KPIs) for monitoring services? 53.2%93.6% 6. What is the impact of a code change or requirements change to the project and its tests? 52.1%94.0% 7. What is the impact of tools on productivity?50.5%97.2% 8. How do I avoid reinventing the wheel by sharing and/or searching for code? 50.0%90.9% 9. What are the common patterns of execution in my application? 48.7%96.6% 10. How well does test coverage correspond to actual code usage by our customers? 48.7%92.0%
© Microsoft Corporation Microsoft’s 10 Most Unwise Questions Unwise 1. Which individual measures correlate with employee productivity (e.g. employee age, tenure, engineering skills, education, promotion velocity, IQ)? 25.5% 2. Which coding measures correlate with employee productivity (e.g. lines of code, time it takes to build software, particular tool set, pair programming, number of hours of coding per day, programming language)? 22.0% 3. What metrics can use used to compare employees?21.3% 4. How can we measure the productivity of a Microsoft employee?20.9% 5. Is the number of bugs a good measure of developer effectiveness?17.2% 6. Can I generate 100% test coverage?14.4% 7. Who should be in charge of creating and maintaining a consistent company- wide software process and tool chain? 12.3% 8. What are the benefits of a consistent, company-wide software process and tool chain? 10.4% 9. When are code comments worth the effort to write them?9.6% 10. How much time and money does it cost to add customer input into your design? 8.3%
© Microsoft Corporation Discipline Differences (Essential %) DevTestPM How many new bugs are introduced for every bug that is fixed?27.3%41.9%12.5% When should we migrate our code from one version of a library to the next? 32.6%16.7%5.1% How much value do customers place on backward compatibility? 14.3%47.1%18.3% What is the tradeoff between frequency and high quality when releasing software? 22.9%48.5%14.5% Role Differences (Essential %) Manager Individual Contributor How much legacy code is in my codebase?36.7%65.2% When in the development cycle should we test performance?63.3%81.4% How can we measure the productivity of a Microsoft employee?57.1%77.3% What are the most commonly used tools on our software team?95.8%67.8%
© Microsoft Corporation Region Differences (Essential %) AsiaEurope North America How can we measure the productivity of a Microsoft employee? 52.9%30.0%11.0% How do software methodologies affect the success and customer satisfaction of shrinkwrapped and service-oriented products? 52.9%10.0%24.7% Can I generate 100% test coverage?60.0%0.0%9.0% What is the effectiveness, reliability, and cost of automated testing? 71.4%12.5%23.6% Mgmt Experience Differences Years as Manager (change in odds per year) How much cloned code is ok to have in my codebase? (Essential) 36% How does the age of code affect its quality, complexity, maintainability, and security? (Essential + Worthwhile) -28%
© Microsoft Corporation MSFT Experience Differences Years at Microsoft (change in odds per year) What criteria should we use to decide when to use managed code or native code (e.g., speed, productivity, functionality, newer language features, code quality)? (Essential) -23% What are the best tools and processes for sharing knowledge and task status? (Essential) -18% Should we do Test-Driven Development?(Essential) -19% How much distinction should there be between developer and tester roles? (Essential + Worthwhile) -14% Who should write unit tests, developers or testers?(Essential + Worthwhile) -13% How much time went into testing vs. development?(Essential + Worthwhile) -12%
© Microsoft Corporation Takeaway: Focus on Important Questions Research: Academic-industry collaborations Tech Transfer Guidance for academic research Practice: Collect metrics and build tools at large scale Spread knowledge of existing tools Make tools more applicable and easier to use. Education: Contextualize data science education Illustrate real-life challenges of software professionals Model analytics process for students: pose high-quality, specific, actionable questions, write up coherent reports, follow up with teams to help them implement improvements Please replicate our study at your company!
Top Down View of Estimation Test Managers Forum 25 th April 2007.
Stoimen Stoimenov QA Engineer QA Engineer SitefinityLeads,SitefinityTeam6 Telerik QA Academy Telerik QA Academy.
Does Distributed Development Affect Software Quality???? An Empirical Case Study of Windows Vista Christian Bird, Premkumar Devanbu, Harald Gall, Brendan.
SE 450 Software Processes & Product Metrics 1 Defect Removal.
A COMPETENCY APPROACH TO HUMAN RESOURCE MANAGEMENT.
CS 1120: Computer Science II Software Life Cycle Slides courtesy of: Prof. Ajay Gupta and Prof. James Yang (format and other minor modifications by by.
CS223: Software Engineering Lecture 18: The XP. Recap Introduction to Agile Methodology Customer centric approach Issues of Agile methodology Where to.
Advanced Software Engineering Lecture 4: Process & Project Metrics.
Extreme/Agile Programming Prabhaker Mateti. ACK These slides are collected from many authors along with a few of mine. Many thanks to all these authors.
Hussein Alhashimi. “If you can’t measure it, you can’t manage it” Tom DeMarco,
Requirements Engineering Requirements Engineering in Agile Methods Lecture-28.
Swami NatarajanJune 23, 2015 RIT Software Engineering Defect Removal.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 17 Slide 1 Extreme Programming.
Facts and Fallacies of Software Engineering (Rob Glass) CSE301 University of Sunderland Discussed by Harry R. Erwin, PhD.
Software Maintenance Speaker: Jerry Gao Ph.D. San Jose State University URL: Sept., 2001.
Agile SOA Agile EAI How do we achieve agility in Enterprise Integration?
Rapid software development 1. Topics covered Agile methods Extreme programming Rapid application development Software prototyping 2.
Chapter 10 Systems Development © 2008 Pearson Prentice Hall, Experiencing MIS, David Kroenke.
High Level: Generic Test Process (from chapter 6 of your text and earlier lesson) Test Planning & Preparation Test Execution Goals met? Analysis & Follow-up.
Computer Engineering 203 R Smith Agile Development 1/ Agile Methods What are Agile Methods? – Extreme Programming is the best known example – SCRUM.
Accelerating Development Using Open Source Software Black Duck Software Company Presentation.
CODING Research Data Management. Research Data Management Coding When writing software or analytical code it is important that others and your future.
Compuware Corporation Deliver Reliable Applications Faster Dave Kapelanski Automated Testing Manager.
1 Design, Implementation and Maintenance The “lower end” of the lifecycle focuses on –Detailed logical and technical specification of the system or system.
May 29 th, 2003 Curtis Anderson Sivaprasad Padisetty.
Applying Adaptive Software Development (ASD) Agile Modeling on Predictive Data Mining Applications: ASD-DM Methodology M. Alnoukari 1 Z.Alzoabi 2 S.Hanna.
Chapter 2 Software processes. Topics covered Software process models Process activities Coping with change.
SE 450 Software Processes & Product Metrics Reliability Engineering.
Usability Process for eBP at Intel Eric Townsend, Intel.
Testing Challenges in an Agile Environment Biraj Nakarja Sogeti UK 28 th October 2009.
Systems Analysis & Design 5 th Edition Chapter 1 Introduction to Systems Analysis and Design.
Test Automation Success: Choosing the Right People & Process Kiran Pyneni, Automation Manager Aetna, Inc.
Configuration Control (Aliases: change control, change management )
Copyright © The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the OWASP License. The OWASP.
Extreme Programming Team Members Gowri Devi Yalamanchi Sandhya Ravi.
Achieving (and Maintaining) Compliance With Secure Software Development Compliance Requirements (ISC)² SecureSDLC May 17, 2012.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 17 Slide 1 Rapid software development.
An Introduction to Software Engineering (Chapter 1 from the textbook)
Problems with reuse – Increased maintenance costs; lack of tool support; not-invented- here syndrome; creating, maintaining, and using a component library.
Alan Page Director, Test Excellence Microsoft. A little about the book A little about the “SDET” A little about me Some other stuff.
What is Software Engineering? the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software”
1 CS 501 Spring 2002 CS 501: Software Engineering Lecture 23 Reliability III.
Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software Brian Chess Fredrick.
Software evolution. Objectives l To explain why change is inevitable if software systems are to remain useful l To discuss software maintenance and maintenance.
CS350/550 Software Engineering Lecture 1. Class Work The main part of the class is a practical software engineering project, in teams of 3-5 people There.
This information is confidential and proprietary to Lawson ® and cannot be reproduced without permission. Reducing Resolution Time with Advanced Electronic.
CSE 308 Software Engineering Software Engineering Strategies.
CS 5150 Software Engineering Lecture 22 Reliability 3.
SONIC-3: Creating Large Scale Installations & Deployments Andrew S. Neumann Principal Engineer, Progress Sonic.
Objective ICT : Internet of Services, Software & Virtualisation FLOSSEvo some preliminary ideas.
© 2017 SlidePlayer.com Inc. All rights reserved.