Presentation on theme: "Sakai 2.6 Post-Mortem Overview, Discussion and Solutions Peter Peterson, QA Director, Sakai Foundation."— Presentation transcript:
Sakai 2.6 Post-Mortem Overview, Discussion and Solutions Peter Peterson, QA Director, Sakai Foundation
2.6.0 Post-mortem “Sometimes a scream is better than a thesis” Ralph Waldo Emerson
Overview Bottom-line Sakai 2.6’s release originally planned for early Jan 2009 is currently set to release in mid July 2009 Many factors contributed to this delay Processes, structure, reporting tools and resource management are all being examined Not reactionary, but rather opportunistic
Overview (cont.) Goals for this discussion Solutions to some of the issues raise in the post- mortem Future of QA and RM in Sakai Consensus And most important buy-in July 200910th Sakai Conference - Boston, MA, U.S.A.4
Observations and Feedback July 200910th Sakai Conference - Boston, MA, U.S.A.5
General Issues Jira process is convoluted and confusing Hiring of QA director mid-stream in release Setting firm release dates is necessary, but can set the wrong expectation Unclear QA reporting
Resources Project abandonment Attrition of key support players QA resource drain Future resource problems
Solutions “The solution of every problem is another problem” Johann Wolfgang von Goethe
Solutions The following solutions are derived from discussions, feedback and proposals surrounding Sakai 2.6 release and past post-mortems. Associated Links 2.6 Post-Mortem http://confluence.sakaiproject.org/x/OQAiAQ http://confluence.sakaiproject.org/x/OQAiAQ Post-Mortems Past and Present (Notes and Ideas) http://confluence.sakaiproject.org/x/OQAiAQ http://confluence.sakaiproject.org/x/OQAiAQ 2.6 QA test reporting pages (new design) http://confluence.sakaiproject.org/x/FYApAQ http://confluence.sakaiproject.org/x/FYApAQ July 200910th Sakai Conference - Boston, MA, U.S.A.10
General Solution Ideas Shared QA resources across early adopter installations Conscription: new Sakai members Automation Ownership Ideas and feedback from the 2.6 release and other post-mortemspost-mortems July 200910th Sakai Conference - Boston, MA, U.S.A.11
Automation Have a bug, write unit test first, then commit the fix Required unit tests for all bugs and fixes. These will be incorporated into the regression suites Build comprehensive regression suites that can be easily run and rerun by testers and developers July 200910th Sakai Conference - Boston, MA, U.S.A.12
Separate Core and Tool July 200910th Sakai Conference - Boston, MA, U.S.A.13
Problems We lump everyone in the same risk profile. Different users have different tolerances of risk - some will run contrib tools early others wont. At the moment its very difficult for a user with a high tolerance of risk to run new code early. We're in essence breaking the OS mantra of "release early, release often", and loose the informal QA we'd get from these users. July 200910th Sakai Conference - Boston, MA, U.S.A.14
Problems (cont.) Too much code in a release. Sakai is huge and we don't differentiate between code that hasn't changed over several release cycles and code that is evolving. We use maven very badly. Refer to how K1 is managed currently. We break the dependency of Sakai tools on master for building. This is a bizarre pattern that gives every tool a dependency on every other tool. July 200910th Sakai Conference - Boston, MA, U.S.A.15
Solution Tools should inherit from the kernel and where more complex dependencies are needed we can define and release poms to meet the needs of developer (e.g. Sakai-gradable-velocity-tool.pom) Identify stable pieces of code that are not deployed to tomcat themselves but are used by other Sakai tools and cut binary releases of them. There are several of these, mostly libraries and poms in velocity and JSF but I'm sure there are others. July 200910th Sakai Conference - Boston, MA, U.S.A.16
Solution (cont.) Learn from the K1 experience - this has worked well - is there other code we could do this with? I'd like to tentatively suggest we may want 2 other bundles that provide stable widely used functionality: 1) Common services (commons project (SakaiPerson & Type Service, Taggable Service, Privacy Service, Email Template service) 2) Common edu service (Course Management, Content Review, Gradebook Service ? ) July 200910th Sakai Conference - Boston, MA, U.S.A.17
Solution (cont.) Identify very stable projects (such as the admin tools) and tag them with their own version and leave them between releases. For instance its been 8 months since a change to the Admin user tool, bar translation updates and housekeeping. July 200910th Sakai Conference - Boston, MA, U.S.A.18
Summary and Comment QA process becomes more of oversight and due diligence process in these cases our role is to satisfy ourselves that the code we include is good quality, and to support those teams in achieving this. July 200910th Sakai Conference - Boston, MA, U.S.A.19
Summary and Comment (cont.) Each tool must begin performing their own releases and generating their own artifacts. The Sakai release process needs to become more of an assemblage of already released artifacts than where we are today. I have been using Redhat as an example for years in this regard - At each release cycle they are looking around to see which projects are mature enough, and high quality to include in a packaged release. IMHO, this is where we need to be. July 200910th Sakai Conference - Boston, MA, U.S.A.20
Separate Releases for Mature Projects Less monolithic release practice -- separate releases for mature projects, Reassemble for general release--more like K1. July 200910th Sakai Conference - Boston, MA, U.S.A.21
Problem Our current "rationed" maintenance release strategy. Adds an unnecessary level of complexity to release management process. Resulting artifacts from a fix perspective tend to fall behind the maintenance branch due to delays in the release cycle. For example, a completed fix may sit unimplemented for many months before it finally makes it into a maintenance release.
Solution Strict up front branch management. What is merged into the maintenance branch is solid code, with a test case, and sign off from QA. Maintenance releases could then be generated rather quickly: we decide on a revision point that constitutes the next release (2.6.1), we copy to a 2.6.1 release branch, clean up the poms, write up release notes and cut the release. Update the 2.6.x poms version number to the next SNAPSHOT version (e.g., 2.6.2-SNAPSHOT) and recommence the cycle.
Summary By performing the QA upfront, there would be no more alpah/beta/rc tags for maintenance releases (e.g., the Tomcat team has not put out an beta tag since 5.5.17). No additional testing on the release branch is required other than ensuring the artifacts generated from it (demo, bin, src) start up in Tomcat.
Summary (cont.) To operationalize this requires an increased commitment from the community to support active, current and ongoing QA of maintenance work Inclusion of test cases with every fix/patch Increased sharing and coordination of development activities, especially with tools with implicit dependencies.
Problem Incentives to participate in the release process are ineffective. Release managers can offer no incentives to developers to have them maintain code and developers have little incentive to do the "last mile" changes and testing when that work doesn't have local consequences. Altruism is a wonderful but weak incentive.
Solution CORE Only the code and tools necessary to run a basic instance Foundation will ensure QA, maintenance and updates are performed Create clear technical quality standards for code and QA to assess whether or not additional projects can be considered to be "Sakai quality".
Solution (cont.) Evaluate non-core projects based on these standards. Include non-core projects in a release only if they meet these objective standards. (No tools are grandfathered in.) Create a general installer so that projects that are not in the core release are easily added to their locally builds
Summary Ensures that a) inclusion in a release is an assurance of tool quality and b) inclusion in a release is up to the developers themselves. Exclusion from a release doesn't mean that a project is low quality, just that it hasn't met the “Sakai quality”. The risk/benefit tradeoff of using these projects needs to be assessed by the local adopter based on their needs.
Summary (cont.) Simplifies and regularizes the release process considerably. Assures items in a release will be high quality. Installations can easily be customized for local needs by adding tools not in the release. Anyone who wants a tool in the release need only to look at the criteria and make sure that the project meets them. If no one is concerned enough to ensure the project meets the appropriate standards this project simply doesn't belong in the release.
Comments Each tool must begin performing their own releases and generating their own artifacts. The Sakai release process needs to become more of an assemblage of already released artifacts than where we are today. I have been using Redhat as an example for years in this regard - At each release cycle they are looking around to see which projects are mature enough, and high quality to include in a packaged release. IMHO, this is where we need to be. (Lance Speelmon)
Comments (cont.) It's not unreasonable to ask contributors or committers (or their managers) about their availability to support changes to new code or complex fixes through a release, and if that availability is uncertain, it wouldn't be unreasonable to keep such a feature out of the release. (Stephen Marquard)
Conclusion July 200910th Sakai Conference - Boston, MA, U.S.A.35