2. A Top Ten You Can Use: Ten Key Fixes that Avoid Big Problems in Procurements and Proposals Breakout Session #: G02 Presented by: Jacqueline Owens Lancaster Date: 24 July 2013 Time: 9:45 a.m.

3. About Your Speaker 2 Jacqueline Owens Lancaster (Jackie O.)  CBDO/VP, ASI Government, since 2011  BD/Capture/Proposal executive roles in major Government contracting firms since 1996  Worked on first proposal in 1985  Started career supporting “Big A” acquisition for U.S. Navy (NAVAIR)  MS, Management, University of Maryland  BS, Mass Communications, Virginia Commonwealth University

4. Who’s in the audience? World Congress 2013 3 I have written or contributed to developing RFPs/ RFQs for the Government I have written proposals in response to Government RFPs/RFQs I’ve done both! Right hand up Left hand up Both hands up More and more of us have been on both sides, so let’s face it… …we need to stop hurting each other!

5. Little things can lead to big trouble 4 World Congress 2013 GovernmentIndustry Failure to allot adequate market research time Generic proposal pricing requirements Delays caused by need to amend RFP Discussions required to complete evaluation Winner selected, but paper trail very messy Limited debriefs to prevent protest Failure to respond with good pricing data Multiple questions on pricing requirements Confusing proposals based on RFP changes Further confusion in Final Proposal Revisions Protest based on conflicting requirements Everyone but winner angry and dissatisfied 4

6. 5 World Congress 2013 5 How do we recognize, avoid, or -- as a last resort – fix these “little things” before they become big problems? So here’s the multi-million-dollar question:

7. 6 World Congress 2013 6 Start by learning from the past … with a little help from late night TV Top 10 List: Ten Key Fixes that Avoid Big Problems in Procurements and Proposals

8. 7 World Congress 2013 7 (First person to match one on my list gets a prize!) But before we get started… Let me ask you… What do you do to avoid problems in acquisitions before they occur?

9. 8 World Congress 2013 8 Number 10: Example In 2011-2012, an in-depth review of the Joint Tactical Radio System (JTRS) program revealed that inadequate market research had been conducted as changes in commercial items and user needs occurred. Upon conduct of market research that revealed user needs had dramatically changed and non-developmental items were available to address them, the existing contract was modified to end all remaining development efforts. - GAO Report, “Defense Acquisitions: Assessments of Selected Weapons Programs,” March 2013 If you don’t do it, what happens?  No understanding of “realm of the possible”  No picture of support requirements  Unawareness of commercial products  Failure to bundle for efficiency  Limited understanding of potential risks  No insight on industry standards/benchmarks  Inadequate number of qualified vendors  Unfamiliarity with small business providers  No visibility into potential incentives  Inclusion of unnecessary constraints  Few criteria to evaluate good/not so good  Unreasonably high prices Conduct market research to enable selection of effective, differentiating evaluation criteria

10. 9 World Congress 2013 9 Number 10: Conduct market research to enable selection of effective, differentiating evaluation criteria Do: Conduct market research across all acquisition phases Maximize contact with industry – don’t fall victim to myths! Keep your mind open – don’t assume you know the answer Focus research on identifying differentiating characteristics Target what you need to know – avoid “free form” research Contact the right people from industry – multiple disciplines Iterate – it’s generally not a “one-and-done” approach Research existing research – don’t start from scratch

11. 10 World Congress 2013 10 Number 9: Example Since 2009, DoD has cancelled three MDA programs due to concerns about technical issues, schedule delays, and cost-effectiveness and operational roles. A GAO investigation documented in 2013 revealed that the issues that ultimately caused these cancellations would most likely have been discovered before the programs began if more robust Analyses of Alternatives (AoAs) had been conducted as part of acquisition planning. - GAO Report, “Missile Defense: Opportunity to Refocus on Strengthening Acquisition Management,” April 2013 If you don’t do it, what happens?  Lack of input from some stakeholders  Lack of market research to support strategy  Inability to integrate (logically) inputs received  Inconsistency among requirements  Failure to address security requirements  Failure to consider support needs  Inadequate or unfounded IGCE  Discovery that funding is unavailable  No use of performance-based techniques  Discouragement of competition  Requirements that don’t yield desired result Take the time required to develop a comprehensive, integrated Acquisition Plan (AP)

12. 11 World Congress 2013 11 Number 9: Take the time required to develop a comprehensive, integrated Acquisition Plan (AP) Do: Use iterative market research to make AP outcomes-focused Consider full realm of needs (e.g., support, security, IT) Reach out to varied experts to get the IGCE right Consider criteria early – what separates good from better? Use cross-disciplinary reviews to check for integration Maximize up-front, cross-disciplinary stakeholder input to AP Use the “So what?” test – is the AP going to improve results? Make senior level sign-off/supervisory approval mandatory

13. 12 World Congress 2013 12 Establish and enforce accountability for the acquisition Example In 2005, GAO sustained protests from two companies challenging the Air Force's award of a sole-source contract based on an unusual and compelling urgency J&A. GAO agreed with the protestors’ claim that the agency's circumstances did not fulfill the criteria for an unusual and compelling urgency; that the urgency of the requirements was the result of the agency's inadequate procurement planning; that the agency unreasonably concluded that the awardee was the only firm capable of meeting the requirements; and that the agency failed to obtain competition to the maximum extent practicable. - GAO Protest Decision, Worldwide Language Resources; SOS International Limited, November 2005 If you don’t do it, what happens?  Rushed, “thrown over” requirements  No ownership of requirements integration  Failure to consider connection to strategy  Loss of perspective on foundational need  No consideration of past approaches to need  Failure to consider impact on competition  Loss of connection between needs and costs  Inadequate knowledge base for acquisition  Loss of focus on supportive services  No consequences for unsatisfactory input  Unbalanced acquisition that is hard to defend Number 8:

14. 13 World Congress 2013 13 Do: Use systems that provide visibility of status (e.g., SharePoint) Evaluate and provide feedback on quality of input Employ “color review” approach to build in quality Define key milestones at which authority must sign off Number 8: Establish and enforce accountability for the acquisition Identify ultimate acquisition authority early Make sure COR is identified early and involved throughout Use methods that build stakeholder engagement while cutting cycle time (e.g., Acquisition Acceleration Centers)

15. 14 World Congress 2013 14 Intensify up-front stakeholder engagement to build consensus on requirements Example In 2010, GAO sustained a protest challenging the elimination of company’s proposal from competition for a Federal Transit Administration PMO contract. GAO agreed with the protest claim that FTA’s RFP criteria were internally in conflict. Specifically, the RFP did not require the submission of cost or pricing data, but the protesting firm was evaluated against, and excluded for, not meeting the requirements of FAR Section 15.408, Table 15-2, which only apply when cost or pricing data are required. - GAO Protest Decision, PMO Partnership Joint Venture, October 2010 If you don’t do it, what happens?  Inconsistent internal view of requirement  No ID of “must haves” versus “nice-to-haves”  Many inconsistencies/contradictions in terms  Limited understanding of overall timeline  Delays caused by over-the-transom approach  Unbalanced view based on involvement level  Failure to consider novel approaches  Loss of focus on supportive services  Many difficult questions after RFP release  Inconsistent proposals  Failure to meet objectives with acquisition Number 7:

16. 15 World Congress 2013 15 Do: Identify approaches that are collaborative versus sequential Use performance-based methods to focus on outcomes Use tools that improve visibility of status (e.g., SharePoint) Consider every discipline that will be involved in the program Identify key stakeholders early and bring them together Identify Authoritative, Responsible, Consulted, and Informed (ARCI) stakeholders by task Number 7: Intensify up-front stakeholder engagement to build consensus on requirements Establish an acquisition schedule that promotes collaboration Integrate efforts through the ultimate acquisition authority

17. 16 World Congress 2013 16 Number 6: Example In 2013, DOE issued a Policy Flash that identified the importance of DRFPs in helping match the proposal and selection process to the requirement. Specifically, DOE noted that using DRFPs to guide selection of the proposal type can lead to significant cost savings and productivity enhancements; reduced proposal preparation and evaluation time; reduced need for solicitation amendments that disrupt timely completion; and better proposals, end products, and services. -DOE Policy Flash 2013-45, April 2013 If you don’t do it, what happens?  Too few bidders based on overly limiting reqs  Unqualified bidders based on overly loose reqs  Too time-consuming considering need  Unnecessarily paper-intensive  Too lengthy considering scope of requirement  Misalignment between prop & execution needs  No ability to meet key staff during bid process  Inappropriate re-use of existing materials  Interpretation that “it’s locked” for someone  Failure to consider oral presentation approach  Inability to select the best possible solution Select a proposal type and process that aligns with overarching objectives

18. 17 World Congress 2013 17 Do: Promote electronic submittal to greatest extent possible For small bids, keep it simple (complexity drives out bidders) Carefully review re-used artifacts – are they appropriate? Use market research to gain specific RFP recommendations Relate the proposal requirements to the work requirements Put DRFPs out early; include as much content as possible Number 6: Select a proposal type and process that meets overarching objectives Consider schedule in selecting response methods (e.g., orals) Directly ask for specific RFP recommendations from industry

19. 18 World Congress 2013 18 Number 5: Example In 2009, GAO sustained a protest claiming that an RFP from the Army Medical Research Acquisition Activity unreasonably required certification of proposed equipment by the Joint Interoperability Test Command at the time of proposal submittal. “The Army has simply not explained why the apparent purpose behind the certification requirement -- to ensure that a system be certified prior to the time it must be fielded -- requires that evidence of certification be provided as early in the process as the time at which quotations are submitted.” -GAO Protest Decision, SMARTnet,Inc., January 2009 If you don’t do it, what happens?  Many questions based on evaluation criteria  Industry self-elimination for the wrong reasons  Inappropriate focus on less significant aspects  Uneven, difficult-to-evaluate proposals  Unintended conflicts with other RFP elements  Misalignment with Acquisition Plan  Loss of confidence in fair and open process  Loss of focus on overarching objectives  Potential protest before proposal submittal  Poor props that inhibit selection of best option Use critical thinking skills when selecting evaluation criteria

20. 19 World Congress 2013 19 Do: Consider use of sample tasks to promote realism Provide adequate time for offerors to meet expectations Think about how you would choose if it was your money Limit the number of criteria to focus on key discriminators Make criteria specific to needs/objectives (limit re-use!) Define to greatest extent possible what makes an offer great Define key expectations (schedules, processes, templates) Make sure proposal instructions align with selected criteria Number 5: Use critical thinking skills when selecting evaluation criteria

21. 20 World Congress 2013 20 Number 4: Example In 2012, GAO partially sustained a protest claiming that an RFP from the Defense Intelligence Agency did not clearly state that use of uncleared personnel by the awardee demonstrated a lack of requirements understanding and should have been considered in a price realism evaluation. GAO also upheld the protestor’s claim that a cost realism evaluation must be conducted if it is defined in the evaluation criteria, even if the evaluation criteria do not specifically state by name that a cost realism evaluation will be conducted. -GAO Protest Decision, Science Applications International Corporation, November 2012 If you don’t do it, what happens?  Annoyed, contentious offerors  Perception that ambiguity is deliberate  High potential for unintended conflicts  Multiple questions focused on gaining clarity  Difficulty in tracking back to AP for answers  Many different but reasonable interpretations  Uneven, difficult-to-evaluate proposals  Increased need for lengthy discussions  Delays in completing acquisition process  High potential for successful protests Challenge ambiguity in all aspects of process

22. 21 World Congress 2013 21 Do: Assign at least one person to search for ambiguity issues Use a color team type approach to build in clarity in phases Document your thinking so it is defensible downstream If you find ambiguity, enlist fresh perspective to help fix it Remember: if it seems ambiguous to you, it is ambiguous Use industry to identify ambiguities and propose solutions Use an iterative approach that considers holistic alignment Write short, direct statements – less is often more (clear) Number 4: Challenge ambiguity in all aspects of process

23. 22 World Congress 2013 22 Number 3: Example In 2011, the U.S. Court of Federal Claims determined that GSA’s inclusion of a 15- year fixed-pricing schedule violated customary commercial practice and therefore also violated FAR 12.301(a)(2). The court found that GSA’s market research failed to show that the solicitation requirement for providing fixed prices at contract outset was consistent with commercial practice, and it therefore found the pricing requirement to be in violation of FAR 12.301(a)(2). -COFC Decision, CW Government Travel v. the United States, August 2011 If you don’t do it, what happens?  Use of outdated guidance/regulatory artifacts  Conflicts among accurate/inaccurate clauses  Improper application of clauses  Multiple questions from annoyed bidders  Loss of bidder confidence in process  Automatic necessity for RFP revision  Need (often) for re-review of entire solicitation  Delays in completing acquisition process  “Technicalities” with potential to derail process  High potential for successful protests Review all clauses for currency, accuracy, and applicability

24. 23 World Congress 2013 23 Do: Schedule a final clause review immediately before release Apply tools and references that focus on clause updates Use automation for generation, but review manually Provide resources/training on clause changes/updates Make an individual responsible for clause accuracy Document identified issues/inconsistencies and resolutions Number 3: Review all clauses for currency, accuracy, and applicability Conduct research and seek guidance if you identify an issue

25. 24 World Congress 2013 24 Number 2: Example In 2009, GAO sustained a protest claiming that TRICARE failed to apply its evaluation criteria properly in the areas of technical approach, past performance, and cost realism in selecting the initial awardee for a contract. In its decision, GAO noted that the instructions and evaluation criteria were complex and could be interpreted differently. The initial awardee followed all RFP instructions, but GAO held that TRICARE failed to apply its evaluation criteria appropriately. GAO upheld the protest. -GAO Decision, Health Net Federal Services, LLC, November 2009 If you don’t do it, what happens?  Confusion over what to cover/to what extent  Varied proposals that are hard to evaluate  High potential for conflicts  Multiple questions from annoyed bidders  “Everything is equally important” mentality  Loss of focus on key acquisition objectives  Delays in the proposal development timeline  Challenging evaluation process  Confusing, overly complicated proposals Make Section L (Instructions) and Section M (Evaluation Criteria) the same

26. 25 World Congress 2013 25 Do: Consider pass/fail for some elements of the proposal Match what you will evaluate with what you need in proposals Write Sections L and M concurrently with SOO/SOW Limit the information required to what will be evaluated Use past RFPs as references – What worked? What didn’t? Complete evaluation plan before RFP release; map L/M to it Consider offerors’ reactions to differing Sections L and M Number 2: Make Section L (Instructions) and Section M (Evaluation Criteria) the same If Sections L and M differ, document your rationale

27. 26 World Congress 2013 26 Number 1: Example In 2007, GAO sustained a protest claiming that the Army improperly evaluated proposals with regard to each of the solicitation’s evaluation factors (fill rate, experience, and transition). Specifically, the evaluations of offerors’ responses to the evaluation factors could not be aligned to specific criteria in the evaluation factors. It was therefore determined that there was inadequate support for the Army’s source selection determination. -GAO Decision, L-3 Communications Titan Corporation, March 2007 If you don’t do it, what happens?  No “red-threading” of changes to solicitation  No “fresh eyes” to find errors/inconsistencies  Missed errors/inconsistencies  Misalignment with AP and Evaluation Plan  Disappointment from requirement initiator  “Amend it later” mentality  Delays from questions/amendments  Confusion caused by multiple changes  Increased potential for successful protests Allow sufficient time to conduct iterative reviews and to respond to flaws they disclose

28. 27 World Congress 2013 27 Do: Consider process changes and training based on reviews Consider and correct ambiguities – even if they’re not “wrong” Establish a timeline that supports an iterative review cycle Remember: it is always better to fix it when you find it Use tools to promote rapid, effective, timely collaboration Create teams that combine historic and fresh perspectives Document, document, document – maintain your file Number 1: Allow sufficient time to conduct iterative reviews and respond to flaws they disclose Use review results as learning opportunities

29. Little things can lead to big successes 28 World Congress 2013 GovernmentIndustry Market research identifies key criteria Schedule/Acquisition Plan/DRFP reflect input Unambiguous, direct RFP includes best ideas Fewer amendments keep timeline intact Winner selected based on clear eval plan Debriefs clearly explain logical rationale Feedback sharpens focus on discriminators DRFP yields good suggestions from industry Well-prepared offerors have fewer questions Offerors submit clear, consistent proposals Program awarded! Protests avoided! Offerors receive notification/request debriefs 28

30. 29 World Congress 2013 29 Success begins with the basics Significant problems are generally avoidable or resolvable by focusing on simple-to-implement acquisition best practices

31. 30 World Congress 2013 30 Questions?