Page 30 - FCW, August 2019

Acquisition
“during evaluation, the government may trigger arbitrary failures to see how the solution responds.”
Bidders were also required to share their prototype as source code for work- ing software.
Tech demos: Sprints vs. hackathons
The tech demo was not the only compo- nent of the award process. Evaluating the quality of the company’s previous work was described in the RFQ as being more important than the results of the tech demo. Second-phase bidders were also required to submit a document of 20 pages or less presenting the com- pany’s “agile technical approach doc- ument, and evaluation of this was of equal importance to the tech demo.”
When asked whether the weight given the technical approach docu- ment departed from the “do, don’t say” philosophy of tech demos, one person familiar with the effort said: “I think this is reasonable if the evaluators are experienced enough to differenti- ate boilerplate text from showing real technical understanding in writing that is centered on the agency and on the scope of the BPA.”
For the tech demo, bidders were given two weeks to develop a prototype, and they could have anyone work on it they chose. The work was performed at the company’s site; bidders didn’t need to come to a government location.
Other tech demos have been four- or eight-hour efforts on one day at a government site. Each approach has upsides and downsides. The two-week effort can produce a solution that can be tested, which seldom happens with a one-day effort. It also more closely resembles an agile sprint rather than a hackathon. The pros of the shorter demo include being able to observe a team in action, which gives government decision-makers valuable information about team dynamics. In addition, in the one-day on-site events, firms cannot bring in talented team members who are unlikely to be involved in the work
after an award is made.
One concern about all tech demos
is the government’s ability to evaluate them, given the shortage of talented IT professionals. In this case, USDS employees did the evaluation, but this is not a scalable solution. There is obvi- ously less to evaluate in a one-day effort, and the cost of that approach is gener- ally lower because although there are airfare and hotel expenses, there is far less need for expensive developer or engineer time.
ADELE-QRI had one protest from a more traditional contractor, but it was resolved before going to adjudication. One could argue that nontraditional firms are less prone to protest and that the entry of more nontraditional firms into the marketplace could reduce the government’s exposure to protests.
New firms entering the federal mar- ketplace, including nontraditional ones, often ask how they can know whether it is worthwhile for them to bid on an agency’s project. I asked a number of winners how they determine when the government is interested in mak- ing awards to nontraditional contrac- tors. Two said their companies look for a reference in requests for proposals to the Digital Services Playbook that USDS and the Office of Federal Pro- curement Policy have developed, and two others said they look for refer- ences to “user-centered design,” “user research” or “minimum viable product.” Another winner said his company looks for “who in government is fun to work with. We work hard to see who’s doing fun things.”
The CMS source selection is an early sign that being a nontraditional contrac- tor is becoming a positive brand name. We may soon get to the point, as we are with agile development, where everyone starts claiming to be a non- traditional contractor. With regard to agile methodology, the Defense Inno- vation Board issued a document called “Detecting Agile BS” to help agencies spot false assertions. Companies’ claims to be nontraditional are unlikely to be convincing when a government contrac- tor has been around for decades. For newer firms, the government needs to look at evidence of their mission orien- tation and agile credentials. n
Steve Kelman is a professor of public management at Harvard University’s Kennedy School of Government and former administrator of the Office of Federal Procurement Policy. His blog can be found at fcw.com/thelectern.
For the tech demo, bidders were given two weeks to develop a prototype, and they could have anyone work on it they chose.
Nontraditional: ‘A positive brand name’
Tech demos have often been promoted as a way for young firms to bid on work without being familiar with the complex world of government proposals, and they are sometimes promoted as a way to streamline source selection because bidders do not need to prepare lengthy documents. Above all, they embody “do, don’t say.” In the MPSM competition, the first and third claims are true but not necessarily the second. Given the time spent on the design challenge and the time involved in preparing the written material for the procurement, it does not seem to have been less resource- intensive for bidders. (I asked two of the successful bidders about this, and one said it was more work than a tra- ditional proposal while the other said it was less work.)
Interestingly, there were no protests.
26 August 2019 FCW.COM