Contract Protest Diagnostic Tool - MITRE (In Partnership with ODNI)

I can see benefit to those mid range contracting personnel that lack years of experience.  It brings up lots of potential pitfalls that can get overlooked.

The more constructive feedback it receives, the better the tool.   Perhaps OFPP can take it over at some point to manage and maintain.

Thanks, formerfed.  Good comments.  You raise an important question regarding which entity should ultimately be responsible for this capability.  Perhaps OFPP is the best fit, or perhaps OFPP could determine that it should be housed elsewhere within the federal government. 

Since contracting professionals need to understand bid protest case law, yet agencies (to my knowledge) are not providing them with (expensive) subscriptions to Lexis and/or Westlaw, the need for this capability is rather obvious and long overdue. 

Some may argue that knowledge of bid protest case law is just the agency legal counsel’s job, but in my experience agency legal counsel with procurement experience are few and far between.  As such, they can’t possibly review every instance where knowledge of bid protest case law is required. 

I note that FAR 1.102(d) indicates that the “acquisition team” must be knowledgeable about (inter alia) bid protest case law … so they can exercise initiative and use sound business judgment without inadvertently falling into a protest pitfall.

@Oyster  I totally agree.  One major deficiency in contract training courses that people usually take is with contract law.  The most common courses spends way too much time on the basic stuff and doesn’t touch the really meaningful issues.

As a minimum I think contracting management should require mid level and senior specialists to take more intensive law courses as well as subscribe to Lexis and Westlaw as you suggested.  Sure it’s expensive but that’s a drop compared to the governments cost of handling protests and court decisions.

But with the subscriptions or even this tool, contracting personnel need to understand the basics first to apply cases to their work.

This is great and needs the attention of all Contracts Directors/Commanders.  No matter how limited its functionality may be in the early stages of development, it breaks the ice on a topic no one knows how to address (most don't even know it's a problem).  And with more attention on it, it will have to improve.

I agree with you Voyager.

When I was in the federal government, I was involved with a lot of protests over my long career.

After the protest was complete, we were required to send up a lengthy (and time consuming) report setting out the protest information in painful detail and explaining the “lessons learned” from the protest in a lengthy narrative.

Do you want to know how much guidance I saw being returned to the field from the policy office that required it?  Nada!  Zippo!  Nothing!

This program is doing something that the Government (writ large) should have been doing for years ... analyzing the protest issues, explaining them in plain English, providing one-click access to the actual cases that are being cited, and providing a way to input general information about real life contracts in order to allow the user to focus on the critical protest pitfalls to avoid.

Yes, due to access-of-information problems, not even the following basic principles are well understood - and not even by policy reviewers!

• "The evaluation of past performance, including an agency’s determination of the relevance and scope of a firm’s performance history to be considered, is a matter of agency discretion, which [GAO] will not find improper unless unreasonable, inconsistent with the solicitation criteria, or undocumented."  So don't ask me to remove documentation explaining my reasonableness and consistency!
• Contemporaneous documentation is king.  So don't tell me to make my written evaluations briefer just for brevity's sake!
• "Competitive prejudice is an essential element of a viable protest; where the protester fails to demonstrate that, but for the agency’s actions, it would have had a substantial chance of receiving the award, there is no basis for finding prejudice, and our Office will not sustain the protest."  So don't waste my time protest-proofing the written evaluations of an offeror with no reasonable competitive prejudice!

I only know there is an information problem because I have had to defend against policy reviewers' edits to my source selection documents, violating the above principles.  Maybe other COs that are directed to use this CPDT tool will find the confidence needed to similarly defend their source selection teams from bad edits.

Good afternoon.
 
My name is Pat Staresina and I am the MITRE project leader for the Contract Protest Diagnostic Tool (CPDT) ... and a long time WIFCON fan.  
 
I appreciate reading all of these comments regarding the Protest Tool.  This tool is designed to help the acquisition workforce navigate around pitfalls connected to the complex universe of protest decisions from GAO, the COFC and the CAFC.  Everyone talks about the "crawl, walk, run" approach to learning acquisition.  This tool is designed to help with the "run"  aspect by helping answer some of the hardest contracting questions.
 
As stated earlier, this tool is 100% free, and we are not selling anything.  (It is funded with federal research dollars as part of our FFRDC Charter).  This idea started as part of MITRE's IR&D program and was then further developed in partnership with the Naval Postgraduate School via CRADA.  It then went through a pilot program with contracting and legal communities in the Intel Community, which helped further refine it and is now being operationalized under contract with ODNI for the IC Acquisition Workforce.  We also presented it at this year’s NCMA World Congress.  

Our goal right now is to make this capability better known (and used) by the acquisition workforce - and get through the so called innovation "valley of death" and achieve significant user adoption.  It’s intended to be another tool in the contracting professionals tool kit.  The tool is a “living document” so to speak, and we are always seeking feedback from users to keep improving it.   
 
We recently added subcategory 8.5 on “self-scoring,” which has been a hot topic lately.  We are also about to add some additional extensive analysis on Federal Supply Schedule protest issues to help demystify protest grounds connected with FAR Subpart 8.4 contracting (i.e., exchanges, scope issues, responsibility determinations, mapping of labor categories, inclusion of open market items, etc.) .

I'm happy to answer any questions on the Protest Tool and am very interested to see more comments on this thread.  We're eager to spread the word!
 
If you or your agency are interested in a demo of the tool, please contact me at pstaresina@mitre.org.

@Patrick S Really good effort.  My only suggestions if you have funding and support is engage agencies outside the IC community as well as expand the cited decisions caseload.  So many decisions have a narrow focus and a literal reading can skew conclusions based upon an erroneous interpretation.  Maybe a drill down capability with expanded coverage for various subjects?

Thank you, formerfed.  Good comments!

We have been engaging with agencies outside of the IC community.  We have provided demos to multiple agencies, including DAU, the Army JAG School, DHS, and recently gave a deep dive presentation to a packed audience at NCMA World Congress.   That said, we are seeking to provide as many demos to government agencies as possible, so I again welcome any requests for a demonstration.  
 
Also, I hear you regarding increasing the number of case citations.  We are expanding the case citations on the tool on a continual basis.  That said, I recognize that some of the subcategories that were completed early on in this three-year process need to be expanded with more “Hot Spots” and corresponding case citations.  The “self-scoring” subcategory is the most recent addition to the tool, and you will notice that it is more robust than some of the other subcategories.  That subcategory is more indicative of the way that all of the subcategories will look in the end.  In other words, I agree with you, and we are doing our level best to increase the substantive information in the tool. 

The Hot Spots are designed to be the “drill down” capability that you mention.  We will be adding more and more as time goes on.   

Finally, please note that at the bottom of the protest tool’s homepage, there is a “feedback” button.  I know that a lot of people believe that a “feedback” response will just go into the ether.  That is not the case with this program.  We are earnestly seeking feedback from experienced acquisition professionals (such as yourself), and we take all feedback comments very seriously.  We really want to partner with the acquisition community on this project.  After all, it is designed to help those who are dealing with these issues, so feedback is critical to the success of this program.

Again, Thank you!

Patrick S said:

Also, I hear you regarding increasing the number of case citations.

Just dipped my big toe in.  First thought off the bat is I did not see a Sealed Bid choice.  Maybe missed it, if I did sorry, if not should it not be included? 

More decisions, I wonder?  I am use to attorney desk books issue by various entities, along with the GAO Red Book.   For these two they seem to concentrate on primary cases that are based on historical case history.  Too many might muddy the waters.   

I will keep looking as I can and pass along other thoughts if I have them.

Patrick S, I went through many of self-scoring subcategories.  That’s a big help in getting to the specifics.   I know it’s a lot of dialogue for the contract specialist/contracting officer to go through but that’s the only individual(s) on the team that’s likely aware of all the details.  Skimping might overlook important potential protest points.  The “hot spots” are good.

C Culham said:

Just dipped my big toe in.  First thought off the bat is I did not see a Sealed Bid choice.  Maybe missed it, if I did sorry, if not should it not be included? 

More decisions, I wonder?  I am use to attorney desk books issue by various entities, along with the GAO Red Book.   For these two they seem to concentrate on primary cases that are based on historical case history.  Too many might muddy the waters.   

I will keep looking as I can and pass along other thoughts if I have them.

Thank you C Culham!  You are correct.  The “Sealed Bid” subcategory is Subcategory 9.8, but it is grayed out with an explanation of “research ongoing.”  That is one of the subcategories that we are still working on.  There are several others that are likewise grayed out with the “research ongoing” explanation.  

I agree, the attorney desk books (particularly the Army JAG School’s Contract Attorneys Deskbook) is an outstanding resource.  I recognize that there are a lot of case citations in the protest tool, but we’re now breaking them all out under each “Hot Spot” within the subcategory.  We didn’t do that originally, which we realized led the user to spend too much time searching for the decisions that support the explanation in the Hot Spot.  If you click on the “snorkel” button and go to “self scoring,” (subcategory 8.5) you will see how we are trying to do this currently.  Its hard to get around the fact that there are so many decisions out there, and we are concerned about cutting out cases that the user may find useful.  Many of the decisions are cited in the text of the “Hot Spot,” so allowing the user to click on the drop down for the cases allows quick access to the actual decision cited in the Hot Spot.  

Yes, please keep looking and passing along your thoughts.  We truly appreciate it!  Thank you.

formerfed said:

Patrick S, I went through many of self-scoring subcategories.  That’s a big help in getting to the specifics.   I know it’s a lot of dialogue for the contract specialist/contracting officer to go through but that’s the only individual(s) on the team that’s likely aware of all the details.  Skimping might overlook important potential protest points.  The “hot spots” are good.

Thank you, formerfed.  I agree that there is a lot to read on the more complex protest topics.  Unfortunately there is a significant amount of nuance in these opinions, which makes it difficult to skimp on the narrative, as you noted.  We try to strike the right balance, but there is just so much info to consider.  To address this, we use hot spots, tags, and sub-topics to make it more readable.  It's definitely a challenge distilling hundreds of cases into a few paragraphs/pages.  We're trying to save the contracting officer/contracting specialist a considerable amount of time without chasing them away with overlong content.  I'm not sure if we're hitting the right balance, which is why this feedback is so important to us.  Thank you again for your comments and feedback!

Great comments by all.  Thank you. 

This is a timely discussion in light of the fact that GAO just released its FY23 bid protest report to Congress: B-158766, GAO Bid Protest Annual Report to Congress for Fiscal Year 2023.

Controlling for the number of CIO-SP4 protests filed, the total number of protests filed appears to be only a small uptick over last year.  (I recognize that the GAO doesn't give us the full picture because it obviously doesn't cover the protest filed at the COFC.)  

Returning to the MITRE protest tool, it seems like the underlying problem is as follows:  The three "main" pillars of the laws governing federal government contract formation are (1) statutory authorities, (2) regulatory authorities, and (3) decisional authorities.  Of those three, contracting professionals have easy access to the first two, but the third (decisional authorities) is murkier ... it consists of thousands of decisions across many years, with new ones churned out every day, the decisions are tied to specific facts, they are published by three separate forums, the COFC cases tend to be much lengthier and more difficult to digest quickly, agencies aren't giving most contracting officers/specialists access to Lexis or Westlaw, brown-bag lunch training sessions aren't sufficient, etc.  In other words, the first two (statutes and regs) are neatly tucked into a box, while the third is like a hoarder's attic, which is understandably confusing to many in the acquisition workforce.  

Meanwhile, the GAO "effectiveness rate" keeps going up indicating that agencies are either losing protests (or taking corrective action) more frequently.  

The protest tool seems to me to be a commonsense way get our collective hands around this aspect of the problem.

Good summary of protests

Wash Tech on protests

I wonder how the less experienced workforce of recent years impacts the protest effectiveness rating.