Highest Technically-Rated, Reasonable Price/Offer (HTRO) and Responsibility Determination

govt2310 said:

Highest Technically-Rated, Reasonable Price/Offer (HTRO), also called HTRFP or HTRRP, is an evaluation method where the offeror must self-score its own proposal.

Self-scoring is not an essential feature of the highest-technically-rated with a fair and reasonable price method of source selection.

govt2310 said:

My question is, how does HTRO work with responsibility-type evaluation factors and small businesses?  If they self-score themselves out of contention, so they are found unacceptable for a factor that would make them also "not responsible," does the agency still have to send it to SBA for COC?

An offeror cannot determine its own responsibility or nonresponsibility. Only a CO may do that.

If a CO determines an offeror to be nonresponsible based on the offeror's own self-scoring sheet, and if that is the only thing keeping the otherwise successful offeror from receiving an award, then the CO would have to refer the matter to the SBA in accordance with FAR 19.602.

I have not yet seen any bid protest in which a CO determined an offeror to be nonresponsible based on the offeror's own self-scoring sheet, but there may be one or more that I have missed.

If the RFP instructions require the offeror to self-score itself on a point system for the non-price factors (which, for this example, include traditional responsibility factors such as Past Performance), and the agency will only evaluate the proposals that are at or above a numerical cut-off point, could the agency argue that, an offeror that self-scores itself below that cut-off point, that does not mean that offeror's proposal is "technically unacceptable"?  So therefore, the agency has not found the offeror not capable to perform the requirement, which means the agency is not required to contact the SBA for a COC?  I'm saying that, in a HTRO RFP that involves a self-scoring sheet, that is not a situation where the evaluation factors are pass/fail.  Rather, the RFP involves a numerical point system.

If the solicitation specifies that the agency will only evaluate the proposals that are at or above a numerical cut-off point, then every offeror will know before submitting whether they are above or below the line, right?  If an offeror self-scores itself below the line, there seems no reason to even submit its proposal -- maybe it should protest before offers are due if it thinks the approach is unfair -- but if it does not protest and submits its protest, maybe it has forfeited its right to complain later when it is not selected?

These kind of hypotheticals are hard, and maybe even unfair -- no one here has read the text of the real or hypothetical solicitation.  Hard decisions need to be made based on facts.  We don't know your facts.

• Is this a single-award or a multiple-award scenario?
• Will the agency validate the scores of the highest-self scored offers, or simply rely on them as submitted?
• If the agency will validate the scores, and one of the highest self-scores is lowered by the agency's evaluation to where it is lower than another offer who was not initially evaluated, will that lower offer be brought into the evaluation pool?
• Is there a numerical point cut-off for each factor, or is there a single cut-off for total points?  If the latter, then maybe the agency will posit that it is not treating past performance on a pass-fail basis but rather is using it in its racking and stacking for selection -- In such a case, maybe referral to SBA for a small business unsuccessful offeror is inappropriate.

Part of the problem with COCs is that practitioners sometimes overlook fundamental correct principles.  The fundamental correct principle is that if we select a small business offeror as the winner based on the evaluation factors, and then find that offeror to be non-responsible, we involve the SBA's COC process -- but only for that one offeror --  we do not refer all the other unsuccessful small business offerors.  I hope the contracting officer who issued your solicitation is aware of this basic, fundamental correct principle and that he planned the evaluation and selection process accordingly.

I presume that the acquisition is for a multiple-award contract. Is that correct?

It appears that what you want to know is how to avoid having to refer offerors for a COC. Is that correct?

i think this is a multiple award procurement still in the draft pre-RFP release stage.  Wise strategy to think through everything before it goes final.

Well, if you're doing "highest-technically-rated with a fair and reasonable price" in a multiple-award acquisition, the best way to avoid the COC process is to not find any offeror to be nonresponsible, either on an actual or de facto pass/fail basis. Instead, say that a low score in comparison with other offerors may position the offeror outside the set of the highest-technically-rated and, therefore, not among the prospective awardees.

During source selection planning, you must establish a rational procedure for determining the dividing line between the set of highest-technically-rated offerors and everyone else. But I would not establish a predetermined (before receipt of proposals) cut-off score. That might be considered arbitrary, "mechanical", or noncompetitive.

Yes, assume this is a solicitation for a multiple-award contract.

Yes, what I want to know is how to structure the solicitation to avoid having to refer offerors to the SBA for a COC.

Hmm, so if the agency does not establish a pre-determined cut-off score, but instead, says that there will be a comparison done, that would avoid the need to do a COC?  Ok, thank you.  I will have to think about how to carefully phrase this comparison language.

• Is this a single-award or a multiple-award scenario? YES
• Will the agency validate the scores of the highest-self scored offers, or simply rely on them as submitted? AGENCY WILL VALIDATE THE SCORES
• If the agency will validate the scores, and one of the highest self-scores is lowered by the agency's evaluation to where it is lower than another offer who was not initially evaluated, will that lower offer be brought into the evaluation pool?  YES
• Is there a numerical point cut-off for each factor, or is there a single cut-off for total points?  If the latter, then maybe the agency will posit that it is not treating past performance on a pass-fail basis but rather is using it in its racking and stacking for selection -- In such a case, maybe referral to SBA for a small business unsuccessful offeror is inappropriate.  THERE IS A NUMERICAL CUT-OFF FOR THE TOTAL POINTS, NOT FOR EACH FACTOR

I realize that FAR 19.602 says that the CO only has to get a SBA COC for the "apparent awardee."  But if the RFP used "responsibility-type factors" in the evaluation factors, then doesn't the CO have a duty to get a SBA COC for each of the small business offerors that did not make it to the "apparent awardee" stage because the agency evaluated their proposals as unacceptable regarding a responsibility-type factor?  In essence, the agency has made a finding of non-responsibility for those offerors.

govt2310 said:

But if the RFP used "responsibility-type factors" in the evaluation factors, then doesn't the CO have a duty to get a SBA COC for each of the small business offerors that did not make it to the "apparent awardee" stage because the agency evaluated their proposals as unacceptable regarding a responsibility-type factor?

You're making this too hard.  You cannot submit every unsuccessful small business offeror for a COC -- if you did, you would be obliged to award a contract to each of them for whom the SBA grants a COC -- that would be an absurd outcome in a single-award scenario, and the rules are written for single award scenarios.  You are in a multiple-award situation, so you make it fit.

Correct principle: You refer the apparent awardee for a COC if, after evaluation and selection, you later disqualify that offeror for non-responsibility before award.  You do not refer all unsuccessful offerors, and you do not refer those who don't make it to the "apparent awardee" stage, as you styled it.

The COC process does not apply when an offeror is not selected based on a comparative and subjective analysis, even when a responsibility-type factor is used.  Don't think in terms of pass-fail for your factors -- instead, think in terms of good-better-best -- think in terms of selecting the best (or the highest technically rated) multiple awardees considering all the factors (or all of the non-price factors) together.

If you select the best offerors using some subjective and/or objective comparison of offers, you will avoid the problem you are imagining.  You create your own problem when you try to combine the technical evaluation and the responsibility review.  And you create your own problem when you try to have a mechanical selection process instead of a human and subjective comparison and selection.  An offeror with weak past performance might still be selected by your human SSA if it excels under the other factors.  Weak past performance is not disqualifying in a tradeoff.  There is a difference between an offeror being uncompetitive and that offeror being disqualified or ineligible -- you may be conflating these.

After you select your apparent awardees, you will still need to determine them to be responsible before award.  The responsibility review will be different than the evaluation/selection review.  The responsibility review occurs after the SSA has made the best-value selections.

I used more words than Vern did -- he was right, but it didn't work for you, so I explained more at length.  I hope this is helpful.

You may be carrying some baggage of incorrect principles, such as thinking that all small business offerors not making it to the "apparent awardee" stage must be referred to the SBA for a COC if a responsibility-type factor is used.  Shake off that baggage!  Let correct principles be your guide.

@govt2310 You have been provided good advice and thoughts.  I suggest as further help that you may want to visit the GAO  and CAFC cases found here as further reinforcement.  /legacy/reg/0f3db0e9dc7138a4.html   Suggest you not just read the synopsis but pull up the actual decisions and give them a read.

Then why did GAO rule the way it did in this case: https://www.gao.gov/products/b-413104.10?

The agency set a competitive range.  The protester was excluded.  There is no mention of being in line for award.  Why?

govt2310 said:

Then why did GAO rule the way it did in this case: https://www.gao.gov/products/b-413104.10?

The agency set a competitive range.  The protester was excluded.  There is no mention of being in line for award.  Why?

Why? Note that in the case the GAO cited the Small Business Act, 15 USC 637(b)(7)(A), which says:

Quote

[The SBA] is empowered, whenever it determines such action is necessary... (7)(A) To certify to Government procurement officers, and officers engaged in the sale and disposal of Federal property, with respect to all elements of responsibility, including, but not limited to, capability, competency, capacity, credit, integrity, perseverance, and tenacity, of any small business concern or group of such concerns to receive and perform a specific Government contract. A Government procurement officer or an officer engaged in the sale and disposal of Federal property may not, for any reason specified in the preceding sentence preclude any small business concern or group of such concerns from being awarded such contract without referring the matter for a final disposition to the Administration.

and the SBA's implementing regulation, 13 CFR 125.5(a)(2)(ii), which says:

Quote

(2) A contracting officer must refer a small business concern to SBA for a possible COC, even if the next apparent successful offeror is also a small business, when the contracting officer... (ii) Refuses to consider a small business concern for award of a contract or order after evaluating the concern's offer on a non-comparative basis (e.g., a pass/fail, go/no go, or acceptable/unacceptable) under one or more responsibility type evaluation factors (such as experience of the company or key personnel or past performance)....

Note that the GAO did not cite FAR 19.602-1, which makes no mention of that rule. The FAR coverage is inadequate. A CO must be aware of the SBA's rule. See Canfield, "Traditional Responsibility Factor Doctrine: From Interpretive Gloss To Regulatory Enshrinement," Procurement Lawyer, Summer 2021:

Quote

Traditional responsibility factor doctrine appears to have begun as an interpretive gloss on the intent of the Small Business Act's conferring of authority on the SBA to determine whether small businesses are competent to perform work for the federal government. Over the years, that gloss has evolved, but it has remained a fairly fixed principle in federal procurement law, culminating in its enshrinement in the SBA's regulations. There is a wealth of statutory, regulatory, and decisional law that outlines and refines the governing principles, and practitioners in all areas of federal procurement would do well to familiarize themselves with it. As the foregoing discussion demonstrates, however, there still remain questions to be answered about the doctrine's application.

FAR 19.602-1 makes it seem like COs must refer a small business for a COC only when they have determined the apparently successful offeror to be nonresponsible. But that is not true.

Also see FAR 15.101-2:

Quote

If the contracting officer elects to consider past performance as an evaluation factor, it shall be evaluated in accordance with 15.305. However, the comparative assessment in 15.305(a)(2)(i) does not apply. If the contracting officer determines that a small business’ past performance is not acceptable, the matter shall be referred to the Small Business Administration for a Certificate of Competency determination, in accordance with the procedures contained in subpart  19.6 and 15 U.S.C.637(b)(7)).

The problem is that the same is true of any responsibility-type evaluation factor.

govt2310 said:

Then why did GAO rule the way it did in this case: https://www.gao.gov/products/b-413104.10?

Because, in the GAO's own words,

Quote

In sum, the agency evaluated the small business offerors on an acceptable/unacceptable basis, as opposed to a comparative basis, with respect to relevant experience, a responsibility-type evaluation factor, and found CRS’s proposal unacceptable under that factor.  See 13 C.F.R. § 125.5(a)(2)(ii).  As such, rejection of CRS’s proposal based on its rating of unacceptable under the management approach, domain-specific capability in a health-related mission subfactor, without first referring the matter to the SBA for a COC determination, was improper.  Accordingly, we sustain the protest.

That agency evaluated on a pass-fail basis.  If you want to avoid that outcome for your procurement, then don't evaluate on a pass-fail basis.  Instead, consider using a comparative basis.  It's easy.  

From the decision--

Quote

Here, the solicitation provided that in phase 1 of the evaluation, proposals were to be evaluated on a go/no-go basis, i.e., non-comparative basis.  See RFP at M-3, M-4.  The solicitation further provided that if rated unacceptable under any factor in phase 1, the entire proposal would be rendered unacceptable and ineligible for award.

So, you should consider using a comparative basis (instead of a non-comparative basis) -- and instead of finding the proposal unacceptable and ineligible for award, simply observe that it does not provide the best value among the competitors.  Only the best go to the next phase.  The loser is not unacceptable or ineligible, it just isn't among the best.  Problem avoided.

ji20874 said:

So, you should consider using a comparative basis (instead of a non-comparative basis) -- and instead of finding the proposal unacceptable and ineligible for award, simply observe that it does not provide the best value among the competitors.  Only the best go to the next phase.  The loser is not unacceptable or ineligible, it just isn't among the best.  Problem avoided.

You cannot do "comparative" with FAR Part 15.  I know it can be done under FAR 16.505, but not FAR Part 15.

You most assuredly can do an evaluation and selection on a comparative basis under FAR Part 15.

ji20874 said:

You most assuredly can do an evaluation and selection on a comparative basis under FAR Part 15.

More specifically 15.101-2(b)(1)  - I know not on point to HTRO but a can do.

15.305(a)(2) and this quote which I believe others emphasized in this thread - "This comparative assessment of past performance information is separate from the responsibility determination required under subpart  9.1."

15.308!

govt2310 said:

You cannot do "comparative" with FAR Part 15.  I know it can be done under FAR 16.505, but not FAR Part 15.

@govt2310That comment was one comment too many. Your ignorance of the rules proves you to be unqualified for the acquisition under discussion. You are in over your head, and I am embarrassed for you. Do yourself a favor and leave the thread. It's dead anyway.

govt2310 said:

You cannot do "comparative" with FAR Part 15.  I know it can be done under FAR 16.505, but not FAR Part 15.

I believe this comment is based on common guidance from policy offices and other authoritative sources. The Procurement Innovation Lab (PIL) stops short of saying that comparative evaluations are prohibited and states that comparative evaluations are “Not recommended for use under FAR part 15.”

I believe the problem comes from the lack of a standardized definition and usage of the term. After all, it should be clear that comparisons are commonly used if not necessary in any competitive acquisition … FAR 15.308 and the DOD SSP expressly talks about comparative assessments and analyses.

So what do you mean when you say ‘comparative’?

Jamaal Valentine said:

So what do you mean when you say ‘comparative’?

I don't know what govt2310 meant, but it doesn't matter. FAR Part 15 does not prohibit comparative evaluation.

Here is one explanation of "comparative":

In general, an agency can evaluate offerors and their offers ("proposals") by comparison to a standard or by direct comparison to each other.

For the sake of illustration, suppose that an evaluation factor is "distance of the contractor's office from the work site," the lesser the distance the greater the value. Suppose further that you decide to rate offerors on a numerical scale of 1 to 100 points, 100 being best.

One way to rate offerors and offers would be to establish a distance/points rating standard (scale), compare each offeror and offer to the standard, assign points accordingly, and then rank them by comparing the ratings and underlying facts.

A different way would be to compare the offerors directly to each other, giving the closest offeror 100 points, and then giving every other offeror fewer points on a proportional basis. So, to use a simple example, assume that you receive offers from five offers, the closet being five miles away and that the farthest being 36 miles away. You could assign the five-mile offeror 100 points and the 36-mile offeror 1 point and score those in-between based on the linear function.

Once upon a time some agencies—most notably the Air Force—prohibited evaluation by direct comparison, because they feared that the best offeror and offer might not be particularly good in relation to the agency's requirement. When you're conducting a source selection based on proposed weapon system design concepts, one design concept might be better than all the others, but not particularly good in terms of desired weapon system performance. (Keep in mind that the process we call "source selection" was developed for use in the acquisition of aeronautical, nautical, and space systems.) So the Air Force developed the concept of evaluation standards and required the evaluation of each offeror by comparison to the standards, not to each other. See Air Force Regulation 70-15, Source Selection Policies and Procedures (1976), para. 3-4b:

Quote

The evaluation of proposals is conducted at the factor level under each item and compared against measurable objective standards, and not by comparing proposals to each other.

That Air Force policy was adopted by some other agencies, but prohibitions against evaluation by direct comparison were eventually abandoned by most as the use of source selection expanded to a wider range of acquisition types after the enactment of CICA in 1984.

The prohibition has never appeared in statute. It did not appear in the Armed Services Procurement Regulation, the Defense Acquisition Regulation, or the Federal Procurement Regulation, and it has never appeared in the FAR.

Evaluation of offerors and offers by direct comparison is a perfectly legal and reasonable method to use for many source selections, especially those for commercial products and services in which evaluation should be based on what is available in the market. Evaluation by comparison to standards is much more challenging, because the development of evaluation standards is challenging.

In any case, every source selection decision ultimately requires the direct comparison of offerors and their offers, whether they were evaluated by direct comparison or comparison to standards.

govt2310 said:

You cannot do "comparative" with FAR Part 15.  I know it can be done under FAR 16.505, but not FAR Part 15.

Whatever govt2310 meant by that, it is wrong.

To Jamaal Valentine: thank you.  Yes, I was just getting ready to post the link to the DHS PIL Boot Camp Workbook that says FAR 15 is not supposed to involve "comparative" evaluation.  You are correct, the DHS PIL does not say that one cannot use it with FAR 15, it just says it is "not recommended."  See pages 13 of 44, 22 of 44 in the Workbook at https://www.dhs.gov/publication/pil.

Jamaal Valentine said:

I believe this comment is based on common guidance from policy offices and other authoritative sources. The Procurement Innovation Lab (PIL) stops short of saying that comparative evaluations are prohibited and states that comparative evaluations are “Not recommended for use under FAR part 15.”

This comparison is inapt.  The PIL encourages a technique called comparative evaluation (or something like that) where a direct comparison of proposals without assigning adjectival ratings is appropriate -- that is the key to the PIL technique, no adjectival ratings, just head to head comparison.  What we're talking about in this thread is not that.  Yes, the PIL encourages adjectival ratings for formal Part 15 tradeoff source selections because that is so ingrained and the PIL doesn't want to rock that boat too much -- but for tradeoffs in simplified acquisitions or ordering situations where there are only a few factors and a few proposals, the PIL encourages a direct offeror-to-offeror comparison and ranking without adjectival ratings.

Every Part 15 tradeoff is done on a comparative basis -- usually with qualitative adjectival ratings rather than pass/fail ratings.  LPTA is pass/fail.

govt2310 said:

You are correct, the DHS PIL does not say that one cannot use it with FAR 15, it just says it is "not recommended."  See pages 13 of 44, 22 of 44 in the Workbook at https://www.dhs.gov/publication/pil.

The phrase "not recommended" does not appear on PILS Workbook page 13. However, that page seems to suggest that FAR Part 15 requires the use of ratings and precludes the use of evaluation by direct comparison. 

PILS Workbook page 22 says that the "comparative evaluation" technique is:

Quote

Ideal for task/delivery orders under FAR subpart 8.4 and § 16.505, but also for part 13 simplified acquisitions (incl. subpart 13.5 for commercial items up to $7 Million). Not recommended for use under FAR part 15.

The Workbook doesn't provide any rationale for saying that evaluation by direct comparison without ratings is not recommended under FAR part 15. So no critical thinker would concur with that "not recommended."

FAR Part 15 does not require the use of ratings or scores. The PILS workbook is wrong to suggest that it does. Ratings have never been required by procurement statute or by any procurement regulation in the Code of Federal Regulations. There are several good reasons to discard ratings. One reason is to eliminate protests about the assignment of ratings.

The purpose of ratings is to simplify and facilitate the aggregation of evaluation findings. (The best way to do that is with numerical ratings, not adjectives or colors.) See "Scoring or Rating in Source Selection: A Continuing Source of Confusion," The Nash & Cibinic Report, February 2006. See also "Source Selection Decisions: Ratings Should Not Be Used," The Nash & Cibinic Report, April 2018. If you don't use too many evaluation factors, there is no reason to assign ratings.

I have to suppose the PIL's "not recommended for use under FAR part 15" for the direct offeror-to-offeror comparison was not because the PIL believed it was prohibited -- rather, I suppose the PIL simply didn't want to engage in those arguments for formal selections and tactically chose to encourage the technique in simplified acquisitions (including commercial items up to the FAR 13.501 threshold) and ordering situations instead.  I believe the use of words was very carefully done.  The PIL Workbook does not require the use of ratings or scores for Part 15 procurements.  The PIL Workbook does not prohibit comparison of offers for FAR Part 15 procurements.

The Workbook's cover says, "Everything here is intended to be helpful; nothing here is intended as policy."  The cover also says, "This document is a training aid to support the PIL Boot Camp all-day experience for the DHS acquisition community. This is not a stand-alone document."

I appreciate the on-going efforts of the PIL to loosen things up a little and to throw off excess baggage.

@ji20874  I understand your support for the PIL, but the PIL Workbook could be a lot better. While they don't say that you cannot evaluate by direct comparison under Part 15, you have seen that unsophisticated readers can only wonder about that given the way they put things.

If you're going to lead, lead by clear direction.

ji20874 said:

This comparison is inapt.

Do you mean the example I provided of the assertion/guidance found in the PIL Workbook? Why is it inapt? The Workbook’s technique doesn’t qualify or provide rationale for its assertion. Moreover, nothing in the FAR system—that I’m aware of—requires ratings so the technique is not a novel idea for limited application where the limitation would clearly make sense. Comparative evaluations are not limited to direct comparisons without ratings.

ji20874 said:

What we're talking about in this thread is not that.

I’m not sure what you have limited your discussion to but, comparisons were first mentioned in the eighth post and the original poster mentioned comparison language in the ninth post. The comparisons mentioned related to qualitative evaluation of responsibility type factors (e.g., not pass/fail or unacceptable/acceptable). I assumed, seemingly correctly, that the original poster was under the belief that comparative evaluation [of factors] weren’t allowed under FAR part 15. The original poster’s reference to the PIL Workbook is not surprising even if you believe it’s misapplied.

Vern Edwards said:

you have seen that unsophisticated readers can only wonder about that given the way they put things.

I wonder who the PIL’s target audience is within the DHS? I know a lot of people outside of DHS reference their work when making arguments for or against a practice.

The original poster erred in asserting that evaluations and selections under FAR Part 15 cannot be made on a comparative basis.  Up to that point in this thread, a comparative basis was used to contrast with a pass-fail basis. The original poster's pointing to the PIL Workbook to save his or her assertion fails, as the comparative technique described therein is not identical to the comparative basis discussion throughout this thread.  Yes, the word comparative exists in both but the contexts are different.

It is true that the PIL Workbook does not recommend the idea of head-to-head-comparisons-without-adjectival-ratings for Part 15 procurements, but "not recommended" does not mean "prohibited." 

I am glad people are reading the PIL Workbook.  If it is generating professional dialogue, that is good.  If it is helping people shake off baggage, that is good.  I'm not aware that there really is anything else out there doing anything helpful.  But the PIL Workbook does not present itself as policy or as a textbook; rather, it presents itself as a hopefully helpful tool for those who are willing to help make the acquisition process more responsive.  For those practitioners, I think the PIL Workbook is one of the best things out there.  If there are other useful aids in other places, I hope our readers will make mention of them here.

I hope this discussion has been helpful to the original poster.  There is good advice in this thread.

ji20874 said:

I am glad people are reading the PIL Workbook... I'm not aware that there really is anything else out there doing anything helpful.

@ji20874If you are not aware of "anything else" doing anything helpful, maybe you're not looking in the right places.

I will see if I can arrange for Bob to be able to post some helpful things.

ji20874 said:

I am glad people are reading the PIL Workbook.  If it is generating professional dialogue, that is good.  If it is helping people shake off baggage, that is good.

Agreed.

Another maybe unnoticed benefit of the PIL workbook is it shows doing some action is alright.  It’s sometimes tough for an 1102 contract specialist to try something different and get approval from policy people and contract law attorneys that review actions without proof that it’s an acceptable practice.

formerfed said:

Another maybe unnoticed benefit of the PIL workbook is it shows doing some action is alright.  It’s sometimes tough for an 1102 contract specialist to try something different and get approval from policy people and contract law attorneys that review actions without proof that it’s an acceptable practice.

The PIL Workbook discussion of comparative evaluation is not proof of anything. That's not to say that it's wrong about anything, but that it is short on explanation. That's my main complaint about it. It is incomplete. Look at page 22, which presents "Innovation Technique 5, Comparative Evaluation."

WHY is comparative evaluation "Not recommended for use under FAR Part 15"? That's a natural question, which the Workbook should answer. It should explain.

In what way does comparative evaluation provide "ultimate subjectivity/flexibility"? Explain. Is subjectivity better than objectivity?

More fundamentally, what the heck is comparative evaluation and how does it work? The workbook says it's "comparing offers to each  other." Don't we always compare offers to each other when making a source selection decision? Isn't that the very essence of competition? What I think they mean is compare offerors to each other, not against a standard or rating scale. They could have explained that in a few sentences. Maybe they could have used some of the space they gave to the two big and not especially useful text boxes that take up three fourths of the page.

It refers readers to the "GAO Guide," but when you get there the only reference to comparative evaluation is to a single protest decision about a task order competition. That 14-page decision does describe direct comparative evaluation in two short paragraphs on page 3. But some explanation in the PIL Workbook about what to look for in that decision and what to see would have helped. (The protest did not challenge evaluation by direct comparison.)

Ideation without explanation is practically useless, unless the people you are communicating with already know and understand the underlying concepts and principles. 

The Procurement Innovation Lab is a good thing, and my objective in writing this is not to trash them. But they need to do better for a workforce whose leadership has failed to provide them with essential professional education and training and essential reference tools of their trade. The PIL Workbook is interesting and pretty, but it's not enough.

Okay, so Vern doesn't like the PIL Workbook.  He seems to wants a exhaustive comprehensive textbook.  Okay.  The Workbook's cover says, "Everything here is intended to be helpful; nothing here is intended as policy."  The cover also says, "This document is a training aid to support the PIL Boot Camp all-day experience for the DHS acquisition community. This is not a stand-alone document."  One errs when judging it as a stand-alone policy document, and outside the Boot Camp setting.  But still, if it is prompting professional dialogue, that is good. 

I am glad that many DHS practitioners (contracting officers and procurement attorneys) and maybe practitioners in other agencies, are successfully using comparative evaluations without assigning adjectival ratings in simplified acquisitions and ordering situations.  That, I think, was the PIL's goal, and that is an honorable goal.  It seems the PIL Workbook is fulfilling its intended purpose.  It would be unfair to expect the PIL Workbook to fulfill other purposes beyond its intended purpose.

Vern hopes to get some other helpful things posted here -- that's good.  I have never heard of anyone other than the PIL providing help to practitioners in doing comparative evaluations without assigning adjectival ratings, but I will be gratified to learn that others are also teaching this approach.

ji20874 said:

I have never heard of anyone other than the PIL providing help to practitioners in doing comparative evaluations without assigning adjectival ratings, but I will be gratified to learn that others are also teaching this approach.

I didn't know whether to laugh or cry when I read that comment.

PIL did not provide "help" practitioners in doing comparative evaluations without assigning adjectival ratings. They just told them about the possibility of doing it in certain kinds of acquisitions. They did not describe the process or the pros and cons. Aside from one GAO protest decision cited on another page, they did not refer practitioners to a source of practical information, like Hammond, Keeney, and Raiffa, Smart Choices: A Practical Guide to Making Better Decisions, Harvard Business Review Press (1999), Chapter 5, or Goodwin and Wright, Decision Analysis for Management Judgment, 5th ed., Wiley (2014), p. 40, "Direct rating." Neither book teaches the use of adjectival ratings.

In 1993, Ralph Nash and John Cibinic wrote this, in Competitive Negotiation: The Source Selection Process, p. 350, The George Washington University.

Quote

Ranking [by direct comparison, without scoring] runs the risk of not identifying the key deficiencies in the proposals unless it is accompanied by detailed narratives. Nonetheless, the direct ranking of proposals has been upheld, Development Assocs., Inc., Comp. Gen. Dec. B-205380, 82-2 CPD ¶ 37. The Comptroller has even stated that "ranking proposals may be a more direct and meaningful method" than numerical scoring, Maximus, Comp. Gen. Dec. B-195806, 81-1 CPD ¶ 285.

See _Development Associates, In_c., B-205380, July 12, 1982:

Quote

The protester also contests the failure to use scoring or rating procedures in the final evaluation. The RFP did not indicate, however, that proposals would be evaluated on the basis of numerical point scores. Moreover, although a point scoring system may be useful as a guide to intelligent decision-making, numerical scores merely reflect the disparate judgments of the evaluators and, as such, do not transform the technical evaluation into a more objective process. Ranking proposals directly, that is, without scores, may be a more meaningful method if ranking permits the contracting activity to gain a clearer understanding of the relative merits of the proposals. See MAXIMUS, B–195806, April 15, 1981, 81–1 CPD 285. Thus, we find nothing inherently improper with ranking proposals without the aid of numerical scores.

(Use of adjectival ratings was first mandated by the Air Force in the early 1980s because there were too many foul-ups using numerical scores, and because they thought it would foster more subjectivity.)

According to Westlaw I have written something like 50 published articles in which I discussed scoring or rating. Fifteen years ago, In 2006, I wrote an article entitled, Scoring or Rating in Source Selection: A Continuing Source of Confusion, which Bob will post at Wifcon as soon as the publisher grants permission. In that article I wrote this:

Quote

It is essential that everyone involved in contractor selection understands the distinction between evaluating offerors and their offers and scoring or rating them. Evaluation is the process of determining the relative value of a thing. Scoring or rating is the use of words or symbols to express evaluation findings in simple terms. The Federal Acquisition Regulation requires agencies to evaluate offerors and their offers, but it does not require that they score or rate them. FAR 15.305(a) is misleading in saying that evaluations “may be conducted using any rating method or combination of methods, including color or adjective ratings, numerical weights, and ordinal rankings,” because rating, in the sense of the assignment of adjectives or symbols, is not a method of conducting an evaluation, it is a method of expressing the results of an evaluation.

I wrote this in 2009:

Quote

If an agency is considering only one or two factors, say, engine thrust and price then it does not make much sense to bother with converting evaluation findings to scores or ratings. The evaluators can simply report their findings to the decisionmaker, who should find it easy to make nonprice/price tradeoffs and decide which competitor offers the best value. But if there are multiple evaluation factors, and if each factor is measured on a different scale, then the SSA will face a rather complex tradeoff analysis problem. The greater the number of evaluation factors and the greater the number of offerors the more complex the problem. A preliminary and conditional ranking based on scores or ratings might help an SSA get oriented. But if an agency is going to use scores or ratings in that way, it should want to develop a rational scheme. In order to be valid and useful, the scores or ratings must preserve the differences among competitors and make them discernible.

In a lengthy analysis of a protest against agency proposal scoring, Prof. Nash wrote this in 2010:

Quote

This takes us back to our premise in 23 N&CR ¶ 61--that scoring systems tend to do more harm than good. Wackenhut is the perfect example because there the agency lost months litigating the accuracy of scoring and further damaging the procurement process by inducing the court to demand an excessive amount of documentation supporting the accuracy of the scoring system. Dropping scoring from the process thus has two benefits. First, it forces source selection officials to base their decision on the real substantive data resulting from the evaluation process. Second, it keeps losing offerors from filing protests addressing the accuracy of the scoring. That's a double benefit that seems compelling to us.

PIL did not invent the wheel. There is nothing new about evaluation by direct comparison without using ratings or scores.

If you have never heard, then maybe you should review your reading program.

BTW, I didn't say I don't like the PIL Workbook. I said I'm not satisfied with it. Is it bad to want more?

That's fair, Vern.

The PIL modus operandi is to assign a coach to an interested acquisition team that wants to try a new technique with a coach's assistance -- ideally, the coach will be up to date on the latest state of the practice for a particular technique.  That information is relayed orally.

@ji20874 That's a good idea!

The PIL has done and is doing good things. But written explanatory material is of enormous value, especially when widely disseminated.

The power of ideation to drive innovation when combined with clear and targeted explanation is virtually limitless.

One of the greatest examples of the 20th Century. Four pages.

https://psychology.okstate.edu/faculty/jgrice/psyc3214/Stevens_FourScales_1946.pdf

Every one of my source selection students gets a copy.