Comparative Evaluation vs. Confidence Ratings under GSA Schedule

On 1/13/2023 at 7:39 AM, far8.4 said:

please share the pros/cons/risks of using confidence ratings vs. comparative evaluation?

There are those that may think that my response is to general.  If so I hope that they provide a response with more detail.   

 I would suggest that the pro/con/risk is equal.  Where the rubber meets road in either process is in the ability to perform the evaluation in accord with the evaluation method chosen for the procurement and then document the result adequately to support the decision.

I offer the following to demonstrate my general view....

https://www.gao.gov/assets/b-415575.pdf

wifcon.com/pd8_4052.htm

As it appears that you are doing thorough research I offer another thought that I will leave with you to wrestle with to see if you can find the answer.  Where within FAR 8.4 does it provide that either method is to be used to evaluate quotes from GSA Schedule contractors?

On 1/13/2023 at 7:39 AM, far8.4 said:

For a procurement of technical services under a GSA schedule, can you please share the pros/cons/risks of using confidence ratings vs. comparative evaluation?

It's not a matter of confidence ratings versus comparative evaluations. Confidence ratings are often used to make comparative evaluations.

In a procurement with a many proposals and/or many evaluation factors, the assignment of adjectival ratings by evaluators can be helpful during the comparative evaluation by the selecting official.  

However, in a procurement with few proposals and few evaluation factors, it might be more efficient to skip the adjectival ratings and to simply compare each proposal to the others and to rank-order the proposals.

The real distinction is between (1) evaluation and selection by direct comparison of proposals and (2) evaluation and selection on the basis of pre-established rating standards.

To use a crude but simple example, suppose that you are conducting a competition for an item of equipment for which the only evaluation factors are mean-time-between-failure (MTBF), measured in hours, and price. Suppose further that your specification requires that in order to be considered acceptable the proposed product must meet a threshold value of at least 2,500 hours MTBF, but that the buyer's objective is a MTBF of 5,000 hours. You intend to make a tradeoff decision.

If you evaluate by direct comparison, you can rank the offers from best to worst based on their proposed MTBF. The price for each is stated in parentheses:

1. 4,205 ($415,000)
2. 3,997 ($406,000)
3. 3,229 ($397,000)
4. 2,914 ($345,000)

You do not assign pre-established ratings. Instead, you make a tradeoff analysis based on direct comparisons of proposal content in order to determine which is the best value. Is Offeror 1's 8 hour MTBF advantage worth $9,000? If not, is Offeror 2's 768 hour advantage over Offeror 3 worth $9,000, and so forth.

As ji20874 pointed out, that method is expedient if you don't have more than two or three evaluation factors and the factors themselves are simple𑁋no subfactors.

Instead of evaluation by direct comparisons, you could use a pre-established system of ordinal adjectival rating standards, such as:

1. 5,000 hrs - 4,500 hrs = Outstanding
2. 4,499 - 4,000 = Very Good
3. 3,999 - 3,750 = Good
4. 3,750 - 2,500 = Acceptable
5. < 2,500 = Unacceptable

The objections to adjectival (ordinal) rating systems are: (1) that two proposals can have the same rating, but one could be better or worse than the other, or very close to the next-higher rating, and (2) you cannot aggregate adjectives to come up with a summary rating. So if you have several factors, the best value tradeoff analysis can be complicated.

Numerical (interval or ratio) rating systems are generally held in distain by Federal agencies, except NASA and a very few others.

There are many methods of evaluation. For more on this, see Decision Analysis for Management Judgment, 5th ed., by Goodwin and Wright, and other good textbooks about decision analysis.