Interpreting what "Neutral" means in the eyes of GAO- Past Performance-

"In the case of an offeror without a record of relevant past performance or for whom information on past performance is not available, the offeror may not be evaluated favorably or unfavorably on past performance."

Read FAR 15.305(a)(2)(iv) again, slowly: "without a record of relevant past performance."

On what do you base your stated beliefs? You mentioned "Commonsense" (sic) but we have the plain language.

In the MDA case GAO states

Quote

"We note initially that, since the record confirms that the prior contracts MDA listed to establish its past performance were not similar to the current requirement, MDA Proposal, Tab 2, at 1-3, the agency reasonably determined that MDA's prior contracts did not provide a basis for assessing the firm's past performance. However, it appears this aspect of MDA's past performance warranted a neutral, rather than an unacceptable, rating; under FAR Sec. 15.305(a)(2)(iv), where an offeror does not have a record of relevant past performance, the offeror may not be evaluated either favorably or unfavorably. On the other hand, even if this is the case, and even if we also agreed with MDA that the agency should have clarified its experience or obtained additional information from its own records, as discussed below, GSA reasonably rated the proposal unacceptable based on MDA's failure to establish adequate experience and past performance for its subcontractors and key employees............

To the extent MDA contends that GSA was required to assign a neutral rating to its proposal based on the absence of information relating to its key subcontractors, we disagree. Although FAR Sec. 15.305(a)(2)(iv) requires an agency to assign a neutral rating where past performance information is not "available," here, the protester's proposal represented that its proposed subcontractors are engaged in projects that would illustrate their performance capability. The information thus was available, but MDA chose not to present the information in its proposal, in direct contravention of the terms of the RFP. In our view, an offeror cannot simply choose to withhold past performance information--and thereby obtain a neutral rating--where the solicitation expressly requires that the information be furnished, and where the information is readily available to the offeror."

Based on what I have read,  it's the offeror's burden to submit an adequately written proposal for the agency to evaluate and further noted that an agency may reject a proposal for information deficiencies that prevent the agency from fully evaluating the proposal. Based on all my reading, there is a difference between having no relevant past performance (FAR 15.305(a)(2)) (therefore the offeror submitted no past performance) vs. submitting a proposal that has past performance that is not even relevant in size scope or complexity.

On ‎4‎/‎26‎/‎2016 at 2:37 PM, FARmer said:

I believe that if an offeror submits past performance to a RFP and that past performance submission was evaluated and determined NOT relevant in size, scope, and complexity, then the past performance would be something less than a default to a "neutral".

I agree with Pepe.  The FAR is clear.  If past performance is not relevant or is not available, they can not be evaluated favorably or unfavorably.  In your question, the determination was already made that the past performance was not relevant, so there does not appear to be a debate regarding whether it is relevant or not.  You may want to look at these other resources on Wifcon if you haven't already.

Blog entry from Emptor Cautus:

Protest decisions and court opinions:

   /legacy/reg/e26de99505114521.html

FARmer said:

Based on what I have read,  it's the offeror's burden to submit an adequately written proposal for the agency to evaluate and further noted that an agency may reject a proposal for information deficiencies that prevent the agency from fully evaluating the proposal. Based on all my reading, there is a difference between having no relevant past performance (FAR 15.305(a)(2)) (therefore the offeror submitted no past performance) vs. submitting a proposal that has past performance that is not even relevant in size scope or complexity.

PepeTheFrog does not know where to start, and is hop, hop, hopping away from this discussion.

FARmer said:

Based on what I have read,  it's the offeror's burden to submit an adequately written proposal for the agency to evaluate and further noted that an agency may reject a proposal for information deficiencies that prevent the agency from fully evaluating the proposal. Based on all my reading, there is a difference between having no relevant past performance (FAR 15.305(a)(2)) (therefore the offeror submitted no past performance) vs. submitting a proposal that has past performance that is not even relevant in size scope or complexity.

In the case you cited the issue was that they did have relevant past performance which they withheld in an attempt to gain a "neutral" rating.  This is different than having no past performance or past performance that was not relevant.  Your initial question did not state they had relevant past performance that they withheld.

Believe it or not, the firm that only cut grass, but wanted to build an aircraft carrier would not receive an unfavorable rating with regard to a past performance element.  However, I suspect they would receive an unacceptable rating on an experience or technical approach factor.

DA Defense Logistics HQ  File: B-411153.3  Date: December 2, 2015 (http://www.gao.gov/products/D12323#mt=e-report) _(_This case lead me to the MDA case). 

An excerpt form the case, GAO states: _"_Furthermore, we find no basis to conclude that the Army violated FAR § 15.305(a)(2)(iv) by “unfavorably” evaluating the protester based on its lack of relevant past performance as a prime contractor.  The agency did not “unfavorably” evaluate the protester for its lack of relevant past performance as a prime contractor; rather, the Army reasonably determined that the protester had not demonstrated sufficiently relevant past performance to warrant the highest possible confidence assessment under the RFP’s qualitative evaluation criteria. "

Todd- Thank you for the link to Emptor's blog entry link I am reading it now.

You can read years of GAO's views on neutral ratings at the protest page here:  FAR 15.305 (a)(2)(iv): Past performance - Neutral rating.

FARmer,

In the case you cited (and in the text you quoted above) the agency determined the offeror did have relevant past performance (the GAO agreed), it just was not sufficiently relevant (experience they had was not as a prime).  The requirement to not rate favorable or unfavorable applies when past performance does not exist or is not relevant at all. 

"The Army found that DA Defense submitted relevant past performance demonstrating its prior experience as a subcontractor performing the performance work statement’s maintenance, supply, and transportation requirements. The agency, however, found that DA Defense did not demonstrate relevant past performance with the overarching program management tasks that the protester, as the prime, would have to fulfill on the resulting order."

Did the solicitation distinguish between experience and past performance? Sometimes the terms are used interchangeably and sometimes they have specific meanings.

Experience - opportunity to learn by doing

Past Performance - Quality assessment of recent, relevant experience

First, "neutral rule" is an obsolete term.

Second, the easiest way to think about the past performance rule at FAR 15.305(a)(2)(iv) is this: You cannot evaluate something that does not exist or that you have no information about.

Third, evaluate experience and past performance separately.

• Relevant experience is the number of chances that a firm has had in the past to do the kind of work that they will be required to do under the prospective contract and to learn about the kinds of problems that will be involved in performance and how best to handle them. Evaluation of relevant experience does not entail any assessment of how well the firm performed.
• There are two facets to experience: depth and breadth. Depth is reflected in the number of times a firm has done the work. Breadth is reflected in the extent of the work that it has done. If a prospective contract entails performance of five distinct tasks, the more times a firm has performed a particular task, the deeper its experience with that task. The more of the five tasks the contractor has performed, the broader its experience with those tasks. See the matrix below. (Crude, but the best I could do here.) You can use a number of different rating schemes with that matrix and make it as complicated as you like.

BREADTH

TASKS

DEPTH

REFERENCES

1

2

3

4

5

A

✔

✔

✔

B

✔

✔

✔

✔

C

✔

✔

D

✔

✔

E

✔

✔

✔

✔

✔

• Relevant past performance is the record of how well a firm performed doing the kind of work that it will have to do under the prospective contract and that confronted the firm with the kinds of problems that it is likely to encounter in the future. Evaluation of relevant past perfomance does not entail any assessment of how many times the firm has done that kind of work.

Always assign more importance to experience than to past performance, because experience is easier to verify than the quality of performance. You can't evaluate past performance that doesn't exist or for which you have no information, but you can evaluate the lack of verifiable experience.

Ok, so Past Performance, FAR 15.305(a)(2)(iv) has been a hot topic. I truly understand that. So in laymans terms and using Vern's example above as well as what Todd Davis cited right before Vern's post-

Offeror A submitted past performance for the requirement. Offeror A's past performance specifically states that they have similar work that is relevant (using Vern's definition above) in all facets. When evaluated, the evaluators determine that the past performance submission is only relevant to Task 1, and is completely not relevant to any of the other tasks. As a matter of fact, base on the proposal, if Offeror A is considering their past performance relevant to Tasks 2, 3, 4, and 5 then they clearly don't understand the requirement they are proposing to.  To the evaluators, this past performance submission was unacceptable...... It wasn't that there was no information to provided to access, it was just completely irrelevant to the requirements of the RFP. 

Now Offeror B submitted NO past performance. They clearly stated in their proposal that they have no past performance and therefore Offeror B was not rated favorably or unfavorably.... they received a neutral per the DoD source selection guide.

So based on everything above, are you telling me that Offeror A should be rated the same as Offeror B?

As quoted from my earlier post in regards to the MDA case, GAO stated:

Quote

In our view, an offeror cannot simply choose to withhold past performance information--and thereby obtain a neutral rating--where the solicitation expressly requires that the information be furnished, and where the information is readily available to the offeror.

  Is it then unreasonable to state that an offeror cannot simply choose to submit irrelevant past performance in order to obtain a neutral rating?

So I am going to post again because after reading many more cases, it appears that in my example above, both Offerors A and B should/would be treated the same........... So I guess what I thought is common sense isn't necessarily the case.

FARmer:

The evaluation of past performance is supposed to be an evaluation of how well an offeror performed in the past. An RFP can limit consideration of past performance to "relevant" past performance, which means that the agency is interested only in how well offerors performed when doing "relevant" work. If an offeror describes performance history that the agency determines is not relevant, then there is nothing for it to evaluate. If there is nothing for it to evaluate, then the agency cannot say anything good or bad about nothing. But if the agency finds out on its own that there really was something, and that the offeror didn't report it, then the agency can evaluate the something that it found.

It really is quite simple.

Lesson? Don't rely on offerors to tell you about things that they didn't do well. Do some checking on your own.

Vern- Yes indeed. I realized and will learn from my mistake.

Thank you everyone for contributing!

Vern, et al. Thanks once again for clearly differentiating between "relevant experience" and "past performance". This is one of my pet peeves.

A construction firm can excel at building houses, which is light, residential construction. But if the instant solicitation is for construction of an aircraft hangar (medium to heavy industrial), the firm's experience is not really relevant.

It's past performance record, if using the DOD's evaluation criteria for assigning a "confidence rating" would be of low relevance of those type projects to the instant project, thus would not lead one to assign the firm a high level of confidence rating.

I believe that certain organizations have contributed to the confusion by devising evaluation schemes that don't clearly differentiate between PP and level of experience, such as the Air Force "Past Performance- Price Trade-off" scheme, at least the one they were using a few years ago.  That was one of my pet peeves.

Indeed, I couldn't find in the Wifcon Protest pages, a separate category for experience, either.  I may have missed it.

I believe all past performance is relevant -- but some of it is more relevant than others.

For example, the grass-cutting company has past performance in managing work and managing schedules and being committed to success.  Its past performance might be of little relevance in an aircraft carrier acquisition, but little relevance is not irrelevant.  If I were drafting an aircraft carrier solicitation, and if I were evaluating past performance, I would want to include text in the solicitation inviting the offeror to describe its past performance without limiting relevance to previous aircraft carrier contracts in the last year.  Then, I could do a comparative assessment of past performance information among all the offerors and identify those offerors in whom I have the highest confidence of future success, based on past performance.  In such a comparative assessment based on past performance, it is likely that the grass-cutting company's past performance (maybe very good, but of little relevance) would give me less confidence than a firm with more relevant past performance.

I think we err when we treat a past performance evaluation (FAR 15.305( a )( 2 )) like a technical evaluation (FAR 15.305( a )( 3 )).  The past performance is supposed to be a comparative assessment, in contrast to the technical evaluation which is supposed to assess each offeror's ability to accomplish the technical requirements (the word "comparative" is not used in the FAR text on technical evaluations).  We religiously make sure that our technical evaluations are not comparative, and that is good; but in our zeal we also make sure that our past performance evaluations are not comparative, and we err in doing so.  Even so, the source selection authority will also make a comparative assessment of proposals as part of the decision-making process (FAR 15.308).

If we did our past performance evaluations as comparative assessments, as the FAR tells us we should do, we would avoid some of the problems we seem to create for ourselves.

The biggest problem is experience, not past performance. The first mistake is measuring experience in terms of years. Years are not a valid unit of experience, because the passage of time is not what educates a firm. The second problem is that people have not thought deeply enough about the nature and value of experience. What educates firms is the encounter with specific work. More specifically, the encounter with problems that must be solved in order to complete the work as specified, on time, and within budget, if possible. An experience is an instance of such an encounter, and the proper measure of experience is the number of such encounters.

When contracting, the evaluation team must think about the work that they want done and then think clearly about how that work will challenge a contractor. What problems will have to be solved in order to finish satisfactorily? By "problem" I don't just mean trouble, but also those circumstances, conditions, and situations that must be dealt with through the effective application of know-how or development of new know-how in order to make progress. Every performance of particular task is unique to some degree due to variations in customer, workforce, environment, initial conditions, events, etc. The more times a firm performs a particular task the more opportunities it will have to encounter those variations and the more it will learn about what works and what doesn't.

Past performance is nothing more than an assessment of how well the contractor has done that work in the past. Once you know what experience is relevant to the work to be done, you know what past performance is relevant.

The main problems associated with experience and past performance is that information about them is based on memories and subject to distortion and exaggeration. Past performance has an additional problem -- evaluation depends more heavily on opinion and is more vulnerable to bias, pro and con.

ji20874 said:

The past performance is supposed to be a comparative assessment, in contrast to the technical evaluation which is supposed to assess each offeror's ability to accomplish the technical requirements (the word "comparative" is not used in the FAR text on technical evaluations).  We religiously make sure that our technical evaluations are not comparative, and that is good; but in our zeal we also make sure that our past performance evaluations are not comparative, and we err in doing so.  Even so, the source selection authority will also make a comparative assessment of proposals as part of the decision-making process (FAR 15.308).

That is a misdescription of evaluation. All evaluations--except those conducted on a pass/fail, acceptable/unacceptable basis--entail a two-step process. In the first step you measure or assess each proposal on the basis of the applicable evaluation factors and factor standards. That's what you do when you determine a proposal to be exceptional, acceptable, etc., with respect to a factor. In the second step you compare each proposal to the others on the basis of the results of the first step in order to rank it based on its relative value. That is true for both the technical facet of proposals and for past performance when past performance is used as an evaluation factor as opposed to a responsibility consideration. In short, we do compare technical proposals, but only after we have evaluated them on the basis of factor standards.

Vern,

Are your two steps the evaluation described in FAR 15.305 (done by the evaluators) and the selection decision in FAR 15.308 (done by the source selection authority)?  I am keeping them purposefully separate.  It is not a universal practice for evaluators to do a comparative assessment for the technical evaluation -- FAR 15.305**_( a )( 3 )** doesn't call for a comparative assessment or a qualitative ranking for the technical evaluation.  However, and in contrast, FAR 15.305_**( a )( 2 )** does call for a comparative assessment just for the past performance evaluation.  After all of this is done by the evaluators, the selecting official then does a comparative assessment of the proposals under FAR 15.308.

(edits made)

My two steps are:

1. Assigning value to each proposal based on factor standards.
2. Comparing and ranking proposals based on the results of Step 1.

I consider both steps to be part of what FAR describes in 15.305, "Proposal evaluation." Under DOD procedures, Step 1 is performed by the SSET and Step 2  is performed by the SSAC. The results are then given to the SSA, who makes the source selection.

What FAR are you using? In the one at acquisition.gov, FAR 15.305(c) does not describe technical evaluation. It states restrictions on the use of support contractor personnel. FAR 15.305(a)(3), which is entitled "Technical evaluation," states only what must be included in the source selection records. It does not describe any process. But 15.305(a)(3)(ii) expressly states that the records of technical evaluation must include some form of ranking, which is produced in Step 2.

I'll correct my FAR citations...

FAR 15.305( a )( 3 ) doesn't require a comparative assessment or a qua_litative ranking for the technical evaluation.  It allows for, but does not require, a quan_titative ranking.

In contrast, FAR 15.305( a )( 2 ) explicitly calls for a comparative assessment for the past performance evaluation.

Sometimes, in my observation, past performance evaluations do not include a comparative assessment.  This comparative assessment asked for by FAR 15.305( a )( 2 ) is part of the past performance evaluation, and is separate from the comparative assessment of the proposals as a whole required by FAR 15.308.

ji20874:

It appears that what I have said has caused you to experience some degree of cognitive dissonance. But I can assure you, on the basis of my experience and past performance in having conducted (for DOD and DOE), observed and studied, taught, consulted, and publicly written about source selection in several books and who knows how many articles, that I know what I'm talking about. I'll brag and say that there is only one person alive who has written more about this or been cited more frequently about this than me.

Your reading of FAR 15.305 is simply wrong. Here is 15.305(a)(3) in its entirety:

Quote

(3) Technical evaluation. When tradeoffs are performed (see 15.101-1), the source selection records shall include—

(i) An assessment of each offeror’s ability to accomplish the technical requirements; and

(ii) A summary, matrix, or quantitative ranking, along with appropriate supporting narrative, of each technical proposal using the evaluation factors.

There shall be a ranking, and it shall be either "summary, matrix, or quantitative." Most agencies use summary or matrix rankings.

I know of no way to rank proposals other than by comparing them. See also FAR 15.305(a):

Quote

Proposal evaluation is an assessment of the proposal and the offeror’s ability to perform the prospective contract successfully.An agency shall evaluate competitive proposals and then assess their relative qualities solely on the factors and subfactors specified in the solicitation. Evaluations may be conducted using any rating method or combination of methods, including color or adjectival ratings, numerical weights, and ordinal rankings. The relative strengths, deficiencies, significant weaknesses, and risks supporting proposal evaluation shall be documented in the contract file.

There are two steps in each technical evaluation: 1) determination of the technical merits of each proposal on the basis of evaluation factor standards and 2) ranking of proposals from best to worst on the basis of the results of Step 1. Perhaps in your experience "technical evaluation" is Step 1 and Step 2 is called something else. I don't know. As I said, DOD organizationally separates Step 1 and Step 2 in some cases, but does not require that separation in all cases.

By the way, from a legal standpoint, there is no law against evaluating and rating proposals by direct comparison on the basis of the factors without the use of factor standards. In fact, that method is commonly taught in courses in decision analysis and is probably best for many commercial products.

Vern,

This thread is about past performance evaluations.  My point was that we don't always do past performance evaluations as a comparative assessment.  If we did always do past performance evaluations as a comparative assessment, rather than evaluating against a factor standard, and if we didn't define relevance so narrowly in our solicitations, then a lot of the angst about neutral ratings in past performance evaluations would disappear.

What is true from an academic perspective doesn't always show itself in practice in all the contracting offices around the country.

I've been encouraging direct comparisons (rather than evaluation against a standard) for technical evaluations for a while.  I hope the practice spreads.  Part of the hindrance is a notion in the community, as a result of all the "training" that has occurred, that we absolutely muist evaluate against the standard first, before the comparative assessment of the selecting official, even in the Subpart 8.4 world.  That notion isn't true, but the perception exists.  The perception also exists that past performance evaluations have to be done against a factor standard, rather than as a comparative assessment -- but as I have been trying to point out, FAR 15.305( a )( 2 ) clearly calls for a comparative assessment for the past performance evaluation.  Notwithstanding all the "training" that occurs, many contracting officers do not know this.

ji20874 said:

Vern,

This thread is about past performance evaluations.  My point was that we don't always do past performance evaluations as a comparative assessment.  If we did always do past performance evaluations as a comparative assessment, rather than evaluating against a factor standard, and if we didn't define relevance so narrowly in our solicitations, then a lot of the angst about neutral ratings in past performance evaluations would disappear.

... The perception also exists that past performance evaluations have to be done against a factor standard, rather than as a comparative assessment -- but as I have been trying to point out, FAR 15.305( a )( 2 ) clearly calls for a comparative assessment for the past performance evaluation.  Notwithstanding all the "training" that occurs, many contracting officers do not know this.

ji20874:

I am confused, and I think others may be as well, about what you mean by "comparative assessment." In order to help me understand, please describe such an approach. How would it be done?

ji20874:

I would like to know more about what you are saying to see if it can be applied within DoD.

It sounds like you are calling for past performance evaluations to comparatively assess which offer has the most relevant past performance.

Right now, seems as though you are advocating for COs to:

Step 1) gather past performance data on offerors

Step 2) review data and determine which offeror has more relevant or better relevant past performance as compared to each other versus a factor standard

DoD acquisitions under the Source Selection Procedures are very structured and might not facilitate such a scheme for past performance ratings. Particularly, I don't know that DoDs three legged stool approach to evaluating past performance facilitates such evaluations.

DoD relevancy ratings consider scope and magnitude and complexities. Offerors are required to have all three legs or the stool doesn't stand. Hypothetically, an offeror could be spot on in scope and complexities, but not have the magnitude and end up with an unknown/neutral rating.

Absent a mechanism to waive one of the legs of the stool, I don't see how we can comparatively assess offerors without first evaluating them against a factor standard to ensure they meet the agencies requirements. Otherwise, seems like you could be highly ranking the best available offeror without regard if they satisfy agency requirements.

I'm not limiting you to DoD SSP, just looking at it from my perspective.

I'm not constrained by the DoD source selection procedures guide.  As  I understand the guide, it calls for evaluation against a factor standard and assignment of adjectival ratings, with supporting rationale.  Maybe everyone comes out SUBSTANTIAL CONFIDENCE, or maybe everyone comes out NO CONFIDENCE, because everyone is evaluated against the standard.

Para. 1.4.4.4.3 of the DoD Source Selection Procedures prohibits the evaluators from performing a comparative assessment for the past performance evaluation, contrary to my reading of FAR 15.305( a )( 2 ) which expressly calls for a comparative assessment.  The DoD procedures require the evaluators to simply evaluate against the standard and assign adjectival ratings -- the comparative assessment is done by the SSAC (if one is appointed) and/or the SSA after the evaluation is complete and is done at the proposal level considering all factors.

Yes, Jamaal, I think a past performance evaluation is more useful and more in keeping with FAR 15.305( a )( 2 ) when it is done on a comparative assessment basis rather than an evaluation against the standard basis.  I realize that most contracting officers are constrained by their agency supplements and guides, as well as the prevalent notion that all evaluations (including past performance evaluations) must be against a standard and cannot be on a comparative assessment basis.  When opportunities arise, I remind contracting officers that FAR 15.305( a )( 2 ) expressly calls for a comparative assessment for the past performance evaluation.  I cannot mandate a particular approach to doing the comparative assessment, and would not mandate a particular approach even if I was empowered to do so.  But I do want contracting officers to be aware of what the FAR says, and I want for myself the privilege of doing what the FAR says (doing the past performance evaluation on a comparative assessment basis).

Vern, turning to the technical evaluation, I read FAR 15.305( a )( 3 )( ii ) differently than you do -- to me, it calls for:

• a summary of each technical proposal using the evaluation factors, along with appropriate supporting narrative;
• a matrix of each technical proposal using the evaluation factors, along with appropriate supporting narrative; or
• a quantitative ranking of each technical proposal using the evaluation factors, along with appropriate supporting narrative.

I think you read it as calling for:

• a summary ranking of each technical proposal using the evaluation factors, along with appropriate supporting narrative;
• a matrix ranking of each technical proposal using the evaluation factors, along with appropriate supporting narrative; or
• a quantitative ranking of each technical proposal using the evaluation factors, along with appropriate supporting narrative.

I am not accustomed to seeing evaluation teams prepare rankings within the technical evaluation, and our attorneys don't ask for them.  Even when I was in DoD, I never saw rankings within a technical evaluation.  Of course, I didn't see everything.

ji20874:

You told me in no uncertain terms that this thread is about past performance evaluations. And you have now told us three times how important you think it is to do a comparative evaluation of past performance without evaluation standards. So I asked you how you would conduct a comparative evaluation of past performance. Instead of answering, you now want to tell me how you interpret FAR 15.305(a)(3)(ii) about technical evaluations.

I frankly do not care how you interpret FAR 15.305(a)(3)(ii). I know what it says and what it means. I disagree with you, but I won't waste a moment trying to change your mind. Instead, I want to follow your lead about past performance evaluation and understand your thinking about how to conduct comparative evaluations of past performance without evaluation standards.

So are you going to describe how you would do that or are you not? If so, please proceed. I'm interested in your ideas about this.

Quote

Past Performance versus Responsibility Determinations

It is important to distinguish comparative
past performance evaluations used in the tradeoff process from pass/fail performance evaluations.

Pre-award surveys and pass/fail evaluations in the lowest price technically acceptable process help you determine whether an offeror is responsible. Responsibility is a broad concept that addresses whether an offeror has the capability to perform a particular contract based upon an analysis of many areas including financial resources, operational controls, technical skills, quality assurance, and past performance. These surveys and evaluations provide a “yes/no,” “pass/fail,” or “go/no-go” answer to the question, “Can the offeror do the work?” and thus help you determine whether the offeror is responsible.

Referral to the Small Business Administration may be necessary if a small business is eliminated from the competitive range solely on the basis of past performance. SBA referral is not required as long as the use of past performance information requires a comparative assessment with other evaluation factors and not as a pass or fail decision. The comparative assessment of past performance information is separate from a responsibility determination required by the Federal Acquisition Regulation.

Unlike a pass/fail responsibility determination, a comparative past performance evaluation conducted using the tradeoff process is a very specific endeavor that seeks to identify the degree of risk associated with each competing offeror. Rather than asking whether an offeror can do the work, you should ask, whether it will do that work successfully. In short, the evaluation describes the degree of confidence the Government has in the offeror’s likelihood of success. If properly conducted, the comparative past performance evaluation and the responsibility determination will complement each other and provide you with a more complete picture of an offeror than either one could by itself.

Joel,

Is your quote from the DoD May 1999 "A Guide to Collection and Use of Past Performance Information"?

On April 28, 2016 at 7:33 AM, ji20874 said:

Joel,

Is your quote from the DoD May 1999 "A Guide to Collection and Use of Past Performance Information"?

I was trying to edit on my iPad .  No it is from the 2001 Version 2. I hate iPad.  I hate Apple.  Not for Dummies, I guess.

I can only conclude that ji20874 does not know how he would do what he says ought to be done, which is to evaluate past performance comparatively, but without the use of evaluation standards.

The following excerpt is from the 2001 "DOD Guide to the Collection and Use of Past Performance Information."  It explains the difference between a comparative evaluation and a pass/fail evaluation. In both methods one must evaluate past performance against some criteria. 

Quote

Past Performance versus Responsibility Determinations 

It is important to distinguish comparative
past performance evaluations used in the tradeoff process from pass/fail performance evaluations. 

Pre-award surveys and pass/fail evaluations in the lowest price technically acceptable process help you determine whether an offeror is responsible. Responsibility is a broad concept that addresses whether an offeror has the capability to perform a particular contract based upon an analysis of many areas including financial resources, operational controls, technical skills, quality assurance, and past performance. These surveys and evaluations provide a “yes/no,” “pass/fail,” or “go/no-go” answer to the question, “Can the offeror do the work?” and thus help you determine whether the offeror is responsible. 

Referral to the Small Business Administration may be necessary if a small business is eliminated from the competitive range solely on the basis of past performance. SBA referral is not required as long as the use of past performance information requires a comparative assessment with other evaluation factors and not as a pass or fail decision. The comparative assessment of past performance information is separate from a responsibility determination required by the Federal Acquisition Regulation. 

Unlike a pass/fail responsibility determination, a comparative past performance evaluation conducted using the tradeoff process is a very specific endeavor that seeks to identify the degree of risk associated with each competing offeror. Rather than asking whether an offeror can do the work, you should ask, whether it will do that work successfully. In short, the evaluation describes the degree of confidence the Government has in the offeror’s likelihood of success. If properly conducted, the comparative past performance evaluation and the responsibility determination will complement each other and provide you with a more complete picture of an offeror than either one could by itself.

I gave up trying to edit my previous post on iPad. Everytime i tried to submit,  someone else had posted, freezing the edit  😡

I think it is important for all participants in a source selection to agree that essentially all past performance is relevant, but some past performance is more relevant than other past performance. If this is agreed to, then one offeror's past performance can be seen as more relevant than another offeror's past performance in a comparative way. That is a bedrock principle that allows a past performance evaluation team to approach its work in such a way that allows it to produce a comparative assessment product or document that can be of use to a source selection authority.

I agree with the point of Joel's quotation. I disagree with the point asserted earlier in this thread that "Past performance is nothing more than an assessment of how well the contractor has done that work in the past." No, a past performance evaluation is supposed to assess the likelihood of the offeror's future success for the instant procurement, based on its past performance. I do not like past performance evaluations that are nothing more than an assessment of how well the offeror has done that work in the past -- I want past performance evaluations that give a professional, subjective, and comparative assessment looking to the future -- I want an assessment of the likelihood of each offeror's success based both on each offeror's own past performance but also in comparison with each other. Because of my feeling in this matter, I am taking steps towards giving meaning to the comparative assessment called for in FAR 15.305( a )( 2 ). I am challenging the prevalent notion that past performance evaluations must be done against a standard, with no comparisons of offers, and that a comparative assessment can only be done by the source selection authority.

And since this thread is about neutral past performance ratings, I am trying to raise awareness in our professional community that FAR 15.305( a )( 2 ) expressly calls for a comparative assessment for the past performance evaluation. I believe that if more contracting officers looked for ways to do past performance evaluations as a comparative assessment, then much of the angst regarding neutral ratings and past performance evaluations would disappear, and we would get more meaningful and more useful past performance evaluations. I hope this discussion will encourage contracting officers to read or re-read FAR 15.305( a )( 2 ), and to find the "comparative assessment" words for themselves, and to try to put meaning into those words. I don't want any reader to think that I am opposed to evaluation standards -- I'm not -- I just am not satisfied with past performance evaluations that are only evaluations against a standard or that are nothing more than an assessment of how well the offeror has done that workin the past, so I generally want to ask for more from my past performance evaluators.

Ji20874:

How do you do a comparative past performance evaluation without standards? That's what you said should be done. Tell us how.

ji, It is my belief, at least from reading the DoD Guide, that a "comparative assessment" as used in 15.305 (a)(2) meant not using a pass/fail evaluation standards for PP in a BV trade-off.  Instead, use some type of comparative evaluation standards to evaluate EACH proposal and then make a comparative analysis between proposals.  Paragraph (a) (2) falls under 15.305:

Quote

15.305 Proposal evaluation.

(a) Proposal evaluation is an assessment of the proposal and the offeror’s ability to perform the prospective contract successfully. An agency shall evaluate competitive proposals and then assess their relative qualities solely on the factors and subfactors specified in the solicitation.

.

Joel:

That's exactly right. The comparative assessment is "my" Step 2.

Vern Edwards said:

Ji20874:

How do you do a comparative past performance evaluation without standards? That's what you said should be done. Tell us how.

That's not what I said.

ji20874 said:

I don't want any reader to think that I am opposed to evaluation standards -- I'm not -- I just am not satisfied with past performance evaluations that are only evaluations against a standard or that are nothing more than an assessment of how well the offeror has done that work in the past, so I generally want to ask for more from my past performance evaluators.

Joel, I understand your point about comparative not being pass/fail, and I accept that as a reasonable approach.  However, I understand from the DoD Source Selection Procedures that a contracting officer can do a proposal evaluation under FAR 15.305 with past performance on a past/fail basis (notwithstanding FAR 15.305( a )( 2 )'s call for a comparative assessment).  To me, FAR 15.305 doesn't allow for a past performance evaluation on a pass/fail basis -- this is in contrast to the technical evaluation text in 15.305( a )( 3 ) "When tradeoffs are performed..." -- no similar words appear in ( a )( 2 ).  Even so, I have never done past performance on a pass/fail basis as a matter of professional choice -- I don't think it makes any sense.

ji20874:

Agency Alpha conducts a past performance evaluation using formal, objective standards-- the method you don't like. Then, in the source selection decision, "The source selection authority’s (SSA) decision [is] based on a comparative assessment of proposals against all source selection criteria in the solicitation...the documentation shall include the rationale for any business judgments and tradeoffs made or relied on by the SSA, including benefits associated with additional costs" (FAR 15.308).

Agency Beta conducts a past performance evaluation using the method you prefer (whatever comparative analysis you're trying to convey in this thread). Then, in the source selection decision, "The source selection authority’s (SSA) decision [is] based on a comparative assessment of proposals against all source selection criteria in the solicitation...the documentation shall include the rationale for any business judgments and tradeoffs made or relied on by the SSA, including benefits associated with additional costs" (FAR 15.308).

Are there any important differences between Alpha and Beta? To be clear, PepeTheFrog is emphasizing two stages: stage (1) proposal evaluation (FAR 15.305) and stage (2) source selection decision, including a comparative assessment of the proposals against all source selection criteria (FAR 15.308).

The only thing that hops out at PepeTheFrog is that under the method you prefer, more past performance will meet the relevancy test, and therefore be evaluated, based on your statement here, and others:

ji20874 said:

I think it is important for all participants in a source selection to agree that essentially all past performance is relevant, but some past performance is more relevant than other past performance.

It sounds like you want to lower the standard for relevancy, and therefore spend more time and effort evaluating a greater amount of past performance. To accomplish that goal, you need not mention comparative assessment of past performance.

ji20874 said:

I want an assessment of the likelihood of each offeror's success based both on each offeror's own past performance but also in comparison with each other.

Comparative assessment of past performance will happen in stage (2), mentioned above.

What is PepeTheFrog missing? What is the harm in having the comparative assessment (of past performance) happen at the tradeoff analysis or source selection decision stage?

ji20874 is auditioning for the role of the Artful Dodger. It's laughable.

Vern Edwards said:

ji20874 is auditioning for the role of the Artful Dodger. It's laughable.

Give him a break, Vern. You only asked him to explain himself three times.

On April 27, 2016 at 10:04 AM, ji20874 said:

Vern,

... If we did always do past performance evaluations as a comparative assessment, rather than evaluating against a factor standard, and if we didn't define relevance so narrowly in our solicitations, then a lot of the angst about neutral ratings in past performance evaluations would disappear.

*     *     *

I've been encouraging direct comparisons (rather than evaluation against a standard) for technical evaluations for a while.  I hope the practice spreads.  Part of the hindrance is a notion in the community, as a result of all the "training" that has occurred, that we absolutely muist evaluate against the standard first, before the comparative assessment of the selecting official, even in the Subpart 8.4 world.  That notion isn't true, but the perception exists.  The perception also exists that past performance evaluations have to be done against a factor standard, rather than as a comparative assessment -- but as I have been trying to point out, FAR 15.305( a )( 2 ) clearly calls for a comparative assessment for the past performance evaluation.  Notwithstanding all the "training" that occurs, many contracting officers do not know this.

ji20874 said:

Yes, Jamaal, I think a past performance evaluation is more useful and more in keeping with FAR 15.305( a )( 2 ) when it is done on a comparative assessment basis rather than an evaluation against the standard basis.

ji20874 has been confused. I think he knows it, and now he's in denial. 

Here is the straight skinny, but we have to study some history:

Traditionally, past performance was used as a pass/fail responsibility consideration. But the GAO stated on a few occasions that responsibility considerations could be used as evaluation factors in source selection, and when used as such on a comparative basis the SBA's certificate of competency (COC) program did not apply.

After enactment of CICA, agencies began to see an advantage in using responsibility considerations as source selection evaluation factors as a way to to side-step the COC program. They thought that SBA was too generous with COCs and resented SBA forcing them to do business with firms that they did not think were qualified. GAO said it was okay as long as such factors were not evaluated on a pass/fail basis. If used as source selection factors, but evaluated on a pass/fail basis, then the COC procedure would still apply. See FitNet Purchasing Alliance, B-410263, 2014 CPD ¶ 344, Nov. 26, 2013.

Quote

Under the Small Business Act, agencies may not find a small business nonresponsible without referring the matter to the SBA, which has the ultimate authority to determine the responsibility of small businesses under its COC procedures. 15 U.S.C. §637(b)(7); FAR subpart 19.6; Federal Support Corp., B–245573, Jan. 16, 1992, 92–1 CPD ¶81 at 4. Past performance traditionally is considered a responsibility factor, that is, a matter relating to the offeror's ability to perform the contract. See FAR §9.104–1(c); Sanford & Sons Co., B–231607, Sept. 20, 1988,88–2 CPD ¶266 at 2. Traditional responsibility factors may be used as technical evaluation factors in a negotiated procurement, but only when a comparative evaluation of those areas is to be made. See, e.g., Medical Info.Servs., B–287824, July 10, 2002, 2001 CPD ¶122 at 5; Nomura Enter., Inc., B–277768, Nov. 19, 1997, 97–2 CPD ¶148 at 3. Comparative evaluation in this context means that competing proposals will be rated on a scale, relative to each other, as opposed to a pass/fail basis. Docusort, Inc., B–254852, Jan. 25, 1994, 94–1 CPD ¶38 at 6. We have cautioned that an agency may not disqualify a small business under the guise of a relative assessment of responsibility-based technical factors in an attempt to avoid referral to the SBA. Federal Support Corp., supra, at 4; Sanford & Sons Co., supra, at 3.

See also Cibinic, Nash, and Yukins, Formation of Government Contracts 4th, pp. 702-713.

In 1994, OFPP proposed a program for the use of past performance information in source selections. See 59 FR 18168, April 15, 1994. The proposal reflected the GAO case law, and thus stated, in pertinent part:

Quote

Responsibility Determinations vs. Source Selection Decisions. Some confusion exists because past performance is used in making responsibility determinations as well as contract award decisions. Federal Acquisition Regulation (FAR) 9.104-3(c) requires a satisfactory performance record before a contractor can be determined to be responsible. Similarly, *18169 FAR 9.104-1(e) requires that the contractor “have the necessary organization, experience, accounting and operational controls . . .,” in order to be determined responsible. If an agency determines that a small business is “not responsible” (i.e., not capable of performing) that determination may be appealed to the Small Business Administration (SBA), under “Certificate of Competency” (COC) procedures.

Contract award or evaluation decisions are (in comparison to responsibility determination), made pursuant to evaluation criteria stated in the RFP. When treated as an evaluation factor, past performance information is used to make comparisons among competing firms to determine relative ratings or rankings; e.g., firm A is better (presents less risk to the government) then firm B; firm B is better than firm C, etc. Evaluation decisions assess the relative capability of firms. They are not “go/no go” decisions and are not subject to the COC process or referral to the SBA.

The program would eventually become the basis for coverage in FAR Parts 15 and 42.

What about the "neutral" problem? According to OFPP:

Quote

“New” Firms. One of the most frequently asked questions about using past performance information in source selections is, “How are new firms to be treated?” In OFPP's view, new firms should be neither rewarded nor penalized as a result of their lack of performance history. If, for example, past performance is to be rated on a scale of one to ten, a new firm should be given the average score of the other competing offerors. Unless the RFP contains a specific requirement for prior performance based on safety, health, national security, or mission essential considerations, agencies should “neutralize” the past performance factor and evaluate the merits of proposals received from new firms in accordance with other stated evaluation criteria.

It was the use of past performance as a comparative evaluation factor instead of a pass/fail responsibility consideration that gave rise to the "neutral" problem, which is why ji20874 was off base in saying:

Quote

If we did always do past performance evaluations as a comparative assessment, rather than evaluating against a factor standard, and if we didn't define relevance so narrowly in our solicitations, then a lot of the angst about neutral ratings in past performance evaluations would disappear.

That's nonsense. It was comparative assessment that caused the angst. A "neutral" rule was thought to be necessary so as not to be unfair to new firms.

ji20874 confuses the idea of evaluating against standards with comparative evaluation. As the quotes above show, be believes (or believed) that evaluation against standards is inconsistent with comparative evaluation. He's wrong. He simply doesn't understand that evaluation against standards is usually Step 1 of evaluation and comparison (and ranking) is Step 2.

However, ji20874 is right that evaluation against standards is not essential. Steps 1 and 2 can be done simultaneously by comparing proposals in a series of pairs and going directly to ranking without application of standards. How could that be done? First, you look at the information about Offeror A's past performance. You do not assign a rating or score. You then look at the information about Offeror B's past performance. You then decide which you think is better, A or B, and rank them 1st and 2nd without assigning a rating. Suppose that you decide that B is better than A. You then compare Offeror B to Offeror C. If you decide that C is better than B it stands to reason that C is also better than A, and you now have a ranking 1st, 2nd, and 3rd. But if C is not as good as B, you then compare C to A to determine which of them is better. You have to do that in order to make tradeoffs later in the process. You continue in that way until you have a complete ranking of all offerors. Since you do not assign ratings, you do not need rating standards. The OFPP proposal addressed this, but somewhat confusedly:

Quote

Should Past Performance Be Scored or Not Scored? The evaluation of past performance is a subjective assessment based on specific facts and circumstances. Past performance evaluations are not generally based on absolute standards of acceptable performance and there is no requirement that such assessments be scored. The decision of whether to score past performance or to use other assessment methods such as color codings, adjectival descriptions or rankings is a determination that must be made by each procuring agency.

Comparison without ratings or scores is easy to do with a quantitative factor, but hard to do, impracticable in most cases, with something like past performance. That's because of the complexity of the information on which comparisons are to be based. Keep in mind that the sole function of raings and scores is to simplify more complex information in order to facilitate comparison.

What about risk? Evaluation of past performance and evaluation of risk are separate matters. Presumably, past performance has predictive value. But risk is a more complex idea than predictive value. That's why DOD uses its "confidence assessment" scheme. I proposed such a scheme more than  20 years ago in a monograph for The George Washington University Law School: How to Evaluate Past Performance: A Best-Value Approach (1994), which went into a second edition (my copy of which I can't find at the moment). I called the scheme the Level of Confidence Assessment Rating or LOCAR, which has been used successfully. See AdapTech General Scientific, LLC, B-293867, 2004 CPD ¶ 126, June 4, 2004, and Colmek Systems Engineering, B-291931.2, 2003 CPD ¶ 123, June 9, 2003. However, my LOCAR scheme used numbers, and most agencies are not comfortable with the use of numbers. Adjectival or color rating is the norm.

There's a lot more history of the use of past performance as an evaluation factor, but a forum post is not the place to provide all of it. Perhaps I'll write a blog post some day.

Finally, contrary to ji20874's claim, FAR does not require the use of a comparative approach. In devising approaches to source selection within the best value continuum, past performance can be evaluated using either a comparative or a past/fail approach, or not evaluated at all. But if evaluated on a pass/fail basis, the COC procedure will apply if an offeror is found to be unacceptable due to poor past performance. See FAR 15.101-2(b)(1). That's true of all traditional responsibility-type considerations.

FAR 15.305( a )( 2 ), Past Performance Evaluation, calls for a "comparative assessment of past performance information."

FAR 15.308, Source Selection Decision, calls for a "comparative assessment of proposals against all source selection criteria."

I am suggesting that these are two separate comparative assessments -- the first done as part of the past performance evaluation (hence its mention in FAR 15.305( a )( 2 ) and the second done as part of the selection decision (hence its mention in FAR 15.308).  Others seem to believe that these are the same.

Vern, I absolutely disagree with your notion expressed earlier in this thread that past performance is nothing more than an assessment of how well the contractor has done that work in the past.  No amount of mocking will get me to change my mind on this.  I hope for intellectual honesty in this forum -- I'm okay with different opinions, but I hope for intellectual honesty.  I have already pointed out the following--

ji20874 said:

I don't want any reader to think that I am opposed to evaluation standards -- I'm not -- I just am not satisfied with past performance evaluations that are only evaluations against a standard or that are nothing more than an assessment of how well the offeror has done that work in the past, so I generally want to ask for more from my past performance evaluators.

I don't see anything wrong with this statement.  For those here who are satisfied with past performance evaluations that are only evaluations against a standard or that are nothing more than an assessment of how well the offeror has done the work in the past, more power to you.  But I want more from my past performance evaluators.

ji20874:

Don't talk to me about intellectual honesty. You're conveniently skipping over the other comments you made about evaluating against standards, which I quoted, and emphasizing something you said late in the game, after you realized that you'd spoken too soon and said too much. You're not opposed to evaluation standards? Wasn't it you who said:

Quote

I've been encouraging direct comparisons (rather than evaluation against a standard) for technical evaluations for a while.  I hope the practice spreads... The perception also exists that past performance evaluations have to be done against a factor standard, rather than as a comparative assessment -- but as I have been trying to point out, FAR 15.305( a )( 2 ) clearly calls for a comparative assessment for the past performance evaluation... I think a past performance evaluation is more useful and more in keeping with FAR 15.305( a )( 2 ) when it is done on a comparative assessment basis rather than an evaluation against the standard basis.

Was that written by someone who stole your identity,  And what's with this new thing about "two separate comparative assessments"? Man, you've been all over the map. Get a grip. Read a book: Competitive Negotiation: The Source Selection Process 3d, by Nash, Cibinic, and O'Brien (2011).

Just a note: other observers are interested on how ji20874 is executing past performance evaluation without standards in his office. A method was mentioned by another post, but I think it is worth further discussion. Thanks.

In light of JMG’s request, it might be helpful to understand what “standards” are and why agencies use them.

Standards are criteria for judgement. Suppose that you are evaluating different candidates for office chairs and one of your criteria is comfort. The more comfort the better. Well, comfort is a highly subjective criterion. Different people will likely have different ideas about what is and is not comfortable in a given context. Thus, if you are going to appoint a panel to evaluate the relative comfort of competing chairs you will either have (a) to aggregate what may be widely differing opinions or (b) set a common standard for comfort in order to ensure (1) that everyone is evaluating comfort the same way and (2) that the competing chairs will be evaluated on the same basis. Thus, you might establish standards for ratings such as:

• Comfortable
• Uncomfortable

or

• Extremely comfortable
• Very comfortable
• Acceptably comfortable
• Somewhat uncomfortable
• Extremely uncomfortable

Et cetera. Each standard should describe what the evaluators must think or experience in order to assign a particular comfort rating.

Standards are generally set before receipt of proposals in order to avoid complaints of that personal biases affect the development of standards after evaluators learn or suspect the identities of the competitors. Evaluators are told to evaluate strictly on the basis of the standards and to avoid direct comparisons in order to (1) ensure that minimum requirements are met and (2) avoid protests that personal preferences affected the evaluation outcome.

However, neither statute nor FAR require the development of evaluation standards (some agencies require them), and evaluation by direct comparison is not generally illegal. Each panel member could sit in each chair and personally rank the chairs from most to least comfortable. The panel members could then share and discuss their rankings and either reach a consensus or vote on a ranking.

If an evaluation factor is simple enough, direct comparison might work perfectly well. If a factor is multi-faceted or otherwise complex, it generally becomes harder to reach a consensus among evaluators, and a majority-rule vote may be necessary. Consensus is generally preferred to voting, because majority-rule decisions are not considered the best way to determine best value, but voting is not illegal.

One problem with setting standards before receipt of proposals is the possibility of short-sightedness or the emergence of an unanticipated proposal feature that is not encompassed by the standard. Such an event could give rise to an issue as to what to do: proceed on the basis of the original standard or change the standard after the fact.

The GAO has indicated that evaluation standards must be disclosed to prospective offerors. See RJO Enterprises, Inc., B-260126, 95-2 CPD ¶ 93, footnote 13.

Now we can all wait for ji20874’s response to JMG’s request.

So this topic took off like no tomorrow. Based on my original posting, my scenario involves a past performance evaluation factor that puts experience and performance together but fails to have a confidence assessment scheme; like whats in the DoD SSP.  Because of this, the past performance evaluation factor is less than effective.

Thanks Vern. Would another example of standards, taken from the DOD Source Selection Procedures, be relevancy of past performance (p.27) ? The standards for relevancy being :

Very relevant; relevant; somewhat relevant;  not relevant 

(setting aside your opinion of the document)

Ref: https://acc.dau.mil/docs/DoDSSP/Source%20Selection%20Guide%20and%20Memo%201%20April%202016%20ljm.pdf

JMG:

Yes, those are evaluation standards, e.g.:

Quote

Very relevant: Present/past performance effort involved essentially the same scope and magnitude of effort and complexities this solicitation requires.

I can't help but say that those standards are dubious. But they're definitely standards.

Vern Edwards said:

Keep in mind that the sole function of raings and scores is to simplify more complex information in order to facilitate comparison.

PepeTheFrog thinks Vern Edwards' statement is worth repeating. Many contracting professionals don't understand this key concept. The ratings are shorthand for the underlying, more complex information-- the strengths, weaknesses, etc. The underlying, more complex information is what a sound decision should be based upon, not a formulaic "conclusion" based on the ratings in isolation.

What good is a conclusion with no premises? How useful is an answer (to a complex question) that provides no explanation? Check the Wifcon forums regularly to find out.

*Rimshot*

Thanks, folks, you're too kind! PepeTheFrog will be here all night, tip your waiter!