Down-Selects Vs. 15.202 Advisory Multi-step & Competitive Ranges - FAR part 15

QUESTION 1

There is no FAR-defined downselect process. The FAR 15.202 advisory multi-step process is not the one-and-only downselect process; rather, it is an advisory mult-step process.

If you will go to the GAO website at www.gao.gov and do a search under bid protests for "downselect," you will see all sorts of examples where downselects were done, using all sorts of approaches that differ from the FAR 15.202 advisory multi-step process.

In summary, you should not assume that the “down-select” process is the Advisory Multi-Step Process now covered in FAR 15.202.

QUESTION 2

lf your market research suggests you might get 200 proposals, then it makes sense for you to design you acquisition strategy accordingly. I recently issued a solicitation where we had five factors and we asked for complete proposals, but we said that we will evaluate two of the factors in the first step and only those with certain ratings would proceed to the second step where the other factors would be evaluated. You could call this a downselect process (alothough we didn't use that term -- I prefer to look at it as pre-screening criteria -- I think they call it a quality ranking factor in the personnel world).

EDIT -- Yes, Vern's term of phased evaluation is the right way to say it...

You should not speak of a competitive range for the purpose of limiting the number of offers on which you will do the cost evaluation -- you should only speak of a competitive range in the context of conducting discussions.

I think if they are technically unacceptable, they by definition are not in your competitive range.

Then you have to do some mail-merging to generate 100 rejection letters, but you shouldn't have to do a cost analysis on those proposals. Shoot, a simple price analysis ought to be sufficient. Am I comprehending your question?

Forget the term "downselect" and don't use "prescreening criteria." You're talking about phased evaluation.

Go find a copy of Source Selection Answer Book, 2d ed., published by Management Concepts. Look up "phased evaluation" in the index and read the entries.

Here is some solicitation verbiage for one approach to phased evaluation:

We will evaluate offerors in phases and progressively reduce the number being considered for contract award. The purpose of this procedure is to reduce the time and cost of source selection, both for ourselves and for the competing offerors. We will execute the procedure as follows:

1. We will determine the acceptability of each offer in accordance with the acceptability criteria in this RFP. We will eliminate from further consideration any offerors that submitted unacceptable offers.

2. We will then perform a price analysis and eliminate offerors whose prices we consider to be too high to be competitive in light of the prices offered by other offerors and our estimate of price.

3. We will then evaluate the experience, past performance, and prices of the remaining offerors and eliminate any that are not sufficiently competitive to be selected for contract award.

4. Once we have reduced the number of offerors to what we consider to be an appropriate number we will schedule oral presentations by the remaining offerors and select the contractor by comparing those offerors on the basis of experience, past performance, understanding of the work, and price, and make an award without discussions.

The decision to eliminate any offeror through this procedure will not constitute the establishment of a competitive range, as described in FAR 15.306[c], and it will not obligate us to conduct discussions or to solicit or entertain proposal revisions. However, we reserve the right to establish a competitive range and conduct discussions if we decide that it is in the best interest of the Government to do so.

Note that price is considered in each step.

Excellent approach!

Thanks Vern!

That is exactly what I was looking for. I have the book in hand and may follow up if I have any questions.

So, using the downselect or contingency procedure described in the Source Selection Answer Book (Second ed), If used on a MA IDIQ, what element of price/cost would us suggest in evaluating in Step 2 of the approach listed by Vern above?

Would it be a labor rate analysis proposed for the Labor Categories or would it potentially be the total cost proposed for the Seed Task Order?

Any ideas?

A price evaluation should be based on price or total estimate cost, not rates.

Thanks Vern!

Then reading between the lines, Phase 2 = evaluation on total Price/cost. Phase 3 = compelet analysis of Price/Cost; Correct? May seem like a stupid question, but I have to ask.

Also, Vern, the book is awsome. I hear that you are handing these out at Boot Camp. Is that still the case?

FARmer - Think. What makes sense for your current requirement? Processes are often not defined and should not be because mandating a standard procedure often leads to bad results because some requirements do not fit. Phase 3 as you mention would not make sense for something that is a competitive FFP contract because you are not doing a cost analysis there. Not everything fits into a defined process, think through the requirement to see what best fits your current situation.

dcarver - Thanks for the response...... I do understand that processes are often not defined and its mostly because all requirements are not the same. HOWEVER, please understand that the purpose of my post is to learn more about this subject. There really isn't much information nor many subject matter experts that I can have a conversation with on the use of this phased approach. Hence the reason why people post on this forum.... To ask questions.

Overall, I was just trying to get ideas on doing this phased approach to a predominantly COST type MA IDIQ.

Vern et al,

I had a very interesting conversation with legal on some phased evaluation language and it revolved around the past performance evaluation aspect in a phased approach.

For this example, lets use Phase #3 in the example provided by Vern on 9/15. If you have an offeror with no past performance, and per FAR 15.305(a)(2)(iv), "...the offeror may not be evaluated favorably or unfavorably on past performance," how do you eliminate an offeror in this phase because of "neutral" rated performance? Isn't this essentially rating them "unfavorably."? (hopefully this made sense.... it's been a long day)

My thought on this is that you are not necessarily rating them unfavorably and therefor eliminating them from the next phase, but based on your analysis, the neutral rating may be determined to not sufficiently be competitive to be selected for contract award. In my opinion, this could be seen as two different things a rating vs. what is sufficient to be competitive.

Thoughts?

Sounds about right. Here's my 30 second a assessment of what I believe you said.

Offeror 'A' is neutral, which could result in the government having unknown confidence in their ability to perform, however, the remaining offerors are [insert your agency's rating nomenclature], which produces [insert confidence rating]. The Government performed some sort of trade-off/comparative analysis and as a result, offeror 'A' is excluded from the 'competitive range'.

Jamaal - Spot on my friend. Now the question is, that using a phased approach and eliminating Offeror 'A', wouldn't this going against FAR 15.305(a)(2)(iv) and "evaluating" them unfavorably?

My thought is No. Since the agency followed the FAR and gave Offeror 'A' the appropriate evaluation/rating for having no past performance, the neutral rating may be determined to not sufficiently be competitive to be selected for contract award. Therefore the Offeror is eliminated from further consideration.

In support of FARmer's thought, see GAO Bid Protest decision Richen Management LLC, B-409697, July 11, 2014. The lower-priced offer had a "neutral" rating for past performance, but the selecting official selected the higher-priced offer as the best value.

Richen Sparkle

Past Performance Neutral Very Good

Price $451,425 $535,371

The contracting officer, the source selection authority for this procurement, determined that Sparkle’s qualitative advantage (i.e., superior past performance) outweighed Richen’s $83,946 (19%) price advantage, and that Sparkle’s proposal represented the best value to the government.

Jamaal - Spot on my friend. Now the question is, that using a phased approach and eliminating Offeror 'A', wouldn't this going against FAR 15.305(a)(2)(iv) and "evaluating" them unfavorably?

My thought is No. Since the agency followed the FAR and gave Offeror 'A' the appropriate evaluation/rating for having no past performance, the neutral rating may be determined to not sufficiently be competitive to be selected for contract award. Therefore the Offeror is eliminated from further consideration.

FARmer, you do know that you can downgrade proposal for lack of recent, relevant experience, separately from lack of a past performance record, right? That is why you want to evaluate "experience" under a separate factor or subfactor than lumping it into the umbrella of "past performance".

The DOD mmandated method for evaluating past performance based upon a confidence rating can confuse the distinction between pp and experience. In the first step of that pp evaluation scheme, the government determines the degree of relevancy of past performance on cited project experience to the scope of the instant solicitation. In the second step the government evaluates the past performance of the various projects.

In step/phase 2, past performance on those projects more relevant are given more consideration in assigning the rating for degree of confidence that the offeror/proposer will succeed in performing the scope of the instant contract/task order.

Thus, the past performance record of a candy maker with outstanding past performance should not carry as much significance in assessing the government's degree of confidence for performance on a construction project, for example, as the past performance record of a company on construction projects that bear relevance to the instant project/scope. And the degree of relevance of construction project experience to the instant solicitation between firms with construction experience will also be a discriminator. The past performance record on more relevant projects is given more consideration (positive or negative) than the record on less relevant projects.

However, you can use the extent of recent, relative experience as a separate factor/sub factor in a multi phased/multi stepped evaluation scheme. You CAN treat the extent of recent, relative experience of a firm with little or no relevancy favorably or unfavorably in determining who will make it to the next step.

In support of FARmer's thought, see GAO Bid Protest decision Richen Management LLC, B-409697, July 11, 2014. The lower-priced offer had a "neutral" rating for past performance, but the selecting official selected the higher-priced offer as the best value.

Richen Sparkle

Past Performance Neutral Very Good

Price $451,425 $535,371

The contracting officer, the source selection authority for this procurement, determined that Sparkles qualitative advantage (i.e., superior past performance) outweighed Richens $83,946 (19%) price advantage, and that Sparkles proposal represented the best value to the government.

ji, your cited example concerns the best value tradeoff comparison for award, not a decision to eliminate a firm from further competition based upon a neutral or unknown" past performance/confidence rating. The former is a relative comparison, while the latter usually is a cut-off determination in a step preceding the best value comparison between proposals.

FARmer:

I don't think it's contrary or rating the contractor unfavorably. The contractor scored a neutral. In turn, you're rating the government and your confidence in the contractor based on what you know. I see the two actions as distinct and separate.

Given the discussion, is it odd that FAR 1.102-2( a )(3) states: When selecting contractors to provide products or perform services, the Government will use contractors who have a track record of successful past performance or who demonstrate a current superior ability to perform?

EDITED: Jamaal, I checked a January 1, 1996 hard copy of the FAR, which contains the same wording in 1.102 ( a )(3) through FAC 90-26 and 90-29 as of 7/3/1995. It also includes the requirement to rate firms lacking relevant past performance under a "neutral" rating in the pre-FAR 15 rewrite version at paragraph 15.608 ( a ) (2) III.

FAC 90-26 added the neutral rating requirement a couple of months prior to FAC 90-29. Paragraph 1.102-( a )(3) was not part of either of those FACs, therefore it must have preceded them. The "neutral" term in Part 15 was subsequently replaced by a requirement to not rate them favorably or unfavorably but to consider it an "unknown" performance risk.

Don Mansfield has a blog entry on the subject at /blogs/don-mansfields-blog/myth-information-neutral-past-performance

At any rate these paragraphs in Parts 1 and 15 are uncoordinated, due to the first paragraph preceding the "neutral" rating language and subsequent updates to those rating requirements.

For this example, lets use Phase #3 in the example provided by Vern on 9/15. If you have an offeror with no past performance, and per FAR 15.305(a)(2)(iv), "...the offeror may not be evaluated favorably or unfavorably on past performance," how do you eliminate an offeror in this phase because of "neutral" rated performance? Isn't this essentially rating them "unfavorably."? (hopefully this made sense.... it's been a long day)

Don't confuse the process of "rating" an offeror with the process of comparing it to other offerors. An offeror with no past performance and a price of $1,000,000 might not be competitive with an offeror with excellent past performance and a price of $1,010,000 or $1,100,000. You might eliminate an offeror because it compares poorly, because you don't have enough information to determine the risk associated with choosing it, because it's an unknown.

joel,

I think a contracting officer can reasonably choose not to include an offeror in the competitive range because of a neutral or unknown past performance rating. Contracting agencies are not required to retain in the competitive range proposals that are not among the most highly rated or that the agency otherwise reasonably concludes have no realistic prospect of being selected for award. A competitive range decision can include a comparison -- a competitive range need not be mechanistic.

A competitive range decision must not be mechanistic (mechanical). Thus, a CO should not exclude an offeror from the competitive range because of a lack of past performance, but because its lack of past performance makes it less competitive than firms with records of good performance. Don't exclude a firm from the competitive range because of a rating, but because of the result of a comparison made on the basis of ratings and facts.

To go back to the original posting, here is some text that I used for a phased-evaluation fair opportunity consideration back in March. IDIQ contractors had to submit complete written proposals (covering Factors 1, 2, 3, and 5) up front. We anticipated a large number of offers for a very high-dollar acquisition.

Factor 1: Corporate Experience of the Prime Contractor.

Factor 2: Technical Approach of Contractor Team.

Factor 3: Management Approach of the Contractor Team.

Factor 4: Oral Presentation.

Factor 5: Price.

The Government intends to use a three-step approach to evaluate offers under 16.505( b )( 1 ) for each portfolio, as follows—

First-Step Consideration. The Government will consider all offers using Factors 1 and 5. Those offers rated as “high confidence” for Factor 1 and “reasonable” for Factor 5 will proceed to the second step consideration. If necessary to provide for adequate competition in the second step, the Government may select one or more offers rated as “some confidence” for Factor 1 and “reasonable” for Factor 5 to proceed to the second step consideration.

Second-Step Consideration. The Government will continue its consideration by evaluating Factors 2 and 3. The Government intends to select at least two but no more than four offers as being most likely to provide the best value solution based on a trade-off of Factors 1, 2, 3 and 5 to proceed to the third step consideration. For this step, all four factors are of relatively equal importance.

Third-Step Consideration. The Government will continue its consideration by inviting oral presentations and evaluating Factor 4. The Government intends to select the best value offer based a trade-off of Factors 1, 2, 3, 4, and 5. For this step, all five factors are of relatively equal importance.

Two questions:

1. Do you evaluate oral presentations or do you evaluate the content of statements made in oral presentations? If the latter, what kinds of statements do you solicit and evaluate?

2. Why bother with written descriptions of "approach'? Why not get those descriptions in the oral presentations? The idea behind oral presentations is to replace written descriptions, not to get written descriptions and oral presentations.

WOW! Thanks for all the great info everyone.

So to expand a little more on my question, I do understand that the evaluation and rating given for past performance has nothing to do with the comparison being made in the phased evaluation process, however, the evaluation that is happening during the phased evaluation to past performance, is this a simple comparison of past performance, or is/could this a full blown evaluation that results in a rating? Then that rating is used to do the comparison?

Somewhere down the line we got back into talking about competitive ranges. My understanding is that the phased evaluation has nothing to do with a competitive range. A competitive range still may be applicable after all phases of the phased evaluation has been done; Correct?

Lastly, Vern, I know that your example provided in this discussion and the one provided in the book (just examples) mention "we will then perform a price analysis and eliminate offerors whose prices we consider to be too high...." What about eliminating offerors that propose prices that are too low? Is this because we would then have to go into a realism analysis to make that determination or does it further add to the complexity of the phased evaluation?

Lastly, Vern, What about eliminating offerors that propose prices that are too low? Is this because we would then have to go into a realism analysis to make that determination or does it further add to the complexity of the phased evaluation?

If you're thinking that you might want to eliminate offerors that propose unrealistically low prices, then you'll have to conduct a price realism analysis. Yes, that will further add to the complexity of a phased evaluation.

Good luck to you. Simplicity is dead. I'm off to the funeral.

ji20874:

Why not combine factors 1, 3, and 4? If you are lucky enough to be able to use oral presentations, why not leverage it to the max?

Might be able to have factor 2 in there as well if structured right.

Taking i20874's factors as given, I would rework them as follows, in order of importance:

Assent to the Government's terms (on a pass/fail basis).

Corporate experience

Price (or cost and fee)

Technical/management approach

(Oral presentation is not an evaluation factor; it is an information delivery method. You don't evaluate oral presentations. You evaluate something else based on information obtained through oral presentations.)

I'd ask for the assent, experience, and price information up front. I would rank offerors on those bases. I might eliminate some. I would tell the survivors to make oral presentations that will include Q&A sessions. I would evaluate technical/management approach based on the oral presentations and then do my combined assessments and make tradeoffs to choose the winner on the basis of all factors.

The key time and money saver is the elimination of written "approach" proposals.

A competitive range and its determination are for the purpose of conducting discussions (15.306) and for requesting revisions to proposals, not for short listing procedures or other phased evaluation procedures prior to selecting firms to conduct discussions with.

I apologize for assuming earlier that you were conducting a multiple phase proposal submission something akin to two phase design- build process in 36.3.

I don't understand what prompted the last comment. One need not make a competitive range determination in order to eliminate one or more firms from a competition in a phased evaluation.

In a few of the earlier posts we started to discuss competitive ranges in this phased evaluation process discussion. I just wanted to make it clear to anyone that came upon this discussion that the competitive range is not the same as a phased evaluation. Joel's response was spot on as this discussion was more towards conducting phased evaluations outside of a two phased design-build process per FAR subpart 36.6 and I would say I was hoping to steer this discussion of phased evaluations more towards services under FAR part 15.

When discussing this phased evaluation with other contracting folks, it's easy for them to jump to the conclusion that this is a competitive range........ and it's not.

Vern, back to your comment about simplicity being dead.... Sorry and I agree 100%. But as your book states, the decision to use this phased approach "may be controversial"...... and it is in my world. So after rereading your book, it is a full blown evaluation (that leads to a rating) of the non-price factor being evaluated and then a simple comparison to other offerors during the phased approach. That's my take away.

I don't understand what prompted the last comment. One need not make a competitive range determination in order to eliminate one or more firms from a competition in a phased evaluation.

Vern, I agree with your last statement. Posts 1,3,13, and 21 - all by separate posters - appear to refer to the competitive range in the context of the multiple phased evaluation/elimination process. Post #21 was in response to an earlier post by me, which discussed eliminating proposals on the basis of past performance and/or lack of experience. I never used the term "competitive range".

Your own post 22 appeared to be in response to post #21 - which, as I said, was discussing a competitive range in a context unrelated to conduct of discussions.

In post #25, FARmer asked a question concerning the correct use of the term. Having been out of cell coverage area for a few hours, I was responding to those 5 or 6 posts.

EDIT: FARmer answered while I was posting this response. I agree with FARmer.

FARmer:

Phased evaluation is simply progressive winnowing of the field of competitors through phased application of the evaluation factors.

The idea is to start with the easiest factors to evaluate -- generally: legal acceptability, experience, and bottom-line price -- so that by the time you reach more difficult factors to evaluate you won't have as much work to do. There is no reason for the concept to be controversial, except to the eternally blockheaded.

You must consider price in each phase before eliminating anyone. However, if your solicitation says that you are going to award without discussions, then you do not have to consider the price of any offeror whose proposal is ineligible for award as submitted. For example, you don't have to consider the price of any firm whose offer does not conform to any material term of the prospective contract as stated in the solicitation.

Thanks for the response everyone!

So based on the phase approach, some "blockheads" (lol) may think that by putting the easiest factor (for instance Past Performance) in the phased approach may make it appear that past performance is the most important factor when it's on the bottom of the list in order of importance. I counter this argument by saying that the phased evaluation has nothing to do with the order or importance.

Furthermore, "blockheads" may think that this phased approach isn't designed to achieve "best value" as someone could receive an excellent rating on all other factors, but be eliminated in the phased approach for lack luster past performance. In addition, it would be seem arbitrary to eliminate that offeror based on the least important factor and unreasonable....... I was dumbfounded by this. What say you?

I have found no GAO cases that support these thoughts......

My thought is that their could be a potential that someone that is rated as an "excellent" on the other factors but be eliminated via this phased approach because they were not determined to be sufficiently competitive. However, the purpose of this approach is to make a source selection more efficient when a large number of proposals is expected to be received. So this same argument could be said for any offerors eliminated using this approach.

FARmer, you're right.

For example, in Phase One, you might say that price and past performance are of relatively equal importance.

In Phase Two, you might say that technical approach, price, and past performance are in descending order of importance.

Yes, there are a lot of "blockheads" (to use your term) in the process.