Immediate Comparative Analysis for Fair Opportunity Selection

Are you familiar with "Even Swaps"? https://hbr.org/1998/03/even-swaps-a-rational-method-for-making-trade-offs

Make sure you read "Ben Franklin's Moral or Prudential Algebra" that is linked in the article.

There's no need to evaluate each factor on a scale to use these methods.

https://www.dhs.gov/sites/default/files/publications/pil_boot_camp_workbook.pdf

Technique 5 in the DHS Procurement Innovation Lab Boot Camp Workbook (link above) is on comparative evaluation -- and Technique 4 is on down-select.  You can do a down-select on a single non-price factor (or a couple of non-price factors), and you can make your down-select decision on a comparative evaluation basis (no adjectival ratings).

One of the key messages from the DHS procurement innovation community is to "stay in your lane" and to keep FAR subpart 15.3 procedures out of your fair opportunity considerations.  DHS is also encouraging streamlined evaluation and selection documentation, and has won GAO bid protest decisions where unsuccessful offerors say the evaluation and selection documentation are too short.

From p. 14 T/F table titled "Fair Opportunity/Orders Under Multiple-Award Contracts"

Quote

A notice/solicitation for a competitive order in a fair opportunity consideration less than $5.5 Million must list the relative order of importance of the evaluation factors

Some people would call that sloppy.

I actually like the construction — it combines both “notice” which is what it really is/what the FAR calls it and “solicitation” which is what some people think of it as when they’re in a hurry, so everyone is happy and no one is offended.  It helps reinforce correct principles, and is a good compromise for a document where the purpose is meaningful learning rather than pedantry.

ji20874 said:

I actually like the construction — it combines both “notice” which is what it really is/what the FAR calls it and “solicitation” which is what some people think of it as when they’re in a hurry, so everyone is happy and no one is offended.  It helps reinforce correct principles, and is a good compromise for a document where the purpose is meaningful learning rather than pedantry.

Not surprised you like it--aren't you one of the PIL team members?

In any case, I didn't say I didn't like it. It's just someone who wrote this in another thread:

Quote

A fair opportunity notice is not a solicitation within the construct of the FAR, even though it may be informally (sloppily?) referred to as such within our community.

A pedant like that might take issue with your document.

Don Mansfield said:

There's no need to evaluate each factor on a scale to use these methods.

 I understand, Don.  It seems the "even swaps" method would work better using rankings as the HBR article states.  I like the ranking or scaling method for this approach.  However, I know a lot of contracts managers are averse to the idea of ranking.  Why do you think this is?  Perhaps it is equated with numeric scoring, which has become a bit of an anathema.  For example, no one wants to explain to the GAO why an offeror lost to an awardee by a single point.  It seems to me we rank offerors regardless as evaluators and selection authorities, if in no other way, in our minds as we are performing the evaluation.  The question is whether we document it as part of the file, making it something subject to discovery.  If we don't rank, then we have to document our evaluation narratively, which some might consider a more difficult exercise, leading to something less definitive and more abstract.

Ibn Battuta said:

Do you think you can't do "immediate comparisons" when conducting a source selection under FAR Subpart 15.3?

No.

ji20874 said:

Technique 5 in the DHS Procurement Innovation Lab Boot Camp Workbook (link above) is on comparative evaluation -- and Technique 4 is on down-select.  You can do a down-select on a single non-price factor (or a couple of non-price factors), and you can make your down-select decision on a comparative evaluation basis (no adjectival ratings).

Some contracts managers believe that we have to document risks and benefits as part of every evaluation, something pushed by our attorneys, as well. Are the attorneys at your agency encouraged to attend these seminars? I appreciate the innovation lab and the work it does.  Unfortunately, I think the continued aversion to risk and unwillingness to adopt the techniques the lab suggests still emanate from the torso down.  Some view the concept of innovation as the innovation lab having provided them with a finite list of techniques. FAR 16.5's use of the term, "broad discretion," allows for much more, i.e., the lab is but a starting point and does not preclude the application of further innovation and other techniques not therein described.  A division will adopt an innovative approach, but then becomes reliant on only that one approach.  Contracts managers become unwilling to flex and further adapt that approach.  Program offices become lazy, recycling the same evaluation method time and again as if it is a one-size fits all approach to selection.  It can be rather stressful for the thinkers and creative types in the 1102 community who do not want to be relegated to factory work.  Those for whom employee retention is a concern somehow remain baffled over the attrition rates.  The question for the practitioner is whether it is better around the bend.

The workbook shows a way to do a comparative evaluation. It neither pretends nor presumes to show the only way.  The workbook is not a policy document; rather, it is an aid to practice (but everything in it conforms to DHS policy requirements).

PIL Boot Camps are for contracting officers, program managers, and procurement attorneys.  Everything is legal and honorable, and is offered for those who are looking to innovate.

If you like the technique, feel free to use it or let it inform your own approach.  Afterwards, share your story with your colleagues so others can benefit from your learning.

FAR 15.305 speaks of evaluating offers and FAR 15.308 speaks of a comparative assessment, both as part of the formal 15.3 source selection process.  Generally, in formal 15.3 source selections, adjectival ratings are assigned in the evaluation process before the comparative assessment occurs.  

In contrast, comparative evaluation is a practice to skip the adjectival ratings step completely.

I love the fact that a comparative evaluation used in phase one of a multi-phase source selection has serious mathematics hiding just below the surface. 

Bottom Line: Don't try comparative evaluations if you expect to get a lot of proposals.  Its a much more challenging math problem that it may first appear.  

Its a special case of what I think its called a comparison sorting algorithm (note: I like math, but am no mathematician) that is known in math and computer science.   When the numbers are low, just basic math required to figure out the number of comparisons necessary.  But as the number increases, this problem gets exponentially more difficult.  Quickly goes from intuitive to computationally hard to God-level AI needed.

Number To Be Compared & Sorted

Number of Highest Rated You Need To Know

Who Can Do The Math To Figure Out How Many Comparisons Are Needed

3

1

Child

5

2

Educated Adult

8

2

Well-Educated Adult

15

5

Computer Scientist

30

5

PhD in Combinatorics

100

10

Von Neumann + Quantum Supercomputer

1000

50

Deep Thought

Ibn Battuta said:

So, for example, if evaluating past performance you would make findings about each offeror's past and then compare their pasts directly without first converting your findings to ratings and then comparing offerors on the basis of the ratings. Is that the idea?

Yes.  My only caution would be not to put too much formality on the word "findings."

For example, for the past performance factor in a selection process, we might say that "A" is better than "B" because the CPARS history for "A" shows 100 satisfactory or better ratings and zero marginal or unacceptable ratings, of which 80 for were similar services, while the history for "B" shows zero satisfactory ratings and 14 marginal or unsatisfactory ratings, of which all 14 were for similar services.

I believe comparative evaluations can be done subjectively, without point scores, mathematics, or algorithms.

I would aver that I decided subjectively that "A" is better than "B" based on the information available to me. 

Then, assuming the other factors were performance, delivery, and price, and "A" was better for past performance and price and "B" was better for performance and delivery, then I would do a subjective tradeoff to select the successful firm.

There is an old thread on comparative evaluation at--

I know we all know where to find this but as this thread wraps itself around the axle I figured it was good to post it as a reminder.  No comparison just "Fair Opportunity" to be considered!  Or in other words do not give someone an unfair advantage to be considered but once you have all to be considered then just figure out who you are going to award to (consider them), document why you are awarding to one over the others and do it.  In my view the discussioin complicates it just like the OP's higher ups do.   

(Emphasis added)

FAR 16.505  "...(b) Orders under multiple-award contracts—

(1) Fair opportunity.

(i) The contracting officer must provide each awardee a fair opportunity to be considered for each order exceeding $3,500 issued under multiple delivery-order contracts or multiple task-order contracts, except as provided for in paragraph (b)(2) of this section.

                (ii) The contracting officer may exercise broad discretion in developing appropriate order placement procedures. The contracting officer should keep submission requirements to a minimum. Contracting officers may use streamlined procedures, including oral presentations. If the order does not exceed the simplified acquisition threshold, the contracting officer need not contact each of the multiple awardees under the contract before selecting an order awardee if the contracting officer has information available to ensure that each awardee is provided a fair opportunity to be considered for each order. The competition requirements in part  6 and the policies in subpart  15.3 do not apply to the ordering process. However, the contracting officer must—

                     (A) Develop placement procedures that will provide each awardee a fair opportunity to be considered for each order and that reflect the requirement and other aspects of the contracting environment;

                     (B) Not use any method (such as allocation or designation of any preferred awardee) that would not result in fair consideration being given to all awardees prior to placing each order;

                     (C) Tailor the procedures to each acquisition;

                     (D) Include the procedures in the solicitation and the contract; and

                     (E) Consider price or cost under each order as one of the factors in the selection decision.

                (iii) Orders exceeding the simplified acquisition threshold.

(A) Each order exceeding the simplified acquisition threshold shall be placed on a competitive basis in accordance with paragraph (b)(1)(iii)(B) of this section, unless supported by a written determination that one of the circumstances described at 16.505(b)(2)(i) applies to the order and the requirement is waived on the basis of a justification that is prepared in accordance with 16.505(b)(2)(ii)(B);

                     (B) The contracting officer shall—

                          (1) Provide a fair notice of the intent to make a purchase, including a clear description of the supplies to be delivered or the services to be performed and the basis upon which the selection will be made to all contractors offering the required supplies or services under the multiple-award contract; and

                          (2) Afford all contractors responding to the notice a fair opportunity to submit an offer and have that offer fairly considered.

..."

A read here is probably in order as well /legacy/reg/6f91c03d66aa4a89.html

Carl,

FAR 16.505(b)(1)(iv) speaks of offers and factors and evaluation and relative order and best value — how does one simply consider and then choose?  Can you sketch out a strawman of the process you envision?  If it is simpler and better than my practice, I’ll adopt it.

Or, are you stopping short of 16.505(b)(1)(iv) and only covering up to (iii)?  That’s okay if that is your thought process, as (iii) is different ftom (iv) (for a reason, right?) and (ii) is even different from (iii).

Seems pedantic to distinguish between "decided subjectively" and "found".

ibn battuta (1),

• I didn't want to put too much emphasis on the word "facts," because the process works just as well for facts as it does with other information.  You spoke of a finding based on facts, and it could be an observation based on other information.  The comparative evaluation process doesn't require facts -- it requires information, and that information sometimes will be factual and sometimes might not be, and it may be judged objectively or subjectively as appropriate.  I wouldn't want a simple-minded reader to go through this thread and think that "facts" are required for a comparative evaluation and a selection (the formal 15.3 source selection process also doesn't require facts).  So rather than saying that one vendor is better than another based on facts, I prefer to say that I subjectively decided one vendor was better than another based on the information available to me.

don,

• You are not contributing to this thread's discussion.

Ibn battuta (2),

• The "notice" versus "solicitation" discussion was another thread, not this one.  Many do use "solicitation," but some use "notice" consistent with the FAR usage.  But I agree that the fair opportunity process was initially intended to be a matter of post-award administration, and we* have turned it into a pre-award process.
  *not you and me personally, but our community generally

guardian (original poster),

• You can be as innovative as you and your organizational culture will allow.  Many organizations are getting better at staying true to the FAR and leaving 15.3 methodology out of their ordering situations.  Professional dialogue is important.  Maybe you can convince your superiors and reviewers to let you do a comparative evaluation on a fair opportunity below the protest threshold?  Your success there might help start a thaw in your organization.

Facts are nice, and they should be used wherever they are available.  But often, we have to make acquisition decisions based on information that may fall short of being factual.  I'm okay with that.

When one person quotes someone else, the quoter should take care not to faithfully convey the essential thought and to avoid skewing the quoted person’s thought.  I wrote, “The comparative evaluation process doesn't require facts -- it requires information, and that information sometimes will be factual and sometimes might not be...” (emphasis added on my text that you dropped in your quote, but that essential to faithfully convey my thought).

Okay.

ji20874 said:

don,

• You are not contributing to this thread's discussion.

Is that a fact?

Apparently, there may be more than one definition of "fact."

Do you have anything to contribute to help the original poster with comparative evaluation?

Ibn Battuta said:

That was written almost 14 years ago, and it apparently hasn't much good given some of the threads at Wifcon.

A group of the people at the company I work for was tasked to facilitate and oversee proposal evaluations.  We did training and then watched evaluators.  We kept telling them to stop scoring first and then doing narrative supporting the score.  We could see evaluators frustrations because we kept stopping them - “focus on identifying attributes first.”  They complained that’s not the way they do it.  One morning we showed up and the door was locked.   We called the CO and he promised to straighten it out.  The next morning the building was dark and the evaluators moved.  Our engagement ended.

ji20874 said:

Apparently, there may be more than one definition of "fact."

Yes, I've heard of alternative facts. Sometimes people contribute them in this forum.

@ji20874,

Might an evaluators confidence rating be an example of what you’re talking about?  I can see that subjectivity plays into assessing how well an offeror understands the requirement, the degree of soundness in a technical approach, and the likelihood of success.

ji20874 said:

Can you sketch out a strawman of the process you envision?

– My sole attempt, take it or leave it. 

1)Multiple award IDIQ – Value exceeds the SAT.  For parent contracts you pick any evaluation factors to place contractors with a parent contract that you want but the factors are to embrace the imperatives of FAR 15.304.

2)Fair Opportunity – The FAR guiding principles of 16.505 are applied but nothing from the FAR rules the contract imperatives rule.

3)Proposed Fair Opportunity Placement Procedures Clause –

Each contractor will be given a fair notice of the need for a task order under the contract.  The notice shall contain the description of work, a response period of not less than 10 days with the length of response period beyond 10 days determined per task order. 

Factors for selection will be stated in each notice.  Factors included in every notice will be - price, past performance on previous task orders including the sub factors of quality of work, timeliness of work and cost control (if applicable), and other orders currently awarded to the contractor.  Additional evaluation factors and sub factors may be included by the government as they relate to the specific work anticipated.  All factors and associated sub factors will be stated in their relative order of importance in the notice.  Oral presentations may be requested as appropriate at the discretion of the Government.

Contractors submitting a response to a notice of fair opportunity shall provide pricing or cost information along with information that addresses the other evaluation factors.  The information provided for price/cost and other factors shall be that minimally necessary for evaluation.  The Government reserves the right to consider any information available to the Contracting Officer in addition to that provided by the contractor.

Selection for award will be on the basis of a basic comparison of each response received to the other responses received with regard to the factors stated in the notice.  A debriefing will be afforded on all award decisions that are more than $5,000,000.

4. The internal process?  Do a comparison any old way that you want but document it in accordance with

“16.505 (b)(7) Decision documentation for orders.

(i) The contracting officer shall document in the contract file the rationale for placement and price of each order, including the basis for award and the rationale for any tradeoffs among cost or price and non-cost considerations in making the award decision. This documentation need not quantify the tradeoffs that led to the decision.”

5. Any old way you want?– First the caveats are - exercise broad discretion, streamlined procedures, policies in subpart  15.3 do not apply and formal evaluation plans or scoring is not required.  While the following may border on a “formal” effort  consider this approach from a March 16, 2017 post by Vern Edwards regarding evaluation of quotes understanding that we are talking here responses to fair opportunity notices so appropriately edited to fit the  fair opportunity construct..….

“Assume that we get four quotes, from companies A, B, C. and D. We'll evaluate using a method that is sometimes called "pairwise comparisons" and we'll assume transitivity.

First, we'll evaluate for experience. We'll compare A's description of its experience to B's and decide, subjectively, which has the better experience by taking note of asserted facts, identifying differences, determining their significance to us, and documenting our conclusions. Let's say we decide that A is better than B. We'll then compare A to C. This time we think C is better than A. Since C is better than A and A is better than B, we assume that C is also better than B. We'll then compare C to D. We decide that D is better than C. Since D is better than C and C is better than A and B, D is also beter than A and B. So D is best on experience. Since there were four offerors, and since D is best, we'll give D four points. Since C is better than A and B we'll give C three points. Since A is better than B we'll give A two points. Finally, we'll give B one point.

Experience: D = 4, C = 3, A = 2. and B = 1.

Second, we'll evaluate for past performance, using the same procedure as we did for experience. This time the result is as follows:

Past performance: D = 4, A = 3, B = 2, and C = 1.

Third, we now compare the four quoters' prices. This is easy. The lowest price gets four points and the highest gets 1 point. The result is:

Price: B = 4, A = 3, C = 2, and D = 1.

Fourth****_,_ we total the points.

A = 8, B = 7, C = 6, D - 9.

Fifth****_, D_ is best overall, so we award to D.

We evaluated on the basis of direct comparisons, based on subjective assessments and without standards. We made no tradeoffs, so we have none to document.

That's one way to do it. There are other ways. Is it a good way to do it? That depends on what you're buying and on your notions of value.

The hardest part is writing up the rationale for your subjective assessments. It takes some thinking and word-smithing. If you don't know how to do that, then don't try this method.

Do you want to weight the factors differently? If so, assign each a decimal weight, such as 0.5 for experience, 0.4 for past performance, and 0.1 for price, so that they add up to 1.0. Then multiply the points by the weights before totaling the points.

The fact that you and your colleagues have not heard of this approach should not be surprising. Government agencies make millions of simplified acquisitions every single fiscal year. Millions. No one knows, much less keeps track of, how they do them. They're rarely protested or the subject of explanatory articles. You've worked for the government for five and one-half years and in two agencies. In the words of BoB Dylan, your experience "is limited and underfed." Your lack of experience, knowledge, or passing familiarity with the method does not make the method nontraditional.”

Are there any real efficiencies? Compared with what other methods? What I described might take a skilled buyer no more than a couple of days so to complete, depending on what you're buying and assuming that the buyer can write and won't have to go to the boss to ask questions every hour or so. That's longer than a price-only approach but shorter than a tradeoff process type approach. Forget efficiency if you're going to create an "evaluation board."

Risk? There is no appreciable risk for any but the brain dead. Less risk than a tradeoff approach."

On ‎1‎/‎16‎/‎2020 at 4:26 PM, ji20874 said:

https://www.dhs.gov/sites/default/files/publications/pil_boot_camp_workbook.pdf

Technique 5 in the DHS Procurement Innovation Lab Boot Camp Workbook (link above) is on comparative evaluation -- and Technique 4 is on down-select.  You can do a down-select on a single non-price factor (or a couple of non-price factors), and you can make your down-select decision on a comparative evaluation basis (no adjectival ratings).

JI, Innovation Technique 5, Comparative Analysis, states "Probably more suited to acquisitions with a few quotes and a few evaluation factors."  The point of Phase One, a downselect, is to narrow a larger pool of offerors down to a more manageable pool, i,e, best qualified_._  Based on our market research, I can conjecture as to how many task order proposals we might receive.  However, I cannot say with any certainty.  I have to presuppose that all contractors under the socio-economic category for which we are setting our FOPR aside might submit task order proposals.  It should be noted that there will be a page limit placed on their responses to Phase One.  Given that we will evaluate only one non-price factor, while also considering price, we have met the second condition of the above statement, which is "Probably more suited to acquisition with...a few evaluation factors." 

During Phase Two, we will also apply a comparative analysis.  In this subsequent phase, both criterium recommended above will be met, that is, less offerors (only the most qualified) and few evaluation criteria (we will only evaluate technical approach in Phase Two).  Per the PIL's recommendation, an immediate comparative analysis seems more appropriate in Phase Two when we are guaranteed less offerors to evaluate.  However, we are proposing to apply an immediate comparative analysis in both phases one and two.

Given that we do not know how many offerors will submit responses to Phase One, might there be a better approach than immediate comparative analysis to incorporate into the downselect phase, Phase One?

On ‎1‎/‎18‎/‎2020 at 12:54 AM, C Culham said:

First, we'll evaluate for experience. We'll compare A's description of its experience to B's and decide, subjectively, which has the better experience by taking note of asserted facts, identifying differences, determining their significance to us, and documenting our conclusions. Let's say we decide that A is better than B. We'll then compare A to C. This time we think C is better than A. Since C is better than A and A is better than B, we assume that C is also better than B. We'll then compare C to D. We decide that D is better than C. Since D is better than C and C is better than A and B, D is also beter than A and B. So D is best on experience. Since there were four offerors, and since D is best, we'll give D four points. Since C is better than A and B we'll give C three points. Since A is better than B we'll give A two points. Finally, we'll give B one point.

Experience: D = 4, C = 3, A = 2. and B = 1.

Second, we'll evaluate for past performance, using the same procedure as we did for experience. This time the result is as follows:

Past performance: D = 4, A = 3, B = 2, and C = 1.

Third, we now compare the four quoters' prices. This is easy. The lowest price gets four points and the highest gets 1 point. The result is:

Price: B = 4, A = 3, C = 2, and D = 1.

Fourth****_,_ we total the points.

A = 8, B = 7, C = 6, D - 9.

Fifth****_, D_ is best overall, so we award to D.

It is in the Government's best interest to apply a trade off approach for my agency's requirement.  The approach described above would not work well for our requirement as it does not apply the weighing found in a tradeoff.  Instead it consistently represents differences between offerors, be they marginal or wide, by a single point in every case.  For example, let's say Company D's experience is only slightly better than that of Company C.  However, Company C's experience is significantly better than Company A's.  In both comparison, we assign only a one point difference between the two.  The same shortcomings of this approach would manifest in past performance or any other non-price factor.  The sequential points approach plays out with even greater imbalance when we factor in our scores for price.  One company might receive a "4", whereas the next lowest priced company receives a "3".  The difference in between their prices might be less than $100 or a some near inconsequential amount in a multi-million dollar acquisition.  In fact, Offerors B, A and C could all be neck-in-neck on pricing, while D could be 100% more than the next lowest price.  However, as you can see from the hypothetical scenario above, D is selected for an award.  The net result might be that the Government ends up paying significantly more for an offeror that is only slightly better in the other factors.

Guardian said:

...we are proposing to apply an immediate comparative analysis in both phases one and two...

That’s fine.  The process may become cumbersome with too many offers or factors, but there is no defined cut-off.  I hope it works for you.

I wouldn’t try to use numerical points — just subjective comparative assessments with a tradeoff inherent in the process.  But that’s me.

On 1/17/2020 at 8:35 PM, formerfed said:

Might an evaluators confidence rating be an example of what you’re talking about?  I can see that subjectivity plays into assessing how well an offeror understands the requirement, the degree of soundness in a technical approach, and the likelihood of success.

Yes.  Really, most of our evaluations and selections are more subjective than objective, and I am okay with that.  A point scheme may seem objective (85 is better than 75, right?), but it might be just a veneer.  If a company got 40 out of 50 points for factor one, 30 out of 30 points for factor two, and 15 out of 20 points for factor three, for a total of 85 points, well, assigning 40 instead of 38 or 42 was probably a subjective process in the first place.   Truly objective evaluations and selections can be done, but I think most major acquisitions are really subjective (even if there is an objective veneer).  Really, I don’t like veneers — I much prefer honesty, so I am okay with admitting that there is real subjectivity in our evaluation and selection processes.  Subjectivity is not inherently bad — it can be professional, legitimate, honorable, and so forth.  Subjectivity is a positive value that acquisition professionals bring to the table.

Guardian said:

Given that we do not know how many offerors will submit responses to Phase One, might there be a better approach than immediate comparative analysis to incorporate into the downselect phase, Phase One?

Probably yes.  Phase One's purpose is to quickly screen for proposals that are worth evaluating in more detail during Phase Two.   So what we are looking for are:

• - Criteria that are pretty good at indicating high-value
• - Criteria that is relatively easy to evaluate - objective, minimal interpretation needed, this usually means quantitative data.
• - Proposal information that is relatively low-cost for offerors.
• Avoiding FAR 15 procedures, or the appearance of using FAR 15 procedures.

Pretty Good Screening Criteria I Have Actually Used At Least Once, And Did Not Regret Using

• Samples.   I had a requirement that involved, basically, scrubbing and QCing some messy and complicated data.   Our scientists made some fake data, with some subtle errors in it.  An evaluation factor was 'analyze this fake data and tell us what you found.'  If they don't find the errors - Fail.  If they find the errors - Meh.  If they find the errors and explain them - Good.   This took less than 10 minutes to evaluate per proposal.  Everyone involved (Program personnel and Offerors) liked it.

• Quasi-Past-Performance.  Criteria about offeror's experience and performance on similar work, but not FAR 15 Past Performance exactly.  PIL has something about this. 

• Offerors have some essential credentials, warrants, licenses, etc.  CMMI Level II.  FedRAMP.  SCIF.  Has appropriate State licenses, etc.  Pretty close to responsibility determination though, so be careful.

• Challenge Questions / Short Q&As.  Offeror's give brief (like really brief - 1/2 page,  a few slides, 10 min. presentation) pitches for how the offeror proposes to do the work.  This is much more subjective, but if done right, this can be very productive.  In my area, IT, we would probably ask about proposed architecture, platforms, software development methodology, etc.  Often our technical folks will immediately know they do or do not like a particular approach based on these answers. 

• Summary "Technical Approach."Give us the short version.  1 page.

• Key Personnel.  Sometimes in my area (IT), at the end of the day we are buying smart people that can code.  The rest is window dressing.  You may consider just asking for KP info at first.  But be careful about this approach, it has some serious down-sides and, generally,and is appropriate only for select circumstances (small & highly technical).

Ibn Battuta said:

What do you mean by subjective and objective?

I would draw your attention to FAR 1.108(a), Words and terms, which states, "Definitions in part  2 apply to the entire regulation unless specifically defined in another part, subpart, section, provision, or clause. Words or terms defined in a specific part, subpart, section, provision, or clause have that meaning when used in that part, subpart, section, provision, or clause. Undefined words retain their common dictionary meaning."

Moreover, I would point you to the link JI included, specifically the cited GAO case, AlliantCorps, LLC; B-417126; B417126.3; B417126.4; February 27, 2019 (see page 31 under the GAO Guide for Comparative Evaluations).  The GAO has rarely held contracting officers to a standard of "beyond reproach," or "perfection in how we document of our selection decisions."  In fact, such standards are antithetical to streamlining as contemplated by FASA and initiatives such as the DHS PIL.  I'm all for semantics and a good hardy debate, but rhetorical interchanges over the meaning of commonly used words border on an exercising of "picking fly feces out of pepper with boxing gloves."  That is an exercise, which I for one, wish not to engage. 

The GAO has set a standard affording KOs generous latitude.  In documenting a decision, we are to meet the minimum requirements as set forth within the solicitation's language (read AlliantCorps, LLC, above cited). Sometimes less is more.  I have see instances in which our attorneys ask us to create multiple layers documenting our evaluation decisions under fair opportunity.  Since they are equally worried about including something in the subsequent layer that was not in the previous one, it becomes an unnecessary and redundant exercise of cutting and pasting.  We are all human.  We do the best we can and if we read GAO precedent, then we know we worry far too much at the sake of efficiency and innovation.  Maybe it's our agency attorneys and others giving rise to such worry.  When an attorney gives his or her opinion, we should not be afraid to request the legal basis of that opinion.  Sometimes I get a reply of, "that is the way we have always done it.  I tend to shudder at such a response.  Every selection decision is going to be tainted with at least a teaspoon of subjectivity.  More often than not, the subjectivity is from the evaluation team and we as KOs do our best to minimize it.  I have spent a long time thinking about the perfect evaluation criteria.  To date, no one, regardless of age, experience or intellect, has been able to gift it to me.  I am not confident that day will ever come.  We must also bear in mind, that it is never a one-size-fits-all approach.  The GAO focuses on points of protest that would make a substantive difference in the results of an evaluation, i.e., that which might tip the scales in favor of another offeror.  That ought to continue to be our focus.

ibn battuta,

Hmmm, a few days ago, you refused to discuss definitions.  I'm glad you're more open now.  Below are some quotes from GAO bid protest decisions where GAO uses the word "subjective."  I think my use the word is the same as the GAO's.  I hope these are helpful to you in your continuous learning...

• The determination of the relative merits of proposals is the responsibility of the agency that solicited them, and requires weighing competing subjective considerations and exercising sound discretion...
• For the most part, the evaluative conclusions to which Price Waterhouse objects are precisely the type of subjective judgments reserved to contracting officials, not our Office...
• The evaluation of past performance, by its very nature, is subjective...
• The protester primarily challenges the differences in the findings between the SSEB and SSAC and contends that the SSA should have adopted the findings of the SSEB with regard to the evaluation of its proposal, rather than the SSAC’s evaluation, which the protester argued was “subjective” and unreasonable.
• Finally, the solicitation advised offerors that the agency intended to make the source selection decision without conducting discussions and noted that the best value evaluation is, in and of itself, a subjective assessment by the Government.”
• When conducting a best-value tradeoff analysis . . . an agency may not simply rely on the assigned adjectival ratings to determine which proposal offers the best value because evaluation scores–whether they are numerical scores, colors, or adjectival ratings–are merely guides to intelligent decision-making and often reflect the disparate, subjective judgments of the evaluators...

GAO testimony before a House subcommittee...

• Procuring agencies are obligated to conduct proposal evaluations in accordance with the evaluation scheme set forth in the solicitation. Such proposal evaluation judgments are by their nature often subjective; nevertheless, the exercise of these judgments in the evaluation of proposals must be reasonable and must bear a rational relationship to the announced criteria upon which the successful competitor is to be selected...

Guardian said:

I would draw your attention to FAR 1.108(a), Words and terms, which states, "Definitions in part  2 apply to the entire regulation unless specifically defined in another part, subpart, section, provision, or clause. Words or terms defined in a specific part, subpart, section, provision, or clause have that meaning when used in that part, subpart, section, provision, or clause. Undefined words retain their common dictionary meaning."

I

Guardian,

That convention would apply to the use of words or terms in the FAR. Ibn is asking ji what he means when he uses those words. I don't think ji has explained what he means--he merely provided examples of the use of "subjective". But his answer may satisfy Ibn.

But I did explain what I meant -- I just did it through illustration...

Don Mansfield said:

Guardian,

That convention would apply to the use of words or terms in the FAR. Ibn is asking ji what he means when he uses those words. I don't think ji has explained what he means--he merely provided examples of the use of "subjective". But his answer may satisfy Ibn.

Fair enough.

Ibn Battuta said:

Nor did I ask for your silly lecture or your unwarranted and vulgar criticism.

Ibn Battuta,

My dear father was a sergeant in the Army.  I believe this is a term he brought back from his enlistment which my brother and I heard countless time growing up, among many of his other favorite sayings and aphorisms from the service.  I am sorry, no harm or insult was intended  I chuckle every time I think of that saying, which in this case, I replaced with a euphemism.  As you can guess, it means focused on minor (perhaps trivial) details.  I never meant to come across as vulgar, nor subject you to a lecture.  This forum can get energetic and mildly sarky, as I'm sure you are aware; but it was not my goal to make you feel anything less than someone with whom I would want to share insights and take advice.

ibn battuta,

I’m glad my illustration was helpful.

Ibn Battuta said:

If you are satisfied with your response to my questions, then I am willing to let it go at that.

I am satisfied.  I don’t have a special definition, but follow ordinary usage.

Ibn Battuta said:

@ji20874

1. What do you mean by subjective and objective?
2. What makes an evaluation subjective versus objective?
3. What are the respective characteristics of subjective and objective evaluations?

A short answer from Jeffrey Glen at the Businessdictionary.com:

”Subjective refers to personal perspectives, feelings, or opinions entering the decision making process.[*]

Objective refers to the elimination of subjective perspectives and a process that is purely based on hard facts.”

*Some other characteristics of subjective thinking may include being influenced by perceptions or a “gut feel”, intuition, prejudices.

Other characteristic could be basing opinion of something or somebody upon your past  experiences - regardless of considerations for how a firm may have changed, addressed prior performance, improved, etc. Or - relying upon prior great experiences without regard to degradation of a person’s or company’s qualifications or recent less than stellar performance; “reading something into” a proposal;

Not all subjectivity is bad, but may be challenged and then would require justification that would survive the challenge. If one has current, ongoing experience with a person or company, irrespective of what is or isn’t specifically stated in a proposal (e.g., “past performance” and “current performance”), one might well use subjective reasoning to  form a Degree of confidence in that person or firm.

Not all persons or firms are great proposal writers but may be very reliable performers.

The opposite is also true. There are companies with flowery or great proposal writers that sometimes or often don’t or can’t  deliver. As as been said before , proposal writers write proposals to win jobs, while often, others who may or may not have even read or have been briefed, execute the work.

Plus, my experience with government performance evaluations is that they are often poorly or hastily written as an obligation rather than a reliable, valuable source of factual information.

Tired of writing, won’t go into a treatise on objective thinking or the mechanics of subjective and objective “evaluations” as that relates to the acquisition process.

Suffice it to say that there are dangers and possible pitfalls to relying solely upon objective or subjective thinking processes during evaluations of proposal submissions or any major decision making.

That’s why I like to negotiate,  to request clarifications, and to conduct discussions, either face to face or at least telephonically (with someone I know or have met, face to face).

Subjectively, I “know” that firms “”often”  don’t provide their best pricing in initial competitive proposals especially for design-build and construction contracts,. Firms have told me that.

I can often bargain for better performance, where the initial proposal meets the minimum Government requirements but is less than optimum or is not favorable to the stakeholders. We have also helped firms better understand aspects of the job that will help them improve efficiency, note mistakes or omissions  or improve performance to positively affect their bottom line earnings. Win win is the goal! .

General.Zhukov said:

Probably yes.  Phase One's purpose is to quickly screen for proposals that are worth evaluating in more detail during Phase Two.   So what we are looking for are:

• - Criteria that are pretty good at indicating high-value
• - Criteria that is relatively easy to evaluate - objective, minimal interpretation needed, this usually means quantitative data.
• - Proposal information that is relatively low-cost for offerors.
• Avoiding FAR 15 procedures, or the appearance of using FAR 15 procedures.

Pretty Good Screening Criteria I Have Actually Used At Least Once, And Did Not Regret Using

• Samples.   I had a requirement that involved, basically, scrubbing and QCing some messy and complicated data.   Our scientists made some fake data, with some subtle errors in it.  An evaluation factor was 'analyze this fake data and tell us what you found.'  If they don't find the errors - Fail.  If they find the errors - Meh.  If they find the errors and explain them - Good.   This took less than 10 minutes to evaluate per proposal.  Everyone involved (Program personnel and Offerors) liked it.

• Quasi-Past-Performance.  Criteria about offeror's experience and performance on similar work, but not FAR 15 Past Performance exactly.  PIL has something about this. 

• Offerors have some essential credentials, warrants, licenses, etc.  CMMI Level II.  FedRAMP.  SCIF.  Has appropriate State licenses, etc.  Pretty close to responsibility determination though, so be careful.

• Challenge Questions / Short Q&As.  Offeror's give brief (like really brief - 1/2 page,  a few slides, 10 min. presentation) pitches for how the offeror proposes to do the work.  This is much more subjective, but if done right, this can be very productive.  In my area, IT, we would probably ask about proposed architecture, platforms, software development methodology, etc.  Often our technical folks will immediately know they do or do not like a particular approach based on these answers. 

• Summary "Technical Approach."Give us the short version.  1 page.

• Key Personnel.  Sometimes in my area (IT), at the end of the day we are buying smart people that can code.  The rest is window dressing.  You may consider just asking for KP info at first.  But be careful about this approach, it has some serious down-sides and, generally,and is appropriate only for select circumstances (small & highly technical).

General, you offer a lot of good suggestions above for evaluation criteria one might use to conduct a downselect.  My original question is focused on how to support a selection decision through documentation after the Government has determined which criteria it will use to evaluate.  Our criteria for evaluation must always be stated in our solicitation.  However, the way we document our evaluation and selection does not necessarily have to be included in the solicitation language.  I have found that many decision authorities prefer to tell contractors a lot about how they will document their decisions.  Whether this is a good idea is arguable.  When was the last time anybody went on a job interview and the hiring manager and panel disclosed to the candidate exactly how they were going to document her evaluation behind the scenes?  Most of the time, she'd be lucky to get a call back.

In my office, we tend to take the stated criteria (for example, any of the criteria you suggested above) and apply it to each individual contractor performing a "thorough" evaluation.  The evaluation team then documents each contractor relative to that criteria.  We do that for each contractor, be it two or twenty-two.  We then take all that information and perform a comparative analysis.  Generally General, this involves comparing each contractor that will move on to Phase Two to each contractor that will not.  We tend to cut and paste our individual findings for each contractor and string those statements (findings) together using comparative language.  The following serves as a truncated example:

Phase 1 - Individual Evaluations of Experience

Contractor A

Contractor A has recent experience in three contracts (within the past three years) performing work largely comparable to that described in the SOW.

Contractor B

Contractor B has recent experience in one contract (within the past three years) performing work largely comparable to that described in the SOW.

Phase 2 - Comparative Evaluations of Experience

Contractor A is superior to Contract B in the area of Experience as Contractor A has recent experience in three contracts (within the past three years) performing work largely comparable to that described in the SOW versus Contractor B, which has recent experience in one contract (within the past three years) performing work largely comparable to that described in the SOW.

Granted, this is an overly general example short on specifics.  But it is an example of the model my office seems to favor.

The Individual Evaluations can run into double digit pages (in a Fair Opportunity selection) depending on how many offerors we are required to evaluate.  The comparative evaluations, which tend to regurgitate the same information found in the Individual Evaluations can also run into double digits page-wise.

The template in the PIL Boot Camp Workbook, specifically Technique 5, consolidates the above information into a single document, which it refers to as a "Comparative Evaluation."  I prefer this approach on the surface, as it provides a model and rhythm for moving through evaluations of multiple offerors with a level of efficiency and less documentation.

General, based on the criteria you suggested above, how would you document your evaluations? Would you not use a comparative analysis to downselect? How then might you otherwise document your downselect decision?

Guardian said:

General, you offer a lot of good suggestions above for evaluation criteria one might use to conduct a downselect.  My original question is focused on how to support a selection decision through documentation after the Government has determined which criteria it will use to evaluate.  Our criteria for evaluation must always be stated in our solicitation.  However, the way we document our evaluation and selection does not necessarily have to be included in the solicitation language.  I have found that many decision authorities prefer to tell contractors a lot about how they will document their decisions.  Whether this is a good idea is arguable.  When was the last time anybody went on a job interview and the hiring manager and panel disclosed to the candidate exactly how they were going to document her evaluation behind the scenes?  Most of the time, she'd be lucky to get a call back.

In my office, we tend to take the stated criteria (for example, any of the criteria you suggested above) and apply it to each individual contractor performing a "thorough" evaluation.  The evaluation team then documents each contractor relative to that criteria.  We do that for each contractor, be it two or twenty-two.  We then take all that information and perform a comparative analysis.  Generally General, this involves comparing each contractor that will move on to Phase Two to each contractor that will not.  We tend to cut and paste our individual findings for each contractor and string those statements (findings) together using comparative language.  The following serves as a truncated example:

Phase 1 - Individual Evaluations of Experience

Contractor A

Contractor A has recent experience in three contracts (within the past three years) performing work largely comparable to that described in the SOW.

Contractor B

Contractor B has recent experience in one contract (within the past three years) performing work largely comparable to that described in the SOW.

Phase 2 - Comparative Evaluations of Experience

Contractor A is superior to Contract B in the area of Experience as Contractor A has recent experience in three contracts (within the past three years) performing work largely comparable to that described in the SOW versus Contractor B, which has recent experience in one contract (within the past three years) performing work largely comparable to that described in the SOW.

Granted, this is an overly general example short on specifics.  But it is an example of the model my office seems to favor.

The Individual Evaluations can run into double digit pages (in a Fair Opportunity selection) depending on how many offerors we are required to evaluate.  The comparative evaluations, which tend to regurgitate the same information found in the Individual Evaluations can also run into double digits page-wise.

The template in the PIL Boot Camp Workbook, specifically Technique 5, consolidates the above information into a single document, which it refers to as a "Comparative Evaluation."  I prefer this approach on the surface, as it provides a model and rhythm for moving through evaluations of multiple offerors with a level of efficiency and less documentation.

General, based on the criteria you suggested above, how would you document your evaluations? Would you not use a comparative analysis to downselect? How then might you otherwise document your downselect decision?

Just addressing here one aspect of the criteria stated above - 

A firm may have more experience (what they did)  than another.  But how WELL did they perform - how successful were they in meeting the customers’ requirements and expectations?

Recent, relevant experience and past performance are related but separate aspects of evaluating and developing a confidence rating.

@Guardian

I waded through all the responses after my latest post a couple of days ago.  Understanding that the single approach I offered as just one of possibly tens of ways did not appear to meet your needs just be aware that I posted it in response to @ji's challenge to me and to not necessarily provide a process that met your needs.   In reading the rest of your posts I completely understand your thoughts and concerns.   I have a gut reaction to this entire thread and it is - Everyone gets too wrapped around the axle on what the comparative process needs to be.  They read the language and specific words of FAR 16.505 as if it is promoting FAR 15.3 procedures.  I take that other view as 16.505 says it is not.   Noting this I then seriously pose ...... 

Why couldn't one use my suggested "Fair Opportunity Placement Procedures Clause", get responses, lay them out on a table and read through them to compare them then sit down and write out an award decision rational to be signed by the CO that is made on a best value basis that states basis for award and the relative importance of quality (non-cost factor(s)) and price or cost factors and the tradeoffs made with out quantifying the tradeoffs made?  

After all the only thing that 16.505 requires is that I have to give everyone fair opportunity to be compared, how I estimate, measure, or note the similarity or dissimilarity between the responses I receive is completely up to my discretion.   Every day I believe we do this in a personal way with every decision we make to buy a car, paint a house, get a lawyer, pick a doctor, etc. etc. etc. and if someone were to require me to justify why I did get the car, etc. etc. I believe I could simply document or express the measure of my tradeoff decision without having to have some dang rating system to quantify it.

Ibn Battuta said:

If someone had asked me what I think the difference is between subjective and objective evaluation, I would have said that an objective evaluation is one that turns on the observable and measurable attributes of the thing being evaluated---the object of the evaluation---and not on the opinions of the person conducting the evaluation--the subject.

FWIW, I think that is a great and usable definition, which can easily be mapped to acquisition scenarios.

Subjective = I'm looking for intangibles; AKA, the "I'll know it when I see it" philosophy, which assumes that the proposal/quote will educate me on things or approaches of which I was not previously aware. 

Objective =  I know what constitutes superior performance before I ever look at a single proposal.

One of my concerns is that I have seen COs state that they are being "innovative" by having the prospective vendors provide oral presentations in response to silly on-the-spot challenges which have nothing to do with the scope of the effort ("contractor team will have 3 hours to develop a response to an example XXX scenario") .  Might as well have them engage in Feats of Strength.  Too often 1102s get wrapped up in everything except what actually matters.  "Lazy" and "innovative" are sometimes too close for comfort.

Ibn Battuta said:

source selection.

At the hazard of starting another big debate I can see the value of those readings as they apply to a selection of contractors for the multiple award but as they apply to application of fair opportunity (and this thread) I do not agree.  Anything beyond comparing and having the ability to write up the selection decision without quantifying the tradeoff is again akin to FAR 15.3 processes.

C Culham said:

At the hazard of starting another big debate I can see the value of those readings as they apply to a selection of contractors for the multiple award but as they apply to application of fair opportunity (and this thread) I do not agree.  Anything beyond comparing and having the ability to write up the selection decision without quantifying the tradeoff is again akin to FAR 15.3 processes.

Amen. And competitive  task order procedures are NOT source selections.

C Culham said:

At the hazard of starting another big debate I can see the value of those readings as they apply to a selection of contractors for the multiple award but as they apply to application of fair opportunity (and this thread) I do not agree.  Anything beyond comparing and having the ability to write up the selection decision without quantifying the tradeoff is again akin to FAR 15.3 processes.

joel hoffman said:

Amen. And competitive  task order procedures are NOT source selections.

Exactly.  Two things bother me about multiple award contracts.  This ordering process is one.  The other is awarding contracts to many, many sources.  IMO, the number of awards shouldn’t exceed the capability of providing around three competitive TO responses for each order need.  There are exceptions of course.

Ibn Battuta said:

pessimistic

Yes

Ibn Battuta said:

realistic.

Nope not on my watch because I lived to see the day that many things were called "pipe dreams" have become reality.  I for one believe change is in the hand of us all and we can make it happen.  It is a fact that has been proven that "pipe dreams" can become reality.

C Culham said:

Nope not on my watch because I lived to see the day that many things were called "pipe dreams" have become reality.  I for one believe change is in the hand of us all and we can make it happen.  It is a fact that has been proven that "pipe dreams" can become reality.

I hope you’re right as well.  What makes change happen in the government is rewarding those that try new things.  Recognize and set examples of the kind of behavior that’s wanted.  Let the rank and file see risk takers and innovators are those that get ahead.

Ibn Battuta said:

One question: What are you, personally, doing to make the pipe dream come true?

Participating in this Forum!

FWIW Virtual Acquisition Office did an Advisory on comparative evaluations a couple years ago. Those with a subscription may want to take a look to get a fresh perspective on the discussion.

Ibn:

Your wrote

Quote

You write a lot. Why not write an article about "fair opportunity" for NCMA's Contract Management magazine? They are eager for articles from 1102s, and you'll reach an audience that you might not reach here. It can just be an opinion piece.

I've just counted the reading statistics for the articles posted on Wifcon.com's Articles page for 2019.  Articles written nearly 2 decades ago and posted here are still being read thousands of times each year--including in 2019.

Atta-Ibn!  I have one more annual update and then I go back to writing for Wifcon.com.