Retaining Evaluators' Notes in Light of 44 U.S.C. 3301

I always considered considered everything a record.

It depends on whether the PCO (or the agency) (1) understands the law or (2) blindly obeys the agency's overly-cautious and not-respectful-of-case-law attorneys.  I esteem the GAO case law as correct, and PCOs who understand the law do not feel a need to automatically retain individual evaluator notes -- they understand that those notes might not be records*.  Overly-cautious and not-respectful-of-case-law attorneys (unfortunately, some agencies have them) will say every single scrap of paper is a record, and will even insist that evaluators create notes just so they can save them as records -- those attorneys are wrong, but no one can say they are wrong because they are attorneys.

*The source selection evaluation report, decision document, and so forth are records.  Individual evaluator notes are not records.

The most prudent and wisest practice in my opinion is preparing a consensus report and that should be the only document retained.  Too often an individual evaluator notes reflect misunderstandings and errors from their initial reviews.  When discussed in a group setting, evaluators often discover why their first hunch was wrong.  What matters is the consensus report that all evaluator acknowledge as the group opinions (however individuals can still express their disagreement if they like as dissenting opinions).  

The problem with retaining notes is a protestors attorney may find misleading statements, confusion, and misunderstandings in the discovery process which don’t reflect the evaluators final opinion.  

Besides when a PCO uses on-the-spot consensus, which is a favorite of mine, individual evaluators worksheets don’t usually exist.

On 8/7/2023 at 4:27 PM, formerfed said:

The most prudent and wisest practice in my opinion is preparing a consensus report and that should be the only document retained.  Too often an individual evaluators notes reflect misunderstandings and errors from their initial reviews.  When discussed in a group setting, evaluators often discover why their first hunch was wrong.  What matters is the consensus report that all evaluator acknowledge as the group opinions (however individuals can still express their disagreement if they like as dissenting opinions).  

The problem with retaining notes is a protestors attorney may find misleading statements, confusion, and misunderstandings in the discovery process which don’t reflect the evaluators final opinion.  

Besides when a PCO uses on-the-spot consensus, which is a favorite of mine, individual evaluators worksheets don’t usually exist.

Exactly!! (Regarding individual evaluation notes and consensus evaluations).

In fact that is what I taught, when teaching source selection procedures to various classes. I learned early on not to have or allow the members to individually assign ratings to the various factors/subfactors. Some initial ratings often bore little resemblance to the final ratings anyway.

I was part of the subject of one protest of a services contract source selection, circa 2005, as the Chairperson of the SSEB. After I wrote the official consensus evaluation report, I threw away the individual evaluators’ pre-group notes.   

On 8/7/2023 at 4:27 PM, formerfed said:

The problem with retaining notes is a protestors attorney may find misleading statements, confusion, and misunderstandings in the discovery process which don’t reflect the evaluators final opinion.

This is exactly what the protestor’s attorney was looking for. I was involved in one previous GSBCA protest where that tactic was used - to use individual notes to twist and confuse the facts of the official, consensus evaluation. 

Steve Feldman was our attorney handling this later protest. He tried to rake me over the coals for not retaining the notes of the  individual, initial evaluations. These notes were basically the individual, initial thoughts after hastily reading the proposals but before the group discussion and official, consensus evaluation.

The KO and I explained to Steve that several of the evaluators changed their opinions after the notes were discussed in the group, admitting that they either didn’t have a good understanding of those aspects of the proposal or had misunderstood the proposal and that the lawyers would only try to twist the evaluation.

We reminded Steve that the RFP solicitation clearly stated that the evaluation and subsequent ratings of proposals would be based upon the SSEB’s consensus evaluation and that we would document any dissent from the conclusion- of which there was none here.

Steve Feldman then changed his mind and strongly supported us in the government’s reply.  Steve convinced the GAO’s attorney. EDIT: The GAO denied that aspect of the protest  and every other basis of the protest. The PCO and I spent a Saturday with Steve formulating the Government response to the Protest.

ji20874 said:

The source selection evaluation report, decision document, and so forth are records.  Individual evaluator notes are not records.

Are you going to support your assertion or just insist?

Don Mansfield said:

Are you going to support your assertion or just insist?

Don Mansfield said:

I always considered considered everything a record.

Well, Don- are you going to support your assertion or opinion?

Why are individual notes made, before the official group consensus evaluation,  a “record” of the source selection evaluation? If so, then why shouldn’t the give and take discussions of each source selection team during the official, group consensus evaluations and reviews, if any by other teams, be recorded and preserved?

Because they are all pre-decision preliminary notes and opinions.

joel hoffman said:

Well, Don- are you going to support your assertion or opinion?

Why are notes made before the official evaluation a “record” of the source selection evaluation? If so, then why shouldn’t the give and take discussions of each source selection team during the official, group consensus evaluations and reviews, if any by other teams, be recorded and preserved?

@joel hoffman, I related what I did--I didn't make any claims about my knowledge of case law or how much smarter I was than my agency attorneys.

But since you are asking, here you go:

Quote

[T]he destruction of the individual TEP [technical evaluation panel] members' score sheets is barred by the FAR provisions. The current contract file for the challenged procurement does not "constitute a complete history of the transaction," FAR § 4.801(b) (emphasis added), nor does it “[f]urnish[] essential facts in the event of litigation. FAR § 4.801(b)(4). FAR § 4.801(b) expressly refers to § 4.803, which provides "examples of the records normally contained … in contract files." FAR § 4.803. Specifically, the record as submitted does not contain all "[s]ource selection documentation," as required by FAR § 4.803(a)(13). It was foreseeable that the individual rating sheets could well become relevant to issues arising in a bid protest, particularly in a situation where, as here, the bias of one or more of the panel members is alleged. No preturnatural clairvoyance would be required to envision that possibility. Although the ratings of the individual members of the TEP presumably were taken into account by, and wrapped into, the consensus report of the TEP, without the separate score sheets of the individual panel members, the court is unable to assess any divergence in the ratings which produced that consensus, or in turn, determine whether there existed personal bias in favor of [the winning contractors] on the part of one or more of the panel members. The argument by the government and the intervenor that the individual members' rating sheets were in effect no more than drafts of the final consensus report of the [TEP] is not supportable. The consensus report necessarily represented an amalgam of the views of the panel members and would have tended to suppress individual views.

Pitney Bowes Gov't Solutions, Inc. v. United States, 93 Fed. Cl. 327 (2010)

"Individual evaluator notes" is undefined. They may or may not be the types of records required to be retained by the FAR. It all depends on what information they contain. A broad assertion that "individual evaluator notes" are, in fact, not records demonstrates a profound ignorance of case law.

Don Mansfield said:

@joel hoffman, I related what I did--I didn't make any claims about my knowledge of case law or how much smarter I was than my agency attorneys.

But since you are asking, here you go:

Pitney Bowes Gov't Solutions, Inc. v. United States, 93 Fed. Cl. 327 (2010)

"Individual evaluator notes" is undefined. They may or may not be the types of records required to be retained by the FAR. It all depends on what information they contain. A broad assertion that "individual evaluator notes" are, in fact, not records demonstrates a profound ignorance of case law.

That’s consistent with my prohibition on individual “scoring” (numbers) or “ratings” (numbers or adjectives). The ratings and the underlying basis for the ratings, which precedes the ratings are done by the group in a consensus…as clearly stated in the Solicitation.

Last night, I received a note on this topic.

Before that, the power went out here and was restored this morning.

This morning, the first thing I did was add Federal Acquisition Circular 2023-05.  

I have locked this topic until I can look at it.

I edited my OP to try to get to the bottom of this.

A broad assertion by an acquisition professional that he or she "always considered considered (sic.) everything a record" seems to me to demonstrate a profound ignorance of case law and fundamental correct principles.  That's my opinion.

But clearly, some in our community (like Don) do indeed consider every scrap of paper, including individual evaluator notes, to be a record.  Others (like every other poster in this thread) do not. 

Each practitioner needs to decide for him- or herself, based on the facts and within whatever leeway is allowed by his or her agency.  There is no universally-agreed-upon answer.

Voyager said:

I edited my OP to try to get to the bottom of this.

First, like others that have responded my usual practice was that the score sheets were retained otherwise notes etc were either tossed or redacted.   My recollection was that this was based on "training".  I did research training curriculum the best I could but could not place my finger on the anything concrete.

I am not sure Don's reference with regard to the FAR answers the 44 USC question.  Yet, it was interesting and begs further research on my part.

Finally in my following of the discussion my thought turned to what is the implementing regulation for the statute.   36 CFR 1222 seems to be it.  Not sure if this helps or not but there is this excerpt that seems to apply....

36 CFR 1222.12(c)

  "Working files and similar materials. Working files, such as preliminary drafts and rough notes, and other similar materials, are records that must be maintained to ensure adequate and proper documentation if:

(1) They were circulated or made available to employees, other than the creator, for official purposes such as approval, comment, action, recommendation, follow-up, or to communicate with agency staff about agency business; and

(2) They contain unique information, such as substantive annotations or comments that adds to a proper understanding of the agency's formulation and execution of basic policies, decisions, actions, or responsibilities."

I no longer participate at Wifcon and I don't read the Forum anymore, but someone called me about this thread and, having read it, I have a few things to say.

Here is the OP's original question:

On 8/7/2023 at 2:53 PM, Voyager said:

Established case law from GAO says this about source selection evaluators' notes: "[T]here is no requirement that an agency retain individual evaluator’s notes or worksheets, provided the agency’s final evaluation documentation reasonably explains the basis for the agency’s judgments." (Government Acquisitions, Inc., B-401048; B-401048.2; B-401048.3, May 4, 2009)

44 U.S.C. 3301, however, requires agencies to retain all "records", which are defined and trained to very broadly include documents that "contain informational value as evidence of your organization's...decisions," or even just, "An original document related to agency business that does not exist elsewhere."

Which of these do PCOs follow when deciding what evaluation documentation to retain in the official source selection file?

Emphasis added.

The question is based on false assumptions. There is no conflict between GAO case law and 44 USC 3301 with respect to retention of records, and there is no choice between compliance with the GAO case law and the statute in this matter. The proper response to the question asked  is that COs must always comply with the statute. As I hope to show, below, they will have no problems under GAO's case law if they do, because the case law is just an evidentiary policy and is less demanding than the statute. A CO might be okay under GAO case law, but not under a federal court's jurisdiction if they did not comply with statute.

Now, with respect to 44 USC 3301, we must ask ourselves some questions

1. What is a record?

2. What records must agencies make?

3. What records must agencies keep?

WHAT IS A RECORD?

According to 44 USC 3301:

Quote

(a) Records Defined.-

(1) In general.-As used in this chapter, the term "records"-

(A) includes all recorded information, regardless of form or characteristics, made or received by a Federal agency under Federal law or in connection with the transaction of public business and preserved or appropriate for preservation by that agency or its legitimate successor as evidence of the organization, functions, policies, decisions, procedures, operations, or other activities of the United States Government or because of the informational value of data in them; and

(B) does not include-

(i) library and museum material made or acquired and preserved solely for reference or exhibition purposes; or

(ii) duplicate copies of records preserved only for convenience.

(2) Recorded information defined.-For purposes of paragraph (1), the term "recorded information" includes all traditional forms of records, regardless of physical form or characteristics, including information created, manipulated, communicated, or stored in digital or electronic form.

That's pretty broad. The National Archives and Records Administration (NARA) has specified its implementation of the statute in Title 36 of the Code of Federal Regulations, Chapter XII, National Archives and Records Administration, Subchapter B, Records Management. According to the regulation, all Federal agencies must comply with Subchapter B.

NARA elaborates on the statutory definition or record in 36 CFR 1220.18, What definitions apply to the regulations in Subchapter B?

Quote

Records or Federal records is defined in 44 USC 3301 as including “all books, papers, maps, photographs, machine readable materials, or other documentary materials, regardless of physical form or characteristics, made or received by an agency of the United States Government under Federal law or in connection with the transaction of public business and preserved or appropriate for preservation by that agency or its legitimate successor as evidence of the organization, functions, policies, decisions, procedures, operations or other activities of the Government or because of the informational value of the data in them 44 USC 3301.” (See also §1220.10 of this part for an explanation of this definition).

And see 36 CFR 1220.18:

Quote

Adequate and proper documentation means a record of the conduct of Government business that is complete and accurate to the extent required to document the organization, functions, policies, decisions, procedures, and essential transactions of the agency and that is designed to furnish the information necessary to protect the legal and financial rights of the Government and of persons directly affected by the agency's activities.

***

Documentary materials is a collective term that refers to recorded information, regardless of the medium or the method or circumstances of recording.

Emphasis added.

See further, 36 CFR 1222.10, How should agencies apply the statutory definition of Federal records?

Quote

(a) The statutory definition of Federal records is contained in 44 U.S.C. 3301 and provided in § 1220.18 of this subchapter.

(b) Several key terms, phrases, and concepts in the statutory definition of a Federal record are further explained as follows:

(1) Documentary materials has the meaning provided in § 1220.18 of this subchapter.

(2) Regardless of physical form or characteristics means that the medium may be paper, film, disk, or other physical type or form; and that the method of recording may be manual, mechanical, photographic, electronic, or any other combination of these or other technologies.

(3) Made means the act of creating and recording information by agency personnel in the course of their official duties, regardless of the method(s) or the medium involved. [Emphasis added.]

(4) Received means the acceptance or collection of documentary materials by or on behalf of an agency or agency personnel in the course of their official duties regardless of their origin (for example, other units of their agency, private citizens, public officials, other agencies, contractors, Government grantees) and regardless of how transmitted (in person or by messenger, mail, electronic means, or by any other method). In this context, the term does not refer to misdirected materials. It may or may not refer to loaned or seized materials depending on the conditions under which such materials came into agency custody or were used by the agency. Advice of legal counsel should be sought regarding the “record” status of loaned or seized materials.

(5) Preserved means the filing, storing, or any other method of systematically maintaining documentary materials in any medium by the agency. This term covers materials not only actually filed or otherwise systematically maintained but also those temporarily removed from existing filing systems.

(6) Appropriate for preservation means documentary materials made or received which, in the judgment of the agency, should be filed, stored, or otherwise systematically maintained by an agency because of the evidence of agency activities or information they contain, even if the materials are not covered by its current filing or maintenance procedures.

And see 36 CFR 1222.12, What types of documentary materials are Federal records?

Quote

(a) General. To ensure that complete and accurate records are made and retained in the Federal Government, agencies must distinguish between records and nonrecord materials by applying the definition of records (see 44 U.S.C. 3301 and 36 CFR 1220.18 and 1222.10 of this subchapter) to agency documentary materials in all formats and media.

(b) Record status. Documentary materials are records when they meet the conditions specified in § 1222.10(b).

(c) Working files and similar materials. Working files, such as preliminary drafts and rough notes, and other similar materials, are records that must be maintained to ensure adequate and proper documentation if:

(1) They were circulated or made available to employees, other than the creator, for official purposes such as approval, comment, action, recommendation, follow-up, or to communicate with agency staff about agency business; and

(2) They contain unique information, such as substantive annotations or comments that adds to a proper understanding of the agency's formulation and execution of basic policies, decisions, actions, or responsibilities.

(d) Record status of copies. The determination as to whether a particular document is a record does not depend upon whether it contains unique information. Multiple copies of the same document and documents containing duplicative information may each have record status depending on how they are used in conducting agency business.

Emphasis added. 

WHAT RECORDS MUST AGENCIES MAKE?

See 44 USC 3101, Records management by agency heads; general duties  

Quote

The head of each Federal agency shall make and preserve records containing adequate and proper documentation of the organization, functions, policies, decisions, procedures, and essential transactions of the agency and designed to furnish the information necessary to protect the legal and financial rights of the Government and of persons directly affected by the agency's activities.

Emphasis added.

WHAT RECORDS MUST AGENCIES KEEP?

See 36 CFR 1220.30, What are an agency's record management responsibilities?

Quote

(a) Under 44 USC 3101, the head of each Federal agency must make and preserve records containing adequate and proper documentation of the organization, functions, policies, decisions, procedures, and essential transactions of the agency. These records must be designed to furnish the information necessary to protect the legal and financial rights of the Government and of persons directly affected by the agency's activities.

Emphasis added.

Now suppose that an evaluator reading a technical proposal sees an assertion by the contractor of an engineering or scientific nature that she knows to be false based or more current research than the contractor has done. She scribbles a note to that effect and later shares that note with other evaluation team members during a team consensus meeting. Can the CO toss it after the announcement of the source selection decision? Should the CO tell evaluators not to make such notes?

My conclusion: The determination of what constitutes a source selection "record" and the determination of which records must be retained must be decided on a case-by-case basis. I think best practice is to retain everything produced by the evaluation team, at least pending review. That's what I was trained to do almost 50 years ago, and I still believe that it is prudent practice. If you do, you'll comply with the law and you'll be okay in that respect with the GAO. If you don't, you might find yourself in difficulty in more than one way. And once something is gone, it's gone, unless you're hiding a copy.

• I disagree with Don that "everything" is a record. That's too broad.
• I disagree with ji20874 that the GAO case law is "correct." Correct about what? It's own evidence policy?
• I disagree with ji20874 that individual evaluator notes are not records. That's too broad in the opposite direction from Don.
• I also disagree with formerfed when he says, "The most prudent and wisest practice in my opinion is preparing a consensus report and that should be the only document retained." That's so unwise it's scary. And it's scary that Joel would agree.
• And I disagree with Joel when he agrees with ji20874 that individual evaluator findings are not records

There is a troubling failure of professionalism on the part of some of the veterans of this forum when it comes to responding to questions. Too much opinion, not enough facts and analysis. Yes, some of the responses provided in this thread have been unprofessional. Wifcon Forum is not a reliable source of information. It won't be until the old timers act on their moral responsibility to be more thoughtful in responding to questions than they have done in this thread.

I have enjoyed meeting people in the Forum, and I have made many friends over the years.

I wish you all well. All of you. I would not have written this if I didn't. Goodbye and good luck.

Vern Edwards said:

I disagree with Don that "everything" is a record. That's too broad.

I didn't say "everything was a record." I said that I considered everything a record--meaning I retained everything produced by the evaluation team.

Voyager said:

I edited my OP to try to get to the bottom of this.

Based on the responses you've received, do you agree with the following assertion:

On 8/7/2023 at 12:32 PM, ji20874 said:

The source selection evaluation report, decision document, and so forth are records.  Individual evaluator notes are not records.

I stand by my prior comment.  If evaluators are instructed to read and/or listen to proposals, examine in light of the evaluation criteria, make notes and come prepared to discuss and produce a consensus report, the report of the only “record.”  That’s all that should be in the contract file.

formerfed said:

I stand by my prior comment.  If evaluators are instructed to read and/or listen to proposals, examine in light of the evaluation criteria, make notes and come prepared to discuss and produce a consensus report, the report of the only “record.”  That’s all that should be in the contract file.

Wouldn't it matter what was contained in the "notes"? Or is the characterization of the information as "notes" sufficient to conclude that they are not "records"?

On 8/9/2023 at 12:01 PM, Vern Edwards said:

And I disagree with Joel when he agrees with ji20874 that individual evaluator findings are not records

Vern, where did I “agree with ji20874 that individual evaluator findings are not records” ??  I have reread my comments in this thread and can’t find that “broad assertion”. I didn’t make any reference to ji’s statement.

I composed this during Bob’s temporary freeze on further comments as my response to Don Mansfield’s comment concerning  “a broad assertion that individual evaluator notes are not a record”.  

I didn’t think that I made any “broad assertion” that individual evaluator notes are [never] records.

in this post I should clarify my opinion concerning retaining evaluator notes. 

I will agree that they can be records under some circumstances. But I don’t think that they are a record under our specific methodology.

“Don Mansfield said : “I always considered considered everything a record.” 

I asked Don if he was going to support his opinion or assertion.

In response, Don said on the next day: “Individual evaluator notes" is undefined. They may or may not be the types of records required to be retained by the FAR. It all depends on what information they contain. A broad assertion that "individual evaluator notes" are, in fact, not records demonstrates a profound ignorance of case law.”

Don apparently changed his long held opinion that he “always considered everything a record”. 

 Don cited a case and concluded that it all depends upon what information they contain.

I don’t have a problem with that statement.

The 2010 Pitney Bowes Case that Don cited concerned individual evaluator “rating sheets”, where an evaluator would presumably assign a “rating” to a factor, (and subfactors and/or elements?) likely with some basis for the rating(?)

I described our process.  The solicitation doesn’t state or hint that the evaluation team members will individually evaluate or rate proposals.  To the contrary, our solicitations specifically stated that the source selection evaluation team will evaluate and rate proposals using a consensus procedure.

In addition, we specifically directed team members not to “assign ratings” to the factors and subfactors during their individual reviews.

Under our procedures at the time, ratings could only be determined under a defined and published rating system - AFTER the source selection Evaluation Board meets, discusses and collectively determines and documents the technical acceptability, strengths, weaknesses, uncertainties, deviations, deficiencies, etc.- if any,  during the group evaluation and documentation process.

Our Source Selection Evaluation Boards were composed of various internal organization members representing different career fields and functions plus one or more customer agency representatives. Not every member had the same technical or business background. 

Under our process, it would have been a waste of time and resources for individual members to try to document detailed conclusions or to come up with initial ratings. They read and familiarized themselves with the proposal, the proposal submission requirements, evaluation factors and evaluation criteria.

They would note what they thought to be a deviation, deficiency, question or uncertainty, an omission, something they thought was a strength or weakness, etc.  (as defined in the evaluation criteria) to discuss during the group consensus evaluation.

Under those circumstances, the notes of the individuals’ first look at the proposals, without the benefit of group discussions and better understanding of the solicitation requirements and more familiarity with the proposals, were mostly just that - “notes”. They weren’t determinative. Usually first impressions.

Those procedures were based upon lessons learned early on and applied over a course of about 90 source selections between 1989 and 2012 or so that I either conducted, led, participated in or supervised others leading a SS.   

In addition, I generally read the GAO protest decisions most every day from GAO and as Bob posted every day on the home page.

My source selections were only protested three times, one GSBCA (construction, Trade-off, we ultimately prevailed after a Court ruling that GSBCA had no jurisdiction over the acquisition), one Agency (construction, LPTA set-Aside for Disadvantaged SB, denied), and the one GAO (services, Trade-off, denied) that I referred to earlier. And Feldman was brilliant in that defense.

joel hoffman said:

Don apparently changed his long held opinion that he “always considered everything a record”.

No, I didn't. I would still consider everything to be a record. That's different than asserting everything is, in fact, a record.

The reason I considered everything a record was because I only had a layman's understanding of statutory records retention policy. My attorneys advised me to retain everything, so I did. I didn't feel the need to second-guess them on something that didn't really have anything to do with acquisition.

If I had a fellow PCO tell me that they knew better than our attorneys based on GAO case law that had nothing to do with interpreting statutorily-mandated records retention policy, then I hope I would have had the good sense to ignore them.

joel hoffman said:

Vern, where did I “agree with ji20874 that individual evaluator findings are not records” ??  I have reread my comments in this thread and can’t find that “broad assertion”. I didn’t make any reference to ji’s statement.

FWIW, I don't think you said that either.

Don Mansfield said:

Wouldn't it matter what was contained in the "notes"? Or is the characterization of the information as "notes" sufficient to conclude that they are not "records"?

I think it might if someone looked critically in hindsight and they were kept with the file.  But in all honesty, I didn’t envision “notes” were something that might be kept.  The best practice I referred to focused on preparing the consensus report.  Evaluators are told to be prepared and contribute to the overall discussion.  It’s assumed that most would bring written comments including strengths and weaknesses for each factor but that’s not required.  The group holds discussions, agrees on the results, and everyone signs.  In my opinion, any individual “notes” aren’t relevant after the group conversation and consensus.

@formerfed

Because it's you, and because we were members together for so long, I'll respond to your last two posts..

My post focused on the record retention requirements of 44 USC 3301, et seq, and the NARA regs, as distinct from GAO case law. See FAR 1.602-1(b). But you can see the importance of retaining documentation in the GAO decisions. What do they say about evaluators's notes?

A search of GAO decisions for the words evaluators and notes in a single sentence turned up 75 hits between 1976 and 2023. See, for example, Accenture Federal Services, LLC, B-421134.2, April 12, 2023. The issue was evaluators' notes taken during oral presentations:

Quote

Finally, Accenture challenges the agency's evaluation of proposals under the technical evaluation factor, asserting that, because the agency chose not to record the oral presentations, there is “no adequate record” of those presentations and maintaining “[t]his is in plain violation of FAR 15.102(e).” Second Supp. Protest at 22; First Supp. Protest at 32 n.8. More specifically, Accenture asserts that there is “no written record” regarding the offerors' responses to the “on-the-spot” scenarios, characterizing the evaluators' contemporaneous notes as “rang[ing] from completely illegible chicken scratch to legible but uninformative and ambiguous bullets.” First Supp. Protest at 49, Second Supp. Protest at 22.

We reject Accenture's assertion that the agency's documentation of oral presentations was inadequate. First, Accenture's reliance on the provisions of FAR part 15 is misplaced, since this task order procurement was conducted under the procedures of FAR subpart 16.5, which provide for a streamlined procurement process requiring less rigorous documentation.27 Our Office recently discussed the extent of an agency's obligation to document oral presentations in a FAR subpart 16.5 procurement under facts that are strikingly similar to those presented here. See Booz Allen Hamilton, Inc., B-419210, B-419210.2, Dec. 22, 2020, 2020 CPD ¶ 409 at 4-5. There, the solicitation (also issued by the Department of Homeland Security) required offerors to make oral presentations that were not recorded, but during which the agency evaluators took notes; permitted offerors to submit a limited number of slides addressing five questions disclosed in the solicitation, but also required responses to undisclosed “on-the-spot” and “scenario-based” questions; and subsequently prepared an evaluation report. Id. at 2-3. There too, the protester asserted that the agency's decision not to record the oral presentations resulted in an inadequate record of the oral presentations. We denied the protest, concluding that the combination of offerors' slides for a portion of the oral presentations, evaluator notes, and a subsequent evaluation report complied with the requirement for the agency to create an adequate record. Id. at 4-5.

Here, the offerors submitted written slides responding to the scenarios disclosed in the solicitation. Further, with regard to the “on-the-spot” scenarios, the agency evaluators created 60 pages of notes; discussed each offeror's presentation immediately after the presentation; and subsequently prepared an evaluation report documenting the basis for the assigned ratings. We have reviewed the record, including the evaluator notes and the evaluation report, and find no basis to meaningfully distinguish the facts presented here from those we considered in Booz Allen Hamilton. Accordingly, the protester's assertion that the agency failed to establish an adequate record of the offerors' oral presentations is denied.

And that was just a "fair opportunity" competition. The protest was denied.

Do you think those notes were "records" as defined by statute? If so, Why? If you think they were records, do you think 44 USC 3301 et seq and the NARA regulations require that they be retained in the file and not just summarized in a report? Do you think destroying them would be consistent with the law?

And see Southwest Marine, Inc.; American System Engineering Corp., B-265865, January 23, 1996:

Quote

Here, the Navy's destruction of the evaluators' notes and workpapers left a written record that did not meet the requirements of FAR § 15.608(a)(3)(ii). The contemporaneous documentation retained by the Navy—the TEC's final report, the TEC chair's cost/technical tradeoff document, and source selection decision—did not adequately explain why BAV's proposal was judged to be technically superior to SWM's and AMSEC's. Specifically, while the TEC report described BAV's proposal's strengths and advantages to justify its “highly acceptable” rating, it did not explain why the protesters' proposals, which appeared from this document to have similar strengths, were not essentially technically equivalent. Accordingly, the retained documentation was not sufficient to support the evaluation results. Indeed, a hearing was convened at our request, in part, because of the inadequacy of the evaluation documentation in the record, which did not fully explain the agency's evaluation, Tr. at 286–287, and the hearing testimony revealed that a number of identified strengths and weaknesses for all offerors' proposals were not recorded in the TEC's final evaluation report. Tr. at 336–338, 397–398.

That protest was sustained. Now the GAO did that without reference to 44 USC 3301. But do you think, given the GAO's findings, that destroying those notes and workpapers was consistent with the records retention law?

Now, yesterday, after I made my post and said that whether documentation constituted a "record" had to be determined on a case-by-case basis, you wrote:

On 8/9/2023 at 11:41 AM, formerfed said:

I stand by my prior comment.  If evaluators are instructed to read and/or listen to proposals, examine in light of the evaluation criteria, make notes and come prepared to discuss and produce a consensus report, the report of the only “record.”  That’s all that should be in the contract file.

I shook my head when I read your response to my post. You wrote it as if there is a national standard for the content of consensus reports. Do you know of one? Did you do any research before coming back with that? Can you see now how your statement, which strikes me as overly broad, but coming as it did from an experienced veteran of contracting and Wifcon Forum might mislead and misdirect less knowledgeable readers? That kind of thing is why I warn readers that Wifcon Forum in not a reliable source of information for anyone other than the foolishly credulous.

I can see that some of members of the Forum think hanging on to documentation is anal retentive. I can see that others fear that keeping documentation might reveal flaws in an evaluation and, like certain politicians, prefer to get rid of that which might be embarrassing. I don't expect to change such minds, but I want to provide information that might help others make sound decisions. As a former contracting professional, I happen to believe in transparent government and legal compliance. I don't think government employees should start taking bypasses from compliance.

I don't think I have your current email address, but if you send it to me via the Wifcon Forum message channel, or through Bob, I'll send you a list of all 75 GAO decisions. And then you can decide if a consensus report, whatever information that might contain, is the only document that need be kept in the contract file.

In any case, best to you.

Vern

This discussion prompted me to take some online training on records management offered by NARA: https://www.archives.gov/records-mgmt/training/online-lessons

I recommend "Recognizing Records, Non-records, and Personal Files" and the review and practice lesson. The modules are L1.012 and L1.013. This should be basic Federal employee training.

The distinctions that some are making in this thread between source selection decision documents, evaluation reports, individual evaluator notes, etc., are meaningless in the context of Federal Records Management policy. The distinction that matters is between records and nonrecords. There's nothing about an individual evaluator "note" scribbled on a scrap of paper that categorically excludes it from the definition of "record". If it meets the definition of "record" it's a record.

Vern,

Thank you for taking the time and effort to respond.  I now see my response was overly broad and general and definitely can be misleading.  This one part you quoted of Southwest Marine, Inc.; American System Engineering Corp., B-265865, January 23, 1996,  really made that clear

Quote

Accordingly, the retained documentation was not sufficient to support the evaluation results. Indeed, a hearing was convened at our request, in part, because of the inadequacy of the evaluation documentation in the record, which did not fully explain the agency's evaluation, Tr. at 286–287, and the hearing testimony revealed that a number of identified strengths and weaknesses for all offerors' proposals were not recorded in the TEC's final evaluation report. Tr. at 336–338, 397–398.

Looking over your post and Don’s most recent, I think your prior statement of your conclusion is most reasonable and sound for anyone to follow

Quote

My conclusion: The determination of what constitutes a source selection "record" and the determination of which records must be retained must be decided on a case-by-case basis. I think best practice is to retain everything produced by the evaluation team, at least pending review. That's what I was trained to do almost 50 years ago, and I still believe that it is prudent practice. If you do, you'll comply with the law and you'll be okay in that respect with the GAO

I really appreciate you showing the error in my statement.

Stan

Vern Edwards said:

its “highly acceptable” rating

Its what again?

Vern Edwards said:

Navy

Ah.

Just hoping to bring a little levity to my thread.

I'll see myself out.

In the interest of professional education, some last thoughts.

• What is a "note"?
• Why make ("take") them?
• Are there different kinds of notes (lecture, reading, observation, discussion, interview... )?
• If so, do they have different kinds of contents, styles, and formats? Are there standards?
• How do you make notes?
• Do your evaluators know what "good" notes are and how to make them?
• Do you?

Believe it or not, there is a voluminous literature about all that.

Making notes is something people take for granted, because they figure they know how to do it.

But do they?

Most people receive no formal instruction in note-making. 

See:

• "The Note-Taking Habits of Highly Successful People" https://observer.com/2017/07/the-note-taking-habits-of-highly-successful-people-richard-branson-david-allen-tim-ferriss-writing/
• "Duly Noted: The Past, Present, and Future of Note-taking," The Atlantic, September 2012

Anyone heard of Niklas Luhmann and the Zettelkästen note-taking method?  https://en.wikipedia.org/wiki/Zettelkasten

Thomas Edison produced more than 3,000 200-page notebooks.

Lawyers keep trial notebooks. What do they put in them? See https://www.fastcase.com/secondarymaterials/building-trial-notebooks/ and https://www.blumberg.com/downloads/TrialNotebook.pdf, https://www.illinoistrialpractice.com/2004/05/a_method_of_org.html

Should COs keep (a) source selection or (b) contract/modification negotiation or (c) claim/dispute resolution notebooks? Should CORs keep contract admin notebooks?

Take care, all.

We're getting older. Don't waste time. Be more knowledgable and smarter than the others in your office.

An interesting discussion.  While monitoring I was doing my own reading.  As this thread trails into the sunset one thought that I had based on the reading I did is interpretation and application of 44 USC 3301 post award with regard to both the contractor and the government.   The mere suggestion of the National Archives of a "contract term and condition" in itself is the start of another rabbit hole.   Quite a Pandoras Box.......

https://www.archives.gov/records-mgmt/policy/records-mgmt-language

Vern Edwards said:

Anyone heard of Niklas Luhmann and the Zettelkästen note-taking method?  https://en.wikipedia.org/wiki/Zettelkasten

Should COs keep (a) source selection or (b) contract/modification negotiation or (c) claim/dispute resolution notebooks? Should CORs keep contract admin notebooks?

These two thoughts from Vern reminded me of an excellent tool that is rarely used in contract offices.  The Wikipedia link mentions knowledge management and one valuable km technique is after action reviews.  Everyone involved in the project (in the context of this thread it would be source selection), participates in a “hotwash” session.  It’s an evaluation of performance where lessons learned, what worked and didn’t work identified, suggestions for improvements made, and pertinent other knowledge gets captured.  In this case, a portion of the session might involve effective use of notes, note taking, incorporating individual evaluators comments into the consensus, documenting information into a report, etc.   The session also includes items like the suitability and effectiveness of the SSP and individual evaluation factors, identification of topics for discussions, the process of reaching consensus, and on and on.  The importance is learning and sharing among others with the goal of continual improvements.

Just think back in your careers all the source selections you’ve been involved with.  Was there any formal means of capturing what worked well and what didn’t?  Was there any means to share those experiences throughout your organization and perhaps with still others?

As a post-script to this discussion:

If your organization states in a solicitation that the government will utilize a consensus evaluation and rating method, my strong recommendation is NEVER require or allow individual source selection team or evaluation team members to make or include a preliminary score or rating for any factors, subfactors, etc. in any notes that they create.

Technical evaluators and advisors should never make rating recommendations. That’s the SS team’s role.

The formal evaluation factors, rating system, etc. will provide rating criteria that the consensus evaluation process will use to document the official rating. The assigned ratings should reflect the underlying, documented basis for each rating. Start with the listed criteria, then assign a rating, not the other way around. 

If there is a protest, the protesting party’s attorneys will seize upon differences between an individual preliminary rating and the team concensus rating, attempting to confuse or show confusion and inconsistencies in the evaluations.

Anybody who has participated on an evaluation team will know that there are often initial thoughts that get hashed out and refined during a consensus discussion and formal evaluation and may differ from individual first impressions based upon level of one’s understanding, evaluator background knowledge and experience, etc.

Lessons learned from leading or participating in approximately 90 source selections. Of those only three were protested.  All those protests were eventually unsuccessful.  In two of them, the protestor’s counsel tried to make an issue between initial individual notes and the concensus evaluation without success. But it was very frustrating for the government team, lawyer and KO and necessitated additional time and effort that could have been spent on other work.

On 11/7/2023 at 10:28 AM, joel hoffman said:

If there is a protest, the protesting party’s attorneys will seize upon differences between an individual preliminary rating and the team concensus rating, attempting to confuse or show confusion and inconsistencies in the evaluations.

Anybody who has participated on an evaluation team will know that there are often initial thoughts that get hashed out and refined during a consensus discussion and formal evaluation and may differ from individual first impressions based upon level of one’s understanding, evaluator background knowledge and experience, etc.

This reminded me of an approach used successfully by one office during COVID.  Other offices adopted it later.  Evaluations were completed using virtual team software.  The evaluation lead loaded a proposal each morning for evaluators to read but do nothing more.  In the afternoon the proposal was discussed, scored, and documented.  The same happened each day until all proposals were finished.  The only record generated was by the lead during consensus discussions.

formerfed said:

This reminded me of an approach used successfully by one office during COVID.  Other offices adopted it later.  Evaluations were completed using virtual team software.  The evaluation lead loaded a proposal each morning for evaluators to read but do nothing more.  In the afternoon the proposal was discussed, scored, and documented.  The same happened each day until all proposals were finished.  The only record generated was by the lead during consensus discussions.

Excellent approach. Takes a lot of time at one per day but that could probably be adjusted, depending upon numbers and complexity…