Experience As An Evaluation Factor in Source Selection
Started by Vern Edwards · May 18, 2026 · 5 replies
- VOriginal post
Vern Edwards
May 18, 2026 · 19d ago
I have long been an advocate of experience as an evaluation factor in source selection. So I was surprised to read the following in the newest edition of NEW PHILOSOPHER, Issue 50, 2026, which is devoted to wisdom, under the heading The Myth of Experience:
We tend to assume that wisdom accumulates naturally with experience. Time passes, mistakes are made, lessons are learned, and judgment improves. This belief is deeply reassuring. It suggests that simply living long enough wil refine our understanding of the world. Yet evidence from psychology and behavioral science suggests that experience alone is a poor teacher.
In many domains, repeated experience does not sharpen judgment ⸺ rather, it dulls it.
The article mentions a book written in 2020 entitled, The Myth of Experience: Why We Learn the Wrong Lessons, and Ways Correct Them, by Soyer and Hogarth. According to a blurb at Amazon:
Our personal experience is key to who we are and what we do. We judge others by their experience and are judged by ours. Society venerates experience. From doctors to teachers to managers to presidents, the more experience the better. It's not surprising then, that we often fall back on experience when making decisions, an easy way to make judgements about the future, a constant teacher that provides clear lessons. Yet, this intuitive reliance on experience is misplaced.
Interesting?
Make sense when you think about it?
- f
formerfed
May 18, 2026 · 19d ago
Demonstrated prior completion of a task can be an important indicator of likely success. Placing significantly greater emphasis on completing the same task a dozen times can distort the value of experience in source selection.
- j
joel hoffman
May 26, 2026 · 11d ago
When past performance (how well you performed) is correlated with relative project experience (what you performed), they can be valuable indicators of probable success .
- V
Vern Edwards
May 27, 2026 · 10d ago
Let me play devil's advocate:
What can a CO really know about a contractor's experience and past performance other than what they personally witnessed?
(Do COs get factual information about past performance or just expressions of opinion?)
What can a CO really know about the reliability of third party reports and ratings?
I'm just asking for purposes of discussion. I'm not arguing against policy.
- f
formerfed
May 27, 2026 · 10d ago
The only meaningful and relevant performance data I’ve ever received was through personal contact (phone and in person conversations). This was both as a government employee and as a consultant. It wasn’t easy because it took effort. I had to really go in unscripted and let the conversation lead into the real issues. I met with COs, CORs, PMs and technical experts to find out candidly how the companies did. Not surprisedly, the feedback was much less favorable than the CPARS.
I compared it to checking employment references when hiring. It takes effort to dig out the important points.
I’ll add though, it’s still mostly an expression of opinion with very little actual facts.
- f
formerfed
May 29, 2026 · 7d ago
A friend in consulting that supports several CIOs said CIOs, in general, want more objective data to evaluate experience in source selections. He couldn’t provide specifics because of NDAs but mentioned this article from Government Executive.
https://about.govexec.com/insights/federal-buyers-case-studies-2026/
He added than CIOs don’t want more written proposal fluff about experience but demand data as proof of success.