Enjoy this.

I'm writing an article about proposal scoring in source selection and I'm going to quote the from 1963 Senate hearings about the famous TFX aircraft source selection dispute of the early 1960s. The quote is a series of exchanges between the head of the Air Force source selection team, Colonel Charles Gayle (abbr. as "G"), Senator John McClellan (abbr. as "M"), and the committee general counsel Jerome Adlerman (abbr. as "A"). Colonel Gayle tries to explain proposal scoring. Enjoy.

Mr. Alderman: When you would evaluate these things, and I don’t want you to give me any point scores or anything of the sort⎯well, I would like to get from you how you would evaluate, how you would score. These, I think are called raw scores, is that right?

Colonel Gayle: That is correct.

A: Without going into any detail by giving numbers or breaking it down too much to a particular level, let me⎯

G: I can briefly cover it.

A: I don’t want to do it in such a way it might give any actual scores or go too deeply into detail so that the contractors may know a system if the Defense Department doesn’t want them to know it.

G: We were given a criteria by the [source Selection] Board to cover the areas which we had teams on. Within this criteria, we had certain scores on a large or broad basis, you might say. From this criteria, we developed standards and a book of standards. The contractors were to be measured against the standards and not against each other. Within these standards and within the criteria⎯I will say it this way: In the criteria we had certain items that we were to measure. We took those items and had certain factors under each item that we would, in turn, measure. In some cases, we went down into the subfactors which we again would measure.

McClellan: Did you say measure?

G: Measure by points.

M: Measure or weigh?

G: Evaluate; yes, sir.

M: I don’t know which is a better word, “measure” or “weigh.”

G: Well, “weigh” is related to a number and probably is better.

M: I wanted to get it clear in my own mind.

G: Each item was an item that a team evaluated. If it was broken down into factors, factors were weighted on relative importance.

A: You didn’t weigh the scores, did you?

G: The team came up with their proposed weight of factors within an item.

A: You didn’t put any weight. In the evaluation scores, you had raw scores, you didn’t have weighted scores?

G: They are weighted even in the evaluation by the evaluation group. In other words, each item has a different importance. Let’s take the propulsion subsystem, for example.

A: Take the propulsion subsystem, and you have a scoring device of 1 to 10, is that right?

G: For the total, that is correct.

A: Then you assign a weight or what you think is the weight to give on a raw score without weighing what the propulsion system is valued at as to the whole system, is that right? Am I explaining myself correctly? You just evaluate one thing at a time. Everything that you evaluate is scored 1 to 10?

G: That is correct.

A: And that is called a raw score. You don’t say the propulsion system as to the whole ball of wax is worth 500 points or 1,000 points or 700 points?

G: That is correct.

A: You just evaluate each item. If you were to pick out, say, a propeller, you would say one propeller as against another propeller may be worth 7 and the other one worth 5, is that right?

M: That is against the standard?

G: Sir, if I could just make this point, in the propulsion area, the propulsion subsystem was an item. We then had factors in that propulsion subsystem, such as the installation and other things. I don’t recall exactly, but say there were 8 factors within an item which has these 10 points. Those factors were not all of the same importance on that one single item. Therefore, there is a weighting in the 10 points that you are talking about. That was all I was trying to point out.

C: You did not weigh it as against the overall part of it, but only as against that particular section or phase of it.

G: That is right, and from this we came up with raw scores for each item.

A: In other words, you might come up with a raw score, say, for the placement of the doors or an airplane at 1 to 10 and you might give one airplane nine credits against one credit to another one because you liked the door latch on that door, and you weighted it and felt that the door latch was very, very important on that door, is that right?

G: We did not compare it between the two airplanes. We compared it against the standards.

A: But you might say against the standard, one had nine points and the other had one point?

G: That is correct.

A: You might also go into ferry range, and you might say that one plane is rated 7 to 1 on ferry range, on the standard, and the other may be rated 1 to 7 on the ferry range. But as far as you are concerned, when it comes up to the addition, the ferry range weight and the door rate would receive the same number of credits, is that right?

G: Each item, individual item, receives a raw score. We do not weigh one against the other.

A: Again, to make the record clear, you did not take the two planes and the two proposals and put them side by side and say, “Looking at this one, it amounts to so much, and that one so much,” but you took each one and measured it or weighted it against the standards you had set up and then came out with your scores.

G: That is correct.

M: But if one had an advantage, it showed it had an advantage on this particular item or particular unit?

G: Basically, that is correct.

M: What do you call it? This particular component is a better word.

G: Item is correct, sir. That is the way we refer to them, as items.

M: Well, an item may be a component and a component an item?

G: That is correct.

M: But what you did was to take each component or each item and weighted that, measured it as you say, against the standards that you were looking for. If it was perfect, you gave it 10 points. If it was something else, you gave it nine, or seven, and so forth.

G: That is correct.

M: But you did not take the two planes or two proposals and lay them down and measure one against the other. You measured each against that standard.

G: That is correct.

M: And whatever it scored against the standard reflected whatever the difference was between the two proposals, is that right?

G: Yes, sir.

M: I think I understand that much now.

And I hope all of you understand, too.