Contracting Officer Testing

I support written exams for CO appointment. But...

First, contracting officers buy and do many things. Some buy base support services while others buy major systems. Some mainly issue and definitize change orders. Some don't buy much of anything, but negotiate forward pricing rate agreements and final indirect cost rates. Some are PCOs, some are ACOs, and some are TCOs. While all should possess the same fundamental knowledge, any notion that one test is suitable for all is simply absurd, and the people who support such exams are either insincerely advocating pro forma testing or do not understand the contracting field.

Second, any test made up entirely of true or false and multiple choice questions is an insult to the intelligence of truly qualified people. Such questions are sufficient to test only the most basic knowledge and are mainly aids to efficiency in testing. Any test that does not include both short and long form essay questions is not respectable. CO candidates should be tested for the ability to reason, argue, and explain. Candidates should be tested for knowledge of basic mathematics. Ideally, they also should be tested for oral communication skill. So, as for the test described by rmeman2k, I ask: What's the point?

Third, lawyers don't have to retake the bar exam, so why should COs routinely have to retake the qualifying test? There is a reason for continuing education. Instead, a CO's continued qualification should be evaluated on the basis of performance.

Fourth, a score of 85 percent isn't saying much. Is it okay for a CO to be right only 85 percent of the time?

Finally, if the 1102 community has any sense of self-respect and if it wants respect from other functional personnel, it should not merely assent to such testing, it should demand it, and it should demand that the testing be challenging. Agencies are giving CO appointments to unqualified people because they need signatures. It is shameful and a sign of disrespect for the contracting function.

Sadly, many people are afraid of high standards. I doubt we will ever have serious testing of CO candidates.

Second, any test made up entirely of true or false and multiple choice questions is an insult to the intelligence of truly qualified people. Such questions are sufficient to test only the most basic knowledge and are mainly aids to efficiency in testing.

I disagree with that statement as it pertains to multiple choice questions. Such questions can be used to assess more than just basic knowledge. For an example of multiple-choice questions at advanced levels of learning click here.

Don,

I think your examples prove my point. Those questions are high school. Here is my idea of an "advanced level" multiple choice question:

===============

In response to a request for proposals for 2,000 two-way radios to be manufactured to a government specification, Radio Corp. proposed to manufacture and deliver the radios for a unit price of $404.00 each. Each radio included a battery case, which the government?s specification required to be .064 inch thick. The cost of the battery case was included in the unit price for the radio. The battery case was not separately priced.

Radio Corp. had manufactured the radios for the government before. The last time, the government?s specification had required the battery case to be .051 inch thick. For this procurement, however, the government engineers had increased the thickness of the battery case to .064 inch thick. Radio Corp. planned to subcontract the manufacture of the battery case, as it had before. But in seeking quotes, Radio Corp.?s purchasing department failed to notice that the government had changed its specification, and solicited subcontractor quotes based on the old specification of .051 inch thick instead of the newly specified .064 inch thick. (This was an honest mistake.) The lowest quote Radio Corp. received from prospective subcontractors for a .051 inch thick case was $13.97 per case. When preparing its proposal, Radio Corp. assumed that it could bargain that price down to $12.48 per case, so it decremented the subcontractor quote to $12.48, which is the amount that it included in its proposed $404.00 unit price.

Radio Corp. won the contract at its proposed price of $404.00 per radio. The government did not require offerors to submit cost or pricing data or information other than cost or pricing data with their proposals. At the time of contract award neither the government nor Radio Corp?s proposal team was aware of Radio Corp.?s mistake about the case.

After contract award, but before Radio Corp. could award the subcontract for the cases, another prospective subcontractor called Radio Corp.?s purchasing department and asked to submit a quote. The purchasing department then detected its mistake about the thickness of the case and asked for a quote for cases .064 inch thick, instead of .051 inch. The new quote was $7.52 per case. Since the contract unit price included $12.48 for the case, the purchasing department?s mistake meant that Radio Corp. would make an additional profit of $4.96 per radio.

But before Radio Corp. ordered any cases, the government?s engineers discovered that an .064 inch thick case would not fit properly in the radio. The contract included FAR clause 52.243-1, Changes?Fixed-Price (AUG 1987), so the contracting officer issued a change order, changing the thickness of the case from .064 back to .051 inch thick. Only at this point did the government learn about the $13.97 quote that Radio Corp. had originally obtained for the .051 inch cases, about the $12.48 it had included in its proposal, and about the $7.52 quote for the .064 cases. The contracting officer asked Radio Corp. to get new quotes for the .051 cases. The lowest new quote Radio Corp. received was $11.63 per case.

Which of the following is the correct measure of the proper equitable adjustment under FAR 52.243-1, excluding adjustment to profit:

a. Reduction of $ 0.85 per unit.

b. Increase of $4.11 per unit.

c. Reduction of $2.34 per unit.

d. Increase of $4.45 per unit.

Explain the rationale for your answer in 500 words of less. Your choice must be correct and your rationale must support it.

===============

Vern

Vern,

Really? If you think that those questions assess "basic knowledge", then we must differ on the definition of "knowledge".

Below is a sample multiple-choice question from the LSAT:

Exactly six trade representatives negotiate a treaty: Klosnik, Londi, Manley, Neri, Osata, Poirier. There are exactly six chairs evenly spaced around a circular table. The chairs are numbered 1 through 6, with successively numbered chairs next to each other and chair number 1 next to chair number 6. Each chair is occupied by exactly one of the representatives. The following conditions apply:

Poirier sits immediately next to Neri.

Londi sits immediately next to Manley, Neri, or both.

Klosnik does not sit immediately next to Manley.

If Osata sits immediately next to Poirier, Osata does not sit immediately next to Manley.

1. Which one of the following seating arrangements of the six representative in chairs 1 through 6 would NOT violate the stated conditions?

(A) Klosnik, Poirier, Neri, Manley, Osata, Londi

( Klosnik, Londi, Manley, Poirier, Neri, Osata

( C ) Klosnik, Londi, Manley, Osata, Poirier, Neri

(D) Klosnik, Osata, Poirier, Neri, Londi, Manley

(E) Klosnik, Neri, Londi, Osata, Manley, Poirier.

Do you think that this question assesses "basic knowledge"?

Don:

I added to my prior post after you posted your response. Sorry. As for the LSAT question, that is part of a test to decide who can go to law school, not who can defend a person up on a capital murder charge.

Vern

Vern,

Really? If you think that those questions assess "basic knowledge", then we must differ on the definition of "knowledge".

Below is a sample multiple-choice question from the LSAT:

Exactly six trade representatives negotiate a treaty: Klosnik, Londi, Manley, Neri, Osata, Poirier. There are exactly six chairs evenly spaced around a circular table. The chairs are numbered 1 through 6, with successively numbered chairs next to each other and chair number 1 next to chair number 6. Each chair is occupied by exactly one of the representatives. The following conditions apply:

Poirier sits immediately next to Neri.

Londi sits immediately next to Manley, Neri, or both.

Klosnik does not sit immediately next to Manley.

If Osata sits immediately next to Poirier, Osata does not sit immediately next to Manley.

1. Which one of the following seating arrangements of the six representative in chairs 1 through 6 would NOT violate the stated conditions?

(A) Klosnik, Poirier, Neri, Manley, Osata, Londi

( Klosnik, Londi, Manley, Poirier, Neri, Osata

( C ) Klosnik, Londi, Manley, Osata, Poirier, Neri

(D) Klosnik, Osata, Poirier, Neri, Londi, Manley

(E) Klosnik, Neri, Londi, Osata, Manley, Poirier.

Do you think that this question assesses "basic knowledge"?

I think the answer is ( - Klosnik, Londi, Manley, Poirier, Neri, Osata.

1. Poirier sits immediately next to Neri.

2. Londi sits immediately next to Manley.

3. Klosnik does not sit immediately next to Manley.

4. Osata does not sit immediately next to Poirier or Manley.

If that's right, then how hard can it be if I can figure it out?

Will the Army exam require a regulatory citation in support of each response?

My opinion - It is not reasonable to test a KO with a limited warrant who is performing SAP Commercial installation-level acquisitions on acquisition procedures for non-commercial R&D or Construction. As a KO I am in support of primarily SAP supplies and services. I have little to no experience in non-commercial R&D or Construction contracting and would not accept a warrant in support of either.

Vern's Post #2 made 5 points. People have taken exception to one of them.

Let me add my endorsement to: "Fourth, a score of 85 percent isn't saying much. Is it okay for a CO to be right only 85 percent of the time?"

During SCUBA training I felt much the same way. After scoring 99 on a test, the 1% you missed can kill you.

Any system of testing has flaws, but also has the potential to measure knowledge and establish standards. Is putting an admittedly imperfect system in place better for the taxpayers than the status quo?

What?s the answer to Vern?s question ? I say d) $4.45. I use the difference between the lowest quote before the change and the lowest quote after the change ? $4.11 ($11.63 -$7.52). Plus costs incurred to prepare a request for price adjustment proposal are allowable, per CAM 12.502, and since there is no other value above $4.11, the multiple choice answer of $4.45 must reflect the additive proposal costs.

I do think that Vern met to assign $6.45 to d) ($13.97 - $7.52) if so, then d) is incorrect and is correct if proposal costs are not meant to be part of this question.

In reality probably any answer is OK, because we can't see the contractor's equitable adjustment proposal. They may have asked for a price reduction - and I am sure that the Government would not have said no you should get more money.

DON's question LSAT question for would be jailhouse lawyers - Answer B

Don:

I added to my prior post after you posted your response. Sorry. As for the LSAT question, that is part of a test to decide who can go to law school, not who can defend a person up on a capital murder charge.

Vern

Vern,

The difficulty of the question and the purpose of the test it appears on are beside the point. This is an example of a multiple-choice question that does not assess basic knowledge. One cannot "know" the correct order of the trade representatives and recall that fact from memory. One can only arrive at the answer through analytical reasoning.

The question that you wrote (minus the essay requirement) would be another example of a multiple-choice question that assesses more than basic knowledge.

Don,

A person could get the answer to my question right by guessing, which is why I would tack on the essay. I despise multiple choice tests and the institutions that use them. Nothing you have said or could say is going to change my opinion about multiple choice testing, which is summed up nicely by this:

By means of sample questions it is shown how the tests use ambiguity, and worse, as substitutes for worthwhile difficulty, how they favour quick-witted superficiality while penalizing depth, subtlety and creativity, how they encourage intellectual dishonesty and how they corrupt education.

Hoffman, "Multiple Choice Tests," 1967 Phys. Educ. 2, 247.

A person who gets a warrant because he or she could get a score of 85 on a 50-question multiple choice test gets no professional respect from me on that account.

We have hijacked this thread, so I am not going to discuss this further with you here. You can take this up with me privately if you wish.

Vern

Don,

A person could get the answer to my question right by guessing, which is why I would tack on the essay. I despise multiple choice tests and the institutions that use them. Nothing you have said or could say is going to change my opinion about multiple choice testing, which is summed up nicely by this:

Hoffman, "Multiple Choice Tests," 1967 Phys. Educ. 2, 247.

A person who gets a warrant because he or she could get a score of 85 on a 50-question multiple choice test gets no professional respect from me on that account.

We have hijacked this thread, so I am not going to discuss this further with you here. You can take this up with me privately if you wish.

Vern

I had developed a lenghty explanation of how the National Council of Engineering Examiners tested for Engineer in Training certification and for Professional Engineer Registration. but I pushed the wron button on the BB and it erased it. Will try again tomorrow at office on the computer. Short version - relying only on multiple choice questions seems to be for the convenience of the testing organization and is not the best way to test knowledge and qualifications.

Short version - relying only on multiple choice questions seems to be for the convenience of the testing organization and is not the best way to test knowledge and qualifications.

That is right on the money.

"1) Do you think that a test consisting solely of essay questions would be the best way to test knowledge and qualifications?"

It might or might not be. I had some college classes which used that method.

If memory serves me correctly (1977), the 8 hour National Council of Engineering Examiners standardized Professional Engineers' Examination at least partially consisted of engineering problems, in which we had to develop a solution. For the afternoon portion, we had to select 5 problems, worth 10 points, each. We had to show the full methodology used and the answer. No calculators were allowed. We used slide rules and logarithm books (The Chemical Rubber Company - "CRC-Handbook"). Other reference books and textbooks were allowed. Memorization of all engineering formulas, etc. wasn't necessary or even practical.

The morning session consisted of multi-part problems. I think we had to select 5 different problems to answer. Each had 10 questions. I don't remember if they built on each other.

And yes - even if we got the wrong answer, we could get partial credit. I remember a V-notch weir problem as part of a dam. I couldn't find the dad-gummed formula to calculate the flow capacity of such a weir, but I remembered that the flow was directly proportional to various factors, including a modifier. I admitted that I couldn't find the correct formula and said that I had developed the formula and included a "K-factor". I told them that and that my assumed K-factor was made-up. I remember getting something like 8 out of 10 points on that problem. I believe that we had to score at least 70 or 75 points to pass.

For example, other problems included steel design and reinforced concrete design. The complex formulas for the concrete design required numerous iterations to arrive at the optimum, most economical combination of size, concrete strength, rebar strength, rebar size, rebar spacing spacing, etc. One could get partial credit, even if we didn't have time to optimize the design elements.

There was also a State-specific exam, which included questions on state laws and regulations, including a section on handicapped accessible design standards. We had four years of engineering experience before taking the PE exam. With the subsequent advent of personal computers, design of reinforced concrete structural members and the involved math has greatly simplified the task!! If I could have even used a calculator or a programmable calculator, I would have enjoyed math, concrete design, structural design, back then. As it was, it was pure drudgery at times.

Now, the Engineer-in-Training exam was a prerequisite for the PE exam. We could take it in college. Unfortunately, I waited 5 years after college to take it (1976) and had to attend a semester of night refresher classes at the University of Wisconsin in Milwaukee - about 60 miles from our home at the time. It was also an 8 hour exam. The morning consisted of - yes - multiple choice questions. I dont remember how many there were but they covered all engineering disciplines. Each correct answer was worth 1/2 point with a maximum possible score of 50 points. Yes - we could guess - that was a weakness in the test. The afternoon session consisted of a selection of 5 problems with 10 answers. I think these were multiple choice but they all seemed to be related, so one couldn't just guess and expect to get the rest.

"2) What method was used when you got your warrant?"

There was no "test" for a Contracting Officer's Warrant in the 1990's, Army-wide. Individual Commands may have tested. I attended the DAWIA mandatory training for Levels I and II (later Level III) and had to have a college degree, including at least 25 semester hours in various business, law and other classes, as well as a number of years of related experience in various areas of acquisition - procurement - contracting - call it what you want. As an engineer, I obtained a KO warrant that was limited to ACO authority on construction contracts because I wasn't an 1102.

I described the test methods for the engineering registration process to show the difference between various testing methods. A 50 question multiple choice exam just doesn't seem like it would really be a useful measure of competency, knowledge and skill. The test method seems to be set up for expediency and the convenience of the test providers.

Plus, I do'nt think that periodic retesting would be very useful, either. A requirement for continuous learning and regular ethics and appropriations law training is more meaningful to me.

I spent this afternoon at the Air Force Judge Advocates Conference, where I served on a panel with Ty Hughes, the USAF Deputy General Counsel, Prof. Steve Schooner of The George Washington University Law School, and Rand Allen, Chair of the Government Contracts Practice at Wiley Rein, LLP. Rand is the attorney who represented Boeing before GAO in the tanker process. We were talking about challenges in Air Force contracting.

Before our panel started I listened to a presentation by USAF litigation attorneys in which a single theme came up again and again -- the importance of getting good documentation from contracting officers: statements of fact for protests and final decisions for claims. After our presentation, in which I discussed the difficulties USAF COs have producing good documentation, I was approached by several attorneys who complained that Air Force COs could not develop well-reasoned explanations for their decisions. I spoke with one who has written guidance for COs about how to prepare statements of fact for protests and who despaired about the inability of COs to write coherently. We have seen some evidence of this in posts at Wifcon.

This is why I want essay questions for CO exams. If you cannot explain your answers in well-written essays, you are not qualified to be a CO for acquisitions in excess of the simplified acquisition threshold.

The good news is that reasoning and writing can be learned by anyone who wants to do so.

As an intern, I am very interested and open to the testing methods prescribed by Mr. Edwards. The current ?status quo? involves assembly-line style training, which appears to be focused on presentation and speed rather than substance. One can only refer to the sheer amount of badly planned online training as an example of what is wrong with government acquisition training today.

As you all know, DAU offers most basic CON courses such as 110, 111, and 112 online. From the onset, this would make sense to a time-conscious manager. These exams have no required essays and no presentations whatsoever. As a matter of fact, a student may finish the course without taking much away due to being allowed to ?fail? the exam 3 times before failing the class. By the way, the online exams are all untimed, open-book and seem to be focused on rote-learning methods rather than critical thinking.

I will dare to go one further than Mr. Edwards and offer some sort of teamwork section of an exam where you are assigned a certain task to perform with other students. Maybe this could even be combined with an oral/presentation section of the examination? Maybe I am being a bit naive, but teamwork seems to be almost as important as self-reliance in the world of government contracts. I am sure the veterans of contracting will rip the prevois statement apart though.

A lot of people seem to think that having an exam at the end of every class substitutes the need for having one final ?bar? examination. I can?t say that I have benefited from multiple choice exams at the end of a long class. From my experience, most people tend to take away more from a course that pushes critical thinking skills rather than memorization. Lectures with no form or clarity go in one ear and out the other when hands-on learning isn't invloved. That is why I have to read and explore on my own since I can't really trust the instructor to give me everything I need.

Mr. Edwards also mentioned about how lawyers are not required to take the bar exam after one has passed. That is true, but COs are required to complete X amount of continuous learning credit hours every so many years. I have taken one of these elective courses filled with experienced COs just looking for a blow-off class just to use as filler. These ?continuous education? modules serve as nothing more than a means for external vendors to line their pockets. Most of them are 2-3 days long and offer no examination at the end.

Motorcity, detroit rock city, you stated, "Mr. Edwards also mentioned about how lawyers are not required to take the bar exam." For the most part attorneys are required to take a bar exam except for Wisconsin (And possibly other states). When you have a moment please explain your statement.

Will the Army exam require a regulatory citation in support of each response?

If the new Army Contracting Command testing procedures completely mirrors the Air Force Mandatory Procedures (MP) 5301.603 dated Apr 2010, it should require a citation. Per MP 5301.603, 3.3.7., "A correct answer and citation of a valid reference, to the paragraph level (e.g. FAR 15.403-1(), as applicable, indicating where the candidate located the answer to each question must be provided to earn two points......." Therefore, the candidate is given an opportunity to provide rationale to support the answer, which is good. However, I still think essay questions should be included in the exam so that the candidate can demonstrate her/his thought process, logic, documentation skills, and even creativity.

As to documentation skills, when I started my very first 1102 job, my CO told me documentation is essential to contracting and I still religiously believe in that. If a sound business decision is orally stated with strong supporting information, but if it is not documented in file, the job is half-done. Speech skills are as necessary as writing skills, but any oral communications that resulted in important decisions shall be followed up by detailed documentation. You need to document: to justify your business judgment; to protect yourself and your agency by storing evidence; and to leave the history of the contract in the file for those who inherit the contract from you.

What I said was that lawyers are not required to retake the bar exam.

In California, in order to be an attorney you must pass a first-year law student exam and the bar exam, Here is a sample question from 2010's first-year exam. Students were given four hours to write an answer:

Adam accepted an invitation from his friend Dot to attend a baseball game. The seats

Dot had purchased were very good, a few rows up from the field, and just past first

base. Adam had recently immigrated to the United States and knew nothing about

baseball.

During the game, a player, Brad, hit a ground ball toward third base. The third baseman

fielded the ball and threw to the first baseman. Brad thought he was ?safe,? but the first

base umpire called him ?out.? Brad began to argue with the umpire, and in frustration,

Brad threw his batting helmet to the ground. The helmet, made of a strong, hard plastic

material, bounced on the ground and flew into the stands, striking Adam on the side of

the head, causing a serious injury. Adam never saw the helmet coming towards him

because he was looking around the stadium at the time rather than at the play on the

field. As he was unfamiliar with the game of baseball, he did not know that objects

occasionally fly into the stands during a baseball game.

What arguments would Adam make in support of a claim of negligence, what defense(s)

can reasonably be asserted, and who is likely to prevail in a lawsuit filed by:

1. Adam against Brad? Discuss.

2. Adam against Dot? Discuss.

Here is the guidance provided to students taking the exam:

Your answer should demonstrate your ability to analyze the facts in the question,

to tell the difference between material facts and immaterial facts, and to discern the

points of law and fact upon which the case turns. Your answer should show that you

know and understand the pertinent principles and theories of law, their qualifications

and limitations, and their relationships to each other.

Your answer should evidence your ability to apply the law to the given facts and

to reason in a logical, lawyer-like manner from the premises you adopt to a sound

conclusion. Do not merely show that you remember legal principles. Instead, try to

demonstrate your proficiency in using and applying them.

If your answer contains only a statement of your conclusions, you will receive

little credit. State fully the reasons that support your conclusions, and discuss all points

thoroughly.

Your answer should be complete, but you should not volunteer information or

discuss legal doctrines which are not pertinent to the solution of the problem.

You should answer the questions according to legal theories and principles of

general application.

Here is a sample question from the 2010 bar exam. Candidates must answer three such questions in one hour:

On April 1, Pat, a computer software consultant, entered into a written services contract

with Danco, Inc. to write four computer programs for use by Danco in controlling its

automated manufacturing machines. The contract provided that Danco would pay Pat

$25,000 on completion of the work and that the programs were to be delivered to Danco

no later than May 1. The contract stated, ?This is the complete and entire contract

between the parties, and no modification of this contract shall be valid unless it is in

writing and signed by both parties.?

Pat entered into the contract in anticipation that it would lead to significant work from

Danco in the future, and he consequently turned away opportunities to take on more

lucrative work.

On April 15, Pat called Chelsea, the President of Danco, who had executed the contract

on behalf of Danco, and told her, ?I?m having some problems with program number 3,

and I won?t have it ready to deliver to you until at least May 8 ? maybe closer to May 15.

Also, I have some doubt about whether I can even write program number 4 at all

because your computer hardware is nearly obsolete. But I?ll get programs numbers 1

and 2 to you by May 1.?

Chelsea said in response, ?I?m sorry to hear that. We really need all four programs. If

you can?t deliver until May 15, I guess I?ll have to live with that.?

On April 28, Pat called Chelsea and said, ?I?ve worked out the problems with programs

numbers 3 and 4. I?ll deliver them to you on May 12.?

Chelsea responded, ?I?ve been meaning to call you. I?m going to start looking around

for another consultant to do the work because I consider what you said in our April 15

telephone discussion to be a repudiation of our contract. My lawyer tells me that,

because of the language in the contract, nothing I said to you in that conversation

matters. You repudiated the contract, so we don?t owe you anything.?

Can Pat prevail in a suit against Danco for breach of contract, and, if so, what is the

measure of his damages? Discuss.

Here is the guidance given to persons taking the bar exam:

Your answer should demonstrate your ability to analyze the facts in question, to tell the

difference between material and immaterial facts, and to discern the points of law and

fact upon which the case turns. Your answer should show that you know and

understand the pertinent principles and theories of law, their qualifications and

limitations, and their relationships to each other.

Your answer should evidence your ability to apply law to the given facts and to

reason in a logical, lawyer-like manner from the premises you adopt to a sound

conclusion. Do not merely show that you remember legal principles. Instead, try to

demonstrate your proficiency in using and applying them.

If your answer contains only a statement of your conclusions, you will receive little

credit. State fully the reasons that support your conclusions, and discuss all points

thoroughly.

Your answer should be complete, but you should not volunteer information or discuss

legal doctrines which are not pertinent to the solution of the problem.

Unless a question expressly asks you to use California law, you should answer

according to legal theories and principles of general application.

Those are the kind of "long form" essay question that I would give to candidates for an unlimited warrant. However, my questions would be geared to contracting officer responsibilities, not those of lawyers. For example, candidates could be given a claim and the facts involved and be required to write a final decision.

The California bar hires graders. Here are the minimum qualifications:

The Committee of Bar Examiners solicits applications from attorneys to grade the California Bar Examination twice yearly. Minimum qualifications require that applicants:

Be active members in good standing of The State Bar of California;

Have been admitted to The State Bar of California not less than one year prior to the examination for which the grader is to be selected, and:

Have passed the California Bar Examination on the first or second attempt.

Graders are paid a token amount.

Motorcity:

You are right about DAU training. DAU has been given mission impossible -- to train almost 30,000 contract specialists, which is about the number of undergraduates at one of the California universities. Even to train just the interns would be a daunting task. Moreover, they don't have a real university faculty. Most of the instructors are contracting personnel who applied for the job and were hired. They teach mainly by rote to standard instructional materials, which are not very deep. The curriculum is very poor. Online learning could be improved dramatically. They might be able to contract with a university to develop the program. But it would cost a lot and I don't know if they have the budget to do it.

Your idea about a teamwork segment to a CO exam is intriguing, but probably impractical. The main reason for using standardized multiple-choice questions is that it is a practical way to test large numbers of people. You don't need experts to grade the tests. You can do it with automation. Practicality is one of the objections to essay questions for COs: If you must test a large number of people, who will grade the essays and how can you ensure consistent grading? However, the problem is clearly solvable, as shown by the California bar exam.

I agree with Vern on most of his points but I have had experience with a multiple choice test system that did test comprehensive knowledge and reasoning abilities. That test was the Series 7 General Securities Representative Examination, an 8 hour 2 session test that had a higher failure rate than the bar exam in California when I took the test. The questions in that test required a lot of knowledge, logical reasoning, and also an ability to understand a complex written problem in order to answer the questions.

I used more scratch paper in passing that test than there were test pages and while the test did not test my ability to communicate orally, I could not have passed it without a very good knowledge of the subject AND an ability to interpret written problems sucessfully. Would a series of essay questions improve such a test? Perhaps, but I don't think the improvement would be significant given the complexity of the questions in that test. Also, how would the subjectivity of the grading official play into the grading of an essay question? Yes, in a perfect world that would not be in question. The trouble is we do not live in such a world, as I clearly understand facing some particularly subjective grades I have received in my doctoral program over the last year or so.

Moreover, they don't have a real university faculty. Most of the instructors are contracting personnel who applied for the job and were hired. They teach mainly by rote to standard instructional materials, which are not very deep. The curriculum is very poor. Online learning could be improved dramatically. They might be able to contract with a university to develop the program. But it would cost a lot and I don't know if they have the budget to do it.

I have wondered about what you wrote above for a number of years now. I have applied for a number of positions at DAU, emphasizing the fact that I was a doctoral student and have experience teaching at a community college. That MAY have been a mistake, as I never even received an invitation for an interview. It appears that DAU does NOT WANT real university faculty, they seem to want rote instructors who may or may not have any experience in what they are teaching. Nothing against those individuals, but it seems to be the truth from my viewpoint.

I agree with Vern on most of his points but I have had experience with a multiple choice test system that did test comprehensive knowledge and reasoning abilities. That test was the Series 7 General Securities Representative Examination, an 8 hour 2 session test that had a higher failure rate than the bar exam in California when I took the test. The questions in that test required a lot of knowledge, logical reasoning, and also an ability to understand a complex written problem in order to answer the questions.

I used more scratch paper in passing that test than there were test pages and while the test did not test my ability to communicate orally, I could not have passed it without a very good knowledge of the subject AND an ability to interpret written problems sucessfully. Would a series of essay questions improve such a test? Perhaps, but I don't think the improvement would be significant given the complexity of the questions in that test. Also, how would the subjectivity of the grading official play into the grading of an essay question? Yes, in a perfect world that would not be in question. The trouble is we do not live in such a world, as I clearly understand facing some particularly subjective grades I have received in my doctoral program over the last year or so.

My main issue with multiple choice questions is that they do not require the candidate to demonstrate his or her ability to reason and explain. It is possible to construct a very difficult multiple choice test. But the answers will not show how the candidate arrived at his or her conclusion. COs must explain their decisions in writing and those documents are subject to court review for reasonableness. The most important questions faced by COs do not always have straightforward right or wrong answers.

I have no argument that selection of contract specialists and purchasing agents for warranting could be improved. In a general view everything can be improved.

The argument I have with essay questions is that their scoring is subjective and can be biased, as biased as many of the posts to WIFCON itself. I have through my own experiences seen this bias with regard to essay questions. A bias is an element in issuing warrants today under current application, testing and oral interview process used by the hundreds of agencies that issue warrants. Simply the essay tests along with oral interviews are a way for folks to apply their own bias as to who should and who should not get a warrant so be careful what you ask for unless you implicitly trust your upper level acquisition personnel. Essay questions and orals might in one persons mind be applied fairly but in another person?s mind bias will occur, no doubt about it.

Discussion of including essay questions to evaluate an individual for a warrant is easy. Developing and scoring essay questions to ensure they fairly evaluate ones knowledge is a whole other matter especially across the whole of the Federal government. Take a look even today there are different standards among the hundreds of agencies even with the implied cookie cutter approach of DAWIA and FAI regarding such things as FAC-C.

In noting my concerns I do not make the claim that I am an expert in the field, I simply offer that a detailed study of the value of essay questions and their application to determining who should and who should not get a warrant and the varied levels where the essay question standard should be applied needs a complete analysis and evaluation before applying across the board.

Example - Is the standard the same for the 1105 purchasing agent seeking a warrant at the SAT, and for the senior management level person who by position is simply given warrant authority (and for all the levels in between)?

My conclusion is that in an organization as large and varied as the Federal government the objective test should be give priority and not the subjective test implied by a standard of essay testing in part or in full.

In keeping with intent of posts made in WIFCON discussions I provide one reference: http://www.personal.psu.edu/faculty/s/r/sr.../essayexams.htm

I found this at the Sparknotes website. It was apparently written for high school studebts. Some of you might find it amusing.

Essay vs. Multiple Choice: Battle of the Exams

By this point in the semester, your tests are probably starting to pile up like dirty laundry. Just when you think you're in the clear and have a few days to relax and focus on homework, you remember that your foreign language teacher added a 50-point "quiz"?which, as far as we're concerned, is a full-on test.

We decided to probe the pros and cons of the two main test types: multiple choice and essay. We pulled an all-nighter comparing these exam options. Here's our crib sheet:

Preparation

Preparing for a multiple choice test is generally pretty easy. All you have to do is recognize the information when you see it.

An essay exam requires that you harness a thorough, working knowledge of the subject, such that you can respond on the spot to any prompt with an interesting and coherent presentation of your ideas. Unless, of course your teacher gives you the essay topic the night before the test, in which case you can compose your essay at home, commit it to memory, and spit it back out the next day. Simple.

Speed

You can complete a multiple choice test in a matter of minutes?whether you know the answers or not. The rest of the period can be used either to nap or to do that math homework you forgot about.

Even if you know what you're talking about, an essay exam takes time. You have to make sure that your thoughts are all in order. And handwriting counts! If your teacher can't read your essay, it doesn't matter how amazing it is.

Smudging

If your multiple choice test is administered using a fill-in-the-bubble sheet, you will need to use a pencil. Pencils increase the likelihood of smudging...which can confuse the electronic-grading-robot.

If you're lucky, your teacher will let you write your essay exam in pen, giving your test a clean, polished look. If your teacher insists on pencil-written exams, make sure you don't wipe your face with the back of your hand.

Hand-Eye Coordination

If you accidentally skip a bubble on a multiple choice test, all your subsequent answers will be off. Hopefully, you will notice your mistake before handing in the test. If not, we're sorry in advance for that "D" that you were sure was going to be an "A" because you knew the material so well after weeks of preparation. (Stupid bubble-sheet!)

If you accidentally skip a line on your essay exam, your teacher will just think you wanted a little extra space for your large, loopy letters. So if you get a "D," you've pretty much earned it.

Creativity

An essay exam lets you express your creativity through language, crafting sentences that convey meaning and beauty all at once.

With a multiple choice test, the only way you can do express yourself is by sacrificing your score to create interesting patterns on the scantron sheet.

BS

If you don't know the answer on a multiple choice test, you've got a 1 in 4 chance of guessing the right one.

If you don't know the answer on an essay exam, you can still come up with a convincing response that makes it sound like you know what you're talking about. Finally! Here's your chance to use all those SAT vocab words.

Giving Up

If you don't want to take your multiple choice test, you can just pick "C" for every answer. Odds are you'll get at least 25% of the test right, and you can make up the difference with extra credit.

Giving in on your way through an essay exam is a little tougher. Either you repeat the same phrase over and over on your paper?which is likely to earn you a trip to the school counselor?or you sheepishly hand in a blank page and slink away.

Celebration

When you bring home a good grade on an essay exam, your parents can bask in the glory of your success by reading and re-reading your essay at the dinner table, heaping praise onto your impeccable prose.

Your parents will be just as proud of your multiple choice triumph, but the experience isn't the same. It's just not as exciting to recite your multiple choice answers out loud. "A, A, C, B, D...Oh wow, I wouldn't have expected a D next."

So which would you prefer? If you could choose, would you pick an essay or a multiple choice test?

I just came across this goal in the DAU Strategic Plan:

Develop standardized DoD warranting program and develop ?back to basics? OJT for new inexperienced contracting workforce by 31 Aug 11.

Sparknotes.

Sparknote reference was funny. I am a firm believer in "when in doubt Charlie out."

California Attorney Qualifications.

Vern said, "In California, in order to be an attorney you must pass a first-year law student exam and the bar exam." This statement is wrong. In California, provided you went to law school, in order to be an attorney you need to pass the CA bar exam. The first year law student exam refers to persons who want to become attorneys without going to lawschool.

Charles

The first year law student exam refers to persons who want to become attorneys without going to lawschool.

Charles

See http://www.calbar.ca.gov/Public/Pamphlets/...ingALawyer.aspx number 7. Essentially, if you go to an ABA-approved law school, and you have at least two years of undergraduate study, you can be exempt from this exam. Otherwise, you must take it. This exam is often referred to as the "Baby Bar."

Back to the OT, why go through the extra work of an exam for a warranted officer? What about the work that is currently being performed? Contracts, contract mods, PNMs, etc, so many things out there that can already be "graded."

The first year law student exam refers to persons who want to become attorneys without going to lawschool.

Charles

Thanks, Charles, but that, too, is wrong. The first-year law student exam is also required for persons who attend an unaccredited law school. See the Rules of the California State Bar, Title 4, Admissions and Educational Standards, Division 1, Admission to Practice Law in California, Rule 4.55, First-Year Law Students? Examination requirement:

A general applicant intending to seek admission to practice law in California must take the First-Year Law Students? Examination unless the applicant

(A) has satisfactorily completed

(1) at least two years of college work as defined by these rules; and

(2) the first-year course of instruction

(a) at a law school that was approved by the American Bar Association or accredited by the Committee when the study was begun or completed; and

( the law school has advanced the person, whether or not on probation, to the second-year of instruction; or

( is exempt by reason of study in a foreign law school as provided by these rules.

Rule 4.55 adopted effective September 1, 2008.

In California you can study law (1) in an accredited law school, (2) in an unaccredited law school (which includes correspondence schools), (3) in a judge's chambers or a law office, or (4) by any combination of (2) and (3). See Rule 4.26, Legal Education:

General applicants for the California Bar Examination must

(A) be graduates of law schools approved by the American Bar Association or accredited by the Committee; or

( demonstrate that in accordance with these rules they have

(1) studied law diligently and in good faith for at least four years in a law school registered with the Committee; in a law office; in a judge?s chambers; or by some combination of these methods; or

(2) met the requirements of these rules for legal education in a foreign state or country; and

( C) have passed or established exemption from the First-Year Law Students' Examination.

Rule 4.26 adopted effective September 1, 2008.

http://rules.calbar.ca.gov/LinkClick.aspx?...&tabid=1227

I learned that the following lawyers did not go to (or graduate from) law school:

Abraham Lincoln,

John Jay (first chief justice of the Supreme Court)

William Wirt (former U.S. attorney general)

John Marshall (another chief justice of the Supreme Court),

Roger Taney (yet another chief justice),

Robert Storey (a former president of the American Bar Association), and

Clarence Darrow (!)

Clarence Darrow is the one that surprised me. He quit after one year of law school and then studied in a law office. And no, that does not change my views about CO exams.

Vern, my response was intended for the majority of persons who take this monster exam. I should have been more specific. What the "basia" can I do? Also, active attorneys who have practiced for so many years are allowed to take the California attorneys' exam. So, it appears anyone in CA can become an attorney

If anyone is considering Law School, please go to an ABA accredited school. Your options are limited if you do not attend an ABA school.

My incoherent ramblings on KO testing

IMO, bar exams are a form of Intellectual-Darwinism. I do not think bar exams other than the MPTs test what attorneys do. Should KOs have a similar test? If I am a bad test taker then I would argue no we should not have bar exam type tests. It is unfair, it is biased, it does not test what KOs know and do. If I am a good test taker then I may argue KOs must maintain minimum levels of professional competency, blah, blah, therefore, a test is necessary to measure, maintain, and perform quality control on these standards. Consequentially or perhaps beneficially, having bar exam type tests limits persons who can become KOs. Limiting KO's could potential increase KOs salaries (supply and demand). For example, AMA. Bad example, ABA. Most professions have tests. Why should KOs be any different? Or are KOs merely quasi-professionals?

charles:

I think that in the fifth sentence of your fourth paragraph you meant to say: It is unfair, it is biased, it does not test what KOs know and do.

Yes, I meant not thank you!

This topic was closed, modified, and an account deleted. The account was deleted because it was created and used as a bogus multi-user account. The rule on the Wifcon.com Forum is: one account, one individual user. I will be adding that to the list of rules later today.