NAS Polygraph Report

[George W. Maschke]

This message thread is for discussion of the National Academy of Sciences polygraph report and press conference, which took place at 11:00 A.M. Eastern. A recording of the conference is available at the NAS website in RealPlayer format at:
 
http://video.nationalacademies.org/ramgen/news/100802.rm  
 

[George W. Maschke]

The full text of the NAS polygraph report may be read on-line here:
 
http://www.nap.edu/books/0309084369/html/
 

[Drew Richardson]

Although there is much to be absorbed and gleaned from the NAS report and press conference over the coming days and weeks, there are a few take home points that are immediately available to us on face value:

(1) The committee distinguished between specific issue testing (criminal testing) and screening applications, specifically the counterintelligence screening exams used by various federal agencies for employees and prospective employees.

(2) The committee suggested that there was a great need for basic and applied research, but very clearly indicated that it should not be conducted by those who had any connection or vested interest in polygraphy.

(3) Although the committee did allow that specific issue testing had an accuracy somewhat greater than chance and substantially less than perfection, it was clear that it associated absolutely no validity with polygraph screening.

(4) Of great significance to the readers of this site, the panel suggested that there was no basis for any agency using polygraph screening as a diagnostic instrument for screening job applicants.  Although they were hesitant to put a number on the number of false-positive stung victims, they suggested it could be a substantial number of the tens of thousands of people subjected yearly to this form of testing.

(5) The committee was particularly concerned with the damage likely being done to national security by relying on polygraph results in terms of catching spies.  It clearly stated that no spy had ever been caught through the use of polygraphy and there was no reason to believe that any ever would be so identified.  It further expressed concern that a spy who had passed a polygraph exam would, in essence, be given a free pass to commit additional acts of espionage due to agency overconfidence given to polygraph results.

(6) The committee discussed the notion of polygraph utility and deterrence.  Although they indicated that there was some anecdotal suggestion of such, that it was not supported an any way in the research literature, it was not/had not been quantified, and that it was dependent (at least in the long term) on the concept of validity which was lacking for this application (polygraph screening).  Presumably even their press conference reduced the utility and deterrent factor by letting the public know in unmistakable terms that polygraphy (screening) was not a valid instrument for detecting lies.

(7) The committee was very careful, in following its original charter, not to tell federal and other agencies what to do with their individual programs.  They were also very diligent to make the scientific considerations abundantly clear.  The conclusions to be drawn and actions to be taken are unmistakable.  Polygraph screening should be discontinued and never should have been a sole basis for denying employment, placing employment in jeopardy, or of implicating or clearing an individual with regard to espionage or any other activity for which polygraph screening has been used as a diagnostic instrument.



I believe the ultimate take-home message will be that polygraphy should be limited to specific issue testing, polygraph screening should be eliminated as any sort of diagnostic tool/instrument and that considerably more effort and monies should be expended upon research (basic science and applied) and that these efforts should in no way be associated with the polygraph community.  The one suggested source for coordination and funding of these research activities was the National Institutes for Health.  

Drew Richardson

[Marty]

No surprises other than the GKT seems to have less significance than I expected, but then the research there is pretty skimpy.

I was curious about the very small group that is doing research on countermeasures (Honts). Research in that area is even more confounded by uncontrolled variables and having a small number of researchers makes bias, even unintentional bias, a more likely component. The report noted the need for independent lab confirmations here to assess countermeasure impact.

I completely agree with the need to do serious, non-incestuous research which is so typical of marginally scientific processes. Look what we have today. People take homeopathic medicine seriously and it is total garbage.  Chiropractic isn't much better. They both tend to produce such "research."

How about magnet therapy?

-Marty

[Twoblock (Guest)]

Dr. Richardson

I haven't read all or begun to digest this report but, it appears to me that the NAS is saying, even in a criminal issue test, that polygraphy is valid only if the examinee is naive or dumb enough to believe lies told to him/her by the operator and gives a confession. (I have difficulty labeling them interrogators). If an examinee maintains innocence, then there is no validity given to the machine or it's operator. Guilt still cannot be proven by polygraphy. Voo-doo is still voo-doo.

Your comments please.

[Mark Mallah]

Drew, thanks for the quick take on the report.  I only scanned the executive summary so far, but I think you hit the salient points.  The polygraph community will probably claim victory because specific issue testing was "vindicated", albeit cautiously.

There may be a way for them to get around the unsupportable use of screening: a screening followed by a "specific issue" exam.  They do a general screening on the employee population.  Let's say they get 1,000 deception indicated results.  Then they give all of those 1,000 what they call a specific issue test, say on espionage.

Are those follow up tests truly "specific issue" tests, or are they screening tests masquerading as specific issue tests?  Your comments Drew, would be appreciated.

[Skeptic]

I note that, in an AP report on the subject, Prof. Frank Horvath desperately tried to change the subject with the following quote:

"I would have to ask what tool they have in mind to replace polygraph screening,'' Horvath said. ``If we wanted to catch a spy or an applicant bent on spying, what would we do? What technique would we use to do that? There is no alternative right now to polygraph testing, and that's why it is used in spite of its shortcomings.''

(U.S. Advised Against Polygraphs--AP)

One might as well justify Tarot card readings for applicant screening on the grounds that "we have nothing better".  What a specious thing to say, especially since the report specifically recommended alternatives to continued polygraph screening.  Horvath obviously hasn't even read that on which he's commenting.

It's a bad day for polygraph proponents.

I also noticed the AP story got several facts wrong, including the statements:

"Polygraphs measure heartbeat, blood pressure and other factors that are known to change when people are under stress, as they are when they lie."

(Certainly, this may happen, but people don't necessarily react with stress when they lie, nor do they fail to do so when they don't.)

"Most uses involve examining individuals about a specific crime, and in those cases the machines can tell the difference between lies and truth 'at rates well above chance, though well below perfection,'' the panel concluded."

(The vast majority of polygraphs given in this country are for security screening purposes, not specific-issue testing.)

At least they did mention that no spy has ever been caught by the polygraph.

Skeptic

[Drew Richardson]

Twoblock,

I believe you are correct in your recollection that Dr. Fienberg did in this morning's press conference mention that the validity of specific issue (criminal) testing might be affected by examiner and examinee perceptions.  As I have opportunity to read the report (largely and properly focused on polygraph screening), I will try to revisit and answer your question more completely.  


Mark,

That which distinguishes a specific issue test from a screening test is whether or not the issues covered by the polygraph exam are known to have occurred.  If not, the exam is a screening exam or fishing expedition.  A second or any number of subsequent examinations that focus on one or more issues (still) not known to have occurred would still amount to nothing but a fishing expedition and certainly not a confirmatory test in any recognized form.

[Skeptic]

That which distinguishes a specific issue test from a screening test is whether or not the issues covered by the polygraph exam are known to have occurred.  If not, the exam is a screening exam or fishing expedition.  A second or any number of subsequent examinations that focus on one or more issues (still) not known to have occurred would still amount to nothing but a fishing expedition and certainly not a confirmatory test in any recognized form.

Dr. Richardson,
The only change I could see is that your breakdown test may now involve a population with a higher percentage of truly deceptive people (higher false positive than false negative rate).  If this is true, then by default you should now improve the number of people falsely flagged as deceptive.

Of course, your test also now involves a population of innocent people who have had their confidence in the machine and their ability to pass a polygraph shaken.  Those innocent people may also now feel that they are under suspicion or that fingers are being pointed in their direction.  Additionally, a breakdown test may ask questions that are more emotionally charged than the initial polygraph.

Skeptic

[Marty]

Drew, George,

Did anyone notice that the report pointedly opposed employee screening? Please note, this was not prospective employee screening, where different statistics apply, as Martin and Terris(1981) pointed out.

-Marty

[Skeptic]

Pages 93 through 95 of the NAS report were especially telling (transcribed here in part):

"In the course of our study we have seen or heard numerous disturbing allegations about the way polygraph research decisions have been made, particularly in federal agencies that have supported this reasearch.  We have seen or heard reports of researchers being prohibited from presenting studies at professional society meetings (see, e.g., Honts, 1994: Note 5); a report of a researcher being required to remove his name from a refereed journal article, apparently because the content displeased his employer (Furedy, 1993); a report of potentially inflammatory findings being suppressed and recalled from distribution; and various reports of researchers having been removed summarily from their duties or their positions, with reasons to believe that this might have been done because of the directions or results of their research.  These reports are not ancient history, though they are not current either: most appear to have dated from the early 1990s...

"Also, we were adivsed by officials from DOE and DoDPI that there was information relevant to our work, classified at the secret level, particularly with regard to polygraph countermeasures.  In order to review such information, several committee members and staff obtained national security clearances at the secret level.  We were subsequently told by officials of the Central Intelligence Agency and DoDPI that there were no completed studies of polygraph countermeasures at the secret level...

"These experiences leave us with unresolved concerns about whether federal agencies sponsoring polygraph research have acted in ways that suppress or conceal research results or that drive out researchers whose results might have questioned the validity of current polygraph practice.  If the agencies have done or are doing these things, the result would be to introduce a pro-polygraph bias into polygraph research in general...

"Issues of conflict of interest reflect a serious structural problem with polygraph research.  For the most part, the scientists involved in this area and the agencies involved in sponsoring and funding this research have a vested interest in supporting particular sets of conclusions about the reliability and validity of the polygraph (Levey, 1988)...

"We find the general quality of research on the criterion validity of the polygraph to be relatively low.  This assessment agrees with those of previous reviewers of this field."

[Marty]

Skeptic,

Pages 93 through 95 of the NAS report were especially telling (transcribed here in part):

"Also, we were adivsed by officials from DOE and DoDPI that there was information relevant to our work, classified at the secret level, particularly with regard to polygraph countermeasures.  In order to review such information, several committee members and staff obtained national security clearances at the secret level.  We were subsequently told by officials of the Central Intelligence Agency and DoDPI that there were no completed studies of polygraph countermeasures at the secret level...

This is entirely consistent with the belief that the effectivity of the polygraph depends on widespread, public acceptance of it as being nearly 100% accurate. Probably medical doctors would feel the same about placebo in the absence of more effective modalities.

-Marty

[Skeptic]

Drew, George,

Did anyone notice that the report pointedly opposed employee screening? Please note, this was not prospective employee screening, where different statistics apply, as Martin and Terris(1981) pointed out.

-Marty

Marty,
I think the NAS report was intended primarily to address DOE screening of existing employees.  However, they did write in their conclusion (page 170, transcribed here in part):

"Preemployment Screening The relevance of available research to preemployment polygraph screening is highly questionable because such screening involves inferences about future behavior on the basis of polygraph evidence about past behaviors that are probably quite different in kind.  The validity for such inferences depends on specifying and testing a plausible theory that links evidence of past behavior, such as illegal drug use, to future behavior of a different kind, such as revealing classified information.  We have not found any explicit statement of a plausible theory, let alone evidence appropriate for judging either construct or criterion validity for this application.  Conclusions about polygraph accuracy for these applications must be drawn by educated extrapolation from research that addresses situations that differ systematically from the intended applications."

Thus, while they attacked polygraph screening in general for its lack of validation, they went further and called into question the entire rationale for polygraph interrogations as typically used in pre-employment screening.

However, they do point out that false positives in preemployment screening are likely to cost both government and applicant less money.

Skeptic

[Skeptic]

Skeptic,


This is entirely consistent with the belief that the effectivity of the polygraph depends on widespread, public acceptance of it as being nearly 100% accurate. Probably medical doctors would feel the same about placebo in the absence of more effective modalities.

That last sentence is key -- "in the absence of more effective modalities".  Furthermore, if your placebo is killing off a good chunk of the people on whom you're using it, perhaps secrecy wouldn't be the best idea...

Skeptic

[polypoppolypop]

Skeptic, that is one powerful excerpt.  I love it.