Polygraph Utility – How Does Polygraph Compare to Other Forensic Tools?

[skip.webb]

In an article published in the Journal of Forensic Science 23 (3), 596-601, published in 1978 (almost 30 years ago) by Jan Widacki and Frank Forvath, entitled "An experimental investigation of the relative validity and utility of the polygraph technique and three other common methods of criminal identification.", researchers compared the polygraph, handwriting analysis, eye witness testimony and fingerprints in a mock crime scenario.  80 subjects (college students) were divided into 20 groups of 4.  In each sub-group there was one guilty participant and three innocent ones.  The task of the guilty participant was to go to a particular building; present themselves to a person whom they did not know before hand;  give an envelope to the specified person; receive a package using a fictitious name; sign a receipt for the package using the fictitious signature and then steal the contents of the package.

All participants were then subjected to a polygraph.  The document receipts and envelopes were subjected to handwriting analysis compared against handwriting standards supplied by all participants.  Fingerprints collected from the envelope and receipts were compared to record prints of all participants.  The eye witnesses to the crimes were provided full front photographs from which to select the perpetrator who signed for and stole the packages.  All forensic technicians were blind to the identity of the perpetrators as was the eye witness but all technicians were aware that one in each group of four was the perpetrator and three were innocent as were the eyewitnesses.

The following results were recorded with inconclusive results removed for all:

Polygraph -18 correct identifications, 1 error and 1 inconclusive result for an over all accuracy of 95%.

Handwriting – 17 correct identifications, 1 error and 2 inconclusive results for an overall accuracy of 94%.

Eyewitness – 7 correct identifications, 4 errors and 9 inconclusive results for an over all accuracy of 64%.

Fingerprints – 4 correct identifications, 0 errors and 16 inconclusive results for an overall accuracy of 100%.

Submitter's comments:  I have provided this study for thoughtful consideration of those on this site who require sound evidence rather than unsupported assertions or ad hominid attacks on polygraph examiners. I realize that students were used so Mr. Maschke may object but as I previously pointed out, students are used in many studies as such studies are an academic endeavor usually conducted in a university setting.  I also realize that this is a mock crime paradigm and as such Mr. Maschke will assert that as there was no real fear of detection or consequences for the perpetrators, the results may not generalize to the public.  Actually, the reverse should be true.  One would assume if polygraph is able to correctly identify the perpetrator in a mock crime scenario then the intensity of reaction in a real life scenario would be greater and therefore easier to discern. I also realize that in the instant study, the polygraph examiners, although blind to the identity of the perpetrator in each group were aware that one of the four was the guilty party.  This does create an advantage but each of the forensic technicians as well as the eye witnesses shared that same advantage.   In summary, polygraph works.  Its utility in criminal investigations is very high.  Its validity compares very favorably with other forensic tools.  Its error rate compares very favorably with other forensic tools. One can argue that any error rate is unacceptable if it does harm to one innocent person.  If only we lived in such a perfect world.  Look back at the eye witness testimony results and ask your self the following questions: "If I am ever on a jury and I am faced with the testimony of an eye witness, how much weight will I give it?"  If I am ever a defendant in a case how much credence do I want the jury to give eye witness testimony?  Well, my friends, such testimony forms a major part of many trials.

[George W. Maschke]

Mr. Webb,

I don't deny that the polygraph may have utility as an aid to interrogation, to the extent that it may help to obtain confessions/admissions that an examinee might not have otherwise been willing to make. But validity is an altogether different question.

You write:

...I also realize that this is a mock crime paradigm and as such Mr. Maschke will assert that as there was no real fear of detection or consequences for the perpetrators, the results may not generalize to the public.  Actually, the reverse should be true.  One would assume if polygraph is able to correctly identify the perpetrator in a mock crime scenario then the intensity of reaction in a real life scenario would be greater and therefore easier to discern....

But the National Academy of Sciences, in its review  of the scientific evidence on the polygraph, disagrees (at pp. 3-4):

Laboratory studies tend to overestimate accuracy because laboratory conditions involve much less variation in test implementation, in the characteristics of examinees, and in the nature and context of investigations than arises in typical field applications.

Dr. Alan Zelicoff, M.D., conducted a statistical analysis (255 kb PDF) of data from polygraph field studies selected from an impeccably pro-polygraph source and found the data indicate that "if a subject fails a polygraph, the probability that she is, in fact, being deceptive is little more than chance alone; that is, one could flip a coin and get virtually the same result for a positive test based on the published data."

[nopolycop]

But Mr. Webb, in the study by Honts attempting to ascertain if countermeasures worked, the resultant rate for polygraphy in general was only 73% accurate.  Now, I realize that accuracy wasn't being specifically tested in that study, but it certainly was a "finding" of the study, and reported as such.

There also in not nearly enough information given regarding how the polygraph was administered to form any conclusions.  Did the person confess?  Did the polygraph give them away.  How were they interrogated?  

Lastly, what exactly were the "guilty" persons guilty of?  Following instructions.  No Mr. Webb, I am not impressed.  With all due respect,  if this is the type of "science" that needs to be resorted to to prove the utility of the polygraph, the polygraphy is in big trouble.

I wonder if this is one of the studies the NAS threw out as not having enough scientific protocols to even be considered.  Anyone know?

[nopolycop]

A few additional observations.  In the first paragraph quoted below, I believe you are WAY OFF BASE.  From what I have read about polygraphy, the more emotion that is attached to a particular event, the more likelyhood of incorrect findings, not the reverse.  Remember, the students were not really guilty of anything, they just followed directions.

#2. Huh?  How can you conclude this, Mr. Webb, based on this study?

#3.  I actually agree with you here, as long as the investigators do not rule out the person as a suspect if they "pass".  Can you say "Gary Ridgway?"

#4.  That is why a good criminal investigator does not rely solely on any investigative tool.

#5.  I am not sure I can agree with this claim, since the protocols of the study were not given.  I also find it interesting that in this analysis, an inconclusive is removed, whereas in the real world, investigators of crime have to deal with an inconclusive finding, both in the street and in court.  And, in the context of an employment screening, an inconclusive is oft times viewed as a failure.

I also realize that this is a mock crime paradigm and as such Mr. Maschke will assert that as there was no real fear of detection or consequences for the perpetrators, the results may not generalize to the public.  Actually, the reverse should be true.  One would assume if polygraph is able to correctly identify the perpetrator in a mock crime scenario then the intensity of reaction in a real life scenario would be greater and therefore easier to discern.

In summary, polygraph works.  

Its utility in criminal investigations is very high.  

Its validity compares very favorably with other forensic tools.  

Its error rate compares very favorably with other forensic tools.

[Sergeant1107]

I think the fact that all the technicians were told that in each group of four people there was one guilty person hopelessly skews the resulting data.  I think this study is worth very little in determining the effectiveness of anything included in it.

Knowing that one and only one person in each four-person group is guilty, the efficacy of randomly picking one person from each group would be 25%.  Would that make random choice effective enough to appear on the list of tools for criminal investigators?

If the technicians knew that one and only one person from each group was "guilty" all they had to do was look for any indications of guilt in each person.  They could have done that with a simple interview, or via a polygraph, CVSA, handwriting analysis, Tarot card reading, or whatever.  At the end of all four interviews they could just point to the person who exhibited the most "tells" that they were being deceptive.  I think that any police cadet with at least a couple of weeks at the academy could have correctly identified the majority of the guilty people.

It would have been somewhat more impressive if there had been one guilty party in the entire group of eighty people.  If there were, say, ten examiners and each one tested eight people, it would be interesting if seven of those examiners concluded that none of the subjects they tested were being deceptive.  It would be even more interesting if the one examiner who was testing the guilty party was able to correctly identify him or her.

[George W. Maschke]

I wonder if this is one of the studies the NAS threw out as not having enough scientific protocols to even be considered.  Anyone know?

The 1978 Widacki & Horvath study (note the mis-spelling of the latter's name in the initial post of this thread) didn't make the cut for serious consideration by the National Academy of Sciences and is not mentioned in its report, The Polygraph and Lie Detection.

[skip.webb]

George, I'm disappointed!  Are we down to critiquing typos on someone's name now?  The study cited here was not designed or intended to discover or investigate polygraph accuracy, but rather to compare polygraph utility and accuracy when compared to several other common forms of investigative tools routinely used in investigations.  I didn't present it as anything other than that.  As the NAS was interested in polygraph screening and that was their mandate, I doubt that their failure to include this study in their review was an indictment of the study, (as I'm sure you know) but your insulation might be misleading to others less knowledgeable.

[George W. Maschke]

Skip,

I pointed out the misspelling of Horvath's name not to embarrass you (I make make such mistakes too), because it could be significant for anyone searching for the report.

I'm perplexed by your argument that "(t)he study cited here was not designed or intended to discover or investigate polygraph accuracy, but rather to compare polygraph utility and accuracy when compared to several other common forms of investigative tools..." How can you seriously contend that on the one hand this study attempted a comparison of the utility and accuracy of investigative tools (or techniques) including polygraphy, and yet on the other hand maintain that it was not intended to investigate (among other things) polygraph accuracy? Especially when the study provides a table of accuracy rates obtained for each of the techniques (including polygraphy)?

The NAS report's omission of the Widacki & Horvath study from consideration didn't stem from the fact that their focus was on polygraph screening. Other polygraph studies not specifically addressing screening were indeed considered. The Widacki & Horvath study simply didn't make the cut.

[Sergeant1107]

The following results were recorded with inconclusive results removed for all:

Polygraph -18 correct identifications, 1 error and 1 inconclusive result for an over all accuracy of 95%.

Handwriting – 17 correct identifications, 1 error and 2 inconclusive results for an overall accuracy of 94%.

Eyewitness – 7 correct identifications, 4 errors and 9 inconclusive results for an over all accuracy of 64%.

Fingerprints – 4 correct identifications, 0 errors and 16 inconclusive results for an overall accuracy of 100%.

Why would the inconclusive results be removed, other than to favorably skew the results for the polygraph supporters?

If you believe it is appropriate to remove inconclusive results, you could easily be left with the following scenario:
One hundred subjects tested, one person correctly scored as "No Deception Indicated", the other ninety-nine scored as "inconclusive" = overall accuracy of 100%???

That doesn't seem to make a lot of sense.

[Barry_C]

If we counted "inconclusives" as errors, then coin-flipping would be more accurate than fingerprinting - and I think those cited numbers are high for what I've seen in the field.

If your doctor said he didn't know what was wrong with you after one test, would he be wrong in his diagnosis?  Wait, there is no diagnosis.

[Hunter]

So lets include the inconclusives and you have the following figures, and polygraph really wins big time Sgt.  


Polygraph -18 correct identifications, 1 error and 1 inconclusive result for an over all accuracy of 95%

18=90%  Correct
1= 5%    Inconclusive
1=5%.    Incorrect

Handwriting – 17 correct identifications, 1 error and 2 inconclusive results for an overall accuracy of 94%.

17=85%
  1= 5%
  2+10%
Eyewitness – 7 correct identifications, 4 errors and 9 inconclusive results for an over all accuracy of 64%.

7=35%
4=20%
9=45%


Fingerprints – 4 correct identifications, 0 errors and 16 inconclusive results for an overall accuracy of 100%.
4=20%
0=0%
16=80%

[raymond.nelson]

This is a good example of some of the complexities surrounding the generic term "accuracy."

digithead has alluded to this before, though the discussion is grossly incomplete.

Test designers know that all tests are imperfect, and have identifiable limitations in terms of types of accuracy they can achieve.

Accuracy with inconclusives included is commonly referred to as sensitivity and specificity. Accuracy without inconclusives might be thoughts of as NPV or PPV. Of course, calculation methods for these statistics can differ, and includes within-set and cross-set variants, depending on your bayesian or inferential emphasis or orientation, and the exact question which one seeks to answer.

One of the dangers here is the gross error of substituting classification accuracy (sometimes called PPV or NPV), for the likelihood or probability of a correct classification of any single case. They are not at all the same according to some of the more common systems of mathematical thinking.

Fingerprinting is a good example of a test with very low sensitivity, but very high accuracy. With some tests, like fingerprinting, specificity is of little interest. Positive matches are the only result of any usefulness.

more later.

r

[Brettski]

If we counted "inconclusives" as errors, then coin-flipping would be more accurate than fingerprinting - and I think those cited numbers are high for what I've seen in the field.

If your doctor said he didn't know what was wrong with you after one test, would he be wrong in his diagnosis?  Wait, there is no diagnosis.

I agree, inconclusive results should not be classified as errors. There are plenty of stronger arguments against polygraphs without approaching an argument that is an easy win for pro-polygraphs. However, if inconclusive results weren't held against the examinee as the APA claims, then hypothetically it should be possible for an applicant to get nothing but inconclusive results on a polygraph screening tests, and still get hired by XYZ governement agency. To my understanding, this is not the case.

[Sergeant1107]

Looking at this from a layman's point of view, it seems that in every polygraph exam the subject is either being truthful or deceptive.  The results of the polygraph should be a binary solution set; either truthful or deceptive.  If you polygraph one hundred subjects, correctly identify one as DI or NDI, and score the other ninety-nine as inconclusive, I don't see how that can indicate the polygraph is 100% accurate.  I think that would indicate the polygraph is 1% accurate, according to those numbers.

If you want to compare it to a medical diagnosis, you would have to compare it to a test that looks to identify the presence of one specific thing.  You would want to compare it to a test that also had results in a binary solution set.  If you tested one hundred people for the presence of influenza, and you correctly diagnosed one person as having the virus, but said you simply didn't know about the other ninety-nine, I don't think you'd be saying the influenza test was 100% accurate.  An accurate test for the presence of influenza would either detect the presence of the virus or confirm that the virus was not present.

It seems like a given that in every polygraph the subject is either being truthful or deceptive.  If the subjects in the study were asked a question such as, "Did you steal the contents of the package," it is likely they would all answer that they did not.  Their denial would either be truthful or a lie.  A test purported to detect deception should be able to determine which it is.

[Hunter]

According to your logic, fingerprints should never be used, they are not 100% accurate when you include the inconclusive rate.Somewhere there is a fallacy in your logic.  Maybe you should re evaluate what you have posted and see if you can correct it.