Very Hot Topic (More than 25 Replies) As a full member, I call BS on the APA's polygraph "testing" accuracy claims (Read 71026 times)
Paste Member Name in Quick Reply Box Dan Mangan
God Member
*****
Offline



Posts: 640
Joined: Jul 31st, 2014
As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Sep 9th, 2016 at 12:07am
Mark & QuoteQuote Print Post  
On its own web site, the APA makes this most suspect claim:

"APA examiners are able to attain accuracy rates exceeding 90 percent."

People, I've been studying, analyzing and conducting polygraph "testing" for nearly twelve years.

In my professional opinion, the APA's 90+%-accuracy claim is a grotesque -- and commercially motivated -- assertion that lacks tangible proof.

Attention, Ray Nelson: Speaking in "Wizard of Oz" terms, you are the little man behind the big curtain that showcases the APA's exuberant claim.

So...show us APA members -- and others -- proof of your findings.

A few questions about your "meta-analysis"...

>Who conducted the "tests"?
>Were the "tests" QA reviewed? If so, by whom? Like-minded polygraph operators employed by the same entity?
>How many different polygraph-school graduates were represented?
>What was used to measure ground truth? Confession?
> What percentage of private examiners are in the APA's "meta-analysis" accuracy formula?
>How many unique government examiners were "used" in your meta-analysis?

We look forward to your explanation.

[cue crickets]
« Last Edit: Sep 9th, 2016 at 1:36am by Dan Mangan »  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Aunty Agony
Very Senior User
****
Offline



Posts: 134
Joined: Aug 15th, 2011
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #1 - Sep 9th, 2016 at 12:26pm
Mark & QuoteQuote Print Post  
Dan Mangan wrote on Sep 9th, 2016 at 12:07am:
[the APA site claims] APA examiners are able to attain accuracy rates exceeding 90 percent.

Well, if a polygrapher were presented only with subjects who were guilty, and if he pursued a policy of failing everyone, he would perforce attain an accuracy of 100%. If this describes more than one APA member then the APA claim is true.

There are a lot of examiners out there. *Some* of them *somewhere* may be hitting any accuracy rate you care to name.

Aunty remains unimpressed by liars pretending to expose liars.
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Raymond Nelson
Guest


Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #2 - Sep 9th, 2016 at 1:14pm
Mark & QuoteQuote Print Post  
I call BS on Dan Mangan's published claims of ~100% accuracy,

http://www.mattepolygraph.com/2008_fieldstudy_quadritrack.html

and published claims that he is impervious to countermeasures

http://www.mattepolygraph.com/2008_rebuttal_quadritrack.html

Seems ironic that the man (Dan Mangan) who published claims of ~100% accuracy is critical of others.

But then again maybe not so ironic after all...

Dan Mangan's rhetoric and publications, together with his  money-back guarantee, seems to be nothing more than scraping the market for desperate customers.

If one were to actually read the publications there are some important details to notice. For example: we've encouraged people to try to learn think about test accuracy (polygraph or any test) as an issue of confidence intervals and not point estimates. There are a lot of other interesting and important examples and details and it would be nice to someday engage an intelligent conversation...

In the meantime, please tell us more about your money-back guaranteed polygraph results Dan.

/rn
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box George W. Maschke
Global Moderator
*****
Offline


Make-believe science yields
make-believe security.

Posts: 5925
Location: The Hague, The Netherlands
Joined: Sep 29th, 2000
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #3 - Sep 9th, 2016 at 1:41pm
Mark & QuoteQuote Print Post  
I think Dan's request for proof is reasonable. If the American Polygraph Association cannot prove its claim that "APA examiners are able to attain accuracy rates exceeding 90 percent" it should withdraw it.
  

George W. Maschke
Tel/SMS: 1-202-810-2105 (Please use Signal Private Messenger or WhatsApp to text or call.)
E-mail/iMessage/FaceTime: antipolygraph.org@protonmail.com
Wire: @ap_org
PGP Public Key: 316A947C
PGP Public Key (offline): 2BF4374B
Personal Statement: "Too Hot of a Potato"
Back to top
IP Logged
 
Paste Member Name in Quick Reply Box Dan Mangan
God Member
*****
Offline



Posts: 640
Joined: Jul 31st, 2014
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #4 - Sep 9th, 2016 at 2:17pm
Mark & QuoteQuote Print Post  
Ray, my money-back guarantee works like this:

If, subsequent to my "test" and opinion, countervailing ground truth is discovered in the form of evidence that is beyond reproach, the fee is fully refunded.

Now, Ray...please tell us about your "research."

>Was a countermeasure component involved? If not, why not?

>What's the split of govt/LE/private examiners in your research?

>How many unique examiners are in your research caldron? [Double, double toil and trouble; Fire burn, and caldron bubble...]
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Raymond Nelson
Guest


Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #5 - Sep 9th, 2016 at 6:40pm
Mark & QuoteQuote Print Post  
George,

If you can give me a link with the 90% point estimate I am willing to look into it. I do not have direct control over these things, but I do, as do others, have a voice.

If there is such a point estimate at the APA website I will have to assume it was a rounding of point estimates from the 2003 and 2011 report coupled with some copywriting by someone who did not analyze the data.

Unfortunately, people like rounded numbers and tend to prefer a single point estimate instead of learning to understand the meaning of scientific test data. This is similar to the problems where people prefer to buy a sense of confidence (over-confidence) even where it is not substantiated - they love to be told what they already think. and they love to be told they are right. Let's face it, being open-minded is a very difficult thing for a large number of people. All of this leads to a situation where a fella with questionable scruples could make a bit of money off of some other desperate and confused folks by offering to sell them something they really really want (for example: a sense of certainty, or a sense of security to fill a void of uncertainty or existential dread).

Here is a link to some summary information that includes confidence intervals.

http://www.polygraph.org/polygraph-validity-research

As is often the case when using tests to measure amorphous phenomena,  for which there is neither physical substance nor physical/linear unit of measurement (which is the reason we use a test instead of a measurement or deterministic observation), we are sometimes interested in the worst case scenario. We have to be careful about the impulse to focus on the happy numbers simply because we like them better, and we need to be especially careful to remember that all test results are probability statements and by definition imperfect.

Using point estimates is simplistic and not a smart way to describe a test result. Test results are statistical classifiers that attempt to make predictions about something that cannot be subject to deterministic inspection or physical measurement - as is often the case when attempting to discuss future events such as the proportions of voter and outcomes in an upcoming election or the proportion of future polygraph test results that can be reasonably expected to be correct or incorrect.

As you know, the way science works is that we can have evidence to support a conclusion, and often the evidence is noisy and somewhat confusing. So we like lots of evidence and replication. And we like to see the evidence and understand how it was analyzed. Sometimes we are satisfied by the evidence. Sometimes we are not.

And there is always more to learn because it is probably not humanly possible to know everything (which is why claims of ~100% are immediately suspicious for scientific fraud because they would seem to suggest we have somehow learned everything.)

as always,

.02

/rn
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Dan Mangan
God Member
*****
Offline



Posts: 640
Joined: Jul 31st, 2014
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #6 - Sep 9th, 2016 at 7:54pm
Mark & QuoteQuote Print Post  
Ray, the APA's 90+% accuracy propaganda statement can be found on the APA web site under "About Us."

Looky here... http://www.polygraph.org/about-the-apa

The APA's capriciously exuberant accuracy claim -- APA examiners are able to attain accuracy rates exceeding 90 percent [ROTFLMAO] is roughly midway through the page.

Indeed, the APA has a sordid history of exaggerated polygraph accuracy.

So why do they keep on doing it?

Here's my theory: Polygraph "testing" is mainly about money.

Do you disagree, Ray?
« Last Edit: Sep 10th, 2016 at 2:22am by Dan Mangan »  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Wandersmann
Very Senior User
****
Offline



Posts: 228
Joined: Jun 2nd, 2015
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #7 - Sep 10th, 2016 at 5:28am
Mark & QuoteQuote Print Post  
George W. Maschke wrote on Sep 9th, 2016 at 1:41pm:
If the American Polygraph Association cannot prove its claim that "APA examiners are able to attain accuracy rates exceeding 90 percent" it should withdraw it.


I do not understand how they can claim accuracy percentage of any kind.  To prove accuracy, wouldn't they need to conduct and independent investigation of polygraph test results and prove through separate physical evidence that the results of the polygraph match the results of scientific investigation ?  In other words, 90 % of polygraph interviews would have to result in confessions of guilt that can be verified through parallel investigation.  That doesn't happen.
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Dan Mangan
God Member
*****
Offline



Posts: 640
Joined: Jul 31st, 2014
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #8 - Sep 10th, 2016 at 12:45pm
Mark & QuoteQuote Print Post  
Quote:
Using point estimates is simplistic and not a smart way to describe a test result. Test results are statistical classifiers that attempt to make predictions about something that cannot be subject to deterministic inspection or physical measurement - as is often the case when attempting to discuss future events such as the proportions of voter and outcomes in an upcoming election or the proportion of future polygraph test results that can be reasonably expected to be correct or incorrect.


If that's the way you really feel, Ray, then it appears that NAS was far more realistic than the APA when they said:

...we conclude that in populations of examinees such as those represented in the polygraph research literature, untrained in countermeasures, specific-incident polygraph tests can discriminate lying from truth telling at rates well above chance, though well below perfection.

Why then, does the APA insist on using point estimates at the upper end of the "happy numbers" to characterize polygraph accuracy?
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Raymond Nelson
Guest


Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #9 - Sep 12th, 2016 at 6:51pm
Mark & QuoteQuote Print Post  
Here is a link to the Mangan publication that claims ~100% accuracy - and also claims to "nullify the effects of countermeasures" (last sentence of the report, on page 7).

http://www.mattepolygraph.com/2008_fieldstudy_quadritrack.html

And here is a link to the APA 2011 report that seems to be peppered with confidence intervals and a lot of information about the difficulty/impossibility in attempting to describe the complexities of test accuracy with a single number.

https://apoa.memberclicks.net/assets/docs/polygraph_404.pdf

And a more recent APA publication on confidence intervals - including an advisement (page 76)  to consider the worst case scenario, meaning the lower limit for accuracy estimations and the upper limit for test error estimation.

https://www.researchgate.net/publication/306292708_Five_minute_science_lesson_St...

So, I suggest we let readers decide for themselves about who here is selling happy numbers ...

, as always

.02

/rn


  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Dan Mangan
God Member
*****
Offline



Posts: 640
Joined: Jul 31st, 2014
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #10 - Sep 12th, 2016 at 7:51pm
Mark & QuoteQuote Print Post  
Ray, my study -- actually, it was more of a "micro-survey" in that it reflected the experience of just one (1) highly seasoned examiner using the MQTZCT -- is clearly an outlier. The study's conclusion should not be seen as being applicable to the polygraph operator population at large, certainly not anyone who wasn't trained by Backster and Matte.

Unlike your dumbed-down, simplified, paint-by-numbers approach to (allegedly) scientifically robust polygraph "testing" -- designed for the high-school graduates that make up the vast majority of APA's membership -- the Quadri-Track technique is purely expertise driven.

On its own web site -- www.polygraph.org --the American Polygraph Association brazenly claims the following: APA examiners are able to attain accuracy rates exceeding 90 percent.

Is that bold assertion by the APA true, Ray? Do you stand by that statement? Should polygraph consumers believe it?

As for the happy numbers, I'm not selling anything other than reality about polygraph "testing."

As the FAQ page on my web site clearly states, incident-specific polygraph "testing" is only about 65% accurate, generally speaking, even under the most favorable of conditions.
« Last Edit: Sep 12th, 2016 at 10:11pm by Dan Mangan »  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Dan Mangan
God Member
*****
Offline



Posts: 640
Joined: Jul 31st, 2014
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #11 - Sep 13th, 2016 at 11:31pm
Mark & QuoteQuote Print Post  
Quote:
And here is a link to the APA 2011 report that seems to be peppered with confidence intervals and a lot of information about the difficulty/impossibility in attempting to describe the complexities of test accuracy with a single number.

https://apoa.memberclicks.net/assets/docs/polygraph_404.pdf

And a more recent APA publication on confidence intervals - including an advisement (page 76)  to consider the worst case scenario, meaning the lower limit for accuracy estimations and the upper limit for test error estimation.

https://www.researchgate.net/publication/306292708_Five_minute_science_lesson_St....



Ray, as past president (and a years-long political operative) of the American polygraph association, please explain why the APA chooses to bury the critical information that you cite in obscure publications that no polygraph consumer is likely to ever read, and features this brazen claim -- APA examiners are able to attain accuracy rates exceeding 90 percent -- on its web site under "About the APA."

Is that choice of positioning a classic "bait-and-switch" tactic, perhaps driven by marketing considerations, on the part of the APA?


Please explain.
« Last Edit: Sep 14th, 2016 at 12:52am by Dan Mangan »  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Raymond Nelson
Guest


Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #12 - Sep 14th, 2016 at 9:05pm
Mark & QuoteQuote Print Post  
Dan,

I do not write or manage the content for the AP website.

What I do is try to learn more and provide information and knowledge to the polygraph profession and to the public. I wish more people would read. And I wish more people would read intelligently. Toward that objective, part of what I have done is to try to make instructional materials available to help people better understand the direct application of science and testing principles to the context of  polygraph, lie detection and credibility assessment.

Your strategy would seem to be the one about "tell a lie long enough and people will believe it, and the bigger the lie they more they believe it."

The first lie was a published claim of ~100% accuracy. Followed by the supporting lie that your favorite technique can "nullify the effects of countermeasures." (Dan Mangan’s written and published words). 

The next lie was in re-inventing yourself as the anti-polygraph polygraph examiner  accusing everyone else of exaggerated claims (hoping they will either not notice or forget your first lie about ~100% accuracy).

Follow that with the lie of professing to be the champion of reason while publishing conclusions that are un-replicatable (~100% accuracy, “nulify countermeasures,” etc.) and unaccountable (only those who are similarly anointed by the guru could possibly understand you), and therefore inconsistent with and disconnected from reason (having more to do with mystified experteeism.)

Probably there are some people who are desperate enough to want your services and pay your fees on the off chance that you can magically pull a rabbit out of the hat for them. 

I don't know whether you actually believe your published claims of ~100% accuracy, or whether that is just convenient marketing hype. I do know that Matte's published hypothesis are not consistent with the evidence - polygraph machines cannot discriminate between fear and hope, nor can they determine the reasons for these emotions. And so your reliance on an unscientific claim would seem you put you solidly in the realm of pseudoscience. 

In the end what you offer is this: "trust me because I am an expert, having been anointed by the hand of the guru.."  The corollary to this is the message "don't trust anyone else without your approval." 

The problem with unscientific expertise as the basis for your test results and conclusions is that you are basically free to give subjective results - perhaps even any result you want to give or any result someone wants to purchase.

I do understand the financial and business and professional economic motivation that would make a person want to perpetuate the business model of "trust me I'm an expert" - when subjective, unscientific, unquantifiable, un-replicatable, "expertise" is all you have to sell.

And of course - when selling subjective expertise and unrealistic solutions - the more you sell it and the more you boastful and outrageous your claims the more likely you are to succeed. Think about it, when selling nonsense, it would sell nothing if one were to advise people that it is nonsense. 

Development of a scientific test requires that we first understand that a test and a test result is always going to be an imperfect and probabilistic assessment of some interesting and important thing for which we cannot evaluate with simple and perfect deterministic observation nor with a direct physical/linear measurement.  So if the test result is greater than chance and less than perfection, then the goal of studying the test is to obtain and study data to improve our knowledge about the confidence intervals that describe the things we might say about the test result.

The difference between old-school-experteeism and a scientific test is that a test result is something that can be reproduced with some expected frequency. The old-school mystified expert approach tends to be so esoteric that reproduction of analytic conclusions is not a realistic thing (think back to the decades before numerical and statistical analysis when polygraph examiners might have been reluctant to allow other examiners to look at there data).  Even today we sometimes professionals play this game we will see a lot of subjective experteeism when different experts disagree while using subjective, inscruitable, and under-quantified (or unquantified) analytic methods.

In the worst cases esoteric process give way to wholesale mysticism in which people tend to make all kinds of outrageous and laughable claims (for example: ~100% accuracy, or the ability to "nullify the effects of countermeasures.") for the convenience of self-promotion or for the convenience of a single case outcome.

Of course if a test result is based on a structured process then it is reproducible (and non-mystical) – and we can begin to study and know the range of probabilities for which we can expect the test to lead us to correct conclusions and effective decisions.

In the words of W. Edward Demming "If you can't describe what you do as a process, you don't know what you are doing." And, of course, if we can describe what we do as a process then we could teach most intelligent persons to do it successfully using the structured process (no mysticism needed). And ultimately we begin to automate any well structured process.

About a half a century or more ago there was discussion was about the need for standardization of processes. This was in a day when people could not imagine the potential for computerized automation. Today we can much more easily imagine what computers, machines, robots and algorithms can do. Automation is today what standardization was 50 years ago. Of course there are some ethical and scientific discussions to be had around how exactly we use computers in human decision making – but ignoring this is not a wise solution.

Automation of a test, either completely or wherever possible would reduce the impact of both competency and random human variation. But automation will only succeed if a test actually works at rates significantly greater than chance. Automating the process and result of a pseudoscientific test would not be fun, because then the “expert” would no longer be free to subjectively adapt the test results to the solution that is most socially and professionally convenient to the expert.

Standardization of processes and process automation will only succeed if results can be reasonably expected to occur most often in a usable range (can you say "confidence interval"). If not, if results are mere random chaos or random guessing, then neither process standardization nor process automation will improve the consistency of outcomes. If test data and test results are mere random chaos then professional and economic survival will depend wholly on salesmanship (the ability to sell confidence in nothing more than one's expertise). 

Which begins to beg the question, why, if even the NAS agrees that polygraph results are significantly greater than chance and still less than perfect (which could be said about any and all tests), why does someone cling to an impression that the test and test result is mere subjective chaos for which he can inject his esoterica and mystified "expert" opinion in ways that are beyond the scrutiny of others who have not been anointed by the guru??? 

One possible answer has to do with a lack of competence. I'll explain. A standardized (non-automated) test process still requires a level of competency for test results to occur within some expected confidence interval (e.g., greater than chance, less than perfect). Without some competency in test administration the results might be mere chaos, in which case professional economic survival will depend on some fast talking and slick marketing - and a customer base that prefers to purchase "expertise" as if it is disconnected from science.


,hoping for the future,



/rn
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Dan Mangan
God Member
*****
Offline



Posts: 640
Joined: Jul 31st, 2014
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #13 - Sep 14th, 2016 at 11:25pm
Mark & QuoteQuote Print Post  
Ray, did the American Polygraph Association lie when they claimed 98% accuracy for fifteen years?

The American Polygraph Association now makes this claim: APA examiners are able to attain accuracy rates exceeding 90 percent.

Is the APA's current claim of 90% accuracy a lie?


Also, Ray...   

If two different -- but equally qualified (in their own right) -- polygraph examiners tested the same subject on the same specific incident using the same RQs, whereby one examiner ran a Utah technique and the other conducted a MQTZCT, what is the likelihood the test results would be in agreement?

If the likelihood of agreement is high, does that make the Utah test pseudoscience, or does it make the MQTZCT a scientifically robust "process"?

Years ago, Krapohl told me in an email that the MQTZCT was as good as the Utah test. Was he wrong?

« Last Edit: Sep 15th, 2016 at 1:47am by Dan Mangan »  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Aunty Agony
Very Senior User
****
Offline



Posts: 134
Joined: Aug 15th, 2011
Re: As a full member, I call BS on the APA's polygraph "testing" accuracy claims
Reply #14 - Sep 15th, 2016 at 2:53pm
Mark & QuoteQuote Print Post  
The question on the floor is:

Dan Mangan wrote on Sep 9th, 2016 at 12:07am:
On its own web site, the APA makes this most suspect claim:

"APA examiners are able to attain accuracy rates exceeding 90 percent."

In my professional opinion, the APA's 90+%-accuracy claim is a grotesque -- and commercially motivated -- assertion that lacks tangible proof.

So...show us APA members -- and others -- proof of your findings.

Answers submitted so far include:

(1) The person posing the question has himself claimed 100% accuracy in a similar context. 475 words.

(2) The person posing the question has claimed himself immune to countermeasures. 591 words.

(3) Any accuracy claim in the form published by the APA should be ignored because it reduces accuracy to a single number. 1364 words.

(4) The person who provided (1) thru (3) is not in a position to speak knowledgeably on the question. 110 words.

Aunty submits a thesis for our readers consideration: if a respondent is not willing nor able to provide the answer to a question then he should not filibuster the discussion with over 2500 words about something else.
  
Back to top
 
IP Logged
 
As a full member, I call BS on the APA's polygraph "testing" accuracy claims

Please type the characters that appear in the image. The characters must be typed in the same order, and they are case-sensitive.
Open Preview Preview

You can resize the textbox by dragging the right or bottom border.
Insert Hyperlink Insert FTP Link Insert Image Insert E-mail Insert Media Insert Table Insert Table Row Insert Table Column Insert Horizontal Rule Insert Teletype Insert Code Insert Quote Edited Superscript Subscript Insert List /me - my name Insert Marquee Insert Timestamp No Parse
Bold Italicized Underline Insert Strikethrough Highlight
                       
Insert Preformatted Text Left Align Centered Right Align
resize_wb
resize_hb







Max 200000 characters. Remaining characters:
Text size: pt
More Smilies
View All Smilies
Collapse additional features Collapse/Expand additional features Smiley Wink Cheesy Grin Angry Sad Shocked Cool Huh Roll Eyes Tongue Embarrassed Lips Sealed Undecided Kiss Cry
Attachments More Attachments Allowed file types: txt doc docx psd pdf bmp jpe jpg jpeg gif png swf zip rar tar gz 7z odt ods mp3 mp4 wav avi mov 3gp html maff pgp gpg
Maximum Attachment size: 500000 KB
Attachment 1:
X