Dan,

I cannot read minds any better than you or anyone else. For this reason I cannot say what people knew or believed when they published the 98% accuracy rate so many years ago. 

What we do know is that attempting to construct field studies from confession confirmations can tend to inflate our observed sampling accuracy because we are not likely to obtain confessions from false-negative errors (unless an examinee is unusually conscientious about informing us of the FN error) and not likely to obtain confessions from false-positive errors (unless we engage in activities that lead to false confessions). 

So I can only assume that people were perhaps optimistic when they observed sampling results that seemed to concur with a desired narrative about test accuracy. 

The same thing seems to have occurred with your publication - which was called a "...field study on the validity..." (with no mention about a survey - though the design was clearly non-experimental) - when you rushed to publish ~100% accuracy and followed up with a rebuttal that re-argued your published conclusions in response to published objections from the scientific community. 

As to the expected concordance between MQTZCT and other techniques such as those developed at the University of Utah - I don't think we know. The reason we do not know is because we actually have no realistic published estimate of the test sensitivity, specificity, error rate and precision of the MQTZCT. Any statement would be mere speculation - which would be an inherently inadequate assertion. Or we have to endorse the unrealistic and obviously flawed published survey information on the MQTZCT. 

Strictly speaking, point estimates from sample data are always wrong. This is because the probability is asymptotically zero that a sampling point estimate is an exact representation of the population. So instead we are supposed to use confidence intervals that describe the upper and lower limit of the range of expected accuracy. Which is where we get the concepts behind such phrases as “significantly greater than chance and less than perfect.” It is better to actually describe the numbers in these situations, along with how the numbers are derived and calculated. 

It is my opinion that Don  may have been trying to be nice when he offered that a kind of over-simplified diplomatic olive-branch about different tests being somewhat similar in aspects that actually make a difference - perhaps in attempt to reduce social/professional friction by re-orienting the discussion around the more open-minded and realistic perspective that proprietors with interests in named polygraph techniques have tended to overlook the fact that they may be more similar than different in terms of the basic procedures and components that have their basis in science and evidence. 

Make no mistake about this: the MQTZCT cannot be realistically expected to produce ~100% accuracy rates and the publications (your's included) appear to offer us absolutely nothing towards the objective of realistically understanding what to expect from the test accuracy. 

Furthermore: the claimed/published foundations of the MQTZCT - the things that authors like yourself and Matte have claimed to make it special and different and better - are inconsistent with the scientific evidence and incompatible with reality. The premises of the MQTZCT is are known to be false hypothesis - that the polygraph machine can discriminate fear and hope - that questions about fear and hope can discriminate the reasons for fear and hope. 

Even more: the analytic model for the MQTZCT is a manual scoring protocol based on visual (i.e.,  subjective) feature extraction with on 23 scoring features. Most people can think in 1 linear dimmension, and many people can think in 2 dimensions ( up/down and right/left). It takes a bit more intelligence and abstracting thinking capacity to work cognitively in 3 dimensions (up/down, right/left, front/back), and smart people can usually handle 4 dimensions when we add a time variable. But what we have with 23 scoring features is a rather high-dimensional feature space that goes well beyond the cognitive capacity of almost everyone. Humans – even smart humans – don’t work well in high-dimensional spaces unless we actually do the math. And the math can get a bit complex – so we very often use computers and algorithms to work with high dimensional data. We do not often try to analyze high dimensional data visually, though we may use visualizations or graphics to illustrate. 

Then couple the high-dimensional (23) MQTZCT feature space with 23 (or is it 25) scoring rules – executed manually of course (by an expert). Many people could have some difficulty remembering all 23 MQTZCT scoring features and 23 MQTZCT scoring rules (which is why we like computers for complex analytic tasks). But here we have the added problem in that the 23 (or is it 25) MQTZCT scoring rules are not well structured to the point where we could construct a logical diagram for flow-control and execution of all those rules. What this means is that it will be impossible to ever use computers to improve the reliability of how those 23 MQTZCT scoring features and 23 (or is it 25) MQTZCT scoring rules get applied. 

In practice this means that these 23 MQTZCT scoring features and 23 (or is it 25) MQTZCT scoring rules will be applied more or less according the subjective whim of the expert. Now this could be fun if we are only in the business of selling expertise and confidence – especially if people are willing to believe claims of ~100% accuracy. But in science we can see problems with inter-scorer reliability. Now we know that Matte reported his conclusions about inter-scorer reliability – with a correlation coefficient about r = .99 (I think he may have wrote that it exceeded). So given the sampling size that Matte reported, and the array of sensors and scores he had something like 2700 numerical scores for which himself and his other expert almost never ever disagreed – while using these 23 visual/subjective scoring features and 23 (or is it 25) scoring rules. This is the kind of delightful result that is simply unheard of in science. It almost seems too good to be true. 

Regardless of whether realistic or not, there may be some experts who enjoy market themselves and their conclusions using techniques such as the MQTZCT for the very reason that the method seems to empower subjective experteeism and the opportunity to pretend ~100% certainty. 
 
The choice to continue to rely on known false hypotheses - coupled with unreal and claims of ~100% decision accuracy -  coupled with an unscientific analytic model - makes the  the MQTZCT vulnerable to accusations that it is pseudo-scientific charlantry  (intended to sell confidence that is disconnected from science while claiming to be scientific). 

The missing piece in most of this is a realistic understanding of what tests are supposed to do (quantify interesting amorphous phenomena that cannot be more easily subject to direct physical measurement or simple/perfect deterministic observation), and a realistic attempt to apply probabilistic thinking and probability models to the polygraph. 

And just in case there is any remaining confusion, Matte's self-publication of a probability model claiming ~100% accuracy is not a realistic application of a probability model and probabilistic thinking. As you have so often emphasized, the meta-message is one of expertise (trust me I'm an expert). It is not that of a scientist. It is the message of a confidence-man not  whose conclusions are inscrutable because they are based on esoterica that is disconnected from science and reality. 

Perhaps readers can decide for themselves if they believe the MQTZCT (with claims of ~100% accuracy based on some unscientific black-magic that polygraph questions can discriminate both fear and hope and the reasons for these emotions) is or is not as good - as a scientific test of credibility - compared to a probabilistic test of credibility developed at the University of Utah (which attempts to apply a simple probability model without making unrealistic claims about accuracy nor unrealistic claims about the capabilities of the recording instrumentation as somehow capable of discriminating between the proximate causes of different emotions). 

The Utah manual scoring protocol is much simpler with simple robust features and simple rules that can be more easily learned by most reasonably intelligent people. The Utah scoring features can also be analyzed using an automated algorithmic probability model. The simple Utah probability model is a linear discriminate analysis that most statisticians and scientists will easily recognize. In fact I was able to replicate it from the published information and achieve results with my sampling data that were similar to that reported by the Utah scientists. In the interest of full disclosure: my work on the OSS-3 and ESS probability models is a derivative of the work at Utah, though I have no proprietary or financial interest in these  models. 

Perhaps well informed readers can decide for themselves if they are willing to believe the MQTZCT to be as good as the methods developed at the University of Utah.


.02

/rn (l500 words)

Dan,

Misleading others is when one says or publishes something with the knowledge or believe to be inconsistent with reality, in attempt to encourage others to accept the information as consistent with reality. 

For example: if one were to publish claims that a scientific test can provide ~100% accuracy with the knowledge or belief that this level of accuracy is not reproducible by others then it would be misleading. 

On the other hand, if one publishes claims of ~100% accuracy because one has enthusiastically accepted an observed sampling result because of of a lack of competency and knowledge about science and testing, then the incorrect information would seem to be about naiveté (perhaps coupled with either impulsiveness or arrogance). 

In the case of the APA statistic, and without information about exactly who wrote it, it would appear that the information is presented by a person who may not adequately understand how to think about scientific test results. 

This is an area of statistics called inference, in which we use our available data to try to understand what we may expect to observe in our future data that is yet unavailable to us. 

Statistical inference is not an attempt to predict the future in a deterministic way, but is similar to attempts to infer the outcome of say a political election where we expect some noisy data/information/vote to go one way and some to go another way. Attempts to predict the exact proportion tend not to work. Instead, we are more successful with attempts to predict the range of proportions (confidence intervals) in which we may expect to observe the eventual data. We can also do this with data-based simulations in which we describe the proportion of simulation outcomes that occurred in a certain range. Anyone who saw the recent move "Sully" may have notices a dramatization of some discussion of repeated simulation outcomes. 

In the case of polygraph accuracy estimates the challenge is complex because test accuracy cannot be not adequately characterized by a single numerical index. 

Notwithstanding the complication of attempting a single numerical index to characterize test accuracy, the statement on the APA website appears to be intended to convey information about point estimates of accuracy under certain circumstances, and is somewhat consistent with some published information on point estimates. 

The real issue in all this is that point estimates are not an adequate way to think about scientific test results. It would be better to use confidence intervals, for which readers can then more easily understand worst-case and best-case outcomes. 

Whereas understanding confidence intervals will require that readers have more basic knowledge about these things, a single proportional index will be simpler for some people to understand but will not adequately convey some important information that can be useful to better informed readers.

.02

/rn

Dan,

In my opinion your "validation study" should be retracted from publication as ~100% accuracy level you reported cannot be realistically accepted as an estimate of what to expect (as has already been described, confession confirmation can systematically exclude error cases for which no confession is likely to be obtained and can lead to the kind of unrealistic accuracy that you have reported) and you have already explained that the study was merely a survey of confession confirmed cases.

/rn

Ray,

Given that you are a past president and former chairman of the quasi-professional 2,800-member American Polygraph Association -- where two out of three members reportedly have only a high school diploma -- please tell us...

How did the data feed that served as the basis of your "meta-analytic survey" go down?

I have a hunch, but it's just a hunch.

A gummint operative, who just happens to be a polygraph advocate, supplied you with a pile of stuff, and said something like, "Trust me, the data is cool. There's no need to investigate any further."

Tell us, Ray... Was there any fact checking? Any oversight? Any third-party QA review?

I think not. 

In other words, I suspect you took the pile of "stuff" purely as a matter of faith.

But as Krapohl said at an APA seminar I attended in 2004, polygraph is BS (Belief System) driven.

If my suspicion is incorrect, please set the record straight.

[cue crickets]

My questions is.

If polygraph enjoys a 90% accuracy rate, why was my idea to let polygraph settle an issue, ongoing in the industry, ridiculed by other examiners, including the high ranking examiners in the industry and within the APA?  

I still have yet to get a straight answer every time I ask this question.  

If 90% is the real number, why did APA members in Texas run from my offer to settle the issue in Texas with polygraph?  

How does one reconcile this?

It seems the only person in Texas that believes in the accuracy in reliability of the product we sell, is me.  

One would think that would be embarrassing for the APA.  That it's members in texas either done believe in the 90% accuracy rate, or the APA has a lot of liars on it's member roles in Texas and in the Texas Association of Polygraph Examiners.

There is a big difference between making claims that the 90% accuracy rates is questionable, as compared to the actions, or lack thereof, of actual examiners who advertises that accuracy rate, but then runs when challenged.  

All Dan can do is make claims.  I can actually point to acts, by examiners, in Texas, who are APA members, have held offices, and sat on committees, who either don't believe those accuracy rates, or are scared of being exposed as liars.

Proof is in action or lack thereof.

I was willing to step forward when I was accused of lying; and people claim I am a discredit to the industry?  I am the only examiner that trusts our test enough to have put my future on the table, based on a test result. 

Joe McCarthy wrote on Sep 29^th, 2016 at 9:16pm:


Probably because the APA's claim of 90+% accuracy is not the "real number."

As the NAS report implies, the so-called real number is only about 65%, roughly speaking -- and that's with CM-ignorant subjects.

In my opinion, the APA's claim of 90+% accuracy is marketing hyperbole.

Oh, and another good argument. If counter measures were that damn good, you would think the Texas examiners would have had no problem in performing them, and thus got rid of me.  If they are so non detectable, you would think that an examiner can do it better then anyone, and not get caught.  

Personally, if I had been them, I would have jumped all over it; and enjoyed being rid of a nemesis, along with having the ability to humiliate said nemesis publicly once and for all.

Yet they avoided the test, probability to engage in countermeasures and all.



hmmmmm, if countermeasures really were that good and undetectable, why wouldn't the use them?  

it is a valid question

Hey Texas examiners.... Maria, Clayton, Stuart, Andy, chime in here any time with the answers.