- Welcome to AntiPolygraph.org Message Board.
A forum for free and open discussion of polygraph-related topics
Posted by Drew Richardson
- Apr 11, 2006, 01:00 PMDigithead,
You write:
Absolutely correct and right on! The only questionable assumption (I realize it was not yours) in the aforementioned example would be the percent of individuals deceptive in our population (base rate) of examinees (i.e. 2 per cent assuming we are still talking about national security issues). In a population of 10,000 FBI agents that would calculate to 200 spies--I would hope not! Using a more realistic base rate and the Bayes Theorem calculation would (as you obviously know) lead to an even higher percent rate of false positives. Best Regards...
You write:
Quote
Oneismus,
Your numbers and formulas are wrong...
And the polygraph is detecting deception/no deception. So your base rate is set by the percentage that are deceptive, not the percentage that are truthful...
First a couple of definitions from medical test screening using polygraph terms:
Sensitivity=P(Deceptive|Deception Indicated)
False Positive Rate=1-Sensitivity
Specificity=P(Not Deceptive|Test is Negative)
False Negative Rate=1-Specificity
Base Rate= P(Deceptive)
Then through Bayes Theorem, we can calculate
Positive Predictive Value, which tells you how often the test is correct given that you have the condition:
PPV=P(Deception Indicated|Deceptive)=(Base Rate*Sensitivity)/(Base Rate*Sensitivity+(1-Base Rate)*Specificity)
Similarly, we can get the same for Negative Predicative Value...
So using Oneismus' first numbers with 2% Deceptive, 85% Sensitivity, 85% Specificity we get:
Positive Predictive Value=(.02*.85)/(.02*.85+.98*.85)
=2%
Which means that a Polygraph with 85% accuracy and 2% deceptive population tested will have only 2% classified correctly, meaning that 98% of the positives will have been falsely accused of being deceptive and will be dismissed under your employment scenario...
And no matter how you spin it, PPV/NPV are dependent on the base rate in addition to test accuracy, the only way you can get past the base rate problem is to increase the accuracy which is impossible for the polygraph because it's not accurate to begin with...
This was the point that was hammered home in the NAS report but the polygraph community ignores it...
And if you don't believe me, check out any epidemiology or biostatistics textbook. All of them use PPV to demonstrate Bayes Theorem and that PPV/NPV is the true indicator of the usefulness of a screening test...
And all of this presupposes that the polygraph is actually dectecting deception which this site, the NAS report, and countless articles have shown, has no scientific basis to support it...
Absolutely correct and right on! The only questionable assumption (I realize it was not yours) in the aforementioned example would be the percent of individuals deceptive in our population (base rate) of examinees (i.e. 2 per cent assuming we are still talking about national security issues). In a population of 10,000 FBI agents that would calculate to 200 spies--I would hope not! Using a more realistic base rate and the Bayes Theorem calculation would (as you obviously know) lead to an even higher percent rate of false positives. Best Regards...
Posted by digithead
- Apr 11, 2006, 06:22 AMOneismus,
Your numbers and formulas are wrong...
And the polygraph is detecting deception/no deception. So your base rate is set by the percentage that are deceptive, not the percentage that are truthful...
First a couple of definitions from medical test screening using polygraph terms:
Sensitivity=P(Deceptive|Deception Indicated)
False Positive Rate=1-Sensitivity
Specificity=P(Not Deceptive|Test is Negative)
False Negative Rate=1-Specificity
Base Rate= P(Deceptive)
Then through Bayes Theorem, we can calculate
Positive Predictive Value, which tells you how often the test is correct given that you have the condition:
PPV=P(Deception Indicated|Deceptive)=(Base Rate*Sensitivity)/(Base Rate*Sensitivity+(1-Base Rate)*Specificity)
Similarly, we can get the same for Negative Predicative Value...
So using Oneismus' first numbers with 2% Deceptive, 85% Sensitivity, 85% Specificity we get:
Positive Predictive Value=(.02*.85)/(.02*.85+.98*.85)
=2%
Which means that a Polygraph with 85% accuracy and 2% deceptive population tested will have only 2% classified correctly, meaning that 98% of the positives will have been falsely accused of being deceptive and will be dismissed under your employment scenario...
And no matter how you spin it, PPV/NPV are dependent on the base rate in addition to test accuracy, the only way you can get past the base rate problem is to increase the accuracy which is impossible for the polygraph because it's not accurate to begin with...
This was the point that was hammered home in the NAS report but the polygraph community ignores it...
And if you don't believe me, check out any epidemiology or biostatistics textbook. All of them use PPV to demonstrate Bayes Theorem and that PPV/NPV is the true indicator of the usefulness of a screening test...
And all of this presupposes that the polygraph is actually dectecting deception which this site, the NAS report, and countless articles have shown, has no scientific basis to support it...
Your numbers and formulas are wrong...
And the polygraph is detecting deception/no deception. So your base rate is set by the percentage that are deceptive, not the percentage that are truthful...
First a couple of definitions from medical test screening using polygraph terms:
Sensitivity=P(Deceptive|Deception Indicated)
False Positive Rate=1-Sensitivity
Specificity=P(Not Deceptive|Test is Negative)
False Negative Rate=1-Specificity
Base Rate= P(Deceptive)
Then through Bayes Theorem, we can calculate
Positive Predictive Value, which tells you how often the test is correct given that you have the condition:
PPV=P(Deception Indicated|Deceptive)=(Base Rate*Sensitivity)/(Base Rate*Sensitivity+(1-Base Rate)*Specificity)
Similarly, we can get the same for Negative Predicative Value...
So using Oneismus' first numbers with 2% Deceptive, 85% Sensitivity, 85% Specificity we get:
Positive Predictive Value=(.02*.85)/(.02*.85+.98*.85)
=2%
Which means that a Polygraph with 85% accuracy and 2% deceptive population tested will have only 2% classified correctly, meaning that 98% of the positives will have been falsely accused of being deceptive and will be dismissed under your employment scenario...
And no matter how you spin it, PPV/NPV are dependent on the base rate in addition to test accuracy, the only way you can get past the base rate problem is to increase the accuracy which is impossible for the polygraph because it's not accurate to begin with...
This was the point that was hammered home in the NAS report but the polygraph community ignores it...
And if you don't believe me, check out any epidemiology or biostatistics textbook. All of them use PPV to demonstrate Bayes Theorem and that PPV/NPV is the true indicator of the usefulness of a screening test...
And all of this presupposes that the polygraph is actually dectecting deception which this site, the NAS report, and countless articles have shown, has no scientific basis to support it...
Posted by suethem
- Aug 09, 2003, 02:18 AMOnesimus,
Even with my shoes off and my toes a wigglin' I couldn't understand all that math.
Do you think you could do my taxes?
Interesting stuff. I wish I had paid more attention in school.
Even with my shoes off and my toes a wigglin' I couldn't understand all that math.
Do you think you could do my taxes?
Interesting stuff. I wish I had paid more attention in school.
Posted by Onesimus
- Aug 09, 2003, 02:03 AMOk, here's the new numbers, still assuming 15 percent false positive, 15 percent false negative. Now I'm also assuming that 75% of the guilty are persuaded by their polygrapher to confess, and the hiring organization is willing to deny employment solely on the basis of polygraph results (i.e. no hire if fail polygraph even if no confession)
| Percentage of Applicants Telling Truth on Relevant Questions | Percentage Rightfully Denied Employment Based Soley On Polygraph Results |
| 98 | 2.809917 |
| 95 | 6.938776 |
| 90 | 13.6 |
| 80 | 26.15385 |
| 70 | 37.77778 |
| 60 | 48.57143 |
| 50 | 58.62069 |
| 40 | 68 |
| 30 | 76.77419 |
| 20 | 85 |
| 10 | 92.72727 |
| 5 | 96.41791 |
| 2 | 98.57988 |
Posted by Onesimus
- Aug 09, 2003, 01:40 AM<<My problem with it is that it assumes the machine can distiguish between truth and lies, which it can't!>>
No, I'm assuming that 85% of the time when someone is lying, their polygrapher will accuse them of lying, and 85% percent of the time when a person is truthful, their polygrapher will find them truthful. (regardless of the machine's ability to determine this) I know the numbers aren't correct, but they are reasonable estimates. Even if the numbers were different, we would still see the same trend – that as the truthfulness of your applicants increases, the percentage of those accused of lying who are actually lying will decrease. (Thus supporting the notion that polygraph is most flawed when applied to pre-employment screening).
<<I wonder what the official stance on false negatives is? 15% Then were already down to 70% percent accuracy.>>
If we assume
X = Fraction who are telling the truth
Y = Fraction False Positive (.15 in our case)
Z = Fraction False Negative (.15 in our case)
Then,
1-X = Fraction who are lying
XY + (1-X)Z = Fraction False Accusation
XY + Z -XZ = Fraction False Accusation
So when Y=Z,
Fraction False Accusation = Fraction False Positive = Fraction False Negative
If Y is not Z, then the Fraction False Accusation is dependent on the fraction of people telling the truth
So in our case the accuracy is 85% if false positive and false negative rates are both 15%. But I think that the more important percentages are the ones I showed in the chart.
In fact, the situation could be worse than the numbers shown in the chart. Lets suppose that 75% of the guilty ones are persuaded by their polygrapher to fess-up. That's going to drop all the percentages in the second column if we make it "Percentage of Applicants that don't confess to anything and are accused of guilt that are guilty". So an even higher percentage who are rejected solely on the basis of polygraph results (bad poly but no confession) will have been unfairly terminated. Perhaps I'll make a chart for this too.
<<if your probable lie question is "have you ever stolen from an employer", well, this might be hard to believe, but some people haven't.>>
Gee, I haven't and I assume most of my coworkers haven't!!
No, I'm assuming that 85% of the time when someone is lying, their polygrapher will accuse them of lying, and 85% percent of the time when a person is truthful, their polygrapher will find them truthful. (regardless of the machine's ability to determine this) I know the numbers aren't correct, but they are reasonable estimates. Even if the numbers were different, we would still see the same trend – that as the truthfulness of your applicants increases, the percentage of those accused of lying who are actually lying will decrease. (Thus supporting the notion that polygraph is most flawed when applied to pre-employment screening).
<<I wonder what the official stance on false negatives is? 15% Then were already down to 70% percent accuracy.>>
If we assume
X = Fraction who are telling the truth
Y = Fraction False Positive (.15 in our case)
Z = Fraction False Negative (.15 in our case)
Then,
1-X = Fraction who are lying
XY + (1-X)Z = Fraction False Accusation
XY + Z -XZ = Fraction False Accusation
So when Y=Z,
Fraction False Accusation = Fraction False Positive = Fraction False Negative
If Y is not Z, then the Fraction False Accusation is dependent on the fraction of people telling the truth
So in our case the accuracy is 85% if false positive and false negative rates are both 15%. But I think that the more important percentages are the ones I showed in the chart.
In fact, the situation could be worse than the numbers shown in the chart. Lets suppose that 75% of the guilty ones are persuaded by their polygrapher to fess-up. That's going to drop all the percentages in the second column if we make it "Percentage of Applicants that don't confess to anything and are accused of guilt that are guilty". So an even higher percentage who are rejected solely on the basis of polygraph results (bad poly but no confession) will have been unfairly terminated. Perhaps I'll make a chart for this too.
<<if your probable lie question is "have you ever stolen from an employer", well, this might be hard to believe, but some people haven't.>>
Gee, I haven't and I assume most of my coworkers haven't!!
Posted by suethem
- Aug 08, 2003, 11:08 PMMr. Truth,
interesting link.
I think that one of the NAS scientists was a statistician. The link you cited brought up the same bad choice between catching a few spies at the cost of thousands of loyal employees that NAS study mentions.
If Americas best scientists say that polygraphy is a joke, it makesyou wonder about the quality of the "professional polygrapher."
" I don't care what science says.." is never a very good opening arguement.
interesting link.
I think that one of the NAS scientists was a statistician. The link you cited brought up the same bad choice between catching a few spies at the cost of thousands of loyal employees that NAS study mentions.
If Americas best scientists say that polygraphy is a joke, it makesyou wonder about the quality of the "professional polygrapher."
" I don't care what science says.." is never a very good opening arguement.
Posted by Mr. Truth
- Aug 08, 2003, 09:07 PMA quick little primer on Bayesian statistics, even includes references to the junk science called polygraphy:
http://www.abelard.org/briefings/bayes.htm
http://www.abelard.org/briefings/bayes.htm
Posted by Human Subject
- Aug 08, 2003, 08:25 PMQuote from: suethem on Aug 08, 2003, 03:40 PM
Probable lies-- are not lies for everyone- (Probable?)
Exactly... if your probable lie question is "have you ever stolen from an employer", well, this might be hard to believe, but some people haven't. If such a person answers this question and the relevant questions truthfully, he or she is screwed since there is no contrast between responses.
Posted by suethem
- Aug 08, 2003, 03:40 PMOnesimus,
Enjoyed looking at the stats.
My problem with it is that it assumses the machine can distiguish between truth and lies, which it can't!
If an experiment is flawed from the start, then results can have no merit.
I have brought up AG Ashcroft's statement of up to 15% false positive rate hoping that the pro-side would address it.
I personally believe that the admission of the 15% false positive rate is simply the gov allowing itself some wiggle room, when its own results (negative or positive) are disproved.
I wonder what the official stance on false negatives is? 15% Then were already down to 70% percent accuracy.
Regardless- since its not scientific , its not accurate!!
Probable lies-- are not lies for everyone- (Probable?)
Control Questions- are not scientific controls
are based on probable lies which
are not givens.
Relevant questions- are often provocative and could
get a spiked response from a
person who finds the question
exciting, scary, funny, stupid...
Polygraphy is an interesting idea. But when you put the theory to the test it fails.
Why agenices use it is an ethical question.
Enjoyed looking at the stats.
My problem with it is that it assumses the machine can distiguish between truth and lies, which it can't!
If an experiment is flawed from the start, then results can have no merit.
I have brought up AG Ashcroft's statement of up to 15% false positive rate hoping that the pro-side would address it.
I personally believe that the admission of the 15% false positive rate is simply the gov allowing itself some wiggle room, when its own results (negative or positive) are disproved.
I wonder what the official stance on false negatives is? 15% Then were already down to 70% percent accuracy.
Regardless- since its not scientific , its not accurate!!
Probable lies-- are not lies for everyone- (Probable?)
Control Questions- are not scientific controls
are based on probable lies which
are not givens.
Relevant questions- are often provocative and could
get a spiked response from a
person who finds the question
exciting, scary, funny, stupid...
Polygraphy is an interesting idea. But when you put the theory to the test it fails.
Why agenices use it is an ethical question.
Posted by Onesimus
- Aug 06, 2003, 02:20 AMFrom my own personal experiences, I should add to the list of pre-screening polygraph accomplishments:
1)Discourages many qualified persons from applying to some jobs in the first place
2)Increases the likelihood that someone will get frustrated and leave.
3)Wastes time at work as people complain about their polygraph experience.
1)Discourages many qualified persons from applying to some jobs in the first place
2)Increases the likelihood that someone will get frustrated and leave.
3)Wastes time at work as people complain about their polygraph experience.
Posted by Onesimus
- Aug 06, 2003, 02:14 AMWell I was hoping for some comments on how to refine the numbers, but here's how I'd start to interpret it. I think a lot of people on this board are more against the use of the polygraph for pre-employment purposes than they are for its use in criminal investigations. I think the chart I posted supports this notion since a much higher percentage of people polygraphed for pre-employment purposes are going to be truthful on the relevant questions that those for criminal investigations.
Basically, column 2 shows us that a deception indicated score becomes less reliable as the percentage of applicants who are telling the truth on relevant questions increases. For example, if 60%of your applicants are truthful, then 79% of the ones that are accused of lying are lying. However, if 95% of your applicants are truthful, then only 23% of those accused of lying are actually lying!
Now suppose you're a federal agency that is polygraphing your contractor's technical guys (software engineers, mechanical engineers, mathematicians, etc.). A pretty high percentage of these guys are going to be telling the truth, so you're going to get a lousy accuracy rate with your accusations of lying.
I have more comments on this, but don't have time to elaborate. To summarize, I would suggest that the methods taken to alleviate this problem would allow all spies to pass through, only weeding out some of the drug abusers, child molesters, killers, etc.
But a much smaller percentage of this technical crowd is going to be of the sex offender/violent crime variety than your general population, so there is still less of a reason to poly the technical crowd then the general public. And I'm honestly not that concerned if the guy sitting next to me has used drugs too many times if he can do his job well. I think its more of a risk to our national security to take a less qualified candidate than one who is qualified but used drugs to much ("Well, he screwed up the project, but at least he doesn't do drugs")
So basically you're left with something that costs money, can't catch spies, tortures innocent people, and catches some bad guys at a smaller rate than you could catch them from the general public.
Do you understand how the numbers were calculated?
Looking back at TLBTLD, I see that something similar to this is mentioned on pages 20 and 21under "False Positives and the Base Rate Problem."
Basically, column 2 shows us that a deception indicated score becomes less reliable as the percentage of applicants who are telling the truth on relevant questions increases. For example, if 60%of your applicants are truthful, then 79% of the ones that are accused of lying are lying. However, if 95% of your applicants are truthful, then only 23% of those accused of lying are actually lying!
Now suppose you're a federal agency that is polygraphing your contractor's technical guys (software engineers, mechanical engineers, mathematicians, etc.). A pretty high percentage of these guys are going to be telling the truth, so you're going to get a lousy accuracy rate with your accusations of lying.
I have more comments on this, but don't have time to elaborate. To summarize, I would suggest that the methods taken to alleviate this problem would allow all spies to pass through, only weeding out some of the drug abusers, child molesters, killers, etc.
But a much smaller percentage of this technical crowd is going to be of the sex offender/violent crime variety than your general population, so there is still less of a reason to poly the technical crowd then the general public. And I'm honestly not that concerned if the guy sitting next to me has used drugs too many times if he can do his job well. I think its more of a risk to our national security to take a less qualified candidate than one who is qualified but used drugs to much ("Well, he screwed up the project, but at least he doesn't do drugs")
So basically you're left with something that costs money, can't catch spies, tortures innocent people, and catches some bad guys at a smaller rate than you could catch them from the general public.
Do you understand how the numbers were calculated?
Looking back at TLBTLD, I see that something similar to this is mentioned on pages 20 and 21under "False Positives and the Base Rate Problem."
Posted by Human Subject
- Aug 05, 2003, 06:10 PMJust looking at this, I don't get it. You want to walk us through the interpretation of these results?
Posted by Onesimus
- Aug 05, 2003, 01:37 AMJust some simple number crunching assuming a 15 percent false positive and false negative rate.
| Percentage of ApplicantsTelling Truth on Relevant Questions | Percentage of Applicants Accused of Guilt that are Guilty | Percentage of Applicants Found Truthful that are Truthful |
| 98 | 10.36585 | 99.64115 |
| 95 | 22.97297 | 99.07975 |
| 90 | 38.63636 | 98.0769298.07692 |
| 80 | 58.62069 | 95.77465 |
| 70 | 70.83333 | 92.96875 |
| 60 | 79.06977 | 89.47368 |
| 50 | 85 | 85 |
| 40 | 89.47368 | 79.06977 |
| 30 | 92.96875 | 70.83333 |
| 20 | 95.77465 | 58.62069 |
| 10 | 98.07692 | 38.63636 |
| 5 | 99.07975 | 22.97297 |
| 2 | 99.64115 | 10.36585 |