Normal Topic Numbers game (Read 6339 times)
Paste Member Name in Quick Reply Box Onesimus
Very Senior User
****
Offline



Posts: 110
Joined: May 10th, 2003
Numbers game
Aug 5th, 2003 at 5:37am
Mark & QuoteQuote Print Post  
Just some simple number crunching assuming a 15 percent false positive and false negative rate.



Percentage of ApplicantsTelling Truth on Relevant Questions Percentage of  Applicants Accused of Guilt that are Guilty Percentage of Applicants Found Truthful that are Truthful
98 10.36585 99.64115
95 22.97297 99.07975
90 38.63636 98.07692
80 58.62069 95.77465
70 70.83333 92.96875
60 79.06977 89.47368
50 85 85
40 89.47368 79.06977
30 92.96875 70.83333
20 95.77465 58.62069
10 98.07692 38.63636
5 99.07975 22.97297
2 99.64115 10.36585

  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Human Subject
Senior User
***
Offline



Posts: 73
Joined: May 31st, 2003
Re: Numbers game
Reply #1 - Aug 5th, 2003 at 10:10pm
Mark & QuoteQuote Print Post  
Just looking at this, I don't get it.  You want to walk us through the interpretation of these results?
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Onesimus
Very Senior User
****
Offline



Posts: 110
Joined: May 10th, 2003
Re: Numbers game
Reply #2 - Aug 6th, 2003 at 6:14am
Mark & QuoteQuote Print Post  
Well I was hoping for some comments on how to refine the numbers, but hereís how Iíd start to interpret it.  I think a lot of people on this board are more against the use of the  polygraph for pre-employment purposes than they are for its use in criminal investigations.  I think the chart I posted supports this notion since a much higher percentage of people polygraphed for pre-employment purposes are going to be truthful on the relevant questions that those for criminal investigations.

Basically, column 2 shows us that a deception indicated score becomes less reliable as the percentage of applicants who are telling the truth on relevant questions increases.  For example, if 60%of your applicants are truthful, then 79% of the ones that are accused of lying are lying.  However, if 95% of your applicants are truthful, then only 23% of those accused of lying are actually lying!

Now suppose youíre a federal agency that is polygraphing your contractorís technical guys (software engineers, mechanical engineers, mathematicians, etc.).  A pretty high percentage of these guys are going to be telling the truth, so youíre going to get a lousy accuracy rate with your accusations of lying.

I have more comments on this, but donít have time to elaborate.  To summarize, I would suggest that the methods taken to alleviate this problem would allow all spies to pass through, only weeding out some of  the drug abusers, child molesters, killers, etc.

But a much smaller percentage of this technical crowd is going to be of the sex offender/violent crime variety than your general population, so there is still less of a reason to poly the technical crowd then the general public.  And Iím honestly not that concerned if the guy sitting next to me has used drugs too many times if he can do his job well.  I think its more of a risk to our national security to take a less qualified candidate than one who is qualified but used drugs to much ("Well, he screwed up the project, but at least he doesnít do drugs")

So basically youíre left with something that costs money, canít catch spies, tortures innocent people, and catches some bad guys at a smaller rate than you could catch them from the general public.

Do you understand how the numbers were calculated?

Looking back at TLBTLD, I see that something similar to this is mentioned on pages 20 and 21under "False Positives and the Base Rate Problem." 
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Onesimus
Very Senior User
****
Offline



Posts: 110
Joined: May 10th, 2003
Re: Numbers game
Reply #3 - Aug 6th, 2003 at 6:20am
Mark & QuoteQuote Print Post  
From my own personal experiences, I should add to the list of pre-screening polygraph accomplishments:

1)Discourages many qualified persons from applying to some jobs in the first place
2)Increases the likelihood that someone will get frustrated and leave.
3)Wastes time at work as people complain about their polygraph experience.
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box suethem
Very Senior User
****
Offline



Posts: 196
Joined: Apr 29th, 2003
Re: Numbers game
Reply #4 - Aug 8th, 2003 at 7:40pm
Mark & QuoteQuote Print Post  
Onesimus,

Enjoyed looking at the stats. †

My problem with it is that it assumses the machine can distiguish between truth and lies, which it can't!

If an experiment is flawed from the start, then results can have no merit.

I have brought up AG Ashcroft's statement of up to 15% false positive rate hoping that the pro-side would address it.

I personally believe that the admission of the 15% †false positive rate is simply the gov allowing itself some wiggle room, when its own results (negative or positive) are disproved.

I wonder what the official stance on false negatives is? †15% †Then were already down to 70% percent accuracy.

Regardless- since its not scientific , its not accurate!!

Probable lies-- are not lies for everyone- (Probable?)

Control Questions- are not scientific controls
† † † † † † † † † † † † † † † are based on probable lies which
† † † † † † † † † † † † † † † †are not givens.

Relevant questions- †are often provocative and could
† † † † † † † † † † † † † † † † †get a spiked response from a
† † † † † † † † † † † † † † † † †person who finds the question
† † † † † † † † † † † † † † † † † exciting, scary, funny, stupid...

Polygraphy is an interesting idea. †But when you put the theory to the test it fails.

Why agenices use it is an ethical question.
« Last Edit: Aug 8th, 2003 at 10:18pm by suethem »  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Human Subject
Senior User
***
Offline



Posts: 73
Joined: May 31st, 2003
Re: Numbers game
Reply #5 - Aug 9th, 2003 at 12:25am
Mark & QuoteQuote Print Post  
suethem wrote on Aug 8th, 2003 at 7:40pm:
Probable lies-- are not lies for everyone- (Probable?)


Exactly... if your probable lie question is "have you ever stolen from an employer", well, this might be hard to believe, but some people haven't.  If such a person answers this question and the relevant questions truthfully, he or she is screwed since there is no contrast between responses.
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Mr. Truth
Very Senior User
****
Offline



Posts: 194
Joined: Aug 4th, 2003
Re: Numbers game
Reply #6 - Aug 9th, 2003 at 1:07am
Mark & QuoteQuote Print Post  
A quick little primer on Bayesian statistics, even includes references to the junk science called polygraphy:
http://www.abelard.org/briefings/bayes.htm
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box suethem
Very Senior User
****
Offline



Posts: 196
Joined: Apr 29th, 2003
Re: Numbers game
Reply #7 - Aug 9th, 2003 at 3:08am
Mark & QuoteQuote Print Post  
Mr. Truth,

interesting link.

I think that one of the NAS scientists was a statistician.   The link you cited brought up the same bad choice between catching a few spies at the cost of thousands of loyal employees that NAS study mentions.

If Americas best scientists say that polygraphy is a joke, it makesyou wonder about the quality of  the "professional polygrapher."

" I don't care what science says.." is never a very good opening arguement.
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Onesimus
Very Senior User
****
Offline



Posts: 110
Joined: May 10th, 2003
Re: Numbers game
Reply #8 - Aug 9th, 2003 at 5:40am
Mark & QuoteQuote Print Post  
<<My problem with it is that it assumes the machine can distiguish between truth and lies, which it can't!>>

No, Iím assuming that 85% of the time when someone is lying, their polygrapher will accuse them of lying, and 85% percent of the time when a person is truthful, their polygrapher will find them truthful.  (regardless of the machineís ability to determine this)   I know the numbers arenít correct, but they are reasonable estimates.  Even if the numbers were different, we would still see the same trend Ė that as the truthfulness of your applicants increases, the percentage of those accused of lying who are actually lying will decrease.  (Thus supporting the notion that polygraph is most flawed when applied to pre-employment screening).

<<I wonder what the official stance on false negatives is? 15% Then were already down to 70% percent accuracy.>>

If we assume
X = Fraction who are telling the truth
Y = Fraction False Positive  (.15 in our case)
Z = Fraction  False Negative (.15 in our case)

Then,

1-X = Fraction who are lying
XY + (1-X)Z = Fraction False Accusation
XY + Z -XZ = Fraction False Accusation

So when Y=Z,
Fraction False Accusation = Fraction False Positive = Fraction False Negative
If Y is not Z, then the Fraction False Accusation is dependent on the fraction of people telling the truth

So in our case the accuracy is 85%  if false positive and false negative rates are both 15%.  But I think that the more important percentages are the ones I showed in the chart.

In fact, the situation could be worse than the numbers shown in the chart.  Lets suppose that 75% of the guilty ones are persuaded by their polygrapher to fess-up.   Thatís going to drop all the percentages in the second column if we make it "Percentage of Applicants that donít confess to anything and are accused of guilt that are guilty".  So an even higher percentage who are rejected solely on the basis of polygraph results (bad poly but no confession) will have been unfairly terminated.  Perhaps Iíll make a chart for this too.

<<if your probable lie question is "have you ever stolen from an employer", well, this might be hard to believe, but some people haven't.>>

Gee, I havenít and I assume most of my coworkers havenít!!
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Onesimus
Very Senior User
****
Offline



Posts: 110
Joined: May 10th, 2003
Re: Numbers game
Reply #9 - Aug 9th, 2003 at 6:03am
Mark & QuoteQuote Print Post  
Ok, hereís the new numbers, still assuming 15 percent false positive, 15 percent false negative.  Now Iím also assuming that 75% of the guilty are persuaded by their polygrapher to confess, and the hiring organization is willing to deny employment solely on the basis of polygraph results (i.e. no hire if fail polygraph even if no confession)
Percentage of Applicants Telling Truth on Relevant Questions Percentage Rightfully Denied Employment Based Soley On Polygraph Results
98 2.809917
95 6.938776
90 13.6
80 26.15385
70 37.77778
60 48.57143
50 58.62069
40 68
30 76.77419
20 85
10 92.72727
5 96.41791
2 98.57988
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box suethem
Very Senior User
****
Offline



Posts: 196
Joined: Apr 29th, 2003
Re: Numbers game
Reply #10 - Aug 9th, 2003 at 6:18am
Mark & QuoteQuote Print Post  
Onesimus,

Even with my shoes off and my toes a wigglin' I couldn't  understand all that  math.

Do you think you could do my taxes?

Interesting stuff.  I wish I had paid more attention in school.
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box digithead
Very Senior User
****
Offline



Posts: 150
Joined: Apr 11th, 2006
Re: Numbers game
Reply #11 - Apr 11th, 2006 at 10:22am
Mark & QuoteQuote Print Post  
Oneismus,

Your numbers and formulas are wrong...

And the polygraph is detecting deception/no deception. So your base rate is set by the percentage that are deceptive, not the percentage that are truthful...

First a couple of definitions from medical test screening using polygraph terms:

Sensitivity=P(Deceptive|Deception Indicated)
False Positive Rate=1-Sensitivity
Specificity=P(Not Deceptive|Test is Negative)
False Negative Rate=1-Specificity
Base Rate= P(Deceptive)

Then through Bayes Theorem, we can calculate
Positive Predictive Value, which tells you how often the test is correct given that you have the condition:

PPV=P(Deception Indicated|Deceptive)=(Base Rate*Sensitivity)/(Base Rate*Sensitivity+(1-Base Rate)*Specificity)

Similarly, we can get the same for Negative Predicative Value...

So using Oneismus' first numbers with 2% Deceptive, 85% Sensitivity, 85% Specificity we get:

Positive Predictive Value=(.02*.85)/(.02*.85+.98*.85)
=2%

Which means that a Polygraph with 85% accuracy and 2% deceptive population tested will have only 2% classified correctly, meaning that 98% of the positives will have been falsely accused of being deceptive and will be dismissed under your employment scenario...

And no matter how you spin it, PPV/NPV are dependent on the base rate in addition to test accuracy, the only way you can get past the base rate problem is to increase the accuracy which is impossible for the polygraph because it's not accurate to begin with...

This was the point that was hammered home in the NAS report but the polygraph community ignores it...

And if you don't believe me, check out any epidemiology or biostatistics textbook. All of them use PPV to demonstrate Bayes Theorem and that PPV/NPV is the true indicator of the usefulness of a screening test...

And all of this presupposes that the polygraph is actually dectecting deception which this site, the NAS report, and countless articles have shown, has no scientific basis to support it...
  
Back to top
 
IP Logged
 
Paste Member Name in Quick Reply Box Drew Richardson
Especially Senior User
*****
Offline



Posts: 427
Joined: Sep 7th, 2001
Re: Numbers game
Reply #12 - Apr 11th, 2006 at 5:00pm
Mark & QuoteQuote Print Post  
Digithead,

You write:
Quote:
Oneismus,

Your numbers and formulas are wrong...

And the polygraph is detecting deception/no deception. So your base rate is set by the percentage that are deceptive, not the percentage that are truthful...

First a couple of definitions from medical test screening using polygraph terms:

Sensitivity=P(Deceptive|Deception Indicated)
False Positive Rate=1-Sensitivity
Specificity=P(Not Deceptive|Test is Negative)
False Negative Rate=1-Specificity
Base Rate= P(Deceptive)

Then through Bayes Theorem, we can calculate
Positive Predictive Value, which tells you how often the test is correct given that you have the condition:

PPV=P(Deception Indicated|Deceptive)=(Base Rate*Sensitivity)/(Base Rate*Sensitivity+(1-Base Rate)*Specificity)

Similarly, we can get the same for Negative Predicative Value...

So using Oneismus' first numbers with 2% Deceptive, 85% Sensitivity, 85% Specificity we get:

Positive Predictive Value=(.02*.85)/(.02*.85+.98*.85)
=2%

Which means that a Polygraph with 85% accuracy and 2% deceptive population tested will have only 2% classified correctly, meaning that 98% of the positives will have been falsely accused of being deceptive and will be dismissed under your employment scenario...

And no matter how you spin it, PPV/NPV are dependent on the base rate in addition to test accuracy, the only way you can get past the base rate problem is to increase the accuracy which is impossible for the polygraph because it's not accurate to begin with...

This was the point that was hammered home in the NAS report but the polygraph community ignores it...

And if you don't believe me, check out any epidemiology or biostatistics textbook. All of them use PPV to demonstrate Bayes Theorem and that PPV/NPV is the true indicator of the usefulness of a screening test... †

And all of this presupposes that the polygraph is actually dectecting deception which this site, the NAS report, and countless articles have shown, has no scientific basis to support it...


Absolutely correct and right on! †The only questionable assumption (I realize it was not yours) in the aforementioned example would be the percent of individuals deceptive in our population (base rate) of examinees (i.e. 2 per cent assuming we are still talking about national security issues). †In a population of 10,000 FBI agents that would calculate to 200 spies--I would hope not! †Using a more realistic base rate and the Bayes Theorem calculation would (as you obviously know) lead to an even higher percent rate of false positives. †Best Regards...
« Last Edit: Apr 11th, 2006 at 6:25pm by Drew Richardson »  
Back to top
 
IP Logged
 
Numbers game

Please type the characters that appear in the image. The characters must be typed in the same order, and they are case-sensitive.
Open Preview Preview

You can resize the textbox by dragging the right or bottom border.
Insert Hyperlink Insert FTP Link Insert Image Insert E-mail Insert Media Insert Table Insert Table Row Insert Table Column Insert Horizontal Rule Insert Teletype Insert Code Insert Quote Edited Superscript Subscript Insert List /me - my name Insert Marquee Insert Timestamp No Parse
Bold Italicized Underline Insert Strikethrough Highlight
                       
Insert Preformatted Text Left Align Centered Right Align
resize_wb
resize_hb







Max 200000 characters. Remaining characters:
Text size: pt
More Smilies
View All Smilies
Collapse additional features Collapse/Expand additional features Smiley Wink Cheesy Grin Angry Sad Shocked Cool Huh Roll Eyes Tongue Embarrassed Lips Sealed Undecided Kiss Cry
Attachments More Attachments Allowed file types: txt doc docx ics psd pdf bmp jpe jpg jpeg gif png swf zip rar tar gz 7z odt ods mp3 mp4 wav avi mov 3gp html maff pgp gpg
Maximum Attachment size: 500000 KB
Attachment 1:
X