- Welcome to AntiPolygraph.org Message Board.
A forum for free and open discussion of polygraph-related topics
Posted by Drew Richardson
- Aug 24, 2011, 04:37 PMLane99,
You say in part:
Precisely.
Another (of many) issue of concern would be the selection process for the 100 charts presented. Are they randomly selected from a sufficiently large pool? Or are they cherry picked to be "clear," i.e., what one former examiner use to call "Stevie Wonder" charts (even a blind man could read). If the latter were the case, obviously these would not represent "real world" charts and the accuracy involved in scoring same.
Who selects these charts and what professional relationship does that person or persons have to those evaluating the charts. Is there a conflict of interest, e.g., having someone from the polygraph community (that tends to benefit from higher/inflated accuracy rates) select these charts, etc?
Because ASTM (as far as I know) does not conduct polygraph exams, it might be rather easy to have such considerations glossed over.
You say in part:
Quote
"...Perhaps you misunderstand the significance of this. It's not that that information alone would ensure testees of an 86% score.
The point is that that information alone does ensure that the test is not truly blind. But, rather, rigged in favour of producing a higher score than a legitimately blind test would...."
Precisely.
Another (of many) issue of concern would be the selection process for the 100 charts presented. Are they randomly selected from a sufficiently large pool? Or are they cherry picked to be "clear," i.e., what one former examiner use to call "Stevie Wonder" charts (even a blind man could read). If the latter were the case, obviously these would not represent "real world" charts and the accuracy involved in scoring same.
Who selects these charts and what professional relationship does that person or persons have to those evaluating the charts. Is there a conflict of interest, e.g., having someone from the polygraph community (that tends to benefit from higher/inflated accuracy rates) select these charts, etc?
Because ASTM (as far as I know) does not conduct polygraph exams, it might be rather easy to have such considerations glossed over.
Posted by lane99
- Aug 24, 2011, 02:03 PMQuote from: Bill_Brown on Aug 23, 2011, 01:19 AMI did not know how many were deceptive or truthful when I became certified
Fair enough. But it might have been better if you had confined your comments to that in the first place. Rather than making a very misleading sweeping generalization about what other examiners did or didn't know.
Quote from: Bill_Brown on Aug 23, 2011, 01:19 AMEven with...that information available to an examiner how would the examiner be able to obtain a minimum of 86% to qualify.
Perhaps you misunderstand the significance of this. It's not that that information alone would ensure testees of an 86% score.
The point is that that information alone does ensure that the test is not truly blind. But, rather, rigged in favour of producing a higher score than a legitimately blind test would.
Posted by figs
- Aug 23, 2011, 10:29 AMAnyone know if organizations certifying examiners under the Marin Protocol these days are using Evidentiary or Investigative Decision Rules?
ASTM Standard E2324-04 doesn't say, so either the organizations themselves decide this little detail - or the examiners are using whatever decision rules they wish.
ASTM Standard E2324-04 doesn't say, so either the organizations themselves decide this little detail - or the examiners are using whatever decision rules they wish.
Posted by Bill_Brown
- Aug 23, 2011, 01:19 AMlane99
You may assume what you will. I did not know how many were deceptive or truthful when I became certified under the Marin Protocol. I scored in the "blind" totally.
Even with the that information available to an examiner how would the examiner be able to obtain a minimum of 86% to qualify. The "Center" is not a center it is ASTM. It has nothing to do specifically with polygraph other than establishing scientific standards for all industries.
However, you may believe as you desire. I will not argue the point further.
You may assume what you will. I did not know how many were deceptive or truthful when I became certified under the Marin Protocol. I scored in the "blind" totally.
Even with the that information available to an examiner how would the examiner be able to obtain a minimum of 86% to qualify. The "Center" is not a center it is ASTM. It has nothing to do specifically with polygraph other than establishing scientific standards for all industries.
However, you may believe as you desire. I will not argue the point further.
Posted by lane99
- Aug 22, 2011, 01:19 PMQuote from: Bill_Brown on Aug 15, 2011, 11:11 AM...The Center certifies qualified examiners...
Examiners applying for certification must blind-score 100 sets of charts generated in real-world examinations...fifty of which are from examinations where the subject was deceptive and the rest from examinations where the subject was non-deceptive...
As I understand it, there is some "Center" which is granting some "certification" to some "examiners".
And the examiners applying for certification have access to the information that is quoted above.
If so, then surely the examiners know, in advance of the certification test, that, of the 100 charts they're shown, 50 need to be judged deceptive, and 50 non-deceptive.
Posted by Bill_Brown
- Aug 20, 2011, 12:08 PMExaminers were not aware that 50% were deceptive and 50% were truthful. They did score Blind.
Posted by lane99
- Aug 20, 2011, 03:42 AMThanks for the interesting comments, guys.
I might wonder if, in the certification process, the testees know, in advance, that 1/2 the charts are from deceptive subjects, and 1/2 are from truthful subjects? And, further, know that they won't be confronted with any charts that are considered inconclusive.
If so, the certification process doesn't seem legitimately "blind". Rather it's bit rigged, it seems to me.
I might wonder if, in the certification process, the testees know, in advance, that 1/2 the charts are from deceptive subjects, and 1/2 are from truthful subjects? And, further, know that they won't be confronted with any charts that are considered inconclusive.
If so, the certification process doesn't seem legitimately "blind". Rather it's bit rigged, it seems to me.
Posted by Bill_Brown
- Aug 15, 2011, 11:11 AMWe were discussing the topic of inter rater reliability. I used the protocol known as the Marin protocol to demonstrate there is inter rater reliability demonstrated by this protocol.
Certifies:
The Center certifies qualified examiners, in accordance with
ASTM Standard E2324-04 (The Marin Protocol)
Examiners applying for certification must submit an end-to-
end videotape of an examination showing their proficient
adherence to a validated methodology, and the interpretable
chart produced by that examination. They must then blind-
score 100 sets of charts generated in real-world examinations
in cases where the truth is now known, fifty of which are
from examinations where the subject was deceptive and the
rest from examinations where the subject was non-deceptive.
The test sets are selected from a collection of more than 400
sets. The applicant is certified if and only if he declares a
conclusive result for at least 80 per cent of each group of
fifty, and at least 86 of his conclusive results for each
group are correct.
Certifies:
The Center certifies qualified examiners, in accordance with
ASTM Standard E2324-04 (The Marin Protocol)
Examiners applying for certification must submit an end-to-
end videotape of an examination showing their proficient
adherence to a validated methodology, and the interpretable
chart produced by that examination. They must then blind-
score 100 sets of charts generated in real-world examinations
in cases where the truth is now known, fifty of which are
from examinations where the subject was deceptive and the
rest from examinations where the subject was non-deceptive.
The test sets are selected from a collection of more than 400
sets. The applicant is certified if and only if he declares a
conclusive result for at least 80 per cent of each group of
fifty, and at least 86 of his conclusive results for each
group are correct.
Posted by figs
- Aug 15, 2011, 10:10 AMQuote from: Bill_Brown on Aug 14, 2011, 12:14 PMfigs,Quote
The Marin protocol is irelevent to interrater reliability. So are all the studies you cite.
The Marin protocol requires the examiner score 100 known solution cases with a minimum of 86% accuracy. That is inter rater reliability. This is to qualify under that system as an expert witness in court using paired testing. And other studies I have seen and not quoted are in the same area of inter rater reliability.
I will look later and find other studies, sorry I just don't have the time right now.
May be you should tell the polygraphers who practice teh Marin protocol that they dont know what it means.
http://www.gsrsystems.com/examtypes/marin_protocol.html
http://www.veritascenter.org/
That, or it means one thing in practice and some thing els in you're post.
Posted by Bill_Brown
- Aug 14, 2011, 12:42 PMfigs,
Found this on the APA website, another study to look at:
Found this on the APA website, another study to look at:
QuoteA Replication and Validation Study on an Empirically Based Manual Scoring System1
Ben Blalock, Barry Cushman & Raymond Nelson
Abstract
This is a replication of a study validating the hand scoring system for comparison question polygraph examinations proposed by Nelson, Krapohl and Handler (2008). Nine polygraph examiner trainees at an American Polygraph Association accredited polygraph school used an empirically based three-position manual scoring system involving three evaluative criteria and a reduced set of basic rules to evaluate 100 confirmed event-specific single-issue criminal investigation polygraph examinations from the Department of Defense Polygraph Institute confirmed case archive. Average decision accuracy for the inexperienced examiners was 88% with 13.1% inconclusives. Sensitivity and specificity levels achieved by the trainees did not differ significantly, suggesting they achieved balanced accuracy characteristics using the empirically based scoring system. All nine of the inexperienced examiners scored the sample cases with sufficient accuracy to meet the accuracy requirements specified by the Marin protocol (Krapohl, 2005; Marin, 2000). Results from this study parallel the results reported in the previous experiment and support the validity of an empirically based three-position manual scoring method.Quote
Posted by Bill_Brown
- Aug 14, 2011, 12:22 PMStefano,
We are trying to get a handle on those examiners you mention. It is difficult to dictate procedures to individuals that have no respect for proper procedures. There is no legal authority vested in any examiner that allows censorship. New rules and guidelines are being enacted by the APA, but again they are not enforceable.
I am not blind to this problem and do not condone examiners that are not totally professional, I have no power to stop them. I have lobbied for State and Federal regulation of polygraph. Even these State Polygraph Examiner Boards do not dictate or enforce "best practices" because we as examiners have not pushed hard enough against our opposition to have them enacted. Hopefully we will get a handle on this problem in the future, it does not look good right now.
We are trying to get a handle on those examiners you mention. It is difficult to dictate procedures to individuals that have no respect for proper procedures. There is no legal authority vested in any examiner that allows censorship. New rules and guidelines are being enacted by the APA, but again they are not enforceable.
I am not blind to this problem and do not condone examiners that are not totally professional, I have no power to stop them. I have lobbied for State and Federal regulation of polygraph. Even these State Polygraph Examiner Boards do not dictate or enforce "best practices" because we as examiners have not pushed hard enough against our opposition to have them enacted. Hopefully we will get a handle on this problem in the future, it does not look good right now.
Posted by Bill_Brown
- Aug 14, 2011, 12:14 PMfigs,
The Marin protocol requires the examiner score 100 known solution cases with a minimum of 86% accuracy. That is inter rater reliability. This is to qualify under that system as an expert witness in court using paired testing. And other studies I have seen and not quoted are in the same area of inter rater reliability.
I will look later and find other studies, sorry I just don't have the time right now.
Quote
The Marin protocol is irelevent to interrater reliability. So are all the studies you cite.
The Marin protocol requires the examiner score 100 known solution cases with a minimum of 86% accuracy. That is inter rater reliability. This is to qualify under that system as an expert witness in court using paired testing. And other studies I have seen and not quoted are in the same area of inter rater reliability.
I will look later and find other studies, sorry I just don't have the time right now.
Posted by stefano
- Aug 14, 2011, 04:11 AMQuote from: Bill_Brown on Aug 13, 2011, 01:39 PMThat CBS Expose was in 1986, over 20 years ago.It was also a setup.Of course it was a setup; that was the whole idea. Bill, you know that you are one of the few polygraphists that I have come to respect, but you seem to be blind to the plethora of arrogant polygraphists out there who have taken it upon themselves to do whatever they please apparently without peer scrutiny. It seems that simply going through the 320 hour course somehow makes them untouchable and immune to peer criticism--much like the cops that support each other regardless of whatever travesty they choose to precipitate. There is an examiner in my area who takes the $500, accuses the examinee of attempting countermeasures and gives an Inconclusive. There is no oversight, no scrutiny, they consider themselves demigods and will do as they please. You and others in your profession refuse to take the shitbirds to the woodshed.
Posted by figs
- Aug 14, 2011, 01:47 AMQuote from: Bill_Brown on Aug 13, 2011, 05:49 PMQuote from: 6966687C0F0 on Aug 13, 2011, 12:13 AM
Re: Are they even this objective?
Reply #4 - Today at 5:13am Bill_Brown wrote on Yesterday at 5:46pm:
lane99,
I have been involved in studies of examiners scoring charts, and we found examiners do have high agreement rates when scoring charts. Properly trained examiners are in agreement on results of polygraph charts.
Publication citation, please.
Podlesny & Raskin (1978) Rovner et al. (1979) Kircher & Raskin (1988) Honts et al. (1994) Horowitz et al. (1997)
There are newer studies also, check the Marin protocol also. Inter rater reliability is about 86% in that particular study by Krapol.
The Marin protocol is irelevent to interrater reliability. So are all the studies you cite.
Teh Horowitz study you cite even concludes "The R-I test produced an unacceptable rate of false positive decisions."
You dont offer a cit by Donald Krapohl. Maybe you can dig up something responsive? I can't, but your the expert.
Posted by Bill_Brown
- Aug 13, 2011, 05:49 PMQuote from: 6966687C0F0 on Aug 13, 2011, 12:13 AM
Re: Are they even this objective?
Reply #4 - Today at 5:13am Bill_Brown wrote on Yesterday at 5:46pm:
lane99,
I have been involved in studies of examiners scoring charts, and we found examiners do have high agreement rates when scoring charts. Properly trained examiners are in agreement on results of polygraph charts.
Publication citation, please.
Podlesny & Raskin (1978) Rovner et al. (1979) Kircher & Raskin (1988) Honts et al. (1994) Horowitz et al. (1997)
There are newer studies also, check the Marin protocol also. Inter rater reliability is about 86% in that particular study by Krapol.