This is just a guess.. but I would think that the most people would be weeded out in the initial interview.  The one where they ask you about criminal history and drug usage...  At a career fair last year everyone that wanted to apply had a face-to-face interview with a special agent then and there.  The people that got past that first interview came out one door, and those that did not came out another.  I'd say the vast majority of people came out the door that meant "no go"....

You sure are right the polygraph is the most toughest part of the process. I bet you that at of all those that take the polygraph at least 50% of them fail even though they are telling the truth. I would also say that An agency like the Florida Highway Patrol tends to disqualify many people with their polygraph. I personally believe they disqualify good applicants, At least 50 % percent of those who take their polygraph. This tells me that polygraphs are not accurate machines