Yes, you should worry about the FBI polygraph. About half of special agent applicants who make it as far along in the hiring process as the polygraph end up being branded as liars and disqualified for life from FBI employment. Given polygraphy's complete lack of validity, it's clear that many of those who fail are being falsely accused of deception. As I've mentioned before, I don't think the risks are worth the potential rewards, and my advice to anyone considering FBI employment would be to look elsewhere until such time as the Bureau terminates its cynical (and ultimately counterproductive) reliance on the pseudoscience of polygraphy.