I had a very similar experience a year ago with a USSS poly.  I was late to the interview because I was having difficulty finding the place; despite calling on my cell phone and asking for directions and letting them know I would be late, the examiner clearly had a bad attitude for the rest of the day.  He kept saying throughout the interview that he was having a hard time understanding why I would be late to such an important interview if I really cared about getting the job.  

The actual poly itself was just as you described.  Of course, after the poly was finished he left the room to "read the charts", which is a customary tactic.  He came back in, and just as in your case, told me I failed the drug question.  At the time I had never used an illegal drug in my life.

He kept saying over and over that he could only help me if I would "come clean", and I kept telling him that I had answered his questions fully and had nothing to add.  He even told me that my answers seemed "Clintonian", i.e. like the dodgy answers Bill Clinton used to give!  It seemed ironic to have a Secret Service agent speak so mockingly about a former President.

I openly told the examiner that I didn't believe in polygraph and that my prior experiences with it, combined with the reading I had done about it, had convinced me that there was no validity to the science of it.  This really seemed to bother him.  

I received my rejection letter not long after.

I have had other experiences applying for law enforcement positions, and most have been negative.  Despite being an excellent candidate for law enforcement employment, and despite having been offered one police job (which I turned down), the bulk of my application/interview experiences have been negative.  My experiences with the US Secret Service and the US Capitol Police rank pretty high on my list of lousy experiences in dealing with law enforcement agencies.

I have now abandoned my law enforcement ambitions in favor of other pursuits that pay more and don't employ such unsophisticated people and processes.

Best luck to you.