The World Wide Words has an extended discussion:

"From John Branch in the USA; related questions came from Danny Child and Rodney Breen in the UK, and Midge Peltonen in the USA: "For years I've wondered about the origin of the term eighty-six, which I've heard mainly among restaurant workers. It seems to mean either that the restaurant is out of something ('we're eighty-six on flounder') or, less often, that something should be gotten rid of ('eighty-six that monkey-the health department is outside'). A television program recently reported that it originated at a New York City speakeasy, Chumley's, located at 86 Bedford Street. During Prohibition, when a raid was imminent, a cop on the take would call and warn the proprietor to 'eighty-six it': hide the booze and get the customers out. The story sounds plausible, but I wonder whether you can confirm it."

"One of the standard stories about the origin of this puzzling expression does connect it to Chumley's, though the one I've heard is that when a customer was forcefully ejected from the premises, he would find himself lying woozily on the sidewalk looking up at the number 86 on the door. Neither story, I'm sorry to have to tell you, is likely to be true.

"There are other explanations: that it derives from British merchant shipping, in which the standard crew was 85, so that the 86th man was left behind; that 86 was the number of the American law that forbade bartenders to serve anyone who was drunk (stories disagree about which state it had been enacted in); that a fashionable New York restaurant only had 85 tables, so the eighty-sixth was the one you gave to somebody you didn't want to serve; or that a restaurant (usually said to be in New York) had an especially popular item as number 86 on the menu, so that it frequently ran out. All but the last send my bullshit detector into overload.

"Another explanation frequently given relates the expression to the strengths of spirits served in bars. It is said that these were normally 100 degrees proof but that when a customer was getting over-heated they served instead a weaker brew that was only 86 degrees proof. However, nobody so far as I know has yet produced even halfway convincing evidence that this the origin.

"After this piece first appeared, John Tracy McGrath, a subscriber, wrote with a suggestion that, if not true, certainly deserves to be: "The term was current in the late 1930s when I was a teenager in New York City. It was supposed to have derived from the street-car line that operated on First Avenue on the East Side of Manhattan. The line ran from 14th Street to 86th Street (both major east-west streets). As a north-bound car came to the last stop, the motorman would call out (usually in a rich brogue), "Eighty-six! End of the line! All out!"

"Whatever its origin, it does seem that eighty-six was first used in restaurants and bars, either in the late 1920s or early 1930s; the first firmly attested source is in the journal American Speech for February 1936; another example may be from the mid 1920s-the date is uncertain-which would rule out Chumley's, as it didn't open until 1927. The original sense was that the establishment had run out of some item on the menu.

"The Oxford English Dictionary suggests it may have been rhyming slang for nix, which seems plausible. Although it's often thought of as typically American, nix actually entered the language in the latter part of the eighteenth century in Britain; it was borrowed from a version of the German nichts, nothing. But it seems that eighty-six was created as rhyming slang in the United States.

"The sense that indicated a patron was not to be served because he was drunk or obnoxious appeared later (the first written example is only from 1943); the verb meaning to discard or get rid of something is even more recent, from the 1950s.

"Many people quote other examples of number slang used by hard-pressed servers: 99 meant "the manager is prowling about" and 98 similarly referred to the assistant manager (was 97 a busybody child who wanted to grow up to be a manager?); 19 is a banana split; 55 is root beer, and so on. Presumably some of these related to the numbering on a standard menu somewhere at some time, but the details have been lost."

And, of course, it's the reason Maxwell Smart was named "Agent 86" in the old "Get Smart" series.