Bill, last time I was in the keys was 2001. wow, time flies. Anyway Key West is kinda like a tiny little new orleans. It biggest feature is a laid back lifestyle, and if you are so inclined all the pole you could smoke. I would not take my kids out at night. I remember seeing some 6'2" black guy dressed in a white nurses uniform walking down the street in stilettos. But don't let me scare you, there's lots of straight people, and pretty girls (well, they looked like girls to me anyway) to look at too.
Anyway, the big thing is to go and kick back. Watch the sunset. A crowd and carnival atmosphere develops down at the pier. There are many bars to go to, As far as the day Hemingway house, the southernmost point (just to go there) Mel Fisher's treasure museum come to mind. If you like the water, diving and snorkeling would be great, but better in the upper keys.
I stayed at a place with "character" while I was down there- called Courtney's place. It was in old town and was basically a few shotgun houses turned into a B&B. There was a little courtyard with a tiny pool to swim in. Kinda neat, but not for everyone, especially those that like the relative sterility of a Hampton Inn. Be sure and stay in old town to get the feel of the place. Parking is a bitch in old town.
http://www.courtneysplacekeywest.com/
BTW, all the upper keys are the polar opposite of key west, they pretty much roll up the streets at night.