I definitely wouldn't do impact work barefoot.
And if you pronate or suppinate at all, barefoot isn't a good idea.
I would do yoga, some no-impact martial arts, barre work and stretching barefoot, but nothing else.
But I'm someone who jams my toes againts dumbbells, or steps on used (for testing baked goods) toothpicks, or slams the top of my foot against my lap top when I'm trying to walk over it and don't have my glasses on....guess you call that 'klutzy'!