From reading these forums, it seems the consensus is that it's not possible to create customized flyouts on the tool palettes.
I was messing around with opening my .atc files (the ones for each individual palette) in XML Notepad.
I started by dragging a command button from a toolbar into the palette, then turning on the flyouts and narrowing the flyout commands down to the first 4 (line, arc, circle, ellipse). Then I dug into the XML file for the tool palette to see how the tools were defined (see screenshots).
Each one seems to be identified by one of those long (presumably hexadecimal) strings, under:
Palette -> Tools -> Tool -> Flyout -> ActiveShapes -> Shape -> idValue
But I don't know enough about how these idValues work to intelligently change it. (Just pasting in the idValue for another tool will non-fatally break the tool--the flyout doesn't work at all.)
Has anyone made any progress (or seen any tips) on manually customizing these Tool Palette files?