You should be able to 'link' text to what Object Style or Line Style it is annotating.
e.g. I have a wall that has some leader text pointing to it, I then turn off the wall visibility, the 'associated' text should then turn off with the wall.
I know that there are work arounds, but I think some text intelligence is needed.
Maybe there could be a 'Host' button on the Options Bar. Select the text, then press 'Host', then select your object / line.