My drawing file has grown HUGE!

I am working with a file that has three external references, several Layer States, and 35+ Layout tabs.
After Submittal, some details changed, so I did a "Save As", to revise details and Layouts, but not lose the previous details.
At this point, the file was around 5MB in size.

Then, I started mucking around with the Design Center, to have an old version of a layout in the new file, or some such nonsense.
The file has steadily grown from 5 to 8 to 15 to 27 to 44 and now to 54 MB in size.

I even started a new file, and started pulling in Layouts from ADC. That file quickly grew to 138MB, with only 10 Layouts, and nothing in modelspace at all. Forget that file. Each Layout brought in seemed to be trying to bring in the xref's as well, but there are none in there. (I draw in modelspace, and dimension and annotate in each Layout's paperspace.)

I have purged the ex ref's, deleted unused image files, detached and re-attached the xref's, saved, purged again; everything I can think of, but I cannot get the file back to a manageable size.

Now the program keeps crashing - "Out Of Memory" or nothing at all, just GONE. Saving every two minutes eats up 20-30 seconds just for that!

Before you get too technical, I'm running full ACAD 2004, with 3.5G of RAM, NVIDIA Quatro NVS 285 Display (?).
The file is mostly 2D, with a small bit of 3D on one Layout.

Does anyone have any suggestions for how to get the file back to its original size?