Or, if you want to keep it open, you can toggle between open windows by using CTRL+ Tab. Usually, even if you have several windows open, using CTRL + Tab from an open Xref will put you back to the drawing you were in when you opened it. Somehow CAD knows to put the open Xref just after the current drawing.
Also, if you use CTRL + Shift + Tab, then the drawings cycle in the reverse direction.
Also Also
, if you find you're opening up Xref's a lot, like I do, I'd suggest setting a double click action so when you double click the Xref, it opens. No need to right click and select open or go to the Xref manager. It doesn't seem like it would, but it saves a lot of time.