Welcome to AUGI.
How were the VP's set up? Did you use the Zoom 1/50xp method, or did you simply choose the scale in the status bar? If the former, the VP's Anno Scale doesn't update accordingly.
You mention having several XRefs, are all the scales duplicated between the layout & base drawings? Usually an xref's scale is imported into the layout (if it doesn't already exist), but then the imported scale gets a _XREF suffix. So even if you then later add say the 1:50 scale to the viewport - it's not the scale the original object is applied to (even though the factor is the same), because the scales' names differ. It's stupid I know, but that's what you can expect when programmers never use their own product.
So try to remove the XRef scales from your layout and then copy-n-paste anything (say a line) from the base files into it - that should duplicate the scale names correctly. Set the VP to the correct scale. After which reload the xref and see if it now works.