I have my mouse set for left-handed mode with the mouse buttons reversed (i.e. so that my mouse button 1 is on the right side of the mouse), however when starting up the session selector dialog is still using a right-handed mouse configuration. It would appear that the mouse settings are not being set before that dialog is popped up.
If by "session selector" you are referring to your DM (display manager, like GDM, LightDM, XDM, Slim etc), then yes, they are controlled by X11's systemwide settings which Xfce doesn't touch. More generally speaking, the DM is not part of Xfce. So yeah, the settings will only be applied within Xfce, that is, as soon as your session has started.