It should be possible to use the scroll wheel on a background window without losing focus in the window I'm working in. (Or at least be able to raise a window by clicking in it, but not have it rise from scrolling in it)
Xfce Settings > Window Manager Tweaks > Accessibility > uncheck the box "Raise windows when mouse is pressed".