A Debian user reported: xfce4-mixer allows you to switch between two sound devices using File->Option->Device, but to actually modify the sliders for the newly selected device, it must be restarted. You can see the full bug at: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=382441
Yeah, I'm sorry, that's by design (... omission). What I mean is that if I changed that in the stable branch now, something else would break (it just never was meant to be changed that often). You can always ask Jannis to add it to the new GStreamer-based Xfce Mixer, that's still in development.