Now that xfce4-session supports the SmRestartImmediately hint, the panel should set it when connecting to the SM. I was going to cook up a patch for this, but it seems a bit complicated with the shutdown/restart logic in evaluate_run_state(). If the user requests that the panel quits via 'xfce4-panel -x' (that is, a quit that doesn't involve logging out of the session), the panel should reset its restart style hint to SmRestartIfRunning so the session manager doesn't automatically restart it.
Committed in revision 28386. Hopefully it works. Tried -r and -x and then the panel properly exits, when running 'killall -9 xfce4-panel' it is restarted by the session manager.
Err, well please reopen if you think this implementation is crap ;-).