I have found something neat in the GNOME panel, to move an item from the panel one just have to press middle click and start to DND. I believe the Xfce panel should have such a same feature, but since most plugins would have some actions connected to the middle click I think it should be something more safe like CTRL+Middle click. Or maybe Middle click but detect a start of DND, which is probably more of headache.
I think it would be nice if panel items could be moved simply by drag and drop, like in Windows.
See also bug 8096. I think leaving panel editing to the specialized panel editor is fine, so we don't have to worry about it when the panel editor is not open.
Moving panel plugins is not an action people take all the time, but something you only do when changing the setup of your panel. Also, the "move" action is in the context menu, so it's easily accessible. Closing as wontfix.