Attached is a patch to add a "remove-plugin" signal to xfce-panel-plugin-iface so that an instance of a plugin will be notified when it is removed from the panel.
Created attachment 2516 Add remove-plugin signal to xfce-panel-plugin-iface.c
Tested against 4.6.0 (Xubuntu 9.04) and works. Patch applies cleanly to SVN, so should work fine with SVN.
Devel branch has been merged in master. A 4.7.0 release will follow soon. If you think this bug is not fixed? Feel free to reopen the bug.