When restarting the datetime plugin, there is a GLib assertion failure.
Reproduce as follows:
Add one instance of the plugin to the panel with the default configuration.
From a terminal window exit, start, then restart the panel.
$ xfce4-panel -r
(xfce4-panel:1774): GLib-CRITICAL **: g_source_remove: assertion `tag > 0' failed
** Message: Restarting xfce4-panel...
DBG[datetime.c:670] datetime_new(): Starting datetime panel plugin
On subsequent trials, "xfce4-panel -x" produces this message too.
Revision is 4921.
I was easily able to reproduce the symptoms, thanks.
Fixed in svn r4994, using part of your patch from Bug #4145. Thanks again :)