In case pulseaudio crashes or is restarted for whatever reason, the panel plugin becomes useless. It's necessary to restart the entire panel (xfce4-panel -r) or delete and re-add the plugin for it to become functional again. It would be great if the plugin could detect that pulseaudio has restarted and reconnect itself to the daemon automatically.
This should be fixed in git. Can you check if that works for you?
It appears to be working. Thanks very much!