When one clicks an entry from xfce4-settings-manager startup-notification should not show up in the panel because the settings dialog is embedded into the settings-manager. Of course one can disable the notification in the desktop files, but this also removes it when run in standalone mode. So IMHO xfce4-settings should skip the notification somehow. If there still should be kind of startup feedback it should be inside of the settings-manager but not in the panel.
Fixed in git master.