The 'Logout' menu item seems out of place in the right-click menu of a panel. It has no relation at all with the other items. My suggestion is to remove it completely. Related commit: http://git.xfce.org/xfce/xfce4-panel/commit/?id=c6d5ba57471e9b092111eb1499569b871b92dfaf
I think this was introduced for very minimal sessions where you only want to have a panel without application menu but still be able to log out. This is also the first and only complaint I've ever read about this menu item. Closing as wontfix for now (if you still feel strongly about this, please feel free to reopen).