How to reproduce: Add a custom shortcut with single quotes in it, like xdotool key --window '$(xdotool getactivewindow)' shift+ctrl+a and put it on some unused F-Key. The settings GUI will exit afterwards. I assume there's some kind of missing single quote escaping. Might be related to bug #12933.
I followed the steps, but the GUI did not exit automatically. Please run "xfce4-keyboard-settings" via terminal window instead and see if any log output is generated when the bug occurs.
For whatever reason I'm no longer able to replicate the crashes, neither when started via the menu, nor via the terminal. The only unusual output I was able to see was this... (xfce4-keyboard-settings:2518): Gtk-CRITICAL **: 08:42:55.898: gtk_list_store_remove: assertion 'iter_is_valid (iter, list_store)' failed ...when I edited an existing entry and replaced the single quotes with double quotes.
That one is a known bug. See: Bug 15958