I'm not sure if it is everytime task grouping is working or just if the group always option is enabled, but it gives a segfault, and crashes the panel.

If you tell me how, I can give you a more detailed report.

However, I solved it changing the configuration with the xfce4-settings-editor.