I have compositing enabled in the Window Manager Tweaks dialog, though I don't have anything else checked there and everything is set to opaque (so I hope that if some program wants to have itself fancy, then it can, but I don't have any translucency by default). And Alt-Tab is painfully slow: switching to the next window in the list (after the switcher is already shown) takes a noticeable amount of time, tenths of a second. If I turn off the compositing, it becomes instant.
Could be related to the wire frame drawing which uses XShape and thus causing additional redraws. I don't see that on any of the systems I use, but I guess it may happen with some older/slower hardware or drivers. Please check against current trunk (xfce 4.6) to see if disabling the wire frame does fix the issue.