Some panel plugins use too much CPU:

On a very fast laptop, after 4 hours of uptime I have
00:01:46 xfce4-systemload-plugin
00:00:44 xfce4-battery-plugin
00:00:33 xfce4-netload-plugin

Compared to only
00:00:07 xfce4-diskperf-plugin 
00:00:06 xfce4-cpugraph-plugin

One would expect systemload, netload and diskperf to consume approximately the same amount of CPU (as they roughly seem to do the same amount of work), and all of them should burn less CPU than cpugraph (because cpugraph obviously does more complex work than the simple bar plugins).

In relation to for example
00:00:42 Terminal
for a terminal which was used heavily most of the time,
the plugins clearly consume too much CPU.

I don't know the refresh interval of the systemload plugin,
because it is not shown in the properties dialog.
Netload, diskperf and cpugraph are running at 1 sec refresh,
perhaps systemload is refreshing too often?
(but even at half the refresh rate, it would still consume too much CPU
compared to diskperf and cpugraph)

The battery plugin should consume even less CPU and could run at even longer refresh intervals (battery status won't change dramatically within 1 second...).