I installed a new system recently and noticed the X server was hogging the CPU.  Strace showed a select() loop, and the client was Terminal.

To reproduce: close all existing Terminals.  Move ~/.config/Terminal temporarily to simulate a fresh install.  Start a Terminal, set background to "image" but leave File: as (none).  On my dual core 2.2Ghz box, this will consume 30-40% of one processor per Terminal. Ie, it easily causes a very noticeable slow down, but just using "top" will point to the X server.