If you are starting from a server distro (say ubuntu server) and you install the distro xfce4 and gdm, what will happen is that you will get a happily running distro with NO way to stop the xfce desktop and no way to get to a terminal so that you can add more functionality to it (like the terminal which is packaged separately) leaving you with a pretty busted desktop unless you can determine the IP address of the machine. If the machine is part of a server farm (on a virtual server) - good luck with that since you can't get to the terminal to do ifconfig or anything else.

If xterm is available - add THAT to the menu.
If there is a way to stop the window manager and that to the logout screen. There is no way to stop it :(