First of all, I'm not too keen on the default setting for the fonts in xffm. They are HUGE on my system (Debian Sarge (testing)). The only way for me to turn them down is to change the view to "Detailed view". I don't necessarily want the detailed view, but I would like smaller fonts, so I think a smaller default font (the font specified in the User Interface configuration thing) would be a Good Thing (tm). If you for some reason can't do that very easily, at least an option to change the font without changing the level of detail would be very nice.
this bug has been noted for some time. Unfortunately the fix applied to CVS was to add a configuration option "resize fonts to match icons" and this involved a translation that could not be accepted into the 4.0.x branch. So you could either 1- update to the 4.1.1 tarball at http://www.xfce.org/~edscott/tarballs, or 2- just edit the file misc.c, look for the function "set_font" or "setfont" and put in a "return" for the first line and recompile.