I'm running Xubuntu Vivid Vervet 15.04 with Xfce 4.12 I have purged Firefox and replaced it with Chrome Browser... clicking on magnet links should open Transmission but it opened instead another instance of Chrome. I found a fix, as follows... but this should be fixed out of the box imho....

modify /usr/bin/xdg-open. At the end of the file, you'll find a portion that starts with:

detectDE
if [ x"$DE" = x"" ]; then
   DE=generic
change generic to one of gnome, kde, xfce, or lxde.


as I am running xfce I changed generic to it and now it works...