If my network is down when I start Xfce, the weather plugin causes the panel not to load for upwards of a minute just because it cannot contact its information site. I've put this under panel because it seems like the panel should load regardless of the status of the plugins. However, feel free to reassign. ;P
There's no way the panel can know about this. It's the same with the mail checker and POP3. The real solution will probably involve making the plugins separate processes. Because you left me the option, I'm reassigning this ;-)
This actually seems to be fixed in the current release.
*** Bug 1064 has been marked as a duplicate of this bug. ***
Re-assigning to Colin Leroy who is the new maintainer of the plugin.