Scrolling while hovering adjusts volume perfectly but as soon as the dropdown is opened via left click it fails to do so. After closing the menu it works. The same is for having the contect menu open, but that's inteded I guess - or at least no issue for me. Distribution is Debian Buster x86_64 with xfce4-pulseaudio-plugin 0.3.1-1 from Debian Experimental.
*** Bug 13899 has been marked as a duplicate of this bug. ***
This is actually a limitation imposed by GTK+. When any menu is open it takes all focus, scrolling, and other events. This can be observed by opening the menu in any other application and attempting to adjust the volume with the mousewheel while it is open.