When I have this plugin in my panel, scrolling the mouse wheel on top of it does not work, until I left-click the icon and close it again. After doing that, the mouse wheel will work (for a while)
The brightness plugin has been dropped and the remaining panel plugin handles this the way you expected, scrolling on top of it changes the brightness.