While this feature may have been meaningful in Gtk+2 (i.e. setting custom icon sizes via gtkrc), we will either have to re-implement it for Gtk+3 in a different or live without it. First step is to move to fixed standard icon sizes for the respective contexts.
This bug was referenced in this commit: https://git.xfce.org/xfce/xfce4-panel/commit?id=ee7381e4feb1c009c23e8936a351bf07d303d6e3
This bug was referenced in this commit: https://git.xfce.org/xfce/xfce4-panel/commit?id=d6744ac6480f0a540b217b4dd72e502dae0b244c