Created attachment 7961 patch I could swear this was already addressed by now, seems I was mistaken.
Created attachment 7973 opposite patch This patch does the opposite, it adds the missing icons.
Sorry, didn't notice your patch and fixed it myself: https://git.xfce.org/xfce/xfce4-panel/commit/?id=c4e5767f54ab4984aa05a965d6ebee5d0141a91a