terminal-screen.c sets COLORTERM to "xfce4-terminal". I just checked, and the default ncurses database has no entry for "xfce4-terminal". ncurses only knows about "xfce", and does not know anything about "xfce4-terminal". Additionally, COLORTERM appears to be an obsolete environment variable. It is not on the default list of environment variables forwarded across ssh sessions, and according to https://bugzilla.redhat.com/show_bug.cgi?id=1173688 gnome-terminal does not want to have anything to do with it, either. I'll also copy this to the ncurses mailing list. xfce4-terminal and ncurses need to be on the same page. It doesn't look to me like ncurses even looks at COLORTERM anymore.
For vte-based terminals TERM should be set to "xterm-256color". Once xfce-terminal migrates to newest gtk3-based vte, it'll no longer be necessary to set this since vte sets this value on its own. COLORTERM's old value is indeed obsolete. If it's set at all, it should be set to "truecolor", see https://bugzilla.gnome.org/show_bug.cgi?id=754521.
Resolved by migrating to vte3.