I haven't been able to get an useful trace yet. 4.13.0, bug still present.

Comment 2 Eric Koegel 2017-06-11 12:25:31 CEST

Best bet for memory leaks is to run it in valgrind. Something like:
xfdesktop --quit;
G_DEBUG=gc-friendly G_SLICE=always-malloc,debug-blocks valgrind -v --leak-check=full \
    --leak-resolution=high --num-callers=500 --track-origins=yes  \
    --read-var-info=yes --show-leak-kinds=all \
    -log-file=~/Desktop/xfdesktop-valgrind.log xfdesktop &

Then trigger the leak and gracefully kill xfdesktop with xfdesktop --quit. Note: it will be super slow running in valgrind but once you have xfdesktop-valgrind.log file feel free to attach it and I'll take a look. I also fixed a memory leak a couple days ago if you're running the 4.13/git master stuff.

No luck with valgrind either, but I've managed to attach a debugger and interrupt the process as it was looping today. See the backtrace below. I wonder if old_size = 34359737792 is the culprit. Possible under/overflow?

(gdb) bt full
#0  0x00007ffff4792a80 in ?? () from /lib64/libc.so.6
No symbol table info available.
#1  0x000000000043a894 in xfdesktop_move_all_icons_to_pending_icons_list (icon_view=0x974310)
    at xfdesktop-icon-view.c:3079
        l = 0x0
#2  0x000000000043b0f4 in xfdesktop_grid_do_resize (icon_view=0x974310) at xfdesktop-icon-view.c:3250
        xorigin = 8
        yorigin = 8
        width = 1904
        height = 1037
        new_rows = 9
        new_cols = 18
        old_size = 34359737792
        new_size = 1296
        gscreen = 0x7fffffff52e0
#3  0x0000000000438f20 in xfdesktop_rootwin_watch_workarea (gxevent=0x7fffffff5350, event=0xb5a3d0, 
    user_data=0x974310) at xfdesktop-icon-view.c:2644
        icon_view = 0x974310
        xevt = 0x7fffffff5350
        __func__ = "xfdesktop_rootwin_watch_workarea"
#4  0x00007ffff5fed1d1 in ?? () from /usr/lib64/libgdk-3.so.0
No symbol table info available.
#5  0x00007ffff5fed48d in ?? () from /usr/lib64/libgdk-3.so.0
No symbol table info available.
#6  0x00007ffff5fb8ad9 in gdk_display_get_event () from /usr/lib64/libgdk-3.so.0
No symbol table info available.
#7  0x00007ffff5fed242 in ?? () from /usr/lib64/libgdk-3.so.0
No symbol table info available.
#8  0x00007ffff4d0e36d in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#9  0x00007ffff4d0e618 in ?? () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#10 0x00007ffff4d0e932 in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#11 0x00007ffff6468ec5 in gtk_main () from /usr/lib64/libgtk-3.so.0
No symbol table info available.
#12 0x00000000004307e1 in xfdesktop_application_start (app=0x68e150) at xfdesktop-application.c:702
        gdpy = 0x6a9010
        error = 0x0
        buf = "/backdrop/screen0/\000\000\000\000\000\000 \000\000\000\000\000\000\000\000\354\225\000\000\000\000\000\204\224x\364\377\177\000\000\300\251\205\367\377\177\000\000 ", '\000' <repeats 15 times>, "\365SL\363\377\177\000\000 \000\000\000\000\000\000\000 \000\000\000\377\177\000\000\264V\377\377\377\177\000\000ۆ\325\355\377\177\000\000\337\000\000\000\000\000\000\000\222\212@\000\000\000\000\000\310[@\000\000\000\000\000\270V\377\377\377\177\000\000n\345\000/\000\000\000\000P\223k\000\000\000\000\000\001\000\b", '\000' <repeats 13 times>, "t\354\225\000\000\000\000\000 \000\000\000\000\000\000\000\000\354\225\000\000\000\000\000\001"...
        __func__ = "xfdesktop_application_start"
#13 0x000000000043020e in cb_wait_for_window_manager_destroyed (data=0x7b5830) at xfdesktop-application.c:570
        wfwm = 0x7b5830
        __func__ = "cb_wait_for_window_manager_destroyed"
#14 0x00007ffff4d0ae78 in ?? () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#15 0x00007ffff4d0b8a7 in ?? () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#16 0x00007ffff4d0e2b8 in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#17 0x00007ffff4d0e618 in ?? () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#18 0x00007ffff4d0e6bc in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
No symbol table info available.
#19 0x00007ffff54dec4d in g_application_run () from /usr/lib64/libgio-2.0.so.0
No symbol table info available.
#20 0x000000000043085f in xfdesktop_application_run (app=0x68e150, argc=2, argv=0x7fffffff5cf8)
    at xfdesktop-application.c:711
No locals.
#21 0x0000000000421f2d in main (argc=2, argv=0x7fffffff5cf8) at main.c:58
        app = 0x68e150
        ret = 0

I'm debugging the (currently still running) xfdesktop instance with my limited knowledge.

It's stuck on memset inside xfdesktop_move_all_icons_to_pending_icons_list(). The last parameter to memset is

(guint)icon_view->priv->nrows * icon_view->priv->ncols * sizeof(XfdesktopIcon *))

The problem appears to be that nrows is negative (-4)! Casting that into an uint is bound to be the problem. Now the question is how it ends up being negative in the first place...

icon_view->priv->height = -443
icon_view->priv->ymargin = -12

These are also negative.

Some further info:

Sometimes when I suspend, and intel-virtual-output gets killed, I end up with these monitors:

LVDS1 connected 1920x1080+0+0 (normal left inverted right x axis y axis) 340mm x 190mm
VIRTUAL7 connected 320x240+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
VIRTUAL8 connected 1440x2560+4480+0 right (normal left inverted right x axis y axis) 0mm x 0mm

Now when intel-virtual-output is restarted at a different location and updates the monitor list, This changes to:

LVDS1 connected 1920x1080+0+0 (normal left inverted right x axis y axis) 340mm x 190mm
VIRTUAL6 connected 2048x1536+0+0 (normal left inverted right x axis y axis) 0mm x 0mm

which triggers this bug:

DBG[xfdesktop-icon-view.c:2609] xfdesktop_setup_grids(): grid size is -4x18

which then tries to allocate infinite memory.

Comment 7 Eric Koegel 2017-07-18 08:03:12 CEST

Created attachment 7230 
Don't try to allocate all the memory

Can you try out this patch? Thanks for all the troubleshooting info!

Looks like it's working:

DBG[xfdesktop-icon-view.c:2609] xfdesktop_setup_grids(): grid size is 0x18

no memory problems and xfdesktop recovers with the new screen layout that follows.

Thanks a lot!

Comment 9 Git Bot 2017-07-18 17:18:10 CEST

Eric Koegel referenced this bugreport in commit 424208b3e150c48797faeeac7693c5ad6594b3e8

Don't try to allocate all the memory (Bug #12805)

https://git.xfce.org/xfce/xfdesktop/commit?id=424208b3e150c48797faeeac7693c5ad6594b3e8

Comment 10 Eric Koegel 2017-07-19 05:00:40 CEST

Thanks again for all the debugging and testing on this!

Comment 11 Skunnyk 2018-06-09 10:43:00 CEST

Does this fix need to be backported to 4.12 branch ?