When you select a new file, if you hold down your mouse and move it up / down it will select the file you had selected previously too. Just deleted accidentally an important file because of this (used APPLY TO ALL), fix it asap.
The reselection occurs because of the multiple files selection method used for drag and drop. If you do not release the mouse button after selecting a file, the previous selection will be reselected, allowing you to select multiple files easily for drag an drop or another operation (like delete). This might be a problem for some, so the multiple file selection mode can be toggled back to the normal gtk way of multiple selecting. This option is not in the 4.0 branch because it requires a translation, but you can find it in 4.1 at CVS HEAD.
The selection method works this way. In xffm 4.1 you can choose to use normal GTK selection method. This option was added but removed from 4.0.x because it involved a new translation in the options menu.