ReadyNAS Wouldn't Load Large Folder - and how I diagnosed this and solved it :)

Had an issue where explorer on Windows 10 would not complete loading the folder, it would give up midway and the mapped network drives would sometimes disappear.. downloading the log ("Download logs" on log page and opening up the ZIP in 7-zip) showed the device saying it had run out of memory:

 

Repeatedly featuring lines in kernel.log like:

 

Dec 17 11:52:07 NAS kernel: Out of memory: Kill process 21301 (tracker-miner-f) score 26 or sacrifice child
Dec 17 11:52:07 NAS kernel: Killed process 21301 (tracker-miner-f) total-vm:707032kB, anon-rss:113488kB, file-rss:0kB

 and

 

Dec 17 16:43:32 NAS kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x2008000)
Dec 17 16:43:32 NAS kernel:   cache: cfq_queue, object size: 232, buffer size: 232, default order: 0, min order: 0
Dec 17 16:43:32 NAS kernel:   node 0: slabs: 285, objs: 4845, free: 0

This was on a folder of around 11,118 subfolders and files.

 

Solution was to create folders such as "A", "B", "C", "D" and move the files and folders into them using the ReadyNAS web interface.. and being really patient and ignoring the browser warnings about it saying a script is taking too long.

 

And also turned off "File Search" as that seemed to be consuming a lot of memory (https://packages.debian.org/sid/utils/tracker-miner-fs). There were also a lot of errors in tracker-miner-fs.log of:

 

Dec 17 02:13:01 NAS tracker-miner-fs[13596]: (tracker-miner-fs:13596): Tracker-CRITICAL **:   (Sparql buffer) Error in task 0 (file:///data/Music/<<PATH>>) of the array-update: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not necessarily related): Resource temporarily unavailable)
Dec 17 02:13:01 NAS tracker-miner-fs[13596]: (tracker-miner-fs:13596): Tracker-CRITICAL **: Could not execute sparql: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not necessarily related): Resource temporarily unavailable)

and it looks like it tried to start indexing the folder.. ran out of memory.. was killed then tried re-indexing again and re-inserting the same records.. might be related to https://bugs.launchpad.net/ubuntu/+source/tracker-miners/+bug/1723677