Deleting large file fails and crashes NAS

[Zippyduda]

Hi all.

I have a 300GB file that I want to delete however when I try to do it via Windows or SSH it fails and crashes the NAS, meaning I have to remove the power to reboot it (it doesn't respond to software or physical reboot). When it comes back on the file is still there.

Is there a way I can remove this file?

Or do I just need to delete the share it resides on?

I'm on the latest stable firmware version for my NETGEAR RN10400-100EUS: 6.9.3

Thanks and kind regards.

[StephenB · Accepted solution]

You could try a single truncation via ssh

truncate -s 0 myfile.ext

Progressive truncation just uses different parameters for -s (size).  For instance, -s 100G or -s 100M.

Echo is another tool that can truncate a file.

echo > myfile.ext

Either way, turn snapshots off before you do this.  Perhaps also disable checksums and quota if they are enabled (look on the volume settings wheel).

If it works, I suggest running a balance afterwards.  If that also locks up, then post back.

[StephenB]

I suggest running the disk test next - it's one of the options on the volume settings wheel.

Also, download the log zip, and see if there are any disk or BTRFS errors in system.log and kernel.log.

[Zippyduda]

Running a disk text now. For a 4TB setup in RAID1 (4TBx2) how long should I expect this to take?

[StephenB]

The last one I ran took 13 hours with a 4x6TB array with healthy disks.  I'm not sure if it tests the drives sequentially or in parallel.  Maybe assume about 8 hours.

If you have spindown enabled, maybe temporarily turn that off.

[Zippyduda]

Thanks Stephen, will leave it going then! 🙂 Have turned off disk spin down, had it set to 240 minutes inactivity.

No disk errors in kernel.log or system.log which is reassuring.

Only errors I see are these:

Line 4502: May 02 12:36:50 wsdd2[2091]: error: wsdd-mcast-v4: wsd_send_soap_msg: send
Line 4503: May 02 12:36:50 wsdd2[2091]: error: wsdd-mcast-v6: wsd_send_soap_msg: send
Line 4504: May 02 12:36:50 wsdd2[2091]: error: llmnr-mcast-v4: open_ep: IP_ADD_MEMBERSHIP
Line 4521: May 02 12:36:52 wsdd2[2091]: error: wsdd-mcast-v4: wsd_send_soap_msg: send
Line 4522: May 02 12:36:52 wsdd2[2091]: error: wsdd-mcast-v6: wsd_send_soap_msg: send
Line 4597: May 02 12:47:22 smbd[4481]: list_snaps error response: error.no_permissions
Line 4597: May 02 12:47:22 smbd[4481]: list_snaps error response: error.no_permissions
Line 4853: May 02 14:38:16 wsdd2[2073]: error: wsdd-mcast-v4: wsd_send_soap_msg: send
Line 4854: May 02 14:38:16 wsdd2[2073]: error: wsdd-mcast-v6: wsd_send_soap_msg: send
Line 4855: May 02 14:38:16 wsdd2[2073]: error: llmnr-mcast-v4: open_ep: IP_ADD_MEMBERSHIP
Line 4868: May 02 14:38:18 wsdd2[2073]: error: wsdd-mcast-v4: wsd_send_soap_msg: send
Line 4869: May 02 14:38:18 wsdd2[2073]: error: wsdd-mcast-v6: wsd_send_soap_msg: send

[mdgm-ntgr]

There is a method to progressively delete huge files via SSH by repeatedly truncating it. Have you tried that?

[Zippyduda]

Looks like the disk test went through fine from the Logs section:

Thu May 3 2018 0:25:10 Volume: Disk test completed for volume data.
Wed May 2 2018 15:22:17 Volume: Disk test started for volume data.

Overnight the admin panel closed so it closed I can't see the "Info, Warning, Errors" popup any more.

Unsure how I can get to this again or if I need to as the disk test completed (would it show any warnings or errors in Logs?)

I have not tried the a method to progressively delete huge files via SSH by repeatedly truncating it. 
How would I go about doing this?

[StephenB]

---

@Zippyduda wrote:

Overnight the admin panel closed so it closed I can't see the "Info, Warning, Errors" popup any more.

Unsure how I can get to this again or if I need to as the disk test completed (would it show any warnings or errors in Logs?)

---

I suggest downloading the log zip file, and looking in volume.log.  That will give you the smart stats, and the results of the last disk self-test.

[Zippyduda]

All results showing as "passed" 🙂

[StephenB]

---

@Zippyduda wrote:

All results showing as "passed" 🙂

---

Good.  Are you seeing reallocated or pending sectors, command timeouts, or ATA errors in disk_info.log?

Also any btrfs errors in system.log or kernel.log?

[Zippyduda]

I've replied twice now and my reply has been deleted! Don't know what's going on.

[Zippyduda]

No issues from disk test.

No reallocated or pending sectors, command timeouts, or ATA errors in disk_info.log

 No btrfs errors in system.log or kernel.log

[StephenB]

---

@Zippyduda wrote:

I've replied twice now and my reply has been deleted! Don't know what's going on.

---

There is an automated spam filter that sometimes misbehaves.  Netgear is supposed to monitor it, and release legit posts - but sometimes they fall behind or miss them.  Often there is a lot of spam.

It's fine to PM me on this (esp. if I am in the conversation); I can also release them.

[StephenB · Accepted solution]

You could try a single truncation via ssh

truncate -s 0 myfile.ext

Progressive truncation just uses different parameters for -s (size).  For instance, -s 100G or -s 100M.

Echo is another tool that can truncate a file.

echo > myfile.ext

Either way, turn snapshots off before you do this.  Perhaps also disable checksums and quota if they are enabled (look on the volume settings wheel).

If it works, I suggest running a balance afterwards.  If that also locks up, then post back.

[Zippyduda]

I've disabled quotas, however I cannot disable checksum as it is stating I have bit rot protection enabled on one or more shares. How do I go about turning this off?

[Zippyduda]

Nevermind, I've found out how to disable it on each share 🙂

[Zippyduda]

Currently running the truncate command. Got the file I want to remove from 297GB to 200GB successfully. Now going to 100GB. Then will go to 0.

[Zippyduda]

That's worked, thank you so much 🙂 One last question, will it cause any issues now I've turned checksum, quotas and Bit Rot Protection off when I go to Balance the disks? Should I turn these all back on before doing a Balance.

[mdgm-ntgr]

Whatever you prefer.

Though when the file was deleted any chunks that were emptied completely would have automatically been returned to unallocated space. If you look in btrfs.log you may well find that the space allocated is already close to what's consumed.

A balance will move data around so that chunks are emptied and those emptied chunks are returned to unallocated space.

[StephenB]

---

@mdgm-ntgr wrote:

Whatever you prefer.

---

My impression is that you haven't been running balances regularly - if that's the case, I'd leave them off for now.  Quotas and checksums both add some additional load to the NAS, so the balance might proceed a bit more quickly w/o them. 

Then you can turn them back on if you wish, and also set up a maintenance schedule for defrag, scrub, balance, and perhaps disk tests via the volume settings.  I run them each once a quarter myself.

---

@mdgm-ntgr wrote:

If you look in btrfs.log you may well find that the space allocated is already close to what's consumed.

A balance will move data around so that chunks are emptied and those emptied chunks are returned to unallocated space.

To add to this - BTRFS free space can either be allocated or unallocated.  The system works best when there is a healthy amount of unallocated free space.  As you delete files, you normally end up with more and more allocated free space.   The balance takes care of that. 

If you run them regularly, then they generally don't take a long time.  But if you wait until all the space is allocated, it can take quite a while (and the NAS won't be very responsive while it is running). 

[Zippyduda]

Having trouble copying files onto my NAS as it thinks it's lacking space, when I know it is not.

Windows comes back with a message saying:

""There is not enough space on <Volume> (NAS name). You need an additional X.XX GB to copy these files"

I've attached screenshots to show what I mean.

Would it be worth trying to FTP/SFTP the files on as a workaround?

[Zippyduda]

Attached multiple images showing the free space, relevant settings are off and the Windows message.

[StephenB]

It looks like it shouldn't be doing this.

Perhaps try rebooting the NAS (mostly under the theory that it is simple to do).

How old is your data volume?  If it is very old, there might be some hidden snapshots.

[Zippyduda]

Already tried rebooting it sadly 😞

Is it possibly because of all these snapshots?

I swear I used to be able to see these as yellow on the key/legend on the left.

[StephenB]

Normally they are shown in yellow.  

One option is to back up the data, reset the NAS, and restore the data from the backup.  Not elegant, and if it helps you'll never know the root cause.  But it might be the fastest way to solve it.