× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

ReadyNAS 316 not responding after disk balance task started

jaffacake
Tutor

ReadyNAS 316 not responding after disk balance task started

I scheduled the ReadyNAS 316 disk balance task to run this morning.

The task started as expected, i got an email confirmation and could see the notice on the top of the Web UI. I could also see an entry in the log that confirmed the task started.

I looked at the disk monitors for a bit to see how much the drives ( 5 x 3TB WD RED) were being hammered. Quite a lot. I then left it to run.

After lunch, I noticed all my LUNs and network drives have gone off-line. We admin doesn't respond and I can't even ping the NAS any more.

On the unit itself, the Netgear logo is lit (as it always has been since upgrading to 6.2.0) and the LED display shows the hostname and expected IP address. The touch buttons on the unit do not respond, they do not even light up.

Inside, two of the drives are flashing with activity, other drives seem inactive.

I'm assuming this isn't normal? I didn't expect the unit to go off-line during maintenance? How should I resolve? I don't want to pull the power if it is busy but it does appear to be completely hung.

Any help appreciated, I've got total loss of service right now.
Message 1 of 27
vandermerwe
Master

Re: ReadyNAS 316 not responding after disk balance task star

Do you have a backup?
Can you get it to respond to power down by using the front power button, should be 2 or 3 presses?


Btw the Netgear logo behaviour is back to off in 6.2.1 RC1&2.
Message 2 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

Pretty much all of the critical data is backed up but it would take many days to get back to where I am now and I would lose all of the non-critical data, which is undesirable and why I store it on RAID5.

I haven't pushed any buttons yet, as I wanted to take advice before proceeding. I simply tried to get the front panel to respond and failed. Two hours later, still not responding to network pings.
Message 3 of 27
vandermerwe
Master

Re: ReadyNAS 316 not responding after disk balance task star

I don't know exactly how long a balance should take, it depends on the amount of space in the file system.
I would not expect it to take longer than a full resync which on your volume should take less than 24 hours.
Perhaps wait for 24 hours and see what happens.
If it doesn't come back online I would try to power off using the front button ( in fact I would do that now, but I have a full and up to date backup of my critical and non- critical data).
When did you buy the 316?
Message 4 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

I just want to confirm if this is expected behaviour for a balance, I never expected it to go offline. It was offline for the first several minutes after the task kicked off and I'd at least expect the front panel to respond or show "balancing" or similar.

I bought the device in April this year.
Message 5 of 27
vandermerwe
Master

Re: ReadyNAS 316 not responding after disk balance task star

No it should not be offline.
Message 6 of 27
vandermerwe
Master

Re: ReadyNAS 316 not responding after disk balance task star

I'll run a balance on my 316 now and see how long it takes (5x4 tb drives in raid 6)
Message 7 of 27
vandermerwe
Master

Re: ReadyNAS 316 not responding after disk balance task star

In fact my 316 did a scheduled balance about 2 weeks ago. It was finished within 2 minutes.
I would try to power down.
Message 8 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

vandermerwe wrote:
In fact my 316 did a scheduled balance about 2 weeks ago. It was finished within 2 minutes.
I would try to power down.


How much free space do you have on your system? This would be the first time I have run a balance since new, does it take longer the first time?
Message 9 of 27
vandermerwe
Master

Re: ReadyNAS 316 not responding after disk balance task star

I'm using 6.4/10.9.
Message 10 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

OK, thanks, in the absence of any more responses...I'm going to have a try with the power button.
Message 11 of 27
vandermerwe
Master

Re: ReadyNAS 316 not responding after disk balance task star

That 2 minute balance was the first time I've run a balance. Remember it's a new feature on 6.2.
Message 12 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

I tried pulling the LAN cables out - I remembered that once woke my NV+ up when it was frozen. Nothing.
I pushed the power button. Nothing.
I pushed it again. Nothing.
I pushed it again. Nothing.
I held it down. Predictably, the power went off.

I then pushed it again..."Booting".
Seemed to boot successfully.
Went back to my workstation and it's up, but rebuilding. Email confirmation of this.

Looking at the web UI, the volume details shows me this:



Confirms that the rebalance doesn't appear to have completed and now I have to wait for a rebuild too.

No obvious information in the logs:

Mon Dec 15 2014 18:47:57	
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__14_00_07' from share or LUN 'Shared'.
Mon Dec 15 2014 18:47:57
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__15_00_13' from share or LUN 'Shared'.
Mon Dec 15 2014 18:47:57
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__16_00_22' from share or LUN 'Shared'.
Mon Dec 15 2014 18:47:56
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__17_00_31' from share or LUN 'Shared'.
Mon Dec 15 2014 18:47:56
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__18_00_39' from share or LUN 'Shared'.
Mon Dec 15 2014 18:47:32
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__14_00_05' from share or LUN 'joey'.
Mon Dec 15 2014 18:47:31
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__15_00_11' from share or LUN 'joey'.
Mon Dec 15 2014 18:47:31
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__16_00_19' from share or LUN 'joey'.
Mon Dec 15 2014 18:47:30
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__17_00_29' from share or LUN 'joey'.
Mon Dec 15 2014 18:47:30
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__18_00_37' from share or LUN 'joey'.
Mon Dec 15 2014 18:47:30
Volume: Rebuilding started for Volume 'data'.
Mon Dec 15 2014 18:47:19
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__14_00_04' from share or LUN 'Frazer'.
Mon Dec 15 2014 18:47:19
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__15_00_10' from share or LUN 'Frazer'.
Mon Dec 15 2014 18:47:19
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__16_00_18' from share or LUN 'Frazer'.
Mon Dec 15 2014 18:47:19
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__17_00_28' from share or LUN 'Frazer'.
Mon Dec 15 2014 18:47:18
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__18_00_36' from share or LUN 'Frazer'.
Mon Dec 15 2014 18:47:06
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__14_00_03' from share or LUN 'DropboxCopy'.
Mon Dec 15 2014 18:47:06
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__15_00_10' from share or LUN 'DropboxCopy'.
Mon Dec 15 2014 18:47:06
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__16_00_18' from share or LUN 'DropboxCopy'.
Mon Dec 15 2014 18:47:06
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__17_00_27' from share or LUN 'DropboxCopy'.
Mon Dec 15 2014 18:47:05
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__18_00_35' from share or LUN 'DropboxCopy'.
Mon Dec 15 2014 18:47:05
Volume: Volume 'data' usage exceeds 80 % of size (10.9 TB).
Mon Dec 15 2014 18:46:58
System: ReadyNASOS background service started.
Mon Dec 15 2014 18:46:58
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__14_00_03' from share or LUN 'ben'.
Mon Dec 15 2014 18:46:58
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__15_00_09' from share or LUN 'ben'.
Mon Dec 15 2014 18:46:57
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__16_00_16' from share or LUN 'ben'.
Mon Dec 15 2014 18:46:57
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__17_00_25' from share or LUN 'ben'.
Mon Dec 15 2014 18:46:56
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__18_00_34' from share or LUN 'ben'.
Mon Dec 15 2014 18:46:40
System: ReadyNASOS service or process (smbd) was restarted.
Mon Dec 15 2014 13:13:08
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__13_01_01' from share or LUN 'Shared'.
Mon Dec 15 2014 13:12:47
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__13_00_59' from share or LUN 'joey'.
Mon Dec 15 2014 13:12:37
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__13_00_58' from share or LUN 'Frazer'.
Mon Dec 15 2014 13:12:27
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__13_00_58' from share or LUN 'DropboxCopy'.
Mon Dec 15 2014 13:12:17
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__13_00_57' from share or LUN 'ben'.
Mon Dec 15 2014 13:01:09
Miscellaneous: Snapshot 'c_1418648467' is successfully created for share or LUN 'torrents'.
Mon Dec 15 2014 13:01:06
Miscellaneous: Snapshot 'c_1418648464' is successfully created for share or LUN 'Shared'.
Mon Dec 15 2014 13:01:03
Miscellaneous: Snapshot 'c_1418648461' is successfully created for share or LUN 'media'.
Mon Dec 15 2014 13:00:59
Miscellaneous: Snapshot 'c_1418648457' is successfully created for share or LUN 'joey'.
Mon Dec 15 2014 13:00:54
Miscellaneous: Snapshot 'c_1418648452' is successfully created for share or LUN 'Frazer'.
Mon Dec 15 2014 13:00:51
Miscellaneous: Snapshot 'c_1418648449' is successfully created for share or LUN 'DropboxCopy'.
Mon Dec 15 2014 13:00:47
Miscellaneous: Snapshot 'c_1418648443' is successfully created for share or LUN 'ben'.
Mon Dec 15 2014 12:11:49
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__12_00_55' from share or LUN 'Shared'.
Mon Dec 15 2014 12:11:29
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__12_00_53' from share or LUN 'joey'.
Mon Dec 15 2014 12:11:18
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__12_00_52' from share or LUN 'Frazer'.
Mon Dec 15 2014 12:11:08
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__12_00_52' from share or LUN 'DropboxCopy'.
Mon Dec 15 2014 12:10:59
snapshot: Smart Snapshot successfully delete snapshot '2014_12_13__12_00_51' from share or LUN 'ben'.
Mon Dec 15 2014 12:00:38
Miscellaneous: Snapshot 'c_1418644837' is successfully created for share or LUN 'torrents'.
Mon Dec 15 2014 12:00:36
Miscellaneous: Snapshot 'c_1418644833' is successfully created for share or LUN 'Shared'.
Mon Dec 15 2014 12:00:32
Miscellaneous: Snapshot 'c_1418644829' is successfully created for share or LUN 'media'.
Mon Dec 15 2014 12:00:26
Miscellaneous: Snapshot 'c_1418644823' is successfully created for share or LUN 'joey'.
Mon Dec 15 2014 12:00:21
Miscellaneous: Snapshot 'c_1418644819' is successfully created for share or LUN 'Frazer'.
Mon Dec 15 2014 12:00:16
Miscellaneous: Snapshot 'c_1418644815' is successfully created for share or LUN 'DropboxCopy'.
Mon Dec 15 2014 12:00:13
Miscellaneous: Snapshot 'c_1418644809' is successfully created for share or LUN 'ben'.
Mon Dec 15 2014 11:25:01
Volume: Balance started for Volume 'data'.


I've used the download logs button in case it's of use for diagnosis of why the 316 froze up.
Message 13 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

Since the 6.2.0 upgrade, I'm seeing this a lot too:



This happened before today's server side lock-up and has occurred several times. Never before 6.2.0.
Message 14 of 27
vandermerwe
Master

Re: ReadyNAS 316 not responding after disk balance task star

You have snapshots enabled on a lot of shares and it looks like they're hourly.
One of the shares says torrents, is this a transmission share . In any case you should not have snapshots enabled for this type of share.
What apps do you have installed?

Do you need hourly snapshots on so many shares?
Message 15 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

vandermerwe wrote:
You have snapshots enabled on a lot of shares and it looks like they're hourly.
One of the shares says torrents, is this a transmission share . In any case you should not have snapshots enabled for this type of share.
What apps do you have installed?

Do you need hourly snapshots on so many shares?


I have snapshots enabled on all my shares pretty much, I see no reason not to. If I'm going to do it, why not as often as possible? It's saved me from a few cockups.

The torrents share is pretty much a WORM area for seeding. My PC has limited local SSD storage so I move completed torrents to this share as I don't otherwise have enough room to host big distros for long. Why not have snapshots enabled for this? My slowdowns don't coincide with snapshot events.

No apps installed.

I see no reason not to have hourly snapshots, I bought a more powerful device for this purpose. If it's not capable, this should be documented.
Message 16 of 27
mdgm-ntgr
NETGEAR Employee Retired

Re: ReadyNAS 316 not responding after disk balance task star

Can you send me your logs (see the Sending Logs link in my sig)?

The reason we don't recommend CoW or snapshots for the Transmission share is that they don't work well with data that is modified a huge number of times.

With CoW each write does not overwrite the previous data. So with a huge number of writes a file will quickly become very fragmented.

Once the file has downloaded you could then copy it to a share which has CoW enabled.

Note that disabling CoW does not affect existing files.
Message 17 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

mdgm wrote:
Can you send me your logs (see the Sending Logs linK in my sig)?

The reason we don't recommend CoW or snapshots for the Transmission share is that they don't work well with data that is modified a huge number of times.

With CoW each write does not overwrite the previous data. So with a huge number of writes a file will quickly become very fragmented.

Once the file has downloaded you could then copy it to a share which has CoW enabled.

Note that disabling CoW does not affect existing files.


Thanks for the response.

I read about the torrent/CoW stuff on another thread but I'm not using the NAS in this way. I run uTorrent on a Workstation which uses local SSD. Only when the torrent is completed and seeding does it get moved to my NAS share.

In real terms, I find myself using snapshots a lot as I miss the trash function from my NV+. Without the trash function, if you accidentally delete something it's gone for good. With snapshots, I can easily recover that.

Logs have been sent over as requested.

Thanks,

Ben
Message 18 of 27
mdgm-ntgr
NETGEAR Employee Retired

Re: ReadyNAS 316 not responding after disk balance task star

Ah, thanks for clarifying.

Your volume is very full.

Total devices 1 FS bytes used 9.34TiB
devid 1 size 10.90TiB used 9.98TiB path /dev/md127


You should have got warnings about volume usage once your usage exceeded 70%. Depending on how much you have been making in the way of changes deleting some older snapshots then doing another balance may help.

Snapshots can provide some protection against accidental file deletions but you still need backups of important data. RAID and snapshots are great but there are problems that they can't protect against such as too many disk failures, fire, flood, theft etc.

You may find that with a very full volume, a balance works better on 6.2.1-RC3: http://www.readynas.com/forum/viewtopic.php?f=154&t=78866
Message 19 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

In the past, 70% full was a concern - these days it means I still have around 3TB of storage left, or a whole drive in my configuration. It's not all going to be used up over night, far from it.

I backup all critical data to external drives (stored offsite) and the cloud. There is a practical limit to how often these tasks are performed though. If I download something in the morning and then inadvertently delete it in the afternoon, then the overnight backup isn't going to cut it. That's what I use snapshots for - previously the trash was sufficient.

Are you implying only 70% of the storage in my device is useable without getting into a "warning" situation? That's really quite a waste of storage if I can't use it without things going wrong.

Sounds like I should try the latest beta, but one assumes I should wait for this rebuild to complete before upgrading?

Web interface implies the drive balance is no longer running, but nothing in the logs to show it completed.
Message 20 of 27
mdgm-ntgr
NETGEAR Employee Retired

Re: ReadyNAS 316 not responding after disk balance task star

jaffacake wrote:

Are you implying only 70% of the storage in my device is useable without getting into a "warning" situation? That's really quite a waste of storage if I can't use it without things going wrong.

You can use more than 70% of the space, however if you let it get very full like yours is currently you are more likely to run into issues. We have been adding changes to improve how things work when the volume is quite full. We aim to warn well in advanced of there being a problem in order to provide time for
jaffacake wrote:
I should wait for this rebuild to complete before upgrading?

Yes
jaffacake wrote:

Web interface implies the drive balance is no longer running, but nothing in the logs to show it completed.

Well I guess it has completed then.
Message 21 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

OK, thanks. Still another 12+ hours until rebuild complete...will feedback if I find out any more.
Message 22 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

Rebuild completed so I patched up to 6.2.1 RC3.

Now running a scrub to make sure all is healthy before running a balance/defrag.
Message 23 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

Think I may have resolved my performance issues.

1) AFP was enabled. I never used this and wouldn't have turned it on. Perhaps a firmware update turned this back on? I've turned this off now.

2) DLNA was enabled. Given the volume of home videos etc. stored on here, this would likely slow things down a lot. DIsabled now.

Edit - still scrubbing. 22% has taken 6 hours.
Message 24 of 27
jaffacake
Tutor

Re: ReadyNAS 316 not responding after disk balance task star

Scrubbing completed so I ran a balance.

It took 1hr 40mins and completed successfully. I originally built the system with 4 drives and added a fifth, so I guess there may have been some optimisation to do in this area.

Whatever problem I had now appears to have gone away, so thanks to all for the input.

Now for a defrag.
Message 25 of 27
Top Contributors
Discussion stats
  • 26 replies
  • 3205 views
  • 0 kudos
  • 5 in conversation
Announcements