× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Something seems broken with iSCSI since 6.7.0

mangrove
Apprentice

Something seems broken with iSCSI since 6.7.0

Upgraded from 6.6.1 stable to 6.7.0 stable yesterday. Since then I can no longer mount my TrueCrypt containers residing on Readynas iSCSI volumes. Or, they mount after several minutes, but I can no longer see any data. What has changed regarding iSCSI?

Thick provision, no sync writes, no snapshots, no CHAP or connection lists.

Model: ReadyNAS RNDU4000|ReadyNAS Ultra 4 Chassis only
Message 1 of 26
mdgm-ntgr
NETGEAR Employee Retired

Re: Something seems broken with iSCSI since 6.7.0

Can you send in your logs (see the Sending Logs link in my sig)?

Message 2 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

Logs sent!

Message 3 of 26
mdgm-ntgr
NETGEAR Employee Retired

Re: Something seems broken with iSCSI since 6.7.0

Your data volume is 98% full. We don't recommend filling a volume that full especially when using iSCSI.

Do you have data in ordinary shares? If so, can you backup some of those files and free up some space and see if that helps?

Message 4 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

The device has only ever been used as an iSCSI drive. As such, it is created with the maximum LUN size allowed by NTGR, which for my unit is 7.354 TB according to the GUI. Only one LUN. There is nothing else in /data (verified with WinSCP) but a couple of NTGR settings files in /data/.apps etc. Only two apps are installed, smbplus and phpr6.

 

The GUI says "217.29 GB free of 8.17 TB" which indeed is a discrepancy, but that discrepancy seems to be precisely because of the upgrade from 6.6.1 to 6.7.0. New version of BTRFS acting up?

Message 5 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

root@link:~# btrfs fi df /data
Data, single: total=7.97TiB, used=7.96TiB
System, DUP: total=32.00MiB, used=992.00KiB
Metadata, DUP: total=512.00MiB, used=262.50MiB
GlobalReserve, single: total=96.00MiB, used=0.00B

 

root@link:/data# du -sh *
7.4T    8TB
0       Documents
0       home
0       Music
0       Pictures
0       Videos

Message 6 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

As you can see from initrd.log the device was factory defaulted in 2015/02/15 with 6.2.2 and the iSCSI volume was created and has been in use since then.

Message 7 of 26
mdgm-ntgr
NETGEAR Employee Retired

Re: Something seems broken with iSCSI since 6.7.0

Do/did you have Bit-rot protection or snapshots enabled on the LUN?

Message 8 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

No and no (snapshots are set to Smart/Never and that is not an UI I remember seeing, when created I think the smart/custom setting didn't exist, correct?).

Message 9 of 26
mdgm-ntgr
NETGEAR Employee Retired

Re: Something seems broken with iSCSI since 6.7.0

Still it looks like there's been some fragmentation over time. iSCSI doesn't work well when the volume gets this full.

Message 10 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

Frankly speaking, if there has been fragmentation, Netgear has created it by using the data drive as a temporary drive for updates or such. The data drive basically contains ONE FILE (the iSCSI file).

 

So how do I fix this?

 

Have you tried replicating the problem?

Message 11 of 26
Skywalker
NETGEAR Expert

Re: Something seems broken with iSCSI since 6.7.0

What is your initiator?

 

I see a steady stream of messages about the initiator sending FUA requests, but the iSCSI target is configured to claim no support for FUA.  I assume your Sync Writes setting is set to Disabled?  Can you try changing it to Allowed?

Message 12 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

Microsoft initiator (on Windows 7 64-bit).

 

Interestingly, with sync writes enabled, I can now mount the volume again. Thanks for that!

 

Doesn't solve the space free problem though Cat Sad and that clearly happened when upgrading the firmware. Is it risk free in this situation to try rebalancing (from the UI)?

Message 13 of 26
mdgm-ntgr
NETGEAR Employee Retired

Re: Something seems broken with iSCSI since 6.7.0

Your status.log shows your volume being over 95% full as far back as April 2016. It's not an issue caused by the upgrade.

Balancing shouldn't be attempted on a volume that full. That's not the solution. For optimal performance you should consider expanding your data volume (after updating your regular backup, of course).

Message 14 of 26
Skywalker
NETGEAR Expert

Re: Something seems broken with iSCSI since 6.7.0

Is there anything in particular this iSCSI volume is doing?  We tried to reproduce the issue, but only got a single FUA write request during mount time, and none after that.

Message 15 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

> Your status.log shows your volume being over 95% full as far back as April 2016. It's not an issue caused by the upgrade.

 

Whatever. As stated before, the volume has been created using Netgear recommendations, and never used for anything but this iSCSI volume.

 

> Is there anything in particular this iSCSI volume is doing?

 

No, its only purpose is to host a TrueCrypt volume. Sync Writes has been off for performance reasons for a long time.

Message 16 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

Now the free space is down to 210.09 GB, and the only thing done is upgrading to 6.7.3 and my normal iSCSI traffic.

Message 17 of 26
mdgm-ntgr
NETGEAR Employee Retired

Re: Something seems broken with iSCSI since 6.7.0

I would run 6.7.4 rather than 6.7.3.


Message 18 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

Updated to 6.7.4. Ran a balance for good measure. 210.02 GB free.

Message 19 of 26
mdgm-ntgr
NETGEAR Employee Retired

Re: Something seems broken with iSCSI since 6.7.0

Please send in fresh logs.

Message 20 of 26
ruffy91
Aspirant

Re: Something seems broken with iSCSI since 6.7.0

I have the same problem.

My ReadyNas is solely used as iSCSI Target.

I have created a LUN with 3.269TB (90% of Space) and activated Bit-Rot Protection for this LUN.

 

root@NAS:~# btrfs fi du /data/RB00_READYNAS02/.iscsi/
     Total   Exclusive  Set shared  Filename
   3.27TiB     3.19TiB           -  /data/RB00_READYNAS02/.iscsi//iscsi_lun_backing_store
     0.00B       0.00B           -  /data/RB00_READYNAS02/.iscsi//.serial_number
   3.27TiB     3.19TiB    75.14GiB  /data/RB00_READYNAS02/.iscsi/
root@NAS:~# btrfs fi df /data
Data, single: total=3.62TiB, used=3.62TiB
System, DUP: total=8.00MiB, used=480.00KiB
Metadata, DUP: total=5.00GiB, used=4.16GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
root@NAS:~# du -h --max-depth=2 /data | sort -hr
3.3T    /data/RB00_READYNAS02/.iscsi
3.3T    /data/RB00_READYNAS02
3.3T    /data
1.7M    /data/.apps
1.6M    /data/.apps/.freeapps
76K     /data/.apps/.readydlna
20K     /data/._share
16K     /data/._share/RB00_READYNAS02
0       /data/.vault
0       /data/.timemachine
0       /data/.purge
0       /data/home
0       /data/.apps/.xdg
0       /data/.apps/.forked-daapd
0       /data/.apps/DO_NOT_DELETE

Where have the missing 350GiB gone?

 

Best Regards

 

 

Message 21 of 26
ruffy91
Aspirant

Re: Something seems broken with iSCSI since 6.7.0

Got it!

The iSCSI File is heavily fragmented (over 1 Million Extents!)

I disabled CoW for the file (chattr +C <file>) and defragmented extents smaller than 100M (btrfs fi defrag -f -t 100M /data/RB00_READYNAS01/.iscsi/iscsi_lun_backing_store).

When enough space was freed up (>50GB) I stopped defragmentation and started again but this time for extents smaller than 10G.

 

I hope the file won't fragment again this heavy after i disabled CoW.

Message 22 of 26
Skywalker
NETGEAR Expert

Re: Something seems broken with iSCSI since 6.7.0

You can't disable CoW on existing files that contain data.  So the chattr command you ran had no effect.

 

You may be able to improve fragmentation characteristics by adding the "autodefrag" mount option to the /etc/fstab entry.

Message 23 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

Well, well, well.

iscsi_lun_backing_store: 1616701 extents found

Message 24 of 26
mangrove
Apprentice

Re: Something seems broken with iSCSI since 6.7.0

Even though repeated defragging has freed up some space on the drive (835 GB "free" as of now), the originally reported problem still stands: when "sync writes" are disabled, TrueCrypt iSCSI volumes are no longer mountable since 6.7.0.

Message 25 of 26
Top Contributors
Discussion stats
  • 25 replies
  • 6774 views
  • 1 kudo
  • 4 in conversation
Announcements