× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

iSCSI issues

HuyTong
Aspirant

iSCSI issues

hi

I have had this unit in production for a few years now. I used it as an iSCSI target for a network backup. now the issue in the last few months that has happened is that all my backups have failed,

I cannot recall firmware it was on before but its on the lastest one now and the issue is still there.

when it connects via iSCSI, and you try to write something to it, you can hear the hard drives going crazy. and it doesnt go crazy until you write something to it.

I also have an SMB share on and I can write to that fine it seems.

i've checked the each hard drive via the NAS, mouse over it, and it doesnt report any SMART errors on any drive.

i've taken the NAS off of the network and tried to connect to it from another computer instead of the server, same issue.

you can hear the hard drives just going mad once the iSCSI connection is made/writes are made, starts at 300MB/sec then drops to nothing and just hangs the PC.

 

any ideas?

Model: RN31400|ReadyNAS 300 Series 4-Bay
Message 1 of 13

Accepted Solutions
evan2
NETGEAR Expert

Re: iSCSI issues

@HuyTong

You may delete snapshots in NAS admin GUI,

Steps:

1. Login NAS admin GUI,

2. Open Shares=>Snapshots page,

3. Click Shares or iSCSI LUN, you will see all snapshots in the Share/LUN,

4. Select Share/LUN and right click, delete button will shown, then you can delete snapshots you selected.

iscsi.jpg

if you don't want to use snapshot, then you can disable in Share/LUN's setting,

Go to Share's page in NAS admin GUI, then right click a Share or LUN, open settings page, there is Snapshots settings.

You may set smart snapshot to Never.

View solution in original post

Message 4 of 13

All Replies
StephenB
Guru

Re: iSCSI issues

Download the log zip file from the web ui.  Then start by looking at the SMART stats in disk_info.log.  Perhaps look in the some other log files for clues on what's going wrong (system.log and kernel.log would be reasonable places to start).

 


@HuyTong wrote:

starts at 300MB/sec then drops to nothing and just hangs the PC.

 


Gigabit ethernet only carries about 110 MB/sec, so this starting value is caching in the client, and not a real speed.

Message 2 of 13
HuyTong
Aspirant

Re: iSCSI issues

ok

so I had reply and I thought I replied but it never made it here for some reason.

anyways the error I had are as follows,

Volume: Less than 5% of volume data's capacity is free. Performance on volume data is degraded. To improve performance, you must add capacity.

so what I did was delete the snapshots to the iSCSI and its working but the error is still at 10% but its working. so my question is this.

NAS has 4 1TB drives in a raid5, so 2.7TB total

I have a

iSCSI connection that is 2.3 TB in two drives.  2TB/1.7TB free and 350GB/ all free

SMB share thats 275GB/175 GB free

I dont understand the snapshot features and where does it store it? I can lower the iSCSI volume as I'm not using much of it .

what do you recommend?

 

Message 3 of 13
evan2
NETGEAR Expert

Re: iSCSI issues

@HuyTong

You may delete snapshots in NAS admin GUI,

Steps:

1. Login NAS admin GUI,

2. Open Shares=>Snapshots page,

3. Click Shares or iSCSI LUN, you will see all snapshots in the Share/LUN,

4. Select Share/LUN and right click, delete button will shown, then you can delete snapshots you selected.

iscsi.jpg

if you don't want to use snapshot, then you can disable in Share/LUN's setting,

Go to Share's page in NAS admin GUI, then right click a Share or LUN, open settings page, there is Snapshots settings.

You may set smart snapshot to Never.

Message 4 of 13
StephenB
Guru

Re: iSCSI issues


@evan2 wrote:

 

You may set smart snapshot to Never.


Another option is to switch to Custom Snapshot.  Then you can set the retention (either by number of snapshots or by age). I found that manages the snapshot space for me.  Generally I use 3 weeks retention on most shares, though of course the space needed depends on how much "churn" there is.

 

Also, I suggest running the balance function regularly (using the maintenance schedule in the settings wheel).  Running defrag will increase the snapshot space needed, especially on your iSCSI LUN.  So if you keep snapshots enabled on the LUN, you should keep that in mind.

Message 5 of 13
HuyTong
Aspirant

Re: iSCSI issues

hi, thank you for the replies but i'm still at a loss on what the issue was, and where this lack of space is being reported at/from? as I have stated before

NAS has four - 1TB drives in a raid5, so 2.7TB total

I have a

iSCSI connection that is 2.3 TB in two drives.  2TB/1.7TB free and 350GB/ all free

SMB share thats 275GB/175 GB free

where does the snapshot system use its space? I  have a lot of free space still, or is the reporting error is at the SMB share as I have it set for 2 snapshots and if it keeps the files for say 2 weekly and it uses all of the space then its full.

 

how do I reduce the LUN so I can add it to the SMB?  my iSCSI is disconnected already but it wont actually reduce in size, I can move it down from 2.3 TB to say 1 even as I'm not using much of it at all. or do I have to delete the LUN and start over?

Message 6 of 13
StephenB
Guru

Re: iSCSI issues


@HuyTong wrote:

 

iSCSI connection that is 2.3 TB in two drives.

 


Do you mean you have two LUNs?  If not, what do you mean?

 

Are the LUNs thick or thin?

 

if you have 2.3 TiB of thick LUNs and 275 GiB of files in shares, then you have 2.56 TiB of space used.  That's 95% full, and we haven't gotten to the snapshots yet. 

 

Note the LUNs are just block storage (e.g. opaque) to the NAS - free space that the iSCSI client sees isn't free space on the NAS.  A  LUN is just a file as far as storage system is concerned.

 

As far as shrinking the LUNs go, perhaps you should offload the data on the iSCSI client system, delete the LUNs and create new ones.

 


@HuyTong wrote:

 

 

where does the snapshot system use its space?


Snapshots are part of the BTRFS file system, and the space they take comes from the same free space pool that is used for files and metadata.  There's a post here that might help you understand how snapshots work: https://community.netgear.com/t5/ReadyNAS-in-Business/ReadyNAS-312-Need-Help-Understanding-Snapshots...

Message 7 of 13
HuyTong
Aspirant

Re: iSCSI issues

its 1 LUN, thick, its connected as 1 drive, but partitioned to two, 2TiB and 300GB, I think I was playing with something.

could this be why I cannot resize it down?

so the snapshot system uses free space but only makes copies fo the used space?

its only a backup target so I can just delete the LUN and recreate it. if I want to use the snapshot system, on the iSCSI LUN, how big should I make it? just 1 TB?

I attached a screenshot of the drive usage, I just dont want this error to come up and slow it down again.

Message 8 of 13
StephenB
Guru

Re: iSCSI issues


@HuyTong wrote:

its 1 LUN, thick, its connected as 1 drive, but partitioned to two, 2TiB and 300GB, I think I was playing with something.

could this be why I cannot resize it down?

 


That would be one reason. It's also thick. 

 

Do you actually need a LUN?  If you aren't using a VM, then you probably are better off using a share instead.

 

Another possibility is that you can get bigger disks, and expand the volume size.

 


@HuyTong wrote:

 

so the snapshot system uses free space but only makes copies fo the used space?

 


Your question doesn't make any sense to me, so I am not sure what you are asking.  When you first create a snapshot it uses no space.  If you then delete a file in the main share, that file remains in the snapshot - so the free space does not increase. The space used by the snapshot increases, and the space used in the share decreases.  If you then modify a file in the main share, the original data remains in the snapshot, plus you have the new data in the share.  The space in the snapshot goes up, and the free space goes down.  There are some more use cases outlined in the post I linked in above.

 

Your problem isn't because of your three snapshots anyway - the share is taking only 100 GB of data.  Your problem is that the LUN is too big.

 

 

 

Message 9 of 13
HuyTong
Aspirant

Re: iSCSI issues

its a VM, I attached the iSCSI drive to it.

the issue I tried to use a \\UNC\ share but it just didnt work as well. maybe I can go back to it and recreate my backup targets.

is there a point in doing snapshots on an iSCSI LUN? as its mentioned its just a page file does it snapshot the page file or the contents of the page file?

 

 

Message 10 of 13
StephenB
Guru

Re: iSCSI issues


@HuyTong wrote:

does snapshot the page file or the contents of the page file?

  


Let's try again.  A LUN is a big opaque data file as far as the NAS is concerned - just a bunch of blocks. It has no idea what the file system inside the LUN is.  All it sees are blocks being read and written by iSCSI client.

 

Let's imagine that snapshots are on for the LUN, with one snapshot created. Let's also imagine that the LUN container is not fragmented, and that the snapshot and LUN are identical.  All the blocks are held in common between the "real" LUN and the snapshot.  BTRFS only reports the blocks held in common once, and it shows them all as being part of the "real" LUN.  So as far as BTRFS is concerned the snapshot is taking no space.

 

Now the iSCSI client updates block X - some block in the middle. Blocks < X and blocks > X aren't touched. 

 

The original LUN container remains in the snapshot, and it is not fragmented.  As far as storage goes, the snapshot only uses one block of unique space - and that is the original block X. So BTRFS shows the snapshot as consuming one block.

 

The "real" LUN is now fragmented into three pieces - blocks 1 to X-1, block X, and blocks X+1 to the end.  The beginning and ending pieces are held in common with the snapshot - BTRFS reports them only once, and they are shown as being in the main data volume.  The updated block X is unique (only in the real LUN) - so it is also in the main data volume.  BTRFS shows the LUN as consuming the same space it had before.

 

The overall impact on free space in the example is that it has been reduced by 1 block.  The bigger impact might be the fragmentation of the "real" LUN.  That fragmentation gets worse every time you update another block. 

 

Now if you defrag in the VM using the iSCSI client, then the fragmentation in the "real" LUN container gets horribly worse.  You can defrag the "real" LUN container after that completes, but when you do that BTRFS duplicates all the blocks in common. The snapshot and the main LUN become entirely separate copies, with no blocks in common.  Space usage doubles.  That won't work for you because your data volume isn't big enough.

 

This suggests that in most cases using snapshots with a LUN will kill performance and might also consume a lot of free space.  So I think you should turn them off for the LUN.

 


@HuyTong wrote:

the issue I tried to use a \\UNC\ share but it just didnt work as well.


Is this a linux VM?  If so, you could back it up with rsync in the VM (using a NAS share with rsync enabled?)?

 

If backup to a share won't work reliably, then you'll need to make a new, smaller LUN.  Ideally you'd have enough free space to let you defrag the LUN occasionally (keeping snapshots off).

 

Message 11 of 13
HuyTong
Aspirant

Re: iSCSI issues

ok

i'll probably rebuild the LUN and expand the SMB share, have it as a backup.

disable the snapshot on the LUN, make it a thin LUN then?

does the HDD SMART reporting work fine on this unit?

I have another smaller unit where I can hear pinging from the box but the SMART isnt reporting any HDD errors.

Message 12 of 13
StephenB
Guru

Re: iSCSI issues


@HuyTong wrote:

make it a thin LUN then?

 


Not sure about that, it might be better to make it thick, but set the size more appropriately.

 


@HuyTong wrote:

 

does the HDD SMART reporting work fine on this unit?


The SMART reports are good, but the thresholds for alerts are much too high for my liking.  Perhaps periodically download the log zip file.  The SMART stats are in a couple of places, disk_info.log is one.

 


@HuyTong wrote:

 

I have another smaller unit where I can hear pinging from the box but the SMART isnt reporting any HDD errors.


Often disks with those symptoms don't report SMART errors. But if a disk is pinging or making clicking sounds, then it definitely needs to be replaced. 

 

Message 13 of 13
Top Contributors
Discussion stats
  • 12 replies
  • 3194 views
  • 0 kudos
  • 3 in conversation
Announcements