Reply

Size of files/folders vs "size on disk" MASSIVELY different?

r00x
Aspirant

Size of files/folders vs "size on disk" MASSIVELY different?

Hi there, I'm on my NVX via a windows 8.1 machine, just going through some image folders and tidying up a little, when I realised there is a simply massive disparity between the "size" and the "size on disk" reported by the folder/file properties when browsing the share in Windows.

I really do mean massive. A folder with 144kB of data is listed as being 400MB in size, for example. More: 6MB > 1.05GB. 102MB > 7.03GB. 152MB > 74GB.

I've read somewhere that the ReadyNAS may report unusually high filesizes as this somehow speeds up SMB. Is this what's happening here? All my data seems to be intact (it is also backed up, by the way), I'm mostly wondering if this is an effect of having just expanded the array (went from 1.36TB to 2.7ish today as I swapped in a larger drive). I suppose I should add that I've never noticed this before, either, although my observational skills are not stellar so perhaps it's slipped under my radar over the last year or so 😄

Oh, should also add that this is only really happening in the pictures and documents share as far as I can see right now. No hidden files are present (that I can find, anyway). Does not seem to correlate with file quantity but does seem worse where smaller files are involved (documents, pics). Big files in the movies, software folders etc are not showing the same disparity.

Hoping this is normal!! 🙂
Message 1 of 24

Accepted Solutions
rjameson
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

I ran again the volume consistency check manually and it found volume scan was needed, so rebooted with the vol scan, it reported fixed problems...re-ran the scrub - that finished good in approx 7hrs. Problem still there on file count....so I installed TreeSize Pro - to find out in the 100+ folders - which one tripped the size, and low & behold - found the single file showing 16TB in size (yes, impossible on a NAS with only 2.6TB). It was a quickbooks file. Zero clue how, but I opened the file, backed it up, resaved and now the file size reports normal at 58mb. Whew!!! TG for treesize pro finding exactly what file. The wonders of why or how this can happen, but guess any corrupt file can produce strange results.

I am sure CP now will be happy I am not trying to push 17.1TB of data ...thru our unlimited data plan. 🙂

View solution in original post

Message 23 of 24

All Replies
mdgm-ntgr
NETGEAR Employee Retired

Re: Size of files/folders vs "size on disk" MASSIVELY differ

I think this will be addressed in 4.2.26 (or was it already addressed in 4.2.25? What firmware are you running?), and yes it is to improve performance.

Your data should all be intact.
Message 2 of 24
ReadyAS
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

I observe the same big difference.
On my RNDP4000 with firmware 4.2.24 I see my Pictures folder:

1. Under Windows 8:
Size: 328 GB (352 863 936 767)
Size on disk: 328 GB (352 903 155 712)

2. Under Windows 8.1:
Size: 328 GB (352 863 936 767)
Size on disk: 7,72 TB (8 498 498 764 800)

Under Windows 8 everything is ok, but under Windows 8.1 there is a big difference as you can see above. Why??
Message 3 of 24
StephenB
Guru

Re: Size of files/folders vs "size on disk" MASSIVELY differ

The issue is described here: viewtopic.php?f=51&t=72934&p=406274#p406274

It is fixed in 4.2.25.
Message 4 of 24
ReadyAS
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

Thanks StephenB. I have upgraded to 4.2.25 and now I can see:

1. Under Windows 8:
Size: 328 GB (352 863 936 767)
Size on disk: 328 GB (352 903 155 712)

2. Under Windows 8.1:
Size: 328 GB (352 863 936 767)
Size on disk: 331 GB (355 647 881 216)

So still there is difference in size on disk. Could you explain me why, please?

Whereas I use 'du -b' command in the system shell I can see another size than Windows presents:
352 868 872 447

We have 2014 and such basic problems with calculate the size of files??? It's very irritating. Now we can't be sure which value out of the above is correct.
Message 5 of 24
fastfwd
Virtuoso

Re: Size of files/folders vs "size on disk" MASSIVELY differ

ReadyAS wrote:
1. Under Windows 8:
Size: 328 GB (352 863 936 767)
Size on disk: 328 GB (352 903 155 712)

2. Under Windows 8.1:
Size: 328 GB (352 863 936 767)
Size on disk: 331 GB (355 647 881 216)

So still there is difference in size on disk. Could you explain me why, please?

The smallest unit of space that can be allocated on your drives is called a "cluster" or "allocation unit". The size of a cluster varies from one drive to another, although it is typically around 4K bytes.

With a 4KB cluster size, even a one-byte file will require 4KB of disk space. You can see this for yourself: Create a text file containming a single character, then look at its properties in Windows. It'll be shown as something like "Size: 1, Size on disk: 4096". That "size on disk" indicates that a full cluster was allocated even though only one byte of storage was needed.

What you're seeing with your NAS is that Windows 8 assumes one cluster size for its "Size on disk" display and Windows 8.1 assumes another. I don't know what sizes they're using, but if I had to guess from your numbers I would say that Windows 8 is assuming 4K cluster sizes and Windows 8.1 is assuming 256K cluster sizes. If you tell me how many files are on the disk, I can give you a more certain answer.

ReadyAS wrote:
Whereas I use 'du -b' command in the system shell I can see another size than Windows presents:
352 868 872 447

That size is the same as is displayed by Windows 8 and Windows 8.1 on the "Size:" line.

ReadyAS wrote:
We have 2014 and such basic problems with calculate the size of files??? It's very irritating. Now we can't be sure wchich value out of the above is correct.

It depends on what you mean by "correct". If you want to know the sum of the actual file sizes of all your files, the Windows "Size:" value (for both Win8 and Win8.1) is the correct one. If you want to know exactly how much space on disk is being used, the Windows 8 number is probably correct.
2 × Pro 6 [Main & Backup] – XRAID2 Dual-Redundant (RAID6): 14.53TB (110MB/s Write, 123MB/s Read)
  • CPU: 2.4GHz E6600 SL9ZL - RAM: 8GB (PSD28G800K) - BIOS: 07/26/2010 - OS: 6.10.3 w/ custom kernel - Drives: 6 × ST4000VN000 / ST4000VN008
    Fans: Noctua NF-S12A FLX (Case), NF-R8 PWM (CPU), NF-R8 w/LNA (Power Supply) - USB: STGY8000400 - UPS: APC BR1000G
Ultra 2 Plus [for experimentation and testing] – JBOD: 8TB (110MB/s Write, 123MB/s Read)
  • CPU: Standard 1.8GHz Atom D525 - RAM: 2GB (CT25664BC1067) - BIOS: 11/02/2011 - OS: 4.2.31 - Drives: 2 × ST4000VN000 - Fan: Noctua NF-B9 - USB: ST31000340
Message 6 of 24
ReadyAS
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

fastfwd wrote:
ReadyAS wrote:
1. Under Windows 8:
Size: 328 GB (352 863 936 767)
Size on disk: 328 GB (352 903 155 712)

2. Under Windows 8.1:
Size: 328 GB (352 863 936 767)
Size on disk: 331 GB (355 647 881 216)

So still there is difference in size on disk. Could you explain me why, please?

The smallest unit of space that can be allocated on your drives is called a "cluster" or "allocation unit". The size of a cluster varies from one drive to another, although it is typically around 4K bytes.

With a 4KB cluster size, even a one-byte file will require 4KB of disk space. You can see this for yourself: Create a text file containming a single character, then look at its properties in Windows. It'll be shown as something like "Size: 1, Size on disk: 4096". That "size on disk" indicates that a full cluster was allocated even though only one byte of storage was needed.

I know what the cluster mean, but thanks for explanation.

fastfwd wrote:

What you're seeing with your NAS is that Windows 8 assumes one cluster size for its "Size on disk" display and Windows 8.1 assumes another. I don't know what sizes they're using, but if I had to guess from your numbers I would say that Windows 8 is assuming 4K cluster sizes and Windows 8.1 is assuming 256K cluster sizes. If you tell me how many files are on the disk, I can give you a more certain answer.

There is 80935 files in 592 folders.

fastfwd wrote:

ReadyAS wrote:
Whereas I use 'du -b' command in the system shell I can see another size than Windows presents:
352 868 872 447

That size is the same as is displayed by Windows 8 and Windows 8.1 on the "Size:" line.

No, it's different: 352 868 872 447 vs 352 863 936 767 (not so much, but different)

fastfwd wrote:

ReadyAS wrote:
We have 2014 and such basic problems with calculate the size of files??? It's very irritating. Now we can't be sure wchich value out of the above is correct.

It depends on what you mean by "correct". If you want to know the sum of the actual file sizes of all your files, the Windows "Size:" value (for both Win8 and Win8.1) is the correct one. If you want to know exactly how much space on disk is being used, the Windows 8 number is probably correct.


The difference between W8 and W8.1 is not acceptable in my opinion, but it is Microsoft fault.
Message 7 of 24
fastfwd
Virtuoso

Re: Size of files/folders vs "size on disk" MASSIVELY differ

ReadyAS wrote:
There is 80935 files in 592 folders.

In that case, Windows 8 is using 1KB allocation units and Windows 8.1 is using 64KB.

ReadyAS wrote:
No, [the du -b result and the Windows Size reports are] different: 352 868 872 447 vs 352 863 936 767 (not so much, but different)

Ah, my mistake.

That difference is a mystery to me as well; I don't know enough about the Linux filesystems to know where the extra ~60 bytes per file (or ~8K per directory) is coming from.

ReadyAS wrote:
The difference between W8 and W8.1 [size on disk] is not acceptable in my opinion, but it is Microsoft fault.

I do not know for sure -- this is only a guess -- but I assume that the difference is due to Windows 8.1 being "smarter" about calculating cluster size. Rather than simply accepting the 1KB cluster size reported by the NAS, perhaps Windows 8.1 assumes that the cluster size is the GREATER of that 1KB and some block-size parameter used by SMB. (I don't know anything about SMB, so I can't say exactly what parameter that might be.) If the SMB protocol requires an acknowledgement (or a similar time-wasting action) after every block, it would make sense for Netgear to set the SMB block size to an absurdly high value in order to make transfers happen as quickly as possible... And the original, pre-OS4.2.25, Windows 8.1 size on disk that you posted -- 8498498764800 -- indicates that Windows 8.1 was indeed seeing a huge block size, somewhere around 32MB.

I guess that the SMB block size was decreased in OS4.2.25 to make the Win8.1 size on disk more accurate, but that Netgear couldn't make it as small as the actual 1KB cluster size without hurting the transfer speed too much, so they compromised at 64KB: Close to the correct size on disk, but still fast.

As I said, though, only a guess.
2 × Pro 6 [Main & Backup] – XRAID2 Dual-Redundant (RAID6): 14.53TB (110MB/s Write, 123MB/s Read)
  • CPU: 2.4GHz E6600 SL9ZL - RAM: 8GB (PSD28G800K) - BIOS: 07/26/2010 - OS: 6.10.3 w/ custom kernel - Drives: 6 × ST4000VN000 / ST4000VN008
    Fans: Noctua NF-S12A FLX (Case), NF-R8 PWM (CPU), NF-R8 w/LNA (Power Supply) - USB: STGY8000400 - UPS: APC BR1000G
Ultra 2 Plus [for experimentation and testing] – JBOD: 8TB (110MB/s Write, 123MB/s Read)
  • CPU: Standard 1.8GHz Atom D525 - RAM: 2GB (CT25664BC1067) - BIOS: 11/02/2011 - OS: 4.2.31 - Drives: 2 × ST4000VN000 - Fan: Noctua NF-B9 - USB: ST31000340
Message 8 of 24
ReadyAS
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

Thank you fastfwd for your explanations.
Allocation unit is established during the process of formatting the partition and every system should read it correctly.
Tha allocation unit is the same for specific partition so what's the problem to read it, even through SMB. It's strange.
It's very bad if system assumes another size of allocation unit that in fact is.
In my opinion there are simply a weak points in the operating systems.
Message 9 of 24
StephenB
Guru

Re: Size of files/folders vs "size on disk" MASSIVELY differ

This discrepancy is not about the physical allocation units on the disks. Windows 8 is calculating everything based on what the SAMBA server in the NAS reports.

So the differences are a combination of the assumptions Windows 8 is making about the the SMB server (in its calculations), and the details of what SAMBA is reporting. Netgear already stated that SAMBA "fudges" some values. I believe that is to help overall interoperability, though the size reports are clearly not exact.

The actual file size is of course what is reported through du. That is looking at the native file system, and is not filtered through SAMBA to an OS that has no idea of what the real file system is.
Message 10 of 24
rjameson
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

I just loaded on a Pro6, the 4.2.26 - but still am having the file size issue in a much grander scale. I have only 630gb total used (5x1tb drive in dual redundancy), yet after in windows 2003 I check file space on the folders on the NAS - it shows 930,000 files, and 17.1TB. As it counts away, when it gets near just over 700,000 files - it jumps from 20gb to 17.1TB. This is causing some havoc with our crashplan load as well - when we backup to the cloud - it shows 17.1TB as well. Is this bug still in the latest version?
Message 11 of 24
StephenB
Guru

Re: Size of files/folders vs "size on disk" MASSIVELY differ

This sounds very different from the windows 8 problem - which is specific to SMB. I am running Crashplan with 4.2.26, and am not seeing any symptoms like this.

It sounds to me like something weird has happened with your file system. Maybe you should poke around with SSH?
Message 12 of 24
rjameson
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

I noticed it by adding crashplan, and even if I used the windows explorer - same thing. Not sure what to even look for thru SSH (which I have turned on) to know what is bad/good. Hate to do a full rebuild if it can be avoided. Any pointers what I should be looking for? I am not any type of linux pro, but have gone thru enough to get info out of it.
Message 13 of 24
StephenB
Guru

Re: Size of files/folders vs "size on disk" MASSIVELY differ

start with
cd /c/
df -h .

and tell us what it says.
Message 14 of 24
rjameson
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

Filesystem Size Used Avail Use% Mounted on
/dev/md0 4.0G 1.6G 2.3G 42% /
tmpfs 16K 0 16K 0% /USB
/dev/c/c 2.7T 630G 2.0T 24% /c
/dev/c/c_2014_02_25_03_00
2.7T 630G 2.0T 24% /c_2014_02_25_03_00
/dev/sdf1 280G 280G 0 100% /USB/USB_HDD_1
ma-storage:/c#
Message 15 of 24
StephenB
Guru

Re: Size of files/folders vs "size on disk" MASSIVELY differ

Thanks. So ssh confirms that you are only using 630 GB.

Are you running crashplan on the Windows Server, or did you install it on the NAS?
Message 16 of 24
rjameson
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

CP is loaded on the NAS. However, it was only when I loaded it that I saw it misrepresent the size. Even if I use the windows file explorer to the readynas, it shows the same bogus behaviour when you right click/properties on the share.
Message 17 of 24
StephenB
Guru

Re: Size of files/folders vs "size on disk" MASSIVELY differ

You are set up for only one share? Or only one share shows the problem?

I can't think of any straightforward mechanism for CP to corrupt the file system in this way.

Perhaps there are links in the share that are creating the discrepancy? You can try navigating to the share, and then entering

du . -sh

That should align with the df command you entered before.

Then try

du . -shD

and see if you get a different answer.

From practical point of view, the simplest approach might be to save the share to a PC or USB drive, delete it, and re-create it. Perhaps then run the maintenance programs (scrub and volume checks).
Message 18 of 24
rjameson
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

I have multiple shares, just this one is the largest. I navigated to the share, did the du -sh, it shows 167G, I did the same at the root and it matches from the df - which is 630G.

I did the du -shD at the root folder - and the response was: 630G

Doing the same at the share - and the response was: 167G

Is it possible the readynas does not handle this many files/folders? I would be surprised, but something it tripping this up. I have the room, I may copy all the files from this share, to a new share - then try a directory size check....I am also going to try to get a directory size from a windows 7 PC - maybe 2003 server does not like it.

Note to add - I did a directory check for size on the snap folder - it does the same thing from my 2003 server - shows bogus 15TB total size.

I went to a win7 pc, highlighted all the folders in the share to get properties - same issue, after it gets to about 100gb, and around 770,000 files, it jumps rapidly to 105gb, 110gb, 130gb then 15.1TB in a matter of seconds after letting it count for about 10 minutes.
Message 19 of 24
StephenB
Guru

Re: Size of files/folders vs "size on disk" MASSIVELY differ

It is weird that CP is getting a different size from du and df, since CP is accessing the ext file system, and not using SAMBA.

I am thinking this is perhaps an issue in a specific share - whichever one that Win7 was scanning when it suddenly jumped. Can you remove that one share from the highlight, and see if that is the case?

Did you try opening a support ticket?
Message 20 of 24
rjameson
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

This is only the one share of a half dozen I have the issue with, and with the -snap folder of this share also having the same issue - is it possible a folder inside the share can do this? If so, that will be next to impossible to figure out. 🙂

If CP does not use Samba to get its directory info, then this does seem to be beyond a windows thing (as evidenced I get the same result from Win7 as I so 2003 server file explorer).

I ran the online consistency check - that came up clean. Although I ran the disk scrub with auto parity fix , after about 3hrs - it bombed at 64%. In checking SMART, 3 of the 5 drives show 2 reallocated sectors...not horrible - but wonder if that may be it.
Message 21 of 24
StephenB
Guru

Re: Size of files/folders vs "size on disk" MASSIVELY differ

Something wrong with the disks could possibly explain it.

The disk scrub failed to complete? That worries me more than the sector count by itself. Is there anything in the log that indicates why it failed?

What is your backup status on the data? Though it will be tedious, you might need to start copying folder by folder to get everything off that you can.
Message 22 of 24
rjameson
Aspirant

Re: Size of files/folders vs "size on disk" MASSIVELY differ

I ran again the volume consistency check manually and it found volume scan was needed, so rebooted with the vol scan, it reported fixed problems...re-ran the scrub - that finished good in approx 7hrs. Problem still there on file count....so I installed TreeSize Pro - to find out in the 100+ folders - which one tripped the size, and low & behold - found the single file showing 16TB in size (yes, impossible on a NAS with only 2.6TB). It was a quickbooks file. Zero clue how, but I opened the file, backed it up, resaved and now the file size reports normal at 58mb. Whew!!! TG for treesize pro finding exactly what file. The wonders of why or how this can happen, but guess any corrupt file can produce strange results.

I am sure CP now will be happy I am not trying to push 17.1TB of data ...thru our unlimited data plan. 🙂
Message 23 of 24
StephenB
Guru

Re: Size of files/folders vs "size on disk" MASSIVELY differ

I'm glad you found it.
Message 24 of 24
Top Contributors
Discussion stats
  • 23 replies
  • 51210 views
  • 0 kudos
  • 6 in conversation
Announcements