Re: OS6 now works on x86 Legacy WARNING: NO NTGR SUPPORT!
Okay. . . next stupid question:
In my ongoing quest to determine if OS6 is ready to house any actual data that I give a hoot about, I decided to do a little "Fault Tolerance" testing.
Here's the test platform:
A "new out of the box" ReadyNAS Pro with stock CPU, and Memory upgraded to 4GB's
4 X 2TB Western Digital WD2002FYPS RE4 Hard Drives populated in drive bays 1 through 4 (RAID 5 - X-RAID2)
RAID Array build process completed and stable, then rested for 12 hours
BIOS recently upgraded to v2.0 (Thanks mdgm!)
After confirmation that test data could be written to and read from the NAS, I pulled the drive out of bay 4 (hot) to pseudo-simulate a drive failure.
Waited 15 minutes, reconfirmed that test data could be written to and read from the NAS, then I replaced the drive in bay 4 with a different unformatted Western Digital WD2002FYPS RE4 Hard Drive. I checked the GUI which had the expected notifications of drive removals and insertions, but that's about it. I kind of expected the new drive to be added back into the array after the obligatory re-sync period. To the contrary, the new drive is just sitting there and the RAID5 array seems oblivious to the fact that a fresh drive is available for reintegration back into the array. I'm obviously missing something here. . .
Just for S's&G's I've rebooted, pulled and reinserted the blank drive a couple of times, skimmed the official OS6 documentation; I can't figure out how to reintegrate the new drive into the exsisting array.
This GUI currently shows this:
School me please! This should not be this counterintuitive. . . :nashammer:
In my ongoing quest to determine if OS6 is ready to house any actual data that I give a hoot about, I decided to do a little "Fault Tolerance" testing.
Here's the test platform:
A "new out of the box" ReadyNAS Pro with stock CPU, and Memory upgraded to 4GB's
4 X 2TB Western Digital WD2002FYPS RE4 Hard Drives populated in drive bays 1 through 4 (RAID 5 - X-RAID2)
RAID Array build process completed and stable, then rested for 12 hours
BIOS recently upgraded to v2.0 (Thanks mdgm!)
After confirmation that test data could be written to and read from the NAS, I pulled the drive out of bay 4 (hot) to pseudo-simulate a drive failure.
Waited 15 minutes, reconfirmed that test data could be written to and read from the NAS, then I replaced the drive in bay 4 with a different unformatted Western Digital WD2002FYPS RE4 Hard Drive. I checked the GUI which had the expected notifications of drive removals and insertions, but that's about it. I kind of expected the new drive to be added back into the array after the obligatory re-sync period. To the contrary, the new drive is just sitting there and the RAID5 array seems oblivious to the fact that a fresh drive is available for reintegration back into the array. I'm obviously missing something here. . .
Just for S's&G's I've rebooted, pulled and reinserted the blank drive a couple of times, skimmed the official OS6 documentation; I can't figure out how to reintegrate the new drive into the exsisting array.
This GUI currently shows this:
School me please! This should not be this counterintuitive. . . :nashammer:
mdgm wrote: So are you using X-RAID2 or not?
X-RAID2 = YES
mdgm wrote: What options do you see if you click on the data volume?
If I click Settings I get this:
mdgm wrote: What happens of you click on the drive that is not highlighted in blue?
I get something that looks like a "Thought Bubble" (like you would see in the Sunday Comics) with a bunch of Hard Drive statistics, like - ID, Model Number, Serial number, Firmware version, RPM, Sectors, etc, etc. I can't screenshot it because it disappears as soon as I move my mouse away from the image 🙂
It just keeps getting curiouser and curiouser. . .
So, since no one chimed in with any suggestions, I just kept playing with stuff. Here is what I've done so far:
1.) I shut the ReadyNAS Pro down
2.) I removed the fresh drive that was in Bay 4 which was supposed to be the replacement for the pseudo-failed drive
a. So at this point all that is in the box are drives in Bays 1, 2, and 3
3.) I moved the drive in Bay 3 to Bay 4, 2 to Bay 3 and 1 to Bay 2, essencially shifting the entire array down one drive slot (so now Bay 1 is empty)
4.) I powered the ReadyNAS back on. Much to my surprise, it actually booted up just fine.
5.) I waited about 10 minutes, logged into the GUI, checked the logs, and everything seems okay
6.) I reinserted the new drive that did not get incorporated into the array previously, this time into Bay 5. Still no magic happening. . . :-?
7.) Now I'm getting annoyed. Since I was about 15 minutes away from doing a sledgehammer reset on the damn thing (Factory Default and start over) anyway, I decided to take the original drive (that started out in Bay 4), connect it to a computer, erase the three partitions that the ReadyNAS OS created on it, and shoved THAT drive into Bay 6.
8.) Guess what? See the screenshot below. . .
So, in case anyone is wondering what my plans are now:
9.) Once the array finishes Re-syncing (about 6 hours from now) I plan to re-remove the screwy drive currently in Bay 5 and run some basic forensics on it to see if I can figure out why the ReadyNAS is treating it like a "Red-Headed Step Child"
10.) Then I intend on DBAN'ing it (http://www.dban.org/), formatting it, and surface testing it
11.) If all that is a success, I'll clear any remaining partitions and put it back into Bay 5 and see what happens. . .
Oh what fun it is to. . . :wiggle:
So, since no one chimed in with any suggestions, I just kept playing with stuff. Here is what I've done so far:
1.) I shut the ReadyNAS Pro down
2.) I removed the fresh drive that was in Bay 4 which was supposed to be the replacement for the pseudo-failed drive
a. So at this point all that is in the box are drives in Bays 1, 2, and 3
3.) I moved the drive in Bay 3 to Bay 4, 2 to Bay 3 and 1 to Bay 2, essencially shifting the entire array down one drive slot (so now Bay 1 is empty)
4.) I powered the ReadyNAS back on. Much to my surprise, it actually booted up just fine.
5.) I waited about 10 minutes, logged into the GUI, checked the logs, and everything seems okay
6.) I reinserted the new drive that did not get incorporated into the array previously, this time into Bay 5. Still no magic happening. . . :-?
7.) Now I'm getting annoyed. Since I was about 15 minutes away from doing a sledgehammer reset on the damn thing (Factory Default and start over) anyway, I decided to take the original drive (that started out in Bay 4), connect it to a computer, erase the three partitions that the ReadyNAS OS created on it, and shoved THAT drive into Bay 6.
8.) Guess what? See the screenshot below. . .
So, in case anyone is wondering what my plans are now:
9.) Once the array finishes Re-syncing (about 6 hours from now) I plan to re-remove the screwy drive currently in Bay 5 and run some basic forensics on it to see if I can figure out why the ReadyNAS is treating it like a "Red-Headed Step Child"
10.) Then I intend on DBAN'ing it (http://www.dban.org/), formatting it, and surface testing it
11.) If all that is a success, I'll clear any remaining partitions and put it back into Bay 5 and see what happens. . .
Oh what fun it is to. . . :wiggle:
So for any of you who have not been following along, I've been putting a brand new (excess inventory) ReadyNAS Pro through its paces with a fresh install of OS6 version 6.0.8.
About eight posts ago I ended with a list of eleven steps that I either had done or were planning on doing. The conclusion of that little experiment was simple. . . It turned out to be a BAD HARD DRIVE. Even though all the short/quick diagnostic tests I attempted said that the drive was okay the ReadyNAS knew otherwise. Once I resorted to a FULL Diagnostic / Surface Scan the problems eventually came to light. So I set that Hard Drive aside to send back to Western Digital for warranty replacement.
Fortunately I had a spare hard drive of the same brand and model that I could use in place of the newly discovered BAD drive. So, for the past week I have been pushing and pulling Terabytes and Terabytes worth of data to and from this ReadyNAS Pro using different protocols and methods trying to test every possible scenario before putting OS6 into a production environment. I also experimented with some really crazy drive swapping scenarios that I would NEVER attempt in a production environment. All in all, the vast majority of testing went pretty well, until. . . I get this E-Mail out of the blue from the NAS under test:
"Detected increasing ATA error count: [24636] on disk 5 [WDC WD2002FYPS-02W3B0, WD-WCAVY5956816] 2 times in the past 30 days. This condition often indicates an impending failure. Please be prepared to replace this disk to maintain data redundancy."
I'm like "What the Hell?" The distressing thing about the timing of this message is it took place about 20 minutes into a volume expansion experiment that I was performing. You see, all the testing that I was doing previously was with four 2TB Drives (Bays 2,3,4,5 populated. 1 and 6 empty). So I filled the volume with test data to ensure that I would get an alert stating that it was getting full. Once I was satisfied that the volume capacity alerts were working properly, I thought it would be a good time to do a Volume Expansion test. I inserted a fresh drive into Bay 6. Along with the message above I received another that said:
"Disk in channel '5' (Internal) changed state from ONLINE to FAILED."
So I removed the drive from Bay 5 and started a failure diagnosis on it. Here comes the weird part - After doing a FULL diagnostic on the drive with two different tools (to include a full SURFACE SCAN) both diagnostics reported zero errors or problems of any kind. Again I say: What the hell? So I wiped the first and last million sectors of the drive in preparation for reintroduction to the array.
Since performing a full diagnostic on a 2TB drive takes about 5 to 6 hours, the drive that I popped into Bay 6 had finished the integration process. Unfortunately, since the drive that was in Bay 5 failed during a Volume Expansion, the Array as a whole was still considered "Degraded". So, in an attempt to "fool" the ReadyNAS into forgetting about the drive that failed from Bay 5, I shut the unit down and moved the new drive that was in Bay 6 to Bay 5, and powered it back on, waited 15 or 20 minutes and reinserted the alleged "BAD" drive that was originally in Bay 5 into Bay 6. The insertion was detected and this is what I saw on the GUI:
Which leads me to three questions, which are:
1.) Why do you suppose the ReadyNAS thinks that this drive has issues when two different Hard Drive diagnostic programs say otherwise?
2.) Does OS6 have some sort Hard Drive Database that remembers every Hard Drive that has ever been in it?
3.) If number 2 above = YES, is there some way to clear that data so that the ReadyNAS reassess the condition of a particular drive from scratch sometime in the future?
As always. . . Thanks in advance,
About eight posts ago I ended with a list of eleven steps that I either had done or were planning on doing. The conclusion of that little experiment was simple. . . It turned out to be a BAD HARD DRIVE. Even though all the short/quick diagnostic tests I attempted said that the drive was okay the ReadyNAS knew otherwise. Once I resorted to a FULL Diagnostic / Surface Scan the problems eventually came to light. So I set that Hard Drive aside to send back to Western Digital for warranty replacement.
Fortunately I had a spare hard drive of the same brand and model that I could use in place of the newly discovered BAD drive. So, for the past week I have been pushing and pulling Terabytes and Terabytes worth of data to and from this ReadyNAS Pro using different protocols and methods trying to test every possible scenario before putting OS6 into a production environment. I also experimented with some really crazy drive swapping scenarios that I would NEVER attempt in a production environment. All in all, the vast majority of testing went pretty well, until. . . I get this E-Mail out of the blue from the NAS under test:
"Detected increasing ATA error count: [24636] on disk 5 [WDC WD2002FYPS-02W3B0, WD-WCAVY5956816] 2 times in the past 30 days. This condition often indicates an impending failure. Please be prepared to replace this disk to maintain data redundancy."
I'm like "What the Hell?" The distressing thing about the timing of this message is it took place about 20 minutes into a volume expansion experiment that I was performing. You see, all the testing that I was doing previously was with four 2TB Drives (Bays 2,3,4,5 populated. 1 and 6 empty). So I filled the volume with test data to ensure that I would get an alert stating that it was getting full. Once I was satisfied that the volume capacity alerts were working properly, I thought it would be a good time to do a Volume Expansion test. I inserted a fresh drive into Bay 6. Along with the message above I received another that said:
"Disk in channel '5' (Internal) changed state from ONLINE to FAILED."
So I removed the drive from Bay 5 and started a failure diagnosis on it. Here comes the weird part - After doing a FULL diagnostic on the drive with two different tools (to include a full SURFACE SCAN) both diagnostics reported zero errors or problems of any kind. Again I say: What the hell? So I wiped the first and last million sectors of the drive in preparation for reintroduction to the array.
Since performing a full diagnostic on a 2TB drive takes about 5 to 6 hours, the drive that I popped into Bay 6 had finished the integration process. Unfortunately, since the drive that was in Bay 5 failed during a Volume Expansion, the Array as a whole was still considered "Degraded". So, in an attempt to "fool" the ReadyNAS into forgetting about the drive that failed from Bay 5, I shut the unit down and moved the new drive that was in Bay 6 to Bay 5, and powered it back on, waited 15 or 20 minutes and reinserted the alleged "BAD" drive that was originally in Bay 5 into Bay 6. The insertion was detected and this is what I saw on the GUI:
Which leads me to three questions, which are:
1.) Why do you suppose the ReadyNAS thinks that this drive has issues when two different Hard Drive diagnostic programs say otherwise?
2.) Does OS6 have some sort Hard Drive Database that remembers every Hard Drive that has ever been in it?
3.) If number 2 above = YES, is there some way to clear that data so that the ReadyNAS reassess the condition of a particular drive from scratch sometime in the future?
As always. . . Thanks in advance,
mdgm wrote: Did you check the SMART stats for ATA errors using LifeGuard Diagnostics or another tool?
YES. Both tools reported that the drive was in PERFECT Health.
Check out this screen shot:
The Array rebuild is almost 2/3rds completed and has been running for several hours now. Wouldn't you suppose that if this drive was REALLY BAD, that the ATA errors would have incremented (even slightly) by now? Upon drive insertion the ATA Error count was 24638. I just checked it again three plus hours into the rebuild and it is still at 24638. Doing an Array rebuild is a VERY Disk I/O intensive operation, is it not? Wouldn't you have thought the errors would have increased by now?
:werd:
whelp, after several tries and 3x readynas ultra pro 2; ReadyNAS OS6.0.x is NOT stable for production. i knew it was a bad ideal to try to run it in an production environment. but particularly, after trying multiple times running iscsi lun with mpio connections from multiple hosts, initially after x amount of days the lun would literally drop the filesystem. this is an hyper-v 3.x environment with the lun as an CSV; this happened three times, with the same result.
fortunately i learned a trick to be able to remount the lun as a readable filesystem, and all the data is intact, but any further use would render the filesystem useless again.
having this occur with running VM's is simply not worth the risk. i think there is something inherently wrong with the way iscsi is handle in readynasos6 compared to raidiator 4x for the readynas models. so just a warning to others, please be advise about using iscsi lun on an ultra pro 2;if you have ran into this issue and trying to get your data back from the raw filesystem, message me and i'll help you through it
fortunately i learned a trick to be able to remount the lun as a readable filesystem, and all the data is intact, but any further use would render the filesystem useless again.
having this occur with running VM's is simply not worth the risk. i think there is something inherently wrong with the way iscsi is handle in readynasos6 compared to raidiator 4x for the readynas models. so just a warning to others, please be advise about using iscsi lun on an ultra pro 2;if you have ran into this issue and trying to get your data back from the raw filesystem, message me and i'll help you through it
