ReadyNAS Pioneer Pro corrupt flash no boot menu

No thoughts about DIMM placement.

But it is a very interesting thread, and please keep updating us on your progress.

The RSYNC of the logical unit finished.

I downloaded all of the logs to have a copy of the logs after a rebuild so that I have a clean copy of the logs.  When the NAS does not boot, the OS is not up so I suspect there would be no clues added to the logs from a non-boot event.  Regardless, a clean copy of the logs may come in handy at a later date.

The NAS was setup via READYCloud.  That was all normal. 

Test #1:

I did a reboot via the HTTP GUI and it booted normaly.

Test #2:

I did a shutdown via the HTTP GUI and then waited 5 minutes.  Then powered the unit on via the power button on the front of the NAS.  The NAS booted normally.

Test #3:

I did a shutdown via the HTTP GUI and waited 5 minutes.  Then powered off the power supply switch at the rear of the NAS.  Waited 15 minutes and applied power to the NAS by the rear power button.  The NAS booted normally.  I then did a reboot via the HTTP GUI and it booted normally.

Although, these are all positve results, I doubt the issue is resolved.

Test #4: 

I have shut the NAS down and disconnected power from the NAS.  This gives that new 3V lithium battery something to do.  I will wait several hours and then plug the NAS back in and see if it boots. 

Note, I have tried these tests with and without a powerstrip and the problem happens regardless of if the powerstrip is there or not.  I have also tried differenct power outlets in the house.  I have not tried a different powercord yet, but I just added that (for inclusiveness) as another long shot item.  I am using the powercord that came with the NAS. 

I will wait for a boot to fail before moving the DIMM.  Does anyone know where the manual is for this motherboard?  After a quick search, I can not find the manual online.  It is a FLAME6 Ver 1.1 motherboard.  I want to check the manual to see if it says anything about if you only have one DIMM that the DIMM needs to be in DIMM slot 1.  Here is a picture of the motherboard model:

StephenB, I agree that this is an interesting issue.  Thanks to you and everyone else for following along and offering any advice or insights.

I waited 7 hours and plugged the NAS powercord back in.  I powered on the NAS via the power supply switch on the rear of the NAS.  The NAS booted fine.  I shut the NAS down via the HTTP GUI.  I waited 5 minutes and powered the NAS on via the power button on the front of the NAS.  The NAS booted fine.  Although I am an optimist, based on the history of this NAS, I am still not convinced the issue is resolved.  I will continue testing tomorrow after the NAS sits longer in the powered off state; this time with the powercord connected.

I plugged the powercord into the NAS the next morning.  The NAS booted fine.  According to the logs, I have powered the NAS on and off 5 times since then.  I will contintue testing.  If the NAS makes it to 15 times of booting after applying power, I will put data on the system.  Until then, the jury is still out.

I booted the NAS two more times on different days and it booted each time.  With things looking good on the power-on testing, I decided to do some more of the setup for the NAS:

     1.  Added users.

     2.  Set permissions for the built-in shares.

     3.  Created new shares.

     4.  Set pemissions of new shares.

     5.  Enabled RSYNC for specific shares.

     6.  Created backup jobs to get data on to the NAS.

     7.  Ran backup jobs and got all the data off of the NV+ NAS and on the Pioneer Pro NAS.

At this point, I decided to work on replacing disk 3.  For a refresher, the current disk 3 shows this in diags:

     Disk 3 has 2 Reallocated Sectors
     Disk 3 has 2 Reallocation Events
     Disk 3 has 1 Command Timeouts

That is not catastrophic (especially since I know that all the failed power-on attempts is what gave the disk drives the trauma they experienced), but my intent is to retire the NV+ from my backups and normal use.  I used the HTTP GUI on the NV+ to run the S.M.A.R.T. test on all drives in the NV+.  I picked disk 2 from the NV+ to be the new disk 3 in the Pioneer Pro.  Here is the S.M.A.R.T. info for the NV+'s disk 2:

This drive does not have any issues with the main 5 items detected by S.M.A.R.T. that typically show a drive has potential to fail.  Those items are:

     1.  Reallocated Sector Count

     2.  Reported Uncorrected Errors

     3.  Command Timeout

     4.  Current Pending Sector Count

     5.  Offline Uncorrectable

You can read more about that at this website where someone analyzed the S.M.A.R.T. data from 40,000 disk drives. 

Disk 3 was removed from the Pioneer Pro while the unit was powered on.  The system noticed the drive being removed and gave the following message in the logs of the HTTP GUI:

     Volume: Volume data health changed from Redundant to Degraded

     Disk: Disk Model: ST31000528AS was removed from Channel 3 of the head unit.

The LED display on the front of the NAS said "data DEGRADED".

Disk 2 was removed from the NV+ NAS while that NAS was powered on.  The LCD on the NV+ then said "Disk 2 Failed".  The logs for the NV+ showed:

     Disk remove event occurred on SATA channel 2.

     Disk fail event occurred on SATA channel 2. If the failed disk is used in a RAID level 1, 5, or X-RAID volume, please note that volume is now unprotected, and an additional disk failure may render that volume dead.

The disk drive carriers are different in the two NASs so those were swapped.  The NV+ disk 2 was placed in the Pioneer Pro disk 3 drive slot while the NAS was powered on.  The Pioneer Pro saw the new disk and said this in the logs:

     Disk: Disk Model: ST31000528AS was added to Channel 3 of the head unit.

     Volume: Resyncing started for Volume data.

That NAS is currently resyncing. 

The Pioneer Pro original disk 3 was placed in the NV+ disk slot 2 while the NAS was powered on.  That showed this in the logs of the NV+:

     Disk add event occurred on SATA channel 2.

     Disk initialization successfully finished.

     RAID sync started on volume C.

Both NASs are currently syncing.  I will let that finish and then run diags on the Pioneer Pro to confirm that the new disk 3 shows no issues. 

The syncing finished on both NASs.  Everything is as expected.  At the completion of the sycning, the Pioneer Pro showed this in the logs:

     Volume: Volume data is resynced.

     Volume: Volume data health changed from Degraded to Redundant.

I rebooted the Pioneer Pro to make sure that it would survive a reboot with the replacement disk drive and that worked fine.  I also tried another poweroff and poweron test with the Pioneer Pro and that worked as well.  I ran diags (via RAIDar) on the Pioneer Pro and all the messages for disk 3 are now gone since the disk was replaced:

     Successfully completed diagnostics
      System
      Disk 4 has 1 Reallocated Sectors
      Disk 4 has 1 Reallocation Events

Although disk 4 only has 1 reallocated sector, I will repeat the drive swap process again to replace disk 4 in the Pioneer Pro by swapping another disk drive between the NV+ and the Pioneer Pro.  Then there will be no visible hardware errors with the Pioneer Pro. 

I swapped disk 4 in the Pioneer Pro with disk 3 in the NV+.  Everything completed like normal.  I ran diags on the Pioneer Pro and all looks good:

      Successfully completed diagnostics
      System
         No errors found.
      Logs
         No errors found

I will do a couple more poweron tests tomorrow for completeness. 

Every ReadyNAS Pro BE or Pro6 I have purchased (all used) have had the DIMM in slot 2 except for when there were already two.  I suspect it's just because it's easier to access that slot.  The motherboard does not appear to use dual-channel RAM, so which slot is used first does not matter.

When a unit is off for a very long time, all capacitors will fully discharge.  That might cause a problem with the first boot, but I'm not sure why subsequent ones would have an issue.  The flash should be able to survive a voltage fluxuation at power-on.

One of the used machines I bought had no 3V battery at all, likely having been removed because it was put in storage.  Other than the date/time coming up wrong, it booted just fine.  Apparently, all the default CMOS settings are just fine for booting.  But maybe a weak battery could cause it to only partially retain the CMOS settings, resulting in it trying to boot from corrupt settings. Whether it would simply reload the defaults if the CMOS checksum failed or fail to boot, I don't know.

The motherboard does not seem to do a lot of diagnistics in the boot process.  It will boot just fine with a bad or missing fan.  It can just lock up on bad RAM. 

Sandshark, thanks for the information that other ReadyNAS Pro's have their only DIMM in DIMM slot 2.  DIMM slot 2 is definitely more accessible than DIMM slot 1 (which is close to a lot of cables).

All - I checked the logs and the NAS has been booted 16 times in a row with no issues.  The last NAS power off was 5 days ago so I decided to try the NAS this morning.  I powered the NAS on this morning and the LED just said "READYNAS", the USB activity light by the backup button stayed lit solid the whole time, the link light on the rear of the NAS for the NIC was not lit, and the row of tick-marks below the word READYNAS kept going from left to right like the picture below for 45 minutes. 

As you can see from the previous postings in this thread, this has happened before and is the periodic non-boot issue that I am trying to fix.  It looks like replacing the 3V battery did not resolve the issue.  When the system is in this state, it can not be accessed by RAIDar.  I powered the NAS off, waited 10 seconds, powered the NAS on, and it booted fine.  Why it boots sometimes and not others, remains a mystery.

The "ReadyNAS" on the display is the default.  All you need is 5V from the power supply for that to be there.  It certainly takes some action to start the progress bar moving, but it likely doesn't take any to sustain it.  You are getting no clues on the VGA as to the status at that point?

No lights on the NIC is interesting.  Do you have WOL enabled?  If not, enable it when you do get it to boot and then verify ou have NIC activity when the unit is off.  The NIC uses the standby power bus in order for WOL to work, but I don't know what else also uses it.

It's possible it's a power issue.  You can use a standard ATX supply externally to check and see if that makes a difference.  If it does, the supply is readily available and you will find messages here about that.  Not enough evidence to say it's power for sure, so I wouldn't recommend buying a replacement supply until you've tried the external method to see if it will make a difference unless ou just don't have a supply to use for that.  Power supplies on older units are starting to go, as attested by messages here about that.  I bought one that I was pretty sure only needed a replacement supply, and I was right.  And one of my others had the supply fail.

All - I decided to move the memory from DIMM 2's slot to DIMM 1's slot.  I did this with a grounding strap and with the NAS unplugged.  The NAS booted fine with the memory in DIMM 1's slot.  I will continue to do power-on tests and reboots to see if moving the memory helps or not.  If there is a race condition, I would expect the motherboard to check DIMM 1's slot before DIMM 2's slot. 

Sandshark - the BIOS settings were reset to factory default recently.  If the system does not boot again, I will check WOL and enable it if it is disabled.  I like to only change one variable at a time to identify the root cause of the issue.  I have several new and used spare power supples from/for PCs.  I already tried another Seasonic power supply with the same model number of the existing power supply and that yielded the same results.  I get what you are saying though and I will keep this in mind as testing continues. Thanks for the input.

With a PC style power supply added to the testing list, I am down to three things to try. 

I powered on the NAS today and it did not boot.  It just said READYNAS and the row of neverending tickmarks.  Therefore, moving the only DIMM to DIMM slot 1 did not resolve the issue.  Note, it has only been 3 days since my last post, so in this case, it did not take too long to reproduce the issue.  For grins, I swapped the powercord with another powercord that has equal current ratings.  I just wanted to get that off the list of things to try.  I doubt that will make a difference.

After several power-on tests, I was able to prove that the issue still happens with the replacement power cord.  This time it was the infinite rows of tick marks issue (as previous described).  The next chance I get I will put a monitor and keyboard on it again to look for any new clues during a failed boot sequence.  I will check the WOL setting then as well.  I am also going to open it up again and look around more inside.  I am down to two things to try to resolve this.

Hi Yoda1492,

Thanks for updating, please keep us posted as well.

Regards,

Just a small update.  I had at least 10 good boots in a row over the last several days.  I used the NAS, wrote data to it at 111MB/sec, and updated the firmware to the new verson that came out.  All of that was fine day after day.  Today, the NAS does bot and does the infinite row of tick marks.  I tried to boot it at least 10 times.  I will put a monitor and keyboard on it this weekend and open it up to look for more clues.

Thanks for the update. Glad this is now resolved. You may wish to mark your post as the solution.