ReadyNAS Pioneer Pro corrupt flash no boot menu
HELP! When I power on my ReadyNAS Pioneer Pro, the LED screen just says ReadyNAS (does not show the drive LEDS) and the yellow activity light (on the front of the system above the USB port) lights and stays lit (there is no USB device plugged into this port). I do not care about any of the data on the drives. I just want to get the system working again. I have a VGA connector on the 16 pin port on the motherboard so I can see what happens when the system powers on. I can also access and change the BIOS with a USB keyboard. The BIOS shows my disk drives. I can not (repeat not) get to the Boot Menu. That used to work, but currently does not. When I power the unit on with the reset button pressed in and no (repeat no) hard drives in the system, I see "MBR H" on the monitor. Based on searches on the Internet, it looks like that is related to the internal flash needing to be reset. I created a USB boot recovery for x86, with a USB stick that has an LED on it to show when the stick is being accessed. That runs to completion and powers the NAS off. With only two Seagate 1TB drives (they are on the hardware compatibility list) in the system, this created the drives in a logical unit (XRAID) and the system was able to be powered on and off many times. When the unit is powered off, I remove the power cord to move the system to it's final place and then when I power the unit on it goes back to just saying READYNAS on the LED with the USB activity light lit.
This unit was powered off and disconnected from power for three months. The hardware health all looks good in the BIOS (to include the 3V battery). When I first plugged in the unit after it was unplugged for all that time, the LED said ERR: Could not find internal fla (which I am guessing is flash image).
I see there is a TFTP Boot option to help recover systems. I have not tried that since holding the power button in on the front of the unit as mentioned in the README file just powers my unit off.
I do not care if the system is restored with OS 4 or OS 6, but I need the unit to be able to boot consistantly and after the power cord is removed from the system (as power outages will eventually happen). My drives are good as I have tested them in a SATA to USB adapter with Seagate's SeaTools. I also tried other drives (on the hardware compatibility list) and I get the same results.
The process of the system being able to create the logical unit with the same two drives right after a USB recovery is not always repeatable. After powering off the NAS after a USB recovery and then removing the USB stick, sometimes when the unit is powered back on, it just says ReadyNAS on the LED screen with the activity light lit again.
With just one disk drive in the system (and no USB sticks), the system goes through it's POST test, picture included below, and then the monitor just shows a blinking cursor. Please advise if anyone has ideas on this issue.
Accepted Solutions
Hello. I found some time to work on this NAS. The issue may now be permanently resolved. I moved the NAS to a place where I could get a monitor and keyboard on it. I created a video of the boot sequence and took pictures of all the BIOS settings. While I was in the BIOS, I did enable Wake-On-LAN as previously requested. I also turned off the NAS from coming back on after a power hit; that was annoying as the power would go off and then on and then off again and that is bad for electronics and data volumes. I will get an UPS sometime this year.
First, I will start with some information on those tick marks that were going across the NAS LED infinitely when the NAS would not boot. As shown in the picture below, that is from the Kernel loading and the initrd.gz loading:
I kept going back to what someone said about it being a hardware problem. Armed with that and the clue that the NAS was shipped as part of a move and never worked consistantly after it was plugged in after the move, I looked more closely at the hardware. I took off both sides of the NAS and when I was inspecting all the cables, I saw this:
At first glance, it looks mostly ok. However, that top connector is supposed to go all the way through the cutout hole in the metal and fully into the circuit board on the other side of the metal. On the other side of the metal, you can see that the connection is not fully made:
Pulling the connector out just a little bit shows that it has no chance of making this connection without hitting the metal:
The NAS was then taken even further apart. There are 3 screws on each side of the NAS that holds the front panel of the NAS in place. All of those screws were taken out and that provided access to the cicrucit board behind the front panel that this cable plugs into. The circuit board looks like this:
The board has three screws holding it into place. However, the board has a lot of play with regards to how it lines up with the cutout hole in the metal. The connector was pushed through the hole and fully connected it to the metal pins. Then the screws were put back in for the circuit board and now the connector makes a full connection with these pins. The NAS was reassembled and it has booted ten times in a row. The power was disconnected for a few hours and the NAS booted fine after power was reapplied. I can not explain why the NAS would boot some times and not boot other times when the cable was partially connected. The NAS will be tested over the next several days before declaring this a permanent fix.
All Replies
Hello,
Thank you for responding so quickly! I really appreciate that. I have a ReadyNAS Pioneer Pro and it has an Intel processor and is the x86 architecture. Thank you for the link for the USB Recovery Tool process. That is the process that I used to create the USB stick. The instructions in that link say: "Units enabled with LCD will indicate that it has entered USB Recovery mode". That never happens for my unit. My LCD never says that.
I think it would help to show what I see on the VGA monitor when doing a USB recovery. It says:
Loading kernel ...
Loading initrd.gz .... ready
and then the USB stick LED flashes for a while and then the NAS powers off. I wish it would say Loading RAIDiator-x86-4.2.28 or something about the firmware being copied to the internal flash, but it never says that.
Here is a picture of the files on the USB Recovery stick:
If you see anything wrong or missing with the files on the USB Recovery stick, please advise. I am thinking it would be worth it to do this USB recovery with no disk drives in the NAS. The point of this process is to copy data from the USB stick to the internal flash inside of the NAS. I do not need any disk drives for that. Then I can test the factory reset option via the reset button to see if there are any changes getting that to work. It does not hurt to remake the USB Recovery stick so I will do that if the test with no drives does not help. It is Halloween so I will do this tomorrow and report the findings. FYI - I previously mentioned the TFTP Boot information as I saw on this forum where people had fixed this same issue (MBR H on the monitor when holding in the reset button) by following an "undocumented procedure" after talking to Netgear Support. I am not sure what that process is, but I will follow any path that leads to a working NAS. Thanks again for contacting me and please keep in touch until the issue is resolved.
Hello StephenB / FramerV,
Welcome to the Fix-My-NAS party. I will start with a quick update on the test that I was doing after my last post and then I will answer StephenB's question.
I tried the USB Recovery Tool with no disk drives in the NAS and got the same results (the LCD just says READYNAS, the system goes through the recovery process and shuts off the NAS, but I never could get to the Boot Menu via the reset button). I also reset the BIOS setitngs via the "Load optimal settings" in the BIOS of the NAS, but that did not help either.
Now on to StephenB's question. The creation date on the files on my USB Recovery stick are from January 2016. I made that stick a long time ago and I can not guarantee where I got all the files from for that USB stick. With the link from you and FramerV, I decided to just recreate the USB Recovery stick. For your information, those instructions may need to be tweaked. The instructions say to:
"Extract the firmware file and release notes from the zip file to the same folder you extracted the Recovery tool."
That is easy to do. However, when you run the "usbrecovery.exe" tool to create the USB stick with both the release notes HTML file and the firmware in the same directory, you can not select the firmware to load into the USB Recovery tool. Here is a picture showing what I am talking about:
See how the HTML Release Notes file is automatically selected and in the screen grab I have the arrow selected showing that there is no other file to choose from in the list. The "Create" button is still available but I never tried that as it is asking you to choose the firmware image and not the Release Notes HTML file. Who knows, maybe that works as well. I just deleted the HTML file from the directory, re-ran the tool (it automatically selects the firmware file since the HTML file is not there), and re-made the USB Recovery stick. Note, I decided to use a different USB stick then I was using before so that I could maintain the old one as well. People are right when they say these NAS's are very picky on the type of USB stick that will work with this process. I went through several before I found one that the NAS would recognize as a boot device. It seems the older the USB stick and the smaller in storage space size the better. The instructions say to use a USB stick between 256MB and 32GB. I used a 512MB imation USB stick with an activity light and it worked fine.
Anyway, good news! I was able to change the state of the NAS. I did the USB Recovery with the new USB stick and no drives in the NAS and that completed (very quickly actually) and shut the NAS off. I then held in the reset button while powering on the NAS and was able to get to the Boot Menu! Note, this means the USB Recovery process worked and restored the firmware onto the internal flash inside the NAS. I went through the Factory Reset process and it worked flawlessly. At the end it said "Err: no drives" on the LCD which was true. I shut it all down, put in 4 of the 6 1TB Seagate disk drives and did another Factory Reset. It went to completion and the NAS booted fine. It then started resyncing the volume. That is taking awhile. Tomorrow after the resync finishes, I will do some shutdown/startup tests before moving the NAS to where I can get an ethernet cable to it. Those tests will include disconnecting the NAS from a power source for some time and then seeing if the entire process starts over where it just says READYNAS on the LCD screen when powered on. I am hoping those days are behind me.
Note this NAS has been dead like this for 5 years and 5 months. The key to fixing it was to get the motherboard to VGA adapter that allows you to see things on the monitor. I will post an update after further testing. Thanks for listening.
Hello mdgm,
Welcome to the Fix-My-NAS party! I will start with an update on the testing and then address mdgm's comments.
I let the NAS resync overnight. That completed as expected. I shut the NAS down via the power button on the front (pressing it multiple times to do a graceful shutdown), waited a little bit, powered it back on, and everything was normal. I shut the NAS down gracefully via the power button on the front of the NAS, turned off the power switch on the back of the NAS, and disconnected the NAS from power. I let the NAS sit for 12 hours, reconnected the power, powered the NAS on, and everything was perfect.
I then shut the NAS down gracefully via the power button on the front of the NAS, put the side panel on the NAS, moved the NAS to it's final destination, connected it to my LAN and a powerstrip, and powered on the NAS. The NAS just said READYNAS and sat there for a long time (greater than 30 minutes). I should have used RAIDar to try to connect to it, but I powered it off instead. The second time I powered it on, it said READYNAS, the USB activity light by the backup button stayed lit solid the whole time, and the row of tick-marks below the word READYNAS kept going from left to right like the picture below for over 30 minutes:
Over time, I have been able to get the NAS restored with various drive configurations. Later, after the NAS loses power from a power hit or being moved to a new location, the NAS seems to forget how to boot and just says READYNAS on the LCD. This has happened at least 10 times over the last 5 years.
I downloaded RAIDar 4.3.8 for windows from here: https://www.readynas.com/download/RAIDar/RAIDar_windows_4_3_8.exe
and installed it (a pop-up said it had to be installed as an admin so I installed it as an admin). It did not see the NAS. Looking at the NAS ethernet cable there was no link light on the port on the switch for the NAS or on the LED by the RJ-45 connector plugged into the back of the NAS. I let it sit for 30 minutes while I typed up this email and it just kept doing the same thing. Can someone please explain what the NAS is doing when these tick marks are going from left to right forever? I suppose if I connected the monitor I would get a clue to that. I powered the NAS off, waited a little bit and powered it back on. It booted like normal. The link light was lit for the NIC and I was able to connect to the system via RAIDar.
Notice how the disk status section is blank? That is odd for a system with 4 disk drives in it? The LCD shows 4 disk drives, the C drive, and the free space and appears normal.
I run the setup wizard in RAIDar and am greeted by a blank screen in Firefox 49.0.2.
I did add the exception to connect to the page. This used to happen with Windows 7 as well and I would just refresh the page and it would fill in the page properly. However, that is not happening with Windows 10 Home 64bit. I tried using Microsoft Edge as well and I get the same results. The system gets it's IP correctly via DHCP from my router which is set to statically provide the same IP to this MAC address. I can ping the IP of the NAS, but that is expected since the GUI connected and accepted the default login information.
Now I will address Mdgm's comments, which hopefully leads to more clues to solving the root cause of this issue. First, thanks for responding! I agree that you do not need the cable to do the USB Recovery. However, the cable enabled me to see information on the monitor (drives seen by BIOS, NAS boot order, power settings, etc.) that are needed to get the full picture for troubleshooting. But you right, in a perfect world, it is not needed to do a USB Recovery. I would recommend to remove all hard drives though as when I first connected the VGA adapter I noticed that one of the disk drives was not recognized by the NAS. I just removed that one from the rest of this testing. You asked an interesting question about the NAS showing up with the correct model number in RAIDar. That section is blank in RAIDar.
What does that mean? Thanks for the reminder that 4.2.30 is the latest firmware. I was going to use FrontView to upgrade the firmware to prove the upgrade process worked as well. Alas, FrontView is not displaying correctly so I was not able to do that. I could always do a USB Recovery with the newer firmware, but I will wait to hear from someone before doing that.
This is a great NAS model, which is why I want to get it fixed. The reason I did not ask for help before is because I have 4 Netgear NASs. I have two of them mirrored to two others as I hope to never lose data. But with this one dead, one NAS stands tall by itself living with pride and mocking NAS redundancy. If I could not do the factory reset on this one, I was going to suggest formatting the drives in the READYNAS ULTRA 6 NAS as that is pretty close to the Pioneer Pro and then moving the drives over. I was not sure if the NAS controller kept the metadata or not so I was not sure if that would work or not. Anyway, the firmware was flashed on the Pro so there is no need to do that.
Here is an interesting tidbit. The model number of the Ultra 6 does show up in RAIDar and that NAS is running firmware version 4.2.30
I can connect to FrontView with no problems for the Ultra 6 NAS. So that tells me my brower supports displaying the data and it is not a JAVA thing or something like that. The other NAS is still a blank screen. Any help from anyone would be appreciated. Like I said, welcome to the Fix-My-NAS party. Whoever can fix it permanently, wins bragging rights!
All,
After letting the NAS sit there in Tech Mode for 7.5 hours, I decided to shut the NAS down. Then I used the reset button on the back of the NAS and did an OS Re-install. That said "Upgrading Firmware" and went to completion. After that, the NAS booted up to where the drives are all shown on the LCD, the free space is shown, and the IP address of the NAS is shown. I can ping the NAS, but the admin web page shows up blank in Firefox and in Edge. RAIDar can connect to the NAS, but the Model number is blank (which I think is the most important clue to resolving this as I have seen that fixed in other posts via mdgm). The drives also do not show up in the picure below:
When I try to telnet to the NAS from my PC, it says: "Could not open connection to the host, on port 23: Connect failed"
I am not sure if I need to use Putty or if I should be able to telnet to the system after a Factory Reset and OS Re-Install. The tests I have not done yet are:
1. Install 4.2.30 firmware via USB. The release notes for that show these two fixes:
A. Fixed Badlock (CVE-2016-2118)
B. Various security SSL fixes.
I am not sure what the first one is or if it applies to my situation? Please advise if you think that will help my situation.
2. I could go through the whole process of a Factory Reset and OS Re-Install with just one disk drive in the NAS. That rules out more things and reduces the system to a smaller configuration.
3. I could put the monitor and keyboard on the NAS now that the NAS is connected to a network cable. This would help to look for more clues.
4. I can try the Tech Mode option again now that I did the OS Re-Install.
5. I have tried the memory and disk tests via the Boot Menu in the past, but I could do those again.
As to if it is a hardware problem or not, I could see that if it was the 3V battery being dead inside the NAS, but that gave good voltage in the BIOS. It seems more like a logic problem to me, but you guys are the experts. What I would like to know is the following:
1. What exactly is the common side effect of doing a USB Recovery and how is that fixed?
2. If the above is different than setting the model number and serial number on the NAS, where does that need to be set? I read on the posting below that this info is contained somewhere in the flash. See this posting:
If this information can not be posted, then please PM the information to me. I am able to get the NAS into various states at various times. I would rather have the information handy should I be able to access the system from the command line. I am VERY comfortable editting files by hand. Any help would be appreciated. Thanks.
Hello,
The RSYNC finished, I ran the setup via ReadyCLOUD and completed all of that. I did a reboot and it rebooted fine. I like all the extra feedback on the LCD in OS 6.X vs OS 4.X. I thought I would add a disk drive to the system as the system only has 4 of the 6 drives in it and I have a lot of spare drives for the NAS. I put in the 5th disk drive with ReadyCLOUD open and the NAS powered on. The NAS saw the disk, the LCD said disk added, and ReadyCLOUD showed this:
Disk model ... serial ... was added to channel 5 of the head unit.
I would have thought that the drive would have automatically been added to the XRAID volume, but that does not seem to be the case. The volume section in ReadyCLOUD, looks like this:
I am suprised by the RAID 5 comment under the picture and in the data section on the left. XRAID is colored green on the right. When I click on the 5th disk (the new disk which is red), it gives me a bunch of data about the drive and under the "disk state" section it says "unknown". The volume section of ReadyCLOUD has this message:
So I am guessing that disk had some data on it that the NAS is seeing. When I click on that disk I can click on format on the right or XRAID on the right. Which one am I supposed to click on? I get that format will delete any data on disk 5, that is fine. I just want to add it to the XRAID volume that I thought was already there, but is showing up as RAID level 5? I am a bit new to OS 6.6.0 so sorry for bugging you on something that seems like a normal process (adding a disk). The plan is to add these last two drives and then disconnect the power cord for the night and plug it back in the next day. If it survives that, then there will be much rejoicing and I will close the ticket.
Here is an update. Mdgm was not able to mount the USB flash with the NAS not showing it's model number or serial number correctly in RAIDar. I was asked to reboot the NAS and it booted fine and displayed all information correctly. Mdgm checked the internal flash and that appeared fine. Why it boots fine one time and not the other times is the issue that needs to be resolved before I put data on the system.
As for things to do at this time, here are my thoughts:
1. Put the monitor back on it and check the POST and boot sequence for any clues.
2. Disable quick boot in the BIOS to see if that shows any clues on the monitor.
3. Check all the disk drives against the hardware compatibility list. I have used different drives over time in this NAS. These are the drives that are currently in the NAS:
4. Run diags via RAIDar.
5. Look for firmware updates for the disk drives. Here are the firmware version for the drives:
Disk1: CC49
Disk 2: SD1A
Disk 3: CC49
Disk 4: CC49
Disk 5: HP63
Disk 6: SD1A
6. All drives are Seagate disk drive. Remove each drive and run SeaTools long and short tests on each drive. I did that a while ago but I can do it again.
7. Try 6 other drives. I have 8 spare hard drives from upgrading the other NASs to 3TB drives
8. Go back to 4 disk drives and recreate the Logical Unit. I did several reboots with only 4 disk drives and those all worked.
9. Do a memory test on the NAS via the boot menu.
10. Maybe the booting issue is a race condition. Perhaps one of the drives is weighing down the SATA bus or the SATA controller and causing boot issues. I can go down to just 1 disk drive and do 10 boot attempts and see if they all work.
I decided to do option 4 and here are the results of running diagnostics via RAIDar:
Successfully completed diagnostics
System
Disk 2 has 6 Reallocated Sectors
Disk 2 has 6 Reallocation Events
Disk 2 has 7 Spin Retries
Disk 3 has 577 Reallocated Sectors
Disk 3 has 577 Reallocation Events
Disk 4 has 97 Reallocated Sectors
Disk 4 has 97 Reallocation Events
Disk 6 has 21 Spin Retries
Disk 6 has 1 Command Timeouts
Should I be concerned about any of that?
