Reply

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

fastfwd
Virtuoso

Problem getting Intel microcode to load early on my Frankenstein NAS

So... After running multiple legacy ReadyNAS boxes on OS4 for 5+ years, I've decided to upgrade to OS6.  I wish I knew who made the decision to include software support for legacy devices in OS6 (and also, btw, to ship a security update for OS4 after the "final" 4.2.28 release), because that person is awesome.

 

Anyway, here's my situation: The upgraded E6600 CPUs in my 6-bay boxes want microcode updates that Intel first made available on 9/30/2010, but they can't get the microcode in the usual way -- from the BIOS -- because the latest BIOS for those boxes is dated 7/26/2010.

 

Fortunately, Linux has a solution: There's a set of packages -- iucode-tools and intel-microcode -- that can load microcode updates into the CPU at boot time (or anytime afterward, although obviously it's best to load the updates as early in the boot process as possible).  Unfortunately, those packages require support from the kernel, and OS 6.6.1's kernel was built without that support (probably because the current-production ReadyNAS devices are still getting BIOS updates).

 

Ok, so I fired up Google and learned how to build the Linux kernel.  Compiled it with built-in support for microcode updates, then copied it to the internal flash as an additional boot-menu option (thanks to IcyK's genius post for that idea) and it booted up just fine.  I installed the Debian microcode packages and used the tool to update the CPU microcode from the command line; that worked fine, too.

 

But here's the problem: Updating the CPU's microcode after it's already executed a few billion instructions isn't very safe.  To get the kernel to update the microcode at the safest time (i.e., as early as possible, right at the start of the boot sequence), all the instructions on the web say that the microcode update must be added to the initramfs.  But those instructions also all say that the initramfs file is in the /boot directory, and on my machine there's nothing in that directory at all.

 

There's an initrd.gz file on the internal flash, but the update-initramfs tool doesn't recognize it.  Maybe I'm misusing the tool (I probably am), but I don't even know whether that initrd.gz file is used by OS6 or is just a vestigial artifact from my box's original OS4.  I have no idea how a new (RN516, etc.) box's boot flash compares to mine, and I don't have a new box to look at.  Plus I don't really understand my box's boot flash, anyway.

 

So three things:

 

1. If anyone's modified the initramfs on a legacy NAS running OS6, I'd love to know how to do it.

 

2. If there's a way to get the early-microcode hook compiled directly into the kernel, I'd love to know how to do that, too.  My microcode updates are from 6+ years ago; there aren't going to be any newer ones for these CPUs, so hard-coding them into the kernel would be fine.

 

3. If anyone from Netgear sees this... So long as you're releasing unsupported code for legacy devices anyway, I hope you'll consider adding some even less-supported features to future builds -- specifically, kernel options that would benefit those of us who love our old ReadyNAS boxes so much that we've installed not only a new OS but also a new CPU. Things like this early microcode-update support, explicit Intel Core2 support, any optimizations for compatible CPUs newer than the original legacy E2160 but older than the currently shipping Corei5 (or whatever), etc.

 

2 × Pro 6 [Main & Backup] – XRAID2 Dual-Redundant (RAID6): 14.53TB (110MB/s Write, 123MB/s Read)
  • CPU: 2.4GHz E6600 SL9ZL - RAM: 8GB (PSD28G800K) - BIOS: 07/26/2010 - OS: 6.10.3 w/ custom kernel - Drives: 6 × ST4000VN000 / ST4000VN008
    Fans: Noctua NF-S12A FLX (Case), NF-R8 PWM (CPU), NF-R8 w/LNA (Power Supply) - USB: STGY8000400 - UPS: APC BR1000G
Ultra 2 Plus [for experimentation and testing] – JBOD: 8TB (110MB/s Write, 123MB/s Read)
  • CPU: Standard 1.8GHz Atom D525 - RAM: 2GB (CT25664BC1067) - BIOS: 11/02/2011 - OS: 4.2.31 - Drives: 2 × ST4000VN000 - Fan: Noctua NF-B9 - USB: ST31000340
Model: ReadyNAS-OS6|
Message 1 of 12

Accepted Solutions
fastfwd
Virtuoso

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

Ok, I've answered my own question.  It is NOT necessary to lzma-compress the initramfs file on the boot flash.

 

As described in my previous post, I created a cpio archive just like the one inside initrd.gz, except that it also contains the /kernel/x86/microcode/GenuineIntel.bin file with my CPU's microcode update.  Copied it to the boot flash without lzma-compressing it first (called it "initrduc"), edited SYSLINUX.CFG to load it along with my new kernel ("k4-1-30.uc") as a boot-menu option, and boom:

 

root@NAS3:~# dmesg | head
[    0.000000] microcode: CPU0 microcode updated early to revision 0xd0, date = 2010-09-30
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.1.30.x86_64.1 (root@NAS3) (gcc version 4.9.2 (Debian 4.9.2-10) ) #2 SMP Tue Jan 17 01:33:26 PST 2017
[    0.000000] Command line: initrd=initrduc reason=normal BOOT_IMAGE=k4-1-30.uc
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000] Disabled fast string operations
[    0.000000] e820: BIOS-provided physical RAM map:

 

Now my NAS not only has a CPU updated to the latest microcode, it's also running a kernel built specifically for its hardware (Core2 Duo CPU, Yukon2 Ethernet chip) and with a few other optimizations for that hardware.

 

If everything's still running fine a few days from now, I'll make this kernel/initramfs combo the default, and make the original combo a boot-menu option.  Then I'll run through the process again for my main NAS, and maybe also for the Atom D525 in my little Ultra 2 Plus.

 

2 × Pro 6 [Main & Backup] – XRAID2 Dual-Redundant (RAID6): 14.53TB (110MB/s Write, 123MB/s Read)
  • CPU: 2.4GHz E6600 SL9ZL - RAM: 8GB (PSD28G800K) - BIOS: 07/26/2010 - OS: 6.10.3 w/ custom kernel - Drives: 6 × ST4000VN000 / ST4000VN008
    Fans: Noctua NF-S12A FLX (Case), NF-R8 PWM (CPU), NF-R8 w/LNA (Power Supply) - USB: STGY8000400 - UPS: APC BR1000G
Ultra 2 Plus [for experimentation and testing] – JBOD: 8TB (110MB/s Write, 123MB/s Read)
  • CPU: Standard 1.8GHz Atom D525 - RAM: 2GB (CT25664BC1067) - BIOS: 11/02/2011 - OS: 4.2.31 - Drives: 2 × ST4000VN000 - Fan: Noctua NF-B9 - USB: ST31000340

View solution in original post

Message 8 of 12

All Replies
mdgm-ntgr
NETGEAR Employee Retired

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

1. Modifying that would require building your own /init for it as well, I think

2. I think it would be unlikely that that would happen. The number of users that would benefit would be quite small. Unsupported unit for OS6 with an unsupported CPU upgrade

Message 2 of 12
fastfwd
Virtuoso

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

Thanks, mdgm.

 

Your comment helped me to find what I think is a way to do what I want, and it's actually super-easy.  I'll try it after lunch and post an update here.

 

And, yeah, I agree that it's unlikely that OS6 will include kernel features not only for products no longer supported by Netgear, but also for products that were NEVER supported by Netgear... But I made the suggestion anyway, under the basic principle of "It can't hurt to ask".

 

2 × Pro 6 [Main & Backup] – XRAID2 Dual-Redundant (RAID6): 14.53TB (110MB/s Write, 123MB/s Read)
  • CPU: 2.4GHz E6600 SL9ZL - RAM: 8GB (PSD28G800K) - BIOS: 07/26/2010 - OS: 6.10.3 w/ custom kernel - Drives: 6 × ST4000VN000 / ST4000VN008
    Fans: Noctua NF-S12A FLX (Case), NF-R8 PWM (CPU), NF-R8 w/LNA (Power Supply) - USB: STGY8000400 - UPS: APC BR1000G
Ultra 2 Plus [for experimentation and testing] – JBOD: 8TB (110MB/s Write, 123MB/s Read)
  • CPU: Standard 1.8GHz Atom D525 - RAM: 2GB (CT25664BC1067) - BIOS: 11/02/2011 - OS: 4.2.31 - Drives: 2 × ST4000VN000 - Fan: Noctua NF-B9 - USB: ST31000340
Message 3 of 12
Sandshark
Sensei

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

There is a BIOS update that runs on OS 4.2.x that lets other CPU's run OS6.  I have the E6600 on two of my units and an E7500 on one.  I've seen instructions for updating by extracting the update from the .bin package and using Linux toold to update, but downgrading back to 4.2.x and using the .bin package could be easier.  Just put in a small spare drive to downgrade, update BIOS, and go back to OS6.  I'm afraid I don't have a pointer to the package, but it's called BIOS_Update_Package_0.5-x86.bin, so try a search.

 

There were some very early Pro BE and Pioneer units that have a different revision motherboard, and you are out of luck if you have one of those.  But I don't think that one ran the E6600 even on OS4.2.x.

 

Take a look at bios_ver.log in the logs download .zip.  If it says 10/03/2008 FLAME6-MB V1.6, as most of the Pro BE/Pioneer units did, you are in luck.  The update will move you up to 07/26/2010 FLAME6-MB V2.0, which has the required microcode.  If it says something else, you are on your own.  But the package seems to be pretty good at recognizing your system and only updating when it can.

Message 4 of 12
mdgm-ntgr
NETGEAR Employee Retired

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

The system is already running the 07/26/2010 BIOS as stated in the opening post of the thread.

Message 5 of 12
Sandshark
Sensei

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

Hmmm.  I missed that.

 

Maybe he mistakenly got the Pentium E6600 instead of the Core 2 Duo E6600.  The Pentium does need additional microcode over the Core 2 Duo.  I have two that are running the Core 2 Duo E6600 and they need no microcode mods, even with the previous BIOS version.  I have no idea what Intel was thinking re-using the E6600 moniker,

 

Besides the microcode issue, the BIOS may not select the proper frequency multiplier for that Pentium chip.  The Core 2 Duo E7500 is definately a better choice than the Pentium E6600 for a ReadyNAS.  Similar performance with full support in the BIOS (provided you have the latest update).  They're under $5 on eBay these days.  I got the E6600's when the E7500 was a lot more.  The E7500 is a huge step up from the Pentium E2160 in the original Pro BE/Pioneer.  Not as much from the Pentium E5300 of the Pro6/Ultra6Plus.  The Core 2 Duo E6600 is still a sizable step up from the E2160, but negligable from the E5300.

 

FWIW, the datasheet on the E5000/E6000 says "The Intel® Pentium® dual-core processor E5000 series operates at a 800 MHz FSB frequency (selected by a 200 MHz BCLK[1:0] frequency). The Intel® Pentium® dualcore processor E6000 series operates at a 1066 MHz FSB frequency (selected by a 266 MHz BCLK[1:0] frequency). Individual processors will only operate at their specified FSB frequency."

 

That last statement differs from the usual "Individual processors operate only at or below the rated frequency." (which is what's in the E7500 spec sheet.)  Perhaps Netgear chose an E5000 series for the newer systems over an E6000 because the motherbord won't do a 266MHz BCLK needed for the 1033MHz FSB.

Message 6 of 12
fastfwd
Virtuoso

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

Thanks, Sandshark.  I guess I should have been clearer about my setup:  My 6-bay ReadyNAS devices have been running on Core2 Duo E6600 SL9ZL CPUs with the 07/26/2010 BIOS since early 2013. They're functional, but the BIOS only brings them up to update "cb" while Intel's microcode-update package will bing them to "d0".

 

More news on my progress:

 

I now know just enough about initramfs and the syslinux loader to be really dangerous.  Yesterday I discovered that syslinux since version 3.71 has had the ability to load multiple initramfs files specified in the SYSLINUX.CFG file, so I planned to use that facility.  Unfortunately, I later discovered that syslinux on my boot flash is only version 3.31:

 

root@NAS3:/media/INT_FLASH# strings ./LDLINUX.SYS
SYSLINUX 3.31 v1.07 Tue Feb 15 11:31:02 PST 2011
fBf9
 Load error -
Debug Serial Port Enabled
....

So that idea's a non-starter.

 

My current plan is to modify a copy of the existing initrd.gz to include kernel/x86/microcode/GenuineIntel.bin so everything can be loaded at once, in the single file that syslinux 3.31 supports.  I've unpacked the existing LZMA-compressed initrd.gz into a cpio achive, unpacked that into the initramfs directory tree, added the kernel directory, and repacked the tree into a cpio archive.  But now I'm stuck because lzma has a ton of options and I haven't found the right ones to replicate what Netgear used for their initrd.gz.

 

So I have two new questions:

 

1. Executing "file" on Netgear's original initrd.gz produces "initrd.gz: LZMA compressed data, streamed", and 7-Zip identifies the compression method as "LZMA:26".  Does anyone know what lzma options will produce that format?

 

2. Alternatively, does the initramfs file on the boot flash even need to be compressed with lzma, or will syslinux load a raw cpio archive?

 

2 × Pro 6 [Main & Backup] – XRAID2 Dual-Redundant (RAID6): 14.53TB (110MB/s Write, 123MB/s Read)
  • CPU: 2.4GHz E6600 SL9ZL - RAM: 8GB (PSD28G800K) - BIOS: 07/26/2010 - OS: 6.10.3 w/ custom kernel - Drives: 6 × ST4000VN000 / ST4000VN008
    Fans: Noctua NF-S12A FLX (Case), NF-R8 PWM (CPU), NF-R8 w/LNA (Power Supply) - USB: STGY8000400 - UPS: APC BR1000G
Ultra 2 Plus [for experimentation and testing] – JBOD: 8TB (110MB/s Write, 123MB/s Read)
  • CPU: Standard 1.8GHz Atom D525 - RAM: 2GB (CT25664BC1067) - BIOS: 11/02/2011 - OS: 4.2.31 - Drives: 2 × ST4000VN000 - Fan: Noctua NF-B9 - USB: ST31000340
Message 7 of 12
fastfwd
Virtuoso

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

Ok, I've answered my own question.  It is NOT necessary to lzma-compress the initramfs file on the boot flash.

 

As described in my previous post, I created a cpio archive just like the one inside initrd.gz, except that it also contains the /kernel/x86/microcode/GenuineIntel.bin file with my CPU's microcode update.  Copied it to the boot flash without lzma-compressing it first (called it "initrduc"), edited SYSLINUX.CFG to load it along with my new kernel ("k4-1-30.uc") as a boot-menu option, and boom:

 

root@NAS3:~# dmesg | head
[    0.000000] microcode: CPU0 microcode updated early to revision 0xd0, date = 2010-09-30
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.1.30.x86_64.1 (root@NAS3) (gcc version 4.9.2 (Debian 4.9.2-10) ) #2 SMP Tue Jan 17 01:33:26 PST 2017
[    0.000000] Command line: initrd=initrduc reason=normal BOOT_IMAGE=k4-1-30.uc
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000] Disabled fast string operations
[    0.000000] e820: BIOS-provided physical RAM map:

 

Now my NAS not only has a CPU updated to the latest microcode, it's also running a kernel built specifically for its hardware (Core2 Duo CPU, Yukon2 Ethernet chip) and with a few other optimizations for that hardware.

 

If everything's still running fine a few days from now, I'll make this kernel/initramfs combo the default, and make the original combo a boot-menu option.  Then I'll run through the process again for my main NAS, and maybe also for the Atom D525 in my little Ultra 2 Plus.

 

2 × Pro 6 [Main & Backup] – XRAID2 Dual-Redundant (RAID6): 14.53TB (110MB/s Write, 123MB/s Read)
  • CPU: 2.4GHz E6600 SL9ZL - RAM: 8GB (PSD28G800K) - BIOS: 07/26/2010 - OS: 6.10.3 w/ custom kernel - Drives: 6 × ST4000VN000 / ST4000VN008
    Fans: Noctua NF-S12A FLX (Case), NF-R8 PWM (CPU), NF-R8 w/LNA (Power Supply) - USB: STGY8000400 - UPS: APC BR1000G
Ultra 2 Plus [for experimentation and testing] – JBOD: 8TB (110MB/s Write, 123MB/s Read)
  • CPU: Standard 1.8GHz Atom D525 - RAM: 2GB (CT25664BC1067) - BIOS: 11/02/2011 - OS: 4.2.31 - Drives: 2 × ST4000VN000 - Fan: Noctua NF-B9 - USB: ST31000340
Message 8 of 12
mdgm-ntgr
NETGEAR Employee Retired

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

If you want to use this modification you're going to have to redo this work for every firmware update you wish to install.

The reason we compress the initrd cpio archive using lzma rather than gzip is to keep the size of the download as small as we can. It also allows for more potential growth in the size of the firmware over time whilst still being able to fit it on the internal flash.

Devices that shipped with OS6 have a newer syslinux bootloader.

Message 9 of 12
Sandshark
Sensei

Re: Problem getting Intel microcode to load early on my Frankenstein NAS

OK, so what do you gain with the additional microcode?  I've not been having any problems with OS 6.6.0 (haven't moved up to 6.6.1 yet).

Message 10 of 12
fastfwd
Virtuoso

Re: Problem getting Intel microcode to load early on my Frankenstein NAS


@Sandshark wrote:

OK, so what do you gain with the additional microcode?

 

No one knows.  Intel never says.

 

 

-Andrew

 

2 × Pro 6 [Main & Backup] – XRAID2 Dual-Redundant (RAID6): 14.53TB (110MB/s Write, 123MB/s Read)
  • CPU: 2.4GHz E6600 SL9ZL - RAM: 8GB (PSD28G800K) - BIOS: 07/26/2010 - OS: 6.10.3 w/ custom kernel - Drives: 6 × ST4000VN000 / ST4000VN008
    Fans: Noctua NF-S12A FLX (Case), NF-R8 PWM (CPU), NF-R8 w/LNA (Power Supply) - USB: STGY8000400 - UPS: APC BR1000G
Ultra 2 Plus [for experimentation and testing] – JBOD: 8TB (110MB/s Write, 123MB/s Read)
  • CPU: Standard 1.8GHz Atom D525 - RAM: 2GB (CT25664BC1067) - BIOS: 11/02/2011 - OS: 4.2.31 - Drives: 2 × ST4000VN000 - Fan: Noctua NF-B9 - USB: ST31000340
Message 11 of 12
fastfwd
Virtuoso

Re: Problem getting Intel microcode to load early on my Frankenstein NAS


@mdgm wrote:

If you want to use this modification you're going to have to redo this work for every firmware update you wish to install.

The reason we compress the initrd cpio archive using lzma rather than gzip is to keep the size of the download as small as we can. It also allows for more potential growth in the size of the firmware over time whilst still being able to fit it on the internal flash.

Devices that shipped with OS6 have a newer syslinux bootloader.


Fortunately, it'll only take a few minutes now that I know how to do it.  And at least on my old boot flash, there's plenty of free space -- two copies of the kernel plus two copies of initramfs (one uncompressed) only uses half of the flash.

 

2 × Pro 6 [Main & Backup] – XRAID2 Dual-Redundant (RAID6): 14.53TB (110MB/s Write, 123MB/s Read)
  • CPU: 2.4GHz E6600 SL9ZL - RAM: 8GB (PSD28G800K) - BIOS: 07/26/2010 - OS: 6.10.3 w/ custom kernel - Drives: 6 × ST4000VN000 / ST4000VN008
    Fans: Noctua NF-S12A FLX (Case), NF-R8 PWM (CPU), NF-R8 w/LNA (Power Supply) - USB: STGY8000400 - UPS: APC BR1000G
Ultra 2 Plus [for experimentation and testing] – JBOD: 8TB (110MB/s Write, 123MB/s Read)
  • CPU: Standard 1.8GHz Atom D525 - RAM: 2GB (CT25664BC1067) - BIOS: 11/02/2011 - OS: 4.2.31 - Drives: 2 × ST4000VN000 - Fan: Noctua NF-B9 - USB: ST31000340
Message 12 of 12
Top Contributors
Discussion stats
  • 11 replies
  • 4886 views
  • 0 kudos
  • 3 in conversation
Announcements