Orbi WiFi 7 RBE973
Reply

MySQLDB will not start code 22002010000

BuckoA512
Aspirant

MySQLDB will not start code 22002010000

Good morning everyone

 

ReadyNAS 314

Firmare 6.10.5 Hotfix 1

 

Installed apps:

MySQLDB 1.0.5 from ReadyNAS Xtras

Nextcloud 20.0.1-rnx1 from ReadyNAS Xtras

PHP7.4 from ReadyNAS Xtras

PHPMyAdmin 4 from ReadyNAS Xtras

 

This morning I woke up to this annoying error. MySQLDB turned itself off and will no longer start. "The app failed to start, Please update or contact developer. Code 22002010000"

 

I see in this thread https://community.netgear.com/t5/ReadyNAS-Storage-Apps-Current/MYSQL/m-p/1113575 that the answer is to "bind address to my IP but after changing it back to 127.0.0.1 it is now back working."

 

Unfortunately, I've no idea how to do that.

 

I've e-mailed ReadyNAS Xtras for help but they did not respond yet.

 

If anyone could help I'd be greatful. 

Message 1 of 18
StephenB
Guru

Re: MySQLDB will not start code 22002010000


@BuckoA512 wrote:

 

I see in this thread https://community.netgear.com/t5/ReadyNAS-Storage-Apps-Current/MYSQL/m-p/1113575 that the answer is to "bind address to my IP but after changing it back to 127.0.0.1 it is now back working."

 


Not quite.  The original poster was saying that he had changed the binding to his IP address, but that changing it back to 127.0.0.1 resolved his problem.  It's a 5 year-old thread, and it is possible that your problem is caused by something different.  In any event, that setting should be in /etc/mysql/my.cnf  It's conceivable it's in a different folder - if so, mysqld --help --verbose should find it.  Note you need to use ssh.

 

Did this happen immediately after you updated to 6.10.5?

Message 2 of 18
BuckoA512
Aspirant

Re: MySQLDB will not start code 22002010000

Hi StephenB thank you for replying.

I've never had to ssh since my university days so I'm a little rusty on that to say the least.

No, it didn't happen right after I updated. Everything was working fine yesterday, then today this happens and MySQLDB won't start.

 

ReadyNAS Xtras has replied to me by e-mail too so I'll update if he can figure anything out.

 

Message 3 of 18
BuckoA512
Aspirant

Re: MySQLDB will not start code 22002010000

It looks like the failure is down to a bad disk in the array, or so it seems from the logs. I didn't realise that downloading the logs gave you more information than you get on the web page.

 

Aug 31 10:13:48 ReadyNAS mysqld[6853]: 2021-08-31 10:13:48 7fd324f26780 InnoDB: Error: Write to file ./nextcloud/oc_filecache.ibd failed at offset 25690112.
Aug 31 10:13:48 ReadyNAS mysqld[6853]: InnoDB: 8192 bytes should have been written, only 0 were written.
Aug 31 10:13:48 ReadyNAS mysqld[6853]: InnoDB: Operating system error number 5.
Aug 31 10:13:48 ReadyNAS mysqld[6853]: InnoDB: Check that your OS and file system support files of this size.
Aug 31 10:13:48 ReadyNAS mysqld[6853]: InnoDB: Check also that the disk is not full or a disk quota exceeded.
Aug 31 10:13:48 ReadyNAS mysqld[6853]: InnoDB: Error number 5 means 'Input/output error'.
Aug 31 10:13:48 ReadyNAS mysqld[6853]: InnoDB: Some operating system error numbers are described at
Aug 31 10:13:48 ReadyNAS mysqld[6853]: InnoDB: (link removed)
Aug 31 10:13:48 ReadyNAS mysqld[6853]: 2021-08-31 10:13:48 7fd324f26780  InnoDB: Operating system error number 5 in a file

 

I thought if a bad disk was detected the ReadyNAS would alert me via notiifcation, rather surprised that it didn't.

Message 4 of 18
StephenB
Guru

Re: MySQLDB will not start code 22002010000


@BuckoA512 wrote:

It looks like the failure is down to a bad disk in the array, or so it seems from the logs. I didn't realise that downloading the logs gave you more information than you get on the web page.

 

Aug 31 10:13:48 ReadyNAS mysqld[6853]: 2021-08-31 10:13:48 7fd324f26780 InnoDB: Error: Write to file ./nextcloud/oc_filecache.ibd failed at offset 25690112.
Aug 31 10:13:48 ReadyNAS mysqld[6853]: InnoDB: 8192 bytes should have been written, only 0 were written.

Not convinced it's a failed disk, though is it clearly a failed write.

How full is your OS partition (root)?  You can see this by looking at volume.log - scroll down to the df -h section, and look at the /dev/md0 line.  

 

=== df -h ===
Filesystem      Size  Used Avail Use% Mounted on
udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  524M  3.2G  15% /

Similarly, how full is your data volume?

Message 5 of 18
BuckoA512
Aspirant

Re: MySQLDB will not start code 22002010000

=== df -h ===
Filesystem      Size  Used Avail Use% Mounted on
udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  2.1G  1.6G  58% /

ReadyNAS data has 4.40tb free space. It has 4 x 3tb drives in Xraid.

 

Message 6 of 18
StephenB
Guru

Re: MySQLDB will not start code 22002010000


@BuckoA512 wrote:
=== df -h ===
Filesystem      Size  Used Avail Use% Mounted on
udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  2.1G  1.6G  58% /

ReadyNAS data has 4.40tb free space. It has 4 x 3tb drives in Xraid.

 


Fuller than usual, but not dangerously full.

 

Are you seeing any concerning smart stats for your disks in volume.log?

Also, are you seeing any other I/O errors in system.log and kernel.log

Message 7 of 18
BuckoA512
Aspirant

Re: MySQLDB will not start code 22002010000

Nothing in volume log but I'm getting read/write errors in backup logs and stuff.

 

Also see this in kernel.log

 

Aug 30 16:58:20 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 6, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:17:43 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 7, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:18:42 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 8, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:18:45 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 9, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:18:46 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 10, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:22:24 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 11, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:22:32 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 12, rd 359, flush 0, corrupt 0, gen 0

Message 8 of 18
StephenB
Guru

Re: MySQLDB will not start code 22002010000


@BuckoA512 wrote:

 

Aug 30 16:58:20 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 6, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:17:43 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 7, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:18:42 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 8, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:18:45 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 9, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:18:46 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 10, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:22:24 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 11, rd 359, flush 0, corrupt 0, gen 0
Aug 30 17:22:32 ReadyNAS kernel: BTRFS error (device md126): bdev /dev/md127 errs: wr 12, rd 359, flush 0, corrupt 0, gen 0


Ok.  Try running the disk test from the volume settings wheel next.  It's a long test (can be a few days if you have large disks).

Message 9 of 18
BuckoA512
Aspirant

Re: MySQLDB will not start code 22002010000

Ran the check no errors detected, but backup still failed with read errors from my data so obviously something is wrong.

 

At this point I no longer trust the disks. I'm going to swap the disks one by one and rebuild the array. Hopefully that completes without further errors.

 

Have to say quite disappointed with this, you never know how reliable your systems are until something goes wrong like this and the ReadyNAS has not handled this gracefully at all. All I can hope now is there's no bit rot on data I don't use/access very often that doesn't turn up until years later.

Message 10 of 18
StephenB
Guru

Re: MySQLDB will not start code 22002010000


@BuckoA512 wrote:

Ran the check no errors detected


You mean you ran it earlier?  There's no way it would have completed so quickly (with a pass) if you'd run it when I suggested it a few minutes ago.  It runs in the background, and will eventually give you an indication of completion in the logs.

 


@BuckoA512 wrote:

 

At this point I no longer trust the disks. I'm going to swap the disks one by one and rebuild the array.

 


If you have a disk problem, your process will almost certainly result in a lost volume.

 

You could alternatively do a BTRFS scrub from the volume settings wheel, or test them in a Windows PC with vendor tools (with the NAS powered down).

 

FWIW, I am thinking file system corruption is more likely than a disk failure (though I can't rule out the disk failure from what you've posted so far).  Normally I'd expect to see another error telling me what disk couldn't be read or written, and we are not seeing that here.  You might check kernel.log and look for disk errors as well as btrfs errors.

 

 

 

 

Message 11 of 18
BuckoA512
Aspirant

Re: MySQLDB will not start code 22002010000

I started the disk test yesterday and it completed today.

 

"If you have a disk problem, your process will almost certainly result in a lost volume."

 

Well that's great, I thought the whole point of RAID was to protect against single disk failure?

What's even the point of this thing if I will almost certainly get a lost volume as soon as an error like this comes up?

 

Lucky I have a full backup I suppose!

 

Message 12 of 18
StephenB
Guru

Re: MySQLDB will not start code 22002010000


@BuckoA512 wrote:

 

Well that's great, I thought the whole point of RAID was to protect against single disk failure?

 


Single redundancy RAID does protect against a routine disk failure.  But your process requires dual redundancy.

 

When you re-insert a disk, it is rebuilt from scratch from the remaining disks.  During that process you have no redundancy at all.  If one of the other disks is the problem, then the rebuild will fail and you lose everything. This is normal RAID stuff, not related to the ReadyNAS specifically.

 

A scrub from the volume settings wheel is a better option.

 


@BuckoA512 wrote:

 

Lucky I have a full backup I suppose!


Hopefully not lucky - RAID isn't enough to keep your data safe, so you do need to back up the ReadyNAS if it is used for primary storage.

 

FWIW, if the disks turn out to be ok, then I suggest doing a factory default, reconfiguring the NAS, and restoring the data from the backup.  Make sure you have all the app deb files, as you will need to reinstall them.

Message 13 of 18
BuckoA512
Aspirant

Re: MySQLDB will not start code 22002010000

I have started a data scrub as per your advice.

 

Yeah I understand RAID isn't a backup in its own right. You're suggesting two or more disks may be faulty here, that's always a possibility yet the ReadyNAS can't seem to decide/detect where the problem may lie.

 

Doing a factory default and reconfiguring everything from scratch is quite time consuming and the kind of situation I hoped haivng a RAID array would avoid.

 

Guess we'll see what the scrub comes back with.

Message 14 of 18
StephenB
Guru

Re: MySQLDB will not start code 22002010000


@BuckoA512 wrote:

You're suggesting two or more disks may be faulty here, that's always a possibility yet the ReadyNAS can't seem to decide/detect where the problem may lie.

Actually I haven't suggested that.  I said that you have no RAID redundancy at all when you are rebuilding a disk. If we assume that one disk is bad, then the volume will fail if you rebuild one that isn't bad.  The odds are that the first disk you try to rebuild will be one of the healthy ones.

 

And I have said a couple of times that I suspect the problem is not the disks at all, but might instead be file system corruption (which has other possible causes).  But it is important to rule the disks out.  I agree that the OS 6 ReadyNAS could do a better job in reporting/finding disk and file system problems.

 

FWIW, after you get past this, you might want to schedule the disk test and the scrub.  Personally I run all four of the maintenance tasks on schedule (one per month).  The scrub doubles as a reasonable disk test, as it does require every sector in the data volume to be either read or written.  So scheduling those two every now and then is a good way to give you earlier warning on disk issues.

 


@BuckoA512 wrote:

 

Doing a factory default and reconfiguring everything from scratch is quite time consuming and the kind of situation I hoped haivng a RAID array would avoid.

 


Totally get that, and generally it isn't needed.  But sometimes it's the best path forward.

 

Message 15 of 18
BuckoA512
Aspirant

Re: MySQLDB will not start code 22002010000

"Actually I haven't suggested that.  I said that you have no RAID redundancy at all when you are rebuilding a disk. If we assume that one disk is bad, then the volume will fail if you rebuild one that isn't bad.  The odds are that the first disk you try to rebuild will be one of the healthy ones."

 

Ah yes of course since we can't tell which disk (if any) is bad, now I think about it that makes total sense.

 

"FWIW, after you get past this, you might want to schedule the disk test and the scrub.  Personally I run all four of the maintenance tasks on schedule (one per month).  The scrub doubles as a reasonable disk test, as it does require every sector in the data volume to be either read or written.  So scheduling those two every now and then is a good way to give you earlier warning on disk issues."

 

Good advice, thanks.

 

"Totally get that, and generally it isn't needed.  But sometimes it's the best path forward."

 

Do you happen to know, If I back up the configuration, factory reset, and then re-load it, does that restore everything to how it was, users, passwords etc, just with all the shares empty and the apps uninstalled?

Message 16 of 18
Sandshark
Sensei

Re: MySQLDB will not start code 22002010000

Actually, you need to reinstall the apps before you restore the saved configuration because some of the files in the configuration .zip may have been modified by an app and putting them in without the app installed can result in problems.

Message 17 of 18
BuckoA512
Aspirant

Re: MySQLDB will not start code 22002010000

Makes sense, though I think I'll start from scratch with Nextcloud anyway if it comes to that.

Message 18 of 18
Discussion stats
  • 17 replies
  • 3623 views
  • 0 kudos
  • 3 in conversation
Announcements