Anyone has difficulties connecting their shelly relay to Orbi RBK852 wifi network?

the problem is with coding. A properly coded IoT devices doesn't care if your phone is on 2.4ghz or 5ghz. The ones that are coded poorly or by a lazy coder do care. So if you're phone isn't on the 2.4ghz it won't setup .

You have options. You tried turning down the broadcast power. You can also disable the ssid broadcast (doesn't stop the radio). And then move as far from the router as you can. Then disconnect your phone from the wifi and reconnect. see if it connect to the 2.4ghz. if it does, try setup. If it doesn't, then we'd need another option. 

I've hotspotted my phone before with the same ssid as my router (shut off the router). I've used that to get the IoT device setup on the ssid. then I simply shut off the hotspot and turned the router back on. I haven't had to do that in years and it was many on cheap/poorly coded devices.