SRX5308 VPN to FVS318v3

For what it's worth, I do have this exact type of IPSec tunnel configured and working well, with the 5308 located in our datacenter and the 318 at my home.  The tunnel between these two routers has "just worked" since the earliest 5308 firmware I had through 4.3.4-2 just this month while the 318 is on 3.0_28.  I'm not sure if that info gives you encouragement or is just frustrating.  I did not use the wizard on either side to set them up, just manually set the parameters to match.

Unfortunately I often see similar errors to what you're quoting in your logs (malformed packets, then phase1 failed...) when configuring new VPN tunnels on the 5308, for sure when a Cisco ASA or a simple linux box running racoon is on the far end of the tunnel.  It's very, very frustrating.

When I updated the firmware on our 5308 recently, I decided to wipe it completely clean and rebuild all our firewall rules and ipsec parms from scratch rather than restoring from a backup config or anything.  I had hoped there was something that had been malconfigured in the older firmware versions and then never cleared which could cause this, but unfortunately many IPSec tunnels appear to be impossible to set up and neither me nor the far-end admins can figure out why because other ones do work.  There doesn't seem to be a reason.