The word "need" in this comment seems a bit exaggerated. Yes, we'd want to have some proof and verification, but that is not an inescapable requirement.
I'm fairly sure that we as a society in general are willing to launch systems where we have only a reasonable expectation and trust that it will most likely work properly instead of total proof and verification that it definitely will do so. Just take a look at pretty much any life-critical system in use today; some verification and testing is required, but formal proof of correctness is a very high bar that's never required for practical systems. It's highly desirable, but it will only be an absolute requirement if it's reasonably easy to achieve.
I'm fairly sure that if it turns out that we can't provide guarantees and proof about what the machine might do, then we'll just do our best even if the result is not provable and verifiable and launch such systems anyway.
And I think that even if we have a choice between two systems where one is verified and proved that it never can do anything bad but is otherwise inferior in what it can do, and the second system has no such proof but seems safe in general testing and simply performs better than the first one.... then it's quite likely that we'll choose the second one anyway.
That is my opinion as well. And indeed the post talks mainly about dynamic verification and achieving some "good enough" verification quality (as determined by coverage and other metrics).
And it is in that context that "soft" techniques like ML can help a lot, and thus the question of how to connect them to "hard" rules (which are also part of dynamic verification) becomes interesting.
I'm fairly sure that we as a society in general are willing to launch systems where we have only a reasonable expectation and trust that it will most likely work properly instead of total proof and verification that it definitely will do so. Just take a look at pretty much any life-critical system in use today; some verification and testing is required, but formal proof of correctness is a very high bar that's never required for practical systems. It's highly desirable, but it will only be an absolute requirement if it's reasonably easy to achieve.
I'm fairly sure that if it turns out that we can't provide guarantees and proof about what the machine might do, then we'll just do our best even if the result is not provable and verifiable and launch such systems anyway.
And I think that even if we have a choice between two systems where one is verified and proved that it never can do anything bad but is otherwise inferior in what it can do, and the second system has no such proof but seems safe in general testing and simply performs better than the first one.... then it's quite likely that we'll choose the second one anyway.