Eh, he probably hit Fable’s guardrails when he promoted “secure this project”

The 2k line expression evaluator is kind of an actual language here. I think its error messages and devtools will matter much when people literally still write raw JS.

You're 100% right. Good error messages are important AF for something like this. I'm take a deeper look into this and focus into proper devtools support ASAP to make sure the quality is where it needs to be. Thanks for the comment ;)

Btw, its OSS, feel free to evaluate yourself and contribute!

I would personally love to see the best score over time, not only the final. Then IMO it's also useful to know about effectiveness of /goal

If I had one I would definitely try creating a separate VLAN for it to control, otherwise it's isolated from your files but still has access to your network and devices in it.

Put it in a faraday cage with some kind of explosive device set to trip if it escapes containment. It's the only way to be sure.

This can actually accidentally become the best debugging tool for map files, ngl you should cook

Maybe the backup job should periodically restore into a temporary DB and run PRAGMA integrity_check. Theoretically should help

I think it's actually interesting how you can use the same model to both find and falsify the findings

you can, but it's better to use a different model or higher effort level at the very least to do the verifying part. (haven't checked to see what it is they are doing exactly), but doing vulnerability validation with the same model and effort you used to find the vulnerabilities isn't going to be a true second set of eyes and the model will likely always just try and justify it was right in the first place. ime, it's better to start with a fresh, clean context and at the very least a higher effort level on the same model. pass the finding(s) to a model that hasn't seen it before and it will judge it with an unbiased perspective.

It will be valuable to have two types of benchmarks: ones that evolve alongside the models and ones that never change. You probably can't get historical stability and resistance to flooding and training on at least some parts of it from the same test

I wonder if very cheap code generation will make software monocultures less relevant here. Because lots of incompatible devices is awful to work with, security stuff may also hurt

For the sake of interest you could try to expose periodically rotated keyed hashes of IPs and credentials instead of the raw values. It would still let people correlate events within a limited time window