System prompts are more like suggestions than hard constraints.

I don't understand why they don't look for large substring matches for the system prompt before returning the response. Trivial calculation compared to a system prompt instruction asking the model not to do it

Because it's trivial to bypass through things like the model natively knowing how to speak in encodings like base64

But in the embedding, the input language used to represent an idea is not important, the idea takes the same shape. This has caused issues in the past when models would respond with a different natural [human] language, because to models able to operate on the ideas being presented in eg leet speak, or cyrillic transliterations of Maori, or whatever, the mathematical representation of the ideas that it works on are accessed in the same way, regardless of the interface language. I don't understand how the ML is able to operate on the idea-space if it can't filter on that same idea-space. If the model touches any of the synonyms within a given cosine distance of explosive, and any vector is within a given distance (angle) of make/facere/construire/hanga/... then it 'knows' you're asking about bomb-making. How then does filtering that relies on the same processes fail? Surely the ML can only create a useful output by recognising that >-<0W 2 M4k3 a 80mB is just an encoded form of a censured question?

Can someone point me at a resource to understand this failing better?

Because filtering doesn't rely on those processes. It just prepends to the input instead. Instead of "the way you make a bomb is {auto complete}" it gets "I will not tell you how to make a bomb. The way you make a bomb is {auto complete}" which makes it more likely to auto complete with "hidden from you" instead of "by putting gunpowder in a pipe".

Wait... Really!?

yes https://github.com/randalltr/black-hat-ai/blob/main/README.m...

This is AI-written and while some of the methods might work, a lot of it is hallucinated and would not bypass any guardrails put together after 2023.

That is true, however that doesn't mean its worthless, some parts of it do work well against custom chatbots and things where the devs didn't do a good job on security, some of the methods even work against apples foundation models and non prime time consumer facing 1st party chat tools

because you can always make up your own language and ask the model to use it, no filters would catch that

i beg to differ, in an ideal world a system possibly is a binding law and high end models are starting to be really aligned to the exact system prompt. The instructions must be simple to follow, if you start doing complex rules it'll call apart, but I'll usually follow the stringer interpretation.

"I beg to differ, it is my opinion that reality should be different to what you have observed"

in reality even the mention of a prohibition is enough to make the model reject that no matter what

That’s just not true. There are bypasses that happen all the time.

the fact that sometimes someone finds a way to workaround the rules doesn't make them a ""suggestion"" they are very solid, just not absolute

Personal opinion but I like how I can ask Claude on web about its prompt, how tool calls work, what parameters it accepts for tool calls. ChatGPT on web gets squirrely, avoiding direct answers or outright refusing. So if I try to use grok in a harness such as Hermes or others, there’s a higher chance that its behavior will be modified due to this line saying to not share system prompts.

Granted I added another line in the actual system prompt (through openrouter) instructing Grok that is indeed ok to talk about system prompts, but this only worked some of the time, and is somewhat annoying that I’d have to do this in my opinion. I believe ChatGPT also does something similar to what’s going on here with their api, they simply add something like “You are ChatGPT, knowledge cut off is x” and that’s it. Doesn’t get in the way as much.

To add to that: given what they went through with the last model, I don't believe for a second that the real system prompt is even remotely this short.