"If you know what those types of article are you can separate them out and make sure they're represented in the prompt. But now we're back at needing a tonne of domain knowledge about your problem -- that's like having to write custom features to make your model work"
I think this is where the different perspectives come into play.
If you're an NLP practitioner you are thinking, oh no! I need to know a lot about audience intention and how the articles are represented navigationally and the kind of variety people are looking for and how articles might fit multiple categories, etc, etc. And you have to think about these things on a meta level ("prompt engineering"), because you have to instruct the model on how to act in an abstract way.
If you're someone who wants to run a news site then you already are thinking a ton about these things, and probably have a dozen things you'd like to change and adjust, new ways of presenting content, etc. You _wish_ you could be thinking about these domain-specific topics.
What feels like a bug to the NLP practitioner – needing a deep understanding of the domain – is a feature to... everyone else. It's a feature to the people who care most about the results.
The other big perspective difference here, I believe, is how you think about goals. How many tasks ARE categorization? My intuition is that it's a quite small number. There are many tasks that can be implemented with one step as categorization, but that is seldom the task. To the NLP practitioner categorization might seem very prominent – that's when someone calls you up or hands over the work. But with an LLM you might be able to do a much larger portion of the real task, with or without categorization.
Even with a categorization task, when I'm working with an LLM I usually don't produce just a "category", but produce other information at the same time, often using natural language as a first-class data type because it can be fed back into an LLM. In my experience the results are often (usually?) much higher quality because I'm not breaking things down into steps where the inaccuracies propagate between steps, but instead going right for the result, and using a model that can basically "check" itself against general knowledge, scrubbing out nonsensical results during inference. (As a result the remaining inaccuracies often appear plausible and are labeled "hallucinations"... it can make things more challenging, but what we don't see are the multitude of obvious inaccuracies that a more traditional NLP system would create, and which in a sense exist momentarily during the LLM inference.)
I know what you mean about the domain knowledge, and it's a thing that's a bit different from the previous situation with the feature engineering. The problem with feature engineering for linear models was you really had to understand the domain _and_ the ML.
I do think there's a similarity in how creative you need to be though. It means that applying LLMs to new problems isn't as straight-forward as people make it seem at first, and isn't necessarily reliable. In contrast, labelling data is something that has a much smoother effort to reward curve for most problems. The experience of labelling the data, training a model and getting it hosted isn't as seamless as it could be -- we're working on that.
I do think classification is pretty fundamental though. The way I see it is, model outputs can be either human-facing, machine facing, or both. If you're going to feed the output into another system, that system wants the data to obey some limited schema, so that you can run logic based on it. For instance, let's say you want to trigger some alert when a particular kind of article is published or a particular kind of message is sent. Triggering the alert is a boolean thing, so that has to be a classification task. You might want to also attach text in the alert, so that's a human-facing part.
I agree that there's lots of ways that LLMs can be used iteratively, allowing more trade-off of computational cost for accuracy. I just think in a lot of cases, the best way to exploit that is to trade towards as much accuracy as you can get, and use that to create training samples. You can then manually correct the training samples as well --- if they're mostly correct, reviewing them is pretty quick. You can then train and evaluate a smaller model.
"For instance, let's say you want to trigger some alert when a particular kind of article is published or a particular kind of message is sent. Triggering the alert is a boolean thing, so that has to be a classification task."
I think this is a good example of how decomposed tasks can feel very different from goal-oriented task definitions.
We can imagine a goal like "I want to know when one of my competitors shows up in the news." Now you have a bunch of tasks: entity extraction, determining the subject of an article, maybe categorizing articles. Then you can define a pipeline and conditional to trigger a notification. And you might get great accuracy on each of these.
But the goal is really about getting actionable information. In practice the approach above creates a ton of alerts, and the person receiving them will filter through them, ignore a bunch, have to figure out what is really new, etc. An LLM could do things like accumulate a running set of background knowledge, identifying what information is truly "new" (and in a granular way, not just detecting duplicate articles). You can tell the LLM all kinds of details about what you are interested in, "categories" that are completely inaccessible to traditional NLP because they are described by higher-level concepts or have to be combined with history or user-provided context (something that happens naturally in a prompt).
Traditional NLP feels very industrial to me. Factories can be very productive and high volume, but they redefine the tasks to satisfy their processes. Individuals don't interface with factories, and factories don't serve individuals.
I think the 'industrial' or assembly-line analogy is probably good, and I see what you mean about the alternative system design. Thanks for explaining the other approach.
I'll put it this way. If you want to integrate ML into a product, or even a system with lots of internal users, you end up increasingly towards the 'factory' approach.
I think this is where the different perspectives come into play.
If you're an NLP practitioner you are thinking, oh no! I need to know a lot about audience intention and how the articles are represented navigationally and the kind of variety people are looking for and how articles might fit multiple categories, etc, etc. And you have to think about these things on a meta level ("prompt engineering"), because you have to instruct the model on how to act in an abstract way.
If you're someone who wants to run a news site then you already are thinking a ton about these things, and probably have a dozen things you'd like to change and adjust, new ways of presenting content, etc. You _wish_ you could be thinking about these domain-specific topics.
What feels like a bug to the NLP practitioner – needing a deep understanding of the domain – is a feature to... everyone else. It's a feature to the people who care most about the results.
The other big perspective difference here, I believe, is how you think about goals. How many tasks ARE categorization? My intuition is that it's a quite small number. There are many tasks that can be implemented with one step as categorization, but that is seldom the task. To the NLP practitioner categorization might seem very prominent – that's when someone calls you up or hands over the work. But with an LLM you might be able to do a much larger portion of the real task, with or without categorization.
Even with a categorization task, when I'm working with an LLM I usually don't produce just a "category", but produce other information at the same time, often using natural language as a first-class data type because it can be fed back into an LLM. In my experience the results are often (usually?) much higher quality because I'm not breaking things down into steps where the inaccuracies propagate between steps, but instead going right for the result, and using a model that can basically "check" itself against general knowledge, scrubbing out nonsensical results during inference. (As a result the remaining inaccuracies often appear plausible and are labeled "hallucinations"... it can make things more challenging, but what we don't see are the multitude of obvious inaccuracies that a more traditional NLP system would create, and which in a sense exist momentarily during the LLM inference.)