While nowhere near compared to their scale, I run a consumer app where most of our users are using the mobile app, with the web app getting perhaps 10-15% of the mobile active users.
However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.
Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.
However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.
I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.
Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.
I still haven't found a way to deal with rotating residential IPs however, and most likely never will.
My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.
Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?
I had the same problem, tried a bunch of stuff, banned millions of IPs, but nothing really worked. I ended up spending a bunch of time rewriting the website with Codex so that it is more efficient. Now I'm still getting the same junk traffic, but it barely affects the CPU anymore. I think this might be the way to go.
Had the same experience with upwork, feels like the door has closed on easy api access to big site data, ironically funneling people's money to proxies and other scrape helpers instead of their own api.
Yeah, perhaps its due to noone wants LLMs trained on their data for free. I don't know, at least for reddit there are still ways to get the data for free.
To me it's a useful signal not to read an article that someone didn't bother to write.
Which is a shame as real insights are buried inside some of these articles, which if the author bothered to write in his own words could have reached an audience that would have appreciated them.
Writing is one of the areas where I want no LLM involvement.
The number of things that make it to the top of HN/Reddit/wherever now that are devoid a human's touch is exhausting. Whether it's a site that's got that Claude frontend smell, or a repo that's got a burst of 10 claude commits before getting shared and abandoned, or a series of blog posts that were written by LLMs... it's all, at this point, a flag for me that the human behind the LLM doesn't really want to engage with others or share; in some ways it dehumanises their entire (supposed) audience.
IDK. Maybe having Claude contribute writing about something novel to the general blogosphere is useful in some dimension, but it usually gives me no confidence in the truth of the post.
More often it's the difference between finishing something and not finishing it - so often LLM's are helping them reach an audience that would appreciate them, even if that audience doesn't include you
I agree with the copy-pasted slop that you see sometimes, that is probably generated from a short prompt and therefore has no real substance to it.
If an article has interesting content (which comes from an human) and the LLM is just used to help the author finish off the article, I don't have any problem with that.
Labeling both scenarios in the same category feels completely wrong to it, as equating vibe coded stuff (as in no human ever read the produced code) and agent-assisted good old software engineering
That's great but I really wish you guys would do something about the llm integration, I tried using it two days ago to create a cohort of users using a sql query, and I was surprised to see that it said that it could not create cohorts for me and i had to resort to exporting data from a sql insight as a cohort cannot use a sql query.
However the worst part was it just writing in the text input slowed down my m4 pro chip to less than 1 fps after 2 prompts and it really left a bad taste in my mouth.
I'm still on their old 500 requests/month plan and the value is simply unbeatable for $20. I've been able to use agents without worrying about usage for my job and personal projects paired with the $20 codex sub and I dread the day when they finally get rid of the requests plan.
Deploying software has been reduced to just running 'vercel' in the terminal, something the agent has zero problems with if they were to just ask.
Distributing desktop software is a bit harder depending on the platform.
The gap between a pet project and great software is still very wide and I have a hard time believing that it will ever be bridged.
I don't see how a solved problem even before ai is the thing that won't be replaced first. I struggle to believe a personal project requires complex infra
TBF the anti-AI artists are insisting AI will never create great works of art. I disagree. But I do agree with Richard Sutton that it would need a reward function steering it towards a popular perception of great art, whatever that may be. I can approximate that with sampling and using my subjectivity to pick the ones I like most and then iterate on variants thereof until I am exhausted/satisfied. That's not the same thing because I am in the loop. But RLHF and RLVR demonstrate the path is feasible IMO. Sadly though, if you tell people a work by Monet is AI art, they do the predictably stupid thing about it so the game is rigged.
W/r to code, I do enjoy playing designer and product manager on personal projects after decades of full-stack development and design, and by full-stack I mean all the way down to machine code. The anti-AI retort is complaining I wouldn't be doing anywhere near as well without those decades of experience. Fair point IMO, but the kids who grow up around this technology will likely expend neural plasticity on wielding it far more effectively than the boomers and GenX ever will. And getting an AI to work with them once again seems like a reward function problem rather than an intrinsic roadblock to me.
All while acknowledging it's much easier said than done. I'm just not going to bet against ingenuity both organic and electronic. However, the incessant mania and panic episodes around AI that just keep happening seem to be the algorithms monetizing enragement and fear to me. That's a real and bigger problem IMO.
AI is unlikely to ever create great works of art because art is not, and has never been just about technical excellence; it is fundamentally a human thing: one human communicating to another. Just even knowing that some art was AI produced is enough to not see it as great: there is no human story or experience behind it, no human context, etc. You can definitely appreciate it in the category of AI work, however.
AI still struggles with technical excellence in some genres of art, but even if they master this, this human element they cannot overcome, by definition.
It's like piano performance: AI can already generate a "prefect" performance audio, a MIDI file can already encode that. But, I hate MIDI files, none of the live-ness, the weirdness, and non-repeatable nuances of an actual performance by an actual pianist.
Your entire argument hinges on being able to tell the extent to which generative AI was used in the creation of any given piece of art, which capability you will not have. You therefore fall into the category the commenter mentioned in that your perception of the value of a piece of art can be heavily influenced by someone convincing you it was AI generated, regardless of the facts.
Jut recently, there was a thread discussing Persepolis, a series of autobiographical graphic novels by Marjane Satrapi (recently deceased) that depict her childhood and early adult years in Iran and Austria during and after the Islamic Revolution. People remarked it was deeply moving.
Part of what makes it deeply moving is the actuality of it. This is a human story based on actual lived experiences. How does an AI produce this?? If it came out later that it was written by an AI (assume for the purpose of argument that we had AI when it was written), then of course m=it's impact would be different.
If a seemingly powerful piece of non-fiction is later exposed as fiction, and AI written fiction at that, won't that change your perception of it? Or if a nice anecdote someone likes to tell is exposed as made up, I would hope that matters.
I guess since we're not talking about art anymore, and are instead talking about the veracity of information, we can safely agree. If I read a news story, I do value that information higher to the extent I am convinced it is actually true information. If I read someone's autobiography, I do value that to the extent that I trust them and that it is coming from them. A piece of fiction, however, or music, or visual art, is something that can stand on its own and be appreciated or not without having to assume this context. The context and provenance can certainly color the appreciation, but it is no longer necessary.
But I am not just taking about factual information. You still don't get it. I was just using that example to show if you take out key components of what makes something what is it, it is no longer the same thing.
I am saying the an intrinsic component of art is the human context of it. You can call AI generated "art" by another name if you want, and enjoy it for what it is, but the reasons why you might enjoy it are different from the reasons you might enjoy human art.
Why do we still enjoy art in spite of the fact that we have photography? Or foot races even though we have cars?
Why is it that we still enjoy music despite not knowing how it was created? Why is it that we still enjoy visual art despite not knowing how it was created?
We can keep throwing counterexamples at each other forever. You can find an endless number of examples wherein the provenance of an activity or piece of work is important to the enjoyment of it, and I can find endless examples where the provenance is unnecessary. What will this prove?
So what you're saying is it will never be about the art itself, it's entirely about selling you on its origin story? Yeah, that sounds about right. So once it can create human-level art, it just needs to spew human-level BS script along with it for the artist to act out, got it.
You misunderstand art. If art were solely about the outcome, we’d all be staring at photographs instead, as no art will ever have as high fidelity.
Art is about the journey of the artist; the meaning with which that art is impregnated is the point. What you cynically refer to as “human-level BS,” others refer to as “the human condition” because we can relate to other humans, and empathy is a thing.
It’s okay not to like art. But pretending art is just “the painting at the end” is nonsense.
I'm doing no such thing. By invalidating the Monet piece if it is described as AI-generated, art becomes entirely about the creation story. So that moves the bar to telling a convincing story. And LLMs can absolutely do that at the level of those museum placards next to each painting. So if you add an actor to pose as the artist, the art is the performance now.
What makes you think I don't like art? Spent 3 hours at an art museum event last weekend staring in details at paintings whilst rich drunk fools kept taking selfies next to them.
But I no longer believe people care about empathy. The US wouldn't have elected a grifting performance artist president twice if they valued empathy. We're much more hindbrain-driven than I suspect you think we are.
> I'm doing no such thing. By invalidating the Monet piece if it is described as AI-generated, art becomes entirely about the creation story. So that moves the bar to telling a convincing story.
No. Art isn’t about the “origin story,” but about what it makes you feel. In part, what it makes you feel is due to the fact that it was made by a human, with human emotion and intent, to communicate an idea to the world. AI has no such emotional backing.
> And LLMs can absolutely do that at the level of those museum placards next to each painting. So if you add an actor to pose as the artist, the art is the performance now.
Museum placards are not what make art interesting.
> But I no longer believe people care about empathy. The US wouldn't have elected a grifting performance artist president twice if they valued empathy.
Irrelevant; two things can be true at once. Also, not everyone has empathy, and not everyone likes art.
That’s why most people don’t have art on their walls, but random prints from IKEA.
> We're much more hindbrain-driven than I suspect you think we are.
Many people are! But “people who like art” is a relatively small subset of people, and I maintain that nearly everyone who likes art likes it because it makes them feel something.
Otherwise, there would be no difference between a Picasso and a print from EBay.
People who like art don’t buy NFTs. They buy art made by humans.
I am reminded of The Doctor Who episode where the original Mona Lisa was destroyed and only one of many copies clearly marked "THIS IS A FAKE" underneath the pigments survived.
Photographs made technical skill at recreating reality with paint less relevant, while making point of view, staging, etc, more important. You can absolutely create a photograph that is art, but a camera is not an artist, the person holding it is.
Only someone with little appreciation of music will describe the difference between an actual performance and an AI generated one as "human-level BS". It makes a large difference in my enjoyment of the music.
Have you listened to a MIDI file before? And have you listened to (or attended, preferably in person) a piano concert before? You can't compare them, AI changes nothing at all about this.
> Only someone with little appreciation of music will describe the difference between an actual performance and an AI generated one as "human-level BS".
Unfortunately, this seems to apply to quite a few people on HN, and for all types of art in general. I fear this worldview (or maybe the common root cause) is what leads to the all too prevalent lack of care or outright disdain for people and society in general we're seeing from the tech industry (amongst others) these days.
And you're telling me that as long as you believe a human struggled and went through a process, the nature of the outcome is irrelevant. Hence a Monet piece is instantly slop if one believes an AI created it. That rings true. But it also explains the perception that Piss Christ was art instead of rubbish.
We just disagree as to whether an AI can write a script for a human to portray a struggle that never happened and that a human actor can make you believe it. This almost demands a performance art piece to do exactly that. The art of course being in the performance by a human until it can be replaced by an indistinguishable robot. And then the artist becomes the robot's creator I guess. Why it's creators all the down, no?
I guess we also disagree about all these lines in the sand you keep drawing when it's seemingly entirely subjective experience of a world that may or may not exist according to Plato's Cave.
And sure, I love concerts, I love live performances, but I also love to listen to a much wider range of music when I'm not in a theatre. I can understand that you might feel differently, but we are all entitled to our own opinions and tastes. And what we do in our own lives if it isn't hurting others anymore than anyone else in the west is doing with their egregious carbon footprints is none of your business.
And I don't draw such lines in the sand. I am saying an experience that would be perceived as authentic human communication can be artificially generated until proven otherwise. Good luck with that.
I really wonder if humanity will be able to process alien life when it finally encounters it. Because it's likely to have a very different origin story, nature, and outlook.
Yeah, what you're saying is true but only to the extent of simulacra AI can produce. But it's not really art if you know what I mean, it's an artefact of training on other people's art.
So what I do with AI art is start with a photo I took. I then extract its Lin Canny lines or other descriptive features algorithmically. From there, I apply various models to re-express the image in an interesting way. IMO it looks more like Digital Art out of photoshop and I would refute any claim that this isn't remotely artistic given I was the one that took the photo that is the base image. And it doesn't end up looking like obvious AI art, but rather more Andy Warholish than anything else IMO.
And sure, the model was trained on other people's art but so are we.
It may not be your cup of tea, but it's a lot more effort than typing in a prompt and calling it a day. But, ya know, Rembrandt and Hans Zimmer were the OG prompt engineers in their respective media IMO.
The day spotify wrapped dropped, I found out accidentally that they had used Rive as their team was happy to share this on X.
At the same time I was going through it trying to implement a handful of Rive animations in our app built with React Native, with not so stellar results and a taste in my mouth so bad that it inspired me to write about it.
Mostly to discourage other small teams(with a solo dev like us) saving their roadmaps in process and to share how I intend to tackle animations in the future.
Since last year i've been working on something that surprisingly didn't exist before, an app where you can rate sports games.
https://rategame.io/ and on the app stores(I really recommend the app if you want to check it out, especially on a phone)
We've expanded on the concept with rating stadiums, creating lists, voting for player of the game and more as we are trying to become the letterboxd for sports.
wow.. this is cool. what's the endgame? I built a site that logs my lifetime in person attendance along with statcast tracking for homeruns, launch angle and more. wonder if theres some crossover
https://baseball.willbaxter.info/
The endgame is to become the ultimate source of truth regarding the quality of a sports game.
Dreaming big with hopes of becoming a household name similar to imdb or letterboxd.
Looking at your site I think you would like to hear that when rating games you can mark a game as 'attended', 'watched' or other categories to show others how you experienced the game or just for your own personal log.
Other than that we are light on classic sports stats and our stats are more focused on the user ratings of the games rather than the game stats.
We have been using Meilisearch with firebase for years and it has always worked great.
I just wish they would update the extension on the firebase extensions hub[1] because the current version available uses node 14 which is not supported by cloud functions on GCP so the extension is not usable at all.
What's weird is that the latest version available on their repo has upgraded the node version but they are not offering it in the extensions hub.
However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.
Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.
However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.
I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.
Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.
I still haven't found a way to deal with rotating residential IPs however, and most likely never will.
My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.
Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?