justAnotherHero

While nowhere near compared to their scale, I run a consumer app where most of our users are using the mobile app, with the web app getting perhaps 10-15% of the mobile active users.

However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.

Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.

However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.

I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.

Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.

I still haven't found a way to deal with rotating residential IPs however, and most likely never will.

My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.

Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?

Had the same experience with upwork, feels like the door has closed on easy api access to big site data, ironically funneling people's money to proxies and other scrape helpers instead of their own api.

Are you using the reddit api or scraping new reddit posts/replies?

To me it's a useful signal not to read an article that someone didn't bother to write.

Which is a shame as real insights are buried inside some of these articles, which if the author bothered to write in his own words could have reached an audience that would have appreciated them.

Writing is one of the areas where I want no LLM involvement.

That's great but I really wish you guys would do something about the llm integration, I tried using it two days ago to create a cohort of users using a sql query, and I was surprised to see that it said that it could not create cohorts for me and i had to resort to exporting data from a sql insight as a cohort cannot use a sql query. However the worst part was it just writing in the text input slowed down my m4 pro chip to less than 1 fps after 2 prompts and it really left a bad taste in my mouth.

Perhaps the next target for a 100x improvement

I'm still on their old 500 requests/month plan and the value is simply unbeatable for $20. I've been able to use agents without worrying about usage for my job and personal projects paired with the $20 codex sub and I dread the day when they finally get rid of the requests plan.

Deploying software has been reduced to just running 'vercel' in the terminal, something the agent has zero problems with if they were to just ask. Distributing desktop software is a bit harder depending on the platform.

The gap between a pet project and great software is still very wide and I have a hard time believing that it will ever be bridged.

I don't see how a solved problem even before ai is the thing that won't be replaced first. I struggle to believe a personal project requires complex infra

I've forked it so that I can see the old cursor 500 requests plan and haven't merged any of their changes after that for the last few months.

Seems to me the way to go with these apps that keep pushing daily updates.

The day spotify wrapped dropped, I found out accidentally that they had used Rive as their team was happy to share this on X.

At the same time I was going through it trying to implement a handful of Rive animations in our app built with React Native, with not so stellar results and a taste in my mouth so bad that it inspired me to write about it.

Mostly to discourage other small teams(with a solo dev like us) saving their roadmaps in process and to share how I intend to tackle animations in the future.

The endgame is to become the ultimate source of truth regarding the quality of a sports game.

Dreaming big with hopes of becoming a household name similar to imdb or letterboxd.

Looking at your site I think you would like to hear that when rating games you can mark a game as 'attended', 'watched' or other categories to show others how you experienced the game or just for your own personal log.

Other than that we are light on classic sports stats and our stats are more focused on the user ratings of the games rather than the game stats.

I'm from Skopje myself and I see this building often as I bike to work or go anywhere near the center.

As you get closer to it, the post office stands out and the tower is the one thing that always catches my attention in a sea of blandness (not to complain as these old apartment blocks are nice to live in, I live in one myself).

The windows become orange and it feels like a villain is plotting his world domination somewhere up there. The type of villain whose plans never succeed and his tower is beginning to crumble.