za_creature

> How do you even implement an ua check?

For this purpose, it looks like this:

  if 'googlebot' in request.headers.get('user-agent', 'googlebot'):
      return 403

Eh, it's not as clear cut as you make it to be.

For one, you can make PoW progressive based on the number of requests you've encountered from the same IP address. This does hurt users behind NAT so it's not ideal, but it might be time we moved past NAT.

Secondly, 5 seconds is a lot but not unheard of; you could start PoW in the first lines of your html and load the rest of the assets while you're otherwise busy burning cpu time. 20ms quickly adds up at the scale scrapers operate. Which, according to cloudflare is > 50% of web requests.

All of the above fall under "very little". We're talking in the ballpark of 5% extra sales max, at the expense of double the infrastructure costs.

They're also stealing your curated dataset: if you take the time to correctly label your products, add specifications and manuals because you want this to be a competitive advantage, your work was just re-appropriated for the benefit of humanity. But primarily for your larger competitors willing to pay for it.

> You cannot determine intent by looking at an HTTP request.

I believe this is the crux of our disagreement. Clearly you can't, which is why you must assume good faith and service all innocent-looking requests.

What you can do however is temporarily log and search for patterns in multiple HTTP requests. If you can then prove that these requests came from a bot via the plethora of fingerprinting measures already in use, you have grounds to make a complaint to the FTC, which would have the ability to subpoena the identity of the offending entity and, depending on their history, issue a warning, a fine or file a lawsuit.

The current status quo is that attackers have an advantage in that the worst thing that you can do to them is to temporarily block them, and you don't really want to do this because you're never 100% sure whether any _individual_ request is automated.

While I can't say that this law is good for this purpose, I am pointing out that the solution looks something like it. The goal is not blocking the individual, but doing something about infra-level abusers, at a volume where 95% accuracy is sufficient proof of wrongdoing. All of your arguments are valid and would be tried in court in response to this passing, hopefully for the better but ultimately it depends on how good the justice system is. This last thing is something no amount of technology can fix but in that case illegal scraping is the least of our problems.

> you're proposing a law that says anybody with an inaccurate user agent string is in violation

No, I am proposing that if googlebot is not within that inaccurate query string, google is in violation.

> Sites could then try to either blacklist or whitelist user agents.

Yes, that is a desired outcome.

> If they do the first one, that doesn't work, because the scraping service can change it whenever they want.

That would put them in violation, and make them prosecutable. Of course you need to start collecting data to prove that it's automatic, but this is already happening today for the purpose of temporarily blocking an IP. Having this law means that you also have some punitive measures at the end of the rainbow.

> If they do the second one ...

Anyone pretending to be a common browser when they're not is in violation. In addition, under this law there's no incentive for anyone to pretend to be googlebot since _that_ can get them blocked. You are intentionally and repeatedly trying to obfuscate these facts.

There is, this entire thread is peanuts in the grand scheme of things. The real problem is whether it'll be possible to write to the web without some form of ID, and there's some good news there too:

There's also always going to be an anonymous unmoderated "dark web" where everyone and their LLM can shitpost all day long, it's just questionable how many humans are involved in it. It's also broadly irrelevant since the important messages will eventually be reposted by "legitimate" users on their moderated channels for the "normies" to read.

You can get pretty far with messages signed by certificates issued by authorities that check one-another. You can go further with a gossip protocol on how many bytes have been signed by each certificate on each website.

It's just an unfortunate reality that people with disabilities live with.

Some of them are already prevented from operating heavy machinery because doing so would put everyone at risk, so there is precedent here. I'm in favor of making concessions to the very unfortunate, though at some point you have to look at percentages, expected impact and decide what is reasonable and what isn't.

To your other question, in the last week there have been entire CPUs embedded within weights. Clearly it's possible to publish "open models" with an agenda so then what assurances you have that the enhancements provided by local ai would be more focused on the plot than what the big browsers do?

Specifically how would a person with disabilities be able to tell the difference between improved instrumentation and conditional access to information, as we have in other media? Consider this in the context of "compared to superintelligence, we all have an intelectual disability"

As a human being,

"stealth" means that it's not trivial for other people to block you from crawling their shit and

"damage, impair or burden" is in the eye of the beholder, the above person trying to block you from crawling their shit

Yes there will be patent-trolls attempting to squeeze this. Which is why it would be in the crawling entities' best interest to provide electronic means to opt-out and abide by them.

> this would necessarily target the people who pay for election campaigns, so it will never happen.

statesmen (and women) do the right thing in the end, but only after trying every other option

User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.

That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.

As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.

> All of the code in a browser is "automation"

I don't think you are interpreting what I said in good faith, but here goes:

If you use an API to control a browser into navigating to a certain webpage, as opposed to using human interface devices to assert said control.

And no, AI is not a human interface device.

The compromised smart tvs in your country were manufactured by someone that could be held liable for operating a "stealth" bot network, or for allowing someone to do so through negligence.

As for journalists, nothing is stopping their legal department from petitioning a judge to issue a temporarily exemption from the law for a specific period of time under a specific written purpose.

A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.

You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.

Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.

A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.

A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.