I recently stood up a wiki for a project I do computer-y stuff for (Shousetsu Bang*Bang, one of the longest running NSFW queer webzines, so thank them for my blocklist if you’re inclined) and, like pretty much every other MediaWiki instance stood up on the internet this is getting absolutely pestered with AI scraper bots. It’s not crippling amounts of traffic, but it’s ridiculous and unnecessary amounts and frankly I am completely done with all these people. (Does facebook need to hit a wiki with a thousand or so pages 400k times in a day? No. No it does not)
To not bury the lede too much, there’s a link to my htaccess file you can snag that blocks a lot of these crappy things.
The initial way we’re told to deal with them is to just add entries into robots.txt. Since this is possibly the URL hit the least on the site I’d say Press X to Doubt. Additionally a number of these crawlers are actively lying about themselves, with bogus referers and user agent strings and I think it’s safe to assume that these choads wouldn’t pay attention even if they did fetch the file.
So! Time for a blockin’ on these things. The wiki is user facing, so all the traffic we care about comes from people and places people’s devices are — that is, residential IPs and mobile carriers. While I don’t have root access on the server so I can’t black hole this crap at the network layer, it is running a fairly well appointed Apache2 setup, and that gives us some options for blocking at the app level. The crawlers, while badly written, do at least recognize 404 errors and give up after a while.
Data center based crawlers (and other not totally skeevy crawlers)
The first block set are the crawlers that are abominably incompetently written (such as Facebook and Amazon’s crawlers, whoever did these honestly should quit IT and do something more in line with their skillset and talents. Telephone sanitizers, perhaps) but still identify themselves. Pulling out IP addresses from the logs on them is pretty trivial, and a quick run through some ICANN based tooling and we can turn the IP addresses into their CIDRs. Apache lets us block by both individual IP address or by entire CIDR, so… we block these by CIDR. Yes, this means that the wiki rejects all traffic from these datacenters, but we’re user facing, so fuck these guts.
The second block set are the crawlers that don’t identify themselves but still come from data center IPs. Alibaba’s crawler, in particular, is in this set — those assholes lie about their referer and their user agent, but there’s no lying about where you come from. Filtering the logs to find these is pretty simple, we just extract out all the source IP addresses, chop off the bottom byte (all the data centers have at least a /24 for their ranges, most have something like a /15 or /16, and AWS even has a /9) and count the repetitions. From there it’s pretty obvious, and we can snag a full IP address and grab its real CIDR and block that.
The third block set are high traffic individual IP addresses coming from a business block. (There are a few crawlers that identify themselves that do this) I’m somewhat loathe to block a whole CIDR when traffic is only coming from a single IP address, so these go into the list all on their own. Woo!
The net result of all of this is a not too large .htaccess file. You’re welcome to grab it, a recent version is on my server. You’re welcome to it, I’ll work on keeping this up to date but no promises. If you’re not running Apache you can still snag it and extract out the addresses from it.
Skeevy crawlers
Past that you come to the skeevy crawlers. These are the ones that come from all over, presumably using one of the many residential botnets out there for rent. (One more reason to never plug your smart TV into any network) Unfortunately there are a lot of these, and blocking them via your .htaccess file isn’t an option. (It’s read every request, so I wouldn’t want to put 50k IP addresses in it) Still! It’s worth characterizing these, and I suspect I’ll be switching from Apache to Nginx at some point to block them.
The first of these skeevy crawlers have a referer of just the site root. I assume they’re trying to pretend some kind of security thing is happening and the referring URL is being scrubbed just leaving the originating site, but that is just not how it works, and definitely not how it works for tens of thousands of different originating sites that also haven’t actually fetched anything else. (That referer would be a lot more believable if there was an earlier fetch of something, but these are all one fetch per IP so clearly they’re just lying)
The second of these skeevy crawlers have a referer that’s identical to the URL being fetched. That’s also not how things work for the wiki, so these are also blockable.
Sadly with Apache there’s no easy way (unless you have access to the base config, which I don’t because managed hosting) to block these that I’m aware of, though I’m definitely going to work on something which probably involves Nginx (which my hosting provider actually allows more config access to, so I can probably do something) and maybe some kind of cheap cleverness if I can write Apache htaccess things that can cheaply match the two different characteristics above.
Regardless, definitely making a list of these IP addresses. The current plan is to have traffic from them redirect to a static “you’re hosting a botnet node so you’ve been blocked” page on the very unlikely off-chance that someone actually comes to the site from one of these IP addresses. Which I don’t expect, but you never know. (There’ll probably be a 30 or 60 day expiry on an entry unlike for datacenters, residential botnets might die but datacenters are forever-ish)