rarias/jungle
2
0
Fork 3
Code Issues 63 Pull Requests 10 Actions Packages Projects Releases Wiki Activity

Ban LLM crawler bots #228

New Issue
Open
opened 2026-02-20 11:26:18 +01:00 by rarias · 0 comments
rarias commented 2026-02-20 11:26:18 +01:00
Owner
Copy Link

They seem to have gotten way worse in the last weeks. It seems we can just match the User-Agent for the worst part of the DDoS, the rest indentify as Chrome and use residential IP blocks (likely from residential proxies).

tent% sudo ls -lhv /var/log/nginx/ | head
total 4,5G
-rw-r----- 1 nginx nginx 2,2G Feb 20 11:09 access.log
-rw-r----- 1 nginx nginx 1,1G Feb 15 00:00 access.log.1
-rw-r----- 1 nginx nginx  79M Feb  8 00:00 access.log.2.gz
-rw-r----- 1 nginx nginx  71M Feb  1 00:00 access.log.3.gz
-rw-r----- 1 nginx nginx  78M Jan 25 00:00 access.log.4.gz
-rw-r----- 1 nginx nginx  50M Jan 18 00:00 access.log.5.gz
-rw-r----- 1 nginx nginx  29M Jan 11 00:00 access.log.6.gz
-rw-r----- 1 nginx nginx  33M Jan  4 00:00 access.log.7.gz
-rw-r----- 1 nginx nginx  53M Dec 28 00:00 access.log.8.gz

tent% sudo head /var/log/nginx/access.log
216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/rarias/jungle-website/src/commit/afcb594775ecf37df25fb721fc200cc04df616f9/themes/PaperMod/layouts/_default/baseof.html?display=source HTTP/1.1" 200 41928 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/varcila/jungle/rss/commit/025f82cbcd753c59e2734481322116d45270fde0/pkgs/vite/cmake.patch HTTP/1.1" 404 11 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
105.159.151.145 - - [15/Feb/2026:00:00:28 +0100] "GET /git/git/rarias/bscpkgs/src/commit/dbdcfea01904fed21381f4d8546caaa1cb688bfe/garlic/stages/exec.nix HTTP/1.1" 404 9183 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/arocanon/bscpkgs/src/commit/e84e961b9fec3879c4abbea0adaa7270961936f9/pkgs/mcxx/git.nix HTTP/1.1" 200 49719 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
74.7.227.159 - - [15/Feb/2026:00:00:28 +0100] "GET /git/arocanon/jungle/commits/commit/834d3187e5a832d977827d3b15a0e468ddfe8b34/m/common/rev.nix HTTP/1.1" 200 25085 "https://jungle.bsc.es/git/arocanon/jungle/src/commit/834d3187e5a832d977827d3b15a0e468ddfe8b34/m/common/rev.nix" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"
216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/arocanon/bscpkgs/src/commit/76ddd85afeac894937824fe44f47d35315ec9e45/pkgs/nosv/default.nix?display=source HTTP/1.1" 200 53620 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/varcila/bscpkgs/src/commit/bf28263cc5959bdff4978b6838c33ec097cdbdf0/garlic/doc/ug.ms?display=source HTTP/1.1" 200 446168 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
84.88.53.236 - - [15/Feb/2026:00:00:28 +0100] "POST /git/api/actions/runner.v1.RunnerService/FetchTask HTTP/1.1" 200 3 "-" "connect-go/1.16.2 (go1.25.5)"
216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/abonerib/bscpkgs/src/commit/2d1f1ee5c71b52c5e765a781d48a2553031b1c45/pkgs/ovni HTTP/1.1" 200 30469 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/rarias/bscpkgs/commits/commit/7995ca0a9918bf18615dcc43278b124f437df998/pkgs/intel-compiler HTTP/1.1" 200 28775 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
They seem to have gotten way worse in the last weeks. It seems we can just match the User-Agent for the worst part of the DDoS, the rest indentify as Chrome and use residential IP blocks (likely from residential proxies). ``` tent% sudo ls -lhv /var/log/nginx/ | head total 4,5G -rw-r----- 1 nginx nginx 2,2G Feb 20 11:09 access.log -rw-r----- 1 nginx nginx 1,1G Feb 15 00:00 access.log.1 -rw-r----- 1 nginx nginx 79M Feb 8 00:00 access.log.2.gz -rw-r----- 1 nginx nginx 71M Feb 1 00:00 access.log.3.gz -rw-r----- 1 nginx nginx 78M Jan 25 00:00 access.log.4.gz -rw-r----- 1 nginx nginx 50M Jan 18 00:00 access.log.5.gz -rw-r----- 1 nginx nginx 29M Jan 11 00:00 access.log.6.gz -rw-r----- 1 nginx nginx 33M Jan 4 00:00 access.log.7.gz -rw-r----- 1 nginx nginx 53M Dec 28 00:00 access.log.8.gz tent% sudo head /var/log/nginx/access.log 216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/rarias/jungle-website/src/commit/afcb594775ecf37df25fb721fc200cc04df616f9/themes/PaperMod/layouts/_default/baseof.html?display=source HTTP/1.1" 200 41928 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" 216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/varcila/jungle/rss/commit/025f82cbcd753c59e2734481322116d45270fde0/pkgs/vite/cmake.patch HTTP/1.1" 404 11 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" 105.159.151.145 - - [15/Feb/2026:00:00:28 +0100] "GET /git/git/rarias/bscpkgs/src/commit/dbdcfea01904fed21381f4d8546caaa1cb688bfe/garlic/stages/exec.nix HTTP/1.1" 404 9183 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/arocanon/bscpkgs/src/commit/e84e961b9fec3879c4abbea0adaa7270961936f9/pkgs/mcxx/git.nix HTTP/1.1" 200 49719 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" 74.7.227.159 - - [15/Feb/2026:00:00:28 +0100] "GET /git/arocanon/jungle/commits/commit/834d3187e5a832d977827d3b15a0e468ddfe8b34/m/common/rev.nix HTTP/1.1" 200 25085 "https://jungle.bsc.es/git/arocanon/jungle/src/commit/834d3187e5a832d977827d3b15a0e468ddfe8b34/m/common/rev.nix" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)" 216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/arocanon/bscpkgs/src/commit/76ddd85afeac894937824fe44f47d35315ec9e45/pkgs/nosv/default.nix?display=source HTTP/1.1" 200 53620 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" 216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/varcila/bscpkgs/src/commit/bf28263cc5959bdff4978b6838c33ec097cdbdf0/garlic/doc/ug.ms?display=source HTTP/1.1" 200 446168 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" 84.88.53.236 - - [15/Feb/2026:00:00:28 +0100] "POST /git/api/actions/runner.v1.RunnerService/FetchTask HTTP/1.1" 200 3 "-" "connect-go/1.16.2 (go1.25.5)" 216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/abonerib/bscpkgs/src/commit/2d1f1ee5c71b52c5e765a781d48a2553031b1c45/pkgs/ovni HTTP/1.1" 200 30469 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" 216.73.216.128 - - [15/Feb/2026:00:00:28 +0100] "GET /git/rarias/bscpkgs/commits/commit/7995ca0a9918bf18615dcc43278b124f437df998/pkgs/intel-compiler HTTP/1.1" 200 28775 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" ```
rarias added the netops labels 2026-02-20 11:26:27 +01:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
ceph
ci
config
cross
hw
io
kernel
mpi
net
nix
ops
repair
slurm
No Label net ops
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
abonerib anavarro arocanon varcila rarias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rarias/jungle#228
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?