Block what I think are headless requests?
Hello! I am trying to block someone currently spamming an API post endpoint for one of my webistes, but trying to do it in a way that isnt just IP banning that one user. He is somehow making 1000s of requests to my site despite my rate limit of 25 requests per 10 seconds. I have checked the local real time logs and there is no user agent listed, so I am thinking this is some sort of headless scan. I did some googling and I cant find out how to block headless requests with WAF. I am on the Free tier as well.
Here is an example of one of the real time logs for better help, I have removed all identifying information.
51 Replies
http.user_agent eq ""Will CF not register a user agent in the logs if it
eq ""what do you mean?
I added a traffic log above. There is no listed useragent, so will cloudflare not log that as an empty value in the traffic log if its a headless/agentless request? Just trying to better understand as I am new to this prodct 🙂
cloudflare will log empty useragents
or maybe i misunderstood you
So for the above request, what method should I implement to block? I do think I could add a function to my cloudflare function for that page to only allow traffic from a referrer of that page.
From WAF
headers can be spoofed (...), if all of the requests spamming your API don't have a UA then just add a WAF rule to block all requests without a UA
but i'm curious about how they are circumventing your ratelimit rules, if they truly all come from the same origin IP
So they might be using curl or grep, i dont remember if they have a default header now. on the free plan you should be able to block it with this rule
Just added that to my bulk rule
Lol jeez, this is last 10 mins
want me to test it for you?
i can curl it with an explicitly blank header
lmao one sec
got challenged with useragent
Mozilla/5.0 (X11; Linux x86_64; rv:134.0) Gecko/20100101 Firefox/134.0
Yeah I have a managed challenge rule set as well. But that didnt stop whatever he was using.
Useragent null
seems fine
(BTW ray ID is hidden for me by https://github.com/pabli24/CloudFlareError)
do your rules apply to /passcode?
(they do)
the ray id is non identifying
not quite sure what purpose this serves
i know, also hides ip just the same userstyle
I have applied the rule globally.
is this for game ARG or something?
its already hidden by default
the button bugs me, (yes i know, its a non-issue)
Any of yall have any good block lists for things like this? Selenium stealth, etc etc
blocklist, wdym?
Well, just a list of block expressions that will cover a lot of bases
These.
there's no cure-it-all for this
you adapt your waf rules to the traffic you receive
this
Oh I know that for sure, id have to monitor logs and do my own, im just looking for a place to start
i guess if it becomes too much of an issue you can block their IP (and for the nuclear option ASN not a great idea tho)
my guess someone is trying to brute the code on the page
Also, this is a sidenote, how are you guys getting more accurate real time logs out of cloudflare? And haha yeah I am trying to simply discourage bruteforcing of a key. 🙂
a nice place to start would be to just challenge (managed) requests that don't have a nearly perfect threat score.
(i do not recommend this for production)
then filter your WAF for that rule
look at the WAF logs that you get, and try to find patterns in requests that seem suspicious to you.
as you define your own rules to block requests you can decrease the threat sensitivity of your challenging rule, rinse and repeat.
(this is just my personal strategy, this is not recommended anywhere)
would also recommend the cf guide on ddos mitigation
?ddos
If you are under a DDoS attack then you can take a look at these threads for first steps and help with mitigation:
- https://community.cloudflare.com/t/under-ddos-attack-first-steps/89476
- https://community.cloudflare.com/t/mitigating-an-http-ddos-attack-manually-with-cloudflare/302366
side note: pro (and above) provide a lot more information both in waf logs, and configuration options for your WAF...
I might just have to suck it up and go pro lol. Was trying to avoid the expense, but the cdn features would be nice
if your keys are generated sufficiently secure, all you have to do is enforce rate limits
The issue with that may lay in how I coded the page, each button press seemingly acts as a request. I may have to fine tune that to capture a request within a certain time period and send it as a batch request to the server.
if you are referencing the passcode lock for your site, i would highly recommend you look into zero trust
which essentially does what you have implemented right now, except much more convenient and secure
(and free)
you can impliment that in the site side too, make it so after like 10 tries in 20 seconds or something like that it kicks an error with a different response syntax some bruteforce systems dont like output syntax changes too much
or better yet as i did on one of mine, add red herrings, that go to a page that says something like "if you are a human, this is the wrong code, you entered a wrong code that is used to prevent bruteforcing, There are no ARG/Lore secrets on this page, please go back"
This is all a CTF style game, so its all just for fun. Nothing commercial, just trying to prevent threat actors and evil-doers type of deal
Hmm, not sure how id pull that one off. Its a bad idea to run an arg on a static webpage anyway aha
thats true
My experience with WAFs is all commercial coorporate stuff, so first time screwing with something like Cloudflare, kinda fun, that whole cat and mouse sort of deal.
Yall have been great, only thing im wondering is if I can export logs to some free python tool or something. I set up KV logging via a cloudflare function to track total requests.
Feel free to drop a screenshot of that setup btw, that sounds interesting. I may buy premium or w/e because the 10 logs at a time for free kinda stinks.