Under attack from ByteSpider bots from Singapore
Hi, I have a small Ghost blog at https://ghost-tech.up.railway.app/ that's serving the blog for https://techinterview.coach/blog (through a Netlify redirect).
I am under attack from a ByteSpider crawler from Singapore, that's causing my Cloudinary usage (and bills) to spike. I am trying to block them. So far:
- I have put https://techinterview.coach/ behind Cloudflare, but I think bots are hitting the railway.app domain directly so I am not sure this will help.
- I've modified the https://ghost-tech.up.railway.app/robots.txt to disallow crawling.
Can I do anything else? Is there a way to put the railway app domain itself behind CloudFlare?
Thanks,
Andrea
13 Replies
Project ID:
3e1d8e5a-85d2-453c-ae03-c5983d51b5e03e1d8e5a-85d2-453c-ae03-c5983d51b5e0
you can change the railway domain
As brody suggested change the domain.
you can also block any traffic that isn't going through cloudflare, just validates if the origin IP is from Cloudflare: https://www.cloudflare.com/ips/
Also, you can probably block their user agent
https://stackoverflow.com/questions/57908900/what-is-the-bytespider-user-agent
Here's a guide for doing that in Cloudflare: https://developers.cloudflare.com/waf/tools/user-agent-blocking/
Amazing - thank you! How do I validate that the IPs are from Cloudflare?
I will change the domain in the meantime, but keen to learn alternatives in case they find the new one...
that depends entirely on what kind of app you are running
maybe there's a middleware available that will only respond to cloudflare ip's
I am running a fork of this https://github.com/railwayapp-templates/ghost
GitHub
GitHub - railwayapp-templates/ghost: A self-hosted version of Ghost...
A self-hosted version of Ghost with a MySQL database. - GitHub - railwayapp-templates/ghost: A self-hosted version of Ghost with a MySQL database.
btw, looks like the robots.txt and putting https://techinterview.coach behind cloudflare solved the problem... can't tell which of the two though...
traffic before
traffic after
62.7 thousand
holy moly
I know, brutal...
maybe you can modify your spam options?
https://github.com/TryGhost/Ghost/blob/c667620d8f2e32c96fe376ad0f3dabc79488532a/ghost/core/core/shared/config/defaults.json#L61