CF blocking RSS bot from email service provider despite custom rule
Hey, so I have an RSS automation in my ConvertKit email service provider account that automatically pulls new blog posts and sends them out via email. Since switching to CF, this automation has broken.
I have spent hours trying everything to fix it. IP allow rules on my server. IP allow rules on CF. I checked all the boxes with a skip rule. I checked only some. I tried allowing just the user agent. Nothing works.
I'm not sure why it's so hard to tell CF to back off a list of 3 IP addresses or a certain bot without disabling Bot Fight Mode altogether. If anybody could please help me, I'd be very grateful.
Thanks!
23 Replies
I'm not sure why it's so hard to tell CF to back off a list of 3 IP addresses or a certain bot without disabling Bot Fight Mode altogether. If anybody could please help me, I'd be very grateful.The answer is very simple but not good news: Bot Fight Mode (the free version, not Super Bot Fight Mode which pro or above have) is not configurable/skippable at all, by anything. It's either on, or off. Custom Rules can only skip Super Bot Fight Mode (fwiw they're both options that are recommended only for use while under attack/and are "high security" options)
Oh, so I have to turn off BFM and that's the only way? But that should also be safe? It's not a super high traffic site anyway.
Oh, so I have to turn off BFM and that's the only way?Yes
But that should also be safe?"Safe" is relative, but BFM is a really heavy handed solution and going to make normal visitors/etc go through challenges occasionally, etc. Just not necessary unless you're constantly under attack and you can't better fight it off with more targeted custom rules
Hmm, I have Ninja Firewall and some custom rules
So if I have some level of protection via these, is that usually enough for a "normal" site like mine? I get less than 10k visits/mo
probably way overkill if anything. With small sites I wouldn't worry as much about attacks as I would about keeping stuff up to date/not getting hit by some big vuln that people scan the web for
blocking non-standard ports is also a good idea if you don't use them, ex:
not cf.edge.server_port in {80 443}Oh, okay, thank you! So I could add this blocking non-standard ports rule instead of the custom one I have now trying to fix the RSS and turn off BFM?
that'd work
Amazing!
Oh, is there a way to verify if I'm using those standard ports?
Just so I don't accidentally kill anything that's being used.
when you visit your website, do you ever manually specify an alternative port, for example:
https://example.com:8443 (cf only supports a few like 8443, 5443, etc)
if you don't, and you just type http:// and https:// that's just 80/443 respectivelyOh, no, I don't! Ahh, so if people just use the basic http/s then they are using only those 2 ports? Wow, amazing, I didn't know. Learned something!
Okay, should I paste the expresseion under hostname? Or what type of rule should I make this?
Ah nvm I just edited the expression directly
ah sorry yea just edit expression
ok yeah, got it I think!
this is the first rule
and it just looks like this
yup looks good, yea you'll get some amount of random bots/crawlers trying those alt ports. It makes connections to your origin which either fail, or work (if bound to them) and could let them bypass cache
Oh, okay, but the "good" bots like Google search crawler etc., those use the normal ports as well?
yea
google can index non-standard ports (if you had an app on them) but it's not going to do so unless something links to it/it has a reason to
Makes sense
okay, now just turning off BFM to see if the automation works
awww hell yeah, it works!! Thank you so much!!!
lifesaver @Chaika 🙏
I spent so much time on this lol
really made my day!
really appreciate your help, thank you!
no problem, yea sadly this isn't the most obvious and people miss the "Super" part from the bot fight mode rule skip