RunPod•3w ago

Are webhooks fired from Digital Ocean?

I setup a WAF in AWS to block bots and I am getting a bunch of requests to my RunPod Serverless Webhook blocked by AWS#AWSManagedRulesBotControlRuleSet#SignalKnownBotDataCenter . The IP address in these requests seems to be a Digital Ocean Data Center. I have disabled the WAF for my ALB for my RunPod webhooks temporarily, but hoping that someone can confirm whether these are legitimate requests or not, because I was under the impression that RunPod uses AWS and not Digital Ocean.

17 Replies

nerdylive•3w ago

Yeah they can use multiple clouds @flash-singh can you confirm this

digigoblin•3w ago

Obviously, but the question is DO they? 🙂

nerdylive•3w ago

Wait ya, maybe support staffs can confirm im not sure

flash-singh•3w ago

yes we use DO, AWS and Cloudflare

digigoblin•3w ago

Thanks for confirming, do you just have 2 IPs in DO or multiple?

nerdylive•3w ago

Maybe you can give the IP range for whitelisting

digigoblin•3w ago

I just removed my webhook load balancer from the WAF and added an auth token to the webhook receiver

flash-singh•2d ago

many different ips, we don't recommend whitelisting them since they can and will change over time we can introduce more security into the http request if it can help, rather avoid using ip filtering as a security measure since those change

digigoblin•2d ago

Yeah it will be nice to add an auth token to the headers rather than in the query string as I am currently doing

flash-singh•2d ago

why is that? something you can use to filter at load balancer level?

digigoblin•2d ago

Just feels more secure putting the auth token in the header rather than in the query string. Do you use a specific user agent string for the requests or just the default of the library you use? Because then I can filter for user agent string at WAF level before it reaches the load balancer.

flash-singh•2d ago

we can define specific one, right now its whatever is default in golang, can also define auth header

digigoblin•2d ago

Thanks, this is useful indo, and auth header will be amazing 🫶

Arjun•2d ago

@flash-singh @digigoblin Funny I was just implementing webhooks on our project today and the exact subject came up for us too. Putting auth tokens in a query param can be insecure and make the token highly visible if there are logs being generated within RunPod that log the url. Adding as a header: Authorization: Bearer <token> pattern would be standard and more secure. This could either be set on the endpoint config in the console, or how it currently is in the initial run post:

{
  input: {...},
  webhook: {url: 'https...', auth_token: 'your auth token'}
}

{
  input: {...},
  webhook: {url: 'https...', auth_token: 'your auth token'}
}

~~Even better if we could add additional body params to the webhook:~~ Scrap this idea

{
  input: {...},
  webhook: {
    url: 'https...',
    auth_token: 'your auth token'
    params: {
      custom_identifier_1: '',
      custom_identifier_2: '',
      ...
    }
  }
}

{
  input: {...},
  webhook: {
    url: 'https...',
    auth_token: 'your auth token'
    params: {
      custom_identifier_1: '',
      custom_identifier_2: '',
      ...
    }
  }
}

Given the current limitations, another way we were thinking we could do is just encrypt a single state param that contains a hash of a few things (incl. auth token).

nerdylive•2d ago

Or maybe just headers field that we can customize, with some examples to use.. Like for Auth headers, and req. bodies are kind of useless here right?

Arjun•2d ago

@nerdylive True, a headers field would be the most versatile and simplest. Anything else might get muddied with the webhook post body.

flash-singh•22h ago

thanks for the input, will dive into this more next week and how we can improve it

Gaming

Programming

Are webhooks fired from Digital Ocean?