Software Mansion•3mo ago

Fly.io + UDP

I've got an membrane_webrtc server setup where someone can "call" an LLM and talk with them (audio only, no video). It largely works, though my users are reporting random disconnects. The console errors match the attached image. I'm a little thrown since the url in that message specifies UDP as the transport. I deployed to fly.io, and explicitly did not open up the UDP ports in my fly.toml, so I'm wondering why the app is failing with a UDP timeout. Am I incorrect in assuming that I can force all traffic over TCP by just not opening it up? Shoudl I also figure out UDP? On UDP, I read through this: https://github.com/fishjam-dev/fishjam-docs/blob/main/docs/deploying/fly_io.md But it's Fishjam specific, and it doesn't line up neatly with my app which is based on the old membrane video room repo (https://github.com/membraneframework-labs/membrane_videoroom). Where does fly-global-services get specified in that case? I'm not explicitly setting a TURN_LISTEN_IP. I traced through things I think it could be here in turn_ip (and then the turn_mock_ip is my external IPv4 address)

    turn_mock_ip = Application.fetch_env!(:smartvox, :integrated_turn_ip)
    turn_ip = 'fly-global-services'
    #turn_ip = {0, 0, 0, 0} #prior value

    integrated_turn_options = [
      ip: turn_ip,
      mock_ip: turn_mock_ip,
      ports_range: Application.fetch_env!(:smartvox, :integrated_turn_port_range),
      cert_file: turn_cert_file
    ]

    turn_mock_ip = Application.fetch_env!(:smartvox, :integrated_turn_ip)
    turn_ip = 'fly-global-services'
    #turn_ip = {0, 0, 0, 0} #prior value

    integrated_turn_options = [
      ip: turn_ip,
      mock_ip: turn_mock_ip,
      ports_range: Application.fetch_env!(:smartvox, :integrated_turn_port_range),
      cert_file: turn_cert_file
    ]

Which is in my room.ex -- those options are used here:

TURNManager.ensure_tcp_turn_launched(integrated_turn_options, port: tcp_turn_port)

TURNManager.ensure_tcp_turn_launched(integrated_turn_options, port: tcp_turn_port)

But that throws an error when I override turn_ip that way. So, (1) does that console error suggest I should be looking into why UDP is failing and (2) any tips on trying to configure UDP on Fly.io specifically?

GitHub

fishjam-docs/docs/deploying/fly_io.md at main · fishjam-dev/fishjam...

Fishjam documentation. Contribute to fishjam-dev/fishjam-docs development by creating an account on GitHub.

12 Replies

Michał Śledź•3mo ago

Hi @TonyLikeSocks We don't support Fly.io Their network configuration was problematic for WebRTC based apps at least the last time we checked and we also no longer maintain RTC Engine 😦

TonyLikeSocksOP•3mo ago

😅 yeah, I heard. Migrating right now isn’t in the cards to ex webrtc though. I’ve got too much built out. Does the error message suggest I need UDP?

Michał Śledź•3mo ago

as long as you have TCP enabled, I belive that chrome should fallback to it when it cannot connect via UDP 🤔

TonyLikeSocksOP•3mo ago

Yeah, that was our assumption when we launched. That error message is really throwing me

Michał Śledź•3mo ago

you can go under chrome://webrtc-internals and see wheter tcp turn is on and what's its status

TonyLikeSocksOP•3mo ago

Ok - so the advice is to migrate off fly. It should work, but the evidence suggests it is not At least not reliably

Michał Śledź•3mo ago

It's hard to say something more without looking at the code are going deeper into the problem 😦 We always recomand bare providers like Hetzner for example, which we use for all our apps :/

TonyLikeSocksOP•3mo ago

Thanks @mickel8 Super helpful convo. I’ll work on moving to a bare provider.

Michał Śledź•3mo ago

Hope it will work! 🙂

TonyLikeSocksOP•3mo ago

Thanks for the tip on chrome://webrtc-internals. Testing now (with no changes, still on fly) -- Chrome thinks UDP is connected and sending / receiving bytes. But I also see the same timeout errors, and I never exposed UDP in my fly.toml. Very odd. For future searches, Fly.io is weird.

shuntrho•3mo ago

Hi @TonyLikeSocks , I agree with mickel8 that it's probably best to migrate away from fly.io, as we've been having issues with deployment there. Anyway, while you're there you can try to fix the error with turn ip. You cannot passfly-global-services address to turn options. The ip in the integrated_turn_options has to be a tuple. That means that you have resolve the address into an ipv4 before passing it to turn options. For this, you can use functions from Fishjam's ConfigParser: https://github.com/fishjam-dev/fishjam/blob/main/lib/fishjam/config_reader.ex#L39 and https://github.com/fishjam-dev/fishjam/blob/main/lib/fishjam/config_reader.ex#L368 . Make sure to read https://fly.io/docs/networking/udp-and-tcp/ guide. If this approach fails, you may also try somehow forcing chrome to use TCP turn

Fly

Running apps on UDP and TCP

Documentation and guides from the team at Fly.io.

GitHub

fishjam/lib/fishjam/config_reader.ex at main · fishjam-dev/fishjam

General purpose media server. Supports WebRTC, HLS, RTSP, SIP - fishjam-dev/fishjam

TonyLikeSocksOP•3mo ago

Thanks @shuntrho. That’s helpful to know on needing to pass in the tuple not a domain name. I am migrating away from fly.

Gaming

Programming

Fly.io + UDP