Clustering and scale out behaviour
Context: I've a membrane application running in an elixir cluster of 1. It receives a RTSP stream (UDP packets) from a client and does the whole streaming thing - awesome!
If/when the cluster expands, there will be multiple nodes receiving UDP packets (the packets are load-balanced between nodes). Does Membrane have any handling to route the packet to the correct node? 🤔
6 Replies
If I understand correctly, you're using an Erlang distribution, having pipelines on each node that are capable of receiving UDP packets, and you want a Membrane load-balancer to decide, to which node redirect the newly arriving stream?
Please, correct me, if I have misunderstood your problem
Yep that's the thinking. Inbound streams need to be load balanced somehow, right? 🤔
But then an incoming stream needs to be associated with the same node after the load balancer decides and UDP doesn't have sticky packets 🤔
Playing through a hypothetical scenario where:
- there is an incoming RTP stream from a client
- server side, there's two nodes behind a UDP load balancer.
The initial packet (which triggers the first
:new_rtp_stream) lands on node 1, which creates the membrane pipeline.
The next packet could land on node 2. Currently I think this triggers another :new_rtp_stream and if RTSP is being used, it will probably cause :auth_fail errors since the encrypted stream is mismatched between node 1 and 2.
Ideally, there probably needs to be a mechanism for node 2 to pass the packet to node 1 for processing (?).I don't think we have such a load balancer. Are
node 1 and node 2 running on the same machine? If yes, there might be no need for a starting many nodes, because BEAM provides very good concurrency and you should be able to spawn many pipelines on the single node.
Ideally, there probably needs to be a mechanism for node 2 to pass the packet to node 1 for processing (?).Moreover, if you want to send such a packets between nodes, take into account that it will happen over TCP (if you want to use
send/2), what will worsen the performance of your system.In the scenario provided,
node 1 and node 2 are on different machines.
I totally understand many pipelines can be spawned on a single node but there's going to be a resource limit eventually.
Are there plans to support load balancing of some sort, or does the network basically need to know which node to send it to? 🤔I'm sorry, but currently we don't have plans to implement a load-balancing mechanism, that could solve your problem
No worries, I think I've worked out a viable solution (but will have to address some of my own quirks).
Thanks!