Distributing pipeline in erlang cluster.
I started working with membrane few days ago and am implementing this scenario using rtp to send streams between machines but I am wondering if it would be possible to simplify it by using erlang distribution to send data between machines.
Scenario:
Machine 1: Produces h264 stream from camera and sends it to Machine2
Machine 2: Recevies stream, transcodes it to h265 (ideally VP9 too but I couldn't find membrane plugin to do it), stores both versions and forwards streams to Machine3
Machine 3: Recieves h264 and h265 streams and serves them via HLS and WebRTC
4 Replies
👋 I'm not sure what are the consequences of sending large amounts of data through Erlang distribution, however it's possible. The simplest way is to pass
node option when spawning elements, for example {child(MyElement) |> child(AnotherElement), node: node}. The problem is that it uses cross-node supervision, so if a node becomes unreachable, the pipeline will see it as if all the children on that node crashed, which may or may not be desired. Another way would be to have a pair of elements (sink and source) that would send data via Erlang messages and have separate pipelines. Membrane doesn't provide those, however they should be fairly easy to implement.Regarding VP9, we're working on a plugin based on libvpx and we already got VP8/9 decoding working. It's early stage, but we should have encoding support very soon as well. See https://github.com/membraneframework/membrane_vpx_plugin/tree/setup-project
GitHub
GitHub - membraneframework/membrane_vpx_plugin at setup-project
Contribute to membraneframework/membrane_vpx_plugin development by creating an account on GitHub.
Erlang distribution is implemented on top of
TCP/IP so there are usual downsides of tcp present, otherwise sending binary data is generally efficient, sending structured data has serialization->deserialization step so it depends of how messages between processes are structured.
Thx for the info, don't think I would have found node option on my own. Could you point me to some code where I could get better understanding of what Membrane does in that case?
Implementing sink and source version already crossed my mind. I'll do research and benchmarks to see which version works best in my case.
Looking forward to V8/9 en/decoders 🙂Could you point me to some code where I could get better understanding of what Membrane does in that case?It just spawns processes via rpc https://github.com/membraneframework/membrane_core/blob/010f29e2afb7a806ea378375935da0649379f9b6/lib/membrane/core/element.ex#L87