Lowest latency h264 UDP video stream possible.
Hello,
To give some context I am trying to replicate parts of what https://openhd.gitbook.io/open-hd
is doing using elixir, nerves and membrane.
I have rpi mountend on drone on one side and laptop on the other side connected in network
by two specially configured RTL8812AU wifi dongles.
Point here is that there is single direct network (no routers, network switches... in between)
but signal gets lower with distance and obstructions in the way. That results in varible bandwith,
packet loss and sometimes complete signal loss for few seconds.
For current prototype I am using RTP stream with RTPSessionBin, but latency is quite high,
I need it to be less than 100ms and just jitter buffer is more than that.
I treied digging deeper through RTP code but got ovewhelmed quickly.
I would like to start from building simplest h264 stream over UDP possible. No jitter control no variable bandwith...
I would probably go with just repeating last video frame until I get new one.
And then start adding features on top and measure how they impact latency as I want to show stream stats in UI
I guess webrtc has most of what I need already implemented but my question is what resources to read and
where to start. I am also willing to make a blog post and/or presentation covering my journey.
3 Replies
Hi @Damirados , yeah RTP.SessionBin is quite complex and could use a refactor. It also doesn't allow configuring jitter buffer latency, but you can disable depayloading by setting
depayloader option to nil on the output pad and plugging depayloader and jitter buffer after the bin manually - then you can configure the latency option - see https://hexdocs.pm/membrane_rtp_plugin/Membrane.RTP.JitterBuffer.html. Contrary to the name, jitter buffer actually doesn't control jitter - its main job is to make sure that packets are output in order, otherwise the stream could be undecodable, so I'd advise against removing it.
WebRTC may be a good option - it also uses RTP over UDP, but adds more features, including retransmissions, which may be necessary in a poor network. Generally, its hard to achieve low latency in a poor network, as subsequent video frames depend on previous ones and dropping part of the stream, so you either wait for retransmission or the stream is undecodable.Regarding WebRTC, have a look at https://blog.swmansion.com/11-resources-that-will-help-you-start-with-webrtc-b7e48c178b76
Medium
11 resources that will help you start with WebRTC
A collection of must-read links for every WebRTC developer.
There's also https://github.com/membraneframework/membrane_webrtc_plugin. If you want to minimize latency, set the
depayload: false option in the WebRTC Source and plug the depayloader and jitter buffer manually, lowering the latency, just like in case of plain RTP.