High WebRTC CPU consumption

Hey! We are performing some benchmarks with ExWebRTC-based pipeline and on 4 CPU droplet (with dedicated cores) and 10 incoming streams consumes 100% of CPU.

Is that expected behaviour? Our pipeline is essentially consuming H264 video and decodes the AAC to Opus, nothing else.

The second question is - how do we trace the membrane elements CPU consumption? I was trying to https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#module-visualizing-the-supervision-tree, but I doesn't see the pipeline at all in

:observer

:observer

. The Live dashboard on the other hand lists the processes and I can sort by "Number of Reductions", but over there all elements is just

Membrane.Core.Element

Membrane.Core.Element

so it's kind of makes it impossible to distinguish which process causes the most CPU consumption.

Please help

odingrailOP•3/28/25, 1:05 PM

Also, it is unclear how to consume the telemetry metrics even after reading the documentation(. I'm trying this and I got 0 events reported with working pipeline, what is the issue here?:

Also, my config.exs:

odingrailOP•3/28/25, 1:07 PM

The docs suggested the format:

varsill•3/28/25, 1:46 PM

Hello! Definitely something is suspicious with that high CPU consumption for 10 streams and it requires some inspection. I assume that you don't do any transcoding of the video track, do you?
Concerning the CPU usage monitoring, there are 2 places I would inspect:

use
```
:observer
```
```
:observer
```
, find a process with suspiciously high number of reductions, double click on that process, visit Dictionary tab and read the entry. (Side note: we plan to set label (https://hexdocs.pm/elixir/Process.html#set_label/1) for the element processes to their but it's not available yet so as of now I am afraid you need to read manually.)
see (where <pid> is the OS process of BEAM) and inspect if some OS threads are using extraordinarily high amount of CPU. There are two options here: either these are some "worker" threads spawned in NIFs (then you should see high CPU usage for some custom threads) or NIFs themselves are using much CPU (then you should see that or threads are using much CPU)

varsill•3/28/25, 1:52 PM

Concerning telemetry events, I think it might have something to do with a recent bug report. It turns out that by mistake the second element of the "suggested format" you mentioned is the component's module instead of the "component type" (by "component type" I mean :element, :bin or :pipeline). We fixed it here: https://github.com/membraneframework/membrane_core/pull/958 and we will soon release v1.2.3 with that change. As of now could you check if telemetry is working for you with membrane_core fixed on branch?

GitHub

Fixed telemetry metadata bug of component_type and covered it by te...

Closes #957

Feliks•3/31/25, 9:21 AM

@odingrail have you tried recompiling all dependencies?

Feliks•3/31/25, 9:25 AM

and there is no event named

Feliks•3/31/25, 9:26 AM

All Membrane events names match the convention , more docs are there: https://hexdocs.pm/membrane_core/Membrane.Telemetry.html

Feliks•3/31/25, 9:28 AM

to get events from a specific module, you can filter on

telemetry metadata type is https://hexdocs.pm/membrane_core/Membrane.Telemetry.html#t:callback_span_metadata/0

Feliks•3/31/25, 9:31 AM

What do you mean by ? Do you use ?

odingrailOP•3/31/25, 9:53 AM

Hey! Thank you for detalied answers @Feliks and @varsill ! I was able to see which processes consumes most resources by using this config: https://github.com/membraneframework/membrane_core/pull/960 in this way I see the labels in the LiveDashboard and in

:observer

:observer

. But from this info I don't see any anomalies in resource consumption.

We are using the , also we are converting the audio from AAC to Opus, and doesn't do any transcoding for this benchmarks.

GitHub

Correct documentation for unsafely_name_processes_for_observer by m...

odingrailOP•3/31/25, 9:56 AM

We are streaming RTSP with TCP transport, but similar resource consumption is also noted with RTMP streaming.

odingrailOP•3/31/25, 10:11 AM

We are trying to do the WebRTC distributed streaming. The architecture is roughtly the following: the have some node that consumes the stream (running the webrtc_sink and other elements), and then we are sending packets to "Room" process, which in turns distributes the packets accorss the peers, for that we are using the Phoenix PubSub.

odingrailOP•3/31/25, 4:53 PM

Regarding this debug technique you described:

top -H -p <pid> (where <pid> is the OS process of BEAM) and inspect if some OS threads are using extraordinarily high amount of CPU. There are two options here: either these are some "worker" threads spawned in NIFs (then you should see high CPU usage for some custom threads) or NIFs themselves are using much CPU (then you should see that ertsdcpus* or ertssched*threads are using much CPU)

Hmm,

varsill•4/1/25, 8:08 AM

Hello! Oh, I've forgotten about that config option, it has one main drawback as it works only for a single node, but it's great for debugging. Also thanks for spotting the bug in the documentation!

Concerning the results from , it's definitely odd that the main thread is using that much of CPU. At first sight I would say that it might have something to do with BEAM spending too much time on threads synchronization (which used to be a case with the desired number of schedulers improperly resolved in environments with limits assigned), but since you are using a droplet I don't think that's an issue.
Could you try to gather microstate accounting statistics for let's say a minute of the system running? (https://www.erlang.org/doc/apps/erts/erlang.html#statistics_microstate_accounting)
We should be able to see more precisely what BEAM is busy at.
If it doesn't tell us much, I am afraid we might need to use to see particular calls that use most of the CPU.

Concerning CPU usage you can try experimenting with disabling busy waiting for the schedulers with option (https://www.erlang.org/doc/apps/erts/erl_cmd.html#+sbt) to see (more or less) how much of CPU is indeed spent on code execution.

odingrailOP•4/1/25, 11:19 AM

Thanks everyone, I think that issue was that in the element we set the TCP socket to active mode. After switching to the and manual flow control the machine is able to sustain 28 WebRTC streams without significant growth of memory. (The CPU's are on 100%)

varsill•4/2/25, 8:06 AM

Hello, great you got it figured out!
Am I right that it was somewhere in your custom element where the TCP socket was operating in an active mode?

Vvarsill Hello, great you got it figured out! Am I right that it was somewhere in your c...

odingrailOP•4/2/25, 10:43 AM

Hey! Yes, you are right. When we were prototyping the pipeline we just set the socket to mode and forgot about that