Software Mansion•14mo ago

RTP demo with RawAudio

Hello friends, I'm trying to get microphone input (via Membrane.PortAudio.Source) packaged into an RTP stream and sent to a server and can't quite seem to get it right. Excerpt below based on the demo in membrane-demo/rtp but with microphone input substituted and newer syntax.

alias Membrane.{RTP, UDP, PortAudio, RawAudio}
...
links = [
      child(:mic_input, %PortAudio.Source{
        channels: 2,
        sample_rate: 24_000
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
        application: :audio,
        input_stream_format: %RawAudio{
          channels: 2,
          sample_format: :s16le,
          sample_rate: 24_000
        }
      })
      |> via_in(Pad.ref(:input, audio_ssrc), options: [payloader: RTP.Opus.Payloader])
      |> child(:rtp, %RTP.SessionBin{
        secure?: secure?,
        srtp_policies: [
          %ExLibSRTP.Policy{
            ssrc: :any_inbound,
            key: srtp_key
          }
        ]
      })
      |> via_out(Pad.ref(:rtp_output, audio_ssrc), options: [encoding: :OPUS])
      |> child(:audio_realtimer, Membrane.Realtimer)
      |> child(:audio_sink, %UDP.Sink{
        destination_port_no: destination_port,
        destination_address: destination_address
      })
    ]

alias Membrane.{RTP, UDP, PortAudio, RawAudio}
...
links = [
      child(:mic_input, %PortAudio.Source{
        channels: 2,
        sample_rate: 24_000
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
        application: :audio,
        input_stream_format: %RawAudio{
          channels: 2,
          sample_format: :s16le,
          sample_rate: 24_000
        }
      })
      |> via_in(Pad.ref(:input, audio_ssrc), options: [payloader: RTP.Opus.Payloader])
      |> child(:rtp, %RTP.SessionBin{
        secure?: secure?,
        srtp_policies: [
          %ExLibSRTP.Policy{
            ssrc: :any_inbound,
            key: srtp_key
          }
        ]
      })
      |> via_out(Pad.ref(:rtp_output, audio_ssrc), options: [encoding: :OPUS])
      |> child(:audio_realtimer, Membrane.Realtimer)
      |> child(:audio_sink, %UDP.Sink{
        destination_port_no: destination_port,
        destination_address: destination_address
      })
    ]

This seems to throw an error when generating the headers because buffer.pts is nil:

[error] <0.815.0>/:rtp/{:stream_send_bin, 1236}/:payloader/:header_generator Error handling action {:split, {:handle_buffer, [[:input, %Membrane.Buffer{payload: <<220, 255, 254>>, pts: nil, dts: nil, metadata: %{}}]]}} returned by callback Membrane.RTP.HeaderGenerator.handle_buffers_batch

[error] GenServer #PID<0.849.0> terminating
** (FunctionClauseError) no function clause matching in Ratio.mult/2
    (ratio 2.4.2) lib/ratio.ex:418: Ratio.mult(nil, 48000)
    (membrane_rtp_plugin 0.23.0) lib/membrane/rtp/header_generator.ex:71: Membrane.RTP.HeaderGenerator.handle_process/4

[error] <0.815.0>/:rtp/{:stream_send_bin, 1236}/:payloader/:header_generator Error handling action {:split, {:handle_buffer, [[:input, %Membrane.Buffer{payload: <<220, 255, 254>>, pts: nil, dts: nil, metadata: %{}}]]}} returned by callback Membrane.RTP.HeaderGenerator.handle_buffers_batch

[error] GenServer #PID<0.849.0> terminating
** (FunctionClauseError) no function clause matching in Ratio.mult/2
    (ratio 2.4.2) lib/ratio.ex:418: Ratio.mult(nil, 48000)
    (membrane_rtp_plugin 0.23.0) lib/membrane/rtp/header_generator.ex:71: Membrane.RTP.HeaderGenerator.handle_process/4

Does anyone have any tips on what I might be doing wrong?

3 Replies

Calms•14mo ago

Ah, just stumbled across this issue: https://github.com/membraneframework/membrane_opus_plugin/issues/22 Looks like I wasn't going mad after all - opus plugin doesn't support it yet. Although I don't quite understand why I'm hitting this issue when membrane-demo/rtp works fine - perhaps a difference with the underlying RawAudio stream instead of a file?

varsill•14mo ago

Hello @calms! I believe that adding child(:parser, Membrane.Opus.Parser) right after the encoder should solve your problem, since the Opus.Parser adds some metadata and PTS to the buffers flowing in the stream.

Calms•14mo ago

Ah, that does indeed solve the problem with buffer.pts! Many thanks! With that in place, this now works (although it's super quiet...) - happy to mark this as solved. If you had any tips on volume, I'd be all ears though!

Gaming

Programming

RTP demo with RawAudio