SM
Software Mansion•2y ago
Calms

RTP demo with RawAudio

Hello friends, I'm trying to get microphone input (via Membrane.PortAudio.Source) packaged into an RTP stream and sent to a server and can't quite seem to get it right. Excerpt below based on the demo in membrane-demo/rtp but with microphone input substituted and newer syntax. 
alias Membrane.{RTP, UDP, PortAudio, RawAudio}
...
links = [
      child(:mic_input, %PortAudio.Source{
        channels: 2,
        sample_rate: 24_000
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
        application: :audio,
        input_stream_format: %RawAudio{
          channels: 2,
          sample_format: :s16le,
          sample_rate: 24_000
        }
      })
      |> via_in(Pad.ref(:input, audio_ssrc), options: [payloader: RTP.Opus.Payloader])
      |> child(:rtp, %RTP.SessionBin{
        secure?: secure?,
        srtp_policies: [
          %ExLibSRTP.Policy{
            ssrc: :any_inbound,
            key: srtp_key
          }
        ]
      })
      |> via_out(Pad.ref(:rtp_output, audio_ssrc), options: [encoding: :OPUS])
      |> child(:audio_realtimer, Membrane.Realtimer)
      |> child(:audio_sink, %UDP.Sink{
        destination_port_no: destination_port,
        destination_address: destination_address
      })
    ]
alias Membrane.{RTP, UDP, PortAudio, RawAudio}
...
links = [
      child(:mic_input, %PortAudio.Source{
        channels: 2,
        sample_rate: 24_000
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
        application: :audio,
        input_stream_format: %RawAudio{
          channels: 2,
          sample_format: :s16le,
          sample_rate: 24_000
        }
      })
      |> via_in(Pad.ref(:input, audio_ssrc), options: [payloader: RTP.Opus.Payloader])
      |> child(:rtp, %RTP.SessionBin{
        secure?: secure?,
        srtp_policies: [
          %ExLibSRTP.Policy{
            ssrc: :any_inbound,
            key: srtp_key
          }
        ]
      })
      |> via_out(Pad.ref(:rtp_output, audio_ssrc), options: [encoding: :OPUS])
      |> child(:audio_realtimer, Membrane.Realtimer)
      |> child(:audio_sink, %UDP.Sink{
        destination_port_no: destination_port,
        destination_address: destination_address
      })
    ]
This seems to throw an error when generating the headers because buffer.pts is nil: 
[error] <0.815.0>/:rtp/{:stream_send_bin, 1236}/:payloader/:header_generator Error handling action {:split, {:handle_buffer, [[:input, %Membrane.Buffer{payload: <<220, 255, 254>>, pts: nil, dts: nil, metadata: %{}}]]}} returned by callback Membrane.RTP.HeaderGenerator.handle_buffers_batch

[error] GenServer #PID<0.849.0> terminating
** (FunctionClauseError) no function clause matching in Ratio.mult/2
    (ratio 2.4.2) lib/ratio.ex:418: Ratio.mult(nil, 48000)
    (membrane_rtp_plugin 0.23.0) lib/membrane/rtp/header_generator.ex:71: Membrane.RTP.HeaderGenerator.handle_process/4
[error] <0.815.0>/:rtp/{:stream_send_bin, 1236}/:payloader/:header_generator Error handling action {:split, {:handle_buffer, [[:input, %Membrane.Buffer{payload: <<220, 255, 254>>, pts: nil, dts: nil, metadata: %{}}]]}} returned by callback Membrane.RTP.HeaderGenerator.handle_buffers_batch

[error] GenServer #PID<0.849.0> terminating
** (FunctionClauseError) no function clause matching in Ratio.mult/2
    (ratio 2.4.2) lib/ratio.ex:418: Ratio.mult(nil, 48000)
    (membrane_rtp_plugin 0.23.0) lib/membrane/rtp/header_generator.ex:71: Membrane.RTP.HeaderGenerator.handle_process/4
Does anyone have any tips on what I might be doing wrong?
3 Replies
Calms
CalmsOP•2y ago
Ah, just stumbled across this issue: https://github.com/membraneframework/membrane_opus_plugin/issues/22 Looks like I wasn't going mad after all - opus plugin doesn't support it yet. Although I don't quite understand why I'm hitting this issue when membrane-demo/rtp works fine - perhaps a difference with the underlying RawAudio stream instead of a file?
varsill
varsill•2y ago
Hello @calms! I believe that adding child(:parser, Membrane.Opus.Parser) right after the encoder should solve your problem, since the Opus.Parser adds some metadata and PTS to the buffers flowing in the stream.
Calms
CalmsOP•2y ago
Ah, that does indeed solve the problem with buffer.pts! Many thanks! With that in place, this now works (although it's super quiet...) - happy to mark this as solved. If you had any tips on volume, I'd be all ears though!

Did you find this page helpful?