TonyLikeSocks Posts - Answer Overflow

TonyLikeSocks

•Created by TonyLikeSocks on 10/3/2024 in #membrane-help

Pipeline Error: Pipeline Failed to Terminate within Timeout (5000ms)

This is a bit of a head scratcher for me. I'm in the process of writing a new element for my pipeline that uses the Silero VAD module for speech detection (rather than the built in WebRTC Engine VAD extension). I've got it working, but hitting a wierd bug. Now when my engine terminates (peer leaves), I'm getting this error: ** (Membrane.PipelineError) Pipeline #PID<0.1499.0> hasn't terminated within given timeout (5000 ms). The only thing that's changed is my new element in the pipeline (it's setup as a Membrane.Filter). If I remove the element from the pipeline, then the error goes away. I could obviously bump the timeout, but before I do that I thought to ask for advice. Why would adding an element increase the engine shutdown time? What's the right way to dig into this?

13 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 9/4/2024 in #membrane-help

Fly.io + UDP

17 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 6/12/2024 in #membrane-help

Issue Membrane Upgrade to 1.1 (from 0.12.9)

I'm trying to go from Membrane 0.12.9 to 1.1 and hitting a wall. I followed the upgrade guide[1], and everything compiles successfully. Though I'm getting the following error in my pipeline:

19:50:34.066 [error] <0.4709.0>/{:endpoint, "conversation_endpoint"}/:opus_payloader/:header_generator Error occured in Membrane Element:
%UndefinedFunctionError{
  module: Coerce.Implementations.Atom.Integer,
  function: :coerce,
  arity: 2,
  reason: nil,
  message: nil
}
    Coerce.Implementations.Atom.Integer.coerce(nil, 48000)
    (numbers 5.2.4) lib/numbers.ex:98: Numbers.mult/2
    (membrane_rtp_plugin 0.24.1) lib/membrane/rtp/header_generator.ex:71: Membrane.RTP.HeaderGenerator.handle_buffer/4
    (membrane_core 1.1.0) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 1.1.0) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (elixir 1.15.6) lib/enum.ex:2510: Enum."-reduce/3-lists^foldl/2-0-"/3

...trimming error stack

Last message: {Membrane.Core.Message, :buffer, [%Membrane.Buffer{payload: <<72, 11, 239, 4, 254, 218, 15, 39, 69, 221, 236, 72, 21, 31, 80>>, pts: nil, dts: nil, metadata: %{duration: 20000000}}], [for_pad: :input]}, metadata: line=1373 pid=<0.4945.0> file=gen_server.erl domain=otp mfa=:gen_server.error_info/8 mb_prefix=<0.4709.0>/{:endpoint, "conversation_endpoint"}/:opus_payloader/:header_generator rtc_engine=4CE007265278E76558B69EA3354CCCF4 room_id=4CE007265278E76558B69EA3354CCCF4

19:50:34.066 [error] <0.4709.0>/{:endpoint, "conversation_endpoint"}/:opus_payloader/:header_generator Error occured in Membrane Element:
%UndefinedFunctionError{
  module: Coerce.Implementations.Atom.Integer,
  function: :coerce,
  arity: 2,
  reason: nil,
  message: nil
}
    Coerce.Implementations.Atom.Integer.coerce(nil, 48000)
    (numbers 5.2.4) lib/numbers.ex:98: Numbers.mult/2
    (membrane_rtp_plugin 0.24.1) lib/membrane/rtp/header_generator.ex:71: Membrane.RTP.HeaderGenerator.handle_buffer/4
    (membrane_core 1.1.0) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 1.1.0) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (elixir 1.15.6) lib/enum.ex:2510: Enum."-reduce/3-lists^foldl/2-0-"/3

...trimming error stack

Last message: {Membrane.Core.Message, :buffer, [%Membrane.Buffer{payload: <<72, 11, 239, 4, 254, 218, 15, 39, 69, 221, 236, 72, 21, 31, 80>>, pts: nil, dts: nil, metadata: %{duration: 20000000}}], [for_pad: :input]}, metadata: line=1373 pid=<0.4945.0> file=gen_server.erl domain=otp mfa=:gen_server.error_info/8 mb_prefix=<0.4709.0>/{:endpoint, "conversation_endpoint"}/:opus_payloader/:header_generator rtc_engine=4CE007265278E76558B69EA3354CCCF4 room_id=4CE007265278E76558B69EA3354CCCF4

I looked at the code in lib/membrane/rtp/header_generator.ex:71 and it seems to assume that buffer.pts won't be nil. Though you can see in the last message it's definitely nil, which I think it what's causing the error to throw. More details in thread. Any advice would be appreciated. [1] https://hexdocs.pm/membrane_core/1.1.0/v1-0-0.html

16 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 6/7/2024 in #membrane-help

WebRTC Endpoint + Mixing Multiple Tracks into a single mp4

I have a working app that allows a user to "talk" to an LLM. I'm using Membrane to help coordinate the audio. For QA purposes, we record the tracks (one for each endpoint). I'm trying to setup a bin that mixes the two tracks using the Membrane.LiveAudioMixer so I can have a single file. There's no errors thrown, but the resulting file is only 40 bytes, so I suspect I have something misconfigured. Each time a pad is added, I try piping it into the LiveAudioMixer and then take that output, encode it and write it to the file.

def handle_setup(_context, state) do
    log_path = Application.fetch_env!(:smartvox, :log_path)
    File.mkdir(log_path)

    spec = [
      child(:mixer, %Membrane.LiveAudioMixer{
        stream_format: %Membrane.RawAudio{
          channels: 1,
          sample_rate: 16_000,
          sample_format: :s16le
        }
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
          application: :voip,
          input_stream_format: %Membrane.RawAudio{
            channels: 1,
            sample_rate: 16_000,
            sample_format: :s16le
          }
      })
      |> child(:parser, Membrane.Opus.Parser)
      |> child({:muxer, state.room_id}, Membrane.MP4.Muxer.ISOM)
      |> child({:sink, state.room_id}, %Membrane.File.Sink{
        location: "#{log_path}/#{state.room_id}.mp4"
      })
    ]

    {[spec: spec], state}
  end

  def handle_pad_added(Pad.ref(:input, track_id) = pad, _ctx, state) do
    track = state.tracks[track_id]

    spec = [
      bin_input(pad)
      |> child({:track_receiver, track_id}, Smartvox.Endpoints.Conversation.TrackRecevier)
      |> child({:depayloader, track_id}, Track.get_depayloader(track))
      |> child({:decoder, track_id}, %Membrane.Opus.Decoder{
        sample_rate: 16_000,
      })
      |> via_in(:input)
      |> get_child(:mixer)
    ]

    {[spec: spec], state}
  end

def handle_setup(_context, state) do
    log_path = Application.fetch_env!(:smartvox, :log_path)
    File.mkdir(log_path)

    spec = [
      child(:mixer, %Membrane.LiveAudioMixer{
        stream_format: %Membrane.RawAudio{
          channels: 1,
          sample_rate: 16_000,
          sample_format: :s16le
        }
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
          application: :voip,
          input_stream_format: %Membrane.RawAudio{
            channels: 1,
            sample_rate: 16_000,
            sample_format: :s16le
          }
      })
      |> child(:parser, Membrane.Opus.Parser)
      |> child({:muxer, state.room_id}, Membrane.MP4.Muxer.ISOM)
      |> child({:sink, state.room_id}, %Membrane.File.Sink{
        location: "#{log_path}/#{state.room_id}.mp4"
      })
    ]

    {[spec: spec], state}
  end

  def handle_pad_added(Pad.ref(:input, track_id) = pad, _ctx, state) do
    track = state.tracks[track_id]

    spec = [
      bin_input(pad)
      |> child({:track_receiver, track_id}, Smartvox.Endpoints.Conversation.TrackRecevier)
      |> child({:depayloader, track_id}, Track.get_depayloader(track))
      |> child({:decoder, track_id}, %Membrane.Opus.Decoder{
        sample_rate: 16_000,
      })
      |> via_in(:input)
      |> get_child(:mixer)
    ]

    {[spec: spec], state}
  end

7 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 4/24/2024 in #membrane-help

ex_dtls won't compile

I'm sure this is a me issue, but I'm stumped. I've got a membrane project that worked on a different computer. Both are Macs Running mix deps.compile throws an error:

ld: library 'ssl' not found
clang: error: linker command failed with exit code 1 (use -v to see invocation)
could not compile dependency :ex_dtls, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile ex_dtls --force", update it with "mix deps.update ex_dtls" or clean it with "mix deps.clean ex_dtls"

I've got openssl@3 installed via homebrew, and have the following env variables set

export LDFLAGS="-L${BREW_CELLAR}/openssl@3/${OPENSSL_VERSION}/lib"
export CFLAGS="-I${BREW_CELLAR}/openssl@3/${OPENSSL_VERSION}/include/"
export CPPFLAGS="-I${BREW_CELLAR}/openssl@3/${OPENSSL_VERSION}/include/"

What am I missing? Does ex_dtls actually need [email protected] instead? Thanks for the help.

6 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 4/9/2024 in #membrane-help

Wiring up Javascript FE Using membrane-webrtc-js

Sorry if this obvious. I'm looking through the example in the membrane_rtc_engine (link below). It's not obvious to me how the audio playback for remote endpoints is managed. Does the membrane-webrtc-js take care of that magically? I see addVideoElement -- but that just seems to add an HTMLVideo element, but doesn't actually connect it to anything from the endpoint / tracks. https://github.com/jellyfish-dev/membrane_rtc_engine/blob/master/examples/webrtc_videoroom/assets/src/room.ts

2 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 1/30/2024 in #membrane-help

Testing Membrane Element

I'm trying to setup a simple test of a membrane element, but I'm stumped a bit on how to assert that the element is sending the correct output. For context, this is an element that accepts an audio stream, sends it to a speech 2 text api and then forwards along a text buffer. The test fails, as there's no 'buffer' in the mailbox. Though I'm positive that's what my element emits when the stream is complete. I've tried longer timeouts (10s) but that doesn't alter the test. I could use some advice.

    test "Converts sample audio to text" do
      current_dir = __DIR__
      file_path = Path.join([current_dir, "../../fixtures/test.wav"])
      absolute_path = Path.expand(file_path)
      sample_audio = File.read!(absolute_path)

      expected_text =
        "Adventure 1 a scandal in Bohemia from The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle. This is a LibriVox recording."

      links = [
        child(:source, %Source{output: [sample_audio], stream_format: %RawAudio{sample_format: :s16le, sample_rate: 16_000, channels: 1}})
        |> child(:speech_2_text, Speech2Text)
        |> child(:sink, %Sink{})
      ]

      options = [
        spec: links
      ]

      pipeline = Pipeline.start_link_supervised!(options)

      assert_sink_buffer(pipeline, :sink, {:buffer, %Buffer{payload: ^expected_text}}, 5_000)

      Pipeline.terminate(pipeline)
    end

    test "Converts sample audio to text" do
      current_dir = __DIR__
      file_path = Path.join([current_dir, "../../fixtures/test.wav"])
      absolute_path = Path.expand(file_path)
      sample_audio = File.read!(absolute_path)

      expected_text =
        "Adventure 1 a scandal in Bohemia from The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle. This is a LibriVox recording."

      links = [
        child(:source, %Source{output: [sample_audio], stream_format: %RawAudio{sample_format: :s16le, sample_rate: 16_000, channels: 1}})
        |> child(:speech_2_text, Speech2Text)
        |> child(:sink, %Sink{})
      ]

      options = [
        spec: links
      ]

      pipeline = Pipeline.start_link_supervised!(options)

      assert_sink_buffer(pipeline, :sink, {:buffer, %Buffer{payload: ^expected_text}}, 5_000)

      Pipeline.terminate(pipeline)
    end

25 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 9/25/2023 in #membrane-help

React-Native connection?

I'm struggling to get a react-native client to connect to my membrane server. I'm just running locally right now. I start my membrane server with EXTERNAL_IP={my ip} mix phx.server. I'm using the @jellyfish-dev/react-native-membrane-webrtc client in my react native code Then I have the following connection code in my react-native view. I see the console log statements for init connect and attempting connect. But it never connects. I don't see a connection message in my phoenix server, nor the successful connection message. I tried increasing the log verbosity, but didn't get anything out of the logs from react-native. Is there something obviously wrong with my connection string? Is it expecting something different for the server URL?

const startServerConnection = async () => {
    const deviceID = await getUniqueId();
    
    try {
      console.log('attempting connnect')
      //Should make this environment aware at some point
      connect('https://192.168.121.87:4001/socket/', "room:" + deviceID, {
        endpointMetadata: {
          displayName: deviceID,
        },
        socketChannelParams: {
          childrenNames: params.childrenNames,
          talkAbout: params.talkAbout          
        }
      }).then(() => {
        console.log('connected. starting mic')
        startMicrophone({ audioTrackMetadata: { active: true, type: 'audio' } });
      }).catch((e) => {
        console.log('connection error: ' + e);  
      });
      
    } catch (e) {
      console.log('connection error: ' + e);
    }
  };

  useEffect(()=>{
    let isMounted = true;
    const logLevel: LoggingSeverity = LoggingSeverity.Verbose
    changeWebRTCLoggingSeverity(logLevel)
    const initConnection = async ()=>{
      try {
        console.log('init connnect')
        await startServerConnection();
        if (isMounted) {
          setIsConnected(true)
        }
      } catch (error) {
        console.log('init connection error: ' + error)
      }
    }

const startServerConnection = async () => {
    const deviceID = await getUniqueId();
    
    try {
      console.log('attempting connnect')
      //Should make this environment aware at some point
      connect('https://192.168.121.87:4001/socket/', "room:" + deviceID, {
        endpointMetadata: {
          displayName: deviceID,
        },
        socketChannelParams: {
          childrenNames: params.childrenNames,
          talkAbout: params.talkAbout          
        }
      }).then(() => {
        console.log('connected. starting mic')
        startMicrophone({ audioTrackMetadata: { active: true, type: 'audio' } });
      }).catch((e) => {
        console.log('connection error: ' + e);  
      });
      
    } catch (e) {
      console.log('connection error: ' + e);
    }
  };

  useEffect(()=>{
    let isMounted = true;
    const logLevel: LoggingSeverity = LoggingSeverity.Verbose
    changeWebRTCLoggingSeverity(logLevel)
    const initConnection = async ()=>{
      try {
        console.log('init connnect')
        await startServerConnection();
        if (isMounted) {
          setIsConnected(true)
        }
      } catch (error) {
        console.log('init connection error: ' + error)
      }
    }

4 replies

Gaming

Programming