TonyLikeSocks Comments - Answer Overflow

TonyLikeSocks

Posts Comments

SMSoftware Mansion

•Created by TonyLikeSocks on 10/3/2024 in #membrane-help

Pipeline Error: Pipeline Failed to Terminate within Timeout (5000ms)

The inference time is ~1 ms according to my logging.

13 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 10/3/2024 in #membrane-help

Pipeline Error: Pipeline Failed to Terminate within Timeout (5000ms)

It's stable insofar as it works. The delayed shutdown is just puzzling. I'm guessing it's Ortex + the use of Rustler. Though I've looked through those docs and the code, and I didn't see an "unload" or cleanup type function. Though tellingly, if I replace my do_predict/3 function with a dummy function like so:

  defp do_predict(model, model_state, audio) do
    {0.5, "dummy_state"}
  end

  defp do_predict(model, model_state, audio) do
    {0.5, "dummy_state"}
  end

I don't get the same timeout failure. So it's defintely that Ortex.run/2 call.

13 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 10/3/2024 in #membrane-help

Pipeline Error: Pipeline Failed to Terminate within Timeout (5000ms)

Here's a gist with the module. I was thinking of open sourcing it, if I could get it stable. https://gist.github.com/Tonyhaenn/66c9148b2ae73a5009894250a0b6f6d7

13 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 10/3/2024 in #membrane-help

Pipeline Error: Pipeline Failed to Terminate within Timeout (5000ms)

08:59:34.478 [error] GenServer "ac02dc54-14ba-4791-86c9-fc45bd38ea32" terminating
** (Membrane.PipelineError) Pipeline #PID<0.1504.0> hasn't terminated within given timeout (5000 ms).
If you want to kill it anyway, use `force?: true` option.


Last message: :terminate_request, metadata: line=1391 pid=<0.1500.0> file=gen_server.erl domain=otp mfa=:gen_server.error_info/8 room_id=ac02dc54-14ba-4791-86c9-fc45bd38ea32  
Termination time: 2024-10-03 12:59:34.480162Z
[lib/smartvox/endpoints/conversation/vad.ex:59: Smartvox.Endpoints.Conversation.SileroVAD.handle_init/2]
binding() #=> [
  _ctx: %{
    name: :silero_vad,
    resource_guard: #PID<0.1636.0>,
    pads: %{},
    playback: :stopped,
    clock: nil,
    utility_supervisor: #PID<0.1634.0>,
    parent_clock: #PID<0.1510.0>
  },
  _pid: #PID<0.1635.0>,
  element_pid: #PID<0.1635.0>,
  options: %Smartvox.Endpoints.Conversation.SileroVAD{
    threshold: 0.5,
    min_speech_duration: 0.05,
    min_silence_duration: 0.25,
    smoothing_factor: 0.7
  },
  reason: :shutdown,
  ref: #Reference<0.3075342379.1387790343.180566>,
  state: %{
    chunk_size: nil,
    threshold: 0.5,
    min_speech_duration: 0.05,
    min_silence_duration: 0.25,
    smoothing_factor: 0.7,
    model: nil,
    audio_buffer: "",
    lstm: #Nx.Tensor<
      f32[2][1][128]
      [
        [
          [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...]
        ],
        ...
      ]
    >,
    last_vad_state: :silence,
    speech_duration: 0.0,
    silence_duration: 0.0,
    last_prob: 0
  }
]

08:59:34.478 [error] GenServer "ac02dc54-14ba-4791-86c9-fc45bd38ea32" terminating
** (Membrane.PipelineError) Pipeline #PID<0.1504.0> hasn't terminated within given timeout (5000 ms).
If you want to kill it anyway, use `force?: true` option.


Last message: :terminate_request, metadata: line=1391 pid=<0.1500.0> file=gen_server.erl domain=otp mfa=:gen_server.error_info/8 room_id=ac02dc54-14ba-4791-86c9-fc45bd38ea32  
Termination time: 2024-10-03 12:59:34.480162Z
[lib/smartvox/endpoints/conversation/vad.ex:59: Smartvox.Endpoints.Conversation.SileroVAD.handle_init/2]
binding() #=> [
  _ctx: %{
    name: :silero_vad,
    resource_guard: #PID<0.1636.0>,
    pads: %{},
    playback: :stopped,
    clock: nil,
    utility_supervisor: #PID<0.1634.0>,
    parent_clock: #PID<0.1510.0>
  },
  _pid: #PID<0.1635.0>,
  element_pid: #PID<0.1635.0>,
  options: %Smartvox.Endpoints.Conversation.SileroVAD{
    threshold: 0.5,
    min_speech_duration: 0.05,
    min_silence_duration: 0.25,
    smoothing_factor: 0.7
  },
  reason: :shutdown,
  ref: #Reference<0.3075342379.1387790343.180566>,
  state: %{
    chunk_size: nil,
    threshold: 0.5,
    min_speech_duration: 0.05,
    min_silence_duration: 0.25,
    smoothing_factor: 0.7,
    model: nil,
    audio_buffer: "",
    lstm: #Nx.Tensor<
      f32[2][1][128]
      [
        [
          [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...]
        ],
        ...
      ]
    >,
    last_vad_state: :silence,
    speech_duration: 0.0,
    silence_duration: 0.0,
    last_prob: 0
  }
]

13 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 10/3/2024 in #membrane-help

Pipeline Error: Pipeline Failed to Terminate within Timeout (5000ms)

Ahh. Interesting, I did your method and got an entirely different result.

13 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 10/3/2024 in #membrane-help

Pipeline Error: Pipeline Failed to Terminate within Timeout (5000ms)

I didn't do that, but I did just try;

  @impl true
  def handle_terminate_request(_ctx, state) do
    start_time = System.monotonic_time(:millisecond)
    IO.puts("========== VAD terminating ==========")

    state
    |> Map.delete(:model)
    |> Map.delete(:audio_buffer)
    |> Map.delete(:lstm)

    :erlang.garbage_collect()

    end_time = System.monotonic_time(:millisecond)
    IO.puts("VAD terminated in #{end_time - start_time}ms")

    {[], state}
  end

  @impl true
  def handle_terminate_request(_ctx, state) do
    start_time = System.monotonic_time(:millisecond)
    IO.puts("========== VAD terminating ==========")

    state
    |> Map.delete(:model)
    |> Map.delete(:audio_buffer)
    |> Map.delete(:lstm)

    :erlang.garbage_collect()

    end_time = System.monotonic_time(:millisecond)
    IO.puts("VAD terminated in #{end_time - start_time}ms")

    {[], state}
  end

I get a VAD shutdown time of 0ms, and then 5000ms later, I get the error message. Unless this test isn't reliable

13 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 10/3/2024 in #membrane-help

Pipeline Error: Pipeline Failed to Terminate within Timeout (5000ms)

How should I go about determining why my element is taking too long to terminate? I don’t have a handle_terminate callback defined.

13 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 10/3/2024 in #membrane-help

Pipeline Error: Pipeline Failed to Terminate within Timeout (5000ms)

I'm loading the Ortex model within the element init and storing it within the state model = Ortex.load(@model_path) But beyond that, everything else seems pretty straight forward logic / state management (if prob > threshold blah, etc.)

13 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 9/4/2024 in #membrane-help

Fly.io + UDP

Thanks @shuntrho. That’s helpful to know on needing to pass in the tuple not a domain name. I am migrating away from fly.

17 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 9/4/2024 in #membrane-help

Fly.io + UDP

Thanks for the tip on chrome://webrtc-internals. Testing now (with no changes, still on fly) -- Chrome thinks UDP is connected and sending / receiving bytes. But I also see the same timeout errors, and I never exposed UDP in my fly.toml. Very odd. For future searches, Fly.io is weird.

17 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 9/4/2024 in #membrane-help

Fly.io + UDP

Thanks @mickel8 Super helpful convo. I’ll work on moving to a bare provider.

17 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 9/4/2024 in #membrane-help

Fly.io + UDP

At least not reliably

17 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 9/4/2024 in #membrane-help

Fly.io + UDP

Ok - so the advice is to migrate off fly. It should work, but the evidence suggests it is not

17 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 9/4/2024 in #membrane-help

Fly.io + UDP

Yeah, that was our assumption when we launched. That error message is really throwing me

17 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 9/4/2024 in #membrane-help

Fly.io + UDP

😅 yeah, I heard. Migrating right now isn’t in the cards to ex webrtc though. I’ve got too much built out. Does the error message suggest I need UDP?

17 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 6/12/2024 in #membrane-help

Issue Membrane Upgrade to 1.1 (from 0.12.9)

Thank you. That got us past that error, but now I see a new one:

20:05:56.275 [error] <0.1543.0>/{:endpoint, "recorder"}/:mixer Error occured in Membrane Element:
%Protocol.UndefinedError{
  protocol: String.Chars,
  value: #Reference<0.2482563602.4096262146.236672>,
  description: ""
}
    (elixir 1.15.6) lib/string/chars.ex:3: String.Chars.impl_for!/1
    (elixir 1.15.6) lib/string/chars.ex:22: String.Chars.to_string/1
    (membrane_audio_mix_plugin 0.16.1) lib/membrane_live_audio_mixer/live_queue.ex:57: Membrane.LiveAudioMixer.LiveQueue.remove_queue/2
    (membrane_audio_mix_plugin 0.16.1) lib/membrane_live_audio_mixer.ex:221: Membrane.LiveAudioMixer.handle_end_of_stream/3
    (membrane_core 1.1.0) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 1.1.0) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (membrane_core 1.1.0) lib/membrane/core/element/event_controller.ex:131: Membrane.Core.Element.EventController.exec_handle_event/4
    (membrane_core 1.1.0) lib/membrane/core/element.ex:243: Membrane.Core.Element.handle_info/2
    (stdlib 5.1.1) gen_server.erl:1077: :gen_server.try_handle_info/3
    (stdlib 5.1.1) gen_server.erl:1165: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3

20:05:56.275 [error] <0.1543.0>/{:endpoint, "recorder"}/:mixer Error occured in Membrane Element:
%Protocol.UndefinedError{
  protocol: String.Chars,
  value: #Reference<0.2482563602.4096262146.236672>,
  description: ""
}
    (elixir 1.15.6) lib/string/chars.ex:3: String.Chars.impl_for!/1
    (elixir 1.15.6) lib/string/chars.ex:22: String.Chars.to_string/1
    (membrane_audio_mix_plugin 0.16.1) lib/membrane_live_audio_mixer/live_queue.ex:57: Membrane.LiveAudioMixer.LiveQueue.remove_queue/2
    (membrane_audio_mix_plugin 0.16.1) lib/membrane_live_audio_mixer.ex:221: Membrane.LiveAudioMixer.handle_end_of_stream/3
    (membrane_core 1.1.0) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 1.1.0) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (membrane_core 1.1.0) lib/membrane/core/element/event_controller.ex:131: Membrane.Core.Element.EventController.exec_handle_event/4
    (membrane_core 1.1.0) lib/membrane/core/element.ex:243: Membrane.Core.Element.handle_info/2
    (stdlib 5.1.1) gen_server.erl:1077: :gen_server.try_handle_info/3
    (stdlib 5.1.1) gen_server.erl:1165: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3

16 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 6/12/2024 in #membrane-help

Issue Membrane Upgrade to 1.1 (from 0.12.9)

%Membrane.ParentError{
  message: "Duplicated names in children specification: [:audio_parser]"
}

%Membrane.ParentError{
  message: "Duplicated names in children specification: [:audio_parser]"
}

I added the new element as part of the spec defined in handle_pad_added. However, I get the error above.

  @impl true
  def handle_pad_added(Pad.ref(:input, track_id) = pad, _ctx, state) do
    track = state.tracks[track_id]

    spec = [
      bin_input(pad)
      |> child({:track_receiver, track_id}, Smartvox.Endpoints.Conversation.TrackRecevier)
      |> child({:depayloader, track_id}, Track.get_depayloader(track))
      |> child({:decoder, track_id}, %Membrane.Opus.Decoder{
        sample_rate: 16_000
      })
      |> child(:audio_parser, %Membrane.RawAudioParser{overwrite_pts?: true})
      |> via_in(:input)
      |> get_child(:mixer)
    ]

    {[spec: spec], state}
  end

  @impl true
  def handle_pad_added(Pad.ref(:input, track_id) = pad, _ctx, state) do
    track = state.tracks[track_id]

    spec = [
      bin_input(pad)
      |> child({:track_receiver, track_id}, Smartvox.Endpoints.Conversation.TrackRecevier)
      |> child({:depayloader, track_id}, Track.get_depayloader(track))
      |> child({:decoder, track_id}, %Membrane.Opus.Decoder{
        sample_rate: 16_000
      })
      |> child(:audio_parser, %Membrane.RawAudioParser{overwrite_pts?: true})
      |> via_in(:input)
      |> get_child(:mixer)
    ]

    {[spec: spec], state}
  end

16 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 6/12/2024 in #membrane-help

Issue Membrane Upgrade to 1.1 (from 0.12.9)

This is my setup:

def handle_setup(_context, state) do
    log_path = Application.fetch_env!(:smartvox, :log_path)
    File.mkdir(log_path)

    spec = [
      child(:mixer, %Membrane.LiveAudioMixer{
        stream_format: %Membrane.RawAudio{
          channels: 1,
          sample_rate: 16_000,
          sample_format: :s16le
        }
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
          application: :voip,
          input_stream_format: %Membrane.RawAudio{
            channels: 1,
            sample_rate: 16_000,
            sample_format: :s16le
          }
      })
      |> child(:parser, Membrane.Opus.Parser)
      |> child({:muxer, state.room_id}, Membrane.MP4.Muxer.ISOM)
      |> child({:sink, state.room_id}, %Membrane.File.Sink{
        location: "#{log_path}/#{state.room_id}.mp4"
      })
    ]

    {[spec: spec], state}
  end

  def handle_pad_added(Pad.ref(:input, track_id) = pad, _ctx, state) do
    track = state.tracks[track_id]

    spec = [
      bin_input(pad)
      |> child({:track_receiver, track_id}, Smartvox.Endpoints.Conversation.TrackRecevier)
      |> child({:depayloader, track_id}, Track.get_depayloader(track))
      |> child({:decoder, track_id}, %Membrane.Opus.Decoder{
        sample_rate: 16_000,
      })
      |> via_in(:input)
      |> get_child(:mixer)
    ]

    {[spec: spec], state}
  end

def handle_setup(_context, state) do
    log_path = Application.fetch_env!(:smartvox, :log_path)
    File.mkdir(log_path)

    spec = [
      child(:mixer, %Membrane.LiveAudioMixer{
        stream_format: %Membrane.RawAudio{
          channels: 1,
          sample_rate: 16_000,
          sample_format: :s16le
        }
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
          application: :voip,
          input_stream_format: %Membrane.RawAudio{
            channels: 1,
            sample_rate: 16_000,
            sample_format: :s16le
          }
      })
      |> child(:parser, Membrane.Opus.Parser)
      |> child({:muxer, state.room_id}, Membrane.MP4.Muxer.ISOM)
      |> child({:sink, state.room_id}, %Membrane.File.Sink{
        location: "#{log_path}/#{state.room_id}.mp4"
      })
    ]

    {[spec: spec], state}
  end

  def handle_pad_added(Pad.ref(:input, track_id) = pad, _ctx, state) do
    track = state.tracks[track_id]

    spec = [
      bin_input(pad)
      |> child({:track_receiver, track_id}, Smartvox.Endpoints.Conversation.TrackRecevier)
      |> child({:depayloader, track_id}, Track.get_depayloader(track))
      |> child({:decoder, track_id}, %Membrane.Opus.Decoder{
        sample_rate: 16_000,
      })
      |> via_in(:input)
      |> get_child(:mixer)
    ]

    {[spec: spec], state}
  end

Would you add a parser after the opus decoder in the handle_pad_added callback?

16 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 6/12/2024 in #membrane-help

Issue Membrane Upgrade to 1.1 (from 0.12.9)

membrane_rtp_plugin ~> 0.24.1, which is a child dependency of membrane_rtc_engine ~> 0.22.0 and of membrane_rtc_engine_webrtc ~> 0.8.0

16 replies

SMSoftware Mansion

•Created by TonyLikeSocks on 6/12/2024 in #membrane-help

Issue Membrane Upgrade to 1.1 (from 0.12.9)

I think I’m on latest for everything but the file plugin. I don’t explicitly pull in :membrane_rtp_plugin Membrane {:membrane_audio_mix_plugin, "~> 0.16.1"}, {:membrane_fake_plugin, "~> 0.11"}, {:membrane_ffmpeg_swresample_plugin, "~> 0.20.2"}, {:membrane_file_plugin, "~> 0.16"}, {:membrane_mp3_mad_plugin, "~> 0.18.3"}, {:membrane_mp4_plugin, "~> 0.34.0"}, {:membrane_opus_plugin, "~> 0.20.2"}, {:membrane_raw_audio_format, "~> 0.12.0"}, {:membrane_rtc_engine, "~> 0.22.0"}, {:membrane_rtc_engine_webrtc, "~> 0.8.0"}, {:membrane_tee_plugin, "~> 0.12"},

16 replies

Gaming

Programming