Software Mansion•8mo ago

Loop Audio File

I have a little Membrane Pipeline with a video, and some audio tracks. I want to add some background music to it. A short track, that just loops over and over, until the video is done. I've looked at doing something like:

          child(:mp3_source_bg, %Membrane.File.Source{
            location: state.background_audio.path,
            seekable?: true
          })

          child(:mp3_source_bg, %Membrane.File.Source{
            location: state.background_audio.path,
            seekable?: true
          })

and now I need to send a seek event to that child as far as I understand. However, I can't find any examples of that. How do I send this child an event?

    event = %Membrane.File.SeekSourceEvent{
      start: :bof,
      size_to_read: 100,
      last?: false
    }
    get_child(:mp3_source_bg) |> ????

    event = %Membrane.File.SeekSourceEvent{
      start: :bof,
      size_to_read: 100,
      last?: false
    }
    get_child(:mp3_source_bg) |> ????

Any tips?

14 Replies

mat_hek•8mo ago

Hi @Aske, events are meant to be sent by other elements through pads, using https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:event/0 So you'd need a filter right after the file that would intercept end_of_stream and send the seek event through its input. However, I'm not sure if it works, because concatenated MP3 files are not necessarily a valid MP3 - for example some ID3 tags are only valid at the beginning of file

AskeOP•8mo ago

I h

      |> child(:decoder_mp3_bg, Membrane.MP3.MAD.Decoder)
      |> child(:converter_bg, %Membrane.FFmpeg.SWResample.Converter{
        # input_stream_format: %Membrane.RawAudio{
        #   channels: 1,
        #   sample_rate: 48_000,
        #   sample_format: :s24le
        # },
        output_stream_format: %Membrane.RawAudio{
          channels: 2,
          sample_rate: 48_000,
          sample_format: :s16le
        }
      })
      |> via_in(:input)
      |> get_child(:mixer)

      |> child(:decoder_mp3_bg, Membrane.MP3.MAD.Decoder)
      |> child(:converter_bg, %Membrane.FFmpeg.SWResample.Converter{
        # input_stream_format: %Membrane.RawAudio{
        #   channels: 1,
        #   sample_rate: 48_000,
        #   sample_format: :s24le
        # },
        output_stream_format: %Membrane.RawAudio{
          channels: 2,
          sample_rate: 48_000,
          sample_format: :s16le
        }
      })
      |> via_in(:input)
      |> get_child(:mixer)

mat_hek•8mo ago

So the preferred solution would be to remove all the elements that process the MP3 and spawn them again

AskeOP•8mo ago

*I have more stuff after the file here Ah.. yeah, I see. It would still be an issue for the MAD.Decoder Can I remove and add named elements in one go? Or how would I go about doing that? It complains over this:

    children = [...from above]
    {[remove_children: [:mp3_source_bg, :decoder_mp3_bg, :converter_bg], spec: children], state}

    children = [...from above]
    {[remove_children: [:mp3_source_bg, :decoder_mp3_bg, :converter_bg], spec: children], state}

Duplicated names in children specification: [:mp3_source_bg, :decoder_mp3_bg, :converter_bg]

Duplicated names in children specification: [:mp3_source_bg, :decoder_mp3_bg, :converter_bg]

mat_hek•8mo ago

Nope, since the removal is not atomic. You need to wrap the names in tuples like {:mp3_source_bg, make_ref()}. Usually it's better to make them a group, like spec = {[child(..), ...], group: :bg_audio} and then do remove_children: :bg_audio.

AskeOP•8mo ago

Ah, did not know that was possible! Thanks for the help! Hmm.. I never got this to work. I keep running into this unlink error. It looks like it's running correctly at least once, but perhaps the last run isn't cleaned up correctly (playback: :stopped in the logs below):

[error] GenServer #PID<0.8714.0> terminating
** (Membrane.PadError) Tried to unlink a static pad :output, before it was linked. Static pads cannot be unlinked unless element is terminating
    (membrane_core 1.0.1) lib/membrane/core/element/pad_controller.ex:253: Membrane.Core.Element.PadController.handle_unlink/2
    (membrane_core 1.0.1) lib/membrane/core/element.ex:254: Membrane.Core.Element.handle_info/2
    (stdlib 4.3.1.3) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.3.1.3) gen_server.erl:1200: :gen_server.handle_msg/6
    (stdlib 4.3.1.3) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {Membrane.Core.Message, :handle_unlink, :output, []}
State: %Membrane.Core.Element.State{module: Membrane.FFmpeg.SWResample.Converter, name: {:converter_bg, #Reference<0.1532085730.115867653.111238>}, parent_pid: #PID<0.8559.0>, playback: :stopped, type: :filter, internal_state: %{input_stream_format: nil, input_stream_format_provided?: false, native: nil, output_stream_format: %Membrane.RawAudio{channels: 2, sample_rate: 48000, sample_format: :s16le}, pts_queue: [], queue: ""}, pad_refs: [:input], pads_info: %{input: %{accepted_formats_str:...

[error] GenServer #PID<0.8714.0> terminating
** (Membrane.PadError) Tried to unlink a static pad :output, before it was linked. Static pads cannot be unlinked unless element is terminating
    (membrane_core 1.0.1) lib/membrane/core/element/pad_controller.ex:253: Membrane.Core.Element.PadController.handle_unlink/2
    (membrane_core 1.0.1) lib/membrane/core/element.ex:254: Membrane.Core.Element.handle_info/2
    (stdlib 4.3.1.3) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.3.1.3) gen_server.erl:1200: :gen_server.handle_msg/6
    (stdlib 4.3.1.3) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {Membrane.Core.Message, :handle_unlink, :output, []}
State: %Membrane.Core.Element.State{module: Membrane.FFmpeg.SWResample.Converter, name: {:converter_bg, #Reference<0.1532085730.115867653.111238>}, parent_pid: #PID<0.8559.0>, playback: :stopped, type: :filter, internal_state: %{input_stream_format: nil, input_stream_format_provided?: false, native: nil, output_stream_format: %Membrane.RawAudio{channels: 2, sample_rate: 48000, sample_format: :s16le}, pts_queue: [], queue: ""}, pad_refs: [:input], pads_info: %{input: %{accepted_formats_str:...

I don't even kill the group right now, just add it:

  @impl true
  def handle_element_end_of_stream({:decoder_mp3_bg, ref}, :input, ctx, state) do
    if ctx.playback == :playing do
      children = [make_bg_audio_group(state)]

      {[spec: children], state}
    else
      {[], state}
    end
  end

  @impl true
  def handle_element_end_of_stream({:decoder_mp3_bg, ref}, :input, ctx, state) do
    if ctx.playback == :playing do
      children = [make_bg_audio_group(state)]

      {[spec: children], state}
    else
      {[], state}
    end
  end

I don't think I really understand what's happening here 😬

mat_hek•8mo ago

Are you sure you're removing the converter element?

AskeOP•8mo ago

I tried to comment out the remove_children, to rule out that cleanup being the source of the error. The only time I remove anything is with {[terminate: :normal], state}

mat_hek•8mo ago

Oh I see. Can you provide the code so I can reproduce?

AskeOP•8mo ago

Yes, will do. Shaving it down to the minimums now.

AskeOP•6mo ago

Attempted again using a Filter as you described, but I still cannot get it to work. This time with raw audio to eliminate the mp3 issue: https://gist.github.com/Doerge/77bc2a10e53da07a4124a861fb7526bd On the end_of_stream event in the Filter on the input pad, I send an event:

  @impl Membrane.Element.WithInputPads
  def handle_end_of_stream(:input, _ctx, state) do
    Logger.warning("end of stream in filter.input")

    seek_event =
      %Membrane.File.SeekSourceEvent{
        start: :bof,
        size_to_read: :infinity,
        last?: true
      }

    events =
      [
        {:event, {:input, seek_event}}
      ]

    {events, %{loops: state.loops - 1}}
  end

  @impl Membrane.Element.WithInputPads
  def handle_end_of_stream(:input, _ctx, state) do
    Logger.warning("end of stream in filter.input")

    seek_event =
      %Membrane.File.SeekSourceEvent{
        start: :bof,
        size_to_read: :infinity,
        last?: true
      }

    events =
      [
        {:event, {:input, seek_event}}
      ]

    {events, %{loops: state.loops - 1}}
  end

which looks like it is being received, because I see other elements (:mixer) receiving it in the logs:

[debug] <0.4856.0>/:loop_filter Received end of stream on pad :input
[debug] <0.4856.0>/:mixer Received event %Membrane.File.NewSeekEvent{} on pad {Membrane.Pad, :input, #Reference<0.448509681.3489398788.16262>}
[warning] end of stream in filter.input
[warning] got end of stream for loop_filter on pad input
[debug] <0.4856.0>/:mixer Received event %Membrane.File.NewSeekEvent{} on pad {Membrane.Pad, :input, #Reference<0.448509681.3489398788.16262>}

[debug] <0.4856.0>/:loop_filter Received end of stream on pad :input
[debug] <0.4856.0>/:mixer Received event %Membrane.File.NewSeekEvent{} on pad {Membrane.Pad, :input, #Reference<0.448509681.3489398788.16262>}
[warning] end of stream in filter.input
[warning] got end of stream for loop_filter on pad input
[debug] <0.4856.0>/:mixer Received event %Membrane.File.NewSeekEvent{} on pad {Membrane.Pad, :input, #Reference<0.448509681.3489398788.16262>}

But it's only received once, so I assume it didn't work out.. I think I'm unfortunately still misunderstanding something conceptually here, but what? I've dug through documentation, and examples, but can't find anything similar to what I want to do here. Any help would be appreciated.

Gist

Elixir Membrane Loop

Elixir Membrane Loop. GitHub Gist: instantly share code, notes, and snippets.

mat_hek•6mo ago

Well, so apparently I was wrong - you cannot rely on the end of stream for that, because when the source sends it, it cannot send anything else anymore. I added an EndOfSeek event for that purpose and here's a simple example showing how to use it. Run with elixir loop.exs

loop.exs

AskeOP•6mo ago

You're amazing! This also works for raw audio! Thank you! 🙏

mat_hek•6mo ago

glad to help 😉

Gaming

Programming

Loop Audio File