Aske Posts - Answer Overflow

Aske

•Created by Aske on 7/4/2024 in #membrane-help

H265 choppy playback in QuickTime on Mac

I have issues with h265 files and I've reduced the bug to just demuxing and muxing an mp4. The input file plays fine in QuickTime, but the output plays with stutters. It does play fine in another player like IINA. See here for a simple demux/mux pipeline: https://gist.github.com/Doerge/903243abde51bda3468f20ca27fc966f QuickTime is a piece of garbage, but it's the default player for Mac users, so I can't get around it. Anyone experienced this, and resolved it somehow? Files below are input and output file respectively.

14 replies

SMSoftware Mansion

•Created by Aske on 5/6/2024 in #membrane-help

Loop Audio File

I have a little Membrane Pipeline with a video, and some audio tracks. I want to add some background music to it. A short track, that just loops over and over, until the video is done. I've looked at doing something like:

          child(:mp3_source_bg, %Membrane.File.Source{
            location: state.background_audio.path,
            seekable?: true
          })

          child(:mp3_source_bg, %Membrane.File.Source{
            location: state.background_audio.path,
            seekable?: true
          })

and now I need to send a seek event to that child as far as I understand. However, I can't find any examples of that. How do I send this child an event?

    event = %Membrane.File.SeekSourceEvent{
      start: :bof,
      size_to_read: 100,
      last?: false
    }
    get_child(:mp3_source_bg) |> ????

    event = %Membrane.File.SeekSourceEvent{
      start: :bof,
      size_to_read: 100,
      last?: false
    }
    get_child(:mp3_source_bg) |> ????

Any tips?

21 replies

SMSoftware Mansion

•Created by Aske on 4/11/2024 in #membrane-help

Dynamically starting children to Demux Mp4 tracks

I want to convert this to take arbitrary user uploaded Mp4 files where the tracks can have different indexes:

    structure = [
      child(:video_source, %Membrane.File.Source{
        location: @input_file
      })
      |> child(:demuxer, Membrane.MP4.Demuxer.ISOM)
      # Video
      |> via_out(Pad.ref(:output, 1))
      |> child(:vido_tmp, %Membrane.H264.Parser{
        output_stream_structure: :annexb
      })
      |> do_video()
      |> child(:video_out, %Membrane.H264.Parser{
        generate_best_effort_timestamps: %{framerate: {25, 1}},
        output_stream_structure: :avc1
      }),
      # Audio
      get_child(:demuxer)
      |> via_out(Pad.ref(:output, 2))
      |> do_audio(),
      # Mux
      child(:muxer, Membrane.MP4.Muxer.ISOM)
      |> child(:sink, %Membrane.File.Sink{location: @out_file})
      get_child(:video_out) |> get_child(:muxer),
      get_child(:audio_out) |> get_child(:muxer)
    ]

    structure = [
      child(:video_source, %Membrane.File.Source{
        location: @input_file
      })
      |> child(:demuxer, Membrane.MP4.Demuxer.ISOM)
      # Video
      |> via_out(Pad.ref(:output, 1))
      |> child(:vido_tmp, %Membrane.H264.Parser{
        output_stream_structure: :annexb
      })
      |> do_video()
      |> child(:video_out, %Membrane.H264.Parser{
        generate_best_effort_timestamps: %{framerate: {25, 1}},
        output_stream_structure: :avc1
      }),
      # Audio
      get_child(:demuxer)
      |> via_out(Pad.ref(:output, 2))
      |> do_audio(),
      # Mux
      child(:muxer, Membrane.MP4.Muxer.ISOM)
      |> child(:sink, %Membrane.File.Sink{location: @out_file})
      get_child(:video_out) |> get_child(:muxer),
      get_child(:audio_out) |> get_child(:muxer)
    ]

I tried converting the above to:

  @impl true
  def handle_init(_ctx, _path) do
    structure = [
      child(:video_source, %Membrane.File.Source{
        location: @input_file
      })
      |> child(:demuxer, Membrane.MP4.Demuxer.ISOM),
      # Mux
      child(:muxer, Membrane.MP4.Muxer.ISOM)
      |> child(:sink, %Membrane.File.Sink{location: @out_file})
    ]

    {[spec: structure], %{}}
  end

  @impl true
  def handle_child_notification({:new_tracks, tracks}, :demuxer, _ctx, state) do
    spec =
      tracks
      |> Enum.map(fn
        {track_id, %Membrane.H264{}} ->
          get_child(:demuxer)
          |> via_out(Pad.ref(:output, track_id))
          |> do_video()

        {track_id, %Membrane.AAC{}} ->
          get_child(:demuxer)
          |> via_out(Pad.ref(:output, track_id))
          |> do_audio()
      end)

    {[spec: spec], state}
  end

  @impl true
  def handle_init(_ctx, _path) do
    structure = [
      child(:video_source, %Membrane.File.Source{
        location: @input_file
      })
      |> child(:demuxer, Membrane.MP4.Demuxer.ISOM),
      # Mux
      child(:muxer, Membrane.MP4.Muxer.ISOM)
      |> child(:sink, %Membrane.File.Sink{location: @out_file})
    ]

    {[spec: structure], %{}}
  end

  @impl true
  def handle_child_notification({:new_tracks, tracks}, :demuxer, _ctx, state) do
    spec =
      tracks
      |> Enum.map(fn
        {track_id, %Membrane.H264{}} ->
          get_child(:demuxer)
          |> via_out(Pad.ref(:output, track_id))
          |> do_video()

        {track_id, %Membrane.AAC{}} ->
          get_child(:demuxer)
          |> via_out(Pad.ref(:output, track_id))
          |> do_audio()
      end)

    {[spec: spec], state}
  end

but now the demuxer doesn't run, because there are no outputs to pull data.. How do I add dynamic children, based on the track types?

5 replies

SMSoftware Mansion

•Created by Aske on 4/10/2024 in #membrane-help

H264.FFmpeg.Decoded frames to MP4.Muxer

I'm attempting to open a local mp4, demux it, and write it back to an mp4, just to get started. I want to do stuff with overlay images and add sound clips once this basic thing is working. This is my spec:

    structure = [
      child(:video_source, %Membrane.File.Source{
        location: "example_data/example.mp4"
      })
      # Video Parser
      |> child(:demuxer, Membrane.MP4.Demuxer.ISOM)
      |> via_out(Pad.ref(:output, 1))
      |> child(:video_parser, %Membrane.H264.Parser{
        # Required for the FFmpeg.Decoder/Encoder
        output_stream_structure: :annexb
      })
      |> child(:video_decoder, Membrane.H264.FFmpeg.Decoder),
      # Audio Parser
      get_child(:demuxer)
      |> via_out(Pad.ref(:output, 2))
      |> child(:audio_parser, %Membrane.AAC.Parser{
        # Required for MP4.Muxer
        output_config: :esds
      }),
      # TODO: Overlays + audio clips
      # Mux to back to MP4
      child(:muxer, %Membrane.MP4.Muxer.ISOM{})
      |> child(:sink, %Membrane.File.Sink{location: "example_data/processed.mp4"}),
      get_child(:audio_parser) |> get_child(:muxer),
      get_child(:video_decoder)
      |> child(%Membrane.H264.FFmpeg.Encoder{
        preset: :ultrafast
      })
      |> get_child(:muxer)
    ]

    structure = [
      child(:video_source, %Membrane.File.Source{
        location: "example_data/example.mp4"
      })
      # Video Parser
      |> child(:demuxer, Membrane.MP4.Demuxer.ISOM)
      |> via_out(Pad.ref(:output, 1))
      |> child(:video_parser, %Membrane.H264.Parser{
        # Required for the FFmpeg.Decoder/Encoder
        output_stream_structure: :annexb
      })
      |> child(:video_decoder, Membrane.H264.FFmpeg.Decoder),
      # Audio Parser
      get_child(:demuxer)
      |> via_out(Pad.ref(:output, 2))
      |> child(:audio_parser, %Membrane.AAC.Parser{
        # Required for MP4.Muxer
        output_config: :esds
      }),
      # TODO: Overlays + audio clips
      # Mux to back to MP4
      child(:muxer, %Membrane.MP4.Muxer.ISOM{})
      |> child(:sink, %Membrane.File.Sink{location: "example_data/processed.mp4"}),
      get_child(:audio_parser) |> get_child(:muxer),
      get_child(:video_decoder)
      |> child(%Membrane.H264.FFmpeg.Encoder{
        preset: :ultrafast
      })
      |> get_child(:muxer)
    ]

I get this error:

[error] GenServer #PID<0.1357.0> terminating
** (Membrane.StreamFormatError) Stream format: %Membrane.H264{width: 640, height: 360, profile: nil, alignment: :au, nalu_in_metadata?: false, framerate: nil, stream_structure: :annexb} is not matching accepted format pattern "%Membrane.AAC{config: {:esds, _esds}}%Membrane.H264{stream_structure: {:avc1, _dcr}, alignment: :au}%Membrane.H265{stream_structure: {_hevc, _dcr}, alignment: :au}%Membrane.Opus{self_delimiting?: false}" in def_input_pad
for pad :input in Membrane.MP4.Muxer.ISOM

[error] GenServer #PID<0.1357.0> terminating
** (Membrane.StreamFormatError) Stream format: %Membrane.H264{width: 640, height: 360, profile: nil, alignment: :au, nalu_in_metadata?: false, framerate: nil, stream_structure: :annexb} is not matching accepted format pattern "%Membrane.AAC{config: {:esds, _esds}}%Membrane.H264{stream_structure: {:avc1, _dcr}, alignment: :au}%Membrane.H265{stream_structure: {_hevc, _dcr}, alignment: :au}%Membrane.Opus{self_delimiting?: false}" in def_input_pad
for pad :input in Membrane.MP4.Muxer.ISOM

The muxer expects :avc1 while the FFmpeg.Encoder outputs annexb. Either I'm missing a node, or I've misunderstood something somewhere. Any tips?

2 replies

Gaming

Programming