Software Mansion•8mo ago

H264.FFmpeg.Decoded frames to MP4.Muxer

I'm attempting to open a local mp4, demux it, and write it back to an mp4, just to get started. I want to do stuff with overlay images and add sound clips once this basic thing is working. This is my spec:

    structure = [
      child(:video_source, %Membrane.File.Source{
        location: "example_data/example.mp4"
      })
      # Video Parser
      |> child(:demuxer, Membrane.MP4.Demuxer.ISOM)
      |> via_out(Pad.ref(:output, 1))
      |> child(:video_parser, %Membrane.H264.Parser{
        # Required for the FFmpeg.Decoder/Encoder
        output_stream_structure: :annexb
      })
      |> child(:video_decoder, Membrane.H264.FFmpeg.Decoder),
      # Audio Parser
      get_child(:demuxer)
      |> via_out(Pad.ref(:output, 2))
      |> child(:audio_parser, %Membrane.AAC.Parser{
        # Required for MP4.Muxer
        output_config: :esds
      }),
      # TODO: Overlays + audio clips
      # Mux to back to MP4
      child(:muxer, %Membrane.MP4.Muxer.ISOM{})
      |> child(:sink, %Membrane.File.Sink{location: "example_data/processed.mp4"}),
      get_child(:audio_parser) |> get_child(:muxer),
      get_child(:video_decoder)
      |> child(%Membrane.H264.FFmpeg.Encoder{
        preset: :ultrafast
      })
      |> get_child(:muxer)
    ]

    structure = [
      child(:video_source, %Membrane.File.Source{
        location: "example_data/example.mp4"
      })
      # Video Parser
      |> child(:demuxer, Membrane.MP4.Demuxer.ISOM)
      |> via_out(Pad.ref(:output, 1))
      |> child(:video_parser, %Membrane.H264.Parser{
        # Required for the FFmpeg.Decoder/Encoder
        output_stream_structure: :annexb
      })
      |> child(:video_decoder, Membrane.H264.FFmpeg.Decoder),
      # Audio Parser
      get_child(:demuxer)
      |> via_out(Pad.ref(:output, 2))
      |> child(:audio_parser, %Membrane.AAC.Parser{
        # Required for MP4.Muxer
        output_config: :esds
      }),
      # TODO: Overlays + audio clips
      # Mux to back to MP4
      child(:muxer, %Membrane.MP4.Muxer.ISOM{})
      |> child(:sink, %Membrane.File.Sink{location: "example_data/processed.mp4"}),
      get_child(:audio_parser) |> get_child(:muxer),
      get_child(:video_decoder)
      |> child(%Membrane.H264.FFmpeg.Encoder{
        preset: :ultrafast
      })
      |> get_child(:muxer)
    ]

I get this error:

[error] GenServer #PID<0.1357.0> terminating
** (Membrane.StreamFormatError) Stream format: %Membrane.H264{width: 640, height: 360, profile: nil, alignment: :au, nalu_in_metadata?: false, framerate: nil, stream_structure: :annexb} is not matching accepted format pattern "%Membrane.AAC{config: {:esds, _esds}}%Membrane.H264{stream_structure: {:avc1, _dcr}, alignment: :au}%Membrane.H265{stream_structure: {_hevc, _dcr}, alignment: :au}%Membrane.Opus{self_delimiting?: false}" in def_input_pad
for pad :input in Membrane.MP4.Muxer.ISOM

[error] GenServer #PID<0.1357.0> terminating
** (Membrane.StreamFormatError) Stream format: %Membrane.H264{width: 640, height: 360, profile: nil, alignment: :au, nalu_in_metadata?: false, framerate: nil, stream_structure: :annexb} is not matching accepted format pattern "%Membrane.AAC{config: {:esds, _esds}}%Membrane.H264{stream_structure: {:avc1, _dcr}, alignment: :au}%Membrane.H265{stream_structure: {_hevc, _dcr}, alignment: :au}%Membrane.Opus{self_delimiting?: false}" in def_input_pad
for pad :input in Membrane.MP4.Muxer.ISOM

The muxer expects :avc1 while the FFmpeg.Encoder outputs annexb. Either I'm missing a node, or I've misunderstood something somewhere. Any tips?

1 Reply

AskeOP•8mo ago

Ah... Adding an H264.Parser after the Encoder did the trick.. Was confused by the name.

Gaming

Programming

H264.FFmpeg.Decoded frames to MP4.Muxer