Al
Al
PostsComments
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
Hi @Radosław, once again thank you! Your working version helped me figure out why my approach wasn't working - I'd cloned the demuxer repo back in December and tried to upgrade it to core 1.0, not entirely successfully it would appear! I should have checked back with the original and picked up the uplift to core 1.0.
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
Hi @Radosław I've only just managed to get some time to look at this and am still struggling to get this to work with a ts file. I've stripped it back to a simple variation on the SDL file player example, all works if I play your example file through it. When I try the ts file I get it all working up to the point I get the :mpeg_ts_pmt child notification. Then it stops. I think the issue is that the output pin of the demuxer is flow_control: :manual so I'm missing something to trigger the flow once the video output is hooked up to the demuxer in the pipeline. This worked fine with the http_adaptive_stream plugin but I guess that's because the SinkBin triggered the flow. Is my intuition here correct?
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
That's great, thank you. I'd almost got there. I think this will be a great help, next step is to try and modify it to take a udp stream rather than a file - I'll let you know how I get on 🙂
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
Ok, I've decided to simplify everything for my benefit. I've cloned the videoroom example that comes with the RTC engine. I'd now like to add a file endpoint when the room is instantiated so that in the browser when I join a room I get a video feed for the file endpoint. My possibly dumb question - where do I add the file endpoint? I've tried to add it at the end of the room GenServer init: 
...
{:ok, pid} = Membrane.RTC.Engine.start(rtc_engine_options, [])
Engine.register(pid, self())
Process.monitor(pid)

file_peer_id = UUID.uuid4()
file_endpoint = create_file_endpoint(pid, file_peer_id)
:ok = Engine.add_endpoint(pid, file_endpoint, id: file_peer_id)
...
...
{:ok, pid} = Membrane.RTC.Engine.start(rtc_engine_options, [])
Engine.register(pid, self())
Process.monitor(pid)

file_peer_id = UUID.uuid4()
file_endpoint = create_file_endpoint(pid, file_peer_id)
:ok = Engine.add_endpoint(pid, file_endpoint, id: file_peer_id)
...
with create_file_endpoint defined as: 
defp create_file_endpoint(rtc_engine, peer_id) do
  IO.puts("Adding File #{inspect peer_id}")
  video_track_config = %Isr.Pipeline.TrackConfig{
    type: :video,
    encoding: :H264,
    clock_rate: 90_000,
    fmtp: %ExSDP.Attribute.FMTP{
      pt: 96
    },
    opts: [framerate: {60, 1}]
  }

  file_endpoint =
    %Isr.Pipeline.Endpoint.File{
      rtc_engine: rtc_engine,
      file_path: "./test/fixtures/video.h264",
      track_config: video_track_config,
      payload_type: 96,
      autoplay: true
    }

end
defp create_file_endpoint(rtc_engine, peer_id) do
  IO.puts("Adding File #{inspect peer_id}")
  video_track_config = %Isr.Pipeline.TrackConfig{
    type: :video,
    encoding: :H264,
    clock_rate: 90_000,
    fmtp: %ExSDP.Attribute.FMTP{
      pt: 96
    },
    opts: [framerate: {60, 1}]
  }

  file_endpoint =
    %Isr.Pipeline.Endpoint.File{
      rtc_engine: rtc_engine,
      file_path: "./test/fixtures/video.h264",
      track_config: video_track_config,
      payload_type: 96,
      autoplay: true
    }

end
But I clearly still misunderstand the webrtc process because I then get an error on the client side because it can't find the endpoint when handling the tracksAdded mediaEvent because there isn't an entry in this.idToEndpoint map for the new endpointId.
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
Thanks for your help btw, very much appreciated!
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
Sorry, I should have been clearer - yes my original H264.Parser was configured with :au alignment. But following your comment I changed it to :nalu and it had no impact. The application is aimed at interfacing to a uav camera with the video payload delivered as an mpegts stream over udp. I've been using ex_nvr as the model. In particular the main pipeline https://github.com/evercam/ex_nvr/blob/master/apps/ex_nvr/lib/ex_nvr/pipelines/main.ex, the web_rtc bin element https://github.com/evercam/ex_nvr/blob/master/apps/ex_nvr/lib/ex_nvr/pipeline/output/web_rtc.ex and associated stream endpoint https://github.com/evercam/ex_nvr/blob/master/apps/ex_nvr/lib/ex_nvr/pipeline/output/webrtc/stream_endpoint.ex. It does seem quite complicated. Perhaps, as you suggest, it might be easier to instantiate a room GenServer and add a single stream endpoint.
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
I''m using :nalu alignment: 
|> child(:video_parser, %Membrane.H264.Parser{
    generate_best_effort_timestamps: %{framerate: {30, 1}},
    output_alignment: :nalu
})
|> child(:video_parser, %Membrane.H264.Parser{
    generate_best_effort_timestamps: %{framerate: {30, 1}},
    output_alignment: :nalu
})
So I don't think it's that. I can only think that my pipeline isn't getting hooked up end to end since the Membrane.Debug.Filter I've added on the video out of the demuxer isn't showing any data - so i guess there's nothing generating a demand (at least that's what my limited understanding of Membrane is suggesting to me!).
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
I guess I must be doing something dumb with my pipeline. Trying to use Membrane.Debug.Filter to inspect the output from the demuxer and I get nothing: 
get_child(:demuxer)
|> via_out(Pad.ref(:output, {:stream_id, sid}))
|> child(:filter, %Membrane.Debug.Filter{
  handle_buffer: &IO.inspect(&1, label: "buffer"),
  handle_stream_format: &IO.inspect(&1, label: "stream format")
})
|> child(:video_parser, %Membrane.H264.Parser{
  generate_best_effort_timestamps: %{framerate: {30, 1}},
  output_alignment: :au
})
|> via_in(:input, toilet_capacity: 1_000)
|> child(:realtimer, Membrane.Realtimer)
|> via_in(Pad.ref(:input, :main_stream), options: [media_track: track])
|> get_child(:webrtc)
get_child(:demuxer)
|> via_out(Pad.ref(:output, {:stream_id, sid}))
|> child(:filter, %Membrane.Debug.Filter{
  handle_buffer: &IO.inspect(&1, label: "buffer"),
  handle_stream_format: &IO.inspect(&1, label: "stream format")
})
|> child(:video_parser, %Membrane.H264.Parser{
  generate_best_effort_timestamps: %{framerate: {30, 1}},
  output_alignment: :au
})
|> via_in(:input, toilet_capacity: 1_000)
|> child(:realtimer, Membrane.Realtimer)
|> via_in(Pad.ref(:input, :main_stream), options: [media_track: track])
|> get_child(:webrtc)
This worked fine when the output was to HLS: 
get_child(:demuxer)
|> via_out(Pad.ref(:output, {:stream_id, sid}))
|> child(:video_nal_parser, %Membrane.H264.Parser{
  generate_best_effort_timestamps: %{framerate: {30, 1}},
  output_alignment: :au
})
|> via_in(Pad.ref(:input, :video),
  options: [encoding: :H264, segment_duration: Membrane.Time.seconds(4)]
)
|>child(:video_out,
  %Membrane.HTTPAdaptiveStream.SinkBin{
    manifest_module: Membrane.HTTPAdaptiveStream.HLS,
    mode: :vod,
    target_window_duration: :infinity,
    persist?: true,
    storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{directory: state.hls_path}
  },
  get_if_exists: true)
get_child(:demuxer)
|> via_out(Pad.ref(:output, {:stream_id, sid}))
|> child(:video_nal_parser, %Membrane.H264.Parser{
  generate_best_effort_timestamps: %{framerate: {30, 1}},
  output_alignment: :au
})
|> via_in(Pad.ref(:input, :video),
  options: [encoding: :H264, segment_duration: Membrane.Time.seconds(4)]
)
|>child(:video_out,
  %Membrane.HTTPAdaptiveStream.SinkBin{
    manifest_module: Membrane.HTTPAdaptiveStream.HLS,
    mode: :vod,
    target_window_duration: :infinity,
    persist?: true,
    storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{directory: state.hls_path}
  },
  get_if_exists: true)
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
Running ffprobe on the test fixture video.h264 file that's part of the file source endpoint test (https://github.com/jellyfish-dev/membrane_rtc_engine/tree/master/file/test/fixtures) shows that it's profile is High: 
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
...
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
...
So I wonder is it not the profile that's the issue.
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
Transcoding with ffmpeg to get this with ffprobe: 
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=Constrained Baseline
codec_type=video
codec_tag_string=[27][0][0][0]
codec_tag=0x001b
...
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=Constrained Baseline
codec_type=video
codec_tag_string=[27][0][0][0]
codec_tag=0x001b
...
I still get Didn't receive keyframe for variant: high in 500. Retrying..
23 replies
SMSoftware Mansion
Created by Al on 2/19/2024 in #membrane-help
WebRTC stream not working
@Radosław, thanks looks like that may be the problem. ffprobe shows it's Main profile: 
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=Main
codec_type=video
...
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=Main
codec_type=video
...
23 replies
SMSoftware Mansion
Created by Al on 12/28/2023 in #membrane-help
HTTPAdaptiveStream issue with hls.js
I swapped hls.js with video.js and it works which suggests that it was an hls.js issue. Τurning on logging for hls.js I was getting the following: [log] > live playlist - outdated PTS, unknown sliding which seems to be an issue others have seen with hls.js. (e.g. https://github.com/video-dev/hls.js/issues/583)
4 replies
SMSoftware Mansion
Created by Al on 12/5/2023 in #membrane-help
RTP stream
At the moment just passing the output from the RTP SessionBin to the demuxer from the membrane_mpeg_ts_plugin seems to work fine.
12 replies
SMSoftware Mansion
Created by Al on 12/5/2023 in #membrane-help
RTP stream
Video from a UAV in STANAG 4609 format
12 replies
SMSoftware Mansion
Created by Al on 12/5/2023 in #membrane-help
RTP stream
Thanks @mat_hek . I can now see what I was doing wrong - adding the |> child(:out, %Membrane.File.Sink{location: "tmp.ts"}) causes the error. As you say I need to add this once the pipeline has received handle_child_notification({:new_rtp_stream, ssrc, 33, _extensions}, :rtp, _ctx, state). Am I correct in assuming I'll need to implement a deplayloader for TS packets?
12 replies