Grab keyframe image data from h264 stream?
We are looking at some h264 coming from an RTSP stream. Membrane is doing a fine job with the HLS demo turning it into HLS. But we want to grab images for processing and such. I didn't find anything conclusive on how to do this. I remember being able to do something with keyframes and images but can't find it.
16 Replies
Hah! essentially I want the thing you do in the iOS blog post from way back but as an example 😄
Hi @lawik , if you decode an H264 stream you get a series of YUV pictures. I worked with the author of Elixir Image on using the library to manipulate them and the most straightforward way requires converting YUV to RGB. The guy did that, but it's quite slow, see https://github.com/elixir-image/image/discussions/121. Since my use case was specifically about applying an image overlay, I managed to do that without converting to RGB, see https://github.com/membraneframework/membrane_overlay_plugin (merged today)
GitHub
GitHub - membraneframework/membrane_overlay_plugin: Filter for appl...
Filter for applying overlay image or text on top of video - membraneframework/membrane_overlay_plugin
Regarding key frames, instead of decoding the whole stream you can throw away all frames that are not key frames and decode only the key frames, for example if you want thumbnails, that's going to save a lot of CPU compared to decoding the whole stream and then choosing frames for thumbnails.
YUV is fine, I want to feed it into a model. I can change color spaces and stuff if I need to..
I can't find a scissors plugin example for keyframes only..
cocoa's evision can use opencv to give you yuv to rgb I think
Anyway, I have a tee working, I want to filter to only keep keyframes, then decode and turbojpeg them or something
Ah yeah, this plugin doesn't have the best API :/ When I needed to have key frames last time I wrote a simple filter. It's enough to check for
buffer.metadata.h264.key_frame? and only send these buffersgotcha
cocoa's evision can use opencv to give you yuv to rgb I thinkthat's possible, I didn't want to include whole evision for that purpose
Hmm, getting errors due to some bad stuff going into the ffmpeg decoder..
Same :send_pkt from here: https://elixirforum.com/t/membrane-rtsp-source/28461/4
Elixir Programming Language Forum
Membrane RTSP source
@mat-hek I’ve successfully wired up my first pipeline. It seems like it almost works, but i’ve ran into an issue. My stream only supports interleaving, however the membrane-element-rtp-h264 element does not support interleaving. There is only one channel, so i tried to hack it together but just filtering out the interleaving packets with someth...
The problem there was that SPS and PPS were not in the stream
In that case they need to be passed like here https://github.com/membraneframework/membrane_demo/blob/master/rtsp_to_hls/lib/pipeline.ex#L103
GitHub
membrane_demo/rtsp_to_hls/lib/pipeline.ex at master · membraneframe...
Examples of using the Membrane Framework. Contribute to membraneframework/membrane_demo development by creating an account on GitHub.
I wonder if the camera was giving us rtsp 2 or something.. will look at that though
Hah, Nope those cameras don't send that
Could we get Membrane to do this?
https://stackoverflow.com/questions/72343926/identify-if-an-rtsp-stream-contains-pps-and-sps-information
Stack Overflow
Identify if an RTSP Stream contains PPS and SPS information
I am ingesting different kinds of security camera RTSP streams with gstreamer. In most of the cases I use a pipeline similar to this
gst-launch-1.0 rtspsrc location=rtsp://<username:password@$RT...
I might have a chance to contrib that. We will see
I won't have access to the device for a bit. Curious if we could detect that failure and just make it clear what the problem is, or solve it for the dev
We plan to create an RTSP source bin that would be passed an URL and did everything for you, no time for that yet though :/ contributions very welcome as always 😉