nickdichev_fw
nickdichev_fw
PostsComments
SMSoftware Mansion
Created by harrisi on 8/8/2024 in #membrane-help
Background loop with sound effect playback on event
You could decompose it by using the existing source elements like this: 
child(%Source{})
|> child(%LoopingBufferQueue{}) # this just loops forever
|> child(%Mixer{}) # this has a static pad which consumes the looping buffer, and has dynamic pads which would be linked for sound effects
|> child(%Sink{}) # output, in my case portaudio
child(%Source{})
|> child(%LoopingBufferQueue{}) # this just loops forever
|> child(%Mixer{}) # this has a static pad which consumes the looping buffer, and has dynamic pads which would be linked for sound effects
|> child(%Sink{}) # output, in my case portaudio
but yes that is the general idea. Sorry I cant share the code its from an app at work 🙂 I think the membrane guides have an introduction to creating elements (and iirc dynamic pads are covered).
13 replies
SMSoftware Mansion
Created by harrisi on 8/8/2024 in #membrane-help
Background loop with sound effect playback on event
In our pipeline we have implemented some similar mechanisms. At a high level our elements work as follows: We wrote a "LoopingBufferQueue" element. it is placed after a source and will read the entire input until it sees end_of_stream. Every time it sees a buffer before end_of_stream it stores in an in-memory ring buffer. The idea then is when the LoopingBufferQueue gets demand it just serves it from its ring buffer forever instead of propogating the end_of_stream from the source. Then we feed this LoopingBufferQueue input into a mixer (`custom element) which has a "primary" static pad and dynamic pads. Then you can wire it up such that the mixer will produce buffers from the primary input until an extra pad is added to the mixer. When the pad is added to the mixer it undergoes a state change and starts producing buffers from the extra input pad. The tricky thing to keep in mind are that generally you should produce buffers with a monotonicly increasing presentation time stamp. You must take that into account in several places with this architecture for example every time the LoopingBufferQueue loops the timestamp resets to 0. Similarly, when an extra pad is attached its timestamp will reset to 0.
13 replies
SMSoftware Mansion
Created by nickdichev_fw on 5/9/2023 in #membrane-help
Intermittent Failures with RTMP Sink
After some additional testing today, we notice there seems to be some correlation between the chunk size passed to the file source we are reading the h264/aac from and the file size. I also noticed that on each crash the buffer starts with the binary <<0, 0, ....>> -- is there maybe some correlation/ratio between the file size, chunk size, bit/sample rate we need to be concerned with?
4 replies
SMSoftware Mansion
Created by nickdichev_fw on 5/9/2023 in #membrane-help
Intermittent Failures with RTMP Sink
Any ideas on how to debug or fix this? Is there a constraint on the protocol which makes the element raise/crash on an error writing a frame? In our use case, I think we would rather drop the frame than crash the stream. However, I can understand there can be problems with dropping a keyframe.
4 replies
SMSoftware Mansion
Created by nickdichev_fw on 3/22/2023 in #membrane-help
Confusion on usage of MP4.Demuxer.ISOM
Sure!
12 replies
SMSoftware Mansion
Created by nickdichev_fw on 3/22/2023 in #membrane-help
Confusion on usage of MP4.Demuxer.ISOM
Hi @.wol yes I think it would be nice to eventually transition our pipelines to use the mp4 demuxer, however, this shouldn't be a blocker for a v1 release we have coming up.
12 replies
SMSoftware Mansion
Created by nickdichev_fw on 3/28/2023 in #membrane-help
Problems specifying toilet capacity of Realtimer
Thanks, that solves it! I am a bit confused on why there is such a large "toilet" accumulating on the rtmp sink as the pipeline starts up. I notice when I test against a local rtmp server running locally it will never overflow, however, when I point it at a server in the cloud (IVS) it will intermittently backup. Maybe some sort of delay with the tls handshake? Will have to take a deeper look!
4 replies
SMSoftware Mansion
Created by nickdichev_fw on 3/22/2023 in #membrane-help
Confusion on usage of MP4.Demuxer.ISOM
Awesome thanks Mat, if it helps here's a pipeline I've tried after incorporating some of your other tips, however, it's not working! Looking forward to seeing what you all come up with 🙂 https://gist.github.com/nickdichev-firework/ffdc3ca3bd6ce9efa6207dec952c5c47
12 replies
SMSoftware Mansion
Created by nickdichev_fw on 3/22/2023 in #membrane-help
Confusion on usage of MP4.Demuxer.ISOM
Hm, apologies -- I created a minimal example repo and could not reproduce. I tried again in my application repo and couldn't reproduce.. odd, sorry for the confusion. If you could point me in the direction of how to parse the mp4 payloads that would be much appreciated 🙂 thanks for all the awesome work you and the team have put out!!
12 replies
SMSoftware Mansion
Created by nickdichev_fw on 3/22/2023 in #membrane-help
Confusion on usage of MP4.Demuxer.ISOM
Awesome, thanks @mat_hek -- an example of a pipeline of elements downstream of the demuxer would be awesome. I haven't been able to figure it out, and right now I have a very hacky solution where I ffprobe to get stream indicies and then shell out to ffmpeg to manually demux the h264/aac -- not great. For 2, yes, that makes sense I've seen a similar pattern in some membrane examples but in the opposite direction (waiting for EOS from all elements before terminating the pipeline). In my initial testing I thought it would be a nice abstraction to do the demuxing and any decoding/parsing in a bin, however, I noticed that the bin doesn't get the :new_track child notification. When I converted that simple bin into a pipeline, suddenly I was getting the :new_track notification -- any ideas on why that might be happening? Having to hoist the demuxer into a pipeline isn't a problem, however, I just don't understand why the bin isn't handling the child notification.
12 replies
SMSoftware Mansion
Created by nickdichev_fw on 1/18/2023 in #membrane-help
Confusion on video compositor timestamp offset
Hi @mat_hek that change does work for the issue with the player! I pulled your changed down into my local deps and had to update it for core 0.11 though!
6 replies
SMSoftware Mansion
Created by nickdichev_fw on 1/18/2023 in #membrane-help
Confusion on video compositor timestamp offset
@.wol That does make a lot more sense thanks for clarifying that. I started playing around with the video merger plugin and made some progress there but I am experiencing some strange behavior that I can't really reason about with my limited knowledge of how membrane works. I uploaded some code snippets from the POC in a gist: https://gist.github.com/nickdichev-firework/fb41d5d789b13b056fe536a4c9a27aa5 There's two versions of the application in the gist: 1. merger_to_compositor.ex pipes the VideoCutAndMerge bin into the video compositor. This one displays strange behavior of: paused playback between the cue points, playback not starting for the second clip, and no clips/cue points after the third displaying. 2. merger_to_player.ex pipes the VideoCutAndMerge directly to the SDL player. This one behaves a bit closer to what I expect to happen, however, there is a strange black screen between the second and third clips. Additionally, a new window is opened for every clip -- unlike the version with the compositor. Could you take a look and let me know if the application makes sense at all for what I'm trying to accomplish? After re-reading your comment I think maybe I misinterpreted and should be directly feeding the RawVideo from just a VideoCutter into the compositor? Thanks so much, appreciate the help 🙂
6 replies