Software Mansion•3mo ago

Background loop with sound effect playback on event

Hello, I'm using Membrane for background music playing locally (portaudio), and I want to be able to play sound effects at will (as in, not related to any timer, just events like pressing a key, for example). Would it make sense to have multiple pipelines, one for each sound effect? I think it would make more sense to have my background loop pipeline and a separate sound effect pipeline, which can play any of the various sound effects, but I'm not sure how that would work. Any help would be great, thanks!

7 Replies

harrisi•3mo ago

I guess I'm also confused on how to keep the background loop playing when the sound effects play. I suppose multiple sound effects could play overlapped as well. I'd still like some help with this if anyone has any suggestions, but what I'm doing for now is I have my looping background as it's own pipeline, and another sound effect pipeline that takes a path. in places where I want to play a sound effect, I just call Membrane.Pipeline.start/3. the background pipeline is in my application's supervision tree but the sound effect pipeline isn't. if anyone wants to see the code, https://github.com/harrisi/elixir_breakout/pull/1/files is just membrane stuff, shouldn't need to look at the rest of the project to understand.

Aske•3mo ago

One thing that made it a lot easier for me was to not use MP3s as input, but RAW audio instead. When you seek in the MP3 file you risk incomplete data being emitted to your MP3 decoder. To "merge" multiple audios you need the Membrane.AudioMixer

harrisi•3mo ago

That's true, I do have some various warnings about that. Good tip! Is this actually needed? If I have two different pipelines to play the audio, it seems to play things correctly. It does seem like mixing the audio could improve things, but I'm not sure how to do that with the background loop. Maybe dynamic pads or something?

nickdichev_fw•3mo ago

In our pipeline we have implemented some similar mechanisms. At a high level our elements work as follows: We wrote a "LoopingBufferQueue" element. it is placed after a source and will read the entire input until it sees end_of_stream. Every time it sees a buffer before end_of_stream it stores in an in-memory ring buffer. The idea then is when the LoopingBufferQueue gets demand it just serves it from its ring buffer forever instead of propogating the end_of_stream from the source. Then we feed this LoopingBufferQueue input into a mixer (`custom element) which has a "primary" static pad and dynamic pads. Then you can wire it up such that the mixer will produce buffers from the primary input until an extra pad is added to the mixer. When the pad is added to the mixer it undergoes a state change and starts producing buffers from the extra input pad. The tricky thing to keep in mind are that generally you should produce buffers with a monotonicly increasing presentation time stamp. You must take that into account in several places with this architecture for example every time the LoopingBufferQueue loops the timestamp resets to 0. Similarly, when an extra pad is attached its timestamp will reset to 0.

harrisi•3mo ago

ohh, so you have something like this?:

child(%LoopingBufferQueue{source: "wherever"}) # this just loops forever
|> child(%Mixer{}) # this has a static pad which consumes the looping buffer, and has dynamic pads which would be linked for sound effects
|> child(%Sink{}) # output, in my case portaudio

child(%LoopingBufferQueue{source: "wherever"}) # this just loops forever
|> child(%Mixer{}) # this has a static pad which consumes the looping buffer, and has dynamic pads which would be linked for sound effects
|> child(%Sink{}) # output, in my case portaudio

if I understand correctly, the Mixer would have dynamic pads which would mix (somehow) the looping buffer when linked. I haven't used dynamic pads (I just started with membrane yesterday), but that flow does make sense to me. there's some hand waving, but I think I kinda get it.

nickdichev_fw•3mo ago

You could decompose it by using the existing source elements like this:

child(%Source{})
|> child(%LoopingBufferQueue{}) # this just loops forever
|> child(%Mixer{}) # this has a static pad which consumes the looping buffer, and has dynamic pads which would be linked for sound effects
|> child(%Sink{}) # output, in my case portaudio

child(%Source{})
|> child(%LoopingBufferQueue{}) # this just loops forever
|> child(%Mixer{}) # this has a static pad which consumes the looping buffer, and has dynamic pads which would be linked for sound effects
|> child(%Sink{}) # output, in my case portaudio

but yes that is the general idea. Sorry I cant share the code its from an app at work 🙂 I think the membrane guides have an introduction to creating elements (and iirc dynamic pads are covered).

harrisi•3mo ago

that makes sense. thanks a lot for the suggestion!

Gaming

Programming

Background loop with sound effect playback on event