Mix receiver streams efficiently
Anyone know how to mix receiver streams efficiently (in real time by keeping it a stream, no ffmpeg)?
I tried using an npm library, audio-mixer, works but sounds horrible. Is there other libraries, maybe something built in...? I heard web audio api is a thing, but dont understand how to combine that with streams and then use the mixed stream in my node.js app.
Any other ideas or examples?
101 Replies
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Solutions would be great in v14 since the audio player/receiver is a little more funny to set up there in case you could send example code. Else, every input is welcome ofc
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Thats exactly what I want
You can merge streams in real time like that? I thought ffmpeg was for files only and not streams like voice receiver streams on Discord. Waiting for all users to stop talking so I can then merge everything together to create a readable stream out of that could take from a second to minutes if they dont stop talking for a while.
audio-mixer takes buffers from streams and then creates another readable stream out of that, which I can then use with createAudioResoure() and then play it
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Ive been stuck on that for a while lmao like days
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Well it works but when multiple people start talking it creates a constant cracking sound in the background. Imagine your headphone cables are not connected properly to the pc, thats how it sounds like but a little worse.
U dont understand anything anymore like that
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yeah it sounds perfectly fine when only someone talks
So I don't really think it's that
Maybe I could test out if creating two read streams of already existing audio files and usind audio-mixer on them would do the same thing. Cause if thats the case, the package is bad
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Great answers
I can try lowering the volume thats easily done, changing the sample rate and bit depth aint difficult too but I think they are the same from what I know.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yeah
About the github thing: Im not at home rn and all but I will be in like 4 hours from now on so I can send it to u by then
As well as give you other updates if I have any
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
The code is very bad tho xD it just listens to a message event by a user and then joins the vc and thats where the magic happens. Its only a badly written code so excuse me for that lmfao
Nice thanks
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I might be understanding something wrong since I'm rather new to streams, but you cant extract a readstream out of that right?
Like something I could use to instantly replay
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
So thats what PassThrough streams are for I see
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I understand the concept it's pretty smart. But honestly I don't really understand how that'd look in code like as I havent worked with discord.js in a while and also startet coding just recently. Is there maybe an example to what you have explained somewhere?😅
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Not at all to be very honest
All I have done is replay audio and receive audio through discord and maybe use audio-mixer (not effectively as it seems), thats all
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Well the entire "project" is just a single index.js file. (ofc also with the package json, dotenv and all). Im not on my computer right now but will be in around 40 minutes.
I was testing at how far I'd come with replaying audio in real time as this would be something I could maybe use in the future. I can send you the git link once I'm there, the code is just very ugly lmao
If you could help that would be very awesome of you. The help I already received is very nice of all of you
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
https://github.com/Jiron/Discord-Audio-Replay-Test
That would be the necessary part. All it needs is a simple
npm i and you can start it through node(mon). the login function is ran on the last line and needs a dotenv file with a token in it, but of course can just be replaced with the actual token string.
Line 18 checks for the author id. Currently its mine but you might want to change it to yours or completely change it.
I took 99% of the playStream function from an other dev that tried the same. He couldn't get it to keep playing audio after the vc was silence for a bit so I looped everything with the initAudioPlayer part which is very terribly solved (but works somehow).
I extracted everything that was related to the replay bot into the git repo from my original code, since the actual code had a few more off topic functions in it and is currently set on private in git but thats like everything I had for the bot.
(the second last line is the worst thing I've done in my coding "career" since I couldnt fix it but it was also not really affecting the code)GitHub
GitHub - Jiron/Discord-Audio-Replay-Test
Contribute to Jiron/Discord-Audio-Replay-Test development by creating an account on GitHub.
vcmembers doesnt do anything, its from something else as well oops
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Oh god yeah sure take your time. Have a good night and thanks for the help
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yeah I'll look into it :) I never coded with C++ but with C# but just by looking at it its not really hard to understand. Thank you a lot!
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I see
Well then I'll look if I can install it as a package
And then apply what was mentioned there?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yeah I'll have to look into that. I've seen how someone extended Transform to read out a json readStream. I don't really understand everything yet though and especially not how to apply it to my code.
Anyways, why has my code been working when only a single person was speaking though?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I understand now
Basically when you have two chunks, it will take the size of largest one in order not to cut it, but the smaller chunk doesnt fit into that size since it's too small.
It's like having chunk1 with size 15 and chunk2 with size 13
13 won't fit into 15 entirely so it will be silent for 2 (just a random example). This happens many times in a row and that causes the stuttering but with a continuous stream you can control the stream so that it will always send a defined chunk size and like that audio-mixer's largest chunk size is the same as the smallest chunk size and all chunks fill in the "gap".
That's basically what Transform does then? Is that correct?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
That too dam-
Well I guess my level is not high enough to understand all that yet. I understand the concepts, kind of. Pretty frustrating how much you have to know just to combine two simple streams from an npm package
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yea-
how come there is no solution only based on exactly that though yet
Ive seen many people asking the same thing and no one received (proper) answers. I think yours rn is the best
As well as Kirdocks example
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
But what actually is off?
The timing, the size (as in theres a audio chunk with better quality taking up more space)
Like it's always 20ms from what I know. Only thing I could think off is maybe delays or something like if a chunk got sent a bit later but that would make the entire discord app not work in general as it would stutter
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I guess thats the only thing I can do-. I heard from multiple people that ffmpeg could solve this but it wouldnt work with discord streams in real time right?
Or there's no other simpler way like using another package whatsoever right?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I've seen a stackoverflow example on how to combine streams through passthrough. Wouldn't work with audio streams though, right?
https://stackoverflow.com/questions/16431163/concatenate-two-or-n-streams
Stack Overflow
Concatenate two (or n) streams
2 streams:
Given readable streams stream1 and stream2, what's an idiomatic (concise) way to get a stream containing stream1 and stream2 concatenated?
I cannot do stream1.pipe(outStream); stream2....
Second last answer even mentions audio and is quite up to date, that'd still lead to the same issue I have with audio mixer tho right
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Oh so like playing after each other
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Alright sadge
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Ohhhh I see
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
True that would make a long audio file play in like a second (if it even is possible to do)
Oh one thing that I also have is that i get a "possible EventEmitter memory leak detected" error.
That one coming from the Mixer. I then destroyed the mixer and re created it every time the input went silent because of that problem but that only solves it partly. Now I get that error in the network section, specifically the part with the connection.stateChanged event on the last line so I then proceeded to add a tiny line of code which removes warnings as a very bad temporary solution
Im almost tempted not to use audio-mixer at all and finding a better alternative to it but oh well I dont really know any
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I see alright, thank you!
Good idea I guess, I'll do that
Yeah, though I kinda wanna finish my project too so that bothers me but I mean without experience I wont finish it anyways I guess hahaha
Thats also true yeah
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I logged the chunks being read. Doesn't that mean all chunks have the same size and thus aren't really the problem?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I'll do exactly that in a sec, but I just logged out the mixers data output, as in the chunks and I guess that already doesn't look good..?
That's how it looks like when a single person talks
that when multiple
but Ill write it to a file with code just in a sec
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Oh alright we'll see us another time then maybe. Good night then :)
and thanks for everything
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Oh I managed to do it
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Sounds just like its being played in the vc
well I havent changed the code yet
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yeah
thats a close up
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
looks like the reverse of the saw wave thing
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yeah I could maybe Record myself saying "test" into two different devices / two different accounts or like play an audio while my microphone picks it up
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I dont know how the extension is called for those types of files but I mean it doesnt matter
Just me saying this is a test but on two different devices
causing the cracking effect
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yeah sure
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I seeee
yeaa..?
But theres still audio there, its just more silent
And that explains why the audio "crack" sounds have the same space in between them. Like it's not random its always after 20ms then when a chunk arrives
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
lets goo
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yes sure!
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yes ofc send me a dm
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Jup ofc vittee found the issue
So the issue was that the decoder (the OpusEncoder.decode(...) function which I used to decode the chunks) were shared for every user instead of me creating a separate decoder for each user.
vittee then changed it so each user has it's own and then pushed it into the repo (https://github.com/Jiron/Discord-Audio-Replay-Test)
GitHub
GitHub - Jiron/Discord-Audio-Replay-Test
Contribute to Jiron/Discord-Audio-Replay-Test development by creating an account on GitHub.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
tiny addition that vittee found out: Make sure that when implementing all this into your own code, that everything only runs once as it will kill running listeners since it "re-joins" the voice channel, meaning if you are speaking and you run it again, it will only start listening again once you go silent really quick (so the green ring disappears ofc) and then start talking again so the bot can get the stream
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Oh true that was the last pull request done by @vittee. I just accepted it out of haslte. He for sure can answer it lmao
But from what I can telly that's just debugging stuff which you teeechnically could leave out. Like the class extensions and the pipeline