Kord•6d ago

User audio processing only works after rejoin

I am trying to do some audio processing using

voiceConnection.streams.incomingUserStreams

voiceConnection.streams.incomingUserStreams

but I am facing the weird issue that when, for example I am in a voice channel and have the bot join me, the flow only starts emitting packets after I rejoin the channel. In general it seems to only work if the bot is in the channel before the target user joins. Is there a way to fix that?

16 Replies

lost•5d ago

voice connections send audio packets linked to an ssrc (a unique identifier for each user). whats happening is that a cache which builds the ssrc <-> discord user map has to be populated and isnt known when the bot joins (until, for example, you rejoin and discord lets us know of this new ssrc <-> user mapping). not sure if this is intentional or a bug, its been a few years. if a flow that provides packets mapped to a unique ssrc will suffice, use the incomingAudioFrames flow. in the mean time ill see what we could to build this map instantly, since obviously the discord client can do it too. @Marc just looked at the code. i imagine it still works. do you have the voice state intent enabled?

MarcOP•5d ago

Oh sorry, didn't get a notification. Technically I only need a unique mapping, I don't think the user is needed for my usage

MarcOP•5d ago

MarcOP•5d ago

This is what my login looks like

lost•5d ago

yeah i got a test bot running and can repro it race condition basically

MarcOP•5d ago

So this UInt is just a number without meaning basically? Or can I map it to a user without much hassle?

lost•5d ago

the uint is the ssrc which is unique per user kord does the ssrc <-> user mapping but i guess discord changed some stuff to where theres a race condition between when kord begins listening for this mapping notice (speaking event), versus when its actually sent you could maintain your own map of ssrc to an actual discord user if you need. just listen to the speaking event from the voice gateway. or if youre fine with the ssrc being your identifier per user, that works fine too

MarcOP•5d ago

Is that just using voiceConnection.voiceGateway.on<Speaking> { }? Doesn't seem to fire for me, is there something I need to configure to get that event?

lost•5d ago

works for me. the speaking event would inform you of a user's ssrc. it gets sent right when the bot joins, or when a new user joins.

MarcOP•5d ago

Alright, I tried using

voiceGateway {
    eventFlow = voiceGatewayEvents
}

voiceGateway {
    eventFlow = voiceGatewayEvents
}

inside my connect() function and that seems to get me the events. I did the mapping by hand and it seems to work, even if what I did is probably the worst way to do it:

val userSsrcToUserIdMapping = ConcurrentHashMap<UInt, Snowflake>()

voiceConnection.streams
    .incomingAudioFrames
    .onEach { (userSsrc, audioFrame) ->
        val userId = userSsrcToUserIdMapping[userSsrc] ?: run {
            voiceGatewayEvents
                .filterIsInstance<Speaking>()
                .first { it.ssrc == userSsrc }
                .userId
                .also {
                    userSsrcToUserIdMapping[userSsrc] = it
                }
        }

val userSsrcToUserIdMapping = ConcurrentHashMap<UInt, Snowflake>()

voiceConnection.streams
    .incomingAudioFrames
    .onEach { (userSsrc, audioFrame) ->
        val userId = userSsrcToUserIdMapping[userSsrc] ?: run {
            voiceGatewayEvents
                .filterIsInstance<Speaking>()
                .first { it.ssrc == userSsrc }
                .userId
                .also {
                    userSsrcToUserIdMapping[userSsrc] = it
                }
        }

lost•5d ago

weird that u had to supply your own event flow. and yeah personally just:

val s2u = mutableMapOf<UInt, Snowflake>()
connection.voiceGateway.on<Speaking> { speaking ->
  s2u.putIfAbsent(speaking.ssrc, speaking.userId)
}

val s2u = mutableMapOf<UInt, Snowflake>()
connection.voiceGateway.on<Speaking> { speaking ->
  s2u.putIfAbsent(speaking.ssrc, speaking.userId)
}

would probably suffice. thats more or less what kord is doing. and then just use s2u inside your onEach or wherever

MarcOP•5d ago

Am I guaranteed to always get the speaking event first?

lost•5d ago

not sure what you mean exactly by first? but i think yesish? assuming you create this collector soon after the #connect call. discord sends you a speaking event for every user when you join a channel, and then for any new user that joins later

lost•5d ago

see https://github.com/kordlib/kord/pull/1008 for the fix and if/when it is merged.

GitHub

fix(voice): ssrc user mapping by psmarko · Pull Request #1008 · k...

At some point within the last few years, Discord began to send (or always has?) initial voice Speaking events instantly after the voice Ready event. The speaking events are used to build a voice SS...

MarcOP•5d ago

Alright, thanks a lot!

Gaming

Programming

User audio processing only works after rejoin

Did you find this page helpful?