discord.js - Imagine ❄•6d ago

Not emitting audio received from websocket (webm opus)

I'm trying to make a voice bot PoC that takes browser audio stream (webm opus) and sends it over websocket to be spoken into a channel via the bot. I have the stream working with the website and the server, but the bot is not emitting the audio in the channel. I see the following state transition logs

read 16384
[08:43:02.268] DEBUG (17160): got interaction: join-voice-channel
[08:43:02.269] DEBUG (17160): got join-voice-channel for channel: 1323318616145264651
[08:43:02.318] INFO (17160): Connection transitioned from signalling to signalling
[08:43:02.320] DEBUG (17160): no audio data
[08:43:02.502] INFO (17160): Connection transitioned from signalling to connecting
[08:43:02.582] INFO (17160): Connection transitioned from connecting to connecting
[08:43:02.593] INFO (17160): Connection transitioned from connecting to connecting
[08:43:02.603] INFO (17160): Connection transitioned from connecting to connecting
[08:43:02.619] INFO (17160): Connection transitioned from connecting to ready
[08:43:02.619] DEBUG (17160): Connection is ready, starting playback
[08:43:02.619] INFO (17160): Audio player transitioned from idle to buffering
[08:46:02.661] INFO (17160): Audio player transitioned from buffering to playing
[08:46:02.802] INFO (17160): Audio player transitioned from playing to idle

read 16384
[08:43:02.268] DEBUG (17160): got interaction: join-voice-channel
[08:43:02.269] DEBUG (17160): got join-voice-channel for channel: 1323318616145264651
[08:43:02.318] INFO (17160): Connection transitioned from signalling to signalling
[08:43:02.320] DEBUG (17160): no audio data
[08:43:02.502] INFO (17160): Connection transitioned from signalling to connecting
[08:43:02.582] INFO (17160): Connection transitioned from connecting to connecting
[08:43:02.593] INFO (17160): Connection transitioned from connecting to connecting
[08:43:02.603] INFO (17160): Connection transitioned from connecting to connecting
[08:43:02.619] INFO (17160): Connection transitioned from connecting to ready
[08:43:02.619] DEBUG (17160): Connection is ready, starting playback
[08:43:02.619] INFO (17160): Audio player transitioned from idle to buffering
[08:46:02.661] INFO (17160): Audio player transitioned from buffering to playing
[08:46:02.802] INFO (17160): Audio player transitioned from playing to idle

As you can see the player doesn't change from buffering to playing until 3 minutes later, and it never plays any audio. I can verify that the audio is being received. Some code:

3 Replies

DanTheGoodmanOP•6d ago

const audioPlayer = createAudioPlayer({
  behaviors: {
    maxMissedFrames: 1000,
    noSubscriber: NoSubscriberBehavior.Play,
  },
})

const connection = joinVoiceChannel({
        channelId: channel.id,
        guildId: interaction.guildId!,
        adapterCreator: channel.guild
          .voiceAdapterCreator as DiscordGatewayAdapterCreator,
        debug: true,
        selfDeaf: false,
        selfMute: false,
      })

      connection.on("stateChange", (oldState, newState) => {
        logger.info(
          `Connection transitioned from ${oldState.status} to ${newState.status}`
        )
      })

      audioPlayer.on("stateChange", (oldState, newState) => {
        logger.info(
          `Audio player transitioned from ${oldState.status} to ${newState.status}`
        )
      })

      let audioQueue: Buffer[] = []

      const audioStream = new Readable({
        read(size) {
          console.log("read", size)
          if (audioQueue.length > 0) {
            logger.debug("sending audio data")
            const chunk = audioQueue.shift()
            return chunk
          }
          logger.debug("no audio data")
          return null
        },
      })

      emitter.on("audio", (audioData: AudioData) => {
        audioQueue.push(audioData.audioBuffer)
        audioStream.emit("readable", audioData.audioBuffer)
        audioStream.emit("data", audioData.audioBuffer)
      })

      const resource = createAudioResource(audioStream, {
        inputType: StreamType.WebmOpus,
        inlineVolume: true,
      })

      audioPlayer.on("error", (error) => {
        logger.error({ error }, "Error in audio player")
      })

      connection.on("error", (error) => {
        logger.error({ error }, "Error in connection")
      })

      connection.subscribe(audioPlayer)

      connection.on(VoiceConnectionStatus.Ready, () => {
        logger.debug("Connection is ready, starting playback")
        audioPlayer.play(resource)
      })

const audioPlayer = createAudioPlayer({
  behaviors: {
    maxMissedFrames: 1000,
    noSubscriber: NoSubscriberBehavior.Play,
  },
})

const connection = joinVoiceChannel({
        channelId: channel.id,
        guildId: interaction.guildId!,
        adapterCreator: channel.guild
          .voiceAdapterCreator as DiscordGatewayAdapterCreator,
        debug: true,
        selfDeaf: false,
        selfMute: false,
      })

      connection.on("stateChange", (oldState, newState) => {
        logger.info(
          `Connection transitioned from ${oldState.status} to ${newState.status}`
        )
      })

      audioPlayer.on("stateChange", (oldState, newState) => {
        logger.info(
          `Audio player transitioned from ${oldState.status} to ${newState.status}`
        )
      })

      let audioQueue: Buffer[] = []

      const audioStream = new Readable({
        read(size) {
          console.log("read", size)
          if (audioQueue.length > 0) {
            logger.debug("sending audio data")
            const chunk = audioQueue.shift()
            return chunk
          }
          logger.debug("no audio data")
          return null
        },
      })

      emitter.on("audio", (audioData: AudioData) => {
        audioQueue.push(audioData.audioBuffer)
        audioStream.emit("readable", audioData.audioBuffer)
        audioStream.emit("data", audioData.audioBuffer)
      })

      const resource = createAudioResource(audioStream, {
        inputType: StreamType.WebmOpus,
        inlineVolume: true,
      })

      audioPlayer.on("error", (error) => {
        logger.error({ error }, "Error in audio player")
      })

      connection.on("error", (error) => {
        logger.error({ error }, "Error in connection")
      })

      connection.subscribe(audioPlayer)

      connection.on(VoiceConnectionStatus.Ready, () => {
        logger.debug("Connection is ready, starting playback")
        audioPlayer.play(resource)
      })

d.js toolkit•6d ago

- What are your intents? GuildVoiceStates is required to receive voice data! - Show what dependencies you are using -- generateDependencyReport() is exported from @discordjs/voice. - Try looking at common examples: https://github.com/discordjs/voice-examples. - Consider reading #how-to-get-help to improve your question! - Explain what exactly your issue is. - Post the full error stack trace, not just the top part! - Show your code! - Issue solved? Press the button!

DanTheGoodmanOP•6d ago

I can see in logs I am getting the data, and I do have the voicestates:

export const client = new Client({
  intents: [GatewayIntentBits.Guilds, GatewayIntentBits.GuildVoiceStates],
})

export const client = new Client({
  intents: [GatewayIntentBits.Guilds, GatewayIntentBits.GuildVoiceStates],
})

dependencies:

[08:42:59.931] DEBUG (17160): --------------------------------------------------
Core Dependencies
- @discordjs/voice: 0.16.1
- prism-media: 1.3.5

Opus Libraries
- @discordjs/opus: 0.9.0
- opusscript: not found

Encryption Libraries
- sodium-native: 4.3.1
- sodium: not found
- libsodium-wrappers: 0.7.15
- tweetnacl: not found

FFmpeg
- version: 7.1
- libopus: yes
--------------------------------------------------

[08:42:59.931] DEBUG (17160): --------------------------------------------------
Core Dependencies
- @discordjs/voice: 0.16.1
- prism-media: 1.3.5

Opus Libraries
- @discordjs/opus: 0.9.0
- opusscript: not found

Encryption Libraries
- sodium-native: 4.3.1
- sodium: not found
- libsodium-wrappers: 0.7.15
- tweetnacl: not found

FFmpeg
- version: 7.1
- libopus: yes
--------------------------------------------------

I can also see that the read method is only being called once, and not again when I emit the event that it should be called again according to https://nodejs.org/api/stream.html#readablereadsize my suspicion is that the resource is not getting audio from the Readable, because it sits buffering for so long I tried using

const resource = createAudioResource(
        "https://www.soundhelix.com/examples/mp3/SoundHelix-Song-1.mp3",
        {
          inputType: StreamType.Arbitrary,
        }
      )

const resource = createAudioResource(
        "https://www.soundhelix.com/examples/mp3/SoundHelix-Song-1.mp3",
        {
          inputType: StreamType.Arbitrary,
        }
      )

and confirmed that does play audio, so connection is fine ok so it seems i had to start the discord bot first, then send audio, so the header frame for the audio was sent to discord

Gaming

Programming

Not emitting audio received from websocket (webm opus)

Did you find this page helpful?