discord.js - Imagine an app•6mo ago

Recording a user

Has anyone made something similar to a Discord bot that you can talk to in VC? 1. You say a message 2. The bot converts to text 3. Bot sends to API (e.g. GPT) 4. Bot converts to voice 5. Bot sends to voice channel I'm unsure how to proceed between steps 1 & 2 - is there a way for the bot to automatically detect when the user has finished speaking before processing? I do not want a button press or anything of the sort - just automatic.

10 Replies

d.js toolkit•6mo ago

- What's your exact discord.js npm list discord.js and node node -v version? - Not a discord.js issue? Check out #other-js-ts. - Consider reading #how-to-get-help to improve your question! - Explain what exactly your issue is. - Post the full error stack trace, not just the top part! - Show your code! - Issue solved? Press the button!

duck•6mo ago

you can listen to the end event on <VoiceConnection>.receiver.speaking for 2, you'd need something else to handle the speech-to-text since this is not something @discordjs/voice would handle

adamOP•6mo ago

Thank you! Do you have any suggestions on how to record the user? I'm thinking of something like this 1. Start recording the user 2. Once user stops talking, stop recording the user 3. Use external API to convert to text 4. etc etc

adamOP•6mo ago

Hi, not sure if I'm doing something wrong with my code

if (interaction.commandName === 'vc') {
        if (!interaction.member.voice.channelId) {
            return interaction.reply('You need to be in a voice channel to use this command.');
        }

        const voiceChannel = interaction.member.voice.channel;
        const connection = joinVoiceChannel({
            channelId: voiceChannel.id,
            guildId: voiceChannel.guildId,
            adapterCreator: voiceChannel.guild.voiceAdapterCreator
        });

        const receiver = connection.receiver;
        const audioPlayer = createAudioPlayer();

        connection.subscribe(audioPlayer);

        receiver.speaking.on('start', (userId) => {
            interaction.channel.send(`<@${userId}> started speaking`);
        });

        receiver.speaking.on('end', (userId) => {
            interaction.channel.send(`<@${userId}> stopped speaking`);
        });

        interaction.reply('Joined the voice channel and listening for speech.');
    }

if (interaction.commandName === 'vc') {
        if (!interaction.member.voice.channelId) {
            return interaction.reply('You need to be in a voice channel to use this command.');
        }

        const voiceChannel = interaction.member.voice.channel;
        const connection = joinVoiceChannel({
            channelId: voiceChannel.id,
            guildId: voiceChannel.guildId,
            adapterCreator: voiceChannel.guild.voiceAdapterCreator
        });

        const receiver = connection.receiver;
        const audioPlayer = createAudioPlayer();

        connection.subscribe(audioPlayer);

        receiver.speaking.on('start', (userId) => {
            interaction.channel.send(`<@${userId}> started speaking`);
        });

        receiver.speaking.on('end', (userId) => {
            interaction.channel.send(`<@${userId}> stopped speaking`);
        });

        interaction.reply('Joined the voice channel and listening for speech.');
    }

Zerls•6mo ago

I've never used djs voice before, but there is this example https://github.com/discordjs/voice-examples/tree/main/recorder

Hogan.CaroNet•6mo ago

this is actually a pretty cool idea if done properly, as zerls said above, you can use those examples to get an audio file, then maybe use Google's Speech to Text API to get the actual text from what the person said.

Google Cloud

Speech-to-Text AI: speech recognition and transcription

Accurately convert voice to text in over 125 languages and variants using Google AI and an easy-to-use API.

Pkmmte•6mo ago

Hi, this is possible! :Tga_YESS: I already created and open sourced something like this a year ago. It’s pretty outdated now, but I wrote an article about it last year: https://blog.waveplay.com/let-your-robo-speak-introducing-ai-voice-plugin/ https://docs.roboplay.dev/plugins/ai-voice It uses GPT for conversing, Whisper for transcription, and Azure Speech Services for voice. I’d like to redo this to use only OpenAI APIs in the future. :UwU:

adamOP•6mo ago

Wow, really nice find! I really love this blog, it is extremely interesting. I'll have to take a look at the source code, thanks so much! Do you know if I can try this out myself? Is it hosted on a public bot at this time?

Pkmmte•6mo ago

:beanthankyou: We have more recent posts in our Dev.to blog if you’re interested It’s not in a public bot, but it’s in a private bot right now if you’d like to try it out yourself. Let me know if you’re interested and I’ll bring it online!

adamOP•6mo ago

Sent you a DM :)

Gaming

Programming

Recording a user