Kord•4y ago

Kord doesn't reconnect

I pretty much have issues since day 1 with my bot reconnecting, sometimes it just spams RetryLimitReachedEvent in my log

I don't know whether this is just an issue with my logging, but the bot actually doesn't work until a full restart

SchlaubiBusOP•9/19/22, 11:38 AM

Also in the process of connecting 14 shards it pretty much happens all the time that when the last shard got connected other shards have been disconnected again

g•9/19/22, 11:53 AM

I haven't noticed this with an unsharded bot so it might be related to that

LustigerLurch•9/19/22, 3:27 PM

Hm, can you call the endpoint to see the

max_concurrency

max_concurrency

for your bot? Maybe it's because we don't properly handle it.

Gg I haven't noticed this with an unsharded bot so it might be related to that

SchlaubiBusOP•9/19/22, 5:45 PM

It's hard to experience that issue with an unsharded bot as it only connects one shard

LLustigerLurch Hm, can you call the `GatewayBotGet` endpoint to see the `max_concurrency` for y...

SchlaubiBusOP•9/19/22, 5:46 PM

SchlaubiBusOP•9/19/22, 5:47 PM

I think we do connect shards in parallel

LustigerLurch•9/19/22, 5:47 PM

and you are only allowed to connect one concurrently

LustigerLurch•9/19/22, 5:48 PM

so it's this issue again, you already opened https://github.com/kordlib/kord/issues/625

GitHub

Properly implement login rate limiting · Issue #625 · kordlib/kord

Currently we connect all shards at once and rate limit the identify command, which causes session resets and timeouts (see #624) A better solution would be to implement rate limiting for logins

SchlaubiBusOP•9/19/22, 5:48 PM

IG that's what that means yes

SchlaubiBusOP•9/19/22, 5:48 PM

yes

SchlaubiBusOP•9/19/22, 5:49 PM

Kord seems to eventually give up reconnecting

SchlaubiBusOP•9/19/22, 5:49 PM

I have an all shards ready event
And it consumes all other ready events and if all shards on the current instance are ready it fires

SchlaubiBusOP•9/19/22, 5:50 PM

And it has logging and that logging tells me "waiting for [<all shards>]

Kord doesn't log anything

SchlaubiBusOP•9/19/22, 5:50 PM

So it could be that which is broken

SchlaubiBusOP•9/19/22, 5:50 PM

But that shouldn't affect commands

SchlaubiBusOP•9/19/22, 5:50 PM

That event doesn't change some state

SchlaubiBusOP•9/19/22, 8:45 PM

2022-09-19T20:28:01.550055599Z 2022-09-19 20:28:01.546 [DefaultDispatcher-worker-8] WARN dev.schlaubi.musicbot.core.Bot - Shard got disconnected 4 RetryLimitReachedEvent, Awaiting login from: [4, 12]

SchlaubiBusOP•9/19/22, 8:45 PM

@LustigerLurch this also happens

LustigerLurch•9/19/22, 8:50 PM

does this log come from kord?

SchlaubiBusOP•9/19/22, 8:51 PM

It doesn't

SchlaubiBusOP•9/19/22, 8:51 PM

It's my logging

SchlaubiBusOP•9/19/22, 8:51 PM

but that basically means a RetryLimitReachedEvent was fired

SchlaubiBusOP•9/19/22, 8:51 PM

And it never fires a ready event for that shard again

LustigerLurch•9/19/22, 9:20 PM

what does the trace logging for gateway events show?

SchlaubiBusOP•9/19/22, 9:45 PM

gonna download .4 GB of LOGS real qiuck

SchlaubiBusOP•9/19/22, 9:46 PM

however since enabeling trace logging the issue hasn't occurred yet

SchlaubiBusOP•9/19/22, 9:49 PM

Fleet doesn't want to open it http://rice.by.devs-from.asia/u/4869Mq.png

SchlaubiBusOP•9/19/22, 9:49 PM

srv-captain--votebot.1.kyi5nfvreqag@v220210987031163663 | 2022-09-19 21:41:12.302 [DefaultDispatcher-worker-6] TRACE dev.kord.gateway.DefaultGateway - Gateway >>> {"op":2,"d":{"token":"token","properties":{"os":"Linux","browser":"Kord","device":"Kord"},"compress":false,"large_threshold":250,"shard":[12,15],"presence":{"status":"dnd","afk":false,"game":{"name":"Starting ...","type":0}},"intents":"3243773"}}
srv-captain--votebot.1.kyi5nfvreqag@v220210987031163663 | 2022-09-19 21:41:12.521 [DefaultDispatcher-worker-22] TRACE dev.kord.gateway.DefaultGateway - Gateway <<< {"t":null,"s":null,"op":9,"d":false}
srv-captain--votebot.1.kyi5nfvreqag@v220210987031163663 | 2022-09-19 21:41:12.540 [DefaultDispatcher-worker-10] TRACE dev.kord.gateway.DefaultGateway - gateway connection closing
srv-captain--votebot.1.kyi5nfvreqag@v220210987031163663 | 2022-09-19 21:41:12.543 [DefaultDispatcher-worker-10] TRACE dev.kord.gateway.DefaultGateway - Gateway closed: 4900 reconnecting
srv-captain--votebot.1.kyi5nfvreqag@v220210987031163663 | 2022-09-19 21:41:12.544 [DefaultDispatcher-worker-4] WARN dev.schlaubi.musicbot.core.Bot - Shard got disconnected 12 SessionReset, Awaiting login from: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14]
srv-captain--votebot.1.kyi5nfvreqag@v220210987031163663 | 2022-09-19 21:41:12.544 [DefaultDispatcher-worker-10] TRACE dev.kord.gateway.DefaultGateway - handled gateway connection closed
srv-captain--votebot.1.kyi5nfvreqag@v220210987031163663 | 2022-09-19 21:41:12.546 [DefaultDispatcher-worker-13] WARN dev.schlaubi.musicbot.core.Bot - Shard got disconnected 12 RetryLimitReachedEvent, Awaiting login from: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14]
srv-captain--votebot.1.kyi5nfvreqag@v220210987031163663 | 2022-09-19 21:41:12.701 [DefaultDispatcher-worker-10] TRACE dev.kord.gateway.DefaultGateway - Gateway <<< {"t":null,"s":null,"op":10,"d":{"heartbeat_interval":41250,"_trace":["["gateway-prd-main-gh00",{"micros":0.0}]"]}}
srv-captain--votebot.1.kyi5nfvreqag@v220210987031163663 | 2022-09-19 21:41:12.703 [DefaultDispatcher-worker-12] TRACE dev.kord.gateway.DefaultGateway - Gateway >>> {"op":1,"d":null}

SchlaubiBusOP•9/19/22, 9:50 PM

Okay so op 9 is "The session has been invalidated. You should reconnect and identify/resume accordingly."

SchlaubiBusOP•9/19/22, 9:51 PM

That's what happens during startup

LustigerLurch•9/19/22, 9:58 PM

yeah, pretty sure it's because you are only allowed to login one shard at a time

LustigerLurch•9/19/22, 9:59 PM

we should fix this!

SchlaubiBusOP•9/19/22, 10:17 PM

yeah

SchlaubiBusOP•9/19/22, 10:17 PM

maybe this also causes the other disconnects

SchlaubiBusOP•9/19/22, 10:17 PM

because if one shard disconnects

SchlaubiBusOP•9/19/22, 10:18 PM

I get rate limmited

LustigerLurch•9/19/22, 11:01 PM

@I love Gradle files this is what I have so far as a replacement for the existing :

SchlaubiBusOP•9/19/22, 11:03 PM

I am on my phone rn

SchlaubiBusOP•9/19/22, 11:03 PM

But looks good at first glance

LustigerLurch•9/20/22, 2:01 AM

Could you try out with ?

SchlaubiBusOP•9/20/22, 9:48 AM

sure

SchlaubiBusOP•9/20/22, 3:15 PM

I did all of this in a hurry but I still have session resets

LustigerLurch•9/20/22, 4:02 PM

do you use kordex?

LustigerLurch•9/20/22, 4:02 PM

it might have to be recompiled

LustigerLurch•9/20/22, 4:03 PM

cause there was an inline functions somewhere in this pr

LustigerLurch•9/20/22, 4:21 PM

@MrPowerGamerBR are you familiar with sharding? If yes, could you take a look at this PR too? https://github.com/kordlib/kord/pull/693

GitHub

Implement login rate limiting by Lukellmann · Pull Request #693 · k...

fixes #625