K
Kord•4w ago
Tschis

java.io.EOFException on dev.kord.gateway.DefaultGateway

Hey! I am getting some spam of errors on Kord, and after a while it seems to just render my application dead and I have to restart it. 
2025-01-04 17:44:33 ​2025-01-04 16:44:33 | ERROR |      dev.kord.gateway.DefaultGateway | 
2025-01-04 17:44:33 java.io.EOFException: null
2025-01-04 17:44:33     at okio.RealBufferedSource.require(RealBufferedSource.kt:204)
2025-01-04 17:44:33     at okio.RealBufferedSource.readByte(RealBufferedSource.kt:214)
2025-01-04 17:44:33     at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.kt:119)
2025-01-04 17:44:33     at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:102)
2025-01-04 17:44:33     at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
2025-01-04 17:44:33     at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
2025-01-04 17:44:33     at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
2025-01-04 17:44:33     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
2025-01-04 17:44:33     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
2025-01-04 17:44:33     at java.base/java.lang.Thread.run(Unknown Source)
2025-01-04 17:44:33 ​2025-01-04 16:44:33 | ERROR |      dev.kord.gateway.DefaultGateway | 
2025-01-04 17:44:33 java.io.EOFException: null
2025-01-04 17:44:33     at okio.RealBufferedSource.require(RealBufferedSource.kt:204)
2025-01-04 17:44:33     at okio.RealBufferedSource.readByte(RealBufferedSource.kt:214)
2025-01-04 17:44:33     at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.kt:119)
2025-01-04 17:44:33     at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:102)
2025-01-04 17:44:33     at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
2025-01-04 17:44:33     at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
2025-01-04 17:44:33     at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
2025-01-04 17:44:33     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
2025-01-04 17:44:33     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
2025-01-04 17:44:33     at java.base/java.lang.Thread.run(Unknown Source)
As the stacktrace doesn't show much, I was wondering if anyone could have a better idea of how I could troubleshoot or fix this 🙂 I am a newbie with Kord/Kotlin, so I might ask a lot of follow ups
67 Replies
viztea
viztea•4w ago
this is just okhttp being bad personally i use the java ktor client for gateway stuff ktor-client-java
Tschis
TschisOP•4w ago
ok, atm I can not change it would you think it has any chance to be related to long latency times to respond to commands? my bot is having issues sometimes where it takes up to 3 minutes for command events to be received (since their createdAt timestamp)
No description
viztea
viztea•4w ago
shouldn't unless the entire jvm freezes during that period and the connection drops but I think that would be connection reset instead i've gotten EOF exceptions just running my private bot locally https://canary.discord.com/channels/556525343595298817/587324906702766226/1233451713613135982 just looked at okhttp's github and many other people have reported similar issues but the team can't reproduce so there's nothing for them to do :clueless: one of the contributors said v5 includes a fix for websocket concurrency issues ktor-client-okhttp uses okhttp v4 though https://github.com/square/okhttp/issues/3113
I think your server is kicking you!
can you elaborate on
after a while it seems to just render my application dead and I have to restart it.
is there anything else in your logs to indicate why it might be "rendering your application dead" Kord attempts to reconnect on exceptions
Tschis
TschisOP•4w ago
well i just notice nothing else was being handled, no new events we have a gateway microservice handling some kord events, it's usually spammy and sometimes it would just stop writing anything to logs but this picture i sent above is from another application, the actual bot, which also handles other events and these should mostly be command from users, the ones with high interaction latency above we have this issue every Sunday, somewhat linked it to some brazilian discord servers (our bot is for a game with heavy brazilian population) it seems these servers have a lot of crypto bots that spammed memberupdatevents and it seemd handling these was causing this latency spike in our bot however, after a lot of support and discussion on kordex side https://discord.com/channels/1121419906995458098/1323320267031838791 we are running out of ideas we have improved the current code to skip events we don't care about, and for the ones we do, we also skip if they come from bot users but still, our bot behaves like this
viztea
viztea•4w ago
i'm fairly certain the eof exceptions are separate from your other issues (but dont quote me on that :p) i do recommend swapping to the ktor-client-java engine though
Tschis
TschisOP•4w ago
will note it as an enhancement for us to do however we are trying to focus on figuring out this latency issue as it affects us and all users every Sunday for around 24h (00:00 to 23:59 BRT)
Moon
Moon•3w ago
@Tschis Hello 
val kord = Kord(token) {
    stackTraceRecovery = true
}
val kord = Kord(token) {
    stackTraceRecovery = true
}
enable the stackTraceRecovery flag, it might be eaten up by coroutine
Tschis
TschisOP•3w ago
we already have this set
Moon
Moon•3w ago
is it a private server?
Tschis
TschisOP•3w ago
if you mean bot, it is private, yes it can be added to any server
Moon
Moon•3w ago
is it in many servers? can you please share the logs after and before the error too?
Tschis
TschisOP•3w ago
many is relative, but it is in more than 16k There is nothing relevant, just normal logic logging 
2025-01-09 16:38:57 |  INFO |       x.n.d.g.e.UserServersExtension | [addMemberToDatabase] Shard: 3 | Guild: xxx| Member: xxx
2025-01-09 16:39:29 | ERROR |      dev.kord.gateway.DefaultGateway
java.io.EOFException: null
    at okio.RealBufferedSource.require(RealBufferedSource.kt:204)
    at okio.RealBufferedSource.readByte(RealBufferedSource.kt:214)
    at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.kt:119)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:102)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
    at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

2025-01-09 16:39:29 | ERROR |      dev.kord.gateway.DefaultGateway
java.io.EOFException: null
    at okio.RealBufferedSource.require(RealBufferedSource.kt:204)
    at okio.RealBufferedSource.readByte(RealBufferedSource.kt:214)
    at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.kt:119)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:102)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
    at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
2025-01-09 16:45:08 |  INFO |       x.n.d.g.e.UserServersExtension | [addMemberToDatabase] Shard: 3 | Guild: xxx| Member: xxx
2025-01-09 16:38:57 |  INFO |       x.n.d.g.e.UserServersExtension | [addMemberToDatabase] Shard: 3 | Guild: xxx| Member: xxx
2025-01-09 16:39:29 | ERROR |      dev.kord.gateway.DefaultGateway
java.io.EOFException: null
    at okio.RealBufferedSource.require(RealBufferedSource.kt:204)
    at okio.RealBufferedSource.readByte(RealBufferedSource.kt:214)
    at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.kt:119)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:102)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
    at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

2025-01-09 16:39:29 | ERROR |      dev.kord.gateway.DefaultGateway
java.io.EOFException: null
    at okio.RealBufferedSource.require(RealBufferedSource.kt:204)
    at okio.RealBufferedSource.readByte(RealBufferedSource.kt:214)
    at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.kt:119)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:102)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
    at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
2025-01-09 16:45:08 |  INFO |       x.n.d.g.e.UserServersExtension | [addMemberToDatabase] Shard: 3 | Guild: xxx| Member: xxx
Moon
Moon•3w ago
are you sending custom events down the gateway? (asking since that's what I got from the info logs) the addMembertodb here
Tschis
TschisOP•3w ago
no, this gateway only listens to Kord events this addMemberToDatabase is just a reaction to a MemberJoinEvent
Moon
Moon•3w ago
does it happen after certain events? I became rusty with replies, excuse me
Tschis
TschisOP•3w ago
I do not think so, it seems to happen after some time the gateway is running I have not noticed any specific trends
Moon
Moon•3w ago
what is your logging level @Tschis ?
Tschis
TschisOP•3w ago
at the moment it is running on INFO
Moon
Moon•3w ago
set it to trace and let it run for a bit it might show some interesting trends / discord errors that may help us finding out the actual issue my suspection is that discord is sending no-reconnect error codes that renders a connection dead over time
Tschis
TschisOP•3w ago
I will need to get back to you on that becase trace is quite spammy and Im not sure if this affects some of our grafana quotas
Moon
Moon•3w ago
Mmmmm
Tschis
TschisOP•3w ago
i can run it locally and see if it's reproduceable
Moon
Moon•3w ago
yes, that will do
viztea
viztea•3w ago
kord-gateway logs error codes though so unless there are "Gateway closed: <code>" logs this is literally just okhttp being bad
Moon
Moon•3w ago
@viztealong time no see
viztea
viztea•3w ago
do i know you?
Moon
Moon•3w ago
I'll leave that to your imagination
viztea
viztea•3w ago
ok you're weird
NoComment
NoComment•3w ago
I too am regularly coming across this it is slightly irking me
Moon
Moon•3w ago
What engine are you using?
NoComment
NoComment•3w ago
whatever the kord default is 
10-01-2025 14:27:13 | DefaultDispatcher-worker-1     | ERROR | dev.kord.gateway.DefaultGateway
    |
java.io.EOFException: null
        at okio.RealBufferedSource.require(RealBufferedSource.kt:204)
        at okio.RealBufferedSource.readByte(RealBufferedSource.kt:214)
        at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.kt:119)
        at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:102)
        at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
        at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
10-01-2025 14:27:13 | DefaultDispatcher-worker-1     | ERROR | dev.kord.gateway.DefaultGateway
    |
java.io.EOFException: null
        at okio.RealBufferedSource.require(RealBufferedSource.kt:204)
        at okio.RealBufferedSource.readByte(RealBufferedSource.kt:214)
        at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.kt:119)
        at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:102)
        at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
        at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
:)
Moon
Moon•3w ago
can you set your trace level to trace
NoComment
NoComment•3w ago
this is prod and i can't repro locally so it'll take me a little while to get back to you and it seems to take a long time to appear
Moon
Moon•3w ago
I did realize that
NoComment
NoComment•3w ago
Ok bot has restarted , I'll let you know when i get one
Tschis
TschisOP•3w ago
didn't get the error over night on my local
Moon
Moon•3w ago
Thank ya'll
NoComment
NoComment•3w ago
The error has come up in my logs, I'll look at the surrounding trace once I finish work 👍
NoComment
NoComment•3w ago
viztea
viztea•3w ago
@NoComment is it possible for you to swap to java client?
NoComment
NoComment•3w ago
possibly I don't know how but I can try tomorrow?
viztea
viztea•3w ago
that fixed the problem for me
viztea
viztea•3w ago
should be able to just do something like this
No description
viztea
viztea•3w ago
websocket_client is just my HttpClient(Java) with WebSockets installed
NoComment
NoComment•3w ago
can you show plz
viztea
viztea•3w ago
?
No description
viztea
viztea•3w ago
HttpClient(Java) { install(WebSockets) }
NoComment
NoComment•3w ago
thenk I will try tomorrow actually fuck it i'll do it over night Where is the gateway builder?
viztea
viztea•3w ago
kord block
NoComment
NoComment•3w ago
idea disagrees
viztea
viztea•3w ago
yeah wait thats specific to my fork oop
NoComment
NoComment•3w ago
:wubble_explode:
viztea
viztea•3w ago
one sec i should really pr this into kord 
 
val javaClient = HttpClient(Java) {
    install(WebSockets)
}

gateways { resources, shards ->
    val rateLimiter = IdentifyRateLimiter(resources.maxConcurrency, defaultDispatcher)
    shards.map {
        DefaultGateway {
            identifyRateLimiter = rateLimiter
            client = javaClient
        }
    }
}
 
val javaClient = HttpClient(Java) {
    install(WebSockets)
}

gateways { resources, shards ->
    val rateLimiter = IdentifyRateLimiter(resources.maxConcurrency, defaultDispatcher)
    shards.map {
        DefaultGateway {
            identifyRateLimiter = rateLimiter
            client = javaClient
        }
    }
}
NoComment
NoComment•3w ago
what lib is the http client there's like 4 options and none of them want java lmao
viztea
viztea•3w ago
ktor-client-java
NoComment
NoComment•3w ago
and what is the java? doesn't exist for me
viztea
viztea•3w ago
its this artifact
NoComment
NoComment•3w ago
ah duh
viztea
viztea•3w ago
io.ktor:ktor-client-java:<ktor version>
NoComment
NoComment•3w ago
time to find out what kordex uses
viztea
viztea•3w ago
kord uses 3.0.0
NoComment
NoComment•3w ago
a cool, overnight test begins momentarily
viztea
viztea•3w ago
@NoComment any update?
NoComment
NoComment•3w ago
Seems to be good Irritatingly I went to bed before discovering that apication commands failed to sync So like 10 hours of it was errors of people trying commands But since then all seems well
viztea
viztea•3w ago
:omegalul:
NoComment
NoComment•3w ago
Looks like the java client has fixed the funny
Tschis
TschisOP•3w ago
I am not sure where the okhttp client is coming from Our gateway shoud be usingCIO engine from ktor 🤔

Did you find this page helpful?