Frequent Timeouts On Velocity
So I have a private fabric server network behind a velocity proxy hosted on OCI.
The backend(s) uses fabric 1.21 and fabricproxylite for velocity support
It works, but my players keep getting timeouts randomly while playing.
136 Replies
Thanks for asking your question!
Make sure to provide as much helpful information as possible such as logs/what you tried and what your exact issue is
Make sure to mark solved when issue is solved!!!
/close
!close
!solved
!answered
Requested by _.dominic#0
I guess i can run a spark report, gimme 5 mins
1. Is it while playing or on connection?
2. Is it specific players or all of them?
3. Do you have anecdotal experience from a time that you did not use Velocity and did not have this issue?
4. What are you hosting your server and proxy on?
5. How long as this issue been happening?
6. Has there ever been a time when it was not happening?
and yes, please do
1. Is it while playing or on connection?
While playing, connection is fine
2. Is it specific players or all of them?
All players (myself included)
3. Do you have anecdotal experience from a time that you did not use Velocity and did not have this issue?
It happens, but the frequency of occurrences reduced dramatically compared to when Velocity is introduced
4. What are you hosting your server and proxy on?
Backend(s): Home
Proxy: OCI
5. How long as this issue been happening?
Ever since I started using velocity
6. Has there ever been a time when it was not happening?
Not really
spark is running
will provide link when its done
#3 and #5 directly contradict each other. Did you previously have your server open to players without using Velocity?
oop i misread
spark
spark is a performance profiler for Minecraft clients, servers, and proxies.
Spark Profile Analysis
β Processing Error
The bot cannot process this Spark profile. It appears that the platform is not supported for analysis. Platform: Fabric
Requested by _.dominic
My instincts are telling me there's probably something breaking between the OCI server and your backend
Ive been considering the possibility if OCI is the problem
Your server is poorly optimized.
1. Shenandoah GC is NOT recommended for Minecraft. There are some misinformed people who put a lot of effort into recommending it, but even so, they are wrong. G1GC with Aikar's flags as generated by https://flags.sh is still preferable.
2. Your memory flags are wrong. -Xms and -Xmx should never be different. Always set them equal. The JVM avoids expanding the heap as much as possible, so you are essentially forcing it to try to minmax the initial allocation space. Once it finally accepts that it needs to grow the heap, this wastes CPU time. Set them equal.
Fix those issues first and then we can talk.
however I couldn't verify this possibility
fair, will change that
Wait for #2. That still matters?
Thought the jvm just took what it needed dynamically and just kept it
On G1GC, yes. Expanding the heap takes cycles. It's not free.
Makes sense ig
might be different on Z or Shen, but they're both shit for MC, so doesn't matter.
should I keep large pages
You should use the flags exactly as generated without changes
I pray one day mcjava server software becomes as optimized as bedrock alternatives (specifically dragonfly)
@ππππΎππΎπΈ Oh and #4, your OS is swapping memory. You'll want to turn swappiness to 0 in your OS.
https://help.ubuntu.com/community/SwapFaq#What_is_swappiness_and_how_do_I_change_it.3F
I see
is it related to the disconnects by chance?
All of this can be related to disconnects, yes.
We have to eliminate all of these normal performance issues to eliminate that possibility.
If the disconnects persist, then we move on to other possibilities.
Will do, I'll patch the issues mentioned, then report back if it happens again
Thanks for the heads up
Btw try to spam reconnect
Might be some ratelimiting bs
Sounds good. Feel free to leave the thread open for a day or two if necessary. I'm also interested in this, so I'd love it if you pinged me if something new comes up.
Yeah, for a network heavy node like a Velocity box, I'd sooner use a small AWS EC2 instance than Oracle.
Tbh I've had similar issues back when I was a free tier user
or Azure, or Digital Ocean, etc
I use one of their 1c1g instances in oracle free
π
Oh. So that's why
We've found the cause π
That's definitely not helping.
Ive heard the routing is suboptimal, yes
Their networking on those is dogshit.
"suboptimal" is an understatement
Grab a 1c/4G arm server and throw velocity on thst
Seriously, get some free credits for a big enterprise host, and put your Velocity server on that. EC2 (and similar) instances are dogshit for MC servers, but great for Velocity and other networking heavy applications.
At least you get gigabit networking then
This. 200x over
so I should consider AWS?
Aws, digital ocean, linode, azure
Anything you want really
AWS, Azure, Digital Ocean, etc. Something big with really solid networking infrastructure.
But only for the proxy.
Never for a Minecraft server. Their performance profile is wrong for Minecraft servers.
I've heard DO is more budget compared to azure/aws
Btw how are you connecting it to ur backend
Yes
But iirc no free credits?
DO frequently hands out free credits.
I've gotten DO credits before.
Oh nvm, guess I'm just unlucky
Never gotten em
Skill issue tbh.
so you think my issue is mostly caused by OCI?
It's definitely a strong contributor lmfao
but the other issues I highlighted absolutely did not help
and likely contributed
shenGC without velocity is actually not that bad compared to G1GC
ive tried both
Please don't make me search through a bunch of posts to find the performance comparison that absolutely eviscerates Shen as a viable contender for MC servers. I know I have the link somewhere, but I don't remember if it's in my bookmarks, reddit saved posts, screenshots, github stars, or whatever else. Please just take my word for it. It LOOKS viable at a glance if you don't know how to test it. It is NOT viable.
Can you π₯Ί
Ugh.
I wanna read
I fucking hate you Ashank.
π
I'll eventually dredge it up for the AC wiki project.
β€οΈβ€οΈβ€οΈ ly2
π
Why did discord move edit
Nvm huh? Electron bugging
can you rewrite that so it makes sense to my brain?
OH
nevermind. got it.
Lol
@ππππΎππΎπΈ ANYWAY, make those performance changes, move off of Oracle, and then tag me for followup. I'm hella curious.
guess ill use my aws free creds then
(and if you really don't wanna use Velocity, nginx isn't it. You wanna use Fast Reverse Proxy and Byerwall.)
GitHub
GitHub - fatedier/frp: A fast reverse proxy to help you expose a lo...
A fast reverse proxy to help you expose a local server behind a NAT or firewall to the internet. - fatedier/frp
uhm.
how could it be OOM
nvm, adding large pages back into the mix worked
spark
spark is a performance profiler for Minecraft clients, servers, and proxies.
Spark Profile Analysis
β Processing Error
The bot cannot process this Spark profile. It appears that the platform is not supported for analysis. Platform: Fabric
Requested by _.dominic
AWS EC2 proxy, changed flags
No issues so far
nvm, just as I said it, another timeout
@Rhythmatic as you have requested
π
I am using this if that matters
What's your upload speed?
its gbit bidirectional at home
and its also gbit bidirectional on my EC2 instance (via speedtest-cli)
Yeah, that's good.
hmmm
Do these timeouts happen to everyone or just certain people?
everyone
at the same time simultaneously, or just that it could be anyone at any given time?
Hm, I wonder if this is the issue
Is that your router log?
randomly, so at any given time
IDPS, yeah
but not everyone simultaneously
okay
the IP is from my client pc to the proxy server
very interesting
This is getting a bit outside of my wheelhouse.
@Snow Kit you good with networking?
or can think of someone who is?
I'll try disabling that specific rule and see if it helps
yeah give it a try
if not, I can try to summon a turbo nerd
I see how it is
you didn't answer, turbo nerd
however those packets being blocked doesnt seem to affect my gameplay
Or alternatively, doesn't seem to affect your gameplay all of the time.
Maybe there's some critical mass of blocked packets that results in a timeout
TCP does tend to resend packets that get lost
fair
I had to read context, smh
imagine not being omniscient and omnipresent, smh
I'll just afk a while on the server while I go have my dinner, and pray the issue is solved
I guess there is more problem than just a timeout
lmao
https://spark.lucko.me/WDPFuWym6B
and the spark profiler with another player
spark
spark is a performance profiler for Minecraft clients, servers, and proxies.
Spark Profile Analysis
β Processing Error
The bot cannot process this Spark profile. It appears that the platform is not supported for analysis. Platform: Fabric
Requested by _.dominic
I'm really concerned about your physical memory being almost full and your server swapping memory.
What all are you running on that box? There's no way it's only Minecraft on fresh Ubuntu.
And you should really turn Ubuntu's swappiness down to 0.
ah i forgot about that
it is actually a fresh ubuntu vm, with 4gb overhead
Are you running like a full GUI on it?
ubuntu server 2204
that is not an answer
so, I guess not?
full cli based with no GUI
The distro and variant do not imply the presence of--
Okay, thanks.
So how the fuck do you have 4GB of overhead?
That's astronomical. It should be like 200MB.
jvm uses 8192M, but the VM is given 12gbs
yesssss...?
but
that shows that it's almost all being used
Might have something to do with my huge pages reserve
eh, I guess it doesn't matter. if nothing else is running on it, I suppose it's probably just memory cache
we'll ignore that for now
change swappiness for sure
Now that I double check, I don't think that's part of normal Aikar's flags. What's the rationale for that?
nope, that is indeed not a part of the regular aikar's flags
I was using hiltty's flags before moving over to flags.sh
I just had large pages reserved, so using it here too
Gotcha. I won't advise one way or another on that, as I don't have data or experience to support any particular recommendation.
https://www.reddit.com/r/technicalfactorio/comments/mvn88v/over_20_actually_30_performance_gain_by_using/
I did see this while trying to find out on whether or not I should use huge (or large) pages
Reddit
From the technicalfactorio community on Reddit: over 20% (actually ...
Explore this post and more from the technicalfactorio community
Yeah, but that's not Java.
So.
afaik factorio also runs on a java based server
but ye
wait does it
try removing antixray
I saw "malloc" which to me, means C or C++
*waves* thanks for answering my call
:wave_animated:
i just read it and wanted to see whats up
im personally suffering myself setting up another program lol
@ππππΎππΎπΈ ?
antixray?
the mod
^
may I ask why?
Antixrays tend to work by manipulating packet data. And they do be heavy af.
i forgot the main reason but somewhere in bloom docs i remember reading on antixray being a source of ping lag
or ping inflation
https://docs.bloom.host/extras/ping-issues/#paper-anti-xray-engine-mode-2
yep here
very interesting indeed
anyways im going to continue suffering putting serverpack creator on my pterodactyl instance
@ me if it worked or not
So far not seeing timeouts from me nor other players after disabling the IDPS rule
Niiiiiice. Might've been false pos-ing a Minecraft packet
are pings better in spark now, too?
Right until packetloss/it drops the keepalive packet
That was my guess too. Or there's a finite number of those packets that TCP allows the connection to survive, but if it does one too many, poof.
Would mean users actions wouldnt happen for a few seconds before kick probably
Which doesnt sound to be the case
See, this is why I tell people that Staff != smart.
Nah its not staff its just that youre old sir
eh about the same
spark
spark is a performance profiler for Minecraft clients, servers, and proxies.
Spark Profile Analysis
β Processing Error
The bot cannot process this Spark profile. It appears that the platform is not supported for analysis. Platform: Fabric
Requested by _.dominic
*an older profiler
my old proxy was in frankfurt
I meant better since fixing the IDPS filter
seeing one from 3-4 days ago is...not much help lol
Will mark as solved until further issues arise
Thanks for the input
Closed post!
Your post has been marked as solved!
Requested by _.dominic#0