My socket won't send data to 600 endpoints
My application sends a packet to 600 servers in order to gather data about active players and such.
My old code would use a parallel loop and this worked fine but was incredibly slow.
After removing it my code is much faster, but only 58 out of the 600 servers now return data.
Is there some limitation on how much data a socket can handle at once?
This is the method used. The endPoints array would contain 600 endPoints.
I verified the method only parsed about 58 servers by checking how many of the pending builders actually ende dup getting parsed.
At the bottom you see a loop that checks for pending builders that did not get parsed and this shows the timed out servers.
https://gdl.space/irafuvuvam.cs
17 Replies
I mostly wonder if I possibly need to change the way I ask the servers for data in case there is some limit
well every time the method is called it creates the socket, which is really necessary? how often is this method called?
also why are you using SendTo and not SendToAsync
if you need to speed this stuff you can at least use one socket/core and divide the connections, but for sure a persistent connection would help a lot too
im not directly suggesting a framework because i dont know the context; at least can you give a hint on what is the average data size?
I only call the method once, but the endpoints array contains 600 endpoints
The average data size is no more than 1kb per packet. It contains data about the server like how many players play on the server on that port
It seems like it might be possible that packets are dropped when this code runs because of the quanity of servers, but this is hard to say and I have no idea what is really going wrong
I just wonder why it would only parse a fraction of the expected servers
For context, here and onwards I talked about the issue https://discord.com/channels/143867839282020352/169726586931773440/1228389539584671838
FusedQyou
I hope you understand the code:
This first parallel loop invoked for all the IP addresses that were fetched: https://github.com/RoyDefined/WebDoomer/blob/main/src/WebDoomer/WebDoomerApi/Scheduling/ServerDataFetchJob.cs#L85
This next one then runs in parallel for each individual port: https://github.com/RoyDefined/WebDoomer/blob/main/src/WebDoomer/WebDoomerApi/Scheduling/ServerDataFetchJob.cs#L107
This is what is called for each IPEndPoint: https://github.com/RoyDefined/WebDoomer/blob/main/src/WebDoomer/WebDoomer/Services/Zandronum/Server/ZandronumServerService.cs#L29
Quoted by
<@191282187016863744> from #help-0 (click here)
React with ❌ to remove this embed.
I also linked the project so it's possible to see what the working code was and what the "broken" code is
If packets get dropped because it's UDP but also because they are received at the same time as others, then how could I fix this code so it manages to read all incoming packets?
Surely this is a pretty normal issue that has a fix for it?
judging from the arguments it seems like this is called more than once per lifetime of the application
Technically yes, I call it for two different "engines" which have servers. Other than that it's something that is scheduled to be called once every 5 minutes
var socketResultTask = socket.ReceiveFromAsync(bufferData, endPoint, cancellationToken).AsTask();
.AsTask() 😐
i don't think i see anything that checks if the data is all there?
eitiher a beging/end message or a message lengthIt's not supposed to be all there
If it reads a packet it will know the remote endpoint and parse it in the builder assigned to the endpoint
The packet iself will have an indicator if additional data is expected
It's just that some packets are lost and never read
you mean not at the first packet
eventually
did you count how many times
while (true) cycles? is that exactly 58?Yes it is possible for more packets to arrive and the previous packet would indicate if more arrives
Let me check
This iteration fetched half the servers and out of the 305 fetched it did 353 loops
so this logic would be in
ServerResultBuilder?
but then what i would like to see is
if (serverResult.IsCompleted()) yield return serverResult;
ok and you recovered some data from the loop that there are 58 distinct endpoints from which you received something?
and is the endpoints that receives socket.SendTo(packet, endPoint); made by you?
i mean i guess not but asking doesn't hurtYes, this part determines if it should continue https://github.com/RoyDefined/WebDoomer/blob/main/src/WebDoomer/WebDoomer/Services/Zandronum/Server/Builder/ServerResultBuilder.cs#L118
The loop checks this and then builds the final class if nothing else is expected, as the code explains
Specifically this https://github.com/RoyDefined/WebDoomer/blob/1.1.0/src/WebDoomer/WebDoomer/Services/Zandronum/Server/ZandronumServerService.cs#L116
Well in the previous example this was 300. It is very random
But it should fetch at least 95% of them. It is not normal over half do not respond
No, these are Zandronum servers running online
I know their endPoints from a fetch request from the master server
There are existing applications that allow me to verify endPoints that I am unable to communicate with do exist and should be fetchable
I also have a console app in the project which should fetch data by specifying an endPoint and this works fine
wait a minute
are you send udp over internet?!
Yes
They're UDP packets
If you mean that the issue is the packets dropping because of that then it's not because I can see the packets returning in wireshark
So it's like the socket drops the packets
i thought udp was not supported (on internet) anymore
first thing i would do is separating sending and receiving
you can't start receving after you've sent all the data, especially if sending is not cheap
first you setup the listening, and then start sending, like
second one is going to sleep