C#•15mo ago

❔ Read large data over TCP

Team need inputs on reading large data using TCPClient. I need to read data using TCPClient , I am sending a command and receiving data. but not sure whether I have read complete data. How do i check whether all the data has been read and how do I get the buffer size at receiver end ..can I use int.Max like Byte[Int32.Max]. But i noticed I am not getting the whole data at once.Should I loop it until I get the complete data? using (TcpClient client = new TcpClient("192.168.100.123", 10001)) { using (NetworkStream stream = client.GetStream()) { using (StreamReader reader = new StreamReader(stream)) {
byte[] outStream = System.Text.Encoding.ASCII.GetBytes(";"); stream.Write(outStream, 0, outStream.Length);
// Buffer to store the response bytes. Byte[] data = new Byte[5000]; // how do i know this size //string content = new StreamReader(stream, Encoding.Unicode).ReadToEnd();

Int32 bytes = stream.Read(data, 0, data.Length); var responseData = System.Text.Encoding.ASCII.GetString(data, 0, bytes); } } }

83 Replies

Omnissiah•15mo ago

format the code using triple backtick

SinFluxx•15mo ago

$code

MODiX•15mo ago

To post C# code type the following: ```cs // code here ``` Get an example by typing $codegif in chat If your code is too long, post it to: https://paste.mod.gg/

Omnissiah•15mo ago

can I use int.Max like Byte[Int32.Max]

please don't

How do i check whether all the data has been read

tcp sends packets so yes you would have to continue to read all the packets it depends on how your connection works, if it's one connection per request or if it's permanent in the last case, you could send the size of the object before sending the object so that the receiving part knows how much to read

Pobiega•15mo ago

Is it just straight raw TCP data, or is some kind of messaging protocol being used, like MLLP?

Unknown User•15mo ago

Message Not Public

MODiX•15mo ago

see $pipeline

Unknown User•15mo ago

Message Not Public

MODiX•15mo ago

(Contains example for Socket TCP Echo) https://devblogs.microsoft.com/dotnet/system-io-pipelines-high-performance-io-in-net/ https://docs.microsoft.com/en-us/dotnet/standard/io/pipelines if you're handling data from/to a stream, and dealing with intermediate buffer / parser / writer, you probably should consider using System.IO.Pipelines It's handling internally * temporary buffer * memory pool to re-use memory and avoid allocation * accumulate buffer until you decide you have enough data to do something (deserialize ?) There's dedicated extensions for Stream for specific use case like : (Contains a sample to read file / dump to console) Reading (eg: to a file):

using var stream = File.OpenRead("lorem-ipsum.txt");
var reader = PipeReader.Create(stream);

using var stream = File.OpenRead("lorem-ipsum.txt");
var reader = PipeReader.Create(stream);

Writing (eg: to Console):

var writer = PipeWriter.Create(Console.OpenStandardOutput(), new StreamPipeWriterOptions(leaveOpen: true));

var writer = PipeWriter.Create(Console.OpenStandardOutput(), new StreamPipeWriterOptions(leaveOpen: true));

ShivOP•15mo ago

@TeBeClone @Pobiega Sorry for late response. I am not sure about MLLP. But we have a Command to Query Real Time Sensor Values i.e, "!" and ";" for download of data Real time data is fine, I am getting 35 char of data which I am processing but for download I am not sure how much is the accumulated data... Is there we can check before . Can I use something like this :

 int responseBytes = 0;
                            do
                            {
                                responseBytes = stream.Read(data, 0, data.Length); // store the response to the buffer
                                responseData += System.Text.Encoding.ASCII.GetString(data, 0, responseBytes);

                            } while (responseBytes > 0);

 int responseBytes = 0;
                            do
                            {
                                responseBytes = stream.Read(data, 0, data.Length); // store the response to the buffer
                                responseData += System.Text.Encoding.ASCII.GetString(data, 0, responseBytes);

                            } while (responseBytes > 0);

I have no control over sending part...Its a sensor data which I am reading... so only receiving part can i read until stream.Read is 0 ?

Pobiega•15mo ago

in general, what you would do is read until either your buffer is full or you've found an "end of packet" signifier if the buffer is full, you're in trouble - it means a single packet was larger than your buffer. There are ways to get around this, like dumping the buffer somewhere else, flushing it and keep reading, but it gets annoying so what you'd usually do is just set a somewhat large buffer, ideally large enough to store the largest possible packet but if thats an unknown size, you'll have to guesstimate

ShivOP•15mo ago

OK Ending part is clear now.... I am stopping it when I get ";" but Will the Sender send the data randomly or is it in synchronous way.. because everytime I am getting different starting point of data... will it get cleared at sender once we read the data? is it like that or is it device specific?

Pobiega•15mo ago

Your questions don't really make sense to me I'm afraid think of the TCP stream as... well a stream it sends one byte after the other the trick is knowing when it stops so if my buffer is empty, and I see...

123456;789

123456;789

I know that 123456 is one message, and I have a partial(unfinished) message that starts with 789 what I would do while reading this is as soon as I see ; I stop reading, build a finished package, clear my buffer and keep going repeat forever the sender will know how TCP works for sure, so it wont send random bytes it will start a package, send it, then send the ending separator

Unknown User•15mo ago

Message Not Public

Accord•15mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity. Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

ShivOP•15mo ago

@TeBeClone @Pobiega Got it. I am clear now.. Ours is text protocol not binary.But I need to retest couple of usecases...Basically it is a datalogger which will capture weather data from different sensors(like wind direction, speed ,humidity etc) ...We started logging from September 1st.. What I have observed is when I fetch the data sometimes I am getting data from September 7th, next time I am getting data from september 1st...and next time from sep 2nd , that is what confusing me.... According to my understanding it should always be from sep 1st...

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

You have any references or links ?It will be helpful ..

Omnissiah•15mo ago

he said he has no control on the sender

Pobiega•15mo ago

Where are the logs stored? I'd expect once a dataset is collected its not fetchable again but yeah, all this is related to the sender if you cant change the sender, you cant change its sending behaviour :p

ShivOP•15mo ago

Yes sending part I have no control .May be they have some memory inside datalogger(not sure) . It has capacity to store couple of months of data is what I know...

Pobiega•15mo ago

okay

ShivOP•15mo ago

To give you the context , The below is the last 2 rows of data A080923181102635026370263504727047280471500000000000000000026091120000000136 A080923181302634026360263304727047290472600000000000000000026091130000000136; A080923181302<---so date is Sep 08 2023 06:13 pm .... Once I read the data, if it is cleared ,then my issue would have been solved ... I will recollect when required ..but that is not the case, it is giving same random data everytime...

Pobiega•15mo ago

¯\_(ツ)_/¯ All I can suggest is to read the documentation for the sender perhaps you need to ACK the message before its deleted or something idk

ShivOP•15mo ago

Ok let me try that.

Pobiega•15mo ago

dont just try random stuff lol, read the docs you need to understand how the sender works to use it

Omnissiah•15mo ago

i've worked with stuff like this, like the davis wather station but usually i download the file and then parse it locally

ShivOP•15mo ago

Yes even i am trying to download the file and then parsing it ..that part is fine... but everytime I download I am getting random data ( sometimes old data, sometimes new + old data,sometime only new data) ... Will analyse for some more time and then come to the conclusion.

Omnissiah•15mo ago

you know, if this behavior is not documented then you have to ask the manifacturer as others were saying too

ShivOP•15mo ago

Yeah ..Will do that..

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

Yes it is text as I told earlier..

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

"Ours is text protocol not binary"

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

I think You got confused 🙂

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

TO be frank, Not good at it ...

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

Yes first time I am dealing with this kind of stuff

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

no it is fixed

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

Ok !!!

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

Yes

Unknown User•15mo ago

Message Not Public

Pobiega•15mo ago

There is a package I've used a few times that wrap TCP in a very very nice way called "SuperSocket". Unforunately the documentation isnt amazing and its very opinionated, but its very good

Unknown User•15mo ago

Message Not Public

Pobiega•15mo ago

very easy to set up a package encoder/decoder for something like this

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

Yes parsing part is clear. Every row starts with A and have 76 chars ...

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

I think "protocol error" is what i need to check

Omnissiah•15mo ago

to me this return new Message (date, foo, bar ...) would not be a return but a message passing to the next block in the chain

Unknown User•15mo ago

Message Not Public

MODiX•15mo ago

see $pipeline

Unknown User•15mo ago

Message Not Public

MODiX•15mo ago

using var stream = File.OpenRead("lorem-ipsum.txt");
var reader = PipeReader.Create(stream);

using var stream = File.OpenRead("lorem-ipsum.txt");
var reader = PipeReader.Create(stream);

Writing (eg: to Console):

var writer = PipeWriter.Create(Console.OpenStandardOutput(), new StreamPipeWriterOptions(leaveOpen: true));

var writer = PipeWriter.Create(Console.OpenStandardOutput(), new StreamPipeWriterOptions(leaveOpen: true));

Unknown User•15mo ago

Message Not Public

Pobiega•15mo ago

It's probably worth the effort to do it with Pipeline/Channel/Parser manually, since at least you have control over the blackbox then 🙂 SuperSocket would easily work here, but its very opinionated and blackboxy

ShivOP•15mo ago

Yes understood the concept. Thanks for the detailed explaination.... Still have few doubts on blackbox ... Let me try with PipeReader and SuperSocket ... @TeBeClone @Pobiega

Pobiega•15mo ago

We used it to partially implement HL7 2.8, so we had thousands of different message types where it made more sense

Unknown User•15mo ago

Message Not Public

Pobiega•15mo ago

yeah

Accord•15mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

ShivOP•15mo ago

Sorry for reopening the same thread... I got information on what happening at datalogger..Just need info on couple of things .. They are using only two commands "!" for real time data and ";" for full data download . Once we issue ";" , its starts giving data from start to end. this is happy path. if its get interrupted in the middle , next time when I connect ,even before I issue the command, its starts reading the data from where ever it has stopped before. So there might be a buffer from where my client is reading the data. can we clear the buffer before i read? @Pobiega @TeBeClone

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

They told they dont have control over the buffer..Its upto us to clear the buffer.... Will my laptop have buffer( its just a wild guess ) because I connected from two different client... hercules and my c# client once i disconnect from hercules , my c# client started reading from where hercules has stopped..

ShivOP•15mo ago

RIght is hercules client and left is log from our client...so it has continued downloading from two different client ... there has to be a common buffer....

ShivOP•15mo ago

re-establish not connection / new pipe etc ....? can you please tell about reestablish ? yes reconnecting is not working .... reconnect starts download even before i give the command

Unknown User•15mo ago

Message Not Public

ShivOP•15mo ago

hercules is just a tcpclient Ok let me do a rough flow diagram See if you can understand better

ShivOP•15mo ago

ShivOP•15mo ago

So the point is data is not lost even if i connect from two different client . but what i need is complete data. not half data. so i need to clear the buffer... can we clear our local buffer from our code?

Unknown User•15mo ago

Message Not Public

Omnissiah•15mo ago

and this happens even if you connect with hercules, stop, and reconnect always with hercules?

ShivOP•15mo ago

Not a problem 🙂 Yes it happens even on hercules. I executed ":" command and disconnected. As soon i connect again Its start downloading until it reaches end.. Once it reaches the end, it will wait for a new command..

Omnissiah•15mo ago

well that's weird, if i understood correctly at this point i would try connecting from two different pcs just to check if it's maintaining a sort of global session

ShivOP•15mo ago

Yes!! that will be a good check .I can be sure from which buffer it is reading... My pc or datalogger.. Thank you....Will try and update you tomo...

Accord•15mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

❔ Read large data over TCP