C#•15mo ago

❔ FTP Uploading Files bigger than 2GB

this is quite a difficult one to put to paper, but i have teams/discord and if need me and I can live share the code. this may be an easier option in my WinForms Application I have the ability to upload using the WinSCP API. now part of this is its generates a file list so that if I push the repository through again it will only upload new files or modified files. to do this it puts the whole file in memory to generate some kind of checksum of the file using File Information (I'm still not sure how this code works as I have used some of it from a Game Updater. when I wasn't using files over 2GB this was working flawlessly, how ever I'm now moving on to push larger files I'm reaching the 2GB limit with File.ReadAllBytes (and the 32bit windows memory issue) so some research suggests I should be using File Stream, so I've modified my code for that but now its not copying the byte data over correctly. I tried a couple 5gb+ files and it only wrote between 1gb and 1.7gb on the FTP server. I think it will be something completely obvious I but I'm loosing the will to live lol!!!!!

        public byte[] FileToByteArray(string fileName)
        {
            byte[] fileData = null;

            using (FileStream fs = File.OpenRead(fileName))
            {
                using (BinaryReader binaryReader = new BinaryReader(fs))
                {
                    fileData = binaryReader.ReadBytes((int)fs.Length);
                }
            }
            return fileData;
        }

        public byte[] FileToByteArray(string fileName)
        {
            byte[] fileData = null;

            using (FileStream fs = File.OpenRead(fileName))
            {
                using (BinaryReader binaryReader = new BinaryReader(fs))
                {
                    fileData = binaryReader.ReadBytes((int)fs.Length);
                }
            }
            return fileData;
        }

my byte reading code. TYIA , any and all help appreciated

18 Replies

Mayor McCheese•15mo ago

Don't try to ftp such a large file; break it up into segments; send smaller segments and reassemble on the other end

GibboOP•15mo ago

il be honest im not sure how to do that and my brains fried from getting this far lol so a bit more reserching suggest i should be using byte buffers.?

Mayor McCheese•15mo ago

something like...

using FileStream fs = File.OpenRead(fileName);
var bytesRead = 0;
byte[] bytes = new byte[10_000];
do
{
 bytesRead = fs.Read(bytes, 0, bytes.Length);
 SendBytesViaFTP(bytes, bytesRead);
}
while(bytesRead > 0);

using FileStream fs = File.OpenRead(fileName);
var bytesRead = 0;
byte[] bytes = new byte[10_000];
do
{
 bytesRead = fs.Read(bytes, 0, bytes.Length);
 SendBytesViaFTP(bytes, bytesRead);
}
while(bytesRead > 0);

the above is pretty old school and hand-jammed; I just wouldn't try and send a 2GB file over FTP; that's a long time for something to go wrong and have to resend a lot of bytes again; also FTP is unencrypted; just an fyi.

GibboOP•15mo ago

ive got tls setup, will eventually switch out to SFTP il take a proper gander at this and see what i can do, thank you for helping

Mayor McCheese•15mo ago

NB: there are a number of gotchas with a many-files approach; make sure you name files appropriately and with justified zeros; you'll want enough padding that sorting works appropriately; you'll need something to reassemble those files on the other side.

filepart.0001
filepart.0002
filepart.0003

filepart.0001
filepart.0002
filepart.0003

MODiX•15mo ago

Hazel | へいぜる

REPL Result: Success

byte[] sampleBytes = new byte[1_000];
IEnumerable<byte[]> chunks = sampleBytes.Chunk(100);
Console.WriteLine(chunks.Count());

byte[] sampleBytes = new byte[1_000];
IEnumerable<byte[]> chunks = sampleBytes.Chunk(100);
Console.WriteLine(chunks.Count());

Console Output

Compile: 608.439ms | Execution: 42.594ms | React with ❌ to remove this embed.

Hazel 🌊💃•15mo ago

@Mayor McCheese wouldn't using .Chunk prevent the need for do-while (as in you could use foreach instead)? Granted I think its an extra allocation, so there's that.

GibboOP•15mo ago

i shall go away and look at both examples and see if i can work out how to progress thank you both

Mayor McCheese•15mo ago

Yeah probably; I said it was old school 🙂

Hazel 🌊💃•15mo ago

I'm curious now lol This is why I have a sandbox 😄 Because I didn't know Chunk existed until Nox mentioned it over the weekend

Mayor McCheese•15mo ago

But that presupposes you can read 2gb into memory 🙂 you should be able to; but honestly I tend to shy aware from reading all the bytes at once

Hazel 🌊💃•15mo ago

That's fair 😄

Accord•15mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

GibboOP•15mo ago

thanks all, turns out uploading wasnt the issue the code was writing the file to another location using a byte array ive done this as a alternative now ,which has given me another issue when it comes to compressing the files but il put that in my next thread lol

using (FileStream sourceStream = new FileStream(fileLocation, FileMode.Open, FileAccess.Read))
using (FileStream destinationStream = new FileStream(tempFilePath, FileMode.Create, FileAccess.Write))
{
    byte[] buffer = new byte[4096]; // 4KB buffer size (can be adjusted)

    int bytesRead;

    if (fileName != patcherFilename && patcherEnableCompression)
    {
        fileName += ".gz";
    }

    while ((bytesRead = sourceStream.Read(buffer, 0, buffer.Length)) > 0)
    {
        //buffer = (!patcherEnableCompression || fileName == patcherFilename) ? buffer : Compress(buffer);
        //info.Compressed = buffer.Length;
        destinationStream.Write(buffer, 0, bytesRead);
    }
}

using (FileStream sourceStream = new FileStream(fileLocation, FileMode.Open, FileAccess.Read))
using (FileStream destinationStream = new FileStream(tempFilePath, FileMode.Create, FileAccess.Write))
{
    byte[] buffer = new byte[4096]; // 4KB buffer size (can be adjusted)

    int bytesRead;

    if (fileName != patcherFilename && patcherEnableCompression)
    {
        fileName += ".gz";
    }

    while ((bytesRead = sourceStream.Read(buffer, 0, buffer.Length)) > 0)
    {
        //buffer = (!patcherEnableCompression || fileName == patcherFilename) ? buffer : Compress(buffer);
        //info.Compressed = buffer.Length;
        destinationStream.Write(buffer, 0, bytesRead);
    }
}

Mayor McCheese•15mo ago

I'd still be wary of uploading a massive file via ftp; ftp is notorious for unexpected disconnects.

GibboOP•15mo ago

im using WinSCP for ftp management now

Mayor McCheese•15mo ago

I don't know if that's more reliable or not

Accord•15mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

❔ FTP Uploading Files bigger than 2GB