C#•2y ago

❔ Best way to you Read lot of files?

I need something like a Filestream but where I can change the file thats read but each time I want to read another file I need to create a new FileStream which does extra allocation. See here:

    public void T2()
    {
        var fs = new FileStream(@"./File.txt", FileMode.Open);
        byte[] buffer = new byte[500];

        for (int i = 0; i < fs.Length; i++)
        {
            buffer[i] = (byte)fs.ReadByte();
        }

        // do somthing with bytes in buffer where I get another file location

        // need to do this cause we want another File but
        fs = new FileStream(@"./AnotherFile.txt", FileMode.Open);

        // Do someting with file

        // need to do this cause we want another File but
        fs = new FileStream(@"./YetAnotherFile.txt", FileMode.Open);

        // and so on

        fs.Close();

    }

    public void T2()
    {
        var fs = new FileStream(@"./File.txt", FileMode.Open);
        byte[] buffer = new byte[500];

        for (int i = 0; i < fs.Length; i++)
        {
            buffer[i] = (byte)fs.ReadByte();
        }

        // do somthing with bytes in buffer where I get another file location

        // need to do this cause we want another File but
        fs = new FileStream(@"./AnotherFile.txt", FileMode.Open);

        // Do someting with file

        // need to do this cause we want another File but
        fs = new FileStream(@"./YetAnotherFile.txt", FileMode.Open);

        // and so on

        fs.Close();

    }

Is there a better way which dont generate as much or even better, no(extra) garbage?

8 Replies

Lord Jinxy•2y ago

I guess you might try File.ReadAllText() method.

the_sea_jayOP•2y ago

Seems like this allocates even more per use than a FileStream

ACiDCA7•2y ago

did you try to set the filestreamoption buffersize to 0, according to docs this would disable the buffer and hopefully the allocation for it.. never used it tho so take it with a grain of salt what im more interested in tho is, what is it for? are you trying read thousands of files at once? btw it is recommended to use the usingpattern since filestream implements idisposable, even the docs say to better use dispose btw you are always creating new filestreams but never close them, so potential memoryleak

the_sea_jayOP•2y ago

setting it to 0 or 1 does disable it but there are still other allocations, not that big anymore (without the buffer Filestream does allocate ~200b instead of 4kb) Well I dont know how many Files need to be read(also because of user input) but I wanted to make it flexible so if I need to read thousands of files it should do so and possible without much or any extra allocations so it wont run the Collector 2 much I Could make a static Filestream , but that does not change the problem with new Files

ACiDCA7•2y ago

i dont think you can completely get rid of all allocations.. to my understanding, to be able to read files the OS has to provide handles which would have to be managed, therefore at least some allocations per file to be honest, did you do any benchmarking/testing? is it even a problem currently? if not use your time more productively instead of saving couple megs of ram

the_sea_jayOP•2y ago

Yea I did Benchmark with Benchmarkdotnet and at this point it is more in my own interest rather than an actual real scenario problem. But is there maybe some unsafe way can solve this? Thanks for any info so far!

ACiDCA7•2y ago

my personal opinion is that as long you arent running into any problems you should let the garbagecollector do its job. just properly dispose the streams and the collector will most likely free the memory in batches when necessary.

Accord•2y ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

❔ Best way to you Read lot of files?