C#•2y ago

IAsyncEnumerable taking to much proccess memory [Answered]

Is there a way to free up memory after each iteration?

        public async Task TransferFileAsync(string fileId, IAsyncEnumerable<byte[]> chunks)
        {
            var file = await _fileService.GetFileAsync(Context.ConnectionId, fileId);

            if (file == null)
                return;

            if (file.Acceptance != TransferAcceptance.Accepted)
                return;

            await foreach (var chunk in chunks)
            {
                await Clients.Client(file.RecipientId).SendAsync("OnReceiveFile", new FileChunk()
                {
                    FileId = file.Id,
                    Chunk = chunk
                });
            }

            await _fileService.RemoveFileAsync(Context.ConnectionId, fileId);
        }

        public async Task TransferFileAsync(string fileId, IAsyncEnumerable<byte[]> chunks)
        {
            var file = await _fileService.GetFileAsync(Context.ConnectionId, fileId);

            if (file == null)
                return;

            if (file.Acceptance != TransferAcceptance.Accepted)
                return;

            await foreach (var chunk in chunks)
            {
                await Clients.Client(file.RecipientId).SendAsync("OnReceiveFile", new FileChunk()
                {
                    FileId = file.Id,
                    Chunk = chunk
                });
            }

            await _fileService.RemoveFileAsync(Context.ConnectionId, fileId);
        }

39 Replies

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

So this method is being called by the client to upload a file. Now even when process is done. It doesn't seem to free up memory

MODiX•2y ago

tebeco#0205

need the full code

Quoted by

<@!689473681302224947> from #IAsyncEnumerable taking to much proccess memory (click here)

React with ❌ to remove this embed.

SwaggerLife•2y ago

This is the full.

var file = await _fileService.GetFileAsync(Context.ConnectionId, fileId);

var file = await _fileService.GetFileAsync(Context.ConnectionId, fileId);

is only getting details about the file. It doesn't hold the data.

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

Yes the file isn't being sent all at once. It's being streamed into small chunks. The client-is is a flutter application. Not sure if the code is going to help.

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

Client:

  Future<void> transferFileAsync(FileItem file) async {
    try {
      var stream = File(file.path).openRead();
      Subject subject = Subject();
      stream.listen((event) {
        subject.next(event);
      }, onDone: () => subject.complete());
      await _connection.invokeAsync("TransferFileAsync", [file.id, subject]);
    } catch (_) {}
  }

  Future<void> transferFileAsync(FileItem file) async {
    try {
      var stream = File(file.path).openRead();
      Subject subject = Subject();
      stream.listen((event) {
        subject.next(event);
      }, onDone: () => subject.complete());
      await _connection.invokeAsync("TransferFileAsync", [file.id, subject]);
    } catch (_) {}
  }

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

Yes, now before you begin. I know signalR is bad for file transferring 😂

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

Any idea, how I could fix this?

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

Sure let me try it.

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

It now looks like this

        public async Task TransferFileAsync(string fileId, ChannelReader<int[]> stream)
        {
            var file = await _fileService.GetFileAsync(Context.ConnectionId, fileId);

            if (file == null)
                return;

            if (file.Acceptance != TransferAcceptance.Accepted)
                return;


            while (await stream.WaitToReadAsync())
            {
                while (stream.TryRead(out var item))
                {
                    if(item != null)
                    {
                        await Clients.Client(file.RecipientId).SendAsync("OnReceiveFile", new FileChunk()
                        {
                            FileId = file.Id,
                            Chunk = item
                        });
                    }
                }
            }

            await _fileService.RemoveFileAsync(Context.ConnectionId, fileId);
        }

        public async Task TransferFileAsync(string fileId, ChannelReader<int[]> stream)
        {
            var file = await _fileService.GetFileAsync(Context.ConnectionId, fileId);

            if (file == null)
                return;

            if (file.Acceptance != TransferAcceptance.Accepted)
                return;


            while (await stream.WaitToReadAsync())
            {
                while (stream.TryRead(out var item))
                {
                    if(item != null)
                    {
                        await Clients.Client(file.RecipientId).SendAsync("OnReceiveFile", new FileChunk()
                        {
                            FileId = file.Id,
                            Chunk = item
                        });
                    }
                }
            }

            await _fileService.RemoveFileAsync(Context.ConnectionId, fileId);
        }

But the process memory is not dropping 😦

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

I sent data twice. It went from 130 mb to 460 mb

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

Okay let's see 😂

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

The client is sending the bytes as an collection of int's. JsonSerializer had problem converting it. So I had to change from byte[] to int[].

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

I tried using MessagePackProtocol. But the methods on the hub aren't getting triggered. It could be because of the signalR client that I'm using. Let me try MessagePackProtocol again. Maybe it will work. By the I tried calling GC.Collect after each iteration and the memory never went up. But I don't think it's a good idea.

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

You were right, it seems like the GC is getting activated after a certain amount of bytes.

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

I really appreciate your help mate. Sincerely blobthumbsup

Unknown User•2y ago

Message Not Public

MODiX•2y ago

tebeco#0205

i would have to re-open signalr code but I think it's because you can't put bare byte[] at Channel level

public class Foo { public byte[] Data{get;set;} }
ChannelReader<Foo>

public class Foo { public byte[] Data{get;set;} }
ChannelReader<Foo>

Quoted by

<@!689473681302224947> from #IAsyncEnumerable taking to much proccess memory (click here)

React with ❌ to remove this embed.

SwaggerLife•2y ago

Yeah I will take a look into why it's not working and get it to work. It's probably some form of bug from the client side.

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

I see so something like this

    public record struct FileData
    {
        public byte[] Bytes { get; set; }
    }

    public record struct FileData
    {
        public byte[] Bytes { get; set; }
    }

        public async Task TransferFileAsync(string fileId, IAsyncEnumerable<FileData> chunks)
        {
            var file = await _fileService.GetFileAsync(Context.ConnectionId, fileId);

            if (file == null)
                return;

            if (file.Acceptance != TransferAcceptance.Accepted)
                return;

            await foreach (var chunk in chunks)
            {
                await Clients.Client(file.RecipientId).SendAsync("OnReceiveFile", new FileChunk()
                {
                    FileId = file.Id,
                    Chunk = chunk.Bytes
                });
            }

            await _fileService.RemoveFileAsync(Context.ConnectionId, fileId);
        }

        public async Task TransferFileAsync(string fileId, IAsyncEnumerable<FileData> chunks)
        {
            var file = await _fileService.GetFileAsync(Context.ConnectionId, fileId);

            if (file == null)
                return;

            if (file.Acceptance != TransferAcceptance.Accepted)
                return;

            await foreach (var chunk in chunks)
            {
                await Clients.Client(file.RecipientId).SendAsync("OnReceiveFile", new FileChunk()
                {
                    FileId = file.Id,
                    Chunk = chunk.Bytes
                });
            }

            await _fileService.RemoveFileAsync(Context.ConnectionId, fileId);
        }

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

Just one last question. It seems like I have got MessagePack to work. Is there a way to tell it to serialize enums as name rather than index?

Unknown User•2y ago

Message Not Public

SwaggerLife•2y ago

I see, I will take a look at the readme of messagepack. Thank you very much mate.

Unknown User•2y ago

Message Not Public

Accord•2y ago

✅ This post has been marked as answered!

Gaming

Programming

IAsyncEnumerable taking to much proccess memory [Answered]