C#•2y ago

❔ streaming an IAsyncEnumerable<string> from complex type

hey. i have a minimal api that looks like this:

public static async IAsyncEnumerable<string> AskOpenAi(
    [FromQuery] string question,
    [FromServices] IOpenAIService openAiService)
{
    var completionResult = openAiService.ChatCompletion.CreateCompletionAsStream(new ChatCompletionCreateRequest
    {
        Messages = new List<ChatMessage>
        {
            new(StaticValues.ChatMessageRoles.System, "You are a helpful assistant."),
            new(StaticValues.ChatMessageRoles.User, question),
        },
        Model = Models.ChatGpt3_5Turbo,
        MaxTokens = 150
    });

    await foreach (var completion in completionResult)
        yield return completion.Choices.First().Message.Content;
}

public static async IAsyncEnumerable<string> AskOpenAi(
    [FromQuery] string question,
    [FromServices] IOpenAIService openAiService)
{
    var completionResult = openAiService.ChatCompletion.CreateCompletionAsStream(new ChatCompletionCreateRequest
    {
        Messages = new List<ChatMessage>
        {
            new(StaticValues.ChatMessageRoles.System, "You are a helpful assistant."),
            new(StaticValues.ChatMessageRoles.User, question),
        },
        Model = Models.ChatGpt3_5Turbo,
        MaxTokens = 150
    });

    await foreach (var completion in completionResult)
        yield return completion.Choices.First().Message.Content;
}

it's registered like this:

app.MapGet("", AskOpenAiEndpoint.AskOpenAi);

app.MapGet("", AskOpenAiEndpoint.AskOpenAi);

but whenever i visit the url, the answer is not being streamed but rather returend as one large string. i tried making a dummy endpoint that works like expected, but im unsure how to modify the above code so it works. this is the test that worked for me:

public static async IAsyncEnumerable<string> Test()
{
    var values = new[] { "value1", "value2", "value3", "value4" };
    foreach (var item in values)
    {
        await Task.Delay(1000);
        yield return item;
    }
}

public static async IAsyncEnumerable<string> Test()
{
    var values = new[] { "value1", "value2", "value3", "value4" };
    foreach (var item in values)
    {
        await Task.Delay(1000);
        yield return item;
    }
}

6 Replies

cumslvt13•2y ago

You have misconception about IAsyncEnumerable and it's role in asp.net core. Returning IAsynEnumerable from endpoint doesn't make it stream the data to the client, but rather stream to the json serializer that will serialize it and return it to a client If you're interested in streaming - http is not the right thing to do that You can take a look at websockets and especially SignalR

cumslvt13•2y ago

https://learn.microsoft.com/en-us/aspnet/core/signalr/streaming?view=aspnetcore-7.0

Use streaming in ASP.NET Core SignalR

Learn how to stream data between the client and the server.

ægteemilOP•2y ago

ahh gotcha. how come it works like i expected in my second example? i considered doing it with signalr, but since my simple example worked i thought i could get away without setting it up

cumslvt13•2y ago

I give a test run to your second example, and aparently it works, which honestly wasn't was I expected It sends results in chunks, but still it's not proper streaming, since connection is closed afterwards My guess you you just see a string is because delay is too short If you change Task.Delay to lik 10ms, you won't notice anything

ægteemilOP•2y ago

okay, i'll spin up a signalr hub and get it working that way. thanks 🙂

Accord•2y ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

❔ streaming an IAsyncEnumerable<string> from complex type