C#•5mo ago

XML in an API

I'm creating an API to facing with External Restrictive List and this list is a XML in browser, but I don't wanna to facing with entire XML and I need to break in pieces to get performance, anyone can help me?

13 Replies

theeuzinOP•5mo ago

this is my service

public class CsnuService : ICsnuService
{
    private readonly HttpClient _httpClient;

    public CsnuService(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<IEnumerable<XElement>> GetIndividualElementsAsync(string url)
    {
        if (string.IsNullOrEmpty(url))
        {
            throw new ArgumentException("URL cannot be empty.", nameof(url));
        }

        using var response = await _httpClient.GetStreamAsync(url);
        using var reader = XmlReader.Create(response);
        return GetIndividualElements(reader);
    }

    public IEnumerable<XElement> GetIndividualElements(XmlReader reader)
    {
        while (reader.Read())
        {
            if (reader.NodeType == XmlNodeType.Element && reader.Name == "INDIVIDUALS")
            {
                while (reader.ReadToFollowing("INDIVIDUAL"))
                {
                    yield return (XElement)XElement.ReadFrom(reader);
                }
                
            }
        }
    }
}

public class CsnuService : ICsnuService
{
    private readonly HttpClient _httpClient;

    public CsnuService(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<IEnumerable<XElement>> GetIndividualElementsAsync(string url)
    {
        if (string.IsNullOrEmpty(url))
        {
            throw new ArgumentException("URL cannot be empty.", nameof(url));
        }

        using var response = await _httpClient.GetStreamAsync(url);
        using var reader = XmlReader.Create(response);
        return GetIndividualElements(reader);
    }

    public IEnumerable<XElement> GetIndividualElements(XmlReader reader)
    {
        while (reader.Read())
        {
            if (reader.NodeType == XmlNodeType.Element && reader.Name == "INDIVIDUALS")
            {
                while (reader.ReadToFollowing("INDIVIDUAL"))
                {
                    yield return (XElement)XElement.ReadFrom(reader);
                }
                
            }
        }
    }
}

HtmlCompiler•5mo ago

what is wrong with performance

theeuzinOP•5mo ago

i'm facing with a huge xml, you know?

HtmlCompiler•5mo ago

ok but at what point the performance breaks?

theeuzinOP•5mo ago

because if I work with entire xml, i'm leave in memory and that's not good, right? So i want to break this xml in pieces this is the main question, i'm not sure if i work this entire xml I lost performance

HtmlCompiler•5mo ago

ok so you would like to parse xml while streaming it or parsing "well defined" chunks you would have to save it to file and trying opening a stream im not sure about the xmlreader capability to parse it in that way but the only way to know is trying

theeuzinOP•5mo ago

yes because i have a XML and a I need to put the information in a database i was trying to do this, but i always receive exception about the length, so i decide to ask here

HtmlCompiler•5mo ago

🤔 is it bigger than 2 GB?

theeuzinOP•5mo ago

i'm not sure, but i have more than 200 pages i'm looking for to know that i saw here, have 2,2mb its so light, but i'm so confuse with the exceptions

theeuzinOP•5mo ago

look at that

HtmlCompiler•5mo ago

yeah i wouldn't use a stream returned from httpclient in that way, i don't think it can evaluate Length you really ought to save it to a temporary file

theeuzinOP•5mo ago

ok, ty

Cracker•5mo ago

you can try to read document as string and use Regex, benchmark whichever is faster (for sure after getting the pages in temp file or variable or smt)

Gaming

Programming

XML in an API