C#•2y ago

✅ Newtonsoft JSON and unknown data

Say I have an Input set which looks like this: Array of Unknown Json Objects as request body to my API How am I going to count the amount of unique properties in all the Json Objects within the Array that the request body contains? And how am I going to count the occurrences of each of those unique properties within the entire array of json objects? I'm hoping I can do this with Newtonsoft.Json with as little "dynamic" data as possible, but I am aware that since it's unknown what kind of data is passed into the API I am going to have to generalize a lot. Any ideas?

34 Replies

phaseshift•2y ago

Parse the various JsonValue etc and keep a count. Good luck

D.Mentia•2y ago

Dictionary<string, object>

malkavOP•2y ago

Okay so I have this:

public class RequestBody
{
  // string[], JObject, JArray, JToken[] ???
  [JsonProperty("data")] public string[] Data {get;set;}
}

public class RequestBody
{
  // string[], JObject, JArray, JToken[] ???
  [JsonProperty("data")] public string[] Data {get;set;}
}

so in my Api Methods I just go:

var requestBody = await new StreamReader(req.Body).ReadToEndAsync();
RequestBody data = JsonConvert.DeserializeObject<RequestBody>(requestBody);

foreach (var item in data.Data)
{
  // Convert Key-Value pairs into a dictionary??
}

var requestBody = await new StreamReader(req.Body).ReadToEndAsync();
RequestBody data = JsonConvert.DeserializeObject<RequestBody>(requestBody);

foreach (var item in data.Data)
{
  // Convert Key-Value pairs into a dictionary??
}

I'm not entirely sure how I can parse this

foreach (var item in request.Data)
{
    JObject i = JObject.Parse(item);
    List<JProperty> list = i.Properties().ToList();
    foreach (var prop in list)
    {
        Console.WriteLine(prop.Name);
    }
}

foreach (var item in request.Data)
{
    JObject i = JObject.Parse(item);
    List<JProperty> list = i.Properties().ToList();
    foreach (var prop in list)
    {
        Console.WriteLine(prop.Name);
    }
}

Something like this? I think I have something now but it returns a count of 1 each time, and with the data I provide I'm sure the count should be 4 (except for a single prop that should be 1)

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
RequestBody request = JsonConvert.DeserializeObject<RequestBody>(requestBody);

string[] props = Array.Empty<string>();
string[] distinctProps = Array.Empty<string>();
foreach (string item in request.Data)
{
    JProperty[] list = JObject.Parse(item).Properties().ToArray();
    props = list.Select(prop => prop.Name).ToArray();
    distinctProps = list.Select(prop => prop.Name).Distinct().ToArray();
}

AllResponseBody[] resultItems = distinctProps.Distinct().Select(property => new AllResponseBody() { UniqueProperty = property, Count = props.Count(x => x == property) }).ToArray();

return new OkObjectResult(new AllResponse { Results = resultItems });

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
RequestBody request = JsonConvert.DeserializeObject<RequestBody>(requestBody);

string[] props = Array.Empty<string>();
string[] distinctProps = Array.Empty<string>();
foreach (string item in request.Data)
{
    JProperty[] list = JObject.Parse(item).Properties().ToArray();
    props = list.Select(prop => prop.Name).ToArray();
    distinctProps = list.Select(prop => prop.Name).Distinct().ToArray();
}

AllResponseBody[] resultItems = distinctProps.Distinct().Select(property => new AllResponseBody() { UniqueProperty = property, Count = props.Count(x => x == property) }).ToArray();

return new OkObjectResult(new AllResponse { Results = resultItems });

D.Mentia•2y ago

var dictionary = JsonConvert.DeserializeObject<Dictionary<string,object>>(content);
int numProps = CountNested(dictionary);

int CountNested(Dictionary<string,object> dictionary)
{
  int numProperties = 0;
  foreach (var kvp in dictionary)
  {
    numProperties++;
    if (kvp.Value is Dictionary<string,object> childDict)
      numProperties += CountNested(childDict); 
    else if (kvp.Value is IEnumerable<Dictionary<string,object>> childDictList)
      foreach(var innerChildDict in childDictList)
        numProperties += CountNested(innerChildDict);
  }
  return numProperties;
}

var dictionary = JsonConvert.DeserializeObject<Dictionary<string,object>>(content);
int numProps = CountNested(dictionary);

int CountNested(Dictionary<string,object> dictionary)
{
  int numProperties = 0;
  foreach (var kvp in dictionary)
  {
    numProperties++;
    if (kvp.Value is Dictionary<string,object> childDict)
      numProperties += CountNested(childDict); 
    else if (kvp.Value is IEnumerable<Dictionary<string,object>> childDictList)
      foreach(var innerChildDict in childDictList)
        numProperties += CountNested(innerChildDict);
  }
  return numProperties;
}

malkavOP•2y ago

Can't do this with Newtonsoft.Json though? I think in JObject you get KeyValuePair<string, string> if you iterate over it.. But I'll see if I can use your example and edit slightly for my usage.

D.Mentia•2y ago

That is newtonsoft. JObjects are complicated and too much work (and still not strongly typed), when json already perfectly matches Dictionary<string,object> format Also worth adding is that if you do know some of the properties, you can make the model with those, and there's a newtonsoft attribute to dump any properties not in your model, into a Dictionary<string,object> on the model

malkavOP•2y ago

but the input will always be of type string[] (I checked the requirements for the API, it has to be.

D.Mentia•2y ago

a string[] deserialized from json would have just one key then

malkavOP•2y ago

the string would look something like this:

"{\n    \"key_name\": \"value\",\n    ..... \n}"

"{\n    \"key_name\": \"value\",\n    ..... \n}"

so I can parse that as a JObject

D.Mentia•2y ago

right, but the value could possibly itself be an array, or a nested dictionary object You could probably parse it into a List<JObject> and check them individually like that, the dictionaries are just kind of easier

malkavOP•2y ago

The value of "data" is an array of the string objects I just exampled, these can contain their own arrays or nested objects yea Oh, and I see a flaw in my explanation already too... feck...

D.Mentia•2y ago

Also I find it hard to believe that your API doesn't tell you what properties to expect Not that I don't believe you just that they really should have a better way to send those than as unidentified properties

D.Mentia•2y ago

if they even have an example json that contains all possible ones, there's this

malkavOP•2y ago

I forgot to add that I have to return a count for each of the properties so for example I have these input strings:

"{\n    \"name\": \"SomeName\",\n    \"last_name\": \"SomeLastName\",\n    \"street\": \"SomeStreetistan\",\n    \"number\": \"1111\",\n    \"zip_code\": \"8888AA\",\n    \"city\": \"Cityname\"\n}",

"{\n    \"name\": \"SomeName\",\n    \"last_name\": \"SomeLastName\",\n    \"street\": \"SomeStreetistan\",\n    \"number\": \"1111\",\n    \"zip_code\": \"8888AA\",\n    \"city\": \"Cityname\"\n}",

like 600 times this object with different "values" for each of the items. What I want is a return body that looks like this:

{
  "results": [
    { "property": "name", "count": 600 },
    { "property": "last_name", "count": 600 },
    { "property": "street", "count": 600 },
    { "property": "number", "count": 600 },
    { "property": "zip_code", "count": 600 },
    { "property": "city", "count": 600 },
  ]
}

{
  "results": [
    { "property": "name", "count": 600 },
    { "property": "last_name", "count": 600 },
    { "property": "street", "count": 600 },
    { "property": "number", "count": 600 },
    { "property": "zip_code", "count": 600 },
    { "property": "city", "count": 600 },
  ]
}

and if any of the objects have another property in them that does not exist in the rest, it's just added to this results array with its own count

D.Mentia•2y ago

Yeah that's a lot easier if you can just build the models Like that's just a list of

public class Person
    {
        public string name { get; set; }
        public string last_name { get; set; }
        public string street { get; set; }
        public string number { get; set; }
        public string zip_code { get; set; }
        public string city { get; set; }
    }

public class Person
    {
        public string name { get; set; }
        public string last_name { get; set; }
        public string street { get; set; }
        public string number { get; set; }
        public string zip_code { get; set; }
        public string city { get; set; }
    }

malkavOP•2y ago

I can't because the json objects can litteraly contain anything. Basically the use-case would be: User has a database full of data, grabs one of their tables and converts each row into a json object. Then calls my API to count the number of occurrences of each column in the dataset. So the user grabs an array of JSON objects from the database (convert row into object, or use document based database) and sends it off. My API has to tell the amount of occurrences in that dataset without knowing (or caring) what property names and values are in there. The example I gave is just that, and example. But the properties and their values can be anything so no, I can't create a model ahead of time 😅 so I need to be able to count this without caring about the model

D.Mentia•2y ago

Then sure, dictionary or jobject or whatever. Sounds like you're only interested in properties that are nested in the top level objects so it doesn't have to be that recursive

malkavOP•2y ago

Yea I do need to be that recursive, I needed to give an example that was easy to setup, I'll grab some deeper nested examples later. I just needed to have an initial starting point first

D.Mentia•2y ago

then yeah just, as above. Your input, being valid json, will have to look like

{ Results = [ { "name": "SomeName", ... } ] }

{ Results = [ { "name": "SomeName", ... } ] }

So you can at least, knowing that, parse it into a List<Dictionary<string,object>> or List<JObject> Though that's a bit less generic so maybe just do it as above. The initial dictionary will just have one key and one value (the array), and the nested stuff gets recursed and then probably have another Dictionary<string,int> to store the counts

malkavOP•2y ago

Okay, but how do I parse a string into a Dictionary<string, object>?

D.Mentia•2y ago

deserialize as normal to that type

malkavOP•2y ago

I get a feeling this method will return a total count of the entire object, instead of a count for a property. So count = 20.000 instead of "name" count = 600 but I'll give this a shot, and see if this works out

D.Mentia•2y ago

I would say that's up to you to rework but I was already reworking it

var dictionary = JsonConvert.DeserializeObject<Dictionary<string,object>>(content);
Dictionary<string,int> countedProps = CountNested(dictionary);

Dictionary<string,int> CountNested(Dictionary<string,object> dictionary)
{
  var countedProps = new Dictionary<string,int>();
  foreach (var kvp in dictionary)
  {
    countedProps.IncrementCount(kvp.Key);
        
    if (kvp.Value is Dictionary<string,object> childDict)
    {
      var childProps = CountNested(childDict);
      foreach(var ckvp in childProps)
        countedProps.IncrementCount(ckvp.Key, ckvp.Value);
    }
    else if (kvp.Value is IEnumerable<Dictionary<string,object>> childDictList)
    {
        foreach(var innerChildDict in childDictList)
        {
            var innerChildProps = CountNested(innerChildDict);
            foreach(var ickvp in innerChildProps)
                countedProps.IncrementCount(ickvp.Key, ickvp.Value);
  }
  return countedProps;
}


// In extension class...
static void IncrementCount(this Dictionary<string,object> dict, string key, int amount = 1) {
    if (dict.ContainsKey(key))
        dict[key] += amount;
    else
        dict[key] = amount;
}

var dictionary = JsonConvert.DeserializeObject<Dictionary<string,object>>(content);
Dictionary<string,int> countedProps = CountNested(dictionary);

Dictionary<string,int> CountNested(Dictionary<string,object> dictionary)
{
  var countedProps = new Dictionary<string,int>();
  foreach (var kvp in dictionary)
  {
    countedProps.IncrementCount(kvp.Key);
        
    if (kvp.Value is Dictionary<string,object> childDict)
    {
      var childProps = CountNested(childDict);
      foreach(var ckvp in childProps)
        countedProps.IncrementCount(ckvp.Key, ckvp.Value);
    }
    else if (kvp.Value is IEnumerable<Dictionary<string,object>> childDictList)
    {
        foreach(var innerChildDict in childDictList)
        {
            var innerChildProps = CountNested(innerChildDict);
            foreach(var ickvp in innerChildProps)
                countedProps.IncrementCount(ickvp.Key, ickvp.Value);
  }
  return countedProps;
}


// In extension class...
static void IncrementCount(this Dictionary<string,object> dict, string key, int amount = 1) {
    if (dict.ContainsKey(key))
        dict[key] += amount;
    else
        dict[key] = amount;
}

malkavOP•2y ago

Sorry for interrupting 😅

D.Mentia•2y ago

Though tbh JObject could be cleaner. I just never worked with them and the dictionaries make more sense to me (and I don't have to look it up when I'm writing freehand code in discord 😛 ). Either way

malkavOP•2y ago

I think doing a foreach over a JObject also turns into a KeyValuePair<T,P> Let me see if I can translate this 😅

D.Mentia•2y ago

Basically it iterates over the main dictionary, incrementing something like countedProps["Name"] when it sees a property with key "Name". Then if the value contains properties (a dictionary) or is a list containing properties, run the method on the inner thing, and add its results into countedProps But, this doesn't catch all cases; if there is an array containing arrays of objects with keys on them, it wouldn't get to those, for example Unfortunately I've done this before... lol

malkavOP•2y ago

lol yea 😅 I am starting to get the "unfortunately" part lol

D.Mentia•2y ago

Very very similar goal actually, in my case I wanted to take lots of unknown properties and flatten them so that I can output a single 'row', with each column being a property (or an inner-object's property)

malkavOP•2y ago

I've got a meeting to dive into that will last me another 3 hours, after which I'll check again if I can come up with something lol yea it's almost the same.. Mine needs to basically return an array of simple objects

{
  "prop_name": "actual name",
  "prop_count": "actual count"
}

{
  "prop_name": "actual name",
  "prop_count": "actual count"
}

but I'll be back later, thanks for the patience!

D.Mentia•2y ago

It's a lot easier with JObjects actually

MODiX•2y ago

D.Mentia#0614

REPL Result: Success

var jo = JsonConvert.DeserializeObject<JObject>(@"
{
    ""People"": [
        { ""Name"": ""Bob"", ""Thing"": ""Yes"" },
        { ""Name"": ""Alice"", ""Thing2"": ""No""}
    ]
}");
var countedProps = CountNested(jo);
Console.WriteLine(JsonConvert.SerializeObject(countedProps));

Dictionary<string, int> CountNested(JObject jo)
{
    var countedProps = new Dictionary<string, int>();
    foreach (var tokenPair in jo)
    {
        IncrementCount(countedProps, tokenPair.Key);
        foreach(JObject childJo in tokenPair.Value.Children().OfType<JObject>())
        {
            var childResults = CountNested(childJo);
            foreach (var resultKvp in childResults)
IncrementCount(countedProps, resultKvp.Key, resultKvp.Value);
        }
    }
    
    return countedProps;
}

void IncrementCount(Dictionary<string, int> dict, string key, int amount = 1)
{
    if (dict.ContainsKey(key))
        dict[key] += amount;
    else
        dict[key] = amount;
}

var jo = JsonConvert.DeserializeObject<JObject>(@"
{
    ""People"": [
        { ""Name"": ""Bob"", ""Thing"": ""Yes"" },
        { ""Name"": ""Alice"", ""Thing2"": ""No""}
    ]
}");
var countedProps = CountNested(jo);
Console.WriteLine(JsonConvert.SerializeObject(countedProps));

Dictionary<string, int> CountNested(JObject jo)
{
    var countedProps = new Dictionary<string, int>();
    foreach (var tokenPair in jo)
    {
        IncrementCount(countedProps, tokenPair.Key);
        foreach(JObject childJo in tokenPair.Value.Children().OfType<JObject>())
        {
            var childResults = CountNested(childJo);
            foreach (var resultKvp in childResults)
IncrementCount(countedProps, resultKvp.Key, resultKvp.Value);
        }
    }
    
    return countedProps;
}

void IncrementCount(Dictionary<string, int> dict, string key, int amount = 1)
{
    if (dict.ContainsKey(key))
        dict[key] += amount;
    else
        dict[key] = amount;
}

Console Output

{"People":1,"Name":2,"Thing":1,"Thing2":1}

{"People":1,"Name":2,"Thing":1,"Thing2":1}

Compile: 673.815ms | Execution: 150.629ms | React with ❌ to remove this embed.

malkavOP•2y ago

Hey man, sorry I eventually returned home by 11pm, so I didn't get back to it. I'm back on it now, and your method seems solid, I'm just trying to figure out how to translate string[] into JObject 😅 Because I'm passing the class RequestBody into my initial JsonConvert.DeserializeObject<RequestBody>(req.Body); Which looks like this:

public class RequestBody
{
  [OpenApiProperty][JsonProperty("data")] public string[] Data {get;set;}
}

public class RequestBody
{
  [OpenApiProperty][JsonProperty("data")] public string[] Data {get;set;}
}

Which technically I should be able to convert string[] into JObject] but then I get this error saying

Can not add Newtonsoft.Json.Linq.JValue to Newtonsoft.Json.Linq.JObject.

Can not add Newtonsoft.Json.Linq.JValue to Newtonsoft.Json.Linq.JObject.

So I tried just transforming each item of the string[] into its own JObject (which is possible) but then I get numerous amounts of seperate Dictionaries instead of a single one. So what I also tried was Dictionary<string, int> countedProperties = CountNested(request.Data); but then I get the following error:

System.Private.CoreLib: Exception while executing function: All. Newtonsoft.Json: Deserialized JSON type 'Newtonsoft.Json.Linq.JArray' is not compatible with expected type 'Newton
soft.Json.Linq.JObject'. Path 'data', line 7, position 3.

System.Private.CoreLib: Exception while executing function: All. Newtonsoft.Json: Deserialized JSON type 'Newtonsoft.Json.Linq.JArray' is not compatible with expected type 'Newton
soft.Json.Linq.JObject'. Path 'data', line 7, position 3.

so I'm a little confused 😅 Ooh, I think I got it! So here's my initial input from the user:

{
  "data": [
    "{\n    \"name\": \"First name\",\n    \"last_name\": \"Some Last Name\",\n    \"street\": \"Street-ishtan\",\n    \"number\": \"9999\",\n    \"zip_code\": \"1234AA\",\n    \"city\": \"Some City\"\n}",
  // Repeat as many times as neccecary, and yes I am aware this is only first level deep. I'm working on a test case with nested Items
  ]
}

{
  "data": [
    "{\n    \"name\": \"First name\",\n    \"last_name\": \"Some Last Name\",\n    \"street\": \"Street-ishtan\",\n    \"number\": \"9999\",\n    \"zip_code\": \"1234AA\",\n    \"city\": \"Some City\"\n}",
  // Repeat as many times as neccecary, and yes I am aware this is only first level deep. I'm working on a test case with nested Items
  ]
}

and here's the initial input at the API:

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
RequestBody request = JsonConvert.DeserializeObject<RequestBody>(requestBody);


Dictionary<string, int> countedProperties = new();
countedProperties = request.Data.Aggregate(countedProperties, (current, item) => JObject.Parse(item).CountNested(current));

return new OkObjectResult(countedProperties);

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
RequestBody request = JsonConvert.DeserializeObject<RequestBody>(requestBody);


Dictionary<string, int> countedProperties = new();
countedProperties = request.Data.Aggregate(countedProperties, (current, item) => JObject.Parse(item).CountNested(current));

return new OkObjectResult(countedProperties);

And the CountNested method you wrote, but I turned it into an extension

private static Dictionary<string, int> CountNested(this JObject obj, Dictionary<string, int> countedProperties = null)
{
    Dictionary<string, int> countedProps = countedProperties ?? new();
    foreach (KeyValuePair<string, JToken> kvp in obj)
    {
        countedProps.IncrementCount(kvp.Key);
        foreach (JObject childJobject in kvp.Value.Children().OfType<JObject>())
        {
            Dictionary<string, int> childResults = CountNested(childJobject);
            foreach (KeyValuePair<string, int> resultKvp in childResults)
            {
                countedProps.IncrementCount(resultKvp.Key, resultKvp.Value);
            }
        }
    }
    return countedProps;
}

private static Dictionary<string, int> CountNested(this JObject obj, Dictionary<string, int> countedProperties = null)
{
    Dictionary<string, int> countedProps = countedProperties ?? new();
    foreach (KeyValuePair<string, JToken> kvp in obj)
    {
        countedProps.IncrementCount(kvp.Key);
        foreach (JObject childJobject in kvp.Value.Children().OfType<JObject>())
        {
            Dictionary<string, int> childResults = CountNested(childJobject);
            foreach (KeyValuePair<string, int> resultKvp in childResults)
            {
                countedProps.IncrementCount(resultKvp.Key, resultKvp.Value);
            }
        }
    }
    return countedProps;
}

as well as the IncrementCount method:

private static void IncrementCount(this Dictionary<string, int> dict, string key, int amount = 1)
{
    if (dict.ContainsKey(key))
        dict[key] += amount;
    else
        dict[key] = amount;
}

private static void IncrementCount(this Dictionary<string, int> dict, string key, int amount = 1)
{
    if (dict.ContainsKey(key))
        dict[key] += amount;
    else
        dict[key] = amount;
}

And then the result: (After repeating it 4 times with a single object containing a unique property)

{
  "name": 4,
  "last_name": 4,
  "street": 4,
  "number": 4,
  "zip_code": 4,
  "city": 4,
  "email": 1
}

{
  "name": 4,
  "last_name": 4,
  "street": 4,
  "number": 4,
  "zip_code": 4,
  "city": 4,
  "email": 1
}

I'll leave the ticket open until you see my thank you here 😅 you've been an amazing patient one to help me, so thank you very much!! Oh, and some nested data:

{
  "data": [
    "{\r\n\"month\": \"january\",\r\n\"days_in_month\":\r\n[\r\n { \"day\": \"monday\", \"date\": \"01-01-2000\"\r\n },\r\n { \"day\": \"tuesday\", \"date\": \"02-01-2000\"\r\n }, { \"day\": \"wednesday\", \"date\": \"03-01-2000\"\r\n }]\r\n}"
// repeat
  ]
}

{
  "data": [
    "{\r\n\"month\": \"january\",\r\n\"days_in_month\":\r\n[\r\n { \"day\": \"monday\", \"date\": \"01-01-2000\"\r\n },\r\n { \"day\": \"tuesday\", \"date\": \"02-01-2000\"\r\n }, { \"day\": \"wednesday\", \"date\": \"03-01-2000\"\r\n }]\r\n}"
// repeat
  ]
}

results in:

{
  "month": 4,
  "days_in_month": 4,
  "day": 122,
  "date": 122
}

{
  "month": 4,
  "days_in_month": 4,
  "day": 122,
  "date": 122
}

so, awesome!

Accord•2y ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

✅ Newtonsoft JSON and unknown data