C#•2mo ago

Dynamic parsing different document types from Elastic Search

I am trying to fetch documents with different types of fields and nested fields from elastic search using NEST SDK and I am having issues parsing the documents into their appropriate types for example YoutubeVideo or TrelloCard which have different fields and structure. I am doing a free text search across all fields for the different types of documents in a single index and want to dynamically map them to specific C# model class for example Class TrelloCard. I have tried using Generics, pattern matching and so on. However I am unable to return the Source object eg. TrelloCard or YoutubeVideo from my search method. The JSON search result from Elastic search looks like this for the YoutubeVideo document type and the document data that I want to extract and parse is in the "_source" object:

{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 205,
      "relation" : "eq"
    },
    "max_score" : 1.5206873,
    "hits" : [
      {
        "_index" : "global-dev",
        "_type" : "_doc",
        "_id" : "23qNtJIBgw6_7scS9GAD",
        "_score" : 1.5206873,
        "_source" : {
          "kind" : "youtube#playlistItem",
          "etag" : "iHgQaZpSuxlCTGAHG8zsESDCVDA",
          "id" : "UExnS01oN1JBZkVWY2GAek1USk04MlhEVHZ1ZlV5MDBuZS4yODlGNEE0NkRGMEEzMEQy",               
          "type" : "youtube-videos",
          "url": "https://www.youtube.com/watch?v=KAH_D3ELDA",
            "NestedObject": {
                "Description": "Test"
             }
        }
      }   
    ]
  }
}

{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 205,
      "relation" : "eq"
    },
    "max_score" : 1.5206873,
    "hits" : [
      {
        "_index" : "global-dev",
        "_type" : "_doc",
        "_id" : "23qNtJIBgw6_7scS9GAD",
        "_score" : 1.5206873,
        "_source" : {
          "kind" : "youtube#playlistItem",
          "etag" : "iHgQaZpSuxlCTGAHG8zsESDCVDA",
          "id" : "UExnS01oN1JBZkVWY2GAek1USk04MlhEVHZ1ZlV5MDBuZS4yODlGNEE0NkRGMEEzMEQy",               
          "type" : "youtube-videos",
          "url": "https://www.youtube.com/watch?v=KAH_D3ELDA",
            "NestedObject": {
                "Description": "Test"
             }
        }
      }   
    ]
  }
}

3 Replies

HtmlCompiler•2mo ago

how exactly are you searching in elastic, do you use ElasticClient.SearchAsync or...?

CasperOP•2mo ago

Yes I use NEST ElasticClient.SearchAsync:

        var searchResponse = await _client.SearchAsync<SearchResult>(s => s
var searchResponse = await _client.SearchAsync<SearchResult>(s => s
    .Index("global-dev")  // Specify your index name
    .From(0)              // Starting from the first result
    .Size(100)            // Limit the number of results to 100
    .Query(q => q
        .MultiMatch(m => m
            .Query("Bean")    // Search for the term "Bean"
            .Fields(f => f.Field("*"))  // Search across all fields
        )
    )
);

        var searchResponse = await _client.SearchAsync<SearchResult>(s => s
var searchResponse = await _client.SearchAsync<SearchResult>(s => s
    .Index("global-dev")  // Specify your index name
    .From(0)              // Starting from the first result
    .Size(100)            // Limit the number of results to 100
    .Query(q => q
        .MultiMatch(m => m
            .Query("Bean")    // Search for the term "Bean"
            .Fields(f => f.Field("*"))  // Search across all fields
        )
    )
);

I can show the model schemas as well for the classes in use from the SearchResult Hey, I just wanted to follow up 🙂

HtmlCompiler•2mo ago

by "i tried using generics" you mean you tried SearchAsync<TDocument>? did you look at the query done from the client? tbh i don't remember at the moment if it excludes the _source but i can parse objects with no issues so i think it doesn't

Gaming

Programming

Dynamic parsing different document types from Elastic Search