How to search database for matching text efficiently with C# (maybe with tokens)
In python there is something called a tokenizer that convert strings to tokens and we can use search through those tokens to get the text.
(I used it in my last python job).
Is there a c# equivalent?
12 Replies
what kind of tokens?
can you provide an example like what kind of searching you're looking for .
NLTK, ( Natural Language Tool kit) Thats what I used at that time
Let's say I am searching for name "Daniel Andrew Simons", and I send in "andrew daniel" as a request. It is supposed to return list of items with "Daniel Andrew Simons" included.
Word Tokens
I doubt if you can query the database or collection like that in c#, since the query should match one of the entries in the db to be able to return something. "andrew daniel" and "daniel andrew" are two different things. At least I'm not aware if this can be done .
Damn. Thank you very much
Is there a way i can structure my database to do this though?
try asking this in #database as well
Okay
Use a vector database for search indexing maybe?
Okay, I will try that. Thanks
I understand you want to do similarity search?
If so, I believe there are ways to achieve that using sqlite or postgresql extensions for vector db usage.
Though I don't know if there are dotnet wrappers you could use, maybe you'll find something useful anyway.
Alright, The keyword is "similarity" search, so I will do some research and tell you what I come up with
Good Luck :)