Andys1814
ATApache TinkerPop
•Created by Andys1814 on 8/26/2024 in #questions
Very slow regex query (AWS Neptune)
@Dave Thank you for the information! We have seen that OpenSearch is the preferred strategy, but this would take a while for us to implement. Do you have any shorter term recommendations to improve this lookup before we can do OpenSearch?
6 replies
ATApache TinkerPop
•Created by Andys1814 on 11/7/2023 in #questions
Sequential IDs in Neptune?
This is a great idea. We're not really worried if there's some gaps. The range idea definitely mitigates the negative affects of when collisions do hit and retries are necessary.
16 replies
ATApache TinkerPop
•Created by Andys1814 on 11/7/2023 in #questions
Sequential IDs in Neptune?
This is a good note. 16 characters might be a little more than I would hope for, but maybe I can find a balance between the characters count, vs. a practically low chance at a collision. Of course I'd still like to do uniqueness validation before saving to the database, but luckily we're still going to be using UUID for the actual primary ID, so it's not the absolute end of the world if we get a very rare collision
16 replies
ATApache TinkerPop
•Created by Andys1814 on 11/7/2023 in #questions
Sequential IDs in Neptune?
I have considered this, because I think it'd be great to have the built-in uniqueness constraint, but I'm a little worried that vertexes with conflicting IDs will cause us to miss certain data being ingested. We would definitely need to handle whatever exception Neptune throws when attempting to add vertexes with non-unique IDs
16 replies
ATApache TinkerPop
•Created by Andys1814 on 11/7/2023 in #questions
Sequential IDs in Neptune?
We're not replacing the UUID convention that Neptune uses at all. The UUID will still be the primary identifier for vertices and edges. We are simply adding an additional property called
vertexNumber. The purpose of sequential valued properties is for user experience and human readability. The best example I can give you on what we're trying to achieve is how ServiceNow does it: https://i.imgur.com/zYOFnE5.png . However, they still use a UUID on the backend, probably for security and scaling purposes: https://i.imgur.com/tXNBNiT.png. This is exactly what we're going for. UUIDs are a nice easy solution but we need a way for our users to identify something without us needing to display a 32-character string of random non-human readable junk on the frontend.16 replies
ATApache TinkerPop
•Created by Andys1814 on 11/7/2023 in #questions
Sequential IDs in Neptune?
Thanks for the comment. And yes, the performance of
max() was absolutely a concern for us, which is why we're putting that query behind a Redis cache. In an ideal situation, the DB won't need to be hit very often as the Redis cache should have the most up-to-date sequential ID.16 replies
ATApache TinkerPop
•Created by Andys1814 on 11/7/2023 in #questions
Sequential IDs in Neptune?
And yes, we would absolutely have concurrent threads running the provided example. Like I said we have a distributed architechture in which tens, or hundreds of vertexes can be ingested per second.
16 replies
ATApache TinkerPop
•Created by Andys1814 on 11/7/2023 in #questions
Sequential IDs in Neptune?
Thanks for the information. I've read all the Neptune docs and didn't feel like it provided a lot of clarity as to whether or not my use case would work.
16 replies