Phantom Unique Data / Data Too Large?
I've been chasing down a weird writing bug for our graph writer, but I feel like I've stumbled into conflicting realities in my graph and wanted to share in case I'm missing something larger. For context the environment in play is my local development environment, all running inside docker compose (host machine is MacOS, on the off chance it matters).
- JanusGraph 1.0.0
- Cassandra 4
- Elasticsearch 8
I'm writing vertices into a graph. I've got a property that is enforced uniqueness via a composite index with
unique() enabled. I don't have locking turned on, I know that's documented (https://docs.janusgraph.org/schema/index-management/index-performance/#index-uniqueness) but as a single local Cassandra I was thinking that wouldn't be required since there's no other instances to sync with, maybe that's the root of the whole issue.
The vertices also use custom vertex ids derived off this property. My thought was this way knowing the property I could go ahead and issue traversals against the vertex id instead of having to detour to an index, but still leverage the index when it was more convenient.
Trying to rerun some data for processing I'm having a violation of the aforementioned composite unique index, but I'm puzzled as to why.
Using gDotV I'm able to determine the following:
1. Querying based on the property that should use the unique index returns the vertex as expected with the expected custom id. However the properties off the vertex are empty like so when shown in gDotV: "properties": [],
2. Running just a query for the custom id as in g.V('id_goes_here') paradoxically seems to not return the vertex.
So this leaves me in a weird state that seems to be implying:
- The properties of the vertex are empty, but the unique index is still populated with the property, even when the vertex itself is not
- I'm not able to find the vertex with its vertex id8 Replies
This does ultimately put my writer in an effectively stuck state since it tries to do its write based off the vertex id as an upsert style mutation.
First it does a lookup based on the vertex id, but fails to find it, so it then tries to create a vertex (with that same id) and with the same property value, but that is then rejected because it's breaking the apparently unique constraint on a vertex that doesn't seem to otherwise exist.
I think I may have uncovered what caused this weird partial write state.
I modified my write to try the vertex id and then fallback on lookup using the property, perfore attempting to create the vertex. Caught this in the Cassandra log now that I was no longer failing to the unique constraint:
So looks like the initial write failed and the ensuing retries likewise don't have a way to proceed due to the weird phantom partial write
Did a temporary work around removing a different field that was too large and likely causing the write to fail.
Though it looks like even with this temporary work around this vertex seems to be completely stuck, I can't assign the ID to the vertex once it's been created so this vertex seemingly is not able to be healed and be retrievable via its intended custom id ever again
I am a little perplexed by this event however. What I am writing is a large string to leverage the full text search provided by Elasticsearch over the property. But I have test cases that write a string that's over 2x of this one in size in my CI, so this is weirdly a size limit I'm hitting below what I've consisistently successfully written.
It's like this one didn't get divided in a manner that if it were a little larger it would have.
Here's the stack trace for the write from JG's perspective
It looks like it's attempting to run mutliple mutations, I am starting to wonder if one of them didn't get split quite right, but getting lost in the weeds of where the mutation preparation happens since this stacktrace is all about queueing it to be written
@Oleksandr Porunov hope it's okay I pinged you. Git blame had your name around lots of the cql backend code I was sifting through. Thought maybe you'd be the one to ask if this is even in the right neighborhood.
It's not exact match of the log message but I think this doc is referring to the same issue: https://support.datastax.com/s/article/Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y
The official recommendation is to break the batch into smaller chunks or make the batch asynchronous and do single-row writes.On the surface it seems like maybe somewhere around the code like here (https://github.com/JanusGraph/janusgraph/blob/2dba4307d492e288b12420a99a087f31473bc58b/janusgraph-cql/src/main/java/org/janusgraph/diskstorage/cql/function/mutate/AbstractCQLMutateManyFunction.java#L53-L54) where CQL statements are being populated for the batch that members of the batch should be split up depending on the size of the value being submitted? But it doesn't seem to have an explanation for why I have confirmed success for writing over 2x this particular property's size (around 36MB) in my CI environment (where one of my test cases is about 80MB for this property). Seems like there should be some kind of column single write splitting already in play, and this particular case just got an unlucky split.
GitHub
janusgraph/janusgraph-cql/src/main/java/org/janusgraph/diskstorage/...
JanusGraph: an open-source, distributed graph database - JanusGraph/janusgraph
Hey @criminosis . Sorry I'm confused regarding what issues you are referencing to right now because it seems there are multiple problems you are facing.
I will reply regarding your last message, so that we can focus on a specific problem.
You can write data into Cassandra in 2 modes (with atomic transaction enabled or disabled). If you disable atomic transaction then your transaction may be split to multiple CQL batch requests of the configured size. Meaning that part of you data could be mutated and another part of your data may fail. With atomic transactions you won't need to deal with such issues, however it also means that all the mutations of your transaction should fit into a single batch request. In that case, all your mutations (no matter what batch size you configure) are going to be batched together which in some cases may result in very big batch requests. However, different storage backends (Cassandra, ScyllaDB, AstraDB, Amazon Keyspaces, etc.) may have their own batch size limitations. In some cases you can configure you servers to accept larger batch requests (i.e. in your own managed Cassandra or ScyllaDB), however in other cases you need to reach out to customer support team so that they can increase batch size limitation (i.e. AstraDB or Amazon Keyspaces managed clusters).
The reason why it could potentially fail in one environment but work in another is that this limit is applied to multi-partition statements. Meaning, in some situations your batch request may still be bigger while it is touching a single partition (a single vertex without index mutation). When you have a composite index mutation it already means that you are affecting multiple partitions (at least a sinlge partition for a vertex and a single partition for the index record).
Please, refer to
batch_size_fail_threshold_in_kb in cassandra.yaml configuration for more information.Yeah after trying to see if I could split apart the mutation in JG as a possible other PR I opted to just increase the cassandra setting mentioned in the docs I linked to and moved on.
It really puzzled me because I have a test case that's able to write a known 80MiB value successfully, and could query for the string length using gremlin to confirm. But then other smaller values, larger than 16 MiB would fail.
It's not the solution the cassandra docs say you should do, but I had to move on to other issues