J
JanusGraph•2mo ago
HailDevil

CQL query timeout issues

I am trying to create an edge between two vertices, where one of the vertex contains more than 450K edges. But everytime the operation fails with Query Timeout error as shown below: 
Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT12S
    at com.datastax.oss.driver.internal.core.cql.CqlRequestHandler.lambda$scheduleTimeout$1(CqlRequestHandler.java:207)
    at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
    at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
    at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
    at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
    at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT12S
    at com.datastax.oss.driver.internal.core.cql.CqlRequestHandler.lambda$scheduleTimeout$1(CqlRequestHandler.java:207)
    at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
    at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
    at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
    at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
    at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
We went through the documentation and found that we can increase the threshold by increasing the default value for "storage.cql.request-timeout". But what if my edge count exceeds even beyond 800K?, I cannot keep increasing the threshold forever. My understanding is, as it is trying to create a lock on the edges to create another edge, is there any other better way to do it? that is, something like lazy loading where the graph do not query the entire edges and just create the new one instead.
3 Replies
HailDevil
HailDevilOP•2mo ago
@Bo @Oleksandr Porunov @Florian Hockmann @Natalia Your help will be appreciated here 🙂
porunov
porunov•2mo ago
I don't know your graph configuration, tx configuration, and the queries performed, but I assume it could be related to the Super-Node problem which Boxuan was working here: https://github.com/JanusGraph/janusgraph/pull/4420
GitHub
Super Node Series One: Introduce proxy node type by li-boxuan · Pul...
This is a first attempt towards solving a famous problem in graph databases world - super node. A brief introduction to super node problem, and the status quo of JanusGraph, is documented at #2717....
Natalia
Natalia•2mo ago
Hi @HailDevil ! I think having more context might be helpful. Are you using JanusGraphs APIs directly to operate with the edges or you are using Gremlin sever? From my experience, we have some usecases in PROD with 1 million edges between verticies and we don't observe timeouts. Normally, during edge addition, JanusGraph doesn't read all edges back. Maybe worth checking your Cassandra consistency level, if it cross DC QUORUM, that might be expensive to get a lock when edge is being added. To experiment and check if this is due to lock, you can turn this configuration on: storage.cql.use-external-locking=true if this helps with insertions, might be worth looking into that more. Also, if the latencies come from reading back the edges, sometimes building a vertex centric index helps: https://docs.janusgraph.org/schema/index-management/index-performance/

Did you find this page helpful?