Apache TinkerPop•7mo ago

Unable to deserialize results with Gremlin-go client + JanusGraph

Hi all - I'm trying to set up a JanusGraph database and use the Gremlin-go client to run some gremlin traversals over it. I'm facing some serialization error. My docker-compose file to start a JanusGraph at localhost:8182:

version: '3'
services:
  janusgraph:
    image: janusgraph/janusgraph:1.0.0  # Use the latest JanusGraph image
    container_name: jce-janusgraph
    ports:
      - "8182:8182"  # Map the Gremlin server port to the host
    environment:
      JANUS_PROPS_TEMPLATE: cql
      JANUSGRAPH_RELATION_DELIMITER: "@"
      janusgraph.storage.hostname: jce-cassandra
      janusgraph.graph.set-vertex-id: "true"
      janusgraph.graph.allow-custom-vid-types: "true"
    networks:
      - jce-network
    restart: unless-stopped
    depends_on:
      - cassandra
  
  cassandra:
    image: cassandra:3
    container_name: jce-cassandra
    ports:
      - "9042:9042"
      - "9160:9160"
    volumes:
      - graph_data:/var/lib/cassandra
    networks:
      - jce-network

volumes:
  graph_data:  # Named volume to store JanusGraph data

networks:
  jce-network:

version: '3'
services:
  janusgraph:
    image: janusgraph/janusgraph:1.0.0  # Use the latest JanusGraph image
    container_name: jce-janusgraph
    ports:
      - "8182:8182"  # Map the Gremlin server port to the host
    environment:
      JANUS_PROPS_TEMPLATE: cql
      JANUSGRAPH_RELATION_DELIMITER: "@"
      janusgraph.storage.hostname: jce-cassandra
      janusgraph.graph.set-vertex-id: "true"
      janusgraph.graph.allow-custom-vid-types: "true"
    networks:
      - jce-network
    restart: unless-stopped
    depends_on:
      - cassandra
  
  cassandra:
    image: cassandra:3
    container_name: jce-cassandra
    ports:
      - "9042:9042"
      - "9160:9160"
    volumes:
      - graph_data:/var/lib/cassandra
    networks:
      - jce-network

volumes:
  graph_data:  # Named volume to store JanusGraph data

networks:
  jce-network:

My minimal example Golang code that is not working:

package main

import (
    "time"

    gremlingo "github.com/apache/tinkerpop/gremlin-go/driver"
)

func main() {
    endpoint := "ws://localhost:8182/gremlin"
    var remoteConnection *gremlingo.DriverRemoteConnection

    remoteConnection, err := gremlingo.NewDriverRemoteConnection(endpoint, func(settings *gremlingo.DriverRemoteConnectionSettings) {
        settings.TraversalSource = "g"
        settings.KeepAliveInterval = time.Duration(15 * time.Minute)
    })
    if err != nil {
        panic(err)
    }

    query := `g.V().limit(2)`

    result, err := remoteConnection.Submit(query)
    if err != nil {
        panic(err)
    }

    resultList, err := result.All()
    if err != nil {
        panic(err)
    }

    for _, item := range resultList {
        println(item.String())
    }

    defer remoteConnection.Close()
}

package main

import (
    "time"

    gremlingo "github.com/apache/tinkerpop/gremlin-go/driver"
)

func main() {
    endpoint := "ws://localhost:8182/gremlin"
    var remoteConnection *gremlingo.DriverRemoteConnection

    remoteConnection, err := gremlingo.NewDriverRemoteConnection(endpoint, func(settings *gremlingo.DriverRemoteConnectionSettings) {
        settings.TraversalSource = "g"
        settings.KeepAliveInterval = time.Duration(15 * time.Minute)
    })
    if err != nil {
        panic(err)
    }

    query := `g.V().limit(2)`

    result, err := remoteConnection.Submit(query)
    if err != nil {
        panic(err)
    }

    resultList, err := result.All()
    if err != nil {
        panic(err)
    }

    for _, item := range resultList {
        println(item.String())
    }

    defer remoteConnection.Close()
}

Solution:

For Python, you can also use JanusGraph-Python to get better support for JanusGraph types: https://github.com/JanusGraph/janusgraph-python (JanusGraph-Python only extends Gremlin-Python so you will still be using that. It just adds serializers for JanusGraph types)...

GitHub

GitHub - JanusGraph/janusgraph-python: JanusGraph Python Gremlin La...

JanusGraph Python Gremlin Language Variant (GLV). Contribute to JanusGraph/janusgraph-python development by creating an account on GitHub.

Jump to solution

10 Replies

zlfbenOP•7mo ago

Output (at result.All()):

panic: E0408: unknown data type to deserialize 0x0

goroutine 1 [running]:
main.main()
    /home/apr16-2024/main.go:30 +0x44d

panic: E0408: unknown data type to deserialize 0x0

goroutine 1 [running]:
main.main()
    /home/apr16-2024/main.go:30 +0x44d

Some notes: - ~~I didn't have the above issue if I used the Gremlin-go's bytecode APIs (i.e., not a string, but constructing steps within Golang)~~. - The above code also didn't have any issue when connecting to Tinkerpop or Neptune instead of JanusGraph. - The same query under the groovy console works. Ran into the same error using Gremlin-go's bytecode APIs:

package main

import (
    "time"

    gremlingo "github.com/apache/tinkerpop/gremlin-go/driver"
)

func main() {
    endpoint := "ws://localhost:8182/gremlin"
    var remoteConnection *gremlingo.DriverRemoteConnection

    remoteConnection, err := gremlingo.NewDriverRemoteConnection(endpoint, func(settings *gremlingo.DriverRemoteConnectionSettings) {
        settings.TraversalSource = "g"
        settings.KeepAliveInterval = time.Duration(15 * time.Minute)
    })
    if err != nil {
        panic(err)
    }

    g := gremlingo.Traversal_().WithRemote(remoteConnection)

    resultList, err := g.V().Limit(2).ToList()
    if err != nil {
        panic(err)
    }

    for _, item := range resultList {
        println(item.String())
    }

    defer remoteConnection.Close()
}

package main

import (
    "time"

    gremlingo "github.com/apache/tinkerpop/gremlin-go/driver"
)

func main() {
    endpoint := "ws://localhost:8182/gremlin"
    var remoteConnection *gremlingo.DriverRemoteConnection

    remoteConnection, err := gremlingo.NewDriverRemoteConnection(endpoint, func(settings *gremlingo.DriverRemoteConnectionSettings) {
        settings.TraversalSource = "g"
        settings.KeepAliveInterval = time.Duration(15 * time.Minute)
    })
    if err != nil {
        panic(err)
    }

    g := gremlingo.Traversal_().WithRemote(remoteConnection)

    resultList, err := g.V().Limit(2).ToList()
    if err != nil {
        panic(err)
    }

    for _, item := range resultList {
        println(item.String())
    }

    defer remoteConnection.Close()
}

I noticed in gremlin_python, I faced the same issue, but I was able to fix it by setting message_serializer=serializer.GraphSONMessageSerializer() in the DriverRemoteConnection to fix it. The issue seems to come from the default binary serializer after reading some Github Issues and the general channel. I suspect the issue is with the serializer my Golang code uses (the binary one), but I have no idea how it could be set to another serializer (Graphson) like gremlin_python does. Does anyone have some clue how to change the serializer in gremlin-go? Thanks!

spmallette•7mo ago

can someone who knows @janusgraph help on this one?

Yang Xia•7mo ago

One note on gremlin-go, it only supports the graphBinary serialization format, so there is no way to change the serializer. Not sure if folks from JanusGraph know any workaround?

spmallette•7mo ago

really need to get rid of the need for custom serializers in the future. it's too much burden on providers to have to build them and too complicated for users to figure them out. providers should make sure that their ids can be coerced to/from string or otherwise operate with the set of datatypes TinkerPop allows.

Bo•7mo ago

@zlfben As a workaround - could you try JanusGraph 0.6.3?

providers should make sure that their ids can be coerced to/from string

That will be a huge breaking change 😦

zlfbenOP•7mo ago

Thanks. Just tried JanusGraph 0.6.3. With out of box configuration it worked well with gremlin-go's graphBinary serialization. However, it doesn't support my old configuration (shown below). After setting these for the JanusGraph, it seemed to have a lot of issues setting any types of vertex ids (long, string, etc.). I do need to use UUID as my vertex IDs - is there any way to configure JanusGraph 0.6.3 to support that, or is it only available in 1.0 or later?

      JANUSGRAPH_RELATION_DELIMITER: "@"
      janusgraph.graph.set-vertex-id: "true"
      janusgraph.graph.allow-custom-vid-types: "true"

      JANUSGRAPH_RELATION_DELIMITER: "@"
      janusgraph.graph.set-vertex-id: "true"
      janusgraph.graph.allow-custom-vid-types: "true"

Bo•7mo ago

I do need to use UUID as my vertex IDs - is there any way to configure JanusGraph 0.6.3 to support that, or is it only available in 1.0 or later

That's only supported in 1.0 😦

zlfbenOP•7mo ago

Thanks for the replies! In the end I just built a small gremlin broker using gremlin_python + Graphson serializer and called the broker in my Golang code. Hopefully future updates of gremlin-go or JanusGraph can solve this issue.🤞

Solution

Florian Hockmann•7mo ago

GitHub

GitHub - JanusGraph/janusgraph-python: JanusGraph Python Gremlin La...

JanusGraph Python Gremlin Language Variant (GLV). Contribute to JanusGraph/janusgraph-python development by creating an account on GitHub.

Florian Hockmann•7mo ago

And if you want to continue Go, then maybe you are interested in implementing basic support for JanusGraph types yourself there? In that case you could check out the GraphBinary support for JanusGraph in JanusGraph.Net: https://github.com/JanusGraph/janusgraph-dotnet/tree/master/src/JanusGraph.Net/IO/GraphBinary to get an idea of what needs to be implemented. Most types are even optional, e.g., you don't need support for Geoshapes / Point if you aren't using those

GitHub

janusgraph-dotnet/src/JanusGraph.Net/IO/GraphBinary at master · Jan...

JanusGraph .NET Gremlin Language Variant (GLV). Contribute to JanusGraph/janusgraph-dotnet development by creating an account on GitHub.

Gaming

Programming

Unable to deserialize results with Gremlin-go client + JanusGraph