Apache TinkerPop•3mo ago

Best practices for local development with Neptune.

I would like to use a local Gremlin server with TinkerGraph for local development, and then deploy changes to Neptune later. However, there are several differences between TinkerGraph and Neptune that impact the portability of the code. The most important one is probably the fact that in Tinkergraph vertex and edge ids are numeric, but they are strings in Neptune. Also, I think there are some differences in how properties are handled if the cardinality is a list. What is the recommended workflow to minimize discrepancies between my local environment and Neptune?

Solution:

There's a blog post here that contains some of the details on what properties you can change in TinkerGraph to get close: https://aws.amazon.com/blogs/database/automated-testing-of-amazon-neptune-data-access-with-apache-tinkerpop-gremlin/ It's unlikely that you'll find anything that emulates things like the result cache, lookup cache, full-text-search, features, etc.
I would be curious to hear what the needs are for local dev....

Amazon Web Services

Automated testing of Amazon Neptune data access with Apache TinkerP...

Amazon Neptune, a fully managed graph database, is purpose built to work with highly connected data such as relationships between customers and products, or between pieces of equipment within a complex industrial plant. Neptune is designed to support highly concurrent online transaction processing (OLTP) over graph data models. Neptune supports ...

Jump to solution

7 Replies

gdotv•3mo ago

There's https://docs.localstack.cloud/user-guide/aws/neptune/ if you're looking for a quick solution that emulates the Neptune querying API.

Docs

Neptune

Get started with Neptune on LocalStack

MaxOP•3mo ago

Thanks. Does it support Neptune specific apis like g.with('Neptune#enableResultCache', true)? By the way I figured out that I can start gremlin server container with modified TinkerGraph properties:

gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
gremlin.tinkergraph.vertexIdManager=ANY
gremlin.tinkergraph.edgeIdManager=ANY

gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
gremlin.tinkergraph.vertexIdManager=ANY
gremlin.tinkergraph.edgeIdManager=ANY

This allows to have ids as strings, which solves one of the major pain points.

Solution

triggan•3mo ago

Amazon Web Services

Automated testing of Amazon Neptune data access with Apache TinkerP...

MaxOP•3mo ago

Could you possibly clarify why the example code for Neptune includes a different gremlin-server.yaml configuration compared to the default gremlin server? I understand that it needs to reference a distinct tinkergraph.properties file (to allow IDs as strings), but are there any additional configuration changes that justify the divergence from the default? Neptune sample code: https://github.com/aws-samples/automated-testing-graph-queries/blob/main/test/db/conf/gremlin-server.yaml Default config: https://github.com/apache/tinkerpop/blob/master/gremlin-server/conf/gremlin-server.yaml

GitHub

automated-testing-graph-queries/test/db/conf/gremlin-server.yaml at...

This repository contains code that provides an approach to the automated testing of Gremlin queries that are intended to run against an Amazon Neptune database - aws-samples/automated-testing-graph...

GitHub

tinkerpop/gremlin-server/conf/gremlin-server.yaml at master · apach...

Apache TinkerPop - a graph computing framework. Contribute to apache/tinkerpop development by creating an account on GitHub.

triggan•3mo ago

Many of the differences there are a little opinionated from the person that wrote that blog post. When I create a Gremlin Server docker container to emulate Neptune (as close as possible), I typically just use something like the following (Dockerfile) and comments explain the changes:

FROM tinkerpop/gremlin-server:latest

# Add support for both websockets and http requests
RUN sed -i "s|^channelizer:.*|channelizer: org.apache.tinkerpop.gremlin.server.channel.WsAndHttpChannelizer|" ./conf/gremlin-server.yaml

# Allow for string based IDs
RUN sed -i "s|^gremlin.tinkergraph.vertexIdManager=.*|gremlin.tinkergraph.vertexIdManager=ANY|" ./conf/tinkergraph-empty.properties

# Remove ReferenceElementStrategy for use with graph-explorer - return all properties
RUN sed -i "s|^globals << \[g.*|globals << [g : traversal().withEmbedded(graph).withStrategies()]|" ./scripts/empty-sample.groovy

# Increase thread stack to 2m
ENV JAVA_OPTIONS="-Xss2m -Xms512m -Xmx4096m"

FROM tinkerpop/gremlin-server:latest

# Add support for both websockets and http requests
RUN sed -i "s|^channelizer:.*|channelizer: org.apache.tinkerpop.gremlin.server.channel.WsAndHttpChannelizer|" ./conf/gremlin-server.yaml

# Allow for string based IDs
RUN sed -i "s|^gremlin.tinkergraph.vertexIdManager=.*|gremlin.tinkergraph.vertexIdManager=ANY|" ./conf/tinkergraph-empty.properties

# Remove ReferenceElementStrategy for use with graph-explorer - return all properties
RUN sed -i "s|^globals << \[g.*|globals << [g : traversal().withEmbedded(graph).withStrategies()]|" ./scripts/empty-sample.groovy

# Increase thread stack to 2m
ENV JAVA_OPTIONS="-Xss2m -Xms512m -Xmx4096m"

Daniel Weber•3mo ago

In an upcoming version of GremlinServer, you should even be able to use "STRING" instead of "ANY" for an idManager, so that whenever you don't provide an id during element creation, the autogenerated id will also be of type string (instead of UUID).

ColeGreer•3mo ago

To add to Daniel's comment, the String ID manager will be included in the 3.7.3 release which is expected to be published by the end of October.

Gaming

Programming

Best practices for local development with Neptune.