Apache TinkerPop•11mo ago

AWS Neptune updating gremlin driver to 3.6.2 introduced many bugs to working queries

After updating Amazon Neptune engine version from 1.2.0.2 to 1.2.1.0 and the Gremlin.Net (C# nuget) driver from 3.5.2 to 3.6.2, suddenly queries started throwing exceptions, specifically exceptions about serialization errors. To pinpoint the cause, I downgraded the Gremlin.Net driver to 3.5.2 while leaving the engine version updated to 1.2.1.0, the queries started working like before. The problem is, according to the AWS documentation, the minimum required Gremlin.net version is 3.6.2 Will there be a problem keeping the Gremlin.net driver version to 3.5.2? Will there be any side effects? Resources: Aws documentation: https://docs.aws.amazon.com/neptune/latest/userguide/engine-releases-1.2.1.0.html#engine-releases-1.2.1.0-query-versions

12 Replies

spmallette•11mo ago

The preference would be to go with the recommended version of 3.6.2+ on 1.2.1.0. do you have other examples of the types of errors you are getting? or are they all like that one?

HaieketounOP•11mo ago

Thank you for your answer @spmallette, as of my last knowledge, all error types are like this. Specifically, these errors occurring when my query contains Enumerations and DateTime. A solution to this is converting these objects to strings with the ToString() function.

spmallette•11mo ago

strange. i'm not aware of any changes to DateTime serialization that would cause a problem here with an upgrade. could you share an example of a failing query?

Florian Hockmann•11mo ago

I guess you don't explicitly configure the serializer in Gremlin.Net? Because then you're getting the default serializer which was GraphSON before 3.6.1 and GraphBinary after that. That probably explains the changed behavior. You can now simply configure Gremlin.Net to continue using GraphSON instead of GraphBinary or you could use DateTimeOffset instead of DateTime as we have a serializer in GraphBinary for DateTimeOffset I guess we should also add one for DateTime, but the default is DateTimeOffset as that's the recommended data type for dates

HaieketounOP•11mo ago

C#
 return DoWithRetry(async (conn, _) =>
        {
            var g = conn.TraversalSource;
            var eightStagePostIds = await g.V()
                .HasLabel(EightLabel.CreatorsContent)
                .Has(GraphProperty.IsEightStage, true)
                .Has(GraphProperty.PublishedDate, P.Lte(DateTime.UtcNow)) //This breaks 3.6.2 driver
                .Where(__
                    .Not(__
                        .In(EightEdge.CreatedContent)
                        .Both(EightEdge.Blocks)
                        .HasId(citizenId.ToString()))
                    .Not(__
                        .In(EightEdge.CreatedContent)
                        .In(EightEdge.Mutes)
                        .HasId(citizenId.ToString()))
                    .Not(__
                        .Has(GraphProperty.ModerationDecision, 
                            P.Within(ModerationDecisions.Rejected))))
                .Order().By(GraphProperty.PublishedDate, Order.Desc)
                .Limit<Vertex>(limit)
                .ProjectBy(new Dictionary<string, ITraversal>
                {
                    { "id",  __.Id() }
                })
                .Promise(x => x.ToList());
            
            return eightStagePostIds?.Select(x => 
                           x["id"].ToString().ToGuid())
                       .ToReadOnlyList()
                   ?? new List<Guid>();
        }, cancellationToken);

C#
 return DoWithRetry(async (conn, _) =>
        {
            var g = conn.TraversalSource;
            var eightStagePostIds = await g.V()
                .HasLabel(EightLabel.CreatorsContent)
                .Has(GraphProperty.IsEightStage, true)
                .Has(GraphProperty.PublishedDate, P.Lte(DateTime.UtcNow)) //This breaks 3.6.2 driver
                .Where(__
                    .Not(__
                        .In(EightEdge.CreatedContent)
                        .Both(EightEdge.Blocks)
                        .HasId(citizenId.ToString()))
                    .Not(__
                        .In(EightEdge.CreatedContent)
                        .In(EightEdge.Mutes)
                        .HasId(citizenId.ToString()))
                    .Not(__
                        .Has(GraphProperty.ModerationDecision, 
                            P.Within(ModerationDecisions.Rejected))))
                .Order().By(GraphProperty.PublishedDate, Order.Desc)
                .Limit<Vertex>(limit)
                .ProjectBy(new Dictionary<string, ITraversal>
                {
                    { "id",  __.Id() }
                })
                .Promise(x => x.ToList());
            
            return eightStagePostIds?.Select(x => 
                           x["id"].ToString().ToGuid())
                       .ToReadOnlyList()
                   ?? new List<Guid>();
        }, cancellationToken);

spmallette•11mo ago

i think @Florian Hockmann is on the right track with his reply @Valentyn Kahamlyk i know you are tracking some differences among the drivers/serializers/types.......here's another 😦

HaieketounOP•11mo ago

Thank you very much for your answer @Florian Hockmann, I will change to GraphSON serializer and get back to you with any updates.

Valentyn Kahamlyk•11mo ago

in .net GLV both Date and Timestamp GraphBinary types mapped to DateTimeOffset, I don't know why it's done this way. DateAdd/DateDiff/asDate steps also works only with DateTimeOffset

HaieketounOP•11mo ago

Hello guys, indeed changing to GraphSON serializer fixed the issue, thank you all kindly for your help

Florian Hockmann•11mo ago

in .net GLV both Date and Timestamp GraphBinary types mapped to DateTimeOffset, I don't know why it's done this way.

What else do you propose in .NET? The IO docs even state that both types are the same:

Timestamp Format: The same as Date.

https://tinkerpop.apache.org/docs/current/dev/io/#_timestamp_3

Valentyn Kahamlyk•11mo ago

For Java Timestamp is A thin wrapper that allows the JDBC API to identify this as an SQL TIMESTAMP value GraphBinary type | Java type | .net type DATE(0X04) | java.util.Date | DateTime TIMESTAMP(0X05) | java.sql.Timestamp extends java.util.Date | DateTime OFFSETDATETIME(0X88) | java.time.OffsetDateTime | DateTimeOffset Looks like we need cross-GLV tests to verify all ~12 date related types handled in same way. Or keep only 1-2 most used types.

spmallette•11mo ago

Or keep only 1-2 most used types.

must get rid of some types going forward

Gaming

Programming

AWS Neptune updating gremlin driver to 3.6.2 introduced many bugs to working queries