JanusGraph•2y ago

Plans for the Hadoop integration

Notice https://docs.janusgraph.org/advanced-topics/hadoop/ from JanusGraph documentation about JanusGraph and Hadoop integration, would like to know where we could get more information on the project plan/status. Thanks,

4 Replies

Florian HockmannOP•2y ago

This was originally asked by @chiujl in our previous questions chat: https://discord.com/channels/981533699378135051/982256175196295180/1110320366368346203 Hi @chiujl, the story with JanusGraph's Hadoop integration is that JanusGraph mostly relies on TinkerPop's Hadoop-Gremlin to execute Gremlin traversals or so called Vertex Programs on Spark. JanusGraph mostly adds the functionality to read data from your storage backend, like Cassandra or HBase, to make it available for Hadoop/Spark. So, what you can do with that is mostly described in the TinkerPop reference docs: Hadoop-Gremlin: https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin GraphComputer: https://tinkerpop.apache.org/docs/current/reference/#graphcomputer

paritoshgote•2y ago

@HadoopMarc , @florianhockmann , @boxuanli : We are running into an issue while launching the gremlin.sh with hadoop (ref: https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin) using janus. Would you folks have any tips for this issue? Error with Janus 0.6.3: 20:00:36 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: wasbs I tried using the latest janus 1.0.0-rc2 as well, and that gave a different error Error with Janus 1.0.0-rc2: 21:32:16 WARN org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found

sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ bin/gremlin.sh

plugin activated: tinkerpop.tinkergraph
21:32:16 INFO  org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.getCustomizers - HADOOP_GREMLIN_LIBS is set to: /home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib
21:32:16 WARN  org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
        at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:125)

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2665)
        ... 67 more

sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ bin/gremlin.sh

plugin activated: tinkerpop.tinkergraph
21:32:16 INFO  org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.getCustomizers - HADOOP_GREMLIN_LIBS is set to: /home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib
21:32:16 WARN  org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
        at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:125)

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2665)
        ... 67 more

Florian HockmannOP•2y ago

@paritoshgote Could you please create a dedicated post for your question? Having only one question per post makes it much easier to follow

paritoshgote•2y ago

@florianhockmann : Created a separate post here https://discord.com/channels/981533699378135051/1113947036664995880, thanks 🙂

Discord

Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

Gaming

Programming

Plans for the Hadoop integration