JanusGraph on Azure HDInsights Spark Cluster
We are following the instructions here https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin and are working on setting up the hadoop-gremlin on the spark cluster with JanusGraph 1.0.0-rc2. We are working through some of the issues with this integration where we needed to copy the two binaries in the Janus lib location
/usr/hdp/4.1.14.2/hadoop/hadoop-azure-3.1.1.4.1.14.2.jar
/usr/hdp/4.1.14.2/hadoop/azure-storage-7.0.1.jar
Now the current issue we are facing is the following. Would anyone know what might be going on, am I missing some more jar references here? Any tips/help greatly appreciated, thanks!
org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating plugin: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink
at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:212) ~[hadoop-client-api-3.3.1.jar:?]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink
at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_362]
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProvider(Ljava/lang/String;Ljava/lang/String;)[C
at org.apache.hadoop.fs.azure.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:55)
at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:1018)
@boxuanli , @florianhockmann , @HadoopMarc who have been helping on various channels regarding this effort.
1 Reply
Actually, after copying all the binaries from the /usr/hdp/4.1.14.2/hadoop/ location to the Janus distro lib location, we see this error. Any tips/recommendations appreciated!
Exception in thread "main" java.lang.AbstractMethodError: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink.init(Lorg/apache/hadoop/shaded/org/apache/commons/configuration2/SubsetConfiguration;)V
at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:209)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.newSink(MetricsSystemImpl.java:535)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSinks(MetricsSystemImpl.java:507)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:483)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163)
at org.apache.hadoop.fs.azure.metrics.AzureFileSystemMetricsSystem.fileSystemStarted(AzureFileSystemMetricsSystem.java:41)