VaygrEmpire
VaygrEmpire
PostsComments
JCHJava Community | Help. Code. Learn.
Created by VaygrEmpire on 1/7/2025 in #java-help
send message to slack without token
resolved by sending token from airflow dag with xcom variable to scala class, then connect to HttpPost
5 replies
JCHJava Community | Help. Code. Learn.
Created by VaygrEmpire on 6/2/2023 in #java-help
question about spark submit
what I learned so far is, for spark submit job to work, it needs main class to execute codes, however, if main class is importing 3rd party libraries, files like pom.xml needs to be included in jar file, or will throw Class Not Found Exception. But I'm continuously told to "find a way to make it work" without creating jar file that includes libraries, and without adjusting existing script; just in command line level with parameters. And I have no idea how to make this work, let alone if it's even possible. been stuck for entire week
17 replies
JCHJava Community | Help. Code. Learn.
Created by VaygrEmpire on 6/2/2023 in #java-help
question about spark submit
main class is there with all the spark properties defined and methods as well. Jar file is, as mentioned, empty jar file. It has nothing inside the jar.
17 replies
JCHJava Community | Help. Code. Learn.
Created by VaygrEmpire on 6/2/2023 in #java-help
question about spark submit
current command is like following: 
spark-submit --master yarn --jars <jar url> --class <class url> --repositories <repository url> --packages <package url>
spark-submit --master yarn --jars <jar url> --class <class url> --repositories <repository url> --packages <package url>
17 replies
JCHJava Community | Help. Code. Learn.
Created by VaygrEmpire on 6/2/2023 in #java-help
question about spark submit
i need to extract latest file from some website through spark submit, that senior engineer did that in previous project and want me to test it, but i just don't understand how that's possible
17 replies
JCHJava Community | Help. Code. Learn.
Created by VaygrEmpire on 6/2/2023 in #java-help
question about spark submit
they showcase me to create 
touch dummy.jar
touch dummy.jar
and proceeded to spark submit, and been communicating that I cannot proceed with empty jar but they keep saying to try again...
17 replies
JCHJava Community | Help. Code. Learn.
Created by VaygrEmpire on 6/2/2023 in #java-help
question about spark submit
I'm a junior dev and seniors told me to test spark submit job specifically with empty jar...
17 replies
JCHJava Community | Help. Code. Learn.
Created by VaygrEmpire on 4/3/2023 in #java-help
question about multithreading
figured
5 replies