question about spark submit
I've been working on PoC to find a way to test spark submit with empty jar but with proper class file in CLI. I'm supposed to extract data from some other website. Is this even possible thing to do with empty jar? (empty as in nothing in the jar. All you do is and that's it)
Everywhere I look I need a jar that's functioning, that includes dependencies for class file. With empty jar. is there way to bypass all those class not found exceptions? Or is this impossible to do?
11 Replies
⌛ This post has been reserved for your question.
Hey @VaygrEmpire! Please useTIP: Narrow down your issue to simple and precise questions to maximize the chance that others will reply in here./closeor theClose Postbutton above when you're finished. Please remember to follow the help guidelines. This post will be automatically closed after 300 minutes of inactivity.
what do you need the JAR for?
an empty file isn't a valid JAR...
I'm a junior dev and seniors told me to test spark submit job specifically with empty jar...
Are you sure they meant an empty file?
they showcase me to create and proceeded to spark submit, and been communicating that I cannot proceed with empty jar but they keep saying to try again...
what is saying try again?
i need to extract latest file from some website through spark submit, that senior engineer did that in previous project and want me to test it, but i just don't understand how that's possible
current command is like following:
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
So what's the problem?
main class is there with all the spark properties defined and methods as well. Jar file is, as mentioned, empty jar file. It has nothing inside the jar.
what I learned so far is, for spark submit job to work, it needs main class to execute codes, however, if main class is importing 3rd party libraries, files like pom.xml needs to be included in jar file, or will throw Class Not Found Exception.
But I'm continuously told to "find a way to make it work" without creating jar file that includes libraries, and without adjusting existing script; just in command line level with parameters. And I have no idea how to make this work, let alone if it's even possible. been stuck for entire week
💤 Post marked as dormant
This post has been inactive for over 300 minutes, thus, it has been archived. If your question was not answered yet, feel free to re-open this post or create a new one.