question about multithreading
Hello. I'm using multithreading with spark and have a question. I'm using Scala but shouldn't matter.
above is simple multithreading from geeksforgeeks.org. I just changed n to 3 because that's what i'm using in Scala and put custom function in run() as an example. I was working with spark and saw unexpected behavior.
I have a function called Foo that takes in dataframe and return dataframe. Inside Foo, modifications and optimizations are applied to dataframe.
when I run the code I thought I would be assigning spark job to each thread, therefore I would be able to run 3 jobs that require dataframes at the same time. But I just tried to print something inside that Foo and it printed out 7 times. I was expecting 3 times. What's causing it to print out 7 times rather than 3 times? Am I misunderstanding something about multithreading?
3 Replies
⌛
This post has been reserved for your question.
Hey @VaygrEmpire! Please useTIP: Narrow down your issue to simple and precise questions to maximize the chance that others will reply in here./close
or theClose Post
button above when you're finished. Please remember to follow the help guidelines. This post will be automatically closed after 300 minutes of inactivity.
figured
Post Closed
This post has been closed by <@205489120984825857> for the following reason:
figured out myself