Anyone here a data scientist? I got some questi...
Anyone here a data scientist? I got some questions on how to best learn data science so as a DE I can better support data scientists at my job.
12 Replies
Thread automatically created by ThugNasty - NastyThug in #🤔｜questions
Probably better off leaving the questions so people can answer
extended-salmon•2w ago
I’m a DA rather than a DS but the one thing that’s most difficult about dealing with data engineers is the complete lack of any expectation that they have subject matter expertise. I realize this isn’t their job but it will be very helpful to me if I can come to a DE and say, “This airline data says that the plane landed with more fuel than it took off with.” or “The data says that this plane carried 190 passengers from New York to LA but it only has 160 seats.” and have them immediately understand that this is a sign something is wrong.
For a DA (and probably a DS) it’s very frustrating when you spend a lot of time trying to diagnose why your own program gave a nonsensical result, trace it to an issue like this with the raw data, and then have the engineer not understand what you are talking about.
If I say that something doesn’t make sense from a subject matter point of view, I don’t want to hear that the spec says it’s ok and therefore it’s not the DE’s problem. The STTM may indeed say it’s supposed to be like that but an acknowledgement from the DE that I actually found something odd and my work doing so was worthwhile is valuable.
underlying-yellow•2w ago
If your job is asking you to help ds, then they're offloading some ds work to you. The ds should be responsible for gathering their own data. De should really only present it to them raw imo. The exception being if you're a de that is living on or very near a ds team
xenial-black•2w ago
The challenge with that is the DE doesn’t deal with Business processes on a day to day basis like analysts do. Personally, I always have Analysts explain me the business and how things work and take notes.
That way I can do some analysis to validate the data pipeline outputs after all the transformations.
But what I more so need help with here is how to get up to speed with data science because sometimes they ask us for stuff and idk if it’s even feasible.
Yea but if I did that I wouldn’t have this job lol
So I wanna learn just enough data science so that I can support without hindering if you know what I mean.
underlying-yellow•2w ago
Hm yea depends on what they're making you do
Ds shouldn't throw out all the requirements to de smh
I understand in the real world it's a different story tho
xenial-black•2w ago
Any recommendations on how to learn data science from bottom up? Preferably through books and projects.
Just so that I know enough to support the DS
underlying-yellow•2w ago
Hmm
When I was first learning DS, I started off with edX and datacamp.
https://www.edx.org/
https://www.datacamp.com
Datacamp is cool beacuse you learn by doing, similar to the codeacademy.com method but datacamp is dedicated to DS.
For edX, they are a good source to learn mroe of the fundamentals and core concepts of DS, like statistics. I know a lot of people get bad memories when they think of school but a course or two in statistics will take you far, whatever your level may be, whether it's 101 or Masters or PhD level statistics. A single course can take you pretty far pretty quick.
While I was learning the fundamentals there, I used kaggle to put it into practice.
https://www.kaggle.com
Yes, there are a lot of people with fake high scores on Kaggle but just ignore all of that and focus on using it for your goal of learning and improving your skills.
xenial-black•2w ago
Thank you!
rival-black•2w ago
Your passenger data seems so obvious that something is wrong. A DE still wouldn’t care?!
extended-salmon•2w ago
The problem is it’s not so obvious unless you know about airlines. Imagine if I actually said that the data indicated a 90 degree AOA. If you’ve never had a job involving airplanes you wouldn’t realize that means the plane is heading forward but has its nose pointed straight up.
My hope is that a DE working with airline data should know a little bit so that something like this would perk his ears up. If he doesn’t know, he should have a sufficiently open mind to realize when I explain what this means that the data is wrong and needs looking into.