Need help with opening and reading a 122 GB JSON file
Dose anyone know a website or app or anything to open a json file and also read it without it crashing?
66 Replies
So here's what you can do
first to get a basic idea you can open your terminal
but uh....are you on windows, mac or linux?
in a terminal you could peek at the first few lines of a file by doing
cat myfile.json
and the end of the file like tail myfile.json
I'm on windows
no idea, I think powershell supports those commands? or WSL if you're using that
Time for google time 😄
this can show you the main keys in your data and you can go from there
anyway you can see the basic idea of how you open files and read it
so you can take peeks at the data and decide what to do with it, based on it's contents....? idk your assignment is weird
yeah 😦
I mean it's not uncommon as a data scientist to have files that you don't even know what they contain or how they are structured... but idk if that's what they're teaching you
So basically were helping to create this app that reads information about the json but we can't open it or read it, but the client dosen't know what a json is since his just a messenger, but we can't ask the company cause the messenger is a one way. This is confusing XD
oh fun
well anyway, take the commands I told you about or the script I sent and work from there, hopefully the key names are descriptive of what the file contains and if not you at least know how to access the data on it and go from there
Cool cheers, time to see what is in here XD
It just returned 3000. Oh boy
3k keys? oof
rip MEEEEEEEEEEEEEEEEEEEEEE
yeah kinda expected a lot of them
Is there a website or video on how to look in the json file on what to do next? I looked into json files before but not this way without the key names XD
well depends
you have a list of the keys now Object.keys() returns an array so you could loop through that array and peek each of those keys
I just notice I was running the wrong file XD
it's running my typescript for some reason, I'm fixing it now tho
Oh... thats a problem
I'm just gonna gather my team and just be like "Yeah um.... file is too big to open"
ok we need to change how to open the file then
oh?
this usually opens the entire file and puts it in a buffer
but I guess node doesn't want 122GB in memory...
who would've thought
when something is 122GB but it's not a game XD
so instead you need to open it as streams, not very familiar with big files and node js completely slipped my mind
oh ok, how do I open it while it streams? I think it try to open it up on a browser to see what was inside. MY PC crashed XD
npm
event-stream
construct pipes of streams of events. Latest version: 4.0.1, last published: 4 years ago. Start using event-stream in your project by running
npm i event-stream
. There are 1945 other projects in the npm registry using event-stream.Not too terribly familiar with how Node works with file streams, but I do know that's how to do it :p
it's a bit of a pain
you can use the npm package I sent to have a nice render of the data
if you want to do it yourself however
https://itnext.io/using-node-js-to-read-really-really-large-files-pt-1-d2057fe76b33
here's something that can help
Actually quesiton, is this common to have a huge file like this? Cause with what the client has given us, is it even like realistic? XD
More than several MB of a JSON is just bad form
Like, what is this supposed to be, anyway?
I'll have to like talk to the teacher, cause if this is unrealistic then I don't think this should even be a thing for an exam XD
Even if it's an array of JSON objects that's a fuckton of info
Oh yeah, heres the image of the file size XD
Thanks for the information, I'll still learn about the stream to learn that incase I need it. But for now, me and my team are going to get out of it XD.
See if you can find a way to read the first few lines of the file
is this common to have a huge file like this?in webdev? I'd fire you other areas of dev? sometimes
O.O
again, that's what data science is, there this would be common
tho huge JSON files would not be that common that would be possible
That way we can see how the file is organized. I'm hoping that it's not one JSON object and they just appended object after object. If that's the case you can read the file line by line
yeah like I said, powershell probably supports the
cat
command
but not sureso I tried cat. ummmmmmm
Can you do this
head -n 5 file.json
, show the output?
This is to get the first 5 lines
Hopefully it'll show us how the file is formattedforget the format look at that data
I already hate it
YOU CAN'T MAKE ME
Damn, no
head
commandsad no head
Can you copy the first little bit of what
cat
showed you?Um I can't really do that. When I press enter it ZOOMS down and when I scroll up even after 1 sec of pausing, I can't scroll back to the top XD
What I'm hoping to discern is if it's one huge JSON object or if the file is several objects either in an array or just one object, newline, second object, etc
I can try again tho
Well, have a look at the end of the file then
See if you can tell if it's one large object or not
HAHA auto clicker wins
Damn, just one gigantic object
For the first time I can finally read the file. ðŸ˜
not gonna lie, that object is bigger than even my ego
Well, at least you can read it now :p
But seriously, how does one expect you to parse a file that large?
usually? not with javascript lol
try the streams tho, particularly with even-stream
it'll be a pain but get a new assignment
It is what it is
Well, thankyou all for teaching me powercell with cat I like how to read json another way. Um... I think I'm gonna see if there's a "smaller" version or another exam that our team can do cause now I don't even know of our teacher knows the limits of javascript XD
It is a while back and i've only used it once. But this helped me out in a simular situation.
https://dadroit.com/
Dadroit JSON Viewer
View, find, and export within BIG JSON data in a blink. Open 1GB JSON in seconds. Find with RegEX and JSONPath. Export JSON to XML, minified and formatted.