Need help with opening and reading a 122 GB JSON file

Dose anyone know a website or app or anything to open a json file and also read it without it crashing?
66 Replies
ErickO
ErickO•3y ago
So here's what you can do first to get a basic idea you can open your terminal but uh....are you on windows, mac or linux? in a terminal you could peek at the first few lines of a file by doing cat myfile.json and the end of the file like tail myfile.json
kingtigerknight
kingtigerknightOP•3y ago
I'm on windows
ErickO
ErickO•3y ago
no idea, I think powershell supports those commands? or WSL if you're using that
kingtigerknight
kingtigerknightOP•3y ago
Time for google time 😄
ErickO
ErickO•3y ago
const fs = require("fs")

fs.readFile("./myfile.json", "utf8", (err,data) => {
if(err) {
console.error(err)
} else {
const content = JSON.parse(data)
console.log(Object.keys(content))
}
})
const fs = require("fs")

fs.readFile("./myfile.json", "utf8", (err,data) => {
if(err) {
console.error(err)
} else {
const content = JSON.parse(data)
console.log(Object.keys(content))
}
})
this can show you the main keys in your data and you can go from there anyway you can see the basic idea of how you open files and read it so you can take peeks at the data and decide what to do with it, based on it's contents....? idk your assignment is weird
kingtigerknight
kingtigerknightOP•3y ago
yeah 😦
ErickO
ErickO•3y ago
I mean it's not uncommon as a data scientist to have files that you don't even know what they contain or how they are structured... but idk if that's what they're teaching you
kingtigerknight
kingtigerknightOP•3y ago
So basically were helping to create this app that reads information about the json but we can't open it or read it, but the client dosen't know what a json is since his just a messenger, but we can't ask the company cause the messenger is a one way. This is confusing XD
ErickO
ErickO•3y ago
oh fun well anyway, take the commands I told you about or the script I sent and work from there, hopefully the key names are descriptive of what the file contains and if not you at least know how to access the data on it and go from there
kingtigerknight
kingtigerknightOP•3y ago
Cool cheers, time to see what is in here XD It just returned 3000. Oh boy
ErickO
ErickO•3y ago
3k keys? oof
kingtigerknight
kingtigerknightOP•3y ago
rip MEEEEEEEEEEEEEEEEEEEEEE
ErickO
ErickO•3y ago
yeah kinda expected a lot of them
kingtigerknight
kingtigerknightOP•3y ago
Is there a website or video on how to look in the json file on what to do next? I looked into json files before but not this way without the key names XD
ErickO
ErickO•3y ago
well depends you have a list of the keys now Object.keys() returns an array so you could loop through that array and peek each of those keys
kingtigerknight
kingtigerknightOP•3y ago
I just notice I was running the wrong file XD
ErickO
ErickO•3y ago
wat
kingtigerknight
kingtigerknightOP•3y ago
it's running my typescript for some reason, I'm fixing it now tho
kingtigerknight
kingtigerknightOP•3y ago
kingtigerknight
kingtigerknightOP•3y ago
Oh... thats a problem
ErickO
ErickO•3y ago
OMEGALUL
kingtigerknight
kingtigerknightOP•3y ago
I'm just gonna gather my team and just be like "Yeah um.... file is too big to open"
ErickO
ErickO•3y ago
ok we need to change how to open the file then
kingtigerknight
kingtigerknightOP•3y ago
oh?
ErickO
ErickO•3y ago
this usually opens the entire file and puts it in a buffer but I guess node doesn't want 122GB in memory... who would've thought
kingtigerknight
kingtigerknightOP•3y ago
when something is 122GB but it's not a game XD
ErickO
ErickO•3y ago
so instead you need to open it as streams, not very familiar with big files and node js completely slipped my mind
kingtigerknight
kingtigerknightOP•3y ago
oh ok, how do I open it while it streams? I think it try to open it up on a browser to see what was inside. MY PC crashed XD
ErickO
ErickO•3y ago
npm
event-stream
construct pipes of streams of events. Latest version: 4.0.1, last published: 4 years ago. Start using event-stream in your project by running npm i event-stream. There are 1945 other projects in the npm registry using event-stream.
13eck
13eck•3y ago
Not too terribly familiar with how Node works with file streams, but I do know that's how to do it :p
ErickO
ErickO•3y ago
it's a bit of a pain you can use the npm package I sent to have a nice render of the data if you want to do it yourself however https://itnext.io/using-node-js-to-read-really-really-large-files-pt-1-d2057fe76b33 here's something that can help
kingtigerknight
kingtigerknightOP•3y ago
Actually quesiton, is this common to have a huge file like this? Cause with what the client has given us, is it even like realistic? XD
13eck
13eck•3y ago
More than several MB of a JSON is just bad form Like, what is this supposed to be, anyway?
kingtigerknight
kingtigerknightOP•3y ago
I'll have to like talk to the teacher, cause if this is unrealistic then I don't think this should even be a thing for an exam XD
13eck
13eck•3y ago
Even if it's an array of JSON objects that's a fuckton of info
kingtigerknight
kingtigerknightOP•3y ago
Oh yeah, heres the image of the file size XD
kingtigerknight
kingtigerknightOP•3y ago
Thanks for the information, I'll still learn about the stream to learn that incase I need it. But for now, me and my team are going to get out of it XD.
13eck
13eck•3y ago
See if you can find a way to read the first few lines of the file
ErickO
ErickO•3y ago
is this common to have a huge file like this?
in webdev? I'd fire you other areas of dev? sometimes
kingtigerknight
kingtigerknightOP•3y ago
O.O
ErickO
ErickO•3y ago
again, that's what data science is, there this would be common tho huge JSON files would not be that common that would be possible
13eck
13eck•3y ago
That way we can see how the file is organized. I'm hoping that it's not one JSON object and they just appended object after object. If that's the case you can read the file line by line
ErickO
ErickO•3y ago
yeah like I said, powershell probably supports the cat command but not sure
kingtigerknight
kingtigerknightOP•3y ago
so I tried cat. ummmmmmm
kingtigerknight
kingtigerknightOP•3y ago
13eck
13eck•3y ago
Can you do this head -n 5 file.json, show the output? This is to get the first 5 lines Hopefully it'll show us how the file is formatted
ErickO
ErickO•3y ago
forget the format look at that data I already hate it
kingtigerknight
kingtigerknightOP•3y ago
13eck
13eck•3y ago
YOU CAN'T MAKE ME Damn, no head command
kingtigerknight
kingtigerknightOP•3y ago
sad no head
kingtigerknight
kingtigerknightOP•3y ago
13eck
13eck•3y ago
Can you copy the first little bit of what cat showed you?
kingtigerknight
kingtigerknightOP•3y ago
Um I can't really do that. When I press enter it ZOOMS down and when I scroll up even after 1 sec of pausing, I can't scroll back to the top XD
13eck
13eck•3y ago
What I'm hoping to discern is if it's one huge JSON object or if the file is several objects either in an array or just one object, newline, second object, etc
kingtigerknight
kingtigerknightOP•3y ago
I can try again tho
13eck
13eck•3y ago
Well, have a look at the end of the file then See if you can tell if it's one large object or not
kingtigerknight
kingtigerknightOP•3y ago
HAHA auto clicker wins
kingtigerknight
kingtigerknightOP•3y ago
13eck
13eck•3y ago
Damn, just one gigantic object
kingtigerknight
kingtigerknightOP•3y ago
For the first time I can finally read the file. 😭
13eck
13eck•3y ago
not gonna lie, that object is bigger than even my ego Well, at least you can read it now :p But seriously, how does one expect you to parse a file that large?
ErickO
ErickO•3y ago
usually? not with javascript lol try the streams tho, particularly with even-stream it'll be a pain but shrug get a new assignment OMEGALUL
13eck
13eck•3y ago
It is what it is
kingtigerknight
kingtigerknightOP•3y ago
Well, thankyou all for teaching me powercell with cat I like how to read json another way. Um... I think I'm gonna see if there's a "smaller" version or another exam that our team can do cause now I don't even know of our teacher knows the limits of javascript XD
MarkBoots
MarkBoots•3y ago
It is a while back and i've only used it once. But this helped me out in a simular situation. https://dadroit.com/
Dadroit JSON Viewer
View, find, and export within BIG JSON data in a blink. Open 1GB JSON in seconds. Find with RegEX and JSONPath. Export JSON to XML, minified and formatted.
Want results from more Discord servers?
Add your server