Need help with opening and reading a 122 GB JSON file

Dose anyone know a website or app or anything to open a json file and also read it without it crashing?

66 Replies

ErickO•2y ago

So here's what you can do first to get a basic idea you can open your terminal but uh....are you on windows, mac or linux? in a terminal you could peek at the first few lines of a file by doing cat myfile.json and the end of the file like tail myfile.json

kingtigerknight•2y ago

I'm on windows

ErickO•2y ago

no idea, I think powershell supports those commands? or WSL if you're using that

kingtigerknight•2y ago

Time for google time 😄

ErickO•2y ago

const fs = require("fs")

fs.readFile("./myfile.json", "utf8", (err,data) => {
  if(err) {
   console.error(err)
  } else {
    const content = JSON.parse(data)
    console.log(Object.keys(content))
  }
})

const fs = require("fs")

fs.readFile("./myfile.json", "utf8", (err,data) => {
  if(err) {
   console.error(err)
  } else {
    const content = JSON.parse(data)
    console.log(Object.keys(content))
  }
})

this can show you the main keys in your data and you can go from there anyway you can see the basic idea of how you open files and read it so you can take peeks at the data and decide what to do with it, based on it's contents....? idk your assignment is weird

kingtigerknight•2y ago

yeah 😦

ErickO•2y ago

I mean it's not uncommon as a data scientist to have files that you don't even know what they contain or how they are structured... but idk if that's what they're teaching you

kingtigerknight•2y ago

So basically were helping to create this app that reads information about the json but we can't open it or read it, but the client dosen't know what a json is since his just a messenger, but we can't ask the company cause the messenger is a one way. This is confusing XD

ErickO•2y ago

oh fun well anyway, take the commands I told you about or the script I sent and work from there, hopefully the key names are descriptive of what the file contains and if not you at least know how to access the data on it and go from there

kingtigerknight•2y ago

Cool cheers, time to see what is in here XD It just returned 3000. Oh boy

ErickO•2y ago

3k keys? oof

kingtigerknight•2y ago

rip MEEEEEEEEEEEEEEEEEEEEEE

ErickO•2y ago

yeah kinda expected a lot of them

kingtigerknight•2y ago

Is there a website or video on how to look in the json file on what to do next? I looked into json files before but not this way without the key names XD

ErickO•2y ago

well depends you have a list of the keys now Object.keys() returns an array so you could loop through that array and peek each of those keys

kingtigerknight•2y ago

I just notice I was running the wrong file XD

ErickO•2y ago

kingtigerknight•2y ago

it's running my typescript for some reason, I'm fixing it now tho

kingtigerknight•2y ago

kingtigerknight•2y ago

Oh... thats a problem

ErickO•2y ago

kingtigerknight•2y ago

I'm just gonna gather my team and just be like "Yeah um.... file is too big to open"

ErickO•2y ago

ok we need to change how to open the file then

kingtigerknight•2y ago

oh?

ErickO•2y ago

this usually opens the entire file and puts it in a buffer but I guess node doesn't want 122GB in memory... who would've thought

kingtigerknight•2y ago

when something is 122GB but it's not a game XD

ErickO•2y ago

so instead you need to open it as streams, not very familiar with big files and node js completely slipped my mind

13eck•2y ago

https://nodejs.org/api/fs.html#fscreatereadstreampath-options

kingtigerknight•2y ago

oh ok, how do I open it while it streams? I think it try to open it up on a browser to see what was inside. MY PC crashed XD

ErickO•2y ago

https://www.npmjs.com/package/event-stream

npm

event-stream

construct pipes of streams of events. Latest version: 4.0.1, last published: 4 years ago. Start using event-stream in your project by running npm i event-stream. There are 1945 other projects in the npm registry using event-stream.

13eck•2y ago

Not too terribly familiar with how Node works with file streams, but I do know that's how to do it :p

ErickO•2y ago

it's a bit of a pain you can use the npm package I sent to have a nice render of the data if you want to do it yourself however https://itnext.io/using-node-js-to-read-really-really-large-files-pt-1-d2057fe76b33 here's something that can help

kingtigerknight•2y ago

Actually quesiton, is this common to have a huge file like this? Cause with what the client has given us, is it even like realistic? XD

13eck•2y ago

More than several MB of a JSON is just bad form Like, what is this supposed to be, anyway?

kingtigerknight•2y ago

I'll have to like talk to the teacher, cause if this is unrealistic then I don't think this should even be a thing for an exam XD

13eck•2y ago

Even if it's an array of JSON objects that's a fuckton of info

kingtigerknight•2y ago

Oh yeah, heres the image of the file size XD

kingtigerknight•2y ago

Thanks for the information, I'll still learn about the stream to learn that incase I need it. But for now, me and my team are going to get out of it XD.

13eck•2y ago

See if you can find a way to read the first few lines of the file

ErickO•2y ago

is this common to have a huge file like this?

in webdev? I'd fire you other areas of dev? sometimes

kingtigerknight•2y ago

O.O

ErickO•2y ago

again, that's what data science is, there this would be common tho huge JSON files would not be that common that would be possible

13eck•2y ago

That way we can see how the file is organized. I'm hoping that it's not one JSON object and they just appended object after object. If that's the case you can read the file line by line

ErickO•2y ago

yeah like I said, powershell probably supports the cat command but not sure

kingtigerknight•2y ago

so I tried cat. ummmmmmm

kingtigerknight•2y ago

13eck•2y ago

Can you do this head -n 5 file.json, show the output? This is to get the first 5 lines Hopefully it'll show us how the file is formatted

ErickO•2y ago

forget the format look at that data I already hate it

kingtigerknight•2y ago

13eck•2y ago

YOU CAN'T MAKE ME Damn, no head command

kingtigerknight•2y ago

sad no head

kingtigerknight•2y ago

13eck•2y ago

Can you copy the first little bit of what cat showed you?

kingtigerknight•2y ago

Um I can't really do that. When I press enter it ZOOMS down and when I scroll up even after 1 sec of pausing, I can't scroll back to the top XD

13eck•2y ago

What I'm hoping to discern is if it's one huge JSON object or if the file is several objects either in an array or just one object, newline, second object, etc

kingtigerknight•2y ago

I can try again tho

13eck•2y ago

Well, have a look at the end of the file then See if you can tell if it's one large object or not

kingtigerknight•2y ago

HAHA auto clicker wins

kingtigerknight•2y ago

13eck•2y ago

Damn, just one gigantic object

kingtigerknight•2y ago

For the first time I can finally read the file. 😭

13eck•2y ago

not gonna lie, that object is bigger than even my ego Well, at least you can read it now :p But seriously, how does one expect you to parse a file that large?

ErickO•2y ago

usually? not with javascript lol try the streams tho, particularly with even-stream it'll be a pain but shrug

get a new assignment OMEGALUL

13eck•2y ago

It is what it is

kingtigerknight•2y ago

Well, thankyou all for teaching me powercell with cat I like how to read json another way. Um... I think I'm gonna see if there's a "smaller" version or another exam that our team can do cause now I don't even know of our teacher knows the limits of javascript XD

MarkBoots•2y ago

It is a while back and i've only used it once. But this helped me out in a simular situation. https://dadroit.com/

Dadroit JSON Viewer

View, find, and export within BIG JSON data in a blink. Open 1GB JSON in seconds. Find with RegEX and JSONPath. Export JSON to XML, minified and formatted.

Gaming

Programming

Need help with opening and reading a 122 GB JSON file