How Can I Pull 900K Records From My Database & Not Have The Page Slow Down When Rendering The Record
I have a project that is pulling 900K records from a database & displaying them to a view. Whenever I pull these records, I get a connection timeout error. I fixed that by pulling the top 50000 records, it works but then I have a problem that it takes time to fully display the page. I also have a problem that I'm displaying these records using DataTables & takes time to load the JavaScript. How can I fix this?
39 Replies
Whyever would you be pulling this many records all at once...?
Use pagination
rendering 900K records in a datatable will never ever be fast lol
and also entirely useless if you ask me
its just too much data to meaningfully work with in a browser/manually
you could offer an export of data instead
It's required for the user to see all these records & choose which ever record they want to use
Hw would they be even able to pick one record out of 900k lmao
You can't even read that many and parse them
Can you redirect me to where I can learn about this
lmao! i figured it out the hard way
Are you using EF or raw SQL?
mhhh! maybe export the data to an excel file or so?
they will search using an ID & the datatable will filter the records because it has this feature
Filter when fetching the data
Search when fetching the data
yeah, CSV would be a solid choice and works with excel
one solution is pagination, how does this work?
and yeah, if the user alreayd knows the ID, just have them enter it before you fetch the data
You get the first 50 records
Then the second batch of 50
pagination just "chunks" the results into managable bits - it will not help in this case
Third batch of 50
and so on
Requested by the user
Raw SQL, I used EF but then i needed to pull from multiple databases so i turned to raw SQL