Theo's Typesafe Cult•8mo ago

Optimizing importing json into sqlite with or without drizzle

Hi guys, I have this script running over my 60k row json file and somehow this is really really slow. is there an optimized way for turso?

  const jsonData = JSON.parse(data);
  const parsedJsonData = citySchema.parse(jsonData);

  for (const city of parsedJsonData) {
    await db.insert(schema.city).values({
      id: city.id,
      lon: city.lon,
      admin1: city.admin1,
      admin2: city.admin2,
      lat: city.lat,
      country: city.country,
      name: city.name,
    });
  }

  const jsonData = JSON.parse(data);
  const parsedJsonData = citySchema.parse(jsonData);

  for (const city of parsedJsonData) {
    await db.insert(schema.city).values({
      id: city.id,
      lon: city.lon,
      admin1: city.admin1,
      admin2: city.admin2,
      lat: city.lat,
      country: city.country,
      name: city.name,
    });
  }

I'm now 15 min into this script and were able to only add 4k lines

3 Replies

Neto•8mo ago

https://orm.drizzle.team/docs/insert#insert-multiple-rows

Drizzle ORM - Insert

Drizzle ORM is a lightweight and performant TypeScript ORM with developer experience in mind.

Neto•8mo ago

You can also run multiple of those in parallel

FleetAdmiralJakob 🗕 🗗 🗙OP•8mo ago

nice, down to 15 seconds

Gaming

Programming

Optimizing importing json into sqlite with or without drizzle