Nuxt•2mo ago

Loading in large dataset

Hi, I currently have a huge .json file (40mb) which I'm trying to import. const { default: availableCities } = await import('~/assets/cities.json') console.log(Object.keys(availableCities)) But I run out of memory. What would be the best way to get this data into Nuxt.

8 Replies

kapa.ai•2mo ago

Hi @Kyllian. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave: ---------------------- For loading a large JSON dataset (40MB) in Nuxt, you're encountering memory issues with direct imports. Based on the knowledge sources, there are a few approaches you could consider: Use Nuxt Content Nuxt Content v3 would be a good solution for managing large datasets. It uses an SQL-based storage system which is more efficient than file-based approaches:

// Define a collection in content.config.ts
import { defineCollection, defineContentConfig, z } from '@nuxt/content'

export default defineContentConfig({
  collections: {
    cities: defineCollection({
      type: 'data',
      source: 'cities/**.json',
      schema: z.object({
        // Define your schema here based on your data structure
      })
    })
  }
})

// Define a collection in content.config.ts
import { defineCollection, defineContentConfig, z } from '@nuxt/content'

export default defineContentConfig({
  collections: {
    cities: defineCollection({
      type: 'data',
      source: 'cities/**.json',
      schema: z.object({
        // Define your schema here based on your data structure
      })
    })
  }
})

Then you can query it:

<script lang="ts" setup>
// Get all cities or use filters to get specific ones
const { data: cities } = await useAsyncData('cities', () => {
  return queryCollection('cities').all()
})
</script>

<script lang="ts" setup>
// Get all cities or use filters to get specific ones
const { data: cities } = await useAsyncData('cities', () => {
  return queryCollection('cities').all()
})
</script>

This approach is recommended because: 1. It's optimized for large datasets 2. It supports serverless environments 3. It improves client-side navigation performance Nuxt Content v3 specifically mentions that it "moves away from a file-based storage approach to an SQL database system" which "enhances the way your data files are stored and structured, ensuring better performance and scalability." Alternative Approaches If you prefer not to use Nuxt Content: 1. Split your data: Break your large JSON file into smaller chunks and load only what you need. 2. Server API route: Create an API endpoint in the server/api directory that returns only the necessary data:

// server/api/cities.ts
export default defineEventHandler(() => {
  const availableCities = require('~/assets/cities.json')
  // Return only what you need, perhaps paginated
  return Object.keys(availableCities).slice(0, 100) // Example: return first 100 cities
})

// server/api/cities.ts
export default defineEventHandler(() => {
  const availableCities = require('~/assets/cities.json')
  // Return only what you need, perhaps paginated
  return Object.keys(availableCities).slice(0, 100) // Example: return first 100 cities
})

Then fetch it:

<script setup>
const { data } = await useFetch('/api/cities')
</script>

<script setup>
const { data } = await useFetch('/api/cities')
</script>

3. Use useAsyncData: If you're using a third-party SDK or need more control:

const { data: cities } = await useAsyncData('cities', async () => {
  // Load and process your data here
  // Return only what you need
})

const { data: cities } = await useAsyncData('cities', async () => {
  // Load and process your data here
  // Return only what you need
})

The key is to avoid loading the entire 40MB file at once, and instead focus on loading only the data you need when you need it. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ----------------------

kapa.ai•2mo ago

<@166602211780984832> Kapa.ai is still learning and improving, please let me know how I did by reacting below.

KyllianOP•2mo ago

@kapa.ai using nuxt content, how can I fetch content on server side

kapa.ai•2mo ago

Let me check the knowledge base for you again. One moment! :books: ---------------------- To fetch content on the server side using Nuxt Content, you need to use the server-side versions of the query utilities. Here's how you can do it: Server-Side Content Fetching with Nuxt Content When working with Nuxt Content on the server side, you use the same query utilities as on the client side, but with a slight difference - you need to pass the event parameter as the first argument. Basic Server API Example

// server/api/[slug].ts
export default eventHandler(async (event) => {
  const { slug } = getRouterParams(event)
  const page = await queryCollection(event, 'docs').path(slug).first()
  return page
})

// server/api/[slug].ts
export default eventHandler(async (event) => {
  const { slug } = getRouterParams(event)
  const page = await queryCollection(event, 'docs').path(slug).first()
  return page
})

This creates an API endpoint that fetches a specific content item based on the slug parameter from the URL. Nuxt Content documentation shows that you need to create a server/tsconfig.json file to avoid type errors:

{
  "extends": "../.nuxt/tsconfig.server.json"
}

{
  "extends": "../.nuxt/tsconfig.server.json"
}

Other Server-Side Query Utilities Nuxt Content provides several server-side query utilities: 1. queryCollection - For fetching content items:

   // server/api/content.ts
   export default eventHandler(async (event) => {
     const content = await queryCollection(event, 'content').all()
     return content
   })

   // server/api/content.ts
   export default eventHandler(async (event) => {
     const content = await queryCollection(event, 'content').all()
     return content
   })

2. queryCollectionNavigation - For generating navigation:

   // server/api/navigation.ts
   export default eventHandler(async (event) => {
     const navigation = await queryCollectionNavigation(event, 'docs')
     return navigation
   })

   // server/api/navigation.ts
   export default eventHandler(async (event) => {
     const navigation = await queryCollectionNavigation(event, 'docs')
     return navigation
   })

Learn more about navigation queries 3. queryCollectionItemSurroundings - For finding sibling content:

   // server/api/surroundings.ts
   export default eventHandler(async (event) => {
     const surroundings = await queryCollectionItemSurroundings(event, 'docs', '/foo')
     return surroundings
   })

   // server/api/surroundings.ts
   export default eventHandler(async (event) => {
     const surroundings = await queryCollectionItemSurroundings(event, 'docs', '/foo')
     return surroundings
   })

Learn more about surroundings queries These server-side utilities allow you to create API endpoints that can be called from your client-side code or used in server-side rendering. Remember that when using these utilities on the server side, the main difference is passing the event parameter as the first argument to each function. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ----------------------

kapa.ai•2mo ago

<@166602211780984832> Kapa.ai is still learning and improving, please let me know how I did by reacting below.

johntanzer•2mo ago

Hey have come across this before in nitro didnt find a solution.

KyllianOP•2mo ago

yeah me neither, ended up scrapping the functionality

johntanzer•2mo ago

what did come out every time i had this issue is that this is prob not the correct/best thing/way to do! did spk to pooya about it cant find the issue though 🙈

Gaming

Programming

Loading in large dataset

Did you find this page helpful?