Theo's Typesafe Cult•4mo ago

Cost Efficient XL Generation for Large Products

Hey guys, title could have been better, but I am basically trying to setup an XML product feed for my store, which has 20k+ products including variants, and it constantly grows and shrinks throughout each day. I want the feed to rerun every 15 minutes, and I want to host this on vercel. I am worried about serverles costs since this is a function, below is my code with my approach I am doing caching, is this the most efficient way to handle this to reduce costs in Vercel? Is serverless (vercel) hosting bad for this use case?

// app/api/product-feed/route.ts

import { NextResponse } from 'next/server';
import { kv } from '@vercel/kv';
import { BigCommerceClient } from './bigcommerce-client';
import { generateXMLFeed } from './xml-generator';

const CACHE_KEY = 'product-feed-cache';
const CACHE_TTL = 15 * 60; // 15 minutes in seconds

export async function GET() {
  try {
    // Check cache first
    const cachedFeed = await kv.get(CACHE_KEY);
    if (cachedFeed) {
      return new NextResponse(cachedFeed, {
        headers: { 'Content-Type': 'application/xml' },
      });
    }

    // If not in cache, generate new feed
    const client = new BigCommerceClient();
    const products = await client.getAllProducts();
    const xmlFeed = generateXMLFeed(products);

    // Cache the new feed
    await kv.set(CACHE_KEY, xmlFeed, { ex: CACHE_TTL });

    return new NextResponse(xmlFeed, {
      headers: { 'Content-Type': 'application/xml' },
    });
  } catch (error) {
    console.error('Error generating product feed:', error);
    return new NextResponse('Error generating product feed', { status: 500 });
  }
}

// Scheduled job to update cache (run every 15 minutes)
export async function POST() {
  try {
    const client = new BigCommerceClient();
    const products = await client.getAllProducts();
    const xmlFeed = generateXMLFeed(products);
    await kv.set(CACHE_KEY, xmlFeed, { ex: CACHE_TTL });
    return new NextResponse('Cache updated successfully', { status: 200 });
  } catch (error) {
    console.error('Error updating cache:', error);
    return new NextResponse('Error updating cache', { status: 500 });
  }
}

// app/api/product-feed/route.ts

import { NextResponse } from 'next/server';
import { kv } from '@vercel/kv';
import { BigCommerceClient } from './bigcommerce-client';
import { generateXMLFeed } from './xml-generator';

const CACHE_KEY = 'product-feed-cache';
const CACHE_TTL = 15 * 60; // 15 minutes in seconds

export async function GET() {
  try {
    // Check cache first
    const cachedFeed = await kv.get(CACHE_KEY);
    if (cachedFeed) {
      return new NextResponse(cachedFeed, {
        headers: { 'Content-Type': 'application/xml' },
      });
    }

    // If not in cache, generate new feed
    const client = new BigCommerceClient();
    const products = await client.getAllProducts();
    const xmlFeed = generateXMLFeed(products);

    // Cache the new feed
    await kv.set(CACHE_KEY, xmlFeed, { ex: CACHE_TTL });

    return new NextResponse(xmlFeed, {
      headers: { 'Content-Type': 'application/xml' },
    });
  } catch (error) {
    console.error('Error generating product feed:', error);
    return new NextResponse('Error generating product feed', { status: 500 });
  }
}

// Scheduled job to update cache (run every 15 minutes)
export async function POST() {
  try {
    const client = new BigCommerceClient();
    const products = await client.getAllProducts();
    const xmlFeed = generateXMLFeed(products);
    await kv.set(CACHE_KEY, xmlFeed, { ex: CACHE_TTL });
    return new NextResponse('Cache updated successfully', { status: 200 });
  } catch (error) {
    console.error('Error updating cache:', error);
    return new NextResponse('Error updating cache', { status: 500 });
  }
}

Then I would setup a cron job in vercel

5 Replies

oz•4mo ago

I guess a non serverless service will always be cheaper for an API like this

Neto•4mo ago

store as blobs do you need constant access to it?

oz•4mo ago

No its for a search engine, they told us they need it. They will access it once every 15min.

Neto•4mo ago

if they need all the data and you won't be mutating and accessing it directly storing as a blob should be good enough, "super cheap"

oz•4mo ago

Thanks!

Gaming

Programming

Cost Efficient XL Generation for Large Products