SEO

Hi everyone It's possible to generate the robots.txt and sitemap.xml files on Solid Start project?

7 Replies

From ChatGPT. Maybe you can give this a try: #### Step 1: Create a sitemap.xml route 1. Create a New Route: Add a new file in the src/routes directory called sitemap.xml.tsx:

// src/routes/sitemap.xml.tsx
import { APIEvent } from "solid-start";

export async function GET({ request }: APIEvent) {
  const baseUrl = new URL(request.url).origin;

  const routes = [
    "/",
    "/about",
    "/contact",
    // Add more static routes here
  ];

  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    ${routes
      .map(
        (route) => `
    <url>
      <loc>${baseUrl}${route}</loc>
      <lastmod>${new Date().toISOString()}</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
    </url>
    `
      )
      .join("")}
  </urlset>`;

  return new Response(sitemap, {
    headers: {
      "Content-Type": "application/xml",
    },
  });
}

// src/routes/sitemap.xml.tsx
import { APIEvent } from "solid-start";

export async function GET({ request }: APIEvent) {
  const baseUrl = new URL(request.url).origin;

  const routes = [
    "/",
    "/about",
    "/contact",
    // Add more static routes here
  ];

  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    ${routes
      .map(
        (route) => `
    <url>
      <loc>${baseUrl}${route}</loc>
      <lastmod>${new Date().toISOString()}</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
    </url>
    `
      )
      .join("")}
  </urlset>`;

  return new Response(sitemap, {
    headers: {
      "Content-Type": "application/xml",
    },
  });
}

- This code creates an XML sitemap dynamically when you access the /sitemap.xml route. - Dynamic Routes: If you have dynamic routes (like blog posts), you can fetch data inside this route to generate the URLs. #### Step 2: Add Dynamic Routes (if needed) If your app has dynamic content (e.g., blog posts), you can fetch these in the sitemap.xml.tsx file:

// Example to add dynamic routes
const posts = await fetch("https://api.yourwebsite.com/posts").then(res => res.json());

const routes = [
  "/",
  "/about",
  ...posts.map(post => `/blog/${post.slug}`),
];

// Example to add dynamic routes
const posts = await fetch("https://api.yourwebsite.com/posts").then(res => res.json());

const routes = [
  "/",
  "/about",
  ...posts.map(post => `/blog/${post.slug}`),
];

2. Creating robots.txt Similar to Nuxt, you can create a robots.txt manually or programmatically: #### Option 1: Manual Creation Place a robots.txt file in your public/ directory:

User-agent: *
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xml

User-agent: *
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xml

#### Option 2: Programmatic Generation Create a robots.txt.tsx route:

// src/routes/robots.txt.tsx
import { APIEvent } from "solid-start";

export function GET({ request }: APIEvent) {
  const baseUrl = new URL(request.url).origin;

  const robotsTxt = `
    User-agent: *
    Allow: /
    Sitemap: ${baseUrl}/sitemap.xml
  `;

  return new Response(robotsTxt.trim(), {
    headers: {
      "Content-Type": "text/plain",
    },
  });
}

// src/routes/robots.txt.tsx
import { APIEvent } from "solid-start";

export function GET({ request }: APIEvent) {
  const baseUrl = new URL(request.url).origin;

  const robotsTxt = `
    User-agent: *
    Allow: /
    Sitemap: ${baseUrl}/sitemap.xml
  `;

  return new Response(robotsTxt.trim(), {
    headers: {
      "Content-Type": "text/plain",
    },
  });
}

3. Testing and Deployment - Testing Locally: Run your Solid Start app and check http://localhost:3000/sitemap.xml and http://localhost:3000/robots.txt to ensure everything is working correctly. - Deployment: When deploying your app (on platforms like Vercel, Netlify, or other edge platforms), Solid Start and Nitro will handle serving these files efficiently. 4. Advanced: Edge and Serverless Considerations If you're deploying to an edge or serverless environment (supported natively by Nitro), ensure that your routes for sitemap.xml and robots.txt are well-optimized for quick response times. Conclusion With Solid Start using Nitro, you can manually create and serve a sitemap and robots.txt file by defining custom routes. This gives you flexibility and control over how these files are generated, especially in dynamic environments. This way you could also fetch the data from sanity. If you don’t want to create it every time, you can also save the file in the public directory in the api route and then create a webhook in sanity which calls the api route if a page/post is created or deleted to update the sitemap.

// src/routes/api/generate-sitemap.tsx
import { APIEvent } from "solid-start";
import fs from "fs";
import path from "path";

export async function GET({ request }: APIEvent) {
  const baseUrl = new URL(request.url).origin;

  const routes = [
    "/",
    "/about",
    "/contact",
    // Add more static routes or fetch dynamic routes
  ];

  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    ${routes
      .map(
        (route) => `
    <url>
      <loc>${baseUrl}${route}</loc>
      <lastmod>${new Date().toISOString()}</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
    </url>
    `
      )
      .join("")}
  </urlset>`;

  // Save to public directory
  const sitemapPath = path.join(process.cwd(), "public", "sitemap.xml");
  fs.writeFileSync(sitemapPath, sitemap);

  return new Response("Sitemap generated and saved.", {
    status: 200,
  });
}

// src/routes/api/generate-sitemap.tsx
import { APIEvent } from "solid-start";
import fs from "fs";
import path from "path";

export async function GET({ request }: APIEvent) {
  const baseUrl = new URL(request.url).origin;

  const routes = [
    "/",
    "/about",
    "/contact",
    // Add more static routes or fetch dynamic routes
  ];

  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    ${routes
      .map(
        (route) => `
    <url>
      <loc>${baseUrl}${route}</loc>
      <lastmod>${new Date().toISOString()}</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
    </url>
    `
      )
      .join("")}
  </urlset>`;

  // Save to public directory
  const sitemapPath = path.join(process.cwd(), "public", "sitemap.xml");
  fs.writeFileSync(sitemapPath, sitemap);

  return new Response("Sitemap generated and saved.", {
    status: 200,
  });
}

Daniel Sousa @TutoDSOP•5mo ago

Thanks @Madaxen86 , but I can use createAsync inside the API route? Or the cache() function that I have in my requests will work without the createAsync?

Madaxen86•5mo ago

Not 100% sure. But i think you can call the cached function. Just give it a try. If not just make extract the function and export an uncached and a cached version.

Daniel Sousa @TutoDSOP•5mo ago

yeah I think it works: https://impulsionar.devserver7.com/sitemap.xml I did for the pages and for each blog post/project/service

Madaxen86•5mo ago

Yeah it is valid: You can test here https://www.xml-sitemaps.com/validate-xml-sitemap.html

Validate XML Sitemap - XML Sitemaps Generator

Validate XML Sitemap - Free Online Google Sitemap Generator. XML-sitemaps.com provides free online sitemap generator service, creating an XML sitemap that can be submitted to Google, Bing and other search engines to help them crawl your website better. It will also generate an HTML site map to allow your website visitors to navigate easier.

Daniel Sousa @TutoDSOP•5mo ago

Apparently not

Madaxen86•5mo ago

You need to use the url to the sitemap.xml, not to the page itself

Gaming

Programming

SEO