Guides › Platforms

Next.js: robots.ts and sitemap.ts (App Router)

In the App Router, add app/robots.ts and app/sitemap.ts. Next.js serves them at /robots.txt and /sitemap.xml.

// app/robots.ts
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: '/api/' },
      { userAgent: 'GPTBot', disallow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

// app/sitemap.ts
import type { MetadataRoute } from 'next'
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getPosts()
  return [
    { url: 'https://example.com/', lastModified: new Date('2026-10-01') },
    ...posts.map(p => ({ url: 'https://example.com/blog/' + p.slug, lastModified: p.updatedAt })),
  ]
}

For more than 50,000 URLs, export generateSitemaps() to split the sitemap into several files. A static public/robots.txt also works but can't be dynamic. Don't keep both. Watch for preview deployments being indexed; Vercel adds X-Robots-Tag: noindex on preview URLs by default.

Related