Valkyrian Labs logo

Dynamic Sitemap Helper

Add canonical docs pages to a Next App Router sitemap.

Dynamic Sitemap Helper

Use getDocsForSitemap from the /next export when a Next App Router site has a dynamic src/app/sitemap.ts file.

The helper reads published docs sets, resolves group paths, includes the generated docs records inside each set, prepends siteUrl, merges optional static routes, and returns a ready-to-use MetadataRoute.Sitemap array. By default, the output is limited to canonical crawler-discoverable docs pages.

1import type { MetadataRoute } from 'next'2 3import config from '@payload-config'4import { getPayload } from 'payload'5 6import { getDocsForSitemap } from '@valkyrianlabs/payload-markdown-docs/next'7 8const siteUrl = process.env.NEXT_PUBLIC_SITE_URL ?? 'https://example.com'9 10export default async function sitemap(): Promise<MetadataRoute.Sitemap> {11  const payload = await getPayload({ config })12  return getDocsForSitemap({13    payload,14    siteUrl,15  })16}

Combine With Other Routes

Most apps also include static routes, Pages collection routes, AI discovery files, or other dynamic content in the same sitemap. Use additionalRoutes for site-relative path entries or absolute url entries.

1import type { MetadataRoute } from 'next'2 3import config from '@payload-config'4import { getPayload } from 'payload'5 6import { getDocsForSitemap } from '@valkyrianlabs/payload-markdown-docs/next'7 8const siteUrl = process.env.NEXT_PUBLIC_SITE_URL ?? 'https://example.com'9 10export default async function sitemap(): Promise<MetadataRoute.Sitemap> {11  const payload = await getPayload({ config })12  const docs = await getDocsForSitemap({13    additionalRoutes: [14      { path: '/agent-index.txt' },15    ],16    payload,17    siteUrl,18  })19 20  return [21    {22      url: siteUrl,23    },24    ...docs,25  ]26}

The sitemap helper dedupes by final URL. When the same URL appears more than once, the newest lastModified value is kept. Output remains sorted by URL.

AI Discovery Routes

sitemap.xml and llms.txt serve different jobs:

  • sitemap.xml is crawler discovery.
  • llms.txt is an AI-readable entrypoint.
  • native skills are agent workflow artifacts.

getDocsForSitemap does not include raw AI-facing artifacts by default. Human docs routes remain in sitemap.xml; /llms.txt, /llms-full.txt, and native skill routes can stay publicly served without being pushed into search crawler discovery.

Opt in only when the app intentionally wants raw artifacts in sitemap.xml:

1const docs = await getDocsForSitemap({2  includeLlms: true,3  includeSkills: true,4  payload,5  siteUrl,6})

Options:

  • includeLlms: includes generated and stored llms.txt / llms-full.txt routes.
  • includeSkills: includes stored native skill artifact routes and generated skill indexes such as /skills/codex.
  • includeAssets: includes stored generic static assets only. It does not imply includeLlms or includeSkills.

For static files that are not synced, keep using additionalRoutes.

Use getPayloadMarkdownDocsAiSitemapRoutes to build common AI/static routes:

1import type { MetadataRoute } from 'next'2 3import config from '@payload-config'4import {5  getDocsForSitemap,6  getPayloadMarkdownDocsAiSitemapRoutes,7} from '@valkyrianlabs/payload-markdown-docs/next'8import { getPayload } from 'payload'9 10const aiRoutes = getPayloadMarkdownDocsAiSitemapRoutes({11  includeLlmsFull: true,12  skills: [13    {14      basePath: '/plugins/payload-markdown-docs/skills',15      agents: ['codex', 'claude'],16      files: [17        'SKILL.md',18        'reference/docs-package.md',19        'reference/frontmatter.md',20        'reference/workflow.md',21        'reference/sync.md',22        'reference/routing.md',23        'reference/admin.md',24        'reference/troubleshooting.md',25        'examples/github-actions.md',26      ],27    },28  ],29})30 31export default async function sitemap(): Promise<MetadataRoute.Sitemap> {32  const payload = await getPayload({ config })33 34  return getDocsForSitemap({35    additionalRoutes: aiRoutes,36    payload,37    siteUrl,38  })39}

additionalRoutes is always explicit. Only pass aiRoutes when the sitemap is intentionally opting raw AI artifacts into crawler discovery.

Skill artifacts can be hosted under plugin docs routes, such as /plugins/payload-markdown-docs/skills/codex/SKILL.md, or under a top-level route such as /skills/payload-markdown-docs/codex/SKILL.md. Set basePath to the public route your site owns.

Serving Synced Assets

The plugin registers Payload-owned GET endpoints for generated AI discovery files and synced skills:

  • /llms.txt
  • /llms-full.txt
  • <computed-docs-set-route>/llms.txt
  • <computed-docs-set-route>/llms-full.txt
  • <computed-docs-set-route>/skills/<agent>
  • <computed-docs-set-route>/skills/<agent>.zip
  • <computed-docs-set-route>/skills/<agent>/<path...>

For example, a docs set served at /plugins/payload-markdown-docs exposes /plugins/payload-markdown-docs/llms.txt, /plugins/payload-markdown-docs/skills/codex, and /plugins/payload-markdown-docs/skills/codex.zip; raw skill files remain available at /plugins/payload-markdown-docs/skills/codex/SKILL.md and supporting file paths after the assets are synced. Consuming apps should install the public Next route files that delegate to the package asset route handler:

1pnpm exec payload-markdown-docs install routes --payload-app "src/app/(payload)"

Use --payload-app "app/(payload)" for apps without src/. If the public routes return an asset schema error, migrate the Payload database so the payload-markdown-docs-assets collection table exists.

The /api/... asset URLs are implementation/internal fallback URLs. Public sitemap entries should use the canonical routes outside /api.

Cache Keys And Tags

The helper wraps its read in unstable_cache. Override cacheKey or tags when the app uses different sitemap invalidation tags.

1const docs = await getDocsForSitemap({2  cacheKey: ['sitemap-docs-v2'],3  payload,4  siteUrl,5  tags: ['sitemap', 'docs'],6})

Defaults:

  • cacheKey: sitemap-docs-v2
  • includeAssets: false
  • includeLlms: false
  • includeSkills: false
  • recursive: true
  • tags: sitemap, sitemap:docs

Recursive Docs

By default, docs set indexes and generated child docs are included in the sitemap. Disable recursion only when the app intentionally wants the base docs set URLs.

1const docs = await getDocsForSitemap({2  payload,3  recursive: false,4  siteUrl,5})

Custom Collection Slugs

If the plugin uses custom collection slugs, pass them through collections.

1const docs = await getDocsForSitemap({2  collections: {3    docs: 'knowledge-docs',4    docsGroups: 'knowledge-groups',5    docsSets: 'knowledge-sets',6  },7  payload,8  siteUrl,9})

Paginated Result

Use getPaginatedDocsForSitemap when the app needs the original Payload-style paginated result instead of the mapped Next sitemap array.

1import { getPaginatedDocsForSitemap } from '@valkyrianlabs/payload-markdown-docs/next'2 3const result = await getPaginatedDocsForSitemap({4  payload,5  siteUrl,6})7 8// result.docs: Array<{ url?: string | null; lastModified?: string | null }>