robots.ts

Generate /robots.txt from code: app/robots.ts returns the crawler rules and GioJS writes the file.

import type { MetadataRoute } from '@gio.js/core';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/', disallow: ['/account', '/api'] },
    sitemap: '/sitemap.xml',
  };
}

Reference

File name and location

app/robots.ts or app/robots.js, at the root of app/ only. It answers /robots.txt.

Exports

The Robots object

The types are MetadataRoute.Robots and RobotsRule, from @gio.js/core.

Response

• 200, Content-Type: text/plain; charset=utf-8. Blocks are separated by a blank line.
• Only GET and HEAD: other methods get 405 with Allow: GET, HEAD.
• Cached by the server for revalidate seconds, with an ETag and no automatic Cache-Control.
• A value with a line break is refused (it could smuggle in a directive of its own), as is a negative crawlDelay or a result that is not an object. Those, and a function that throws, answer 500 Internal Server Error (ref <digest>), never cached.

Examples

Different rules per crawler

import type { MetadataRoute } from '@gio.js/core';

export const revalidate = false;

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/account', '/api'] },
      { userAgent: 'GPTBot', disallow: '/' },
    ],
    sitemap: '/sitemap.xml',
  };
}

With GIO_SITE_URL=https://example.com, /robots.txt is:

User-Agent: *
Allow: /
Disallow: /account
Disallow: /api

User-Agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Block crawlers outside production

import type { MetadataRoute } from '@gio.js/core';

export default function robots(): MetadataRoute.Robots {
  if (process.env.DEPLOY_ENV !== 'production') {
    return { rules: { userAgent: '*', disallow: '/' } };
  }
  return { rules: { userAgent: '*', allow: '/' }, sitemap: '/sitemap.xml' };
}

DEPLOY_ENV here is a variable of your own, set per deployment (a staging server runs with NODE_ENV=production too).

Good to know

• A public/robots.txt wins over app/robots.ts: it is served first, the module never runs, and startup warns. A page or route.ts at /robots.txt stops startup.
• gio export writes out/robots.txt from it. Without one (and without public/robots.txt), the export writes User-agent: * / Allow: /, plus a Sitemap: line when GIO_SITE_URL is set. The server writes nothing in that case: /robots.txt is a 404.
• Rules only ask crawlers to stay away. Protect private pages with guards or authentication, not with Disallow.

Related

• Metadata & SEO - the guide.
• sitemap.ts, manifest.ts, public/
• metadata.robots - per-page <meta name="robots">.

Version history