Versioned robots.txt¶

Your site has exactly one robots.txt, but its contents change over time, and a bad change (a stray Disallow: /) can quietly wreck your search traffic. django-seo-suite serves robots.txt from the database and keeps every version as a retained snapshot, so when traffic drops you can line the timeline up against your analytics and see whether a robots.txt change is to blame.

Key ideas:

• Each edit is a new version (a new row). History is never overwritten.
• One version is active per site — the one that gets served.
• You switch the live version with an "Activate" action in the admin; you never hand-edit the active flag, so you can't accidentally end up with two (or zero) active versions.
• Every version records who created it and when, plus when it was activated.

Setup¶

The model lives in the core seo_suite app, so it's available once the package is installed and migrated:

python manage.py migrate

Add the route to your root URLconf:

# urls.py
from django.urls import include, path

urlpatterns = [
    # ...
    path("", include("seo_suite.urls")),   # serves /robots.txt
]

GET /robots.txt now returns the active version as text/plain.

Managing versions in the admin¶

Open SEO Suite → robots.txt versions in the Django admin.

To change robots.txt, add a version: the form is pre-filled with the current live content, so you edit it, add an optional label (e.g. "Block /staging") and a note explaining why, then Save. Saving publishes it immediately as the live robots.txt. The version that was live becomes a history row, so nothing is overwritten.

Past versions are read-only: you can't change a snapshot's served content in place. The live robots.txt only ever changes by adding a new version, which keeps the history honest.

The changelist shows version, label, site, is_active, created_at, activated_at, and created_by, so you have a complete audit trail.

Fallback¶

When no version is active (for the current site), the view serves SEO_SUITE["ROBOTS_TXT_FALLBACK"]:

SEO_SUITE = {
    "ROBOTS_TXT_FALLBACK": "User-agent: *\nAllow: /",
}

Pointing crawlers at your sitemap¶

Append Sitemap: lines to every served robots.txt without editing the content:

SEO_SUITE = {
    "ROBOTS_SITEMAP_URLS": [
        "https://example.com/sitemap.xml",
        "/news-sitemap.xml",   # relative URLs are made absolute against the request
    ],
}

Output:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/news-sitemap.xml

Multiple sites¶

If you run several sites, set the site on a version to scope it to that site. Leave it blank for a version that applies to all sites. Resolution prefers a site-specific active version and falls back to the global one. Site resolution follows the same chain as the rest of the package (django.contrib.sites if installed, otherwise settings.SITE_ID, otherwise treated as a single site).

Caching¶

robots.txt is low-traffic, so caching is off by default. If you want it:

SEO_SUITE = {
    "ROBOTS_CACHE_TTL": 3600,   # seconds; 0 disables (default)
}

The served body is cached per site and automatically invalidated whenever any version is saved or activated, so activating a new version takes effect immediately.

Settings summary¶