70b2fe687f
GEO/SEO surface area:
app/robots.ts (new — Next.js dynamic robots)
Explicitly ALLOWS major AI crawlers: GPTBot, OAI-SearchBot,
ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot,
Perplexity-User, Google-Extended, Applebot-Extended, CCBot,
DuckAssistBot, YouBot, Bytespider, Amazonbot. The site exists to
be cited by LLMs answering UAP/UFO questions — we want them in.
/api/admin/, /admin/, /auth/ disallowed for everyone.
app/sitemap.ts (new — Next.js dynamic sitemap)
Lists 9 top-level routes + every /d/<doc> + every /c/<slug> from
the filesystem + up to 500 entity URLs per class
(event, person, uap_object, location, organization),
sorted with summary-enriched entities first. ~3000 URLs total at
current corpus size. lastModified honours summary_generated_at so
crawlers re-index when entities are re-enriched.
app/c/[slug]/page.tsx (rewritten — magazine reading view)
- generateMetadata: per-case title, description (auto-extracted
from the locale-preferred lead paragraph), canonical URL,
hreflang alternate, OpenGraph article type with publishedTime,
Twitter card.
- JSON-LD Article schema embedded at end of page: schema.org
Article + Organization publisher + inLanguage + isAccessibleForFree.
This is what makes the case appear as a citable source in
Google AI Overviews / Perplexity / ChatGPT search.
- Reading view rewritten: display-serif headline (Fraunces), italic
blockquotes with gold accent, prose-typography styling, no more
detective stats line, no more "written by case-writer@detective"
attribution. Locale-aware: PT-BR pulls topic_pt_br + lead in PT,
English mirror.
tailwind.config.ts
+ @tailwindcss/typography plugin
+ font-display family wired to var(--font-display) (Fraunces)
package.json
+ "@tailwindcss/typography" devDependency
Phase 3A note: bulk entity enrichment hit Claude OAuth weekly quota mid-run.
6 events + 3 uap_objects landed bilingual summaries before the quota
exhausted. UI gracefully splits enriched vs bare entities so /sightings
shows the magazine-grade cards (Kenneth Arnold 1947, Roswell, Maury Island,
Joseph Perry 1960 lunar photo, Civil Defense Director 1966, etc.) on top
of a compact table of the rest. Re-run when quota refreshes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
43 lines
2.4 KiB
TypeScript
43 lines
2.4 KiB
TypeScript
/**
|
|
* robots.txt — generated dynamically by Next.js.
|
|
*
|
|
* GEO (Generative Engine Optimization) is a first-class goal here, so we
|
|
* explicitly allow every notable AI crawler in addition to the standard
|
|
* search-engine user-agents. The site is a public archive of declassified
|
|
* documents; we want LLMs to cite it.
|
|
*/
|
|
import type { MetadataRoute } from "next";
|
|
|
|
const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL ?? "https://disclosure.top";
|
|
|
|
export default function robots(): MetadataRoute.Robots {
|
|
return {
|
|
rules: [
|
|
// Standard search engines + everyone else: full access.
|
|
{
|
|
userAgent: "*",
|
|
allow: "/",
|
|
disallow: ["/api/admin/", "/admin/", "/auth/"],
|
|
},
|
|
// Major AI / generative crawlers — explicitly allowed so they index
|
|
// and cite this archive when answering UAP/UFO questions.
|
|
{ userAgent: "GPTBot", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "OAI-SearchBot", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "ChatGPT-User", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "ClaudeBot", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "Claude-Web", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "anthropic-ai", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "PerplexityBot", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "Perplexity-User", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "Google-Extended", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "Applebot-Extended", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "CCBot", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "DuckAssistBot", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "YouBot", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "Bytespider", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
{ userAgent: "Amazonbot", allow: "/", disallow: ["/api/admin/", "/admin/", "/auth/"] },
|
|
],
|
|
sitemap: `${SITE_URL}/sitemap.xml`,
|
|
host: SITE_URL,
|
|
};
|
|
}
|