# robots.txt for PayPeek (https://www.paypeek.ai) # Docs: https://developers.google.com/search/docs/crawling-indexing/robots/intro # -------------------------------------------------------------------------- # Default policy: everything is crawlable except private app surfaces, # authentication flows, embed views (duplicate content), and tracking relays. # -------------------------------------------------------------------------- User-agent: * Allow: / Allow: /salaries/ Disallow: /api/ Disallow: /auth/ Disallow: /auth/callback Disallow: /verify-email Disallow: /test-verification Disallow: /forgot-password Disallow: /reset-password Disallow: /relay Disallow: /fb Disallow: /embed/ Disallow: /p/profiles Disallow: /p/profile/ Disallow: /p/estimate-result/ Disallow: /p/history Disallow: /p/leaderboard Disallow: /p/invitations Disallow: /p/settings Disallow: /p/activity Disallow: /p/usage Disallow: /p/onboarding Disallow: /p/help Disallow: /*?tr_id= Disallow: /*?tr_lid= Disallow: /*?pc_uid= Disallow: /*?utm_ Disallow: /*?ref= # -------------------------------------------------------------------------- # Search engines — give Google/Bing explicit permission for public surfaces. # -------------------------------------------------------------------------- User-agent: Googlebot Allow: / Disallow: /api/ Disallow: /auth/ Disallow: /embed/ User-agent: Googlebot-Image Allow: / User-agent: Bingbot Allow: / Disallow: /api/ Disallow: /auth/ Disallow: /embed/ User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / User-agent: YandexBot Allow: / # -------------------------------------------------------------------------- # AI / LLM crawlers. # PayPeek explicitly allows answer-engine and RAG crawlers to index public # marketing, blog, and help content so the product is discoverable in AI # search results. Adjust if your content policy changes. # -------------------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: meta-externalagent Allow: / User-agent: FacebookBot Allow: / User-agent: Diffbot Allow: / User-agent: YouBot Allow: / User-agent: DuckAssistBot Allow: / User-agent: Amazonbot Allow: / User-agent: MistralAI-User Allow: / # -------------------------------------------------------------------------- # Sitemaps & LLM index # -------------------------------------------------------------------------- Sitemap: https://www.paypeek.ai/sitemap.xml # LLM-readable site summary: https://llmstxt.org/ # https://www.paypeek.ai/llms.txt