TutusooBot

TutusooBot crawler policy

TutusooBot builds the public web index used by TutuSoo. It respects robots.txt by default, refuses private-network targets and limits response size and pages per crawl task.

Security overview Back to search

Identify our crawler

Production requests use the User-Agent below. Publishers can target TutusooBot specifically or use the common User-agent: * rules.

TutusooBot/0.1 (+https://tutusoo.com/bot)

Allow or block crawling

A standard robots.txt rule is enough. Changes take effect on a later recrawl; previously indexed pages that become disallowed are removed.

User-agent: TutusooBot
Disallow: /private/
Crawl-delay: 5

Polite access and resource limits

Requests are spaced per site, declared Crawl-delay values are read, and oversized pages, redirects and sitemaps are bounded.

  • The default delay between pages is one second.
  • Crawl-delay is respected with a safety cap against permanently occupied jobs.
  • Per-page content, sitemap depth and pages per task are limited.