SpuuunBot

Who we are

SpuuunBot is the feed reader behind Spuuun, a service that groups news coverage into stories and shows readers which outlets reported each one, with links to the originals. Its requests carry this user agent:

SpuuunBot/1.0 (+https://spuuun.ai/bot)

What it fetches

From news publishers it requests two things only: the RSS or Atom feeds they publish, and their robots.txt. It never requests article pages. When your server sends an ETag or a Last-Modified date, we send it back on the next request, so a feed that has not changed costs you a short reply.

How often

We poll each feed about once every 60 minutes, and feeds on some shared publishing platforms less often, so that a platform hosting many sites is not asked for all of them at once.

robots.txt

We honour robots.txt. Before SpuuunBot first requests a feed from your site, it reads your robots.txt and follows the group written for User-agent: SpuuunBot if you have one, and otherwise the group for User-agent: *. If that group disallows your feed's address, we do not fetch it. Rules written only for other named crawlers do not cover SpuuunBot.

How we read the answer:

• If robots.txt answers 401 or 403, we treat your whole site as off limits.
• If it is missing (any other 4xx answer), there are no rules to follow, and we read the feed.
• If your server returns a server error (5xx), or does not answer within 8 seconds, we read the feed anyway.
• We keep the copy we read while our feed reader runs, so a change to your robots.txt can take a while to reach us. Email us if you need it to apply at once.

Opting out, and copyright

To stop SpuuunBot reading your feeds, add this to your robots.txt:

User-agent: SpuuunBot
Disallow: /

Or email support@spuuun.ai with the feed or site address, and we will stop. The same address takes copyright questions and takedown requests: tell us the page, the material and the rights you hold, and we will review it and reply.