For publishers: what our crawler does

Our user-agent

Every request we send identifies itself with this exact User-Agent header, which carries a contact URL back to this page:

How we discover pages

We read your RSS or Atom feed, falling back to your sitemap.xml filtered to recently-changed URLs. We never spider a site looking for links — we do not follow links from one of your pages to another looking for more content.

How often we fetch

We enforce a minimum interval of 2.5 seconds between two requests to the same host, and we send conditional requests (If-None-Match/ETag and If-Modified-Since) so a page that has not changed since our last visit costs you a small 304 response rather than a full page load.

robots.txt

We honour robots.txt. If your robots.txt is unreachable when we check it, we treat that as disallowing everything on your site until we can read it, rather than assuming permission.

What we store

• Recipe title
• Source URL
• Publisher name
• Published time
• A resized thumbnail image (fetched, resized and stored — not hot-linked)
• Ingredients
• Prep/cook/total times
• Yield/servings
• Nutrition information
• Diet/course/main-ingredient categories (a small set we derive by keyword match — not your original tag text)
• Rating value, rating count, and comment count

We also keep the underlying structured recipe markup you publish (your page's JSON-LD or microdata) nearly as-is, with instruction steps removed. We do this so we can reprocess a recipe later — for example, improve how we parse an ingredient — without fetching your page again, which is a politeness to you as much as a convenience to us. Because it is your own markup, it can include fields beyond the list above, such as an author bio, review text, or a video object, if your page embeds them on the same recipe block.

What we never store

We never store your instruction steps — not in the fields above, and not in the structured markup we retain either; they are stripped before anything is saved. Ingredient lists are largely factual; instructions are your creative expression, and the click-through to read them on your site is how you get paid. We link to your page for anyone who wants to cook the recipe — we do not reproduce the steps ourselves.

Opting out

Once you opt out, we never fetch anything from your site again — we do not fetch it and then filter the results afterward; the check happens before any request is sent. Visit your publisher page (replace example.com with your domain) to see what we currently have indexed and to submit a takedown request. Every request is recorded and reviewed by a person — nothing is actioned automatically from that form alone.

Reach us

The fastest way to reach us about your specific site is the takedown form on your publisher page. If we have not indexed anything from you yet, the same form still works and lets you tell us never to start.