VeskonBot

Veskon lists websites the moment they launch. When we learn that a new site has appeared, our crawler visits it to read the public homepage (title, description, icon) for the directory.

What it requests

• /robots.txt, then the homepage (following redirects). Nothing else: no deep crawling, no forms, no logins.
• A visit when we first learn about a new site, plus an occasional recheck of sites already listed.
• If your server refuses the request (403, 429 or 503) we wait at least six hours, then one and three days, and stop after three tries. A Retry-After header is honored.
• If the site does not answer at all, we try a few more times over about three days, then stop.

How it identifies itself

Mozilla/5.0 (compatible; VeskonBot/1.0; +https://veskon.com/bot)

Requests come from the server that hosts veskon.com.

Opting out

We read robots.txt before every visit. To keep VeskonBot out of your whole site:

User-agent: VeskonBot
Disallow: /

Or write to info@ardeyazilim.com with your domain and we will stop visiting it.