{"id":1709,"date":"2024-10-23T17:32:25","date_gmt":"2024-10-23T15:32:25","guid":{"rendered":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/?page_id=1709"},"modified":"2025-01-29T12:45:37","modified_gmt":"2025-01-29T11:45:37","slug":"crawling-control","status":"publish","type":"page","link":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/en\/crawling-control\/","title":{"rendered":"Crawling control"},"content":{"rendered":"\n

SeznamBot fully complies with the robots exclusion standard<\/a> (or simply robots.txt<\/strong>), which specifies the rules  of robots behavior through a robots.txt<\/code> file. A robots.txt<\/code> file contains instructions that specify which content of the website the robots are \/ are not allowed to access and download. All robots visiting your web that follow this standard read this file first when accessing the web and they adjust their behavior according to the directives in the file. You can find detailed description of its syntax on the official website<\/a> of the standard.<\/p>\n\n\n\n

Using the robots.txt standard you can stop all crawling performed by SeznamBot on your web or only stop downloading of exactly specified pages. It typically takes several days for our crawler to recheck the restriction in the robots.txt<\/code> file and eventually update the index, though for some sites that are not visited often, it can take up to several weeks. In case you only want to stop indexing<\/em> of a page but allow SeznamBot to download and explore it, see Indexing Control<\/a>. In that case you should also allow SeznamBot to download the page in the robots.txt<\/code> file so that it is able to read the restrictions in the HTML code.<\/p>\n\n\n\n

\n

Tip<\/h2>\n\n\n\n

If you want to keep SeznamBot from accessing your site altogether, use the following directives in your robots.txt<\/code> file:<\/p>\n\n\n\n

\n
\n

User-agent: SeznamBot
Disallow: \/<\/code><\/p>\n<\/div>\n\n\n\n

<\/div>\n<\/div>\n<\/div>\n\n\n\n

Nonstandard Extensions of robots.txt Syntax Recognized By SeznamBot<\/h2>\n\n\n\n

On top of the official version 1.0 standard<\/a> SeznamBot recognizes other directives. These extensions are listed below in separate sections.<\/p>\n\n\n\n

Allow Directive<\/h2>\n\n\n\n

The syntax of the Allow directive is the same as with the standard Disallow directive, except for the name. Use of the directive explicitly allows robots‘ access to a given URL(s). This is useful when you want to instruct robots to avoid an entire directory but still want some HTML documents in that directory crawled and indexed.<\/p>\n\n\n\n

Examples<\/h4>\n\n\n\n
User-agent: *
Disallow:<\/code><\/td>		All robots can access and download all pages of the web. The empty space following the Disallow<\/code>\/Allow<\/code> directive means that the directive doesn’t apply at all. This is the default (empty or nonexistent robots.txt<\/code> file has the same effect).<\/td><\/tr>
User-agent: *
Allow:<\/code><\/td><\/tr>
User-agent: *
Disallow: \/ <\/code><\/td>		No robot can download any page.<\/td><\/tr>
User-agent: *
Disallow: \/archive\/
Disallow: \/abc<\/code><\/td>		No robot can enter the \/archive\/<\/em> directory of the website. Furthermore, no robot can download any page with a name starting with „abc<\/em>„.<\/td><\/tr>
User-agent: *
Disallow: \/
Allow: \/A\/
Disallow: \/A\/B\/<\/code><\/td>		All robots can download files only from the \/A\/<\/em> directory and its subdirectories except for the subdirectory B\/<\/strong>. The order of the directives is not important.<\/td><\/tr>
User-agent: SeznamBot
Disallow: \/ <\/code><\/td>		SeznamBot can’t download anything from the website. Other robots are allowed by default.<\/td><\/tr>
User-agent: SeznamBot
Disallow: \/discussion\/ <\/code><\/td>		SeznamBot can’t download the \/discussion\/<\/em> directory. Other robots are allowed by default.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Wildcards<\/h2>\n\n\n\n

You can use the following wildcards in a robots.txt<\/code> file:<\/p>\n\n\n\n

*<\/td>any number of any characters (an arbitrary string). Can be used multiple times in a directive.<\/td><\/tr>
$<\/td>the end of the address string<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Examples<\/h4>\n\n\n\n
User-agent: SeznamBot
Disallow: *.pdf$<\/code><\/td>		Disallow downloading all files with addresses ending with \u201e.pdf<\/em>\u201c (regardless of the characters preceding it).<\/td><\/tr>
User-agent: SeznamBot
Disallow: \/*\/discussion\/$<\/code><\/td>		Disallow downloading the default document in any of the \/discussion\/<\/em> subdirectories while still allowing downloading all other files in those subdirectories.<\/td><\/tr>
User-agent: SeznamBot
Disallow: \/discussion$<\/code><\/td>		Disallow \/discussion<\/em>, allowing \/discussion-01<\/em>, \/discussion\/-02 <\/em>etc.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Request-rate Directive<\/h2>\n\n\n\n

The Request-rate directive is used to tell robots how many documents from a website they can download during a given time period. Seznambot fully respects this directive, which enables you to set this download rate in a way that prevents your servers from being overloaded or even crushed. On the other hand, if you want your files to be processed by SeznamBot at a faster rate, you can set the Request-rate to a higher value.<\/p>\n\n\n\n

The request rate directive syntax is: Request-rate: <number of documents>\/<time><\/code><\/p>\n\n\n\n

You can also specify a time period in the day, during which the robot will observe the rate set by the directive. In the rest of the day, it will return to its regular behavior.<\/p>\n\n\n\n

The general syntax in this case is: Request-rate: <rate> <time of day><\/code><\/p>\n\n\n\n

Examples<\/h4>\n\n\n\n
Request-rate: 1\/10s<\/strong><\/code><\/td>Robots are allowed to download one document every ten seconds.<\/td><\/tr>
Request-rate: 100\/15m<\/strong><\/code><\/td>100 documents every 15 minutes<\/td><\/tr>
Request-rate: 400\/1h<\/strong><\/code><\/td>400 documents every hour<\/td><\/tr>
Request-rate: 9000\/1d<\/strong><\/code><\/td>9000 documents every day<\/td><\/tr>
Request-rate: 1\/10s<\/strong> 1800-1900<\/code><\/td>Robots are allowed to download one document every ten seconds between 18:00 and 19:00 (UTC). In other times there are no limits for download rate.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n

Caution<\/h2>\n\n\n\n

The minimum download rate for SeznamBot is 1 document every 10 seconds. If you specify a lower value, SeznamBot will interpret it as this minimum rate. The maximum rate is only limited by the current speed of SeznamBot.<\/p>\n<\/div>\n\n\n\n

User-agent: * \nDisallow: \/images\/ \nRequest-rate: 30\/1m \n# all robots except for SeznamBot and Googlebot: do not access \/images\/ directory,\n# rate 30 URLs per minute\n\nUser-agent: SeznamBot \nDisallow: \/cz\/chat\/ \nRequest-rate: 300\/1m \n# SeznamBot: do not access \/cz\/chat\/ directory, rate 300 URLs per minute \n\nUser-agent: Googlebot \nDisallow: \/logs\/ \nRequest-rate: 10\/1m \n# Googlebot: do not access \/logs\/, rate 10 URLs per minute<\/code><\/pre>\n\n\n\n
User-agent: * \nDisallow: \/ \n# all robots except for SeznamBot: do not access anything \n\nUser-agent: Seznambot \nRequest-rate: 300\/1m \n# Seznambot: access everything, rate 300 URLs per minute<\/code><\/pre>\n\n\n\n
Sitemaps<\/h2>\n\n\n\n
Sitemaps allow you to fine-tune the movement of SeznamBot around your web. Through a sitemap you can tell SeznamBot which pages change frequently, when a given page was last updated or what is its indexing priority within the site. Sitemaps are implemented through the Sitemap protocol which uses XML files that contain all information needed. You can find more information on sitemaps, including the exact syntax, on the official website sitemaps.org<\/a>.<\/p>\n\n\n\n
The sitemap directive syntax is: Sitemap: <absolute URL><\/code><\/p>\n","protected":false},"excerpt":{"rendered":"
SeznamBot fully complies with the robots exclusion standard (or simply robots.txt), which specifies the rules  of robots behavior through a robots.txt file. A robots.txt file contains instructions that specify which content of the website the robots are \/ are not allowed to access and download. All robots visiting your web that follow this standard read […]<\/p>\n","protected":false},"author":22,"featured_media":0,"parent":0,"menu_order":2,"comment_status":"closed","ping_status":"closed","template":"page-sidemenu","meta":{"_acf_changed":false,"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","footnotes":""},"page_category":[28],"page_tag":[],"service_status_category":[],"class_list":["post-1709","page","type-page","status-publish","hentry"],"acf":[],"lang":"en","translations":{"en":1709,"cs":73},"pll_sync_post":[],"_links":{"self":[{"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/pages\/1709","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/comments?post=1709"}],"version-history":[{"count":4,"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/pages\/1709\/revisions"}],"predecessor-version":[{"id":1992,"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/pages\/1709\/revisions\/1992"}],"wp:attachment":[{"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/media?parent=1709"}],"wp:term":[{"taxonomy":"page_category","embeddable":true,"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/page_category?post=1709"},{"taxonomy":"page_tag","embeddable":true,"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/page_tag?post=1709"},{"taxonomy":"service_status_category","embeddable":true,"href":"https:\/\/o-seznam.cz\/napoveda\/vyhledavani\/wp-json\/wp\/v2\/service_status_category?post=1709"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}