ITWissen.info - Tech know how online

robot exclusion standard (RES)

Many websites do not want to appear in the search engineresults pages

(SERP). To prevent this, websites can be marked in such a way that the robots accessing them recognise that the marked web page must not be captured. This is done with the RES marking, which stands for Robot Exclusion Standard

(RES). The behavior of the robots can be influenced by header entries and the ASCII file robots.txt in the rootdirectory of the web server. If the robots.txt file is stored in the web server, it prohibits the robots from accessing the web page.

Informationen zum Artikel
Englisch: robot exclusion standard - RES
Updated at: 15.11.2009
#Words: 123
Links:
Translations: DE