robot exclusion standard (RES)

Many websites do not want to appear in the search engine results pages( SERP).

To prevent this, you can mark websites in such a way that the robots accessing them recognize that the marked website must not be captured. This is done with RES marking, which stands for Robot Exclusion Standard (RES).

The behavior of the robots can be influenced by header entries and the ASCII file robots.txt in the root directory of the web server. If the robots.txt file is stored in the web server, it prohibits the robots from accessing the web page.