To stay away from unwanted material from the look for indexes, site owners can instruct spiders not to crawl certain data files or directories through the conventional robots.txt file in the basis directory with the domain. On top of that, a web site might be explicitly excluded from the internet search engine's databases by utilizing a meta tag pa