Posts tagged as:

Robots.txt

Robots.txt Meta Tag Versus X-Robots-Tag HTTP Header .htaccess

December 17, 2009

It has become common practice to implement robot.txt files to prevent the search engine spiders from crawling your webpage and having it show up in the search engine results page (SERP). Unfortunately, robots.txt does not prevent your site from showing up in the search engine results. That means your web page is passing on link [...]

Read the full article →