Robots.txt Meta Tag Versus X-Robots-Tag HTTP Header .htaccess

It has become common practice to implement robot.txt files to prevent the search engine spiders from crawling your webpage and having it show up in the search engine results page (SERP). Unfortunately, robots.txt does not prevent your site from showing up in the search engine results. That means … [Read more...]