Robots Meta Tag

Robot Meta Tag is used to give instructions to the web spiders. It is implemented on a page-to-page basis by incorporating a meta tag in the head of an HTML file. It was proposed in 1996 at the W3C workshop, and its use was described in December 1999 in Appendix B of HTML 4.01.

Robots’ meta tag plays an important role in on-page optimization as it is the robot’s meta tag that tells the search engines how to treat the content of a page.

Why are robots meta tags important for SEO?

These tags are mostly used to prevent pages from showing up in search engine result pages, although it can be used for various other purposes. There are many types of content that you may not want to be indexed by the search engine to improve the SEO of your site. Such as:

• Thin pages with little or no information for the users
• Admin and thank-you pages
• PPC landing pages
• Pages related to product launches, upcoming events, promotions, etc.
• Duplicate content if you are not using canonical tags to suggest the main version for indexing.

Basic Syntax: It has a very simple basic syntax.

So, a robot Meta tag consists of two attributes: name and content. You have to specify values for each of these attributes.

In the syntax, you can put different values as a placeholder as we have written “instructions to be followed” as a placeholder. The different values can be index, noindex, follow, nofollow, all, or none-thing.

Implementation and meaning of Robot meta tags:

This tag says, “Do not index the page and do not follow the links on the page.” It is used to instruct the web spider not to index the page and not to follow the links to other pages”. So, your page will be dropped out of the search index, and the rest of your page will not be crawled. This tag is mostly used when a site is in development mode, and the developers don’t want the spider to pick the page for indexing.

This tag says, “Index the page but do not follow the links on the page.” It instructs the spider to index the given page and not to follow the links to other pages. Thus, your page will be included in the search index, but the link path to other pages will not be created.

This tag says, “Do not index the page but follow the links on the page.” So, the spider will not index the given page but will follow the links to other pages, i.e., your page will not be included in the search index, but a link path to other pages will be established.

This tag says, “Index the page and follow the links on the page.” So, the spider will index the given page as well as follow the links to other pages, i.e., your page will be included in the search index, and a link path to other pages will be established.