‘X-Robots-Tag’ http header

X Robots Tag is used to control more data then Robots.txt file and Robots Meta Tag. The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any directive that can be used in an robots meta tag can also be specified as an X-Robots-Tag. Here’s an example of an HTTP response with an X-Robots-Tag instructing crawlers not to index a page:

X-Robots-Tag: noindex

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)

Multiple X-Robots-Tag headers can be combined within the HTTP response, or you can specify a comma-separated list of directives. Here’s an example of an HTTP header response which has a noarchive X-Robots-Tag combined with an unavailable_after X-Robots-Tag.

X-Robots-Tag: noarchive & unavailable_after

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noarchive
X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PST
(…)

The X-Robots-Tag may optionally specify a user-agent before the directives. For instance, the following set of X-Robots-Tag HTTP headers can be used to conditionally allow showing of a page in search results for different search engines:

X-Robots-Tag: googlebot

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow
(…)

Directives specified without a user-agent are valid for all crawlers. The section below demonstrates how to handle combined directives. Both the name and the specified values are not case sensitive.

October 30, 2019

0 responses on "'X-Robots-Tag' http header"

Leave a Message

Your email address will not be published. Required fields are marked *

Award winning
Digital Marketing Institute
in India

Seven Boats Academy (A Unit of Seven Boats Info-System Private Limited) is an award-winning digital marketing institute located in Kolkata India, offering digital marketing courses for job seekers, working professionals, and entrepreneurs. This digital marketing institute has set a new benchmark for digital marketing education in India by training over 100K+ students in digital marketing through their online, classroom, corporate training and workshops. Seven Boats provides digital marketing solutions tailored to each student’s requirements while ensuring they acquire the latest industry skills that are required to kickstart their digital career. With their committed team of experienced digital professionals by their side and interactive teaching pedagogy, it’s no wonder Seven Boats has become one of the premier digital marketing institutes in India.

Seven Boats Academy Centres

Nagerbazar

Unit No.304, Diamond Arcade 1/72, Cal, Jessore Rd, Kolkata, West Bengal 700055, India

Phone: 08017049042
Secondary phone: 09674937499
Email: [email protected]

Khardah

P.O, 1095, Lodh House, Arunachal, Khardaha, Rahara, Kolkata, West Bengal 700118, India

Phone: 08017049042
Secondary phone: 09766470193
Email: [email protected]

Bhowanipore

SHIVAYAN, Flat No. G-2, 41B, Suhasini Ganguly Sarani, Bhowanipore, Kolkata, West Bengal 700025, India

Phone: 08017049042
Email: [email protected]

Baranagar

14/4/1, Behari Lal Paul St, Baranagar, Kolkata, West Bengal 700036, India
Phone: 08017049042
Email: [email protected]

top
Copyright © 2011-2024 Seven Boats Academy. All rights reserved.
Login / Sign up
Download Brochure
WhatsApp Chat

Alert: Scammers are impersonating & pretending to be Seven Boats Info-System Private Limited. Be Aware. Learn How to Protect Yourself.

Read more →