Put simply, unavailable_after tags send a signal to search engines letting them know that the contents of a page should not be crawled after a certain date. Unlike “noindex” tags which are directives, the “unavailable_after” tag is a hint instead that could technically be ignored. However, if it is followed, this means that a web crawler such as Googlebot will not crawl the page as frequently, allowing the more important pages to take up the majority of the site’s crawl budget. Since the crawler could still visit the page occasionally, the page could remain in the index.
It is important to note that unavailable after tags should not be used for evergreen pages. Just because the content of a page is not changing, does not mean that it should not be crawled regularly. This tag should only be used for pages that are not important anymore and should not be ranking in the search results.
In this tutorial, we will go over when to use unavailable_after tags, how to implement them using the RankSense Cloudflare app, and how to track the efficiency of the tags with RankSense.
Suppose you have an events page on your website, linking to a different event page every week. Perhaps you post job listings that are only available for a month. Pages like these, showcasing a limited-time event, are great examples of pages that you would not want Google to crawl frequently after a certain date.
In these cases, implementing an unavailable_after tag would be the perfect solution as the page will not be crawled as often and could remain in the index for anyone who happens to search for the exact content on the page. (Note that these pages should be moved to an “archived” part of the website so that they are not on any main pages as they are no longer important.)
Implementing “Unavailable_After” Using the RankSense SEO Tool
Step 1: Installation and rule sheets
If you do not have the RankSense app installed, you can do so by following the steps in this article. If you are not familiar with creating custom “rules” or SEO changes using the RankSense app, I recommend that you read this article.
Step 2: Filling out your rule sheet
Unavailable_after tags are implemented on the “Meta Robots Tags” rule sheet. On this Google sheet, you will see a column for the URL, the robots tag, and comments.
In the “URL” column, simply paste the URL for the page that you would like to implement the unavailable_after tag for. In the “robots” column we will specify the “unavailable_after” tag and the exact date for the tag to take action.
These are the ways the date can be formatted:
Specifying the exact time and timezone is optional. If the user specifies a time but does not list a timezone, the rule will default to the user’s timezone. If the user does not specify a time or a timezone, the default time will be used.
Here are some examples of ways to format the unavailable_after tag rule:
Once the rule sheet is complete, navigate to “Settings>SEO rules” on the RankSense Dashboard and click the “+ File” button to add your rule sheet. Paste the URL of your Google sheet and add the appropriate tags.
(Note: We recommended that users publish to staging first to make sure that the changes are applied properly)
Using the RankSense SEO tool to verify your changes
To verify whether the unavailable_after tags are working as intended, simply open up the RankSense Cloudflare app and scroll to the bottom of the “Dashboard” tab until you see the “Crawler statistics” section. The crawler statistics provide you with information regarding every URL that Google Bot, or your search engine of preference, has recently crawled on your page.
Under “Latest pages crawled”, you can see the URL path and the “Last crawl” of that page. After the date that you set the unavailable_after tag for has passed, which can be checked by using our RankSense Chrome Extension, the page should not be crawled as frequently. This will let you know the tags are working as intended.
You can even use the search tool to filter for specific URLs to check their last crawl date. You can filter these URLs by selecting the tags you applied to them when importing the rules. This way you do not need to go through all of the pages to find the ones with the unavailable_after tags.
We learned that:
Unavailable_after tags are used to send a signal to search engines to let them know which pages are no longer important and should therefore not be crawled often.
The web crawler may still visit the page occasionally.
Limited time event pages are great examples of pages that should contain an unavailable_after tag once they are no longer important.
You can apply unavailable_after tags to pages by filling out a rule sheet and importing it through the RankSense Cloudflare app.
It is easy to verify your changes are working by using the “Crawler statistics” section of the RankSense Dashboard.
As we continue to improve the RankSense app for Cloudflare, we are always working to make the app more intuitive and easy to use. I'm pleased to share that we have made significant changes to our SEO rules interface in the settings tab of our app. It is now easier to publish multiple rules sheets and to see which changes have not yet been published to production.
For the following Ranksense Webinar, we were joined by Antoine Eripret, who works at Liligo as an SEO lead. Liligo.com is a travel search engine which instantly searches all available flight, bus and train prices on an exhaustive number of travel sites such as online travel agencies, major and low-cost airlines and tour-operators. In this...
In this RankSense Webinar, we were joined by Charly Wargnier who is a member of the Streamlit Creators Program. Streamlit is a Python open-source library used to make data apps quickly. Charly takes us on his Python journey from starting out in digital marketing with image editing software like Photoshop all the way to working...