All the popular Search Engines on the Internet have agreed on using a text file provided by each website which instructs them on which part of the website they should index and which they should not. This text file is called robots.txt.
In this post we are going to talk about Joomla’s robots.txt specifically
What is the robots.txt
The robots.txt file is a file which contains a set of rules for the Search Engines. Each rule contains a folder path and whether this folder path is Allowed or Disallowed to be indexed.
Although you should know that not all Search Engines respect this file. So keep in mind that you should not use this file to hide information.
What does a Joomla robots.txt contain
As you can see in the code below, a Joomla robots.txt file contains a rather large set of folders which are Disallowed for the Search Engines to be indexed.
# If the Joomla site is installed within a folder
# eg www.example.com/joomla/ then the robots.txt file
# MUST be moved to the site root
# eg www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to all of the
# paths.
# eg the Disallow rule for the /administrator/ folder MUST
# be changed to read
# Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
The line User-agent: * basically says that the rules below are to be followed by ALL the Search Engines.
The set of folders that are disallowed are basically folders which hold Joomla’s code and have nothing to do with your content. Therefore they hold no valuable information for the Search Engines anyway. That’s why it is wise to Disallow any access to them and to keep the search engines results for your website, relevant only to your content.
How to change it from Joomla's global configuration
Joomla’s Global Configuration contains a way to quickly change the basic functionality of your robots.txt file. Let’s see what is available.
First you need to navigate to Joomla’s Global Configuration like in the screenshot below
Then, at the bottom of your screen there will be a section called Metadata Settings and in there a selectbox called Robots like in the screenshot below
As you can see, this selectbox holds 4 options which are basically all the combined options of Index / No index and Follow / No Follow.
The most common value of this option is the Index, Follow which instructs the Search Engines to crawl through the whole website and index everything.
The question is, that does Index and Follow mean?
Index: Indexing means that the Search Engine will use any part of your website in it’s search results.
Follow: Follow means that the Search Engine will use any link provided by any part of your website to traverse further into your Website and therefore a create a tree-like structure of it.
Generally you would use No Follow if your website had only one level and you wouldn’t want any Search Engine to use any links in that page which may or may not lead to a completely different website.
Finally, once you’ve chosen the choice which suits you, just go ahead and Save.
What to do if you recently updated from Joomla 3.3
Joomla 3.3 and lower versions had a different robots.txt which was using a different set of instructions. After the release of Joomla 3.4 those instructions have been changed to conform with the newest changes in the Search Engines and Joomla’s folder structure.
The problem is that the automated update process does not change the robots.txt file and therefore you should manually grab a new a copy of a robots.txt from the latest Joomla version and use that instead.
That was it! If you are a fun of our tutorials for Joomla Beginners, you can go ahead and subscribe in our Newsletter and follow us on Twitter to get notified first about new posts!
