menuOptimising for search engines
Search engine optimisation (SEO) is the process of ensuring that your website is properly indexed by search engines and consequently that it has the best chance of ranking well on relevant keywords. The goal is to have your site listed high in search results so that it is seen by more people, thus driving more traffic to your site.
It is beyond the scope of this site to go deep into SEO strategies, but there are a few basic principles that will help your site to play nicely with search engines. Many of these are covered elsewhere on this site as they also improve the user experience for your visitors which should always be your primary goal when building a website.
Navigation
Menu & breadcrumbs
Use a good, clear menu structure to help visitors and search engines to find their way around your site easily. In addition, breadcrumbs can help visitors to quickly go to the section's parent or root page.
Broken links
It should go without saying that all links on your site should work. Broken links cause frustration for visitors and block their access to information, and hinder search engines from crawling all the pages in your site.
A tool we use to check for broken links is Xenu Link Sleuth.
Descriptive link text
You might be tempted to use 'click here' links or simply display the URL. However, it is far more helpful for both visitors and search engines to use link text which describes where the link is going.
Meta tags
Use the description meta tag as it gives search engines a summary of what the page is about, and the summary might be shown to potential visitors in the search results.
Page titles
Use unique, descriptive page titles (via <title> tags) to tell both visitors and search engines the topic of the page. Page titles are shown prominently in search results, so using an appropriate title will make it easier for people to know whether your page will contain the information they are looking for.
Headings
Use heading tags to show how the content on your page is structured, and the hierarchy and relative importance.
Images
Use alt attributes to describe any images which contain content. This will help screen reader users as well as search engines to understand the non-text content of your site.
URLs
Use descriptive URLs rather than generic page names, and structure your content into directories which reflect the hierarchy of your site. It is also best practice to gather similar types of files, such as images, together in the same directory.
Restricting crawler access with robots.txt
If you have content that you do not want indexed, use a robots.txt file to tell search engines (including Funnelback, the ANU search engine) which parts of your site they should not crawl. While this can be particularly useful for development sites, be aware that robots.txt can be ignored by disreputable bots, so you should not rely on it in place of password protection.
ANU only sites
If your site is already restricted to ANU only (or similar) then search engines external to ANU (such as Google) will not have access to your pages.
Restricting access to all crawler bots
If you do not want any robots to visit your site (including Funnelback):
- Create a file called robots.txt
- Insert the following text:
User agent: * Disallow: /
- Save the file at the top (root) level of your website
Allowing access only to the ANU search engine
If you only want the ANU search engine to index your pages, and not other search engines:
- Create a file called robots.txt
- Insert the following text:
User-agent: anu-gsa-prodprimary-crawler Disallow: User-agent: * Disallow: /
- Save the file at the top (root) level of your website
In this example the ANU search engine (User-agent: anu-gsa-prodprimary-crawler) is allowed to search your whole site (because there are no values after Disallow). All search engines (User-agent: *) are not allowed to search anything on your site from the root folder downwards (Disallow: /).
Further information
There are more options for disallowing search engines such as specifying particular search engines, or only disallowing certain folders. There are a number of tutorials on this topic available on the web.
For example:
