What Is Crawlability in SEO?

Technical SEO is the strange hybrid area of digital marketing that falls squarely between the roles of the marketer and the IT professional. However, the technical side of your search engine optimization is a key piece of the online marketing puzzle. To reach and remain at the top spot on the search engine rankings, you need to have your technical side running as seamlessly as your social media, your advertising campaigns, and your content.

Fortunately, if you outsource your digital marketing, then your SEO agency should be able to help you with the technicals. It will still benefit you to know what is going on behind the scenes though, so we have created this guide to one of the fundamentals of technical SEO: crawlability.

Want to learn more about the Digital Authority Partners approach to SEO? Watch this video!

What Is Crawlability?

This is the term used to describe how easy it is for a web crawler to index and ‘understand’ your website. So – the logical next question: what is a web crawler? A web crawler, also known as a spider bot, spider, or bot, is an automatic process that constantly and systematically browses the entirety of the world wide web. On its travels, the crawler will essentially copy the pages that you have published so that they can then be fully indexed and subsequently ranked on the search engine results pages.

In the simplest of terms, crawlability is the ease with which the search engine can access, navigate, and correctly identify and index each and every part of your site. So, in terms of technical SEO, it is the first and last thing you should be checking for – because if the Google bot (for example) cannot find your pages, they will never be ranked at all, let alone on top of page one.

How Often Are Sites Crawled?

This can differ. Some sites are crawled daily, others every couple of days, with some pages being crawled only once every six months. Clearly, you want to be in the ‘more frequently’ crawled category, particularly if you have made changes to your web pages and want to see how your ranking and traffic are affected.

There is some good news here, as the bots will typically check your site more frequently if you do make changes often – if after a few months of checking your site every day there has never been any change, the site crawler quite rightly allocates the resources elsewhere and simply drops in once in a while.

You can also request a web crawl from Google, although this may still take a few days to be completed. The quickest and simplest way to request a recrawl is by using the Google Search Console. Go to the test live URL function and then select request indexing.

Can Crawling Be Monitored?

It can! Returning to the trusty free Google tools, you can go to the Google Search Console and click on ‘Settings’ and then ‘Crawl Stats’. The report will show you how many times Google tried to crawl your site in the last 90 days (whether or not the crawl was completed) and the average time it took to crawl a page.

You can use these statistics to ensure that everything is running as it should – essentially, any major deviations from the norm could be an indicator that something is wrong and an adjustment needs to be made.

Next, use the ‘Host Status’ query to show if your web pages were successfully accessed and crawled. You will find three categories here, each of which can potentially mean that your site has major technical issues.

Firstly, you have the robots.txt failures, which indicates that there is an issue accessing your robots.txt file. Next, there is the DNS report, which tells you if your pages were not recognized or were non-responsive when the crawl occurred. Lastly, there is the server data – it shows if your server did not have the capacity to process the crawl request.

Can Crawling Be Prevented?

Absolutely. You can request that certain (or all) search engines leave your pages alone, simply by editing your robot.txt file. This is a standard file that each crawler is supposed to access before it accesses any other part of your website, and inside the file are the instructions that you want the robots to obey. Here you can state that you do not want a specific page crawling, for example.

How does this help your SEO? It may seem counterintuitive – surely you want all your content to be looked at and ranked by all of the search engines? Well, in a roundabout way, crawling certain pages could actually damage your ranking. For example, if you have several pages that you have designed and built for optimum positioning, and then a page that is auto-generated, or a contact page, which is not so well optimized, then your good pages will suffer by association.

Similarly, you may want to reduce the chances of your pages competing against each other (keyword cannibalization) by having main pages indexed and by having sub-pages ignored. Why? Sub-pages are more likely to be of value when accessed directly from their parent and therefore do not need to be ranked.

Of course, these settings can be accidentally (or maliciously) misused and the pages that you definitely want to be ranked on might not even be accessible by the bots. It pays to check your robot settings fairly frequently, particularly if you notice a sudden drop in traffic.

How Can the Results of Crawling Be Improved?

As mentioned previously, crawling is the process that underpins all other SEO. It stands to reason, then, that to improve the results of your site being crawled, you need to improve everything else in the digital marketing mix – your keyword use, your page speed, and your mobile optimization, for example.

The easiest, quickest, and most effective way to make this happen is to engage an experienced SEO agency like Digital Authority Partners. We have an award-winning track record of helping businesses of all sizes to reach their digital goals, and we can help you too.