We’ve run over 100 technical audits this year.
Through this we’ve gained deep insights into how technical structure impacts a website’s performance in search.
This article will highlight the most common technical SEO issues we encounter and which have the largest impact on organic traffic when corrected.
1. Mismanaged 404 errors
This happens quite a bit on eCommerce sites. When a product is removed or expires, it’s easily forgotten and the page “404s”.
Although 404 errors can erode your crawl budget, they won’t necessarily kill your SEO. Google understands that sometimes you HAVE to delete pages on your site.
However, 404 pages can be a problem when they:
Are getting traffic (internally and from organic search)
Have external links pointing to them
Have internal links pointing to them
Have a large number of them on a larger website
Are shared on social media / around the web
The best practice is to set up a 301 redirect from the deleted page into another relevant page on your site. This will preserve the SEO equity and make sure users can seamlessly navigate.
How to find these errors
Run a full website crawl (SiteBulb, DeepCrawl or Screaming Frog) to find all 404 pages
Check Google Search Console reporting (Crawl > Crawl Errors)
How to fix these errors
Analyze the list of “404” errors on your website
Crosscheck those URLs with Google Analytics to understand which pages were getting traffic
Crosscheck those URLs with Google Search Console to understand which pages had inbound links from outside websites
For those pages of value, identify an existing page on your website that is most relevant to the deleted page
Setup "server-side" 301 redirects from the 404 page into the existing page you've identified - If you are going to use a 4XX page - make sure that page is actually functional so it doesn’t impact user experience
2. Website migrations issues
When launching a new website, design changes or new pages, there are a number of technical aspects that should be addressed ahead of time.
Common errors we see:
Use of 302 (temporary redirect) instead of 301 (permanent) redirects. While Google recently stated that 302 redirects pass SEO equity, we hedge based on internal data that shows 301 redirects are the better option
Improper setup of HTTPS on a website. Specifically, not redirecting the HTTP version of the site into HTTPS which can cause issues with duplicate pages
Not carrying over 301 redirects from the previous site to new site. This often happens if you're using a plugin for 301 redirects - 301 redirects should always be setup through a website's cPanel
Leaving legacy tags on the site from the staging domain. For example, canonical tags, NOINDEX tags, etc. that prevent pages on your staging domain from being indexed
Leaving staging domains indexed. The opposite of the previous item, when you do NOT place the proper tags on staging domains (or subdomains) to NOINDEX them from SERPs (either with NOINDEX tags or blocking crawl via Robots.txt file)
Creating “redirect chains” when cleaning up legacy websites. In other words, not properly identifying pages that were previously redirected and moving forward with a new set of redirects
Not saving the force www or non www of the site in the .htaccess file. This causes 2 (or more) instances of your website to be indexed in Google, causing issues with duplicate pages being indexed
How to find these errors
Run a full website crawl (SiteBulb, DeepCrawl or Screaming Frog) to get the needed data inputs
How to fix these errors
Triple check to make sure your 301 redirects migrated properly
Test your 301 and 302 redirects to make sure they go to the right place on the first step
Check canonical tags in the same way and ensure you have the right canonical tags in place
Given a choice between canonicalizing a page and 301 redirecting a page - a 301 redirect is a safer, stronger option
Check your code to ensure you remove NOINDEX tags (if used on staging domain). Don’t just uncheck the options the plugins. Your developer may have hardcoded NOINDEX into the theme header - Appearance > Themes > Editor > header.php
Update your robots.txt file
Check and update your .htaccess file
3. Website speed
Google has confirmed that website speed is a ranking factor - they expect pages to load in 2 seconds or less. More importantly, website visitors won’t wait around for a page to load.
In other words, slow websites don’t make money.
Optimizing for website speed will require the help of a developer, as the most common issues slowing down websites are:
Large, unoptimized images
Poorly written (bloated) website code
Too many plugins
Heavy Javascript and CSS
How to find these errors
Check your website in Google PageSpeed Insights, Uptime.com, GTMetrix or Pingdom
How to fix these errors
Hire a developer with experience in this area (find out how FTF can help)
Make sure you have a staging domain setup so website performance isn’t hindered
Where possible make sure you have upgraded PHP to PHP7 where you use Wordpress or a PHP CMS. This will have a big impact on speed.
4. Not optimizing the mobile User Experience (UX)
Google’s index is officially mobile first, which means that the algorithm is looking at the mobile version of your site first when ranking for queries.
With that being said, don’t exclude the desktop experience (UX) or simplify the mobile experience significantly compared to the desktop.
Google has said it wants both experiences to be the same. Google has also stated that those using responsive or dynamically served pages should not be affected when the change comes.
How to find these errors
Use Google’s Mobile-Friendly Test to check if Google sees your site as mobile-friendly
Check to see if "smartphone Googlebot" is crawling your site - it hasn't rolled out everywhere yet
Does your website respond to different devices? If your site doesn't work on a mobile device, now is the time to get that fixed
Got unusable content on your site? Check to see if it loads or if you get error messages. Make sure you fully test all your site pages on mobile
How to fix these errors
Understand the impact of mobile on your server load
Focus on building your pages from a mobile-first perspective. Google likes Responsive sites and is their preferred option for delivering mobile sites. If you currently run a standalone subdirectory, m.yourdomain.com look at potential impact of increased crawling on your server
If you need to, consider a template update to make the theme responsive. Just using a plugin might not do what you need or cause other issues. Find a developer who can scratch build responsive themes
Focus on multiple mobile breakpoints, not just your brand new iPhone X. 320px wide (iPhone 5 and SE) is still super important
Test across iPhone and Android
If you have content that needs “fixing” - flash or other proprietary systems that don’t work on your mobile journey - consider moving to HTML5 which will render on mobile - Google web designer will allow you to reproduce FLASH files in HTML
5. XML Sitemap issues
An XML Sitemap lists out URLs on your site that you want to be crawled and indexed by search engines. You’re allowed to include information about when a page:
Was last updated
How often it changes
How important it is in relation to other URLs in the site (i.e., priority)
While Google admittedly ignores a lot of this information, it’s still important to optimize properly, particularly on large websites with complicated architectures.
Sitemaps are particularly beneficial on websites where:
Some areas of the website are not available through the browsable interface
Webmasters use rich Ajax, Silverlight or Flash content that is not normally processed by search engines
The site is very large, and there is a chance for the web crawlers to overlook some of the new or recently updated content
When websites have a huge number of pages that are isolated or not well linked together
Misused “crawl budget” on unimportant pages. If this is the case, you’ll want to block crawl / NOINDEX
How to find these errors
Make sure you have submitted your sitemap to your GSC
Also remember to use Bing webmaster tools to submit your sitemap
Check your sitemap for errors Crawl > Sitemaps > Sitemap Errors
Check the log files to see when your sitemap was last accessed by bots
How to fix these errors
Make sure your XML sitemap is connected to your Google Search Console
Run a server log analysis to understand how often Google is crawling your sitemap. There are lots of other things we will cover using our server log files later on
Google will show you the issues and examples of what it sees as an error so you can correct
If you are using a plugin for sitemap generation, make sure it is up to date and that the file it generates works by validating it
If you don’t want to use Excel to check your server logs - you can use a server log analytics tool such as Logz.io, Greylog, SEOlyzer (great for WP sites) or Loggly to see how your XML sitemap is being used
6. URL Structure issues
As your website grows, it’s easy to lose track of URL structures and hierarchies. Poor structures make it difficult for both users and bots navigate, which will negatively impact your rankings.
Issues with website structure and hierarchy
Not using proper folder and subfolder structure
URLs with special characters, capital letters or not useful to humans
How to find these errors
404 errors, 302 redirects, issues with your XML sitemap are all signs of a site that needs its structure revisited
Run a full website crawl (SiteBulb, DeepCrawl or ScreamingFrog) and manually review for quality issues
Check Google Search Console reporting (Crawl > Crawl Errors)
User testing - ask people to find content on your site or make a test purchase - use a UX testing service to record their experiences
How to fix these errors
Plan your site hierarchy - we always recommend parent-child folder structures
Make sure all content is placed in its correct folder or subfolder
Make sure your URL paths are easy to read and make sense
Remove or consolidate any content that looks to rank for the same keyword
Try to limit the number of subfolders / directories to no more than three levels
7. Issues with robots.txt file
A Robots.txt file controls how search engines access your website. It’s a commonly misunderstood file that can crush your website’s indexation if misused.
Most problems with the robots.txt tend to arise from not changing it when you move from your development environment to live or miscoding the syntax.
How to find these errors
Check your site stats - i.e. Google Analytics for big drops in traffic
Check Google Search Console reporting (Crawl > robots.txt tester)
How to fix these errors
Check Google Search Console reporting (Crawl > robots.txt tester) This will validate your file
Check to make sure the pages/folders you DON'T want to be crawled are included in your robots.txt file
Make sure you are not blocking any important directories (JS, CSS, 404, etc.)
8. Too much thin content
It’s no longer a good idea to crank out pages for “SEO” purposes. Google wants to rank pages that are deep, informative and provide value.
Having too much “thin” (i.e. less than 500 words, no media, lack of purpose) can negatively impact your SEO. Some of the reasons:
Content that doesn't resonate with your target audience will kill conversion and engagement rates. Google's algorithm looks heavily at content quality, trust and relevancy (aka having crap content can hurt rankings)
Too much low quality content can decrease search engine crawl rate, indexation rate and ultimately, traffic
Rather than generating content around each keyword, collect the content into common themes and write much more detailed, useful content.
How to find these errors
Run a crawl to find pages with word count less than 500
Check your GSC for manual messages from Google (GSC > Messages)
Not ranking for the keywords you are writing content for or suddenly lose rankings
Check your page bounce rates and user dwell time - pages with high bounce rates
How to fix these errors
Cluster keywords into themes so rather than writing one keyword per page you can place 5 or 6 in the same piece of content and expand it.
Work on pages that try and keep the user engaged with a variety of content - consider video or audio, infographics or images - if you don’t have these skills find them on Upwork, Fiverr, or PPH.
Think about your user first - what do they want? Create content around their needs.
9. Too much irrelevant content
In addition to “thin” pages, you want to make sure your content is “relevant”. Irrelevant pages that don't help the user, can also detract from the good stuff you have on site.
This is particularly important if you have a small, less authoritative website. Google crawls smaller website less than more authoritative ones. We want to make sure we’re only serving Google our best content to increase that trust, authority and crawl budget.
Some common instances
Creating boring pages with low engagement
Letting search engines crawl of "non-SEO" pages
How to find these errors
Review at your content strategy. Focus on creating better pages as opposed to more
Check your Google crawl stats and see what pages are being crawled and indexed
How to fix these errors
Remove quotas in your content planning. Add content that adds value rather than the six blogs posts you NEED to post because that is what your plan says
Add pages to your Robots.txt file that you would rather not see Google rank. In this way, you are focussing Google on the good stuff
10. Misuse of canonical tags
A canonical tag (aka "rel=canonical") is a piece of HTML that helps search engines decipher duplicate pages. If you have two pages that are the same (or similar), you can use this tag to tell search engines which page you want to show in search results.
If your website runs on a CMS like WordPress or Shopify, you can easily set canonical tags using a plugin (we like Yoast).
We often find websites that misuse canonical tags, in a number of ways:
Canonical tags pointing to the wrong pages (i.e.pages not relevant to the current page)
Canonical tags pointing to 404 pages (i.e., pages that no longer exist)
Missing a canonical tag altogether
eCommerce and "faceted navigation"
When a CMS create two versions of a page
This is significant, as you're telling search engines to focus on the wrong pages on your website. This can cause massive indexation and ranking issues. The good news is, it's an easy fix.
How to find these errors
Run a site crawl in DeepCrawl
Compare “Canonical link element” to the root URL to see which pages are using canonical tags to point to a different page
How to fix these errors
Review pages to determine if canonical tags are pointing to the wrong page
Also, you will want to run a content audit to understand pages that are similar and need a canonical tag
11. Misuse of robots tags
As well as your robots.txt file, there are also robots tags that can be used in your header code. We see a lot of potential issues with this used at file level and on individual pages. In some cases, we have seen multiple robots tags on the same page.
Google will struggle with this and it can prevent a good, optimized page from ranking.
How to find these errors
Check your source code in a browser to see if robots tag added more than once
Check the syntax and don’t confuse the nofollow link attribute with the nofollow robots tag
How to fix these errors
Decide how you will manage/control robots activity. Yoast SEO gives you some pretty good abilities to manage robot tags at a page level
Make sure you use one plugin to manage robot activity
Make sure you amend any file templates where robot tags have been added manually Appearance > Themes >Editor > header.php
You can add Nofollow directives to the robots.txt file instead of going file by file
12. Mismanaged crawl budget
It’s a challenge for Google to crawl all the content on the internet. In order to save time, the Googlebot has a budget it allocates to sites depending on a number of factors.
A more authoritative site will have a bigger crawl budget (it crawls and indexes more content) than a lower authority site, which will have fewer pages and fewer visits. Google itself defines this as “Prioritizing what to crawl, when and how much resource the server hosting the site can allocate to crawling.”
How to find these errors
Find out what your crawl stats are in GSC Search Console > Select your domain > Crawl > Crawl Stats
Use your server logs to find out what the Googlebot is spending the most time doing on your site - this will then tell you if it is on the right pages - use a tool such as botify if spreadsheets make you nauseous.
How to fix these errors
Reduce the errors on your site
Block pages you don’t really want Google crawling
Reduce redirect chains by finding all those links that link to a page that itself is redirected and update all links to the new final page
Fixing some of the other issues we have discussed above will go a long way to help increase your crawl budget or focus your crawl budget on the right content
For ecommerce specifically, not blocking parameter tags that are used for faceted navigation without changing the actual content on a page
13. Not leveraging internal links to pass equity
Internal links help to distribute "equity" across a website. Lots of sites, especially those with thin or irrelevant content tend to have a lower amount of cross-linking within the site content.
Cross-linking articles and posts help Google and your site traffic moves around your website. The added value of this from a technical SEO perspective is that you can pass equity across the website. This helps with improved keyword ranking.
How to find these errors
For pages you are trying to rank, look at what internal pages link to them. This can be done in Google Analytics - do they have any internal inbound links?
Run an inlinks crawl using Screaming Frog
You will know yourself if you actively link to other pages on your site
Are you adding internal nofollow links via a plugin that is applying this to all links? Check the link code in a browser by inspecting or viewing the source code
Use the same small number of anchor tags and links on your site
How to fix these errors
For pages you are trying to rank, find existing site content (pages and posts) that can link to the page you want to improve ranking for, and add internal links
Use the crawl data from Screaming Frog to identify opportunities for more internal linking
Don’t overcook the number of links and the keywords used to link - make it natural and across the board
Check your nofollow link rules in any plugin you are using to manage linking
14. Errors with page “on page” markup
Title tags and metadata are some of the most abused code on websites and have been since Google has been crawling websites. With this in mind, site owners have pretty much forgotten about the relevance and importance of title tags and metadata.
How to find these errors
Use Yoast to see how well your site titles and metadata work - red and amber mean more work can be done
Keyword stuffing the keyword tag - are you using keywords in your keyword tag that do not appear in the page content?
Use SEMRush and Screaming Frog to identify duplicate title tags or missing title tags
How to fix these errors
Use Yoast to see how to rework the titles and metadata , especially the meta description, which has undergone a bit of a rebirth thanks to Google increase of character count. Meta description data used to be set to 255 characters, but now the average length it is displaying is over 300 characters - take advantage of this increase
Use SEMrush to identify and fix any missing or duplicate page title tags - make sure every page has a unique title tag and page meta description
Remove any non-specific keywords from the meta keyword tag
Bonus: Structured data
With Google becoming more sophisticated and offering webmasters the ability to add different markup data to display in different places within their sites, it is easy to see how schema markup can get messy. From:
Map data
Review data
Rich snippet data
Product data
Book Reviews
It is easy to see how this can break a website or just get overlooked as the focus is elsewhere. The correct schema markup data can in effect allow you to dominate the onscreen element of a SERP.
How to find these errors
Use your GSC to identify what schema is being picked up by Google and where the errors are. Search Appearance > Structured Data - If no snippets are found this means the code is wrong or you need to add schema code
Use your GSC to identify what schema is being picked up by Google and where the errors are. Search Appearance > Rich Cards - If no Rich Cards are found this means the code is wrong or you need to add schema code
Test your schema with Google’s own markup helper
How to fix these errors
Identify what schema you want to use on your website, then find a relevant plugin to assist. All in One Schema plugin or RichSnippets Plugin can be used to manage and generate a schema.
Once the code is built, test with Google Markup helper below
If you aren’t using Wordpress, you can get a developer to build this code for you. Google prefers the JSON-LD format so ensure your developer knows this format
Test your schema with Google’s own markup helper
Wrapping it up
As search engine algorithms continue to advance, so does the need for technical SEO.
If your website needs an audit, consulting or improvements, contact us directly more help.