You’re reading an article online and click on one of the links in the post only to find that the page that the article links to cannot be found. This is because of Broken Links and it’s SEO best practice to regularly check your website for such links and fix them.
In this post, I’ll show you how you can use Screaming Frog’s Free SEO Spider to find and fix these broken links/404s. But before we understand how to locate these, let us first understand what they are and why are they bad for SEO.
What are Broken Links?
A broken link directs the user to a web page that can’t be found or can’t be accessed on the server, for various reasons. These links are often called “dead links”, “HTTP 404” or “404 code”.
There are two types of Broken Links or 404s:
- Hard 404s: These are the regular type of “404 errors” that you typically find. It tells both the user and the search engines that the page does not exist.
- Soft 404s: A soft 404 occurs when a page displays “404” or “not found” error to the users but returns a 200 OK status to search engines. This makes the search engines assume that there is a page at that location and they waste time crawling such pages.
What can be the reason for Broken Links?
Broken links can occur due to a various number of reasons:
- The website owner has misspelled the URL.
- The website owner has recently deleted a page without setting up a redirect.
- The site is no longer available (is offline or has been moved to a new location).
- Links to content that has been moved or deleted.
Do Broken Links affect SEO?
Before we talk about whether or not broken links affect your website’s SEO, let’s look at them from the user’s perspective.
Directing users to pages that no longer exist is not a good user experience. And if the user runs into a lot of them on your site, they would probably never want to visit your site again.
Consider this scenario:
You have an events management business. A person looking to host an event lands on your website from Google. They go through the services that you provide and are ready to discuss business with you.
They click on the Contact Us button and are redirected to a page that does not exist. You just lost a potential customer due to a broken link.
Broken links can negatively impact your conversion rates. So it’s extremely important to find and fix them on a regular basis.
Broken Links or 404 errors are not a ranking factor but they can cause a drop in rankings.
How?
The more broken links your website has, the harder it will be for search engine crawlers to crawl your website. They also won’t be able to pass around link equity. This can negatively impact your website’s SEO.
If users leave your website after encountering a 404 – you can also have extremely high bounce rates.
How to use Screaming Frog to find Broken Links?
Screaming Frog SEO Spider is a website crawler that helps improve onsite SEO. The crawler helps in extracting and auditing website data to fix common SEO issues.
The free version of SEO Spider lets you crawl and analyse up to 500 URLs. So you can use this as a free broken link checker tool.
You can purchase a license and remove this limit or you can analyse your links in batches and make do with the free version.
You can download the Screaming Frog SEO Spider from here.
So lets see how to find out broken links in your website using Screaming Frog’s SEO Spider tool:
Step 1: Crawl your Website
Just enter your website’s URL in the search bar and click on Start to start crawling your website.
Step 2: Set Filters to View Broken Links
Once the crawling has finished, Click on Response Code at the top navigation and then from the filter select Client Error (4xx).
Step 3: Analyse the Broken Links
Now it’s time to start analysing your 404 errors. The thing that you’ll want to know to fix these errors is the source (which pages on your website have these broken links). To find these, simply click on the URL in the list and then select Inlinks in the bottom navigation.
In this example, you can see that 3 pages on my website (https://www.blog.nimitkapoor.com/deep-dive/pagespeed-insights-monitor/), (https://www.blog.nimitkapoor.com/bite-sized-info/organic-search-traffic-drops/) and (https://blog.nimitkapoor.com/deep-dive/off-page-seo-optimisation-2021/) link to a page (https://www.blog.nimitkapoor.com/deep-dive/off-page-seo-optimisation-2021/) that cannot be found.
Note: The last one is a redirect from non-www to www version.
I had moved this page to another URL (updated 2022 version) but forgot to change the links in the other posts that linked to this one. But imagine if there were 50 or 100 posts that had an internal link to this post. It’s extremely time-consuming to go to each of these posts and update the URL there.
The easiest way to fix these errors is to set up a 301 redirect (permanently moved). This type of redirect indicates that the page has permanently moved to another location and all the link equity of the old URL is passed on to the redirected page.
There are different ways to set up a 301 redirect, just jump to the section that works for you:
How to set up a 301 redirect using cPanel
If you have access to cPanel through your hosting, then you can add 301/302 redirects easily using cPanel. This will automatically add all the necessary code for the redirect to the htaccess file.
Step 1: Log in to your cPanel and click on Redirects
Step 2: Enter your old and new URL details.
Click on Add/Save and that’s it. Your redirect will start working.
How to set up a 301 redirect using htaccess
Find your .htaccess file and add the following code:
Redirect 301 /oldpage.html http://www.example.com/newpage.htmlReplace oldpage with the URL that you want to redirect and http://www.example.com/newpage.html with the URL that you want the oldpage to be redirected to.
For instance, we want to redirect https://www.blog.nimitkapoor.com/deep-dive/off-page-seo-optimisation-2021/ to https://www.blog.nimitkapoor.com/deep-dive/off-page-seo-optimisation-2022/ so the code will look something like this:
Redirect 301 /deep-dive/off-page-seo-optimisation-2021/ https://www.blog.nimitkapoor.com/deep-dive/off-page-seo-optimisation-2022/How to set up a 301 redirect on WordPress
Step 1: Download a Redirection Plugin
Just search for a Redirection plugin and install it.
Step 2: Open the Plugin Settings
Step 3: Set up the Redirection
Setting up the redirect is pretty straightforward. Just enter the URL that you want to redirect in the Source URL. And the URL that you want the old URL to be redirected to in the target URL.
How to set up a 301 redirect on Wix
It’s extremely easy to set up a 301 redirect on Wix.
Step 1: Just open your SEO Tools
Go to your SEO Tools section and select the site that you’re working on.
Step 2: Click on URL Redirect Manager
Step 3: Create a New Redirect
Click on Create New Redirect and just enter your Source (old) and Target (new) URLs.
Click on Save and it’s done! Your old URL will now redirect to the new URL.
How to set up a 301 redirect on Shopify
Step 1: Go to your Online Store’s Navigation
Click on Navigation and then Click on View URL Redirects.
Step 2: Click on Create URL redirect
Step 3: Enter your Old and New URL
Just enter your old and new URL and Save the Redirect.
Broken Links Best Practice
1. You don’t have to fix all your Broken Links
Broken links are natural and not all of them should be fixed. Big websites have thousands of broken links and it’s not worth spending time and effort in fixing all of them.
If you don’t have a relevant page that you can redirect the old URL to, then don’t do it. Just have a nice and helpful 404 error page that gives the user some guidance on what they can do next.
If you want some inspiration to create your 404 page, here’s a good post on that.
2. Don’t redirect all your Broken Links to the Homepage
It’s not recommended to redirect your Broken links to your homepage as it does not provide a good user experience.
3. Prioritise pages with high authority
If you have pages that have a lot of other pages linking to it, or the page itself has a high value, priortise fixing that first and then move on to your other broken links.
