1. Introduction

    In the ever-evolving world of SEO (Search Engine Optimization), getting your web pages indexed by Google is a crucial step toward achieving online visibility.

    However, there are instances when Google decides not to index certain pages. Understanding the reasons behind this can help you determine if action needs to be taken, or if this was the intended behaviour.

    In this blog post, we will explore various reasons why Google might not index specific pages, whether it is likely to be intentional, and if action should be taken.

  2. Google Search Console Reports

    If you are a verified owner of a website in Google Search console, you will regularly receive reports on page indexing.

    This will often include a list of reasons why certain pages have not been indexed. It is important to note that this is not necessarily an issue, as some pages are designed not to be indexed or Google have been specifically asked not to index said pages.

  3. VB Media Specific

    This blog post has been designed to be specific for the VB Media platform, and as such may discuss options or defaults that differ to other systems, platforms, and/or websites.

  1. Reasons that (probably) require no action.

  1. Robots.txt File

    "Your page is disallowed in robots.txt"

    The robots.txt file is a method used to control how search engines crawl and index a site.

    By specifying certain pages or sections of the site in the robots.txt file, website owners can instruct search engines not to visit or index those areas.

    VB Media automatically generates and serves your robots.txt file. In it, we instruct crawlers not to index certain pages or areas. Either they are not relevant for search engine results pages, or are too sensitive to crawl. Example:

    User-agent: * Disallow: /private/

    In this case, the /private/ directory and all its contents would be excluded from indexing.

    There is no action required for any pages not indexed for this reason, as Google have explicitly been asked not to do so.

  1. Canonically Linked Pages

    "Alternate page with proper canonical tag"
    "Your page has a canonical tag pointing to a different page"

    Canonical links play a crucial role in managing duplicate content and ensuring that the preferred version of a page is indexed.

    A canonical link is an HTML element that helps webmasters prevent duplicate content issues by specifying the "canonical" or preferred version of a webpage. Example:

    <link rel="canonical" href="https://example.com/preferred-page">

    When Google encounters a canonical link, it understands that the page in question is a duplicate or a variation of the specified canonical URL.

    Therefore, it will index the canonical page rather than the duplicate. This is the correct behaviour because it consolidates ranking signals and ensures that the best version of the content appears in search results.

    VB Media will serve all pages both with and without a trailing slash. If you are using a top level domain, we will also serve the page both with and without the leading www..

    For example, you can access this sites homepage at all of these locations.

    https://printhome.co.uk
    https://printhome.co.uk/
    https://www.printhome.co.uk
    https://www.printhome.co.uk/

    Although these are the exact same page, they are accessible via 4 unique URLs. We therefore canonically link all versions to https://printhome.co.uk.

    There is no action required for any pages not indexed for this reason, as Google have explicitly been asked not to do so.

  2. Read more on Canonical Links in our dedicated post here.

  1. Noindex Directive

    "Your page has a noindex meta tag"
    "Excluded by ‘noindex’ tag"

    One of the primary reasons Google may not index a page is the presence of a noindex directive. This directive is used in the HTML code of a webpage to tell search engines not to index the page. This can be implemented through a meta tag in the head of the HTML document:

    <meta name="robots" content="noindex">

    This tag will not appear by default but VB Media does allow for custom meta tags to be added in the page editor.

    If this has been added, it is likely on purpose to specifically ask Google not to index it. Just review if this is still the case and remove the tag if necessary.

  1. Currently Not Indexed

    "Crawled - currently not indexed"

    The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.

  1. Reasons to look into.

  1. Page does not exist (404s)

    "Not found (404)"

    If a requested page does not exist, a server will respond with a 404 Error, to indicate that the page could not be found. Google therefore have nothing to index, as essentially there is no page at that location.

    Whilst serving a 404 is the correct behaviour for the server, there may be an underlying issue at play that needs resolving.

    Firstly, identify where the link for the page comes from and try to amend said link at the source. For example, if you have a link in your footer that goes to a page that has since been removed, remove or edit that link. Common places to check are:

    • The navigation bar
    • The footer
    • Custom pages
    • Social media profiles
    • Business listings

    After you have conducted a link review, consider adding 301/302 redirects so that future requests for the page that does not exist are redirected to the new location, or a similar page should one exist.

  1. Redirects

    "Page with redirect"

    A redirect is when a request for a page does not respond with content, instead it instructs the requester to go to a different page.

    This is particularly useful if a page moves locations and you still have links to the old location (on social media for example). Instead of a visitor seeing a 404 page, they are redirected to a different URL.

    The redirect may be completely genuine, but it does mean that Google has nothing to index and will therefore show in your indexing reports.

    Whilst this is normal behaviour it is recommended that any internal links on your site go to a page that exists and not a redirected page. Conduct a link review on your site to ensure they go to the correct URL, some key areas to check are:

    • The navigation bar
    • Footer links
    • Links on custom pages

  1. Duplicate Content

    "Your page is a duplicate or near duplicate of a different page"

    Google strives to provide unique and relevant content in its search results. If a page contains content that is very similar or identical to other pages on the web, it might not be indexed to avoid redundancy.

    This includes content duplicated within the same site or across different sites. Ensuring that each page has unique, valuable content can help improve the chances of indexing.

  1. Low-Quality Content

    "The quality of your page is too low"

    Pages with low-quality content, such as "thin" content with little to no added value, might not be indexed by Google.

    This includes pages with only a few sentences, excessive advertisements, or content that is auto-generated without human oversight. Google aims to index pages that provide substantial value to users.

    Consider reviewing the following on your site:

    • Custom pages provide unique and valuable content
    • Similar products have different names and descriptions

  1. Crawl Budget Limitations

    "Your crawl budget is too low"

    Google allocates a specific crawl budget to each site, determining how many pages it will crawl and index within a given timeframe.

    Much larger sites with many pages might find that not all pages are indexed due to these limitations.

    If you have an abundance of low-quality pages on your site then consider removing those that do not provide the user with valuable content.

    If you believe your website has ongoing issues with the crawl budget, you should try to find the cause of this situation. An experienced SEO Specialist will definitely help you with that.

  1. 403 Errors

    "Blocked due to access forbidden (403)"

    A 403 Forbidden Error occurs when you do not have permission to access a web page or something else on a web server.

    VB Media serve a 403 error if the requester has been blocked by our Website Application Firewall (WAF). Whilst the WAF is in place to protect your site, our platform, and any data held within it. There can be "false positives" and may require an IP address to be white listed.

    If you see any pages not indexed because of this reason please contact our support team.

  1. Server Errors

    "Server error (5xx)"
    "Server error prevents Googlebot from crawling"

    When Google tries to crawl a web page, it sends a request to retrieve the page’s content. If the server encounters an issue, it will respond with a server error code, indicating that it could not provide the requested content.

    Googlebot usually interprets this as a temporary unavailability. However, consistent ongoing errors can be detrimental and as a result, some of your pages may not be indexed by the search engine.

    If you see a report of a 5xx error please send a list of all affected pages to our support team.

  1. Conclusion.

  1. Understanding why Google might not index certain pages can help you better optimise your site and ensure that your most important content is visible in search results. Also allowing you to concentrate efforts where they are needed most.

    If a page is not indexed because that is the desired behaviour, there is little to be concerned about.

    However, if a page that should be indexed has not been, then this will need further investigation and quite possibly some action is needed.