Articles

Guide to Ensuring a Sub‑Domain Gets Indexed and Crawled Correctly

Inventiva Creative Studio™ World leaders in AI, Systems Automation & Digital Marketing 5cd34b2a Bad8 4408 86eb E61e640f2beb

Guide to Ensuring a Sub‑Domain Gets Indexed and Crawled Correctly

Illustration of a search engine crawler exploring interconnected pages

Contents

Introduction

Owning a website with multiple sub‑domains adds flexibility to your online presence. A sub‑domain can host a dedicated shop, a customer portal or a corporate services site. For these pages to rank in search results, search engines must be able to discover, crawl and index them. When a sub‑domain does not appear in Google, it often means that something is preventing crawlers from accessing or indexing its pages. This guide explains how search engines work and provides step‑by‑step instructions for diagnosing and fixing indexing issues on WordPress sites running the Flatsome theme with the WP Rocket caching plugin.

Who should read this

This eBook is aimed at site owners and web managers who want to ensure that a sub‑domain is visible to search engines. It assumes basic familiarity with WordPress but no advanced technical knowledge. Each task is broken down into clear, numbered actions so you can follow along and verify each step.

Understanding crawling and indexing

Before addressing your indexing problem, it is helpful to understand what happens when a search engine discovers a web page. A crawler (also called a spider or robot) fetches pages, follows links and interprets instructions from the site. If the crawler is allowed to access the page and finds no instruction to exclude it, the page will be stored in the search engine’s index. The index is a vast catalogue of web pages that search engines query to return results to users. Some of the key mechanisms involved are:

  • Robots.txt – a plain‑text file at the root of your domain that tells crawlers which paths they can and cannot fetch. Google’s documentation notes that the rules in robots.txt apply only to the host, protocol and port where the file is hosted. This means the robots file for your main domain cannot block pages on your sub‑domain.
  • Meta robots tag and X‑Robots‑Tag header – instructions embedded in the HTML <head> or sent via HTTP header that tell search engines whether to index a page. A noindex tag prevents a page from being indexed.
  • Canonical tags – these tags tell search engines which version of similar pages to index. If a page’s canonical tag points to a different domain, the crawler may drop the current page and index the canonical target instead.

Understanding these mechanisms will help you diagnose why your sub‑domain is not appearing in search results. The following sections walk through each area in turn.

Step 1 – Confirm that the sub‑domain is publicly accessible

  1. Open a private browsing window or use a browser where you are not logged in.
  2. Type the sub‑domain URL (for example, https://pro.company.com/) and press Enter.
  3. Ensure that a normal page loads and not a login screen. If you see a WordPress login form, your site is gating visitors, and crawlers cannot see the content. You should change the URL settings or membership plugin that is causing the redirection so that public pages remain open.
  4. If you are using WP Rocket, clear caches after any change to ensure the latest version of the page is served. In the WordPress dashboard, go to Settings → WP Rocket and click Clear Cache.

Step 2 – Check the robots.txt file

  1. Visit https://subdomain.example.com/robots.txt in your browser, replacing the example with your sub‑domain. For example, if your sub‑domain is pro.company.com, the file is at https://pro.company.com/robots.txt.
  2. Look for any rule that might block search engines. A rule such as Disallow: / blocks the entire site. Remember that the robots file on your main domain does not affect the sub‑domain.
  3. Confirm that important directories like /wp-content/ and /wp-includes/ are not disallowed. Google needs to fetch CSS and JavaScript files to render pages properly. A safer robots.txt might look like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /readme.html
Disallow: /license.txt
Disallow: /xmlrpc.php
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: */disclaimer/*
Disallow: *?attachment_id=
Disallow: /trackback/
Disallow: /feed/
Sitemap: https://subdomain.example.com/sitemap_index.xml
  1. If you make changes to robots.txt, save the file and upload it to the root of the sub‑domain via FTP or your hosting file manager. Clear your caches with WP Rocket to ensure crawlers see the updated file.

Step 3 – Inspect WordPress search engine visibility settings

  1. Log in to your sub‑domain’s WordPress dashboard.
  2. Navigate to Settings → Reading.
  3. Locate the Search Engine Visibility checkbox labelled Discourage search engines from indexing this site. WordPress documentation explains that enabling this box modifies your robots.txt file and adds a meta tag telling search engines not to index any content. Ensure the checkbox is unchecked.
  4. Click Save Changes if you modified the setting.
  5. Purge caches via WP Rocket to serve the updated pages.

Step 4 – Remove noindex directives

  1. View the source of a representative page by right‑clicking the page and choosing View Page Source.
  2. Search for <meta name="robots" or noindex. An entry like <meta name="robots" content="noindex"> tells search engines not to index the page.
  3. If present, determine which plugin or theme is adding it. Common sources include SEO plugins (Yoast SEO, Rank Math, All in One SEO) or maintenance modes.
  4. In your SEO plugin settings, ensure each page or post is set to Index. For example, in Yoast SEO, open the page editor, scroll to the Advanced tab and confirm that Allow search engines to show this Page in search results? is set to Yes.
  5. Check your server’s HTTP response headers for X‑Robots‑Tag: noindex. You can use the curl command in a terminal:
curl -I https://subdomain.example.com/your-page/
  1. If the header appears, remove it in your server configuration or plugin. For Apache, you might find a line like this in .htaccess:
Header set X-Robots-Tag "noindex, nofollow"
  1. Comment out or remove that line and reload your server.
  2. After removal, clear your cache via WP Rocket and re‑check the page source. If the noindex tag remains, trace through your theme and plugins for custom code.

Step 5 – Verify canonical tags

  1. In the page source, look for a line that starts <link rel="canonical" href="..." />. Canonical tags tell search engines which version of a page is the main one.
  2. The href value should match the current sub‑domain URL exactly. If it points to the main domain or to another site, search engines will index the canonical target and ignore the sub‑domain page.
  3. In your SEO plugin, enable self‑referencing canonical tags. Most plugins generate these automatically, but check that the Canonical URL field in the page editor matches the sub‑domain.
  4. Avoid using cross‑domain canonical tags unless you intend to consolidate content. Fix any misconfigured canonical entries in theme templates or custom code.
  5. Clear caches in WP Rocket after making changes.

Step 6 – Create and submit a sitemap

  1. If you use an SEO plugin, enable the XML sitemap feature. For example, in Yoast SEO go to SEO → General → Features and switch XML Sitemaps to On.
  2. Visit https://subdomain.example.com/sitemap_index.xml to confirm the sitemap is accessible.
  3. Sign in to Google Search Console. Add a Domain or URL prefix property for your sub‑domain. Google treats sub‑domains as separate properties, so you must verify it separately.
  4. In Search Console, open Sitemaps in the left menu, enter the sitemap URL (for example https://subdomain.example.com/sitemap_index.xml) and click Submit.
  5. Check the Coverage report to see which pages are indexed, excluded or blocked.

Step 7 – Improve discovery through internal and external links

  1. Link to your sub‑domain from your main domain using descriptive anchor text. This signals to Google that the sub‑domain is part of your site.
  2. Include links from your sitemap pages to each important page on the sub‑domain. Avoid orphan pages that are not linked from anywhere.
  3. Secure relevant external links by sharing the sub‑domain content on social media, submitting to industry directories and collaborating with partners. External backlinks help crawlers discover and prioritise your pages.

Step 8 – Clear and optimise caching with WP Rocket

WP Rocket improves site speed by caching pages, minifying assets and optimising delivery. However, if changes are made to robots.txt, meta tags or canonical tags, the cache may still serve old content. Follow these steps to ensure WP Rocket does not interfere with indexing:

  1. After editing settings, click WP Rocket → Dashboard → Clear Cache. This removes cached copies of pages.
  2. Under File Optimisation, uncheck Combine JavaScript Files or Delay JavaScript Execution if they break page rendering. Search engines need to see the fully rendered page to index it correctly.
  3. Use Exclude Files from Delay to ensure analytics scripts or SEO plugin scripts are not deferred.
  4. In Preload, enable Activate Preloading so the cache is refreshed automatically after changes. Preloading ensures that crawlers see the latest version of your pages.
  5. Test your site’s pages with Google’s URL Inspection Tool to confirm that the correct version is served. If the rendered HTML still shows outdated tags, clear your browser cache and WP Rocket cache again.

Step 9 – Monitor with Google Search Console

  1. Open URL Inspection and enter a page URL from your sub‑domain. Google will report whether the page is indexed and highlight any issues (for example, Excluded by robots.txt, Alternate page with proper canonical tag, Crawled – currently not indexed).
  2. For each excluded page, note the reason and return to the relevant step in this guide to fix it.
  3. Use the Page Indexing report to see aggregated reasons for exclusion across your site. Address the most common problems first.
  4. Re‑request indexing from the URL Inspection page after you fix an issue. Google will recrawl the page more quickly.

Advanced troubleshooting and best practices

Sometimes a sub‑domain remains unindexed despite following the standard steps. In such cases, consider the following:

  • Server misconfiguration – If your server serves the wrong content or redirects incorrectly, Google may pick a different canonical. Check your host’s configuration and ensure the sub‑domain points to the correct directory.
  • Duplicate content – If your sub‑domain contains the same content as the main domain, Google may choose one canonical and drop the other. Provide unique, high‑quality content for each domain.
  • Hreflang and multilingual sites – If you serve multiple language versions, implement hreflang tags correctly to avoid confusion about regional versions.
  • Security and access control – Ensure that firewall rules, IP restrictions or password protection do not block crawlers. Keep a staging site separate from the live sub‑domain.
  • Malicious or injected code – Rarely, hackers may add cross‑domain canonical tags or meta directives. Regularly scan your site for malware and keep plugins and themes up to date.

Conclusion

Getting a sub‑domain indexed is not a one‑time task but an ongoing process. By understanding how search engines interpret robots instructions, meta tags and canonical links, you can avoid common pitfalls that hide your content from users. The steps in this guide help you verify that your pages are accessible, allowed to be crawled and marked as indexable. Regularly monitor Google Search Console for feedback, maintain clear internal linking and update caches through WP Rocket whenever you make changes. With patience and attention to detail, your sub‑domain will earn its rightful place in search results.

Further Reading

  1. Google’s robots.txt specification:contentReference[oaicite:0]{index=0}
  2. WordPress “Discourage search engines” option:contentReference[oaicite:1]{index=1}
  3. Blocking indexing with noindex tags:contentReference[oaicite:2]{index=2}
  4. Fixing canonicalisation issues:contentReference[oaicite:3]{index=3}
  5. Canonical links and malicious canonical tags:contentReference[oaicite:4]{index=4}

Leave a Reply