Separate access, indexing and performance problems. Trace one important URL through the system before turning a crawler’s warning list into development work.
Start with the affected business pages
Identify the symptom and the pages that matter: a service page has disappeared, a redesign changed URLs, or organic landing pages are returning errors. Record when the change began and what was released around that time. A crawler can find many warnings; it cannot decide which ones threaten the business without that context.
At Keyforge, technical findings are part of the agreed SEO service scope. The output should name the issue, evidence, affected pages, priority, owner and acceptance test. Advice that stops at “fix technical SEO” leaves the implementation decision with the client.
Collect a small set of reliable evidence
Use live HTTP responses, a crawl of relevant routes, Search Console and rendered page checks where needed. Compare the public page with the intended template. Server logs and deployment history can add useful detail when access is available. Treat unavailable data as a limit, not a reason to invent a diagnosis.
Keep the original response and configuration before changing routing or templates. A repair needs a rollback path, especially when the same system serves commercial pages, editorial content and forms.
Separate crawling from indexing
Crawling is fetching a URL. Indexing is processing and selecting content for the search index. A working page can remain unindexed, and a crawl block does not necessarily remove a known URL from search. Check the specific stage before choosing a control.
| Control or signal | Main purpose | Important limit |
|---|---|---|
| robots.txt disallow | Ask compliant crawlers not to fetch a route | Google cannot read a page-level noindex rule on a blocked page. |
| noindex | Ask search engines not to index an accessible page | The crawler must be able to access the directive. |
| Canonical | Indicate the preferred version of duplicate or similar content | It is a signal; Google can select a different canonical. |
| 301 or 308 redirect | Move visitors and crawlers to a permanent destination | The destination should be relevant and work correctly. |
| 404 or 410 status | Report content that is unavailable or removed | Returning 200 with an error message sends a conflicting signal. |
Google’s technical requirements explain basic eligibility, while its robots meta documentation covers page and HTTP indexing directives. Meeting the requirements does not guarantee indexing.
For a missing commercial page, check the final status, redirect path, robots access and indexing directive before rewriting its copy. Use Search Console’s inspected URL and selected canonical when account access is available.
Verify redirects and error responses
Follow a URL from the first request to the final response. Record each redirect and check whether host, HTTPS and trailing-slash normalization create unnecessary hops. A permanent move should reach the intended replacement directly where practical. Avoid loops and redirects to irrelevant pages.
A custom error design needs to return the right status. A branded missing-page template that responds with 200 can look healthy in a browser while sending an incorrect success signal. Equally, an actual service page returning 404 needs investigation even if its content appears on screen.
A verified example from Keyforge’s own site
During our October 15, 2026 site check, unknown URLs returned HTTP 404 but displayed the previous WordPress design. The redesigned template existed; the helper that selected it was missing. We installed the helper, cleared the edge cache and rechecked the public response.
The acceptance test had two parts: the new design had to appear, and the HTTP status had to remain 404. We also checked a HEAD request and existing content routes. The repair corrected the error-page presentation; it did not restore content at other missing URLs.
Make canonical and sitemap signals agree
Review the canonical on the live page and compare it with the preferred URL, redirect destination, internal links and sitemap entry. Conflicting signals can make a migration harder to interpret. A template that points every page at the homepage deserves immediate attention.
Use canonical annotations for duplicate or very similar content, not to merge unrelated pages. They do not replace a redirect when visitors need to move to a new address. Google’s canonicalization documentation explains the relative signals and why consistency matters.
Keep the sitemap selective
List canonical, indexable URLs you want discovered. Remove accidental errors and obsolete redirect destinations. A submitted sitemap is a discovery aid, not an instruction that every listed page must be indexed. Check the live XML and its referenced children after a release.
For a redesign, map old URLs before moving them. Preserve established addresses when the content still fits. Where a move is necessary, identify the equivalent replacement and check redirects, internal links and sitemap entries as a single release task.
Check the page search engines can access
The content in a CMS editor is not always the content served publicly. A cache can retain an old template. A script can fail before it inserts important text. A consent tool can hide a resource the page depends on. Compare the response HTML with the rendered page and investigate any difference that changes its meaning.
Check that important navigation uses working links and that core content does not depend on a click before it exists. For JavaScript-heavy pages, inspect rendered output using the available search tools and browser evidence. Google’s JavaScript SEO guidance describes the rendering process and common constraints.
Test representative routes, including the exceptions
A homepage check is insufficient when the failure affects a template or rewrite rule. Sample a service page, article, archive, pagination page and a missing URL as appropriate. Include query or preview behavior when the change touches it. A fix should not break editing access, forms or content that intentionally stays on another system.
If the rendered page is present but lacks the information a buyer needs, the next task is content and page review. Our on-page SEO guide covers that work.
Measure page experience before choosing a fix
Core Web Vitals describe loading, responsiveness and visual stability through Largest Contentful Paint (LCP), Interaction to Next Paint (INP) and Cumulative Layout Shift (CLS). Good thresholds are LCP at or below 2.5 seconds, INP at or below 200 milliseconds and CLS at or below 0.1, evaluated at the 75th percentile of page loads. See web.dev’s Core Web Vitals reference.
Distinguish field data from a lab test. Field data reflects measured user experiences over its reporting window. A local or synthetic test helps diagnose a particular load under defined conditions. A high lab score does not prove that real users have good performance, and low-traffic pages may have no field data available.
Fix the resource or behavior behind the symptom
For slow loading, inspect the main content resource, server response and request dependencies. For delayed interactions, investigate work on the main thread and the interaction itself. For layout shifts, look for missing media dimensions, late content and unstable interface elements. Retest the affected behavior after the change.
Performance work belongs alongside accessibility and usability. Do not hide core content, remove useful interactions or break a form to improve a score. If the work needs template or asset changes, coordinate it with the website design and development scope.
Give each finding an owner and an acceptance test
Prioritize by effect on important pages, confidence in the diagnosis and the effort or dependencies involved. A sitewide accidental noindex can outweigh dozens of minor metadata warnings. A single error on a high-value service page can also deserve early work. Severity depends on the actual site.
| Finding | Proposed action | Acceptance evidence |
|---|---|---|
| Important service URL returns 404 | Restore the intended route or map a confirmed equivalent replacement | Correct status, content and destination on the live URL. |
| Template canonical points to the wrong page | Correct the canonical source and check other template instances | Live canonical, sitemap and internal links agree. |
| Opening image delays main content | Investigate delivery, dimensions and loading behavior | Repeatable diagnostic improvement; field results reviewed when available. |
| Obsolete URLs remain in a sitemap | Update the sitemap generator or source list | Live XML lists the intended canonical, indexable URLs. |
A developer needs a reproducible URL, observed behavior, expected behavior and a test. Identify who approves the change and who owns the cache, CMS or hosting layer. An independent SEO review can help when several systems or teams are involved.
After deployment, test the public route again and record the result. Monitor relevant indexing and search data over time. A correct technical release creates the conditions for search access; it does not guarantee a particular ranking or recovery date.
Finish with a release check
- Important URLs return the intended status and content.
- Redirects reach relevant working destinations without loops.
- Robots directives and canonical annotations match the indexing plan.
- Sitemaps and internal links use the intended URLs.
- Mobile content, navigation and enquiry paths remain usable.
- Backups, implementation owners and post-release evidence are recorded.
The value of a technical audit is the work it makes possible. A short list of verified, owned fixes can be more useful than a large export of unexplained warnings. For delivery through an agency, our SEO outsourcing service can be scoped around those implementation and review responsibilities.