Sites that may be blocking AI bots by accident
Owners who inherited an old robots.txt or a security rule from a past migration. You get a list of which crawlers are allowed or blocked, and which rules to change.
AI crawler and technical audit
For businesses that want AI search tools to find, read and cite their pages. Our AI crawler access audit checks what bots can reach and what they can parse.
Get a free audit planTalk to our team
- Review of robots.txt and crawler permissions
- Check of indexability, rendering and page speed
- Written fix plan agreed before changes go live
Quick answer
An AI crawler access audit is a technical review of whether AI search bots can reach, render and read your website's pages. Pugmarks Media checks robots.txt rules, indexability, JavaScript rendering and page speed, then delivers a written fix plan. You agree the plan with us before any changes go live.
01Overview
What is AI crawler and technical audit?
An AI crawler and technical audit is a structured check of whether the automated bots behind AI search tools can reach your pages, load them and understand the content. An AI crawler access audit covers the technical layer that sits underneath any content or citation work. If bots cannot read a page, that page is unlikely to be used in AI-generated answers.
Crawlers are programs that fetch web pages so a search or AI system can process them. Each one identifies itself with a name, and your site can allow or block it. Many sites block bots by accident, through an old robots.txt rule, a firewall setting or a security plugin.
Here is a concrete example. A company runs a product catalog built with JavaScript. The pages look fine in a browser, but the product descriptions only appear after scripts run, and some bots do not run scripts. Those bots see an empty page, so the products are not read at all.
An audit finds problems like this by testing the site the way a bot sees it. We review the robots.txt file, which tells bots what they may fetch. We check indexability, meaning whether a page is allowed into a search index. We also test rendering and page speed, since slow or script-dependent pages are harder for bots to read.
The output is a written list of issues in priority order, with a recommended fix for each one. You review the plan with us, and nothing changes on your site until you agree. Our parent service, answer and generative engine optimization, shows where this audit fits in the wider work.
02Fit
Who AI crawler and technical audit is for
JavaScript-heavy sites and web apps
Teams whose content loads only after scripts run. We check whether bots see the full page or an empty shell, and give developers a written fix list.
Businesses after a redesign or migration
Owners whose pages changed URLs, templates or hosting. The audit checks indexability, redirects and canonical tags so important pages remain readable.
Brands missing from AI answers
Companies with good content that AI tools rarely cite. We rule technical causes in or out before you spend more on content.
Multi-location and large catalog sites
Retail, service and ecommerce sites with many similar pages. You get a review of crawl paths, duplicates and page speed, prioritized by page type.
In-house marketing teams without technical support
Teams that need a plain-language report. We write the fix plan and agree it with you before any changes go live. Contact us to start.
03Scope
What's included
- Robots.txt and permissions reviewWe read your robots.txt and server rules and list which AI crawlers are allowed or blocked. Owners get a clear view of which rules to keep, change or remove.
- Indexability and canonical checksWe check noindex tags, canonical tags, redirects and sitemaps so important pages can be found. This helps sites that changed URLs, templates or hosting.
- JavaScript rendering testWe compare what a bot receives with what a visitor sees. Teams with script-loaded content get a written list showing which sections bots may miss.
- Page speed and server responseWe review load times, server errors and timeouts that can cause crawlers to give up. Developers receive specific pages and issues to fix first.
- Written fix planWe deliver a prioritized document of fixes with the reason for each. You review and agree it with us before any changes go live.
- Monthly reportingWe report monthly on fixes completed and crawler access status. Business owners can see what changed and what remains open.
04Process
How our AI crawler and technical audit works
6 steps. You always know what's next.
The same path for every engagement.
1About 30–60 minutes (estimate)
Hold a discovery call
We ask which pages matter most, what has changed recently, and which bots you want to reach. You provide site access details and any known migration history. You receive an agreed audit scope.
21–2 days (estimate)
Review robots.txt and server rules
We read your robots.txt, server and security rules, then record which AI crawlers are allowed or blocked. You provide hosting or firewall access where needed. You receive a keep, change or remove list.
32–3 days (estimate)
Check indexability and canonicals
We test noindex tags, canonical tags, redirects and sitemaps on your priority pages. You provide a list of key URLs if you have one. You receive a list of pages bots may skip.
42–4 days (estimate)
Test rendering and page speed
We compare the raw HTML a bot receives with the page a visitor sees, and note slow templates. You provide developer contacts if scripts load content. You receive a list of missed sections.
52–3 days (estimate)
Write the fix plan
We rank every issue by effort and likely impact, with plain-language notes developers can act on. You review the draft and tell us about constraints. You receive a written fix plan.
6About 1 hour, plus approval time (estimate)
Agree and hand over the plan
We walk through the plan on a call and confirm what changes first. Nothing goes live until you approve it. You receive the final plan and a clear next step, with timings depending on site size.
05Deliverables
What you get
| Deliverable | What it is | Cadence | Starts in |
|---|---|---|---|
| Audit scope note | Agreed list of priority pages, recent site changes and target bots | One-time | Discovery call |
| Robots.txt and permissions report | Keep, change or remove list for crawler rules and server settings | One-time | Site audit |
| Indexability and canonical review | Findings on noindex tags, canonicals, redirects and sitemaps | One-time | Site audit |
| JavaScript rendering findings | Comparison of what bots receive and what visitors see, with a developer fix list | One-time | Site audit |
| Page speed review | Notes on load issues that may slow or block bot fetching | One-time | Site audit |
| Written fix plan | Prioritized actions and work plan, agreed with you before changes go live | One-time | After the site audit |
| Implementation of agreed fixes | Changes made to rules, tags or templates as set out in the plan | One-time | After plan approval |
| Progress report | Summary of fixes made, open items and next steps | Monthly | After the first fixes |
| Crawler access re-check | Repeat test of bot access after releases, migrations or rule changes | Quarterly | After the first fixes |
06Compare
AI crawler and technical audit: in-house vs agency vs Pugmarks
Swipe to compare →
| Criteria | In-house | Typical agency | Pugmarks |
|---|---|---|---|
| Time to start | Depends on team capacity and priorities | Depends on sales and onboarding process | Discovery call, then site audit |
| Specialization | Developers or SEO generalists, varies by team | Broad service mix, varies by agency | 25+ SEO specialists; AI crawler checks are one focus |
| Reporting | Internal updates, format set by your team | Varies by contract | Monthly reporting; written fix plan for the audit |
| Cost model | Salaries and tool subscriptions | Retainer or project fee | Depends on scope; see our pricing page or contact us |
| Risk | Gaps if no one owns bot access rules | Generic checklists may miss site-specific rules | Plan agreed with you before changes go live |
| Flexibility | High control, limited by staff time | Often fixed packages | Scope set on the discovery call and adjusted to your site |
07Cost
What does AI crawler and technical audit cost?
The cost of an AI crawler and technical audit depends on the size of your site, the number of pages and locations to check, and how complex your setup is, such as script-heavy pages or several domains. Market competitiveness can also change how many pages and bots we prioritize. We confirm scope on a discovery call, then send a quote. To start, contact our team or see our answer and generative engine optimization services.
Package prices are published so you can compare before you call.
See all packages11Years in business
08Proof
Results for clients like you
Only results our clients have approved for publication. No made-up logos or numbers.
FAQ
Frequently asked questions
Still have a question?
Ask us directly. We reply with honest next steps, even if that means we're not the right fit.
Ask our teamWhat is an AI crawler access audit?
An AI crawler access audit is a technical check of whether the bots behind AI search tools can reach, load and read your pages. We review robots.txt rules, server and security settings, indexability, JavaScript rendering and page speed, then document what is blocked and why.
How do I know if my website is blocking AI crawlers?
Start by reading your robots.txt file and checking for rules that disallow specific bots or all bots. Firewall, CDN and security settings can also block crawlers without appearing in robots.txt. An audit tests both layers and lists which crawlers are allowed or blocked.
Should I allow AI crawlers on my website?
It depends on your goals. If you want AI tools to find and cite your pages, allowing the relevant crawlers is usually necessary. Some businesses prefer to restrict certain content. We help you decide which bots to allow, then write rules that match that decision.
Can AI bots read JavaScript-heavy websites?
Some can, but many handle scripts poorly or not at all. If your content appears only after scripts run, a bot may receive an empty shell. Our rendering test compares what a bot receives with what a visitor sees and gives developers a written fix list.
How long does an AI crawler and technical audit take?
Timing depends on the size of your site and how quickly we get access to your accounts. Every engagement starts with a discovery call and a site audit. We then agree a work plan with you before any changes go live.
How much does an AI crawler audit cost?
Cost depends on the size of your site, the number of pages and locations to check, and how complex your setup is, such as several domains or script-heavy pages. We confirm scope on a discovery call, then send a quote. You can contact our team to start that conversation.
Will fixing crawler access get my pages cited in AI answers?
Not on its own, and no one can guarantee citations. Crawler access is the technical foundation: if bots cannot read a page, it is unlikely to be used. Content quality, authority and relevance still matter. Our answer and generative engine optimization work covers those layers.
Who needs an AI crawler access audit?
Sites that inherited old robots.txt or security rules, JavaScript-heavy sites and web apps, and businesses that recently redesigned or migrated are the usual candidates. Any business that wants AI search tools to read its pages can benefit from confirming that bots have access.
09Keep reading
Related services and guides
Related pages cover the content and citation side of AI search. Our AI search optimization, AI Overviews and AI visibility audit pages sit alongside this audit.
Get in touch
Talk to our team about your AI crawler access audit
Start with a discovery call to agree which pages and AI bots matter most. We then run the site audit and agree a written fix plan before any changes go live.
