Free SEO, AEO and GEO website crawler
Silkpath is a free website crawler that visits every page of a site and checks it for technical SEO, AEO (answer engine optimization) and GEO (generative engine optimization). It reports status codes, indexability, titles, headings, links, images, structured data, AI crawler access and llms.txt, page by page, with a site health score and PDF, Excel and CSV exports.
How it works
Four steps from a web address to a report you can hand to a client.
Type a website address and press Start Crawl. Queue several domains at once, or open Crawl Options to set page limits, speed, robots.txt, sitemap pages, asset download and your own extraction rules.
The crawler follows every link from the home page the way a search engine does, reads each page's HTML, respects robots.txt and retries pages that fail for a moment. Watch it live in the Scan Queue.
The report opens with a site health score out of 100, then every issue, worst first. Each section has filter chips with counts, search and sortable columns.
Save PDF, Excel or CSV, share a file through a link, download the site's HTML as a ZIP, compare with the last crawl, or schedule the next one.
The report
Sample data from a made-up coffee shop site. Pick a view, then click the numbers, chips, issues and pages to see how a report works.
Report for
harbourroasters.example
Site health 40 out of 100 (Poor): 125 URLs with errors, 289 with warnings and 286 with notices, up 1 point since the last crawl.
Every issue found, worst first: Validation Errors on 43 pages, Missing Return Tag on 30, Broken Internal Links on 24, 4xx Client Error on 18, Missing H1 on 8, Missing Title on 5.
A map of how the pages link together, from the home page out, with an issue log that lights up the pages that have each issue.
AEO checks per page: JSON-LD, a direct answer in the first paragraph, an author byline, FAQ or HowTo schema and question-style headings.
GEO checks: GPTBot is blocked in robots.txt, the firewall test passed, llms.txt was found and the entity markup is linked.
Compared with the crawl two weeks earlier: health 39 to 40, 30 issues new or worse, 8 fixed or better, 9 URLs added and 4 removed.
Every number above comes from a real kind of check in Silkpath. Run one on your own site.
Audit Your SiteFeatures
Each URL is checked once per section. Filters with nothing to show are hidden, so you only see what needs work.
How healthy is the site, at a glance?
Is every page found, reachable and allowed in search?
Does every page say clearly what it is about?
Do the links, images and files all work?
Can search engines and social sites read the page data?
Can answer engines and AI tools reach, read and quote the site?
What did the crawler do, and what else should it look for?
What changed since the last crawl?
Silkpath reads raw HTML and does not run JavaScript. If a site builds its pages in the browser, or a firewall answers with a "prove you are human" page, the report says so at the top and in every export, so you never trust numbers about the wrong page.
AI search
AEO, answer engine optimization, makes pages easy for answer boxes, featured snippets and voice assistants to quote. A page is ready when it opens with a direct answer, uses question-style headings, carries FAQ or HowTo schema and names its author.
GEO, generative engine optimization, makes sure AI tools such as ChatGPT, Perplexity, Gemini and Google AI Overviews can reach, read and cite a site. That means AI crawlers allowed in robots.txt and through the firewall, content in the raw HTML, an llms.txt file and clear entity schema.
Read more: What is AEO · What is GEO · AI crawler checker · llms.txt checker
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlWhat Silkpath reports
Site health
Simple on purpose. Each URL counts once, however many checks it fails. A URL with an error costs a full point, a URL with only warnings costs a third of a point, and notices cost nothing. The score is 100 minus those points as a share of all URLs crawled.
AEO and GEO checks are listed with the issues but never change the score.
Try it: a site of 1,000 URLs
Functions
From the first crawl to the hundredth: options, exports, history, schedules and settings.
Set it up once
Max pages per domain, parallel requests (1 to 8), delay between requests, domains at once (1 to 5), respect robots.txt, check external links, fetch image sizes and crawl sitemap pages.
Crawl limits, thin content and heavy image thresholds, storage, dark, light or system theme, 12 or 24 hour time, and a reset to defaults.
The crawler server can be locked with a password, so only people its owner trusts can start crawls. Lock Now locks the app at once.
Crawls that look after themselves
Progress is saved while a crawl runs. If the server restarts, the crawl carries on from the last save by itself.
Pick a date and time and the crawl runs by itself, even while the app is closed.
Crawl a site again every 1 to 365 days and keep the latest 1 to 50 results, so health over time builds itself.
Take the report anywhere
Export what you are looking at: the whole report from the Overview, one tab, or one section exactly as filtered. Branded PDFs with a cover and contents, formatted Excel sheets, plain CSV.
Turn a PDF, Excel or CSV file into a link with your own name, lasting 1 to 365 days. Delete a link and it stops working at once.
Download every page's HTML as a ZIP, optionally with the site's CSS, JavaScript, images, fonts and media (up to 499 MB a crawl) and links rewritten for offline browsing.
Every crawl kept, nothing lost
Every crawl is saved and listed by website, with health, errors, warnings and indexable share. Open, rescan, compare, export a summary or delete.
Crawls are kept in shared cloud storage, so every device shows the same History. Without it, crawls stay in the browser.
Deleted crawls wait 30 days before they are erased. Restore one exactly as it was, on every device.
Find anything, check more
Press / or Ctrl+K on any screen and jump straight to any feature, setting or button.
A two-part illustrated guide inside the app, with search: features and functions, then every column and filter explained.
Optional PageSpeed Insights with Core Web Vitals, Google Safe Browsing for the site and its links, and AI visibility with Gemini.
| Limit | Default | What it controls |
|---|---|---|
| Pages per crawl | Up to 10,000 | The free server's maximum. Set a lower limit per domain in Crawl Options. |
| Crawl depth | No limit | How many clicks from the start page to follow. |
| Parallel requests | 3 (1 to 8) | Pages fetched at once. Lower is gentler on the site. |
| External links checked | 3,000 | Status of links to other websites. 0 skips the check. |
| Images checked | 3,000 | Status, size and format of images. |
| Script and style files | 500 | Status and size of JavaScript and CSS files. |
| Wait for each page | 20 seconds (up to 60) | Slower pages are marked as failed. Short failures are retried twice. |
| Largest page read | 8 MB | Bigger pages are cut off at this size. |
| Website assets | 499 MB a crawl | Asset download for the ZIP stops at this total. |
The app
Install Silkpath on Windows, macOS, Android, iPhone or iPad, with its own window and icon. It opens fast, even offline. On iPhone, tap Share, then Add to Home Screen.
Start a long crawl and walk away. When it ends you get a notification, even with Silkpath closed. Tap it and the report opens. It also tells you when a crawl stopped or failed.
Notifications belong to the device. Turn them on or off in Settings, App and Notifications. On iPhone and iPad they need iOS 16.4 or later.
Honest comparison
Silkpath is a free, browser-based alternative to Screaming Frog SEO Spider for most audits, with AEO and GEO built in.
| Silkpath | Screaming Frog SEO Spider | |
|---|---|---|
| Where it runs | In the browser, installable | Desktop app |
| Price | Free | Free up to 500 URLs, paid licence beyond |
| AEO and GEO checks | Built in: AI crawlers, firewall test, llms.txt, FAQ schema | With custom extraction |
| Crawl history and compare | Automatic, with a Recycle Bin | With a licence |
| Notifications on your phone | Yes | No |
| JavaScript rendering | No, reads raw HTML and warns | Yes, with a licence |
| Settings and integrations | The essentials | Very many |
Screaming Frog and SEO Spider are trademarks of Screaming Frog Ltd. Silkpath is not affiliated with Screaming Frog. Read the full comparison.
Questions
Silkpath is a free website crawler for SEO, AEO and GEO audits, created by Raj Meena. Enter a website address and it visits every page, reports technical problems page by page with a site health score, and exports the report as PDF, Excel or CSV.
Yes. Silkpath is free to use, with nothing to install. Its crawler server runs on free hosting plans.
Status codes and redirects, security, indexability, canonicals, XML sitemaps, page titles, meta descriptions, headings, content and duplicates, mobile and accessibility basics, internal and external links, images, scripts, structured data, Open Graph, hreflang, and AEO and GEO signals such as AI crawler access and llms.txt.
AEO (answer engine optimization) makes pages easy for answer boxes and voice assistants to quote. GEO (generative engine optimization) makes sure AI tools such as ChatGPT, Perplexity, Gemini and Google AI Overviews can reach, read and cite a site.
Up to 10,000 pages per crawl on the free crawler server. You can set a lower limit per domain, a crawl depth and other limits before a crawl.
No. Silkpath reads each page's raw HTML, the way most crawlers and AI tools first see it. A site that builds its pages in the browser looks nearly empty, and Silkpath warns about it so you can check it with a rendering crawler.
Yes. Respect robots.txt is on by default for every crawl. Turn it off only on a site you own.
PDF, Excel and CSV for one section, one tab or the whole report, share links for those files, and a ZIP of every page's HTML with optional CSS, JavaScript, images, fonts and media.
Yes. Silkpath installs as an app on computers and phones. Turn on notifications and each crawl you start sends a notification when it ends, even with Silkpath closed. On iPhone and iPad, add Silkpath to the Home Screen first (iOS 16.4 or later).
The crawler server can be protected with a password, so only the people its owner trusts can start crawls. If you need access, contact Raj Meena on LinkedIn.
Silkpath was created, designed and built by its founder, Raj Meena.
Silkpath runs in the browser and is free, with no install, and includes AEO and GEO checks such as llms.txt, AI crawler access and FAQ schema. Screaming Frog is a mature desktop crawler with more settings and JavaScript rendering, and a paid licence for large crawls. Many people use both.
One crawl. Every page. A report you can act on today.
Open SilkpathCrawl only websites you own or have permission to audit.