Skip to main content
URL sources can fetch one page or crawl linked pages. The agent uses HTTP parsing for straightforward content and Playwright-based crawling where a browser is required.

​
Source settings

​
string
required
Absolute http or https URL to index.
​
string
required
single for one page or crawl to follow eligible links.
​
number
Maximum link depth used by crawl mode. Start small and widen only when needed.
​
string
One of manual, 1hour, 6hours, or daily.
1

Choose a narrow entry URL

Prefer the documentation or help-center root that contains authoritative customer-facing content.
2

Start with single-page mode

Confirm extraction quality before enabling a crawl. Dynamic menus, cookie banners, and repeated navigation can add noise.
3

Set crawl boundaries

Keep depth low, avoid account/logout/admin URLs, and ensure the crawler follows only intended origins and paths.
4

Index and review counts

Watch the source state, word/chunk counts, error detail, and retrieval results. Pause or delete sources that import irrelevant data.
The ingestion service uses a Redis lock for URL work so the same source is not crawled concurrently. The lock TTL defaults to one hour and can be configured with URL_INGEST_LOCK_TTL_SECONDS.
A crawler can access far more content than a human expects. Do not target private applications, authenticated pages, or hosts that prohibit automated access. Protect against internal-network and metadata-service URLs when accepting user-supplied sources.
Last modified on July 17, 2026