Crawl and add websites to an agent’s knowledge base. Supports customizable crawling parameters including depth limit, URL filtering, and page limits.
User ID
Agent ID
Base URL to crawl
Maximum number of pages to crawl (max 10)
1 <= x <= 10Maximum crawling depth (max 10)
1 <= x <= 10Array of URL paths to exclude from crawling
Array of URL paths to include in crawling (if specified, only these paths will be crawled)
Websites added successfully