The customer’s website becomes a knowledge base
The website crawler is the starting point that works for every end customer. You enter the address; it retrieves the page list and indexes the text. From then on, the chat answers questions about services, prices and opening hours using what is already on the site.
- The page list comes from the sitemap. The crawler discovers it automatically via robots.txt, /sitemap.xml or /sitemap_index.xml.
- If a site has no sitemap, you add the addresses as a list. Up to 2,000 pages per source.
- One sync per day keeps content current. Unchanged pages are identified by a checksum and are not processed again.
- robots.txt is respected. Anything the site operator has blocked for bots stays out.
Good to know: The crawler reads HTML. It cannot see content loaded later by JavaScript, and PDFs on the site are not recognized as documents. Upload manuals and data sheets directly to the knowledge base.