Algolia CrawlerUnleash your content

Algolia Crawler is a hosted and highly customizable web crawler that makes sense of any content of a website and makes it deliverable through a seamless experience

World’s leading brands use Algolia to power their Site Search and Discovery

Accelerate time to value

Great Site Search experiences are based on various types of content, but this content is siloed in disparate systems managed by different teams.

By automatically extracting content from your websites, Algolia Crawler removes the need for building data pipelines between each of your content repository and Algolia, and avoids complex internal project management, saving time and resources.

Turn web pages into structured content

Tailor the crawler to make sure it accurately interprets your content. It allows your users to search and navigate news articles, job posts, FAQ answers, financial reports or any type of content your website offers, including JavaScript, PDFs and Docs, instead of generic web pages.

Extract content without editing your website

Extract structured content without the need to add any metatag to your website. Algolia Crawler provides an easy to use editor for your technical team, so they can define what content to extract and how to structure it, ensuring an optimal end user experience.

Enrich your content to improve the experience

Algolia Crawler can enrich the extracted content with business data, including Google Analytics data, to enhance the relevance of the end user experience. From using your visitor behaviors and page performance to adjust the search rankings, to attaching categories to your content to power advanced navigation, possibilities are endless.

Configure the crawler to your needs

Algolia Crawler gives you the options to index the parts of your websites you need, when you need it.

Discover the business value of Algolia Crawler

Schedule automatic crawls at the timing of your choice - illustration

Manually trigger a crawl of part or all your websites when necessary - illustration

Define what parts of your websites the crawler should or should not explore, or let it explore your websites automatically - illustration

Configure the crawler to explore login protected pages when necessary - illustration

Schedule automatic crawls at the timing of your choice

Manually trigger a crawl of part or all your websites when necessary

Define what parts of your websites the crawler should or should not explore, or let it explore your websites automatically

Configure the crawler to explore login protected pages when necessary

Rely on a Production Ready crawler

Algolia Crawler comes with a complete set of tools to make sure you always fuel your site search experience with up to date and accurate content.

URL Inspector

Search and inspect all the crawled URLs. For each URL, check when it was last crawled, whether the crawl was successful, and the records it generated.

Monitoring

Get a detailed report of the errors encountered during the last crawl.

Data analysis

Assess the quality of the extracted data. For each type of content, the Data Analyser compares all the extracted content to identify missing data.

Path Explorer

Assess which paths the Crawler explores, and for each path, how many URLs were crawled, how many records were extracted, what errors happened.

“We realized that search should be a core competence of the LegalZoom enterprise, and we see Algolia as a revenue generating product.”

Mrinal Murari
Tools team lead & senior software engineer

Read the full story

Webinar: The next best way to increase revenue

Industries

Platform Solutions