What are our data sources?
We use the data sources on the side for ranking solutions and awarding badges in web crawler category. Our data sources in web crawler category include;
Web is the largest source of public information however due to formatting issues and UX changes, it requires manual effort to get consistent/high quality data from web sources. Web crawlers, with the help of pattern recognition techniques, help users overcome these difficulties and leverage the largest source of public information
Web crawlers are also called web scrapers, web data extractors or collectors.
To be categorized as a web crawler, a product must provide an:
AIMultiple is data driven. Evaluate 27 services based on
comprehensive, transparent and objective AIMultiple scores.
For any of our scores, click the information icon to learn how it is
calculated based on objective data.
*Products with visit website buttons are sponsored
We use the data sources on the side for ranking solutions and awarding badges in web crawler category. Our data sources in web crawler category include;
review websites
social media websites
search engine data for branded queries
According to the weighted combination of 7 data sources
Bright Data Web Scraper IDE
Hevo Data
Price2Spy
Octoparse
Phantombuster
Taking into account the latest metrics outlined below, these are the current web crawler market leaders. Market leaders are not the overall leaders since market leadership doesn’t take into account growth rate.
Bright Data Web Scraper IDE
Price2Spy
Hevo Data
Phantombuster
Octoparse
These are the number of queries on search engines which include the brand name of the solution. Compared to other Data categories, Web Crawler is more concentrated in terms of top 3 companies’ share of search queries. Top 3 companies receive 69%, 7% more than the average of search queries in this area.
26 employees work for a typical company in this solution category which is 5 more than the number of employees for a typical company in the average solution category.
In most cases, companies need at least 10 employees to serve other businesses with a proven tech product or service. 19 companies with >10 employees are offering web crawler. Top 3 products are developed by companies with a total of 1k employees. The largest company building web crawler is Bright Data with more than 400 employees.
Taking into account the latest metrics outlined below, these are the fastest growing solutions:
Bright Data Web Scraper IDE
Octoparse
Hevo Data
Price2Spy
Phantombuster
We have analyzed reviews published in the last months. These were published in 4 review platforms as well as vendor websites where the vendor had provided a testimonial from a client whom we could connect to a real person.
These solutions have the best combination of high ratings from reviews and number of reviews when we take into account all their recent reviews.
This data is collected from customer reviews for all Web Crawler companies. The most positive word describing Web Crawler is “Easy to use” that is used in 11% of the reviews. The most negative one is “Difficult” with which is used in 1.00% of all the Web Crawler reviews.
According to customer reviews, most common company size for web crawler customers is 1-50 Employees. Customers with 1-50 Employees make up 49% of web crawler customers. For an average Data solution, customers with 1-50 Employees make up 21% of total customers.
These scores are the average scores collected from customer reviews for all Web Crawlers. Web Crawlers is most positively evaluated in terms of "Customer Service" but falls behind in "Likelihood to Recommend".
This category was searched on average for 52.2k times per month on search engines in 2022. This number has decreased to 51.7k in 2023. If we compare with other data solutions, a typical solution was searched 1k times in 2022 and this increased to 1.2k in 2023.
Web crawling is a true Swiss army knife like Excel, therefore we will stick to the most obvious use cases here:
First, user needs to communicate the relevant content to the crawler. For the technically savvy, this can be done by programming a crawler. For those with less technical skills, there are tens of web crawlers with GUIs (Graphical User Interface) which let users select the relevant data
Then, user starts the crawler using a bot management module. Crawling tends to take time (e.g. 10-20 pages per minute in the starter packages of most crawlers). This is because the web crawler visits the pages to be crawled like a regular browser and copies the relevant information. If you tried doing this manually, you would quickly get visual tests to verify that you are human. This test is called a CAPTCHA "Completely Automated Public Turing test to tell Computers and Humans Apart". Websites have variety of methods like CAPTCHA to stop such automated behavior. Web crawlers rely on methods like changing their IP adresses and digital fingerprints to make their automated behavior less noticeable
Web crawlers extract data from websites. Websites are designed for human interaction so they include a mix of structured data like tables, semi-structured data like lists and unstructured data like text. Web crawlers analyze the patterns in websites to extract and transform all these different types of data.
Crawlers are useful when data is spread over multiple pages which makes it difficult for a human to copy the data
Legality of crawling is currently a gray area and the Linkedin's lawsuit against hiQ which is still in progress, will likely create the first steps of a legal framework around data crawling. In case you are betting your business on crawling, for now don't.
Unless severe restrictions are placed crawling, crawling will remain an important tool in the corporate toolbox. Leading web crawling companies claim to work with Fortune 500 companies like PwC and P&G. BusinessInsider claims in a paywalled article that hedgefunds spend billions on crawling.
We will update this as the Linkedin vs HiQ case comes to a close. Please note that this does not constitute legal advice.