This site shows the results of a weekly demo run of the distcrawl project. Every week on Monday, we use Terraform to automatically spin up the infrastructure across providers (Hetzner, Cloudflare, SaladCloud), then run the demo crawl and show the results on this site. Then all infrastructure is destroyed. I chose wikipedia, duckduckgo and my personal website. The last crawl of our 3 example websites happened on 2026-08-10 and took 34s.
| Site | Meta Description | Status | Requests | Responses | Crawled from Country | Crawl Time |
|---|---|---|---|---|---|---|
| wikipedia.com | Wikipedia is a free online encyclopedia, created and edited by volunteers around the world and hosted by the Wikimedia Foundation. | crawled | 6 | 6 | US | 10th of August, at 1:44AM |
| duckduckgo.com | The Internet privacy company that empowers you to seamlessly take control of your personal information online, without any tradeoffs. | crawled | 92 | 92 | US | 10th of August, at 1:44AM |
| landmann.ph | Cloud & Platform Engineer and Computer Science graduate focused on infrastructure automation, observability, and distributed systems. | crawled | 12 | 12 | US | 10th of August, at 1:45AM |
See the distcrawl GitHub repository to learn more about the project and how to deploy it for your own crawls. Because we can split the crawling task across dozens / hundreds of machines, collecting large-scale datasets can be done very quickly and cheaply by taking advantage of consumer-grade hardware (e.g. SaladCloud nodes) or spot instances. I have successfully tested it on a crawl of different configurations (browser choice, cookie acceptance enabled, disabled) to collect a large scale (~23GB) dataset of requests and responses for my bachelor's thesis "Web Crawling Analysis of Cookie Consent and Subpage Navigation Effects on Web Tracking". The dataset is available here.