crawler.sh

GDPR overview

The privacy policy, last updated on 14 January 2025, carries an explicit GDPR section: users in the EU may access, update or delete their personal data by writing to support@crawler.sh, and the vendor undertakes to answer within 30 days and to keep data only as long as the service requires. Traffic between the device and the vendor's API is encrypted with HTTPS/TLS, cookies are limited to essential ones, and no advertising or cross-site tracking is used. The documentation stops there. No Data Processing Agreement is published or offered, no subprocessor is named beyond two unattributed categories, no hosting country or region is disclosed, and no Article 27 EU representative is designated. Compliance is asserted; the paperwork that usually evidences it is absent.

Who owns the data?

The split is unusually clear. Crawl data, results and exported files stay on the user's own machine and, according to the privacy policy, never reach the vendor: browsing history, crawl targets, crawl results and extracted page content are explicitly not collected. What the vendor does hold is account data — an email address, a subscription status and authentication tokens, the tokens themselves stored locally in ~/.crawler/auth.json or in browser storage. The terms of service reserve all software, designs and site content as the intellectual property of Crawler, and state that personally identifiable information is never shared without consent except where the law requires it.

Reuse rights

Nothing in the terms restricts what a user does with the pages a crawl produces: the software is licensed for use on the user's own projects, subject to applicable law, and the vendor never takes possession of the output. Responsibility travels with that freedom — users agree to lawful use only and to complying with the regulations that apply to them, which matters because an --ignore-robots flag exists and the rights attached to crawled pages belong to their publishers, not to crawler.sh. On the vendor's own side, account and usage data may be used to run the service, and clause 4.2 of the terms adds that usage data may serve to improve it. Reselling, reverse engineering or redistributing the software itself is prohibited.

Data retention & training

Hosting summary

There are two very different answers depending on which data you mean. Crawl data is hosted nowhere: pages, extracted Markdown and exports are written to the user's own disk and, according to the privacy policy, are never uploaded to the vendor's servers. Authentication tokens are equally local, kept in ~/.crawler/auth.json for the CLI or in browser storage for the desktop app. Account data — email address and subscription status — is the exception: it is processed by the vendor's own API, and neither the hosting country nor the region is disclosed anywhere on the site. Payments run through an unnamed third-party processor, with no card details stored by the vendor. The website itself is served behind Cloudflare on an anycast address, which says nothing about where account data is actually processed. No jurisdiction is stated, no DPA is offered and no subprocessor is named, so the hosting posture for the small amount of personal data involved has to be taken on trust.