DataEdge crawler
DataEdge is the website crawler operated by Nexa Equity. It reads the public home pages
of businesses to describe what each company does.
Identification
- User agent:
DataEdge/crawler (+https://crawler.nexaequity.app/)
- robots.txt token:
DataEdge
- Requests are signed with HTTP Message Signatures (Web Bot Auth). The
Signature-Agent header is "https://crawler.nexaequity.app", and the
public key is published at
https://crawler.nexaequity.app/.well-known/http-message-signatures-directory.
- Requests come from Amazon Web Services addresses in us-east-2 that change over time, and
some retries come from a residential address. Identify the crawler by its signature or user
agent, not by IP address.
What it fetches
- Your robots.txt first, then your home page. It does not follow links to other pages.
- If the home page redirects to another host, that host's robots.txt is read and obeyed.
- The page is loaded in a headless Chromium browser, so the page's own scripts and
stylesheets run. Images, fonts, audio and video are not downloaded.
- One request at a time to a site, at least 2 seconds apart.
- Where a live page cannot be fetched, we may use a public archived copy from the Internet
Archive or Common Crawl, unless your robots.txt disallows us.
robots.txt
- Rules for
DataEdge apply first; otherwise the rules for *
apply.
- If your robots.txt disallows other crawlers but allows Googlebot, we read that as consent
to be indexed and treat DataEdge as allowed. A rule that names
DataEdge always
takes precedence. We never identify as Googlebot.
Crawl-delay and Request-rate are honoured up to 30 seconds. A site
asking for more is skipped.- If robots.txt returns a server error (5xx or 429) or does not answer, the page is not
fetched. A 4xx response means there is no robots.txt, as RFC 9309 specifies.
Opting out
Add this to your robots.txt:
User-agent: DataEdge
Disallow: /
The crawler reads it before its next visit. To have stored pages from your site removed,
write to the contact address below.
Use of the data
The page text is used internally to categorise businesses and is not sold or shared.
Contact
automation@nexaequity.com