Crawler Disclosure
MetaDataCrawler/1.0 operates under stringent operational guidelines designed to respect public infrastructure. This page documents our scope, bandwidth posture, and stewardship commitments to the agencies whose archives we analyze.
Purpose & Function
Our primary objective is to help State and Local agencies protect public integrity. We analyze metadata and structural markers in public archives to identify ADA non-compliance and accidental information leaks, enabling agencies to remediate risks before they become liabilities.
Operational Protocol
Standards Adherence
We strictly prioritize robots.txt directives and HTTP headers. Our crawler identifies as MetaDataCrawler/1.0 and will honor all Disallow instructions.
Scope Containment
For domains explicitly in a client's scope, we perform full recursive auditing. For all other systems, analysis is strictly limited to directly referenced URIs identified within the client's authorized environment. Metadata Minder does not perform discovery on unrelated third-party domains.
Bandwidth Efficiency
To conserve host bandwidth, re-fetches utilize HEAD requests to evaluate ETag or Last-Modified headers. We only request the full document body if a change is detected.
Resource Stewardship
Re-examination of external documents is restricted to roughly once per calendar month. Requests are distributed non-predictably across time to prevent spikes in server load.
Need to reach the Rietta security team regarding crawler activity?
(770) 623-2059|Alpharetta, Georgia