Policy documents — privacy policies, terms of service, community guidelines, transparency reports, and more — are the legal documents that govern how social media platforms collect, store, and use your data. They affect your rights and your access to justice. Most people never read them. And it turns out that even those who want to often have difficulty: the documents are frequently buried behind hard-to-find URLs, written in dense legal language, and rarely include any record of what changed between versions.
These gaps are exactly what Transparency Hub was built to address. Today, we’re excited to announce that the Transparency Hub codebase is now open source.
What Transparency Hub Does
Transparency Hub aggregates and archives policy documents from over 300 social media companies, giving researchers, journalists, educators, and curious users a single place to search, compare, and track how platform governance evolves over time. Our project primarily covers US-based platforms and US-facing policy documents; we recognize that platform governance varies significantly across jurisdictions, and we encourage researchers and developers elsewhere to adapt the codebase for their own locale.
The platform’s core capabilities include:
- Automated document collection that detects and downloads updated policy documents in real time, eliminating the manual overhead of tracking policy changes across hundreds of companies
- Cross-version comparison tools that let users place documents side by side across time, exposing specific updates and wording changes
- Full archive access to historical versions of documents, making it possible to trace how policies have shifted in response to regulatory changes, public pressure, or major product decisions
Why We’re Opening the Code
Building Transparency Hub taught us a few things about the broader landscape of policy research tooling. First, this kind of research is genuinely hard: previous versions of documents are rarely preserved, meaningful changelogs almost never exist, and links to the documents themselves are sometimes very hard to find. Second, the few open-source tools in this space have largely gone unmaintained.
We think the appropriate response to this need is to put what we’ve built into the community’s hands.
Opening the source code means researchers can audit how we collect and store documents, extend the system with new capabilities, deploy their own instances for specific research contexts, and build on top of our infrastructure rather than starting from scratch. Transparency Hub is a tool to enable research, not to replace the interpretive work of lawyers, journalists, or policy analysts. We’re engineers, and we built something we hope makes their work easier.
Who This Is For
If you work in privacy research, platform governance, digital rights advocacy, or journalism, Transparency Hub gives you structured, accessible data that has historically required significant manual effort to gather. If you’re a developer interested in contributing or in building applications that consume structured policy data, the open codebase is a starting point.
To get up and running, you’ll need Docker. Full setup instructions are available in the repository.
If you’re an everyday user who has ever clicked “I agree” without reading what you agreed to, the Transparency Hub web interface is available without any technical setup required.
Get Involved
The code is released under the AGPL-3.0 license with a fresh version history. It spans three repositories:
- transparency-hub — the website codebase
- transparency-hub-engine — the main archiving engine, hosted on GCP, which captures policy documents from across the web
- transparency-hub-browsertrix-crawler — our fork of the Browsertrix crawler, extended with three runtime parameters (–detachedHTML, –detachedText, and –generatePDF) that produce those respective document formats during capture for extraction by the sync pipeline; each can be configured to run only at depth 0 (the initial page) or across all pages discovered during a crawl
We welcome contributions, bug reports, and feedback across all three repos. You can also use this form to suggest platforms you’d like us to track, or reach out here to request database access for research purposes.
Policy documents are one of the primary mechanisms by which platforms communicate their governance decisions to the public. Making those documents findable, comparable, and trackable over time is a small but important step toward a more accountable social media ecosystem.