CertTransparencySearch/README.md

Certificate Transparency Search

This project builds a publication-grade monograph from Certificate Transparency and public DNS:

• it finds currently valid leaf certificates whose SAN values contain configured search terms
• it verifies locally that the certificates are real leaf certificates rather than CA certificates or precertificates
• it assesses intended usage from EKU and KeyUsage
• it scans the DNS names exposed by the SAN corpus
• it evaluates the effective CAA policy for those DNS names to show where public CA issuance is governed, delegated, or unrestricted
• it can analyse a second local-only Subject-CN cohort file against the wider estate
• it produces one primary readable output set: a monograph in Markdown, LaTeX, and PDF

The project is designed for public source control:

• real search terms live only in domains.local.txt
• real focused Subject-CN cohorts live only in focus_subjects.local.txt
• generated artefacts live only in output/
• caches live only in .cache/

None of those paths should be committed.

What You Need On A Fresh macOS Machine

Required software

• git
• python3
• make
• dig
• xelatex

What each dependency is for

• python3: runs the scanners and report generators
• make: gives you short repeatable commands instead of long manual command lines
• dig: performs the live DNS and CAA scans
• xelatex: compiles the PDF reports

If xelatex is missing, the Markdown and LaTeX outputs can still be generated, but the PDF targets will fail.

Network access required

• outbound TCP access to crt.sh:5432
• public DNS resolution for dig

The scanner reads Certificate Transparency data directly from the public certwatch PostgreSQL service on crt.sh using guest access. If that TCP path is blocked, the certificate part of the run will fail even if normal web browsing works.

Clean-Room Operator Checklist

Use this sequence if you need to reproduce the same output structure on another Mac without any extra guidance:

1. Install the required macOS system tools.
2. Clone the repository.
3. Create the Python virtual environment and install Python dependencies.
4. Create the local-only config files.
5. Put the real search terms into domains.local.txt.
6. Optionally put the focused Subject-CN cohort into focus_subjects.local.txt.
7. Run make monograph.
8. Read the outputs from output/corpus/.

Expected final outputs:

• output/corpus/monograph.md
• output/corpus/monograph.tex
• output/corpus/monograph.pdf

The PDF build no longer depends on macOS-only fonts.

macOS Install Recipe

Install Apple command-line tools first:

xcode-select --install

If Homebrew is not already installed, install it from https://brew.sh, then install the required tools:

brew install python make
brew install --cask mactex-no-gui

Notes:

• git, make, and dig are usually already present once Apple command-line tools are installed.
• mactex-no-gui provides xelatex.
• If xelatex is still not on your PATH after installation, open a new shell and re-run which xelatex.

Preflight Checks

Run these checks before the first full build:

python3 --version
git --version
make --version
dig -v
xelatex --version
nc -vz crt.sh 5432

If the last command fails, the CT query layer will not be able to reach the public certwatch database.

Fresh Install On Another Mac

Clone the repository from your chosen remote and enter the directory:

git clone <repository-url>
cd CertTransparencySearch

Create the local Python environment and install dependencies:

make bootstrap

Create the local-only search-term file:

make init-config

Then edit domains.local.txt and replace the placeholder values with the real search terms you want to scan.

If you want the monograph to analyse a remembered or suspicious Subject-CN cohort as well, edit focus_subjects.local.txt too. The format is one Subject CN per line, optionally followed by analyst notes in parentheses.

Fastest End-To-End Run

If the Mac already has the required system tools installed, this is the shortest full path:

git clone <repository-url>
cd CertTransparencySearch
make bootstrap
make init-config
# edit domains.local.txt
# optionally edit focus_subjects.local.txt
make monograph

The canonical results will then be in:

• output/corpus/monograph.md
• output/corpus/monograph.tex
• output/corpus/monograph.pdf

Local Search Terms

The tracked file is:

• domains.example.txt
• focus_subjects.example.txt

The local-only file is:

• domains.local.txt
• focus_subjects.local.txt

Rules:

• keep real search terms only in domains.local.txt
• keep real focused Subject-CN cohorts only in focus_subjects.local.txt
• do not rename that file unless you also pass DOMAINS=... to make
• do not commit either local file

One-Command Runs

Main publication

This is the single canonical publication. The appendices are embedded into the same monograph, so you do not need to manage separate visible appendix artefacts:

make monograph

Outputs:

• output/corpus/monograph.md
• output/corpus/monograph.tex
• output/corpus/monograph.pdf

Internal helper artefacts used during PDF assembly are written only under .cache/monograph-temp/.

Supporting purpose assessment

This is optional. Its findings are already woven into the monograph, but the standalone output can still be useful during development:

make purpose

Outputs:

• output/corpus/certificate-purpose-assessment.md
• output/corpus/certificate-purpose-assessment.json

Historical lineage analysis

This is optional. Its findings are already woven into the monograph, but the standalone output can still be useful during development:

This report extends the analysis across current and expired certificates to study:

• repeated issuance under the same Subject CN
• Subject CN with different Subject DN over time
• Subject CN with different issuing CA or vendor over time
• Subject CN with different SAN profiles over time
• issuance bursts and step-change start dates

make lineage

Outputs:

• output/corpus/certificate-lineage-report.md
• output/corpus/certificate-lineage-report.tex
• output/corpus/certificate-lineage-report.pdf

Shorter executive report

make consolidated

Outputs:

• output/corpus/consolidated-corpus-report.md
• output/corpus/consolidated-corpus-report.tex
• output/corpus/consolidated-corpus-report.pdf

Full operator run

This creates the local config if missing, then builds the full monograph:

make all

Reproducibility And Run Behaviour

The default Makefile values are:

• DOMAINS=domains.local.txt
• FOCUS_SUBJECTS=focus_subjects.local.txt
• CACHE_TTL=0
• DNS_CACHE_TTL=86400
• CAA_CACHE_TTL=86400
• MAX_CANDIDATES=10000

This means:

• Certificate Transparency is refreshed live on every normal run.
• DNS results are reused for up to one day unless you override the DNS cache TTL.
• CAA results are reused for up to one day unless you override the CAA cache TTL.
• The query cap is high enough for the current corpus and the scanner will refuse to run if the live raw match count exceeds the cap.

If you want to override values:

make monograph CACHE_TTL=86400 DNS_CACHE_TTL=86400 CAA_CACHE_TTL=86400

Or:

make monograph DOMAINS=/path/to/other.local.txt

Or override both local inputs:

make monograph DOMAINS=/path/to/domains.local.txt FOCUS_SUBJECTS=/path/to/focus_subjects.local.txt

Manual Commands

If you do not want to use make, the equivalent commands are:

Inventory appendix source

This is only needed if you want the raw family inventory outside the monograph:

.venv/bin/python ct_scan.py \
  --domains-file domains.local.txt \
  --cache-ttl-seconds 0 \
  --max-candidates-per-domain 10000 \
  --output output/corpus/current-valid-certificates.md \
  --latex-output output/corpus/current-valid-certificates.tex \
  --pdf-output output/corpus/current-valid-certificates.pdf

Purpose assessment

.venv/bin/python ct_usage_assessment.py \
  --domains-file domains.local.txt \
  --cache-ttl-seconds 0 \
  --max-candidates 10000 \
  --markdown-output output/corpus/certificate-purpose-assessment.md \
  --json-output output/corpus/certificate-purpose-assessment.json

Consolidated report

.venv/bin/python ct_master_report.py \
  --domains-file domains.local.txt \
  --cache-ttl-seconds 0 \
  --dns-cache-ttl-seconds 86400 \
  --max-candidates-per-domain 10000 \
  --markdown-output output/corpus/consolidated-corpus-report.md \
  --latex-output output/corpus/consolidated-corpus-report.tex \
  --pdf-output output/corpus/consolidated-corpus-report.pdf

Historical lineage report

.venv/bin/python ct_lineage_report.py \
  --domains-file domains.local.txt \
  --cache-ttl-seconds 0 \
  --max-candidates-per-domain 10000 \
  --markdown-output output/corpus/certificate-lineage-report.md \
  --latex-output output/corpus/certificate-lineage-report.tex \
  --pdf-output output/corpus/certificate-lineage-report.pdf

Full monograph

.venv/bin/python ct_monograph_report.py \
  --domains-file domains.local.txt \
  --focus-subjects-file focus_subjects.local.txt \
  --cache-ttl-seconds 0 \
  --dns-cache-ttl-seconds 86400 \
  --caa-cache-ttl-seconds 86400 \
  --max-candidates-per-domain 10000 \
  --markdown-output output/corpus/monograph.md \
  --latex-output output/corpus/monograph.tex \
  --pdf-output output/corpus/monograph.pdf

Project Structure

• ct_scan.py: core CT scan, leaf verification, grouping, and detailed inventory report
• ct_usage_assessment.py: EKU and KeyUsage assessment
• ct_lineage_report.py: historical Subject CN, Subject DN, issuer, SAN, and issuance-burst analysis
• ct_dns_utils.py: DNS scanning and provider-signature logic
• ct_caa_analysis.py: CAA discovery, caching, and issuance-policy analysis
• ct_master_report.py: shorter consolidated report
• ct_monograph_report.py: publication-grade monograph with embedded appendices
• Makefile: reproducible operator workflow

Safety Against Silent Undercounts

The scanner checks the live raw identity-row count before it executes the capped query. If the configured cap is too low, it stops with an error instead of silently returning an incomplete corpus.

Public Repo Rules

• keep domains.local.txt local only
• keep focus_subjects.local.txt local only
• never commit output/
• never commit .cache/
• if you need a sample config in git, update domains.example.txt, not domains.local.txt
• if you need a sample focused-cohort config in git, update focus_subjects.example.txt, not focus_subjects.local.txt