Results Storage

PII Crawler scan results are stored in an SQLite Database. SQLite is a single-file database. Most programing languages have built-in support for connecting to it. By default results will be stored in the user home directory under .piicrawler/scans/<scan-id>.sqlite.

PII Crawler intentionally exposes the application's database to the user so the user can manually or programmatically:

• View the data that is being collected
• Run custom queries on the data
• Create alerts from findings
• Customize which files are scanned
• Customize how files are scanned
• Customize the behavior of PII Crawler

You can think of this database as an API to PII Crawler.

Schema

Files Table

CREATE TABLE IF NOT EXISTS files (
    path TEXT PRIMARY KEY,
    sha256 TEXT,
    size INTEGER NOT NULL,
    extension TEXT NOT NULL,
    mime_type TEXT NOT NULL DEFAULT '',
    skip BOOLEAN DEFAULT 0 NOT NULL,
    last_modified INTEGER DEFAULT 0 NOT NULL,
    parent_path TEXT DEFAULT '' NOT NULL,
    last_scanned_at INTEGER DEFAULT 0 NOT NULL,
    scan_attempts INTEGER DEFAULT 0 NOT NULL,
    last_error TEXT,
    scan_started_at INTEGER,
    scan_finished_at INTEGER
);

Column	Description
path	Absolute path to file (primary key)
sha256	SHA256 hash of file content (used for deduplication)
size	Size of file in bytes
extension	File extension (ex: .pdf, .csv)
mime_type	Detected file MIME type (ex: application/json, image/jpeg)
skip	Boolean flag - if true, file will not be scanned
last_modified	Unix timestamp of when the file was last modified
parent_path	Path to the parent directory
last_scanned_at	Unix timestamp of the last scan attempt
scan_attempts	Number of times scanning has been attempted for this file
last_error	Error message from the last failed scan attempt (if any)
scan_started_at	Unix timestamp when the current/last scan started
scan_finished_at	Unix timestamp when the current/last scan finished

Matches Table

CREATE TABLE IF NOT EXISTS matches (
    id INTEGER PRIMARY KEY,
    text TEXT NOT NULL,
    kind TEXT NOT NULL,
    ignored INTEGER DEFAULT 0 NOT NULL,
    UNIQUE(text, kind)
);

Column	Description
id	Auto-incrementing primary key
text	The matched PII text (e.g., actual email address, name, etc.)
kind	Type of PII match (e.g., "emails", "names", "ssns", "addresses")
ignored	Boolean flag (0 or 1) - if 1, this match will be excluded from results

File Matches Table

CREATE TABLE IF NOT EXISTS file_matches (
    sha256 TEXT,
    match_id INTEGER,
    PRIMARY KEY (sha256, match_id)
);

Column	Description
sha256	SHA256 hash of file content (references files.sha256)
match_id	ID of the match (references matches.id)

This junction table links files to their PII matches using content hashing for deduplication. Multiple files with identical content share the same matches.