Reference

Watch Mode & Policies

Last updated September 2026

piicrawler watch is the daemon mode of PII Crawler. It monitors one or more paths for file changes, scans new or modified files for PII, evaluates each finding against a set of policies, and dispatches violations to JSON stdout, a webhook, and the local database.

A watched path is usually a directory, which is walked recursively. It may also be a single file — piicrawler watch /srv/exports/customers.csv watches that one file and nothing else in its directory.

watch reports changes. The files that are already there when the daemon starts are its starting point, not its first result, so a daemon pointed at a share full of SSNs stays quiet until something is written to it. Add --scan-existing to check those files first, then watch.

This page is the complete reference for the policy file format, the violation payload schema, and the operational behaviour of the daemon. For the CLI flags themselves see piicrawler watch.

Watching scans files, so it needs a registered installation, the same as piicrawler scan. Register the machine first with piicrawler register [email protected]; an unregistered copy exits immediately instead of starting the daemon.

How the pipeline works

At startup, with --scan-existing, the daemon scans every file already under the watched paths, one at a time, through steps 2 to 5 below. Without the flag it skips straight to watching.

Then, each cycle:

Polls every watched path for file changes (created, modified, removed). A directory is walked recursively; a path that is a file is checked on its own. The daemon polls rather than relying on OS file-change notifications, so watching behaves identically across Linux, macOS, and Windows.
For each created or modified file, extracts text using the same extraction stack as a one-shot scan (PDF, Office, archive, image OCR, etc.). Symlinks and offline cloud-only files are skipped.
Scans the extracted text for PII.
Evaluates every finding against the loaded policies. A finding becomes a violation when it matches a policy.
Dispatches the violations: writes them to the watch_violations table, emits them on stdout as JSON (unless --no-json), and POSTs them to the webhook (if --webhook was given).

The daemon runs until you send Ctrl+C.

Scanning what is already there

--scan-existing scans every file already under the watched paths before the daemon settles into watching for changes. Use it when the watched location is not new: an existing share, a directory of last quarter's exports, anywhere the PII you care about may have arrived before the daemon did.

piicrawler watch /srv/uploads --policy ./policies.toml --scan-existing

Watching 1 path for PII:
  /srv/uploads
Scanning the files that are already there first, then watching for changes.
2 alert policies active.
Press Ctrl-C to stop.
PII found in /srv/uploads/2024-handover.csv (3 findings)
PII found in /srv/uploads/archive/payroll.xlsx (11 findings)
Checked 412 files already in place, 2 with PII.

Each file goes through the same extraction, scan and policy evaluation as a changed one, so violations from the pass reach stdout, the webhook and the watch_violations table exactly like any other. Checked counts the files actually scanned; unsupported types and files whose text could not be extracted are not counted.

The pass runs one file at a time, like the rest of the daemon, and file changes that happen while it is running are queued and handled as soon as it finishes. Nothing is scanned twice: the pass and the change poller share one walk of the watched paths.

For a first inventory of a large share, a one-shot piicrawler scan is still the better tool. It scans in parallel and writes a saved scan you can report on, whereas the pass exists so a long-lived daemon does not begin its life ignoring what it is pointed at.

The policy file

--policy <file> loads alert policies from a TOML-style config file. Each policy describes a condition (which PII types in which paths) and metadata (an action label and a severity label) that gets attached to every matching violation.

Example

# policies.toml: use any path you like; the file is read and parsed.

[[policy]]
name = "no-ssn-in-public-shares"
pii_type = "ssn"
path_pattern = "/srv/public/.*"
action = "deny"
severity = "critical"

[[policy]]
name = "any-credit-card"
pii_type = "credit-card"
action = "alert"
severity = "high"

[[policy]]
# Catch-all: alert on anything PII-shaped landing in /srv/uploads/
name = "uploads-anything"
path_pattern = "/srv/uploads/.*"
severity = "medium"

Load it on startup:

piicrawler watch /srv/public /srv/uploads --policy ./policies.toml

Schema

Each [[policy]] table accepts the following keys:

Key	Required	Default	Type	Purpose
`name`	yes	—	string	Identifier shown in alerts and stored with each violation row. A table without a `name` is skipped, and reported on stderr by its position in the file.
`pii_type`	no	(any)	string	Restrict the policy to a single PII type slug (see PII type slugs below). Compared case-insensitively. Omit to match every PII type. A slug PII Crawler does not produce is loaded as written and can never match, so it is reported on stderr at load time.
`path_pattern`	no	(any)	regex	Restrict the policy to files whose path matches this Rust regex. The match uses `is_match` semantics: it succeeds if the regex matches anywhere in the path, so `/public/.` matches `/srv/public/data.txt`. Omit to match every path. An invalid regex is skipped with a warning on stderr at load time, and the resulting policy then matches every* path rather than the ones you meant — a scoped policy fails open, not closed, so read the warning.
`action`	no	`"deny"`	string	Free-form label attached to each violation. Pass-through only; see Actions and severity are labels.
`severity`	no	`"high"`	string	Free-form severity label attached to each violation. Pass-through only.
`max_risk`	no	—	string	Currently parsed but not evaluated. Reserved for a future risk-score filter; setting it has no effect today.

Keys are read line-by-line; values may be quoted with " or ' (name = "no-ssn", name = 'no-ssn') or left bare (name = no-ssn). A # begins a comment anywhere outside a quoted string, so it may follow a header ([[policy]] # public shares) or a value (severity = "high" # page the on-call) as well as start a line of its own — and a # inside quotes is part of the value, so path_pattern = "/srv/ticket-#[0-9]+/.*" keeps its hash. Whitespace inside the brackets ([[ policy ]]) is allowed. Blank lines are allowed.

Everything the parser cannot use is reported on stderr, and nothing is dropped in silence. A key it does not recognise, a key before the first [[policy]] header, a table header that is not [[policy]], a line that is neither a header nor key = value, and a file with no [[policy]] table in it each produce a warning naming the line. The reason is that a policy file is not run, it is loaded: a rule that fails to parse is not an error you see, it is an alert that never fires.

What loading reports

Nothing about a bad policy fails: it is stored and evaluated like any other, and then matches nothing (a pii_type that does not exist) or everything (a path_pattern that does not compile). So --policy says on stderr what it made of the file before the daemon starts:

$ piicrawler watch /srv/uploads --policy ./policies.toml
warning: policy "typo-type": pii_type "credit_card" is not a type PII Crawler produces, so this policy can never match — did you mean "credit-card"?
warning: policy "bad-regex": path_pattern "/srv/[unclosed" is not a valid regex (unclosed character class), so the policy matches every path instead of the ones you meant
warning: line 14: policy table #3 has no name and was skipped; every [[policy]] needs a name = "..."
warning: line 19: [[polcy]] is not a policy table — only [[policy]] tables are read. The 2 keys under it were ignored.
Loaded 2 policies from ./policies.toml.
Watching 1 path for PII:
  /srv/uploads
2 alert policies active.
Press Ctrl-C to stop.

Loaded N policies from <file> counts this file. N alert policies active counts every policy in the database, including ones loaded by earlier runs and since removed from the file (see Reloading policies) — the two numbers differ on purpose, and only the first one is about the file you just passed.

That difference is why the warnings matter more than the counts. A typo in a table header, or a header this file does not use at all, does not stop the daemon: the tables it could read are loaded, the rest are reported, and everything already in the database from a previous run stays there. So a file that was fine yesterday and has a mistake in it today can print Loaded 0 policies above 2 alert policies active — the daemon is running, on yesterday's rules. Read the warnings, and read Loaded N as being about the file rather than about the daemon.

A finding becomes a violation for a given policy when both of the following hold:

The finding's PII type equals pii_type (case-insensitive), or pii_type is omitted.
The file's path matches path_pattern, or path_pattern is omitted.

A single finding can violate multiple policies. Each match produces its own violation row and its own alert.

PII type slugs

pii_type accepts any of the built-in slugs:

Slug	What it detects
`ssn`	U.S. Social Security Number
`credit-card`	Credit card number (Luhn-checked)
`aws-credential`	AWS access keys / secret keys
`passport`	U.S. passport number
`ein`	Employer ID Number
`drivers-license`	Driver's license number
`dob`	Date of birth
`address`	Street address
`phone`	Phone number
`city_state_zip`	City / state / ZIP cluster
`email`	Email address
`name`	Full name

It also accepts the dynamic slugs PII Crawler generates for user-defined detectors:

regex-<slug>: a custom regex rule
terms-list-<slug>: a terms list match
proximity-<slug>: a proximity regex group

The slug is whatever the rule was named, lowercased and dashed. For example, a custom regex rule called "Account Number" produces the slug regex-account-number.

Actions and severity are labels

action and severity are emitted to alerts and stored with each violation, but PII Crawler itself does not act on them. Setting action = "deny" does not block, quarantine, or modify the file. The file is left exactly where it is. Both fields exist so a downstream system (your webhook receiver, SIEM, ticketing pipeline, etc.) can decide what to do based on the label.

If you want the daemon to actually move or delete a file when PII is found, do it on the receiving end of your webhook.

Reloading policies

--policy <file> is upsert-by-name: each [[policy]] table is inserted if its name is new, or its fields are updated in place if a policy with that name already exists. Re-running the daemon with the same file is therefore idempotent: no duplicate alerts.

Policies that you remove from the file are not deleted from the database automatically. If you rename or drop a policy and want the old row gone, delete it explicitly with your SQLite client of choice. The database lives at ~/.piicrawler/piicrawler.db (see Results Storage):

sqlite3 ~/.piicrawler/piicrawler.db "DELETE FROM watch_policies WHERE name = 'old-name';"

Policies are stored in the local database and shared across runs, but the TUI and Web UI do not currently expose a policy editor. --policy <file> is the only way to load or change them.

Webhook payload

When --webhook <url> is set, the daemon sends each batch of violations from a single file event as one HTTP request:

POST <url>
Content-Type: application/json

{"violations":[
  {
    "event": "policy_violation",
    "policy": "no-ssn-in-public-shares",
    "file": "/srv/public/handover/employees.csv",
    "pii_type": "ssn",
    "term": "***-**-6789",
    "severity": "critical",
    "action": "deny"
  }
]}

Field reference

Field	Source
`event`	Always the literal string `"policy_violation"`.
`policy`	The policy's `name`.
`file`	Absolute path of the changed file.
`pii_type`	The detector slug (e.g. `ssn`, `email`, `regex-account-number`).
`term`	The matched text, masked for transport (e.g. `*--6789`, `jo***e@ex*le.com`). The unmasked term is never sent over the wire.
`severity`	The policy's `severity` label, verbatim.
`action`	The policy's `action` label, verbatim.

Batching

One POST is made per file event. If a single file produces N policy violations (e.g. it contains both an SSN and a credit card and you have policies for both), all N appear in the same request's violations array. Files that produce zero violations result in no request at all.

Delivery semantics

The webhook is fire-and-forget. There is no retry on failure: a non-2xx response or a connection error is logged at error level and the violations are dropped from the webhook stream. They are still recorded in watch_violations and (if enabled) printed to stdout.
The request is synchronous: the daemon blocks on the POST before processing the next file event. A slow webhook will throttle scanning. Run your receiver behind a fast queue if alert volume could be high.
There is no signing header or shared secret. If the receiver is reachable from anywhere other than localhost, terminate it behind a reverse proxy that enforces auth.

JSON stdout stream

Unless you pass --no-json, every violation is also written to stdout as a single line of JSON, in the same shape as one element of the webhook violations array:

{"event":"policy_violation","policy":"no-ssn-in-public-shares","file":"/srv/public/handover/employees.csv","pii_type":"ssn","term":"***-**-6789","severity":"critical","action":"deny"}

This is JSONL (one violation per line). Pipe it through jq -c or into a log shipper like Vector or Filebeat. As with the webhook payload, the term field is masked.

Status output is written to stderr, so stdout stays clean for piping:

piicrawler watch /srv/uploads --policy ./policies.toml > violations.jsonl 2> watch.log

The daemon could not start

In that pipeline, stderr goes to a file nobody is watching. So if the daemon cannot start, it says so on stdout as well, as one more line of the same stream:

{"error":"no such path to watch: /srv/uplods","event":"error"}

Check event to tell the two apart: violations carry "event":"policy_violation", and this is the only other kind of line the stream produces. There is at most one, it is always the last, and the daemon exits non-zero straight after it.

Handle it. Without it, a mistyped path, a policy file that moved, or a trial that expired leaves an empty violations.jsonl, which is indistinguishable from a quiet day:

piicrawler watch /srv/uploads --policy ./policies.toml \
  | jq -c 'select(.event == "error") | "watch failed: \(.error)"'

With --no-json there is no stream to put it in, so nothing is written to stdout and the message is on stderr alone.

What the daemon reports on stderr

On startup the daemon prints what it is watching, whether it is scanning the files already there, and how many policies are active:

Watching 1 path for PII:
  /srv/uploads
Only files created or changed from now on are scanned. Pass --scan-existing to check the files that are already there first.
2 alert policies active.
Press Ctrl-C to stop.

The third line is the one that explains a quiet daemon. watch diffs against the files that are there when it starts, so silence over a directory full of PII means nothing has changed yet, not that nothing was found. With --scan-existing that line becomes Scanning the files that are already there first, then watching for changes., and the pass closes with a Checked N files already in place, M with PII. line.

Then, every time a changed file turns out to hold PII, it prints one line:

PII found in /srv/uploads/handover.csv (3 findings)

This line appears whether or not any policy matched, which makes it the quickest way to confirm the daemon is seeing your file events. Violations, which need a matching policy, are the JSON on stdout.

Full structured logs (including per-file extraction failures) go to the log file rather than stderr. See Troubleshooting for its location, or set PIICRAWLER_LOG_FILE to choose one.

Database persistence

Independent of stdout and the webhook, every violation is inserted into the watch_violations table in the local database. Each row records the policy that fired, the file path, the PII type, the unmasked matched term, the severity, and a timestamp.

The unmasked term is kept locally so triage and DSAR workflows have the underlying value, but it is never transmitted off the host. The schema lives in Results Storage → Watch mode.

Operational notes

Watched paths are resolved to absolute paths. piicrawler watch ./uploads reports and stores /home/you/uploads/..., not ./uploads/.... A relative path names a file only in combination with the directory the daemon was started in, which is not part of a violation, a watch_violations row, or a path_pattern — so the daemon resolves each path once at startup and the startup banner prints what it resolved to.
Only changes are scanned, unless you ask. The first walk of a watched path is the poller's baseline, so files already in place are not scanned. --scan-existing scans them once at startup and then watches as usual.
Polling, not OS events. The daemon walks every watched directory each cycle and diffs against the previous snapshot on (size, mtime). This keeps behaviour identical across platforms but means CPU cost grows with the size of the watched tree.
Debounce floor. --debounce <ms> sets the polling interval. Values below 500 are clamped to 500ms, so --debounce 100 and --debounce 500 behave identically. Defaults to 500.
PII Crawler's own files are skipped. Watching a path that contains them (piicrawler watch ~) does not report its data directory (~/.piicrawler, which holds the database), the log file the daemon is writing, or the run folders a quarantine has filled. The database is the one that matters: the daemon stores a row there for every violation it raises, so without this each alert was a file change, each change was a scan of a file holding a copy of every finding ever recorded, and that scan raised more alerts. Name one of them as a watched path outright and it is watched as asked. See Exclusion patterns for the full list.
Symlinks are skipped. This prevents traversal escape from the watched root.
Offline cloud files are skipped. Files marked as cloud-only / placeholder by macOS iCloud, OneDrive, Dropbox, etc. are not downloaded by the daemon.
Unsupported file types are skipped. Only files matching PII Crawler's supported file types are extracted and scanned; everything else short-circuits before any work is done.
Removed files don't trigger scans. Deletions are detected but are not surfaced as violations because there is no PII to find.
File events are handled one at a time. The daemon extracts, scans and dispatches each event before it looks at the next one. A file that takes the full extraction timeout (120 seconds, the same as a one-shot scan) therefore delays every event behind it by that much, and a slow webhook receiver does the same, since the POST is synchronous. If one tree can hold up alerts for another, watch them from separate piicrawler watch processes.

Examples

Dry-run on stdout, no webhook, no policies

Useful for verifying file events are firing as you expect:

piicrawler watch /srv/uploads

With no --policy, no policies are loaded, so no violations are produced and stdout stays empty. The daemon says so on startup rather than looking like it has hung:

Watching 1 path for PII:
  /srv/uploads
Only files created or changed from now on are scanned. Pass --scan-existing to check the files that are already there first.
No alert policies are loaded, so no alerts will be sent and nothing will be written to stdout.
Load alert rules with --policy <file>: https://www.piicrawler.com/docs/watch-mode/
Press Ctrl-C to stop.

Each file holding PII is still reported on stderr as it changes, which is what makes this useful as a dry run. Touch a file to see it work; nothing is reported for the files that were already in place unless you add --scan-existing.

Single-policy alert to a webhook, no JSON stream

piicrawler watch /srv/uploads \
  --policy ./policies.toml \
  --webhook https://alerts.example.com/piicrawler \
  --no-json

Catch-all policy, all PII types, only in a sensitive subtree

[[policy]]
name = "anything-in-finance"
path_pattern = "/srv/shared/finance/.*"
severity = "high"

CI-style: fail loudly on first violation

watch itself does not exit on a violation. It is a daemon. For "fail the build if PII is present" use piicrawler dsar --assert-clean or run a one-shot scan and check the output.