Exclusion Patterns

Overview

Exclusion patterns keep paths out of a scan. Use them to skip dependency folders, build output, version control data, or anything else you do not want scanned.

A pattern is a regular expression matched against the whole path, not a glob and not a directory name. node_modules matches any path containing that text; \.log$ matches any path ending in .log.

Excluded files are never opened, never scanned, and never stored, so they do not appear in the file list, the counts, or the results. Because an exclusion silently makes a scan smaller, every scan reports what each pattern kept out. See What your patterns excluded.

Setting exclusion patterns

Patterns belong to a scan and are set when the scan is created.

• Web UI: open New Scan, expand File Filtering, and enter one pattern per line under Exclusion Patterns.
• TUI: the exclusions field on the form that starts the scan, comma separated. Each of the three forms (local, network share, mailbox) has its own, and a scan runs with the patterns typed into the form that started it. Because the field splits on commas, a pattern that contains one — a repetition count such as \d{1,2} — cannot be written here; use the web UI or --exclude, which take a pattern whole.
• CLI: --exclude, repeatable, on scan, smb, and imap:

piicrawler scan ~/share --exclude '(^|/)node_modules/' --exclude '(^|/)\.git/'

A pattern that is not a valid regular expression is refused before the scan starts, naming the pattern and the reason. So is a pattern that would exclude the whole scan — see Patterns that match everything. To change the patterns on an existing scan, create a new scan with the patterns you want.

Writing a pattern

The whole path is searched, so a pattern does not have to match the whole path. That makes short patterns broader than they look:

Pattern	Also matches	Because
build/	/srv/app/rebuild/out.js	the text build/ appears in rebuild/
.git/	/srv/legit/notes.txt	. is "any character" in a regex
bin/	/home/robin/tax.pdf	the text bin/ appears in robin/
$RECYCLE.BIN/	nothing at all	$ means "end of path", so this can never match

Anchor a directory name with (^|/) and end it with / to mean exactly that directory, at any depth:

(^|/)node_modules/
(^|/)build/
(^|/)\.git/

Those match /srv/app/build/out.js and leave /srv/app/rebuild/out.js alone. Escape any character a regex treats specially: . (any character), $ ^ (anchors), * + ? (repetition), ( ) [ ] { } | \.

To match by extension instead, anchor the end:

\.log$
\.(tmp|bak)$

Glob syntax is not accepted. *.log is not a valid regular expression and is refused; the pattern you want is \.log$.

Patterns that match everything

A pattern is searched for anywhere in the path, so a pattern that matches the empty string matches every path there is, and a scan carrying one excludes its whole tree. Those patterns are refused, naming the pattern and why:

• An empty pattern, from --exclude "$SKIP" with SKIP unset: the flag is still there, with nothing in it.
• A pattern ending in |, such as node_modules| — a list joined into one alternation with a trailing separator, leaving an empty branch.
• .*, a glob habit written as a regex. * already means "zero or more", so .* matches zero characters at the start of every path.
• A fragment left behind while editing a longer pattern: x?, ^, $.

Before this was refused, such a scan ran to completion and reported no PII found across 0 files — a clean result over a folder full of PII, and a clean record if it was saved. To narrow a scan, point it at a smaller path, or use --ext to name the extensions you want.

Note the one case this does not catch: . on its own matches any path with at least one character in it, which is also everything, but it does not match the empty string and so is accepted. Read the excluded counts (below) on any scan whose file count surprises you.

Common patterns

Development dependencies

(^|/)node_modules/
(^|/)vendor/
(^|/)\.bundle/
(^|/)packages/
(^|/)target/

Version control

(^|/)\.git/
(^|/)\.svn/
(^|/)\.hg/

Build output

(^|/)build/
(^|/)dist/
(^|/)out/
(^|/)obj/
(^|/)__pycache__/

IDE, editor, and system files

(^|/)\.idea/
(^|/)\.vscode/
(^|/)\.Trash/
(^|/)\$RECYCLE\.BIN/

The files PII Crawler owns are skipped automatically, by scan, textextract and watch alike, and you do not need a pattern for any of them: its data directory (~/.piicrawler), the log file it is writing as the scan runs (~/.local/state/piicrawler/piicrawler.log on Linux, ~/Library/Logs/PIICrawler/piicrawler.log on macOS, %LOCALAPPDATA%\PIICrawler\Logs\piicrawler.log on Windows), the --out file the run is streaming its results into, and the run folders a quarantine has filled, wherever you sent them. Naming one of them as the path to scan still scans it: skipping is for what happens to be in the way, not for what you asked for.

This matters most in watch mode, where the daemon writes a row to its own database for every violation it raises. piicrawler watch ~ used to treat that write as a file change, scan the database, alert on what it found there, and write more rows: the alert stream filled with one file nobody asked about and the database grew for as long as the daemon ran.

What your patterns excluded

An excluded file leaves no other trace, so the scan reports each pattern with the number of paths it removed:

  ✓  All clear   no PII found across 84 files  (2.1s)
     37 paths excluded by pattern
       (^|/)node_modules/  36
       (^|/)build/          1

Two numbers are worth reading:

• More than you expected means the pattern is broader than you meant, usually an unanchored directory name catching a longer one.
• Zero means the pattern matched nothing. Either there was nothing to exclude, or the pattern does not say what you thought it did.

The same figures appear on a saved scan's page in the web UI, beside each pattern: a pattern that matched nothing is labelled matched nothing rather than left to look effective.

On the CLI a matched directory is pruned, so the walk never descends into it and the whole directory counts as one excluded path. A scan started from the web UI or TUI counts excluded files individually.