Exclusion Patterns

Overview

Exclusion patterns keep paths out of a scan. Use them to skip dependency folders, build output, version control data, or anything else you do not want scanned.

A pattern is a regular expression matched against the whole path, not a glob and not a directory name. node_modules matches any path containing that text; \.log$ matches any path ending in .log.

Excluded files are never opened, never scanned, and never stored, so they do not appear in the file list, the counts, or the results. Because an exclusion silently makes a scan smaller, every scan reports what each pattern kept out. See What your patterns excluded.

Setting exclusion patterns

Patterns belong to a scan and are set when the scan is created.

• Web UI: open New Scan, expand File Filtering, and enter one pattern per line under Exclusion Patterns.
• TUI: the exclusions field on the form that starts the scan, comma separated. Each of the three forms (local, network share, mailbox) has its own, and a scan runs with the patterns typed into the form that started it.
• CLI: --exclude, repeatable, on scan, smb, and imap:

piicrawler scan ~/share --exclude '(^|/)node_modules/' --exclude '(^|/)\.git/'

A pattern that is not a valid regular expression is refused before the scan starts, naming the pattern and the reason. To change the patterns on an existing scan, create a new scan with the patterns you want.

Writing a pattern

The whole path is searched, so a pattern does not have to match the whole path. That makes short patterns broader than they look:

Pattern	Also matches	Because
build/	/srv/app/rebuild/out.js	the text build/ appears in rebuild/
.git/	/srv/legit/notes.txt	. is "any character" in a regex
bin/	/home/robin/tax.pdf	the text bin/ appears in robin/
$RECYCLE.BIN/	nothing at all	$ means "end of path", so this can never match

Anchor a directory name with (^|/) and end it with / to mean exactly that directory, at any depth:

(^|/)node_modules/
(^|/)build/
(^|/)\.git/

Those match /srv/app/build/out.js and leave /srv/app/rebuild/out.js alone. Escape any character a regex treats specially: . (any character), $ ^ (anchors), * + ? (repetition), ( ) [ ] { } | \.

To match by extension instead, anchor the end:

\.log$
\.(tmp|bak)$

Glob syntax is not accepted. *.log is not a valid regular expression and is refused; the pattern you want is \.log$.

Common patterns

Development dependencies

(^|/)node_modules/
(^|/)vendor/
(^|/)\.bundle/
(^|/)packages/
(^|/)target/

Version control

(^|/)\.git/
(^|/)\.svn/
(^|/)\.hg/

Build output

(^|/)build/
(^|/)dist/
(^|/)out/
(^|/)obj/
(^|/)__pycache__/

IDE, editor, and system files

(^|/)\.idea/
(^|/)\.vscode/
(^|/)\.Trash/
(^|/)\$RECYCLE\.BIN/

PII Crawler's own data directory, ~/.piicrawler, is skipped automatically. You do not need a pattern for it.

What your patterns excluded

An excluded file leaves no other trace, so the scan reports each pattern with the number of paths it removed:

  ✓  All clear   no PII found across 84 files  (2.1s)
     37 paths excluded by pattern
       (^|/)node_modules/  36
       (^|/)build/          1

Two numbers are worth reading:

• More than you expected means the pattern is broader than you meant, usually an unanchored directory name catching a longer one.
• Zero means the pattern matched nothing. Either there was nothing to exclude, or the pattern does not say what you thought it did.

The same figures appear on a saved scan's page in the web UI, beside each pattern: a pattern that matched nothing is labelled matched nothing rather than left to look effective.

On the CLI a matched directory is pruned, so the walk never descends into it and the whole directory counts as one excluded path. A scan started from the web UI or TUI counts excluded files individually.