Scan Email Over IMAP

PII Crawler can scan an email account directly over IMAP. It connects to the mail server, lists your folders, and scans every message where it lives: each message is fetched into memory, its body and attachments are scanned, and the bytes are discarded. Nothing is downloaded to disk, so scanning your mailbox never creates a second copy of your PII.

The mailbox is opened read-only. PII Crawler uses the IMAP EXAMINE command and BODY.PEEK fetches, which means your messages are never marked as read, no flags change, and nothing is moved or deleted. Your mail client will not notice anything happened.

This works with any standard IMAP server: Fastmail, Gmail (with an app password), iCloud, Zoho, Dovecot, Courier, a NAS mail server, and most corporate mail systems that still expose IMAP.

What you'll need

• The IMAP server hostname (for example imap.fastmail.com or imap.gmail.com). Your provider's help pages list it, usually under "IMAP settings".
• The port: 993 for regular IMAP over TLS (the default), or 143 with STARTTLS.
• Your username, usually your full email address.
• An app password in most cases. Providers with two-factor authentication (Gmail, Fastmail, iCloud, Zoho) do not accept your normal account password over IMAP; they let you generate a per-app password instead:
  • Gmail: myaccount.google.com, Security, 2-Step Verification, App passwords. If your organization has disabled app passwords, use the Google Takeout export path instead.
  • Fastmail: Settings, Privacy & Security, Integrations, New app password.
  • iCloud: appleid.apple.com, Sign-In and Security, App-Specific Passwords.
• Microsoft 365 / Outlook.com: Microsoft has retired basic authentication for IMAP, so most Microsoft accounts cannot connect this way. Exchange Online support is on the roadmap.

How your password is protected

Scans started from the TUI or web UI store the account password in the same encrypted credential store used for network shares, protected by a credential password you choose. Someone who steals the database file cannot read the password without it. See How credentials are protected for the full details, including the no-recovery warning.

CLI scans never persist the password at all: it is read from an environment variable, handed to the scan engine in memory, and forgotten when the process exits.

The connection always uses TLS with certificate validation. There is no insecure mode. On Linux and Windows, if your mail server uses a self-signed certificate, install that certificate into the operating system trust store and PII Crawler will accept it. On macOS, PII Crawler validates against its own bundled set of publicly trusted certificate authorities and does not read the system Keychain, so servers with self-signed or private-CA certificates are not supported there.

Which folders are scanned

By default, every selectable folder is scanned, including Junk and Trash. That is deliberate: leaked PII loves to hide in spam replies and deleted mail.

Virtual duplicate views are skipped automatically so messages are not scanned twice. On Gmail that means All Mail, Important, and Starred are excluded; the real folders (Inbox, Sent Mail, labels) cover everything they contain.

You can narrow the scan to specific folders in every interface. Selecting a folder also includes everything nested under it, so choosing Work covers Work/Clients too. Folder names are matched exactly, with one exception: INBOX is matched whatever its case, because the IMAP protocol reserves that name and makes it case-insensitive, so inbox and Inbox reach the same mailbox.

A selection that would leave nothing to read is refused before the scan starts, and the message lists the account's folders so you can see the name you meant. That covers a folder the account does not have, a selection made up only of folders the scan skips (such as a Gmail duplicate view), and excludes that between them cover the whole account — --exclude-folder INBOX --exclude-folder Sent --exclude-folder Archive on a three-folder account, or a single --exclude-folder "[Gmail]" where the account nests everything under one name. This is the same rule the network-share scanner follows for a share it cannot reach: a scan fails up front rather than reporting an empty result.

Scanning from the web UI

1. Open Create Scan and choose the Email (IMAP) scan type.
2. Enter the server, username, and app password. Check Use STARTTLS only if your server uses port 143.
3. Click Test & List Folders. On success you get a checkbox list of your folders, with duplicate views unchecked. Adjust the selection if you want a narrower scan; the line under the list counts what is ticked. Unticking every folder is refused rather than read as "scan them all" — leave them ticked for the whole account.
4. Adjust Network Scan Performance Options if your provider needs a gentler pace, or your own server can take a faster one. See Being polite to your mail server.
5. Start the scan. Progress, findings, and the files view work exactly like any other scan; each message appears as its own entry named after its folder, message number, and subject.

Scanning from the TUI

1. From the scan list, press M (new email scan). If the credential store is locked you'll be prompted to unlock it first.
2. Fill in the server, username, and password. Alt+S toggles STARTTLS; Alt+C cycles through previously saved email accounts.
3. Optionally list folders to include (comma-separated) and adjust the throttle fields.
4. Optionally fill in Exclude Patterns, an Extensions Filter, and a Max File Size. These apply to the attachments the scan reads, and are recorded on the scan, so resuming it skips the same ones. A pattern that is not a valid regular expression, or a size the field cannot read, is refused while the form is still up.
5. Pick the data types to look for in the PII Types panel on the right: Up/Down moves the cursor, Alt+Space turns the type under it on or off. It starts on the default set.
6. Press Enter to start. You land on the scan detail view with live progress.

On a short window the form shows the fields it has room for and scrolls to the one you are typing into, and its bottom border says which part of the list you are looking at (fields 1-5 of 8). Alt+C scrolls to the saved-account selector as it cycles it.

The filters and the PII-type selection belong to this form, and are recorded on the scan, so its page reports them and a resume repeats them.

Scanning from the CLI

export PIICRAWLER_IMAP_PASSWORD='your-app-password'
piicrawler imap imap.fastmail.com -u [email protected]

The CLI prints the saved scan's ID to stdout, so it chains into reports and triage:

SCAN_ID=$(piicrawler imap imap.fastmail.com -u [email protected] --quiet)
piicrawler report "$SCAN_ID"
piicrawler findings list --scan "$SCAN_ID"

Scan only some folders, or skip one:

piicrawler imap imap.fastmail.com -u [email protected] --folders INBOX,Archive
piicrawler imap imap.fastmail.com -u [email protected] --exclude-folder Newsletters

See CLI Reference for every flag.

What gets scanned inside a message

Each message is scanned as a complete email: headers (From, To, Cc, Subject), the text and HTML bodies, and every attachment, using the same extraction pipeline as file scans. An HTML body is read as the message a reader sees rather than as its markup, so a table in one keeps its column headings and the card numbers under them are reported. A PDF or spreadsheet attached to a five-year-old email gets the same treatment as one sitting in a folder, including OCR for scanned documents and images (disable with the OCR option if you don't need it).

One caveat: .pst attachments (Outlook archive files, rare inside mail) are briefly written to a temporary file during extraction because the PST parser requires a file on disk. Every other format is processed fully in memory.

Being polite to your mail server

Mail providers limit how many IMAP connections a client may open (Gmail allows 15). PII Crawler opens at most one connection per fetch worker, which defaults to 2 concurrent fetches with a 100 ms delay between messages. That is gentle enough for any provider. If you are scanning your own mail server, you can raise the concurrency and drop the delay in the scan's performance options.

The three throttle settings are on the mailbox form in all three interfaces: the web UI's Network Scan Performance Options panel, the TUI's throttle row, and the CLI's --max-concurrent, --delay-ms, and --bandwidth-mbps flags. They are recorded on the scan, so resuming or re-running a mailbox scan reads at the pace you set rather than falling back to the defaults.

Rescanning

Every scan is a full pass over the selected folders. Messages that have not changed still benefit from the content cache within a scan, but there is no incremental "only new mail since last time" mode yet; it is on the roadmap along with OAuth sign-in.

Troubleshooting

• No password given or The password given is empty: the scan stopped before connecting, because --password was absent or blank and PIICRAWLER_IMAP_PASSWORD was empty or unset. IMAP has no guest login, so a blank password is refused rather than sent. Check that the variable is exported in the shell you are actually running in: sudo and most cron setups drop it.
• Authentication failed: double-check the username and confirm you are using an app password, not your account password. Gmail requires 2-Step Verification to be on before app passwords appear.
• Certificate verification failed: your server presented a certificate PII Crawler does not trust. On Linux and Windows, install the server's certificate (or your company's CA) into the OS trust store. On macOS, only publicly trusted certificate authorities are accepted, so use a mail server with a publicly issued certificate. PII Crawler intentionally has no option to skip certificate checks.
• Could not connect: the message ends with the reason, such as a refused connection or a hostname that does not resolve. Check the hostname and port, and whether the network between you and the server allows IMAP (some corporate networks block port 993/143 outbound).
• ... would read no mail: the folders left after your includes and excludes are all ones the scan skips, so there would be nothing to scan. The message names each one and why it was skipped (excluded, or a duplicate view), and lists the account's folders. Drop the exclude that went too wide.
• No folder on this account is named ...: the folder you asked for is not on the server, and the message lists the folders that are. Check the spelling, and note that nesting is always written with / here whatever separator your server uses. INBOX is the one name matched case-insensitively; every other folder name has to match exactly.
• A folder shows as an error: your account lacks permission to open it (common with shared or system folders). The rest of the scan is unaffected; the folder is recorded in the files view so you know it was skipped.