Supported File Types
PII Crawler reads every plain-text file, plus the document, image, mail, archive and database formats below.
📄 Documents
| Format | Extensions |
|---|---|
| PDF (with OCR) | .pdf |
| Word | .docx, .doc, .dot |
| Excel | .xlsx, .xls, .xlsb |
| PowerPoint | .pptx |
| Rich Text Format | .rtf |
| OpenDocument text | .odt, .ott |
| OpenDocument spreadsheet | .ods |
| OpenDocument presentation | .odp, .otp |
| OpenDocument drawing | .odg, .otg |
| EPUB e-book | .epub |
| Apple Pages | .pages |
| Apple Keynote | .key |
| Apple Numbers | .numbers |
| Web page | .html, .htm, .xhtml, .shtml |
| Markdown, CSV, plain text | .md, .csv, .txt |
What gets read inside a document
Documents are scanned beyond the main body of the page. Sensitive values often sit in the parts of a file nobody looks at before sharing it.
A web page is read as the page a browser shows, not as the markup it is written in. That matters most for tables. A number is only reported as a card or a phone number when a supporting term sits near it, and in a table that term is the column heading, so a page whose markup was read literally reported the email addresses in an exported customer table and none of the card numbers. Reading the page instead puts each row's values under their own headings, the way the same table in a spreadsheet or a CSV is read.
It also keeps a value in one piece. A number with one bold digit in it, or a hyphen written as -, is one number on the page and is read as one number here. The same goes for a character written by name: a page that spells Hernández as Hernández, or an apostrophe as ’, holds the name the browser shows, so that is the name reported and searched for. Every character reference a browser understands is read the same way. Values that never appear as text are read too: the address behind a mailto: link, a tooltip, an image's alt text, the value already filled into a form field, and the contents of an HTML comment, which is where a commented-out record tends to sit.
Word documents are scanned including their headers and footers, footnotes and endnotes, reviewers' comments, text boxes, and text that was struck out with track changes but never accepted. That last one matters: deleting an SSN with track changes on leaves the SSN in the file until the revision is accepted, and anyone who opens the document can still read it. PowerPoint decks are scanned including speaker notes and slide comments, and Excel workbooks including cell comments and notes on every sheet, hidden sheets included.
Legacy Word documents, the .doc files Word saved before 2007 and still saves today when someone picks "Word 97-2003 Document", are covered as well, headers, footnotes, comments and text boxes included. So are the Word 6.0 and Word 95 files that turn up in long-lived archives. A file named .doc is often not a Word document at all: report tools write rich text, mail systems write HTML, and people rename spreadsheets. PII Crawler reads the file to see what it actually is and scans it accordingly, rather than trusting the name.
Apple Pages, Keynote and Numbers
Documents from Apple's iWork apps are scanned, including the ones people brought over from old Macs. Pages documents are read with their text boxes, headers, footers and footnotes; Keynote presentations with their speaker notes and slide comments; Numbers spreadsheets with their cell values, table names and the figures behind a chart. Reviewers' comments are read in all three.
Both generations of the format are covered: what Pages, Keynote and Numbers write today, and the older files saved by iWork '09 and earlier that are still sitting in shared folders.
A .key file is more often a private key or a config file than a Keynote presentation. PII Crawler reads the file to see what it actually is, so a .key that holds text is scanned as text.
macOS stores an iWork document as a bundle, which is a folder the Finder draws as a single file. When one is copied to a Windows server or a NAS it arrives as an ordinary folder, and PII Crawler scans it either way.
The address behind a link is scanned along with the text of the link. A "Contact payroll" hyperlink pointing at mailto:[email protected] yields the email address, which is invisible on the page.
Files pasted into a document
When someone pastes a spreadsheet, a PDF or another document into a Word file, the whole embedded file is stored inside it. A quarterly report can be nothing but a page of placeholders, with every figure and every name in the objects pasted onto the page.
PII Crawler reads those objects, in Word, Excel and PowerPoint files alike, in both the current formats and the legacy .doc and .xls. Whatever was pasted in is read with the same reader it would get on disk, so a pasted-in spreadsheet is read sheet by sheet and a pasted-in PDF is read page by page. Objects pasted into pasted-in files are read too, four levels deep by default.
Findings from an embedded object are reported against the document that carries it, the same way an email attachment's findings are reported against the message. When the object kept the name it had before it was embedded, that name appears with its contents, so payroll-statement.txt is visible in the results.
To change how deep this goes, set the PIICRAWLER_EMBEDDED_DEPTH environment variable before starting PII Crawler. Setting it to 0 turns embedded objects off entirely and scans only each document's own text.
PIICRAWLER_EMBEDDED_DEPTH=2 piicrawler scan ~/DocumentsOpenDocument files, the format LibreOffice and OpenOffice save in and Google Docs exports to, are covered the same way: page headers and footers, footnotes and endnotes, comments, table cells, text boxes, hidden sections, speaker notes on slides, and text deleted with track changes but not yet accepted. A spreadsheet or chart pasted into a document is a document in its own right inside the file, and it is scanned too.
PDFs are scanned beyond their visible page text. If a PDF has fillable form fields (an intake form, tax form, or direct deposit form, for example), the values typed into those fields are stored separately from the page text, and many scanners miss them. PII Crawler extracts every filled-in field along with its label, so an SSN typed into an "Employee/Worker Number" box is detected just like one appearing in the body of the document. Scanned PDFs with little or no embedded text automatically fall back to OCR.
Forms built with Adobe LiveCycle, which many government and bank forms are, keep what somebody typed in a separate XML record rather than in the fields above. Those values are read as well.
Comments, sticky notes and the text of a highlight or a review markup are scanned along with the page they sit on.
Files attached to a PDF
A PDF can carry whole files inside it: the spreadsheet clipped to an invoice, the documents bound together in a PDF Package or portfolio, the paperclip attachment stuck on a page. A portfolio in particular is only a cover sheet and its attachments, so its cover page says nothing about what it holds.
PII Crawler reads them. Each attached file goes to the reader its own contents call for, so an attached spreadsheet is read sheet by sheet, an attached PDF is read page by page, and an attached scan goes through OCR. Attachments inside attachments are read too, to the same depth as objects pasted into documents, and the attachment's file name is reported with its findings.
This is covered by the same PIICRAWLER_EMBEDDED_DEPTH setting described above, so setting it to 0 turns attachments off with everything else.
E-books are read chapter by chapter in reading order, including chapters left out of the table of contents.
Pictures inside documents
A scan pasted into a Word file, a screenshot on a slide, a photographed form sitting on a spreadsheet: the text in those pictures is read with OCR, the same way a picture on disk is. It applies to Word, Excel and PowerPoint files, OpenDocument files, EPUB books and Apple Pages, Keynote and Numbers documents, and to pictures inside documents that were themselves pasted into a document.
Pictures too small to hold text, such as bullets, icons and dividing rules, are skipped, and a picture repeated through a document is read once however many times it appears. To change the size floor, set PIICRAWLER_OCR_MIN_IMAGE_PIXELS to the smallest picture area worth reading, in pixels. The default is 4096, the area of a 64 by 64 picture. Setting it to 0 reads every picture in every document, which is slower.
PIICRAWLER_OCR_MIN_IMAGE_PIXELS=1024 piicrawler scan ~/DocumentsFindings from a picture are reported against the document that carries it. Running with --no-ocr skips pictures inside documents along with image files.
Password-protected documents and DRM-protected e-books cannot be read without the password. They are reported as protected, so they are never mistaken for files that were read and turned out to be clean.
🗃️ Archives
| Format | Extensions |
|---|---|
| ZIP | .zip |
| Java archive | .jar, .war, .ear |
| 7z | .7z |
| Tarball | .tar |
| Compressed tarball | .tar.gz, .tgz, .tar.bz2, .tbz2, .tar.xz, .txz |
| Compressed file | .gz, .bz2, .xz |
| Mail archive | .mbox |
.zip archives are flattened during enumeration: each supported file inside becomes its own scan unit, identified by a virtual path like bundle.zip!/reports/people.txt. Entries are scanned in parallel rather than one after another, and progress counts entries rather than the archive as a single file. Encrypted or corrupt zips are reported as a single error entry.
Tarballs work the same way, whichever compression they use, and so does a single compressed file such as a rotated log or a database dump. customers.csv.gz is scanned as the CSV inside it and reported as customers.csv.gz!/customers.csv, so results point at the file you would restore rather than at the compressed blob. When gzip recorded the original name, that name is used, which is what makes archive-2024.gz show up as archive-2024.gz!/payroll.csv.
Archives nested inside other archives are opened too: a .tar inside a .zip, a .gz inside a .tar, or a .zip pasted into a Word document. Nesting is followed four levels deep by default; set PIICRAWLER_EMBEDDED_DEPTH to change that, or to 0 to switch it off.
The contents decide how an archive is read, not its name. A file that a backup script wrote with bzip2 and named .tar.gz is read correctly, a backup.gz that turns out to hold a tar is expanded into the files inside it, and an ordinary export that somebody renamed to .gz is still scanned instead of being written off as corrupt.
In the web UI and TUI, virtual zip-entry paths are first-class: each entry has its own row in the file list, opens to its own findings page, and previews its decoded contents through the "Re-scan this file" action. The "delete from disk" action is disabled for entries inside an archive, since removing one entry would mean rewriting the whole .zip.
🖼️ Images (with OCR text extraction)
| Format | Extensions |
|---|---|
| JPEG | .jpg, .jpeg |
| PNG | .png |
| GIF | .gif |
| TIFF | .tiff, .tif |
| BMP | .bmp |
| WebP | .webp |
A screenshot or a scanned page is read with OCR, so an SSN in a picture dropped into a shared folder is found the same as one in a text file.
Multi-page images are read in full. A scanner or a fax that writes a whole job into one TIFF is read page by page, not just the cover sheet, and an animated GIF is read frame by frame. Set PIICRAWLER_OCR_MAX_PAGES to raise or lower the 64-page limit.
A photo taken with a phone is turned the right way up first, using the orientation the camera recorded, so a picture of a form reads as well as a flatbed scan. A page that was scanned sideways with no orientation recorded still reads poorly; rotate it before scanning.
Every TIFF a scanner or a fax writes is readable, whichever way it was compressed: Group 4, Group 3 (one and two dimensional), Modified Huffman, LZW, Deflate, PackBits, JPEG and uncompressed, in colour, greyscale, black and white, CMYK or colour-mapped.
An image that cannot be decoded, such as a truncated download, is reported as an error against that file rather than counted as scanned and empty.
✉️ Email / Messaging
| Format | Extensions |
|---|---|
| Email message | .eml, .emlx |
| Outlook message | .msg |
| Gmail / Thunderbird mailbox | .mbox |
| Outlook data file | .pst, .ost |
| Saved web page | .mht, .mhtml |
Attachments are read whether or not the sending program named them. A pasted screenshot, a forwarded message, a report a workflow tool generated: all three routinely arrive with no filename, and each one is opened by what its content type or its own bytes say it is. A forwarded message is scanned as a message, so the original's sender, recipients, subject, body and its own attachments are all covered, however many hops deep the thread goes.
.emlx is the single-message format Apple Mail stores on disk. .mht and .mhtml are what browsers write for "save page complete": one MIME file holding the page and everything it referenced, which is scanned the same way a message is.
.msg files are single Outlook messages, the format you get when you drag a message out of Outlook to your desktop or export individual items from an eDiscovery search. The sender, recipients, subject, and body are all scanned. Bodies are read whether they are stored as plain text, HTML, or rich text, so messages composed in Outlook's rich-text mode are covered. Attachments are scanned in place using the extractor for their own file type, meaning a spreadsheet attached to a message is scanned as a spreadsheet, and a message forwarded as an attachment is scanned as a message. Findings are attributed to the .msg file itself.
One limit worth knowing: if a message is encrypted, either with S/MIME or by a sensitivity label that applies encryption, the body is unreadable without the decryption rights and no PII will be found in it. Labels that only classify a message, without encrypting it, do not affect scanning.
.mbox archives are treated as containers: each message inside becomes its own scan unit, just like an entry inside a .zip. Headers (From, To, Cc, Bcc, Subject, Reply-To), bodies, and decoded attachments are all scanned. The file_path in JSONL output carries the message ordinal (and Message-ID: when present), for example mail.mbox::message-000042::<[email protected]>. Multi-gigabyte mailboxes such as Gmail Takeout exports are streamed rather than loaded into memory. See Scan Gmail for PII for a walkthrough.
📇 Contacts and calendars
| Format | Extensions |
|---|---|
| vCard contact | .vcf, .vcard |
| iCalendar | .ics, .ical, .ifb |
| vCalendar | .vcs |
A contacts export is one of the densest PII files an organisation has: names, home addresses, personal phone numbers, dates of birth, and whatever somebody typed into the notes field. Calendar exports carry the same thing in meeting descriptions.
Both formats wrap long values across lines. An exporter breaks a line at a fixed length, wherever that falls, and continues it on the next line behind a single space, so a phone number, an address or a national ID is routinely stored in two pieces. Those pieces are put back together before the file is scanned, which is what makes a value that straddles a break findable at all. Contacts exported from Outlook go a step further and store long values quoted-printable, hiding the punctuation inside them behind escapes like =2D; those are decoded too.
An export that was renamed, saved as .txt, or left with no extension is recognised by its opening BEGIN:VCARD or BEGIN:VCALENDAR line and read the same way. A .vcf that turns out to be a Variant Call Format genomics file, which is an unrelated format sharing the extension, is read as the plain text it is.
🗂️ Directory exports
| Format | Extensions |
|---|---|
| LDAP export | .ldif, .ldf |
An LDIF file is what ldifde writes on a Windows domain controller and what ldapsearch and slapcat write on OpenLDAP. It holds every user's name, title, office address, work and mobile numbers, and whatever an administrator typed into description or an extension attribute.
Two things in the format hide values from a literal read, and both are undone before the file is scanned. Any line may be folded at any position, so an exporter wrapping at a fixed column routinely stores a phone number or a national ID in two pieces. And a value that is not plain ASCII is base64 encoded behind a second colon, which is where every name with an accent in it ends up. Binary attributes such as objectGUID and thumbnailPhoto are base64 too, and those are left encoded rather than fed to a detector as raw bytes.
An export saved as .txt or with no extension is recognised by its opening version: or dn: line.
🗄️ Database files
| Format | Extensions |
|---|---|
| SQLite | .sqlite, .sqlite3, .db, .db3, .s3db, .sqlitedb |
| Microsoft Access | .mdb, .accdb |
| dBase and FoxPro | .dbf, and the .dbt and .fpt memo files beside them |
Database files are opened read-only, each user table is dumped to text, and the result is run through PII detection. Up to 1,000 rows are taken from each table, which is enough to tell you a table holds customer records without writing the whole table out again. When a table has more rows than that, the scan says so rather than leaving you to guess.
Files stored inside a database. Applications often keep whole documents in a database rather than on disk: the PDF a helpdesk attached to a ticket, a photographed licence from a signup form, a JSON payload nobody gave a column of its own. Those are read too. A stored file goes to the reader its contents call for, so a PDF is read as a PDF and a scanned image goes through OCR, and its findings are reported against the database file with the table, column and row they came from.
A column usually holds one kind of thing all the way down, so a few files are taken from each column rather than all of them, and identical copies are read once. The scan tells you which columns held more than it read, so you know where to look if a sample turns something up.
Access databases. Every user table is read, in all four generations of the format: Access 97, 2000, 2002-2003 and the .accdb files Access has written since 2007. Text, Long Text (memo) and number columns all come out, including memo values too long to sit in the row itself. System tables are skipped, so what you get is the data somebody typed rather than the database's own bookkeeping. Password-protected databases are reported as protected rather than scanned.
A Long Text column set to Rich Text stores formatting alongside the words, and the scan reads what the column shows rather than how it is stored. This matters more than it sounds: a value typed with one character in bold is stored with the formatting sitting in the middle of it, and read literally that would split a Social Security number in two and hide it. Plain Text columns are read exactly as they are stored, so a column that genuinely holds markup keeps it.
Attachment and OLE Object columns hold whole files, and those are read the same way a stored file in any other database is: the Word document somebody pasted into a record, the PDF attached to a row, the scanned form. The file goes to the reader its contents call for, and the same table, column and row are reported with its findings.
dBase and FoxPro tables. A .dbf is a table of people in fixed-width binary records, and accounting, payroll, shipping and point-of-sale packages have been writing them since the 1980s. Plenty still do, and plenty more have left one on a share. Every generation is read: dBase III and IV, dBase 5 and 7, FoxBASE, FoxPro 2 and Visual FoxPro. Column names come out with the rows, so the values keep the context that says a number is a phone number rather than an account balance, and dates, logical columns, currency and timestamps are read as what they are rather than as raw bytes.
Three things about these files are worth knowing, because each one hides data from a scanner that takes the file at face value:
- The long values are not in the table. A memo column stores a block number pointing into a separate file beside it, named
.dbtfor dBase and.fptfor FoxPro, and that file is where the notes somebody typed a card number into actually live. The memo file is read along with its table, so a note arrives with the account it belongs to. It is also scanned on its own, which is what covers a memo file copied without its table, or one inside an archive where the table cannot be reached. - A deleted record is not deleted. Deleting a row in dBase marks it and leaves every byte of it in the file until somebody packs the table, which is often never. Those records are read and reported, marked
(deleted)so you can tell them from live rows. A table somebody "cleaned up" years ago can still be full of national IDs. - The text is in a DOS code page. One byte in the header says which. Read as anything else, a name with an accent in it comes out as control characters, so
José Muñozin a table written under DOS is decoded as it was typed.
Up to 1,000 rows are read from a table, the same as for any other database, and the scan says when there were more. A .dbf that turns out to be a CSV or fixed-width text export, which is common because plenty of tools claim the extension, is read as what it actually is.
A SQLite database is also recognised by its contents, so the ones that carry no extension at all are scanned too: Chrome and Edge store browsing history, cookies and saved logins in files named History, Cookies and Login Data, and iOS backups are full of them. A file named .db that turns out to be something else, which is common because plenty of programs claim that extension, is read as whatever it actually is instead of being reported as unreadable.
👨💻 Code, config and data files
Source files, configuration, logs and data exports are text, so they are scanned as they are: .java, .c, .cpp, .py, .js, .php, .sh, .json, .xml, .yaml, .env, .sql, and any other text file, whatever its extension. A hard-coded SSN in a test fixture or a customer export sitting in a log is exactly the kind of thing worth finding.
Files with no extension at all are scanned too. README, Makefile, Dockerfile, a mail spool, an export somebody saved as customers with no suffix: all of them are read as text. A file with no extension that is not text is identified by its contents, so a database, document or archive that lost its name is still scanned as one. Anything else with no readable text in it, such as a folder of executables, is listed as skipped rather than reported as an error. That check happens while the file is scanned, not while files are being counted, so finding files stays fast even in folders full of extensionless files.
🪶 Very large files
A single text file can be enormous: a multi-gigabyte web server log, a database .sql dump, a giant .csv or .json export. Text files at or above 32 MB are read in bounded, overlapping windows rather than all at once, so memory stays flat however large the file is, and a value that lands on a window boundary is still found exactly once. A 100 GB log scans in the same memory footprint as a 100 MB one, over a network share as well as on local disk.
There is no size limit unless you set one. If you would rather skip large files than scan them, set a Max File Size on the scan (or pass --max-size <MB>) and anything above that limit is ignored. The scan reports how many files the limit skipped, so a clean result over a tree whose biggest export was passed over is never silent. A limit of zero is refused everywhere it can be typed — the field, the flag and the API — since it would skip every file with anything in it; leave the limit off for no limit.
🔤 Text encodings
Text files are read as UTF-8, which also covers plain ASCII, and as UTF-16, which matters on Windows: PowerShell's Out-File and > redirection, Excel's "Unicode Text (*.txt)" export, and Notepad's "Unicode" encoding all write UTF-16. Both byte orders are handled, at any size, including files written without a byte-order mark, which is what SQL Server's bcp -w and many .NET and Java exports produce. A CSV exported from Excel as Unicode Text is scanned the same as a UTF-8 one.
UTF-32 is not decoded. If you have UTF-32 text to scan, convert it first with iconv -f UTF-32 -t UTF-8.
📛 What is not scanned
Programs, media files, fonts, disk images and other formats that hold no readable text are skipped, along with formats not listed above.
A file that should have been readable but was not, because it is corrupt, password-protected, or genuinely binary despite its extension, is listed in the scan results with the reason. "No findings" against a file means it was read and nothing was found.
Occasionally a document is malformed in a way that makes a format reader fail badly. A file that defeats the reader is recorded as unreadable and the scan carries on, rather than the scan ending. This needs no configuration.