You must find every Social Security number (SSN) on a file share. The reason can be an audit, a breach-response plan, a data-retention cleanup or a merger. The share has thousands of folders and nobody knows what is in them.
The first attempt is usually a text search for the pattern NNN-NN-NNNN. On most shares, that search finds a few numbers in text files and CSV exports. It does not find the SSNs in Word documents, Excel workbooks, PDFs or ZIP files. On a typical file share, those formats hold most of the files.
This guide shows why a text search misses these SSNs, which problems a real scan must solve, and how to do the scan without copying the data to a cloud service.
Why a text search finds so few SSNs
We made a small test share with four files. Each file contains at least one SSN. The SSNs are made-up numbers that pass the issue rules.
| File | Where the SSN is |
|---|---|
HR/notes.txt |
Plain text: Her SSN is 212-45-6781 |
HR/offer-letter.docx |
Word document: SSN 212-45-6780 |
HR/payroll-2019.xlsx |
Excel column headed SSN, shown as 021-34-5678 |
HR/old/2018-archive.zip |
A CSV file in a ZIP file: 301-22-4455 |
Then we did the usual search:
$ grep -rlE '[0-9]{3}-[0-9]{2}-[0-9]{4}' /srv/share
/srv/share/HR/notes.txt
The search found one file of four. The other three files contain SSNs, but the search did not see them. Windows findstr and PowerShell Select-String have the same problem, because they also read the raw bytes of each file. There are five causes.
1. Office files are compressed
A .docx or .xlsx file is a ZIP file of XML parts. The text is compressed, so the bytes on the disk do not contain 212-45-6780. A search of the raw bytes cannot find it.
You can search inside the ZIP with zipgrep, but this is not a full solution. Word can divide one number between two parts of the XML, for example after an edit or a spelling check. A pattern on the XML then does not match the number.
2. Excel stores SSNs as numbers
Many HR and payroll workbooks use Excel's Social Security Number format, 000-00-0000. The cell shows 021-34-5678, but the file keeps the value as the number 21345678. There are no dashes, and the leading zero is gone:
$ unzip -p HR/payroll-2019.xlsx xl/worksheets/sheet1.xml
... <c r="B2" s="1"><v>21345678</v></c> ...
A pattern for NNN-NN-NNNN cannot match this value. A pattern for nine digits cannot match it either, because it has only eight digits. To find it, a scanner must read the workbook as a table, see the SSN header, and examine every cell in that column.
3. PDFs and images need OCR
A text PDF also keeps its text in compressed streams. A scanned PDF is worse: it is a picture of a page, and it has no text at all. W-2 forms, I-9 forms and benefit forms often arrive as scans. Only optical character recognition (OCR) can read the SSNs in them.
4. Archives and mailboxes hide files
ZIP, 7z and tar files contain more files. Outlook PST files and .mbox files contain thousands of messages. A file name such as 2018-archive.zip tells you nothing about its contents. A scan must open each archive and mailbox and examine each item in it.
5. A nine-digit search finds too much
Not all SSNs have dashes. Some systems write 212456781. But a search for any nine digits also finds order numbers, ZIP+4 codes, phone numbers and account numbers. On a real share, this search gives thousands of results. Nobody can examine all of them, so the real SSNs are lost in the noise.
A better pattern for SSNs with dashes
The Social Security Administration never issues some numbers. The area number (the first three digits) is never 000, 666 or 900 to 999. The group number (the middle two digits) is never 00. The serial number (the last four digits) is never 0000. This pattern skips those numbers. It needs grep -P (PCRE):
grep -rnP '\b(?!000|666|9\d\d)\d{3}-(?!00)\d{2}-(?!0000)\d{4}\b' /srv/share
On our test share, the pattern found two lines in notes.txt:
HR/notes.txt:1:Called about W-2. Her SSN is 212-45-6781.
HR/notes.txt:3:Example on the form: 123-45-6789
The second line is not a real SSN. 123-45-6789 is the example number on many forms, and the SSA has never issued it. Expect many of these examples on a real share. Our PII regex patterns post has more SSN patterns, for example a version for engines that do not support lookaheads.
A better pattern removes some false results. It does not solve the first four problems. It still cannot read Office files, Excel numbers, scanned PDFs or archives.
What a real SSN scan must do
Use this list to examine a scanner, or your own script, before you trust its result:
- Read the text in Word, Excel, PowerPoint and PDF files, not their raw bytes.
- Read spreadsheets as tables, and find SSNs that Excel stores as numbers in a column headed SSN.
- Use OCR on scanned PDFs and images.
- Open ZIP files, other archives and mailboxes, and examine each item.
- Skip numbers that the SSA never issues, and skip the usual form examples.
- Report nine digits without dashes only when a word such as "SSN" or "Social Security" is near them.
- Record the files that it could not read. A locked file is not a clean file.
- Keep the results on your own network. The results contain the SSNs.
The last two items are important for an audit. If a scan cannot open a folder, the folder looks clean in the results. If the scan sends file content to a cloud service, you made a new copy of the SSNs while you looked for them.
How to scan a file share for SSNs with PII Crawler
PII Crawler does the eight steps in the list above, with one limit for Outlook data files that the next section gives. It runs on your own computer, so no file content goes to a cloud service.
Scan the share
To scan an SMB share without mounting it, use the smb command. Put the password in the PIICRAWLER_SMB_PASSWORD environment variable, so that it does not go into your shell history. --only ssn limits the scan to Social Security numbers:
piicrawler smb fileserver HR -u svc-scan --domain CORP --only ssn --name "SSN sweep: HR share"
The command saves the results and writes the scan ID to standard output. If the share is already mounted, or if you scan a local folder, use scan with --save:
piicrawler scan /mnt/hr --only ssn --save --name "SSN sweep: HR share"
OCR is on by default. The scan reads Office files, PDFs, images, archives, Outlook PST and OST files, .msg, .eml and .mbox files, and Access and SQLite databases. The supported file types page has the full list. One limit: in PST and OST files, the scan reads the subject, body, sender and recipients of each message, but it does not read attachments. A .msg file is read with its attachments. To scan several shares, run one command for each share. Our guide to scanning shared drives and NAS devices shows how to make a list of shares and how to get read access to them.
How PII Crawler decides that a number is an SSN
- A number with dashes, such as
212-45-6781, is reported in all locations. Numbers with a never-issued area, group or serial are not reported. - Nine digits without dashes are reported only when a label is near them. Labels include
SSN,SS#,Social Security,Soc Sec,Tax IDandTIN. - In a spreadsheet or CSV column headed SSN, every cell is examined. Seven- and eight-digit values in that column are reported as SSNs that lost their leading zeros.
- Placeholders are not reported: counting digits such as
123-45-6789, and numbers such as111-22-3333, where each group is one repeated digit. - Digits inside a file hash, a serial code or a timestamp are not reported.
The data types reference gives the full rules.
On our test share, these rules found all five SSNs in the four files, including the two in the Excel column. They did not report the form example 123-45-6789. They did report one false result: the order number 412557890 in notes.txt, because the word "SSN" is on the line above it. Expect some false results of this type, and mark them in the next step.
Examine the results
Open the scan in the web UI (piicrawler serve) or in the terminal UI (piicrawler). Mark each false result as a false positive. The mark stays on the finding, so the next scan does not ask you again. See triaging findings.
Then decide what to do with each file. You can delete it, move it to a quarantine folder or redact the SSNs in it. Action lists show a preview before they change a file. Delete and quarantine also work on SMB shares.
Give the auditor proof of the scan
An auditor usually wants to know three things: what you scanned, what you found, and what you did about it. Two commands give you this proof:
piicrawler report 42 --out hr-share-ssn-report.html
piicrawler export 42 --out hr-share-ssn-findings.csv
Replace 42 with your scan ID. piicrawler scans shows the IDs.
- The HTML report masks every SSN, so you can attach it to an email. It shows the risk score, the findings for each file, and a "Could Not Be Read" section. That section lists each file and folder that the scan could not open, with the reason.
- The CSV export has one row for each finding. It includes the
verdictcolumn, which records whether a person marked the finding as true or false. The export contains the full SSNs, so keep it as safe as the share.
After you remove the SSNs, scan the share again and keep the second report. Two reports, before and after, show that the cleanup worked. When you do not need the results any more, piicrawler scans delete 42 removes the scan and its stored findings.
Frequently asked questions
Can Windows Explorer search find SSNs on a share?
Not reliably. A search for the pattern in raw file content has the problems described above. It cannot find SSNs that Excel stores as numbers, SSNs in scanned PDFs, or SSNs in files inside a ZIP file.
How long does an SSN scan of a file share take?
That depends on the number of files, the network and the number of scanned PDFs. OCR is the slowest step. To make a large scan faster, scan from a computer near the file server, or use --no-ocr for a first pass and scan the image folders later.
Does the scan change my files?
No. A scan only reads files. Files change only when you run an action list, and the action list shows a preview first.
What about ITINs?
An Individual Taxpayer Identification Number (ITIN) has the same shape as an SSN, but its first digit is 9. PII Crawler does not report numbers that start with 9 as SSNs. To find ITINs, add your own pattern with --regex. Our regex post has an ITIN pattern.