What 8 Million PERM Applications Tell Us About Who Gets Approved
A data engineering + statistical pipeline on a decade of US labor-certification data — built in a datathon window.
Overview
Cleaned, entity-resolved, and statistically analyzed a decade of US Department of Labor PERM filings — 8M+ applications, 1.5GB+ of raw data — to identify the factors most strongly associated with approval.
The PERM (Permanent Labor Certification) dataset is the first step in the US employment-based green card process. The Department of Labor publishes every application annually with hundreds of fields. It's an unusually rich dataset for asking who actually gets approved and why — but it's also messy in the specific ways that matter most for analysis: institution names spelled inconsistently across years, missing wage fields, outliers from clerical errors, and no clean linkage to external reference data like university rankings. This project built a multi-stage Python pipeline to clean, harmonize, and entity-resolve the raw data, then ran four parallel statistical cuts to identify approval correlates. Originally pitched at a 2025 UMich datathon centered on lender risk-modeling for international students; the analytical machinery is broader, and this page focuses on it.
Stack
The Question
A decade of public PERM data exists, but it's messy enough that most ad-hoc analyses don't trust it.
The DOL publishes every PERM application annually going back over a decade. In principle, that's enough to answer concrete questions: which employers approve most consistently, how wage-offered relative to prevailing wage affects outcomes, whether applicant background correlates with approval, how trends shift across administrations. In practice, the dataset has three structural problems that make naive analysis unreliable: (1) entity ambiguity — institution and employer names are inconsistent across years and require resolution to compare meaningfully; (2) schema drift — field schemas drift between years, so a column called the same thing in 2014 may not be the same in 2023; (3) outlier contamination — clerical-error outliers are mixed with real outliers from unusual filings, and treating them the same biases results.
Methodology
Pipeline-first design: independent stages for cleaning, entity resolution, merging, and analysis — slower to set up, but tunable without re-running the 1.5GB ingest each time.
| Stage | Method | Why this design |
|---|---|---|
| Cleaning | State-name normalization, schema harmonization | Field semantics drift between years; without harmonization, "the same column" is silently different across the decade. |
| Entity resolution | RapidFuzz against canonical US News ranking list | The same university appears with dozens of variant spellings. Without resolution, institutional-prestige analyses report spelling noise, not signal. |
| Merging | Joined cleaned PERM filings ↔ resolved institutional metadata | Allows downstream analyses to filter and group by ranking bucket, state, year — pre-computed once, queried four ways. |
| Analysis (4 parallel cuts) | Per-factor decomposition · YoY trend · institutional / wage-bracket comparison · outlier-sensitive distributions | Each script answers a separable question and can be re-run independently. Outlier-sensitive runs surface where naive averages were misleading. |
Pipeline
Each pipeline stage is its own script in the repo — one module, one responsibility. Decoupling cleaning from analysis is what made fuzzy-match iteration possible in a datathon window.
The fuzzy-match stage was the slowest and most error-prone part of the pipeline — without separating it from the ingest, every threshold tweak would have meant re-loading the 1.5GB CSV. Keeping stages independent was the design call that made the datathon timeline feasible.
Findings
Institutional prestige and wage-offered structure emerged as among the strongest correlates of approval — but the more interesting finding was how much the raw data misleads if entity resolution and outlier handling are skipped.
The substantive findings pointed to institutional prestige (after entity resolution to canonical names + ranking buckets) and the wage-offered field as strong correlates of approval outcomes, with notable variation across years. These are correlational findings, not causal — the dataset doesn't support claims about what causes approval in any individual case. The methodological finding ended up being equally important: a substantial fraction of the cross-year and cross-institution patterns that appear striking in the raw data are partly artifacts of inconsistent name spellings and outlier records. Naive analyses of PERM data that skip entity resolution risk reporting spelling noise as real signal.
Limitations
What this pipeline and the analyses on top of it cannot claim.
- Correlational, not causal. No counterfactual design — observed differences may reflect selection into PERM filings rather than causal effects of any covariate.
- Entity resolution was tuned by eye. RapidFuzz similarity thresholds were set against spot-checks during the datathon window. Precision/recall on a held-out labeled set isn't measured.
- Schema drift is partially handled. Fields renamed or restructured across years were reconciled where I caught them; silent changes that look like the same field but differ semantically may remain.
- Outlier handling is heuristic. Clerical errors and real-outlier filings are difficult to separate without case-level review. The pipeline flags but doesn't fully classify.
Lessons Learned
On large, messy, public datasets, the cleaning and entity-resolution stage is usually where the analysis wins or loses — not the choice of statistical method.
What worked. Separating the pipeline into independent scripts rather than one monolithic notebook. The fuzzy-matching stage took dozens of iterations to tune the similarity threshold and the canonical reference list; if every iteration had required rerunning the 1.5GB CSV ingest, the project wouldn't have finished in the datathon window.
What didn't. The analysis stops at correlation. With 8M+ records I had enough data to attempt causal inference techniques (matching, instrumental variables, regression discontinuity around wage-offered thresholds), but not enough time. The findings as-stated are useful as descriptive empirical work; they aren't enough to support claims about individual case outcomes.
What I'd do differently. Build the entity-resolution evaluation harness before tuning thresholds — a held-out test set of known matches lets you measure precision/recall directly rather than estimating by eye. Same principle as building model evaluation before model training: it saves days of guessing.