Network Propagation Score Cutoff
Replacing the arbitrary "top-N or raw-score threshold" with a permutation-based empirical p-value cutoff — degree-matched nulls, gold-standard validation, ~225 FinnGen R12 diseases.
Overview
Network propagation gives every gene a score. Picking which scores count as "significant" usually defaults to an arbitrary top-N or raw-score threshold — both of which fail under different network shapes. This framework replaces that with a per-gene empirical p-value from degree-matched permutations, then validates the resulting cutoff against external biology.
Random-walk-with-restart (personalized PageRank) over a protein–protein interaction network is a standard way to spread known disease-gene signal across the genome — every non-seed gene gets a "guilt-by-association" propagation score. The hard part isn't running the propagation; it's deciding which of those scores are signal vs. background. Picking top-50 or p < 0.05 on raw scores is convention, not statistics. This project replaces convention with a permutation framework: replace the seeds with degree-matched random gene sets, re-run propagation N_PERM times, build a per-gene null distribution, and report p = (#{null ≥ observed} + 1) / (N_PERM + 1). The output cutoff is then validated against OpenTargets associations and ChEMBL clinical precedence to confirm it captures real biology rather than network topology artifacts.
Stack
The Problem
Network propagation outputs a score per gene. The standard "cutoff" picks are conventional, not statistical — and they break under different network shapes.
Three common heuristics for "which propagated genes count": top-N (pick the highest scoring N), raw-score threshold (everyone above some absolute value), or top quantile. All three are insensitive to the structural fact that high-degree genes drift toward high propagation scores regardless of biology — they have more neighbors, so signal flows into them by topology alone. A defensible cutoff has to compare each gene's observed score to a null distribution that controls for its degree.
Methodology
For each disease: build a degree-matched null by permutation, compute per-gene empirical p-values, then validate the resulting cutoff externally.
| Step | What it does & why |
|---|---|
| 1. Seeds & weights | Known disease genes are rows where padj ≠ 0. Their padj values become the personalization weights for the PageRank-style propagation. |
| 2. Propagation | igraph page_rank, weighted and personalized over the combined STRING v11 + OTAR network. Every non-seed gene gets a propagation score. |
| 3. Degree-matched null | Replace seeds with N_PERM random gene sets selected by nearest-neighbor degree matching (not quantile bins — bins miss the local structure of degree distributions). Re-run propagation each time to build a per-gene null distribution. |
| 4. Empirical p-value | p = (#{null ≥ observed} + 1) / (N_PERM + 1). Genes with emp_pval < 0.05 are significant; diseases with very few seeds (n_seeds ≤ 2) are flagged low_confidence rather than silently included. |
N_PERM = 1000, the minimum possible p-value is 0.001. Bonferroni / BH against ~18,400 tested genes would multiply that to ≈ 18 — mathematically, nothing could survive. The empirical p-value is used directly as the cutoff, and the framework's defensibility comes from external validation (Method 3 below), not from a correction that can't apply at this permutation count.
Pipeline
One central R script produces the empirical p-value table. Five numbered analysis stages (w1–w5) consume it and answer separate validation questions.
Scale. Final runs use N_PERM = 10,000 across ~225 diseases on the parallel R script (mclapply auto-splits cores between disease and permutation parallelism). The single-threaded R script exists as a reference implementation at N_PERM = 1000 for quick tests; both share the same make_nearest_match_sampler() degree-matching routine. Reproducibility relies on set.seed(42) + mc.set.seed = TRUE in the parallel paths.
Validation
Five separable validation stages (w1–w5). Together they test whether the cutoff captures real biology, not just propagation-score topology.
| Stage | Question | Method |
|---|---|---|
| w1 | Does propagation score relate to empirical significance? | Per-disease Pearson correlation between page.rank and −log10(emp_pval); tested against seed count for reliability; gold-standard overlay on scatter; linearity test (NP(A+B) ≈ NP(A)+NP(B)?) to see if per-seed runs could replace per-disease runs. |
| w2 | Does the cutoff recover known disease genes? | Leave-One-Out: drop each seed in turn, re-run propagation + permutation on the remainder, check whether the held-out gene is recovered at emp_pval < 0.05. Parallelized per (disease, seed) pair. Also: 1K vs 10K permutations to quantify p-value sharpening. |
| w3 | Does the cutoff match external biology? | Pulls OpenTargets associations (GraphQL API, EFO/MONDO mapping) per disease and builds gold sets from globalScore + ChEMBL clinical precedence. Computes TP / FN / Novel / precision / sensitivity; sweeps cutoff combos; compares against top-N baseline. |
| w4 | Empirical-p-value cutoff vs. raw propagation rank? | Augments LOO results with the held-out gene's propagation rank (single propagation run, no permutation). Quantifies recovery by empirical p-value vs. recovery by raw rank. |
| w5 | Is the cutoff signal-bearing across the score range? | Splits genes into four quadrants by (propagation-score quantile × significance) and measures gold-standard density per quadrant. Hypergeometric enrichment test (pooled + per-trait). The "low propagation but significant" quadrant should still be gold-enriched; "low + non-significant" is the negative control. |
Findings
Active research output. Headline results land at the disease level — each of the ~225 FinnGen R12 endpoints gets its own per-gene emp_pval and the validation tells whether the cutoff is biologically meaningful for that disease.
The central output table — result_np_cutoff/np_cutoff_results.csv — has one row per (disease, gene) and is consumed by all five downstream analyses. Numerical headline metrics (gold-standard precision/sensitivity, LOO recovery rates) are still in active iteration as the OpenTargets/ChEMBL gold standards are refined and the per-disease results stabilize.
Limitations
- BH / Bonferroni can't apply at this permutation count. Empirical p-value floor is
1 / (N_PERM + 1) ≈ 1e-4; multiplied by ~18,400 genes, nothing survives multiple-testing correction mathematically. The framework's defensibility comes from external (Method 3) validation, not from a correction that can't apply. - Low-seed diseases are fragile. Diseases with
n_seeds ≤ 2are flaggedlow_confidence, but they're not removed — the user has to honor the flag downstream. A more aggressive policy would drop them outright. - Seed scope is FinnGen R12 only. Pipeline is built around FinnGen R12 endpoints. Adding UKB / MGI sources would require additional EFO/MONDO mapping work and re-running the full permutation pass.
- Gold standard is OpenTargets + ChEMBL. Both are imperfect ground truths; OpenTargets aggregates many evidence types with varying quality, and ChEMBL clinical precedence is weighted toward well-studied diseases. Cross-disease comparisons inherit those biases.
- Compute cost. 10K permutations × ~225 diseases requires the parallel R script and a real server. Single-threaded runs at 1K permutations exist for quick tests but produce noticeably noisier p-values.
Lessons Learned
The two decisions that made the framework defensible: degree-matched (not quantile-bin) sampling, and external validation as the only legitimate decision metric.
Degree matching matters more than people think. A propagation null built from quantile-binned degree (e.g. "high-degree" vs. "low-degree" buckets) misses the local structure of the degree distribution and undercorrects for the bias it's supposed to control. Nearest-neighbor degree matching forces each permuted seed to have a degree close to a real seed — that's what makes the null comparable to the observed signal.
External validation is non-negotiable. Inside the framework, every internal metric (reconstruction quality, score-vs-significance correlation, top-N agreement) is somewhat circular — they all derive from the same matrix and the same propagation. Only the OpenTargets / ChEMBL gold standards are independent of the implementation choices. That's why w3 ranks methodology choices and w1/w2 don't.
What I'd do differently. Build the validation harness (Method 3 / w3) at the same time as the core permutation framework, not after. Several iterations of the permutation script (parallelization strategy, low_confidence flagging, ID schema) would have been informed earlier if the gold-standard precision/sensitivity numbers were already producing.