LinkedIn Scraper
Profile and post extraction to power persona capture inside Maetel's product.
Overview
A Python scraper that extracts LinkedIn profile data and up to 30 original posts per creator — built to support Maetel's persona-capture product, where the model needs a creator's actual writing voice and history to produce on-brand outputs.
Maetel's product captures a target user's "persona" from their LinkedIn presence so the system can generate content in their voice. That requires three things the LinkedIn UI doesn't expose cleanly: a structured profile snapshot, a representative sample of their original (non-reposted) writing, and metadata like engagement and timing. The scraper handles all three — Playwright with a persistent Chrome session for login, HTML extractors for profile and post content, snowflake-ID decoding for upload dates, and a normalized PostgreSQL schema (users and posts) with CSV fallback. It runs against a JSON list of target creators and dedupes on unique media IDs across runs.
Stack
The Problem
Persona-driven content generation needs the writer's actual voice and history. LinkedIn's UI doesn't expose that as structured data.
For Maetel's persona-capture model to produce on-brand content, it needs three signals that aren't available from the LinkedIn UI as clean data: (1) a structured profile snapshot — headline, follower count, key experience and education entries; (2) a representative sample of original (non-reposted) posts so the voice the model learns is the creator's own; and (3) metadata around each post — engagement, timing, content type. The scraper exists to produce that structured view consistently across many target creators, deduped, and ingestable into the downstream pipeline.
Design Choices
Three deliberate choices to keep the scraper maintainable, safe to operate, and useful as production input.
| Choice | Why |
|---|---|
| Playwright over raw HTTP | LinkedIn content is rendered client-side and gated behind authentication. A browser-based approach with a logged-in session was the only realistic option. |
| Persistent Chrome profile, not credential storage | The scraper never handles passwords. Login happens once, manually, in a visible browser; the session cookie is reused across subsequent runs. |
| Normalized schema + CSV fallback | PostgreSQL is the primary store for production. CSV mode supports development and small one-off collections without DB setup. |
| Reposts detected and labeled, not collected | The voice we want to capture is the creator's own — reposts contaminate the persona signal. Detection happens at the extractor stage. |
What It Does
- Profile extraction. Follower count, headline, up to 5 experience entries, up to 5 education entries.
- Post collection. Up to 30 original posts per creator (reposts detected and labeled separately), with content (preserved line breaks), likes, comments, media URLs, and decoded upload dates from snowflake IDs.
- Data persistence. PostgreSQL with normalized
usersandpoststables and CSV fallback; duplicate prevention via unique media IDs across runs. - Session management. Persistent Chrome profile so login happens once and is reused across runs; configurable scroll behavior and delays.
Limitations
- Subject to LinkedIn's UI changes. Extractors are tied to current page structure; LinkedIn DOM changes require extractor updates. This is the cost of any scraping approach against a non-public-API target.
- Rate-limited by design. Configurable delays prevent triggering anti-scraping detection. Throughput is intentionally modest — this is a per-creator persona collector, not a bulk crawler.
- 30-post cap per creator. The persona signal saturates well before unbounded collection. Diminishing returns past ~30 posts, and unbounded crawls are both unkind and operationally risky.
Lessons Learned
Persistent sessions beat credential handling. The choice to never touch passwords — login once, reuse the cookie — eliminated a whole category of operational risk and security review concerns. The same pattern (manual auth, automated session reuse) generalizes to many scraping-adjacent tasks.
Repost detection at the extractor, not the warehouse. Filtering reposts at collection means the downstream pipeline never has to wonder whether a given post is the creator's voice or someone else's. Pushing that distinction upstream simplifies every consumer.
Schema + CSV duality. Having both PostgreSQL and CSV outputs from day one meant the same scraper served production (DB) and dev/debugging (CSV) without a separate code path. Cheap optionality.