Private
Public Access
1
0

docs: add README, CLAUDE.md, and anonymised profiles.txt for public release

This commit is contained in:
belisards
2026-03-29 17:13:24 -03:00
parent 9996fa5f6d
commit 8db07570b0
3 changed files with 190 additions and 5 deletions

77
CLAUDE.md Normal file
View File

@@ -0,0 +1,77 @@
# Instagram Scraper — Agent Context
## What this project is
A Playwright-based Instagram scraper. Reads `profiles.txt`, visits each profile in a real Chromium browser, scrapes the last N posts, and writes `output.csv` + `output.md`.
## How to run
```bash
uv run python scraper.py # run the scraper
uv run pytest tests/ -v # run unit tests
uv run playwright install chromium # install browser (first time only)
```
Always use `uv run` — never `python` directly or `pip`.
## Key files
| File | Purpose |
|---|---|
| `scraper.py` | All logic: auth, profile scraping, post scraping, output |
| `profiles.txt` | Input: one Instagram URL per line |
| `auth_state.json` | Saved Playwright session (gitignored, created on first run) |
| `output.csv` | Scraped results (gitignored) |
| `output.md` | Scraped results in Markdown (gitignored) |
| `tests/test_parsers.py` | Unit tests for pure parsing functions |
## Architecture
Single script, no framework. Key functions in `scraper.py`:
- `extract_hashtags(text)` / `extract_mentions(text)` — pure regex, fully tested
- `profile_slug_from_url(url)` — extracts username from URL
- `read_profiles()` — reads `profiles.txt`, strips line number prefixes
- `is_logged_in(page)` / `ensure_authenticated(browser)` — session management
- `get_post_urls(page, profile_url)` — collects post links from profile grid
- `scrape_post(page, post_url, profile_slug)` — extracts all fields from a post
- `write_csv(posts)` / `write_markdown(posts)` — output writers
- `main()` — orchestrates everything
## Output fields
`profile`, `post_url`, `date` (ISO 8601), `caption`, `likes`, `image_urls` (comma-separated CDN URLs), `hashtags`, `mentions`, `location`, `media_type` (photo/video/carousel)
## DOM selectors (verified against live Instagram)
Instagram uses atomic CSS — class names change. These structural selectors are stable:
- **Post grid links**: `a[href*="/p/"]`
- **Date**: `time[datetime]``.get_attribute('datetime')` (first element = post date)
- **Caption**: JS tree walker over text nodes in `section` containing a link to the profile slug; filter `len > 20`, skip relative times
- **Likes**: First `span` with purely numeric text
- **Images**: JS `img[src*="cdninstagram"]` filtered by `width > 100` and excluding `/s150x150/`
- **Carousel**: `button[aria-label="Next"]` exists
- **Video**: `video` element exists
- **Location**: `a[href*="/explore/locations/"]` — filter out text "Locations" (footer link)
## Auth flow
1. Check for `auth_state.json` → load with `browser.new_context(storage_state=...)`
2. If missing or expired → open visible browser, wait for user to log in manually, save with `context.storage_state(path="auth_state.json")`
## Modifying behaviour
- Change number of posts: edit `POSTS_PER_PROFILE` constant at top of `scraper.py`
- Change input/output paths: edit `PROFILES_FILE`, `OUTPUT_CSV`, `OUTPUT_MD` constants
- To re-authenticate: delete `auth_state.json` and re-run
## Testing policy
Pure functions (`extract_hashtags`, `extract_mentions`, `profile_slug_from_url`, `read_profiles`) have unit tests. Browser functions are tested end-to-end only — do not mock the browser.
## Dependencies
- `playwright>=1.40.0` — browser automation
- `pytest>=7.0.0` (dev) — test runner
- Standard library only: `re`, `csv`, `sys`, `pathlib`, `itertools`