devplacepy/README.md

# DevPlace - The Developer Social Network

Server-rendered social network for developers. FastAPI backend serving Jinja2 templates with ES6 interactivity. Avatar generation via Multiavatar (local, no network). SQLite via `dataset` with WAL mode and concurrency tuning.

[![CI](https://git.example.com/api/badges/DevPlace/ci/status.svg)](.gitea/workflows/test.yaml)

## Quick start

```bash
make install        # pip install -e .
make dev            # uvicorn --reload on port 10500
make test           # Playwright integration + unit tests, headless, fail-fast
make test-headed    # same tests in visible browser
```

Open `http://localhost:10500`.

## Stack

| Layer | Technology |
|-------|-----------|
| Backend | Python 3.13+, FastAPI, Uvicorn (single worker) |
| Templates | Jinja2 (server-side rendered) |
| Frontend | Pure ES6 JavaScript, one class per file |
| Database | SQLite via `dataset` (auto-sync schema, `uid` PKs, WAL mode, 30s busy timeout) |
| Auth | Session cookies, SHA256+SALT via passlib |
| Avatars | Multiavatar (local SVG generation, no external API, <5ms) |
| Validation | `hawk` (Python/JS/CSS/HTML) |
| Load testing | Locust (locustfile.py) |

## Project structure

```
devplacepy/
  main.py             # FastAPI app, router registration
  config.py           # Settings from env vars + .env
  database.py         # dataset connection, index creation
  templating.py       # Shared Jinja2 environment + globals
  avatar.py           # Multiavatar generation, URL builder
  utils.py            # Password hashing, session mgmt, time_ago, notification hook
  models.py           # Pydantic schemas
  push.py             # Web push crypto, VAPID keys, encrypt/send/register
  routers/            # One file per domain (auth, feed, posts, push, ...)
  templates/          # Jinja2 HTML templates
  static/css/         # Page-specific CSS files
  static/js/          # Application.js (ES6 module)
  services/           # Background service framework (base, manager, news)
```

## Routes

| Prefix | Purpose |
|--------|---------|
| `/auth` | Signup, login, logout, forgot/reset password |
| `/feed` | Post feed with topic/tab filtering (public) |
| `/news` | Developer news listing, detail page with comments |
| `/posts` | Post detail, creation |
| `/comments` | Comment creation, deletion |
| `/projects` | Project listing, creation |
| `/profile` | Profile view, editing |
| `/messages` | Direct messaging |
| `/notifications` | Notification list, mark read |
| `/votes` | Upvote/downvote on posts, comments, projects |
| `/follow` | Follow/unfollow users |
| `/leaderboard` | Contributor ranking by total stars earned |
| `/avatar` | Multiavatar proxy with in-memory cache |
| `/bugs` | Bug reports listing, creation |
| `/services` | Background service monitoring (status, logs) |
| `/admin` | Admin panel (user management, news curation, settings) |
| `(none)` | `/robots.txt`, `/sitemap.xml` (SEO) |
| `(none)` | `/push.json` (VAPID key + subscribe), `/service-worker.js`, `/manifest.json` (push + PWA) |

## Gamification

Member progression is driven by activity and peer recognition.

- **Stars** are the net vote score (`upvotes - downvotes`) on a post, project, or gist. A member's total stars is the sum across all their content and is the basis for ranking.
- **XP and levels.** Members earn XP for contributing: posting (10), commenting (2), publishing a project (15) or gist (5), receiving an upvote (5), and gaining a follower (5). Each level requires 100 XP (`level = 1 + xp // 100`). The profile shows the current level and progress to the next.
- **Badges** are awarded once per milestone: first post/comment/project/gist, 10 posts (Prolific), 25 stars (Rising Star), 100 stars (Star Author), 10 followers (Popular), and reaching levels 5 and 10. Badges render with an icon and description on the profile.
- **Leaderboard** (`/leaderboard`) ranks the top 50 members by total stars (single page, no pagination); a member's own rank is shown on their profile.
- **Reward notifications** fire when a member levels up or earns a badge.

XP awards are wired at the existing content-creation, vote, and follow hook points in the routers and centralized in `award_xp()` / `check_milestone_badges()` (`devplacepy/utils.py`). Existing accounts have their XP and levels backfilled once from prior activity at startup (`init_db()`).

## Configuration

| Env var | Default | Purpose |
|---------|---------|---------|
| `DEVPLACE_DATABASE_URL` | `sqlite:///devplace.db` | Database connection string |
| `SECRET_KEY` | hardcoded fallback | Session signing key |
| `DEVPLACE_VAPID_SUB` | `mailto:retoor@molodetz.nl` | Contact address in the VAPID JWT `sub` claim |

## Background Services

`devplacepy/services/` provides a generic async service framework. Services start on server boot, run at configurable intervals, and are gracefully cancelled on shutdown.

### Service framework

- **`BaseService`** - abstract class with async run loop, `deque(maxlen=20)` log buffer, `name`, `interval_seconds`
- **`ServiceManager`** - singleton that registers, starts, and stops all services
- **`NewsService`** - fetches news from `news.app.molodetz.nl/api`, grades with AI, stores articles >= threshold

### Adding a service

Create a class extending `BaseService`, override `run_once()`, register in `main.py`:
```python
service_manager.register(YourService())
```
The service appears at `/services` with live status and log tail.

### News service

Fetches news from a configurable API, grades each article via AI, stores ALL articles in the `news` table (nothing is silently skipped). Articles with grade >= the configurable threshold are auto-published; the rest go to draft. Admin can manually publish/draft, toggle for landing page appearance, and delete. Images are extracted from article URLs.

Configuration via admin site settings:

| Setting key | Default | Purpose |
|-------------|---------|---------|
| `news_grade_threshold` | `7` | Minimum AI grade for auto-publish |
| `news_api_url` | `https://news.app.molodetz.nl/api` | News source |
| `news_ai_url` | `https://openai.app.molodetz.nl/v1/chat/completions` | AI grading endpoint |
| `news_ai_model` | `molodetz` | AI model |

News articles have detail pages at `/news/{slug}` with full comment support (same component as posts/projects). The landing page can display curated articles toggled from admin.

CLI: `devplace news clear` - delete all news from local database.

## Push notifications & PWA

Authenticated users can receive native web push notifications, and the site is an
installable Progressive Web App. Push uses only standard libraries (`cryptography`,
`PyJWT`, `httpx`) against the Web Push Protocol — no third-party push wrapper.

### Events

Every event that already produces an in-app notification also sends a web push,
because both share a single funnel — `create_notification()` in `utils.py`:

| Event | Recipient |
|-------|-----------|
| Direct message received | receiver |
| Comment on your post | post author |
| Reply to your comment | comment author |
| `@mention` in any content | mentioned user |
| Upvote on your content | content owner |
| New follower | followed user |

`create_notification` schedules delivery as a fire-and-forget async task, so a dead
subscription or push-service error never blocks the triggering request. Delivery
(`push.notify_user`) iterates a user's subscriptions, encrypts the payload
(legacy `aesgcm` content encoding), and POSTs to each endpoint; subscriptions that
return `404`/`410` are soft-deleted.

### VAPID keys

The server identity is three PEM files generated once at startup in the repository
root: `notification-private.pem`, `notification-private.pkcs8.pem`,
`notification-public.pem`. They are git-ignored.

**These keys are the application's identity to the push services. If they are lost or
regenerated, every existing subscription becomes permanently undeliverable.** Persist
them across deployments and back them up; do not regenerate them.

### Opt-in

The browser requires the first permission prompt to originate from a user gesture, so
opt-in is exposed as a button in both the top navigation (bell-with-slash icon) and on
the `/notifications` page. After opt-in, the subscription is refreshed silently on
every page load. `PushManager.js` owns registration, subscription, and the opt-in UI.

### PWA

`manifest.json` (192/512 and maskable icons), `service-worker.js`, and an install
button (`PwaInstaller.js`) make the app installable. The service worker uses a
network-first strategy for navigations and falls back to `static/offline.html` when
offline. Installation requires a secure origin (HTTPS, or `localhost` for development).

| File | Role |
|------|------|
| `devplacepy/push.py` | VAPID keys, payload encryption, send, register |
| `devplacepy/routers/push.py` | `/push.json`, `/service-worker.js`, `/manifest.json` |
| `static/js/PushManager.js` | Service-worker registration + subscribe + opt-in UI |
| `static/js/PwaInstaller.js` | `beforeinstallprompt` capture + install button |
| `static/service-worker.js` | Receives push, shows notification, offline fallback |
| `static/manifest.json` | PWA manifest (icons, display, theme) |
| `static/offline.html` | Offline fallback page |

## Database

SQLite via `dataset` with production-oriented pragmas set on every connection:

```sql
PRAGMA journal_mode=WAL;      -- concurrent readers + writers
PRAGMA synchronous=NORMAL;     -- safe with WAL, faster than FULL
PRAGMA busy_timeout=30000;     -- wait 30s instead of failing on lock
PRAGMA cache_size=-8000;       -- 8MB page cache
PRAGMA temp_store=MEMORY;      -- temp tables in memory
PRAGMA mmap_size=268435456;    -- 256MB memory map for reads
```

All indexes are created via `CREATE INDEX IF NOT EXISTS` wrapped in try/except - safe to run on every startup regardless of table state.

## Testing

- **274 tests** across 23 files: Playwright integration + unit tests
- Playwright (NOT pytest-playwright plugin - conflicts, uninstall it)
- Server starts as subprocess on port 10501 with isolated temp database
- Test users `alice_test` / `bob_test` seeded via HTTP at session start
- Tests stop at first failure (`-x` flag)
- Failure screenshots auto-save to `/tmp/devplace_test_screenshots/`
- Headed mode: `PLAYWRIGHT_HEADLESS=0 make test`

### Key test patterns

- Every `page.goto()` and `page.wait_for_url()` uses `wait_until="domcontentloaded"` - avatar images don't block test execution
- Session-scoped browser context with per-test cookie clearing
- `page.locator(...).wait_for(state="visible")` preferred over bare selectors

## Avatars

Uses [Multiavatar](https://github.com/multiavatar/multiavatar-python) - generates deterministic SVG avatars locally from a seed string (the username). No external API calls, no network dependency. Generation takes <5ms. Results are cached in-memory (cleared on server restart).

Avatar URL format: `/avatar/multiavatar/{username}?size={size}`

## CI/CD

Gitea Actions workflow at `.gitea/workflows/test.yaml`:
- Runs on push/PR to main
- Sets up Python 3.13, installs dependencies + Playwright
- Validates all source files with `hawk`
- Runs all tests with fail-fast
- Uploads failure screenshots as artifacts

## Feature workflow

1. Implement the feature (router + template + CSS + JS)
2. `hawk .` - validate all source files
3. `make test` - run all tests (fail-fast)
4. Add Playwright tests in `tests/test_*.py` for new functionality
5. For visual features: `falcon take --output /tmp/verify.png && falcon describe /tmp/verify.png`
6. Update `AGENTS.md` and `README.md` if new conventions were introduced

## License

MIT - DevPlace