# When creating a new app

> # 🔴 LEGACY — on the new host (CT 116) this checklist gains a step 0.5, and step "push to deploy" is gone
>
> Since the 2026-08-02 rebuild onto self-hosted **CT 116**, two things below no longer hold:
>
> 1. **A new app must first choose its AUDIENCE**, because nginx is now **two** vhosts, not one:
>    **public** (through the Cloudflare Tunnel — `adampowell.pro`/`www`) or **tailnet-only**
>    (`app-ct.tail94c6c8.ts.net` / `app.lan`). Anything confidential goes in the tailnet vhost, where
>    it is *structurally* unreachable from the internet rather than relying on the login gate. Adding
>    a route to the wrong file is how something private becomes public — decide before you write it.
> 2. **Push-to-deploy reaches CT 116 within 5 minutes** (`ap-deploy.timer`: read-only deploy key →
>    ff-only merge → `rebuild-symlinks.sh` → restarts `remote`/`rosters`/`janda` → ntfy). No webhook
>    there, so no instant deploy, no CF cache purge, no deploy email. A new systemd backend still
>    needs its restart line added — the timer restarts only those three.
>
> Also: apps run as the non-root **`app`** user, secrets go in `/etc/app/secrets.env` (not
> `/root/secrets.env`), and new units should use `Restart=on-failure` + `StartLimitBurst`, not
> `Restart=always`. Current-state truth: `c:\dev\homelab\HOMELAB_MASTER_INFRASTRUCTURE.md`.

A practical checklist for adding a new application to `adampowell.pro` without breaking anything. Derived from the 2026-04-10 restructure work — every step here exists because of a mistake we made or nearly made at some point.

---

## 0. Decide what KIND of app it is

This dictates almost everything else:

| Kind | Examples | What you need |
|---|---|---|
| **Static HTML/CSS/JS** | POCUS cardiac-anes, mac, shift-calc, Blake & Powell Photography | Just files in `apps/<name>/`, a symlink from `html/`, a nav.json entry. Zero backend. |
| **PHP** | nda, purchase-orders, bp, mickey, photography | Same as static + a PHP-handling nginx location block (`location ~ \.php$`) or falls through to the existing `location ~ ^/(bp\|mickey\|photo)/.*\.php$` regex if you squeeze it into one of those. |
| **Node backend only (API)** | case-tracker-api | Source in `apps/server/<name>/`, a `systemd` unit file, an `nginx` proxy_pass block. |
| **Node backend + static PWA frontend** | KVH PULSE, Atlas | Two pieces — `apps/<name>/` for the frontend, `apps/server/<name>/` for the backend. |
| **Node backend + Node frontend (same process)** | janda | One place under `apps/<name>/`, `pm2` or `systemd`, an `nginx` proxy_pass block. |
| **Go single-binary + embedded SPA** | command-center | Source in `apps/<name>/` (Go + Vite SPA via `go:embed`); build the web bundle then `go build`; a `systemd` unit runs the binary; an `nginx` proxy_pass block. The droplet has **no CGO toolchain** — build pure-Go (e.g. `-tags no_duckdb`, `modernc.org/sqlite`). |

If you don't know which one, stop and pick. Shoving a PHP thing into a Node path causes hours of pain.

---

## 1. Put the source in the right place

```
/var/www/adampowell.pro/
├── apps/
│   ├── <newapp>/                 ← static / frontend / self-contained Node
│   └── server/
│       └── <newapp>/             ← pure backend service (systemd-managed)
└── html/
    └── <newapp>                  ← symlink into apps/<newapp> (see step 3)
```

**NEVER** put a live app at the repo root. Everything lives under `apps/`. The repo root is for `github-webhook.js`, `.gitignore`, `.git/`, and nothing else.

**NEVER** put a backend outside the repo** in `/var/www/<thing>/` like the old layout. Backends belong in `apps/server/<name>/`. The 2026-04-10 Tier 4 move documented the pain of fixing the previous setup and there is no excuse to repeat it.

---

## 2. Handle secrets correctly — this is the most important step

The repo has had the same OpenAI key committed in 11 different files for months before the 2026-04-10 audit. Don't let it happen again.

### Rules

1. **No literal secret strings in tracked files. Ever.** Not in `.js`, not in markdown, not in systemd units, not in `pm2 ecosystem` files, not in `OLD BACKUPS/`.
2. **Secrets live in one place and one place only:** `/root/secrets.env` on the server. Mode `600`, root-only. Not in git. (See SYSTEM.md §9 for the current key inventory — names only.) Command-center is the exception: it stores its own service credentials **age-encrypted** in its SQLite, keyed by `/etc/command-center/age.key`.
3. **Services read secrets via `EnvironmentFile=/root/secrets.env`** in their systemd unit (`[Service]` section). Node code reads them via `process.env.X`.
4. **For pm2-managed processes**, use `pm2 set <app>:<var> <value>` to set per-process env, then `pm2 restart <app> --update-env`. Verify with `pm2 env <id> | grep X`.
5. **If a secret MUST be accessible from a client-side script** (like the OpenAI key for the investment calculator's GPT verify button), put it in a server-only JS file at `/var/www/adampowell.pro/html/<app>/secrets.js` loaded by `<script src="secrets.js">`. That file is gitignored by pattern (`*/secrets.js`). The HTML page itself is gated by nginx auth_request so only authenticated users can fetch it.

### Adding a new secret

```bash
ssh root@198.211.114.12 '
  echo "
# ----- <new service> secret -----
# Used by: apps/server/<name>/
# Rotation: <how>
<NEW_VAR>=<value>
" >> /root/secrets.env
'
```

Then either:

- Add `EnvironmentFile=/root/secrets.env` to the systemd unit's `[Service]` section (one line, idempotent — check if it's already there). Do **not** use `Environment=` for the secret; that puts the literal in the unit file which is readable by any user.
- Or `pm2 set <app>:<NEW_VAR> <value>` for pm2-managed processes.

### Pre-commit secret check

Before **every** commit that touches a new app, run this grep against the staged files:

```bash
cd /c/ADAMANT/ADAMPOWELL\ PRO/adampowell.pro
git diff --cached | grep -iE 'sk-[a-z0-9_]{30,}|sk-proj-|-----BEGIN|api[_-]?key.*=.*[a-z0-9]{20,}|password.*=.*[a-z0-9]{10,}|secret.*=.*[a-z0-9]{20,}|bearer [a-z0-9]{20,}|[A-Za-z0-9+/]{40,}=' | head
```

If that prints anything, **do not commit**. Fix the leak first.

---

## 3. Wire it into the web root

All live URLs on the site come from symlinks inside `/var/www/adampowell.pro/html/`. Never write actual content files directly into `html/` except `index.html`, `favicon.ico`, `nav.json`, `shift-calc/`, and the server-only `secrets.js` files.

### Add your app to `scripts/rebuild-symlinks.sh`

Edit the script and add one `link` line for your new app:

```bash
link "$HTML/<newapp>" "$(resolve <newapp>)"
```

For nested targets (like `pulse/client/dist`):

```bash
root="$(resolve <newapp>)"
[ -n "$root" ] && link "$HTML/<newapp>" "$root/<subpath>"
```

The `resolve` helper tries `apps/<newapp>` first, then falls back to a legacy top-level path. If your app is only ever at `apps/<newapp>` (which it should be), `resolve` still works.

Commit the change to `rebuild-symlinks.sh`. The webhook runs the script automatically on the next push. Verify the symlink was created afterward:

```bash
ssh root@198.211.114.12 'ls -la /var/www/adampowell.pro/html/<newapp>'
```

It should point at `/var/www/adampowell.pro/apps/<newapp>` (or the nested subpath).

---

## 4. nginx — only if you need a new location

**You probably don't need to touch nginx.** The existing HTTPS catch-all `location /` handles any static file under `html/` (including your new symlinked directory), and it already has auth_request gating + the `@login_redirect` error page. So a new static or PHP app that lives under `html/<newapp>` via symlink **just works** — auth-gated, served with the right content type, no nginx edit.

You only need an nginx edit if:

- **You have a backend that needs a proxy_pass** (port 3XXX → `/<newapp>/`). Add one location block, `nginx -t`, reload. Always back up first:
  ```bash
  ssh root@198.211.114.12 'cp /etc/nginx/sites-enabled/adampowell.pro /root/nginx-backups/adampowell.pro.pre-<newapp>.$(date +%Y%m%d_%H%M%S)'
  ```
- **Your app has special caching requirements** that the default catch-all doesn't cover.
- **You want the app publicly accessible without login** (rare — most things should be gated). In that case you add a specific location block WITHOUT `auth_request`, placed BEFORE the catch-all.

**Always update BOTH `/etc/nginx/sites-available/adampowell.pro` AND `/etc/nginx/sites-enabled/adampowell.pro`.** They are separate file copies on this server, not a symlink. If you only update one, `sudo nginx -t` will pass against the wrong file and the reload will use the other.

Test the change:

```bash
ssh root@198.211.114.12 'nginx -t && systemctl reload nginx'
```

If `nginx -t` fails, restore from the backup immediately and do NOT reload.

---

## 5. systemd unit — only if you have a backend service

For a Node backend in `apps/server/<newapp>/`, create `/etc/systemd/system/<newapp>.service`:

```ini
[Unit]
Description=<human-readable description>
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/var/www/adampowell.pro/apps/server/<newapp>
EnvironmentFile=/root/secrets.env
Environment=NODE_ENV=production
Environment=PORT=30XX
ExecStart=/usr/bin/node /var/www/adampowell.pro/apps/server/<newapp>/server.js
Restart=on-failure
RestartSec=10
StandardOutput=append:/var/log/<newapp>-app.log
StandardError=append:/var/log/<newapp>-error.log

[Install]
WantedBy=multi-user.target
```

Pick a **new, unused port**. Current port map (verified 2026-06-06 — see SYSTEM.md §3/§4 for live status):

| Port | Service | Status |
|---|---|---|
| 3001 | (retired — was maps) | gone |
| 3002 | messenger.service | inactive |
| 3003 | auth.service | active |
| 3004 | pm2 janda-app | errored |
| 3005 | pm2 github-webhook-adampowell | active |
| 3006 | kvh-pulse.service | inactive |
| 3007 | kvh-pulse-webhook.service | inactive |
| 3008 | todos.service | active |
| 3009 | randemail.service | active |
| 3010 | remote.service | active |
| 3011 | petfeed.service | inactive |
| 3013 | atlas.service (broker) | active |
| 3014 | sharedtodo.service | inactive |
| 3015 | command-center.service (Go) | active |
| 3016 | npa.service (NPA/NDA e-signing) | active (added 2026-06-30) |
| 3017 | rosters.service (AHA class rosters + signup) | active (added 2026-06-30) |
| 3100 | case-tracker.service | active |
| 3110 / 3111 / 3112 | ekg-tutor api / tutor / interpret | inactive |
| 8081 | AdGuard Home | inactive |

Use `ss -tlnp | grep :30XX` to confirm a port is free before assigning it. (A port whose service is
"inactive" is still claimed — pick a genuinely unused number.) **Bind new backends to `127.0.0.1`**, not
`0.0.0.0` — there is no host firewall (ufw is inactive), so a `0.0.0.0` bind is internet‑exposed on the raw IP.

**Critical:** do NOT use `Restart=always; RestartSec=10` without a backoff. The previous maps and aiomail services burned 830,358 and 717,260 restarts respectively before we noticed. Use `Restart=on-failure; RestartSec=10; StartLimitIntervalSec=60; StartLimitBurst=3` so a broken service stops trying after 3 failures in a minute and you get a loud failure instead of a silent resource drain.

**Critical — deploy restart:** the GitHub webhook only restarts pm2 `janda-app`. A **systemd** node backend will NOT
pick up its own `server.js` changes on deploy unless `scripts/rebuild-symlinks.sh` restarts it. So for every new
systemd node app, add a line near the bottom of that script:
`systemctl restart <newapp>.service 2>/dev/null || true`. Skip this and your code deploys to disk but the running
process keeps serving the old build — the classic symptom is a **route you just added returning 404 in prod** even
though it's in git (this bit rosters' `resend-followup` route on 2026-07-01). `remote.service` + `rosters.service`
already have their lines there as the pattern to copy.

Enable and start:

```bash
ssh root@198.211.114.12 '
  systemctl daemon-reload
  systemctl enable <newapp>.service
  systemctl start <newapp>.service
  sleep 1
  systemctl is-active <newapp>.service
  ss -tlnp | grep :30XX
'
```

---

## 6. pm2 — only for apps you want pm2 to manage instead of systemd

systemd is preferred for anything new. pm2 is currently only used for janda-app and the github-webhook because they were set up that way before. New services go in systemd unless you have a specific reason.

If you really need pm2:

```bash
ssh root@198.211.114.12 '
  cd /var/www/adampowell.pro
  pm2 start apps/<newapp>/server.js --name <newapp> --update-env
  pm2 save   # persist across reboots via pm2-startup
'
```

Set env vars via `pm2 set <newapp>:<VAR> <value>` BEFORE starting. Never bake secrets into `ecosystem.config.js`.

---

## 7. Add it to `nav.json`

`html/nav.json` is tracked in git (as of 2026-04-10, not server-only anymore). Edit it locally, commit, push. The webhook deploys it and Cloudflare gets purged automatically.

```json
{
  "name": "New App",
  "icon": "🔧",
  "desc": "Short description",
  "href": "/<newapp>/"
}
```

Put it in the right category (Medical Tools, Finance, Personal, Utilities, Business) or add a new category. The landing page renders categories in array order.

If you're linking to a URL-gated page (like POCUS or investing), include the key in the `href`:

```json
"href": "/<newapp>/?key=<long-random-hex>"
```

Never put login credentials in nav.json. Keys that just open a public-behind-auth page (like the POCUS cardiac-anes URL key) are OK because everything is already behind the main `/login` gate.

Purge Cloudflare cache for `nav.json` after pushing if you want the change visible immediately instead of waiting for TTL:

```bash
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/b41bcfddcd002e9d2841aa6591abd66e/purge_cache" \
  -H "Authorization: Bearer $CF_PURGE_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"files":["https://adampowell.pro/nav.json"]}'
```

(`CF_PURGE_TOKEN` is stored in `/root/secrets.env` as `CF_API_TOKEN`.)

---

## 8. `.gitignore` any runtime state

Before your first commit, add gitignore rules for anything the app writes at runtime:

- Log files: covered by the global `*.log`
- SQLite databases: covered by the global `*.db`, `*.sqlite*`
- Uploads / user content: add an explicit rule for the path
- `.env` files: covered by the global `.env`, `.env.*` (with a `!.env.example` exception)
- `node_modules/`: covered globally
- Any app-specific cache / temp dirs

Example addition for a new app that writes to `apps/server/<newapp>/uploads/`:

```gitignore
# <newapp> runtime user uploads (server-only, NEVER commit)
apps/server/<newapp>/uploads/
apps/server/<newapp>/cache/
```

---

## 9. Pre-push checklist

Before `git push`, run through this:

- [ ] No hardcoded secrets in any staged file (run the grep from section 2)
- [ ] `.gitignore` covers every runtime state path
- [ ] `nginx -t` passes (if you touched nginx)
- [ ] `node -c apps/server/<newapp>/server.js` passes (if Node backend)
- [ ] `systemctl start <newapp>.service && systemctl is-active <newapp>.service` = `active` (if new service)
- [ ] Port listener check: `ss -tlnp | grep :30XX` shows your process
- [ ] Direct smoke test: `curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:30XX/<some-endpoint>`
- [ ] Via-nginx smoke test: `curl -s -o /dev/null -w '%{http_code}\n' https://adampowell.pro/<newapp>/` (expect 302 unauthenticated)
- [ ] Authenticated smoke test with a real cookie (see `docs/SYSTEM.md` for the curl recipe)
- [ ] `nav.json` entry added in the correct category
- [ ] `rebuild-symlinks.sh` updated (if you need a new symlink)

---

## 10. After the push

There are **two** deploy paths and both are live: the GitHub webhook (push-triggered) and a
`git-auto-pull-adampowell.timer` that runs `git pull origin main` every 5 minutes. So a push lands within
5 minutes even if the webhook misses it — but it also means the server tree must stay clean (a plain
`git pull` will conflict on a dirty tree just like the webhook's `--ff-only`). Watch the webhook deploy:

```bash
# Watch the webhook deploy the commit
ssh root@198.211.114.12 'tail -f /var/log/github-webhook-adampowell.log'
```

You should see:

```
[timestamp] Received push event from t3h28/adampowell.pro
[timestamp] Executing git pull...
  rebuild-symlinks: N ok, 0 warn, 0 err
[timestamp] Git pull completed successfully
[timestamp] Email sent: 250 2.0.0 OK ...
```

If `rebuild-symlinks` reports warnings or errors, SSH in and run the script manually to see why:

```bash
ssh root@198.211.114.12 'bash /var/www/adampowell.pro/scripts/rebuild-symlinks.sh'
```

After the deploy, run the full authenticated smoke test (see `docs/SYSTEM.md` → Smoke Test section) and verify your new app returns `200` when authenticated and `302` when not.

---

## 11. Document it

Add a section to `docs/SYSTEM.md` under Application Inventory:

```
### N. <App Name>
- **Domain:** https://adampowell.pro/<newapp>/
- **Source:** /var/www/adampowell.pro/apps/<newapp>/
- **Backend:** /var/www/adampowell.pro/apps/server/<newapp>/ (if applicable)
- **Type:** Static / PHP / Node.js / Node + PWA
- **Port:** 30XX (localhost only, proxied by nginx) / N/A (static)
- **Service:** systemd `<newapp>.service` / pm2 `<newapp>` / N/A
- **Status:** ✅ Fully operational
- **Features:** …
- **Auth:** Main adampowell.pro login + optional URL key gate
- **Added:** YYYY-MM-DD
```

Also update the services table at the top of `docs/SYSTEM.md` if you added a new port.

---

## Anti-patterns (things NOT to do)

Every one of these has burned someone in this repo's history:

1. **Hardcoding secrets in source files.** Always `process.env.X`. Always.
2. **Putting runtime state in a tracked file.** Databases, logs, session files, user uploads, chat history, photo uploads — gitignore all of them.
3. **Editing the live file on the server without committing the same change locally.** The next `git pull` will either revert your change or fail to merge because of local modifications. Always: edit local → commit → push → let the webhook deploy.
4. **Using `git stash` on the server.** The 2026-04-10 near-miss incident where `git stash clear` destroyed untracked state should never be repeated. The webhook explicitly uses `git fetch + merge --ff-only` to avoid stash entirely. If a merge fails, investigate and fix; don't paper over it.
5. **Restarting `auth.service` repeatedly.** Since SESSION_SECRET now comes from `/root/secrets.env`, this is less dangerous than it used to be, but unnecessary restarts still kick everyone's in-memory session state via the SQLiteStore's garbage collection. Restart only when necessary.
6. **Putting backend services outside the repo** (back in `/var/www/<name>/` the old way). They belong in `apps/server/<name>/` so they're versioned.
7. **`Restart=always; RestartSec=10`** on a broken service. Use `Restart=on-failure` + `StartLimitBurst=3` so a broken service stops trying instead of burning 800k restarts over months.
8. **`action=""` forms that rely on JS preventDefault for submit** without verifying the JS actually runs. The whole "page flashes and refreshes" login bug was symptoms of this kind of issue (though the actual cause was different). Always set `action="/auth/api/login"` and `method="POST"` on the `<form>` element as a belt-and-braces fallback.
9. **Two locations with the same content.** The maps/case-tracker/pulse/janda-node/etc. debacles all stemmed from having two live copies that drifted apart. One canonical location, period. Everything else is a symlink to it.
10. **Committing `.env` files.** Covered by gitignore, but double-check before every commit.
11. **Committing ML model weights to the repo.** For client‑side ML (in‑browser models, e.g. transformers.js/
    Whisper — see `apps/preop/`), load the model + runtime from a public CDN at runtime (dynamic ESM `import()`
    of e.g. `@huggingface/transformers`) and let the **browser** cache the weights. **NEVER commit weights** —
    they're tens–hundreds of MB and balloon the monorepo + the 88%‑full server disk; deploy stays a symlink +
    nav.json tile, no nginx/port change. After the one‑time download the app runs offline. **transformers.js
    gotcha (Debian/iOS):** the **q8 dtype fails to load on the WASM/ORT‑Web backend** (the q8 QDQ Whisper decoder
    throws `TransposeDQWeightsForMatMulNBits / Missing required scale`). Use a best‑first backend chain instead:
    **WASM + decoder `q4`** (4‑bit MatMulNBits, ~25 MB, no WebGPU dependency) first, then **WebGPU + `fp32`** only
    if `navigator.gpu.requestAdapter()` succeeds, then **WASM + `fp32`** as the universal fallback — wrap each
    `pipeline()` init in a timeout and fall through on failure.

---

## Quick reference: a 100%-new static app in 5 minutes

```bash
# 1. Create the source dir
cd /c/ADAMANT/ADAMPOWELL\ PRO/adampowell.pro
mkdir -p apps/newapp
echo '<!DOCTYPE html><html><body><h1>Hello</h1></body></html>' > apps/newapp/index.html

# 2. Add to scripts/rebuild-symlinks.sh (one line)
#    link "$HTML/newapp" "$(resolve newapp)"

# 3. Add to html/nav.json (one tile in the right category)

# 4. Pre-push secret audit
git diff --cached | grep -iE 'sk-[a-z0-9_]{30,}|api[_-]?key|password|secret' | head
# (expect no output)

# 5. Commit + push
git add apps/newapp scripts/rebuild-symlinks.sh html/nav.json
git commit -m "Add newapp — <description>"
git push origin main

# 6. Watch the deploy
ssh root@198.211.114.12 'tail -f /var/log/github-webhook-adampowell.log'
# Look for "rebuild-symlinks: N ok, 0 warn, 0 err"

# 7. Verify
COOKIE=$(curl -sD - --max-time 15 -H 'Content-Type: application/json' \
  -d '{"username":"adam","password":"<your-password>"}' \
  https://adampowell.pro/auth/api/login | awk '/^[Ss]et-[Cc]ookie:.*apsess/ {sub(/^[Ss]et-[Cc]ookie: /, ""); sub(/;.*/, ""); print}')
curl -s -H "Cookie: $COOKIE" -o /dev/null -w '%{http_code}\n' https://adampowell.pro/newapp/
# Expect: 200

curl -s -o /dev/null -w '%{http_code}\n' https://adampowell.pro/newapp/
# Expect: 302 (auth-gated)
```

Done.

---

**Last updated:** 2026-06-26 (added anti-pattern #11 on client-side ML / in-browser model loading — see `apps/preop/`; port map + app-kind table + secrets + deploy reality were refreshed against the live server 2026-06-06; see SYSTEM.md / SERVER_NGINX_ROUTES.md for the full current state)
**Maintainer:** Adam Powell