honestly its been months, i found unstaged changes and forgot

docs/.gitignore

@@ -0,0 +1 @@
+demo_files_do_not_delete/

docs/compose-update-policy.md

@@ -7,8 +7,7 @@ upstream files are reference snapshots only and must not be deployed directly.
 
 Each Docker-backed role has:
 
-- `templates/compose.yaml.j2`: base deployed Compose file
-- `templates/compose.override.yaml.j2`: NBOPS-specific local override
+- `templates/compose.yaml.j2`: deployed Compose template
 - `templates/env.j2`: generated environment file
 - `upstream/`: copied upstream reference files
 
@@ -49,19 +48,28 @@ Entries that use `readme_paths` or `compose_paths` must include both `url` and
 
    The script also looks for Compose-shaped fenced YAML blocks in README files.
    When exactly one candidate is found, `--apply` writes it as
-   `compose.from-readme.yaml`. Compose YAML fetched from `compose_urls` is
-   written as `compose.from-url.yaml`, or `compose.from-url-N.yaml` when a
-   repository has multiple fallback URLs. A role-level `local_compose_file` is
-   copied under `roles/<role>/upstream/local/`.
+   `docker-compose.yml` inside that upstream purpose directory. Compose YAML
+   copied from `compose_paths` or fetched from `compose_urls` is also normalized
+   to `docker-compose.yml`. URL sources are fallback material and do not
+   overwrite a Compose file already staged from the repository or README parse.
+   A role-level `local_compose_file` is copied under `roles/<role>/upstream/local/`.
+
+   Repository snapshots are written under `roles/<role>/upstream/<purpose>/`.
+   On `--apply`, each `upstream/<purpose>/` directory is replaced with a fresh
+   snapshot for that source, so stale README and Compose files are removed.
+   The script displays a single-line progress bar on interactive terminals while
+   processing sources.
 4. Update `roles/<role>/upstream/SOURCE.md` with source URL, ref, snapshot date,
    fetched paths, and any manual verification notes.
 5. Commit upstream snapshot changes separately from NBOPS deployment template
    changes.
 6. Diff upstream snapshots against current role templates.
-7. Apply upstream structural changes to `templates/compose.yaml.j2`.
-8. Keep NBOPS-local changes in `templates/compose.override.yaml.j2`, including:
-   Traefik labels, internal networks, host paths, ports, restart policy, logging,
-   resource limits, and deployment-specific environment wiring.
+7. Apply upstream structural changes to `templates/compose.yaml.j2`. If a
+   role-level `upstream/docker-compose.override.yml` exists, treat it as the
+   primary source for the template; otherwise use `upstream/docker-compose.yml`.
+8. Keep NBOPS-local changes in the deployed template, including Traefik labels,
+   internal networks, host paths, ports, restart policy, logging, resource
+   limits, and deployment-specific environment wiring.
 9. Replace mutable image tags with pinned versions where practical.
 10. Document breaking changes or operator actions in the role README.
 
@@ -70,6 +78,27 @@ Entries that use `readme_paths` or `compose_paths` must include both `url` and
 - Do not edit generated Compose files on VMs.
 - Do not deploy files from `upstream/` directly.
 - Do not treat `status: verify` sources as trusted until manually checked.
+- Treat `upstream/<purpose>/` as generated snapshot output; do not keep manual
+  edits there.
+- To remove generated purpose snapshots and any template files explicitly marked
+  with `nbops: derived-from-upstream`, run:
+
+  ```sh
+  scripts/vendor-upstream-snapshots.py --clean-upstreams <role>
+  ```
+
+  This deletes directories like `upstream/application/` and
+  `upstream/container/`, but leaves files such as `upstream/SOURCE.md` and
+  root-level override YAMLs alone. The old `--clean-purposes` flag is retained
+  as a compatibility alias.
+- To target a specific upstream purpose, add `--upstream`:
+
+  ```sh
+  scripts/vendor-upstream-snapshots.py --clean-upstreams --upstream container <role>
+  ```
+
+  `--upstream` may be repeated, or `--upstreams container,application` may be
+  used for comma-separated values.
 - Prefer pinned tags or digests over `latest` and other mutable tags.
 - Keep HTTP application ports unpublished in Traefik mode; document internal
   service ports as comments in generated Compose.

docs/proxmox-template.md

@@ -10,11 +10,67 @@ Expected VM defaults:
 - CPU type `host`
 - QEMU Guest Agent enabled
 - Cloud-Init drive attached
+- Serial console attached as `serial0: socket`
+- Display set to `vga: serial0` for Proxmox xterm.js console access
 - 2 vCPU, 2 GB RAM, 32 GB disk unless a service needs more
 
+Use the helper script on a Proxmox node to create the base template:
+
+```sh
+scripts/create-proxmox-template.sh
+```
+
+The script prompts for VMID, name, storage, bridge, CPU, memory, disk size, and
+image source. Press Enter to accept the displayed defaults. It downloads the
+Debian 13 GenericCloud qcow2 image unless a local `--image-file` is provided,
+imports it into Proxmox storage, attaches a Cloud-Init drive, configures serial
+console support, and converts the VM to a template.
+
+For automation or previews:
+
+```sh
+scripts/create-proxmox-template.sh --dry-run --yes
+scripts/create-proxmox-template.sh --non-interactive --vmid 9000 --storage local-lvm --bridge vmbr0
+```
+
+If a PBS storage/datastore ID is supplied, the template's main disk is created
+with the Proxmox disk `backup=1` flag. Leaving it empty keeps the main disk at
+`backup=0`.
+
+```sh
+scripts/create-proxmox-template.sh --pbs-storage pbs-main
+```
+
+To initialise all VMs declared in `host_vars/*.yml`, run:
+
+```sh
+scripts/init-proxmox-vms.sh
+```
+
+The VM initialiser checks whether the template VMID exists. If it does not, it
+runs `scripts/create-proxmox-template.sh` first. Each host var file with a
+`proxmox_vmid` is cloned from the template and configured from its `proxmox_*`
+hardware settings.
+
 Cloud-Init is only responsible for Day 0 bootstrap: user creation, SSH
 hardening, Docker installation, baseline packages, firewall defaults, directory
-layout, PKI directory preparation, and `ansible-pull`.
+layout, PKI directory preparation, enabling `serial-getty@ttyS0.service`, and
+`ansible-pull`.
+
+Proxmox hardware settings must be applied outside Cloud-Init. For a VM or
+template, set:
+
+```sh
+qm set <vmid> --serial0 socket --vga serial0
+```
+
+The Cloud-Init template enables `serial-getty@ttyS0.service` by default and sets
+`TERM=xterm-256color` for that getty. Override these variables if needed:
+
+```yaml
+cloud_init_serial_console_enabled: true
+cloud_init_serial_console_tty: ttyS0
+cloud_init_serial_console_term: xterm-256color
+```
 
 Applications and durable service configuration belong in Ansible roles.
-

docs/upstream-sources.yml

@@ -65,7 +65,7 @@ sources:
     vendor_dir: roles/authentik/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/goauthentik/authentik.git
         ref: main
         readme_paths: [README.md]
@@ -77,7 +77,7 @@ sources:
     vendor_dir: roles/authentik_ldap/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/goauthentik/authentik.git
         ref: main
         readme_paths: [README.md]
@@ -89,7 +89,7 @@ sources:
     vendor_dir: roles/authentik_proxy/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/goauthentik/authentik.git
         ref: main
         readme_paths: [README.md]
@@ -101,7 +101,7 @@ sources:
     vendor_dir: roles/authentik_rac/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/goauthentik/authentik.git
         ref: main
         readme_paths: [README.md]
@@ -113,7 +113,7 @@ sources:
     vendor_dir: roles/cleanuparr/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/cleanuparr/cleanuparr.git
         ref: main
         readme_paths: [README.md]
@@ -167,7 +167,7 @@ sources:
     vendor_dir: roles/flaresolverr/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/FlareSolverr/FlareSolverr.git
         ref: master
         readme_paths: [README.md]
@@ -179,7 +179,7 @@ sources:
     vendor_dir: roles/gitea/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/go-gitea/gitea.git
         ref: main
         readme_paths: [README.md]
@@ -191,7 +191,7 @@ sources:
     vendor_dir: roles/gitea_act_runner/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://gitea.com/gitea/act_runner.git
         ref: main
         readme_paths: [README.md]
@@ -203,7 +203,7 @@ sources:
     vendor_dir: roles/grafana/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/grafana/grafana.git
         ref: main
         readme_paths: [README.md]
@@ -215,7 +215,7 @@ sources:
     vendor_dir: roles/grafana_renderer/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/grafana/grafana-image-renderer.git
         ref: master
         readme_paths: [README.md]
@@ -227,7 +227,7 @@ sources:
     vendor_dir: roles/homarr/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/homarr-labs/homarr.git
         ref: dev
         readme_paths: [README.md]
@@ -239,7 +239,7 @@ sources:
     vendor_dir: roles/immich/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/immich-app/immich.git
         ref: main
         readme_paths: [README.md]
@@ -281,7 +281,7 @@ sources:
     vendor_dir: roles/jellyseerr/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/Fallenbagel/jellyseerr.git
         ref: develop
         readme_paths: [README.md]
@@ -305,7 +305,7 @@ sources:
     vendor_dir: roles/kiwix_serve/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/kiwix/kiwix-tools.git
         ref: main
         readme_paths: [README.md]
@@ -317,7 +317,7 @@ sources:
     vendor_dir: roles/loki/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/grafana/loki.git
         ref: main
         readme_paths: [README.md]
@@ -341,7 +341,7 @@ sources:
     vendor_dir: roles/ntfy/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/binwiederhier/ntfy.git
         ref: main
         readme_paths: [README.md]
@@ -353,7 +353,7 @@ sources:
     vendor_dir: roles/ocis/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/owncloud/ocis.git
         ref: master
         readme_paths: [README.md]
@@ -365,7 +365,7 @@ sources:
     vendor_dir: roles/paperless_ngx/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/paperless-ngx/paperless-ngx.git
         ref: main
         readme_paths: [README.md]
@@ -377,7 +377,7 @@ sources:
     vendor_dir: roles/pelican_panel/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/pelican-dev/panel.git
         ref: main
         readme_paths: [README.md]
@@ -401,7 +401,7 @@ sources:
     vendor_dir: roles/pinchflat/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/kieraneglin/pinchflat.git
         ref: main
         readme_paths: [README.md]
@@ -437,7 +437,7 @@ sources:
     vendor_dir: roles/pterodactyl/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/pterodactyl/panel.git
         ref: 1.0-develop
         readme_paths: [README.md]
@@ -461,7 +461,7 @@ sources:
     vendor_dir: roles/requarks/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/requarks/wiki.git
         ref: master
         readme_paths: [README.md]
@@ -485,7 +485,7 @@ sources:
     vendor_dir: roles/traefik/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/traefik/traefik.git
         ref: master
         readme_paths: [README.md]
@@ -509,7 +509,7 @@ sources:
     vendor_dir: roles/uptime_kuma/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/louislam/uptime-kuma.git
         ref: master
         readme_paths: [README.md]
@@ -521,7 +521,7 @@ sources:
     vendor_dir: roles/valkey/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/valkey-io/valkey.git
         ref: unstable
         readme_paths: [README.md]
@@ -533,7 +533,7 @@ sources:
     vendor_dir: roles/vaultwarden/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/dani-garcia/vaultwarden.git
         ref: main
         readme_paths: [README.md]
@@ -545,7 +545,7 @@ sources:
     vendor_dir: roles/wizarr/upstream
     local_compose_file: ""
     repositories:
-      - purpose: application
+      - purpose: container
         url: https://github.com/Wizarrrr/wizarr.git
         ref: main
         readme_paths: [README.md]