implemented compose by url, parse, local yaml.

2026-07-10 13:32:40 +01:00
1 parent 53695f4fff
commit 784fff8faf
@@ -0,0 +1,87 @@
+# Compose Update Policy
+
+Compose-backed services are deployed from role-owned Jinja templates. Vendored
+upstream files are reference snapshots only and must not be deployed directly.
+
+## Layout
+
+Each Docker-backed role has:
+
+- `templates/compose.yaml.j2`: base deployed Compose file
+- `templates/compose.override.yaml.j2`: NBOPS-specific local override
+- `templates/env.j2`: generated environment file
+- `upstream/`: copied upstream reference files
+
+`docs/upstream-sources.yml` is the authoritative inventory of upstream Git
+repositories, README paths, Compose paths, fallback Compose URLs, local Compose
+references, refs, and verification status.
+
+Role entries support one local fallback:
+
+- `local_compose_file`: repo-local Compose file to copy as role reference
+  material when upstream does not provide a useful Compose example.
+
+Repository entries support three upstream Compose discovery sources:
+
+- `compose_paths`: paths to Compose files inside the checked-out repository.
+- `compose_urls`: direct `http` or `https` URLs that return Compose YAML.
+- README fenced YAML blocks, parsed when a README is listed in `readme_paths`.
+
+Repository entries that only use `compose_urls` may omit `url` and `ref`.
+Entries that use `readme_paths` or `compose_paths` must include both `url` and
+`ref` so the script can clone the source repository.
+
+## Update Workflow
+
+1. Read `docs/upstream-sources.yml` for the role being updated.
+2. Test the upstream inventory without writing snapshots:
+
+   ```sh
+   scripts/vendor-upstream-snapshots.py <role>
+   ```
+
+3. Fetch or copy the listed upstream README and Compose files into
+   `roles/<role>/upstream/`:
+
+   ```sh
+   scripts/vendor-upstream-snapshots.py --apply <role>
+   ```
+
+   The script also looks for Compose-shaped fenced YAML blocks in README files.
+   When exactly one candidate is found, `--apply` writes it as
+   `compose.from-readme.yaml`. Compose YAML fetched from `compose_urls` is
+   written as `compose.from-url.yaml`, or `compose.from-url-N.yaml` when a
+   repository has multiple fallback URLs. A role-level `local_compose_file` is
+   copied under `roles/<role>/upstream/local/`.
+4. Update `roles/<role>/upstream/SOURCE.md` with source URL, ref, snapshot date,
+   fetched paths, and any manual verification notes.
+5. Commit upstream snapshot changes separately from NBOPS deployment template
+   changes.
+6. Diff upstream snapshots against current role templates.
+7. Apply upstream structural changes to `templates/compose.yaml.j2`.
+8. Keep NBOPS-local changes in `templates/compose.override.yaml.j2`, including:
+   Traefik labels, internal networks, host paths, ports, restart policy, logging,
+   resource limits, and deployment-specific environment wiring.
+9. Replace mutable image tags with pinned versions where practical.
+10. Document breaking changes or operator actions in the role README.
+
+## Rules
+
+- Do not edit generated Compose files on VMs.
+- Do not deploy files from `upstream/` directly.
+- Do not treat `status: verify` sources as trusted until manually checked.
+- Prefer pinned tags or digests over `latest` and other mutable tags.
+- Keep HTTP application ports unpublished in Traefik mode; document internal
+  service ports as comments in generated Compose.
+
+## Validation
+
+Run after every Compose template update:
+
+```sh
+ansible-playbook --syntax-check playbooks/site.yml
+git diff --check
+```
+
+Later, add a render check that runs `docker compose config` against generated
+`compose.yaml` plus `compose.override.yaml` for selected roles.
@@ -54,10 +54,19 @@ Monitoring:
 - `prometheus`
 - `uptime_kuma`

-Most roles use the shared `docker_app` implementation and support:
+Compose-backed roles use the shared `compose_project` lifecycle directly or via
+the `docker_app` single-container adapter. They support:

 - `host` routing mode: publish ports directly and open UFW rules.
 - `traefik` routing mode: attach to the proxy network and render Traefik labels.
+- `traefik_file` routing mode: publish service ports for central edge Traefik,
+  without rendering Docker labels. Prefer `traefik` for normal multi-VM
+  deployments so edge Traefik forwards to internal Traefik, and internal Traefik
+  routes to applications by labels.

 Override role defaults in `host_vars` or `group_vars`; do not edit generated
 Compose files on the VM.
+
+Upstream README and Compose examples are tracked as copied snapshots under each
+role's `upstream/` directory. See `docs/upstream-sources.yml` and
+`docs/compose-update-policy.md`.
@@ -21,9 +21,9 @@ authentik_domain: auth.example.invalid

 ## Downstream VMs

-Downstream VMs also run Docker. Instead of publishing each application port on
-the host, run Traefik on the VM and attach application containers to the shared
-Docker proxy network.
+Downstream VMs also run Docker and normally run an internal Traefik instance.
+Application containers route through that local Traefik using Docker labels and
+the shared Docker proxy network.

 ```yaml
 nbops_roles:
@@ -47,13 +47,85 @@ Traefik labels and join `nbops_proxy_network`, which defaults to `proxy`.
 Only Traefik entrypoints should be exposed on the downstream host firewall. Use
 `traefik_bind_address` to bind those entrypoints to the VM's internal address.

+For central edge routing, the edge Traefik uses the file provider to route
+machine-to-machine traffic to each downstream VM's internal Traefik entrypoint.
+The internal Traefik then routes to applications using Docker labels. This keeps
+container discovery local to the VM and gives the edge-to-internal leg a stable
+place for future mTLS.
+
+Edge VM:
+
+```yaml
+nbops_roles:
+  - base
+  - ssh
+  - ufw
+  - updates
+  - docker
+  - traefik
+
+traefik_docker_provider_enabled: false
+traefik_file_provider_enabled: true
+traefik_dynamic_inventory_group: nbops
+```
+
+Downstream VM:
+
+```yaml
+nbops_service_routing_mode: traefik
+nbops_traefik_entrypoint: web
+nbops_traefik_tls: false
+
+traefik_bind_address: "10.0.10.25"
+traefik_publish_entrypoints: true
+
+nbops_traefik_file_routes:
+  - name: jellyfin
+    domain: jellyfin.example.invalid
+    scheme: http
+    port: 80
+  - name: gitea
+    domain: git.example.invalid
+    scheme: http
+    port: 80
+```
+
+When the play runs against the edge VM, Traefik renders
+`dynamic/http-services.yml` from `nbops_traefik_file_routes` declared across the
+inventory group. Each route targets `ansible_host` for that inventory host by
+default. Set `target_host` for a different VM address, or set `url` to provide
+the complete upstream URL.
+
+For future mTLS between edge Traefik and internal Traefik, routes can reference a
+Traefik `serversTransport`:
+
+```yaml
+nbops_traefik_file_routes:
+  - name: jellyfin
+    domain: jellyfin.example.invalid
+    scheme: https
+    port: 443
+    servers_transport: internal-mtls
+
+traefik_dynamic_servers_transports:
+  - name: internal-mtls
+    server_name: apps-01.example.internal
+    root_cas:
+      - /etc/pki/internal/certs/ca.crt
+    certificates:
+      - cert_file: /etc/pki/internal/certs/edge.crt
+        key_file: /etc/pki/internal/private/edge.key
+```
+
 ## Routing Modes

 Application roles support:

 - `host`: publish application ports directly and open those ports in UFW.
 - `traefik`: do not publish HTTP ports; attach to Traefik and render labels.
+- `traefik_file`: publish a service port without labels. This is available for
+  exceptional direct edge-to-app routes, but the preferred multi-VM topology is
+  edge Traefik -> internal Traefik -> application.

 Gitea is special: HTTP can be routed through Traefik while SSH remains published
 with `gitea_publish_ssh: true`.
-
@@ -0,0 +1,553 @@
+---
+version: 1
+policy:
+  vendor_method: copied_snapshots
+  vendor_root: roles
+  deployed_from: role_templates
+  notes:
+    - Upstream files are reference snapshots only and are not deployed directly.
+    - "Use compose_paths: [] when upstream documents Compose in README files only."
+    - "Use compose_urls: [] for direct raw HTTP Compose YAML fallback URLs."
+    - "Use role-level local_compose_file as a repo-local fallback Compose reference."
+    - "status: verify means the source mapping must be checked before pinning."
+
+sources:
+  alloy:
+    role: alloy
+    vendor_dir: roles/alloy/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/grafana/alloy.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  arr:
+    role: arr
+    vendor_dir: roles/arr/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: prowlarr-container
+        url: https://github.com/linuxserver/docker-prowlarr.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+      - purpose: sonarr-container
+        url: https://github.com/linuxserver/docker-sonarr.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+      - purpose: radarr-container
+        url: https://github.com/linuxserver/docker-radarr.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+      - purpose: lidarr-container
+        url: https://github.com/linuxserver/docker-lidarr.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+      - purpose: bazarr-container
+        url: https://github.com/linuxserver/docker-bazarr.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  authentik:
+    role: authentik
+    vendor_dir: roles/authentik/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/goauthentik/authentik.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: [https://docs.goauthentik.io/compose.yml]
+
+  authentik_ldap:
+    role: authentik_ldap
+    vendor_dir: roles/authentik_ldap/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/goauthentik/authentik.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: [https://docs.goauthentik.io/add-secure-apps/outposts/manual-deploy-docker-compose/]
+
+  authentik_proxy:
+    role: authentik_proxy
+    vendor_dir: roles/authentik_proxy/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/goauthentik/authentik.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: [https://docs.goauthentik.io/add-secure-apps/outposts/manual-deploy-docker-compose/]
+
+  authentik_rac:
+    role: authentik_rac
+    vendor_dir: roles/authentik_rac/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/goauthentik/authentik.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: [https://docs.goauthentik.io/add-secure-apps/outposts/manual-deploy-docker-compose/]
+
+  cleanuparr:
+    role: cleanuparr
+    vendor_dir: roles/cleanuparr/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/cleanuparr/cleanuparr.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  code_server:
+    role: code_server
+    vendor_dir: roles/code_server/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/linuxserver/docker-code-server.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  filebrowser_quantum:
+    role: filebrowser_quantum
+    status: verify
+    vendor_dir: roles/filebrowser_quantum/upstream
+    local_compose_file: ""
+    notes:
+      - Verify upstream repository for the gtstef/filebrowser image before pinning.
+    repositories:
+      - purpose: container
+        url: https://github.com/gtsteffaniak/filebrowser.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: [https://filebrowserquantum.com/en/docs/user-guides/other/standalone/]
+
+  fiveetools:
+    role: fiveetools
+    status: verify
+    vendor_dir: roles/fiveetools/upstream
+    local_compose_file: ""
+    notes:
+      - Verify upstream repository for the jafner/5etools image before pinning.
+    repositories:
+      - purpose: container
+        url: https://github.com/Jafner/5etools-docker.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: [docker-compose.yml]
+        compose_urls: []
+
+  flaresolverr:
+    role: flaresolverr
+    vendor_dir: roles/flaresolverr/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/FlareSolverr/FlareSolverr.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: [docker-compose.yml]
+        compose_urls: []
+
+  gitea:
+    role: gitea
+    vendor_dir: roles/gitea/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/go-gitea/gitea.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: [https://docs.gitea.com/installation/install-with-docker-rootless]
+
+  gitea_act_runner:
+    role: gitea_act_runner
+    vendor_dir: roles/gitea_act_runner/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://gitea.com/gitea/act_runner.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: [https://docs.gitea.com/usage/actions/act-runner#start-the-runner-using-docker-compose]
+
+  grafana:
+    role: grafana
+    vendor_dir: roles/grafana/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/grafana/grafana.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: [] # Host own, all examples are bad
+
+  grafana_renderer:
+    role: grafana_renderer
+    vendor_dir: roles/grafana_renderer/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/grafana/grafana-image-renderer.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  homarr:
+    role: homarr
+    vendor_dir: roles/homarr/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/homarr-labs/homarr.git
+        ref: dev
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  immich:
+    role: immich
+    vendor_dir: roles/immich/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/immich-app/immich.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: [docker/docker-compose.yml]
+        compose_urls: []
+
+  ipv6nat:
+    role: ipv6nat
+    vendor_dir: roles/ipv6nat/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/robbertkl/docker-ipv6nat.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  jellyfin:
+    role: jellyfin
+    vendor_dir: roles/jellyfin/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/linuxserver/docker-jellyfin.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+      - purpose: application
+        url: https://github.com/jellyfin/jellyfin.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  jellyseerr:
+    role: jellyseerr
+    vendor_dir: roles/jellyseerr/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/Fallenbagel/jellyseerr.git
+        ref: develop
+        readme_paths: [README.md]
+        compose_paths: [docker-compose.yml]
+        compose_urls: []
+
+  kavita:
+    role: kavita
+    vendor_dir: roles/kavita/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/linuxserver/docker-kavita.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  kiwix_serve:
+    role: kiwix_serve
+    vendor_dir: roles/kiwix_serve/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/kiwix/kiwix-tools.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  loki:
+    role: loki
+    vendor_dir: roles/loki/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/grafana/loki.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: [production/docker-compose.yaml]
+        compose_urls: []
+
+  nginx_content:
+    role: nginx_content
+    vendor_dir: roles/nginx_content/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/nginx/nginx.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  ntfy:
+    role: ntfy
+    vendor_dir: roles/ntfy/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/binwiederhier/ntfy.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: [docker-compose.yml]
+        compose_urls: []
+
+  ocis:
+    role: ocis
+    vendor_dir: roles/ocis/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/owncloud/ocis.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: [deployments/examples/ocis_full/docker-compose.yml]
+        compose_urls: []
+
+  paperless_ngx:
+    role: paperless_ngx
+    vendor_dir: roles/paperless_ngx/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/paperless-ngx/paperless-ngx.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: [docker/compose/docker-compose.postgres.yml]
+        compose_urls: []
+
+  pelican_panel:
+    role: pelican_panel
+    vendor_dir: roles/pelican_panel/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/pelican-dev/panel.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: [compose-full-stack.yml]
+        compose_urls: []
+
+  php_fpm:
+    role: php_fpm
+    vendor_dir: roles/php_fpm/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/docker-library/php.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  pinchflat:
+    role: pinchflat
+    vendor_dir: roles/pinchflat/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/kieraneglin/pinchflat.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  postgres:
+    role: postgres
+    vendor_dir: roles/postgres/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/docker-library/postgres.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  prometheus:
+    role: prometheus
+    vendor_dir: roles/prometheus/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/prometheus/prometheus.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: [documentation/examples/prometheus.yml]
+        compose_urls: []
+
+  pterodactyl:
+    role: pterodactyl
+    vendor_dir: roles/pterodactyl/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/pterodactyl/panel.git
+        ref: 1.0-develop
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  qbittorrent:
+    role: qbittorrent
+    vendor_dir: roles/qbittorrent/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/linuxserver/docker-qbittorrent.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  requarks:
+    role: requarks
+    vendor_dir: roles/requarks/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/requarks/wiki.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  syncthing:
+    role: syncthing
+    vendor_dir: roles/syncthing/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/linuxserver/docker-syncthing.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  traefik:
+    role: traefik
+    vendor_dir: roles/traefik/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/traefik/traefik.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  transmission:
+    role: transmission
+    vendor_dir: roles/transmission/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: container
+        url: https://github.com/linuxserver/docker-transmission.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  uptime_kuma:
+    role: uptime_kuma
+    vendor_dir: roles/uptime_kuma/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/louislam/uptime-kuma.git
+        ref: master
+        readme_paths: [README.md]
+        compose_paths: [compose.yaml]
+        compose_urls: []
+
+  valkey:
+    role: valkey
+    vendor_dir: roles/valkey/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/valkey-io/valkey.git
+        ref: unstable
+        readme_paths: [README.md]
+        compose_paths: []
+        compose_urls: []
+
+  vaultwarden:
+    role: vaultwarden
+    vendor_dir: roles/vaultwarden/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/dani-garcia/vaultwarden.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: [docker/docker-compose.yml]
+        compose_urls: []
+
+  wizarr:
+    role: wizarr
+    vendor_dir: roles/wizarr/upstream
+    local_compose_file: ""
+    repositories:
+      - purpose: application
+        url: https://github.com/Wizarrrr/wizarr.git
+        ref: main
+        readme_paths: [README.md]
+        compose_paths: [docker-compose.yml]
+        compose_urls: []