ntbops-ansible/docs/compose-update-policy.md

Compose Update Policy

Compose-backed services are deployed from role-owned Jinja templates. Vendored upstream files are reference snapshots only and must not be deployed directly.

Layout

Each Docker-backed role has:

• templates/compose.yaml.j2: deployed Compose template
• templates/env.j2: generated environment file
• upstream/: copied upstream reference files

docs/upstream-sources.yml is the authoritative inventory of upstream Git repositories, README paths, Compose paths, fallback Compose URLs, local Compose references, refs, and verification status.

Role entries support one local fallback:

• local_compose_file: repo-local Compose file to copy as role reference material when upstream does not provide a useful Compose example.

Repository entries support three upstream Compose discovery sources:

• compose_paths: paths to Compose files inside the checked-out repository.
• compose_urls: direct http or https URLs that return Compose YAML.
• README fenced YAML blocks, parsed when a README is listed in readme_paths.

Repository entries that only use compose_urls may omit url and ref. Entries that use readme_paths or compose_paths must include both url and ref so the script can clone the source repository.

Update Workflow

1. Read docs/upstream-sources.yml for the role being updated.

2. Test the upstream inventory without writing snapshots:

   scripts/vendor-upstream-snapshots.py <role>
   

3. Fetch or copy the listed upstream README and Compose files into roles/<role>/upstream/:

   scripts/vendor-upstream-snapshots.py --apply <role>
   

   The script also looks for Compose-shaped fenced YAML blocks in README files. When exactly one candidate is found, --apply writes it as docker-compose.yml inside that upstream purpose directory. Compose YAML copied from compose_paths or fetched from compose_urls is also normalized to docker-compose.yml. URL sources are fallback material and do not overwrite a Compose file already staged from the repository or README parse. A role-level local_compose_file is copied under roles/<role>/upstream/local/.

   Repository snapshots are written under roles/<role>/upstream/<purpose>/. On --apply, each upstream/<purpose>/ directory is replaced with a fresh snapshot for that source, so stale README and Compose files are removed. The script displays a single-line progress bar on interactive terminals while processing sources.

4. Update roles/<role>/upstream/SOURCE.md with source URL, ref, snapshot date, fetched paths, and any manual verification notes.

5. Commit upstream snapshot changes separately from NBOPS deployment template changes.

6. Diff upstream snapshots against current role templates.

7. Apply upstream structural changes to templates/compose.yaml.j2. If a role-level upstream/docker-compose.override.yml exists, treat it as the primary source for the template; otherwise use upstream/docker-compose.yml.

8. Keep NBOPS-local changes in the deployed template, including Traefik labels, internal networks, host paths, ports, restart policy, logging, resource limits, and deployment-specific environment wiring.

9. Replace mutable image tags with pinned versions where practical.

10. Document breaking changes or operator actions in the role README.

Rules

• Do not edit generated Compose files on VMs.

• Do not deploy files from upstream/ directly.

• Do not treat status: verify sources as trusted until manually checked.

• Treat upstream/<purpose>/ as generated snapshot output; do not keep manual edits there.

• To remove generated purpose snapshots and any template files explicitly marked with nbops: derived-from-upstream, run:

  scripts/vendor-upstream-snapshots.py --clean-upstreams <role>
  

  This deletes directories like upstream/application/ and upstream/container/, but leaves files such as upstream/SOURCE.md and root-level override YAMLs alone. The old --clean-purposes flag is retained as a compatibility alias.

• To target a specific upstream purpose, add --upstream:

  scripts/vendor-upstream-snapshots.py --clean-upstreams --upstream container <role>
  

  --upstream may be repeated, or --upstreams container,application may be used for comma-separated values.

• Prefer pinned tags or digests over latest and other mutable tags.

• Keep HTTP application ports unpublished in Traefik mode; document internal service ports as comments in generated Compose.

Validation

Run after every Compose template update:

ansible-playbook --syntax-check playbooks/site.yml
git diff --check

Later, add a render check that runs docker compose config against generated compose.yaml plus compose.override.yaml for selected roles.