code/yieldomega
Watch
4
0
Fork
You've already forked yieldomega
 0
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity

DevOps: Cloud Agent Docker API permission denied (bootstrap + verify) #288

New issue
Closed
opened 2026-06-03 13:43:01 +00:00 by PlasticDigits · 9 comments
PlasticDigits commented 2026-06-03 13:43:01 +00:00 (Migrated from gitlab.com)

Problem

Cursor Cloud Agents report Docker failures when running stack or QA scripts:

permission denied while trying to connect to the docker API at unix:///var/run/docker.sock

scripts/start-local-anvil-stack.sh and scripts/start-qa-local-full-stack.sh expect yieldomega-pg via docker run. Agents without working Docker must fall back to native Postgres (AGENTS.md § Postgres without Docker), but many agents still attempt Docker first and burn time on opaque socket errors.

Current state

Piece Behavior
scripts/bootstrap-cloud-vm-toolchain.sh Installs Docker, fuse-overlayfs / vfs fallback, starts dockerd, usermod -aG docker, chmod 666 on /var/run/docker.sock
scripts/verify-cloud-vm-toolchain.sh FAIL if docker info or docker run hello-world fails
.cursor/environment.json Runs bootstrap on agent boot

Observed on nested Cloud VMs: dockerd may run as root while the agent user lacks group membership until re-login; chmod 666 may not persist across snapshots; overlay mount errors leave daemon up but docker run broken.

Goals

  1. Reliable non-sudo Docker for the Cloud Agent user after bootstrap (or clear, early skip).
  2. Actionable diagnostics when socket permission fails (user, groups, socket mode, docker info stderr).
  3. Document when to use native Postgres only — do not block indexer-only / Foundry-only tasks on Docker.

Recommended direction

  • Extend fix_docker_socket_permissions / ensure_docker to verify as $USER (not only root), re-apply group + socket perms, optionally newgrp docker hint in log.
  • Add scripts/verify-docker-cloud-agent.sh (or extend verify-cloud-vm-toolchain) that prints remediation steps on failure.
  • Bootstrap: if Docker still fails after vfs retry, write marker file e.g. /tmp/yieldomega-docker-unavailable and log native Postgres one-liner from AGENTS.md.
  • Consider Cloud image / environment.json hook: run verify-cloud-vm-toolchain.sh post-bootstrap and surface FAIL in agent instructions.
  • Optional: detect permission-denied vs overlay error in start-local-anvil-stack.sh and auto-suggest PG_HOST_PORT native path.

Acceptance criteria

  • Fresh Cloud VM: after bash scripts/bootstrap-cloud-vm-toolchain.sh, docker info and docker run --rm hello-world succeed without sudo for the dev user, or bootstrap exits with explicit "use native Postgres" and verify script documents SKIP vs FAIL.
  • bash scripts/verify-cloud-vm-toolchain.sh distinguishes Docker required vs optional (today: hard FAIL on docker).
  • AGENTS.md troubleshooting subsection: socket permission denied + remediation steps.
  • Agent checklist row: Docker → command → PASS/FAIL/SKIP with reason.

Out of scope

  • Production Docker / K8s deploy paths.
  • Fixing every nested-VM overlay limitation (vfs fallback already exists; document limits).

Labels

devops, testing, cloud-agent

## Problem Cursor Cloud Agents report Docker failures when running stack or QA scripts: ```text permission denied while trying to connect to the docker API at unix:///var/run/docker.sock ``` [`scripts/start-local-anvil-stack.sh`](scripts/start-local-anvil-stack.sh) and [`scripts/start-qa-local-full-stack.sh`](scripts/start-qa-local-full-stack.sh) expect **`yieldomega-pg`** via `docker run`. Agents without working Docker must fall back to native Postgres ([`AGENTS.md`](AGENTS.md) § Postgres without Docker), but many agents still attempt Docker first and burn time on opaque socket errors. ## Current state | Piece | Behavior | |-------|----------| | [`scripts/bootstrap-cloud-vm-toolchain.sh`](scripts/bootstrap-cloud-vm-toolchain.sh) | Installs Docker, `fuse-overlayfs` / `vfs` fallback, starts `dockerd`, `usermod -aG docker`, `chmod 666` on `/var/run/docker.sock` | | [`scripts/verify-cloud-vm-toolchain.sh`](scripts/verify-cloud-vm-toolchain.sh) | **FAIL** if `docker info` or `docker run hello-world` fails | | [`.cursor/environment.json`](.cursor/environment.json) | Runs bootstrap on agent boot | Observed on nested Cloud VMs: `dockerd` may run as root while the **agent user** lacks group membership until re-login; `chmod 666` may not persist across snapshots; overlay mount errors leave daemon up but `docker run` broken. ## Goals 1. **Reliable non-sudo Docker** for the Cloud Agent user after bootstrap (or clear, early skip). 2. **Actionable diagnostics** when socket permission fails (user, groups, socket mode, `docker info` stderr). 3. **Document** when to use native Postgres only — do not block indexer-only / Foundry-only tasks on Docker. ## Recommended direction - Extend `fix_docker_socket_permissions` / `ensure_docker` to verify as **`$USER`** (not only root), re-apply group + socket perms, optionally `newgrp docker` hint in log. - Add `scripts/verify-docker-cloud-agent.sh` (or extend verify-cloud-vm-toolchain) that prints remediation steps on failure. - Bootstrap: if Docker still fails after vfs retry, write marker file e.g. `/tmp/yieldomega-docker-unavailable` and log **native Postgres** one-liner from AGENTS.md. - Consider Cloud image / environment.json hook: run `verify-cloud-vm-toolchain.sh` post-bootstrap and surface FAIL in agent instructions. - Optional: detect permission-denied vs overlay error in `start-local-anvil-stack.sh` and auto-suggest `PG_HOST_PORT` native path. ## Acceptance criteria - [ ] Fresh Cloud VM: after `bash scripts/bootstrap-cloud-vm-toolchain.sh`, `docker info` and `docker run --rm hello-world` succeed **without sudo** for the dev user, **or** bootstrap exits with explicit "use native Postgres" and verify script documents SKIP vs FAIL. - [ ] [`bash scripts/verify-cloud-vm-toolchain.sh`](scripts/verify-cloud-vm-toolchain.sh) distinguishes **Docker required** vs **optional** (today: hard FAIL on docker). - [ ] [`AGENTS.md`](AGENTS.md) troubleshooting subsection: socket permission denied + remediation steps. - [ ] Agent checklist row: Docker → command → PASS/FAIL/SKIP with reason. ## Out of scope - Production Docker / K8s deploy paths. - Fixing every nested-VM overlay limitation (vfs fallback already exists; document limits). ## Labels `devops`, `testing`, `cloud-agent`
PlasticDigits commented 2026-06-03 13:43:13 +00:00 (Migrated from gitlab.com)

mentioned in issue #287

mentioned in issue #287
ghost1 commented 2026-06-03 14:06:28 +00:00 (Migrated from gitlab.com)

mentioned in commit e696a74855ef01bbbfc02dee00703bd06ef35e5f

mentioned in commit e696a74855ef01bbbfc02dee00703bd06ef35e5f
PlasticDigits commented 2026-06-03 14:07:02 +00:00 (Migrated from gitlab.com)

mentioned in merge request !421

mentioned in merge request !421
ghost1 commented 2026-06-03 14:15:39 +00:00 (Migrated from gitlab.com)

mentioned in commit 134128aa7e53840b5b636f7274cee75917f0bdc0

mentioned in commit 134128aa7e53840b5b636f7274cee75917f0bdc0
ghost1 commented 2026-06-03 14:24:39 +00:00 (Migrated from gitlab.com)

mentioned in commit 8f6a936faa

mentioned in commit 8f6a936faabb34155e090f3025690665e05a2349
PlasticDigits commented 2026-06-03 14:29:01 +00:00 (Migrated from gitlab.com)

mentioned in commit d9b1ed959c

mentioned in commit d9b1ed959c15b0567bcfac551e6edfcd4883068f
PlasticDigits commented 2026-06-03 14:45:33 +00:00 (Migrated from gitlab.com)

Verification (#288) — all acceptance criteria PASS

Implementation is already on main (merge d9b1ed9 from cursor/cloud-agent-docker-perms-288-6d29). No additional MR from this agent run.

Acceptance criteria

Criterion Command / check Result
Non-sudo Docker after bootstrap or explicit native Postgres + SKIP/FAIL semantics bash scripts/bootstrap-cloud-vm-toolchain.sh (socket fix via yieldomega_fix_docker_socket_permissions); bash scripts/verify-docker-cloud-agent.sh PASS after socket chmod 666 (bootstrap path); SKIP (exit 2, docker_permission_denied) before fix — marker written on bootstrap failure via yieldomega_mark_docker_unavailable
verify-cloud-vm-toolchain.sh optional vs required Docker bash scripts/verify-cloud-vm-toolchain.sh (docker lines) PASS — SKIP when optional broken; YIELDOMEGA_DOCKER_REQUIRED=1 → FAIL with diagnosis
AGENTS.md troubleshooting Manual PASS — § Docker troubleshooting
Agent checklist row (Docker → command → PASS/FAIL/SKIP) Manual PASS — Cloud agent verification table

Commands run (this VM, ubuntu@ Cloud Agent)

$ bash scripts/test-docker-cloud-agent-lib.sh
test-docker-cloud-agent-lib: PASS

$ bash scripts/verify-docker-cloud-agent.sh
# Before socket fix: SKIP  Docker optional — docker_permission_denied; exit 2
# After:  PASS  docker info + hello-world (fuse-overlayfs); exit 0

$ YIELDOMEGA_DOCKER_REQUIRED=1 bash scripts/verify-docker-cloud-agent.sh
FAIL  Docker required but not usable (docker_permission_denied)
# stderr: USER=ubuntu, groups=ubuntu, docker.sock: 660 root:docker, remediation steps

$ bash scripts/verify-cloud-vm-toolchain.sh 2>&1 | grep -i docker
PASS  docker (fuse-overlayfs) — required only for start-local-anvil-stack / full QA stack

Deliverables (on main)

Third-party verification checklist

  1. bash scripts/test-docker-cloud-agent-lib.sh → PASS
  2. bash scripts/verify-docker-cloud-agent.sh → PASS or SKIP (not FAIL unless YIELDOMEGA_DOCKER_REQUIRED=1)
  3. bash scripts/verify-cloud-vm-toolchain.sh → Docker line PASS or SKIP (toolchain may FAIL on unrelated layers, e.g. Postgres #287)
  4. Broken socket: confirm stderr diagnosis (user, groups, socket mode, remediation)
  5. Optional stack: start-local-anvil-stack.sh prints native Postgres hint when Docker fails

Closing as complete on main.

## Verification (#288) — all acceptance criteria **PASS** Implementation is already on **`main`** (merge `d9b1ed9` from `cursor/cloud-agent-docker-perms-288-6d29`). No additional MR from this agent run. ### Acceptance criteria | Criterion | Command / check | Result | |-----------|-----------------|--------| | Non-sudo Docker after bootstrap **or** explicit native Postgres + SKIP/FAIL semantics | `bash scripts/bootstrap-cloud-vm-toolchain.sh` (socket fix via `yieldomega_fix_docker_socket_permissions`); `bash scripts/verify-docker-cloud-agent.sh` | **PASS** after socket `chmod 666` (bootstrap path); **SKIP** (exit 2, `docker_permission_denied`) before fix — marker written on bootstrap failure via `yieldomega_mark_docker_unavailable` | | `verify-cloud-vm-toolchain.sh` optional vs required Docker | `bash scripts/verify-cloud-vm-toolchain.sh` (docker lines) | **PASS** — `SKIP` when optional broken; `YIELDOMEGA_DOCKER_REQUIRED=1` → **FAIL** with diagnosis | | `AGENTS.md` troubleshooting | Manual | **PASS** — [§ Docker troubleshooting](AGENTS.md#docker-troubleshooting-gitlab-288) | | Agent checklist row (Docker → command → PASS/FAIL/SKIP) | Manual | **PASS** — [Cloud agent verification table](AGENTS.md#cloud-agent-verification-when-to-run-what) | ### Commands run (this VM, `ubuntu@` Cloud Agent) ```text $ bash scripts/test-docker-cloud-agent-lib.sh test-docker-cloud-agent-lib: PASS $ bash scripts/verify-docker-cloud-agent.sh # Before socket fix: SKIP Docker optional — docker_permission_denied; exit 2 # After: PASS docker info + hello-world (fuse-overlayfs); exit 0 $ YIELDOMEGA_DOCKER_REQUIRED=1 bash scripts/verify-docker-cloud-agent.sh FAIL Docker required but not usable (docker_permission_denied) # stderr: USER=ubuntu, groups=ubuntu, docker.sock: 660 root:docker, remediation steps $ bash scripts/verify-cloud-vm-toolchain.sh 2>&1 | grep -i docker PASS docker (fuse-overlayfs) — required only for start-local-anvil-stack / full QA stack ``` ### Deliverables (on `main`) - [`scripts/lib/docker_cloud_agent.sh`](scripts/lib/docker_cloud_agent.sh) — diagnosis, error kind, socket fix, marker, stack hints - [`scripts/verify-docker-cloud-agent.sh`](scripts/verify-docker-cloud-agent.sh) — exit 0/1/2 (PASS/FAIL/SKIP) - [`scripts/verify-cloud-vm-toolchain.sh`](scripts/verify-cloud-vm-toolchain.sh) — delegates Docker; no hard FAIL when optional - [`scripts/bootstrap-cloud-vm-toolchain.sh`](scripts/bootstrap-cloud-vm-toolchain.sh) — verify as `$USER`, vfs fallback, marker + native Postgres one-liner - [`scripts/start-local-anvil-stack.sh`](scripts/start-local-anvil-stack.sh) — `yieldomega_docker_stack_failure_hint` on socket failure - Invariants **`INV-DEVOPS-288-*`**: [docs/testing/invariants-and-business-logic.md §288](docs/testing/invariants-and-business-logic.md#cloud-agent-docker-gitlab-288) - Cross-links: [AGENTS.md](AGENTS.md), [`.cursor/skills/yieldomega-guardrails/SKILL.md`](.cursor/skills/yieldomega-guardrails/SKILL.md), [qa-local-full-stack.md](docs/testing/qa-local-full-stack.md) - [`.cursor/environment.json`](.cursor/environment.json) — post-bootstrap `verify-cloud-vm-toolchain.sh` ### Third-party verification checklist 1. `bash scripts/test-docker-cloud-agent-lib.sh` → PASS 2. `bash scripts/verify-docker-cloud-agent.sh` → PASS or SKIP (not FAIL unless `YIELDOMEGA_DOCKER_REQUIRED=1`) 3. `bash scripts/verify-cloud-vm-toolchain.sh` → Docker line PASS or SKIP (toolchain may FAIL on unrelated layers, e.g. Postgres #287) 4. Broken socket: confirm stderr diagnosis (user, groups, socket mode, remediation) 5. Optional stack: `start-local-anvil-stack.sh` prints native Postgres hint when Docker fails Closing as complete on `main`.
PlasticDigits (Migrated from gitlab.com) closed this issue 2026-06-03 14:45:40 +00:00
PlasticDigits commented 2026-06-08 08:45:13 +00:00 (Migrated from gitlab.com)

mentioned in commit 8b1fdf8c79

mentioned in commit 8b1fdf8c7967bba100b73d0b3d8fe62d644ad812
PlasticDigits commented 2026-06-08 08:45:13 +00:00 (Migrated from gitlab.com)

mentioned in commit 7698e97f5a

mentioned in commit 7698e97f5aa16dda2598c34633df48542a29cdeb
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
cac-design-issue-544
renovate/configure
No results found.
Labels
Clear labels   
agent:fix_bugfix

   
agent:fix_conflicts

   
agent:fix_security

   
agent:gap_analysis

   
agent:implement

   
agent:open_issues

   
agents

   
agent:security_audit

   
agent:verify
  
area
timecurve

  
arena

   
arena-v2

   
audit

   
block:feature_review

   
block:security

   
bug

   
contracts

   
design

   
devops

   
docs
Documentation

  
documentation

   
enhancement

   
epic
Epic / umbrella issue

  
feature

   
frontend

   
indexer

   
low priority
Low priority, investigate other issues first

  
megaeth

   
operations

   
ops
Deploy and operations

  
performance

   
QA

   
qa checklist

   
referrals

   
removal
Legacy removal

  
security

   
smart-contracts

   
tech-debt

   
testing

   
time-arena
TimeArena v2 revamp

  
timecurve

   
visual-qa
No labels
agent:fix_bugfix
agent:fix_conflicts
agent:fix_security
agent:gap_analysis
agent:implement
agent:open_issues
agents
agent:security_audit
agent:verify
area
timecurve
arena
arena-v2
audit
block:feature_review
block:security
bug
contracts
design
devops
docs
documentation
enhancement
epic
feature
frontend
indexer
low priority
megaeth
operations
ops
performance
QA
qa checklist
referrals
removal
security
smart-contracts
tech-debt
testing
time-arena
timecurve
visual-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
code/yieldomega#288
No description provided.