Investigate: Redeployment Requirements on Code Changes. QA Environment Decision Guide #325

Closed
opened 2026-06-05 09:46:57 +00:00 by totdking · 12 comments
totdking commented 2026-06-05 09:46:57 +00:00 (Migrated from gitlab.com)
No description provided.
totdking commented 2026-06-05 09:48:33 +00:00 (Migrated from gitlab.com)

Summary

Certain code changes in this repo require partial or full redeployment of the LocalTerra QA environment to produce valid test results. Without knowing which changes require what level of reset, QA can silently test against stale contract state, an orphaned indexer DB, or a mis-compiled indexer binary — producing results that don't reflect the actual code under test.

This document maps change types to the correct reset action and explains why.


Background

The QA environment is made up of three layers, each with its own state:

Layer State lives in Reset action
LocalTerra chain Docker volume (localterra) make reset-qa (wipes volume)
Postgres / indexer DB Docker volume (postgres) make reset-qa (wipes volume)
Deployed contracts On-chain at specific addresses make deploy-local (new instantiation)
Indexer binary Compiled Rust (cargo run --release) make start-qa or make reset-qa
Frontend env frontend-dapp/.env.local make deploy-local (rewrites file)

make deploy-local only redeploys contracts and rewrites the frontend env. It does not wipe the chain, wipe postgres, or restart the indexer.

make reset-qa wipes both Docker volumes, then runs the full start-qa flow which includes make deploy-local, indexer recompile, and indexer restart.

make start only starts Docker containers. It does not build or deploy anything.


Change Types and Required Reset Level

Requires make reset-qa (full wipe + redeploy)

Change type Reason
Indexer Rust source changed (indexer/src/**) Running binary is stale; must recompile. deploy-local alone does not restart indexer. Also, new contract addresses from the redeploy make prior postgres data orphaned — wipe is cleaner.
Smart contract source changed (smartcontracts/contracts/**) New WASM must be built and uploaded. New addresses invalidate all prior indexer DB data for old addresses.
Deploy script changes that affect factory instantiation params (scripts/deploy-dex-local.sh) Existing on-chain factory retains its old instantiation config (e.g. pair_creation_fee_uluna=0). The only way to change it is to redeploy. Prior postgres data is then stale against new addresses.
Chain state is corrupted or in a known-bad state Volume wipe is the only reliable fix.
Multiple layers changed in one branch/commit When contracts + indexer + deploy script all change together, staggered partial resets risk inconsistency. Full wipe guarantees a clean baseline.

Requires make deploy-local only (no chain wipe)

Change type Reason
Deploy script logic changes that do not affect instantiation params e.g. gas flags, wait loops, output formatting — new addresses needed, but chain and postgres data remain valid.
Frontend source changes that require updated env addresses deploy-local rewrites frontend-dapp/.env.local.
Partial or failed prior deploy left contracts in an incomplete state Redeploy creates fresh addresses without wiping history.

No redeployment needed

Change type Reason
Frontend source only (frontend-dapp/src/**) No on-chain state changes. Restart make dev or the frontend dev server.
Docs only (docs/**, *.md, AGENTS.md, skills/**) No runtime impact.
QA scripts only (scripts/qa/**, not deploy script) Test scripts, not deployment.
Makefile target changes unrelated to deploy/build No state impact.
Contract unit tests only (smartcontracts/**/tests/**) Tests run against local cargo, not the live chain.

Real Example: Commit 019ded6 (merge qa/318-315-deploy-fee-gas)

This commit illustrates a case where make reset-qa was the correct action, not just make deploy-local.

What changed:

  • scripts/deploy-dex-local.sh — factory now instantiated with real pair_creation_fee_uluna (100 LUNC default) instead of 0; gas flag changed from --fees to --gas-prices
  • indexer/src/api/mod.rs — IPv6 /64 rate-limit key extractor and its tests removed
  • packages/localnet-trading-swarm/src/funding.ts — gas flag updated
  • Docs, skills, and Makefile updated

Why deploy-local alone was insufficient:

  1. The indexer binary needed recompilation (Rust source changed)
  2. The new deploy creates new contract addresses, making any prior postgres data for old addresses orphaned
  3. The existing on-chain factory retained pair_creation_fee_uluna=0 — the wrong config for validating the #318 fix

Correct action: make reset-qa


Quick Decision Reference

Did indexer Rust source change?          → make reset-qa
Did contract Rust source change?         → make reset-qa
Did deploy script change instantiation
  params (fees, governance, code IDs)?   → make reset-qa
Did deploy script change non-param
  logic only (gas flags, waits)?         → make deploy-local
Did only frontend / docs / scripts
  change?                                → no redeploy needed

  • scripts/deploy-dex-local.sh — local deploy entrypoint
  • scripts/qa/start-qa.sh — full QA stack start (includes deploy-local + indexer)
  • scripts/qa/reset-qa.sh — volume wipe + start-qa
  • GitLab issues: #318 (closed), #315, #276 (closed)

cc: @PlasticDigits

As requested per setting up phase time taking long before active investigation goes on .

## Summary Certain code changes in this repo require partial or full redeployment of the LocalTerra QA environment to produce valid test results. Without knowing which changes require what level of reset, QA can silently test against stale contract state, an orphaned indexer DB, or a mis-compiled indexer binary — producing results that don't reflect the actual code under test. This document maps change types to the correct reset action and explains why. --- ## Background The QA environment is made up of three layers, each with its own state: | Layer | State lives in | Reset action | |-------|----------------|--------------| | LocalTerra chain | Docker volume (`localterra`) | `make reset-qa` (wipes volume) | | Postgres / indexer DB | Docker volume (`postgres`) | `make reset-qa` (wipes volume) | | Deployed contracts | On-chain at specific addresses | `make deploy-local` (new instantiation) | | Indexer binary | Compiled Rust (`cargo run --release`) | `make start-qa` or `make reset-qa` | | Frontend env | `frontend-dapp/.env.local` | `make deploy-local` (rewrites file) | `make deploy-local` only redeploys contracts and rewrites the frontend env. It does not wipe the chain, wipe postgres, or restart the indexer. `make reset-qa` wipes both Docker volumes, then runs the full `start-qa` flow which includes `make deploy-local`, indexer recompile, and indexer restart. `make start` only starts Docker containers. It does not build or deploy anything. --- ## Change Types and Required Reset Level ### Requires `make reset-qa` (full wipe + redeploy) | Change type | Reason | |-------------|--------| | Indexer Rust source changed (`indexer/src/**`) | Running binary is stale; must recompile. `deploy-local` alone does not restart indexer. Also, new contract addresses from the redeploy make prior postgres data orphaned — wipe is cleaner. | | Smart contract source changed (`smartcontracts/contracts/**`) | New WASM must be built and uploaded. New addresses invalidate all prior indexer DB data for old addresses. | | Deploy script changes that affect factory instantiation params (`scripts/deploy-dex-local.sh`) | Existing on-chain factory retains its old instantiation config (e.g. `pair_creation_fee_uluna=0`). The only way to change it is to redeploy. Prior postgres data is then stale against new addresses. | | Chain state is corrupted or in a known-bad state | Volume wipe is the only reliable fix. | | Multiple layers changed in one branch/commit | When contracts + indexer + deploy script all change together, staggered partial resets risk inconsistency. Full wipe guarantees a clean baseline. | ### Requires `make deploy-local` only (no chain wipe) | Change type | Reason | |-------------|--------| | Deploy script logic changes that do not affect instantiation params | e.g. gas flags, wait loops, output formatting — new addresses needed, but chain and postgres data remain valid. | | Frontend source changes that require updated env addresses | `deploy-local` rewrites `frontend-dapp/.env.local`. | | Partial or failed prior deploy left contracts in an incomplete state | Redeploy creates fresh addresses without wiping history. | ### No redeployment needed | Change type | Reason | |-------------|--------| | Frontend source only (`frontend-dapp/src/**`) | No on-chain state changes. Restart `make dev` or the frontend dev server. | | Docs only (`docs/**`, `*.md`, `AGENTS.md`, `skills/**`) | No runtime impact. | | QA scripts only (`scripts/qa/**`, not deploy script) | Test scripts, not deployment. | | Makefile target changes unrelated to deploy/build | No state impact. | | Contract unit tests only (`smartcontracts/**/tests/**`) | Tests run against local cargo, not the live chain. | --- ## Real Example: Commit `019ded6` (merge `qa/318-315-deploy-fee-gas`) This commit illustrates a case where `make reset-qa` was the correct action, not just `make deploy-local`. **What changed:** - `scripts/deploy-dex-local.sh` — factory now instantiated with real `pair_creation_fee_uluna` (100 LUNC default) instead of `0`; gas flag changed from `--fees` to `--gas-prices` - `indexer/src/api/mod.rs` — IPv6 `/64` rate-limit key extractor and its tests removed - `packages/localnet-trading-swarm/src/funding.ts` — gas flag updated - Docs, skills, and Makefile updated **Why `deploy-local` alone was insufficient:** 1. The indexer binary needed recompilation (Rust source changed) 2. The new deploy creates new contract addresses, making any prior postgres data for old addresses orphaned 3. The existing on-chain factory retained `pair_creation_fee_uluna=0` — the wrong config for validating the #318 fix **Correct action:** `make reset-qa` --- ## Quick Decision Reference ``` Did indexer Rust source change? → make reset-qa Did contract Rust source change? → make reset-qa Did deploy script change instantiation params (fees, governance, code IDs)? → make reset-qa Did deploy script change non-param logic only (gas flags, waits)? → make deploy-local Did only frontend / docs / scripts change? → no redeploy needed ``` --- ## Related - `scripts/deploy-dex-local.sh` — local deploy entrypoint - `scripts/qa/start-qa.sh` — full QA stack start (includes deploy-local + indexer) - `scripts/qa/reset-qa.sh` — volume wipe + start-qa - GitLab issues: #318[ (closed)](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/318), [#315](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/315), [#276 (closed)](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/276) cc: @PlasticDigits As requested per setting up phase time taking long before active investigation goes on .
PlasticDigits commented 2026-06-05 10:06:08 +00:00 (Migrated from gitlab.com)

Might want to consider having brebuilt binaries available. This is a research and design question for how to optimize for qa team

Might want to consider having brebuilt binaries available. This is a research and design question for how to optimize for qa team
PlasticDigits commented 2026-06-05 11:10:27 +00:00 (Migrated from gitlab.com)

Investigation: QA setup slowness

I traced the QA/LocalTerra bring-up paths. The slow path is mostly deterministic setup work, not a single hang.

Where time is going

  1. make start-qa always pays optimizer + deploy:

    • scripts/qa/start-qa.sh stops the stack, starts compose, then runs make deploy-local.
    • Makefile has deploy-local: build-optimized, so every start-qa invokes the CosmWasm workspace optimizer.
    • The repo docs call out the first optimized wasm build as roughly 10-15 min cold; Docker cache volumes help but do not make it free.
  2. scripts/deploy-dex-local.sh is intentionally heavyweight:

    • Current script has 46 terrad_tx submissions and 31 fixed sleep 3 waits, before block inclusion and query time.
    • It uploads/instantiates core contracts, creates whitelisted/non-whitelisted/unpaired tokens, configures fee tiers, creates pairs and liquidity, runs swaps for history, then writes frontend-dapp/.env.local, indexer/.env, and .qa-deploy-stamp.
    • If auxiliary wasm is missing (cw20_mintable.wasm, treasury.wasm, wrap_mapper.wasm), deploy can detour into clone + optimizer builds inside the deploy script.
  3. Indexer cold start can add minutes:

    • start-qa uses cargo run --release unless INDEXER_QA_BIN points at a prebuilt binary.
    • The Cloud Agent setup path starts the same release run in tmux and waits for HTTP readiness.
  4. make reset-qa adds volume re-init:

    • It is correct for stale chain/schema cases, but expensive if used for frontend/docs/test-script-only work.
  5. Cloud Agent setup is more idempotent than QA server setup:

    • scripts/setup-cloud-agent-localterra.sh can skip build/deploy when .qa-deploy-stamp, .env.local, artifacts, and a live factory LCD probe all match HEAD.
    • scripts/qa/start-qa.sh always runs make deploy-local, so it always pays optimizer + deploy even when state is already valid.

Immediate operating recommendations

  • Default to make start-qa, not make reset-qa, unless contract/genesis/indexer schema changed or qa-verify-deploy reports stale on-chain schema.
  • If schema probes pass but the deploy stamp does not match HEAD, use make deploy-local && make qa-verify-deploy; do not wipe volumes.
  • On Cloud Agents, prefer make setup-cloud-localterra; use ./scripts/setup-cloud-agent-localterra.sh --skip-build when wasm artifacts are already fresh, or --infra-only when only LocalTerra/Postgres are needed.
  • Preserve Docker volumes and optimizer cache volumes; avoid docker volume prune on QA hosts unless an explicit fresh reset is needed.
  • Build the indexer once and reuse it for QA starts:
    • cd indexer && cargo build --release
    • from repo root: export INDEXER_QA_BIN=$PWD/indexer/target/release/cl8y-dex-indexer
    • then run make start-qa.
  • Ensure cw20_mintable.wasm, treasury.wasm, and wrap_mapper.wasm are present in smartcontracts/artifacts/ before deploy to avoid nested optimizer detours.
  • For frontend-only work, keep the existing LocalTerra/indexer when Q1 holds and run the frontend against the existing .env.local instead of re-running the full QA bring-up.

Design recommendations

  1. Add the Cloud Agent _deploy_up_to_date style skip to start-qa: if stamp git_sha == HEAD, env factory matches, artifacts exist, and the factory LCD probe succeeds, skip make deploy-local.
  2. Split deploy-local into build and no-build variants, for example deploy-local-no-build, so QA can skip the optimizer when artifacts are already fresh.
  3. Publish or cache optimized wasm artifacts per commit, e.g. GitLab package registry or CI artifact, and download them during QA setup with local optimizer as a cache-miss fallback.
  4. Publish or cache the indexer release binary per commit/toolchain hash and wire start-qa to prefer it through INDEXER_QA_BIN.
  5. Consider deploy seed profiles: minimal base pair, chart-history seed, wallet-edge-case seed. Many QA checks do not need all tokens, all pairs, and all swap history.
  6. Replace fixed sleep 3 waits with a tx wait/poll helper where possible. This will not remove block time, but it can reduce additive delay when a tx is already queryable.
  7. Add phase timing logs around start-qa and deploy-dex-local.sh so future slowness reports identify the exact phase without manual tracing.

Bottom line: the largest win is avoiding unnecessary optimizer + full deploy + release compile for QA sessions where deployed contract state is already valid.

## Investigation: QA setup slowness I traced the QA/LocalTerra bring-up paths. The slow path is mostly deterministic setup work, not a single hang. ### Where time is going 1. `make start-qa` always pays optimizer + deploy: - `scripts/qa/start-qa.sh` stops the stack, starts compose, then runs `make deploy-local`. - `Makefile` has `deploy-local: build-optimized`, so every `start-qa` invokes the CosmWasm workspace optimizer. - The repo docs call out the first optimized wasm build as roughly 10-15 min cold; Docker cache volumes help but do not make it free. 2. `scripts/deploy-dex-local.sh` is intentionally heavyweight: - Current script has 46 `terrad_tx` submissions and 31 fixed `sleep 3` waits, before block inclusion and query time. - It uploads/instantiates core contracts, creates whitelisted/non-whitelisted/unpaired tokens, configures fee tiers, creates pairs and liquidity, runs swaps for history, then writes `frontend-dapp/.env.local`, `indexer/.env`, and `.qa-deploy-stamp`. - If auxiliary wasm is missing (`cw20_mintable.wasm`, `treasury.wasm`, `wrap_mapper.wasm`), deploy can detour into clone + optimizer builds inside the deploy script. 3. Indexer cold start can add minutes: - `start-qa` uses `cargo run --release` unless `INDEXER_QA_BIN` points at a prebuilt binary. - The Cloud Agent setup path starts the same release run in tmux and waits for HTTP readiness. 4. `make reset-qa` adds volume re-init: - It is correct for stale chain/schema cases, but expensive if used for frontend/docs/test-script-only work. 5. Cloud Agent setup is more idempotent than QA server setup: - `scripts/setup-cloud-agent-localterra.sh` can skip build/deploy when `.qa-deploy-stamp`, `.env.local`, artifacts, and a live factory LCD probe all match `HEAD`. - `scripts/qa/start-qa.sh` always runs `make deploy-local`, so it always pays optimizer + deploy even when state is already valid. ### Immediate operating recommendations - Default to `make start-qa`, not `make reset-qa`, unless contract/genesis/indexer schema changed or `qa-verify-deploy` reports stale on-chain schema. - If schema probes pass but the deploy stamp does not match `HEAD`, use `make deploy-local && make qa-verify-deploy`; do not wipe volumes. - On Cloud Agents, prefer `make setup-cloud-localterra`; use `./scripts/setup-cloud-agent-localterra.sh --skip-build` when wasm artifacts are already fresh, or `--infra-only` when only LocalTerra/Postgres are needed. - Preserve Docker volumes and optimizer cache volumes; avoid `docker volume prune` on QA hosts unless an explicit fresh reset is needed. - Build the indexer once and reuse it for QA starts: - `cd indexer && cargo build --release` - from repo root: `export INDEXER_QA_BIN=$PWD/indexer/target/release/cl8y-dex-indexer` - then run `make start-qa`. - Ensure `cw20_mintable.wasm`, `treasury.wasm`, and `wrap_mapper.wasm` are present in `smartcontracts/artifacts/` before deploy to avoid nested optimizer detours. - For frontend-only work, keep the existing LocalTerra/indexer when Q1 holds and run the frontend against the existing `.env.local` instead of re-running the full QA bring-up. ### Design recommendations 1. Add the Cloud Agent `_deploy_up_to_date` style skip to `start-qa`: if stamp `git_sha == HEAD`, env factory matches, artifacts exist, and the factory LCD probe succeeds, skip `make deploy-local`. 2. Split `deploy-local` into build and no-build variants, for example `deploy-local-no-build`, so QA can skip the optimizer when artifacts are already fresh. 3. Publish or cache optimized wasm artifacts per commit, e.g. GitLab package registry or CI artifact, and download them during QA setup with local optimizer as a cache-miss fallback. 4. Publish or cache the indexer release binary per commit/toolchain hash and wire `start-qa` to prefer it through `INDEXER_QA_BIN`. 5. Consider deploy seed profiles: minimal base pair, chart-history seed, wallet-edge-case seed. Many QA checks do not need all tokens, all pairs, and all swap history. 6. Replace fixed `sleep 3` waits with a tx wait/poll helper where possible. This will not remove block time, but it can reduce additive delay when a tx is already queryable. 7. Add phase timing logs around `start-qa` and `deploy-dex-local.sh` so future slowness reports identify the exact phase without manual tracing. Bottom line: the largest win is avoiding unnecessary optimizer + full deploy + release compile for QA sessions where deployed contract state is already valid.
PlasticDigits commented 2026-06-05 12:27:09 +00:00 (Migrated from gitlab.com)

All 7 approved including publishing wasm, indexer artifacts

All 7 approved including publishing wasm, indexer artifacts
ghost1 commented 2026-06-05 12:31:36 +00:00 (Migrated from gitlab.com)

mentioned in commit 6763580ad2

mentioned in commit 6763580ad2c83c6ee0d9f3d34346708d775f2781
PlasticDigits commented 2026-06-05 12:32:02 +00:00 (Migrated from gitlab.com)

mentioned in merge request !802

mentioned in merge request !802
PlasticDigits commented 2026-06-05 12:32:07 +00:00 (Migrated from gitlab.com)

Implementation complete — MR !802

MR: https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/84

Delivered for #325:

  1. Decision guide — skills/AGENTS_QA_REDEPLOY_DECISION.md (change type → reset-qa / deploy-local / no redeploy)
  2. start-qa deploy skip — shared deploy_up_to_date probe (stamp == HEAD + factory LCD)
  3. deploy-local-no-build — deploy without optimizer when artifacts exist
  4. CI wasm packages — .gitlab-ci.yml qa-wasm-artifacts + publish script
  5. CI indexer binary — qa-indexer-binary job + INDEXER_QA_BIN / build-indexer-release
  6. Deploy seed profiles — QA_DEPLOY_SEED=minimal|charts|wallet|full
  7. Tx poll — scripts/lib/terrad-wait-tx.sh replaces fixed sleep 3
  8. Phase timing — [timing] logs in start-qa and deploy-dex-local.sh

Verification (agent VM)

Check Command Result
Wiring / decision guide make test-qa-redeploy-decision PASS
Fresh volumes helpers make test-qa-fresh-volumes PASS
LCD verify helpers make test-qa-verify-deploy (verify-deploy only) PASS
Live LocalTerra curl probe test-localterra-host-curl SKIP (no compose on agent VM)
Full start-qa on QA server manual SKIP (needs QA host post-merge)
Package fetch E2E QA_FETCH_CI_ARTIFACTS=1 SKIP (needs CI publish after merge)

Issue left open for QA host sign-off after MR merge.

## Implementation complete — MR !802 **MR:** https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/84 Delivered for #325: 1. **Decision guide** — `skills/AGENTS_QA_REDEPLOY_DECISION.md` (change type → `reset-qa` / `deploy-local` / no redeploy) 2. **`start-qa` deploy skip** — shared `deploy_up_to_date` probe (stamp == HEAD + factory LCD) 3. **`deploy-local-no-build`** — deploy without optimizer when artifacts exist 4. **CI wasm packages** — `.gitlab-ci.yml` `qa-wasm-artifacts` + publish script 5. **CI indexer binary** — `qa-indexer-binary` job + `INDEXER_QA_BIN` / `build-indexer-release` 6. **Deploy seed profiles** — `QA_DEPLOY_SEED=minimal|charts|wallet|full` 7. **Tx poll** — `scripts/lib/terrad-wait-tx.sh` replaces fixed `sleep 3` 8. **Phase timing** — `[timing]` logs in `start-qa` and `deploy-dex-local.sh` ### Verification (agent VM) | Check | Command | Result | |-------|---------|--------| | Wiring / decision guide | `make test-qa-redeploy-decision` | **PASS** | | Fresh volumes helpers | `make test-qa-fresh-volumes` | **PASS** | | LCD verify helpers | `make test-qa-verify-deploy` (verify-deploy only) | **PASS** | | Live LocalTerra curl probe | `test-localterra-host-curl` | **SKIP** (no compose on agent VM) | | Full `start-qa` on QA server | manual | **SKIP** (needs QA host post-merge) | | Package fetch E2E | `QA_FETCH_CI_ARTIFACTS=1` | **SKIP** (needs CI publish after merge) | Issue left **open** for QA host sign-off after MR merge.
PlasticDigits commented 2026-06-05 12:39:24 +00:00 (Migrated from gitlab.com)

mentioned in commit 775ad0e620

mentioned in commit 775ad0e62003034e6ad6570f5144629a133aa756
PlasticDigits (Migrated from gitlab.com) closed this issue 2026-06-05 12:39:25 +00:00
ghost1 commented 2026-06-06 08:16:07 +00:00 (Migrated from gitlab.com)

mentioned in merge request !829

mentioned in merge request !829
PlasticDigits commented 2026-06-12 04:46:03 +00:00 (Migrated from gitlab.com)

mentioned in issue #361

mentioned in issue #361
PlasticDigits commented 2026-06-12 07:23:08 +00:00 (Migrated from gitlab.com)

mentioned in merge request !879

mentioned in merge request !879
PlasticDigits commented 2026-08-24 05:54:00 +00:00 (Migrated from gitlab.com)

mentioned in issue #620

mentioned in issue #620
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
code/cl8y-dex-terraclassic#325
No description provided.