Skip to content

chore(mongo): upgrade embedded MongoDB from 6.0 to 7.0#41743

Open
wyattwalter wants to merge 12 commits intoreleasefrom
chore/mongo-7-upgrade
Open

chore(mongo): upgrade embedded MongoDB from 6.0 to 7.0#41743
wyattwalter wants to merge 12 commits intoreleasefrom
chore/mongo-7-upgrade

Conversation

@wyattwalter
Copy link
Copy Markdown
Contributor

@wyattwalter wyattwalter commented Apr 16, 2026

Summary

Upgrades the embedded MongoDB in the Appsmith Docker image from 6.0 (EOL July 2024) to 7.0.

  • base.dockerfile — switches apt repo, GPG key, and list file from the MongoDB 6.0 stream to 7.0. Stays on the jammy (22.04) packages because MongoDB doesn't publish a noble (24.04) apt repo yet — same pattern the 6.0 install already used on Ubuntu 24.04. Local smoke build installs mongodb-org 7.0.31 and mongosh 2.8.2.

  • mongodb-fixer.sh — writes a .appsmith-mongo-fcv-min marker file into the Mongo data directory once mongod is confirmed running under this release. Contents record the minimum FCV this release commits to preserve (constant 6.0) — a release-level contract, not a live FCV reading. FCV is deliberately not raised to 7.0 — keeping it at 6.0 preserves the ability to downgrade back to a 6.x Appsmith release. Raise-FCV scaffolding and a marker-value bump can come back when MongoDB 8 needs it.

  • entrypoint.sh — adds ensure_mongodb_fcv_compatible, a pre-flight check that runs before supervisord starts mongod. MongoDB 7.0 refuses to start on data with FCV < 6.0; without this check, supervisord would retry mongod three times and give up, leaving the container in a confusing degraded state with no clear signal to the administrator. Reader only — never writes the marker. Only runs when there's existing local-Mongo data (gated on shouldPerformInitdb=0 && isUriLocal=0 inside the function).

Decision matrix

Data dir Marker Action
Fresh install n/a Skipped — nothing to check. Fixer writes the marker (constant 6.0) on first real boot under supervisord.
Has data Present Fast path — proceed, zero overhead.
Has data Missing One-time mongod --fork probe. If it starts, proceed (fixer writes the marker on first boot under supervisord). If it fails, hard-fail with an actionable error pointing users to roll back to Appsmith v1.99 first.

Marker presence alone is what's trusted — the fixer writes it only after mongod is confirmed running under this release, so presence is proof this release successfully boots on this data. The value (6.0) is diagnostic: future Appsmith releases can inspect it to reason about upgrade safety.

Linear: APP-14867

Test plan

  • Fresh install: empty /appsmith-stacks → entrypoint skips the FCV check (shouldPerformInitdb=0 gate is false), init_replica_set initializes the data, supervisord starts mongod, fixer writes .appsmith-mongo-fcv-min with 6.0. Second boot: marker present → fast path, skip probe.
  • Happy-path upgrade: boot appsmith-ce:v1.99 (last 6.x release), let the fixer run (FCV stays at 6.0), stop. Swap to this image and boot — entrypoint runs the probe (marker from prior image is missing), probe succeeds, fixer writes the marker. Subsequent boots are fast.
  • Failed + remediated upgrade: boot appsmith-ce:v1.69 (last mongo-5 release, pre-v1.70 cutover) to seed FCV 5.0 data. Swap to this image → expected hard fail with error block pointing to v1.99 rollback. Swap to appsmith-ce:v1.99 → fixer raises FCV 5.0 → 6.0. Swap back to this image → probe succeeds, marker written. Verifies the error path and that the remediation instructions actually work end-to-end.
  • Marker-missing transitional path: with the marker present, rm /appsmith-stacks/data/mongodb/.appsmith-mongo-fcv-min, reboot. Entrypoint should log running one-time compatibility probe, probe succeeds, fixer re-writes the marker.
  • Base image smoke build: docker build -f deploy/docker/base.dockerfile . succeeds on linux/arm64. Image contains mongodb-org 7.0.31 and mongosh 2.8.2.
  • CE→EE sync: cherry-pick onto community/release and merge into EE release both apply cleanly (verified locally).
  • Fresh install smoke-tested locally with the current DP image — probe skipped on shouldPerformInitdb=0 gate, marker written by fixer after supervisord starts mongod.

Out of scope

  • deploy/helm/ chart still references Bitnami MongoDB 6.0.27 in Chart.yaml / values.yaml. That's tracked under a separate effort.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores

    • Upgraded bundled MongoDB from 6.0 to 7.0.
  • New Features

    • Added a startup compatibility probe that detects incompatible local/embedded MongoDB feature versions and halts startup with operator guidance when needed.
    • Added a persistent marker to record confirmed DB compatibility and skip repeated probes on subsequent startups.
    • Improved post-startup DB feature-version handling to record and persist the committed minimum FCV.

Warning

Tests have not run on the HEAD af34f2d yet


Tue, 05 May 2026 22:44:32 UTC

MongoDB 6.0 reached end of life, so we are bumping the embedded MongoDB
in the Docker image to 7.0. This is a major version upgrade and requires
the existing data to be at featureCompatibilityVersion (FCV) >= 6.0 —
MongoDB 7.0 refuses to start otherwise.

Changes:

- base.dockerfile: switch the apt repo, GPG key, and list file from the
  6.0 stream to 7.0. Stay on the jammy (22.04) packages since MongoDB
  does not publish a noble (24.04) apt repo — same pattern the 6.0
  install already used on Ubuntu 24.04.

- mongodb-fixer.sh: after confirming / raising FCV, write the observed
  value to a marker file (.appsmith-fcv) in the Mongo data directory.
  Keep the target FCV at 6.0 (not 7.0) to preserve the ability to
  downgrade to a 6.x Appsmith release. Add confirm:true to the
  setFeatureCompatibilityVersion call, required as of MongoDB 7.0.

- entrypoint.sh: add ensure_mongodb_fcv_compatible, a pre-flight check
  that reads the marker before handing off to supervisord. Fast path
  when the marker is present and >= 6.0; hard-fail with an actionable
  error (roll back to last 6.x release, let it boot, upgrade again) if
  the marker indicates FCV < 6.0. On the rare transitional case of
  marker-missing-with-data (first boot after upgrading from a pre-
  marker Appsmith release), do a one-time mongod --fork probe to
  confirm compatibility, then let the fixer write the marker after
  supervisord brings mongod up. Also seed the marker on fresh install
  inside init_replica_set so the probe is skipped on the very first
  boot of a new deployment.

This gives operators a clear error message when their data is
incompatible, without paying the cost of multiple mongod starts on
every container boot.
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 16, 2026

Walkthrough

Upgrades embedded MongoDB to 7.0 in the Docker base image; adds a one‑time startup pre‑flight probe in the container entrypoint to verify FCV compatibility for existing local MongoDB data; and adds a committed‑FCV marker + simplified FCV handling to mongodb-fixer.

Changes

MongoDB upgrade + FCV compatibility probe

Layer / File(s) Summary
Base image / repo
deploy/docker/base.dockerfile
Replaced MongoDB 6.0 GPG/keyring and APT source with MongoDB 7.0 equivalents (jammy) and updated comments about using jammy packages on Ubuntu 24.04.
Entrypoint: probe impl
deploy/docker/fs/opt/appsmith/entrypoint.sh
Added ensure_mongodb_fcv_compatible() which: gates on shouldPerformInitdb and isUriLocal, fast‑paths if marker $MONGO_DB_PATH/.appsmith-mongo-fcv-min exists, otherwise writes probe log, starts mongod --fork bound to localhost against $MONGO_DB_PATH, attempts shutdown, and on failure extracts FCV/upgrade logs, prints guidance, and exits 1.
Entrypoint: wiring
deploy/docker/fs/opt/appsmith/entrypoint.sh
Calls ensure_mongodb_fcv_compatible in the non‑Heroku startup path after init_mongodb and init_replica_set when isMongoUrl==1.
Fixer: marker & FCV handling
deploy/docker/fs/opt/appsmith/mongodb-fixer.sh
Introduced MONGO_FCV_MIN_MARKER and FCV_MIN; added write_fcv_marker() to atomically persist the FCV marker (non‑fatal on write failure). Changed readiness wait to loop until supervisor reports RUNNING and simplified FCV enforcement to compare against FCV_MIN, then record the committed FCV via the marker.
Notes / Docs
deploy/docker/...
No test or user‑facing docs changed in this diff.

Sequence Diagram

sequenceDiagram
    participant Entrypoint as Container Entrypoint
    participant MongoDB as Local MongoDB
    participant FS as File System

    Entrypoint->>Entrypoint: init_mongodb()
    Entrypoint->>Entrypoint: init_replica_set()
    Entrypoint->>FS: check `$MONGO_DB_PATH/.appsmith-mongo-fcv-min`
    alt Marker present
        FS-->>Entrypoint: marker exists
        Entrypoint->>Entrypoint: skip probe, continue startup
    else Marker absent and existing local DB
        Entrypoint->>MongoDB: start `mongod --fork` (bind localhost)
        alt Probe runs successfully
            MongoDB-->>Entrypoint: mongod running
            Entrypoint->>MongoDB: shutdown probe
            Entrypoint->>FS: write `.appsmith-mongo-fcv-min` marker
        else Probe fails
            MongoDB-->>Entrypoint: probe logs
            Entrypoint->>Entrypoint: extract FCV/upgrade log, print guidance, exit 1
        end
    else Fresh init or external URI
        Entrypoint->>Entrypoint: skip FCV probe
    end
Loading

Estimated Code Review Effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🔧 From six to seven the base is sewn,
A cautious probe wakes mongod alone.
A marker keeps the FCV true,
If logs complain, the startup rues.
Operators guided, containers shown.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main change: upgrading embedded MongoDB from 6.0 to 7.0, which aligns with all three modified files.
Description check ✅ Passed The PR description is comprehensive and well-structured with clear summaries of each file change, a detailed decision matrix, test plan with checkboxes, and links to tracking issues. However, it does not include an explicit 'Fixes #Issue Number' statement as required by the template.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch chore/mongo-7-upgrade

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

The entrypoint should only read the FCV marker, never write it — that
responsibility belongs solely to mongodb-fixer.sh. Fresh installs pay a
cheap one-time pre-flight probe on the first boot; the fixer writes the
marker on the same boot once supervisord brings mongod up, so subsequent
boots take the fast path.
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@deploy/docker/fs/opt/appsmith/entrypoint.sh`:
- Around line 341-344: Fresh installs hit the probe-only path because
entrypoint.sh's check for "$MONGO_DB_PATH/WiredTiger" runs after
init_replica_set() even though the .appsmith-fcv marker is only created later by
mongodb-fixer.sh; fix by either creating the .appsmith-fcv marker during
init_replica_set() (seed the marker when you initialize the embedded replica in
init_replica_set()) or by moving the WiredTiger/marker existence check earlier
(before local Mongo initialization in entrypoint.sh) so the fast-path applies to
brand-new installs; update the logic in init_replica_set() or reorder the check
in entrypoint.sh accordingly and ensure the marker name ".appsmith-fcv" is
consistently referenced.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9557b9a3-a1d9-4433-b412-3a373b0cf288

📥 Commits

Reviewing files that changed from the base of the PR and between da07ad2 and f73a3e0.

📒 Files selected for processing (3)
  • deploy/docker/base.dockerfile
  • deploy/docker/fs/opt/appsmith/entrypoint.sh
  • deploy/docker/fs/opt/appsmith/mongodb-fixer.sh

Comment thread deploy/docker/fs/opt/appsmith/entrypoint.sh Outdated
…anches

Three related simplifications:

- entrypoint: gate ensure_mongodb_fcv_compatible on shouldPerformInitdb=0
  at the call site. A fresh install has no data to be incompatible with.

- entrypoint: delete the "marker says FCV < 6.0" hard-fail branch. The
  fixer writes the marker only after observing current FCV on a running
  mongod 7, so marker presence implies a compatible FCV. The branch was
  unreachable.

- mongodb-fixer: drop the raise-to-6.0 FCV branch. The entrypoint's
  pre-flight probe ensures mongod won't start on FCV < 6.0, so the fixer
  never sees that state. The raise-FCV scaffolding can come back when the
  MongoDB 8 upgrade needs to bump FCV to 7.0.
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@deploy/docker/fs/opt/appsmith/entrypoint.sh`:
- Around line 345-347: The temporary mongod probe in entrypoint.sh currently
swallows shutdown errors with "mongod --shutdown >/dev/null 2>&1 || true", which
can leave dbpath/port locked and break the later supervisord start; change the
cleanup to detect a failed "mongod --shutdown" (referencing the probe run that
used "mongod --fork --port 27017 --dbpath \"$MONGO_DB_PATH\" --logpath
\"$probe_log\" --bind_ip localhost") and, on failure, log an explicit error (use
tlog or similar) and exit non‑zero instead of returning success; ensure the
shutdown command's exit status is checked and propagated so the entrypoint
aborts when shutdown fails.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 986e1e6b-a02d-4182-8e17-474fdd1c7abb

📥 Commits

Reviewing files that changed from the base of the PR and between f73a3e0 and a6a96df.

📒 Files selected for processing (2)
  • deploy/docker/fs/opt/appsmith/entrypoint.sh
  • deploy/docker/fs/opt/appsmith/mongodb-fixer.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • deploy/docker/fs/opt/appsmith/mongodb-fixer.sh

Comment thread deploy/docker/fs/opt/appsmith/entrypoint.sh Outdated
…atible

Matches the pattern used by sibling mongo-bootstrap logic in the same
file (e.g., init_replica_set gates its fresh-init branch on
shouldPerformInitdb/isUriLocal internally). Keeps "when does this run"
in one place instead of splitting it between the function body and the
main section.

No behavior change.
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
deploy/docker/fs/opt/appsmith/entrypoint.sh (1)

348-350: ⚠️ Potential issue | 🟠 Major

Don’t ignore probe shutdown failures.

Line 349 treats probe cleanup failure as success. If that mongod stays alive, the later supervisord start can fail with a stale lock or busy port instead of the real cause.

Suggested fix
   if mongod --fork --port 27017 --dbpath "$MONGO_DB_PATH" --logpath "$probe_log" --bind_ip localhost >/dev/null 2>&1; then
-    mongod --dbpath "$MONGO_DB_PATH" --shutdown >/dev/null 2>&1 || true
+    if ! mongod --dbpath "$MONGO_DB_PATH" --shutdown >/dev/null 2>&1; then
+      tlog "== ERROR: failed to stop the MongoDB compatibility probe" >&2
+      exit 1
+    fi
     tlog "Pre-flight probe succeeded; mongodb-fixer will write the FCV marker after supervisord starts mongod"
     return
   fi
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@deploy/docker/fs/opt/appsmith/entrypoint.sh` around lines 348 - 350, The
probe cleanup currently ignores failures when shutting down the probe mongod
(the command using mongod --dbpath "$MONGO_DB_PATH" --shutdown || true), which
can leave a live mongod and cause supervisord to fail later; modify the
entrypoint.sh logic around the probe (the initial mongod --fork ... using
$MONGO_DB_PATH and $probe_log and the subsequent shutdown command) to check the
shutdown exit status instead of swallowing it: if shutdown fails, log a clear
error via tlog (including the probe_log path and exit code), attempt a safe
fallback (e.g., try a graceful kill or report the PID if available), and exit
non‑zero so the container fails fast rather than proceeding to supervisord with
a stale mongod.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@deploy/docker/fs/opt/appsmith/entrypoint.sh`:
- Around line 354-367: The current error block prints only the probe logfile
path ($probe_log) when probe_err is empty, making post-failure debugging hard;
modify the failure-handling block (the code referencing probe_err, probe_log and
tlog in entrypoint.sh) to print the actual contents of the probe log to stderr
instead of or in addition to the temp-file path: after computing probe_err, if
probe_err is empty print a clear header via tlog and then stream the contents of
$probe_log to stderr (e.g., using cat or tail) so the full mongod probe output
is visible in the container logs; keep the existing messages (FCV hint and Full
mongod log header) but replace the lone filepath emission with the log dump.

---

Duplicate comments:
In `@deploy/docker/fs/opt/appsmith/entrypoint.sh`:
- Around line 348-350: The probe cleanup currently ignores failures when
shutting down the probe mongod (the command using mongod --dbpath
"$MONGO_DB_PATH" --shutdown || true), which can leave a live mongod and cause
supervisord to fail later; modify the entrypoint.sh logic around the probe (the
initial mongod --fork ... using $MONGO_DB_PATH and $probe_log and the subsequent
shutdown command) to check the shutdown exit status instead of swallowing it: if
shutdown fails, log a clear error via tlog (including the probe_log path and
exit code), attempt a safe fallback (e.g., try a graceful kill or report the PID
if available), and exit non‑zero so the container fails fast rather than
proceeding to supervisord with a stale mongod.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e7dfca11-861c-46b1-8aff-2fa7b9df2803

📥 Commits

Reviewing files that changed from the base of the PR and between a6a96df and 2f95cfe.

📒 Files selected for processing (1)
  • deploy/docker/fs/opt/appsmith/entrypoint.sh

Comment thread deploy/docker/fs/opt/appsmith/entrypoint.sh
…n probe shutdown

- Use -gt 0 for the gate check to match the style used by init_replica_set
  elsewhere in this file.
- Drop the "rare" framing on the no-marker branch — every upgraded instance
  takes this path exactly once, so it's the expected first-boot flow.
- "operator" -> "administrator" to avoid collision with the Kubernetes
  operator pattern.
- Abort explicitly if the probe's --shutdown fails instead of swallowing
  the error with || true. Probe mongod and supervisord's mongod share port
  27017 and the data lock, so a silent shutdown failure would make
  supervisord's mongod fail to start — exactly the "confusing degraded
  state" this pre-flight exists to prevent.
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
deploy/docker/fs/opt/appsmith/entrypoint.sh (1)

356-369: ⚠️ Potential issue | 🟠 Major

Print the probe log contents before exiting.

If probe_err is empty, the container logs still end with just a temp-file path. That makes unexpected probe failures hard to diagnose once the container exits.

Suggested diff
   tlog "== To upgrade safely:" >&2
   tlog "==   1. Roll back to Appsmith v1.99 (the last release shipped with MongoDB 6.x)" >&2
   tlog "==   2. Let the container start fully — it will raise the compatibility version to 6.0 automatically" >&2
   tlog "==   3. Shut down, then upgrade to this release" >&2
   tlog "==" >&2
-  tlog "== Full mongod log: $probe_log" >&2
+  tlog "== Full mongod log follows (from $probe_log):" >&2
+  sed 's/^/==   /' "$probe_log" >&2 || tlog "==   (unable to read $probe_log)" >&2
   tlog "====================================================================================================" >&2
   exit 1
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@deploy/docker/fs/opt/appsmith/entrypoint.sh` around lines 356 - 369, The
current exit path prints only the probe_log path when probe_err is empty, making
debugging difficult; modify the block around probe_err/probe_log (references:
probe_err, probe_log, tlog) so that before exiting you always output the actual
contents of "$probe_log" to stderr (use tlog to print a header/footer and then
cat the file to >&2), and ensure this happens whether or not probe_err is
non-empty so callers see the full mongod probe output instead of just a
temp-file path.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@deploy/docker/fs/opt/appsmith/entrypoint.sh`:
- Around line 356-369: The current exit path prints only the probe_log path when
probe_err is empty, making debugging difficult; modify the block around
probe_err/probe_log (references: probe_err, probe_log, tlog) so that before
exiting you always output the actual contents of "$probe_log" to stderr (use
tlog to print a header/footer and then cat the file to >&2), and ensure this
happens whether or not probe_err is non-empty so callers see the full mongod
probe output instead of just a temp-file path.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 17995bfe-a780-4ef7-b98e-f79144361501

📥 Commits

Reviewing files that changed from the base of the PR and between 2f95cfe and 912ad5d.

📒 Files selected for processing (1)
  • deploy/docker/fs/opt/appsmith/entrypoint.sh

@wyattwalter
Copy link
Copy Markdown
Contributor Author

/build-deploy-preview base-image-tag=chore-mongo-7-upgrade skip-tests=true

@github-actions
Copy link
Copy Markdown

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/24525894004.
Workflow: On demand build Docker image and deploy preview.
skip-tests: true.
env: ``.
PR: 41743.
recreate: .
base-image-tag: chore-mongo-7-upgrade.

@github-actions
Copy link
Copy Markdown

Deploy-Preview-URL: https://ce-41743.dp.appsmith.com

The old marker (.appsmith-fcv) recorded "the observed FCV at the moment
mongod came up." That was misleading in two ways:
- On fresh installs, mongod 7.0 sets FCV to 7.0 by default, so the file
  read "7.0" even though this release commits to preserving 6.0 (for
  downgrade safety).
- Nothing was actually enforcing the written value — FCV can drift via
  setFeatureCompatibilityVersion, and the marker never updates.

Reframed: the marker now records the *minimum FCV this Appsmith release
commits to preserve* — a release-level contract, constant across all
installs of this release. Value is "6.0" until MongoDB 8 forces a bump
to 7.0, at which point the constant and the corresponding setFCV call
both change together. Useful for future releases deciding "can this
data still be upgraded?"

Changes:
- File renamed: .appsmith-fcv -> .appsmith-mongo-fcv-min.
- Fixer writes the FCV_MIN constant instead of querying mongod. The
  mongosh retry loop that did the FCV query is gone — the marker's
  value no longer depends on a live reading.
- The "wait for mongod RUNNING" supervisorctl loop in the fixer was
  previously semantically inverted (`while ... RUNNING; do sleep ...`
  looped WHILE running, exited on not-running). It happened to work
  because the mongosh retry loop after it did the real wait. With the
  mongosh loop removed, the supervisorctl loop becomes load-bearing, so
  fix it with the missing `!`.
- entrypoint.sh's pre-flight check still only reads presence of the
  marker; only the path and the header comment change.
@wyattwalter
Copy link
Copy Markdown
Contributor Author

/build-deploy-preview base-image-tag=chore-mongo-7-upgrade skip-tests=true recreate=true

@github-actions
Copy link
Copy Markdown

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/24534289736.
Workflow: On demand build Docker image and deploy preview.
skip-tests: true.
env: ``.
PR: 41743.
recreate: true.
base-image-tag: chore-mongo-7-upgrade.

@github-actions
Copy link
Copy Markdown

This PR has not seen activitiy for a while. It will be closed in 7 days unless further activity is detected.

@github-actions github-actions Bot added the Stale label Apr 24, 2026
tlog "supervisor.sock found"

while supervisorctl status mongodb | grep -q RUNNING; do
while ! supervisorctl status mongodb | grep -q RUNNING; do
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a max attempt counter here to avoid unbounded looping?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't get to supervisord at least running, it'll kill the container because the health checks will fail anyway. I think it's not worth the risk of this dying too early.

Comment thread deploy/docker/fs/opt/appsmith/entrypoint.sh
fi

tlog "No MongoDB FCV marker found on existing data; running one-time compatibility probe"
local probe_log="$TMP/mongo-fcv-probe.log"
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this error log be lost because the container exited for any possible reasons?

wyattwalter and others added 2 commits May 1, 2026 14:17
Two PR-feedback fixes:

1. Move the pre-flight probe log from $TMP/mongo-fcv-probe.log to
   $MONGO_DB_PATH/fcv-probe.log so it survives container restarts.
   $TMP gets wiped, which left admins with no forensic trail when
   investigating why their container exited. The log now lives next
   to the MongoDB data and uses --logappend with a UTC timestamp
   header per probe run, so repeated upgrade attempts all stay on
   record.

2. Restore the active setFeatureCompatibilityVersion call in
   mongodb-fixer.sh, gated on FCV being below $FCV_MIN. In the
   steady state this is a no-op (the pre-flight probe already
   guarantees mongod won't start on data below the supported FCV),
   but keeping the call live makes the upgrade scaffolding
   self-documenting and reduces the next major-version bump to a
   constant change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@wyattwalter
Copy link
Copy Markdown
Contributor Author

/build-deploy-preview base-image-tag=chore-mongo-7-upgrade skip-tests=true

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 4, 2026

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/25331209961.
Workflow: On demand build Docker image and deploy preview.
skip-tests: true.
env: ``.
PR: 41743.
recreate: .
base-image-tag: chore-mongo-7-upgrade.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 4, 2026

Deploy-Preview-URL: https://ce-41743.dp.appsmith.com

@github-actions github-actions Bot removed the Stale label May 4, 2026
PR feedback — make the recovery instructions more actionable for
admins seeing this for the first time:

- Reassure up front that data has NOT been altered. The container
  exits before mongod starts, so on-disk files are untouched and no
  backup restore is needed. This was implicit before; now explicit.
- Frame each step as an Appsmith deployment action ("Alter your
  Appsmith deployment to use v1.99 ...") rather than a generic
  "roll back". Operators running Helm, ECS, k8s manifests, etc.
  recognize this as their own task immediately.
- Step 2 names "MongoDB compatibility version" explicitly so it's
  clear what's being changed under the hood.
- Add blank `==` separator lines around the steps so the recovery
  block stands out in a wall of stderr.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@wyattwalter
Copy link
Copy Markdown
Contributor Author

/build-deploy-preview base-image-tag=chore-mongo-7-upgrade skip-tests=true

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 4, 2026

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/25339757487.
Workflow: On demand build Docker image and deploy preview.
skip-tests: true.
env: ``.
PR: 41743.
recreate: .
base-image-tag: chore-mongo-7-upgrade.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 4, 2026

Deploy-Preview-URL: https://ce-41743.dp.appsmith.com

Reframe the error block to convey the full upgrade story rather than
just the recovery steps. Feedback was that the previous wording
under-explained why this happens and over-promised on data safety.

Now covers:
- Appsmith 2.x ships MongoDB 7.x; FCV must be 6.0 or higher.
- 1.93–1.99 raise FCV to 6.0 automatically; only pre-1.66-only
  instances are at risk.
- This check applies only to embedded MongoDB. External MongoDB
  instances bypass it entirely.
- The failure happens before any Appsmith service comes online, so
  no Appsmith database migrations have been attempted — rolling back
  to a 1.x release is just an image version swap.

Drops the prior "your data has NOT been altered" line, which
overpromised: mongod 7.0 may have touched journal/lock files just
attempting to start. The accurate and more useful guarantee is that
Appsmith's own migrations haven't run, which is what's stated now.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@wyattwalter
Copy link
Copy Markdown
Contributor Author

/build-deploy-preview base-image-tag=chore-mongo-7-upgrade skip-tests=true

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 5, 2026

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/25395925559.
Workflow: On demand build Docker image and deploy preview.
skip-tests: true.
env: ``.
PR: 41743.
recreate: .
base-image-tag: chore-mongo-7-upgrade.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 5, 2026

Deploy-Preview-URL: https://ce-41743.dp.appsmith.com

The stepping-stone range and FCV-5.x risk threshold in the error
message were both off:

- Stepping stone for raising FCV to 6.0 is releases 1.96 to 1.99
  (was: 1.93 to 1.99). Earlier 1.9x releases didn't yet have the
  FCV-bump logic.
- The MongoDB 5 → 6 upgrade landed in 1.70, not 1.66, so the
  at-risk population is instances that have only ever run Appsmith
  older than 1.70.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (4)
deploy/docker/fs/opt/appsmith/entrypoint.sh (2)

373-382: 💤 Low value

Verify port 27017 is free at probe time.

This relies on the invariant that supervisord hasn't started yet and init_replica_set() left no mongod running on 27017. Currently true because init_replica_set only forks mongod when shouldPerformInitdb=1, and ensure_mongodb_fcv_compatible short-circuits on that same condition. Worth a brief comment so a future refactor doesn't quietly break the assumption.

♻️ Suggested tweak
   tlog "No MongoDB FCV marker found on existing data; running one-time compatibility probe"
+  # Safe to bind 27017 here: supervisord hasn't started yet, and init_replica_set
+  # only forks mongod on a fresh install — a path this function early-returns on.
   # Persist the probe log inside the Mongo data directory so it survives container
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/docker/fs/opt/appsmith/entrypoint.sh` around lines 373 - 382, The
pre-flight probe assumes port 27017 is free because supervisord hasn't started
mongod and init_replica_set() does not leave a mongod running; add a brief
inline comment above the probe block referencing this invariant and the related
functions (init_replica_set and ensure_mongodb_fcv_compatible) so future changes
don't break the assumption — e.g., note that init_replica_set() only forks
mongod when shouldPerformInitdb=1 and ensure_mongodb_fcv_compatible
short-circuits on the same condition, therefore the probe can safely bind to
port 27017 using the local probe_log and mongod invocation.

384-408: 💤 Low value

Optional: tail the probe log to stderr on failure.

docker logs already shows everything written here, but the actual mongod failure lines live only in $probe_log on the volume. Streaming the last ~50 lines inline saves a docker exec/volume-mount roundtrip when triaging. The volume copy still wins for full forensics.

♻️ Suggested tweak
   tlog "==" >&2
-  tlog "== Full mongod log: $probe_log" >&2
+  tlog "== Last lines of the mongod probe log (full log at $probe_log):" >&2
+  tail -n 50 "$probe_log" 2>/dev/null | sed 's/^/==   /' >&2 || true
   tlog "====================================================================================================" >&2
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/docker/fs/opt/appsmith/entrypoint.sh` around lines 384 - 408, The
failure message block prints the path to the mongod probe log ($probe_log) but
doesn't stream recent lines; update the error handling (the block that computes
probe_err and prints via tlog and exits) to also tail the last ~50 lines of
$probe_log to stderr (e.g. run tail -n 50 "$probe_log" 2>/dev/null >&2 after
printing "Full mongod log: $probe_log") so triage can see the actual mongod
failure lines without an extra docker exec; ensure the tail is guarded to not
fail if $probe_log is missing.
deploy/docker/fs/opt/appsmith/mongodb-fixer.sh (2)

51-66: 💤 Low value

FCV-floor loop swallows ultimate failure, then writes the marker anyway.

If all 60 mongosh attempts fail (e.g., transient auth/replica-set hiccup), the loop falls through silently and write_fcv_marker still records the floor. Per the marker's documented "release-level contract" semantics this may be intentional — but if the intent is "marker = mongod confirmed booted under this release at floor", a soft-warn on exhaustion would aid diagnosis without changing behavior.

♻️ Suggested tweak
 tlog "Ensuring MongoDB featureCompatibilityVersion is at least $FCV_MIN"
+fcv_confirmed=0
 for _ in {1..60}; do
   if mongosh --quiet "$APPSMITH_DB_URL" --eval '
     const floor = '"$FCV_MIN"';
     const current = parseFloat(db.adminCommand({getParameter: 1, featureCompatibilityVersion: 1}).featureCompatibilityVersion.version);
     if (current < floor) {
       db.adminCommand({setFeatureCompatibilityVersion: "'"$FCV_MIN"'", confirm: true});
     }
   '; then
     tlog "MongoDB featureCompatibilityVersion floor of $FCV_MIN confirmed"
+    fcv_confirmed=1
     break
   fi
   sleep 1
 done
+if [[ $fcv_confirmed -ne 1 ]]; then
+  tlog "warning: could not confirm FCV floor after 60 attempts; recording marker regardless (release-level contract)"
+fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/docker/fs/opt/appsmith/mongodb-fixer.sh` around lines 51 - 66, The
loop that runs mongosh to enforce FCV_MIN may exit after 60 failed attempts but
currently proceeds to write_fcv_marker("$FCV_MIN") regardless; update the block
around the for-loop (the mongosh loop using APPSMITH_DB_URL and FCV_MIN with
tlog) to detect exhaustion (e.g., set a local flag like fcv_confirmed=false and
set it true on successful break or check the last exit status) and if not
confirmed emit a soft warning via tlog (e.g., tlog "WARNING: failed to confirm
FCV floor $FCV_MIN after retries") before calling write_fcv_marker so the marker
is still written but operators get a clear warning.

23-31: 💤 Low value

Nit: stale .tmp file may linger if printf fails.

If the temp write fails (no space, perms), the partially-created ${MONGO_FCV_MIN_MARKER}.tmp could be left on disk. Cheap insurance to clean it up.

♻️ Suggested tweak
   if ! printf '%s\n' "$value" > "$tmp" 2>/dev/null; then
     tlog "warning: failed to write FCV marker temp file"
+    rm -f "$tmp" 2>/dev/null || true
     return 0
   fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/docker/fs/opt/appsmith/mongodb-fixer.sh` around lines 23 - 31, The
write_fcv_marker function can leave a stale temp file if the printf write fails;
modify write_fcv_marker (and keep using the MONGO_FCV_MIN_MARKER and tmp
variables) so that when the printf '> "$tmp"' command fails you remove the tmp
file (e.g., rm -f "$tmp") before returning, and keep the existing warning
log/return behavior so the temp file is not left on disk.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@deploy/docker/fs/opt/appsmith/entrypoint.sh`:
- Around line 373-382: The pre-flight probe assumes port 27017 is free because
supervisord hasn't started mongod and init_replica_set() does not leave a mongod
running; add a brief inline comment above the probe block referencing this
invariant and the related functions (init_replica_set and
ensure_mongodb_fcv_compatible) so future changes don't break the assumption —
e.g., note that init_replica_set() only forks mongod when shouldPerformInitdb=1
and ensure_mongodb_fcv_compatible short-circuits on the same condition,
therefore the probe can safely bind to port 27017 using the local probe_log and
mongod invocation.
- Around line 384-408: The failure message block prints the path to the mongod
probe log ($probe_log) but doesn't stream recent lines; update the error
handling (the block that computes probe_err and prints via tlog and exits) to
also tail the last ~50 lines of $probe_log to stderr (e.g. run tail -n 50
"$probe_log" 2>/dev/null >&2 after printing "Full mongod log: $probe_log") so
triage can see the actual mongod failure lines without an extra docker exec;
ensure the tail is guarded to not fail if $probe_log is missing.

In `@deploy/docker/fs/opt/appsmith/mongodb-fixer.sh`:
- Around line 51-66: The loop that runs mongosh to enforce FCV_MIN may exit
after 60 failed attempts but currently proceeds to write_fcv_marker("$FCV_MIN")
regardless; update the block around the for-loop (the mongosh loop using
APPSMITH_DB_URL and FCV_MIN with tlog) to detect exhaustion (e.g., set a local
flag like fcv_confirmed=false and set it true on successful break or check the
last exit status) and if not confirmed emit a soft warning via tlog (e.g., tlog
"WARNING: failed to confirm FCV floor $FCV_MIN after retries") before calling
write_fcv_marker so the marker is still written but operators get a clear
warning.
- Around line 23-31: The write_fcv_marker function can leave a stale temp file
if the printf write fails; modify write_fcv_marker (and keep using the
MONGO_FCV_MIN_MARKER and tmp variables) so that when the printf '> "$tmp"'
command fails you remove the tmp file (e.g., rm -f "$tmp") before returning, and
keep the existing warning log/return behavior so the temp file is not left on
disk.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 88564e7f-adbd-441e-8e4e-40ac31ebe56f

📥 Commits

Reviewing files that changed from the base of the PR and between 912ad5d and af34f2d.

📒 Files selected for processing (3)
  • deploy/docker/base.dockerfile
  • deploy/docker/fs/opt/appsmith/entrypoint.sh
  • deploy/docker/fs/opt/appsmith/mongodb-fixer.sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants