Skip to content

Support synchronizing a flattened superproject with upstream Git commits and submodule changes #184

Closed

Description

@olfway

What you are trying to do

I maintain a fork of an upstream Git repository that uses Git submodules, including recursive/nested submodules.

I want to flatten all submodules into regular directories, while keeping the original repository structure and continuing to synchronize with the upstream superproject.

Importantly, I want different history handling for the superproject and its submodules:

  • Superproject: Preserve individual upstream commits and their history (do not squash them into a single commit).
  • Submodules: Import their contents as regular files, squashing upstream changes into one commit per synchronization rather than importing their entire history.
  • Local modifications: Allow local commits inside flattened submodule directories and preserve them across upstream updates.

The main problem arises when upstream changes .gitmodules or submodule gitlinks.

For example, upstream might add or remove a submodule, change its URL, or update its pinned commit. These changes should be reflected in my flattened repository without restoring Git submodules.

What you would like git-nested to do

Support synchronizing a flattened superproject at the repository root with its original upstream repository, including changes to .gitmodules and gitlinks.

For example, a hypothetical command:

git fetch upstream
git nested sync-upstream upstream/main --recurse-submodules

Expected behavior:

  1. Incorporate upstream superproject commits individually, preserving their commit boundaries, messages, and authors where possible (even if rewritten commit SHAs are necessary).
  2. Detect changes to .gitmodules and submodule gitlinks.
  3. Automatically flatten newly added submodules, recursively.
  4. Update existing flattened submodules to their upstream-pinned commits.
  5. Handle removed, renamed, or relocated submodules and changed remote URLs.
  6. Preserve locally committed modifications inside flattened submodules, using Git's normal merge/conflict-resolution mechanisms.
  7. Keep all content as regular files and directories, without reintroducing gitlinks.
  8. Squash imported submodule changes into one commit per submodule update, without squashing unrelated superproject commits.

The distinction between preserving the superproject's commit history and squashing submodule updates is particularly important.

What you do instead today

git nested init --from-submodule can convert individual submodules, but it does not appear to solve subsequent synchronization with the original upstream superproject.

After flattening the submodules, a normal:

git fetch upstream
git merge upstream/main

can introduce conflicts between upstream gitlinks and local regular directories, or reintroduce submodules when upstream adds new ones.

The workaround appears to require manually inspecting .gitmodules and gitlink changes, resolving structural conflicts, and updating each flattened submodule separately.

Alternatively, I could import the entire upstream repository using:

git nested clone --recurse-submodules \
  https://example.com/upstream.git project

However, this has two important limitations:

  1. It changes the repository's root layout, placing the whole superproject inside an additional directory.
  2. It squashes the superproject's history. The initial import produces a single commit, and subsequent pulls also squash upstream superproject commits into one commit per pull. I need to retain the individual superproject commits, while squashing only the submodule updates.

Is there already a supported workflow for maintaining such a flattened fork, preserving the superproject's commit granularity while recursively flattening and synchronizing its submodules?

Activity

  1. thorsten-klein commented on Nov 10, 2026

    @thorsten-klein
    Owner

    Hi @olfway, thanks for the clear write-up. A first version is ready in #202. Can you try it on your fork, and tell me if it fits what you need?

    Install the branch

    In a virtual environment:

    uv pip install "git+https://github.com/thorsten-klein/git-nested@feat/184-sync-gitlinks"
    

    Try it on your fork

    Best on a new branch, so nothing changes on your main branch:

    cd <your fork>
    git switch -c try-git-nested
    git remote add upstream <url of the project you forked>   # if you don't have it yet
    
    git nested submodules init --recursive                # nest all submodules, also deeper ones
    git nested submodules pull --dry-run upstream main    # see what a sync would do
    git nested submodules pull upstream main              # sync
    

    If your .gitmodules has relative urls like ../lib.git, init stops. Give it the url of the project you forked: git nested submodules init --recursive --resolve-relative <url>.

    To nest only some submodules, use --only <path> or --ignore <path>, more than once if needed. The others stay submodules.

    If you fetch yourself, this works too:

    git fetch upstream
    git nested submodules merge upstream/main
    

    Or merge with git itself. git stops on the changed submodules. Then submodules pull finishes the merge:

    git merge upstream/main
    git nested submodules pull
    

    What you get

    • Each upstream commit stays, with its message and author.
    • Each changed submodule gets one commit per sync.
    • New, removed, moved and re-pointed submodules are handled, also inside other submodules.
    • Your changes in the submodule folders stay. On conflicts you fix them with git, then run git nested submodules pull --continue.
    • No gitlink comes back in your own commits.

    Two things differ from your idea

    1. The upstream commits are merged as they are. Their SHAs stay the same. But they still have their gitlinks. A checkout of an old upstream commit shows empty submodule folders. Use git bisect --first-parent. We did this on purpose: with rewritten commits, every later sync would need a map of old to new SHAs. It would also get much slower and break more easily.
    2. By default, the merge commit comes first, then one commit per submodule. If every commit must build, use --single-commit.

    The docs are in doc/cli/submodules.md.

    One question: is your fork public? If yes, please share the link. Then I can try it on your real repo and fine-tune it.

  2. olfway commented on Nov 12, 2026

    @olfway
    Author

    Hi @thorsten-klein , thanks for your work!

    I'll try to test it soon, and will write you how it works

    My forks are local only for now, but upstreams are public:

    they both has a lot of submodules

  3. olfway commented on Nov 13, 2026

    @olfway
    Author

    I tested it on tdesktop, and it already works well enough
    to replace my own update scripts. Thanks a lot for building this!

    I used feat/184-sync-gitlinks (8639728) on a fresh clone of tdesktop:

    • submodules init --recursive at c2694a2
    • merge 7381fb1: 11 pins moved, 3 new submodules, deeper ones too
    • pull origin dev to 346f10f

    After each step, the tree matched a git submodule update --init --recursive
    checkout, except for the items below.
    Local changes in a nested repo stayed.
    A conflict there worked with --continue.

    Problems I found:

    1. update = none submodules get nested.
      git skips them.
      In tdesktop, quickjs/.gitmodules has update = none for test262,
      but git-nested nests it: 57k of 77k files (74%), 239 MB.

    2. --abort leaves a half sync.
      After a conflict in a nested repo, --abort keeps the merge commit
      and the pulls before the conflict.
      The conflicted submodule and the ones after it stay at the old pins.
      Then submodules merge <same target> says already up to date,
      so nothing updates them until upstream moves them again.
      The conflict worktree .git/tmp/nested/<path> also stays
      (git nested clean removes it).

    3. The # git-nested: block conflicts in .gitattributes.
      It is at the end of the file, where upstream adds rules too.
      In tdesktop, upstream added *.bin binary, and the merge stopped.
      git-nested owns the block, so it could take upstream's file
      and add the block again at the end.

    4. --dry-run misses deeper submodules.
      It showed widget-sandbox: would pull,
      but the real run also pulled widget-sandbox/.../quickjs.

    5. CRLF in the checkout (minor).
      With *.bat text eol=crlf in the parent, LF .bat files in nested repos
      are checked out with CRLF. A submodule keeps them LF.
      The blobs are the same.
      The -text lines cover only files stored with CRLF.

    6. The conflict message during submodules is confusing (minor).
      It first shows the 7 steps of a normal pull, which end with
      git nested commit <path>. These steps do not say that you must
      then run git nested submodules merge --continue, so the other
      submodules stay at the old pins. Then "Shorter: ... run
      git nested submodules merge --continue" follows.
      For submodules, showing only the --continue way would be clearer.

  4. thorsten-klein commented on Nov 14, 2026

    @thorsten-klein
    Owner

    Hi @olfway, thanks a lot for testing it on tdesktop, and for the clear feedback and explanations. The first version is now merged in #202. I tried to tackle the findings from your last comment. If you find more, please open a new issue, so we can tackle it in a follow-up PR.

    The steps are the same as before, but pull is now update. It takes a commit and does not fetch:

    uv pip install "git+https://github.com/thorsten-klein/git-nested"
    
    cd <your fork>
    git switch -c try-git-nested
    git remote add upstream <url of the project you forked>   # if you don't have it yet
    
    git nested submodules init --recursive                  # nest all submodules, also deeper ones
    git fetch upstream
    git nested submodules update --dry-run upstream/main    # see what a sync would do
    git nested submodules update upstream/main              # sync
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions