{"thread":{"id":"65715","subject":"Suggetsions for collaboration workflows in large repos","startedAt":"2026-05-29T16:31:20Z","lastAt":"2026-06-03T13:44:27Z","messageCount":5,"participants":["Matthew Hughes","Ben Knoble","Toon Claes"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"544275","messageId":"20260529163117.z2auhbg4sdxxgmis@archP14s","threadId":"65715","inReplyTo":null,"subject":"Suggetsions for collaboration workflows in large repos","fromName":"Matthew Hughes","fromEmail":"matthewhughes934@gmail.com","sentAt":"2026-05-29T16:31:17Z","receivedAt":"2026-05-29T16:31:20Z","isPatch":false,"body":"Hi,\n\nI'm looking for some git workflow suggestions to help cut down on unnecessary\nfetching when working in a large repo with many (hundreds) of other devs and\nthousands of branches. Specifically, if in this repo I use the common config to\njust fetch all the remote heads:\n\n    $ git config set remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'\n\nThen I find I get a lot of noise from the all the branches being\ncreated/updated/deleted as well as an increase in the size of my local repo due\nto all the objects I need to fetch across all those branches.\n\nTo clarify the general performance of git in this repo is reasonable (shoutout\nto `scalar`) but I am interested in cutting down on this fetching since when\nworking in this repo I'm generally only interested in a tiny subset of all\nbranches:\n\n1. The `main` branch (that everyone merges into)\n2. Any of _my_ branches\n3. Occasionally, one of my colleagues branches, so e.g. I can check out their\n   code locally to review (most reviewing I do in the web UI, this is\n   GitHub)\n\nI have a prefix for all my branches: `mhughes-`, so to sort out just the\nfirst two points I can configure git to fetch `main` and references with that\nprefix:\n\n    $ git config set --comment 'fetch main' remote.origin.fetch '+refs/heads/main:refs/remotes/origin/main'\n    $ git config set --append --comment 'fetch my branches' remote.origin.fetch '+refs/heads/mhughes-*:refs/remotes/origin/mhughes-*'\n\nBut then when I do want to check out a colleague's branch I need to explicitly\nfetch the exact ref like:\n\n    $ git fetch origin some-colleague-branch\n    $ git checkout FETCH_HEAD -b some-colleague-branch\n\nWhich is ok (it's my current workflow), but it means I have to re-fetch the\nexact ref if I want to bring in changes that they make after my initial fetch\n\nI could add an explicit fetch of their branch like:\n\n    $ git config set --append remote.origin.fetch '+refs/heads/some-colleague-branch:refs/remotes/origin/some-colleague-branch'\n\nSo that each `git fetch` also brings in updates to that branch, but in the\nremote we delete branches once their changes are merged, so if I leave that\nconfig I'll eventually (once they merge their change and delete the branch) run\ninto errors when fetching like:\n\n    fatal: couldn't find remote ref refs/heads/some-colleague-branch\n\nDoes anyone have suggestions to make this smoother? Or alternative workflows\nfor achieving this goal? I'd also be curious to hear about other approaches\npeople take went working in large repos with lots of other collaborators.\nOr am I just using git wrong in a repo like this, and should adopt another\napproach?\n\nI thought about doing something like tracking\n`refs/heads*/some-colleague-branch` from the remote, since with the wildcard\n`*` I at least won't the fatal error on the missing reference during fetch, but\nthat risks my config containing an ever growing list of such wildcards, or a\nbunch of manual work occasionally cleaning up old ones (or maybe that could be\nautomated).\n\nThanks,\nMatt\n"},{"id":"544278","messageId":"82F556A1-A5C6-414E-8EFB-13F83FA30E44@gmail.com","threadId":"65715","inReplyTo":"20260529163117.z2auhbg4sdxxgmis@archP14s","subject":"Re: Suggetsions for collaboration workflows in large repos","fromName":"Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2026-05-29T17:56:02Z","receivedAt":"2026-05-29T17:56:13Z","isPatch":false,"body":"\n> Le 29 mai 2026 à 12:47, Matthew Hughes <matthewhughes934@gmail.com> a écrit :\n> \n> Hi,\n> \n> I'm looking for some git workflow suggestions to help cut down on unnecessary\n> fetching when working in a large repo with many (hundreds) of other devs and\n> thousands of branches. Specifically, if in this repo I use the common config to\n> just fetch all the remote heads:\n> \n>    $ git config set remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'\n> \n> Then I find I get a lot of noise from the all the branches being\n> created/updated/deleted as well as an increase in the size of my local repo due\n> to all the objects I need to fetch across all those branches.\n> \n> To clarify the general performance of git in this repo is reasonable (shoutout\n> to `scalar`) but I am interested in cutting down on this fetching since when\n> working in this repo I'm generally only interested in a tiny subset of all\n> branches:\n> \n> 1. The `main` branch (that everyone merges into)\n> 2. Any of _my_ branches\n> 3. Occasionally, one of my colleagues branches, so e.g. I can check out their\n>   code locally to review (most reviewing I do in the web UI, this is\n>   GitHub)\n> \n> I have a prefix for all my branches: `mhughes-`, so to sort out just the\n> first two points I can configure git to fetch `main` and references with that\n> prefix:\n> \n>    $ git config set --comment 'fetch main' remote.origin.fetch '+refs/heads/main:refs/remotes/origin/main'\n>    $ git config set --append --comment 'fetch my branches' remote.origin.fetch '+refs/heads/mhughes-*:refs/remotes/origin/mhughes-*'\n> \n> But then when I do want to check out a colleague's branch I need to explicitly\n> fetch the exact ref like:\n> \n>    $ git fetch origin some-colleague-branch\n>    $ git checkout FETCH_HEAD -b some-colleague-branch\n> \n> Which is ok (it's my current workflow), but it means I have to re-fetch the\n> exact ref if I want to bring in changes that they make after my initial fetch\n> \n> I could add an explicit fetch of their branch like:\n> \n>    $ git config set --append remote.origin.fetch '+refs/heads/some-colleague-branch:refs/remotes/origin/some-colleague-branch'\n> \n> So that each `git fetch` also brings in updates to that branch, but in the\n> remote we delete branches once their changes are merged, so if I leave that\n> config I'll eventually (once they merge their change and delete the branch) run\n> into errors when fetching like:\n> \n>    fatal: couldn't find remote ref refs/heads/some-colleague-branch\n> \n> Does anyone have suggestions to make this smoother? Or alternative workflows\n> for achieving this goal? I'd also be curious to hear about other approaches\n> people take went working in large repos with lots of other collaborators.\n> Or am I just using git wrong in a repo like this, and should adopt another\n> approach?\n> \n> I thought about doing something like tracking\n> `refs/heads*/some-colleague-branch` from the remote, since with the wildcard\n> `*` I at least won't the fatal error on the missing reference during fetch, but\n> that risks my config containing an ever growing list of such wildcards, or a\n> bunch of manual work occasionally cleaning up old ones (or maybe that could be\n> automated).\n> \n> Thanks,\n> Matt\n\nMy current advice is to enable git-maintenance on such a repo, where prefetches and commit graphs and so on will give you a nice perf boost. Then I keep the default fetch all heads config and don’t mind the noise too much."},{"id":"544279","messageId":"ahnUeESE1x802Z9N@desktop","threadId":"65715","inReplyTo":"20260529163117.z2auhbg4sdxxgmis@archP14s","subject":"Re: Suggetsions for collaboration workflows in large repos","fromName":"Matthew Hughes","fromEmail":"matthewhughes934@gmail.com","sentAt":"2026-05-29T18:06:29Z","receivedAt":"2026-05-29T18:06:33Z","isPatch":false,"body":"On Fri, May 29, 2026 at 05:31:17PM +0100, Matthew Hughes wrote:\n> I thought about doing something like tracking\n> `refs/heads*/some-colleague-branch` from the remote, since with the wildcard\n> `*` I at least won't the fatal error on the missing reference during fetch, but\n> that risks my config containing an ever growing list of such wildcards, or a\n> bunch of manual work occasionally cleaning up old ones (or maybe that could be\n> automated).\n\nI hacked some scripts to automate this. Firstly, one for fetching:\n\n1. Fetches the branch\n2. Adds a fetch config with wildcard hacks so `git fetch` brings in updates for\n  that branch (the refspec should match _exactly_ that branch and never\n  anything more)\n3. Adds a separate ref to record that we're tracking this branch (so something\n  knows to clean it up later)\n\n    #!/usr/bin/env bash\n\n    set -o errexit -o pipefail -o nounset\n\n    # save command as e.g. git-fetch-other\n    CMD_NAME=\"$(basename \"$0\" | sed 's/git-//g')\"\n    if [ $# -lt 1 ]\n    then\n        echo \"usage: git $CMD_NAME branch-name [ remote-name ]\" >&2\n        exit 1\n    fi\n\n    BRANCH_NAME=\"$1\"\n    REMOTE_NAME=\"${2:-origin}\"\n    FETCH_CONFIG_NAME=\"remote.$REMOTE_NAME.fetch\"\n\n    git fetch \"$REMOTE_NAME\" \"$BRANCH_NAME\"\n    git checkout -b \"$BRANCH_NAME\"\n\n    # we want to record that we are tracking this branch, to do this create\n    # a new ref whose name tells us what we're tracking, but whose value is\n    # unimportant. So as a placeholder value just use the hash of an empty tree\n    # taken from https://git.kernel.org/pub/scm/git/git.git/commit/?id=9c8a294a1ae1335511475db9c0eb8841c0ec9738\n    EMPTY_TREE_REF=\"$(git hash-object -t tree /dev/null)\"\n\n    # refspec used to track the branch: we expect branches to be deleted from the\n    # upstream when merged so tracking exactly:\n    # \"+refs/heads/$BRANCH_NAME:refs/remotes/$REMOTE_NAME/$BRANCH_NAME\" will error\n    # when we go to fetch that exact ref after its removed upstream.\n    # so HACK around this: add wildcards that we still expect to only ever match\n    # this exact branch (but doesn't have the issue of git complaining when it\n    # tries to fetch an _exact_ ref)\n    TRACKING_REFSPEC=\"+refs/heads*/$BRANCH_NAME:refs/remotes*/$REMOTE_NAME/$BRANCH_NAME\"\n\n    # record that we're tracking this branch. First check we've not already\n    # recorded this, then ...\n    if ! git config get --local --fixed-value --value \"$TRACKING_REFSPEC\" \"$FETCH_CONFIG_NAME\" >/dev/null\n    then\n        # ... set the config to track it for fetching, and ...\n        git config set --comment \"$CMD_NAME: tracking at $(date -I)\"  --local --append \"$FETCH_CONFIG_NAME\" \"$TRACKING_REFSPEC\"\n        # ... record that we have special cased this tracking\n        git update-ref \"refs/tracked/$REMOTE_NAME/$BRANCH_NAME\" \"$EMPTY_TREE_REF\"\n    fi\n\nAnd the cleanup script (needs to be run periodically):\n\n1. Collects all the remote branches we know about\n2. Checks all the references from step 3. above and checks if any branches\ndefined there are missing remotes (I have fetch.prune=true to keep the remote\ntracking references up-to-date)\n3. If they are, drops the tracking config for that branch\n\n    #!/usr/bin/env bash\n\n    set -o errexit -o pipefail -o nounset\n\n    REMOTE_NAME=\"${1:-origin}\"\n    TRACKED_REF_PREFIX=\"refs/tracked/$REMOTE_NAME\"\n    REMOTE_REF_PREFIX=\"refs/remotes/$REMOTE_NAME\"\n\n    declare -A remote_branch_lookup\n    while read -r remote_ref\n    do\n        # strip prefix, e.g. 'refs/remotes/origin/some-branch' -> 'some-branch'\n        branch_name=\"${remote_ref#$REMOTE_REF_PREFIX/}\"\n        remote_branch_lookup[\"$branch_name\"]=1\n    done < <(git for-each-ref --format='%(refname)' \"$REMOTE_REF_PREFIX/\")\n\n    while read -r tracking_info\n    do\n        tracked_branch=\"${tracking_info#$TRACKED_REF_PREFIX/}\"\n        if ! [[ -v \"remote_branch_lookup[$tracked_branch]\" ]]\n        then\n            echo \"branch $tracked_branch has been removed from the remote, untracking it\"\n            git update-ref -d \"$TRACKED_REF_PREFIX/$tracked_branch\"\n\n            tracking_refspec=\"+refs/heads*/$tracked_branch:refs/remotes*/$REMOTE_NAME/$tracked_branch\"\n            git config unset --local --fixed-value --value \"$tracking_refspec\" \"remote.$REMOTE_NAME.fetch\"\n        fi\n    done < <(git for-each-ref --format='%(refname)' \"$TRACKED_REF_PREFIX/\")\n\nSo functionally I think this allows for the workflow I want, but does feel like\na big ol' hack :>\n\n"},{"id":"544548","messageId":"ah8W5BL714h9r3_c@desktop","threadId":"65715","inReplyTo":"82F556A1-A5C6-414E-8EFB-13F83FA30E44@gmail.com","subject":"Re: Suggetsions for collaboration workflows in large repos","fromName":"Matthew Hughes","fromEmail":"matthewhughes934@gmail.com","sentAt":"2026-06-02T18:35:32Z","receivedAt":"2026-06-02T18:35:36Z","isPatch":false,"body":"On Fri, May 29, 2026 at 01:56:02PM -0400, Ben Knoble wrote:\n> My current advice is to enable git-maintenance on such a repo, where\n> prefetches and commit graphs and so on will give you a nice perf boost. Then\n> I keep the default fetch all heads config and don’t mind the noise too much.\n\nThanks, I do have maintenance activated (I believe `scalar` handled that for\nme) and that does noticeable speed up some operations, and I have find the\nperformance in general for almost all operations to be much better than I\nexpected (having not worked in such a large repo before). My only real issue is\nin fetching, since I really don't want to waste the time pulling down all the\nother branches in the repo that I almost certainly will never need locally.\n"},{"id":"544617","messageId":"87tsrj20yc.fsf@emacs.iotcl.com","threadId":"65715","inReplyTo":"ahnUeESE1x802Z9N@desktop","subject":"Re: Suggetsions for collaboration workflows in large repos","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2026-06-03T13:44:11Z","receivedAt":"2026-06-03T13:44:27Z","isPatch":false,"body":"Matthew Hughes <matthewhughes934@gmail.com> writes:\n\n> On Fri, May 29, 2026 at 05:31:17PM +0100, Matthew Hughes wrote:\n>> I thought about doing something like tracking\n>> `refs/heads*/some-colleague-branch` from the remote, since with the wildcard\n>> `*` I at least won't the fatal error on the missing reference during fetch, but\n>> that risks my config containing an ever growing list of such wildcards, or a\n>> bunch of manual work occasionally cleaning up old ones (or maybe that could be\n>> automated).\n\nI feel your problem, although a lot less in the project I'm working on\nlately. I have these refspecs by the way:\n\n\tfetch = +refs/heads/master:refs/remotes/origin/master\n\tfetch = +refs/heads/toon-*:refs/remotes/origin/toon-*\n\n\n> I hacked some scripts to automate this. Firstly, one for fetching:\n>\n> 1. Fetches the branch\n> 2. Adds a fetch config with wildcard hacks so `git fetch` brings in updates for\n>   that branch (the refspec should match _exactly_ that branch and never\n>   anything more)\n> 3. Adds a separate ref to record that we're tracking this branch (so something\n>   knows to clean it up later)\n>\n>     #!/usr/bin/env bash\n>\n>     set -o errexit -o pipefail -o nounset\n>\n>     # save command as e.g. git-fetch-other\n>     CMD_NAME=\"$(basename \"$0\" | sed 's/git-//g')\"\n>     if [ $# -lt 1 ]\n>     then\n>         echo \"usage: git $CMD_NAME branch-name [ remote-name ]\" >&2\n>         exit 1\n>     fi\n>\n>     BRANCH_NAME=\"$1\"\n>     REMOTE_NAME=\"${2:-origin}\"\n>     FETCH_CONFIG_NAME=\"remote.$REMOTE_NAME.fetch\"\n>\n>     git fetch \"$REMOTE_NAME\" \"$BRANCH_NAME\"\n>     git checkout -b \"$BRANCH_NAME\"\n>\n>     # we want to record that we are tracking this branch, to do this create\n>     # a new ref whose name tells us what we're tracking, but whose value is\n>     # unimportant. So as a placeholder value just use the hash of an empty tree\n>     # taken from https://git.kernel.org/pub/scm/git/git.git/commit/?id=9c8a294a1ae1335511475db9c0eb8841c0ec9738\n>     EMPTY_TREE_REF=\"$(git hash-object -t tree /dev/null)\"\n>\n>     # refspec used to track the branch: we expect branches to be deleted from the\n>     # upstream when merged so tracking exactly:\n>     # \"+refs/heads/$BRANCH_NAME:refs/remotes/$REMOTE_NAME/$BRANCH_NAME\" will error\n>     # when we go to fetch that exact ref after its removed upstream.\n>     # so HACK around this: add wildcards that we still expect to only ever match\n>     # this exact branch (but doesn't have the issue of git complaining when it\n>     # tries to fetch an _exact_ ref)\n>     TRACKING_REFSPEC=\"+refs/heads*/$BRANCH_NAME:refs/remotes*/$REMOTE_NAME/$BRANCH_NAME\"\n>\n>     # record that we're tracking this branch. First check we've not already\n>     # recorded this, then ...\n>     if ! git config get --local --fixed-value --value \"$TRACKING_REFSPEC\" \"$FETCH_CONFIG_NAME\" >/dev/null\n>     then\n>         # ... set the config to track it for fetching, and ...\n>         git config set --comment \"$CMD_NAME: tracking at $(date -I)\"  --local --append \"$FETCH_CONFIG_NAME\" \"$TRACKING_REFSPEC\"\n>         # ... record that we have special cased this tracking\n>         git update-ref \"refs/tracked/$REMOTE_NAME/$BRANCH_NAME\" \"$EMPTY_TREE_REF\"\n>     fi\n\nIt seems to be a bit more advanced than the alias I have:\n\n    cofetch = !sh -c 'git fetch $1 $2:remotes/$1/$2 && git switch -c $2 remotes/$1/$2' -\n\nYou need to pass it the remote and the branch name (in reverse order of\nyours, which makes sense if you want the remote to be optional).\n\n> And the cleanup script (needs to be run periodically):\n>\n> 1. Collects all the remote branches we know about\n> 2. Checks all the references from step 3. above and checks if any branches\n> defined there are missing remotes (I have fetch.prune=true to keep the remote\n> tracking references up-to-date)\n> 3. If they are, drops the tracking config for that branch\n>\n>     #!/usr/bin/env bash\n>\n>     set -o errexit -o pipefail -o nounset\n>\n>     REMOTE_NAME=\"${1:-origin}\"\n>     TRACKED_REF_PREFIX=\"refs/tracked/$REMOTE_NAME\"\n>     REMOTE_REF_PREFIX=\"refs/remotes/$REMOTE_NAME\"\n>\n>     declare -A remote_branch_lookup\n>     while read -r remote_ref\n>     do\n>         # strip prefix, e.g. 'refs/remotes/origin/some-branch' -> 'some-branch'\n>         branch_name=\"${remote_ref#$REMOTE_REF_PREFIX/}\"\n>         remote_branch_lookup[\"$branch_name\"]=1\n>     done < <(git for-each-ref --format='%(refname)' \"$REMOTE_REF_PREFIX/\")\n>\n>     while read -r tracking_info\n>     do\n>         tracked_branch=\"${tracking_info#$TRACKED_REF_PREFIX/}\"\n>         if ! [[ -v \"remote_branch_lookup[$tracked_branch]\" ]]\n>         then\n>             echo \"branch $tracked_branch has been removed from the remote, untracking it\"\n>             git update-ref -d \"$TRACKED_REF_PREFIX/$tracked_branch\"\n>\n>             tracking_refspec=\"+refs/heads*/$tracked_branch:refs/remotes*/$REMOTE_NAME/$tracked_branch\"\n>             git config unset --local --fixed-value --value \"$tracking_refspec\" \"remote.$REMOTE_NAME.fetch\"\n>         fi\n>     done < <(git for-each-ref --format='%(refname)' \"$TRACKED_REF_PREFIX/\")\n>\n> So functionally I think this allows for the workflow I want, but does feel like\n> a big ol' hack :>\n\nI agree it feels hacky, but I don't really see how we can generalize it\nmore so it will become a standard feature in git?\n\nI was thinking you can already pass `-c remote.origin.fetch=<refspec>`\n(multiple times) to git-clone(1), but in practice it doesn't seem to\nwork because that config is additive, so it adds the refspec, instead of\noverwriting, so you're getting:\n\n  fatal: multiple updates for ref 'refs/remotes/origin/main' not allowed\n\nAnd you cannot combine it with `--single-branch`, although you could do\na single branch clone and then add additional refspecs later.\n\n-- \nCheers,\nToon\n"}]}