{"thread":{"id":"59915","subject":"Bug: fetch with deepen shortens history","startedAt":"2023-06-26T21:24:19Z","lastAt":"2023-07-31T10:58:40Z","messageCount":3,"participants":["Benjamin Stein","Bagas Sanjaya"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"478860","messageId":"PH8P223MB06996E110EF25C386AA314E19F26A@PH8P223MB0699.NAMP223.PROD.OUTLOOK.COM","threadId":"59915","inReplyTo":null,"subject":"Bug: fetch with deepen shortens history","fromName":"Benjamin Stein","fromEmail":"benji_stein@hotmail.com","sentAt":"2023-06-26T21:24:13Z","receivedAt":"2023-06-26T21:24:19Z","isPatch":false,"sender":{"key":"benji_stein@hotmail.com","avatar":null},"body":"Hello git gurus,\n\nHere's an atypical bug report for you. I'm sorry for not starting with the template, but the context/setup are longer than felt useful in that format.\n\nI have what I believe to be a (relatively) simple, reproducible test case (repo setup/steps below) around shallow checkouts at merge commits and deepening where the behavior is quite surprising - I end up with a smaller history after a fetch operation than when I started!\n\nI've tried this on multiple OSes (Linux, Mac/Darwin), shells (zsh, bash) and various git versions ranging from 2.40.1 to 2.34.1, but I suspect it's older than that.\n\nScenario:\nI'm using GitHub actions to look through some commit history and generate a report of commits relative to another branch. The specifics aren't super important, just that I start off in a shallow repo (depth=1) because (1) that's what GHA drops me into by default, and (2) since this is a large mono-repo I don't want to fetch all history every time - so I want to minimize the amount of data fetched.\nSo I used `fetch --shallow-exclude=other-branch HEAD` to get the relevant commits, and ran into my bug: when I do an extra `fetch --deepen` I end up with only a single commit in history instead of the N I had right before the call. If that's not super clear, I think the reproduction steps below should help.\n\nEven more confusingly, if I check out a merge commit into trunk (the default) it misbehaves, but if I instead start out on the branch I'm inspecting, the same sequence of commands works correctly! This detail threw me for a long time as I tried to understand why behavior after a merge wasn't consistent.\n\nMy hunch is that this has to do with .git/shallow and the way some of the SHAs are coalescing when histories intersect/are combined, but I'm not too familiar with the inner workings of shallow. I even tried running the same sequence of steps with various combinations of --update-shallow on the fetches, but that doesn't seem to address the underlying issue of history shortening.\n\nI hope that's all clear. If I can give more detail, definitely let me know, and I'm happy to try and explore some solutions, but I'm not certain where to begin.\nI'm also not sure if it's an issue on the client or server - since I was initially testing with GitHub - but in my reduced example everything is local.\n\nThanks for your time and help with this!\n\n-Benji\n\n\n------ requisite bugreport answers ------\n\nThank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\nSee setup below.\nIn my shallow checkout, I ran:\n    git log --oneline | wc -l => N commits of history.\nThen I ran\n    git fetch --deepen=1 <branch>\nFollowed by\n    git log --oneline | wc -l => now just one commit.\n\nWhat did you expect to happen? (Expected behavior)\nI expected to see N+1 commits of history because I deepened by 1.\n\nWhat happened instead? (Actual behavior)\nI instead had just one commit of history.\n\nWhat's different between what you expected and what actually happened?\nI expected not to have my local history shortened by ~N commits despite using --deepen instead of --depth.\n\nAnything else you want to add:\nSee additional notes/steps in this email and attached script to reproduce\n\n\n[System Info]\ngit version:\ngit version 2.40.1\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Linux 6.2.6-76060206-generic #202303130630~1681329778~22.04~d824cd4 SMP PREEMPT_DYNAMIC Wed A x86_64\ncompiler info: gnuc: 11.3\nlibc info: glibc: 2.35\n$SHELL (typically, interactive shell): /bin/zsh\n\n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show\n(it's irrelevant, my example starts from nothing, so no hooks)\n\n\n------- bug-setup.sh ----- (better to to run in chunks, rather than all at once, but provided for convenience)\n\nset -x\n# Setup working folder for easy cleanup\nmkdir git-test && cd git-test\n\n# Setup sample repo\nmkdir source-repo && cd source-repo\ngit init\ngit branch -m trunk\nfor i in {01..05}; do echo \"start${i}\" >> start; git add start; git commit -m \"start${i}\"; done\ngit branch old-checkpoint\nfor i in {01..10}; do echo \"new${i}\" >> new; git add new; git commit -m \"new${i}\"; done\ngit checkout -b feature HEAD~2\nfor i in {01..03}; do echo \"feature${i}\" >> feature; git add feature; git commit -m \"feature${i}\"; done\ngit checkout trunk\ngit merge --no-edit feature\ncd ..\nsleep 1\n\n\n# Checkout shallow clone at feature branch - this works as desired\ngit clone --no-local source-repo --depth=1 --branch feature shallow-clone-feature\ncd shallow-clone-feature\ngit remote set-branches --add origin '*'\ngit fetch origin --shallow-exclude=old-checkpoint feature\ngit log --oneline origin/feature | wc -l # 11, expected\ngit fetch --deepen=1 origin feature\ngit log --oneline origin/feature | wc -l # 12, also as expected\ncd ..\nsleep 1\n\n\n# Checkout shallow clone at merge commit - this illustrates the bug\ngit clone --no-local source-repo --depth=1 --branch trunk shallow-clone-merge\ncd shallow-clone-merge\ngit remote set-branches --add origin '*'\ngit fetch origin --shallow-exclude=old-checkpoint feature\ngit log --oneline origin/feature | wc -l # 11, expected\ngit fetch --deepen=1 origin feature\ngit log --oneline origin/feature | wc -l # 1, unexpected\n\n# Wait, what? Let's try that again\ngit fetch origin --shallow-exclude=old-checkpoint feature\ngit log --oneline origin/feature | wc -l # 11, still as expected\ngit fetch --deepen=1 origin feature\ngit log --oneline origin/feature | wc -l # 13, different this time. Unexpected.\ncd ..\nsleep 1\n\n\n# What if we expand depth first?\ngit clone --no-local source-repo --depth=1 --branch trunk shallow-clone-with-depth\ncd shallow-clone-with-depth\ngit remote set-branches --add origin '*'\ngit fetch --depth=2 origin feature\ngit fetch origin --shallow-exclude=old-checkpoint feature\ngit log --oneline origin/feature | wc -l # 11, expected\ngit fetch --deepen=1 origin feature\ngit log --oneline origin/feature | wc -l # 1, still unexpected\ncd ..\nsleep 1\n\n\n# It turns out the depth query sometimes works if I also manually include HEAD, but still strangely.\n# If I use depth=2, it's fine, but if I keep depth=3 it's not.\ngit clone --no-local source-repo --depth=1 --branch trunk shallow-clone-with-depth\ncd shallow-clone-with-depth\ngit remote set-branches --add origin '*'\ngit fetch --depth=3 origin HEAD feature # it works if I use or include HEAD, but *not* if I include old-checkpoint\ngit fetch origin --shallow-exclude=old-checkpoint feature\ngit log --oneline origin/feature | wc -l # 11, expected\ngit fetch --deepen=1 origin feature\ngit log --oneline origin/feature | wc -l # 4, different from before, still wrong\ncd ..\nsleep 1\n\n# If we start with deepen, instead\ngit clone --no-local ./source-repo --depth=1 --branch trunk shallow-clone-deepen\ncd shallow-clone-deepen\ngit remote set-branches --add origin '*'\ngit fetch --deepen=1 origin HEAD feature # this also works if we use feature\ngit fetch origin --shallow-exclude=old-checkpoint feature\ngit log --oneline origin/feature | wc -l # 11, expected\ngit fetch --deepen=1 origin feature\ngit log --oneline origin/feature | wc -l # 12, that's finally correct\ncd ..\nsleep 1\n\n# manually clear everything out by running\n# cd .. && rm -rf git-test"},{"id":"479835","messageId":"PH8P223MB0699B4956F3FE9DC736419FA9F03A@PH8P223MB0699.NAMP223.PROD.OUTLOOK.COM","threadId":"59915","inReplyTo":"PH8P223MB06996E110EF25C386AA314E19F26A@PH8P223MB0699.NAMP223.PROD.OUTLOOK.COM","subject":"Re: Bug: fetch with deepen shortens history","fromName":"Benjamin Stein","fromEmail":"benji_stein@hotmail.com","sentAt":"2023-07-25T00:03:31Z","receivedAt":"2023-07-25T00:03:36Z","isPatch":false,"sender":{"key":"benji_stein@hotmail.com","avatar":null},"body":"> Hello git gurus,\n\n> Here's an atypical bug report for you. I'm sorry for not starting with the template, but the context/setup are longer \n> than felt useful in that format.\n> \n> I have what I believe to be a (relatively) simple, reproducible test case (repo setup/steps below) around shallow \n> checkouts at merge commits and deepening where the behavior is quite surprising - I end up with a smaller history \n> after a fetch operation than when I started!\n>\n>  [...snip...]\n\nHello again. It's been a month, and I didn't even get a \"yes, we tested this and confirm the problem\", so I thought I'd check in on this.\n\nI also found a commit setup where even my \"working\" solution steps (only using deepen) still ends up with the unexpected behavior, so I thought I'd add that in here as a simpler scenario to experiment with. It happens when both sides of the merge are the same number of commits to the merge base.\n\nLet me know if there's any additional information I can provide or something I can do to help resolve.\n\nThanks,\nBenji\n\n---------simple-bug-setup.sh---------\nset -x\n# Setup working folder for easy cleanup\nmkdir git-test && cd git-test\n\n# Setup repo\nmkdir source-repo &&  cd source-repo\ngit init\ngit branch -m trunk\nfor i in {01..05}; do echo \"start${i}\" >> start; git add start; git commit -m \"start${i}\"; done\ngit branch old-checkpoint\nfor i in {01..10}; do echo \"new${i}\" >> new; git add new; git commit -m \"new${i}\"; done\ngit checkout -b feature HEAD~4\nfor i in {01..03}; do echo \"feature${i}\" >> feature; git add feature; git commit -m \"feature${i}\"; done\ngit checkout trunk\ngit merge --no-edit feature\ncd ..\nsleep 1\n\n# simple checkout\ngit clone --no-local source-repo --depth=1 --branch trunk shallow-clone-only-deepen\ncd shallow-clone-only-deepen\ngit remote set-branches --add origin '*'\ngit fetch --deepen=4 origin HEAD feature # this also works if we use feature\ngit fetch origin --shallow-exclude=old-checkpoint feature\ngit log --oneline origin/feature | wc -l # 9, expected\ngit fetch --deepen=1 origin feature\ngit log --oneline origin/feature | wc -l # 4, unexpected\ncd ..\nsleep 1\n\n"},{"id":"479989","messageId":"ZMeTR1rj9RH27CUL@debian.me","threadId":"59915","inReplyTo":"PH8P223MB0699B4956F3FE9DC736419FA9F03A@PH8P223MB0699.NAMP223.PROD.OUTLOOK.COM","subject":"Re: Bug: fetch with deepen shortens history","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2023-07-31T10:56:07Z","receivedAt":"2023-07-31T10:58:40Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On Tue, Jul 25, 2023 at 12:03:31AM +0000, Benjamin Stein wrote:\n> > Hello git gurus,\n> \n> > Here's an atypical bug report for you. I'm sorry for not starting with the template, but the context/setup are longer \n> > than felt useful in that format.\n> > \n> > I have what I believe to be a (relatively) simple, reproducible test case (repo setup/steps below) around shallow \n> > checkouts at merge commits and deepening where the behavior is quite surprising - I end up with a smaller history \n> > after a fetch operation than when I started!\n> >\n> >  [...snip...]\n> \n> Hello again. It's been a month, and I didn't even get a \"yes, we tested this and confirm the problem\", so I thought I'd check in on this.\n> \n> I also found a commit setup where even my \"working\" solution steps (only using deepen) still ends up with the unexpected behavior, so I thought I'd add that in here as a simpler scenario to experiment with. It happens when both sides of the merge are the same number of commits to the merge base.\n> \n> Let me know if there's any additional information I can provide or something I can do to help resolve.\n\n(also Cc: Junio and Taylor.)\n\nI think in most cases, people doing shallow clones for testing mainline\nbranch of whatever project they're hacking. If at some later date they\nwish to deepen their clones, they can do that up to specified date:\n\n```\n$ git clone <clone url> project && cd project\n(hack, hack, hack)\n$ git checkout main\n$ git fetch --shallow-since=2023-01-01 && git repack -A -d\n```\n\nI often use this method when cloning quite large repos when I'm on mobile\ndata (via WiFi tethering) due to bandwidth limitation.\n\n> \n> Thanks,\n> Benji\n> \n> ---------simple-bug-setup.sh---------\n> set -x\n> # Setup working folder for easy cleanup\n> mkdir git-test && cd git-test\n> \n> # Setup repo\n> mkdir source-repo &&  cd source-repo\n> git init\n> git branch -m trunk\n> for i in {01..05}; do echo \"start${i}\" >> start; git add start; git commit -m \"start${i}\"; done\n> git branch old-checkpoint\n> for i in {01..10}; do echo \"new${i}\" >> new; git add new; git commit -m \"new${i}\"; done\n> git checkout -b feature HEAD~4\n> for i in {01..03}; do echo \"feature${i}\" >> feature; git add feature; git commit -m \"feature${i}\"; done\n> git checkout trunk\n> git merge --no-edit feature\n> cd ..\n> sleep 1\n> \n> # simple checkout\n> git clone --no-local source-repo --depth=1 --branch trunk shallow-clone-only-deepen\n> cd shallow-clone-only-deepen\n> git remote set-branches --add origin '*'\n> git fetch --deepen=4 origin HEAD feature # this also works if we use feature\n> git fetch origin --shallow-exclude=old-checkpoint feature\n> git log --oneline origin/feature | wc -l # 9, expected\n> git fetch --deepen=1 origin feature\n> git log --oneline origin/feature | wc -l # 4, unexpected\n> cd ..\n> sleep 1\n> \n\nThanks for the script that you provided.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"}]}