threads / discuss / 51532

Possible to unshallow or deepen based on local objects?

Subject: Possible to unshallow or deepen based on local objects?

## tl;dr

3 messages between Jul 26, 2019 and Aug 8, 2019.

replies: 2people: 2as markdown or json

Mark Rushakoff· Jul 26, 2019, 18:43 UTC · lore

I am building some CI tooling, and I am working with a large-ish repository, so I am trying to start with a shallow clone of the repository and deepen it on demand. I am finding it very difficult to correctly switch between fetch --depth and fetch --deepen.

I am looking for a way to "recover" from a fetch --depth call that made my history shallower than before. Ultimately, I may take a different approach to just work around this altogether, but I thought I'd ask here first.

In particular, if there is history like this:
A --- B --- C --- D --- E --- F
       \                 \
         ---------------- G
And my shallow history includes D..F, then I believe I am limited in my options
for fetching G:
- If I do a plain fetch of G, or if I use --deepen, then I pull in A and all of
  its history, which I don't strictly need or want yet.
- If I know G's parent is B, and I fetch --depth=1, then (I think) I will see
  B, E, F, and G, effectively losing sight of D
As a more contrived example, suppose I have a shallow clone of a repository:

bash-3.2$ git clone --depth=4 -q git@github.com:git/git.git bash-3.2$ cd git bash-3.2$ git log --format='%h %d' 3034dab (HEAD -> master, origin/master, origin/HEAD) 98e06de 352253a 4098130 (grafted) 7b974e3 df63c2e 75ce486 (grafted) 70b39fb (grafted) afc3bf6 (grafted) bash-3.2$ cat .git/shallow 409813088ad55ae4a60f55412f6b5ba6a89d89e7 70b39fbede78313656e8a6bd9b38b238ab10db2f 75ce48674889df6a2bb493fb5d6bef0ef60ca7ae afc3bf6eb13d9fc489b569164819cff44db8ac17

And then suppose I fetch with depth smaller than before:

bash-3.2$ git fetch --depth=1 -q bash-3.2$ git log --format='%h %d' 3034dab (grafted, HEAD -> master, origin/master, origin/HEAD) bash-3.2$ cat .git/shallow 3034dab9ed6b11970a53099a7b3ca981f1461365 409813088ad55ae4a60f55412f6b5ba6a89d89e7 70b39fbede78313656e8a6bd9b38b238ab10db2f 75ce48674889df6a2bb493fb5d6bef0ef60ca7ae afc3bf6eb13d9fc489b569164819cff44db8ac17

I can still see the commits I had before the --depth=1 fetch:

bash-3.2$ git log -1 --format=oneline 75ce486 75ce48674889df6a2bb493fb5d6bef0ef60ca7ae (grafted) Merge branch 'di/readme-markup-fix'

Short of looking through .git/shallow and removing each entry whose parents I can resolve, is there a built-in command to "unshallow based on local objects"? I have tried many web search terms, I have looked through the docs for many low-level git commands, I have tried variations of "git fetch .", and I am stumped.

Mark Rushakoff· Aug 8, 2019, 22:17 UTC · re: Mark Rushakoff · lore

Re: Possible to unshallow or deepen based on local objects?

On Fri, Jul 26, 2019 at 11:43 AM Mark Rushakoff <mark@influxdata.com> wrote:
Show 66 quoted lines
>
> I am building some CI tooling, and I am working with a large-ish repository, so
> I am trying to start with a shallow clone of the repository and deepen it on
> demand. I am finding it very difficult to correctly switch between fetch
> --depth and fetch --deepen.
>
> I am looking for a way to "recover" from a fetch --depth call that made my
> history shallower than before. Ultimately, I may take a different approach to
> just work around this altogether, but I thought I'd ask here first.
>
> In particular, if there is history like this:
>
> A --- B --- C --- D --- E --- F
>        \                 \
>          ---------------- G
> And my shallow history includes D..F, then I believe I am limited in my options
> for fetching G:
>
> - If I do a plain fetch of G, or if I use --deepen, then I pull in A and all of
>   its history, which I don't strictly need or want yet.
> - If I know G's parent is B, and I fetch --depth=1, then (I think) I will see
>   B, E, F, and G, effectively losing sight of D
>
> As a more contrived example, suppose I have a shallow clone of a repository:
>
> bash-3.2$ git clone --depth=4 -q git@github.com:git/git.git
> bash-3.2$ cd git
> bash-3.2$ git log --format='%h %d'
> 3034dab  (HEAD -> master, origin/master, origin/HEAD)
> 98e06de
> 352253a
> 4098130  (grafted)
> 7b974e3
> df63c2e
> 75ce486  (grafted)
> 70b39fb  (grafted)
> afc3bf6  (grafted)
> bash-3.2$ cat .git/shallow
> 409813088ad55ae4a60f55412f6b5ba6a89d89e7
> 70b39fbede78313656e8a6bd9b38b238ab10db2f
> 75ce48674889df6a2bb493fb5d6bef0ef60ca7ae
> afc3bf6eb13d9fc489b569164819cff44db8ac17
>
> And then suppose I fetch with depth smaller than before:
>
> bash-3.2$ git fetch --depth=1 -q
> bash-3.2$ git log --format='%h %d'
> 3034dab  (grafted, HEAD -> master, origin/master, origin/HEAD)
> bash-3.2$ cat .git/shallow
> 3034dab9ed6b11970a53099a7b3ca981f1461365
> 409813088ad55ae4a60f55412f6b5ba6a89d89e7
> 70b39fbede78313656e8a6bd9b38b238ab10db2f
> 75ce48674889df6a2bb493fb5d6bef0ef60ca7ae
> afc3bf6eb13d9fc489b569164819cff44db8ac17
>
> I can still see the commits I had before the --depth=1 fetch:
>
> bash-3.2$ git log -1 --format=oneline 75ce486
> 75ce48674889df6a2bb493fb5d6bef0ef60ca7ae (grafted) Merge branch
> 'di/readme-markup-fix'
>
> Short of looking through .git/shallow and removing each entry whose parents I
> can resolve, is there a built-in command to "unshallow based on local objects"?
> I have tried many web search terms, I have looked through the docs for many
> low-level git commands, I have tried variations of "git fetch .", and I am
> stumped.
I've run into another flavor of this problem now.

I have a shallow clone of a repository. Then sometime later, I obtain a bundle containing full history.

I thought I could use git fetch --unshallow /path/to/repo.bundle mybranch:mybranch, but I've found that git does not support shallow fetches from a bundle and that unshallow is actually an alias for --depth=2147483647 in 4dcb167fc3 (fetch: add --unshallow for turning shallow repo into complete one, 2013-01-11).

If I do a plain fetch from the bundle, I can manually modify .git/shallow to "unshallow" my repository like the previous email in this thread. But what I think I still want here is a way to tell git "here are all the objects I have; update the shallow boundaries accordingly."

Junio C Hamano· Aug 8, 2019, 22:33 UTC · re: Mark Rushakoff · lore

Re: Possible to unshallow or deepen based on local objects?

Mark Rushakoff <mark@influxdata.com> writes:
Show 13 quoted lines
> I have a shallow clone of a repository. Then sometime later, I obtain a bundle
> containing full history.
>
> I thought I could use git fetch --unshallow /path/to/repo.bundle
> mybranch:mybranch, but I've found that git does not support shallow fetches
> from a bundle and that unshallow is actually an alias for --depth=2147483647 in
> 4dcb167fc3 (fetch: add --unshallow for turning shallow repo into complete one,
> 2013-01-11).
>
> If I do a plain fetch from the bundle, I can manually modify .git/shallow to
> "unshallow" my repository like the previous email in this thread. But what I
> think I still want here is a way to tell git "here are all the objects I have;
> update the shallow boundaries accordingly."
It sounds like a quite sensible request to me.  

Offhand, I do not think of anything that would make it fundamentally impossible to allow us to first add objects to the .git/objects directory by whatever means available to you (including, but not limited to, exploding the object data from a full bundle) and then to tell Git to see if there (still) are missing commits and objects that ought to exist but don't that necessitates the repository to be marked "shallow", and adjust the boundary.

I however do not think there is such a code already written X-<.

← back to recent threads