{"thread":{"id":"62596","subject":"Some feedback on 'git clone create'","startedAt":"2024-12-04T05:28:42Z","lastAt":"2024-12-05T18:45:27Z","messageCount":6,"participants":["Sainan","Justin Tobler","Patrick Steinhardt","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"508601","messageId":"AfLIcOv4X1AxLOaODNS89HA-bMeI7bj0xUGsLD-6xmVAS_a_2xOzy2uX-wXxpNA7kCpKYudELCEKv73roW_-HTd83Fcz3FZ_yJQOswQHW48=@calamity.inc","threadId":"62596","inReplyTo":null,"subject":"Some feedback on 'git clone create'","fromName":"Sainan","fromEmail":"sainan@calamity.inc","sentAt":"2024-12-04T05:28:27Z","receivedAt":"2024-12-04T05:28:42Z","isPatch":false,"sender":{"key":"sainan@calamity.inc","avatar":null},"body":"Hi, I hope this email finds you well.\n\nI think Git bundles/packfiles are an exceptional compression format, but I find there are some rough edges with the tool to create them:\n\n1. There is no way to specify that you want a shallow bundle, instead you are only able to a) pack the entire tree at a given head or b) pack new/updated objects in a specified range. Anecdotally, this could store data in ~67% of the size of an equivalent .zip file.\n\n2. It seems that when specifiying a commit hash, it raises an error:\n$ git bundle create repo.bundle $(git rev-list HEAD | head -n 1)\n> fatal: Refusing to create empty bundle.\nThis confuses me slightly because I thought a commit hash should also be a valid head _pointer_. 'git rev-list' also seems to agree with me on this.\n\n-- Sainan\n"},{"id":"508632","messageId":"pgz37cgfssouykfqxvmjt6pc2lycta6tmyjkovsceqwibww4fw@424usnt3dlh2","threadId":"62596","inReplyTo":"AfLIcOv4X1AxLOaODNS89HA-bMeI7bj0xUGsLD-6xmVAS_a_2xOzy2uX-wXxpNA7kCpKYudELCEKv73roW_-HTd83Fcz3FZ_yJQOswQHW48=@calamity.inc","subject":"Re: Some feedback on 'git clone create'","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-12-04T23:19:53Z","receivedAt":"2024-12-04T23:21:59Z","isPatch":false,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/12/04 05:28AM, Sainan wrote:\n> Hi, I hope this email finds you well.\n> \n> I think Git bundles/packfiles are an exceptional compression format, but I find there are some rough edges with the tool to create them:\n> \n> 1. There is no way to specify that you want a shallow bundle, instead you are only able to a) pack the entire tree at a given head or b) pack new/updated objects in a specified range. Anecdotally, this could store data in ~67% of the size of an equivalent .zip file.\n\nYou can create an incremental bundle covering a specified range.\nSomething like the following example might help you achieve what you are\nlooking for:\n\n  $ git bundle create inc.bundle main~10..main\n\n> \n> 2. It seems that when specifiying a commit hash, it raises an error:\n> $ git bundle create repo.bundle $(git rev-list HEAD | head -n 1)\n> > fatal: Refusing to create empty bundle.\n> This confuses me slightly because I thought a commit hash should also be a valid head _pointer_. 'git rev-list' also seems to agree with me on this.\n\nA bundle is essentially a pack file with a header indicating the\nreferences contained within the bundle. If no reference is provided, the\nbundle is considered empty and git refuses to create it. I think this\nmakes sense in the context of unbundling as you probably would not want\nto add new objects without updating references in the target repository.\n\nFrom the git-bundle(1) docs for \"create\", the usage does say it accepts\n<git-rev-list-args> which may be a bit misleading because\ngit-rev-list(1) does consider the commit hash as valid. Maybe that\nshould be updated to indicate that proper references are expected.\n\n-Justin\n"},{"id":"508640","messageId":"oEOA5yj-xh-iOJdB-TSYMB2RpjLqNmu-jqrQekiII-mTEYSXfC2Fqz71iRyriWaUiED-r4KyDOypwu47cu0v0ZGxoVHBexKkMMMFalfP7IQ=@calamity.inc","threadId":"62596","inReplyTo":"pgz37cgfssouykfqxvmjt6pc2lycta6tmyjkovsceqwibww4fw@424usnt3dlh2","subject":"Re: Some feedback on 'git clone create'","fromName":"Sainan","fromEmail":"sainan@calamity.inc","sentAt":"2024-12-05T07:03:03Z","receivedAt":"2024-12-05T07:03:20Z","isPatch":false,"sender":{"key":"sainan@calamity.inc","avatar":null},"body":"> You can create an incremental bundle covering a specified range.\n\nYeah, I know that you can bundle a delta, but I mean an analogue to a shallow clone, which is kind of like traversing the tree in a different direction.\n\n(Sorry for double-send, forgot to CC git@vger.kernel.org on the reply.)\n\n-- Sainan\n"},{"id":"508644","messageId":"Z1Fma7OLPOkxRncd@pks.im","threadId":"62596","inReplyTo":"pgz37cgfssouykfqxvmjt6pc2lycta6tmyjkovsceqwibww4fw@424usnt3dlh2","subject":"Re: Some feedback on 'git clone create'","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-05T08:38:03Z","receivedAt":"2024-12-05T08:38:20Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Dec 04, 2024 at 05:19:53PM -0600, Justin Tobler wrote:\n> On 24/12/04 05:28AM, Sainan wrote:\n> > Hi, I hope this email finds you well.\n> > \n> > I think Git bundles/packfiles are an exceptional compression format, but I find there are some rough edges with the tool to create them:\n> > \n> > 1. There is no way to specify that you want a shallow bundle, instead you are only able to a) pack the entire tree at a given head or b) pack new/updated objects in a specified range. Anecdotally, this could store data in ~67% of the size of an equivalent .zip file.\n> \n> You can create an incremental bundle covering a specified range.\n> Something like the following example might help you achieve what you are\n> looking for:\n> \n>   $ git bundle create inc.bundle main~10..main\n\nYup. The thing that might not be immediately obvious is that\ngit-bundle(1) accepts git-rev-list(1) arguments, so you can influence\nwhat is and isn't included via that. You can for example even generate\npartial bundles without blobs:\n\n    $ git bundle create partial.bundle main~10..main \\\n        --filter=blob:none\n\nWhat you can do with the resulting bundle might be a different story.\n\n> > 2. It seems that when specifiying a commit hash, it raises an error:\n> > $ git bundle create repo.bundle $(git rev-list HEAD | head -n 1)\n> > > fatal: Refusing to create empty bundle.\n> > This confuses me slightly because I thought a commit hash should also be a valid head _pointer_. 'git rev-list' also seems to agree with me on this.\n> \n> A bundle is essentially a pack file with a header indicating the\n> references contained within the bundle. If no reference is provided, the\n> bundle is considered empty and git refuses to create it. I think this\n> makes sense in the context of unbundling as you probably would not want\n> to add new objects without updating references in the target repository.\n> \n> From the git-bundle(1) docs for \"create\", the usage does say it accepts\n> <git-rev-list-args> which may be a bit misleading because\n> git-rev-list(1) does consider the commit hash as valid. Maybe that\n> should be updated to indicate that proper references are expected.\n\nThat's somewhat weird indeed. I don't see a strong reason why the first\nof the following commands works while the second one doesn't:\n\n    $ git bundle create inc.bundle master~..master\n    $ git bundle create inc.bundle $(git rev-parse master~)..$(git rev-parse master)\n\nIt's not like the bundle has \"master\" in its header in the first command\nanyway, it only lists HEAD in there. So I'd claim that we could do the\nsame for the second command, as well.\n\nPatrick\n"},{"id":"508671","messageId":"6okfwtbgcve77xwjdqjnvsa4yigo5tymlsrypjgztbrmabjtrz@o7dfhaltih35","threadId":"62596","inReplyTo":"Z1Fma7OLPOkxRncd@pks.im","subject":"Re: Some feedback on 'git clone create'","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-12-05T15:27:15Z","receivedAt":"2024-12-05T15:29:21Z","isPatch":false,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/12/05 09:38AM, Patrick Steinhardt wrote:\n> On Wed, Dec 04, 2024 at 05:19:53PM -0600, Justin Tobler wrote:\n> > On 24/12/04 05:28AM, Sainan wrote:\n> > > Hi, I hope this email finds you well.\n> > > \n> > > I think Git bundles/packfiles are an exceptional compression format, but I find there are some rough edges with the tool to create them:\n> > > \n> > > 1. There is no way to specify that you want a shallow bundle, instead you are only able to a) pack the entire tree at a given head or b) pack new/updated objects in a specified range. Anecdotally, this could store data in ~67% of the size of an equivalent .zip file.\n> > \n> > You can create an incremental bundle covering a specified range.\n> > Something like the following example might help you achieve what you are\n> > looking for:\n> > \n> >   $ git bundle create inc.bundle main~10..main\n> \n> Yup. The thing that might not be immediately obvious is that\n> git-bundle(1) accepts git-rev-list(1) arguments, so you can influence\n> what is and isn't included via that. You can for example even generate\n> partial bundles without blobs:\n> \n>     $ git bundle create partial.bundle main~10..main \\\n>         --filter=blob:none\n> \n> What you can do with the resulting bundle might be a different story.\n\nWhen trying to unbundle an incremental bundle into a repository that\nlacks the prerequisite objects, Git fails. These prerequisite objects\nare also listed in the bundle header. Maybe it would be nice if we were\nable to create a shallow repository from this bundle.\n\n> \n> > > 2. It seems that when specifiying a commit hash, it raises an error:\n> > > $ git bundle create repo.bundle $(git rev-list HEAD | head -n 1)\n> > > > fatal: Refusing to create empty bundle.\n> > > This confuses me slightly because I thought a commit hash should also be a valid head _pointer_. 'git rev-list' also seems to agree with me on this.\n> > \n> > A bundle is essentially a pack file with a header indicating the\n> > references contained within the bundle. If no reference is provided, the\n> > bundle is considered empty and git refuses to create it. I think this\n> > makes sense in the context of unbundling as you probably would not want\n> > to add new objects without updating references in the target repository.\n> > \n> > From the git-bundle(1) docs for \"create\", the usage does say it accepts\n> > <git-rev-list-args> which may be a bit misleading because\n> > git-rev-list(1) does consider the commit hash as valid. Maybe that\n> > should be updated to indicate that proper references are expected.\n> \n> That's somewhat weird indeed. I don't see a strong reason why the first\n> of the following commands works while the second one doesn't:\n> \n>     $ git bundle create inc.bundle master~..master\n>     $ git bundle create inc.bundle $(git rev-parse master~)..$(git rev-parse master)\n> \n> It's not like the bundle has \"master\" in its header in the first command\n> anyway, it only lists HEAD in there. So I'd claim that we could do the\n> same for the second command, as well.\n\nI'm not quite sure I follow. According to gitformat-bundle(5), we should\nsee \"obj-id SP refname LF\" in the header. Inspecting the header of a\nbundle created from `git bundle create inc.bundle master~..master` also\nshows \"refs/heads/master\" in the header.\n\nLooking at git-bundle(1) docs more closely it also does mention this\nlimitation:\n\n       A revision name or a range whose right-hand-side cannot be\n       resolved to a reference is not accepted:\n\n           $ git bundle create HEAD.bundle $(git rev-parse HEAD)\n           fatal: Refusing to create empty bundle.\n           $ git bundle create master-yesterday.bundle master~10..master~5\n           fatal: Refusing to create empty bundle.\n\nIt looks like when creating a bundle in `bundle.c:create_bundle()`, if\nthe call to `write_bundle_refs()` returns a reference count of 0, git\ndies with the error seen. When a commit hash is used for the\nrev-list-arg, it is not able to determine a reference from it.\n\n-Justin\n"},{"id":"508675","messageId":"xmqqttbi0y0r.fsf@gitster.g","threadId":"62596","inReplyTo":"6okfwtbgcve77xwjdqjnvsa4yigo5tymlsrypjgztbrmabjtrz@o7dfhaltih35","subject":"Re: Some feedback on 'git clone create'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-05T18:45:24Z","receivedAt":"2024-12-05T18:45:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler <jltobler@gmail.com> writes:\n\n> When trying to unbundle an incremental bundle into a repository that\n> lacks the prerequisite objects, Git fails. These prerequisite objects\n> are also listed in the bundle header. Maybe it would be nice if we were\n> able to create a shallow repository from this bundle.\n\nThe prerequisite objects are required for two reasons when unpacking\na bundle.  One is to ensure that the resulting history is complete,\nas the central idea of bundle was to \"freeze the over-the-wire data\ntransferred during a fetch\" but it predates \"shallow clone\", which\nallows a clone to be lacking history beyond certain point in the\ntopology.  The other is that the pack data stream recorded in a\nbundle is allowed to be a \"thin pack\", with objects represented as a\ndelta against other objects that do not exist in the same pack---the\nrecipient is expected to supply these delta base objects to ensure\nthat these deltified objects can be reconstituted, which is where\nthe \"prerequisite\" comes from.\n\nIn order to allow creating a shallow clone out of a bundle when\nusing revision exclusions (see \"git bundle --help\"), we'd update the\nbundle format to allow us to create a bundle using \"thick pack\",\nwhich we do not currently do.  And more importantly, the recipient\nmust be able to identify such a \"thick pack\", as the data currently\ndefined, a bundle with \"prerequisite\" by definition is \"thin\" and\ncannot be used to create a shallow clone out of.  Which means we'd\nneed an updated bundle file format.\n\n> I'm not quite sure I follow. According to gitformat-bundle(5), we should\n> see \"obj-id SP refname LF\" in the header. Inspecting the header of a\n> bundle created from `git bundle create inc.bundle master~..master` also\n> shows \"refs/heads/master\" in the header.\n\nYes, if there is one thing I regret in the way the bundle header was\ndesigned and wish I could fix is to that we lack \"HEAD\" unless we\nforce to include it, and \"git clone\" out of a bundle sometimes fails\nbecause of it (you can \"git fetch\" out of the bundle, naming the\nrefs you find in the bundle header instaed).\n\n> It looks like when creating a bundle in `bundle.c:create_bundle()`, if\n> the call to `write_bundle_refs()` returns a reference count of 0, git\n> dies with the error seen. When a commit hash is used for the\n> rev-list-arg, it is not able to determine a reference from it.\n\nYes.  It is not fundamental, though.  There are occasions where \"git\nfetch\" that does not transfer any object data is still useful (e.g.,\nwhen they create a new branch that points at a commit that already\nis included in the history of another branch, you only need to learn\nwhat that new ref is pointing at, but you already have all the\nobject data).  As the central idea of \"bundle\" is to freeze the data\nthat goes over the wire for a fetch (to allow you sneaker-net\ninstead of \"git fetch\"), a bundle that says \"you are expected to\nhave the history reachable from these commits, now you be aware of\nthe fact that this and that ref points at this and that objects\"\nshould be possible.  Such a bundle would have the prerequisite\nsection and the ref advertisement section, but there is no need for\nany pack data.\n\n"}]}