{"thread":{"id":"52723","subject":"[PATCH] doc: describe Git bundle format","startedAt":"2020-01-30T22:58:32Z","lastAt":"2020-02-12T22:43:21Z","messageCount":16,"participants":["Masaya Suzuki","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"390864","messageId":"20200130225818.193825-1-masayasuzuki@google.com","threadId":"52723","inReplyTo":null,"subject":"[PATCH] doc: describe Git bundle format","fromName":"Masaya Suzuki","fromEmail":"masayasuzuki@google.com","sentAt":"2020-01-30T22:58:18Z","receivedAt":"2020-01-30T22:58:32Z","isPatch":true,"sender":{"key":"masayasuzuki@google.com","avatar":"https://avatars.githubusercontent.com/u/21878?v=4"},"body":"The bundle format was not documented. Describe the format with ABNF and\nexplain the meaning of each part.\n\nSigned-off-by: Masaya Suzuki \n---\n Documentation/technical/bundle-format.txt | 40 +++++++++++++++++++++++\n 1 file changed, 40 insertions(+)\n create mode 100644 Documentation/technical/bundle-format.txt\n\ndiff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt\nnew file mode 100644\nindex 0000000000..dbb80225b5\n--- /dev/null\n+++ b/Documentation/technical/bundle-format.txt\n@@ -0,0 +1,40 @@\n+= Git bundle v2 format\n+\n+The Git bundle format is a format that represents both refs and Git objects.\n+\n+== Format\n+\n+We will use ABNF notation to define the Git bundle format. See\n+protocol-common.txt for the details.\n+\n+----\n+bundle = signature references pack\n+signature = \"# v2 git bundle\" LF\n+\n+references = *(prerequisite / ref) LF\n+prerequisite = \"-\" obj-id SP comment LF\n+comment = *CHAR\n+ref = obj-id SP refname LF\n+\n+pack = ... ; packfile\n+----\n+\n+== Semantics\n+\n+A Git bundle consists of three parts.\n+\n+* Prerequisites: Optional list of objects that are not included in the bundle\n+ file. A bundle can reference these prerequisite objects (or it can reference\n+ the objects reachable from the prerequisite objects). The bundle itself\n+ might not contain those objects.\n+* References: Mapping of ref names to objects.\n+* Git objects: Commit, tree, blob, and tags. These are included in the pack\n+ format.\n+\n+If a bundle contains prerequisites, it means the bundle has a thin pack and the\n+bundle alone is not enough for resolving all objects. When you read such\n+bundles, you should have those missing objects beforehand.\n+\n+In the bundle format, there can be a comment following a prerequisite obj-id.\n+This is a comment and it has no specific meaning. When you write a bundle, you\n+can put any string here. When you read a bundle, you can ignore this part.\n-- \n2.25.0.341.g760bfbb309-goog\n\n"},{"id":"390904","messageId":"nycvar.QRO.7.76.6.2001311456220.46@tvgsbejvaqbjf.bet","threadId":"52723","inReplyTo":"20200130225818.193825-1-masayasuzuki@google.com","subject":"Re: [PATCH] doc: describe Git bundle format","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-01-31T13:56:35Z","receivedAt":"2020-01-31T13:56:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 30 Jan 2020, Masaya Suzuki wrote:\n\n> The bundle format was not documented. Describe the format with ABNF and\n> explain the meaning of each part.\n\nLGTM,\nDscho\n\n>\n> Signed-off-by: Masaya Suzuki \n> ---\n> Documentation/technical/bundle-format.txt | 40 +++++++++++++++++++++++\n> 1 file changed, 40 insertions(+)\n> create mode 100644 Documentation/technical/bundle-format.txt\n>\n> diff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt\n> new file mode 100644\n> index 0000000000..dbb80225b5\n> --- /dev/null\n> +++ b/Documentation/technical/bundle-format.txt\n> @@ -0,0 +1,40 @@\n> += Git bundle v2 format\n> +\n> +The Git bundle format is a format that represents both refs and Git objects.\n> +\n> +== Format\n> +\n> +We will use ABNF notation to define the Git bundle format. See\n> +protocol-common.txt for the details.\n> +\n> +----\n> +bundle = signature references pack\n> +signature = \"# v2 git bundle\" LF\n> +\n> +references = *(prerequisite / ref) LF\n> +prerequisite = \"-\" obj-id SP comment LF\n> +comment = *CHAR\n> +ref = obj-id SP refname LF\n> +\n> +pack = ... ; packfile\n> +----\n> +\n> +== Semantics\n> +\n> +A Git bundle consists of three parts.\n> +\n> +* Prerequisites: Optional list of objects that are not included in the bundle\n> + file. A bundle can reference these prerequisite objects (or it can reference\n> + the objects reachable from the prerequisite objects). The bundle itself\n> + might not contain those objects.\n> +* References: Mapping of ref names to objects.\n> +* Git objects: Commit, tree, blob, and tags. These are included in the pack\n> + format.\n> +\n> +If a bundle contains prerequisites, it means the bundle has a thin pack and the\n> +bundle alone is not enough for resolving all objects. When you read such\n> +bundles, you should have those missing objects beforehand.\n> +\n> +In the bundle format, there can be a comment following a prerequisite obj-id.\n> +This is a comment and it has no specific meaning. When you write a bundle, you\n> +can put any string here. When you read a bundle, you can ignore this part.\n> --\n> 2.25.0.341.g760bfbb309-goog\n>\n>\n"},{"id":"390959","messageId":"xmqqk1579xa4.fsf@gitster-ct.c.googlers.com","threadId":"52723","inReplyTo":"20200130225818.193825-1-masayasuzuki@google.com","subject":"Re: [PATCH] doc: describe Git bundle format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-01-31T20:38:27Z","receivedAt":"2020-01-31T20:38:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Masaya Suzuki writes:\n\n> The bundle format was not documented. Describe the format with ABNF and\n> explain the meaning of each part.\n\nThanks.\n\n>\n> Signed-off-by: Masaya Suzuki \n> ---\n> Documentation/technical/bundle-format.txt | 40 +++++++++++++++++++++++\n> 1 file changed, 40 insertions(+)\n> create mode 100644 Documentation/technical/bundle-format.txt\n>\n> diff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt\n> new file mode 100644\n> index 0000000000..dbb80225b5\n> --- /dev/null\n> +++ b/Documentation/technical/bundle-format.txt\n> @@ -0,0 +1,40 @@\n> += Git bundle v2 format\n> +\n> +The Git bundle format is a format that represents both refs and Git objects.\n> +\n> +== Format\n> +\n> +We will use ABNF notation to define the Git bundle format. See\n> +protocol-common.txt for the details.\n> +\n> +----\n> +bundle = signature references pack\n> +signature = \"# v2 git bundle\" LF\n\nGood. \"signature\" is the name used by bundle.c::create_bundle() to\ncall this part.\n\n> +references = *(prerequisite / ref) LF\n\nThis allows prereq and ref can come inter-mixed, but I think we show\nall prerequisites first before refs.\n\n> +prerequisite = \"-\" obj-id SP comment LF\n> +comment = *CHAR\n\nDo readers know what CHAR consists of? Anything other than NUL and\nLF?\n\n> +ref = obj-id SP refname LF\n\nOK.\n\n\"prerequisite\" and \"ref\" are both used in bundle.c::create_bundle(),\nso calling these parts with these names is consistent with the code.\n\"head\" is also a good name for the latter as \"git bundle list-heads\"\nis the way the end-users access them from outside.\n\n> +\n> +pack = ... ; packfile\n> +----\n> +\n> +== Semantics\n> +\n> +A Git bundle consists of three parts.\n> +\n> +* Prerequisites: Optional list of objects that are not included in the bundle\n> + file. A bundle can reference these prerequisite objects (or it can reference\n> + the objects reachable from the prerequisite objects). The bundle itself\n> + might not contain those objects.\n\nWhile not incorrect per-se, the above misses the more important\npoints (and defers the description to a later paragraph). It is\nbetter to describe what it means to have prereqs upfront. \n\n> +* References: Mapping of ref names to objects.\n> +* Git objects: Commit, tree, blob, and tags. These are included in the pack\n> + format.\n> +\n\nMatch the name you used to descibe the parts in the earlier ABNF\ndescription, so that the correspondence is clear to the readers.\nYou somehow used \"references\" to mean both prereqs and heads, but in\nthe above you are describing only \"heads\" under the label of\n\"references\".\n\nPerhaps something like this?\n\n * \"Prerequisites\" lists the objects that are NOT included in the\n bundle and the receiver of the bundle MUST already have, in\n order to use the data in the bundle. The objects stored in\n the bundle may refer to prerequiste objects and anything\n reachable from them and/or expressed as a delta against\n prerequisite objects.\n\n * \"Heads\" record the tips of the history graph, iow, what the\n receiver of the bundle CAN \"git fetch\" from it.\n\n * \"Pack\" is the pack data stream \"git fetch\" would send, if you\n fetch from a repository that has the references recorded in\n the \"Heads\" above into a repository that has references\n pointing at the objects listed in \"Prerequisites\" above.\n\n> +If a bundle contains prerequisites, it means the bundle has a thin pack and the\n> +bundle alone is not enough for resolving all objects. When you read such\n> +bundles, you should have those missing objects beforehand.\n\nWith the above rewrite, this paragraph is unneeded.\n\n> +In the bundle format, there can be a comment following a prerequisite obj-id.\n> +This is a comment and it has no specific meaning. When you write a bundle, you\n> +can put any string here. When you read a bundle, you can ignore this part.\n\nIs it \"you can\"? At least the last one should be \"readers of a\nbundle MUST ignore the comment\", I think.\n"},{"id":"390967","messageId":"CAJB1erXnNe0yGvL+wgU9RXAA6Vyx7T2dwM9NgCmUChOtL102NQ@mail.gmail.com","threadId":"52723","inReplyTo":"xmqqk1579xa4.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH] doc: describe Git bundle format","fromName":"Masaya Suzuki","fromEmail":"masayasuzuki@google.com","sentAt":"2020-01-31T21:49:53Z","receivedAt":"2020-01-31T21:50:07Z","isPatch":true,"sender":{"key":"masayasuzuki@google.com","avatar":"https://avatars.githubusercontent.com/u/21878?v=4"},"body":"On Fri, Jan 31, 2020 at 12:38 PM Junio C Hamano wrote:\n>\n> Masaya Suzuki writes:\n>\n> > The bundle format was not documented. Describe the format with ABNF and\n> > explain the meaning of each part.\n>\n> Thanks.\n>\n> >\n> > Signed-off-by: Masaya Suzuki \n> > ---\n> > Documentation/technical/bundle-format.txt | 40 +++++++++++++++++++++++\n> > 1 file changed, 40 insertions(+)\n> > create mode 100644 Documentation/technical/bundle-format.txt\n> >\n> > diff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt\n> > new file mode 100644\n> > index 0000000000..dbb80225b5\n> > --- /dev/null\n> > +++ b/Documentation/technical/bundle-format.txt\n> > @@ -0,0 +1,40 @@\n> > += Git bundle v2 format\n> > +\n> > +The Git bundle format is a format that represents both refs and Git objects.\n> > +\n> > +== Format\n> > +\n> > +We will use ABNF notation to define the Git bundle format. See\n> > +protocol-common.txt for the details.\n> > +\n> > +----\n> > +bundle = signature references pack\n> > +signature = \"# v2 git bundle\" LF\n>\n> Good. \"signature\" is the name used by bundle.c::create_bundle() to\n> call this part.\n>\n> > +references = *(prerequisite / ref) LF\n>\n> This allows prereq and ref can come inter-mixed, but I think we show\n> all prerequisites first before refs.\n\nBased on bundle.c::parse_bundle_header(), I infer that this can be\nmixed. If that's not intended, this can be changed to have\nprerequisites first.\n\n>\n> > +prerequisite = \"-\" obj-id SP comment LF\n> > +comment = *CHAR\n>\n> Do readers know what CHAR consists of? Anything other than NUL and\n> LF?\n\nRFC 5234 defines core rules\n(https://tools.ietf.org/html/rfc5234#appendix-B.1), and these CHAR etc\nare defined there. It should be OK to use these rules.\n\n>\n> > +ref = obj-id SP refname LF\n>\n> OK.\n>\n> \"prerequisite\" and \"ref\" are both used in bundle.c::create_bundle(),\n> so calling these parts with these names is consistent with the code.\n> \"head\" is also a good name for the latter as \"git bundle list-heads\"\n> is the way the end-users access them from outside.\n>\n> > +\n> > +pack = ... ; packfile\n> > +----\n> > +\n> > +== Semantics\n> > +\n> > +A Git bundle consists of three parts.\n> > +\n> > +* Prerequisites: Optional list of objects that are not included in the bundle\n> > + file. A bundle can reference these prerequisite objects (or it can reference\n> > + the objects reachable from the prerequisite objects). The bundle itself\n> > + might not contain those objects.\n>\n> While not incorrect per-se, the above misses the more important\n> points (and defers the description to a later paragraph). It is\n> better to describe what it means to have prereqs upfront.\n>\n> > +* References: Mapping of ref names to objects.\n> > +* Git objects: Commit, tree, blob, and tags. These are included in the pack\n> > + format.\n> > +\n>\n> Match the name you used to descibe the parts in the earlier ABNF\n> description, so that the correspondence is clear to the readers.\n> You somehow used \"references\" to mean both prereqs and heads, but in\n> the above you are describing only \"heads\" under the label of\n> \"references\".\n\nYes. It should match with the ABNF definition above.\n\nI usually use \"heads\" to mean \"references under refs/heads/*\" (not\nsure if this is true for other people). Since a bundle can contain\ntags etc., using \"heads\" here seems confusing. With prerequisites and\nreferences split you mentioned above, I think I can make ABNF and this\nsemantics section consistent in terms of wording.\n\nbundle = signature *prerequisite *ref LF pack\nprerequisite = \"-\" obj-id SP comment LF\ncomment = *CHAR\nreference = obj-id SP refname LF\npack = ... ; packfile\n\nThe terms (\"prerequisite\" and \"reference\") are consistent with\nbundle.h::ref_list.\n\n>\n> Perhaps something like this?\n>\n> * \"Prerequisites\" lists the objects that are NOT included in the\n> bundle and the receiver of the bundle MUST already have, in\n> order to use the data in the bundle. The objects stored in\n> the bundle may refer to prerequiste objects and anything\n> reachable from them and/or expressed as a delta against\n> prerequisite objects.\n\nI want to make sure the meaning of prerequisites.\n\n1. Are they meant for a delta base? Or are they meant to represent a\npartial/shallow state?\n\nIf these prerequisites are used as a delta base, the receiver of the\nbundle MUST have them. If these prerequisites are the indicators of\nthe shallowness or the partialness of the repository, the pack data\nwould have complete data in terms of deltification (e.g. all objects\nin the pack file can be undeltified with just the pack file), and the\nbundle can be treated as a shallow-cloned/partially-cloned repository\nsnapshot.\n\nFrom what I can see from bundle.c, I think it's an indicator of a\ndelta base, not an indicator of a shallow/partial state, but I want to\nmake sure.\n\n2. Do they need to be commits? Or can they be any object type?\n\nFrom what I can see, it seems that they should always be commits.\n\n3. Does the receiver have to have all reachable objects from prerequisites?\n\nMy understanding is \"Yes, the receiver must have all reachable objects\nfrom prerequisites.\" This means that if a receiver has a\nshallow-cloned repository, they might not be able to proceess a bundle\nwith prerequisites. The bundle's pack part can deltify against the\nobjects that exist beyond the shallow depth.\n\n>\n> * \"Heads\" record the tips of the history graph, iow, what the\n> receiver of the bundle CAN \"git fetch\" from it.\n>\n> * \"Pack\" is the pack data stream \"git fetch\" would send, if you\n> fetch from a repository that has the references recorded in\n> the \"Heads\" above into a repository that has references\n> pointing at the objects listed in \"Prerequisites\" above.\n\nI'll adopt this in the next patch.\n\n>\n> > +If a bundle contains prerequisites, it means the bundle has a thin pack and the\n> > +bundle alone is not enough for resolving all objects. When you read such\n> > +bundles, you should have those missing objects beforehand.\n>\n> With the above rewrite, this paragraph is unneeded.\n>\n> > +In the bundle format, there can be a comment following a prerequisite obj-id.\n> > +This is a comment and it has no specific meaning. When you write a bundle, you\n> > +can put any string here. When you read a bundle, you can ignore this part.\n>\n> Is it \"you can\"? At least the last one should be \"readers of a\n> bundle MUST ignore the comment\", I think.\n\nI'll change this to MUST.\n"},{"id":"390973","messageId":"20200131221800.240352-1-masayasuzuki@google.com","threadId":"52723","inReplyTo":"20200130225818.193825-1-masayasuzuki@google.com","subject":"[PATCH v2] doc: describe Git bundle format","fromName":"Masaya Suzuki","fromEmail":"masayasuzuki@google.com","sentAt":"2020-01-31T22:18:00Z","receivedAt":"2020-01-31T22:18:24Z","isPatch":true,"sender":{"key":"masayasuzuki@google.com","avatar":"https://avatars.githubusercontent.com/u/21878?v=4"},"body":"The bundle format was not documented. Describe the format with ABNF and\nexplain the meaning of each part.\n\nSigned-off-by: Masaya Suzuki \n---\nChanges from v1:\n\n* Update the ABNF definition so that prerequisites come before references.\n* Adopt Junio's suggestion on the semantics section.\n* State that the receiver MUST ignore the comments in the prereqs.\n* Change \"you\" to \"the receiver\" and \"the sender\" (I wonder if this should be\n \"writer\" and \"reader\").\n\n Documentation/technical/bundle-format.txt | 41 +++++++++++++++++++++++\n 1 file changed, 41 insertions(+)\n create mode 100644 Documentation/technical/bundle-format.txt\n\ndiff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt\nnew file mode 100644\nindex 0000000000..f568fcd7d1\n--- /dev/null\n+++ b/Documentation/technical/bundle-format.txt\n@@ -0,0 +1,41 @@\n+= Git bundle v2 format\n+\n+The Git bundle format is a format that represents both refs and Git objects.\n+\n+== Format\n+\n+We will use ABNF notation to define the Git bundle format. See\n+protocol-common.txt for the details.\n+\n+----\n+bundle = signature *prerequisite *reference LF pack\n+signature = \"# v2 git bundle\" LF\n+\n+prerequisite = \"-\" obj-id SP comment LF\n+comment = *CHAR\n+reference = obj-id SP refname LF\n+\n+pack = ... ; packfile\n+----\n+\n+== Semantics\n+\n+A Git bundle consists of three parts.\n+\n+* \"Prerequisites\" lists the objects that are NOT included in the bundle and the\n+ receiver of the bundle MUST already have, in order to use the data in the\n+ bundle. The objects stored in the bundle may refer to prerequisite objects and\n+ anything reachable from them and/or expressed as a delta against prerequisite\n+ objects.\n+\n+* \"References\" record the tips of the history graph, iow, what the receiver of\n+ the bundle CAN \"git fetch\" from it.\n+\n+* \"Pack\" is the pack data stream \"git fetch\" would send, if you fetch from a\n+ repository that has the references recorded in the \"References\" above into a\n+ repository that has references pointing at the objects listed in\n+ \"Prerequisites\" above.\n+\n+In the bundle format, there can be a comment following a prerequisite obj-id.\n+This is a comment and it has no specific meaning. The sender of the bundle MAY\n+put any string here. The receiver of the bundle MUST ignore the comment.\n-- \n2.25.0.341.g760bfbb309-goog\n\n"},{"id":"390974","messageId":"xmqqy2tn8c3w.fsf@gitster-ct.c.googlers.com","threadId":"52723","inReplyTo":"CAJB1erXnNe0yGvL+wgU9RXAA6Vyx7T2dwM9NgCmUChOtL102NQ@mail.gmail.com","subject":"Re: [PATCH] doc: describe Git bundle format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-01-31T23:01:07Z","receivedAt":"2020-01-31T23:01:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Masaya Suzuki writes:\n\n>> > +prerequisite = \"-\" obj-id SP comment LF\n>> > +comment = *CHAR\n>>\n>> Do readers know what CHAR consists of? Anything other than NUL and\n>> LF?\n>\n> RFC 5234 defines core rules\n> (https://tools.ietf.org/html/rfc5234#appendix-B.1), and these CHAR etc\n> are defined there. It should be OK to use these rules.\n\nThat's not what I asked. Do readers know that? Did you tell them\nthat we expect they are familiar with the RFC convention?\n\nIt might be easier to make the above simple ABNF understandable to\nthose without knowledge of RFC 5234 by spelling out what CHAR in the\ncontext of the above description means. Or to tell them \"go over\nthere and learn CHAR then come back\". We need to do one of them.\n\n> I want to make sure the meaning of prerequisites.\n>\n> 1. Are they meant for a delta base? Or are they meant to represent a\n> partial/shallow state?\n\nThey are meant as the \"bottom boundary\" of the range of the pack\ndata stored in the bundle.\n\nThink of \"git rev-list --objects $heads --not $prerequisites\". If\nwe limit ourselves to commits, in the simplest case, \"git log\nmaint..master\". Imagine your repository has everything up to\n'maint' (and nothing else) and then you are \"git fetch\"-ing from\nanother repository that advanced the tip that now points at\n'master'. Imagine the data transferred over the network. Imagine\nthat data is frozen on disk somehow. That is what a bundle is.\n\nSo, 'maint' is the prerequisite---for the person who builds the\nbundle, it can safely be assumed that the bundle will be used only\nby those who already has 'maint'.\n\nThere is nothing about 'partial' or 'shallow'. And even though a\nbundle typically has deltified objects in the packfile, it does not\nhave to. Some objects are delitifed against prerequisite, and the\nlogic to generate thin packs may even prefer to use the\nprerequisites as the delta base, but it is merely a side effect that\nthe prerequisites are at the \"bottom boundary\" of the range.\n\n> 2. Do they need to be commits? Or can they be any object type?\n>\n> From what I can see, it seems that they should always be commits.\n>\n> 3. Does the receiver have to have all reachable objects from prerequisites?\n\nI would say that the receiver needs to have everything that is\nneeded to \"complete\" prereqs.\n\nBundle transfer predates shallow or incomplete repositories, but I\nthink that we can (and we should if needed) update it to adjust to\nthese situations by using the appropriate definition of what it\nmeans to \"complete\". In a lazy clone, it may be sufficient to have\npromisor remote that has everything reachable from them. In a\nshallow clone, the repository may have to be deep enough to have\nthem and objects immediately reachable from them (e.g. trees and\nblobs for a commit at the \"bottom boundary\").\n\nThanks.\n"},{"id":"390975","messageId":"xmqqtv4b8bus.fsf@gitster-ct.c.googlers.com","threadId":"52723","inReplyTo":"20200131221800.240352-1-masayasuzuki@google.com","subject":"Re: [PATCH v2] doc: describe Git bundle format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-01-31T23:06:35Z","receivedAt":"2020-01-31T23:06:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Masaya Suzuki writes:\n\n> * Change \"you\" to \"the receiver\" and \"the sender\" (I wonder if this should be\n> \"writer\" and \"reader\").\n\nI come from \"a bundle is a frozen snapshot of 'git fetch' transfer\"\nschool, and consider the act of reading a \"fetch\" too much, but I\nthink \"writer\" and \"reader\" is more understandable for new readers.\n\nThanks.\n"},{"id":"390978","messageId":"CAJB1erXqK-a2uDPPQDLpdLYnPC8Mcxjo2ER0qSAsD9DOVHSmGQ@mail.gmail.com","threadId":"52723","inReplyTo":"xmqqy2tn8c3w.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH] doc: describe Git bundle format","fromName":"Masaya Suzuki","fromEmail":"masayasuzuki@google.com","sentAt":"2020-01-31T23:57:55Z","receivedAt":"2020-02-01T00:01:16Z","isPatch":true,"sender":{"key":"masayasuzuki@google.com","avatar":"https://avatars.githubusercontent.com/u/21878?v=4"},"body":"On Fri, Jan 31, 2020 at 3:01 PM Junio C Hamano wrote:\n>\n> Masaya Suzuki writes:\n>\n> >> > +prerequisite = \"-\" obj-id SP comment LF\n> >> > +comment = *CHAR\n> >>\n> >> Do readers know what CHAR consists of? Anything other than NUL and\n> >> LF?\n> >\n> > RFC 5234 defines core rules\n> > (https://tools.ietf.org/html/rfc5234#appendix-B.1), and these CHAR etc\n> > are defined there. It should be OK to use these rules.\n>\n> That's not what I asked. Do readers know that? Did you tell them\n> that we expect they are familiar with the RFC convention?\n\nThe patch says \"We will use ABNF notation to define the Git bundle\nformat. See protocol-common.txt for the details.\", and\nprotocol-common.txt says \"ABNF notation as described by RFC 5234 is\nused within the protocol documents, except the following replacement\ncore rules are used:\". In order to interpret this ABNF definition,\nit's not enough to read RFC 5234, but the reader has to read\nprotocol-common.txt. Otherwise, they cannot understand what `obj-id`\nis and what `refname` is. Those are not defined in RFC 5234. They're\ndefined in protocol-common.txt.\n\nBased on the fact that (1) this document instructs the reader to see\nprotocol-common.txt in the beginning and (2) protocol-common.txt is\nneeded to interpret this definition and protocol-common.txt says RFC\n5234 describes ABNF format, the readers should know ABNF is defined in\nRFC 5234 and ABNF includes those LF, CHAR, and SP as a part of the\ndefinition after reading the first sentence and referenced documents.\n\n>\n> It might be easier to make the above simple ABNF understandable to\n> those without knowledge of RFC 5234 by spelling out what CHAR in the\n> context of the above description means. Or to tell them \"go over\n> there and learn CHAR then come back\". We need to do one of them.\n\nAs I said above, the first sentence says \"See protocol-common.txt\"\nwhich includes the reference to the RFC and other non-terminals. Note\nthat, not only CHAR, but obj-id and refname are not defined here as\nwell. The readers need to reference protocol-common.txt to get the\ndefinition of them.\n\n>\n> > I want to make sure the meaning of prerequisites.\n> >\n> > 1. Are they meant for a delta base? Or are they meant to represent a\n> > partial/shallow state?\n>\n> They are meant as the \"bottom boundary\" of the range of the pack\n> data stored in the bundle.\n>\n> Think of \"git rev-list --objects $heads --not $prerequisites\". If\n> we limit ourselves to commits, in the simplest case, \"git log\n> maint..master\". Imagine your repository has everything up to\n> 'maint' (and nothing else) and then you are \"git fetch\"-ing from\n> another repository that advanced the tip that now points at\n> 'master'. Imagine the data transferred over the network. Imagine\n> that data is frozen on disk somehow. That is what a bundle is.\n>\n> So, 'maint' is the prerequisite---for the person who builds the\n> bundle, it can safely be assumed that the bundle will be used only\n> by those who already has 'maint'.\n>\n> There is nothing about 'partial' or 'shallow'. And even though a\n> bundle typically has deltified objects in the packfile, it does not\n> have to. Some objects are delitifed against prerequisite, and the\n> logic to generate thin packs may even prefer to use the\n> prerequisites as the delta base, but it is merely a side effect that\n> the prerequisites are at the \"bottom boundary\" of the range.\n\nOK. Then, it's better to make this clear. If you follow the analogy of\nsaved git-fetch response, it's possible that these prerequisites are\ninterpreted same as \"shallow\" lines of the shallow clone response.\nIt's more like \"have\" lines of git-fetch request.\n\n> > 2. Do they need to be commits? Or can they be any object type?\n> >\n> > From what I can see, it seems that they should always be commits.\n> >\n> > 3. Does the receiver have to have all reachable objects from prerequisites?\n>\n> I would say that the receiver needs to have everything that is\n> needed to \"complete\" prereqs.\n>\n> Bundle transfer predates shallow or incomplete repositories, but I\n> think that we can (and we should if needed) update it to adjust to\n> these situations by using the appropriate definition of what it\n> means to \"complete\". In a lazy clone, it may be sufficient to have\n> promisor remote that has everything reachable from them. In a\n> shallow clone, the repository may have to be deep enough to have\n> them and objects immediately reachable from them (e.g. trees and\n> blobs for a commit at the \"bottom boundary\").\n\nI think there are two completeness of a packfile:\n\n* Delta complete: If an object in a packfile is deltified, the delta\nbase exists in the same packfile.\n* Object complete: If an object in a packfile contains a reference to\nanother object, that object exists in the same packfile.\n\nFor example, initial shallow clone response should contain a\ndelta-complete object-incomplete packfile. Incremental fetch response\nand bundles with prereqs would have a delta-incomplete\nobject-incomplete packfile. Creating delta-incomplete object-complete\npackfile is possible (e.g. create a parallel history with all blobs\nslightly modified and deltify against the original branch. I can\ncreate a packfile with all objects in one history with all objects\ndeltified with the other history), but it's a rare case.\n\nThe reader of a bundle SHOULD have all objects reachable from prereqs.\n"},{"id":"391086","messageId":"xmqqpneu8ba7.fsf@gitster-ct.c.googlers.com","threadId":"52723","inReplyTo":"CAJB1erXqK-a2uDPPQDLpdLYnPC8Mcxjo2ER0qSAsD9DOVHSmGQ@mail.gmail.com","subject":"Re: [PATCH] doc: describe Git bundle format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-04T18:20:16Z","receivedAt":"2020-02-04T18:20:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Masaya Suzuki writes:\n\n> * Delta complete: If an object in a packfile is deltified, the delta\n> base exists in the same packfile.\n\nYes, even though \"thin\" packs delierately violate this to save size,\nnormal packs, and more importantly, on-disk packs, are complete in\nthis sense.\n\n> * Object complete: If an object in a packfile contains a reference to\n> another object, that object exists in the same packfile.\n\nA single packfile that would result from a full clone at some time\nin the project's history would be \"complete\" in this sense. Such a\npackfile may contain all objects that are needed to reproduce the\nhistory up to v1.0.1, or another larger \"object complete\" packfile\nmay contain everything needed for the history up to v3.0.0. So as a\nconcept, this can be defined sensibly. In the original packfile\ndesign, however, this concept was not useful (iow, there was nowhere\nthat cared if a packfile is \"object complete\" or not), so I do not\nthink there is no explicit \"support\" to ensure or validate this\ntrait in the system.\n\nObviously, a bundle that stores object incomplete pack must have\nbeen created with the bottom boundary.\n\n> The reader of a bundle SHOULD have all objects reachable from prereqs.\n\nPerhaps. \n\nIt _might_ be possible to teach \"git clone\" to produce a shallow\nclone whose shallow cut-off points match the prerequisites of the\nbundle, so it depends on what the reader wants to do with the data,\nthough.\n\nThanks.\n\n"},{"id":"391350","messageId":"20200207204225.123764-1-masayasuzuki@google.com","threadId":"52723","inReplyTo":"20200131221800.240352-1-masayasuzuki@google.com","subject":"[PATCH v3] doc: describe Git bundle format","fromName":"Masaya Suzuki","fromEmail":"masayasuzuki@google.com","sentAt":"2020-02-07T20:42:25Z","receivedAt":"2020-02-07T20:42:36Z","isPatch":true,"sender":{"key":"masayasuzuki@google.com","avatar":"https://avatars.githubusercontent.com/u/21878?v=4"},"body":"The bundle format was not documented. Describe the format with ABNF and\nexplain the meaning of each part.\n\nSigned-off-by: Masaya Suzuki \n---\nChanges from v2:\n\n* Change \"sender\" and \"receiver\" to \"writer\" and \"reader\".\n* Add an example of a case that a bundle can reference an object outside of the\n bundle.\n* Mention that the prerequisites are different from the shallow\n boundary, and the bundle format cannot represent a shallow clone repository.\n\n\n Documentation/technical/bundle-format.txt | 48 +++++++++++++++++++++++\n 1 file changed, 48 insertions(+)\n create mode 100644 Documentation/technical/bundle-format.txt\n\ndiff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt\nnew file mode 100644\nindex 0000000000..0e828151a5\n--- /dev/null\n+++ b/Documentation/technical/bundle-format.txt\n@@ -0,0 +1,48 @@\n+= Git bundle v2 format\n+\n+The Git bundle format is a format that represents both refs and Git objects.\n+\n+== Format\n+\n+We will use ABNF notation to define the Git bundle format. See\n+protocol-common.txt for the details.\n+\n+----\n+bundle = signature *prerequisite *reference LF pack\n+signature = \"# v2 git bundle\" LF\n+\n+prerequisite = \"-\" obj-id SP comment LF\n+comment = *CHAR\n+reference = obj-id SP refname LF\n+\n+pack = ... ; packfile\n+----\n+\n+== Semantics\n+\n+A Git bundle consists of three parts.\n+\n+* \"Prerequisites\" lists the objects that are NOT included in the bundle and the\n+ reader of the bundle MUST already have, in order to use the data in the\n+ bundle. The objects stored in the bundle may refer to prerequisite objects and\n+ anything reachable from them (e.g. a tree object in the bundle can reference\n+ a blob that is reachable from a prerequisite) and/or expressed as a delta\n+ against prerequisite objects.\n+\n+* \"References\" record the tips of the history graph, iow, what the reader of the\n+ bundle CAN \"git fetch\" from it.\n+\n+* \"Pack\" is the pack data stream \"git fetch\" would send, if you fetch from a\n+ repository that has the references recorded in the \"References\" above into a\n+ repository that has references pointing at the objects listed in\n+ \"Prerequisites\" above.\n+\n+In the bundle format, there can be a comment following a prerequisite obj-id.\n+This is a comment and it has no specific meaning. The writer of the bundle MAY\n+put any string here. The reader of the bundle MUST ignore the comment.\n+\n+=== Note on the shallow clone and a Git bundle\n+\n+Note that the prerequisites does not represent a shallow-clone boundary. The\n+semantics of the prerequisites and the shallow-clone boundaries are different,\n+and the Git bundle v2 format cannot represent a shallow clone repository.\n-- \n2.25.0.341.g760bfbb309-goog\n\n"},{"id":"391351","messageId":"CAJB1erXMR_aCqVPsH6WQZdC7yybOBCUbwkJDv3LtU2f0ymNmbQ@mail.gmail.com","threadId":"52723","inReplyTo":"20200207204225.123764-1-masayasuzuki@google.com","subject":"Re: [PATCH v3] doc: describe Git bundle format","fromName":"Masaya Suzuki","fromEmail":"masayasuzuki@google.com","sentAt":"2020-02-07T20:44:12Z","receivedAt":"2020-02-07T20:44:27Z","isPatch":true,"sender":{"key":"masayasuzuki@google.com","avatar":"https://avatars.githubusercontent.com/u/21878?v=4"},"body":"On Fri, Feb 7, 2020 at 12:42 PM Masaya Suzuki wrote:\n> +=== Note on the shallow clone and a Git bundle\n> +\n> +Note that the prerequisites does not represent a shallow-clone boundary. The\n\nthe prerequisites do not\n"},{"id":"391353","messageId":"xmqqlfpe2jx3.fsf@gitster-ct.c.googlers.com","threadId":"52723","inReplyTo":"CAJB1erXMR_aCqVPsH6WQZdC7yybOBCUbwkJDv3LtU2f0ymNmbQ@mail.gmail.com","subject":"Re: [PATCH v3] doc: describe Git bundle format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-07T20:59:20Z","receivedAt":"2020-02-07T20:59:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Masaya Suzuki writes:\n\n> On Fri, Feb 7, 2020 at 12:42 PM Masaya Suzuki wrote:\n>> +=== Note on the shallow clone and a Git bundle\n>> +\n>> +Note that the prerequisites does not represent a shallow-clone boundary. The\n>\n> the prerequisites do not\n\nGrammo aside, I am not sure if that particular Note is beneficial to\nbegin with. I would imagine that you can get a bundle that holds\nall the objects in a shallow repository by specifying the range that\nmatch the shallow-clone boundary when you run \"git bundle create\"\nwhile disabling thin-pack generation.\n\nThe support of shallow-clone by Git may be incomplete and it may not\nbe easy to form such a range, and \"git bundle create\" command may\nnot have a knob to disable thin-pack generation, but that does not\nmean that the bundle *format* cannot be used to represent the\nshallow boundary.\n\n"},{"id":"391357","messageId":"CAJB1erVZ5E6FxtD8gJXXpzJjquvksLxpz+h1TzG52Yr9DFmDRQ@mail.gmail.com","threadId":"52723","inReplyTo":"xmqqlfpe2jx3.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3] doc: describe Git bundle format","fromName":"Masaya Suzuki","fromEmail":"masayasuzuki@google.com","sentAt":"2020-02-07T22:21:37Z","receivedAt":"2020-02-07T22:21:51Z","isPatch":true,"sender":{"key":"masayasuzuki@google.com","avatar":"https://avatars.githubusercontent.com/u/21878?v=4"},"body":"On Fri, Feb 7, 2020 at 12:59 PM Junio C Hamano wrote:\n>\n> Masaya Suzuki writes:\n>\n> > On Fri, Feb 7, 2020 at 12:42 PM Masaya Suzuki wrote:\n> >> +=== Note on the shallow clone and a Git bundle\n> >> +\n> >> +Note that the prerequisites does not represent a shallow-clone boundary. The\n> >\n> > the prerequisites do not\n>\n> Grammo aside, I am not sure if that particular Note is beneficial to\n> begin with. I would imagine that you can get a bundle that holds\n> all the objects in a shallow repository by specifying the range that\n> match the shallow-clone boundary when you run \"git bundle create\"\n> while disabling thin-pack generation.\n\nYes. The reason that I've been trying to check the semantics of the\nprerequisites is that I DO recognize that this is possible\nformat-wise. I'm not sure if this Git implementation can create such\nbundles, but format-wise such bundles can be created.\n\nWhen writing a Git bundle parser in other implementations (like JGit),\nit's not clear whether, as a library, I should support such use cases.\nIf such usage is supported in the format, then the semantics of the\nprerequisites changes. Currently the prerequisites are defined as the\nobjects that are NOT included in the bundle, and the reader of the\nbundle MUST already have, in order to use the data in the bundle. If\nthe format supports shallow-cloned repository, this will be defined as\nthe objects that are NOT included in the bundle. If the reader wants\nto read this bundle as if it's a non-shallow clone, the reader of the\nbundle MUST have the objects that are reachable from these\nprerequisites. If the reader wants to read this bundle as if it's a\nshallow clone, the reader MUST treat these as a shallow boundary.\n\nAlso, this change will put further restrictions on the pack. \"Pack\" is\nthe pack data stream \"git fetch\" would send. If the writer of a bundle\nwants to write as a shallow-clone pack, the pack MUST NOT reference\nobjects outside of the shallow boundary from the pack file as a delta\nbase. The writer MAY reference the commit objects outside of the\nshallow boundary as a parent.\n\nThe readers and the writers of bundles MUST communicate whether a\nbundle represents a shallow clone repository in other means. The\nbundle file does not have any indicator whether it's a shallow clone\nbundle or not.\n\n> The support of shallow-clone by Git may be incomplete and it may not\n> be easy to form such a range, and \"git bundle create\" command may\n> not have a knob to disable thin-pack generation, but that does not\n> mean that the bundle *format* cannot be used to represent the\n> shallow boundary.\n\nAs I wrote above, if this bundle format supports the shallow clone\nstate, the semantics will change and writers and readers have\ndifferent constraints on the packs. In order to do so, the readers and\nthe writers have to agree whether it's a shallow clone or not in other\nmean since the bundle file doesn't have such indicators. I think it's\nbetter to prohibit such use cases (or at least make it as unintended\nusage), and then create a different bundle format version that\nsupports shallow clone boundary (so that the bundle file can be more\nclose to the frozen git-fetch response).\n"},{"id":"391359","messageId":"xmqqa75t3l1h.fsf@gitster-ct.c.googlers.com","threadId":"52723","inReplyTo":"CAJB1erVZ5E6FxtD8gJXXpzJjquvksLxpz+h1TzG52Yr9DFmDRQ@mail.gmail.com","subject":"Re: [PATCH v3] doc: describe Git bundle format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-08T01:49:46Z","receivedAt":"2020-02-08T01:49:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Masaya Suzuki writes:\n\n> Yes. The reason that I've been trying to check the semantics of the\n> prerequisites is that I DO recognize that this is possible\n> format-wise. I'm not sure if this Git implementation can create such\n> bundles, but format-wise such bundles can be created.\n\nYeah, now I get it. \n\nThe problem is *not* that v2 format \"cannot represent a shallow\nclone repository\", but is that there is nothing that prevents a\nbundle in v2 format from depending on objects behind (not just at)\nthe shallow boundary, making it impossible for a reader to guarantee\nthat a bundle with prereqs can be used to create an equivalent\nshallow repository with shallow boundary at the same place as\nprereqs. IOW, bundle with prereqs in the v2 format allows more\nobjects to be omitted than an equivalent shallow repository omits,\nbecause prereqs and shallow cutoff points mean different things.\n\nWhile we are at it, I suspect that with reachability bitmap, a \"git\nfetch\" that updates a history up to commit A to a new history up to\ncommit B can omit more objects than what is directly reachable from\nthe commit A. That is, if A's direct child (call it C) is a commit\nthat reverts A, a blob in A's tree won't be in the bundle (because A\nis a prereq), but the blob at the same path in C is the same blob as\nthe blob at the same path in A's parent (that is what it means for\nthat A's direct child to be a revert of A). In the normal\nenumeration based on object-walk to decide which objects to send,\nsuch a blob in C will be included in the pack, but a reachability\nbitmap can say \"if we assume the reader has A, it must have A^1, so\nthat blob should exist at the reader, hence can be omitted from the\ntransfer even though we are sending commit C.\"\n"},{"id":"391625","messageId":"CAJB1erWuCTYtGdjiwBT3hhbH5Jrdr+cE8duwtoH=P7Fe6BmUpg@mail.gmail.com","threadId":"52723","inReplyTo":"xmqqa75t3l1h.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3] doc: describe Git bundle format","fromName":"Masaya Suzuki","fromEmail":"masayasuzuki@google.com","sentAt":"2020-02-12T22:13:29Z","receivedAt":"2020-02-12T22:13:42Z","isPatch":true,"sender":{"key":"masayasuzuki@google.com","avatar":"https://avatars.githubusercontent.com/u/21878?v=4"},"body":"On Fri, Feb 7, 2020 at 5:49 PM Junio C Hamano wrote:\n>\n> Masaya Suzuki writes:\n>\n> > Yes. The reason that I've been trying to check the semantics of the\n> > prerequisites is that I DO recognize that this is possible\n> > format-wise. I'm not sure if this Git implementation can create such\n> > bundles, but format-wise such bundles can be created.\n>\n> Yeah, now I get it.\n>\n> The problem is *not* that v2 format \"cannot represent a shallow\n> clone repository\", but is that there is nothing that prevents a\n> bundle in v2 format from depending on objects behind (not just at)\n> the shallow boundary, making it impossible for a reader to guarantee\n> that a bundle with prereqs can be used to create an equivalent\n> shallow repository with shallow boundary at the same place as\n> prereqs. IOW, bundle with prereqs in the v2 format allows more\n> objects to be omitted than an equivalent shallow repository omits,\n> because prereqs and shallow cutoff points mean different things.\n\nYes. So, I think it's better to say prereqs and shallow boundaries are\ndifferent.\n\n> While we are at it, I suspect that with reachability bitmap, a \"git\n> fetch\" that updates a history up to commit A to a new history up to\n> commit B can omit more objects than what is directly reachable from\n> the commit A. That is, if A's direct child (call it C) is a commit\n> that reverts A, a blob in A's tree won't be in the bundle (because A\n> is a prereq), but the blob at the same path in C is the same blob as\n> the blob at the same path in A's parent (that is what it means for\n> that A's direct child to be a revert of A). In the normal\n> enumeration based on object-walk to decide which objects to send,\n> such a blob in C will be included in the pack,\n\nThat's interesting. I have never looked CGit's implementation, but I\nthink JGit would omit those objects. (At least that was my\nunderstanding. Not confirmed with the code.)\n\nAnyway. Is it OK with adding this small note on \"prereq is not a\nshallow boundary\"? In practice, there are not many Git implementations\nthat handle Git bundles, so it's not that big deal as long those few\nimplementers recognize this, but this document is meant for those\nimplementers.\n"},{"id":"391630","messageId":"xmqqftfftoje.fsf@gitster-ct.c.googlers.com","threadId":"52723","inReplyTo":"CAJB1erWuCTYtGdjiwBT3hhbH5Jrdr+cE8duwtoH=P7Fe6BmUpg@mail.gmail.com","subject":"Re: [PATCH v3] doc: describe Git bundle format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-12T22:43:17Z","receivedAt":"2020-02-12T22:43:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Masaya Suzuki writes:\n\n> Anyway. Is it OK with adding this small note on \"prereq is not a\n> shallow boundary\"?\n\nI thought the text in the latest round is good as-is.\n\nThanks.\n"}]}