{"thread":{"id":"28781","subject":"[RFC/PATCH] define the way new representation types are encoded in the pack","startedAt":"2011-10-28T06:04:40Z","lastAt":"2011-10-30T09:31:15Z","messageCount":12,"participants":["Junio C Hamano","Jakub Narebski","Shawn Pearce","Nicolas Pitre","Nguyen Thai Ngoc Duy"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"178397","messageId":"7v62j9veh3.fsf@alter.siamese.dyndns.org","threadId":"28781","inReplyTo":null,"subject":"[RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-28T06:04:40Z","receivedAt":"2011-10-28T06:04:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"In addition to four basic types (commit, tree, blob and tag), the pack\nstream can encode a few other \"representation\" types, such as REF_DELTA\nand OFS_DELTA. As we allocate 3 bits in the first byte for this purpose,\nwe do not have much room to add new representation types in place, but we\ndo have one value reserved for future expansion.\n\nThis patch is about defining how that reserved value is used.\n\nThe first byte in the pack stream data consists of the following for the\ncurrent representation types:\n\n - Bit 0-3 are used for the low 4-bit of \"some\" size (not necessarily the\n size of the representation);\n\n - Bit 4-6 are used for object types 0-7, but we have not used type 5 so\n far and reserved it for future expansion (we could also use type 0\n recorded in the pack stream for future expansion, just like how I\n convert 5 into the real \"extended\" representation type in this patch);\n\n - Bit 7 is used to signal if the second byte needs to be read for sizes\n that do not fit in the 4-bit.\n\nWhen bit 4-6 encodes type 5, the first byte is used this way:\n\n - Bit 0-3 denotes the real \"extended\" representation type. Because types\n 0-7 can already be encoded without using the extended format, we can\n offset the type by 8 (i.e. if bit 0-3 says 3, it means representation\n type 11 = 3 + 8);\n\n - Bit 4-6 has the value \"5\";\n\n - Bit 7 is used to signal if the _third_ byte needs to be read for larger\n size that cannot be represented with 8-bit.\n\nAs it is unlikely for us to pack things that do not need to record any\nsize, the second byte is always used in full to encode the low 8-bit of\nthe size.\n\nI haven't started using type=8 and upwards for anything yet, but because\nwe have only one \"future expansion\" value left, I want us to be extremely\ncareful in order to avoid painting us into a corner that we cannot get out\nof, so I am sending this out early for a preliminary review.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 3 ++-\n sha1_file.c | 36 ++++++++++++++++++++++++++++++++----\n 2 files changed, 34 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 2e6ad36..b02139b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -380,9 +380,10 @@ enum object_type {\n \tOBJ_TREE = 2,\n \tOBJ_BLOB = 3,\n \tOBJ_TAG = 4,\n-\t/* 5 for future expansion */\n+\tOBJ_EXT = 5, /* 5 for future expansion */\n \tOBJ_OFS_DELTA = 6,\n \tOBJ_REF_DELTA = 7,\n+\tOBJ_CAT_TREE = 8,\n \tOBJ_ANY,\n \tOBJ_MAX\n };\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 27f3b9b..4dcd023 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1254,16 +1254,43 @@ static int experimental_loose_object(unsigned char *map)\n }\n \n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n-\t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n+\tunsigned long len, enum object_type *typep, unsigned long *sizep)\n {\n \tunsigned shift;\n \tunsigned long size, c;\n \tunsigned long used = 0;\n+\tenum object_type type;\n \n+\t/*\n+\t * MSB of the first byte is used to tell if the second byte\n+\t * needs to be read for the size, so type field is only 3-bit\n+\t * wide.\n+\t */\n \tc = buf[used++];\n-\t*type = (c >> 4) & 7;\n-\tsize = c & 15;\n-\tshift = 4;\n+\ttype = (c >> 4) & 7;\n+\n+\tif (type != OBJ_EXT) {\n+\t\t/*\n+\t\t * For basic types of object representations, the low\n+\t\t * 4-bit of the first byte is used for the lowermost\n+\t\t * 4-bit of the size. The MSB of the first byte tells\n+\t\t * if the second byte needs to be read for size.\n+\t\t */\n+\t\tsize = c & 15;\n+\t\tshift = 4;\n+\t} else {\n+\t\t/*\n+\t\t * For extended types, the low 4-bit of the first byte\n+\t\t * is used for the representation type (offset by 8),\n+\t\t * and the size begins at the second byte. The MSB of\n+\t\t * the first byte is still used to indicate the next\n+\t\t * byte (i.e. the third byte) needs to be read for the\n+\t\t * size.\n+\t\t */\n+\t\ttype = (c & 15) + 8;\n+\t\tsize = buf[used++];\n+\t\tshift = 8;\n+\t}\n \twhile (c & 0x80) {\n \t\tif (len <= used || bitsizeof(long) <= shift) {\n \t\t\terror(\"bad object header\");\n@@ -1274,6 +1301,7 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t\tshift += 7;\n \t}\n \t*sizep = size;\n+\t*typep = type;\n \treturn used;\n }\n \n"},{"id":"178399","messageId":"7vwrbptzjm.fsf@alter.siamese.dyndns.org","threadId":"28781","inReplyTo":"7v62j9veh3.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-28T06:12:29Z","receivedAt":"2011-10-28T06:12:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> I haven't started using type=8 and upwards for anything yet, but because\n> we have only one \"future expansion\" value left, I want us to be extremely\n> careful in order to avoid painting us into a corner that we cannot get out\n> of, so I am sending this out early for a preliminary review.\n>\n> Signed-off-by: Junio C Hamano \n> ---\n> cache.h | 3 ++-\n> sha1_file.c | 36 ++++++++++++++++++++++++++++++++----\n> 2 files changed, 34 insertions(+), 5 deletions(-)\n>\n> diff --git a/cache.h b/cache.h\n> index 2e6ad36..b02139b 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -380,9 +380,10 @@ enum object_type {\n> \tOBJ_TREE = 2,\n> \tOBJ_BLOB = 3,\n> \tOBJ_TAG = 4,\n> -\t/* 5 for future expansion */\n> +\tOBJ_EXT = 5, /* 5 for future expansion */\n> \tOBJ_OFS_DELTA = 6,\n> \tOBJ_REF_DELTA = 7,\n> +\tOBJ_CAT_TREE = 8,\n> \tOBJ_ANY,\n> \tOBJ_MAX\n> };\n\nAs people may be able to guess from the name, CAT_TREE is envisioned to\nencode a large data (primarily of type \"blob\") by recording the object\nname of a tree object and probably the total length, and would represent\nthe concatenation of all blobs contained in the tree object when the tree\nis traversed in some fixed order (e.g. Avery's \"bup split\"). I am guessing\nthat the payload for CAT_TREE representation type will be:\n\n - 20-byte object name for the top-level tree object;\n\n - type of the basic object (commit, tree, blob, or tag) it represents,\n even though it is unlikely that we would want to record such a large\n commit or tag that needs CAT_TREE representation;\n\n - the total length of the basic object it represents, even though it is\n redundant (you could traverse and sum the sizes of blobs contained in\n the tree object), it would help sha1_object_info() and friends. This\n will be the \"some size\" I mentioned in the previous message for this\n representation type.\n\nWe would probably add loose object representation for CAT_TREE, which may\nlook like:\n\n \"cattree\" NUL\n NUL\n \n\nand would need to teach unpack_sha1_file() about it. One caveat is that\nwe would want to keep the \"contents name the object\" invariant, so even if\na large blob is expressed as a CAT_TREE, its object name must still be\nwhat we would get by hashing '\"blob\" NUL '.\n\nA loose object file in \"cattree\" representation will not hash to the value\na naïve implementation would expect, and fsck_sha1() needs to be aware of\nit. I haven't thought things through in this area.\n\nFurther work would involve (no way exhaustive, of course):\n\n - Teach fsck and connectivity tools that objects that are reachable from\n any object (even a blob) that is represented as a CAT_TREE are needed\n and reachable by that object;\n\n - Teach pack-objects that anything that is represented as a CAT_TREE does\n not need to be deltified (the objects used as its representation would\n go through the usual deltification rules);\n\n - Teach unpack-objects to expand CAT_TREE representation into a \"cattree\"\n loose object.\n\n - Perhaps teach the attributes mechanism to lie to anybody who asks that\n any object in CAT_TREE representation is a binary file to trigger the\n \"we do not unnecessarily look at binary\" logic in \"git diff\" machinery.\n\n - Teach fast-import to write out CAT_TREE representation. This is\n probably the quickest and least-impact way to exploit existing support\n for large files in index_fd().\n\n - Update grep.c::grep_buffer() to take not but git_istream,\n and rewrite builtin/grep.c::grep_sha1() to use streaming interface.\n \n"},{"id":"178467","messageId":"m3wrbp44kr.fsf@localhost.localdomain","threadId":"28781","inReplyTo":"7v62j9veh3.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-10-28T13:41:57Z","receivedAt":"2011-10-28T13:41:57Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano writes:\n\n> When bit 4-6 encodes type 5, the first byte is used this way:\n> \n> - Bit 0-3 denotes the real \"extended\" representation type. Because types\n> 0-7 can already be encoded without using the extended format, we can\n> offset the type by 8 (i.e. if bit 0-3 says 3, it means representation\n> type 11 = 3 + 8);\n\nWhy not use third byte for that instead?\n\n-- \nJakub Narębski\n"},{"id":"178469","messageId":"7vsjmdta00.fsf@alter.siamese.dyndns.org","threadId":"28781","inReplyTo":"m3wrbp44kr.fsf@localhost.localdomain","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-28T15:24:15Z","receivedAt":"2011-10-28T15:24:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski writes:\n\n> Junio C Hamano writes:\n>\n>> When bit 4-6 encodes type 5, the first byte is used this way:\n>> \n>> - Bit 0-3 denotes the real \"extended\" representation type. Because types\n>> 0-7 can already be encoded without using the extended format, we can\n>> offset the type by 8 (i.e. if bit 0-3 says 3, it means representation\n>> type 11 = 3 + 8);\n>\n> Why not use third byte for that instead?\n\nIs it a good enough reason that there is no upside for doing so?\n"},{"id":"178472","messageId":"CAJo=hJt-YZcdxw+D=1S4haPmY-8-LLjXD=MvDGeWbdJ88_VOGw@mail.gmail.com","threadId":"28781","inReplyTo":"7v62j9veh3.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-10-28T15:44:56Z","receivedAt":"2011-10-28T15:44:56Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Thu, Oct 27, 2011 at 23:04, Junio C Hamano wrote:\n> In addition to four basic types (commit, tree, blob and tag), the pack\n> stream can encode a few other \"representation\" types, such as REF_DELTA\n> and OFS_DELTA. As we allocate 3 bits in the first byte for this purpose,\n> we do not have much room to add new representation types in place, but we\n> do have one value reserved for future expansion.\n\nWe have 2 values reserved, 0 and 5.\n\n> When bit 4-6 encodes type 5, the first byte is used this way:\n>\n>  - Bit 0-3 denotes the real \"extended\" representation type. Because types\n>   0-7 can already be encoded without using the extended format, we can\n>   offset the type by 8 (i.e. if bit 0-3 says 3, it means representation\n>   type 11 = 3 + 8);\n>\n>  - Bit 4-6 has the value \"5\";\n>\n>  - Bit 7 is used to signal if the _third_ byte needs to be read for larger\n>   size that cannot be represented with 8-bit.\n\nThis is very complicated. We don't need more complex logic in the pack\nencoding. We _especially_ do not need yet another variant of how to\nstore a variable length integer in the pack file. I'm sorry, but we\nalready have two different variants and this just adds a third. It is\nbeyond crazy.\n\nLast time (this is now years ago but whatever) Nico and I discussed\nadding a new type to packs it was for the alternate tree encoding in\n\"pack v4\". Trees happen so often that type code 5 is a good value to\nuse for these. Later you talked about using the extended type to store\na cattree blob thing, which would not appear nearly as often as a\nnormal directory listing type tree that was encoded using the pack v4\nstyle encoding... I think saving type 5 for a small frequently\noccurring type is a good thing.\n\n> As it is unlikely for us to pack things that do not need to record any\n> size, the second byte is always used in full to encode the low 8-bit of\n> the size.\n>\n> I haven't started using type=8 and upwards for anything yet, but because\n> we have only one \"future expansion\" value left, I want us to be extremely\n> careful in order to avoid painting us into a corner that we cannot get out\n> of, so I am sending this out early for a preliminary review.\n\nI would have said something more like:\n\nWhen bit 4-6 encodes \"0\", then:\n\n- Bit 0-3 and bit 7 are used normally to encode a variable length\n\"size\" integer. These may be 0 indicating no size information.\n\n- 2nd-nth bytes store remaining \"size\" information, if bit 7 was set.\n\n- The immediate next byte encodes the extended type. This type is\nstored using the OFS_DELTA offset varint encoding, and thus may be\nlarger than 256 if we ever need it to be.\n"},{"id":"178494","messageId":"alpine.LFD.2.02.1110290031540.30467@xanadu.home","threadId":"28781","inReplyTo":"CAJo=hJt-YZcdxw+D=1S4haPmY-8-LLjXD=MvDGeWbdJ88_VOGw@mail.gmail.com","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-10-28T22:48:03Z","receivedAt":"2011-10-28T22:48:03Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 28 Oct 2011, Shawn Pearce wrote:\n\n> On Thu, Oct 27, 2011 at 23:04, Junio C Hamano wrote:\n> > In addition to four basic types (commit, tree, blob and tag), the pack\n> > stream can encode a few other \"representation\" types, such as REF_DELTA\n> > and OFS_DELTA. As we allocate 3 bits in the first byte for this purpose,\n> > we do not have much room to add new representation types in place, but we\n> > do have one value reserved for future expansion.\n> \n> We have 2 values reserved, 0 and 5.\n> \n> > When bit 4-6 encodes type 5, the first byte is used this way:\n> >\n> >  - Bit 0-3 denotes the real \"extended\" representation type. Because types\n> >   0-7 can already be encoded without using the extended format, we can\n> >   offset the type by 8 (i.e. if bit 0-3 says 3, it means representation\n> >   type 11 = 3 + 8);\n> >\n> >  - Bit 4-6 has the value \"5\";\n> >\n> >  - Bit 7 is used to signal if the _third_ byte needs to be read for larger\n> >   size that cannot be represented with 8-bit.\n> \n> This is very complicated. We don't need more complex logic in the pack\n> encoding. We _especially_ do not need yet another variant of how to\n> store a variable length integer in the pack file. I'm sorry, but we\n> already have two different variants and this just adds a third. It is\n> beyond crazy.\n> \n> Last time (this is now years ago but whatever) Nico and I discussed\n> adding a new type to packs it was for the alternate tree encoding in\n> \"pack v4\". Trees happen so often that type code 5 is a good value to\n> use for these. Later you talked about using the extended type to store\n> a cattree blob thing, which would not appear nearly as often as a\n> normal directory listing type tree that was encoded using the pack v4\n> style encoding... I think saving type 5 for a small frequently\n> occurring type is a good thing.\n> \n> > As it is unlikely for us to pack things that do not need to record any\n> > size, the second byte is always used in full to encode the low 8-bit of\n> > the size.\n> >\n> > I haven't started using type=8 and upwards for anything yet, but because\n> > we have only one \"future expansion\" value left, I want us to be extremely\n> > careful in order to avoid painting us into a corner that we cannot get out\n> > of, so I am sending this out early for a preliminary review.\n> \n> I would have said something more like:\n> \n> When bit 4-6 encodes \"0\", then:\n> \n> - Bit 0-3 and bit 7 are used normally to encode a variable length\n> \"size\" integer. These may be 0 indicating no size information.\n> \n> - 2nd-nth bytes store remaining \"size\" information, if bit 7 was set.\n> \n> - The immediate next byte encodes the extended type. This type is\n> stored using the OFS_DELTA offset varint encoding, and thus may be\n> larger than 256 if we ever need it to be.\n\nI'd say it is just a byte. No encoding needed. Let's not be silly \nabout it. If we really have more than 255 object types one day (and I \nreally hope this will never happen) then the value 0 in that byte could \nindicate yet another extended object type encoding. But I truly hope \nwe'll have pack v9 or v10 by then and that we'll have obsoleted the \ncurrent 3-bit encoding completely at that point anyway.\n\nFor the record, I spent around 20 hours working on pack v4 while in the \nCaribbeans for a week last winter as I said I would. Maybe I'll repeat \nthe operation this year.\n\n\nNicolas\n"},{"id":"178495","messageId":"CAJo=hJsEzkFV9k8N+GAwWddmEZH8pQeJZrg_MXD72stbAW0ceQ@mail.gmail.com","threadId":"28781","inReplyTo":"alpine.LFD.2.02.1110290031540.30467@xanadu.home","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-10-28T23:07:43Z","receivedAt":"2011-10-28T23:07:43Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Fri, Oct 28, 2011 at 15:48, Nicolas Pitre wrote:\n> On Fri, 28 Oct 2011, Shawn Pearce wrote:\n>> - The immediate next byte encodes the extended type. This type is\n>> stored using the OFS_DELTA offset varint encoding, and thus may be\n>> larger than 256 if we ever need it to be.\n>\n> I'd say it is just a byte.  No encoding needed.  Let's not be silly\n> about it.  If we really have more than 255 object types one day (and I\n> really hope this will never happen) then the value 0 in that byte could\n> indicate yet another extended object type encoding.  But I truly hope\n> we'll have pack v9 or v10 by then and that we'll have obsoleted the\n> current 3-bit encoding completely at that point anyway.\n\nYes. I probably wouldn't code the parser to use a varint here. I would\nsay the extended types stored in this byte must be >= 8, and must be\n<= 127. Any values out of this range are unsupported and should be\nrejected. We can later reserve the right to set the high bit and\nswitch to the OFS_DELTA varint encoding if we need that many more\ntypes, and we explicitly define codes 0-7 as illegal if detected here\nin the extended byte field.\n"},{"id":"178496","messageId":"alpine.LFD.2.02.1110290128320.30467@xanadu.home","threadId":"28781","inReplyTo":"CAJo=hJsEzkFV9k8N+GAwWddmEZH8pQeJZrg_MXD72stbAW0ceQ@mail.gmail.com","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-10-28T23:30:55Z","receivedAt":"2011-10-28T23:30:55Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 28 Oct 2011, Shawn Pearce wrote:\n\n> On Fri, Oct 28, 2011 at 15:48, Nicolas Pitre wrote:\n> > On Fri, 28 Oct 2011, Shawn Pearce wrote:\n> >> - The immediate next byte encodes the extended type. This type is\n> >> stored using the OFS_DELTA offset varint encoding, and thus may be\n> >> larger than 256 if we ever need it to be.\n> >\n> > I'd say it is just a byte.  No encoding needed.  Let's not be silly\n> > about it.  If we really have more than 255 object types one day (and I\n> > really hope this will never happen) then the value 0 in that byte could\n> > indicate yet another extended object type encoding.  But I truly hope\n> > we'll have pack v9 or v10 by then and that we'll have obsoleted the\n> > current 3-bit encoding completely at that point anyway.\n> \n> Yes. I probably wouldn't code the parser to use a varint here. I would\n> say the extended types stored in this byte must be >= 8, and must be\n> <= 127. Any values out of this range are unsupported and should be\n> rejected. We can later reserve the right to set the high bit and\n> switch to the OFS_DELTA varint encoding if we need that many more\n> types, and we explicitly define codes 0-7 as illegal if detected here\n> in the extended byte field.\n\nI wouldn't go as far as rejecting codes 1-7 as illegal though, but I \notherwise agree with what you say.\n\n\nNicolas\n"},{"id":"178517","messageId":"CACsJy8Cz0R_s+VYRd+1wTTfbt_vH5dd3ALgZip0xn7rfYf6gpw@mail.gmail.com","threadId":"28781","inReplyTo":"7vwrbptzjm.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-10-30T05:40:27Z","receivedAt":"2011-10-30T05:40:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Oct 28, 2011 at 1:12 PM, Junio C Hamano wrote:\n> As people may be able to guess from the name, CAT_TREE is envisioned to\n> encode a large data (primarily of type \"blob\") by recording the object\n> name of a tree object and probably the total length, and would represent\n> the concatenation of all blobs contained in the tree object when the tree\n> is traversed in some fixed order (e.g. Avery's \"bup split\"). I am guessing\n> that the payload for CAT_TREE representation type will be:\n>\n>  - 20-byte object name for the top-level tree object;\n\nBecause all blobs in this tree object must be in a fixed order, and\nthey won't likely have meaningful names nor permission, should\nCAT_TREE payload is a SHA-1 sequence of all blobs (or cat-trees if we\nwant nested trees) instead? IOW the tree is integrated into cat-tree\nobject, not as a separate tree object.\n\n>  - type of the basic object (commit, tree, blob, or tag) it represents,\n>   even though it is unlikely that we would want to record such a large\n>   commit or tag that needs CAT_TREE representation;\n>\n>  - the total length of the basic object it represents, even though it is\n>   redundant (you could traverse and sum the sizes of blobs contained in\n>   the tree object), it would help sha1_object_info() and friends. This\n>   will be the \"some size\" I mentioned in the previous message for this\n>   representation type.\n\nNot sure if it's related to representation types, but is there any way\n(perhaps FLAT_BLOB type?) we can mark an object uncompressed, so we\ncan mmap() and access it directly?\n-- \nDuy\n"},{"id":"178522","messageId":"7v4nyrrm1w.fsf@alter.siamese.dyndns.org","threadId":"28781","inReplyTo":"CACsJy8Cz0R_s+VYRd+1wTTfbt_vH5dd3ALgZip0xn7rfYf6gpw@mail.gmail.com","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-30T07:11:23Z","receivedAt":"2011-10-30T07:11:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy writes:\n\n> Because all blobs in this tree object must be in a fixed order, and\n> they won't likely have meaningful names nor permission, should\n> CAT_TREE payload is a SHA-1 sequence of all blobs (or cat-trees if we\n> want nested trees) instead? IOW the tree is integrated into cat-tree\n> object, not as a separate tree object.\n\nI have no problem with that (I am not worried about minor details of the\nactual implementation of cat-tree yet).\n\n> Not sure if it's related to representation types, but is there any way\n> (perhaps FLAT_BLOB type?) we can mark an object uncompressed, so we\n> can mmap() and access it directly?\n\nIn pack? Loose? Both?\n\nWhat kind of payload and use case do you have in mind?\n"},{"id":"178523","messageId":"7vzkgjq7dc.fsf@alter.siamese.dyndns.org","threadId":"28781","inReplyTo":"7v4nyrrm1w.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-30T07:13:51Z","receivedAt":"2011-10-30T07:13:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Nguyen Thai Ngoc Duy writes:\n>\n>> Because all blobs in this tree object must be in a fixed order, and\n>> they won't likely have meaningful names nor permission, should\n>> CAT_TREE payload is a SHA-1 sequence of all blobs (or cat-trees if we\n>> want nested trees) instead? IOW the tree is integrated into cat-tree\n>> object, not as a separate tree object.\n>\n> I have no problem with that (I am not worried about minor details of the\n> actual implementation of cat-tree yet).\n\nSide note. It should be renamed \"split-object\" or something if we go the\nroute you suggest, as \"tree\"-ness of the actual representation is not\nessential.\n"},{"id":"178527","messageId":"CACsJy8A1QtHS2Y-O+NmO6MNDTcG_Wqc+Q1zk0RDGei=VTuxkPg@mail.gmail.com","threadId":"28781","inReplyTo":"7v4nyrrm1w.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] define the way new representation types are encoded in the pack","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-10-30T09:31:15Z","receivedAt":"2011-10-30T09:31:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Oct 30, 2011 at 2:11 PM, Junio C Hamano wrote:\n>> Not sure if it's related to representation types, but is there any way\n>> (perhaps FLAT_BLOB type?) we can mark an object uncompressed, so we\n>> can mmap() and access it directly?\n>\n> In pack? Loose? Both?\n>\n> What kind of payload and use case do you have in mind?\n\nHmm.. big files in general with partial file transfer (e.g. git\ntorrent). But if CAT_TREE is properly used, then all blob pieces\nshould be reasonably small and decompress time should not be a\nproblem. I think we won't need this. Sorry for the noise.\n-- \nDuy\n"}]}