git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 5/5] support writing uncompressed loose object

From
LYLiu Yubao <yubao.liu@gmail.com>
Date
Dec 3, 2008, 04:22 UTC
Message-ID
<49360996.40106@gmail.com>
In-Reply-To
<20081202160706.GN23984@spearce.org>
Shawn O. Pearce wrote:
Show 21 quoted lines
> Liu Yubao <yubao.liu@gmail.com> wrote:
>> Signed-off-by: Liu Yubao <yubao.liu@gmail.com>
> 
> IMHO, this needs more description in the commit message.
> 
>> diff --git a/sha1_file.c b/sha1_file.c
>> index 05a9fa3..053b564 100644
>> --- a/sha1_file.c
>> +++ b/sha1_file.c
>> @@ -2328,7 +2328,7 @@ static int create_tmpfile(char *buffer, size_t bufsiz, const char *filename)
>>  }
>>  
>>  static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
>> -			      void *buf, unsigned long len, time_t mtime)
>> +			      void *buf, unsigned long len, time_t mtime, int dont_deflate)
> 
> Passing this as an argument is pointless.  It should be a repository
> wide configuration option in core, so you can declare it a static and
> allow git_config to populate it.  Defaulting to 1 (no compression)
> like you do elsewhere in the patch isn't good.
> 
Aha, sorry again, I sent the patch series as separate topics by mistake.

I considered adding a configuration variable, the patch series are sent just to see whether the idea is worth.

Show 9 quoted lines
> I'm still against this file format change.  The series itself isn't
> that bad, and the buffer overflow catch in parse_sha1_header()
> may be something worthwhile fixing.  But I'm still not sold that
> introducing a new loose object format is worth it.
> 
> I'd rather use a binary header encoding like the new-style/in-pack
> format rather than the older style text headers.  Its faster to
> parse for one thing.
> 

The key point I suggest is to use *uncompressed* loose object, I didn't change the format of uncompressed loose object because I don't want to distract your attention and keep the patches small.

Show 7 quoted lines
> Your changes in the reading code cause a copy of the buffer we
> mmap()'d.  That sort of ruins your argument that this change is
> worthwhile because concurrent processes on the same host can mmap the
> same buffer and save memory.  We're still copying the buffer anyway.
> I probably should have commented on that in patch 4/5, but I just
> realized it, so I'm saying it here.
> 
Yes, I mentioned it in the cover letter(sigh, sorry!)

I didn't use the mapped buffer directly because other functions required a null terminated buffer to parse data part of loose object. It can be fixed but I don't want to make the patches too big.

The two big pros of uncompressed loose object are:

*) avoid compressing and uncompressing loose objects (I have implemented it) *) use memory mapped loose object directly (I havn't implemented it)

Thank you for reviewing my patches, seems the idea to use uncompressed loose object isn't attractive enough, I will keep the patches locally.

Best regards,
Liu Yubao
Previous: Shawn O. PearceNext: Liu Yubao
Message 21 of 27 in “two questions about the format of loose object”
  1. Liu YubaoDec 1, 2008
  2. Junio C HamanoDec 1, 2008
  3. Liu YubaoDec 1, 2008
  4. Jakub NarebskiDec 1, 2008
  5. Liu YubaoDec 2, 2008
  6. Shawn O. PearceDec 1, 2008
  7. Liu YubaoDec 2, 2008
  8. 0/5 support reading and writing uncompressed loose objectLiu Yubao, Dec 2, 2008
  9. 1/5 avoid parse_sha1_header() accessing memory out of boundLiu Yubao, Dec 2, 2008
  10. Shawn O. PearceDec 2, 2008
  11. Liu YubaoDec 3, 2008
  12. 2/5 don't die immediately when convert an invalid type nameLiu Yubao, Dec 2, 2008
  13. 3/5 optimize parse_sha1_header() a little by detecting object typeLiu Yubao, Dec 2, 2008
  14. Shawn O. PearceDec 2, 2008
  15. Liu YubaoDec 3, 2008
  16. 4/5 support reading uncompressed loose objectLiu Yubao, Dec 2, 2008
  17. Shawn O. PearceDec 2, 2008
  18. Liu YubaoDec 3, 2008
  19. 5/5 support writing uncompressed loose objectLiu Yubao, Dec 2, 2008
  20. Shawn O. PearceDec 2, 2008
  21. Liu YubaoDec 3, 2008
  22. Liu YubaoDec 2, 2008
  23. Nick AndrewDec 1, 2008
  24. Liu YubaoDec 2, 2008
  25. Shawn O. PearceDec 1, 2008
  26. Liu YubaoDec 2, 2008
  27. Nicolas PitreDec 4, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.