git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Packfile can't be mapped

From
Shawn Pearce <spearce@spearce.org>
Date
Aug 28, 2006, 05:33 UTC
Message-ID
<20060828053301.GA25285@spearce.org>
In-Reply-To
<Pine.LNX.4.64.0608280014190.3683@localhost.localdomain>
Nicolas Pitre <nico@cam.org> wrote:
Show 16 quoted lines
> On Sun, 27 Aug 2006, Shawn Pearce wrote:
> 
> > I'm going to try to get tree deltas written to the pack sometime this
> > week. That should compact this intermediate pack down to something
> > that git-pack-objects would be able to successfully mmap into a
> > 32 bit address space.  A complete repack with no delta reuse will
> > hopefully generate a pack closer to 400 MB in size.  But I know
> > Jon would like to get that pack even smaller.  :)
> 
> One thing to consider in your code (if you didn't implement that 
> already) is to _not_ attempt any delta on any object whose size is 
> smaller than 50 bytes, and then limit the maximum delta size to 
> object_size/2 - 20 (use that for the last argument to diff-delta() and 
> store the undeltified object when diff-delta returns NULL).  This way 
> you'll avoid creating delta objects that are most likely to end up being 
> _larger_ than the undeltified object.

I haven't tried this. Should be trivial to implement. Thanks for the suggestion.

Show 8 quoted lines
> > I should point out that the input stream to fast-import was 20 GB
> > (completely decompressed revisions from RCS) plus all commit data.
> > The original CVS ,v files are around 3 GB.  An archive .tar.gz'ing
> > the ,v files is around 550 MB.  Going to only 1.7 GB without tree
> > or commit deltas is certainly pretty good.  :)
> 
> Good job indeed.  Oh and you probably should not bother trying to 
> deltify commit objects at all since that would be a waste of time.

I wasn't going to bother even trying to delta the commits. In this import the 200k commits isn't a very large percentage of the data. As I'm sure you are well aware its pretty much a waste time to try with the commits, especially with an "intermediate" pack such as this one.

-- 
Shawn.
Previous: Nicolas PitreNext: Shawn Pearce
Message 12 of 16 in “Packfile can't be mapped”
  1. Jon SmirlAug 28, 2006
  2. Shawn PearceAug 28, 2006
  3. Nicolas PitreAug 28, 2006
  4. Linus TorvaldsAug 28, 2006
  5. Shawn PearceAug 28, 2006
  6. Jon SmirlAug 28, 2006
  7. Nicolas PitreAug 28, 2006
  8. Jon SmirlAug 28, 2006
  9. Nicolas PitreAug 28, 2006
  10. Shawn PearceAug 28, 2006
  11. Nicolas PitreAug 28, 2006
  12. Shawn PearceAug 28, 2006
  13. Shawn PearceAug 28, 2006
  14. Nicolas PitreAug 28, 2006
  15. Shawn PearceAug 29, 2006
  16. Shawn PearceAug 29, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.