Re: [PATCH] RFC: git lazy clone proof-of-concept
- From
Jon Smirl <jonsmirl@gmail.com>
- Date
- Feb 12, 2008, 21:25 UTC
- Message-ID
- <9e4733910802121325p7ce6b58axae71f698f76dbfd2@mail.gmail.com>
- In-Reply-To
- <alpine.LSU.1.00.0802122036150.3870@racer.site>
On 2/12/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
Show 21 quoted lines
> Hi, > > On Sun, 10 Feb 2008, Johannes Schindelin wrote: > > > $ /usr/bin/time git repack -a -d -f --window=150 --depth=150 > > Counting objects: 2477715, done. > > Compressing objects: 19% (481551/2411764) > > Compressing objects: 19% (482333/2411764) > > fatal: Out of memory, malloc failed411764) > > Command exited with non-zero status 1 > > 7118.37user 54.15system 2:01:44elapsed 98%CPU (0avgtext+0avgdata > > 0maxresident)k > > 0inputs+0outputs (29834major+17122977minor)pagefaults 0swaps > > I made the window much smaller (512 megabyte), and it still runs, after 27 > hours: > > Compressing objects: 20% (484132/2411764) > > However, it seems that it only worked on about 4000 objects in the last > 20(!) hours.
I found that out with gcc. 95% went down in no time and the last 5% took two hours. The 5% that got stuck were chains with 2000+ entries.
The neat thing about the multithread code is that it will keep splitting the work load. That lets all of the easy deltas finish and not get stuck behind the problem objects.
With quad core on gcc one core would get stuck on the problem objects. The other three would finish their list and start splitting the problem list. This effectively sorts the problems to the end of the work load. By printing the object hash out as they are completed you can easily identify the problem objects. If I recall right on gcc the problem was a configure file that had 2000 entries in its delta chain. That one delta chain took over an hour to process.
Could there be an N squared type problem when 2000 entry delta chains are encountered? Maybe something that just isn't noticeable when depth/window=50. Has testing been done with really long object chains to make sure that only the minimal amount of work is being done? It seems like something is breaking down when the chain length exceeds the window size.
So, the first 19% were relatively quick. The next percent
Show 10 quoted lines
> not at all. > > Will keep you posted, > Dscho > > - > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >
-- Jon Smirl jonsmirl@gmail.com