threads / discuss / 51719

[Question] clone performance

Subject: [Question] clone performance

## tl;dr

5 messages between Aug 24, 2019 and Aug 26, 2019.

replies: 4people: 4as markdown or json

randall.s.becker@rogers.com· Aug 24, 2019, 01:59 UTC · lore

Hi All,

I'm trying to answer a question for a customer on clone performance. They are doing at least 2-3 clones a day, of repositories with about 2500 files and 10Gb of content. This is stressing the file system. I have tried to convince them that their process is not reasonable and should stick with existing clones, using branch checkout rather that re=cloning for each feature branch. Sadly, I have not been successful - not for a lack of trying. Is there any way to improve raw clone performance in a situation like this, where status really doesn't matter, because the clone's life span is under 48 hours.

TIA, Randall

Bryan Turner· Aug 24, 2019, 21:00 UTC · re: randall.s.becker@rogers.com · lore

Re: [Question] clone performance

On Fri, Aug 23, 2019 at 6:59 PM <randall.s.becker@rogers.com> wrote:

Show 6 quoted lines

>
> Hi All,
>
> I'm trying to answer a question for a customer on clone performance. They
> are doing at least 2-3 clones a day, of repositories with about 2500 files
> and 10Gb of content. This is stressing the file system.

Can you go into a bit more detail about what "stress" means? Using too much disk space? Too many IOPS reading/packing? Since you specifically called out the filesystem, does that mean the CPU/memory usage is acceptable?

Depending on how well-packed the repository is, Git will reuse a lot of the existing pack (and a "perfectly" packed repository can achieve complete reuse, with no "Compressing objects" phase at all). Delta islands[1] can help increase reuse and reduce the need for on-the-fly compression, if the repository includes a lot of refs that aren't generally cloned.

Another relatively recent addition is uploadpack.packobjectshook[2], which can simplify caching of packfiles so they can be reused on subsequent requests. Whether or not this will be beneficial is likely to be influenced by how many times the exact same commits are cloned and how much extra disk space is available for storing cached packs.

Not sure if any of this is helpful, but I hope it will be! Bryan

[1] https://git-scm.com/docs/git-pack-objects#_delta_islands [2] https://git-scm.com/docs/git-config#Documentation/git-config.txt-uploadpackpackObjectsHook

Show 12 quoted lines

> I have tried to
> convince them that their process is not reasonable and should stick with
> existing clones, using branch checkout rather that re=cloning for each
> feature branch. Sadly, I have not been successful - not for a lack of
> trying. Is there any way to improve raw clone performance in a situation
> like this, where status really doesn't matter, because the clone's life span
> is under 48 hours.
>
> TIA,
> Randall
>
>

randall.s.becker@rogers.com· Aug 26, 2019, 14:16 UTC · re: Bryan Turner · lore

RE: [Question] clone performance

On August 24, 2019 5:00 PM, Bryan Turner wrote:

Show 12 quoted lines

> On Fri, Aug 23, 2019 at 6:59 PM <randall.s.becker@rogers.com> wrote:
> >
> > Hi All,
> >
> > I'm trying to answer a question for a customer on clone performance.
> > They are doing at least 2-3 clones a day, of repositories with about
> > 2500 files and 10Gb of content. This is stressing the file system.
> 
> Can you go into a bit more detail about what "stress" means? Using too
> much disk space? Too many IOPS reading/packing? Since you specifically
> called out the filesystem, does that mean the CPU/memory usage is
> acceptable?

The upstream is BitBucket, which does a gc frequently. I'm not sure any of this is relating to the pack structure. Git is spending most of its time writing the large number of large files into the working directory - it is stress mostly the disk, with a bit on the CPU (neither is acceptable to the customer). I am really unsure there is any way to make things better. The core issue is that the customer insists on doing a clone for every feature branch instead of using pull/checkout. I have been unable to change their mind - to this point anyway.

We are going to be setting up a detailed performance analysis that may lead to some data the git team can use.

Regards, Randall

Jeff King· Aug 26, 2019, 18:21 UTC · re: randall.s.becker@rogers.com · lore

Re: [Question] clone performance

On Mon, Aug 26, 2019 at 10:16:48AM -0400, randall.s.becker@rogers.com wrote:

Show 23 quoted lines

> On August 24, 2019 5:00 PM, Bryan Turner wrote:
> > On Fri, Aug 23, 2019 at 6:59 PM <randall.s.becker@rogers.com> wrote:
> > >
> > > Hi All,
> > >
> > > I'm trying to answer a question for a customer on clone performance.
> > > They are doing at least 2-3 clones a day, of repositories with about
> > > 2500 files and 10Gb of content. This is stressing the file system.
> > 
> > Can you go into a bit more detail about what "stress" means? Using too
> > much disk space? Too many IOPS reading/packing? Since you specifically
> > called out the filesystem, does that mean the CPU/memory usage is
> > acceptable?
> 
> The upstream is BitBucket, which does a gc frequently. I'm not sure
> any of this is relating to the pack structure. Git is spending most of
> its time writing the large number of large files into the working
> directory - it is stress mostly the disk, with a bit on the CPU
> (neither is acceptable to the customer). I am really unsure there is
> any way to make things better. The core issue is that the customer
> insists on doing a clone for every feature branch instead of using
> pull/checkout. I have been unable to change their mind - to this point
> anyway.

Yeah, at the point of checkout there's basically no impact from anything the server is doing or has done (technically it could make things worse for you by returning a pack with absurdly long delta chains or something, but that would be CPU and not disk stress).

I doubt there's much to optimize in Git here. It's literally just writing files to disk as quickly as it can, and it sounds like disk performance is your bottleneck.

-Peff

Elijah Newren· Aug 26, 2019, 19:27 UTC · re: Jeff King · lore

Re: [Question] clone performance

On Mon, Aug 26, 2019 at 12:04 PM Jeff King <peff@peff.net> wrote:

Show 35 quoted lines

>
> On Mon, Aug 26, 2019 at 10:16:48AM -0400, randall.s.becker@rogers.com wrote:
>
> > On August 24, 2019 5:00 PM, Bryan Turner wrote:
> > > On Fri, Aug 23, 2019 at 6:59 PM <randall.s.becker@rogers.com> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I'm trying to answer a question for a customer on clone performance.
> > > > They are doing at least 2-3 clones a day, of repositories with about
> > > > 2500 files and 10Gb of content. This is stressing the file system.
> > >
> > > Can you go into a bit more detail about what "stress" means? Using too
> > > much disk space? Too many IOPS reading/packing? Since you specifically
> > > called out the filesystem, does that mean the CPU/memory usage is
> > > acceptable?
> >
> > The upstream is BitBucket, which does a gc frequently. I'm not sure
> > any of this is relating to the pack structure. Git is spending most of
> > its time writing the large number of large files into the working
> > directory - it is stress mostly the disk, with a bit on the CPU
> > (neither is acceptable to the customer). I am really unsure there is
> > any way to make things better. The core issue is that the customer
> > insists on doing a clone for every feature branch instead of using
> > pull/checkout. I have been unable to change their mind - to this point
> > anyway.
>
> Yeah, at the point of checkout there's basically no impact from anything
> the server is doing or has done (technically it could make things worse
> for you by returning a pack with absurdly long delta chains or
> something, but that would be CPU and not disk stress).
>
> I doubt there's much to optimize in Git here. It's literally just
> writing files to disk as quickly as it can, and it sounds like disk
> performance is your bottleneck.

Well, if it's just checkout, Stolee's sparse-checkout series he just posted may be of interest to them...once it's polished up and included in git, of course.