git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] git.txt: document limitations on non-typical repos (and hints)

From
Junio C Hamano <gitster@pobox.com>
Date
Oct 6, 2010, 16:32 UTC
Message-ID
<7vsk0j46zk.fsf@alter.siamese.dyndns.org>
In-Reply-To
<4cac8659.0541730a.0ef0.3ff9@mx.google.com>
pclouds@gmail.com writes:
Show 27 quoted lines
> From: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
>
> ---
>  Revised version. I dropped shallow clone because it does not really
>  relate to performance.
>
>  Documentation/git.txt |   41 +++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 41 insertions(+), 0 deletions(-)
>
> diff --git a/Documentation/git.txt b/Documentation/git.txt
> index dd57bdc..129947f 100644
> --- a/Documentation/git.txt
> +++ b/Documentation/git.txt
> @@ -729,6 +729,47 @@ The index is also capable of storing multiple entries (called "stages")
>  for a given pathname.  These stages are used to hold the various
>  unmerged version of a file when a merge is in progress.
>  
> +Performance concerns
> +--------------------
> +
> +Git is written with performance in mind and it works extremely well
> +with its typical repositories (i.e. source code repositories, with
> +a moderate number of small text files, possibly with long history).
> +Non-typical repositories (a lot of files, or very large files...)
> +may experience mild performance degradation. This section describes
> +how Git behaves in such repositories and how to reduce impact.
> +

I have seen this "mild" suggested in the discussion, but do we want any adjective here? The runtime for, say, "git log" from the tip to the root obviously would grow proportionally to the length of the history, i.e. the number of records you would want to see, and it may not be "mild" if your history is very deep. Same for the runtime for "git diff" in a wide project with many changed paths.

More importantly, what is "degradation"? It is not a degradation if "git log" took 100x as long for a project with 100k commits compared to a similar project with 1k commits.

If you do not have enough core to hold the part of the ancestry graph that is involved to compute "git log A..B" to show a gazillion commits, it will eat into the swap, take a lot more time than it takes "git log B" to show the same number of commits. That _is_ degradation, and I suspect it won't be mild at all.

> +For repositories with a large number of files (~50k files or more),
How did you come up with this 50k number?
> +but you only need a few of them present in working tree, you can use
> +sparse checkout (see linkgit:git-read-tree[1], section 'Sparse
> +checkout').

Is "sparse checkout" a real feature that has been made usable by mere mortals, battle tested, and shown to be reliable?

It feels funny that we have to refer to the documentation of plumbing read-tree when the key verb in this paragraph is "checkout". With the current documentation set, you can follow read-tree page that mentions some magic called skip-worktree-bit, get tempted to jump to update-index page and get lost in the implementation details of the feature, which is irrelevant to the end user. If you resisted the temptation and keep reading read-tree page, you see the description of info/sparse-checkout to learn how to control the feature, but it does not come with an easy-to-follow example. A few concrete suggestions to "Sparse checkout" section in read-tree:

    - Move the section to a separate file and include it in read-tree
      page, so that we can include it later in checkout page;
    - Drop the first paragraph;
    - Move the second and third paragraph, that still describe the
      machinery more than the usage, much later in the section;
    - Start the section with the description of info/sparse-checkout; the
      first sentence ("while ... is usually used") need to be rewritten,
      because (1) it is not a complete sentence and grammatically
      incorrect, and (2) it reads as if you will say an alternative file
      can be used instead of info/sparse-checkout, which is not what you
      wanted to do; perhaps "$GIT_DIR/info/sparse-checkout is used to
      specify which paths are to be (and not to be) checked out. It lists
      glob patterns to match paths to be checked out. Prefix the pattern
      with a '!' to specify a pattern to match paths not to be checked
      out.  Note: a bug in the implementation requires you to end a
      pattern with a trailing slash to match a directory".
    - Show examples; not just the samples of how contents of that control
      file looks like, but also with a concrete command sequence (e.g. (1)
      run "git clone -n", (2) edit info/sparse-checkout to contain this,
      (3) run "git checkout", (4) here is how to widen/narrow the sparse
      checkout--first edit info/sparse-checkout to look like this and then
      run "git xxx" to match the updated definition, etc.).
    - Drop BUGS section from read-tree documentation (but see Note: above
      in my example); the bug mentioned there is not a bug of read-tree,
      but is a bug in the sparse-checkout feature.

I think the suggestion to use Sparse checkout in git(1)---i.e. your patch we are discussing here, is a bit premature before the above happens.

Show 5 quoted lines
> +... If you need all of them present in working tree, but you
> +know in advance only a few of them may be modified, please consider
> +using assume-unchanged bit (see linkgit:git-update-index[1]).
> +... The following commands are
> +however known to do full index refresh in some cases:
It is "need to", not "are known to", isn't it?
> +Some commands need entire file content in memory to process.
> +Files that have size a significant portion of physical RAM may
> +affect performance. You may want to avoid using the following
> +commands if possible on such large files:

"If possible" is not a good excuse. How would one _avoid_ checkout of a file if one wants to use it? You can't. Similarly to "diff". This advice is pretty much useless, isn't it? It's not much better than saying "if your machine has too little RAM, things will get slow---deal with it".

Show 13 quoted lines
> +* All checkout commands (linkgit:git-checkout[1],
> +  linkgit:git-checkout-index[1], linkgit:git-read-tree[1],
> +  linkgit:git-clone[1]...)
> +* All diff-related commands (linkgit:git-diff[1],
> +  linkgit:git-log[1] with diff, linkgit:git-show[1] on commits...)
> +* All commands that need file conversion processing (see
> +  linkgit:gitattributes[5])
> +
>  Authors
>  -------
>  * git's founding father is Linus Torvalds <torvalds@osdl.org>.
> -- 
> 1.7.0.2.445.gcbdb3
Previous: pclouds@gmail.comNext: Nguyen Thai Ngoc Duy
Message 7 of 8 in “git.txt: document limitations on non-typical repos (and hints)”
  1. git.txt: document limitations on non-typical repos (and hints)Nguyễn Thái Ngọc Duy, Oct 5, 2010
  2. Alex RiesenOct 5, 2010
  3. Chris PackhamOct 5, 2010
  4. Nguyen Thai Ngoc DuyOct 5, 2010
  5. git.txt: document limitations on non-typical repos (and hints)Nguyễn Thái Ngọc Duy, Oct 6, 2010
  6. git.txt: document limitations on non-typical repos (and hints)pclouds@gmail.com, Oct 6, 2010
  7. Junio C HamanoOct 6, 2010
  8. Nguyen Thai Ngoc DuyOct 7, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.