Re: [PATCHv1 3/3] git-p4: auto-size the block

> git-p4 originally would fetch changes in one query. On large repos this
> could fail because of the limits that Perforce imposes on the number of
> items returned and the number of queries in the database.
>
> To fix this, git-p4 learned to query changes in blocks of 512 changes,
> However, this can be very slow - if you have a few million changes,
> with each chunk taking about a second, it can be an hour or so.
>
> Although it's possible to tune this value manually with the
> "--changes-block-size" option, it's far from obvious to ordinary users
> that this is what needs doing.
>
> This change alters the block size dynamically by looking for the
> specific error messages returned from the Perforce server, and reducing
> the block size if the error is seen, either to the limit reported by the
> server, or to half the current block size.
>
> That means we can start out with a very large block size, and then let
> it automatically drop down to a value that works without error, while
> still failing correctly if some other error occurs.
>
> Signed-off-by: Luke Diamand <luke@diamand.org>
> ---
> diff --git a/git-p4.py b/git-p4.py
> @@ -48,7 +48,8 @@ def __str__(self):
>  # Grab changes in blocks of this many revisions, unless otherwise requested
> -defaultBlockSize = 512
> +# The code should now automatically reduce the block size if it is required
> +defaultBlockSize = 1<<20

> @@ -983,10 +985,24 @@ def p4ChangesForPaths(depotPaths, changeRange, requestedBlockSize):
> +        try:
> +            result = p4CmdList(cmd, errors_as_exceptions=True)
> +        except P4RequestSizeException as e:
> +            if not block_size:
> +                block_size = e.limit
> +            elif block_size > e.limit:
> +                block_size = e.limit
> +            else:
> +                block_size = max(2, block_size // 2)
> +
> +            if verbose: print("block size error, retry with block size {0}".format(block_size))

> diff --git a/t/t9818-git-p4-block.sh b/t/t9818-git-p4-block.sh
> @@ -129,6 +129,7 @@ test_expect_success 'Create a repo with multiple depot paths' '
> +test_expect_success 'Clone repo with self-sizing block size' '
> +       test_when_finished cleanup_git &&
> +       git p4 clone --changes-block-size=1000000 //depot@all --destination="$git" &&
> +       (
> +               cd "$git" &&
> +               git log --oneline > log &&
> +               test_line_count \> 10 log
> +       )
> +'