Re: git-p4 fails when cloning a p4 depo.

From

BSBenjamin Sergeant <bsergean@gmail.com>

Date

Jun 9, 2007, 00:32 UTC

Message-ID

<1621f9fa0706081732k7a31782cv26f3295245057b6f@mail.gmail.com>

In-Reply-To

<4669E73F.2040702@xs4all.nl>

On 6/8/07, Han-Wen Nienhuys <hanwen@xs4all.nl> wrote:

> Benjamin Sergeant escreveu:
>
> > So are you saying that in the old days, git-p4 was importing the p4
> > depo in small slices to not overkill the process memory (in case the
> > depo is big) ?
>
> no, in the "old days" git-p4 used a separate p4 invocation for each file.
>

Benjamin.

[bsergean@flanders fast-export]$ git format-patch -k -m --stdout origin
From 45f2dbdb9a8c0b3beb007ae892613cdc4afab80a Mon Sep 17 00:00:00 2001
From: Benjamin Sergeant <bsergean@flanders.(none)>
Date: Fri, 8 Jun 2007 09:58:57 -0700
Subject: Split p4 print call into multiple call to not exceed the
command line lenght maximum

---
 git-p4 |   21 ++++++++++++++++++---
 1 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/git-p4 b/git-p4
index 36fe69a..906b193 100755
--- a/git-p4
+++ b/git-p4
@@ -703,9 +703,22 @@ class P4Sync(Command):
         if not files:
             return

-        filedata = p4CmdList('print %s' % ' '.join(['"%s#%s"' % (f['path'],
-                                                                 f['rev'])
-                                                    for f in files]))
+        # We cannot put all the files on the command line
+        # OS have limitations on the max lenght of arguments
+        # POSIX says it's 4096 bytes, default for Linux seems to be 130 K.
+        # and all OS from the table below seems to be higher than POSIX.
+        # See http://www.in-ulm.de/~mascheck/various/argmax/
+        chunk = ''
+        filedata = []
+        for i in xrange(len(files)):
+            f = files[i]
+            chunk += '"%s#%s" ' % (f['path'], f['rev'])
+            if len(chunk) > 4000 or i == len(files)-1:
+                data = p4CmdList('print %s' % chunk)
+                if "p4ExitCode" in data[0]:
+                    die("Problems executing p4. Error: [%d]." %
(data[0]['p4ExitCode']));
+                filedata.extend(data)
+                chunk = ''

         j = 0;
         contents = {}
@@ -1486,3 +1499,5 @@ def main():

 if __name__ == '__main__':
     main()
+
+# vim: set filetype=python sts=4 sw=4 et si :
--
1.5.0.4


>From dd9975708433efeec37b608755f54fbeaedf0f3f Mon Sep 17 00:00:00 2001
From: Benjamin Sergeant <bsergean@flanders.(none)>
Date: Fri, 8 Jun 2007 10:20:39 -0700
Subject: Use os.sysconf('SC_ARG_MAX') to retrieve the max value, and
build the string using join (faster)

---
 git-p4 |   17 ++++++++++-------
 1 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/git-p4 b/git-p4
index 906b193..8dc1963 100755
--- a/git-p4
+++ b/git-p4
@@ -705,20 +705,23 @@ class P4Sync(Command):

         # We cannot put all the files on the command line
         # OS have limitations on the max lenght of arguments
-        # POSIX says it's 4096 bytes, default for Linux seems to be 130 K.
-        # and all OS from the table below seems to be higher than POSIX.
         # See http://www.in-ulm.de/~mascheck/various/argmax/
-        chunk = ''
+        chunks = []
+        chunkLenght = 0
         filedata = []
+        maxlenght = max(int(os.sysconf('SC_ARG_MAX') * 0.90), 4000)
+        print maxlenght
         for i in xrange(len(files)):
             f = files[i]
-            chunk += '"%s#%s" ' % (f['path'], f['rev'])
-            if len(chunk) > 4000 or i == len(files)-1:
-                data = p4CmdList('print %s' % chunk)
+            chunkLenght += len(f['path']) + len(f['rev'])
+            chunks.append('"%s#%s" ' % (f['path'], f['rev']))
+            if chunkLenght > maxlenght or i == len(files)-1:
+                data = p4CmdList('print %s' % ' '.join(chunks))
                 if "p4ExitCode" in data[0]:
                     die("Problems executing p4. Error: [%d]." %
(data[0]['p4ExitCode']));
                 filedata.extend(data)
-                chunk = ''
+                chunks = []
+                chunkLenght = 0

         j = 0;
         contents = {}
--
1.5.0.4