{"thread":{"id":"8505","subject":"git-p4 fails when cloning a p4 depo.","startedAt":"2007-06-08T16:41:35Z","lastAt":"2007-06-17T16:09:00Z","messageCount":14,"participants":["Benjamin Sergeant","Scott Lamb","Simon Hausmann","Han-Wen Nienhuys"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"44359","messageId":"1621f9fa0706080941k67d2878dud8cf06436c67aea0@mail.gmail.com","threadId":"8505","inReplyTo":null,"subject":"git-p4 fails when cloning a p4 depo.","fromName":"Benjamin Sergeant","fromEmail":"bsergean@gmail.com","sentAt":"2007-06-08T16:41:35Z","receivedAt":"2007-06-08T16:41:35Z","isPatch":false,"sender":{"key":"bsergean@gmail.com","avatar":null},"body":"I attached a lame patch to die without showing the Python Traceback,\nbut I'd rather succeed :)\nMaybe there is a different mailing list for git-p4. If there is tell\nme and I'll post there.\n\nBenjamin.\n\n[bsergean@flanders sandbox]$ rm -rf dev ; git-p4 clone\n//Work/Users/Capture3D/A3D810/pdfl/Common/a3d/dev\nImporting from //Work/Users/Capture3D/A3D810/pdfl/Common/a3d/dev into dev\nInitialized empty Git repository in .git/\nDoing initial import of\n//Work/Users/Capture3D/A3D810/pdfl/Common/a3d/dev/ from revision #head\n[{'p4ExitCode': 32512}]\nTraceback (most recent call last):\n File \"/home/bsergean/src/fast-export/git-p4\", line 1489, in \n main()\n File \"/home/bsergean/src/fast-export/git-p4\", line 1484, in main\n if not cmd.run(args):\n File \"/home/bsergean/src/fast-export/git-p4\", line 1395, in run\n if not P4Sync.run(self, depotPaths):\n File \"/home/bsergean/src/fast-export/git-p4\", line 1203, in run\n self.commit(details, self.extractFilesFromCommit(details),\nself.branch, self.depotPaths)\n File \"/home/bsergean/src/fast-export/git-p4\", line 744, in commit\n self.readP4Files(files)\n File \"/home/bsergean/src/fast-export/git-p4\", line 722, in readP4Files\n contents[stat['depotFile']] = text\nKeyError: 'depotFile'\n\n\ndiff --git a/git-p4 b/git-p4\nindex 36fe69a..3e1a878 100755\n--- a/git-p4\n+++ b/git-p4\n@@ -707,6 +707,9 @@ class P4Sync(Command):\n f['rev'])\n for f in files]))\n \n+\tif \"p4ExitCode\" in filedata[0]:\n+ die(\"Problems executing p4\");\n+\n j = 0;\n contents = {}\n while j < len(filedata):\n"},{"id":"44364","messageId":"1621f9fa0706081113w7bb765ebx74f03a7407b753cb@mail.gmail.com","threadId":"8505","inReplyTo":"1621f9fa0706080941k67d2878dud8cf06436c67aea0@mail.gmail.com","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Benjamin Sergeant","fromEmail":"bsergean@gmail.com","sentAt":"2007-06-08T18:13:55Z","receivedAt":"2007-06-08T18:13:55Z","isPatch":false,"sender":{"key":"bsergean@gmail.com","avatar":null},"body":"A perforce command with all the files in the repo is generated to get\nall the file content.\nHere is a patch to break it into multiple successive perforce command\nwho uses 4K of parameter max, and collect the output for later.\n\nIt works, but not for big depos, because the whole perforce depo\ncontent is stored in memory in P4Sync.run(), and it looks like mine is\nbigger than 2 Gigs, so I had to kill the process.\n\n\ndiff --git a/git-p4 b/git-p4\nindex 36fe69a..906b193 100755\n--- a/git-p4\n+++ b/git-p4\n@@ -703,9 +703,22 @@ class P4Sync(Command):\n if not files:\n return\n\n- filedata = p4CmdList('print %s' % ' '.join(['\"%s#%s\"' % (f['path'],\n- f['rev'])\n- for f in files]))\n+ # We cannot put all the files on the command line\n+ # OS have limitations on the max lenght of arguments\n+ # POSIX says it's 4096 bytes, default for Linux seems to be 130 K.\n+ # and all OS from the table below seems to be higher than POSIX.\n+ # See http://www.in-ulm.de/~mascheck/various/argmax/\n+ chunk = ''\n+ filedata = []\n+ for i in xrange(len(files)):\n+ f = files[i]\n+ chunk += '\"%s#%s\" ' % (f['path'], f['rev'])\n+ if len(chunk) > 4000 or i == len(files)-1:\n+ data = p4CmdList('print %s' % chunk)\n+ if \"p4ExitCode\" in data[0]:\n+ die(\"Problems executing p4. Error: [%d].\" %\n(data[0]['p4ExitCode']));\n+ filedata.extend(data)\n+ chunk = ''\n\n j = 0;\n contents = {}\n@@ -1486,3 +1499,5 @@ def main():\n\n if __name__ == '__main__':\n main()\n+\n+# vim: set filetype=python sts=4 sw=4 et si :\n\n\n\n\n\n\n\n\n\n\n\nOn 6/8/07, Benjamin Sergeant wrote:\n> I attached a lame patch to die without showing the Python Traceback,\n> but I'd rather succeed :)\n> Maybe there is a different mailing list for git-p4. If there is tell\n> me and I'll post there.\n>\n> Benjamin.\n>\n> [bsergean@flanders sandbox]$ rm -rf dev ; git-p4 clone\n> //Work/Users/Capture3D/A3D810/pdfl/Common/a3d/dev\n> Importing from //Work/Users/Capture3D/A3D810/pdfl/Common/a3d/dev into dev\n> Initialized empty Git repository in .git/\n> Doing initial import of\n> //Work/Users/Capture3D/A3D810/pdfl/Common/a3d/dev/ from revision #head\n> [{'p4ExitCode': 32512}]\n> Traceback (most recent call last):\n> File \"/home/bsergean/src/fast-export/git-p4\", line 1489, in \n> main()\n> File \"/home/bsergean/src/fast-export/git-p4\", line 1484, in main\n> if not cmd.run(args):\n> File \"/home/bsergean/src/fast-export/git-p4\", line 1395, in run\n> if not P4Sync.run(self, depotPaths):\n> File \"/home/bsergean/src/fast-export/git-p4\", line 1203, in run\n> self.commit(details, self.extractFilesFromCommit(details),\n> self.branch, self.depotPaths)\n> File \"/home/bsergean/src/fast-export/git-p4\", line 744, in commit\n> self.readP4Files(files)\n> File \"/home/bsergean/src/fast-export/git-p4\", line 722, in readP4Files\n> contents[stat['depotFile']] = text\n> KeyError: 'depotFile'\n>\n>\n"},{"id":"44379","messageId":"4669CAB4.5080507@slamb.org","threadId":"8505","inReplyTo":"1621f9fa0706081113w7bb765ebx74f03a7407b753cb@mail.gmail.com","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Scott Lamb","fromEmail":"slamb@slamb.org","sentAt":"2007-06-08T21:31:32Z","receivedAt":"2007-06-08T21:31:32Z","isPatch":false,"sender":{"key":"slamb@slamb.org","avatar":null},"body":"Benjamin Sergeant wrote:\n> A perforce command with all the files in the repo is generated to get\n> all the file content.\n> Here is a patch to break it into multiple successive perforce command\n> who uses 4K of parameter max, and collect the output for later.\n> \n> It works, but not for big depos, because the whole perforce depo\n> content is stored in memory in P4Sync.run(), and it looks like mine is\n> bigger than 2 Gigs, so I had to kill the process.\n\nHmm. I tried git-p4 out on Sunday, and it definitely didn't do this \nthen...this commit must have showed up since:\n\n commit 6460cf12df4556f889888b5f0b49e07040747e6f\n Author: Han-Wen Nienhuys \n Date: Wed May 23 18:49:35 2007 -0300\n\n Read p4 files in one batch.\n\nI believe HEAD at the time was this:\n\n commit 458e0545cb3dd03af9cd1a61480cbb764639043a\n Author: Simon Hausmann \n Date: Mon May 28 19:24:57 2007 +0200\n\n Fix typo in listExistingP4Branches that broke sync.\n\nso you might try checking out an old version, and I'll go RTFM on \nreading merge history, because I can't figure out when this happened.\n\n> \n> \n> diff --git a/git-p4 b/git-p4\n> index 36fe69a..906b193 100755\n> --- a/git-p4\n> +++ b/git-p4\n> @@ -703,9 +703,22 @@ class P4Sync(Command):\n> if not files:\n> return\n> \n> - filedata = p4CmdList('print %s' % ' '.join(['\"%s#%s\"' % \n> (f['path'],\n> - f['rev'])\n> - for f in files]))\n> + # We cannot put all the files on the command line\n> + # OS have limitations on the max lenght of arguments\n> + # POSIX says it's 4096 bytes, default for Linux seems to be 130 K.\n> + # and all OS from the table below seems to be higher than POSIX.\n> + # See http://www.in-ulm.de/~mascheck/various/argmax/\n\nNo need to hardcode - from Python this is \nos.sysconf(os.sysconf_names['SC_ARG_MAX'])\n\n> + chunk = ''\n> + filedata = []\n> + for i in xrange(len(files)):\n> + f = files[i]\n> + chunk += '\"%s#%s\" ' % (f['path'], f['rev'])\n> + if len(chunk) > 4000 or i == len(files)-1:\n> + data = p4CmdList('print %s' % chunk)\n> + if \"p4ExitCode\" in data[0]:\n> + die(\"Problems executing p4. Error: [%d].\" %\n> (data[0]['p4ExitCode']));\n> + filedata.extend(data)\n> + chunk = ''\n> \n> j = 0;\n> contents = {}\n> @@ -1486,3 +1499,5 @@ def main():\n> \n> if __name__ == '__main__':\n> main()\n> +\n> +# vim: set filetype=python sts=4 sw=4 et si :\n"},{"id":"44381","messageId":"4669CB75.7060009@slamb.org","threadId":"8505","inReplyTo":"4669CAB4.5080507@slamb.org","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Scott Lamb","fromEmail":"slamb@slamb.org","sentAt":"2007-06-08T21:34:45Z","receivedAt":"2007-06-08T21:34:45Z","isPatch":false,"sender":{"key":"slamb@slamb.org","avatar":null},"body":"Scott Lamb wrote:\n> No need to hardcode - from Python this is \n> os.sysconf(os.sysconf_names['SC_ARG_MAX'])\n\nIn fact, just os.sysconf('SC_ARG_MAX') will do.\n"},{"id":"44384","messageId":"1621f9fa0706081504l6106c639oe57c9fd74ebd097a@mail.gmail.com","threadId":"8505","inReplyTo":"4669CB75.7060009@slamb.org","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Benjamin Sergeant","fromEmail":"bsergean@gmail.com","sentAt":"2007-06-08T22:04:22Z","receivedAt":"2007-06-08T22:04:22Z","isPatch":false,"sender":{"key":"bsergean@gmail.com","avatar":null},"body":"On 6/8/07, Scott Lamb wrote:\n> Scott Lamb wrote:\n> > No need to hardcode - from Python this is\n> > os.sysconf(os.sysconf_names['SC_ARG_MAX'])\n>\n> In fact, just os.sysconf('SC_ARG_MAX') will do.\n>\n\nmagic number are lot of fun, why would you want to use the clean method :)\n\nSo are you saying that in the old days, git-p4 was importing the p4\ndepo in small slices to not overkill the process memory (in case the\ndepo is big) ?\n\nBTW, there is the whole universe in my depot, so using -//Work/Users\nin my client specification I usually manage to have less megs of code\non my disk after a sync.\nThis way the git-p4 clone would not use too much memory. But we would\nhave to change the way git-p4 works, it should be able to read a full\nclient view instead of just a single perforce path.\n\nWould you give me the git command to fetch the up\ngit clone --date ?\n\nThanks,\nBenjamin.\n"},{"id":"44387","messageId":"1621f9fa0706081525n64e5d3a3i6379030e1f619ef0@mail.gmail.com","threadId":"8505","inReplyTo":"1621f9fa0706081504l6106c639oe57c9fd74ebd097a@mail.gmail.com","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Benjamin Sergeant","fromEmail":"bsergean@gmail.com","sentAt":"2007-06-08T22:25:14Z","receivedAt":"2007-06-08T22:25:14Z","isPatch":false,"sender":{"key":"bsergean@gmail.com","avatar":null},"body":"I got my git-p4 from here BTW.\nhttp://repo.or.cz/w/fast-export.git\n\nOn 6/8/07, Benjamin Sergeant wrote:\n> On 6/8/07, Scott Lamb wrote:\n> > Scott Lamb wrote:\n> > > No need to hardcode - from Python this is\n> > > os.sysconf(os.sysconf_names['SC_ARG_MAX'])\n> >\n> > In fact, just os.sysconf('SC_ARG_MAX') will do.\n> >\n>\n> magic number are lot of fun, why would you want to use the clean method :)\n>\n> So are you saying that in the old days, git-p4 was importing the p4\n> depo in small slices to not overkill the process memory (in case the\n> depo is big) ?\n>\n> BTW, there is the whole universe in my depot, so using -//Work/Users\n> in my client specification I usually manage to have less megs of code\n> on my disk after a sync.\n> This way the git-p4 clone would not use too much memory. But we would\n> have to change the way git-p4 works, it should be able to read a full\n> client view instead of just a single perforce path.\n>\n> Would you give me the git command to fetch the up\n> git clone --date ?\n>\n> Thanks,\n> Benjamin.\n>\n"},{"id":"44391","messageId":"200706090038.33799.simon@lst.de","threadId":"8505","inReplyTo":"1621f9fa0706081113w7bb765ebx74f03a7407b753cb@mail.gmail.com","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Simon Hausmann","fromEmail":"simon@lst.de","sentAt":"2007-06-08T22:38:27Z","receivedAt":"2007-06-08T22:38:27Z","isPatch":false,"sender":{"key":"hausmann@kde.org","avatar":"https://gravatar.com/avatar/bc9aad4fb31dce17eb66e690e7b51fe980c62da3c225c785da35dd806b8da778?d=mp&s=160"},"body":"On Friday 08 June 2007 20:13:55 Benjamin Sergeant wrote:\n> A perforce command with all the files in the repo is generated to get\n> all the file content.\n> Here is a patch to break it into multiple successive perforce command\n> who uses 4K of parameter max, and collect the output for later.\n>\n> It works, but not for big depos, because the whole perforce depo\n> content is stored in memory in P4Sync.run(), and it looks like mine is\n> bigger than 2 Gigs, so I had to kill the process.\n\nI'd be generally fine with splitting up the \"p4 print ...\" calls into chunks \nbut you have a good point with the memory usage. The old approach of calling \nprint per file did not have any of those limitations. Han-Wen, what do you \nthink? How much of a performance improvement is the batched print?\n\n(I didn't notice any immediate difference, but then I have a very fast \nconnection to the perforce server and usually small changesets)\n\nSimon\n"},{"id":"44395","messageId":"4669E73F.2040702@xs4all.nl","threadId":"8505","inReplyTo":"1621f9fa0706081504l6106c639oe57c9fd74ebd097a@mail.gmail.com","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2007-06-08T23:33:19Z","receivedAt":"2007-06-08T23:33:19Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Benjamin Sergeant escreveu:\n\n> So are you saying that in the old days, git-p4 was importing the p4\n> depo in small slices to not overkill the process memory (in case the\n> depo is big) ?\n\nno, in the \"old days\" git-p4 used a separate p4 invocation for each file.\n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"44424","messageId":"1621f9fa0706081732k7a31782cv26f3295245057b6f@mail.gmail.com","threadId":"8505","inReplyTo":"4669E73F.2040702@xs4all.nl","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Benjamin Sergeant","fromEmail":"bsergean@gmail.com","sentAt":"2007-06-09T00:32:16Z","receivedAt":"2007-06-09T00:32:16Z","isPatch":false,"sender":{"key":"bsergean@gmail.com","avatar":null},"body":"On 6/8/07, Han-Wen Nienhuys wrote:\n> Benjamin Sergeant escreveu:\n>\n> > So are you saying that in the old days, git-p4 was importing the p4\n> > depo in small slices to not overkill the process memory (in case the\n> > depo is big) ?\n>\n> no, in the \"old days\" git-p4 used a separate p4 invocation for each file.\n>\n\nAnyway, in case you hit command line lenght limit here it is. That\nmight be interesting for the \"next days\" :)\n\nBenjamin.\n\n\n[bsergean@flanders fast-export]$ git format-patch -k -m --stdout origin\nFrom 45f2dbdb9a8c0b3beb007ae892613cdc4afab80a Mon Sep 17 00:00:00 2001\nFrom: Benjamin Sergeant \nDate: Fri, 8 Jun 2007 09:58:57 -0700\nSubject: Split p4 print call into multiple call to not exceed the\ncommand line lenght maximum\n\n---\n git-p4 | 21 ++++++++++++++++++---\n 1 files changed, 18 insertions(+), 3 deletions(-)\n\ndiff --git a/git-p4 b/git-p4\nindex 36fe69a..906b193 100755\n--- a/git-p4\n+++ b/git-p4\n@@ -703,9 +703,22 @@ class P4Sync(Command):\n if not files:\n return\n\n- filedata = p4CmdList('print %s' % ' '.join(['\"%s#%s\"' % (f['path'],\n- f['rev'])\n- for f in files]))\n+ # We cannot put all the files on the command line\n+ # OS have limitations on the max lenght of arguments\n+ # POSIX says it's 4096 bytes, default for Linux seems to be 130 K.\n+ # and all OS from the table below seems to be higher than POSIX.\n+ # See http://www.in-ulm.de/~mascheck/various/argmax/\n+ chunk = ''\n+ filedata = []\n+ for i in xrange(len(files)):\n+ f = files[i]\n+ chunk += '\"%s#%s\" ' % (f['path'], f['rev'])\n+ if len(chunk) > 4000 or i == len(files)-1:\n+ data = p4CmdList('print %s' % chunk)\n+ if \"p4ExitCode\" in data[0]:\n+ die(\"Problems executing p4. Error: [%d].\" %\n(data[0]['p4ExitCode']));\n+ filedata.extend(data)\n+ chunk = ''\n\n j = 0;\n contents = {}\n@@ -1486,3 +1499,5 @@ def main():\n\n if __name__ == '__main__':\n main()\n+\n+# vim: set filetype=python sts=4 sw=4 et si :\n--\n1.5.0.4\n\n\n>From dd9975708433efeec37b608755f54fbeaedf0f3f Mon Sep 17 00:00:00 2001\nFrom: Benjamin Sergeant \nDate: Fri, 8 Jun 2007 10:20:39 -0700\nSubject: Use os.sysconf('SC_ARG_MAX') to retrieve the max value, and\nbuild the string using join (faster)\n\n---\n git-p4 | 17 ++++++++++-------\n 1 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/git-p4 b/git-p4\nindex 906b193..8dc1963 100755\n--- a/git-p4\n+++ b/git-p4\n@@ -705,20 +705,23 @@ class P4Sync(Command):\n\n # We cannot put all the files on the command line\n # OS have limitations on the max lenght of arguments\n- # POSIX says it's 4096 bytes, default for Linux seems to be 130 K.\n- # and all OS from the table below seems to be higher than POSIX.\n # See http://www.in-ulm.de/~mascheck/various/argmax/\n- chunk = ''\n+ chunks = []\n+ chunkLenght = 0\n filedata = []\n+ maxlenght = max(int(os.sysconf('SC_ARG_MAX') * 0.90), 4000)\n+ print maxlenght\n for i in xrange(len(files)):\n f = files[i]\n- chunk += '\"%s#%s\" ' % (f['path'], f['rev'])\n- if len(chunk) > 4000 or i == len(files)-1:\n- data = p4CmdList('print %s' % chunk)\n+ chunkLenght += len(f['path']) + len(f['rev'])\n+ chunks.append('\"%s#%s\" ' % (f['path'], f['rev']))\n+ if chunkLenght > maxlenght or i == len(files)-1:\n+ data = p4CmdList('print %s' % ' '.join(chunks))\n if \"p4ExitCode\" in data[0]:\n die(\"Problems executing p4. Error: [%d].\" %\n(data[0]['p4ExitCode']));\n filedata.extend(data)\n- chunk = ''\n+ chunks = []\n+ chunkLenght = 0\n\n j = 0;\n contents = {}\n--\n1.5.0.4\n"},{"id":"44778","messageId":"466DF1E1.9060408@xs4all.nl","threadId":"8505","inReplyTo":"200706090038.33799.simon@lst.de","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2007-06-12T01:07:45Z","receivedAt":"2007-06-12T01:07:45Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Simon Hausmann escreveu:\n> On Friday 08 June 2007 20:13:55 Benjamin Sergeant wrote:\n>> A perforce command with all the files in the repo is generated to get\n>> all the file content.\n>> Here is a patch to break it into multiple successive perforce command\n>> who uses 4K of parameter max, and collect the output for later.\n>>\n>> It works, but not for big depos, because the whole perforce depo\n>> content is stored in memory in P4Sync.run(), and it looks like mine is\n>> bigger than 2 Gigs, so I had to kill the process.\n> \n> I'd be generally fine with splitting up the \"p4 print ...\" calls into chunks \n> but you have a good point with the memory usage. The old approach of calling \n> print per file did not have any of those limitations. Han-Wen, what do you \n> think? How much of a performance improvement is the batched print?\n\nOne unscientific measurement (getting 1 file vs. 30 files)\nindicates that this is about 5x times faster. \n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"44779","messageId":"466DF1FB.6010001@xs4all.nl","threadId":"8505","inReplyTo":"200706090038.33799.simon@lst.de","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2007-06-12T01:08:11Z","receivedAt":"2007-06-12T01:08:11Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Simon Hausmann escreveu:\n> On Friday 08 June 2007 20:13:55 Benjamin Sergeant wrote:\n>> A perforce command with all the files in the repo is generated to get\n>> all the file content.\n>> Here is a patch to break it into multiple successive perforce command\n>> who uses 4K of parameter max, and collect the output for later.\n>>\n>> It works, but not for big depos, because the whole perforce depo\n>> content is stored in memory in P4Sync.run(), and it looks like mine is\n>> bigger than 2 Gigs, so I had to kill the process.\n> \n> I'd be generally fine with splitting up the \"p4 print ...\" calls into chunks \n> but you have a good point with the memory usage. The old approach of calling \n> print per file did not have any of those limitations. Han-Wen, what do you \n> think? How much of a performance improvement is the batched print?\n\nOne unscientific measurement (getting 1 file vs. 30 files)\nindicates that this is about 5x times faster. \n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"44780","messageId":"466DF32D.802@xs4all.nl","threadId":"8505","inReplyTo":"1621f9fa0706081113w7bb765ebx74f03a7407b753cb@mail.gmail.com","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2007-06-12T01:13:17Z","receivedAt":"2007-06-12T01:13:17Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Benjamin Sergeant escreveu:\n> A perforce command with all the files in the repo is generated to get\n> all the file content.\n> Here is a patch to break it into multiple successive perforce command\n> who uses 4K of parameter max, and collect the output for later.\n> \n> It works, but not for big depos, because the whole perforce depo\n> content is stored in memory in P4Sync.run(), and it looks like mine is\n> bigger than 2 Gigs, so I had to kill the process.\n\nGeneral idea of the patch is ok. some nits:\n\n> + chunk = ''\n> + filedata = []\n> + for i in xrange(len(files)):\n\nwhy not \n\n for f in files:\n\n?\n\n> + f = files[i]\n> + chunk += '\"%s#%s\" ' % (f['path'], f['rev'])\n> + if len(chunk) > 4000 or i == len(files)-1:\n\n4k seems reasonable enough, but can you take the min() with\nos.sysconf('SC_ARG_MAX') ?\n\nCan you address this and resend so we can apply the patch? \nThanks.\n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"45211","messageId":"200706171011.52492.simon@lst.de","threadId":"8505","inReplyTo":"466DF32D.802@xs4all.nl","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Simon Hausmann","fromEmail":"simon@lst.de","sentAt":"2007-06-17T08:11:48Z","receivedAt":"2007-06-17T08:11:48Z","isPatch":false,"sender":{"key":"hausmann@kde.org","avatar":"https://gravatar.com/avatar/bc9aad4fb31dce17eb66e690e7b51fe980c62da3c225c785da35dd806b8da778?d=mp&s=160"},"body":"On Tuesday 12 June 2007 03:13:17 Han-Wen Nienhuys wrote:\n> Benjamin Sergeant escreveu:\n> > A perforce command with all the files in the repo is generated to get\n> > all the file content.\n> > Here is a patch to break it into multiple successive perforce command\n> > who uses 4K of parameter max, and collect the output for later.\n> >\n> > It works, but not for big depos, because the whole perforce depo\n> > content is stored in memory in P4Sync.run(), and it looks like mine is\n> > bigger than 2 Gigs, so I had to kill the process.\n>\n> General idea of the patch is ok. some nits:\n> > + chunk = ''\n> > + filedata = []\n> > + for i in xrange(len(files)):\n>\n> why not\n>\n> for f in files:\n>\n> ?\n\nIt seems 'i' is used a bit later. Is there a nicer way to express this in \npython?\n\n> > + f = files[i]\n> > + chunk += '\"%s#%s\" ' % (f['path'], f['rev'])\n> > + if len(chunk) > 4000 or i == len(files)-1:\n>\n> 4k seems reasonable enough, but can you take the min() with\n> os.sysconf('SC_ARG_MAX') ?\n>\n> Can you address this and resend so we can apply the patch?\n> Thanks.\n\nSince I ran into the very problem of a too long commandline myself yesterday I \ntook the liberty of adding the SC_ARG_MAX bit to Benjamin's patch and \ncomitting it then.\n\n\nSimon\n"},{"id":"45229","messageId":"1621f9fa0706170909s779b471fl872c027b8e7b446c@mail.gmail.com","threadId":"8505","inReplyTo":"200706171011.52492.simon@lst.de","subject":"Re: git-p4 fails when cloning a p4 depo.","fromName":"Benjamin Sergeant","fromEmail":"bsergean@gmail.com","sentAt":"2007-06-17T16:09:00Z","receivedAt":"2007-06-17T16:09:00Z","isPatch":false,"sender":{"key":"bsergean@gmail.com","avatar":null},"body":"On 6/17/07, Simon Hausmann wrote:\n> On Tuesday 12 June 2007 03:13:17 Han-Wen Nienhuys wrote:\n> > Benjamin Sergeant escreveu:\n> > > A perforce command with all the files in the repo is generated to get\n> > > all the file content.\n> > > Here is a patch to break it into multiple successive perforce command\n> > > who uses 4K of parameter max, and collect the output for later.\n> > >\n> > > It works, but not for big depos, because the whole perforce depo\n> > > content is stored in memory in P4Sync.run(), and it looks like mine is\n> > > bigger than 2 Gigs, so I had to kill the process.\n> >\n> > General idea of the patch is ok. some nits:\n> > > + chunk = ''\n> > > + filedata = []\n> > > + for i in xrange(len(files)):\n> >\n> > why not\n> >\n> > for f in files:\n> >\n> > ?\n>\n> It seems 'i' is used a bit later. Is there a nicer way to express this in\n> python?\n>\n> > > + f = files[i]\n> > > + chunk += '\"%s#%s\" ' % (f['path'], f['rev'])\n> > > + if len(chunk) > 4000 or i == len(files)-1:\n> >\n> > 4k seems reasonable enough, but can you take the min() with\n> > os.sysconf('SC_ARG_MAX') ?\n> >\n> > Can you address this and resend so we can apply the patch?\n> > Thanks.\n>\n> Since I ran into the very problem of a too long commandline myself yesterday I\n> took the liberty of adding the SC_ARG_MAX bit to Benjamin's patch and\n> comitting it then.\n>\n\nCool.\n\n(probably useless but)\nFor what it's worth ; Here is a tar file with 2 patchs:\n - The original one\n - The second one that adds the SC_ARG_MAX\n\nBTW, doing a += on a string is not supposed to be fast, appending elem\nto a sequence and then using ' '.join on them to build the big string\nis said to be faster (I did not timed it thought). (in the second\npatch also)\n\nThanks,\nBenjamin.\n\n\n>\n> Simon\n>\n>\n"}]}