{"thread":{"id":"35907","subject":"Fwd: git p4: feature request - branch check filtering","startedAt":"2014-02-18T12:42:47Z","lastAt":"2014-04-23T19:46:35Z","messageCount":5,"participants":["Dan Porter","Pete Wyckoff","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"234963","messageId":"CADtnS+weco6Lvk3hHuM7BcaRsvMkeDCmqH26s19TrgWvBYXAvA@mail.gmail.com","threadId":"35907","inReplyTo":"CADtnS+zWzPY6ftwxWUE+Gb-OKq_Kzf9y+fFfgJ-demWyX3azCg@mail.gmail.com","subject":"Fwd: git p4: feature request - branch check filtering","fromName":"Dan Porter","fromEmail":"dpreid@gmail.com","sentAt":"2014-02-18T12:42:47Z","receivedAt":"2014-02-18T12:42:47Z","isPatch":false,"sender":{"key":"dpreid@gmail.com","avatar":"https://gravatar.com/avatar/8f07f55d438cc7cb8eceff243f5d1f97afe1d52b5daa37d0016791b1aa921dd5?d=mp&s=160"},"body":"Hi,\n\nI'm unable to find a similar issue, and if it's raised on the mailing\nlist I apologize.\n\nI work at a company that has recently moved all CVS, SVN, and git\nrepositories to Perforce.  Depots have not been setup correctly in\nevery case, and there is one depot that contains literally hundreds of\nprojects under commercial development (and hundreds of branches as a\nresult)\n\nMy project may be in //stupid_depot/commercial/teamporter/rok.  This\nis the path I clone with git-p4.  The only branches in this depot that\ncontain files at this path are titled as\n'rok_porter_branch/release_1.x' or similar.\n\nWhen using '--detect-branches' git-p4 checks each key of branches to\nsee if any of them have files in the path I've cloned.  Whilst this is\ngood in practice there is unfortunately 6,809 branches, git-p4\nprocesses about 2 a second and just under an hour to perform any\ngit-p4 rebase, submit, or similar operation.\n\nI propose the addition of a branch list filtering option\n(--filter-branches) that takes either a regular expression or list of\nbranches it should check.  This may be useful in sane situations where\nyou don't want to scan every branch in a Perforce repository, or\nblacklist branches that have undesirable content (for example, one of\nthe branches is called 'svn-backup'.  It contains a single, multi-GB\ntarball.)\n\nIt would be ideal to have this information (after initial clone or\nsync) stored somewhere in the git config where is appropriate so that\nfuture submit/rebase operations adhere to this list.\n\nHas something like this been worked on, or has been considered in the\npast?  If not I will consider implementing this after reading up on\nthe Git code guidelines.\n\nThanks for keeping the Git workflow accessible in painful areas.\n\nDan\n"},{"id":"235205","messageId":"20140223151247.GA1272@padd.com","threadId":"35907","inReplyTo":"CADtnS+weco6Lvk3hHuM7BcaRsvMkeDCmqH26s19TrgWvBYXAvA@mail.gmail.com","subject":"Re: Fwd: git p4: feature request - branch check filtering","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2014-02-23T15:12:47Z","receivedAt":"2014-02-23T15:12:47Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"dpreid@gmail.com wrote on Tue, 18 Feb 2014 12:42 +0000:\n> I work at a company that has recently moved all CVS, SVN, and git\n> repositories to Perforce.  Depots have not been setup correctly in\n> every case, and there is one depot that contains literally hundreds of\n> projects under commercial development (and hundreds of branches as a\n> result)\n\nMy condolences.\n\n> My project may be in //stupid_depot/commercial/teamporter/rok.  This\n> is the path I clone with git-p4.  The only branches in this depot that\n> contain files at this path are titled as\n> 'rok_porter_branch/release_1.x' or similar.\n> \n> When using '--detect-branches' git-p4 checks each key of branches to\n> see if any of them have files in the path I've cloned.  Whilst this is\n> good in practice there is unfortunately 6,809 branches, git-p4\n> processes about 2 a second and just under an hour to perform any\n> git-p4 rebase, submit, or similar operation.\n\nThis is in getBranchMapping() presumably.  Where it loops\nover each branch doing \"p4 branch -o\".  Yuk.\n\nYou could always avoid the --detect-branches if you don't really\nneed it, instead doing, say, multiple \"git p4 sync\" for the\ndifferent areas of the repo that interest you, each with its own\ndestination branch in git (\"p4/depot-part1\", \"p4/depot-part3\",\n...).  Or --use-client-spec to cobble together an exact mapping\nof where p4 files should land in git, all in a single git branch\nthen.\n\n> I propose the addition of a branch list filtering option\n> (--filter-branches) that takes either a regular expression or list of\n> branches it should check.  This may be useful in sane situations where\n> you don't want to scan every branch in a Perforce repository, or\n> blacklist branches that have undesirable content (for example, one of\n> the branches is called 'svn-backup'.  It contains a single, multi-GB\n> tarball.)\n\nThere is the existing git-p4.branchList option that explicitly\nadds (or overrides) branch information, beyond the ones auto-discovered.\n\nYou might be able to use that option, but change its behavior\nto avoid the scan.  So that if that option is set in the config,\np4 is not asked anything about its branches.  Not sure if this\nwould break anyone's setup though.\n\nAnother approach would be to add a config option\ngit-p4.branchScan that defaults to True.  You could turn it off\nand use branchList.\n\n> It would be ideal to have this information (after initial clone or\n> sync) stored somewhere in the git config where is appropriate so that\n> future submit/rebase operations adhere to this list.\n> \n> Has something like this been worked on, or has been considered in the\n> past?  If not I will consider implementing this after reading up on\n> the Git code guidelines.\n> \n> Thanks for keeping the Git workflow accessible in painful areas.\n\nIt would be great if you could get something like this to work.\nStart in getBranchMapping() and don't forget to write up your\nwork in Documentation/git-p4.txt.  Also, this is sort of a messy\narea of the code, unfortunately.  t/t9801 tries to make sure some\nof it keeps working.\n\n\t\t-- Pete\n"},{"id":"239319","messageId":"CADtnS+w9q0dmnGsZoDr12GZ-RSZzcfPs6rfii-4eK7Hhn2byag@mail.gmail.com","threadId":"35907","inReplyTo":"20140223151247.GA1272@padd.com","subject":"Re: Fwd: git p4: feature request - branch check filtering","fromName":"Dan Porter","fromEmail":"dpreid@gmail.com","sentAt":"2014-04-22T09:29:19Z","receivedAt":"2014-04-22T09:29:19Z","isPatch":false,"sender":{"key":"dpreid@gmail.com","avatar":"https://gravatar.com/avatar/8f07f55d438cc7cb8eceff243f5d1f97afe1d52b5daa37d0016791b1aa921dd5?d=mp&s=160"},"body":"Hi Pete,\n\nI should have updated on this earlier, but I wished to refine my work\non this feature before submitting.  With 2.0 looming I'll submit\nwhat's there so far.\n\nThere is a patch viewable at this link:\nhttps://github.com/Stealthii/git/commit/f7a2e611262fd977ac99e066872d3d0743b7df3c\n\nFor the use case this works perfectly - if I define branch mappings\nwith git config, followed by setting 'git-p4.skipBranchScan' to true,\ngit-p4 will skip scanning of all remote branches and limit to what's\ndefined in the map.  An example config:\n\n[git-p4]\n        skipBranchScan = true\n        branchList = release_1.0.0:release_1.1.0\n        branchList = release_1.1.0:release_1.2.0\n\nIf there is any more information I need to provide let me know. I have\nbeen using this patch for over two months, testing both use cases with\nand without git-p4.skipBranchScan and I have noticed no issues.  Logic\nof git-p4 is not changed from default behaviour, unless the user\nexplicitly sets the boolean flag to skip scanning.\n\nDan\n\n\nThis email and any files transmitted with it are confidential and\nintended solely for the use of the individual or entity to whom they\nare addressed. If you have received this email in error please notify\nthe system manager. This message contains confidential information and\nis intended only for the individual named. If you are not the named\naddressee you should not disseminate, distribute or copy this e-mail.\nPlease notify the sender immediately by e-mail if you have received\nthis e-mail by mistake and delete this e-mail from your system. If you\nare not the intended recipient you are notified that disclosing,\ncopying, distributing or taking any action in reliance on the contents\nof this information is strictly prohibited.\n\n\nOn 23 February 2014 15:12, Pete Wyckoff <pw@padd.com> wrote:\n> dpreid@gmail.com wrote on Tue, 18 Feb 2014 12:42 +0000:\n>> I work at a company that has recently moved all CVS, SVN, and git\n>> repositories to Perforce.  Depots have not been setup correctly in\n>> every case, and there is one depot that contains literally hundreds of\n>> projects under commercial development (and hundreds of branches as a\n>> result)\n>\n> My condolences.\n>\n>> My project may be in //stupid_depot/commercial/teamporter/rok.  This\n>> is the path I clone with git-p4.  The only branches in this depot that\n>> contain files at this path are titled as\n>> 'rok_porter_branch/release_1.x' or similar.\n>>\n>> When using '--detect-branches' git-p4 checks each key of branches to\n>> see if any of them have files in the path I've cloned.  Whilst this is\n>> good in practice there is unfortunately 6,809 branches, git-p4\n>> processes about 2 a second and just under an hour to perform any\n>> git-p4 rebase, submit, or similar operation.\n>\n> This is in getBranchMapping() presumably.  Where it loops\n> over each branch doing \"p4 branch -o\".  Yuk.\n>\n> You could always avoid the --detect-branches if you don't really\n> need it, instead doing, say, multiple \"git p4 sync\" for the\n> different areas of the repo that interest you, each with its own\n> destination branch in git (\"p4/depot-part1\", \"p4/depot-part3\",\n> ...).  Or --use-client-spec to cobble together an exact mapping\n> of where p4 files should land in git, all in a single git branch\n> then.\n>\n>> I propose the addition of a branch list filtering option\n>> (--filter-branches) that takes either a regular expression or list of\n>> branches it should check.  This may be useful in sane situations where\n>> you don't want to scan every branch in a Perforce repository, or\n>> blacklist branches that have undesirable content (for example, one of\n>> the branches is called 'svn-backup'.  It contains a single, multi-GB\n>> tarball.)\n>\n> There is the existing git-p4.branchList option that explicitly\n> adds (or overrides) branch information, beyond the ones auto-discovered.\n>\n> You might be able to use that option, but change its behavior\n> to avoid the scan.  So that if that option is set in the config,\n> p4 is not asked anything about its branches.  Not sure if this\n> would break anyone's setup though.\n>\n> Another approach would be to add a config option\n> git-p4.branchScan that defaults to True.  You could turn it off\n> and use branchList.\n>\n>> It would be ideal to have this information (after initial clone or\n>> sync) stored somewhere in the git config where is appropriate so that\n>> future submit/rebase operations adhere to this list.\n>>\n>> Has something like this been worked on, or has been considered in the\n>> past?  If not I will consider implementing this after reading up on\n>> the Git code guidelines.\n>>\n>> Thanks for keeping the Git workflow accessible in painful areas.\n>\n> It would be great if you could get something like this to work.\n> Start in getBranchMapping() and don't forget to write up your\n> work in Documentation/git-p4.txt.  Also, this is sort of a messy\n> area of the code, unfortunately.  t/t9801 tries to make sure some\n> of it keeps working.\n>\n>                 -- Pete\n>\n"},{"id":"239348","messageId":"xmqqoazt71nb.fsf@gitster.dls.corp.google.com","threadId":"35907","inReplyTo":"CADtnS+w9q0dmnGsZoDr12GZ-RSZzcfPs6rfii-4eK7Hhn2byag@mail.gmail.com","subject":"Re: Fwd: git p4: feature request - branch check filtering","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-22T16:46:32Z","receivedAt":"2014-04-22T16:46:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Dan Porter <dpreid@gmail.com> writes:\n\n> I should have updated on this earlier, but I wished to refine my work\n> on this feature before submitting.  With 2.0 looming I'll submit\n> what's there so far.\n\nI am not Pete, but...\n\nThe pre-release time is to find and fix regressions that may have\nbeen introduced since the last release while we tried to add new\nfeatures and fixes to 2.0 until now.\n\n\"With 2.0 looming\" is not a good reason to send out a new work.  In\nfact, if 2.0 is \"looming\", it is already too late for the upcoming\nrelease.  Of course \"With 2.0 looming\" does not mean that you must\nnot work on things that are regression fixes---it is your own time\nand effort, and it will be great if that can help later releases.\n\nThanks for contributing anyway ;-)\n"},{"id":"239469","messageId":"20140423194635.GA19597@padd.com","threadId":"35907","inReplyTo":"CADtnS+w9q0dmnGsZoDr12GZ-RSZzcfPs6rfii-4eK7Hhn2byag@mail.gmail.com","subject":"Re: Fwd: git p4: feature request - branch check filtering","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2014-04-23T19:46:35Z","receivedAt":"2014-04-23T19:46:35Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"dpreid@gmail.com wrote on Tue, 22 Apr 2014 10:29 +0100:\n> There is a patch viewable at this link:\n> https://github.com/Stealthii/git/commit/f7a2e611262fd977ac99e066872d3d0743b7df3c\n> \n> For the use case this works perfectly - if I define branch mappings\n> with git config, followed by setting 'git-p4.skipBranchScan' to true,\n> git-p4 will skip scanning of all remote branches and limit to what's\n> defined in the map.  An example config:\n> \n> [git-p4]\n>         skipBranchScan = true\n>         branchList = release_1.0.0:release_1.1.0\n>         branchList = release_1.1.0:release_1.2.0\n> \n> If there is any more information I need to provide let me know. I have\n> been using this patch for over two months, testing both use cases with\n> and without git-p4.skipBranchScan and I have noticed no issues.  Logic\n> of git-p4 is not changed from default behaviour, unless the user\n> explicitly sets the boolean flag to skip scanning.\n\nThanks, Dan.  This looks good and is a fine compromise\nconsidering the various choices we discussed earlier.\n\nJunio's comments about 2.0 non-withstanding, I think this change\nshould go into the next convenient release.  So 2.1 or 2.0.1;\nhowever the numbers end up working post-2.0.\n\nIf you could take a look at Documentation/SubmittingPatches,\nand do a few things:\n\n    1.  Write a nice commit message, say:\n\n\tgit p4: add skipBranchScan to avoid p4 branch scan\n\n\tSome more useful text.\n\n    2.  Include at the bottom of that message:\n\n\tAcked-by: Pete Wyckoff <pw@padd.com>\n\n    3.  Inline the text of your patch, not just a link to github.\n\n    4.  Consider adding a t98xx test.  This isn't required for\n\ta fairly minor change like yours, but if you think TDD\n\tis fun, have at it.  Might protect your feature against\n\tfuture hackers who would try to break it.  :)\n\nThen send it to vger, cc junio (and me), and he will be kind\nenough to queue it up appropriately.\n\nThanks!\n\n\t\t-- Pete\n"}]}