{"thread":{"id":"29668","subject":"git-svn show-externals and svn version","startedAt":"2012-02-19T18:53:56Z","lastAt":"2012-03-07T09:39:40Z","messageCount":5,"participants":["Nikolaus Demmel","Jakub Narebski","Andreas Stricker"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"184950","messageId":"E59CCE45-6F92-4748-9B6E-2A562647904B@nikolaus-demmel.de","threadId":"29668","inReplyTo":null,"subject":"git-svn show-externals and svn version","fromName":"Nikolaus Demmel","fromEmail":"nikolaus@nikolaus-demmel.de","sentAt":"2012-02-19T18:53:56Z","receivedAt":"2012-02-19T18:53:56Z","isPatch":false,"sender":{"key":"nikolaus@nikolaus-demmel.de","avatar":"https://gravatar.com/avatar/2c9741b87d0f456387a06f577a904933d4614758202292a48ca1c739ebb48916?d=mp&s=160"},"body":"Hi,\n\nI am currently investigating getting support for svn externals in git-svn (you might have noticed my other mails).\n\nIt turns out that there are quite a few scripts floating around that use the output of show-externals and then try to pull these externals with git-svn into independent repositories and add the folders as submodules to the root repository.\n\nHowever, none of them work for me, and the primary reason AFAICT is that they were written for the pre svn1.5 format of svn:externals. From 1.5 svn supports a new format of svn:externals, which changes the order of revision, repository-url, and local folder, and also adds the posibility to add relative urls, peg-revisions, etc [1].\n\nOn top that it seems to me that the output of show-externals was purely designed for the old format. For example, if you compare the output of \"git svn show-externals\" with \"git svn propget svn:externals\" in an example repository using the new format [2], you find that the in the show-externals output the prepended \"/\" and \"/instantiations/\" at the beginning of each line does not make sense. If the target url (all relative with the ^ syntax in this case) and the sub-folder were swapped in order, as of pre 1.5 svn, it would make much more sense. Also apparently the code for show-externals was added in 2007 and not changed since, whereas svn 1.5 was released in 2008.\n\nWhat I am not completely clear about is, whether svn 1.5 and later enforces the new syntax, or whether it just adds the new syntax and still has to support the old syntax (which could be distinguished, I guess, by checking of the last part on an entry is an absolute URL instead of a subfolder). Also, I'm not sure if the format depends on the version of the svn-server or the client. I would assume you can check out a repository hosted with svn 1.4 with a 1.5 client. Does the client process the svn:externals and present it in the new format, or is this the text string just taken from the server unaltered (I have not much knowledge of how svn actually works internally)?\n\nAnother question is whether the perl svn bindings present the svn:externals in some parsed, standard format, or do they just give you the raw text string?\n\nIn order to make show-externals more useful with the svn 1.5 and later syntax, one would maybe need to check the underlying svn version. I guess it is also quite important to retain backwards compatibility, such that users of externals with the old syntax would still get the same output as before.\n\nI would suggest that the show-externals output should be as close as possible to the svn:externals syntax, possibly adapting the subfolder path for nested folders. However here the recursive display of externals for subfolders becomes a bit more tricky, since the URL can also be relative to the subfolder as of the new syntax. Maybe the easiest way to deal with the new syntax in show-externals would be to have each line like it is in the svn-properties, but add a space separated relative path to the corresponding subfolder at the beginning. A tool that uses this is then responsible for making sure the relative URLs are resolved correctly.\n\nTo sum up, given that all the questions I have are answered like I think is most likely, it would boil down to changing the output of show-externals for svn 1.5  and later just slightly, namely by inserting an additional space between the prepended subfolder and the actual svn:externals definition in each line.\n\nAny thoughts and/or answers?\n\nCheers,\nNikolaus\n\n\n[1] http://svnbook.red-bean.com/en/1.7/svn.advanced.externals.html\n[2] http://paste.lisp.org/display/127858"},{"id":"185070","messageId":"994D6101-DD43-4CD9-BB96-34807E3087C4@nikolaus-demmel.de","threadId":"29668","inReplyTo":"E59CCE45-6F92-4748-9B6E-2A562647904B@nikolaus-demmel.de","subject":"Re: git-svn show-externals and svn version","fromName":"Nikolaus Demmel","fromEmail":"nikolaus@nikolaus-demmel.de","sentAt":"2012-02-21T11:14:49Z","receivedAt":"2012-02-21T11:14:49Z","isPatch":false,"sender":{"key":"nikolaus@nikolaus-demmel.de","avatar":"https://gravatar.com/avatar/2c9741b87d0f456387a06f577a904933d4614758202292a48ca1c739ebb48916?d=mp&s=160"},"body":"Hi,\n\nas a followup just another example of when the current show-externals gives a flaky output, namely when the line in the external definition is commented.\n\n\n$ git svn show-externals\n[...]\n# /src/\n/src/#https://codex.cs.bham.ac.uk/svn/nah/cogx/code/subarchitectures/vision.sa/tags/matlab-cosy-2008     matlab_cosy_2008\n/src/#https://codex.cs.bham.ac.uk/svn/nah/cogx/code/subarchitectures/vision.sa/tags/matlab-review-2009     matlab_review_2009\n\n\nRegards,\nNikolaus\n\n\nAm 19.02.2012 um 19:53 schrieb Nikolaus Demmel:\n\n> Hi,\n> \n> I am currently investigating getting support for svn externals in git-svn (you might have noticed my other mails).\n> \n> It turns out that there are quite a few scripts floating around that use the output of show-externals and then try to pull these externals with git-svn into independent repositories and add the folders as submodules to the root repository.\n> \n> However, none of them work for me, and the primary reason AFAICT is that they were written for the pre svn1.5 format of svn:externals. From 1.5 svn supports a new format of svn:externals, which changes the order of revision, repository-url, and local folder, and also adds the posibility to add relative urls, peg-revisions, etc [1].\n> \n> On top that it seems to me that the output of show-externals was purely designed for the old format. For example, if you compare the output of \"git svn show-externals\" with \"git svn propget svn:externals\" in an example repository using the new format [2], you find that the in the show-externals output the prepended \"/\" and \"/instantiations/\" at the beginning of each line does not make sense. If the target url (all relative with the ^ syntax in this case) and the sub-folder were swapped in order, as of pre 1.5 svn, it would make much more sense. Also apparently the code for show-externals was added in 2007 and not changed since, whereas svn 1.5 was released in 2008.\n> \n> What I am not completely clear about is, whether svn 1.5 and later enforces the new syntax, or whether it just adds the new syntax and still has to support the old syntax (which could be distinguished, I guess, by checking of the last part on an entry is an absolute URL instead of a subfolder). Also, I'm not sure if the format depends on the version of the svn-server or the client. I would assume you can check out a repository hosted with svn 1.4 with a 1.5 client. Does the client process the svn:externals and present it in the new format, or is this the text string just taken from the server unaltered (I have not much knowledge of how svn actually works internally)?\n> \n> Another question is whether the perl svn bindings present the svn:externals in some parsed, standard format, or do they just give you the raw text string?\n> \n> In order to make show-externals more useful with the svn 1.5 and later syntax, one would maybe need to check the underlying svn version. I guess it is also quite important to retain backwards compatibility, such that users of externals with the old syntax would still get the same output as before.\n> \n> I would suggest that the show-externals output should be as close as possible to the svn:externals syntax, possibly adapting the subfolder path for nested folders. However here the recursive display of externals for subfolders becomes a bit more tricky, since the URL can also be relative to the subfolder as of the new syntax. Maybe the easiest way to deal with the new syntax in show-externals would be to have each line like it is in the svn-properties, but add a space separated relative path to the corresponding subfolder at the beginning. A tool that uses this is then responsible for making sure the relative URLs are resolved correctly.\n> \n> To sum up, given that all the questions I have are answered like I think is most likely, it would boil down to changing the output of show-externals for svn 1.5  and later just slightly, namely by inserting an additional space between the prepended subfolder and the actual svn:externals definition in each line.\n> \n> Any thoughts and/or answers?\n> \n> Cheers,\n> Nikolaus\n> \n> \n> [1] http://svnbook.red-bean.com/en/1.7/svn.advanced.externals.html\n> [2] http://paste.lisp.org/display/127858--\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"185179","messageId":"5B8386D7-3C01-4A58-A7AB-9AA43BB45572@nikolaus-demmel.de","threadId":"29668","inReplyTo":"994D6101-DD43-4CD9-BB96-34807E3087C4@nikolaus-demmel.de","subject":"Re: git-svn show-externals and svn version","fromName":"Nikolaus Demmel","fromEmail":"nikolaus@nikolaus-demmel.de","sentAt":"2012-02-22T15:27:23Z","receivedAt":"2012-02-22T15:27:23Z","isPatch":false,"sender":{"key":"nikolaus@nikolaus-demmel.de","avatar":"https://gravatar.com/avatar/2c9741b87d0f456387a06f577a904933d4614758202292a48ca1c739ebb48916?d=mp&s=160"},"body":"Hi,\n\nI feel a bit like I am talking to myself, but I see from the high\ntraffic on this list that people are busy doing great things :-). I will\nwrite anyway in case someone interested in git-svn listens.\n\nIs there an appropriate place to file these kinds of feature/enhancement\nrequests?\n\nSo I've investigated the matter a bit further. Turns out in the\nsubversion SWIG language bindings there is an API function that parses\nsvn:externals definitions for you. See [1] for a recent (minimal) change\nto make this function available in python. I guess supporting Perl\nrequires equally minimal changes. I haven't attempted it myself since I\nknow neither Perl nor SWIG.\n\nHow could this be used in git-svn show-externals? As layed out before, I\nbelieve that the current output for the svn1.5 syntax is inherently\nbroken and we should not worry about backwards compatibility for\nthat. To maintain backwards compatibility with the output for the old\nformat and to give a canonical, easy to parse, output for any external\ndefinition, I suggest sticking to the current format, just inserting the\nparsed definition at the appropriate place with relative URLs completely\nresolved to absolute ones.\n\nThe pre-svn1.5 syntax for external definitions was:\n\nLOCAL-PATH [-r REVISION] ABSOLUTE-URL\n\nThe output for show-externals was thus (note that there is no parsing of\nthe external definition going on yet):\n\nDIRECTORY-PREFIX/LOCAL-PATH [-rREVISION] ABSOLUTE-URL\n\nThe DIRECTORY-PREFIX was added because show-externals shows the external\ndefinitions for all subdirectories recursively. With this prefix, every\nline can be processed on its own. I suggest extending this output to:\n\nDIRECTORY-PREFIX/LOCAL-PATH [-rREVISION] ABSOLUTE-URL[@PEG-REV]\n\nAgain, as mentioned above, show-externals should parse the definitions\nand resolve relative URLs. Any lines that the svn API call cannot parse\nshould be completely ommited (e.g. commented lines and empty lines).\n\nAs I understand it show-externals is intended primarily for scripts for\nfurther processing. With this extension existing scripts for the old\nsyntax should keep working also long as they don't feature\npeg-revisions. With relative URLs resolved and a standard ordering old\nand new syntax cannot be distinguished in terms of show-externals output\n(except when there are peg-revsion are there).\n\nThoughts, comments, opinions?\n\nCheers,\nNikolaus\n\n[1] http://thread.gmane.org/gmane.comp.version-control.subversion.user/109033/focus=109036\n\nAm 21.02.2012 um 12:14 schrieb Nikolaus Demmel:\n\n> Hi,\n> \n> as a followup just another example of when the current show-externals gives a flaky output, namely when the line in the external definition is commented.\n> \n> \n> $ git svn show-externals\n> [...]\n> # /src/\n> /src/#https://codex.cs.bham.ac.uk/svn/nah/cogx/code/subarchitectures/vision.sa/tags/matlab-cosy-2008     matlab_cosy_2008\n> /src/#https://codex.cs.bham.ac.uk/svn/nah/cogx/code/subarchitectures/vision.sa/tags/matlab-review-2009     matlab_review_2009\n> \n> \n> Regards,\n> Nikolaus\n> \n> \n> Am 19.02.2012 um 19:53 schrieb Nikolaus Demmel:\n> \n>> Hi,\n>> \n>> I am currently investigating getting support for svn externals in git-svn (you might have noticed my other mails).\n>> \n>> It turns out that there are quite a few scripts floating around that use the output of show-externals and then try to pull these externals with git-svn into independent repositories and add the folders as submodules to the root repository.\n>> \n>> However, none of them work for me, and the primary reason AFAICT is that they were written for the pre svn1.5 format of svn:externals. From 1.5 svn supports a new format of svn:externals, which changes the order of revision, repository-url, and local folder, and also adds the posibility to add relative urls, peg-revisions, etc [1].\n>> \n>> On top that it seems to me that the output of show-externals was purely designed for the old format. For example, if you compare the output of \"git svn show-externals\" with \"git svn propget svn:externals\" in an example repository using the new format [2], you find that the in the show-externals output the prepended \"/\" and \"/instantiations/\" at the beginning of each line does not make sense. If the target url (all relative with the ^ syntax in this case) and the sub-folder were swapped in order, as of pre 1.5 svn, it would make much more sense. Also apparently the code for show-externals was added in 2007 and not changed since, whereas svn 1.5 was released in 2008.\n>> \n>> What I am not completely clear about is, whether svn 1.5 and later enforces the new syntax, or whether it just adds the new syntax and still has to support the old syntax (which could be distinguished, I guess, by checking of the last part on an entry is an absolute URL instead of a subfolder). Also, I'm not sure if the format depends on the version of the svn-server or the client. I would assume you can check out a repository hosted with svn 1.4 with a 1.5 client. Does the client process the svn:externals and present it in the new format, or is this the text string just taken from the server unaltered (I have not much knowledge of how svn actually works internally)?\n>> \n>> Another question is whether the perl svn bindings present the svn:externals in some parsed, standard format, or do they just give you the raw text string?\n>> \n>> In order to make show-externals more useful with the svn 1.5 and later syntax, one would maybe need to check the underlying svn version. I guess it is also quite important to retain backwards compatibility, such that users of externals with the old syntax would still get the same output as before.\n>> \n>> I would suggest that the show-externals output should be as close as possible to the svn:externals syntax, possibly adapting the subfolder path for nested folders. However here the recursive display of externals for subfolders becomes a bit more tricky, since the URL can also be relative to the subfolder as of the new syntax. Maybe the easiest way to deal with the new syntax in show-externals would be to have each line like it is in the svn-properties, but add a space separated relative path to the corresponding subfolder at the beginning. A tool that uses this is then responsible for making sure the relative URLs are resolved correctly.\n>> \n>> To sum up, given that all the questions I have are answered like I think is most likely, it would boil down to changing the output of show-externals for svn 1.5  and later just slightly, namely by inserting an additional space between the prepended subfolder and the actual svn:externals definition in each line.\n>> \n>> Any thoughts and/or answers?\n>> \n>> Cheers,\n>> Nikolaus\n>> \n>> \n>> [1] http://svnbook.red-bean.com/en/1.7/svn.advanced.externals.html\n>> [2] http://paste.lisp.org/display/127858--\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"186027","messageId":"m3sjhotzhj.fsf@localhost.localdomain","threadId":"29668","inReplyTo":"5B8386D7-3C01-4A58-A7AB-9AA43BB45572@nikolaus-demmel.de","subject":"Re: git-svn show-externals and svn version","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2012-03-04T10:46:53Z","receivedAt":"2012-03-04T10:46:53Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Nikolaus Demmel <nikolaus@nikolaus-demmel.de> writes:\n\n> I feel a bit like I am talking to myself, but I see from the high\n> traffic on this list that people are busy doing great things :-). I will\n> write anyway in case someone interested in git-svn listens.\n> \n> Is there an appropriate place to file these kinds of feature/enhancement\n> requests?\n> \n> So I've investigated the matter a bit further. Turns out in the\n> subversion SWIG language bindings there is an API function that parses\n> svn:externals definitions for you. See [1] for a recent (minimal) change\n> to make this function available in python. I guess supporting Perl\n> requires equally minimal changes. I haven't attempted it myself since I\n> know neither Perl nor SWIG.\n> \n> How could this be used in git-svn show-externals? As layed out before, I\n> believe that the current output for the svn1.5 syntax is inherently\n> broken and we should not worry about backwards compatibility for\n> that. To maintain backwards compatibility with the output for the old\n> format and to give a canonical, easy to parse, output for any external\n> definition, I suggest sticking to the current format, just inserting the\n> parsed definition at the appropriate place with relative URLs completely\n> resolved to absolute ones.\n[...]\n\n> Thoughts, comments, opinions?\n\nPerhaps better support for svn:externals would be a good project for\nGit in Google Summer of Code 2012?  You could write a proposal, see\n\n* GSoC 2012 application process\n  Message-ID: <20120302091114.GA3984@sigill.intra.peff.net>\n  http://thread.gmane.org/gmane.comp.version-control.git/192014 \n\n* [RFC] \"Remote helper for Subversion\" project\n  Message-ID: <1330777646-28381-1-git-send-email-davidbarr@google.com>\n  http://thread.gmane.org/gmane.comp.version-control.git/192106\n\n* https://github.com/peff/git/wiki/SoC-2012-Ideas\n\n-- \nJakub Narebski\n"},{"id":"186294","messageId":"4F572CDC.6060303@futurelab.ch","threadId":"29668","inReplyTo":"5B8386D7-3C01-4A58-A7AB-9AA43BB45572@nikolaus-demmel.de","subject":"Re: git-svn show-externals and svn version","fromName":"Andreas Stricker","fromEmail":"astricker@futurelab.ch","sentAt":"2012-03-07T09:39:40Z","receivedAt":"2012-03-07T09:39:40Z","isPatch":false,"sender":{"key":"astricker@futurelab.ch","avatar":"https://gravatar.com/avatar/007ac93f9abc20ee78c5169783fcf3f065d470d6b53d47050e30cf7ebad5025b?d=mp&s=160"},"body":"Nikolaus Demmel <nikolaus@nikolaus-demmel.de> wrote:\n> I feel a bit like I am talking to myself, but I see from the high\n> traffic on this list that people are busy doing great things :-). I will\n> write anyway in case someone interested in git-svn listens.\n\nUm I'm always a bit behind in reading this list. A long time ago my\ncolleague and I implemented a parser for the new svn:externals format\nas a proof of concept[1]. I never took the time to finish it.\n\n> So I've investigated the matter a bit further. Turns out in the\n> subversion SWIG language bindings there is an API function that parses\n> svn:externals definitions for you.\n\nThis looks like a sane approach. I ended with a bunch of complicated\nparsing code [2].\n\n> How could this be used in git-svn show-externals? As layed out before, I\n> believe that the current output for the svn1.5 syntax is inherently\n> broken and we should not worry about backwards compatibility for\n> that.\n\nI second that. The output for the new syntax is just plain broken and\ncan't be used in a sane way. I know that because I tried...\n\n> To maintain backwards compatibility with the output for the old\n> format and to give a canonical, easy to parse, output for any external\n> definition, I suggest sticking to the current format, just inserting the\n> parsed definition at the appropriate place with relative URLs completely\n> resolved to absolute ones.\n\nThis is exactly what my proof of concept does. The output format keeps\nthe same as for pre subversion 1.5 format.\n\n> The pre-svn1.5 syntax for external definitions was:\n> \n> LOCAL-PATH [-r REVISION] ABSOLUTE-URL\n> \n> The output for show-externals was thus (note that there is no parsing of\n> the external definition going on yet):\n> \n> DIRECTORY-PREFIX/LOCAL-PATH [-rREVISION] ABSOLUTE-URL\n\nWasn't there also a line commented with a hash \"#\" before that? Like:\n# DIRECTORY-PREFIX\n\n> The DIRECTORY-PREFIX was added because show-externals shows the external\n> definitions for all subdirectories recursively. With this prefix, every\n> line can be processed on its own. I suggest extending this output to:\n> \n> DIRECTORY-PREFIX/LOCAL-PATH [-rREVISION] ABSOLUTE-URL[@PEG-REV]\n> \n> Again, as mentioned above, show-externals should parse the definitions\n> and resolve relative URLs. Any lines that the svn API call cannot parse\n> should be completely ommited (e.g. commented lines and empty lines).\n\nA sane approach. What about a warning about lines skipped?\n\n> As I understand it show-externals is intended primarily for scripts for\n> further processing. With this extension existing scripts for the old\n> syntax should keep working also long as they don't feature\n> peg-revisions. With relative URLs resolved and a standard ordering old\n> and new syntax cannot be distinguished in terms of show-externals output\n> (except when there are peg-revsion are there).\n\nTrue. So external tools like git-svn-clone-externals will still work\nwith this. I verified this with my proof of concept.\n\nRegards, Andy\n\n[1] https://github.com/AndyStricker/git\n[2]\nhttps://github.com/AndyStricker/git/commit/9981b3b8313fb831247a16a04d5040bd6a8660b1\n"}]}