{"thread":{"id":"25664","subject":"import determinism","startedAt":"2010-11-07T20:25:36Z","lastAt":"2010-11-11T13:09:14Z","messageCount":13,"participants":["Enrico Weigelt","Ævar Arnfjörð Bjarmason","Andreas Schwab","Martin Langhoff","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"155344","messageId":"20101107202535.GA18766@nibiru.local","threadId":"25664","inReplyTo":null,"subject":"import determinism","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-11-07T20:25:36Z","receivedAt":"2010-11-07T20:25:36Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"\nHi folks,\n\n\nI'm curious on how deterministic the imports (git-cvsimport and\ngit-svn) are. Suppose I close the same cvs repo twice (assuming\nno write access in between), are the resulting object SHA-1's\nthe same ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"155345","messageId":"AANLkTi=mx0AAKo2Asn5XJVcs30-PLuwhTbM=o0y36Wa_@mail.gmail.com","threadId":"25664","inReplyTo":"20101107202535.GA18766@nibiru.local","subject":"Re: import determinism","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-11-07T20:46:57Z","receivedAt":"2010-11-07T20:46:57Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sun, Nov 7, 2010 at 21:25, Enrico Weigelt wrote:\n> I'm curious on how deterministic the imports (git-cvsimport and\n> git-svn) are. Suppose I close the same cvs repo twice (assuming\n> no write access in between), are the resulting object SHA-1's\n> the same ?\n\nNo, because the committer dates will be different.\n"},{"id":"155346","messageId":"m2lj54u9uj.fsf@igel.home","threadId":"25664","inReplyTo":"AANLkTi=mx0AAKo2Asn5XJVcs30-PLuwhTbM=o0y36Wa_@mail.gmail.com","subject":"Re: import determinism","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2010-11-07T21:01:24Z","receivedAt":"2010-11-07T21:01:24Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Sun, Nov 7, 2010 at 21:25, Enrico Weigelt wrote:\n>> I'm curious on how deterministic the imports (git-cvsimport and\n>> git-svn) are. Suppose I close the same cvs repo twice (assuming\n>> no write access in between), are the resulting object SHA-1's\n>> the same ?\n>\n> No, because the committer dates will be different.\n\nThe committer info in every commit will be the same as the author info,\nso the repository conversion is completely deterministic.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"155348","messageId":"20101107215652.GB18766@nibiru.local","threadId":"25664","inReplyTo":"m2lj54u9uj.fsf@igel.home","subject":"Re: import determinism","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-11-07T21:56:52Z","receivedAt":"2010-11-07T21:56:52Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Andreas Schwab wrote:\n\n> The committer info in every commit will be the same as the author info,\n> so the repository conversion is completely deterministic.\n\nThanks, that helps a lot.\n\nMy point is that I'm running a buch of automatic mirrors and like\nto get them clustered. In theory, it should be enough - for cvs - \nrunning git-cvsimport regularily on all nodes, right ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"155349","messageId":"AANLkTikXxM=CfU2dKAY9khi1_tAsGDdUEc8S5AxooGH9@mail.gmail.com","threadId":"25664","inReplyTo":"m2lj54u9uj.fsf@igel.home","subject":"Re: import determinism","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2010-11-07T22:20:17Z","receivedAt":"2010-11-07T22:20:17Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Sun, Nov 7, 2010 at 4:01 PM, Andreas Schwab wrote:\n> The committer info in every commit will be the same as the author info,\n> so the repository conversion is completely deterministic.\n\nFor cvsimport, it is not deterministic. Given cvs'\nambiguous/buggy/inconsistent internal semantics around some\noperations, cvsps makes educated guesses about what happened.\n\nLater commits can affect those educated guesses.\n\nIIRC, this affects the exact tree at the birth of a branch, and files\nadded on a branch (instead of on HEAD).\n\nIf you use cvsimport in 'incremental' runs you won't have this\nproblem. Whatever it decided to do, sticks.\n\nI haven't worked in detail with Keith Packard's parsecvs but I assume\nit has similar issues.\n\nhth,\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"155350","messageId":"m2d3qgu50c.fsf@igel.home","threadId":"25664","inReplyTo":"AANLkTikXxM=CfU2dKAY9khi1_tAsGDdUEc8S5AxooGH9@mail.gmail.com","subject":"Re: import determinism","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2010-11-07T22:45:55Z","receivedAt":"2010-11-07T22:45:55Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Martin Langhoff writes:\n\n> On Sun, Nov 7, 2010 at 4:01 PM, Andreas Schwab wrote:\n>> The committer info in every commit will be the same as the author info,\n>> so the repository conversion is completely deterministic.\n>\n> For cvsimport, it is not deterministic. Given cvs'\n> ambiguous/buggy/inconsistent internal semantics around some\n> operations, cvsps makes educated guesses about what happened.\n>\n> Later commits can affect those educated guesses.\n\nThe OP was assuming an unchanging repository.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"155488","messageId":"20101109134337.GA19430@nibiru.local","threadId":"25664","inReplyTo":"m2d3qgu50c.fsf@igel.home","subject":"Re: import determinism","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-11-09T13:43:37Z","receivedAt":"2010-11-09T13:43:37Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Andreas Schwab wrote:\n\n> > For cvsimport, it is not deterministic. Given cvs'\n> > ambiguous/buggy/inconsistent internal semantics around some\n> > operations, cvsps makes educated guesses about what happened.\n> >\n> > Later commits can affect those educated guesses.\n> \n> The OP was assuming an unchanging repository.\n\nMy assumption is:\n\n* the original cvs repo will have later additions\n (so I'm incrementally importing)\n* no commints (besides cvsimport) in the git mirror, but others\n for off from there\n* the mirror could get lost in an desaster (no separate backup)\n and should be recreated afresh in that case.\n\n\nThe point behind this is: I'm running a growing number of cvs2git\nmirrors and dont want to do full backups of them.\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"155571","messageId":"AANLkTi=y_DKRRU43ro3WDz2rSDGL9xASfsinoj8Ya9PT@mail.gmail.com","threadId":"25664","inReplyTo":"20101109134337.GA19430@nibiru.local","subject":"Re: import determinism","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2010-11-10T04:40:52Z","receivedAt":"2010-11-10T04:40:52Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"Every full clone is a good backup. If nobody is doing a full clone...\nthe do back them up. That's my recommendation.\n\n\nM\n\nOn Tuesday, November 9, 2010, Enrico Weigelt wrote:\n> * Andreas Schwab wrote:\n>\n>> > For cvsimport, it is not deterministic. Given cvs'\n>> > ambiguous/buggy/inconsistent internal semantics around some\n>> > operations, cvsps makes educated guesses about what happened.\n>> >\n>> > Later commits can affect those educated guesses.\n>>\n>> The OP was assuming an unchanging repository.\n>\n> My assumption is:\n>\n> * the original cvs repo will have later additions\n>  (so I'm incrementally importing)\n> * no commints (besides cvsimport) in the git mirror, but others\n>  for off from there\n> * the mirror could get lost in an desaster (no separate backup)\n>  and should be recreated afresh in that case.\n>\n>\n> The point behind this is: I'm running a growing number of cvs2git\n> mirrors and dont want to do full backups of them.\n>\n>\n> cu\n> --\n> ----------------------------------------------------------------------\n>  Enrico Weigelt, metux IT service -- http://www.metux.de/\n>\n>  phone:  +49 36207 519931  email: weigelt@metux.de\n>  mobile: +49 151 27565287  icq:   210169427         skype: nekrad666\n> ----------------------------------------------------------------------\n>  Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n> ----------------------------------------------------------------------\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"155597","messageId":"20101110161804.GA29619@nibiru.local","threadId":"25664","inReplyTo":"AANLkTi=y_DKRRU43ro3WDz2rSDGL9xASfsinoj8Ya9PT@mail.gmail.com","subject":"Re: import determinism","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-11-10T16:18:31Z","receivedAt":"2010-11-10T16:18:31Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Martin Langhoff wrote:\n> Every full clone is a good backup. If nobody is doing a full clone...\n> the do back them up. That's my recommendation.\n\nDoes the clone have all information necessary for the incremental\ncvsimport runs ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"155627","messageId":"AANLkTin-SxiTMZOeucS5_KKpaCz=PJ3yWjPFxB0bhCQ2@mail.gmail.com","threadId":"25664","inReplyTo":"20101110161804.GA29619@nibiru.local","subject":"Re: import determinism","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2010-11-10T21:25:40Z","receivedAt":"2010-11-10T21:25:40Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Wed, Nov 10, 2010 at 11:18 AM, Enrico Weigelt wrote:\n> * Martin Langhoff wrote:\n>> Every full clone is a good backup. If nobody is doing a full clone...\n>> the do back them up. That's my recommendation.\n>\n> Does the clone have all information necessary for the incremental\n> cvsimport runs ?\n\nYes -- all cvsimport tracks is the branches. No extra import-tracking\ndata is maintained. (Other importers may or may not keep additional\ndata, YMMV, etc)\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"155630","messageId":"20101110220416.GA26994@nibiru.local","threadId":"25664","inReplyTo":"AANLkTin-SxiTMZOeucS5_KKpaCz=PJ3yWjPFxB0bhCQ2@mail.gmail.com","subject":"Re: import determinism","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-11-10T22:04:16Z","receivedAt":"2010-11-10T22:04:16Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Martin Langhoff wrote:\n\n> > Does the clone have all information necessary for the incremental\n> > cvsimport runs ?\n> \n> Yes -- all cvsimport tracks is the branches. No extra import-tracking\n> data is maintained. (Other importers may or may not keep additional\n> data, YMMV, etc)\n\nThat's good news. Now I'll have to sit down and code some little\ncluster management tool (mirrors are getting more and more), or\ndoes anyone already did that ? ...\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"155645","messageId":"4CDB70D0.6000405@alum.mit.edu","threadId":"25664","inReplyTo":"20101109134337.GA19430@nibiru.local","subject":"Re: import determinism","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2010-11-11T04:28:00Z","receivedAt":"2010-11-11T04:28:00Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 11/07/2010 09:25 PM, Enrico Weigelt wrote:\n> I'm curious on how deterministic the imports (git-cvsimport and\n> git-svn) are. Suppose I close the same cvs repo twice (assuming\n> no write access in between), are the resulting object SHA-1's\n> the same ?\n\nOn 11/09/2010 02:43 PM, Enrico Weigelt wrote:\n> The point behind this is: I'm running a growing number of cvs2git\n> mirrors and dont want to do full backups of them.\n\nIf you are using cvs2git, why are you asking about git-cvsimport and\ngit-svn?\n\nNo tool that imports from CVS or Subversion can make a blanket guarantee\nabout consistency across conversions because both CVS and SVN allow\nretroactive changes to the project history. For example:\n\n* Both CVS and SVN allow commit messages and other metadata of old\ncommits to be changed\n\n* CVS allows files to be added retroactively to tags and branches with\nno timestamp indicating that the file was not part of the original tag.\n\n* CVS allows old revisions to be \"obsoleted\" (i.e., expunged from history).\n\n* In CVS it is common practice for people to muck about directly in the\nrepository, for example renaming *,v files.\n\nSo (in the general case) there is no way to guarantee that two\nindependent conversions will have consistent results for the overlapping\nparts of their history. And even incremental conversions will\nnecessarily have to decide between converting the current state of the\nrepository accurately and converting in a way that is consistent with\nearlier conversions.\n\nIn practice, especially if you are willing to constrain what the CVS\nusers are allowed to do, the overlapping parts of two conversions should\nusually be identical or at least very similar (with older history more\nlikely to be identical). Perhaps an rsync-style backup would be smart\nenough to copy only the changed part of the history without excluding\nthe possibility that there are retroactive changes between subsequent\nconversions.\n\nIf you run two cvs2git conversions on *exactly* the same CVS repository,\nthen the results *should* be identical. I have tried always to process\ndata in a defined order rather than, say, in filesystem or\nhashmap-determined order. But AFAIK this property has not been tested\nand could easily be buggy if I overlooked some source of indeterminism\nsomewhere in the cvs2git code.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"155685","messageId":"20101111130914.GA23423@nibiru.local","threadId":"25664","inReplyTo":"4CDB70D0.6000405@alum.mit.edu","subject":"Re: import determinism","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-11-11T13:09:14Z","receivedAt":"2010-11-11T13:09:14Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Michael Haggerty wrote:\n> On 11/07/2010 09:25 PM, Enrico Weigelt wrote:\n> > I'm curious on how deterministic the imports (git-cvsimport and\n> > git-svn) are. Suppose I close the same cvs repo twice (assuming\n> > no write access in between), are the resulting object SHA-1's\n> > the same ?\n> \n> On 11/09/2010 02:43 PM, Enrico Weigelt wrote:\n> > The point behind this is: I'm running a growing number of cvs2git\n> > mirrors and dont want to do full backups of them.\n> \n> If you are using cvs2git, why are you asking about git-cvsimport and\n> git-svn?\n\nNo, I'm not using the \"cvs2git\" tool (wasn't aware of that at all) ;-)\nI'm using git-cvsimport.\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"}]}