{"thread":{"id":"1942","subject":"rsync deprecated but promoted?","startedAt":"2005-09-25T16:32:01Z","lastAt":"2005-11-10T23:17:04Z","messageCount":26,"participants":["Zack Brown","H. Peter Anvin","Martin Coxall","Petr Baudis","Brian Gerst","Linus Torvalds","walt","Johannes Schindelin","Junio C Hamano","Daniel Barkalow","Matthias Urlichs","Sergey Vlasov","A Large Angry SCM"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"9261","messageId":"20050925163201.GA29198@tumblerings.org","threadId":"1942","inReplyTo":null,"subject":"rsync deprecated but promoted?","fromName":"Zack Brown","fromEmail":"zbrown@tumblerings.org","sentAt":"2005-09-25T16:32:01Z","receivedAt":"2005-09-25T16:32:01Z","isPatch":false,"sender":{"key":"zbrown@tumblerings.org","avatar":null},"body":"Hi folks,\n\nWhen I use cogito, it gives a warning saying the rsync method is deprecated and\nwill be removed in the future. But when I visit kernel.org/git, the page says to\nuse an rsync URL with cg-clone.\n\nMaybe kernel.org should be updated?\n\nBe well,\nZack\n\n-- \nZack Brown\n"},{"id":"9264","messageId":"4336D957.4040201@zytor.com","threadId":"1942","inReplyTo":"20050925163201.GA29198@tumblerings.org","subject":"Re: rsync deprecated but promoted?","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-09-25T17:07:35Z","receivedAt":"2005-09-25T17:07:35Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Zack Brown wrote:\n> Hi folks,\n> \n> When I use cogito, .  it gives a warning saying the rsync method is deprecated and\n> will be removed in the future. But when I visit kernel.org/git, the page says to\n> use an rsync URL with cg-clone.\n> \n> Maybe kernel.org should be updated?\n\nNo, since it's currently the only method available to the general \npublic. git-daemon still needs some tweaking before I trust to enable \nit; I've been meaning to do this but I've been personally very busy.\n\n\t-hpa\n"},{"id":"9269","messageId":"4d4586301dca616f42880612fae01492@cream.org","threadId":"1942","inReplyTo":"20050925163201.GA29198@tumblerings.org","subject":"Re: rsync deprecated but promoted?","fromName":"Martin Coxall","fromEmail":"quasi@cream.org","sentAt":"2005-09-25T19:06:37Z","receivedAt":"2005-09-25T19:06:37Z","isPatch":false,"sender":{"key":"quasi@cream.org","avatar":null},"body":"\nOn 25 Sep 2005, at 17:32, Zack Brown wrote:\n\n> Hi folks,\n>\n> When I use cogito, it gives a warning saying the rsync method is \n> deprecated and\n> will be removed in the future. But when I visit kernel.org/git, the \n> page says to\n> use an rsync URL with cg-clone.\n>\n> Maybe kernel.org should be updated?\n>\n\nIt does seem to be sending out a confusing message to us users too, \nsince an initial clone of Linus's tree with rsync is on my machine 10x \nfaster than an http clone, so it seems to be sending out something of a \nconfused/confusing message re: rsync.\n\nAm I right in thinking it's because rsync didn't originally have pack \nsupport, but now it does, Petr has simply forgotten to deprecate the \ndeprecation message?\n\nKind Regards,\n\nMartin\n"},{"id":"9298","messageId":"20050926133204.GB21019@pasky.or.cz","threadId":"1942","inReplyTo":"4d4586301dca616f42880612fae01492@cream.org","subject":"Re: rsync deprecated but promoted?","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-09-26T13:32:04Z","receivedAt":"2005-09-26T13:32:04Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sun, Sep 25, 2005 at 09:06:37PM CEST, I got a letter\nwhere Martin Coxall <quasi@cream.org> told me that...\n> On 25 Sep 2005, at 17:32, Zack Brown wrote:\n> >Hi folks,\n> >\n> >When I use cogito, it gives a warning saying the rsync method is \n> >deprecated and\n> >will be removed in the future. But when I visit kernel.org/git, the \n> >page says to\n> >use an rsync URL with cg-clone.\n> >\n> >Maybe kernel.org should be updated?\n> >\n> \n> It does seem to be sending out a confusing message to us users too, \n> since an initial clone of Linus's tree with rsync is on my machine 10x \n> faster than an http clone, so it seems to be sending out something of a \n> confused/confusing message re: rsync.\n> \n> Am I right in thinking it's because rsync didn't originally have pack \n> support, but now it does, Petr has simply forgotten to deprecate the \n> deprecation message?\n\nNope. rsync always did packs, I actually un-deprecated it for the time\nperiod when HTTP didn't. The thing is, rsync is bad - it will happily\nput duplicate, redundant, and especially unwanted data to your\nrepository, especially when the shared GIT repositories happen. HTTP and\ngit-daemon are much better access methods in this regard - actually, I\nstill like HTTP the most:\n\n+ Works everywhere - no special setup, no dedicated service, firewalls\nand proxies won't stop it\n+ Works properly, i.e. only getting stuff you want, unlike rsync\n+ Replicates packs setup - would be even better if it would kill objects\nand packs which the new pack makes redundant\n\n\tIt would be best to have some smarter git-prune-packed, which\n\twould process just a single pack. The other alternative would be\n\tthat it would prune packs being subsets of other packs as well,\n\tbut that scaled bad. I will write another mail about that.\n\n- It is slow. Actually, I think it should be much faster for incremental\nfetches, and the initial fetch should take about the same time if you\nuse packs. But the question is, did we already hit the limit? Are we\nusing HTTP keepalive connections, do we parallelize the requests?\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"9299","messageId":"433808B2.3070508@didntduck.org","threadId":"1942","inReplyTo":"20050926133204.GB21019@pasky.or.cz","subject":"Re: rsync deprecated but promoted?","fromName":"Brian Gerst","fromEmail":"bgerst@didntduck.org","sentAt":"2005-09-26T14:41:54Z","receivedAt":"2005-09-26T14:41:54Z","isPatch":false,"sender":{"key":"bgerst@didntduck.org","avatar":null},"body":"Petr Baudis wrote:\n> Dear diary, on Sun, Sep 25, 2005 at 09:06:37PM CEST, I got a letter\n> where Martin Coxall <quasi@cream.org> told me that...\n> \n>>On 25 Sep 2005, at 17:32, Zack Brown wrote:\n>>\n>>>Hi folks,\n>>>\n>>>When I use cogito, it gives a warning saying the rsync method is \n>>>deprecated and\n>>>will be removed in the future. But when I visit kernel.org/git, the \n>>>page says to\n>>>use an rsync URL with cg-clone.\n>>>\n>>>Maybe kernel.org should be updated?\n>>>\n>>\n>>It does seem to be sending out a confusing message to us users too, \n>>since an initial clone of Linus's tree with rsync is on my machine 10x \n>>faster than an http clone, so it seems to be sending out something of a \n>>confused/confusing message re: rsync.\n>>\n>>Am I right in thinking it's because rsync didn't originally have pack \n>>support, but now it does, Petr has simply forgotten to deprecate the \n>>deprecation message?\n> \n> \n> Nope. rsync always did packs, I actually un-deprecated it for the time\n> period when HTTP didn't. The thing is, rsync is bad - it will happily\n> put duplicate, redundant, and especially unwanted data to your\n> repository, especially when the shared GIT repositories happen. HTTP and\n> git-daemon are much better access methods in this regard - actually, I\n> still like HTTP the most:\n> \n> + Works everywhere - no special setup, no dedicated service, firewalls\n> and proxies won't stop it\n> + Works properly, i.e. only getting stuff you want, unlike rsync\n> + Replicates packs setup - would be even better if it would kill objects\n> and packs which the new pack makes redundant\n> \n> \tIt would be best to have some smarter git-prune-packed, which\n> \twould process just a single pack. The other alternative would be\n> \tthat it would prune packs being subsets of other packs as well,\n> \tbut that scaled bad. I will write another mail about that.\n> \n> - It is slow. Actually, I think it should be much faster for incremental\n> fetches, and the initial fetch should take about the same time if you\n> use packs. But the question is, did we already hit the limit? Are we\n> using HTTP keepalive connections, do we parallelize the requests?\n> \n\nThe current HTTP fetch doesn't do asynchronous requests (using \ncurl_multi_*).  This means that no transfers occur while processing \nreceived objects.\n\nThe other problem with HTTP vs. rsync is that the HTTP fetch will walk \nthe entire tree down to the root to verify it has every object.  While \nthis isn't a bad thing it's usually unnecessary when it's all in one big \npack file.\n\n--\n\t\t\t\tBrian Gerst\n"},{"id":"9300","messageId":"Pine.LNX.4.58.0509260801430.3308@g5.osdl.org","threadId":"1942","inReplyTo":"20050926133204.GB21019@pasky.or.cz","subject":"Re: rsync deprecated but promoted?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-26T15:04:25Z","receivedAt":"2005-09-26T15:04:25Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 26 Sep 2005, Petr Baudis wrote:\n> \n> Nope. rsync always did packs, I actually un-deprecated it for the time\n> period when HTTP didn't. The thing is, rsync is bad - it will happily\n> put duplicate, redundant, and especially unwanted data to your\n> repository, especially when the shared GIT repositories happen.\n\nWorse than that, rsync will happily sync up to a remote repository without \neven getting _all_ the object files, and never tell you anything is wrong.\n\nThis happened to several people when the kernel.org mirroring was \nbroken/delayed.\n\nSo yes, rsync is fast. But it's fast exactly _because_ it is broken. Very \nvery fundamentally broken.\n\nYou basically have to run fsck on your repository after an rsync. And if \nit returns errors, you're screwed unless you remember what your old heads \nwere.\n\n\t\tLinus\n"},{"id":"9301","messageId":"20050926163604.GC21019@pasky.or.cz","threadId":"1942","inReplyTo":"433808B2.3070508@didntduck.org","subject":"Re: rsync deprecated but promoted?","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-09-26T16:36:04Z","receivedAt":"2005-09-26T16:36:04Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Mon, Sep 26, 2005 at 04:41:54PM CEST, I got a letter\nwhere Brian Gerst <bgerst@didntduck.org> told me that...\n> The current HTTP fetch doesn't do asynchronous requests (using \n> curl_multi_*).  This means that no transfers occur while processing \n> received objects.\n\nThat should be fixed then, so that we fully utilize the network.\n\n> The other problem with HTTP vs. rsync is that the HTTP fetch will walk \n> the entire tree down to the root to verify it has every object.  While \n> this isn't a bad thing it's usually unnecessary when it's all in one big \n> pack file.\n\nIs that really the case? I believe it will walk only to the original ref\nand assume everything before is complete. (Actually, it doesn't even\nseem to honor the --recover patch anymore, which isn't so nice\nespecially in case some objects disappeared from your database and you\nwould like to get them back. Happenned to me.)\n\nBut there were changes in that not so long ago, so maybe I'm still\nconfused.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"9302","messageId":"20050926163846.GD21019@pasky.or.cz","threadId":"1942","inReplyTo":"Pine.LNX.4.58.0509260801430.3308@g5.osdl.org","subject":"Re: rsync deprecated but promoted?","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-09-26T16:38:46Z","receivedAt":"2005-09-26T16:38:46Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Mon, Sep 26, 2005 at 05:04:25PM CEST, I got a letter\nwhere Linus Torvalds <torvalds@osdl.org> told me that...\n> You basically have to run fsck on your repository after an rsync. And if \n> it returns errors, you're screwed unless you remember what your old heads \n> were.\n\nActually, it would be nice to be able to tell git-fsck-objects to only\nverify objects which are referenced between given two commits (perhaps\njust make it support the ^object notation). Then I wouldn't mind running\nthat after each rsync fetch in Cogito.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"9303","messageId":"Pine.LNX.4.58.0509260941340.3308@g5.osdl.org","threadId":"1942","inReplyTo":"20050926163846.GD21019@pasky.or.cz","subject":"Re: rsync deprecated but promoted?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-26T16:43:08Z","receivedAt":"2005-09-26T16:43:08Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 26 Sep 2005, Petr Baudis wrote:\n> \n> Actually, it would be nice to be able to tell git-fsck-objects to only\n> verify objects which are referenced between given two commits (perhaps\n> just make it support the ^object notation). Then I wouldn't mind running\n> that after each rsync fetch in Cogito.\n\nYou can kind of do it.\n\nDo\n\n\tgit-rev-list --objects $oldheads --not $newheads >& /dev/null\n\techo \"$?\"\n\nand it _should_ largely work. Untested, of course, but I _hope_ that if \nany object is missing, git-rev-list should die with an error. And if it \ndoesn't, I should fix it ;)\n\n\t\tLinus\n"},{"id":"9305","messageId":"dh98gk$6rp$1@sea.gmane.org","threadId":"1942","inReplyTo":"Pine.LNX.4.58.0509260801430.3308@g5.osdl.org","subject":"Re: rsync deprecated but promoted?","fromName":"walt","fromEmail":"wa1ter@myrealbox.com","sentAt":"2005-09-26T16:44:04Z","receivedAt":"2005-09-26T16:44:04Z","isPatch":false,"sender":{"key":"wa1ter@myrealbox.com","avatar":null},"body":"Linus Torvalds wrote:\n[...]\n> You basically have to run fsck on your repository after an rsync. And if \n> it returns errors, you're screwed unless you remember what your old heads \n> were.\n\nJust because you mentioned it, I did a git-fsck-objects on my local\ncopies of your kernel tree and Junio's git tree.\n\n From git I got this:\n$git-fsck-objects\nmissing commit 00d8bbd3c4bba72a6dfd48c2c0c9cbaa000f13c2\nbroken link from     tag 02b2acff8bafb6d73c6513469cdda0c6c18c4138\n               to  commit d5bc7eecbbb0b9f6122708bf5cd62f78ebdaafd8\n<similar lines snipped>\n\n From your tree I got only this single line:\ndangling commit 02459eaab98a6a57717bc0cacede148fc76af881\n\nYet both trees compile and run perfectly.  Are these messages\nworrisome?  (BTW, git was cloned and updated using http.)\n"},{"id":"9304","messageId":"43382614.6080907@didntduck.org","threadId":"1942","inReplyTo":"20050926163604.GC21019@pasky.or.cz","subject":"Re: rsync deprecated but promoted?","fromName":"Brian Gerst","fromEmail":"bgerst@didntduck.org","sentAt":"2005-09-26T16:47:16Z","receivedAt":"2005-09-26T16:47:16Z","isPatch":false,"sender":{"key":"bgerst@didntduck.org","avatar":null},"body":"Petr Baudis wrote:\n> Dear diary, on Mon, Sep 26, 2005 at 04:41:54PM CEST, I got a letter\n> where Brian Gerst <bgerst@didntduck.org> told me that...\n> \n>>The other problem with HTTP vs. rsync is that the HTTP fetch will walk \n>>the entire tree down to the root to verify it has every object.  While \n>>this isn't a bad thing it's usually unnecessary when it's all in one big \n>>pack file.\n> \n> \n> Is that really the case? I believe it will walk only to the original ref\n> and assume everything before is complete. (Actually, it doesn't even\n> seem to honor the --recover patch anymore, which isn't so nice\n> especially in case some objects disappeared from your database and you\n> would like to get them back. Happenned to me.)\n\nI was talking about the initial pull.  It does stop at the previous head \nfor updates.\n\n--\n\t\t\t\tBrian Gerst\n"},{"id":"9310","messageId":"Pine.LNX.4.58.0509261038460.3308@g5.osdl.org","threadId":"1942","inReplyTo":"dh98gk$6rp$1@sea.gmane.org","subject":"Re: rsync deprecated but promoted?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-26T17:55:49Z","receivedAt":"2005-09-26T17:55:49Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 26 Sep 2005, walt wrote:\n> \n> Just because you mentioned it, I did a git-fsck-objects on my local\n> copies of your kernel tree and Junio's git tree.\n> \n>  From git I got this:\n> $git-fsck-objects\n> missing commit 00d8bbd3c4bba72a6dfd48c2c0c9cbaa000f13c2\n> broken link from     tag 02b2acff8bafb6d73c6513469cdda0c6c18c4138\n>                to  commit d5bc7eecbbb0b9f6122708bf5cd62f78ebdaafd8\n> <similar lines snipped>\n> \n>  From your tree I got only this single line:\n> dangling commit 02459eaab98a6a57717bc0cacede148fc76af881\n\nThat commit shouldn't be dangling, but I suspect it is harmless and is\nmost likely because you have pack-files. Use \"git-fsck-cache --full\" if\nyou are downloading with http/rsync (since that gets packs without\nunpacking them, and you haven't re-packed everything).\n\nThe git thing may be similar, although it sounds unlikely. A more likely\nreason is that earlier http pulling got incomplete trees if you ever\ninterrupted it with ^C.\n\n> Yet both trees compile and run perfectly.  Are these messages\n> worrisome?  (BTW, git was cloned and updated using http.)\n\nYes, they can be worrisome. Some of it may be normal (I really suspect \nthat the kernel tree is that kind - a \"dangling commit\" is almost always \neither because you've lost a tag or because of a pack-file that wasn't \nexamined).\n\nYour git tree is quote possibly corrupted.\n\nThe good news is that if \"git checkout\" works, then the corruption is all\nold - you may not have all of the history, but the corruption is\n\"harmless\".\n\nThere's nothing fundamentally wrong with not having all of history: it\nwill cause fsck to complain (unless you \"plug\" the history by using a\ngraft file). And obviously it means that you may not be able to go back in \ntime - but you may never even care. \n\nA \"git-http-fetch --recover HEAD <url>\" _should_ fix it, but I don't think \nthat works right now. It's documented, but it doesn't do anything. Junio?\n\n\t\t\tLinus\n"},{"id":"9316","messageId":"dh9hqs$6nl$1@sea.gmane.org","threadId":"1942","inReplyTo":"Pine.LNX.4.58.0509261038460.3308@g5.osdl.org","subject":"Re: rsync deprecated but promoted?","fromName":"walt","fromEmail":"wa1ter@myrealbox.com","sentAt":"2005-09-26T19:23:07Z","receivedAt":"2005-09-26T19:23:07Z","isPatch":false,"sender":{"key":"wa1ter@myrealbox.com","avatar":null},"body":"Linus Torvalds wrote:\n> \n> On Mon, 26 Sep 2005, walt wrote:\n>> Just because you mentioned it, I did a git-fsck-objects on my local\n>> copies of your kernel tree and Junio's git tree.\n>>\n>>  From git I got this:\n>> $git-fsck-objects\n>> missing commit 00d8bbd3c4bba72a6dfd48c2c0c9cbaa000f13c2\n>> broken link from     tag 02b2acff8bafb6d73c6513469cdda0c6c18c4138\n>>                to  commit d5bc7eecbbb0b9f6122708bf5cd62f78ebdaafd8\n>> <similar lines snipped>\n>>\n>>  From your tree I got only this single line:\n>> dangling commit 02459eaab98a6a57717bc0cacede148fc76af881\n> \n> That commit shouldn't be dangling, but I suspect it is harmless and is\n> most likely because you have pack-files. Use \"git-fsck-cache --full\"\n\nUsing the --full flag made the error disappear for your kernel tree,\nbut had no effect on the git tree.\n\nI neglected to mention that I use cg-clone and cg-update rather than\nthe git equivalents.  (cogito 0.15.1 from kernel.org)\n\n\n> Your git tree is quote possibly corrupted.\n\nI recloned from http://kernel.org and I still get exactly the same fsck\nerrors for git, with or without the --full flag.\n\n(I mention this only FYI.  I'm not having any problems compiling or\nusing either git or the kernel.)\n"},{"id":"9318","messageId":"Pine.LNX.4.63.0509262208550.12539@wgmdd8.biozentrum.uni-wuerzburg.de","threadId":"1942","inReplyTo":"dh9hqs$6nl$1@sea.gmane.org","subject":"Re: rsync deprecated but promoted?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2005-09-26T20:12:09Z","receivedAt":"2005-09-26T20:12:09Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 26 Sep 2005, walt wrote:\n\n> Using the --full flag made the error disappear for your kernel tree,\n> but had no effect on the git tree.\n\nI think it is because of the \"pu\" branch, which gets fetched using rsync, \nbut no ref is pointing to it. Since the \"pu\" branch is rebased quite \noften, it would also happen if you fetched the \"pu\" branch, though.\n\nCiao,\nDscho\n"},{"id":"9320","messageId":"7v7jd3inuz.fsf@assigned-by-dhcp.cox.net","threadId":"1942","inReplyTo":"dh9hqs$6nl$1@sea.gmane.org","subject":"Re: rsync deprecated but promoted?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-26T20:19:00Z","receivedAt":"2005-09-26T20:19:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"walt <wa1ter@myrealbox.com> writes:\n\n>>>  From git I got this:\n>>> $git-fsck-objects\n>>> missing commit 00d8bbd3c4bba72a6dfd48c2c0c9cbaa000f13c2\n>>> broken link from     tag 02b2acff8bafb6d73c6513469cdda0c6c18c4138\n>>>                to  commit d5bc7eecbbb0b9f6122708bf5cd62f78ebdaafd8\n>>> <similar lines snipped>\n\n>> Your git tree is quote possibly corrupted.\n>\n> I recloned from http://kernel.org and I still get exactly the same fsck\n> errors for git, with or without the --full flag.\n\nThat 00d8bbd3c4bba72a6dfd48c2c0c9cbaa000f13c2 is v0.99.7c commit.\n\nWhile I do not doubt your git repository is missing that object,\nit probably is a buggy clone method.  Let me see if I can\nreproduce.\n\n    $ git clone http://kernel.org/pub/scm/git/git.git/ git-clone\n    (says a lot of \"got\" and \"walk\" here)\n    $ cd git-clone\n    $ git-cat-file -t 00d8bbd3c4bba72a6dfd48c2c0c9cbaa000f13c2\n    commit\n    $ git-fsck-objects\n    $ git-fsck-objects --full\n    $ exit\n\nNope.  Things look OK from here.\n"},{"id":"9324","messageId":"20050926204304.GE26340@pasky.or.cz","threadId":"1942","inReplyTo":"dh98gk$6rp$1@sea.gmane.org","subject":"Re: rsync deprecated but promoted?","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-09-26T20:43:04Z","receivedAt":"2005-09-26T20:43:04Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Mon, Sep 26, 2005 at 06:44:04PM CEST, I got a letter\nwhere walt <wa1ter@myrealbox.com> told me that...\n> Linus Torvalds wrote:\n> [...]\n> >You basically have to run fsck on your repository after an rsync. And if \n> >it returns errors, you're screwed unless you remember what your old heads \n> >were.\n> \n> Just because you mentioned it, I did a git-fsck-objects on my local\n> copies of your kernel tree and Junio's git tree.\n> \n> From git I got this:\n> $git-fsck-objects\n> missing commit 00d8bbd3c4bba72a6dfd48c2c0c9cbaa000f13c2\n> broken link from     tag 02b2acff8bafb6d73c6513469cdda0c6c18c4138\n>               to  commit d5bc7eecbbb0b9f6122708bf5cd62f78ebdaafd8\n> <similar lines snipped>\n\nThis isn't too harmful. It just means that you have a tag ref and the\ncorresponding tag object, but not the commit tagged by that object.\nThis is nothing harmful as long as you don't try to reference the tag,\nand if you don't have the commit object already, it's actually not quite\nlikely that you would, since you don't have the branch the bug belongs\nto anyway. I'll hopefully fix this bug during the weekend.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"},{"id":"9337","messageId":"Pine.LNX.4.63.0509261808530.23242@iabervon.org","threadId":"1942","inReplyTo":"Pine.LNX.4.58.0509261038460.3308@g5.osdl.org","subject":"Re: rsync deprecated but promoted?","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-09-26T22:13:43Z","receivedAt":"2005-09-26T22:13:43Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 26 Sep 2005, Linus Torvalds wrote:\n\n> A \"git-http-fetch --recover HEAD <url>\" _should_ fix it, but I don't think \n> that works right now. It's documented, but it doesn't do anything. Junio?\n\nI should actually implement that now that it's easy; you just skip the \n\"for_each_ref(mark_complete);\" on line 217 of fetch.c, and it'll make sure \nthat it has everything.\n\n(I'll make a patch tonight if nobody beats me to it.)\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"9345","messageId":"7vy85j8nfk.fsf@assigned-by-dhcp.cox.net","threadId":"1942","inReplyTo":"Pine.LNX.4.63.0509261808530.23242@iabervon.org","subject":"Re: rsync deprecated but promoted?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-26T22:38:23Z","receivedAt":"2005-09-26T22:38:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> writes:\n\n> I should actually implement that now that it's easy; you just skip the \n> \"for_each_ref(mark_complete);\" on line 217 of fetch.c, and it'll make sure \n> that it has everything.\n>\n> (I'll make a patch tonight if nobody beats me to it.)\n\nThanks.\n"},{"id":"9367","messageId":"pan.2005.09.27.06.35.35.834134@smurf.noris.de","threadId":"1942","inReplyTo":"20050926133204.GB21019@pasky.or.cz","subject":"hared GIT repos (was Re: rsync deprecated but promoted?)","fromName":"Matthias Urlichs","fromEmail":"smurf@smurf.noris.de","sentAt":"2005-09-27T06:35:37Z","receivedAt":"2005-09-27T06:35:37Z","isPatch":false,"sender":{"key":"matthias@urlichs.de","avatar":"https://gravatar.com/avatar/2708905af227313eba6f2b2ae0f7d0259b5ac5d71baef58fe5a13c699ce0bbf0?d=mp&s=160"},"body":"Hi, Petr Baudis wrote:\n\n> The thing is, rsync is bad - it will happily put\n> duplicate, redundant, and especially unwanted data to your repository,\n> especially when the shared GIT repositories happen.\n\nSpeaking of which -- is anybody working on that one?\n\nI find myself in need of a multiuser shared repository that cannot\nbe corrupted (i.e. I want to prevent the users from removing objects,\nand replacing a ref with something that is not a child of the sha1 that's\nalready there should also be prevented).\n\n-- \nMatthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de\nDisclaimer: The quote was selected randomly. Really. | http://smurf.noris.de\n - -\nBeware of bugs in the above code; I have only proved it correct, not tried it.\n\t\t-- Donald Knuth\n"},{"id":"9369","messageId":"7vu0g70yqg.fsf_-_@assigned-by-dhcp.cox.net","threadId":"1942","inReplyTo":"pan.2005.09.27.06.35.35.834134@smurf.noris.de","subject":"Re: shared GIT repos","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-27T07:13:43Z","receivedAt":"2005-09-27T07:13:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthias Urlichs <smurf@smurf.noris.de> writes:\n\n> Speaking of which -- is anybody working on that one?\n>\n> I find myself in need of a multiuser shared repository that cannot\n> be corrupted (i.e. I want to prevent the users from removing objects,\n> and replacing a ref with something that is not a child of the sha1 that's\n> already there should also be prevented).\n\nDo you want to guard the repository from malicious users?  Or is\nit enough to guard a casual/careless user from making mistakes?\n\nIf one has commit privileges, then one can already do enough\nharm to the project without being able to remove objects nor\nupdating a ref with non-fast-forward ref.  So let's assume for\nnow that malicious users are not something we worry about.  In\nthat case, \"working on\" might be too scary a word.  I think most\nof the pieces are already there and you only need to assemble\nthem and write a howto ;-).\n\n - Place the users that has write access to the repository in\n   the same Unix group, and have the repository owned by that\n   group;\n\n - Give the users ssh access, perhaps with authorized_keys set\n   up to only allow running git-receive-pack and nothing else\n   (like normal shell access);\n\n - Set up hooks/update to make sure the ref updates are fast\n   forward.  Additionally, you could set up a mapping that says\n   which user can/cannot update which refs if you wanted to.\n\n-jc\n"},{"id":"9375","messageId":"20050927084513.GU31276@kiste.smurf.noris.de","threadId":"1942","inReplyTo":"7vu0g70yqg.fsf_-_@assigned-by-dhcp.cox.net","subject":"Re: shared GIT repos","fromName":"Matthias Urlichs","fromEmail":"smurf@smurf.noris.de","sentAt":"2005-09-27T08:45:13Z","receivedAt":"2005-09-27T08:45:13Z","isPatch":false,"sender":{"key":"matthias@urlichs.de","avatar":"https://gravatar.com/avatar/2708905af227313eba6f2b2ae0f7d0259b5ac5d71baef58fe5a13c699ce0bbf0?d=mp&s=160"},"body":"Hi,\n\nJunio C Hamano:\n> Do you want to guard the repository from malicious users?  Or is\n> it enough to guard a casual/careless user from making mistakes?\n> \nWell, s/malicious users/somebody who wants to cover up an ugly mistake/\nwould be more accurate.\n\nWhat I am doing: I'm writing a system management frontend which allows\npeople to install version-controlled stuff (like, the configuration for\na backup server, or Yet Another PHPBB Installation) on servers -- without\neven having a login there.\n\nSome of these contain login scripts that might need root privileges or\nsimilar (like, \"restart Apache\"). I want people to be unable to simply\nremove the commit that included the \"rm -rf /\" command, move the ref\nback, upload a new version, and pretend that nothing happened *la la la*.\n\n> If one has commit privileges, then one can already do enough\n> harm to the project without being able to remove objects nor\n> updating a ref with non-fast-forward ref.\n\nBut in that case it's traceable what happened and whodunit.\n\n> I think most of the pieces are already there and you only need to\n> assemble them and write a howto ;-).\n> [ list ]\n\nOK, thanks, that helps. I'll write something up.\n\n-- \nMatthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de\nDisclaimer: The quote was selected randomly. Really. | http://smurf.noris.de\n - -\nWhen angry, count four; when very angry, swear.\n"},{"id":"9379","messageId":"20050927135903.6b20a76b.vsu@altlinux.ru","threadId":"1942","inReplyTo":"20050927084513.GU31276@kiste.smurf.noris.de","subject":"Re: shared GIT repos","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2005-09-27T09:59:03Z","receivedAt":"2005-09-27T09:59:03Z","isPatch":false,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Tue, 27 Sep 2005 10:45:13 +0200 Matthias Urlichs wrote:\n\n> > If one has commit privileges, then one can already do enough\n> > harm to the project without being able to remove objects nor\n> > updating a ref with non-fast-forward ref.\n> \n> But in that case it's traceable what happened and whodunit.\n\nDon't forget that the user who has rights to invoke git-receive-pack\ncan set the \"author\" and \"committer\" fields in his commits to anything\nhe wants - unless you check these fields in hooks/update.\n"},{"id":"9383","messageId":"pan.2005.09.27.10.29.32.813139@smurf.noris.de","threadId":"1942","inReplyTo":"20050927135903.6b20a76b.vsu@altlinux.ru","subject":"Re: shared GIT repos","fromName":"Matthias Urlichs","fromEmail":"smurf@smurf.noris.de","sentAt":"2005-09-27T10:29:32Z","receivedAt":"2005-09-27T10:29:32Z","isPatch":false,"sender":{"key":"matthias@urlichs.de","avatar":"https://gravatar.com/avatar/2708905af227313eba6f2b2ae0f7d0259b5ac5d71baef58fe5a13c699ce0bbf0?d=mp&s=160"},"body":"Hi, Sergey Vlasov wrote:\n\n> On Tue, 27 Sep 2005 10:45:13 +0200 Matthias Urlichs wrote:\n> \n>> > If one has commit privileges, then one can already do enough\n>> > harm to the project without being able to remove objects nor\n>> > updating a ref with non-fast-forward ref.\n>> \n>> But in that case it's traceable what happened and whodunit.\n> \n> Don't forget that the user who has rights to invoke git-receive-pack\n> can set the \"author\" and \"committer\" fields in his commits to anything\n> he wants - unless you check these fields in hooks/update.\n\nSure. I plan to; \"committer\" at least should match one of the user's known\nemail addresses. In addition to that, the files will belong to the user.\n\n-- \nMatthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de\nDisclaimer: The quote was selected randomly. Really. | http://smurf.noris.de\n - -\nNever count your chickens before they rip your lips off.\n"},{"id":"9391","messageId":"Pine.LNX.4.58.0509270807580.3308@g5.osdl.org","threadId":"1942","inReplyTo":"20050927084513.GU31276@kiste.smurf.noris.de","subject":"Re: shared GIT repos","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-27T15:21:16Z","receivedAt":"2005-09-27T15:21:16Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 27 Sep 2005, Matthias Urlichs wrote:\n> \n> Junio C Hamano:\n> > Do you want to guard the repository from malicious users?  Or is\n> > it enough to guard a casual/careless user from making mistakes?\n> > \n> Well, s/malicious users/somebody who wants to cover up an ugly mistake/\n> would be more accurate.\n\nHmm.\n\nWhat you _can_ do is to make your object and refs directories sticky.\n\nThat automatically means that only the owner of a file can remove it.\n\nNow, people can still cover up their _own_ mistakes in that case, but they \ncan't change other peoples branches (since that involves overwriting \nsomebody elses ref), and they can't remove objects that somebody else has \nwritten.\n\nBut they can, for example, change their _own_ branch to not have a ref to \nthat object, of course.\n\nA more draconian option is to make the git programs setgid to a \"git\" \ngroup, and making the object and ref directories only writable by the git \ngroup. And then you change all the git programs to verify whatever rules \nyou have. That requires pretty big changes, though.\n\nFor example, you'd have to make all the scripts use the new git-update-ref\nthing, and if you want to enforce that any new ref is a proper child of\nthe old ref, then you'd have to make git-update-ref test that one \nexplicitly (instead of leaving it to the scripts).\n\nQuite frankly, I'd rather avoid that.\n\nOh. One thing you can do: don't allow direct filesystem access at _all_.  \nUse ssh to log in (even if it's on the same machine) as a special user\nwhich is the only one that is allowed to touch the repo, and make that\nspecial users login shell only accept git-receive-pack.\n\nI wrote and posted an untested \"git-sh\" that did that some time ago, \nholler if you want it again.\n\nAdd logging, and testing, and it should give you a safe write-only\nalternative to \"git-daemon\" that only allows people to append to the git\nhistory (oh, you'd still have to add some _small_ code to git-receive-pack\nto not allow the \"ignore old ref contents\" case, but that's like two lines\nof code).\n\n\t\t\tLinus\n"},{"id":"9407","messageId":"43399124.8020706@gmail.com","threadId":"1942","inReplyTo":"pan.2005.09.27.06.35.35.834134@smurf.noris.de","subject":"Re: hared GIT repos (was Re: rsync deprecated but promoted?)","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2005-09-27T18:36:20Z","receivedAt":"2005-09-27T18:36:20Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Matthias Urlichs wrote:\n> Hi, Petr Baudis wrote:\n> \n>>The thing is, rsync is bad - it will happily put\n>>duplicate, redundant, and especially unwanted data to your repository,\n>>especially when the shared GIT repositories happen.\n> \n> Speaking of which -- is anybody working on that one?\n> \n> I find myself in need of a multiuser shared repository that cannot\n> be corrupted (i.e. I want to prevent the users from removing objects,\n> and replacing a ref with something that is not a child of the sha1 that's\n> already there should also be prevented).\n> \n\nYes. Actually just the protocol spec so far. Although, progress has been \ninterrupted by the process of moving to the SoCal area from Fla.\n"},{"id":"11524","messageId":"20051110231704.GB30496@pasky.or.cz","threadId":"1942","inReplyTo":"Pine.LNX.4.58.0509260941340.3308@g5.osdl.org","subject":"Re: rsync deprecated but promoted?","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-11-10T23:17:04Z","receivedAt":"2005-11-10T23:17:04Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Mon, Sep 26, 2005 at 06:43:08PM CEST, I got a letter\nwhere Linus Torvalds <torvalds@osdl.org> said that...\n> \n> \n> On Mon, 26 Sep 2005, Petr Baudis wrote:\n> > \n> > Actually, it would be nice to be able to tell git-fsck-objects to only\n> > verify objects which are referenced between given two commits (perhaps\n> > just make it support the ^object notation). Then I wouldn't mind running\n> > that after each rsync fetch in Cogito.\n> \n> You can kind of do it.\n> \n> Do\n> \n> \tgit-rev-list --objects $oldheads --not $newheads >& /dev/null\n> \techo \"$?\"\n> \n> and it _should_ largely work. Untested, of course, but I _hope_ that if \n> any object is missing, git-rev-list should die with an error. And if it \n> doesn't, I should fix it ;)\n\nIt should obviously be\n\n\tgit-rev-list $(git-rev-parse $newheads --not $oldheads) >& /dev/null\n\nbut it is indeed broken:\n\n\t$ git-rev-list --objects ... ^... && echo ':)'\n\t001439c6a797461c3e75018d95744d463077ae33\n\t841e3297d8df764da417da81dbfe1044e24d4082\n\tcf11a3d561e76c8ba273cb0bb62d46a4b2959c1f file\n\t:)\n\t$ git-cat-file -t cf11a3d561e76c8ba273cb0bb62d46a4b2959c1f\n\terror: unable to find cf11a3d561e76c8ba273cb0bb62d46a4b2959c1f\n\tfatal: git-cat-file cf11a3d561e76c8ba273cb0bb62d46a4b2959c1f: bad file\n\nCurrently I just modified it so that I iterate through all the sha1s\ngit-rev-list spits out, and test them by git-cat-file -t.\n\nThanks for the hint,\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nVI has two modes: the one in which it beeps and the one in which\nit doesn't.\n"}]}