{"thread":{"id":"3683","subject":"Question about possible git races","startedAt":"2006-03-20T16:24:05Z","lastAt":"2006-03-23T20:51:46Z","messageCount":10,"participants":["Radoslaw Szkodzinski","Andreas Ericsson","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"17709","messageId":"200603201724.12442.astralstorm@o2.pl","threadId":"3683","inReplyTo":null,"subject":"Question about possible git races","fromName":"Radoslaw Szkodzinski","fromEmail":"astralstorm@o2.pl","sentAt":"2006-03-20T16:24:05Z","receivedAt":"2006-03-20T16:24:05Z","isPatch":false,"sender":{"key":"astralstorm@o2.pl","avatar":null},"body":"I'd like to write a multithreaded application using git, so I'd like to see if \nthere are any races:\n\n- push vs pull\nOne thread pushes to the repository while another is pulling from it at the \nsame time. I should get the older commit.\n\n- push vs push\nBoth threads push at the same time. What happens?\nAny good way to merge those pushes?\n(I have full access to both repos)\n\nPossibly those two aren't fast-forward of each other.\nI think one of the pushes should abort in this case unless I force it.\n\n- fetch vs fetch\nI mean that two threads try to fetch from different repositories to a single \none. Possibly those two aren't fast-forward of each other.\nAny good way to merge those fetches?\n(I have full access to both repos)\n\nI'm meaning really bare git there, w/o bash+perl scripts.\n\n-- \nGPG Key id: 0xD1F10BA2\nFingerprint: 96E2 304A B9C4 949A 10A0 9105 9543 0453 D1F1 0BA2\n\nAstralStorm\n"},{"id":"17782","messageId":"200603222146.25395.astralstorm@o2.pl","threadId":"3683","inReplyTo":"200603201724.12442.astralstorm@o2.pl","subject":"Re: Question about possible git races","fromName":"Radoslaw Szkodzinski","fromEmail":"astralstorm@o2.pl","sentAt":"2006-03-22T20:46:21Z","receivedAt":"2006-03-22T20:46:21Z","isPatch":false,"sender":{"key":"astralstorm@o2.pl","avatar":null},"body":"On Monday 20 March 2006 17:24, Radoslaw Szkodzinski wrote yet:\n> I'd like to write a multithreaded application using git, so I'd like to see\n> if there are any races:\n>\n> - push vs fetch\n> ...\n> - push vs push\n> ...\n> - fetch vs fetch\n> ...\n>\n> I'm meaning really bare git there, w/o bash+perl scripts.\n\nCould anyone try to answer the question? \nI'd really like to know, because it's crucial to my application.\n\n-- \nGPG Key id: 0xD1F10BA2\nFingerprint: 96E2 304A B9C4 949A 10A0 9105 9543 0453 D1F1 0BA2\n\nAstralStorm\n"},{"id":"17785","messageId":"4421DD9E.7030201@op5.se","threadId":"3683","inReplyTo":"200603201724.12442.astralstorm@o2.pl","subject":"Re: Question about possible git races","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-03-22T23:28:30Z","receivedAt":"2006-03-22T23:28:30Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Radoslaw Szkodzinski wrote:\n> I'd like to write a multithreaded application using git,\n\nWhy on earth? The git tools aren't written to be thread-safe in that \nmanner, so you'll run into all sorts of problems. Unless you're talking \nabout managing the code for a multithreaded application with git, in \nwhich case you should just go read the tutorial. However, feeling \nslightly tipsy and in a distinctly good mood, I shall try to answer your \nquestions anyway.\n\n\n> so I'd like to see if \n> there are any races:\n> \n> - push vs pull\n> One thread pushes to the repository while another is pulling from it at the \n> same time. I should get the older commit.\n> \n\nYou will. Git atomizes (atomicizes? atomicifies?) pushes by updating the \nbranch head being pushed to after all the commit-, tree- and \nblob-objects are written. Tags are handled separately but equally \natomically.\n\n\n> - push vs push\n> Both threads push at the same time. What happens?\n> Any good way to merge those pushes?\n> (I have full access to both repos)\n> \n> Possibly those two aren't fast-forward of each other.\n> I think one of the pushes should abort in this case unless I force it.\n> \n\nRead the source to find out if it's locking the repo while updating or \nnot (I think it is, but I'm not sure). If it isn't the last one to \nfinish pushing wins out since the branch head update from that push will \noverwrite the previous one.\n\n\n> - fetch vs fetch\n> I mean that two threads try to fetch from different repositories to a single \n> one. Possibly those two aren't fast-forward of each other.\n> Any good way to merge those fetches?\n> (I have full access to both repos)\n> \n\ngit help octopus\n\nYou can fetch those two remote branch heads to local branches \nsimultaneously and then do the octopus in the master-thread while no \nother updates are happening. Doing several simultanous merges to a \nsingle branch is quite frankly so insane I have to go get myself a drink \njust from thinking about it.\n\n> I'm meaning really bare git there, w/o bash+perl scripts.\n> \n\nI don't think you can do it without Python. The default merge strategy \nis written in python, so.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"17787","messageId":"4421E406.5030700@op5.se","threadId":"3683","inReplyTo":"200603222146.25395.astralstorm@o2.pl","subject":"Re: Question about possible git races","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-03-22T23:55:50Z","receivedAt":"2006-03-22T23:55:50Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Radoslaw Szkodzinski wrote:\n> On Monday 20 March 2006 17:24, Radoslaw Szkodzinski wrote yet:\n> \n> Could anyone try to answer the question? \n> I'd really like to know, because it's crucial to my application.\n> \n\nI believe the reasons no-one answered your first mail are, in the \nfollowing order:\n\n1. Since I'm sure you're truly capable of writing such an application \nand finishing it before the git API has changed completely you should \nhave gotten your answers from trial-and-error, reading the source, or by \njust trying the app and seeing where it fails.\n\n2. You didn't say *why* you want to write a multi-threaded layer on top \nof git. Is it to implement a redundant file-server with revision \ncontrol? If so you don't need multi-threading. You need clever update \nhooks, a master repo and plenty of bandwidth with fast disks on the \nfile-servers.\n\n3. You didn't mention what you've tried to find the answers yourself, \nwhich makes me think you want me and the rest of us gitizens (yay! I \ncoined a phrase) do your homework for you. I personally find it very \nrude that you send another email again so shortly after the first one \nclaiming that \"you need this info for your app\", when the 500-odd people \nyou're asking clearly need their time for their families, hobbies, \ndaytime jobs, beer, etc. etc...\n\n4. Noone felt like answering since they saw no use for a multi-threaded \nlayer on top of git, especially without knowing what it was for. \nFriendliness only goes so far when met by such lack of respect for other \npeoples time, but if someone had seen the uses for the app you're \nwriting they probably would have taken time to at least ask you for some \nof the answers you left out in your original mail. If nothing else for \nthe sake of curiosity.\n\n\nSome more pointers on how to get answers to questions posted in online \nforums can be found on the links below.\n\nhttp://catb.org/~esr/faqs/smart-questions.html\nhttp://www.catb.org/~esr/faqs/hacker-howto.html\n\n\nBtw. I'm assuming you're aware you'll have to GPL this app of yours, \nsince git is GPL and you'll be using the git produce in a way that makes \nit vital to your app.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"17789","messageId":"7vacbi6m91.fsf@assigned-by-dhcp.cox.net","threadId":"3683","inReplyTo":"200603201724.12442.astralstorm@o2.pl","subject":"Re: Question about possible git races","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-03-23T00:24:58Z","receivedAt":"2006-03-23T00:24:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Radoslaw Szkodzinski writes:\n\n> - push vs pull\n>\n> - push vs push\n>\n> - fetch vs fetch\n\nAbout push vs push, with \"really bare git\", I take it that you\nmean two send-pack from remote sites running two receive-pack\nsimultaneously.\n\nThere is an explicit race avoidance between the receive-pack\nprocesses. When a ref (either branch head or a tag) is updated,\nit does:\n\n - read the current value from the ref.\n - do its work.\n - lock to prevent others to create the temporary file for\n updating the ref.\n - create the temporary file for the ref and write the new value.\n - check if the ref's value has not changed from what it\n initially read;\n - rename the temporary file to the ref to unlock.\n\nRead receive-pack.c::update() for exact details if you are\ninterested.\n\n> I'm meaning really bare git there, w/o bash+perl scripts.\n\nThe question does not make any sense for other cases, because\nbranch update by fetch and pull are all scripts based.\n"},{"id":"17792","messageId":"200603230222.38978.astralstorm@o2.pl","threadId":"3683","inReplyTo":"7vacbi6m91.fsf@assigned-by-dhcp.cox.net","subject":"Re: Question about possible git races","fromName":"Radoslaw Szkodzinski","fromEmail":"astralstorm@o2.pl","sentAt":"2006-03-23T01:22:34Z","receivedAt":"2006-03-23T01:22:34Z","isPatch":false,"sender":{"key":"astralstorm@o2.pl","avatar":null},"body":"On Thursday 23 March 2006 01:24, Junio C Hamano wrote yet:\n> Radoslaw Szkodzinski writes:\n> > - push vs pull\n> >\n> > - push vs push\n> >\n> > - fetch vs fetch\n>\n> About push vs push, with \"really bare git\", I take it that you\n> mean two send-pack from remote sites running two receive-pack\n> simultaneously.\n>\n> There is an explicit race avoidance between the receive-pack\n> processes. When a ref (either branch head or a tag) is updated,\n> it does:\n>\n> - read the current value from the ref.\n> - do its work.\n> - lock to prevent others to create the temporary file for\n> updating the ref.\n\n\n> - create the temporary file for the ref and write the new value.\n> - check if the ref's value has not changed from what it\n> initially read;\n> - rename the temporary file to the ref to unlock.\n>\n> Read receive-pack.c::update() for exact details if you are\n> interested.\n\nSo there is locking I've missed while reading through the source.\nGuess all the coffee doesn't help.\n\nThere is a lock, so the other git-receive-pack for given ref will fail. \nDoes that also work for git-local-fetch with -l option?\n\nLooks good though that I can fetch to another ref.\n\n>\n> > I'm meaning really bare git there, w/o bash+perl scripts.\n>\n> The question does not make any sense for other cases, because\n> branch update by fetch and pull are all scripts based.\n\nI should have known better than to use vague words.\nFor me fetch = git-*-fetch. Which in turn calls git-receive-pack.\n\nThank you for the precise answer.\n\n-- \nGPG Key id: 0xD1F10BA2\nFingerprint: 96E2 304A B9C4 949A 10A0 9105 9543 0453 D1F1 0BA2\n\nAstralStorm\n"},{"id":"17793","messageId":"200603230224.54736.astralstorm@o2.pl","threadId":"3683","inReplyTo":"4421DD9E.7030201@op5.se","subject":"Re: Question about possible git races","fromName":"Radoslaw Szkodzinski","fromEmail":"astralstorm@o2.pl","sentAt":"2006-03-23T01:24:50Z","receivedAt":"2006-03-23T01:24:50Z","isPatch":false,"sender":{"key":"astralstorm@o2.pl","avatar":null},"body":"On Thursday 23 March 2006 00:28, Andreas Ericsson wrote yet:\n> However, feeling slightly tipsy and in a distinctly good mood, I shall try \nto answer your questions anyway.\n\nThe golden liquid works for me, great. Thank you.\n\n>\n> > so I'd like to see if\n> > there are any races:\n> >\n> > - push vs pull\n> > One thread pushes to the repository while another is pulling from it at\n> > the same time. I should get the older commit.\n>\n> You will. Git atomizes (atomicizes? atomicifies?) pushes by updating the\n> branch head being pushed to after all the commit-, tree- and\n> blob-objects are written. \n\nJust as I expected. Good.\n\n> Tags are handled separately but equally \n> atomically.\n\nGood too.\n\n>\n> > - fetch vs fetch\n> > I mean that two threads try to fetch from different repositories to a\n> > single one. Possibly those two aren't fast-forward of each other.\n> > Any good way to merge those fetches?\n> > (I have full access to both repos)\n>\n> git help octopus\n>\n> You can fetch those two remote branch heads to local branches\n> simultaneously and then do the octopus in the master-thread while no\n> other updates are happening.\n\nI could slurp the unrelated commits with an octopus of course...\nBut the others pose a problem.\n\n> Doing several simultanous merges to a \n> single branch is quite frankly so insane I have to go get myself a drink\n> just from thinking about it.\n\nIt's a rare situation in the app, but not impossible. (I want to avoid \nlocking) That's why I was asking about it.\n\n>\n> > I'm meaning really bare git there, w/o bash+perl scripts.\n>\n> I don't think you can do it without Python. The default merge strategy\n> is written in python, so.\n\nYou've hit it, the app is written in Python (as of yet).\n\nThe default merge strategy is simply... calling merge and also detecting \nnaming/existence conflicts with a simple scalar merge.\n\nThe only bit more complicated thing is detecting the LCA for 3-way merge.\n\n\nI'd like to build a decentralised collaborative web application, as scalable \nas it gets. (I expect a lot of traffic)\nI also need to only check out parts of the tree. (many SCMs can't do that)\nGit, with its speed, seems well-suited to the task.\nMerging will be uncommon in the workload, so can be slow, but shouldn't break \nor present weird conflicts to the users.\n\nUnless the accidental case starts to dominate - then I'll have a problem.\n\nSorry for spam and cutting out major questions, then answering at the end of \nthe post.\n\n\n> Btw. I'm assuming you're aware you'll have to GPL this app of yours,\n> since git is GPL and you'll be using the git produce in a way that makes\n> it vital to your app.\n\nIt will be, but not because of git (it's execve()ing it), but rather because \nof the principle.\n\nIntermediate results will probably be:\n- a good Python object interface to git\n- another porcelain, portable, coded in Python\n- maybe a new merge strategy\n- later yet another porcelain, written in C\n\n-- \nGPG Key id: 0xD1F10BA2\nFingerprint: 96E2 304A B9C4 949A 10A0 9105 9543 0453 D1F1 0BA2\n\nAstralStorm\n"},{"id":"17797","messageId":"7vmzfi53w7.fsf@assigned-by-dhcp.cox.net","threadId":"3683","inReplyTo":"200603230222.38978.astralstorm@o2.pl","subject":"Re: Question about possible git races","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-03-23T01:46:48Z","receivedAt":"2006-03-23T01:46:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Radoslaw Szkodzinski writes:\n\n> For me fetch = git-*-fetch. Which in turn calls git-receive-pack.\n\nDoes anything other than git-send-pack call git-receive-pack?\n\nFor fetch, git-fetch-pack is called from the core level, but it\ndoes not update refs itself. It writes out enough information\nto its standard output so that the script calling it can update\nthe refs. So at the core level there cannot be any race, but\nthat does not necessarily mean existing scripts are race free.\n\nOur barebone Porcelainish scripts _do_ use update-ref to do the\nsame lock - re-read - rename-to-update cycle when updating the\nrefs using that information, but that is something you\nexplicitly said you are not interested in ;-).\n"},{"id":"17802","messageId":"44220E21.7040204@op5.se","threadId":"3683","inReplyTo":"200603230224.54736.astralstorm@o2.pl","subject":"Re: Question about possible git races","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-03-23T02:55:29Z","receivedAt":"2006-03-23T02:55:29Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Radoslaw Szkodzinski wrote:\n> On Thursday 23 March 2006 00:28, Andreas Ericsson wrote yet:\n> \n>>Btw. I'm assuming you're aware you'll have to GPL this app of yours,\n>>since git is GPL and you'll be using the git produce in a way that makes\n>>it vital to your app.\n> \n> \n> It will be, but not because of git (it's execve()ing it), but rather because \n> of the principle.\n> \n\nAh, Good Thing. Just to clarify for the archives though, it *is* \nrequired since it's using git in a way that makes git a fundamental, \nnon-replaceable part of its core operations.\n\n> Intermediate results will probably be:\n> - later yet another porcelain, written in C\n> \n\nyagit? yagp? What I wanna know is when jigit's gonna hit the streets. :)\n\n\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"17823","messageId":"200603232151.51493.astralstorm@o2.pl","threadId":"3683","inReplyTo":"44220E21.7040204@op5.se","subject":"Re: Question about possible git races","fromName":"Radoslaw Szkodzinski","fromEmail":"astralstorm@o2.pl","sentAt":"2006-03-23T20:51:46Z","receivedAt":"2006-03-23T20:51:46Z","isPatch":false,"sender":{"key":"astralstorm@o2.pl","avatar":null},"body":"On Thursday 23 March 2006 03:55, Andreas Ericsson wrote yet:\n> Radoslaw Szkodzinski wrote:\n> > On Thursday 23 March 2006 00:28, Andreas Ericsson wrote yet:\n> >>Btw. I'm assuming you're aware you'll have to GPL this app of yours,\n> >>since git is GPL and you'll be using the git produce in a way that makes\n> >>it vital to your app.\n> >\n> > It will be, but not because of git (it's execve()ing it), but rather\n> > because of the principle.\n>\n> Ah, Good Thing. Just to clarify for the archives though, it *is*\n> required since it's using git in a way that makes git a fundamental,\n> non-replaceable part of its core operations.\n>\n> > Intermediate results will probably be:\n> > - later yet another porcelain, written in C\n>\n> yagit? yagp? What I wanna know is when jigit's gonna hit the streets. :)\n\nI need a memorable three-letter acronym, so it will be gip. (Git porcelain In \nPython) and the C one... later.\n\nGIP also stands for Good Informatics Practices :P or \"to swindle\", but it's \nthe deprecated usage, I think. It has some racial ties.\n\nConflicts with: GNOME IP calculator - http://www.debain.org/software/gip/\n\nI'd get myself a better commandline IP calculator than give a good TLA to \nthis. Gnip should be the name.\n\nWhat's jigit? Did I miss something?\n\n-- \nGPG Key id: 0xD1F10BA2\nFingerprint: 96E2 304A B9C4 949A 10A0 9105 9543 0453 D1F1 0BA2\n\nAstralStorm\n"}]}