{"thread":{"id":"38379","subject":"[RFC] Introducing different handling for small/large transactions","startedAt":"2015-01-15T22:36:11Z","lastAt":"2015-01-18T12:13:15Z","messageCount":8,"participants":["Stefan Beller","Jeff King","Junio C Hamano","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"254781","messageId":"1421361371-30221-1-git-send-email-sbeller@google.com","threadId":"38379","inReplyTo":null,"subject":"[RFC] Introducing different handling for small/large transactions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-01-15T22:36:11Z","receivedAt":"2015-01-15T22:36:11Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"For everyday use we want git to be fast. Creating one commit should not\ntouch the packed refs file. If we do other stuff involving more than\none ref, we may accept touching the packed refs file and have a process\nwhich takes slightly longer but can handle more complex requests correctly,\nsuch as renaming into and from directories (topic/1 -> topic and reverse).\nRenaming is currently not part of the transaction API because of the (D/F)\nproblems. This proposed change would enable having renames being part of\nthe transactions API.\n\nA transaction covers creating, deleting and updating a ref and its reflog.\nRenaming would be a deletion followed by creating a new ref atomically.\n\nSo for here is my proposal for small transactions:\n(just one ref [and/or reflog] touched):\n\nIn ref_transaction_update:\n\t* create $REF.lock file\n\t* write new content to the lock file\n\nIn ref_transaction_commit\n\t* commit the .lock file to its destination\n\t* in case this is a deletion:\n\t\t* remove the loose ref\n\t\t* and repack the packed refs file if necessary\n\nThe larger transactions would be handled differently by relying\non the packed refs file:\nIn ref_transaction_update:\n\t* detect if we transition to a large transaction\n\t  (by having more than one entry in transaction->updates)\n\t  if so:\n\t\t* Pack all currently existing refs into the packed\n\t\t  refs file, commit the packed refs file and delete\n\t\t  all loose refs. This will avoid (d/f) conflicts.\n\n\t\t* Keep the packed-refs file locked and move the first\n\t\t  transaction update into the packed-refs.lock file\n\n\t* Any update(delete, create, update) is put into the locked\n\t  packed refs file.\n\t* Additionally we need to obtain the .lock for the loose refs\n\t  file to keep guarantees, though we should close the file\n\t  descriptor as we don't wand to run out of file descriptors.\n\nIn ref_transaction_commit:\n\t* We only need to commit the packed refs file\n\t* Discard all other lock files as the changes get committed as a whole\n\t  by the packed refs file\n\nAny feedback would be welcome!\nThanks,\nStefan\n"},{"id":"254784","messageId":"20150115224605.GD19021@peff.net","threadId":"38379","inReplyTo":"1421361371-30221-1-git-send-email-sbeller@google.com","subject":"Re: [RFC] Introducing different handling for small/large transactions","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-01-15T22:46:05Z","receivedAt":"2015-01-15T22:46:05Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 15, 2015 at 02:36:11PM -0800, Stefan Beller wrote:\n\n> So for here is my proposal for small transactions:\n> (just one ref [and/or reflog] touched):\n\nThe implication being that a \"large\" transaction is any with more than\none update.\n\nI think performance may suffer if you do not also take into account the\nsize of the packed-refs file. If you are updating 5 refs and there are\n10 in the packed-refs file, rewriting the extra 5 is probably not a big\ndeal. If there are 400,000 in the packed-refs file, it probably is. I'm\nnot sure where the cutoff is (certainly the per-ref cost is less for\npacked-refs once you have started writing the file, so there is probably\nsome crossover percentage that you could measure).\n\n> \t* detect if we transition to a large transaction\n> \t  (by having more than one entry in transaction->updates)\n> \t  if so:\n> \t\t* Pack all currently existing refs into the packed\n> \t\t  refs file, commit the packed refs file and delete\n> \t\t  all loose refs. This will avoid (d/f) conflicts.\n> \n> \t\t* Keep the packed-refs file locked and move the first\n> \t\t  transaction update into the packed-refs.lock file\n\nThis increases lock contention, as now independent ref updates all need\nto take the same packed-refs.lock. This can be a problem on a busy\nrepository, especially because we never retry the packed-refs lock.\nWe already see this problem somewhat on GitHub. Ref deletions need the\npacked-refs.lock file, which can conflict with another deletion, or with\nrunning `git pack-refs`.\n\n-Peff\n"},{"id":"254787","messageId":"CAGZ79kbXYm66StsAwHFdTovkYeDVHWV2PdCK0abVXwWiD7sAAg@mail.gmail.com","threadId":"38379","inReplyTo":"20150115224605.GD19021@peff.net","subject":"Re: [RFC] Introducing different handling for small/large transactions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-01-15T23:24:15Z","receivedAt":"2015-01-15T23:24:15Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Jan 15, 2015 at 2:46 PM, Jeff King <peff@peff.net> wrote:\n> On Thu, Jan 15, 2015 at 02:36:11PM -0800, Stefan Beller wrote:\n>\n>> So for here is my proposal for small transactions:\n>> (just one ref [and/or reflog] touched):\n>\n> The implication being that a \"large\" transaction is any with more than\n> one update.\n\nExactly.\n\n>\n> I think performance may suffer if you do not also take into account the\n> size of the packed-refs file. If you are updating 5 refs and there are\n> 10 in the packed-refs file, rewriting the extra 5 is probably not a big\n> deal. If there are 400,000 in the packed-refs file, it probably is. I'm\n> not sure where the cutoff is (certainly the per-ref cost is less for\n> packed-refs once you have started writing the file, so there is probably\n> some crossover percentage that you could measure).\n>\n>>       * detect if we transition to a large transaction\n>>         (by having more than one entry in transaction->updates)\n>>         if so:\n>>               * Pack all currently existing refs into the packed\n>>                 refs file, commit the packed refs file and delete\n>>                 all loose refs. This will avoid (d/f) conflicts.\n>>\n>>               * Keep the packed-refs file locked and move the first\n>>                 transaction update into the packed-refs.lock file\n>\n> This increases lock contention, as now independent ref updates all need\n> to take the same packed-refs.lock. This can be a problem on a busy\n> repository, especially because we never retry the packed-refs lock.\n> We already see this problem somewhat on GitHub. Ref deletions need the\n> packed-refs.lock file, which can conflict with another deletion, or with\n> running `git pack-refs`.\n>\n> -Peff\n\nI see the performance problem as well as the contention problem\nyou're pointing out. Dealing with loose refs however creates other\nproblems such as directory/file conflicts on renaming. I am trying to\nthink of a way which moves most of the failures to the transaction_update\nphase, such that the transaction_commit is rather easy and not expected\nto produce many errors.\n\nSo I think dealing with a generic large transaction cannot be really solved\noutside the packed refs file. There could be another special case for mass\ndeleting refs however. Or retries for the packed refs file. Or we first check if\nwe *really* need to lock the packed refs file (just realized we\nalready do that :/)\n\n(just curious:)\nMay I ask on which kinds of repository you can see packed-refs.lock contention?\n\nI want to improve git atomicity, specially for 'weird' cases as presented in my\nprevious mail[1]. Eventually I even want to have cross repository atomicty in\ngit, so an example could be:\n(\n    cd API-Provider &&\n    edit files # new changes breaking the API\n    git commit -a -m \"...\"\n) &&\n(\n    cd API-consumer\n    edit files # use new and shiny API\n    git commit -a -m \"...\"\n) &&\ngit multipush --atomic API-Provider:origin:master API-consumer:origin:master\n\nWhen having such a goal a reliable and easy to use ref transaction API makes\nlife lots more easier. By reliable I mean that there are no sudden problems\nas pointed out in [1], these kinds of rejections make users unhappy. And by\neasy to use I mean there are only a few functions I need to know and no\nproliferation of functions exposed in the API. Internally we can do all we want\nsuch as special cases for delete-only transactions.\n\nAs another unrelated thought (400,000 refs is quite a lot)\nWould it make sense to have packed-refs files grouped by topic directory, i.e.\none packed-ref for\n    topic/1\n    topic/2\n    topic/*\nand another packed ref for\n    feature/a\n    feature/b\n    feature/*\n\nThis would rather help your problems with contention instead of making things\nmore atomic though. But that would avoid 400,000 refs in one packed refs file,\nwhich then may still be easier to handle for larger transactions.\n\nThanks,\nStefan\n\n[1] http://www.mail-archive.com/git@vger.kernel.org/msg63919.html\n"},{"id":"254789","messageId":"xmqqppaf4o04.fsf@gitster.dls.corp.google.com","threadId":"38379","inReplyTo":"1421361371-30221-1-git-send-email-sbeller@google.com","subject":"Re: [RFC] Introducing different handling for small/large transactions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-01-15T23:34:19Z","receivedAt":"2015-01-15T23:34:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n> In ref_transaction_commit\n> \t* commit the .lock file to its destination\n> \t* in case this is a deletion:\n> \t\t* remove the loose ref\n> \t\t* and repack the packed refs file if necessary\n\nDon't you need to repack and then remove the loose one, though?\nOtherwise you would expose a stale packed ref in the middle to the\nother readers, no?\n\n> The larger transactions would be handled differently by relying\n> on the packed refs file:\n> In ref_transaction_update:\n> \t* detect if we transition to a large transaction\n> \t  (by having more than one entry in transaction->updates)\n> \t  if so:\n> \t\t* Pack all currently existing refs into the packed\n> \t\t  refs file, commit the packed refs file and delete\n> \t\t  all loose refs. This will avoid (d/f) conflicts.\n>\n> \t\t* Keep the packed-refs file locked and move the first\n> \t\t  transaction update into the packed-refs.lock file\n>\n> \t* Any update(delete, create, update) is put into the locked\n> \t  packed refs file.\n\nI am not sure if you mean (a) keep updates only in-core, to be\nflushed at the commit time, or (b) each and every update in the\nlarge transaction results in rewriting the entire packed-refs.lock\nfile, only to be renamed to the final name at the commit time.\nI am hoping it would be the former.\n\n> \t* Additionally we need to obtain the .lock for the loose refs\n> \t  file to keep guarantees, though we should close the file\n> \t  descriptor as we don't wand to run out of file descriptors.\n\nYes, this last point is important.\n"},{"id":"254791","messageId":"20150115235310.GA25120@peff.net","threadId":"38379","inReplyTo":"CAGZ79kbXYm66StsAwHFdTovkYeDVHWV2PdCK0abVXwWiD7sAAg@mail.gmail.com","subject":"Re: [RFC] Introducing different handling for small/large transactions","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-01-15T23:53:10Z","receivedAt":"2015-01-15T23:53:10Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 15, 2015 at 03:24:15PM -0800, Stefan Beller wrote:\n\n> I see the performance problem as well as the contention problem\n> you're pointing out. Dealing with loose refs however creates other\n> problems such as directory/file conflicts on renaming. I am trying to\n> think of a way which moves most of the failures to the transaction_update\n> phase, such that the transaction_commit is rather easy and not expected\n> to produce many errors.\n> \n> So I think dealing with a generic large transaction cannot be really solved\n> outside the packed refs file. There could be another special case for mass\n> deleting refs however. Or retries for the packed refs file. Or we first check if\n> we *really* need to lock the packed refs file (just realized we\n> already do that :/)\n\nIt sounds like there are certain _types_ of transactions that trigger\nproblems (and that we can handle better). E.g., renames. Which must\ninvolve a deletion. And we already have to touch the packed-refs file\nfor a deletion. Once we know that we must do that, we might as well\nwrite the rest of the ref updates into it for \"free\", right? There is no\nextra lock contention (there is contention, but we are not making it\nworse than what is already there, and what is necessary for the on-disk\nformat).\n\nCould the rule be something like:\n\n  if there are deletions in the transaction\n    /* to handle potential D/F conflicts */\n    update_type = packed\n  else if # of transactions / # of packed-refs entries > 0.8\n    /*\n     * for performance, we are better off just writing the whole file;\n     * extra lock contention probably doesn't matter because a\n     * simultaneous write would probably conflict with one of our\n     * refs anyway.\n     */\n     update_type = packed\n  else\n    /*\n     * small-ish ref update. Just write out the loose refs.\n     */\n    update_type = loose\n\nI'm not sure I'm completely comfortable with all of the \"probably\"'s in\nthe second comment, though.\n\n> May I ask on which kinds of repository you can see packed-refs.lock contention?\n\nI'm not sure what you mean by \"kinds\" (i.e., how you want me to\ncharacterize our repos). Busy ones, certainly. :) And this is all\nserver-side, where the main write operations are client pushes and\nrunning \"git gc\". And of course mainly ones that are shared between many\nusers (so many users pushing to a central repo, not each one forking and\nmaking pull requests from their own publishing points).\n\nSome people do a lot of deletions. E.g., they may be constantly pushing\nand deleting tags from a CI process that kicks off whenever a \"real\"\npush happens.\n\nA lot of our contention is from the pack-refs process itself. It has to\ntake a lock on each of the loose refs it prunes. It's smart enough\n(since 06839515, anyway) to avoid complaining when a real writer is\nholding the lock. But when the race goes the other way, the \"real\"\nwriter has no idea that the process holding the lock is \"weak\". That is,\nif it were another push, we would not want to block and wait to acquire\nthe lock. We would only find out then that the sha1 had moved, and we\nhave to abort the operation. But for a weak writer like packed-refs, the\nref value is not changing at all, and we could probably succeed if we\nretried the lock after a delay.\n\n> I want to improve git atomicity, specially for 'weird' cases as presented in my\n> previous mail[1]. Eventually I even want to have cross repository atomicty in\n> git, so an example could be:\n> (\n>     cd API-Provider &&\n>     edit files # new changes breaking the API\n>     git commit -a -m \"...\"\n> ) &&\n> (\n>     cd API-consumer\n>     edit files # use new and shiny API\n>     git commit -a -m \"...\"\n> ) &&\n> git multipush --atomic API-Provider:origin:master API-consumer:origin:master\n\nI think that's a reasonable goal, but I am not sure what packed-refs has\nto do with it. The client sees only the push interface. So you need:\n\n  1. Atomic push to each remote (which I think you have already\n     implemented).\n\n  2. Some kind of 2-phase commit on each remote. To say \"take the lock,\n     tell me when you have it, but _don't_ update until I ask you to\".\n     And then when you have confirmation from all remotes, send each one\n     the instruction to commit the update.\n\n     Of course that isn't foolproof if the commit step fails (lost\n     connection, write error on disk, etc), but it's basically what you\n     want (and this kind of commit is well-studied in the literature).\n\nThe actions that the remote takes to do (1) are not important, and\neither way it needs to take all of the locks for each ref. Using\npacked-refs here is just a quality of implementation issue (there are\nsome transactions we can take as a single atomic change that we\notherwise would not be able to).\n\nCoincidentally, we are looking at something similar at GitHub to (2) for\ndoing repository replication (where you'd want to make sure that each of\nyour replicas moves in lockstep). It hasn't been written yet, but we\nimagined that the result would be way too gross and too GitHub-specific\nto make it into the upstream protocol. But maybe not.\n\n> When having such a goal a reliable and easy to use ref transaction API makes\n> life lots more easier. By reliable I mean that there are no sudden problems\n> as pointed out in [1], these kinds of rejections make users unhappy. And by\n> easy to use I mean there are only a few functions I need to know and no\n> proliferation of functions exposed in the API. Internally we can do all we want\n> such as special cases for delete-only transactions.\n\nYeah, I agree with this. But I have a nagging feeling as we deal with\nthese problems that the right solution is some better ref-store than the\nfilesystem + packed-refs. It's a great, simple solution, but the scaling\nand transactional considerations are really making it not-simple. These\nare solved problems in the database world. Moving onto a ready-made\nsolution has its own problems, and I do not think we would ever\ndeprecate the filesystem + packed-refs storage format. It works very\nwell for client repositories, and has no dependencies. But it would be\nnice if there was a pluggable option for busy shared server repos.\n\n> As another unrelated thought (400,000 refs is quite a lot)\n> Would it make sense to have packed-refs files grouped by topic directory, i.e.\n> one packed-ref for\n>     topic/1\n>     topic/2\n>     topic/*\n> and another packed ref for\n>     feature/a\n>     feature/b\n>     feature/*\n> \n> This would rather help your problems with contention instead of making things\n> more atomic though. But that would avoid 400,000 refs in one packed refs file,\n> which then may still be easier to handle for larger transactions.\n\nCertainly the thought has occurred to me (and others at GitHub). But if\nwe are going to break compatibility, I think I'd rather see us move onto\na more mature data-storage backend than invent something even more\ncomplicated (and that does not really solve the problem, just makes them\nless bad).\n\n-Peff\n"},{"id":"254825","messageId":"CAGZ79kZ9vFebD9nsNkfU759msT9CvF4gRS_Apmi5-u5XWsiGYQ@mail.gmail.com","threadId":"38379","inReplyTo":"xmqqppaf4o04.fsf@gitster.dls.corp.google.com","subject":"Re: [RFC] Introducing different handling for small/large transactions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-01-16T19:00:46Z","receivedAt":"2015-01-16T19:00:46Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Jan 15, 2015 at 3:34 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Stefan Beller <sbeller@google.com> writes:\n>\n>> In ref_transaction_commit\n>>       * commit the .lock file to its destination\n>>       * in case this is a deletion:\n>>               * remove the loose ref\n>>               * and repack the packed refs file if necessary\n>\n> Don't you need to repack and then remove the loose one, though?\n> Otherwise you would expose a stale packed ref in the middle to the\n> other readers, no?\n\nYou're right.\n\n>\n>> The larger transactions would be handled differently by relying\n>> on the packed refs file:\n>> In ref_transaction_update:\n>>       * detect if we transition to a large transaction\n>>         (by having more than one entry in transaction->updates)\n>>         if so:\n>>               * Pack all currently existing refs into the packed\n>>                 refs file, commit the packed refs file and delete\n>>                 all loose refs. This will avoid (d/f) conflicts.\n>>\n>>               * Keep the packed-refs file locked and move the first\n>>                 transaction update into the packed-refs.lock file\n>>\n>>       * Any update(delete, create, update) is put into the locked\n>>         packed refs file.\n>\n> I am not sure if you mean (a) keep updates only in-core, to be\n> flushed at the commit time, or (b) each and every update in the\n> large transaction results in rewriting the entire packed-refs.lock\n> file, only to be renamed to the final name at the commit time.\n> I am hoping it would be the former.\n\nI wasn't sure which I mean. The first one is obviously better to do.\n\n>\n>>       * Additionally we need to obtain the .lock for the loose refs\n>>         file to keep guarantees, though we should close the file\n>>         descriptor as we don't wand to run out of file descriptors.\n>\n> Yes, this last point is important.\n>\n"},{"id":"254828","messageId":"CAGZ79kYc97xa6YRJ_KTUj7z2_aNa4uTDSEAA0+DiLuMPE8WuBw@mail.gmail.com","threadId":"38379","inReplyTo":"20150115235310.GA25120@peff.net","subject":"Re: [RFC] Introducing different handling for small/large transactions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-01-16T19:23:22Z","receivedAt":"2015-01-16T19:23:22Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Jan 15, 2015 at 3:53 PM, Jeff King <peff@peff.net> wrote:\n> On Thu, Jan 15, 2015 at 03:24:15PM -0800, Stefan Beller wrote:\n>\n>> I see the performance problem as well as the contention problem\n>> you're pointing out. Dealing with loose refs however creates other\n>> problems such as directory/file conflicts on renaming. I am trying to\n>> think of a way which moves most of the failures to the transaction_update\n>> phase, such that the transaction_commit is rather easy and not expected\n>> to produce many errors.\n>>\n>> So I think dealing with a generic large transaction cannot be really solved\n>> outside the packed refs file. There could be another special case for mass\n>> deleting refs however. Or retries for the packed refs file. Or we first check if\n>> we *really* need to lock the packed refs file (just realized we\n>> already do that :/)\n>\n> It sounds like there are certain _types_ of transactions that trigger\n> problems (and that we can handle better). E.g., renames. Which must\n> involve a deletion. And we already have to touch the packed-refs file\n> for a deletion. Once we know that we must do that, we might as well\n> write the rest of the ref updates into it for \"free\", right? There is no\n> extra lock contention (there is contention, but we are not making it\n> worse than what is already there, and what is necessary for the on-disk\n> format).\n>\n> Could the rule be something like:\n>\n>   if there are deletions in the transaction\n>     /* to handle potential D/F conflicts */\n>     update_type = packed\n>   else if # of transactions / # of packed-refs entries > 0.8\n>     /*\n>      * for performance, we are better off just writing the whole file;\n>      * extra lock contention probably doesn't matter because a\n>      * simultaneous write would probably conflict with one of our\n>      * refs anyway.\n>      */\n>      update_type = packed\n>   else\n>     /*\n>      * small-ish ref update. Just write out the loose refs.\n>      */\n>     update_type = loose\n>\n> I'm not sure I'm completely comfortable with all of the \"probably\"'s in\n> the second comment, though.\n>\n>> May I ask on which kinds of repository you can see packed-refs.lock contention?\n>\n> I'm not sure what you mean by \"kinds\" (i.e., how you want me to\n> characterize our repos). Busy ones, certainly. :) And this is all\n> server-side, where the main write operations are client pushes and\n> running \"git gc\". And of course mainly ones that are shared between many\n> users (so many users pushing to a central repo, not each one forking and\n> making pull requests from their own publishing points).\n>\n> Some people do a lot of deletions. E.g., they may be constantly pushing\n> and deleting tags from a CI process that kicks off whenever a \"real\"\n> push happens.\n>\n> A lot of our contention is from the pack-refs process itself. It has to\n> take a lock on each of the loose refs it prunes. It's smart enough\n> (since 06839515, anyway) to avoid complaining when a real writer is\n> holding the lock. But when the race goes the other way, the \"real\"\n> writer has no idea that the process holding the lock is \"weak\". That is,\n> if it were another push, we would not want to block and wait to acquire\n> the lock. We would only find out then that the sha1 had moved, and we\n> have to abort the operation. But for a weak writer like packed-refs, the\n> ref value is not changing at all, and we could probably succeed if we\n> retried the lock after a delay.\n>\n>> I want to improve git atomicity, specially for 'weird' cases as presented in my\n>> previous mail[1]. Eventually I even want to have cross repository atomicty in\n>> git, so an example could be:\n>> (\n>>     cd API-Provider &&\n>>     edit files # new changes breaking the API\n>>     git commit -a -m \"...\"\n>> ) &&\n>> (\n>>     cd API-consumer\n>>     edit files # use new and shiny API\n>>     git commit -a -m \"...\"\n>> ) &&\n>> git multipush --atomic API-Provider:origin:master API-consumer:origin:master\n>\n> I think that's a reasonable goal, but I am not sure what packed-refs has\n> to do with it. The client sees only the push interface. So you need:\n\nI was assuming origin to be the same for both repos, which then makes\nit a lot easier.\nSo on the server side you need to do:\n* prepare the ref transaction in the first repo\n* \"cd second repo\"\n* prepare the ref transaction the second repo\n* ref transaction commit.\n\nNote that we're in the second repository when committing the ref transaction,\nbut we also want to commit things the first repo. So we cannot keep branch\nnames in memory but need to have the lockfiles already as a lockfile is pointed\nto by its filename or optionally by its file descriptor, so we still\nknow which of\nthe repos we're adressing there.\n\n>\n>   1. Atomic push to each remote (which I think you have already\n>      implemented).\n>\n>   2. Some kind of 2-phase commit on each remote. To say \"take the lock,\n>      tell me when you have it, but _don't_ update until I ask you to\".\n>      And then when you have confirmation from all remotes, send each one\n>      the instruction to commit the update.\n>\n>      Of course that isn't foolproof if the commit step fails (lost\n>      connection, write error on disk, etc), but it's basically what you\n>      want (and this kind of commit is well-studied in the literature).\n\nIt is even taught in universities, so it must be at least 15 years old\nalready. ;)\n\n>\n> The actions that the remote takes to do (1) are not important, and\n> either way it needs to take all of the locks for each ref. Using\n> packed-refs here is just a quality of implementation issue (there are\n> some transactions we can take as a single atomic change that we\n> otherwise would not be able to).\n\nYeah I think it's a good goal to have atomic not shrinking the\npossible set of operations,\ni.e. what works without --atomic should also work with --atomic.\n\n>\n> Coincidentally, we are looking at something similar at GitHub to (2) for\n> doing repository replication (where you'd want to make sure that each of\n> your replicas moves in lockstep). It hasn't been written yet, but we\n> imagined that the result would be way too gross and too GitHub-specific\n> to make it into the upstream protocol. But maybe not.\n\nRonnie has written code for that and already posted it here in the mailing list\na few times, but it was lacking reviews and feedback.\nEssentially he proposed the following:\n* do a push to a hidden part of the git server. This hidden place is\nhidden by default\nand doesn't need special setup. (refs/hidden/ iirc)\n* do an push which makes the previous push visible. This one doesn't need to\n  wait for pack uploads, so it's fast. Hence it could lock all it wants.\n\n>\n>> When having such a goal a reliable and easy to use ref transaction API makes\n>> life lots more easier. By reliable I mean that there are no sudden problems\n>> as pointed out in [1], these kinds of rejections make users unhappy. And by\n>> easy to use I mean there are only a few functions I need to know and no\n>> proliferation of functions exposed in the API. Internally we can do all we want\n>> such as special cases for delete-only transactions.\n>\n> Yeah, I agree with this. But I have a nagging feeling as we deal with\n> these problems that the right solution is some better ref-store than the\n> filesystem + packed-refs. It's a great, simple solution, but the scaling\n> and transactional considerations are really making it not-simple. These\n> are solved problems in the database world. Moving onto a ready-made\n> solution has its own problems, and I do not think we would ever\n> deprecate the filesystem + packed-refs storage format. It works very\n> well for client repositories, and has no dependencies. But it would be\n> nice if there was a pluggable option for busy shared server repos.\n\nThat's what Jonathan said here as well. I though about getting the files backend\nimproved, but you're the second who talks me into implementing a database\nbackend. I'll look into that.\n\nThanks,\nStefan\n"},{"id":"254862","messageId":"54BBA35B.50101@alum.mit.edu","threadId":"38379","inReplyTo":"1421361371-30221-1-git-send-email-sbeller@google.com","subject":"Re: [RFC] Introducing different handling for small/large transactions","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-01-18T12:13:15Z","receivedAt":"2015-01-18T12:13:15Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/15/2015 11:36 PM, Stefan Beller wrote:\n> For everyday use we want git to be fast. Creating one commit should not\n> touch the packed refs file. If we do other stuff involving more than\n> one ref, we may accept touching the packed refs file and have a process\n> which takes slightly longer but can handle more complex requests correctly,\n> such as renaming into and from directories (topic/1 -> topic and reverse).\n> Renaming is currently not part of the transaction API because of the (D/F)\n> problems. This proposed change would enable having renames being part of\n> the transactions API.\n> \n> A transaction covers creating, deleting and updating a ref and its reflog.\n> Renaming would be a deletion followed by creating a new ref atomically.\n\nA rename is a little bit more than a generic delete+create pair; it also\nmoves the reflog from the old name to the new name. Is your plan to add\nan extra \"rename\" operation to the refs-transactions API, or to\nautomatically detect delete+create pairs and treat them as renames?\n\n> So for here is my proposal for small transactions:\n> (just one ref [and/or reflog] touched):\n> \n> In ref_transaction_update:\n> \t* create $REF.lock file\n> \t* write new content to the lock file\n> \n> In ref_transaction_commit\n> \t* commit the .lock file to its destination\n> \t* in case this is a deletion:\n> \t\t* remove the loose ref\n> \t\t* and repack the packed refs file if necessary\n\nThe above describes the current algorithm, but FYI it is not entirely\ncorrect. The deletion of the loose ref might expose an old version of\nthe reference in the packed-refs file (which might even point at an\nobject that has been garbage-collected. So the reference has to be\ndeleted from the packed-refs file before the loose ref version is deleted.\n\nHowever, it is important that the packed-ref lock be held during the\nwhole procedure, so that a pack-refs process doesn't rewrite the loose\nref version of the reference into the (now-unlocked) packed-refs file,\ncausing the reference to survive its supposed deletion. (At least that\nwas the status a while ago; I don't know if recent changes to pack-refs\nmight have removed this problem in another way.)\n\nBut activating a new packed-refs file while still holding the\npacked-refs lock is not supported by our current lockfile API. In fact,\nworking towards enabling this was one of the reasons for the big\nlockfile refactoring that I did a while back. Though I never got as far\nas fixing this bug.\n\n> The larger transactions would be handled differently by relying\n> on the packed refs file:\n> In ref_transaction_update:\n> \t* detect if we transition to a large transaction\n> \t  (by having more than one entry in transaction->updates)\n> \t  if so:\n> \t\t* Pack all currently existing refs into the packed\n> \t\t  refs file, commit the packed refs file and delete\n> \t\t  all loose refs. This will avoid (d/f) conflicts.\n> \n> \t\t* Keep the packed-refs file locked and move the first\n> \t\t  transaction update into the packed-refs.lock file\n\nNB: this requires not just one but two rewrites of the packed-refs file,\nsharpening the performance concerns expressed elsewhere in this thread.\n\nBut couldn't one of the rewrites be avoided if the transaction doesn't\ninvolve any deletes?\n\n> \t* Any update(delete, create, update) is put into the locked\n> \t  packed refs file.\n> \t* Additionally we need to obtain the .lock for the loose refs\n> \t  file to keep guarantees, though we should close the file\n> \t  descriptor as we don't wand to run out of file descriptors.\n> \n> In ref_transaction_commit:\n> \t* We only need to commit the packed refs file\n> \t* Discard all other lock files as the changes get committed as a whole\n> \t  by the packed refs file\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"}]}