{"thread":{"id":"49011","subject":"Question regarding quarantine environments","startedAt":"2018-08-02T17:58:57Z","lastAt":"2018-08-03T13:30:02Z","messageCount":7,"participants":["Liam Decker","Jeff King","Jonathan Nieder","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"354284","messageId":"CALcD6sYfcaftciYU+2AHpzabyOS9ohRcZoE6LDuMyht+dxktHw@mail.gmail.com","threadId":"49011","inReplyTo":null,"subject":"Question regarding quarantine environments","fromName":"Liam Decker","fromEmail":"ldecker@indeed.com","sentAt":"2018-08-02T17:58:52Z","receivedAt":"2018-08-02T17:58:57Z","isPatch":false,"sender":{"key":"ldecker@indeed.com","avatar":null},"body":"Hi all,\n\nI've been working on a git hook in golang recently. However, the library I\nwas using did not support a possible quarantine directory, which I would\nuse for my hook.\n\nI have been trying to find out how git finds this incoming directory in the\nobjects folder, as their code simply assumed it resided in\n.git/objects/<1st byte>/<last 19 bytes>\nI read the documentation describing the git repository layout here [1] as\nwell as the objects documentation here [2] and the git hooks documentation\nhere [3] directed me to the receive-pack documentation here [4]\nI have also tried googling, but this is a pretty specific question\n\nThe solution that I implemented was to check the objects directory for the\nobject, and if it was not there, to look for a quarantine directory and try\nthere. However, that feels fairly inefficient.\nFor the curious, the library and solution I attempted are both here [5]\n\nIf anyone could help direct me to find specifically how git looks for\nobjects in the repository, I would be very grateful\n\n[1] https://git-scm.com/docs/gitrepository-layout\n[2] https://git-scm.com/book/en/v2/Git-Internals-Git-Objects\n[3] https://git-scm.com/docs/githooks\n[4] https://git-scm.com/docs/git-receive-pack\n[5] https://github.com/src-d/go-git/pull/887\n"},{"id":"354291","messageId":"20180802183924.GB23690@sigill.intra.peff.net","threadId":"49011","inReplyTo":"CALcD6sYfcaftciYU+2AHpzabyOS9ohRcZoE6LDuMyht+dxktHw@mail.gmail.com","subject":"Re: Question regarding quarantine environments","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-02T18:39:25Z","receivedAt":"2018-08-02T18:39:28Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 02, 2018 at 12:58:52PM -0500, Liam Decker wrote:\n\n> I've been working on a git hook in golang recently. However, the library I\n> was using did not support a possible quarantine directory, which I would\n> use for my hook.\n> \n> I have been trying to find out how git finds this incoming directory in the\n> objects folder, as their code simply assumed it resided in\n> .git/objects/<1st byte>/<last 19 bytes>\n\nWhen you're running a hook inside the quarantine environment, then\n$GIT_OBJECT_DIRECTORY in the environment will be set to the quarantine\ndirectory, and $GIT_ALTERNATE_OBJECT_DIRECTORIES will point to the main\nrepository object directory (possibly alongside other alternates, if\nthere were any already set).\n\nAny Git commands you run should therefore find objects from either\nlocation, but any writes would go to the quarantine (most notably, Git's\nown index-pack/unpack-objects processes, which is the point of the\nquarantine in the first place).\n\n> The solution that I implemented was to check the objects directory for the\n> object, and if it was not there, to look for a quarantine directory and try\n> there. However, that feels fairly inefficient.\n\nThat's more or less what Git will do under the hood (though in the\nopposite order).\n\n> For the curious, the library and solution I attempted are both here [5]\n\nJust skimming, but it sounds like go-git does not support the\nGIT_OBJECT_DIRECTORY environment variable.\n\n-Peff\n"},{"id":"354345","messageId":"20180803024902.GC197924@aiede.svl.corp.google.com","threadId":"49011","inReplyTo":"20180802183924.GB23690@sigill.intra.peff.net","subject":"Re: Question regarding quarantine environments","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-08-03T02:49:02Z","receivedAt":"2018-08-03T02:49:06Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff King wrote:\n> On Thu, Aug 02, 2018 at 12:58:52PM -0500, Liam Decker wrote:\n\n>> The solution that I implemented was to check the objects directory for the\n>> object, and if it was not there, to look for a quarantine directory and try\n>> there. However, that feels fairly inefficient.\n>\n> That's more or less what Git will do under the hood (though in the\n> opposite order).\n>\n>> For the curious, the library and solution I attempted are both here [5]\n>\n> Just skimming, but it sounds like go-git does not support the\n> GIT_OBJECT_DIRECTORY environment variable.\n\nTo be clear: we don't guarantee that the quarantine directory in the\nfuture will be where it is today.\n\nSo as Peff hinted, supporting GIT_OBJECT_DIRECTORY in go-git is likely\nto be the best way forward for your tool.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"354379","messageId":"878t5ntxqs.fsf@evledraar.gmail.com","threadId":"49011","inReplyTo":"20180802183924.GB23690@sigill.intra.peff.net","subject":"Re: Question regarding quarantine environments","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-08-03T12:56:11Z","receivedAt":"2018-08-03T12:56:16Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 02 2018, Jeff King wrote:\n\n> On Thu, Aug 02, 2018 at 12:58:52PM -0500, Liam Decker wrote:\n>\n>> I've been working on a git hook in golang recently. However, the library I\n>> was using did not support a possible quarantine directory, which I would\n>> use for my hook.\n>>\n>> I have been trying to find out how git finds this incoming directory in the\n>> objects folder, as their code simply assumed it resided in\n>> .git/objects/<1st byte>/<last 19 bytes>\n>\n> When you're running a hook inside the quarantine environment, then\n> $GIT_OBJECT_DIRECTORY in the environment will be set to the quarantine\n> directory, and $GIT_ALTERNATE_OBJECT_DIRECTORIES will point to the main\n> repository object directory (possibly alongside other alternates, if\n> there were any already set).\n>\n> Any Git commands you run should therefore find objects from either\n> location, but any writes would go to the quarantine (most notably, Git's\n> own index-pack/unpack-objects processes, which is the point of the\n> quarantine in the first place).\n\nTo add to this, one interesting thing that you can do with hooks because\nof this quarantine is to answer certain questions about the push that\nwere prohibitively expensive before it existed, but there's no explicit\ndocumentation for this.\n\nE.g. for a hook that wants to ban big blobs in the repo, but wants to\nallow all existing blobs (you don't want to block e.g. a revert of a\ncommit that removed it from the checkout), you can juggle these two env\nvariables and hide the \"main\" object dir from the hook for some\noperations, so e.g. if a blob lookup succeeds in the alternate\nquarantine dir, but not the main object dir, you know it's new.\n\n>> The solution that I implemented was to check the objects directory for the\n>> object, and if it was not there, to look for a quarantine directory and try\n>> there. However, that feels fairly inefficient.\n>\n> That's more or less what Git will do under the hood (though in the\n> opposite order).\n>\n>> For the curious, the library and solution I attempted are both here [5]\n>\n> Just skimming, but it sounds like go-git does not support the\n> GIT_OBJECT_DIRECTORY environment variable.\n>\n> -Peff\n"},{"id":"354380","messageId":"20180803130058.GA4671@sigill.intra.peff.net","threadId":"49011","inReplyTo":"878t5ntxqs.fsf@evledraar.gmail.com","subject":"Re: Question regarding quarantine environments","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-03T13:00:58Z","receivedAt":"2018-08-03T13:01:01Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 03, 2018 at 02:56:11PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > Any Git commands you run should therefore find objects from either\n> > location, but any writes would go to the quarantine (most notably, Git's\n> > own index-pack/unpack-objects processes, which is the point of the\n> > quarantine in the first place).\n> \n> To add to this, one interesting thing that you can do with hooks because\n> of this quarantine is to answer certain questions about the push that\n> were prohibitively expensive before it existed, but there's no explicit\n> documentation for this.\n> \n> E.g. for a hook that wants to ban big blobs in the repo, but wants to\n> allow all existing blobs (you don't want to block e.g. a revert of a\n> commit that removed it from the checkout), you can juggle these two env\n> variables and hide the \"main\" object dir from the hook for some\n> operations, so e.g. if a blob lookup succeeds in the alternate\n> quarantine dir, but not the main object dir, you know it's new.\n\nI'd be a bit careful with that, though, as the definition of \"new\" is\nvague there.\n\nFor example, completing a thin pack may mean that the receiver creates a\ncopy of a base object found in the main repo. That object isn't new as\npart of the push, nor was it even sent on the wire, but it will appear\nin the quarantine directory. But only sometimes, depending on whether we\nkept the sender's pack or exploded it to loose objects.\n\n-Peff\n"},{"id":"354384","messageId":"877el7twej.fsf@evledraar.gmail.com","threadId":"49011","inReplyTo":"20180803130058.GA4671@sigill.intra.peff.net","subject":"Re: Question regarding quarantine environments","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-08-03T13:25:08Z","receivedAt":"2018-08-03T13:25:13Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Aug 03 2018, Jeff King wrote:\n\n> On Fri, Aug 03, 2018 at 02:56:11PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> > Any Git commands you run should therefore find objects from either\n>> > location, but any writes would go to the quarantine (most notably, Git's\n>> > own index-pack/unpack-objects processes, which is the point of the\n>> > quarantine in the first place).\n>>\n>> To add to this, one interesting thing that you can do with hooks because\n>> of this quarantine is to answer certain questions about the push that\n>> were prohibitively expensive before it existed, but there's no explicit\n>> documentation for this.\n>>\n>> E.g. for a hook that wants to ban big blobs in the repo, but wants to\n>> allow all existing blobs (you don't want to block e.g. a revert of a\n>> commit that removed it from the checkout), you can juggle these two env\n>> variables and hide the \"main\" object dir from the hook for some\n>> operations, so e.g. if a blob lookup succeeds in the alternate\n>> quarantine dir, but not the main object dir, you know it's new.\n>\n> I'd be a bit careful with that, though, as the definition of \"new\" is\n> vague there.\n>\n> For example, completing a thin pack may mean that the receiver creates a\n> copy of a base object found in the main repo. That object isn't new as\n> part of the push, nor was it even sent on the wire, but it will appear\n> in the quarantine directory. But only sometimes, depending on whether we\n> kept the sender's pack or exploded it to loose objects.\n\nRight, I mean:\n\n    is_new = !in_quarantine() && in_main()\n\nOr:\n\n    is_new = !in_main()\n\nShould work, in the latter case if the object really is missing from the\nquarnatine too, other fsck bits will stop the push.\n\nBut as you point out:\n\n    is_new = in_quarantine()\n\nCannot be relied upon, although it'll be true most of the time.\n\nPerhaps I'm missing some edge case above, but I wanted to reword it to\nmake sure I understood it correctly (and perhaps you have a correction).\n"},{"id":"354386","messageId":"20180803132959.GB28939@sigill.intra.peff.net","threadId":"49011","inReplyTo":"877el7twej.fsf@evledraar.gmail.com","subject":"Re: Question regarding quarantine environments","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-03T13:29:59Z","receivedAt":"2018-08-03T13:30:02Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 03, 2018 at 03:25:08PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > I'd be a bit careful with that, though, as the definition of \"new\" is\n> > vague there.\n> >\n> > For example, completing a thin pack may mean that the receiver creates a\n> > copy of a base object found in the main repo. That object isn't new as\n> > part of the push, nor was it even sent on the wire, but it will appear\n> > in the quarantine directory. But only sometimes, depending on whether we\n> > kept the sender's pack or exploded it to loose objects.\n> \n> Right, I mean:\n> \n>     is_new = !in_quarantine() && in_main()\n> \n> Or:\n> \n>     is_new = !in_main()\n> \n> Should work, in the latter case if the object really is missing from the\n> quarnatine too, other fsck bits will stop the push.\n\nAh, OK. Yes, I agree that should work to cover new objects (including\nones that the other side but aren't actually needed to update the refs,\nthough hopefully that is rare).\n\nThere may also be other object stores, if the main repository used\nalternates (or if somebody set GIT_ALTERNATE_OBJECT_DIRECTORIES). You\ncan probably disregard that, though, as:\n\n  1. If you ignore the main repo, presumably you ignore its\n     recursive info/alternates, too.\n\n  2. The easy mechanism for ignoring the main repo is to ignore\n     GIT_ALTERNATE_OBJECT_DIRECTORIES, so you'd already be handling\n     that.\n\n> But as you point out:\n> \n>     is_new = in_quarantine()\n> \n> Cannot be relied upon, although it'll be true most of the time.\n> \n> Perhaps I'm missing some edge case above, but I wanted to reword it to\n> make sure I understood it correctly (and perhaps you have a correction).\n\nNope, I just didn't think through what you were saying carefully enough. ;)\n\n-Peff\n"}]}