{"thread":{"id":"47361","subject":"[RFE] Inverted sparseness","startedAt":"2017-12-01T17:22:14Z","lastAt":"2017-12-04T11:54:41Z","messageCount":6,"participants":["Randall S. Becker","Jeff Hostetler","Philip Oakley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"333922","messageId":"004201d36ac8$db62b900$92282b00$@nexbridge.com","threadId":"47361","inReplyTo":null,"subject":"[RFE] Inverted sparseness","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2017-12-01T17:21:37Z","receivedAt":"2017-12-01T17:22:14Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"I recently encountered a really strange use-case relating to sparse clone/fetch that is really backwards from the discussion that has been going on, and well, I'm a bit embarrassed to bring it up, but I have no good solution including building a separate data store that will end up inconsistent with repositories (a bad solution).  The use-case is as follows:\n\nGiven a backbone of multiple git repositories spread across an organization with a server farm and upstream vendors.\nThe vendor delivers code by having the client perform git pull into a specific branch.\nThe customer may take the code as is or merge in customizations.\nThe vendor wants to know exactly what commit of theirs is installed on each server, in near real time.\nThe customer is willing to push the commit-ish to the vendor's upstream repo but does not want, by default, to share the actual commit contents for security reasons.\n\tRealistically, the vendor needs to know that their own commit id was put somewhere (process exists to track this, so not part of the use-case) and whether there is a subsequent commit contributed by the customer, but the content is not relevant initially.\n\nAfter some time, the vendor may request the commit contents from the customer in order to satisfy support requirements - a.k.a. a defect was found but has to be resolved.\nThe customer would then perform a deeper push that looks a lot like a \"slightly\" symmetrical operation of a deep fetch following a prior sparse fetch to supply the vendor with the specific commit(s).\n\nThis is not hard to realize if the sparse commit is HEAD on a branch, but if its inside a tree, well, I don't even know where to start. To self-deprecate, this is likely a bad idea, but it has come up a few times.\n\nThoughts? Nasty Remarks?\n\nRandall\n\n-- Brief whoami: NonStop&UNIX developer since approximately UNIX(421664400)/NonStop(211288444200000000) \n-- In my real life, I talk too much.\n\n\n\n"},{"id":"333929","messageId":"bdd01692-198a-f5ec-3c88-7d99e4adced5@jeffhostetler.com","threadId":"47361","inReplyTo":"004201d36ac8$db62b900$92282b00$@nexbridge.com","subject":"Re: [RFE] Inverted sparseness","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-01T18:19:01Z","receivedAt":"2017-12-01T18:19:52Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 12/1/2017 12:21 PM, Randall S. Becker wrote:\n> I recently encountered a really strange use-case relating to sparse clone/fetch that is really backwards from the discussion that has been going on, and well, I'm a bit embarrassed to bring it up, but I have no good solution including building a separate data store that will end up inconsistent with repositories (a bad solution).  The use-case is as follows:\n> \n> Given a backbone of multiple git repositories spread across an organization with a server farm and upstream vendors.\n> The vendor delivers code by having the client perform git pull into a specific branch.\n> The customer may take the code as is or merge in customizations.\n> The vendor wants to know exactly what commit of theirs is installed on each server, in near real time.\n> The customer is willing to push the commit-ish to the vendor's upstream repo but does not want, by default, to share the actual commit contents for security reasons.\n> \tRealistically, the vendor needs to know that their own commit id was put somewhere (process exists to track this, so not part of the use-case) and whether there is a subsequent commit contributed by the customer, but the content is not relevant initially.\n> \n> After some time, the vendor may request the commit contents from the customer in order to satisfy support requirements - a.k.a. a defect was found but has to be resolved.\n> The customer would then perform a deeper push that looks a lot like a \"slightly\" symmetrical operation of a deep fetch following a prior sparse fetch to supply the vendor with the specific commit(s).\n> \n> This is not hard to realize if the sparse commit is HEAD on a branch, but if its inside a tree, well, I don't even know where to start. To self-deprecate, this is likely a bad idea, but it has come up a few times.\n> \n> Thoughts? Nasty Remarks?\n> \n> Randall\n\nPerhaps I'm not understanding the subtleties of what you're describing,\nbut could you do this with stock git functionality.\n\nLet the vendor publish a \"well known branch\" for the client.\nLet the client pull that and build.\nLet the client create a branch set to the same commit that they fetched.\nLet the client push that branch as a client-specific branch to\nthe vendor to indicate that that is the official release they\nare based on.\n\nThen the vendor would know the official commit that the client was\nusing.\n\nIf the client makes local changes, does the vendor really need the\nSHA of those -- without the actual content?  I mean any SHA would\ndo right?  Perhaps let the client create a second client-specific\nbranch (set to the same commit as the first) to indicate they had\nmods.\n\nLater, when the vendor needs the actual client changes, the client\ndoes a normal push to this 2nd client-specific branch at the vendor.\nThis would send everything that the client has done to the code\nsince the official release.\n\nI'm not sure what you mean about \"it is inside a tree\".\n\nHope this helps,\nJeff\n\n"},{"id":"333933","messageId":"004701d36ad2$9a7fcea0$cf7f6be0$@nexbridge.com","threadId":"47361","inReplyTo":"bdd01692-198a-f5ec-3c88-7d99e4adced5@jeffhostetler.com","subject":"RE: [RFE] Inverted sparseness","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2017-12-01T18:31:23Z","receivedAt":"2017-12-01T18:31:36Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On December 1, 2017 1:19 PM, Jeff Hostetler wrote:\n>On 12/1/2017 12:21 PM, Randall S. Becker wrote:\n>> I recently encountered a really strange use-case relating to sparse clone/fetch that is really backwards from the discussion that has been going on, and well, I'm a bit embarrassed to bring it up, but I have no good solution including building a separate data store that will end up inconsistent with repositories (a bad solution).  The use-case is as follows:\n>> \n>> Given a backbone of multiple git repositories spread across an organization with a server farm and upstream vendors.\n>> The vendor delivers code by having the client perform git pull into a specific branch.\n>> The customer may take the code as is or merge in customizations.\n>> The vendor wants to know exactly what commit of theirs is installed on each server, in near real time.\n>> The customer is willing to push the commit-ish to the vendor's upstream repo but does not want, by default, to share the actual commit contents for security reasons.\n>> \tRealistically, the vendor needs to know that their own commit id was put somewhere (process exists to track this, so not part of the use-case) and whether there is a subsequent commit contributed >by the customer, but the content is not relevant initially.\n>> \n>> After some time, the vendor may request the commit contents from the customer in order to satisfy support requirements - a.k.a. a defect was found but has to be resolved.\n>> The customer would then perform a deeper push that looks a lot like a \"slightly\" symmetrical operation of a deep fetch following a prior sparse fetch to supply the vendor with the specific commit(s).\n\n>Perhaps I'm not understanding the subtleties of what you're describing, but could you do this with stock git functionality.\n\n>Let the vendor publish a \"well known branch\" for the client.\n>Let the client pull that and build.\n>Let the client create a branch set to the same commit that they fetched.\n>Let the client push that branch as a client-specific branch to the vendor to indicate that that is the official release they are based on.\n\n>Then the vendor would know the official commit that the client was using.\nThis is the easy part, and it doesn't require anything sparse to exist.\n\n>If the client makes local changes, does the vendor really need the SHA of those -- without the actual content?\n>I mean any SHA would do right?  Perhaps let the client create a second client-specific branch (set to\n> the same commit as the first) to indicate they had mods.\n>Later, when the vendor needs the actual client changes, the client does a normal push to this 2nd client-specific branch at the vendor.\n>This would send everything that the client has done to the code since the official release.\n\nWhat I should have added to the use-case was that there is a strong audit requirement (regulatory, actually) involved that the SHA is exact, immutable, and cannot be substitute or forged (one of the reasons git is in such high regard). So, no I can't arrange a fake SHA to represent a SHA to be named later. It SHA of the installed commit is part of the official record of what happened on the specific server, so I'm stuck with it.\n\n>I'm not sure what you mean about \"it is inside a tree\".\n\nm---a---b---c---H1\n          `---d---H2\n\nd would be at a head. b would be inside. Determining content of c is problematic if b is sparse, so I'm really unsure that any of this is possible.\n\nCheers,\nRandall\n\n-- Brief whoami: NonStop&UNIX developer since approximately UNIX(421664400)/NonStop(211288444200000000) \n-- In my real life, I talk too much.\n\n\n\n"},{"id":"334026","messageId":"A05AA29AD20041A2B31E7A5CD56BB682@PhilipOakley","threadId":"47361","inReplyTo":"004701d36ad2$9a7fcea0$cf7f6be0$@nexbridge.com","subject":"Re: [RFE] Inverted sparseness","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-12-03T23:14:17Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Randall S. Becker\" <rsbecker@nexbridge.com>\nSent: Friday, December 01, 2017 6:31 PM\n> On December 1, 2017 1:19 PM, Jeff Hostetler wrote:\n>>On 12/1/2017 12:21 PM, Randall S. Becker wrote:\n>>> I recently encountered a really strange use-case relating to sparse \n>>> clone/fetch that is really backwards from the discussion that has been \n>>> going on, and well, I'm a bit embarrassed to bring it up, but I have no \n>>> good solution including building a separate data store that will end up \n>>> inconsistent with repositories (a bad solution).  The use-case is as \n>>> follows:\n>>>\n>>> Given a backbone of multiple git repositories spread across an \n>>> organization with a server farm and upstream vendors.\n>>> The vendor delivers code by having the client perform git pull into a \n>>> specific branch.\n>>> The customer may take the code as is or merge in customizations.\n>>> The vendor wants to know exactly what commit of theirs is installed on \n>>> each server, in near real time.\n>>> The customer is willing to push the commit-ish to the vendor's upstream \n>>> repo but does not want, by default, to share the actual commit contents \n>>> for security reasons.\n>>> Realistically, the vendor needs to know that their own commit id was put \n>>> somewhere (process exists to track this, so not part of the use-case) \n>>> and whether there is a subsequent commit contributed >by the customer, \n>>> but the content is not relevant initially.\n>>>\n>>> After some time, the vendor may request the commit contents from the \n>>> customer in order to satisfy support requirements - a.k.a. a defect was \n>>> found but has to be resolved.\n>>> The customer would then perform a deeper push that looks a lot like a \n>>> \"slightly\" symmetrical operation of a deep fetch following a prior \n>>> sparse fetch to supply the vendor with the specific commit(s).\n>\n>>Perhaps I'm not understanding the subtleties of what you're describing, \n>>but could you do this with stock git functionality.\n>\n>>Let the vendor publish a \"well known branch\" for the client.\n>>Let the client pull that and build.\n>>Let the client create a branch set to the same commit that they fetched.\n>>Let the client push that branch as a client-specific branch to the vendor \n>>to indicate that that is the official release they are based on.\n>\n>>Then the vendor would know the official commit that the client was using.\n> This is the easy part, and it doesn't require anything sparse to exist.\n>\n>>If the client makes local changes, does the vendor really need the SHA of \n>>those -- without the actual content?\n>>I mean any SHA would do right?  Perhaps let the client create a second \n>>client-specific branch (set to\n>> the same commit as the first) to indicate they had mods.\n>>Later, when the vendor needs the actual client changes, the client does a \n>>normal push to this 2nd client-specific branch at the vendor.\n>>This would send everything that the client has done to the code since the \n>>official release.\n>\n> What I should have added to the use-case was that there is a strong audit \n> requirement (regulatory, actually) involved that the SHA is exact, \n> immutable, and cannot be substitute or forged (one of the reasons git is \n> in such high regard). So, no I can't arrange a fake SHA to represent a SHA \n> to be named later. It SHA of the installed commit is part of the official \n> record of what happened on the specific server, so I'm stuck with it.\n>\n>>I'm not sure what you mean about \"it is inside a tree\".\n>\n> m---a---b---c---H1\n>          `---d---H2\n>\n> d would be at a head. b would be inside. Determining content of c is \n> problematic if b is sparse, so I'm really unsure that any of this is \n> possible.\n>\n> Cheers,\n> Randall\n>\n> -- Brief whoami: NonStop&UNIX developer since approximately \n> UNIX(421664400)/NonStop(211288444200000000)\n> -- In my real life, I talk too much.\n\nI think I get the jist of your use case. Would I be right that you don't \nhave a true working solution yet? i.e. that it's a problem that is almost \nsorted but falls down at the last step.\n\nIf one pretended that this was a single development shop, and the various \nvendors, clients and customers as being independent devolopers, each of whom \nis over protective of their code, it may give a better view that maps onto \nclassic feature development diagrams. (i.e draw the answer for local devs, \nthen mark where the splits happen)\n\nIn particular, I think you could use a notional regulator's view that the \nwhole code base is part of a large Git heirarchy of branches and merges, and \nthat some of the feature loops are only available via the particular \ndeveloper that worked on that feature.\n\nThis would mean that from a regulatory overview there is a merge commit in \nthe 'main' (master) heirachy that has the main and feature commits listed, \nand the feature commit is probably an --allow-empty commit (that has an \nempty tree if they are that paranoid) that says 'function X released' (and \nprobably tagged), and that release commit then has, as its parent, the true \nrelease commit, with the true code tree. The latter commit isn't actually \nbeing shown to you!\n\nAt this point the potential for using the graft capability comes in (as a \nregulated method!). Locally the graft records the missing line of pearls for \nthat paranoid dev/vendor/customer/client. The whole git heirachy still \nworks.\n\nThe question is how to get that  release commit with its empty tree, and its \ntag, to you from the dev. I'd guess that a fast-export of just that \ntag/commit/empty tree would allow you to bring in that sentinel point to \nyour heirachy (initially as a psuedo --root), and then graft it on. (I \nhaven't checked if fast-export allows such specificity, but it's a method)\n\nYou can now form the merge commit and have regulatory oversight and the full \ngit validation and verification capability, as long as your web of trust \nextends to the regulator looking effectively across the air gap. It's a \nfresh way of seeing the web of trust.\n\nThus you/they have various \"shallow clones\", but with gaps and islands in \nthe shallowness....  and those gaps are spanned by grafts (which are \naudited). The `git-replace` may also be an option, but I don't think it's \nquite right for this case. You just have a temporary gap in the history, and \nwith fast export\n\nIf using the empty tree part doesn't pass muster (i.e. showing nothing isn't \nsufficient), then the narrow clone could come into play to limit what parts \nof the trees are widely visible, but mainly its using the grafts to cover \nthe regulatory gap, and (for the moment) using fast-export to transfer the \nsingleton commit / tags\n\nOh Just remembered, there is the newish capability to fetch random blobs, so \nthat may help.\n\nPhilip\n\n\n"},{"id":"334029","messageId":"004c01d36c90$a4f27580$eed76080$@nexbridge.com","threadId":"47361","inReplyTo":"A05AA29AD20041A2B31E7A5CD56BB682@PhilipOakley","subject":"RE: [RFE] Inverted sparseness","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2017-12-03T23:44:16Z","receivedAt":"2017-12-03T23:44:38Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On December 3, 2017 6:14 PM, Philip Oakley wrote a nugget of wisdom: \n>From: \"Randall S. Becker\" <rsbecker@nexbridge.com>\n>Sent: Friday, December 01, 2017 6:31 PM\n>> On December 1, 2017 1:19 PM, Jeff Hostetler wrote:\n>>>On 12/1/2017 12:21 PM, Randall S. Becker wrote:\n>>>> I recently encountered a really strange use-case relating to sparse \n>>>> clone/fetch that is really backwards from the discussion that has \n>>>> been going on, and well, I'm a bit embarrassed to bring it up, but I \n>>>> have no good solution including building a separate data store that \n>>>> will end up inconsistent with repositories (a bad solution).  The \n>>>> use-case is as\n>>>> follows:\n>>>>\n>>>> Given a backbone of multiple git repositories spread across an \n>>>> organization with a server farm and upstream vendors.\n>>>> The vendor delivers code by having the client perform git pull into \n>>>> a specific branch.\n>>>> The customer may take the code as is or merge in customizations.\n>>>> The vendor wants to know exactly what commit of theirs is installed \n>>>> on each server, in near real time.\n>>>> The customer is willing to push the commit-ish to the vendor's \n>>>> upstream repo but does not want, by default, to share the actual \n>>>> commit contents for security reasons.\n>>>> Realistically, the vendor needs to know that their own commit id was \n>>>> put somewhere (process exists to track this, so not part of the \n>>>> use-case) and whether there is a subsequent commit contributed >by \n>>>> the customer, but the content is not relevant initially.\n>>>>\n>>>> After some time, the vendor may request the commit contents from the \n>>>> customer in order to satisfy support requirements - a.k.a. a defect \n>>>> was found but has to be resolved.\n>>>> The customer would then perform a deeper push that looks a lot like \n>>>> a \"slightly\" symmetrical operation of a deep fetch following a prior \n>>>> sparse fetch to supply the vendor with the specific commit(s).\n>>\n>>>Perhaps I'm not understanding the subtleties of what you're \n>>>describing, but could you do this with stock git functionality.\n>>\n>>>Let the vendor publish a \"well known branch\" for the client.\n>>>Let the client pull that and build.\n>>>Let the client create a branch set to the same commit that they fetched.\n>>>Let the client push that branch as a client-specific branch to the \n>>>vendor to indicate that that is the official release they are based on.\n>>\n>>>Then the vendor would know the official commit that the client was using.\n>> This is the easy part, and it doesn't require anything sparse to exist.\n>>\n>>>If the client makes local changes, does the vendor really need the SHA \n>>>of those -- without the actual content?\n>>>I mean any SHA would do right?  Perhaps let the client create a second \n>>>client-specific branch (set to  the same commit as the first) to \n>>>indicate they had mods.\n>>>Later, when the vendor needs the actual client changes, the client \n>>>does a normal push to this 2nd client-specific branch at the vendor.\n>>>This would send everything that the client has done to the code since \n>>>the official release.\n>>\n>> What I should have added to the use-case was that there is a strong \n>> audit requirement (regulatory, actually) involved that the SHA is \n>> exact, immutable, and cannot be substitute or forged (one of the \n>> reasons git is in such high regard). So, no I can't arrange a fake SHA \n>> to represent a SHA to be named later. It SHA of the installed commit \n>> is part of the official record of what happened on the specific server, so I'm stuck with it.\n>>\n>>>I'm not sure what you mean about \"it is inside a tree\".\n>>\n>> m---a---b---c---H1\n>>          `---d---H2\n>>\n>> d would be at a head. b would be inside. Determining content of c is \n>> problematic if b is sparse, so I'm really unsure that any of this is \n>> possible.\n\n>I think I get the jist of your use case. Would I be right that you don't have a true working\n>solution yet? i.e. that it's a problem that is almost sorted but falls down at the last step.\n\n>If one pretended that this was a single development shop, and the various vendors, clients\n>and customers as being independent devolopers, each of whom is over protective of their\n>code, it may give a better view that maps onto classic feature development diagrams.\n>(i.e draw the answer for local devs, then mark where the splits happen)\n\n>In particular, I think you could use a notional regulator's view that the whole code base is\n>part of a large Git heirarchy of branches and merges, and that some of the feature loops\n>are only available via the particular developer that worked on that feature.\n\n>This would mean that from a regulatory overview there is a merge commit in the 'main'\n>(master) heirachy that has the main and feature commits listed, and the feature commit\n>is probably an --allow-empty commit (that has an empty tree if they are that paranoid) that\n>says 'function X released' (and probably tagged), and that release commit then has, as its\n>parent, the true release commit, with the true code tree. The latter commit isn't actually being\n>shown to you!\n\n>At this point the potential for using the graft capability comes in (as a regulated method!). \n>Locally the graft records the missing line of pearls for that paranoid dev/vendor/customer/client. \n>The whole git heirachy still works.\n\n>The question is how to get that  release commit with its empty tree, and its tag, to you from\n>the dev. I'd guess that a fast-export of just that tag/commit/empty tree would allow you to\n>bring in that sentinel point to your heirachy (initially as a psuedo --root), and then graft it\n>on. (I haven't checked if fast-export allows such specificity, but it's a method)\n\n>You can now form the merge commit and have regulatory oversight and the full git validation\n>and verification capability, as long as your web of trust extends to the regulator looking\n>effectively across the air gap. It's a fresh way of seeing the web of trust.\n\n>Thus you/they have various \"shallow clones\", but with gaps and islands in the shallowness.... \n>and those gaps are spanned by grafts (which are audited). The `git-replace` may also be an\n>option, but I don't think it's quite right for this case. You just have a temporary gap in\n>the history, and with fast export\n\n>If using the empty tree part doesn't pass muster (i.e. showing nothing isn't sufficient), \n>then the narrow clone could come into play to limit what parts of the trees are widely\n>visible, but mainly its using the grafts to cover the regulatory gap, and (for the moment) \n>using fast-export to transfer the singleton commit / tags\n\n>Oh Just remembered, there is the newish capability to fetch random blobs, so that may help.\n\nI think you hit the nail on the head pretty well. We're currently at 2.3.7, with a push to 2.15.1 this week, so I'm looking forward to trying this. My two worries are whether the empty tree is acceptable (it should be to the client, and might be to the vendor), and doing this reliably (semi-automated) so the user base does not have to worry about the gory details of doing this. The unit tests for it are undoubtedly going to give me headaches.\n\nThanks for the advice. Islands of shallowness are a really descriptive image for what this is. So identifying that there are shoals (to extend the metaphor somewhat), will be crucial to this adventure.\n\nCheers,\nRandall\n\n-- Brief whoami: NonStop&UNIX developer since approximately UNIX(421664400)/NonStop(211288444200000000) \n-- In my real life, I talk too much.\n\n\n\n"},{"id":"334039","messageId":"0789FA44BEE345F3B1E664A317C79564@PhilipOakley","threadId":"47361","inReplyTo":"004c01d36c90$a4f27580$eed76080$@nexbridge.com","subject":"Re: [RFE] Inverted sparseness","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-12-04T11:54:41Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Randall S. Becker\"  :December 03, 2017 11:44 PM\nOn December 3, 2017 6:14 PM, Philip Oakley wrote a nugget of wisdom:\n>From: \"Randall S. Becker\" <rsbecker@nexbridge.com>\n>Sent: Friday, December 01, 2017 6:31 PM\n>> On December 1, 2017 1:19 PM, Jeff Hostetler wrote:\n>>>On 12/1/2017 12:21 PM, Randall S. Becker wrote:\n>>>> I recently encountered a really strange use-case relating to sparse\n>>>> clone/fetch that is really backwards from the discussion that has\n>>>> been going on, and well, I'm a bit embarrassed to bring it up, but I\n>>>> have no good solution including building a separate data store that\n>>>> will end up inconsistent with repositories (a bad solution).  The\n>>>> use-case is as\n>>>> follows:\n>>>>\n>>>> Given a backbone of multiple git repositories spread across an\n>>>> organization with a server farm and upstream vendors.\n>>>> The vendor delivers code by having the client perform git pull into\n>>>> a specific branch.\n>>>> The customer may take the code as is or merge in customizations.\n>>>> The vendor wants to know exactly what commit of theirs is installed\n>>>> on each server, in near real time.\n>>>> The customer is willing to push the commit-ish to the vendor's\n>>>> upstream repo but does not want, by default, to share the actual\n>>>> commit contents for security reasons.\n>>>> Realistically, the vendor needs to know that their own commit id was\n>>>> put somewhere (process exists to track this, so not part of the\n>>>> use-case) and whether there is a subsequent commit contributed >by\n>>>> the customer, but the content is not relevant initially.\n>>>>\n>>>> After some time, the vendor may request the commit contents from the\n>>>> customer in order to satisfy support requirements - a.k.a. a defect\n>>>> was found but has to be resolved.\n>>>> The customer would then perform a deeper push that looks a lot like\n>>>> a \"slightly\" symmetrical operation of a deep fetch following a prior\n>>>> sparse fetch to supply the vendor with the specific commit(s).\n>>\n>>>Perhaps I'm not understanding the subtleties of what you're\n>>>describing, but could you do this with stock git functionality.\n>>\n>>>Let the vendor publish a \"well known branch\" for the client.\n>>>Let the client pull that and build.\n>>>Let the client create a branch set to the same commit that they fetched.\n>>>Let the client push that branch as a client-specific branch to the\n>>>vendor to indicate that that is the official release they are based on.\n>>\n>>>Then the vendor would know the official commit that the client was using.\n>> This is the easy part, and it doesn't require anything sparse to exist.\n>>\n>>>If the client makes local changes, does the vendor really need the SHA\n>>>of those -- without the actual content?\n>>>I mean any SHA would do right?  Perhaps let the client create a second\n>>>client-specific branch (set to  the same commit as the first) to\n>>>indicate they had mods.\n>>>Later, when the vendor needs the actual client changes, the client\n>>>does a normal push to this 2nd client-specific branch at the vendor.\n>>>This would send everything that the client has done to the code since\n>>>the official release.\n>>\n>> What I should have added to the use-case was that there is a strong\n>> audit requirement (regulatory, actually) involved that the SHA is\n>> exact, immutable, and cannot be substitute or forged (one of the\n>> reasons git is in such high regard). So, no I can't arrange a fake SHA\n>> to represent a SHA to be named later. It SHA of the installed commit\n>> is part of the official record of what happened on the specific server,\n>> so I'm stuck with it.\n>>\n>>>I'm not sure what you mean about \"it is inside a tree\".\n>>\n>> m---a---b---c---H1\n>>          `---d---H2\n>>\n>> d would be at a head. b would be inside. Determining content of c is\n>> problematic if b is sparse, so I'm really unsure that any of this is\n>> possible.\n\n>I think I get the jist of your use case. Would I be right that you don't\n>have a true working\n>solution yet? i.e. that it's a problem that is almost sorted but falls down\n>at the last step.\n\n>If one pretended that this was a single development shop, and the various\n>vendors, clients\n>and customers as being independent devolopers, each of whom is over\n>protective of their\n>code, it may give a better view that maps onto classic feature development\n>diagrams.\n>(i.e draw the answer for local devs, then mark where the splits happen)\n\n>In particular, I think you could use a notional regulator's view that the\n>whole code base is\n>part of a large Git heirarchy of branches and merges, and that some of the\n>feature loops\n>are only available via the particular developer that worked on that\n>feature.\n\n>This would mean that from a regulatory overview there is a merge commit in\n>the 'main'\n>(master) heirachy that has the main and feature commits listed, and the\n>feature commit\n>is probably an --allow-empty commit (that has an empty tree if they are\n>that paranoid) that\n>says 'function X released' (and probably tagged), and that release commit\n>then has, as its\n>parent, the true release commit, with the true code tree. The latter commit\n>isn't actually being\n>shown to you!\n\n>At this point the potential for using the graft capability comes in (as a\n>regulated method!).\n>Locally the graft records the missing line of pearls for that paranoid\n>dev/vendor/customer/client.\n>The whole git heirachy still works.\n\n>The question is how to get that  release commit with its empty tree, and\n>its tag, to you from\n>the dev. I'd guess that a fast-export of just that tag/commit/empty tree\n>would allow you to\n>bring in that sentinel point to your heirachy (initially as a\n>psuedo --root), and then graft it\n>on. (I haven't checked if fast-export allows such specificity, but it's a\n>method)\n\n>You can now form the merge commit and have regulatory oversight and the\n>full git validation\n>and verification capability, as long as your web of trust extends to the\n>regulator looking\n>effectively across the air gap. It's a fresh way of seeing the web of\n>trust.\n\n>Thus you/they have various \"shallow clones\", but with gaps and islands in\n>the shallowness....\n>and those gaps are spanned by grafts (which are audited). The `git-replace`\n>may also be an\n>option, but I don't think it's quite right for this case. You just have a\n>temporary gap in\n>the history, and with fast export\n\n>If using the empty tree part doesn't pass muster (i.e. showing nothing\n>isn't sufficient),\n>then the narrow clone could come into play to limit what parts of the trees\n>are widely\n>visible, but mainly its using the grafts to cover the regulatory gap, and\n>(for the moment)\n>using fast-export to transfer the singleton commit / tags\n\n>Oh Just remembered, there is the newish capability to fetch random blobs,\n>so that may help.\n\nRandall said>\nI think you hit the nail on the head pretty well. We're currently at 2.3.7,\nwith a push to 2.15.1 this week, so I'm looking forward to trying this. My\ntwo worries are whether the empty tree is acceptable (it should be to the\nclient, and might be to the vendor), and doing this reliably\n(semi-automated) so the user base does not have to worry about the gory\ndetails of doing this. The unit tests for it are undoubtedly going to give\nme headaches.\n\nThanks for the advice. Islands of shallowness are a really descriptive image\nfor what this is. So identifying that there are shoals (to extend the\nmetaphor somewhat), will be crucial to this adventure.\n---\n\nAn overnight sleep remined me that the ideal way of transferring across the \nair gap is *Obviously* the use of `git bundle`\n\nBundle allows you to specify the exact revisions (the tag and commits) in \nthe bundle that is sneakernet transferred (or email) between the repos.\n\nI'm also thinking that the vendor/client/customet should also have, on their \nside, one of the empty merge commits that shows both the original fork-point \n(aka merge base) and their current (empty) release commit. This provides the \nauthentication and verification that they have used the right base commit \nfor their ox-bow lake of disconnected development (Oh the metaphors just \nkeep coming). It also provies a place for the automated/scripted graft to \nget the two ends of the graft from, and check they are valid.\n\nIt would be very easy for transcription errors to sneak in at that step of \nrecording the fork-point (which would be a bit of unexpected river capture - \nhttps://phys.org/news/2017-04-retreating-yukon-glacier-river.html), so \nmaking the client also do it removes that concern.\n\nThe creation of client side and your side empty-merges should also create a \ncriss-cross plaiting that locks the two processes together - it's almost a \nblock-chain!\n\nHope it goes well. It would be great to hear the result.\n--\nPhilip\n\n\n"}]}