{"thread":{"id":"35415","subject":"Pack transfer negotiation to tree and blob level?","startedAt":"2013-11-27T22:08:27Z","lastAt":"2013-11-28T08:32:36Z","messageCount":3,"participants":["Philip Oakley","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"231218","messageId":"1E0A3B7F917445C7A968C81C35892FD4@PhilipOakley","threadId":"35415","inReplyTo":null,"subject":"Pack transfer negotiation to tree and blob level?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2013-11-27T22:08:27Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"In the pack transfer protocol \n(Documentation\\technical\\pack-protocol.txt) the negotiation for refs is \ndiscussed, but its unclear to me if the negotiation explicitly navigates \ndown into the trees and blobs of each commit that need to go into the \npack.\n\n>From one perspective I can see that, in the main, it's only commit \nobjects that are being negotiated, and the DAG is used to imply which \ncommit objects are to be sent between the wants and haves end points, \nwithout need to descend into their trees and blobs. The tags and the \nobjects they point to are explicitly given so are negotiated easily.\n\nThe other view is that the negotiation should be listing every object of \nany type between the wants and haves as part of the negotiation. I just \ncouldn't tell from the docs which assumption is appropriate. Is there \nany extra clarifications on this?\n\nI ask as I was cogitating on options for a 'narrow' clone  (to \ncomplement shallow clones ;-) that could, say, in some way limit the \nsize of blobs downloaded, or the number of tree levels downloaded, or \neven path limiting.\n\nPhilip Oakley \n"},{"id":"231219","messageId":"CACsJy8C5om91ObOHvSjDyDtHfCB9Y2gSbgF8Dsi2oXCbO55Czw@mail.gmail.com","threadId":"35415","inReplyTo":"1E0A3B7F917445C7A968C81C35892FD4@PhilipOakley","subject":"Re: Pack transfer negotiation to tree and blob level?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-27T23:50:05Z","receivedAt":"2013-11-27T23:50:05Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Nov 28, 2013 at 5:52 AM, Philip Oakley <philipoakley@iee.org> wrote:\n> In the pack transfer protocol (Documentation\\technical\\pack-protocol.txt)\n> the negotiation for refs is discussed, but its unclear to me if the\n> negotiation explicitly navigates down into the trees and blobs of each\n> commit that need to go into the pack.\n>\n> From one perspective I can see that, in the main, it's only commit objects\n> that are being negotiated, and the DAG is used to imply which commit objects\n> are to be sent between the wants and haves end points, without need to\n> descend into their trees and blobs. The tags and the objects they point to\n> are explicitly given so are negotiated easily.\n>\n> The other view is that the negotiation should be listing every object of any\n> type between the wants and haves as part of the negotiation. I just couldn't\n> tell from the docs which assumption is appropriate. Is there any extra\n> clarifications on this?\n\nother object negotiation is inferred from commits because sending full\nlisting is too much. If you say you have commit A, you imply you have\neverything from commit A down to the bottom. With this knowledge, when\nyou want commit B, the sender only needs to send trees and objects\nthat do not exist in commit A or any of its predecessors. Although to\ncut cost at the sender, we do something less than optimized (check out\nthe edge concept in documents, or else in pack-objects.c). Pack\nbitmaps are supposed to provide cheap object traversal and make the\ntransfered pack even smaller.\n\n> I ask as I was cogitating on options for a 'narrow' clone  (to complement\n> shallow clones ;-) that could, say, in some way limit the size of blobs\n> downloaded, or the number of tree levels downloaded, or even path limiting.\n\nsize limiting is easy because you don't need to traverse object dag at\nall. Inside pack-objects it calls rev-list to collect objects to be\nsent. You just filter by size at that phase. Support for raising or\nlowering size limit is also workable, just like how shallow\ndeepen/shorten is done: you let the sender know you have size limit A,\nnow you want to raise to B and the sender just collects extra objects\nin A..B range for all \"have\" refs.\n\nThe problem is how to let the client know what objects are not sent\ndue to the size limit, so it could set up refs/replace to stop the\nuser from running into missing objects. If there are too many excluded\nobjects, sending all those SHA-1 with pkt-line is inefficient. (path\nlimit does not have problem, it can infer from the command line\narguments most of the time). Maybe you could send this listing in\nbinary format just before sending the pack.\n\nBTW another way to deal with large blobs in clone is git-annex. I was\nthinking the other day if we could sort of integrate it to git to\nprovide smooth UI (the user does not have to type \"git annex\nsomething\", or at least not often). Of course git-annex is still\noptional and the UI integration is only activated via config key,\nafter git-annex is installed.\n--\nDuy\n"},{"id":"231229","messageId":"4A4C52E5034347EF80B0D63DF71AE4C9@PhilipOakley","threadId":"35415","inReplyTo":"CACsJy8C5om91ObOHvSjDyDtHfCB9Y2gSbgF8Dsi2oXCbO55Czw@mail.gmail.com","subject":"Re: Pack transfer negotiation to tree and blob level?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2013-11-28T08:32:36Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Duy Nguyen\" <pclouds@gmail.com>\nSent: Wednesday, November 27, 2013 11:50 PM\n> On Thu, Nov 28, 2013 at 5:52 AM, Philip Oakley <philipoakley@iee.org> \n> wrote:\n>> In the pack transfer protocol \n>> (Documentation\\technical\\pack-protocol.txt)\n>> the negotiation for refs is discussed, but its unclear to me if the\n>> negotiation explicitly navigates down into the trees and blobs of \n>> each\n>> commit that need to go into the pack.\n>>\n>> From one perspective I can see that, in the main, it's only commit \n>> objects\n>> that are being negotiated, and the DAG is used to imply which commit \n>> objects\n>> are to be sent between the wants and haves end points, without need \n>> to\n>> descend into their trees and blobs. The tags and the objects they \n>> point to\n>> are explicitly given so are negotiated easily.\n>>\n>> The other view is that the negotiation should be listing every object \n>> of any\n>> type between the wants and haves as part of the negotiation. I just \n>> couldn't\n>> tell from the docs which assumption is appropriate. Is there any \n>> extra\n>> clarifications on this?\n>\n> other object negotiation is inferred from commits because sending full\n> listing is too much. If you say you have commit A, you imply you have\n> everything from commit A down to the bottom. With this knowledge, when\n> you want commit B, the sender only needs to send trees and objects\n> that do not exist in commit A or any of its predecessors.\n\nI presume that in the case of a tag that points to a tree, the pack will \ncontain not only the specific tree that was tagged, but also its \nsub-trees and blobs that it references.\n\nPlus I'm wondering if the commit(s) that contain that tagged tree are \nalso sent (given that such a tree could be repeated/reused in other \ncommits this may be expensive, and hence not done)\n\n> Although to\n> cut cost at the sender, we do something less than optimized (check out\n> the edge concept in documents, or else in pack-objects.c). Pack\n> bitmaps are supposed to provide cheap object traversal and make the\n> transfered pack even smaller.\n>\n>> I ask as I was cogitating on options for a 'narrow' clone  (to \n>> complement\n>> shallow clones ;-) that could, say, in some way limit the size of \n>> blobs\n>> downloaded, or the number of tree levels downloaded, or even path \n>> limiting.\n>\n> size limiting is easy because you don't need to traverse object dag at\n> all. Inside pack-objects it calls rev-list to collect objects to be\n> sent. You just filter by size at that phase. Support for raising or\n> lowering size limit is also workable, just like how shallow\n> deepen/shorten is done: you let the sender know you have size limit A,\n> now you want to raise to B and the sender just collects extra objects\n> in A..B range for all \"have\" refs.\n>\n> The problem is how to let the client know what objects are not sent\n> due to the size limit, so it could set up refs/replace to stop the\n> user from running into missing objects. If there are too many excluded\n> objects, sending all those SHA-1 with pkt-line is inefficient. (path\n> limit does not have problem, it can infer from the command line\n> arguments most of the time). Maybe you could send this listing in\n> binary format just before sending the pack.\n\nThis was part of the problem I was thinking about ;-)\n\n>\n> BTW another way to deal with large blobs in clone is git-annex. I was\n> thinking the other day if we could sort of integrate it to git to\n> provide smooth UI (the user does not have to type \"git annex\n> something\", or at least not often). Of course git-annex is still\n> optional and the UI integration is only activated via config key,\n> after git-annex is installed.\n\nI'll have a look.\n> --\n> Duy\n> --\n\nPhilip \n"}]}