{"thread":{"id":"30503","subject":"merge unpack-objects to index-pack?","startedAt":"2012-05-11T12:02:14Z","lastAt":"2012-05-11T17:51:33Z","messageCount":2,"participants":["Nguyen Thai Ngoc Duy","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"191386","messageId":"CACsJy8BjA-L=jOZ7G9TbDY6D0rqhuzCCRwAjazEfBx_=HcqcqA@mail.gmail.com","threadId":"30503","inReplyTo":null,"subject":"merge unpack-objects to index-pack?","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-05-11T12:02:14Z","receivedAt":"2012-05-11T12:02:14Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Hi,\n\nI'm looking at adding large file support to unpack-objects. A simple\nway is just to stream large blobs into loose large blobs. But I'd\nrather keep those blobs in pack because pack-objects is happier that\nway. I'm looking at unpack-objects and thinking maybe it's best to\njust merge it back to index-pack.\n\nIn normal mode (all small objects), index-pack receives the pack\nstream. Objects will be unpacked in phase two, resolving objects in\nindex-pack. The only downside I can see is the new unpack-objects now\nwrite temporary pack on disk, which does not sound too bad to me.\nunpack-packs is called on small packs so extra space is small. For\nsingle-huge-blob packs, it's good to keep them on disk anyway. When\nthe pack has large blobs, we could either just keep full pack.\n\nAfter this, the only pack receiver at client side is index-pack.\nfetch-pack does not have to choose between unpack-objects and\nindex-pack, just pass --unpack-limit <n> to index-pack.\n\nWhat do you think?\n-- \nDuy\n"},{"id":"191405","messageId":"7vsjf6li6y.fsf@alter.siamese.dyndns.org","threadId":"30503","inReplyTo":"CACsJy8BjA-L=jOZ7G9TbDY6D0rqhuzCCRwAjazEfBx_=HcqcqA@mail.gmail.com","subject":"Re: merge unpack-objects to index-pack?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-05-11T17:51:33Z","receivedAt":"2012-05-11T17:51:33Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n\n> I'm looking at adding large file support to unpack-objects. A simple\n> way is just to stream large blobs into loose large blobs. But I'd\n> rather keep those blobs in pack because pack-objects is happier that\n> way. I'm looking at unpack-objects and thinking maybe it's best to\n> just merge it back to index-pack.\n>\n> In normal mode (all small objects), index-pack receives the pack\n> stream. Objects will be unpacked in phase two, resolving objects in\n> index-pack. The only downside I can see is the new unpack-objects now\n> write temporary pack on disk, which does not sound too bad to me.\n> unpack-packs is called on small packs so extra space is small. For\n> single-huge-blob packs, it's good to keep them on disk anyway. When\n> the pack has large blobs, we could either just keep full pack.\n>\n> After this, the only pack receiver at client side is index-pack.\n> fetch-pack does not have to choose between unpack-objects and\n> index-pack, just pass --unpack-limit <n> to index-pack.\n>\n> What do you think?\n\nI think it is beneficial to step back a bit.  What is the _real_ reason\nwhy we call unpack-objects instead of index-pack when we receive only a\nhandful of objects?\n\nI think we did this to avoid littering the receiving repository with too\nmany packs from individual transfers.  As long as that goal is met, a\nsolution that replaces the current \"if we are going to get less than N\nobjects, explode them to loose objects\" does not actually have to explode\nthem to loose objects.  It could explode normal objects into loose ones,\nwhile appending large ones into an existing pack (and it has to fix up the\npack .idx after doing so), for example.  Or it could even choose to\n_always_ append into an existing pack designated for appending new\nobjects.  Or it could punt the \"appending\" part, declaring that large\nobject problem is a rare event, and create/leave a new pack in the\nrepository that stores a large object (this however would not satisfy \"do\nnot litter the receiving repository with too many packs\" goal if \"large\nobject problem\" is not rare enough).\n\nAnd the first step to make that happen would be to let a single receiver\nprogram, instead of receive-pack/fetch-pack, make the decision.  That\nreceiver program _might_ benefit from knowing how many objects it is going\nto receive when making the decision before seeing a single byte from the\npackstream, but there are other more meaningful data you can learn only\nafter looking at what is in the pack.\n\nSo I like the general direction you are heading.\n\nProbably the first step in the right structure of such a series would\nintroduce a new helper program that builtin/receive-pack.c::unpack() and\nbuiltin/fetch-pack.c::get_pack() call, remove the header-peeking these\ncalling processes currently do, and make that new helper responsible for\nswitching between unpack/index-pack (the new helper may peek the header\ninstead).  The first implementation of the new helper may decide exactly\nlike how these two functions choose between the two.\n\nOnce that is done, it will be an implementation detail of how objects in\nthe incoming packstream is stored locally from the point of view of\nfetch-pack and receive-pack, and nobody should notice when the new helper\nis updated to call only the updated index-pack that knows to stream large\n(or all) objects into a pack.\n"}]}