{"thread":{"id":"38123","subject":"git bundle vs git rev-list","startedAt":"2014-12-05T22:36:18Z","lastAt":"2014-12-06T05:16:51Z","messageCount":6,"participants":["Jesse Hopkins","Junio C Hamano","brian m. carlson","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"253198","messageId":"CAL3By--xYnXFUdDP3MDxAxvfeBT3ArFrdAV=apzdWg6_kiD2Yg@mail.gmail.com","threadId":"38123","inReplyTo":null,"subject":"git bundle vs git rev-list","fromName":"Jesse Hopkins","fromEmail":"jesse.hops@gmail.com","sentAt":"2014-12-05T22:36:18Z","receivedAt":"2014-12-05T22:36:18Z","isPatch":false,"sender":{"key":"jesse.hops@gmail.com","avatar":null},"body":"Hello all –\n\n\n\nI am working to create a wrapper around git bundle to synchronize of\ngit repos via sneakernet from network ‘a’ to network ‘b’ transfer on a\nfairly frequent basis (daily to weekly). Network ‘b’ has a\ngatekeeper who is persnickety about what content might end up on his\nnetwork. The gatekeeper wants to know about the content being\ntransferred.\n\n\n\nI’ve come up with a scheme to list the final form of all files\nincluded in the bundle in whole or in part, see the psuedo code below:\n\n\n\n# BEGIN PSEUDOCODE\n\n#Create the bundle\ngit bundle create out.bundle --all \"--since=\"\n\n#Get list of commits\nincluded_commits = git rev-list --all \"--since=\"\n\n\n#For each commit, get the immediate parent(s), and find objects in its\nparents' tree that are not in its tree\nforeach commit in included_commits:\n #Get all blobs in this commit's tree, map blob to file name\n CommitBlobsMapToFilename = Process(git ls-tree -r commit)\n\n #Now find the parent commit(s)\n ParentCommits = git rev-list --parents -n 1 commit\n\n foreach parent in ParentCommits:\n #Get all blobs in the parent's tree\n ParentBlobsMapToFilename = Process(git\nls-tree -r parent)\n\n #Find blobs in this commit's tree that\nare not in the parent's commit tree\n NewBlobs =\nsetdiff(CommitBlobsMapToFilename , ParentBlobsMapToFilename);\n\n #Write each new blob contents to a unique filename\n foreach blob in NewBlobs\n filename =\nCommitBlobsMapToFilename(blob)\n filename = makeUnique(filename)\n git show blob > filename\n # END PSEUDOCODE\n\n\nThis scheme has worked well, but this is approach is predicated on the\nassumption that\n\ngit bundle create –all –since=\n\nuses the same commits that are returned by\n\ngit rev-list --all --since=\n\nHowever, I’ve noticed a scenario where that is not the case. I create\na bundle using --since=yesterday, where no activity has been made\nwithin the past few days. As expected, 'git rev-list --all\n--since=yesterday' returns 0 commits. However, the command 'git\nbundle create --all --since=yesterday' creates a bundle containing the\nfull history.\n\nTags seem to be the culprit, but I don’t know why. I do notice in the\noutput of git bundle that it mentions “skipping ref …” and “skipping\ntag …”, and sure enough all branches and most tags are shown as being\nskipped. However there are a few tags that are missing from that\nlist.\n\nIf I use --branches rather than --all as the limiter, then all is\nwell. In that case, git rev-list still returns 0 commits, and git\nbundle reports that it is refusing to make an empty bundle, as\nexpected.\n\nSo after all that, I have a two questions:\n\n1. Any thoughts on why a tag would be included by 'git bundle', when\n'git rev-list' with the same arguments returns empty?\n\n2. Is there a way to list commits contained in the bundle file itself?\n This seems like it would be more robust than trying to re-create the\ncommit list via 'git rev-list'.\n\nThanks,\n\nJesse\n"},{"id":"253204","messageId":"xmqqlhmlog4l.fsf@gitster.dls.corp.google.com","threadId":"38123","inReplyTo":"CAL3By--xYnXFUdDP3MDxAxvfeBT3ArFrdAV=apzdWg6_kiD2Yg@mail.gmail.com","subject":"Re: git bundle vs git rev-list","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-12-05T23:01:46Z","receivedAt":"2014-12-05T23:01:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jesse Hopkins writes:\n\n> 2. Is there a way to list commits contained in the bundle file itself?\n> This seems like it would be more robust than trying to re-create the\n> commit list via 'git rev-list'.\n\n\"git bundle list-heads o.bndl\" shows the positive endpoints, but\nthere is no corresponding \"git bundle list-prereq\" that shows the\nprerequisite commits.\n\nRunning \"git bundle verify o.bndl\" in an empty directory will show\nthe negative endpoints that are required to be in the receiving\nrepository in its error message, e.g.\n\n $ git bundle verify ~/w/git.git/o.bndle\n error: Repository lacks these prerequisite commits:\n error: bf404025edf1d7f5a69aa07cbaa88622e9d528df \n error: 15ab2081fff5b234ec5705a8645d39c1fdcf204c \n ...\n\nso collecting them would be one way to substitute \"list-prereq\".\n\nOnce you learned the positive and negative endpoints, running \"git\nrev-list --objects $positive_ones --not $negative_ones\" should list\nall the objects contained in the bundle.\n"},{"id":"253206","messageId":"20141205231350.GB223328@vauxhall.crustytoothpaste.net","threadId":"38123","inReplyTo":"CAL3By--xYnXFUdDP3MDxAxvfeBT3ArFrdAV=apzdWg6_kiD2Yg@mail.gmail.com","subject":"Re: git bundle vs git rev-list","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-05T23:13:50Z","receivedAt":"2014-12-05T23:13:50Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Fri, Dec 05, 2014 at 03:36:18PM -0700, Jesse Hopkins wrote:\n> 1. Any thoughts on why a tag would be included by 'git bundle', when\n> 'git rev-list' with the same arguments returns empty?\n\nI think the answer to this is found in the git rev-list manpage:\n\n List commits that are reachable by following the parent links from the\n given commit(s), but exclude commits that are reachable from the\n one(s) given with a ^ in front of them.\n\nThe operative word here is \"commits\". A bundle might include one or\nmore tag objects, or unannotated tags, even though no new commits were\navailable within the time frame.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"253210","messageId":"xmqq4mt9oecp.fsf@gitster.dls.corp.google.com","threadId":"38123","inReplyTo":"20141205231350.GB223328@vauxhall.crustytoothpaste.net","subject":"Re: git bundle vs git rev-list","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-12-05T23:40:06Z","receivedAt":"2014-12-05T23:40:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> On Fri, Dec 05, 2014 at 03:36:18PM -0700, Jesse Hopkins wrote:\n>> 1. Any thoughts on why a tag would be included by 'git bundle', when\n>> 'git rev-list' with the same arguments returns empty?\n>\n> I think the answer to this is found in the git rev-list manpage:\n>\n> List commits that are reachable by following the parent links from the\n> given commit(s), but exclude commits that are reachable from the\n> one(s) given with a ^ in front of them.\n>\n> The operative word here is \"commits\". A bundle might include one or\n> more tag objects, or unannotated tags, even though no new commits were\n> available within the time frame.\n\nIs this what a recent \"git bundle create\" change in 2.1.1 and 2.2\nfixed? The Release Notes to them seem to have this entry:\n\n * \"git bundle create\" with date-range specification were meant to\n exclude tags outside the range, but it did not work correctly.\n"},{"id":"253211","messageId":"20141205234201.GC223328@vauxhall.crustytoothpaste.net","threadId":"38123","inReplyTo":"xmqq4mt9oecp.fsf@gitster.dls.corp.google.com","subject":"Re: git bundle vs git rev-list","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-05T23:42:02Z","receivedAt":"2014-12-05T23:42:02Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Fri, Dec 05, 2014 at 03:40:06PM -0800, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> \n> > On Fri, Dec 05, 2014 at 03:36:18PM -0700, Jesse Hopkins wrote:\n> >> 1. Any thoughts on why a tag would be included by 'git bundle', when\n> >> 'git rev-list' with the same arguments returns empty?\n> >\n> > I think the answer to this is found in the git rev-list manpage:\n> >\n> > List commits that are reachable by following the parent links from the\n> > given commit(s), but exclude commits that are reachable from the\n> > one(s) given with a ^ in front of them.\n> >\n> > The operative word here is \"commits\". A bundle might include one or\n> > more tag objects, or unannotated tags, even though no new commits were\n> > available within the time frame.\n> \n> Is this what a recent \"git bundle create\" change in 2.1.1 and 2.2\n> fixed? The Release Notes to them seem to have this entry:\n> \n> * \"git bundle create\" with date-range specification were meant to\n> exclude tags outside the range, but it did not work correctly.\n\nThat certainly could be the case. I was thinking that perhaps someone\nhad created a tag recently, but your explanation is more likely.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"253229","messageId":"20141206051651.GC31301@peff.net","threadId":"38123","inReplyTo":"CAL3By--xYnXFUdDP3MDxAxvfeBT3ArFrdAV=apzdWg6_kiD2Yg@mail.gmail.com","subject":"Re: git bundle vs git rev-list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-12-06T05:16:51Z","receivedAt":"2014-12-06T05:16:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 05, 2014 at 03:36:18PM -0700, Jesse Hopkins wrote:\n\n> #Create the bundle\n> git bundle create out.bundle --all \"--since=\"\n\nOthers pointed out that a bug in the handling of --since may be the\nculprit here. However, I'd encourage you to use actual sha1s, as they\nare going to be more robust (especially in the face of any clock skew in\nthe commit timestamps).\n\nYou should be able to follow a procedure like:\n\n 1. On day 1, create a bundle from scratch:\n\n git bundle create out.bundle --all\n\n 2. Before you send it out, record its tips in the local repository\n for later reference:\n\n git fetch out.bundle +refs/*:refs/remotes/bundle/*\n\n 3. On day 2, create a bundle from the previously recorded tips:\n\n git bundle create out.bundle --all --not --remotes=bundle\n\n 4. Update your tips in the same way:\n\n git fetch out.bundle +refs/*:refs/remotes/bundle/*\n\nand so on for day 3 and onward.\n\nNote that this is not the only way to store those tips (I just did it\nusing git refs because it's simple to manipulate). You could also just\nstore it in a file:\n\n # checkpoint\n git ls-remote out.bundle | cut -f1 | sort -u >tips\n\n # make incremental bundle\n git bundle create out.bundle --all --not $(cat tips)\n\nThis also makes it easy to recover if the other side ever gets out of\nsync (say you create and checkpoint a bundle on the sending side, but it\nnever makes it to the remote; how do you know where to start from?). You\ncan always get the latest set of tips from the remote by running:\n\n git ls-remote . | cut -f1 | sort -u >tips\n\non it and then sneaker-netting the tips file back to the sender.\n\n-Peff\n"}]}