{"thread":{"id":"1221","subject":"GIT overlay repositories","startedAt":"2005-07-13T18:03:57Z","lastAt":"2005-07-13T22:35:35Z","messageCount":2,"participants":["James Ketrenos","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"6079","messageId":"42D5578D.3000301@linux.intel.com","threadId":"1221","inReplyTo":null,"subject":"GIT overlay repositories","fromName":"James Ketrenos","fromEmail":"jketreno@linux.intel.com","sentAt":"2005-07-13T18:03:57Z","receivedAt":"2005-07-13T18:03:57Z","isPatch":false,"sender":{"key":"jketreno@linux.intel.com","avatar":"https://gravatar.com/avatar/f4a487ad53c70943ed10ba220390cc7a004e4ba617c962bc4b7afcbd2bc08cc7?d=mp&s=160"},"body":"Ok...\n\nI started working on this a few weeks ago when it didn't appear \nanyone else had worked on support for this feature.  Since then, I\nhaven't kept pace with GIT development, so someone may have done \nsomething similar in the interim.  That said...\n\nStart with two repositories, let's call them Repo-A and Repo-B.  Repo-A\nis hosted on some server somewhere and contains lots of code \n(let's say its a kernel source repository).  Repo-B is only adding a \nsmall amount of changes to the repo (for argument sake, let's say the \nIPW2100 and IPW2200 projects) on top of what is already provided by Repo-A. \n\nFor several reasons, we would like users to be able to get just the \ndifferences between Repo-A and Repo-B from me.\n\nFor example, the user gets the full Repo-A:\n\n\trsync -avpr rsync://somewhere.com/repo-a/.git/objects/ \\ \n\t\t.git/objects/\n\trsync -avpr rsync://somewhere.com/repo-a/.git/refs/heads/master \\\n\t\t.git/refs/heads/parent\n\nand then overlays just the delta, which they obtain from me:\n\n\trsync -avpr rsync://myserver.com/repo-delta/.git/objects/ \\\n\t\t.git/objects/\n\trsync -avpr rsync://myserver.com/repo-delta/.git/refs/heads/master \\\n\t\t.git/refs/heads/master\n\nThe user can then run:\n\n\tln -s .git/HEAD refs/heads/master\n\tgit-read-tree -m HEAD\n\tgit-checkout-cache -a -q -f -u\n\nAnd now their disk has the entire Repo-A + Repo-B object \ntree as if Repo-B had been entirely stored on myserver.com.\n\nAdding this recursively is pretty straightforward if we introduce an \n'ancestors' file stored on the overlay repository that contains the URL \nand SHA of the tip of the parent when it was used to build the full \nrepository.  For example:\n\n% cat .git/refs/ancestors (edited to fit 80 columns)\nrsync://rsync.kernel.org/...-2.6.git/#ieee80211\t0c16...7f3\nrsync://bughost.org/repos/ieee80211-delta/.git/ 5cb5...b84\nrsync://bughost.org/repos/ipw-delta/.git/       f3ed...abe\n\nA simple script (attached as 'git-grab-overlay') can then walk this \nancestor file and pull down updates as needed from each of the parent \nrepositories.\n\nThe process of creating the overlays is pretty straight forward (for \nsimple cases anyway)\n\nFor example:\n\n\tA1   B  <-- HEAD SHA1\n\t|    |\n\t|    |\n\t| .->'  <-- parent SHA1\n\t|/     \n\t|\n\tA\n\nIn this state, the list of objects is easily obtained by walking the \ntree from HEAD back to the parent's SHA1 at the time the branch was \nmade.  This can be done with something like: \n\n\tgit-rev-list --objects HEAD ^parent\n\nTo simplify things, I created the attached script 'git-make-overlay' \nwhich will create an overlay repository with for all objects needed \nbetween a given SHA1 and the HEAD of the repository:\n\n\tcd repository\n\tgit-make-overlay ../target\n\nThis script will take the current HEAD and cached parent SHA (stored via \nthe git-grab-overlay or manually when you create the intial tree) and \ncreate an overlay that you can then store on a remote server for others \nto pull.\n\nObviously you don't want the overlay repository to remain out of sync \nwith the parent repository for too long.  The process of updating the \nfull local repository is then a matter of:\n\n  * Grabbing the latest parent objects\n  * Caching the parent's latest master (store as new-parent)\n  * Perform a merge of the child back into the master tree\n  * Updating the cached heads (parent, master, etc.)\n\nThe merge is accomplished via:\n\n\tchild=$(cat .git/HEAD)\n\tparent=$(cat .git/refs/heads/new-parent)\n\tbase=$(git-merge-base $child $parent)\n\t# NOTE: base should be the same as .git/refs/heads/parent!\n\nIs it best to merge the smaller subset back into the merge base, ie:\n\n\tgit-read-tree -m $base $parent $child\n\nIn either case, the merge will fail if you have your local tree in a \nstate consistent with child (vs. base).  So before you do the merge you \nneed to either nuke the local tree or use paramters to the git- tools \nthat I don't know about (definitely wouldn't suprise me).  When I run it, \nI create a temporary directory with a symlink of .git to the .git directory\nwhere the merge will occur.  After the merge is done, I go back to the main\ndirectory and resync the local file system with the GIT repository.\n\nAnyway, I put together another simple script that attempts to do the \nabove--minus the symlink hack (attached as git-merge-parent).  It invokes \nthe git-merge-heads script which does some of what Cogito's cg-merge does, \nonly with support for automatically launching an external merge conflict \nresolution program.  It relies on cogito's cg-Xmergefile so you need to \nhave cogito to use git-merge-parent and git-merge-heads.\n\t\nFrequently when remerging with a parent repository--especially one that \nmay contain a version of the child within it, there will be merge \nconflicts.  The git-merge-parent accepts (well, requries) a MERGE \nenvironment variable to invoke as a merge conflict resolver.  If a \nconflict arrises, it will prompt if you want to resolve the merge, \nundo the update, or exit and allow you to finish the process manually.\n\nIf you chose to run the conflict editor, it will invoke it by providing \nthree versions of the conflicting file:\n\n1.  The base version (as captured in the tree identified \n    with git-merge-base on the parent and child) -- named FILE.base\n2.  The parent version named FILE.parent\n3.  The child version named FILE\n\nAfter all merge conflicts have been manually updated, the \ngit-merge-parent script will commit those changes, along with the rest \nof the merge, and move the child's HEAD to the new tree identity.  In \naddition it will cache the HEAD of the parent's merge.\n\nI haven't really tested the whole merge extensiveloy; it has worked \n(more or less) with some simple parent updates I've done here.  However, \nit may be of use to some folks out there, and if others can take it \nforward beyond what I have, or merge the functionality into cogito (or \nwherever) great. \n\nI have a tree currently up that demonstrates the above in use.  \nYou can access it via:\n\n  git-grab-overlay rsync://bughost.org/repos/ipw-delta/.git/ ipw-dev/\n\nJames\n\n\n\n\n\n#!/usr/bin/env bash\n# Copyright (c) 2005, Intel Corporation, James Ketrenos\n\nfunction die()\n{\n\tret=$1\n\tshift\n\techo $@ >&2\n\texit $ret\n}\n\nfunction pull-delta-repo()\n{\n\tparent=$1\n\n\tif echo \"$parent\" | egrep -q \"#\"; then\n\t\tbranch=$(echo $parent | sed -ne \"s,.*#\\(.*\\),\\1,p\")\n\t\tparent=$(echo $parent | sed -ne \"s,\\([^#]*\\).*,\\1,\")\n\telse\n\t\tbranch=master\n\tfi\n\techo \"$parent\" | grep -q \"/\\$\" || parent=\"$parent/\"\n\n\tmkdir -p ${dst}{refs/{heads,branches},objects}\n\n\techo \"Grabbing repo:\"\n\techo -en \"\\tparent...\"\n\trsync ${RSYNC_FLAGS} -ap ${parent}refs/heads/${branch} \\\n\t\t${2}refs/heads/parent || \n\t\tdie 5 \"Could not pull $parent $branch\"\n\techo \"done.\"\n\n\techo -en \"\\tancestors...\"\n\trsync ${RSYNC_FLAGS} -ap ${parent}refs/ancestors \\\n\t\t${2}refs/ancestors || \n\t\tdie 5 \"Could not pull $parent $branch\"\n\techo \"done.\"\n\n\tcat ${2}refs/ancestors |\n\twhile read url sha; do\n\t\techo $url | grep \"^#.*\" && continue\n\t\turl=$(echo $url | sed -e \"s,\\([^#]*\\).*,\\1,\")\n\t\techo $url | grep -q \"/\\$\" || url=\"$url/\"\n\t\tdir=$(echo $sha | sed -ne \"s#^\\(..\\).*#\\1#p\")\n\t\tobj=$(echo $sha | sed -ne \"s#^..\\(.*\\)#\\1#p\")\n\t\techo -e \"Checking for updates on:\\n\\t$url\\nfor\\n\\t$sha...\"\n\t\tif [ ! -e ${2}objects/$dir/$obj ]; then\n\t\t\techo \"Updating objects from $url...\"\n\t\t\trsync ${RSYNC_FLAGS} -avpr \\\n\t\t\t\t${url}objects/ \\\n\t\t\t\t${2}objects/ ||\n\t\t\t\tdie 6 \"Could not pull objects from $url.\"\n\t\t\techo \"Updated.\"\n\t\telse\n\t\t\techo \"$url up to date.\"\n\t\tfi\n\tdone || exit 1\n\n\turl=$parent\n\tsha=$(cat ${2}/refs/heads/parent)\n\tdir=$(echo $sha | sed -ne \"s#^\\(..\\).*#\\1#p\")\n\tobj=$(echo $sha | sed -ne \"s#^..\\(.*\\)#\\1#p\")\n\tif [ ! -e ${2}objects/$dir/$obj ]; then\n\t\techo \"Updating objects from $url...\"\n\t\trsync ${RSYNC_FLAGS} -avpr \\\n\t\t\t${url}objects/ \\\n\t\t\t${2}objects/ ||\n\t\tdie 6 \"Coult not pull objects from $url.\"\n\telse\n\t\techo \"$url up to date.\"\n\tfi\n\n\techo -e \"$parent\\t$sha\" >> ${2}refs/ancestors\n\tcp ${2}refs/heads/parent ${2}refs/heads/master\n\tln -sf refs/heads/master ${2}HEAD\n#\trsync ${RSYNC_FLAGS} -ap ${1}refs/branches ${2}refs/branches/\n}\n\n([ ! \"$1\" ] || [ ! \"$2\" ]) && echo \\\n\"usage: git-grab-overlay src_delta_repo target_full_repo\" && exit 1\n\nsrc=\"${1}\"\necho ${src} | grep -q \"/\\$\" || src=\"${src}/\"\necho ${src} | grep -q \"\\.git/\\$\" || src=\"${src}.git/\"\n\ndst=\"${2}\"\necho ${dst} | grep -q \"/\\$\" || dst=\"${dst}/\"\necho ${dst} | grep -q \"\\.git/\\$\" || dst=\"${dst}.git/\"\n[ ! -d ${dst} ] && (mkdir -p ${dst} || die 5 \"Error creating '${dst}'\")\ndst=`readlink -f ${dst}`/\n\npull-delta-repo $src $dst || exit 1\n\necho \"\"\necho \"${2} now contains the full tree from ${1}.\"\necho -e \"\\nTo sync the files to the cache state, run:\"\necho -e \"\\t% cd ${2}\"\necho -e \"\\t% git-read-tree -m HEAD && git-checkout-cache -q -f -u -a\"\nwhile true; do\n\tread -p \"Do you want to do this now? [Y/n] :\" reply\n\tcase $reply in\n\t\"n\")\texit 0\n\t\t;;\n\t\"Y\"|\"y\"|\"\")\n\t\tdir=$PWD\n\t\tcd ${2}\n\t\tgit-read-tree -m HEAD\n\t\tgit-checkout-cache -q -f -u -a\n\t\tcd $PWD\n\t\texit 0\n\tesac\ndone\n\n\n#!/usr/bin/env bash\n# Copyright (c) 2005, Intel Corporation, James Ketrenos\n\nfunction die()\n{\n\tret=$1\n\tshift\n\techo -e $@>&2\n\texit $ret\n}\n\n\nfunction copy-object()\n{\n        dir=$(echo $1 | sed -e 's#\\(..\\).*#\\1/#')\n        obj=$(echo $1 | sed -e 's#..\\(.*\\)#\\1#')\n\n        [ ! -e ${2}objects/${dir}${obj} ] && \\\n                return 1\n\n        if [ -e ${3}objects/${dir}${obj} ]; then\n\t\t[ \"$warn\" ] && echo \"$1 already exists in $3.\"\n                return 0\n\tfi\n\n        [ ! -d ${3}objects/${dir} ] && \\\n                (mkdir -p ${3}objects/${dir} || return 1)\n\n        cp -a ${2}objects/${dir}${obj} ${3}objects/${dir} || return 1\n}\n\n\n[ \"$1\" ] ||\n\tdie 1 \\\n\"usage: git-make-overlay dst-repo [SHA1]\\n\\n\"\\\n\"Where dst-repo is the target directory to store the overlay\\n\"\\\n\"repository and SHA1 is the 'base' of the tree you want to create the\\n\"\\\n\"delta against.\\n\"\\\n\"\\n\"\\\n\"If not provided, the script will compute the merge base between\\n\"\\\n\"the parent (.git/refs/heads/parent) and the current HEAD (which would\\n\"\\\n\"typically be the same).\\n\"\\\n\"\\n\"\\\n\"For example:\\n\"\\\n\"  % child=\\$(cat .git/HEAD)\\n\"\\\n\"  % parent=\\$(cat .git/refs/heads/parent)\\n\"\\\n\"  % git-make-overlay ../overlay \\$(git-merge-base \\$child \\$parent)\\n\"\\\n\"\\n\"\\\n\"The resulting overlay will be populated with the following files:\\n\"\\\n\"\\n\"\\\n\"  .git/objects\\t\\tContaining all objects not contained in parent.\\n\"\\\n\"  .git/refs/ancestors\\tParental lineage to root repository.\\n\"\\\n\"  .git/refs/heads/master\\tSHA for the tip of this overlay.\\n\"\\\n\"\\n\"\n\n[ -d \".git\" ] ||\n\tdie 2 \".git repository not found.\"\n\ndst=$1\necho $dst | grep -ne \"/\\$\" || dst=\"$1/\"\necho $dst | grep -ne \"\\.git/\\$\" || dst=\"${dst}.git/\"\n[ ! -d $dst ] && (mkdir -p $dst || die 3 \"mkdir: error creating '$dst'\")\nshift\n\nif [ \"$1\" ]; then \n\tbase=\"$1\"\nelse \n\tbase=\"$(git-merge-base $(cat .git/HEAD) $(cat .git/refs/heads/parent))\"\nfi\n\nhead=$(cat .git/HEAD)\n\necho -e \\\n\"Creating overlay from:\\n\"\\\n\"\\t$base ->\\n\"\\\n\"\\t\\t$head (HEAD)\\nin\\n\"\\\n\"\\t$dst\\n\"\n\n(git-rev-list --objects $head ^$base ||\n                die 2 \"git-rev-list failed!\") |\nwhile read sha rest; do\n\tcopy-object $sha .git/ $dst || \n\t\tdie 5 \"copy-object failed on $sha!\"\n\ttype=$(git-cat-file -t $sha)\n\techo -e \"$sha\\t$type\\t$rest\"\ndone\n\nmkdir -p ${dst}refs/heads || die 5 \"Error creating refs/heads\"\ncat .git/HEAD > ${dst}refs/heads/master\nln -sf refs/heads/master ${dst}HEAD\ncp .git/refs/ancestors ${dst}refs/ancestors\n\necho \"done.\"\n\n\n#!/usr/bin/env bash\n#\n# Portions based on cg-merge Copyright (c) Petr Baudis, 2005\n# Copyright (c) 2005, Intel Corporation, James Ketrenos\n\n. ${COGITO_LIB:-/usr/lib/cogito/}cg-Xlib\n\nfunction die()\n{\n\tret=$1\n\tshift\n\techo $@ >&2\n\texit $ret\n}\n\n([ \"$1\" ] && [ \"$2\" ]) || \n\tdie 1 \"usage: git-merge-heads parent-sha1 child-sha1\"\n\ngit-read-tree HEAD ||\n\tdie 8 \"Failed to read tree\"\n\nparent=$1\nchild=$2\nbase=$(git-merge-base $1 $2) || \n\tdie 2 \"Failed to obtain common base for trees\"\n\ngit-read-tree -m $base $parent $child ||\n\tdie 3 \"Failed to merge trees\"\n\nif ! git-merge-cache -o ${COGITO_LIB:-/usr/lib/cogito/}cg-Xmergefile -a; then\n\tgit-resolve-conflicts ||\n\t\tdie 4 \"Failed to resolve conflicts\"\nfi\n\ntree=$(git-write-tree) || \n\tdie 5 \"Failed to write merged tree.\"\n\necho \"Merge succeeded as tree $tree.\"\n\nwhile true; do\n\tread -p \"Do you wish to commit this tree? [y]n :\" reply <&1\n\tcase $reply in\n\tn)\tbreak\n\t\t;;\n\ty|*)\n\t\techo \"Type your COMMIT message followed by CTRL-D:\"\n\t\tcommit=$(git-commit-tree $tree -p $parent -p $child) ||\n\t\t\tdie 6 \"Commit failed.\"\n\t\techo \"Commit succeeded as commit $commit.\"\n\t\techo \"Moving HEAD to $commit\"\n\t\techo $commit > .git/HEAD\n\t\tbreak\n\t\t;;\n\tesac\ndone\n\ngit-read-tree -m HEAD\n\nwhile true; do\n\tread -p \"Do you wish to update the files on disk to reflect repository? [y]n :\" reply <&1\n\tcase $reply in\n\tn)\tbreak\n\t\t;;\n\ty|*)\n\t\tgit-read-tree -m HEAD ||\n\t\t\tdie 9 \"Failed to read tree.\"\n\n\t\tgit-checkout-cache -q -u -f -a ||\n\t\t\tdie 7 \"Checkout failed.\"\n\n\t\tbreak\n\t\t;;\n\tesac\ndone\n\n\n\n#!/usr/bin/env bash\n# Copyright (c) 2005, Intel Corporation, James Ketrenos\n\nfunction die()\n{\n\tret=$1\n\tshift\n\techo $@ >&2\n\texit $ret\n}\n\n[ ! \"$MERGE\" ] && \n\tdie 1 \"MERGE must be set to merge program (ie, export MERGE=kdiff3)\"\n\nparent=\nbranch=\nfunction get_parent_and_branch()\n{\n\t[ ! -e .git/refs/parent ] && return 1\n\tparent=$(cat .git/refs/parent)\n\t[ ! \"$parent\" ] && return 1\n\n\tif echo ${parent} | egrep -q \"#\"; then\n\t\tbranch=\"refs/heads/\"$(echo $parent | sed -ne \"s,.*#\\(.*\\),\\1,p\")\n\t\tparent=$(echo $parent | sed -e \"s,\\([^#]*\\).*,\\1,\")\n\telse\n\t\tbranch=\"HEAD\"\n\tfi\n\n\techo $parent | grep \"/\\$\" || parent=\"${parent}/\"\n\n\treturn 0\n}\n\nfunction merge_parent()\n{\n\t[ -e \".git/refs/heads/repository.merge\" ] && \n\t\tdie 9 \"Prior merge failed and not un-done.\"\n\n\trepo=$1\n\tbranch=$2\n\techo $repo | grep \"/\\$\" || repo=\"${repo}/\"\n#\tFirst verify there are no uncommitted changes, then proceed...\n#\t(TODO: Add script commands for checking for uncommitted changes)\n#\n#\tObtain the parent's HEAD and store as .git/refs/heads/parent.tip and\n#\t\t.git/refs/heads/parent.merge\n\techo \"Checking for updates from ${repo}${branch}...\"\n\n\trsync ${RSYNC_FLAGS} -vpL ${repo}${branch} .git/refs/heads/parent.tip || return 1\n\n\tparent=$(cat .git/refs/heads/parent)\n\tnew_parent=$(cat .git/refs/heads/parent.tip)\n\techo $new_parent > .git/refs/heads/parent.merge || return 2\n\n#\tCheck for an update; if no update, we're done\n\t[ \"$parent\" == \"$new_parent\" ] && return 0\n\n\tchild=$(cat .git/HEAD)\n\n#\tGrab the latest objects from the parent\n\techo \"Pulling object files\"\n\trsync ${RSYNC_FLAGS} -avpr ${repo}objects/ .git/objects/ || return 3\n\n#\tCopy the current HEAD to .git/refs/heads/repository.merge\n\techo \"Caching HEAD in repository.merge\"\n\tcp .git/HEAD .git/refs/heads/repository.merge || return 4\n\n\n#\tUpdate the GIT index\n\techo \"Setting local repository to mirror parent\"\n\techo $new_parent > .git/HEAD || return 5\n\trm .git/index\n\tgit-read-tree -m HEAD || return 6\n#\tgit-checkout-cache -f -u -a\n\n# \tAt this point, our local repository is now a mirror of the \n#\tparent repository.  \n\n#\tMerge and commit the repository changes back into the current parent\n\techo \"Merging child back into parent\"\n\tgit-merge-heads $new_parent $child || return 7\n\n#\tUpdate parent / repo index links\n\tmv -f .git/refs/heads/parent.tip .git/refs/heads/parent || return 12\n\n\tmv .git/refs/heads/repository.merge .git/refs/heads/pre-merge\n\t\n\techo \"Updating ancestors file...\"\n\turl=$(cat .git/refs/parent)\n\tsed -i -e \"s#$url\\t$parent#$url\\t$new_parent#g\" \\\n\t\t.git/refs/ancestors\n\treturn 0\n}\n\nfunction merge_cleanup()\n{\n#\tIf any of the above fails, the repository should be changed back \n#\tto its original state:\n\t\n\trm -f .git/refs/heads/{parent,repository}.merge \\\n\t\t.git/refs/heads/parent.tip || \\\n\t\treturn 1\n\tln -sf refs/heads/master .git/HEAD || return 2\n\tgit-read-tree -m HEAD || return 3\n\tgit-checkout-files -q -f -u -a || return 4\n\n\treturn 0\n}\n\nif ! get_parent_and_branch; then\n#\tmerge_cleanup\n\texit 1\nfi\n\necho -e \"Attempting merging with:\\n\\t$parent\\n...\"\nmerge_parent $parent $branch || \n\tdie 12 \"Error merging parent.\"\n\nrm -rf .git/merge*\n\nexit 0\n"},{"id":"6126","messageId":"7vmzoqqqq0.fsf@assigned-by-dhcp.cox.net","threadId":"1221","inReplyTo":"42D5578D.3000301@linux.intel.com","subject":"Re: GIT overlay repositories","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-07-13T22:35:35Z","receivedAt":"2005-07-13T22:35:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"James Ketrenos <jketreno@linux.intel.com> writes:\n\n> Start with two repositories, let's call them Repo-A and Repo-B.  Repo-A\n> is hosted on some server somewhere and contains lots of code \n> (let's say its a kernel source repository).  Repo-B is only adding a \n> small amount of changes to the repo (for argument sake, let's say the \n> IPW2100 and IPW2200 projects) on top of what is already provided by Repo-A. \n>\n> For several reasons, we would like users to be able to get just the \n> differences between Repo-A and Repo-B from me.\n\nI have done something like this in late April - early May when I\nran git-jc repository.  I took the then-current Linus head from\ngit.git, placed only my commits and objects that are absent from\nhis tree to a public place, and asked pullers to pull from Linus\nfirst and then from me to get a usable repository.  My\nunderstanding is that you are formalizing and automating the\npart \"please pull from X, Y, Z and then from me to complete what\nI have, making it usable\".  If that is the case, I agree with\nthe intentions [*1*].\n\nThis is related to the reason Linus wanted to have .git/config,\nsomething that records \"this object database depends on these\nother object databases to be complete\", when we talked about\nALTERNATE_OBJECT_DIRECTORIES last time.  I am wondering if there\nis a way to solve these two related problems in a unified way.\n\nOne minor problem is that ALTERNATE_OBJECT_DIRECTORIES is a\nlocal thing and we do not want to be able to express URLs in\nthere, because we do not want to run rsync nor curl from inside\nsha1_file.c.  The \"partial repository\" problem, on the other\nhand, is to publish such a repository and you _do_ want to have\nURLs, likely to be somewhere completely different from where\nsuch a partial repository is hosted at, reachable from your\npullers.\n\nRegardless of whatever we end up doing, I have one proposal to\nmake.  How about having .git/objects/info/ directory for housing\nvarious \"object database\" specific (and not repository specific)\ninformation?\n\nThe set of files I would see immediate benefits are:\n\n    objects/info/ext  -- this is your .git/refs/ancestors file [*2*],\n                         that lists external URLs that the objects\n                         in this object database depends upon,\n                         along with the set of head commits\n                         there to start pulling from to complete\n                         this partial object database [*3*].  This\n                         _should_ name URLs accessible to\n                         expected pullers from this repository.\n\n    objects/info/alt  -- list of local alternate object\n                         directories; probably we should\n                         deprecate ALTERNATE_OBJECT_DIRECTORIES\n                         environment variable with this, and\n                         rewrite parts of sha1_file.c.  I'd\n                         volunteer to do it if we have\n                         consensus.\n\n    objects/info/pack -- list of pack files in objects/pack/;\n                         this would be useful for discovery\n                         through really dumb web servers [*4*].\n\nUsing something like the above structure, pulling from this\n\"partial\" repository at rsync://abc.xz/x.git would go this way:\n\n   (1) Sync from rsync://abc.xz/x.git/objects/\n      \n   (2) Read objects/info/ext just slurped from there.  Run the\n       procedure (1) thru (3) against the URLs listed in the\n       file, recursively.\n\n   (3) [*5*] Read objects/info/alt just slurped from there.  Say\n       it contained ../a.git and ../b.git.  Run the procedure\n       (1) thru (3) against rsync://abc.xz/a.git/objects/ and\n       rsync://abc.xz/b.git/objects/ recursively.\n\n   (4) Sync from rsync://abc.xz/x.git/refs/ as needed.\n\nNon-rsync transfer can and should be done the same way.  In\neither case, updating the puller's refs/ is done solely based on\nthe information from rsync://abc.xz/x.git/refs/ and not from\nrefs in the depended-upon repositories.\n\nAm I basically on the same page as you are?\n\n\n[Footnote]\n\n*1* But you are conflating it something else.  I will not\ncomment on the part you talk about merges in this message,\nbecause forward-porting your own changes to updated upstream is\northogonal to the \"partial object database\" issue.  It needs be\ntaken care of independently even if you maintain a fully\npopulated object database for your development.\n\n*2* Ancestry is an overused word, so I would propose to call\nthis \"external dependency\": your partial object database depends\non them to be complete and usable.\n\n*3* I was wondering why you wanted to record the foreign head in\naddition to the URLs first, but you need that information (and\nprobabaly that needs to be a set of heads, not just a single\nhead) because their head may move, and in the worst case may\neven be rewound to something that is not a descendant of what\nyou depend upon.\n\n*4* This is not related to the current topic.\n\n*5* This part is optional, because some alternates used locally\nat the partial repository site may not be exposed to the same\npullers, or even when exposed, the alternate site may be behind\na slow link and it is preferable to get the same information\nfrom somewhere else listed in info/ext file.  On the other hand,\nit is simpler to just maintain info/alt file to serve for both\nlocal and remote purposes.  I don't offhand know the tradeoffs.\n \n"}]}