{"thread":{"id":"2922","subject":"GIT 1.0.4 and what is in master and proposed updates.","startedAt":"2005-12-24T09:42:11Z","lastAt":"2005-12-24T10:03:32Z","messageCount":2,"participants":["Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"13997","messageId":"7vd5jmvnkc.fsf@assigned-by-dhcp.cox.net","threadId":"2922","inReplyTo":null,"subject":"GIT 1.0.4 and what is in master and proposed updates.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-12-24T09:42:11Z","receivedAt":"2005-12-24T09:42:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"GIT 1.0.4 contains the following updates since GIT 1.0.3:\n\n * check_packed_git_idx(): check integrity of the idx file itself.\n\n   Earlier we did not use SHA1 checksum at the end of the index\n   file that records the checksum of the index file itself.\n\n * show-branch: usability updates.\n\n   Fixes the bug that made 'git show-branch --heads' show the\n   same heads twice, among other things.\n\n * merge --no-commit: tweak message.\n\n   Instead of saying \"failed/prevented\", say which one.\n\n * ls-files --full-name: usage string and documentation.\n\n   The option was not documented.\n\n * mailinfo: iconv does not like \"latin-1\" -- should spell it \"latin1\"\n\n   Noted by Marco; when a message does not identify its own\n   charset, we tried to assume latin1 but misspelled it as\n   latin-1, and iconv barfed.\n\n\nThe master branch has all of the above, and in addition contains\nthe following:\n\n * git-clone: do not special case dumb http.\n\n   This is to remove the remnant from the days http-fetch did\n   not handle packed repositories natively.\n\n * rev-parse: --show-cdup\n\n   This is a low-impact shorthand for:\n\n        git rev-parse --show-prefix | sed -e 's|[^/][^/]*|..|g'\n\n   and will be used by \"git checkout works from a subdirectory\"\n   patch.\n\n\nThe proposed updates branch has the following updates.\n\n * git-daemon --base-path\n\n   Lack of the command line verification bugs me somewhat with\n   this patch.  Also we might want to apply the base-path to the\n   whitelist as well, which the patch currently does not do.\n   Almost there, but not quite ready.\n\n * help with group permission in a shared repository.\n\n   This series consists of these commits:\n\n      Introduce core.sharedrepository\n      git-init-db: initialize shared repositories with --shared\n      Tutorial: mention shared repository management.\n\n   I think this one is ready.  I just am taking things slowly.\n\n * make git-checkout work from a subdirectory.\n\n   This series consists of these commits:\n\n      checkout: sometimes work from a subdirectory.\n      ls-tree: chomp leading directories when run from a subdirectory\n      Adjust to ls-tree --full-name when run from a subdirectory.\n\n   This seems to work pretty well.  I just am taking things\n   slowly.\n\n * require packfiles to follow the naming convention more strictly.\n\n   This rejects a packfile whose name does not begin with\n   \"pack-\" followed by 40-byte hexadecimal digits.  The core\n   never had such limitation (the only thing it required was\n   that the pack and the idx are named with the same basename\n   with .pack and .idx extention), but the way http-fetch\n   handles packs requires this.\n\n   An alternative strategy to handle the current mess would be\n   to fix http-fetch so that it does not assume that pack names\n   are always in the format \"pack-[0-9a-f]{40}.pack\", getting\n   rid of sha1[] column from struct packed_git.  This is\n   probably more work, but would be cleaner.  On the other hand,\n   the cleanliness probably would not buy us much.  We do not\n   need to insist on \"pack-\" prefix, but there is no strong\n   reason to hate it, and we need to ensure packnames do not\n   collide with each other anyway, so having SHA1 hash in the\n   name makes sense.  However, we currently allow arbitrary 40\n   hexadecimal digits there and the user is free to say 0{40}.\n   Continuing to allow this defeats the initial purpose of why\n   40 hexadecimal digits are there: collision avoidance.\n\n   This series starts warning if the 40-byte hex that follows\n   \"pack-\" is not the correct hash for contained object names\n   that was given by pack-objects when the pack was created, and\n   has a provision to later start rejecting such packs.\n\n   This is probably controversial because it would eventually\n   require users to rename their packs created using pre v0.99.9\n   tools.  There is a small utility program that helps the\n   conversion process, so it may not be that painful.\n\n * Disable USE_SYMLINK_HEAD by default.\n\n   This uses symref for .git/HEAD everywhere.  Some Porcelain\n   may not be ready to deal with it, so it is kept in the\n   proposed updates branch for now.\n\n   If your Porcelain \"cat .git/HEAD\", or \"readlink .git/HEAD\",\n   it is already broken in repositories that use symref for\n   HEAD, but it will be broken everywhere by this.  Use\n   git-symbolic-ref if you want to find out the name of the\n   branch HEAD points at, or git-rev-parse if you want to find\n   out the commit object name HEAD points at.\n\n   Most notably, git_read_hash(\".../HEAD\") in gitweb is probably\n   affected (meaning, it is already broken).\n"},{"id":"14001","messageId":"7vfyoiu80b.fsf@assigned-by-dhcp.cox.net","threadId":"2922","inReplyTo":"7vd5jmvnkc.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH] Require packfiles to follow the naming convention.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-12-24T10:03:32Z","receivedAt":"2005-12-24T10:03:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <junkio@cox.net> writes:\n\n> * require packfiles to follow the naming convention more strictly.\n>\n>   This rejects a packfile whose name does not begin with\n>   \"pack-\" followed by 40-byte hexadecimal digits.  The core\n>   never had such limitation (the only thing it required was\n>   that the pack and the idx are named with the same basename\n>   with .pack and .idx extention), but the way http-fetch\n>   handles packs requires this.\n> ...\n>   This is probably controversial because it would eventually\n>   require users to rename their packs created using pre v0.99.9\n>   tools.  There is a small utility program that helps the\n>   conversion process, so it may not be that painful.\n\nAnd this is the patch in question.  When your packs are the ones\nproduced by pre v0.99.9 tools, you can ask git-convert-packs to\nsuggest you the \"right name\" to use.\n\n$ git-convert-packs\nwarning: index name '.git/objects/pack/pack-a3b60db8bb6dad970f6dcdc03b0addf3988e8556.idx' does not match contents of the pack '83b60db8bb6dad970f6dcdc03b0addf3988e8556'\n# Packname does not match contents.\nmv '.git/objects/pack/pack-a3b60db8bb6dad970f6dcdc03b0addf3988e8556.pack' '.git/objects/pack/pack-83b60db8bb6dad970f6dcdc03b0addf3988e8556.pack'\nmv '.git/objects/pack/pack-a3b60db8bb6dad970f6dcdc03b0addf3988e8556.idx' '.git/objects/pack/pack-83b60db8bb6dad970f6dcdc03b0addf3988e8556.idx'\n$ git-convert-packs | sh \nwarning: index name '.git/objects/pack/pack-a3b60db8bb6dad970f6dcdc03b0addf3988e8556.idx' does not match contents of the pack '83b60db8bb6dad970f6dcdc03b0addf3988e8556'\n$ git-convert-packs\n$ exit\n\nComments?\n\n-- >8 --\nPack files names are required to be \"pack-[0-9a-f]{40}.pack\",\nwith corresponding .idx file.  This is not strictly needed at\nthe core level, but the packed file transfer over HTTP relies on\nthis format.  With this commit, the core would ignore otherwise\na valid packfile in .git/objects/packs that \"git-unpack-objects\"\nwould expand properly, if it does not follow the naming\nconvention.\n\nThe hexadecimal part should be what git-pack-objects computed\nwhen the pack was created originally, and it is the SHA1 sum\nover sorted object names (i.e. 20-byte binary SHA1 checksum\neach).  However, this condition is checked but not enforced\ncurrently, because git-pack-objects before v0.99.9 computed a\nchecksum over object names without sorting (i.e. the order\ngit-rev-list fed the object names), just to make pack names\nunlikely to collide.  This made less than ideal use of SHA1 sum\nto name packs, which we would want to rectify in the future.\n\nMany packfiles in the wild were created with git before v0.99.9,\nso the check would trigger and will give a warning message like\nthis:\n\nwarning: index name '.git/objects/pack/pack-3b60db8....idx' does\nnot match contents of the pack '83b60db...'\n\nYou can rename the existing pack that has this old name\n(3b60db8...) to new name (83b60db...) to squelch this warning\nmessage.  For that, a new command git-convert-packs is provided\nto produce suitable \"mv\" command you can paste into your shell.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n\n---\n\n .gitignore             |    1 +\n Makefile               |    2 +\n convert-packs.c        |   49 ++++++++++++++++++++++++++++++\n sha1_file.c            |   41 ++++++++++++++++++++++++-\n t/t5300-pack-object.sh |   79 ++++++++++++++++++++++++++----------------------\n 5 files changed, 133 insertions(+), 39 deletions(-)\n create mode 100644 convert-packs.c\n\n9c9f23b012e4030f433c7d4260a3fae93c302a0e\ndiff --git a/.gitignore b/.gitignore\nindex 6bd508e..d713f62 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -18,6 +18,7 @@ git-clone-pack\n git-commit\n git-commit-tree\n git-convert-objects\n+git-convert-packs\n git-count-objects\n git-cvsexportcommit\n git-cvsimport\ndiff --git a/Makefile b/Makefile\nindex 3395a9e..2209f97 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -121,7 +121,7 @@ SIMPLE_PROGRAMS = \\\n PROGRAMS = \\\n \tgit-apply$X git-cat-file$X \\\n \tgit-checkout-index$X git-clone-pack$X git-commit-tree$X \\\n-\tgit-convert-objects$X git-diff-files$X \\\n+\tgit-convert-objects$X git-convert-packs$X git-diff-files$X \\\n \tgit-diff-index$X git-diff-stages$X \\\n \tgit-diff-tree$X git-fetch-pack$X git-fsck-objects$X \\\n \tgit-hash-object$X git-index-pack$X git-init-db$X \\\ndiff --git a/convert-packs.c b/convert-packs.c\nnew file mode 100644\nindex 0000000..13f5502\n--- /dev/null\n+++ b/convert-packs.c\n@@ -0,0 +1,49 @@\n+#include \"cache.h\"\n+\n+void verify_pack_name(struct packed_git *p)\n+{\n+\tSHA_CTX ctx;\n+\tunsigned char sha1[20];\n+\tint i, nr, len;\n+\tconst char *hex;\n+\n+\tnr = num_packed_objects(p);\n+\n+\tSHA1_Init(&ctx);\n+\tfor (i = 0; i < nr; i++) {\n+\t\tunsigned char *ent = (((unsigned char *)p->index_base) +\n+\t\t\t\t      4*256 + i * 24 + 4);\n+\t\tSHA1_Update(&ctx, ent, 20);\n+\t}\n+\tSHA1_Final(sha1, &ctx);\n+\t\n+\tlen = strlen(p->pack_name);\n+\thex = sha1_to_hex(sha1);\n+\t/* \"pack-X{40}.pack\" */\n+\tif (len < 50 ||\n+\t    memcmp(p->pack_name + len - 5 - 40, hex, 40)) {\n+\t\tchar *d = strrchr(p->pack_name, '/');\n+\t\tint dlen;\n+\t\t\n+\t\tif (d)\n+\t\t\tdlen = (d - p->pack_name) + 1;\n+\t\telse\n+\t\t\tdlen = 0;\n+\t\tprintf(\"# Packname does not match contents.\\n\");\n+\t\tprintf(\"mv '%s' '%.*spack-%s.pack'\\n\",\n+\t\t       p->pack_name, dlen, p->pack_name, hex);\n+\t\tprintf(\"mv '%.*sidx' '%.*spack-%s.idx'\\n\",\n+\t\t       len-4, p->pack_name, dlen, p->pack_name, hex);\n+\t}\n+}\n+\n+int main(int ac, char **av)\n+{\n+\tstruct packed_git *p;\n+\n+\tsetup_git_directory();\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tverify_pack_name(p);\n+\treturn 0;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 6b7577d..fc24d0e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -323,6 +323,7 @@ static int check_packed_git_idx(const ch\n {\n \tSHA_CTX ctx;\n \tunsigned char sha1[20];\n+\tunsigned char packname[20];\n \tvoid *idx_map;\n \tunsigned int *index;\n \tunsigned long idx_size;\n@@ -337,6 +338,16 @@ static int check_packed_git_idx(const ch\n \t\tclose(fd);\n \t\treturn -1;\n \t}\n+\n+\t/* We require \"...../pack-XXXX{40}XXXX.idx\" now.\n+\t * 5-byte \"pack-\", 40-byte hex and 4-byte .idx.\n+\t */\n+\ti = strlen(path);\n+\tif (i < 49 || strcmp(path + i - 4, \".idx\") ||\n+\t    strncmp(path + i - 49, \"pack-\", 5) ||\n+\t    get_sha1_hex(path + i - 44, packname))\n+\t\treturn error(\"non index file %s ignored\", path);\n+\n \tidx_size = st.st_size;\n \tidx_map = mmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n@@ -369,7 +380,7 @@ static int check_packed_git_idx(const ch\n \t\treturn error(\"wrong index file size\");\n \n \t/*\n-\t * File checksum.\n+\t * File checksum and index name.\n \t */\n \tSHA1_Init(&ctx);\n \tSHA1_Update(&ctx, idx_map, idx_size-20);\n@@ -378,6 +389,34 @@ static int check_packed_git_idx(const ch\n \tif (memcmp(sha1, idx_map + idx_size - 20, 20))\n \t\treturn error(\"index checksum mismatch\");\n \n+/*\n+ * 0: do not check, do not do anything.\n+ * 1: check and warn, but do not reject.\n+ * 2: check and reject if pack name is obsolete.\n+ */\n+#define PACK_NAME_ENFORCEMENT_PHASE 1\n+\n+#if PACK_NAME_ENFORCEMENT_PHASE != 0\n+\tSHA1_Init(&ctx);\n+\tfor (i = 0; i < nr; i++) {\n+\t\tunsigned char *ent = (idx_map + 4*256) + i * 24 + 4;\n+\t\tSHA1_Update(&ctx, ent, 20);\n+\t}\n+\tSHA1_Final(sha1, &ctx);\n+\t\n+\tif (memcmp(sha1, packname, 20)) {\n+#if PACK_NAME_ENFORCEMENT_PHASE == 1\n+\t\tfprintf(stderr, \"warning: index name '%s' does not match \"\n+\t\t\t\"contents of the pack '%s'\\n\",\n+\t\t\tpath, sha1_to_hex(sha1));\n+#else\n+\t\treturn error(\"index name '%s' does not match \"\n+\t\t\t     \"contents of the pack '%s'\\n\",\n+\t\t\t     path, sha1_to_hex(sha1));\n+#endif\n+\t}\n+#endif\n+\n \treturn 0;\n }\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 7dfb1ab..dba8227 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -10,6 +10,8 @@ test_description='git-pack-object\n \n TRASH=`pwd`\n \n+mkdir test-1 test-2\n+\n test_expect_success \\\n     'setup' \\\n     'rm -f .git/index*\n@@ -35,7 +37,7 @@ test_expect_success \\\n \n test_expect_success \\\n     'pack without delta' \\\n-    'packname_1=$(git-pack-objects --window=0 test-1 <obj-list)'\n+    'packname_1=$(git-pack-objects --window=0 test-1/pack <obj-list)'\n \n rm -fr .git2\n mkdir .git2\n@@ -45,8 +47,8 @@ test_expect_success \\\n     \"GIT_OBJECT_DIRECTORY=.git2/objects &&\n      export GIT_OBJECT_DIRECTORY &&\n      git-init-db &&\n-     git-unpack-objects -n <test-1-${packname_1}.pack &&\n-     git-unpack-objects <test-1-${packname_1}.pack\"\n+     git-unpack-objects -n <test-1/pack-${packname_1}.pack &&\n+     git-unpack-objects <test-1/pack-${packname_1}.pack\"\n \n unset GIT_OBJECT_DIRECTORY\n cd \"$TRASH/.git2\"\n@@ -66,7 +68,8 @@ cd \"$TRASH\"\n test_expect_success \\\n     'pack with delta' \\\n     'pwd &&\n-     packname_2=$(git-pack-objects test-2 <obj-list)'\n+     packname_2=$(git-pack-objects test-2/pack <obj-list) &&\n+     test \"$packname_2\" = \"$packname_1\"'\n \n rm -fr .git2\n mkdir .git2\n@@ -76,8 +79,8 @@ test_expect_success \\\n     'GIT_OBJECT_DIRECTORY=.git2/objects &&\n      export GIT_OBJECT_DIRECTORY &&\n      git-init-db &&\n-     git-unpack-objects -n <test-2-${packname_2}.pack &&\n-     git-unpack-objects <test-2-${packname_2}.pack'\n+     git-unpack-objects -n <test-2/pack-${packname_2}.pack &&\n+     git-unpack-objects <test-2/pack-${packname_2}.pack'\n \n unset GIT_OBJECT_DIRECTORY\n cd \"$TRASH/.git2\"\n@@ -101,7 +104,8 @@ test_expect_success \\\n     'GIT_OBJECT_DIRECTORY=.git2/objects &&\n      export GIT_OBJECT_DIRECTORY &&\n      git-init-db &&\n-     cp test-1-${packname_1}.pack test-1-${packname_1}.idx .git2/objects/pack && {\n+     cp test-1/pack-${packname_1}.pack test-1/pack-${packname_1}.idx \\\n+     \t.git2/objects/pack && {\n \t git-diff-tree --root -p $commit &&\n \t while read object\n \t do\n@@ -116,8 +120,9 @@ test_expect_success \\\n     'use packed deltified objects' \\\n     'GIT_OBJECT_DIRECTORY=.git2/objects &&\n      export GIT_OBJECT_DIRECTORY &&\n-     rm -f .git2/objects/pack/test-?.idx &&\n-     cp test-2-${packname_2}.pack test-2-${packname_2}.idx .git2/objects/pack && {\n+     rm -f .git2/objects/pack/pack-* &&\n+     cp test-2/pack-${packname_2}.pack test-2/pack-${packname_2}.idx \\\n+     \t.git2/objects/pack && {\n \t git-diff-tree --root -p $commit &&\n \t while read object\n \t do\n@@ -131,41 +136,41 @@ unset GIT_OBJECT_DIRECTORY\n \n test_expect_success \\\n     'verify pack' \\\n-    'git-verify-pack test-1-${packname_1}.idx test-2-${packname_2}.idx'\n+    'git-verify-pack test-?/pack-${packname_1}.idx'\n \n test_expect_success \\\n     'corrupt a pack and see if verify catches' \\\n-    'cp test-1-${packname_1}.idx test-3.idx &&\n-     cp test-2-${packname_2}.pack test-3.pack &&\n-     if git-verify-pack test-3.idx\n+    'cp test-1/pack-${packname_1}.idx . &&\n+     cp test-2/pack-${packname_2}.pack . &&\n+     if git-verify-pack pack-${packname_1}.idx\n      then false\n      else :;\n      fi &&\n \n-     cp test-1-${packname_1}.pack test-3.pack &&\n-     dd if=/dev/zero of=test-3.pack count=1 bs=1 conv=notrunc seek=2 &&\n-     if git-verify-pack test-3.idx\n+     cp test-1/pack-${packname_1}.pack . &&\n+     dd if=/dev/zero of=pack-${packname_1}.pack count=1 bs=1 conv=notrunc seek=2 &&\n+     if git-verify-pack pack-${packname_1}.idx\n      then false\n      else :;\n      fi &&\n \n-     cp test-1-${packname_1}.pack test-3.pack &&\n-     dd if=/dev/zero of=test-3.pack count=1 bs=1 conv=notrunc seek=7 &&\n-     if git-verify-pack test-3.idx\n+     cp test-1/pack-${packname_1}.pack . &&\n+     dd if=/dev/zero of=pack-${packname_1}.pack count=1 bs=1 conv=notrunc seek=7 &&\n+     if git-verify-pack pack-${packname_1}.idx\n      then false\n      else :;\n      fi &&\n \n-     cp test-1-${packname_1}.pack test-3.pack &&\n-     dd if=/dev/zero of=test-3.pack count=1 bs=1 conv=notrunc seek=12 &&\n-     if git-verify-pack test-3.idx\n+     cp test-1/pack-${packname_1}.pack . &&\n+     dd if=/dev/zero of=pack-${packname_1}.pack count=1 bs=1 conv=notrunc seek=12 &&\n+     if git-verify-pack pack-${packname_1}.idx\n      then false\n      else :;\n      fi &&\n \n-     cp test-1-${packname_1}.pack test-3.pack &&\n-     dd if=/dev/zero of=test-3.idx count=1 bs=1 conv=notrunc seek=1200 &&\n-     if git-verify-pack test-3.pack\n+     cp test-1/pack-${packname_1}.pack . &&\n+     dd if=/dev/zero of=pack-${packname_1}.idx count=1 bs=1 conv=notrunc seek=1200 &&\n+     if git-verify-pack pack-${packname_1}.idx\n      then false\n      else :;\n      fi &&\n@@ -174,19 +179,19 @@ test_expect_success \\\n \n test_expect_success \\\n     'build pack index for an existing pack' \\\n-    'cp test-1-${packname_1}.pack test-3.pack &&\n-     git-index-pack -o tmp.idx test-3.pack &&\n-     cmp tmp.idx test-1-${packname_1}.idx &&\n-\n-     git-index-pack test-3.pack &&\n-     cmp test-3.idx test-1-${packname_1}.idx &&\n-\n-     cp test-2-${packname_2}.pack test-3.pack &&\n-     git-index-pack -o tmp.idx test-2-${packname_2}.pack &&\n-     cmp tmp.idx test-2-${packname_2}.idx &&\n+    'cp test-1/pack-${packname_1}.pack unknown.pack &&\n+     git-index-pack -o tmp.idx unknown.pack &&\n+     cmp tmp.idx test-1/pack-${packname_1}.idx &&\n+\n+     git-index-pack unknown.pack &&\n+     cmp unknown.idx test-1/pack-${packname_1}.idx &&\n+\n+     cp test-2/pack-${packname_2}.pack unknown.pack &&\n+     git-index-pack -o tmp.idx unknown.pack &&\n+     cmp tmp.idx test-2/pack-${packname_2}.idx &&\n \n-     git-index-pack test-3.pack &&\n-     cmp test-3.idx test-2-${packname_2}.idx &&\n+     git-index-pack unknown.pack &&\n+     cmp unknown.idx test-2/pack-${packname_2}.idx &&\n \n      :'\n \n-- \n1.0.GIT\n"}]}