{"thread":{"id":"48668","subject":"[PATCH 00/23] Multi-pack-index (MIDX)","startedAt":"2018-06-07T14:03:47Z","lastAt":"2018-07-20T22:28:39Z","messageCount":192,"participants":["Derrick Stolee","Ævar Arnfjörð Bjarmason","Duy Nguyen","Stefan Beller","Junio C Hamano","Eric Sunshine","SZEDER Gábor","brian m. carlson"],"isPatch":true,"patchVersion":1,"patchTotal":23},"messages":[{"id":"349612","messageId":"20180607140338.32440-1-dstolee@microsoft.com","threadId":"48668","inReplyTo":null,"subject":"[PATCH 00/23] Multi-pack-index (MIDX)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:15Z","receivedAt":"2018-06-07T14:03:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series includes a rewrite of the previous\nmulti-pack-index RFC [1] using the feedback from the\ncommit-graph feature.\n\nI based this series on 'next' as it requires the\nrecent object-store patches.\n\nThe multi-pack-index (MIDX) is explained fully in\nthe design document 'Documentation/technical/midx.txt'.\nThe short description is that the MIDX stores the\ninformation from all of the IDX files in a pack\ndirectory. The crucial design decision is that the\nIDX files still exist, so we can fall back to the IDX\nfiles if there is any issue with the MIDX (or core.midx\nis set to false, or a user downgrades Git, etc.)\n\nThe MIDX feature has been part of our GVFS releases\nfor a few months (since the RFC). It has behaved well,\nindexing over 31 million commits and trees across up\nto 250 packfiles. These MIDX files are nearly 1GB in\nsize and take ~20 seconds to rewrite when adding new\nIDX information. This ~20s mark is something I'd like\nto improve, and I mention how to make the file\nincremental (similar to split-index) in the design\ndocument. I also want to make the commit-graph file\nincremental, so I'd like to do that at the same time\nafter both the MIDX and commit-graph are stable.\n\n\nLookup Speedups\n---------------\n\nWhen looking for an object, Git uses an most-recently-\nused (MRU) cache of packfiles. This does pretty well to\nminimize the number of misses when searching through\npackfiles for an object, especially if there is one\n\"big\" packfile that contains most of the objets (so it\nwill rarely miss and is usually one of the first two\npackfiles in the list). The MIDX does provide a way\nto remove these misses, improving lookup time. However,\nthis lookup time greatly depends on the arrangement of\nthe packfiles.\n\nFor instance, if you take the Linux repository and repack\nusing `git repack -adfF --max-pack-size=128m` then all\ncommits will be in one packfile, all trees will be in\na small set of packfiles and organized well so 'git\nrev-list --objects HEAD^{tree}' only inspects one or two\npackfiles.\n\nGVFS has the notion of a \"prefetch packfile\". These are\npackfiles that are precomputed by cache servers to\ncontain the commits and trees introduced to the remote\neach day. GVFS downloads these packfiles and places them\nin an alternate. Since these are organized by \"first\ntime introduced\" and the working directory is so large,\nthe MRU misses are significant when performing a checkout\nand updating the .git/index file.\n\nTo test the performance in this situation, I created a\nscript that organizes the Linux repository in a similar\nfashion. I split the commit history into 50 parts by\ncreating branches on every 10,000 commits of the first-\nparent history. Then, `git rev-list --objects A ^B`\nprovides the list of objects reachable from A but not B,\nso I could send that to `git pack-objects` to create\nthese \"time-based\" packfiles. With these 50 packfiles\n(deleting the old one from my fresh clone, and deleting\nall tags as they were no longer on-disk) I could then\ntest 'git rev-list --objects HEAD^{tree}' and see:\n\n Before: 0.17s\n After: 0.13s\n % Diff: -23.5%\n\nBy adding logic to count hits and misses to bsearch_pack,\nI was able to see that the command above calls that\nmethod 266,930 times with a hit rate of 33%. The MIDX\nhas the same number of calls with a 100% hit rate.\n\n\n\nAbbreviation Speedups\n---------------------\n\nTo fully disambiguate an abbreviation, we must iterate\nthrough all packfiles to ensure no collision exists in\nany packfile. This requires O(P log N) time. With the\nMIDX, this is only O(log N) time. Our standard test [2]\nis 'git log --oneline --parents --raw' because it writes\nmany abbreviations while also doing a lot of other work\n(walking commits and trees to compute the raw diff).\n\nFor a copy of the Linux repository with 50 packfiles\nsplit by time, we observed the following:\n\n Before: 100.5 s\n After: 58.2 s\n % Diff: -59.7%\n\n\nRequest for Review Attention\n----------------------------\n\nI tried my best to take the feedback from the commit-graph\nfeature and apply it to this feature. I also worked to\nfollow the object-store refactoring as I could. I also have\nsome local commits that create a 'verify' subcommand and\nintegrate with 'fsck' similar to the commit-graph, but I'll\nleave those for a later series (and review is still underway\nfor that part of the commit-graph).\n\nOne place where I could use some guidance is related to the\ncurrent state of 'the_hash_algo' patches. The file format\nallows a different \"hash version\" which then indicates the\nlength of the hash. What's the best way to ensure this\nfeature doesn't cause extra pain in the hash-agnostic series?\nThis will inform how I go back and make the commit-graph\nfeature better in this area, too.\n\n\nThanks,\n-Stolee\n\n[1] https://public-inbox.org/git/20180107181459.222909-1-dstolee@microsoft.com/T/#u\n Previous MIDX RFC.\n\n[2] https://public-inbox.org/git/20171012120220.226427-1-dstolee@microsoft.com/\n A patch series on abbreviation speedups\n\n\nDerrick Stolee (23):\n midx: add design document\n midx: add midx format details to pack-format.txt\n midx: add midx builtin\n midx: add 'write' subcommand and basic wiring\n midx: write header information to lockfile\n midx: struct midxed_git and 'read' subcommand\n midx: expand test data\n midx: read packfiles from pack directory\n midx: write pack names in chunk\n midx: write a lookup into the pack names chunk\n midx: sort and deduplicate objects from packfiles\n midx: write object ids in a chunk\n midx: write object id fanout chunk\n midx: write object offsets\n midx: create core.midx config setting\n midx: prepare midxed_git struct\n midx: read objects from multi-pack-index\n midx: use midx in abbreviation calculations\n midx: use existing midx when writing new one\n midx: use midx in approximate_object_count\n midx: prevent duplicate packfile loads\n midx: use midx to find ref-deltas\n midx: clear midx on repack\n\n .gitignore | 1 +\n Documentation/config.txt | 4 +\n Documentation/git-midx.txt | 60 ++\n Documentation/technical/midx.txt | 109 +++\n Documentation/technical/pack-format.txt | 82 +++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/midx.c | 88 +++\n builtin/repack.c | 8 +\n cache.h | 1 +\n command-list.txt | 1 +\n config.c | 5 +\n environment.c | 1 +\n git.c | 1 +\n midx.c | 923 ++++++++++++++++++++++++\n midx.h | 23 +\n object-store.h | 35 +\n packfile.c | 47 +-\n packfile.h | 1 +\n sha1-name.c | 70 ++\n t/t5319-midx.sh | 192 +++++\n 21 files changed, 1652 insertions(+), 3 deletions(-)\n create mode 100644 Documentation/git-midx.txt\n create mode 100644 Documentation/technical/midx.txt\n create mode 100644 builtin/midx.c\n create mode 100644 midx.c\n create mode 100644 midx.h\n create mode 100755 t/t5319-midx.sh\n\n-- \n2.18.0.rc1\n\n"},{"id":"349613","messageId":"20180607140338.32440-2-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 01/23] midx: add design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:16Z","receivedAt":"2018-06-07T14:03:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/midx.txt | 109 +++++++++++++++++++++++++++++++\n 1 file changed, 109 insertions(+)\n create mode 100644 Documentation/technical/midx.txt\n\ndiff --git a/Documentation/technical/midx.txt b/Documentation/technical/midx.txt\nnew file mode 100644\nindex 0000000000..789f410d71\n--- /dev/null\n+++ b/Documentation/technical/midx.txt\n@@ -0,0 +1,109 @@\n+Multi-Pack-Index (MIDX) Design Notes\n+====================================\n+\n+The Git object directory contains a 'pack' directory containing\n+packfiles (with suffix \".pack\") and pack-indexes (with suffix\n+\".idx\"). The pack-indexes provide a way to lookup objects and\n+navigate to their offset within the pack, but these must come\n+in pairs with the packfiles. This pairing depends on the file\n+names, as the pack-index differs only in suffix with its pack-\n+file. While the pack-indexes provide fast lookup per packfile,\n+this performance degrades as the number of packfiles increases,\n+because abbreviations need to inspect every packfile and we are\n+more likely to have a miss on our most-recently-used packfile.\n+For some large repositories, repacking into a single packfile\n+is not feasible due to storage space or excessive repack times.\n+\n+The multi-pack-index (MIDX for short) stores a list of objects\n+and their offsets into multiple packfiles. It contains:\n+\n+- A list of packfile names.\n+- A sorted list of object IDs.\n+- A list of metadata for the ith object ID including:\n+ - A value j referring to the jth packfile.\n+ - An offset within the jth packfile for the object.\n+- If large offsets are required, we use another list of large\n+ offsets similar to version 2 pack-indexes.\n+\n+Thus, we can provide O(log N) lookup time for any number\n+of packfiles.\n+\n+Design Details\n+--------------\n+\n+- The MIDX is stored in a file named 'multi-pack-index' in the\n+ .git/objects/pack directory. This could be stored in the pack\n+ directory of an alternate. It refers only to packfiles in that\n+ same directory.\n+\n+- The core.midx config setting must be on to consume MIDX files.\n+\n+- The file format includes parameters for the object ID hash\n+ function, so a future change of hash algorithm does not require\n+ a change in format.\n+\n+- The MIDX keeps only one record per object ID. If an object appears\n+ in multiple packfiles, then the MIDX selects the copy in the most-\n+ recently modified packfile.\n+\n+- If there exist packfiles in the pack directory not registered in\n+ the MIDX, then those packfiles are loaded into the `packed_git`\n+ list and `packed_git_mru` cache.\n+\n+- The pack-indexes (.idx files) remain in the pack directory so we\n+ can delete the MIDX file, set core.midx to false, or downgrade\n+ without any loss of information.\n+\n+- The MIDX file format uses a chunk-based approach (similar to the\n+ commit-graph file) that allows optional data to be added.\n+\n+Future Work\n+-----------\n+\n+- Add a 'verify' subcommand to the 'git midx' builtin to verify the\n+ contents of the multi-pack-index file match the offsets listed in\n+ the corresponding pack-indexes.\n+\n+- The multi-pack-index allows many packfiles, especially in a context\n+ where repacking is expensive (such as a very large repo), or\n+ unexpected maintenance time is unacceptable (such as a high-demand\n+ build machine). However, the multi-pack-index needs to be rewritten\n+ in full every time. We can extend the format to be incremental, so\n+ writes are fast. By storing a small \"tip\" multi-pack-index that\n+ points to large \"base\" MIDX files, we can keep writes fast while\n+ still reducing the number of binary searches required for object\n+ lookups.\n+\n+- The reachability bitmap is currently paired directly with a single\n+ packfile, using the pack-order as the object order to hopefully\n+ compress the bitmaps well using run-length encoding. This could be\n+ extended to pair a reachability bitmap with a multi-pack-index. If\n+ the multi-pack-index is extended to store a \"stable object order\"\n+ (a function Order(hash) = integer that is constant for a given hash,\n+ even as the multi-pack-index is updated) then a reachability bitmap\n+ could point to a multi-pack-index and be updated independently.\n+\n+- Packfiles can be marked as \"special\" using empty files that share\n+ the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n+ We can add an optional chunk of data to the multi-pack-index that\n+ records flags of information about the packfiles. This allows new\n+ states, such as 'repacked' or 'redeltified', that can help with\n+ pack maintenance in a multi-pack environment. It may also be\n+ helpful to organize packfiles by object type (commit, tree, blob,\n+ etc.) and use this metadata to help that maintenance.\n+\n+- The partial clone feature records special \"promisor\" packs that\n+ may point to objects that are not stored locally, but available\n+ on request to a server. The multi-pack-index does not currently\n+ track these promisor packs.\n+\n+Related Links\n+-------------\n+[0] https://bugs.chromium.org/p/git/issues/detail?id=6\n+ Chromium work item for: Multi-Pack Index (MIDX)\n+\n+[1] https://public-inbox.org/git/20180107181459.222909-1-dstolee@microsoft.com/\n+ An earlier RFC for the multi-pack-index feature\n+\n+[2] https://public-inbox.org/git/alpine.DEB.2.20.1803091557510.23109@alexmv-linux/\n+ Git Merge 2018 Contributor's summit notes (includes discussion of MIDX)\n-- \n2.18.0.rc1\n\n"},{"id":"349614","messageId":"20180607140338.32440-4-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 03/23] midx: add midx builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:18Z","receivedAt":"2018-06-07T14:03:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This new 'git midx' builtin will be the plumbing access for writing,\nreading, and checking multi-pack-index (MIDX) files. The initial\nimplementation is a no-op.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-midx.txt | 29 +++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/midx.c | 38 ++++++++++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n 7 files changed, 72 insertions(+)\n create mode 100644 Documentation/git-midx.txt\n create mode 100644 builtin/midx.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 388cc4beee..e309644d6b 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -97,6 +97,7 @@\n /git-merge-subtree\n /git-mergetool\n /git-mergetool--lib\n+/git-midx\n /git-mktag\n /git-mktree\n /git-name-rev\ndiff --git a/Documentation/git-midx.txt b/Documentation/git-midx.txt\nnew file mode 100644\nindex 0000000000..2bd886f1a2\n--- /dev/null\n+++ b/Documentation/git-midx.txt\n@@ -0,0 +1,29 @@\n+git-midx(1)\n+============\n+\n+NAME\n+----\n+git-midx - Write and verify multi-pack-indexes (MIDX files).\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git midx' [--object-dir ]\n+\n+DESCRIPTION\n+-----------\n+Write or verify a MIDX file.\n+\n+OPTIONS\n+-------\n+\n+--object-dir ::\n+\tUse given directory for the location of Git objects. We check\n+\t/packs/multi-pack-index for the current MIDX file, and\n+\t/packs for the pack-files to index.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 1d27f36365..88958c7b42 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1045,6 +1045,7 @@ BUILTIN_OBJS += builtin/merge-index.o\n BUILTIN_OBJS += builtin/merge-ours.o\n BUILTIN_OBJS += builtin/merge-recursive.o\n BUILTIN_OBJS += builtin/merge-tree.o\n+BUILTIN_OBJS += builtin/midx.o\n BUILTIN_OBJS += builtin/mktag.o\n BUILTIN_OBJS += builtin/mktree.o\n BUILTIN_OBJS += builtin/mv.o\ndiff --git a/builtin.h b/builtin.h\nindex 4e0f64723e..7b5bd46c7d 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -189,6 +189,7 @@ extern int cmd_merge_ours(int argc, const char **argv, const char *prefix);\n extern int cmd_merge_file(int argc, const char **argv, const char *prefix);\n extern int cmd_merge_recursive(int argc, const char **argv, const char *prefix);\n extern int cmd_merge_tree(int argc, const char **argv, const char *prefix);\n+extern int cmd_midx(int argc, const char **argv, const char *prefix);\n extern int cmd_mktag(int argc, const char **argv, const char *prefix);\n extern int cmd_mktree(int argc, const char **argv, const char *prefix);\n extern int cmd_mv(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/midx.c b/builtin/midx.c\nnew file mode 100644\nindex 0000000000..59ea92178f\n--- /dev/null\n+++ b/builtin/midx.c\n@@ -0,0 +1,38 @@\n+#include \"builtin.h\"\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"git-compat-util.h\"\n+#include \"parse-options.h\"\n+\n+static char const * const builtin_midx_usage[] ={\n+\tN_(\"git midx [--object-dir ]\"),\n+\tNULL\n+};\n+\n+static struct opts_midx {\n+\tconst char *object_dir;\n+} opts;\n+\n+int cmd_midx(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_midx_options[] = {\n+\t\t{ OPTION_STRING, 0, \"object-dir\", &opts.object_dir,\n+\t\t N_(\"dir\"),\n+\t\t N_(\"The object directory containing set of packfile and pack-index pairs.\") },\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_midx_usage, builtin_midx_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_midx_options,\n+\t\t\t builtin_midx_usage, 0);\n+\n+\tif (!opts.object_dir)\n+\t\topts.object_dir = get_object_directory();\n+\n+\treturn 0;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex e1c26c1bb7..a21bd7470e 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -123,6 +123,7 @@ git-merge-index plumbingmanipulators\n git-merge-one-file purehelpers\n git-mergetool ancillarymanipulators complete\n git-merge-tree ancillaryinterrogators\n+git-midx plumbingmanipulators\n git-mktag plumbingmanipulators\n git-mktree plumbingmanipulators\n git-mv mainporcelain worktree\ndiff --git a/git.c b/git.c\nindex c2f48d53dd..400fadd677 100644\n--- a/git.c\n+++ b/git.c\n@@ -503,6 +503,7 @@ static struct cmd_struct commands[] = {\n \t{ \"merge-recursive-theirs\", cmd_merge_recursive, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"merge-subtree\", cmd_merge_recursive, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"merge-tree\", cmd_merge_tree, RUN_SETUP | NO_PARSEOPT },\n+\t{ \"midx\", cmd_midx, RUN_SETUP },\n \t{ \"mktag\", cmd_mktag, RUN_SETUP | NO_PARSEOPT },\n \t{ \"mktree\", cmd_mktree, RUN_SETUP },\n \t{ \"mv\", cmd_mv, RUN_SETUP | NEED_WORK_TREE },\n-- \n2.18.0.rc1\n\n"},{"id":"349615","messageId":"20180607140338.32440-5-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 04/23] midx: add 'write' subcommand and basic wiring","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:19Z","receivedAt":"2018-06-07T14:03:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of writing multi-pack-indexes (MIDX files), add a\n'git midx write' subcommand and send the options to a write_midx_file()\nmethod. Also create a basic test file that tests the 'write' subcommand.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-midx.txt | 22 +++++++++++++++++++++-\n Makefile | 1 +\n builtin/midx.c | 9 ++++++++-\n midx.c | 9 +++++++++\n midx.h | 4 ++++\n t/t5319-midx.sh | 10 ++++++++++\n 6 files changed, 53 insertions(+), 2 deletions(-)\n create mode 100644 midx.c\n create mode 100644 midx.h\n create mode 100755 t/t5319-midx.sh\n\ndiff --git a/Documentation/git-midx.txt b/Documentation/git-midx.txt\nindex 2bd886f1a2..dcaeb1a91b 100644\n--- a/Documentation/git-midx.txt\n+++ b/Documentation/git-midx.txt\n@@ -9,7 +9,7 @@ git-midx - Write and verify multi-pack-indexes (MIDX files).\n SYNOPSIS\n --------\n [verse]\n-'git midx' [--object-dir ]\n+'git midx' [--object-dir ] \n \n DESCRIPTION\n -----------\n@@ -23,6 +23,26 @@ OPTIONS\n \t/packs/multi-pack-index for the current MIDX file, and\n \t/packs for the pack-files to index.\n \n+write::\n+\tWhen given as the verb, write a new MIDX file to\n+\t/packs/multi-pack-index.\n+\n+\n+EXAMPLES\n+--------\n+\n+* Write a MIDX file for the packfiles in the current .git folder.\n++\n+-------------------------------------------\n+$ git midx write\n+-------------------------------------------\n+\n+* Write a MIDX file for the packfiles in an alternate.\n++\n+-------------------------------------------\n+$ git midx --object-dir write\n+-------------------------------------------\n+\n \n GIT\n ---\ndiff --git a/Makefile b/Makefile\nindex 88958c7b42..aa86fcd8ec 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -890,6 +890,7 @@ LIB_OBJS += merge.o\n LIB_OBJS += merge-blobs.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += mergesort.o\n+LIB_OBJS += midx.o\n LIB_OBJS += name-hash.o\n LIB_OBJS += notes.o\n LIB_OBJS += notes-cache.o\ndiff --git a/builtin/midx.c b/builtin/midx.c\nindex 59ea92178f..dc0a5acd3f 100644\n--- a/builtin/midx.c\n+++ b/builtin/midx.c\n@@ -3,9 +3,10 @@\n #include \"config.h\"\n #include \"git-compat-util.h\"\n #include \"parse-options.h\"\n+#include \"midx.h\"\n \n static char const * const builtin_midx_usage[] ={\n-\tN_(\"git midx [--object-dir ]\"),\n+\tN_(\"git midx [--object-dir ] [write]\"),\n \tNULL\n };\n \n@@ -34,5 +35,11 @@ int cmd_midx(int argc, const char **argv, const char *prefix)\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \n+\tif (argc == 0)\n+\t\treturn 0;\n+\n+\tif (!strcmp(argv[0], \"write\"))\n+\t\treturn write_midx_file(opts.object_dir);\n+\n \treturn 0;\n }\ndiff --git a/midx.c b/midx.c\nnew file mode 100644\nindex 0000000000..616af66b13\n--- /dev/null\n+++ b/midx.c\n@@ -0,0 +1,9 @@\n+#include \"git-compat-util.h\"\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"midx.h\"\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nnew file mode 100644\nindex 0000000000..3a63673952\n--- /dev/null\n+++ b/midx.h\n@@ -0,0 +1,4 @@\n+#include \"cache.h\"\n+#include \"packfile.h\"\n+\n+int write_midx_file(const char *object_dir);\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nnew file mode 100755\nindex 0000000000..a590137af7\n--- /dev/null\n+++ b/t/t5319-midx.sh\n@@ -0,0 +1,10 @@\n+#!/bin/sh\n+\n+test_description='multi-pack-indexes'\n+. ./test-lib.sh\n+\n+test_expect_success 'write midx with no pakcs' '\n+\tgit midx --object-dir=. write\n+'\n+\n+test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349616","messageId":"20180607140338.32440-10-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 09/23] midx: write pack names in chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:24Z","receivedAt":"2018-06-07T14:04:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index (MIDX) needs to track which pack-files are covered\nby the MIDX file. Store these in our first required chunk. Since\nfilenames are not well structured, add padding to keep good alignment in\nlater chunks.\n\nModify the 'git midx read' subcommand to output the existence of the\npack-file name chunk. Modify t5319-midx.sh to reflect this new output\nand the new expected number of chunks.\n\nDefense in depth: A pattern we are using in the multi-pack-index feature\nis to verify the data as we write it. We want to ensure we never write\ninvalid data to the multi-pack-index. There are many checks during the\nwrite of a MIDX file that double-check that the values we are writing\nfit the format definitions. If any value is incorrect, then we notice\nbefore writing invalid data. This mainly helps developers while working\non the feature, but it can also identify issues that only appear when\ndealing with very large data sets. These large sets are hard to encode\ninto test cases.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 6 +\n builtin/midx.c | 7 +\n midx.c | 176 +++++++++++++++++++++++-\n object-store.h | 2 +\n t/t5319-midx.sh | 3 +-\n 5 files changed, 188 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 17666b4bfc..2b37be7b33 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -296,6 +296,12 @@ CHUNK LOOKUP:\n \n CHUNK DATA:\n \n+\tPackfile Names (ID: {'P', 'N', 'A', 'M'})\n+\t Stores the packfile names as concatenated, null-terminated strings.\n+\t Packfiles must be listed in lexicographic order for fast lookups by\n+\t name. This is the only chunk not guaranteed to be a multiple of four\n+\t bytes in length, so should be the last chunk for alignment reasons.\n+\n \t(This section intentionally left incomplete.)\n \n TRAILER:\ndiff --git a/builtin/midx.c b/builtin/midx.c\nindex c7002f664a..fe56560853 100644\n--- a/builtin/midx.c\n+++ b/builtin/midx.c\n@@ -28,6 +28,13 @@ static int read_midx_file(const char *object_dir)\n \t m->num_chunks,\n \t m->num_packs);\n \n+\tprintf(\"chunks:\");\n+\n+\tif (m->chunk_pack_names)\n+\t\tprintf(\" pack_names\");\n+\n+\tprintf(\"\\n\");\n+\n \tprintf(\"object_dir: %s\\n\", m->object_dir);\n \n \treturn 0;\ndiff --git a/midx.c b/midx.c\nindex 9fb89c80a2..d4f4a01a51 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -13,6 +13,11 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n+#define MIDX_MAX_CHUNKS 1\n+#define MIDX_CHUNK_ALIGNMENT 4\n+#define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \tstruct strbuf midx_name = STRBUF_INIT;\n@@ -29,6 +34,7 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \tsize_t midx_size;\n \tvoid *midx_map;\n \tconst char *midx_name = get_midx_filename(object_dir);\n+\tuint32_t i;\n \n \tfd = git_open(midx_name);\n \tif (fd < 0)\n@@ -74,6 +80,31 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \tm->num_chunks = *(m->data + 6);\n \tm->num_packs = get_be32(m->data + 8);\n \n+\tfor (i = 0; i < m->num_chunks; i++) {\n+\t\tuint32_t chunk_id = get_be32(m->data + 12 + MIDX_CHUNKLOOKUP_WIDTH * i);\n+\t\tuint64_t chunk_offset = get_be64(m->data + 16 + MIDX_CHUNKLOOKUP_WIDTH * i);\n+\n+\t\tswitch (chunk_id) {\n+\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n+\t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase 0:\n+\t\t\t\tdie(\"terminating MIDX chunk id appears earlier than expected\");\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\t/*\n+\t\t\t\t * Do nothing on unrecognized chunks, allowing future\n+\t\t\t\t * extensions to add optional chunks.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (!m->chunk_pack_names)\n+\t\tdie(\"MIDX missing required pack-name chunk\");\n+\n \treturn m;\n \n cleanup_fail:\n@@ -99,18 +130,88 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+struct pack_pair {\n+\tuint32_t pack_int_id;\n+\tchar *pack_name;\n+};\n+\n+static int pack_pair_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_pair *a = (struct pack_pair *)_a;\n+\tstruct pack_pair *b = (struct pack_pair *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n+static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n+{\n+\tuint32_t i;\n+\tstruct pack_pair *pairs;\n+\n+\tALLOC_ARRAY(pairs, nr_packs);\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\tpairs[i].pack_int_id = i;\n+\t\tpairs[i].pack_name = pack_names[i];\n+\t}\n+\n+\tQSORT(pairs, nr_packs, pack_pair_compare);\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\tpack_names[i] = pairs[i].pack_name;\n+\t\tperm[pairs[i].pack_int_id] = i;\n+\t}\n+}\n+\n+static size_t write_midx_pack_names(struct hashfile *f,\n+\t\t\t\t char **pack_names,\n+\t\t\t\t uint32_t num_packs)\n+{\n+\tuint32_t i;\n+\tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < num_packs; i++) {\n+\t\tsize_t writelen = strlen(pack_names[i]) + 1;\n+\n+\t\tif (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n+\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n+\t\t\t pack_names[i - 1],\n+\t\t\t pack_names[i]);\n+\n+\t\thashwrite(f, pack_names[i], writelen);\n+\t\twritten += writelen;\n+\t}\n+\n+\t/* add padding to be aligned */\n+\ti = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n+\tif (i < MIDX_CHUNK_ALIGNMENT) {\n+\t\tbzero(padding, sizeof(padding));\n+\t\thashwrite(f, padding, i);\n+\t\twritten += i;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n-\tunsigned char num_chunks = 0;\n+\tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n \tstruct hashfile *f;\n \tstruct lock_file lk;\n \tstruct packed_git **packs = NULL;\n+\tchar **pack_names = NULL;\n+\tuint32_t *pack_perm;\n \tuint32_t i, nr_packs = 0, alloc_packs = 0;\n+\tuint32_t alloc_pack_names = 0;\n \tDIR *dir;\n \tstruct dirent *de;\n \tstruct strbuf pack_dir = STRBUF_INIT;\n \tsize_t pack_dir_len;\n+\tuint64_t pack_name_concat_len = 0;\n+\tuint64_t written = 0;\n+\tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n+\tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -132,12 +233,14 @@ int write_midx_file(const char *object_dir)\n \tstrbuf_addch(&pack_dir, '/');\n \tpack_dir_len = pack_dir.len;\n \tALLOC_ARRAY(packs, alloc_packs);\n+\tALLOC_ARRAY(pack_names, alloc_pack_names);\n \twhile ((de = readdir(dir)) != NULL) {\n \t\tif (is_dot_or_dotdot(de->d_name))\n \t\t\tcontinue;\n \n \t\tif (ends_with(de->d_name, \".idx\")) {\n \t\t\tALLOC_GROW(packs, nr_packs + 1, alloc_packs);\n+\t\t\tALLOC_GROW(pack_names, nr_packs + 1, alloc_pack_names);\n \n \t\t\tstrbuf_setlen(&pack_dir, pack_dir_len);\n \t\t\tstrbuf_addstr(&pack_dir, de->d_name);\n@@ -145,21 +248,83 @@ int write_midx_file(const char *object_dir)\n \t\t\tpacks[nr_packs] = add_packed_git(pack_dir.buf,\n \t\t\t\t\t\t\t pack_dir.len,\n \t\t\t\t\t\t\t 0);\n-\t\t\tif (!packs[nr_packs])\n+\t\t\tif (!packs[nr_packs]) {\n \t\t\t\twarning(\"failed to add packfile '%s'\",\n \t\t\t\t\tpack_dir.buf);\n-\t\t\telse\n-\t\t\t\tnr_packs++;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tpack_names[nr_packs] = xstrdup(de->d_name);\n+\t\t\tpack_name_concat_len += strlen(de->d_name) + 1;\n+\t\t\tnr_packs++;\n \t\t}\n \t}\n+\n \tclosedir(dir);\n \tstrbuf_release(&pack_dir);\n \n+\tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n+\tALLOC_ARRAY(pack_perm, nr_packs);\n+\tsort_packs_by_name(pack_names, nr_packs, pack_perm);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\twrite_midx_header(f, num_chunks, nr_packs);\n+\tcur_chunk = 0;\n+\tnum_chunks = 1;\n+\n+\twritten = write_midx_header(f, num_chunks, nr_packs);\n+\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n+\tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n+\n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = 0;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n+\n+\tfor (i = 0; i <= num_chunks; i++) {\n+\t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n+\t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n+\t\t\t chunk_offsets[i - 1],\n+\t\t\t chunk_offsets[i]);\n+\n+\t\tif (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n+\t\t\tBUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n+\t\t\t chunk_offsets[i]);\n+\n+\t\thashwrite_be32(f, chunk_ids[i]);\n+\t\thashwrite_be32(f, chunk_offsets[i] >> 32);\n+\t\thashwrite_be32(f, chunk_offsets[i]);\n+\n+\t\twritten += MIDX_CHUNKLOOKUP_WIDTH;\n+\t}\n+\n+\tfor (i = 0; i < num_chunks; i++) {\n+\t\tif (written != chunk_offsets[i])\n+\t\t\tBUG(\"inccrrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n+\t\t\t chunk_offsets[i],\n+\t\t\t written,\n+\t\t\t chunk_ids[i]);\n+\n+\t\tswitch (chunk_ids[i]) {\n+\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n+\t\t\t\twritten += write_midx_pack_names(f, pack_names, nr_packs);\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n+\t\t\t\t chunk_ids[i]);\n+\t\t}\n+\t}\n+\n+\tif (written != chunk_offsets[num_chunks])\n+\t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n+\t\t written,\n+\t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n@@ -170,5 +335,6 @@ int write_midx_file(const char *object_dir)\n \t}\n \n \tFREE_AND_NULL(packs);\n+\tFREE_AND_NULL(pack_names);\n \treturn 0;\n }\ndiff --git a/object-store.h b/object-store.h\nindex 77cb82621a..199cf4bd44 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -100,6 +100,8 @@ struct midxed_git {\n \tuint32_t num_packs;\n \tuint32_t num_objects;\n \n+\tconst unsigned char *chunk_pack_names;\n+\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex abe545c7c4..fdf4f84a90 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -6,7 +6,8 @@ test_description='multi-pack-indexes'\n midx_read_expect() {\n \tNUM_PACKS=$1\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 1 0 $NUM_PACKS\n+\theader: 4d494458 1 1 1 $NUM_PACKS\n+\tchunks: pack_names\n \tobject_dir: .\n \tEOF\n \tgit midx read --object-dir=. >actual &&\n-- \n2.18.0.rc1\n\n"},{"id":"349617","messageId":"20180607140338.32440-13-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 12/23] midx: write object ids in a chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:27Z","receivedAt":"2018-06-07T14:04:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 4 ++\n builtin/midx.c | 2 +\n midx.c | 50 +++++++++++++++++++++++--\n object-store.h | 1 +\n t/t5319-midx.sh | 4 +-\n 5 files changed, 55 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 29bf87283a..de9ac778b6 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -307,6 +307,10 @@ CHUNK DATA:\n \t name. This is the only chunk not guaranteed to be a multiple of four\n \t bytes in length, so should be the last chunk for alignment reasons.\n \n+\tOID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n+\t The OIDs for all objects in the MIDX are stored in lexicographic\n+\t order in this chunk.\n+\n \t(This section intentionally left incomplete.)\n \n TRAILER:\ndiff --git a/builtin/midx.c b/builtin/midx.c\nindex 3a261e9bbf..86edd30174 100644\n--- a/builtin/midx.c\n+++ b/builtin/midx.c\n@@ -35,6 +35,8 @@ static int read_midx_file(const char *object_dir)\n \t\tprintf(\" pack_lookup\");\n \tif (m->chunk_pack_names)\n \t\tprintf(\" pack_names\");\n+\tif (m->chunk_oid_lookup)\n+\t\tprintf(\" oid_lookup\");\n \n \tprintf(\"\\n\");\n \ndiff --git a/midx.c b/midx.c\nindex b20d52713c..d06bc6876a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -14,10 +14,11 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 2\n+#define MIDX_MAX_CHUNKS 3\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKLOOKUP 0x504c4f4f /* \"PLOO\" */\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n \n static char *get_midx_filename(const char *object_dir)\n@@ -95,6 +96,10 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n+\t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n \t\t\tcase 0:\n \t\t\t\tdie(\"terminating MIDX chunk id appears earlier than expected\");\n \t\t\t\tbreak;\n@@ -112,6 +117,8 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \t\tdie(\"MIDX missing required pack lookup chunk\");\n \tif (!m->chunk_pack_names)\n \t\tdie(\"MIDX missing required pack-name chunk\");\n+\tif (!m->chunk_oid_lookup)\n+\t\tdie(\"MIDX missing required OID lookup chunk\");\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n \tfor (i = 0; i < m->num_packs; i++) {\n@@ -370,6 +377,32 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \treturn written;\n }\n \n+static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n+\t\t\t\t struct pack_midx_entry *objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tuint32_t i;\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\tif (i < nr_objects - 1) {\n+\t\t\tstruct pack_midx_entry *next = list;\n+\t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n+\t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n+\t\t\t\toid_to_hex(&obj->oid),\n+\t\t\t\toid_to_hex(&next->oid));\n+\t\t}\n+\n+\t\thashwrite(f, obj->oid.hash, (int)hash_len);\n+\t\twritten += hash_len;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n@@ -389,6 +422,7 @@ int write_midx_file(const char *object_dir)\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n+\tstruct pack_midx_entry *entries;\n \tuint32_t nr_entries;\n \n \tmidx_name = get_midx_filename(object_dir);\n@@ -448,14 +482,14 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(pack_perm, nr_packs);\n \tsort_packs_by_name(pack_names, nr_packs, pack_perm);\n \n-\tget_sorted_entries(packs, pack_perm, nr_packs, &nr_entries);\n+\tentries = get_sorted_entries(packs, pack_perm, nr_packs, &nr_entries);\n \n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 2;\n+\tnum_chunks = 3;\n \n \twritten = write_midx_header(f, num_chunks, nr_packs);\n \n@@ -467,9 +501,13 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_packs * sizeof(uint32_t);\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = 0;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n \n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = 0;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n+\n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n \t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n@@ -503,6 +541,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_pack_names(f, pack_names, nr_packs);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n+\t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tdefault:\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\ndiff --git a/object-store.h b/object-store.h\nindex 1ba50459ca..7d14d3586e 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -102,6 +102,7 @@ struct midxed_git {\n \n \tconst uint32_t *chunk_pack_lookup;\n \tconst unsigned char *chunk_pack_names;\n+\tconst unsigned char *chunk_oid_lookup;\n \n \tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex a31c387c8f..e71aa52b80 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -6,8 +6,8 @@ test_description='multi-pack-indexes'\n midx_read_expect() {\n \tNUM_PACKS=$1\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 1 2 $NUM_PACKS\n-\tchunks: pack_lookup pack_names\n+\theader: 4d494458 1 1 3 $NUM_PACKS\n+\tchunks: pack_lookup pack_names oid_lookup\n \tpacks:\n \tEOF\n \tif [ $NUM_PACKS -ge 1 ]\n-- \n2.18.0.rc1\n\n"},{"id":"349618","messageId":"20180607140338.32440-11-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 10/23] midx: write a lookup into the pack names chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:25Z","receivedAt":"2018-06-07T14:04:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 5 +++\n builtin/midx.c | 7 ++++\n midx.c | 56 +++++++++++++++++++++++--\n object-store.h | 2 +\n t/t5319-midx.sh | 11 +++--\n 5 files changed, 75 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 2b37be7b33..29bf87283a 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -296,6 +296,11 @@ CHUNK LOOKUP:\n \n CHUNK DATA:\n \n+\tPackfile Name Lookup (ID: {'P', 'L', 'O', 'O'}) (P * 4 bytes)\n+\t P * 4 bytes storing the offset in the packfile name chunk for\n+\t the null-terminated string containing the filename for the\n+\t ith packfile.\n+\n \tPackfile Names (ID: {'P', 'N', 'A', 'M'})\n \t Stores the packfile names as concatenated, null-terminated strings.\n \t Packfiles must be listed in lexicographic order for fast lookups by\ndiff --git a/builtin/midx.c b/builtin/midx.c\nindex fe56560853..3a261e9bbf 100644\n--- a/builtin/midx.c\n+++ b/builtin/midx.c\n@@ -16,6 +16,7 @@ static struct opts_midx {\n \n static int read_midx_file(const char *object_dir)\n {\n+\tuint32_t i;\n \tstruct midxed_git *m = load_midxed_git(object_dir);\n \n \tif (!m)\n@@ -30,11 +31,17 @@ static int read_midx_file(const char *object_dir)\n \n \tprintf(\"chunks:\");\n \n+\tif (m->chunk_pack_lookup)\n+\t\tprintf(\" pack_lookup\");\n \tif (m->chunk_pack_names)\n \t\tprintf(\" pack_names\");\n \n \tprintf(\"\\n\");\n \n+\tprintf(\"packs:\\n\");\n+\tfor (i = 0; i < m->num_packs; i++)\n+\t\tprintf(\"%s\\n\", m->pack_names[i]);\n+\n \tprintf(\"object_dir: %s\\n\", m->object_dir);\n \n \treturn 0;\ndiff --git a/midx.c b/midx.c\nindex d4f4a01a51..923acda72e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -13,8 +13,9 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 1\n+#define MIDX_MAX_CHUNKS 2\n #define MIDX_CHUNK_ALIGNMENT 4\n+#define MIDX_CHUNKID_PACKLOOKUP 0x504c4f4f /* \"PLOO\" */\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n \n@@ -85,6 +86,10 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \t\tuint64_t chunk_offset = get_be64(m->data + 16 + MIDX_CHUNKLOOKUP_WIDTH * i);\n \n \t\tswitch (chunk_id) {\n+\t\t\tcase MIDX_CHUNKID_PACKLOOKUP:\n+\t\t\t\tm->chunk_pack_lookup = (uint32_t *)(m->data + chunk_offset);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n \t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n \t\t\t\tbreak;\n@@ -102,9 +107,32 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \t\t}\n \t}\n \n+\tif (!m->chunk_pack_lookup)\n+\t\tdie(\"MIDX missing required pack lookup chunk\");\n \tif (!m->chunk_pack_names)\n \t\tdie(\"MIDX missing required pack-name chunk\");\n \n+\tm->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tif (i) {\n+\t\t\tif (ntohl(m->chunk_pack_lookup[i]) <= ntohl(m->chunk_pack_lookup[i - 1])) {\n+\t\t\t\terror(\"MIDX pack lookup value %d before %d\",\n+\t\t\t\t ntohl(m->chunk_pack_lookup[i - 1]),\n+\t\t\t\t ntohl(m->chunk_pack_lookup[i]));\n+\t\t\t\tgoto cleanup_fail;\n+\t\t\t}\n+\t\t}\n+\n+\t\tm->pack_names[i] = (const char *)(m->chunk_pack_names + ntohl(m->chunk_pack_lookup[i]));\n+\n+\t\tif (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0) {\n+\t\t\terror(\"MIDX pack names out of order: '%s' before '%s'\",\n+\t\t\t m->pack_names[i - 1],\n+\t\t\t m->pack_names[i]);\n+\t\t\tgoto cleanup_fail;\n+\t\t}\n+\t}\n+\n \treturn m;\n \n cleanup_fail:\n@@ -162,6 +190,20 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n \t}\n }\n \n+static size_t write_midx_pack_lookup(struct hashfile *f,\n+\t\t\t\t char **pack_names,\n+\t\t\t\t uint32_t nr_packs)\n+{\n+\tuint32_t i, cur_len = 0;\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\thashwrite_be32(f, cur_len);\n+\t\tcur_len += strlen(pack_names[i]) + 1;\n+\t}\n+\n+\treturn sizeof(uint32_t) * (size_t)nr_packs;\n+}\n+\n static size_t write_midx_pack_names(struct hashfile *f,\n \t\t\t\t char **pack_names,\n \t\t\t\t uint32_t num_packs)\n@@ -275,13 +317,17 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 1;\n+\tnum_chunks = 2;\n \n \twritten = write_midx_header(f, num_chunks, nr_packs);\n \n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKLOOKUP;\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_packs * sizeof(uint32_t);\n+\n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = 0;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n@@ -311,6 +357,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t chunk_ids[i]);\n \n \t\tswitch (chunk_ids[i]) {\n+\t\t\tcase MIDX_CHUNKID_PACKLOOKUP:\n+\t\t\t\twritten += write_midx_pack_lookup(f, pack_names, nr_packs);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n \t\t\t\twritten += write_midx_pack_names(f, pack_names, nr_packs);\n \t\t\t\tbreak;\ndiff --git a/object-store.h b/object-store.h\nindex 199cf4bd44..1ba50459ca 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -100,8 +100,10 @@ struct midxed_git {\n \tuint32_t num_packs;\n \tuint32_t num_objects;\n \n+\tconst uint32_t *chunk_pack_lookup;\n \tconst unsigned char *chunk_pack_names;\n \n+\tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex fdf4f84a90..a31c387c8f 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -6,10 +6,15 @@ test_description='multi-pack-indexes'\n midx_read_expect() {\n \tNUM_PACKS=$1\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 1 1 $NUM_PACKS\n-\tchunks: pack_names\n-\tobject_dir: .\n+\theader: 4d494458 1 1 2 $NUM_PACKS\n+\tchunks: pack_lookup pack_names\n+\tpacks:\n \tEOF\n+\tif [ $NUM_PACKS -ge 1 ]\n+\tthen\n+\t\tls pack/ | grep idx | sort >> expect\n+\tfi\n+\tprintf \"object_dir: .\\n\" >>expect &&\n \tgit midx read --object-dir=. >actual &&\n \ttest_cmp expect actual\n }\n-- \n2.18.0.rc1\n\n"},{"id":"349619","messageId":"20180607140338.32440-17-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 16/23] midx: prepare midxed_git struct","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:31Z","receivedAt":"2018-06-07T14:04:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 22 ++++++++++++++++++++++\n midx.h | 2 ++\n object-store.h | 7 +++++++\n packfile.c | 6 +++++-\n 4 files changed, 36 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex a49300bf75..5e9290ca8f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -175,6 +175,28 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \texit(1);\n }\n \n+int prepare_midxed_git_one(struct repository *r, const char *object_dir)\n+{\n+\tstruct midxed_git *m = r->objects->midxed_git;\n+\tstruct midxed_git *m_search;\n+\n+\tif (!core_midx)\n+\t\treturn 0;\n+\n+\tfor (m_search = m; m_search; m_search = m_search->next)\n+\t\tif (!strcmp(object_dir, m_search->object_dir))\n+\t\t\treturn 1;\n+\n+\tr->objects->midxed_git = load_midxed_git(object_dir);\n+\n+\tif (r->objects->midxed_git) {\n+\t\tr->objects->midxed_git->next = m;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static size_t write_midx_header(struct hashfile *f,\n \t\t\t\tunsigned char num_chunks,\n \t\t\t\tuint32_t num_packs)\ndiff --git a/midx.h b/midx.h\nindex a1d18ed991..793203fc4a 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -5,8 +5,10 @@\n #include \"cache.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n+#include \"repository.h\"\n \n struct midxed_git *load_midxed_git(const char *object_dir);\n+int prepare_midxed_git_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n \ndiff --git a/object-store.h b/object-store.h\nindex 9b671f1b0a..7908d46e34 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -130,6 +130,13 @@ struct raw_object_store {\n \t */\n \tstruct oidmap *replace_map;\n \n+\t/*\n+\t * private data\n+\t *\n+\t * should only be accessed directly by packfile.c and midx.c\n+\t */\n+\tstruct midxed_git *midxed_git;\n+\n \t/*\n \t * private data\n \t *\ndiff --git a/packfile.c b/packfile.c\nindex 1a714fbde9..b91ca9b9f5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -15,6 +15,7 @@\n #include \"tree-walk.h\"\n #include \"tree.h\"\n #include \"object-store.h\"\n+#include \"midx.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -893,10 +894,13 @@ static void prepare_packed_git(struct repository *r)\n \n \tif (r->objects->packed_git_initialized)\n \t\treturn;\n+\tprepare_midxed_git_one(r, r->objects->objectdir);\n \tprepare_packed_git_one(r, r->objects->objectdir, 1);\n \tprepare_alt_odb(r);\n-\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next)\n+\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\tprepare_midxed_git_one(r, alt->path);\n \t\tprepare_packed_git_one(r, alt->path, 0);\n+\t}\n \trearrange_packed_git(r);\n \tprepare_packed_git_mru(r);\n \tr->objects->packed_git_initialized = 1;\n-- \n2.18.0.rc1\n\n"},{"id":"349620","messageId":"20180607140338.32440-12-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 11/23] midx: sort and deduplicate objects from packfiles","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:26Z","receivedAt":"2018-06-07T14:04:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before writing a list of objects and their offsets to a multi-pack-index\n(MIDX), we need to collect the list of objects contained in the\npackfiles. There may be multiple copies of some objects, so this list\nmust be deduplicated.\n\nIt is possible to artificially get into a state where there are many\nduplicate copies of objects. That can create high memory pressure if we\nare to create a list of all objects before de-duplication. To reduce\nthis memory pressure without a significant performance drop,\nautomatically group objects by the first byte of their object id. Use\nthe IDX fanout tables to group the data, copy to a local array, then\nsort.\n\nCopy only the de-duplicated entries. Select the duplicate based on the\nmost-recent modified time of a packfile containing the object.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 138 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 923acda72e..b20d52713c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -4,6 +4,7 @@\n #include \"csum-file.h\"\n #include \"lockfile.h\"\n #include \"object-store.h\"\n+#include \"packfile.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n@@ -190,6 +191,140 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n \t}\n }\n \n+static uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)\n+{\n+\tconst uint32_t *level1_ofs = p->index_data;\n+\n+\tif (!level1_ofs) {\n+\t\tif (open_pack_index(p))\n+\t\t\treturn 0;\n+\t\tlevel1_ofs = p->index_data;\n+\t}\n+\n+\tif (p->index_version > 1) {\n+\t\tlevel1_ofs += 2;\n+\t}\n+\n+\treturn ntohl(level1_ofs[value]);\n+}\n+\n+struct pack_midx_entry {\n+\tstruct object_id oid;\n+\tuint32_t pack_int_id;\n+\ttime_t pack_mtime;\n+\tuint64_t offset;\n+};\n+\n+static int midx_oid_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_midx_entry *a = (struct pack_midx_entry *)_a;\n+\tstruct pack_midx_entry *b = (struct pack_midx_entry *)_b;\n+\tint cmp = oidcmp(&a->oid, &b->oid);\n+\n+\tif (cmp)\n+\t\treturn cmp;\n+\n+\tif (a->pack_mtime > b->pack_mtime)\n+\t\treturn -1;\n+\telse if (a->pack_mtime < b->pack_mtime)\n+\t\treturn 1;\n+\n+\treturn a->pack_int_id - b->pack_int_id;\n+}\n+\n+static void fill_pack_entry(uint32_t pack_int_id,\n+\t\t\t struct packed_git *p,\n+\t\t\t uint32_t cur_object,\n+\t\t\t struct pack_midx_entry *entry)\n+{\n+\tif (!nth_packed_object_oid(&entry->oid, p, cur_object))\n+\t\tdie(\"failed to located object %d in packfile\", cur_object);\n+\n+\tentry->pack_int_id = pack_int_id;\n+\tentry->pack_mtime = p->mtime;\n+\n+\tentry->offset = nth_packed_object_offset(p, cur_object);\n+}\n+\n+/*\n+ * It is possible to artificially get into a state where there are many\n+ * duplicate copies of objects. That can create high memory pressure if\n+ * we are to create a list of all objects before de-duplication. To reduce\n+ * this memory pressure without a significant performance drop, automatically\n+ * group objects by the first byte of their object id. Use the IDX fanout\n+ * tables to group the data, copy to a local array, then sort.\n+ *\n+ * Copy only the de-duplicated entries (selected by most-recent modified time\n+ * of a packfile containing the object).\n+ */\n+static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n+\t\t\t\t\t\t uint32_t *perm,\n+\t\t\t\t\t\t uint32_t nr_packs,\n+\t\t\t\t\t\t uint32_t *nr_objects)\n+{\n+\tuint32_t cur_fanout, cur_pack, cur_object;\n+\tuint32_t nr_fanout, alloc_fanout, alloc_objects, total_objects = 0;\n+\tstruct pack_midx_entry *entries_by_fanout = NULL;\n+\tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\n+\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\tif (open_pack_index(p[cur_pack]))\n+\t\t\tcontinue;\n+\n+\t\ttotal_objects += p[cur_pack]->num_objects;\n+\t}\n+\n+\t/*\n+\t * As we de-duplicate by fanout value, we expect the fanout\n+\t * slices to be evenly distributed, with some noise. Hence,\n+\t * allocate slightly more than one 256th.\n+\t */\n+\talloc_objects = alloc_fanout = total_objects > 3200 ? total_objects / 200 : 16;\n+\n+\tALLOC_ARRAY(entries_by_fanout, alloc_fanout);\n+\tALLOC_ARRAY(deduplicated_entries, alloc_objects);\n+\t*nr_objects = 0;\n+\n+\tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n+\t\tnr_fanout = 0;\n+\n+\t\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\t\tuint32_t start = 0, end;\n+\n+\t\t\tif (cur_fanout)\n+\t\t\t\tstart = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n+\t\t\tend = get_pack_fanout(p[cur_pack], cur_fanout);\n+\n+\t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n+\t\t\t\tfill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tnr_fanout++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tQSORT(entries_by_fanout, nr_fanout, midx_oid_compare);\n+\n+\t\t/*\n+\t\t * The batch is now sorted by OID and then mtime (descending).\n+\t\t * Take only the first duplicate.\n+\t\t */\n+\t\tfor (cur_object = 0; cur_object < nr_fanout; cur_object++) {\n+\t\t\tif (cur_object && !oidcmp(&entries_by_fanout[cur_object - 1].oid,\n+\t\t\t\t\t\t &entries_by_fanout[cur_object].oid))\n+\t\t\t\tcontinue;\n+\n+\t\t\tALLOC_GROW(deduplicated_entries, *nr_objects + 1, alloc_objects);\n+\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n+\t\t\t &entries_by_fanout[cur_object],\n+\t\t\t sizeof(struct pack_midx_entry));\n+\t\t\t(*nr_objects)++;\n+\t\t}\n+\t}\n+\n+\tFREE_AND_NULL(entries_by_fanout);\n+\treturn deduplicated_entries;\n+}\n+\n static size_t write_midx_pack_lookup(struct hashfile *f,\n \t\t\t\t char **pack_names,\n \t\t\t\t uint32_t nr_packs)\n@@ -254,6 +389,7 @@ int write_midx_file(const char *object_dir)\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n+\tuint32_t nr_entries;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -312,6 +448,8 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(pack_perm, nr_packs);\n \tsort_packs_by_name(pack_names, nr_packs, pack_perm);\n \n+\tget_sorted_entries(packs, pack_perm, nr_packs, &nr_entries);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n-- \n2.18.0.rc1\n\n"},{"id":"349621","messageId":"20180607140338.32440-22-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 21/23] midx: prevent duplicate packfile loads","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:36Z","receivedAt":"2018-06-07T14:04:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If the multi-pack-index contains a packfile, then we do not need to add\nthat packfile to the packed_git linked list or the MRU list.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 23 +++++++++++++++++++++++\n midx.h | 1 +\n packfile.c | 7 +++++++\n 3 files changed, 31 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 388d79b7d9..3242646fe0 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -278,6 +278,29 @@ int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct mi\n \treturn nth_midxed_pack_entry(m, e, pos);\n }\n \n+int midx_contains_pack(struct midxed_git *m, const char *idx_name)\n+{\n+\tuint32_t first = 0, last = m->num_packs;\n+\n+\twhile (first < last) {\n+\t\tuint32_t mid = first + (last - first) / 2;\n+\t\tconst char *current;\n+\t\tint cmp;\n+\n+\t\tcurrent = m->pack_names[mid];\n+\t\tcmp = strcmp(idx_name, current);\n+\t\tif (!cmp)\n+\t\t\treturn 1;\n+\t\tif (cmp > 0) {\n+\t\t\tfirst = mid + 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlast = mid;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int prepare_midxed_git_one(struct repository *r, const char *object_dir)\n {\n \tstruct midxed_git *m = r->objects->midxed_git;\ndiff --git a/midx.h b/midx.h\nindex 497bdcc77c..c1db58d8c4 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -13,6 +13,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct midxed_git *m,\n \t\t\t\t\tuint32_t n);\n int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct midxed_git *m);\n+int midx_contains_pack(struct midxed_git *m, const char *idx_name);\n int prepare_midxed_git_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\ndiff --git a/packfile.c b/packfile.c\nindex 059b2aa097..479cb69b9f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -746,6 +746,11 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n \tDIR *dir;\n \tstruct dirent *de;\n \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n+\tstruct midxed_git *m = r->objects->midxed_git;\n+\n+\t/* look for the multi-pack-index for this object directory */\n+\twhile (m && strcmp(m->object_dir, objdir))\n+\t\tm = m->next;\n \n \tstrbuf_addstr(&path, objdir);\n \tstrbuf_addstr(&path, \"/pack\");\n@@ -772,6 +777,8 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n \t\tbase_len = path.len;\n \t\tif (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n \t\t\t/* Don't reopen a pack we already have. */\n+\t\t\tif (m && midx_contains_pack(m, de->d_name))\n+\t\t\t\tcontinue;\n \t\t\tfor (p = r->objects->packed_git; p;\n \t\t\t p = p->next) {\n \t\t\t\tsize_t len;\n-- \n2.18.0.rc1\n\n"},{"id":"349622","messageId":"20180607140338.32440-24-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 23/23] midx: clear midx on repack","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:38Z","receivedAt":"2018-06-07T14:04:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If a 'git repack' command replaces existing packfiles, then we must\nclear the existing multi-pack-index before moving the packfiles it\nreferences.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/repack.c | 8 ++++++++\n midx.c | 8 ++++++++\n midx.h | 1 +\n 3 files changed, 17 insertions(+)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 6c636e159e..66a7d8e8ea 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -8,6 +8,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"argv-array.h\"\n+#include \"midx.h\"\n \n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -174,6 +175,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n \tint local = 0;\n+\tint midx_cleared = 0;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -340,6 +342,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\t}\n \n+\t\t\tif (!midx_cleared) {\n+\t\t\t\t/* if we move a packfile, it will invalidated the midx */\n+\t\t\t\tclear_midx_file(get_object_directory());\n+\t\t\t\tmidx_cleared = 1;\n+\t\t\t}\n+\n \t\t\tfname_old = mkpathdup(\"%s/old-%s%s\", packdir,\n \t\t\t\t\t\titem->string, exts[ext].name);\n \t\t\tif (file_exists(fname_old))\ndiff --git a/midx.c b/midx.c\nindex e46f392fa4..1043c01fa7 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -913,3 +913,11 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(pack_names);\n \treturn 0;\n }\n+\n+void clear_midx_file(const char *object_dir)\n+{\n+\tchar *midx = get_midx_filename(object_dir);\n+\n+\tif (remove_path(midx))\n+\t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n+}\ndiff --git a/midx.h b/midx.h\nindex 6996b5ff6b..46f9f44c94 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -18,5 +18,6 @@ int midx_contains_pack(struct midxed_git *m, const char *idx_name);\n int prepare_midxed_git_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n+void clear_midx_file(const char *object_dir);\n \n #endif\n-- \n2.18.0.rc1\n\n"},{"id":"349623","messageId":"20180607140338.32440-23-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 22/23] midx: use midx to find ref-deltas","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:37Z","receivedAt":"2018-06-07T14:04:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 2 +-\n midx.h | 1 +\n packfile.c | 15 +++++++++++++++\n 3 files changed, 17 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3242646fe0..e46f392fa4 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -214,7 +214,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \treturn oid;\n }\n \n-static off_t nth_midxed_offset(struct midxed_git *m, uint32_t pos)\n+off_t nth_midxed_offset(struct midxed_git *m, uint32_t pos)\n {\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\ndiff --git a/midx.h b/midx.h\nindex c1db58d8c4..6996b5ff6b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -9,6 +9,7 @@\n \n struct midxed_git *load_midxed_git(const char *object_dir);\n int bsearch_midx(const struct object_id *oid, struct midxed_git *m, uint32_t *result);\n+off_t nth_midxed_offset(struct midxed_git *m, uint32_t n);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct midxed_git *m,\n \t\t\t\t\tuint32_t n);\ndiff --git a/packfile.c b/packfile.c\nindex 479cb69b9f..9b814c89c7 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1794,6 +1794,21 @@ off_t find_pack_entry_one(const unsigned char *sha1,\n \tuint32_t result;\n \n \tif (!index) {\n+\t\t/*\n+\t\t * If we have a MIDX, then we want to\n+\t\t * check the MIDX for the offset instead.\n+\t\t */\n+\t\tstruct midxed_git *m;\n+\n+\t\tfor (m = get_midxed_git(the_repository); m; m = m->next) {\n+\t\t\tif (midx_contains_pack(m, p->pack_name)) {\n+\t\t\t\tif (bsearch_midx(&oid, m, &result))\n+\t\t\t\t\treturn nth_midxed_offset(m, result);\n+\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (open_pack_index(p))\n \t\t\treturn 0;\n \t}\n-- \n2.18.0.rc1\n\n"},{"id":"349624","messageId":"20180607140338.32440-14-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 13/23] midx: write object id fanout chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:28Z","receivedAt":"2018-06-07T14:04:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 5 +++\n builtin/midx.c | 4 +-\n midx.c | 53 +++++++++++++++++++++++--\n object-store.h | 1 +\n t/t5319-midx.sh | 18 +++++----\n 5 files changed, 69 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex de9ac778b6..77e88f85e4 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -307,6 +307,11 @@ CHUNK DATA:\n \t name. This is the only chunk not guaranteed to be a multiple of four\n \t bytes in length, so should be the last chunk for alignment reasons.\n \n+\tOID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n+\t The ith entry, F[i], stores the number of OIDs with first\n+\t byte at most i. Thus F[255] stores the total\n+\t number of objects (N).\n+\n \tOID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n \t The OIDs for all objects in the MIDX are stored in lexicographic\n \t order in this chunk.\ndiff --git a/builtin/midx.c b/builtin/midx.c\nindex 86edd30174..e1fd0e0de4 100644\n--- a/builtin/midx.c\n+++ b/builtin/midx.c\n@@ -35,10 +35,12 @@ static int read_midx_file(const char *object_dir)\n \t\tprintf(\" pack_lookup\");\n \tif (m->chunk_pack_names)\n \t\tprintf(\" pack_names\");\n+\tif (m->chunk_oid_fanout)\n+\t\tprintf(\" oid_fanout\");\n \tif (m->chunk_oid_lookup)\n \t\tprintf(\" oid_lookup\");\n \n-\tprintf(\"\\n\");\n+\tprintf(\"\\nnum_objects: %d\\n\", m->num_objects);\n \n \tprintf(\"packs:\\n\");\n \tfor (i = 0; i < m->num_packs; i++)\ndiff --git a/midx.c b/midx.c\nindex d06bc6876a..9458ced208 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -14,12 +14,14 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 3\n+#define MIDX_MAX_CHUNKS 4\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKLOOKUP 0x504c4f4f /* \"PLOO\" */\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n+#define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n \n static char *get_midx_filename(const char *object_dir)\n {\n@@ -96,6 +98,10 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n+\t\t\t\tm->chunk_oid_fanout = (uint32_t *)(m->data + chunk_offset);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n \t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n \t\t\t\tbreak;\n@@ -117,9 +123,13 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \t\tdie(\"MIDX missing required pack lookup chunk\");\n \tif (!m->chunk_pack_names)\n \t\tdie(\"MIDX missing required pack-name chunk\");\n+\tif (!m->chunk_oid_fanout)\n+\t\tdie(\"MIDX missing required OID fanout chunk\");\n \tif (!m->chunk_oid_lookup)\n \t\tdie(\"MIDX missing required OID lookup chunk\");\n \n+\tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n+\n \tm->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n \tfor (i = 0; i < m->num_packs; i++) {\n \t\tif (i) {\n@@ -377,6 +387,35 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \treturn written;\n }\n \n+static size_t write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t struct pack_midx_entry *objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tstruct pack_midx_entry *last = objects + nr_objects;\n+\tuint32_t count = 0;\n+\tuint32_t i;\n+\n+\t/*\n+\t* Write the first-level table (the list is sorted,\n+\t* but we use a 256-entry lookup to be able to avoid\n+\t* having to do eight extra binary search iterations).\n+\t*/\n+\tfor (i = 0; i < 256; i++) {\n+\t\tstruct pack_midx_entry *next = list;\n+\n+\t\twhile (next < last && next->oid.hash[0] == i) {\n+\t\t\tcount++;\n+\t\t\tnext++;\n+\t\t}\n+\n+\t\thashwrite_be32(f, count);\n+\t\tlist = next;\n+\t}\n+\n+\treturn MIDX_CHUNK_FANOUT_SIZE;\n+}\n+\n static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n \t\t\t\t struct pack_midx_entry *objects,\n \t\t\t\t uint32_t nr_objects)\n@@ -489,7 +528,7 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 3;\n+\tnum_chunks = 4;\n \n \twritten = write_midx_header(f, num_chunks, nr_packs);\n \n@@ -501,9 +540,13 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_packs * sizeof(uint32_t);\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n \n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n+\n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = 0;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n@@ -541,6 +584,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_pack_names(f, pack_names, nr_packs);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n+\t\t\t\twritten += write_midx_oid_fanout(f, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n \t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n \t\t\t\tbreak;\ndiff --git a/object-store.h b/object-store.h\nindex 7d14d3586e..c613ff2571 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -102,6 +102,7 @@ struct midxed_git {\n \n \tconst uint32_t *chunk_pack_lookup;\n \tconst unsigned char *chunk_pack_names;\n+\tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \n \tconst char **pack_names;\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex e71aa52b80..d4ae988479 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -5,9 +5,11 @@ test_description='multi-pack-indexes'\n \n midx_read_expect() {\n \tNUM_PACKS=$1\n+\tNUM_OBJECTS=$2\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 1 3 $NUM_PACKS\n-\tchunks: pack_lookup pack_names oid_lookup\n+\theader: 4d494458 1 1 4 $NUM_PACKS\n+\tchunks: pack_lookup pack_names oid_fanout oid_lookup\n+\tnum_objects: $NUM_OBJECTS\n \tpacks:\n \tEOF\n \tif [ $NUM_PACKS -ge 1 ]\n@@ -23,7 +25,7 @@ test_expect_success 'write midx with no packs' '\n \tgit midx --object-dir=. write &&\n \ttest_when_finished rm pack/multi-pack-index &&\n \ttest_path_is_file pack/multi-pack-index &&\n-\tmidx_read_expect 0\n+\tmidx_read_expect 0 0\n '\n \n test_expect_success 'create objects' '\n@@ -54,18 +56,18 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test wide_delta_$iii &&\n@@ -92,7 +94,7 @@ test_expect_success 'write midx with two packs' '\n \tpack1=$(git pack-objects --index-version=1 pack/test-1 \n---\n Documentation/technical/pack-format.txt | 15 +++-\n builtin/midx.c | 4 +\n midx.c | 100 +++++++++++++++++++++++-\n object-store.h | 2 +\n t/t5319-midx.sh | 45 ++++++++---\n 5 files changed, 151 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 77e88f85e4..0256cfb5e0 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -316,7 +316,20 @@ CHUNK DATA:\n \t The OIDs for all objects in the MIDX are stored in lexicographic\n \t order in this chunk.\n \n-\t(This section intentionally left incomplete.)\n+\tObject Offsets (ID: {'O', 'O', 'F', 'F'}) (N * 8 bytes)\n+\t Stores two 4-byte values for every object.\n+\t 1: The pack-int-id for the pack storing this object.\n+\t 2: The offset within the pack.\n+\t\tIf all offsets are less than 2^31, then the large offset chunk\n+\t\twill not exist and offsets are stored as in IDX v1.\n+\t\tIf there is at least one offset value larger than 2^32-1, then\n+\t\tthe large offset chunk must exist. If the large offset chunk\n+\t\texists and the 31st bit is on, then removing that bit reveals\n+\t\tthe row in the large offsets containing the 8-byte offset of\n+\t\tthis object.\n+\n+\t[Optional] Object Large Offsets (ID: {'L', 'O', 'F', 'F'})\n+\t 8-byte offsets into large packfiles.\n \n TRAILER:\n \ndiff --git a/builtin/midx.c b/builtin/midx.c\nindex e1fd0e0de4..607d2b3544 100644\n--- a/builtin/midx.c\n+++ b/builtin/midx.c\n@@ -39,6 +39,10 @@ static int read_midx_file(const char *object_dir)\n \t\tprintf(\" oid_fanout\");\n \tif (m->chunk_oid_lookup)\n \t\tprintf(\" oid_lookup\");\n+\tif (m->chunk_object_offsets)\n+\t\tprintf(\" object_offsets\");\n+\tif (m->chunk_large_offsets)\n+\t\tprintf(\" large_offsets\");\n \n \tprintf(\"\\nnum_objects: %d\\n\", m->num_objects);\n \ndiff --git a/midx.c b/midx.c\nindex 9458ced208..a49300bf75 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -14,14 +14,19 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 4\n+#define MIDX_MAX_CHUNKS 6\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKLOOKUP 0x504c4f4f /* \"PLOO\" */\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n+#define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n+#define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n+#define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n+#define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n+#define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n static char *get_midx_filename(const char *object_dir)\n {\n@@ -106,6 +111,14 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n+\t\t\t\tm->chunk_object_offsets = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n+\t\t\t\tm->chunk_large_offsets = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n \t\t\tcase 0:\n \t\t\t\tdie(\"terminating MIDX chunk id appears earlier than expected\");\n \t\t\t\tbreak;\n@@ -127,6 +140,8 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \t\tdie(\"MIDX missing required OID fanout chunk\");\n \tif (!m->chunk_oid_lookup)\n \t\tdie(\"MIDX missing required OID lookup chunk\");\n+\tif (!m->chunk_object_offsets)\n+\t\tdie(\"MIDX missing required object offsets chunk\");\n \n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n@@ -442,6 +457,56 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n \treturn written;\n }\n \n+static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n+\t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tuint32_t i, nr_large_offset = 0;\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\thashwrite_be32(f, obj->pack_int_id);\n+\n+\t\tif (large_offset_needed && obj->offset >> 31)\n+\t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n+\t\telse if (!large_offset_needed && obj->offset >> 32)\n+\t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n+\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t obj->offset);\n+\t\telse\n+\t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n+\n+\t\twritten += MIDX_CHUNK_OFFSET_WIDTH;\n+\t}\n+\n+\treturn written;\n+}\n+\n+static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_offset,\n+\t\t\t\t struct pack_midx_entry *objects, uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tsize_t written = 0;\n+\n+\twhile (nr_large_offset) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\t\tuint64_t offset = obj->offset;\n+\n+\t\tif (!(offset >> 31))\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be32(f, offset >> 32);\n+\t\thashwrite_be32(f, offset & 0xffffffff);\n+\t\twritten += 2 * sizeof(uint32_t);\n+\n+\t\tnr_large_offset--;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n@@ -462,7 +527,8 @@ int write_midx_file(const char *object_dir)\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tstruct pack_midx_entry *entries;\n-\tuint32_t nr_entries;\n+\tuint32_t nr_entries, num_large_offsets = 0;\n+\tint large_offsets_needed = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -522,13 +588,19 @@ int write_midx_file(const char *object_dir)\n \tsort_packs_by_name(pack_names, nr_packs, pack_perm);\n \n \tentries = get_sorted_entries(packs, pack_perm, nr_packs, &nr_entries);\n+\tfor (i = 0; i < nr_entries; i++) {\n+\t\tif (entries[i].offset > 0x7fffffff)\n+\t\t\tnum_large_offsets++;\n+\t\tif (entries[i].offset > 0xffffffff)\n+\t\t\tlarge_offsets_needed = 1;\n+\t}\n \n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 4;\n+\tnum_chunks = large_offsets_needed ? 6 : 5;\n \n \twritten = write_midx_header(f, num_chunks, nr_packs);\n \n@@ -548,9 +620,21 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = 0;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n \n+\tcur_chunk++;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_CHUNK_OFFSET_WIDTH;\n+\tif (large_offsets_needed) {\n+\t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n+\n+\t\tcur_chunk++;\n+\t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n+\t\t\t\t\t num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n+\t}\n+\n+\tchunk_ids[cur_chunk] = 0;\n+\n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n \t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n@@ -592,6 +676,14 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n+\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n+\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tdefault:\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\ndiff --git a/object-store.h b/object-store.h\nindex c613ff2571..9b671f1b0a 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -104,6 +104,8 @@ struct midxed_git {\n \tconst unsigned char *chunk_pack_names;\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n+\tconst unsigned char *chunk_object_offsets;\n+\tconst unsigned char *chunk_large_offsets;\n \n \tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex d4ae988479..709652c635 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -6,18 +6,21 @@ test_description='multi-pack-indexes'\n midx_read_expect() {\n \tNUM_PACKS=$1\n \tNUM_OBJECTS=$2\n+\tNUM_CHUNKS=$3\n+\tOBJECT_DIR=$4\n+\tEXTRA_CHUNKS=\"$5\"\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 1 4 $NUM_PACKS\n-\tchunks: pack_lookup pack_names oid_fanout oid_lookup\n+\theader: 4d494458 1 1 $NUM_CHUNKS $NUM_PACKS\n+\tchunks: pack_lookup pack_names oid_fanout oid_lookup object_offsets$EXTRA_CHUNKS\n \tnum_objects: $NUM_OBJECTS\n \tpacks:\n \tEOF\n \tif [ $NUM_PACKS -ge 1 ]\n \tthen\n-\t\tls pack/ | grep idx | sort >> expect\n+\t\tls $OBJECT_DIR/pack/ | grep idx | sort >> expect\n \tfi\n-\tprintf \"object_dir: .\\n\" >>expect &&\n-\tgit midx read --object-dir=. >actual &&\n+\tprintf \"object_dir: $OBJECT_DIR\\n\" >>expect &&\n+\tgit midx read --object-dir=$OBJECT_DIR >actual &&\n \ttest_cmp expect actual\n }\n \n@@ -25,7 +28,7 @@ test_expect_success 'write midx with no packs' '\n \tgit midx --object-dir=. write &&\n \ttest_when_finished rm pack/multi-pack-index &&\n \ttest_path_is_file pack/multi-pack-index &&\n-\tmidx_read_expect 0 0\n+\tmidx_read_expect 0 0 5 .\n '\n \n test_expect_success 'create objects' '\n@@ -56,14 +59,14 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test []\n+corrupt_data() {\n+\tfile=$1\n+\tpos=$2\n+\tdata=\"${3:-\\0}\"\n+\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n+}\n+\n+# Force 64-bit offsets by manipulating the idx file.\n+# This makes the IDX file _incorrect_ so be careful to clean up after!\n+test_expect_success 'force some 64-bit offsets with pack-objects' '\n+\tmkdir objects64 &&\n+\tmkdir objects64/pack &&\n+\tpack64=$(git pack-objects --index-version=2,0x40 objects64/pack/test-64 \n---\n packfile.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 638e113972..059b2aa097 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -819,11 +819,14 @@ unsigned long approximate_object_count(void)\n {\n \tif (!the_repository->objects->approximate_object_count_valid) {\n \t\tunsigned long count;\n+\t\tstruct midxed_git *m;\n \t\tstruct packed_git *p;\n \n \t\tprepare_packed_git(the_repository);\n \t\tcount = 0;\n-\t\tfor (p = the_repository->objects->packed_git; p; p = p->next) {\n+\t\tfor (m = get_midxed_git(the_repository); m; m = m->next)\n+\t\t\tcount += m->num_objects;\n+\t\tfor (p = get_packed_git(the_repository); p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n \t\t\tcount += p->num_objects;\n-- \n2.18.0.rc1\n\n"},{"id":"349627","messageId":"20180607140338.32440-20-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 19/23] midx: use existing midx when writing new one","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:34Z","receivedAt":"2018-06-07T14:04:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 63 insertions(+), 5 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 25d8142c2a..388d79b7d9 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -389,6 +389,23 @@ static int midx_oid_compare(const void *_a, const void *_b)\n \treturn a->pack_int_id - b->pack_int_id;\n }\n \n+static int nth_midxed_pack_midx_entry(struct midxed_git *m,\n+\t\t\t\t uint32_t *pack_perm,\n+\t\t\t\t struct pack_midx_entry *e,\n+\t\t\t\t uint32_t pos)\n+{\n+\tif (pos >= m->num_objects)\n+\t\treturn 1;\n+\n+\tnth_midxed_object_oid(&e->oid, m, pos);\n+\te->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];\n+\te->offset = nth_midxed_offset(m, pos);\n+\n+\t/* consider objects in midx to be from \"old\" packs */\n+\te->pack_mtime = 0;\n+\treturn 0;\n+}\n+\n static void fill_pack_entry(uint32_t pack_int_id,\n \t\t\t struct packed_git *p,\n \t\t\t uint32_t cur_object,\n@@ -414,7 +431,8 @@ static void fill_pack_entry(uint32_t pack_int_id,\n * Copy only the de-duplicated entries (selected by most-recent modified time\n * of a packfile containing the object).\n */\n-static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n+static struct pack_midx_entry *get_sorted_entries(struct midxed_git *m,\n+\t\t\t\t\t\t struct packed_git **p,\n \t\t\t\t\t\t uint32_t *perm,\n \t\t\t\t\t\t uint32_t nr_packs,\n \t\t\t\t\t\t uint32_t *nr_objects)\n@@ -423,8 +441,9 @@ static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n \tuint32_t nr_fanout, alloc_fanout, alloc_objects, total_objects = 0;\n \tstruct pack_midx_entry *entries_by_fanout = NULL;\n \tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\tuint32_t start_pack = m ? m->num_packs : 0;\n \n-\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\tif (open_pack_index(p[cur_pack]))\n \t\t\tcontinue;\n \n@@ -445,7 +464,23 @@ static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tnr_fanout = 0;\n \n-\t\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\tif (m) {\n+\t\t\tuint32_t start = 0, end;\n+\n+\t\t\tif (cur_fanout)\n+\t\t\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n+\t\t\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n+\n+\t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n+\t\t\t\tnth_midxed_pack_midx_entry(m, perm,\n+\t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n+\t\t\t\t\t\t\t cur_object);\n+\t\t\t\tnr_fanout++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\t\tuint32_t start = 0, end;\n \n \t\t\tif (cur_fanout)\n@@ -654,6 +689,7 @@ int write_midx_file(const char *object_dir)\n \tstruct pack_midx_entry *entries;\n \tuint32_t nr_entries, num_large_offsets = 0;\n \tint large_offsets_needed = 0;\n+\tstruct midxed_git *m = NULL;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -662,6 +698,8 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n+\tm = load_midxed_git(object_dir);\n+\n \tstrbuf_addf(&pack_dir, \"%s/pack\", object_dir);\n \tdir = opendir(pack_dir.buf);\n \n@@ -676,11 +714,27 @@ int write_midx_file(const char *object_dir)\n \tpack_dir_len = pack_dir.len;\n \tALLOC_ARRAY(packs, alloc_packs);\n \tALLOC_ARRAY(pack_names, alloc_pack_names);\n+\n+\tif (m) {\n+\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\tALLOC_GROW(packs, nr_packs + 1, alloc_packs);\n+\t\t\tALLOC_GROW(pack_names, nr_packs + 1, alloc_pack_names);\n+\n+\t\t\tpacks[nr_packs] = NULL;\n+\t\t\tpack_names[nr_packs] = xstrdup(m->pack_names[i]);\n+\t\t\tpack_name_concat_len += strlen(pack_names[nr_packs]) + 1;\n+\t\t\tnr_packs++;\n+\t\t}\n+\t}\n+\n \twhile ((de = readdir(dir)) != NULL) {\n \t\tif (is_dot_or_dotdot(de->d_name))\n \t\t\tcontinue;\n \n \t\tif (ends_with(de->d_name, \".idx\")) {\n+\t\t\tif (m && midx_contains_pack(m, de->d_name))\n+\t\t\t\tcontinue;\n+\n \t\t\tALLOC_GROW(packs, nr_packs + 1, alloc_packs);\n \t\t\tALLOC_GROW(pack_names, nr_packs + 1, alloc_pack_names);\n \n@@ -705,6 +759,9 @@ int write_midx_file(const char *object_dir)\n \tclosedir(dir);\n \tstrbuf_release(&pack_dir);\n \n+\tif (m && nr_packs == m->num_packs)\n+\t\tgoto cleanup;\n+\n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n@@ -712,7 +769,7 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(pack_perm, nr_packs);\n \tsort_packs_by_name(pack_names, nr_packs, pack_perm);\n \n-\tentries = get_sorted_entries(packs, pack_perm, nr_packs, &nr_entries);\n+\tentries = get_sorted_entries(m, packs, pack_perm, nr_packs, &nr_entries);\n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n@@ -823,7 +880,8 @@ int write_midx_file(const char *object_dir)\n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n \n-\tfor (i = 0; i < nr_packs; i++) {\n+cleanup:\n+\tfor (i = m ? m->num_packs : 0; i < nr_packs; i++) {\n \t\tclose_pack(packs[i]);\n \t\tFREE_AND_NULL(packs[i]);\n \t}\n-- \n2.18.0.rc1\n\n"},{"id":"349628","messageId":"20180607140338.32440-16-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 15/23] midx: create core.midx config setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:30Z","receivedAt":"2018-06-07T14:04:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The core.midx config setting controls the multi-pack-index (MIDX)\nfeature. If false, the setting will disable all reads from the\nmulti-pack-index file.\n\nAdd comparison commands in t5319-midx.sh to check typical Git behavior\nremains the same as the config setting is turned on and off. This\ncurrently includes 'git rev-list' and 'git log' commands to trigger\nseveral object database reads.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 4 +++\n cache.h | 1 +\n config.c | 5 ++++\n environment.c | 1 +\n t/t5319-midx.sh | 57 ++++++++++++++++++++++++++++++++--------\n 5 files changed, 57 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ab641bf5a9..e78150e452 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -908,6 +908,10 @@ core.commitGraph::\n \tEnable git commit graph feature. Allows reading from the\n \tcommit-graph file.\n \n+core.midx::\n+\tEnable multi-pack-index feature. Allows reading from the multi-\n+\tpack-index file.\n+\n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n \tlinkgit:git-read-tree[1] for more information.\ndiff --git a/cache.h b/cache.h\nindex 89a107a7f7..c7967f7643 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -814,6 +814,7 @@ extern char *git_replace_ref_base;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_commit_graph;\n+extern int core_midx;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex fbbf0f8e9f..0df3dbdf74 100644\n--- a/config.c\n+++ b/config.c\n@@ -1313,6 +1313,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.midx\")) {\n+\t\tcore_midx = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.sparsecheckout\")) {\n \t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 2a6de2330b..dcb4417604 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -67,6 +67,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_commit_graph;\n+int core_midx;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex 709652c635..1a50987778 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -3,6 +3,8 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n+objdir=.git/objects\n+\n midx_read_expect() {\n \tNUM_PACKS=$1\n \tNUM_OBJECTS=$2\n@@ -62,13 +64,42 @@ test_expect_success 'write midx with one v1 pack' '\n \tmidx_read_expect 1 17 5 .\n '\n \n+midx_git_two_modes() {\n+\tgit -c core.midx=false $1 >expect &&\n+\tgit -c core.midx=true $1 >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx() {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\"\n+\t'\n+}\n+\n test_expect_success 'write midx with one v2 pack' '\n-\tpack=$(git pack-objects --index-version=2,0x40 pack/test expect &&\n+\tgit -c core.midx=true $1 >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx() {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\"\n+\t'\n+}\n+\n+compare_results_with_midx \"one v2 pack\"\n+\n test_expect_success 'Add more objects' '\n \tfor i in `test_seq 6 10`\n \tdo\n@@ -94,12 +125,13 @@ test_expect_success 'Add more objects' '\n '\n \n test_expect_success 'write midx with two packs' '\n-\tpack1=$(git pack-objects --index-version=1 pack/test-1 obj-list &&\n \t\tgit update-ref HEAD $commit &&\n-\t\tgit pack-objects --index-version=2 pack/test-pack []\n corrupt_data() {\n-- \n2.18.0.rc1\n\n"},{"id":"349629","messageId":"20180607140338.32440-19-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 18/23] midx: use midx in abbreviation calculations","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:33Z","receivedAt":"2018-06-07T14:04:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 11 ++++++++\n midx.h | 3 +++\n packfile.c | 6 +++++\n packfile.h | 1 +\n sha1-name.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++\n t/t5319-midx.sh | 3 ++-\n 6 files changed, 93 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 6eca8f1b12..25d8142c2a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -203,6 +203,17 @@ int bsearch_midx(const struct object_id *oid, struct midxed_git *m, uint32_t *re\n \t\t\t MIDX_HASH_LEN, result);\n }\n \n+struct object_id *nth_midxed_object_oid(struct object_id *oid,\n+\t\t\t\t\tstruct midxed_git *m,\n+\t\t\t\t\tuint32_t n)\n+{\n+\tif (n >= m->num_objects)\n+\t\treturn NULL;\n+\n+\thashcpy(oid->hash, m->chunk_oid_lookup + m->hash_len * n);\n+\treturn oid;\n+}\n+\n static off_t nth_midxed_offset(struct midxed_git *m, uint32_t pos)\n {\n \tconst unsigned char *offset_data;\ndiff --git a/midx.h b/midx.h\nindex 0c66812229..497bdcc77c 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -9,6 +9,9 @@\n \n struct midxed_git *load_midxed_git(const char *object_dir);\n int bsearch_midx(const struct object_id *oid, struct midxed_git *m, uint32_t *result);\n+struct object_id *nth_midxed_object_oid(struct object_id *oid,\n+\t\t\t\t\tstruct midxed_git *m,\n+\t\t\t\t\tuint32_t n);\n int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct midxed_git *m);\n int prepare_midxed_git_one(struct repository *r, const char *object_dir);\n \ndiff --git a/packfile.c b/packfile.c\nindex 73f8cc28ee..638e113972 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -919,6 +919,12 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packed_git;\n }\n \n+struct midxed_git *get_midxed_git(struct repository *r)\n+{\n+\tprepare_packed_git(r);\n+\treturn r->objects->midxed_git;\n+}\n+\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\ndiff --git a/packfile.h b/packfile.h\nindex e0a38aba93..01e14b93fd 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -39,6 +39,7 @@ extern void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n+struct midxed_git *get_midxed_git(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/sha1-name.c b/sha1-name.c\nindex 60d9ef3c7e..d975a186c9 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -12,6 +12,7 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"repository.h\"\n+#include \"midx.h\"\n \n static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);\n \n@@ -149,6 +150,32 @@ static int match_sha(unsigned len, const unsigned char *a, const unsigned char *\n \treturn 1;\n }\n \n+static void unique_in_midx(struct midxed_git *m,\n+\t\t\t struct disambiguate_state *ds)\n+{\n+\tuint32_t num, i, first = 0;\n+\tconst struct object_id *current = NULL;\n+\tnum = m->num_objects;\n+\n+\tif (!num)\n+\t\treturn;\n+\n+\tbsearch_midx(&ds->bin_pfx, m, &first);\n+\n+\t/*\n+\t * At this point, \"first\" is the location of the lowest object\n+\t * with an object name that could match \"bin_pfx\". See if we have\n+\t * 0, 1 or more objects that actually match(es).\n+\t */\n+\tfor (i = first; i < num && !ds->ambiguous; i++) {\n+\t\tstruct object_id oid;\n+\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n+\t\tif (!match_sha(ds->len, ds->bin_pfx.hash, current->hash))\n+\t\t\tbreak;\n+\t\tupdate_candidates(ds, current);\n+\t}\n+}\n+\n static void unique_in_pack(struct packed_git *p,\n \t\t\t struct disambiguate_state *ds)\n {\n@@ -177,8 +204,12 @@ static void unique_in_pack(struct packed_git *p,\n \n static void find_short_packed_object(struct disambiguate_state *ds)\n {\n+\tstruct midxed_git *m;\n \tstruct packed_git *p;\n \n+\tfor (m = get_midxed_git(the_repository); m && !ds->ambiguous;\n+\t m = m->next)\n+\t\tunique_in_midx(m, ds);\n \tfor (p = get_packed_git(the_repository); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n@@ -527,6 +558,42 @@ static int extend_abbrev_len(const struct object_id *oid, void *cb_data)\n \treturn 0;\n }\n \n+static void find_abbrev_len_for_midx(struct midxed_git *m,\n+\t\t\t\t struct min_abbrev_data *mad)\n+{\n+\tint match = 0;\n+\tuint32_t num, first = 0;\n+\tstruct object_id oid;\n+\tconst struct object_id *mad_oid;\n+\n+\tif (!m->num_objects)\n+\t\treturn;\n+\n+\tnum = m->num_objects;\n+\tmad_oid = mad->oid;\n+\tmatch = bsearch_midx(mad_oid, m, &first);\n+\n+\t/*\n+\t * first is now the position in the packfile where we would insert\n+\t * mad->hash if it does not exist (or the position of mad->hash if\n+\t * it does exist). Hence, we consider a maximum of two objects\n+\t * nearby for the abbreviation length.\n+\t */\n+\tmad->init_len = 0;\n+\tif (!match) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t} else if (first < num - 1) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t}\n+\tif (first > 0) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t}\n+\tmad->init_len = mad->cur_len;\n+}\n+\n static void find_abbrev_len_for_pack(struct packed_git *p,\n \t\t\t\t struct min_abbrev_data *mad)\n {\n@@ -565,8 +632,11 @@ static void find_abbrev_len_for_pack(struct packed_git *p,\n \n static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n {\n+\tstruct midxed_git *m;\n \tstruct packed_git *p;\n \n+\tfor (m = get_midxed_git(the_repository); m; m = m->next)\n+\t\tfind_abbrev_len_for_midx(m, mad);\n \tfor (p = get_packed_git(the_repository); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex 1a50987778..e3873da7d6 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -94,7 +94,8 @@ compare_results_with_midx() {\n \tMSG=$1\n \ttest_expect_success \"check normal git operations: $MSG\" '\n \t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n-\t\tmidx_git_two_modes \"log --raw\"\n+\t\tmidx_git_two_modes \"log --raw\" &&\n+\t\tmidx_git_two_modes \"log --oneline\"\n \t'\n }\n \n-- \n2.18.0.rc1\n\n"},{"id":"349630","messageId":"20180607140338.32440-18-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 17/23] midx: read objects from multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:32Z","receivedAt":"2018-06-07T14:04:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++--\n midx.h | 2 ++\n object-store.h | 1 +\n packfile.c | 8 ++++-\n 4 files changed, 104 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 5e9290ca8f..6eca8f1b12 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"csum-file.h\"\n #include \"lockfile.h\"\n+#include \"sha1-lookup.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n #include \"midx.h\"\n@@ -64,7 +65,7 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \n \tm = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n \tstrcpy(m->object_dir, object_dir);\n-\tm->data = midx_map;\n+\tm->data = (const unsigned char*)midx_map;\n \n \tm->signature = get_be32(m->data);\n \tif (m->signature != MIDX_SIGNATURE) {\n@@ -145,7 +146,9 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n-\tm->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n+\tm->packs = xcalloc(m->num_packs, sizeof(*m->packs));\n+\n+\tALLOC_ARRAY(m->pack_names, m->num_packs);\n \tfor (i = 0; i < m->num_packs; i++) {\n \t\tif (i) {\n \t\t\tif (ntohl(m->chunk_pack_lookup[i]) <= ntohl(m->chunk_pack_lookup[i - 1])) {\n@@ -175,6 +178,95 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n \texit(1);\n }\n \n+static int prepare_midx_pack(struct midxed_git *m, uint32_t pack_int_id)\n+{\n+\tstruct strbuf pack_name = STRBUF_INIT;\n+\n+\tif (pack_int_id >= m->num_packs)\n+\t\tBUG(\"bad pack-int-id\");\n+\n+\tif (m->packs[pack_int_id])\n+\t\treturn 0;\n+\n+\tstrbuf_addstr(&pack_name, m->object_dir);\n+\tstrbuf_addstr(&pack_name, \"/pack/\");\n+\tstrbuf_addstr(&pack_name, m->pack_names[pack_int_id]);\n+\n+\tm->packs[pack_int_id] = add_packed_git(pack_name.buf, pack_name.len, 1);\n+\tstrbuf_release(&pack_name);\n+\treturn !m->packs[pack_int_id];\n+}\n+\n+int bsearch_midx(const struct object_id *oid, struct midxed_git *m, uint32_t *result)\n+{\n+\treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n+\t\t\t MIDX_HASH_LEN, result);\n+}\n+\n+static off_t nth_midxed_offset(struct midxed_git *m, uint32_t pos)\n+{\n+\tconst unsigned char *offset_data;\n+\tuint32_t offset32;\n+\n+\toffset_data = m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH;\n+\toffset32 = get_be32(offset_data + sizeof(uint32_t));\n+\n+\tif (m->chunk_large_offsets && offset32 & MIDX_LARGE_OFFSET_NEEDED) {\n+\t\tif (sizeof(offset32) < sizeof(uint64_t))\n+\t\t\tdie(_(\"multi-pack-index stores a 64-bit offset, but off_t is too small\"));\n+\n+\t\toffset32 ^= MIDX_LARGE_OFFSET_NEEDED;\n+\t\treturn get_be64(m->chunk_large_offsets + sizeof(uint64_t) * offset32);\n+\t}\n+\n+\treturn offset32;\n+}\n+\n+static uint32_t nth_midxed_pack_int_id(struct midxed_git *m, uint32_t pos)\n+{\n+\treturn get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);\n+}\n+\n+static int nth_midxed_pack_entry(struct midxed_git *m, struct pack_entry *e, uint32_t pos)\n+{\n+\tuint32_t pack_int_id;\n+\tstruct packed_git *p;\n+\n+\tif (pos >= m->num_objects)\n+\t\treturn 0;\n+\n+\tpack_int_id = nth_midxed_pack_int_id(m, pos);\n+\n+\tif (prepare_midx_pack(m, pack_int_id))\n+\t\tdie(_(\"error preparing packfile from multi-pack-index\"));\n+\tp = m->packs[pack_int_id];\n+\n+\t/*\n+\t* We are about to tell the caller where they can locate the\n+\t* requested object. We better make sure the packfile is\n+\t* still here and can be accessed before supplying that\n+\t* answer, as it may have been deleted since the MIDX was\n+\t* loaded!\n+\t*/\n+\tif (!is_pack_valid(p))\n+\t\treturn 0;\n+\n+\te->offset = nth_midxed_offset(m, pos);\n+\te->p = p;\n+\n+\treturn 1;\n+}\n+\n+int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct midxed_git *m)\n+{\n+\tuint32_t pos;\n+\n+\tif (!bsearch_midx(oid, m, &pos))\n+\t\treturn 0;\n+\n+\treturn nth_midxed_pack_entry(m, e, pos);\n+}\n+\n int prepare_midxed_git_one(struct repository *r, const char *object_dir)\n {\n \tstruct midxed_git *m = r->objects->midxed_git;\ndiff --git a/midx.h b/midx.h\nindex 793203fc4a..0c66812229 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -8,6 +8,8 @@\n #include \"repository.h\"\n \n struct midxed_git *load_midxed_git(const char *object_dir);\n+int bsearch_midx(const struct object_id *oid, struct midxed_git *m, uint32_t *result);\n+int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct midxed_git *m);\n int prepare_midxed_git_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\ndiff --git a/object-store.h b/object-store.h\nindex 7908d46e34..5af2a852bc 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -108,6 +108,7 @@ struct midxed_git {\n \tconst unsigned char *chunk_large_offsets;\n \n \tconst char **pack_names;\n+\tstruct packed_git **packs;\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/packfile.c b/packfile.c\nindex b91ca9b9f5..73f8cc28ee 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1857,11 +1857,17 @@ static int fill_pack_entry(const struct object_id *oid,\n int find_pack_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e)\n {\n \tstruct list_head *pos;\n+\tstruct midxed_git *m;\n \n \tprepare_packed_git(r);\n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packed_git && !r->objects->midxed_git)\n \t\treturn 0;\n \n+\tfor (m = r->objects->midxed_git; m; m = m->next) {\n+\t\tif (fill_midx_entry(oid, e, m))\n+\t\t\treturn 1;\n+\t}\n+\n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (fill_pack_entry(oid, e, p)) {\n-- \n2.18.0.rc1\n\n"},{"id":"349631","messageId":"20180607140338.32440-6-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 05/23] midx: write header information to lockfile","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:20Z","receivedAt":"2018-06-07T14:04:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As we begin writing the multi-pack-index format to disk, start with\nthe basics: the 12-byte header and the 20-byte checksum footer. Start\nwith these basics so we can add the rest of the format in small\nincrements.\n\nAs we implement the format, we will use a technique to check that our\ncomputed offsets within the multi-pack-index file match what we are\nactually writing. Each method that writes to the hashfile will return\nthe number of bytes written, and we will track that those values match\nour expectations.\n\nCurrently, write_midx_header() returns 12, but is not checked. We will\ncheck the return value in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++\n t/t5319-midx.sh | 5 +++--\n 2 files changed, 56 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 616af66b13..3e55422a21 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,9 +1,62 @@\n #include \"git-compat-util.h\"\n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"csum-file.h\"\n+#include \"lockfile.h\"\n #include \"midx.h\"\n \n+#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n+#define MIDX_VERSION 1\n+#define MIDX_HASH_VERSION 1 /* SHA-1 */\n+#define MIDX_HEADER_SIZE 12\n+\n+static char *get_midx_filename(const char *object_dir)\n+{\n+\tstruct strbuf midx_name = STRBUF_INIT;\n+\tstrbuf_addstr(&midx_name, object_dir);\n+\tstrbuf_addstr(&midx_name, \"/pack/multi-pack-index\");\n+\treturn strbuf_detach(&midx_name, NULL);\n+}\n+\n+static size_t write_midx_header(struct hashfile *f,\n+\t\t\t\tunsigned char num_chunks,\n+\t\t\t\tuint32_t num_packs)\n+{\n+\tchar byte_values[4];\n+\thashwrite_be32(f, MIDX_SIGNATURE);\n+\tbyte_values[0] = MIDX_VERSION;\n+\tbyte_values[1] = MIDX_HASH_VERSION;\n+\tbyte_values[2] = num_chunks;\n+\tbyte_values[3] = 0; /* unused */\n+\thashwrite(f, byte_values, sizeof(byte_values));\n+\thashwrite_be32(f, num_packs);\n+\n+\treturn MIDX_HEADER_SIZE;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n+\tunsigned char num_chunks = 0;\n+\tuint32_t num_packs = 0;\n+\tchar *midx_name;\n+\tstruct hashfile *f;\n+\tstruct lock_file lk;\n+\n+\tmidx_name = get_midx_filename(object_dir);\n+\tif (safe_create_leading_directories(midx_name)) {\n+\t\tUNLEAK(midx_name);\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t midx_name);\n+\t}\n+\n+\thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n+\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\tFREE_AND_NULL(midx_name);\n+\n+\twrite_midx_header(f, num_chunks, num_packs);\n+\n+\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tcommit_lock_file(&lk);\n+\n \treturn 0;\n }\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex a590137af7..80f9389837 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -3,8 +3,9 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n-test_expect_success 'write midx with no pakcs' '\n-\tgit midx --object-dir=. write\n+test_expect_success 'write midx with no packs' '\n+\tgit midx --object-dir=. write &&\n+\ttest_path_is_file pack/multi-pack-index\n '\n \n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349632","messageId":"20180607140338.32440-9-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 08/23] midx: read packfiles from pack directory","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:23Z","receivedAt":"2018-06-07T14:04:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When constructing a multi-pack-index file for a given object directory,\nread the files within the enclosed pack directory and find matches that\nend with \".idx\" and find the correct paired packfile using\nadd_packed_git().\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++--\n t/t5319-midx.sh | 15 ++++++++-------\n 2 files changed, 57 insertions(+), 9 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex fa18770f1d..9fb89c80a2 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -102,10 +102,15 @@ static size_t write_midx_header(struct hashfile *f,\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char num_chunks = 0;\n-\tuint32_t num_packs = 0;\n \tchar *midx_name;\n \tstruct hashfile *f;\n \tstruct lock_file lk;\n+\tstruct packed_git **packs = NULL;\n+\tuint32_t i, nr_packs = 0, alloc_packs = 0;\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tstruct strbuf pack_dir = STRBUF_INIT;\n+\tsize_t pack_dir_len;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -114,14 +119,56 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n+\tstrbuf_addf(&pack_dir, \"%s/pack\", object_dir);\n+\tdir = opendir(pack_dir.buf);\n+\n+\tif (!dir) {\n+\t\terror_errno(\"unable to open pack directory: %s\",\n+\t\t\t pack_dir.buf);\n+\t\tstrbuf_release(&pack_dir);\n+\t\treturn 1;\n+\t}\n+\n+\tstrbuf_addch(&pack_dir, '/');\n+\tpack_dir_len = pack_dir.len;\n+\tALLOC_ARRAY(packs, alloc_packs);\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tif (is_dot_or_dotdot(de->d_name))\n+\t\t\tcontinue;\n+\n+\t\tif (ends_with(de->d_name, \".idx\")) {\n+\t\t\tALLOC_GROW(packs, nr_packs + 1, alloc_packs);\n+\n+\t\t\tstrbuf_setlen(&pack_dir, pack_dir_len);\n+\t\t\tstrbuf_addstr(&pack_dir, de->d_name);\n+\n+\t\t\tpacks[nr_packs] = add_packed_git(pack_dir.buf,\n+\t\t\t\t\t\t\t pack_dir.len,\n+\t\t\t\t\t\t\t 0);\n+\t\t\tif (!packs[nr_packs])\n+\t\t\t\twarning(\"failed to add packfile '%s'\",\n+\t\t\t\t\tpack_dir.buf);\n+\t\t\telse\n+\t\t\t\tnr_packs++;\n+\t\t}\n+\t}\n+\tclosedir(dir);\n+\tstrbuf_release(&pack_dir);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\twrite_midx_header(f, num_chunks, num_packs);\n+\twrite_midx_header(f, num_chunks, nr_packs);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n \n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\tclose_pack(packs[i]);\n+\t\tFREE_AND_NULL(packs[i]);\n+\t}\n+\n+\tFREE_AND_NULL(packs);\n \treturn 0;\n }\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex 2c25a69744..abe545c7c4 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -4,8 +4,9 @@ test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n midx_read_expect() {\n+\tNUM_PACKS=$1\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 1 0 0\n+\theader: 4d494458 1 1 0 $NUM_PACKS\n \tobject_dir: .\n \tEOF\n \tgit midx read --object-dir=. >actual &&\n@@ -16,7 +17,7 @@ test_expect_success 'write midx with no packs' '\n \tgit midx --object-dir=. write &&\n \ttest_when_finished rm pack/multi-pack-index &&\n \ttest_path_is_file pack/multi-pack-index &&\n-\tmidx_read_expect\n+\tmidx_read_expect 0\n '\n \n test_expect_success 'create objects' '\n@@ -47,14 +48,14 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test obj-list &&\n \t\tgit update-ref HEAD $commit &&\n-\t\tgit pack-objects --index-version=2 test-pack \n---\n t/t5319-midx.sh | 101 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 101 insertions(+)\n\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex e78514d8e9..2c25a69744 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -14,8 +14,109 @@ midx_read_expect() {\n \n test_expect_success 'write midx with no packs' '\n \tgit midx --object-dir=. write &&\n+\ttest_when_finished rm pack/multi-pack-index &&\n \ttest_path_is_file pack/multi-pack-index &&\n \tmidx_read_expect\n '\n \n+test_expect_success 'create objects' '\n+\tfor i in `test_seq 1 5`\n+\tdo\n+\t\tiii=$(printf '%03i' $i)\n+\t\ttest-tool genrandom \"bar\" 200 > wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\ti=$(expr $i + 1) || return 1\n+\tdone &&\n+\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree obj-list &&\n+\tgit update-ref HEAD $commit\n+'\n+\n+test_expect_success 'write midx with one v1 pack' '\n+\tpack=$(git pack-objects --index-version=1 pack/test wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\ti=$(expr $i + 1) || return 1\n+\tdone &&\n+\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree -p HEADobj-list2 &&\n+\tgit update-ref HEAD $commit\n+'\n+\n+test_expect_success 'write midx with two packs' '\n+\tpack1=$(git pack-objects --index-version=1 pack/test-1 wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\t\tgit update-index --add file_101 &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcommit=$(git commit-tree $tree -p HEADobj-list &&\n+\t\tgit update-ref HEAD $commit &&\n+\t\tgit pack-objects --index-version=2 test-pack \n---\n Documentation/git-midx.txt | 11 +++++++\n builtin/midx.c | 23 +++++++++++++-\n midx.c | 65 ++++++++++++++++++++++++++++++++++++++\n midx.h | 9 ++++++\n object-store.h | 19 +++++++++++\n t/t5319-midx.sh | 12 ++++++-\n 6 files changed, 137 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-midx.txt b/Documentation/git-midx.txt\nindex dcaeb1a91b..919283fdd8 100644\n--- a/Documentation/git-midx.txt\n+++ b/Documentation/git-midx.txt\n@@ -23,6 +23,11 @@ OPTIONS\n \t/packs/multi-pack-index for the current MIDX file, and\n \t/packs for the pack-files to index.\n \n+read::\n+\tWhen given as the verb, read the current MIDX file and output\n+\tbasic information about its contents. Used for debugging\n+\tpurposes only.\n+\n write::\n \tWhen given as the verb, write a new MIDX file to\n \t/packs/multi-pack-index.\n@@ -43,6 +48,12 @@ $ git midx write\n $ git midx --object-dir write\n -------------------------------------------\n \n+* Read the MIDX file in the .git/objects folder.\n++\n+-------------------------------------------\n+$ git midx read\n+-------------------------------------------\n+\n \n GIT\n ---\ndiff --git a/builtin/midx.c b/builtin/midx.c\nindex dc0a5acd3f..c7002f664a 100644\n--- a/builtin/midx.c\n+++ b/builtin/midx.c\n@@ -6,7 +6,7 @@\n #include \"midx.h\"\n \n static char const * const builtin_midx_usage[] ={\n-\tN_(\"git midx [--object-dir ] [write]\"),\n+\tN_(\"git midx [--object-dir ] [read|write]\"),\n \tNULL\n };\n \n@@ -14,6 +14,25 @@ static struct opts_midx {\n \tconst char *object_dir;\n } opts;\n \n+static int read_midx_file(const char *object_dir)\n+{\n+\tstruct midxed_git *m = load_midxed_git(object_dir);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tprintf(\"header: %08x %d %d %d %d\\n\",\n+\t m->signature,\n+\t m->version,\n+\t m->hash_version,\n+\t m->num_chunks,\n+\t m->num_packs);\n+\n+\tprintf(\"object_dir: %s\\n\", m->object_dir);\n+\n+\treturn 0;\n+}\n+\n int cmd_midx(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_midx_options[] = {\n@@ -38,6 +57,8 @@ int cmd_midx(int argc, const char **argv, const char *prefix)\n \tif (argc == 0)\n \t\treturn 0;\n \n+\tif (!strcmp(argv[0], \"read\"))\n+\t\treturn read_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"write\"))\n \t\treturn write_midx_file(opts.object_dir);\n \ndiff --git a/midx.c b/midx.c\nindex 3e55422a21..fa18770f1d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -3,12 +3,15 @@\n #include \"dir.h\"\n #include \"csum-file.h\"\n #include \"lockfile.h\"\n+#include \"object-store.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n #define MIDX_HASH_VERSION 1 /* SHA-1 */\n #define MIDX_HEADER_SIZE 12\n+#define MIDX_HASH_LEN 20\n+#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n static char *get_midx_filename(const char *object_dir)\n {\n@@ -18,6 +21,68 @@ static char *get_midx_filename(const char *object_dir)\n \treturn strbuf_detach(&midx_name, NULL);\n }\n \n+struct midxed_git *load_midxed_git(const char *object_dir)\n+{\n+\tstruct midxed_git *m;\n+\tint fd;\n+\tstruct stat st;\n+\tsize_t midx_size;\n+\tvoid *midx_map;\n+\tconst char *midx_name = get_midx_filename(object_dir);\n+\n+\tfd = git_open(midx_name);\n+\tif (fd < 0)\n+\t\treturn NULL;\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn NULL;\n+\t}\n+\tmidx_size = xsize_t(st.st_size);\n+\n+\tif (midx_size < MIDX_MIN_SIZE) {\n+\t\tclose(fd);\n+\t\tdie(\"multi-pack-index file %s is too small\", midx_name);\n+\t}\n+\n+\tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\n+\tm = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n+\tstrcpy(m->object_dir, object_dir);\n+\tm->data = midx_map;\n+\n+\tm->signature = get_be32(m->data);\n+\tif (m->signature != MIDX_SIGNATURE) {\n+\t\terror(\"multi-pack-index signature %X does not match signature %X\",\n+\t\t m->signature, MIDX_SIGNATURE);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tm->version = *(m->data + 4);\n+\tif (m->version != MIDX_VERSION) {\n+\t\terror(\"multi-pack-index version %d not recognized\",\n+\t\t m->version);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tm->hash_version = *(m->data + 5);\n+\tif (m->hash_version != MIDX_HASH_VERSION) {\n+\t\terror(\"hash version %d not recognized\", m->hash_version);\n+\t\tgoto cleanup_fail;\n+\t}\n+\tm->hash_len = MIDX_HASH_LEN;\n+\n+\tm->num_chunks = *(m->data + 6);\n+\tm->num_packs = get_be32(m->data + 8);\n+\n+\treturn m;\n+\n+cleanup_fail:\n+\tFREE_AND_NULL(m);\n+\tmunmap(midx_map, midx_size);\n+\tclose(fd);\n+\texit(1);\n+}\n+\n static size_t write_midx_header(struct hashfile *f,\n \t\t\t\tunsigned char num_chunks,\n \t\t\t\tuint32_t num_packs)\ndiff --git a/midx.h b/midx.h\nindex 3a63673952..a1d18ed991 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -1,4 +1,13 @@\n+#ifndef MIDX_H\n+#define MIDX_H\n+\n+#include \"git-compat-util.h\"\n #include \"cache.h\"\n+#include \"object-store.h\"\n #include \"packfile.h\"\n \n+struct midxed_git *load_midxed_git(const char *object_dir);\n+\n int write_midx_file(const char *object_dir);\n+\n+#endif\ndiff --git a/object-store.h b/object-store.h\nindex d683112fd7..77cb82621a 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -84,6 +84,25 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+struct midxed_git {\n+\tstruct midxed_git *next;\n+\n+\tint fd;\n+\n+\tconst unsigned char *data;\n+\tsize_t data_len;\n+\n+\tuint32_t signature;\n+\tunsigned char version;\n+\tunsigned char hash_version;\n+\tunsigned char hash_len;\n+\tunsigned char num_chunks;\n+\tuint32_t num_packs;\n+\tuint32_t num_objects;\n+\n+\tchar object_dir[FLEX_ARRAY];\n+};\n+\n struct raw_object_store {\n \t/*\n \t * Path to the repository's object store.\ndiff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\nindex 80f9389837..e78514d8e9 100755\n--- a/t/t5319-midx.sh\n+++ b/t/t5319-midx.sh\n@@ -3,9 +3,19 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n+midx_read_expect() {\n+\tcat >expect <<- EOF\n+\theader: 4d494458 1 1 0 0\n+\tobject_dir: .\n+\tEOF\n+\tgit midx read --object-dir=. >actual &&\n+\ttest_cmp expect actual\n+}\n+\n test_expect_success 'write midx with no packs' '\n \tgit midx --object-dir=. write &&\n-\ttest_path_is_file pack/multi-pack-index\n+\ttest_path_is_file pack/multi-pack-index &&\n+\tmidx_read_expect\n '\n \n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349635","messageId":"20180607140338.32440-3-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH 02/23] midx: add midx format details to pack-format.txt","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:03:17Z","receivedAt":"2018-06-07T14:04:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index (MIDX) feature generalizes the existing pack-\nindex (IDX) feature by indexing objects across multiple pack-files.\n\nDescribe the basic file format, using a 12-byte header followed by\na lookup table for a list of \"chunks\" which will be described later.\nThe file ends with a footer containing a checksum using the hash\nalgorithm.\n\nThe header allows later versions to create breaking changes by\nadvancing the version number. We can also change the hash algorithm\nusing a different version value.\n\nWe will add the individual chunk format information as we introduce\nthe code that writes that information.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 49 +++++++++++++++++++++++++\n 1 file changed, 49 insertions(+)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 70a99fd142..17666b4bfc 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -252,3 +252,52 @@ Pack file entry: <+\n corresponding packfile.\n \n 20-byte SHA-1-checksum of all of the above.\n+\n+== midx-*.midx files have the following format:\n+\n+The meta-index files refer to multiple pack-files and loose objects.\n+\n+In order to allow extensions that add extra data to the MIDX, we organize\n+the body into \"chunks\" and provide a lookup table at the beginning of the\n+body. The header includes certain length values, such as the number of packs,\n+the number of base MIDX files, hash lengths and types.\n+\n+All 4-byte numbers are in network order.\n+\n+HEADER:\n+\n+\t4-byte signature:\n+\t The signature is: {'M', 'I', 'D', 'X'}\n+\n+\t1-byte version number:\n+\t Git only writes or recognizes version 1\n+\n+\t1-byte Object Id Version\n+\t Git only writes or recognizes verion 1 (SHA-1)\n+\n+\t1-byte number (C) of \"chunks\"\n+\n+\t1-byte number (I) of base multi-pack-index files:\n+\t This value is currently always zero.\n+\n+\t4-byte number (P) of pack files\n+\n+CHUNK LOOKUP:\n+\n+\t(C + 1) * 12 bytes providing the chunk offsets:\n+\t First 4 bytes describe chunk id. Value 0 is a terminating label.\n+\t Other 8 bytes provide offset in current file for chunk to start.\n+\t (Chunks are provided in file-order, so you can infer the length\n+\t using the next chunk position if necessary.)\n+\n+\tThe remaining data in the body is described one chunk at a time, and\n+\tthese chunks may be given in any order. Chunks are required unless\n+\totherwise specified.\n+\n+CHUNK DATA:\n+\n+\t(This section intentionally left incomplete.)\n+\n+TRAILER:\n+\n+\tH-byte HASH-checksum of all of the above.\n-- \n2.18.0.rc1\n\n"},{"id":"349636","messageId":"e4542116-f48a-c5c2-de8d-74c53de46ae0@gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"Re: [PATCH 00/23] Multi-pack-index (MIDX)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:06:42Z","receivedAt":"2018-06-07T14:06:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/7/2018 10:03 AM, Derrick Stolee wrote:\n> This patch series includes a rewrite of the previous\n> multi-pack-index RFC [1] using the feedback from the\n> commit-graph feature.\n\nSorry to everyone who got a duplicate copy of this series. I misspelled \n'kernel.org' and it didn't go to the list.\n\nI also have this series available as a GitHub PR [1]\n\n[1] https://github.com/derrickstolee/git/pull/7\n\n"},{"id":"349640","messageId":"877enazmdd.fsf@evledraar.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"Re: [PATCH 00/23] Multi-pack-index (MIDX)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-06-07T14:45:18Z","receivedAt":"2018-06-07T14:45:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jun 07 2018, Derrick Stolee wrote:\n\n> To test the performance in this situation, I created a\n> script that organizes the Linux repository in a similar\n> fashion. I split the commit history into 50 parts by\n> creating branches on every 10,000 commits of the first-\n> parent history. Then, `git rev-list --objects A ^B`\n> provides the list of objects reachable from A but not B,\n> so I could send that to `git pack-objects` to create\n> these \"time-based\" packfiles. With these 50 packfiles\n> (deleting the old one from my fresh clone, and deleting\n> all tags as they were no longer on-disk) I could then\n> test 'git rev-list --objects HEAD^{tree}' and see:\n>\n> Before: 0.17s\n> After: 0.13s\n> % Diff: -23.5%\n>\n> By adding logic to count hits and misses to bsearch_pack,\n> I was able to see that the command above calls that\n> method 266,930 times with a hit rate of 33%. The MIDX\n> has the same number of calls with a 100% hit rate.\n\nDo you have the script you used for this? It would be very interesting\nas something we could stick in t/perf/ to test this use-case in the\nfuture.\n\nHow does this & the numbers below compare to just a naïve\n--max-pack-size= on linux.git?\n\nIs it possible for you to tar this test repo up and share it as a\none-off? I've been polishing the core.validateAbbrev series I have, and\nit would be interesting to compare some of the (abbrev) numbers.\n\n> Abbreviation Speedups\n> ---------------------\n>\n> To fully disambiguate an abbreviation, we must iterate\n> through all packfiles to ensure no collision exists in\n> any packfile. This requires O(P log N) time. With the\n> MIDX, this is only O(log N) time. Our standard test [2]\n> is 'git log --oneline --parents --raw' because it writes\n> many abbreviations while also doing a lot of other work\n> (walking commits and trees to compute the raw diff).\n>\n> For a copy of the Linux repository with 50 packfiles\n> split by time, we observed the following:\n>\n> Before: 100.5 s\n> After: 58.2 s\n> % Diff: -59.7%\n>\n>\n> Request for Review Attention\n> ----------------------------\n>\n> I tried my best to take the feedback from the commit-graph\n> feature and apply it to this feature. I also worked to\n> follow the object-store refactoring as I could. I also have\n> some local commits that create a 'verify' subcommand and\n> integrate with 'fsck' similar to the commit-graph, but I'll\n> leave those for a later series (and review is still underway\n> for that part of the commit-graph).\n>\n> One place where I could use some guidance is related to the\n> current state of 'the_hash_algo' patches. The file format\n> allows a different \"hash version\" which then indicates the\n> length of the hash. What's the best way to ensure this\n> feature doesn't cause extra pain in the hash-agnostic series?\n> This will inform how I go back and make the commit-graph\n> feature better in this area, too.\n>\n>\n> Thanks,\n> -Stolee\n"},{"id":"349646","messageId":"b81eedfe-e789-7ce3-cdd5-2754c8bbc40a@gmail.com","threadId":"48668","inReplyTo":"877enazmdd.fsf@evledraar.gmail.com","subject":"Re: [PATCH 00/23] Multi-pack-index (MIDX)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-07T14:54:13Z","receivedAt":"2018-06-07T15:30:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/7/2018 10:45 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Thu, Jun 07 2018, Derrick Stolee wrote:\n>\n>> To test the performance in this situation, I created a\n>> script that organizes the Linux repository in a similar\n>> fashion. I split the commit history into 50 parts by\n>> creating branches on every 10,000 commits of the first-\n>> parent history. Then, `git rev-list --objects A ^B`\n>> provides the list of objects reachable from A but not B,\n>> so I could send that to `git pack-objects` to create\n>> these \"time-based\" packfiles. With these 50 packfiles\n>> (deleting the old one from my fresh clone, and deleting\n>> all tags as they were no longer on-disk) I could then\n>> test 'git rev-list --objects HEAD^{tree}' and see:\n>>\n>> Before: 0.17s\n>> After: 0.13s\n>> % Diff: -23.5%\n>>\n>> By adding logic to count hits and misses to bsearch_pack,\n>> I was able to see that the command above calls that\n>> method 266,930 times with a hit rate of 33%. The MIDX\n>> has the same number of calls with a 100% hit rate.\n> Do you have the script you used for this? It would be very interesting\n> as something we could stick in t/perf/ to test this use-case in the\n> future.\n>\n> How does this & the numbers below compare to just a naïve\n> --max-pack-size= on linux.git?\n>\n> Is it possible for you to tar this test repo up and share it as a\n> one-off? I've been polishing the core.validateAbbrev series I have, and\n> it would be interesting to compare some of the (abbrev) numbers.\n\nHere is what I used. You will want to adjust your constants for whatever \nrepo you are using. This is for the Linux kernel which has a \nfirst-parent history of ~50,000 commits. It also leaves a bunch of extra \nfiles around, so it is nowhere near incorporating into the code.\n\n#!/bin/bash\n\nfor i in `seq 1 50`\ndo\n         ORDER=$((51 - $i))\n         NUM_BACK=$((1000 * ($i - 1)))\n         echo creating batch/$ORDER\n         git branch -f batch/$ORDER HEAD~$NUM_BACK\n         echo batch/$ORDER\n         git rev-parse batch/$ORDER\ndone\n\nlastbranch=\"\"\nfor i in `seq 1 50`\ndo\n         branch=batch/$i\n         if [$lastbranch -eq \"\"]\n         then\n                 echo \"$branch\"\n                 git rev-list --objects $branch | sed 's/ .*//' \n >objects-$i.txt\n         else\n                 echo \"$lastbranch\"\n                 echo \"$branch\"\n                 git rev-list --objects $branch ^$lastbranch | sed 's/ \n.*//' >objects-$i.txt\n         fi\n\n         git pack-objects --no-reuse-delta \n.git/objects/pack/branch-split2 wrote:\n> diff --git a/Documentation/git-midx.txt b/Documentation/git-midx.txt\n> new file mode 100644\n> index 0000000000..2bd886f1a2\n> --- /dev/null\n> +++ b/Documentation/git-midx.txt\n> @@ -0,0 +1,29 @@\n> +git-midx(1)\n> +============\n> +\n> +NAME\n> +----\n> +git-midx - Write and verify multi-pack-indexes (MIDX files).\n\nNo full stop. This head line is collected automatically with others\nand its having a full stop while the rest does not looks strange/\n\n> diff --git a/builtin/midx.c b/builtin/midx.c\n> new file mode 100644\n> index 0000000000..59ea92178f\n> --- /dev/null\n> +++ b/builtin/midx.c\n> @@ -0,0 +1,38 @@\n> +#include \"builtin.h\"\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"git-compat-util.h\"\n\nYou only need either cache.h or git-compat-util.h. If cache.h is here,\ngit-compat-util can be removed.\n\n> +#include \"parse-options.h\"\n> +\n> +static char const * const builtin_midx_usage[] ={\n> + N_(\"git midx [--object-dir ]\"),\n> + NULL\n> +};\n> +\n> +static struct opts_midx {\n> + const char *object_dir;\n> +} opts;\n> +\n> +int cmd_midx(int argc, const char **argv, const char *prefix)\n> +{\n> + static struct option builtin_midx_options[] = {\n> + { OPTION_STRING, 0, \"object-dir\", &opts.object_dir,\n\nFor paths (including dir), OPTION_FILENAME may be a better option to\nhandle correctly when the command is run in a subdir. See df217ed643\n(parse-opts: add OPT_FILENAME and transition builtins - 2009-05-23)\nfor more info.\n\n> + N_(\"dir\"),\n> + N_(\"The object directory containing set of packfile and pack-index pairs.\") },\n\nOther help strings do not have full stop either (I only checked a\ncouple commands though)\n\nAlso, doesn't OPT_STRING() work here too (if you avoid OPTION_FILENAME\nfor some reason)?\n\n> + OPT_END(),\n> + };\n> +\n> + if (argc == 2 && !strcmp(argv[1], \"-h\"))\n> + usage_with_options(builtin_midx_usage, builtin_midx_options);\n> +\n> + git_config(git_default_config, NULL);\n> +\n> + argc = parse_options(argc, argv, prefix,\n> + builtin_midx_options,\n> + builtin_midx_usage, 0);\n> +\n> + if (!opts.object_dir)\n> + opts.object_dir = get_object_directory();\n> +\n> + return 0;\n> +}\n\n> diff --git a/git.c b/git.c\n> index c2f48d53dd..400fadd677 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -503,6 +503,7 @@ static struct cmd_struct commands[] = {\n> { \"merge-recursive-theirs\", cmd_merge_recursive, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n> { \"merge-subtree\", cmd_merge_recursive, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n> { \"merge-tree\", cmd_merge_tree, RUN_SETUP | NO_PARSEOPT },\n> + { \"midx\", cmd_midx, RUN_SETUP },\n\nIf it's a plumbing and can take an --object-dir, then I don't think\nyou should require it to run in a repo (with RUN_SETUP).\nRUN_SETUP_GENTLY may be better. You could even leave it empty here and\nonly call setup_git_directory() only when --object-dir is not set.\n\n> { \"mktag\", cmd_mktag, RUN_SETUP | NO_PARSEOPT },\n> { \"mktree\", cmd_mktree, RUN_SETUP },\n> { \"mv\", cmd_mv, RUN_SETUP | NEED_WORK_TREE },\n-- \nDuy\n"},{"id":"349654","messageId":"CACsJy8DwNeYQUdJPnD694v3cEanq00f=zWz_bx53syJ47=ekyQ@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-5-dstolee@microsoft.com","subject":"Re: [PATCH 04/23] midx: add 'write' subcommand and basic wiring","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-07T17:27:35Z","receivedAt":"2018-06-07T17:28:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n> diff --git a/builtin/midx.c b/builtin/midx.c\n> index 59ea92178f..dc0a5acd3f 100644\n> --- a/builtin/midx.c\n> +++ b/builtin/midx.c\n> @@ -3,9 +3,10 @@\n> #include \"config.h\"\n> #include \"git-compat-util.h\"\n> #include \"parse-options.h\"\n> +#include \"midx.h\"\n>\n> static char const * const builtin_midx_usage[] ={\n> - N_(\"git midx [--object-dir ]\"),\n> + N_(\"git midx [--object-dir ] [write]\"),\n> NULL\n> };\n>\n> @@ -34,5 +35,11 @@ int cmd_midx(int argc, const char **argv, const char *prefix)\n> if (!opts.object_dir)\n> opts.object_dir = get_object_directory();\n>\n> + if (argc == 0)\n> + return 0;\n\nIsn't it better to die here when no verb is given? I don't see any\ngood use case for running a no-op \"git midx\" without verbs. It's more\nlikely a mistake (e.g. \"git midx $foo\" where foo happens to be empty)\n\n> +\n> + if (!strcmp(argv[0], \"write\"))\n> + return write_midx_file(opts.object_dir);\n> +\n> return 0;\n> }\n> diff --git a/midx.c b/midx.c\n> new file mode 100644\n> index 0000000000..616af66b13\n> --- /dev/null\n> +++ b/midx.c\n> @@ -0,0 +1,9 @@\n> +#include \"git-compat-util.h\"\n> +#include \"cache.h\"\n\nOnly one of the two is needed\n\n> +#include \"dir.h\"\n\nNot needed yet. It's better to include it in the patch that actually needs it.\n\n> +#include \"midx.h\"\n> +\n> +int write_midx_file(const char *object_dir)\n> +{\n> + return 0;\n> +}\n> diff --git a/midx.h b/midx.h\n> new file mode 100644\n> index 0000000000..3a63673952\n> --- /dev/null\n> +++ b/midx.h\n> @@ -0,0 +1,4 @@\n> +#include \"cache.h\"\n> +#include \"packfile.h\"\n\nThese includes are not needed, at least not now. And please protect\nthe header file with #ifndef __MINDX_H__ .. #endif.\n\n> +\n> +int write_midx_file(const char *object_dir);\n> diff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\n> new file mode 100755\n> index 0000000000..a590137af7\n> --- /dev/null\n> +++ b/t/t5319-midx.sh\n> @@ -0,0 +1,10 @@\n> +#!/bin/sh\n> +\n> +test_description='multi-pack-indexes'\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'write midx with no pakcs' '\n\nno packs\n\n\n> + git midx --object-dir=. write\n> +'\n> +\n> +test_done\n> --\n> 2.18.0.rc1\n>\n\n\n\n-- \nDuy\n"},{"id":"349655","messageId":"CACsJy8DZpe0=onuKxmM2NTN9SwKhwonv75E9T6vdB0S-WOdoDQ@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-6-dstolee@microsoft.com","subject":"Re: [PATCH 05/23] midx: write header information to lockfile","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-07T17:35:37Z","receivedAt":"2018-06-07T17:36:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n> +static char *get_midx_filename(const char *object_dir)\n> +{\n> + struct strbuf midx_name = STRBUF_INIT;\n> + strbuf_addstr(&midx_name, object_dir);\n> + strbuf_addstr(&midx_name, \"/pack/multi-pack-index\");\n> + return strbuf_detach(&midx_name, NULL);\n> +}\n\nI think this whole function can be written as\nxstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n\n> +\n> +static size_t write_midx_header(struct hashfile *f,\n> + unsigned char num_chunks,\n> + uint32_t num_packs)\n> +{\n> + char byte_values[4];\n\nunsigned char just to be on the safe side? 'char' is signed on ARM if\nI remember correctly.\n-- \nDuy\n"},{"id":"349656","messageId":"CACsJy8A3qVRXrQ9YiiN3Ggrcc5XFJWaWRrTbdO8E_48yRmzrqQ@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-7-dstolee@microsoft.com","subject":"Re: [PATCH 06/23] midx: struct midxed_git and 'read' subcommand","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-07T17:54:51Z","receivedAt":"2018-06-07T17:55:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n> As we build the multi-pack-index feature by adding chunks at a time,\n> we want to test that the data is being written correctly.\n>\n> Create struct midxed_git to store an in-memory representation of a\n\nA word play on 'packed_git'? Amusing. Some more descriptive name would\nbe better though. midxed looks almost like random letters thrown\ntogether.\n\n> multi-pack-index and a memory-map of the binary file. Initialize this\n> struct in load_midxed_git(object_dir).\n\n> +static int read_midx_file(const char *object_dir)\n> +{\n> + struct midxed_git *m = load_midxed_git(object_dir);\n> +\n> + if (!m)\n> + return 0;\n\nThis looks like an error case, please don't just return zero,\ntypically used to say \"success\". I don't know if this command stays\n\"for debugging purposes\" until the end. Of course in that case it does\nnot really matter.\n\n> +struct midxed_git *load_midxed_git(const char *object_dir)\n> +{\n> + struct midxed_git *m;\n> + int fd;\n> + struct stat st;\n> + size_t midx_size;\n> + void *midx_map;\n> + const char *midx_name = get_midx_filename(object_dir);\n\nmem leak? This function returns allocated memory if I remember correctly.\n\n> +\n> + fd = git_open(midx_name);\n> + if (fd < 0)\n> + return NULL;\n\ndo an error_errno() so we know what went wrong at least.\n\n> + if (fstat(fd, &st)) {\n> + close(fd);\n> + return NULL;\n\nsame here, we should know why fstat() fails.\n\n> + }\n> + midx_size = xsize_t(st.st_size);\n> +\n> + if (midx_size < MIDX_MIN_SIZE) {\n> + close(fd);\n> + die(\"multi-pack-index file %s is too small\", midx_name);\n\n_()\n\nThe use of die() should be discouraged though. Many people still try\n(or wish) to libify code and new die() does not help. I think error()\nhere would be enough then you can return NULL. Or you can go fancier\nand store the error string in a strbuf like refs code.\n\n> + }\n> +\n> + midx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n> +\n> + m = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n> + strcpy(m->object_dir, object_dir);\n> + m->data = midx_map;\n> +\n> + m->signature = get_be32(m->data);\n> + if (m->signature != MIDX_SIGNATURE) {\n> + error(\"multi-pack-index signature %X does not match signature %X\",\n> + m->signature, MIDX_SIGNATURE);\n\n_(). Maybe 0x%08x instead of %x\n\n> + goto cleanup_fail;\n> + }\n> +\n> + m->version = *(m->data + 4);\n\nm->data[4] instead? shorter and easier to understand.\n\nSame comment on \"*(m->data + x)\" and error() without _() for the rest.\n\n> + if (m->version != MIDX_VERSION) {\n> + error(\"multi-pack-index version %d not recognized\",\n> + m->version);\n\n_()\n\n> + goto cleanup_fail;\n> + }\n> +\n> + m->hash_version = *(m->data + 5);\n\nm->data[5]\n\n> +cleanup_fail:\n> + FREE_AND_NULL(m);\n> + munmap(midx_map, midx_size);\n> + close(fd);\n> + exit(1);\n\nIt's bad enough that you die() but exit() in this code seems too much.\nPlease just return NULL and let the caller handle the error.\n\n> diff --git a/midx.h b/midx.h\n> index 3a63673952..a1d18ed991 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -1,4 +1,13 @@\n> +#ifndef MIDX_H\n> +#define MIDX_H\n> +\n> +#include \"git-compat-util.h\"\n> #include \"cache.h\"\n> +#include \"object-store.h\"\n\nI don't really think you need object-store here (git-compat-util.h\ntoo). \"struct mixed_git;\" would be enough for load_midxed_git\ndeclaration below.\n\n> #include \"packfile.h\"\n>\n> +struct midxed_git *load_midxed_git(const char *object_dir);\n> +\n> int write_midx_file(const char *object_dir);\n> +\n> +#endif\n> diff --git a/object-store.h b/object-store.h\n> index d683112fd7..77cb82621a 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -84,6 +84,25 @@ struct packed_git {\n> char pack_name[FLEX_ARRAY]; /* more */\n> };\n>\n> +struct midxed_git {\n> + struct midxed_git *next;\n\nDo we really have multiple midx files?\n\n> +\n> + int fd;\n> +\n> + const unsigned char *data;\n> + size_t data_len;\n> +\n> + uint32_t signature;\n> + unsigned char version;\n> + unsigned char hash_version;\n> + unsigned char hash_len;\n> + unsigned char num_chunks;\n> + uint32_t num_packs;\n> + uint32_t num_objects;\n> +\n> + char object_dir[FLEX_ARRAY];\n\nWhy do you need to keep object_dir when it could be easily retrieved\nwhen the repo is available?\n\n> +};\n> +\n> struct raw_object_store {\n> /*\n> * Path to the repository's object store.\n-- \nDuy\n"},{"id":"349657","messageId":"CACsJy8BS1x2yqzongv7iPsBbQxxZ5fO4ZbwdX5BPrzePvpJB=Q@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-9-dstolee@microsoft.com","subject":"Re: [PATCH 08/23] midx: read packfiles from pack directory","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-07T18:03:04Z","receivedAt":"2018-06-07T18:03:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n> @@ -114,14 +119,56 @@ int write_midx_file(const char *object_dir)\n> midx_name);\n> }\n>\n> + strbuf_addf(&pack_dir, \"%s/pack\", object_dir);\n> + dir = opendir(pack_dir.buf);\n> +\n> + if (!dir) {\n> + error_errno(\"unable to open pack directory: %s\",\n> + pack_dir.buf);\n\n_()\n\n> + strbuf_release(&pack_dir);\n> + return 1;\n> + }\n> +\n> + strbuf_addch(&pack_dir, '/');\n> + pack_dir_len = pack_dir.len;\n> + ALLOC_ARRAY(packs, alloc_packs);\n> + while ((de = readdir(dir)) != NULL) {\n> + if (is_dot_or_dotdot(de->d_name))\n> + continue;\n> +\n> + if (ends_with(de->d_name, \".idx\")) {\n> + ALLOC_GROW(packs, nr_packs + 1, alloc_packs);\n> +\n> + strbuf_setlen(&pack_dir, pack_dir_len);\n> + strbuf_addstr(&pack_dir, de->d_name);\n> +\n> + packs[nr_packs] = add_packed_git(pack_dir.buf,\n> + pack_dir.len,\n> + 0);\n> + if (!packs[nr_packs])\n> + warning(\"failed to add packfile '%s'\",\n> + pack_dir.buf);\n> + else\n> + nr_packs++;\n> + }\n> + }\n> + closedir(dir);\n> + strbuf_release(&pack_dir);\n\nCan we refactor and share this scanning-for-packs code with\npackfile.c? I'm pretty sure it does something similar in there.\n\n> - write_midx_header(f, num_chunks, num_packs);\n> + write_midx_header(f, num_chunks, nr_packs);\n\nHmm.. could have stuck to one name from the beginning...\n-- \nDuy\n"},{"id":"349658","messageId":"CACsJy8ARDLeWaWq1_npZq5ARCVgcETpS--YUVLqZkZA52E7Msw@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-10-dstolee@microsoft.com","subject":"Re: [PATCH 09/23] midx: write pack names in chunk","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-07T18:26:52Z","receivedAt":"2018-06-07T18:27:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n> @@ -74,6 +80,31 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n> m->num_chunks = *(m->data + 6);\n> m->num_packs = get_be32(m->data + 8);\n>\n> + for (i = 0; i < m->num_chunks; i++) {\n> + uint32_t chunk_id = get_be32(m->data + 12 + MIDX_CHUNKLOOKUP_WIDTH * i);\n> + uint64_t chunk_offset = get_be64(m->data + 16 + MIDX_CHUNKLOOKUP_WIDTH * i);\n\nWould be good to reduce magic numbers like 12 and 16, I think you have\nsome header length constants for those already.\n\n> + switch (chunk_id) {\n> + case MIDX_CHUNKID_PACKNAMES:\n> + m->chunk_pack_names = m->data + chunk_offset;\n> + break;\n> +\n> + case 0:\n> + die(\"terminating MIDX chunk id appears earlier than expected\");\n\n_()\n\n> + break;\n> +\n> + default:\n> + /*\n> + * Do nothing on unrecognized chunks, allowing future\n> + * extensions to add optional chunks.\n> + */\n\nI wrote about the chunk term reminding me of PNG format then deleted\nit. But it may help to do similar to PNG here. The first letter can\nlet us know if the chunk is optional and can be safely ignored. E.g.\nuppercase first letter cannot be ignored, lowercase go wild.\n\n> + break;\n> + }\n> + }\n> +\n> + if (!m->chunk_pack_names)\n> + die(\"MIDX missing required pack-name chunk\");\n\n_()\n\n> +\n> return m;\n>\n> cleanup_fail:\n> @@ -99,18 +130,88 @@ static size_t write_midx_header(struct hashfile *f,\n> return MIDX_HEADER_SIZE;\n> }\n>\n> +struct pack_pair {\n> + uint32_t pack_int_id;\n\ncan this be just pack_id?\n\n> + char *pack_name;\n> +};\n> +\n> +static int pack_pair_compare(const void *_a, const void *_b)\n> +{\n> + struct pack_pair *a = (struct pack_pair *)_a;\n> + struct pack_pair *b = (struct pack_pair *)_b;\n> + return strcmp(a->pack_name, b->pack_name);\n> +}\n> +\n> +static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n> +{\n> + uint32_t i;\n> + struct pack_pair *pairs;\n> +\n> + ALLOC_ARRAY(pairs, nr_packs);\n> +\n> + for (i = 0; i < nr_packs; i++) {\n> + pairs[i].pack_int_id = i;\n> + pairs[i].pack_name = pack_names[i];\n> + }\n> +\n> + QSORT(pairs, nr_packs, pack_pair_compare);\n> +\n> + for (i = 0; i < nr_packs; i++) {\n> + pack_names[i] = pairs[i].pack_name;\n> + perm[pairs[i].pack_int_id] = i;\n> + }\n\npairs[] is leaked?\n\n> +}\n> +\n> +static size_t write_midx_pack_names(struct hashfile *f,\n> + char **pack_names,\n> + uint32_t num_packs)\n> +{\n> + uint32_t i;\n> + unsigned char padding[MIDX_CHUNK_ALIGNMENT];\n> + size_t written = 0;\n> +\n> + for (i = 0; i < num_packs; i++) {\n> + size_t writelen = strlen(pack_names[i]) + 1;\n> +\n> + if (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n> + BUG(\"incorrect pack-file order: %s before %s\",\n> + pack_names[i - 1],\n> + pack_names[i]);\n> +\n> + hashwrite(f, pack_names[i], writelen);\n> + written += writelen;\n\nside note. This pattern happens a lot. It may be a good idea to make\nhashwrite() return writelen so we can just write\n\nwritten += hashwrite(f, ..., writelen);\n\n> + }\n> +\n> + /* add padding to be aligned */\n> + i = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n> + if (i < MIDX_CHUNK_ALIGNMENT) {\n> + bzero(padding, sizeof(padding));\n> + hashwrite(f, padding, i);\n> + written += i;\n> + }\n> +\n> + return written;\n> +}\n> +\n> int write_midx_file(const char *object_dir)\n> {\n> - unsigned char num_chunks = 0;\n> + unsigned char cur_chunk, num_chunks = 0;\n> char *midx_name;\n> struct hashfile *f;\n> struct lock_file lk;\n> struct packed_git **packs = NULL;\n> + char **pack_names = NULL;\n> + uint32_t *pack_perm;\n> uint32_t i, nr_packs = 0, alloc_packs = 0;\n> + uint32_t alloc_pack_names = 0;\n> DIR *dir;\n> struct dirent *de;\n> struct strbuf pack_dir = STRBUF_INIT;\n> size_t pack_dir_len;\n> + uint64_t pack_name_concat_len = 0;\n> + uint64_t written = 0;\n> + uint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n> + uint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n\nThis long list of local vars may be a good indicator that this\nfunction needs split up into smaller ones.\n\n>\n> midx_name = get_midx_filename(object_dir);\n> if (safe_create_leading_directories(midx_name)) {\n> @@ -132,12 +233,14 @@ int write_midx_file(const char *object_dir)\n> strbuf_addch(&pack_dir, '/');\n> pack_dir_len = pack_dir.len;\n> ALLOC_ARRAY(packs, alloc_packs);\n> + ALLOC_ARRAY(pack_names, alloc_pack_names);\n> while ((de = readdir(dir)) != NULL) {\n> if (is_dot_or_dotdot(de->d_name))\n> continue;\n>\n> if (ends_with(de->d_name, \".idx\")) {\n> ALLOC_GROW(packs, nr_packs + 1, alloc_packs);\n> + ALLOC_GROW(pack_names, nr_packs + 1, alloc_pack_names);\n>\n> strbuf_setlen(&pack_dir, pack_dir_len);\n> strbuf_addstr(&pack_dir, de->d_name);\n> @@ -145,21 +248,83 @@ int write_midx_file(const char *object_dir)\n> packs[nr_packs] = add_packed_git(pack_dir.buf,\n> pack_dir.len,\n> 0);\n> - if (!packs[nr_packs])\n> + if (!packs[nr_packs]) {\n> warning(\"failed to add packfile '%s'\",\n> pack_dir.buf);\n> - else\n> - nr_packs++;\n> + continue;\n> + }\n> +\n> + pack_names[nr_packs] = xstrdup(de->d_name);\n> + pack_name_concat_len += strlen(de->d_name) + 1;\n> + nr_packs++;\n> }\n> }\n> +\n> closedir(dir);\n> strbuf_release(&pack_dir);\n>\n> + if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n> + pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n> + (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n> +\n> + ALLOC_ARRAY(pack_perm, nr_packs);\n> + sort_packs_by_name(pack_names, nr_packs, pack_perm);\n> +\n> hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n> f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n> FREE_AND_NULL(midx_name);\n>\n> - write_midx_header(f, num_chunks, nr_packs);\n> + cur_chunk = 0;\n> + num_chunks = 1;\n> +\n> + written = write_midx_header(f, num_chunks, nr_packs);\n> +\n> + chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n> + chunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n> +\n> + cur_chunk++;\n> + chunk_ids[cur_chunk] = 0;\n> + chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n> +\n> + for (i = 0; i <= num_chunks; i++) {\n> + if (i && chunk_offsets[i] < chunk_offsets[i - 1])\n> + BUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n> + chunk_offsets[i - 1],\n> + chunk_offsets[i]);\n> +\n> + if (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n> + BUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n> + chunk_offsets[i]);\n> +\n> + hashwrite_be32(f, chunk_ids[i]);\n> + hashwrite_be32(f, chunk_offsets[i] >> 32);\n> + hashwrite_be32(f, chunk_offsets[i]);\n> +\n> + written += MIDX_CHUNKLOOKUP_WIDTH;\n> + }\n> +\n> + for (i = 0; i < num_chunks; i++) {\n> + if (written != chunk_offsets[i])\n> + BUG(\"inccrrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n\nincorrect\n\n> + chunk_offsets[i],\n> + written,\n> + chunk_ids[i]);\n> +\n> + switch (chunk_ids[i]) {\n> + case MIDX_CHUNKID_PACKNAMES:\n> + written += write_midx_pack_names(f, pack_names, nr_packs);\n> + break;\n> +\n> + default:\n> + BUG(\"trying to write unknown chunk id %\"PRIx32,\n> + chunk_ids[i]);\n> + }\n> + }\n> +\n> + if (written != chunk_offsets[num_chunks])\n> + BUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n> + written,\n> + chunk_offsets[num_chunks]);\n>\n> finalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n> commit_lock_file(&lk);\n> @@ -170,5 +335,6 @@ int write_midx_file(const char *object_dir)\n> }\n>\n> FREE_AND_NULL(packs);\n> + FREE_AND_NULL(pack_names);\n\nWhat about the strings in this array? I think they are xstrdup() but I\ndidn't spot them being freed.\n\nAnd maybe just use string_list...\n\n> return 0;\n> }\n-- \nDuy\n"},{"id":"349659","messageId":"CACsJy8DG1-YTSBSFkpM8BPcnKmvC-BjjK0W0+EckkdENPC7Ohg@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-7-dstolee@microsoft.com","subject":"Re: [PATCH 06/23] midx: struct midxed_git and 'read' subcommand","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-07T18:31:05Z","receivedAt":"2018-06-07T18:31:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n> diff --git a/Documentation/git-midx.txt b/Documentation/git-midx.txt\n> index dcaeb1a91b..919283fdd8 100644\n> --- a/Documentation/git-midx.txt\n> +++ b/Documentation/git-midx.txt\n> @@ -23,6 +23,11 @@ OPTIONS\n> /packs/multi-pack-index for the current MIDX file, and\n> /packs for the pack-files to index.\n>\n> +read::\n> + When given as the verb, read the current MIDX file and output\n> + basic information about its contents. Used for debugging\n> + purposes only.\n\nOn second thought. If you just need a temporary debugging interface,\nadding a program in t/helper may be a better option. In the end we\nmight still need 'read' to dump a file out, but we should have some\nstable output format (and json might be a good choice).\n\nThat's it I'm done for today. I will continue on the rest some day, hopefully.\n-- \nDuy\n"},{"id":"349806","messageId":"CACsJy8C0K_3-=S-sV+jfKdRoF0Ta8eafbOT_B1zOfWCftwz8Bw@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-11-dstolee@microsoft.com","subject":"Re: [PATCH 10/23] midx: write a lookup into the pack names chunk","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T16:43:55Z","receivedAt":"2018-06-09T16:44:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 7:01 PM Derrick Stolee wrote:\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/pack-format.txt | 5 +++\n> builtin/midx.c | 7 ++++\n> midx.c | 56 +++++++++++++++++++++++--\n> object-store.h | 2 +\n> t/t5319-midx.sh | 11 +++--\n> 5 files changed, 75 insertions(+), 6 deletions(-)\n>\n> diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\n> index 2b37be7b33..29bf87283a 100644\n> --- a/Documentation/technical/pack-format.txt\n> +++ b/Documentation/technical/pack-format.txt\n> @@ -296,6 +296,11 @@ CHUNK LOOKUP:\n>\n> CHUNK DATA:\n>\n> + Packfile Name Lookup (ID: {'P', 'L', 'O', 'O'}) (P * 4 bytes)\n> + P * 4 bytes storing the offset in the packfile name chunk for\n> + the null-terminated string containing the filename for the\n> + ith packfile.\n> +\n\nCommit message is too light on this one. Why does this need to be\nstored? Isn't the cost of rebuilding this in-core cheap?\n\nAdding this chunk on disk in my opinion only adds more burden. Now you\nhave to verify that these offsets actually point to the right place.\n\n> Packfile Names (ID: {'P', 'N', 'A', 'M'})\n> Stores the packfile names as concatenated, null-terminated strings.\n> Packfiles must be listed in lexicographic order for fast lookups by\n> diff --git a/builtin/midx.c b/builtin/midx.c\n> index fe56560853..3a261e9bbf 100644\n> --- a/builtin/midx.c\n> +++ b/builtin/midx.c\n> @@ -16,6 +16,7 @@ static struct opts_midx {\n>\n> static int read_midx_file(const char *object_dir)\n> {\n> + uint32_t i;\n> struct midxed_git *m = load_midxed_git(object_dir);\n>\n> if (!m)\n> @@ -30,11 +31,17 @@ static int read_midx_file(const char *object_dir)\n>\n> printf(\"chunks:\");\n>\n> + if (m->chunk_pack_lookup)\n> + printf(\" pack_lookup\");\n> if (m->chunk_pack_names)\n> printf(\" pack_names\");\n>\n> printf(\"\\n\");\n>\n> + printf(\"packs:\\n\");\n> + for (i = 0; i < m->num_packs; i++)\n> + printf(\"%s\\n\", m->pack_names[i]);\n> +\n> printf(\"object_dir: %s\\n\", m->object_dir);\n>\n> return 0;\n> diff --git a/midx.c b/midx.c\n> index d4f4a01a51..923acda72e 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -13,8 +13,9 @@\n> #define MIDX_HASH_LEN 20\n> #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n>\n> -#define MIDX_MAX_CHUNKS 1\n> +#define MIDX_MAX_CHUNKS 2\n> #define MIDX_CHUNK_ALIGNMENT 4\n> +#define MIDX_CHUNKID_PACKLOOKUP 0x504c4f4f /* \"PLOO\" */\n> #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n> #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n>\n> @@ -85,6 +86,10 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n> uint64_t chunk_offset = get_be64(m->data + 16 + MIDX_CHUNKLOOKUP_WIDTH * i);\n>\n> switch (chunk_id) {\n> + case MIDX_CHUNKID_PACKLOOKUP:\n> + m->chunk_pack_lookup = (uint32_t *)(m->data + chunk_offset);\n> + break;\n> +\n> case MIDX_CHUNKID_PACKNAMES:\n> m->chunk_pack_names = m->data + chunk_offset;\n> break;\n> @@ -102,9 +107,32 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n> }\n> }\n>\n> + if (!m->chunk_pack_lookup)\n> + die(\"MIDX missing required pack lookup chunk\");\n> if (!m->chunk_pack_names)\n> die(\"MIDX missing required pack-name chunk\");\n>\n> + m->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n> + for (i = 0; i < m->num_packs; i++) {\n> + if (i) {\n> + if (ntohl(m->chunk_pack_lookup[i]) <= ntohl(m->chunk_pack_lookup[i - 1])) {\n> + error(\"MIDX pack lookup value %d before %d\",\n> + ntohl(m->chunk_pack_lookup[i - 1]),\n> + ntohl(m->chunk_pack_lookup[i]));\n> + goto cleanup_fail;\n> + }\n> + }\n> +\n> + m->pack_names[i] = (const char *)(m->chunk_pack_names + ntohl(m->chunk_pack_lookup[i]));\n> +\n> + if (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0) {\n> + error(\"MIDX pack names out of order: '%s' before '%s'\",\n> + m->pack_names[i - 1],\n> + m->pack_names[i]);\n> + goto cleanup_fail;\n> + }\n> + }\n> +\n> return m;\n>\n> cleanup_fail:\n> @@ -162,6 +190,20 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n> }\n> }\n>\n> +static size_t write_midx_pack_lookup(struct hashfile *f,\n> + char **pack_names,\n> + uint32_t nr_packs)\n> +{\n> + uint32_t i, cur_len = 0;\n> +\n> + for (i = 0; i < nr_packs; i++) {\n> + hashwrite_be32(f, cur_len);\n> + cur_len += strlen(pack_names[i]) + 1;\n> + }\n> +\n> + return sizeof(uint32_t) * (size_t)nr_packs;\n> +}\n> +\n> static size_t write_midx_pack_names(struct hashfile *f,\n> char **pack_names,\n> uint32_t num_packs)\n> @@ -275,13 +317,17 @@ int write_midx_file(const char *object_dir)\n> FREE_AND_NULL(midx_name);\n>\n> cur_chunk = 0;\n> - num_chunks = 1;\n> + num_chunks = 2;\n>\n> written = write_midx_header(f, num_chunks, nr_packs);\n>\n> - chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n> + chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKLOOKUP;\n> chunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n>\n> + cur_chunk++;\n> + chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n> + chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_packs * sizeof(uint32_t);\n> +\n> cur_chunk++;\n> chunk_ids[cur_chunk] = 0;\n> chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n> @@ -311,6 +357,10 @@ int write_midx_file(const char *object_dir)\n> chunk_ids[i]);\n>\n> switch (chunk_ids[i]) {\n> + case MIDX_CHUNKID_PACKLOOKUP:\n> + written += write_midx_pack_lookup(f, pack_names, nr_packs);\n> + break;\n> +\n> case MIDX_CHUNKID_PACKNAMES:\n> written += write_midx_pack_names(f, pack_names, nr_packs);\n> break;\n> diff --git a/object-store.h b/object-store.h\n> index 199cf4bd44..1ba50459ca 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -100,8 +100,10 @@ struct midxed_git {\n> uint32_t num_packs;\n> uint32_t num_objects;\n>\n> + const uint32_t *chunk_pack_lookup;\n> const unsigned char *chunk_pack_names;\n>\n> + const char **pack_names;\n> char object_dir[FLEX_ARRAY];\n> };\n>\n> diff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\n> index fdf4f84a90..a31c387c8f 100755\n> --- a/t/t5319-midx.sh\n> +++ b/t/t5319-midx.sh\n> @@ -6,10 +6,15 @@ test_description='multi-pack-indexes'\n> midx_read_expect() {\n> NUM_PACKS=$1\n> cat >expect <<- EOF\n> - header: 4d494458 1 1 1 $NUM_PACKS\n> - chunks: pack_names\n> - object_dir: .\n> + header: 4d494458 1 1 2 $NUM_PACKS\n> + chunks: pack_lookup pack_names\n> + packs:\n> EOF\n> + if [ $NUM_PACKS -ge 1 ]\n> + then\n> + ls pack/ | grep idx | sort >> expect\n> + fi\n> + printf \"object_dir: .\\n\" >>expect &&\n> git midx read --object-dir=. >actual &&\n> test_cmp expect actual\n> }\n> --\n> 2.18.0.rc1\n>\n\n\n-- \nDuy\n"},{"id":"349807","messageId":"CACsJy8CJ0GNkciHVVUm_7a_MtG5RnZSWpV_zNAwtvMR8aRq42A@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-12-dstolee@microsoft.com","subject":"Re: [PATCH 11/23] midx: sort and deduplicate objects from packfiles","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T17:07:38Z","receivedAt":"2018-06-09T17:08:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:06 PM Derrick Stolee wrote:\n>\n> Before writing a list of objects and their offsets to a multi-pack-index\n> (MIDX), we need to collect the list of objects contained in the\n> packfiles. There may be multiple copies of some objects, so this list\n> must be deduplicated.\n\nCan you just do merge-sort with a slight modification to ignore duplicates?\n\n>\n> It is possible to artificially get into a state where there are many\n> duplicate copies of objects. That can create high memory pressure if we\n> are to create a list of all objects before de-duplication. To reduce\n> this memory pressure without a significant performance drop,\n> automatically group objects by the first byte of their object id. Use\n> the IDX fanout tables to group the data, copy to a local array, then\n> sort.\n>\n> Copy only the de-duplicated entries. Select the duplicate based on the\n> most-recent modified time of a packfile containing the object.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> midx.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 138 insertions(+)\n>\n> diff --git a/midx.c b/midx.c\n> index 923acda72e..b20d52713c 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -4,6 +4,7 @@\n> #include \"csum-file.h\"\n> #include \"lockfile.h\"\n> #include \"object-store.h\"\n> +#include \"packfile.h\"\n> #include \"midx.h\"\n>\n> #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n> @@ -190,6 +191,140 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n> }\n> }\n>\n> +static uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)\n> +{\n> + const uint32_t *level1_ofs = p->index_data;\n> +\n> + if (!level1_ofs) {\n> + if (open_pack_index(p))\n> + return 0;\n> + level1_ofs = p->index_data;\n> + }\n> +\n> + if (p->index_version > 1) {\n> + level1_ofs += 2;\n> + }\n> +\n> + return ntohl(level1_ofs[value]);\n> +}\n\nMaybe keep this in packfile,c, refactor fanout code in there if\nnecessary, keep .idx file format info in that file instead of\nspreading out more.\n\n> +\n> +struct pack_midx_entry {\n> + struct object_id oid;\n> + uint32_t pack_int_id;\n> + time_t pack_mtime;\n> + uint64_t offset;\n> +};\n> +\n> +static int midx_oid_compare(const void *_a, const void *_b)\n> +{\n> + struct pack_midx_entry *a = (struct pack_midx_entry *)_a;\n> + struct pack_midx_entry *b = (struct pack_midx_entry *)_b;\n\nTry not to lose \"const\" while typecasting.\n\n> + int cmp = oidcmp(&a->oid, &b->oid);\n> +\n> + if (cmp)\n> + return cmp;\n> +\n> + if (a->pack_mtime > b->pack_mtime)\n> + return -1;\n> + else if (a->pack_mtime < b->pack_mtime)\n> + return 1;\n> +\n> + return a->pack_int_id - b->pack_int_id;\n> +}\n> +\n> +static void fill_pack_entry(uint32_t pack_int_id,\n> + struct packed_git *p,\n> + uint32_t cur_object,\n> + struct pack_midx_entry *entry)\n> +{\n> + if (!nth_packed_object_oid(&entry->oid, p, cur_object))\n> + die(\"failed to located object %d in packfile\", cur_object);\n\n_()\n\n> +\n> + entry->pack_int_id = pack_int_id;\n> + entry->pack_mtime = p->mtime;\n> +\n> + entry->offset = nth_packed_object_offset(p, cur_object);\n> +}\n> +\n> +/*\n> + * It is possible to artificially get into a state where there are many\n> + * duplicate copies of objects. That can create high memory pressure if\n> + * we are to create a list of all objects before de-duplication. To reduce\n> + * this memory pressure without a significant performance drop, automatically\n> + * group objects by the first byte of their object id. Use the IDX fanout\n> + * tables to group the data, copy to a local array, then sort.\n> + *\n> + * Copy only the de-duplicated entries (selected by most-recent modified time\n> + * of a packfile containing the object).\n> + */\n> +static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n> + uint32_t *perm,\n> + uint32_t nr_packs,\n> + uint32_t *nr_objects)\n> +{\n> + uint32_t cur_fanout, cur_pack, cur_object;\n> + uint32_t nr_fanout, alloc_fanout, alloc_objects, total_objects = 0;\n> + struct pack_midx_entry *entries_by_fanout = NULL;\n> + struct pack_midx_entry *deduplicated_entries = NULL;\n> +\n> + for (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n> + if (open_pack_index(p[cur_pack]))\n> + continue;\n\nIs it a big problem if you fail to open .idx for a certain pack?\nShould we error out and abort instead of continuing on? Later on in\nthe second pack loop code when get_fanout return zero (failure), you\ndon't seem to catch it and skip the pack.\n\n> +\n> + total_objects += p[cur_pack]->num_objects;\n> + }\n> +\n> + /*\n> + * As we de-duplicate by fanout value, we expect the fanout\n> + * slices to be evenly distributed, with some noise. Hence,\n> + * allocate slightly more than one 256th.\n> + */\n> + alloc_objects = alloc_fanout = total_objects > 3200 ? total_objects / 200 : 16;\n> +\n> + ALLOC_ARRAY(entries_by_fanout, alloc_fanout);\n> + ALLOC_ARRAY(deduplicated_entries, alloc_objects);\n> + *nr_objects = 0;\n> +\n> + for (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n> + nr_fanout = 0;\n\nKeep variable scope small, declare nr_fanout here instead of at the\ntop of the function.\n\n> +\n> + for (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n> + uint32_t start = 0, end;\n> +\n> + if (cur_fanout)\n> + start = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n> + end = get_pack_fanout(p[cur_pack], cur_fanout);\n> +\n> + for (cur_object = start; cur_object < end; cur_object++) {\n> + ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n> + fill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n> + nr_fanout++;\n> + }\n> + }\n> +\n> + QSORT(entries_by_fanout, nr_fanout, midx_oid_compare);\n> +\n> + /*\n> + * The batch is now sorted by OID and then mtime (descending).\n> + * Take only the first duplicate.\n> + */\n> + for (cur_object = 0; cur_object < nr_fanout; cur_object++) {\n> + if (cur_object && !oidcmp(&entries_by_fanout[cur_object - 1].oid,\n> + &entries_by_fanout[cur_object].oid))\n> + continue;\n> +\n> + ALLOC_GROW(deduplicated_entries, *nr_objects + 1, alloc_objects);\n> + memcpy(&deduplicated_entries[*nr_objects],\n> + &entries_by_fanout[cur_object],\n> + sizeof(struct pack_midx_entry));\n> + (*nr_objects)++;\n> + }\n> + }\n> +\n> + FREE_AND_NULL(entries_by_fanout);\n> + return deduplicated_entries;\n> +}\n> +\n> static size_t write_midx_pack_lookup(struct hashfile *f,\n> char **pack_names,\n> uint32_t nr_packs)\n> @@ -254,6 +389,7 @@ int write_midx_file(const char *object_dir)\n> uint64_t written = 0;\n> uint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n> uint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n> + uint32_t nr_entries;\n>\n> midx_name = get_midx_filename(object_dir);\n> if (safe_create_leading_directories(midx_name)) {\n> @@ -312,6 +448,8 @@ int write_midx_file(const char *object_dir)\n> ALLOC_ARRAY(pack_perm, nr_packs);\n> sort_packs_by_name(pack_names, nr_packs, pack_perm);\n>\n> + get_sorted_entries(packs, pack_perm, nr_packs, &nr_entries);\n\nIntentional ignoring return value (and temporary leaking as a result)\nshould have a least a comment to acknowledge it and save reviewers\nsome head scratching. Or even better, just free it now, even if you\ndon't use it.\n\n> +\n> hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n> f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n> FREE_AND_NULL(midx_name);\n> --\n> 2.18.0.rc1\n>\n\n\n-- \nDuy\n"},{"id":"349808","messageId":"CACsJy8AVV9bJdC_uh3fGWfoQSCvMexFA5VTSL+cZ+hSULrXc6Q@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-13-dstolee@microsoft.com","subject":"Re: [PATCH 12/23] midx: write object ids in a chunk","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T17:25:01Z","receivedAt":"2018-06-09T17:25:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:07 PM Derrick Stolee wrote:\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/pack-format.txt | 4 ++\n> builtin/midx.c | 2 +\n> midx.c | 50 +++++++++++++++++++++++--\n> object-store.h | 1 +\n> t/t5319-midx.sh | 4 +-\n> 5 files changed, 55 insertions(+), 6 deletions(-)\n>\n> diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\n> index 29bf87283a..de9ac778b6 100644\n> --- a/Documentation/technical/pack-format.txt\n> +++ b/Documentation/technical/pack-format.txt\n> @@ -307,6 +307,10 @@ CHUNK DATA:\n> name. This is the only chunk not guaranteed to be a multiple of four\n> bytes in length, so should be the last chunk for alignment reasons.\n>\n> + OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n\nSo N is the number of objects and H is hash size? Please don't let me guess.\n\n> + The OIDs for all objects in the MIDX are stored in lexicographic\n> + order in this chunk.\n\nThe reason we keep all hashes together, packed right, is to reduce\ncache footprint. Another observation is it takes us usually just 12\nbytes or less to uniquely identify an object, which means we could\npack even tighter if we split he object hash into two chunks, do\nbsearch in the first chunk with just bytes then verify that the\nremaining 20- bytes is matched in the second chunk. This may matter\nmore when we move to larger hashes. The split would of course be\nconfigurable since different project may have different optimal value,\nbut default value could be 10/10 bytes.\n\n> +\n> (This section intentionally left incomplete.)\n>\n> TRAILER:\n> diff --git a/builtin/midx.c b/builtin/midx.c\n> index 3a261e9bbf..86edd30174 100644\n> --- a/builtin/midx.c\n> +++ b/builtin/midx.c\n> @@ -35,6 +35,8 @@ static int read_midx_file(const char *object_dir)\n> printf(\" pack_lookup\");\n> if (m->chunk_pack_names)\n> printf(\" pack_names\");\n> + if (m->chunk_oid_lookup)\n> + printf(\" oid_lookup\");\n>\n> printf(\"\\n\");\n>\n> diff --git a/midx.c b/midx.c\n> index b20d52713c..d06bc6876a 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -14,10 +14,11 @@\n> #define MIDX_HASH_LEN 20\n> #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n>\n> -#define MIDX_MAX_CHUNKS 2\n> +#define MIDX_MAX_CHUNKS 3\n> #define MIDX_CHUNK_ALIGNMENT 4\n> #define MIDX_CHUNKID_PACKLOOKUP 0x504c4f4f /* \"PLOO\" */\n> #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n> +#define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n> #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n>\n> static char *get_midx_filename(const char *object_dir)\n> @@ -95,6 +96,10 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n> m->chunk_pack_names = m->data + chunk_offset;\n> break;\n>\n> + case MIDX_CHUNKID_OIDLOOKUP:\n> + m->chunk_oid_lookup = m->data + chunk_offset;\n> + break;\n\n\nI just now realized, how do you protect from duplicate chunks? From\nthis patch, it looks like you could accept two oidlookup chunks just\nfine then siliently ignore the first one.\n\n> case 0:\n> die(\"terminating MIDX chunk id appears earlier than expected\");\n> break;\n> @@ -112,6 +117,8 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n> die(\"MIDX missing required pack lookup chunk\");\n> if (!m->chunk_pack_names)\n> die(\"MIDX missing required pack-name chunk\");\n> + if (!m->chunk_oid_lookup)\n> + die(\"MIDX missing required OID lookup chunk\");\n\n_()\n\n>\n> m->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n> for (i = 0; i < m->num_packs; i++) {\n> @@ -370,6 +377,32 @@ static size_t write_midx_pack_names(struct hashfile *f,\n> return written;\n> }\n>\n> +static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n> + struct pack_midx_entry *objects,\n> + uint32_t nr_objects)\n> +{\n> + struct pack_midx_entry *list = objects;\n> + uint32_t i;\n> + size_t written = 0;\n> +\n> + for (i = 0; i < nr_objects; i++) {\n> + struct pack_midx_entry *obj = list++;\n> +\n> + if (i < nr_objects - 1) {\n> + struct pack_midx_entry *next = list;\n> + if (oidcmp(&obj->oid, &next->oid) >= 0)\n> + BUG(\"OIDs not in order: %s >= %s\",\n> + oid_to_hex(&obj->oid),\n> + oid_to_hex(&next->oid));\n\nIndentation. I almost thought oid_to_hex() was a separate statement.\n\n> + }\n> +\n> + hashwrite(f, obj->oid.hash, (int)hash_len);\n\nIs (int) really necessary? There's no loss in automatically casting\nunsigned char to int. But I didn't check C spec, maybe there's some\nrules...\n\n> + written += hash_len;\n> + }\n> +\n> + return written;\n> +}\n> +\n> int write_midx_file(const char *object_dir)\n> {\n> unsigned char cur_chunk, num_chunks = 0;\n> @@ -389,6 +422,7 @@ int write_midx_file(const char *object_dir)\n> uint64_t written = 0;\n> uint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n> uint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n> + struct pack_midx_entry *entries;\n> uint32_t nr_entries;\n>\n> midx_name = get_midx_filename(object_dir);\n> @@ -448,14 +482,14 @@ int write_midx_file(const char *object_dir)\n> ALLOC_ARRAY(pack_perm, nr_packs);\n> sort_packs_by_name(pack_names, nr_packs, pack_perm);\n>\n> - get_sorted_entries(packs, pack_perm, nr_packs, &nr_entries);\n> + entries = get_sorted_entries(packs, pack_perm, nr_packs, &nr_entries);\n\nThis change should belong to the previous patch. This patch alone\ncan't tell me that entries is a new allocation. If I didn't remember\nthe last patch, I could not realize that entries should be freed (and\nit does not look like it is here)\n\n>\n> hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n> f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n> FREE_AND_NULL(midx_name);\n>\n> cur_chunk = 0;\n> - num_chunks = 2;\n> + num_chunks = 3;\n>\n> written = write_midx_header(f, num_chunks, nr_packs);\n>\n> @@ -467,9 +501,13 @@ int write_midx_file(const char *object_dir)\n> chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_packs * sizeof(uint32_t);\n>\n> cur_chunk++;\n> - chunk_ids[cur_chunk] = 0;\n> + chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n> chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n>\n> + cur_chunk++;\n> + chunk_ids[cur_chunk] = 0;\n> + chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n> +\n> for (i = 0; i <= num_chunks; i++) {\n> if (i && chunk_offsets[i] < chunk_offsets[i - 1])\n> BUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n> @@ -503,6 +541,10 @@ int write_midx_file(const char *object_dir)\n> written += write_midx_pack_names(f, pack_names, nr_packs);\n> break;\n>\n> + case MIDX_CHUNKID_OIDLOOKUP:\n> + written += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n> + break;\n> +\n> default:\n> BUG(\"trying to write unknown chunk id %\"PRIx32,\n> chunk_ids[i]);\n> diff --git a/object-store.h b/object-store.h\n> index 1ba50459ca..7d14d3586e 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -102,6 +102,7 @@ struct midxed_git {\n>\n> const uint32_t *chunk_pack_lookup;\n> const unsigned char *chunk_pack_names;\n> + const unsigned char *chunk_oid_lookup;\n>\n> const char **pack_names;\n> char object_dir[FLEX_ARRAY];\n> diff --git a/t/t5319-midx.sh b/t/t5319-midx.sh\n> index a31c387c8f..e71aa52b80 100755\n> --- a/t/t5319-midx.sh\n> +++ b/t/t5319-midx.sh\n> @@ -6,8 +6,8 @@ test_description='multi-pack-indexes'\n> midx_read_expect() {\n> NUM_PACKS=$1\n> cat >expect <<- EOF\n> - header: 4d494458 1 1 2 $NUM_PACKS\n> - chunks: pack_lookup pack_names\n> + header: 4d494458 1 1 3 $NUM_PACKS\n> + chunks: pack_lookup pack_names oid_lookup\n> packs:\n> EOF\n> if [ $NUM_PACKS -ge 1 ]\n> --\n> 2.18.0.rc1\n>\n\n\n-- \nDuy\n"},{"id":"349809","messageId":"CACsJy8DW8foQ8+KDdjf4g_3UNi5oSqJyHYo5d9j4cth_yBXDCA@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-14-dstolee@microsoft.com","subject":"Re: [PATCH 13/23] midx: write object id fanout chunk","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T17:28:57Z","receivedAt":"2018-06-09T17:29:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:06 PM Derrick Stolee wrote:\n> @@ -117,9 +123,13 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n> die(\"MIDX missing required pack lookup chunk\");\n> if (!m->chunk_pack_names)\n> die(\"MIDX missing required pack-name chunk\");\n> + if (!m->chunk_oid_fanout)\n> + die(\"MIDX missing required OID fanout chunk\");\n\n_()\n\n> @@ -501,9 +540,13 @@ int write_midx_file(const char *object_dir)\n> chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_packs * sizeof(uint32_t);\n>\n> cur_chunk++;\n> - chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n> + chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n\nErr.. mistake?\n\n> chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n>\n> + cur_chunk++;\n> + chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n\nSame here.\n\n> + chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n> +\n> cur_chunk++;\n> chunk_ids[cur_chunk] = 0;\n> chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n>\n-- \nDuy\n"},{"id":"349810","messageId":"CACsJy8CX5HVNXfiFjOSVJUkyKEfMTcrGKN3T93kyJoG8L00kPg@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-15-dstolee@microsoft.com","subject":"Re: [PATCH 14/23] midx: write object offsets","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T17:41:33Z","receivedAt":"2018-06-09T17:42:03Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 7:02 PM Derrick Stolee wrote:\n> +static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_offset,\n> + struct pack_midx_entry *objects, uint32_t nr_objects)\n> +{\n> + struct pack_midx_entry *list = objects;\n> + size_t written = 0;\n> +\n> + while (nr_large_offset) {\n> + struct pack_midx_entry *obj = list++;\n> + uint64_t offset = obj->offset;\n> +\n> + if (!(offset >> 31))\n> + continue;\n> +\n> + hashwrite_be32(f, offset >> 32);\n> + hashwrite_be32(f, offset & 0xffffffff);\n\nNot sure if you need UL suffix or something here on 32-bit platform.\n\n> + written += 2 * sizeof(uint32_t);\n> +\n> + nr_large_offset--;\n> + }\n> +\n> + return written;\n> +}\n> +\n-- \nDuy\n"},{"id":"349811","messageId":"CACsJy8BD+hiA61gUb-EjoUnx2E9WtCrSBAtHDZuEVFp3M+jOWw@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-17-dstolee@microsoft.com","subject":"Re: [PATCH 16/23] midx: prepare midxed_git struct","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T17:47:50Z","receivedAt":"2018-06-09T17:48:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 7:02 PM Derrick Stolee wrote:\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> midx.c | 22 ++++++++++++++++++++++\n> midx.h | 2 ++\n> object-store.h | 7 +++++++\n> packfile.c | 6 +++++-\n> 4 files changed, 36 insertions(+), 1 deletion(-)\n>\n> diff --git a/midx.c b/midx.c\n> index a49300bf75..5e9290ca8f 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -175,6 +175,28 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n> exit(1);\n> }\n>\n> +int prepare_midxed_git_one(struct repository *r, const char *object_dir)\n> +{\n> + struct midxed_git *m = r->objects->midxed_git;\n> + struct midxed_git *m_search;\n> +\n> + if (!core_midx)\n> + return 0;\n> +\n> + for (m_search = m; m_search; m_search = m_search->next)\n> + if (!strcmp(object_dir, m_search->object_dir))\n> + return 1;\n> +\n> + r->objects->midxed_git = load_midxed_git(object_dir);\n> +\n> + if (r->objects->midxed_git) {\n> + r->objects->midxed_git->next = m;\n> + return 1;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> static size_t write_midx_header(struct hashfile *f,\n> unsigned char num_chunks,\n> uint32_t num_packs)\n> diff --git a/midx.h b/midx.h\n> index a1d18ed991..793203fc4a 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -5,8 +5,10 @@\n> #include \"cache.h\"\n> #include \"object-store.h\"\n> #include \"packfile.h\"\n> +#include \"repository.h\"\n>\n> struct midxed_git *load_midxed_git(const char *object_dir);\n> +int prepare_midxed_git_one(struct repository *r, const char *object_dir);\n>\n> int write_midx_file(const char *object_dir);\n>\n> diff --git a/object-store.h b/object-store.h\n> index 9b671f1b0a..7908d46e34 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -130,6 +130,13 @@ struct raw_object_store {\n> */\n> struct oidmap *replace_map;\n>\n> + /*\n> + * private data\n> + *\n> + * should only be accessed directly by packfile.c and midx.c\n> + */\n> + struct midxed_git *midxed_git;\n> +\n> /*\n> * private data\n> *\n> diff --git a/packfile.c b/packfile.c\n> index 1a714fbde9..b91ca9b9f5 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -15,6 +15,7 @@\n> #include \"tree-walk.h\"\n> #include \"tree.h\"\n> #include \"object-store.h\"\n> +#include \"midx.h\"\n>\n> char *odb_pack_name(struct strbuf *buf,\n> const unsigned char *sha1,\n> @@ -893,10 +894,13 @@ static void prepare_packed_git(struct repository *r)\n>\n> if (r->objects->packed_git_initialized)\n> return;\n> + prepare_midxed_git_one(r, r->objects->objectdir);\n> prepare_packed_git_one(r, r->objects->objectdir, 1);\n> prepare_alt_odb(r);\n> - for (alt = r->objects->alt_odb_list; alt; alt = alt->next)\n> + for (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n> + prepare_midxed_git_one(r, alt->path);\n> prepare_packed_git_one(r, alt->path, 0);\n> + }\n\nAh, so the object path and the linked list in midxed_git is for\nalternates. Makes sense. Would have saved me the trouble if you only\nintroduced those fields now, when they are actually used (and become\nself explanatory)\n\n> rearrange_packed_git(r);\n> prepare_packed_git_mru(r);\n> r->objects->packed_git_initialized = 1;\n> --\n> 2.18.0.rc1\n>\n-- \nDuy\n"},{"id":"349812","messageId":"CACsJy8CqoKWgd=ekxozaKzRjrU3NN9KUxrspk_iP+zxBVcV5pA@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-18-dstolee@microsoft.com","subject":"Re: [PATCH 17/23] midx: read objects from multi-pack-index","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T17:56:13Z","receivedAt":"2018-06-09T17:56:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 6:55 PM Derrick Stolee wrote:\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> midx.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++--\n> midx.h | 2 ++\n> object-store.h | 1 +\n> packfile.c | 8 ++++-\n> 4 files changed, 104 insertions(+), 3 deletions(-)\n>\n> diff --git a/midx.c b/midx.c\n> index 5e9290ca8f..6eca8f1b12 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -3,6 +3,7 @@\n> #include \"dir.h\"\n> #include \"csum-file.h\"\n> #include \"lockfile.h\"\n> +#include \"sha1-lookup.h\"\n> #include \"object-store.h\"\n> #include \"packfile.h\"\n> #include \"midx.h\"\n> @@ -64,7 +65,7 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n>\n> m = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n> strcpy(m->object_dir, object_dir);\n> - m->data = midx_map;\n> + m->data = (const unsigned char*)midx_map;\n\nHmm? Why is this typecast only needed now? Or is it not really needed at all?\n\n>\n> m->signature = get_be32(m->data);\n> if (m->signature != MIDX_SIGNATURE) {\n> @@ -145,7 +146,9 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n>\n> m->num_objects = ntohl(m->chunk_oid_fanout[255]);\n>\n> - m->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n> + m->packs = xcalloc(m->num_packs, sizeof(*m->packs));\n> +\n> + ALLOC_ARRAY(m->pack_names, m->num_packs);\n\nPlease make this ALLOC_ARRAY change in the patch that adds\nxcalloc(m->num_packs).\n\n> for (i = 0; i < m->num_packs; i++) {\n> if (i) {\n> if (ntohl(m->chunk_pack_lookup[i]) <= ntohl(m->chunk_pack_lookup[i - 1])) {\n> @@ -175,6 +178,95 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n> exit(1);\n> }\n>\n> +static int prepare_midx_pack(struct midxed_git *m, uint32_t pack_int_id)\n> +{\n> + struct strbuf pack_name = STRBUF_INIT;\n> +\n> + if (pack_int_id >= m->num_packs)\n> + BUG(\"bad pack-int-id\");\n> +\n> + if (m->packs[pack_int_id])\n> + return 0;\n> +\n> + strbuf_addstr(&pack_name, m->object_dir);\n> + strbuf_addstr(&pack_name, \"/pack/\");\n> + strbuf_addstr(&pack_name, m->pack_names[pack_int_id]);\n\nJust use strbuf_addf()\n\n> +\n> + m->packs[pack_int_id] = add_packed_git(pack_name.buf, pack_name.len, 1);\n> + strbuf_release(&pack_name);\n> + return !m->packs[pack_int_id];\n\nThis is a weird return value convention. Normally we go zero/negative\nor non-zero/zero for success/failure.\n\n> +}\n-- \nDuy\n"},{"id":"349814","messageId":"CACsJy8BQ5xxhQ8L9K84KO2EkYELsKKKkdJ0mvc8p427sA8bapQ@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-19-dstolee@microsoft.com","subject":"Re: [PATCH 18/23] midx: use midx in abbreviation calculations","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T18:01:28Z","receivedAt":"2018-06-09T18:01:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:06 PM Derrick Stolee wrote:\n> @@ -565,8 +632,11 @@ static void find_abbrev_len_for_pack(struct packed_git *p,\n>\n> static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n> {\n> + struct midxed_git *m;\n> struct packed_git *p;\n>\n> + for (m = get_midxed_git(the_repository); m; m = m->next)\n> + find_abbrev_len_for_midx(m, mad);\n\nIf all the packs are in midx, we don't need to run the second loop\nbelow, do we? Otherwise I don't see why we waste cycles on finding\nabbrev length on midx at all.\n\n> for (p = get_packed_git(the_repository); p; p = p->next)\n> find_abbrev_len_for_pack(p, mad);\n> }\n-- \nDuy\n"},{"id":"349815","messageId":"CACsJy8AtkVwGw0+tgpv2AzNZMHGTOfkjHUN-37owaRWuQ_2wKw@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-21-dstolee@microsoft.com","subject":"Re: [PATCH 20/23] midx: use midx in approximate_object_count","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T18:03:19Z","receivedAt":"2018-06-09T18:03:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:06 PM Derrick Stolee wrote:\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> packfile.c | 5 ++++-\n> 1 file changed, 4 insertions(+), 1 deletion(-)\n>\n> diff --git a/packfile.c b/packfile.c\n> index 638e113972..059b2aa097 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -819,11 +819,14 @@ unsigned long approximate_object_count(void)\n> {\n> if (!the_repository->objects->approximate_object_count_valid) {\n> unsigned long count;\n> + struct midxed_git *m;\n> struct packed_git *p;\n>\n> prepare_packed_git(the_repository);\n> count = 0;\n> - for (p = the_repository->objects->packed_git; p; p = p->next) {\n> + for (m = get_midxed_git(the_repository); m; m = m->next)\n> + count += m->num_objects;\n> + for (p = get_packed_git(the_repository); p; p = p->next) {\n\nPlease don't change this line, it's not related to this patch. Same\nconcern applies, if we have already counted objects in midx we should\nignore packs that belong to it or we double count.\n\n> if (open_pack_index(p))\n> continue;\n> count += p->num_objects;\n> --\n> 2.18.0.rc1\n>\n\n\n-- \nDuy\n"},{"id":"349816","messageId":"CACsJy8DbF3AXuHK9yaEtYPwkhP_BC47b50iTT0tHqODGVp7FCA@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-22-dstolee@microsoft.com","subject":"Re: [PATCH 21/23] midx: prevent duplicate packfile loads","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T18:05:39Z","receivedAt":"2018-06-09T18:06:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:07 PM Derrick Stolee wrote:\n>\n> If the multi-pack-index contains a packfile, then we do not need to add\n> that packfile to the packed_git linked list or the MRU list.\n\nBecause...?\n\nI think I see the reason, but I'd like it spelled out to avoid any\nmisunderstanding.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> midx.c | 23 +++++++++++++++++++++++\n> midx.h | 1 +\n> packfile.c | 7 +++++++\n> 3 files changed, 31 insertions(+)\n>\n> diff --git a/midx.c b/midx.c\n> index 388d79b7d9..3242646fe0 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -278,6 +278,29 @@ int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct mi\n> return nth_midxed_pack_entry(m, e, pos);\n> }\n>\n> +int midx_contains_pack(struct midxed_git *m, const char *idx_name)\n> +{\n> + uint32_t first = 0, last = m->num_packs;\n> +\n> + while (first < last) {\n> + uint32_t mid = first + (last - first) / 2;\n> + const char *current;\n> + int cmp;\n> +\n> + current = m->pack_names[mid];\n> + cmp = strcmp(idx_name, current);\n> + if (!cmp)\n> + return 1;\n> + if (cmp > 0) {\n> + first = mid + 1;\n> + continue;\n> + }\n> + last = mid;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> int prepare_midxed_git_one(struct repository *r, const char *object_dir)\n> {\n> struct midxed_git *m = r->objects->midxed_git;\n> diff --git a/midx.h b/midx.h\n> index 497bdcc77c..c1db58d8c4 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -13,6 +13,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n> struct midxed_git *m,\n> uint32_t n);\n> int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct midxed_git *m);\n> +int midx_contains_pack(struct midxed_git *m, const char *idx_name);\n> int prepare_midxed_git_one(struct repository *r, const char *object_dir);\n>\n> int write_midx_file(const char *object_dir);\n> diff --git a/packfile.c b/packfile.c\n> index 059b2aa097..479cb69b9f 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -746,6 +746,11 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n> DIR *dir;\n> struct dirent *de;\n> struct string_list garbage = STRING_LIST_INIT_DUP;\n> + struct midxed_git *m = r->objects->midxed_git;\n> +\n> + /* look for the multi-pack-index for this object directory */\n> + while (m && strcmp(m->object_dir, objdir))\n> + m = m->next;\n>\n> strbuf_addstr(&path, objdir);\n> strbuf_addstr(&path, \"/pack\");\n> @@ -772,6 +777,8 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n> base_len = path.len;\n> if (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n> /* Don't reopen a pack we already have. */\n> + if (m && midx_contains_pack(m, de->d_name))\n> + continue;\n> for (p = r->objects->packed_git; p;\n> p = p->next) {\n> size_t len;\n> --\n> 2.18.0.rc1\n>\n\n\n-- \nDuy\n"},{"id":"349818","messageId":"CACsJy8BHgqqrDNvU_oy0+_QmDW-wt-jBvWT9oJ9H1pA_tmuVFg@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-24-dstolee@microsoft.com","subject":"Re: [PATCH 23/23] midx: clear midx on repack","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-09T18:13:47Z","receivedAt":"2018-06-09T18:14:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 4:07 PM Derrick Stolee wrote:\n>\n> If a 'git repack' command replaces existing packfiles, then we must\n> clear the existing multi-pack-index before moving the packfiles it\n> references.\n\nI think there are other places where we add or remove pack files and\nneed to reprepare_packed_git(). Any midx invalidation should be part\nof that as well.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/repack.c | 8 ++++++++\n> midx.c | 8 ++++++++\n> midx.h | 1 +\n> 3 files changed, 17 insertions(+)\n>\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 6c636e159e..66a7d8e8ea 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -8,6 +8,7 @@\n> #include \"strbuf.h\"\n> #include \"string-list.h\"\n> #include \"argv-array.h\"\n> +#include \"midx.h\"\n>\n> static int delta_base_offset = 1;\n> static int pack_kept_objects = -1;\n> @@ -174,6 +175,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> int no_update_server_info = 0;\n> int quiet = 0;\n> int local = 0;\n> + int midx_cleared = 0;\n>\n> struct option builtin_repack_options[] = {\n> OPT_BIT('a', NULL, &pack_everything,\n> @@ -340,6 +342,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> continue;\n> }\n>\n> + if (!midx_cleared) {\n> + /* if we move a packfile, it will invalidated the midx */\n\nWhat about removing packs, which also happens in repack? If the\nremoved pack is part of midx, then midx becomes invalid as well.\n\n> + clear_midx_file(get_object_directory());\n> + midx_cleared = 1;\n> + }\n> +\n> fname_old = mkpathdup(\"%s/old-%s%s\", packdir,\n> item->string, exts[ext].name);\n> if (file_exists(fname_old))\n> diff --git a/midx.c b/midx.c\n> index e46f392fa4..1043c01fa7 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -913,3 +913,11 @@ int write_midx_file(const char *object_dir)\n> FREE_AND_NULL(pack_names);\n> return 0;\n> }\n> +\n> +void clear_midx_file(const char *object_dir)\n\ndelete_ may be more obvious than clear_\n\n> +{\n> + char *midx = get_midx_filename(object_dir);\n> +\n> + if (remove_path(midx))\n> + die(_(\"failed to clear multi-pack-index at %s\"), midx);\n\ndie_errno()\n\n> +}\n> diff --git a/midx.h b/midx.h\n> index 6996b5ff6b..46f9f44c94 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -18,5 +18,6 @@ int midx_contains_pack(struct midxed_git *m, const char *idx_name);\n> int prepare_midxed_git_one(struct repository *r, const char *object_dir);\n>\n> int write_midx_file(const char *object_dir);\n> +void clear_midx_file(const char *object_dir);\n>\n> #endif\n> --\n> 2.18.0.rc1\n>\n\n\n-- \nDuy\n"},{"id":"349930","messageId":"CAGZ79ka4RD_E5NpEf8-7BUCcoqEc+hDrHNR+YAqT=6m5TYg7mg@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-2-dstolee@microsoft.com","subject":"Re: [PATCH 01/23] midx: add design document","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-06-11T19:04:02Z","receivedAt":"2018-06-11T19:04:18Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Jun 7, 2018 at 7:03 AM Derrick Stolee wrote:\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/midx.txt | 109 +++++++++++++++++++++++++++++++\n> 1 file changed, 109 insertions(+)\n> create mode 100644 Documentation/technical/midx.txt\n>\n> diff --git a/Documentation/technical/midx.txt b/Documentation/technical/midx.txt\n> new file mode 100644\n> index 0000000000..789f410d71\n> --- /dev/null\n> +++ b/Documentation/technical/midx.txt\n> @@ -0,0 +1,109 @@\n> +Multi-Pack-Index (MIDX) Design Notes\n> +====================================\n> +\n> +The Git object directory contains a 'pack' directory containing\n> +packfiles (with suffix \".pack\") and pack-indexes (with suffix\n> +\".idx\"). The pack-indexes provide a way to lookup objects and\n> +navigate to their offset within the pack, but these must come\n> +in pairs with the packfiles. This pairing depends on the file\n> +names, as the pack-index differs only in suffix with its pack-\n> +file. While the pack-indexes provide fast lookup per packfile,\n> +this performance degrades as the number of packfiles increases,\n> +because abbreviations need to inspect every packfile and we are\n> +more likely to have a miss on our most-recently-used packfile.\n> +For some large repositories, repacking into a single packfile\n> +is not feasible due to storage space or excessive repack times.\n\nThis leads to the question how MIDX will cope with large repos or\na large number of packs. As it is just an index and not a pack itself,\nI guess it is smaller by some orders of magnitude, such that it\nis ok for now.\n\n> +The multi-pack-index (MIDX for short) stores a list of objects\n> +and their offsets into multiple packfiles. It contains:\n> +\n> +- A list of packfile names.\n> +- A sorted list of object IDs.\n> +- A list of metadata for the ith object ID including:\n> + - A value j referring to the jth packfile.\n> + - An offset within the jth packfile for the object.\n> +- If large offsets are required, we use another list of large\n> + offsets similar to version 2 pack-indexes.\n> +\n> +Thus, we can provide O(log N) lookup time for any number\n> +of packfiles.\n\nThis sounds great for the lookup case!\nThough that is for the repo-read case.\nLet's read on how the dynamics of a repository are dealt with,\ne.g. integrating new packs into the MIDX, or how we deal with\nobjects in multiple packs.\n\n> +\n> +Design Details\n> +--------------\n> +\n> +- The MIDX is stored in a file named 'multi-pack-index' in the\n> + .git/objects/pack directory. This could be stored in the pack\n> + directory of an alternate. It refers only to packfiles in that\n> + same directory.\n\nSo there is one and only one multi pack index?\nThat makes the case of preparing the next MIDX that contains more\npack references more interesting, as then we have to atomically update\nthat file.\n\n> +- The core.midx config setting must be on to consume MIDX files.\n\nLooking through current config options, I would rename this to a more\nsuggestive name. I searched for the core.idx counterpart that enables\nidx files -- it turns out that is named pack.indexVersion.\n\nSo maybe pack.MultiIndex ? That could start out as a boolean as in this\nseries and then evolve into a version number or such later.\n\n> +- The file format includes parameters for the object ID hash\n> + function, so a future change of hash algorithm does not require\n> + a change in format.\n> +\n> +- The MIDX keeps only one record per object ID. If an object appears\n> + in multiple packfiles, then the MIDX selects the copy in the most-\n> + recently modified packfile.\n\nOkay. That answers the question from above. Though this is just the tie\nbreaking decision and not a hard limitation? (i.e. we could change this\nthis later to that pack that has e.g. shortest delta chain for that object or\nsuch)\n\n> +- If there exist packfiles in the pack directory not registered in\n> + the MIDX, then those packfiles are loaded into the `packed_git`\n> + list and `packed_git_mru` cache.\n\nNot sure I understand the implications of this?\nDoes that mean we first look at the multi index and if an object is not\nfound, we'll search linearly through all packs that are not part of the\nMIDX? That would require the MIDX to be kepot up to date reasonably\nto be useful.\n\n> +- The pack-indexes (.idx files) remain in the pack directory so we\n> + can delete the MIDX file, set core.midx to false, or downgrade\n> + without any loss of information.\n\nIn the future will it be possible to have no .idx files and just have the .midx?\n(I guess that depends on the strategy of how to integrate new packs into\nthe MIDX?)\n\n> +- The MIDX file format uses a chunk-based approach (similar to the\n> + commit-graph file) that allows optional data to be added.\n\n... or the index files v2 (or reftable files)? Sure, you are most familiar with\ncommit-graph files, but others may find it easier to have some older\nfile formats to relate to.\n\n> +Future Work\n> +-----------\n> +\n> +- Add a 'verify' subcommand to the 'git midx' builtin to verify the\n> + contents of the multi-pack-index file match the offsets listed in\n> + the corresponding pack-indexes.\n> +\n> +- The multi-pack-index allows many packfiles, especially in a context\n> + where repacking is expensive (such as a very large repo), or\n> + unexpected maintenance time is unacceptable (such as a high-demand\n> + build machine).\n\nSupposedly maintenance (git gc) can be run in the background without\ninterfering with day-to-day life, how is the regeneration of commit graph\nor MIDX files impacting the work here?\n\n> However, the multi-pack-index needs to be rewritten\n> + in full every time. We can extend the format to be incremental, so\n> + writes are fast. By storing a small \"tip\" multi-pack-index that\n> + points to large \"base\" MIDX files, we can keep writes fast while\n> + still reducing the number of binary searches required for object\n> + lookups.\n\nSo we can have multiple MIDX files? How would that work? Would there\nbe a chunk that refers to other MIDX files?\n\n> +- The reachability bitmap is currently paired directly with a single\n> + packfile, using the pack-order as the object order to hopefully\n> + compress the bitmaps well using run-length encoding. This could be\n> + extended to pair a reachability bitmap with a multi-pack-index. If\n> + the multi-pack-index is extended to store a \"stable object order\"\n> + (a function Order(hash) = integer that is constant for a given hash,\n\nThis stable object order doesn't fly well with integrating new packs?\n\n> + even as the multi-pack-index is updated) then a reachability bitmap\n> + could point to a multi-pack-index and be updated independently.\n> +\n> +- Packfiles can be marked as \"special\" using empty files that share\n> + the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n> + We can add an optional chunk of data to the multi-pack-index that\n> + records flags of information about the packfiles. This allows new\n> + states, such as 'repacked' or 'redeltified', that can help with\n> + pack maintenance in a multi-pack environment. It may also be\n> + helpful to organize packfiles by object type (commit, tree, blob,\n> + etc.) and use this metadata to help that maintenance.\n> +\n> +- The partial clone feature records special \"promisor\" packs that\n> + may point to objects that are not stored locally, but available\n> + on request to a server. The multi-pack-index does not currently\n> + track these promisor packs.\n"},{"id":"349932","messageId":"CAGZ79kaDiXnAuRym3HHQPEZh9z0-sPL5V4Yur5-_RojsyLJsYA@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-3-dstolee@microsoft.com","subject":"Re: [PATCH 02/23] midx: add midx format details to pack-format.txt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-06-11T19:19:13Z","receivedAt":"2018-06-11T19:23:55Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Hi Derrick,\nOn Thu, Jun 7, 2018 at 7:03 AM Derrick Stolee wrote:\n>\n> The multi-pack-index (MIDX) feature generalizes the existing pack-\n> index (IDX) feature by indexing objects across multiple pack-files.\n>\n> Describe the basic file format, using a 12-byte header followed by\n> a lookup table for a list of \"chunks\" which will be described later.\n> The file ends with a footer containing a checksum using the hash\n> algorithm.\n>\n> The header allows later versions to create breaking changes by\n> advancing the version number. We can also change the hash algorithm\n> using a different version value.\n>\n> We will add the individual chunk format information as we introduce\n> the code that writes that information.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/pack-format.txt | 49 +++++++++++++++++++++++++\n> 1 file changed, 49 insertions(+)\n>\n> diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\n> index 70a99fd142..17666b4bfc 100644\n> --- a/Documentation/technical/pack-format.txt\n> +++ b/Documentation/technical/pack-format.txt\n> @@ -252,3 +252,52 @@ Pack file entry: <+\n> corresponding packfile.\n>\n> 20-byte SHA-1-checksum of all of the above.\n> +\n> +== midx-*.midx files have the following format:\n> +\n> +The meta-index files refer to multiple pack-files and loose objects.\n\nSo is it meta or multi?\n\n> +In order to allow extensions that add extra data to the MIDX, we organize\n> +the body into \"chunks\" and provide a lookup table at the beginning of the\n> +body. The header includes certain length values, such as the number of packs,\n> +the number of base MIDX files, hash lengths and types.\n> +\n> +All 4-byte numbers are in network order.\n> +\n> +HEADER:\n> +\n> + 4-byte signature:\n> + The signature is: {'M', 'I', 'D', 'X'}\n> +\n> + 1-byte version number:\n> + Git only writes or recognizes version 1\n> +\n> + 1-byte Object Id Version\n> + Git only writes or recognizes verion 1 (SHA-1)\n\ns/verion/version/\n\n> + 1-byte number (C) of \"chunks\"\n> +\n> + 1-byte number (I) of base multi-pack-index files:\n> + This value is currently always zero.\n\nOh? Are meta-index and multi-index files different things?\n\n> + 4-byte number (P) of pack files\n> +\n> +CHUNK LOOKUP:\n> +\n> + (C + 1) * 12 bytes providing the chunk offsets:\n> + First 4 bytes describe chunk id. Value 0 is a terminating label.\n> + Other 8 bytes provide offset in current file for chunk to start.\n> + (Chunks are provided in file-order, so you can infer the length\n> + using the next chunk position if necessary.)\n\nIt is so nice to have the header also have 12 bytes, so it fits right into the\nlookup table. So an alternative point of view:\n\n If a chunk needs to store more than 8 bytes, we'll have an offset after\n the first 4 bytes that describe the chunk, otherwise you can store the 8 bytes\n of information directly after the 4 bytes.\n \"MIDX\" is a special chunk and must come first (does it?) and only once\n as it contains the version number.\n\n> + The remaining data in the body is described one chunk at a time, and\n> + these chunks may be given in any order. Chunks are required unless\n> + otherwise specified.\n> +\n> +CHUNK DATA:\n> +\n> + (This section intentionally left incomplete.)\n> +\n> +TRAILER:\n> +\n> + H-byte HASH-checksum of all of the above.\n\nThis means we have to rehash the whole file for updating its contents.\nokay.\n"},{"id":"349941","messageId":"CAGZ79kaH7XzbtPZqVwGgP774GJNxF7ZsHqLzWEmb2NLSxG3Njg@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-4-dstolee@microsoft.com","subject":"Re: [PATCH 03/23] midx: add midx builtin","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-06-11T21:02:08Z","receivedAt":"2018-06-11T21:02:24Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Hi Derrick,\nOn Thu, Jun 7, 2018 at 7:03 AM Derrick Stolee wrote:\n>\n> This new 'git midx' builtin will be the plumbing access for writing,\n> reading, and checking multi-pack-index (MIDX) files. The initial\n> implementation is a no-op.\n\nLet's talk about the name for a second:\n\n.idx files are written by git-index-pack or as part of\ngit-pack-objects (which just calls write_idx_file as part\nof finish_tmp_packfile), and the name actually suggests\nit writes the index files. I have a hard time understanding\nwhat the git-midx command does[1].\n\nWith both commit graph as well as multi index we introduce\na command that is centered around that concept (similar to\ngit-remote or git-config that are centered around a concept,\nthat is closely resembled by a file), but for indexes for packs\nit was integrated differently into Git. So I am not sure if I want\nto suggest to integrate it into the packfile commands as that\ndoesn't really fit. But maybe we can have a name that is human\nreadable instead of the file suffix? Maybe\n\n git multi-pack-index ?\n\nI suppose that eventually this command is not really used by\nusers as it will be used by other porcelain commands in the\nbackground or even as part of repack/gc so I am not worried\nabout a long name, but I'd be more worried about understandability.\n\n[1] While these names are not perfect for the layman, it is okay?\n I am sure you are aware of https://git-man-page-generator.lokaltog.net/\n\n\n> new file mode 100644\n> index 0000000000..2bd886f1a2\n> --- /dev/null\n> +++ b/Documentation/git-midx.txt\n> @@ -0,0 +1,29 @@\n> +git-midx(1)\n> +============\n> +\n> +NAME\n> +----\n> +git-midx - Write and verify multi-pack-indexes (MIDX files).\n\nThe reading is done as part of all other commands.\n\n> +\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git midx' [--object-dir ]\n> +\n> +DESCRIPTION\n> +-----------\n> +Write or verify a MIDX file.\n> +\n> +OPTIONS\n> +-------\n> +\n> +--object-dir ::\n> + Use given directory for the location of Git objects. We check\n> + /packs/multi-pack-index for the current MIDX file, and\n> + /packs for the pack-files to index.\n> +\n> +\n\nMaybe we could have a SEE ALSO section that points at\nthe explanation of multi index files?\n(c.f. man git-submodule that has a SEE ALSO\ngitsubmodules(7), gitmodules(5) explaining concepts(7)\nand the file(5))\n\nBut as this is plumbing and users should not need to worry about it\nthis is optional, I would think.\n"},{"id":"349983","messageId":"CACsJy8ALMjiyjcEdFPnR7GTzBVpqVB72VQNkpUWrj9p4nm-OGg@mail.gmail.com","threadId":"48668","inReplyTo":"20180607140338.32440-6-dstolee@microsoft.com","subject":"Re: [PATCH 05/23] midx: write header information to lockfile","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-12T15:00:14Z","receivedAt":"2018-06-12T15:00:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 7, 2018 at 7:01 PM Derrick Stolee wrote:\n> diff --git a/midx.c b/midx.c\n> index 616af66b13..3e55422a21 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -1,9 +1,62 @@\n> #include \"git-compat-util.h\"\n> #include \"cache.h\"\n> #include \"dir.h\"\n> +#include \"csum-file.h\"\n> +#include \"lockfile.h\"\n> #include \"midx.h\"\n>\n> +#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n> +#define MIDX_VERSION 1\n> +#define MIDX_HASH_VERSION 1 /* SHA-1 */\n...\n> +static size_t write_midx_header(struct hashfile *f,\n> + unsigned char num_chunks,\n> + uint32_t num_packs)\n> +{\n> + char byte_values[4];\n> + hashwrite_be32(f, MIDX_SIGNATURE);\n> + byte_values[0] = MIDX_VERSION;\n> + byte_values[1] = MIDX_HASH_VERSION;\n\nQuoting from \"State of NewHash work, future directions, and discussion\" [1]\n\n* If you need to serialize an algorithm identifier into your data\n format, use the format_id field of struct git_hash_algo. It's\n designed specifically for that purpose.\n\n[1] https://public-inbox.org/git/20180612024252.GA141166@aiede.svl.corp.google.com/T/#m5fdd09dcaf31266c45343fb6c0beaaa3e928bc60\n\n> + byte_values[2] = num_chunks;\n> + byte_values[3] = 0; /* unused */\n> + hashwrite(f, byte_values, sizeof(byte_values));\n> + hashwrite_be32(f, num_packs);\n> +\n> + return MIDX_HEADER_SIZE;\n> +}\n-- \nDuy\n"},{"id":"350409","messageId":"e251bcff-8638-19d8-2937-0e913b19215a@gmail.com","threadId":"48668","inReplyTo":"CAGZ79ka4RD_E5NpEf8-7BUCcoqEc+hDrHNR+YAqT=6m5TYg7mg@mail.gmail.com","subject":"Re: [PATCH 01/23] midx: add design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-18T18:48:25Z","receivedAt":"2018-06-18T18:48:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/11/2018 3:04 PM, Stefan Beller wrote:\n> On Thu, Jun 7, 2018 at 7:03 AM Derrick Stolee wrote:\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/technical/midx.txt | 109 +++++++++++++++++++++++++++++++\n>> 1 file changed, 109 insertions(+)\n>> create mode 100644 Documentation/technical/midx.txt\n>>\n>> diff --git a/Documentation/technical/midx.txt b/Documentation/technical/midx.txt\n>> new file mode 100644\n>> index 0000000000..789f410d71\n>> --- /dev/null\n>> +++ b/Documentation/technical/midx.txt\n>> @@ -0,0 +1,109 @@\n>> +Multi-Pack-Index (MIDX) Design Notes\n>> +====================================\n>> +\n>> +The Git object directory contains a 'pack' directory containing\n>> +packfiles (with suffix \".pack\") and pack-indexes (with suffix\n>> +\".idx\"). The pack-indexes provide a way to lookup objects and\n>> +navigate to their offset within the pack, but these must come\n>> +in pairs with the packfiles. This pairing depends on the file\n>> +names, as the pack-index differs only in suffix with its pack-\n>> +file. While the pack-indexes provide fast lookup per packfile,\n>> +this performance degrades as the number of packfiles increases,\n>> +because abbreviations need to inspect every packfile and we are\n>> +more likely to have a miss on our most-recently-used packfile.\n>> +For some large repositories, repacking into a single packfile\n>> +is not feasible due to storage space or excessive repack times.\n> This leads to the question how MIDX will cope with large repos or\n> a large number of packs. As it is just an index and not a pack itself,\n> I guess it is smaller by some orders of magnitude, such that it\n> is ok for now.\n\nThe MIDX file is only slightly larger than the union of the IDX files \nfor those packfiles.\n\n>> +The multi-pack-index (MIDX for short) stores a list of objects\n>> +and their offsets into multiple packfiles. It contains:\n>> +\n>> +- A list of packfile names.\n>> +- A sorted list of object IDs.\n>> +- A list of metadata for the ith object ID including:\n>> + - A value j referring to the jth packfile.\n>> + - An offset within the jth packfile for the object.\n>> +- If large offsets are required, we use another list of large\n>> + offsets similar to version 2 pack-indexes.\n>> +\n>> +Thus, we can provide O(log N) lookup time for any number\n>> +of packfiles.\n> This sounds great for the lookup case!\n> Though that is for the repo-read case.\n> Let's read on how the dynamics of a repository are dealt with,\n> e.g. integrating new packs into the MIDX, or how we deal with\n> objects in multiple packs.\n>\n>> +\n>> +Design Details\n>> +--------------\n>> +\n>> +- The MIDX is stored in a file named 'multi-pack-index' in the\n>> + .git/objects/pack directory. This could be stored in the pack\n>> + directory of an alternate. It refers only to packfiles in that\n>> + same directory.\n> So there is one and only one multi pack index?\n> That makes the case of preparing the next MIDX that contains more\n> pack references more interesting, as then we have to atomically update\n> that file.\n\nThere is only one, but we can make the file incremental without changing \nthis name similar to how the split index works.\n\n>\n>> +- The core.midx config setting must be on to consume MIDX files.\n> Looking through current config options, I would rename this to a more\n> suggestive name. I searched for the core.idx counterpart that enables\n> idx files -- it turns out that is named pack.indexVersion.\n>\n> So maybe pack.MultiIndex ? That could start out as a boolean as in this\n> series and then evolve into a version number or such later.\n\nI'll use that name and rename this file to \nDocumentation/technical/multi-pack-index.txt\n\n>\n>> +- The file format includes parameters for the object ID hash\n>> + function, so a future change of hash algorithm does not require\n>> + a change in format.\n>> +\n>> +- The MIDX keeps only one record per object ID. If an object appears\n>> + in multiple packfiles, then the MIDX selects the copy in the most-\n>> + recently modified packfile.\n> Okay. That answers the question from above. Though this is just the tie\n> breaking decision and not a hard limitation? (i.e. we could change this\n> this later to that pack that has e.g. shortest delta chain for that object or\n> such)\n\nThis is a soft requirement. It is an easy thing to track at the moment. \nWe can compute the MIDX without opening a packfile, for instance.\n\n>\n>> +- If there exist packfiles in the pack directory not registered in\n>> + the MIDX, then those packfiles are loaded into the `packed_git`\n>> + list and `packed_git_mru` cache.\n> Not sure I understand the implications of this?\n> Does that mean we first look at the multi index and if an object is not\n> found, we'll search linearly through all packs that are not part of the\n> MIDX? That would require the MIDX to be kepot up to date reasonably\n> to be useful.\n\nIf you add a packfile to the pack directory, you can immediately start \nconsuming it. You do not need to wait for the MIDX to be updated. The \nmore asynchronous these auxiliary data structures (MIDX, commit-graph) \ncan be, the better. This is in direct contrast to the reachability \nbitmap which is useless without its corresponding packfile.\n\n>\n>> +- The pack-indexes (.idx files) remain in the pack directory so we\n>> + can delete the MIDX file, set core.midx to false, or downgrade\n>> + without any loss of information.\n> In the future will it be possible to have no .idx files and just have the .midx?\n> (I guess that depends on the strategy of how to integrate new packs into\n> the MIDX?)\n\nThis may be part of a future plan, but we need to know a user will never \nset pack.multiIndex to false if they deleted their IDX files.\n\n>> +- The MIDX file format uses a chunk-based approach (similar to the\n>> + commit-graph file) that allows optional data to be added.\n> ... or the index files v2 (or reftable files)? Sure, you are most familiar with\n> commit-graph files, but others may find it easier to have some older\n> file formats to relate to.\n\nI specifically mean that we have a \"table of contents\" describing the \nlist of chunks. IDX v2 relies on a fixed ordering of the tables, and the \noffsets are computed by consuming the last fanout value (number of \nobjects). Also, I'm not sure how to add optional data (data that can \nsafely be ignored by an earlier version of Git) to an IDX without \nincrementing the version.\n\n>> +Future Work\n>> +-----------\n>> +\n>> +- Add a 'verify' subcommand to the 'git midx' builtin to verify the\n>> + contents of the multi-pack-index file match the offsets listed in\n>> + the corresponding pack-indexes.\n>> +\n>> +- The multi-pack-index allows many packfiles, especially in a context\n>> + where repacking is expensive (such as a very large repo), or\n>> + unexpected maintenance time is unacceptable (such as a high-demand\n>> + build machine).\n> Supposedly maintenance (git gc) can be run in the background without\n> interfering with day-to-day life, how is the regeneration of commit graph\n> or MIDX files impacting the work here?\n\nAssuming infinite RAM and disk, then yes we could not interfere with \ndaily life. A big problem we see is that users frequently don't have the \ndisk space to store a second copy of their packfiles on disk, even if we \ncould organize a new packfile in reasonable time.\n\nThe MIDX is only invalid when a packfile it references is deleted.\n\nThe commit-graph is never invalid, except if a commit is deleted by GC. \nBut even in that case, how did we \"see\" the commit ID? As long as we \ndon't access these commits, the commit-graph feature doesn't violate \nexpectations and can be generated asynchronously with a GC and repack.\n\n>\n>> However, the multi-pack-index needs to be rewritten\n>> + in full every time. We can extend the format to be incremental, so\n>> + writes are fast. By storing a small \"tip\" multi-pack-index that\n>> + points to large \"base\" MIDX files, we can keep writes fast while\n>> + still reducing the number of binary searches required for object\n>> + lookups.\n> So we can have multiple MIDX files? How would that work? Would there\n> be a chunk that refers to other MIDX files?\n\nWe can have an optional chunk that refers to a list of \"base\" MIDX \nfiles. We then load that full list into multiple 'midxed_git' structs, \nand iterate through the list. VSTS keeps this list to a maximum length \nof 3 (LARGE, Medium, tiny) and merging files as necessary.\n\n>\n>> +- The reachability bitmap is currently paired directly with a single\n>> + packfile, using the pack-order as the object order to hopefully\n>> + compress the bitmaps well using run-length encoding. This could be\n>> + extended to pair a reachability bitmap with a multi-pack-index. If\n>> + the multi-pack-index is extended to store a \"stable object order\"\n>> + (a function Order(hash) = integer that is constant for a given hash,\n> This stable object order doesn't fly well with integrating new packs?\n\nWhen you integrate new packs, the lexicographic order changes as the new \nobjects are inserted into the list. However, if we track a separate \ninteger value (order[obj]) associated with the object, and keep that \nconstant, we can track a stable order for an object across multiple \ngenerations of MIDX files. New objects are assigned order values larger \nthan the previous order values. We can select a \"good\" ordering of the \nnew objects as we extend the list.\n\nThis requires a new chunk in the file format. It also helps to store the \nreverse-lookup lex[i] which provides the lexicographic position of the \nobject 'obj' with stable-order order[obj] == i.\n\nI'm being intentionally vague in this document to hint towards a \nvaluable feature without giving robust details of something that may \nnever get built. But, I do think this is one feature of the MIDX that \nwould be of the most value for services that use Git as a server \nprocess, as it allows the reachability bitmap to be set to this stable \norder instead of a single pack order. This is speculation on my part, as \nI don't know how such servers are maintained in the background.\n\n>\n>> + even as the multi-pack-index is updated) then a reachability bitmap\n>> + could point to a multi-pack-index and be updated independently.\n>> +\n>> +- Packfiles can be marked as \"special\" using empty files that share\n>> + the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n>> + We can add an optional chunk of data to the multi-pack-index that\n>> + records flags of information about the packfiles. This allows new\n>> + states, such as 'repacked' or 'redeltified', that can help with\n>> + pack maintenance in a multi-pack environment. It may also be\n>> + helpful to organize packfiles by object type (commit, tree, blob,\n>> + etc.) and use this metadata to help that maintenance.\n>> +\n>> +- The partial clone feature records special \"promisor\" packs that\n>> + may point to objects that are not stored locally, but available\n>> + on request to a server. The multi-pack-index does not currently\n>> + track these promisor packs.\n\n"},{"id":"350412","messageId":"a0dbd78c-507b-3005-c2eb-2fa82e7d29ec@gmail.com","threadId":"48668","inReplyTo":"CAGZ79kaDiXnAuRym3HHQPEZh9z0-sPL5V4Yur5-_RojsyLJsYA@mail.gmail.com","subject":"Re: [PATCH 02/23] midx: add midx format details to pack-format.txt","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-18T19:01:53Z","receivedAt":"2018-06-18T19:02:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/11/2018 3:19 PM, Stefan Beller wrote:\n> Hi Derrick,\n> On Thu, Jun 7, 2018 at 7:03 AM Derrick Stolee wrote:\n>> The multi-pack-index (MIDX) feature generalizes the existing pack-\n>> index (IDX) feature by indexing objects across multiple pack-files.\n>>\n>> Describe the basic file format, using a 12-byte header followed by\n>> a lookup table for a list of \"chunks\" which will be described later.\n>> The file ends with a footer containing a checksum using the hash\n>> algorithm.\n>>\n>> The header allows later versions to create breaking changes by\n>> advancing the version number. We can also change the hash algorithm\n>> using a different version value.\n>>\n>> We will add the individual chunk format information as we introduce\n>> the code that writes that information.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/technical/pack-format.txt | 49 +++++++++++++++++++++++++\n>> 1 file changed, 49 insertions(+)\n>>\n>> diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\n>> index 70a99fd142..17666b4bfc 100644\n>> --- a/Documentation/technical/pack-format.txt\n>> +++ b/Documentation/technical/pack-format.txt\n>> @@ -252,3 +252,52 @@ Pack file entry: <+\n>> corresponding packfile.\n>>\n>> 20-byte SHA-1-checksum of all of the above.\n>> +\n>> +== midx-*.midx files have the following format:\n>> +\n>> +The meta-index files refer to multiple pack-files and loose objects.\n> So is it meta or multi?\n\nGood catch. We were calling this the meta-index internally before \nchanging to \"multi-pack-index\" (helps to not change the acronym).\n\n>\n>> +In order to allow extensions that add extra data to the MIDX, we organize\n>> +the body into \"chunks\" and provide a lookup table at the beginning of the\n>> +body. The header includes certain length values, such as the number of packs,\n>> +the number of base MIDX files, hash lengths and types.\n>> +\n>> +All 4-byte numbers are in network order.\n>> +\n>> +HEADER:\n>> +\n>> + 4-byte signature:\n>> + The signature is: {'M', 'I', 'D', 'X'}\n>> +\n>> + 1-byte version number:\n>> + Git only writes or recognizes version 1\n>> +\n>> + 1-byte Object Id Version\n>> + Git only writes or recognizes verion 1 (SHA-1)\n> s/verion/version/\n>\n>> + 1-byte number (C) of \"chunks\"\n>> +\n>> + 1-byte number (I) of base multi-pack-index files:\n>> + This value is currently always zero.\n> Oh? Are meta-index and multi-index files different things?\n\nNot intended to be different things, but this number is related to \nmaking the feature incremental.\n\n>\n>> + 4-byte number (P) of pack files\n>> +\n>> +CHUNK LOOKUP:\n>> +\n>> + (C + 1) * 12 bytes providing the chunk offsets:\n>> + First 4 bytes describe chunk id. Value 0 is a terminating label.\n>> + Other 8 bytes provide offset in current file for chunk to start.\n>> + (Chunks are provided in file-order, so you can infer the length\n>> + using the next chunk position if necessary.)\n> It is so nice to have the header also have 12 bytes, so it fits right into the\n> lookup table. So an alternative point of view:\n>\n> If a chunk needs to store more than 8 bytes, we'll have an offset after\n> the first 4 bytes that describe the chunk, otherwise you can store the 8 bytes\n> of information directly after the 4 bytes.\n> \"MIDX\" is a special chunk and must come first (does it?) and only once\n> as it contains the version number.\n\nThis sounds feasible, but unnecessarily complicated. I don't think any \nother chunk will be this small.\n\n>> + The remaining data in the body is described one chunk at a time, and\n>> + these chunks may be given in any order. Chunks are required unless\n>> + otherwise specified.\n>> +\n>> +CHUNK DATA:\n>> +\n>> + (This section intentionally left incomplete.)\n>> +\n>> +TRAILER:\n>> +\n>> + H-byte HASH-checksum of all of the above.\n> This means we have to rehash the whole file for updating its contents.\n> okay.\n\n"},{"id":"350415","messageId":"b0484a3f-95ab-f0c1-0dc5-3c75c766ff6e@gmail.com","threadId":"48668","inReplyTo":"CACsJy8Ca4B_xjpuTU=sFS+fOEin5fFiSmH+Z7JKjoUA6ZMnv4A@mail.gmail.com","subject":"Re: [PATCH 03/23] midx: add midx builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-18T19:23:13Z","receivedAt":"2018-06-18T19:23:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/7/2018 1:20 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n>> diff --git a/Documentation/git-midx.txt b/Documentation/git-midx.txt\n>> new file mode 100644\n>> index 0000000000..2bd886f1a2\n>> --- /dev/null\n>> +++ b/Documentation/git-midx.txt\n>> @@ -0,0 +1,29 @@\n>> +git-midx(1)\n>> +============\n>> +\n>> +NAME\n>> +----\n>> +git-midx - Write and verify multi-pack-indexes (MIDX files).\n> No full stop. This head line is collected automatically with others\n> and its having a full stop while the rest does not looks strange/\n>\n>> diff --git a/builtin/midx.c b/builtin/midx.c\n>> new file mode 100644\n>> index 0000000000..59ea92178f\n>> --- /dev/null\n>> +++ b/builtin/midx.c\n>> @@ -0,0 +1,38 @@\n>> +#include \"builtin.h\"\n>> +#include \"cache.h\"\n>> +#include \"config.h\"\n>> +#include \"git-compat-util.h\"\n> You only need either cache.h or git-compat-util.h. If cache.h is here,\n> git-compat-util can be removed.\n>\n>> +#include \"parse-options.h\"\n>> +\n>> +static char const * const builtin_midx_usage[] ={\n>> + N_(\"git midx [--object-dir ]\"),\n>> + NULL\n>> +};\n>> +\n>> +static struct opts_midx {\n>> + const char *object_dir;\n>> +} opts;\n>> +\n>> +int cmd_midx(int argc, const char **argv, const char *prefix)\n>> +{\n>> + static struct option builtin_midx_options[] = {\n>> + { OPTION_STRING, 0, \"object-dir\", &opts.object_dir,\n> For paths (including dir), OPTION_FILENAME may be a better option to\n> handle correctly when the command is run in a subdir. See df217ed643\n> (parse-opts: add OPT_FILENAME and transition builtins - 2009-05-23)\n> for more info.\nThanks for the pointer!\n\n>\n>> + N_(\"dir\"),\n>> + N_(\"The object directory containing set of packfile and pack-index pairs.\") },\n> Other help strings do not have full stop either (I only checked a\n> couple commands though)\n>\n> Also, doesn't OPT_STRING() work here too (if you avoid OPTION_FILENAME\n> for some reason)?\n>\n>> + OPT_END(),\n>> + };\n>> +\n>> + if (argc == 2 && !strcmp(argv[1], \"-h\"))\n>> + usage_with_options(builtin_midx_usage, builtin_midx_options);\n>> +\n>> + git_config(git_default_config, NULL);\n>> +\n>> + argc = parse_options(argc, argv, prefix,\n>> + builtin_midx_options,\n>> + builtin_midx_usage, 0);\n>> +\n>> + if (!opts.object_dir)\n>> + opts.object_dir = get_object_directory();\n>> +\n>> + return 0;\n>> +}\n>> diff --git a/git.c b/git.c\n>> index c2f48d53dd..400fadd677 100644\n>> --- a/git.c\n>> +++ b/git.c\n>> @@ -503,6 +503,7 @@ static struct cmd_struct commands[] = {\n>> { \"merge-recursive-theirs\", cmd_merge_recursive, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n>> { \"merge-subtree\", cmd_merge_recursive, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n>> { \"merge-tree\", cmd_merge_tree, RUN_SETUP | NO_PARSEOPT },\n>> + { \"midx\", cmd_midx, RUN_SETUP },\n> If it's a plumbing and can take an --object-dir, then I don't think\n> you should require it to run in a repo (with RUN_SETUP).\n> RUN_SETUP_GENTLY may be better. You could even leave it empty here and\n> only call setup_git_directory() only when --object-dir is not set.\n\nI agree. Good point. This could be run to maintain an alternate without \nany .git folder.\n\n>\n>> { \"mktag\", cmd_mktag, RUN_SETUP | NO_PARSEOPT },\n>> { \"mktree\", cmd_mktree, RUN_SETUP },\n>> { \"mv\", cmd_mv, RUN_SETUP | NEED_WORK_TREE },\n\n"},{"id":"350418","messageId":"160f10c8-3016-8466-22dd-14cbd1a2c2f6@gmail.com","threadId":"48668","inReplyTo":"CAGZ79kaH7XzbtPZqVwGgP774GJNxF7ZsHqLzWEmb2NLSxG3Njg@mail.gmail.com","subject":"Re: [PATCH 03/23] midx: add midx builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-18T19:40:44Z","receivedAt":"2018-06-18T19:40:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/11/2018 5:02 PM, Stefan Beller wrote:\n> Hi Derrick,\n> On Thu, Jun 7, 2018 at 7:03 AM Derrick Stolee wrote:\n>> This new 'git midx' builtin will be the plumbing access for writing,\n>> reading, and checking multi-pack-index (MIDX) files. The initial\n>> implementation is a no-op.\n> Let's talk about the name for a second:\n>\n> .idx files are written by git-index-pack or as part of\n> git-pack-objects (which just calls write_idx_file as part\n> of finish_tmp_packfile), and the name actually suggests\n> it writes the index files. I have a hard time understanding\n> what the git-midx command does[1].\n>\n> With both commit graph as well as multi index we introduce\n> a command that is centered around that concept (similar to\n> git-remote or git-config that are centered around a concept,\n> that is closely resembled by a file), but for indexes for packs\n> it was integrated differently into Git. So I am not sure if I want\n> to suggest to integrate it into the packfile commands as that\n> doesn't really fit. But maybe we can have a name that is human\n> readable instead of the file suffix? Maybe\n>\n> git multi-pack-index ?\n>\n> I suppose that eventually this command is not really used by\n> users as it will be used by other porcelain commands in the\n> background or even as part of repack/gc so I am not worried\n> about a long name, but I'd be more worried about understandability.\n\nI'll use \"git multi-pack-index\" in v2. I'll keep \"midx.c\" in the root, \nthough, if that is OK.\n\n> [1] While these names are not perfect for the layman, it is okay?\n> I am sure you are aware of https://git-man-page-generator.lokaltog.net/\n\nI was not, and enjoyed that quite a bit.\n\nThanks,\n-Stolee\n\n>\n>\n>> new file mode 100644\n>> index 0000000000..2bd886f1a2\n>> --- /dev/null\n>> +++ b/Documentation/git-midx.txt\n>> @@ -0,0 +1,29 @@\n>> +git-midx(1)\n>> +============\n>> +\n>> +NAME\n>> +----\n>> +git-midx - Write and verify multi-pack-indexes (MIDX files).\n> The reading is done as part of all other commands.\n\nI like to think the 'read' verb is a subset of \"verify\" because we are \nchecking for information about the MIDX, and mostly for tests or debugging.\n\n>\n>> +\n>> +\n>> +SYNOPSIS\n>> +--------\n>> +[verse]\n>> +'git midx' [--object-dir ]\n>> +\n>> +DESCRIPTION\n>> +-----------\n>> +Write or verify a MIDX file.\n>> +\n>> +OPTIONS\n>> +-------\n>> +\n>> +--object-dir ::\n>> + Use given directory for the location of Git objects. We check\n>> + /packs/multi-pack-index for the current MIDX file, and\n>> + /packs for the pack-files to index.\n>> +\n>> +\n> Maybe we could have a SEE ALSO section that points at\n> the explanation of multi index files?\n> (c.f. man git-submodule that has a SEE ALSO\n> gitsubmodules(7), gitmodules(5) explaining concepts(7)\n> and the file(5))\n>\n> But as this is plumbing and users should not need to worry about it\n> this is optional, I would think.\n\nThe design document is also in 'Documentation/technical' instead of just \n'Documentation/'. Do we have a pattern of linking to the technical \ndocuments?\n\nThanks,\n-Stolee\n"},{"id":"350419","messageId":"CAGZ79ka+RA2jUb-VmDMGwCN80LARTz1Ku5ZMP-PbabEhpNtGCQ@mail.gmail.com","threadId":"48668","inReplyTo":"a0dbd78c-507b-3005-c2eb-2fa82e7d29ec@gmail.com","subject":"Re: [PATCH 02/23] midx: add midx format details to pack-format.txt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-06-18T19:41:38Z","receivedAt":"2018-06-18T19:41:53Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> >> + (C + 1) * 12 bytes providing the chunk offsets:\n> >> + First 4 bytes describe chunk id. Value 0 is a terminating label.\n> >> + Other 8 bytes provide offset in current file for chunk to start.\n> >> + (Chunks are provided in file-order, so you can infer the length\n> >> + using the next chunk position if necessary.)\n> > It is so nice to have the header also have 12 bytes, so it fits right into the\n> > lookup table. So an alternative point of view:\n> >\n> > If a chunk needs to store more than 8 bytes, we'll have an offset after\n> > the first 4 bytes that describe the chunk, otherwise you can store the 8 bytes\n> > of information directly after the 4 bytes.\n> > \"MIDX\" is a special chunk and must come first (does it?) and only once\n> > as it contains the version number.\n>\n> This sounds feasible, but unnecessarily complicated. I don't think any\n> other chunk will be this small.\n\nI was just writing it as a way to test if I really understood what you said\nin the doc, not as a suggestion to incorporate it.\n\nStefan\n"},{"id":"350422","messageId":"CAGZ79kbb699FHoD1X5gXjtBv0r7zpd5PRuh1R2FQ8ABySV8znQ@mail.gmail.com","threadId":"48668","inReplyTo":"160f10c8-3016-8466-22dd-14cbd1a2c2f6@gmail.com","subject":"Re: [PATCH 03/23] midx: add midx builtin","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-06-18T19:55:18Z","receivedAt":"2018-06-18T19:55:33Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> > But as this is plumbing and users should not need to worry about it\n> > this is optional, I would think.\n>\n> The design document is also in 'Documentation/technical' instead of just\n> 'Documentation/'. Do we have a pattern of linking to the technical\n> documents?\n\nApparently we do (and I was not aware of it):\n\n $ git -C Documentation/ grep link:technical\n git-credential.txt:23:link:technical/api-credentials.html[the Git\ncredential API] for more\n git.txt:839:link:technical/api-index.html[Git API documentation].\n gitcredentials.txt:184:link:technical/api-credentials.html[credentials\nAPI] for details.\n technical/http-protocol.txt:517:link:technical/pack-protocol.html\n technical/http-protocol.txt:518:link:technical/protocol-capabilities.html\n user-manual.txt:3220:found in link:technical/pack-format.html[pack format].\n"},{"id":"350423","messageId":"87d4a3ef-cc42-daef-66cb-46ae0f4d7ffe@gmail.com","threadId":"48668","inReplyTo":"CAGZ79kbb699FHoD1X5gXjtBv0r7zpd5PRuh1R2FQ8ABySV8znQ@mail.gmail.com","subject":"Re: [PATCH 03/23] midx: add midx builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-18T19:58:07Z","receivedAt":"2018-06-18T19:58:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/18/2018 3:55 PM, Stefan Beller wrote:\n>>> But as this is plumbing and users should not need to worry about it\n>>> this is optional, I would think.\n>> The design document is also in 'Documentation/technical' instead of just\n>> 'Documentation/'. Do we have a pattern of linking to the technical\n>> documents?\n> Apparently we do (and I was not aware of it):\n>\n> $ git -C Documentation/ grep link:technical\n> git-credential.txt:23:link:technical/api-credentials.html[the Git\n> credential API] for more\n> git.txt:839:link:technical/api-index.html[Git API documentation].\n> gitcredentials.txt:184:link:technical/api-credentials.html[credentials\n> API] for details.\n> technical/http-protocol.txt:517:link:technical/pack-protocol.html\n> technical/http-protocol.txt:518:link:technical/protocol-capabilities.html\n> user-manual.txt:3220:found in link:technical/pack-format.html[pack format].\n\nThanks! I'll add some links.\n"},{"id":"350471","messageId":"fa036efb-2164-ee0a-8cbb-4144487b6617@gmail.com","threadId":"48668","inReplyTo":"CACsJy8ALMjiyjcEdFPnR7GTzBVpqVB72VQNkpUWrj9p4nm-OGg@mail.gmail.com","subject":"Re: [PATCH 05/23] midx: write header information to lockfile","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-19T12:54:28Z","receivedAt":"2018-06-19T12:54:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/12/2018 11:00 AM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 7:01 PM Derrick Stolee wrote:\n>> diff --git a/midx.c b/midx.c\n>> index 616af66b13..3e55422a21 100644\n>> --- a/midx.c\n>> +++ b/midx.c\n>> @@ -1,9 +1,62 @@\n>> #include \"git-compat-util.h\"\n>> #include \"cache.h\"\n>> #include \"dir.h\"\n>> +#include \"csum-file.h\"\n>> +#include \"lockfile.h\"\n>> #include \"midx.h\"\n>>\n>> +#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n>> +#define MIDX_VERSION 1\n>> +#define MIDX_HASH_VERSION 1 /* SHA-1 */\n> ...\n>> +static size_t write_midx_header(struct hashfile *f,\n>> + unsigned char num_chunks,\n>> + uint32_t num_packs)\n>> +{\n>> + char byte_values[4];\n>> + hashwrite_be32(f, MIDX_SIGNATURE);\n>> + byte_values[0] = MIDX_VERSION;\n>> + byte_values[1] = MIDX_HASH_VERSION;\n> Quoting from \"State of NewHash work, future directions, and discussion\" [1]\n>\n> * If you need to serialize an algorithm identifier into your data\n> format, use the format_id field of struct git_hash_algo. It's\n> designed specifically for that purpose.\n>\n> [1] https://public-inbox.org/git/20180612024252.GA141166@aiede.svl.corp.google.com/T/#m5fdd09dcaf31266c45343fb6c0beaaa3e928bc60\n\nThanks! I'll also use the_hash_algo->rawsz to infer the length of the \nhash function.\n"},{"id":"350476","messageId":"CACsJy8B+9WnWJASkd19BSjRdpAECo7Kva8OTC_ZbjiMH4wXi-w@mail.gmail.com","threadId":"48668","inReplyTo":"fa036efb-2164-ee0a-8cbb-4144487b6617@gmail.com","subject":"Re: [PATCH 05/23] midx: write header information to lockfile","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-19T14:59:14Z","receivedAt":"2018-06-19T14:59:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jun 19, 2018 at 2:54 PM Derrick Stolee wrote:\n>\n> On 6/12/2018 11:00 AM, Duy Nguyen wrote:\n> > On Thu, Jun 7, 2018 at 7:01 PM Derrick Stolee wrote:\n> >> diff --git a/midx.c b/midx.c\n> >> index 616af66b13..3e55422a21 100644\n> >> --- a/midx.c\n> >> +++ b/midx.c\n> >> @@ -1,9 +1,62 @@\n> >> #include \"git-compat-util.h\"\n> >> #include \"cache.h\"\n> >> #include \"dir.h\"\n> >> +#include \"csum-file.h\"\n> >> +#include \"lockfile.h\"\n> >> #include \"midx.h\"\n> >>\n> >> +#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n> >> +#define MIDX_VERSION 1\n> >> +#define MIDX_HASH_VERSION 1 /* SHA-1 */\n> > ...\n> >> +static size_t write_midx_header(struct hashfile *f,\n> >> + unsigned char num_chunks,\n> >> + uint32_t num_packs)\n> >> +{\n> >> + char byte_values[4];\n> >> + hashwrite_be32(f, MIDX_SIGNATURE);\n> >> + byte_values[0] = MIDX_VERSION;\n> >> + byte_values[1] = MIDX_HASH_VERSION;\n> > Quoting from \"State of NewHash work, future directions, and discussion\" [1]\n> >\n> > * If you need to serialize an algorithm identifier into your data\n> > format, use the format_id field of struct git_hash_algo. It's\n> > designed specifically for that purpose.\n> >\n> > [1] https://public-inbox.org/git/20180612024252.GA141166@aiede.svl.corp.google.com/T/#m5fdd09dcaf31266c45343fb6c0beaaa3e928bc60\n>\n> Thanks! I'll also use the_hash_algo->rawsz to infer the length of the\n> hash function.\n\nBTW, since you're the author of commit-graph.c and may notice it has\nthe same problem. Don't touch that code. Brian already has some WIP\nchanges [1]. We just make sure new code does not add extra work for\nhim. I expect he'll send all those patches out soon.\n\n[1] https://github.com/bk2204/git/commit/3f9031e06cfb21534eb7dfff7b54e7598ac1149f\n\n-- \nDuy\n"},{"id":"350479","messageId":"a026d956-eb93-7034-dfd9-25d94e7dce84@gmail.com","threadId":"48668","inReplyTo":"CACsJy8B+9WnWJASkd19BSjRdpAECo7Kva8OTC_ZbjiMH4wXi-w@mail.gmail.com","subject":"Re: [PATCH 05/23] midx: write header information to lockfile","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-19T15:24:20Z","receivedAt":"2018-06-19T15:24:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/19/2018 10:59 AM, Duy Nguyen wrote:\n> On Tue, Jun 19, 2018 at 2:54 PM Derrick Stolee wrote:\n>> On 6/12/2018 11:00 AM, Duy Nguyen wrote:\n>>> On Thu, Jun 7, 2018 at 7:01 PM Derrick Stolee wrote:\n>>>> diff --git a/midx.c b/midx.c\n>>>> index 616af66b13..3e55422a21 100644\n>>>> --- a/midx.c\n>>>> +++ b/midx.c\n>>>> @@ -1,9 +1,62 @@\n>>>> #include \"git-compat-util.h\"\n>>>> #include \"cache.h\"\n>>>> #include \"dir.h\"\n>>>> +#include \"csum-file.h\"\n>>>> +#include \"lockfile.h\"\n>>>> #include \"midx.h\"\n>>>>\n>>>> +#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n>>>> +#define MIDX_VERSION 1\n>>>> +#define MIDX_HASH_VERSION 1 /* SHA-1 */\n>>> ...\n>>>> +static size_t write_midx_header(struct hashfile *f,\n>>>> + unsigned char num_chunks,\n>>>> + uint32_t num_packs)\n>>>> +{\n>>>> + char byte_values[4];\n>>>> + hashwrite_be32(f, MIDX_SIGNATURE);\n>>>> + byte_values[0] = MIDX_VERSION;\n>>>> + byte_values[1] = MIDX_HASH_VERSION;\n>>> Quoting from \"State of NewHash work, future directions, and discussion\" [1]\n>>>\n>>> * If you need to serialize an algorithm identifier into your data\n>>> format, use the format_id field of struct git_hash_algo. It's\n>>> designed specifically for that purpose.\n>>>\n>>> [1] https://public-inbox.org/git/20180612024252.GA141166@aiede.svl.corp.google.com/T/#m5fdd09dcaf31266c45343fb6c0beaaa3e928bc60\n>> Thanks! I'll also use the_hash_algo->rawsz to infer the length of the\n>> hash function.\n> BTW, since you're the author of commit-graph.c and may notice it has\n> the same problem. Don't touch that code. Brian already has some WIP\n> changes [1]. We just make sure new code does not add extra work for\n> him. I expect he'll send all those patches out soon.\n>\n> [1] https://github.com/bk2204/git/commit/3f9031e06cfb21534eb7dfff7b54e7598ac1149f\n\nThanks for the link. It seems he is creating an oid_version() method \nthat returns a 1-byte version for the hash version instead of the 4-byte \nsignature of the_hash_algo->format_id. I look forward to incorporating \nthat into the MIDX format. I'll keep my macros for now, as we work out \nthe other details, and while Brain's patch is cooking.\n\nThanks,\n-Stolee\n"},{"id":"350561","messageId":"3827f4ce-7f78-6566-d9c8-1830afb3c09c@gmail.com","threadId":"48668","inReplyTo":"CACsJy8A3qVRXrQ9YiiN3Ggrcc5XFJWaWRrTbdO8E_48yRmzrqQ@mail.gmail.com","subject":"Re: [PATCH 06/23] midx: struct midxed_git and 'read' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-20T13:13:36Z","receivedAt":"2018-06-20T13:13:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/7/2018 1:54 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n>> As we build the multi-pack-index feature by adding chunks at a time,\n>> we want to test that the data is being written correctly.\n>>\n>> Create struct midxed_git to store an in-memory representation of a\n> A word play on 'packed_git'? Amusing. Some more descriptive name would\n> be better though. midxed looks almost like random letters thrown\n> together.\n\nI'll use 'struct multi_pack_index'.\n\n>\n>> multi-pack-index and a memory-map of the binary file. Initialize this\n>> struct in load_midxed_git(object_dir).\n>> +static int read_midx_file(const char *object_dir)\n>> +{\n>> + struct midxed_git *m = load_midxed_git(object_dir);\n>> +\n>> + if (!m)\n>> + return 0;\n> This looks like an error case, please don't just return zero,\n> typically used to say \"success\". I don't know if this command stays\n> \"for debugging purposes\" until the end. Of course in that case it does\n> not really matter.\n\nIt is intended for debugging and testing. Generally, it is not an error \nto not have a MIDX in an object directory.\n\n>> +struct midxed_git *load_midxed_git(const char *object_dir)\n>> +{\n>> + struct midxed_git *m;\n>> + int fd;\n>> + struct stat st;\n>> + size_t midx_size;\n>> + void *midx_map;\n>> + const char *midx_name = get_midx_filename(object_dir);\n> mem leak? This function returns allocated memory if I remember correctly.\n>\n>> +\n>> + fd = git_open(midx_name);\n>> + if (fd < 0)\n>> + return NULL;\n> do an error_errno() so we know what went wrong at least.\n>\n>> + if (fstat(fd, &st)) {\n>> + close(fd);\n>> + return NULL;\n> same here, we should know why fstat() fails.\n>\n>> + }\n>> + midx_size = xsize_t(st.st_size);\n>> +\n>> + if (midx_size < MIDX_MIN_SIZE) {\n>> + close(fd);\n>> + die(\"multi-pack-index file %s is too small\", midx_name);\n> _()\n>\n> The use of die() should be discouraged though. Many people still try\n> (or wish) to libify code and new die() does not help. I think error()\n> here would be enough then you can return NULL. Or you can go fancier\n> and store the error string in a strbuf like refs code.\n>\n>> + }\n>> +\n>> + midx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n>> +\n>> + m = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n>> + strcpy(m->object_dir, object_dir);\n>> + m->data = midx_map;\n>> +\n>> + m->signature = get_be32(m->data);\n>> + if (m->signature != MIDX_SIGNATURE) {\n>> + error(\"multi-pack-index signature %X does not match signature %X\",\n>> + m->signature, MIDX_SIGNATURE);\n> _(). Maybe 0x%08x instead of %x\n>\n>> + goto cleanup_fail;\n>> + }\n>> +\n>> + m->version = *(m->data + 4);\n> m->data[4] instead? shorter and easier to understand.\n>\n> Same comment on \"*(m->data + x)\" and error() without _() for the rest.\n>\n>> + if (m->version != MIDX_VERSION) {\n>> + error(\"multi-pack-index version %d not recognized\",\n>> + m->version);\n> _()\n>> + goto cleanup_fail;\n>> + }\n>> +\n>> + m->hash_version = *(m->data + 5);\n> m->data[5]\n>\n>> +cleanup_fail:\n>> + FREE_AND_NULL(m);\n>> + munmap(midx_map, midx_size);\n>> + close(fd);\n>> + exit(1);\n> It's bad enough that you die() but exit() in this code seems too much.\n> Please just return NULL and let the caller handle the error.\n\nWill do.\n\n>\n>> diff --git a/midx.h b/midx.h\n>> index 3a63673952..a1d18ed991 100644\n>> --- a/midx.h\n>> +++ b/midx.h\n>> @@ -1,4 +1,13 @@\n>> +#ifndef MIDX_H\n>> +#define MIDX_H\n>> +\n>> +#include \"git-compat-util.h\"\n>> #include \"cache.h\"\n>> +#include \"object-store.h\"\n> I don't really think you need object-store here (git-compat-util.h\n> too). \"struct mixed_git;\" would be enough for load_midxed_git\n> declaration below.\n>\n>> #include \"packfile.h\"\n>>\n>> +struct midxed_git *load_midxed_git(const char *object_dir);\n>> +\n>> int write_midx_file(const char *object_dir);\n>> +\n>> +#endif\n>> diff --git a/object-store.h b/object-store.h\n>> index d683112fd7..77cb82621a 100644\n>> --- a/object-store.h\n>> +++ b/object-store.h\n>> @@ -84,6 +84,25 @@ struct packed_git {\n>> char pack_name[FLEX_ARRAY]; /* more */\n>> };\n>>\n>> +struct midxed_git {\n>> + struct midxed_git *next;\n> Do we really have multiple midx files?\n\nThere is one per object directory currently, but you may have one \nlocally and one in each of your alternates. I do need to double-check \nthat we populate this list later in the series. (And I'll remove it from \nthis commit and save it for when it is needed.)\n\n>\n>> +\n>> + int fd;\n>> +\n>> + const unsigned char *data;\n>> + size_t data_len;\n>> +\n>> + uint32_t signature;\n>> + unsigned char version;\n>> + unsigned char hash_version;\n>> + unsigned char hash_len;\n>> + unsigned char num_chunks;\n>> + uint32_t num_packs;\n>> + uint32_t num_objects;\n>> +\n>> + char object_dir[FLEX_ARRAY];\n> Why do you need to keep object_dir when it could be easily retrieved\n> when the repo is available?\n>\n>> +};\n>> +\n>> struct raw_object_store {\n>> /*\n>> * Path to the repository's object store.\n\n"},{"id":"350562","messageId":"2906e8e8-ca32-828f-e8d0-35b0dff7db81@gmail.com","threadId":"48668","inReplyTo":"CACsJy8DG1-YTSBSFkpM8BPcnKmvC-BjjK0W0+EckkdENPC7Ohg@mail.gmail.com","subject":"Re: [PATCH 06/23] midx: struct midxed_git and 'read' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-20T13:33:50Z","receivedAt":"2018-06-20T13:33:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/7/2018 2:31 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n>> diff --git a/Documentation/git-midx.txt b/Documentation/git-midx.txt\n>> index dcaeb1a91b..919283fdd8 100644\n>> --- a/Documentation/git-midx.txt\n>> +++ b/Documentation/git-midx.txt\n>> @@ -23,6 +23,11 @@ OPTIONS\n>> /packs/multi-pack-index for the current MIDX file, and\n>> /packs for the pack-files to index.\n>>\n>> +read::\n>> + When given as the verb, read the current MIDX file and output\n>> + basic information about its contents. Used for debugging\n>> + purposes only.\n> On second thought. If you just need a temporary debugging interface,\n> adding a program in t/helper may be a better option. In the end we\n> might still need 'read' to dump a file out, but we should have some\n> stable output format (and json might be a good choice).\n\nMy intention with this 'read' pattern in the MIDX (and commit-graph) is \ntwo-fold:\n\n1. We can test that we are writing the correct data in our test suite. A \ntest-tool builtin would suffice for this purpose.\n\n2. We can help trouble-shoot users who may be having trouble with their \nMIDX files. Having the subcommand in a plumbing command allows us to do \nthis in the shipped versions of Git.\n\nMaybe this second purpose isn't enough to justify the feature in Git and \nwe should move this to the test-tool, especially with the 'verify' mode \ncoming in a second series. Note that a 'verify' mode doesn't satisfy \nitem (1).\n\nThanks,\n-Stolee\n"},{"id":"350565","messageId":"CACsJy8B4HBrbuog-38o4YA2dKSQejhd4LOQn4+nswi=Wnod-iQ@mail.gmail.com","threadId":"48668","inReplyTo":"2906e8e8-ca32-828f-e8d0-35b0dff7db81@gmail.com","subject":"Re: [PATCH 06/23] midx: struct midxed_git and 'read' subcommand","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-20T15:07:34Z","receivedAt":"2018-06-20T15:08:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jun 20, 2018 at 3:33 PM Derrick Stolee wrote:\n>\n> On 6/7/2018 2:31 PM, Duy Nguyen wrote:\n> > On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n> >> diff --git a/Documentation/git-midx.txt b/Documentation/git-midx.txt\n> >> index dcaeb1a91b..919283fdd8 100644\n> >> --- a/Documentation/git-midx.txt\n> >> +++ b/Documentation/git-midx.txt\n> >> @@ -23,6 +23,11 @@ OPTIONS\n> >> /packs/multi-pack-index for the current MIDX file, and\n> >> /packs for the pack-files to index.\n> >>\n> >> +read::\n> >> + When given as the verb, read the current MIDX file and output\n> >> + basic information about its contents. Used for debugging\n> >> + purposes only.\n> > On second thought. If you just need a temporary debugging interface,\n> > adding a program in t/helper may be a better option. In the end we\n> > might still need 'read' to dump a file out, but we should have some\n> > stable output format (and json might be a good choice).\n>\n> My intention with this 'read' pattern in the MIDX (and commit-graph) is\n> two-fold:\n>\n> 1. We can test that we are writing the correct data in our test suite. A\n> test-tool builtin would suffice for this purpose.\n>\n> 2. We can help trouble-shoot users who may be having trouble with their\n> MIDX files. Having the subcommand in a plumbing command allows us to do\n> this in the shipped versions of Git.\n>\n> Maybe this second purpose isn't enough to justify the feature in Git and\n> we should move this to the test-tool, especially with the 'verify' mode\n> coming in a second series. Note that a 'verify' mode doesn't satisfy\n> item (1).\n\nYeah I think normally we just have some \"fsck\" thing to verify when\nthings go bad. If you need more than that I think you just ask the\nuser to send the .midx to you (with full understanding of potentially\nrevealing confidential info and stuff). It'll be faster than\ninstructing them to \"run this command\", \"ok, run another command\"....\nI thought of suggesting a command to dump the midx file in readable\nform (like json), but I think if fsck fails then chances of that\ncommand successfully dumping may be very low.\n\nEither way, if the command is meant for troubleshooting, I think it\nshould be added at the end when the whole midx file is implemented and\nunderstood and we see what we need to troubleshoot. Adding small\npieces of changes from patch to patch makes it really hard to see if\nit helps troubleshooting at all, it just helps the first purpose.\n-- \nDuy\n"},{"id":"350571","messageId":"20180620163325.10393-1-dstolee@microsoft.com","threadId":"48668","inReplyTo":"CACsJy8BS1x2yqzongv7iPsBbQxxZ5fO4ZbwdX5BPrzePvpJB=Q@mail.gmail.com","subject":"[PATCH] packfile: generalize pack directory list","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-20T16:33:44Z","receivedAt":"2018-06-20T16:33:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of sharing the pack directory listing with the\nmulti-pack-index, generalize prepare_packed_git_one() into\nfor_each_file_in_pack_dir().\n\nSigned-off-by: Derrick Stolee \n---\n\nDuy,\n\nI think this is what you mean by sharing code between packfile.c and\nmidx.c for reading the packfiles from a pack directory. This does make\nthe code in midx.c much simpler. Is this change worth it?\n\nThis patch could stand on its own, or can be incorporated into the next\nversion of the MIDX series.\n\nThanks,\n-Stolee\n\n packfile.c | 103 +++++++++++++++++++++++++++++++++--------------------\n packfile.h | 6 ++++\n 2 files changed, 71 insertions(+), 38 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 7cd45aa4b2..db61c8813b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -738,13 +738,14 @@ static void report_pack_garbage(struct string_list *list)\n \treport_helper(list, seen_bits, first, list->nr);\n }\n \n-static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tsize_t dirnamelen;\n \tDIR *dir;\n \tstruct dirent *de;\n-\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \n \tstrbuf_addstr(&path, objdir);\n \tstrbuf_addstr(&path, \"/pack\");\n@@ -759,53 +760,79 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n \tstrbuf_addch(&path, '/');\n \tdirnamelen = path.len;\n \twhile ((de = readdir(dir)) != NULL) {\n-\t\tstruct packed_git *p;\n-\t\tsize_t base_len;\n-\n \t\tif (is_dot_or_dotdot(de->d_name))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&path, dirnamelen);\n \t\tstrbuf_addstr(&path, de->d_name);\n \n-\t\tbase_len = path.len;\n-\t\tif (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n-\t\t\t/* Don't reopen a pack we already have. */\n-\t\t\tfor (p = r->objects->packed_git; p;\n-\t\t\t p = p->next) {\n-\t\t\t\tsize_t len;\n-\t\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n-\t\t\t\t len == base_len &&\n-\t\t\t\t !memcmp(p->pack_name, path.buf, len))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tif (p == NULL &&\n-\t\t\t /*\n-\t\t\t * See if it really is a valid .idx file with\n-\t\t\t * corresponding .pack file that we can map.\n-\t\t\t */\n-\t\t\t (p = add_packed_git(path.buf, path.len, local)) != NULL)\n-\t\t\t\tinstall_packed_git(r, p);\n-\t\t}\n-\n-\t\tif (!report_garbage)\n-\t\t\tcontinue;\n-\n-\t\tif (ends_with(de->d_name, \".idx\") ||\n-\t\t ends_with(de->d_name, \".pack\") ||\n-\t\t ends_with(de->d_name, \".bitmap\") ||\n-\t\t ends_with(de->d_name, \".keep\") ||\n-\t\t ends_with(de->d_name, \".promisor\"))\n-\t\t\tstring_list_append(&garbage, path.buf);\n-\t\telse\n-\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path.buf);\n+\t\tfn(path.buf, path.len, de->d_name, data);\n \t}\n+\n \tclosedir(dir);\n-\treport_pack_garbage(&garbage);\n-\tstring_list_clear(&garbage, 0);\n \tstrbuf_release(&path);\n }\n \n+struct prepare_pack_data\n+{\n+\tstruct repository *r;\n+\tstruct string_list *garbage;\n+\tint local;\n+};\n+\n+static void prepare_pack(const char *full_name, size_t full_name_len, const char *file_name, void *_data)\n+{\n+\tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n+\tstruct packed_git *p;\n+\tsize_t base_len = full_name_len;\n+\n+\tif (strip_suffix_mem(full_name, &base_len, \".idx\")) {\n+\t\t/* Don't reopen a pack we already have. */\n+\t\tfor (p = data->r->objects->packed_git; p; p = p->next) {\n+\t\t\tsize_t len;\n+\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n+\t\t\t len == base_len &&\n+\t\t\t !memcmp(p->pack_name, full_name, len))\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (p == NULL &&\n+\t\t /*\n+\t\t * See if it really is a valid .idx file with\n+\t\t * corresponding .pack file that we can map.\n+\t\t */\n+\t\t (p = add_packed_git(full_name, full_name_len, data->local)) != NULL)\n+\t\t\tinstall_packed_git(data->r, p);\n+\t}\n+\n+\tif (!report_garbage)\n+\t return;\n+\n+\tif (ends_with(file_name, \".idx\") ||\n+\t ends_with(file_name, \".pack\") ||\n+\t ends_with(file_name, \".bitmap\") ||\n+\t ends_with(file_name, \".keep\") ||\n+\t ends_with(file_name, \".promisor\"))\n+\t\tstring_list_append(data->garbage, full_name);\n+\telse\n+\t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\n+}\n+\n+static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+{\n+\tstruct prepare_pack_data data;\n+\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n+\n+\tdata.r = r;\n+\tdata.garbage = &garbage;\n+\tdata.local = local;\n+\n+\tfor_each_file_in_pack_dir(objdir, prepare_pack, &data);\n+\n+\treport_pack_garbage(data.garbage);\n+\tstring_list_clear(data.garbage, 0);\n+}\n+\n static void prepare_packed_git(struct repository *r);\n /*\n * Give a fast, rough count of the number of objects in the repository. This\ndiff --git a/packfile.h b/packfile.h\nindex e0a38aba93..d2ad30300a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -28,6 +28,12 @@ extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n+typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len,\n+\t\t\t\t const char *file_pach, void *data);\n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data);\n+\n /* A hook to report invalid files in pack directory */\n #define PACKDIR_FILE_PACK 1\n #define PACKDIR_FILE_IDX 2\n-- \n2.18.0.rc1\n\n"},{"id":"350572","messageId":"4baf0a39-36ec-56b1-2cc4-400128324c4a@gmail.com","threadId":"48668","inReplyTo":"CACsJy8B4HBrbuog-38o4YA2dKSQejhd4LOQn4+nswi=Wnod-iQ@mail.gmail.com","subject":"Re: [PATCH 06/23] midx: struct midxed_git and 'read' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-20T16:39:29Z","receivedAt":"2018-06-20T16:39:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/20/2018 11:07 AM, Duy Nguyen wrote:\n> On Wed, Jun 20, 2018 at 3:33 PM Derrick Stolee wrote:\n>> On 6/7/2018 2:31 PM, Duy Nguyen wrote:\n>>> On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n>>>> diff --git a/Documentation/git-midx.txt b/Documentation/git-midx.txt\n>>>> index dcaeb1a91b..919283fdd8 100644\n>>>> --- a/Documentation/git-midx.txt\n>>>> +++ b/Documentation/git-midx.txt\n>>>> @@ -23,6 +23,11 @@ OPTIONS\n>>>> /packs/multi-pack-index for the current MIDX file, and\n>>>> /packs for the pack-files to index.\n>>>>\n>>>> +read::\n>>>> + When given as the verb, read the current MIDX file and output\n>>>> + basic information about its contents. Used for debugging\n>>>> + purposes only.\n>>> On second thought. If you just need a temporary debugging interface,\n>>> adding a program in t/helper may be a better option. In the end we\n>>> might still need 'read' to dump a file out, but we should have some\n>>> stable output format (and json might be a good choice).\n>> My intention with this 'read' pattern in the MIDX (and commit-graph) is\n>> two-fold:\n>>\n>> 1. We can test that we are writing the correct data in our test suite. A\n>> test-tool builtin would suffice for this purpose.\n>>\n>> 2. We can help trouble-shoot users who may be having trouble with their\n>> MIDX files. Having the subcommand in a plumbing command allows us to do\n>> this in the shipped versions of Git.\n>>\n>> Maybe this second purpose isn't enough to justify the feature in Git and\n>> we should move this to the test-tool, especially with the 'verify' mode\n>> coming in a second series. Note that a 'verify' mode doesn't satisfy\n>> item (1).\n> Yeah I think normally we just have some \"fsck\" thing to verify when\n> things go bad. If you need more than that I think you just ask the\n> user to send the .midx to you (with full understanding of potentially\n> revealing confidential info and stuff). It'll be faster than\n> instructing them to \"run this command\", \"ok, run another command\"....\n> I thought of suggesting a command to dump the midx file in readable\n> form (like json), but I think if fsck fails then chances of that\n> command successfully dumping may be very low.\n>\n> Either way, if the command is meant for troubleshooting, I think it\n> should be added at the end when the whole midx file is implemented and\n> understood and we see what we need to troubleshoot. Adding small\n> pieces of changes from patch to patch makes it really hard to see if\n> it helps troubleshooting at all, it just helps the first purpose.\n\nI'll abandon point (2) for the later 'verify' patch series. Adding the \ntest helper early allows tests to demonstrate that each patch does the \nright thing, and that we don't miss testing something.\n\nThanks,\n-Stolee\n"},{"id":"350649","messageId":"4082c785-9fb2-7b15-b646-3c658b54abbe@gmail.com","threadId":"48668","inReplyTo":"CACsJy8ARDLeWaWq1_npZq5ARCVgcETpS--YUVLqZkZA52E7Msw@mail.gmail.com","subject":"Re: [PATCH 09/23] midx: write pack names in chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-21T15:25:38Z","receivedAt":"2018-06-21T15:25:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/7/2018 2:26 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n>> @@ -74,6 +80,31 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n>> m->num_chunks = *(m->data + 6);\n>> m->num_packs = get_be32(m->data + 8);\n>>\n>> + for (i = 0; i < m->num_chunks; i++) {\n>> + uint32_t chunk_id = get_be32(m->data + 12 + MIDX_CHUNKLOOKUP_WIDTH * i);\n>> + uint64_t chunk_offset = get_be64(m->data + 16 + MIDX_CHUNKLOOKUP_WIDTH * i);\n> Would be good to reduce magic numbers like 12 and 16, I think you have\n> some header length constants for those already.\n>\n>> + switch (chunk_id) {\n>> + case MIDX_CHUNKID_PACKNAMES:\n>> + m->chunk_pack_names = m->data + chunk_offset;\n>> + break;\n>> +\n>> + case 0:\n>> + die(\"terminating MIDX chunk id appears earlier than expected\");\n> _()\n\nThis die() and others like it are not marked for translation on purpose, \nas they should never be seen by an end-user.\n\n>\n>> + break;\n>> +\n>> + default:\n>> + /*\n>> + * Do nothing on unrecognized chunks, allowing future\n>> + * extensions to add optional chunks.\n>> + */\n> I wrote about the chunk term reminding me of PNG format then deleted\n> it. But it may help to do similar to PNG here. The first letter can\n> let us know if the chunk is optional and can be safely ignored. E.g.\n> uppercase first letter cannot be ignored, lowercase go wild.\n\nThat's an interesting way to think about it. That way you could add a \nnew \"required\" chunk and earlier versions could die() realizing they \ndon't know how to parse that required chunk.\n\nI think for this format, we should update the file version value when a \nrequired chunk is needed.\n\n>\n>> + break;\n>> + }\n>> + }\n>> +\n>> + if (!m->chunk_pack_names)\n>> + die(\"MIDX missing required pack-name chunk\");\n> _()\n>\n>> +\n>> return m;\n>>\n>> cleanup_fail:\n>> @@ -99,18 +130,88 @@ static size_t write_midx_header(struct hashfile *f,\n>> return MIDX_HEADER_SIZE;\n>> }\n>>\n>> +struct pack_pair {\n>> + uint32_t pack_int_id;\n> can this be just pack_id?\n\nSince packfiles are usually named pack-{hash}.pack, I chose to be \nspecific here.\n\n>\n>> + char *pack_name;\n>> +};\n>> +\n>> +static int pack_pair_compare(const void *_a, const void *_b)\n>> +{\n>> + struct pack_pair *a = (struct pack_pair *)_a;\n>> + struct pack_pair *b = (struct pack_pair *)_b;\n>> + return strcmp(a->pack_name, b->pack_name);\n>> +}\n>> +\n>> +static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n>> +{\n>> + uint32_t i;\n>> + struct pack_pair *pairs;\n>> +\n>> + ALLOC_ARRAY(pairs, nr_packs);\n>> +\n>> + for (i = 0; i < nr_packs; i++) {\n>> + pairs[i].pack_int_id = i;\n>> + pairs[i].pack_name = pack_names[i];\n>> + }\n>> +\n>> + QSORT(pairs, nr_packs, pack_pair_compare);\n>> +\n>> + for (i = 0; i < nr_packs; i++) {\n>> + pack_names[i] = pairs[i].pack_name;\n>> + perm[pairs[i].pack_int_id] = i;\n>> + }\n> pairs[] is leaked?\n\nGood catch!\n\n>\n>> +}\n>> +\n>> +static size_t write_midx_pack_names(struct hashfile *f,\n>> + char **pack_names,\n>> + uint32_t num_packs)\n>> +{\n>> + uint32_t i;\n>> + unsigned char padding[MIDX_CHUNK_ALIGNMENT];\n>> + size_t written = 0;\n>> +\n>> + for (i = 0; i < num_packs; i++) {\n>> + size_t writelen = strlen(pack_names[i]) + 1;\n>> +\n>> + if (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n>> + BUG(\"incorrect pack-file order: %s before %s\",\n>> + pack_names[i - 1],\n>> + pack_names[i]);\n>> +\n>> + hashwrite(f, pack_names[i], writelen);\n>> + written += writelen;\n> side note. This pattern happens a lot. It may be a good idea to make\n> hashwrite() return writelen so we can just write\n>\n> written += hashwrite(f, ..., writelen);\n\nIf I change the prototype of hashwrite(), will other callers get \nwarnings about not inspecting the return value (for some build options \non some platforms)?\n\n>\n>> + }\n>> +\n>> + /* add padding to be aligned */\n>> + i = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n>> + if (i < MIDX_CHUNK_ALIGNMENT) {\n>> + bzero(padding, sizeof(padding));\n>> + hashwrite(f, padding, i);\n>> + written += i;\n>> + }\n>> +\n>> + return written;\n>> +}\n>> +\n>> int write_midx_file(const char *object_dir)\n>> {\n>> - unsigned char num_chunks = 0;\n>> + unsigned char cur_chunk, num_chunks = 0;\n>> char *midx_name;\n>> struct hashfile *f;\n>> struct lock_file lk;\n>> struct packed_git **packs = NULL;\n>> + char **pack_names = NULL;\n>> + uint32_t *pack_perm;\n>> uint32_t i, nr_packs = 0, alloc_packs = 0;\n>> + uint32_t alloc_pack_names = 0;\n>> DIR *dir;\n>> struct dirent *de;\n>> struct strbuf pack_dir = STRBUF_INIT;\n>> size_t pack_dir_len;\n>> + uint64_t pack_name_concat_len = 0;\n>> + uint64_t written = 0;\n>> + uint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n>> + uint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n> This long list of local vars may be a good indicator that this\n> function needs split up into smaller ones.\n\nOr the data needs to be grouped into structs, which is happening in my \nlocal branch due to the for_each_file_in_pack_dir() method.\n\n>> midx_name = get_midx_filename(object_dir);\n>> if (safe_create_leading_directories(midx_name)) {\n>> @@ -132,12 +233,14 @@ int write_midx_file(const char *object_dir)\n>> strbuf_addch(&pack_dir, '/');\n>> pack_dir_len = pack_dir.len;\n>> ALLOC_ARRAY(packs, alloc_packs);\n>> + ALLOC_ARRAY(pack_names, alloc_pack_names);\n>> while ((de = readdir(dir)) != NULL) {\n>> if (is_dot_or_dotdot(de->d_name))\n>> continue;\n>>\n>> if (ends_with(de->d_name, \".idx\")) {\n>> ALLOC_GROW(packs, nr_packs + 1, alloc_packs);\n>> + ALLOC_GROW(pack_names, nr_packs + 1, alloc_pack_names);\n>>\n>> strbuf_setlen(&pack_dir, pack_dir_len);\n>> strbuf_addstr(&pack_dir, de->d_name);\n>> @@ -145,21 +248,83 @@ int write_midx_file(const char *object_dir)\n>> packs[nr_packs] = add_packed_git(pack_dir.buf,\n>> pack_dir.len,\n>> 0);\n>> - if (!packs[nr_packs])\n>> + if (!packs[nr_packs]) {\n>> warning(\"failed to add packfile '%s'\",\n>> pack_dir.buf);\n>> - else\n>> - nr_packs++;\n>> + continue;\n>> + }\n>> +\n>> + pack_names[nr_packs] = xstrdup(de->d_name);\n>> + pack_name_concat_len += strlen(de->d_name) + 1;\n>> + nr_packs++;\n>> }\n>> }\n>> +\n>> closedir(dir);\n>> strbuf_release(&pack_dir);\n>>\n>> + if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n>> + pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n>> + (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n>> +\n>> + ALLOC_ARRAY(pack_perm, nr_packs);\n>> + sort_packs_by_name(pack_names, nr_packs, pack_perm);\n>> +\n>> hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n>> f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n>> FREE_AND_NULL(midx_name);\n>>\n>> - write_midx_header(f, num_chunks, nr_packs);\n>> + cur_chunk = 0;\n>> + num_chunks = 1;\n>> +\n>> + written = write_midx_header(f, num_chunks, nr_packs);\n>> +\n>> + chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n>> + chunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n>> +\n>> + cur_chunk++;\n>> + chunk_ids[cur_chunk] = 0;\n>> + chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n>> +\n>> + for (i = 0; i <= num_chunks; i++) {\n>> + if (i && chunk_offsets[i] < chunk_offsets[i - 1])\n>> + BUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n>> + chunk_offsets[i - 1],\n>> + chunk_offsets[i]);\n>> +\n>> + if (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n>> + BUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n>> + chunk_offsets[i]);\n>> +\n>> + hashwrite_be32(f, chunk_ids[i]);\n>> + hashwrite_be32(f, chunk_offsets[i] >> 32);\n>> + hashwrite_be32(f, chunk_offsets[i]);\n>> +\n>> + written += MIDX_CHUNKLOOKUP_WIDTH;\n>> + }\n>> +\n>> + for (i = 0; i < num_chunks; i++) {\n>> + if (written != chunk_offsets[i])\n>> + BUG(\"inccrrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n> incorrect\n>\n>> + chunk_offsets[i],\n>> + written,\n>> + chunk_ids[i]);\n>> +\n>> + switch (chunk_ids[i]) {\n>> + case MIDX_CHUNKID_PACKNAMES:\n>> + written += write_midx_pack_names(f, pack_names, nr_packs);\n>> + break;\n>> +\n>> + default:\n>> + BUG(\"trying to write unknown chunk id %\"PRIx32,\n>> + chunk_ids[i]);\n>> + }\n>> + }\n>> +\n>> + if (written != chunk_offsets[num_chunks])\n>> + BUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n>> + written,\n>> + chunk_offsets[num_chunks]);\n>>\n>> finalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n>> commit_lock_file(&lk);\n>> @@ -170,5 +335,6 @@ int write_midx_file(const char *object_dir)\n>> }\n>>\n>> FREE_AND_NULL(packs);\n>> + FREE_AND_NULL(pack_names);\n> What about the strings in this array? I think they are xstrdup() but I\n> didn't spot them being freed.\n>\n> And maybe just use string_list...\n\nWill do! Thanks.\n"},{"id":"350655","messageId":"72b3e833-0071-ee94-b91e-c558af4927a0@gmail.com","threadId":"48668","inReplyTo":"CACsJy8C0K_3-=S-sV+jfKdRoF0Ta8eafbOT_B1zOfWCftwz8Bw@mail.gmail.com","subject":"Re: [PATCH 10/23] midx: write a lookup into the pack names chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-21T17:23:38Z","receivedAt":"2018-06-21T17:23:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\n\nOn 6/9/2018 12:43 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 7:01 PM Derrick Stolee wrote:\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/technical/pack-format.txt | 5 +++\n>> builtin/midx.c | 7 ++++\n>> midx.c | 56 +++++++++++++++++++++++--\n>> object-store.h | 2 +\n>> t/t5319-midx.sh | 11 +++--\n>> 5 files changed, 75 insertions(+), 6 deletions(-)\n>>\n>> diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\n>> index 2b37be7b33..29bf87283a 100644\n>> --- a/Documentation/technical/pack-format.txt\n>> +++ b/Documentation/technical/pack-format.txt\n>> @@ -296,6 +296,11 @@ CHUNK LOOKUP:\n>>\n>> CHUNK DATA:\n>>\n>> + Packfile Name Lookup (ID: {'P', 'L', 'O', 'O'}) (P * 4 bytes)\n>> + P * 4 bytes storing the offset in the packfile name chunk for\n>> + the null-terminated string containing the filename for the\n>> + ith packfile.\n>> +\n> Commit message is too light on this one. Why does this need to be\n> stored? Isn't the cost of rebuilding this in-core cheap?\n>\n> Adding this chunk on disk in my opinion only adds more burden. Now you\n> have to verify that these offsets actually point to the right place.\nThis is a very good point. I'll drop the chunk and just read the names \ndirectly to construct the array of strings.\n\nThanks,\n-Stolee\n"},{"id":"350656","messageId":"xmqqfu1gt4ye.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"4082c785-9fb2-7b15-b646-3c658b54abbe@gmail.com","subject":"Re: [PATCH 09/23] midx: write pack names in chunk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-21T17:38:49Z","receivedAt":"2018-06-21T17:38:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 6/7/2018 2:26 PM, Duy Nguyen wrote:\n>> On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n>>> @@ -74,6 +80,31 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n>>> m->num_chunks = *(m->data + 6);\n>>> m->num_packs = get_be32(m->data + 8);\n>>>\n>>> + for (i = 0; i < m->num_chunks; i++) {\n>>> + uint32_t chunk_id = get_be32(m->data + 12 + MIDX_CHUNKLOOKUP_WIDTH * i);\n>>> + uint64_t chunk_offset = get_be64(m->data + 16 + MIDX_CHUNKLOOKUP_WIDTH * i);\n>> Would be good to reduce magic numbers like 12 and 16, I think you have\n>> some header length constants for those already.\n>>\n>>> + switch (chunk_id) {\n>>> + case MIDX_CHUNKID_PACKNAMES:\n>>> + m->chunk_pack_names = m->data + chunk_offset;\n>>> + break;\n\n(style: aren't these case arms indented one level too deep)?\n\n>>> + case 0:\n>>> + die(\"terminating MIDX chunk id appears earlier than expected\");\n>> _()\n>\n> This die() and others like it are not marked for translation on\n> purpose, as they should never be seen by an end-user.\n\nShould never be seen because it indicates a software bug, in which\ncase this should be BUG() instead of die()?\n\nOr did we just find a file corruption on the filesystem? If so,\nthen the error is end-user facing and should tell the user something\nthat hints what is going on in the language the user understands, I\nwould guess.\n\n>>> + default:\n>>> + /*\n>>> + * Do nothing on unrecognized chunks, allowing future\n>>> + * extensions to add optional chunks.\n>>> + */\n>> I wrote about the chunk term reminding me of PNG format then deleted\n>> it. But it may help to do similar to PNG here. The first letter can\n>> let us know if the chunk is optional and can be safely ignored. E.g.\n>> uppercase first letter cannot be ignored, lowercase go wild.\n>\n> That's an interesting way to think about it. That way you could add a\n> new \"required\" chunk and earlier versions could die() realizing they\n> don't know how to parse that required chunk.\n\nThat is how the index extension sections work and may be a good\nexample to follow.\n"},{"id":"350659","messageId":"dcd19f97-51de-8254-9977-56ad9a54cbb4@gmail.com","threadId":"48668","inReplyTo":"CACsJy8CJ0GNkciHVVUm_7a_MtG5RnZSWpV_zNAwtvMR8aRq42A@mail.gmail.com","subject":"Re: [PATCH 11/23] midx: sort and deduplicate objects from packfiles","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-21T17:54:11Z","receivedAt":"2018-06-21T17:54:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/9/2018 1:07 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 4:06 PM Derrick Stolee wrote:\n>> Before writing a list of objects and their offsets to a multi-pack-index\n>> (MIDX), we need to collect the list of objects contained in the\n>> packfiles. There may be multiple copies of some objects, so this list\n>> must be deduplicated.\n> Can you just do merge-sort with a slight modification to ignore duplicates?\n\nAre you proposing we consider a multi-way merge of the existing sorted \nlists of packfiles (skipping duplicates)? In my head, this would work \nthis way:\n\n1. Keep an array of positions within each of the pack-indexes for the \n\"current lex-least OID not already in my sorted list\"\n\n2. Scan the list of P pack-indexes to find the lex-least OID among all \ncandidates. Advance the position of that pack-index as we put that OID \nin the list (and advance the position of pack-indexes with duplicates).\n\nThis would have O(P * N) performance, where P is the number of packfiles \nand N is the total number of objects. This gets slightly better when \nthere are duplicates; in the world where we have P identical lists of n \nobjects, then N = n * P and we actually get N steps because we can \nadvance the position on a duplicate value and not revisit duplicates. \nHowever, we do not expect duplicates in this density.\n\nBy adding some complexity to the algorithm, we could sort the \npack-indexes in order of their lex-least OIDs, and update the order as \nwe advance -- or rather use a min-heap to have access to the proper \npack-index. This case is most likely to be valuable when updating a \nlarge MIDX by adding a list of smaller IDX files (which we expect to not \nbe the \"best\" choice for most of the selections). I'm not sure the \ncomplexity is worth it (would need to measure!).\n\nBy concatenating the lists within the fanout values and sorting, we do \n256 sorts of size ~N/256, giving O(N * log(N/256)) performance. This \nmethod also has an extra array of size ~N/200 to store the batches, \nresulting in extra copies being pushed around.\n\nYou've convinced me that your approach may be better, especially in the \ntypical case of adding a small number of packfiles to an existing MIDX \nfile. Some work is needed to be sure it is better in general (such as \nreported cases of 5000 packfiles!). I'll leave a note to revisit this \nbetween v2 and v3.\n\n>\n>> It is possible to artificially get into a state where there are many\n>> duplicate copies of objects. That can create high memory pressure if we\n>> are to create a list of all objects before de-duplication. To reduce\n>> this memory pressure without a significant performance drop,\n>> automatically group objects by the first byte of their object id. Use\n>> the IDX fanout tables to group the data, copy to a local array, then\n>> sort.\n>>\n>> Copy only the de-duplicated entries. Select the duplicate based on the\n>> most-recent modified time of a packfile containing the object.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> midx.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n>> 1 file changed, 138 insertions(+)\n>>\n>> diff --git a/midx.c b/midx.c\n>> index 923acda72e..b20d52713c 100644\n>> --- a/midx.c\n>> +++ b/midx.c\n>> @@ -4,6 +4,7 @@\n>> #include \"csum-file.h\"\n>> #include \"lockfile.h\"\n>> #include \"object-store.h\"\n>> +#include \"packfile.h\"\n>> #include \"midx.h\"\n>>\n>> #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n>> @@ -190,6 +191,140 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n>> }\n>> }\n>>\n>> +static uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)\n>> +{\n>> + const uint32_t *level1_ofs = p->index_data;\n>> +\n>> + if (!level1_ofs) {\n>> + if (open_pack_index(p))\n>> + return 0;\n>> + level1_ofs = p->index_data;\n>> + }\n>> +\n>> + if (p->index_version > 1) {\n>> + level1_ofs += 2;\n>> + }\n>> +\n>> + return ntohl(level1_ofs[value]);\n>> +}\n> Maybe keep this in packfile,c, refactor fanout code in there if\n> necessary, keep .idx file format info in that file instead of\n> spreading out more.\n>\n>> +\n>> +struct pack_midx_entry {\n>> + struct object_id oid;\n>> + uint32_t pack_int_id;\n>> + time_t pack_mtime;\n>> + uint64_t offset;\n>> +};\n>> +\n>> +static int midx_oid_compare(const void *_a, const void *_b)\n>> +{\n>> + struct pack_midx_entry *a = (struct pack_midx_entry *)_a;\n>> + struct pack_midx_entry *b = (struct pack_midx_entry *)_b;\n> Try not to lose \"const\" while typecasting.\n>\n>> + int cmp = oidcmp(&a->oid, &b->oid);\n>> +\n>> + if (cmp)\n>> + return cmp;\n>> +\n>> + if (a->pack_mtime > b->pack_mtime)\n>> + return -1;\n>> + else if (a->pack_mtime < b->pack_mtime)\n>> + return 1;\n>> +\n>> + return a->pack_int_id - b->pack_int_id;\n>> +}\n>> +\n>> +static void fill_pack_entry(uint32_t pack_int_id,\n>> + struct packed_git *p,\n>> + uint32_t cur_object,\n>> + struct pack_midx_entry *entry)\n>> +{\n>> + if (!nth_packed_object_oid(&entry->oid, p, cur_object))\n>> + die(\"failed to located object %d in packfile\", cur_object);\n> _()\n>\n>> +\n>> + entry->pack_int_id = pack_int_id;\n>> + entry->pack_mtime = p->mtime;\n>> +\n>> + entry->offset = nth_packed_object_offset(p, cur_object);\n>> +}\n>> +\n>> +/*\n>> + * It is possible to artificially get into a state where there are many\n>> + * duplicate copies of objects. That can create high memory pressure if\n>> + * we are to create a list of all objects before de-duplication. To reduce\n>> + * this memory pressure without a significant performance drop, automatically\n>> + * group objects by the first byte of their object id. Use the IDX fanout\n>> + * tables to group the data, copy to a local array, then sort.\n>> + *\n>> + * Copy only the de-duplicated entries (selected by most-recent modified time\n>> + * of a packfile containing the object).\n>> + */\n>> +static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n>> + uint32_t *perm,\n>> + uint32_t nr_packs,\n>> + uint32_t *nr_objects)\n>> +{\n>> + uint32_t cur_fanout, cur_pack, cur_object;\n>> + uint32_t nr_fanout, alloc_fanout, alloc_objects, total_objects = 0;\n>> + struct pack_midx_entry *entries_by_fanout = NULL;\n>> + struct pack_midx_entry *deduplicated_entries = NULL;\n>> +\n>> + for (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n>> + if (open_pack_index(p[cur_pack]))\n>> + continue;\n> Is it a big problem if you fail to open .idx for a certain pack?\n> Should we error out and abort instead of continuing on? Later on in\n> the second pack loop code when get_fanout return zero (failure), you\n> don't seem to catch it and skip the pack.\n>\n>> +\n>> + total_objects += p[cur_pack]->num_objects;\n>> + }\n>> +\n>> + /*\n>> + * As we de-duplicate by fanout value, we expect the fanout\n>> + * slices to be evenly distributed, with some noise. Hence,\n>> + * allocate slightly more than one 256th.\n>> + */\n>> + alloc_objects = alloc_fanout = total_objects > 3200 ? total_objects / 200 : 16;\n>> +\n>> + ALLOC_ARRAY(entries_by_fanout, alloc_fanout);\n>> + ALLOC_ARRAY(deduplicated_entries, alloc_objects);\n>> + *nr_objects = 0;\n>> +\n>> + for (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n>> + nr_fanout = 0;\n> Keep variable scope small, declare nr_fanout here instead of at the\n> top of the function.\n>\n>> +\n>> + for (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n>> + uint32_t start = 0, end;\n>> +\n>> + if (cur_fanout)\n>> + start = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n>> + end = get_pack_fanout(p[cur_pack], cur_fanout);\n>> +\n>> + for (cur_object = start; cur_object < end; cur_object++) {\n>> + ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n>> + fill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n>> + nr_fanout++;\n>> + }\n>> + }\n>> +\n>> + QSORT(entries_by_fanout, nr_fanout, midx_oid_compare);\n>> +\n>> + /*\n>> + * The batch is now sorted by OID and then mtime (descending).\n>> + * Take only the first duplicate.\n>> + */\n>> + for (cur_object = 0; cur_object < nr_fanout; cur_object++) {\n>> + if (cur_object && !oidcmp(&entries_by_fanout[cur_object - 1].oid,\n>> + &entries_by_fanout[cur_object].oid))\n>> + continue;\n>> +\n>> + ALLOC_GROW(deduplicated_entries, *nr_objects + 1, alloc_objects);\n>> + memcpy(&deduplicated_entries[*nr_objects],\n>> + &entries_by_fanout[cur_object],\n>> + sizeof(struct pack_midx_entry));\n>> + (*nr_objects)++;\n>> + }\n>> + }\n>> +\n>> + FREE_AND_NULL(entries_by_fanout);\n>> + return deduplicated_entries;\n>> +}\n>> +\n>> static size_t write_midx_pack_lookup(struct hashfile *f,\n>> char **pack_names,\n>> uint32_t nr_packs)\n>> @@ -254,6 +389,7 @@ int write_midx_file(const char *object_dir)\n>> uint64_t written = 0;\n>> uint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n>> uint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n>> + uint32_t nr_entries;\n>>\n>> midx_name = get_midx_filename(object_dir);\n>> if (safe_create_leading_directories(midx_name)) {\n>> @@ -312,6 +448,8 @@ int write_midx_file(const char *object_dir)\n>> ALLOC_ARRAY(pack_perm, nr_packs);\n>> sort_packs_by_name(pack_names, nr_packs, pack_perm);\n>>\n>> + get_sorted_entries(packs, pack_perm, nr_packs, &nr_entries);\n> Intentional ignoring return value (and temporary leaking as a result)\n> should have a least a comment to acknowledge it and save reviewers\n> some head scratching. Or even better, just free it now, even if you\n> don't use it.\n>\n>> +\n>> hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n>> f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n>> FREE_AND_NULL(midx_name);\n>> --\n>> 2.18.0.rc1\n>>\n>\n\n"},{"id":"350665","messageId":"8e7c491c-81d5-abf0-dae6-6d605afb1153@gmail.com","threadId":"48668","inReplyTo":"CACsJy8DW8foQ8+KDdjf4g_3UNi5oSqJyHYo5d9j4cth_yBXDCA@mail.gmail.com","subject":"Re: [PATCH 13/23] midx: write object id fanout chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-21T19:49:23Z","receivedAt":"2018-06-21T19:49:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/9/2018 1:28 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 4:06 PM Derrick Stolee wrote:\n>> @@ -117,9 +123,13 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n>> die(\"MIDX missing required pack lookup chunk\");\n>> if (!m->chunk_pack_names)\n>> die(\"MIDX missing required pack-name chunk\");\n>> + if (!m->chunk_oid_fanout)\n>> + die(\"MIDX missing required OID fanout chunk\");\n> _()\n>\n>> @@ -501,9 +540,13 @@ int write_midx_file(const char *object_dir)\n>> chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_packs * sizeof(uint32_t);\n>>\n>> cur_chunk++;\n>> - chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n>> + chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n> Err.. mistake?\n\nNot a mistake, just a side-effect of inserting the fanout before the lookup.\n\nThe commits are in this order because we need to construct the list \nbefore we build the fanout, but it makes sense to have the smaller \nfanout chunk before the lookup chunk.\n\n>\n>> chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n>>\n>> + cur_chunk++;\n>> + chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n> Same here.\n>\n>> + chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n>> +\n>> cur_chunk++;\n>> chunk_ids[cur_chunk] = 0;\n>> chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n>>\n\n"},{"id":"350667","messageId":"b9d875e2-452d-6492-fd25-9d114af39a25@gmail.com","threadId":"48668","inReplyTo":"CACsJy8CqoKWgd=ekxozaKzRjrU3NN9KUxrspk_iP+zxBVcV5pA@mail.gmail.com","subject":"Re: [PATCH 17/23] midx: read objects from multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-21T20:03:32Z","receivedAt":"2018-06-21T20:03:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/9/2018 1:56 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 6:55 PM Derrick Stolee wrote:\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> midx.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++--\n>> midx.h | 2 ++\n>> object-store.h | 1 +\n>> packfile.c | 8 ++++-\n>> 4 files changed, 104 insertions(+), 3 deletions(-)\n>>\n>> diff --git a/midx.c b/midx.c\n>> index 5e9290ca8f..6eca8f1b12 100644\n>> --- a/midx.c\n>> +++ b/midx.c\n>> @@ -3,6 +3,7 @@\n>> #include \"dir.h\"\n>> #include \"csum-file.h\"\n>> #include \"lockfile.h\"\n>> +#include \"sha1-lookup.h\"\n>> #include \"object-store.h\"\n>> #include \"packfile.h\"\n>> #include \"midx.h\"\n>> @@ -64,7 +65,7 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n>>\n>> m = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n>> strcpy(m->object_dir, object_dir);\n>> - m->data = midx_map;\n>> + m->data = (const unsigned char*)midx_map;\n> Hmm? Why is this typecast only needed now? Or is it not really needed at all?\n>\n>> m->signature = get_be32(m->data);\n>> if (m->signature != MIDX_SIGNATURE) {\n>> @@ -145,7 +146,9 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n>>\n>> m->num_objects = ntohl(m->chunk_oid_fanout[255]);\n>>\n>> - m->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n>> + m->packs = xcalloc(m->num_packs, sizeof(*m->packs));\n>> +\n>> + ALLOC_ARRAY(m->pack_names, m->num_packs);\n> Please make this ALLOC_ARRAY change in the patch that adds\n> xcalloc(m->num_packs).\n>\n>> for (i = 0; i < m->num_packs; i++) {\n>> if (i) {\n>> if (ntohl(m->chunk_pack_lookup[i]) <= ntohl(m->chunk_pack_lookup[i - 1])) {\n>> @@ -175,6 +178,95 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n>> exit(1);\n>> }\n>>\n>> +static int prepare_midx_pack(struct midxed_git *m, uint32_t pack_int_id)\n>> +{\n>> + struct strbuf pack_name = STRBUF_INIT;\n>> +\n>> + if (pack_int_id >= m->num_packs)\n>> + BUG(\"bad pack-int-id\");\n>> +\n>> + if (m->packs[pack_int_id])\n>> + return 0;\n>> +\n>> + strbuf_addstr(&pack_name, m->object_dir);\n>> + strbuf_addstr(&pack_name, \"/pack/\");\n>> + strbuf_addstr(&pack_name, m->pack_names[pack_int_id]);\n> Just use strbuf_addf()\n>\n>> +\n>> + m->packs[pack_int_id] = add_packed_git(pack_name.buf, pack_name.len, 1);\n>> + strbuf_release(&pack_name);\n>> + return !m->packs[pack_int_id];\n> This is a weird return value convention. Normally we go zero/negative\n> or non-zero/zero for success/failure.\n\nWe are inconsistent.\n\n* open_pack_index() returns non-zero on error. (This was my reference \npoint.)\n* bsearch_pack() and find_pack_entry() return non-zero when an entry is \nfound.\n\nSince the use is \"if (error) die()\", similar to open_pack_index(), I'll \nkeep the current behavior. To switch would require using \n\"!!m->packs[pack_int_id]\" here and \"if (!prepare_midx_pack()) die()\" in \nthe consumer.\n\nThanks,\n-Stolee\n"},{"id":"350750","messageId":"08667213-02fa-6104-d8de-20ee92b6bdce@gmail.com","threadId":"48668","inReplyTo":"xmqqfu1gt4ye.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 09/23] midx: write pack names in chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-22T18:25:26Z","receivedAt":"2018-06-22T18:25:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/21/2018 1:38 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> On 6/7/2018 2:26 PM, Duy Nguyen wrote:\n>>> On Thu, Jun 7, 2018 at 4:03 PM, Derrick Stolee wrote:\n>>>> @@ -74,6 +80,31 @@ struct midxed_git *load_midxed_git(const char *object_dir)\n>>>> m->num_chunks = *(m->data + 6);\n>>>> m->num_packs = get_be32(m->data + 8);\n>>>>\n>>>> + for (i = 0; i < m->num_chunks; i++) {\n>>>> + uint32_t chunk_id = get_be32(m->data + 12 + MIDX_CHUNKLOOKUP_WIDTH * i);\n>>>> + uint64_t chunk_offset = get_be64(m->data + 16 + MIDX_CHUNKLOOKUP_WIDTH * i);\n>>> Would be good to reduce magic numbers like 12 and 16, I think you have\n>>> some header length constants for those already.\n>>>\n>>>> + switch (chunk_id) {\n>>>> + case MIDX_CHUNKID_PACKNAMES:\n>>>> + m->chunk_pack_names = m->data + chunk_offset;\n>>>> + break;\n> (style: aren't these case arms indented one level too deep)?\n>\n>>>> + case 0:\n>>>> + die(\"terminating MIDX chunk id appears earlier than expected\");\n>>> _()\n>> This die() and others like it are not marked for translation on\n>> purpose, as they should never be seen by an end-user.\n> Should never be seen because it indicates a software bug, in which\n> case this should be BUG() instead of die()?\n>\n> Or did we just find a file corruption on the filesystem? If so,\n> then the error is end-user facing and should tell the user something\n> that hints what is going on in the language the user understands, I\n> would guess.\n>\n>>>> + default:\n>>>> + /*\n>>>> + * Do nothing on unrecognized chunks, allowing future\n>>>> + * extensions to add optional chunks.\n>>>> + */\n>>> I wrote about the chunk term reminding me of PNG format then deleted\n>>> it. But it may help to do similar to PNG here. The first letter can\n>>> let us know if the chunk is optional and can be safely ignored. E.g.\n>>> uppercase first letter cannot be ignored, lowercase go wild.\n>> That's an interesting way to think about it. That way you could add a\n>> new \"required\" chunk and earlier versions could die() realizing they\n>> don't know how to parse that required chunk.\n> That is how the index extension sections work and may be a good\n> example to follow.\n\nThe index extension documentation doesn't appear to be clear about which \nextensions are optional or required, but it seems the split-index is the \nonly \"required\" one and uses lowercase for its extension id.\n\nSince the multi-pack-index has similar structure to the commit-graph \nfile, and that file includes an optional chunk with no special casing of \nthe chunk id, I think we should stick with the existing model: chunks \nthat are added later are optional and if Git _must_ understand it, then \nwe increment the version number. Hence, for each version number there is \na fixed list of required chunks, but an extendible list of optional chunks.\n\nThanks,\n-Stolee\n"},{"id":"350751","messageId":"xmqqzhzmptab.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"08667213-02fa-6104-d8de-20ee92b6bdce@gmail.com","subject":"Re: [PATCH 09/23] midx: write pack names in chunk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-22T18:31:24Z","receivedAt":"2018-06-22T18:31:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> The index extension documentation doesn't appear to be clear about\n> which extensions are optional or required, but it seems the\n> split-index is the only \"required\" one and uses lowercase for its\n> extension id.\n\nread-cache.c::\n\n /* Index extensions.\n *\n * The first letter should be 'A'..'Z' for extensions that are not\n * necessary for a correct operation (i.e. optimization data).\n * When new extensions are added that _needs_ to be understood in\n * order to correctly interpret the index file, pick character that\n * is outside the range, to cause the reader to abort.\n */\n\n"},{"id":"350752","messageId":"d79e4b0f-eeaa-0995-1ee6-8565903f2899@gmail.com","threadId":"48668","inReplyTo":"xmqqzhzmptab.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 09/23] midx: write pack names in chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-22T18:32:56Z","receivedAt":"2018-06-22T18:33:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/22/2018 2:31 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> The index extension documentation doesn't appear to be clear about\n>> which extensions are optional or required, but it seems the\n>> split-index is the only \"required\" one and uses lowercase for its\n>> extension id.\n> read-cache.c::\n>\n> /* Index extensions.\n> *\n> * The first letter should be 'A'..'Z' for extensions that are not\n> * necessary for a correct operation (i.e. optimization data).\n> * When new extensions are added that _needs_ to be understood in\n> * order to correctly interpret the index file, pick character that\n> * is outside the range, to cause the reader to abort.\n> */\n\nThanks! I was reading Documentation/technical/index-format.txt and \noptional extensions are mentioned but not described precisely.\n"},{"id":"350753","messageId":"56acee9d-ff91-94d7-3bbd-4089e8775c14@gmail.com","threadId":"48668","inReplyTo":"CACsJy8BQ5xxhQ8L9K84KO2EkYELsKKKkdJ0mvc8p427sA8bapQ@mail.gmail.com","subject":"Re: [PATCH 18/23] midx: use midx in abbreviation calculations","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-22T18:38:39Z","receivedAt":"2018-06-22T18:38:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/9/2018 2:01 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 4:06 PM Derrick Stolee wrote:\n>> @@ -565,8 +632,11 @@ static void find_abbrev_len_for_pack(struct packed_git *p,\n>>\n>> static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n>> {\n>> + struct midxed_git *m;\n>> struct packed_git *p;\n>>\n>> + for (m = get_midxed_git(the_repository); m; m = m->next)\n>> + find_abbrev_len_for_midx(m, mad);\n> If all the packs are in midx, we don't need to run the second loop\n> below, do we? Otherwise I don't see why we waste cycles on finding\n> abbrev length on midx at all.\n\nWe put all packs _at time of writing_ into the midx. More packs may be \nadded later that are not in the midx. There are tests in \nt5319-multi-pack-index.sh that verify everything works in this \"mixed mode\".\n\nIt is important that the packfiles are not loaded into the packed_git \nlist if they are managed by the midx.\n\n>\n>> for (p = get_packed_git(the_repository); p; p = p->next)\n>> find_abbrev_len_for_pack(p, mad);\n>> }\n\n"},{"id":"350754","messageId":"466e191e-6e0c-692c-ff8c-b847f3c4823a@gmail.com","threadId":"48668","inReplyTo":"CACsJy8AtkVwGw0+tgpv2AzNZMHGTOfkjHUN-37owaRWuQ_2wKw@mail.gmail.com","subject":"Re: [PATCH 20/23] midx: use midx in approximate_object_count","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-22T18:39:24Z","receivedAt":"2018-06-22T18:39:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/9/2018 2:03 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 4:06 PM Derrick Stolee wrote:\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> packfile.c | 5 ++++-\n>> 1 file changed, 4 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/packfile.c b/packfile.c\n>> index 638e113972..059b2aa097 100644\n>> --- a/packfile.c\n>> +++ b/packfile.c\n>> @@ -819,11 +819,14 @@ unsigned long approximate_object_count(void)\n>> {\n>> if (!the_repository->objects->approximate_object_count_valid) {\n>> unsigned long count;\n>> + struct midxed_git *m;\n>> struct packed_git *p;\n>>\n>> prepare_packed_git(the_repository);\n>> count = 0;\n>> - for (p = the_repository->objects->packed_git; p; p = p->next) {\n>> + for (m = get_midxed_git(the_repository); m; m = m->next)\n>> + count += m->num_objects;\n>> + for (p = get_packed_git(the_repository); p; p = p->next) {\n> Please don't change this line, it's not related to this patch.\n\nSure. I'll revert that line.\n\n> Same\n> concern applies, if we have already counted objects in midx we should\n> ignore packs that belong to it or we double count.\n\nSince we do not put packfiles into the packed_git list if they are \ntracked by the midx, we will not double count.\n\n>\n>> if (open_pack_index(p))\n>> continue;\n>> count += p->num_objects;\n>> --\n>> 2.18.0.rc1\n>>\n>\n\n"},{"id":"350755","messageId":"9aa18ceb-672b-31e5-ebfc-0e369ef9bc80@gmail.com","threadId":"48668","inReplyTo":"CACsJy8BHgqqrDNvU_oy0+_QmDW-wt-jBvWT9oJ9H1pA_tmuVFg@mail.gmail.com","subject":"Re: [PATCH 23/23] midx: clear midx on repack","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-22T18:44:47Z","receivedAt":"2018-06-22T18:44:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/9/2018 2:13 PM, Duy Nguyen wrote:\n> On Thu, Jun 7, 2018 at 4:07 PM Derrick Stolee wrote:\n>> If a 'git repack' command replaces existing packfiles, then we must\n>> clear the existing multi-pack-index before moving the packfiles it\n>> references.\n> I think there are other places where we add or remove pack files and\n> need to reprepare_packed_git(). Any midx invalidation should be part\n> of that as well.\n\nThe other places where we call reprepare_packed_git() are for when we \nmay have added a packfile, such as in fetch-pack.c, or sha1_file.c. The \nother candidate to consider is 'git gc', but the packfile deletion is \nhandled by a call to 'git repack'.\n\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/repack.c | 8 ++++++++\n>> midx.c | 8 ++++++++\n>> midx.h | 1 +\n>> 3 files changed, 17 insertions(+)\n>>\n>> diff --git a/builtin/repack.c b/builtin/repack.c\n>> index 6c636e159e..66a7d8e8ea 100644\n>> --- a/builtin/repack.c\n>> +++ b/builtin/repack.c\n>> @@ -8,6 +8,7 @@\n>> #include \"strbuf.h\"\n>> #include \"string-list.h\"\n>> #include \"argv-array.h\"\n>> +#include \"midx.h\"\n>>\n>> static int delta_base_offset = 1;\n>> static int pack_kept_objects = -1;\n>> @@ -174,6 +175,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n>> int no_update_server_info = 0;\n>> int quiet = 0;\n>> int local = 0;\n>> + int midx_cleared = 0;\n>>\n>> struct option builtin_repack_options[] = {\n>> OPT_BIT('a', NULL, &pack_everything,\n>> @@ -340,6 +342,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n>> continue;\n>> }\n>>\n>> + if (!midx_cleared) {\n>> + /* if we move a packfile, it will invalidated the midx */\n> What about removing packs, which also happens in repack? If the\n> removed pack is part of midx, then midx becomes invalid as well.\n>\n>> + clear_midx_file(get_object_directory());\n>> + midx_cleared = 1;\n>> + }\n>> +\n>> fname_old = mkpathdup(\"%s/old-%s%s\", packdir,\n>> item->string, exts[ext].name);\n>> if (file_exists(fname_old))\n>> diff --git a/midx.c b/midx.c\n>> index e46f392fa4..1043c01fa7 100644\n>> --- a/midx.c\n>> +++ b/midx.c\n>> @@ -913,3 +913,11 @@ int write_midx_file(const char *object_dir)\n>> FREE_AND_NULL(pack_names);\n>> return 0;\n>> }\n>> +\n>> +void clear_midx_file(const char *object_dir)\n> delete_ may be more obvious than clear_\n>\n>> +{\n>> + char *midx = get_midx_filename(object_dir);\n>> +\n>> + if (remove_path(midx))\n>> + die(_(\"failed to clear multi-pack-index at %s\"), midx);\n> die_errno()\n>\n>> +}\n>> diff --git a/midx.h b/midx.h\n>> index 6996b5ff6b..46f9f44c94 100644\n>> --- a/midx.h\n>> +++ b/midx.h\n>> @@ -18,5 +18,6 @@ int midx_contains_pack(struct midxed_git *m, const char *idx_name);\n>> int prepare_midxed_git_one(struct repository *r, const char *object_dir);\n>>\n>> int write_midx_file(const char *object_dir);\n>> +void clear_midx_file(const char *object_dir);\n>>\n>> #endif\n>> --\n>> 2.18.0.rc1\n>>\n>\n\n"},{"id":"350807","messageId":"20180625143434.89044-1-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180607140338.32440-1-dstolee@microsoft.com","subject":"[PATCH v2 00/24] Multi-pack-index (MIDX)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:10Z","receivedAt":"2018-06-25T14:34:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis v2 patch has several significant changes from v1. Thanks for all\nthe feedback that informed these changes:\n\n* The 'midx' builtin is renamed to 'multi-pack-index'\n\n* The 'core.midx' config setting is renamed to 'core.multiPackIndex'\n\n* Many die() or error() statements are marked for translation\n\n* The packfile name lookup chunk is dropped in favor of dynamic\n calculation on read.\n\n* The 'read' verb in the builtin is moved to a test tool.\n\nAnd one item that I'm saving for investigation while v2 is under review:\n\n* Consider using a merge sort when constructing/deduplicating the list\n of objects and offsets instead of sorting the objects in batches by\n first byte.\n\nThanks,\n-Stolee\n\n---\n\nThe multi-pack-index (MIDX) is explained fully in\nthe design document 'Documentation/technical/multi-pack-index.txt'.\nThe short description is that the MIDX stores the\ninformation from all of the IDX files in a pack\ndirectory. The crucial design decision is that the\nIDX files still exist, so we can fall back to the IDX\nfiles if there is any issue with the MIDX (or core.midx\nis set to false, or a user downgrades Git, etc.)\n\nThe MIDX feature has been part of our GVFS releases\nfor a few months (since the RFC). It has behaved well,\nindexing over 31 million commits and trees across up\nto 250 packfiles. These MIDX files are nearly 1GB in\nsize and take ~20 seconds to rewrite when adding new\nIDX information. This ~20s mark is something I'd like\nto improve, and I mention how to make the file\nincremental (similar to split-index) in the design\ndocument. I also want to make the commit-graph file\nincremental, so I'd like to do that at the same time\nafter both the MIDX and commit-graph are stable.\n\n\nLookup Speedups\n---------------\n\nWhen looking for an object, Git uses an most-recently-\nused (MRU) cache of packfiles. This does pretty well to\nminimize the number of misses when searching through\npackfiles for an object, especially if there is one\n\"big\" packfile that contains most of the objets (so it\nwill rarely miss and is usually one of the first two\npackfiles in the list). The MIDX does provide a way\nto remove these misses, improving lookup time. However,\nthis lookup time greatly depends on the arrangement of\nthe packfiles.\n\nFor instance, if you take the Linux repository and repack\nusing `git repack -adfF --max-pack-size=128m` then all\ncommits will be in one packfile, all trees will be in\na small set of packfiles and organized well so 'git\nrev-list --objects HEAD^{tree}' only inspects one or two\npackfiles.\n\nGVFS has the notion of a \"prefetch packfile\". These are\npackfiles that are precomputed by cache servers to\ncontain the commits and trees introduced to the remote\neach day. GVFS downloads these packfiles and places them\nin an alternate. Since these are organized by \"first\ntime introduced\" and the working directory is so large,\nthe MRU misses are significant when performing a checkout\nand updating the .git/index file.\n\nTo test the performance in this situation, I created a\nscript that organizes the Linux repository in a similar\nfashion. I split the commit history into 50 parts by\ncreating branches on every 10,000 commits of the first-\nparent history. Then, `git rev-list --objects A ^B`\nprovides the list of objects reachable from A but not B,\nso I could send that to `git pack-objects` to create\nthese \"time-based\" packfiles. With these 50 packfiles\n(deleting the old one from my fresh clone, and deleting\nall tags as they were no longer on-disk) I could then\ntest 'git rev-list --objects HEAD^{tree}' and see:\n\n Before: 0.17s\n After: 0.13s\n % Diff: -23.5%\n\nBy adding logic to count hits and misses to bsearch_pack,\nI was able to see that the command above calls that\nmethod 266,930 times with a hit rate of 33%. The MIDX\nhas the same number of calls with a 100% hit rate.\n\n\n\nAbbreviation Speedups\n---------------------\n\nTo fully disambiguate an abbreviation, we must iterate\nthrough all packfiles to ensure no collision exists in\nany packfile. This requires O(P log N) time. With the\nMIDX, this is only O(log N) time. Our standard test [2]\nis 'git log --oneline --parents --raw' because it writes\nmany abbreviations while also doing a lot of other work\n(walking commits and trees to compute the raw diff).\n\nFor a copy of the Linux repository with 50 packfiles\nsplit by time, we observed the following:\n\n Before: 100.5 s\n After: 58.2 s\n % Diff: -59.7%\n\n\nDerrick Stolee (24):\n multi-pack-index: add design document\n multi-pack-index: add format details\n multi-pack-index: add builtin\n multi-pack-index: add 'write' verb\n midx: write header information to lockfile\n multi-pack-index: load into memory\n multi-pack-index: expand test data\n packfile: generalize pack directory list\n multi-pack-index: read packfile list\n multi-pack-index: write pack names in chunk\n midx: read pack names into array\n midx: sort and deduplicate objects from packfiles\n midx: write object ids in a chunk\n midx: write object id fanout chunk\n midx: write object offsets\n config: create core.multiPackIndex setting\n midx: prepare midxed_git struct\n midx: read objects from multi-pack-index\n midx: use midx in abbreviation calculations\n midx: use existing midx when writing new one\n midx: use midx in approximate_object_count\n midx: prevent duplicate packfile loads\n packfile: skip loading index if in multi-pack-index\n midx: clear midx on repack\n\n .gitignore | 3 +-\n Documentation/config.txt | 4 +\n Documentation/git-multi-pack-index.txt | 56 ++\n Documentation/technical/multi-pack-index.txt | 109 +++\n Documentation/technical/pack-format.txt | 77 ++\n Makefile | 3 +\n builtin.h | 1 +\n builtin/multi-pack-index.c | 46 +\n builtin/repack.c | 8 +\n cache.h | 1 +\n command-list.txt | 1 +\n config.c | 5 +\n environment.c | 1 +\n git.c | 1 +\n midx.c | 900 +++++++++++++++++++\n midx.h | 20 +\n object-store.h | 33 +\n packfile.c | 173 +++-\n packfile.h | 9 +\n sha1-name.c | 70 ++\n t/helper/test-read-midx.c | 54 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5319-multi-pack-index.sh | 191 ++++\n 24 files changed, 1724 insertions(+), 44 deletions(-)\n create mode 100644 Documentation/git-multi-pack-index.txt\n create mode 100644 Documentation/technical/multi-pack-index.txt\n create mode 100644 builtin/multi-pack-index.c\n create mode 100644 midx.c\n create mode 100644 midx.h\n create mode 100644 t/helper/test-read-midx.c\n create mode 100755 t/t5319-multi-pack-index.sh\n\n\nbase-commit: 53f9a3e157dbbc901a02ac2c73346d375e24978c\n-- \n2.18.0.rc1\n\n"},{"id":"350808","messageId":"20180625143434.89044-2-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 01/24] multi-pack-index: add design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:11Z","receivedAt":"2018-06-25T14:34:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/multi-pack-index.txt | 109 +++++++++++++++++++\n 1 file changed, 109 insertions(+)\n create mode 100644 Documentation/technical/multi-pack-index.txt\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nnew file mode 100644\nindex 0000000000..d7e57639f7\n--- /dev/null\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -0,0 +1,109 @@\n+Multi-Pack-Index (MIDX) Design Notes\n+====================================\n+\n+The Git object directory contains a 'pack' directory containing\n+packfiles (with suffix \".pack\") and pack-indexes (with suffix\n+\".idx\"). The pack-indexes provide a way to lookup objects and\n+navigate to their offset within the pack, but these must come\n+in pairs with the packfiles. This pairing depends on the file\n+names, as the pack-index differs only in suffix with its pack-\n+file. While the pack-indexes provide fast lookup per packfile,\n+this performance degrades as the number of packfiles increases,\n+because abbreviations need to inspect every packfile and we are\n+more likely to have a miss on our most-recently-used packfile.\n+For some large repositories, repacking into a single packfile\n+is not feasible due to storage space or excessive repack times.\n+\n+The multi-pack-index (MIDX for short) stores a list of objects\n+and their offsets into multiple packfiles. It contains:\n+\n+- A list of packfile names.\n+- A sorted list of object IDs.\n+- A list of metadata for the ith object ID including:\n+ - A value j referring to the jth packfile.\n+ - An offset within the jth packfile for the object.\n+- If large offsets are required, we use another list of large\n+ offsets similar to version 2 pack-indexes.\n+\n+Thus, we can provide O(log N) lookup time for any number\n+of packfiles.\n+\n+Design Details\n+--------------\n+\n+- The MIDX is stored in a file named 'multi-pack-index' in the\n+ .git/objects/pack directory. This could be stored in the pack\n+ directory of an alternate. It refers only to packfiles in that\n+ same directory.\n+\n+- The pack.multiIndex config setting must be on to consume MIDX files.\n+\n+- The file format includes parameters for the object ID hash\n+ function, so a future change of hash algorithm does not require\n+ a change in format.\n+\n+- The MIDX keeps only one record per object ID. If an object appears\n+ in multiple packfiles, then the MIDX selects the copy in the most-\n+ recently modified packfile.\n+\n+- If there exist packfiles in the pack directory not registered in\n+ the MIDX, then those packfiles are loaded into the `packed_git`\n+ list and `packed_git_mru` cache.\n+\n+- The pack-indexes (.idx files) remain in the pack directory so we\n+ can delete the MIDX file, set core.midx to false, or downgrade\n+ without any loss of information.\n+\n+- The MIDX file format uses a chunk-based approach (similar to the\n+ commit-graph file) that allows optional data to be added.\n+\n+Future Work\n+-----------\n+\n+- Add a 'verify' subcommand to the 'git midx' builtin to verify the\n+ contents of the multi-pack-index file match the offsets listed in\n+ the corresponding pack-indexes.\n+\n+- The multi-pack-index allows many packfiles, especially in a context\n+ where repacking is expensive (such as a very large repo), or\n+ unexpected maintenance time is unacceptable (such as a high-demand\n+ build machine). However, the multi-pack-index needs to be rewritten\n+ in full every time. We can extend the format to be incremental, so\n+ writes are fast. By storing a small \"tip\" multi-pack-index that\n+ points to large \"base\" MIDX files, we can keep writes fast while\n+ still reducing the number of binary searches required for object\n+ lookups.\n+\n+- The reachability bitmap is currently paired directly with a single\n+ packfile, using the pack-order as the object order to hopefully\n+ compress the bitmaps well using run-length encoding. This could be\n+ extended to pair a reachability bitmap with a multi-pack-index. If\n+ the multi-pack-index is extended to store a \"stable object order\"\n+ (a function Order(hash) = integer that is constant for a given hash,\n+ even as the multi-pack-index is updated) then a reachability bitmap\n+ could point to a multi-pack-index and be updated independently.\n+\n+- Packfiles can be marked as \"special\" using empty files that share\n+ the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n+ We can add an optional chunk of data to the multi-pack-index that\n+ records flags of information about the packfiles. This allows new\n+ states, such as 'repacked' or 'redeltified', that can help with\n+ pack maintenance in a multi-pack environment. It may also be\n+ helpful to organize packfiles by object type (commit, tree, blob,\n+ etc.) and use this metadata to help that maintenance.\n+\n+- The partial clone feature records special \"promisor\" packs that\n+ may point to objects that are not stored locally, but available\n+ on request to a server. The multi-pack-index does not currently\n+ track these promisor packs.\n+\n+Related Links\n+-------------\n+[0] https://bugs.chromium.org/p/git/issues/detail?id=6\n+ Chromium work item for: Multi-Pack Index (MIDX)\n+\n+[1] https://public-inbox.org/git/20180107181459.222909-1-dstolee@microsoft.com/\n+ An earlier RFC for the multi-pack-index feature\n+\n+[2] https://public-inbox.org/git/alpine.DEB.2.20.1803091557510.23109@alexmv-linux/\n+ Git Merge 2018 Contributor's summit notes (includes discussion of MIDX)\n\nbase-commit: 53f9a3e157dbbc901a02ac2c73346d375e24978c\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350809","messageId":"20180625143434.89044-3-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 02/24] multi-pack-index: add format details","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:12Z","receivedAt":"2018-06-25T14:34:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index feature generalizes the existing pack-index\nfeature by indexing objects across multiple pack-files.\n\nDescribe the basic file format, using a 12-byte header followed by\na lookup table for a list of \"chunks\" which will be described later.\nThe file ends with a footer containing a checksum using the hash\nalgorithm.\n\nThe header allows later versions to create breaking changes by\nadvancing the version number. We can also change the hash algorithm\nusing a different version value.\n\nWe will add the individual chunk format information as we introduce\nthe code that writes that information.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 49 +++++++++++++++++++++++++\n 1 file changed, 49 insertions(+)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 70a99fd142..e060e693f4 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -252,3 +252,52 @@ Pack file entry: <+\n corresponding packfile.\n \n 20-byte SHA-1-checksum of all of the above.\n+\n+== multi-pack-index (MIDX) files have the following format:\n+\n+The multi-pack-index files refer to multiple pack-files and loose objects.\n+\n+In order to allow extensions that add extra data to the MIDX, we organize\n+the body into \"chunks\" and provide a lookup table at the beginning of the\n+body. The header includes certain length values, such as the number of packs,\n+the number of base MIDX files, hash lengths and types.\n+\n+All 4-byte numbers are in network order.\n+\n+HEADER:\n+\n+\t4-byte signature:\n+\t The signature is: {'M', 'I', 'D', 'X'}\n+\n+\t1-byte version number:\n+\t Git only writes or recognizes version 1.\n+\n+\t1-byte Object Id Version\n+\t Git only writes or recognizes version 1 (SHA1).\n+\n+\t1-byte number of \"chunks\"\n+\n+\t1-byte number of base multi-pack-index files:\n+\t This value is currently always zero.\n+\n+\t4-byte number of pack files\n+\n+CHUNK LOOKUP:\n+\n+\t(C + 1) * 12 bytes providing the chunk offsets:\n+\t First 4 bytes describe chunk id. Value 0 is a terminating label.\n+\t Other 8 bytes provide offset in current file for chunk to start.\n+\t (Chunks are provided in file-order, so you can infer the length\n+\t using the next chunk position if necessary.)\n+\n+\tThe remaining data in the body is described one chunk at a time, and\n+\tthese chunks may be given in any order. Chunks are required unless\n+\totherwise specified.\n+\n+CHUNK DATA:\n+\n+\t(This section intentionally left incomplete.)\n+\n+TRAILER:\n+\n+\t20-byte SHA1-checksum of the above contents.\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350810","messageId":"20180625143434.89044-5-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 04/24] multi-pack-index: add 'write' verb","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:14Z","receivedAt":"2018-06-25T14:35:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of writing multi-pack-indexes, add a\n'git multi-pack-index write' subcommand and send the options to a\nwrite_midx_file() method. Also create a basic test file that tests\nthe 'write' subcommand.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 22 +++++++++++++++++++++-\n Makefile | 1 +\n builtin/multi-pack-index.c | 10 +++++++++-\n midx.c | 7 +++++++\n midx.h | 6 ++++++\n t/t5319-multi-pack-index.sh | 10 ++++++++++\n 6 files changed, 54 insertions(+), 2 deletions(-)\n create mode 100644 midx.c\n create mode 100644 midx.h\n create mode 100755 t/t5319-multi-pack-index.sh\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 9877f9c441..c4dc92ddd9 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir ]\n+'git multi-pack-index' [--object-dir ] \n \n DESCRIPTION\n -----------\n@@ -23,6 +23,26 @@ OPTIONS\n \t/packs/multi-pack-index for the current MIDX file, and\n \t/packs for the pack-files to index.\n \n+write::\n+\tWhen given as the verb, write a new MIDX file to\n+\t/packs/multi-pack-index.\n+\n+\n+EXAMPLES\n+--------\n+\n+* Write a MIDX file for the packfiles in the current .git folder.\n++\n+-----------------------------------------------\n+$ git multi-pack-index write\n+-----------------------------------------------\n+\n+* Write a MIDX file for the packfiles in an alternate.\n++\n+-----------------------------------------------\n+$ git multi-pack-index --object-dir write\n+-----------------------------------------------\n+\n \n SEE ALSO\n --------\ndiff --git a/Makefile b/Makefile\nindex 54610875ec..f5636c711d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -890,6 +890,7 @@ LIB_OBJS += merge.o\n LIB_OBJS += merge-blobs.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += mergesort.o\n+LIB_OBJS += midx.o\n LIB_OBJS += name-hash.o\n LIB_OBJS += notes.o\n LIB_OBJS += notes-cache.o\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex f101873525..c8f1f19d1f 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -2,9 +2,10 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"parse-options.h\"\n+#include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] ={\n-\tN_(\"git multi-pack-index [--object-dir ]\"),\n+\tN_(\"git multi-pack-index [--object-dir ] [write]\"),\n \tNULL\n };\n \n@@ -34,5 +35,12 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \n+\tif (argc == 0)\n+\t\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t\t builtin_multi_pack_index_options);\n+\n+\tif (!strcmp(argv[0], \"write\"))\n+\t\treturn write_midx_file(opts.object_dir);\n+\n \treturn 0;\n }\ndiff --git a/midx.c b/midx.c\nnew file mode 100644\nindex 0000000000..32468db1a2\n--- /dev/null\n+++ b/midx.c\n@@ -0,0 +1,7 @@\n+#include \"cache.h\"\n+#include \"midx.h\"\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nnew file mode 100644\nindex 0000000000..dbdbe9f873\n--- /dev/null\n+++ b/midx.h\n@@ -0,0 +1,6 @@\n+#ifndef __MIDX_H__\n+#define __MIDX_H__\n+\n+int write_midx_file(const char *object_dir);\n+\n+#endif\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nnew file mode 100755\nindex 0000000000..ec3ddbe79c\n--- /dev/null\n+++ b/t/t5319-multi-pack-index.sh\n@@ -0,0 +1,10 @@\n+#!/bin/sh\n+\n+test_description='multi-pack-indexes'\n+. ./test-lib.sh\n+\n+test_expect_success 'write midx with no packs' '\n+\tgit multi-pack-index --object-dir=. write\n+'\n+\n+test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350811","messageId":"20180625143434.89044-4-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 03/24] multi-pack-index: add builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:13Z","receivedAt":"2018-06-25T14:35:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This new 'git multi-pack-index' builtin will be the plumbing access\nfor writing, reading, and checking multi-pack-index files. The\ninitial implementation is a no-op.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 3 +-\n Documentation/git-multi-pack-index.txt | 36 ++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/multi-pack-index.c | 38 ++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n 7 files changed, 80 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/git-multi-pack-index.txt\n create mode 100644 builtin/multi-pack-index.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 388cc4beee..25633bc515 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -99,8 +99,9 @@\n /git-mergetool--lib\n /git-mktag\n /git-mktree\n-/git-name-rev\n+/git-multi-pack-index\n /git-mv\n+/git-name-rev\n /git-notes\n /git-p4\n /git-pack-redundant\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nnew file mode 100644\nindex 0000000000..9877f9c441\n--- /dev/null\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -0,0 +1,36 @@\n+git-multi-pack-index(1)\n+======================\n+\n+NAME\n+----\n+git-multi-pack-index - Write and verify multi-pack-indexes\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git multi-pack-index' [--object-dir ]\n+\n+DESCRIPTION\n+-----------\n+Write or verify a multi-pack-index (MIDX) file.\n+\n+OPTIONS\n+-------\n+\n+--object-dir ::\n+\tUse given directory for the location of Git objects. We check\n+\t/packs/multi-pack-index for the current MIDX file, and\n+\t/packs for the pack-files to index.\n+\n+\n+SEE ALSO\n+--------\n+See link:technical/multi-pack-index.html[The Multi-Pack-Index Design\n+Document] and link:technical/pack-format.html[The Multi-Pack-Index\n+Format] for more information on the multi-pack-index feature.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex e4b503d259..54610875ec 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1047,6 +1047,7 @@ BUILTIN_OBJS += builtin/merge-recursive.o\n BUILTIN_OBJS += builtin/merge-tree.o\n BUILTIN_OBJS += builtin/mktag.o\n BUILTIN_OBJS += builtin/mktree.o\n+BUILTIN_OBJS += builtin/multi-pack-index.o\n BUILTIN_OBJS += builtin/mv.o\n BUILTIN_OBJS += builtin/name-rev.o\n BUILTIN_OBJS += builtin/notes.o\ndiff --git a/builtin.h b/builtin.h\nindex 4e0f64723e..70997d7ace 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -191,6 +191,7 @@ extern int cmd_merge_recursive(int argc, const char **argv, const char *prefix);\n extern int cmd_merge_tree(int argc, const char **argv, const char *prefix);\n extern int cmd_mktag(int argc, const char **argv, const char *prefix);\n extern int cmd_mktree(int argc, const char **argv, const char *prefix);\n+extern int cmd_multi_pack_index(int argc, const char **argv, const char *prefix);\n extern int cmd_mv(int argc, const char **argv, const char *prefix);\n extern int cmd_name_rev(int argc, const char **argv, const char *prefix);\n extern int cmd_notes(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nnew file mode 100644\nindex 0000000000..f101873525\n--- /dev/null\n+++ b/builtin/multi-pack-index.c\n@@ -0,0 +1,38 @@\n+#include \"builtin.h\"\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+\n+static char const * const builtin_multi_pack_index_usage[] ={\n+\tN_(\"git multi-pack-index [--object-dir ]\"),\n+\tNULL\n+};\n+\n+static struct opts_multi_pack_index {\n+\tconst char *object_dir;\n+} opts;\n+\n+int cmd_multi_pack_index(int argc, const char **argv,\n+\t\t\t const char *prefix)\n+{\n+\tstatic struct option builtin_multi_pack_index_options[] = {\n+\t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n+\t\t N_(\"The object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t\t builtin_multi_pack_index_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_multi_pack_index_options,\n+\t\t\t builtin_multi_pack_index_usage, 0);\n+\n+\tif (!opts.object_dir)\n+\t\topts.object_dir = get_object_directory();\n+\n+\treturn 0;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex e1c26c1bb7..61071f8fa2 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -123,6 +123,7 @@ git-merge-index plumbingmanipulators\n git-merge-one-file purehelpers\n git-mergetool ancillarymanipulators complete\n git-merge-tree ancillaryinterrogators\n+git-multi-pack-index plumbingmanipulators\n git-mktag plumbingmanipulators\n git-mktree plumbingmanipulators\n git-mv mainporcelain worktree\ndiff --git a/git.c b/git.c\nindex c2f48d53dd..a7509fa5f7 100644\n--- a/git.c\n+++ b/git.c\n@@ -505,6 +505,7 @@ static struct cmd_struct commands[] = {\n \t{ \"merge-tree\", cmd_merge_tree, RUN_SETUP | NO_PARSEOPT },\n \t{ \"mktag\", cmd_mktag, RUN_SETUP | NO_PARSEOPT },\n \t{ \"mktree\", cmd_mktree, RUN_SETUP },\n+\t{ \"multi-pack-index\", cmd_multi_pack_index, RUN_SETUP_GENTLY },\n \t{ \"mv\", cmd_mv, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"name-rev\", cmd_name_rev, RUN_SETUP },\n \t{ \"notes\", cmd_notes, RUN_SETUP },\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350812","messageId":"20180625143434.89044-6-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 05/24] midx: write header information to lockfile","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:15Z","receivedAt":"2018-06-25T14:35:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As we begin writing the multi-pack-index format to disk, start with\nthe basics: the 12-byte header and the 20-byte checksum footer. Start\nwith these basics so we can add the rest of the format in small\nincrements.\n\nAs we implement the format, we will use a technique to check that our\ncomputed offsets within the multi-pack-index file match what we are\nactually writing. Each method that writes to the hashfile will return\nthe number of bytes written, and we will track that those values match\nour expectations.\n\nCurrently, write_midx_header() returns 12, but is not checked. We will\ncheck the return value in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 49 +++++++++++++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh | 3 ++-\n 2 files changed, 51 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 32468db1a2..393d526881 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,7 +1,56 @@\n #include \"cache.h\"\n+#include \"csum-file.h\"\n+#include \"lockfile.h\"\n #include \"midx.h\"\n \n+#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n+#define MIDX_VERSION 1\n+#define MIDX_HASH_VERSION 1\n+#define MIDX_HEADER_SIZE 12\n+\n+static char *get_midx_filename(const char *object_dir)\n+{\n+\treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n+}\n+\n+static size_t write_midx_header(struct hashfile *f,\n+\t\t\t\tunsigned char num_chunks,\n+\t\t\t\tuint32_t num_packs)\n+{\n+\tunsigned char byte_values[4];\n+\thashwrite_be32(f, MIDX_SIGNATURE);\n+\tbyte_values[0] = MIDX_VERSION;\n+\tbyte_values[1] = MIDX_HASH_VERSION;\n+\tbyte_values[2] = num_chunks;\n+\tbyte_values[3] = 0; /* unused */\n+\thashwrite(f, byte_values, sizeof(byte_values));\n+\thashwrite_be32(f, num_packs);\n+\n+\treturn MIDX_HEADER_SIZE;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n+\tunsigned char num_chunks = 0;\n+\tchar *midx_name;\n+\tstruct hashfile *f;\n+\tstruct lock_file lk;\n+\n+\tmidx_name = get_midx_filename(object_dir);\n+\tif (safe_create_leading_directories(midx_name)) {\n+\t\tUNLEAK(midx_name);\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t midx_name);\n+\t}\n+\n+\thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n+\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\tFREE_AND_NULL(midx_name);\n+\n+\twrite_midx_header(f, num_chunks, 0);\n+\n+\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tcommit_lock_file(&lk);\n+\n \treturn 0;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex ec3ddbe79c..8622a7cdce 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -4,7 +4,8 @@ test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n test_expect_success 'write midx with no packs' '\n-\tgit multi-pack-index --object-dir=. write\n+\tgit multi-pack-index --object-dir=. write &&\n+\ttest_path_is_file pack/multi-pack-index\n '\n \n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350813","messageId":"20180625143434.89044-9-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 08/24] packfile: generalize pack directory list","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:18Z","receivedAt":"2018-06-25T14:35:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of sharing the pack directory listing with the\nmulti-pack-index, generalize prepare_packed_git_one() into\nfor_each_file_in_pack_dir().\n\nSigned-off-by: Derrick Stolee \n---\n packfile.c | 103 +++++++++++++++++++++++++++++++++--------------------\n packfile.h | 6 ++++\n 2 files changed, 71 insertions(+), 38 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 7cd45aa4b2..db61c8813b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -738,13 +738,14 @@ static void report_pack_garbage(struct string_list *list)\n \treport_helper(list, seen_bits, first, list->nr);\n }\n \n-static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tsize_t dirnamelen;\n \tDIR *dir;\n \tstruct dirent *de;\n-\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \n \tstrbuf_addstr(&path, objdir);\n \tstrbuf_addstr(&path, \"/pack\");\n@@ -759,53 +760,79 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n \tstrbuf_addch(&path, '/');\n \tdirnamelen = path.len;\n \twhile ((de = readdir(dir)) != NULL) {\n-\t\tstruct packed_git *p;\n-\t\tsize_t base_len;\n-\n \t\tif (is_dot_or_dotdot(de->d_name))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&path, dirnamelen);\n \t\tstrbuf_addstr(&path, de->d_name);\n \n-\t\tbase_len = path.len;\n-\t\tif (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n-\t\t\t/* Don't reopen a pack we already have. */\n-\t\t\tfor (p = r->objects->packed_git; p;\n-\t\t\t p = p->next) {\n-\t\t\t\tsize_t len;\n-\t\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n-\t\t\t\t len == base_len &&\n-\t\t\t\t !memcmp(p->pack_name, path.buf, len))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tif (p == NULL &&\n-\t\t\t /*\n-\t\t\t * See if it really is a valid .idx file with\n-\t\t\t * corresponding .pack file that we can map.\n-\t\t\t */\n-\t\t\t (p = add_packed_git(path.buf, path.len, local)) != NULL)\n-\t\t\t\tinstall_packed_git(r, p);\n-\t\t}\n-\n-\t\tif (!report_garbage)\n-\t\t\tcontinue;\n-\n-\t\tif (ends_with(de->d_name, \".idx\") ||\n-\t\t ends_with(de->d_name, \".pack\") ||\n-\t\t ends_with(de->d_name, \".bitmap\") ||\n-\t\t ends_with(de->d_name, \".keep\") ||\n-\t\t ends_with(de->d_name, \".promisor\"))\n-\t\t\tstring_list_append(&garbage, path.buf);\n-\t\telse\n-\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path.buf);\n+\t\tfn(path.buf, path.len, de->d_name, data);\n \t}\n+\n \tclosedir(dir);\n-\treport_pack_garbage(&garbage);\n-\tstring_list_clear(&garbage, 0);\n \tstrbuf_release(&path);\n }\n \n+struct prepare_pack_data\n+{\n+\tstruct repository *r;\n+\tstruct string_list *garbage;\n+\tint local;\n+};\n+\n+static void prepare_pack(const char *full_name, size_t full_name_len, const char *file_name, void *_data)\n+{\n+\tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n+\tstruct packed_git *p;\n+\tsize_t base_len = full_name_len;\n+\n+\tif (strip_suffix_mem(full_name, &base_len, \".idx\")) {\n+\t\t/* Don't reopen a pack we already have. */\n+\t\tfor (p = data->r->objects->packed_git; p; p = p->next) {\n+\t\t\tsize_t len;\n+\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n+\t\t\t len == base_len &&\n+\t\t\t !memcmp(p->pack_name, full_name, len))\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (p == NULL &&\n+\t\t /*\n+\t\t * See if it really is a valid .idx file with\n+\t\t * corresponding .pack file that we can map.\n+\t\t */\n+\t\t (p = add_packed_git(full_name, full_name_len, data->local)) != NULL)\n+\t\t\tinstall_packed_git(data->r, p);\n+\t}\n+\n+\tif (!report_garbage)\n+\t return;\n+\n+\tif (ends_with(file_name, \".idx\") ||\n+\t ends_with(file_name, \".pack\") ||\n+\t ends_with(file_name, \".bitmap\") ||\n+\t ends_with(file_name, \".keep\") ||\n+\t ends_with(file_name, \".promisor\"))\n+\t\tstring_list_append(data->garbage, full_name);\n+\telse\n+\t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\n+}\n+\n+static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+{\n+\tstruct prepare_pack_data data;\n+\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n+\n+\tdata.r = r;\n+\tdata.garbage = &garbage;\n+\tdata.local = local;\n+\n+\tfor_each_file_in_pack_dir(objdir, prepare_pack, &data);\n+\n+\treport_pack_garbage(data.garbage);\n+\tstring_list_clear(data.garbage, 0);\n+}\n+\n static void prepare_packed_git(struct repository *r);\n /*\n * Give a fast, rough count of the number of objects in the repository. This\ndiff --git a/packfile.h b/packfile.h\nindex e0a38aba93..d2ad30300a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -28,6 +28,12 @@ extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n+typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len,\n+\t\t\t\t const char *file_pach, void *data);\n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data);\n+\n /* A hook to report invalid files in pack directory */\n #define PACKDIR_FILE_PACK 1\n #define PACKDIR_FILE_IDX 2\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350814","messageId":"20180625143434.89044-10-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 09/24] multi-pack-index: read packfile list","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:19Z","receivedAt":"2018-06-25T14:35:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When constructing a multi-pack-index file for a given object directory,\nread the files within the enclosed pack directory and find matches that\nend with \".idx\" and find the correct paired packfile using\nadd_packed_git().\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 45 ++++++++++++++++++++++++++++++++++++-\n t/t5319-multi-pack-index.sh | 16 ++++++-------\n 2 files changed, 52 insertions(+), 9 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 0977397d6a..e79ffb5576 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,6 +1,8 @@\n #include \"cache.h\"\n #include \"csum-file.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n+#include \"packfile.h\"\n #include \"object-store.h\"\n #include \"midx.h\"\n \n@@ -107,12 +109,40 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+struct pack_list\n+{\n+\tstruct packed_git **list;\n+\tuint32_t nr;\n+\tuint32_t alloc;\n+};\n+\n+static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n+\t\t\t const char *file_name, void *data)\n+{\n+\tstruct pack_list *packs = (struct pack_list *)data;\n+\n+\tif (ends_with(file_name, \".idx\")) {\n+\t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc);\n+\n+\t\tpacks->list[packs->nr] = add_packed_git(full_path,\n+\t\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t\t 0);\n+\t\tif (!packs->list[packs->nr])\n+\t\t\twarning(_(\"failed to add packfile '%s'\"),\n+\t\t\t\tfull_path);\n+\t\telse\n+\t\t\tpacks->nr++;\n+\t}\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char num_chunks = 0;\n \tchar *midx_name;\n+\tuint32_t i;\n \tstruct hashfile *f;\n \tstruct lock_file lk;\n+\tstruct pack_list packs;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -121,14 +151,27 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n+\tpacks.nr = 0;\n+\tpacks.alloc = 16;\n+\tpacks.list = NULL;\n+\tALLOC_ARRAY(packs.list, packs.alloc);\n+\n+\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\twrite_midx_header(f, num_chunks, 0);\n+\twrite_midx_header(f, num_chunks, packs.nr);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n \n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tclose_pack(packs.list[i]);\n+\t\tFREE_AND_NULL(packs.list[i]);\n+\t}\n+\n+\tFREE_AND_NULL(packs.list);\n \treturn 0;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex d533fd0dbc..4d4d6ca0a6 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -4,8 +4,9 @@ test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n midx_read_expect() {\n+\tNUM_PACKS=$1\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 0 0\n+\theader: 4d494458 1 0 $NUM_PACKS\n \tobject_dir: .\n \tEOF\n \ttest-tool read-midx . >actual &&\n@@ -15,8 +16,7 @@ midx_read_expect() {\n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\ttest_path_is_file pack/multi-pack-index &&\n-\tmidx_read_expect\n+\tmidx_read_expect 0\n '\n \n test_expect_success 'create objects' '\n@@ -47,13 +47,13 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test obj-list &&\n \t\tgit update-ref HEAD $commit &&\n-\t\tgit pack-objects --index-version=2 test-pack \n---\n Makefile | 1 +\n midx.c | 78 +++++++++++++++++++++++++++++++++++++\n midx.h | 4 ++\n object-store.h | 16 ++++++++\n t/helper/test-read-midx.c | 34 ++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5319-multi-pack-index.sh | 12 +++++-\n 8 files changed, 146 insertions(+), 1 deletion(-)\n create mode 100644 t/helper/test-read-midx.c\n\ndiff --git a/Makefile b/Makefile\nindex f5636c711d..0b801d1b16 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -717,6 +717,7 @@ TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n TEST_BUILTINS_OBJS += test-prio-queue.o\n TEST_BUILTINS_OBJS += test-read-cache.o\n+TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\ndiff --git a/midx.c b/midx.c\nindex 393d526881..0977397d6a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,18 +1,96 @@\n #include \"cache.h\"\n #include \"csum-file.h\"\n #include \"lockfile.h\"\n+#include \"object-store.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n #define MIDX_HASH_VERSION 1\n #define MIDX_HEADER_SIZE 12\n+#define MIDX_HASH_LEN 20\n+#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m = NULL;\n+\tint fd;\n+\tstruct stat st;\n+\tsize_t midx_size;\n+\tvoid *midx_map = NULL;\n+\tuint32_t hash_version;\n+\tchar *midx_name = get_midx_filename(object_dir);\n+\n+\tfd = git_open(midx_name);\n+\n+\tif (fd < 0) {\n+\t\terror_errno(_(\"failed to read %s\"), midx_name);\n+\t\tFREE_AND_NULL(midx_name);\n+\t\treturn NULL;\n+\t}\n+\tif (fstat(fd, &st)) {\n+\t\terror_errno(_(\"failed to read %s\"), midx_name);\n+\t\tFREE_AND_NULL(midx_name);\n+\t\tclose(fd);\n+\t\treturn NULL;\n+\t}\n+\n+\tmidx_size = xsize_t(st.st_size);\n+\n+\tif (midx_size < MIDX_MIN_SIZE) {\n+\t\tclose(fd);\n+\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tFREE_AND_NULL(midx_name);\n+\n+\tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\n+\tm = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n+\tstrcpy(m->object_dir, object_dir);\n+\tm->data = midx_map;\n+\n+\tm->signature = get_be32(m->data);\n+\tif (m->signature != MIDX_SIGNATURE) {\n+\t\terror(_(\"multi-pack-index signature 0x%08x does not match signature 0x%08x\"),\n+\t\t m->signature, MIDX_SIGNATURE);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tm->version = m->data[4];\n+\tif (m->version != MIDX_VERSION) {\n+\t\terror(_(\"multi-pack-index version %d not recognized\"),\n+\t\t m->version);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\thash_version = m->data[5];\n+\tif (hash_version != MIDX_HASH_VERSION) {\n+\t\terror(_(\"hash version %u does not match\"), hash_version);\n+\t\tgoto cleanup_fail;\n+\t}\n+\tm->hash_len = MIDX_HASH_LEN;\n+\n+\tm->num_chunks = *(m->data + 6);\n+\n+\tm->num_packs = get_be32(m->data + 8);\n+\n+\treturn m;\n+\n+cleanup_fail:\n+\tFREE_AND_NULL(m);\n+\tFREE_AND_NULL(midx_name);\n+\tmunmap(midx_map, midx_size);\n+\tclose(fd);\n+\treturn NULL;\n+}\n+\n static size_t write_midx_header(struct hashfile *f,\n \t\t\t\tunsigned char num_chunks,\n \t\t\t\tuint32_t num_packs)\ndiff --git a/midx.h b/midx.h\nindex dbdbe9f873..2d83dd9ec1 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -1,6 +1,10 @@\n #ifndef __MIDX_H__\n #define __MIDX_H__\n \n+struct multi_pack_index;\n+\n+struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n+\n int write_midx_file(const char *object_dir);\n \n #endif\ndiff --git a/object-store.h b/object-store.h\nindex d683112fd7..4f410841cc 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -84,6 +84,22 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+struct multi_pack_index {\n+\tint fd;\n+\n+\tconst unsigned char *data;\n+\tsize_t data_len;\n+\n+\tuint32_t signature;\n+\tunsigned char version;\n+\tunsigned char hash_len;\n+\tunsigned char num_chunks;\n+\tuint32_t num_packs;\n+\tuint32_t num_objects;\n+\n+\tchar object_dir[FLEX_ARRAY];\n+};\n+\n struct raw_object_store {\n \t/*\n \t * Path to the repository's object store.\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nnew file mode 100644\nindex 0000000000..5abf969175\n--- /dev/null\n+++ b/t/helper/test-read-midx.c\n@@ -0,0 +1,34 @@\n+/*\n+ * test-mktemp.c: code to exercise the creation of temporary files\n+ */\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"midx.h\"\n+#include \"repository.h\"\n+#include \"object-store.h\"\n+\n+static int read_midx_file(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tprintf(\"header: %08x %d %d %d\\n\",\n+\t m->signature,\n+\t m->version,\n+\t m->num_chunks,\n+\t m->num_packs);\n+\n+\tprintf(\"object_dir: %s\\n\", m->object_dir);\n+\n+\treturn 0;\n+}\n+\n+int cmd__read_midx(int argc, const char **argv)\n+{\n+\tif (argc != 2)\n+\t\tusage(\"read-midx \");\n+\n+\treturn read_midx_file(argv[1]);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 805a45de9c..1c3ab36e6c 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -27,6 +27,7 @@ static struct test_cmd cmds[] = {\n \t{ \"path-utils\", cmd__path_utils },\n \t{ \"prio-queue\", cmd__prio_queue },\n \t{ \"read-cache\", cmd__read_cache },\n+\t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"regex\", cmd__regex },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 7116ddfb94..6af8c08a66 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -21,6 +21,7 @@ int cmd__online_cpus(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n int cmd__prio_queue(int argc, const char **argv);\n int cmd__read_cache(int argc, const char **argv);\n+int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 8622a7cdce..0372704c96 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,9 +3,19 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n+midx_read_expect() {\n+\tcat >expect <<- EOF\n+\theader: 4d494458 1 0 0\n+\tobject_dir: .\n+\tEOF\n+\ttest-tool read-midx . >actual &&\n+\ttest_cmp expect actual\n+}\n+\n test_expect_success 'write midx with no packs' '\n \tgit multi-pack-index --object-dir=. write &&\n-\ttest_path_is_file pack/multi-pack-index\n+\ttest_path_is_file pack/multi-pack-index &&\n+\tmidx_read_expect\n '\n \n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350816","messageId":"20180625143434.89044-12-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 11/24] midx: read pack names into array","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:21Z","receivedAt":"2018-06-25T14:35:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 31 +++++++++++++++++++++++++++++++\n object-store.h | 1 +\n t/helper/test-read-midx.c | 5 +++++\n t/t5319-multi-pack-index.sh | 7 ++++++-\n 4 files changed, 43 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 2fad99d1b8..8bbc966f6b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tuint32_t hash_version;\n \tchar *midx_name = get_midx_filename(object_dir);\n \tuint32_t i;\n+\tconst char *cur_pack_name;\n \n \tfd = git_open(midx_name);\n \n@@ -116,6 +117,22 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tif (!m->chunk_pack_names)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n \n+\tm->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n+\n+\tcur_pack_name = (const char *)m->chunk_pack_names;\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tm->pack_names[i] = cur_pack_name;\n+\n+\t\tcur_pack_name += strlen(cur_pack_name) + 1;\n+\n+\t\tif (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0) {\n+\t\t\terror(\"MIDX pack names out of order: '%s' before '%s'\",\n+\t\t\t m->pack_names[i - 1],\n+\t\t\t m->pack_names[i]);\n+\t\t\tgoto cleanup_fail;\n+\t\t}\n+\t}\n+\n \treturn m;\n \n cleanup_fail:\n@@ -210,6 +227,20 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n \tFREE_AND_NULL(pairs);\n }\n \n+static size_t write_midx_pack_lookup(struct hashfile *f,\n+\t\t\t\t char **pack_names,\n+\t\t\t\t uint32_t nr_packs)\n+{\n+\tuint32_t i, cur_len = 0;\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\thashwrite_be32(f, cur_len);\n+\t\tcur_len += strlen(pack_names[i]) + 1;\n+\t}\n+\n+\treturn sizeof(uint32_t) * (size_t)nr_packs;\n+}\n+\n static size_t write_midx_pack_names(struct hashfile *f,\n \t\t\t\t char **pack_names,\n \t\t\t\t uint32_t num_packs)\ndiff --git a/object-store.h b/object-store.h\nindex c87d051849..88169b33e9 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -99,6 +99,7 @@ struct multi_pack_index {\n \n \tconst unsigned char *chunk_pack_names;\n \n+\tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex a9232d8219..0b53a9e8b5 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -9,6 +9,7 @@\n \n static int read_midx_file(const char *object_dir)\n {\n+\tuint32_t i;\n \tstruct multi_pack_index *m = load_multi_pack_index(object_dir);\n \n \tif (!m)\n@@ -27,6 +28,10 @@ static int read_midx_file(const char *object_dir)\n \n \tprintf(\"\\n\");\n \n+\tprintf(\"packs:\\n\");\n+\tfor (i = 0; i < m->num_packs; i++)\n+\t\tprintf(\"%s\\n\", m->pack_names[i]);\n+\n \tprintf(\"object_dir: %s\\n\", m->object_dir);\n \n \treturn 0;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 1b2778961c..800fa7749c 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -8,8 +8,13 @@ midx_read_expect() {\n \tcat >expect <<- EOF\n \theader: 4d494458 1 1 $NUM_PACKS\n \tchunks: pack_names\n-\tobject_dir: .\n+\tpacks:\n \tEOF\n+\tif [ $NUM_PACKS -ge 1 ]\n+\tthen\n+\t\tls pack/ | grep idx | sort >> expect\n+\tfi\n+\tprintf \"object_dir: .\\n\" >>expect &&\n \ttest-tool read-midx . >actual &&\n \ttest_cmp expect actual\n }\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350817","messageId":"20180625143434.89044-13-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 12/24] midx: sort and deduplicate objects from packfiles","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:22Z","receivedAt":"2018-06-25T14:35:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before writing a list of objects and their offsets to a multi-pack-index,\nwe need to collect the list of objects contained in the packfiles. There\nmay be multiple copies of some objects, so this list must be deduplicated.\n\nIt is possible to artificially get into a state where there are many\nduplicate copies of objects. That can create high memory pressure if we\nare to create a list of all objects before de-duplication. To reduce\nthis memory pressure without a significant performance drop,\nautomatically group objects by the first byte of their object id. Use\nthe IDX fanout tables to group the data, copy to a local array, then\nsort.\n\nCopy only the de-duplicated entries. Select the duplicate based on the\nmost-recent modified time of a packfile containing the object.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.c | 17 +++++++\n packfile.h | 2 +\n 3 files changed, 146 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 8bbc966f6b..648a501d74 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -4,6 +4,7 @@\n #include \"lockfile.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"packfile.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n@@ -187,6 +188,13 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\treturn;\n \t\t}\n \n+\t\tif (open_pack_index(packs->list[packs->nr])) {\n+\t\t\twarning(_(\"failed to open pack-index '%s'\"),\n+\t\t\t\tfull_path);\n+\t\t\tclose_pack(packs->list[packs->nr]);\n+\t\t\tFREE_AND_NULL(packs->list[packs->nr]);\n+\t\t}\n+\n \t\tpacks->names[packs->nr] = xstrdup(file_name);\n \t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n@@ -227,6 +235,120 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n \tFREE_AND_NULL(pairs);\n }\n \n+struct pack_midx_entry {\n+\tstruct object_id oid;\n+\tuint32_t pack_int_id;\n+\ttime_t pack_mtime;\n+\tuint64_t offset;\n+};\n+\n+static int midx_oid_compare(const void *_a, const void *_b)\n+{\n+\tconst struct pack_midx_entry *a = (const struct pack_midx_entry *)_a;\n+\tconst struct pack_midx_entry *b = (const struct pack_midx_entry *)_b;\n+\tint cmp = oidcmp(&a->oid, &b->oid);\n+\n+\tif (cmp)\n+\t\treturn cmp;\n+\n+\tif (a->pack_mtime > b->pack_mtime)\n+\t\treturn -1;\n+\telse if (a->pack_mtime < b->pack_mtime)\n+\t\treturn 1;\n+\n+\treturn a->pack_int_id - b->pack_int_id;\n+}\n+\n+static void fill_pack_entry(uint32_t pack_int_id,\n+\t\t\t struct packed_git *p,\n+\t\t\t uint32_t cur_object,\n+\t\t\t struct pack_midx_entry *entry)\n+{\n+\tif (!nth_packed_object_oid(&entry->oid, p, cur_object))\n+\t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n+\n+\tentry->pack_int_id = pack_int_id;\n+\tentry->pack_mtime = p->mtime;\n+\n+\tentry->offset = nth_packed_object_offset(p, cur_object);\n+}\n+\n+/*\n+ * It is possible to artificially get into a state where there are many\n+ * duplicate copies of objects. That can create high memory pressure if\n+ * we are to create a list of all objects before de-duplication. To reduce\n+ * this memory pressure without a significant performance drop, automatically\n+ * group objects by the first byte of their object id. Use the IDX fanout\n+ * tables to group the data, copy to a local array, then sort.\n+ *\n+ * Copy only the de-duplicated entries (selected by most-recent modified time\n+ * of a packfile containing the object).\n+ */\n+static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n+\t\t\t\t\t\t uint32_t *perm,\n+\t\t\t\t\t\t uint32_t nr_packs,\n+\t\t\t\t\t\t uint32_t *nr_objects)\n+{\n+\tuint32_t cur_fanout, cur_pack, cur_object;\n+\tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n+\tstruct pack_midx_entry *entries_by_fanout = NULL;\n+\tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\n+\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\ttotal_objects += p[cur_pack]->num_objects;\n+\t}\n+\n+\t/*\n+\t * As we de-duplicate by fanout value, we expect the fanout\n+\t * slices to be evenly distributed, with some noise. Hence,\n+\t * allocate slightly more than one 256th.\n+\t */\n+\talloc_objects = alloc_fanout = total_objects > 3200 ? total_objects / 200 : 16;\n+\n+\tALLOC_ARRAY(entries_by_fanout, alloc_fanout);\n+\tALLOC_ARRAY(deduplicated_entries, alloc_objects);\n+\t*nr_objects = 0;\n+\n+\tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n+\t\tuint32_t nr_fanout = 0;\n+\n+\t\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\t\tuint32_t start = 0, end;\n+\n+\t\t\tif (cur_fanout)\n+\t\t\t\tstart = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n+\t\t\tend = get_pack_fanout(p[cur_pack], cur_fanout);\n+\n+\t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n+\t\t\t\tfill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tnr_fanout++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tQSORT(entries_by_fanout, nr_fanout, midx_oid_compare);\n+\n+\t\t/*\n+\t\t * The batch is now sorted by OID and then mtime (descending).\n+\t\t * Take only the first duplicate.\n+\t\t */\n+\t\tfor (cur_object = 0; cur_object < nr_fanout; cur_object++) {\n+\t\t\tif (cur_object && !oidcmp(&entries_by_fanout[cur_object - 1].oid,\n+\t\t\t\t\t\t &entries_by_fanout[cur_object].oid))\n+\t\t\t\tcontinue;\n+\n+\t\t\tALLOC_GROW(deduplicated_entries, *nr_objects + 1, alloc_objects);\n+\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n+\t\t\t &entries_by_fanout[cur_object],\n+\t\t\t sizeof(struct pack_midx_entry));\n+\t\t\t(*nr_objects)++;\n+\t\t}\n+\t}\n+\n+\tFREE_AND_NULL(entries_by_fanout);\n+\treturn deduplicated_entries;\n+}\n+\n static size_t write_midx_pack_lookup(struct hashfile *f,\n \t\t\t\t char **pack_names,\n \t\t\t\t uint32_t nr_packs)\n@@ -284,6 +406,8 @@ int write_midx_file(const char *object_dir)\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n+\tuint32_t nr_entries;\n+\tstruct pack_midx_entry *entries;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -309,6 +433,8 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n+\tentries = get_sorted_entries(packs.list, pack_perm, packs.nr, &nr_entries);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n@@ -376,5 +502,6 @@ int write_midx_file(const char *object_dir)\n \n \tFREE_AND_NULL(packs.list);\n \tFREE_AND_NULL(packs.names);\n+\tFREE_AND_NULL(entries);\n \treturn 0;\n }\ndiff --git a/packfile.c b/packfile.c\nindex db61c8813b..39d6b66337 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -196,6 +196,23 @@ int open_pack_index(struct packed_git *p)\n \treturn ret;\n }\n \n+uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)\n+{\n+\tconst uint32_t *level1_ofs = p->index_data;\n+\n+\tif (!level1_ofs) {\n+\t\tif (open_pack_index(p))\n+\t\t\treturn 0;\n+\t\tlevel1_ofs = p->index_data;\n+\t}\n+\n+\tif (p->index_version > 1) {\n+\t\tlevel1_ofs += 2;\n+\t}\n+\n+\treturn ntohl(level1_ofs[value]);\n+}\n+\n static struct packed_git *alloc_packed_git(int extra)\n {\n \tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\ndiff --git a/packfile.h b/packfile.h\nindex d2ad30300a..b0eed44c0b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -69,6 +69,8 @@ extern int open_pack_index(struct packed_git *);\n */\n extern void close_pack_index(struct packed_git *);\n \n+extern uint32_t get_pack_fanout(struct packed_git *p, uint32_t value);\n+\n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void close_pack_windows(struct packed_git *);\n extern void close_pack(struct packed_git *);\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350818","messageId":"20180625143434.89044-14-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 13/24] midx: write object ids in a chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:23Z","receivedAt":"2018-06-25T14:35:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 4 ++\n midx.c | 51 ++++++++++++++++++++++---\n object-store.h | 1 +\n t/helper/test-read-midx.c | 2 +\n t/t5319-multi-pack-index.sh | 4 +-\n 5 files changed, 55 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 6c5a77475f..78ee0489c6 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -302,6 +302,10 @@ CHUNK DATA:\n \t name. This is the only chunk not guaranteed to be a multiple of four\n \t bytes in length, so should be the last chunk for alignment reasons.\n \n+\tOID Lookup (ID: {'O', 'I', 'D', 'L'})\n+\t The OIDs for all objects in the MIDX are stored in lexicographic\n+\t order in this chunk.\n+\n \t(This section intentionally left incomplete.)\n \n TRAILER:\ndiff --git a/midx.c b/midx.c\nindex 648a501d74..aec85b8181 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -14,9 +14,10 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 1\n+#define MIDX_MAX_CHUNKS 2\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n \n static char *get_midx_filename(const char *object_dir)\n@@ -102,6 +103,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n+\t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n \t\t\tcase 0:\n \t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n \t\t\t\tbreak;\n@@ -117,6 +122,8 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tif (!m->chunk_pack_names)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n+\tif (!m->chunk_oid_lookup)\n+\t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n \n@@ -127,7 +134,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\tcur_pack_name += strlen(cur_pack_name) + 1;\n \n \t\tif (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0) {\n-\t\t\terror(\"MIDX pack names out of order: '%s' before '%s'\",\n+\t\t\terror(_(\"multi-pack-index pack names out of order: '%s' before '%s'\"),\n \t\t\t m->pack_names[i - 1],\n \t\t\t m->pack_names[i]);\n \t\t\tgoto cleanup_fail;\n@@ -394,6 +401,32 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \treturn written;\n }\n \n+static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n+\t\t\t\t struct pack_midx_entry *objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tuint32_t i;\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\tif (i < nr_objects - 1) {\n+\t\t\tstruct pack_midx_entry *next = list;\n+\t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n+\t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n+\t\t\t\toid_to_hex(&obj->oid),\n+\t\t\t\toid_to_hex(&next->oid));\n+\t\t}\n+\n+\t\thashwrite(f, obj->oid.hash, (int)hash_len);\n+\t\twritten += hash_len;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n@@ -407,7 +440,7 @@ int write_midx_file(const char *object_dir)\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tuint32_t nr_entries;\n-\tstruct pack_midx_entry *entries;\n+\tstruct pack_midx_entry *entries = NULL;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -440,7 +473,7 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 1;\n+\tnum_chunks = 2;\n \n \twritten = write_midx_header(f, num_chunks, packs.nr);\n \n@@ -448,9 +481,13 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = 0;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n \n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = 0;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n+\n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n \t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n@@ -480,6 +517,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n+\t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tdefault:\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\ndiff --git a/object-store.h b/object-store.h\nindex 88169b33e9..25f8530eb4 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -98,6 +98,7 @@ struct multi_pack_index {\n \tuint32_t num_objects;\n \n \tconst unsigned char *chunk_pack_names;\n+\tconst unsigned char *chunk_oid_lookup;\n \n \tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 0b53a9e8b5..60bca5b668 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -25,6 +25,8 @@ static int read_midx_file(const char *object_dir)\n \n \tif (m->chunk_pack_names)\n \t\tprintf(\" pack_names\");\n+\tif (m->chunk_oid_lookup)\n+\t\tprintf(\" oid_lookup\");\n \n \tprintf(\"\\n\");\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 800fa7749c..47e1c7d99e 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -6,8 +6,8 @@ test_description='multi-pack-indexes'\n midx_read_expect() {\n \tNUM_PACKS=$1\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 1 $NUM_PACKS\n-\tchunks: pack_names\n+\theader: 4d494458 1 2 $NUM_PACKS\n+\tchunks: pack_names oid_lookup\n \tpacks:\n \tEOF\n \tif [ $NUM_PACKS -ge 1 ]\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350819","messageId":"20180625143434.89044-15-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 14/24] midx: write object id fanout chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:24Z","receivedAt":"2018-06-25T14:35:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 5 +++\n midx.c | 53 +++++++++++++++++++++++--\n object-store.h | 1 +\n t/helper/test-read-midx.c | 4 +-\n t/t5319-multi-pack-index.sh | 18 +++++----\n 5 files changed, 69 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 78ee0489c6..3215f7bfcd 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -302,6 +302,11 @@ CHUNK DATA:\n \t name. This is the only chunk not guaranteed to be a multiple of four\n \t bytes in length, so should be the last chunk for alignment reasons.\n \n+\tOID Fanout (ID: {'O', 'I', 'D', 'F'})\n+\t The ith entry, F[i], stores the number of OIDs with first\n+\t byte at most i. Thus F[255] stores the total\n+\t number of objects.\n+\n \tOID Lookup (ID: {'O', 'I', 'D', 'L'})\n \t The OIDs for all objects in the MIDX are stored in lexicographic\n \t order in this chunk.\ndiff --git a/midx.c b/midx.c\nindex aec85b8181..0f773e2585 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -14,11 +14,13 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 2\n+#define MIDX_MAX_CHUNKS 3\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n+#define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n \n static char *get_midx_filename(const char *object_dir)\n {\n@@ -103,6 +105,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n+\t\t\t\tm->chunk_oid_fanout = (uint32_t *)(m->data + chunk_offset);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n \t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n \t\t\t\tbreak;\n@@ -122,9 +128,13 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tif (!m->chunk_pack_names)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n+\tif (!m->chunk_oid_fanout)\n+\t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n \tif (!m->chunk_oid_lookup)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n \n+\tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n+\n \tm->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n \n \tcur_pack_name = (const char *)m->chunk_pack_names;\n@@ -401,6 +411,35 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \treturn written;\n }\n \n+static size_t write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t struct pack_midx_entry *objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tstruct pack_midx_entry *last = objects + nr_objects;\n+\tuint32_t count = 0;\n+\tuint32_t i;\n+\n+\t/*\n+\t* Write the first-level table (the list is sorted,\n+\t* but we use a 256-entry lookup to be able to avoid\n+\t* having to do eight extra binary search iterations).\n+\t*/\n+\tfor (i = 0; i < 256; i++) {\n+\t\tstruct pack_midx_entry *next = list;\n+\n+\t\twhile (next < last && next->oid.hash[0] == i) {\n+\t\t\tcount++;\n+\t\t\tnext++;\n+\t\t}\n+\n+\t\thashwrite_be32(f, count);\n+\t\tlist = next;\n+\t}\n+\n+\treturn MIDX_CHUNK_FANOUT_SIZE;\n+}\n+\n static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n \t\t\t\t struct pack_midx_entry *objects,\n \t\t\t\t uint32_t nr_objects)\n@@ -473,7 +512,7 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 2;\n+\tnum_chunks = 3;\n \n \twritten = write_midx_header(f, num_chunks, packs.nr);\n \n@@ -481,9 +520,13 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n \n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n+\n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = 0;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n@@ -517,6 +560,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n+\t\t\t\twritten += write_midx_oid_fanout(f, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n \t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n \t\t\t\tbreak;\ndiff --git a/object-store.h b/object-store.h\nindex 25f8530eb4..3357e51100 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -98,6 +98,7 @@ struct multi_pack_index {\n \tuint32_t num_objects;\n \n \tconst unsigned char *chunk_pack_names;\n+\tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \n \tconst char **pack_names;\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 60bca5b668..d1bb7290ae 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -25,10 +25,12 @@ static int read_midx_file(const char *object_dir)\n \n \tif (m->chunk_pack_names)\n \t\tprintf(\" pack_names\");\n+\tif (m->chunk_oid_fanout)\n+\t\tprintf(\" oid_fanout\");\n \tif (m->chunk_oid_lookup)\n \t\tprintf(\" oid_lookup\");\n \n-\tprintf(\"\\n\");\n+\tprintf(\"\\nnum_objects: %d\\n\", m->num_objects);\n \n \tprintf(\"packs:\\n\");\n \tfor (i = 0; i < m->num_packs; i++)\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 47e1c7d99e..ad0d447522 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -5,9 +5,11 @@ test_description='multi-pack-indexes'\n \n midx_read_expect() {\n \tNUM_PACKS=$1\n+\tNUM_OBJECTS=$2\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 2 $NUM_PACKS\n-\tchunks: pack_names oid_lookup\n+\theader: 4d494458 1 3 $NUM_PACKS\n+\tchunks: pack_names oid_fanout oid_lookup\n+\tnum_objects: $NUM_OBJECTS\n \tpacks:\n \tEOF\n \tif [ $NUM_PACKS -ge 1 ]\n@@ -22,7 +24,7 @@ midx_read_expect() {\n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\tmidx_read_expect 0\n+\tmidx_read_expect 0 0\n '\n \n test_expect_success 'create objects' '\n@@ -53,17 +55,17 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test wide_delta_$iii &&\n@@ -89,7 +91,7 @@ test_expect_success 'Add more objects' '\n test_expect_success 'write midx with two packs' '\n \tgit pack-objects --index-version=1 pack/test-2 \n---\n Documentation/technical/pack-format.txt | 15 +++-\n midx.c | 100 +++++++++++++++++++++++-\n object-store.h | 2 +\n t/helper/test-read-midx.c | 4 +\n t/t5319-multi-pack-index.sh | 45 ++++++++---\n 5 files changed, 151 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 3215f7bfcd..cab5bdd2ff 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -311,7 +311,20 @@ CHUNK DATA:\n \t The OIDs for all objects in the MIDX are stored in lexicographic\n \t order in this chunk.\n \n-\t(This section intentionally left incomplete.)\n+\tObject Offsets (ID: {'O', 'O', 'F', 'F'})\n+\t Stores two 4-byte values for every object.\n+\t 1: The pack-int-id for the pack storing this object.\n+\t 2: The offset within the pack.\n+\t\tIf all offsets are less than 2^31, then the large offset chunk\n+\t\twill not exist and offsets are stored as in IDX v1.\n+\t\tIf there is at least one offset value larger than 2^32-1, then\n+\t\tthe large offset chunk must exist. If the large offset chunk\n+\t\texists and the 31st bit is on, then removing that bit reveals\n+\t\tthe row in the large offsets containing the 8-byte offset of\n+\t\tthis object.\n+\n+\t[Optional] Object Large Offsets (ID: {'L', 'O', 'F', 'F'})\n+\t 8-byte offsets into large packfiles.\n \n TRAILER:\n \ndiff --git a/midx.c b/midx.c\nindex 0f773e2585..71ca493107 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -14,13 +14,18 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 3\n+#define MIDX_MAX_CHUNKS 5\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n+#define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n+#define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n+#define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n+#define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n+#define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n static char *get_midx_filename(const char *object_dir)\n {\n@@ -113,6 +118,14 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n+\t\t\t\tm->chunk_object_offsets = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n+\t\t\t\tm->chunk_large_offsets = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n \t\t\tcase 0:\n \t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n \t\t\t\tbreak;\n@@ -132,6 +145,8 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n \tif (!m->chunk_oid_lookup)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n+\tif (!m->chunk_object_offsets)\n+\t\tdie(_(\"multi-pack-index missing required object offsets chunk\"));\n \n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n@@ -466,6 +481,56 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n \treturn written;\n }\n \n+static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n+\t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tuint32_t i, nr_large_offset = 0;\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\thashwrite_be32(f, obj->pack_int_id);\n+\n+\t\tif (large_offset_needed && obj->offset >> 31)\n+\t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n+\t\telse if (!large_offset_needed && obj->offset >> 32)\n+\t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n+\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t obj->offset);\n+\t\telse\n+\t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n+\n+\t\twritten += MIDX_CHUNK_OFFSET_WIDTH;\n+\t}\n+\n+\treturn written;\n+}\n+\n+static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_offset,\n+\t\t\t\t struct pack_midx_entry *objects, uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tsize_t written = 0;\n+\n+\twhile (nr_large_offset) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\t\tuint64_t offset = obj->offset;\n+\n+\t\tif (!(offset >> 31))\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be32(f, offset >> 32);\n+\t\thashwrite_be32(f, offset & 0xffffffffUL);\n+\t\twritten += 2 * sizeof(uint32_t);\n+\n+\t\tnr_large_offset--;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n@@ -478,8 +543,9 @@ int write_midx_file(const char *object_dir)\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t nr_entries;\n+\tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n+\tint large_offsets_needed = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -506,13 +572,19 @@ int write_midx_file(const char *object_dir)\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n \tentries = get_sorted_entries(packs.list, pack_perm, packs.nr, &nr_entries);\n+\tfor (i = 0; i < nr_entries; i++) {\n+\t\tif (entries[i].offset > 0x7fffffff)\n+\t\t\tnum_large_offsets++;\n+\t\tif (entries[i].offset > 0xffffffff)\n+\t\t\tlarge_offsets_needed = 1;\n+\t}\n \n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 3;\n+\tnum_chunks = large_offsets_needed ? 5 : 4;\n \n \twritten = write_midx_header(f, num_chunks, packs.nr);\n \n@@ -528,9 +600,21 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = 0;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n \n+\tcur_chunk++;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_CHUNK_OFFSET_WIDTH;\n+\tif (large_offsets_needed) {\n+\t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n+\n+\t\tcur_chunk++;\n+\t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n+\t\t\t\t\t num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n+\t}\n+\n+\tchunk_ids[cur_chunk] = 0;\n+\n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n \t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n@@ -568,6 +652,14 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n+\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n+\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tdefault:\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\ndiff --git a/object-store.h b/object-store.h\nindex 3357e51100..07bcc80e02 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -100,6 +100,8 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_pack_names;\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n+\tconst unsigned char *chunk_object_offsets;\n+\tconst unsigned char *chunk_large_offsets;\n \n \tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex d1bb7290ae..20771d1c1d 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -29,6 +29,10 @@ static int read_midx_file(const char *object_dir)\n \t\tprintf(\" oid_fanout\");\n \tif (m->chunk_oid_lookup)\n \t\tprintf(\" oid_lookup\");\n+\tif (m->chunk_object_offsets)\n+\t\tprintf(\" object_offsets\");\n+\tif (m->chunk_large_offsets)\n+\t\tprintf(\" large_offsets\");\n \n \tprintf(\"\\nnum_objects: %d\\n\", m->num_objects);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex ad0d447522..ccde83bca4 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -6,25 +6,28 @@ test_description='multi-pack-indexes'\n midx_read_expect() {\n \tNUM_PACKS=$1\n \tNUM_OBJECTS=$2\n+\tNUM_CHUNKS=$3\n+\tOBJECT_DIR=$4\n+\tEXTRA_CHUNKS=\"$5\"\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 3 $NUM_PACKS\n-\tchunks: pack_names oid_fanout oid_lookup\n+\theader: 4d494458 1 $NUM_CHUNKS $NUM_PACKS\n+\tchunks: pack_names oid_fanout oid_lookup object_offsets$EXTRA_CHUNKS\n \tnum_objects: $NUM_OBJECTS\n \tpacks:\n \tEOF\n \tif [ $NUM_PACKS -ge 1 ]\n \tthen\n-\t\tls pack/ | grep idx | sort >> expect\n+\t\tls $OBJECT_DIR/pack/ | grep idx | sort >> expect\n \tfi\n-\tprintf \"object_dir: .\\n\" >>expect &&\n-\ttest-tool read-midx . >actual &&\n+\tprintf \"object_dir: $OBJECT_DIR\\n\" >>expect &&\n+\ttest-tool read-midx $OBJECT_DIR >actual &&\n \ttest_cmp expect actual\n }\n \n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\tmidx_read_expect 0 0\n+\tmidx_read_expect 0 0 4 .\n '\n \n test_expect_success 'create objects' '\n@@ -55,13 +58,13 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test []\n+corrupt_data() {\n+\tfile=$1\n+\tpos=$2\n+\tdata=\"${3:-\\0}\"\n+\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n+}\n+\n+# Force 64-bit offsets by manipulating the idx file.\n+# This makes the IDX file _incorrect_ so be careful to clean up after!\n+test_expect_success 'force some 64-bit offsets with pack-objects' '\n+\tmkdir objects64 &&\n+\tmkdir objects64/pack &&\n+\tpack64=$(git pack-objects --index-version=2,0x40 objects64/pack/test-64 \n---\n Documentation/config.txt | 4 +++\n cache.h | 1 +\n config.c | 5 ++++\n environment.c | 1 +\n t/t5319-multi-pack-index.sh | 55 +++++++++++++++++++++++++++++++------\n 5 files changed, 57 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ab641bf5a9..ab895ebb32 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -908,6 +908,10 @@ core.commitGraph::\n \tEnable git commit graph feature. Allows reading from the\n \tcommit-graph file.\n \n+core.multiPackIndex::\n+\tUse the multi-pack-index file to track multiple packfiles using a\n+\tsingle index. See linkgit:technical/multi-pack-index[1].\n+\n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n \tlinkgit:git-read-tree[1] for more information.\ndiff --git a/cache.h b/cache.h\nindex 89a107a7f7..d12aa49710 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -814,6 +814,7 @@ extern char *git_replace_ref_base;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_commit_graph;\n+extern int core_multi_pack_index;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex fbbf0f8e9f..95d8da4243 100644\n--- a/config.c\n+++ b/config.c\n@@ -1313,6 +1313,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.multipackindex\")) {\n+\t\tcore_multi_pack_index = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.sparsecheckout\")) {\n \t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 2a6de2330b..b9bc919cdb 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -67,6 +67,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_commit_graph;\n+int core_multi_pack_index;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex ccde83bca4..f7f55ea181 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,6 +3,8 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n+objdir=.git/objects\n+\n midx_read_expect() {\n \tNUM_PACKS=$1\n \tNUM_OBJECTS=$2\n@@ -61,12 +63,42 @@ test_expect_success 'write midx with one v1 pack' '\n \tmidx_read_expect 1 17 4 .\n '\n \n+midx_git_two_modes() {\n+\tgit -c core.multiPackIndex=false $1 >expect &&\n+\tgit -c core.multiPackIndex=true $1 >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx() {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\"\n+\t'\n+}\n+\n test_expect_success 'write midx with one v2 pack' '\n-\tgit pack-objects --index-version=2,0x40 pack/test expect &&\n+\tgit -c core.multiPackIndex=true $1 >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx() {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\"\n+\t'\n+}\n+\n+compare_results_with_midx \"one v2 pack\"\n+\n test_expect_success 'Add more objects' '\n \tfor i in `test_seq 6 10`\n \tdo\n@@ -92,11 +124,13 @@ test_expect_success 'Add more objects' '\n '\n \n test_expect_success 'write midx with two packs' '\n-\tgit pack-objects --index-version=1 pack/test-2 obj-list &&\n \t\tgit update-ref HEAD $commit &&\n-\t\tgit pack-objects --index-version=2 pack/test-pack []\n corrupt_data() {\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350822","messageId":"20180625143434.89044-19-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 18/24] midx: read objects from multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:28Z","receivedAt":"2018-06-25T14:35:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++--\n midx.h | 2 ++\n object-store.h | 1 +\n packfile.c | 8 ++++-\n 4 files changed, 101 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3dd5027dc6..14514d6828 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -4,7 +4,7 @@\n #include \"lockfile.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n-#include \"packfile.h\"\n+#include \"sha1-lookup.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n@@ -150,7 +150,8 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n-\tm->pack_names = xcalloc(m->num_packs, sizeof(const char *));\n+\tm->packs = xcalloc(m->num_packs, sizeof(*m->packs));\n+\tALLOC_ARRAY(m->pack_names, m->num_packs);\n \n \tcur_pack_name = (const char *)m->chunk_pack_names;\n \tfor (i = 0; i < m->num_packs; i++) {\n@@ -176,6 +177,94 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \treturn NULL;\n }\n \n+static int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id)\n+{\n+\tstruct strbuf pack_name = STRBUF_INIT;\n+\n+\tif (pack_int_id >= m->num_packs)\n+\t\tBUG(\"bad pack-int-id\");\n+\n+\tif (m->packs[pack_int_id])\n+\t\treturn 0;\n+\n+\tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n+\t\t m->pack_names[pack_int_id]);\n+\n+\tm->packs[pack_int_id] = add_packed_git(pack_name.buf, pack_name.len, 1);\n+\tstrbuf_release(&pack_name);\n+\treturn !m->packs[pack_int_id];\n+}\n+\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)\n+{\n+\treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n+\t\t\t MIDX_HASH_LEN, result);\n+}\n+\n+static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n+{\n+\tconst unsigned char *offset_data;\n+\tuint32_t offset32;\n+\n+\toffset_data = m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH;\n+\toffset32 = get_be32(offset_data + sizeof(uint32_t));\n+\n+\tif (m->chunk_large_offsets && offset32 & MIDX_LARGE_OFFSET_NEEDED) {\n+\t\tif (sizeof(offset32) < sizeof(uint64_t))\n+\t\t\tdie(_(\"multi-pack-index stores a 64-bit offset, but off_t is too small\"));\n+\n+\t\toffset32 ^= MIDX_LARGE_OFFSET_NEEDED;\n+\t\treturn get_be64(m->chunk_large_offsets + sizeof(uint64_t) * offset32);\n+\t}\n+\n+\treturn offset32;\n+}\n+\n+static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n+{\n+\treturn get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);\n+}\n+\n+static int nth_midxed_pack_entry(struct multi_pack_index *m, struct pack_entry *e, uint32_t pos)\n+{\n+\tuint32_t pack_int_id;\n+\tstruct packed_git *p;\n+\n+\tif (pos >= m->num_objects)\n+\t\treturn 0;\n+\n+\tpack_int_id = nth_midxed_pack_int_id(m, pos);\n+\n+\tif (prepare_midx_pack(m, pack_int_id))\n+\t\tdie(_(\"error preparing packfile from multi-pack-index\"));\n+\tp = m->packs[pack_int_id];\n+\n+\t/*\n+\t* We are about to tell the caller where they can locate the\n+\t* requested object. We better make sure the packfile is\n+\t* still here and can be accessed before supplying that\n+\t* answer, as it may have been deleted since the MIDX was\n+\t* loaded!\n+\t*/\n+\tif (!is_pack_valid(p))\n+\t\treturn 0;\n+\n+\te->offset = nth_midxed_offset(m, pos);\n+\te->p = p;\n+\n+\treturn 1;\n+}\n+\n+int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m)\n+{\n+\tuint32_t pos;\n+\n+\tif (!bsearch_midx(oid, m, &pos))\n+\t\treturn 0;\n+\n+\treturn nth_midxed_pack_entry(m, e, pos);\n+}\n+\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n {\n \tstruct multi_pack_index *m = r->objects->multi_pack_index;\ndiff --git a/midx.h b/midx.h\nindex 731ad6f094..6b74a0640f 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -6,6 +6,8 @@\n struct multi_pack_index;\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\ndiff --git a/object-store.h b/object-store.h\nindex 7d67ad7aa9..03cc278758 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -106,6 +106,7 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_large_offsets;\n \n \tconst char **pack_names;\n+\tstruct packed_git **packs;\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/packfile.c b/packfile.c\nindex ff2df22a0b..946d0c241f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1904,11 +1904,17 @@ static int fill_pack_entry(const struct object_id *oid,\n int find_pack_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e)\n {\n \tstruct list_head *pos;\n+\tstruct multi_pack_index *m;\n \n \tprepare_packed_git(r);\n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packed_git && !r->objects->multi_pack_index)\n \t\treturn 0;\n \n+\tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n+\t\tif (fill_midx_entry(oid, e, m))\n+\t\t\treturn 1;\n+\t}\n+\n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (fill_pack_entry(oid, e, p)) {\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350823","messageId":"20180625143434.89044-11-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 10/24] multi-pack-index: write pack names in chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:20Z","receivedAt":"2018-06-25T14:35:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index needs to track which packfiles it indexes. Store\nthese in our first required chunk. Since filenames are not well\nstructured, add padding to keep good alignment in later chunks.\n\nModify the 'git multi-pack-index read' subcommand to output the\nexistence of the pack-file name chunk. Modify t5319-multi-pack-index.sh\nto reflect this new output and the new expected number of chunks.\n\nDefense in depth: A pattern we are using in the multi-pack-index feature\nis to verify the data as we write it. We want to ensure we never write\ninvalid data to the multi-pack-index. There are many checks that verify\nthat the values we are writing fit the format definitions. This mainly\nhelps developers while working on the feature, but it can also identify\nissues that only appear when dealing with very large data sets. These\nlarge sets are hard to encode into test cases.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 6 +\n midx.c | 190 ++++++++++++++++++++++--\n object-store.h | 2 +\n t/helper/test-read-midx.c | 7 +\n t/t5319-multi-pack-index.sh | 3 +-\n 5 files changed, 198 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex e060e693f4..6c5a77475f 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -296,6 +296,12 @@ CHUNK LOOKUP:\n \n CHUNK DATA:\n \n+\tPackfile Names (ID: {'P', 'N', 'A', 'M'})\n+\t Stores the packfile names as concatenated, null-terminated strings.\n+\t Packfiles must be listed in lexicographic order for fast lookups by\n+\t name. This is the only chunk not guaranteed to be a multiple of four\n+\t bytes in length, so should be the last chunk for alignment reasons.\n+\n \t(This section intentionally left incomplete.)\n \n TRAILER:\ndiff --git a/midx.c b/midx.c\nindex e79ffb5576..2fad99d1b8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -13,6 +13,11 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n+#define MIDX_MAX_CHUNKS 1\n+#define MIDX_CHUNK_ALIGNMENT 4\n+#define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n@@ -27,6 +32,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tvoid *midx_map = NULL;\n \tuint32_t hash_version;\n \tchar *midx_name = get_midx_filename(object_dir);\n+\tuint32_t i;\n \n \tfd = git_open(midx_name);\n \n@@ -83,6 +89,33 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tm->num_packs = get_be32(m->data + 8);\n \n+\tfor (i = 0; i < m->num_chunks; i++) {\n+\t\tuint32_t chunk_id = get_be32(m->data + MIDX_HEADER_SIZE +\n+\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n+\t\tuint64_t chunk_offset = get_be64(m->data + MIDX_HEADER_SIZE + 4 +\n+\t\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n+\n+\t\tswitch (chunk_id) {\n+\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n+\t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase 0:\n+\t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\t/*\n+\t\t\t\t * Do nothing on unrecognized chunks, allowing future\n+\t\t\t\t * extensions to add optional chunks.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (!m->chunk_pack_names)\n+\t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n+\n \treturn m;\n \n cleanup_fail:\n@@ -112,8 +145,11 @@ static size_t write_midx_header(struct hashfile *f,\n struct pack_list\n {\n \tstruct packed_git **list;\n+\tchar **names;\n \tuint32_t nr;\n-\tuint32_t alloc;\n+\tuint32_t alloc_list;\n+\tuint32_t alloc_names;\n+\tsize_t pack_name_concat_len;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -122,27 +158,101 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \tstruct pack_list *packs = (struct pack_list *)data;\n \n \tif (ends_with(file_name, \".idx\")) {\n-\t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc);\n+\t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n+\t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n \n \t\tpacks->list[packs->nr] = add_packed_git(full_path,\n \t\t\t\t\t\t\t full_path_len,\n \t\t\t\t\t\t\t 0);\n-\t\tif (!packs->list[packs->nr])\n+\t\tif (!packs->list[packs->nr]) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n-\t\telse\n-\t\t\tpacks->nr++;\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tpacks->names[packs->nr] = xstrdup(file_name);\n+\t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n+\t\tpacks->nr++;\n+\t}\n+}\n+\n+struct pack_pair {\n+\tuint32_t pack_int_id;\n+\tchar *pack_name;\n+};\n+\n+static int pack_pair_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_pair *a = (struct pack_pair *)_a;\n+\tstruct pack_pair *b = (struct pack_pair *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n+static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n+{\n+\tuint32_t i;\n+\tstruct pack_pair *pairs;\n+\n+\tALLOC_ARRAY(pairs, nr_packs);\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\tpairs[i].pack_int_id = i;\n+\t\tpairs[i].pack_name = pack_names[i];\n+\t}\n+\n+\tQSORT(pairs, nr_packs, pack_pair_compare);\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\tpack_names[i] = pairs[i].pack_name;\n+\t\tperm[pairs[i].pack_int_id] = i;\n+\t}\n+\n+\tFREE_AND_NULL(pairs);\n+}\n+\n+static size_t write_midx_pack_names(struct hashfile *f,\n+\t\t\t\t char **pack_names,\n+\t\t\t\t uint32_t num_packs)\n+{\n+\tuint32_t i;\n+\tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < num_packs; i++) {\n+\t\tsize_t writelen = strlen(pack_names[i]) + 1;\n+\n+\t\tif (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n+\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n+\t\t\t pack_names[i - 1],\n+\t\t\t pack_names[i]);\n+\n+\t\thashwrite(f, pack_names[i], writelen);\n+\t\twritten += writelen;\n+\t}\n+\n+\t/* add padding to be aligned */\n+\ti = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n+\tif (i < MIDX_CHUNK_ALIGNMENT) {\n+\t\tbzero(padding, sizeof(padding));\n+\t\thashwrite(f, padding, i);\n+\t\twritten += i;\n \t}\n+\n+\treturn written;\n }\n \n int write_midx_file(const char *object_dir)\n {\n-\tunsigned char num_chunks = 0;\n+\tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n \tuint32_t i;\n \tstruct hashfile *f;\n \tstruct lock_file lk;\n \tstruct pack_list packs;\n+\tuint32_t *pack_perm;\n+\tuint64_t written = 0;\n+\tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n+\tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -152,17 +262,77 @@ int write_midx_file(const char *object_dir)\n \t}\n \n \tpacks.nr = 0;\n-\tpacks.alloc = 16;\n+\tpacks.alloc_list = 16;\n+\tpacks.alloc_names = 16;\n \tpacks.list = NULL;\n-\tALLOC_ARRAY(packs.list, packs.alloc);\n+\tpacks.pack_name_concat_len = 0;\n+\tALLOC_ARRAY(packs.list, packs.alloc_list);\n+\tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n+\tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t(packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n+\tALLOC_ARRAY(pack_perm, packs.nr);\n+\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\twrite_midx_header(f, num_chunks, packs.nr);\n+\tcur_chunk = 0;\n+\tnum_chunks = 1;\n+\n+\twritten = write_midx_header(f, num_chunks, packs.nr);\n+\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n+\tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n+\n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = 0;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n+\n+\tfor (i = 0; i <= num_chunks; i++) {\n+\t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n+\t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n+\t\t\t chunk_offsets[i - 1],\n+\t\t\t chunk_offsets[i]);\n+\n+\t\tif (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n+\t\t\tBUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n+\t\t\t chunk_offsets[i]);\n+\n+\t\thashwrite_be32(f, chunk_ids[i]);\n+\t\thashwrite_be32(f, chunk_offsets[i] >> 32);\n+\t\thashwrite_be32(f, chunk_offsets[i]);\n+\n+\t\twritten += MIDX_CHUNKLOOKUP_WIDTH;\n+\t}\n+\n+\tfor (i = 0; i < num_chunks; i++) {\n+\t\tif (written != chunk_offsets[i])\n+\t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n+\t\t\t chunk_offsets[i],\n+\t\t\t written,\n+\t\t\t chunk_ids[i]);\n+\n+\t\tswitch (chunk_ids[i]) {\n+\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n+\t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n+\t\t\t\t chunk_ids[i]);\n+\t\t}\n+\t}\n+\n+\tif (written != chunk_offsets[num_chunks])\n+\t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n+\t\t written,\n+\t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n@@ -170,8 +340,10 @@ int write_midx_file(const char *object_dir)\n \tfor (i = 0; i < packs.nr; i++) {\n \t\tclose_pack(packs.list[i]);\n \t\tFREE_AND_NULL(packs.list[i]);\n+\t\tFREE_AND_NULL(packs.names[i]);\n \t}\n \n \tFREE_AND_NULL(packs.list);\n+\tFREE_AND_NULL(packs.names);\n \treturn 0;\n }\ndiff --git a/object-store.h b/object-store.h\nindex 4f410841cc..c87d051849 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -97,6 +97,8 @@ struct multi_pack_index {\n \tuint32_t num_packs;\n \tuint32_t num_objects;\n \n+\tconst unsigned char *chunk_pack_names;\n+\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 5abf969175..a9232d8219 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -20,6 +20,13 @@ static int read_midx_file(const char *object_dir)\n \t m->num_chunks,\n \t m->num_packs);\n \n+\tprintf(\"chunks:\");\n+\n+\tif (m->chunk_pack_names)\n+\t\tprintf(\" pack_names\");\n+\n+\tprintf(\"\\n\");\n+\n \tprintf(\"object_dir: %s\\n\", m->object_dir);\n \n \treturn 0;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 4d4d6ca0a6..1b2778961c 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -6,7 +6,8 @@ test_description='multi-pack-indexes'\n midx_read_expect() {\n \tNUM_PACKS=$1\n \tcat >expect <<- EOF\n-\theader: 4d494458 1 0 $NUM_PACKS\n+\theader: 4d494458 1 1 $NUM_PACKS\n+\tchunks: pack_names\n \tobject_dir: .\n \tEOF\n \ttest-tool read-midx . >actual &&\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350824","messageId":"20180625143434.89044-20-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 19/24] midx: use midx in abbreviation calculations","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:29Z","receivedAt":"2018-06-25T14:35:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 11 ++++++\n midx.h | 3 ++\n packfile.c | 6 ++++\n packfile.h | 1 +\n sha1-name.c | 70 +++++++++++++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh | 3 +-\n 6 files changed, 93 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 14514d6828..c258e3ebdf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -201,6 +201,17 @@ int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32\n \t\t\t MIDX_HASH_LEN, result);\n }\n \n+struct object_id *nth_midxed_object_oid(struct object_id *oid,\n+\t\t\t\t\tstruct multi_pack_index *m,\n+\t\t\t\t\tuint32_t n)\n+{\n+\tif (n >= m->num_objects)\n+\t\treturn NULL;\n+\n+\thashcpy(oid->hash, m->chunk_oid_lookup + m->hash_len * n);\n+\treturn oid;\n+}\n+\n static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n {\n \tconst unsigned char *offset_data;\ndiff --git a/midx.h b/midx.h\nindex 6b74a0640f..f7c2ec7893 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -7,6 +7,9 @@ struct multi_pack_index;\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+struct object_id *nth_midxed_object_oid(struct object_id *oid,\n+\t\t\t\t\tstruct multi_pack_index *m,\n+\t\t\t\t\tuint32_t n);\n int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \ndiff --git a/packfile.c b/packfile.c\nindex 946d0c241f..20b743da91 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -963,6 +963,12 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packed_git;\n }\n \n+struct multi_pack_index *get_multi_pack_index(struct repository *r)\n+{\n+\tprepare_packed_git(r);\n+\treturn r->objects->multi_pack_index;\n+}\n+\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\ndiff --git a/packfile.h b/packfile.h\nindex b0eed44c0b..046280caf3 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -45,6 +45,7 @@ extern void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n+struct multi_pack_index *get_multi_pack_index(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/sha1-name.c b/sha1-name.c\nindex 60d9ef3c7e..7dc71201e6 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -12,6 +12,7 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"repository.h\"\n+#include \"midx.h\"\n \n static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);\n \n@@ -149,6 +150,32 @@ static int match_sha(unsigned len, const unsigned char *a, const unsigned char *\n \treturn 1;\n }\n \n+static void unique_in_midx(struct multi_pack_index *m,\n+\t\t\t struct disambiguate_state *ds)\n+{\n+\tuint32_t num, i, first = 0;\n+\tconst struct object_id *current = NULL;\n+\tnum = m->num_objects;\n+\n+\tif (!num)\n+\t\treturn;\n+\n+\tbsearch_midx(&ds->bin_pfx, m, &first);\n+\n+\t/*\n+\t * At this point, \"first\" is the location of the lowest object\n+\t * with an object name that could match \"bin_pfx\". See if we have\n+\t * 0, 1 or more objects that actually match(es).\n+\t */\n+\tfor (i = first; i < num && !ds->ambiguous; i++) {\n+\t\tstruct object_id oid;\n+\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n+\t\tif (!match_sha(ds->len, ds->bin_pfx.hash, current->hash))\n+\t\t\tbreak;\n+\t\tupdate_candidates(ds, current);\n+\t}\n+}\n+\n static void unique_in_pack(struct packed_git *p,\n \t\t\t struct disambiguate_state *ds)\n {\n@@ -177,8 +204,12 @@ static void unique_in_pack(struct packed_git *p,\n \n static void find_short_packed_object(struct disambiguate_state *ds)\n {\n+\tstruct multi_pack_index *m;\n \tstruct packed_git *p;\n \n+\tfor (m = get_multi_pack_index(the_repository); m && !ds->ambiguous;\n+\t m = m->next)\n+\t\tunique_in_midx(m, ds);\n \tfor (p = get_packed_git(the_repository); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n@@ -527,6 +558,42 @@ static int extend_abbrev_len(const struct object_id *oid, void *cb_data)\n \treturn 0;\n }\n \n+static void find_abbrev_len_for_midx(struct multi_pack_index *m,\n+\t\t\t\t struct min_abbrev_data *mad)\n+{\n+\tint match = 0;\n+\tuint32_t num, first = 0;\n+\tstruct object_id oid;\n+\tconst struct object_id *mad_oid;\n+\n+\tif (!m->num_objects)\n+\t\treturn;\n+\n+\tnum = m->num_objects;\n+\tmad_oid = mad->oid;\n+\tmatch = bsearch_midx(mad_oid, m, &first);\n+\n+\t/*\n+\t * first is now the position in the packfile where we would insert\n+\t * mad->hash if it does not exist (or the position of mad->hash if\n+\t * it does exist). Hence, we consider a maximum of two objects\n+\t * nearby for the abbreviation length.\n+\t */\n+\tmad->init_len = 0;\n+\tif (!match) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t} else if (first < num - 1) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t}\n+\tif (first > 0) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t}\n+\tmad->init_len = mad->cur_len;\n+}\n+\n static void find_abbrev_len_for_pack(struct packed_git *p,\n \t\t\t\t struct min_abbrev_data *mad)\n {\n@@ -565,8 +632,11 @@ static void find_abbrev_len_for_pack(struct packed_git *p,\n \n static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n {\n+\tstruct multi_pack_index *m;\n \tstruct packed_git *p;\n \n+\tfor (m = get_multi_pack_index(the_repository); m; m = m->next)\n+\t\tfind_abbrev_len_for_midx(m, mad);\n \tfor (p = get_packed_git(the_repository); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex f7f55ea181..d8a636c7b7 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -93,7 +93,8 @@ compare_results_with_midx() {\n \tMSG=$1\n \ttest_expect_success \"check normal git operations: $MSG\" '\n \t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n-\t\tmidx_git_two_modes \"log --raw\"\n+\t\tmidx_git_two_modes \"log --raw\" &&\n+\t\tmidx_git_two_modes \"log --oneline\"\n \t'\n }\n \n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350825","messageId":"20180625143434.89044-18-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 17/24] midx: prepare midxed_git struct","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:27Z","receivedAt":"2018-06-25T14:35:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 22 ++++++++++++++++++++++\n midx.h | 3 +++\n object-store.h | 9 +++++++++\n packfile.c | 6 +++++-\n 4 files changed, 39 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 71ca493107..3dd5027dc6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -176,6 +176,28 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \treturn NULL;\n }\n \n+int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n+{\n+\tstruct multi_pack_index *m = r->objects->multi_pack_index;\n+\tstruct multi_pack_index *m_search;\n+\n+\tif (!core_multi_pack_index)\n+\t\treturn 0;\n+\n+\tfor (m_search = m; m_search; m_search = m_search->next)\n+\t\tif (!strcmp(object_dir, m_search->object_dir))\n+\t\t\treturn 1;\n+\n+\tr->objects->multi_pack_index = load_multi_pack_index(object_dir);\n+\n+\tif (r->objects->multi_pack_index) {\n+\t\tr->objects->multi_pack_index->next = m;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static size_t write_midx_header(struct hashfile *f,\n \t\t\t\tunsigned char num_chunks,\n \t\t\t\tuint32_t num_packs)\ndiff --git a/midx.h b/midx.h\nindex 2d83dd9ec1..731ad6f094 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -1,9 +1,12 @@\n #ifndef __MIDX_H__\n #define __MIDX_H__\n \n+#include \"repository.h\"\n+\n struct multi_pack_index;\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n+int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n \ndiff --git a/object-store.h b/object-store.h\nindex 07bcc80e02..7d67ad7aa9 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -85,6 +85,8 @@ struct packed_git {\n };\n \n struct multi_pack_index {\n+\tstruct multi_pack_index *next;\n+\n \tint fd;\n \n \tconst unsigned char *data;\n@@ -126,6 +128,13 @@ struct raw_object_store {\n \t */\n \tstruct oidmap *replace_map;\n \n+\t/*\n+\t * private data\n+\t *\n+\t * should only be accessed directly by packfile.c and midx.c\n+\t */\n+\tstruct multi_pack_index *multi_pack_index;\n+\n \t/*\n \t * private data\n \t *\ndiff --git a/packfile.c b/packfile.c\nindex 39d6b66337..ff2df22a0b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -15,6 +15,7 @@\n #include \"tree-walk.h\"\n #include \"tree.h\"\n #include \"object-store.h\"\n+#include \"midx.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -937,10 +938,13 @@ static void prepare_packed_git(struct repository *r)\n \n \tif (r->objects->packed_git_initialized)\n \t\treturn;\n+\tprepare_multi_pack_index_one(r, r->objects->objectdir);\n \tprepare_packed_git_one(r, r->objects->objectdir, 1);\n \tprepare_alt_odb(r);\n-\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next)\n+\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\tprepare_multi_pack_index_one(r, alt->path);\n \t\tprepare_packed_git_one(r, alt->path, 0);\n+\t}\n \trearrange_packed_git(r);\n \tprepare_packed_git_mru(r);\n \tr->objects->packed_git_initialized = 1;\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350826","messageId":"20180625143434.89044-24-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 23/24] packfile: skip loading index if in multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:33Z","receivedAt":"2018-06-25T14:35:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n packfile.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex f2b8d6f8a7..acd02430a8 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -469,8 +469,19 @@ static int open_packed_git_1(struct packed_git *p)\n \tssize_t read_result;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \n-\tif (!p->index_data && open_pack_index(p))\n-\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\tif (!p->index_data) {\n+\t\tstruct multi_pack_index *m;\n+\t\tconst char *pack_name = strrchr(p->pack_name, '/');\n+\n+\t\tfor (m = the_repository->objects->multi_pack_index;\n+\t\t m; m = m->next) {\n+\t\t\tif (midx_contains_pack(m, pack_name))\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!m && open_pack_index(p))\n+\t\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\t}\n \n \tif (!pack_max_fds) {\n \t\tunsigned int max_fds = get_max_fd_limit();\n@@ -521,6 +532,10 @@ static int open_packed_git_1(struct packed_git *p)\n \t\t\t\" supported (try upgrading GIT to a newer version)\",\n \t\t\tp->pack_name, ntohl(hdr.hdr_version));\n \n+\t/* Skip index checking if in multi-pack-index */\n+\tif (!p->index_data)\n+\t\treturn 0;\n+\n \t/* Verify the pack matches its index. */\n \tif (p->num_objects != ntohl(hdr.hdr_entries))\n \t\treturn error(\"packfile %s claims to have %\"PRIu32\" objects\"\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350827","messageId":"20180625143434.89044-25-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 24/24] midx: clear midx on repack","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:34Z","receivedAt":"2018-06-25T14:35:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If a 'git repack' command replaces existing packfiles, then we must\nclear the existing multi-pack-index before moving the packfiles it\nreferences.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/repack.c | 8 ++++++++\n midx.c | 8 ++++++++\n midx.h | 1 +\n 3 files changed, 17 insertions(+)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 6c636e159e..66a7d8e8ea 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -8,6 +8,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"argv-array.h\"\n+#include \"midx.h\"\n \n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -174,6 +175,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n \tint local = 0;\n+\tint midx_cleared = 0;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -340,6 +342,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\t}\n \n+\t\t\tif (!midx_cleared) {\n+\t\t\t\t/* if we move a packfile, it will invalidated the midx */\n+\t\t\t\tclear_midx_file(get_object_directory());\n+\t\t\t\tmidx_cleared = 1;\n+\t\t\t}\n+\n \t\t\tfname_old = mkpathdup(\"%s/old-%s%s\", packdir,\n \t\t\t\t\t\titem->string, exts[ext].name);\n \t\t\tif (file_exists(fname_old))\ndiff --git a/midx.c b/midx.c\nindex 02cbfc5bd5..ef9fb38610 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -890,3 +890,11 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(pack_perm);\n \treturn 0;\n }\n+\n+void clear_midx_file(const char *object_dir)\n+{\n+\tchar *midx = get_midx_filename(object_dir);\n+\n+\tif (remove_path(midx))\n+\t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n+}\ndiff --git a/midx.h b/midx.h\nindex 5faffb7bc6..5a42cbed1d 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -15,5 +15,6 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n+void clear_midx_file(const char *object_dir);\n \n #endif\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350828","messageId":"20180625143434.89044-21-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 20/24] midx: use existing midx when writing new one","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:30Z","receivedAt":"2018-06-25T14:35:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++++++-------\n midx.h | 1 +\n 2 files changed, 96 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex c258e3ebdf..02cbfc5bd5 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -47,7 +47,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tfd = git_open(midx_name);\n \n \tif (fd < 0) {\n-\t\terror_errno(_(\"failed to read %s\"), midx_name);\n \t\tFREE_AND_NULL(midx_name);\n \t\treturn NULL;\n \t}\n@@ -276,6 +275,29 @@ int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct mu\n \treturn nth_midxed_pack_entry(m, e, pos);\n }\n \n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_name)\n+{\n+\tuint32_t first = 0, last = m->num_packs;\n+\n+\twhile (first < last) {\n+\t\tuint32_t mid = first + (last - first) / 2;\n+\t\tconst char *current;\n+\t\tint cmp;\n+\n+\t\tcurrent = m->pack_names[mid];\n+\t\tcmp = strcmp(idx_name, current);\n+\t\tif (!cmp)\n+\t\t\treturn 1;\n+\t\tif (cmp > 0) {\n+\t\t\tfirst = mid + 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlast = mid;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n {\n \tstruct multi_pack_index *m = r->objects->multi_pack_index;\n@@ -322,6 +344,7 @@ struct pack_list\n \tuint32_t alloc_list;\n \tuint32_t alloc_names;\n \tsize_t pack_name_concat_len;\n+\tstruct multi_pack_index *m;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -330,6 +353,9 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \tstruct pack_list *packs = (struct pack_list *)data;\n \n \tif (ends_with(file_name, \".idx\")) {\n+\t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n+\t\t\treturn;\n+\n \t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n \t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n \n@@ -413,6 +439,23 @@ static int midx_oid_compare(const void *_a, const void *_b)\n \treturn a->pack_int_id - b->pack_int_id;\n }\n \n+static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n+\t\t\t\t uint32_t *pack_perm,\n+\t\t\t\t struct pack_midx_entry *e,\n+\t\t\t\t uint32_t pos)\n+{\n+\tif (pos >= m->num_objects)\n+\t\treturn 1;\n+\n+\tnth_midxed_object_oid(&e->oid, m, pos);\n+\te->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];\n+\te->offset = nth_midxed_offset(m, pos);\n+\n+\t/* consider objects in midx to be from \"old\" packs */\n+\te->pack_mtime = 0;\n+\treturn 0;\n+}\n+\n static void fill_pack_entry(uint32_t pack_int_id,\n \t\t\t struct packed_git *p,\n \t\t\t uint32_t cur_object,\n@@ -438,7 +481,8 @@ static void fill_pack_entry(uint32_t pack_int_id,\n * Copy only the de-duplicated entries (selected by most-recent modified time\n * of a packfile containing the object).\n */\n-static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n+static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n+\t\t\t\t\t\t struct packed_git **p,\n \t\t\t\t\t\t uint32_t *perm,\n \t\t\t\t\t\t uint32_t nr_packs,\n \t\t\t\t\t\t uint32_t *nr_objects)\n@@ -447,8 +491,9 @@ static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n \tstruct pack_midx_entry *entries_by_fanout = NULL;\n \tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\tuint32_t start_pack = m ? m->num_packs : 0;\n \n-\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\ttotal_objects += p[cur_pack]->num_objects;\n \t}\n \n@@ -466,7 +511,23 @@ static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tuint32_t nr_fanout = 0;\n \n-\t\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\tif (m) {\n+\t\t\tuint32_t start = 0, end;\n+\n+\t\t\tif (cur_fanout)\n+\t\t\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n+\t\t\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n+\n+\t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n+\t\t\t\tnth_midxed_pack_midx_entry(m, perm,\n+\t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n+\t\t\t\t\t\t\t cur_object);\n+\t\t\t\tnr_fanout++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\t\tuint32_t start = 0, end;\n \n \t\t\tif (cur_fanout)\n@@ -661,7 +722,7 @@ int write_midx_file(const char *object_dir)\n \tstruct hashfile *f;\n \tstruct lock_file lk;\n \tstruct pack_list packs;\n-\tuint32_t *pack_perm;\n+\tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n@@ -676,24 +737,42 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n+\tpacks.m = load_multi_pack_index(object_dir);\n+\n \tpacks.nr = 0;\n-\tpacks.alloc_list = 16;\n-\tpacks.alloc_names = 16;\n+\tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n+\tpacks.alloc_names = packs.alloc_list;\n \tpacks.list = NULL;\n \tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n \tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n+\tif (packs.m) {\n+\t\tfor (i = 0; i < packs.m->num_packs; i++) {\n+\t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n+\t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n+\n+\t\t\tpacks.list[packs.nr] = NULL;\n+\t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n+\t\t\tpacks.nr++;\n+\t\t}\n+\t}\n+\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n+\tif (packs.m && packs.nr == packs.m->num_packs)\n+\t\tgoto cleanup;\n+\n \tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n-\t\t\t\t\t(packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n \n-\tALLOC_ARRAY(pack_perm, packs.nr);\n+\tALLOC_ARRAY(pack_perm, packs.alloc_list);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n-\tentries = get_sorted_entries(packs.list, pack_perm, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n+\n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n@@ -796,14 +875,18 @@ int write_midx_file(const char *object_dir)\n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n \n+cleanup:\n \tfor (i = 0; i < packs.nr; i++) {\n-\t\tclose_pack(packs.list[i]);\n-\t\tFREE_AND_NULL(packs.list[i]);\n+\t\tif (packs.list[i]) {\n+\t\t\tclose_pack(packs.list[i]);\n+\t\t\tFREE_AND_NULL(packs.list[i]);\n+\t\t}\n \t\tFREE_AND_NULL(packs.names[i]);\n \t}\n \n \tFREE_AND_NULL(packs.list);\n \tFREE_AND_NULL(packs.names);\n \tFREE_AND_NULL(entries);\n+\tFREE_AND_NULL(pack_perm);\n \treturn 0;\n }\ndiff --git a/midx.h b/midx.h\nindex f7c2ec7893..5faffb7bc6 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -11,6 +11,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n);\n int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350829","messageId":"20180625143434.89044-23-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 22/24] midx: prevent duplicate packfile loads","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:32Z","receivedAt":"2018-06-25T14:35:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index, when present, tracks the existence of objects and\ntheir offsets within a list of packfiles. This allows us to use the\nmulti-pack-index for object lookups, abbreviations, and object counts.\n\nWhen the multi-pack-index tracks a packfile, then we do not need to add\nthat packfile to the packed_git linked list or the MRU list.\n\nWe still need to load the packfiles that are not tracked by the\nmulti-pack-index.\n\nSigned-off-by: Derrick Stolee \n---\n packfile.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/packfile.c b/packfile.c\nindex e72e8a685d..f2b8d6f8a7 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -796,6 +796,7 @@ struct prepare_pack_data\n \tstruct repository *r;\n \tstruct string_list *garbage;\n \tint local;\n+\tstruct multi_pack_index *m;\n };\n \n static void prepare_pack(const char *full_name, size_t full_name_len, const char *file_name, void *_data)\n@@ -805,6 +806,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len, const char\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\")) {\n+\t\tif (data->m && midx_contains_pack(data->m, file_name))\n+\t\t\treturn;\n \t\t/* Don't reopen a pack we already have. */\n \t\tfor (p = data->r->objects->packed_git; p; p = p->next) {\n \t\t\tsize_t len;\n@@ -841,6 +844,12 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n \tstruct prepare_pack_data data;\n \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \n+\tdata.m = r->objects->multi_pack_index;\n+\n+\t/* look for the multi-pack-index for this object directory */\n+\twhile (data.m && strcmp(data.m->object_dir, objdir))\n+\t\tdata.m = data.m->next;\n+\n \tdata.r = r;\n \tdata.garbage = &garbage;\n \tdata.local = local;\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350830","messageId":"20180625143434.89044-22-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 21/24] midx: use midx in approximate_object_count","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:31Z","receivedAt":"2018-06-25T14:35:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n packfile.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 20b743da91..e72e8a685d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -863,11 +863,14 @@ unsigned long approximate_object_count(void)\n {\n \tif (!the_repository->objects->approximate_object_count_valid) {\n \t\tunsigned long count;\n+\t\tstruct multi_pack_index *m;\n \t\tstruct packed_git *p;\n \n \t\tprepare_packed_git(the_repository);\n \t\tcount = 0;\n-\t\tfor (p = the_repository->objects->packed_git; p; p = p->next) {\n+\t\tfor (m = get_multi_pack_index(the_repository); m; m = m->next)\n+\t\t\tcount += m->num_objects;\n+\t\tfor (p = get_packed_git(the_repository); p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n \t\t\tcount += p->num_objects;\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"350831","messageId":"20180625143434.89044-8-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v2 07/24] multi-pack-index: expand test data","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:34:17Z","receivedAt":"2018-06-25T14:36:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As we build the multi-pack-index file format, we want to test the format\non real repoasitories. Add tests to t5319-multi-pack-index.sh that\ncreate repository data including multiple packfiles with both version 1\nand version 2 formats.\n\nThe current 'git multi-pack-index write' command will always write the\nsame file with no \"real\" data. This will be expanded in future commits,\nalong with the test expectations.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 99 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 99 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 0372704c96..d533fd0dbc 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -13,9 +13,108 @@ midx_read_expect() {\n }\n \n test_expect_success 'write midx with no packs' '\n+\ttest_when_finished rm pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n \ttest_path_is_file pack/multi-pack-index &&\n \tmidx_read_expect\n '\n \n+test_expect_success 'create objects' '\n+\tfor i in `test_seq 1 5`\n+\tdo\n+\t\tiii=$(printf '%03i' $i)\n+\t\ttest-tool genrandom \"bar\" 200 > wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\ti=$(expr $i + 1) || return 1\n+\tdone &&\n+\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree obj-list &&\n+\tgit update-ref HEAD $commit\n+'\n+\n+test_expect_success 'write midx with one v1 pack' '\n+\tpack=$(git pack-objects --index-version=1 pack/test wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\ti=$(expr $i + 1) || return 1\n+\tdone &&\n+\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree -p HEADobj-list2 &&\n+\tgit update-ref HEAD $commit\n+'\n+\n+test_expect_success 'write midx with two packs' '\n+\tgit pack-objects --index-version=1 pack/test-2 wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\t\tgit update-index --add file_101 &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcommit=$(git commit-tree $tree -p HEADobj-list &&\n+\t\tgit update-ref HEAD $commit &&\n+\t\tgit pack-objects --index-version=2 test-pack writes:\n\n> This new 'git multi-pack-index' builtin will be the plumbing access\n> for writing, reading, and checking multi-pack-index files. The\n> initial implementation is a no-op.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> .gitignore | 3 +-\n> Documentation/git-multi-pack-index.txt | 36 ++++++++++++++++++++++++\n> Makefile | 1 +\n> builtin.h | 1 +\n> builtin/multi-pack-index.c | 38 ++++++++++++++++++++++++++\n> command-list.txt | 1 +\n> git.c | 1 +\n> 7 files changed, 80 insertions(+), 1 deletion(-)\n> create mode 100644 Documentation/git-multi-pack-index.txt\n> create mode 100644 builtin/multi-pack-index.c\n>\n> diff --git a/.gitignore b/.gitignore\n> index 388cc4beee..25633bc515 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -99,8 +99,9 @@\n> /git-mergetool--lib\n> /git-mktag\n> /git-mktree\n> -/git-name-rev\n> +/git-multi-pack-index\n> /git-mv\n> +/git-name-rev\n\nNice attention to the detail (even though the patch as an\nincremental change gets distracting, the result is better\nand future changes to the file will read cleaner).\n\n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> new file mode 100644\n> index 0000000000..9877f9c441\n> --- /dev/null\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -0,0 +1,36 @@\n> +git-multi-pack-index(1)\n> +======================\n> +\n> +NAME\n> +----\n> +git-multi-pack-index - Write and verify multi-pack-indexes\n> +\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git multi-pack-index' [--object-dir ]\n> +\n> +DESCRIPTION\n> +-----------\n> +Write or verify a multi-pack-index (MIDX) file.\n> +\n> +OPTIONS\n> +-------\n> +\n> +--object-dir ::\n> +\tUse given directory for the location of Git objects. We check\n> +\t/packs/multi-pack-index for the current MIDX file, and\n> +\t/packs for the pack-files to index.\n\nDo we want to `quote` these constant strings?\n\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> new file mode 100644\n> index 0000000000..f101873525\n> --- /dev/null\n> +++ b/builtin/multi-pack-index.c\n> @@ -0,0 +1,38 @@\n> +#include \"builtin.h\"\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"parse-options.h\"\n> +\n> +static char const * const builtin_multi_pack_index_usage[] ={\n\nERROR: spaces required around that '=' (ctx:WxV)\n#112: FILE: builtin/multi-pack-index.c:6:\n+static char const * const builtin_multi_pack_index_usage[] ={\n ^\n\n"},{"id":"350893","messageId":"xmqqmuvioeqq.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"20180625143434.89044-6-dstolee@microsoft.com","subject":"Re: [PATCH v2 05/24] midx: write header information to lockfile","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-25T19:19:57Z","receivedAt":"2018-06-25T19:20:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n> +#define MIDX_VERSION 1\n> +#define MIDX_HASH_VERSION 1\n> +#define MIDX_HEADER_SIZE 12\n> +\n> +static char *get_midx_filename(const char *object_dir)\n> +{\n> +\treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n> +}\n> +\n> +static size_t write_midx_header(struct hashfile *f,\n> +\t\t\t\tunsigned char num_chunks,\n> +\t\t\t\tuint32_t num_packs)\n> +{\n> +\tunsigned char byte_values[4];\n> +\thashwrite_be32(f, MIDX_SIGNATURE);\n\nWARNING: Missing a blank line after declarations\n#48: FILE: midx.c:21:\n+ unsigned char byte_values[4];\n+ hashwrite_be32(f, MIDX_SIGNATURE);\n\n> +\tbyte_values[0] = MIDX_VERSION;\n> +\tbyte_values[1] = MIDX_HASH_VERSION;\n> +\tbyte_values[2] = num_chunks;\n> +\tbyte_values[3] = 0; /* unused */\n> +\thashwrite(f, byte_values, sizeof(byte_values));\n> +\thashwrite_be32(f, num_packs);\n> +\n> +\treturn MIDX_HEADER_SIZE;\n> +}\n> +\n> int write_midx_file(const char *object_dir)\n> {\n> +\tunsigned char num_chunks = 0;\n> +\tchar *midx_name;\n> +\tstruct hashfile *f;\n> +\tstruct lock_file lk;\n> +\n> +\tmidx_name = get_midx_filename(object_dir);\n> +\tif (safe_create_leading_directories(midx_name)) {\n> +\t\tUNLEAK(midx_name);\n> +\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n> +\t\t\t midx_name);\n> +\t}\n> +\n> +\thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n> +\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n> +\tFREE_AND_NULL(midx_name);\n\nI am not sure why people prefer FREE_AND_NULL over free() for things\nlike this. It is on stack; it's not like this is a static variable\nvisible after this function returns or anything like that.\n\n> +\twrite_midx_header(f, num_chunks, 0);\n> +\n> +\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n> +\tcommit_lock_file(&lk);\n> +\n> \treturn 0;\n> }\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index ec3ddbe79c..8622a7cdce 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -4,7 +4,8 @@ test_description='multi-pack-indexes'\n> . ./test-lib.sh\n> \n> test_expect_success 'write midx with no packs' '\n> -\tgit multi-pack-index --object-dir=. write\n> +\tgit multi-pack-index --object-dir=. write &&\n> +\ttest_path_is_file pack/multi-pack-index\n> '\n> \n> test_done\n"},{"id":"350895","messageId":"xmqqfu1aodw3.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"20180625143434.89044-7-dstolee@microsoft.com","subject":"Re: [PATCH v2 06/24] multi-pack-index: load into memory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-25T19:38:20Z","receivedAt":"2018-06-25T19:38:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +#define MIDX_HASH_LEN 20\n> +#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n> \n> static char *get_midx_filename(const char *object_dir)\n> {\n> \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n> }\n> \n> +struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n> +{\n> +\tstruct multi_pack_index *m = NULL;\n> +\tint fd;\n> +\tstruct stat st;\n> +\tsize_t midx_size;\n> +\tvoid *midx_map = NULL;\n> +\tuint32_t hash_version;\n> +\tchar *midx_name = get_midx_filename(object_dir);\n> +\n> +\tfd = git_open(midx_name);\n> +\n> +\tif (fd < 0) {\n> +\t\terror_errno(_(\"failed to read %s\"), midx_name);\n> +\t\tFREE_AND_NULL(midx_name);\n> +\t\treturn NULL;\n> +\t}\n> +\tif (fstat(fd, &st)) {\n> +\t\terror_errno(_(\"failed to read %s\"), midx_name);\n> +\t\tFREE_AND_NULL(midx_name);\n> +\t\tclose(fd);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\tmidx_size = xsize_t(st.st_size);\n> +\n> +\tif (midx_size < MIDX_MIN_SIZE) {\n> +\t\tclose(fd);\n> +\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name);\n> +\t\tgoto cleanup_fail;\n> +\t}\n> +\n> +\tFREE_AND_NULL(midx_name);\n\nError handling in the above part looks a bit inconsistent. I first\nthought that the earlier ones manually clean up and leave because\njumping to cleanup_fail would need a successfully opened fd and\nsuccessfully mmapped midx_map, but the above \"goto\" forces\ncleanup_fail: to munmap NULL and close an already closed fd.\n\nI wonder if it is simpler to do\n\n\tcleanup_fail:\n\t\t/* no need to check for NULL when freeing */\n\t\tfree(m);\n\t\tfree(midx_name);\n\t\tif (midx_map)\n\t\t\tmunmap(midx_map, midx_size);\n\t\tif (0 <= fd)\n\t\t\tclose(fd);\n\t\treturn NULL;\n\nand have all of the above error codepath to jump there.\n\n> +\tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n> +\n> +\tm = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n> +\tstrcpy(m->object_dir, object_dir);\n> +\tm->data = midx_map;\n> +\n> +\tm->signature = get_be32(m->data);\n> +\tif (m->signature != MIDX_SIGNATURE) {\n> +\t\terror(_(\"multi-pack-index signature 0x%08x does not match signature 0x%08x\"),\n> +\t\t m->signature, MIDX_SIGNATURE);\n> +\t\tgoto cleanup_fail;\n> +\t}\n> +\n> +\tm->version = m->data[4];\n> +\tif (m->version != MIDX_VERSION) {\n> +\t\terror(_(\"multi-pack-index version %d not recognized\"),\n> +\t\t m->version);\n> +\t\tgoto cleanup_fail;\n> +\t}\n> +\n> +\thash_version = m->data[5];\n\nIs there a good existing example to show a better way to avoid these\nhard-coded constants that describe/define the file format?\n\n> +\tif (hash_version != MIDX_HASH_VERSION) {\n> +\t\terror(_(\"hash version %u does not match\"), hash_version);\n> +\t\tgoto cleanup_fail;\n> +\t}\n> +\tm->hash_len = MIDX_HASH_LEN;\n> +\n> +\tm->num_chunks = *(m->data + 6);\n\nBy the way, this mixture of m->data[4] and *(m->data + 6) is even\nworse. You could do get_be32(&8[m->data]) if you want to irritate\nreaders even more ;-)\n\n> +\tm->num_packs = get_be32(m->data + 8);\n> +\n> +\treturn m;\n> +\n> +cleanup_fail:\n> +\tFREE_AND_NULL(m);\n> +\tFREE_AND_NULL(midx_name);\n> +\tmunmap(midx_map, midx_size);\n> +\tclose(fd);\n> +\treturn NULL;\n> +}\n> +\n\n\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index 8622a7cdce..0372704c96 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -3,9 +3,19 @@\n> test_description='multi-pack-indexes'\n> . ./test-lib.sh\n> \n> +midx_read_expect() {\n\n\"midx_read_expect () {\", i.e. SP on both sides of (), please.\n\n> +\tcat >expect <<- EOF\n\n\"<<-\\EOF\", i.e. make it easy for readers to spot that there is no\nfunny substitutions happening in the here-doc body.\n\n\n> +\theader: 4d494458 1 0 0\n> +\tobject_dir: .\n> +\tEOF\n> +\ttest-tool read-midx . >actual &&\n> +\ttest_cmp expect actual\n> +}\n> +\n> test_expect_success 'write midx with no packs' '\n> \tgit multi-pack-index --object-dir=. write &&\n> -\ttest_path_is_file pack/multi-pack-index\n> +\ttest_path_is_file pack/multi-pack-index &&\n> +\tmidx_read_expect\n> '\n> \n> test_done\n"},{"id":"350896","messageId":"xmqqbmbyodjg.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"20180625143434.89044-8-dstolee@microsoft.com","subject":"Re: [PATCH v2 07/24] multi-pack-index: expand test data","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-25T19:45:55Z","receivedAt":"2018-06-25T19:46:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> test_expect_success 'write midx with no packs' '\n> +\ttest_when_finished rm pack/multi-pack-index &&\n\nIt is generally a good idea to give \"-f\" to \"rm\" used in\ntest_when_finished. The main command sequence may have failed\nbefore it has a chance to create that file; even though an error\nwill be ignored by the when_finished handler, it is a good code\nhygiene to mark expected condition (e.g. the file to be removed may\nnot exist at this point) to signal to future readers that the author\nknew what s/he was writing.\n\n> \tgit multi-pack-index --object-dir=. write &&\n> \ttest_path_is_file pack/multi-pack-index &&\n> \tmidx_read_expect\n> '\n> \n> +test_expect_success 'create objects' '\n> +\tfor i in `test_seq 1 5`\n\nPlease write it as \"$(test_seq 1 5)\"\n\n> +\tdo\n> +\t\tiii=$(printf '%03i' $i)\n> +\t\ttest-tool genrandom \"bar\" 200 > wide_delta_$iii &&\n\n\ttest-tool genrandom \"bar\" 200 >\"wide_delta_$iii\" &&\n\ni.e. no SP between the redirection operator and the target file\nname. Also dq around target file name that depends on variable\nsubstitution to tell bash that we know what we are doing (some\nvintage of bash will throw warning at us unless we do so).\n\n"},{"id":"350897","messageId":"xmqq7emmod0y.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"20180625143434.89044-9-dstolee@microsoft.com","subject":"Re: [PATCH v2 08/24] packfile: generalize pack directory list","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-25T19:57:01Z","receivedAt":"2018-06-25T19:57:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +struct prepare_pack_data\n> +{\n\nERROR: open brace '{' following struct go on the same line\n#88: FILE: packfile.c:777:\n+struct prepare_pack_data\n+{\n\n> +\tstruct repository *r;\n> +\tstruct string_list *garbage;\n> +\tint local;\n> +};\n> +\n> +static void prepare_pack(const char *full_name, size_t full_name_len, const char *file_name, void *_data)\n> +{\n> +\tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n> +\tstruct packed_git *p;\n> +\tsize_t base_len = full_name_len;\n> +\n> +\tif (strip_suffix_mem(full_name, &base_len, \".idx\")) {\n> +\t\t/* Don't reopen a pack we already have. */\n> +\t\tfor (p = data->r->objects->packed_git; p; p = p->next) {\n> +\t\t\tsize_t len;\n> +\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n> +\t\t\t len == base_len &&\n> +\t\t\t !memcmp(p->pack_name, full_name, len))\n> +\t\t\t\tbreak;\n> +\t\t}\n> +\n> +\t\tif (p == NULL &&\n> +\t\t /*\n> +\t\t * See if it really is a valid .idx file with\n> +\t\t * corresponding .pack file that we can map.\n> +\t\t */\n> +\t\t (p = add_packed_git(full_name, full_name_len, data->local)) != NULL)\n> +\t\t\tinstall_packed_git(data->r, p);\n> +\t}\n\nThis is merely a moved code and the issue was inherited from the\noriginal, but can we make it easier to read and at the same time\nremove that assignment inside if() condition (which generally makes\nthe code harder to read)? The most naïve\n\n\tif (!p) {\n\t\tp = add_packed_git(full_name, full_name_len, data->local);\n\t\tif (p)\n\t\t\tinstall_packed_git(data->r, p);\n\t}\n\nisn't all that bad, but there may be even better ways.\n\n> +\tif (!report_garbage)\n> +\t return;\n\nThis \"return;\" is indented with a run of SP not with HT?\n\n"},{"id":"350912","messageId":"CAPig+cRWwTAFRJOwQOi4USk5UZke=2sz_JVDh3+XRKCcBGD5ow@mail.gmail.com","threadId":"48668","inReplyTo":"20180625143434.89044-12-dstolee@microsoft.com","subject":"Re: [PATCH v2 11/24] midx: read pack names into array","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-06-25T23:52:32Z","receivedAt":"2018-06-25T23:52:48Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Jun 25, 2018 at 10:35 AM Derrick Stolee wrote:\n> diff --git a/midx.c b/midx.c\n> @@ -210,6 +227,20 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n> +static size_t write_midx_pack_lookup(struct hashfile *f,\n> + char **pack_names,\n> + uint32_t nr_packs)\n> +{\n> + uint32_t i, cur_len = 0;\n> +\n> + for (i = 0; i < nr_packs; i++) {\n> + hashwrite_be32(f, cur_len);\n> + cur_len += strlen(pack_names[i]) + 1;\n> + }\n> +\n> + return sizeof(uint32_t) * (size_t)nr_packs;\n> +}\n\nThis static function is never used, thus breaks the build with DEVELOPER=1:\n\n midx.c:567:15: error: ‘write_midx_pack_lookup’ defined but not used\n [-Werror=unused-function]\n cc1: all warnings being treated as errors\n"},{"id":"351704","messageId":"2bb2a12f-f22d-990b-636c-73b2b805d12d@gmail.com","threadId":"48668","inReplyTo":"xmqqfu1aodw3.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 06/24] multi-pack-index: load into memory","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-05T14:19:58Z","receivedAt":"2018-07-05T14:20:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/25/2018 3:38 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>> +\tcat >expect <<- EOF\n> \"<<-\\EOF\", i.e. make it easy for readers to spot that there is no\n> funny substitutions happening in the here-doc body.\n\nWhile I don't use substitutions in this patch, I do use them in later \npatches. Here is the final version of this method:\n\nmidx_read_expect () {\n         NUM_PACKS=$1\n         NUM_OBJECTS=$2\n         NUM_CHUNKS=$3\n         OBJECT_DIR=$4\n         EXTRA_CHUNKS=\"$5\"\n         cat >expect <<-\\EOF\n         header: 4d494458 1 $NUM_CHUNKS $NUM_PACKS\n         chunks: pack_names oid_fanout oid_lookup \nobject_offsets$EXTRA_CHUNKS\n         num_objects: $NUM_OBJECTS\n         packs:\n         EOF\n         if [ $NUM_PACKS -ge 1 ]\n         then\n                 ls $OBJECT_DIR/pack/ | grep idx | sort >> expect\n         fi\n         printf \"object_dir: $OBJECT_DIR\\n\" >>expect &&\n         test-tool read-midx $OBJECT_DIR >actual &&\n         test_cmp expect actual\n}\n\nUsing <<-\\EOF causes these substitutions to fail. Is there a different \nway I should construct this method?\n\nThanks,\n-Stolee\n"},{"id":"351711","messageId":"CAPig+cTVD5OpVnxAqQnO_CkzFRRkZ3YWsRC4ZwUmAELSZzcDKw@mail.gmail.com","threadId":"48668","inReplyTo":"2bb2a12f-f22d-990b-636c-73b2b805d12d@gmail.com","subject":"Re: [PATCH v2 06/24] multi-pack-index: load into memory","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-05T18:58:22Z","receivedAt":"2018-07-05T18:58:37Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 10:20 AM Derrick Stolee wrote:\n> On 6/25/2018 3:38 PM, Junio C Hamano wrote:\n> While I don't use substitutions in this patch, I do use them in later\n> patches. Here is the final version of this method:\n>\n> midx_read_expect () {\n> NUM_PACKS=$1\n> NUM_OBJECTS=$2\n> NUM_CHUNKS=$3\n> EXTRA_CHUNKS=\"$5\"\n> cat >expect <<-\\EOF\n> header: 4d494458 1 $NUM_CHUNKS $NUM_PACKS\n> chunks: pack_names oid_fanout oid_lookup\n> object_offsets$EXTRA_CHUNKS\n> num_objects: $NUM_OBJECTS\n> packs:\n> EOF\n>\n> Using <<-\\EOF causes these substitutions to fail. Is there a different\n> way I should construct this method?\n\nWhen you need to interpolate variables into the here-doc, use <<-EOF;\nwhen you don't, use <<-\\EOF.\n"},{"id":"351712","messageId":"41812292-3f1e-adaa-3faa-33f5e8f8e47a@gmail.com","threadId":"48668","inReplyTo":"xmqqmuvioeqq.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 05/24] midx: write header information to lockfile","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-05T19:13:07Z","receivedAt":"2018-07-05T19:13:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/25/2018 3:19 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> +#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n>> +#define MIDX_VERSION 1\n>> +#define MIDX_HASH_VERSION 1\n>> +#define MIDX_HEADER_SIZE 12\n>> +\n>> +static char *get_midx_filename(const char *object_dir)\n>> +{\n>> +\treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n>> +}\n>> +\n>> +static size_t write_midx_header(struct hashfile *f,\n>> +\t\t\t\tunsigned char num_chunks,\n>> +\t\t\t\tuint32_t num_packs)\n>> +{\n>> +\tunsigned char byte_values[4];\n>> +\thashwrite_be32(f, MIDX_SIGNATURE);\n> WARNING: Missing a blank line after declarations\n> #48: FILE: midx.c:21:\n> + unsigned char byte_values[4];\n> + hashwrite_be32(f, MIDX_SIGNATURE);\n>\n>> +\tbyte_values[0] = MIDX_VERSION;\n>> +\tbyte_values[1] = MIDX_HASH_VERSION;\n>> +\tbyte_values[2] = num_chunks;\n>> +\tbyte_values[3] = 0; /* unused */\n>> +\thashwrite(f, byte_values, sizeof(byte_values));\n>> +\thashwrite_be32(f, num_packs);\n>> +\n>> +\treturn MIDX_HEADER_SIZE;\n>> +}\n>> +\n>> int write_midx_file(const char *object_dir)\n>> {\n>> +\tunsigned char num_chunks = 0;\n>> +\tchar *midx_name;\n>> +\tstruct hashfile *f;\n>> +\tstruct lock_file lk;\n>> +\n>> +\tmidx_name = get_midx_filename(object_dir);\n>> +\tif (safe_create_leading_directories(midx_name)) {\n>> +\t\tUNLEAK(midx_name);\n>> +\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n>> +\t\t\t midx_name);\n>> +\t}\n>> +\n>> +\thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n>> +\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n>> +\tFREE_AND_NULL(midx_name);\n> I am not sure why people prefer FREE_AND_NULL over free() for things\n> like this. It is on stack; it's not like this is a static variable\n> visible after this function returns or anything like that.\n\nI default to FREE_AND_NULL(X) because a later change may introduce logic \nto use X later in the same code block. In this case, we add a 'cleanup:' \nat the end which would fail in a success case if we don't set midx_name \nto NULL here.\n\nI think there are some other FREE_AND_NULLs that are currently at the \nend of a code block, so I'll work to clean those up.\n\n>\n>> +\twrite_midx_header(f, num_chunks, 0);\n>> +\n>> +\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n>> +\tcommit_lock_file(&lk);\n>> +\n>> \treturn 0;\n>> }\n>> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n>> index ec3ddbe79c..8622a7cdce 100755\n>> --- a/t/t5319-multi-pack-index.sh\n>> +++ b/t/t5319-multi-pack-index.sh\n>> @@ -4,7 +4,8 @@ test_description='multi-pack-indexes'\n>> . ./test-lib.sh\n>> \n>> test_expect_success 'write midx with no packs' '\n>> -\tgit multi-pack-index --object-dir=. write\n>> +\tgit multi-pack-index --object-dir=. write &&\n>> +\ttest_path_is_file pack/multi-pack-index\n>> '\n>> \n>> test_done\n\n"},{"id":"351717","messageId":"20180706005321.124643-1-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180625143434.89044-1-dstolee@microsoft.com","subject":"[PATCH v3 00/24] Multi-pack-index (MIDX)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:52:57Z","receivedAt":"2018-07-06T00:53:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks for the feedback on v2. I cleaned up the patch to respond to all\nthat feedback. I'll include the version diff in a follow-up email.\n\nYou can see the CI builds for Linux, Mac, and Windows linked from the\nGitHub pull request [1].\n\nBiggest changes in this version:\n\n* Deleted the extra static method.\n\n* Use $(...) instead of `...` in test script\n\n* Fixed spaces in tabbing (hopefully I caught them all)\n\n* Cleaned up some inter-patch diff issues\n\n* `...` quotes in git-multi-pack-index.txt\n\n* Replace FREE_AND_NULL() with free() when obvious that a variable is\n going out of scope\n\n* Due to how Windows handles open file handles when replacing a\n lockfile, be sure to close the midx before writing a new one.\n\nI do still want to revisit the \"order by fanout\" vs \"merge sort\"\ndiscussion that we had on v1, but this series is very big already. I'd\nlike to come back to that as a follow-up.\n\nThanks,\n-Stolee\n\n[1] https://github.com/gitgitgadget/git/pull/5\n\nDerrick Stolee (24):\n multi-pack-index: add design document\n multi-pack-index: add format details\n multi-pack-index: add builtin\n multi-pack-index: add 'write' verb\n midx: write header information to lockfile\n multi-pack-index: load into memory\n multi-pack-index: expand test data\n packfile: generalize pack directory list\n multi-pack-index: read packfile list\n multi-pack-index: write pack names in chunk\n midx: read pack names into array\n midx: sort and deduplicate objects from packfiles\n midx: write object ids in a chunk\n midx: write object id fanout chunk\n midx: write object offsets\n config: create core.multiPackIndex setting\n midx: prepare midxed_git struct\n midx: read objects from multi-pack-index\n midx: use midx in abbreviation calculations\n midx: use existing midx when writing new one\n midx: use midx in approximate_object_count\n midx: prevent duplicate packfile loads\n packfile: skip loading index if in multi-pack-index\n midx: clear midx on repack\n\n .gitignore | 3 +-\n Documentation/config.txt | 4 +\n Documentation/git-multi-pack-index.txt | 56 ++\n Documentation/technical/multi-pack-index.txt | 109 +++\n Documentation/technical/pack-format.txt | 77 ++\n Makefile | 3 +\n builtin.h | 1 +\n builtin/multi-pack-index.c | 46 +\n builtin/repack.c | 8 +\n cache.h | 1 +\n command-list.txt | 1 +\n config.c | 5 +\n environment.c | 1 +\n git.c | 1 +\n midx.c | 896 +++++++++++++++++++\n midx.h | 20 +\n object-store.h | 33 +\n packfile.c | 169 +++-\n packfile.h | 9 +\n sha1-name.c | 70 ++\n t/helper/test-read-midx.c | 54 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5319-multi-pack-index.sh | 191 ++++\n 24 files changed, 1717 insertions(+), 43 deletions(-)\n create mode 100644 Documentation/git-multi-pack-index.txt\n create mode 100644 Documentation/technical/multi-pack-index.txt\n create mode 100644 builtin/multi-pack-index.c\n create mode 100644 midx.c\n create mode 100644 midx.h\n create mode 100644 t/helper/test-read-midx.c\n create mode 100755 t/t5319-multi-pack-index.sh\n\n\nbase-commit: 53f9a3e157dbbc901a02ac2c73346d375e24978c\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351718","messageId":"20180706005321.124643-2-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 01/24] multi-pack-index: add design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:52:58Z","receivedAt":"2018-07-06T00:53:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/multi-pack-index.txt | 109 +++++++++++++++++++\n 1 file changed, 109 insertions(+)\n create mode 100644 Documentation/technical/multi-pack-index.txt\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nnew file mode 100644\nindex 0000000000..d7e57639f7\n--- /dev/null\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -0,0 +1,109 @@\n+Multi-Pack-Index (MIDX) Design Notes\n+====================================\n+\n+The Git object directory contains a 'pack' directory containing\n+packfiles (with suffix \".pack\") and pack-indexes (with suffix\n+\".idx\"). The pack-indexes provide a way to lookup objects and\n+navigate to their offset within the pack, but these must come\n+in pairs with the packfiles. This pairing depends on the file\n+names, as the pack-index differs only in suffix with its pack-\n+file. While the pack-indexes provide fast lookup per packfile,\n+this performance degrades as the number of packfiles increases,\n+because abbreviations need to inspect every packfile and we are\n+more likely to have a miss on our most-recently-used packfile.\n+For some large repositories, repacking into a single packfile\n+is not feasible due to storage space or excessive repack times.\n+\n+The multi-pack-index (MIDX for short) stores a list of objects\n+and their offsets into multiple packfiles. It contains:\n+\n+- A list of packfile names.\n+- A sorted list of object IDs.\n+- A list of metadata for the ith object ID including:\n+ - A value j referring to the jth packfile.\n+ - An offset within the jth packfile for the object.\n+- If large offsets are required, we use another list of large\n+ offsets similar to version 2 pack-indexes.\n+\n+Thus, we can provide O(log N) lookup time for any number\n+of packfiles.\n+\n+Design Details\n+--------------\n+\n+- The MIDX is stored in a file named 'multi-pack-index' in the\n+ .git/objects/pack directory. This could be stored in the pack\n+ directory of an alternate. It refers only to packfiles in that\n+ same directory.\n+\n+- The pack.multiIndex config setting must be on to consume MIDX files.\n+\n+- The file format includes parameters for the object ID hash\n+ function, so a future change of hash algorithm does not require\n+ a change in format.\n+\n+- The MIDX keeps only one record per object ID. If an object appears\n+ in multiple packfiles, then the MIDX selects the copy in the most-\n+ recently modified packfile.\n+\n+- If there exist packfiles in the pack directory not registered in\n+ the MIDX, then those packfiles are loaded into the `packed_git`\n+ list and `packed_git_mru` cache.\n+\n+- The pack-indexes (.idx files) remain in the pack directory so we\n+ can delete the MIDX file, set core.midx to false, or downgrade\n+ without any loss of information.\n+\n+- The MIDX file format uses a chunk-based approach (similar to the\n+ commit-graph file) that allows optional data to be added.\n+\n+Future Work\n+-----------\n+\n+- Add a 'verify' subcommand to the 'git midx' builtin to verify the\n+ contents of the multi-pack-index file match the offsets listed in\n+ the corresponding pack-indexes.\n+\n+- The multi-pack-index allows many packfiles, especially in a context\n+ where repacking is expensive (such as a very large repo), or\n+ unexpected maintenance time is unacceptable (such as a high-demand\n+ build machine). However, the multi-pack-index needs to be rewritten\n+ in full every time. We can extend the format to be incremental, so\n+ writes are fast. By storing a small \"tip\" multi-pack-index that\n+ points to large \"base\" MIDX files, we can keep writes fast while\n+ still reducing the number of binary searches required for object\n+ lookups.\n+\n+- The reachability bitmap is currently paired directly with a single\n+ packfile, using the pack-order as the object order to hopefully\n+ compress the bitmaps well using run-length encoding. This could be\n+ extended to pair a reachability bitmap with a multi-pack-index. If\n+ the multi-pack-index is extended to store a \"stable object order\"\n+ (a function Order(hash) = integer that is constant for a given hash,\n+ even as the multi-pack-index is updated) then a reachability bitmap\n+ could point to a multi-pack-index and be updated independently.\n+\n+- Packfiles can be marked as \"special\" using empty files that share\n+ the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n+ We can add an optional chunk of data to the multi-pack-index that\n+ records flags of information about the packfiles. This allows new\n+ states, such as 'repacked' or 'redeltified', that can help with\n+ pack maintenance in a multi-pack environment. It may also be\n+ helpful to organize packfiles by object type (commit, tree, blob,\n+ etc.) and use this metadata to help that maintenance.\n+\n+- The partial clone feature records special \"promisor\" packs that\n+ may point to objects that are not stored locally, but available\n+ on request to a server. The multi-pack-index does not currently\n+ track these promisor packs.\n+\n+Related Links\n+-------------\n+[0] https://bugs.chromium.org/p/git/issues/detail?id=6\n+ Chromium work item for: Multi-Pack Index (MIDX)\n+\n+[1] https://public-inbox.org/git/20180107181459.222909-1-dstolee@microsoft.com/\n+ An earlier RFC for the multi-pack-index feature\n+\n+[2] https://public-inbox.org/git/alpine.DEB.2.20.1803091557510.23109@alexmv-linux/\n+ Git Merge 2018 Contributor's summit notes (includes discussion of MIDX)\n\nbase-commit: 53f9a3e157dbbc901a02ac2c73346d375e24978c\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351719","messageId":"20180706005321.124643-3-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 02/24] multi-pack-index: add format details","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:52:59Z","receivedAt":"2018-07-06T00:53:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index feature generalizes the existing pack-index\nfeature by indexing objects across multiple pack-files.\n\nDescribe the basic file format, using a 12-byte header followed by\na lookup table for a list of \"chunks\" which will be described later.\nThe file ends with a footer containing a checksum using the hash\nalgorithm.\n\nThe header allows later versions to create breaking changes by\nadvancing the version number. We can also change the hash algorithm\nusing a different version value.\n\nWe will add the individual chunk format information as we introduce\nthe code that writes that information.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 49 +++++++++++++++++++++++++\n 1 file changed, 49 insertions(+)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 70a99fd142..e060e693f4 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -252,3 +252,52 @@ Pack file entry: <+\n corresponding packfile.\n \n 20-byte SHA-1-checksum of all of the above.\n+\n+== multi-pack-index (MIDX) files have the following format:\n+\n+The multi-pack-index files refer to multiple pack-files and loose objects.\n+\n+In order to allow extensions that add extra data to the MIDX, we organize\n+the body into \"chunks\" and provide a lookup table at the beginning of the\n+body. The header includes certain length values, such as the number of packs,\n+the number of base MIDX files, hash lengths and types.\n+\n+All 4-byte numbers are in network order.\n+\n+HEADER:\n+\n+\t4-byte signature:\n+\t The signature is: {'M', 'I', 'D', 'X'}\n+\n+\t1-byte version number:\n+\t Git only writes or recognizes version 1.\n+\n+\t1-byte Object Id Version\n+\t Git only writes or recognizes version 1 (SHA1).\n+\n+\t1-byte number of \"chunks\"\n+\n+\t1-byte number of base multi-pack-index files:\n+\t This value is currently always zero.\n+\n+\t4-byte number of pack files\n+\n+CHUNK LOOKUP:\n+\n+\t(C + 1) * 12 bytes providing the chunk offsets:\n+\t First 4 bytes describe chunk id. Value 0 is a terminating label.\n+\t Other 8 bytes provide offset in current file for chunk to start.\n+\t (Chunks are provided in file-order, so you can infer the length\n+\t using the next chunk position if necessary.)\n+\n+\tThe remaining data in the body is described one chunk at a time, and\n+\tthese chunks may be given in any order. Chunks are required unless\n+\totherwise specified.\n+\n+CHUNK DATA:\n+\n+\t(This section intentionally left incomplete.)\n+\n+TRAILER:\n+\n+\t20-byte SHA1-checksum of the above contents.\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351720","messageId":"20180706005321.124643-5-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 04/24] multi-pack-index: add 'write' verb","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:01Z","receivedAt":"2018-07-06T00:53:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of writing multi-pack-indexes, add a\n'git multi-pack-index write' subcommand and send the options to a\nwrite_midx_file() method. Also create a basic test file that tests\nthe 'write' subcommand.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 22 +++++++++++++++++++++-\n Makefile | 1 +\n builtin/multi-pack-index.c | 10 +++++++++-\n midx.c | 7 +++++++\n midx.h | 6 ++++++\n t/t5319-multi-pack-index.sh | 10 ++++++++++\n 6 files changed, 54 insertions(+), 2 deletions(-)\n create mode 100644 midx.c\n create mode 100644 midx.h\n create mode 100755 t/t5319-multi-pack-index.sh\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex a83c0496a6..be97c9372e 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir ]\n+'git multi-pack-index' [--object-dir ] \n \n DESCRIPTION\n -----------\n@@ -23,6 +23,26 @@ OPTIONS\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n \n+write::\n+\tWhen given as the verb, write a new MIDX file to\n+\t`/packs/multi-pack-index`.\n+\n+\n+EXAMPLES\n+--------\n+\n+* Write a MIDX file for the packfiles in the current .git folder.\n++\n+-----------------------------------------------\n+$ git multi-pack-index write\n+-----------------------------------------------\n+\n+* Write a MIDX file for the packfiles in an alternate.\n++\n+-----------------------------------------------\n+$ git multi-pack-index --object-dir write\n+-----------------------------------------------\n+\n \n SEE ALSO\n --------\ndiff --git a/Makefile b/Makefile\nindex 54610875ec..f5636c711d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -890,6 +890,7 @@ LIB_OBJS += merge.o\n LIB_OBJS += merge-blobs.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += mergesort.o\n+LIB_OBJS += midx.o\n LIB_OBJS += name-hash.o\n LIB_OBJS += notes.o\n LIB_OBJS += notes-cache.o\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 4853291477..14b32e1373 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -2,9 +2,10 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"parse-options.h\"\n+#include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir ]\"),\n+\tN_(\"git multi-pack-index [--object-dir ] [write]\"),\n \tNULL\n };\n \n@@ -34,5 +35,12 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \n+\tif (argc == 0)\n+\t\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t\t builtin_multi_pack_index_options);\n+\n+\tif (!strcmp(argv[0], \"write\"))\n+\t\treturn write_midx_file(opts.object_dir);\n+\n \treturn 0;\n }\ndiff --git a/midx.c b/midx.c\nnew file mode 100644\nindex 0000000000..32468db1a2\n--- /dev/null\n+++ b/midx.c\n@@ -0,0 +1,7 @@\n+#include \"cache.h\"\n+#include \"midx.h\"\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nnew file mode 100644\nindex 0000000000..dbdbe9f873\n--- /dev/null\n+++ b/midx.h\n@@ -0,0 +1,6 @@\n+#ifndef __MIDX_H__\n+#define __MIDX_H__\n+\n+int write_midx_file(const char *object_dir);\n+\n+#endif\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nnew file mode 100755\nindex 0000000000..ec3ddbe79c\n--- /dev/null\n+++ b/t/t5319-multi-pack-index.sh\n@@ -0,0 +1,10 @@\n+#!/bin/sh\n+\n+test_description='multi-pack-indexes'\n+. ./test-lib.sh\n+\n+test_expect_success 'write midx with no packs' '\n+\tgit multi-pack-index --object-dir=. write\n+'\n+\n+test_done\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351721","messageId":"20180706005321.124643-4-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 03/24] multi-pack-index: add builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:00Z","receivedAt":"2018-07-06T00:53:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This new 'git multi-pack-index' builtin will be the plumbing access\nfor writing, reading, and checking multi-pack-index files. The\ninitial implementation is a no-op.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 3 +-\n Documentation/git-multi-pack-index.txt | 36 ++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/multi-pack-index.c | 38 ++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n 7 files changed, 80 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/git-multi-pack-index.txt\n create mode 100644 builtin/multi-pack-index.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 388cc4beee..25633bc515 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -99,8 +99,9 @@\n /git-mergetool--lib\n /git-mktag\n /git-mktree\n-/git-name-rev\n+/git-multi-pack-index\n /git-mv\n+/git-name-rev\n /git-notes\n /git-p4\n /git-pack-redundant\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nnew file mode 100644\nindex 0000000000..a83c0496a6\n--- /dev/null\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -0,0 +1,36 @@\n+git-multi-pack-index(1)\n+======================\n+\n+NAME\n+----\n+git-multi-pack-index - Write and verify multi-pack-indexes\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git multi-pack-index' [--object-dir ]\n+\n+DESCRIPTION\n+-----------\n+Write or verify a multi-pack-index (MIDX) file.\n+\n+OPTIONS\n+-------\n+\n+--object-dir ::\n+\tUse given directory for the location of Git objects. We check\n+\t`/packs/multi-pack-index` for the current MIDX file, and\n+\t`/packs` for the pack-files to index.\n+\n+\n+SEE ALSO\n+--------\n+See link:technical/multi-pack-index.html[The Multi-Pack-Index Design\n+Document] and link:technical/pack-format.html[The Multi-Pack-Index\n+Format] for more information on the multi-pack-index feature.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex e4b503d259..54610875ec 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1047,6 +1047,7 @@ BUILTIN_OBJS += builtin/merge-recursive.o\n BUILTIN_OBJS += builtin/merge-tree.o\n BUILTIN_OBJS += builtin/mktag.o\n BUILTIN_OBJS += builtin/mktree.o\n+BUILTIN_OBJS += builtin/multi-pack-index.o\n BUILTIN_OBJS += builtin/mv.o\n BUILTIN_OBJS += builtin/name-rev.o\n BUILTIN_OBJS += builtin/notes.o\ndiff --git a/builtin.h b/builtin.h\nindex 4e0f64723e..70997d7ace 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -191,6 +191,7 @@ extern int cmd_merge_recursive(int argc, const char **argv, const char *prefix);\n extern int cmd_merge_tree(int argc, const char **argv, const char *prefix);\n extern int cmd_mktag(int argc, const char **argv, const char *prefix);\n extern int cmd_mktree(int argc, const char **argv, const char *prefix);\n+extern int cmd_multi_pack_index(int argc, const char **argv, const char *prefix);\n extern int cmd_mv(int argc, const char **argv, const char *prefix);\n extern int cmd_name_rev(int argc, const char **argv, const char *prefix);\n extern int cmd_notes(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nnew file mode 100644\nindex 0000000000..4853291477\n--- /dev/null\n+++ b/builtin/multi-pack-index.c\n@@ -0,0 +1,38 @@\n+#include \"builtin.h\"\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+\n+static char const * const builtin_multi_pack_index_usage[] = {\n+\tN_(\"git multi-pack-index [--object-dir ]\"),\n+\tNULL\n+};\n+\n+static struct opts_multi_pack_index {\n+\tconst char *object_dir;\n+} opts;\n+\n+int cmd_multi_pack_index(int argc, const char **argv,\n+\t\t\t const char *prefix)\n+{\n+\tstatic struct option builtin_multi_pack_index_options[] = {\n+\t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n+\t\t N_(\"The object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t\t builtin_multi_pack_index_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_multi_pack_index_options,\n+\t\t\t builtin_multi_pack_index_usage, 0);\n+\n+\tif (!opts.object_dir)\n+\t\topts.object_dir = get_object_directory();\n+\n+\treturn 0;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex e1c26c1bb7..61071f8fa2 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -123,6 +123,7 @@ git-merge-index plumbingmanipulators\n git-merge-one-file purehelpers\n git-mergetool ancillarymanipulators complete\n git-merge-tree ancillaryinterrogators\n+git-multi-pack-index plumbingmanipulators\n git-mktag plumbingmanipulators\n git-mktree plumbingmanipulators\n git-mv mainporcelain worktree\ndiff --git a/git.c b/git.c\nindex c2f48d53dd..a7509fa5f7 100644\n--- a/git.c\n+++ b/git.c\n@@ -505,6 +505,7 @@ static struct cmd_struct commands[] = {\n \t{ \"merge-tree\", cmd_merge_tree, RUN_SETUP | NO_PARSEOPT },\n \t{ \"mktag\", cmd_mktag, RUN_SETUP | NO_PARSEOPT },\n \t{ \"mktree\", cmd_mktree, RUN_SETUP },\n+\t{ \"multi-pack-index\", cmd_multi_pack_index, RUN_SETUP_GENTLY },\n \t{ \"mv\", cmd_mv, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"name-rev\", cmd_name_rev, RUN_SETUP },\n \t{ \"notes\", cmd_notes, RUN_SETUP },\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351722","messageId":"20180706005321.124643-6-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 05/24] midx: write header information to lockfile","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:02Z","receivedAt":"2018-07-06T00:53:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As we begin writing the multi-pack-index format to disk, start with\nthe basics: the 12-byte header and the 20-byte checksum footer. Start\nwith these basics so we can add the rest of the format in small\nincrements.\n\nAs we implement the format, we will use a technique to check that our\ncomputed offsets within the multi-pack-index file match what we are\nactually writing. Each method that writes to the hashfile will return\nthe number of bytes written, and we will track that those values match\nour expectations.\n\nCurrently, write_midx_header() returns 12, but is not checked. We will\ncheck the return value in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 50 +++++++++++++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh | 3 ++-\n 2 files changed, 52 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 32468db1a2..f85f2d334d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,7 +1,57 @@\n #include \"cache.h\"\n+#include \"csum-file.h\"\n+#include \"lockfile.h\"\n #include \"midx.h\"\n \n+#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n+#define MIDX_VERSION 1\n+#define MIDX_HASH_VERSION 1\n+#define MIDX_HEADER_SIZE 12\n+\n+static char *get_midx_filename(const char *object_dir)\n+{\n+\treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n+}\n+\n+static size_t write_midx_header(struct hashfile *f,\n+\t\t\t\tunsigned char num_chunks,\n+\t\t\t\tuint32_t num_packs)\n+{\n+\tunsigned char byte_values[4];\n+\n+\thashwrite_be32(f, MIDX_SIGNATURE);\n+\tbyte_values[0] = MIDX_VERSION;\n+\tbyte_values[1] = MIDX_HASH_VERSION;\n+\tbyte_values[2] = num_chunks;\n+\tbyte_values[3] = 0; /* unused */\n+\thashwrite(f, byte_values, sizeof(byte_values));\n+\thashwrite_be32(f, num_packs);\n+\n+\treturn MIDX_HEADER_SIZE;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n+\tunsigned char num_chunks = 0;\n+\tchar *midx_name;\n+\tstruct hashfile *f = NULL;\n+\tstruct lock_file lk;\n+\n+\tmidx_name = get_midx_filename(object_dir);\n+\tif (safe_create_leading_directories(midx_name)) {\n+\t\tUNLEAK(midx_name);\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t midx_name);\n+\t}\n+\n+\thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n+\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\tFREE_AND_NULL(midx_name);\n+\n+\twrite_midx_header(f, num_chunks, 0);\n+\n+\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tcommit_lock_file(&lk);\n+\n \treturn 0;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex ec3ddbe79c..8622a7cdce 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -4,7 +4,8 @@ test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n test_expect_success 'write midx with no packs' '\n-\tgit multi-pack-index --object-dir=. write\n+\tgit multi-pack-index --object-dir=. write &&\n+\ttest_path_is_file pack/multi-pack-index\n '\n \n test_done\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351723","messageId":"20180706005321.124643-9-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 08/24] packfile: generalize pack directory list","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:05Z","receivedAt":"2018-07-06T00:53:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of sharing the pack directory listing with the\nmulti-pack-index, generalize prepare_packed_git_one() into\nfor_each_file_in_pack_dir().\n\nSigned-off-by: Derrick Stolee \n---\n packfile.c | 101 +++++++++++++++++++++++++++++++++--------------------\n packfile.h | 6 ++++\n 2 files changed, 69 insertions(+), 38 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 7cd45aa4b2..ee1ab9b804 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -738,13 +738,14 @@ static void report_pack_garbage(struct string_list *list)\n \treport_helper(list, seen_bits, first, list->nr);\n }\n \n-static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tsize_t dirnamelen;\n \tDIR *dir;\n \tstruct dirent *de;\n-\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \n \tstrbuf_addstr(&path, objdir);\n \tstrbuf_addstr(&path, \"/pack\");\n@@ -759,53 +760,77 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n \tstrbuf_addch(&path, '/');\n \tdirnamelen = path.len;\n \twhile ((de = readdir(dir)) != NULL) {\n-\t\tstruct packed_git *p;\n-\t\tsize_t base_len;\n-\n \t\tif (is_dot_or_dotdot(de->d_name))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&path, dirnamelen);\n \t\tstrbuf_addstr(&path, de->d_name);\n \n-\t\tbase_len = path.len;\n-\t\tif (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n-\t\t\t/* Don't reopen a pack we already have. */\n-\t\t\tfor (p = r->objects->packed_git; p;\n-\t\t\t p = p->next) {\n-\t\t\t\tsize_t len;\n-\t\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n-\t\t\t\t len == base_len &&\n-\t\t\t\t !memcmp(p->pack_name, path.buf, len))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tif (p == NULL &&\n-\t\t\t /*\n-\t\t\t * See if it really is a valid .idx file with\n-\t\t\t * corresponding .pack file that we can map.\n-\t\t\t */\n-\t\t\t (p = add_packed_git(path.buf, path.len, local)) != NULL)\n-\t\t\t\tinstall_packed_git(r, p);\n-\t\t}\n-\n-\t\tif (!report_garbage)\n-\t\t\tcontinue;\n-\n-\t\tif (ends_with(de->d_name, \".idx\") ||\n-\t\t ends_with(de->d_name, \".pack\") ||\n-\t\t ends_with(de->d_name, \".bitmap\") ||\n-\t\t ends_with(de->d_name, \".keep\") ||\n-\t\t ends_with(de->d_name, \".promisor\"))\n-\t\t\tstring_list_append(&garbage, path.buf);\n-\t\telse\n-\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path.buf);\n+\t\tfn(path.buf, path.len, de->d_name, data);\n \t}\n+\n \tclosedir(dir);\n-\treport_pack_garbage(&garbage);\n-\tstring_list_clear(&garbage, 0);\n \tstrbuf_release(&path);\n }\n \n+struct prepare_pack_data {\n+\tstruct repository *r;\n+\tstruct string_list *garbage;\n+\tint local;\n+};\n+\n+static void prepare_pack(const char *full_name, size_t full_name_len,\n+\t\t\t const char *file_name, void *_data)\n+{\n+\tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n+\tstruct packed_git *p;\n+\tsize_t base_len = full_name_len;\n+\n+\tif (strip_suffix_mem(full_name, &base_len, \".idx\")) {\n+\t\t/* Don't reopen a pack we already have. */\n+\t\tfor (p = data->r->objects->packed_git; p; p = p->next) {\n+\t\t\tsize_t len;\n+\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n+\t\t\t len == base_len &&\n+\t\t\t !memcmp(p->pack_name, full_name, len))\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!p) {\n+\t\t\tp = add_packed_git(full_name, full_name_len, data->local);\n+\t\t\tif (p)\n+\t\t\t\tinstall_packed_git(data->r, p);\n+\t\t}\n+\t}\n+\n+\tif (!report_garbage)\n+\t\treturn;\n+\n+\tif (ends_with(file_name, \".idx\") ||\n+\t ends_with(file_name, \".pack\") ||\n+\t ends_with(file_name, \".bitmap\") ||\n+\t ends_with(file_name, \".keep\") ||\n+\t ends_with(file_name, \".promisor\"))\n+\t\tstring_list_append(data->garbage, full_name);\n+\telse\n+\t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\n+}\n+\n+static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+{\n+\tstruct prepare_pack_data data;\n+\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n+\n+\tdata.r = r;\n+\tdata.garbage = &garbage;\n+\tdata.local = local;\n+\n+\tfor_each_file_in_pack_dir(objdir, prepare_pack, &data);\n+\n+\treport_pack_garbage(data.garbage);\n+\tstring_list_clear(data.garbage, 0);\n+}\n+\n static void prepare_packed_git(struct repository *r);\n /*\n * Give a fast, rough count of the number of objects in the repository. This\ndiff --git a/packfile.h b/packfile.h\nindex e0a38aba93..d2ad30300a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -28,6 +28,12 @@ extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n+typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len,\n+\t\t\t\t const char *file_pach, void *data);\n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data);\n+\n /* A hook to report invalid files in pack directory */\n #define PACKDIR_FILE_PACK 1\n #define PACKDIR_FILE_IDX 2\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351724","messageId":"20180706005321.124643-7-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 06/24] multi-pack-index: load into memory","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:03Z","receivedAt":"2018-07-06T00:53:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Create a new multi_pack_index struct for loading multi-pack-indexes into\nmemory. Create a test-tool builtin for reading basic information about\nthat multi-pack-index to verify the correct data is written.\n\nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n midx.c | 82 +++++++++++++++++++++++++++++++++++++\n midx.h | 4 ++\n object-store.h | 16 ++++++++\n t/helper/test-read-midx.c | 34 +++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5319-multi-pack-index.sh | 12 +++++-\n 8 files changed, 150 insertions(+), 1 deletion(-)\n create mode 100644 t/helper/test-read-midx.c\n\ndiff --git a/Makefile b/Makefile\nindex f5636c711d..0b801d1b16 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -717,6 +717,7 @@ TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n TEST_BUILTINS_OBJS += test-prio-queue.o\n TEST_BUILTINS_OBJS += test-read-cache.o\n+TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\ndiff --git a/midx.c b/midx.c\nindex f85f2d334d..fb388f5858 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,18 +1,100 @@\n #include \"cache.h\"\n #include \"csum-file.h\"\n #include \"lockfile.h\"\n+#include \"object-store.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n+#define MIDX_BYTE_FILE_VERSION 4\n+#define MIDX_BYTE_HASH_VERSION 5\n+#define MIDX_BYTE_NUM_CHUNKS 6\n+#define MIDX_BYTE_NUM_PACKS 8\n #define MIDX_HASH_VERSION 1\n #define MIDX_HEADER_SIZE 12\n+#define MIDX_HASH_LEN 20\n+#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m = NULL;\n+\tint fd;\n+\tstruct stat st;\n+\tsize_t midx_size;\n+\tvoid *midx_map = NULL;\n+\tuint32_t hash_version;\n+\tchar *midx_name = get_midx_filename(object_dir);\n+\n+\tfd = git_open(midx_name);\n+\n+\tif (fd < 0)\n+\t\tgoto cleanup_fail;\n+\tif (fstat(fd, &st)) {\n+\t\terror_errno(_(\"failed to read %s\"), midx_name);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tmidx_size = xsize_t(st.st_size);\n+\n+\tif (midx_size < MIDX_MIN_SIZE) {\n+\t\tclose(fd);\n+\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tFREE_AND_NULL(midx_name);\n+\n+\tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\n+\tm = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n+\tstrcpy(m->object_dir, object_dir);\n+\tm->fd = fd;\n+\tm->data = midx_map;\n+\tm->data_len = midx_size;\n+\n+\tm->signature = get_be32(m->data);\n+\tif (m->signature != MIDX_SIGNATURE) {\n+\t\terror(_(\"multi-pack-index signature 0x%08x does not match signature 0x%08x\"),\n+\t\t m->signature, MIDX_SIGNATURE);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tm->version = m->data[MIDX_BYTE_FILE_VERSION];\n+\tif (m->version != MIDX_VERSION) {\n+\t\terror(_(\"multi-pack-index version %d not recognized\"),\n+\t\t m->version);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\thash_version = m->data[MIDX_BYTE_HASH_VERSION];\n+\tif (hash_version != MIDX_HASH_VERSION) {\n+\t\terror(_(\"hash version %u does not match\"), hash_version);\n+\t\tgoto cleanup_fail;\n+\t}\n+\tm->hash_len = MIDX_HASH_LEN;\n+\n+\tm->num_chunks = m->data[MIDX_BYTE_NUM_CHUNKS];\n+\n+\tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n+\n+\treturn m;\n+\n+cleanup_fail:\n+\t/* no need to check for NULL when freeing */\n+\tfree(m);\n+\tfree(midx_name);\n+\tif (midx_map)\n+\t\tmunmap(midx_map, midx_size);\n+\tif (0 <= fd)\n+\t\tclose(fd);\n+\treturn NULL;\n+}\n+\n static size_t write_midx_header(struct hashfile *f,\n \t\t\t\tunsigned char num_chunks,\n \t\t\t\tuint32_t num_packs)\ndiff --git a/midx.h b/midx.h\nindex dbdbe9f873..2d83dd9ec1 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -1,6 +1,10 @@\n #ifndef __MIDX_H__\n #define __MIDX_H__\n \n+struct multi_pack_index;\n+\n+struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n+\n int write_midx_file(const char *object_dir);\n \n #endif\ndiff --git a/object-store.h b/object-store.h\nindex d683112fd7..4f410841cc 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -84,6 +84,22 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+struct multi_pack_index {\n+\tint fd;\n+\n+\tconst unsigned char *data;\n+\tsize_t data_len;\n+\n+\tuint32_t signature;\n+\tunsigned char version;\n+\tunsigned char hash_len;\n+\tunsigned char num_chunks;\n+\tuint32_t num_packs;\n+\tuint32_t num_objects;\n+\n+\tchar object_dir[FLEX_ARRAY];\n+};\n+\n struct raw_object_store {\n \t/*\n \t * Path to the repository's object store.\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nnew file mode 100644\nindex 0000000000..5abf969175\n--- /dev/null\n+++ b/t/helper/test-read-midx.c\n@@ -0,0 +1,34 @@\n+/*\n+ * test-mktemp.c: code to exercise the creation of temporary files\n+ */\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"midx.h\"\n+#include \"repository.h\"\n+#include \"object-store.h\"\n+\n+static int read_midx_file(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tprintf(\"header: %08x %d %d %d\\n\",\n+\t m->signature,\n+\t m->version,\n+\t m->num_chunks,\n+\t m->num_packs);\n+\n+\tprintf(\"object_dir: %s\\n\", m->object_dir);\n+\n+\treturn 0;\n+}\n+\n+int cmd__read_midx(int argc, const char **argv)\n+{\n+\tif (argc != 2)\n+\t\tusage(\"read-midx \");\n+\n+\treturn read_midx_file(argv[1]);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 805a45de9c..1c3ab36e6c 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -27,6 +27,7 @@ static struct test_cmd cmds[] = {\n \t{ \"path-utils\", cmd__path_utils },\n \t{ \"prio-queue\", cmd__prio_queue },\n \t{ \"read-cache\", cmd__read_cache },\n+\t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"regex\", cmd__regex },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 7116ddfb94..6af8c08a66 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -21,6 +21,7 @@ int cmd__online_cpus(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n int cmd__prio_queue(int argc, const char **argv);\n int cmd__read_cache(int argc, const char **argv);\n+int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 8622a7cdce..2ecc369529 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,9 +3,19 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n+midx_read_expect () {\n+\tcat >expect <<-EOF\n+\theader: 4d494458 1 0 0\n+\tobject_dir: .\n+\tEOF\n+\ttest-tool read-midx . >actual &&\n+\ttest_cmp expect actual\n+}\n+\n test_expect_success 'write midx with no packs' '\n \tgit multi-pack-index --object-dir=. write &&\n-\ttest_path_is_file pack/multi-pack-index\n+\ttest_path_is_file pack/multi-pack-index &&\n+\tmidx_read_expect\n '\n \n test_done\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351725","messageId":"20180706005321.124643-10-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 09/24] multi-pack-index: read packfile list","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:06Z","receivedAt":"2018-07-06T00:53:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When constructing a multi-pack-index file for a given object directory,\nread the files within the enclosed pack directory and find matches that\nend with \".idx\" and find the correct paired packfile using\nadd_packed_git().\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 46 ++++++++++++++++++++++++++++++++++++-\n t/t5319-multi-pack-index.sh | 16 ++++++-------\n 2 files changed, 53 insertions(+), 9 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex fb388f5858..b0722485df 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,6 +1,8 @@\n #include \"cache.h\"\n #include \"csum-file.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n+#include \"packfile.h\"\n #include \"object-store.h\"\n #include \"midx.h\"\n \n@@ -112,12 +114,39 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+struct pack_list {\n+\tstruct packed_git **list;\n+\tuint32_t nr;\n+\tuint32_t alloc_list;\n+};\n+\n+static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n+\t\t\t const char *file_name, void *data)\n+{\n+\tstruct pack_list *packs = (struct pack_list *)data;\n+\n+\tif (ends_with(file_name, \".idx\")) {\n+\t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n+\n+\t\tpacks->list[packs->nr] = add_packed_git(full_path,\n+\t\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t\t 0);\n+\t\tif (!packs->list[packs->nr]) {\n+\t\t\twarning(_(\"failed to add packfile '%s'\"),\n+\t\t\t\tfull_path);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char num_chunks = 0;\n \tchar *midx_name;\n+\tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n+\tstruct pack_list packs;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -126,14 +155,29 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n+\tpacks.nr = 0;\n+\tpacks.alloc_list = 16;\n+\tpacks.list = NULL;\n+\tALLOC_ARRAY(packs.list, packs.alloc_list);\n+\n+\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\twrite_midx_header(f, num_chunks, 0);\n+\twrite_midx_header(f, num_chunks, packs.nr);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n \n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tif (packs.list[i]) {\n+\t\t\tclose_pack(packs.list[i]);\n+\t\t\tfree(packs.list[i]);\n+\t\t}\n+\t}\n+\n+\tfree(packs.list);\n \treturn 0;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 1be7be02b8..fd0a3f3be7 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -4,8 +4,9 @@ test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n midx_read_expect () {\n+\tNUM_PACKS=$1\n \tcat >expect <<-EOF\n-\theader: 4d494458 1 0 0\n+\theader: 4d494458 1 0 $NUM_PACKS\n \tobject_dir: .\n \tEOF\n \ttest-tool read-midx . >actual &&\n@@ -15,8 +16,7 @@ midx_read_expect () {\n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm -f pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\ttest_path_is_file pack/multi-pack-index &&\n-\tmidx_read_expect\n+\tmidx_read_expect 0\n '\n \n test_expect_success 'create objects' '\n@@ -47,13 +47,13 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test obj-list &&\n \t\tgit update-ref HEAD $commit &&\n-\t\tgit pack-objects --index-version=2 test-pack \n---\n Documentation/technical/pack-format.txt | 6 +\n midx.c | 176 +++++++++++++++++++++++-\n object-store.h | 2 +\n t/helper/test-read-midx.c | 7 +\n t/t5319-multi-pack-index.sh | 3 +-\n 5 files changed, 191 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex e060e693f4..6c5a77475f 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -296,6 +296,12 @@ CHUNK LOOKUP:\n \n CHUNK DATA:\n \n+\tPackfile Names (ID: {'P', 'N', 'A', 'M'})\n+\t Stores the packfile names as concatenated, null-terminated strings.\n+\t Packfiles must be listed in lexicographic order for fast lookups by\n+\t name. This is the only chunk not guaranteed to be a multiple of four\n+\t bytes in length, so should be the last chunk for alignment reasons.\n+\n \t(This section intentionally left incomplete.)\n \n TRAILER:\ndiff --git a/midx.c b/midx.c\nindex b0722485df..f78c161422 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -17,6 +17,11 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n+#define MIDX_MAX_CHUNKS 1\n+#define MIDX_CHUNK_ALIGNMENT 4\n+#define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n@@ -31,6 +36,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tvoid *midx_map = NULL;\n \tuint32_t hash_version;\n \tchar *midx_name = get_midx_filename(object_dir);\n+\tuint32_t i;\n \n \tfd = git_open(midx_name);\n \n@@ -84,6 +90,33 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n \n+\tfor (i = 0; i < m->num_chunks; i++) {\n+\t\tuint32_t chunk_id = get_be32(m->data + MIDX_HEADER_SIZE +\n+\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n+\t\tuint64_t chunk_offset = get_be64(m->data + MIDX_HEADER_SIZE + 4 +\n+\t\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n+\n+\t\tswitch (chunk_id) {\n+\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n+\t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase 0:\n+\t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\t/*\n+\t\t\t\t * Do nothing on unrecognized chunks, allowing future\n+\t\t\t\t * extensions to add optional chunks.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (!m->chunk_pack_names)\n+\t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n+\n \treturn m;\n \n cleanup_fail:\n@@ -116,8 +149,11 @@ static size_t write_midx_header(struct hashfile *f,\n \n struct pack_list {\n \tstruct packed_git **list;\n+\tchar **names;\n \tuint32_t nr;\n \tuint32_t alloc_list;\n+\tuint32_t alloc_names;\n+\tsize_t pack_name_concat_len;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -127,6 +163,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \tif (ends_with(file_name, \".idx\")) {\n \t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n+\t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n \n \t\tpacks->list[packs->nr] = add_packed_git(full_path,\n \t\t\t\t\t\t\t full_path_len,\n@@ -136,17 +173,90 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n+\n+\t\tpacks->names[packs->nr] = xstrdup(file_name);\n+\t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n+\t\tpacks->nr++;\n+\t}\n+}\n+\n+struct pack_pair {\n+\tuint32_t pack_int_id;\n+\tchar *pack_name;\n+};\n+\n+static int pack_pair_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_pair *a = (struct pack_pair *)_a;\n+\tstruct pack_pair *b = (struct pack_pair *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n+static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n+{\n+\tuint32_t i;\n+\tstruct pack_pair *pairs;\n+\n+\tALLOC_ARRAY(pairs, nr_packs);\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\tpairs[i].pack_int_id = i;\n+\t\tpairs[i].pack_name = pack_names[i];\n+\t}\n+\n+\tQSORT(pairs, nr_packs, pack_pair_compare);\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\tpack_names[i] = pairs[i].pack_name;\n+\t\tperm[pairs[i].pack_int_id] = i;\n+\t}\n+\n+\tfree(pairs);\n+}\n+\n+static size_t write_midx_pack_names(struct hashfile *f,\n+\t\t\t\t char **pack_names,\n+\t\t\t\t uint32_t num_packs)\n+{\n+\tuint32_t i;\n+\tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < num_packs; i++) {\n+\t\tsize_t writelen = strlen(pack_names[i]) + 1;\n+\n+\t\tif (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n+\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n+\t\t\t pack_names[i - 1],\n+\t\t\t pack_names[i]);\n+\n+\t\thashwrite(f, pack_names[i], writelen);\n+\t\twritten += writelen;\n+\t}\n+\n+\t/* add padding to be aligned */\n+\ti = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n+\tif (i < MIDX_CHUNK_ALIGNMENT) {\n+\t\tmemset(padding, 0, sizeof(padding));\n+\t\thashwrite(f, padding, i);\n+\t\twritten += i;\n \t}\n+\n+\treturn written;\n }\n \n int write_midx_file(const char *object_dir)\n {\n-\tunsigned char num_chunks = 0;\n+\tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct pack_list packs;\n+\tuint32_t *pack_perm = NULL;\n+\tuint64_t written = 0;\n+\tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n+\tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -157,16 +267,76 @@ int write_midx_file(const char *object_dir)\n \n \tpacks.nr = 0;\n \tpacks.alloc_list = 16;\n+\tpacks.alloc_names = 16;\n \tpacks.list = NULL;\n+\tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n+\tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n+\tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n+\tALLOC_ARRAY(pack_perm, packs.nr);\n+\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\twrite_midx_header(f, num_chunks, packs.nr);\n+\tcur_chunk = 0;\n+\tnum_chunks = 1;\n+\n+\twritten = write_midx_header(f, num_chunks, packs.nr);\n+\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n+\tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n+\n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = 0;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n+\n+\tfor (i = 0; i <= num_chunks; i++) {\n+\t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n+\t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n+\t\t\t chunk_offsets[i - 1],\n+\t\t\t chunk_offsets[i]);\n+\n+\t\tif (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n+\t\t\tBUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n+\t\t\t chunk_offsets[i]);\n+\n+\t\thashwrite_be32(f, chunk_ids[i]);\n+\t\thashwrite_be32(f, chunk_offsets[i] >> 32);\n+\t\thashwrite_be32(f, chunk_offsets[i]);\n+\n+\t\twritten += MIDX_CHUNKLOOKUP_WIDTH;\n+\t}\n+\n+\tfor (i = 0; i < num_chunks; i++) {\n+\t\tif (written != chunk_offsets[i])\n+\t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n+\t\t\t chunk_offsets[i],\n+\t\t\t written,\n+\t\t\t chunk_ids[i]);\n+\n+\t\tswitch (chunk_ids[i]) {\n+\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n+\t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n+\t\t\t\t chunk_ids[i]);\n+\t\t}\n+\t}\n+\n+\tif (written != chunk_offsets[num_chunks])\n+\t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n+\t\t written,\n+\t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n@@ -176,8 +346,10 @@ int write_midx_file(const char *object_dir)\n \t\t\tclose_pack(packs.list[i]);\n \t\t\tfree(packs.list[i]);\n \t\t}\n+\t\tfree(packs.names[i]);\n \t}\n \n \tfree(packs.list);\n+\tfree(packs.names);\n \treturn 0;\n }\ndiff --git a/object-store.h b/object-store.h\nindex 4f410841cc..c87d051849 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -97,6 +97,8 @@ struct multi_pack_index {\n \tuint32_t num_packs;\n \tuint32_t num_objects;\n \n+\tconst unsigned char *chunk_pack_names;\n+\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 5abf969175..a9232d8219 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -20,6 +20,13 @@ static int read_midx_file(const char *object_dir)\n \t m->num_chunks,\n \t m->num_packs);\n \n+\tprintf(\"chunks:\");\n+\n+\tif (m->chunk_pack_names)\n+\t\tprintf(\" pack_names\");\n+\n+\tprintf(\"\\n\");\n+\n \tprintf(\"object_dir: %s\\n\", m->object_dir);\n \n \treturn 0;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex fd0a3f3be7..f458758945 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -6,7 +6,8 @@ test_description='multi-pack-indexes'\n midx_read_expect () {\n \tNUM_PACKS=$1\n \tcat >expect <<-EOF\n-\theader: 4d494458 1 0 $NUM_PACKS\n+\theader: 4d494458 1 1 $NUM_PACKS\n+\tchunks: pack_names\n \tobject_dir: .\n \tEOF\n \ttest-tool read-midx . >actual &&\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351727","messageId":"20180706005321.124643-14-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 13/24] midx: write object ids in a chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:10Z","receivedAt":"2018-07-06T00:53:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 4 +++\n midx.c | 47 +++++++++++++++++++++++--\n object-store.h | 1 +\n t/helper/test-read-midx.c | 2 ++\n t/t5319-multi-pack-index.sh | 4 +--\n 5 files changed, 53 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 6c5a77475f..78ee0489c6 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -302,6 +302,10 @@ CHUNK DATA:\n \t name. This is the only chunk not guaranteed to be a multiple of four\n \t bytes in length, so should be the last chunk for alignment reasons.\n \n+\tOID Lookup (ID: {'O', 'I', 'D', 'L'})\n+\t The OIDs for all objects in the MIDX are stored in lexicographic\n+\t order in this chunk.\n+\n \t(This section intentionally left incomplete.)\n \n TRAILER:\ndiff --git a/midx.c b/midx.c\nindex 028e3aa5e9..7606addab6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -18,9 +18,10 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 1\n+#define MIDX_MAX_CHUNKS 2\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n \n static char *get_midx_filename(const char *object_dir)\n@@ -103,6 +104,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n+\t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n \t\t\tcase 0:\n \t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n \t\t\t\tbreak;\n@@ -118,6 +123,8 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tif (!m->chunk_pack_names)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n+\tif (!m->chunk_oid_lookup)\n+\t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n \n@@ -384,6 +391,32 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \treturn written;\n }\n \n+static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n+\t\t\t\t struct pack_midx_entry *objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tuint32_t i;\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\tif (i < nr_objects - 1) {\n+\t\t\tstruct pack_midx_entry *next = list;\n+\t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n+\t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n+\t\t\t\toid_to_hex(&obj->oid),\n+\t\t\t\toid_to_hex(&next->oid));\n+\t\t}\n+\n+\t\thashwrite(f, obj->oid.hash, (int)hash_len);\n+\t\twritten += hash_len;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n@@ -430,7 +463,7 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 1;\n+\tnum_chunks = 2;\n \n \twritten = write_midx_header(f, num_chunks, packs.nr);\n \n@@ -438,9 +471,13 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = 0;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n \n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = 0;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n+\n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n \t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n@@ -470,6 +507,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n+\t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tdefault:\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\ndiff --git a/object-store.h b/object-store.h\nindex 88169b33e9..25f8530eb4 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -98,6 +98,7 @@ struct multi_pack_index {\n \tuint32_t num_objects;\n \n \tconst unsigned char *chunk_pack_names;\n+\tconst unsigned char *chunk_oid_lookup;\n \n \tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 0b53a9e8b5..60bca5b668 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -25,6 +25,8 @@ static int read_midx_file(const char *object_dir)\n \n \tif (m->chunk_pack_names)\n \t\tprintf(\" pack_names\");\n+\tif (m->chunk_oid_lookup)\n+\t\tprintf(\" oid_lookup\");\n \n \tprintf(\"\\n\");\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 4610352b69..cbe84c74fc 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -6,8 +6,8 @@ test_description='multi-pack-indexes'\n midx_read_expect () {\n \tNUM_PACKS=$1\n \tcat >expect <<-EOF\n-\theader: 4d494458 1 1 $NUM_PACKS\n-\tchunks: pack_names\n+\theader: 4d494458 1 2 $NUM_PACKS\n+\tchunks: pack_names oid_lookup\n \tpacks:\n \tEOF\n \tif [ $NUM_PACKS -ge 1 ]\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351728","messageId":"20180706005321.124643-13-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 12/24] midx: sort and deduplicate objects from packfiles","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:09Z","receivedAt":"2018-07-06T00:53:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before writing a list of objects and their offsets to a multi-pack-index,\nwe need to collect the list of objects contained in the packfiles. There\nmay be multiple copies of some objects, so this list must be deduplicated.\n\nIt is possible to artificially get into a state where there are many\nduplicate copies of objects. That can create high memory pressure if we\nare to create a list of all objects before de-duplication. To reduce\nthis memory pressure without a significant performance drop,\nautomatically group objects by the first byte of their object id. Use\nthe IDX fanout tables to group the data, copy to a local array, then\nsort.\n\nCopy only the de-duplicated entries. Select the duplicate based on the\nmost-recent modified time of a packfile containing the object.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n packfile.c | 17 +++++++\n packfile.h | 2 +\n 3 files changed, 148 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex ffe29af65d..028e3aa5e9 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -4,6 +4,7 @@\n #include \"lockfile.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"packfile.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n@@ -183,14 +184,22 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n \n \t\tpacks->list[packs->nr] = add_packed_git(full_path,\n-\t\t\t\t\t\t\t full_path_len,\n-\t\t\t\t\t\t\t 0);\n+\t\t\t\t\t\t\tfull_path_len, 0);\n+\n \t\tif (!packs->list[packs->nr]) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n+\t\tif (open_pack_index(packs->list[packs->nr])) {\n+\t\t\twarning(_(\"failed to open pack-index '%s'\"),\n+\t\t\t\tfull_path);\n+\t\t\tclose_pack(packs->list[packs->nr]);\n+\t\t\tFREE_AND_NULL(packs->list[packs->nr]);\n+\t\t\treturn;\n+\t\t}\n+\n \t\tpacks->names[packs->nr] = xstrdup(file_name);\n \t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n@@ -231,6 +240,119 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n \tfree(pairs);\n }\n \n+struct pack_midx_entry {\n+\tstruct object_id oid;\n+\tuint32_t pack_int_id;\n+\ttime_t pack_mtime;\n+\tuint64_t offset;\n+};\n+\n+static int midx_oid_compare(const void *_a, const void *_b)\n+{\n+\tconst struct pack_midx_entry *a = (const struct pack_midx_entry *)_a;\n+\tconst struct pack_midx_entry *b = (const struct pack_midx_entry *)_b;\n+\tint cmp = oidcmp(&a->oid, &b->oid);\n+\n+\tif (cmp)\n+\t\treturn cmp;\n+\n+\tif (a->pack_mtime > b->pack_mtime)\n+\t\treturn -1;\n+\telse if (a->pack_mtime < b->pack_mtime)\n+\t\treturn 1;\n+\n+\treturn a->pack_int_id - b->pack_int_id;\n+}\n+\n+static void fill_pack_entry(uint32_t pack_int_id,\n+\t\t\t struct packed_git *p,\n+\t\t\t uint32_t cur_object,\n+\t\t\t struct pack_midx_entry *entry)\n+{\n+\tif (!nth_packed_object_oid(&entry->oid, p, cur_object))\n+\t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n+\n+\tentry->pack_int_id = pack_int_id;\n+\tentry->pack_mtime = p->mtime;\n+\n+\tentry->offset = nth_packed_object_offset(p, cur_object);\n+}\n+\n+/*\n+ * It is possible to artificially get into a state where there are many\n+ * duplicate copies of objects. That can create high memory pressure if\n+ * we are to create a list of all objects before de-duplication. To reduce\n+ * this memory pressure without a significant performance drop, automatically\n+ * group objects by the first byte of their object id. Use the IDX fanout\n+ * tables to group the data, copy to a local array, then sort.\n+ *\n+ * Copy only the de-duplicated entries (selected by most-recent modified time\n+ * of a packfile containing the object).\n+ */\n+static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n+\t\t\t\t\t\t uint32_t *perm,\n+\t\t\t\t\t\t uint32_t nr_packs,\n+\t\t\t\t\t\t uint32_t *nr_objects)\n+{\n+\tuint32_t cur_fanout, cur_pack, cur_object;\n+\tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n+\tstruct pack_midx_entry *entries_by_fanout = NULL;\n+\tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\n+\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++)\n+\t\ttotal_objects += p[cur_pack]->num_objects;\n+\n+\t/*\n+\t * As we de-duplicate by fanout value, we expect the fanout\n+\t * slices to be evenly distributed, with some noise. Hence,\n+\t * allocate slightly more than one 256th.\n+\t */\n+\talloc_objects = alloc_fanout = total_objects > 3200 ? total_objects / 200 : 16;\n+\n+\tALLOC_ARRAY(entries_by_fanout, alloc_fanout);\n+\tALLOC_ARRAY(deduplicated_entries, alloc_objects);\n+\t*nr_objects = 0;\n+\n+\tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n+\t\tuint32_t nr_fanout = 0;\n+\n+\t\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\t\tuint32_t start = 0, end;\n+\n+\t\t\tif (cur_fanout)\n+\t\t\t\tstart = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n+\t\t\tend = get_pack_fanout(p[cur_pack], cur_fanout);\n+\n+\t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n+\t\t\t\tfill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tnr_fanout++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tQSORT(entries_by_fanout, nr_fanout, midx_oid_compare);\n+\n+\t\t/*\n+\t\t * The batch is now sorted by OID and then mtime (descending).\n+\t\t * Take only the first duplicate.\n+\t\t */\n+\t\tfor (cur_object = 0; cur_object < nr_fanout; cur_object++) {\n+\t\t\tif (cur_object && !oidcmp(&entries_by_fanout[cur_object - 1].oid,\n+\t\t\t\t\t\t &entries_by_fanout[cur_object].oid))\n+\t\t\t\tcontinue;\n+\n+\t\t\tALLOC_GROW(deduplicated_entries, *nr_objects + 1, alloc_objects);\n+\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n+\t\t\t &entries_by_fanout[cur_object],\n+\t\t\t sizeof(struct pack_midx_entry));\n+\t\t\t(*nr_objects)++;\n+\t\t}\n+\t}\n+\n+\tfree(entries_by_fanout);\n+\treturn deduplicated_entries;\n+}\n+\n static size_t write_midx_pack_names(struct hashfile *f,\n \t\t\t\t char **pack_names,\n \t\t\t\t uint32_t num_packs)\n@@ -274,6 +396,8 @@ int write_midx_file(const char *object_dir)\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n+\tuint32_t nr_entries;\n+\tstruct pack_midx_entry *entries = NULL;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -299,6 +423,8 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n+\tentries = get_sorted_entries(packs.list, pack_perm, packs.nr, &nr_entries);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n@@ -368,5 +494,6 @@ int write_midx_file(const char *object_dir)\n \n \tfree(packs.list);\n \tfree(packs.names);\n+\tfree(entries);\n \treturn 0;\n }\ndiff --git a/packfile.c b/packfile.c\nindex ee1ab9b804..3d652212c6 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -196,6 +196,23 @@ int open_pack_index(struct packed_git *p)\n \treturn ret;\n }\n \n+uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)\n+{\n+\tconst uint32_t *level1_ofs = p->index_data;\n+\n+\tif (!level1_ofs) {\n+\t\tif (open_pack_index(p))\n+\t\t\treturn 0;\n+\t\tlevel1_ofs = p->index_data;\n+\t}\n+\n+\tif (p->index_version > 1) {\n+\t\tlevel1_ofs += 2;\n+\t}\n+\n+\treturn ntohl(level1_ofs[value]);\n+}\n+\n static struct packed_git *alloc_packed_git(int extra)\n {\n \tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\ndiff --git a/packfile.h b/packfile.h\nindex d2ad30300a..b0eed44c0b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -69,6 +69,8 @@ extern int open_pack_index(struct packed_git *);\n */\n extern void close_pack_index(struct packed_git *);\n \n+extern uint32_t get_pack_fanout(struct packed_git *p, uint32_t value);\n+\n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void close_pack_windows(struct packed_git *);\n extern void close_pack(struct packed_git *);\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351729","messageId":"20180706005321.124643-15-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 14/24] midx: write object id fanout chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:11Z","receivedAt":"2018-07-06T00:54:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 5 +++\n midx.c | 53 +++++++++++++++++++++++--\n object-store.h | 1 +\n t/helper/test-read-midx.c | 4 +-\n t/t5319-multi-pack-index.sh | 16 ++++----\n 5 files changed, 68 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 78ee0489c6..3215f7bfcd 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -302,6 +302,11 @@ CHUNK DATA:\n \t name. This is the only chunk not guaranteed to be a multiple of four\n \t bytes in length, so should be the last chunk for alignment reasons.\n \n+\tOID Fanout (ID: {'O', 'I', 'D', 'F'})\n+\t The ith entry, F[i], stores the number of OIDs with first\n+\t byte at most i. Thus F[255] stores the total\n+\t number of objects.\n+\n \tOID Lookup (ID: {'O', 'I', 'D', 'L'})\n \t The OIDs for all objects in the MIDX are stored in lexicographic\n \t order in this chunk.\ndiff --git a/midx.c b/midx.c\nindex 7606addab6..404147bb9f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -18,11 +18,13 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 2\n+#define MIDX_MAX_CHUNKS 3\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n+#define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n \n static char *get_midx_filename(const char *object_dir)\n {\n@@ -104,6 +106,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n+\t\t\t\tm->chunk_oid_fanout = (uint32_t *)(m->data + chunk_offset);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n \t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n \t\t\t\tbreak;\n@@ -123,9 +129,13 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tif (!m->chunk_pack_names)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n+\tif (!m->chunk_oid_fanout)\n+\t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n \tif (!m->chunk_oid_lookup)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n \n+\tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n+\n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n \n \tcur_pack_name = (const char *)m->chunk_pack_names;\n@@ -391,6 +401,35 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \treturn written;\n }\n \n+static size_t write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t struct pack_midx_entry *objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tstruct pack_midx_entry *last = objects + nr_objects;\n+\tuint32_t count = 0;\n+\tuint32_t i;\n+\n+\t/*\n+\t* Write the first-level table (the list is sorted,\n+\t* but we use a 256-entry lookup to be able to avoid\n+\t* having to do eight extra binary search iterations).\n+\t*/\n+\tfor (i = 0; i < 256; i++) {\n+\t\tstruct pack_midx_entry *next = list;\n+\n+\t\twhile (next < last && next->oid.hash[0] == i) {\n+\t\t\tcount++;\n+\t\t\tnext++;\n+\t\t}\n+\n+\t\thashwrite_be32(f, count);\n+\t\tlist = next;\n+\t}\n+\n+\treturn MIDX_CHUNK_FANOUT_SIZE;\n+}\n+\n static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n \t\t\t\t struct pack_midx_entry *objects,\n \t\t\t\t uint32_t nr_objects)\n@@ -463,7 +502,7 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 2;\n+\tnum_chunks = 3;\n \n \twritten = write_midx_header(f, num_chunks, packs.nr);\n \n@@ -471,9 +510,13 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n \n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n+\n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = 0;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n@@ -507,6 +550,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n+\t\t\t\twritten += write_midx_oid_fanout(f, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n \t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n \t\t\t\tbreak;\ndiff --git a/object-store.h b/object-store.h\nindex 25f8530eb4..3357e51100 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -98,6 +98,7 @@ struct multi_pack_index {\n \tuint32_t num_objects;\n \n \tconst unsigned char *chunk_pack_names;\n+\tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \n \tconst char **pack_names;\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 60bca5b668..d1bb7290ae 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -25,10 +25,12 @@ static int read_midx_file(const char *object_dir)\n \n \tif (m->chunk_pack_names)\n \t\tprintf(\" pack_names\");\n+\tif (m->chunk_oid_fanout)\n+\t\tprintf(\" oid_fanout\");\n \tif (m->chunk_oid_lookup)\n \t\tprintf(\" oid_lookup\");\n \n-\tprintf(\"\\n\");\n+\tprintf(\"\\nnum_objects: %d\\n\", m->num_objects);\n \n \tprintf(\"packs:\\n\");\n \tfor (i = 0; i < m->num_packs; i++)\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex cbe84c74fc..23f653473a 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -5,9 +5,11 @@ test_description='multi-pack-indexes'\n \n midx_read_expect () {\n \tNUM_PACKS=$1\n+\tNUM_OBJECTS=$2\n \tcat >expect <<-EOF\n-\theader: 4d494458 1 2 $NUM_PACKS\n-\tchunks: pack_names oid_lookup\n+\theader: 4d494458 1 3 $NUM_PACKS\n+\tchunks: pack_names oid_fanout oid_lookup\n+\tnum_objects: $NUM_OBJECTS\n \tpacks:\n \tEOF\n \tif [ $NUM_PACKS -ge 1 ]\n@@ -22,7 +24,7 @@ midx_read_expect () {\n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm -f pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\tmidx_read_expect 0\n+\tmidx_read_expect 0 0\n '\n \n test_expect_success 'create objects' '\n@@ -53,13 +55,13 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test \n---\n Documentation/technical/pack-format.txt | 15 +++-\n midx.c | 100 +++++++++++++++++++++++-\n object-store.h | 2 +\n t/helper/test-read-midx.c | 4 +\n t/t5319-multi-pack-index.sh | 44 ++++++++---\n 5 files changed, 150 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 3215f7bfcd..cab5bdd2ff 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -311,7 +311,20 @@ CHUNK DATA:\n \t The OIDs for all objects in the MIDX are stored in lexicographic\n \t order in this chunk.\n \n-\t(This section intentionally left incomplete.)\n+\tObject Offsets (ID: {'O', 'O', 'F', 'F'})\n+\t Stores two 4-byte values for every object.\n+\t 1: The pack-int-id for the pack storing this object.\n+\t 2: The offset within the pack.\n+\t\tIf all offsets are less than 2^31, then the large offset chunk\n+\t\twill not exist and offsets are stored as in IDX v1.\n+\t\tIf there is at least one offset value larger than 2^32-1, then\n+\t\tthe large offset chunk must exist. If the large offset chunk\n+\t\texists and the 31st bit is on, then removing that bit reveals\n+\t\tthe row in the large offsets containing the 8-byte offset of\n+\t\tthis object.\n+\n+\t[Optional] Object Large Offsets (ID: {'L', 'O', 'F', 'F'})\n+\t 8-byte offsets into large packfiles.\n \n TRAILER:\n \ndiff --git a/midx.c b/midx.c\nindex 404147bb9f..cc35abe7a2 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -18,13 +18,18 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 3\n+#define MIDX_MAX_CHUNKS 5\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n+#define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n+#define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n+#define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n+#define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n+#define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n static char *get_midx_filename(const char *object_dir)\n {\n@@ -114,6 +119,14 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n+\t\t\t\tm->chunk_object_offsets = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n+\t\t\t\tm->chunk_large_offsets = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n \t\t\tcase 0:\n \t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n \t\t\t\tbreak;\n@@ -133,6 +146,8 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n \tif (!m->chunk_oid_lookup)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n+\tif (!m->chunk_object_offsets)\n+\t\tdie(_(\"multi-pack-index missing required object offsets chunk\"));\n \n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n@@ -456,6 +471,56 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n \treturn written;\n }\n \n+static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n+\t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tuint32_t i, nr_large_offset = 0;\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\thashwrite_be32(f, obj->pack_int_id);\n+\n+\t\tif (large_offset_needed && obj->offset >> 31)\n+\t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n+\t\telse if (!large_offset_needed && obj->offset >> 32)\n+\t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n+\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t obj->offset);\n+\t\telse\n+\t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n+\n+\t\twritten += MIDX_CHUNK_OFFSET_WIDTH;\n+\t}\n+\n+\treturn written;\n+}\n+\n+static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_offset,\n+\t\t\t\t struct pack_midx_entry *objects, uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tsize_t written = 0;\n+\n+\twhile (nr_large_offset) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\t\tuint64_t offset = obj->offset;\n+\n+\t\tif (!(offset >> 31))\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be32(f, offset >> 32);\n+\t\thashwrite_be32(f, offset & 0xffffffffUL);\n+\t\twritten += 2 * sizeof(uint32_t);\n+\n+\t\tnr_large_offset--;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n@@ -468,8 +533,9 @@ int write_midx_file(const char *object_dir)\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t nr_entries;\n+\tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n+\tint large_offsets_needed = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -496,13 +562,19 @@ int write_midx_file(const char *object_dir)\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n \tentries = get_sorted_entries(packs.list, pack_perm, packs.nr, &nr_entries);\n+\tfor (i = 0; i < nr_entries; i++) {\n+\t\tif (entries[i].offset > 0x7fffffff)\n+\t\t\tnum_large_offsets++;\n+\t\tif (entries[i].offset > 0xffffffff)\n+\t\t\tlarge_offsets_needed = 1;\n+\t}\n \n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 3;\n+\tnum_chunks = large_offsets_needed ? 5 : 4;\n \n \twritten = write_midx_header(f, num_chunks, packs.nr);\n \n@@ -518,9 +590,21 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = 0;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n \n+\tcur_chunk++;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_CHUNK_OFFSET_WIDTH;\n+\tif (large_offsets_needed) {\n+\t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n+\n+\t\tcur_chunk++;\n+\t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n+\t\t\t\t\t num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n+\t}\n+\n+\tchunk_ids[cur_chunk] = 0;\n+\n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n \t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n@@ -558,6 +642,14 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n+\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n+\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tdefault:\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\ndiff --git a/object-store.h b/object-store.h\nindex 3357e51100..07bcc80e02 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -100,6 +100,8 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_pack_names;\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n+\tconst unsigned char *chunk_object_offsets;\n+\tconst unsigned char *chunk_large_offsets;\n \n \tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex d1bb7290ae..20771d1c1d 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -29,6 +29,10 @@ static int read_midx_file(const char *object_dir)\n \t\tprintf(\" oid_fanout\");\n \tif (m->chunk_oid_lookup)\n \t\tprintf(\" oid_lookup\");\n+\tif (m->chunk_object_offsets)\n+\t\tprintf(\" object_offsets\");\n+\tif (m->chunk_large_offsets)\n+\t\tprintf(\" large_offsets\");\n \n \tprintf(\"\\nnum_objects: %d\\n\", m->num_objects);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 23f653473a..ae6c9d4d02 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -6,25 +6,28 @@ test_description='multi-pack-indexes'\n midx_read_expect () {\n \tNUM_PACKS=$1\n \tNUM_OBJECTS=$2\n+\tNUM_CHUNKS=$3\n+\tOBJECT_DIR=$4\n+\tEXTRA_CHUNKS=\"$5\"\n \tcat >expect <<-EOF\n-\theader: 4d494458 1 3 $NUM_PACKS\n-\tchunks: pack_names oid_fanout oid_lookup\n+\theader: 4d494458 1 $NUM_CHUNKS $NUM_PACKS\n+\tchunks: pack_names oid_fanout oid_lookup object_offsets$EXTRA_CHUNKS\n \tnum_objects: $NUM_OBJECTS\n \tpacks:\n \tEOF\n \tif [ $NUM_PACKS -ge 1 ]\n \tthen\n-\t\tls pack/ | grep idx | sort >> expect\n+\t\tls $OBJECT_DIR/pack/ | grep idx | sort >> expect\n \tfi\n-\tprintf \"object_dir: .\\n\" >>expect &&\n-\ttest-tool read-midx . >actual &&\n+\tprintf \"object_dir: $OBJECT_DIR\\n\" >>expect &&\n+\ttest-tool read-midx $OBJECT_DIR >actual &&\n \ttest_cmp expect actual\n }\n \n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm -f pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\tmidx_read_expect 0 0\n+\tmidx_read_expect 0 0 4 .\n '\n \n test_expect_success 'create objects' '\n@@ -55,13 +58,13 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test []\n+corrupt_data() {\n+\tfile=$1\n+\tpos=$2\n+\tdata=\"${3:-\\0}\"\n+\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n+}\n+\n+# Force 64-bit offsets by manipulating the idx file.\n+# This makes the IDX file _incorrect_ so be careful to clean up after!\n+test_expect_success 'force some 64-bit offsets with pack-objects' '\n+\tmkdir objects64 &&\n+\tmkdir objects64/pack &&\n+\tpack64=$(git pack-objects --index-version=2,0x40 objects64/pack/test-64 \n---\n midx.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++++-\n midx.h | 2 ++\n object-store.h | 1 +\n packfile.c | 8 ++++-\n 4 files changed, 100 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d5a61c0c53..84b045060a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -4,7 +4,7 @@\n #include \"lockfile.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n-#include \"packfile.h\"\n+#include \"sha1-lookup.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n@@ -152,6 +152,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n+\tm->packs = xcalloc(m->num_packs, sizeof(*m->packs));\n \n \tcur_pack_name = (const char *)m->chunk_pack_names;\n \tfor (i = 0; i < m->num_packs; i++) {\n@@ -180,6 +181,94 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \treturn NULL;\n }\n \n+static int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id)\n+{\n+\tstruct strbuf pack_name = STRBUF_INIT;\n+\n+\tif (pack_int_id >= m->num_packs)\n+\t\tBUG(\"bad pack-int-id\");\n+\n+\tif (m->packs[pack_int_id])\n+\t\treturn 0;\n+\n+\tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n+\t\t m->pack_names[pack_int_id]);\n+\n+\tm->packs[pack_int_id] = add_packed_git(pack_name.buf, pack_name.len, 1);\n+\tstrbuf_release(&pack_name);\n+\treturn !m->packs[pack_int_id];\n+}\n+\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)\n+{\n+\treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n+\t\t\t MIDX_HASH_LEN, result);\n+}\n+\n+static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n+{\n+\tconst unsigned char *offset_data;\n+\tuint32_t offset32;\n+\n+\toffset_data = m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH;\n+\toffset32 = get_be32(offset_data + sizeof(uint32_t));\n+\n+\tif (m->chunk_large_offsets && offset32 & MIDX_LARGE_OFFSET_NEEDED) {\n+\t\tif (sizeof(offset32) < sizeof(uint64_t))\n+\t\t\tdie(_(\"multi-pack-index stores a 64-bit offset, but off_t is too small\"));\n+\n+\t\toffset32 ^= MIDX_LARGE_OFFSET_NEEDED;\n+\t\treturn get_be64(m->chunk_large_offsets + sizeof(uint64_t) * offset32);\n+\t}\n+\n+\treturn offset32;\n+}\n+\n+static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n+{\n+\treturn get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);\n+}\n+\n+static int nth_midxed_pack_entry(struct multi_pack_index *m, struct pack_entry *e, uint32_t pos)\n+{\n+\tuint32_t pack_int_id;\n+\tstruct packed_git *p;\n+\n+\tif (pos >= m->num_objects)\n+\t\treturn 0;\n+\n+\tpack_int_id = nth_midxed_pack_int_id(m, pos);\n+\n+\tif (prepare_midx_pack(m, pack_int_id))\n+\t\tdie(_(\"error preparing packfile from multi-pack-index\"));\n+\tp = m->packs[pack_int_id];\n+\n+\t/*\n+\t* We are about to tell the caller where they can locate the\n+\t* requested object. We better make sure the packfile is\n+\t* still here and can be accessed before supplying that\n+\t* answer, as it may have been deleted since the MIDX was\n+\t* loaded!\n+\t*/\n+\tif (!is_pack_valid(p))\n+\t\treturn 0;\n+\n+\te->offset = nth_midxed_offset(m, pos);\n+\te->p = p;\n+\n+\treturn 1;\n+}\n+\n+int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m)\n+{\n+\tuint32_t pos;\n+\n+\tif (!bsearch_midx(oid, m, &pos))\n+\t\treturn 0;\n+\n+\treturn nth_midxed_pack_entry(m, e, pos);\n+}\n+\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n {\n \tstruct multi_pack_index *m = r->objects->multi_pack_index;\ndiff --git a/midx.h b/midx.h\nindex 731ad6f094..6b74a0640f 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -6,6 +6,8 @@\n struct multi_pack_index;\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\ndiff --git a/object-store.h b/object-store.h\nindex 7d67ad7aa9..03cc278758 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -106,6 +106,7 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_large_offsets;\n \n \tconst char **pack_names;\n+\tstruct packed_git **packs;\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/packfile.c b/packfile.c\nindex 5d4493dbf4..bc763d91b9 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1902,11 +1902,17 @@ static int fill_pack_entry(const struct object_id *oid,\n int find_pack_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e)\n {\n \tstruct list_head *pos;\n+\tstruct multi_pack_index *m;\n \n \tprepare_packed_git(r);\n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packed_git && !r->objects->multi_pack_index)\n \t\treturn 0;\n \n+\tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n+\t\tif (fill_midx_entry(oid, e, m))\n+\t\t\treturn 1;\n+\t}\n+\n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (fill_pack_entry(oid, e, p)) {\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351732","messageId":"20180706005321.124643-20-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 19/24] midx: use midx in abbreviation calculations","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:16Z","receivedAt":"2018-07-06T00:54:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 11 ++++++\n midx.h | 3 ++\n packfile.c | 6 ++++\n packfile.h | 1 +\n sha1-name.c | 70 +++++++++++++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh | 3 +-\n 6 files changed, 93 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 84b045060a..e66025f066 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -205,6 +205,17 @@ int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32\n \t\t\t MIDX_HASH_LEN, result);\n }\n \n+struct object_id *nth_midxed_object_oid(struct object_id *oid,\n+\t\t\t\t\tstruct multi_pack_index *m,\n+\t\t\t\t\tuint32_t n)\n+{\n+\tif (n >= m->num_objects)\n+\t\treturn NULL;\n+\n+\thashcpy(oid->hash, m->chunk_oid_lookup + m->hash_len * n);\n+\treturn oid;\n+}\n+\n static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n {\n \tconst unsigned char *offset_data;\ndiff --git a/midx.h b/midx.h\nindex 6b74a0640f..f7c2ec7893 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -7,6 +7,9 @@ struct multi_pack_index;\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+struct object_id *nth_midxed_object_oid(struct object_id *oid,\n+\t\t\t\t\tstruct multi_pack_index *m,\n+\t\t\t\t\tuint32_t n);\n int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \ndiff --git a/packfile.c b/packfile.c\nindex bc763d91b9..c0eb5ac885 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -961,6 +961,12 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packed_git;\n }\n \n+struct multi_pack_index *get_multi_pack_index(struct repository *r)\n+{\n+\tprepare_packed_git(r);\n+\treturn r->objects->multi_pack_index;\n+}\n+\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\ndiff --git a/packfile.h b/packfile.h\nindex b0eed44c0b..046280caf3 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -45,6 +45,7 @@ extern void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n+struct multi_pack_index *get_multi_pack_index(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/sha1-name.c b/sha1-name.c\nindex 60d9ef3c7e..7dc71201e6 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -12,6 +12,7 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"repository.h\"\n+#include \"midx.h\"\n \n static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);\n \n@@ -149,6 +150,32 @@ static int match_sha(unsigned len, const unsigned char *a, const unsigned char *\n \treturn 1;\n }\n \n+static void unique_in_midx(struct multi_pack_index *m,\n+\t\t\t struct disambiguate_state *ds)\n+{\n+\tuint32_t num, i, first = 0;\n+\tconst struct object_id *current = NULL;\n+\tnum = m->num_objects;\n+\n+\tif (!num)\n+\t\treturn;\n+\n+\tbsearch_midx(&ds->bin_pfx, m, &first);\n+\n+\t/*\n+\t * At this point, \"first\" is the location of the lowest object\n+\t * with an object name that could match \"bin_pfx\". See if we have\n+\t * 0, 1 or more objects that actually match(es).\n+\t */\n+\tfor (i = first; i < num && !ds->ambiguous; i++) {\n+\t\tstruct object_id oid;\n+\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n+\t\tif (!match_sha(ds->len, ds->bin_pfx.hash, current->hash))\n+\t\t\tbreak;\n+\t\tupdate_candidates(ds, current);\n+\t}\n+}\n+\n static void unique_in_pack(struct packed_git *p,\n \t\t\t struct disambiguate_state *ds)\n {\n@@ -177,8 +204,12 @@ static void unique_in_pack(struct packed_git *p,\n \n static void find_short_packed_object(struct disambiguate_state *ds)\n {\n+\tstruct multi_pack_index *m;\n \tstruct packed_git *p;\n \n+\tfor (m = get_multi_pack_index(the_repository); m && !ds->ambiguous;\n+\t m = m->next)\n+\t\tunique_in_midx(m, ds);\n \tfor (p = get_packed_git(the_repository); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n@@ -527,6 +558,42 @@ static int extend_abbrev_len(const struct object_id *oid, void *cb_data)\n \treturn 0;\n }\n \n+static void find_abbrev_len_for_midx(struct multi_pack_index *m,\n+\t\t\t\t struct min_abbrev_data *mad)\n+{\n+\tint match = 0;\n+\tuint32_t num, first = 0;\n+\tstruct object_id oid;\n+\tconst struct object_id *mad_oid;\n+\n+\tif (!m->num_objects)\n+\t\treturn;\n+\n+\tnum = m->num_objects;\n+\tmad_oid = mad->oid;\n+\tmatch = bsearch_midx(mad_oid, m, &first);\n+\n+\t/*\n+\t * first is now the position in the packfile where we would insert\n+\t * mad->hash if it does not exist (or the position of mad->hash if\n+\t * it does exist). Hence, we consider a maximum of two objects\n+\t * nearby for the abbreviation length.\n+\t */\n+\tmad->init_len = 0;\n+\tif (!match) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t} else if (first < num - 1) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t}\n+\tif (first > 0) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t}\n+\tmad->init_len = mad->cur_len;\n+}\n+\n static void find_abbrev_len_for_pack(struct packed_git *p,\n \t\t\t\t struct min_abbrev_data *mad)\n {\n@@ -565,8 +632,11 @@ static void find_abbrev_len_for_pack(struct packed_git *p,\n \n static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n {\n+\tstruct multi_pack_index *m;\n \tstruct packed_git *p;\n \n+\tfor (m = get_multi_pack_index(the_repository); m; m = m->next)\n+\t\tfind_abbrev_len_for_midx(m, mad);\n \tfor (p = get_packed_git(the_repository); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex fc582c9a59..4c630ecab4 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -93,7 +93,8 @@ compare_results_with_midx() {\n \tMSG=$1\n \ttest_expect_success \"check normal git operations: $MSG\" '\n \t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n-\t\tmidx_git_two_modes \"log --raw\"\n+\t\tmidx_git_two_modes \"log --raw\" &&\n+\t\tmidx_git_two_modes \"log --oneline\"\n \t'\n }\n \n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351733","messageId":"20180706005321.124643-21-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 20/24] midx: use existing midx when writing new one","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:17Z","receivedAt":"2018-07-06T00:54:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Due to how Windows handles replacing a lockfile when there is an open\nhandle, create the close_midx() method to close the existing midx before\nwriting the new one.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n midx.h | 1 +\n 2 files changed, 111 insertions(+), 6 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex e66025f066..7c00b02436 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -181,6 +181,23 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \treturn NULL;\n }\n \n+static void close_midx(struct multi_pack_index *m)\n+{\n+\tuint32_t i;\n+\tmunmap((unsigned char *)m->data, m->data_len);\n+\tclose(m->fd);\n+\tm->fd = -1;\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tif (m->packs[i]) {\n+\t\t\tclose_pack(m->packs[i]);\n+\t\t\tfree(m->packs);\n+\t\t}\n+\t}\n+\tFREE_AND_NULL(m->packs);\n+\tFREE_AND_NULL(m->pack_names);\n+}\n+\n static int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id)\n {\n \tstruct strbuf pack_name = STRBUF_INIT;\n@@ -280,6 +297,29 @@ int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct mu\n \treturn nth_midxed_pack_entry(m, e, pos);\n }\n \n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_name)\n+{\n+\tuint32_t first = 0, last = m->num_packs;\n+\n+\twhile (first < last) {\n+\t\tuint32_t mid = first + (last - first) / 2;\n+\t\tconst char *current;\n+\t\tint cmp;\n+\n+\t\tcurrent = m->pack_names[mid];\n+\t\tcmp = strcmp(idx_name, current);\n+\t\tif (!cmp)\n+\t\t\treturn 1;\n+\t\tif (cmp > 0) {\n+\t\t\tfirst = mid + 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlast = mid;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n {\n \tstruct multi_pack_index *m = r->objects->multi_pack_index;\n@@ -326,6 +366,7 @@ struct pack_list {\n \tuint32_t alloc_list;\n \tuint32_t alloc_names;\n \tsize_t pack_name_concat_len;\n+\tstruct multi_pack_index *m;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -334,6 +375,9 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \tstruct pack_list *packs = (struct pack_list *)data;\n \n \tif (ends_with(file_name, \".idx\")) {\n+\t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n+\t\t\treturn;\n+\n \t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n \t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n \n@@ -418,6 +462,23 @@ static int midx_oid_compare(const void *_a, const void *_b)\n \treturn a->pack_int_id - b->pack_int_id;\n }\n \n+static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n+\t\t\t\t uint32_t *pack_perm,\n+\t\t\t\t struct pack_midx_entry *e,\n+\t\t\t\t uint32_t pos)\n+{\n+\tif (pos >= m->num_objects)\n+\t\treturn 1;\n+\n+\tnth_midxed_object_oid(&e->oid, m, pos);\n+\te->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];\n+\te->offset = nth_midxed_offset(m, pos);\n+\n+\t/* consider objects in midx to be from \"old\" packs */\n+\te->pack_mtime = 0;\n+\treturn 0;\n+}\n+\n static void fill_pack_entry(uint32_t pack_int_id,\n \t\t\t struct packed_git *p,\n \t\t\t uint32_t cur_object,\n@@ -443,7 +504,8 @@ static void fill_pack_entry(uint32_t pack_int_id,\n * Copy only the de-duplicated entries (selected by most-recent modified time\n * of a packfile containing the object).\n */\n-static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n+static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n+\t\t\t\t\t\t struct packed_git **p,\n \t\t\t\t\t\t uint32_t *perm,\n \t\t\t\t\t\t uint32_t nr_packs,\n \t\t\t\t\t\t uint32_t *nr_objects)\n@@ -452,8 +514,9 @@ static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n \tstruct pack_midx_entry *entries_by_fanout = NULL;\n \tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\tuint32_t start_pack = m ? m->num_packs : 0;\n \n-\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++)\n+\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n \t\ttotal_objects += p[cur_pack]->num_objects;\n \n \t/*\n@@ -470,7 +533,23 @@ static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tuint32_t nr_fanout = 0;\n \n-\t\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\tif (m) {\n+\t\t\tuint32_t start = 0, end;\n+\n+\t\t\tif (cur_fanout)\n+\t\t\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n+\t\t\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n+\n+\t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n+\t\t\t\tnth_midxed_pack_midx_entry(m, perm,\n+\t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n+\t\t\t\t\t\t\t cur_object);\n+\t\t\t\tnr_fanout++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\t\tuint32_t start = 0, end;\n \n \t\t\tif (cur_fanout)\n@@ -666,16 +745,34 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n+\tpacks.m = load_multi_pack_index(object_dir);\n+\n \tpacks.nr = 0;\n-\tpacks.alloc_list = 16;\n-\tpacks.alloc_names = 16;\n+\tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n+\tpacks.alloc_names = packs.alloc_list;\n \tpacks.list = NULL;\n+\tpacks.names = NULL;\n \tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n \tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n+\tif (packs.m) {\n+\t\tfor (i = 0; i < packs.m->num_packs; i++) {\n+\t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n+\t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n+\n+\t\t\tpacks.list[packs.nr] = NULL;\n+\t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n+\t\t\tpacks.nr++;\n+\t\t}\n+\t}\n+\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n+\tif (packs.m && packs.nr == packs.m->num_packs)\n+\t\tgoto cleanup;\n+\n \tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n@@ -683,7 +780,8 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n-\tentries = get_sorted_entries(packs.list, pack_perm, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n+\n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n@@ -695,6 +793,9 @@ int write_midx_file(const char *object_dir)\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n+\tif (packs.m)\n+\t\tclose_midx(packs.m);\n+\n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n@@ -786,6 +887,7 @@ int write_midx_file(const char *object_dir)\n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n \n+cleanup:\n \tfor (i = 0; i < packs.nr; i++) {\n \t\tif (packs.list[i]) {\n \t\t\tclose_pack(packs.list[i]);\n@@ -797,5 +899,7 @@ int write_midx_file(const char *object_dir)\n \tfree(packs.list);\n \tfree(packs.names);\n \tfree(entries);\n+\tfree(pack_perm);\n+\tfree(midx_name);\n \treturn 0;\n }\ndiff --git a/midx.h b/midx.h\nindex f7c2ec7893..5faffb7bc6 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -11,6 +11,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n);\n int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351734","messageId":"20180706005321.124643-25-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 24/24] midx: clear midx on repack","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:21Z","receivedAt":"2018-07-06T00:54:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If a 'git repack' command replaces existing packfiles, then we must\nclear the existing multi-pack-index before moving the packfiles it\nreferences.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/repack.c | 8 ++++++++\n midx.c | 12 ++++++++++++\n midx.h | 1 +\n 3 files changed, 21 insertions(+)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 6c636e159e..66a7d8e8ea 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -8,6 +8,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"argv-array.h\"\n+#include \"midx.h\"\n \n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -174,6 +175,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n \tint local = 0;\n+\tint midx_cleared = 0;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -340,6 +342,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\t}\n \n+\t\t\tif (!midx_cleared) {\n+\t\t\t\t/* if we move a packfile, it will invalidated the midx */\n+\t\t\t\tclear_midx_file(get_object_directory());\n+\t\t\t\tmidx_cleared = 1;\n+\t\t\t}\n+\n \t\t\tfname_old = mkpathdup(\"%s/old-%s%s\", packdir,\n \t\t\t\t\t\titem->string, exts[ext].name);\n \t\t\tif (file_exists(fname_old))\ndiff --git a/midx.c b/midx.c\nindex 7c00b02436..8afd08f3fe 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -903,3 +903,15 @@ int write_midx_file(const char *object_dir)\n \tfree(midx_name);\n \treturn 0;\n }\n+\n+void clear_midx_file(const char *object_dir)\n+{\n+\tchar *midx = get_midx_filename(object_dir);\n+\n+\tif (remove_path(midx)) {\n+\t\tUNLEAK(midx);\n+\t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n+\t}\n+\n+\tfree(midx);\n+}\ndiff --git a/midx.h b/midx.h\nindex 5faffb7bc6..5a42cbed1d 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -15,5 +15,6 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n+void clear_midx_file(const char *object_dir);\n \n #endif\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351735","messageId":"20180706005321.124643-22-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 21/24] midx: use midx in approximate_object_count","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:18Z","receivedAt":"2018-07-06T00:54:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n packfile.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/packfile.c b/packfile.c\nindex c0eb5ac885..97e7812b6b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -861,10 +861,13 @@ unsigned long approximate_object_count(void)\n {\n \tif (!the_repository->objects->approximate_object_count_valid) {\n \t\tunsigned long count;\n+\t\tstruct multi_pack_index *m;\n \t\tstruct packed_git *p;\n \n \t\tprepare_packed_git(the_repository);\n \t\tcount = 0;\n+\t\tfor (m = get_multi_pack_index(the_repository); m; m = m->next)\n+\t\t\tcount += m->num_objects;\n \t\tfor (p = the_repository->objects->packed_git; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351736","messageId":"20180706005321.124643-23-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 22/24] midx: prevent duplicate packfile loads","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:19Z","receivedAt":"2018-07-06T00:54:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index, when present, tracks the existence of objects and\ntheir offsets within a list of packfiles. This allows us to use the\nmulti-pack-index for object lookups, abbreviations, and object counts.\n\nWhen the multi-pack-index tracks a packfile, then we do not need to add\nthat packfile to the packed_git linked list or the MRU list.\n\nWe still need to load the packfiles that are not tracked by the\nmulti-pack-index.\n\nSigned-off-by: Derrick Stolee \n---\n packfile.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/packfile.c b/packfile.c\nindex 97e7812b6b..2c819a0ad8 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -795,6 +795,7 @@ struct prepare_pack_data {\n \tstruct repository *r;\n \tstruct string_list *garbage;\n \tint local;\n+\tstruct multi_pack_index *m;\n };\n \n static void prepare_pack(const char *full_name, size_t full_name_len,\n@@ -805,6 +806,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\")) {\n+\t\tif (data->m && midx_contains_pack(data->m, file_name))\n+\t\t\treturn;\n \t\t/* Don't reopen a pack we already have. */\n \t\tfor (p = data->r->objects->packed_git; p; p = p->next) {\n \t\t\tsize_t len;\n@@ -839,6 +842,12 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n \tstruct prepare_pack_data data;\n \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \n+\tdata.m = r->objects->multi_pack_index;\n+\n+\t/* look for the multi-pack-index for this object directory */\n+\twhile (data.m && strcmp(data.m->object_dir, objdir))\n+\t\tdata.m = data.m->next;\n+\n \tdata.r = r;\n \tdata.garbage = &garbage;\n \tdata.local = local;\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351737","messageId":"20180706005321.124643-24-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 23/24] packfile: skip loading index if in multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:20Z","receivedAt":"2018-07-06T00:54:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n packfile.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 2c819a0ad8..e6ecf12ab5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -469,8 +469,19 @@ static int open_packed_git_1(struct packed_git *p)\n \tssize_t read_result;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \n-\tif (!p->index_data && open_pack_index(p))\n-\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\tif (!p->index_data) {\n+\t\tstruct multi_pack_index *m;\n+\t\tconst char *pack_name = strrchr(p->pack_name, '/');\n+\n+\t\tfor (m = the_repository->objects->multi_pack_index;\n+\t\t m; m = m->next) {\n+\t\t\tif (midx_contains_pack(m, pack_name))\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!m && open_pack_index(p))\n+\t\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\t}\n \n \tif (!pack_max_fds) {\n \t\tunsigned int max_fds = get_max_fd_limit();\n@@ -521,6 +532,10 @@ static int open_packed_git_1(struct packed_git *p)\n \t\t\t\" supported (try upgrading GIT to a newer version)\",\n \t\t\tp->pack_name, ntohl(hdr.hdr_version));\n \n+\t/* Skip index checking if in multi-pack-index */\n+\tif (!p->index_data)\n+\t\treturn 0;\n+\n \t/* Verify the pack matches its index. */\n \tif (p->num_objects != ntohl(hdr.hdr_entries))\n \t\treturn error(\"packfile %s claims to have %\"PRIu32\" objects\"\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351738","messageId":"20180706005321.124643-17-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 16/24] config: create core.multiPackIndex setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:13Z","receivedAt":"2018-07-06T00:54:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The core.multiPackIndex config setting controls the multi-pack-\nindex (MIDX) feature. If false, the setting will disable all reads\nfrom the multi-pack-index file.\n\nAdd comparison commands in t5319-multi-pack-index.sh to check\ntypical Git behavior remains the same as the config setting is turned\non and off. This currently includes 'git rev-list' and 'git log'\ncommands to trigger several object database reads.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 4 +++\n cache.h | 1 +\n config.c | 5 ++++\n environment.c | 1 +\n t/t5319-multi-pack-index.sh | 56 +++++++++++++++++++++++++++++++------\n 5 files changed, 58 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ab641bf5a9..ab895ebb32 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -908,6 +908,10 @@ core.commitGraph::\n \tEnable git commit graph feature. Allows reading from the\n \tcommit-graph file.\n \n+core.multiPackIndex::\n+\tUse the multi-pack-index file to track multiple packfiles using a\n+\tsingle index. See linkgit:technical/multi-pack-index[1].\n+\n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n \tlinkgit:git-read-tree[1] for more information.\ndiff --git a/cache.h b/cache.h\nindex 89a107a7f7..d12aa49710 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -814,6 +814,7 @@ extern char *git_replace_ref_base;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_commit_graph;\n+extern int core_multi_pack_index;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex fbbf0f8e9f..95d8da4243 100644\n--- a/config.c\n+++ b/config.c\n@@ -1313,6 +1313,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.multipackindex\")) {\n+\t\tcore_multi_pack_index = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.sparsecheckout\")) {\n \t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 2a6de2330b..b9bc919cdb 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -67,6 +67,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_commit_graph;\n+int core_multi_pack_index;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex ae6c9d4d02..fc582c9a59 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,6 +3,8 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n+objdir=.git/objects\n+\n midx_read_expect () {\n \tNUM_PACKS=$1\n \tNUM_OBJECTS=$2\n@@ -61,12 +63,42 @@ test_expect_success 'write midx with one v1 pack' '\n \tmidx_read_expect 1 17 4 .\n '\n \n+midx_git_two_modes() {\n+\tgit -c core.multiPackIndex=false $1 >expect &&\n+\tgit -c core.multiPackIndex=true $1 >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx() {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\"\n+\t'\n+}\n+\n test_expect_success 'write midx with one v2 pack' '\n-\tgit pack-objects --index-version=2,0x40 pack/test expect &&\n+\tgit -c core.multiPackIndex=true $1 >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx() {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\"\n+\t'\n+}\n+\n+compare_results_with_midx \"one v2 pack\"\n+\n test_expect_success 'Add more objects' '\n \tfor i in $(test_seq 6 10)\n \tdo\n@@ -92,11 +124,13 @@ test_expect_success 'Add more objects' '\n '\n \n test_expect_success 'write midx with two packs' '\n-\tgit pack-objects --index-version=1 pack/test-2 obj-list &&\n \t\tgit update-ref HEAD $commit &&\n-\t\tgit pack-objects --index-version=2 pack/test-pack []\n corrupt_data() {\n \tfile=$1\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351739","messageId":"20180706005321.124643-18-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 17/24] midx: prepare midxed_git struct","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:14Z","receivedAt":"2018-07-06T00:54:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 22 ++++++++++++++++++++++\n midx.h | 3 +++\n object-store.h | 9 +++++++++\n packfile.c | 6 +++++-\n 4 files changed, 39 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex cc35abe7a2..d5a61c0c53 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -180,6 +180,28 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \treturn NULL;\n }\n \n+int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n+{\n+\tstruct multi_pack_index *m = r->objects->multi_pack_index;\n+\tstruct multi_pack_index *m_search;\n+\n+\tif (!core_multi_pack_index)\n+\t\treturn 0;\n+\n+\tfor (m_search = m; m_search; m_search = m_search->next)\n+\t\tif (!strcmp(object_dir, m_search->object_dir))\n+\t\t\treturn 1;\n+\n+\tr->objects->multi_pack_index = load_multi_pack_index(object_dir);\n+\n+\tif (r->objects->multi_pack_index) {\n+\t\tr->objects->multi_pack_index->next = m;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static size_t write_midx_header(struct hashfile *f,\n \t\t\t\tunsigned char num_chunks,\n \t\t\t\tuint32_t num_packs)\ndiff --git a/midx.h b/midx.h\nindex 2d83dd9ec1..731ad6f094 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -1,9 +1,12 @@\n #ifndef __MIDX_H__\n #define __MIDX_H__\n \n+#include \"repository.h\"\n+\n struct multi_pack_index;\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n+int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n \ndiff --git a/object-store.h b/object-store.h\nindex 07bcc80e02..7d67ad7aa9 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -85,6 +85,8 @@ struct packed_git {\n };\n \n struct multi_pack_index {\n+\tstruct multi_pack_index *next;\n+\n \tint fd;\n \n \tconst unsigned char *data;\n@@ -126,6 +128,13 @@ struct raw_object_store {\n \t */\n \tstruct oidmap *replace_map;\n \n+\t/*\n+\t * private data\n+\t *\n+\t * should only be accessed directly by packfile.c and midx.c\n+\t */\n+\tstruct multi_pack_index *multi_pack_index;\n+\n \t/*\n \t * private data\n \t *\ndiff --git a/packfile.c b/packfile.c\nindex 3d652212c6..5d4493dbf4 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -15,6 +15,7 @@\n #include \"tree-walk.h\"\n #include \"tree.h\"\n #include \"object-store.h\"\n+#include \"midx.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -935,10 +936,13 @@ static void prepare_packed_git(struct repository *r)\n \n \tif (r->objects->packed_git_initialized)\n \t\treturn;\n+\tprepare_multi_pack_index_one(r, r->objects->objectdir);\n \tprepare_packed_git_one(r, r->objects->objectdir, 1);\n \tprepare_alt_odb(r);\n-\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next)\n+\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\tprepare_multi_pack_index_one(r, alt->path);\n \t\tprepare_packed_git_one(r, alt->path, 0);\n+\t}\n \trearrange_packed_git(r);\n \tprepare_packed_git_mru(r);\n \tr->objects->packed_git_initialized = 1;\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351740","messageId":"20180706005321.124643-8-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v3 07/24] multi-pack-index: expand test data","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-06T00:53:04Z","receivedAt":"2018-07-06T00:54:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As we build the multi-pack-index file format, we want to test the format\non real repoasitories. Add tests to t5319-multi-pack-index.sh that\ncreate repository data including multiple packfiles with both version 1\nand version 2 formats.\n\nThe current 'git multi-pack-index write' command will always write the\nsame file with no \"real\" data. This will be expanded in future commits,\nalong with the test expectations.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 99 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 99 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 2ecc369529..1be7be02b8 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -13,9 +13,108 @@ midx_read_expect () {\n }\n \n test_expect_success 'write midx with no packs' '\n+\ttest_when_finished rm -f pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n \ttest_path_is_file pack/multi-pack-index &&\n \tmidx_read_expect\n '\n \n+test_expect_success 'create objects' '\n+\tfor i in $(test_seq 1 5)\n+\tdo\n+\t\tiii=$(printf '%03i' $i)\n+\t\ttest-tool genrandom \"bar\" 200 >wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >>wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 >deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >>deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >>deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\ti=$(expr $i + 1) || return 1\n+\tdone &&\n+\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree obj-list &&\n+\tgit update-ref HEAD $commit\n+'\n+\n+test_expect_success 'write midx with one v1 pack' '\n+\tpack=$(git pack-objects --index-version=1 pack/test wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >>wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 >deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >>deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >>deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\ti=$(expr $i + 1) || return 1\n+\tdone &&\n+\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree -p HEADobj-list2 &&\n+\tgit update-ref HEAD $commit\n+'\n+\n+test_expect_success 'write midx with two packs' '\n+\tgit pack-objects --index-version=1 pack/test-2 wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >>wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 >deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >>deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >>deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\t\tgit update-index --add file_101 &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcommit=$(git commit-tree $tree -p HEADobj-list &&\n+\t\tgit update-ref HEAD $commit &&\n+\t\tgit pack-objects --index-version=2 test-pack \n---\n midx.c | 17 +++++++++++++++++\n object-store.h | 1 +\n t/helper/test-read-midx.c | 5 +++++\n t/t5319-multi-pack-index.sh | 7 ++++++-\n 4 files changed, 29 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex f78c161422..ffe29af65d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -37,6 +37,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tuint32_t hash_version;\n \tchar *midx_name = get_midx_filename(object_dir);\n \tuint32_t i;\n+\tconst char *cur_pack_name;\n \n \tfd = git_open(midx_name);\n \n@@ -117,6 +118,22 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tif (!m->chunk_pack_names)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n \n+\tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n+\n+\tcur_pack_name = (const char *)m->chunk_pack_names;\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tm->pack_names[i] = cur_pack_name;\n+\n+\t\tcur_pack_name += strlen(cur_pack_name) + 1;\n+\n+\t\tif (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0) {\n+\t\t\terror(_(\"multi-pack-index pack names out of order: '%s' before '%s'\"),\n+\t\t\t m->pack_names[i - 1],\n+\t\t\t m->pack_names[i]);\n+\t\t\tgoto cleanup_fail;\n+\t\t}\n+\t}\n+\n \treturn m;\n \n cleanup_fail:\ndiff --git a/object-store.h b/object-store.h\nindex c87d051849..88169b33e9 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -99,6 +99,7 @@ struct multi_pack_index {\n \n \tconst unsigned char *chunk_pack_names;\n \n+\tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex a9232d8219..0b53a9e8b5 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -9,6 +9,7 @@\n \n static int read_midx_file(const char *object_dir)\n {\n+\tuint32_t i;\n \tstruct multi_pack_index *m = load_multi_pack_index(object_dir);\n \n \tif (!m)\n@@ -27,6 +28,10 @@ static int read_midx_file(const char *object_dir)\n \n \tprintf(\"\\n\");\n \n+\tprintf(\"packs:\\n\");\n+\tfor (i = 0; i < m->num_packs; i++)\n+\t\tprintf(\"%s\\n\", m->pack_names[i]);\n+\n \tprintf(\"object_dir: %s\\n\", m->object_dir);\n \n \treturn 0;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex f458758945..4610352b69 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -8,8 +8,13 @@ midx_read_expect () {\n \tcat >expect <<-EOF\n \theader: 4d494458 1 1 $NUM_PACKS\n \tchunks: pack_names\n-\tobject_dir: .\n+\tpacks:\n \tEOF\n+\tif [ $NUM_PACKS -ge 1 ]\n+\tthen\n+\t\tls pack/ | grep idx | sort >> expect\n+\tfi\n+\tprintf \"object_dir: .\\n\" >>expect &&\n \ttest-tool read-midx . >actual &&\n \ttest_cmp expect actual\n }\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"351750","messageId":"CAPig+cQS-sFHtgMZdaW2tTPSLAo=pytFJ5_zhVvUU1=8Mq74ng@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-4-dstolee@microsoft.com","subject":"Re: [PATCH v3 03/24] multi-pack-index: add builtin","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T03:54:08Z","receivedAt":"2018-07-06T03:54:23Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:53 PM Derrick Stolee wrote:\n> This new 'git multi-pack-index' builtin will be the plumbing access\n> for writing, reading, and checking multi-pack-index files. The\n> initial implementation is a no-op.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> +SYNOPSIS\n> +--------\n> +'git multi-pack-index' [--object-dir ]\n\nIn Git documentation, this is more typically written: [--object-dir=]\n\n> +OPTIONS\n> +-------\n> +--object-dir ::\n\nDitto: --object-dir=::\n\n> + Use given directory for the location of Git objects. We check\n> + `/packs/multi-pack-index` for the current MIDX file, and\n> + `/packs` for the pack-files to index.\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> @@ -0,0 +1,38 @@\n> +static char const * const builtin_multi_pack_index_usage[] = {\n> + N_(\"git multi-pack-index [--object-dir ]\"),\n\nLikewise.\n\n> +int cmd_multi_pack_index(int argc, const char **argv,\n> + const char *prefix)\n> +{\n> + static struct option builtin_multi_pack_index_options[] = {\n> + OPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n> + N_(\"The object directory containing set of packfile and pack-index pairs\")),\n\nIt's more typical not to capitalize these. Also, keep them short, if\npossible, so perhaps drop \"The\".\n\n> + OPT_END(),\n> + };\n> +\n> + if (argc == 2 && !strcmp(argv[1], \"-h\"))\n> + usage_with_options(builtin_multi_pack_index_usage,\n> + builtin_multi_pack_index_options);\n\nUnless you are planning on adding a short \"-h \" option\nlater in the series, then you can do away with this conditional\naltogether since the below parse_options() will give you \"-h\" as help\nfor free.\n\n> + git_config(git_default_config, NULL);\n> +\n> + argc = parse_options(argc, argv, prefix,\n> + builtin_multi_pack_index_options,\n> + builtin_multi_pack_index_usage, 0);\n"},{"id":"351751","messageId":"CAPig+cSfdbprGHajYZZWT20fW6_+_YN2SAtM86QswaApPHkqPQ@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-5-dstolee@microsoft.com","subject":"Re: [PATCH v3 04/24] multi-pack-index: add 'write' verb","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T04:07:47Z","receivedAt":"2018-07-06T04:08:02Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:53 PM Derrick Stolee wrote:\n> In anticipation of writing multi-pack-indexes, add a\n> 'git multi-pack-index write' subcommand and send the options to a\n> write_midx_file() method.\n\nSince the 'write' command is a no-op at this point, perhaps say so in\nthe commit message. Something like:\n\n ... add a skeleton 'git multi-pack-index write' subcommand,\n which will be fleshed-out by a later commit.\n\nThe bit about sending options to write_midx_file() is superfluous;\nit's a mere implementation detail which is clearly seen by reading the\npatch.\n\n> Also create a basic test file that tests\n> the 'write' subcommand.\n\nMaybe: s/file/script\n\nAnd, as above, perhaps mention that this is a _skeleton_ test script\nso as to avoid confusing readers into thinking that something\nsignificant is happening at this stage.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> +* Write a MIDX file for the packfiles in an alternate.\n\nIn an alternate what?\n\n> +-----------------------------------------------\n> +$ git multi-pack-index --object-dir write\n> +-----------------------------------------------\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> @@ -2,9 +2,10 @@\n> static char const * const builtin_multi_pack_index_usage[] = {\n> - N_(\"git multi-pack-index [--object-dir ]\"),\n> + N_(\"git multi-pack-index [--object-dir ] [write]\"),\n\nIs there going to be some default behavior when no verb is provided?\nThe below implementation seems to suggest that the verb is required,\nso this probably ought to be typeset as:\n\n git multi-pack-index [--object-dir=] write\n\nLater, when you add more (mutually exclusive) verbs, change the typesetting to:\n\n git multi-pack-index [--object-dir=] (write|...|...)\n\nAlternately, just use:\n\n git multi-pack-index [--object-dir=] \n\n> @@ -34,5 +35,12 @@ int cmd_multi_pack_index(int argc, const char **argv,\n> + if (argc == 0)\n> + usage_with_options(builtin_multi_pack_index_usage,\n> + builtin_multi_pack_index_options);\n> +\n> + if (!strcmp(argv[0], \"write\"))\n> + return write_midx_file(opts.object_dir);\n> +\n> return 0;\n\nThis should be throwing an error when an unrecognized verb is provided.\n\nIt also should be throwing an error when 'write' is given too many\narguments (which, at this point, appears to be 0).\n"},{"id":"351752","messageId":"CAPig+cSntb3=YTBm67+4T1FgLCi585CnUuN__DE2mdo4y2n98A@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-7-dstolee@microsoft.com","subject":"Re: [PATCH v3 06/24] multi-pack-index: load into memory","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T04:19:10Z","receivedAt":"2018-07-06T04:19:25Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:53 PM Derrick Stolee wrote:\n> Create a new multi_pack_index struct for loading multi-pack-indexes into\n> memory. Create a test-tool builtin for reading basic information about\n> that multi-pack-index to verify the correct data is written.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\n> @@ -0,0 +1,34 @@\n> +/*\n> + * test-mktemp.c: code to exercise the creation of temporary files\n> + */\n\nMeh. Copy/paste botch.\n\n> +static int read_midx_file(const char *object_dir)\n> +{\n> + struct multi_pack_index *m = load_multi_pack_index(object_dir);\n> +\n> + if (!m)\n> + return 0;\n\nShould this 'return 0' be a die() or BUG() or something?\n\n> + printf(\"header: %08x %d %d %d\\n\",\n> + m->signature,\n> + m->version,\n> + m->num_chunks,\n> + m->num_packs);\n> +\n> + printf(\"object_dir: %s\\n\", m->object_dir);\n> +\n> + return 0;\n> +}\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> @@ -3,9 +3,19 @@\n> +midx_read_expect () {\n> + cat >expect <<-EOF\n\nI guess you're planning on interpolating some variables here in a\nlater patch, which is why you used -EOF rather than -\\EOF?\n\n> + header: 4d494458 1 0 0\n> + object_dir: .\n> + EOF\n> + test-tool read-midx . >actual &&\n> + test_cmp expect actual\n> +}\n> +\n> test_expect_success 'write midx with no packs' '\n> git multi-pack-index --object-dir=. write &&\n> - test_path_is_file pack/multi-pack-index\n> + test_path_is_file pack/multi-pack-index &&\n> + midx_read_expect\n> '\n\nKind of a do-nothing change. I wonder if this step would better be\ndelayed until a later patch. (Not necessarily a big deal.)\n"},{"id":"351753","messageId":"CAPig+cTO=UPuCRUAAWfBwaNhRD+=dgSwpgEB2fQJuWyFEd7Aew@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-8-dstolee@microsoft.com","subject":"Re: [PATCH v3 07/24] multi-pack-index: expand test data","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T04:36:55Z","receivedAt":"2018-07-06T04:37:10Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n> multi-pack-index: expand test data\n\nSince this patch is touching only t5319, a more typical title would be:\n\n t5319: expand test data\n\n> As we build the multi-pack-index file format, we want to test the format\n> on real repoasitories. Add tests to t5319-multi-pack-index.sh that\n\ns/repoasitories/repositories/\n\nAnd, since the title now mentions t5319, this can become simply:\n\n Add tests that...\n\n> create repository data including multiple packfiles with both version 1\n> and version 2 formats.\n>\n> The current 'git multi-pack-index write' command will always write the\n> same file with no \"real\" data. This will be expanded in future commits,\n> along with the test expectations.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> @@ -13,9 +13,108 @@ midx_read_expect () {\n> test_expect_success 'write midx with no packs' '\n> + test_when_finished rm -f pack/multi-pack-index &&\n\nShould this test_when_finished() have been added in an earlier patch?\nIt seems out of place here.\n\n> git multi-pack-index --object-dir=. write &&\n> test_path_is_file pack/multi-pack-index &&\n> midx_read_expect\n> '\n>\n> +test_expect_success 'create objects' '\n> + for i in $(test_seq 1 5)\n> + do\n> + iii=$(printf '%03i' $i)\n> + test-tool genrandom \"bar\" 200 >wide_delta_$iii &&\n> + test-tool genrandom \"baz $iii\" 50 >>wide_delta_$iii &&\n\nAlternately:\n\n {\n test-tool genrandom \"bar\" 200 &&\n test-tool genrandom \"baz $iii\" 50\n } >wide_delta_$iii &&\n\nwhich makes it easier to see at a glance that both commands are\npopulating the same file. Same comment for the other files. (Not worth\na re-roll.)\n\n> + test-tool genrandom \"foo\"$i 100 >deep_delta_$iii &&\n> + test-tool genrandom \"foo\"$(expr $i + 1) 100 >>deep_delta_$iii &&\n> + test-tool genrandom \"foo\"$(expr $i + 2) 100 >>deep_delta_$iii &&\n\nOr, just use POSIX arithmetic expansion:\n\n $(( $i + 1 ))\n\n> + echo $iii >file_$iii &&\n> + test-tool genrandom \"$iii\" 8192 >>file_$iii &&\n> + git update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n> + i=$(expr $i + 1) || return 1\n\nDitto, POSIX arithmetic expansion:\n\n i=$(( $i + 1 ))\n\n(Not worth a re-roll.)\n\n> + done &&\n> + { echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n> + git update-index --add file_101 &&\n> + tree=$(git write-tree) &&\n> + commit=$(git commit-tree $tree + echo $tree &&\n> + git ls-tree $tree | sed -e \"s/.* \\\\([0-9a-f]*\\\\) .*/\\\\1/\"\n> + } >obj-list &&\n\nPerhaps indent the content of the {...} block?\n\n> + git update-ref HEAD $commit\n> +'\n> +\n> +test_expect_success 'write midx with one v1 pack' '\n> + pack=$(git pack-objects --index-version=1 pack/test + test_when_finished rm pack/test-$pack.pack pack/test-$pack.idx pack/multi-pack-index &&\n> + git multi-pack-index --object-dir=. write &&\n> + midx_read_expect\n> +'\n\nIt's odd to see all these tests ending by creating an 'expect' file\nbut not actually doing anything with that file.\n\n> +test_expect_success 'Add more objects' '\n\ns/Add/add/\n\n> + for i in $(test_seq 6 10)\n> + do\n> + iii=$(printf '%03i' $i)\n> + test-tool genrandom \"bar\" 200 >wide_delta_$iii &&\n> + test-tool genrandom \"baz $iii\" 50 >>wide_delta_$iii &&\n> + test-tool genrandom \"foo\"$i 100 >deep_delta_$iii &&\n> + test-tool genrandom \"foo\"$(expr $i + 1) 100 >>deep_delta_$iii &&\n> + test-tool genrandom \"foo\"$(expr $i + 2) 100 >>deep_delta_$iii &&\n> + echo $iii >file_$iii &&\n> + test-tool genrandom \"$iii\" 8192 >>file_$iii &&\n> + git update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n> + i=$(expr $i + 1) || return 1\n> + done &&\n> + { echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n> + git update-index --add file_101 &&\n> + tree=$(git write-tree) &&\n> + commit=$(git commit-tree $tree -p HEAD + echo $tree &&\n> + git ls-tree $tree | sed -e \"s/.* \\\\([0-9a-f]*\\\\) .*/\\\\1/\"\n> + } >obj-list2 &&\n> + git update-ref HEAD $commit\n> +'\n\nThere seems to be a fair bit of duplication in these tests which\ncreate objects. Is it possible to factor out some of this code into a\nshell function?\n\n> +test_expect_success 'write midx with two packs' '\n> + git pack-objects --index-version=1 pack/test-2 + git multi-pack-index --object-dir=. write &&\n> + midx_read_expect\n> +'\n> +\n> +test_expect_success 'Add more packs' '\n\ns/Add/add/\n\n> + for j in $(test_seq 1 10)\n> + do\n> + [...]\n> + done\n> +'\n"},{"id":"351754","messageId":"CAPig+cQ2cz8H9qYn09g1o-ZAVQc98Z=hD9R0c7pfha257BsX3w@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-12-dstolee@microsoft.com","subject":"Re: [PATCH v3 11/24] midx: read pack names into array","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T04:58:13Z","receivedAt":"2018-07-06T04:58:28Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> @@ -8,8 +8,13 @@ midx_read_expect () {\n> cat >expect <<-EOF\n\nBroken &&-chain.\n\n> header: 4d494458 1 1 $NUM_PACKS\n> chunks: pack_names\n> - object_dir: .\n> + packs:\n> EOF\n> + if [ $NUM_PACKS -ge 1 ]\n\nOn this project, use 'test' rather than '['.\n\n> + then\n> + ls pack/ | grep idx | sort >> expect\n> + fi\n\nBroken &&-chain.\n\n> + printf \"object_dir: .\\n\" >>expect &&\n\nAll this code building up 'expect' could be in a {...} block to make\nit clearer and less noisy:\n\n {\n cat <<-EOF &&\n ...\n EOF\n if test $NUM_PACKS -ge 1\n then\n ls -1 pack/ | ...\n fi &&\n printf \"...\"\n } >expect &&\n\nAnd, some pointless bike-shedding while here: perhaps dashes instead\nof underlines? \"pack-names\", \"object-dir\"\n\n> test-tool read-midx . >actual &&\n> test_cmp expect actual\n> }\n"},{"id":"351755","messageId":"CAPig+cRK0GWVueJgM-DKSFSJxiE5OhN98F8ZXkBDnLwrVSDFMA@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-14-dstolee@microsoft.com","subject":"Re: [PATCH v3 13/24] midx: write object ids in a chunk","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T05:04:20Z","receivedAt":"2018-07-06T05:04:34Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/midx.c b/midx.c\n> @@ -18,9 +18,10 @@\n> @@ -384,6 +391,32 @@ static size_t write_midx_pack_names(struct hashfile *f,\n> +static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n> + struct pack_midx_entry *objects,\n> + uint32_t nr_objects)\n> +{\n> + struct pack_midx_entry *list = objects;\n> + uint32_t i;\n> + size_t written = 0;\n> +\n> + for (i = 0; i < nr_objects; i++) {\n> + struct pack_midx_entry *obj = list++;\n> +\n> + if (i < nr_objects - 1) {\n> + struct pack_midx_entry *next = list;\n> + if (oidcmp(&obj->oid, &next->oid) >= 0)\n> + BUG(\"OIDs not in order: %s >= %s\",\n> + oid_to_hex(&obj->oid),\n> + oid_to_hex(&next->oid));\n\nThe above two lines are arguments to BUG(), thus should be indented more.\n\n> + }\n> +\n"},{"id":"351756","messageId":"CAPig+cTg9QqxWpOPefuy4KcP3TTQuHted4m4kHThgSG-ZQHuog@mail.gmail.com","threadId":"48668","inReplyTo":"CAPig+cSntb3=YTBm67+4T1FgLCi585CnUuN__DE2mdo4y2n98A@mail.gmail.com","subject":"Re: [PATCH v3 06/24] multi-pack-index: load into memory","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T05:18:19Z","receivedAt":"2018-07-06T05:18:33Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Jul 6, 2018 at 12:19 AM Eric Sunshine wrote:\n> On Thu, Jul 5, 2018 at 8:53 PM Derrick Stolee wrote:\n> > +midx_read_expect () {\n> > + cat >expect <<-EOF\n> > + header: 4d494458 1 0 0\n> > + object_dir: .\n> > + EOF\n> > + test-tool read-midx . >actual &&\n> > + test_cmp expect actual\n> > +}\n> > +\n> > test_expect_success 'write midx with no packs' '\n> > git multi-pack-index --object-dir=. write &&\n> > - test_path_is_file pack/multi-pack-index\n> > + test_path_is_file pack/multi-pack-index &&\n> > + midx_read_expect\n>\n> Kind of a do-nothing change. I wonder if this step would better be\n> delayed until a later patch. (Not necessarily a big deal.)\n\nNever mind. I missed that midx_read_expect() is comparing the 'expect'\nand 'actual' files, so this is not a do-nothing change. The function\nname may have helped to mislead me (or I was just unobservant or too\nfocused on its creation of the 'expect' file). I wonder if a different\nname would have helped. midx_check(), midx_verify()? Meh.\n"},{"id":"351757","messageId":"CAPig+cRFgLtw-DV4ESnY=Qs6PSqDa2bJozBUOVedS_17KPs-HQ@mail.gmail.com","threadId":"48668","inReplyTo":"CAPig+cTO=UPuCRUAAWfBwaNhRD+=dgSwpgEB2fQJuWyFEd7Aew@mail.gmail.com","subject":"Re: [PATCH v3 07/24] multi-pack-index: expand test data","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T05:20:29Z","receivedAt":"2018-07-06T05:20:42Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Jul 6, 2018 at 12:36 AM Eric Sunshine wrote:\n> On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n> > +test_expect_success 'write midx with one v1 pack' '\n> > + pack=$(git pack-objects --index-version=1 pack/test > + test_when_finished rm pack/test-$pack.pack pack/test-$pack.idx pack/multi-pack-index &&\n> > + git multi-pack-index --object-dir=. write &&\n> > + midx_read_expect\n> > +'\n>\n> It's odd to see all these tests ending by creating an 'expect' file\n> but not actually doing anything with that file.\n\nIgnore this comment. As mentioned in my follow-up to 6/24, I missed\nthe fact that midx_read_expect() is doing more than just creating the\n'expect' file.\n"},{"id":"351758","messageId":"CAPig+cR1BaOTTjg751NO14sAr6UZ0WT_G3Hu59KyiWTASZ8D2A@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-16-dstolee@microsoft.com","subject":"Re: [PATCH v3 15/24] midx: write object offsets","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T05:27:37Z","receivedAt":"2018-07-06T05:27:51Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n> The final pair of chunks for the multi-pack-index file stores the object\n> offsets. We default to using 32-bit offsets as in the pack-index version\n> 1 format, but if there exists an offset larger than 32-bits, we use a\n> trick similar to the pack-index version 2 format by storing all offsets\n> at least 2^31 in a 64-bit table; we use the 32-bit table to point into\n> that 64-bit table as necessary.\n>\n> We only store these 64-bit offsets if necessary, so create a test that\n> manipulates a version 2 pack-index to fake a large offset. This allows\n> us to test that the large offset table is created, but the data does not\n> match the actual packfile offsets. The multi-pack-index offset does match\n> the (corrupted) pack-index offset, so a future feature will compare these\n> offsets during a 'verify' step.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> @@ -6,25 +6,28 @@ test_description='multi-pack-indexes'\n> +# usage: corrupt_data []\n> +corrupt_data() {\n\nStyle: corrupt_data () {\n\n> + file=$1\n> + pos=$2\n> + data=\"${3:-\\0}\"\n> + printf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n> +}\n> +\n> +# Force 64-bit offsets by manipulating the idx file.\n> +# This makes the IDX file _incorrect_ so be careful to clean up after!\n> +test_expect_success 'force some 64-bit offsets with pack-objects' '\n> + mkdir objects64 &&\n> + mkdir objects64/pack &&\n> + pack64=$(git pack-objects --index-version=2,0x40 objects64/pack/test-64 + idx64=objects64/pack/test-64-$pack64.idx &&\n> + chmod u+w $idx64 &&\n\nI guess you don't have to worry about the POSIXPERM prerequisite here\nbecause the file is already writable on Windows, right?\n\n> + corrupt_data $idx64 2899 \"\\02\" &&\n> + midx64=$(git multi-pack-index write --object-dir=objects64) &&\n> + midx_read_expect 1 62 5 objects64 \" large_offsets\"\n> '\n"},{"id":"351759","messageId":"CAPig+cTJuvOjyzkmAj5axAkS3q1UyeV03jQk6-oB==KyzbEK_w@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-17-dstolee@microsoft.com","subject":"Re: [PATCH v3 16/24] config: create core.multiPackIndex setting","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T05:39:02Z","receivedAt":"2018-07-06T05:39:17Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n> The core.multiPackIndex config setting controls the multi-pack-\n> index (MIDX) feature. If false, the setting will disable all reads\n> from the multi-pack-index file.\n>\n> Add comparison commands in t5319-multi-pack-index.sh to check\n> typical Git behavior remains the same as the config setting is turned\n> on and off. This currently includes 'git rev-list' and 'git log'\n> commands to trigger several object database reads.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/cache.h b/cache.h\n> @@ -814,6 +814,7 @@ extern char *git_replace_ref_base;\n> +extern int core_multi_pack_index;\n> diff --git a/config.c b/config.c\n> @@ -1313,6 +1313,11 @@ static int git_default_core_config(const char *var, const char *value)\n> + if (!strcmp(var, \"core.multipackindex\")) {\n> + core_multi_pack_index = git_config_bool(var, value);\n> + return 0;\n> + }\n\nThis is a rather unusual commit. This new configuration is assigned,\nbut it's never actually consulted, which means that it's impossible\nfor it to have any impact on functionality, yet tests are being added\nto check whether it _did_ have any impact on functionality. Confusing.\n\nPatch 17 does consult 'core_multi_pack_index', so it's only at that\npoint that it could have any impact. This situation would be less\nconfusing if you swapped patches 16 and 17 (and, of course, move the\ndeclaration of 'core_multi_pack_index' to patch 17 with a reasonable\ndefault value).\n\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> @@ -61,12 +63,42 @@ test_expect_success 'write midx with one v1 pack' '\n> +midx_git_two_modes() {\n> + git -c core.multiPackIndex=false $1 >expect &&\n> + git -c core.multiPackIndex=true $1 >actual &&\n> + test_cmp expect actual\n> +}\n> +\n> +compare_results_with_midx() {\n> + MSG=$1\n> + test_expect_success \"check normal git operations: $MSG\" '\n> + midx_git_two_modes \"rev-list --objects --all\" &&\n> + midx_git_two_modes \"log --raw\"\n> + '\n> +}\n\nHere, you define midx_git_two_modes() and compare_results_with_midx()...\n\n> test_expect_success 'write midx with one v2 pack' '\n> - git pack-objects --index-version=2,0x40 pack/test - git multi-pack-index --object-dir=. write &&\n> - midx_read_expect 1 17 4 .\n> + git pack-objects --index-version=2,0x40 $objdir/pack/test + git multi-pack-index --object-dir=$objdir write &&\n> + midx_read_expect 1 17 4 $objdir\n> '\n>\n> +midx_git_two_modes() {\n> + git -c core.multiPackIndex=false $1 >expect &&\n> + git -c core.multiPackIndex=true $1 >actual &&\n> + test_cmp expect actual\n> +}\n> +\n> +compare_results_with_midx() {\n> + MSG=$1\n> + test_expect_success \"check normal git operations: $MSG\" '\n> + midx_git_two_modes \"rev-list --objects --all\" &&\n> + midx_git_two_modes \"log --raw\"\n> + '\n> +}\n\n... and here you define both functions again.\n\n> +\n> +compare_results_with_midx \"one v2 pack\"\n"},{"id":"351760","messageId":"CAPig+cREC9Qt7NBjFe3MH2zr_P-aykGJhN4G_diqkm6RaeUJxg@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-18-dstolee@microsoft.com","subject":"Re: [PATCH v3 17/24] midx: prepare midxed_git struct","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T05:41:44Z","receivedAt":"2018-07-06T05:41:58Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n> midx: prepare midxed_git struct\n\nWhat's a \"midxed_git\"? I don't see it in the code anywhere.\n\n> Signed-off-by: Derrick Stolee \n"},{"id":"351761","messageId":"CAPig+cRm0VykJWvMLqaKDDRk1mcG_kDDPeo64jJCoVZ1i+k9wg@mail.gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-25-dstolee@microsoft.com","subject":"Re: [PATCH v3 24/24] midx: clear midx on repack","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-06T05:52:27Z","receivedAt":"2018-07-06T05:54:46Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n> If a 'git repack' command replaces existing packfiles, then we must\n> clear the existing multi-pack-index before moving the packfiles it\n> references.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/repack.c | 8 ++++++++\n> midx.c | 12 ++++++++++++\n> midx.h | 1 +\n> 3 files changed, 21 insertions(+)\n\nThis seems like a pretty important bit of functionality. Is there any\nway to add a test of it to t5319 or would it be too difficult (or not\nimportant enough)?\n"},{"id":"351793","messageId":"xmqq601skw75.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"CAPig+cTVD5OpVnxAqQnO_CkzFRRkZ3YWsRC4ZwUmAELSZzcDKw@mail.gmail.com","subject":"Re: [PATCH v2 06/24] multi-pack-index: load into memory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-07-06T19:20:30Z","receivedAt":"2018-07-06T19:20:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> On Thu, Jul 5, 2018 at 10:20 AM Derrick Stolee wrote:\n>> On 6/25/2018 3:38 PM, Junio C Hamano wrote:\n>> While I don't use substitutions in this patch, I do use them in later\n>> patches. Here is the final version of this method:\n>>\n>> midx_read_expect () {\n>> NUM_PACKS=$1\n>> NUM_OBJECTS=$2\n>> NUM_CHUNKS=$3\n>> EXTRA_CHUNKS=\"$5\"\n>> cat >expect <<-\\EOF\n>> header: 4d494458 1 $NUM_CHUNKS $NUM_PACKS\n>> chunks: pack_names oid_fanout oid_lookup\n>> object_offsets$EXTRA_CHUNKS\n>> num_objects: $NUM_OBJECTS\n>> packs:\n>> EOF\n>>\n>> Using <<-\\EOF causes these substitutions to fail. Is there a different\n>> way I should construct this method?\n>\n> When you need to interpolate variables into the here-doc, use <<-EOF;\n> when you don't, use <<-\\EOF.\n\nI think what was said is \"in an early step there is no need to\ninterpolate but the same here-doc will need substitution in a later\nstep, and that is why I started an early step with a form without\nquoting\", which is different from \"when should I use the form with\nand without quoting?\"\n\nI think a reasonable response would have been \"then please do use\nthe quoted form in the early step to help reviewers to let them know\nthere is not yet any substitutions, and then switch to quote-less form\nat the step that starts needing substitution. That way, reviewers can\nsee the test started to become \"interesting\" by interpolating variable\nbits in the test vector by seeing the line with \"< writes:\n\n> +struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n> +{\n> +\tstruct multi_pack_index *m = NULL;\n> +\tint fd;\n> +\tstruct stat st;\n> +\tsize_t midx_size;\n> +\tvoid *midx_map = NULL;\n> +\tuint32_t hash_version;\n> +\tchar *midx_name = get_midx_filename(object_dir);\n> +\n> +\tfd = git_open(midx_name);\n> +\n> +\tif (fd < 0)\n> +\t\tgoto cleanup_fail;\n> +\tif (fstat(fd, &st)) {\n> +\t\terror_errno(_(\"failed to read %s\"), midx_name);\n> +\t\tgoto cleanup_fail;\n> +\t}\n> +\n> +\tmidx_size = xsize_t(st.st_size);\n> +\n> +\tif (midx_size < MIDX_MIN_SIZE) {\n> +\t\tclose(fd);\n\nWith the use of \"do things normally and jump to cleanup-fail label\"\npattern, I think you do not want the close() here (unless you also\nassign -1 to fd yourself, but that is a pointless workaround).\nAnother goto we see above after fstat() failure correctly omits it.\n\n> +\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name);\n> +\t\tgoto cleanup_fail;\n> +\t}\n> +\n> +\tFREE_AND_NULL(midx_name);\n\nThis correctly calls free-and-null not just free (otherwise we'd\nbreak the cleanup-fail procedure below), which is good.\n\n> +\tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n> +\n> +\tm = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n> +\tstrcpy(m->object_dir, object_dir);\n\nHmph, I thought we had FLEX_ALLOC_*() convenience functions exactly\nfor doing things like this more safely.\n\n> +\tm->fd = fd;\n> +\tm->data = midx_map;\n> +\tm->data_len = midx_size;\n> +\n> +\tm->signature = get_be32(m->data);\n> +\tif (m->signature != MIDX_SIGNATURE) {\n> +\t\terror(_(\"multi-pack-index signature 0x%08x does not match signature 0x%08x\"),\n> +\t\t m->signature, MIDX_SIGNATURE);\n> +\t\tgoto cleanup_fail;\n> +\t}\n> +\n> +\tm->version = m->data[MIDX_BYTE_FILE_VERSION];\n> +\tif (m->version != MIDX_VERSION) {\n> +\t\terror(_(\"multi-pack-index version %d not recognized\"),\n> +\t\t m->version);\n> +\t\tgoto cleanup_fail;\n> +\t}\n> +\n> +\thash_version = m->data[MIDX_BYTE_HASH_VERSION];\n> +\tif (hash_version != MIDX_HASH_VERSION) {\n> +\t\terror(_(\"hash version %u does not match\"), hash_version);\n> +\t\tgoto cleanup_fail;\n> +\t}\n> +\tm->hash_len = MIDX_HASH_LEN;\n> +\n> +\tm->num_chunks = m->data[MIDX_BYTE_NUM_CHUNKS];\n> +\n> +\tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n> +\n> +\treturn m;\n\n> +cleanup_fail:\n> +\t/* no need to check for NULL when freeing */\n\nI wonder who the target reader of this comment is. We certainly are\nnot in the business of C language newbies.\n\nIf this _were_ a commit that looked like this:\n\n\t-\tif (ptr)\n\t-\t\tfree(ptr);\n\t+\t/* no need to check for NULL when freeing */\n\t+\tfree(ptr);\n\nthen it might be more understandable, but it still is wrong (such a\ncomment does not help understanding the new code, which is the only\nthing the people who read the comment sees, without knowing what was\nthere previously---it belongs to the commit log message as a rationale\nto make that change).\n\n> diff --git a/midx.h b/midx.h\n> index dbdbe9f873..2d83dd9ec1 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -1,6 +1,10 @@\n> #ifndef __MIDX_H__\n> #define __MIDX_H__\n> \n> +struct multi_pack_index;\n\nI actually was quite surprised that this struct is defined in\nobject-store.h and not here. It feels the other way around.\n\nThe raw_object_store needs to know that such an in-core structure\nmight exist as an optional feature in an object store, but as an\noptional feature, I suspect that it has a pointer to an instance of\nmulti_pack_index, instead of embedding the struct itself in it, so I\nwould have expected to see an \"I am only telling you that there is a\nstruct with this name, but I am leaving it opaque as you do not have\nany business looking inside the struct yourself. You only need to\nbe aware of the type's existence and a pointer to it so that you can\ncall helpers that know what's inside and that should be sufficient\nfor your needs.\" decl like this in object-store.h and instead an\nactual implementation in here.\n"},{"id":"352224","messageId":"20180711094858.3230-1-szeder.dev@gmail.com","threadId":"48668","inReplyTo":"20180706005321.124643-17-dstolee@microsoft.com","subject":"Re: [PATCH v3 16/24] config: create core.multiPackIndex setting","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-07-11T09:48:58Z","receivedAt":"2018-07-11T09:49:06Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index ab641bf5a9..ab895ebb32 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -908,6 +908,10 @@ core.commitGraph::\n> \tEnable git commit graph feature. Allows reading from the\n> \tcommit-graph file.\n> \n> +core.multiPackIndex::\n> +\tUse the multi-pack-index file to track multiple packfiles using a\n> +\tsingle index. See linkgit:technical/multi-pack-index[1].\n\nThe 'linkgit' macro should be used to create links to other man pages,\nbut 'technical/multi-pack-index' is not a man page and this causes\n'make check-docs' to complain:\n\n LINT lint-docs\n ./config.txt:929: nongit link: technical/multi-pack-index[1]\n Makefile:456: recipe for target 'lint-docs' failed\n make[1]: *** [lint-docs] Error 1\n\n\n> +\n> core.sparseCheckout::\n> \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n> \tlinkgit:git-read-tree[1] for more information.\n"},{"id":"352334","messageId":"dd2f774d-a217-0273-efbe-d133ae23c5bc@gmail.com","threadId":"48668","inReplyTo":"20180711094858.3230-1-szeder.dev@gmail.com","subject":"Re: [PATCH v3 16/24] config: create core.multiPackIndex setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T13:01:47Z","receivedAt":"2018-07-12T13:01:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/11/2018 5:48 AM, SZEDER Gábor wrote:\n>> diff --git a/Documentation/config.txt b/Documentation/config.txt\n>> index ab641bf5a9..ab895ebb32 100644\n>> --- a/Documentation/config.txt\n>> +++ b/Documentation/config.txt\n>> @@ -908,6 +908,10 @@ core.commitGraph::\n>> \tEnable git commit graph feature. Allows reading from the\n>> \tcommit-graph file.\n>> \n>> +core.multiPackIndex::\n>> +\tUse the multi-pack-index file to track multiple packfiles using a\n>> +\tsingle index. See linkgit:technical/multi-pack-index[1].\n> The 'linkgit' macro should be used to create links to other man pages,\n> but 'technical/multi-pack-index' is not a man page and this causes\n> 'make check-docs' to complain:\n>\n> LINT lint-docs\n> ./config.txt:929: nongit link: technical/multi-pack-index[1]\n> Makefile:456: recipe for target 'lint-docs' failed\n> make[1]: *** [lint-docs] Error 1\n>\nThanks for this point. It seems to work using \n\"link:technical/multi-pack-index[1]\", which is what I'll use in the next \nversion.\n\n-Stolee\n\n"},{"id":"352337","messageId":"2270ec32-f5e3-e23a-5605-f5f0e7a44c46@gmail.com","threadId":"48668","inReplyTo":"CAPig+cTJuvOjyzkmAj5axAkS3q1UyeV03jQk6-oB==KyzbEK_w@mail.gmail.com","subject":"Re: [PATCH v3 16/24] config: create core.multiPackIndex setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T13:19:20Z","receivedAt":"2018-07-12T13:19:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/6/2018 1:39 AM, Eric Sunshine wrote:\n> On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n>> The core.multiPackIndex config setting controls the multi-pack-\n>> index (MIDX) feature. If false, the setting will disable all reads\n>> from the multi-pack-index file.\n>>\n>> Add comparison commands in t5319-multi-pack-index.sh to check\n>> typical Git behavior remains the same as the config setting is turned\n>> on and off. This currently includes 'git rev-list' and 'git log'\n>> commands to trigger several object database reads.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> diff --git a/cache.h b/cache.h\n>> @@ -814,6 +814,7 @@ extern char *git_replace_ref_base;\n>> +extern int core_multi_pack_index;\n>> diff --git a/config.c b/config.c\n>> @@ -1313,6 +1313,11 @@ static int git_default_core_config(const char *var, const char *value)\n>> + if (!strcmp(var, \"core.multipackindex\")) {\n>> + core_multi_pack_index = git_config_bool(var, value);\n>> + return 0;\n>> + }\n> This is a rather unusual commit. This new configuration is assigned,\n> but it's never actually consulted, which means that it's impossible\n> for it to have any impact on functionality, yet tests are being added\n> to check whether it _did_ have any impact on functionality. Confusing.\n>\n> Patch 17 does consult 'core_multi_pack_index', so it's only at that\n> point that it could have any impact. This situation would be less\n> confusing if you swapped patches 16 and 17 (and, of course, move the\n> declaration of 'core_multi_pack_index' to patch 17 with a reasonable\n> default value).\n\nYou're right that this commit is a bit too aware of the future, but I \ndisagree with the recommendation to change it.\n\nYes, in this commit there is no possible way that these tests could \nfail. The point is that patches 17-23 all change behavior if this \nsetting is on, and we want to make sure we do not break at any point \nalong that journey (or in future iterations of the multi-pack-index \nfeature).\n\nWith this in mind, I don't think there is a better commit to place these \ntests.\n\n>> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n>> @@ -61,12 +63,42 @@ test_expect_success 'write midx with one v1 pack' '\n>> +midx_git_two_modes() {\n>> + git -c core.multiPackIndex=false $1 >expect &&\n>> + git -c core.multiPackIndex=true $1 >actual &&\n>> + test_cmp expect actual\n>> +}\n>> +\n>> +compare_results_with_midx() {\n>> + MSG=$1\n>> + test_expect_success \"check normal git operations: $MSG\" '\n>> + midx_git_two_modes \"rev-list --objects --all\" &&\n>> + midx_git_two_modes \"log --raw\"\n>> + '\n>> +}\n> Here, you define midx_git_two_modes() and compare_results_with_midx()...\n>\n>> test_expect_success 'write midx with one v2 pack' '\n>> - git pack-objects --index-version=2,0x40 pack/test > - git multi-pack-index --object-dir=. write &&\n>> - midx_read_expect 1 17 4 .\n>> + git pack-objects --index-version=2,0x40 $objdir/pack/test > + git multi-pack-index --object-dir=$objdir write &&\n>> + midx_read_expect 1 17 4 $objdir\n>> '\n>>\n>> +midx_git_two_modes() {\n>> + git -c core.multiPackIndex=false $1 >expect &&\n>> + git -c core.multiPackIndex=true $1 >actual &&\n>> + test_cmp expect actual\n>> +}\n>> +\n>> +compare_results_with_midx() {\n>> + MSG=$1\n>> + test_expect_success \"check normal git operations: $MSG\" '\n>> + midx_git_two_modes \"rev-list --objects --all\" &&\n>> + midx_git_two_modes \"log --raw\"\n>> + '\n>> +}\n> ... and here you define both functions again.\n\nThis was a mistake. Thanks for catching it.\n\n\nThanks,\n\n-Stolee\n\n"},{"id":"352339","messageId":"CAM0VKjkuC5Qc6HtQZP4dpHE4CcCvNkg1D8_AFw7+PQDU6bU5Wg@mail.gmail.com","threadId":"48668","inReplyTo":"dd2f774d-a217-0273-efbe-d133ae23c5bc@gmail.com","subject":"Re: [PATCH v3 16/24] config: create core.multiPackIndex setting","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-07-12T13:31:13Z","receivedAt":"2018-07-12T13:31:28Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jul 12, 2018 at 3:01 PM Derrick Stolee wrote:\n>\n> On 7/11/2018 5:48 AM, SZEDER Gábor wrote:\n> >> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> >> index ab641bf5a9..ab895ebb32 100644\n> >> --- a/Documentation/config.txt\n> >> +++ b/Documentation/config.txt\n> >> @@ -908,6 +908,10 @@ core.commitGraph::\n> >> Enable git commit graph feature. Allows reading from the\n> >> commit-graph file.\n> >>\n> >> +core.multiPackIndex::\n> >> + Use the multi-pack-index file to track multiple packfiles using a\n> >> + single index. See linkgit:technical/multi-pack-index[1].\n> > The 'linkgit' macro should be used to create links to other man pages,\n> > but 'technical/multi-pack-index' is not a man page and this causes\n> > 'make check-docs' to complain:\n> >\n> > LINT lint-docs\n> > ./config.txt:929: nongit link: technical/multi-pack-index[1]\n> > Makefile:456: recipe for target 'lint-docs' failed\n> > make[1]: *** [lint-docs] Error 1\n> >\n> Thanks for this point. It seems to work using\n> \"link:technical/multi-pack-index[1]\", which is what I'll use in the next\n> version.\n\nIt doesn't work, it merely works around the build failure.\n\nThe generated man page looks like this:\n\n core.multiPackIndex\n Use the multi-pack-index file to track multiple packfiles using a\n single index. See 1[1].\n\nAnd the resulting html page looks similar:\n\n core.multiPackIndex\n\n Use the multi-pack-index file to track multiple packfiles using a\n single index. See 1.\n\nwhere that \"1\" is a link pointing to the non-existing URL\nfile:///home/me/src/git/Documentation/technical/multi-pack-index\n"},{"id":"352342","messageId":"3e82dd51-2e30-291f-8753-c1d711459c1d@gmail.com","threadId":"48668","inReplyTo":"CAPig+cTO=UPuCRUAAWfBwaNhRD+=dgSwpgEB2fQJuWyFEd7Aew@mail.gmail.com","subject":"Re: [PATCH v3 07/24] multi-pack-index: expand test data","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T14:10:29Z","receivedAt":"2018-07-12T14:10:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/6/2018 12:36 AM, Eric Sunshine wrote:\n> On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n>\n>> + for i in $(test_seq 6 10)\n>> + do\n>> + iii=$(printf '%03i' $i)\n>> + test-tool genrandom \"bar\" 200 >wide_delta_$iii &&\n>> + test-tool genrandom \"baz $iii\" 50 >>wide_delta_$iii &&\n>> + test-tool genrandom \"foo\"$i 100 >deep_delta_$iii &&\n>> + test-tool genrandom \"foo\"$(expr $i + 1) 100 >>deep_delta_$iii &&\n>> + test-tool genrandom \"foo\"$(expr $i + 2) 100 >>deep_delta_$iii &&\n>> + echo $iii >file_$iii &&\n>> + test-tool genrandom \"$iii\" 8192 >>file_$iii &&\n>> + git update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n>> + i=$(expr $i + 1) || return 1\n>> + done &&\n>> + { echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n>> + git update-index --add file_101 &&\n>> + tree=$(git write-tree) &&\n>> + commit=$(git commit-tree $tree -p HEAD> + echo $tree &&\n>> + git ls-tree $tree | sed -e \"s/.* \\\\([0-9a-f]*\\\\) .*/\\\\1/\"\n>> + } >obj-list2 &&\n>> + git update-ref HEAD $commit\n>> +'\n> There seems to be a fair bit of duplication in these tests which\n> create objects. Is it possible to factor out some of this code into a\n> shell function?\n\nIn addition to the other small changes, this refactor in particular was \na big change (but a good one). I'm sending my current progress in this \ndirection, as I expect this can be improved.\n\nTo make the commit_and_list_objects method more generic to all \nsituations, I had to add an extra commit, which will cause some of the \nnumbers to change in the later 'midx_read_expect' calls.\n\nThanks,\n\n-Stolee\n\n-->8--\n\n From cb38bb284fd05cf2230725b6cb9ead5795c913f2 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Thu, 31 May 2018 15:05:00 -0400\nSubject: [PATCH] t5319: expand test data\n\nAs we build the multi-pack-index file format, we want to test the format\non real repositories. Add tests that create repository data including\nmultiple packfiles with both version 1 and version 2 formats.\n\nThe current 'git multi-pack-index write' command will always write the\nsame file with no \"real\" data. This will be expanded in future commits,\nalong with the test expectations.\n\nSigned-off-by: Derrick Stolee \n---\n  t/t5319-multi-pack-index.sh | 83 +++++++++++++++++++++++++++++++++++++\n  1 file changed, 83 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 2ecc369529..a50be41bc0 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -13,9 +13,92 @@ midx_read_expect () {\n  }\n\n  test_expect_success 'write midx with no packs' '\n+       test_when_finished rm -f pack/multi-pack-index &&\n         git multi-pack-index --object-dir=. write &&\n         test_path_is_file pack/multi-pack-index &&\n         midx_read_expect\n  '\n\n+generate_objects () {\n+       i=$1\n+       iii=$(printf '%03i' $i)\n+       {\n+               test-tool genrandom \"bar\" 200 &&\n+               test-tool genrandom \"baz $iii\" 50\n+       } >wide_delta_$iii &&\n+       {\n+               test-tool genrandom \"foo\"$i 100 &&\n+               test-tool genrandom \"foo\"$(( $i + 1 )) 100 &&\n+               test-tool genrandom \"foo\"$(( $i + 2 )) 100\n+       } >>deep_delta_$iii &&\n+       echo $iii >file_$iii &&\n+       test-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+       git update-index --add file_$iii deep_delta_$iii wide_delta_$iii\n+}\n+\n+commit_and_list_objects () {\n+       {\n+               echo 101 &&\n+               test-tool genrandom 100 8192;\n+       } >file_101 &&\n+       git update-index --add file_101 &&\n+       tree=$(git write-tree) &&\n+       commit=$(git commit-tree $tree -p HEADobj-list &&\n+       git reset --hard $commit\n+}\n+\n+test_expect_success 'create objects' '\n+       test_commit initial &&\n+       for i in $(test_seq 1 5)\n+       do\n+               generate_objects $i\n+       done &&\n+       commit_and_list_objects\n+'\n+\n+test_expect_success 'write midx with one v1 pack' '\n+       pack=$(git pack-objects --index-version=1 pack/test On Thu, Jul 12, 2018 at 3:01 PM Derrick Stolee wrote:\n>> On 7/11/2018 5:48 AM, SZEDER Gábor wrote:\n>>>> diff --git a/Documentation/config.txt b/Documentation/config.txt\n>>>> index ab641bf5a9..ab895ebb32 100644\n>>>> --- a/Documentation/config.txt\n>>>> +++ b/Documentation/config.txt\n>>>> @@ -908,6 +908,10 @@ core.commitGraph::\n>>>> Enable git commit graph feature. Allows reading from the\n>>>> commit-graph file.\n>>>>\n>>>> +core.multiPackIndex::\n>>>> + Use the multi-pack-index file to track multiple packfiles using a\n>>>> + single index. See linkgit:technical/multi-pack-index[1].\n>>> The 'linkgit' macro should be used to create links to other man pages,\n>>> but 'technical/multi-pack-index' is not a man page and this causes\n>>> 'make check-docs' to complain:\n>>>\n>>> LINT lint-docs\n>>> ./config.txt:929: nongit link: technical/multi-pack-index[1]\n>>> Makefile:456: recipe for target 'lint-docs' failed\n>>> make[1]: *** [lint-docs] Error 1\n>>>\n>> Thanks for this point. It seems to work using\n>> \"link:technical/multi-pack-index[1]\", which is what I'll use in the next\n>> version.\n> It doesn't work, it merely works around the build failure.\n>\n> The generated man page looks like this:\n>\n> core.multiPackIndex\n> Use the multi-pack-index file to track multiple packfiles using a\n> single index. See 1[1].\n>\n> And the resulting html page looks similar:\n>\n> core.multiPackIndex\n>\n> Use the multi-pack-index file to track multiple packfiles using a\n> single index. See 1.\n>\n> where that \"1\" is a link pointing to the non-existing URL\n> file:///home/me/src/git/Documentation/technical/multi-pack-index\n\nRight. Sorry. I also see that I use the correct kind of links in \nDocumentation/git-multi-pack-index.txt (see below) so I will use it \nhere, too.\n\nSEE ALSO\n--------\nSee link:technical/multi-pack-index.html[The Multi-Pack-Index Design\nDocument] and link:technical/pack-format.html[The Multi-Pack-Index\nFormat] for more information on the multi-pack-index feature.\n\nThanks,\n\n-Stolee\n\n"},{"id":"352358","messageId":"1fb87ff0-5315-d96e-4671-433311746c31@gmail.com","threadId":"48668","inReplyTo":"xmqq36wsdy75.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 06/24] multi-pack-index: load into memory","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T16:06:02Z","receivedAt":"2018-07-12T16:06:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/9/2018 3:08 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>> diff --git a/midx.h b/midx.h\n>> index dbdbe9f873..2d83dd9ec1 100644\n>> --- a/midx.h\n>> +++ b/midx.h\n>> @@ -1,6 +1,10 @@\n>> #ifndef __MIDX_H__\n>> #define __MIDX_H__\n>> \n>> +struct multi_pack_index;\n> I actually was quite surprised that this struct is defined in\n> object-store.h and not here. It feels the other way around.\n>\n> The raw_object_store needs to know that such an in-core structure\n> might exist as an optional feature in an object store, but as an\n> optional feature, I suspect that it has a pointer to an instance of\n> multi_pack_index, instead of embedding the struct itself in it, so I\n> would have expected to see an \"I am only telling you that there is a\n> struct with this name, but I am leaving it opaque as you do not have\n> any business looking inside the struct yourself. You only need to\n> be aware of the type's existence and a pointer to it so that you can\n> call helpers that know what's inside and that should be sufficient\n> for your needs.\" decl like this in object-store.h and instead an\n> actual implementation in here.\n\nI thought it natural to include the struct definition next to the \ndefinition for 'struct packed_git', but I like your separation of \nconcerns. Perhaps we could move the packed_git definition to packfile.h \nas well (separately). Of course, that sounds like an unnecessary churn.\n\nThanks,\n\n-Stolee\n\n"},{"id":"352362","messageId":"5ff95525-4677-872d-e249-b01b5aaf6c68@gmail.com","threadId":"48668","inReplyTo":"2270ec32-f5e3-e23a-5605-f5f0e7a44c46@gmail.com","subject":"Re: [PATCH v3 16/24] config: create core.multiPackIndex setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T16:30:22Z","receivedAt":"2018-07-12T16:30:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/12/2018 9:19 AM, Derrick Stolee wrote:\n> On 7/6/2018 1:39 AM, Eric Sunshine wrote:\n>> On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n>>> The core.multiPackIndex config setting controls the multi-pack-\n>>> index (MIDX) feature. If false, the setting will disable all reads\n>>> from the multi-pack-index file.\n>>>\n>>> Add comparison commands in t5319-multi-pack-index.sh to check\n>>> typical Git behavior remains the same as the config setting is turned\n>>> on and off. This currently includes 'git rev-list' and 'git log'\n>>> commands to trigger several object database reads.\n>>>\n>>> Signed-off-by: Derrick Stolee \n>>> ---\n>>> diff --git a/cache.h b/cache.h\n>>> @@ -814,6 +814,7 @@ extern char *git_replace_ref_base;\n>>> +extern int core_multi_pack_index;\n>>> diff --git a/config.c b/config.c\n>>> @@ -1313,6 +1313,11 @@ static int git_default_core_config(const char \n>>> *var, const char *value)\n>>> +       if (!strcmp(var, \"core.multipackindex\")) {\n>>> +               core_multi_pack_index = git_config_bool(var, value);\n>>> +               return 0;\n>>> +       }\n>> This is a rather unusual commit. This new configuration is assigned,\n>> but it's never actually consulted, which means that it's impossible\n>> for it to have any impact on functionality, yet tests are being added\n>> to check whether it _did_ have any impact on functionality. Confusing.\n>>\n>> Patch 17 does consult 'core_multi_pack_index', so it's only at that\n>> point that it could have any impact. This situation would be less\n>> confusing if you swapped patches 16 and 17 (and, of course, move the\n>> declaration of 'core_multi_pack_index' to patch 17 with a reasonable\n>> default value).\n>\n> You're right that this commit is a bit too aware of the future, but I \n> disagree with the recommendation to change it.\n>\n> Yes, in this commit there is no possible way that these tests could \n> fail. The point is that patches 17-23 all change behavior if this \n> setting is on, and we want to make sure we do not break at any point \n> along that journey (or in future iterations of the multi-pack-index \n> feature).\n>\n> With this in mind, I don't think there is a better commit to place \n> these tests.\n\nOf course, as I convert this global config variable into an on-demand \ncheck as promised [1] this commit seems even more trivial. I'm going to \nsquash it with PATCH 17.\n\n[1] \nhttps://public-inbox.org/git/b5733625-29c8-4317-ff44-d27c2fca11ce@gmail.com/\n\n"},{"id":"352363","messageId":"e33b0e17-bf32-3ee9-5874-96555ce98cbc@gmail.com","threadId":"48668","inReplyTo":"CAPig+cR1BaOTTjg751NO14sAr6UZ0WT_G3Hu59KyiWTASZ8D2A@mail.gmail.com","subject":"Re: [PATCH v3 15/24] midx: write object offsets","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T16:33:13Z","receivedAt":"2018-07-12T16:33:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/6/2018 1:27 AM, Eric Sunshine wrote:\n> On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee wrote:\n>> The final pair of chunks for the multi-pack-index file stores the object\n>> offsets. We default to using 32-bit offsets as in the pack-index version\n>> 1 format, but if there exists an offset larger than 32-bits, we use a\n>> trick similar to the pack-index version 2 format by storing all offsets\n>> at least 2^31 in a 64-bit table; we use the 32-bit table to point into\n>> that 64-bit table as necessary.\n>>\n>> We only store these 64-bit offsets if necessary, so create a test that\n>> manipulates a version 2 pack-index to fake a large offset. This allows\n>> us to test that the large offset table is created, but the data does not\n>> match the actual packfile offsets. The multi-pack-index offset does match\n>> the (corrupted) pack-index offset, so a future feature will compare these\n>> offsets during a 'verify' step.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n>> @@ -6,25 +6,28 @@ test_description='multi-pack-indexes'\n>> +# usage: corrupt_data []\n>> +corrupt_data() {\n> Style: corrupt_data () {\n>\n>> + file=$1\n>> + pos=$2\n>> + data=\"${3:-\\0}\"\n>> + printf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n>> +}\n>> +\n>> +# Force 64-bit offsets by manipulating the idx file.\n>> +# This makes the IDX file _incorrect_ so be careful to clean up after!\n>> +test_expect_success 'force some 64-bit offsets with pack-objects' '\n>> + mkdir objects64 &&\n>> + mkdir objects64/pack &&\n>> + pack64=$(git pack-objects --index-version=2,0x40 objects64/pack/test-64 > + idx64=objects64/pack/test-64-$pack64.idx &&\n>> + chmod u+w $idx64 &&\n> I guess you don't have to worry about the POSIXPERM prerequisite here\n> because the file is already writable on Windows, right?\n\nCorrect. And I want this test to still run on Windows.\n\nThanks,\n\n-Stolee\n\n"},{"id":"352371","messageId":"xmqqsh4oxsze.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"CAM0VKjkuC5Qc6HtQZP4dpHE4CcCvNkg1D8_AFw7+PQDU6bU5Wg@mail.gmail.com","subject":"Re: [PATCH v3 16/24] config: create core.multiPackIndex setting","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-07-12T17:29:41Z","receivedAt":"2018-07-12T17:29:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n>> Thanks for this point. It seems to work using\n>> \"link:technical/multi-pack-index[1]\", which is what I'll use in the next\n>> version.\n>\n> It doesn't work, it merely works around the build failure.\n\nSorry. I fell into the same trap X-<.\n\nlink:techincal/multi-pack-index.html[the technical documentation\nfor it]\n\nor something like that, perhaps.\n"},{"id":"352375","messageId":"CAPig+cTEcLtvrQ25KDmc4Cmi9C2yCUpLCTWsK9q8SAiZt+N-Jw@mail.gmail.com","threadId":"48668","inReplyTo":"3e82dd51-2e30-291f-8753-c1d711459c1d@gmail.com","subject":"Re: [PATCH v3 07/24] multi-pack-index: expand test data","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-12T18:02:27Z","receivedAt":"2018-07-12T18:02:41Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 12, 2018 at 10:10 AM Derrick Stolee wrote:\n> On 7/6/2018 12:36 AM, Eric Sunshine wrote:\n> > There seems to be a fair bit of duplication in these tests which\n> > create objects. Is it possible to factor out some of this code into a\n> > shell function?\n>\n> In addition to the other small changes, this refactor in particular was\n> a big change (but a good one). I'm sending my current progress in this\n> direction, as I expect this can be improved.\n\nI like the amount of code reduction. A couple minor comments...\n\n> +generate_objects () {\n> + i=$1\n> + iii=$(printf '%03i' $i)\n> + {\n> + test-tool genrandom \"bar\" 200 &&\n> + test-tool genrandom \"baz $iii\" 50\n> + } >wide_delta_$iii &&\n> + {\n> + test-tool genrandom \"foo\"$i 100 &&\n> + test-tool genrandom \"foo\"$(( $i + 1 )) 100 &&\n> + test-tool genrandom \"foo\"$(( $i + 2 )) 100\n> + } >>deep_delta_$iii &&\n\nI think this should be: s/>>/>/\n\n> + echo $iii >file_$iii &&\n> + test-tool genrandom \"$iii\" 8192 >>file_$iii &&\n\nAnd this: s/>>/>/\n\n> + git update-index --add file_$iii deep_delta_$iii wide_delta_$iii\n> +}\n"},{"id":"352376","messageId":"59cbce97-66cf-8332-c901-8a75f375b670@gmail.com","threadId":"48668","inReplyTo":"CAPig+cTEcLtvrQ25KDmc4Cmi9C2yCUpLCTWsK9q8SAiZt+N-Jw@mail.gmail.com","subject":"Re: [PATCH v3 07/24] multi-pack-index: expand test data","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T18:06:08Z","receivedAt":"2018-07-12T18:06:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/12/2018 2:02 PM, Eric Sunshine wrote:\n> On Thu, Jul 12, 2018 at 10:10 AM Derrick Stolee wrote:\n>> On 7/6/2018 12:36 AM, Eric Sunshine wrote:\n>>> There seems to be a fair bit of duplication in these tests which\n>>> create objects. Is it possible to factor out some of this code into a\n>>> shell function?\n>> In addition to the other small changes, this refactor in particular was\n>> a big change (but a good one). I'm sending my current progress in this\n>> direction, as I expect this can be improved.\n> I like the amount of code reduction. A couple minor comments...\n>\n>> +generate_objects () {\n>> + i=$1\n>> + iii=$(printf '%03i' $i)\n>> + {\n>> + test-tool genrandom \"bar\" 200 &&\n>> + test-tool genrandom \"baz $iii\" 50\n>> + } >wide_delta_$iii &&\n>> + {\n>> + test-tool genrandom \"foo\"$i 100 &&\n>> + test-tool genrandom \"foo\"$(( $i + 1 )) 100 &&\n>> + test-tool genrandom \"foo\"$(( $i + 2 )) 100\n>> + } >>deep_delta_$iii &&\n> I think this should be: s/>>/>/\n\nIt should!\n\n>> + echo $iii >file_$iii &&\n>> + test-tool genrandom \"$iii\" 8192 >>file_$iii &&\n> And this: s/>>/>/\n\nIn addition, I should wrap these two commands in { } like the files above.\n\nThanks,\n\n-Stolee\n\n"},{"id":"352392","messageId":"20180712193940.21065-1-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180706005321.124643-1-dstolee@microsoft.com","subject":"[PATCH v4 00/23] Multi-pack-index (MIDX)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:17Z","receivedAt":"2018-07-12T19:40:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"v3 had a lot of interesting feedback, most of which was non-functional,\nbut made a big impact on the shape of the patch, especially the test\nscript.\n\nThese are the important changes:\n\n* 'git multi-pack-index' will report usage if the 'write' verb is not\n provided, or if extra parameters are provided. A later series will\n create the 'verify' verb.\n\n* t5319-multi-pack-index.sh has a reoganized way to generate object\n data, so it has fewer code clones.\n\n* 'test-tool read-midx' uses '-' instead of '_'.\n\n* The global 'core_multi_pack_index' is replaced with a one-time call to\n git_config_bool() per repository that loads a multi-pack-index.\n\n* 'struct multi_pack_index' is now defined in midx.h and kept anonymous\n to object-store.h.\n\n* Added a test that 'git repack' removes the multi-pack-index.\n\n* Fixed a doc bug when linking to the technical docs.\n\nI included the diff between the latest ds/multi-pack-index and this\nseries as part of this message.\n\nYou can see the CI builds for Linux, Mac, and Windows linked from the\nGitHub pull request [1].\n\nThanks,\n-Stolee\n\n[1] https://github.com/gitgitgadget/git/pull/5\n\nDerrick Stolee (23):\n multi-pack-index: add design document\n multi-pack-index: add format details\n multi-pack-index: add builtin\n multi-pack-index: add 'write' verb\n midx: write header information to lockfile\n multi-pack-index: load into memory\n t5319: expand test data\n packfile: generalize pack directory list\n multi-pack-index: read packfile list\n multi-pack-index: write pack names in chunk\n midx: read pack names into array\n midx: sort and deduplicate objects from packfiles\n midx: write object ids in a chunk\n midx: write object id fanout chunk\n midx: write object offsets\n config: create core.multiPackIndex setting\n midx: read objects from multi-pack-index\n midx: use midx in abbreviation calculations\n midx: use existing midx when writing new one\n midx: use midx in approximate_object_count\n midx: prevent duplicate packfile loads\n packfile: skip loading index if in multi-pack-index\n midx: clear midx on repack\n\n .gitignore | 3 +-\n Documentation/config.txt | 5 +\n Documentation/git-multi-pack-index.txt | 56 ++\n Documentation/technical/multi-pack-index.txt | 109 +++\n Documentation/technical/pack-format.txt | 77 ++\n Makefile | 3 +\n builtin.h | 1 +\n builtin/multi-pack-index.c | 47 +\n builtin/repack.c | 9 +\n command-list.txt | 1 +\n git.c | 1 +\n midx.c | 918 +++++++++++++++++++\n midx.h | 44 +\n object-store.h | 9 +\n packfile.c | 169 +++-\n packfile.h | 9 +\n sha1-name.c | 70 ++\n t/helper/test-read-midx.c | 51 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5319-multi-pack-index.sh | 179 ++++\n 21 files changed, 1720 insertions(+), 43 deletions(-)\n create mode 100644 Documentation/git-multi-pack-index.txt\n create mode 100644 Documentation/technical/multi-pack-index.txt\n create mode 100644 builtin/multi-pack-index.c\n create mode 100644 midx.c\n create mode 100644 midx.h\n create mode 100644 t/helper/test-read-midx.c\n create mode 100755 t/t5319-multi-pack-index.sh\n\n\nbase-commit: 53f9a3e157dbbc901a02ac2c73346d375e24978c\n-- \n2.18.0.118.gd4f65b8d14\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9dcde07a34..25f817ca42 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -910,7 +910,8 @@ core.commitGraph::\n \n core.multiPackIndex::\n \tUse the multi-pack-index file to track multiple packfiles using a\n-\tsingle index. See link:technical/multi-pack-index[1].\n+\tsingle index. See link:technical/multi-pack-index.html[the\n+\tmulti-pack-index design document].\n \n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex be97c9372e..a62af1caca 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir ] \n+'git multi-pack-index' [--object-dir=] \n \n DESCRIPTION\n -----------\n@@ -18,7 +18,7 @@ Write or verify a multi-pack-index (MIDX) file.\n OPTIONS\n -------\n \n---object-dir ::\n+--object-dir=::\n \tUse given directory for the location of Git objects. We check\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n@@ -37,7 +37,7 @@ EXAMPLES\n $ git multi-pack-index write\n -----------------------------------------------\n \n-* Write a MIDX file for the packfiles in an alternate.\n+* Write a MIDX file for the packfiles in an alternate object store.\n +\n -----------------------------------------------\n $ git multi-pack-index --object-dir write\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 14b32e1373..6a7aa00cf2 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,7 +5,7 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir ] [write]\"),\n+\tN_(\"git multi-pack-index [--object-dir=] write\"),\n \tNULL\n };\n \n@@ -18,14 +18,10 @@ int cmd_multi_pack_index(int argc, const char **argv,\n {\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n-\t\t N_(\"The object directory containing set of packfile and pack-index pairs\")),\n+\t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n \t\tOPT_END(),\n \t};\n \n-\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n-\t\tusage_with_options(builtin_multi_pack_index_usage,\n-\t\t\t\t builtin_multi_pack_index_options);\n-\n \tgit_config(git_default_config, NULL);\n \n \targc = parse_options(argc, argv, prefix,\n@@ -36,11 +32,16 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\topts.object_dir = get_object_directory();\n \n \tif (argc == 0)\n-\t\tusage_with_options(builtin_multi_pack_index_usage,\n-\t\t\t\t builtin_multi_pack_index_options);\n+\t\tgoto usage;\n+\n+\tif (!strcmp(argv[0], \"write\")) {\n+\t\tif (argc > 1)\n+\t\t\tgoto usage;\n \n-\tif (!strcmp(argv[0], \"write\"))\n \t\treturn write_midx_file(opts.object_dir);\n+\t}\n \n-\treturn 0;\n+usage:\n+\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t builtin_multi_pack_index_options);\n }\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 66a7d8e8ea..7f7cdc8b17 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -335,12 +335,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tfor_each_string_list_item(item, &names) {\n \t\tfor (ext = 0; ext < ARRAY_SIZE(exts); ext++) {\n \t\t\tchar *fname, *fname_old;\n-\t\t\tfname = mkpathdup(\"%s/pack-%s%s\", packdir,\n-\t\t\t\t\t\titem->string, exts[ext].name);\n-\t\t\tif (!file_exists(fname)) {\n-\t\t\t\tfree(fname);\n-\t\t\t\tcontinue;\n-\t\t\t}\n \n \t\t\tif (!midx_cleared) {\n \t\t\t\t/* if we move a packfile, it will invalidated the midx */\n@@ -348,6 +342,13 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tmidx_cleared = 1;\n \t\t\t}\n \n+\t\t\tfname = mkpathdup(\"%s/pack-%s%s\", packdir,\n+\t\t\t\t\t\titem->string, exts[ext].name);\n+\t\t\tif (!file_exists(fname)) {\n+\t\t\t\tfree(fname);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tfname_old = mkpathdup(\"%s/old-%s%s\", packdir,\n \t\t\t\t\t\titem->string, exts[ext].name);\n \t\t\tif (file_exists(fname_old))\ndiff --git a/cache.h b/cache.h\nindex d12aa49710..89a107a7f7 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -814,7 +814,6 @@ extern char *git_replace_ref_base;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_commit_graph;\n-extern int core_multi_pack_index;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex 95d8da4243..fbbf0f8e9f 100644\n--- a/config.c\n+++ b/config.c\n@@ -1313,11 +1313,6 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n-\tif (!strcmp(var, \"core.multipackindex\")) {\n-\t\tcore_multi_pack_index = git_config_bool(var, value);\n-\t\treturn 0;\n-\t}\n-\n \tif (!strcmp(var, \"core.sparsecheckout\")) {\n \t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex b9bc919cdb..2a6de2330b 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -67,7 +67,6 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_commit_graph;\n-int core_multi_pack_index;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\ndiff --git a/midx.c b/midx.c\nindex 8afd08f3fe..19b7df338e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"csum-file.h\"\n #include \"dir.h\"\n #include \"lockfile.h\"\n@@ -60,7 +61,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tmidx_size = xsize_t(st.st_size);\n \n \tif (midx_size < MIDX_MIN_SIZE) {\n-\t\tclose(fd);\n \t\terror(_(\"multi-pack-index file %s is too small\"), midx_name);\n \t\tgoto cleanup_fail;\n \t}\n@@ -69,8 +69,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \n-\tm = xcalloc(1, sizeof(*m) + strlen(object_dir) + 1);\n-\tstrcpy(m->object_dir, object_dir);\n+\tFLEX_ALLOC_MEM(m, object_dir, object_dir, strlen(object_dir));\n \tm->fd = fd;\n \tm->data = midx_map;\n \tm->data_len = midx_size;\n@@ -171,7 +170,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \treturn m;\n \n cleanup_fail:\n-\t/* no need to check for NULL when freeing */\n \tfree(m);\n \tfree(midx_name);\n \tif (midx_map)\n@@ -324,8 +322,10 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n {\n \tstruct multi_pack_index *m = r->objects->multi_pack_index;\n \tstruct multi_pack_index *m_search;\n+\tint config_value;\n \n-\tif (!core_multi_pack_index)\n+\tif (repo_config_get_bool(r, \"core.multipackindex\", &config_value) ||\n+\t !config_value)\n \t\treturn 0;\n \n \tfor (m_search = m; m_search; m_search = m_search->next)\n@@ -382,7 +382,8 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n \n \t\tpacks->list[packs->nr] = add_packed_git(full_path,\n-\t\t\t\t\t\t\tfull_path_len, 0);\n+\t\t\t\t\t\t\tfull_path_len,\n+\t\t\t\t\t\t\t0);\n \n \t\tif (!packs->list[packs->nr]) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n@@ -661,8 +662,8 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n \t\t\tstruct pack_midx_entry *next = list;\n \t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n \t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n-\t\t\t\toid_to_hex(&obj->oid),\n-\t\t\t\toid_to_hex(&next->oid));\n+\t\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t\t oid_to_hex(&next->oid));\n \t\t}\n \n \t\thashwrite(f, obj->oid.hash, (int)hash_len);\ndiff --git a/midx.h b/midx.h\nindex 5a42cbed1d..e3b07f1586 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -3,7 +3,31 @@\n \n #include \"repository.h\"\n \n-struct multi_pack_index;\n+struct multi_pack_index {\n+\tstruct multi_pack_index *next;\n+\n+\tint fd;\n+\n+\tconst unsigned char *data;\n+\tsize_t data_len;\n+\n+\tuint32_t signature;\n+\tunsigned char version;\n+\tunsigned char hash_len;\n+\tunsigned char num_chunks;\n+\tuint32_t num_packs;\n+\tuint32_t num_objects;\n+\n+\tconst unsigned char *chunk_pack_names;\n+\tconst uint32_t *chunk_oid_fanout;\n+\tconst unsigned char *chunk_oid_lookup;\n+\tconst unsigned char *chunk_object_offsets;\n+\tconst unsigned char *chunk_large_offsets;\n+\n+\tconst char **pack_names;\n+\tstruct packed_git **packs;\n+\tchar object_dir[FLEX_ARRAY];\n+};\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\ndiff --git a/object-store.h b/object-store.h\nindex 03cc278758..c2b162489a 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -84,31 +84,7 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n-struct multi_pack_index {\n-\tstruct multi_pack_index *next;\n-\n-\tint fd;\n-\n-\tconst unsigned char *data;\n-\tsize_t data_len;\n-\n-\tuint32_t signature;\n-\tunsigned char version;\n-\tunsigned char hash_len;\n-\tunsigned char num_chunks;\n-\tuint32_t num_packs;\n-\tuint32_t num_objects;\n-\n-\tconst unsigned char *chunk_pack_names;\n-\tconst uint32_t *chunk_oid_fanout;\n-\tconst unsigned char *chunk_oid_lookup;\n-\tconst unsigned char *chunk_object_offsets;\n-\tconst unsigned char *chunk_large_offsets;\n-\n-\tconst char **pack_names;\n-\tstruct packed_git **packs;\n-\tchar object_dir[FLEX_ARRAY];\n-};\n+struct multi_pack_index;\n \n struct raw_object_store {\n \t/*\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 20771d1c1d..8e19972e89 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -1,6 +1,3 @@\n-/*\n- * test-mktemp.c: code to exercise the creation of temporary files\n- */\n #include \"test-tool.h\"\n #include \"cache.h\"\n #include \"midx.h\"\n@@ -13,7 +10,7 @@ static int read_midx_file(const char *object_dir)\n \tstruct multi_pack_index *m = load_multi_pack_index(object_dir);\n \n \tif (!m)\n-\t\treturn 0;\n+\t\treturn 1;\n \n \tprintf(\"header: %08x %d %d %d\\n\",\n \t m->signature,\n@@ -24,15 +21,15 @@ static int read_midx_file(const char *object_dir)\n \tprintf(\"chunks:\");\n \n \tif (m->chunk_pack_names)\n-\t\tprintf(\" pack_names\");\n+\t\tprintf(\" pack-names\");\n \tif (m->chunk_oid_fanout)\n-\t\tprintf(\" oid_fanout\");\n+\t\tprintf(\" oid-fanout\");\n \tif (m->chunk_oid_lookup)\n-\t\tprintf(\" oid_lookup\");\n+\t\tprintf(\" oid-lookup\");\n \tif (m->chunk_object_offsets)\n-\t\tprintf(\" object_offsets\");\n+\t\tprintf(\" object-offsets\");\n \tif (m->chunk_large_offsets)\n-\t\tprintf(\" large_offsets\");\n+\t\tprintf(\" large-offsets\");\n \n \tprintf(\"\\nnum_objects: %d\\n\", m->num_objects);\n \n@@ -40,7 +37,7 @@ static int read_midx_file(const char *object_dir)\n \tfor (i = 0; i < m->num_packs; i++)\n \t\tprintf(\"%s\\n\", m->pack_names[i]);\n \n-\tprintf(\"object_dir: %s\\n\", m->object_dir);\n+\tprintf(\"object-dir: %s\\n\", m->object_dir);\n \n \treturn 0;\n }\n@@ -48,7 +45,7 @@ static int read_midx_file(const char *object_dir)\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (argc != 2)\n-\t\tusage(\"read-midx \");\n+\t\tusage(\"read-midx \");\n \n \treturn read_midx_file(argv[1]);\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 4c630ecab4..5ad6614465 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -11,17 +11,19 @@ midx_read_expect () {\n \tNUM_CHUNKS=$3\n \tOBJECT_DIR=$4\n \tEXTRA_CHUNKS=\"$5\"\n-\tcat >expect <<-EOF\n-\theader: 4d494458 1 $NUM_CHUNKS $NUM_PACKS\n-\tchunks: pack_names oid_fanout oid_lookup object_offsets$EXTRA_CHUNKS\n-\tnum_objects: $NUM_OBJECTS\n-\tpacks:\n-\tEOF\n-\tif [ $NUM_PACKS -ge 1 ]\n-\tthen\n-\t\tls $OBJECT_DIR/pack/ | grep idx | sort >> expect\n-\tfi\n-\tprintf \"object_dir: $OBJECT_DIR\\n\" >>expect &&\n+\t{\n+\t\tcat <<-EOF &&\n+\t\theader: 4d494458 1 $NUM_CHUNKS $NUM_PACKS\n+\t\tchunks: pack-names oid-fanout oid-lookup object-offsets$EXTRA_CHUNKS\n+\t\tnum_objects: $NUM_OBJECTS\n+\t\tpacks:\n+\t\tEOF\n+\t\tif test $NUM_PACKS -ge 1\n+\t\tthen\n+\t\t\tls $OBJECT_DIR/pack/ | grep idx | sort\n+\t\tfi &&\n+\t\tprintf \"object-dir: $OBJECT_DIR\\n\"\n+\t} >expect &&\n \ttest-tool read-midx $OBJECT_DIR >actual &&\n \ttest_cmp expect actual\n }\n@@ -32,35 +34,55 @@ test_expect_success 'write midx with no packs' '\n \tmidx_read_expect 0 0 4 .\n '\n \n+generate_objects () {\n+\ti=$1\n+\tiii=$(printf '%03i' $i)\n+\t{\n+\t\ttest-tool genrandom \"bar\" 200 &&\n+\t\ttest-tool genrandom \"baz $iii\" 50\n+\t} >wide_delta_$iii &&\n+\t{\n+\t\ttest-tool genrandom \"foo\"$i 100 &&\n+\t\ttest-tool genrandom \"foo\"$(( $i + 1 )) 100 &&\n+\t\ttest-tool genrandom \"foo\"$(( $i + 2 )) 100\n+\t} >deep_delta_$iii &&\n+\t{\n+\t\techo $iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192\n+\t} >file_$iii &&\n+\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii\n+}\n+\n+commit_and_list_objects () {\n+\t{\n+\t\techo 101 &&\n+\t\ttest-tool genrandom 100 8192;\n+\t} >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree -p HEADobj-list &&\n+\tgit reset --hard $commit\n+}\n+\n test_expect_success 'create objects' '\n+\ttest_commit initial &&\n \tfor i in $(test_seq 1 5)\n \tdo\n-\t\tiii=$(printf '%03i' $i)\n-\t\ttest-tool genrandom \"bar\" 200 >wide_delta_$iii &&\n-\t\ttest-tool genrandom \"baz $iii\" 50 >>wide_delta_$iii &&\n-\t\ttest-tool genrandom \"foo\"$i 100 >deep_delta_$iii &&\n-\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >>deep_delta_$iii &&\n-\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >>deep_delta_$iii &&\n-\t\techo $iii >file_$iii &&\n-\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n-\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n-\t\ti=$(expr $i + 1) || return 1\n+\t\tgenerate_objects $i\n \tdone &&\n-\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n-\tgit update-index --add file_101 &&\n-\ttree=$(git write-tree) &&\n-\tcommit=$(git commit-tree $tree obj-list &&\n-\tgit update-ref HEAD $commit\n+\tcommit_and_list_objects\n '\n \n test_expect_success 'write midx with one v1 pack' '\n-\tpack=$(git pack-objects --index-version=1 pack/test expect &&\n-\tgit -c core.multiPackIndex=true $1 >actual &&\n-\ttest_cmp expect actual\n-}\n-\n-compare_results_with_midx() {\n-\tMSG=$1\n-\ttest_expect_success \"check normal git operations: $MSG\" '\n-\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n-\t\tmidx_git_two_modes \"log --raw\" &&\n-\t\tmidx_git_two_modes \"log --oneline\"\n-\t'\n-}\n-\n compare_results_with_midx \"one v2 pack\"\n \n-test_expect_success 'Add more objects' '\n+test_expect_success 'add more objects' '\n \tfor i in $(test_seq 6 10)\n \tdo\n-\t\tiii=$(printf '%03i' $i)\n-\t\ttest-tool genrandom \"bar\" 200 >wide_delta_$iii &&\n-\t\ttest-tool genrandom \"baz $iii\" 50 >>wide_delta_$iii &&\n-\t\ttest-tool genrandom \"foo\"$i 100 >deep_delta_$iii &&\n-\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >>deep_delta_$iii &&\n-\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >>deep_delta_$iii &&\n-\t\techo $iii >file_$iii &&\n-\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n-\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n-\t\ti=$(expr $i + 1) || return 1\n+\t\tgenerate_objects $i\n \tdone &&\n-\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n-\tgit update-index --add file_101 &&\n-\ttree=$(git write-tree) &&\n-\tcommit=$(git commit-tree $tree -p HEADobj-list2 &&\n-\tgit update-ref HEAD $commit\n+\tcommit_and_list_objects\n '\n \n test_expect_success 'write midx with two packs' '\n-\tgit pack-objects --index-version=1 $objdir/pack/test-2 wide_delta_$iii &&\n-\t\ttest-tool genrandom \"baz $iii\" 50 >>wide_delta_$iii &&\n-\t\ttest-tool genrandom \"foo\"$i 100 >deep_delta_$iii &&\n-\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >>deep_delta_$iii &&\n-\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >>deep_delta_$iii &&\n-\t\techo $iii >file_$iii &&\n-\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n-\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n-\t\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n-\t\tgit update-index --add file_101 &&\n-\t\ttree=$(git write-tree) &&\n-\t\tcommit=$(git commit-tree $tree -p HEADobj-list &&\n-\t\tgit update-ref HEAD $commit &&\n-\t\tgit pack-objects --index-version=2 $objdir/pack/test-pack []\n-corrupt_data() {\n+corrupt_data () {\n \tfile=$1\n \tpos=$2\n \tdata=\"${3:-\\0}\"\n@@ -180,12 +163,17 @@ corrupt_data() {\n test_expect_success 'force some 64-bit offsets with pack-objects' '\n \tmkdir objects64 &&\n \tmkdir objects64/pack &&\n+\tfor i in $(test_seq 1 11)\n+\tdo\n+\t\tgenerate_objects 11\n+\tdone &&\n+\tcommit_and_list_objects &&\n \tpack64=$(git pack-objects --index-version=2,0x40 objects64/pack/test-64 \n---\n Documentation/technical/pack-format.txt | 49 +++++++++++++++++++++++++\n 1 file changed, 49 insertions(+)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 70a99fd142..e060e693f4 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -252,3 +252,52 @@ Pack file entry: <+\n corresponding packfile.\n \n 20-byte SHA-1-checksum of all of the above.\n+\n+== multi-pack-index (MIDX) files have the following format:\n+\n+The multi-pack-index files refer to multiple pack-files and loose objects.\n+\n+In order to allow extensions that add extra data to the MIDX, we organize\n+the body into \"chunks\" and provide a lookup table at the beginning of the\n+body. The header includes certain length values, such as the number of packs,\n+the number of base MIDX files, hash lengths and types.\n+\n+All 4-byte numbers are in network order.\n+\n+HEADER:\n+\n+\t4-byte signature:\n+\t The signature is: {'M', 'I', 'D', 'X'}\n+\n+\t1-byte version number:\n+\t Git only writes or recognizes version 1.\n+\n+\t1-byte Object Id Version\n+\t Git only writes or recognizes version 1 (SHA1).\n+\n+\t1-byte number of \"chunks\"\n+\n+\t1-byte number of base multi-pack-index files:\n+\t This value is currently always zero.\n+\n+\t4-byte number of pack files\n+\n+CHUNK LOOKUP:\n+\n+\t(C + 1) * 12 bytes providing the chunk offsets:\n+\t First 4 bytes describe chunk id. Value 0 is a terminating label.\n+\t Other 8 bytes provide offset in current file for chunk to start.\n+\t (Chunks are provided in file-order, so you can infer the length\n+\t using the next chunk position if necessary.)\n+\n+\tThe remaining data in the body is described one chunk at a time, and\n+\tthese chunks may be given in any order. Chunks are required unless\n+\totherwise specified.\n+\n+CHUNK DATA:\n+\n+\t(This section intentionally left incomplete.)\n+\n+TRAILER:\n+\n+\t20-byte SHA1-checksum of the above contents.\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352394","messageId":"20180712193940.21065-2-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 01/23] multi-pack-index: add design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:18Z","receivedAt":"2018-07-12T19:40:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/multi-pack-index.txt | 109 +++++++++++++++++++\n 1 file changed, 109 insertions(+)\n create mode 100644 Documentation/technical/multi-pack-index.txt\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nnew file mode 100644\nindex 0000000000..d7e57639f7\n--- /dev/null\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -0,0 +1,109 @@\n+Multi-Pack-Index (MIDX) Design Notes\n+====================================\n+\n+The Git object directory contains a 'pack' directory containing\n+packfiles (with suffix \".pack\") and pack-indexes (with suffix\n+\".idx\"). The pack-indexes provide a way to lookup objects and\n+navigate to their offset within the pack, but these must come\n+in pairs with the packfiles. This pairing depends on the file\n+names, as the pack-index differs only in suffix with its pack-\n+file. While the pack-indexes provide fast lookup per packfile,\n+this performance degrades as the number of packfiles increases,\n+because abbreviations need to inspect every packfile and we are\n+more likely to have a miss on our most-recently-used packfile.\n+For some large repositories, repacking into a single packfile\n+is not feasible due to storage space or excessive repack times.\n+\n+The multi-pack-index (MIDX for short) stores a list of objects\n+and their offsets into multiple packfiles. It contains:\n+\n+- A list of packfile names.\n+- A sorted list of object IDs.\n+- A list of metadata for the ith object ID including:\n+ - A value j referring to the jth packfile.\n+ - An offset within the jth packfile for the object.\n+- If large offsets are required, we use another list of large\n+ offsets similar to version 2 pack-indexes.\n+\n+Thus, we can provide O(log N) lookup time for any number\n+of packfiles.\n+\n+Design Details\n+--------------\n+\n+- The MIDX is stored in a file named 'multi-pack-index' in the\n+ .git/objects/pack directory. This could be stored in the pack\n+ directory of an alternate. It refers only to packfiles in that\n+ same directory.\n+\n+- The pack.multiIndex config setting must be on to consume MIDX files.\n+\n+- The file format includes parameters for the object ID hash\n+ function, so a future change of hash algorithm does not require\n+ a change in format.\n+\n+- The MIDX keeps only one record per object ID. If an object appears\n+ in multiple packfiles, then the MIDX selects the copy in the most-\n+ recently modified packfile.\n+\n+- If there exist packfiles in the pack directory not registered in\n+ the MIDX, then those packfiles are loaded into the `packed_git`\n+ list and `packed_git_mru` cache.\n+\n+- The pack-indexes (.idx files) remain in the pack directory so we\n+ can delete the MIDX file, set core.midx to false, or downgrade\n+ without any loss of information.\n+\n+- The MIDX file format uses a chunk-based approach (similar to the\n+ commit-graph file) that allows optional data to be added.\n+\n+Future Work\n+-----------\n+\n+- Add a 'verify' subcommand to the 'git midx' builtin to verify the\n+ contents of the multi-pack-index file match the offsets listed in\n+ the corresponding pack-indexes.\n+\n+- The multi-pack-index allows many packfiles, especially in a context\n+ where repacking is expensive (such as a very large repo), or\n+ unexpected maintenance time is unacceptable (such as a high-demand\n+ build machine). However, the multi-pack-index needs to be rewritten\n+ in full every time. We can extend the format to be incremental, so\n+ writes are fast. By storing a small \"tip\" multi-pack-index that\n+ points to large \"base\" MIDX files, we can keep writes fast while\n+ still reducing the number of binary searches required for object\n+ lookups.\n+\n+- The reachability bitmap is currently paired directly with a single\n+ packfile, using the pack-order as the object order to hopefully\n+ compress the bitmaps well using run-length encoding. This could be\n+ extended to pair a reachability bitmap with a multi-pack-index. If\n+ the multi-pack-index is extended to store a \"stable object order\"\n+ (a function Order(hash) = integer that is constant for a given hash,\n+ even as the multi-pack-index is updated) then a reachability bitmap\n+ could point to a multi-pack-index and be updated independently.\n+\n+- Packfiles can be marked as \"special\" using empty files that share\n+ the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n+ We can add an optional chunk of data to the multi-pack-index that\n+ records flags of information about the packfiles. This allows new\n+ states, such as 'repacked' or 'redeltified', that can help with\n+ pack maintenance in a multi-pack environment. It may also be\n+ helpful to organize packfiles by object type (commit, tree, blob,\n+ etc.) and use this metadata to help that maintenance.\n+\n+- The partial clone feature records special \"promisor\" packs that\n+ may point to objects that are not stored locally, but available\n+ on request to a server. The multi-pack-index does not currently\n+ track these promisor packs.\n+\n+Related Links\n+-------------\n+[0] https://bugs.chromium.org/p/git/issues/detail?id=6\n+ Chromium work item for: Multi-Pack Index (MIDX)\n+\n+[1] https://public-inbox.org/git/20180107181459.222909-1-dstolee@microsoft.com/\n+ An earlier RFC for the multi-pack-index feature\n+\n+[2] https://public-inbox.org/git/alpine.DEB.2.20.1803091557510.23109@alexmv-linux/\n+ Git Merge 2018 Contributor's summit notes (includes discussion of MIDX)\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352395","messageId":"20180712193940.21065-4-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 03/23] multi-pack-index: add builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:20Z","receivedAt":"2018-07-12T19:40:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This new 'git multi-pack-index' builtin will be the plumbing access\nfor writing, reading, and checking multi-pack-index files. The\ninitial implementation is a no-op.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 3 ++-\n Documentation/git-multi-pack-index.txt | 36 ++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/multi-pack-index.c | 34 ++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n 7 files changed, 76 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/git-multi-pack-index.txt\n create mode 100644 builtin/multi-pack-index.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 388cc4beee..25633bc515 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -99,8 +99,9 @@\n /git-mergetool--lib\n /git-mktag\n /git-mktree\n-/git-name-rev\n+/git-multi-pack-index\n /git-mv\n+/git-name-rev\n /git-notes\n /git-p4\n /git-pack-redundant\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nnew file mode 100644\nindex 0000000000..ec9982cbfc\n--- /dev/null\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -0,0 +1,36 @@\n+git-multi-pack-index(1)\n+======================\n+\n+NAME\n+----\n+git-multi-pack-index - Write and verify multi-pack-indexes\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git multi-pack-index' [--object-dir=]\n+\n+DESCRIPTION\n+-----------\n+Write or verify a multi-pack-index (MIDX) file.\n+\n+OPTIONS\n+-------\n+\n+--object-dir=::\n+\tUse given directory for the location of Git objects. We check\n+\t`/packs/multi-pack-index` for the current MIDX file, and\n+\t`/packs` for the pack-files to index.\n+\n+\n+SEE ALSO\n+--------\n+See link:technical/multi-pack-index.html[The Multi-Pack-Index Design\n+Document] and link:technical/pack-format.html[The Multi-Pack-Index\n+Format] for more information on the multi-pack-index feature.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex e4b503d259..54610875ec 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1047,6 +1047,7 @@ BUILTIN_OBJS += builtin/merge-recursive.o\n BUILTIN_OBJS += builtin/merge-tree.o\n BUILTIN_OBJS += builtin/mktag.o\n BUILTIN_OBJS += builtin/mktree.o\n+BUILTIN_OBJS += builtin/multi-pack-index.o\n BUILTIN_OBJS += builtin/mv.o\n BUILTIN_OBJS += builtin/name-rev.o\n BUILTIN_OBJS += builtin/notes.o\ndiff --git a/builtin.h b/builtin.h\nindex 4e0f64723e..70997d7ace 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -191,6 +191,7 @@ extern int cmd_merge_recursive(int argc, const char **argv, const char *prefix);\n extern int cmd_merge_tree(int argc, const char **argv, const char *prefix);\n extern int cmd_mktag(int argc, const char **argv, const char *prefix);\n extern int cmd_mktree(int argc, const char **argv, const char *prefix);\n+extern int cmd_multi_pack_index(int argc, const char **argv, const char *prefix);\n extern int cmd_mv(int argc, const char **argv, const char *prefix);\n extern int cmd_name_rev(int argc, const char **argv, const char *prefix);\n extern int cmd_notes(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nnew file mode 100644\nindex 0000000000..3161ddae86\n--- /dev/null\n+++ b/builtin/multi-pack-index.c\n@@ -0,0 +1,34 @@\n+#include \"builtin.h\"\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+\n+static char const * const builtin_multi_pack_index_usage[] = {\n+\tN_(\"git multi-pack-index [--object-dir=]\"),\n+\tNULL\n+};\n+\n+static struct opts_multi_pack_index {\n+\tconst char *object_dir;\n+} opts;\n+\n+int cmd_multi_pack_index(int argc, const char **argv,\n+\t\t\t const char *prefix)\n+{\n+\tstatic struct option builtin_multi_pack_index_options[] = {\n+\t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n+\t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_multi_pack_index_options,\n+\t\t\t builtin_multi_pack_index_usage, 0);\n+\n+\tif (!opts.object_dir)\n+\t\topts.object_dir = get_object_directory();\n+\n+\treturn 0;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex e1c26c1bb7..61071f8fa2 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -123,6 +123,7 @@ git-merge-index plumbingmanipulators\n git-merge-one-file purehelpers\n git-mergetool ancillarymanipulators complete\n git-merge-tree ancillaryinterrogators\n+git-multi-pack-index plumbingmanipulators\n git-mktag plumbingmanipulators\n git-mktree plumbingmanipulators\n git-mv mainporcelain worktree\ndiff --git a/git.c b/git.c\nindex c2f48d53dd..a7509fa5f7 100644\n--- a/git.c\n+++ b/git.c\n@@ -505,6 +505,7 @@ static struct cmd_struct commands[] = {\n \t{ \"merge-tree\", cmd_merge_tree, RUN_SETUP | NO_PARSEOPT },\n \t{ \"mktag\", cmd_mktag, RUN_SETUP | NO_PARSEOPT },\n \t{ \"mktree\", cmd_mktree, RUN_SETUP },\n+\t{ \"multi-pack-index\", cmd_multi_pack_index, RUN_SETUP_GENTLY },\n \t{ \"mv\", cmd_mv, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"name-rev\", cmd_name_rev, RUN_SETUP },\n \t{ \"notes\", cmd_notes, RUN_SETUP },\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352396","messageId":"20180712193940.21065-5-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 04/23] multi-pack-index: add 'write' verb","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:21Z","receivedAt":"2018-07-12T19:40:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of writing multi-pack-indexes, add a skeleton\n'git multi-pack-index write' subcommand and send the options to a\nwrite_midx_file() method. Also create a skeleton test script that\ntests the 'write' subcommand.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 22 +++++++++++++++++++++-\n Makefile | 1 +\n builtin/multi-pack-index.c | 17 +++++++++++++++--\n midx.c | 7 +++++++\n midx.h | 6 ++++++\n t/t5319-multi-pack-index.sh | 10 ++++++++++\n 6 files changed, 60 insertions(+), 3 deletions(-)\n create mode 100644 midx.c\n create mode 100644 midx.h\n create mode 100755 t/t5319-multi-pack-index.sh\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex ec9982cbfc..a62af1caca 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=]\n+'git multi-pack-index' [--object-dir=] \n \n DESCRIPTION\n -----------\n@@ -23,6 +23,26 @@ OPTIONS\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n \n+write::\n+\tWhen given as the verb, write a new MIDX file to\n+\t`/packs/multi-pack-index`.\n+\n+\n+EXAMPLES\n+--------\n+\n+* Write a MIDX file for the packfiles in the current .git folder.\n++\n+-----------------------------------------------\n+$ git multi-pack-index write\n+-----------------------------------------------\n+\n+* Write a MIDX file for the packfiles in an alternate object store.\n++\n+-----------------------------------------------\n+$ git multi-pack-index --object-dir write\n+-----------------------------------------------\n+\n \n SEE ALSO\n --------\ndiff --git a/Makefile b/Makefile\nindex 54610875ec..f5636c711d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -890,6 +890,7 @@ LIB_OBJS += merge.o\n LIB_OBJS += merge-blobs.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += mergesort.o\n+LIB_OBJS += midx.o\n LIB_OBJS += name-hash.o\n LIB_OBJS += notes.o\n LIB_OBJS += notes-cache.o\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 3161ddae86..6a7aa00cf2 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -2,9 +2,10 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"parse-options.h\"\n+#include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=]\"),\n+\tN_(\"git multi-pack-index [--object-dir=] write\"),\n \tNULL\n };\n \n@@ -30,5 +31,17 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \n-\treturn 0;\n+\tif (argc == 0)\n+\t\tgoto usage;\n+\n+\tif (!strcmp(argv[0], \"write\")) {\n+\t\tif (argc > 1)\n+\t\t\tgoto usage;\n+\n+\t\treturn write_midx_file(opts.object_dir);\n+\t}\n+\n+usage:\n+\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t builtin_multi_pack_index_options);\n }\ndiff --git a/midx.c b/midx.c\nnew file mode 100644\nindex 0000000000..32468db1a2\n--- /dev/null\n+++ b/midx.c\n@@ -0,0 +1,7 @@\n+#include \"cache.h\"\n+#include \"midx.h\"\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nnew file mode 100644\nindex 0000000000..dbdbe9f873\n--- /dev/null\n+++ b/midx.h\n@@ -0,0 +1,6 @@\n+#ifndef __MIDX_H__\n+#define __MIDX_H__\n+\n+int write_midx_file(const char *object_dir);\n+\n+#endif\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nnew file mode 100755\nindex 0000000000..ec3ddbe79c\n--- /dev/null\n+++ b/t/t5319-multi-pack-index.sh\n@@ -0,0 +1,10 @@\n+#!/bin/sh\n+\n+test_description='multi-pack-indexes'\n+. ./test-lib.sh\n+\n+test_expect_success 'write midx with no packs' '\n+\tgit multi-pack-index --object-dir=. write\n+'\n+\n+test_done\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352398","messageId":"20180712193940.21065-7-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 06/23] multi-pack-index: load into memory","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:23Z","receivedAt":"2018-07-12T19:40:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Create a new multi_pack_index struct for loading multi-pack-indexes into\nmemory. Create a test-tool builtin for reading basic information about\nthat multi-pack-index to verify the correct data is written.\n\nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n midx.c | 79 +++++++++++++++++++++++++++++++++++++\n midx.h | 18 +++++++++\n object-store.h | 2 +\n t/helper/test-read-midx.c | 31 +++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5319-multi-pack-index.sh | 11 +++++-\n 8 files changed, 143 insertions(+), 1 deletion(-)\n create mode 100644 t/helper/test-read-midx.c\n\ndiff --git a/Makefile b/Makefile\nindex f5636c711d..0b801d1b16 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -717,6 +717,7 @@ TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n TEST_BUILTINS_OBJS += test-prio-queue.o\n TEST_BUILTINS_OBJS += test-read-cache.o\n+TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\ndiff --git a/midx.c b/midx.c\nindex f85f2d334d..c1ff5acf85 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,18 +1,97 @@\n #include \"cache.h\"\n #include \"csum-file.h\"\n #include \"lockfile.h\"\n+#include \"object-store.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n+#define MIDX_BYTE_FILE_VERSION 4\n+#define MIDX_BYTE_HASH_VERSION 5\n+#define MIDX_BYTE_NUM_CHUNKS 6\n+#define MIDX_BYTE_NUM_PACKS 8\n #define MIDX_HASH_VERSION 1\n #define MIDX_HEADER_SIZE 12\n+#define MIDX_HASH_LEN 20\n+#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m = NULL;\n+\tint fd;\n+\tstruct stat st;\n+\tsize_t midx_size;\n+\tvoid *midx_map = NULL;\n+\tuint32_t hash_version;\n+\tchar *midx_name = get_midx_filename(object_dir);\n+\n+\tfd = git_open(midx_name);\n+\n+\tif (fd < 0)\n+\t\tgoto cleanup_fail;\n+\tif (fstat(fd, &st)) {\n+\t\terror_errno(_(\"failed to read %s\"), midx_name);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tmidx_size = xsize_t(st.st_size);\n+\n+\tif (midx_size < MIDX_MIN_SIZE) {\n+\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tFREE_AND_NULL(midx_name);\n+\n+\tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\n+\tFLEX_ALLOC_MEM(m, object_dir, object_dir, strlen(object_dir));\n+\tm->fd = fd;\n+\tm->data = midx_map;\n+\tm->data_len = midx_size;\n+\n+\tm->signature = get_be32(m->data);\n+\tif (m->signature != MIDX_SIGNATURE) {\n+\t\terror(_(\"multi-pack-index signature 0x%08x does not match signature 0x%08x\"),\n+\t\t m->signature, MIDX_SIGNATURE);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\tm->version = m->data[MIDX_BYTE_FILE_VERSION];\n+\tif (m->version != MIDX_VERSION) {\n+\t\terror(_(\"multi-pack-index version %d not recognized\"),\n+\t\t m->version);\n+\t\tgoto cleanup_fail;\n+\t}\n+\n+\thash_version = m->data[MIDX_BYTE_HASH_VERSION];\n+\tif (hash_version != MIDX_HASH_VERSION) {\n+\t\terror(_(\"hash version %u does not match\"), hash_version);\n+\t\tgoto cleanup_fail;\n+\t}\n+\tm->hash_len = MIDX_HASH_LEN;\n+\n+\tm->num_chunks = m->data[MIDX_BYTE_NUM_CHUNKS];\n+\n+\tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n+\n+\treturn m;\n+\n+cleanup_fail:\n+\tfree(m);\n+\tfree(midx_name);\n+\tif (midx_map)\n+\t\tmunmap(midx_map, midx_size);\n+\tif (0 <= fd)\n+\t\tclose(fd);\n+\treturn NULL;\n+}\n+\n static size_t write_midx_header(struct hashfile *f,\n \t\t\t\tunsigned char num_chunks,\n \t\t\t\tuint32_t num_packs)\ndiff --git a/midx.h b/midx.h\nindex dbdbe9f873..0e05051bca 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -1,6 +1,24 @@\n #ifndef __MIDX_H__\n #define __MIDX_H__\n \n+struct multi_pack_index {\n+\tint fd;\n+\n+\tconst unsigned char *data;\n+\tsize_t data_len;\n+\n+\tuint32_t signature;\n+\tunsigned char version;\n+\tunsigned char hash_len;\n+\tunsigned char num_chunks;\n+\tuint32_t num_packs;\n+\tuint32_t num_objects;\n+\n+\tchar object_dir[FLEX_ARRAY];\n+};\n+\n+struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n+\n int write_midx_file(const char *object_dir);\n \n #endif\ndiff --git a/object-store.h b/object-store.h\nindex d683112fd7..13a766aea8 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -84,6 +84,8 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+struct multi_pack_index;\n+\n struct raw_object_store {\n \t/*\n \t * Path to the repository's object store.\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nnew file mode 100644\nindex 0000000000..988a487169\n--- /dev/null\n+++ b/t/helper/test-read-midx.c\n@@ -0,0 +1,31 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"midx.h\"\n+#include \"repository.h\"\n+#include \"object-store.h\"\n+\n+static int read_midx_file(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir);\n+\n+\tif (!m)\n+\t\treturn 1;\n+\n+\tprintf(\"header: %08x %d %d %d\\n\",\n+\t m->signature,\n+\t m->version,\n+\t m->num_chunks,\n+\t m->num_packs);\n+\n+\tprintf(\"object-dir: %s\\n\", m->object_dir);\n+\n+\treturn 0;\n+}\n+\n+int cmd__read_midx(int argc, const char **argv)\n+{\n+\tif (argc != 2)\n+\t\tusage(\"read-midx \");\n+\n+\treturn read_midx_file(argv[1]);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 805a45de9c..1c3ab36e6c 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -27,6 +27,7 @@ static struct test_cmd cmds[] = {\n \t{ \"path-utils\", cmd__path_utils },\n \t{ \"prio-queue\", cmd__prio_queue },\n \t{ \"read-cache\", cmd__read_cache },\n+\t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"regex\", cmd__regex },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 7116ddfb94..6af8c08a66 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -21,6 +21,7 @@ int cmd__online_cpus(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n int cmd__prio_queue(int argc, const char **argv);\n int cmd__read_cache(int argc, const char **argv);\n+int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 50e80f8f2c..506bd8abb8 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,10 +3,19 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n+midx_read_expect () {\n+\tcat >expect <<-EOF\n+\theader: 4d494458 1 0 0\n+\tobject-dir: .\n+\tEOF\n+\ttest-tool read-midx . >actual &&\n+\ttest_cmp expect actual\n+}\n+\n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm -f pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\ttest_path_is_file pack/multi-pack-index\n+\tmidx_read_expect\n '\n \n test_done\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352397","messageId":"20180712193940.21065-6-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 05/23] midx: write header information to lockfile","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:22Z","receivedAt":"2018-07-12T19:40:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As we begin writing the multi-pack-index format to disk, start with\nthe basics: the 12-byte header and the 20-byte checksum footer. Start\nwith these basics so we can add the rest of the format in small\nincrements.\n\nAs we implement the format, we will use a technique to check that our\ncomputed offsets within the multi-pack-index file match what we are\nactually writing. Each method that writes to the hashfile will return\nthe number of bytes written, and we will track that those values match\nour expectations.\n\nCurrently, write_midx_header() returns 12, but is not checked. We will\ncheck the return value in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 50 +++++++++++++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh | 4 ++-\n 2 files changed, 53 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 32468db1a2..f85f2d334d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,7 +1,57 @@\n #include \"cache.h\"\n+#include \"csum-file.h\"\n+#include \"lockfile.h\"\n #include \"midx.h\"\n \n+#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n+#define MIDX_VERSION 1\n+#define MIDX_HASH_VERSION 1\n+#define MIDX_HEADER_SIZE 12\n+\n+static char *get_midx_filename(const char *object_dir)\n+{\n+\treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n+}\n+\n+static size_t write_midx_header(struct hashfile *f,\n+\t\t\t\tunsigned char num_chunks,\n+\t\t\t\tuint32_t num_packs)\n+{\n+\tunsigned char byte_values[4];\n+\n+\thashwrite_be32(f, MIDX_SIGNATURE);\n+\tbyte_values[0] = MIDX_VERSION;\n+\tbyte_values[1] = MIDX_HASH_VERSION;\n+\tbyte_values[2] = num_chunks;\n+\tbyte_values[3] = 0; /* unused */\n+\thashwrite(f, byte_values, sizeof(byte_values));\n+\thashwrite_be32(f, num_packs);\n+\n+\treturn MIDX_HEADER_SIZE;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n+\tunsigned char num_chunks = 0;\n+\tchar *midx_name;\n+\tstruct hashfile *f = NULL;\n+\tstruct lock_file lk;\n+\n+\tmidx_name = get_midx_filename(object_dir);\n+\tif (safe_create_leading_directories(midx_name)) {\n+\t\tUNLEAK(midx_name);\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t midx_name);\n+\t}\n+\n+\thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n+\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\tFREE_AND_NULL(midx_name);\n+\n+\twrite_midx_header(f, num_chunks, 0);\n+\n+\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tcommit_lock_file(&lk);\n+\n \treturn 0;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex ec3ddbe79c..50e80f8f2c 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -4,7 +4,9 @@ test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n test_expect_success 'write midx with no packs' '\n-\tgit multi-pack-index --object-dir=. write\n+\ttest_when_finished rm -f pack/multi-pack-index &&\n+\tgit multi-pack-index --object-dir=. write &&\n+\ttest_path_is_file pack/multi-pack-index\n '\n \n test_done\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352399","messageId":"20180712193940.21065-8-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 07/23] t5319: expand test data","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:24Z","receivedAt":"2018-07-12T19:40:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As we build the multi-pack-index file format, we want to test the format\non real repositories. Add tests that create repository data including\nmultiple packfiles with both version 1 and version 2 formats.\n\nThe current 'git multi-pack-index write' command will always write the\nsame file with no \"real\" data. This will be expanded in future commits,\nalong with the test expectations.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 84 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 84 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 506bd8abb8..1240127ec1 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -18,4 +18,88 @@ test_expect_success 'write midx with no packs' '\n \tmidx_read_expect\n '\n \n+generate_objects () {\n+\ti=$1\n+\tiii=$(printf '%03i' $i)\n+\t{\n+\t\ttest-tool genrandom \"bar\" 200 &&\n+\t\ttest-tool genrandom \"baz $iii\" 50\n+\t} >wide_delta_$iii &&\n+\t{\n+\t\ttest-tool genrandom \"foo\"$i 100 &&\n+\t\ttest-tool genrandom \"foo\"$(( $i + 1 )) 100 &&\n+\t\ttest-tool genrandom \"foo\"$(( $i + 2 )) 100\n+\t} >deep_delta_$iii &&\n+\t{\n+\t\techo $iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192\n+\t} >file_$iii &&\n+\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii\n+}\n+\n+commit_and_list_objects () {\n+\t{\n+\t\techo 101 &&\n+\t\ttest-tool genrandom 100 8192;\n+\t} >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree -p HEADobj-list &&\n+\tgit reset --hard $commit\n+}\n+\n+test_expect_success 'create objects' '\n+\ttest_commit initial &&\n+\tfor i in $(test_seq 1 5)\n+\tdo\n+\t\tgenerate_objects $i\n+\tdone &&\n+\tcommit_and_list_objects\n+'\n+\n+test_expect_success 'write midx with one v1 pack' '\n+\tpack=$(git pack-objects --index-version=1 pack/test \n---\n packfile.c | 101 +++++++++++++++++++++++++++++++++--------------------\n packfile.h | 6 ++++\n 2 files changed, 69 insertions(+), 38 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 7cd45aa4b2..ee1ab9b804 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -738,13 +738,14 @@ static void report_pack_garbage(struct string_list *list)\n \treport_helper(list, seen_bits, first, list->nr);\n }\n \n-static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tsize_t dirnamelen;\n \tDIR *dir;\n \tstruct dirent *de;\n-\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \n \tstrbuf_addstr(&path, objdir);\n \tstrbuf_addstr(&path, \"/pack\");\n@@ -759,53 +760,77 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n \tstrbuf_addch(&path, '/');\n \tdirnamelen = path.len;\n \twhile ((de = readdir(dir)) != NULL) {\n-\t\tstruct packed_git *p;\n-\t\tsize_t base_len;\n-\n \t\tif (is_dot_or_dotdot(de->d_name))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&path, dirnamelen);\n \t\tstrbuf_addstr(&path, de->d_name);\n \n-\t\tbase_len = path.len;\n-\t\tif (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n-\t\t\t/* Don't reopen a pack we already have. */\n-\t\t\tfor (p = r->objects->packed_git; p;\n-\t\t\t p = p->next) {\n-\t\t\t\tsize_t len;\n-\t\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n-\t\t\t\t len == base_len &&\n-\t\t\t\t !memcmp(p->pack_name, path.buf, len))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tif (p == NULL &&\n-\t\t\t /*\n-\t\t\t * See if it really is a valid .idx file with\n-\t\t\t * corresponding .pack file that we can map.\n-\t\t\t */\n-\t\t\t (p = add_packed_git(path.buf, path.len, local)) != NULL)\n-\t\t\t\tinstall_packed_git(r, p);\n-\t\t}\n-\n-\t\tif (!report_garbage)\n-\t\t\tcontinue;\n-\n-\t\tif (ends_with(de->d_name, \".idx\") ||\n-\t\t ends_with(de->d_name, \".pack\") ||\n-\t\t ends_with(de->d_name, \".bitmap\") ||\n-\t\t ends_with(de->d_name, \".keep\") ||\n-\t\t ends_with(de->d_name, \".promisor\"))\n-\t\t\tstring_list_append(&garbage, path.buf);\n-\t\telse\n-\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path.buf);\n+\t\tfn(path.buf, path.len, de->d_name, data);\n \t}\n+\n \tclosedir(dir);\n-\treport_pack_garbage(&garbage);\n-\tstring_list_clear(&garbage, 0);\n \tstrbuf_release(&path);\n }\n \n+struct prepare_pack_data {\n+\tstruct repository *r;\n+\tstruct string_list *garbage;\n+\tint local;\n+};\n+\n+static void prepare_pack(const char *full_name, size_t full_name_len,\n+\t\t\t const char *file_name, void *_data)\n+{\n+\tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n+\tstruct packed_git *p;\n+\tsize_t base_len = full_name_len;\n+\n+\tif (strip_suffix_mem(full_name, &base_len, \".idx\")) {\n+\t\t/* Don't reopen a pack we already have. */\n+\t\tfor (p = data->r->objects->packed_git; p; p = p->next) {\n+\t\t\tsize_t len;\n+\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n+\t\t\t len == base_len &&\n+\t\t\t !memcmp(p->pack_name, full_name, len))\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!p) {\n+\t\t\tp = add_packed_git(full_name, full_name_len, data->local);\n+\t\t\tif (p)\n+\t\t\t\tinstall_packed_git(data->r, p);\n+\t\t}\n+\t}\n+\n+\tif (!report_garbage)\n+\t\treturn;\n+\n+\tif (ends_with(file_name, \".idx\") ||\n+\t ends_with(file_name, \".pack\") ||\n+\t ends_with(file_name, \".bitmap\") ||\n+\t ends_with(file_name, \".keep\") ||\n+\t ends_with(file_name, \".promisor\"))\n+\t\tstring_list_append(data->garbage, full_name);\n+\telse\n+\t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\n+}\n+\n+static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+{\n+\tstruct prepare_pack_data data;\n+\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n+\n+\tdata.r = r;\n+\tdata.garbage = &garbage;\n+\tdata.local = local;\n+\n+\tfor_each_file_in_pack_dir(objdir, prepare_pack, &data);\n+\n+\treport_pack_garbage(data.garbage);\n+\tstring_list_clear(data.garbage, 0);\n+}\n+\n static void prepare_packed_git(struct repository *r);\n /*\n * Give a fast, rough count of the number of objects in the repository. This\ndiff --git a/packfile.h b/packfile.h\nindex e0a38aba93..d2ad30300a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -28,6 +28,12 @@ extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n+typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len,\n+\t\t\t\t const char *file_pach, void *data);\n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data);\n+\n /* A hook to report invalid files in pack directory */\n #define PACKDIR_FILE_PACK 1\n #define PACKDIR_FILE_IDX 2\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352401","messageId":"20180712193940.21065-10-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 09/23] multi-pack-index: read packfile list","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:26Z","receivedAt":"2018-07-12T19:40:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When constructing a multi-pack-index file for a given object directory,\nread the files within the enclosed pack directory and find matches that\nend with \".idx\" and find the correct paired packfile using\nadd_packed_git().\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 48 ++++++++++++++++++++++++++++++++++++-\n t/t5319-multi-pack-index.sh | 15 ++++++------\n 2 files changed, 55 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex c1ff5acf85..f742d7ccd7 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,6 +1,8 @@\n #include \"cache.h\"\n #include \"csum-file.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n+#include \"packfile.h\"\n #include \"object-store.h\"\n #include \"midx.h\"\n \n@@ -109,12 +111,41 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+struct pack_list {\n+\tstruct packed_git **list;\n+\tuint32_t nr;\n+\tuint32_t alloc_list;\n+};\n+\n+static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n+\t\t\t const char *file_name, void *data)\n+{\n+\tstruct pack_list *packs = (struct pack_list *)data;\n+\n+\tif (ends_with(file_name, \".idx\")) {\n+\t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n+\n+\t\tpacks->list[packs->nr] = add_packed_git(full_path,\n+\t\t\t\t\t\t\tfull_path_len,\n+\t\t\t\t\t\t\t0);\n+\t\tif (!packs->list[packs->nr]) {\n+\t\t\twarning(_(\"failed to add packfile '%s'\"),\n+\t\t\t\tfull_path);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tpacks->nr++;\n+\t}\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char num_chunks = 0;\n \tchar *midx_name;\n+\tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n+\tstruct pack_list packs;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -123,14 +154,29 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n+\tpacks.nr = 0;\n+\tpacks.alloc_list = 16;\n+\tpacks.list = NULL;\n+\tALLOC_ARRAY(packs.list, packs.alloc_list);\n+\n+\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\twrite_midx_header(f, num_chunks, 0);\n+\twrite_midx_header(f, num_chunks, packs.nr);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n \n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tif (packs.list[i]) {\n+\t\t\tclose_pack(packs.list[i]);\n+\t\t\tfree(packs.list[i]);\n+\t\t}\n+\t}\n+\n+\tfree(packs.list);\n \treturn 0;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 1240127ec1..54117a7f49 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -4,8 +4,9 @@ test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n midx_read_expect () {\n+\tNUM_PACKS=$1\n \tcat >expect <<-EOF\n-\theader: 4d494458 1 0 0\n+\theader: 4d494458 1 0 $NUM_PACKS\n \tobject-dir: .\n \tEOF\n \ttest-tool read-midx . >actual &&\n@@ -15,7 +16,7 @@ midx_read_expect () {\n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm -f pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\tmidx_read_expect\n+\tmidx_read_expect 0\n '\n \n generate_objects () {\n@@ -65,13 +66,13 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test \n---\n midx.c | 17 +++++++++++++++++\n midx.h | 1 +\n t/helper/test-read-midx.c | 5 +++++\n t/t5319-multi-pack-index.sh | 17 ++++++++++++-----\n 4 files changed, 35 insertions(+), 5 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex ca7a32bf95..fcdf6553ce 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -37,6 +37,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tuint32_t hash_version;\n \tchar *midx_name = get_midx_filename(object_dir);\n \tuint32_t i;\n+\tconst char *cur_pack_name;\n \n \tfd = git_open(midx_name);\n \n@@ -115,6 +116,22 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tif (!m->chunk_pack_names)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n \n+\tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n+\n+\tcur_pack_name = (const char *)m->chunk_pack_names;\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tm->pack_names[i] = cur_pack_name;\n+\n+\t\tcur_pack_name += strlen(cur_pack_name) + 1;\n+\n+\t\tif (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0) {\n+\t\t\terror(_(\"multi-pack-index pack names out of order: '%s' before '%s'\"),\n+\t\t\t m->pack_names[i - 1],\n+\t\t\t m->pack_names[i]);\n+\t\t\tgoto cleanup_fail;\n+\t\t}\n+\t}\n+\n \treturn m;\n \n cleanup_fail:\ndiff --git a/midx.h b/midx.h\nindex 38af01fa3b..17b56172e3 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -16,6 +16,7 @@ struct multi_pack_index {\n \n \tconst unsigned char *chunk_pack_names;\n \n+\tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 3f2d2cfa78..76a60d7882 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -6,6 +6,7 @@\n \n static int read_midx_file(const char *object_dir)\n {\n+\tuint32_t i;\n \tstruct multi_pack_index *m = load_multi_pack_index(object_dir);\n \n \tif (!m)\n@@ -24,6 +25,10 @@ static int read_midx_file(const char *object_dir)\n \n \tprintf(\"\\n\");\n \n+\tprintf(\"packs:\\n\");\n+\tfor (i = 0; i < m->num_packs; i++)\n+\t\tprintf(\"%s\\n\", m->pack_names[i]);\n+\n \tprintf(\"object-dir: %s\\n\", m->object_dir);\n \n \treturn 0;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 7512d55c92..e8da082c64 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -5,11 +5,18 @@ test_description='multi-pack-indexes'\n \n midx_read_expect () {\n \tNUM_PACKS=$1\n-\tcat >expect <<-EOF\n-\theader: 4d494458 1 1 $NUM_PACKS\n-\tchunks: pack-names\n-\tobject-dir: .\n-\tEOF\n+\t{\n+\t\tcat <<-EOF &&\n+\t\theader: 4d494458 1 1 $NUM_PACKS\n+\t\tchunks: pack-names\n+\t\tpacks:\n+\t\tEOF\n+\t\tif test $NUM_PACKS -ge 1\n+\t\tthen\n+\t\t\tls pack/ | grep idx | sort\n+\t\tfi &&\n+\t\tprintf \"object-dir: .\\n\"\n+\t} >expect &&\n \ttest-tool read-midx . >actual &&\n \ttest_cmp expect actual\n }\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352403","messageId":"20180712193940.21065-11-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 10/23] multi-pack-index: write pack names in chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:27Z","receivedAt":"2018-07-12T19:40:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index needs to track which packfiles it indexes. Store\nthese in our first required chunk. Since filenames are not well\nstructured, add padding to keep good alignment in later chunks.\n\nModify the 'git multi-pack-index read' subcommand to output the\nexistence of the pack-file name chunk. Modify t5319-multi-pack-index.sh\nto reflect this new output and the new expected number of chunks.\n\nDefense in depth: A pattern we are using in the multi-pack-index feature\nis to verify the data as we write it. We want to ensure we never write\ninvalid data to the multi-pack-index. There are many checks that verify\nthat the values we are writing fit the format definitions. This mainly\nhelps developers while working on the feature, but it can also identify\nissues that only appear when dealing with very large data sets. These\nlarge sets are hard to encode into test cases.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 6 +\n midx.c | 174 +++++++++++++++++++++++-\n midx.h | 2 +\n t/helper/test-read-midx.c | 7 +\n t/t5319-multi-pack-index.sh | 3 +-\n 5 files changed, 189 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex e060e693f4..6c5a77475f 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -296,6 +296,12 @@ CHUNK LOOKUP:\n \n CHUNK DATA:\n \n+\tPackfile Names (ID: {'P', 'N', 'A', 'M'})\n+\t Stores the packfile names as concatenated, null-terminated strings.\n+\t Packfiles must be listed in lexicographic order for fast lookups by\n+\t name. This is the only chunk not guaranteed to be a multiple of four\n+\t bytes in length, so should be the last chunk for alignment reasons.\n+\n \t(This section intentionally left incomplete.)\n \n TRAILER:\ndiff --git a/midx.c b/midx.c\nindex f742d7ccd7..ca7a32bf95 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -17,6 +17,11 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n+#define MIDX_MAX_CHUNKS 1\n+#define MIDX_CHUNK_ALIGNMENT 4\n+#define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n@@ -31,6 +36,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tvoid *midx_map = NULL;\n \tuint32_t hash_version;\n \tchar *midx_name = get_midx_filename(object_dir);\n+\tuint32_t i;\n \n \tfd = git_open(midx_name);\n \n@@ -82,6 +88,33 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n \n+\tfor (i = 0; i < m->num_chunks; i++) {\n+\t\tuint32_t chunk_id = get_be32(m->data + MIDX_HEADER_SIZE +\n+\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n+\t\tuint64_t chunk_offset = get_be64(m->data + MIDX_HEADER_SIZE + 4 +\n+\t\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n+\n+\t\tswitch (chunk_id) {\n+\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n+\t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase 0:\n+\t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\t/*\n+\t\t\t\t * Do nothing on unrecognized chunks, allowing future\n+\t\t\t\t * extensions to add optional chunks.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (!m->chunk_pack_names)\n+\t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n+\n \treturn m;\n \n cleanup_fail:\n@@ -113,8 +146,11 @@ static size_t write_midx_header(struct hashfile *f,\n \n struct pack_list {\n \tstruct packed_git **list;\n+\tchar **names;\n \tuint32_t nr;\n \tuint32_t alloc_list;\n+\tuint32_t alloc_names;\n+\tsize_t pack_name_concat_len;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -124,6 +160,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \tif (ends_with(file_name, \".idx\")) {\n \t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n+\t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n \n \t\tpacks->list[packs->nr] = add_packed_git(full_path,\n \t\t\t\t\t\t\tfull_path_len,\n@@ -134,18 +171,89 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\treturn;\n \t\t}\n \n+\t\tpacks->names[packs->nr] = xstrdup(file_name);\n+\t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n \t}\n }\n \n+struct pack_pair {\n+\tuint32_t pack_int_id;\n+\tchar *pack_name;\n+};\n+\n+static int pack_pair_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_pair *a = (struct pack_pair *)_a;\n+\tstruct pack_pair *b = (struct pack_pair *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n+static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n+{\n+\tuint32_t i;\n+\tstruct pack_pair *pairs;\n+\n+\tALLOC_ARRAY(pairs, nr_packs);\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\tpairs[i].pack_int_id = i;\n+\t\tpairs[i].pack_name = pack_names[i];\n+\t}\n+\n+\tQSORT(pairs, nr_packs, pack_pair_compare);\n+\n+\tfor (i = 0; i < nr_packs; i++) {\n+\t\tpack_names[i] = pairs[i].pack_name;\n+\t\tperm[pairs[i].pack_int_id] = i;\n+\t}\n+\n+\tfree(pairs);\n+}\n+\n+static size_t write_midx_pack_names(struct hashfile *f,\n+\t\t\t\t char **pack_names,\n+\t\t\t\t uint32_t num_packs)\n+{\n+\tuint32_t i;\n+\tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < num_packs; i++) {\n+\t\tsize_t writelen = strlen(pack_names[i]) + 1;\n+\n+\t\tif (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n+\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n+\t\t\t pack_names[i - 1],\n+\t\t\t pack_names[i]);\n+\n+\t\thashwrite(f, pack_names[i], writelen);\n+\t\twritten += writelen;\n+\t}\n+\n+\t/* add padding to be aligned */\n+\ti = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n+\tif (i < MIDX_CHUNK_ALIGNMENT) {\n+\t\tmemset(padding, 0, sizeof(padding));\n+\t\thashwrite(f, padding, i);\n+\t\twritten += i;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n-\tunsigned char num_chunks = 0;\n+\tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct pack_list packs;\n+\tuint32_t *pack_perm = NULL;\n+\tuint64_t written = 0;\n+\tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n+\tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -156,16 +264,76 @@ int write_midx_file(const char *object_dir)\n \n \tpacks.nr = 0;\n \tpacks.alloc_list = 16;\n+\tpacks.alloc_names = 16;\n \tpacks.list = NULL;\n+\tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n+\tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n+\tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n+\tALLOC_ARRAY(pack_perm, packs.nr);\n+\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\twrite_midx_header(f, num_chunks, packs.nr);\n+\tcur_chunk = 0;\n+\tnum_chunks = 1;\n+\n+\twritten = write_midx_header(f, num_chunks, packs.nr);\n+\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n+\tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n+\n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = 0;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n+\n+\tfor (i = 0; i <= num_chunks; i++) {\n+\t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n+\t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n+\t\t\t chunk_offsets[i - 1],\n+\t\t\t chunk_offsets[i]);\n+\n+\t\tif (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n+\t\t\tBUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n+\t\t\t chunk_offsets[i]);\n+\n+\t\thashwrite_be32(f, chunk_ids[i]);\n+\t\thashwrite_be32(f, chunk_offsets[i] >> 32);\n+\t\thashwrite_be32(f, chunk_offsets[i]);\n+\n+\t\twritten += MIDX_CHUNKLOOKUP_WIDTH;\n+\t}\n+\n+\tfor (i = 0; i < num_chunks; i++) {\n+\t\tif (written != chunk_offsets[i])\n+\t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n+\t\t\t chunk_offsets[i],\n+\t\t\t written,\n+\t\t\t chunk_ids[i]);\n+\n+\t\tswitch (chunk_ids[i]) {\n+\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n+\t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n+\t\t\t\t chunk_ids[i]);\n+\t\t}\n+\t}\n+\n+\tif (written != chunk_offsets[num_chunks])\n+\t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n+\t\t written,\n+\t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n@@ -175,8 +343,10 @@ int write_midx_file(const char *object_dir)\n \t\t\tclose_pack(packs.list[i]);\n \t\t\tfree(packs.list[i]);\n \t\t}\n+\t\tfree(packs.names[i]);\n \t}\n \n \tfree(packs.list);\n+\tfree(packs.names);\n \treturn 0;\n }\ndiff --git a/midx.h b/midx.h\nindex 0e05051bca..38af01fa3b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -14,6 +14,8 @@ struct multi_pack_index {\n \tuint32_t num_packs;\n \tuint32_t num_objects;\n \n+\tconst unsigned char *chunk_pack_names;\n+\n \tchar object_dir[FLEX_ARRAY];\n };\n \ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 988a487169..3f2d2cfa78 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -17,6 +17,13 @@ static int read_midx_file(const char *object_dir)\n \t m->num_chunks,\n \t m->num_packs);\n \n+\tprintf(\"chunks:\");\n+\n+\tif (m->chunk_pack_names)\n+\t\tprintf(\" pack-names\");\n+\n+\tprintf(\"\\n\");\n+\n \tprintf(\"object-dir: %s\\n\", m->object_dir);\n \n \treturn 0;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 54117a7f49..7512d55c92 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -6,7 +6,8 @@ test_description='multi-pack-indexes'\n midx_read_expect () {\n \tNUM_PACKS=$1\n \tcat >expect <<-EOF\n-\theader: 4d494458 1 0 $NUM_PACKS\n+\theader: 4d494458 1 1 $NUM_PACKS\n+\tchunks: pack-names\n \tobject-dir: .\n \tEOF\n \ttest-tool read-midx . >actual &&\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352405","messageId":"20180712193940.21065-13-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 12/23] midx: sort and deduplicate objects from packfiles","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:29Z","receivedAt":"2018-07-12T19:40:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before writing a list of objects and their offsets to a multi-pack-index,\nwe need to collect the list of objects contained in the packfiles. There\nmay be multiple copies of some objects, so this list must be deduplicated.\n\nIt is possible to artificially get into a state where there are many\nduplicate copies of objects. That can create high memory pressure if we\nare to create a list of all objects before de-duplication. To reduce\nthis memory pressure without a significant performance drop,\nautomatically group objects by the first byte of their object id. Use\nthe IDX fanout tables to group the data, copy to a local array, then\nsort.\n\nCopy only the de-duplicated entries. Select the duplicate based on the\nmost-recent modified time of a packfile containing the object.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 128 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.c | 17 +++++++\n packfile.h | 2 +\n 3 files changed, 147 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex fcdf6553ce..29f8de5ee6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -4,6 +4,7 @@\n #include \"lockfile.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"packfile.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n@@ -182,12 +183,21 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\tpacks->list[packs->nr] = add_packed_git(full_path,\n \t\t\t\t\t\t\tfull_path_len,\n \t\t\t\t\t\t\t0);\n+\n \t\tif (!packs->list[packs->nr]) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n+\t\tif (open_pack_index(packs->list[packs->nr])) {\n+\t\t\twarning(_(\"failed to open pack-index '%s'\"),\n+\t\t\t\tfull_path);\n+\t\t\tclose_pack(packs->list[packs->nr]);\n+\t\t\tFREE_AND_NULL(packs->list[packs->nr]);\n+\t\t\treturn;\n+\t\t}\n+\n \t\tpacks->names[packs->nr] = xstrdup(file_name);\n \t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n@@ -228,6 +238,119 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n \tfree(pairs);\n }\n \n+struct pack_midx_entry {\n+\tstruct object_id oid;\n+\tuint32_t pack_int_id;\n+\ttime_t pack_mtime;\n+\tuint64_t offset;\n+};\n+\n+static int midx_oid_compare(const void *_a, const void *_b)\n+{\n+\tconst struct pack_midx_entry *a = (const struct pack_midx_entry *)_a;\n+\tconst struct pack_midx_entry *b = (const struct pack_midx_entry *)_b;\n+\tint cmp = oidcmp(&a->oid, &b->oid);\n+\n+\tif (cmp)\n+\t\treturn cmp;\n+\n+\tif (a->pack_mtime > b->pack_mtime)\n+\t\treturn -1;\n+\telse if (a->pack_mtime < b->pack_mtime)\n+\t\treturn 1;\n+\n+\treturn a->pack_int_id - b->pack_int_id;\n+}\n+\n+static void fill_pack_entry(uint32_t pack_int_id,\n+\t\t\t struct packed_git *p,\n+\t\t\t uint32_t cur_object,\n+\t\t\t struct pack_midx_entry *entry)\n+{\n+\tif (!nth_packed_object_oid(&entry->oid, p, cur_object))\n+\t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n+\n+\tentry->pack_int_id = pack_int_id;\n+\tentry->pack_mtime = p->mtime;\n+\n+\tentry->offset = nth_packed_object_offset(p, cur_object);\n+}\n+\n+/*\n+ * It is possible to artificially get into a state where there are many\n+ * duplicate copies of objects. That can create high memory pressure if\n+ * we are to create a list of all objects before de-duplication. To reduce\n+ * this memory pressure without a significant performance drop, automatically\n+ * group objects by the first byte of their object id. Use the IDX fanout\n+ * tables to group the data, copy to a local array, then sort.\n+ *\n+ * Copy only the de-duplicated entries (selected by most-recent modified time\n+ * of a packfile containing the object).\n+ */\n+static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n+\t\t\t\t\t\t uint32_t *perm,\n+\t\t\t\t\t\t uint32_t nr_packs,\n+\t\t\t\t\t\t uint32_t *nr_objects)\n+{\n+\tuint32_t cur_fanout, cur_pack, cur_object;\n+\tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n+\tstruct pack_midx_entry *entries_by_fanout = NULL;\n+\tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\n+\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++)\n+\t\ttotal_objects += p[cur_pack]->num_objects;\n+\n+\t/*\n+\t * As we de-duplicate by fanout value, we expect the fanout\n+\t * slices to be evenly distributed, with some noise. Hence,\n+\t * allocate slightly more than one 256th.\n+\t */\n+\talloc_objects = alloc_fanout = total_objects > 3200 ? total_objects / 200 : 16;\n+\n+\tALLOC_ARRAY(entries_by_fanout, alloc_fanout);\n+\tALLOC_ARRAY(deduplicated_entries, alloc_objects);\n+\t*nr_objects = 0;\n+\n+\tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n+\t\tuint32_t nr_fanout = 0;\n+\n+\t\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\t\tuint32_t start = 0, end;\n+\n+\t\t\tif (cur_fanout)\n+\t\t\t\tstart = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n+\t\t\tend = get_pack_fanout(p[cur_pack], cur_fanout);\n+\n+\t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n+\t\t\t\tfill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tnr_fanout++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tQSORT(entries_by_fanout, nr_fanout, midx_oid_compare);\n+\n+\t\t/*\n+\t\t * The batch is now sorted by OID and then mtime (descending).\n+\t\t * Take only the first duplicate.\n+\t\t */\n+\t\tfor (cur_object = 0; cur_object < nr_fanout; cur_object++) {\n+\t\t\tif (cur_object && !oidcmp(&entries_by_fanout[cur_object - 1].oid,\n+\t\t\t\t\t\t &entries_by_fanout[cur_object].oid))\n+\t\t\t\tcontinue;\n+\n+\t\t\tALLOC_GROW(deduplicated_entries, *nr_objects + 1, alloc_objects);\n+\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n+\t\t\t &entries_by_fanout[cur_object],\n+\t\t\t sizeof(struct pack_midx_entry));\n+\t\t\t(*nr_objects)++;\n+\t\t}\n+\t}\n+\n+\tfree(entries_by_fanout);\n+\treturn deduplicated_entries;\n+}\n+\n static size_t write_midx_pack_names(struct hashfile *f,\n \t\t\t\t char **pack_names,\n \t\t\t\t uint32_t num_packs)\n@@ -271,6 +394,8 @@ int write_midx_file(const char *object_dir)\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n+\tuint32_t nr_entries;\n+\tstruct pack_midx_entry *entries = NULL;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -296,6 +421,8 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n+\tentries = get_sorted_entries(packs.list, pack_perm, packs.nr, &nr_entries);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n@@ -365,5 +492,6 @@ int write_midx_file(const char *object_dir)\n \n \tfree(packs.list);\n \tfree(packs.names);\n+\tfree(entries);\n \treturn 0;\n }\ndiff --git a/packfile.c b/packfile.c\nindex ee1ab9b804..3d652212c6 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -196,6 +196,23 @@ int open_pack_index(struct packed_git *p)\n \treturn ret;\n }\n \n+uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)\n+{\n+\tconst uint32_t *level1_ofs = p->index_data;\n+\n+\tif (!level1_ofs) {\n+\t\tif (open_pack_index(p))\n+\t\t\treturn 0;\n+\t\tlevel1_ofs = p->index_data;\n+\t}\n+\n+\tif (p->index_version > 1) {\n+\t\tlevel1_ofs += 2;\n+\t}\n+\n+\treturn ntohl(level1_ofs[value]);\n+}\n+\n static struct packed_git *alloc_packed_git(int extra)\n {\n \tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\ndiff --git a/packfile.h b/packfile.h\nindex d2ad30300a..b0eed44c0b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -69,6 +69,8 @@ extern int open_pack_index(struct packed_git *);\n */\n extern void close_pack_index(struct packed_git *);\n \n+extern uint32_t get_pack_fanout(struct packed_git *p, uint32_t value);\n+\n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void close_pack_windows(struct packed_git *);\n extern void close_pack(struct packed_git *);\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352404","messageId":"20180712193940.21065-14-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 13/23] midx: write object ids in a chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:30Z","receivedAt":"2018-07-12T19:40:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 4 +++\n midx.c | 47 +++++++++++++++++++++++--\n midx.h | 1 +\n t/helper/test-read-midx.c | 2 ++\n t/t5319-multi-pack-index.sh | 4 +--\n 5 files changed, 53 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 6c5a77475f..78ee0489c6 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -302,6 +302,10 @@ CHUNK DATA:\n \t name. This is the only chunk not guaranteed to be a multiple of four\n \t bytes in length, so should be the last chunk for alignment reasons.\n \n+\tOID Lookup (ID: {'O', 'I', 'D', 'L'})\n+\t The OIDs for all objects in the MIDX are stored in lexicographic\n+\t order in this chunk.\n+\n \t(This section intentionally left incomplete.)\n \n TRAILER:\ndiff --git a/midx.c b/midx.c\nindex 29f8de5ee6..3f113e1beb 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -18,9 +18,10 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 1\n+#define MIDX_MAX_CHUNKS 2\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n \n static char *get_midx_filename(const char *object_dir)\n@@ -101,6 +102,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n+\t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n \t\t\tcase 0:\n \t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n \t\t\t\tbreak;\n@@ -116,6 +121,8 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tif (!m->chunk_pack_names)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n+\tif (!m->chunk_oid_lookup)\n+\t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n \n@@ -382,6 +389,32 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \treturn written;\n }\n \n+static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n+\t\t\t\t struct pack_midx_entry *objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tuint32_t i;\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\tif (i < nr_objects - 1) {\n+\t\t\tstruct pack_midx_entry *next = list;\n+\t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n+\t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n+\t\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t\t oid_to_hex(&next->oid));\n+\t\t}\n+\n+\t\thashwrite(f, obj->oid.hash, (int)hash_len);\n+\t\twritten += hash_len;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n@@ -428,7 +461,7 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 1;\n+\tnum_chunks = 2;\n \n \twritten = write_midx_header(f, num_chunks, packs.nr);\n \n@@ -436,9 +469,13 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = 0;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n \n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = 0;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n+\n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n \t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n@@ -468,6 +505,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n+\t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tdefault:\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\ndiff --git a/midx.h b/midx.h\nindex 17b56172e3..4d3bceafc5 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -15,6 +15,7 @@ struct multi_pack_index {\n \tuint32_t num_objects;\n \n \tconst unsigned char *chunk_pack_names;\n+\tconst unsigned char *chunk_oid_lookup;\n \n \tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 76a60d7882..de6d452a7c 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -22,6 +22,8 @@ static int read_midx_file(const char *object_dir)\n \n \tif (m->chunk_pack_names)\n \t\tprintf(\" pack-names\");\n+\tif (m->chunk_oid_lookup)\n+\t\tprintf(\" oid-lookup\");\n \n \tprintf(\"\\n\");\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex e8da082c64..4813610115 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -7,8 +7,8 @@ midx_read_expect () {\n \tNUM_PACKS=$1\n \t{\n \t\tcat <<-EOF &&\n-\t\theader: 4d494458 1 1 $NUM_PACKS\n-\t\tchunks: pack-names\n+\t\theader: 4d494458 1 2 $NUM_PACKS\n+\t\tchunks: pack-names oid-lookup\n \t\tpacks:\n \t\tEOF\n \t\tif test $NUM_PACKS -ge 1\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352406","messageId":"20180712193940.21065-15-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 14/23] midx: write object id fanout chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:31Z","receivedAt":"2018-07-12T19:40:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n Documentation/technical/pack-format.txt | 5 +++\n midx.c | 53 +++++++++++++++++++++++--\n midx.h | 1 +\n t/helper/test-read-midx.c | 4 +-\n t/t5319-multi-pack-index.sh | 16 ++++----\n 5 files changed, 68 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 78ee0489c6..3215f7bfcd 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -302,6 +302,11 @@ CHUNK DATA:\n \t name. This is the only chunk not guaranteed to be a multiple of four\n \t bytes in length, so should be the last chunk for alignment reasons.\n \n+\tOID Fanout (ID: {'O', 'I', 'D', 'F'})\n+\t The ith entry, F[i], stores the number of OIDs with first\n+\t byte at most i. Thus F[255] stores the total\n+\t number of objects.\n+\n \tOID Lookup (ID: {'O', 'I', 'D', 'L'})\n \t The OIDs for all objects in the MIDX are stored in lexicographic\n \t order in this chunk.\ndiff --git a/midx.c b/midx.c\nindex 3f113e1beb..7a954eb0cd 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -18,11 +18,13 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 2\n+#define MIDX_MAX_CHUNKS 3\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n+#define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n \n static char *get_midx_filename(const char *object_dir)\n {\n@@ -102,6 +104,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n+\t\t\t\tm->chunk_oid_fanout = (uint32_t *)(m->data + chunk_offset);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n \t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n \t\t\t\tbreak;\n@@ -121,9 +127,13 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \n \tif (!m->chunk_pack_names)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n+\tif (!m->chunk_oid_fanout)\n+\t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n \tif (!m->chunk_oid_lookup)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n \n+\tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n+\n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n \n \tcur_pack_name = (const char *)m->chunk_pack_names;\n@@ -389,6 +399,35 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \treturn written;\n }\n \n+static size_t write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t struct pack_midx_entry *objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tstruct pack_midx_entry *last = objects + nr_objects;\n+\tuint32_t count = 0;\n+\tuint32_t i;\n+\n+\t/*\n+\t* Write the first-level table (the list is sorted,\n+\t* but we use a 256-entry lookup to be able to avoid\n+\t* having to do eight extra binary search iterations).\n+\t*/\n+\tfor (i = 0; i < 256; i++) {\n+\t\tstruct pack_midx_entry *next = list;\n+\n+\t\twhile (next < last && next->oid.hash[0] == i) {\n+\t\t\tcount++;\n+\t\t\tnext++;\n+\t\t}\n+\n+\t\thashwrite_be32(f, count);\n+\t\tlist = next;\n+\t}\n+\n+\treturn MIDX_CHUNK_FANOUT_SIZE;\n+}\n+\n static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n \t\t\t\t struct pack_midx_entry *objects,\n \t\t\t\t uint32_t nr_objects)\n@@ -461,7 +500,7 @@ int write_midx_file(const char *object_dir)\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 2;\n+\tnum_chunks = 3;\n \n \twritten = write_midx_header(f, num_chunks, packs.nr);\n \n@@ -469,9 +508,13 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n \n+\tcur_chunk++;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n+\n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = 0;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n@@ -505,6 +548,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n+\t\t\t\twritten += write_midx_oid_fanout(f, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n \t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n \t\t\t\tbreak;\ndiff --git a/midx.h b/midx.h\nindex 4d3bceafc5..8572cf0f4b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -15,6 +15,7 @@ struct multi_pack_index {\n \tuint32_t num_objects;\n \n \tconst unsigned char *chunk_pack_names;\n+\tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \n \tconst char **pack_names;\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex de6d452a7c..f7c17b0940 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -22,10 +22,12 @@ static int read_midx_file(const char *object_dir)\n \n \tif (m->chunk_pack_names)\n \t\tprintf(\" pack-names\");\n+\tif (m->chunk_oid_fanout)\n+\t\tprintf(\" oid-fanout\");\n \tif (m->chunk_oid_lookup)\n \t\tprintf(\" oid-lookup\");\n \n-\tprintf(\"\\n\");\n+\tprintf(\"\\nnum_objects: %d\\n\", m->num_objects);\n \n \tprintf(\"packs:\\n\");\n \tfor (i = 0; i < m->num_packs; i++)\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 4813610115..1cf609958f 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -5,10 +5,12 @@ test_description='multi-pack-indexes'\n \n midx_read_expect () {\n \tNUM_PACKS=$1\n+\tNUM_OBJECTS=$2\n \t{\n \t\tcat <<-EOF &&\n-\t\theader: 4d494458 1 2 $NUM_PACKS\n-\t\tchunks: pack-names oid-lookup\n+\t\theader: 4d494458 1 3 $NUM_PACKS\n+\t\tchunks: pack-names oid-fanout oid-lookup\n+\t\tnum_objects: $NUM_OBJECTS\n \t\tpacks:\n \t\tEOF\n \t\tif test $NUM_PACKS -ge 1\n@@ -24,7 +26,7 @@ midx_read_expect () {\n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm -f pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\tmidx_read_expect 0\n+\tmidx_read_expect 0 0\n '\n \n generate_objects () {\n@@ -74,13 +76,13 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test \n---\n Documentation/technical/pack-format.txt | 15 +++-\n midx.c | 100 +++++++++++++++++++++++-\n midx.h | 2 +\n t/helper/test-read-midx.c | 4 +\n t/t5319-multi-pack-index.sh | 49 +++++++++---\n 5 files changed, 155 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 3215f7bfcd..cab5bdd2ff 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -311,7 +311,20 @@ CHUNK DATA:\n \t The OIDs for all objects in the MIDX are stored in lexicographic\n \t order in this chunk.\n \n-\t(This section intentionally left incomplete.)\n+\tObject Offsets (ID: {'O', 'O', 'F', 'F'})\n+\t Stores two 4-byte values for every object.\n+\t 1: The pack-int-id for the pack storing this object.\n+\t 2: The offset within the pack.\n+\t\tIf all offsets are less than 2^31, then the large offset chunk\n+\t\twill not exist and offsets are stored as in IDX v1.\n+\t\tIf there is at least one offset value larger than 2^32-1, then\n+\t\tthe large offset chunk must exist. If the large offset chunk\n+\t\texists and the 31st bit is on, then removing that bit reveals\n+\t\tthe row in the large offsets containing the 8-byte offset of\n+\t\tthis object.\n+\n+\t[Optional] Object Large Offsets (ID: {'L', 'O', 'F', 'F'})\n+\t 8-byte offsets into large packfiles.\n \n TRAILER:\n \ndiff --git a/midx.c b/midx.c\nindex 7a954eb0cd..e83110ae92 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -18,13 +18,18 @@\n #define MIDX_HASH_LEN 20\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + MIDX_HASH_LEN)\n \n-#define MIDX_MAX_CHUNKS 3\n+#define MIDX_MAX_CHUNKS 5\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n+#define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n+#define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n+#define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n+#define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n+#define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n static char *get_midx_filename(const char *object_dir)\n {\n@@ -112,6 +117,14 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n+\t\t\t\tm->chunk_object_offsets = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n+\t\t\t\tm->chunk_large_offsets = m->data + chunk_offset;\n+\t\t\t\tbreak;\n+\n \t\t\tcase 0:\n \t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n \t\t\t\tbreak;\n@@ -131,6 +144,8 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n \tif (!m->chunk_oid_lookup)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n+\tif (!m->chunk_object_offsets)\n+\t\tdie(_(\"multi-pack-index missing required object offsets chunk\"));\n \n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n@@ -454,6 +469,56 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n \treturn written;\n }\n \n+static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n+\t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tuint32_t i, nr_large_offset = 0;\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\thashwrite_be32(f, obj->pack_int_id);\n+\n+\t\tif (large_offset_needed && obj->offset >> 31)\n+\t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n+\t\telse if (!large_offset_needed && obj->offset >> 32)\n+\t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n+\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t obj->offset);\n+\t\telse\n+\t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n+\n+\t\twritten += MIDX_CHUNK_OFFSET_WIDTH;\n+\t}\n+\n+\treturn written;\n+}\n+\n+static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_offset,\n+\t\t\t\t struct pack_midx_entry *objects, uint32_t nr_objects)\n+{\n+\tstruct pack_midx_entry *list = objects;\n+\tsize_t written = 0;\n+\n+\twhile (nr_large_offset) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\t\tuint64_t offset = obj->offset;\n+\n+\t\tif (!(offset >> 31))\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be32(f, offset >> 32);\n+\t\thashwrite_be32(f, offset & 0xffffffffUL);\n+\t\twritten += 2 * sizeof(uint32_t);\n+\n+\t\tnr_large_offset--;\n+\t}\n+\n+\treturn written;\n+}\n+\n int write_midx_file(const char *object_dir)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n@@ -466,8 +531,9 @@ int write_midx_file(const char *object_dir)\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t nr_entries;\n+\tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n+\tint large_offsets_needed = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -494,13 +560,19 @@ int write_midx_file(const char *object_dir)\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n \tentries = get_sorted_entries(packs.list, pack_perm, packs.nr, &nr_entries);\n+\tfor (i = 0; i < nr_entries; i++) {\n+\t\tif (entries[i].offset > 0x7fffffff)\n+\t\t\tnum_large_offsets++;\n+\t\tif (entries[i].offset > 0xffffffff)\n+\t\t\tlarge_offsets_needed = 1;\n+\t}\n \n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n \tcur_chunk = 0;\n-\tnum_chunks = 3;\n+\tnum_chunks = large_offsets_needed ? 5 : 4;\n \n \twritten = write_midx_header(f, num_chunks, packs.nr);\n \n@@ -516,9 +588,21 @@ int write_midx_file(const char *object_dir)\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n \n \tcur_chunk++;\n-\tchunk_ids[cur_chunk] = 0;\n+\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_HASH_LEN;\n \n+\tcur_chunk++;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_CHUNK_OFFSET_WIDTH;\n+\tif (large_offsets_needed) {\n+\t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n+\n+\t\tcur_chunk++;\n+\t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n+\t\t\t\t\t num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n+\t}\n+\n+\tchunk_ids[cur_chunk] = 0;\n+\n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n \t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n@@ -556,6 +640,14 @@ int write_midx_file(const char *object_dir)\n \t\t\t\twritten += write_midx_oid_lookup(f, MIDX_HASH_LEN, entries, nr_entries);\n \t\t\t\tbreak;\n \n+\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n+\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n+\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, entries, nr_entries);\n+\t\t\t\tbreak;\n+\n \t\t\tdefault:\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\ndiff --git a/midx.h b/midx.h\nindex 8572cf0f4b..e15966272f 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -17,6 +17,8 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_pack_names;\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n+\tconst unsigned char *chunk_object_offsets;\n+\tconst unsigned char *chunk_large_offsets;\n \n \tconst char **pack_names;\n \tchar object_dir[FLEX_ARRAY];\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex f7c17b0940..8e19972e89 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -26,6 +26,10 @@ static int read_midx_file(const char *object_dir)\n \t\tprintf(\" oid-fanout\");\n \tif (m->chunk_oid_lookup)\n \t\tprintf(\" oid-lookup\");\n+\tif (m->chunk_object_offsets)\n+\t\tprintf(\" object-offsets\");\n+\tif (m->chunk_large_offsets)\n+\t\tprintf(\" large-offsets\");\n \n \tprintf(\"\\nnum_objects: %d\\n\", m->num_objects);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 1cf609958f..4a4fa26f7a 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -6,27 +6,30 @@ test_description='multi-pack-indexes'\n midx_read_expect () {\n \tNUM_PACKS=$1\n \tNUM_OBJECTS=$2\n+\tNUM_CHUNKS=$3\n+\tOBJECT_DIR=$4\n+\tEXTRA_CHUNKS=\"$5\"\n \t{\n \t\tcat <<-EOF &&\n-\t\theader: 4d494458 1 3 $NUM_PACKS\n-\t\tchunks: pack-names oid-fanout oid-lookup\n+\t\theader: 4d494458 1 $NUM_CHUNKS $NUM_PACKS\n+\t\tchunks: pack-names oid-fanout oid-lookup object-offsets$EXTRA_CHUNKS\n \t\tnum_objects: $NUM_OBJECTS\n \t\tpacks:\n \t\tEOF\n \t\tif test $NUM_PACKS -ge 1\n \t\tthen\n-\t\t\tls pack/ | grep idx | sort\n+\t\t\tls $OBJECT_DIR/pack/ | grep idx | sort\n \t\tfi &&\n-\t\tprintf \"object-dir: .\\n\"\n+\t\tprintf \"object-dir: $OBJECT_DIR\\n\"\n \t} >expect &&\n-\ttest-tool read-midx . >actual &&\n+\ttest-tool read-midx $OBJECT_DIR >actual &&\n \ttest_cmp expect actual\n }\n \n test_expect_success 'write midx with no packs' '\n \ttest_when_finished rm -f pack/multi-pack-index &&\n \tgit multi-pack-index --object-dir=. write &&\n-\tmidx_read_expect 0 0\n+\tmidx_read_expect 0 0 4 .\n '\n \n generate_objects () {\n@@ -76,13 +79,13 @@ test_expect_success 'write midx with one v1 pack' '\n \tpack=$(git pack-objects --index-version=1 pack/test []\n+corrupt_data () {\n+\tfile=$1\n+\tpos=$2\n+\tdata=\"${3:-\\0}\"\n+\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n+}\n+\n+# Force 64-bit offsets by manipulating the idx file.\n+# This makes the IDX file _incorrect_ so be careful to clean up after!\n+test_expect_success 'force some 64-bit offsets with pack-objects' '\n+\tmkdir objects64 &&\n+\tmkdir objects64/pack &&\n+\tfor i in $(test_seq 1 11)\n+\tdo\n+\t\tgenerate_objects 11\n+\tdone &&\n+\tcommit_and_list_objects &&\n+\tpack64=$(git pack-objects --index-version=2,0x40 objects64/pack/test-64 \n\nmidx: prepare midxed_git struct\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 5 ++++\n midx.c | 25 ++++++++++++++++++\n midx.h | 5 ++++\n object-store.h | 7 +++++\n packfile.c | 6 ++++-\n t/t5319-multi-pack-index.sh | 51 +++++++++++++++++++++++++++----------\n 6 files changed, 85 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ab641bf5a9..25f817ca42 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -908,6 +908,11 @@ core.commitGraph::\n \tEnable git commit graph feature. Allows reading from the\n \tcommit-graph file.\n \n+core.multiPackIndex::\n+\tUse the multi-pack-index file to track multiple packfiles using a\n+\tsingle index. See link:technical/multi-pack-index.html[the\n+\tmulti-pack-index design document].\n+\n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n \tlinkgit:git-read-tree[1] for more information.\ndiff --git a/midx.c b/midx.c\nindex e83110ae92..4090cf4ca4 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"csum-file.h\"\n #include \"dir.h\"\n #include \"lockfile.h\"\n@@ -177,6 +178,30 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \treturn NULL;\n }\n \n+int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n+{\n+\tstruct multi_pack_index *m = r->objects->multi_pack_index;\n+\tstruct multi_pack_index *m_search;\n+\tint config_value;\n+\n+\tif (repo_config_get_bool(r, \"core.multipackindex\", &config_value) ||\n+\t !config_value)\n+\t\treturn 0;\n+\n+\tfor (m_search = m; m_search; m_search = m_search->next)\n+\t\tif (!strcmp(object_dir, m_search->object_dir))\n+\t\t\treturn 1;\n+\n+\tr->objects->multi_pack_index = load_multi_pack_index(object_dir);\n+\n+\tif (r->objects->multi_pack_index) {\n+\t\tr->objects->multi_pack_index->next = m;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static size_t write_midx_header(struct hashfile *f,\n \t\t\t\tunsigned char num_chunks,\n \t\t\t\tuint32_t num_packs)\ndiff --git a/midx.h b/midx.h\nindex e15966272f..9bcfc82d2e 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -1,7 +1,11 @@\n #ifndef __MIDX_H__\n #define __MIDX_H__\n \n+#include \"repository.h\"\n+\n struct multi_pack_index {\n+\tstruct multi_pack_index *next;\n+\n \tint fd;\n \n \tconst unsigned char *data;\n@@ -25,6 +29,7 @@ struct multi_pack_index {\n };\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n+int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n \ndiff --git a/object-store.h b/object-store.h\nindex 13a766aea8..c2b162489a 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -105,6 +105,13 @@ struct raw_object_store {\n \t */\n \tstruct oidmap *replace_map;\n \n+\t/*\n+\t * private data\n+\t *\n+\t * should only be accessed directly by packfile.c and midx.c\n+\t */\n+\tstruct multi_pack_index *multi_pack_index;\n+\n \t/*\n \t * private data\n \t *\ndiff --git a/packfile.c b/packfile.c\nindex 3d652212c6..5d4493dbf4 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -15,6 +15,7 @@\n #include \"tree-walk.h\"\n #include \"tree.h\"\n #include \"object-store.h\"\n+#include \"midx.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -935,10 +936,13 @@ static void prepare_packed_git(struct repository *r)\n \n \tif (r->objects->packed_git_initialized)\n \t\treturn;\n+\tprepare_multi_pack_index_one(r, r->objects->objectdir);\n \tprepare_packed_git_one(r, r->objects->objectdir, 1);\n \tprepare_alt_odb(r);\n-\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next)\n+\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\tprepare_multi_pack_index_one(r, alt->path);\n \t\tprepare_packed_git_one(r, alt->path, 0);\n+\t}\n \trearrange_packed_git(r);\n \tprepare_packed_git_mru(r);\n \tr->objects->packed_git_initialized = 1;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 4a4fa26f7a..601e28a2f0 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,6 +3,8 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n+objdir=.git/objects\n+\n midx_read_expect () {\n \tNUM_PACKS=$1\n \tNUM_OBJECTS=$2\n@@ -76,18 +78,35 @@ test_expect_success 'create objects' '\n '\n \n test_expect_success 'write midx with one v1 pack' '\n-\tpack=$(git pack-objects --index-version=1 pack/test expect &&\n+\tgit -c core.multiPackIndex=true $1 >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx() {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\"\n+\t'\n+}\n+\n test_expect_success 'write midx with one v2 pack' '\n-\tgit pack-objects --index-version=2,0x40 pack/test []\n corrupt_data () {\n \tfile=$1\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352409","messageId":"20180712193940.21065-18-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 17/23] midx: read objects from multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:34Z","receivedAt":"2018-07-12T19:40:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++-\n midx.h | 3 ++\n packfile.c | 8 ++++-\n 3 files changed, 100 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 4090cf4ca4..182535933c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -5,7 +5,7 @@\n #include \"lockfile.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n-#include \"packfile.h\"\n+#include \"sha1-lookup.h\"\n #include \"midx.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n@@ -151,6 +151,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n+\tm->packs = xcalloc(m->num_packs, sizeof(*m->packs));\n \n \tcur_pack_name = (const char *)m->chunk_pack_names;\n \tfor (i = 0; i < m->num_packs; i++) {\n@@ -178,6 +179,94 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \treturn NULL;\n }\n \n+static int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id)\n+{\n+\tstruct strbuf pack_name = STRBUF_INIT;\n+\n+\tif (pack_int_id >= m->num_packs)\n+\t\tBUG(\"bad pack-int-id\");\n+\n+\tif (m->packs[pack_int_id])\n+\t\treturn 0;\n+\n+\tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n+\t\t m->pack_names[pack_int_id]);\n+\n+\tm->packs[pack_int_id] = add_packed_git(pack_name.buf, pack_name.len, 1);\n+\tstrbuf_release(&pack_name);\n+\treturn !m->packs[pack_int_id];\n+}\n+\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)\n+{\n+\treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n+\t\t\t MIDX_HASH_LEN, result);\n+}\n+\n+static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n+{\n+\tconst unsigned char *offset_data;\n+\tuint32_t offset32;\n+\n+\toffset_data = m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH;\n+\toffset32 = get_be32(offset_data + sizeof(uint32_t));\n+\n+\tif (m->chunk_large_offsets && offset32 & MIDX_LARGE_OFFSET_NEEDED) {\n+\t\tif (sizeof(offset32) < sizeof(uint64_t))\n+\t\t\tdie(_(\"multi-pack-index stores a 64-bit offset, but off_t is too small\"));\n+\n+\t\toffset32 ^= MIDX_LARGE_OFFSET_NEEDED;\n+\t\treturn get_be64(m->chunk_large_offsets + sizeof(uint64_t) * offset32);\n+\t}\n+\n+\treturn offset32;\n+}\n+\n+static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n+{\n+\treturn get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);\n+}\n+\n+static int nth_midxed_pack_entry(struct multi_pack_index *m, struct pack_entry *e, uint32_t pos)\n+{\n+\tuint32_t pack_int_id;\n+\tstruct packed_git *p;\n+\n+\tif (pos >= m->num_objects)\n+\t\treturn 0;\n+\n+\tpack_int_id = nth_midxed_pack_int_id(m, pos);\n+\n+\tif (prepare_midx_pack(m, pack_int_id))\n+\t\tdie(_(\"error preparing packfile from multi-pack-index\"));\n+\tp = m->packs[pack_int_id];\n+\n+\t/*\n+\t* We are about to tell the caller where they can locate the\n+\t* requested object. We better make sure the packfile is\n+\t* still here and can be accessed before supplying that\n+\t* answer, as it may have been deleted since the MIDX was\n+\t* loaded!\n+\t*/\n+\tif (!is_pack_valid(p))\n+\t\treturn 0;\n+\n+\te->offset = nth_midxed_offset(m, pos);\n+\te->p = p;\n+\n+\treturn 1;\n+}\n+\n+int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m)\n+{\n+\tuint32_t pos;\n+\n+\tif (!bsearch_midx(oid, m, &pos))\n+\t\treturn 0;\n+\n+\treturn nth_midxed_pack_entry(m, e, pos);\n+}\n+\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n {\n \tstruct multi_pack_index *m = r->objects->multi_pack_index;\ndiff --git a/midx.h b/midx.h\nindex 9bcfc82d2e..377838c9ca 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -25,10 +25,13 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_large_offsets;\n \n \tconst char **pack_names;\n+\tstruct packed_git **packs;\n \tchar object_dir[FLEX_ARRAY];\n };\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\ndiff --git a/packfile.c b/packfile.c\nindex 5d4493dbf4..bc763d91b9 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1902,11 +1902,17 @@ static int fill_pack_entry(const struct object_id *oid,\n int find_pack_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e)\n {\n \tstruct list_head *pos;\n+\tstruct multi_pack_index *m;\n \n \tprepare_packed_git(r);\n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packed_git && !r->objects->multi_pack_index)\n \t\treturn 0;\n \n+\tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n+\t\tif (fill_midx_entry(oid, e, m))\n+\t\t\treturn 1;\n+\t}\n+\n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (fill_pack_entry(oid, e, p)) {\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352410","messageId":"20180712193940.21065-19-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 18/23] midx: use midx in abbreviation calculations","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:35Z","receivedAt":"2018-07-12T19:40:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n midx.c | 11 +++++++++\n midx.h | 3 +++\n packfile.c | 6 +++++\n packfile.h | 1 +\n sha1-name.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 91 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 182535933c..4e014ff6e3 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -203,6 +203,17 @@ int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32\n \t\t\t MIDX_HASH_LEN, result);\n }\n \n+struct object_id *nth_midxed_object_oid(struct object_id *oid,\n+\t\t\t\t\tstruct multi_pack_index *m,\n+\t\t\t\t\tuint32_t n)\n+{\n+\tif (n >= m->num_objects)\n+\t\treturn NULL;\n+\n+\thashcpy(oid->hash, m->chunk_oid_lookup + m->hash_len * n);\n+\treturn oid;\n+}\n+\n static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n {\n \tconst unsigned char *offset_data;\ndiff --git a/midx.h b/midx.h\nindex 377838c9ca..1b976df873 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -31,6 +31,9 @@ struct multi_pack_index {\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+struct object_id *nth_midxed_object_oid(struct object_id *oid,\n+\t\t\t\t\tstruct multi_pack_index *m,\n+\t\t\t\t\tuint32_t n);\n int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \ndiff --git a/packfile.c b/packfile.c\nindex bc763d91b9..c0eb5ac885 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -961,6 +961,12 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packed_git;\n }\n \n+struct multi_pack_index *get_multi_pack_index(struct repository *r)\n+{\n+\tprepare_packed_git(r);\n+\treturn r->objects->multi_pack_index;\n+}\n+\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\ndiff --git a/packfile.h b/packfile.h\nindex b0eed44c0b..046280caf3 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -45,6 +45,7 @@ extern void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n+struct multi_pack_index *get_multi_pack_index(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/sha1-name.c b/sha1-name.c\nindex 60d9ef3c7e..7dc71201e6 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -12,6 +12,7 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"repository.h\"\n+#include \"midx.h\"\n \n static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);\n \n@@ -149,6 +150,32 @@ static int match_sha(unsigned len, const unsigned char *a, const unsigned char *\n \treturn 1;\n }\n \n+static void unique_in_midx(struct multi_pack_index *m,\n+\t\t\t struct disambiguate_state *ds)\n+{\n+\tuint32_t num, i, first = 0;\n+\tconst struct object_id *current = NULL;\n+\tnum = m->num_objects;\n+\n+\tif (!num)\n+\t\treturn;\n+\n+\tbsearch_midx(&ds->bin_pfx, m, &first);\n+\n+\t/*\n+\t * At this point, \"first\" is the location of the lowest object\n+\t * with an object name that could match \"bin_pfx\". See if we have\n+\t * 0, 1 or more objects that actually match(es).\n+\t */\n+\tfor (i = first; i < num && !ds->ambiguous; i++) {\n+\t\tstruct object_id oid;\n+\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n+\t\tif (!match_sha(ds->len, ds->bin_pfx.hash, current->hash))\n+\t\t\tbreak;\n+\t\tupdate_candidates(ds, current);\n+\t}\n+}\n+\n static void unique_in_pack(struct packed_git *p,\n \t\t\t struct disambiguate_state *ds)\n {\n@@ -177,8 +204,12 @@ static void unique_in_pack(struct packed_git *p,\n \n static void find_short_packed_object(struct disambiguate_state *ds)\n {\n+\tstruct multi_pack_index *m;\n \tstruct packed_git *p;\n \n+\tfor (m = get_multi_pack_index(the_repository); m && !ds->ambiguous;\n+\t m = m->next)\n+\t\tunique_in_midx(m, ds);\n \tfor (p = get_packed_git(the_repository); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n@@ -527,6 +558,42 @@ static int extend_abbrev_len(const struct object_id *oid, void *cb_data)\n \treturn 0;\n }\n \n+static void find_abbrev_len_for_midx(struct multi_pack_index *m,\n+\t\t\t\t struct min_abbrev_data *mad)\n+{\n+\tint match = 0;\n+\tuint32_t num, first = 0;\n+\tstruct object_id oid;\n+\tconst struct object_id *mad_oid;\n+\n+\tif (!m->num_objects)\n+\t\treturn;\n+\n+\tnum = m->num_objects;\n+\tmad_oid = mad->oid;\n+\tmatch = bsearch_midx(mad_oid, m, &first);\n+\n+\t/*\n+\t * first is now the position in the packfile where we would insert\n+\t * mad->hash if it does not exist (or the position of mad->hash if\n+\t * it does exist). Hence, we consider a maximum of two objects\n+\t * nearby for the abbreviation length.\n+\t */\n+\tmad->init_len = 0;\n+\tif (!match) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t} else if (first < num - 1) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t}\n+\tif (first > 0) {\n+\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n+\t\t\textend_abbrev_len(&oid, mad);\n+\t}\n+\tmad->init_len = mad->cur_len;\n+}\n+\n static void find_abbrev_len_for_pack(struct packed_git *p,\n \t\t\t\t struct min_abbrev_data *mad)\n {\n@@ -565,8 +632,11 @@ static void find_abbrev_len_for_pack(struct packed_git *p,\n \n static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n {\n+\tstruct multi_pack_index *m;\n \tstruct packed_git *p;\n \n+\tfor (m = get_multi_pack_index(the_repository); m; m = m->next)\n+\t\tfind_abbrev_len_for_midx(m, mad);\n \tfor (p = get_packed_git(the_repository); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352411","messageId":"20180712193940.21065-20-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 19/23] midx: use existing midx when writing new one","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:36Z","receivedAt":"2018-07-12T19:40:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Due to how Windows handles replacing a lockfile when there is an open\nhandle, create the close_midx() method to close the existing midx before\nwriting the new one.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n midx.h | 1 +\n 2 files changed, 111 insertions(+), 6 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 4e014ff6e3..bf2334acc6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -179,6 +179,23 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)\n \treturn NULL;\n }\n \n+static void close_midx(struct multi_pack_index *m)\n+{\n+\tuint32_t i;\n+\tmunmap((unsigned char *)m->data, m->data_len);\n+\tclose(m->fd);\n+\tm->fd = -1;\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tif (m->packs[i]) {\n+\t\t\tclose_pack(m->packs[i]);\n+\t\t\tfree(m->packs);\n+\t\t}\n+\t}\n+\tFREE_AND_NULL(m->packs);\n+\tFREE_AND_NULL(m->pack_names);\n+}\n+\n static int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id)\n {\n \tstruct strbuf pack_name = STRBUF_INIT;\n@@ -278,6 +295,29 @@ int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct mu\n \treturn nth_midxed_pack_entry(m, e, pos);\n }\n \n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_name)\n+{\n+\tuint32_t first = 0, last = m->num_packs;\n+\n+\twhile (first < last) {\n+\t\tuint32_t mid = first + (last - first) / 2;\n+\t\tconst char *current;\n+\t\tint cmp;\n+\n+\t\tcurrent = m->pack_names[mid];\n+\t\tcmp = strcmp(idx_name, current);\n+\t\tif (!cmp)\n+\t\t\treturn 1;\n+\t\tif (cmp > 0) {\n+\t\t\tfirst = mid + 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlast = mid;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir)\n {\n \tstruct multi_pack_index *m = r->objects->multi_pack_index;\n@@ -326,6 +366,7 @@ struct pack_list {\n \tuint32_t alloc_list;\n \tuint32_t alloc_names;\n \tsize_t pack_name_concat_len;\n+\tstruct multi_pack_index *m;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -334,6 +375,9 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \tstruct pack_list *packs = (struct pack_list *)data;\n \n \tif (ends_with(file_name, \".idx\")) {\n+\t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n+\t\t\treturn;\n+\n \t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n \t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n \n@@ -419,6 +463,23 @@ static int midx_oid_compare(const void *_a, const void *_b)\n \treturn a->pack_int_id - b->pack_int_id;\n }\n \n+static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n+\t\t\t\t uint32_t *pack_perm,\n+\t\t\t\t struct pack_midx_entry *e,\n+\t\t\t\t uint32_t pos)\n+{\n+\tif (pos >= m->num_objects)\n+\t\treturn 1;\n+\n+\tnth_midxed_object_oid(&e->oid, m, pos);\n+\te->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];\n+\te->offset = nth_midxed_offset(m, pos);\n+\n+\t/* consider objects in midx to be from \"old\" packs */\n+\te->pack_mtime = 0;\n+\treturn 0;\n+}\n+\n static void fill_pack_entry(uint32_t pack_int_id,\n \t\t\t struct packed_git *p,\n \t\t\t uint32_t cur_object,\n@@ -444,7 +505,8 @@ static void fill_pack_entry(uint32_t pack_int_id,\n * Copy only the de-duplicated entries (selected by most-recent modified time\n * of a packfile containing the object).\n */\n-static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n+static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n+\t\t\t\t\t\t struct packed_git **p,\n \t\t\t\t\t\t uint32_t *perm,\n \t\t\t\t\t\t uint32_t nr_packs,\n \t\t\t\t\t\t uint32_t *nr_objects)\n@@ -453,8 +515,9 @@ static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n \tstruct pack_midx_entry *entries_by_fanout = NULL;\n \tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\tuint32_t start_pack = m ? m->num_packs : 0;\n \n-\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++)\n+\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n \t\ttotal_objects += p[cur_pack]->num_objects;\n \n \t/*\n@@ -471,7 +534,23 @@ static struct pack_midx_entry *get_sorted_entries(struct packed_git **p,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tuint32_t nr_fanout = 0;\n \n-\t\tfor (cur_pack = 0; cur_pack < nr_packs; cur_pack++) {\n+\t\tif (m) {\n+\t\t\tuint32_t start = 0, end;\n+\n+\t\t\tif (cur_fanout)\n+\t\t\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n+\t\t\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n+\n+\t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n+\t\t\t\tnth_midxed_pack_midx_entry(m, perm,\n+\t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n+\t\t\t\t\t\t\t cur_object);\n+\t\t\t\tnr_fanout++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\t\tuint32_t start = 0, end;\n \n \t\t\tif (cur_fanout)\n@@ -667,16 +746,34 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n+\tpacks.m = load_multi_pack_index(object_dir);\n+\n \tpacks.nr = 0;\n-\tpacks.alloc_list = 16;\n-\tpacks.alloc_names = 16;\n+\tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n+\tpacks.alloc_names = packs.alloc_list;\n \tpacks.list = NULL;\n+\tpacks.names = NULL;\n \tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n \tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n+\tif (packs.m) {\n+\t\tfor (i = 0; i < packs.m->num_packs; i++) {\n+\t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n+\t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n+\n+\t\t\tpacks.list[packs.nr] = NULL;\n+\t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n+\t\t\tpacks.nr++;\n+\t\t}\n+\t}\n+\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n+\tif (packs.m && packs.nr == packs.m->num_packs)\n+\t\tgoto cleanup;\n+\n \tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n@@ -684,7 +781,8 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n-\tentries = get_sorted_entries(packs.list, pack_perm, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n+\n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n@@ -696,6 +794,9 @@ int write_midx_file(const char *object_dir)\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n+\tif (packs.m)\n+\t\tclose_midx(packs.m);\n+\n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n@@ -787,6 +888,7 @@ int write_midx_file(const char *object_dir)\n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n \n+cleanup:\n \tfor (i = 0; i < packs.nr; i++) {\n \t\tif (packs.list[i]) {\n \t\t\tclose_pack(packs.list[i]);\n@@ -798,5 +900,7 @@ int write_midx_file(const char *object_dir)\n \tfree(packs.list);\n \tfree(packs.names);\n \tfree(entries);\n+\tfree(pack_perm);\n+\tfree(midx_name);\n \treturn 0;\n }\ndiff --git a/midx.h b/midx.h\nindex 1b976df873..d4cde99473 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -35,6 +35,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n);\n int fill_midx_entry(const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352412","messageId":"20180712193940.21065-21-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 20/23] midx: use midx in approximate_object_count","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:37Z","receivedAt":"2018-07-12T19:40:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n packfile.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/packfile.c b/packfile.c\nindex c0eb5ac885..97e7812b6b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -861,10 +861,13 @@ unsigned long approximate_object_count(void)\n {\n \tif (!the_repository->objects->approximate_object_count_valid) {\n \t\tunsigned long count;\n+\t\tstruct multi_pack_index *m;\n \t\tstruct packed_git *p;\n \n \t\tprepare_packed_git(the_repository);\n \t\tcount = 0;\n+\t\tfor (m = get_multi_pack_index(the_repository); m; m = m->next)\n+\t\t\tcount += m->num_objects;\n \t\tfor (p = the_repository->objects->packed_git; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352413","messageId":"20180712193940.21065-22-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 21/23] midx: prevent duplicate packfile loads","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:38Z","receivedAt":"2018-07-12T19:40:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index, when present, tracks the existence of objects and\ntheir offsets within a list of packfiles. This allows us to use the\nmulti-pack-index for object lookups, abbreviations, and object counts.\n\nWhen the multi-pack-index tracks a packfile, then we do not need to add\nthat packfile to the packed_git linked list or the MRU list.\n\nWe still need to load the packfiles that are not tracked by the\nmulti-pack-index.\n\nSigned-off-by: Derrick Stolee \n---\n packfile.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/packfile.c b/packfile.c\nindex 97e7812b6b..2c819a0ad8 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -795,6 +795,7 @@ struct prepare_pack_data {\n \tstruct repository *r;\n \tstruct string_list *garbage;\n \tint local;\n+\tstruct multi_pack_index *m;\n };\n \n static void prepare_pack(const char *full_name, size_t full_name_len,\n@@ -805,6 +806,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\")) {\n+\t\tif (data->m && midx_contains_pack(data->m, file_name))\n+\t\t\treturn;\n \t\t/* Don't reopen a pack we already have. */\n \t\tfor (p = data->r->objects->packed_git; p; p = p->next) {\n \t\t\tsize_t len;\n@@ -839,6 +842,12 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local\n \tstruct prepare_pack_data data;\n \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \n+\tdata.m = r->objects->multi_pack_index;\n+\n+\t/* look for the multi-pack-index for this object directory */\n+\twhile (data.m && strcmp(data.m->object_dir, objdir))\n+\t\tdata.m = data.m->next;\n+\n \tdata.r = r;\n \tdata.garbage = &garbage;\n \tdata.local = local;\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352414","messageId":"20180712193940.21065-24-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 23/23] midx: clear midx on repack","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:40Z","receivedAt":"2018-07-12T19:40:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If a 'git repack' command replaces existing packfiles, then we must\nclear the existing multi-pack-index before moving the packfiles it\nreferences.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/repack.c | 9 +++++++++\n midx.c | 12 ++++++++++++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 9 +++++++++\n 4 files changed, 31 insertions(+)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 6c636e159e..7f7cdc8b17 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -8,6 +8,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"argv-array.h\"\n+#include \"midx.h\"\n \n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -174,6 +175,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n \tint local = 0;\n+\tint midx_cleared = 0;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -333,6 +335,13 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tfor_each_string_list_item(item, &names) {\n \t\tfor (ext = 0; ext < ARRAY_SIZE(exts); ext++) {\n \t\t\tchar *fname, *fname_old;\n+\n+\t\t\tif (!midx_cleared) {\n+\t\t\t\t/* if we move a packfile, it will invalidated the midx */\n+\t\t\t\tclear_midx_file(get_object_directory());\n+\t\t\t\tmidx_cleared = 1;\n+\t\t\t}\n+\n \t\t\tfname = mkpathdup(\"%s/pack-%s%s\", packdir,\n \t\t\t\t\t\titem->string, exts[ext].name);\n \t\t\tif (!file_exists(fname)) {\ndiff --git a/midx.c b/midx.c\nindex bf2334acc6..19b7df338e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -904,3 +904,15 @@ int write_midx_file(const char *object_dir)\n \tfree(midx_name);\n \treturn 0;\n }\n+\n+void clear_midx_file(const char *object_dir)\n+{\n+\tchar *midx = get_midx_filename(object_dir);\n+\n+\tif (remove_path(midx)) {\n+\t\tUNLEAK(midx);\n+\t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n+\t}\n+\n+\tfree(midx);\n+}\ndiff --git a/midx.h b/midx.h\nindex d4cde99473..e3b07f1586 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -39,5 +39,6 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir);\n \n int write_midx_file(const char *object_dir);\n+void clear_midx_file(const char *object_dir);\n \n #endif\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 601e28a2f0..5ad6614465 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -141,6 +141,15 @@ test_expect_success 'write midx with twelve packs' '\n \n compare_results_with_midx \"twelve packs\"\n \n+test_expect_success 'repack removes multi-pack-index' '\n+\ttest_path_is_file $objdir/pack/multi-pack-index &&\n+\tgit repack -adf &&\n+\ttest_path_is_missing $objdir/pack/multi-pack-index\n+'\n+\n+compare_results_with_midx \"after repack\"\n+\n+\n # usage: corrupt_data []\n corrupt_data () {\n \tfile=$1\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352415","messageId":"20180712193940.21065-23-dstolee@microsoft.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"[PATCH v4 22/23] packfile: skip loading index if in multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-12T19:39:39Z","receivedAt":"2018-07-12T19:40:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n packfile.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 2c819a0ad8..e6ecf12ab5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -469,8 +469,19 @@ static int open_packed_git_1(struct packed_git *p)\n \tssize_t read_result;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \n-\tif (!p->index_data && open_pack_index(p))\n-\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\tif (!p->index_data) {\n+\t\tstruct multi_pack_index *m;\n+\t\tconst char *pack_name = strrchr(p->pack_name, '/');\n+\n+\t\tfor (m = the_repository->objects->multi_pack_index;\n+\t\t m; m = m->next) {\n+\t\t\tif (midx_contains_pack(m, pack_name))\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!m && open_pack_index(p))\n+\t\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\t}\n \n \tif (!pack_max_fds) {\n \t\tunsigned int max_fds = get_max_fd_limit();\n@@ -521,6 +532,10 @@ static int open_packed_git_1(struct packed_git *p)\n \t\t\t\" supported (try upgrading GIT to a newer version)\",\n \t\t\tp->pack_name, ntohl(hdr.hdr_version));\n \n+\t/* Skip index checking if in multi-pack-index */\n+\tif (!p->index_data)\n+\t\treturn 0;\n+\n \t/* Verify the pack matches its index. */\n \tif (p->num_objects != ntohl(hdr.hdr_entries))\n \t\treturn error(\"packfile %s claims to have %\"PRIu32\" objects\"\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352437","messageId":"xmqqin5kupu3.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"20180712193940.21065-17-dstolee@microsoft.com","subject":"Re: [PATCH v4 16/23] config: create core.multiPackIndex setting","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-07-12T21:05:56Z","receivedAt":"2018-07-12T21:06:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> The core.multiPackIndex config setting controls the multi-pack-\n> index (MIDX) feature. If false, the setting will disable all reads\n> from the multi-pack-index file.\n>\n> Read this config setting in the new prepare_multi_pack_index_one()\n> which is called during prepare_packed_git(). This check is run once\n> per repository.\n>\n> Add comparison commands in t5319-multi-pack-index.sh to check\n> typical Git behavior remains the same as the config setting is turned\n> on and off. This currently includes 'git rev-list' and 'git log'\n> commands to trigger several object database reads. Currently, these\n> would only catch an error in the prepare_multi_pack_index_one(), but\n> with later commits will catch errors in object lookups, abbreviations,\n> and approximate object counts.\n>\n> Signed-off-by: Derrick Stolee \n>\n> midx: prepare midxed_git struct\n>\n> Signed-off-by: Derrick Stolee \n\nWhat is going on around here?\n\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index 4a4fa26f7a..601e28a2f0 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -3,6 +3,8 @@\n> test_description='multi-pack-indexes'\n> . ./test-lib.sh\n> \n> +objdir=.git/objects\n> +\n> midx_read_expect () {\n> \tNUM_PACKS=$1\n> \tNUM_OBJECTS=$2\n> @@ -76,18 +78,35 @@ test_expect_success 'create objects' '\n> '\n> \n> test_expect_success 'write midx with one v1 pack' '\n> -\tpack=$(git pack-objects --index-version=1 pack/test -\ttest_when_finished rm pack/test-$pack.pack pack/test-$pack.idx pack/multi-pack-index &&\n> -\tgit multi-pack-index --object-dir=. write &&\n> -\tmidx_read_expect 1 18 4 .\n> +\tpack=$(git pack-objects --index-version=1 $objdir/pack/test +\ttest_when_finished rm $objdir/pack/test-$pack.pack \\\n> +\t\t$objdir/pack/test-$pack.idx $objdir/pack/multi-pack-index &&\n> +\tgit multi-pack-index --object-dir=$objdir write &&\n> +\tmidx_read_expect 1 18 4 $objdir\n\nHmph, so we used to run tests as if $cwd were GIT_OBJECT_DIRECTORY\nbut now we are running them from the top-level of the working tree,\njust like all the other tests? Interesting.\n\n> '\n> \n> +midx_git_two_modes() {\n> +\tgit -c core.multiPackIndex=false $1 >expect &&\n> +\tgit -c core.multiPackIndex=true $1 >actual &&\n> +\ttest_cmp expect actual\n> +}\n> +\n> +compare_results_with_midx() {\n\nStyle: \"compare_results_with_midx () {\", just like mdx_read_expect\nnear the top of the file, but unlike midx_git_two_modes we see\nnearby. Please keep \"git grep 'funcname () {'\" a usable way to\nlocate where a shell function is defined without forcing people to\ntype an asterisk.\n"},{"id":"352438","messageId":"xmqqefg8uplg.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"20180712193940.21065-1-dstolee@microsoft.com","subject":"Re: [PATCH v4 00/23] Multi-pack-index (MIDX)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-07-12T21:11:07Z","receivedAt":"2018-07-12T21:11:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> * The global 'core_multi_pack_index' is replaced with a one-time call to\n> git_config_bool() per repository that loads a multi-pack-index.\n\nOK. I guess that is more consistent, as the configuration would be\nper-repo and multi-pack-index is per object-store.\n\nAfter a quick look at the diff between the application of these\npatches and what has been queued in 'pu', I found most (perhaps all)\nof the changes are strict improvements. I may have more comments\nlater after reading the patches in full.\n\nThanks.\n"},{"id":"352441","messageId":"CAPig+cTU--KrGcv4C_CwBZEuec4dgm_tJqL=CFWKT6vxxR016w@mail.gmail.com","threadId":"48668","inReplyTo":"20180712193940.21065-5-dstolee@microsoft.com","subject":"Re: [PATCH v4 04/23] multi-pack-index: add 'write' verb","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-12T22:56:22Z","receivedAt":"2018-07-12T22:56:35Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 12, 2018 at 3:40 PM Derrick Stolee wrote:\n> In anticipation of writing multi-pack-indexes, add a skeleton\n> 'git multi-pack-index write' subcommand and send the options to a\n> write_midx_file() method. Also create a skeleton test script that\n> tests the 'write' subcommand.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> @@ -30,5 +31,17 @@ int cmd_multi_pack_index(int argc, const char **argv,\n> + if (argc == 0)\n> + goto usage;\n> +\n> + if (!strcmp(argv[0], \"write\")) {\n> + if (argc > 1)\n> + goto usage;\n> +\n> + return write_midx_file(opts.object_dir);\n> + }\n> +\n> +usage:\n> + usage_with_options(builtin_multi_pack_index_usage,\n> + builtin_multi_pack_index_options);\n> }\n\nI realize that some other commands may work this way, but merely\nshowing 'usage' is fairly user-hostile. What would be much more\nhelpful would be to explain to the user what the problem is; for\ninstance, \"no action specified\" and \"'write' takes no arguments\", or\nsomething. That way the user knows exactly what corrective action to\ntake rather than having to try to guess based upon 'usage'. Showing\n'usage' after the actual error message may or may not make sense\n(though, I prefer not doing so since noisy 'usage' tends to swamp the\nactual error message, making it easy to miss).\n\nI wouldn't want to see a re-roll just for this, especially for such a\nlong series. Perhaps such a change can be done as a follow-up patch by\nsomeone at some point.\n\nBy the way, if you ever need to add options or arguments to \"write\" or\n\"verify\", you might want to model it after how it's done in\nbuiltin/worktree.c, in which each subcommand is responsible for its\nown argument processing, rather than handling subcommand arguments\nhere in the top-level function.\n"},{"id":"352446","messageId":"61de4c01-daeb-2fad-491e-aa06facc3db8@gmail.com","threadId":"48668","inReplyTo":"xmqqin5kupu3.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 16/23] config: create core.multiPackIndex setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-07-13T00:50:36Z","receivedAt":"2018-07-13T00:50:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/12/2018 5:05 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> The core.multiPackIndex config setting controls the multi-pack-\n>> index (MIDX) feature. If false, the setting will disable all reads\n>> from the multi-pack-index file.\n>>\n>> Read this config setting in the new prepare_multi_pack_index_one()\n>> which is called during prepare_packed_git(). This check is run once\n>> per repository.\n>>\n>> Add comparison commands in t5319-multi-pack-index.sh to check\n>> typical Git behavior remains the same as the config setting is turned\n>> on and off. This currently includes 'git rev-list' and 'git log'\n>> commands to trigger several object database reads. Currently, these\n>> would only catch an error in the prepare_multi_pack_index_one(), but\n>> with later commits will catch errors in object lookups, abbreviations,\n>> and approximate object counts.\n>>\n>> Signed-off-by: Derrick Stolee \n>>\n>> midx: prepare midxed_git struct\n>>\n>> Signed-off-by: Derrick Stolee \n> What is going on around here?\nSorry. I squashed the commits, and intended to drop this second commit \nmessage.\n>\n>> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n>> index 4a4fa26f7a..601e28a2f0 100755\n>> --- a/t/t5319-multi-pack-index.sh\n>> +++ b/t/t5319-multi-pack-index.sh\n>> @@ -3,6 +3,8 @@\n>> test_description='multi-pack-indexes'\n>> . ./test-lib.sh\n>> \n>> +objdir=.git/objects\n>> +\n>> midx_read_expect () {\n>> \tNUM_PACKS=$1\n>> \tNUM_OBJECTS=$2\n>> @@ -76,18 +78,35 @@ test_expect_success 'create objects' '\n>> '\n>> \n>> test_expect_success 'write midx with one v1 pack' '\n>> -\tpack=$(git pack-objects --index-version=1 pack/test > -\ttest_when_finished rm pack/test-$pack.pack pack/test-$pack.idx pack/multi-pack-index &&\n>> -\tgit multi-pack-index --object-dir=. write &&\n>> -\tmidx_read_expect 1 18 4 .\n>> +\tpack=$(git pack-objects --index-version=1 $objdir/pack/test > +\ttest_when_finished rm $objdir/pack/test-$pack.pack \\\n>> +\t\t$objdir/pack/test-$pack.idx $objdir/pack/multi-pack-index &&\n>> +\tgit multi-pack-index --object-dir=$objdir write &&\n>> +\tmidx_read_expect 1 18 4 $objdir\n> Hmph, so we used to run tests as if $cwd were GIT_OBJECT_DIRECTORY\n> but now we are running them from the top-level of the working tree,\n> just like all the other tests? Interesting.\nThis is the first time we _need_ them in the .git/object directory.\n>> '\n>> \n>> +midx_git_two_modes() {\n>> +\tgit -c core.multiPackIndex=false $1 >expect &&\n>> +\tgit -c core.multiPackIndex=true $1 >actual &&\n>> +\ttest_cmp expect actual\n>> +}\n>> +\n>> +compare_results_with_midx() {\n> Style: \"compare_results_with_midx () {\", just like mdx_read_expect\n> near the top of the file, but unlike midx_git_two_modes we see\n> nearby. Please keep \"git grep 'funcname () {'\" a usable way to\n> locate where a shell function is defined without forcing people to\n> type an asterisk.\n\nSorry that I missed these two.\n\nThanks,\n\n-Stolee\n\n"},{"id":"353166","messageId":"xmqqva99bwdw.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"20180712193940.21065-4-dstolee@microsoft.com","subject":"Re: [PATCH v4 03/23] multi-pack-index: add builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-07-20T18:22:03Z","receivedAt":"2018-07-20T18:22:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> new file mode 100644\n> index 0000000000..ec9982cbfc\n> --- /dev/null\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -0,0 +1,36 @@\n> +git-multi-pack-index(1)\n> +======================\n\nIsn't this underline too short by one column? My copy of AsciiDoc\nseems to be OK with it, but do other versions and reimplementations\ngroke this fine?\n"},{"id":"353200","messageId":"20180720221522.GD18502@genre.crustytoothpaste.net","threadId":"48668","inReplyTo":"xmqqva99bwdw.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/23] multi-pack-index: add builtin","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-07-20T22:15:22Z","receivedAt":"2018-07-20T22:15:31Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Fri, Jul 20, 2018 at 11:22:03AM -0700, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n> > diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> > new file mode 100644\n> > index 0000000000..ec9982cbfc\n> > --- /dev/null\n> > +++ b/Documentation/git-multi-pack-index.txt\n> > @@ -0,0 +1,36 @@\n> > +git-multi-pack-index(1)\n> > +======================\n> \n> Isn't this underline too short by one column? My copy of AsciiDoc\n> seems to be OK with it, but do other versions and reimplementations\n> groke this fine?\n\nI believe AsciiDoc allows a two-character grace (either longer or\nshorter) and Asciidoctor allows one. So this should work both places\n(although I haven't tested).\n\nIt might be nice to fix if a re-roll is needed, but it should be\nfunctional as-is.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"353203","messageId":"xmqqwotp8ru5.fsf@gitster-ct.c.googlers.com","threadId":"48668","inReplyTo":"20180720221522.GD18502@genre.crustytoothpaste.net","subject":"Re: [PATCH v4 03/23] multi-pack-index: add builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-07-20T22:28:34Z","receivedAt":"2018-07-20T22:28:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> On Fri, Jul 20, 2018 at 11:22:03AM -0700, Junio C Hamano wrote:\n>> Derrick Stolee writes:\n>> \n>> > diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n>> > new file mode 100644\n>> > index 0000000000..ec9982cbfc\n>> > --- /dev/null\n>> > +++ b/Documentation/git-multi-pack-index.txt\n>> > @@ -0,0 +1,36 @@\n>> > +git-multi-pack-index(1)\n>> > +======================\n>> \n>> Isn't this underline too short by one column? My copy of AsciiDoc\n>> seems to be OK with it, but do other versions and reimplementations\n>> groke this fine?\n>\n> I believe AsciiDoc allows a two-character grace (either longer or\n> shorter) and Asciidoctor allows one. So this should work both places\n> (although I haven't tested).\n>\n> It might be nice to fix if a re-roll is needed, but it should be\n> functional as-is.\n\n\nThanks.\n\nI've already locally amended it and pushed the result out, as I\nneeded to touch-up another commit in the same series anyway.\n\n\n"}]}