git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file

From
Junio C Hamano <gitster@pobox.com>
Date
Aug 10, 2012, 00:13 UTC
Message-ID
<7v7gt7mwo3.fsf@alter.siamese.dyndns.org>
In-Reply-To
<7vtxwbn2qe.fsf@alter.siamese.dyndns.org>
Junio C Hamano <gitster@pobox.com> writes:
Show 24 quoted lines
> If you found that an entry you read halfway has an inconsistent crc,
> and if you suspect that is because somebody else was writing to the
> same index, it is a _sure_ sign that you are not alone, and all the
> entries you read so far to the core, even if they weren't touched by
> that sombody else when you read them, may be stale, and worse yet,
> what you are going to read may be inconsistent with what you've read
> and have in-core (e.g. you may have read "f" before somebody else
> that is racing with you have turned it into a directory, and your
> next read may find "f/d" in the index without crc error).
>
> One sane way to avoid reading such an inconsistent state may be to
> redo this whole function, starting from the part that calls mmap().
> IOW,
>
> 	do {
> 		fd = open()
> 		mmap = xmmap(fd);
> 		close(fd);
>                 verify_various_fields(mmap);
>                 status = istate->ops->read_index(istate, mmap, mmap_size));
> 	} while (status == READ_AGAIN);
>
> I do not think the "pass fd around so that we can redo the mapping
> deep inside the callchain" is either a good idea or necessary.

By the way, you can only detect such inconsistency when you are lucky enough that you catch the other person in the middle of writing.

If the index you are looking at holds a large tree with very many paths, it is possible that there are two large directories, and after you read all entries from one, the other process starts modifying the paths in that directory, without you ever finding it out. If the goal of the topic is to make the index work better in projects with large trees, it may be wise to think about locking the whole thing, so that you do not have to rely on the per-entry crc and you being lucky to detect such a race. The per-entry crc, as far as I understand, may have been introduced primarily to detect on-disk data corruption; it is not a suitable mechanism to detect conflicting accesses.

Previous: Thomas GummererNext: Nguyen Thai Ngoc Duy
Message 7 of 34 in “Introduce index file format version 5”
  1. 0/13 Introduce index file format version 5Thomas Gummerer, Aug 8, 2012
  2. 01/13 Move index v2 specific functions to their own fileThomas Gummerer, Aug 8, 2012
  3. Nguyen Thai Ngoc DuyAug 8, 2012
  4. Thomas GummererAug 8, 2012
  5. Junio C HamanoAug 9, 2012
  6. Thomas GummererAug 9, 2012
  7. Junio C HamanoAug 10, 2012
  8. Nguyen Thai Ngoc DuyAug 10, 2012
  9. Thomas RastAug 10, 2012
  10. Junio C HamanoAug 10, 2012
  11. Thomas GummererAug 10, 2012
  12. 02/13 t2104: Don't fail for index versions other than [23]Thomas Gummerer, Aug 8, 2012
  13. 03/13 t3700: Avoid interfering with the racy codeThomas Gummerer, Aug 8, 2012
  14. 04/13 Add documentation of the index-v5 file formatThomas Gummerer, Aug 8, 2012
  15. Junio C HamanoAug 9, 2012
  16. Thomas GummererAug 9, 2012
  17. Junio C HamanoAug 9, 2012
  18. 05/13 Make in-memory format aware of stat_crcThomas Gummerer, Aug 8, 2012
  19. 06/13 Read index-v5Thomas Gummerer, Aug 8, 2012
  20. Nguyen Thai Ngoc DuyAug 8, 2012
  21. Johannes SixtAug 8, 2012
  22. Junio C HamanoAug 8, 2012
  23. Thomas GummererAug 8, 2012
  24. 07/13 Read resolve-undo dataThomas Gummerer, Aug 8, 2012
  25. Junio C HamanoAug 9, 2012
  26. Thomas GummererAug 9, 2012
  27. Junio C HamanoAug 10, 2012
  28. Thomas GummererAug 10, 2012
  29. 08/13 Read cache-tree in index-v5Thomas Gummerer, Aug 8, 2012
  30. 09/13 Write index-v5Thomas Gummerer, Aug 8, 2012
  31. 10/13 Write index-v5 cache-tree dataThomas Gummerer, Aug 8, 2012
  32. 11/13 Write resolve-undo data for index-v5Thomas Gummerer, Aug 8, 2012
  33. 12/13 update-index.c: always rewrite the index when index-version is givenThomas Gummerer, Aug 8, 2012
  34. 13/13 p0002-index.sh: add perf test for the index formatsThomas Gummerer, Aug 8, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.