# Potential problem in git-add may corrupt the index file 3 messages from 2012-04-23 to 2012-04-29. Participants: Gang Hu, Junio C Hamano. Thread: https://gitlist.dev/t/30314 ## Gang Hu, 2012-04-23 23:30 Subject: Potential problem in git-add may corrupt the index file Message-ID: URL: https://gitlist.dev/e/CAEJt7hYYJh5dT1Tk99Oze6Hg4QaOqX5ERi0faWjyifQ6dr-pWw%40mail.gmail.com ``` Hi, We are developing a tool to find bugs, and it flagged a potential problem in git-add. We are checking git 1.7.9.4, and this problem still exists in the latest release, git 1.7.10. At the end of cmd_add() in builtin/add.c, there is a call to write_cache(), followed by commit_locked_index(). 474: if (write_cache(newfd, active_cache, active_nr) || 475: commit_locked_index(&lock_file)) In write_cache(), the program wrote to the locked index file, and in commit_locked_index(), it did an atomic rename to rename the locked index file to the real index file. However, the locked index file is not synced to the disk before being renamed to the real index file. This may cause data loss, which has been pointed out in this post: http://stackoverflow.com/questions/7433057/is-rename-without-fsync-safe If a crash happens after the rename(), the data may have not been completely flushed into the index file, so the user may face an empty or corrupted index file. Although the user may just recreate the index file, he would need to add the files to be committed again, which may be inconvenient and disturbing. To fix this problem, we may just do a fsync() before renaming the file. We can add this to close_lock_file() in lockfile.c: 223: int close_lock_file(struct lock_file *lk) 224: { 225: int fd = lk->fd; 226: lk->fd = -1; // we may add // fsync(fd); // here 227: return close(fd); 228:} Is this a real problem? Any clarification or confirmation is appreciated. Thank you. -- Cheers, Gang ``` ## Junio C Hamano, 2012-04-24 01:51 Subject: Re: Potential problem in git-add may corrupt the index file Message-ID: URL: https://gitlist.dev/e/xmqq4ns9j3rf.fsf%40junio.mtv.corp.google.com In-Reply-To: ``` Gang Hu writes: > If a crash happens after the rename(), the data may have not been > completely flushed into the index file, so the user may face an empty > or corrupted index file. Yeah, probably we should add fsync_or_die() at the end of write_index() in read-cache.c, possibly protect it with fsync_object_files just like we do in close_sha1_file() in sha1_file.c for loose object files. I think the real fix would be to update the write_index() codepath to use the csum-file API, though. Then sha1close() will give us the fsync behaviour for free. ``` ## Gang Hu, 2012-04-29 22:39 Subject: Re: Potential problem in git-add may corrupt the index file Message-ID: URL: https://gitlist.dev/e/CAEJt7hbKFKN8J5jjX9Z_Apnco%2BYiN0kYn%3DwrHTYjE9P9M%3Djjaw%40mail.gmail.com In-Reply-To: ``` Hi Junio, Thanks for the fast reply! We've also discovered other similar programs in git, enclosed below so you may want to fix them in one patch. Here is one example in refs.c: 1710: lockpath = mkpath("%s.lock", git_HEAD); fd = open(lockpath, O_CREAT | O_EXCL | O_WRONLY, 0666); .... written = write_in_full(fd, ref, len); // We may add // fsync_or_die(fd, ...); // here if (close(fd) != 0 || written != len) { error("Unable to write to %s", lockpath); goto error_unlink_return; } if (rename(lockpath, git_HEAD) < 0) { ... So there is a problem here, and the head file may be corrupted. In server-info.c, there is a similar problem: 216: namelen = sprintf(infofile, "%s/info/packs", get_object_directory()); strcpy(name, infofile); strcpy(name + namelen, "+"); init_pack_info(infofile, force); safe_create_leading_directories(name); fp = fopen(name, "w"); if (!fp) return error("cannot open %s", name); write_pack_info_file(fp); // We may add // fsync_or_die(fileno(fp), ...); // here fclose(fp); adjust_shared_perm(name); rename(name, infofile); Here, the packs file in the objects/info directory may be corrupted. In the same file, the info/refs file may also be corrupted: 28: char *path0 = git_pathdup("info/refs"); int len = strlen(path0); char *path1 = xmalloc(len + 2); strcpy(path1, path0); strcpy(path1 + len, "+"); safe_create_leading_directories(path0); info_ref_fp = fopen(path1, "w"); if (!info_ref_fp) return error("unable to update %s", path1); for_each_ref(add_info_ref, NULL); // We may add // fsync_or_die(fileno(info_ref_fp), ...); // here fclose(info_ref_fp); adjust_shared_perm(path1); rename(path1, path0); On Mon, Apr 23, 2012 at 9:51 PM, Junio C Hamano wrote: > Gang Hu writes: > >> If a crash happens after the rename(), the data may have not been >> completely flushed into the index file, so the user may face an empty >> or corrupted index file. > > Yeah, probably we should add fsync_or_die() at the end of write_index() > in read-cache.c, possibly protect it with fsync_object_files just like > we do in close_sha1_file() in sha1_file.c for loose object files. > > I think the real fix would be to update the write_index() codepath to > use the csum-file API, though.  Then sha1close() will give us the fsync > behaviour for free. -- Cheers, Gang ```