File extension conflict when working with git and latex

Hi,

I struggled with a really nasty issue today (and yesterday):

I work on a semester project paper which is written in latex and therefor version-controlled with git. We compile the document using pdflatex, we automate the compiling with make.

pdflatex generates some temporary files, which are required if you call it the second and third time (it is a top-down compiler, which needs to be called several times). One of the file extensions is ".idx", which (kind of) conflicts with file extensions in .git/.

Unfortunetely, we wrote our `make clean` task recursively. I think you can imagine what went wrong: The clean-task corrupted the repository, as it removed .idx files from within .git/.

I lost work because of this ugly name collision.

My question now:

I know, I can fix this by fixing the clean task in my Makefile. But maybe someone somewhere on this world doesn't know the git internals as good as me (and, of course, my coworker). Is there _any chance at all_ that this gets mentioned somewhere, so others don't fall into this pit?

On Fri, Mar 21, 2014 at 05:13:24PM +0100, Matthias Beyer wrote:

Hello Matthias,

You can recreate the .idx files by running

    git index-pack .git/objects/pack/pack-<hash>.pack

for each pack file.

Regards Simon

On 21-03-2014 17:36:28, Simon Ruderich wrote:

Hi Simon,

I think so. I executed:

    git fsck # reports N missing blobs, commits, trees and dangling stuff
    git index-pack ...
    git fsck # reports only dangling commits and blobs

I don't know if this means that the repository is fixed now?

On Fri, Mar 21, 2014 at 05:46:51PM +0100, Matthias Beyer wrote:

Hello Matthias,

If nothing else was deleted, then yes, the repository should be fine now.

Regards Simon

Matthias Beyer <mail@beyermatthias.de> writes:

Surely, we are here to please ;-) All of us want to make sure newbies do not shoot themselves in the foot.

But the problem is what exactly should be mentioned. With a fresh wound with your LaTeX project still in your mind, you may be tempted to special case ".idx", but other newbies may inflict the same kind of hurt on themselves with different "find" patterns, e.g.

    $ find . -name '[0-9a-f]*[0-9a-f]' -type f -print | xargs rm -f

when they know their project creates hexadecimal-numbered temoprary files, or whatever other pattern that match the files they do not care about, that also happens to match whatever is in $GIT_DIR. The only common caution that helps us to make sure "others do not fall into this pit" is "Files and directories in $GIT_DIR are used to record your work; do not muck with them unless you know what you are doing e.g. manually repairing a corrupt repository", but that is a bit lame, isn't it?

It is tempting to suggest "git clean '*.idx'", but that is a good fit in the Makefile only when you know everybody involved in the project works in a checkout from Git, not from a tarball extract, and does not apply to projects in general.

On 03/21/2014 07:32 PM, Junio C Hamano wrote:

It might help some people to show the "find" incantation for skipping files under .git, which I think would be something like

    find . -name .git -prune -o \( [...] \) -print0 | xargs -0 rm -f

where "[...]" is whatever condition the user would otherwise use.

Michael

On 2014-03-21 17.13, Matthias Beyer wrote:

The idx files are mentioned here: http://www.git-scm.com/book/en/Git-Internals-Packfiles

(Otherwise, some random thoughts:)
- stay out of the .git directory, whatever you do
   (I lost some data because my faulty scripts using sed to replace
   "s/this/that/" and went into the .git directory) 
- Don't delete "read-only" files
   (The .idx files are all  "r--r--r--")
  Or in other words, do not use "rf -f" in a script, unless you know exactly
  what you and the script are doing.
- Replace "rm -f" whith "mv /tmp" could be an option
- As mentioned before, "git clean" can be an option,
  I used it once when the "make clean", "make distclean", "make reallydistclean"
  did not work as expected by me,
  some auto-generated config files where still not deleted. 
- Make backups of your repo (frequently)
- push the repo to a backup repo (frequently)
- Use $GIT_DIR to move the repo out of your work tree:
  https://www.kernel.org/pub/software/scm/git/docs/git.html

- Prepare a patch that renames the *.idx files used by Git into
  *.gitidx, *.gidx, *.packidx, (and the *.pack into *.packdata ??) 

Hopefully you did not loose too much data, and if you have an idea how to improve things, you may consider sending a patch, improvements are possible and welcome.