{"thread":{"id":"36266","subject":"File extension conflict when working with git and latex","startedAt":"2014-03-21T16:13:24Z","lastAt":"2014-03-22T07:11:50Z","messageCount":7,"participants":["Matthias Beyer","Simon Ruderich","Junio C Hamano","Michael Haggerty","Torsten Bögershausen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"237318","messageId":"20140321161324.GC2965@fu.192.168.200.1","threadId":"36266","inReplyTo":null,"subject":"File extension conflict when working with git and latex","fromName":"Matthias Beyer","fromEmail":"mail@beyermatthias.de","sentAt":"2014-03-21T16:13:24Z","receivedAt":"2014-03-21T16:13:24Z","isPatch":false,"sender":{"key":"mail@beyermatthias.de","avatar":null},"body":"Hi,\n\nI struggled with a really nasty issue today (and yesterday):\n\nI work on a semester project paper which is written in latex and\ntherefor version-controlled with git. We compile the document using\npdflatex, we automate the compiling with make.\n\npdflatex generates some temporary files, which are required if you\ncall it the second and third time (it is a top-down compiler, which\nneeds to be called several times). One of the file extensions is\n\".idx\", which (kind of) conflicts with file extensions in .git/.\n\nUnfortunetely, we wrote our `make clean` task recursively. I think you\ncan imagine what went wrong: The clean-task corrupted the repository,\nas it removed .idx files from within .git/.\n\nI lost work because of this ugly name collision.\n\nMy question now:\n\nI know, I can fix this by fixing the clean task in my Makefile. But\nmaybe someone somewhere on this world doesn't know the git internals\nas good as me (and, of course, my coworker). Is there _any chance\nat all_ that this gets mentioned somewhere, so others don't fall into\nthis pit?\n\n-- \nMit freundlichen Grüßen,\nKind regards,\nMatthias Beyer\n\nProudly sent with mutt.\nHappily signed with gnupg.\n"},{"id":"237319","messageId":"20140321163628.GA28121@ruderich.org","threadId":"36266","inReplyTo":"20140321161324.GC2965@fu.192.168.200.1","subject":"Re: File extension conflict when working with git and latex","fromName":"Simon Ruderich","fromEmail":"simon@ruderich.org","sentAt":"2014-03-21T16:36:28Z","receivedAt":"2014-03-21T16:36:28Z","isPatch":false,"sender":{"key":"simon@ruderich.org","avatar":"https://avatars.githubusercontent.com/u/390994?v=4"},"body":"On Fri, Mar 21, 2014 at 05:13:24PM +0100, Matthias Beyer wrote:\n> Unfortunetely, we wrote our `make clean` task recursively. I think you\n> can imagine what went wrong: The clean-task corrupted the repository,\n> as it removed .idx files from within .git/.\n>\n> I lost work because of this ugly name collision.\n\nHello Matthias,\n\nYou can recreate the .idx files by running\n\n git index-pack .git/objects/pack/pack-.pack\n\nfor each pack file.\n\nRegards\nSimon\n-- \n+ privacy is necessary\n+ using gnupg http://gnupg.org\n+ public key id: 0x92FEFDB7E44C32F9\n"},{"id":"237321","messageId":"20140321164651.GD2965@fu.192.168.200.1","threadId":"36266","inReplyTo":"20140321163628.GA28121@ruderich.org","subject":"Re: File extension conflict when working with git and latex","fromName":"Matthias Beyer","fromEmail":"mail@beyermatthias.de","sentAt":"2014-03-21T16:46:51Z","receivedAt":"2014-03-21T16:46:51Z","isPatch":false,"sender":{"key":"mail@beyermatthias.de","avatar":null},"body":"On 21-03-2014 17:36:28, Simon Ruderich wrote:\n> On Fri, Mar 21, 2014 at 05:13:24PM +0100, Matthias Beyer wrote:\n> > Unfortunetely, we wrote our `make clean` task recursively. I think you\n> > can imagine what went wrong: The clean-task corrupted the repository,\n> > as it removed .idx files from within .git/.\n> >\n> > I lost work because of this ugly name collision.\n>\n> Hello Matthias,\n>\n> You can recreate the .idx files by running\n>\n> git index-pack .git/objects/pack/pack-.pack\n>\n> for each pack file.\n\nHi Simon,\n\nI think so. I executed:\n\n git fsck # reports N missing blobs, commits, trees and dangling stuff\n git index-pack ...\n git fsck # reports only dangling commits and blobs\n\nI don't know if this means that the repository is fixed now?\n\n-- \nMit freundlichen Grüßen,\nKind regards,\nMatthias Beyer\n\nProudly sent with mutt.\nHappily signed with gnupg.\n"},{"id":"237324","messageId":"20140321170142.GA30418@ruderich.org","threadId":"36266","inReplyTo":"20140321164651.GD2965@fu.192.168.200.1","subject":"Re: File extension conflict when working with git and latex","fromName":"Simon Ruderich","fromEmail":"simon@ruderich.org","sentAt":"2014-03-21T17:01:42Z","receivedAt":"2014-03-21T17:01:42Z","isPatch":false,"sender":{"key":"simon@ruderich.org","avatar":"https://avatars.githubusercontent.com/u/390994?v=4"},"body":"On Fri, Mar 21, 2014 at 05:46:51PM +0100, Matthias Beyer wrote:\n> Hi Simon,\n>\n> I think so. I executed:\n>\n> git fsck # reports N missing blobs, commits, trees and dangling stuff\n> git index-pack ...\n> git fsck # reports only dangling commits and blobs\n>\n> I don't know if this means that the repository is fixed now?\n\nHello Matthias,\n\nIf nothing else was deleted, then yes, the repository should be\nfine now.\n\nRegards\nSimon\n-- \n+ privacy is necessary\n+ using gnupg http://gnupg.org\n+ public key id: 0x92FEFDB7E44C32F9\n"},{"id":"237340","messageId":"xmqqr45vtnay.fsf@gitster.dls.corp.google.com","threadId":"36266","inReplyTo":"20140321161324.GC2965@fu.192.168.200.1","subject":"Re: File extension conflict when working with git and latex","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-21T18:32:37Z","receivedAt":"2014-03-21T18:32:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthias Beyer writes:\n\n> I know, I can fix this by fixing the clean task in my Makefile. But\n> maybe someone somewhere on this world doesn't know the git internals\n> as good as me (and, of course, my coworker). Is there _any chance\n> at all_ that this gets mentioned somewhere, so others don't fall into\n> this pit?\n\nSurely, we are here to please ;-) All of us want to make sure\nnewbies do not shoot themselves in the foot.\n\nBut the problem is what exactly should be mentioned. With a fresh\nwound with your LaTeX project still in your mind, you may be tempted\nto special case \".idx\", but other newbies may inflict the same kind\nof hurt on themselves with different \"find\" patterns, e.g.\n\n $ find . -name '[0-9a-f]*[0-9a-f]' -type f -print | xargs rm -f\n\nwhen they know their project creates hexadecimal-numbered temoprary\nfiles, or whatever other pattern that match the files they do not\ncare about, that also happens to match whatever is in $GIT_DIR. The\nonly common caution that helps us to make sure \"others do not fall\ninto this pit\" is \"Files and directories in $GIT_DIR are used to\nrecord your work; do not muck with them unless you know what you are\ndoing e.g. manually repairing a corrupt repository\", but that is a\nbit lame, isn't it?\n\nIt is tempting to suggest \"git clean '*.idx'\", but that is a good\nfit in the Makefile only when you know everybody involved in the\nproject works in a checkout from Git, not from a tarball extract,\nand does not apply to projects in general.\n"},{"id":"237375","messageId":"532CAF94.8040108@alum.mit.edu","threadId":"36266","inReplyTo":"xmqqr45vtnay.fsf@gitster.dls.corp.google.com","subject":"Re: File extension conflict when working with git and latex","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2014-03-21T21:31:00Z","receivedAt":"2014-03-21T21:31:00Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 03/21/2014 07:32 PM, Junio C Hamano wrote:\n> Matthias Beyer writes:\n> \n>> I know, I can fix this by fixing the clean task in my Makefile. But\n>> maybe someone somewhere on this world doesn't know the git internals\n>> as good as me (and, of course, my coworker). Is there _any chance\n>> at all_ that this gets mentioned somewhere, so others don't fall into\n>> this pit?\n> \n> Surely, we are here to please ;-) All of us want to make sure\n> newbies do not shoot themselves in the foot.\n> \n> But the problem is what exactly should be mentioned. With a fresh\n> wound with your LaTeX project still in your mind, you may be tempted\n> to special case \".idx\", but other newbies may inflict the same kind\n> of hurt on themselves with different \"find\" patterns, e.g.\n> \n> $ find . -name '[0-9a-f]*[0-9a-f]' -type f -print | xargs rm -f\n> \n> when they know their project creates hexadecimal-numbered temoprary\n> files, or whatever other pattern that match the files they do not\n> care about, that also happens to match whatever is in $GIT_DIR. The\n> only common caution that helps us to make sure \"others do not fall\n> into this pit\" is \"Files and directories in $GIT_DIR are used to\n> record your work; do not muck with them unless you know what you are\n> doing e.g. manually repairing a corrupt repository\", but that is a\n> bit lame, isn't it?\n\nIt might help some people to show the \"find\" incantation for skipping\nfiles under .git, which I think would be something like\n\n find . -name .git -prune -o \\( [...] \\) -print0 | xargs -0 rm -f\n\nwhere \"[...]\" is whatever condition the user would otherwise use.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"237386","messageId":"532D37B6.1000306@web.de","threadId":"36266","inReplyTo":"20140321161324.GC2965@fu.192.168.200.1","subject":"Re: File extension conflict when working with git and latex","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-03-22T07:11:50Z","receivedAt":"2014-03-22T07:11:50Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2014-03-21 17.13, Matthias Beyer wrote:\n> Hi,\n> \n> I struggled with a really nasty issue today (and yesterday):\n> \n> I work on a semester project paper which is written in latex and\n> therefor version-controlled with git. We compile the document using\n> pdflatex, we automate the compiling with make.\n> \n> pdflatex generates some temporary files, which are required if you\n> call it the second and third time (it is a top-down compiler, which\n> needs to be called several times). One of the file extensions is\n> \".idx\", which (kind of) conflicts with file extensions in .git/.\n> \n> Unfortunetely, we wrote our `make clean` task recursively. I think you\n> can imagine what went wrong: The clean-task corrupted the repository,\n> as it removed .idx files from within .git/.\n> \n> I lost work because of this ugly name collision.\n> \n> My question now:\n> \n> I know, I can fix this by fixing the clean task in my Makefile. But\n> maybe someone somewhere on this world doesn't know the git internals\n> as good as me (and, of course, my coworker). Is there _any chance\n> at all_ that this gets mentioned somewhere, so others don't fall into\n> this pit?\n> \nThe idx files are mentioned here:\nhttp://www.git-scm.com/book/en/Git-Internals-Packfiles\n\n(Otherwise, some random thoughts:)\n- stay out of the .git directory, whatever you do\n (I lost some data because my faulty scripts using sed to replace\n \"s/this/that/\" and went into the .git directory) \n- Don't delete \"read-only\" files\n (The .idx files are all \"r--r--r--\")\n Or in other words, do not use \"rf -f\" in a script, unless you know exactly\n what you and the script are doing.\n- Replace \"rm -f\" whith \"mv /tmp\" could be an option\n- As mentioned before, \"git clean\" can be an option,\n I used it once when the \"make clean\", \"make distclean\", \"make reallydistclean\"\n did not work as expected by me,\n some auto-generated config files where still not deleted. \n- Make backups of your repo (frequently)\n- push the repo to a backup repo (frequently)\n- Use $GIT_DIR to move the repo out of your work tree:\n https://www.kernel.org/pub/software/scm/git/docs/git.html\n\n- Prepare a patch that renames the *.idx files used by Git into\n *.gitidx, *.gidx, *.packidx, (and the *.pack into *.packdata ??) \n\nHopefully you did not loose too much data, and if you have an idea how to improve\nthings, you may consider sending a patch, improvements are possible and welcome.\n"}]}