{"thread":{"id":"11230","subject":"backups with git and inotify","startedAt":"2007-12-10T20:29:11Z","lastAt":"2007-12-11T13:24:16Z","messageCount":6,"participants":["Luciano Rocha","David Tweed","Björn Steinbrink"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"62615","messageId":"20071210202911.GA14738@bit.office.eurotux.com","threadId":"11230","inReplyTo":null,"subject":"backups with git and inotify","fromName":"Luciano Rocha","fromEmail":"luciano@eurotux.com","sentAt":"2007-12-10T20:29:11Z","receivedAt":"2007-12-10T20:29:11Z","isPatch":false,"sender":{"key":"luciano@eurotux.com","avatar":null},"body":"\nHello,\n\nThe following is a work in progress. There are some problems in how I'm\nusing git and recording the history:\n\n1. I use an opened fd for each monitored directory (and subdirectories),\n   (inotify_add_watch_at would be nice).\n   I fchdir(fd) when a change happens to register and commit it.\n\n2. git-rm dir/file also removes <dir> if file was the only entry of\n   <dir>. So, when committing the removal, git complains that it can't\n   find cwd. So I record the parent directory, do the git command, check\n   if getcwd() works, and if not do the commit in the parent directory.\n\n3. git-rm (empty) directory fails\n\n4. Changes aren't atomic, but I can live with that and I doubt I would\n   be able to make it atomic without implementing a filesystem (FUSE or\n   not).\n\nI can work around most of the problems, and rewrite to use recorded path\nnames instead of directories fd, but before I do that, and while I'm\nat the beginning, I'd like to probe for opinions and suggestions.\n\nSo, please, suggest.\n\nRegards,\nLuciano Rocha\n\n-- \nLuciano Rocha <luciano@eurotux.com>\nEurotux Informática, S.A. <http://www.eurotux.com/>\n\n\n/*\n    monitor with inotify, record with git\n\n    Copyright (C) 2007, Luciano Rocha <luciano@nsk.pt>\n    Released under the GPL v2 or later. See LICENSE.GPL.\n \n    This program is free software; you can redistribute it and/or modify\n    it under the terms of the GNU General Public License as published by\n    the Free Software Foundation; either version 2 of the License, or\n    (at your option) any later version.\n\n    This program is distributed in the hope that it will be useful,\n    but WITHOUT ANY WARRANTY; without even the implied warranty of\n    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n    GNU General Public License for more details.\n\n    You should have received a copy of the GNU General Public License\n    along with this program; if not, write to the Free Software\n    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA\n*/\n\n#define _ATFILE_SOURCE 1\n#define _GNU_SOURCE 1\n#include <sys/inotify.h>\n#include <sys/types.h>\n#include <sys/stat.h>\n#include <sys/wait.h>\n#include <errno.h>\n#include <fcntl.h>\n#include <unistd.h>\n#include <dirent.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n\nstruct ilist {\n\tint fd;\n\tuint32_t wd;\n\tstruct ilist *prev, *next;\n};\n\ntypedef struct ilist *ilist;\n\n/* inotify_add_watch macro, with desired mask */\n#define INOTIFY_ADD(ifd, dir) (inotify_add_watch((ifd), (dir), \\\n\t\t\tIN_CLOSE_WRITE | IN_CREATE \\\n\t\t\t| IN_DELETE | IN_DELETE_SELF \\\n\t\t\t| IN_MOVED_FROM | IN_MOVED_TO))\n\n/* add directory to inotify watch list\n * inotify_add_watchat would be nice, but it doesn't exist, so\n * read symlink in /proc/self/fd/<dirfd> instead\n */\nint add_inotify(int fd, int ifd)\n{\n\tchar p[32];\n\tchar *dir;\n\tint l;\n\tint wd;\n\tssize_t ll;\n\n\tsnprintf(p, sizeof p, \"/proc/self/fd/%d\", fd);\n\tdir = NULL;\n\tl = 0;\n\tdo {\n\t\tl += 256;\n\t\tif (dir)\n\t\t\tfree(dir);\n\t\tdir = malloc(l);\n\t\tll = readlink(p, dir, l);\n\t\tif (ll < 0) {\n\t\t\tperror(p);\n\t\t\tfree(dir);\n\t\t\treturn -1;\n\t\t}\n\t} while (ll >= l);\n\tdir[strlen(dir) - 1] = '\\0';\n\twd = INOTIFY_ADD(ifd, dir);\n\tif (wd < 0)\n\t\tperror(dir);\n\tfree(dir);\n\treturn wd;\n}\n\n/* add watch for directory and each sub-directory, unless\n * there's a .git inside\n */\nvoid add_watch(int root, const char *name, ilist *head, int ifd,\n\t\tconst char **except, const char **skip)\n{\n\tilist new;\n\tDIR *dir;\n\tstruct dirent *d;\n\tint dirfd, DIRfd;\n\tint wd;\n\tint i;\n\n\tnew = malloc(sizeof *new);\n\tif (!new) {\n\t\tperror(name);\n\t\treturn;\n\t}\n\n\tdirfd = openat(root, name, O_RDONLY | O_DIRECTORY | O_NOATIME\n\t\t\t| O_NOFOLLOW);\n\n\tif (dirfd < 0) {\n\t\tperror(name);\n\t\tfree(new);\n\t\treturn;\n\t}\n\n\tif (except) {\n\t\tfor (i = 0; except[i] && faccessat(dirfd, except[i],\n\t\t\t\t\tR_OK | X_OK,\n\t\t\t\t\tAT_EACCESS | AT_SYMLINK_NOFOLLOW); i++);\n\t\tif (except[i]) {\n\t\t\tprintf(\"skipping %s (%s exists)\\n\", name, except[i]);\n\t\t\tfree(new);\n\t\t\tclose(dirfd);\n\t\t\treturn;\n\t\t}\n\t}\n\n\twd = add_inotify(dirfd, ifd);\n\tif (wd < 0) {\n\t\tfree(new);\n\t\tclose(dirfd);\n\t\treturn;\n\t}\n\n\tDIRfd = dup(dirfd);\n\tdir = fdopendir(DIRfd);\n\tif (!dir) {\n\t\tperror(name);\n\t\tfree(new);\n\t\tclose(dirfd);\n\t\tclose(DIRfd);\n\t\tinotify_rm_watch(ifd, wd);\n\t\treturn;\n\t}\n\n\twhile ((d = readdir(dir))) {\n\t\tif (!S_ISDIR(d->d_type << 12))\n\t\t\tcontinue;\n\t\tif (d->d_name[0] == '.' && (d->d_name[1] == '\\0'\n\t\t\t\t\t|| (d->d_name[1] == '.'\n\t\t\t\t\t\t&& d->d_name[2] == '\\0')))\n\t\t\tcontinue;\n\n\t\tif (skip) {\n\t\t\tfor (i = 0; skip[i] && strcmp(skip[i], d->d_name); i++);\n\t\t\tif (skip[i])\n\t\t\t\tcontinue;\n\t\t}\n\n\t\tadd_watch(dirfd, d->d_name, head, ifd, except, skip);\n\t}\n\tclosedir(dir);\n\n\t/* add to list */\n\tnew->fd = dirfd;\n\tnew->wd = wd;\n\tnew->next = *head;\n\tif (*head) (*head)->prev = new;\n\tnew->prev = NULL;\n\t*head = new;\n}\n\nstatic const char *default_except[] = {\n\t\".git\",\n\tNULL,\n};\n\n/* add watch to a directory and its sub-directories, complain and do\n * nothing if no .git exists\n */\nvoid git_watch(const char *name, ilist *head, int ifd, const char **skip)\n{\n\tilist new;\n\tDIR *dir;\n\tstruct dirent *d;\n\tint dirfd, DIRfd;\n\tint wd;\n\tint i;\n\n\tnew = malloc(sizeof *new);\n\tif (!new) {\n\t\tperror(name);\n\t\treturn;\n\t}\n\n\tdirfd = open(name, O_RDONLY | O_DIRECTORY | O_NOATIME | O_NOFOLLOW);\n\n\tif (dirfd < 0) {\n\t\tperror(name);\n\t\tfree(new);\n\t\treturn;\n\t}\n\n\tif (faccessat(dirfd, \".git\", R_OK | X_OK,\n\t\t\t\tAT_EACCESS | AT_SYMLINK_NOFOLLOW)) {\n\t\tfprintf(stderr, \"couldn't access .git subdir of %s: %s\\n\",\n\t\t\t\tname, strerror(errno));\n\t\tfree(new);\n\t\tclose(dirfd);\n\t\treturn;\n\t}\n\n\twd = INOTIFY_ADD(ifd, name);\n\tif (wd < 0) {\n\t\tperror(name);\n\t\tfree(new);\n\t\tclose(dirfd);\n\t\treturn;\n\t}\n\n\tDIRfd = dup(dirfd);\n\tdir = fdopendir(DIRfd);\n\tif (!dir) {\n\t\tperror(name);\n\t\tfree(new);\n\t\tclose(dirfd);\n\t\tclose(DIRfd);\n\t\tinotify_rm_watch(ifd, wd);\n\t\treturn;\n\t}\n\n\twhile ((d = readdir(dir))) {\n\t\tif (!S_ISDIR(d->d_type << 12))\n\t\t\tcontinue;\n\t\tif (d->d_name[0] == '.' && (d->d_name[1] == '\\0'\n\t\t\t\t\t|| (d->d_name[1] == '.'\n\t\t\t\t\t\t&& d->d_name[2] == '\\0')))\n\t\t\tcontinue;\n\t\tif (!strcmp(\".git\", d->d_name))\n\t\t\tcontinue;\n\t\tif (skip) {\n\t\t\tfor (i = 0; skip[i] && strcmp(skip[i], d->d_name); i++);\n\t\t\tif (skip[i])\n\t\t\t\tcontinue;\n\t\t}\n\n\t\tadd_watch(dirfd, d->d_name, head, ifd, default_except, skip);\n\t}\n\tclosedir(dir);\n\n\t/* add to list */\n\tnew->fd = dirfd;\n\tnew->wd = wd;\n\tnew->next = *head;\n\tif (*head) (*head)->prev = new;\n\tnew->prev = NULL;\n\t*head = new;\n}\n\n/* run a shell command, abort on error\n */\nvoid run_command(char *argv[])\n{\n\tpid_t pid;\n\tint status;\n\n\tpid = fork();\n\tif (pid < 0) {\n\t\tperror(\"fork(2)\");\n\t\texit(1);\n\t}\n\n\tif (pid == 0) {\n\t\texecvp(argv[0], argv);\n\t\tperror(argv[0]);\n\t\texit(1);\n\t}\n\n\tif (waitpid(pid, &status, 0) < 0) {\n\t\tperror(\"couldn't wait for child\");\n\t\texit(1);\n\t}\n\n\tif (WIFEXITED(status) && WEXITSTATUS(status) == 0)\n\t\treturn;\n\n\tfprintf(stderr, \"sub-command %s returned invalid exit code: %d\\n\",\n\t\t\targv[0], status);\n\texit(1);\n}\n\n/* run git command, and follow it with git-commit\n */\nvoid run_git(int fd, char *cmd, char *what)\n{\n\tint parent;\n\tint cl = strlen(cmd);\n\tint wl = strlen(what);\n\tchar commit[cl + wl + 5];\n\tchar *argv[] = {\n\t\t\"git\",\n\t\tcmd,\n\t\twhat,\n\t\tNULL,\n\t};\n\n\tif (fchdir(fd))\n\t\treturn;\n\n\t/* save parent:\n\t * git-rm of last file in subdir removes the directory, so the\n\t * following git-commit fails\n\t */\n\tparent = open(\"..\", O_RDONLY | O_DIRECTORY | O_NOATIME | O_NOFOLLOW);\n\n\t/* run git sub-command */\n\trun_command(argv);\n\n\tif (getcwd(commit, cl + wl) == NULL && errno != ERANGE && parent >= 0) {\n\t\tprintf(\"errno: %d, %s\\n\", errno, strerror(errno));\n\t\tfchdir(parent);\n\t}\n\n\t/* create commit message */\n\tcommit[0] = '-';\n\tcommit[1] = 'm';\n\tmemcpy(commit + 2, cmd, cl);\n\tcommit[cl + 2] = ':';\n\tcommit[cl + 3] = ' ';\n\tmemcpy(commit + cl + 4, what, wl + 1);\n\n\t/* commit change(s) */\n\targv[1] = \"commit\";\n\targv[2] = commit;\n\trun_command(argv);\n}\n\n/* get inotify events, run git as appropriate\n */\nvoid check_event(void *buffer, int len, ilist *head, int ifd)\n{\n\twhile (len >= sizeof(struct inotify_event)) {\n\t\tstruct inotify_event *p = buffer;\n\t\tilist l;\n\n\t\t/* advance buffer position */\n\t\tlen -= sizeof(struct inotify_event) + p->len;\n\t\tbuffer += sizeof(struct inotify_event) + p->len;\n\n\t\tfor (l = *head; l && l->wd != p->wd; l = l->next);\n\n\t\tif (!l) {\n\t\t\t/* not found in list? */\n\t\t\tcontinue;\n\t\t}\n\n\t\tif (p->mask & (IN_IGNORED | IN_UNMOUNT | IN_DELETE_SELF)) {\n\t\t\t/* remove it */\n\t\t\tinotify_rm_watch(ifd, p->wd);\n\t\t\tclose(l->fd);\n\n\t\t\tif (l->prev)\n\t\t\t\tl->prev->next = l->next;\n\t\t\telse\n\t\t\t\t*head = l->next;\n\n\t\t\tif (l->next)\n\t\t\t\tl->next->prev = l->prev;\n\t\t}\n\n\t\t/* the following events require a file name specification,\n\t\t * changes to the directory itself aren't of our interest\n\t\t */\n\t\tif (p->len == 0)\n\t\t\tcontinue;\n\n\t\tif (p->mask & IN_CREATE) {\n\t\t\t/* add new watch if directory, otherwise ignore,\n\t\t\t * IN_CLOSE_WRITE should follow\n\t\t\t */\n\t\t\tstruct stat st;\n\t\t\tif (!fstatat(l->fd, p->name, &st, AT_SYMLINK_NOFOLLOW)\n\t\t\t\t\t&& S_ISDIR(st.st_mode))\n\t\t\t\tadd_watch(l->fd, p->name, head, ifd,\n\t\t\t\t\t\tdefault_except, NULL);\n\t\t}\n\n\t\tif (p->mask & (IN_CLOSE_WRITE | IN_MOVED_TO)) {\n\t\t\t/* add/commit */\n\t\t\trun_git(l->fd, \"add\", p->name);\n\t\t}\n\n\n\t\tif (p->mask & (IN_DELETE | IN_MOVED_FROM)) {\n\t\t\t/* rm/commit */\n\t\t\trun_git(l->fd, \"rm\", p->name);\n\t\t}\n\t}\n}\n\nint main(int argc, char *argv[])\n{\n\tint fd;\n\tint i;\n\tilist head;\n\tvoid *buffer;\n\n\tfd = inotify_init();\n\tif (fd < 0) {\n\t\tperror(\"init inotify\");\n\t\treturn 1;\n\t}\n\n\n\tbuffer = malloc(1<<20);\n\tif (!buffer) {\n\t\tperror(\"buffer allocation\");\n\t\treturn 1;\n\t}\n\n\thead = NULL;\n\n\twhile (*++argv) {\n\t\tgit_watch(*argv, &head, fd, NULL);\n\t}\n\n\tif (!head) {\n\t\tprintf(\"nothing to do\\n\");\n\t\treturn 0;\n\t}\n\n\t/* loop until there's an error or all watched elements are\n\t * removed or made inaccessible\n\t */\n\twhile (head) {\n\t\ti = read(fd, buffer, 1<<20);\n\t\tif (i == 0 || (i < 0 && errno != EINTR && errno != EAGAIN))\n\t\t\tbreak;\n\t\tif (i < 0)\n\t\t\tcontinue;\n\t\tcheck_event(buffer, i, &head, fd);\n\t}\n\n\tif (i < 0) {\n\t\tperror(\"reading event\");\n\t\treturn 1;\n\t}\n\n\treturn 0;\n}\n"},{"id":"62617","messageId":"e1dab3980712101318v264fcce5pebbb829d8cefb1ac@mail.gmail.com","threadId":"11230","inReplyTo":"20071210202911.GA14738@bit.office.eurotux.com","subject":"Re: backups with git and inotify","fromName":"David Tweed","fromEmail":"david.tweed@gmail.com","sentAt":"2007-12-10T21:18:18Z","receivedAt":"2007-12-10T21:18:18Z","isPatch":false,"sender":{"key":"david.tweed@gmail.com","avatar":null},"body":"Hi, looks interesting project.\n\nI've been doing something similar-ish (but periodically rather than\nupon changes). Here are some rough off-the-cuff observations (probably\ntelling you things you already know).\n\nFirstly, are you doing backups (be able to restore to n previous\nstates upon catastrophe) or archives (being able to lookup arbitrary\npoints in history to compare with current stuff, eg, for regressions)?\n(Archiving is useful if you aren't in a disciplined enough project to\ndo rewritten proper commits but still want to be able to look around\nand try to figure out what's caused regressions, etc.)\n\nThe scripts I use are at\n\nhttp://www.personal.rdg.ac.uk/~sis05dst/chronoversion.tgz\n\nbut they're designed around archiving rather than backups.\n\nOn Dec 10, 2007 8:29 PM, Luciano Rocha <luciano@eurotux.com> wrote:\n> The following is a work in progress. There are some problems in how I'm\n> using git and recording the history:\n>\n> 1. I use an opened fd for each monitored directory (and subdirectories),\n>    (inotify_add_watch_at would be nice).\n>    I fchdir(fd) when a change happens to register and commit it.\n\nI thought about trying to have a daemon using inotify to record the\ngit-add's/git-rm's but keeping the cron driven actual commits, and\nlooked at python support module. I didn't because firstly I wasn't\nsure how far inotify scaled (the fact the Linux VFS maintainer insists\non calling it \"idiotify\" doesn't inspire confidence). If it was me,\nI'd pull the git-commit outside your loop that does the git-add/git-rm\n(see later comment about emacs, etc). Obviously if your buffer isn't\ncompletely emptied you'll get a misleading granularity of commits, but\nthen I guess that'll happen anyway. I think inotify drops events if an\ninternal queue fills: personally I'd try to check for that and\ninitiate manually scanning if I detected that happening.\n\n> 2. git-rm dir/file also removes <dir> if file was the only entry of\n>    <dir>. So, when committing the removal, git complains that it can't\n>    find cwd. So I record the parent directory, do the git command, check\n>    if getcwd() works, and if not do the commit in the parent directory.\n>\n> 3. git-rm (empty) directory fails\n>\n> 4. Changes aren't atomic, but I can live with that and I doubt I would\n>    be able to make it atomic without implementing a filesystem (FUSE or\n>    not).\n\nWith things like emacs that do update writes by writing a new file\nwith a temporary name and then copying it over the top of the old\nfile, you'll get presumably 3 commits. Is that acceptable?\n\n> I can work around most of the problems, and rewrite to use recorded path\n> names instead of directories fd, but before I do that, and while I'm\n> at the beginning, I'd like to probe for opinions and suggestions.\n\nThe only other thing that occurs to me is whether you need any greater\nsupport for stopping the automatic monitoring than just stopping the\ndaemon. Eg, what happens if you decide you need to recover a previous\nversion of a file. Git checks it out, presumably updates the index\nitself and then inotify fires off a git-add that will want to write to\nthe index. Basically, I'm trying to think if there's any situation\nwhere you can have a delete event that git causes, followed by a\ncreating some new content where delay in your program processing the\ndelete will cause the new content to be `lost'? (I know, I sould read\nthe code.) In chronoversion, the first thing it does is check for a\n\"suppress\" file which stops it doing anything automatically and I put\none in there whenever I'm doing anything more than looking at the data\n(eg, switch branch, checkout old version, etc). But I might be being\nhyper-cautious.\n\n-- \ncheers, dave tweed__________________________\ndavid.tweed@gmail.com\nRm 124, School of Systems Engineering, University of Reading.\n\"we had no idea that when we added templates we were adding a Turing-\ncomplete compile-time language.\" -- C++ standardisation committee\n"},{"id":"62623","messageId":"20071210214728.GB17458@bit.office.eurotux.com","threadId":"11230","inReplyTo":"e1dab3980712101318v264fcce5pebbb829d8cefb1ac@mail.gmail.com","subject":"Re: backups with git and inotify","fromName":"Luciano Rocha","fromEmail":"luciano@eurotux.com","sentAt":"2007-12-10T21:47:28Z","receivedAt":"2007-12-10T21:47:28Z","isPatch":false,"sender":{"key":"luciano@eurotux.com","avatar":null},"body":"On Mon, Dec 10, 2007 at 09:18:18PM +0000, David Tweed wrote:\n> Hi, looks interesting project.\n> \n> I've been doing something similar-ish (but periodically rather than\n> upon changes). Here are some rough off-the-cuff observations (probably\n> telling you things you already know).\n\nThanks.\n\n> \n> Firstly, are you doing backups (be able to restore to n previous\n> states upon catastrophe) or archives (being able to lookup arbitrary\n> points in history to compare with current stuff, eg, for regressions)?\n> (Archiving is useful if you aren't in a disciplined enough project to\n> do rewritten proper commits but still want to be able to look around\n> and try to figure out what's caused regressions, etc.)\n\nArchives, only. If the project requires a coherent state, then I don't\nthink that automatic commits are the way to go.\n\n> \n> The scripts I use are at\n> \n> http://www.personal.rdg.ac.uk/~sis05dst/chronoversion.tgz\n> \n> but they're designed around archiving rather than backups.\n\nThanks, I'll take a look, and maybe borrow some ideas. :)\n\n> \n> On Dec 10, 2007 8:29 PM, Luciano Rocha <luciano@eurotux.com> wrote:\n> > The following is a work in progress. There are some problems in how I'm\n> > using git and recording the history:\n> >\n> > 1. I use an opened fd for each monitored directory (and subdirectories),\n> >    (inotify_add_watch_at would be nice).\n> >    I fchdir(fd) when a change happens to register and commit it.\n> \n> I thought about trying to have a daemon using inotify to record the\n> git-add's/git-rm's but keeping the cron driven actual commits, and\n> looked at python support module. I didn't because firstly I wasn't\n> sure how far inotify scaled (the fact the Linux VFS maintainer insists\n> on calling it \"idiotify\" doesn't inspire confidence). If it was me,\n> I'd pull the git-commit outside your loop that does the git-add/git-rm\n> (see later comment about emacs, etc).\n\nI'd like to have the changes committed as soon as an application closes\nits files. As I monitor a small subset of possible inotify events, I\nthink I shouldn't have much problems with scale. I'll have to test it\nwith my Maildir, though, to have a definitive answer.\n\n> Obviously if your buffer isn't\n> completely emptied you'll get a misleading granularity of commits, but\n> then I guess that'll happen anyway. I think inotify drops events if an\n> internal queue fills: personally I'd try to check for that and\n> initiate manually scanning if I detected that happening.\n\nHm, that could be a problem. Maybe a periodic git-status followed by\ngit-add/rm, etc.. Hourly, perhaps.\n\n> \n> > 2. git-rm dir/file also removes <dir> if file was the only entry of\n> >    <dir>. So, when committing the removal, git complains that it can't\n> >    find cwd. So I record the parent directory, do the git command, check\n> >    if getcwd() works, and if not do the commit in the parent directory.\n> >\n> > 3. git-rm (empty) directory fails\n> >\n> > 4. Changes aren't atomic, but I can live with that and I doubt I would\n> >    be able to make it atomic without implementing a filesystem (FUSE or\n> >    not).\n> \n> With things like emacs that do update writes by writing a new file\n> with a temporary name and then copying it over the top of the old\n> file, you'll get presumably 3 commits. Is that acceptable?\n\nNo. Vim also has that behaviour. I plan on accepting ignore patterns,\nand maybe also parse .gitignore, and add those temporary files (*~,\n.*.sw[po]), etc.) implicitly.\n\n> \n> > I can work around most of the problems, and rewrite to use recorded path\n> > names instead of directories fd, but before I do that, and while I'm\n> > at the beginning, I'd like to probe for opinions and suggestions.\n> \n> The only other thing that occurs to me is whether you need any greater\n> support for stopping the automatic monitoring than just stopping the\n> daemon. Eg, what happens if you decide you need to recover a previous\n> version of a file. Git checks it out, presumably updates the index\n> itself and then inotify fires off a git-add that will want to write to\n> the index. Basically, I'm trying to think if there's any situation\n> where you can have a delete event that git causes, followed by a\n> creating some new content where delay in your program processing the\n> delete will cause the new content to be `lost'? (I know, I sould read\n> the code.) In chronoversion, the first thing it does is check for a\n> \"suppress\" file which stops it doing anything automatically and I put\n> one in there whenever I'm doing anything more than looking at the data\n> (eg, switch branch, checkout old version, etc). But I might be being\n> hyper-cautious.\n\nI'll have to think about that. A stop/pause button is a good idea, and\nchecking if the tree is at HEAD. I don't think a commit of changes to a\nfile checked-out to a previous version will lose any information,\nbut I'll check.\n\n-- \nLuciano Rocha <luciano@eurotux.com>\nEurotux Informática, S.A. <http://www.eurotux.com/>\n"},{"id":"62624","messageId":"20071210215746.GA3022@atjola.homenet","threadId":"11230","inReplyTo":"20071210202911.GA14738@bit.office.eurotux.com","subject":"Re: backups with git and inotify","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2007-12-10T21:57:46Z","receivedAt":"2007-12-10T21:57:46Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2007.12.10 20:29:11 +0000, Luciano Rocha wrote:\n> \n> Hello,\n> \n> The following is a work in progress. There are some problems in how I'm\n> using git and recording the history:\n> \n> 1. I use an opened fd for each monitored directory (and subdirectories),\n>    (inotify_add_watch_at would be nice).\n>    I fchdir(fd) when a change happens to register and commit it.\n> \n> 2. git-rm dir/file also removes <dir> if file was the only entry of\n>    <dir>. So, when committing the removal, git complains that it can't\n>    find cwd. So I record the parent directory, do the git command, check\n>    if getcwd() works, and if not do the commit in the parent directory.\n> \n> 3. git-rm (empty) directory fails\n> \n> 4. Changes aren't atomic, but I can live with that and I doubt I would\n>    be able to make it atomic without implementing a filesystem (FUSE or\n>    not).\n> \n> I can work around most of the problems, and rewrite to use recorded path\n> names instead of directories fd, but before I do that, and while I'm\n> at the beginning, I'd like to probe for opinions and suggestions.\n> \n> So, please, suggest.\n\nI posted an extremely simple bash script here:\nhttp://lkml.org/lkml/2007/12/7/279\n\nIt just employs inotifywait to do all watching and just needs to\ntranslate the events to the different git command. Did just glance over\nyour code, but it seems to do basically the same thing, just that it's a\nlot shorter. The overhead of being a shell script is probably neglible,\nas the amount of git calls are likely dominating anyway.\n\nFeel free to ignore my comments on why I think that that is crap anyway\nand do whatever you want with the script.\n\nHTH\nBjörn\n"},{"id":"62696","messageId":"20071211102546.GA5811@bit.office.eurotux.com","threadId":"11230","inReplyTo":"20071210215746.GA3022@atjola.homenet","subject":"Re: backups with git and inotify","fromName":"Luciano Rocha","fromEmail":"luciano@eurotux.com","sentAt":"2007-12-11T10:25:46Z","receivedAt":"2007-12-11T10:25:46Z","isPatch":false,"sender":{"key":"luciano@eurotux.com","avatar":null},"body":"On Mon, Dec 10, 2007 at 10:57:46PM +0100, Björn Steinbrink wrote:\n> On 2007.12.10 20:29:11 +0000, Luciano Rocha wrote:\n<snip>\n> > So, please, suggest.\n> \n> I posted an extremely simple bash script here:\n> http://lkml.org/lkml/2007/12/7/279\n\nThat thread was what motivated me to write the program, but I must've\nmissed your post.\n\n> \n> It just employs inotifywait to do all watching and just needs to\n> translate the events to the different git command. Did just glance over\n> your code, but it seems to do basically the same thing, just that it's a\n> lot shorter. The overhead of being a shell script is probably neglible,\n> as the amount of git calls are likely dominating anyway.\n\nYes, being a shell script isn't a problem. I wasn't aware of\ninotify-tools, so I wrote my own. Still, the inotify-tools programs miss\na check for ignoring directories with predetermined contents (I want to\nignore sub-directories with their own git repository).\n\nI think I'll contribute to inotify-tools a switch for that, and switch\nto perl. That will simplify the development and allow for coalescing\nevents, so that updates with temporary files would be a simple update.\n\n> Feel free to ignore my comments on why I think that that is crap anyway\n> and do whatever you want with the script.\n\nFWIW, I also think that trying to keep a coherent stat with automatic\ncommits isn't possible. As for the temporary, unneeded files, a\nexclusion pattern will suffice, and using .git directly, instead of a\n(FUSE) filesystem, will allow permanent storage of those temporary\nfiles, until explicitly removed.\n\nThanks,\nLuciano Rocha\n\n-- \nLuciano Rocha <luciano@eurotux.com>\nEurotux Informática, S.A. <http://www.eurotux.com/>\n"},{"id":"62712","messageId":"e1dab3980712110524k59437f15pd38f74f55d692a69@mail.gmail.com","threadId":"11230","inReplyTo":"20071211102546.GA5811@bit.office.eurotux.com","subject":"Re: backups with git and inotify","fromName":"David Tweed","fromEmail":"david.tweed@gmail.com","sentAt":"2007-12-11T13:24:16Z","receivedAt":"2007-12-11T13:24:16Z","isPatch":false,"sender":{"key":"david.tweed@gmail.com","avatar":null},"body":"On Dec 11, 2007 10:25 AM, Luciano Rocha <luciano@eurotux.com> wrote:\n> On Mon, Dec 10, 2007 at 10:57:46PM +0100, Björn Steinbrink wrote:\n> > On 2007.12.10 20:29:11 +0000, Luciano Rocha wrote:\n> FWIW, I also think that trying to keep a coherent stat with automatic\n> commits isn't possible. As for the temporary, unneeded files, a\n> exclusion pattern will suffice, and using .git directly, instead of a\n> (FUSE) filesystem, will allow permanent storage of those temporary\n> files, until explicitly removed.\n\nAs a data point, I find it easier for my work to have rules that\nspecify the files you DO want to keep. (I actually have a set of\ncommon suffixes to ignore that's checked first but purely for\nefficiency: it's quicker to throw out the usual suspects like *~, *.o,\n*.pyc, etc, immediately rather than fail to match all the more\ncomplicated \"keep this\" rules every single time.) One advantage of\nthis is that if I do something stupid like drop a 2G video file into\nthe tracked tree it's doesn't get sucked in to the git repo.\n\n-- \ncheers, dave tweed__________________________\ndavid.tweed@gmail.com\nRm 124, School of Systems Engineering, University of Reading.\n\"we had no idea that when we added templates we were adding a Turing-\ncomplete compile-time language.\" -- C++ standardisation committee\n"}]}