{"thread":{"id":"1027","subject":"[ANNOUNCE] gitfs pre-release 0.01","startedAt":"2005-06-26T04:24:57Z","lastAt":"2005-06-26T04:24:57Z","messageCount":1,"participants":["Mitchell Blank Jr"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"5273","messageId":"20050626042457.GB84203@gaz.sfgoth.com","threadId":"1027","inReplyTo":null,"subject":"[ANNOUNCE] gitfs pre-release 0.01","fromName":"Mitchell Blank Jr","fromEmail":"mitch@sfgoth.com","sentAt":"2005-06-26T04:24:57Z","receivedAt":"2005-06-26T04:24:57Z","isPatch":false,"sender":{"key":"mitch@sfgoth.com","avatar":null},"body":"GITFS pre-release version 0.01\n\ngitfs is a FUSE-based filesystem for working with source trees stored in\ngit repositories.  Currently only very basic functionality is implemented\nbut I'm hoping to expand it into a useful tool for managing many builds\nand patches.\n\nOVERALL PLAN\n======= ====\n\nMany years ago I had an idea for a filesystem that would make importing\npatches and spinning kernel builds far more efficient.  The basic idea was:\n\n  1. Store the unchanged part of the source tree in a shared repository;\n     only keep a separate copy of the files and directories that have\n     changed.  This would be similar to doing hardlinked source tree copies\n     but be even faster -- checking out a new tree would be as quick as\n     mkdir'ing a new empty directory.\n\n  2. On top of this, implement a very-fast \"diff\" operator that only worked\n     on changed files\n\n  3. Have ccache-style compiler caching built in.  The filesystem could\n     (using some wrapper programs) watch every file read and written to\n     by a command like \"gcc\".  Since it knows what versions of those files\n     were read at that time it can know very quickly if any of them changed.\n\n     This saves the \"gcc -E\" step that ccache must do to determine that it\n     can use the cached .o result and should be quite a bit faster.\n\nSo basically common operations such as \"compile a test kernel with this\nfix\" and \"produce a well-formed patch describing how my current tree\ndiverges from mainline\" become very fast.\n\nSeveral times I started to implement this idea but every time I got bogged\ndown in the details of making kernel parts of the file system and such work.\nHowever, two things changed recently:\n\n  1. git came along; I realized that I could use it for the backing data\n     store.  Since the linux kernel is already published in git format\n     this is especially handy.  Leveraging the existing git code and\n     design has sped this up immeasurably.\n\n  2. FUSE (filesystem in userspace) has become more widely available --\n     it hasn't make it to mainline yet but it is in the -mm series kernels.\n     This made getting started on actual implementation a lot easier.\n     Writing a userspace filesystem on top of FUSE is really a joy.\n\nI'm currently calling this project \"gitfs\" although perhaps that is a\nbit of a misnomer since I am _absolutely_not_ trying to implement the\nfull SCM workflow as a filesystem.  In fact we present hardly any git\nmetadata like commit messages at all.  Also, I operate on the underlying\nobjects directly - the index file is never touched.\n\nHowever, I decided to stick with the \"gitfs\" name for now -- I'm hoping that\nthis project can grow to become a useful compliment to the git workflow.\nI'm not adverse to giving it a different name if it's an issue, though.\n\nCURRENT STATE\n======= =====\n\nThis is an early pre-release that only demonstrates the most basic of\nfunctionality -- read-only access to the existing tags and objects in\nthe git repository.  Still, it's already a somewhat handy tool which is\nwhy I'm announcing it now.\n\nIn addition to the missing functionality, currently there is a lot of\nperformance work to do -- I've been working on getting it functionally\ncorrect first.  Specific performance work I'm planing includes:\n\n  1. Every time we touch a directory (whether lookup or readdir) we parse\n     the git tree object into a memory structure which then immediately gets\n     thrown away.  There is some infrastructure for caching these objects\n     in memory which will solve the problem, but it's not completed yet.\n\n  2. On a related note, I need to do some data structure work -- in some\n     places I'm using simple linked lists where I really should be using\n     B-tree's or something.  I actually have a lot of this work done already\n     but I need to do some heavy testing before I integrate that into my\n     tree.\n\n  3. Since we cache the uncompressed file data our read/write operations\n     always go straight through to the underlying files.  A large performance\n     boost would be available if at open()-time we could tell the kernel\n     \"here's the file descriptor I opened for you, do I/O to that\"  That\n     way we could avoid the need for all data to make two user/kernel\n     transitions.  However, this would require some extensive work to\n     FUSE to implement.\n\n  4. We are currently single threaded; I eventually am planning on adding\n     service threads for handling CPU bound tasks.  I want to keep the\n     normal filesystem operations single-threaded (they're generally just\n     walking in-memory structures so they're fast anyway), but things like\n     uncompressing a git object should really be done in separate thread\n     so they won't block other filesystem operations.\n\nFinally, since I'm still working on finishing the infrastructure work, please\njust consider this a \"preview release\"  Feel free to play with it, look at\nthe code, poke it with sticks, etc.  However, the code base is still rapidly\nevolving so I probably won't be able to integrate any non-trivial patches yet.\nThe code also needs things like more comments and clear error messages.\n\nBUILDING GITFS\n======== =====\n\n  Gitfs can currently be obtained at:\n\thttp://www.sfgoth.com/~mitch/linux/gitfs/\n\n  Please refer to the included INSTALL file for directions on compiling\n  the gitfs binary.\n\nRUNNING GITFS\n======= =====\n\n  MOUNT:\n    gitfs [-d] [-O object_cache_dir] <gitdir> <mntpoint>\n  UMOUNT:\n    gitfs -u [-d] <dir>\n\n  Options:\n\n    -d -- debugging mode; we run in the foreground and print very verbose\n          messages about what is going on (mostly courtesy of FUSE)\n\n    -O -- specify an object cache directory.  For fast performance we always\n\t  store the result of decompressing a git \"blob\" object in a file.\n\t  This directory is where the decompressed objects live.\n\n    \t  This currently defaults to \"/tmp/gitfs/ocache\"  DO NOT make this\n\t  the same as your \".git/objects\" directory or things will probably\n\t  become horribly broken!\n\n\t  Currently gitfs never removes anything from the ocache so it\n\t  can grow quite large.  However it's safe to prune files from it\n\t  (or even blow away the entire tree) while gitfs is running.\n\nUnder normal operation gitfs would run in the background until you unmount\nit with \"gitfs -u\"  *However*, we currently always run in \"debug\" mode so the\ngitfs program runs in the foreground.  To shut down you just have to send\nit a ctrl-C and it should shut down cleanly.  For now you should only have\nto use \"gitfs -u\" if something goes wrong and it crashes.\n\nEXAMPLE SESSION\n======= =======\n\n  $ gitfs ~/git/linux-2.6 /tmp/fuse\n\n  [then in another window]\n  $ cd /tmp/fuse\n  $ ls -l\n  total 0\n  dr-xr-xr-x  2 mitch mitch 0 Apr 20 16:38 HEADS\n  dr-xr-xr-x  2 mitch mitch 0 May 24 20:32 TAGS\n  $ ls -l TAGS\n  total 0\n  lrwxrwxrwx  1 mitch mitch 43 May  4 16:51 v2.6.11 -> ../5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c\n  lrwxrwxrwx  1 mitch mitch 43 May  4 16:51 v2.6.11-tree -> ../5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c\n  lrwxrwxrwx  1 mitch mitch 43 May  1 17:16 v2.6.12-rc2 -> ../9e734775f7c22d2f89943ad6c745571f1930105f\n  lrwxrwxrwx  1 mitch mitch 43 May  1 17:15 v2.6.12-rc3 -> ../0397236d43e48e821cce5bbe6a80a1a56bb7cc3a\n  lrwxrwxrwx  1 mitch mitch 43 May  6 22:22 v2.6.12-rc4 -> ../ebb5573ea8beaf000d4833735f3e53acb9af844c\n  lrwxrwxrwx  1 mitch mitch 43 May 24 20:32 v2.6.12-rc5 -> ../06f6d9e2f140466eeb41e494e14167f90210f89d\n  $ cd TAGS/v2.6.11\n  $ ls\n  arch     Documentation  init    MAINTAINERS  README          sound\n  COPYING  drivers        ipc     Makefile     REPORTING-BUGS  usr\n  CREDITS  fs             kernel  mm           scripts\n  crypto   include        lib     net          security\n  $ pwd\n  /tmp/fuse/TAGS/v2.6.11\n  $ /bin/pwd\n  /var/tmp/fuse/5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c\n  $ ls -l /tmp/fuse\n  total 0\n  dr-xr-xr-x  18 mitch mitch 352 May  4 16:50 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c\n  dr-xr-xr-x   2 mitch mitch   0 Apr 20 16:38 HEADS\n  dr-xr-xr-x   2 mitch mitch   0 May 24 20:32 TAGS\n"}]}