Re: Can git be tweaked to work cross-platform, on FAT32?

Hi there,

I would like to use git to develop a somewhat larger Perl project which I would need to test and develop under both Linux and Windows, i.e. I would need a repository on a FAT32 partition (to my knowledge, still the only filesystem that can be fully accessed from both OSes - correct me if I'm wrong) which could be accessed by two compatible git executables, such that I could switch back and forth.

Before describing how I (unsuccessfully, which is perhaps not surprising, as I'm C-illiterate) hacked the Makefile to compile two git siblings, I would like to generally ask whether this is possible at all (or should be possible in theory, in which case I'm willing to test extensively), whether with Cygwin or not. The preliminary impression was that the Linux version would not accept a repository on FAT32 (it did work with more Unixish filesystems). - I do not need the networking capabilities, BTW (standalone development), if that is an issue in any way.

Thanks for any hint!

Florian

Hi,

On Wed, 13 Dec 2006, Florian v. Savigny wrote:

I would not use the _same_ repository from both Windows and Linux, but rather fetch/push from/to the Windows partition.

I am using Cygwin to compile git, and I have a USB Stick with FAT32 on which I push regularly. Important: since Linux is more intelligent about permissions, I had to initialize the repos with Windows.

To get started compiling, you might want to use the autoconf script.

Ciao, Dscho

Florian,

a bit of a long shot, but you might be able to access it via Samba, and just use git on linux. I have to confess to having setup a few git checkouts for Win32 users this way... in some cases with cygwin+x11 and a shortcut that will do something like

    ssh linuxhost gitk --all

or

    ssh linuxhost qgit

but then again, I'm rumoured to be insane...

cheers,

Thank you Martin and Johannes -

before proceeding by following either of your two suggestions, I would like to make sure that there is no other way, so I would like to ask:

1. I know, there is absolutely nothing in either of your answers that
   says that the route I conceived is not possible, but anyway: DO
   your answers imply that it is NOT possible (by "possible", I mean
   of course: possible without re-programming the source code ;-)) to
   compile git in a way that would enable it to work DIRECTLY with a
   local repository on FAT32 from both Windows and Linux (or, perhaps
   more simple: to work directly with a repository on FAT32 under
   Linux)?

   I'm repeating my question because it is impossible for me to tell
   whether this would involve a rather simple change (such as: don't
   use symlinks, and/or: don't care about user/group permissions), or
   whether the critical differences are rather an inherent
   characteristic of git, or perhaps even the libraries it uses.

   And: if this were possible, I would consider it a rather attractive
   idea, not only for me, hence I'd like to rule it out before
   dropping it altogether.

2. ad Martin: Ouf! I have no idea of how Samba works (well, yes: I
   thought it was some kind of server or client program permitting to
   connect to Windows servers or clients), hence I don't get a very
   clear idea of what you have in mind. I hope I have made it clear
   that I am using only one computer (double-boot, simply), and that
   everything is local. I am aware that Linux permits to do "virtual"
   networking with localhost, but how would your scheme work on a
   single machine? Can samba "pretend" there is a Windows network
   behind it, while in truth, it accesses a local directory (on
   FAT32)? And can git work directly in a repository it considers to
   be accessible over the network only?

   The interesting thing in this could be that one could use indeed
   the same repository under both OSes. If I got vaguely the right
   idea, that is.

3. ad Johannes: This does sound quite simple and straightforward. If I
   got it right, it would involve having one repository on a, say,
   ext2 partition to work with under Linux, and one on a FAT32
   partition to work with under Windows, and syncing the two after
   booting (fetching from FAT32) and before shutting down (pushing to
   FAT32) Linux.

   It is quite interesting, BTW, that git can /sync/ with a repository
   on FAT32 under Linux, but not work with it.

Thanks very much for both answers! I would still be curious to know whether there is an answer to my first question, but yours should get me working one way or other.

Regards, Florian

Hi,

On Sat, 16 Dec 2006, Florian v. Savigny wrote:

My answer was not so clear...

Question: Is it possible to use git on a FAT32 filesystem?

Answer: Yes.

Question: Is it possible to access such a repository from both Linux and 
Windows?

Answer: Yes.

Question: Is there anything that could cause problems?

Answer: Yes. The working directory is a problem.

	- Symlinks for example. Don't use them.
	- Umlauts: don't use them.
	- The index has to be updated, since the stat information is 
	  unlikely to match. A simple "git status" should suffice, though.
	- Permissions: the permission handling of Linux and Windows are
	  quite different. If you initialize the repository with Cygwin,
	  it automatically sets core.filemode=false, if you initalize(d) 
	  with Linux, you have to adjust manually.

Question: Are there other shortcomings to this approach?

Answer: Definitely. Most are only annoying (like the performance of FAT32, 
which literally sucks), but some are different: for example, if you 
compile things in the working directory, you will be unhappy.

This is how I'd do it.

You can. But I wouldn't. I like to keep separations clean and obvious.

Ciao,

On 12/17/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

And I concur - I had only introduced Samba to the conversation because I thought you were talking about several computers. If you are dual-booting on one machine, I'd do as above.

Note that under windows you can use ext2 -- haven't used it, and don't know how cygwin behaves with it, but it may be *just* what you need to avoid case sensitivity problems and have symlink support.

    http://sourceforge.net/projects/ext2fsd

    http://www.fs-driver.org/

cheers,

I just tried to use git writing to my FAT32 formatted usb stick.

On windows XP, I compiled git with and without NO_D_TYPE_IN_DIRENT, but after the following actions I always got the error "fatal: Unable to write new index file" or "fatal: unable to create '.git/index': File exists":

mkdir git-test-repo cd git-test-repo git-init-db echo iii>test git add test git commit -m "test msg" echo ooo>test1 git add test1 ***error***

converting the stick to NTFS I have no problems.

any hint ???

---

Hi,

On Sun, 17 Dec 2006, Stefano Spinucci wrote:

Come to think of it, I probably never tried to actually _commit_ on FAT32... Sorry.

I will not be able to test this scenario until Tuesday, though. Sorry again!

Ciao,

Stefano Spinucci writes:
 > I just tried to use git writing to my FAT32 formatted usb stick.
 > 
 > On windows XP, I compiled git with and without NO_D_TYPE_IN_DIRENT,
 > but after the
 > following actions I always got the error "fatal: Unable to write new
 > index file" or
 > "fatal: unable to create '.git/index': File exists":

Compile git with NO_MMAP. The problem is that the old index file is mmaped while it is replaced. Cygwin supports this operation only on NTFS. I don't know whether this is a bug in cygwin or a windows limitation (I suspect the latter).

Unfortunately NO_MMAP makes git pretty slow if the pack files get larger than about 10-20 MB.

Johannes claims that there are additional problems with mmap on cygwin, but it passes the complete test suite on NTFS, so it should be ok for most operations

IIRC it has been suggested on the list that mmaping the index isn't that important for git performance and it could be replaced by simply reading the index file into memory on access.

It would be nice to have mmap work on non-NTFS partitions since my tests suggest that FAT32 is up to twice as fast (in the hot cache case). (My test version simply uses an unlink before the rename in lockfile.c to hack around the cygwin rename problem.)

Hi,

On Sun, 17 Dec 2006, Juergen Ruehle wrote:

I encountered a problem with git-log, where the fork() tried to reestablish a mmap()ed file, which unfortunately was renamed (since it was a .lock file[*1*]). This triggered the implementation of NO_MMAP.

I am sure that there are more problems with it. BTW the problem stems from Windows having _no_ equivalent to fork().

Notefoot 1: Isn't it funny that Windows cannot delete files which are still opened, but does not care when it is renamed?

Ciao,

On 12/17/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

I recompiled git with NO_MMAP and used it successfully, then I added a note in the wiki page http://git.or.cz/gitwiki/WindowsInstall on how to compile git to work on a FAT32 filesystem.

Now, two questions...

if you are sure that there are more problems with using MMAP, shouldn't NO_MMAP be commented better in the Makefile and in INSTALL, pointing out to use it always on cygwin ???

finally on cygwin there is a precompiled git package (v 1.4.4.1); however, I found if it was compiled *with* MMAP support... should I write to the cygwin-apps list asking for a different compilation (with NO_MMAP), or for the majority of the users the standard compilation is a better choice ???

bye

---

Hi,

On Mon, 18 Dec 2006, Stefano Spinucci wrote:

I argued for that a long time ago, but there were a couple "works-for-me"s and I just shut up.

It could also be the case that for most operations, git became safe over time. For example, my biggest problem was the fork()ed diff in git-log, which was substituted by a non-forking inbuilt diff.

Given that most Windows setups nowadays run on NTFS, I doubt that this is necessary. I use NO_MMAP myself on Windows, just in case, though.

Ciao, Dscho

Johannes Schindelin writes:
 > On Mon, 18 Dec 2006, Stefano Spinucci wrote:
 > > On 12/17/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
 > > > I am sure that there are more problems with it. BTW the problem stems from
 > > > Windows having _no_ equivalent to fork().
 > > 
 > > if you are sure that there are more problems with using MMAP, shouldn't 
 > > NO_MMAP be commented better in the Makefile and in INSTALL, pointing out 
 > > to use it always on cygwin ???
 > 
 > I argued for that a long time ago, but there were a couple "works-for-me"s 
 > and I just shut up.

It would be nice to have test cases documenting known failures. Then people could decide whether these cases are important for their work, since I don't think it's actually possible to loose data this way.

Personally I've been unsuccessful in trying to recreate failures, but that is probably just an indication of my inability to use git correctly:-)

On 12/17/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:

I've used both of these guys, but not the most recent versions. Both were previously read-only, but now it appears both have write support from looking at the release notes. Both were a bit heavy handed to get setup, but a decently computer literate person could probably handle it. Ext2IFS in particular looks dramatically improved from when I last used it.

Both worked fine through Cygwin for simple usage. I never did anything with serious disk usage on either of them, so I can't say how it stacks up to FAT32 on Linux in terms of really hammering the disk.

A little more reading says ext2ifs can't handle permissions or symbolic links.

http://www.fs-driver.org/faq.html

I don't recall the details of ext2fsd, but I think symbolic links worked. I'd like to look at this again, but I spend all my time in Linux at home.

--

On Wed, 13 Dec 2006, Florian v. Savigny wrote:

there are drivers available for windows to let it access ext2/3 partitions. I don't know if that would help any in this situation.

David Lang