Question about possible git races

I'd like to write a multithreaded application using git, so I'd like to see if there are any races:

- push vs pull
One thread pushes to the repository while another is pulling from it at the 
same time. I should get the older commit.

- push vs push
Both threads push at the same time. What happens?
Any good way to merge those pushes?
(I have full access to both repos)

Possibly those two aren't fast-forward of each other. I think one of the pushes should abort in this case unless I force it.

- fetch vs fetch
I mean that two threads try to fetch from different repositories to a single 
one. Possibly those two aren't fast-forward of each other.
Any good way to merge those fetches?
(I have full access to both repos)

I'm meaning really bare git there, w/o bash+perl scripts.

On Monday 20 March 2006 17:24, Radoslaw Szkodzinski wrote yet:

Could anyone try to answer the question? I'd really like to know, because it's crucial to my application.

Radoslaw Szkodzinski wrote:

I believe the reasons no-one answered your first mail are, in the following order:

1. Since I'm sure you're truly capable of writing such an application 
and finishing it before the git API has changed completely you should 
have gotten your answers from trial-and-error, reading the source, or by 
just trying the app and seeing where it fails.

2. You didn't say *why* you want to write a multi-threaded layer on top 
of git. Is it to implement a redundant file-server with revision 
control? If so you don't need multi-threading. You need clever update 
hooks, a master repo and plenty of bandwidth with fast disks on the 
file-servers.

3. You didn't mention what you've tried to find the answers yourself, 
which makes me think you want me and the rest of us gitizens (yay! I 
coined a phrase) do your homework for you. I personally find it very 
rude that you send another email again so shortly after the first one 
claiming that "you need this info for your app", when the 500-odd people 
you're asking clearly need their time for their families, hobbies, 
daytime jobs, beer, etc. etc...

4. Noone felt like answering since they saw no use for a multi-threaded 
layer on top of git, especially without knowing what it was for. 
Friendliness only goes so far when met by such lack of respect for other 
peoples time, but if someone had seen the uses for the app you're 
writing they probably would have taken time to at least ask you for some 
of the answers you left out in your original mail. If nothing else for 
the sake of curiosity.

Some more pointers on how to get answers to questions posted in online forums can be found on the links below.

http://catb.org/~esr/faqs/smart-questions.html http://www.catb.org/~esr/faqs/hacker-howto.html

Btw. I'm assuming you're aware you'll have to GPL this app of yours, since git is GPL and you'll be using the git produce in a way that makes it vital to your app.

Radoslaw Szkodzinski wrote:

Why on earth? The git tools aren't written to be thread-safe in that manner, so you'll run into all sorts of problems. Unless you're talking about managing the code for a multithreaded application with git, in which case you should just go read the tutorial. However, feeling slightly tipsy and in a distinctly good mood, I shall try to answer your questions anyway.

You will. Git atomizes (atomicizes? atomicifies?) pushes by updating the branch head being pushed to after all the commit-, tree- and blob-objects are written. Tags are handled separately but equally atomically.

Read the source to find out if it's locking the repo while updating or not (I think it is, but I'm not sure). If it isn't the last one to finish pushing wins out since the branch head update from that push will overwrite the previous one.

git help octopus

You can fetch those two remote branch heads to local branches simultaneously and then do the octopus in the master-thread while no other updates are happening. Doing several simultanous merges to a single branch is quite frankly so insane I have to go get myself a drink just from thinking about it.

I don't think you can do it without Python. The default merge strategy is written in python, so.

On Thursday 23 March 2006 00:28, Andreas Ericsson wrote yet:

to answer your questions anyway.

The golden liquid works for me, great. Thank you.

Just as I expected. Good.

Good too.

I could slurp the unrelated commits with an octopus of course... But the others pose a problem.

It's a rare situation in the app, but not impossible. (I want to avoid locking) That's why I was asking about it.

You've hit it, the app is written in Python (as of yet).

The default merge strategy is simply... calling merge and also detecting naming/existence conflicts with a simple scalar merge.

The only bit more complicated thing is detecting the LCA for 3-way merge.

I'd like to build a decentralised collaborative web application, as scalable as it gets. (I expect a lot of traffic) I also need to only check out parts of the tree. (many SCMs can't do that) Git, with its speed, seems well-suited to the task. Merging will be uncommon in the workload, so can be slow, but shouldn't break or present weird conflicts to the users.

Unless the accidental case starts to dominate - then I'll have a problem.

Sorry for spam and cutting out major questions, then answering at the end of the post.

It will be, but not because of git (it's execve()ing it), but rather because of the principle.

Intermediate results will probably be:
- a good Python object interface to git
- another porcelain, portable, coded in Python
- maybe a new merge strategy
- later yet another porcelain, written in C

Radoslaw Szkodzinski wrote:

Ah, Good Thing. Just to clarify for the archives though, it *is* required since it's using git in a way that makes git a fundamental, non-replaceable part of its core operations.

yagit? yagp? What I wanna know is when jigit's gonna hit the streets. :)

On Thursday 23 March 2006 03:55, Andreas Ericsson wrote yet:

I need a memorable three-letter acronym, so it will be gip. (Git porcelain In Python) and the C one... later.

GIP also stands for Good Informatics Practices :P or "to swindle", but it's the deprecated usage, I think. It has some racial ties.

Conflicts with: GNOME IP calculator - http://www.debain.org/software/gip/

I'd get myself a better commandline IP calculator than give a good TLA to this. Gnip should be the name.

What's jigit? Did I miss something?

Radoslaw Szkodzinski <astralstorm@o2.pl> writes:

About push vs push, with "really bare git", I take it that you mean two send-pack from remote sites running two receive-pack simultaneously.

There is an explicit race avoidance between the receive-pack processes. When a ref (either branch head or a tag) is updated, it does:

 - read the current value from the ref.
 - do its work.
 - lock to prevent others to create the temporary file for
   updating the ref.
 - create the temporary file for the ref and write the new value.
 - check if the ref's value has not changed from what it
   initially read;
 - rename the temporary file to the ref to unlock.

Read receive-pack.c::update() for exact details if you are interested.

The question does not make any sense for other cases, because branch update by fetch and pull are all scripts based.

On Thursday 23 March 2006 01:24, Junio C Hamano wrote yet:

So there is locking I've missed while reading through the source. Guess all the coffee doesn't help.

There is a lock, so the other git-receive-pack for given ref will fail. Does that also work for git-local-fetch with -l option?

Looks good though that I can fetch to another ref.

I should have known better than to use vague words. For me fetch = git-*-fetch. Which in turn calls git-receive-pack.

Thank you for the precise answer.

Radoslaw Szkodzinski <astralstorm@o2.pl> writes:

Does anything other than git-send-pack call git-receive-pack?

For fetch, git-fetch-pack is called from the core level, but it does not update refs itself. It writes out enough information to its standard output so that the script calling it can update the refs. So at the core level there cannot be any race, but that does not necessarily mean existing scripts are race free.

Our barebone Porcelainish scripts _do_ use update-ref to do the same lock - re-read - rename-to-update cycle when updating the refs using that information, but that is something you explicitly said you are not interested in ;-).