how about this as a basis for git-annotate?

#!/bin/bash

file="$1" anno="$file.anno" prev="" export GIT_DIFF_OPTS=-u0

rm $anno touch $anno

revs=`git-rev-list HEAD | tac`

for rev in $revs; do
	if [ ! -z "$prev" ]; then
		git-diff-tree -p $prev $rev $file \
		| sed 	-e "/^--- /p" \
			-e "/^+++ /p" \
			-e "/^--- /d" \
			-e "/^+++ /d" \
			-e "s/^-/-$prev /g" \
			-e "s/^+/+$rev /g" | patch $anno
	fi
	prev=$rev
done

* gives a warning when the diff is empty
* has a problem with "\ No newline at end of file"

What I tried to do is to reconstruct the file from the diffs, and prefixing each line with the sha1 of the revision.

After further testing, I see that the implementation I provided does not work, because the context is not necessarily from the previous version, but from /a/ previous version. Still the idea should work. So patch needs to ignore the sha1 of the revision. Maybe git-apply can be modified to do this...?