threads / discuss / 20293

Add Gitweb support for LZMA compressed snapshots

Subject: Add Gitweb support for LZMA compressed snapshots

## tl;dr

17 messages between Jul 30, 2009 and Aug 1, 2009.

replies: 16people: 10as markdown or json

Mark A Rada· Jul 30, 2009, 05:48 UTC · lore
Hi,

I thought I would submit this little patch I made to my gitweb. I am on a relatively slow connection, and so LZMA compression time is less of a concern than bandwidth---I'm guessing that I am not the only person who suffers from slow internet connection syndrome.

-- Mark A Rada (ferrous26) marada@uwaterloo.ca

From a080ce5e8467317a3f20c6de8814549e5c4f46f2 Mon Sep 17 00:00:00 2001
From: Mark Rada <marada@uwaterloo.ca>
Date: Wed, 29 Jul 2009 21:27:32 -0400
Subject: [PATCH] Add Gitweb support for LZMA compressed snapshots
Signed-off-by: Mark Rada <marada@uwaterloo.ca>
---
 gitweb/gitweb.perl |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 38492d0..e131ea4 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -184,6 +184,13 @@ our %known_snapshot_formats = (
 		'format' => 'tar',
 		'compressor' => ['bzip2']},
 
+	'txz' => {
+		'display' => 'tar.lzma',
+		'type' => 'application/x-lzma',
+		'suffix' => '.lzma',
+		'format' => 'tar',
+		'compressor' => ['lzma']},
+
 	'zip' => {
 		'display' => 'zip',
 		'type' => 'application/x-zip',
@@ -196,6 +203,7 @@ our %known_snapshot_formats = (
 our %known_snapshot_format_aliases = (
 	'gzip'  => 'tgz',
 	'bzip2' => 'tbz2',
+	'lzma'  => 'txz',
 
 	# backward compatibility: legacy gitweb config support
 	'x-gzip' => undef, 'gz' => undef,

-- 
1.6.4
J.H.· Jul 30, 2009, 07:44 UTC · re: Mark A Rada · lore

Re: Add Gitweb support for LZMA compressed snapshots

Mark A Rada wrote:
Show 14 quoted lines
> Hi,
> 
> I thought I would submit this little patch I made to my gitweb. I am on 
> a relatively slow
> connection, and so LZMA compression time is less of a  concern than 
> bandwidth---I'm
> guessing that I am not the only person who suffers from slow internet 
> connection
> syndrome.
> 
> 
> -- 
> Mark A Rada (ferrous26)
> marada@uwaterloo.ca

Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it uses the lzma2 format which is, by far, preferable to what 'lzma' outputs. Same compression (lzma) just different file format.

- John 'Warthog9' Hawley
Alex Riesen· Aug 1, 2009, 07:43 UTC · re: J.H. · lore

Re: Add Gitweb support for LZMA compressed snapshots

On Thu, Jul 30, 2009 at 09:44, J.H.<warthog19@eaglescrag.net> wrote:
> Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it uses
The host seems to be down.
> the lzma2 format which is, by far, preferable to what 'lzma' outputs.  Same
> compression (lzma) just different file format.

And the whole project seems to be very young (listen on ohloh since 2007). And it is not in current stable distributions. Exotic thing.

Dmitry Potapov· Aug 1, 2009, 14:34 UTC · re: Alex Riesen · lore

Re: Add Gitweb support for LZMA compressed snapshots

On Sat, Aug 01, 2009 at 09:43:39AM +0200, Alex Riesen wrote:
> 
> And the whole project seems to be very young (listen on ohloh since 2007).
> And it is not in current stable distributions. Exotic thing.

The project seems to be it is developed together with Igor Pavlov, who is the author of 7-Zip (and now 7-Zip also supports LZMA2 and XZ format). Accordingly to Igor Pavlov:

   LZMA2 provides the following advantages over LZMA:
   1) Better compression ratio for data than can't be compressed.
   It can store such blocks of data in uncompressed form.
   Also it decompresses such data faster.
   2) Better multithreading support. If you compress big file, LZMA2 can split
   that file to chunks and compress these chunks in multiple threads.

Clearly XZ is not very popular yet, but I would not call it as exotic. At least, I expect it to be supported by future distributives.

Dmitry
André Goddard Rosa· Aug 1, 2009, 14:38 UTC · re: Alex Riesen · lore

Re: Add Gitweb support for LZMA compressed snapshots

On Sat, Aug 1, 2009 at 4:43 AM, Alex Riesen<raa.lkml@gmail.com> wrote:
Show 5 quoted lines
> On Thu, Jul 30, 2009 at 09:44, J.H.<warthog19@eaglescrag.net> wrote:
>> Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it uses
>
> The host seems to be down.
>
You can clone it here: git://ctrl.tukaani.org/xz.git

Best regards, Andre

Jim Meyering· Aug 1, 2009, 14:58 UTC · re: Alex Riesen · lore

Re: Add Gitweb support for LZMA compressed snapshots

Alex Riesen wrote:
Show 10 quoted lines
> On Thu, Jul 30, 2009 at 09:44, J.H.<warthog19@eaglescrag.net> wrote:
>> Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it uses
>
> The host seems to be down.
>
>> the lzma2 format which is, by far, preferable to what 'lzma' outputs.  Same
>> compression (lzma) just different file format.
>
> And the whole project seems to be very young (listen on ohloh since 2007).
> And it is not in current stable distributions. Exotic thing.

XZ is young, but worthy, imho. It was added to Fedora 11 in the last few days.

Alex Riesen· Aug 1, 2009, 18:51 UTC · re: Jim Meyering · lore

Re: Add Gitweb support for LZMA compressed snapshots

On Sat, Aug 1, 2009 at 16:58, Jim Meyering<jim@meyering.net> wrote:
Show 5 quoted lines
> Alex Riesen wrote:
>> And it is not in current stable distributions. Exotic thing.
>
> XZ is young, but worthy, imho.
> It was added to Fedora 11 in the last few days.

Well, then maybe we should do by its good example and support xz. In addition to .lzma, .7z, .cpio, etc.

Johannes Schindelin· Jul 30, 2009, 07:52 UTC · re: Mark A Rada · lore

Re: Add Gitweb support for LZMA compressed snapshots

Hi,
On Thu, 30 Jul 2009, Mark A Rada wrote:
> I thought I would submit this little patch I made to my gitweb. I am on 
> a relatively slow connection, and so LZMA compression time is less of a 
> concern than bandwidth---I'm guessing that I am not the only person who 
> suffers from slow internet connection syndrome.
That's great!

FWIW on this mailing list, we prefer to have patches inlined so that it is easier for reviewers to comment (it is almost impossible to get any patch into git.git without review, and because reviewing is such an ungratifying job you want to make it easier).

The patch is pretty straight-forward, but I'd love to see some information in the commit message about:

- performance numbers on the _server_ (i.e. how does the RAM and CPU load 
  compare to, say, bzip2?)
- how to disable/enable it (yes, it's in the documentation, but you can 
  make things easier still)
- whether it is turned on by default, and why.

Thanks, Dscho

Jakub Narebski· Jul 30, 2009, 08:31 UTC · re: Mark A Rada · lore

Re: Add Gitweb support for LZMA compressed snapshots

Mark A Rada <marada@uwaterloo.ca> writes:
> I thought I would submit this little patch I made to my gitweb. I am
> on a relatively slow connection, and so LZMA compression time is
> less of a concern than bandwidth---I'm guessing that I am not the
> only person who suffers from slow internet connection syndrome.

First, Documentation/SubmittingPatches states that we prefer inline patches, with additional comments (like the one above) either between "---\n" line and diffstat, or like I did here before patch with patch separated by comment by e.g. "-- >8 --\n" (scissors) line.

If posting patch inline isn't possible, and you have to use attachement (for example because mailer you use wraps lines), please at least use 'text/plain' mimetype so patch can be viewed without need to save it in separate file (this might require changing suffix from '.patch' to '.txt').

Second, if you are using 'txz' as internal name for new snapshot format, why not use 'XZ utils' instead of 'LZMA Utils', and use 'display' => 'txz', 'type' => 'application/x-xz' and 'suffix' => '.txz' (BTW. shouldn't suffix in your case be '.tar.lzma' and not only '.lzma'?) and of course 'compressor' => ['xz']?

BTW. I wonder if it would be good idea to add support for this format directly to git-archive... OTOH it would mean additional dependency.

-- 
Jakub Narebski
Git User's Survey 2009: http://tinyurl.com/GitSurvey2009

-- >8 --
From: Mark Rada <marada@uwaterloo.ca>
Subject: [PATCH] Add Gitweb support for LZMA compressed snapshots

Signed-off-by: Mark Rada <marada@uwaterloo.ca>
---
 gitweb/gitweb.perl |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 38492d0..e131ea4 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -184,6 +184,13 @@ our %known_snapshot_formats = (
 		'format' => 'tar',
 		'compressor' => ['bzip2']},
 
+	'txz' => {
+		'display' => 'tar.lzma',
+		'type' => 'application/x-lzma',
+		'suffix' => '.lzma',
+		'format' => 'tar',
+		'compressor' => ['lzma']},
+
 	'zip' => {
 		'display' => 'zip',
 		'type' => 'application/x-zip',
@@ -196,6 +203,7 @@ our %known_snapshot_formats = (
 our %known_snapshot_format_aliases = (
 	'gzip'  => 'tgz',
 	'bzip2' => 'tbz2',
+	'lzma'  => 'txz',
 
 	# backward compatibility: legacy gitweb config support
 	'x-gzip' => undef, 'gz' => undef,
-- 
1.6.4
Johannes Schindelin· Jul 30, 2009, 08:43 UTC · re: Jakub Narebski · lore

Re: Add Gitweb support for LZMA compressed snapshots

Hi,
On Thu, 30 Jul 2009, Jakub Narebski wrote:
> BTW. I wonder if it would be good idea to add support for this format
> directly to git-archive...  OTOH it would mean additional dependency.

I don't think it would be a good idea; we do not have bzip2 support either.

The only reason we have inbuilt gzip and zip support is because the format is so similar to Git's own compression.

Ciao, Dscho

Felipe Contreras· Jul 31, 2009, 15:45 UTC · re: Johannes Schindelin · lore

Re: Add Gitweb support for LZMA compressed snapshots

On Thu, Jul 30, 2009 at 8:43 AM, Johannes Schindelin<Johannes.Schindelin@gmx.de> wrote:

Show 9 quoted lines
> Hi,
>
> On Thu, 30 Jul 2009, Jakub Narebski wrote:
>
>> BTW. I wonder if it would be good idea to add support for this format
>> directly to git-archive...  OTOH it would mean additional dependency.
>
> I don't think it would be a good idea; we do not have bzip2 support
> either.

bzip2 has no advantages whatsoever. AFAIK xz is superior to other formats and it would be nice to see git make a technological stance encouraging xz.

> The only reason we have inbuilt gzip and zip support is because the format
> is so similar to Git's own compression.
Personally I don't see the point of having zip support.
-- 
Felipe Contreras
Johannes Schindelin· Aug 1, 2009, 12:34 UTC · re: Felipe Contreras · lore

Re: Add Gitweb support for LZMA compressed snapshots

Hi,
On Fri, 31 Jul 2009, Felipe Contreras wrote:
Show 13 quoted lines
> On Thu, Jul 30, 2009 at 8:43 AM, Johannes
> Schindelin<Johannes.Schindelin@gmx.de> wrote:
> > Hi,
> >
> > On Thu, 30 Jul 2009, Jakub Narebski wrote:
> >
> >> BTW. I wonder if it would be good idea to add support for this format
> >> directly to git-archive...  OTOH it would mean additional dependency.
> >
> > I don't think it would be a good idea; we do not have bzip2 support
> > either.
> 
> bzip2 has no advantages whatsoever.

Bzzzzt. Wrong. Just because you cannot see them does not mean it has no advantages.

> AFAIK xz is superior to other formats and it would be nice to see git 
> make a technological stance encouraging xz.

Bzzzt. Wrong again. Git's mission in life is not to encourage one compression over another.

If at all, the only compression Git actually does promote in a sense is zlib compression.

> > The only reason we have inbuilt gzip and zip support is because the 
> > format is so similar to Git's own compression.
> 
> Personally I don't see the point of having zip support.

Personally, I see the point of having zip support. It makes things easy for Windows users. And it's an established format, much more so than tar.gz.

Ciao, Dscho

Felipe Contreras· Aug 1, 2009, 13:10 UTC · re: Johannes Schindelin · lore

Re: Add Gitweb support for LZMA compressed snapshots

On Sat, Aug 1, 2009 at 12:34 PM, Johannes Schindelin<Johannes.Schindelin@gmx.de> wrote:

Show 20 quoted lines
> Hi,
>
> On Fri, 31 Jul 2009, Felipe Contreras wrote:
>
>> On Thu, Jul 30, 2009 at 8:43 AM, Johannes
>> Schindelin<Johannes.Schindelin@gmx.de> wrote:
>> > Hi,
>> >
>> > On Thu, 30 Jul 2009, Jakub Narebski wrote:
>> >
>> >> BTW. I wonder if it would be good idea to add support for this format
>> >> directly to git-archive...  OTOH it would mean additional dependency.
>> >
>> > I don't think it would be a good idea; we do not have bzip2 support
>> > either.
>>
>> bzip2 has no advantages whatsoever.
>
> Bzzzzt.  Wrong.  Just because you cannot see them does not mean it has no
> advantages.

All right, most the time you compress once, and multiple people uncompress multiple times. Therefore the path that should be optimized is the decompression, that leaves bzip2 out of the picture.

You can think of advantages of bzip2 in more esoteric cases where the compression time is also important. I don't think that's the case here. Actually it might be if gitweb is not caching the tarballs, but if that's the case I wouldn't say it's an advantage of bzip2.

Show 5 quoted lines
>> AFAIK xz is superior to other formats and it would be nice to see git
>> make a technological stance encouraging xz.
>
> Bzzzt. Wrong again.  Git's mission in life is not to encourage one
> compression over another.

Git's mission in life is not promoting asciidot either, but it's doing that unintentionally by merely using it.

Show 11 quoted lines
> If at all, the only compression Git actually does promote in a sense is
> zlib compression.
>
>> > The only reason we have inbuilt gzip and zip support is because the
>> > format is so similar to Git's own compression.
>>
>> Personally I don't see the point of having zip support.
>
> Personally, I see the point of having zip support.  It makes things easy
> for Windows users.  And it's an established format, much more so than
> tar.gz.
Windows can't extract .tar.gz?
-- 
Felipe Contreras
Erik Faye-Lund· Aug 1, 2009, 14:04 UTC · re: Felipe Contreras · lore

Re: Add Gitweb support for LZMA compressed snapshots

On Sat, Aug 1, 2009 at 3:10 PM, Felipe Contreras<felipe.contreras@gmail.com> wrote:

> Windows can't extract .tar.gz?

Not without installing 3. party software. Zip-support is built into the operating system.

-- 
Erik "kusma" Faye-Lund
kusmabite@gmail.com
(+47) 986 59 656
Mark A Rada· Aug 1, 2009, 16:07 UTC · re: Erik Faye-Lund · lore

Re: Add Gitweb support for LZMA compressed snapshots

On 1-Aug-09, at 10:04 AM, Erik Faye-Lund wrote:
Show 15 quoted lines
> On Sat, Aug 1, 2009 at 3:10 PM, Felipe
> Contreras<felipe.contreras@gmail.com> wrote:
>> Windows can't extract .tar.gz?
>
> Not without installing 3. party software. Zip-support is built into
> the operating system.
>
> -- 
> Erik "kusma" Faye-Lund
> kusmabite@gmail.com
> (+47) 986 59 656
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

FWIW, Windows doesn't seem to come with a whole lot, and you need 3rd party software for a number of things, BUT that is also one of the primary reasons that I need to be able to quickly get snapshots from my Git repositories at home, and using zip format is convenient if I am in an extremely restrictive environment and cannot get a smaller XZ compressed snapshot :).

-- Mark A Rada (ferrous26) marada@uwaterloo.ca

Erik Faye-Lund· Aug 1, 2009, 21:39 UTC · re: Mark A Rada · lore

Re: Add Gitweb support for LZMA compressed snapshots

On Sat, Aug 1, 2009 at 6:07 PM, Mark A Rada<marada@uwaterloo.ca> wrote:
> FWIW, Windows doesn't seem to come with a whole lot, and you need
> 3rd party software for a number of things
Tell that to the US Department of Justice ;)
-- 
Erik "kusma" Faye-Lund
kusmabite@gmail.com
(+47) 986 59 656
Dmitry Potapov· Aug 1, 2009, 14:13 UTC · re: Felipe Contreras · lore

Re: Add Gitweb support for LZMA compressed snapshots

On Fri, Jul 31, 2009 at 03:45:02PM +0000, Felipe Contreras wrote:
> 
> bzip2 has no advantages whatsoever. AFAIK xz is superior to other
> formats and it would be nice to see git make a technological stance
> encouraging xz.

git is not about encouraging any compression. If you have not noticed, git-archive produces tar and not tgz (though it would not be difficult to implement it, because we use zlib anyway). If you like xz and it is installed on your system, you can always compress tar with it:

git archive HEAD | xz > my-archive.tar.xz
So, I see no reason to have this compression coupled with git.
Show 5 quoted lines
> 
> > The only reason we have inbuilt gzip and zip support is because the format
> > is so similar to Git's own compression.
> 
> Personally I don't see the point of having zip support.

zip is still a popular format on Windows. If git archive was not able to produce then creating would require creating tar first, extracting it to some temporary directory and then creating a zip file. It is not very efficient process and requires extra efforts, so because we use zlib compression anyway, it makes perfect sense that we can create zip file directly.

Dmitry

← back to recent threads