threads / patch / 15398

patchadd option to import series directly from a tar archive

Subject: [StGit PATCH] add option to import series directly from a tar archive

## tl;dr

10 messages between Sep 7, 2008 and Sep 12, 2008. Diffs are folded; open one to read it.

replies: 9people: 3as markdown or json

Clark Williams· Sep 7, 2008, 03:47 UTC · lore
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Catalin/Karl,

Attached is my first cut at adding the ability to import a patch series by specifying the tarball. For example, the following command:

	$ stg import --tarfile patch-2.6.26.3-rt6.bz2
will apply the latest -rt patch series to your current kernel tree.

No Karl, I haven't developed a test for it (yet). I wanted to see what you guys thought first :)

Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAkjDTscACgkQqA4JVb61b9dNRgCZAW+tOCgz5Y+A0IdomcOA4X7v u8MAnRvFWMXRJ0Kxv1rAnBRnheq6Iidi =W7Dl -----END PGP SIGNATURE-----

patch to allow importing a series from a tar archive
From: Clark Williams <williams@redhat.com>
Signed-off-by: Clark Williams <williams@redhat.com>
---
 stgit/commands/imprt.py |   46 +++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 45 insertions(+), 1 deletions(-)
Show changes to stgit/commands/imprt.py +45 −1
diff --git a/stgit/commands/imprt.py b/stgit/commands/imprt.py
index 227743f..0e9bb73 100644
--- a/stgit/commands/imprt.py
+++ b/stgit/commands/imprt.py
@@ -87,7 +87,10 @@ options = [make_option('-m', '--mail',
            make_option('--commname',
                        help = 'use COMMNAME as the committer name'),
            make_option('--commemail',
-                       help = 'use COMMEMAIL as the committer e-mail')
+                       help = 'use COMMEMAIL as the committer e-mail'),
+           make_option('--tarfile',
+                       help = 'import a series from a tar archive',
+                       action = "store_true"),
            ] + make_sign_options()
 
 
@@ -287,6 +290,45 @@ def __import_url(url, options):
     urllib.urlretrieve(url, filename)
     __import_file(filename, options)
 
+def __import_tarfile(tar, options):
+    """Import patch series from a tar archive
+    """
+    import tarfile
+    import tempfile
+
+    if not tarfile.is_tarfile(tar):
+        raise CmdException, "%s is not a tarfile!" % tar
+
+
+    t = tarfile.open(tar, 'r')
+    names = t.getnames()
+
+    # verify paths in the tarfile are safe
+    for n in names:
+        if n.startswith('/'):
+            raise CmdException, "Absolute path found in %s" % tar
+        if n.startswith("../"):
+            raise CmdException, "Relative path found in %s" % tar
+
+    # find the series file
+    seriesfile = '';
+    for m in names:
+        if m.endswith('/series') or m == 'series':
+            seriesfile = m
+            break
+    if seriesfile == '':
+        raise CmdException, "no series file found in %s" % tar
+
+    # unpack into a tmp dir
+    tmpdir = tempfile.mkdtemp('.stg')
+    t.extractall(tmpdir)
+
+    # apply the series
+    __import_series(os.path.join(tmpdir, seriesfile), options)
+
+    # cleanup the tmpdir
+    os.system('rm -rf %s' % tmpdir)
+
 def func(parser, options, args):
     """Import a GNU diff file as a new patch
     """
@@ -308,6 +350,8 @@ def func(parser, options, args):
         __import_mbox(filename, options)
     elif options.url:
         __import_url(filename, options)
+    elif options.tarfile:
+        __import_tarfile(filename, options)
     else:
         __import_file(filename, options)
Karl Hasselström· Sep 8, 2008, 18:03 UTC · re: Clark Williams · lore

Re: [StGit PATCH] add option to import series directly from a tar archive

On 2008-09-06 22:47:19 -0500, Clark Williams wrote:
> Attached is my first cut at adding the ability to import a patch
> series by specifying the tarball.
Thanks!
> No Karl, I haven't developed a test for it (yet). I wanted to see
> what you guys thought first :)

I don't see a problem with it, and if you took the time to code it there is obviously at least one user (I have no idea how common patch series tarballs are). I do have some comments below, but nothing that would prevent you from writing a test or two right away. ;-)

> +           make_option('--tarfile',
> +                       help = 'import a series from a tar archive',
> +                       action = "store_true"),

As I hint below, you might want to autodetect tarballs with --series instead, since a tarball is just a tarred series directory.

> +        if n.startswith("../"):
> +            raise CmdException, "Relative path found in %s" % tar

I guess any occurrence of /../ in the middle of n should be caught as well? Or can't that happen?

By the way, is the separator always '/' in tarfile? Or should you use os.sep? (There is also os.pardir which you could use instead of '..', but that might be overdoing it a little ...)

> +        raise CmdException, "no series file found in %s" % tar

Perhaps "no 'series' file ...", to make it clear what the name should be?

Show 6 quoted lines
> +    # unpack into a tmp dir
> +    tmpdir = tempfile.mkdtemp('.stg')
> +    t.extractall(tmpdir)
> +
> +    # apply the series
> +    __import_series(os.path.join(tmpdir, seriesfile), options)

Hmm. It seems like such a waste to go via the file system here, when tarfile has such nice file extraction methods.

What you could do is something like this:
  1. Make two small classes with the same interface, one backed by a
     tarfile and one backed by a directory, that have two methods:
     get_series() and get_file(filename). Both methods return
     file-like objects (created by either open() or
     tarfile.extractfile()).
  2. Change __import_series() to use objects of this class rather than
     a directory directly -- starting with creating an instance of one
     or the other depending on tarfile.is_tarfile(). This will involve
     teaching __import_file to accept a file-like object instead of
     just a file name, but that's a one-liner.
  3. Drop the --tarfile flag, since you've just taught the --series
     flag to handle tarballs!

That said, if you don't feel like doing it the hard way, I won't insist. The way you coded it is in no way bad (in particular, you chose the right function to create a temp dir).

> +    # cleanup the tmpdir
> +    os.system('rm -rf %s' % tmpdir)
Aaah! My eyes! My _eyes_!!!!!

Seriously, though, you'd want to use something like shutil.rmtree here.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle
Clark Williams· Sep 8, 2008, 18:11 UTC · re: Karl Hasselström · lore

Re: [StGit PATCH] add option to import series directly from a tar archive

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Karl Hasselström wrote:
Show 15 quoted lines
> On 2008-09-06 22:47:19 -0500, Clark Williams wrote:
> 
>> Attached is my first cut at adding the ability to import a patch
>> series by specifying the tarball.
> 
> Thanks!
> 
>> No Karl, I haven't developed a test for it (yet). I wanted to see
>> what you guys thought first :)
> 
> I don't see a problem with it, and if you took the time to code it
> there is obviously at least one user (I have no idea how common patch
> series tarballs are). I do have some comments below, but nothing that
> would prevent you from writing a test or two right away. ;-)
>

Patch series tarballs are quite common from people who use quilt (e.g. many of the kernel -rt series developers). My biggest problem (now that I can directly import them) is to see if I can ease StGit's patch import rules a bit, since quilt accepts pretty much anything as long as there's a diff in there somewhere. I bomb out regularly importing the -rt series using StGit, because some people don't put complete email addresses in their patches.

As to the test, I'll get right on that...:)
Show 6 quoted lines
>> +           make_option('--tarfile',
>> +                       help = 'import a series from a tar archive',
>> +                       action = "store_true"),
> 
> As I hint below, you might want to autodetect tarballs with --series
> instead, since a tarball is just a tarred series directory.

Yeah I thought about that, as well as auto-detecting it in the file case. I'll look into that a bit more.

Show 7 quoted lines
> 
>> +        if n.startswith("../"):
>> +            raise CmdException, "Relative path found in %s" % tar
> 
> I guess any occurrence of /../ in the middle of n should be caught as
> well? Or can't that happen?
>

Hence the "would you guys look at this?". Yeah, I need to detect sneaky stuff like that too.

> By the way, is the separator always '/' in tarfile? Or should you use
> os.sep? (There is also os.pardir which you could use instead of '..',
> but that might be overdoing it a little ...)

I doubt there are many Windows-generated tarballs out there (except for the Cygwin case; I believe they use '/'), but I shouldn't be so Unix-centric. I'll work on cleaning it up.

I did consider adding Zipfile support as well, but didn't get a very good match-up between tar functionality and zip functionality. Maybe later...

Show 6 quoted lines
> 
>> +        raise CmdException, "no series file found in %s" % tar
> 
> Perhaps "no 'series' file ...", to make it clear what the name should
> be?
> 
Yeah, that makes sense.
Show 30 quoted lines
>> +    # unpack into a tmp dir
>> +    tmpdir = tempfile.mkdtemp('.stg')
>> +    t.extractall(tmpdir)
>> +
>> +    # apply the series
>> +    __import_series(os.path.join(tmpdir, seriesfile), options)
> 
> Hmm. It seems like such a waste to go via the file system here, when
> tarfile has such nice file extraction methods.
> 
> What you could do is something like this:
> 
>   1. Make two small classes with the same interface, one backed by a
>      tarfile and one backed by a directory, that have two methods:
>      get_series() and get_file(filename). Both methods return
>      file-like objects (created by either open() or
>      tarfile.extractfile()).
> 
>   2. Change __import_series() to use objects of this class rather than
>      a directory directly -- starting with creating an instance of one
>      or the other depending on tarfile.is_tarfile(). This will involve
>      teaching __import_file to accept a file-like object instead of
>      just a file name, but that's a one-liner.
> 
>   3. Drop the --tarfile flag, since you've just taught the --series
>      flag to handle tarballs!
> 
> That said, if you don't feel like doing it the hard way, I won't
> insist. The way you coded it is in no way bad (in particular, you
> chose the right function to create a temp dir).
I did consider pulling directly from the tarball. I'll look into it.
Show 9 quoted lines
> 
>> +    # cleanup the tmpdir
>> +    os.system('rm -rf %s' % tmpdir)
> 
> Aaah! My eyes! My _eyes_!!!!!
> 
> Seriously, though, you'd want to use something like shutil.rmtree
> here.
>

Man, I could not for the life of me remember which module had that in it. To be fair I wasn't up at work with my Python Essential Reference, which would have pointed me directly at it, but I would have thought I could have gotten there through the Python docs. Sigh...

You can dock my StGit pay for the visit to the eye doctor :)
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAkjFatgACgkQqA4JVb61b9fMRQCeLfK0zhPNEq3t5M4HW+vbRtaG VhgAn0rtszqVLbd1bz12MS0b/3r0OkT2 =gkf1 -----END PGP SIGNATURE-----

Karl Hasselström· Sep 8, 2008, 21:22 UTC · re: Clark Williams · lore

Re: [StGit PATCH] add option to import series directly from a tar archive

On 2008-09-08 13:11:37 -0500, Clark Williams wrote:
> Patch series tarballs are quite common from people who use quilt
> (e.g. many of the kernel -rt series developers).
Ah.
Show 5 quoted lines
> My biggest problem (now that I can directly import them) is to see
> if I can ease StGit's patch import rules a bit, since quilt accepts
> pretty much anything as long as there's a diff in there somewhere. I
> bomb out regularly importing the -rt series using StGit, because
> some people don't put complete email addresses in their patches.
Yes, that would be a welcome addition.
> As to the test, I'll get right on that...:)
Swell!
Show 9 quoted lines
> Karl Hasselström wrote:
>
> > By the way, is the separator always '/' in tarfile? Or should you
> > use os.sep? (There is also os.pardir which you could use instead
> > of '..', but that might be overdoing it a little ...)
>
> I doubt there are many Windows-generated tarballs out there (except
> for the Cygwin case; I believe they use '/'), but I shouldn't be so
> Unix-centric. I'll work on cleaning it up.
Well, it's no big deal, really. Just thought I'd mention it.
> I did consider adding Zipfile support as well, but didn't get a very
> good match-up between tar functionality and zip functionality. Maybe
> later...

I had a quick look at the zipfile module, and it looks like it too could easily be wrapped in a small class like I suggested in point (1).

> I did consider pulling directly from the tarball. I'll look into it.

Just don't let my suggestions take all the fun out of contributing ... only do it my way if you really think it's better.

Show 17 quoted lines
> > On 2008-09-06 22:47:19 -0500, Clark Williams wrote:
> >
> > > +    # cleanup the tmpdir
> > > +    os.system('rm -rf %s' % tmpdir)
> >
> > Aaah! My eyes! My _eyes_!!!!!
> >
> > Seriously, though, you'd want to use something like shutil.rmtree
> > here.
>
> Man, I could not for the life of me remember which module had that
> in it. To be fair I wasn't up at work with my Python Essential
> Reference, which would have pointed me directly at it, but I would
> have thought I could have gotten there through the Python docs.
> Sigh...
>
> You can dock my StGit pay for the visit to the eye doctor :)
:-)
-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle
Samuel Tardieu· Sep 12, 2008, 12:21 UTC · re: Clark Williams · lore

Re: [StGit PATCH] add option to import series directly from a tar archive

>>>>> "Clark" == Clark Williams <clark.williams@gmail.com> writes:

Clark> [...] is to see if I can ease StGit's patch Clark> import rules a bit, since quilt accepts pretty much anything as Clark> long as there's a diff in there somewhere. I bomb out regularly Clark> importing the -rt series using StGit, because some people don't Clark> put complete email addresses in their patches.

Two things that would be great would be:
  - to be able to import patches with "-p0" (people not using git
    often sends such patches)
  - to be able to find where the patch should be applied; I sometimes
    receive patches for GCC directory "gcc/ada/", diffed from there,
    and if StGit could see that the patch only makes sense there and
    not at the top-level it would be great as well
 Sam
-- 
Samuel Tardieu -- sam@rfc1149.net -- http://www.rfc1149.net/
Clark Williams· Sep 12, 2008, 12:57 UTC · re: Samuel Tardieu · lore

Re: [StGit PATCH] add option to import series directly from a tar archive

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Samuel Tardieu wrote:
Show 12 quoted lines
>>>>>> "Clark" == Clark Williams <clark.williams@gmail.com> writes:
> 
> Clark> [...] is to see if I can ease StGit's patch
> Clark> import rules a bit, since quilt accepts pretty much anything as
> Clark> long as there's a diff in there somewhere. I bomb out regularly
> Clark> importing the -rt series using StGit, because some people don't
> Clark> put complete email addresses in their patches.
> 
> Two things that would be great would be:
> 
>   - to be able to import patches with "-p0" (people not using git
>     often sends such patches)

I'm not sure how easy this is going to be. It looks like the patch is applied with 'git --apply' from the file stgit/git.py:apply_patch(). The default '-p' value is 1, so we'd have to figure out how to pass the 0 along and then get it into the apply_patch() function.

Show 6 quoted lines
> 
>   - to be able to find where the patch should be applied; I sometimes
>     receive patches for GCC directory "gcc/ada/", diffed from there,
>     and if StGit could see that the patch only makes sense there and
>     not at the top-level it would be great as well
>

Zowie, I thought I only had to worry about folks sending patches with incomplete information. So you get patches to the ada compiler that are rooted in gcc/ada (e.g. patch in tarball says "./ChangeLog", instead of gcc/ada/ChangeLog) rather than at a top level? Only way I could see to deal with that would be to try and pass in the appropriate prefix from the command line.

My current plans are to clean up the first cut at the tarfile logic, then write a test to keep Karl happy, then try to come up with a way to deal with importing patches that don't have complete email addresses, no descriptions, etc. Once I get through that, I'll see if we can deal with weirdly rooted patch series.

Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAkjKZ0sACgkQqA4JVb61b9eg2ACffDv+FXsL1NifMvxr1tbO2c3s Hc4AoJPb/RZJrpqT6QybeZrj8rNFJg1y =ccj/ -----END PGP SIGNATURE-----

Samuel Tardieu· Sep 12, 2008, 13:59 UTC · re: Clark Williams · lore

Re: [StGit PATCH] add option to import series directly from a tar archive

>>>>> "Clark" == Clark Williams <clark.williams@gmail.com> writes:

Clark> Zowie, I thought I only had to worry about folks sending Clark> patches with incomplete information. So you get patches to the Clark> ada compiler that are rooted in gcc/ada (e.g. patch in tarball Clark> says "./ChangeLog", instead of gcc/ada/ChangeLog) rather than Clark> at a top level? Only way I could see to deal with that would be Clark> to try and pass in the appropriate prefix from the command Clark> line.

Yes, passing the prefix and strip levels would be fine.
  Sam
-- 
Samuel Tardieu -- sam@rfc1149.net -- http://www.rfc1149.net/
Karl Hasselström· Sep 12, 2008, 15:44 UTC · re: Clark Williams · lore

Re: [StGit PATCH] add option to import series directly from a tar archive

On 2008-09-12 07:57:47 -0500, Clark Williams wrote:
> then write a test to keep Karl happy,

Technically, you write the test to make sure that your new feature works as intended and won't break in the future. But since that's rather a mouthful, I guess "Karl" will do as an acronym. ;-)

> then try to come up with a way to deal with importing patches that
> don't have complete email addresses, no descriptions, etc. Once I
> get through that, I'll see if we can deal with weirdly rooted patch
> series.
Nice.
-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle
Karl Hasselström· Sep 12, 2008, 13:07 UTC · re: Samuel Tardieu · lore

Re: [StGit PATCH] add option to import series directly from a tar archive

On 2008-09-12 14:21:13 +0200, Samuel Tardieu wrote:
> Two things that would be great would be:
>
>   - to be able to import patches with "-p0" (people not using git
>     often sends such patches)

This should be trivial to implement, since git-apply (pardon the dash) has a -p flag with precisely this meaning.

>   - to be able to find where the patch should be applied; I
>     sometimes receive patches for GCC directory "gcc/ada/", diffed
>     from there, and if StGit could see that the patch only makes
>     sense there and not at the top-level it would be great as well

I don't believe git-apply can do this (please correct me if I'm wrong), and the right way to teach StGit to do it would arguably be to teach it to git-apply and then make StGit use it. It'd be _possible_ to do it directly in StGit, but it wouldn't be quite the right level, and git users wouldn't benefit.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle
Karl Hasselström· Sep 12, 2008, 13:08 UTC · re: Karl Hasselström · lore

Re: [StGit PATCH] add option to import series directly from a tar archive

On 2008-09-12 15:07:03 +0200, Karl Hasselström wrote:
Show 9 quoted lines
> On 2008-09-12 14:21:13 +0200, Samuel Tardieu wrote:
>
> >   - to be able to find where the patch should be applied; I
> >     sometimes receive patches for GCC directory "gcc/ada/", diffed
> >     from there, and if StGit could see that the patch only makes
> >     sense there and not at the top-level it would be great as well
>
> I don't believe git-apply can do this (please correct me if I'm
> wrong)

It does have a --directory flag, but that requires the user to specify the path manually.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

← back to recent threads