{"thread":{"id":"13454","subject":"[egit / jgit] Implementation of a file tree iteration using ignore rules.","startedAt":"2008-05-09T13:20:08Z","lastAt":"2008-05-11T00:12:31Z","messageCount":4,"participants":["Florian Köberle","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"76455","messageId":"48244F88.8060109@web.de","threadId":"13454","inReplyTo":null,"subject":"[egit / jgit] Implementation of a file tree iteration using ignore rules.","fromName":"Florian Köberle","fromEmail":"florianskarten@web.de","sentAt":"2008-05-09T13:20:08Z","receivedAt":"2008-05-09T13:20:08Z","isPatch":false,"sender":{"key":"florianskarten@web.de","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nHi\n\nI like the idea of a Java implementation of git and would like to\ncontribute to the jgit/egit project.\n\nIn order to get familiar with the code I started to implement a command\nlike tool which works like git using the jgit library. I implemented\nvery simple versions of the commands \"help\", \"init\" and finally wanted\nto implement the \"add\" command. However, I didn't find any tools to\ndetermine the files which should be added.\n\nSo I implemented a factory which returns an Iterable<File> for the\niteration over all the files in a directory.\n\nFor an example see the unit test testRealisticExample() in the class\nFileIterableFactoryForAddCommandTest:\nhttp://repo.or.cz/w/egit/florian.git?a=blob;f=org.spearce.jgit.test/tst/org/spearce/jgit/lib/fileiteration/FileIterableFactoryForAddCommandTest.java;h=d3c78f4422c708f26ccb56434053bb711fa3116b;hb=669fd814d34e2f989b5f8eedbcb0d5bcf9743ce7\n\nYou can view the patches online at:\nhttp://repo.or.cz/w/egit/florian.git?a=shortlog;h=refs/heads/mailinglist-patches-0\n\nI signed all patches and formatted them with the code formatter as I\nshould. It's ok for me to put the patches under a dual license between a\n3-clause BSD and the EPL[*3*]. Currently all files have a GPL 2 notice.\nI hope that is ok.\n\nIf you want I will send the patches to the mailing list, but I don't\nknow any automated way to create all the emails. I am not even sure if I\nwill get them formatted correctly with Thunderbird 2. It would be cool\nif you could tell me how to send patches via command line.\n\nI hope you like my patches,\nFlorian Koeberle\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.4.6 (GNU/Linux)\nComment: Using GnuPG with Mozilla - http://enigmail.mozdev.org\n\niD8DBQFIJE+I59ca4mzhfxMRAqlwAKCSp57SkqvVsBpdt8o3jL6zNdn0kACfeLnZ\nIHErO96fu2rdQcT+JpmroYU=\n=E+vF\n-----END PGP SIGNATURE-----\n"},{"id":"76480","messageId":"20080510001132.GF29038@spearce.org","threadId":"13454","inReplyTo":"48244F88.8060109@web.de","subject":"Re: [egit / jgit] Implementation of a file tree iteration using ignore rules.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-05-10T00:11:32Z","receivedAt":"2008-05-10T00:11:32Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Florian Kberle <FloriansKarten@web.de> wrote:\n> \n> I like the idea of a Java implementation of git and would like to\n> contribute to the jgit/egit project.\n\nEven better, we'd love to have you contribute!  :-)\n \n> In order to get familiar with the code I started to implement a command\n> like tool which works like git using the jgit library. I implemented\n> very simple versions of the commands \"help\", \"init\"\n\nThis is an interesting start.  Did you see the existing \"Main\" class\nin org.spearce.jgit/src/org/spearce/jgit/pgm?  It sets up and invokes\na TextBuiltin, which is sort of like the \"Command\" class you added in\nyour first patch.  Though TextBuiltins are created on-the-fly and thus\nare harder/impossible to use to format a \"jgit help\".\n\nI think your approach of building up a table of commands is likely\nthe better one long term, so I am interested in seeing the two unify,\ntaking the best from each (from Main and MainProgram that is).\n\nPlease note that jgit is restricted to Java 5 APIs only right now.\nThe \"MainProgram\" class you introduced uses Arrays.copyOfRange()\nwhich does not compile under Java 5.  I guess it is new in Java 6?\n\n> and finally wanted\n> to implement the \"add\" command. However, I didn't find any tools to\n> determine the files which should be added.\n\nRight.  We haven't implemented this properly yet.  So I am very\nhappy to see someone starting to approach this.\n\n> So I implemented a factory which returns an Iterable<File> for the\n> iteration over all the files in a directory.\n\nSadly this is a reimplementation of the already existing FileTreeIterator,\nwhich is meant to be used within a TreeWalk instance.\n\nThe TreeWalk API is meant to iterate over a working directory in\ncanonical tree entry name ordering, so that we can walk not just\na working directory but also the index file and one or more tree\nobjects in parallel.  We can even walk multiple working directories\nat once making directory differencing fairly simple.\n\nWhat is missing here is really two things:\n\n #1)  Take .gitignore and .git/info/exclude (and other patterns) into\n      account as WorkingTreeIterator (base class of FileTreeIterator)\n      loops over the entries in a directory.\n\n      Since .gitignore can be per-directory we may need to add rules\n      as we enter into a subtree (createSubtreeIterator method)\n      and pop rules as we exit a subtree.\n\n      Fortunately the pop part is easy if the rules are held within\n      the WorkingTreeIterator as instance members as pop is already\n      dealt with up inside of TreeWalk by simply discarding the\n      subtree instance and returning back to the parent instance.\n\n  #2) Be able to edit one (or more) index files during a\n      TreeWalk.  I am (sort of) in the middle of that work on my\n      egit/spearce.git fork's dircache branch.\n\nWe already have a method for filtering entries during TreeWalk; its\nthe TreeFilter API and its many subclasses.  I wonder if the rules\nfor say .gitignore could simply be implemented through this API and\nthen allow a TreeFilter to be set directly on the WorkingTreeIterator\nto \"pre-filter\" the entries before they get returned for merging\nin the TreeWalk main loop.\n\nI am certain we need to write new subclasses of TreeFilter to handle\nfnmatch(3C) style glob rules.  But they shouldn't be too difficult.\nFor example we already have PathFilter to perform equality testing\non path names.\n\nBuilding onto TreeFilter makes for some more interesting cases,\nas we can then feed globs into the revision machinary and actually\ndo something like `jgit log -- 'path/*.c'`, with the globbing being\ndone _on_the_fly_ at each tree, and not once up front by the shell.\n\n \n> For an example see the unit test testRealisticExample() in the class\n> FileIterableFactoryForAddCommandTest:\n> http://repo.or.cz/w/egit/florian.git?a=blob;f=org.spearce.jgit.test/tst/org/spearce/jgit/lib/fileiteration/FileIterableFactoryForAddCommandTest.java;h=d3c78f4422c708f26ccb56434053bb711fa3116b;hb=669fd814d34e2f989b5f8eedbcb0d5bcf9743ce7\n> \n> You can view the patches online at:\n> http://repo.or.cz/w/egit/florian.git?a=shortlog;h=refs/heads/mailinglist-patches-0\n> \n> I signed all patches and formatted them with the code formatter as I\n> should. It's ok for me to put the patches under a dual license between a\n> 3-clause BSD and the EPL[*3*]. Currently all files have a GPL 2 notice.\n> I hope that is ok.\n\nThanks.\n\nI'll be writing a script to edit the headers to switch GPL notice to\nEDL (3-clause BSD) notice real soon, and apply it to the bleeding\nedge tree.  We may need to use it a few times to cover everyone's\ntopic branches before they merge into the main tree.\n \n> If you want I will send the patches to the mailing list, but I don't\n> know any automated way to create all the emails. I am not even sure if I\n> will get them formatted correctly with Thunderbird 2. It would be cool\n> if you could tell me how to send patches via command line.\n\nI think the command you are looking for is `git send-email`.\n \n-- \nShawn.\n"},{"id":"76530","messageId":"4825BB3F.4060006@web.de","threadId":"13454","inReplyTo":"20080510001132.GF29038@spearce.org","subject":"Re: [egit / jgit] Implementation of a file tree iteration using ignore rules.","fromName":"Florian Köberle","fromEmail":"florianskarten@web.de","sentAt":"2008-05-10T15:11:59Z","receivedAt":"2008-05-10T15:11:59Z","isPatch":false,"sender":{"key":"florianskarten@web.de","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nHi\n\n| This is an interesting start.  Did you see the existing \"Main\" class\n| in org.spearce.jgit/src/org/spearce/jgit/pgm?  It sets up and invokes\n| a TextBuiltin, which is sort of like the \"Command\" class you added in\n| your first patch.  Though TextBuiltins are created on-the-fly and thus\n| are harder/impossible to use to format a \"jgit help\".\nI noticed that the class appeared after a rebase, but didn't have a\ncloser look to it yet.\n\n|\n| I think your approach of building up a table of commands is likely\n| the better one long term, so I am interested in seeing the two unify,\n| taking the best from each (from Main and MainProgram that is).\n|\n| Please note that jgit is restricted to Java 5 APIs only right now.\n| The \"MainProgram\" class you introduced uses Arrays.copyOfRange()\n| which does not compile under Java 5.  I guess it is new in Java 6?\nYes it is new in Java 6. A patch fixing this is contained in the patch\nset I send to the mailing list.\n| What is missing here is really two things:\n|\n|  #1)  Take .gitignore and .git/info/exclude (and other patterns) into\n|       account as WorkingTreeIterator (base class of FileTreeIterator)\n|       loops over the entries in a directory.\n\nI had a look at the WorkingTreeIterator and it seems to me that it is\npossible to reuse my Rules class there.\n\nWe could simply give the iterator a member variable of type Rule.\n\nThe method loadEntries of WorkingTreeIterator could then use the rules\nclass to filter out unwanted files and directories.\n\nThe constructor  WorkingTreeIterator(final WorkingTreeIterator p) could\nuse the Rules#getRulesForSubDirectory to create a Rules instance from\nthe parent Rules instance.\n\nAlso note that my Rules implementation would ignore the directory a in\nthe case of \"/a\\n!/a/b.txt\". This means that a directory may not appear\nin the list entries, but must be used to create another iterator.\n\nI suggest to put all the classes from the package\norg.spearce.jgit.treewalk and the package\norg.spearce.jgit.lib.fileiteration into one package. Please tell me\nwhich package and I will send a patch, or do it yourself. I don't have\nany outstanding changes.\n\nI don't see a easy way of porting my Rules implementation to the\nTreeFilter framework, but as you may noticed it is may not necessary to\ndo so.\n\n| I think the command you are looking for is `git send-email`.\nThanks, after switching to a newer version of git I was able to use that\ncommand.\n\nBest regards,\nFlorian\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.4.6 (GNU/Linux)\nComment: Using GnuPG with Mozilla - http://enigmail.mozdev.org\n\niD8DBQFIJbr659ca4mzhfxMRAn/ZAKCYERVJfHgZmvGFXEP+uCT0rD2RawCgqBIW\nXa4NTcAWjt8j0oMPdKMjbbY=\n=koEN\n-----END PGP SIGNATURE-----\n"},{"id":"76561","messageId":"20080511001231.GI29038@spearce.org","threadId":"13454","inReplyTo":"4825BB3F.4060006@web.de","subject":"Re: [egit / jgit] Implementation of a file tree iteration using ignore rules.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-05-11T00:12:31Z","receivedAt":"2008-05-11T00:12:31Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Florian Kberle <FloriansKarten@web.de> wrote:\n> | This is an interesting start.  Did you see the existing \"Main\" class\n> | in org.spearce.jgit/src/org/spearce/jgit/pgm?  It sets up and invokes\n> | a TextBuiltin, which is sort of like the \"Command\" class you added in\n> | your first patch.  Though TextBuiltins are created on-the-fly and thus\n> | are harder/impossible to use to format a \"jgit help\".\n>\n> I noticed that the class appeared after a rebase, but didn't have a\n> closer look to it yet.\n\nI guess you started working on an older version, only to later find\nout that I had also done a lot of work in the mean-time.  :-)\n\nMy jgit contributions come in huge bursts.  RevWalk/TreeWalk was one\nback in March; the transport API (almost 10,000 lines of code itself)\nis the most recent from late April/early May.  Once its fully into\nthe mainline I'll probably have to slow down for a couple of months.\nI have to move in July and have a lot of things to do between now\nand then.\n \n> | Please note that jgit is restricted to Java 5 APIs only right now.\n> | The \"MainProgram\" class you introduced uses Arrays.copyOfRange()\n> | which does not compile under Java 5.  I guess it is new in Java 6?\n>\n> Yes it is new in Java 6. A patch fixing this is contained in the patch\n> set I send to the mailing list.\n\nTo keep the history bisectable as much as possible it is better\nif you use `git rebase -i` to squash these two changes together,\nso that we never introduce the Java 6 usage into the codebase.\n\n> I had a look at the WorkingTreeIterator and it seems to me that it is\n> possible to reuse my Rules class there.\n> \n> We could simply give the iterator a member variable of type Rule.\n> \n> The method loadEntries of WorkingTreeIterator could then use the rules\n> class to filter out unwanted files and directories.\n\nYea, that sounds right.\n \n> Also note that my Rules implementation would ignore the directory a in\n> the case of \"/a\\n!/a/b.txt\". This means that a directory may not appear\n> in the list entries, but must be used to create another iterator.\n\nOuch.  I forgot about that fun corner case.  In the context of a\nTreeWalk directory \"a\" must actually still be reported as an entry so\nthat the TreeWalk main loop knows to enter into the subtree iterator.\nHowever the subtree iterator needs to only have entry \"b.txt\" within\nits entry list.\n \n> I suggest to put all the classes from the package\n> org.spearce.jgit.treewalk and the package\n> org.spearce.jgit.lib.fileiteration into one package. Please tell me\n> which package and I will send a patch, or do it yourself. I don't have\n> any outstanding changes.\n\nThe treewalk package is already established so I would say add\nthem there.  Since you are the original developer and your code is\nnot yet in mainline I would ask that you perform the renames.\n\n> I don't see a easy way of porting my Rules implementation to the\n> TreeFilter framework, but as you may noticed it is may not necessary to\n> do so.\n\nThe TreeFilter framework is perhaps not the right API for ignore\nrules, that is likely true.  It also works with paths as byte[] and\nnot as String, because we get byte[] (generally UTF-8 encoded) data\nfrom canonical tree objects when reading from the object database.\nAvoiding the conversion for most entries is a huge performance\nimprovement for us.\n\nI still won't give up my silly dream for `jgit log -- 'foo/*.c'`,\nbut maybe we do need two different implementations to make things\nwork out well.\n\n-- \nShawn.\n"}]}