Verilog/ASIC development support is insufficient in git , help!

Hi all,

* This email probably represent the whole hardware ASIC community about git 
*

I'm evaluating Git as the replacement of CVS for the ASIC group in my company, but things are moving along very bumpy.

I (and many others doing the evaluation) love the tool dearly; we love the local repository and inter-db sync'ing . I see a lot of potential in productivity and changes in work model that helps efficiency in ASIC dev.

BUT, my managers, some veterans, and directors are EXTREMELY concerned about the ease-of-use.. so much that they are going to pick SVN ! uh-oh....i m serious =(

Alot of people argued, why not SVN ? it's CVS++ and it's ease of use not a problem when comparing to Git.

here are the things not fitting right in ASIC dev:

- no incremental revision numbers (they are so scared of the 40hex SHA1)

- Inability to reference without SHA1, they want simple numbering (ie, 
version 100, 120, 120.1, 130.4.5)

- Inability to refer to a file by a simple number
(the backend guys will be confused by SHA1; they can't work with anything 
more than 4-5 digits)

- Complexity of commands (although we can have warpper, but real git 
commands for non-sw guys is not going to happen)

Most hardware chip designers were using CVS since their first job. It suited the purpose very well.

Most RTL design veterans only use less then 5-6 cvs commands in their whole life (LOL, i m serious) :

$ cvs checkout $ cvs update $ cvs log $ cvs diff (tkdiff) $ cvs status $ cvs commit

We don't use branches. Our model is strict forward with a centralized, one main branch model to avoid mistakes . We see branches as evil ; some merges in Verilog codes means another 10+ hours of simulation and regression.

I'm a verification engineers for the hardware chips designers, there we use Vera and SystemVerilog which requires much in depth use of SCM functions. So, the choice of tools is much more important on our side (the designers only checkin and out, diff, and minimal merging)

I m frustrated about the situration, i truly want Git in ASIC world !!! (yell out loud... no p4, no svn, no clearcase... or i rather keep cvs)

Is there a way to specify the use of a simple GIT model in config, or like, info/attribute, such that (in git main repository model of course) :

(1) SHA1s are hidden, but replaced by simple numbers (2) Simple, incremental numbers (like 'git-5432' ; what we use 'git-describe' to generate) (3) Reference of simple revision numbers in all git commands and tools like gitk, not SHA1

I personally have no problem with the SHA1 . but many are allergic to it .

As I have learned so much about the power of git, I understand that:

- 'git-show-branch' actually show reversed serialized version numbers (we 
want it the other way, accending)
- 'git-describe' gives you commit numbers since your last annotated tags ( 
ie, git-5423-g7def45b)

so, i understand that a simple numbering scheme can be done .

I truly hope that the in the main repository model of git this can be turned on by a switch or in the git config .

Is it too complicated to incorporate this model ?

I m eager to hear about the options

Thanks,

  Justin Leung

  ASIC Verification Engineering

  Redback Networks
  a company of Ericsson 

On May 11, 2008, at 12:08 AM, Justin Leung wrote:

Honestly, it sounds like SVN is actually a good fit here. Just because git is awesome for many things does not mean it is the end-all-be-all of version control systems. SVN still has its place as the last true centralized system. Given your constraints and workflow, why do you think git is better than SVN?

-Kevin Ballard

Hi Kevin,

The ability of inter-user (designer-verifier) code sync'ing without letting the incomplete or incompatible RTL code to propagate to the main build stream is going to facilitate the design flow and efficiency .

The ability of local revision control means that no more over writing of design files until commit .

I think these 2 things will really buy us alot .

 Justin

On May 11, 2008, at 12:29 AM, Justin Leung wrote:

But they come at an expense - no more linear revision numbers, more complex commands, etc. You can't have it both ways.

You could always use a main SVN repo and then use git-svn to maintain your own private git repos, do all the code syncing you want there, and then push it back to SVN when you want to send it back to the main build stream. If you go this route, be careful to maintain a linear history on the branch that tracks the SVN repo, as SVN cannot handle merges the way git can. You'll want to either do rebasing or squashed merges (to avoid multiple parents).

-Kevin Ballard

Thanks Kevin,

   I think i better give svn another serious look . 

but then, my impression from the rest of the people is that p4 and svn are not the ultimate tools for hardware development either .

I guess we just do not have the perfect tool for asic dev yet .

 Justin

Kevin Ballard wrote:

On Sat, May 10, 2008 at 10:29 PM, Justin Leung <jleung@redback.com> wrote:

Hi Justin,

I was originally drawn to git for the exact reasons you identified in your 2nd email. Namely, it is extremely difficult in a p4-based environment to share intermediate work within a design team without pushing the work out to be visible by the entire team. "Inter-user design sync'ing" is exactly what I wanted. In its absence, we have made all references between files relative. This means you can flip over to someone else's netlist by changing one path (say to the top-level design file) to point into someone's private repository. That top-level file then includes everything else using paths relative to its own location, so you get the correct stuff automatically. Of course, you get tripped up all the time by stuff implicitly used and not named in the top-level file and its children...

Now, it would be far better for this to be a lightweight branch in git, and then having people checkout this branch and use it. (Because, for example, while one person is pointing into another's tree, the latter can't change.) But p4 (and cvs) has trained everyone to think of branches as painful and for wizards only. Plus I am not personally interested in investing any time writing scripts on top of p4; the ideas I outlined in the previous paragraph were easier and almost as good as anything (easily) doable in p4 (but not as good as lightweight branching).

I agree with other responses to your email that you may want to think about writing simple wrapper scripts that add tags to checkins with some simple incrementing numeric part to keep your back-end people happy. Yet other responses were distracted by the linearity of your centralized/shared checkins: the inter-design sync'ing you want, and the lightweight branching it may imply, aren't necessarily incompatible with the linear main public history that most design teams expect (and which is unavoidable in design work containing lots of unmergeable files, such as layout design). So I don't necessarily think you would be happy with Subversion (I'm certainly not happy with p4).

There are two other issues you may want to keep in mind. In our chip design activities, we have a lot of very large files (100MB to ~3GB), and the p4 repository has grown beyond 3TB. Now, this is simply a data set size region which is not used by the git developers. I think the git data model is fine for large projects and files (Linus mused otherwise a few weeks ago, but it seems fine to me), but due to lack of use, various details when handling large files/projects remain to be worked out and/or optimized as much as the rest of git. It is true since I started watching there have been a lot of important improvements in this area.

Secondly, you may also want to discuss with your IT people (or whoever is responsible for back-up) how git packs/repacks repositories. Ours were very uncomfortable with the idea that the _entire_ repository has to get re-arranged frequently. I think they would have been much happier with an approach more similar to how Unix systems were backed up in the 80s: have a level-0 repack which repacks everything, a level-1 which repacks only stuff added since the last level-0, level-2 since level-1, etc. To do this would be a pretty straightforward change to git-repack.sh, probably using .keep files. In each level it is clear what needs to be backed up.

Anyway, good luck! Many of the things you touched on, or which I mentioned above, have been (partially) implemented or at least discussed before, so your requests aren't crazy. Unfortunately, in my case, having 6 surgeries in my family in the last year has kept me from doing that much useful for git along these lines and thus I remain stuck with p4 for now.

Hi Dana,

 My best wishes to your family . having 6 surgeries is no joke.
I wish you have all recovered well .

Thanks for letting me know that I am not the crazy one to try to implement git in asia :)

I believe that this tool is full of potential in our community. All it takes would be just a minor tweak to fit our methodology .

in the mean time, while my team is still happy with cvs , due to the design habits (good or bads.. but they are used to the way it is ) , probably svn and p4 are still the only logical choose ; not that I m happy with these chooses though.

git will get there but i think no big hardware firm would like to be the first to adopt to it .. especially my managements would like to minimize risks .

 Justin

Dana How wrote:

On Sun, May 11, 2008 at 7:08 AM, Justin Leung <jleung@redback.com> wrote:

I'm an ASIC designer too.

this is unimportant: if they want to track a specific release of a file, it's better to look at what was the file's content from this cut to that cut.

this is where tags and branches are useful to point to a specific release/cut.

same answer

just use gitk and git-gui: almost all can be done with these two graphical tools.

for linear development, yes. but when we were requested to perform maintenance on a specific old cut, this was becoming a nightmare.

gitk, git-gui: two commands (actually gitk can be called from git-gui)

this is the wrong approach.

use branches to reference the different ressources (rtl, simulation, layout). then track these branches between them for deliveries and work/flow.

use tags to mark specific releases/cuts.

you can create an alias: git-show-branch | tail -r

yes, I used to be scared by sha1 too: I even created numbered tags for each commit. Until I read more about git, and stopped expecting using git as svn/cvs.

no, it would kill the right approach: embrace the index, and never look back.

you have to adapt your methods instead: trust another ASIC designer :-)

Hi Christian,

 I personally have no problem dealing without revision numbers .
I merely used them directly .

However, I think I gotta admit that this is a world more than just us .

There's an obvious reason why Windows and MS are still standing, and IE is still the market holder :

we are the minority ..  a lot of people just care to get the work done .

surely i felt defeated, but one lesson to know is that the interface is very important for beginners ;

and a simplified numbering scheme is really what the managers/VPs are looking for to avoid rookie mistakes .

Thanks for the support tho =) I know i m not alone

 Justin

Christian MICHON wrote:

"Justin Leung" <jleung@redback.com> writes:

Please note that Git is not the only OSS DVCS. There is Mercurial, which has repute of being easy to use (in my opinion: at least for mainly linear development), and there is Bazaar (formerly Bazaar-NG). Both are mature products.

IMHO the main strength of Git (besides performance, but now I think it is less of an edge, especially for not very large projects) is deling with nonlinear history and branch-based (topic branches) development. If that is the worth paying essential complexness (as opposed to accidental complexness, which unfortunately Git also has some) for...

Perhaps Subversion is the right tool for you.

Incremental revision numbers _require_ single, central authority assigning them. You cannot have simple revision numbers in truly distributed development.

Both Mercurial and Bazaar allow to use incremental revsision numbers, at least for revisions (commits) along trunk (mainline). Note that revision numbers are either local to repository (Mercurial, Bazaar), or require treating one of repository as special, i.e. use different commands for central repository from commands used when merging in other repositories (Bazaar).

You can tag, and use git-describe output, i.e. v1.4.2, or v1.5.5.1-182-g7480b28. Or you can use transitional "reverse" numbering, counting from latest revision, like HEAD^, master~10, or master@{2} (the last has only local meaning, and only if reflogs are enabled which is now default).

Errr... refer to _file_ by a number? Why not a pathname, or revision and pathname (<rev>:<path> format)?

You can use shortened sha1, usually 7 characters... but I think this is an artificial fear. When using git you almost never have to use either SHA1, or shortened SHA1 identifier.

Now that Cogito is unmaintained, you can try EasyGit (eg) or Pyrite, although AFAIK both are in early development stages. And there is also git-gui and other commit tools there...

And is probably responsible for some SCM bad habits...

[...]

Bad habit!

Probably because branching in CVS was very complicated, error prone, and just plain uncomfortable to use (sticky tags, blehhh...).

Subversion makes it easy to branch, but developers forgot to make it easy to merge (this is to be corrected in upcoming 1.5 release, and you can migitate it using third party tools / extensions like svnmerge or SVK; the last also gives replicability, and beginnings of distributed development).

Bad habit!

Branches are there to not stomp on each other changes when deveoping independent features. They are also a must have if you plan to maintain fixes to old releases.

When merging is easy it allows you to merge early, or do a try merge, and fix conflict early, when they are easy to spot and resolve.

Please watch Linus Torvalds opinionated Google Tech Talk on Git:
  http://www.youtube.com/watch?v=4XpnKHJAok8
  http://git.or.cz/gitwiki/LinusTalk200705Transcript

Could you elaborate? What SCM functions do you need?

Subversion _is_ improvement on CVS, some brain damages of it aside (as
for example lack of _unchangeable_ tags; and no, unchangeable by
convention doesn't count).  You can use git-svn to interact with it.
 

I think you can get it by heavy use of tags from within hooks, even without changes to git core, and without using any wrapper / porcelain. But this might affect performance.

Note (again) that "simple numbers" for revision names are possible _only_ in centralized development, with centralized authority assigning those incremental numbers ("monitor" in parallell computing).

Again: in my use of git I almost never need to use SHA1, or shorthened
SHA1, even for copy'n'paste.

On Sat, 10 May 2008, Justin Leung wrote:

As far as I can tell, wherever SVN is more powerful than CVS, it is complicated to use. I think that, if all you want if to rename files and directories, you could use SVN. Otherwise, the way SVN implements advanced features is more confusing than git.

This is pretty much forced by being able to trade changes without going through the central repo combined with the desire to have stable identifiers. If I can get from you a version after the fifth version in the central repo and before the sixth version there, what number is the revision I got?

On the other hand, we should be able to make the sequence of latest commits on the central site be marked with numbers in increasing order. This is slightly different from a linear order (there will be commits without numbers, for example, if a developer does a series of commits and then sends in a batch, or does some commits and a merge; only the ends of these sets get numbers).

Whenever you have an important state, you can tag it with a number. Not everything gets numbered that way, but the other things aren't so interesting for people working at that level anyway.

I think you'd only want to send the backend guys commits that you'd tag with version numbers.

$ git clone / git checkout $ git pull $ git log $ git diff $ git status $ git commit -a $ git push

The main training is:

 - type "git" instead of "cvs"
 - to get a new working directory, you use "clone" (but to revert a 
   working directory file to the committed state, you still use 
   "checkout")
 - "update" is "pull"
 - use "-a" with "commit", unless you're committing only a limited set of 
   paths
 - after "commit", check whether what you did was right, and then do "git 
   push" to publish your changes

There are multiple things that branches do.

 - The first is that, whenever you have different revisions in different 
   places, they're always different branches in some sense. That is, 
   there's a "latest state of the project" as checked into the central 
   server, but also a yet-newer state that's each person's working 
   directory contents, and often a trailing state that's the central 
   server state as of the last time the person updated. Each of these is 
   tracked even in CVS, but in CVS they're implicit and qualitatively 
   different, while in git they're explicit and mostly stored the same 
   way.

 - The second is that there can be multiple lines of development in the 
   central location, such that there's a latest checked-in-and-published 
   state for each that's different. This is explicit in CVS, and doesn't 
   work very well, and is also available in git and is much nicer. But, in 
   either case, you don't need to use it. (In fact, git's own development 
   barely uses it, and only for bugfixes to older versions while the 
   current version is under development; aside from that, all of the 
   branches in git development are of other types.)

 - In git, in addition to having a "last seen state of mainline" branch, 
   you can have "last seen states" of other developers' working 
   directories. And you can have branches as ways of putting changes out 
   for other developers to get without the main branch getting them. For 
   example, Junio's "next" branch is all essentially previews of stuff he 
   might commit to the mainline. This is fundamentally just a unified 
   tracking mechanism for the ability to do developer-to-developer 
   transfers of state. This doesn't need to be any more (or less) visible 
   to project managers than the contents of people's CVS working 
   directories are (that is, it'll let them know how close the developers 
   are to having a version they can get into the mainline).

 - in git, you can use branches to work on multiple different changes 
   without having multiple working directories. This is, again, a private 
   matter for the developer, and entirely optional.

Well, you'd also have the same 10+ hours in order to retest your changes after a "cvs update" if somebody else commits while you're working on something. A merge in the git sense is essentially the same thing, except that you can have multiple committed states on the local side, and it understands if the same change reaches the mainline by multiple routes because people exchanged work directly.

On ther other hand, you can use "git rebase" and it works almost exactly like "cvs update" except that you've saved the changes that are from your local working directory so that, if you mess up resolving conflicts, you can abort and try again.

If you have a post-update hook on the central repo that makes a tag of the form "r<number>", with <number> counting up, that'll pretty much do it. But there will be other commits that don't get tagged because they were already historical when they reached the central server. But people generally don't have to care about those unless they're already interested in doing something complicated.

	-Daniel
*This .sig left intentionally blank*