Blogger.com online editing is awful

I have just spend some VERY frustrating days fighting with Blogspot/Bloggers online editing, as I was finishing my "Guide to OpenJDK" article. Avoid using the "a name" tag at all cost, it is normally used to create internal links to chapters in a webpage. When you switch to the preview or compose page, Blogger scripts automatically duplicate this tag EVERYWHERE throughout the text for some bloody reason. Before header tags, inside header tags, after header tags, before links, before images, before every paragraph, before lists, inside every list item.... You will have to spend hours removing them... and if you miss one, like I did, it will metatasize like a cancer all over your text if you switch editing mode again! Fun!!

Also it doesn't save your original hand edited HTML. If you use the "lesser than"/"greater than" html elements to show, for instance, an email address, when you go to compose window everything looks ok, but when you switch back to editing html it will try to parse what is inside the "tag" characters, fail and delete your text.

The LG DBRH198

Our Samsung digital TV decoder broke down recently, and when looking for a replacement I found the LG DBRH198 - a combined DVD/CD/DivX/WM/jpg player, digital decoder, cd/dvd burner and 120GB media recorder. After spending a very frustrating day and a half trying to get it to work, I am now returning it.

Pros:

• Would have replaced four machines (VHS, DVD, CD, decoder) with one, if it had worked. Compact living, and saves energy!
• Image was sharp, at least on our small LCD TV.

Cons:

• Refused to read our Boxer smartcard connected through a CAD adapter, only the free channels worked.
• Text TV fonts were ugly, an eyestrain to read.
  
• When going through menus it kept "locking up" randomly so we had to pull the plug repeatedly.
• If we accidentally pushed the AV button on the remote we lost the decoder image. It got to AV4 and then refused to cycle back to the beginning no matter what buttons we pushed and we had to do a "reset to factory settings" to get it to read from AV1 (antenna) again. Perhaps there is a better way, but the manual didn't say. I don't know if this is a bug or just really poor user interface design
• Poor interface design in general. We are two engineers in the household, and if we were having problems finding out how the menus work... not a good present for grandma.
  
• Poorly translated menus. Unlike some machines I've owned, this time it wasn't so bad that it made understanding difficult, but it was still distracting. Manual was ok though.
  
• Felt slow and sluggish - slow on startup, slow to change channels, slow to read the DVDs.
  
• We could not get sound to work through the decoder. It generally worked when playing DVDs (see below), and once we stumbled into playing back a recording from the hard drive and then it worked, but we never got it to work when watching TV directly through the decoder for some reason. Perhaps there was some setting hidden somewhere, or we should have used another audio cable, but this is something I think should just work out of the box and not something you have to fiddle around with for hours trying to get to work.
• Once when playing a DVD the sound was suddenly just roaring white noise. The same DVD had worked fine 10 minutes before on the same machine...
  

Previously I have been pleased with LG products. Certainly not high-end stuff, but cheap, fairly robust and lots of features. Seems this machine had too many features crammed into it and not enough quality. I think I'll try to replace it with one really good decoder with build in HD recording, and one really good DVD player. Probably not LG this time.

Beginners guide to OpenJDK contributing

**DEPRECATED**

There is now an official OpenJDK Developer guide which incidentally appeared two days after this article was noticed on the java.net frontpage.
:)

You should go there for up to date information instead. Thanks to Brad Whetmore for useful feedback though, and getting this article noticed by mentioning it on his blog.

**DEPRECATED**

This is an expanded version of a presentation I held at Javaforum Stockholm in December 2007. It is intended as a quick "getting started" guide for those who wish to participate in the OpenJDK project. All the information was already out there, but spread over many different sites and I thought it needed summing up in one place so beginners could see step by step what they need to do to get a patch accepted. A warning - as the OpenJDK project changes quickly the information here may be out of date. Comments, feedback or errata is much appreciated. The article is published under the Creative Commons.

Contents

• What is OpenJDK
• Why contribute?
• Who are in charge?
• Organization - Groups and Projects
• Getting the code
• Preparing for the build
  
• How to build
• Short code overview
  
• Contributor agreement
• Testing your changes
  
• Unit/regression tests
• Committing patches
• Making a good patch
• What I would like to see
• Links
  

What is OpenJDK

OpenJDK is the project to open source the development of the Java platform and virtual machine. Java has always been free to download and use, and the source has been available under the Java Research Licence. Now it is also available under a Free Software Foundation approved licence - GNU General Public Licence 2 with Classpath exception. This is not a separate project to Sun's "ordinary" Java. It is not a "ok kids, go play with this and don't bother us anymore" abandonware - the engineers at Sun will use the organization and tools described here exactly like external contributors do, and this source will be the base for the release of Java 7 and future releases as well as possible alternative distributions.

Note that the OpenJDK project is only for collaboration around the implementation of the Java platform. The specification is still being decided by the Java Community Process, which you really should join incidentally... free for individual members, you get to vote about once a year about who will be in the expert groups that decide the future of the platform.

Why contribute

Why should you waste valuable time contributing to open source when you don't get paid for it? Here are the reasons I have to do that:

• Increase your competence as a programmer. Perhaps you have a few hours a week at work dedicated to increasing your competence? Gaining a better understanding of the JVM and seeing how common algorithms are implemented is a pretty good way to do that.
• You increase the value of your chosen platform. You make it more likely that there are good jobs for you in the future. Considering how much stuff have been written in Java the last 10 years I don't think there will be any lack of jobs for competent Java programmers for as long as I live, but perhaps you want some new development in the future, not just maintenance programming.
• Make new contacts. When I was at JavaOne this year people at Sun who work in the management/JMX areas recognized my name when I came up and talked to them because I had submitted a couple of patches in that area.
• It is an excellent merit. You have code that you are legally allowed to show to potential employers. This is not just a little open source project that you and your closest friends use - if you submit patches to Java (or Linux), your code will be used in critical applications by tens - make that HUNDREDS of millions of end users daily.
• If you have a reoccuring bug that you have had to do an annoying workaround for - here is your chance to get rid of it permanently, not just for yourself but for all programmers around the world.
• You will contribute to a more free and better society. Ok, I realize we are getting dangerously close to starry-eyed idealim territory here, but knowledge is power in today's world and computers are our primary knowledge-management tool. Therefore I think it is unwise to allow all control of our computers to fall to any singular entity, be it a state or a few large companies.

Who are in charge

Currently there is a "interim governance board" which I believe was selected by Sun, with help from the open source community. They will write a constitution and decide the final form the project will take. The members are:

• Simon Phipps - Sun Chief Open Source Officer (Image)
• Fabiane Biznella Nardon – CTO of the company ZILICS. (Image) . She is also the JavaTools community leader
• Dalibor Topič - Max Planck Institute. Well known for his work with Kaffe and Classpath.(Image - center)
• Prof. Doug Lea - State University New York. (Image) Well known for his work on Java concurrency - author of several books, and the java.util.concurrent libraries introduced with Java 5, expanded and improved in Java 6 (and likely Java 7, with the forkJoin library).
  
• Mark Reinhold - Sun JavaSE Chief Engineer (Image)

Organization - Groups and Projects

OpenJDK has two main concepts - Groups and Projects. I think this could be changed in the future by the governance board in theory, but I doubt that will happen. Groups and projects are described on the official OpenJDK site, but here is a brief summary:

Groups

A group is a collection of people with a common interest. There are three levels of participation: 1. Participant 2. Member 3. Moderator. A participant is anyone who has subscribed to the group mailing list. A Member is someone who "has demonstrated a history of significant contributions to a Group, has been granted Membership by that Group, and has signed the SCA" (more on the SCA later). A Moderator looks after the mailing list, counts votes etc.

Groups usually have web pages describing them and mailing lists for discussions, but they DON'T have code repositories. A group can chose to sponsor a project. The initial groups were more or less the engineering teams at Sun - Java2D, Security, Hotspot, etc. Any group Member can suggest new groups. An examples of this is Dalibor Topič's Porters group suggestion which was approved just a few weeks ago.

Projects

Projects exist to create some form of "artifact" (code, documentation). Projects, unlike groups, do not have member lists, they are open to all who want to contribute. Projects often have code repositories, and are often limited in time. A new project is formed if a group decides it wants to sponsor it. An example of this is the JDK7 project proposed by Mark Reinhold.

Getting the code

The Java source code was previously available as a read-only Subversion repository, but the project has now moved toMercurial (hg). Mercurial is a distributed version control system. If you can read the repository, you can create a local clone - indeed, that is what you must do to work with it. Doing experimental forks of the code is therefore trivial, and it is hoped that increased local influence creates happier participants and a decreased risk of permanent "political" forks. If you haven't worked with Mercurial before, the greatest difference to svn/cvs is that you always have a local clone of the repository to work against, and that downloading patches and applying patches are two separate steps (unlike svn/cvs "update" command).

Once you have downloaded it, "hg help" is your first command. There is a plugin in Netbeans (and in Eclipse too I'm sure) but the original program is run from the command line so I recommend you try it out that way first, to get a feel for all that is possible. A short list of useful commands -

• hg clone ${localpath_or_url} - create a local copy of a repository
• hg status - how do your work files diff against your local repository?
• hg commit - save work file changes to local repository
• hg incoming - what has changed in parent repository (the one you cloned from)
• hg pull - download remote patches. Again, this step does not apply them.
  
• hg update - apply downloaded patches
• hg push - send your changes to the parent repository (if you have write permission there)
• hg merge - merge changes, best done with a tool (kdiff3, meld, Netbeans, Eclipse)

All of these commands take lots of different parameters of course, but rather than me repeating it, there is a really good free book available on Mercurial at red-bean.com.

Mercurial extensions

One of the advantages of Mercurial is that it is easy to script extensions using Python. Sun has chosen to use the "forest" extension for OpenJDK, presumably so that subteams can administer themselves more easily. If you have already installed Mercurial, you can download the extension with it, using the following command:

hg clone http://hg.akoha.org/hgforest/ hgforest

Next you should edit your Mercurial configuration file. In your home directory, create a file called .hgrc and add the following lines to it:

#Mercurial configuration
[ui]
username = your_name_and/or_mail_here
[extensions]
forest=/home/your_uid/hgforest/forest.py

The username defined there will identify you as the author in the commit metadata. At the public web repository of the code your mail address will be obfuscated, but your username string is not automatically obfuscated in the patches you submit. If you are sensitive about spam/privacy issues you may want to get a separate mail account. It is not improbable that some spamming scumbag clones the repository one day and combs through it for addresses.

Once the extension is installed and configured, some commands will be replaced with their forest equivalent (fclone, fpush), so to finally download the code you do:

hg fclone http://hg.openjdk.java.net/jdk7/jdk7

Preparing for the build

You should read the README-builds.html page included in the source to begin with, these days it is very informative and helpful. You have to install the usual development libs. What differs OpenJDK from many other projects is that you have to download a bootstrap JDK - I think it is JDK6 now - and also the "binary plugs" which contain proprietary code that Sun doesn't own and did not get permission to release as open source. You do not need the binary plugs if you chose to download the Iced Tea version of the OpenJDK code. The Iced Tea project was created by RedHat and aims to replace all non-open source code so that you can build a completely free JDK from scratch. RedHat has signed the OpenJDK participation agreement I mentioned before, so their fixes will hopefully make it back into the core repository. When that has happened you won't need the binary plugs or a separate download from IcedTea anymore.

I have only built OpenJDK under Linux, but if you are planning to build under Windows there are a few things to consider. First, the compile process is oriented towards a Unix like environment so Windows style file paths, environment variables etc do not work. Therefore you must download and install Cygwin. According to the README-builds file you MUST use Visual Studio .NET 2003 Professional, the 2005 or later versions do not work, at least not out of the box. Luckily, Tim Bell and others are looking at getting the "free-as-in-beer" Visual Studio Express for C++ to work (and in the future hopefully also "free-as-in-freedom" tools).

In theory only tested and perfectly functioning code is in the repository, but in reality people have mistakenly checked in non-compiling/non-working code often enough that it might be an idea to do as Elliotte Rusty Harold suggests - download a source snapshot first and try to get that to build, and then go on and try the latest repository code.

Building

There is a file in the project source code you checked out -

jdk/make/jdk_generic_profile.sh

which sets all environment variables needed for the build. Look through it and edit it if needed, and when everything looks ok try running the following commands:

bash
. jdk/make/jdk_generic_profile.sh
make dev-sanity
make dev

The "dev" version of the build is a bit smaller, if you really really want to build everything, including an installer, replace the last two lines with:
make sanity
make

Once you have built the whole project, you can build just the subprojects you are working on by guild into the corresponding "make" subdirectory and run the "make" command from there, this will save you time. Doing builds of subprojects depends on tools created by the global make, so the first time you must unfortunately build everything.

Code overview

For easier maintenance the source has been split up into separate projects - langtools, jaxp, jaxws projects mainly consist of java code, hotspot is native code, corba is mixed.

The sources for the core Java classes you use daily are usually in
jdk/src/share/classes/java(x)
Some of these classes are just interfaces or abstract classes. The implementations are usually in
jdk/src/share/classes/sun
(These classes in the sun package are sometimes used by applications, but it is strongly adviced that you never do that. It ties your users to Sun's JVM versions, and these classes are not a public API and therefore can change without any warning between JVM releases.)
Operating system/platform specific Java code can be found in
jdk/src/{platform}/classes
JNI created files usually end up in
jdk/src/share/native/
but these are usually just header files. All the fun stuff tend to happen in
hotspot/src/share/vm

TODO - expand this part of the article.

Requesting to participate

A lot of the following information is taken from OpenJDK contribution page. First of all print out the Sun Participation Agreement and read it through carefully. Basically you agree to "dual ownership" of the code, both parties can do whatever they want with it without asking the other part first. More details at the FAQ.

If you decide these terms are acceptable, sign it and mail it back to the provided address, or scan it and fax it to them. If you have signed this for any earlier Sun open source projects (Glassfish, Netbeans, OpenSolaris...) you are already covered and don't have to do it again.

Getting started with a bug

First of all find a suitable bug or Request For Enhancement (RFE) in the bug database. If you know of a bug but can't find it in the database, please start by submitting the bug! Note that even though a bug has been accepted into the database, the submitter's synopsis can be misleading or completely incorrect. If there is an evaluation there you are on firmer ground, but I have known these to be out of date or incorrect also, so don't trust everything you read, you must check the code and the specs and think carefully about how things really work, and how they are intended to work. Sometimes bugs are even fixed but not closed in the database. When you have a good bug, search through the mailing lists and see if you can find any discussion about it - perhaps work on it has already started. If there is no discussion, announce your intention to start. Send a mail to the appropriate mailinglist with the subject "{Bugid: Synopsis}". For instance:
162111: Incorrect Descriptor handling in ModelMBean classes

Body of the mail is usually just something like "My name is... and I'd like to get started on this bug". Wait a few days for comments. Perhaps Sun engineers have already started working on this bug, perhaps they will re-evaluate it now and close it for some reason. In general, changing any public APIs (adding new methods for instance) is VERY DIFFICULT. You have to provide a convincing argument why this is needed, if the bug synopsis doesn't make a convincing case you could try to add some arguments to the body of your "starting intention" mail. Also, changing incorrect behaviour that applications already depend on is pretty much impossible - backwards compatibility has been holy so far. Still, just because Sun won't merge your patch into the core OpenJDK repository doesn't mean that others won't find it useful...

Testing changes locally

Before submitting a patch you probably want to try out your changes. If you have compiled with the default build target and everything worked, you will have a newly compiled JDK located at build/{platform}/. One way is to just set the $PATH and $JAVA_HOME variables there and then try running all your favourite Java applications. On the other hand, perhaps you want to try to isolate your changes so you can test just those classes you changed together with an already installed JDK that you know works fine. You do this with the Xbootclasspath parameter. For instance -
java -Xbootclasspath/p:jarname.jar
The "/p" part of the command is important, it prepends classes. This means that if the JVM finds any of the core java classes in the jar files or directories you specified (jarname.jar in my example above), these class versions will be used instead of the internal ones in your installed JDK. You can specify several jar files or directories as parameters.

Automated tests - Unit/regression testing

If you want your patch accepted you must also provide test classes. Most of you are probably familiar with JUnit/TestNG or similar test frameworks. Sun for a long time used an internal testing framework called JavaTest with the "jtreg" extension to run many of the JDK tests. This framework has now been open sourced. You can define test suits in java files, shellscript files or html files. The tests executed by jtreg can test not only ordinary Java files, but also applets and shellscripts. The Java classes you test do not need to implement any special interface, you can define the test to just run the main method of the class. If the process started by the test returns normally (the main method or shellscript finished) the test passed. If the JVM terminates with an uncaught exception, or the JVM or shellscript returns an errorcode (-1 usually) as it terminates, then the test failed. You can run tests with

jtreg -jdk:${jdk_path} ${testdir_or_testfile_path}

I'd like to write more about this works, but documentation is a bit sparse and when I last submitted my patch it was still JUnit that was asked of external submitters. Hopefully this situation will improve.

It is difficult to give any rules for how much testing you should do... if you are just changing how GUI are drawn or documentation you can't do any tests of course. As a rule of thumb, perhaps you should consider spending at least as much time writing tests as you did fixing the code.

Submitting your patch

Now we are finally getting to the fun part! You submit your patch by sending a mail to the appropriate mailing list, with a subject of the following pattern:

[PATCH] 162111: Incorrect Descriptor handling in MBean classes

The mail should contain the following:

1. A discussion of the change: A rationale (if the bug synopsis didn't have one good enough). Briefly what you have changed and why. If you had any alternative solutions that you rejected, you may want to write a little about why.
2. A diff in the "unified" format (-u). Remember to write which version of the source you created the diff against!
3. Your tests classes.

Within a couple of weeks a Sun engineer (or someone with the same responsibility in a project) should reply and say if the patch will be accepted, or if it needs further work.

Making a good patch

Here are a few rules of thumb to maximize your chances of getting your patch accepted:

• Follow The Java Code Conventions. Your code comments should describe WHY you do something if that needs explaining, not HOW you do it. Writing how you do it in comments is unnecessary duplication, that should be immediately obvious when looking at the code or something is wrong with it. Clear and descriptive variable names, yadda yadda. You know the rest...
  
• Idiomatic Java code that is readable even to junior programmers is preferable. Of course you shouldn't avoid writing smart and concise code, but don't show off for the sake of showing off, abuse "clever" tricks to fit the code into a single cryptic unmaintainable line, and so on.
  
• Write lots of good tests.
• Most important rule last - only change what is necessary, nothing else! This is a mistake I did when working on my first patch... Your patch will be carefully evaluated by core engineering teams, the smaller the diff is, the easier it is for them. Never run "fix imports", "autoindent all code" or similar in your IDE, start cleaning up cryptic variable names elsewhere in the class, etc, even if you think the original code looks like complete crap. The patch will probably be rejected immediately if you do. Stick to fixing your original bug. Do code cleanup as a separate patch later that doesn't change any functionality, if you think this cleanup is absolutely necessary for maintainability.
  

If the patch was not accepted into OpenJDK trunk by Sun, don't be discouraged. You probably learned a lot while doing it. Other projects or repositories may still be interested in your patch.

Another rule of thumb I like is - try to be nice, if you need to criticise try to make it constructive criticism. In my experience Sun engineers are almost always been polite and helpful, but as the community grows some highly opinionated (and also very skilled) open source contributors may not hesitate to let you know if they think your code is poor in no uncertain terms. If that should happen, see it as an exercise in maturity. Thank them (silently if you prefer) for the what you may have learned from the exchange, and disregard any possible personal attacks. Consider that current/future employers may read the exchange one day. The Internet has a very long memory, and your online reputation matters.

What I would like to see

For the OpenJDK to succced, Sun must continue their work to lower the barriers to entry. But a living and vibrant community is not something that Sun can create top-down with a wave of a magic wand, it is something that all of us who are interested in seeing OpenJDK succeed must try to help create. I'd like to see the following:

• More participants! We want YOU for OpenJDK! There must be hundreds of thousands of active Java programmers on the planet. If only a few percent contributed a couple of good patches each....
• More information from Sun employees what they are working with. Previously I have seen new contributors who have had their first patch rejected because a internal rewrite/planned rewrite fixes the bug, and they were disappointed. Luckily this situation has already improved now that Sun's engineers use Mercurial, and the same mailing lists for discussions as everyone else.
• Cleaning up of old and inaccurate information online. The Starter bug list page for instance should be updated, I think half of the bugs on it are closed by now. I would also like it to be expanded so it covers not just individual bugs but whole areas with a low barrier to entry where beginner contributions would be welcome - something like the Linux Kernel Newbies community. Perhaps an OpenJDK Newbies Group could be created?
  

Links

On the Open Road - series of articles about OpenJDK. Rustys article beat me to the punch by a week or so and made this article somewhat redundant, but it was good to see that I was on the right track.
Kelly O'Hair has loads of information on his two blogs about Mercurial, and building the JDK.
Ted Neward blogs about building the OpenJDK on Windows.
Volker Simonis has two very in-depth articles about HotSpot development on Linux with Netbeans.