Notes on dataflow concurrency

At a conference, I heard someone say something to the effect of: "Declarative dataflow concurrency is awesome. You can't get deadlocks! It all just works, and it is so beautiful".

If you are unfamiliar with dataflow concurrency, it is a technique used in some languages, where a program, if it reaches a statement that requires the use of a variable which hasn't been assigned a value yet, pauses until the variable has been assigned by another thread and then continues the computation as if nothing had happened.

It is pretty neat, but I thought to myself - "Hang on, if we have two threads that each wait for the other to assign a variable, won't we get a circular dependency between them? Or if there is just one executing thread, won't that block forever? I don't understand, that doesn't sound like much different from what I'm used to...?"

Now a month later, I am currently reading the book "Concepts, Techniques and Models of Computer Programming" by Ray and Haridi. And indeed, on page 48 they state:

The computation models of this book use [dataflow programming]. This is unreasonable in a sequential system, since the program will wait forever. It is reasonable in a concurrent system, where it could be part of normal operation that some other thread binds the varialbe. [Dataflow programming] introduces a new kind of program error, namely a suspension that waits forever. For example, if a variable name is misspelled then it will never be bound. A good debugger should detect when this occurs.

So even if they wait on an infinitely unbound variable instead of lock that is never released, the experience for the user would appear to be much the same - a program that "hangs" without producing the desired result. I wrote this little snipped in Mozart to demonstrate:

declare

Y

X

thread X=Y+1 end

Y=X+1 end

{Browse X}

And indeed the browse window never appears. The main thread blocks on line 5, the spawned thread on line 4, each waiting for an unbound variable.

But even though it hasn't been stated outright in the book (at least not yet, I'm at page 350), I think there is an important difference compared to threaded programs as they are usually written in, for instance, Java. The Mozart program is deterministic (if written in a completely declarative style). It will *always* block, no matter how the threads are scheduled, which makes this programming error easier to detect and (hopefully) to locate and fix.

So what the person at the conference perhaps meant, was that the programs written in dataflow concurrency style don't have *race conditions*.

Adept updater crash

I had been hoping to blog more about programming stuff, but here has been lots of "yak shaving" activities for me lately. Linux is still a hairy beast, even if things are slowly getting better. And here is another one!

The "Adept updater tool" in KDE, which has a little icon in the systray informing you when important updates are unavailable, is light-years better than the default behavior of the Windows Update tool in Vista -

("I have downloaded updates and will restart now, shutting down your programs". "I am configuring updates, you may not turn off your computer right now.", "Now I'm going to restart your computer again". "This is a blue screen of death" "Could not start Windows. Would you like to try to restore your computer to last known working configuration?" No joke, this happens to me with SP1 for Vista, can't install it on my Dell laptop)

- but Adept is not completely bug- or annoyance-free either. Yesterday when downloading a new Linux kernel to my Kubuntu system, my update progress stopped suddenly. A command line style menu window had popped up asking if I wanted to keep my old locally changed grub conf file, or replace it with a new one. While Adept could show the menu to me if I went to the "details" tab, it could not process any input from me and transmit it to the install script, so I had to kill the Adept process.

Next time I tried to restart Adept, it said the update database was locked because another program was using it. This is the solution to this problem (most of it curtesy of "harleqinguy" at Linuxquestions.org)

sudo rm /var/lib/dpkg/lock
Removes lock file - only do if you are sure all programs which may access the update db are terminated, but the lock still remains.

sudo dpkg --configure -a
Reconfigures downloaded but not yet installed update packages, and backs up configuration files. Mainly to test if you can run update tools, and to try to fix any configuration mistakes you may have done

Now you can restart Adept, but trying to update your system through it will only lead to the same issues again, to get by this problem and get your shiny new kernel installed, go to your trusty command line and run

sudo apt-get upgrade

KDE login jingle

To turn off the annoying KDE login/logout jingles (which doesn't respect your sound settings and plays loudly when you are at the library, at a lecture, a conference, next to someone asleep, etc, unless you remembered to have headphones plugged in):

Go to the KDE system settings program, click "Notifications" icon, "System notifications", "Applications" tab, select "KDE Systems notification" in dropdown menu, then select "login", then uncheck the "Play a sound" checkbox.

Done. Whew. They hid that pretty well.

Wine eject cd/dvd problem solved

A problem that has popped up for me several times and whose solution can be hard to find when Googling -

If you are trying to install a Windows game or application on Linux with Wine, always start the command from "outside" the cd.
I.e. - always DO write wine /media/cdrom/setup.exe
Do NOT write

cd /media/cdrom
wine setup.exe

If you are promted to insert the next disc, but Linux won't let you do the unmount command because "device or resource is busy" - try
wine eject d:

If all else fails, copy the whole disc to hard drive and run from there.

Questions to ask employers

After thoroughly grilling the applicant for a job/project, many interviews often end with the question - "So, do you have any questions for us?" I've read in articles by career coaches that it is a good idea to have a few questions to ask here - it shows that you are interested. But the first times I had problems coming up with any sincere and interesting questions.

After being at a number of different programming jobs now, big and small, I've come up with a list of questions. Some of these I intend to ask next time I'm in that situation. Not all of these questions require a "yes" answer from me, but at least that they know what these items are and can reason around them. If they answer "We don't know what these things are!", or "Yeah, yeah, I'm sure that's great stuff, but we don't have time for that sort of academic stuff, we have urgent deadlines to meet" that will be a strong warning signal to me to reconsider, or at least ask for decent pay to make up for the inevitable stress...

• Do you use a version control system for your code? Which?
  
• Do you use a continuous integration system? Which?
• Development processes - none, Scrum, RUP, agile...? Would you say you follow them closely, or do you have you cherry-picked a subset of practices from them that you like/have found to work well for you?
  
• Do you have an internal Wiki? Does anyone use it for project documentation, or do you generally use Word/PowerPoint/Excel files instead?
  
• How do you handle incidents? Do you use a standard process such as ITIL or have you made your own? Either way, how do you integrate incident handling in development? Separate teams, or...? Branch the source often, or...?
  
• How do you get things deployment ready and how do you deploy, are there many manual steps involved?
• How are your enviroments set up, do you have separate environments for development, dev testing, integration, prod-test, production? (Or do they have a variant of the dreaded developmestuction environment?)
  
• Very rough estimate, on a scale from say 1 to 5, how pleased are you with the unit test coverage of your codebase? Ever used any test coverage tools to check?
• Has your team been required to do any all-nighters (or late-evenings) last year because of critical deadlines or major incidents? How many?
  
• Do you use any code-quality checking tools such as Findbugs?
• Are employees allowed to use work time to improve their skills? How do you handle that - do you decide what they should study, or is that up to them? Ever arrange any lectures, study groups or anything like that internally in the company? Are employees encourged to head these, or do you hire external people to lecture?
  

Feedback appreciated. What are your suggested questions?

Update - two good suggestions from David R. MacIver via Twitter:

• Can you give me a coding sample? :-)
• I also like to give them slightly bizarre questions like "Tell me why I wouldn't want to work here".

Delete the obsolete

A week ago I was in the unenviable situation of having to dig through a large number of legacy apps and libraries in Java, C and VB to find references to a database attribute that was going to be retired. Examining just one file took a long time, since a file containing a dangerous reference might have been imported as a helper lib in other projects - and to make things worse, I couldn't immediately see if these projects were obsolete or not.

In my current project, my fellow employees have a very good overview of which systems are used where, and which versions are deployed, so when I am unsure I can always ask them. However, had they quit and taken that knowledge with them, I would have faced a combinatorial explosion of repository source versions, deployed versions, and production environments that would have made my task impossible; not just very hard and time-consuming as it was now.

For clarity and maintainability, if a system is obsolete you should always at the very minimum comment database tables, tag repositories and check in README files to the repositories that clearly show that these things are no longer used. Even better would be to move the data to separate "graveyard" branches/databases. I think the best of all would be to delete the obsolete files, since you can always reconstruct them from the repositories and backups in the extremely unlikely case that you would need them again. I understand if some people are reluctant to do that, the human urge to hang on to things "just in case" is strong.

It is a project management anti-pattern to keep obsolete projects and modules in your source repositories, and obsolete database tables on your servers. You will soon accumulate a technical debt which, like the debts in your bank accounts, will keep growing over time. Keeping track of possible dependencies between systems and components becomes progressively harder, and it will becomes increasingly difficult for new developers to get an overview of the architecture. More and more valuable development time will be spent on "the debt" instead of useful tasks. Developers will not have the courage to refactor, and this will feed the negative habits of adding stuff but never removing - not very agile.

Luckily the managers at my current company have been convinced to set aside developer time during the summer to clean up the source repository and move obsolete code to a separate view. At this particular company, I won't have the tedious task of searching through obsolete code again. For all our sakes, I urge you to do the same. :)

Hudson and ClearCase

If you want your company to be more agile and want to install a Continuous Integration system, but are stuck on Clear Case, here is a little howto. I am normally a Linux/Unix boy, but Windows was mandated.

Download and install a servlet container such as Tomcat. Secondly, download the Hudson war file. You can (should) define a HUDSON_HOME environment variable. Start Tomcat, deploy the Hudson war to the webapps dir, the "jobs" and "plugins" directories should be created in your HUDSON_HOME directory.

Either use the builtin plugin manager page, or if like me you can't get it to work, download the Clear Case plugin manually and put it in the HUDSON_HOME/plugin directory and restart Tomcat.

Go to http://localhost:8080/hudson/ and choose "Manage Hudson" -> "Configure System". Configure the Cleartool executable home (usually c:\program files\rational\clearcase\bin\cleartool), and other tools/services you might use for your projects (Ant, Maven, SMTP server etc)

Select "New job" -> "Build a free-style software project" and select a name, press "ok".

How to configure stuff here depends on how your Clear Case administrator has set up your system, but this is what worked for me.
Select the "UCM ClearCase" radio button and configure the following fields:
View name - select a new unique view name, for instance "hudson_myproject". This is the view that Hudson will create for you.
Stream selector - this is very project dependent, this is how I figured out my stream name: If you have the ClearCase Explorer tool, select "Toolbox" tab -> "UCM" dropdown -> "Join project" link -> select your project -> "Properties" button. Now directly under the project name you see a string somewhat like the following:
project:myproject_1.0@\servername
Then the following should be your stream name:
myproject_1.0@\servername
If you have checked out a project view before with CC Explorer manually to the default drive (usually M:) you can go to M: with Windows Explorer and rightclick on the directory there and select ClearCase tab, then properties. You should be able to get the Stream string from there too.
Load rules - Backslash + the path to a VOB (database file containing the project) just under the root of the project view. Again, if you have previously created a view using ClearCase explorer and mapped it to drive M:, go to M: and open the view directory there. Select one of the subdirs there as your "load rule". For example, I had
M:\myproject\src
so my "load rule" became
\src

Now press "advanced" button in Hudson config, and configure the following:
Additional mkview arguments - Here I had to use the -vws parameter to the mkview command, plus a UNC name. Something like this:
-vws \\companynetwork\clearcaseviewsserver\AD\myusername\
This is where Hudson will create the view directory, but the source itself will be checked out to, and built from, your HUDSON_HOME/jobs/jobname/builds/ directories.