Nordic Ruby was held outside Stockholm at the Hasseludden Yasuragi spa this year, instead of Gothenburg as has been traditional. Hasseludden is a very nice location with great view, good food, and a spa facilities.
Quite a number of talks were non-Ruby or even non-technology related this year, but they were of high quality and didn't feel like filler. Several of my favourite talks were from this section. My top three talks were:
1. Perfectionists Anonymous by Erin O'Brien and Alan Gardner.
Developers work in a fast moving industry with lots of smart people. It is easy to get stressed out by this and set up unrealistic goals for yourself. Fear of failure can cause you to avoid starting something and becomes a source of procrastination. A common trait is forgetting or downplaying all your successes, but remembering all your failures. Perfectionism is correlated with low self esteem, shame, and in difficult cases depression.
The speakers proposed practicing admitting to yourself that failure is ok, and rarely as bad as you imagined it to be. As an exercise some attendees got up on stage to tell the audience about when they had failed - for instance "I have talked about it for years, but I've yet to do a significant contribution the open source community". The speakers recommended trying mediation to help breaking the negative thought spiral of perfectionism, as daily meditation has been shown in trials to reduce stress and improve moods. They encouraged us to use the #devmed hashtag on Twitter to document progress. In more severe cases of perfectionism, such as when you have depression, sleeping problems or other severe or long term problems that don't seem to improve, they recommended considering councelling.
As I do have problems with perfectionism, this was probably one of the most personally helpful talks I've ever attended, and I've started meditating.
2. Therapeutic Refactoring by Katrina Owen
The speaker had a real world example of a single long method of messy opaque Ruby code, and step by step went through a series of refactorings until she was left with a small number of short, clean, well named, easily understood methods. I was reminded of how readable Ruby code is. When I first started learning Ruby, I didn't agree that semi-colons, brackets etc in Java was a big problem. I dismissed it with "After a while you learn to ignore them and they become invisible". But I've realized than when showing slides, Java code must be kept to a very few lines for the audience to be able to read them quickly. The added symbols in Java make parsing easier for computers, but do in fact increase the cognitive load for humans quite a lot.
Apart from an interesting subject, it was a good example of great presentation technique - starting with "a hook" the first minute to engage the audience, a good even pace, well rehearsed, a nice recap and punchline at the end. This was one of the few successful code-heavy talks I've ever seen at a conference.
3. Adventures on the Golden Path by Steve Klabnick
Release 3 of Rails made the framework more modular and cleaned up a lot of technical debt. The speaker has plans for a number of patches that makes the framwork more user friendly and consistent. For instance with routing, currently a several pages long comment is generated in the routing file. Steve proposed having a few lines showing the routing rules sufficient for 95% of all users, and a link to online documentation. A singleton should be provided so that it can be queried, and developers can test which routes will be generated with the console. There was a number of these small changes, but which all together help make Rails friendlier both for beginners and advanced users.
Lightning talks were great fun, two outstanding ones were "Three kinds of respect", and "Go (the game, not the language)" by Joseph Wilk.
All in all, a very good conference, highly recommend you go next year. Smaller conferences can experiement with topics and speakers while big conferences must attract the most famous speakers to attract more attendees. These speakers are of course very good, but they often have a very "introductory" level to their topics since they speak to large crowds and many conferences and can make few assumptions about the listeners previous experience. In smaller conferences I have also found it easier to get to know people - both the speakers and everyone else.
Edit: Pictures and more comments here.
Sunday, September 23, 2012
Crash & Burn 2012
When I was studying computer science we spent a lot of time learning about programming and theory, but from what I can recall almost nothing was mentioned about deployment and administration, which is kind of absurd. What good are programs that can't be run properly? When talking to others, this is something that is common to many developers. This is exacerbated by the habit of some organizations to separate developers far from the sys-admins. The Crash & Burn conference is about the important topics of continuous integration, testing, deployment and virtualization. It was held at KTH Forum, Kista Stockholm, March 2 2012, and this is a summary of my notes from the conference.
Keynote: Why DevOps? by Morten Nielsen from RemoteX
Morten discussed how if someone at the hosting company made a mistake, it would still reflect badly on the developers and their company when the site becomes inaccessable. A few years ago they therefore chose to host the application themselves. This initially gave them increased quality of service, but as the number of versions of the system in production increased, they started feeling the pain of mainting and upgrading the system. Despite a comprehensive manual, deployment took time and was error prone.
The solution was to throw away any deployment manual and automate everything. This reduced the upgrade cost to being 4 minutes, unsupervised. This in turn led to having many releases. This in turn led to developers having more time over for development of new features, increased focus, and increased confidence that they could react quickly to security threats or other emergencies.
Morten especially recommended the book Continuous Delivery.
Designing for Rapid Release by Sam Newman at ThoughtWorks
A lot of time is spent in software projects worrying about requirements, perfomance and compliance. Sam Newman argues that to be successful, more projects should prioritize how easy it is to release. To do this, three things are needed.
He mentioned "Dark Launches" - launching a new version of a service in secret, but not migrating users to it. Instead you play back transactions from the current service to do testing. Facebook were famous for doing this, adding JavaScript hooks that sent all user transactions both to the live site, and the Dark Launched version.
He then described the necessity to degrade quality of service whenever you do an upgrade. This is complex, but it is something you should take into consideration anyway, since it is needed to make the site tolerant to failure by degrading gracefully. He mentioned the Circuit Breaker design pattern, and for further reading recommended the book Patterns of Enterprise Application Architecture by Martin Fowler.
An alternative design is to use asynchronous behaviour - but this is also complex, perhaps too complex for most usages? If using this architecture, it becomes even more important to degrade gracefully, and keep users informed about progress.
Sam then brought up a couple of anti-patterns that make updating and deploying especially difficult. There is one he called "The Trifle". I would have liked an explanation of the choice of name, because the meaning is not obvious to me. From my understanding, he used it do describe the common fallacy of dividing the development of an application into, for instance, "the web layer" and "the persistence layer" and putting teams on working them concurrently. There is then just the little trifle of merging the two. He described a project he had been on where two geographically separate teams developed those tiers, with disastrous results. The problem is that code changes almost always cross boundaries - both tiers need upgrade at the same time. Furthermore, if you divide your application like this, you often end up with a lot of chatter between the services.
Another common antipattern is The Spider - a single central service, which usually grows into a sort of "god object" which then just polls a lot of dumb services in the periphery.
As a rule of thumb, to design services correctly Sam recommended to think of a service as "A set of capabilities at an endpoint" - something an end user would recognize as relevant to them, for instance a "music recommendation engine". You should model your services based on your business domain (see also: DDD).
Whatever your architecture, you should beware of shared serialization protocols. He quoted "Be conservative in what you do, be liberal in what you expect". As an warning example, he mentioned XML binding libraries which, while giving appearent convenience to programmers, tightly couple services to each other. He said that ThoughtWorks almost universally recommend using XPath or similar technologies to consume SOAP services rather than XML-object binding libraries. This is less brittle because domain object changes don't affect the protocol, and if the protocol changes fewer consumer changes are needed.
At this moment in the presentation I had started to think about my own experiences from upgrading applications, and the fact that databases are often a pain point. I had planned to ask serveral questions about how he would mitigate this, but he was ahead of me as the last part of the presentation covered databases and persistance. He offerend the great quote "Data is cool, databases are generally evil". The problem with dbs is that you can't version the schemas easily. There is one terrible anti-pattern that often pop up in shops that try out doing Service Oriented Architecture - the services are decoupled when sending messages, but all too often they all share the same database (because of architectural mistakes, sending all the necessary state is expensive). Sam urged us, "If you only take one thing from this presentation - PLEASE USE SEPARATE DB SCHEMAS FOR YOUR SERVICES".
A great talk, and an obviously experienced presenter with great flow and nice slides.
Glu: Open Source deployment automation platform by Yan Pujante
Yan Pujante is one of the founders of LinkedIn, and one of the core developers of Glu, a Apache licenced library/tool for deployment. At LinkedIn, many developers and admins who experienced the pain of redeployment had developed their own scripts to automate deployment, using various languages. This library grew out of the need to have one, solid way to redeploy. Essentially you need to install the JVM and deploy one Glu agent (written in Groovy) at each node you want to manage. You can then communicate with the agent using a ready web GUI and a REST interface. Then you can deploy, restart services, or even send OS level signals to any live processes. Yan walked through the architecture and the configuration, and ended with a demo. It was certainly interesting and something I want to look more at, but my suggestion would be to start the talk with a short "WOW!" style demo. When you have a technical in-depth presentation, it is important to start strong to arouse interest so that listeners don't start to drift in their attention, or it is very difficult for them to get back. He had prepared slides for it but didn't have time to go into security unfortunately.
The Ops side of Dev with Mårten Gustavsson
Mårten brought up how much of his professional time has been spent maintaining systems and tracking down bugs. This is valuable experience - too often many developers only have experience of working on projects that are cancelled, or are dumped on a maintenance crew while "the next version which will fix everything" is developed. Leonard Axelsson then observed on Twitter - " ever single dev ever should serve in an operations team for at least a few months." Mårten's talk was oriented around how developers can make the operations side easier, no matter who is in charge of it.
Most of the talk revolved around logging, something that is suprisingly hard to do right (or at least most developers are surprisingly poor at it). Some important pointers:
I asked if he used to instrument manually, or use aspect oriented libraries like AspectJ since logging and metrics are classic cross-cutting concerns and often used as examples in aspect tutorials. He said his experience of using declarative aspects was so-so, they tended to be too fine grained and spew too much info into logging, but that he favored using annotations together with something like Google Guice for good control and minimal code impact.
He also mentioned striving for zero-touch configuration of deployment, and preferring using a maving plugin like Shade to produce an über-jar for deployment rather than expecting a server with all runtime dependencies correctly set up and maintained.
DNS in the Spotify Infrastructure with John Stäck from Spotify
Another technically in-depth talk which was interesting, but difficult to take notes on. Some of the things I remember:
Intiutive and distributed load testing with Locust by Carl Byström
Locust is a load testing framwork written in Python. When evaluating existing load testing framworks, Carl decided he wanted to avoid some comon features in his own library:
It did look very impressive, but a little too much time of the presentation was spent talking about companies using Locust. Who they are and what they do is not that interesting - only how they use Locust! I would have preferred if this part of the presentation instead had been spent on discussing clustering of clients and especially test orchestration - how easy is it to start 50 machines sending requests at the same time, and summarizing their results?
Graphite – The village pump of your team by Leonard Axelsson and Ville Svärd
Graphite is an open source tool written in Python. You instrument you code to write to the Carbon component (based on Twisted), which in turn writes to the Whisper (db) component. You then use a Django based GUI to visualize the data. The talk was split into two parts, a demo of how to configure the visualization and the story of how introducing Graphite into a project changed it. Leonard was initially just running it on his local machine. When he was looking through it each morning, other developers and eventually also managers gathered around and asked about it. It eventually became a highly valuable tool for the team.
Continuous Integration – The Good, Bad and Ugly with Brian Riddle
The good:
Brian Riddle works at TV4 and shared his journey of introducing continuous integration using Hudson (now most use the fork called Jenkins) to a very messy CMS project. It is easy to get started, you can download it and start running it locally with just "java -jar jenkins.war", instead of asking for permission to set up a new server. Brian highly recommended the book Working Effectively with Legacy Code by Michael Feathers. They used emma to document the code coverage, and while the percentage of source code increased, "it was a lie". They were dependent on a large number of jsp pages which could fail to compile at runtime, causing redeployments to fail. Eventually they introduced an Ant task which pre-compiled the jsp pages, which at least caught compilation errors ahead of time. But testing them remained problematic, not to mention Flash which was a large component of the site.
Brian's advice is to "Keep deploying until it doesn't hurt". Previously they had scheduled redepoyments every 6 weeks, but they often went wrong. When they started to seriously fix the underlying problems of redeploying, the first major redeploy was an all-nighter before it worked. The second took one redeploy. After that things started working smoothly. Years ago it took over 2 hours to deploy, now they are down to less than 25 minutes unsupervised for the slowest project. Most projects take less than 10 minutes to run all tests and deploy.
They later replaced much of the jsp with Rails, and when using Ruby they could use rcov to test code coversage. They used GitHub to host the source, and had hooks so that changes were notified to the whole team via Yammer. GitHub has a feature where people with the right access rights can edit the code directly through the web site and commit changes. Brian said this was a feature that kept him awake at night at first, but after a while it became a valuable feature, as managers and other non-coders could easily edit FAQs, web pages etc. They could also redeploy it themselves. This meant the correct people (product owners) could decide when redeployment should occur, and developers were not a resource limiting this. Developers also experienced less stress, and could dedicate more time to development.
The Bad:
The CMS can't be deployed whenever. 20 minutes downtime is disastrous if major news break during the downtime. PHP and Wordpress could be problematic to deploy, and especially Wordpress themes introduced by users led to trouble. mule the ESB can hot-redeploy which is nice, but you can't redeploy the whole system often (?), and it is highly problematic to test through it.
Notifications from GitHub, Jenkins and testing systems are critical, but sometimes devs experience notification overload and start to ignore them.
The ugly:
The closer you are to the user, the harder things become to test. Brian mentioned the special problems of testing:
Scaling Github with Zach Holman
There are two problems with scaling - technological and organizational (or human). Zach chose to focus on the latter with his presentation. Happy employees are productive employees, and vice versa, so how do GitHub score excitement and reduce toxicity (i.e. keep their people happy)? They have more or less elimiated meetings. People are not forced to be in the office but are allowed to work from wherever they feel they are most productive, as long as they contribute and keep in touch through their chats.
They have a number of perks (though some of the ones mentioned like dental plans and generous vacation are mandatory in Sweden so those perhaps didn't feel special for us). The employees can spend time on learning new stuff and get paid to talk at conferences (like this one!) They like to hire independent and self-motivating people. Even so they are sure to help them get started as quickly as possible by automating and documenting as much as possible. Internal presentations are recorded (using a neat Arduino+Kinekt hack) and made available for all to view on the intranet.
Hiring poorly is at least as dangerous as losing experienced people, before you know it they are influencing who are hired in turn. How do you find good people to hire? Zach stressed the importance of extending your personal contact networks. You get to know good people by for instance
This was a fantastic presentation, funny, fast moving and great slides. At the Q&A I asked if they had more or less eliminated managers since they didn't have any meetings, and the answer was yes - they had an extremely flat organization even though they are over 60 employees now.
Summary
This was a small conference, but very well worth attending. It was a nice mix of experienced, internationally famous speakers like Sam Newman and Zach Holman and local talent. If this conference returns next year you should try to go, it deservers to grow.
Edit: Links to presentation slides from Brian Riddle available.
Keynote: Why DevOps? by Morten Nielsen from RemoteX
Morten discussed how if someone at the hosting company made a mistake, it would still reflect badly on the developers and their company when the site becomes inaccessable. A few years ago they therefore chose to host the application themselves. This initially gave them increased quality of service, but as the number of versions of the system in production increased, they started feeling the pain of mainting and upgrading the system. Despite a comprehensive manual, deployment took time and was error prone.
The solution was to throw away any deployment manual and automate everything. This reduced the upgrade cost to being 4 minutes, unsupervised. This in turn led to having many releases. This in turn led to developers having more time over for development of new features, increased focus, and increased confidence that they could react quickly to security threats or other emergencies.
Morten especially recommended the book Continuous Delivery.
Designing for Rapid Release by Sam Newman at ThoughtWorks
A lot of time is spent in software projects worrying about requirements, perfomance and compliance. Sam Newman argues that to be successful, more projects should prioritize how easy it is to release. To do this, three things are needed.
- Make it quick to change
- Make it quick to release
- Make it safe to release.
He mentioned "Dark Launches" - launching a new version of a service in secret, but not migrating users to it. Instead you play back transactions from the current service to do testing. Facebook were famous for doing this, adding JavaScript hooks that sent all user transactions both to the live site, and the Dark Launched version.
He then described the necessity to degrade quality of service whenever you do an upgrade. This is complex, but it is something you should take into consideration anyway, since it is needed to make the site tolerant to failure by degrading gracefully. He mentioned the Circuit Breaker design pattern, and for further reading recommended the book Patterns of Enterprise Application Architecture by Martin Fowler.
An alternative design is to use asynchronous behaviour - but this is also complex, perhaps too complex for most usages? If using this architecture, it becomes even more important to degrade gracefully, and keep users informed about progress.
Sam then brought up a couple of anti-patterns that make updating and deploying especially difficult. There is one he called "The Trifle". I would have liked an explanation of the choice of name, because the meaning is not obvious to me. From my understanding, he used it do describe the common fallacy of dividing the development of an application into, for instance, "the web layer" and "the persistence layer" and putting teams on working them concurrently. There is then just the little trifle of merging the two. He described a project he had been on where two geographically separate teams developed those tiers, with disastrous results. The problem is that code changes almost always cross boundaries - both tiers need upgrade at the same time. Furthermore, if you divide your application like this, you often end up with a lot of chatter between the services.
Another common antipattern is The Spider - a single central service, which usually grows into a sort of "god object" which then just polls a lot of dumb services in the periphery.
As a rule of thumb, to design services correctly Sam recommended to think of a service as "A set of capabilities at an endpoint" - something an end user would recognize as relevant to them, for instance a "music recommendation engine". You should model your services based on your business domain (see also: DDD).
Whatever your architecture, you should beware of shared serialization protocols. He quoted "Be conservative in what you do, be liberal in what you expect". As an warning example, he mentioned XML binding libraries which, while giving appearent convenience to programmers, tightly couple services to each other. He said that ThoughtWorks almost universally recommend using XPath or similar technologies to consume SOAP services rather than XML-object binding libraries. This is less brittle because domain object changes don't affect the protocol, and if the protocol changes fewer consumer changes are needed.
At this moment in the presentation I had started to think about my own experiences from upgrading applications, and the fact that databases are often a pain point. I had planned to ask serveral questions about how he would mitigate this, but he was ahead of me as the last part of the presentation covered databases and persistance. He offerend the great quote "Data is cool, databases are generally evil". The problem with dbs is that you can't version the schemas easily. There is one terrible anti-pattern that often pop up in shops that try out doing Service Oriented Architecture - the services are decoupled when sending messages, but all too often they all share the same database (because of architectural mistakes, sending all the necessary state is expensive). Sam urged us, "If you only take one thing from this presentation - PLEASE USE SEPARATE DB SCHEMAS FOR YOUR SERVICES".
A great talk, and an obviously experienced presenter with great flow and nice slides.
Glu: Open Source deployment automation platform by Yan Pujante
Yan Pujante is one of the founders of LinkedIn, and one of the core developers of Glu, a Apache licenced library/tool for deployment. At LinkedIn, many developers and admins who experienced the pain of redeployment had developed their own scripts to automate deployment, using various languages. This library grew out of the need to have one, solid way to redeploy. Essentially you need to install the JVM and deploy one Glu agent (written in Groovy) at each node you want to manage. You can then communicate with the agent using a ready web GUI and a REST interface. Then you can deploy, restart services, or even send OS level signals to any live processes. Yan walked through the architecture and the configuration, and ended with a demo. It was certainly interesting and something I want to look more at, but my suggestion would be to start the talk with a short "WOW!" style demo. When you have a technical in-depth presentation, it is important to start strong to arouse interest so that listeners don't start to drift in their attention, or it is very difficult for them to get back. He had prepared slides for it but didn't have time to go into security unfortunately.
The Ops side of Dev with Mårten Gustavsson
Mårten brought up how much of his professional time has been spent maintaining systems and tracking down bugs. This is valuable experience - too often many developers only have experience of working on projects that are cancelled, or are dumped on a maintenance crew while "the next version which will fix everything" is developed. Leonard Axelsson then observed on Twitter - " ever single dev ever should serve in an operations team for at least a few months." Mårten's talk was oriented around how developers can make the operations side easier, no matter who is in charge of it.
Most of the talk revolved around logging, something that is suprisingly hard to do right (or at least most developers are surprisingly poor at it). Some important pointers:
- Make all developers agree on consistent log levels.
- Have an action plan ready - who responds to each log level?
- Rotation and retention. It is no fun trying to grep through gigabyte size files (or even worse, try to open them in a gui). It is better to have more and smaller files. Put an upper size on them. Compress older files as they become less relevant.
- Spend some time on formatting - make the log files easy on eyes and tools.
- Destinations - you should definitely consider having multiple logger outputs. If you have clusters of services, you want to correlate logging between them, not have to ssh to dozens of servers in turn to locate the one server where the customer problem occured. You probably also want error/critical logging pings to go to for instance IRC or XMPP services. Remember the fallacies of distributed computing though, you can't assume the network or the service is alway available. Always, always have fallback logging to a local file or error information critical to debugging may be lost.
- Logging and other services should be reconfigurable in runtime. You don't want to take your services down in order to increase log levels after the fact. Many JVM frameworks have good logging and the ability to be instrumented through JMX, but you have to remember to enable them in configuration before you start the server.
I asked if he used to instrument manually, or use aspect oriented libraries like AspectJ since logging and metrics are classic cross-cutting concerns and often used as examples in aspect tutorials. He said his experience of using declarative aspects was so-so, they tended to be too fine grained and spew too much info into logging, but that he favored using annotations together with something like Google Guice for good control and minimal code impact.
He also mentioned striving for zero-touch configuration of deployment, and preferring using a maving plugin like Shade to produce an über-jar for deployment rather than expecting a server with all runtime dependencies correctly set up and maintained.
DNS in the Spotify Infrastructure with John Stäck from Spotify
Another technically in-depth talk which was interesting, but difficult to take notes on. Some of the things I remember:
- "You would think DNS resolver libraries are clever. Most of them are not". He mentioned some of them cashing infinitely, requiring a server restart when they had changed DNS configurations. But they had managed to eliminate almost all those problematic servers.
- He mentioned using Geo DNS to route users to servers geographically near them, but that the db for Geo DNS was bad and getting worse all the time, as the shrinking IP4 address space means fragmentation. Also big domains such as Google give strange results. His end recommendation was that you should "Very carefully consider if you really need Geo DNS".
- DNS can handle great loads, don't be afraid to utilize it.
- When it came to deployment, his advice was "Eliminate humans, they are not to be trusted".
Intiutive and distributed load testing with Locust by Carl Byström
Locust is a load testing framwork written in Python. When evaluating existing load testing framworks, Carl decided he wanted to avoid some comon features in his own library:
- Using a GUI to define tests. Programming through a GUI is a pain (something I totally agree with).
- Declarative. (This surprised me, since declarative is usually considered a good thing. It seems he meant XML and similar test definition formats that are ONLY declarative and are not easy to parameterize or mix with code).
- Expensive scaling.
- Configuration using POPC - Plain Old Python Code
- Small and hackable source
- Distributed & scalable
- An intuitive web GUI to monitor and control test runs.
It did look very impressive, but a little too much time of the presentation was spent talking about companies using Locust. Who they are and what they do is not that interesting - only how they use Locust! I would have preferred if this part of the presentation instead had been spent on discussing clustering of clients and especially test orchestration - how easy is it to start 50 machines sending requests at the same time, and summarizing their results?
Graphite – The village pump of your team by Leonard Axelsson and Ville Svärd
Graphite is an open source tool written in Python. You instrument you code to write to the Carbon component (based on Twisted), which in turn writes to the Whisper (db) component. You then use a Django based GUI to visualize the data. The talk was split into two parts, a demo of how to configure the visualization and the story of how introducing Graphite into a project changed it. Leonard was initially just running it on his local machine. When he was looking through it each morning, other developers and eventually also managers gathered around and asked about it. It eventually became a highly valuable tool for the team.
Continuous Integration – The Good, Bad and Ugly with Brian Riddle
The good:
Brian Riddle works at TV4 and shared his journey of introducing continuous integration using Hudson (now most use the fork called Jenkins) to a very messy CMS project. It is easy to get started, you can download it and start running it locally with just "java -jar jenkins.war", instead of asking for permission to set up a new server. Brian highly recommended the book Working Effectively with Legacy Code by Michael Feathers. They used emma to document the code coverage, and while the percentage of source code increased, "it was a lie". They were dependent on a large number of jsp pages which could fail to compile at runtime, causing redeployments to fail. Eventually they introduced an Ant task which pre-compiled the jsp pages, which at least caught compilation errors ahead of time. But testing them remained problematic, not to mention Flash which was a large component of the site.
Brian's advice is to "Keep deploying until it doesn't hurt". Previously they had scheduled redepoyments every 6 weeks, but they often went wrong. When they started to seriously fix the underlying problems of redeploying, the first major redeploy was an all-nighter before it worked. The second took one redeploy. After that things started working smoothly. Years ago it took over 2 hours to deploy, now they are down to less than 25 minutes unsupervised for the slowest project. Most projects take less than 10 minutes to run all tests and deploy.
They later replaced much of the jsp with Rails, and when using Ruby they could use rcov to test code coversage. They used GitHub to host the source, and had hooks so that changes were notified to the whole team via Yammer. GitHub has a feature where people with the right access rights can edit the code directly through the web site and commit changes. Brian said this was a feature that kept him awake at night at first, but after a while it became a valuable feature, as managers and other non-coders could easily edit FAQs, web pages etc. They could also redeploy it themselves. This meant the correct people (product owners) could decide when redeployment should occur, and developers were not a resource limiting this. Developers also experienced less stress, and could dedicate more time to development.
The Bad:
The CMS can't be deployed whenever. 20 minutes downtime is disastrous if major news break during the downtime. PHP and Wordpress could be problematic to deploy, and especially Wordpress themes introduced by users led to trouble. mule the ESB can hot-redeploy which is nice, but you can't redeploy the whole system often (?), and it is highly problematic to test through it.
Notifications from GitHub, Jenkins and testing systems are critical, but sometimes devs experience notification overload and start to ignore them.
The ugly:
The closer you are to the user, the harder things become to test. Brian mentioned the special problems of testing:
- Firefox, IE, Chrome and Safari and all their versions. You can test manually, but that is highly time consuming. Most test framworks running a browser and scraping pages for tests are brittle, and work with few browsers.
- qt-webkit
- Flash. A nightmare to test, especially though Jenkins. The Flash plugin only works on 32 bit Linux, and since Jenkins and all their other servers run on Linux, they either have to downgrade just for Flash testing or else choose not to test.
Scaling Github with Zach Holman
There are two problems with scaling - technological and organizational (or human). Zach chose to focus on the latter with his presentation. Happy employees are productive employees, and vice versa, so how do GitHub score excitement and reduce toxicity (i.e. keep their people happy)? They have more or less elimiated meetings. People are not forced to be in the office but are allowed to work from wherever they feel they are most productive, as long as they contribute and keep in touch through their chats.
They have a number of perks (though some of the ones mentioned like dental plans and generous vacation are mandatory in Sweden so those perhaps didn't feel special for us). The employees can spend time on learning new stuff and get paid to talk at conferences (like this one!) They like to hire independent and self-motivating people. Even so they are sure to help them get started as quickly as possible by automating and documenting as much as possible. Internal presentations are recorded (using a neat Arduino+Kinekt hack) and made available for all to view on the intranet.
Hiring poorly is at least as dangerous as losing experienced people, before you know it they are influencing who are hired in turn. How do you find good people to hire? Zach stressed the importance of extending your personal contact networks. You get to know good people by for instance
- Contributing to Open Source
- Arranging conferences
- Blogging and other tech posts
- Sponsorships
- Hack nights and other smaller meetups
- Talking at conferences
This was a fantastic presentation, funny, fast moving and great slides. At the Q&A I asked if they had more or less eliminated managers since they didn't have any meetings, and the answer was yes - they had an extremely flat organization even though they are over 60 employees now.
Summary
This was a small conference, but very well worth attending. It was a nice mix of experienced, internationally famous speakers like Sam Newman and Zach Holman and local talent. If this conference returns next year you should try to go, it deservers to grow.
Edit: Links to presentation slides from Brian Riddle available.
Friday, April 15, 2011
Simple SOAP client
I needed a quick and simple SOAP client to verify some odd return values from a web service. I also wanted the client to be able to send and receive badly formatted XML for testing purposes, and preferably pretty print the xml. Didn't feel very appealing to try to do this in Java, so I decided to do it with JRuby.
Ruby already has the soap4r library, which given a wsdl url gives you convenient objects for performing requests. Unfortunately soap4r tries to retrieve the wsdl with a http GET, and the service is configured to only respond to POST (yes, even for wsdl request). So I decided to go directly to the HTTPClient. Here it is, including pretty printing the output, if someone else finds it useful.
There are some more sample usages of HTTPClient here, the class documentation is here.
If you have a valid xml reply and want to pretty print it, add:
Ruby already has the soap4r library, which given a wsdl url gives you convenient objects for performing requests. Unfortunately soap4r tries to retrieve the wsdl with a http GET, and the service is configured to only respond to POST (yes, even for wsdl request). So I decided to go directly to the HTTPClient. Here it is, including pretty printing the output, if someone else finds it useful.
require 'rubygems'
require 'httpclient'
soapbody = <<-eos
<soap:Envelope>
** your soapy stuff goes here **
</soap:Envelope>
eos
header = {'Content-Type' => 'application/soap+xml'}
endpoint = "http://localhost:8080/testService"
client = HTTPClient.new
#Uncomment next line if you want to see requests and responses as sent.
#client.debug_dev=STDOUT
result = client.post(endpoint, soapbody, header)
puts '== Content'
print result.body
There are some more sample usages of HTTPClient here, the class documentation is here.
If you have a valid xml reply and want to pretty print it, add:
require "jrexml"
require "rexml/document"
doc = REXML::Document.new result.body
doc.write( STDOUT, 3 )
Sunday, October 17, 2010
Unconference at Mejsla
We've had another unconference day, this time at Mejsla and arranged by yours truly. Participants were me, Sven Johansson, Thomas Podal, Fredrik Rubensson, Mårten Gustafson and Henrik Engström.
Topics
Topics explored during the day included:
Blocks
A slow wireless network and the time it took to get to a "Hello World" state usually meant that we only had time to do a quick spike for each topic. The day was still very interesting, and we learned enough to see what might be worth to pursue further, but for the next unconference I think we will create a Wiki or similar so we can collect topic suggestions in advance. Then we can see which topics have the most interest in, and people who are sure they are going to spend time on a topic can download tools and libraries in advance. That way they can make sure they have passed the 1st compile and "hello world" steps, and maybe even have prepared a few questions or demos if they feel like it.
Lift
Two topics I participated in but which were examples of these "high overhead" tasks, were node.js and Lift. For Lift, I started downloading and installing the Scala language, but then discovered that the easiest way to get started was instead to download the sbt - the Simple Build Tool jar from Google, create a shell script on path to execute the jar, and then run
node.js
A lot of the time was spent on building node.js itself with "make", so we didn't get very far. But at least we got the "asynchronous web server in 10 lines" up and running. Again I was impressed with IntelliJ though, which had nice support for JavaScript editing. If I'm going to continue using fully featured IDEs, I feel I really should learn to use IntelliJ, and purchase it.
Mirah
Mirah is a very interesting language. We got it up and running very quickly (though if we hadn't had a working JRuby and Git set up it might have been a different story). Syntactically the language resembles Ruby very much, but it turns out it is actually statically typed and compiled. The types are generally inferred (like in Scala) for uncluttered code and a minimal amount of typing. We only created a few Java classes which can be imported and used in Java projects, but apparently you should be able to plug in different compiler backends so that you can emit... anything you may like. Source for other languages, Dalvik bytecode, or whatever. The mappings to Java bytecode seemed to work fine, we only had trouble with creating a "void" method. I'm going to experiment a lot more with this language, expect blog posts and hopefully a lightning speech together with Peter Lind at JFokus 2011.
Guava
This was a revelation for me. I had heard it mentioned in passing before, on a presentation by Enno Runne for instance, but it was one in an ever growing list of "things I ought to spend some time on". It turned out that these libraries are something that will help me enormously in my day-to-day Java programming:
Ruby, Rails and Heroku
A quick introduction to the Ruby language to those who were unfamiliar with it. We then generated a Rails project and I explained the structure of it, what could be done with the scripts, and so on. We then went on to what we thought would be the meat of the session, Heroku. Only, it took less than 5 minutes to integrate Git and Heroku. Pushing changes to Git made Heroku update, autodetect that it was a Rails project, download dependencies and start up an instance of our application in their cloud. Jaw-droppingly simple, as is often the case with Ruby. We then went back and did some more Ruby and Rails programming. Sven, whom I've had a hard time convincing about the excellence of Ruby, seemed to be on the way to becoming a convert.
The future, and books about it
Towards the end of the day we were mentally exhausted and instead of doing another programming session we sat together and talked about good books, fictional and not. The only titles I remember were Kent Beck's Implementation Patterns, The Human Body 2.0, Charles Stross' Accelerando, and Vernor Vinge's Rainbow's End.
What we didn't have time for
Topics that remained on the board, but which we hopefully will be able to do on another day:
Summary
Big thanks to all the participants for making this an excellent day, it was a privilege to learn from and with you.
Topics
Topics explored during the day included:
- the Erlang based key-value store Riak
- the cross-browser WebSocket Socket.IO
- the Scala based web framework Lift,
- JavaScript as a backend language, and especially its asynchronous node.js framework
- the graph database Neo4J
- Google's Guava libraries
- the JVM based Lisp-like Clojure
- getting started with Ruby + Rails + Heroku
- the experimental Mirah language
- JQuery.
Blocks
A slow wireless network and the time it took to get to a "Hello World" state usually meant that we only had time to do a quick spike for each topic. The day was still very interesting, and we learned enough to see what might be worth to pursue further, but for the next unconference I think we will create a Wiki or similar so we can collect topic suggestions in advance. Then we can see which topics have the most interest in, and people who are sure they are going to spend time on a topic can download tools and libraries in advance. That way they can make sure they have passed the 1st compile and "hello world" steps, and maybe even have prepared a few questions or demos if they feel like it.
Lift
Two topics I participated in but which were examples of these "high overhead" tasks, were node.js and Lift. For Lift, I started downloading and installing the Scala language, but then discovered that the easiest way to get started was instead to download the sbt - the Simple Build Tool jar from Google, create a shell script on path to execute the jar, and then run
sbt update ~jetty-run
node.js
A lot of the time was spent on building node.js itself with "make", so we didn't get very far. But at least we got the "asynchronous web server in 10 lines" up and running. Again I was impressed with IntelliJ though, which had nice support for JavaScript editing. If I'm going to continue using fully featured IDEs, I feel I really should learn to use IntelliJ, and purchase it.
Mirah
Mirah is a very interesting language. We got it up and running very quickly (though if we hadn't had a working JRuby and Git set up it might have been a different story). Syntactically the language resembles Ruby very much, but it turns out it is actually statically typed and compiled. The types are generally inferred (like in Scala) for uncluttered code and a minimal amount of typing. We only created a few Java classes which can be imported and used in Java projects, but apparently you should be able to plug in different compiler backends so that you can emit... anything you may like. Source for other languages, Dalvik bytecode, or whatever. The mappings to Java bytecode seemed to work fine, we only had trouble with creating a "void" method. I'm going to experiment a lot more with this language, expect blog posts and hopefully a lightning speech together with Peter Lind at JFokus 2011.
Guava
This was a revelation for me. I had heard it mentioned in passing before, on a presentation by Enno Runne for instance, but it was one in an ever growing list of "things I ought to spend some time on". It turned out that these libraries are something that will help me enormously in my day-to-day Java programming:
- A @VisibleForTesting annotation is helpful for documentation, as I very often relax visibilities on helper methods to enable unit testing on them, this will eliminate a lot of boilerplate javadocs.
- Classes that add functional style methods for manipulating collections (given the syntactical and semantical restrictions of the Java language and platform of course).
- Helper methods for primitives and arrays, which will be extremely helpful for me to clean up a library which is written in C style and does nothing but manipulate bytes and byte arrays.
- Additions to the java.util.concurrent package, including a ListenableFuture that you can chain together for asynchronous workflows.
Ruby, Rails and Heroku
A quick introduction to the Ruby language to those who were unfamiliar with it. We then generated a Rails project and I explained the structure of it, what could be done with the scripts, and so on. We then went on to what we thought would be the meat of the session, Heroku. Only, it took less than 5 minutes to integrate Git and Heroku. Pushing changes to Git made Heroku update, autodetect that it was a Rails project, download dependencies and start up an instance of our application in their cloud. Jaw-droppingly simple, as is often the case with Ruby. We then went back and did some more Ruby and Rails programming. Sven, whom I've had a hard time convincing about the excellence of Ruby, seemed to be on the way to becoming a convert.
The future, and books about it
Towards the end of the day we were mentally exhausted and instead of doing another programming session we sat together and talked about good books, fictional and not. The only titles I remember were Kent Beck's Implementation Patterns, The Human Body 2.0, Charles Stross' Accelerando, and Vernor Vinge's Rainbow's End.
What we didn't have time for
Topics that remained on the board, but which we hopefully will be able to do on another day:
- Testing Java with Ruby's excellent testing tools through JTestR
- Ruby web framework Sinatra + HAML markup.
- Getting started with Python
- Doing the Software Craftsmanship Ruby refactoring excercises from SC 2010
- Amazon's EC2 cloud
- Massive scaling with MOM, especially ActiveMQ
- Getting started with event-driven design and Actors with Akka
- HTML 5
- Devops and OpsCode Chef
- Java 2.0 discussion. (Presumably means - next Java version).
- Cucumber and Cuke4Duke
Summary
Big thanks to all the participants for making this an excellent day, it was a privilege to learn from and with you.
Sunday, June 20, 2010
Unconference
Unconference day
A few weeks ago I was at a learning event arranged by Måns Sandström from Adaptiv together with Ola Ellnestam from Agical. It was called "Learning the learned". The people who arranged it had noticed that senior developers have difficulty finding worthwhile learning events. Courses are often very expensive and is often on such a basic level that any of the participants could have been in charge of the course themselves if given a day to prepare. Conference presentations, while inspiring, are often too short to go into any greater depth, and have to take into consideration that many in the audience may be juniors.
So the solution was to have the senior developers teach each others. The format was inspired by the "unconference" or "Open Spaces" concepts that you may have seen from conferences. In other words, instead of having one person in charge and many passive listeners, people are encouraged to self organize in groups around topics that interest them and then discuss, pair program, draw on whiteboards, or whatever they can come up with. Måns and Ola invited a number of people who they thought had valuable things to contribute to such a day. After a few unfortunate last minute cancellations we were 7 participants at the start of the day.
At the beginning of the day, each participant wrote down a short sentence on a sticky note for every topic he* found interesting and wanted to explore. We put the notes up on a whiteboard in turn and explained a little more in depth what each topic was about. We then wrote our names on every note that sounded interesting. Then we picked two notes with the highest number of votes and formed two groups. One group wanted to explore "Testing frameworks for functional languages", and my group wanted to discuss "Unit testing beyond 'assertEquals'". We discussed how it feels like there are often two sources of non-DRY code when writing tests in Java - test data setup, and secondly the assertion lines themselves (so yes, pretty much all of it, unfortunately). For test data setup we quickly discussed Object Mother and Data Builder patterns, and mock frameworks like Mockito. For the assertions parts, I learned for the first time about Matchers (hamcrest to be specific) and the assertThat method. The combination of the two creates very readable error messages and tests that can be quickly and cleanly written by combining predicating matchers. At the end of the session we did a quick recap of what we had learned for the other group. As people came up with new topic ideas, new sticky notes went up on the board during the day.
Second and third sessions I spent doing the first few exercises of the excellent 15 exercises to know a programming language using Clojure, together with Ola Ellnestam and Peter Hultgren who has since founded the Stockholm Clojure User Group. At the final session of the day people were pretty mentally exhausted, and instead of programming we all did a session together where we discussed "Books every developer should read" - programming related and not.
In between sessions Ola was in charge of short exercises that were either medition and focusing exercises, or little games designed to make the participants get to know each other better. These were great fun, and helped greately to keep concentration up.
For me this was a fantastic day. I learned much more than I have at the conferences I usually attend, plus I got to know new interesting people. Before going I was a bit nervous - the topics I had come up with felt "too easy" - stuff a senior programmer should already know. But considering how expansive our field is (and growing every day), no one can be an expert at everything, so this was really something I needn't have worried about (and neither should you, if you decide to go to something similar).
I think that if I could go to something like this once a month, it would accelerate my learning and my career enormously. Hopefully this will become a regular event, the participants all agreed we should try to do it at least once every couple of months. And there is nothing stopping you from arranging a similar event, the concept is not trademarked or anything (though the people who arranged this day say mentioning them as a source of inspiration is appreciated).
* Only guys present this time unfortunately. Sad that gender imbalance in our sector is actually growing! But that is a topic for another post...
Tuesday, March 23, 2010
Load log4j.properties from jar file in webapp
Having environment specific log4j.properties inside a jar file is hardly an ideal solution, but if you find yourself in a similar situation:
Also remember to add the servlet to web.xml with load-on-startup.
import java.net.URL;
import org.apache.log4j.PropertyConfigurator;
import javax.servlet.http.HttpServlet;
public class Log4jConfLoad extends HttpServlet {
@Override
public void init() {
try {
String prefix = "jar:file:" + getServletContext().getRealPath("/");
String path = "WEB-INF/lib/the.jar!/" + System.getProperty("env") + "/log4j.properties";
PropertyConfigurator.configure(new URL(prefix+path));
catch (Exception ex) {
//Because Servlet init can't be declared as throwing new exception.
//Any thrown Exception here should show up in catalina.out (if Tomcat used)
throw new RuntimeException(ex);
}
}
}
Also remember to add the servlet to web.xml with load-on-startup.
Wednesday, December 23, 2009
"That" Java example and preaching to the converted
There is one example of Java code that keeps coming up. I have seen variants of it at conference presentations several times, in Venkat Subramaniam's Programming Groovy and most recently in an essay by Neal Ford in The ThoughtWorks Anthology. This example compares Java with a scripting language, and goes usually something like this (here I combine all the worst practices from the books in one example):
JAVA VERSION:
GROOVY VERSION:
Wow, 1 line instead of almost 50! Java must really suck! The problem with this is that the Groovy version is obviously just a one-off, there is no attempt to make a reusable module or do robust error handling. If there is an error, the program just terminates with en error message. We can do exactly the same with the Java version and cut down the number of lines significantly if we just throw the exception. We can reduce the number of lines even more if we use a "modern" (1.5) API and the Java coding conventions for where to place brackets, etc:
It can be made a one or two lines shorter still, (at the price of a dependency on a big jar) if you are are willing to use an external library such as Apache Commons IO. Or I could have squeezed the three lines of logic into one, but I think this is a reasonable trade off between readability and verbosity. But this new and improved example is still much longer than the Ruby/Groovy/Scala/etc versions (especially if you count characters instead of lines); and arguably less clear (What's a scanner? What does "\\z" mean?).
The problem with using the original Java example is that will mainly convince those who are firmly in the "Yeah, Java totally sucks" camp. Even a very junior Java developer, who is curious about learning about a new language will see that the original example is either
a) Written by a rank amateur (unlikely considering the caliber of the authors) Or -
b) Wildly exaggerated to make Java look as bad as possible.
And that will make them less receptive to the text as a whole. With the number of active Java programmers probably (still) in the hundreds of thousands, alternative languages have a large number of possible recruits and allies. I would prefer to see a better example to convince them that it is worthwhile to look at other languages.
JAVA VERSION:
import java.io.*;
public class ReadFile
{
public ReadFile(String path)
{
BufferedReader reader = null;
try
{
reader = new BufferedReader(
new FileReader(path));
String line = null;
while((line = reader.readLine()) != null)
{
System.out.println(line);
}
}
catch(FileNotFoundException ex)
{
ex.printStackTrace();
}
catch(IOException ex)
{
ex.printStackTrace();
}
finally
{
try
{
if (reader!=null)
{
reader.close();
}
}
catch (IOException ex)
{
//Ignore
}
}
}
public static void main(String[] args)
{
new ReadFile("thoreau.txt");
}
}
GROOVY VERSION:
println new File('thoreau.txt').text
Wow, 1 line instead of almost 50! Java must really suck! The problem with this is that the Groovy version is obviously just a one-off, there is no attempt to make a reusable module or do robust error handling. If there is an error, the program just terminates with en error message. We can do exactly the same with the Java version and cut down the number of lines significantly if we just throw the exception. We can reduce the number of lines even more if we use a "modern" (1.5) API and the Java coding conventions for where to place brackets, etc:
import java.io.File;
import java.util.Scanner;
public class ReadFile {
public static void main(String[] args) throws Exception {
File in = new File("thoreu.txt");
String contents = new Scanner(in).useDelimiter("\\Z").next();
System.out.println(contents);
}
}
It can be made a one or two lines shorter still, (at the price of a dependency on a big jar) if you are are willing to use an external library such as Apache Commons IO. Or I could have squeezed the three lines of logic into one, but I think this is a reasonable trade off between readability and verbosity. But this new and improved example is still much longer than the Ruby/Groovy/Scala/etc versions (especially if you count characters instead of lines); and arguably less clear (What's a scanner? What does "\\z" mean?).
The problem with using the original Java example is that will mainly convince those who are firmly in the "Yeah, Java totally sucks" camp. Even a very junior Java developer, who is curious about learning about a new language will see that the original example is either
a) Written by a rank amateur (unlikely considering the caliber of the authors) Or -
b) Wildly exaggerated to make Java look as bad as possible.
And that will make them less receptive to the text as a whole. With the number of active Java programmers probably (still) in the hundreds of thousands, alternative languages have a large number of possible recruits and allies. I would prefer to see a better example to convince them that it is worthwhile to look at other languages.
Subscribe to:
Posts (Atom)
