From Scrum to Kanban, doing the legwork

May 7, 2012 § 2 Comments

I’ve enjoyed participating in the talks at Scrum Gathering 2012 in Atlanta. One of the most interesting was Steve Forte’s presentation on Kanban 101. I’ve been exploring Kanban as an alternative to Scrum at AtTask for a while now, but with our adoption of Continuous Delivery and the recent rollout of Feature Toggles, we may be ready to take the plunge.

AtTask has been an Agile shop for several years now… really since we had a team of appreciable size to benefit from the principles. We adopted Scrum as our methodology of choice, primarily because it along with XP was about the only thing that matched our legacy delivery model. While we’re still delivering features into production every 4 weeks, our CI pipeline has piqued the interest of folks within our organization who would like to see even more rapid build-test-learn feedback loops.

Kanban evolved from the Toyota production system, as a model for developing cars more quickly, while minimizing inventory and emphasizing just-in-time delivery. When it comes to software, Kanban is a model that eschews the traditionally time-boxed Sprint metaphor in favor of a pull-based system. Features are sized, and move through phases of production which are carefully limited in order to eliminate inventory (unreleased code) and maximize flow. With Feature Toggles, we’ve gained the ability to deploy virtually anything without affecting the customer experience unintentionally. Because releases are now managed by the business, we can create a Kanban flow which will take us from story conception all the way to deployment.

I think we may have a good first start with our customer bug-fixing team, since their work is inherently request driven, rather than time-boxed. Over time I’d like to see Kanban principles adopted more broadly across AtTask.

Continuously Integrated with Selenium

May 2, 2012 § Leave a comment

The video from my talk at Selenium Conference 2012 is now online. I discuss the challenges and opportunities in using cloud infrastructure to run large Selenium suites every check-in. Here’s the abstract:

Most engineering organizations include some level of continuous integration in their development process. The brutal truth is that far too often these efforts are plagued by unreliable tests, long feedback loops, and bad configuration management. Learn how AtTask decided to radically rethink their software development model, and by using open source and cloud solutions went from 3 days of acceptance testing to just minutes, providing feedback on thousands of selenium tests to every engineer in the organization. See how by leveraging publicly available tools, you can deliver hyper-scalability to your Continuous Integration framework and include selenium testing per commit to drive cycle time down in your organization.

Feature Toggling our way to a better future

May 1, 2012 § Leave a comment

Today we released the first Phase of a major new user interface for AtTask. This was the culmination of the hard work over several teams, over a period of several months, during which we transitioned from monthly to daily releases, hired entire teams of engineers, and created an entirely new deployment pipeline. Needless to say it was a great relief to see the system finally deployed, and released into the eager arms of our users.

With a change this large, one might be led to believe that we inherited a high degree of risk. The truth is that this release was particularly low risk, compared to previous releases of our product. One of the reasons was our adoption of a practice that is becoming more common in the Continuous Delivery community, called “Feature Toggling“.

A Feature Toggle is a relatively simple in concept, but devious in execution. At it’s core, it simply means that everything which is coded that has exposure to end users, must be access-controlled through configuration. When you control access to each developed feature by configuration, you decouple the deployment of your changes from release to customers. The process of releasing software then becomes stage-gated: you first deploy, monitor, and learn from a baseline that excludes all customer facing change. Then you release. The release of the software becomes a business decision, not a technology decision. Companies like Facebook and Etsy use this technology every day to split-test and validate features in live production environments.

The reason Feature Toggles are devious in execution is because the implementation path is highly dependent on the flexibility of the application stack to absorb this model. This means that Feature Toggling is more of a design pattern than a product that can be packaged and sold to any development org. In fact, many companies who have developed a Feature Toggling framework in-house will come up with a clever name for their creation. We call ours “PitBoss.”

There are two primary ways that PitBoss differentiates itself from more typical configuration tools. Instead of choosing a simple “Yes” or “No” for each feature, PitBoss gives us the ability to evaluate an expression. We can look at the specific user logging in, and control availability based on their profile, group, company, or any other value we want. For example, if we want to release a new feature to just our beta-test customers, we could create an expression like:

“pitboss.isEnabled(user.beta == true)”

This is evaluated once when the user logs into the system, and drives the availability of features in the product. These are all configurable while the system is up and running, so features that are intended for a single group, or which we discover to be unstable, can be deactivated just as easily.

The second way we’ve differentiated PitBoss is by how we expose it in different layers of our application. In java we can make calls directly into PitBoss methods, but we also have the ability to control access to tiles through PitBoss annotations, or to reference it using JSTL syntax in our JSPs. PitBoss can be accessed through an internal REST api for controlling features in client applications. We have even implemented a mechanism for test cases that allows us to override a particular PitBoss default, for easy A/B testing of features behind configuration.

I mentioned that we released the first phase of our new user experience today, but the code was actually deployed days ago. We’ve had an opportunity to test it by activating the release for a specific set of users, and smoke testing in a live production environment for the first time.

I’m excited by the possibilities of this great new tool, and would love your comments about similar systems you’ve found success with in your organizations!

Cloud Scale CI with Jenkins

April 27, 2012 § Leave a comment

I’m sitting on the tarmac at JFK, thinking about the next time I’ll be in town. The sponsors of Jenkins User Confence NYC have invited me to speak on Continuous Integration in the Cloud. This talk will be similar to what we shared in Armenia, but with a crowd of hard-core Jenkins hackers, I think we’ll enjoy diving into the weeds a little more.
One of the first things we did after forming a tiger team to tackle the CI problem, was evaluate the vendors in the space both commercial and open-source. We had been using TeamCity from Jetbrains because of its deep integration with IntelliJ (we are a java shop). TeamCity has a lot going for it: a slick UI, great claim/blame functionality, and a built in remote run feature which essentially tests every diff against a shadow branch before merging to mainline. Jenkins was adopted in our QA organization, but was used primarily as a job automation system. It wasn’t connected to our version control, and was running on a single virtual machine.
Comparing these two tools was tough. We could keep what we had been using and begin the process of building out integrations to stand up our vision of a cloud-scale CI pipeline, or we could switch to Jenkins, absorb the crappy UI, and leverage the efforts of the community to give us a head start.
In the end, we went with Jenkins, and I’m glad we did. Yes, we ended up having to write a few plugins ourselves, but the initial hacking required to connect with our chosen cloud vendor, Amazon EC2, both it’s single instance APIs and cloud formation endpoints, was largely finished by folks in the community. Jenkins still stinks at visualization, but we have created views which have increased adoption in our org, and which we are now sharing with the community.
I’m looking forward to joining the Jenkins guys at their conference in NYC, stay tuned for slides, photos, and other content from the show on May 17th!

On the Continuous Integration world tour

April 26, 2012 § Leave a comment

We’ve finished up our visit to AtTask’s offices in Armenia. It will be a long flight home, but it was great to get some face time with the managers and leads there. As part of this trip, I spoke at the American University of Yerevan to a group of students, project managers, and engineers. The talk was on Massively Continuous Integration, the practice of applying scaling principles to your CI system using the Cloud. Definitely the most purple room in which I’ve spoken.

Massively Continuous Integration: From 3 days to 30 minutes

Honestly, the team did a great job putting on the whole event. We had over 100 people there, with lots of participation both before and after. CI principles aren’t as commonly adopted as I had expected.

The engineering program at this university provides an accredited masters degree. Many of the engineers in our Armenian office are graduates, and we’re looking into the possibility of beginning an internship program as well. I’ve found that with the right mentoring, the delta on productivity between fresh engineers and those with a decade or more of experience can be quite small. It will be interesting to see what can be accomplished in Yerevan in the months ahead.

The obligatory blog kick-off

April 26, 2012 § Leave a comment

Well, it’s official. I’m finally leaving the ranks of the un-contributing masses and joining the web 2.0 bandwagon. I thought it was a fad, “this blog thing is overrated.” Unfortunately for me, the world of technology moves at blazing speed, and like other professionals in the software world, there’s just too much going on not to talk about it.

I currently work as a Manager of Development for AtTask Inc. My intent is to share my experiences managing the process, organization, and technology as they unfold. I’ll be obfuscating personal details, but I hope to provide a clear window into the world of the modern software team, and in the process perhaps we can discover together the next “big” thing in the software delivery world. We are an agile company, building a project management tool, so I expect that my commentary will tend to focus on the process of developing software more regularly than the intricacies of different coding techniques.

Right now, we’re in the midst of a massive change: from delivering our product once per month to a Continuous Delivery model. We’re web software after all… why not take advantage of our delivery platform and ship more value to our customers more often? I’m traveling around the world speaking on these topics: last week in London at SeConf 2012, this week in Yerevan, and next week in Atlanta at The Scrum Gathering. I’ll certainly share these events with you here on the blog, as well as through my twitter feed at @dowdlemj, and on

If you’ve made it this far, hopefully you’ll come back soon, as I begin to pen my musings on this new-fangled world wide web…thing.