Hold Core Beliefs -- Part II

In my previous post, "Hold Core Beliefs -- Act on Them", I outlined three core beliefs we act on at my organization.  

• Always be delivering
• Not All Builds are Created Equal
• Consume Your Own Output

This post will will cover the second -- Not all Builds are Created Equal

Build Types

When you make every one of your builds consumable, you open yourself to all kinds of feedback.    Several years ago, I was a bit naive on this topic.   When reviewing my own team's automated test results with one of my consumers, I tried to explain that if ruled out 75% of the builds we produced, I was assisting his team in which builds they should consume.   His response (assuming he was looking at build results instead of automated test results), was, "This is terrible.    I have worked on complex systems before and it built more often than this."   Another consumer said, "I don't understand why each build isn't perfect". 

It was at this time I began studying the project organization and development operations at Eclipse.    They have a nice summary of the different types of builds that they attempt to produce.  You can see the entire list at Eclipse.org, but  In a nutshell they range from:

• Release -- Releases are the right builds for people who want to be on a stable, tested release, and don't need the latest greatest features and improvements
• Stable - builds that have been found to be stable enough for most people to use, such as Milestones and Release Candidate builds
• Integration -- Integration builds are built from these stable component versions that have been specified by each component team as the best version available.
• Maintenance -- for maintenance of a Release
• Nightly -- They are completely untested and will almost always have major problems. Many will not work at all.

It was the bold phrase of the nightly build that I quoted to the consumer that expected perfection from my team.   If a well funded project like Eclipse did not attempt perfection with each and every build, why he expect that from my team?   He still did (or at least said that he he did) and eventually left the company.

While my team has not distinguished between build types, we have captured the essence of "Not all builds are equal" with a concept we call "Build Maturity".    We take every build and put it through a series of increasingly complex automated tests.   As it passes the next more complex test suite, we set a higher maturity.  If we attempt the next round of testing and it does not pass, we record that it failed on this maturity attempt and stop all further testing.    This is very similar to the "Fast Fail" concept of a build system.  The build maturity is fully visible to anyone that wants to consume a build.

It is from the highest level of Build Maturity that our Consumers spend most of their time writing bugs against.    The focus then shifts to addressing all bugs that would stop the product from shipping, while each build is still put through the battery of automated testing to ensure that the Maturity has not regressed.   A release can come from anyone of these builds.

Build Feedback from Consumers

You should also have an issue tracking system and be willing to let anyone report anything to you in this system.  If you ask for feedback you will get it.   However, if you get it, you will need to act on it.  To act on it, your developers will need an issue tracking system that they enjoy using.   They will enjoy it if the tool you choose encourages a collaborative mindset rather than a "bug fix" compliance mindset. Even better if you can use the same tool to distinguish and track feature development.  Collaboration is increased and feedback becomes more fact based rather than emotional statements like, "that build is bad", "that build is buggy", or "that feature is clearly not done."

This kind of collaboration takes trust, understanding, and time.   But with time you can develop a common mindset towards an agile software release process that hits target dates more frequently and produces higher quality software.   Who wouldn't signup for a project with those outcomes?

Hold Core Beliefs -- Act on Them

I had a recent e-mail exchange from a departing employee that sparked an interesting follow-up mail thread about whether my organization was making strides in our Agile journey.  Many may have taken the comments as a punch in the gut.   I took it as an opportunity to point out just a few of the things that our team had accomplished.    Here was the essence of that message:

Who would have thought just two years that in a single week we could:

• Execute a release branch in less than 24 hours.   (for comparison other products in my organization, this event takes weeks)
• Put 9 builds into our automated infrastructure that run 10,000’s of automated tests -- including expanded functional testing through the User Interface.
• Produce 4 builds in row that passed 100% of those tests
• All four of those test runs requiring no manual intervention
• Doing all of the above while physically moving the entire team and all our test client machines!

Not a bad list at all.   Perhaps what had the team a bit down is that we had historically been criticised for executing on the list above rather than congratulated for it.  I went on to say to say:

We have done many things that I am hard pressed to believe have been done anywhere.    

Since this was perhaps a bold statement, it warranted some follow-up discussion.   At various team meetings, I suggested that if anyone knew where this was being done, I wanted to speak with that organization.    One suggestion that came from the team was the Eclipse Project.   I agreed with the suggestion since many of my core beliefs  that drove action towards these accomplishments above are centered on three key principles I first heard from the folks at Eclipse:

• Always Be Delivering
• Not All Builds are Created Equal
• Consume Your Own Output

I will write a blog post about each of these, starting with the first -- Always Be Delivering.

Always be delivering.

Each and every build that you produce should be consumable by anyone that wants to to consume it.    The primary benefit is to shorten the feedback cycle time.   This holds true for both new feature development and bugs that you may have introduced.   A secondary but crucial benefit is that it can significantly increase the stability of the installation --the act of delivering.  I frequently say that on any project I work on, "Day 1 is the developer proof of concept and Day 2 is the install."     Repeated delivery allows you to spot automation opportunities the can again shorten the feedback cycle time.

Frankly it's shocking how much code is written and committed to source repositories with claims made about it being "done" long before any consumer ever touches the code.   A recent quote from a colleague encapsulated this for me, "We need to be proud of our code before we make it available".    I was a consumer of this group's code.    Of the first four milestone builds that we took from that team, two of them did not even install.     The second installation failure was after providing feedback on the installation experience itself!

 My reply to my colleague was, "We get proud by making our code available to anyone that will give us feedback."   A potential downside of seeking feedback is that you generally get it.   You need a system for managing feedback or at least communicate which builds should be consumed by whom.    This is the subject of the next blog post -- Not All Builds are Created Equal.