A bigger ball of mud

The most common architecture

5 years ago I inherited a big ball of mud. It was an exemplar of ‘this most frequently deployed of software architectures’ described in the paper written in 1999 by Brian Foote and Joseph Yoder. It was then 2006 and the system was part of the new strategic architecture. A shiny new system that represented our future. Yet, it was still a big ball of mud. When will we learn?

The project was interesting. It was part of a crucial business transformation. It was to deliver a key architectural component that integrated into a more complex system of over 70 individual components – some new, most existing – and it impacted thousands of people and processes. It was based on an off-the-shelf product that we had never managed to successfully deliver before. It had been outsourced as a fixed-price piece of work to the company that built the software, and we had 3 people ‘managing the supplier’ back in to our business. It was the biggest instance of this piece of software in the world and I was their biggest customer. We had insisted that another 3^rd party service provider be part of a design and integration team to de-risk the situation. And, to make matters worse, a bigger supplier of off-the-shelf software then bought the supplier of our product for strategic reasons. The project was interesting. The project was not simple.

A happy coincidence?

Because of this context and the need to join a few things together, I decided to focus my current studying needs of my MBA dissertation on ‘Relationship Management and Collaboration in delivering large business programmes using Agile Methods’. I thought it was worth killing the proverbial two birds with one stone and these both seemed like big birds. At the time of starting out on the project, I had been reading Implementing Lean Software Development book by Mary Poppendieck which set the tone for my approach to the delivery. In addition, I also read widely on the topic of relationships, partnerships, contracting, alliances and a whole gamut of other agile texts to explore how they might interrelate. For the purpose of this post, I am going to focus on the need for managing relationships and not step into the world of contracts (maybe another time). My position at the time of inheriting the project was to damn the contractual position. If I (and my business) was to be successful then we all had to be successful together and no contract was going to help us or get in the way of doing the right thing.

Understanding the landscape

When I started working on the project my first job was to go and live with the supplier on their premises and get to know them. Ask them questions. Understand what their philosophy was in their approach. Know thy enemy (well… not enemy per se, but it was clear a fight was coming).

My guiding light when starting out was to use the ‘Working software as the primary measure of progress’ as a way to judge the current position of the project. Every person I talked with I asked ‘…and how does the work you do help create working software?’ and ‘…how do you know you’ve done a good and effective job today?’ I was a right pain in their behind. But, the problem was not one person told me about the working software. For me, my approach was simple and very engineering focused. I believe that quality and speed are inextricably linked. So, if there was any dubious approaches to quality, then my belief was that they’d never be able to go fast enough for my business. They told me about their designs, their reporting, the project plan, the paper-based architecture and other very important elements that they thought an IT director should hear about. Frankly, I was horrified. Even more so when I found out that the 3rd version of the project was released into a production environment over 6 months late, missing most of the important functionality and had generated in the region of 400 major defects. In addition, there was NO WAY to deliver that same version of the software again because they didn’t actually know the configuration of the software and the metadata that went with it. Boy, was I in trouble? They were already underway developing version 4 and version 5 in parallel <sigh>.

But, this gets worse… It wasn’t just the people delivering for my project that I needed to worry about. This solution was based on an ‘off-the-shelf’ product that ‘just needed configuring’ (huh hum), so I really needed to look in to that too. Oh, and we also had that extra dimension of a 3rd party doing the integration work that I needed to just cast an eye over.

Conway’s Law

In the paper Big Ball of Mud, Conway’s Law is mentioned a couple of times. It is also discussed in other lean and agile texts when discussing strategies around scaling delivery practices. It says: …organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.

If this law is true (and I believe there is a lot of evidence to support it from my experiences) this has big implications for how you manage relationships, communication structures and how you might choose products to build your systems around. The number of points for communication between a supplier and your organisation can be vast. In this case, my project had over 200 people involved, plus the product engineering department of the supplier and all the management roles involved in the relationship. In addition, there were all of the integrating systems within the rest of the programme to consider. The communication structures were complex and inconsistent. The goals, scope and delivery approach were not well understood. The people working for the suppliers didn’t really understand what needed to be delivered, and the direction set by the supplier was not one destined for success.

Working on the relationships

After assessing the project for the first few days and spending time with the team it was clear they were under a huge amount of pressure from their management and from the management within the programme. My first step was to build relationships with the people who were desperate for a change. These were often the ones who could see what was being done was way below the standard needed or people who were constantly in the firing line of the flame-thrower. Basically, these were the folks who were not resistant to change.

I chose to close down the communication channels from within my own company who were creating the pressure and noise. I chose to take the heat away from the team; I needed to build trust and I could do that by removing pressure and creating a more collaborative environment.

Because of a need to do primary research and analytical study for my MBA I explored a few models that might help me empirically manage to a better situation. I found a model created by the Department of Trade and Industry from 1997 that described 5 dimensions to consider that I decided to use to better understand the situation:

1. Compatibility – the degree to which each parties’ business interest coincide
2. Trust – the degree to which the parties open up to each other and discuss sensitive issues such as pricing and quality problems
3. Closeness – how much the parties know about each other’s businesses
4. Depth of Relationship – the depth and breadth of contact between different levels of the hierarchy and departments
5. Reliance – how dependant are both parties on business output

When setting out, the only measure that scored highly was reliance. This was empirical information from over 200 people. Our destinies were entwined at this stage so we’d better improve the other 4 scores!

Expectation management

It is really easy to tell someone what you are going to do before beginning a project. It is much harder to deliver what you said regardless of your intentions. Parasuraman, Berry and Zeithaml proposed a model in 1985 called the Service Gaps Model. This described simplistically the different areas where gaps can appear between managers, the people who do the work, and the service received by a customer, and what factors might be involved in generating these gaps. Ultimately, any programme delivery is done by the people who do the work and they are the people that you need to build trust with and get to know. You do need to understand what their challenges are and work out how they need to be helped for them to be successful. Sometimes their challenges come from within their own organisation. As a manager of such a programme this was part of my job. I needed to understand who, what and why blockers were coming. There were lots of different reasons. One thing that became clear was that there was an authenticity gap. Most people can spot such gaps when words and behaviours don’t match. The words of the management team didn’t always match the work of the team. Sometimes it did, but there was, obviously, some conflict going on. The excellent Cluetrain Manifesto talks about this in regards to the authenticity gaps of a business. Business is a conversation. It is many conversations. It is a conversation that happens at all levels. The people doing the work are quite happy to tell you about things that are not working. They are happy to share with you the troubles that they suffer from. If they are not happy to do so, then there may even be deeper problems in the culture of your supplier. Silence speaks volumes. There are rarely no problems. You have to be open to listening to them and resolving them together. But, you need to believe that the problems and solutions you are discussing together are for the good of your objectives as well as theirs. If the answer sounds like one that you’ve heard before, it will probably lead you to a similar result you’ve experienced in the past.

The folks doing the work are the people who really define the service and outcome you will receive. An old boss of mine always said ‘Know your people’. I believe this. As with all relationships it is working through the problems and creating shared experiences that improves the bonds.

Resetting expectations

I needed to reset expectations with all parties. I needed to set the standards that I needed based on the poor engineering that had been delivered to-date.  I needed to Demand Technical Excellence. I also need to create an environment where all the parties felt that they could participate effectively (this did have some contractual implications and is for another time). I also needed to insert people into the overall team who knew what it would take to meet the quality standards that I was expecting. There was no point in relying on my suppliers or the people who had managed the project so far as they were the team who had taken us to the position we were in. I decided to hire some agile coaches and technical leadership to help the existing parties really understand. The funny thing about trying to change a situation like this is that people are very quick to think change has happened, and claim success, but my experience is that people want to tell you what you want to hear rather than to show you what has changed in reality. Holding the line on the new standards was difficult. It meant heated discussions and politics at different levels within the relationship. This was an 18-month battle and we made incremental steps of progress in terms of building trust, deepening relationships and delivering on our promises. We did manage to improve the quality of the software, and we did deliver our business objective. Every iteration and release was better than the previous one. We need time to retrospect and continuously improve. If the line wasn’t held though and quality demanded the project would have reverted to its old ways. I am absolutely convinced of that. My job was to take my suppliers on a journey that was fit for my business. The core competence of my suppliers was supposedly software delivery. My experience told me a different story and one that I see repeated in many clients. It wasn’t because the people were not good. They were, but they didn’t have the environment or support to succeed.

Why share this story?

At the moment, I have a number of friends and colleagues involved in a huge Big Ball of Mud fight. I feel for them. It is a massive project recovery that is taking its toll. It is a death-march, for sure. These things don’t come up because of one person’s decision-making capability. They occur because of ‘the system’. They occur because people believe that someone else will fix the problem. They occur because of the short-term need for higher margins. They occur because people still don’t know what great (or even good) looks like. They occur because wishful thinking is still predominant in lots of companies that I see. And they occur because active management of relationships and suppliers is not happening with people who can get under the covers of what is really going on. What does happen when your luck runs out?

As an aide memoir, here are a couple of things that I learned through this experience that I would urge managers to consider:

1. You can outsource work; you cannot outsource your problem. It will end up hurting you more. Help your supplier through the problem.
2. You must build relationships at all levels to gain the best and most complete perspective. Businesses are full of conversations. Don’t believe the brochure. Know the people doing the work.
3. You should explore how well aligned your communication channels and cultures are. How do you know that your partner will be successful and support your needs?
4. Be aware and actively manage all parties interfacing with your supplier and from your supplier into your business. Communication channels are important to look after. Loose lips sink ships. Remember Conway’s Law. If your communication channels are confused, your software will be too.
5. You need to know what great looks like, set that standards and help all parties achieve that standard. If you’ve never seen great, find someone who has. Find examples of great even if they exist in different industries. Don’t rely on the team that has been deployed to your project without validating that what they are doing for you is, indeed, world-class. Just because the supplier says they are doesn’t mean that team knows what awesome looks like or that their company can make great happen. Judge results on working solutions not on hyperbole. And keep the feedback cycles short. As google says at number three in their philosophy, ‘Fast is better than slow.’
6. Get to know what objectives and systems your suppliers have in place. Is your success really (and I mean really) aligned to your success? Change control is still a key business mode for many System Integrators – the more Work-In-Progress that exists, the more expensive change will be to your business. That’s good for them, but terrible for your ability to deliver quickly.