A bigger ball of mud

The most common architecture

5 years ago I inherited a big ball of mud. It was an exemplar of ‘this most frequently deployed of software architectures’ described in the paper written in 1999 by Brian Foote and Joseph Yoder. It was then 2006 and the system was part of the new strategic architecture. A shiny new system that represented our future. Yet, it was still a big ball of mud. When will we learn?

The project was interesting. It was part of a crucial business transformation. It was to deliver a key architectural component that integrated into a more complex system of over 70 individual components – some new, most existing – and it impacted thousands of people and processes. It was based on an off-the-shelf product that we had never managed to successfully deliver before. It had been outsourced as a fixed-price piece of work to the company that built the software, and we had 3 people ‘managing the supplier’ back in to our business. It was the biggest instance of this piece of software in the world and I was their biggest customer. We had insisted that another 3rd party service provider be part of a design and integration team to de-risk the situation. And, to make matters worse, a bigger supplier of off-the-shelf software then bought the supplier of our product for strategic reasons. The project was interesting. The project was not simple.

A happy coincidence?

Because of this context and the need to join a few things together, I decided to focus my current studying needs of my MBA dissertation on ‘Relationship Management and Collaboration in delivering large business programmes using Agile Methods’. I thought it was worth killing the proverbial two birds with one stone and these both seemed like big birds. At the time of starting out on the project, I had been reading Implementing Lean Software Development book by Mary Poppendieck which set the tone for my approach to the delivery. In addition, I also read widely on the topic of relationships, partnerships, contracting, alliances and a whole gamut of other agile texts to explore how they might interrelate. For the purpose of this post, I am going to focus on the need for managing relationships and not step into the world of contracts (maybe another time). My position at the time of inheriting the project was to damn the contractual position. If I (and my business) was to be successful then we all had to be successful together and no contract was going to help us or get in the way of doing the right thing.

Understanding the landscape

When I started working on the project my first job was to go and live with the supplier on their premises and get to know them. Ask them questions. Understand what their philosophy was in their approach. Know thy enemy (well… not enemy per se, but it was clear a fight was coming).

My guiding light when starting out was to use the ‘Working software as the primary measure of progress’ as a way to judge the current position of the project. Every person I talked with I asked ‘…and how does the work you do help create working software?’ and ‘…how do you know you’ve done a good and effective job today?’ I was a right pain in their behind. But, the problem was not one person told me about the working software. For me, my approach was simple and very engineering focused. I believe that quality and speed are inextricably linked. So, if there was any dubious approaches to quality, then my belief was that they’d never be able to go fast enough for my business. They told me about their designs, their reporting, the project plan, the paper-based architecture and other very important elements that they thought an IT director should hear about. Frankly, I was horrified. Even more so when I found out that the 3rd version of the project was released into a production environment over 6 months late, missing most of the important functionality and had generated in the region of 400 major defects. In addition, there was NO WAY to deliver that same version of the software again because they didn’t actually know the configuration of the software and the metadata that went with it. Boy, was I in trouble? They were already underway developing version 4 and version 5 in parallel <sigh>.

But, this gets worse… It wasn’t just the people delivering for my project that I needed to worry about. This solution was based on an ‘off-the-shelf’ product that ‘just needed configuring’ (huh hum), so I really needed to look in to that too. Oh, and we also had that extra dimension of a 3rd party doing the integration work that I needed to just cast an eye over.

Conway’s Law

In the paper Big Ball of Mud, Conway’s Law is mentioned a couple of times. It is also discussed in other lean and agile texts when discussing strategies around scaling delivery practices. It says: …organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.

If this law is true (and I believe there is a lot of evidence to support it from my experiences) this has big implications for how you manage relationships, communication structures and how you might choose products to build your systems around. The number of points for communication between a supplier and your organisation can be vast. In this case, my project had over 200 people involved, plus the product engineering department of the supplier and all the management roles involved in the relationship. In addition, there were all of the integrating systems within the rest of the programme to consider. The communication structures were complex and inconsistent. The goals, scope and delivery approach were not well understood. The people working for the suppliers didn’t really understand what needed to be delivered, and the direction set by the supplier was not one destined for success.

Working on the relationships

After assessing the project for the first few days and spending time with the team it was clear they were under a huge amount of pressure from their management and from the management within the programme. My first step was to build relationships with the people who were desperate for a change. These were often the ones who could see what was being done was way below the standard needed or people who were constantly in the firing line of the flame-thrower. Basically, these were the folks who were not resistant to change.

I chose to close down the communication channels from within my own company who were creating the pressure and noise. I chose to take the heat away from the team; I needed to build trust and I could do that by removing pressure and creating a more collaborative environment.

Because of a need to do primary research and analytical study for my MBA I explored a few models that might help me empirically manage to a better situation. I found a model created by the Department of Trade and Industry from 1997 that described 5 dimensions to consider that I decided to use to better understand the situation:

  1. Compatibility – the degree to which each parties’ business interest coincide
  2. Trust – the degree to which the parties open up to each other and discuss sensitive issues such as pricing and quality problems
  3. Closeness – how much the parties know about each other’s businesses
  4. Depth of Relationship – the depth and breadth of contact between different levels of the hierarchy and departments
  5. Reliance – how dependant are both parties on business output

When setting out, the only measure that scored highly was reliance. This was empirical information from over 200 people. Our destinies were entwined at this stage so we’d better improve the other 4 scores!

Expectation management

It is really easy to tell someone what you are going to do before beginning a project. It is much harder to deliver what you said regardless of your intentions. Parasuraman, Berry and Zeithaml proposed a model in 1985 called the Service Gaps Model. This described simplistically the different areas where gaps can appear between managers, the people who do the work, and the service received by a customer, and what factors might be involved in generating these gaps. Ultimately, any programme delivery is done by the people who do the work and they are the people that you need to build trust with and get to know. You do need to understand what their challenges are and work out how they need to be helped for them to be successful. Sometimes their challenges come from within their own organisation. As a manager of such a programme this was part of my job. I needed to understand who, what and why blockers were coming. There were lots of different reasons. One thing that became clear was that there was an authenticity gap. Most people can spot such gaps when words and behaviours don’t match. The words of the management team didn’t always match the work of the team. Sometimes it did, but there was, obviously, some conflict going on. The excellent Cluetrain Manifesto talks about this in regards to the authenticity gaps of a business. Business is a conversation. It is many conversations. It is a conversation that happens at all levels. The people doing the work are quite happy to tell you about things that are not working. They are happy to share with you the troubles that they suffer from. If they are not happy to do so, then there may even be deeper problems in the culture of your supplier. Silence speaks volumes. There are rarely no problems. You have to be open to listening to them and resolving them together. But, you need to believe that the problems and solutions you are discussing together are for the good of your objectives as well as theirs. If the answer sounds like one that you’ve heard before, it will probably lead you to a similar result you’ve experienced in the past.

The folks doing the work are the people who really define the service and outcome you will receive. An old boss of mine always said ‘Know your people’. I believe this. As with all relationships it is working through the problems and creating shared experiences that improves the bonds.

Resetting expectations

I needed to reset expectations with all parties. I needed to set the standards that I needed based on the poor engineering that had been delivered to-date.  I needed to Demand Technical Excellence. I also need to create an environment where all the parties felt that they could participate effectively (this did have some contractual implications and is for another time). I also needed to insert people into the overall team who knew what it would take to meet the quality standards that I was expecting. There was no point in relying on my suppliers or the people who had managed the project so far as they were the team who had taken us to the position we were in. I decided to hire some agile coaches and technical leadership to help the existing parties really understand. The funny thing about trying to change a situation like this is that people are very quick to think change has happened, and claim success, but my experience is that people want to tell you what you want to hear rather than to show you what has changed in reality. Holding the line on the new standards was difficult. It meant heated discussions and politics at different levels within the relationship. This was an 18-month battle and we made incremental steps of progress in terms of building trust, deepening relationships and delivering on our promises. We did manage to improve the quality of the software, and we did deliver our business objective. Every iteration and release was better than the previous one. We need time to retrospect and continuously improve. If the line wasn’t held though and quality demanded the project would have reverted to its old ways. I am absolutely convinced of that. My job was to take my suppliers on a journey that was fit for my business. The core competence of my suppliers was supposedly software delivery. My experience told me a different story and one that I see repeated in many clients. It wasn’t because the people were not good. They were, but they didn’t have the environment or support to succeed.

Why share this story?

At the moment, I have a number of friends and colleagues involved in a huge Big Ball of Mud fight. I feel for them. It is a massive project recovery that is taking its toll. It is a death-march, for sure. These things don’t come up because of one person’s decision-making capability. They occur because of ‘the system’. They occur because people believe that someone else will fix the problem. They occur because of the short-term need for higher margins. They occur because people still don’t know what great (or even good) looks like. They occur because wishful thinking is still predominant in lots of companies that I see. And they occur because active management of relationships and suppliers is not happening with people who can get under the covers of what is really going on. What does happen when your luck runs out?

As an aide memoir, here are a couple of things that I learned through this experience that I would urge managers to consider:

  1. You can outsource work; you cannot outsource your problem. It will end up hurting you more. Help your supplier through the problem.
  2. You must build relationships at all levels to gain the best and most complete perspective. Businesses are full of conversations. Don’t believe the brochure. Know the people doing the work.
  3. You should explore how well aligned your communication channels and cultures are. How do you know that your partner will be successful and support your needs?
  4. Be aware and actively manage all parties interfacing with your supplier and from your supplier into your business. Communication channels are important to look after. Loose lips sink ships. Remember Conway’s Law. If your communication channels are confused, your software will be too.
  5. You need to know what great looks like, set that standards and help all parties achieve that standard. If you’ve never seen great, find someone who has. Find examples of great even if they exist in different industries. Don’t rely on the team that has been deployed to your project without validating that what they are doing for you is, indeed, world-class. Just because the supplier says they are doesn’t mean that team knows what awesome looks like or that their company can make great happen. Judge results on working solutions not on hyperbole. And keep the feedback cycles short. As google says at number three in their philosophy, ‘Fast is better than slow.’
  6. Get to know what objectives and systems your suppliers have in place. Is your success really (and I mean really) aligned to your success? Change control is still a key business mode for many System Integrators – the more Work-In-Progress that exists, the more expensive change will be to your business. That’s good for them, but terrible for your ability to deliver quickly.

Value, Flow and Quality

The PMI Announcement

As a result of the news that the PMI have endorsed Agile Project Management I was doing some reading around the topic and came across this article.  The question posed seems like a very traditional project management question because it is asked from the basis that everything is delivered at the end of a project.  It doesn’t consider that a scope could be wrong, or the times are impossible or the costs are not enough. This is what project managers are taught and this is one of the reasons that so many projects fail, in my opinion.  To be able to predict an arbitrary date sometime into the future that is months, years or decades away is a nonsense.  Predicting how much money you might spend or what the scope might be during that period seems even more impossible to my mind.  And when you finally throw in poorly managed engineering techniques which diminishes the quality, you are always left at the end increasing time, budgets and reducing the scope. Without having built up any knowledge as to whether your predictions might be accurate or not people are committing to the parameters of time, cost and scope (quality is always high at the beginning of any project when it is just a thought in someone’s head, but diminishes when the reality of building stuff actually kicks in which is why the commitment of any project doesn’t really consider quality).

I know this is quite a glum scenario, but there you are.  It is still the most common scenario that I see in environments where projects are the norm and commitments are made to be concrete on the back of an approved project. I think the fact that the project is really the only construct discussed by the project management community is what creates this thinking – the clue is in the title!  I remember saying to a client a couple of years back ‘I guarantee 100% that you will NOT get 100% of the things you are thinking of right now on the date in your project plan that is over 18 months away.   But, how about we work with a little more uncertainty and we aim to deliver 60% to 70% between this date and this date approximately 18 months away so that we can get something into the field for a real trial with real customers, and we would have already got feedback on what we have been doing at regular intervals beforehand.  Which one would you prefer?’  The debate that this created was interesting, because people started to say: ‘well, it would be good to get something to play with before the end date and something we could show our customers’. Others were still very much: ‘well, we have to deliver everything on the date’.  So, I simply asked: ‘what is everything?’.  Stunned silence followed.

I know many of you will have been in similar surreal conversations in your work too, but it always strikes me as a little… well, stupid.  The proliferation of the idea that complicated projects can be predicted so accurately is absurd.  The XP game has demonstrated many times to many teams that even the simplest tasks (like blowing up balloons) cannot be predicted accurately until we have a few goes at it. If we try the same project over and over again, I guarantee we will get good at that project.

You get what you think about

I have come across this saying a number of times: Watch your thoughts for they become your words. Watch your words for they become your actions. Watch your actions for they become your habits. Watch your habits for they become your character. See your character become your destiny. I don’t know who this should be attributed to, but I do like it. In this context, it brings to my mind the fact that the focus of the project management community is focusing on the wrong things.

I remember when I got my first real budget responsibility as an IT Director.  It was a very significant sum of money and one of my more mature and long-serving colleagues took me aside and said: ‘Phil, always hit your numbers. Never underspend or overspend your budgets. This is the most important aspect of your job.  Hit your numbers and you will be fine.’  The young, idealistic me was a little taken aback by this advice. I thought the point was to deliver the projects and programmes, but apparently I was mistaken. How naive.

For me, the focus on trading time, cost and scope off against one another is the wrong focus.  I think managers should be focusing on Value, Flow and Quality.  Each of these words require many more posts to do them justice, but my intention here is to say that the project managers (or any managers involved in delivering products, services, systems, projects, programmes etc) should focus on creating these three things in their organisations.  I am not arguing that cost control is unimportant.  It is important. But, it is ridiculous to believe that this is the most important thing.  The biggest ‘cost’ of a business is a failure to turn ideas into valuable products and services that meet or exceed the expectations of customers.  Going further, there is an implicit cost of not delivering quickly.  For every extra day a feature takes to be delivered a day has been lost reaping economic benefit for that feature. This is the Cost of Delay and managers need to understand this.

Value is not fluffy

One argument I have heard on numerous occasions is that it is difficult to really understand the value of an individual feature or activity, and that’s why we focus on cost. We know how to deal with costs. Value is difficult to measure. This makes no sense to me. If it does to anyone who reads this, please try to enlighten me. I do want to learn.

In the agile community the word value is used a lot. In fact, in my experience it is overused and abused to argue for the things that people want to do rather than things that have real value (i.e. money/economic benefit). I hear the cries now: ‘What about other value types such as happiness of employees?’. Happy people has an economic benefit. Heskett et al postulated a model called the Service Profit Chain that links employees, customers and quality into a profit model. The challenge is creating a value model that works for your business that allows you to trade off different features, requirements and improvements in what you intend to deliver and the way you intend to deliver them. Please remember that any model you create can be refined as you learn more about how that model is being used. You might not get it right first time, but you can definitely have a good debate with your finance director about how you trade things off for the highest economic return.

Projects are short-sighted

One of the problems with projects is that they are inherently short-sighted.  A project manager is judged on their ability to deliver the project, however they do that. It doesn’t matter what state they leave any underlying IT systems, product architectures or people who were working on the project. Managing the assets of a business with a view for the longer-term is a philosophy that is promoted in lean, and is an important principle to be instilled in managers and should be considered when designing any organisational structures and philosophies. When I see organisations that are set up solely around projects and not the underlying components and systems I get nervous.  If more than one project manager affects a common platform without any need to care about the quality they leave behind and the ability for the next project manager to be successful then there is an issue. Projects do not promote flow.  They should be used more sparingly than they are.

Quality is not negotiable

One of the strongest statements that has come out of the agile community is that Quality is not negotiable. As I mentioned in a previous post you need to demand technical excellence. However, as I mentioned previously, when people think about quality at the inception of a project it is easy to think that your project is going to be of the highest quality, but reality bites when you get going. It is easy to have standards about your engineering when you are in the throws of thrashing out your requirements and design in a traditional project. But, why do we believe spending our time on requirements and design at the beginning of a project is more valuable and important than spending time of creating the right environment for building high quality software? Quality and productivity are inextricably linked. Poor quality now will slow you down in the future. Great quality now will increase the flow of value and improve speed-to-market over the longer-term.

A virtuous circle

Time, cost and scope are often called the Iron Triangle.  It sounds like a very negative phrase.  I prefer to think about something that creates a virtuous cycle.  Value, Flow and Quality creates a virtuous circle. This does not mean you are challenging the laws of physics as the article mentioned at the beginning suggested, but it does change the mindset about how you can manage your product, service and IT deliveries. The idea is to go as fast as you can possibly go. Managing projects to time, cost and quality hasn’t given us the ability to go as fast as we can go. If the PMI’s announcement starts to educate project managers to start thinking this way, then I believe it is a really valuable move for our industry. Time for a change? I think so.



The Quality Responsibility

A few weeks ago I was facilitating a session with a group of people responsible for delivering a software platform to support a number of very large institutions in complex markets.  The session was centred around how you could create more collaboration, predictability, communication and feedback into their delivery cycle and how the different groups and roles might better interact with each other to deliver higher quality, valuable software and solutions to their customers.  They have had problems meeting their commitments to-date and when they have delivered it hasn’t really met their customers’ requirements.  A common problem that we’ve all heard about many times now.

Their normal approach is full of hand-offs, long lead times between specifying something and seeing it in reality to provide real feedback, and very document rich to support people’s ailing memories across these long lead times.  During this session there was a real feeling of parochialism.  The Development Manager was arguing the QA needed to step up to do such and such.  The QA Manager was saying the Dev were always delivering sub-standard work.  Everyone was pointing at the requirements still being worked on.  And the designers and architects had such grand plans that weren’t achievable.  It was the classic example of the silo organisation.  The interesting thing is that all of their backs are against the wall because they have deliveries to hit.  My point to them was:  QA fails, you ALL fail.  Dev fails, you ALL fail.  Design fails, you ALL fail.  There was silence in the room and then the penny dropped.  They realised that they should be a team.

I read two blog posts yesterday that made me think about this situation.  The first was the TradableQualityHypothesis by Martin Fowler.  The second was Mike Gaultieri’s ‘Want Better Quality? Fire Your QA Team‘.  Both posts were ultimately about the ownership, accountability and responsibility of quality.  For some organisations they believe that quality is improved by spending more time on getting the requirements right.  For others it is about getting the design right.  For a lot it is about spending all the time on QA.  Not many firms actually believe (in my experience) that the developers are really the ones who can help because it has become for large enterprises a commodity – which is completely wrong.  However, for the teams and organisations who actually produce high quality work they understand it is about everyone systemically collaborating and working together to develop solutions.  Everyone is responsible for quality.  Interestingly, in Mike’s article, the extreme position of firing the ‘Quality’ team had the amazing effect of everyone else taking accountability and responsibility for the quality of the output.  It would be interesting to understand what actually happened and if it supported the hypothesis of Tradable Quality too.  My guess is that the Internal Quality was increased significantly as a result.

I’m not necessarily espousing that all teams need to fire their QA departments, but I am very much of the mind that all of the roles and responsibilities need to be blended into a one-team mentality whereby everyone succeeds or fails on each others’ quality and performance.  The key thing for me is that Managers and Directors need to understand that they will damage their future business unless they really understand this.  As I said in my last post, Demand Technical Excellence, or you will all have a case of high dynamic complexity and lots of finger waving when the IT platforms grind to a halt.


Demand Technical Excellence

Understanding Complexity

In his book Solving Tough Problems, Adam Kahane describes 3 types of complexity: dynamic, generative, and social.  Dynamic complexity describes the time and distance between when a cause is in introduced within a system and when the effect is stumbled across.  The closer the time and space of cause and effect the lower the dynamic complexity.  Generative complexity is the subject of how well known a problem is and how many time that problem has been solved.  In other words how predictable and familiar a problem is.  The final complexity type, social complexity, is about how aligned people and groups are in terms of assumptions, beliefs, values and objectives.  A well-functioning would ordinarily have low social complexity because people understand the values and objectives of the groups and take the time to share assumptions.

In emergn our jobs allow us to work with teams, projects, programmes and business units who are all looking to solve complex organisational issues that contribute to delivering products and services.  The techniques we deploy from the agile and lean philosophies are geared towards lowering these three areas of complexity.

IT is Business

My main interaction within enterprises are with IT Directors, CIOs, leaders of business change and other Product/Service Directors who have responsibility of delivering new or improved products and services to their customers.  One of the similarities of all our clients is that their businesses are highly reliant on IT deliveries and the quality of the software that forms the fabric of their operations.  In fact, most modern organisations are now so reliant on IT that any IT transformations are really business transformations.  This is quite a problem for a lot of businesses because a lot of people still see them as separate.  The old thinking of ‘we need to outsource software engineering/IT as it’s not a core part of our business’ needs to be re-looked at or, at the least, needs to be understood better so it can be managed better.

10 Years of Agile

I have been visiting a number of our clients over the year who are undergoing major transformations in their IT systems to better support their business need for increasing speed-to-market, lowering the total cost of ownership and creating better customer experiences.  This is a very common set of objectives and most companies are turning to agile and lean for solutions.  Over the last weekend, 33 people from the world of agile got together to celebrate the last 10 years since the agile manifesto was introduced and to discuss the next phase of evolution.   The first area that has been mentioned in every write up of the event is ‘Demand Technical Excellence’.  In my role prior to emergn I was responsible for tens of millions of pounds of software and I recognise this point to be so important.  I am often astounded (and dismayed) by the lack of interest by my peers in the need for excellent software craftsmanship and standards.  Even more discouraging is the belief that this problem can be outsourced to a company whose fulltime occupation is to deliver software yet fail to show the levels of quality you would expect.  This belief is often a bout of wishful thinking and sometimes comes with the belief that the harder someone stamps their foot and demands quality the more likely they are to get it.  My view is that if you want quicker speed-to-market, lower cost-of-ownership and better customer experiences then you’d better invest in quality.  Focusing on lowering the cost and speed of change of your software platforms will have a huge economic benefit for a business and will allow your customers to participate alongside you to develop excellent features that they really, really want.  For me, this renewed focus from the industry is an excellent outcome of the event.

So what?

My advice for enterprises is that they’d better hire people who really care about and understand software and their business, and know how to really manage and monitor the software being delivered and how it is delivered by outsourcing partners. Relying on wishful thinking and contracts that ‘protect’ them will not be enough to remain at the top of increasingly competitive environments.  This is a case of high dynamic complexity.  Poor software engineering has massive implications to a company yet the effects of the cause are noticed years later and for years to come and across many areas of an organisation.  This means the problem is not a simple one to solve, but one that will have a material benefit for a long time to come.

Companies that have moved forward with agile and lean transformations including rolling out Scrum, Kanban and other people/process oriented changes are making good strides forward which shouldn’t be ignored.  However, without the investment in the quality of software and improving the technical excellence competency, the transformation can only go so far and any investments in whizzy new strategic architectures will probably end up in the same place.