Entropy and the backlog

"Lack of order or predictability; gradual decline into disorder." - Google

When we create something, either documents, products or software, entropy has an effect on it. At the conception of an idea entropy already starts. The specifications change when we get further into completing our creation. The world around us changes, new things get invented, old things die, a lot changes in the passing of time. Over longer periods of time cities crumble into decay, governments change and even the way we view the world changes. In short, live is about change and coping with change. A result of this continuous process is entropy.

When we develop software we use specifications, we already found out that a big design up front isn't working very well. We only get what we designed eon's ago and not what we need right now; the world has moved on. A new concept was born about twenty years ago: agile working. To understand how slowly time proceeds sometimes, most big companies just start to adapt this way of working; they see that smaller companies that are agile, lean and adapting fast. Those companies are a danger to their industry. Using an agile method can help plan short and adapt to market changes.

This concept spawned a number of working methods of which one of them was scrum. One part of scrum is working with backlog's. A backlog is an ordered list of items or stories as we mostly call them, which is maintained by somebody that is responsible for the product that is created and maintained. But here come's entropy, the backlog killer. When the backlog is not maintained, items on it begin to fall into decay. They slowly lose their priority and therefor any meaningful existence. This article is not about product ownership, but about backlog's and entropy.

So what happens to an unmaintained backlog? As from my experience, the backlog just keeps on growing. And eventually it will get big enough that nobody understands the purpose of all the stories that are told. Which also means, that nobody knows exactly what the product is that is supposed to get build. In nature old things die. In most applications that manage a backlog, items live forever. There is no dying mechanism, stories that were written down by a stakeholder (someone with a more than average interest in the product) become irrelevant or even not understandable any more. Team members spent too much time looking at the backlog trying to understand what the purpose and demands are for the product in the making. So, entropy happens to the contents of a story. The story dies but is never cleaned up.

I see two camps in backlog maintenance. On one side there is the camp that tries to save everything that is ever said about the product. The backlog might have grown to immense proportions and they have a day job maintaining the log. They find it important to know all the stories, their context and purpose. Perhaps they feel even important because they are the centre of attention when people ask about the product. They are the know-it-all's. I can understand that feeling important might seem important, but it isn't practical, nor is it fun. It would be far more practical if everybody who works on the product has a chance to understand the backlog, the importance of the stories and the context of the product.
On the other side there is a group of people that clean up their backlog, but keep items in parking lots, afraid to lose information. The parking lot becomes covered with dust and the items lose its purpose. Entropy at its best. People place these items in the parking lot because they spend time to create the stories and find it a waste that their time was for nothing. But was it? Is it a waste to throw away a story like that? Does this story deserve to get wrecked and covered by dust over time?

In my opinion a backlog should be clear and up to date. This means that we give stories a right to die. If there is information in stories that we shouldn't lose we must write down that information in a system with a documenting purpose (entropy applies here as well). Within a development team and with the product owner we must create rules about the death of a story. It should be bound to time, to contents, to relevance. When a story hasn't been picked up within the allotted time, when it is not ready to be picked up or when it loses its relevance, the story should be finished. Drag the story to done, won't fix and clear up the backlog. Even better, automate this process, just like nature does. Incorporate entropy as a fact of life, let outdated stories die off and don't waste time and effort on them anymore.

Workflow

So I use Jira a lot. It's the default application for most organisations. In the last ten plus years I have seen many variations on the workflow. One significant factor in all the flows I have seen is the wait state. Wait states are killing for progress, the production process should be moving all the time. When issues are waiting to be picked up, there is loss. Time and entropy will catch up and make the change worthless.

In the above example workflow I tried to bring my knowlegde about the workprocess to a Jira workflow. This workflow can be used by many different teams. Let me explain the thinking behind this flow.

• New: All issues comming in recieve the New state. This state means an issue is on the backlog and de Product Owner needs to review it, when reviewed and when it fulfills the Definition of Ready the issues is transfered to Open.
• Refine: This is an optional status, some Product Owners need extra time to refine and make sure all the information is in the issue or story. They plan extra refinement sessions with key users, stake holders and some members of the team.
• Open: All stories with an open state are either on the backlog, they are ready to be taken into a sprint. Or they are on the sprint log, and are worked upon. When they are in the backlog the team will use it's refinement session to impact the story and places them in the sprint log, or on top of the backlog. Remember that we only need to refine 130% of our velocity.
• In Progress: This step means there is work going on for the story; and I mean all the work, not developer bound, but also testing, verifying, peer reviewing et cetera. When work is done and complies to the Definition of Done, the story can be transfered to Done.
• Review and Testing: Both optional states when teams need to make a distinction. They could be combined in a Verify state. These states are optional because of the following. When a developer assigns a story to another developer with the comment to review, the state doesn't need to change. After the peer review, the story needs to be tested, again assigning to a team member with the question to test should be sufficient. The story can still be in progress, just assigned to a team member with a different skillset.
• Done: The story is done according to the DoD, and acceptance criterea defined for the story are met. It can go into production. On an full Continuous Deployment Pipeline this can be the end state.
• Closed: story is done and in production. When problems arise, the story can be reopened.

In my humble opinion this is all we need in Jira, perhaps even a bit too much. It's working pretty well so far. Any thoughts on improving?

Fun is mandatory

Most time in our lives we spent working. A lot of people complain about their work. What is wrong with this picture? They forgot the most important thing in live: fun. Simple, plain old fun. They get annoyed by colleagues, bosses, managers, customers and start complaining. The worst thing about that is that complaining is that it works just like a virus, it spreads, infecting all other people on the work floor. The working atmosphere will get to sub zero in no time. Some will stay at their jobs because they don't dare leaving. Some flee and find a new team to infect and some, a few ones, will try to lighten up and try to get other colleagues back on track. Don't ever count on having those in your team, if you are lucky, there is one, but that's a rare occasion.

Working without fun is demotivating. Working and having fun while doing it, works in multiple ways. First of all, people having fun will excell. Second they tend to get more work done, third they will get the most out themselves and try to accomplish the same for the team.

How can we make development more fun? Most developers can tell you the answer right away; give me some time for myself to make something cool or fix something that is bothering me. A lot of big companies have done this already in different formats: some with a couple of hours per week, a weekend once every couple of months or a day per month. My guess is that about 5% of work time should at least be dedicated to fun stuff. Using this approach I made some nice achievements with my teams from translation modules to reducing technical debt to creating AI's which solve puzzles. Not everything is always work related, but it seems to me that most developers want to fix things that annoy them or increase there knowledge, anyway it's a win for the company.

When a fun day is planned, it will affect the work planning, there is at least eight hours less time per team member to create feature increments, calculate the impact on the velocity and demo the work that has been done on the fun day.

One on one production

Developing service oriented architectures (SOA) is pretty much how we develop software nowadays. With a firm believe that delivering small increments on software every two weeks and doing some daily stand up's is a goal, they stall in the improvement kata. A kata is a japaneese way of doing something. Think about a kata in martial arts, they are practised until perfection. Work can be seen as a kata too. Besides doing daily chores over and over again, there is also a possibility to use a behaviour kata for improving. By improving behaviour all the work will be improved. What this means is that we actually don't want to use a pattern to do things the same way all the time, we need a pattern to improve all the time and with that improve the work we do.

If we would stuck on the SOA implementation we can build a huge monolith application which we can release every two weeks. But what now if we have a target condition of delivering software throughout the day, up to perhaps ten times. A monolith would be very hard to push to production ten times a day. We should learn to improve each iteration with a goal of one on one production. If we would create one on one production it would be possible to release all day, any day. By creating small standalone services we would be able to release when we want, the only thing to keep in mind here is contract nagotiation. We should define an interface and stick to it, hence we need a versioning system (may I propose semver here?), a dedicated pipeline and knowledge of who our consumers are. By doing this we create owners of the service, a PO, a team and so on who really care about their product.

Creating one on one production might stand in the way we think about flexible processes. These are processen capable of doing more than one thing. Flexibility in processes might seem like a good idea. This makes it possible to work around problems that arise during processing and this will guarantee to meet the delivery dead line. There are many variables in the production pipeline to match all the demands any product needs to be produced. This sounds very flexible and therefor like an ulitmate solution. However, working around problems obfuscates the understanding of problems. Production is met with workarounds, but how are the core problems being identified and fixed? In a flexible system this is the hard part. Dedicating production pipelines and focussing on one on one production makes it possible to continuously improve the pipeline.

So for developing software from conception to deployment we should strive to one on one production, to continuously improve the way we work and practise a lot.

Dunbar

While reading The Lean Mindset: Ask the Right Questions I stumbled upon The Dunbar Number (page 29 in my version of the book). Robin Dunbar studied groups of apes, it seemed that the size of the group was related to the size of the neocortex of the primates. Extrapolated to humans this gives the number of around 150 people that one individual is capable of tracking the social relationship of.

There are some common sizes of groups:

1. An "inner circle" of about three to five very close friends or family members.
2. A "sympathy group" of 12 to 15 close friends who care about each others faith
3. A "hunting group" of 30 to 40 who work together to accomplish a task
4. A "clan" of 150 people who maintain stable interpersonal relationships
5. A "tribe" of about 500 to 2500 people who speak the same language or dialect

How should this work in a company? The inner circle are the direct teammember you work with the most, the sympathy group is your team (scrum says seven people on average per team, this would mean at least two teams here). The hunting group would be all the teams that work on the same product from conception to production (so four to six teams), this would include all disciplines necessary to run the business around the product. The clan would be a small company or department that makes three to five products. The tribe would be the company with company values, vision and mission, the tribe would be the scope of the culture of the organisation.

Now I'm wondering if this would work. What will happen if a company will outgrow the number of the tribe? Should a company split up? Will preformance drop? And on the other hand, if the sympathy group consists of two teams, what would be the composition of those groups and what relation would they have to the hunting group?

Any thoughts?

forming–storming–norming–performing

Making a successfull team is not that hard. Already in 1965 Tuckman described the different stages a team needs to go through to become a successfull team. Because these stages apply to small teams, up to twelve people, his lecture can be applied to scrumteams.

Source: https://en.wikipedia.org/wiki/Tuckman%27s_stages_of_group_development

Deployment Pipeline

I just finished one of the best books about software development ever. It's called The Phoenix Project. The nice thing about this book that it is written like a novel and not like a bone dry study book. I want the share the following quote:

"Your next step should be obvious by now Grasshopper. In order for you to keep up with customer demand, which includes your upstream comrades in Development", he says, "you need to create [...] a deployment pipeline. That's the entire value stream from code check-in to production. That's not an art. That's production. You need to get everything in version control. Everything. Not just the code, but everything required to build the environment. Then you need to automate the entire environment creation process. You need a deployment pipeline where you can create test and production environments, and then deploy code into them, entirely on-demand. That's how you reduce your setup times and eliminate errors, so you can finally match whatever rate of change Development sets the tempo at."

Source: The Phoenix Project