Version control and branching

I sometimes wonder why it was that we started to use a SCM in the first place? And why did we start with branching? What problems did we encounter to start using an SCM and what were the problems that started us to use branches.

At the dawn of time all we worked together on source code in shared folders and a couple of problems arose. First we needed to call out to colleagues which files we were working on, locking out other team members to work on the same files. Second we had no history on the files, when we screwed up, we had to fix it and sometimes that was painstaiking. And third we lacked backups of source files. To solve this we started to use an SCM.
With an SCM we could work on multiple files, without interfering with other team members. There was a track record of our changes through commit messages and we could talk about specific revisions or versions.

An SCM comes with the possiblities of branching and tagging. Soon we used tagging to point to a specific revision in history to mark our releases. And we started using branches to create a personal development space. A personal space was good because most of our team members worked on different changes.

So what was the problem that branches solved for us? When we all worked on different features on the trunk we encountered problems with merging, multiple team members touched the same code, working with our own branch made it possible to develop in parallel. The con is that we introduced merging hell further down the line. Testing was done manually most of the time and we needed to create a solution to waterfall to a release, hence can git flow. A model to overcome the problems with branches. With git flow there are feature branches on which developers can work on their features, run unit tests and deploy to a development area. Then when all integrates well the feature is promoted to the development branch, again tests are run, it get's promoted to a release branch and again some more tests. Then all is merged to trunk and brought to production. Sounds waterfallish to me to be honoust. And git flow is a fix on a fix. It fixes problems on branch development. But shouldn't we just fix branching in the first place?

With a lean thought in mind where waste is evil, branching is a perfect way of introducing waste. It gives developers the opportunity to work on items that are not delivered. When working in teams, expecially in agile teams, the most important thing is focus. Working on features together is more important than working on multiple features at the same time. From XP we learned that pairing results in short feedback loops, improved quality and better architecture. Put this in practise and stop branching. Just work on the trunk. Working with a DVCS over a CVCS makes this even easier.

But when should we branch? Keep in mind that we only work on code that will actually go to production, but sometimes we need to experiment. If the experiments are small, they can be done on the trunk. If they are bigger or oppose the proper functioning of the trunk, they should be branched. Although with feature toggleing an experiment can be done on the trunk as well.
Another reason for branching is that tests are not fully automated or take too long to complete. Improve the testsets and think about what you need to test, instead of running all the tests all the time, be wise.

Altough I was a big fan of the branching model, I now understand that it introduces waste. It overcomes a problem that shouldn't have been there in the first place. Sometimes branching is good, but by exception.

Sources:

1. https://www.git-tower.com/learn/git/ebook/en/desktop-gui/basics/why-use-version-control
2. https://trunkbaseddevelopment.com/
3. https://www.quora.com/Why-do-we-need-branches-in-GIT
4. https://datasift.github.io/gitflow/IntroducingGitFlow.html

Continuous change

"Humans are very proud of their changes, but every imagined transformation turns out to be a new set of excuses for behaving exactly as the individual has always behaved" - Xenocide (The Ender Quintet 3, p36)

Continuous change is the process to facilitate continuous improvement. Improvement can only be achieved by experimenting and failing until there is success. Within lean this is known as plan-do-check-act. We think of new ways to do something, then we try them, we see the results of what we have done and we adjust.

For children this is standard behaviour, somewhere down the line this ability seems to get lost. We grow rigid and conservative when we age. We learn to do things a certain way, no questions asked. How many times didn't one hear: "We have done it like this as long as I can rememeber, why change now?". Perhaps the process is perfect, but it is never wrong to think about it and trying to improve. Only the people who ask questions about what is or what isn't discover new ways of doing things. The people that don't as questions will never grow, never gain new insights and never improve.

Somewhere deep inside most adults the child still lives on, give an adult some lego bricks or a a colouring page and they will be playfull. Only by becomming playfull we can open our minds and think about what we are doing and where we can improve. There are no stupid changes, all changes contribute to a better understanding of how stuff works. Sometimes stupid is good, we learn how not to do something and perhaps we gain insight for the next step to take.

To improve we need to give ourselves time to experiment, small controlled experiments. Change just a little in the product, the process or our lives. Observe and check what the change does and decide to keep the change or revert to the old ways. Some say that we need to change just one percent of what we do each day. The sum will be a remarkable change and hopefully an improvement over a year.

Forbes: What is Agile?

According to this article on Forbes the Agile Manifesto is an historical but outdated document now Agile is no longer only a software development approach but an organizational one. It has become about delighting customers, descaling work, enterprise-wide agility and nurturing culture. Those four principles are easier to understand and remember than the original row of principles in the manifesto. This does not make the manifesto useless, it promotes it to the mindset we need in order to be(come) Agile.

Entropy and the backlog

"Lack of order or predictability; gradual decline into disorder." - Google

When we create something, either documents, products or software, entropy has an effect on it. At the conception of an idea entropy already starts. The specifications change when we get further into completing our creation. The world around us changes, new things get invented, old things die, a lot changes in the passing of time. Over longer periods of time cities crumble into decay, governments change and even the way we view the world changes. In short, live is about change and coping with change. A result of this continuous process is entropy.

When we develop software we use specifications, we already found out that a big design up front isn't working very well. We only get what we designed eon's ago and not what we need right now; the world has moved on. A new concept was born about twenty years ago: agile working. To understand how slowly time proceeds sometimes, most big companies just start to adapt this way of working; they see that smaller companies that are agile, lean and adapting fast. Those companies are a danger to their industry. Using an agile method can help plan short and adapt to market changes.

This concept spawned a number of working methods of which one of them was scrum. One part of scrum is working with backlog's. A backlog is an ordered list of items or stories as we mostly call them, which is maintained by somebody that is responsible for the product that is created and maintained. But here come's entropy, the backlog killer. When the backlog is not maintained, items on it begin to fall into decay. They slowly lose their priority and therefor any meaningful existence. This article is not about product ownership, but about backlog's and entropy.

So what happens to an unmaintained backlog? As from my experience, the backlog just keeps on growing. And eventually it will get big enough that nobody understands the purpose of all the stories that are told. Which also means, that nobody knows exactly what the product is that is supposed to get build. In nature old things die. In most applications that manage a backlog, items live forever. There is no dying mechanism, stories that were written down by a stakeholder (someone with a more than average interest in the product) become irrelevant or even not understandable any more. Team members spent too much time looking at the backlog trying to understand what the purpose and demands are for the product in the making. So, entropy happens to the contents of a story. The story dies but is never cleaned up.

I see two camps in backlog maintenance. On one side there is the camp that tries to save everything that is ever said about the product. The backlog might have grown to immense proportions and they have a day job maintaining the log. They find it important to know all the stories, their context and purpose. Perhaps they feel even important because they are the centre of attention when people ask about the product. They are the know-it-all's. I can understand that feeling important might seem important, but it isn't practical, nor is it fun. It would be far more practical if everybody who works on the product has a chance to understand the backlog, the importance of the stories and the context of the product.
On the other side there is a group of people that clean up their backlog, but keep items in parking lots, afraid to lose information. The parking lot becomes covered with dust and the items lose its purpose. Entropy at its best. People place these items in the parking lot because they spend time to create the stories and find it a waste that their time was for nothing. But was it? Is it a waste to throw away a story like that? Does this story deserve to get wrecked and covered by dust over time?

In my opinion a backlog should be clear and up to date. This means that we give stories a right to die. If there is information in stories that we shouldn't lose we must write down that information in a system with a documenting purpose (entropy applies here as well). Within a development team and with the product owner we must create rules about the death of a story. It should be bound to time, to contents, to relevance. When a story hasn't been picked up within the allotted time, when it is not ready to be picked up or when it loses its relevance, the story should be finished. Drag the story to done, won't fix and clear up the backlog. Even better, automate this process, just like nature does. Incorporate entropy as a fact of life, let outdated stories die off and don't waste time and effort on them anymore.

Workflow

So I use Jira a lot. It's the default application for most organisations. In the last ten plus years I have seen many variations on the workflow. One significant factor in all the flows I have seen is the wait state. Wait states are killing for progress, the production process should be moving all the time. When issues are waiting to be picked up, there is loss. Time and entropy will catch up and make the change worthless.

In the above example workflow I tried to bring my knowlegde about the workprocess to a Jira workflow. This workflow can be used by many different teams. Let me explain the thinking behind this flow.

• New: All issues comming in recieve the New state. This state means an issue is on the backlog and de Product Owner needs to review it, when reviewed and when it fulfills the Definition of Ready the issues is transfered to Open.
• Refine: This is an optional status, some Product Owners need extra time to refine and make sure all the information is in the issue or story. They plan extra refinement sessions with key users, stake holders and some members of the team.
• Open: All stories with an open state are either on the backlog, they are ready to be taken into a sprint. Or they are on the sprint log, and are worked upon. When they are in the backlog the team will use it's refinement session to impact the story and places them in the sprint log, or on top of the backlog. Remember that we only need to refine 130% of our velocity.
• In Progress: This step means there is work going on for the story; and I mean all the work, not developer bound, but also testing, verifying, peer reviewing et cetera. When work is done and complies to the Definition of Done, the story can be transfered to Done.
• Review and Testing: Both optional states when teams need to make a distinction. They could be combined in a Verify state. These states are optional because of the following. When a developer assigns a story to another developer with the comment to review, the state doesn't need to change. After the peer review, the story needs to be tested, again assigning to a team member with the question to test should be sufficient. The story can still be in progress, just assigned to a team member with a different skillset.
• Done: The story is done according to the DoD, and acceptance criterea defined for the story are met. It can go into production. On an full Continuous Deployment Pipeline this can be the end state.
• Closed: story is done and in production. When problems arise, the story can be reopened.

In my humble opinion this is all we need in Jira, perhaps even a bit too much. It's working pretty well so far. Any thoughts on improving?

Fun is mandatory

Most time in our lives we spent working. A lot of people complain about their work. What is wrong with this picture? They forgot the most important thing in live: fun. Simple, plain old fun. They get annoyed by colleagues, bosses, managers, customers and start complaining. The worst thing about that is that complaining is that it works just like a virus, it spreads, infecting all other people on the work floor. The working atmosphere will get to sub zero in no time. Some will stay at their jobs because they don't dare leaving. Some flee and find a new team to infect and some, a few ones, will try to lighten up and try to get other colleagues back on track. Don't ever count on having those in your team, if you are lucky, there is one, but that's a rare occasion.

Working without fun is demotivating. Working and having fun while doing it, works in multiple ways. First of all, people having fun will excell. Second they tend to get more work done, third they will get the most out themselves and try to accomplish the same for the team.

How can we make development more fun? Most developers can tell you the answer right away; give me some time for myself to make something cool or fix something that is bothering me. A lot of big companies have done this already in different formats: some with a couple of hours per week, a weekend once every couple of months or a day per month. My guess is that about 5% of work time should at least be dedicated to fun stuff. Using this approach I made some nice achievements with my teams from translation modules to reducing technical debt to creating AI's which solve puzzles. Not everything is always work related, but it seems to me that most developers want to fix things that annoy them or increase there knowledge, anyway it's a win for the company.

When a fun day is planned, it will affect the work planning, there is at least eight hours less time per team member to create feature increments, calculate the impact on the velocity and demo the work that has been done on the fun day.

One on one production

Developing service oriented architectures (SOA) is pretty much how we develop software nowadays. With a firm believe that delivering small increments on software every two weeks and doing some daily stand up's is a goal, they stall in the improvement kata. A kata is a japaneese way of doing something. Think about a kata in martial arts, they are practised until perfection. Work can be seen as a kata too. Besides doing daily chores over and over again, there is also a possibility to use a behaviour kata for improving. By improving behaviour all the work will be improved. What this means is that we actually don't want to use a pattern to do things the same way all the time, we need a pattern to improve all the time and with that improve the work we do.

If we would stuck on the SOA implementation we can build a huge monolith application which we can release every two weeks. But what now if we have a target condition of delivering software throughout the day, up to perhaps ten times. A monolith would be very hard to push to production ten times a day. We should learn to improve each iteration with a goal of one on one production. If we would create one on one production it would be possible to release all day, any day. By creating small standalone services we would be able to release when we want, the only thing to keep in mind here is contract nagotiation. We should define an interface and stick to it, hence we need a versioning system (may I propose semver here?), a dedicated pipeline and knowledge of who our consumers are. By doing this we create owners of the service, a PO, a team and so on who really care about their product.

Creating one on one production might stand in the way we think about flexible processes. These are processen capable of doing more than one thing. Flexibility in processes might seem like a good idea. This makes it possible to work around problems that arise during processing and this will guarantee to meet the delivery dead line. There are many variables in the production pipeline to match all the demands any product needs to be produced. This sounds very flexible and therefor like an ulitmate solution. However, working around problems obfuscates the understanding of problems. Production is met with workarounds, but how are the core problems being identified and fixed? In a flexible system this is the hard part. Dedicating production pipelines and focussing on one on one production makes it possible to continuously improve the pipeline.

So for developing software from conception to deployment we should strive to one on one production, to continuously improve the way we work and practise a lot.