The cascading effect of the DevOps way of working

When you are going to work in a DevOps way it all starts with removing the first silo. This means removing the wall between Dev and Ops by putting them together in a team. The consequence is that the whole team is responsible for change and run. When the whole team becomes responsible, no individual can fall back on their role, they are all committed to the cause. Dev can do Ops, test can do Dev, Ops can do test and Dev and so on. This means more collaboration in the team and less push off of work to another role. As a DevOps team you are all in it together.

Working together is only possible if it is done in a small team, and with a small team I mean no more than five engineers (I like to call all team members engineers and not by their role). I state a team of five is the best, a team of three is minimum. With five it will still be a team, with six it will become two teams of three. Six or seven is still possible but communication and therefor collaboration will become increasingly harder. 

If you have smaller teams, you also need smaller software components and with smaller components you need more interfaces. These interfaces are twofold, one is the technical one with APIs, versioning, release cycles and so on. The other is the communication interface. How do you, as a team, communicate with other teams, how are your dependencies organised? You need to think about hard and soft dependencies and come up with a plan. Hard dependencies are those that make your team wait for work from another team or individual. Soft dependencies are those that interface with another team but don't require any work from another team. One of the goals for a DevOps team is to lessen hard dependencies and turn them into soft ones. This can only be done if it is clear who your teams customers are, which is one thing the team should know and investigate.

Working in a small team means a lot of collaboration, you need to share information and to improve sharing it is wise to minimise the work in progress by introducing a WIP limit. A WIP limit is a limit on the amount of work that can be in progress. If you need to learn a lot, make it a low number, so all team members need to work on the same thing together. One of the most errors I see is that every engineer works solely on items of the backlog not sharing knowledge and information. When two engineers work on the same item, they need to communicate and explain what they are doing, teaching the other one about their ways and maybe even learn something themselves.

It can happen that an engineer can't collaborate and that the WIP limit is reached. The purpose of DevOps is not to be as efficient as possible (max. resource utilisation) but to be as effective as possible (max. customer value). This means that when an engineer isn't working on items on the backlog, they should investigate improvements, refactor or learn.

Investigating the way of working and finding new ways to work leads to better processes and automation. When a DevOps team starts automating, many quirks in the process will be found. By solving these the quality of work and the speed of work will increase. 

So what are the steps to take?

1. Form mixed teams with the disciplines needed to convert customer requests in a usable product or service
2. Find out all the dependencies of the team
3. Make communication plans with other teams, technical and social
4. Introduce WIP limit to increase learning
5. Automate

Let's start doing DevOps!

Why we do things in Scrum

We do daily scrums by answering the three questions.
We give a only a demo in the review.
We skip retrospective.
Planning is distributing tasks
But we do Scrum.

That is not Scrum, that is doing what we do in another form. For me the basis are the three pillars: transparency, inspection and adaption. Every event in Scrum is based on these pillars. 

Take the standup for instance, it is transparent because everybody is welcome to attend and the work progress is made visible. It is a moment to inspect the work for that day and come up with a plan to deliver stuff. It is a moment to adapt the way we are organized to get things done. 
I can't withstand those annoying questions: what did you do, what are you going to to and do you have impediments. The result is somewhat in the area of: "I did x yesterday, didn't finish so I continue on x today and I need to visit the dentist". We are not interested in your status or your agenda. At a standup we are inspecting if we get work done. Nothing about the individual, all about the team. So stop with those questions! One approach you can use is to talk about the work items on the board. Go from almost done in the top right corner, to stuff that can be started in the bottom left. Ask yourselves: what do we need today to get the top story in the done state. Don't make it individual work, but group up to finish the work. Find a way to finish it, then move to the next item on the list until you run out of team members. The work on the top is the most important, so it is also very important to finish it first.

Next on the agenda is the review. A review is a key moment to talk to the stakeholders, inspect the work that has been done, agree on the next goals. You can do a demo if it adds to the experience of the stakeholder. But most important is that you talk about what needs to be done and what you need from each other. Offer insights in work, show metrics, ask what the stakeholder needs and measure. Become predictable. Stakeholders don't need status updates or reports, they want to be involved and this is the moment to do that.

Then comes the retrospective. Most teams draw a boat or a racing car and talk about what is slowing them down and what can speed them up. It's a nice exercise, but it doesn't get to the point of the retrospective. Where the review is about what we do, the retrospective is about how we do it. What is our way of working, are we a team, can we trust each other, what can we do to improve flow and focus. In the retrospective you inspect and adapt on the way you work as a team.

And the last event in Scrum is planning. In this event we don't distribute tasks, we inspect our parameters. What capacity do we have, how does that relate to our speed of delivery and how much work can we take on. Is there any unplanned work we need to take in account? What stuff are we not going to do. Can we finish all the work and be predictable? So we inspect the work load, we adapt our forecast and we publish our sprint log to make it transparent.

We've talked about four Scrum events and how they relate to the pillars. Transparency is needed to build trust, inspection is needed to learn and adaption is needed to improve.

Working in an agile team

How does working in an agile team differ from working in a traditional team? Working in an agile team for the first time can be very challenging and you might rethink the way you work.

The first rule of forming an agile team is multidiscipline. An agile team should be able to do all the work, but not everybody in the team has to be able to perform all tasks. The second rule is a clear product vision and purpose; what needs to be build and maintained and why. And the third rule is: don't meddle. Let the team decide and give room for mistakes. Help the team get better and never punish.

Traditional roles are fading in a team. A programmer is learning to test, an operator is learning to code and a tester is learning to develop. All roles become less important and the knowledge is shared. Local heroes are replaced by team players; who support each other in getting the job done. 

One of the most important behaviors of an agile team is working in pairs. This makes sharing of knowledge possible, facilitates reviewing and saves time. While this sounds counter intuitive, results show that lead time and cycle time for changes drop significantly when people pair up. There is also a lower defect rate, less technical debt and more customer satisfaction.

Over time the roles are becoming responsibilities. A multidisciplinary team consists of engineers who are responsible for a part of the development and maintenance of a product. They are responsible for the piece of knowledge they bring to the team. They are expected to keep developing that knowledge and share it with the team and similar interest groups within the company. It is not really important anymore what exactly the job description of that person is. When they belong to the team, they become engineers that want to build excellent products and services for their customers.

An agile team is all about customer value. They engage in customer relations and find the best way to improve customer value. Work does not come from managers anymore but directly from customers. An agile team is not part of any project, it is part of an organisation that builds stuff for customers. In a big enterprise vision and goals (long term) are shared by leaders and teams use that to focus goals on a quarterly basis and in sprints. They organise the work they receive from customers around these goals.

Because an agile team is responsible for the work they do they are multi disciplinary, because they need to serve the customer best they have self autonomy and because they need to know where to go they need vision. 

How to fail without fear?

Failing in production is bad. There have been many solutions, but non is guaranteeing a 100% success rate. There is always a chance things will go bad when we deploy to production. What can we do to mitigate this risk? Over the years there have been many solutions, albeit most of them are technical ones. Let's have a look at different solution that we have come up over the years and find out how we can minimize risk when we deploy.

Fix in production

The first thing we did when something fails is to fix it as soon as possible, most of the time this was in a production environment. Although we can bring our service back up real quickly, the uncertainty of the deployment makes us feel weary to actually do it more often than necessary or on any regular basis.

DTAP

In order to tackle all the problems we have encountered in a production fix, we introduce different environments to test every possible situation. I have seen up to seven different environments in one situation; just imagine the cost and the result is that a lot of environments are doing absolutely nothing for most of the time. 

Next thing we see here is that not all the environments are exactly the same. There is always a minor difference in those environments that can bite us in the last step to production. I've seen different size clusters, different machine configurations, different firewalls and even different data connections.

The strength of multiple environments that are dedicated to one phase gets nullified by the increase in maintenance, the extended process and the cost.

Cloud

In order to solve the DTAP-trap we can spin up environments on demand to test. This means we don't need idle DTAP stations. We can actually do with one instance of a cloud platform. Just spin up an environment, run the necessary tests and then kill the instance. We get billed by the time we use the system, the system is identical to production and our process is smaller.

Pipeline

In order to facilitate a solid process of testing and deploying it is mandatory to use an automated pipeline. A pipeline can be seen as all the steps that are needed from development to production within one automated process. When we hook this up to the cloud and we make it as fast as possible. We can almost mimic fixes in production this way. And because we can implement safe guards through the whole process we can be almost sure that our stuff works.

Experimentation

In order to fail without fear we need to move to pipelines and cloud solutions (either on or off premise) and we need to feel relaxed with experimentation. Experimentation early in the process gives us the opportunity to learn new methods, processes and technical solutions. The more experience we get, the less chances there are that anything will fail.

Conclusion

In order to be sure we deploy safely and with the lowest chance of failure we need to focus on cloud-like delivery through automated pipelines. 

Fight technical debt

Many software projects suffer from technical debt and there is not enough priority to solve them. With technical debt comes dependency and versioning problems, security issues and a non sustainable application. The longer technical debt is not treated it becomes exponential harder to solve.

When does technical debt occur?

Technical debt arises when the world around the application changes. For instance when dependencies are upgraded, supporting applications are upgraded, interfaces are renewed or new vulnerabilities are found. 

Another reason for technical debt is shortcutting, this means that we bring out a new feature that a client really needs but we take some shortcuts to make it work. The solution is there for the client, but it hasn't been implemented in a proper technical way. In this case we can reap the benefits of the new feature early, but we need to fix the implementation later on in the game. It gives us a short boost and an advantage over our competitors but it needs rework once released.

How does an application suffer from entropy?

Entropy is the aging of systems and the loss of relevance and compatibility of the application during this process. All systems are suffering from entropy, which is solved by updating those systems on a regular basis. When an application isn't updated while the surrounding infrastructure and application are, the application becomes seriously affected by entropy. Therefor refactoring of the application becomes mandatory.

How can we find technical debt?

Finding technical debt can be hard. Luckily there are applications that can help us spotting technical debt. One of those applications is SonarQube. With SonarQube you can find security flaws when new security insights are found. You can scan for CVE and OWASP vulnerabilities. You can also use this tool to spot deprecated dependencies. 

What can we do to fight technical debt?

Reserving time to refactor the application is important. When someone touches code, they should also solve reported problems in that piece of code. Writing unit tests on all the code we touch helps us in refactoring without breaking the correct working of the code.

Having a clear understanding of the dependencies  we use in the code is important. We need to keep ourselves updated on the versions of the dependencies and we need to do refactoring when a dependency is deprecated. 

By regularly building and testing our code we can get a grip of all the vulnerabilities in de the code. Even when we don't touch the code for a while, we need to regularly test for new vulnerabilities and implement fixes when we spot them. Refactoring and unit testing here is important as always.

Last but not least we need to give ourselves time and priority to do all the above on a regular basis. In other words, setup a build that runs automatically at least every week, preferably every day that inspects against entropy, dependencies and vulnerabilities and act on the results as soon as possible.

Conclusion

By inspecting our code on regular intervals and refactor it, we can keep our application safe and up to date. Keep technical debt as low as possible by acting on a changing world. Implement an automated build pipeline to make life easier and write unit tests so you can easily refactor.

Vision, mission and goals

Your team, department or company is just another fish in the pond. You produce stuff that people buy and that's it. When the world changes you might face a serious problem. Many companies have gone broke and ceased to exist. Out of business, for good. How can this happen? Why do some companies survive while others perish? 

Kodak and Fuji are good examples. The first one produced camera rolls, just like the second one. But when the digital camera disrupted the market, Kodak chose to stick to camera rolls, while Fuji focused on film; all kinds of film, layered plastic film. Kodak could not pivot by lack of vision, they where short sighted and only focused on the analog camera roll. Fuji on the other hand understood that it was making film that could be used in analogue cameras, but also understood that there are many other purposes for film. The first one went out of business while the latter one still produces film.

In this example we see that Fuji had a vision, a broad view of the world and its needs over a long period of time. Kodak did not have a vision for a longer term and they gambled and lost.

Another example. Elon Musk is successful because he has a vision that one day, men will be living among the stars. To make that happen he needs to be able to travel to space, so he makes rockets. But he knows that the planet will probably not survive long enough to reach to the stars due to pollution. So in order to achieve his vision he needs to act on that as well, hence he started Tesla.

Vision

Vision is one of the most important aspects of a company. Small startups have success because their vision is still alive and clear. Big companies that produce a lot of different stuff with a lack of vision might survive, but have a big chance to fail at any market disruption because they are not able to pivot based on their vision.

Developing a vision is imperative to give a company focus. A vision is about what a company wants to achieve in four years or even longer. We should ask ourselves: "What is the purpose of the company?", "Why do we exist?" A vision is about one sentence capturing the essence of the company.

Some examples:

Nike - "Bring inspiration and innovation to every athlete in the world"

Unilever -  "Make sustainable living commonplace" 

Netflix - "Helping content creators around the world to find a global audience."

As we can see from these examples, the companies could pivot, change their products and still fulfill their vision. Nike could go into gyms, food and stadiums. Unilever could go into housing and farmlands. And Netflix isn't bound to the internet. 

Mission

A mission is about how you are going to achieve the vision. Typical questions are: "What will we do?", "How will we do it?", "What do I need to do to reach the stars?"

A clear mission is made for a middle long term, about one to four years. It gives meaning to the vision, makes it less abstract. A mission should be revised as soon as the market is disrupted, while probably the vision will remain the same.

A mission for Nike could be: to start a company that creates composites for new soles or they could produce a movie about the origins of the Olympics.

Goals

Goals are the short term anchors on which to focus in order to achieve the said mission. Goals stretch from three months to about a year. They are clear, market focused and descriptive. Goals are something like a plan for the next quarter or year.

Cascading

This system of vision, mission and goals can be cascaded through the company from C-level to teams. A company will have a vision, mission and goals. Departments will have their own vision, mission and goals to cover the goals of the company. The same goes for teams. They will formulate their own vision, mission and goals based on the goals of the department. The further down the chain, the shorter lived the three elements are.

Just imagine a pivot of a company, this means all department and team elements must be flushed and remade in line with the new company goals.

Battleplan

In order to have your teams, departments and the whole company move in the right direction, you should formulate a goal. Mind you that this is a hard exercise and should not be taken lightly. You are going to be stuck with the vision for a long time, so spent enough time on it to get it right. Get the people involved, sketch, write, bin it and start over. Refactor the outcomes, right until it feels right, gives you meaning and energy. Writing a vision can take days or weeks. It is important to get the feeling right.

Once the vision is done, start with the mission. This is also no more then about a sentence, it describes how we are going to achieve the vision. It is important to stick to the how. Again sketch a lot, throw all away, start over and refactor until it is right.

Based on vision and mission the goals need to be created. There might be many goals, the rule of thumb is that there are about as many goals as there are departments, so every department can fulfill one goal. Do not base the goals on departments though, if there is no match, perhaps the department should be reworked and not the goal. Form the company about the vision, mission and goals, not the other way around.

Refactor

The world is constantly changing, we need to adapt to survive. The only way we can do that is to keep our eyes and ears open and pivot when we need. Pivoting is changing the course of the company to match the market demand. Pivot is what Fuji did and Kodak did not. 

In order to pivot, we need to refactor or rework our mission and goals often. We need to inspect them on a regular basis and we should adapt them when we see a disruption. Managing a company, department or team is hard work, these three mentioned elements should be at the core of that work. 

New world leaders

A new world leaders is someone who owns the vision, mission and goals. That leader can explain why a company is working the way it does. Why stuff needs to be done. A new world leader is always explaining why we are doing the things we do, because of the vision the leader wants to achieve. When you can explain the why as a leader, you build a successful company, department or team.

Vertical silos

"Dev(sec)Ops: everything you do to overcome the friction created by silos ... All the rest is plain engineering" - Patrick Dubois on Twitter.

Removing horizontal silos between different parts of the organisation is getting pretty common. We see ops and test integrated in development teams, shifting all the work needed for a product to one team. Disciplines in the team are shared and taught to each other and greater knowledge of the product and the process is obtained. The customers are benefitting from these results and are receiving better quality products.

But there are still hindrances when it comes to organising all this. We are still keeping old structures of hierarchy in organisations and push operational tasks to the teams. Impediments are solved through escalation processes and there are still change advisory boards and other boards that try to direct the operation. We call this vertical silos.

To get rid of vertical silos we need to do something. We need to scale. One way is to scale with the use of frameworks like SAFe, Nexus or LeSS. The other way is to de-scale. Let's have a look at the latter one.

First step: leading from behind

De-scaling is a multistep operation. The first step is changing how we lead the organisation. We should lead as wolfs do: Lead the pack from behind, not from the front. Teach and show, don't impose. This means that leaders let teams do their work, ask them if they can help them achieve their goals. Leaders must start propagate their visions and goals, and explain why those goals are important. 

They should not do any operational work, or influence the work being done. They are here to support the teams in improving their performance. Leaders need a bit of know how of the work, but not too much. They need to be experts in leading. 

Second step: reduce layers

When the leaders are thinned out, the next step can be taken. We start to remove management layers in the organisation, these layers are making your organisation slow to respond to the market. Each layer is a processing step in the vertical organisation. Information is consumed, processed and given to the next layer. We see that all these layers don't add any more information on problems and only make communication more complex. Thus resulting in a slow performance.

Take a critical view on the management layers that are in place, think outside the box, have a look around to find answers to how to reduce layers. As a rule of thumb, try to have a maximum of three layers from management to teams and less if possible.

Third step: autonomy

In order to make teams perform better, they need autonomy. In a complex IT environment the best engineers thrive if they are given enough autonomy. This is a two way street, autonomy and responsibility. Whilst they get autonomy they are becoming more and more responsible. They need the right people in the team, they need to do the right things and they need to raise impediments when problems occour. 

Fourth step: small startups

With great vision from the top, middle management reduced and teams than can find the way themselves because they have the autonomy to do so you are ready for the third step, creating small organisations within the bigger whole.

Then it is possible to look at teams as small startups, little companies within the company that are trying to achieve something. Those small startup should be fairly self sufficient, they will ask support from the organisation when they need it, not the other way around.

De-scaling in short:

1. Start showing vision, mission and goals
2. Remove surplus of management, reduce layers
3. Reorganise autonomy and responsibility
4. Make teams independent as small lean startups