How to fail without fear?

Failing in production is bad. There have been many solutions, but non is guaranteeing a 100% success rate. There is always a chance things will go bad when we deploy to production. What can we do to mitigate this risk? Over the years there have been many solutions, albeit most of them are technical ones. Let's have a look at different solution that we have come up over the years and find out how we can minimize risk when we deploy.

Fix in production

The first thing we did when something fails is to fix it as soon as possible, most of the time this was in a production environment. Although we can bring our service back up real quickly, the uncertainty of the deployment makes us feel weary to actually do it more often than necessary or on any regular basis.

DTAP

In order to tackle all the problems we have encountered in a production fix, we introduce different environments to test every possible situation. I have seen up to seven different environments in one situation; just imagine the cost and the result is that a lot of environments are doing absolutely nothing for most of the time. 

Next thing we see here is that not all the environments are exactly the same. There is always a minor difference in those environments that can bite us in the last step to production. I've seen different size clusters, different machine configurations, different firewalls and even different data connections.

The strength of multiple environments that are dedicated to one phase gets nullified by the increase in maintenance, the extended process and the cost.

Cloud

In order to solve the DTAP-trap we can spin up environments on demand to test. This means we don't need idle DTAP stations. We can actually do with one instance of a cloud platform. Just spin up an environment, run the necessary tests and then kill the instance. We get billed by the time we use the system, the system is identical to production and our process is smaller.

Pipeline

In order to facilitate a solid process of testing and deploying it is mandatory to use an automated pipeline. A pipeline can be seen as all the steps that are needed from development to production within one automated process. When we hook this up to the cloud and we make it as fast as possible. We can almost mimic fixes in production this way. And because we can implement safe guards through the whole process we can be almost sure that our stuff works.

Experimentation

In order to fail without fear we need to move to pipelines and cloud solutions (either on or off premise) and we need to feel relaxed with experimentation. Experimentation early in the process gives us the opportunity to learn new methods, processes and technical solutions. The more experience we get, the less chances there are that anything will fail.

Conclusion

In order to be sure we deploy safely and with the lowest chance of failure we need to focus on cloud-like delivery through automated pipelines. 

Fight technical debt

Many software projects suffer from technical debt and there is not enough priority to solve them. With technical debt comes dependency and versioning problems, security issues and a non sustainable application. The longer technical debt is not treated it becomes exponential harder to solve.

When does technical debt occur?

Technical debt arises when the world around the application changes. For instance when dependencies are upgraded, supporting applications are upgraded, interfaces are renewed or new vulnerabilities are found. 

Another reason for technical debt is shortcutting, this means that we bring out a new feature that a client really needs but we take some shortcuts to make it work. The solution is there for the client, but it hasn't been implemented in a proper technical way. In this case we can reap the benefits of the new feature early, but we need to fix the implementation later on in the game. It gives us a short boost and an advantage over our competitors but it needs rework once released.

How does an application suffer from entropy?

Entropy is the aging of systems and the loss of relevance and compatibility of the application during this process. All systems are suffering from entropy, which is solved by updating those systems on a regular basis. When an application isn't updated while the surrounding infrastructure and application are, the application becomes seriously affected by entropy. Therefor refactoring of the application becomes mandatory.

How can we find technical debt?

Finding technical debt can be hard. Luckily there are applications that can help us spotting technical debt. One of those applications is SonarQube. With SonarQube you can find security flaws when new security insights are found. You can scan for CVE and OWASP vulnerabilities. You can also use this tool to spot deprecated dependencies. 

What can we do to fight technical debt?

Reserving time to refactor the application is important. When someone touches code, they should also solve reported problems in that piece of code. Writing unit tests on all the code we touch helps us in refactoring without breaking the correct working of the code.

Having a clear understanding of the dependencies  we use in the code is important. We need to keep ourselves updated on the versions of the dependencies and we need to do refactoring when a dependency is deprecated. 

By regularly building and testing our code we can get a grip of all the vulnerabilities in de the code. Even when we don't touch the code for a while, we need to regularly test for new vulnerabilities and implement fixes when we spot them. Refactoring and unit testing here is important as always.

Last but not least we need to give ourselves time and priority to do all the above on a regular basis. In other words, setup a build that runs automatically at least every week, preferably every day that inspects against entropy, dependencies and vulnerabilities and act on the results as soon as possible.

Conclusion

By inspecting our code on regular intervals and refactor it, we can keep our application safe and up to date. Keep technical debt as low as possible by acting on a changing world. Implement an automated build pipeline to make life easier and write unit tests so you can easily refactor.

Vision, mission and goals

Your team, department or company is just another fish in the pond. You produce stuff that people buy and that's it. When the world changes you might face a serious problem. Many companies have gone broke and ceased to exist. Out of business, for good. How can this happen? Why do some companies survive while others perish? 

Kodak and Fuji are good examples. The first one produced camera rolls, just like the second one. But when the digital camera disrupted the market, Kodak chose to stick to camera rolls, while Fuji focused on film; all kinds of film, layered plastic film. Kodak could not pivot by lack of vision, they where short sighted and only focused on the analog camera roll. Fuji on the other hand understood that it was making film that could be used in analogue cameras, but also understood that there are many other purposes for film. The first one went out of business while the latter one still produces film.

In this example we see that Fuji had a vision, a broad view of the world and its needs over a long period of time. Kodak did not have a vision for a longer term and they gambled and lost.

Another example. Elon Musk is successful because he has a vision that one day, men will be living among the stars. To make that happen he needs to be able to travel to space, so he makes rockets. But he knows that the planet will probably not survive long enough to reach to the stars due to pollution. So in order to achieve his vision he needs to act on that as well, hence he started Tesla.

Vision

Vision is one of the most important aspects of a company. Small startups have success because their vision is still alive and clear. Big companies that produce a lot of different stuff with a lack of vision might survive, but have a big chance to fail at any market disruption because they are not able to pivot based on their vision.

Developing a vision is imperative to give a company focus. A vision is about what a company wants to achieve in four years or even longer. We should ask ourselves: "What is the purpose of the company?", "Why do we exist?" A vision is about one sentence capturing the essence of the company.

Some examples:

Nike - "Bring inspiration and innovation to every athlete in the world"

Unilever -  "Make sustainable living commonplace" 

Netflix - "Helping content creators around the world to find a global audience."

As we can see from these examples, the companies could pivot, change their products and still fulfill their vision. Nike could go into gyms, food and stadiums. Unilever could go into housing and farmlands. And Netflix isn't bound to the internet. 

Mission

A mission is about how you are going to achieve the vision. Typical questions are: "What will we do?", "How will we do it?", "What do I need to do to reach the stars?"

A clear mission is made for a middle long term, about one to four years. It gives meaning to the vision, makes it less abstract. A mission should be revised as soon as the market is disrupted, while probably the vision will remain the same.

A mission for Nike could be: to start a company that creates composites for new soles or they could produce a movie about the origins of the Olympics.

Goals

Goals are the short term anchors on which to focus in order to achieve the said mission. Goals stretch from three months to about a year. They are clear, market focused and descriptive. Goals are something like a plan for the next quarter or year.

Cascading

This system of vision, mission and goals can be cascaded through the company from C-level to teams. A company will have a vision, mission and goals. Departments will have their own vision, mission and goals to cover the goals of the company. The same goes for teams. They will formulate their own vision, mission and goals based on the goals of the department. The further down the chain, the shorter lived the three elements are.

Just imagine a pivot of a company, this means all department and team elements must be flushed and remade in line with the new company goals.

Battleplan

In order to have your teams, departments and the whole company move in the right direction, you should formulate a goal. Mind you that this is a hard exercise and should not be taken lightly. You are going to be stuck with the vision for a long time, so spent enough time on it to get it right. Get the people involved, sketch, write, bin it and start over. Refactor the outcomes, right until it feels right, gives you meaning and energy. Writing a vision can take days or weeks. It is important to get the feeling right.

Once the vision is done, start with the mission. This is also no more then about a sentence, it describes how we are going to achieve the vision. It is important to stick to the how. Again sketch a lot, throw all away, start over and refactor until it is right.

Based on vision and mission the goals need to be created. There might be many goals, the rule of thumb is that there are about as many goals as there are departments, so every department can fulfill one goal. Do not base the goals on departments though, if there is no match, perhaps the department should be reworked and not the goal. Form the company about the vision, mission and goals, not the other way around.

Refactor

The world is constantly changing, we need to adapt to survive. The only way we can do that is to keep our eyes and ears open and pivot when we need. Pivoting is changing the course of the company to match the market demand. Pivot is what Fuji did and Kodak did not. 

In order to pivot, we need to refactor or rework our mission and goals often. We need to inspect them on a regular basis and we should adapt them when we see a disruption. Managing a company, department or team is hard work, these three mentioned elements should be at the core of that work. 

New world leaders

A new world leaders is someone who owns the vision, mission and goals. That leader can explain why a company is working the way it does. Why stuff needs to be done. A new world leader is always explaining why we are doing the things we do, because of the vision the leader wants to achieve. When you can explain the why as a leader, you build a successful company, department or team.

Vertical silos

"Dev(sec)Ops: everything you do to overcome the friction created by silos ... All the rest is plain engineering" - Patrick Dubois on Twitter.

Removing horizontal silos between different parts of the organisation is getting pretty common. We see ops and test integrated in development teams, shifting all the work needed for a product to one team. Disciplines in the team are shared and taught to each other and greater knowledge of the product and the process is obtained. The customers are benefitting from these results and are receiving better quality products.

But there are still hindrances when it comes to organising all this. We are still keeping old structures of hierarchy in organisations and push operational tasks to the teams. Impediments are solved through escalation processes and there are still change advisory boards and other boards that try to direct the operation. We call this vertical silos.

To get rid of vertical silos we need to do something. We need to scale. One way is to scale with the use of frameworks like SAFe, Nexus or LeSS. The other way is to de-scale. Let's have a look at the latter one.

First step: leading from behind

De-scaling is a multistep operation. The first step is changing how we lead the organisation. We should lead as wolfs do: Lead the pack from behind, not from the front. Teach and show, don't impose. This means that leaders let teams do their work, ask them if they can help them achieve their goals. Leaders must start propagate their visions and goals, and explain why those goals are important. 

They should not do any operational work, or influence the work being done. They are here to support the teams in improving their performance. Leaders need a bit of know how of the work, but not too much. They need to be experts in leading. 

Second step: reduce layers

When the leaders are thinned out, the next step can be taken. We start to remove management layers in the organisation, these layers are making your organisation slow to respond to the market. Each layer is a processing step in the vertical organisation. Information is consumed, processed and given to the next layer. We see that all these layers don't add any more information on problems and only make communication more complex. Thus resulting in a slow performance.

Take a critical view on the management layers that are in place, think outside the box, have a look around to find answers to how to reduce layers. As a rule of thumb, try to have a maximum of three layers from management to teams and less if possible.

Third step: autonomy

In order to make teams perform better, they need autonomy. In a complex IT environment the best engineers thrive if they are given enough autonomy. This is a two way street, autonomy and responsibility. Whilst they get autonomy they are becoming more and more responsible. They need the right people in the team, they need to do the right things and they need to raise impediments when problems occour. 

Fourth step: small startups

With great vision from the top, middle management reduced and teams than can find the way themselves because they have the autonomy to do so you are ready for the third step, creating small organisations within the bigger whole.

Then it is possible to look at teams as small startups, little companies within the company that are trying to achieve something. Those small startup should be fairly self sufficient, they will ask support from the organisation when they need it, not the other way around.

De-scaling in short:

1. Start showing vision, mission and goals
2. Remove surplus of management, reduce layers
3. Reorganise autonomy and responsibility
4. Make teams independent as small lean startups

How to be successful in the cloud?

Teams switching from traditional environments to a cloud model face many challenges. Working with cloud demands new skills from the team and a radical different approach to the development of software.

What are the factors that contribute to success?

To be able to survive in the cloud, a team needs to understand measuring. One does not simply drive to Italy with the dashboard blacked out and the windshield painted black. A team needs to implement measuring to see where they are going. This means application, infra and team metrics.

In order to take advantage of what the cloud has to offer, a team needs to invest in automation. Fully automated build pipeline, scaling infra, infra as code. These are fundamental for working in a cloud environment. They are also very practical when you work in a traditional environment, so you can start whenever you want.

A team needs to be multifunctional, this means that the team should be able to develop and maintain their product. Not all team members need to be able to do everything, but in total the team should not need external support. 

When a team needs to make decisions they should be able to do so themselves. They should not need to get authorizations from anybody else. After all a team knows best what they and their application is doing. A team should be in control of all their environments, pipelines, application and development.

Learning is the most important factor when it comes to working in the cloud. Continuous improvement can only be achieved by letting things fail, inspect and measuring. Teams need to learn how to fail fast, inspect early and improve skills and architecture.

How to be successful?

Measuring is done by defining KPIs with the team. Define reachable goals on performance, predictability, learning. Think about what you and your team want to improve and start measuring that. Make a simple and broad dashboard on which you show our KPIs and rework them on a regular basis. For the application and infra tools like ELK, Graphana en Prometheus can help a lot in getting insights in the application performance.

Rigorous automation is mandatory. By automating manual processes, a lot of time and quality is gained. Reserve enough time to improve automation. It is important that all team members contribute, all disciplines can reduce work and improve quality by automation. Build pipelines, automate triggers, incorporate quality.

Have you ever seen toddlers playing in a sandbox? Notice that they sit next to each other and play their own game. Only after they grow they learn to play together and share toys. This is the same behavior we see in teams, everybody working hard in their own discipline. It is only after we pair and learn to share knowledge and develop each others skills that we become more mature in our work and can become multifunctional. One of the nicest workshops you can do is to make T-shape profiles with the team. People will find shared interest that they can use to team up. 

To get authorization to the team, we need more than just the team. Management and change advisory boards must be willingly to give away their powers. But with great power comes great responsibility as well. So the team should be very transparent in order to gain the powers. This starts with getting grip on the flow in your work. Create a value stream map for all the products you create. Map the delays, risks and dependencies. Get management involved in the flow of things and your challenges. This will build trust and will grant you more powers.

You can only really be successful if you know how to fail. Make failure small and simple, define quality standards, inspect all failures thoroughly in order to learn. Involve the people around the team in the inspection and learning process. One start is to learn how to read error log files with the team, inspect stack traces and logfiles. You can also use tools to try to break your application like chaos monkey or attack proxies.

Have success

When you implement the points above you have a good jump start at being successful in the cloud. By regular inspection and being transparent, you, your team and the people you depend on can help grow a successful cloud implementation. And the beauty is, that it also works if you working in a more traditional environment.

What are the characteristics of a DevOps team?

Putting Ops in a Dev team or vice versa it is not sufficient for practicing DevOps. There is more to it. So what does a DevOps team typically do?

1. Measure

A DevOps team measures itself, it's product and the business value it generates. The team measures itself via feedback and reviewing of goals. The product is measured via tests through automated pipelines and value is measured via stakeholders and performance indicators in the application it builds.

2. Performance

A DevOps team strives for performance and improvement at all time. By regular inspection a DevOps team looks for performance improvements in the way of working, in the application and in its organisation.

3. Risk and refactoring

A DevOps team will do risk analysis, refactoring and supports secure coding and operation. The team is on top of its game when it comes to security and uses tools to measure the static code and does penetration testing on the running application. It refactors code, rewriting code blocks for better performance, improving code quality and readability. 

4. Automation

A DevOps team will strive for full automation of changes from testing, building, deploying to maintaining. Full automation is achieved by coding a pipeline, using gates to test for quality and automate deployment. They get rid of all manual actions in the process. They try to achieve a self healing system when production falters.

5. Transparency

A DevOps team is always transparent and open in what work and when they perform it. Visual management is available for all who want to see it, that is the team, but also stakeholders and leaders. Opening up the ability for them to help and improve.

6. Vision

A DevOps team has a vision and strategy for each product they support. That vision is derived from the company vision. Strategic goals are the lighthouse on the horizon for the teams, so they can create focus on what needs to be done. Vision created by leaders must be simple and clear for everybody to understand.

7. Mixed roles

Members of DevOps teams work together mixed in multiple roles. Not everybody needs to be able to do everything, but the team should have the capabilities to do everything. No outside support should be needed. With pairing knowledge is shared so there is no local hero or one weak spot.

8. Dependencies

A DevOps team keeps track of all life cycles, versions, licenses and dependencies it uses. By actively managing dependencies a lot of misery is avoided. Understanding versioning and deprecation is mandatory for good life cycle management on dependencies, but also for the application that is built.

9. Autonomy

A DevOps teams strives for full autonomy and loose coupling in the organisation, processes and technology. Hard coupling makes a team dependent on other parts of the organisation or on a certain technology. By trying to interface with dependent things, dependencies are loosened, which makes the team and application more flexible.

10. Responsibility

A DevOps team is solely responsible and in full control for the product they work on. This must be supported by leaders. Leaders are stakeholders and impediment removers, they support the team in improving performance, they should not interfere with the work that needs to be done. If they need to interfere it is via the vision, mission and strategic goals they develop for the team.

Disclaimer: A team practicing DevOps does what's on this list, but a team that does what is on this list is not per se practicing DevOps.

Good is just not good enough

You have tried as hard as you could, only to find out you have been beaten. Sometimes doing your best just isn't good enough. This doesn't mean you didn't try. It just means that someone or something else was better than you. High performance teams want to deliver fantastic products for their customers and they need to work hard for it.

Fantastic products
In order to create great or fantastic products for your customer you need to create quality. You need to deliver your best performance and try for even more. Most customers don't precisely know what they want, but they have a fair understanding of the problem they are facing. They should name their wishes but not the solution. You have a fair understanding of all the possibilities and are in a position to come up with even more after a couple of hours of digging.

In order to create the best solution for a customers problem, find out the problem behind the question they are asking. Try and create a solution to the best of your abilities and include the customer. While you are in the solution process you and the customer are finding even more solutions to the problem you are facing. Fantastic products are fantastic because they seem simple. This often means that the process of coming to that solution is hard. Performance is key to simple solutions. Not text book solutions, but solutions that are the result of a creative process between you, your team and the customer.

What is good, what do our customers think is good
A right solution for the customer doesn't mean it is good enough for us. Do we want to deliver what the customer wants or do we want to deliver solutions that make us proud? Sometimes they match, most of the time they don't. Push your team to the best solution, the one that looks simple but is most of the time hard to obtain. A seemingly simple solution will make the customer happy, make the team better and makes the product easier to maintain.

How do we find the sweet spot
In order to find the solution that fits our customer and our needs we need to find the sweet spot. We cannot achieve victory without making choices. And sometimes we need to make sacrifices. Sacrifices in product creation are abandoning a solution you worked hard on but doesn't do it for you or the customer. It is better to let go and take the loss than to keep on going with a bad solution. Pivot if you need. Our sacrifices can include: putting more hours in, moving a deadline, getting rid of a chosen technology or process, switch people and so on.

Inspect and adapt
View your solution from the eyes of the customer, is it simple enough to understand? Can we use the product without reading an extensive manual? Do we need all the features in order to reach our goal? Is the design or layout correct and intuitive? Have we made a safe or secure product? Can we break it easily?

All these and more questions need to be asked, reviewed and improved upon. Only through regular inspection of the product and the team can we improve.

Do we know better than our customers
Yes and no. We don't know exactly what our customers want, but we know the processes and technology that we need to use in order to make a great product (or we know how to learn it). As a team we know how to work together and get all the important information from the customer. We are ready to change everything we do in order to come up with the right solution. We understand that nothing is written in stone when it comes to delivering the best product to our customers.
Our customer probably know the problem best and needs us to extract it, understand it and solve it. We can do that when we have a high performance team that is capable of customer interaction, that has extended knowledge about the processes and technology involved.

So if you want to be good, if you want to be a winner, if you want to be a high performance team, be ready to inspect and adopt as much as possible. Aim high with the team and the customer. We are not looking for mediocre results, we are not looking for something that just works, we are looking for the creation of seemingly easy to use product, with the highest quality that satisfy our customers more than they expected.