Refinement in Scrum

"Refining breaks crude oil down into its various components, which are then selectively reconfigured into new products." - CEEF

Crude oil is refined into various, better and more defined products. Each end product is of better quality than the crude oil was. Refinement in Scrum is no different. Stories are scrutinized, broken down and up, sliced, inspected and adapted until there is a better quality story. But how does refinement work in Scrum and who should be there?

"Product Backlog refinement is the act of breaking down and further defining Product Backlog items into smaller more precise items. This is an ongoing activity to add details, such as a description, order, and size. Attributes often vary with the domain of work.

The Developers who will be doing the work are responsible for the sizing. The Product Owner may influence the Developers by helping them understand and select trade-offs." - 2020 Scrum guide

It is not clear, by the 2020 Scrum guide, who is responsible and it is not clear who will attend. Since the product owner is responsible for the backlog he/she makes a good candidate to be responsible for the quality of its contents. The developers must use the product backlog items to work with; unclear stories might slow them down. For their own sake, developers are pretty good candidates as well. If we look at who will attend a refinement the best answer would be: those that are needed to make the story understandable. Most of the time this will be the developers or a subset and most of the time with the product owner. Sometimes we might need more developers and sometimes we should even invite the reporter or stakeholder to clarify.

According to the Scrum guide, the refinement (which is mentioned only once) is an ongoing process. And in the old guide (2017) about 10% of the time is reserved for refinement. This is up to four hours a week. This sounds like a lot of time, however in waterfall environments this time has already been spent in the planning before coding even started. Refinement is about understanding what needs to be done and that needs to be done often. My advice is to do refinement each day for an hour, just pick the top stories from the backlog and improve their quality. But don't over do it, refine only a bit more than what the team thinks can fit in a sprint.


To make refinement easier, it's common practice to agree on the quality of the work item in a definitoin of ready. Keep in mind that, just like the definition of done, it is not the purpose of the DoR to be an equivalent of a checklist. But a list of quality requirements.

And one last important aspect of refinement is for the members of the refinement: come prepared. Take some time before refinement to read the work items that are going to be refined, formulate questions or even collect some information up front. In the end it is always about delivering the right product increment the customer asks for (which is always more important than following procedure)

Daily Scrum

The Daily Scrum or Stand-up is a meeting to improve collaboration towards the Sprint goal. In the Daily Scrum we should make a plan for the next 24 hours to get work done which contributes towards the Sprint Goal. In the Daily Scrum we explain to each other which impediments we encountered during the last 24 hours and we discuss how we can resolve those. 

What we shouldn't do is to sum up a list of things we've done. That should have been made clear on the Scrum Board. Scrum is about transparency and the state of our work should always be visible. In the Daily Scrum we ask ourselves what we should do to move closer to the Sprint Goal, we don't discuss all the meetings we are going to today. This Daily event is time boxed at fifteen minutes; those fifteen minutes are all we get to align ourselves in moving towards the Sprint Goal.

The Daily Scrum is held by and for the Development Team (not the Scrum Team). Everybody is allowed to listen and only members of the Development Team are allowed to speak; even if you are a very important manager of some sort, simply hold it. As a Development Team we inspect what we did and what we should do to complete the work we committed to. When there are impediments we should make a plan to resolve those in order to finish the work. When it is clear that work won't get done in the allotted time frame we are obliged to inform the stakeholders to prevent surprises. The Daily Scrum is a tool to prevent (un)happy surprises. When we perform the Daily Scrum well, we encounter problems, but those should not age beyond a day. 

Performing a Daily Scrum well is hard. It will turn into a status meeting if we don't pay attention to the purpose of the Daily Scrum. So here are some tips:

1. Always meet at the same place and at the same time
2. Monitor time
3. Be there
4. Use your Scrum Master to improve the Daily Scrum
5. Don't let the Scrum Master take control of the Daily Scrum
6. Inspect work done
7. Inspect impediments
8. Focus on getting things done
9. Think about WIP-limits: a maximum of jobs that can be in progress
10. Work together, don't sit on your own island of work
11. After the Daily Scrum inform Stakeholders to prevent surprises
    

Scrum, Empiricism and the Scientific Method

The first thing most of us learn is that Scrum is an empirical method. And the first thing we forget about Scrum is that Scrum is an empirical method. What the heck is an empirical method anyway? Let’s find out. An empirical approach is a philosophical approach in science. It is known as the scientific method and is used to find answers when direct experimentation is not possible. The scientific method is an iterative approach. Ask a question, formulate an hypothesis, experiment and analyse. Rinse and repeat when necessary. A part of the scientific method is seen in Scrum when looking at the three pillars: inspection, adaption and transparency. But this is only the last part of the scientific method. The first part is putting up an hypothesis based on a question. So instead of using the User Story technique we will use the Scientific Method to formulate our change request. A User Story is nice to get the features anyone wants on the table, but it lacks the opportunity to fully measure the effects of the feature. The scientific approach would be to formulate a User Story as an hypothesis as an answer to a question. And because the hypothesis needs to be tested and validated, a part of the story would also be defining the sensors or metrics on which to base success or failure. An hypothesis is an educated guess as an answer to a question. Q: “When [we change this] what would happen?” H: “If [we change this] we think [this will happen] which we prove [with this metric]” For example: “If we add a shuffle method to our music player, people will use our player more often which we measure with the increased use of our player in the next three months”. We now have a feature defined in the scientific method with an hypothesis. We also defined the metric to prove or disprove that our change has the desired effect. An hypothesis is true as long as it isn’t proven wrong. So besides proving that the increase in orders in the above example was caused by the new feature, we might as well test if this wasn’t the case. What other factors did contribute? Was the feature solely responsible for the increase in use? Did the usage actually increase? What do we need to investigate next? An hypothesis therefore always has a question and there are always multiple ways to investigate the question with different hypotheses. So the scientific method is about defining the right metrics to prove that a certain feature is making a difference for the customer. Business wise the scientific method would show the hypothesis and the direct benefit of a change. When a metric of the scientific method is defined, there should be a threshold that defines the success of the case, e.a. make it SMART. Using the scientific method will open up the ways we do DevOps, we should be able to integrate even more with the business, with less hassle on feature requests and more prove that what we do actually matters. Sources: https://en.wikipedia.org/wiki/Empiricism https://www.sciencebuddies.org/blog/a-strong-hypothesis https://www.sciencebuddies.org/science-fair-projects/science-fair/steps-of-the-scientific-method?from=Blog

PRINCE is awesome

Using PRINCE for project management is the standard since the beginning of the 1990s. It enforces that projects are done in time and within budget. It provides tools for management to estimate risk, time and outcome of projects.

At the base of PRINCE are the six aspects and project levels. The project levels are very important. Because PRINCE defines different management levels to blame when stuff goes haywire. The aspects give boundaries to the work and can be used as key performance indicators. These aspects are: scope, time, risk, quality, benefits and cost. Aspects are valid for any piece of work that is going to be done, whether using PRINCE or not.

While PRINCE is very flexible in it's implementation there are seven principles that should never be broken. First of all is the business case. This document is sacred, it defines the boundaries of our project and justifies why the project is run. It is a living document and grows with the project till the end. Putting the whole project on paper is a contribution to the success of the project. The next important factor in PRINCE is that there should be a learning experience. So each project maintains a log of lessons learned. This log is delivered and stuffed in a closet, so after the project is finished behind schedule, the log will be incomplete, obsolete and using space. No one will ever look at this log again. Because there is a need to blame people when the project fails, and it will fail, roles and responsibilities are to be defined. Individuals are able to play multiple roles if it suits them, and they should be hierarchical, so propagate problems through a decision tree. At the centre is the project organization, with all contributors to the success of the project as stakeholders. One of the control instruments that is very strong in PRINCE, is stage management. Each stage that will make up the project, is planned and controlled with updating documentation, so there will be no chance to go over time or budget. And because there is a lessons learned log from the last couple of months, there can be a detailed planning of the next stage. Oh, wait, that log is never being maintained, oh well, the planning will then being done as has always been done, by project management. PRINCE depends very strong on a hierarchical structure of the organisation, management by exception is done by escalating problems, until a suit will make a discision on how to solve the issue. PRINCE is not a normal project management solution, instead of focusing on non-deliverables it focuses on products. The products delivered from a PRINCE project are the quality requirements. This excellent product will go right up for sale to the highest bidder. Customers always get thrilled when they can acquire new quality requirements. I might sound a little sceptical on this, but there is nothing more satisfying than brushing your teeth with new quality requirements of a toothbrush. The last principle that makes PRINCE such a strong method, is that it is suited for project environments; it lets you manage size, complexity, importance, time and risk.

With enough management tooling on board, PRINCE is perfect for a hierarchical project organisation where documentation and blaming is at the spill and threads are determined by non-experts. Management is identified as the most important factor in this method and the management products are key to a successfull failure. A lot of time and effort is spend on creating and maintaining these artifacts like approaches, reports and logs. Because a lot of people are working on these management products there is extensive knowledge about the product and problems can be mitigated in advance. By granting control to the least knowledgeable people, and make contributors to the product work on management products, an effective way of working to the end of a product will be omitted. As a result, most projects are a success, but there will be no working product at the end, and a lot of documentation to prove it.

With the new movements like agile software development, scrum, kanban et cetera there is lack of control. Therefore PRINCE2Agile has been developed. While an agile framework is a project environment, it needs to be managed by PRINCE. PRINCE will measure the project outcome, while scrum is used to develop products for customers. With PRINCE decisions will be made by management so the project will be successfull. Risk will be mitigated by placing PRINCE on top of an agile software development framework and makes management responsible instead of software developers.

All in all PRINCE is since it's inception, in either '89 or '96 (depends on who you ask), a leading tool for management. With this tool, management is able to predict when a project should have ended, why the customer get's an outdated product and why it is way over budget and time. While some people might think a customer or the product for that customer might be important, we all know better. Management did what the customer initially asked, the way they found best and have the documents to prove it.

Version control and branching

I sometimes wonder why it was that we started to use a SCM in the first place? And why did we start with branching? What problems did we encounter to start using an SCM and what were the problems that started us to use branches.

At the dawn of time all we worked together on source code in shared folders and a couple of problems arose. First we needed to call out to colleagues which files we were working on, locking out other team members to work on the same files. Second we had no history on the files, when we screwed up, we had to fix it and sometimes that was painstaiking. And third we lacked backups of source files. To solve this we started to use an SCM.
With an SCM we could work on multiple files, without interfering with other team members. There was a track record of our changes through commit messages and we could talk about specific revisions or versions.

An SCM comes with the possiblities of branching and tagging. Soon we used tagging to point to a specific revision in history to mark our releases. And we started using branches to create a personal development space. A personal space was good because most of our team members worked on different changes.

So what was the problem that branches solved for us? When we all worked on different features on the trunk we encountered problems with merging, multiple team members touched the same code, working with our own branch made it possible to develop in parallel. The con is that we introduced merging hell further down the line. Testing was done manually most of the time and we needed to create a solution to waterfall to a release, hence can git flow. A model to overcome the problems with branches. With git flow there are feature branches on which developers can work on their features, run unit tests and deploy to a development area. Then when all integrates well the feature is promoted to the development branch, again tests are run, it get's promoted to a release branch and again some more tests. Then all is merged to trunk and brought to production. Sounds waterfallish to me to be honoust. And git flow is a fix on a fix. It fixes problems on branch development. But shouldn't we just fix branching in the first place?

With a lean thought in mind where waste is evil, branching is a perfect way of introducing waste. It gives developers the opportunity to work on items that are not delivered. When working in teams, expecially in agile teams, the most important thing is focus. Working on features together is more important than working on multiple features at the same time. From XP we learned that pairing results in short feedback loops, improved quality and better architecture. Put this in practise and stop branching. Just work on the trunk. Working with a DVCS over a CVCS makes this even easier.

But when should we branch? Keep in mind that we only work on code that will actually go to production, but sometimes we need to experiment. If the experiments are small, they can be done on the trunk. If they are bigger or oppose the proper functioning of the trunk, they should be branched. Although with feature toggleing an experiment can be done on the trunk as well.
Another reason for branching is that tests are not fully automated or take too long to complete. Improve the testsets and think about what you need to test, instead of running all the tests all the time, be wise.

Altough I was a big fan of the branching model, I now understand that it introduces waste. It overcomes a problem that shouldn't have been there in the first place. Sometimes branching is good, but by exception.

Sources:

1. https://www.git-tower.com/learn/git/ebook/en/desktop-gui/basics/why-use-version-control
2. https://trunkbaseddevelopment.com/
3. https://www.quora.com/Why-do-we-need-branches-in-GIT
4. https://datasift.github.io/gitflow/IntroducingGitFlow.html

Continuous change

"Humans are very proud of their changes, but every imagined transformation turns out to be a new set of excuses for behaving exactly as the individual has always behaved" - Xenocide (The Ender Quintet 3, p36)

Continuous change is the process to facilitate continuous improvement. Improvement can only be achieved by experimenting and failing until there is success. Within lean this is known as plan-do-check-act. We think of new ways to do something, then we try them, we see the results of what we have done and we adjust.

For children this is standard behaviour, somewhere down the line this ability seems to get lost. We grow rigid and conservative when we age. We learn to do things a certain way, no questions asked. How many times didn't one hear: "We have done it like this as long as I can rememeber, why change now?". Perhaps the process is perfect, but it is never wrong to think about it and trying to improve. Only the people who ask questions about what is or what isn't discover new ways of doing things. The people that don't as questions will never grow, never gain new insights and never improve.

Somewhere deep inside most adults the child still lives on, give an adult some lego bricks or a a colouring page and they will be playfull. Only by becomming playfull we can open our minds and think about what we are doing and where we can improve. There are no stupid changes, all changes contribute to a better understanding of how stuff works. Sometimes stupid is good, we learn how not to do something and perhaps we gain insight for the next step to take.

To improve we need to give ourselves time to experiment, small controlled experiments. Change just a little in the product, the process or our lives. Observe and check what the change does and decide to keep the change or revert to the old ways. Some say that we need to change just one percent of what we do each day. The sum will be a remarkable change and hopefully an improvement over a year.

Forbes: What is Agile?

According to this article on Forbes the Agile Manifesto is an historical but outdated document now Agile is no longer only a software development approach but an organizational one. It has become about delighting customers, descaling work, enterprise-wide agility and nurturing culture. Those four principles are easier to understand and remember than the original row of principles in the manifesto. This does not make the manifesto useless, it promotes it to the mindset we need in order to be(come) Agile.