Product Owners don't cut Wood

Does this sound familiar to you? There is this backlog grooming session every week. The product owner would present his user stories. They have title no one in the room understands. They follow the Connextra template of as-a I-want-to so-that. When reading this you sometimes feel like you get a notion of what the title of the whole thing may have meant to mean, but it doesn't grab a hold with you. It just keeps slipping away. And there are these acceptance criteria. In full detail. Describing a solution to a problem that you do not fully understand. Somehow you come to think of it as yet another more or less farfetched special case that has to be added. Everything the product owner tells the team about this story is so rich of detail and so deep into the "How" of it that it resembles a specification. Did I mention that this is the first time you would hear about this particular story?

You feel like you would like to know what problem should be solved, what the customer would like to be able to do, what context this case related to. You would like to know that big picture thing but all you get is tons of technical detail. You feel lost and drop out of conversation.

Ever made this experience? No? Lucky guy you are. The whole thing wouldn't be too bad if it would just be wasted time spent in a boring meeting. But it's worse than that. There are two options.

First the team ignores the very detailed spec like acceptance criteria when implementing and the product owner barely checks what he really gets. The consequences are that the product owner thinks he has got a certain product but the team built a slightly different one. His understanding of the product will run out of date and so do his user stories.

The second option. The team actually builds what was requested but does not take the responsibility for the product. The product will start to resemble a bunch of special cases. Similar things are not done in similar ways. Features start to become mutually exclusive. (Yes. This could happen. Have seen it ...) And the code base reflects this. Adding new features takes longer every time. Even minor changes take ages. Bug rates increase. You are in a mess. You know you should clean this up. But you don't know the direction or any direction at all for all you know about the problem space is detail.

In both options the team will have a hard job to build an identification with the product they build.

Cutting Wood

Do you smell the smell? What's wrong here? Taking it to an analogy the product owner takes the team on a tour to the woods. They walk around and pick a tree to cut, and another, and another. Here and there. Randomly it seems. Then he picks them all up and drops them in another forest where they pick another set of randomly chosen trees to cut.

No one has any idea about what made the tree so special to be cut and why the tree next to it did not qualify. No one has any idea about where they are how they got there in the first place. They know the trees they’ve cut and for there is no real memorable connection between they might have forgotten about a number of them.

What the team misses to see, is the landscape around them, the patches of trees, the forests, the meadows and creeks, lakes, hills and pathways that form the world they have to move in. They have no way navigating their surroundings on their own. They depend on their guide. The depend on their product owner to tell them what to go for next.

This puts the team in an uncomfortable position. It cannot fulfill the role they are supposed to in Scrum. They are not in a position to act at eye level with the product owner.

Shape the Landscape

To stay in the picture the product owners job is to shape the landscape to develop a big picture of where to place a forest, a lake, a creek. Where to lay a path or spread a meadow. The detail work will be done by the team on its own account. If a forest shall be then trees have to be planted. If there has to be a simplification some trees have to be cut.

The team always sees the tasks in context with what else is required. They can bring their detail knowledge to the big picture to help refine and adjust it if needed. They have the chance to understand connections between tasks and might offer shortcuts or optimizations.

What a Product Owner should do

The job of the product owner is to take care for the big picture of the application to be built. She has to balance the sometimes conflicting requirements. She has to balance the stakeholders to keep them happy and satisfied with the product and the way it moves on.

The product owner has to serve as the on-site customer for the development team. She has to convey the requirements and the context they come from to the team. She has to understand the problems that should be solved and what the customer really wants to be able to do. She is the person that connects the team to the problem domain and has to make sure the domain and its terminology is known by the team.

Especially the last one. User stories should reflect the problems within the domain language. Acceptance criteria have to use the domain language to describe what should be possible when implementation is done. The product owner is a major player in building the ubiquitous language both the customer and the development team understands. Just as described by Domain Driven Design. Using this ubiquitous language opens up opportunities for the team. It offers possible abstractions they could use in their implementation. Something a very detailed and technical description of the problem never could provide. It helps the team to see connections between features and to further drive abstraction in the code. And with that it helps to improve extensibility and maintainability of the code make sure the speed of feature development could be persisted over a long period of time.

In a way one could conclude that the way a product owner writes his stories she could drive a healthy code base or a horrible one. It depends.

Read also:

On Coaching Agile - What I've learned from being an agile coach

On Sporadics - How to deal with intermittent tests in continuous delivery

On Agile Testing - Do we need testers?

The opinions expressed in this blog are my own views and not those of SAP

There Are No Agile Testers - Balancing the power

This is a follow-up on my posts regarding developer testing versus tester testing which came to the conclusion that rather than having a QA department or a micro QA embedded in an agile team in form of a number of testers I'd prefer to have developers taking on a QA and QC perspective in their day-to-day work. All of the posts negated the need for a QA department.

I intend to change my mind. A bit sort of. at least.

Feature vs. Quality

The situation that induced this partial change of mind I could observe over quite some time now. In an ideal world development teams would take on their responsibility for testing and a reasonable QA. There are places where this actually happens. And there are many more places where this turns out to be a challenge for the teams even if they are willing to do so.
In many shops development teams are faced with a high feature pressure imposed to them by product owners and/or management. Ever more features are requested and sold before they are built. Development teams get forced to deliver even if the quality of the features are not at a level that would suite their own standard. Nor does it fulfill the standard the customer expects.
The quality often is so poor that merely the happy path works. Any step aside would lead to trouble, meaning bugs being reported, adding to the ever increasing demands the development team has to keep up with. In the end the same people that force the team to deliver buggy software are the same people that complain about poor quality. Often I have been witness to conversations like:


PO: I want that feature end of week.
Dev: We are not able to deliver that soon. We're not done. There are some issues to tackle yet.
PO: Doesn't matter. I take the risk.


To be sure. The PO would say anything to get the pressure of his chest by delivery anything he could lay his hands on. He knows very well that he is not taking the risk at all. The development team will have to handle the outcome of the unduly risk taking.


The White Knight: QA

Here is where a QA department or QA stuff comes into play. Given the QA lead is equipped with the same prestige and rank as the management or product owners that force the development teams into bad behavior, the QA could serve as a support for development teams in their effort to ship quality software instead of premature features. The QA is there to balance the power sort of. Putting weight on the side of quality.
It is not about huge organizations of testers swarming out to test all the bugs out of the features. It is about a department that is able to define minimum quality standards and KPIs that have to be met before anything leaves development and that is able to enforce them.
Sure. This is adding pressure on the development teams from one side but reliefs them on the other side. QA will demand the minimum standards are met by development. And QA will prevent stuff from being shipped if the minimum standard is not fulfilled. Thus achieving what development teams are not able to achieve for lack of power.


Balance of Power
To sum up all of my posts: development teams have to do their testing themselves. It makes no sense to build micro silos within teams by adding testers that do testing and nothing else. There should be a convergence of tester and developer to better be able to load-balance within the teams according to the needs of the features to be built. However there are testers that turn into test enablers by providing test automation tools to the development teams or by educating developers with respect to testing. There may even be some testers that do some very specialized testing like user acceptance testing or usability testing. And finally some of the testing stuff will turn into QA - or remains there - to serve as a counterweight for over-ambitious product owners and managers.


So, finally I admit: There is a need for a QA team. A team to set standards and to enforce them. The obligation to fulfill these standards lies with the development teams. The tools and knowledge they need will be provided by former testers.

Read also:

On Coaching Agile - What I've learned from being an agile coach

On Sporadics - How to deal with intermittent tests in continuous delivery

On Agile Teams

Part 1: Agile Testing Days 2014: There are no Agile Testers

Part 2: Agile Testing Days 2014: There are no agile testers - and no agile developers either

Part 3: Agile Testing Days 2015: There Are No Agile Testers - There Are Testing Facilitators

The opinions expressed in this blog are my own views and not those of SAP

Do experienced developers have to learn?

Just recently I’ve had the not so nice experience of ignorance presented publicly. There has been a session on software engineering a colleague and I compiled for our department. The narrative was around extensible architecture a topic that would need some promotion round here as we are faced with a bunch of legacy code.

Virtually no-one showed up. Afterwards I figured that they were too busy or just not convinced that a topic like this would apply to them for they are experienced developers with some 10, 15 or even 20 years of experience in corporate software development. They just know how to code.

How did actually write all this legacy code, then? I wonder.

What happens here is a nice little mistake: experience == knowledge applicable

This is no new situation to me. I’m faced with claims like that all the time. Just like a UI developer once told me that she would not provide any mock ups for the team to discuss and commit to for she would be an experienced UI developer who knows how to build UIs for years. Again a manifestation of experience outweighs everything else.

What’s wrong with a developer so convinced and self-assured that he would give a statement like this?

Basically nothing when it comes to self-esteem. Basically everything when it comes to the ability and will to learn to progress in ones skills and abilities.

The guys giving these statements were the ones that showed only little impulse to question themselves to reflect on their skills and how they fit to the needs. It definitely has consequences when a skilled procedural programmer tries to use her skill on an object oriented language and environment. This would result in a legacy code base that is not extensible, supportable or testable, as could be observed in our code base.

Why does some otherwise intelligent person fail to see this difference? Do they really not see the difference? Or is there something else?

I didn’t come to a conclusion about this yet. My hypothesis circles around uncertainty, fear of loss of control, self-betrayal and plain ignorance.

Thinking about being in the situation of having 20 odd years of experience in software development (which I have) and considering myself an expert (which I do) and someone not even working in my problem domain coming around trying to tell me how to do things differently (which I basically did) I guess I would have and show my objections to that person.

In the end this person would question me. The expert. Would basically say I did something less optimal and perfect than I would consider it myself to be. I could imagine myself not giving a damn about this. I just would not feel comfortable with admitting my work over years has been less than excellent.

Switch back. While this feelings are just and cannot be ignored there has to happen something to improve the code quality, the design. In my opinion the admittance of less than optimal work, the admittance of failure would be to closely related to failing as a person, to closely related to not getting the bonus granted by the boss. The culture of failure needs to be there. A culture where making and admitting a failure would be understood as a learning opportunity. A culture where there is no need for a never failing expert who tells everybody who things are handled here and who has to approve any idea. We need a culture where true expertise shows itself in the experience that everyone will make mistakes and the simple thing that one single person could not know everything. Even an experienced expert level developer could learn something from a complete newbie every now and then. 

Read also:

On Coaching Agile - What I've learned from being an agile coach

On Sporadics - How to deal with intermittent tests in continuous delivery

On Agile Testing - Do we need testers?

The opinions expressed in this blog are my own views and not those of SAP

Agile Testing Days 2015: There Are No Agile Testers - There Are Testing Facilitators

I had the opportunity to attend Agile Testing Days 2015 in Potsdam. It's been the second time. And again, there haven been great sessions, inspiring talks, eye-opening chats. But still there is something that bothers me. If you following my blog post you might have noticed the "There Are no Agile Testers" blog posts:

Agile Testing Days 2014: There are no Agile Testers and 

Agile Testing Days 2014: There are no agile testers - and no agile developers either

To say the least they have been received a bit controversially. Many testers felt personally offended. I could understand this to some extend. A year has passed since then and the idea circulated in my thoughts. I was trying to understand what really bothers me about the agile tester thing. I'm not sure I'm done with it yet. But new aspects revealed themselves. 

The thing that bothers me the most is the fact that by just bringing testers to agile teams does not solve the issues we've had with the development and QA departments. Testing still comes last in the development cycles and tends to be skipped or blamed for being a bottleneck. This is nothing I made up, but something that has been said by testers at the conference. In a way the testers in agile teams establish a silo just as the developers do. Developers rely on testers to get quality into the product and complain if testers do not manage to handle the amount of user stories done for coding stories tends to be faster then thoroughly testing them.

Over the years I became more and more opposed to silo thinking in teams. This discomfort still grows. I try to find ways that could help to overcome this dangerous tendency. My experience from many years shows that whenever a teams starts to separate into silos team performance, quality, and outcome drop dramatically. The teams start to dissolve and I've even seen teams to dissect.

A second aspect that I grow ever more uncomfortable with is the fact how developers are pictured as guys not willing to test, not willing to care for customer needs, not willing to care for quality. A great many developers may fit this description. But another great many developers care for concepts like Continuous Delivery, Lean Startup and DevOps. Both of them are heavily relying on being responsible and accountable for ones quality. Developers show that they are willing to produce stable, high quality code that covers actual customer needs. That they are willing to measure customer acceptance and to act accordingly. That they are willing to ship to production as often as possible. I reckon (a new generation of) developers understand(s) pretty well that they are no longer sitting in the basement coding all day long without ever bothering themselves with any consequences their work might have for the world around them.

For quite some time now developers proved to be no good at testing. Whether they are just testing agnostics, arrogant my-code-will-not-break guys or anything you might think they are doesn't count. Testing did not take place in a way and amount that would have been desirable. Because of that QA people had to be hired to clean up the mess they left behind. But no one bothered to tackle the root cause: Improving quality of the code from line one. This would have meant one has to deal wit these strange guys in the basement. So, an opportunity has been missed. The mere existence of a QA department that made up for the mistakes developers made encouraged them to code even more carelessly. There simply has been no need to do otherwise.

It is time to reverse this development. Now, as developers develop a sense of responsibility testers are urgently needed to share their knowledge and experience gained over so many years of testing. This knowledge has to be shared with developers. Testers are urgently needed to challenge developers to take testing beyond unit testing seriously. There is far more to testing than that as one could learn form "Agile Testing" by Janet Gregory and Lisa Crispin. 

Me, being a developer for most of my professional life, I would wish if not expect from testers that they make testing an integral part of a developers daily business. Testers in agile teams have to become test facilitators. There is no way around that. If an agile team would be staffed with as many testers as you would need to make sure all user stories are covered with acceptance tests, all new code is covered by unit and component tests, all security, usability, performance and you name it what test are required up to thorough exploratory testing one could easily end up with maybe 3 (or more?) testers per developer. Would this be the way you would like to go?

In my humble and honest opinion we would need to tackle the problem from two sides.

1. Testers Coach Developers

Testers gained insights on lots of different aspects of testing including experience of typical hot spots especially when it comes to integration testing. Testers would need to pair with developers to support them when writing these kinds of tests figuring out what test cases are needed and how to best test them. It may be that while doing so testers become familiar with development itself and eventually cease calling themselves a tester. I would consider this a great achievement for our industry when testers did their coaching job that well that developers are able to do the necessary testing and former testers turn to writing code themselves. In a way we would have to call them all Agile Engineers. Then Continuous Delivery and DevOps would fully unfold their potential.

2. Testers Provide Automation Frameworks

Not all testers would like to turn to bare development. I would propose another direction for them. Many developers do not care for security or performance testing because there are no frameworks and no infrastructure available that would perform these tests reliably and easy to write and evaluate. Any of these frameworks needs to be made available in build and test pipelines. If they are not available that way they will hardly be done. Developers need to be urged into writing  these kinds of tests by making this unavoidably easy. Whether or not these frameworks have to implemented from scratch or could be bought. Someone has to set them up for use in pipelines. Someone has to educate developers of how to utilize them.

3. Testers as Integrators

Developers tend to be off by one, so am I. There is a third category of testers. The ones that never ever wrote a line of code and are not fond of doing so at all. There are businesses that have to fulfill legal requirements with respect to quality assurance. There businesses that built huge products with millions lines of code contributed by teams not at all co-located to each other and often enough not well connected. Products like that tend to have integration issues and no one feeling responsible for them. These are areas testers in a more classical sense still would be needed without the urge to turn into developers.

Conclusion

Testers in agile teams should try to see themselves as coaches and facilitators to spread the art of testing. Developers need to be educated and enabled to do lots of testing on their own while and before writing any code. Developers need to learn to look at what they do from a users side in order to enable them to decide in favor of a users needs.

Testers could provide frameworks for automated security, performance, product life-cycle testing and alike. These frameworks have to be made available to developers in their daily work to make these kinds of testing an integral part of coding well before a user story is labeled DONE.

The tasks testers will face in the future might change. For some this change may even be dramatic. But I think we could not afford moving on as we did before. It is time for Developers 2.0

Read also:

On Coaching Agile - What I've learned from being an agile coach

On Sporadics - How to deal with intermittent tests in continuous delivery

On Agile Teams

Part 1: Agile Testing Days 2014: There are no Agile Testers

Part 2: Agile Testing Days 2014: There are no agile testers - and no agile developers either

The opinions expressed in this blog are my own views and not those of SAP

On Coaching Agile: A Matter of Gray - Don't let prejudice guide you

Just recently I was receiving a mail containing a quote which went like this:

"I get paid for features not for tests"

It has been put in a context where we were discussing something like code coverage or test coverage issues. That's the setting. What happened next?

To me this quote in this very moment and this very context has been proof that there something was totally wrong. Based on this quote we would not need to talk about application of agile methodologies with these guys any longer for they are not prepared for it. And I went off into my standard narrative about ignorant developers and ignorant management, that most of the time were complaining about what went wrong and that everything would have to change but no one would ever start doing anything least did they question themselves. And on and on it went.

"I told them many times and they just don't understand. They just ignore the evidence", is what I could tell myself over and over again.

That very same day while taking a shower in the evening it struck me: What did I know about the quote?

Well basically nothing! Who did say this? I don't know. What has been the context it has been said in? I don't know. Was there anything else that has been said to put this into relation? I don't know. How was it said? Angry? Regretfully? Complainingly? Resignedly? Ironically? I don't know! Anything? I JUST DON'T KNOW!

What I did know, it felt perfect to me. I could be the good guy knowing about all the "right" stuff, doing all the "right" stuff - well most of the time. And they (as anonymous as could be), they are the guys doing it wrong. Again. And over and over again. With that two things are for sure:
1. My work as an agile coach would never ever come to an end, and
2. I would always be the good guy. I could always feel superior to "them"

A question arises here: Could I really be a coach to "them" when I consider myself superior? Would I ever be able to bring my message across? Would I really like to bring my message across to help "them" get better?

To be honest the answers would be something like: I don't think so. No. To some extend, yes.

At least in retrospective I would need to admit that "being better than them" gave me a feeling of comfort, a feeling of importance, somehow even a feeling of doing the right thing which supports me in trying to move on despite the little impact one achieves from time to time. So, in a way this attitude supported my will to strive for a change.

Thinking about this I figured that I would have to deal differently with the quote. First of all. Listen. In this case I wasn't able to listen for I've got this second hand. So it would have been asking. Asking all that questions to fill in the obvious gaps, to understand what the speaker really wanted to say. Only then I would have been able to build an opinion about that. Only then I would be able to decide on which steps to go the mitigate a possible issue that could have been meant by saying what has been said. - You see, all purely conjunctive.

What did I learn from this?

Instead of speculating about the manifold of possible reasons one should try to get a hold on the real reason. If you can't just let it go.
Do not exploit a quote to make a point. You could try to do this with the very person the quote came from. You would be spoiled right away. Nothing could reestablish the lost trust.
Do not preach your solutions. Try to understand the issues your coachees have and help them solving them no matter whether your prepared solutions or methodologies could be applied or not.
Don't exaggerate your position. You are not the guy anyone is supposed to follow, the guy that knows. You are the servant. You should do what many blogs, articles and books tell about coaches:

You are a facilitator, enabler, mentor, partner whatever the coachee needs at this very point.

After all the most important thing you have to be is being

humble.

Read also:

On Agile Teams
Part 1: Agile Testing Days 2014: There are no Agile Testers
Part 2: Agile Testing Days 2014: There are no agile testers - and no agile developers either

On Coaching Agile
Part 1: Killer Questions - What do you want to test?
Part 2: Techniques - Make a Bold Statement
Part 3: Techniques - Show and Tell
Part 4: Agile Methodology Breast Feeding

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

Agile Methodology Breast Feeding

How come that many of us developers are struggling with agile methodologies like test-driven development or componentization/modularization of a software?

I was asking myself this for a long time now. Being an agile coach I frequently see developers fail applying these techniques in their daily work. Intelligent people that are able to write code to take care of the most complex problems around. And yet an apparently simple technique like write-your-test-first doesn't get a grip with them. Why? 

In the past I tended to blame the workload and pressure of management and the urgent need to put out the next feature and QA it afterwards that somehow caves in the people into this treadmill of assembly-line work like feature production. In this treadmill they are not able to get the time or freedom or even will to learn something new, to question what they are doing, or how they could do better.

It's easier to just do what one has always been doing for this they would know, this would require less efforts and less energy. Change on the other hand comes with inconvenience, with insecurity, with the need to work more consciously which slows down compared to working with a routine.

So, changing the approach developers use to get their things done becomes a long hard process. Sometimes it feels like a behavior therapy with me being the therapist. 

But I get carried away - a bit.

Just recently I came up with another source of evil, though: Textbooks!

I was curious about JavaScript and node.js recently. What would you do to get a concise guidance on any new language? Well, me I would buy a good book and work through it - or google a good tutorial on the web or something.

This time though I wanted something else. For me being an agile coach I wanted to do what I am preaching. Do TDD right from the start. So I skimmed through the pages and in the index I found "Testing" at page 10-something. Wow! I thought. There is something about testing and it's quite early in the book.Promising! 

Well. what a disappointment. It simply suggested to create an index.html file where the JavaScript would be written to or linked and to open this in a browser to witness the effects of my changes.

That's all to it.

After this the whole thing was like every time: Gradually introduce the language and write some code to learn it.

That way I learned some other languages as well. It never ever crossed my mind that I should write some tests first. Instead I wrote a few tests afterwards. Just like the others did. Fine!

No!

Thousands of young people learned their first programming language that way. They did not learn about testing. They got the notion that programming meant just writing code that accomplished the task at hand. Then they would discover that there are bugs in what they wrote and they learned how to debug even nasty code. Some of them are rather skillful debuggers and learn about seldom used features of their tools. They feel like the best programmers around for they would nail down even the strangest bugs. Bug fixed. Fine. No test needed. The bug has gone, hasn't it?

I've seen many of these guys entering corporate software development where they would move along their way and would just hack on a bigger scale. 

Try to tell these people that there are other ways of doing it. They imbibed programming with no testing from their infancy. It's hard to change something about it. They would lack the awareness of potential problems with that approach.

I'd plead for a new style of textbooks. 

It came to my mind that the textbook I would like to read (or even write for that matter) would be different. I would like to introduce a language TDD style. Explaining testing first. Why not writing a HelloWorldTest first? It's a program as well, isn't it? It's a minor change of perspective but a huge change in mind. 

Tell you what, while thinking about this idea I stumbled across this book on node.js:

http://leanpub.com/nodecraftsman by Manuel Kiessling.

To my big surprise it did exactly what I would like to do: it started introducing node.js with TDD. Thanks for providing this enlightened little book!

Read also:

On Agile Teams
Part 1: Agile Testing Days 2014: There are no Agile Testers
Part 2: Agile Testing Days 2014: There are no agile testers - and no agile developers either

On Coaching Agile
Part 1: Killer Questions - What do you want to test?
Part 2: Techniques - Make a Bold Statement
Part 3: Techniques - Show and Tell

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

Agile Testing Days 2014: There are no agile testers - and no agile developers either

Just recently I wrote a blog post in response to the Agile Testing Days 2014 I was glad to have attended. This post was received controversially, to say the least. There have been many in favor for what I was saying, and there were many that didn't like it at all. For several reasons. I've got involved with interesting discussions with many of my readers., most of which led to the conclusion that there has been some misconception due to the fact that some of my arguments presented in the conversations were not mentioned in the post. I promised to write a follow-up to get to the point I was trying to make with the first post but didn't manage.

Here it is.

My Background

For there have been some not so polite responses questioning my right to express my thoughts regarding testers and testing I will explain where I come from and what my premises are. Currently I'm in QA for a 2M lines of code rich client application. As QA we define the processes for the development teams to follow to ensure a high quality of the product including everything from continuous integration, automated testing up to E2E and exploratory testing as well as static code checks and security scans. We follow the goal of establishing a continuous delivery process for this application.

My background over the years, however, was in development. As a developer I've come a long road from being a typical developer (from a testers perspective) to becoming an coach for agile methodologies (including agile testing). At certain times I did user acceptance testing and scripted testing. So, I've done my fair share on almost everything a tester would usually do. But my upbringing as a developer certainly makes sure I'm not completely familiar with the terminology a tester would use. Please forgive me this lack of knowledge but rest assured it is not because of ignorance.

Hypothesis

The existence of agile testers (aka testers as members of agile teams) prevents the urgently needed mind shift development has to undergo and the existence of agile testers does prevent agile teams to become what they are supposed to be: agile small units that could build and ship the highest rated new features with nearly constant velocity. 

Waterfall

Development departments have a long record of software products not shipped in time and not having the expected quality. There have been many ideas around how this could be improved. The most successful of these has been the introduction of a QA department. Just like in any other industrial production QA would check the product after it has been assembled, and before it would be shipped to customers. For cars or machines this worked out pretty well. So, why not adopting this approach?

Development was known to deliver bad quality. QA would establish a Q-gate for the product to pass. Testers would test the product thoroughly before they would turn the thumb up. Fixes for issues would be requested and delivered by development. Sounds great. But virtually never worked out. Why? There always is the temptation to squeeze yet another feature in and yet one more, thus stretching the time for development beyond any deadline. QA would suffer for they are required to accomplish their tasks in a much shorter time frame then originally planned. Issues they found did not get fixed anymore for release date approaches to fast.

In such environments the guild of testers had to develop methods, techniques, and tools to somehow manage the increasing workload in a shortening time frame to work in. They had to mitigate the lack of information, the lack of involvement, and the fact that they were forced to test at the wrong end of the cycle. They sure would develop a pride in their work and their abilities. For they would act closer to the customers and would receive complaints and issue reports their understanding of customers needs would be better by far compared to the understanding a typical developer would have. They sure took pride in that as well. And rightly so.

Turning an eye on the developers. Before QA has been around they at least had to do some testing. Now there would be QA testing just enough quality in. Fine! More time to develop the features development was forced to squeeze in. It wouldn't come as a surprise if a developer wouldn't even felt relieved of the burden to spent valuable time on testing instead of walking further down the list of features he was supposed to finish. Once QA came into existence developers were disconnected from customers by yet another layer. On the other side of the process there were architects or system analysts that would lay out the whole product before even the first line of code has been written. They would talk to customers to collect requirements. Then they would put down all their design documents and contracts and stuff. And the developer had to walk down the list. Completely disconnected from the source of the requirements and the consumer of his work. They got cushioned in between these two layers. What would you expect them to become within this sandwich. No time to really change the architecture if they would find a flaw. No time for that. Just keep working around it. What connection to your work would you develop if you would have no means to influence major parts of it? A certain archetype would be required to survive in such environments. Nowadays we call them developers (at least when we stick to the cliché). This typos gathered in development departments. At one hand cushioned in and pampered, kept away from customers. At the other in splendid isolation just doing geeky stuff feeling proud of their ability to work around every problem in the shortest possible time.

To be sure: Not every developer fits into the cliché. Just like not every tester fits into the cliché built by many articles and books and which has been presented to me at Agile Testing Days this year or in replies to my postings. Things are not black or white, as usually.

Agile Teams

Fortunately there has been developers that did not fit into the cliché. Guys who wanted to do things differently as well as other people in this industry from other roles. And over the years ideas like XP, agile, scrum, and lean were invented. Now we are faced with agile teams. Suddenly things like backlogs, priorities, and short cycles matter. Frequent shipment of high quality product increments were expected. Customer feedback which leads to major adjustments in the product would be welcome. Imagine a developer from the safe haven of the good old days when they were supposed to just code and a whole organization built around them would take care of the rest. What would he make of this?

Imagine a tester with his safe haven QA department. Far away from developers. Now she was supposed to join teams with the developers. Two parties were put together that felt comfortable with the prejudices they had on each other. What would you expect to happen? Well, the safest thing would be to keep the respective specialization and organize the agile team like the departments they came from. The developers in the teams would, well, develop and the testers would just test. In fact the agile team would turn into a micro version of what has been the organization back then. I have seen this happen several times. And this does not seem to be special to me. Many of the sessions at Agile Testing Days shared the same experience to some extend.

Even the bible for agile testing the book by Lisa Crispin and Janet Gregory did elaborate on this. I like this book it has got a lot of knowledge and experience in it. I certainly like the insights from their experience over the years they would share in there. And it has the right title for sure: Agile Testing. I will come back to this just a little bit later.

Part I of the book elaborates on testers and what they could bring into an agile team. In what sense they are special and would yield a great benefit for the team. Yes, testers have their unique specialties. Yes, these are badly needed close to development and even before the first line of code has been written. We definitely would need more agile testing in agile teams or in software development in general. 

At some point in this part the "Whole team approach" was mentioned and welcomed. No distinction between testers and developers. Just a team working on the top priority tasks doing whatever has to be done by anyone available when a new task has to be worked on. The agile team as a bunch of generalizing specialists. I very much like this idea. In fact that would be the Holy Grail. An agile team that would be able to really work according to priorities.

Unfortunately the remainder of this part very much keeps the distinction between testers and developers and tries to explain why a tester would be required to write acceptance tests, to talk to customers to understand what he really needs and similar stuff. There is one question popping up my mind: Why would I manifest the old DEV - QA pattern in an agile team? Why would I still keep developers away from activities they would urgently need to do to improve their understanding of the product or the customer needs. If they just be kept away from these experiences, then they are just cushioned in and pampered as before and we are all no better off. 

We would still play this DEV & QA game. Testing happens after implementation. I have seen this happening: Testing gets squeezed in between end of implementation and sprint closing. Often resulting in not properly testing for lack of time. Letting bugs slip through and so on. Where would be the difference to the old days? Testing needs to be first priority. To get this done anyone in the teams needs testing skills to some extend.

If you really want developers to change, to develop new capabilities you need to let them do those things they never did before. A tester would be a great teacher to them if he shares his knowledge with them. A developer needs the feeling of pain a bad quality product gives the customer. A developer needs to write acceptance tests herself to understand what a user story requires them to produce. These first hand experiences are what would trigger insights for them. 

Are developers afraid of this? Yes. Are developers not prepared to do something else than develop? Yes, for sure. We are talking about change. Change leads to a feeling of uncertainty, leads to occurrences of not knowing what would be expected of them. Change brings about fear. Same goes for testers. Are they really required to code? Are they really required to do architecture? Bug fixing? Yes they are. In an agile team. And this brings about fear, uncertainty, feel of loss for them too.

So, what happens most of the time? Developers and testers comprising a general non-aggression pact. And each sticks with what she could do best. Keeping specialism alive. And this specialism yields problems itself. Specialists are only able to work on a limited amount of tasks. Those who they do not cover with their specialization they would not take on. Specialists tend to be bottlenecks if their special knowledge would be required by two concurring tasks one would be blocked. Specialism leads to situations where the user stories not get worked on in the order of decreasing priority but by the need for some specialist. Thus the next increment of the product will not contain what the customer needed the most but the features the team in its combination of specializations was able to get done. This would not comply to the ideas of agile anymore. A team like that wouldn't really be an agile team.

Agile Engineer

What would that be? She would be a generalizing specialist. Willing to learn new methods, techniques, new skills. A member of the team that would be able to take on almost any task belonging to the most important user story on the board. If writing some code it would be, then she will code. If it is testing, she would test. If it is something else, she would do this as well. She would not be afraid of asking when she lacks knowledge or experience. She would be willing and open to share her knowledge.

Okay, lets go with agile testers and agile developers. We need to start somewhere. But support them in becoming an agile engineer. This will be a though transition for many of them. They would need help and guidance on the way. And they need the right environment. Just because an organization decides to have agile teams and to go agile doesn't mean that there is anything else than a new label put on the stuff they have. An agile engineer would not yields the effects he could if the organization around him is not transitioning too. Fixed schedules, fixed resources, and fixed scope do not work with agile. If the organization is not going agile anything else could remain as it has been back then.

Agile engineers would build great agile teams working in great agile organizations. They would DO agile testing, would DO agile development, or agile planning. They would cease to be an agile tester, an agile developer, or an agile someone. This would be the time when there would be 

No agile tester - and no agile developers either

Read also:

On Agile Teams
Part 1: Agile Testing Days 2014: There are no Agile Testers

On Coaching Agile
Part 1: Killer Questions - What do you want to test?
Part 2: Techniques - Make a Bold Statement
Part 3: Techniques - Show and Tell

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

On Coaching Agile: Techniques - Show and tell

When coaching agile methodology as well as with any other coaching you are supposed to make your point in a way that coachees are not only able to repeat what you have said but to get the idea of it. They are supposed to be able to adopt the principles to their own daily work. This will only work if you get their attention and to make them listen instead of just hearing. If you do not manage to achieve that you could as well talk about little elves in the trees. The outcome would be the same with regards to agile methodology.

So, what stands between you and a coachees attention very often sums up to pure prejudice, which could be translate into one simple sentence: "This won't work for our code." Very often this simple statement prevented my coachees from listening in the first place. They would attend the training for they are supposed to do so, and they are willing to stand it, somehow. These people will not listen, they are not open to what you want to say. To get their attention is to get around their prejudices. How would you do that?

Many people these days want to publish their own book. There are a lot of book on demand offerings where anyone who hears the call could publish his or her book dreaming of becoming the next Philip Roth or J.K. Rowling. Unfortunately most of them are not at this level. However, they enjoy what they are doing. Many attend creative writing classes to learn about how a novel would be written, what principles should be followed. The result is there are many novels around that reflect these classes teachings. They share the ever same plot structure:

Plot point 2 - Mid Point (Crisis) - Plot Point 2 - Dark Moment (Even deeper crisis) - Resolution - Wrap up

Very seldom there is development, own ideas to what a plot could look like. Most of the time there is just repetition of what has been told. They heard but did they really listen? Has there been something to listen to?

Another shortcoming of creative writing novels are the fact that they tend to TELL the reader a story. Just like this one here:

"Pat has left. Tom was very angry in one moment. The other he was wiping his eyes out. He felt sad and empty, changing moods with a blink of an eye."

Everything said. Does this catch you? Probably not. There is no flesh to the bones, no room for my thoughts to flow around a scene, to come to conclusions about it. Thinking was done for me. These sentences are gone long before the end of the novel. Same goes for many coaches. They would just stand in front of a room full of coachees and read their slides of have their say. People try to get through it without snoring. If I would be a coachee in there I would still have my prejudice and nothing could be done about it. Once out of the room I would have forgotten what might have been said. Why?

Well I just were told something. No one tried to make me thinking.

This what differentiates a serious writer from a creative writing writer is the ability to create scenes, to show me places and people, to make me think about them, to give room for thoughts and still be in control of the flow of the story. The writer makes me think and guides me through the story. Just like this:

"Photographs littering the floor. Pat and Tom at some lovely mansion up in the hills. Pat and Tom diving. Pat and Tom at many places. Pat kissing Tom. Tom hugging Pat. Some of them are torn, showing either Pat or Tom smiling at someone who's no longer there. Tom was standing at the window staring into emptiness with a blank expression. A vase lying shattered between his feet. His cheeks were wet of tears. Suddenly he turned around, kicked a photograph into the next corner and screamed at the top of his voice."

This scene shows the moods Tom are in. The reader is able to think, to figure out what's going on. She will be involved with the scene for the writer SHOWed it to her. And this is what we want to achieve in coachings as well. Get the people involved by showing what me mean. Give them a chance to come to the conclusions themselves instead of telling them what you already know and they do not know where this has come from.

A very nice example I recently was able to experience has been the session TDD and Refactoring with LEGO at Agile Testing Days 2014 in Potsdam, Germany. Bryan Beecham and Mike Bowler showed the effects of TDD, refactoring, technical debt and lack of communication between teams working on the same project with simple but very effective hands on exercises using LEGO bricks. You could physically feel the heavy load of technical debt. You could physically feel the effects of refactoring and TDD. Whether or not you come with prejudices and the strong feeling not to be willing to get involved with this the pure fact that you would be allowed to play with LEGO makes you letting down your defenses. And this is what opens up your mind just enough to be vulnerable for new ideas. When doing my next class room training I want to try this instead of reading and explaining my slides.

Another approach to get around the prejudice and especially to get around the this-won't-work-for-my-code statement would be to actually show that it works for their code as well. Me and my partner spent some time upfront the training to lay hands on the code of the team to coach. We would let them show us some part of the code they do not unit test and they think not to be unit testable at all. Within a week we would create the moving parts required to build and run a first unit test suite. And we would do just enough legacy code magic to get sample unit tests done that proof the approach works. Within the training we would have a session "Legacy Code" and the examples their are taken from the teams code base. This would be the time when we show them what could be done to their code. And guess what we got each time: Wow! No other part of the raining got them more involved and open for discussions than this one. This usually would be the time to go back to refactoring and TDD principles, to go back to test isolation. And every time we did this the teams picked up from there to grow their unit test base steadily. We just had to plant the seed and SHOW them how it works instead of TELLing them that it would be possible.



A checklist to summarize:

Pro:
- makes people think
- gives room for own conclusions
- opens up peoples minds just enough to make them join the discussion
- this-won't-work-for-us will not work anymore

Con:
- for the LEGO thing: the session is harder to control
- preparation becomes much more expensive
- you would need access to the teams code base and some initial support

Read also:

On Coaching Agile
Part 1: Killer Questions - What do you want to test?
Part 2: Techniques - Make a Bold Statement

On Agile Teams
Part 1: Agile Testing Days 2014: There are no Agile Testers

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

Agile Testing Days 2014: There are no Agile Testers

Fortunately I had the opportunity to attend the Agile Testing Days 2014 in Potsdam, Germany. It has been a great conference. I got so many new ideas out of it. It was very well organized and there were passionate and well chosen speakers around. I regret to only have booked the first and third day, especially for they built a car in the hotel lobby on day two. And I missed that one.

To give you a little background on myself. I'm not a tester. Never have been. Agile or not. Most of my career I've been a developer working on a wide range of topics in very small and very big teams. I have done my fair share of UI development both web and RCP. I have done server development and even system development. So I've seen many things between a slick UI and moving around single bits. What I haven't seen all the time was an agile tester. I barely ran into a tester, old school that is. Nowadays I'm involved with shipping a product of round about 2 million lines of code and coaching agile teams. So, maybe I'm a little bit biased here. But now you know. Just in case.

The whole conference has been arranged around the topic agile testers between past and future. Already the opening keynote of Lisa Crispin and Janet Gregory set the tone to what then was yet to come. To me it seems agile testers experiencing a phase of self-discovery these days. On talks gave insight to the live of a tester in a development team who out of a sudden got supported by a developer to get his testing done (Kristoffer Nordström). Another workshop tried to introduce tools that could help a tester to develop a proper testing strategy and to cover the cube of testing opportunities by identifying what he has to actually test. A major part of this workshop put emphasis on the means to obtain information about the application (Huib Schoots). The talk by Alan Parkinson explained how pull request could be used to communicate with developers and how to make sure only tested code gets into the trunk. These talks very much based in the notion that testers and developers are not really working together, that developers throw their stuff over the wall and testers have to cope with what they have implemented. At least that is how I understood what has been told.

And then there has been the third day opening keynote by Anthony Marcano with the title "Don't put me in the box". He was ventilating the idea that agile tester would be no title or job description but an activity one would do for some time. To me this would be the key idea for the future for testers, agile or not.

I've been around in this industry for over 25 years now. The longer I'm into it the more I lack the understanding what a tester would be good for. Or to put it another way around. I lack the understanding why there should be people called testers that would have the obligation to check the work others (developers) have done a little while ago. I'm very well aware of the fact that testing as an activity probably would be the most important task in software development. As an activity.

In my opinion there is no such thing like an Agile Tester when we assume this to be a person. To me agile means fast feedback and early as well as frequent adaptation to what ever needs to be adapted to. The instance of a tester that would consume whatever a developer has programmed only manifests the old fashioned way of waterfall like setups with late feedback and little adaptation. And it doesn't really matter whether the tester would be part of an agile team or not. As long as he is supposed to test after the implementation work has been done he could as well be half a world away. Well communication would be a little easier when he sits right next to the coders.

The major shift the agile tester has to go through would be the change from being a job title to becoming a role one could take on for some time up to its extreme in TDD where it would become a hat which has to switched often with other hats. In times of XP and TDD and all that stuff we would talk about agile testing instead. Agile testing would be any activity from TDD to ATDD or whatever testing happens while implementing.

And what happens to the people? The real existing agile testers? I understand that this change brings about uncertainty and there are a lot of people who ask themselves how to deal with this situation, how to move on. I've seen operations guys turning into devops applying TDD to writing chef recipes. I've seen operation guys moving from writing scripts into developing a build and test infrastructure in an agile team applying XP methodologies. They brought there special knowledge and they moved on. I'm sure same would be possible for agile testers. They do have valuable skills, unique skills that any team of developers would highly appreciate to have around. Agile testers could bring their knowledge to the developers while they bring their unique knowledge to the former testers. Anyone could draw something out of it. Testers would start to take part in the implementation and developers learn how to test early, frequently and automated. Finished user stories would be delivered in high quality in the first place. No need for subsequent QA anymore.

Okay. This might be to bold to claim. To be honest, there are testing activities that would require special knowledge not available for everyone. Think of usability testing or user acceptance testing where there is much more need to be able to deal with people to guide them through the process. These (and some others) are high value manual tests. These tests are not stupid repetition work like scripted manual testing would be. They are interesting an challenging. A work for specialists. Any other testing especially any testing that could be automated or which would be supported by a tool should, or shall I say: must, be done close to the implementation work to make sure to get early feedback. Only in such setups it would be possible to fix bugs as early and cheap as possible, to change architecture as long as one still has the knowledge about it.

To wrap it up. Although there are some testing tasks that would require specialists to get completed most of the testing would be work a developer could learn and has to learn when he is a member of an agile team with the assumption that agile teams incorporate some or many techniques of XP. Agile testing would be the testing work that would be done while implementing a user story. Agile testers would merge into these agile teams. Their job title would become a role or even better an activity. They themselves would move on to become one more guy to get user stories done. They would bring their unique strengths and capabilities to the teams. They would teach former developers how to test efficiently and they would learn how to code.

Once I've been a member of a real agile team. Anyone in this team would code and do TDD as well as any other testing that was required. Anyone would touch the build scripts or would build stuff to get automation done. Anyone was sharing the same idea: We as a team make something happen. We take on any activity needed to get our work done. That has been a great time. Probably the best, the most inspiring I've ever had. If I would imagine the future of our industry this would it be.




Read also:

On Coaching Agile
Part 1: Killer Questions - What do you want to test?
Part 2: Techniques - Make a Bold Statement

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

On Coaching Agile: Techniques - Make a Bold Statement

In coaching agile methodology or coaching in general making people to leave their comfort zone or pushing them out of it would be key to get your message across, to bring about a change. You want to make them think, open up their minds to the new concepts and ideas. There are several techniques to achieve this. Last time I was talking about the power of a simple question: What do you want to test?

This time I want to focus on another technique I use a lot: Make a bold and simplified statement.

What do I have in mind by that? Well a bold statement would be something that stands middle of the road, massive, hard to get around, a bit provocative too. You have to deal with it when it happens on your way. There is no way of slipping through unnoticed.

When coaching you want to make the people that attend the training or that you coach on the job understand what you have to say. I guess it is a common observation that people that attend a training at some point or the other tend to drop out of it for different reasons. So every now and then it is required to retain their attention, distracting them from whatever they would do at this very point in time. Making a bold statement or repeating it could do the job very well. Here are some examples of what I have in mind: J.B. Rainsberger claims Integration Tests are Scam and David Heinemeier Hansson declared TDD is dead. Long live testing.

One I would use very often would be: "There have to be unit tests". Well, sure it is not as bold as the two prominent examples above. It was meant for a different audience. But what they share, all of them are catchy and promise a message that would make people listen. Either for they think they would agree or not. But whether or not a statement will be received as bold or even disruptive or not very much depends on the context. In a context where there are only a small number of unit tests around but a huge number of long running UI click tests a simple statement like "There have to be unit tests" could be a really bold statement that would get a lot of attention, positive and negative of course. Tell you what, it really did.

If the bold statement you are about to use works or not depends on how well it fits the context you would like to use it in. It is supposed to put emphasis on a hot topic or a pain point the trainees experience in their daily work so they are able to relate themselves to it. It should be simple and rather short to be remembered. It should wrap up a message you want the trainees to understand. And of course there has to be a message in the first place and you should be able to explain it at every length at every time. As coaches we are not supposed to resemble the yellow press - just punchlines and nothing else. A good punchline helps but there has to be more to it.

The statement "There has to be unit tests" clearly is short and one could remember it quite well. But it clearly has an offensive touch to it when presented to a team that works on a code base that is so highly interwoven that it is not at all unit testable just like that. In order to bring in unit tests there have to be major refactorings prior to it. It simplifies the message, which would be to incorporate a test portfolio that resembles the testing pyramid we all know. So, what it definitely does not say and not even mention is that integration tests would be forbidden for all times. But it has been received like that and there were huge discussions following along. To be honest that has been the best that could possibly happen. Out of a sudden I've got all peoples attention. Most of them were disagreeing. But finally I was able to make my point.

Suppose you do have a message, sure you have. You wouldn't be coaching if you wouldn't have any, would you? So, you've got your message. And now? When coming up with your punchline be cautious to make it bold enough otherwise it would go unnoticed. You would not get the attention you want to get. You would not get people out of their comfort zone at all. If it is too bold, on the other hand, it either will be noticed but it does not grab hold in peoples minds for they do not relate to it. Or they will feel really offended which would spoil you as a coach.

There is one side effect I should mention here. A bold statement tends to become a synonym for you. It is like there is a sticky note on your forehead with this statement written to it. You will be cited and you will be misunderstood. People take the statement and try to turn it against you. You have to stand it. You have to preach over and over again. Repeating the message all over again.


A checklist to summarize:

Pro:

• distracts people
• makes you get peoples attention
• wraps up your message in a memorable little piece
• annoys and breaks indifference

Con:

• needs the right tuning for the audience you would like to address
• easy to be misunderstood
• could be offensive and repulsive

Read also:

On Coaching Agile
Part 1: Killer Questions - What do you want to test?

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

To Quarantine or Not To Quarantine Contagious Tests

Something is rotten in (the state of) your code base and spreads like a disease. Where there was only one soon enough there will be many. They are contagious as hell - Sporadic failing tests in already secured code. Regressions that is.

In my last post I extended the Green Mainline Policy by a human factor which might help to bring the Green Mainline Policy forward in environments that are ridden with bugs and regressions and which want to find a way out of it. With this human touch comes the freedom of choice. Instead of having an uncompromisable automatism which would just close mainline until a proven fix arrived a human would be able to weigh different aspects against each other and to decide not to close mainline even if a Sporadic popped up. I also talked about the fact that in cases like this at least an initial analysis would be requested which would be the base for the decision. What could this decision look like? What options does the mainline owner have?

Well obviously mainline could be closed until a fix arrives. This would work if the cause of the test failure could be fixed easily. Mainline wouldn't be closed for long.

The second option would be to accept the fact that a fix would take fairly long, too long to close mainline this long time thus hindering other developers or teams to bring new changes to it. Acceptance of the fact alone wouldn't be enough for the test still fails and will do so with every new build pipeline run. Every subsequent run will or may show this error dependent of it being a Sporadic or a constant regression. In either way this test failure will make it hard to assess the run quickly for in case the run fails it is not obvious whether it fails due to the "known" issue or a not yet known one. A closer analysis would be required. In this situation there needs to be a mechanism that ensures the fix to come as quickly as possible while it makes sure the "known" issue does not shadow other failures. The mechanism has to make sure that a failure in a run really is a failure that needs attention payed to it.

The mechanism is known as Quarantine. I'm not the first to talk about a quarantine. Martin Fowler did cover this briefly in [1] and there are CI servers around that support a quarantine out of the box and I'm sure there are plenty of articles on this topic around. So, what's my take on this?

My feelings towards a quarantine are ambivalent. At the one hand I'd welcome it as a means to separate intermittent tests from the ones that would provide valuable feedback. On the other hand I suspect a quarantine would easily turn into a collection of failing tests that just keeps growing without anyone taking any actions on them. Because of that I would prefer a quarantine to be guarded with strict rules in order for it to be of constant value.

The basic idea of Green Mainline Policy is the fact that code in mainline is and stays regression free for it represents the state of the code that has already been shipped to customers or will be shipped with the next release. Keeping it from showing regressions including and especially sporadic ones is the major task. Usually this would be done while mainline is closed. If this is not possible a regression could be quarantined in very special cases.

I'd see these four reasons: The regression is caused by

• a bug in a test that would need to undergo a major refactoring to fix it which takes some time
• a bug in the application code itself which would need to undergo a major refactoring to fix it which takes some time
• a bug in an external component where the fix of which takes some time to arrive if at all
• a hard to analyze sporadicly failing test which would need to undergo thorough analysis before it could be classified as one of the first three cases

In these cases quarantine would serve as a to-do list of open issues to be worked on with highest priority. Any deterministic regression should fall in one of the first three categories.  

However quarantine will always be the broken window. It holds the failing tests that are not supposed to be there at all. What once have been useful tests that guarded the code base turned into a useless mess with no other feedback than the mere fact that they have become of no value. They could not be trusted anymore. 

Once a quarantine exists there will be the question: Why not add just one more intermittent test? Just one. Really! That is the crucial question of Green Mainline Policy for once you have opened the door for one others might slip in as well. In  my opinion the strict rule set and its strict appliance to each and every regression without any exception will be the only way to keep this under control. Otherwise you would start to hide failing tests in plain sight. 

But it is not sufficient to have a rule set for getting into quarantine. You would also need a clear rule for getting out of it. The first way out of quarantine would be a fix. When the fix arrives the previously failing test will be removed from quarantine. For deterministic regressions it's a matter of priorities. If fixing them is highest priority the according tests will leave quarantine soon enough. It is getting interesting if no fix is available which could have several reasons:

• the priorities for analyzing and fixing complicated issues are not high enough 
• fixing really takes long for a major issue has been detected which requires extended efforts to get fixed
• the analysis of a sporadicly failing test takes very long or seems not feasible at all
• the external component provider does not come up with the required fixes

The first of these reasons could be mitigated by inventing a timeout or a maximum amount limit for the quarantine. If the timeout would be reached the respecting test would move out of quarantine and if still failing mainline will be closed. The second one appears to be a candidate for the maximum amount limit. If the maximum amount limit would be reached the next test to be moved into quarantine would close mainline for at least one other test has to move out of quarantine which by definition only would succeed if a fix would be available. These two mechanisms would work as a priority manager. They would raise the priority of fixes automatically.

And what about the other two reasons? The Sporadic and the issue with the external component? It is not predictable when or if a fix will be available in due time. How long would you wait for a fix? It does not make any sense to wait forever. Release date will come. Would you ship a software with known regressions or issues with an external component? Probably not. So, they have to be dealt with. If quarantine is not empty at release date then the product would have to be shipped with limitations. Timeout or maximum amount do not help with that. At some point in time late enough to remain optimistic and early enough to mitigate the bugs that will not get fixed you need to decide whether or not to wait for the fixes any longer. Then time has come to apply other strategies. In the end you need to decide between:

• finding a workaround to avoid the bug (if possible)
• disable the feature if no workaround exists - one couldn't use it anyway (deliver with limitation, might be solved by later patch)
• go back to earlier version of 3rd party component (if possible)

It might prove difficult to hit the best point in time for this decision. And here is the bad news about a quarantine: It moves the need to handle regressions into the future. It might be a not-so-far-away future but it still has the consequence that regressions not get handled when they occur. When you're striving for a continuous delivery then this would break your neck for you would have to ship a product with known regressions or you wouldn't ship for a long time if you are not prepared to ship with known regressions. Both of which does not comply to the continuous delivery idea at all.

So, what's my conclusion then? I'm convinced that a quarantine could be a useful tool. But it depends. The tool itself will not fix you any bug. We've talked about this earlier. If there is a culture where regressions and Sporadics are considered evil and highest priority then this tool could help cleaning them up by making them visible. If there is a culture where these things are always overruled by something seemingly more important then a quarantine will just be another issue tracker with loads of issues never to be handled or like the to-do lists on dozens of desks which never will be worked on to the very end. It's just another container of things one should try to do when there is some time left. It's the people again and the culture they live up to that makes the difference.

Me personally I'd rather not have a quarantine. It's too many rules to know, to follow, and to think about. But I understand that sometimes you would need it to reach the level of maturity not to need it anymore.

Read also:

Part 1: Sporadics don't matter

Part 2: Let the CI server take care of these Sporadics

Part 3: Where do these Sporadics come from

Part 4: Tune up the speakers - Let Sporadics scream at you

Part 5: Process vs. People - Or: Processes won't fix you bugs

The opinions expressed in this blog are my own views and not those of SAP

[1] http://martinfowler.com/articles/nonDeterminism.html#Quarantine

"Something is rotten in the state of Denmark."

Hamlet (1.4), Marcellus to Horatio