There Are No Agile Testers - Balancing the power

This is a follow-up on my posts regarding developer testing versus tester testing which came to the conclusion that rather than having a QA department or a micro QA embedded in an agile team in form of a number of testers I'd prefer to have developers taking on a QA and QC perspective in their day-to-day work. All of the posts negated the need for a QA department.

I intend to change my mind. A bit sort of. at least.

Feature vs. Quality

The situation that induced this partial change of mind I could observe over quite some time now. In an ideal world development teams would take on their responsibility for testing and a reasonable QA. There are places where this actually happens. And there are many more places where this turns out to be a challenge for the teams even if they are willing to do so.
In many shops development teams are faced with a high feature pressure imposed to them by product owners and/or management. Ever more features are requested and sold before they are built. Development teams get forced to deliver even if the quality of the features are not at a level that would suite their own standard. Nor does it fulfill the standard the customer expects.
The quality often is so poor that merely the happy path works. Any step aside would lead to trouble, meaning bugs being reported, adding to the ever increasing demands the development team has to keep up with. In the end the same people that force the team to deliver buggy software are the same people that complain about poor quality. Often I have been witness to conversations like:


PO: I want that feature end of week.
Dev: We are not able to deliver that soon. We're not done. There are some issues to tackle yet.
PO: Doesn't matter. I take the risk.


To be sure. The PO would say anything to get the pressure of his chest by delivery anything he could lay his hands on. He knows very well that he is not taking the risk at all. The development team will have to handle the outcome of the unduly risk taking.


The White Knight: QA

Here is where a QA department or QA stuff comes into play. Given the QA lead is equipped with the same prestige and rank as the management or product owners that force the development teams into bad behavior, the QA could serve as a support for development teams in their effort to ship quality software instead of premature features. The QA is there to balance the power sort of. Putting weight on the side of quality.
It is not about huge organizations of testers swarming out to test all the bugs out of the features. It is about a department that is able to define minimum quality standards and KPIs that have to be met before anything leaves development and that is able to enforce them.
Sure. This is adding pressure on the development teams from one side but reliefs them on the other side. QA will demand the minimum standards are met by development. And QA will prevent stuff from being shipped if the minimum standard is not fulfilled. Thus achieving what development teams are not able to achieve for lack of power.


Balance of Power
To sum up all of my posts: development teams have to do their testing themselves. It makes no sense to build micro silos within teams by adding testers that do testing and nothing else. There should be a convergence of tester and developer to better be able to load-balance within the teams according to the needs of the features to be built. However there are testers that turn into test enablers by providing test automation tools to the development teams or by educating developers with respect to testing. There may even be some testers that do some very specialized testing like user acceptance testing or usability testing. And finally some of the testing stuff will turn into QA - or remains there - to serve as a counterweight for over-ambitious product owners and managers.


So, finally I admit: There is a need for a QA team. A team to set standards and to enforce them. The obligation to fulfill these standards lies with the development teams. The tools and knowledge they need will be provided by former testers.

Read also:

On Coaching Agile - What I've learned from being an agile coach

On Sporadics - How to deal with intermittent tests in continuous delivery

On Agile Teams

Part 1: Agile Testing Days 2014: There are no Agile Testers

Part 2: Agile Testing Days 2014: There are no agile testers - and no agile developers either

Part 3: Agile Testing Days 2015: There Are No Agile Testers - There Are Testing Facilitators

The opinions expressed in this blog are my own views and not those of SAP

Approaches to Distributed Development of a Software Product

Note: This article will be published as a series of installments. See the installment history at the end of the article to track changes.

Introduction

Just recently I came to think about a proper setup for a product developed by distributed teams. As it happens the use of git was a prerequisite. When working with Distributed Version Control Systems (DVCS) like git teams see themselves faced with the task to figure out what would be their way of organizing source code they contribute to a larger system. As git is a powerful tool with loads of features and means to do things one way or the other it offers both simple and rather complex solutions. In this post I want to explore some major approaches and compare them with each other. I admit that I am biased by concepts like Continuous Integration (CI) and Continuous Delivery (CD) which may influence my conclusions.

I consider this an experiment and will define an example product to set some constraints for the exploration. Some findings may be restricted to this setup others may be more commonly valid. However none of the conclusions claim to be universal. The approaches investigated are taken from daily life. Every single of them crossed my way and I consider all of them worth looking at. Any idea as far fetched or remote as possible would be worth to at least provide reasons why it wouldn't be a good idea.

Sample Product

Let's assume a product of considerable size, say 1M LoC. Let's further assume the product consists of a number of large components, say 5-10, which themselves may be made of smaller components. Team setup follows the top level component structure by and large, although an individual may see the need for changes in several components. The development team consists of 50-500 people actually touching code. Finally the product ships as one. There are no independent releases nor patches of parts of it.

None of the components is intended to be reused by other products. Components are a reflection of the current architecture of the product.

The current dependency structure of our product looks like:

Components A, B and E are top level components forming a collection of services the product consists of. Component F forms a UI framework Components A and B plug in to when present. F does not care for any components plugging into it. Components BA and BB are backends to A and B respectively. Component E is an extension to B.

Components are not necessarily identical with libraries, archives or any other sort of distributable artifact. Basically they are a logical structuring of the source code.

The product has a maintenance obligation with respect to already released versions.

What approaches could be applied to organize the source code of the given product? These are the ones that pop up on my mind:

• Component Repositories
• Topic Branch
• Feature Branch
• Trunk Based Development

Approaches

When exploring the different approaches I will try to shed some light on a bunch of questions that are far too often not getting considered. These questions touch several aspects of a software development life cycle. Think of questions like:

• How do we get access to the component we depend on to make use of it in our component?
• How do we make sure we get information about public API changes soon enough for us to incorporate them?
• Should there be orchestrated schedules for component releases?
• How do we handle splits/merges of components?
• How do new top level components come to life?
• How do top level components cease to exist?
• How often do components ship new versions?
• How do we make sure there will only be one version of each component used inside the product? Or how do we make sure components A and B are developed against the same version of component F?
• When will integration testing be done?
• How will it be done?
• How does a component test itself in the context of the product? Or in the subcontext of its dependency tree?
• How is the product being assembled?
• Which component versions should be used?
• How are component versions get managed?

The list may not be complete. It already holds some tough issues, though. What would be the answers when we work with separate Component Repositories?

Component Repositories

Our development team values highly decoupled components which interact via public APIs only. Any use of non-public APIs is prohibited. The developers understand the temptation of using non-public APIs for the sake of re-use and want to avoid that by obscuring the sources of their components towards other components as much as possible.

Our development team learned that a git repository for a large product worked on by many developers tends to become large which would increase time to clone and fetch. I've seen such repositories exceeding 4GB.

Idea: A small repository only containing one top level component worked on by 5-10 people seems to be a fair trade. The team could work in isolation. Their repository would not be littered with source code they do not own or are likely never touch. Things are simple when it comes to developing their component.

Development

For a developer working on component F with no dependencies or only dependencies to 3rd party life would be rather easy in this world. There are only things present one has to deal with directly. One could build the whole component including tests quite focused. As developers are free to add or remove sub components of their component rather freely they would not feel much of a downside.

Not all of the teams are happy with that setup, though. While the team providing the top level component F is quite happy with this approach, the teams depending on them are not. Why is that?

In order to plug into component F component A and B need to know about and have access to the current valid public API to at least be able to mock the dependency away in their tests and to use the right calls in their production code. However this may be done in the particular language the product is being built with, there has to be some sort of communication. Either interfaces have to be provided as files or as API documentation. Depending on the language these files have to be present during build. Latest when running integration tests of component A or B with component F a real component F would be required.

This would add an obligation to any top level component development teams responsibilities: There has to be some sort of releasing the component in a way other teams could rely on for their development. They have to maintain a release schedule and they have to actually release the component and make it available for the other teams to consume. Usually there will be some stable and some development version available. These versions could be used by components A and B for their development.

Integration

As long as component F publishes new versions on a regular base there will be some sort of "continuous" integration be available. Component A and B could make use of the latest component F version and report bugs if found in component F or fix their components usage of F accordingly. Depending on the release cycle of F the feedback loop length would stretch from rather short to pretty long. During development phase this may not be a problem but when release date closes in it rather certainly will turn into an issue.

A real continuous integration would be hard to achieve. Even if component F publishes release candidates with every pipeline run they would have to be verified by the depending on components before they could turn into released versions. Thus component F depends on the pipelines of each component up the dependency tree to verify successful usage of F and any component that uses F and so forth. The verification pipeline for F will become pretty long and in case of bugs found it would have to start all over again.

Whats more, if component A uses the development version of F to stay close to the newest features of F it relies on these development versions actually being released before release date of the product as no development versions of F will be shipped with the released version of the product.

Another complication would be the possible divergence of F's version used by components A and B. Just to make sure they are actually using the very same version of F there needs to be some governance enforcing this constraint.

Integration Testing

When teams are focused on developing their components they tend to consider any usage of their component the business of someone else. The integration of component A or B with component F will probably tested but the product as a whole will not. Who would be responsible for performing this assembly task with all its required testing?

The product in question is an assembly of the just right versions of all its components. Thus the product would be represented by a bill of material (BOM) only. The product assembly would drag all the named versions of the components and would perform the required packaging. What about the testing then? There would have to be a team that would take care of this assembly and the integration testing to make sure the BOM holds a valid and working combination of component versions. The assembly pipeline would have to run the integration tests of all components and eventually would have to provide for additional integration tests on product level. This team would not at all develop anything in terms of production code which bears the risk of them not knowing about features implemented. A dedicated communication would be required to make sure the assembly (or testing) team knows what to test for.

Another risk would be product level test breaking due to changes in components. As the product level tests run in the product assembly pipeline no component pipeline will run them and thus will not get feedback from them. It is the same as component A running integration tests with component F which could find bugs in F outside the pipeline of F. At any of these points there will be feedback someone would have to communicate to the depending on component. This feedback would have to dribble down the dependency tree with all the communication that comes along with that.

It would be best if the component could test itself in the context of the product within its own pipeline. To do that it would have to get access to the current BOM describing the product and to the product level tests. In order to run the product based tests it would have to build the product based on this BOM and replace itself with the component version under test. Component A's build and test process suddenly needs to know about the product and its assembly thus duplicating knowledge.

Another way would be to trigger the product assembly pipeline by replacing the version of component A in the BOM with the latest release candidate of A. If the product assembly pipeline succeeds the release candidate could be considered verified, it could be released and the BOM of the product could be changed accordingly. In this case the knowledge would not be duplicated but we would need a feedback loop from the product assembly pipeline back to the pipeline of component A. In order to get close to continuous integration any pipeline run of component A would include and wait for a pipeline run of the product assembly.

Release

As said before the product is being represented as a bill of material (BOM) containing the proper components and their versions. At this point uniqueness of components could be enforced, i.e. the version of component F to be used. Following the one repository per top level component the product as the topmost component will reside in its own repository along with the product level tests.

Releasing would include collecting all released versions of components making up the product and to run the product level tests as there would be no other tests available. If the versions of lets say component B and component F do not fit together for component B was using a different version of F in their integration testing there would be the risk that product level test would not discover this mismatch as they will not redo the level of testing done at component B's level. To avoid this means would have to be provided that would mitigate this like: components integration tests will be made available to the using components as well, a component will get hold on the BOM of the product to make sure they will use the proper versions of all components they depend on and so on.

This would introduce yet another set of communications required to mitigate issues induced by the general approach of Component Repositories.

Refactoring

As long as refactoring takes place within the boundaries of a top-level component things are fine. When it comes to structural refactorings of the product, i.e. introduction of new top-level components, removal of top-level components, it becomes cumbersome.

If there will be a new service C we could just create a new component repository and start working on C, adding it to the integration like the other components.

If an existing component ceases to exist in a newer product version we just could not get rid of it as long as the maintenance obligation exists. There will be a legacy component around in a repository no one will work full-time any more. This usually will cause the component to rot for no one will take on responsibility for that one. It is just to far out of sight.

Component Repositories make it hard to factor out new components. Consider a part of component A that would be useful for component B as well. How would B get hold of it? The cost of introducing a new component repository for the new reusable component would be quite high. So, copying the code and adding it to component B's repository seems reasonable especially as long as one would be wondering whether this part really is reusable by B. If it would be reusable and if someone really would avoid the code duplication and open up a new component repository who would be responsible for that? The new found component would not be a top-level component, so there would be no dedicated team despite the one for component A. Would this team be responsible for two repositories now?

Summary

The Component Repositories approach has got its advantages when it comes to the development of a leaf component. As soon as interaction with other components due to dependencies is involved things get messy. Components suddenly need a release management and version governance to make sure every component is on the same page. Especially the product assembly part will become a matter of discussions for no component development team will take responsibility for this integration level. A product assembly team would have to deal with that and would have to take care for product level testing itself.

Communication would be key in this approach. Whether it is done by introduction of additional automatisms to connect component repository pipelines with each other or by human interaction it adds complexity and the "one has to think of it" sort of things which tend to not been thought of.

As long as the component is not a real deliverable in its own right, i.e. will be used outside of the product, will be patched individually, I would consider this approach as not practicable.

Approach	Component Repositories
Development (leaf component)
Development (non-leaf component)
Integration
Release
Refactoring
Organizational Complexity

Conclusion

As I've only considered one approach yet the only conclusion I could offer now is that I would not like to go for component repositories not knowing a better alternative for now.

Change History

This is the first installment.

Agile Testing Days 2015: There Are No Agile Testers - There Are Testing Facilitators

I had the opportunity to attend Agile Testing Days 2015 in Potsdam. It's been the second time. And again, there haven been great sessions, inspiring talks, eye-opening chats. But still there is something that bothers me. If you following my blog post you might have noticed the "There Are no Agile Testers" blog posts:

Agile Testing Days 2014: There are no Agile Testers and 

Agile Testing Days 2014: There are no agile testers - and no agile developers either

To say the least they have been received a bit controversially. Many testers felt personally offended. I could understand this to some extend. A year has passed since then and the idea circulated in my thoughts. I was trying to understand what really bothers me about the agile tester thing. I'm not sure I'm done with it yet. But new aspects revealed themselves. 

The thing that bothers me the most is the fact that by just bringing testers to agile teams does not solve the issues we've had with the development and QA departments. Testing still comes last in the development cycles and tends to be skipped or blamed for being a bottleneck. This is nothing I made up, but something that has been said by testers at the conference. In a way the testers in agile teams establish a silo just as the developers do. Developers rely on testers to get quality into the product and complain if testers do not manage to handle the amount of user stories done for coding stories tends to be faster then thoroughly testing them.

Over the years I became more and more opposed to silo thinking in teams. This discomfort still grows. I try to find ways that could help to overcome this dangerous tendency. My experience from many years shows that whenever a teams starts to separate into silos team performance, quality, and outcome drop dramatically. The teams start to dissolve and I've even seen teams to dissect.

A second aspect that I grow ever more uncomfortable with is the fact how developers are pictured as guys not willing to test, not willing to care for customer needs, not willing to care for quality. A great many developers may fit this description. But another great many developers care for concepts like Continuous Delivery, Lean Startup and DevOps. Both of them are heavily relying on being responsible and accountable for ones quality. Developers show that they are willing to produce stable, high quality code that covers actual customer needs. That they are willing to measure customer acceptance and to act accordingly. That they are willing to ship to production as often as possible. I reckon (a new generation of) developers understand(s) pretty well that they are no longer sitting in the basement coding all day long without ever bothering themselves with any consequences their work might have for the world around them.

For quite some time now developers proved to be no good at testing. Whether they are just testing agnostics, arrogant my-code-will-not-break guys or anything you might think they are doesn't count. Testing did not take place in a way and amount that would have been desirable. Because of that QA people had to be hired to clean up the mess they left behind. But no one bothered to tackle the root cause: Improving quality of the code from line one. This would have meant one has to deal wit these strange guys in the basement. So, an opportunity has been missed. The mere existence of a QA department that made up for the mistakes developers made encouraged them to code even more carelessly. There simply has been no need to do otherwise.

It is time to reverse this development. Now, as developers develop a sense of responsibility testers are urgently needed to share their knowledge and experience gained over so many years of testing. This knowledge has to be shared with developers. Testers are urgently needed to challenge developers to take testing beyond unit testing seriously. There is far more to testing than that as one could learn form "Agile Testing" by Janet Gregory and Lisa Crispin. 

Me, being a developer for most of my professional life, I would wish if not expect from testers that they make testing an integral part of a developers daily business. Testers in agile teams have to become test facilitators. There is no way around that. If an agile team would be staffed with as many testers as you would need to make sure all user stories are covered with acceptance tests, all new code is covered by unit and component tests, all security, usability, performance and you name it what test are required up to thorough exploratory testing one could easily end up with maybe 3 (or more?) testers per developer. Would this be the way you would like to go?

In my humble and honest opinion we would need to tackle the problem from two sides.

1. Testers Coach Developers

Testers gained insights on lots of different aspects of testing including experience of typical hot spots especially when it comes to integration testing. Testers would need to pair with developers to support them when writing these kinds of tests figuring out what test cases are needed and how to best test them. It may be that while doing so testers become familiar with development itself and eventually cease calling themselves a tester. I would consider this a great achievement for our industry when testers did their coaching job that well that developers are able to do the necessary testing and former testers turn to writing code themselves. In a way we would have to call them all Agile Engineers. Then Continuous Delivery and DevOps would fully unfold their potential.

2. Testers Provide Automation Frameworks

Not all testers would like to turn to bare development. I would propose another direction for them. Many developers do not care for security or performance testing because there are no frameworks and no infrastructure available that would perform these tests reliably and easy to write and evaluate. Any of these frameworks needs to be made available in build and test pipelines. If they are not available that way they will hardly be done. Developers need to be urged into writing  these kinds of tests by making this unavoidably easy. Whether or not these frameworks have to implemented from scratch or could be bought. Someone has to set them up for use in pipelines. Someone has to educate developers of how to utilize them.

3. Testers as Integrators

Developers tend to be off by one, so am I. There is a third category of testers. The ones that never ever wrote a line of code and are not fond of doing so at all. There are businesses that have to fulfill legal requirements with respect to quality assurance. There businesses that built huge products with millions lines of code contributed by teams not at all co-located to each other and often enough not well connected. Products like that tend to have integration issues and no one feeling responsible for them. These are areas testers in a more classical sense still would be needed without the urge to turn into developers.

Conclusion

Testers in agile teams should try to see themselves as coaches and facilitators to spread the art of testing. Developers need to be educated and enabled to do lots of testing on their own while and before writing any code. Developers need to learn to look at what they do from a users side in order to enable them to decide in favor of a users needs.

Testers could provide frameworks for automated security, performance, product life-cycle testing and alike. These frameworks have to be made available to developers in their daily work to make these kinds of testing an integral part of coding well before a user story is labeled DONE.

The tasks testers will face in the future might change. For some this change may even be dramatic. But I think we could not afford moving on as we did before. It is time for Developers 2.0

Read also:

On Coaching Agile - What I've learned from being an agile coach

On Sporadics - How to deal with intermittent tests in continuous delivery

On Agile Teams

Part 1: Agile Testing Days 2014: There are no Agile Testers

Part 2: Agile Testing Days 2014: There are no agile testers - and no agile developers either

The opinions expressed in this blog are my own views and not those of SAP

Agile Methodology Breast Feeding

How come that many of us developers are struggling with agile methodologies like test-driven development or componentization/modularization of a software?

I was asking myself this for a long time now. Being an agile coach I frequently see developers fail applying these techniques in their daily work. Intelligent people that are able to write code to take care of the most complex problems around. And yet an apparently simple technique like write-your-test-first doesn't get a grip with them. Why? 

In the past I tended to blame the workload and pressure of management and the urgent need to put out the next feature and QA it afterwards that somehow caves in the people into this treadmill of assembly-line work like feature production. In this treadmill they are not able to get the time or freedom or even will to learn something new, to question what they are doing, or how they could do better.

It's easier to just do what one has always been doing for this they would know, this would require less efforts and less energy. Change on the other hand comes with inconvenience, with insecurity, with the need to work more consciously which slows down compared to working with a routine.

So, changing the approach developers use to get their things done becomes a long hard process. Sometimes it feels like a behavior therapy with me being the therapist. 

But I get carried away - a bit.

Just recently I came up with another source of evil, though: Textbooks!

I was curious about JavaScript and node.js recently. What would you do to get a concise guidance on any new language? Well, me I would buy a good book and work through it - or google a good tutorial on the web or something.

This time though I wanted something else. For me being an agile coach I wanted to do what I am preaching. Do TDD right from the start. So I skimmed through the pages and in the index I found "Testing" at page 10-something. Wow! I thought. There is something about testing and it's quite early in the book.Promising! 

Well. what a disappointment. It simply suggested to create an index.html file where the JavaScript would be written to or linked and to open this in a browser to witness the effects of my changes.

That's all to it.

After this the whole thing was like every time: Gradually introduce the language and write some code to learn it.

That way I learned some other languages as well. It never ever crossed my mind that I should write some tests first. Instead I wrote a few tests afterwards. Just like the others did. Fine!

No!

Thousands of young people learned their first programming language that way. They did not learn about testing. They got the notion that programming meant just writing code that accomplished the task at hand. Then they would discover that there are bugs in what they wrote and they learned how to debug even nasty code. Some of them are rather skillful debuggers and learn about seldom used features of their tools. They feel like the best programmers around for they would nail down even the strangest bugs. Bug fixed. Fine. No test needed. The bug has gone, hasn't it?

I've seen many of these guys entering corporate software development where they would move along their way and would just hack on a bigger scale. 

Try to tell these people that there are other ways of doing it. They imbibed programming with no testing from their infancy. It's hard to change something about it. They would lack the awareness of potential problems with that approach.

I'd plead for a new style of textbooks. 

It came to my mind that the textbook I would like to read (or even write for that matter) would be different. I would like to introduce a language TDD style. Explaining testing first. Why not writing a HelloWorldTest first? It's a program as well, isn't it? It's a minor change of perspective but a huge change in mind. 

Tell you what, while thinking about this idea I stumbled across this book on node.js:

http://leanpub.com/nodecraftsman by Manuel Kiessling.

To my big surprise it did exactly what I would like to do: it started introducing node.js with TDD. Thanks for providing this enlightened little book!

Read also:

On Agile Teams
Part 1: Agile Testing Days 2014: There are no Agile Testers
Part 2: Agile Testing Days 2014: There are no agile testers - and no agile developers either

On Coaching Agile
Part 1: Killer Questions - What do you want to test?
Part 2: Techniques - Make a Bold Statement
Part 3: Techniques - Show and Tell

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

Agile Testing Days 2014: There are no agile testers - and no agile developers either

Just recently I wrote a blog post in response to the Agile Testing Days 2014 I was glad to have attended. This post was received controversially, to say the least. There have been many in favor for what I was saying, and there were many that didn't like it at all. For several reasons. I've got involved with interesting discussions with many of my readers., most of which led to the conclusion that there has been some misconception due to the fact that some of my arguments presented in the conversations were not mentioned in the post. I promised to write a follow-up to get to the point I was trying to make with the first post but didn't manage.

Here it is.

My Background

For there have been some not so polite responses questioning my right to express my thoughts regarding testers and testing I will explain where I come from and what my premises are. Currently I'm in QA for a 2M lines of code rich client application. As QA we define the processes for the development teams to follow to ensure a high quality of the product including everything from continuous integration, automated testing up to E2E and exploratory testing as well as static code checks and security scans. We follow the goal of establishing a continuous delivery process for this application.

My background over the years, however, was in development. As a developer I've come a long road from being a typical developer (from a testers perspective) to becoming an coach for agile methodologies (including agile testing). At certain times I did user acceptance testing and scripted testing. So, I've done my fair share on almost everything a tester would usually do. But my upbringing as a developer certainly makes sure I'm not completely familiar with the terminology a tester would use. Please forgive me this lack of knowledge but rest assured it is not because of ignorance.

Hypothesis

The existence of agile testers (aka testers as members of agile teams) prevents the urgently needed mind shift development has to undergo and the existence of agile testers does prevent agile teams to become what they are supposed to be: agile small units that could build and ship the highest rated new features with nearly constant velocity. 

Waterfall

Development departments have a long record of software products not shipped in time and not having the expected quality. There have been many ideas around how this could be improved. The most successful of these has been the introduction of a QA department. Just like in any other industrial production QA would check the product after it has been assembled, and before it would be shipped to customers. For cars or machines this worked out pretty well. So, why not adopting this approach?

Development was known to deliver bad quality. QA would establish a Q-gate for the product to pass. Testers would test the product thoroughly before they would turn the thumb up. Fixes for issues would be requested and delivered by development. Sounds great. But virtually never worked out. Why? There always is the temptation to squeeze yet another feature in and yet one more, thus stretching the time for development beyond any deadline. QA would suffer for they are required to accomplish their tasks in a much shorter time frame then originally planned. Issues they found did not get fixed anymore for release date approaches to fast.

In such environments the guild of testers had to develop methods, techniques, and tools to somehow manage the increasing workload in a shortening time frame to work in. They had to mitigate the lack of information, the lack of involvement, and the fact that they were forced to test at the wrong end of the cycle. They sure would develop a pride in their work and their abilities. For they would act closer to the customers and would receive complaints and issue reports their understanding of customers needs would be better by far compared to the understanding a typical developer would have. They sure took pride in that as well. And rightly so.

Turning an eye on the developers. Before QA has been around they at least had to do some testing. Now there would be QA testing just enough quality in. Fine! More time to develop the features development was forced to squeeze in. It wouldn't come as a surprise if a developer wouldn't even felt relieved of the burden to spent valuable time on testing instead of walking further down the list of features he was supposed to finish. Once QA came into existence developers were disconnected from customers by yet another layer. On the other side of the process there were architects or system analysts that would lay out the whole product before even the first line of code has been written. They would talk to customers to collect requirements. Then they would put down all their design documents and contracts and stuff. And the developer had to walk down the list. Completely disconnected from the source of the requirements and the consumer of his work. They got cushioned in between these two layers. What would you expect them to become within this sandwich. No time to really change the architecture if they would find a flaw. No time for that. Just keep working around it. What connection to your work would you develop if you would have no means to influence major parts of it? A certain archetype would be required to survive in such environments. Nowadays we call them developers (at least when we stick to the cliché). This typos gathered in development departments. At one hand cushioned in and pampered, kept away from customers. At the other in splendid isolation just doing geeky stuff feeling proud of their ability to work around every problem in the shortest possible time.

To be sure: Not every developer fits into the cliché. Just like not every tester fits into the cliché built by many articles and books and which has been presented to me at Agile Testing Days this year or in replies to my postings. Things are not black or white, as usually.

Agile Teams

Fortunately there has been developers that did not fit into the cliché. Guys who wanted to do things differently as well as other people in this industry from other roles. And over the years ideas like XP, agile, scrum, and lean were invented. Now we are faced with agile teams. Suddenly things like backlogs, priorities, and short cycles matter. Frequent shipment of high quality product increments were expected. Customer feedback which leads to major adjustments in the product would be welcome. Imagine a developer from the safe haven of the good old days when they were supposed to just code and a whole organization built around them would take care of the rest. What would he make of this?

Imagine a tester with his safe haven QA department. Far away from developers. Now she was supposed to join teams with the developers. Two parties were put together that felt comfortable with the prejudices they had on each other. What would you expect to happen? Well, the safest thing would be to keep the respective specialization and organize the agile team like the departments they came from. The developers in the teams would, well, develop and the testers would just test. In fact the agile team would turn into a micro version of what has been the organization back then. I have seen this happen several times. And this does not seem to be special to me. Many of the sessions at Agile Testing Days shared the same experience to some extend.

Even the bible for agile testing the book by Lisa Crispin and Janet Gregory did elaborate on this. I like this book it has got a lot of knowledge and experience in it. I certainly like the insights from their experience over the years they would share in there. And it has the right title for sure: Agile Testing. I will come back to this just a little bit later.

Part I of the book elaborates on testers and what they could bring into an agile team. In what sense they are special and would yield a great benefit for the team. Yes, testers have their unique specialties. Yes, these are badly needed close to development and even before the first line of code has been written. We definitely would need more agile testing in agile teams or in software development in general. 

At some point in this part the "Whole team approach" was mentioned and welcomed. No distinction between testers and developers. Just a team working on the top priority tasks doing whatever has to be done by anyone available when a new task has to be worked on. The agile team as a bunch of generalizing specialists. I very much like this idea. In fact that would be the Holy Grail. An agile team that would be able to really work according to priorities.

Unfortunately the remainder of this part very much keeps the distinction between testers and developers and tries to explain why a tester would be required to write acceptance tests, to talk to customers to understand what he really needs and similar stuff. There is one question popping up my mind: Why would I manifest the old DEV - QA pattern in an agile team? Why would I still keep developers away from activities they would urgently need to do to improve their understanding of the product or the customer needs. If they just be kept away from these experiences, then they are just cushioned in and pampered as before and we are all no better off. 

We would still play this DEV & QA game. Testing happens after implementation. I have seen this happening: Testing gets squeezed in between end of implementation and sprint closing. Often resulting in not properly testing for lack of time. Letting bugs slip through and so on. Where would be the difference to the old days? Testing needs to be first priority. To get this done anyone in the teams needs testing skills to some extend.

If you really want developers to change, to develop new capabilities you need to let them do those things they never did before. A tester would be a great teacher to them if he shares his knowledge with them. A developer needs the feeling of pain a bad quality product gives the customer. A developer needs to write acceptance tests herself to understand what a user story requires them to produce. These first hand experiences are what would trigger insights for them. 

Are developers afraid of this? Yes. Are developers not prepared to do something else than develop? Yes, for sure. We are talking about change. Change leads to a feeling of uncertainty, leads to occurrences of not knowing what would be expected of them. Change brings about fear. Same goes for testers. Are they really required to code? Are they really required to do architecture? Bug fixing? Yes they are. In an agile team. And this brings about fear, uncertainty, feel of loss for them too.

So, what happens most of the time? Developers and testers comprising a general non-aggression pact. And each sticks with what she could do best. Keeping specialism alive. And this specialism yields problems itself. Specialists are only able to work on a limited amount of tasks. Those who they do not cover with their specialization they would not take on. Specialists tend to be bottlenecks if their special knowledge would be required by two concurring tasks one would be blocked. Specialism leads to situations where the user stories not get worked on in the order of decreasing priority but by the need for some specialist. Thus the next increment of the product will not contain what the customer needed the most but the features the team in its combination of specializations was able to get done. This would not comply to the ideas of agile anymore. A team like that wouldn't really be an agile team.

Agile Engineer

What would that be? She would be a generalizing specialist. Willing to learn new methods, techniques, new skills. A member of the team that would be able to take on almost any task belonging to the most important user story on the board. If writing some code it would be, then she will code. If it is testing, she would test. If it is something else, she would do this as well. She would not be afraid of asking when she lacks knowledge or experience. She would be willing and open to share her knowledge.

Okay, lets go with agile testers and agile developers. We need to start somewhere. But support them in becoming an agile engineer. This will be a though transition for many of them. They would need help and guidance on the way. And they need the right environment. Just because an organization decides to have agile teams and to go agile doesn't mean that there is anything else than a new label put on the stuff they have. An agile engineer would not yields the effects he could if the organization around him is not transitioning too. Fixed schedules, fixed resources, and fixed scope do not work with agile. If the organization is not going agile anything else could remain as it has been back then.

Agile engineers would build great agile teams working in great agile organizations. They would DO agile testing, would DO agile development, or agile planning. They would cease to be an agile tester, an agile developer, or an agile someone. This would be the time when there would be 

No agile tester - and no agile developers either

Read also:

On Agile Teams
Part 1: Agile Testing Days 2014: There are no Agile Testers

On Coaching Agile
Part 1: Killer Questions - What do you want to test?
Part 2: Techniques - Make a Bold Statement
Part 3: Techniques - Show and Tell

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

Agile Testing Days 2014: There are no Agile Testers

Fortunately I had the opportunity to attend the Agile Testing Days 2014 in Potsdam, Germany. It has been a great conference. I got so many new ideas out of it. It was very well organized and there were passionate and well chosen speakers around. I regret to only have booked the first and third day, especially for they built a car in the hotel lobby on day two. And I missed that one.

To give you a little background on myself. I'm not a tester. Never have been. Agile or not. Most of my career I've been a developer working on a wide range of topics in very small and very big teams. I have done my fair share of UI development both web and RCP. I have done server development and even system development. So I've seen many things between a slick UI and moving around single bits. What I haven't seen all the time was an agile tester. I barely ran into a tester, old school that is. Nowadays I'm involved with shipping a product of round about 2 million lines of code and coaching agile teams. So, maybe I'm a little bit biased here. But now you know. Just in case.

The whole conference has been arranged around the topic agile testers between past and future. Already the opening keynote of Lisa Crispin and Janet Gregory set the tone to what then was yet to come. To me it seems agile testers experiencing a phase of self-discovery these days. On talks gave insight to the live of a tester in a development team who out of a sudden got supported by a developer to get his testing done (Kristoffer Nordström). Another workshop tried to introduce tools that could help a tester to develop a proper testing strategy and to cover the cube of testing opportunities by identifying what he has to actually test. A major part of this workshop put emphasis on the means to obtain information about the application (Huib Schoots). The talk by Alan Parkinson explained how pull request could be used to communicate with developers and how to make sure only tested code gets into the trunk. These talks very much based in the notion that testers and developers are not really working together, that developers throw their stuff over the wall and testers have to cope with what they have implemented. At least that is how I understood what has been told.

And then there has been the third day opening keynote by Anthony Marcano with the title "Don't put me in the box". He was ventilating the idea that agile tester would be no title or job description but an activity one would do for some time. To me this would be the key idea for the future for testers, agile or not.

I've been around in this industry for over 25 years now. The longer I'm into it the more I lack the understanding what a tester would be good for. Or to put it another way around. I lack the understanding why there should be people called testers that would have the obligation to check the work others (developers) have done a little while ago. I'm very well aware of the fact that testing as an activity probably would be the most important task in software development. As an activity.

In my opinion there is no such thing like an Agile Tester when we assume this to be a person. To me agile means fast feedback and early as well as frequent adaptation to what ever needs to be adapted to. The instance of a tester that would consume whatever a developer has programmed only manifests the old fashioned way of waterfall like setups with late feedback and little adaptation. And it doesn't really matter whether the tester would be part of an agile team or not. As long as he is supposed to test after the implementation work has been done he could as well be half a world away. Well communication would be a little easier when he sits right next to the coders.

The major shift the agile tester has to go through would be the change from being a job title to becoming a role one could take on for some time up to its extreme in TDD where it would become a hat which has to switched often with other hats. In times of XP and TDD and all that stuff we would talk about agile testing instead. Agile testing would be any activity from TDD to ATDD or whatever testing happens while implementing.

And what happens to the people? The real existing agile testers? I understand that this change brings about uncertainty and there are a lot of people who ask themselves how to deal with this situation, how to move on. I've seen operations guys turning into devops applying TDD to writing chef recipes. I've seen operation guys moving from writing scripts into developing a build and test infrastructure in an agile team applying XP methodologies. They brought there special knowledge and they moved on. I'm sure same would be possible for agile testers. They do have valuable skills, unique skills that any team of developers would highly appreciate to have around. Agile testers could bring their knowledge to the developers while they bring their unique knowledge to the former testers. Anyone could draw something out of it. Testers would start to take part in the implementation and developers learn how to test early, frequently and automated. Finished user stories would be delivered in high quality in the first place. No need for subsequent QA anymore.

Okay. This might be to bold to claim. To be honest, there are testing activities that would require special knowledge not available for everyone. Think of usability testing or user acceptance testing where there is much more need to be able to deal with people to guide them through the process. These (and some others) are high value manual tests. These tests are not stupid repetition work like scripted manual testing would be. They are interesting an challenging. A work for specialists. Any other testing especially any testing that could be automated or which would be supported by a tool should, or shall I say: must, be done close to the implementation work to make sure to get early feedback. Only in such setups it would be possible to fix bugs as early and cheap as possible, to change architecture as long as one still has the knowledge about it.

To wrap it up. Although there are some testing tasks that would require specialists to get completed most of the testing would be work a developer could learn and has to learn when he is a member of an agile team with the assumption that agile teams incorporate some or many techniques of XP. Agile testing would be the testing work that would be done while implementing a user story. Agile testers would merge into these agile teams. Their job title would become a role or even better an activity. They themselves would move on to become one more guy to get user stories done. They would bring their unique strengths and capabilities to the teams. They would teach former developers how to test efficiently and they would learn how to code.

Once I've been a member of a real agile team. Anyone in this team would code and do TDD as well as any other testing that was required. Anyone would touch the build scripts or would build stuff to get automation done. Anyone was sharing the same idea: We as a team make something happen. We take on any activity needed to get our work done. That has been a great time. Probably the best, the most inspiring I've ever had. If I would imagine the future of our industry this would it be.




Read also:

On Coaching Agile
Part 1: Killer Questions - What do you want to test?
Part 2: Techniques - Make a Bold Statement

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

On Coaching Agile: Killer Questions - What do you want to test?

When coaching teams or individuals in agile techniques like TDD, Refactoring, Clean Code, CI, you name it then I've found out that certain questions yield interesting effects. My absolute favorite would be this one here:

What do you want to test?

In the meaning of: What feature or behavior do you want to test?

One of the key take aways in trainings on agile techniques would be the knowledge about and the appliance of the testing pyramid. Many teams I trained would have a test structure that imitates an upside down pyramid or a trapeze standing on the short edge or even a big letter T. When they attend a training they would learn about TDD and unit tests and are eager to write them for their code too. And what seems to be rather easy in the training with a well designed and small sample application tends to become overwhelmingly difficult when they get back to their own code base which usually would be more complex and more ridden with legacy code issues.

Nevertheless they start out and want to write the first unit test themselves. And here is where the trouble starts. How would you do that? Most of them lack experience of what a good unit test would look like. What unit should you begin with? And which tests would make sense? If you never ever tried to write unit tests for your production code these questions are tough. Everything is so much more complicated compared to end-to-end testing where there is no need to understand the inner structure of your code and where design issues are not preventing you from testing.

The following is condensed from several sessions with developers trying to put a piece of their code under test. There is no single developer who behaved exactly this way. I took the freedom to exaggerate the situation for the educational purpose. But please be aware that any part of this is taken from real situations: 

So, when I pair (actually: triple) with my developers before writing one single line of code I ask them: 

What do you want to test?

Most of the developers really have no good answer to this question. They are able to explain the implementation to its tiniest detail but they fail answering this rather simple question. That is why asking it will usually be the begin of a long discussion where they tell me what actually happens in their code which things get to be done and how. This is when I would guide the discussion to the structure of the code and the building blocks it consists of. We try to identify the parts that would not heavily make use of components which are not under control of the team. To begin with we would focus on the components that depend on components under control by the team to make things a little bit easier. Most of the time this initial discussion already brings about a better understanding of their own code and architecture which will help a lot in the upcoming part of the session. Besides, me as a coach I'm able to understand the coding on a relaively high level of abstraction which helps me to not get lost in too much details. I would gain enough knowledge to understand the features and what tests there should be. With that I'm able to guide them further down the road.

Once the component to put under test has been identified I would ask the question again: 

What do you want to test?

Many times this is when I get my tour through the component and what it would be doing one more time. This is when I try to find a nice unit to start testing with. Sometimes they would pick a unit to test themselves and sometimes not. Whatever they would pick I'd try to start with. It is important that they learn to decide themselves. It is important that they take over the lead as much as possible. It is not my show. It is not about how I would proceed. It's more about being a facilitator, to lend a helping hand. When there is a unit to test I ask one more time

What do you want to test?

Now it's about to find the method to test. Often the units picked had one method that would stand for what the whole unit is all about. Anything else in this unit are just private helper methods. Now I would try to nail them down on a certain behavior of this unit, a certain method to test. Usually I would get the same tour through algorithms and underlying layers again. And here it is important to come down to what a unit test is all about: testing a certain behavior for a defined input producing an expected output. And this is what they really have to understand here:

It is about this behavior. That's the reason we write unit tests. To secure a certain behavior of the unit.

It is not about testing all paths and all combinations. It is about testing a behavior which is directly derived from the acceptance criteria of a user story. Any code that is not covered by such acceptance criteria is useless for it implements stuff no one ever asked for. When developers write their tests with the implementation in mind then they might get a high code coverage but what we would really need is a high feature coverage. By asking the question

What (feature/behavior) do you want to test?

one would always stay in the outside-in perspective which helps to keep the acceptance criteria in mind. With that writing down expected results of tests becomes pretty easy for everything follows directly from the acceptance criteria.

So, to get to the point which tests have to be written I very frequently ask my question. It proofed to be of use to not let them go the whole tour again although they always try. Early interruption is required. But be careful. Because this repetition easily will become very annoying for the people it would be wise to guide through additional questions to point them to the expected behavior or to the obvious negative tests that would be needed. If the procedure does not seem to come to an end you should propose something by asking: What if we try this and that? In the end we want to write unit tests. We don't want to drive the developers crazy.

Especially when we talk about legacy code it is more like covering the existing code with negative tests for these are usually difficult to write as end-to-end tests if at all. This would add value to the existing test base and yields fast positive return on investment by disclosing possible fatal errors that would cause the application to crash. There never has been a session without at least one of these. In most cases these tests are quite easy to write and the bugs they are disclosing are often easy to fix. With that the developers would have a successful first unit test session. In later sessions the very same technique could be used to bring about the positive tests as well.

A checklist to summarize:

Pro:

• focuses on feature/behavior instead of algorithms and lines of code 
• opens up your mind for new approaches
• maybe the existing implementation could be improved?
• forces to divide the feature into portions to be tested 
• unveils structure of the feature at hand from an outside in point of view
• discovers possible negative test opportunities instead of focusing on the happy path
• annoys and breaks indifference
• as any question forces the developer to come up with his own answers 
• solution will be better accepted 

Con:

• easily becomes too annoying and causes resistance 
• guidance through hints or additional hint like questions required

Read also:

On Sporadics - How to deal with intermittent tests in continuous delivery

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7

The opinions expressed in this blog are my own views and not those of SAP

Interlude: A practical experience with quarantine

Sometimes one develops nice ideas or puts down some thoughts and then reality kicks in and what has been a nice theory gets under pressure by the events of daily life. Same for me. Recently I was talking about Sporadics and the attitude of shrugging them off. I was talking about people and the need to put them first and processes second. And I was talking about a quarantine to give people a chance to tackle difficult to solve issues without the need to close mainline. I also was talking about my ambivalent feelings towards a quarantine. Just recently these ambivalent feelings proved themselves valid as a quarantine turned into a ever growing bowl of bad apples and regressions entered mainline faster then one would be able to count up to five. What happened?

A quarantine has been set up with these rules:

• any regression would be a possible reason to close mainline
• if the regression turns out to be 
• trivial then mainline will be closed until the fix arrives and proves to remove the regression
• non-trivial and a fix would take some time and it is
• a reproducible regression in a third party component then it will be quarantined for two weeks and mainline remains open
• a reproducible regression in the application itself then it will be quarantined for two days and mainline remains open
• a sporadic regression and
• it showed up the first time then it will be observed as a known regression - no fix will be requested and mainline remains open
• it showed up the second time then it will close mainline and an analysis will be requested, if analysis shows the fix would be
• trivial then mainline remains closed until the fix arrived and proved to remove the regression
• non-trivial then it will be quarantined and mainline will be opened again
• a quarantined regression that gets not fixed until its time limit has been exceeded will close mainline

It turned out to be a hell of a job to enforce this <sarcasm>simple</sarcasm> set of rules and to convince people to consider removal of regressions their primary task and that disabling the test that unveils them wouldn't be the solution if not the test itself turns out to be buggy. That's why I need to touch the topic of quarantine once again.

The original idea would work if certain assumptions would be true:

• "... quarantine would serve as a to-do list of open issues to be worked on with highest priority"
• quarantine rules will be strictly applied

If one of them does not hold the whole concept would be broken. I further expressed my concerns about a quarantine:

"If there is a culture where these things are always overruled by something seemingly more important then a quarantine will just be another issue tracker with loads of issues never to be handled or like the to-do lists on dozens of desks which never will be worked on to the very end. It's just another container of things one should try to do when there is some time left."

And exactly this is what happened. And it made me think. Either the concept of quarantine was a dead end or the way it has been incorporated was plain wrong. What are my observations?

First, the set of rules is non-trivial. It takes some time to understand which possible cases there are and whether or not a regression would close mainline, will be quarantined or just observed. This rule set is hard to remember for the people to enforce it as well as for the people that would need to fix regressions.

Second, if you consider a green mainline the goal to achieve and if you accept the fact that red pipeline runs require a thorough analysis of what went wrong then the design of this quarantine has its flaws. It would allow for red pipeline runs to enter mainline in the case of a Sporadic occurring the first time. Any deterministic regression gets handled by closing mainline or at least putting it under quarantine while non-deterministic regressions could enter mainline.

Third, there has been no tool to put regressions in quarantine and to report on them with every pipeline run in order to make sure newly failing tests show up easily. Instead all test failures were present and required analysis. Many pipeline runs were analyzed that only contained quarantined test failures causing a lot of wasted work.

Forth, the rule set wasn't enforced completely. Regressions that reached their maximum time in quarantine did not cause mainline to be closed. Instead they started littering pipeline runs and along with the missing quarantine reporting it became even harder to find newly failing tests thus regressions a single change introduced.

Obviously this quarantine has been a failure both in concept and in execution. There needs to be another configuration of it to make it work. There needs to be something very simple and straight so that everyone could memorize it and that a tool could enforce it. And the latter would be the basic shift in my mind. Last time I was talking about not having an automatism but to add a human factor:

"Instead of having an uncompromisable automatism which would just close mainline until a proven fix arrived a human would be able to weigh different aspects against each other and to decide not to close mainline even if a Sporadic popped up"

And this was my fifth observation. It seems that at least in an environment where Sporadics are being shrugged of and features are rated higher then bug or regression fixes this human factor would invite people to discuss the rules and to ask for exceptions from them to just be able to do this and that very quickly. A human would be tempted to grant these exceptions and in special cases does grant them. As experience shows in many cases not sticking to the rules turns out to be the cause of a lot of trouble. In the case at hand what was supposed to be a shortcut, a speedup turned out to be a mess and a slowdown of things. Additional regressions were introduced.

With that experience in mind I try to remember what quarantine was all about. Which goals were expected to be achieved?

First of all the baseline of it all is the idea of continuous delivery where release candidates get produced all the time. If a certain pipeline run fails, this run does not yield a release candidate. Continuous delivery allows for frequent releases of small increments of the application thus getting fast feedback on any new features instead of waiting for yearly or half-yearly releases. The increment to be shipped has to be free of regressions. Otherwise it wouldn't be a release candidate for the pipeline to produce it would have been stopped when the error popped up.

So, if a quarantine gets introduced it must make sure

• the software to be shipped is free of regressions and
• the time between two release candidates is as short as possible to allow for small increments

If a regression gets up to two weeks in quarantine then experience shows that it will stay there for pretty much exactly this time. Considering this it would not be possible to quarantine a regression for two weeks because this would spread the time between two release candidates to at least this amount of time. If within two weeks one more regression of this sort gets quarantined the time between two release candidates would grow even longer. One regression every two weeks would make it impossible to produce a release candidate although one regression every two weeks would be a pretty good rate. What's more, the increment would be quite large. If a quarantine of two weeks is not possible then waiting for fixes of 3rd party components would not be possible any more except this 3rd party component comes with releases in a high frequency. With that the application has to mitigate issues in any 3rd party library by not using the buggy feature at all, which may mean to work around of it or to skip the own feature if it is dependent of this very feature the 3rd party component offers and which turned out to be buggy. If this becomes the default mitigation in the application a quarantine of two weeks wouldn't be needed anymore. Independent of the regression being located in the application or in any 3rd party component the fix has to be made in the application itself. The team developing the application would regain control over the application which in itself would be a nice thing to achieve.

So, allowed time in quarantine needs to be short.

Another flaw in the above quarantine rule set has been the fact that there would be a loophole for regressions to enter mainline. The fact that Sporadics need to show up twice before they would cause a close of mainline and with that open up the possibility to quarantine the regression needs to be dealt with. I'd propose to quarantine any regression no matter whether they are deterministic or not. There would be two days to get them fixed. If the fix does not arrive in due time the regression will leave quarantine and mainline will be closed. No test protocols need to be scanned for known not yet quarantined issues. The quarantine tool would report quarantined regressions properly and any new test failure would show up directly. While mainline is closed no other changes could be pushed to it. This would be an optimistic approach for it is based on the assumption that regressions get fixed with highest priority.

The new quarantine rule set would look like this:

• any regression will be quarantined for up to two days
• if the fix arrives within this time and it proved to be valid then the failing test will leave quarantine
• if the fix does not arrive then the failing test leaves quarantine and causes the mainline to be closed
• if the fix does arrive and it proved to be valid mainline will be opened again

that's it. Simple and straight forward. No distinction between types of regressions. If required an upper limit for regressions in quarantined could be added. If this has been reached mainline would be closed also. But the rules to open mainline would get a bit more complex for there are two conditions to consider. That's why I would prefer not to have the upper limit of quarantined regressions in the rule set. The two day limit is tight enough to put pressure on the resolution of regressions.

This rule set is simple enough that a tool could implement it, thus removing the human factor from the decision making whether or not to close mainline. The human factor should be used to decide upon the proper limits for regressions to stay in quarantine for this seems to be dependent from the current situation an application development finds it self in. So in the end the CI server could take care of the regressions by enforcing a quarantine although I wasn't fond of this idea earlier. But it would do so based on a simple rule set which does not require a huge and sophisticated tool and the pipeline runs still would be reported as failures and no release candidate will be produced as long as there are quarantined regressions.

I will report whether this approach works in reality.


Read also:

Part 1: Sporadics don't matter
Part 2: Let the CI server take care of these Sporadics
Part 3: Where do these Sporadics come from
Part 4: Tune up the speakers - Let Sporadics scream at you
Part 5: Process vs. People - Or: Processes won't fix you bugs
Part 6: To Quarantine or Not To Quarantine Contagious Tests


The opinions expressed in this blog are my own views and not those of SAP