Quagmatic | Balancing Quality and Speed

Create a Good Strategy Around Testing

2020-07-23T00:00:00+00:00

When creating a strategy for software testing, it is important to know the ‘Why’ testing is a current priority. For example, if you are working for a startup or a greenfield application with no users on the software, the presence of a defect is not impacting anyone, and could be argued isn’t valuable. While on the other hand you have 1,000 paying customers using a piece of software, that is critical for your customers to run their businesses, the risk is extremely high. Based on where you land on the scale you will have different strategies and approaches. What I will attempt to lay out below are general strategies that can be effectively applied ot many different contexts.

Ditch Your Test Cases

Yes you read that correctly. Ditch them, or at least place them in a nice safe place that you can reference later. This may sound crazy to some but let me just propose this question. How many bugs have your test cases uncovered? This number shouldn’t include that execution where the exact test case passed, but you noticed something was off that wasn’t called out in a test case. Keeping test cases up to date is a very difficult task in itself, and can be really frustrating as requirements and areas of the system change due to new learnings from your customers. So what should you do instead of creating and executing test cases? The remainder of this article will walk through ways to get by without them.

Work Towards a High-Level Understanding of the Software You are Testing

This is the biggest factor needed to ditch the mountain of test cases you and your company have accrued. The team’s mindset has to change from being given exact input and outputs designed months or years ago, to exploring the system for themselves. This can be accomplished in many different ways. Within my company, we have a new team member self-paced training video series that covers customer-facing product features along with administration configuration pages. This has allowed for our testing team to approach testing from a systems-minded perspective. When looking at how the system operates as a whole, our team has a greater understanding of how different areas of the software influence one another. Having a broad understanding of the system gives a great launchpad into deeper learning and exploration of new features or existing features within the system.

Create and Use a Feature Map

Another tool we use to get us to that high-level understanding mindset is a feature map. Our feature map consists of a list of all pages of the system with a brief description or list of links/features that are available on that page. This is another tool that new team members can use to get familiar with the software, or can reference when exploring a new area of the system, that helps visualize the entire system. This tool can also be used to regress the software for code version upgrades, new infrastructure migrations, or risky site-wide changes such as user permission updates. This simple document doesn’t have a lot of detail into the specifics of how the features work but is used as more of a tool to spark testing ideas or as a reminder of the scope of certain features.

Work Towards a Deep Level Understanding of Specific Features Within Your Team

After a new team member has gone through the onboarding process, the real fun begins. Around week three a new tester joins a product delivery team consisting of developers, a designer, a product manager, and tester(s). Within this team, the tester is an integrated part of an agile team participating in sprint planning, refinement meetings, and retrospectives. On these delivery teams, we are adding new functionality to the system through new features, integrations, refactors, and bug fixes. In each of these areas, both developers and testers work together to refine user stories, establish acceptance criteria, and give clear technical direction of how the problem will be solved. This meeting is also a great place for testers to challenge acceptance criteria, to have conversations with developers going over some specific areas in which you plan to apply exploratory testing. By doing this, it is actually possible to find bugs before they are even coded, which is a huge efficiency gain. Identifying these complex or problematic areas come from knowing the new feature that is being built, being involved in the developer discussions. Knowing if the developer is planning on solving the problem with an asynchronous vs synchronous solution, can change the way you test a certain feature. Gaining an in-depth understanding of the new feature or functionality being delivered allows the team to deliver higher quality software products through identifying and fixing those not so obvious defects.

Create Good Documentation for Things That are Not Straight Forward

The agile manifesto contains the statement ’Working software over comprehensive documentation’ with a follow-up statement ’That is, while there is value in the items on the right, we value the items on the left more.’ I’ve found that the majority of the features in well-designed software are very straight forward and don’t require documentation in order to understand what the product is supposed to do. In situations like these having test cases for simple tasks makes no sense to me. However, there are certain things that may require well-written, easy-to-follow, documentation. The question I ask my team on determining if it needs documentation is: “Did you have to have a conversation with anyone to be able to complete the testing?” If the answer is yes, we probably need some sort of documentation. Some examples of this would be testing complex batch jobs, third party integrations, hidden features that may not be listed within the software, or technical testing tasks. For those not so straightforward items, our test team has a shared wiki where we can share and update information on how to test these complex areas. The idea being that once a tester has figured it out, the documentation should be able to walk any other tester without the deep knowledge of the feature through it.

Write Automated Checks for Your Critical Paths

One of the main reasons I believe that test cases became the standard across most test organizations is there are critical paths in our software systems, that should always work. This is a great place to go find that nice safe place you placed your test cases and begin reviewing and prioritizing what are the most critical paths. These are areas that our users rely heavily on and they should behave consistently. For these critical paths, I would recommend having some sort of automated test coverage. This is not a job for your summer test intern to take on. To write reliable, sustainable, consistent automated tests you have to rely on your developers for guidance, or someone who has proven successes in test automation. This is not easy to get right, but once you have a good reliable suite of automated tests you will have higher confidence in the quality of your releases.

Focus on Targeted Regression Testing

In 2014 our full regression testing cycle took one team member two weeks to complete. We would test through our feature map. These were the days when we had one release every two months. A lot of has changed since then. Now we deploy major features on a two-week cadence, with the ability to deploy bug fixes on any day at any hour. With the use of our automated tests for our critical paths, we still do targeted regression. Prior to our release, we have a code freeze, where no new code is added to our proposed release code branch. During this time, the test team works together reviewing the features and fixes that are going to be released to assess the risk of each item. We do quick targeted regression testing to ensure that there were no code merge issues and that what will be deployed is consistent with the previous testing done by the tester on the feature team.

How Do You Track What Has Been Tested

Rather than using test cases, we use test sessions. On every user story or bug ticket, we track what was tested, who tested it, in what environment, and with what test data. We also use these test sessions to communicate everything we tested to mitigate the risks. The idea is that when the ticket is ready to be released, any member of the team can review the session and have a solid understanding of what has been tested.

In Conclusion

I don’t hate test cases. I think there is a time and a place where a solid set of test cases make a lot of sense. I would also never tell a tester that was on my team or another team that their work is less valuable because they are using lots of test cases. I would, however, argue that there are more efficient ways to test a bug or a feature using the techniques outlined above rather than having a batch of test cases executed. Don’t believe me? Go try it for yourself!

Create Quality Metrics that Matter

2020-07-17T00:00:00+00:00

When I look to define what a healthy Software Engineering Culture of Quality looks like, it is the way a team that cares about the external quality (value we bring to the customer) and internal quality (value we bring to other developers) of the product being built today in the context of the longevity of how long the product will be used in the future. When you consider all of these facets, the teams with a great Culture of Quality will ensure that decisions, discussions, and actions are appropriately made within the context of their current situation.

One of the ways we can build a Culture of Quality is by measuring what matters. I’ve identified 3 areas that can be summarized in a Quality Score Card for Software Engineering Teams.

Externally Reported Defects

Within this category, I’ve found that there are typically 2 types of defects that are reported externally (by our customers). The first type are issues that are due to new recently released code (within the last 3 months). The second type are issues that have been in the system for longer than 3 months but potentially ever since the software was created. I treat each of these issue types differently.

Externally Reported Recently Released

For the recently released issues those are linked back to the team who worked on the feature and are planned and prioritized along with their current deliverables, ideally being fixed with the next scheduled release. In this scenario a delivery team broke the boy scout rule, leaving the software/functionality in a worse state that it started with.

Externally Reported Backlog

For the defects that have been in the system for longer than 3 months, these get sent to the product backlog as areas of the system to be prioritized by our customer success team and our product team. A Severity and Priority matrix can be used to discuss how bad our customers are affected (severity) and how upset or willing to accept the defect (priority) our customers are. These are important but its possible that many of these issues weren’t caused by your current team, if your working with software that is over 5 years old.

Externally Reported Created VS Resolved

To see how your team is trending, the first metric I like to track is the Created defects vs Resolved defects. This allows me to answer the question: How many new defects are being reported externally by our customers and how are we responding to those reports? Are we making progress towards reducing that backlog size or is it growing? This data can also be segmented to view only externally reported recently released items, and how many of those are being resolved by the delivery team in a timely manner. I typically have a past 30 day view and a past 90 days view to really see any prevailing trends.

Total Open Count Externally Reported Defects

It’s also important to track track an overall count of Externally Reported Defects that are not resolved. This will give you a pulse or allow discussion and investigation around these recently released regressions. It can also be useful to segment this data by severity, priority, and responsible delivery team.

Test Automation Metrics

Tracking certain metrics for Automation in Testing efforts can be very helpful to show off the value of the work being done in this area.

Issues Discovered by Automated Checks

One simple lagging metric thats worth tracking is issues discovered by automated checks. Any issue that an automated check uncovers should be documented and you can quickly see the value or lack of value your efforts in building automated checks are providing. I tend to track this metric month over month so as to see overall trends. If zero issues are discovered by automation in a given month that is not necessarily a bad thing.

Functional and Unit Automated Check Counts / Coverage

There are ways to determine what areas you should focus your efforts building up automated checks, I will not cover that here but will cover the counts or coverage of checks in place. This metric will be very subjective as what you want your coverage to be is something you have to define. The way I’ve found best to track this is to create a Feature Map of your system, every action/link on a page broken out mapping out the entire system. From this established list, decide if an automated check is necessary to cover this area of the system. If so mark it as something you intend to add an automated check for (todo). If it has any sort of automated check that touches that feature mark it as automated (yes). If it doesn’t have any automated check and you don’t plan to add one in the near future mark it as (no). One caveat to this method is things that are marked as yes automated may not have the appropriate level of coverage, if that is the case feel free to mark things that have coverage but need more as todo on this list. What this will give you is a way to measure the coverage of your automated checks, and give you a nice backlog of items that can be planned and tracked via burn down chart.

The simpler way to track this by total count, and broken down by the area of automated check counts. This again will give you some information around how many checks are in place but this number doesn’t give information about the quality of the checks.

Most unit test frameworks have built in code coverage metrics so getting a coverage % is a lot easier. I tend to use this and set a baseline that the team works towards staying above with each commit.

Release Quality Metrics

The final category of metrics can be useful in gauging how many defects are found in production by your customers due to a recent release, and how quickly the team responsible responds.

Externally Reported Change Fail Percentage

This metric should give us a good indication of how often we break functionality for our customers due to a release. Some may not find it useful to track every single externally reported defect, but only include defects above a certain severity or priority. This is where you can make this metric work for your team. Gather the data and focus in on what you want to improve. I tend to track this metric month over month. I first gather any externally reported recently released defects that have been reported by customers, and verified as actual defects. During the initial research and triage, an attempt is made to identify the root cause, like the ticket, and link the custom field we have added in Jira ‘Related to Fix Version’. Then I will gather every change (user story, bug fix) that was deployed during a release in the month I’m measuring. From there I follow the formula below to calculate the Change Fail Rate Percentage.

X Changes released to production (any tickets linked to the release) |
Y Issues reported against those releases |
(Y / X) * 100 = Change Fail Rate Percentage |

Once you have a few months of established data I recommend setting an aggressive target change fail rate. For the project I’m currently working on, my target is 2%.

Mean Time to Repair (MTTR)

Mean time to repair helps us track how quickly we react to externally reported recently released defects. To measure this we we identify defects above a certain severity or priority that are important for us to respond quickly to. The

4/1 - Defect1 Opened | Resolved 4/5 - 5
4/2 - Defect2 Opened | Resolved 4/5 - 4
4/6 - Defect3 Opened | Resolved 4/6 - 0 (resolved same day)
4/19 - Defect4 Opened | Current day 4/22 Not resolved open (not counting this)
MTTR = 3 | (9 days in progress) / (3 number of issues that have been resolved)

Using this metric can help surface data to the team how quickly they are responding to defects they have released recently. This is a lagging metric, it doesn’t track defects that aren’t resolved yet. Mean time to repair can be measured in many different ways but this is how I tend to measure it when dealing with software delivery teams.

Lead Time for Hotfix

Another useful metric though less around Release quality is, how long does it take for a piece of code to get from test complete to a production environment. This is assuming a followup release hotfix to resolve a released defect. This value should include any CI/CD pipelines (build,deploy,test) along with any release processes needed to get the fix to a production environment. It should not include time to write the code or test the code as those times will never be consistent due to the complexity of the changes. The release process however should be something that is consistent

Deployment Frequency

For this metric I like to track 2 things. How many scheduled deployments do we have per month, and how many deployments did you actually have. The first metric may be less important for a team working on decouple systems already practicing continuous deployments, but for many teams still working on monolithic applications, the release train model is used, with scheduled releases. Tracking these can give insight into how many changes get deployed per deployment, giving insight into batch size.

Conclusion

There you have it! My list of Quality Metrics that Matter. From each of these metrics I’ve found it useful to create a quality score card that get shared with the team and the leadership team. Being able to see positive or negative progress should help drive better decisions, when it comes to considering quality. Should we speed up and allow our externally reported defects to fill the backlog? Should we slow down, and spend more time testing, releasing with less risk, ect? Should we hire another test engineer to help test or add additional checks to the automation framework? Are our automated checks even useful? I see we have 10 recently released defects and we don’t have any recent issues discovered by automation. These metrics will give useful insight into your teams work.

Do you have any useful metrics that I haven’t covered here? I would love to hear about them, please share them in a comment below!

Building a Software Engineering Culture of Quality

2020-07-14T00:00:00+00:00

You may have heard it said “Culture eats strategy for breakfast”. Do you believe it? Honestly if you break it down in the literal sense, after culture has eaten the strategy; strategy becomes a part of the culture, the way we work, the things we do, the way of life. I believe that a good strategy and a good culture are both needed for any team that wants to deliver high quality software to their customers.

Culture is

Before we go too deep let’s define ‘culture’. Wikipedia defines culture as ’a word for the ‘way of life’ of groups of people, meaning the way they do things.’ Thinking of culture applied to a team at work, culture boils down to the words we use, how a teams interact with one another, what the team does on a day to day basis. It’s more than just who they are but it’s how they act.

Strategy is

Now strategy can be defined as ‘a long term plan on what to do to achieve a certain goal.’

When you think about what the strategy of a software delivery team is you will find processes, and systems that have been put in place to accelerate delivery, mitigate risk, or even get faster feedback from customers. Through these processes and rules you will see the controls in place and the overall flow of how software gets delivered to customers.

What you won’t see is how software and test engineers discuss the code in refinement or planning meetings. The specific choices and actions the teams take as they are writing code within each step in the workflow. The back and forth ping pong match between the developers and testers when a developer says the story is ready for testing before all of the acceptance criteria is met within a user story. The corners that get cut when writing automated unit, integration, or ui checks. All of these items are more than ‘strategy’ but encompass the ‘way of life’ of the team, the culture.

Quality is

This bring us to the question what is the definition of quality? Gerald Weinberg says:

‘Quality Is Value To Some Person’

When it comes to quality there is also a concept that the ‘some person’ may not always be the customer. If you have ever worked in a legacy code base that was not maintainable, didn’t have unit tests, was inconsistent, or not easy to make changes to without breaking some other area of the system, you have more than likely experienced poor internal quality. These are all areas that would degrade the ‘quality’ of a codebase.

When I look to define what a healthy Software Engineering Culture of Quality looks like, I would say it is the way a team that cares about the external quality (value we bring to the customer) and internal quality (value we bring to other developers) of the product being built today, in the context of the longevity (how long the product will be used in the future). When you consider all of these facets, the teams with a great Culture of Quality will ensure that decisions, discussions, and actions are appropriately made within the context of their current situation.

Building a Culture of Quality

Create a Good Strategy Around Testing
Lead the Team In Good Coding Practices
Hire the Right People Who Care About Quality
Coach the Team Towards Delivering Value
Create Quality Metrics that Matter

Why Quagmatic Anyway?

2020-07-09T00:00:00+00:00

quag·mire: an awkward, complex, or hazardous situation.

au·to·mat·ic: working by itself with little or no direct human control

During my time as a leader on within software engineering teams I’ve found myself over and over again faced with complex and hazardous situations when it comes to development and delivery of high quality working software. Navigating these waters has shaped the way communicate, empower, and guide my teams to success. One one of the keys to success in navigating these situations falls back to building processes and tools that manage themselves. Whether it’s a release train process with clearly defined roles, or a suite of automated tests that give fast feedback to the engineers consistently, hence the name Quagmatic.

On this site I plan to share things I’ve learned, questions I’m still trying to answer, other things I found interesting in the area of Continuously Delivering Quality Software.