Developing a stress-free software

The most controversial part of software development is testing. Not only this step is the closest to the project deadline, but also, with all the changes to the initial design, is very crucial to delivering a usable solution.

The rant

The idea behind testing is that you get a black box, which is a software project or a module, or some part of software, and then you poke it around until it does not behave as expected, or straight out breaks. In reality, of course, everyone involved in the project tests its parts in one way or another: developers test new features as those are developed, project managers test early prototypes or user experience elements to give constructive feedback, testers make sure the software meets the quality requested.

All of the test cases, whether those are manual, functional, unit, integration tests, follow a similar pattern – you have a block, module or unit, which can 1) accept data in 2) give data out. The testing principle is that for the given conditions, e.g. given specific data, you get certain and previously known outcome. The quality of tests is determined by how many different “edge” and “normal” cases a tester can produce, in order to be “sure” that the feature works as expected.

As you can imagine, most of the testing, whether performed by developers, testers or managers, is not very good. There aren’t many edge cases tested, some software parts are not tested at all. The quality level of “seems to be working” or “It’s done, trust me” is the norm. Turning a blind eye is actually easier and involves much less stress than going the distance and it’s soo much faster! (we’re agile, after all)

The sheer amount of tests needed to launch a qualitative project is just insane, no surprise it never gets done. Most people will test only the “crucial” bits of software and leave out the rest. But that’s just worse. It gives the false hope that the software is tested, even though it’s partially tested, event though those parts may or may not be written in good will.

One of the best stories I have heard is “we write tests, when we have time. we run them before deployment, unless they fail. if they fail – we turn them off and deploy the project anyway, because we need to”. It’s not a hyperbole. Actual work ethic I’ve seen with my own eyes.

The turning point

I painted a dark-ish picture around testing in general and making sure the software is working as expected in particular, because I have seen those problems repeatedly. Bad test cases and, as a result, bad software gets written not because the people are bad, but because they are unprofessional in majority, and, occasionally, just tired or not in the mood. And good tests require 1) understanding of the domain well 2) being a professional, well versed in the testing field 3) being awake and alert.

The thing about professionals is they are good for breakthroughs. The bad thing – you cannot build a consistent system around them. When writing tests, it’s really hard to quantify the quality of the test. How do you know the test is good? By amount of test cases? By the amount of time it takes to test? Because you trust the person who wrote them, he’s a professional? Those do not display quality of work, but just quantity. Even the test coverage is a mere approximation of “well tested”.

From my experience, the best way to measure the quality of software is by measuring it in the business domain. If the software is written well, it produces results for the business. How do you measure the results? You start collecting events from everything that your software does, and actually looking at them, consistently.

Your software is built around getting more registrations? Start collection registration events. Collect event the failed ones. Especially the failed ones! May be you made a “surname” field required, and everyone thought it’s correct, but most of your user base does not have surnames or someone has two. Maybe your registration process is just shift and fails all the time. Count number of successes vs fails.

Your software is built around selling things? Count which products people put into the basket, count how many people abandon your basket and do not go through with the sale. Sure, those are business metrics, and when they drop, or change, there might be a business reason. But when the developers change things, there can be a shitstorm as well. By following the numbers – you are protected.

Not only you, but everyone can look at the numbers . Developers when introducing new features, to make sure the right events are actually collected at the right time. Testers when performing manual (or automated) checks, because data can tell you about what’s happening, and testers job is basically to check that. Managers can view the numbers and decide which ones are crucial for the business, which ones for product development, which ones are just fun to watch.

The best thing, collecting events from your software is cheap. You don’t need to think of every case that may or may not happen with the software. You know what your software should do, you just need to count how many times it does that. And the next time, after the next change, when the registrations go from 10 a minute to a flat zero, you will know there’s a problem, and with enough events collected, you will know exactly how to fix it.