Every software project comes with its own challenges. Most likely, you are balancing cost and development time with the need to release a product. But you can’t just send something to market without making sure everything is working correctly. Even the most reliable software available has some faults. Under the most ideal conditions, software that runs perfectly on the majority of computers is still likely to have minor issues that conflict with some users.
So, how are you supposed to know the best way to test your product to make sure it is ready to go to market? Watch our video to learn about the best ways to test during the design and development process, so you can consider how this applies to your own planning for a successful launch.
Video Transcript:
The Importance of Software Testing in Australia
The ripple effect
Let’s look at software testing. Here are some things that the industry knows to be true. People who work in the industry know this is true, but sometimes people outside the industry might think ‘is this really true’? First, all software has defects. There is no software anywhere in the world that I know of that doesn’t have defects. Enhancements often add defects. So adding something new often makes other things break.
I wish some of these were not true. These are the pain in my life. I wish it wasn’t the case. Any system of complexity cannot be held fully in the mind. So as a developer, once a system reaches any reasonable amount of complexity, you cannot hold all of the details of that system in the mind. You will make a change and you do not know the full implications of all of those changes.
What often happens is we’re working on something and it’s like ripples in a pond. We can appreciate ripple one and ripple two, we know the immediate things but there will be implications of changes that will happen four and five ripples out that there is no way for a human to know in their mind.
Undetected defects get more expensive to fix over time. This is true throughout the software development process. Let me give you an example. We’re building a piece of software. Let’s say we’re doing it for David and there is a defect. He picks it up before it’s released. There is a cost involved in rectifying that. But imagine that doesn’t get picked up and instead gets picked up a month after it’s released. There could be cost in terms of reputational issues and other bits and pieces.
But just the straight cost of development will generally be higher. The reason is you often have to fix the data that was wrong along with the code. There are actually two parts to that. So the sooner we can get to defects, we reduce the cost of the defects.
Testing and code reviews are our best defense
There are other ways to avoid defects in software but they are too cost consuming. You can do mathematical proofs of software and other bits and pieces. But really it comes down to testing and code reviews as the best defense that we have.
So we need software testing. Let’s have a little look at who can test. We’ve got four key aspects of who can test. We have the developer who can test their work and developers should test their work. But the typical life of a developer, they want to work, they will often put their head down and put headphones on and not talk to anyone for two to three hours and that is why we buffer with phone call interruptions in our office. An interruption messes them up for twenty minutes. They get in their zone and they are thinking really hard.
What happens when they come to test, they’re so focused on that bit there, it’s actually hard to think outside the implications of that. We can have someone in our organisation testing who is not the developer. Historically that has been the project manager, we’ve more recently brought in professional testing to help with that.
Clients can test. You’ll hear about a thing called UAT, which is user acceptance testing. There are two sorts of testing that clients can do. There is testing which is just does it work and there is testing that is did I get what I paid for? Did I get the thing that I agreed to buy?
The last one is we can do automated computer based testing. This is where we get a piece of software to do the testing for us. I’m going to talk a little bit more about that. Let me just explain how automated computer testing works. It looks really freaky when you see it.
Automated testing
Imagine that I’ve got your web page and I’ve got a little bit of software and I click a button and all of a sudden, without touching anything else, it opens up, I put a login detail in, it goes to the next page, it clicks on this, it opens that thing up, it does this and it does this. The system is set up to run all of those key interactions and look for standardised outputs. It’s expecting a certain set of outcomes. That is what automated testing looks like. That’s called end to end automated testing.
There is a different sort called unit testing. That’s where you just test a specific and smaller piece of logic on a particular piece. So we’ve got end to end testing and we’ve got unit testing. End to end testing is a really good way to get coverage that something is working.
Let’s have a look at some of the pros and cons.
Pros
The pro is that it is fast. Sometimes these tests we’ve got, are suites that take an hour to run. But I don’t care if something takes an hour to run if a machine is doing it, because I can go and do something else. They don’t take much human time. They give us high reliability. Human testers are still human, they can make mistakes, where computers run reliably.
Something that higher end development firms do and something that we’ve introduced recently is they can become part of your deployment. What this means is, one of the problems with automated tests is, this is a common scenario. I create them, I do something else and I break them. I’ll talk about how automated tests break. But the client is on the phone, so I’ve got to get it out the door quickly and I do it.
What you can do is you can actually make it part of the process by which you save and deploy your code. You can have these tests run without somebody’s choice. You can force some discipline that way.
Cons
Let me give you some of the cons. Firstly they cost money. They have a high upfront cost and they slow the initial development. Let’s talk about some of the implications of that. Earlier today, we talked about wanting to get things to market quickly, getting MVPs out the door. We want to test things quickly. So we have this tension between, hey, I would love to do some automated testing because of the quality it gives us, but it slows us down and has an upfront investment. Anyone who says it doesn’t is wrong. We live in this tension between automated testing and what it gives us with the time that it takes.
Particularly, automated tests are costly to maintain in the face of rapid change. When your interfaces change a bit, say you’ve got a five step process and you work on step three and you’ve got a whole bunch of end to end tests, they all broke. You had to go and fix them all up because that intermediary page changed. So we’ve got some challenges around that and I’ve got some suggestions which I’ll get to in a moment.
The NASA example
I want to give you an example. NASA. This is deciding do I go down the automated testing route. NASA is considered by many as the world’s most reliable software system. They have something like 400,000 lines of code; that’s big, there is a lot of code there. They have only found three post testing bugs over all the time that it has been written. That is the extreme end of quality. But their code took something like twenty to fifty the time the normal budget took to write.
But they live in a space where it’s ok to spend a billion dollars making sure your code is right, because if a shuttle blows up it is worth something between 2-8 billion dollars. The actual payback is completely acceptable in their space. Let’s compare that to an online system. I’ve got a business that does $1,000 a day online. It cost me $10,000 to set up the test suite. Assume that $1,000 a day is pure margin. There is no cost in that. Sometimes there is cost in business, but assume that is pure margin.
My system needs to be down for ten days, we’d notice that generally, it needs to be down for ten days to get that payback in pure economic terms. There are other factors at play, reputational loss, a whole bunch of other bits and pieces. But in some cases, where I’m doing something cheap and cheerful to test a market, to get something out as an early version, the economics of it just don’t stack up. We’ve had that wrestle internally.
One of the things you can do is just use good monitoring to tell you about problems and fix them when they happen. That can sometimes be a better solution.
Key factors to consider
Is your platform likely to change?
This is the big thing for me. What is the issue of reputational loss? How quickly do we need to get to the market? So speed to market and what is our cost over time? If I was waving a magic wand, this is what testing would look like. We’d do some MVP work, we may or may not do automated testing at that point. Once we thought the business had stabilised, we’d make the investment in tests at a stable point and that investment in tests would pay off in terms of reduced defects, improved quality and we’re getting the benefit of it back. The problem is it can be a hard sell with a client to say “ok, now your business is finally stable, you’re finally making some money, let’s make another investment and do a chunk of automated testing”.
One of the things we have done is invested internally in terms of testing resources. There are a number of our long term clients I’d like to have a conversation with that says, “let’s do a deal”. We’ve done a bunch of work, we’ve got your business stable, let’s do something so we can get a suite of tests up and running. If we need to do a deal for that, let’s do that and then we can roll that forward further and I think there would be a great conversation that might come out of that.
Test coverage
This is going to come out like bad news: automated tests are a little bit of an 80:20 scenario. The 80:20 rule says eighty percent of the benefit comes from twenty percent of the activities. When we’re doing automated tests, we typically don’t want to pay for test coverage of everything. We want to pay for test coverage of the key activities that are going to drive our business. Then if we get a bug that comes up, we’ll add a test case so that we resolve the bug and we put a test case in specifically to deal with that bug. So there can be this crazy thing, you have invested in automated testing and you can still get obscure things around the edges. Yes, you can, but overall you decrease your defect rate.
Human vs. automated testing
The other factor is the client’s willingness to human test. There are people in this room who have said, no I don’t want to do automated testing, I’ll do human testing. That is an economic decision that they’ve made and I totally understand that for a given situation.
What do we do? We have two of our own startup businesses for those who didn’t know. We use automated testing on the important stable bits of software. That’s what we do. I’d love that to be a conversation that came out, if not today, then in the following weeks, with people in the room if that was appropriate.
Whether you plan to make one major launch with the occasional update, or you plan to make numerous updates while every version of your software is rolled out, you have to have a solid plan for how to test your software at some point during development. Choosing the wrong process can be costly and may even leave your program open to numerous user problems that could have been identified under better planning conditions.
Make the right choice for testing your software development project by contacting us today.