Automated testing (I\u2019m deliberately ignoring manual testing in this post) has become a hot subject for which many people have strong opinions, especially when unit testing and system testing are involved (integration and component testing seem to be a bit less controversial). The debate, typically, is around what kind of testing is more effective in helping to find and prevent software defects when developing applications.\u00a0I think each type of automated testing has its place. However, in my experience, too much emphasis on system tests can slow a project down to a crawl or even kill it. This applies to system of all sizes: from small ones developed only by one person, to big distributed systems developed by several geographically distributed teams. The only exceptions I\u2019ve seen were quick throwaway applications, for which often there are very few, if any, smoke tests.<\/p>\n
In particular, in the (quite a few) legacy systems I worked on, the difficulty in maintaining and evolving them was never due to the absence of tests per se\u2014system tests, in a way or another, were always present\u2014but the absence of more finely grained ones (e.g., integration tests, component tests, and, especially, unit tests), which made it extremely difficult or impossible to change some parts of those systems safely and in a reasonable amount of time.<\/p>\n
In the following sections I\u2019ll explain the reasons in more detail.<\/p>\n
The reasons for testing a software system can be summarised in two broad points:<\/p>\n
The first point should be obvious as it is what, traditionally, tests are supposed to help with. However, even if that point is very important, I think the second one is more interesting.<\/p>\n
Successful software systems, not only change and evolve for as long as they are used, but, nowadays, they are often delivered incrementally with new functionality added with each increment. Therefore development teams need a mechanism to check that they haven\u2019t broken any existing functionality (or any system quality) in the process.<\/p>\n
Automated testing, albeit having its limitations (e.g., according to some research, formal code inspections can find more bugs), can be a very convenient technique for that purpose, as it is easier to learn and apply, scales better and is cheaper than other techniques (e.g., formal proofs, and formal code inspections).<\/p>\n
However, there are some caveats: in order to help with the development and evolution of the system, the tests need to<\/p>\n
Unfortunately, those are areas were system tests have some serious limitations.<\/p>\n
System tests are very useful\u2014some qualities, like performance, scalability, usability, and other qualities, are, in many cases, better tested at the system level\u2014however:<\/p>\n
The last point is probably the most important one\u2014and missed by many, who tend to focus on execution speed instead. Big systems are composed of smaller components, which, in turn, are composed of smaller components, and so on and so forth until we reach class, function and method levels.<\/p>\n
Using system testing to see if a method or class works as expected can give is equivalent to check that a small cog in the engine of a car works well by fully assembling the car first and then by trying (I\u2019m using \u201ctrying\u201d because, if the various components have not been tested in isolation, many or all of them may not work at all) to run a few laps in a track with it.<\/p>\n
That approach is inefficient, and ineffective. Is inefficient because every small change would require a longer than needed testing procedure (making code refactorings a major chore, therefore avoided as much as possible). Is ineffective because the system test may still pass or break for reasons that have nothing to do with the small changes being tested.<\/p>\n
The limitations above have quite a big impact on what is tested in practice\u2014system tests are generally used to cover some positive scenarios and a few negative ones, leaving the majority of the possible scenarios out\u2014making over-reliance on system testing quite a dangerous practice. In fact, all the systems I\u2019ve come across that were developed relying almost entirely on system testing, were, without exception, bug-ridden and very difficult to maintain and evolve.<\/p>\n
If we add multiple distributed teams to the mix, with each team developing one or more components of the entire system\u2014something quite common these days\u2014the problems above get orders of magnitude worse. I\u2019ve seen those problems first hand several times: I\u2019ve been involved in quite a few distributed projects where the almost total reliance on system testing brought the development activities (not to mention bug fixing and production support) almost to a standstill. The more the teams the worse the situation.<\/p>\n
A better option, in my experience, is to rely mainly on lots of unit testing and an healthy amount of component and integration testing, and use system testing mainly to make sure the system hangs together by running the main acceptance and performance tests.<\/p>\n
In cases of systems developed by multiple teams, unit and component tests become even more important. Teams working on components depending on each other need to agree on how those components will communicate (they will have to do this anyway), and use that information to create the necessary stubs on their own environments so that each team will be able to work (and to run their tests) in isolation. When all the teams are ready then they can run some integration and system tests to verify there were no misunderstandings in the definition of the communication protocols of the components involved.<\/p>\n
I\u2019m not against system tests\u2014I usually like to work outside-in starting with a failing end-to-end test\u2014but I think that they are not the answer to all testing needs. An over-reliance on them can bring development and maintenance activities to a standstill first and to a rewrite from scratch as the next step (which comes with its own problems, and is very rarely a good idea).<\/p>\n
Other types of testing like unit and component tests are more suitable than system ones in many situations.<\/p>\n
A final observation. In the projects I\u2019ve been involved with, the problems in the ones with worthless unit tests were not due to unit testing per-se, but to a lack of design and automated testing skills inside the teams. In fact, the production code in those systems was invariably bad, and the system tests weren\u2019t in a good shape either.<\/p>\n
So, if you are thinking about putting a lot emphasis on system tests for your testing activities, my suggestion is to take a step back and think again, or your project may be killed as a result.<\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction Automated testing (I\u2019m deliberately ignoring manual testing in this post) has become a hot subject for which many people have strong opinions, especially when unit testing and system testing are involved (integration and component testing seem to be a bit less controversial). The debate, typically, is around what kind of testing is more effective […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[4,11],"tags":[],"series":[],"class_list":["post-302","post","type-post","status-publish","format-standard","hentry","category-software-development","category-testing"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p2KmbL-4S","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/posts\/302","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/comments?post=302"}],"version-history":[{"count":6,"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/posts\/302\/revisions"}],"predecessor-version":[{"id":7994,"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/posts\/302\/revisions\/7994"}],"wp:attachment":[{"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/media?parent=302"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/categories?post=302"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/tags?post=302"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/asprotunity.com\/blog\/wp-json\/wp\/v2\/series?post=302"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}