Regarding "when I noticed that some people fixate on a single kind of test, a single point in that space, and claim that those are the only valuable tests"
hope it's clear that I wasn't suggesting that only one kind of test has value. The FIRST qualities of microtests (we didn't have the term back then) suggested that these are the ones best for TDD, because you can afford them in the tight, inner loop of coding.
Other tests have value, but slower tests are unlikely to be run after each minor refactoring step, like a variable rename.
I still remember those 1990s usenet days, when you or Ron or someone would mention these ideas and people would just lose their minds. The development of the coding practices has been joyful for me to participate and to watch. Great stuff.
Inspired by your tweet, I define a unit test as "a unit test is a small test." It is not about the size of the code under test. Small means fewer lines of test code, specifically the first A of the AAA pattern. If the tests can differentiate from each other by just a few lines of code, they are unit tests to me. I usually can achieve everything in your list, even if my test exercises thousands of lines of code, using a database, and there are hundreds of them. My e2e tests using multiple services can achieve almost everything on your list, except they are a bit slow. You might not want to run all of them 2 or 3 times for no reason, but they are bearable.
In summary, I don't do unit testing. My units (whatever that means) do not have their tests. I do unit tests, which are small tests decoupled from the code structure.
Would you consider tests that touch database and/or the file system isolated if they dont affect other tests? I dont have much experience programming professionally but there have been some instances where I found out I wasn't really testing anything by verifying the mocks were called with the right arguments. The "behvaior" I wanted to test was does this record gets inserted into the database given the username is not taken.
Using Docker can help with resetting the database (just start a fresh instance) of course to keep tests fast it does require quite some hardware at the moment. Sometimes I like having one container using multiple databases (one for each test case)
I agree the order wont matter in this case but tests may not be able to run in parallel. The thing that confuses me is: "Isnt the updating a database a part of the behavior? And if so shouldn't tests written in TDD fashion check whether the database was updated rather than checking whether a mock was called with right arguments?"
Do we need to test that the database actually works? I would rather not need a running database for unit tests. I probably would like to have some post deployment smoke tests that prove all my plumbing is in place.
How do I know if my software works as per requirements if I dont test the database? Plumbing is not the only place where database interaction can go wrong. I might execute the wrong insert/update. Maybe I have a new column and I forget to update that?
You have to ask yourself if updating some table in a database really is a part of systems behaviour as per requirements or is it just your implementation detail. What if your database admin decides to normalize/denormalize the schema because it will improve performance? Your tests will fail. And end user doesn’t care about your database at all. Classic example is that if you want to test if user profile update functionality really works, don’t test that you call your database API, but instead, verify that after user has submitted request to edit his profile with new info, when he requests to read his profile data again, he sees that change in the response returned to him
What about attributes like performance, security, etc.? Wouldn't they need to be kept in mind while writing the tests? In which dimension are they included in?
I've never found a way to test for security. Performance can be tested for, usually at considerable expense. The only unit-scale test you can do is, for example, test that a certain block of code doesn't exceed some number of database calls.
Have heard many say that performance, security, etc. have to be consciously kept in mind when every piece of code is written, so, during TDD, I presume these attributes have to be consciously kept in mind.
I had to back up and confirm the thread topic. Venkat, test automation related to performance or security is rarely a micro (unit test). Since this thread is about unit testing, security and performance testing are rarely built/executed at a micro level since:
* performance testing typically wants to answer the question: does the system as a whole operate/respond in a timely fashion
* security testing typically wants to answer the question: is the system as a whole secure.
All that said, yeah, sometimes there is a change in a specific class or function to address performance or security in which case, you could unit test that because the developer kept that principle in mind and knows how to create an automated test.
In the context of unit tests, and especially with TDD (although it is not specifically mentioned in this thread, but that's the idea), it would be tough to separate out 'test automation' and 'writing code'. The idea is to write tests that drive the development of code. While we do this, it is important to keep various aspects like performance and security while we write every line of code, and not keep aside a separate effort called 'test automation', which is the silo we are trying to avoid anyways.
I was reading about DevSecOps, and for that matter any effort that focuses on a specific area of the system, instead of having that as a separate area, it is important for the developer to have that mindset while coding itself (along with the experts on that area to coach during pair programming or ensemble programming), so that the area gets considered while coding itself (read 'unit testing' itself). I find this as a very helpful practice. It helps with the concept of whole team approach towards Quality, and can avoid major issues like the recent log4j vulnerability and the recent Roblox gaming performance issue.
True that 'performance' and 'security' are system-level considerations, but being conscious about the system-level at the unit-level granularity itself is required.
Yes. These are principles from SAMM (Software Assurance Maturity Model https://owaspsamm.org/model/). And we want our developers to do what you said--keep security and performance in mind. As far as test automation goes, finding the right tier (macro or micro) and discovering what can be run in a continuous fashion, daily, or is better left to manual testing varies by situation. But it's getting better with tools like FOSSA, 42Crunch, (though IMO getting worse in the native mobile space). :-)
If anyone can go deeper than what's described in this thread and want's to record a podcast on SAMM model security or performance testing, go tweet @AgileThoughts1.
Great list. Glad it started from isolation. I've seen too many code bases treating tests as grand architecture of reusable crap that no one can reason about.
Regarding "when I noticed that some people fixate on a single kind of test, a single point in that space, and claim that those are the only valuable tests"
hope it's clear that I wasn't suggesting that only one kind of test has value. The FIRST qualities of microtests (we didn't have the term back then) suggested that these are the ones best for TDD, because you can afford them in the tight, inner loop of coding.
Other tests have value, but slower tests are unlikely to be run after each minor refactoring step, like a variable rename.
That wasn’t meant to be a personal callout. This time 😉
:-D
Thanks for mentioning the article.
I'll try to be on my best behavior.
I still remember those 1990s usenet days, when you or Ron or someone would mention these ideas and people would just lose their minds. The development of the coding practices has been joyful for me to participate and to watch. Great stuff.
Inspired by your tweet, I define a unit test as "a unit test is a small test." It is not about the size of the code under test. Small means fewer lines of test code, specifically the first A of the AAA pattern. If the tests can differentiate from each other by just a few lines of code, they are unit tests to me. I usually can achieve everything in your list, even if my test exercises thousands of lines of code, using a database, and there are hundreds of them. My e2e tests using multiple services can achieve almost everything on your list, except they are a bit slow. You might not want to run all of them 2 or 3 times for no reason, but they are bearable.
In summary, I don't do unit testing. My units (whatever that means) do not have their tests. I do unit tests, which are small tests decoupled from the code structure.
Would you consider tests that touch database and/or the file system isolated if they dont affect other tests? I dont have much experience programming professionally but there have been some instances where I found out I wasn't really testing anything by verifying the mocks were called with the right arguments. The "behvaior" I wanted to test was does this record gets inserted into the database given the username is not taken.
If the database is guaranteed to return to the initial state, then the order of the tests can’t matter. The tests may not be able to run in parallel.
Using Docker can help with resetting the database (just start a fresh instance) of course to keep tests fast it does require quite some hardware at the moment. Sometimes I like having one container using multiple databases (one for each test case)
I agree the order wont matter in this case but tests may not be able to run in parallel. The thing that confuses me is: "Isnt the updating a database a part of the behavior? And if so shouldn't tests written in TDD fashion check whether the database was updated rather than checking whether a mock was called with right arguments?"
Do we need to test that the database actually works? I would rather not need a running database for unit tests. I probably would like to have some post deployment smoke tests that prove all my plumbing is in place.
I think that might depend on the complexity of the database. Mocking the database is usually great for performance, but might hide some problems.
Most of my queries run under 5ms locally on a cheap ssd. 200 of my tests take about 3 seconds to run when using 3 threads
How do I know if my software works as per requirements if I dont test the database? Plumbing is not the only place where database interaction can go wrong. I might execute the wrong insert/update. Maybe I have a new column and I forget to update that?
You have to ask yourself if updating some table in a database really is a part of systems behaviour as per requirements or is it just your implementation detail. What if your database admin decides to normalize/denormalize the schema because it will improve performance? Your tests will fail. And end user doesn’t care about your database at all. Classic example is that if you want to test if user profile update functionality really works, don’t test that you call your database API, but instead, verify that after user has submitted request to edit his profile with new info, when he requests to read his profile data again, he sees that change in the response returned to him
What about attributes like performance, security, etc.? Wouldn't they need to be kept in mind while writing the tests? In which dimension are they included in?
Do you mean tests of system attributes like performance & security? Or do you mean that the tests themselves should be performant & secure?
I meant the system attributes. But yes, tests themselves should be performant & secure as well.
I've never found a way to test for security. Performance can be tested for, usually at considerable expense. The only unit-scale test you can do is, for example, test that a certain block of code doesn't exceed some number of database calls.
I've heard about "Secure by Design", although I haven't personally gone through it yet:
https://www.amazon.in/Secure-Design_p1-Dan-Bergh-Johnsson/dp/1617294357/
Also, DevSecOps by Glenn Wilson gives a layered framework on a general overview level:
https://www.amazon.in/DevSecOps-producing-compromising-continuous-improvement-ebook/dp/B08QRRNX6K
Have heard many say that performance, security, etc. have to be consciously kept in mind when every piece of code is written, so, during TDD, I presume these attributes have to be consciously kept in mind.
I had to back up and confirm the thread topic. Venkat, test automation related to performance or security is rarely a micro (unit test). Since this thread is about unit testing, security and performance testing are rarely built/executed at a micro level since:
* performance testing typically wants to answer the question: does the system as a whole operate/respond in a timely fashion
* security testing typically wants to answer the question: is the system as a whole secure.
All that said, yeah, sometimes there is a change in a specific class or function to address performance or security in which case, you could unit test that because the developer kept that principle in mind and knows how to create an automated test.
In the context of unit tests, and especially with TDD (although it is not specifically mentioned in this thread, but that's the idea), it would be tough to separate out 'test automation' and 'writing code'. The idea is to write tests that drive the development of code. While we do this, it is important to keep various aspects like performance and security while we write every line of code, and not keep aside a separate effort called 'test automation', which is the silo we are trying to avoid anyways.
I was reading about DevSecOps, and for that matter any effort that focuses on a specific area of the system, instead of having that as a separate area, it is important for the developer to have that mindset while coding itself (along with the experts on that area to coach during pair programming or ensemble programming), so that the area gets considered while coding itself (read 'unit testing' itself). I find this as a very helpful practice. It helps with the concept of whole team approach towards Quality, and can avoid major issues like the recent log4j vulnerability and the recent Roblox gaming performance issue.
True that 'performance' and 'security' are system-level considerations, but being conscious about the system-level at the unit-level granularity itself is required.
Yes. These are principles from SAMM (Software Assurance Maturity Model https://owaspsamm.org/model/). And we want our developers to do what you said--keep security and performance in mind. As far as test automation goes, finding the right tier (macro or micro) and discovering what can be run in a continuous fashion, daily, or is better left to manual testing varies by situation. But it's getting better with tools like FOSSA, 42Crunch, (though IMO getting worse in the native mobile space). :-)
This podcast episodes describes a mental model (the test pyramid) to help get intentions clear, though it doesn't speak directly to security: https://agilenoir.biz/en/agilethoughts/test-automation-pyramid-series/
If anyone can go deeper than what's described in this thread and want's to record a podcast on SAMM model security or performance testing, go tweet @AgileThoughts1.
Great list. Glad it started from isolation. I've seen too many code bases treating tests as grand architecture of reusable crap that no one can reason about.