I really like this post, in particular the thoughts on N x M.
If a little bit of self-promotion is allowed, I wrote a blog post about a year ago that came to a similar conclusion. I think of it as a way to build robust test suites. In my ideal test suite, only a a single test would fail whenever an error occurred. I also think it could be linked to Separation of Concerns, Don’t Repeat Yourself, or the Single Responsibility Principle.
Adding my 2 cents to this. What I see is there's definitely a group (albeit small) of us that write and compose tests like this. Calling it composability and redundancy wouldn't be my first choice. I've used composability to explain some parts of it in the past. What sticks the most with engineers who are new to this concept is the idea of "1-red-per-cause". Spinning up thesaurus real quick, that's more related to cohesion or exclusivity.
my worry with the composition above is that if there is a bug in doSomething(), then the state in which we execute doSomethingElse() is unknown. Continuing might be generating errors in doSomethinElse(), which are actually not a bug in doSomethinElse() but the result of executing it in a broken state.
One alternative approach could be to make it possible in the test to explicitly establish the state that doSomething() is expected to bring about. E.g.,
object := new Whatever(readyForDoSomethingElse)
result := object.doSomethingElse()
Yet another alternative would be to explicitly assert on the state that we expect object to be in after doSomething(), eg
test2()
object := new Whatever()
object.doSomething()
assumeTrue(object.isReadyToDoSomethingElse())
actual := object.nowSomethingElse()
assertEquals(expected, actual)
Some test frameworks assign a special state to tests that fail because of a failed assumption, so we might know that test2() did not fail because nowSomethingElse is broken, but because the assumptions we made do not hold
@Matteo what's the purpose of making test2 fail? I want my tests to fail only when they can tell me where the problem is. If test2 fails and says "hey, look at test1" I don't find that to be any additional information than test1 failing.
Especially counting in the possibility of them changing out of sync and that alert possibly being false.
I find expressing the "precondition" explicitly to be left in the domain of designing your production code so that is possible, rather than using test framework gimmicks.
the purpose of having assertions on the "arrange" part of the test (I'm referring to the usual arrange-act-assert pattern in unit tests), is to alert us about problems in the "arrange". When such an assertion fails in a test, it simply means "hey, you thought that your setup would bring about state X, but that does not seem to hold. So in essence, a failure in an arrange assertion means there's a bug in the "arrange", while when an assertion in the "assert" part fails, it means there's a bug in the "act". If you don't put arrange assertions, you risk looking for problems in the "act" while the problem is actually in the "arrange".
I usually put arrange assertions only when the arrange is complicated enough that I suspect the state I want might not hold; that hopefully does not happen often. And yes, If I was a better designer, that need might never happen, but I'm not always able to achieve that.
Thanks for your comment; it helped me sharpen my thinking about this
Isn’t the difference changing between “fast tests” that we repeat after every line change (which benefit for single failure, single bug, single reason), while in “slow, end 2 end” where more than one commit might get tested pure “less logs to read” is enough reason?
@Matteo forgive me for being pedantic, but I ask you to be precise here. That said, I enjoy this conversation immensely, don't get wrong. You have a sharp mind!
What does it mean _exactly_ that "there is a bug in arrange"? I don't need an answer per se, just want to invite you to think about it.
- What does a complicated arrange say about your system?
- How is "bug in arrange" different from "bug in assert"?
- What about the situation where there's a bug in the arrange _and_ the assert?
- Is bug in arrange still "red" for the sake of team dynamics and stopping the pipeline?
What these come down to is providing multiple "reds" for tests that make reading test results more complicated and flaky. On a wider scale it might say something about the types of tests you are writing. Maybe they are too small in surface area and too big in implementation knowledge. Perhaps you could benefit from more event sourcing patterns to surface side effects?
When applying Arrange-Act-Assert comments to the tests you are helped to reach exactly this outcome. I've always recommended to team members to make the phases explicit.
test1()
# Arrange
object := new Whatever()
# Act
actual := object.doSomething()
# Assert
assertEquals(expected, actual)
The "Asserted Action" from test one becomes "Arrangement" in test 2, so it gets no assertions:
One potentially nice way of test composition is scuppered in JUnit, and most clones, because test methods aren't allowed to return anything. I think I remember discussing this with you a long time ago, and you saying that there was no technical reason that test methods couldn't return something, but JUnit 1 looked for methods that began with test and returned void, and everything since has just followed suit.
I often copy paste tests like this, but mostly for readability.
The cloned assertion in the second test still could make sense for someone reading the test: they know what the preconditions for the next part of the test is.
But looking at this I wonder if it makes sense to have named "parts" to a stateful test.
This might look hard in Java, but I'll use Common Lisp because it's easier to prototype such ideas in CL:
```
(deftest test1 ()
(... object := new Whatever() ..)
(... actual := object.doSomething() ...)
(... assertEquals(expected, actual) ...)
(deftest test2()
(... actual := object.nowSomethingElse() ...)
(... assertEquals(expected, actual) ...))
(deftest test3 ()
..))
```
Effectively, we now have a tree of test states. The test tooling can run each path independently (test1, test1->test2, test1->test3). Perhaps ignore assertions in the previous path.
Does this make things more complex to read? Is this a common enough pattern to abstract away like this? I don't know.
n Go, for example, when you apply composition, you still have to check whether err is nil (or not nil). Look at this code (it's updated to look like Go code):
test1()
object := new(Whatever)
actual, err := object.doSomething()
assertsNoError(err)
assertEquals(expected, actual)
Most methods in Go return an err. You can ignore it, but such code may be misleading for other engineers who might copy/paste it without realizing that it returns an error.
So, composition will look like this, applying your approach. And it's not exactly how you'd want it to look aesthetically:
test2()
object := new(Whatever)
_, err := object.doSomething()
assertsNoError(err)
actual := object.nowSomethingElse()
assertEquals(expected, actual)
I would prefer to give up on isolation and in Go do this (composition without isolation)
I'm not fluent in Go, but the particular issue you're highlighting seems easily fixed by extracting a helper test function so that the code would look like:
```
test2()
object := new(Whatever)
_ := assertSuccessful(object.doSomething())
actual := assertSuccessful(object.nowSomethingElse())
assertEquals(expected, actual)
```
Since Go now has generics, that should be possible, no?
I prefer being explicit here and use helpers only if I have to set up the same things in many tests or if the helper checks the same things. But from my experience, helpers make tests hard to maintain and rework (a tradeoff with duplication).
I really like this post, in particular the thoughts on N x M.
If a little bit of self-promotion is allowed, I wrote a blog post about a year ago that came to a similar conclusion. I think of it as a way to build robust test suites. In my ideal test suite, only a a single test would fail whenever an error occurred. I also think it could be linked to Separation of Concerns, Don’t Repeat Yourself, or the Single Responsibility Principle.
Feel free to read my thoughts: https://henko.net/blog/verify-only-what-you-need/
Adding my 2 cents to this. What I see is there's definitely a group (albeit small) of us that write and compose tests like this. Calling it composability and redundancy wouldn't be my first choice. I've used composability to explain some parts of it in the past. What sticks the most with engineers who are new to this concept is the idea of "1-red-per-cause". Spinning up thesaurus real quick, that's more related to cohesion or exclusivity.
Hi Kent,
my worry with the composition above is that if there is a bug in doSomething(), then the state in which we execute doSomethingElse() is unknown. Continuing might be generating errors in doSomethinElse(), which are actually not a bug in doSomethinElse() but the result of executing it in a broken state.
One alternative approach could be to make it possible in the test to explicitly establish the state that doSomething() is expected to bring about. E.g.,
object := new Whatever(readyForDoSomethingElse)
result := object.doSomethingElse()
Yet another alternative would be to explicitly assert on the state that we expect object to be in after doSomething(), eg
test2()
object := new Whatever()
object.doSomething()
assumeTrue(object.isReadyToDoSomethingElse())
actual := object.nowSomethingElse()
assertEquals(expected, actual)
Some test frameworks assign a special state to tests that fail because of a failed assumption, so we might know that test2() did not fail because nowSomethingElse is broken, but because the assumptions we made do not hold
@Matteo what's the purpose of making test2 fail? I want my tests to fail only when they can tell me where the problem is. If test2 fails and says "hey, look at test1" I don't find that to be any additional information than test1 failing.
Especially counting in the possibility of them changing out of sync and that alert possibly being false.
I find expressing the "precondition" explicitly to be left in the domain of designing your production code so that is possible, rather than using test framework gimmicks.
Hi Denis,
the purpose of having assertions on the "arrange" part of the test (I'm referring to the usual arrange-act-assert pattern in unit tests), is to alert us about problems in the "arrange". When such an assertion fails in a test, it simply means "hey, you thought that your setup would bring about state X, but that does not seem to hold. So in essence, a failure in an arrange assertion means there's a bug in the "arrange", while when an assertion in the "assert" part fails, it means there's a bug in the "act". If you don't put arrange assertions, you risk looking for problems in the "act" while the problem is actually in the "arrange".
I usually put arrange assertions only when the arrange is complicated enough that I suspect the state I want might not hold; that hopefully does not happen often. And yes, If I was a better designer, that need might never happen, but I'm not always able to achieve that.
Thanks for your comment; it helped me sharpen my thinking about this
Isn’t the difference changing between “fast tests” that we repeat after every line change (which benefit for single failure, single bug, single reason), while in “slow, end 2 end” where more than one commit might get tested pure “less logs to read” is enough reason?
@Matteo forgive me for being pedantic, but I ask you to be precise here. That said, I enjoy this conversation immensely, don't get wrong. You have a sharp mind!
What does it mean _exactly_ that "there is a bug in arrange"? I don't need an answer per se, just want to invite you to think about it.
- What does a complicated arrange say about your system?
- How is "bug in arrange" different from "bug in assert"?
- What about the situation where there's a bug in the arrange _and_ the assert?
- Is bug in arrange still "red" for the sake of team dynamics and stopping the pipeline?
What these come down to is providing multiple "reds" for tests that make reading test results more complicated and flaky. On a wider scale it might say something about the types of tests you are writing. Maybe they are too small in surface area and too big in implementation knowledge. Perhaps you could benefit from more event sourcing patterns to surface side effects?
When applying Arrange-Act-Assert comments to the tests you are helped to reach exactly this outcome. I've always recommended to team members to make the phases explicit.
test1()
# Arrange
object := new Whatever()
# Act
actual := object.doSomething()
# Assert
assertEquals(expected, actual)
The "Asserted Action" from test one becomes "Arrangement" in test 2, so it gets no assertions:
test2()
# Arrange
object := new Whatever()
object.doSomething()
# Act
actual := object.nowSomethingElse()
# Assert
assertEquals(expected, actual)
One potentially nice way of test composition is scuppered in JUnit, and most clones, because test methods aren't allowed to return anything. I think I remember discussing this with you a long time ago, and you saying that there was no technical reason that test methods couldn't return something, but JUnit 1 looked for methods that began with test and returned void, and everything since has just followed suit.
I often copy paste tests like this, but mostly for readability.
The cloned assertion in the second test still could make sense for someone reading the test: they know what the preconditions for the next part of the test is.
But looking at this I wonder if it makes sense to have named "parts" to a stateful test.
This might look hard in Java, but I'll use Common Lisp because it's easier to prototype such ideas in CL:
```
(deftest test1 ()
(... object := new Whatever() ..)
(... actual := object.doSomething() ...)
(... assertEquals(expected, actual) ...)
(deftest test2()
(... actual := object.nowSomethingElse() ...)
(... assertEquals(expected, actual) ...))
(deftest test3 ()
..))
```
Effectively, we now have a tree of test states. The test tooling can run each path independently (test1, test1->test2, test1->test3). Perhaps ignore assertions in the previous path.
Does this make things more complex to read? Is this a common enough pattern to abstract away like this? I don't know.
i usually start my tests with a testHappyPath that can be constantly enhanced and then each test will test an anomaly from the happy path
n Go, for example, when you apply composition, you still have to check whether err is nil (or not nil). Look at this code (it's updated to look like Go code):
test1()
object := new(Whatever)
actual, err := object.doSomething()
assertsNoError(err)
assertEquals(expected, actual)
Most methods in Go return an err. You can ignore it, but such code may be misleading for other engineers who might copy/paste it without realizing that it returns an error.
So, composition will look like this, applying your approach. And it's not exactly how you'd want it to look aesthetically:
test2()
object := new(Whatever)
_, err := object.doSomething()
assertsNoError(err)
actual := object.nowSomethingElse()
assertEquals(expected, actual)
I would prefer to give up on isolation and in Go do this (composition without isolation)
func TestSomething(t *testing.T) {
object := new(Whatever)
actual, err := object.doSomething()
assertsNoError(err)
assertEquals(expected, actual)
t.Run("test2", func(t *testing.T) {
actual := object.nowSomethingElse()
assertEquals(expected2, actual)
})
}
This gives us:
* test2 always requires test1 logic to pass, but without code duplication
* test 1 can be run alone
I'm not fluent in Go, but the particular issue you're highlighting seems easily fixed by extracting a helper test function so that the code would look like:
```
test2()
object := new(Whatever)
_ := assertSuccessful(object.doSomething())
actual := assertSuccessful(object.nowSomethingElse())
assertEquals(expected, actual)
```
Since Go now has generics, that should be possible, no?
I prefer being explicit here and use helpers only if I have to set up the same things in many tests or if the helper checks the same things. But from my experience, helpers make tests hard to maintain and rework (a tradeoff with duplication).