Order of tests: I recently did again the 'wardrobe kata' (https://kata-log.rocks/configure-wardrobe-kata) and there I found that there are two dimensions one can go in TDD-ing it: number of elements needed to fill the wall and number of different elements available. Both lead to completely different implementations for me.
Agree. I usually find that every problem has multiple "dimensions" that creates a "vector space". Each of those dimensions have their own ZOM(BIE) set of points and I try to cover a select number of such points on each of the vectors. Usually selecting the next test that is just one step in a single dimension at the time. This makes me only solve small steps at a time.
Yes, absolutely. I usually try to explore one dimension as long as there is a need in Katas - and in the real thing, I limit myself to two or three steps.
And if I have more than two dimensions, I will try to fix one dimension and extract it towards another area of the design (reducing spaces to planes, which are inherently easier to control). If that is not possible, I want a QE next to me to help me figure out test cases 🙈
It's especially great if you can a) find the one dimension that is "just a loop" or b) have completely distinct solutions for different points in that dimension.
I find the discussion alone can generate deep insight.
I teach and practice TDD. I’d not been consistent about using a list of test scenarios. After having read this article, I’ve changed how I do TDD.
I did a TDD of Fibonacci demo last week in front of a class. I think starting with a thought like “How many tests do you think we’re going to write and why” was a noticeable improvement especially for a problem like that where the input can be so easily partitioned.
No actual criticism of the post, just noticed this:
> surprising experience is that folks out there don’t agree on the definition of TDD. I made it as clear as possible in my book. I thought it was clear. Nope. My bad.
Sadly, just because you came up with a term and defined it with utmost care, means nothing if the term becomes popular. People take it and run with it, rewriting the definition in their head based on vague understanding (or even more often, based on the name alone).
Love the clarity, Kent! Your breakdown of the TDD workflow is gold. Especially appreciate the emphasis on a clear test list—so often overlooked. 👏 Also, Vic Wu's flowchart adds a visual punch to your wisdom!
About this: "Mistake: copying actual, computed values & pasting them into the expected values of the test. That defeats double checking, which creates much of the validation value of TDD.".
I couldn't understand. Is this about pasting value in the code of the test, or into the code of the system under test? If it is the former, I fail to see the problem.
And think hmmm, what is the expected value? Oh, never mind, I'll just copy the value, so I finish:
assertEquals(complicatedFunction(), null)
Then I run it & get an error message:
null does not equal 14.75
Then I go back and write:
assertEquals(complicatedFunction(), 14.75)
I claim this is a mistake, or at least a missed opportunity. The programmer has missed out on the chance to think through the computation in two different ways. Double checking reduces the chance of errors but it takes time.
Oh, ok. I agree with you. I assumed the tested code was not written yet, and I was calculating the value manually, in a calculator, spreadsheet, etc. and copying that to the test. Now what you explained makes total sense. Thanks!
I 100% had to correct someone on their idea of TDD this week.
"I'm almost done writing the unit tests" and "I'm doing TDD" were two sentences uttered in the same context.
I asked..."ah, so you're almost done with this story then?" But no. I suspected as much.
Gave the benefit of the doubt and exhoed your sentiment "if you're writing all the tests first, you're not doing TDD. You're doign something else...maybe you've found that it works for you and that's fine...but it's not TDD."
I care that the job gets done, that we have a good outcome, well organized code, and well written unit tests to support future improvements to the code. At least the tests weren't an afterthought.
I'd like to add an additional mistake at step 2. Mistake : computing asserted values using logic that will then be used to make the test pass. I prefer to hard code values where possible. E.g if order line 1 is $2 and order line 2 is $3 then expected order total is $5.
Thanks, as a technical coach, I do teach the idea of breaking requirements into possible tests.
In practice, I tend to create new tests based on the implications of the previous test.
I do feel like if the next test is "hard", as in "I can't see the simplest way to make it pass", and I can't see a refactoring to make it easier, then I usually look for an easier intermediate test, that will allow the algorithm to change enough to get the next test unblocked.
> Folks seem to have missed this step in the book.
It's me. Normally I just launches into coding.
It seems work for me to avoid over thinking at the beginning. I always have a chance to add new test cases to the list at step 3. I maintain the list in mind, some time a TODO comments.
Maybe write the list is helpful for others, I did see some guys miss the scenario after TDD through happy path. On the other hand people may think too much implementation details when write the test case list, and may code too much when pass the current test case since she cannot forget the cases in the list.
> The initial step in TDD, given a system & a desired change in behavior, is to list all the expected variants in the new behavior. “There’s the basic case & then what if this service times out & what if the key isn’t in the database yet &…”
> Folks seem to have missed this step in the book. “TDD just launches into coding 🚀. You’ll never know when you’re done.” Nope
Huh, I have the book and read it a long while ago, and yeah, this is certainly a step I've been skipping. I've certainly had decent results without it, but I trust you, so I'll give this a try.
I think for me, I personally get into day-dreaming mode. If I sit still I day-dream (sometimes it's about the billions of dollars I'll eventually have, sometimes it's about how I make this beautiful piece of software that everybody loves etc. etc.). So figuring out how to get to coding as quickly as possible gets me out of day dreaming mode.
(But I'm skimming through the book right now, and I don't see this mentioned. You do seem to be jumping straight into the tests. I'm guessing you're referring to the TDD by Example book?)
I’m not sure that I understand this. Are you just saying to not write the code first but just its invocation in the test assertion as the first step? This will ensure that you start with a failing test.
I usually write the code’s method as a stub with no implementation, then poke the “Jump to test” which creates a blank test case for me. But I can see if you’re tempted to start to fill in the implementation then the other is better.
It's more than that. When I'm writing a test, I can get stuck on all the interface design decisions. I may know that the answer should be 4, but not know how to express the inputs. So following the "known to unknown" principle, I write:
assertEquals(actual, expected); // actually I can write this before I even know the answer is 4
expected = 4
assertEquals(actual, expected)
Then figure out how to express the inputs. Does that help?
In Sarah's (my partner) work as a math tutor, she used this trick quite frequently with students (usually teenagers) who felt stuck. She asked them a simple question: "What do you (already) know how to do?"
For the programmer trying to articulate the next example, that might be "Well, I know that I need to check an answer, so assertEquals(expected, actual)." Programmers routinely underestimate the confidence or hope that can give themselves by writing something like that down, even when they expect to erase it in 5 seconds.
Yes. Another example of starting with the simplest thing that could possibly work (or in this case fail), then building from there in small measured steps.
Try to start with the test and poke the 'create method'-button. You'll see that writing the method/function call alone will give you lots of design-insight.
I also find slowing myself down and considering the list of behaviours the hardest part. I always itch to start writing code as soon as I've determined the first handful of behaviours. In my case, that is partly driven also by a concern that if I spend too long thinking about different behaviours I'm going to start doing a full design in my head or on paper before I start on a test. Maintaining the discipline of not trying to come up with a design to cover all behaviours up front is tricky for me.
The most common complaint I've received about TDD is that it leads to bad quality software because there's no attempt to construct well-designed code. The second is that it leads to poorly tested code - there's a belief that people who practice TDD don't bother to think about testing a wide range of behaviour or considering possible edge cases. In both cases it has invariably turned out that the naysayers are not talking about TDD, but about a subset that includes only one or two of the steps.
I tend to move easily between levels of abstraction, so I don’t distinguish. I think there is a difference between tests customers can read & those they can’t.
Yes, I've used snapshot tests & written a couple of snapshot test tools. My point is that *if* you can work out the expected value some other way than running the code, *then* you get the benefit of double checking. That's more valuable than just having a regression test that tells you when something changed unexpectedly. The regression test is still valuable. I write them too. But I don't want to give myself the opportunity to be lazy about double checking.
I draw a flowchart from the article. Hope you like it.
https://whimsical.com/cannon-tdd-M74C15bNBdVmxhkLztnSXa
"end" needs to go back to "start" as we will need to change system all the time :)
Good piece, thanks.
Order of tests: I recently did again the 'wardrobe kata' (https://kata-log.rocks/configure-wardrobe-kata) and there I found that there are two dimensions one can go in TDD-ing it: number of elements needed to fill the wall and number of different elements available. Both lead to completely different implementations for me.
There is also an uncle bob blog entry (in a weird comic style) where he explores order of tests and how it impacts code (once the resulting algorithm is bubble sort, once it's quick sort): https://blog.cleancoder.com/uncle-bob/2013/05/27/TransformationPriorityAndSorting.html
Very, very, interesting topic.
Agree. I usually find that every problem has multiple "dimensions" that creates a "vector space". Each of those dimensions have their own ZOM(BIE) set of points and I try to cover a select number of such points on each of the vectors. Usually selecting the next test that is just one step in a single dimension at the time. This makes me only solve small steps at a time.
Yes, absolutely. I usually try to explore one dimension as long as there is a need in Katas - and in the real thing, I limit myself to two or three steps.
And if I have more than two dimensions, I will try to fix one dimension and extract it towards another area of the design (reducing spaces to planes, which are inherently easier to control). If that is not possible, I want a QE next to me to help me figure out test cases 🙈
I haven't thought about the dimensions as triggers for a possible refactoring move, but I can certainly see that. Good point!
It's especially great if you can a) find the one dimension that is "just a loop" or b) have completely distinct solutions for different points in that dimension.
I find the discussion alone can generate deep insight.
Kent, this article is appreciated.
I teach and practice TDD. I’d not been consistent about using a list of test scenarios. After having read this article, I’ve changed how I do TDD.
I did a TDD of Fibonacci demo last week in front of a class. I think starting with a thought like “How many tests do you think we’re going to write and why” was a noticeable improvement especially for a problem like that where the input can be so easily partitioned.
I like knowing when I’m done. Alternatively I don’t like knowing that I’m not done. The list helps me.
Everything else is style/preference.
I found great peace when I began to believe this. I hope the people who learn with me have found some of that peace.
- We do some of these extra things because we're mostly learning the fundamentals.
- We do some of these extra things because we recognize patterns in our behavior that need particular attention.
- We do some of these extra things because we find the work more pleasant that way.
No actual criticism of the post, just noticed this:
> surprising experience is that folks out there don’t agree on the definition of TDD. I made it as clear as possible in my book. I thought it was clear. Nope. My bad.
Martin Fowler did a whole writeup on it: https://martinfowler.com/bliki/SemanticDiffusion.html
Sadly, just because you came up with a term and defined it with utmost care, means nothing if the term becomes popular. People take it and run with it, rewriting the definition in their head based on vague understanding (or even more often, based on the name alone).
I don't mind if the definition drifts. Part of being successful. What I mind is people saying, "<this other thing entirely> suckz".
Love the clarity, Kent! Your breakdown of the TDD workflow is gold. Especially appreciate the emphasis on a clear test list—so often overlooked. 👏 Also, Vic Wu's flowchart adds a visual punch to your wisdom!
About this: "Mistake: copying actual, computed values & pasting them into the expected values of the test. That defeats double checking, which creates much of the validation value of TDD.".
I couldn't understand. Is this about pasting value in the code of the test, or into the code of the system under test? If it is the former, I fail to see the problem.
I claim that the former is a mistake. I write:
assertEquals(complicatedFunction(),
And think hmmm, what is the expected value? Oh, never mind, I'll just copy the value, so I finish:
assertEquals(complicatedFunction(), null)
Then I run it & get an error message:
null does not equal 14.75
Then I go back and write:
assertEquals(complicatedFunction(), 14.75)
I claim this is a mistake, or at least a missed opportunity. The programmer has missed out on the chance to think through the computation in two different ways. Double checking reduces the chance of errors but it takes time.
Does that make sense?
Oh, ok. I agree with you. I assumed the tested code was not written yet, and I was calculating the value manually, in a calculator, spreadsheet, etc. and copying that to the test. Now what you explained makes total sense. Thanks!
I 100% had to correct someone on their idea of TDD this week.
"I'm almost done writing the unit tests" and "I'm doing TDD" were two sentences uttered in the same context.
I asked..."ah, so you're almost done with this story then?" But no. I suspected as much.
Gave the benefit of the doubt and exhoed your sentiment "if you're writing all the tests first, you're not doing TDD. You're doign something else...maybe you've found that it works for you and that's fine...but it's not TDD."
I care that the job gets done, that we have a good outcome, well organized code, and well written unit tests to support future improvements to the code. At least the tests weren't an afterthought.
Sounds like someone who takes their responsibility seriously. That's good in my book. Maybe they can optimize from there.
I'd like to add an additional mistake at step 2. Mistake : computing asserted values using logic that will then be used to make the test pass. I prefer to hard code values where possible. E.g if order line 1 is $2 and order line 2 is $3 then expected order total is $5.
Thanks, as a technical coach, I do teach the idea of breaking requirements into possible tests.
In practice, I tend to create new tests based on the implications of the previous test.
I do feel like if the next test is "hard", as in "I can't see the simplest way to make it pass", and I can't see a refactoring to make it easier, then I usually look for an easier intermediate test, that will allow the algorithm to change enough to get the next test unblocked.
> Folks seem to have missed this step in the book.
It's me. Normally I just launches into coding.
It seems work for me to avoid over thinking at the beginning. I always have a chance to add new test cases to the list at step 3. I maintain the list in mind, some time a TODO comments.
Maybe write the list is helpful for others, I did see some guys miss the scenario after TDD through happy path. On the other hand people may think too much implementation details when write the test case list, and may code too much when pass the current test case since she cannot forget the cases in the list.
> The initial step in TDD, given a system & a desired change in behavior, is to list all the expected variants in the new behavior. “There’s the basic case & then what if this service times out & what if the key isn’t in the database yet &…”
> Folks seem to have missed this step in the book. “TDD just launches into coding 🚀. You’ll never know when you’re done.” Nope
Huh, I have the book and read it a long while ago, and yeah, this is certainly a step I've been skipping. I've certainly had decent results without it, but I trust you, so I'll give this a try.
I think for me, I personally get into day-dreaming mode. If I sit still I day-dream (sometimes it's about the billions of dollars I'll eventually have, sometimes it's about how I make this beautiful piece of software that everybody loves etc. etc.). So figuring out how to get to coding as quickly as possible gets me out of day dreaming mode.
(But I'm skimming through the book right now, and I don't see this mentioned. You do seem to be jumping straight into the tests. I'm guessing you're referring to the TDD by Example book?)
"protip: trying working backwards from the assertions some time"
So TDD the test then 😉
Thanks for this, it's like a puzzle piece I was missing for inside my brain.
I’m not sure that I understand this. Are you just saying to not write the code first but just its invocation in the test assertion as the first step? This will ensure that you start with a failing test.
I usually write the code’s method as a stub with no implementation, then poke the “Jump to test” which creates a blank test case for me. But I can see if you’re tempted to start to fill in the implementation then the other is better.
It's more than that. When I'm writing a test, I can get stuck on all the interface design decisions. I may know that the answer should be 4, but not know how to express the inputs. So following the "known to unknown" principle, I write:
assertEquals(actual, expected); // actually I can write this before I even know the answer is 4
expected = 4
assertEquals(actual, expected)
Then figure out how to express the inputs. Does that help?
In Sarah's (my partner) work as a math tutor, she used this trick quite frequently with students (usually teenagers) who felt stuck. She asked them a simple question: "What do you (already) know how to do?"
For the programmer trying to articulate the next example, that might be "Well, I know that I need to check an answer, so assertEquals(expected, actual)." Programmers routinely underestimate the confidence or hope that can give themselves by writing something like that down, even when they expect to erase it in 5 seconds.
Yes. Another example of starting with the simplest thing that could possibly work (or in this case fail), then building from there in small measured steps.
That's a very good question I was wondering about too.
Try to start with the test and poke the 'create method'-button. You'll see that writing the method/function call alone will give you lots of design-insight.
I also find slowing myself down and considering the list of behaviours the hardest part. I always itch to start writing code as soon as I've determined the first handful of behaviours. In my case, that is partly driven also by a concern that if I spend too long thinking about different behaviours I'm going to start doing a full design in my head or on paper before I start on a test. Maintaining the discipline of not trying to come up with a design to cover all behaviours up front is tricky for me.
The most common complaint I've received about TDD is that it leads to bad quality software because there's no attempt to construct well-designed code. The second is that it leads to poorly tested code - there's a belief that people who practice TDD don't bother to think about testing a wide range of behaviour or considering possible edge cases. In both cases it has invariably turned out that the naysayers are not talking about TDD, but about a subset that includes only one or two of the steps.
In the early days of XP there was a distinction between Customer Tests (aka Acceptance Tests) vs Unit Tests. Do you still see that distinction @Kent?
I tend to move easily between levels of abstraction, so I don’t distinguish. I think there is a difference between tests customers can read & those they can’t.
TDD was a big breakthrough for me, but I disagree hard with this point:
> Mistake: copying actual, computed values & pasting them into the expected values of the test.
Snapshot tests are super useful! I'm curious if you've ever tried snapshot testing?
Yes, I've used snapshot tests & written a couple of snapshot test tools. My point is that *if* you can work out the expected value some other way than running the code, *then* you get the benefit of double checking. That's more valuable than just having a regression test that tells you when something changed unexpectedly. The regression test is still valuable. I write them too. But I don't want to give myself the opportunity to be lazy about double checking.