Darn. I never heard of your "JUnit Max" product. I think I would have bought it at the time.
I've since moved to IntelliJ IDEA. And (mostly) from Java to the "Gosu" language.
There is an expectation in the industry that such tools should be free and open source. Driven by passion and such. But that's not a reasonable reason for others to expect that of you.
We often do live (work) in hostile corporate environments, unfortunately.
I donated time to vbUnit 3, for "Visual Basic Classic" (v3-6), back in the day. The hostility of my "superious" was ... educational. They couldn't live with it as open source. Or at a very cheap price. Nor were they entirely happy when they made me rewrite it so they'd own it. Nor could they abide by abandoning the tests on my departure, due to their refusal to allow the test runner tool. "Insanity." Not rational at all.
For Gosu, I wrote my own GUnit runner, as the vendor tool was impractically slow. Released it "open source." Most popular tool ever, in their library. My motivation? (1) I, personally needed it. And (2) hoping to embarrass them into improving their tooling. Well over a decade later, they were (kinda, sorta, maybe mostly) able to improve their tooling. And banned the use of my open source free tool.
Great story Jeff. This is classic, having experienced something similar at a large US bank with an effective home grown tool. I think that, somewhat ironically, if your test runner had been backed by a business that sold, maintained and promoted the tool, the situation would have been quite different. If the tool was expensive, had a 3rd party that owned the IP, and sold those business value drivers to your superiors, they likely would have paid for it. If GUnit was a product, Intellij would have taken it more seriously, and either improved their product or simply purchased the company.
I develop with quarkus, which brings its own continuous testing feature. Usually, firing up continuous testing is the first thing I do, before I even begin to look at the code.
So, imagine my surprise when I found out I am the only one in my team who regularly uses it. Everyone else prefers to run single tests or test classes from the IDE.
I like the idea of prioritization and maybe will try to steal it somehow :)
One more thing I like about this is it automatically runs tests, so you don’t have to click buttons. As an example, I have some scripts which monitor file system changes and if some detected, runs the tests in shell for me. So, on every file save I have my test results.
It saves much time as well, because when I switch the focus from code to tests, I nearly have the results ready. Having the tests rests right in IDE and inside of code might be one improvement on focus.
* Optimizing the metric “mean time to first failure”. If a test is going to fail, how quickly can you report it?
* Optimizing task switching cost. That’s why I reported errors right in the IDE, so you didn’t have to move you hands or eyes. The project would just turn red (or not).
It is very interesting optimizations. I am looking at this and trying to understand alternatives:
* We can invest into the tool to collect statistics and last modified tests to optimize time to failure.
* WE can invest time into making the tests fast. Either by introducing some boundaries or maybe improving the infrastructure.
In my specific case, I worked on Chrome, I had only some degree of freedom and it took a while to isolate the set of tests related to my project. At that point, the tests ran so fast that I didn't even think of prioritizing them. The next issues was compilation (linking more specifically), because in C++ world linking takes quite a while.
But, the test execution was pretty quick. Of course, if we are talking about use-case where the project is small and we didn't face scalability issues, then we need to prioritize different things.
Hmmm... I don't use eclipse but if it ran in the background like a service and always popped up errors quickly I would pay $100 for that and it would be platform independent. I mean the rest of what you are talking about it great but fast feedback alone is so valuable!
Hi Kent. Back in 2002 I was hired as VP to turn around a 20 person software startup. Friends had recently sold their company to thoughtworks, and I visited their office, where they demonstrated cruisecontrol.net v0.3, their internally developed CI server that wasn't yet publicly available. I was stunned by what it meant in terms of time savings and understanding the state of the product at any time. I went back to the office and calculated the time running CI we would save in QA and developer hours. I hired thoughtworks to coach/teach TDD and CI to the team and also install that early version of cc.net on a newly purchased high end PC. I think we spent about $20k including the hardware. The return on investment in time savings for developers and testers alone from CI and cc.net was 1 month. The ROI on TDD training and coaching was more complicated to determine, since it pointed out architectural flaws that needed remediation just to enable TDD. TDD did point out the direction we needed to pursue going forward.
At the end of the day I think most people who are responsible for making investments in tools are looking to save time, make something easier, less error prone, provide more certainty and improve quality. Vendors need to cost justify their tool in those dimensions. If those tools are open source, then for many *tool investors* this often isn't seen in a positive light. Open source often means the business model is altruistic vs economic. If I am a leader in a large organization, choosing an open source tool requires me to justify that choice to my peers. That investor needs to explain, often to folks with no understanding of software development, why choosing an open source tool that has no protected IP, is made by known or unknown persons who receive no monetary benefit and may stop working on it tomorrow, is a good idea. The usual response is, "Ok, but can we get that same functionality from <insert preferably large vendor name here> because then we know the tool will be supported." The reason I chose to spend money on TW and cc.net was 1. the value proposition I recognized and 2. that TW was investing in and standing behind cc.net.
I remember discussing this with you when I was at Dr. Dobb's. I was disappointed when you decided not to move ahead with commercializing it.
So I'm glad to see this post. My only suggestion this time around is to consider making a version that is a plugin to JetBrains' IntelliJ IDE. Most professional Java developers (the folks you need for this to succeed) use that IDE. Every Java tools survey in the last few ye4ars show IntelliJ having a commanding lead over the other IDEs and editors. So, IMHO, this should be the first group you target.
I’m not bringing it back, sorry to say. I’d love to see it out there but I’m not going to drive it. If IntelliJ wants me to explain it to them I’d be glad to.
The IT world has always had a very strange relationship with tools, tool vendors, and open source.
Back in the '90s, I worked for an automated QA company: we sold bespoke coding guidelines and an automated tool to enforce them and calculate software metrics. It was $5k/seat (with an X11 UI). We also sold a workflow automation system that sat on top of your source control and provided a consistent, team-wide check-in/check-out with automated coding guideline enforcement, and could be configured to make every check-in "improve" the software (per the metrics and coding guidelines). That cost $20k/seat as I recall and had a "fancy" Motif UI! We sold a bunch.
Fast forward to the aughts and that "tools should be free" mentality was starting to rise. By the tens, the cost of tools had been forced down to minimal, donation level prices -- except for "enterprise" stuff.
And now we're starting to see a value for tools. We starting to see reasonable subscription prices -- which means a guaranteed monthly income -- and we're seeing a huge rise in OSS sponsorship. I know a few OSS devs who are making a living from GitHub sponsorship alone, and folks who've had grants from OSS foundations that have allowed them to do nothing but work on their projects for up to a year.
I pay month sponsorships to three tools I use all day, every day in my work and my OSS projects -- and I get just about enough GitHub sponsorship to cover my car payment each month.
I bet if you were launching JUnit Max today, it would be a different story...
Darn. I never heard of your "JUnit Max" product. I think I would have bought it at the time.
I've since moved to IntelliJ IDEA. And (mostly) from Java to the "Gosu" language.
There is an expectation in the industry that such tools should be free and open source. Driven by passion and such. But that's not a reasonable reason for others to expect that of you.
We often do live (work) in hostile corporate environments, unfortunately.
I donated time to vbUnit 3, for "Visual Basic Classic" (v3-6), back in the day. The hostility of my "superious" was ... educational. They couldn't live with it as open source. Or at a very cheap price. Nor were they entirely happy when they made me rewrite it so they'd own it. Nor could they abide by abandoning the tests on my departure, due to their refusal to allow the test runner tool. "Insanity." Not rational at all.
For Gosu, I wrote my own GUnit runner, as the vendor tool was impractically slow. Released it "open source." Most popular tool ever, in their library. My motivation? (1) I, personally needed it. And (2) hoping to embarrass them into improving their tooling. Well over a decade later, they were (kinda, sorta, maybe mostly) able to improve their tooling. And banned the use of my open source free tool.
:-[
Great story Jeff. This is classic, having experienced something similar at a large US bank with an effective home grown tool. I think that, somewhat ironically, if your test runner had been backed by a business that sold, maintained and promoted the tool, the situation would have been quite different. If the tool was expensive, had a 3rd party that owned the IP, and sold those business value drivers to your superiors, they likely would have paid for it. If GUnit was a product, Intellij would have taken it more seriously, and either improved their product or simply purchased the company.
I develop with quarkus, which brings its own continuous testing feature. Usually, firing up continuous testing is the first thing I do, before I even begin to look at the code.
So, imagine my surprise when I found out I am the only one in my team who regularly uses it. Everyone else prefers to run single tests or test classes from the IDE.
I don't understand that. The more feedback the sooner the better for me.
I like the idea of prioritization and maybe will try to steal it somehow :)
One more thing I like about this is it automatically runs tests, so you don’t have to click buttons. As an example, I have some scripts which monitor file system changes and if some detected, runs the tests in shell for me. So, on every file save I have my test results.
It saves much time as well, because when I switch the focus from code to tests, I nearly have the results ready. Having the tests rests right in IDE and inside of code might be one improvement on focus.
I red to think more about :)
I thought about 2 things:
* Optimizing the metric “mean time to first failure”. If a test is going to fail, how quickly can you report it?
* Optimizing task switching cost. That’s why I reported errors right in the IDE, so you didn’t have to move you hands or eyes. The project would just turn red (or not).
It is very interesting optimizations. I am looking at this and trying to understand alternatives:
* We can invest into the tool to collect statistics and last modified tests to optimize time to failure.
* WE can invest time into making the tests fast. Either by introducing some boundaries or maybe improving the infrastructure.
In my specific case, I worked on Chrome, I had only some degree of freedom and it took a while to isolate the set of tests related to my project. At that point, the tests ran so fast that I didn't even think of prioritizing them. The next issues was compilation (linking more specifically), because in C++ world linking takes quite a while.
But, the test execution was pretty quick. Of course, if we are talking about use-case where the project is small and we didn't face scalability issues, then we need to prioritize different things.
Hmmm... I don't use eclipse but if it ran in the background like a service and always popped up errors quickly I would pay $100 for that and it would be platform independent. I mean the rest of what you are talking about it great but fast feedback alone is so valuable!
I use WallabyJS all the time for the exact same metrics you mention, and I gladly pay for it for my whole team. It's a no brainer in terms of ROI.
Creating a tool with positive ROI is step 1. Turns out there are a bunch of other steps. Who knew 🤷
Hi Kent. Back in 2002 I was hired as VP to turn around a 20 person software startup. Friends had recently sold their company to thoughtworks, and I visited their office, where they demonstrated cruisecontrol.net v0.3, their internally developed CI server that wasn't yet publicly available. I was stunned by what it meant in terms of time savings and understanding the state of the product at any time. I went back to the office and calculated the time running CI we would save in QA and developer hours. I hired thoughtworks to coach/teach TDD and CI to the team and also install that early version of cc.net on a newly purchased high end PC. I think we spent about $20k including the hardware. The return on investment in time savings for developers and testers alone from CI and cc.net was 1 month. The ROI on TDD training and coaching was more complicated to determine, since it pointed out architectural flaws that needed remediation just to enable TDD. TDD did point out the direction we needed to pursue going forward.
At the end of the day I think most people who are responsible for making investments in tools are looking to save time, make something easier, less error prone, provide more certainty and improve quality. Vendors need to cost justify their tool in those dimensions. If those tools are open source, then for many *tool investors* this often isn't seen in a positive light. Open source often means the business model is altruistic vs economic. If I am a leader in a large organization, choosing an open source tool requires me to justify that choice to my peers. That investor needs to explain, often to folks with no understanding of software development, why choosing an open source tool that has no protected IP, is made by known or unknown persons who receive no monetary benefit and may stop working on it tomorrow, is a good idea. The usual response is, "Ok, but can we get that same functionality from <insert preferably large vendor name here> because then we know the tool will be supported." The reason I chose to spend money on TW and cc.net was 1. the value proposition I recognized and 2. that TW was investing in and standing behind cc.net.
I think the motivations for buying a tool are more complicated than that, but hey, I've demonstrated my lack of skill at selling.
Certainly the "all tools should be free" ethos created big pluses & big minuses (tools not written).
I remember discussing this with you when I was at Dr. Dobb's. I was disappointed when you decided not to move ahead with commercializing it.
So I'm glad to see this post. My only suggestion this time around is to consider making a version that is a plugin to JetBrains' IntelliJ IDE. Most professional Java developers (the folks you need for this to succeed) use that IDE. Every Java tools survey in the last few ye4ars show IntelliJ having a commanding lead over the other IDEs and editors. So, IMHO, this should be the first group you target.
Good luck!
I’m not bringing it back, sorry to say. I’d love to see it out there but I’m not going to drive it. If IntelliJ wants me to explain it to them I’d be glad to.
The IT world has always had a very strange relationship with tools, tool vendors, and open source.
Back in the '90s, I worked for an automated QA company: we sold bespoke coding guidelines and an automated tool to enforce them and calculate software metrics. It was $5k/seat (with an X11 UI). We also sold a workflow automation system that sat on top of your source control and provided a consistent, team-wide check-in/check-out with automated coding guideline enforcement, and could be configured to make every check-in "improve" the software (per the metrics and coding guidelines). That cost $20k/seat as I recall and had a "fancy" Motif UI! We sold a bunch.
Fast forward to the aughts and that "tools should be free" mentality was starting to rise. By the tens, the cost of tools had been forced down to minimal, donation level prices -- except for "enterprise" stuff.
And now we're starting to see a value for tools. We starting to see reasonable subscription prices -- which means a guaranteed monthly income -- and we're seeing a huge rise in OSS sponsorship. I know a few OSS devs who are making a living from GitHub sponsorship alone, and folks who've had grants from OSS foundations that have allowed them to do nothing but work on their projects for up to a year.
I pay month sponsorships to three tools I use all day, every day in my work and my OSS projects -- and I get just about enough GitHub sponsorship to cover my car payment each month.
I bet if you were launching JUnit Max today, it would be a different story...