Even if you are self-studying, you can and should occasionally go talk to someone who knows the subject well to help you with the question. If you don't know anyone, post the question online.
Seems like the spirit of this article is "don't debate test coverage, code design, etc. and just get the job done because meeting requirements is all that matters" and that we should focus on making code shorter. No evidence is offered that shorter code is somehow better than code that adheres to "best practices" or has higher test coverage. Has anyone here fit an entire enterprise software system into half a page, thereby obviating the need for unit tests and clean, understandable code?
This feels like someone that just wants to throw whatever code they want over the wall without being bothered to design it well, write tests for it, and adhere to a team's shared design guidelines. I can't stand working with people like this. Just my two cents.
EDIT: I see that this isn't the primary point of the article, and that maybe the author means we could rid ourselves of so much debate by creating more powerful tools and working at a higher level of abstraction. I think that some of these same things are being attempted, though, when someone suggests a design pattern or something within existing code that could raise the level of abstraction, and that it's worthwhile to have that debate rather than label code design best practices as trivialities.
I think it depends on the problem set and where one works, etc. There's a lot of use cases where getting it over the wall is perfectly acceptable and what the business needs. I can cite numerous incidents where a lot of code was design, written, and then out of circulation less than a couple years later. I also know of instances where some crappy code stuck and is in use 14 years later. The hard part is knowing which is what, and some of this can be alleviated by a decent modular design and taking on tech debt to refactoring/swapping out certain implementations for more robust implementations at a future date. It's hard to balance delivery time frames with robust/clean code, especially when so many opinions drive the later. Hopefully you get to work in a place that favors the slower, better planned approach approach.
> and some of this can be alleviated by a decent modular design and taking on tech debt to refactoring/swapping out certain implementations for more robust implementations at a future date
Absolutely, but even that idea of "modular design" would be something that needed to be discussed by the team - but this article would label this a "fixation". This distinction really isn't helpful. What IS helpful is knowing how much time to spend on such a debate, and when to set that stuff aside in service of moving on with the work.
> where some crappy code stuck and is in use 14 years later
I think this boils down to a roll of the dice. Its possible that some code is simple enough, or some engineer is smart enough, that tests are not really needed, but given the choice to not unit test would be made early, and its very hard to know the complexity of the solution at the beginning, you're really just rolling the dice with regards to whether it would work out. Sometimes you get a huge payout by delivering good code fast, but sometimes it can really blow up in your face with lots of prod errors.
Personally, I'd rather smooth out that large payout vs large explosion curve by just writing tests. If we could have pushed out the code faster, that is a loss, but if the code would have had many errors, thats a gain. So probability distribution with regards to payoffs is smoothed out.
"don't debate test coverage, code design, etc. and just get the job done because meeting requirements is all that matters"
I would say:
"Don't debate test coverage, code design, etc., unless they are relevant to meeting the customer's requirements."
Now, many times, these things are very relevant to meeting the requirements! Without tests, the odds of your code actually meeting the requirements is very low. Instead of finding and fixing problems with the code before it gets to customers, you get to fix it after you give it to customers and it fails, and you've probably just lost those customers.
To the "code design" point, I think that is covered in the "Models" section. Modeling the problem correctly leads very straightforwardly to getting the right design.
The problem is the only constant is change. Customer requirements are rarely static. So it's incredibly important to put thought into designing code that's also flexible and maintainable. Because what will happen is customer will inevitably thought of feature X or find bug Y and if they find out it takes a week to fix it, they won't be happy.
But I get what you say about doing those other things to make sure you don't run into the problem of changing requirements.
Part of the problem though is the varied skill level of developers and adequately timing the completion of these tasks. Most companies never budget appropriately for design and just expect banged out code. And there's not enough push back from developers on this, therefore it's become an expectation.
Perhaps I shouldn't have it's the "spirit" of the article, since a larger goal for software engineering is defined with point 5 - regardless, the point is made that, even in the tools we work in today, these things are trivialities and are always a waste of time.
Not sure if this is what segment is doing, but if it’s a worker based on the Kafka Streams library, the k/v state store can be backed by a changelog topic, and if you have a transaction between that topic and the topic that commits offsets, the scenario you described won’t occur. The worker will reprocess the event again, and it rocks won’t have it because it’ll only have the state that was persisted before the crash.
Can anyone who has tried doing this outside of university recommend some good benchmarks, or resources for setting up a personal lab testing environment for this? I want to give this a shot and didn't get to do it in university.
It would be amazing if this integrated with EKS to easily create remote k8s clusters for development. I couldn't tell from this press release if that's a possibility though.
When you set out to do this, had you studied docker's source code at all? Or did you just have a basic understanding of containers? Other than the link from OP, are there any resources you'd recommend to get one to the point where you have enough understanding of the concepts without having to "cheat" and look at the docker implementation?
I want to do this too, but it's not as much fun if you need to go to the source due to not understanding the fundamentals.
I started with a basic understanding about the parts involved, but not so much how they fit together. Most of the necessary information came from the lwn series of articles posted in another comment:
https://lwn.net/Articles/531114/
The actual namespace stuff was easy, the harder part was pivoting the root fs and figuring out all the things to mount. At some point I looked at the source for systemd-nspawn, but I forget exactly what for.
Trying to cover your state space with coarse-grained tests in database-backed applications can lead to slow tests suites that are laborious to set up/tear down, which is a barrier to fast build times in CI. This harms productivity both while writing tests and in deployment.
I've heard a lot of people say this but I don't find it to be true. Realistic and loosely coupled tests that require minimal maintenance improve my productivity much, much more than a fast build time does.
JD Rainsberger's proscriptions in particular sound like a recipe for creating a massive amount of test code, a huge maintenance headache and tests that are not at all realistic.
I'm not sure what you mean by "Realistic" tests. Perhaps you mean integrated tests that exercise, say, the entire stack from handling the HTTP request down to the database, if we're talking about a REST API.
If you don't care much about covering large portions of the state space and your tolerance for defects is high, then I'm sure your claim about improved productivity is true.
My tolerance for defects is low, which compels me to try and cover the state space more - happy paths, unhappy paths, etc. I've tried doing this with integrated tests, and found it to be very difficult to do in a timely manner. When I switched to using DI, coding to interfaces, and using mocks across architectural boundaries, my ability to cover the state space went up, and my time to deliver code with few defects went down.
I think this is because often the code I care about exercising is limited to a small section of the call stack. In an integrated test, I have to set up too much state and have the machine do too much irrelevant work just to verify this small portion of the code. With DI, interfaces, and mocks, I can isolate just this behavior much more quickly.
That leaves coupling and maintenance. This is a valid concern in the mockist style, but I haven't found it to be a limiting factor in maintaining my code. I think it comes down to testing the right things. If you're using mocks to verify a contract between two objects, and you want to change that contract, naturally your tests for the contract will break. In that case, I think it's fine to just delete them and write new tests that establish your contract. This kind of approach isn't right in all situations, but mockist style testing has helped my write some stable code that would have been a real pain to test due to external dependencies otherwise.
EDIT: That is not to say that your tolerance for defects is high, but rather that productivity is relative to your goals. Sometimes the time spent preventing defects and keeping maintainability high isn't worth it if the repercussions don't translate to dollars and cents.
* Where possible, use the real thing - that is, use an actual database in preference to, e.g. mock objects representing an object that you use to interact with a database.
* Where you need to use a mock (when the real thing is too expensive to use), make it a realistic mock.
>Perhaps you mean integrated tests that exercise, say, the entire stack from handling the HTTP request down to the database, if we're talking about a REST API.
Exactly, but if, say, there was some algorithmic code in there that could be exercised realistically without using the HTTP stack then the HTTP stack is not necessarily necessary.
>If you don't care much about covering large portions of the state space
Code coverage is entirely orthogonal to the type of test you are doing.
>your tolerance for defects is high, then I'm sure your claim about improved productivity is true.
You appear to be fairly confused about the distinction between test coverage and modes of testing.
>My tolerance for defects is low
If you put no emphasis on test realism then I suspect that your tolerance will naturally have to be quite high.
>I've tried doing this with integrated tests, and found it to be very difficult to do in a timely manner.
I found this too sometimes. It's a tooling issue. Integration test tooling is more expensive to build initially but it tends to be much more reusable than unit test tooling. SMTP client stubs may be quicker to build but relatively useless in future projects that use different libraries (sometimes even different versions of the same library) whereas a mock SMTP server is useful forever.
>In an integrated test, I have to set up too much state and have the machine do too much irrelevant work just to verify this small portion of the code.
I have no problem with my tests chewing up 10,000% more CPU than yours and taking an extra 5 minutes if they've got even a 2% higher chance of finding bugs. CPU time is cheap, my time is expensive, bugs are expensive. Easy trade off.
>That leaves coupling and maintenance. This is a valid concern in the mockist style, but I haven't found it to be a limiting factor in maintaining my code.
Still means you're writing and maintaining more code. That means more bugs and more work.
> think it comes down to testing the right things. If you're using mocks to verify a contract between two objects, and you want to change that contract, naturally your tests for the contract will break.
Yup, and since changing the contracts between different subsystems is the most critical and important part of refactoring code, you've just tossed out a large chunk of the mainr benefit of having tests - safe refactoring. This is the worst aspect of unit testing IMO - it cements technical debt because refactoring changes contracts that will turn the tests red.
> Code coverage is entirely orthogonal to the type of test you are doing.
I did not say they were equivalent. Rather, I instead made a connection to the effort required to cover the state space in a given mode of testing. I've found it takes more integration tests to cover the state space than it does with unit tests, given the multiplicative nature of code paths in each layer. J.B. explains this in his talk, and I buy his argument, since I've experienced it myself.
For example, you could collapse several different types of failure at a lower layer into one code path in a higher layer using an exception, and verify that the HTTP layer returns the same thing when that exception happens. In this situation, you avoid having to set up the state necessary to cause the exception that the errors have been collapsed to, which may be 3 or 4 layers deeper. This means less test code, and less effort on my part.
> You appear to be fairly confused about the distinction between test coverage and modes of testing.
Nope, as explained above, the effort required to cover the state space is directly influenced by the mode of testing. For example, if you tested via typing on the keyboard with a pencil, you'd have to expend more effort to cover your state space. Of course, integration isn't as bad as pencil-testing, but the point is that one is easier than the other with respect to state space coverage.
Regarding reusable tooling - I've found this true as well. But tooling doesn't solve the whole problem. My main point is the amount of set up necessary to trigger a certain behavior, which tends to be too time consuming for me to justify the time. You can use tooling here as well, perhaps with data builders and object mothers. But that is also too much setup effort for my tastes.
> I have no problem with my tests chewing up 10,000% more CPU than yours and taking an extra 5 minutes if they've got even a 2% higher chance of finding bugs. CPU time is cheap, my time is expensive, bugs are expensive. Easy trade off.
Here you seem to assert that integration tests are more likely to find bugs. This is highly dependent on your skill in designing testable code that can be exercised without irrelevant portions of the stack. However, you do have a point that integration tests can find classes of bugs that unit tests may not (environmental setup, connection issues, etc.). I don't unit test those things of course.
Regarding CPU and the 5 minute difference - try 25 minutes difference. I have seen this in practice, and at that scale it really slows down the team. You can ease the pain with more machines, but I'd rather have focused, fast tests that can run right on my machine in as short a time as possible. I value the fast feedback loop.
> Still means you're writing and maintaining more code. That means more bugs and more work.
I haven't found this to be a problem. Sometimes I view messages between objects as behavior, so if the tests are inspecting that then I would actually want them to break if I change messages between objects. I've often found that if this is a hindrance to refactoring, there are problems in the code itself, not just the tests.
>Yup, and since changing the contracts between different subsystems is the most critical and important part of refactoring code...
I don't think that's true. If I'm changing contracts so often that it's a burden to maintain, I've likely missed an abstraction opportunity. Instead of blaming unit tests, I just fix the problem. Further, refactoring is not equivalent to changing subsystems. It encompasses many more useful activities that improve your code for the better. And further, if you take the stance that object interactions are behavior (sometimes, when useful), then changing that is not a refactor, but a re_work_. My favorite article on refactor vs. rework: http://www.daedtech.com/rewrite-or-refactor/
>I did not say they were equivalent. Rather, I instead made a connection to the effort required to cover the state space in a given mode of testing. I've found it takes more integration tests to cover the state space than it does with unit tests, given the multiplicative nature of code paths in each layer.
This is more about the level of testing than their nature. You can have high level unit tests (not common, but you can) and you can also have low level integration tests (I have many).
I agree that the "combinatorial explosion" Rainsberger talked about is real, sort of (not to integration test, but to automated end to end tests) but it comes with many caveats:
* If your application is, by and large, gluing together a lot of different libraries that are tested independently of your application then you effectively have a test pyramid even if you only write tests for the top layer because the tests for layers beneath are all written by somebody else. i.e. just because you wrote any kind of app on top of the linux kernel means you somewhat are relying upon the kernel being well tested, but that doesn't mean you're the one doing it directly.
* Those components in the 'application pyramid' you are stringing together require testing independently but IMO it's still better to use integration testing all the way down because all of those components are integrating with something else.
* I've found great success mitigating the effects of the combinatorial explosion using randomized property based testing. Even with slow tests you can get very, very far with this.
* The combinatorial explosion can also be mitigated with stronger typing and more frequent and stringent sanity checking which shuts down invalid code paths.
>For example, you could collapse several different types of failure at a lower layer into one code path in a higher layer using an exception, and verify that the HTTP layer returns the same thing when that exception happens. In this situation, you avoid having to set up the state necessary to cause the exception that the errors have been collapsed to, which may be 3 or 4 layers deeper. This means less test code
I agree with this but I still see it as being orthogonal to the kind of test you use to test each layer.
>Nope, as explained above, the effort required to cover the state space is directly influenced by the mode of testing. For example, if you tested via typing on the keyboard with a pencil, you'd have to expend more effort to cover your state space.
>Of course, integration isn't as bad as pencil-testing, but the point is that one is easier than the other with respect to state space coverage.
I personally found that the up front investment for integration tests is usually higher to put the initial infrastructure all in place - but, once you have gotten past an inflexion point, integration tests are typically easier to write.
For unit tests the investment is roughly constant.
>Here you seem to assert that integration tests are more likely to find bugs. This is highly dependent on your skill in designing testable code that can be exercised without irrelevant portions of the stack
Not really. They find more bugs because they test more realistically. When you are testing against a fake model of a database with fake data that you built for a unit test you are going to inevitably miss stuff that you will pick up on if you use a real database with real data. That's just life.
I do think that the less your code is about linking together different pieces of code and systems and the more it is about 'pure algorithmic calculation', the less you have to face this problem. However, with most code in real life the hard part is typically about integrating things together - most business problems aren't facebook's spam filter.
>However, you do have a point that integration tests can find classes of bugs that unit tests may not (environmental setup, connection issues, etc.). I don't unit test those things of course.
Right. I actually track and classify bugs that I see day to day and these are typically way more common than logical bugs. Moreover, it's not like integration tests don't catch logical bugs as well. They do.
>Regarding CPU and the 5 minute difference - try 25 minutes difference. I have seen this in practice, and at that scale it really slows down the team. You can ease the pain with more machines, but I'd rather have focused, fast tests that can run right on my machine in as short a time as possible. I value the fast feedback loop.
I do as well, but I find that until individual tests take > 30 seconds, or regression test suite breakages become very common, the cumulative effect on my productivity is very minor.
If an entire regression test suite takes an hour it doesn't bother me in the slightest. Hell, I used to have a 32 hour regression test suite once (long story) and even that didn't bother me that much - a release schedule of once a day with a small (~10%) chance of skipping a day when there were failures was more than acceptable for the business.
If that project had unit tests they would have caked themselves around a bunch of really horrible API contracts between module layers and they would have required a rewrite pretty much every time we rewrote anything. Faster to run, yes, but vastly more work to maintain.
>I don't think that's true. If I'm changing contracts so often that it's a burden to maintain, I've likely missed an abstraction opportunity. Instead of blaming unit tests, I just fix the problem.
IME it likely means somebody who came before you missed an opportunity. Since I largely work on projects that existed years before I joined them did I don't see the solution of "just fix the API" as viable. One fix leads to another which leads to another and before you know it it's been weeks or months and you haven't delivered anything. Refactoring needs to be staggered.
It's better to start out with the presumption that you will benefit more by not locking down your API contracts more than you need to conserve CPU horsepower.
You say "do what makes you successful" after "I have never let my teams go full TDD".
If someone on your team is most successful with TDD, do you still not allow it?
re: writing 30% less code - I've found TDD can reduce my percentage of lines of code, as you suggested. Adherence to the "refactoring" part encourages that you reduce duplication, which in my experience has been easier to do with good test coverage.
You are correct. My use of the singular "you" was more directed toward people in their own projects. For the purposes of a team at a company, you can think of it as a collective "us". We do not use TDD.
I would say that the most successful teams i have been a part of focus not on automated testing but instead on other collective practices: informal code reviews, diff analysis of every commit, group discussion of database changes and collective manual testing of other's code. Many people point to the refactoring (or initial code organization) as a benefit of TDD. I find these other practices tend to inspire a more collective ownership of the system. Additionally, and more importantly, they spur a lot of conversation around how and why to organize code certain ways. These learning opportunities are probably the most valuable among young and growing teams.
