1) Developers cannot reliably estimate software timelines. That often comes with experience and age, but many never learn that skill. It is a skill like programming itself. It comes with practice.
2) Break down in communication between management (product managers) and engineers who are writing code.
On the first part. Usually junior software developers simply have not had experience shipping software in a full software cycle. From initial requirements, to design, to deployment, to maintenance. They did projects perhaps in school and homework exercises. Later some eventually learn that they chronically underestimate the time, and just start adding a factor (2x) to what their initial estimate is. That is a step up. But many never learn. There is also a race to the bottom -- the ones that promise shorter timelines get promoted and seen as smarter, more aggressive, better team players. The ones that are realistic are seen as maybe "taking it easy", "not working hard enough". Then everyone sees who gets promoted and show doesn't and learn -- "Aha I need to be like cool cowboy coder Joe over there, he is now running his own department". And so it spreads. Notice how there is often no feedback loop to punish the cool cowboy coder after the project is not delivered on time.
On the second part -- even if there are engineers who know how to estimate the time, nobody asks them. Because the person who promised the feature already played golf with the customer and told them they'll get the new shiny toy in 6 months. That person might have never been a developer, they might have an MBA or where a football coach or something like that. And they usually have a higher position in the company. So they promise and then everyone is pushed to deliver on that deadline. If it fails the failure is usually attributed to the engineers doing the work because they haven't worked "hard enough".
> 1) Developers cannot reliably estimate software timelines. That often comes with experience and age, but many never learn that skill. It is a skill like programming itself. It comes with practice.
This is true. But this is the problem that Agile is supposed to help solve -- with the caveat that most companies don't do true Agile. I guess people are now calling it Continuous Delivery, but the idea is the same: you don't tell people when a feature will be done, only when you will be working on it. But even then, even with experienced estimators, the rule of thumb is to add 15-30% to their estimates.
> 2) Break down in communication between management (product managers) and engineers who are writing code.
Totally agree that this is very common. Unfortunately, the best way to solve this is with process, and developers hate process. But developers often don't realize that they can use the process to their advantage: a strong engineering lead can refuse to accept poorly defined requirements. That allows you to push back on product to force them to define the requirement better.
IMO this breakdown is caused by just that: developers who say "oh yeah, I can do that", then start working on the problem and realize that there are a lot of undefined edge cases. If they promised it to the client in 6 months, fine, but define what "it" is.
From the perspective of an MBA, if we have to bring in a contractor to implement it, fine. I know where I can go to hire certified rockstar contract developers at $200/hr, which I will do if the client is lucrative enough. There is no feature I can think of that would take more than 6 months of development.
Process doesn't work, it can't work, because Process isn't the problem, it's top-down management infrastructure. Process is orthogonal to that.
In a top-down management style, your so-called "Lead Developer" is just the first head on the chopping block. They're leaders of puppet regimes. If the real leaders don't like them, they install a new one. It comes in many forms: layoffs, "performance" related dismissal ("you're not getting enough done!" because of all the rejected requirements docs), even reassignment to "re-engineering initiatives" that have no real chance of seeing the light of day.
Finding a management infrastructure that actually cares about feedback is incredibly rare. I have personally never found it, in 10 years across 6 companies. No amount of process is going to change a bad manager.
Any company of a certain size must convert to a top-down management infrastructure. I've seen what happens in large companies that don't -- you get 15 teams building different versions of the same product, and internal politics decide whose version becomes the 'official' one. But invariably, the internal user community settles on the one that's actually best for their use case, so you end up having to support 3 or 4 versions of every product you create. It's an absolute support and integration nightmare.
You talk about a business like it's supposed to be a democracy. It's not. You need clear direction, and democracy is messy and opaque if your company is too big for you to know the name of everyone who works there. Now the way you describe top down management is a bad implementation of it -- executives need to work to develop their middle managers -- but every company eventually has to convert from a democratic decision making process to a chain of command.
I've been working recently with a number of household-name startups that are having this exact issue. They've hit a plateau in their growth because the culture is dead-set against top-down management, but the company is now too big for people to self-organize. Engineering initiatives that would have taken 2 months in startup mode now take 2 years because they have to go back and rebuild half their systems to make them flexible enough to support the multiple use cases that they now have. Or because they were a startup, they didn't worry about geographic redundancy because that wasn't part of the MVP. So it was put off for later, and now they're a billion dollar company operating out of a single datacenter because their databases and applications were built without multi-datacenter being a requirement. Product can't even reliably tell engineering what features to implement because nobody on the product side has enough authority to shout over the crowd.
So having an infrastructure that cares about feedback is good, but the infrastructure also needs to empower people to do something about said feedback.
I know, I really understand what you're saying. And I am not saying a democracy is a better option. But I just don't think the things you're describing are attributable to top-down management. It certainly has never prevented the problems you've mentioned in any of the places I've ever worked. I agree that clear direction is needed, but your garden-variety middle manager isn't going to provide that. What he will usually provide is a sense that the failures of the organization are the fault of the leaf-nodes.
For every SV-brand startup, there are dozens of small-time consultoware shops serving their local market. They're the kind of places that have been around for 20 years and still only have 100 employees each. They can't ever seem to grow beyond that size because the one-way communication flow from sales to development keeps them in massive technical debt.
Middle managers don't provide direction; they are communication points. As to how effective they can be, it all depends on how the org is structured. If it's a product-centric organization (i.e. P&L rolls up under product and operating costs are charged back to the product org) then it can work pretty well. If it's a functional organization (revenue rolls up through a sales org on a regional basis and is combined with costs at the corporate level) then there's really no amount of management that will make it a good place for developers to work -- under the latter model, finance is running the company and you're a pure cost center. The manager's primary focus will be cost control, not product quality. That's a bad situation to work under.
I agree with your second assessment, but it's more of an issue with the consultoware business model than anything. Most consultoware shops start up thinking they're going to get hockey-stick growth, but it turns out every implementation is super-custom and it ends up as a consulting business. Many companies fail to realize this has happened, and end up operating like a software company when they should be operating like a professional services company with hourly billing. Under a professional services model, if a customer wants a feature, you can quote them a price for developing and implementing it. Tech debt gets dealt with because it becomes a bottleneck for revenue growth when you can't charge clients to add functionality.
Very quick reply because I have to get to bed (it's 1:30 am here!). I shouldn't work so hard... :-P
Estimation (i.e. pulling numbers out of your posterier) is not necessary (or usually desirable). What you want to do break down the problems into roughly equal size pieces -- the more the better. Each piece will take a certain amount of time. You can measure this and record the average. This becomes the estimated amount of time for each piece. Obviously the estimate has an error with some kind of distribution (my sleep adled brain is unable to remember what a likely distribution for the amount of time something takes).
It doesn't really matter because the sum of all the pieces will give you an estimate for the overall completion time for all the pieces. This estimate will have a normal distribution.
If you write down a list of all the pieces in priority order, you can draw a line at one piece of functionality and determine when it is likely to be completed, along with error bars.
Of course you are unlikely to actually to do the things on your list because if you were able to correctly guess all your requirements before you started the project then you would be in fantasy land. So what you do is tell the stake holders that this is what you are going to do unless they tell you that they want to change their mind.
Of course when you start they will say, "We're not going to change our mind. We're 100% certain that this is right". To which you reply, "Excellent". Every 2 weeks or so, once you have done some work and can show it to the stake holders, you should also have a list of new ideas. You should explain these new ideas and say, "Do you want to change the list to add these new ideas". Whether they say "Yes" or "No", you should reply with, "Excellent" and carry on.
Often the stake holders will have an "emergency" and decide that they have new opportunities which change your list. To this you should say, "Excellent" and change the list.
You should release whenever your stake holders are happy to release.
In this way, you have given the best estimate you can to help the stake holders plan. Any changes to the plan will be made by the stake holders themselves at the earliest possible time. You don't get caught up trying to deliver at any particular date because the stake holders are aware that they are changing the deliverables. You may have to ask them where they want to draw the line and give them new estimates for completion every time they change something, just to reinforce this idea.
The really tricky bit is breaking the work down into small enough pieces. This is very, very difficult and requires a lot of experience. The sucess or failure of the above will very likely rest on the ability to do this, so expect to pay a lot of money for someone with a proven track record doing it. It is often worth hiring a consultant if you have nobody on staff with this experience.
If your greater point is that some developers really are poor estimators, I agree, but all of them operate under pressure to err on the optimistic side.
I've spent a lot of time thinking about why software projects go over budget so often. I think that developer estimation only plays a minor role, and the big problem is failure to manage project scope.
I'm specifically talking about projects which go like this:
1) A set of high-level business requirements are put together, and given to a tech lead to estimate.
2) The tech lead's estimate, based on the requirements as-given, is used to determine a budget for the project, and probably a schedule as well.
3) Once the budget and schedule are approved, the project commences.
4) As development begins detailed software requirements are worked out, which usually exposes holes in the high-level business requirements that need to be fleshed out as well.
5) The estimates are never updated, and any attempt to increase the budget due to the new requirements are seen as a sign of failure.
What I see happening in #4 is two kinds of scope creep. The first kind is the sort everyone is familiar with: a requirement was missed, a new feature needs to be added, and since this wasn't in the original budget the budget needs to be increased. It's easy to identify these new/changed requirements, easy to see the big chunk of time needed to implement them, and easy to make prioritization decisions about the new requirement vs the other original requirements, so that maybe something can be dropped in order to avoid increasing the budget.
But there's a second kind of scope creep, and it's insidious. It comes from turning high-level requirements into detailed requirements into implemented and tested code. Every decision a developer needs to make about how to implement a feature is an implicit detailed software requirement, and you just can't know how many of those decisions will be needed or what their impacts will be until you get into the details. The time needed for these all add up, and I consider them to be scope creep because they aren't part of the high-level requirements used for the estimate and budget. We try to add padding for them because we know they're going to happen, but if you add sufficient padding across-the-board for all of the detailed requirement decisions that might be needed your project will never get approved. So we pad insufficiently, hoping that some of the requirements will be easier to implement than we estimate to cover the ones that aren't, and we don't manage this kind of scope creep at all. It's invisible, because it's just adding a few minutes here, a few minutes there, until our estimates have been blown away and we're over budget and we can't really say why.
I agree with you. I was mostly thinking of one iteration (a simplistic view) of "we know the scope, we go to work, we deliver (or fail) the result" which happens only in small projects.
The two types of scope creep is a good observation, I'll have to remember that idea.
Good points but there are also company wide reasons, namely self absorption among leadership. For example, fear. Fear of running out of money, fear of losing to competition, fear of tarnishing brand if we don't respond to this service issue right away, fear of insignificance, etc. impatience - we got a really promising thing and I want to fucking make it/make my mark/cash out! Greed - if we squeeze everyone a little harder I can make more money (even though I'm really well off).
These are all real human struggles. I have them for sure. I just wish you find a place with self aware leadership that endeavors to keeps these emotions in check. Usually startups with leaders who have families are a little better because their family essentially forces them to focus on someone else, at least a little.
I agree with what you said. I think there's also a third primary reason and maybe a fourth:
3) Developers are expensive. If you have a team of 5 developers each being paid a salary+benefits of $1,500-$3,000 per week and they tell you a feature will take four weeks, management does a little mental math and says holy shit, that's a lot of money and it's understandably tempting for them to want to see if that team really needs four weeks, or if maybe it can be done in three weeks. (Meanwhile, as you said, the feature will probably actually take six weeks because developers cannot reliably estimate...)
4) Developers have a hard time saying "no." We have a culture of working crazy hours. Management knows this and often intentionally or unintentionally exploits it.
Your point about the overly aggressive golf-made deal: how do we fix that? I really don't have an answer, but if we, as developers, are really so smart, we should be able formulate the problem and find a solution. We will have to avoid imagined constraints and thinking the answer is related to producing more code, but it's got to be there. Is it related to playing the office politics game better? And if so, can that be a learned and tolerable skill? Is it to do with organizing? With preferring self employment or contracting?
I am probably naive, but I think if we say: the situation is now X and I want to be Y, we can find a way to get there.
Company I work for tried to solve it using this rule -- don't hire any MBAs. Every manager, product manager, and even salesperson is (or at least was at some point) an engineer.
This works in our context (and has worked for many decades apparently). But I doubt this scales to a large size.
IIRC Palantir subscribes at least nominally to this philosophy, too. And they're trying to scale it, so we'll get to see how that works out in the near-ish future.
(Unless you work at Palantir, in which case I feel silly).
That sounds like a very silly policy. Would your mind be blown to learn out there exist many MBAs with engineering backgrounds, and that they're actually not idiots?
It's more appropriate to take a "no MBAs" rule as "no one who self identifies primarily as having an MBA". In my experience, an engineer with an MBA who neglects to even mention their engineering schooling/experience in favor of the MBA is a rare animal.
Self-identifies primarily as having an MBA? I don't even know what that means. If I interview with your company, should I not mention the MBA and instead say that during my 15 years of writing software, I took two years off to go wind surfing? Really? That's considered better?
I think the parent comment meant "self-identifies" as "your only qualification for this job is an MBA", not that you would hide having an MBA. If you wrote software for 15 years and applied for a job in tech, I would imagine the software experience was on your resume alongside the MBA -- you're not just an MBA who wants to get into tech because tech is hot but with no prior industry exposure, someone who could equally likely go be a VP of Paper Product Distribution at a Midwest conglomerate.
It is silly I agree but it worked for many decades, in the sense the customers are happy and engineers here seem to be happy. One can also say it didn't work because the company hasn't turned into Google or "existed" with a huge cash windfall for founders.
Of course that was just one variable and it wasn't isolated. There could have been other factors and reasons for the said result and it might not have had anything to do with "no MBAs" rules.
I just presented it as a reference or what others do as a "solution".
1) Developers cannot reliably estimate software timelines. That often comes with experience and age, but many never learn that skill. It is a skill like programming itself. It comes with practice.
2) Break down in communication between management (product managers) and engineers who are writing code.
On the first part. Usually junior software developers simply have not had experience shipping software in a full software cycle. From initial requirements, to design, to deployment, to maintenance. They did projects perhaps in school and homework exercises. Later some eventually learn that they chronically underestimate the time, and just start adding a factor (2x) to what their initial estimate is. That is a step up. But many never learn. There is also a race to the bottom -- the ones that promise shorter timelines get promoted and seen as smarter, more aggressive, better team players. The ones that are realistic are seen as maybe "taking it easy", "not working hard enough". Then everyone sees who gets promoted and show doesn't and learn -- "Aha I need to be like cool cowboy coder Joe over there, he is now running his own department". And so it spreads. Notice how there is often no feedback loop to punish the cool cowboy coder after the project is not delivered on time.
On the second part -- even if there are engineers who know how to estimate the time, nobody asks them. Because the person who promised the feature already played golf with the customer and told them they'll get the new shiny toy in 6 months. That person might have never been a developer, they might have an MBA or where a football coach or something like that. And they usually have a higher position in the company. So they promise and then everyone is pushed to deliver on that deadline. If it fails the failure is usually attributed to the engineers doing the work because they haven't worked "hard enough".