I've done most of the spectrum from Perl webdev to chip design, with a good chunk of embedded C and C++ in the middle. I basically agree, and I think there's roughly these types of challenge:
- Complicated business requirements ("do this every second Tuesday if there is an R in the month, but not if the customer is using the US VAT system and not if the product is a type of cake")
- Critical system requirements (remember, using Java to control your nuclear reactor violates the license). People working in this environment can easily get elitist about it, until you look at their code and find Toyota spaghetti.
- Problems that require Actual Maths or Actual CS, with known solutions. Rare in the webdev world until someone remembers about ACID; more common in the embedded world if you need control systems.
- Problems with unknown solutions. Actually extremely rare; often the first line of attack involves Matlab or Excel or some other "not real programming" technique. Often the smart domain experts you need to solve the problem are not also very good at software engineering.
IME the first two take up 80-90% of the working day. The third is only applicable to some industries.
Different people are better at different types of challenge. This doesn't matter unless you're trying to construct some sort of total ordering over human beings in order to feel better about yourself.
> - Complicated business requirements ("do this every second Tuesday if there is an R in the month, but not if the customer is using the US VAT system and not if the product is a type of cake")
And then you realise there's no consistent definition of "cake" in the system. And when people in the business say "US VAT system" they actually mean "Regulatory system XYZ", not actually anything to do with the US.
Plus the requirement is not neatly in one place ("do X every Y except if Z"), but strewn all over a document (or several), and different requirements contradicting each other, with no priority defined.
Funny. Most of my career was about problems with no known solutions. I wonder why people keep calling them "rare". After all, why even bother solving a solved problen?
And yes, most useful prototyping tools for me were far from C++ and alike - I used Maxima, Mathematica, various Lisps with tons of batteries. Also not a "real programming".
#1 Just because you're constantly tackling interesting problems doesn't mean they're super common. Most of us seem to be working on an application meeting some trivial business need.
#2 Just because you don't know the answer doesn't mean the answer doesn't exist. The world of development is filled with 20 somethings getting paid huge sums of money to learn and deliver something completely alike to all the other things they think are really different.
You probably fit into one of these categories, if it's the first then congrats, because that sounds like an interesting life.
I still cannot understand what justifies re-solving a solved problem over and over again? Solved once? Automate it.
> Most of us seem to be working on an application meeting some trivial business need.
OP says it's somehow "not trivial". For some weird reasons.
Anyway, if the workflow is trivial, if required components are standard, if no customisation is required - it does not need any engineering at all. List those initial requirements, infer the implementation automatically. It is hard to comprehend why the CRUD world is so reluctant to do such an easy thing.
> doesn't mean the answer doesn't exist
It may exist in some highly secretive labs, but in most cases it cannot be found in any of the papers or preprints.
So just because something is trivial doesn't mean that it doesn't require time to implement the rules. You can be entering a bunch of convulted rules into a GUI based system or you can be writing them in code.
I largely actually prefer the code way, believing it's much more flexible, sane and safer. However, the problem is the average person who understands these rules is scared of the code, and the average coder is much more interested in solving a lot of inconsequential problems.
This makes it much harder to pay someone in who's just going to sit down and churn out your CRUD and when folks find these guys, they're either keeping them happy or think they're easily replaceable and will have a fall later on.
I'm doing devops at a big org at the moment. They have several dev teams working on bespoke low-traffic resource managment applications. Most of these have 5 developers, devops, delivery manager, user researcher, designer in addition of going view a technical review process, pen testing, performance review, and functional testing.
Basic math suggests that's going to come out at about £350,000 per project over 3 months. A million quid later, they've all passed reviews without any of these so called highly technical people realising they're dealing with a common problem and that they're developing 3 of them. Also they're all shit because they're done by MEAN stack tech hipsters who haven't quite realise they're leaking important data.
> You can be entering a bunch of convulted rules into a GUI based system or you can be writing them in code.
There are more options than just these two. The best way is to write your rules in a nice, readable, dense DSL, designed specifically for that domain experts who know the rules but are afraid of code. And such a DSL can be very much free form and forgiving, helping a lot along the way, so the experts won't need much assistance.
With such an approach, developers (i.e., those who are not afraid of code) are either not needed or only concerned with maintaining this DSL, while the experts can code their rules directly. It eliminates unnecessary elements of a chain, and cuts costs quite significantly.
> It is hard to comprehend why the CRUD world is so reluctant to do such an easy thing.
It is odd, isn't it? I'm tempted to say if you think it's so easy, go forth and do it: automating away the production of CRUD would earn you billions. SAP is probably the nearest existing product. But every attempt so far just creates a different group of developers (e.g. SAP).
In a sane and rational world - yes, it would have been automated. But in our world everything is done the most twisted way possible, either deliberately or out of some deeply ingrained human stupidity.
I cannot stop shaking every time I interact with pretty much anything designed by supposedly high profile specialists. There is no single day I do not run into an astonishing stupidity. Daily commute? London Oyster system is such a thoroughly shitty design that it is nearly unbelievable. Shopping? It never works as one would expect, and is always far too twisted - all that Amazons, Ebays and alike. Healthcare? Just super stupid, stupider than the trashier of the comedies. Yes, they "automate". They decided that a doctor should not type or write, to save a valuable time. They're expected to record everything on a voice recorder and pass it to an assistant for typing. Guess what? They type or write first, because it's easier, and then read it aloud. Shall I go on?
This world is a pile of stupid shit, throughout. And expecting that something that definitely should have been automated or eliminated altogether would have happened is just totally naive.
And if someone really do automate all the CRUD needs for 80% of the possible users, they'd never know anything about it. So why bother? They must grow up and automate it themselves, it's dead easy.
If you are specific enough then most problems have no known solutions. I work for government and we see a lot of solutions that solve most of our problems but there is always a catch. That cool cloud stuff? Can't use it. The problem in reality is we need a solution for X that also satisfies our privacy and security policy. From this perspective there is no known solution. We have to develop our own solution.
This makes sense for embedded: you want to save every penny since it's a high volume business, the chip manufacturer wants to save every penny , there's a huge amount of variety in chips - so in the end development is hard. Without all those requirements, heck maybe something like the arduino/mbed would have become popular among professionals.
But as for webdev ? not so much. The critical factors are time-to-market and development cost. And in the domain of web based business apps, we already have useful high-level tools[1], and it seems to be technically possible to improve the situation in consumer facing apps too.
For webdev, the biggest inadequate tool is the browser itself: ancient versions, vendor quirks, mandatory single language. This is mostly due to it being the place where various competing interests have fought one another to a standstill.
But a good, high level tool solves/abstracts all those issues for it's developers(with the cost of some performance decrease) . The business oriented tools manage to do so.
You hit the nail on the head with the Actual CS problems. Storing data is probably the only complex area of web deevelopment, but it's something that 90% of developers seem to punt on.
- Inadequate or high-effort tools. This is half the challenge of the embedded world, and the cause of "I've destroyed my tools with my tools!" https://www.usenix.org/system/files/1311_05-08_mickens.pdf
- Complicated business requirements ("do this every second Tuesday if there is an R in the month, but not if the customer is using the US VAT system and not if the product is a type of cake")
- Critical system requirements (remember, using Java to control your nuclear reactor violates the license). People working in this environment can easily get elitist about it, until you look at their code and find Toyota spaghetti.
- Problems that require Actual Maths or Actual CS, with known solutions. Rare in the webdev world until someone remembers about ACID; more common in the embedded world if you need control systems.
- Problems with unknown solutions. Actually extremely rare; often the first line of attack involves Matlab or Excel or some other "not real programming" technique. Often the smart domain experts you need to solve the problem are not also very good at software engineering.
IME the first two take up 80-90% of the working day. The third is only applicable to some industries.
Different people are better at different types of challenge. This doesn't matter unless you're trying to construct some sort of total ordering over human beings in order to feel better about yourself.