It's nearly 35 years now that I have been in the software production and integration business and as an IT consultant, I have witnessed the inner workings of many types of businesses that claim to run their businesses with the aid of software and teams that do software development and integration into the business. I am still regularly surprised how so many businesses, including some of the most venerable banks and public organizations that are accountable to tax-payers, haphazardly fail to deliver functional software solutions that serve the business in this day and age, when we already have so much cumulative experience to learn from.
Way back then, people got into software because some part of their business had to be automated: The typical accountant sooner or later got tired of doing the monthly book runs on accounting-ruled paper and managed to partially automate it by putting the business transactions in a giant spreadsheet. Over time, the accountant got more sophisticated and used multiple sheets for the spreadsheet, congratulated himself for writing his first Excel/Quattro/Lotus123 macro and then had a go at writing the process in BASIC, DBASE or CLIPPER, with little concept nor background of the dynamics of project delivery, and even less understanding of user interface design. Because of the ensuing never-ending list of bugs, the control of the program development was understandably fiercely protected by the originator. This unfortunately endowed a misplaced sense of personal ownership over the program that would have been unheard of today. Rolling forward through many painful corporate evolutions to the present, we have developed the good common sense to abstract ourselves from these automations that we had contrived and severed the personal relationship that we had with what are essentially just invisible robots that provide a so-called "service". We even trust these services to run on machines that are not our own and that are located in places that we do not care where, and for which we only pay for on a per-use basis. These services are hosted on infinitely-capable and extensible machines of which the interfaces are very quick, reliable, documented, well-understood and highly secure. At last we have a way to successfully deliver software-based automation solutions and we call it the "Cloud". The only problem is just that some people in the business have not caught up with this evolution.
When I started off as a bona fide software system developer, I repeatedly witnessed IT projects failing. Rare was the IT project that truly was a success, and this was pretty much everyone's experience. Out of these embers, patterns were st last beginning to emerge of how to architect complex business systems using well-defined services and trusted integration techniques to accomplish the things that had to happen to the data on hand.
The other aspect that we learned was how to predict the likelihood of an IT project being a success. It is now possible to identify an IT project failure from the outset based on both the successes and failures we in the industry have cumulatively witnessed. Here's how to:
Look out for the verbal clues
As a consultant, I often get brought in at last resort to help "streamline" a project / provide "techincal advice" / assist in the "integration" / do a "digital transformation" / etc for those poor, hapless sonovabitches. When I blythely arrive at the first team meeting and hear the project manager open the session with threatening lines like: "I tell you now, guys, senior management is watching and we are in their firing line, yah?", I know the following:
- The project is already in a huge mess
- The project manager is not up to the job of rescuing this mess. Probably got parachuted in because the predessor left in a panic.
- The delivery time scales that we are working to have been halved to suit the business agenda
- There is no "time" to do things properly.
- Most of the issues could be fixed with a series of technical cludges, but the talent is not there to even make this happen.
- There will be a time and cost overrun, no matter which route is chosen.
- There will soon be a new project manager. Rinse. Repeat.
I would have had a much better life, had I just walked away each time the PM resorted to panic.
Buy, don't build, but be careful of the customization
"Can’t we just buy a tool that does X instead of building one?” A Yes-decision to this question is frequently followed by a gazillion lines of customization code to make the bought package do some of the stuff that it was not originally meant to do. I have been a player in this silly game more than once and often programs just do not lend themselves to extending because either the data structures or the processes are not well-documented or understood, or the program is simply not meant to be extended. In case, there was an interface of sorts that only the program vendor had the "keys" to so that no one but the vendor would be employed to extend this program. Sure, there are cases for this with an unequivocal "Yes" to this question, especially in our current shift towards platforming, distributed service architectures and an API-driven world. However, if you are not meeting 90% of your requirements straight-up with your chosen third-party product before any customization starts, this third-party product will extract a price from you by making you do many things that you did not want to do, in order for you to keep your new pet.
The "Not-Invented-Here" vanity fallacy
"They’re charging us X for their tool; why can’t we just build it in-house?” The pointy-haired boss bizarrely then expects the software team to build something similar to Google in six months. This should include testing, support framework and documentation. It usually transpires that PH boss is maths-illiterate and is driven by personal glory of "having written" a new and essential tool in the business' arsenal of warding off those pesky competitors. Upon failure of this initiative, PH-boss joins the competition based on the perceived success of said "new tool".
They are using Ruby, why are we still using PHP?
This was an actual conversation I once had. A Pointy-Haired Boss had a mate in the competition who told him their tech was far better than our tech. He implied that we should consider switching our 95%-complete product from PHP to Ruby, because "it took the competition only a week to build the same thing". This sort of knee-jerk based on FOMO ("fear of missing out") is common among the young and irrational generation who stare all day at their mobile telephones and who feed off the latest crazes.
Testing? Meh! Not 'ere, mate!
It is sometimes easy to forget the number of hours that have gone into testing and fixing business software solutions so that they perform adequately well. If you start a new software initiative, you too will also do the same amount of software testing that your competitor had to do and which is mandated by the complexity metric of the work. You can not avoid testing to save costs. Whether you choose to do it upfront as part of your software delivery cycle, or choose to test after arriving red-faced at your customer's site and trying to put together a fix and then test it while the customer is breathing down your neck while muttering apologies, the choice is entirely yours.
The Silver Bullet Program
In some hill-billy software backwaters there are still folks who believe that one could write a “Silver bullet” program such that one would never need a programmer to program code again, e.g. “programming without programmers”. A pointy-haired boss will frequently try to remind you that you, as a software professional, will soon be out of a job thanks to this impending monstrosity and that you should therefore take a pay-cut / clean the toilets / babysit his alligator.
On the face of it, it does sound like a plausible idea (no, not the alligator bit), so plausible in fact, that IBM set out in the early 1970s to write such a program called "The Last One". All you had to do is configure it to perform whatever task you had in mind. What eventually dawned on those poor IBM software engineers was that the "richer" the capabilities of that program were, the more you had to configure it. In the end, more time and knowledge was required to configure the program than was actually required to write a bespoke program in the first place. Or, as we would do in the modern style, write a series of loosely-coupled micro-services on an enterprise service bus with a thin client and an infinitely-scalable data store, also in the cloud.
The 100% Bug-free System
In any system of some complexity, there will always be bugs. The best systems have fewer of them, yet they exist...
Any software manager that decrees in a high-pitched voice that they demand a 100% bug-free system needs to redo Grade 1 in the School of Reality. Recently, I have had to explain to a pointy-haired clone how bugs differ and that non-deterministic bugs, the ones that proverbially happen only in a blue moon, but which we had many of due to poor system design, are extremely difficult to track down. This unwelcome news was not well-received.
DevOps and Collaboration tools are for the birds
I have rarely found a situation where DevOps and Collaboration systems were overdone: The sensible projects had four or more working environments that were reasonable facsimiles of one another, and with the PRODUCTION and the PRE-PRODUCTION environments being exact replicas of one another, right down to the virtual network configuration and security settings. These projects had across-the-board support for development tools, deployment tools, issue management tools and code repository management. It was beautiful to behold and great software deliveries ensued.
The not-so-sensible projects had a disparate bunch of servers dotted about the estate of various patch levels and software releases, that were given token names like "PROD", "UAT" or "QA". Because the "QA" environment traditionally is not that often used, "QA" inevitably became the ginger-haired orphan-child environment, that due to incomprehensible tightfistedness by one of the wealthiest banks in Switzerland, it also had to act as the "DEV" and the "TEST" environment. If you know about software deployment, you would appreciate that end-to-end testing is an essential step before any roll-out. However, when you have an environment that is essentially a "DEV" environment, it is not connected to any of the peripheral systems that the production system would be connected to, except to the development tools. For this reason, end-to-end testing is impossible. The upshot of this is that the first time that you can do a full integration test is when you deploy to the real PRODUCTION environment. Stuff never works the first time, not so? The rest of the story tells itself...
Let's save some money with that expensive consultant!
"Let's not onboard this expensive consultant because it will take too much of his expensive time!" Awesome idea. What could have been a neatly prepared pack of documents of procedures, contact points, an up-to-date architectural diagram of the IT estate, log-ins to servers and services, and preparing a fully working and configured workstation with the necessary software for the expensive software consultant in advance, was dragged out to a painful journey of discovery of the corporate decision-tree, that stretched across nearly 5 months, for a very troubled Swiss bank, I might add. A huge amount of time was wasted chasing tails and rumours to get the bits that I need installed. The result at the end of the day was that stuff simply could not be delivered. As a result of my fastidious research and eviscerating conversations up and down the organizational monkey tree, the few people that were able to help were moved to other departments and given a cease-and-desist instruction to not help me, since it was deemed that I wasted their time. At the end of my assignment with the little gnomes, I was still missing a few essential programs on my machine.
Beware ye of Fake Prophets of Y.A.M. (Yet Another Methodology)
If you have ever had to design using the SSADM or the WATERFALL methodology, you are probably old enough to have a few good war stories to tell too. These old approaches are thankfully discredited now and things became better in the last 20 years with the advent of various test-driven development methodologies, tighter delivery cycles of smaller units of system logic, and a drive to more realistic resource consumption estimation for this, which is nowadays referred to as AGILE. AGILE worked fine in its early days and all was good. More recently though, AGILE evolved into a clandestine, religious sect that consists of tribes of AGILE developers/integrators/architects and is led by a grand master, known as THE SCRUM MASTER. The SCRUM MASTER's function is to convene the tribe for regular ceremonies of public humiliation and retribution, and to do future planning for the next round of public humiliations, retributions and further assorted and interesting character assassinations. Weird handshakes, the wearing of aprons and silly walks would have been part of this ritual, were it not for the trend for developers to work from home thanks to the Great Lurgy of 2020-2021 that the Chineese unleashed on to the world.
The other function of the esteemed SCRUM MASTER is to save your soul by "removing all your impediments so that yea may do what yea hath come to do", to quote from the hallowed pages of the Scrum Bibbel, "for lo, therein lie the keys to eternal IT project salvation". Probably. I recently experienced how none of this benovalence was bestowed upon myself when I repeatedly raised the issues about the on-boarding problems in the beginning of my assignment with my grand old scrum master. I asked him to "remove my impediments" and to thus "prepare the path of the virtuous", so that I may "deliver my services". His replay was to "stop whining" and "get on with it". I can only imagine that this must have displeased the AGILE gods immensely, since this IT project turned out, all things considered, not a huge success.
Don't allow the business to design technical solutions
Have you also gasped in awe at the clunkyness of proposed technical solutions that non-technical business people propose? These types of solutions should be countered with a sound and simple technical alternative, but don't be surprised when it gets slapped down for having infringed on some small-minded middle-managment business type's little ego. To avoid all this, have a simple rule: No proposed technical solutions in the business requirements. It muddies the lines of responsibilities and abilities, and having the business produce a requirements document limits the damage that they could do on the technical estate, assuming that the production of a requirements document is part of the delivery cycle.
Who needs Requirements?
Another famous cost-despensing falacy.
"Let's start coding some shit, y'all!", says Pointy-Haired Boss.
What, why, for whom?
"Oh, I dunno, I just wanna see some code checked into GIT", says Pointy-Haired Boss.
No requirements. So therefore, no design, no system architecture, no project plan, no test plan, no support. Following the delayed and painful birth of the "minimum viable" prototype, Marketing's slight of hand magically re-casts it as a "product". And then sells it to unwitting customers, the poor bloody sods.
Filling in forms about forms
When your IT project has descended into a red-tape morass like this, run!
Having a meeting about what to call the meeting
When your IT project progress meeting descends into a petty discussion about what to call the meeting, run even faster!
Mission statement alert
So, you clearly have some problems delivering a product by your software team, right?
Here's the cure: Unite the team's goals with a glorious MISSION STATEMENT! A long one. No, really, if it is not at least 30 words long, it ain't a proper mission statement. And all in one sentence.
This abomination of English punctuation-abuse and breath-consuming lugubriousness is usually bestowed upon the lower organizational echelons from a-top. It is a committee effort that was carefully crafted concocted during an off-site management retreat where vast amounts of alcohol was consumed. Mission statements always contain the same, clichéd management phrases that glorify the company's product and its leaders, brown-nose the customer and insult the people who actually help build the product.