It's nearly 35 years now that I have been in the software production and integration business! As an IT consultant, I have witnessed the inner workings of many types of businesses that claim to have a working software development and integration division. I am still regularly surprised how some of the most venerable banks (actually, they are all scum) and public organizations accountable to tax-payers hap-hazardly fail to functional software in this day and age, when we already have so much cumulative experience to learn from. 

Way back then, people got into software because some part of their business had to be automated: The typical accountant sooner or later got tired of doing the monthly book runs on accounting-ruled paper and managed to partially automate it by putting the business transactions in a giant spreadsheet. Over time, the accountant got more sophisticated and used multiple sheets for the spreadsheet, congratulated himself for writing his first Excel/Quattro/Lotus123 macro and then had a go at writing the process in BASIC, DBASE or CLIPPER with no projecting background and little understanding of user interface design. Because of the never-ending list of bugs, the control of the program development was fiercely protected by the originator, which endowed a sense of personal ownership over the program that would have been unheard of today. Rolling forward through many painful corporate evolutions to the present, we have developed the good common sense to abstract ourselves from these automations and sever the personal relationship that we have with what are essentially just invisible robots that provide a service. We even trust these services to run on machines that are not our own and that are located in places that we do not care where, and for which we only pay for on a per-use basis.These services are hosted on infinitely-capable machines of which the interfaces are are very quick, reliable, documented, well-understood and highly secure. At last we have way to successfully deliver software-based automation solutions! It is just that some people in the business have not evolved as quickly...

I started off as a bona fide software system developer, but as I repeatedly witnessed IT projects failing, patterns were beginning to emerge of how to architect complex business systems using well-defined services and trusted integration techniques to accomplish the things that had to happen to the data on hand, and also learned how to identify an IT failure from the outset, thanks to the many IT project I was involved in and the few superb people that I worked with, whose intellect and leadership will forever be remembered and whose fruits of their successful IT projects will serve as a constant reminder of what a successful IT project looks like from day 1 to final delivery. And there there the corporate drones with their clammy paws whose heads were not only beset with lard and gristle, but whose selfish, short-sighted, cowardly and meddlesome attitudes spelt all-round disaster for the IT project in hand. Yes, these hapless souls also contributed to my learnings. Here's how to identify IT project failures in advance, and possibly remove yourself from the equation before it tarnishes you too:

Look out for the verbal clues

As a consultant, I often get brought in at last resort to help "streamline" a project / provide "techincal advice" / assist in the "integration" / etc. So when I blythely arrive at the first team meeting and hear the project manager start with threatening lines like: "I tell you now, guys, senior management is watching and we are in their firing line, yah?", I know the following:

I would have had a much better life if I had just walked away every time that this happened.

Buy, don't build, but be careful of the customization

"Can’t we just buy a tool that does X instead of building one?” A Yes-decision to this question is frequently followed by a gazillion lines of customization code to make the bought package do some of the stuff that it was not originally meant to do. I have been a player in this silly game more than once and often programs just do not lend themselves to extending because either the data structures or the processes are not well-documented or understood, or the program is simply not meant to be extended. In case, there was an interface of sorts that only the program vendor had the "keys" to so that no-one but the vendor would be employed to extend this program. Sure, there are cases for this with an unequivocal "Yes" to this question, especially in our current shift towards platforming, distributed service-architectures and an API-driven world. However, if you are not meeting 90% of your requirements straight-up with your chosen third-party product before any customization start, this third-party product will extract a price from you by making you do many things that you did not want to do, in order for you to keep your new pet.

The "Not-Invented-Here" vanity fallacy

"They’re charging us X for their tool; why can’t we just build it in-house?” The pointy-haired boss bizarrely then expects the software team to build something akin to Google in six months, including testing, support framework and documentation. It usually transpires that PH-boss is maths-illiterate and just driven by pure personal glory of "having written" a new and essential tool in the business' arsenal of warding off those pesky competitors. Upon failure of this initiative, PH-boss joins the competition based on the perceived success of said "new tool". 

They are using Ruby, why are we still using PHP?

This was an actual conversation I had once. A PH-boss of ours had a mate in the competion who told him their tech was far better than our tech. We should consider switching the 95% complete product from PHP to Ruby, because "it took them only a week to build the same thing". This sort of knee-jerk based on FOMO ("fear of missing out") is common among young, irational comsumers of mobile telephones. You would not expect this in a functional business.

Testing? Meh! Not 'ere, mate

It is sometimes easy to forget the number of hours that have gone into testing and fixing business software solutions so that they perform adequately well. If you start a new software initiative, you too will also do the same amount of software testing that your competitor had to do and which is mandated by the complexity metric of the work. You can not avoid testing to save costs. Whether you choose to do it upfront as part of your software delivery cycle, or choose to test after arriving red-faced at your customer's site and trying to put together a fix and then test it while the customer is breathing down your neck while muttering apologies, the choice is entirely yours.

The Silver Bullet Program

In some hill-billy software backwaters there are still folks who believe that one could write a “Silver bullet” program such that one would never need a programmer to program code again, e.g. “programming without programmers”. A pointy-haired boss will frequently try to remind you that you, as a software professional, will soon be out of a job thanks to this impending monstrosity, and that you should therefore take a pay-cut / clean the toilets / babysit his alligator.

On the face of it, it does sound like a plausible idea (no, not the alligator bit), so plausible in fact, that IBM set out in the early 1970's to write such a program called "The Last One". All you had to do is configure it to perform whatever task you had in mind. What eventually dawned on those poor IBM software engineers was that the "richer" the capabilities of that program were, the more you had to configure it. In the end, more time and knowledge was required to configure the program than was actually required to write a bespoke program in the first place. Or, as we would do in the modern style, write a series of loosely-coupled micro-services on an enterprise service bus with a cloud-hosted thin client and an infinitly-scalable data store, also in the cloud.

The 100% Bug-free System

In any system of some complexity, there will always be bugs. The best systems have fewer of them, yet they exist... 

Any software manager that decrees in a high-pitched voice that they demand a 100% bug-free system needs to go back to Grade 1 in the School of Reality. Recently, I have had to explain to a pointy-haired clone how bugs differ and that non-deterministic bugs, the ones that proverbially happen only in a blue moon and that we had many of, are extremely difficult to track down. This unwelcome news was not well-received.

DevOps and Collaboration tools are for the birds

I have rarely found a situation where DevOps and Collaboration system were overdone: The sensible projects had four or more working environments that were reasonable facsimiles of one another, and with the PRODUCTION and the PRE-PRODUCTION environments being exact replicas of one another, right down to the virtual network configuration. These projects had across-the-board support for development tools, deployments tools, issue managements tools and code repository management. It was beautiful to behold and great software deliveries ensued.

The not-so-sensible projects had a disparate bunch of servers dotted about the estate of various patch levels and software releases, that were given token names like "PROD", "UAT" or "QA". By some for-fetched inference, these servers now declared to constitute IT environments. "QA" inevitably was also the ginger-haired orphan child server that had to act as "DEV" server and the "TEST" server, and for this reason it was not surrounded by other servers with which to do any end-to-end testing with. This means that the first time that you could do a full integration tests is when you deploy to the environment just ahead of your PRODUCTION environment. This is too late in the game to do this sort of test! And when it fails, as many first-time-tests do, you need to unwind all this stuff that you installed in there.

Don't allow the business to design technical solutions

Have you also gasped in awe at the clunkyness of proposed technical solutions that non-technical business people propose? These types of solutions should be countered with a sound technical alternative, but don't be surprised when it gets slapped down for having infringed some small-minded senior business type's little ego.  

Let's save some money with that expensive consultant!

"Let's not onboard this expensive consultant because it will take too much of his expensive time!" Awesome idea. What could have been a neatly prepared pack of documents of procedures, contact points, an up-to-date architectural diagram of the IT estate, log-ins to servers and services, and preparing a fully-working and configured workstation for the expensive software consultant in advance, was dragged out to a painful journey of corporate decision-tree discovery stretching across nearly 5 months. Many mishaps occurred, huge amounts of time was wasted chasing tails and rumours to get the bits I need, with the result that stuff simply could not be delivered. As a result of my fastidious research and eviscerating conversations up and down the organizational monkey tree, the few people that were able to help were moved to other departments and given a cease-and-desist instruction to not help me. At the end of the assignment, I still did not have a few essential tools installed on my machine. 

Beware ye of Fake Prophets of Y.A.M. (Yet Another Methodology) 

If you have ever had to design using the SSADM or the WATERFALL methodology, you are probably old enough to have a few good war stories to tell too. These old approaches are thankfully discredited now and things became better in the last 20 years with the advent of test-driven development, tighter delivery cycles of smaller units of system logic, and a drive to more realistic resource consumption estimation for this, which is nowadays referred to as AGILE. AGILE worked fine in its early days and all was good. More recently though, AGILE evolved into a clandestine, religious sect that consists of tribes of AGILE developers/integrators/architects and is led by a grand master, known as THE SCRUM MASTER. The SCRUM MASTER's function is to convene the tribe for regular ceremonies of public humiliation and retribution, and to do future planning for the next round of public humiliations, retributions and further assorted character assassinations.

The other function of the esteemed SCRUM MASTER is to save your soul by "removing all your impediments so that yea may do what yea hath come to do", to quote from the hallowed pages of the Scrum Bibbel, for lo, therein lie the keys to eternal IT project salvation. I recently experienced how none of this benovalence came to pass, since when I raised the issues again about the on-boarding in the beginning of my assignment with my grand old scrun master, so that he may "remove my impediments" and thus "prepare the path of the virtuous", he told me to "stop whining" and "get on with it". I can only imagine that this must have displeased the AGILE gods immensely, since this IT project turned out, all things considered, not a huge success.