AI#04 - How AI will transform the software industry
In a short timeframe, we have seen numerous companies launch new AI offerings. Software vendors are left with no choice but to reinvent themselves...
Introduction
Until recently, the field of AI had limited applications in Software. Companies like Amazon, Google, and Spotify leverage AI and Machine Learning in innovative ways, like Alexa for voice recognition and Amazon Prime, which predicts your delivery date with amazing accuracy!
ChatGPT changed everything. GPT-3.5 woke up the software industry, and GPT-4, with its plugin model, made everyone rethink how to build software. In a short timeframe, we have seen numerous companies launch new AI offerings. In six months, we moved from a world where AI was a great differentiator to one where it is a must-have. To make matters worse, building AI software represents a paradigm shift that risks turning trillions of lines of code obsolete. Software vendors are left with no choice but to reinvent themselves. This is the extent of the software revolution we are facing, which we explore in this article.
Impact of AI in Software Development
I was a software developer in a previous life and learned to code with declarative programming like most of my peers. This approach is deterministic. The code does exactly what you tell it to do: if this, do this; otherwise, do that.
AI introduces a new programming paradigm. There is no code anymore. The LLM embeds the logic and is callable through APIs. The interaction is a simple input-output, with the LLM model doing work in between to generate an inference, also known as a prediction.
ChatGPT-4.0 takes human input and produces an answer using its neural network built with 175 billion parameters! There’s a bit of code needed to build a LLM, but ChatGPT and other private or open models are like black boxes. The magic lies in how you optimize the LLM model with weights and layers in the neural networks. There is no need to invest thousands of man-days to develop the LLM. LLM training, while not cheap, represents the new programming model. It is built on statistical models. Interestingly, experts cannot even explain how ChatGPT and other LLMs come up with their answers.
We are still in the early innings. Building efficient and accurate LLMs remains challenging and costly. Yet, it makes little doubt in my mind that future software apps and services will be AI-based. The implications are massive for our industry. Today’s software is becoming obsolete and will be replaced by a new or augmented AI stack.
AI-to-AI or AI-to-API communication may become the new norm, with AI services calling each other and sometimes orchestrating the execution of multiple AI services to complete a task, e.g.: asking ChatGPT to create a meeting invite for your team, generate a document summarizing the topics to discuss, and attach it to the email.
In this new world, the LLM is the program. You need expertise in building LLMs with the right architecture (activation function, objective function, RNN, or other neural networks), in creating quality training datasets, and in optimizing predictions with reinforcement learning (RLHF) until you reach your objective. There is no bug fix or code refactoring. When LLM predictions need some tweaking, you need to “fine-tune” your model and retrain it.
How much software development will follow the “LLM model” in three to five years? 10%, 25%, 50%, or more? The answer will greatly impact how the developer's job will be affected.
AI Impact On Productivity and Jobs
AI may disrupt our industry in two ways. First, AI assistants will boost engineers productivity; some examples include:
AI-generated code and inkine comments
AI-generated test automation code, removing the need to do this tedious task
AI-generated code refactoring to make the code faster and easier to understand
AI-orchestration of QA tests: create, execute, report, and monitor
AI-orchestration of security, performance and scale tests
AI-generated documentation with limited involvement from technical writers
Next, developers will be retrained to learn how to build AI and LLMs, and new engineering positions will appear like prompt engineers. New AI automation services may accelerate the LLM development cycle and let companies ship AI services faster at a fraction of the cost. For instance, you can already use a pre-trained foundation model to start a project; third-party services let you generate synthetic data for LLM training.
This transformation will extend beyond software developers to implementers (SIs, ISVs, and IT teams) who may tomorrow work in an AI-powered IDE. This will result in faster innovation and higher productivity across IT functions.
The productivity gains will be phenomenal. However, either way I look at it, I only see a scenario where AI eventually displaces a large number of Tech jobs. This is worrisome.
The potential for AI automation in software and IT is such that High-Tech companies and IT teams have no choice but to embrace this change to stay competitive. The result may be a profound redefinition of the software and IT landscape.
My wife, for instance, works as a technical writer and writes software documentation. This is one of the writer's jobs that pays well and is still in demand. It is a no-brainer to use an AI to generate, publish, and maintain the entire software documentation. All it takes is a software vendor to come up with a “Tech Writer Copilot”. We may be three to five years away from a drastic transformation that will reduce documentation teams by 70% to 80%.
My wife agrees with me. We are neither upset nor pessimistic about her employment prospects. You need to embrace change, whether you like it or not. We are much more concerned for our kids. What if our daughter decides to become a Technical Writer like her mother? What if she graduates four years later as a Technical Writer with a pile of debt and no job prospects? I touched on this dilemma in part 3 of my AI series: tomorrow’s good jobs may be today’s underrated jobs.
Throw It Away. Rebuild on an AI Stack
In the Next Generation of Intelligent Applications, Charles Lamanna, VP of Business Apps & Platform at Microsoft, explores how executives are focusing on building AI strategies to weather a storm of innovation. AI is eating the world, and he sees a Cambrian explosion in intelligent applications. Microsoft’s newly rearchitected strategy is to build the ultimate AI-driven power platform, starting with the business application software stack.
Charles Lamanna describes the opportunity for Microsoft in a stunning way: 500 million customer apps need to be rebuilt on AI. Wow! He is implicitly saying that today’s business apps are obsolete in an AI-driven world. Each company needs to rebuild their business apps and services on top of AI.
Microsoft is embracing an AI-first strategy and has started its pivot. Kudos to them; it’s impressive coming from such a large corporation. What about other tech companies? How can they weather this storm of innovation? How can they innovate and defend their moat?
This is a hard one to answer. First, your leadership needs to realize that AI is leveling the field. What used to be true yesterday is gone. With AI, your software IP is quickly becoming obsolete because only AI-first IP can win. The engineering talents that got you there may not be the ones that will get you to the next stage. The business leaders that brought you to the top may not be suited for this journey.
Dozens of AI apps and services are released every day. Customers are confused. Business and IT buyers need time to sort out their AI strategy. There is probably a twenty-four- to thirty-six-month window until businesses all over the world embark on their AI transformation. Once they do, you better be positioned for this race!
A Simple Framework For AI Innovation
In his excellent book Power and Prediction, Ajay Agarwal identifies three levels of AI innovation:
AI Point solutions enhance the existing decision process and can be implemented quickly, e.g. intelligent resume scanner for applicants to assist the recruiter
AI App solutions bring new solutions to the existing process and can be implemented fairly quickly, e.g. marketing app with an AI -first campaign writer
AI System solutions require rethinking the entire process and redefine how it is done, e.g. self-driving cars
He predicts that in the early stages, many companies will jump on the AI bandwagon with little notion of efficacy. They will overlook the quality of predictions and the value of such predictions to customers.
In his mind, established leaders are at a disadvantage. Leaders will resist disruptive innovation (system-level) and go for incremental innovation. This is the Blockbuster story. Despite the Netflix threat, they refused to reinvent their business model. They listened to their franchisees instead of their customers; they failed to recognize the threat of new entrants. They went bankrupt as Netflix turned their business upside down.
Startups represent a huge threat for leading software vendors, who face system-level disruption. On the other hand, small and medium-sized software providers are stuck in between. They have to include some AI in their offering but cannot afford to stop investing in their core product. This means they may be limited to point-solution innovation. In that case, partnerships with pure-play AI providers will help.
Protecting Your Moat
Large and medium-sized software vendors should avoid building AI point solutions. It’s tempting because it’s fast and easy. However, point solutions bring little customer value and are hard to defend against dozens of similar apps that are often cheaper and better. This is not a battle worth fighting in the long run.
Leadership will have to set clear goals and stay laser-focused on delivering true customer value. The risk is real for software companies to get stuck in the AI starting blocks while competitors get a head start.
The trend among large vendors is to embed AI in existing apps and develop in-house expertise. This is a good approach, as long as they do not build AI for the sake of AI. In my opinion, an AI app or service should deliver 2x improvements over existing processes, ideally 3x to 5x. In layman’s terms, only build AI when it excels at solving a customer pain point against the existing solution.
The true moat may come from proprietary models tied to your data. For instance, using GPT-4.0 to build an email writer assistant for your sales rep or marketer is not going to help. Embedding custom logic in the assistant, like dynamically inserting CRM data or helping build a target distribution list, provides a competitive advantage because it is hard to replicate.
What about system transformation? This will give your company the best moat. Because it is hard to build and a risky move (too much unknown, full-on risk), only a few visionary companies will take that path. The field is primed for startups to disrupt business models and oligopolies. Once this happens, the only option left for large software vendors is an acquisition. Interestingly, because of the nature of AI and LLMs, companies may have an easier time incorporating AI into their tech stack and gain from it.
Competing on Cost-to-Serve
Cost-to-serve is critical for building a moat and an unfair competitive advantage. In a recent interview, Sam Altman admitted we may have to turn to smaller LLMs. He was implicitly recognizing that at the current cost of AI infrastructure, they burn massive amounts of cash and do not have a viable business model yet. I use the paid version of ChatGPT (GPT-4), and I am limited to 25 queries every three hours. OpenGPT can afford to lose money for a while to build a moat, but eventually, they need to turn a profit to stay in business.
On the same token, big Tech players could face new low-cost entrants that disrupt their business model. Think of low-cost airlines and budget hotels applied to AI software. If AI computing (CPU, GPU) follows Moore’s laws, we could see innovative LLM Infra that can compute AI prompts at a drastically reduced cost.
Open-source AI as the Ultimate Threat
Nowadays, it is all about OpenAI (private model). ChatGPT plugins let developers build extensions that can be embedded in ChatGPT, similar to Chrome extensions. They are building a marketplace to sell third-party extensions and have signed numerous strategic partnerships. They are racing to build a moat.
Amazon, Google, and Microsoft are not standing still. In the last few months, each company has made multiple announcements and launched new AI offerings. Microsoft announced Office Copilot and developed a strategic bilateral partnership with OpenAI. Google recently launched Bard. Amazon introduced new tools to build Generative AI.
This leaked memo from a Google Engineer explains how open-source AI is threatening the moat of Google and other tech companies. More importantly, AI may redefine how we buy and interact with software. Google and Facebook's advertising models may become obsolete in an AI world where the interface is your AI assistant. Why would you bother looking at Google search results when you can get an instant answer through a prompt?
Open-source LLMs are catching up fast and are already surpassing GPT-3.5. Anyone can build the next AI-based CRM software or HR app in their garage at a low cost. I used to think of privacy, security, and ethics as unique differentiators for B2B software companies. I am not so sure anymore. This may be offered as part of the foundational open-source model in the future at a low cost.
I am betting on open-source AI to take a decent slice of the pie for a simple reason: inter-connectivity of AI models through APIs. In the past, you had to choose your camp (Linux vs. Microsoft) and build a full-stack architecture on one or the other—even though microservices and distributed architecture helped integrate multiple tech stacks.
In the world of AI, you will either rent a private AI service, e.g., ChatGPT, or create your own with a private or open-source foundation model. You will probably add custom models on top to deliver your end service, e.g., an AI assistant for a radiologist. While open and closed AI models have their pros and cons, developers can choose the best AI service for their needs without being locked in for life. I am not a developer anymore. I may be wrong, and this is worth revisiting later.
A New Paradigm To Buy Software
AI predictions are the new KPIs for software: the better your prediction model, the higher the customer ROI. And since each AI prediction has an associated cost for the vendor, minimizing that cost can lead to higher profit margins. Therefore, software companies may compete on two fronts: quality of predictions and cost to serve.
Because customers will probably pay for AI services on a usage-based model, it is not hard to extrapolate that buyers will compare the performance of different AI models based on the net savings per AI prediction (business savings minus cost of AI prediction) to pick a winner.
Let’s take an example. ABC receives 100,000 cases a year and seeks to reduce support costs through better case deflection. AI vendor X has a 95% success rate for their AI predictions, generating a potential $9.5 million in savings for ABC (excluding AI software costs). ACME builds a similar AI model with a 96% success rate. By selecting ACME, ABC saves an additional $100K a year. It is a trivial choice.
ACME’s average AI cost per case deflection is $3, for a total of $30 million a year (10 million cases across their installed base). Their R&D decreases the LLM's cost to serve by 1%. This generates $300K ($0.03*10M) in profit for the company.
In short, accuracy of predictions gives you a competitive moat to gain additional market shares. Lower cost to serve improves your margin and gives you more flexibility to compete on price.
Conclusion
The software industry is on the verge of a cataclysmic revolution that will redefine the software landscape over the next ten years. Customers, still confused today, will soon rebuild their businesses on an AI stack, creating a virtuous cycle of innovation. To be winners, companies need to not only embrace AI but also deliver true customer value. Building and keeping a moat will be challenging with competitive threats from lower-cost entrants, open-source offerings, and startups bringing system-level disruption. I fear that AI will displace a substantial number of tech jobs and leave behind a generation of young graduates searching for their promised land. I wish to be wrong.