Generative AI deployment: Strategies for smooth scaling

After a procession of overhyped technologies like Web3, the metaverse, and blockchain, executives are bracing for the tidal wave of generative AI, a shift some consider to be on par with the advent of the internet or the desktop computer. But with power comes responsibility, and generative AI offers as much risk as reward. The technology is testing legal regimes in copyright and intellectual property, creating new cyber and data governance threats, and setting off automation anxiety in the workforce.

Organizations need to move quickly to keep up with stakeholder expectations, yet they must proceed carefully to ensure they do not fall foul of regulations or ethical standards in areas like data privacy and bias. Operationally, enterprises need to reconfigure their workforce and forge partnerships with tech companies to design safe, effective, and reliable generative AI.

To gauge the thinking of business decision-makers at this crossroads, MIT Technology Review Insights polled 1,000 executives about their current and expected generative AI use cases, implementation barriers, technology strategies, and workforce planning. Combined with insights from an expert interview panel, this poll offers a view into today’s major strategic considerations for generative AI, helping executives reason through the major decisions they are being called upon to make.

Key findings from the poll and interviews include the following:

• Executives recognize the transformational potential of generative AI, but they are moving cautiously to deploy. Nearly all firms believe generative AI will affect their business, with a mere 4% saying it will not affect them. But at this point, only 9% have fully deployed a generative AI use case in their organization. This figure is as low as 2% in the government sector, while financial services (17%) and IT (28%) are the most likely to have deployed a use case. The biggest hurdle to deployment is understanding generative AI risks, selected as a top-three challenge by 59% of respondents.

• Companies will not go it alone: Partnerships with both startups and Big Tech will be critical to smooth scaling. Most executives (75%) plan to work with partners to bring generative AI to their organization at scale, and very few (10%) consider partnering to be a top implementation challenge, suggesting that a strong ecosystem of providers and services is available for collaboration and co-creation. While Big Tech, as developers of generative AI models and purveyors of AI-enabled software, has an ecosystem advantage, startups enjoy advantages in several specialized niches. Executives are somewhat more likely to plan to team up with small AI-focused companies (43%) than large tech firms (32%).

• Access to generative AI will be democratized across the economy. Company size has no bearing on a firm’s likelihood to be experimenting with generative AI, our poll found. Small companies (those with annual revenue less than $500 million) were three times more likely than mid-sized firms ($500 million to $1 billion) to have already deployed a generative AI use case (13% versus 4%). In fact, these small companies had deployment and experimentation rates similar to those of the very largest companies (those with revenue greater than $10 billion). Affordable generative AI tools could boost smaller businesses in the same way as cloud computing, which granted companies access to tools and computational resources that would once have required huge financial investments in hardware and technical expertise.

• One-quarter of respondents expect generative AI’s primary effect to be a reduction in their workforce. The figure was higher in industrial sectors like energy and utilities (43%), manufacturing (34%), and transport and logistics (31%). It was lowest in IT and telecommunications (7%). Overall, this is a modest figure compared to the more dystopian job replacement scenarios in circulation. Demand for skills is increasing in technical fields that focus on operationalizing AI models and in organizational and management positions tackling thorny topics including ethics and risk. AI is democratizing technical skills across the workforce in ways that could lead to new job opportunities and increased employee satisfaction. But experts caution that, if deployed poorly and without meaningful consultation, generative AI could degrade the qualitative experience of human work.

• Regulation looms, but uncertainty is today’s greatest challenge. Generative AI has spurred a flurry of activity as legislators try to get their arms around the risks, but truly impactful regulation will move at the speed of government. In the meantime, many business leaders (40%) consider engaging with regulation or regulatory uncertainty a primary challenge of generative AI adoption. This varies greatly by industry, from a high of 54% in government to a low of 20% in IT and telecommunications.

Download the report.

This content was produced by Insights, the custom content arm of MIT Technology Review. It was not written by MIT Technology Review’s editorial staff.

Laying the foundation for data- and AI-led growth

Enterprise adoption of AI is ready to shift into higher gear. The capabilities of generative AI have captured management attention across the organization, and technology executives are moving quickly to deploy or experiment with it. Many organizations intend to increase their spending on the wider family of AI capabilities and the data infrastructure that supports them by double digits during the next year. And notwithstanding concerns about unfavorable economic conditions, executives see opportunities to leverage data and AI to deliver more growth to their organizations, to both the top and bottom lines.

Based on a global survey of 600 technology leaders and a series of in-depth interviews, this report finds that organizations are sharply focused on retooling for a data and AI-driven future. Everything from data architecture to AI-enabled automation is on the table, as technology executives strive to find new efficiencies and new sources of growth. At the same time, the pressure to democratize the power of data and AI creates renewed urgency to bolster data governance and security.

Following are the study’s key findings:

• CIOs are doubling down on their investments in data and AI. Faced with increasing audience expectations, new competitive pressures, a challenging economic backdrop, and an unprecedented speed of innovation, technology leaders need their data and AI assets to deliver more growth to the business than ever before. They are investing to secure this future: every organization surveyed will boost its spending on modernizing data infrastructure and adopting AI during the next year, and for nearly half (46%), the increase will exceed 25%.

• Consolidation of data and AI systems is a priority. The proliferation of data and AI systems is particularly extensive in the survey’s largest organizations (those with annual revenue of more than $10 billion). Among these, 81% operate 10 or more of these systems, and 28% use more than 20. The executives we interviewed aim to pare down their multiple systems, connecting data from across the enterprise in unified platforms to break down silos and enable AI initiatives to scale.

• Democratization of AI raises the stakes for governance. As business units and their staff clamor to use generative AI, executives seek assurance that governance frameworks for the technology can provide not only the needed data accuracy and integrity but also adequate data privacy and security. That’s probably why 60% of respondents say a single governance model for data and AI is “very important.”

• Executives expect AI adoption to be transformative in the short term. Eighty percent of survey respondents expect AI to boost efficiency in their industry by at least 25% in the next two years. One-third say the gain will be at least 50%.

• As generative AI spreads, flexible approaches are favored. Eighty-eight percent of organizations are using generative AI, with one-quarter (26%) investing in and adopting it and another 62% experimenting with it. The majority (58%) are taking a hybrid approach to developing these capabilities, using vendors’ large language models (LLMs) for some use cases and building their own models when IP ownership, privacy, security, and accuracy requirements are tighter.

• Lakehouse has become the data architecture of choice for the era of generative AI. Nearly three-quarters of surveyed organizations have adopted a lakehouse architecture, and almost all of the rest expect to do so in the next three years. Survey respondents say they need their data architecture to support streaming data workloads for real-time analytics (a capability deemed “very important” by 72%), easy integration of emerging technologies (66%), and sharing of live data across platforms (64%). Ninety-nine percent of lakehouse adopters say the architecture is helping them achieve their data and AI goals, and 74% say the help is “significant.”

• Investment in people will unlock more value from data and AI. In our survey, talent and skills gaps overshadow organizations’ other data and AI challenges. When asked where their company’s data strategy needs to improve, the largest share of respondents (39%) say investing in talent. The number-one difficulty they face with their data and AI platforms, with 40% citing this as a top concern, is training and upskilling staff to use them.

A subsequent report will examine these survey results in detail, accompanied by insights from additional executive interviews across six sectors: financial services, health care and life sciences, retail and consumer packaged goods, manufacturing, media and entertainment, and government.

Download the report.

This content was produced by Insights, the custom content arm of MIT Technology Review. It was not written by MIT Technology Review’s editorial staff.

Driving companywide efficiencies with AI

Autonomous shopping carts that follow grocery store customers and robots that pick ripe cucumbers faster than humans may grab headlines, but the most compelling applications of AI and ML technology are behind the scenes. Increasingly, organizations are finding substantial efficiency gains by applying AI- and ML-powered tools to back-office procedures such as document processing, data entry, employee onboarding, and workflow automation.

The power of automation to augment productivity in the back office has been clear for decades, but the recent emergence of advanced AI and ML tools offers a step change in what automation can accomplish, including in highly regulated industries such as health care.

“In the past, AI was seen as a complex and expensive technology that was only accessible to large companies with deep pockets,” says Himadri Sarkar, executive vice president and global head of consulting at Teleperformance, a digital business services company. “However, the development of easy-to-use generative AI tools has made it possible for businesses of all sizes to experiment with AI and see how it can benefit their operations.”

Organizations are taking note with innovative use cases that not only promise to improve back-office operations but also deliver bottom-line benefits, from cost savings to productivity gains.

AI in action

According to McKinsey’s 2022 Global Survey on AI, AI adoption has more than doubled—from 20% of respondents having adopted AI in at least one business area in 2017 to 50% today. It’s easy to understand this technology’s growing popularity: as challenging economic times meet increasing customer expectations, organizations are being asked to do more with less.

“Companies are trying to optimize their use of resources in an inflationary environment,” says Omer Minkara, vice president and principal analyst with Aberdeen Strategy and Research. “Adding to the pressure is the fact that many companies have to defer their technology spend and headcount increases.”

Fortunately, AI and ML solutions can help bridge this gap for a wide range of industries by automating and optimizing various back-office tasks and processes. A retailer, for example, may use AI-powered chatbots to handle routine customer inquiries, track orders, and respond to refund requests, improving response times, enhancing customer experience, and freeing up contact center agents. At the same time, financial institutions are discovering the power of ML to identify anomalies within large volumes of data that may indicate fraud—an early warning system against financial loss. Organizations across industries can employ AI and ML tools to extract and analyze information from documents, such as invoices, contracts, and reports, and to reduce the burden of manual data entry while speeding up processing times and minimizing human errors.

Download the full report

This content was produced by Insights, the custom content arm of MIT Technology Review. It was not written by MIT Technology Review’s editorial staff.

Bolstering enterprise LLMs with machine learning operations foundations

Generative AI, particularly large language models (LLMs), will play a crucial role in the future of customer and employee experiences, software development, and more. Building a solid foundation in machine learning operations (MLOps) will be critical for companies to effectively deploy and scale LLMs, and generative AI capabilities broadly. In this uncharted territory, improper management can lead to complexities organizations may not be equipped to handle.

Back to basics for emerging AI

To develop and scale enterprise-grade LLMs, companies should demonstrate five core characteristics of a successful MLOps program, starting with deploying ML models consistently. Standardized, consistent processes and controls monitor production models for drift, and data and feature quality. Companies should be able to replicate and retrain ML models with confidence: through quality assurance and governance processes to deployment, without much manual work or rewriting. Lastly, they should ensure their ML infrastructure is resilient (ensuring multiregional availability and failure recovery), consistently scanned for cyber vulnerabilities, and well managed.

Once these components are in place, more complex LLM challenges will require nuanced approaches and considerations—from infrastructure to capabilities, risk mitigation, and talent.

Deploying LLMs as a backend

Inferencing with traditional ML models typically involves packaging a model object as a container and deploying it on an inferencing server. As the demands on the model increase—more requests and more customers require more run-time decisions (higher QPS within a latency bound)—all it takes to scale the model is to add more containers and servers. In most enterprise settings, CPUs work fine for traditional model inferencing. But hosting LLMs is a much more complex process which requires additional considerations.

LLMs are comprised of tokens—the basic units of a word that the model uses to generate human-like language. They generally make predictions on a token-by-token basis in an autoregressive manner, based on previously generated tokens until a stop word is reached. The process can become cumbersome quickly: tokenizations vary based on the model, task, language, and computational resources. Engineers deploying LLMs need not only infrastructure experience, such as deploying containers in the cloud, they also need to know the latest techniques to keep the inferencing cost manageable and meet performance SLAs.

Vector databases as knowledge repositories

Deploying LLMs in an enterprise context means vector databases and other knowledge bases must be established, and they work together in real time with document repositories and language models to produce reasonable, contextually relevant, and accurate outputs. For example, a retailer may use an LLM to power a conversation with a customer over a messaging interface. The model needs access to a database with real-time business data to call up accurate, up-to-date information about recent interactions, the product catalog, conversation history, company policies regarding return policy, recent promotions and ads in the market, customer service guidelines, and FAQs. These knowledge repositories are increasingly developed as vector databases for fast retrieval against queries via vector search and indexing algorithms.

Training and fine-tuning with hardware accelerators

LLMs have an additional challenge: fine-tuning for optimal performance against specific enterprise tasks. Large enterprise language models could have billions of parameters. This requires more sophisticated approaches than traditional ML models, including a persistent compute cluster with high-speed network interfaces and hardware accelerators such as GPUs (see below) for training and fine-tuning. Once trained, these large models also need multi-GPU nodes for inferencing with memory optimizations and distributed computing enabled.

To meet computational demands, organizations will need to make more extensive investments in specialized GPU clusters or other hardware accelerators. These programmable hardware devices can be customized to accelerate specific computations such as matrix-vector operations. Public cloud infrastructure is an important enabler for these clusters.

A new approach to governance and guardrails

Risk mitigation is paramount throughout the entire lifecycle of the model. Observability, logging, and tracing are core components of MLOps processes, which help monitor models for accuracy, performance, data quality, and drift after their release. This is critical for LLMs too, but there are additional infrastructure layers to consider.

LLMs can “hallucinate,” where they occasionally output false knowledge. Organizations need proper guardrails—controls that enforce a specific format or policy—to ensure LLMs in production return acceptable responses. Traditional ML models rely on quantitative, statistical approaches to apply root cause analyses to model inaccuracy and drift in production. With LLMs, this is more subjective: it may involve running a qualitative scoring of the LLM’s outputs, then running it against an API with pre-set guardrails to ensure an acceptable answer. 

Governance of enterprise LLMs will be both an art and science, and many organizations are still understanding how to codify them into actionable risk thresholds. With new advances emerging rapidly, it’s wise to experiment with both open source and commercial solutions that can be tailored for specific use cases and governance requirements. This requires a very flexible ML platform, especially the control plane with high levels of abstraction as a foundation. This allows the platform team to add or subtract capabilities, and keep pace with the broader ecosystem, without impacting its users and applications. Capital One views the importance of building out a scaled, well-managed platform control plane with high levels of abstraction and multitenancy as critical to address these requirements.

Recruiting and retaining specialized talent

Depending on how much context the LLM is trained on and the tokens it generates, performance can vary significantly. Training or fine-tuning very large models and serving them in production at scale poses significant scientific and engineering challenges. This will require companies to recruit and retain a wide array of AI experts, engineers, and researchers.

For example, deploying LLMs and vector databases for a service agent assistant to tens of thousands of employees across a company means bringing together engineers experienced in a wide variety of domains such as low-latency/high throughput serving, distributed computing, GPUs, guardrails, and well-managed APIs. LLMs also need to deploy on well-tailored prompts to provide accurate answers, which requires sophisticated prompt engineering expertise.

A deep bench of AI research experts is required to stay abreast of the latest developments, build and fine-tune models, and contribute research to the AI community. This virtuous cycle of open contribution and adoption is key to a successful AI strategy. Long-term success for any AI program will involve a diverse set of talent and experience combining data science, research, design, product, risk, legal, and engineering experts that keep the human user at the center.

Balancing opportunity with safeguards

While it is still early days for enterprise LLMs and new technical capabilities evolve on a daily basis, one of the keys to success is a solid foundational ML and AI infrastructure.

AI will continue accelerating rapidly, particularly in the LLM space. These advances promise to transform in ways that haven’t been possible before. As with any emerging technology, the potential benefits must be balanced with well-managed operational practices and risk management. A targeted MLOps strategy that considers the entire spectrum of models can offer a comprehensive approach to accelerating broader AI capabilities.

This content was produced by Capital One. It was not written by MIT Technology Review’s editorial staff.