Multimodal: AI’s new frontier

Multimodality is a relatively new term for something extremely old: how people have learned about the world since humanity appeared. Individuals receive information from myriad sources via their senses, including sight, sound, and touch. Human brains combine these different modes of data into a highly nuanced, holistic picture of reality.

“Communication between humans is multimodal,” says Jina AI CEO Han Xiao. “They use text, voice, emotions, expressions, and sometimes photos.” That’s just a few obvious means of sharing information. Given this, he adds, “it is very safe to assume that future communication between human and machine will also be multimodal.”

A technology that sees the world from different angles

We are not there yet. The furthest advances in this direction have occurred in the fledgling field of multimodal AI. The problem is not a lack of vision. While a technology able to translate between modalities would clearly be valuable, Mirella Lapata, a professor at the University of Edinburgh and director of its Laboratory for Integrated Artificial Intelligence, says “it’s a lot more complicated” to execute than unimodal AI.

In practice, generative AI tools use different strategies for different types of data when building large data models—the complex neural networks that organize vast amounts of information. For example, those that draw on textual sources segregate individual tokens, usually words. Each token is assigned an “embedding” or “vector”: a numerical matrix representing how and where the token is used compared to others. Collectively, the vector creates a mathematical representation of the token’s meaning. An image model, on the other hand, might use pixels as its tokens for embedding, and an audio one sound frequencies.

A multimodal AI model typically relies on several unimodal ones. As Henry Ajder, founder of AI consultancy Latent Space, puts it, this involves “almost stringing together” the various contributing models. Doing so involves various techniques to align the elements of each unimodal model, in a process called fusion. For example, the word “tree”, an image of an oak tree, and audio in the form of rustling leaves might be fused in this way. This allows the model to create a multifaceted description of reality.

This content was produced by Insights, the custom content arm of MIT Technology Review. It was not written by MIT Technology Review’s editorial staff.

The top 3 ways to use generative AI to empower knowledge workers 

Though generative AI is still a nascent technology, it is already being adopted by teams across companies to unleash new levels of productivity and creativity. Marketers are deploying generative AI to create personalized customer journeys. Designers are using the technology to boost brainstorming and iterate between different content layouts more quickly. The future of technology is exciting, but there can be implications if these innovations are not built responsibly.

As Adobe’s CIO, I get questions from both our internal teams and other technology leaders: how can generative AI add real value for knowledge workers—at an enterprise level? Adobe is a producer and consumer of generative AI technologies, and this question is urgent for us in both capacities. It’s also a question that CIOs of large companies are uniquely positioned to answer. We have a distinct view into different teams across our organizations, and working with customers gives us more opportunities to enhance business functions.

Our approach

When it comes to AI at Adobe, my team has taken a comprehensive approach that includes investment in foundational AI, strategic adoption, an AI ethics framework, legal considerations, security, and content authentication. ​The rollout follows a phased approach, starting with pilot groups and building communities around AI. ​

This approach includes experimenting with and documenting use cases like writing and editing, data analysis, presentations and employee onboarding, corporate training, employee portals, and improved personalization across HR channels. The rollouts are accompanied by training podcasts and other resources to educate and empower employees to use AI in ways that improve their work and keep them more engaged. ​

Unlocking productivity with documents

While there are innumerable ways that CIOs can leverage generative AI to help surface value at scale for knowledge workers, I’d like to focus on digital documents—a space in which Adobe has been a leader for over 30 years. Whether they are sales associates who spend hours responding to requests for proposals (RFPs) or customizing presentations, marketers who need competitive intel for their next campaign, or legal and finance teams who need to consume, analyze, and summarize massive amounts of complex information—documents are a core part of knowledge workers’ daily work life. Despite their ubiquity and the fact that critical information lives inside companies’ documents (from research reports to contracts to white papers to confidential strategies and even intellectual property), most knowledge workers are experiencing information overload. The impact on both employee productivity and engagement is real.  

Lessons from customer zero

Adobe invented the PDF and we’ve been innovating new ways for knowledge workers to get more productive with their digital documents for decades. Earlier this year, the Acrobat team approached my team about launching an all-employee beta for the new generative AI-powered AI Assistant. The tool is designed to help people consume the information in documents faster and enable them to consolidate and format information into business content.

I faced all the same questions every CIO is asking about deploying generative AI across their business— from security and governance to use cases and value. We discovered the following three specific ways where generative AI helped (and is still helping) our employees work smarter and improve productivity.

1. Faster time to knowledge
   Our employees used AI Assistant to close the gap between understanding and action for large, complicated documents. The generative AI-powered tool’s summary feature automatically generates an overview to give readers a quick understanding of the content. A conversational interface allows employees to “chat” with their documents and provides a list of suggested questions to help them get started. To get more details, employees can ask the assistant to generate top takeaways or surface only the information on a specific topic. At Adobe, our R&D teams used to spend more than 10 hours a week reading and analyzing technical white papers and industry reports. With generative AI, they’ve been able to nearly halve that time by asking questions and getting answers about exactly what they need to know and instantly identifying trends or surfacing inconsistencies across multiple documents.
2. Easy navigation and verification
   AI-powered chat is gaining ground on traditional search when it comes to navigating the internet. However, there are still challenges when it comes to accuracy and connecting responses to the source. Acrobat AI Assistant takes a more focused approach, applying generative AI to the set of documents employees select and providing hot links and clickable citations along with responses. So instead of using the search function to locate random words or trying to scan through dozens of pages for the information they need, AI Assistant generates both responses and clickable citations and links, allowing employees to navigate quickly to the source where they can quickly verify the information and move on, or spend time deep diving to learn more. One example of where generative AI is having a huge productivity impact is with our sales teams who spend hours researching prospects by reading materials like annual reports as well as responding to RFPs. Consuming that information and finding just the right details for RPFs can cost each salesperson more than eight hours a week. Armed with AI Assistant, sales associates quickly navigate pages of documents and identify critical intelligence to personalize pitch decks and instantly find and verify technical details for RFPs, cutting the time they spend down to about four hours.
3. Creating business content
   One of the most interesting use cases we helped validate is taking information in documents and formatting and repurposing that information into business content. With nearly 30,000 employees dispersed across regions, we have a lot of employees who work asynchronously and depend on technology and colleagues to keep them up to date. Using generative AI, employees can now summarize meeting transcripts, surface action items, and instantly format the information into an email for sharing with their teams or a report for their manager. Before starting the beta, our communications teams reported spending a full workday (seven to 10 hours) per week transforming documents like white papers and research reports into derivative content like media briefing decks, social media posts, blogs, and other thought leadership content. Today they’re saving more than five hours a week by instantly generating first drafts with the help of generative AI.

Simple, safe, and responsible

CIOs love learning about and testing new technologies, but at times they can require lengthy evaluations and implementation processes. Acrobat AI Assistant can be deployed in minutes on the desktop, web, or mobile apps employees already know and use every day. Acrobat AI Assistant leverages a variety of processes, protocols, and technologies so our customers’ data remains their data and they can deploy the features with confidence. No document content is stored or used to train AI Assistant without customers’ consent, and the features only deliver insights from documents users provide. For more information about Adobe is deploying generative AI safely, visit here.

Generative AI is an incredibly exciting technology with incredible potential to help every knowledge worker work smarter and more productively. By having the right guardrails in place, identifying high-value use cases, and providing ongoing training and education to encourage successful adoption, technology leaders can support their workforce and companies to be wildly successful in our AI-accelerated world.  

This content was produced by Adobe. It was not written by MIT Technology Review’s editorial staff.

Taking AI to the next level in manufacturing

Few technological advances have generated as much excitement as AI. In particular, generative AI seems to have taken business discourse to a fever pitch. Many manufacturing leaders express optimism: Research conducted by MIT Technology Review Insights found ambitions for AI development to be stronger in manufacturing than in most other sectors.

Manufacturers rightly view AI as integral to the creation of the hyper-automated intelligent factory. They see AI’s utility in enhancing product and process innovation, reducing cycle time, wringing ever more efficiency from operations and assets, improving maintenance, and strengthening security, while reducing carbon emissions. Some manufacturers that have invested to develop AI capabilities are still striving to achieve their objectives.

This study from MIT Technology Review Insights seeks to understand how manufacturers are generating benefits from AI use cases—particularly in engineering and design and in factory operations. The survey included 300 manufacturers that have begun working with AI. Most of these (64%) are currently researching or experimenting with AI. Some 35% have begun to put AI use cases into production. Many executives that responded to the survey indicate they intend to boost AI spending significantly during the next two years. Those who haven’t started AI in production are moving gradually. To facilitate use-case development and scaling, these manufacturers must address challenges with talents, skills, and data.

Following are the study’s key findings:

• Talent, skills, and data are the main constraints on AI scaling. In both engineering and design and factory operations, manufacturers cite a deficit of talent and skills as their toughest challenge in scaling AI use cases. The closer use cases get to production, the harder this deficit bites. Many respondents say inadequate data quality and governance also hamper use-case development. Insufficient access to cloud-based compute power is another oft-cited constraint in engineering and design.
• The biggest players do the most spending, and have the highest expectations. In engineering and design, 58% of executives expect their organizations to increase AI spending by more than 10% during the next two years. And 43% say the same when it comes to factory operations. The largest manufacturers are far more likely to make big increases in investment than those in smaller—but still large—size categories.
• Desired AI gains are specific to manufacturing functions. The most common use cases deployed by manufacturers involve product design, conversational AI, and content creation. Knowledge management and quality control are those most frequently cited at pilot stage. In engineering and design, manufacturers chiefly seek AI gains in speed, efficiency, reduced failures, and security. In the factory, desired above all is better innovation, along with improved safety and a reduced carbon footprint.
• Scaling can stall without the right data foundations. Respondents are clear that AI use-case development is hampered by inadequate data quality (57%), weak data integration (54%), and weak governance (47%). Only about one in five manufacturers surveyed have production assets with data ready for use in existing AI models. That figure dwindles as manufacturers put use cases into production. The bigger the manufacturer, the greater the problem of unsuitable data is.
• Fragmentation must be addressed for AI to scale. Most manufacturers find some modernization of data architecture, infrastructure, and processes is needed to support AI, along with other technology and business priorities. A modernization strategy that improves interoperability of data systems between engineering and design and the factory, and between operational technology (OT) and information technology (IT), is a sound priority.

This content was produced by Insights, the custom content arm of MIT Technology Review. It was not written by MIT Technology Review’s editorial staff.

Open-sourcing generative AI

The views expressed in this video are those of the speakers, and do not represent any endorsement or sponsorship.

Is the open-source approach, which has democratized access to software, ensured transparency, and improved security for decades, now poised to have a similar impact on AI? We dissect the balance between collaboration and control, legal ramifications, ethical considerations, and innovation barriers as the AI industry seeks to democratize the development of large language models.

Explore more from Booz Allen Hamilton on the future of AI

About the speakers

Alison Smith, Director of Generative AI, Booz Allen Hamilton

Alison Smith is a Director of Generative AI at Booz Allen Hamilton where she helps clients address their missions with innovative solutions. Leading Booz Allen’s investments in Generative AI and grounding them in real business needs, Alison employs a pragmatic approach to designing, implementing, and deploying Generative AI that blends existing tools with additional customization. She is also responsible for disseminating best practices and key solutions throughout the firm to ensure that all teams are up-to-date on the latest available tools, solutions, and approaches to common client problems.

In addition to her role at Booz Allen which balances technical solutions and business growth, Alison also enjoys staying connected to and serving her local community. From 2017-2021, Alison served on the board of a non-profit, DC Open Government Coalition (DCOGC), a group that seeks to enhance public access to government information and ensure transparent government operations; in November 2021, Alison was recognized as a Power Woman in Code by DCFemTech.

Alison has an MBA from The University of Chicago Booth School of Business and a BA from Middlebury College.

Tackling AI risks: Your reputation is at stake

Forget Skynet: One of the biggest risks of AI is your organization’s reputation. That means it’s time to put science-fiction catastrophizing to one side and begin thinking seriously about what AI actually means for us in our day-to-day work.

This isn’t to advocate for navel-gazing at the expense of the bigger picture: It’s to urge technologists and business leaders to recognize that if we’re to address the risks of AI as an industry—maybe even as a society—we need to closely consider its immediate implications and outcomes. If we fail to do that, taking action will be practically impossible.

Risk is all about context

Risk is all about context. In fact, one of the biggest risks is failing to acknowledge or understand your context: That’s why you need to begin there when evaluating risk.

This is particularly important in terms of reputation. Think, for instance, about your customers and their expectations. How might they feel about interacting with an AI chatbot? How damaging might it be to provide them with false or misleading information? Maybe minor customer inconvenience is something you can handle, but what if it has a significant health or financial impact?

Even if implementing AI seems to make sense, there are clearly some downstream reputation risks that need to be considered. We’ve spent years talking about the importance of user experience and being customer-focused: While AI might help us here, it could also undermine those things as well.

There’s a similar question to be asked about your teams. AI may have the capacity to drive efficiency and make people’s work easier, but used in the wrong way it could seriously disrupt existing ways of working. The industry is talking a lot about developer experience recently—it’s something I wrote about for this publication—and the decisions organizations make about AI need to improve the experiences of teams, not undermine them.

In the latest edition of the Thoughtworks Technology Radar—a biannual snapshot of the software industry based on our experiences working with clients around the world—we talk about precisely this point. We call out AI team assistants as one of the most exciting emerging areas in software engineering, but we also note that the focus has to be on enabling teams, not individuals. “You should be looking for ways to create AI team assistants to help create the ‘10x team,’ as opposed to a bunch of siloed AI-assisted 10x engineers,” we say in the latest report.

Failing to heed the working context of your teams could cause significant reputational damage. Some bullish organizations might see this as part and parcel of innovation—it’s not. It’s showing potential employees—particularly highly technical ones—that you don’t really understand or care about the work they do.

Tackling risk through smarter technology implementation

There are lots of tools that can be used to help manage risk. Thoughtworks helped put together the Responsible Technology Playbook, a collection of tools and techniques that organizations can use to make more responsible decisions about technology (not just AI).

However, it’s important to note that managing risks—particularly those around reputation—requires real attention to the specifics of technology implementation. This was particularly clear in work we did with an assortment of Indian civil society organizations, developing a social welfare chatbot that citizens can interact with in their native languages. The risks here were not unlike those discussed earlier: The context in which the chatbot was being used (as support for accessing vital services) meant that inaccurate or “hallucinated” information could stop people from getting the resources they depend on.

This contextual awareness informed technology decisions. We implemented a version of something called retrieval-augmented generation to reduce the risk of hallucinations and improve the accuracy of the model the chatbot was running on.

Retrieval-augmented generation features on the latest edition of the Technology Radar. It might be viewed as part of a wave of emerging techniques and tools in this space that are helping developers tackle some of the risks of AI. These range from NeMo Guardrails—an open-source tool that puts limits on chatbots to increase accuracy—to the technique of running large language models (LLMs) locally with tools like Ollama, to ensure privacy and avoid sharing data with third parties. This wave also includes tools that aim to improve transparency in LLMs (which are notoriously opaque), such as Langfuse.

It’s worth pointing out, however, that it’s not just a question of what you implement, but also what you avoid doing. That’s why, in this Radar, we caution readers about the dangers of overenthusiastic LLM use and rushing to fine-tune LLMs.

Rethinking risk

A new wave of AI risk assessment frameworks aim to help organizations consider risk. There is also legislation (including the AI Act in Europe) that organizations must pay attention to. But addressing AI risk isn’t just a question of applying a framework or even following a static set of good practices. In a dynamic and changing environment, it’s about being open-minded and adaptive, paying close attention to the ways that technology choices shape human actions and social outcomes on both a micro and macro scale.

One useful framework is Dominique Shelton Leipzig’s traffic light framework. A red light signals something prohibited—such as discriminatory surveillance—while a green light signals low risk and a yellow light signals caution. I like the fact it’s so lightweight: For practitioners, too much legalese or documentation can make it hard to translate risk to action.

However, I also think it’s worth flipping the framework, to see risks as embedded in contexts, not in the technologies themselves. That way, you’re not trying to make a solution adapt to a given situation, you’re responding to a situation and addressing it as it actually exists. If organizations take that approach to AI—and, indeed, to technology in general—that will ensure they’re meeting the needs of stakeholders and keep their reputations safe.

This content was produced by Thoughtworks. It was not written by MIT Technology Review’s editorial staff.