Setting up multimodal AI feature in Zoom Virtual Agent

The multimodal AI feature in Zoom Virtual Agent enables users to upload images, PDFs, and other files for AI analysis. This allows the AI to extract key information, identify issues, and provide accurate guidance or next steps. For example, a customer can upload a screenshot of an error message, invoice, or email, and the AI can interpret the content to assist with troubleshooting or inquiries. 
 
Notes:

Requirements for multimodal AI feature in Zoom Virtual Agent

Note: Ensure that the chat engine supports file uploads, including PDFs and images. Contact Zoom Support to enable this feature. 

Table of Contents

How to configure the multimodal AI feature

Configure in agent guidance

Agent guidance provides the virtual agent with instructions on how to handle multimodal interactions. This configuration applies to both chat and voice channels.
  1. Sign in to the Zoom web portal as an admin.
  2. In the top-right corner, click your profile picture or initials, then click Admin Center.
  3. In the side menu, click Product configuration then AI Studio.
  4. Click Virtual Agents.
  5. Open a voice agent or chat agent.
  6. In the Agent Guidance section, add instructions that tell the virtual agent how to handle image attachments from consumers. For example:

Configure the image content tool

The image content tool enables the virtual agent to analyze uploaded images and extract relevant information.
  1. In the agent guidance configuration, add a tool for image content extraction.
  2. Configure the tool with the following description: Extract all image content from the attachment uploaded by the consumer.

Create a skill

Skills define the actions the virtual agent can perform after identifying a consumer's intent. In this example, a skill is created for processing return requests.
  1. Sign in to the Zoom web portal as an admin.
  2. In the top-right corner, click your profile picture or initials, then click Admin Center.
  3. In the side menu, click Product configuration then AI Studio.
  4. Click Virtual Agents.
  5. Open a voice agent or chat agent.
  6. In the Skills section, click Create skill.
  7. Configure the following settings:
  8. Add guidance for handling return requests.
Example skill guidance:
When a consumer requests a return, collect the required order information and verify return eligibility. If the consumer uploads an image containing order details, receipts, invoices, shipping labels, or product information, use the extracted information whenever possible instead of asking the consumer to provide it manually.

Configure the Extract_order_details tool

The return_order skill requires an order ID before it can process a return.
To collect this information, configure the Extract_order_details tool.
  1. Open the Extract_order_details tool within the return_order skill.
  2. Configure the order_id parameter.
Example parameter description:
Ask the consumer for their order ID.
If the consumer uploads an attachment, or has already uploaded an attachment, extract the order ID from the image and use that value when available.

How to add file request link to SMS tool for voice agent

  1. Sign in to the Zoom web portal as an admin.
  2. In the top-right corner, click your profile picture or initials, then click Admin Center.
  3. In the side menu, click Product configuration then AI Studio.
  4. Click Virtual Agents.
  5. Open a voice agent.
  6. In the Skills section, click Add skill.
  7. In the Instructions box, type a forward slash (/) and select Tool from the list.
  8. Add and configure the SMS tool.
  9. In the Content box, click the Insert file upload link icon to include a file request link in the SMS message.
  10. Click Save and add to apply the configuration.
Once configured, the voice agent can automatically send SMS messages containing file upload links to customers during interactions.

Customer experience

Using the chat agent

  1. The user opens the chat engine interface.
  2. The user sends a PDF or image file to the chat agent.
  3. The agent analyzes the content and displays the results in the chat history and report.
  4. Restricted images are automatically rejected and blocked from appearing in chat history or tracing records.
  5. The user can refresh the page to confirm that blocked images are hidden from the chat view.

Using the voice agent

  1. The voice agent sends an SMS message containing a link for file uploads.
  2. When a user reports an issue, the agent prompts them to describe the problem or upload a photo using the provided link.
  3. After the user uploads a file, the agent processes the image and provides a response (for example, identifying a blue screen error and suggesting possible causes).
  4. While the agent processes the file, a default ringtone plays. This ringtone can be customized in the Input and output controls under Waiting media while on hold in the voice agent settings.
  5. Uploaded images and related interactions are recorded and displayed in the report for traceability.
note icon
This article applies to Zoom Virtual Agent voice and chat agent. If you are using Zoom Virtual Agent classic chatbot, refer to this documentation for setup and deployment instructions.