MCP Multimodal: How AI Agents Will Read Images, Video, and Audio Through MCP

MCP has been a text protocol. Tool calls return text, structured JSON, or embedded resources — but not images, video, or audio. The 2026 roadmap changes this. MCP is gaining native multimodal content types, which means agents won't just read your documentation — they'll see your diagrams, watch your tutorials, and hear your walkthroughs. Here's what's coming and what it means for how you build agent-ready content.

Why MCP started as text-only

The first version of MCP was designed around the most common agentic use case: text retrieval and tool execution. An agent asks a question, a tool returns an answer, the agent reasons about it. This covers the majority of documentation queries, code lookups, API reference searches, and knowledge retrieval tasks.

Text also has the advantage of being universally renderable. Every MCP client can display text. Not every MCP client can render an image, stream a video, or play audio — and the protocol's early priority was maximum client compatibility, not rich media support.

As multimodal models became standard (Claude, GPT-4o, Gemini all process images and audio natively), the text-only constraint started to become a bottleneck. An agent that can see but can only retrieve text is missing significant information — diagrams, screenshots, architectural charts, video demos, audio walkthroughs. The multimodal roadmap closes that gap.

What multimodal MCP adds

The MCP multimodal extension adds four new content types to tool responses alongside the existing text and resource types:

image

Inline images in tool responses, either as base64-encoded data or as a URL reference. The content type includes the MIME type, dimensions hint, and an optional alt text for clients that can't render images:

{
  "type": "image",
  "url": "https://docs.example.com/architecture-diagram.png",
  "mimeType": "image/png",
  "alt": "System architecture showing the three-tier deployment"
}

This is the most immediately useful addition. Documentation sites are full of diagrams, screenshots, and figures that agents currently can't access. A tool that returns the relevant image alongside text context gives the agent significantly more information to work with.

audio

Audio clips in tool responses — useful for sites with podcast content, voice walkthroughs, or recorded explanations. The audio content type includes the MIME type, duration, and a transcript field (strongly recommended so text-only clients can still access the content).

The practical use case for documentation: products that publish video tutorials or recorded demos alongside written docs. Agents can retrieve the transcript (or the audio directly for audio-capable clients) as part of answering a question about how a feature works.

video

Video references in tool responses. Videos are large, so the spec treats them as references rather than inline data — a URL, MIME type, duration, and an optional transcript. Clients that support video rendering can show the video in context; text-only clients fall back to the transcript.

structured_data

Typed structured data — tables, charts, code blocks with language metadata, or any content that has semantic structure beyond plain text. This type carries a schema hint so agents and clients know how to interpret and render the data.

{
  "type": "structured_data",
  "schema": "table",
  "columns": ["Plan", "Price", "API calls/mo", "Seats"],
  "rows": [
    ["Free", "$0", "1,000", "1"],
    ["Pro", "$79", "50,000", "3"],
    ["Team", "$299", "unlimited", "10"]
  ]
}

Structured data is a meaningful upgrade over returning pricing tables as prose. An agent that receives a typed table can reason about it more reliably than one parsing "Free is $0 with 1,000 API calls, Pro is $79 with 50,000..." from text.

Backward compatibility

Multimodal content types are additive. A tool can return a mix of content types in the same response:

{
  "result": {
    "content": [
      { "type": "text", "text": "The deployment flow works as follows:" },
      { "type": "image", "url": "https://...", "alt": "Deployment flow diagram" },
      { "type": "text", "text": "Step 1: push to main triggers the CI pipeline..." }
    ]
  }
}

Clients that support only text will render the text items and skip or display the alt text for image items. Clients that support multimodal content will render everything. Tools don't need to know which client type is calling — they can return the richest response they have, and clients degrade gracefully.

Client support for multimodal content types is advertised in the server/discover capabilities response. Tools that want to optimize their response for the calling client can check which content types the client declares support for and tailor the response accordingly.

What this means for documentation sites

Most documentation is already multimodal — it just wasn't accessible to agents. Architecture diagrams explain system design better than prose. Screenshots show UI state that's hard to describe. Screencasts walk through complex setup flows in minutes that take pages to write.

When MCP multimodal lands in stable clients, documentation tools will need to surface this content alongside text. The sites that have already invested in rich visual documentation — diagrams, annotated screenshots, step-by-step video walkthroughs — will have a significant advantage in agent query quality.

A few things to do now to prepare:

Add alt text to all documentation images. Alt text is how multimodal MCP tools surface images to text-only clients. It's also how RAG pipelines will index image content for semantic retrieval. Good alt text ("Three-tier architecture: browser → API gateway → microservices") is searchable; bad alt text ("diagram.png") isn't.

Add transcripts to video content. Videos without transcripts are invisible to text-based retrieval and to text-only MCP clients. Even basic auto-generated transcripts significantly improve retrievability.

Structure your data content. If your docs contain pricing tables, feature comparison matrices, or configuration reference tables, make sure they exist in HTML table form (not as images or prose) so they can be parsed into the structured_data content type.

When to expect it

The multimodal content types are on the MCP 2026 roadmap but are not part of the July 28 Release Candidate. The RC focuses on protocol stability and auth — multimodal is a later extension. The current expectation is a draft extension spec in Q4 2026, with stable client support following in 2027.

That makes now the right time to prepare your content — not to implement multimodal MCP tool responses. By the time the spec stabilizes and clients ship support, sites with good alt text, transcripts, and structured data will be immediately better represented in agent queries than sites that have to retrofit this content.