查询音频和视频模型

本页介绍如何向 Azure Databricks 上 Unity Gateway 提供的 Gemini 基础模型发送音频和视频输入。 你可以使用聊天补全 API 或 Google Gemini API,通过 URL 或 base64 编码的内联数据提供媒体。

要求

输入方法

可以使用两种方法提供音频和视频输入:

  • URL:将可公开访问的 URL 传递给媒体文件。 对于视频,还支持 YouTube URL。
  • Base64 内联数据:将文件编码为 base64 字符串,并将其作为数据 URI 传递(例如)。 data:video/mp4;base64,<encoded_data>

注释

以下示例基于 Unity Gateway 和模型服务。 如果使用模型服务终结点而不是模型服务,请将模型服务名称替换为终结点名称。 请参阅 Databricks 基础模型 API 支持的模型详细列表 ,了解可用的基础模型及其模型服务和端点名称。

聊天补全 API

聊天完成 API 允许传递视频和音频输入。 在video_url数组中使用audio_url和messages内容类型来传递媒体输入。 每个内容项都包含一个 url 接受 Web URL 或 base64 数据 URI 的字段。

视频输入

Python

import os
import base64
from openai import OpenAI

DATABRICKS_TOKEN = os.environ.get('DATABRICKS_TOKEN')
DATABRICKS_BASE_URL = os.environ.get('DATABRICKS_BASE_URL')

client = OpenAI(
    api_key=DATABRICKS_TOKEN,
    base_url=DATABRICKS_BASE_URL
)

# Encode a local video file as base64
with open("video.mp4", "rb") as f:
    video_b64 = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="system.ai.gemini-3-1-pro",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Summarize what happens in these videos."},
            {
                "type": "video_url",
                "video_url": {"url": "https://example.com/sample-video.mp4"}
            },
            {
                "type": "video_url",
                "video_url": {"url": f"data:video/mp4;base64,{video_b64}"}
            },
        ]
    }],
    max_tokens=1024
)

print(response.choices[0].message.content)

REST API

curl \
-u token:$DATABRICKS_TOKEN \
-X POST \
-H "Content-Type: application/json" \
-d '{
  "model": "system.ai.gemini-3-1-pro",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Summarize what happens in these videos."},
      {
        "type": "video_url",
        "video_url": {"url": "https://example.com/sample-video.mp4"}
      },
      {
        "type": "video_url",
        "video_url": {"url": "data:video/mp4;base64,<base64_encoded_data>"}
      }
    ]
  }],
  "max_tokens": 1024
}' \
https://<workspace_host>.databricks.com/ai-gateway/mlflow/v1/chat/completions

音频输入

Python

import os
import base64
from openai import OpenAI

DATABRICKS_TOKEN = os.environ.get('DATABRICKS_TOKEN')
DATABRICKS_BASE_URL = os.environ.get('DATABRICKS_BASE_URL')

client = OpenAI(
    api_key=DATABRICKS_TOKEN,
    base_url=DATABRICKS_BASE_URL
)

# Encode a local audio file as base64
with open("audio.mp3", "rb") as f:
    audio_b64 = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="system.ai.gemini-3-1-pro",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Transcribe this audio and summarize the key points."},
            {
                "type": "audio_url",
                "audio_url": {"url": "https://example.com/sample-audio.mp3"}
            },
            {
                "type": "audio_url",
                "audio_url": {"url": f"data:audio/mp3;base64,{audio_b64}"}
            },
        ]
    }],
    max_tokens=1024
)

print(response.choices[0].message.content)

REST API

curl \
-u token:$DATABRICKS_TOKEN \
-X POST \
-H "Content-Type: application/json" \
-d '{
  "model": "system.ai.gemini-3-1-pro",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Transcribe this audio and summarize the key points."},
      {
        "type": "audio_url",
        "audio_url": {"url": "https://example.com/sample-audio.mp3"}
      },
      {
        "type": "audio_url",
        "audio_url": {"url": "data:audio/mp3;base64,<base64_encoded_data>"}
      }
    ]
  }],
  "max_tokens": 1024
}' \
https://<workspace_host>.databricks.com/ai-gateway/mlflow/v1/chat/completions

谷歌双子座API

使用 Google Gemini API 将媒体作为inlineData数组中的 fileData (base64 编码)或 parts (URL 引用)传递。

视频输入

Python

from google import genai
from google.genai import types
import base64
import os

DATABRICKS_TOKEN = os.environ.get('DATABRICKS_TOKEN')

client = genai.Client(
    api_key="databricks",
    http_options=types.HttpOptions(
        base_url="https://example.staging.cloud.databricks.com/ai-gateway/gemini",
        headers={
            "Authorization": f"Bearer {DATABRICKS_TOKEN}",
        },
    ),
)

# Encode a local video file as base64
with open("video.mp4", "rb") as f:
    video_b64 = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.models.generate_content(
    model="databricks-gemini-3-1-pro",
    contents=[
        types.Content(
            role="user",
            parts=[
                types.Part(text="Summarize what happens in these videos."),
                types.Part(
                    file_data=types.FileData(
                        mime_type="video/mp4",
                        file_uri="https://example.com/sample-video.mp4",
                    )
                ),
                types.Part(
                    inline_data=types.Blob(
                        mime_type="video/mp4",
                        data=video_b64,
                    )
                ),
            ],
        ),
    ],
    config=types.GenerateContentConfig(
        max_output_tokens=1024,
    ),
)

print(response.text)

REST API

curl \
-u token:$DATABRICKS_TOKEN \
-X POST \
-H "Content-Type: application/json" \
-d '{
  "contents": [{
    "role": "user",
    "parts": [
      {"text": "Summarize what happens in these videos."},
      {
        "fileData": {
          "mimeType": "video/mp4",
          "fileUri": "https://example.com/sample-video.mp4"
        }
      },
      {
        "inlineData": {
          "mimeType": "video/mp4",
          "data": "<base64_encoded_data>"
        }
      }
    ]
  }]
}' \
https://<workspace_host>.databricks.com/ai-gateway/gemini/v1beta/models/databricks-gemini-3-1-pro:generateContent

音频输入

Python

from google import genai
from google.genai import types
import base64
import os

DATABRICKS_TOKEN = os.environ.get('DATABRICKS_TOKEN')

client = genai.Client(
    api_key="databricks",
    http_options=types.HttpOptions(
        base_url="https://example.staging.cloud.databricks.com/ai-gateway/gemini",
        headers={
            "Authorization": f"Bearer {DATABRICKS_TOKEN}",
        },
    ),
)

# Encode a local audio file as base64
with open("audio.mp3", "rb") as f:
    audio_b64 = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.models.generate_content(
    model="databricks-gemini-3-1-pro",
    contents=[
        types.Content(
            role="user",
            parts=[
                types.Part(text="Transcribe this audio and summarize the key points."),
                types.Part(
                    file_data=types.FileData(
                        mime_type="audio/mp3",
                        file_uri="https://example.com/sample-audio.mp3",
                    )
                ),
                types.Part(
                    inline_data=types.Blob(
                        mime_type="audio/mp3",
                        data=audio_b64,
                    )
                ),
            ],
        ),
    ],
    config=types.GenerateContentConfig(
        max_output_tokens=1024,
    ),
)

print(response.text)

REST API

curl \
-u token:$DATABRICKS_TOKEN \
-X POST \
-H "Content-Type: application/json" \
-d '{
  "contents": [{
    "role": "user",
    "parts": [
      {"text": "Transcribe this audio and summarize the key points."},
      {
        "fileData": {
          "mimeType": "audio/mp3",
          "fileUri": "https://example.com/sample-audio.mp3"
        }
      },
      {
        "inlineData": {
          "mimeType": "audio/mp3",
          "data": "<base64_encoded_data>"
        }
      }
    ]
  }]
}' \
https://<workspace_host>.databricks.com/ai-gateway/gemini/v1beta/models/databricks-gemini-3-1-pro:generateContent

支持的模型

以下 Gemini 按令牌付费的基础模型支持音频和视频输入。 有关地区可用性,请参阅 Databricks Foundation 模型 API 支持的详细模型列表 。

  • databricks-gemini-3-1-pro
  • databricks-gemini-3-1-flash-lite
  • databricks-gemini-3-flash

局限性

  • 音频和视频输入仅适用于 Gemini 按令牌付费的基础模型。 不支持预配置的吞吐量端点。
  • 单个请求中可以包含多个音频或视频输入,但大型文件会增加延迟和令牌使用量。

其他资源