Transcription and image generation

transcription and image are first-class manifest kinds. They use the same versioning, authentication, content upload, retention, and run APIs as LLM agents.

Transcription

agents/social-transcription.yaml

id: social-transcription
kind: transcription
name: Social narration transcription

model:
  provider: openai
  name: gpt-4o-mini-transcribe
  maxRetries: 2

instruction: |
  Cooking video narration. Preserve ingredient amounts, units,
  temperatures, and times exactly.

settings:
  language: en
  temperature: 0
  timestampGranularities: [segment]

retention:
  mode: none
  artifactTtlSeconds: 3600

The call must contain exactly one audio content block. URLs, base64, existing artifact ids, and local files are accepted; downloaded or uploaded audio is limited to 50 MiB.

const result = await client.agents.run({
  agentId: "social-transcription",
  content: [{
    type: "audio",
    file: { path: "./narration.m4a", mediaType: "audio/mp4" },
  }],
  retention: { mode: "none", artifactTtlSeconds: 3_600 },
});

const transcript = result.output as {
  text: string;
  segments?: unknown[];
  language?: string;
  durationInSeconds?: number;
};

The normalized output contains text, segments, language, and durationInSeconds when supplied by the provider. Token usage is zero when the transcription provider does not report token accounting.

Image generation

agents/generated-recipe-cover.yaml

id: generated-recipe-cover
kind: image
name: Generated recipe cover

model:
  provider: openai
  name: gpt-image-1.5
  maxRetries: 2
  providerOptions:
    openai:
      quality: high
      background: opaque

instruction: Create an appetizing editorial food photograph.
prompt: "Recipe: {{title}}. Main ingredients: {{ingredients}}"

inputSchema:
  type: object
  properties:
    title: { type: string }
    ingredients:
      type: array
      items: { type: string }
  required: [title, ingredients]
  additionalProperties: false

settings:
  n: 1
  size: 1024x1024
  seed: 42

retention:
  mode: result
  ttlSeconds: 86400
  artifactTtlSeconds: 86400

prompt is rendered from structured input. When it is omitted, Agntz uses the string input. instruction is prepended to the rendered prompt. Optional image content blocks become reference images for providers and models that support image-to-image generation.

const result = await client.agents.run({
  agentId: "generated-recipe-cover",
  input: {
    title: "Tomato basil pasta",
    ingredients: ["tomatoes", "basil", "spaghetti"],
  },
  content: [{
    type: "image",
    file: { path: "./style-reference.png", mediaType: "image/png" },
  }],
  retention: { mode: "result", artifactTtlSeconds: 86_400 },
});

const [{ artifactId, mediaType, expiresAt }] =
  (result.output as { artifacts: Array<{
    artifactId: string;
    mediaType: string;
    sizeBytes: number;
    expiresAt: string;
  }> }).artifacts;

const generated = await client.artifacts.download(artifactId);

Generated bytes are never embedded into the JSON response. Each output is a managed artifact with artifactId, mediaType, sizeBytes, and expiresAt.

Image settings support n, maxImagesPerCall, size, aspectRatio, and seed. Provider-specific controls belong under the matching providerOptions key.

Provider support and extension

The built-in adapters currently require provider: openai. Self-hosted workers can supply a HostedOperationRegistry to createWorkerAPI for additional host-level operations. Transcription and image generation are the stable public kinds; embeddings, speech synthesis, moderation, realtime, and batch APIs remain extension points rather than portable manifest contracts.

← Previous

Content, artifacts, retention

Results, streaming, errors