AI Agents

One of Synthome's most powerful features is the ability to let AI agents generate execution plans. This enables dynamic, content-aware video creation without hardcoding pipeline logic.

Quick Start with @synthome/ai

The easiest way to integrate Synthome with AI agents is using the @synthome/ai package, designed to work seamlessly with the Vercel AI SDK:

npm install @synthome/ai @synthome/sdk ai @ai-sdk/openai

Basic Usage with Vercel AI SDK

import { executePlanTool, planSystemPrompt } from "@synthome/ai";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

const result = await generateText({
  model: openai("gpt-4o"),
  system: planSystemPrompt,
  tools: {
    executePlan: executePlanTool({
      apiKey: process.env.SYNTHOME_API_KEY,
    }),
  },
  prompt: "Create a video of a sunset over mountains",
});

The planSystemPrompt teaches the AI how to generate valid execution plans, and executePlanTool handles validation and execution.

Available Tools

executePlanTool

Generates and executes media workflows. The AI creates an ExecutionPlan and runs it immediately.

import { executePlanTool, planSystemPrompt } from "@synthome/ai";

const tool = executePlanTool({
  apiKey: process.env.SYNTHOME_API_KEY,
  // Optional: webhook for async execution
  webhook: "https://myapp.com/webhook",
  webhookSecret: process.env.WEBHOOK_SECRET,
  // Optional: provider API keys
  providerApiKeys: {
    replicate: process.env.REPLICATE_API_KEY,
    fal: process.env.FAL_KEY,
  },
});

buildPlanTool

Generates workflow plans without executing them. Use this to design workflows for later execution.

import { buildPlanTool, planSystemPrompt } from "@synthome/ai";
import { executeFromPlan } from "@synthome/sdk";

const result = await generateText({
  model: openai("gpt-4o"),
  system: planSystemPrompt,
  tools: {
    buildPlan: buildPlanTool(),
  },
  prompt: "Design a workflow for creating a product video",
});

// Save the plan for later
const plan = result.toolResults[0].result.plan;
await db.savePlan("product-video", plan);

// Execute whenever you want
const execution = await executeFromPlan(plan, {
  apiKey: process.env.SYNTHOME_API_KEY,
});

Using Both Tools

Let the AI decide whether to execute immediately or build for later:

const result = await generateText({
  model: openai("gpt-4o"),
  system: planSystemPrompt,
  tools: {
    executePlan: executePlanTool({ apiKey: process.env.SYNTHOME_API_KEY }),
    buildPlan: buildPlanTool(),
  },
  prompt: "Create a video now",
  // AI uses executePlan for immediate execution
  // or buildPlan if user says "design for later"
});

The Execution Plan Format

An execution plan is a JSON object with a jobs array:

interface ExecutionPlan {
  jobs: JobNode[];
  baseExecutionId?: string;
}

interface JobNode {
  id: string; // Unique job identifier
  type: OperationType; // "generate", "generateImage", "merge", etc.
  params: Record<string, unknown>; // Job-specific parameters
  dependsOn?: string[]; // IDs of jobs this depends on
  output: string; // Output reference key
}

Example Plan

{
  "jobs": [
    {
      "id": "scene-1",
      "type": "generate",
      "params": {
        "modelId": "bytedance/seedance-1-pro",
        "provider": "replicate",
        "prompt": "A rocket launching into space, cinematic"
      },
      "output": "$scene-1"
    },
    {
      "id": "scene-2",
      "type": "generate",
      "params": {
        "modelId": "bytedance/seedance-1-pro",
        "provider": "replicate",
        "prompt": "Earth from orbit, blue planet"
      },
      "output": "$scene-2"
    },
    {
      "id": "final",
      "type": "merge",
      "params": {
        "items": [
          { "type": "video", "url": "_jobDependency:scene-1" },
          { "type": "video", "url": "_jobDependency:scene-2" }
        ]
      },
      "dependsOn": ["scene-1", "scene-2"],
      "output": "$final"
    }
  ]
}

Direct Usage with executeFromPlan

If you prefer manual control, use executeFromPlan() directly:

import { executeFromPlan } from "@synthome/sdk";

const plan = {
  jobs: [
    {
      id: "image-1",
      type: "generateImage",
      params: {
        modelId: "google/nano-banana",
        provider: "fal",
        prompt: "A futuristic city at night",
      },
      output: "$image-1",
    },
  ],
};

const execution = await executeFromPlan(plan);
console.log("Result:", execution.result?.url);

With Options

const execution = await executeFromPlan(plan, {
  apiKey: "your-synthome-api-key",
  webhook: "https://your-server.com/webhook",
  webhookSecret: "your-secret",
  providerApiKeys: {
    replicate: "your-replicate-key",
    fal: "your-fal-key",
  },
});

Custom AI Integration

If you're not using the Vercel AI SDK, you can use the schemas and prompt directly:

Using Zod Schemas

import { executionPlanSchema, planSystemPrompt } from "@synthome/ai";
import { executeFromPlan } from "@synthome/sdk";

// Use planSystemPrompt as your system message
const systemMessage = planSystemPrompt;

// Validate AI output with the schema
const result = executionPlanSchema.safeParse(aiGeneratedPlan);
if (!result.success) {
  console.error("Invalid plan:", result.error);
  return;
}

// Execute the validated plan
const execution = await executeFromPlan(result.data);

With OpenAI Function Calling

import { executionPlanSchema, planSystemPrompt } from "@synthome/ai";
import { zodToJsonSchema } from "zod-to-json-schema";

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [
    { role: "system", content: planSystemPrompt },
    { role: "user", content: "Create a video of ocean waves" },
  ],
  tools: [
    {
      type: "function",
      function: {
        name: "execute_plan",
        description: "Execute a media workflow",
        parameters: zodToJsonSchema(executionPlanSchema),
      },
    },
  ],
});

Converting Pipelines to Plans

You can also create a pipeline with compose() and export it:

import { compose, generateVideo, merge, videoModel } from "@synthome/sdk";

// Create pipeline programmatically
const pipeline = compose(
  merge([
    generateVideo({
      model: videoModel("bytedance/seedance-1-pro", "replicate"),
      prompt: "Scene 1: Ocean waves",
    }),
    generateVideo({
      model: videoModel("bytedance/seedance-1-pro", "replicate"),
      prompt: "Scene 2: Beach sunset",
    }),
  ]),
);

// Export as JSON
const plan = pipeline.toJSON();
console.log(JSON.stringify(plan, null, 2));

// Modify the plan if needed
plan.jobs[0].params.prompt = "Scene 1: Calm ocean at dawn";

// Execute the modified plan
const execution = await executeFromPlan(plan);

Validation

Both executePlanTool and executeFromPlan() validate plans before execution:

Plan must have a jobs array
Each job must have id, type, and params
At least one job must exist

Invalid plans throw descriptive errors:

try {
  await executeFromPlan({ jobs: [] });
} catch (error) {
  console.error(error.message);
  // "ExecutionPlan must contain at least one job"
}

Best Practices

1. Use the AI Package

The @synthome/ai package handles schema validation, system prompts, and tool integration automatically.

2. Use Webhooks for Long Jobs

Video generation can take time. Use webhooks for non-blocking execution:

executePlanTool({
  apiKey: process.env.SYNTHOME_API_KEY,
  webhook: "https://myapp.com/webhook",
});

3. Handle Errors Gracefully

try {
  const execution = await executeFromPlan(plan);
  console.log("Success:", execution.result?.url);
} catch (error) {
  if (error.message.includes("ExecutionPlan")) {
    // Plan validation error - ask AI to regenerate
    console.error("Invalid plan:", error.message);
  } else {
    // Execution error
    console.error("Execution failed:", error.message);
  }
}

Use Cases

Content generation at scale: Let AI decide scene composition based on content
Dynamic video creation: Generate videos based on user input or data
A/B testing: Generate variations by modifying plans
Workflow automation: Store and replay execution plans
Multi-tenant applications: Generate plans per-user with different parameters

AI Agents

On this page