Skip to main content
Prompts are the initial instructions given to agents at the start of an episode. They provide the context and goal for the task, forming the agent’s initial observation.

What is a Prompt?

A prompt is a sequence of blocks (text and/or images) that describes the task to the agent:
Purpose:
  • Tell the agent what problem to solve
  • Provide initial context
  • Set expectations for the episode

Accessing Prompts

Prompts are retrieved via the API after creating a session:
Response:
Typical flow:
  1. Create session with task
  2. Get prompt for that task
  3. Agent reads prompt to understand what to do
  4. Agent begins calling tools to solve task

Prompt Structure

Text Prompts

Simple text instructions:

Multi-Line Prompts

Complex instructions:

Multi-Modal Prompts

Text + images:

Generating Prompts

Prompts are generated by an environment’s get_prompt() method. For example, in the Python SDK:
Key points:
  • Prompts are task-specific
  • Use self.task_spec to access task data
  • Return list of blocks (even for single text)

Prompt Design Patterns

Pattern 1: Simple Question

Direct question from task:
Example:
“What is the capital of France?”

Pattern 2: Contextual Instructions

Add context and instructions:
Note the tool instructions may be redundant if the available are already passed into the agent’s context.

Pattern 3: Role-Playing

Set agent persona:

Pattern 4: Multi-Modal

Include images:

Pattern 5: Few-Shot Examples

Provide examples in prompt:

Dynamic Prompts

Prompts can be customized based on task properties:

Next Steps

Tools

Design tools agents use after reading prompts

Tasks & Splits

Organize tasks that prompts are generated from

Implementing a Server

Implement get_prompt() in your environment

Data Types

See prompt data structure (Blocks)

Key Takeaway: Prompts are the agent’s starting point for each episode. Design them to be clear, specific, and informative. Good prompts guide agents toward successful task completion by setting context.