Skip to main content
Prompts are the initial instructions given to agents at the start of an episode. They provide the context and goal for the task, forming the agent’s initial observation.

​
What is a Prompt?

A prompt is a sequence of blocks (text and/or images) that describes the task to the agent: 
type Prompt = Blocks  // Array of TextBlock | ImageBlock
Purpose:
  • Tell the agent what problem to solve
  • Provide initial context
  • Set expectations for the episode

​
Accessing Prompts

Prompts are retrieved via the API after creating a session: 
GET /gsm8k/prompt
X-Session-ID: abc-123
Response: 
[
  {
    "text": "What is 2+2?",
    "detail": null,
    "type": "text"
  }
]
Typical flow:
  1. Create session with task
  2. Get prompt for that task
  3. Agent reads prompt to understand what to do
  4. Agent begins calling tools to solve task

​
Prompt Structure

​
Text Prompts

Simple text instructions: 
[
  {
    "text": "Solve this math problem: If x + 5 = 12, what is x?",
    "detail": null,
    "type": "text"
  }
]

​
Multi-Line Prompts

Complex instructions: 
[
  {
    "text": "You are given a Python programming challenge.\n\nWrite a function that reverses a string.\n\nRequirements:\n-Function name: reverse_string\n-Input: string\n-Output: reversed string\n\nYou have access to a Python REPL. Submit your function when ready.",
    "detail": null,
    "type": "text"
  }
]

​
Multi-Modal Prompts

Text + images: 
[
  {
    "text": "What object is shown in this image?",
    "detail": null,
    "type": "text"
  },
  {
    "data": "iVBORw0KGgoAAAANSUhEUgA...",
    "mimeType": "image/jpeg",
    "detail": null,
    "type": "image"
  }
]

​
Generating Prompts

Prompts are generated by an environment’s get_prompt() method. For example, in the Python SDK: 
from ors import Environment, tool, Blocks, TextBlock, ToolOutput

class MathEnvironment(Environment):
    def get_prompt(self) -> Blocks:
        """Generate prompt from task"""
        question = self.task_spec["question"]

        return [
            TextBlock(
                text=f"Solve this math problem: {question}",
                type="text"
            )
        ]
Key points:
  • Prompts are task-specific
  • Use self.task_spec to access task data
  • Return list of blocks (even for single text)

​
Prompt Design Patterns

​
Pattern 1: Simple Question

Direct question from task: 
def get_prompt(self) -> Blocks:
    return [TextBlock(text=self.task_spec["question"])]
Example:
“What is the capital of France?”

​
Pattern 2: Contextual Instructions

Add context and instructions: 
def get_prompt(self) -> Blocks:
    problem = self.task_spec["problem"]

    prompt_text = f"""You are a coding assistant. Solve the following problem:

{problem}

You have access to a Python REPL. Use the 'python' tool to write and test code.
When you have a solution, use the 'submit' tool.
"""

    return [TextBlock(text=prompt_text)]
Note the tool instructions may be redundant if the available are already passed into the agent’s context.

​
Pattern 3: Role-Playing

Set agent persona: 
def get_prompt(self) -> Blocks:
    scenario = self.task_spec["scenario"]

    prompt_text = f"""You are a customer service agent.

A customer has the following issue:
{scenario}

Your goal is to resolve their issue professionally and efficiently.
Use the 'respond' tool to communicate with the customer.
"""

    return [TextBlock(text=prompt_text)]

​
Pattern 4: Multi-Modal

Include images: 
import base64

def get_prompt(self) -> Blocks:
    question = self.task_spec["question"]
    image_path = self.task_spec["image_path"]

    # Load and encode image
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    return [
        TextBlock(text=question),
        ImageBlock(
            data=image_data,
            mimeType="image/png",
            type="image"
        )
    ]

​
Pattern 5: Few-Shot Examples

Provide examples in prompt: 
def get_prompt(self) -> Blocks:
    problem = self.task_spec["problem"]

    prompt_text = f"""Solve the following math problem.

Examples:
-Problem: "2 + 2" → Answer: 4
-Problem: "10 - 3" → Answer: 7
-Problem: "5 * 6" → Answer: 30

Now solve:
Problem: "{problem}"

Use the 'submit' tool with your answer.
"""

    return [TextBlock(text=prompt_text)]

​
Dynamic Prompts

Prompts can be customized based on task properties: 
def get_prompt(self) -> Blocks:
    difficulty = self.task_spec.get("difficulty", "medium")

    if difficulty == "easy":
        instructions = "This is a simple problem. Take your time."
    elif difficulty == "hard":
        instructions = "This is a challenging problem. Think carefully and use all available tools."
    else:
        instructions = "Solve this problem step by step."

    problem = self.task_spec["problem"]

    return [TextBlock(text=f"{instructions}\n\nProblem: {problem}")]

​
Next Steps

Tools

Design tools agents use after reading prompts

Tasks & Splits

Organize tasks that prompts are generated from

Implementing a Server

Implement get_prompt() in your environment

Data Types

See prompt data structure (Blocks)
Key Takeaway: Prompts are the agent’s starting point for each episode. Design them to be clear, specific, and informative. Good prompts guide agents toward successful task completion by setting context.