> ## Documentation Index > Fetch the complete documentation index at: https://openrewardstandard.io/llms.txt > Use this file to discover all available pages before exploring further. # Rewards > How rewards work in ORS A reward signal defines the goal in a reinforcement learning problem. Typically this is a single number (scalar feedback) that tells an agent how good or bad an outcome is in an immediate sense. In ORS, rewards are obtained by executing tools and are returned as part of a `ToolOutput`. They can be used for training in reinforcement learning, or for agentic evaluation - for example, binary reward can be used for computing accuracy. ## What is a Reward? A reward is a **numeric signal** (typically a float) that indicates how good an outcome was: ```typescript theme={null} interface ToolOutput { blocks: Blocks reward?: number // ← The reward signal finished: boolean metadata?: JSONObject } ``` **Characteristics**: * Optional field (can be null) * Often normalised in the range \[-1, 1] or \[0, 1] * Returned with every tool call * Accumulated over an episode ### Reward Examples **Positive reward** (success): ```json theme={null} { "blocks": [{"text": "Correct!", "detail": null, "type": "text"}], "metadata": null, "reward": 1.0, "finished": true } ``` **Zero reward** (neutral): ```json theme={null} { "blocks": [{"text": "File listed successfully", "detail": null, "type": "text"}], "metadata": null, "reward": 0.0, "finished": false } ``` **Negative reward** (penalty): ```json theme={null} { "blocks": [{"text": "Error: Command failed", "detail": null, "type": "text"}], "metadata": null, "reward": -0.1, "finished": false } ``` ## Reward Design Patterns ### Pattern 1: Sparse Rewards **Only reward at episode end**: ```python theme={null} def submit(self, params) -> ToolOutput: correct = (params.answer == self.task["answer"]) return ToolOutput( blocks=[TextBlock(text="Correct!" if correct else "Incorrect")], reward=1.0 if correct else 0.0, finished=True ) ``` **All intermediate steps**: ```python theme={null} def bash(self, params) -> ToolOutput: output = execute_command(params.command) return ToolOutput( blocks=[TextBlock(text=output)], reward=0.0, # No reward for intermediate steps finished=False ) ``` **Pros**: * Simple to implement * Clear success/failure signal * Easy to understand **Cons**: * Hard for agent to learn (delayed feedback) * Credit assignment problem * Slow learning **Best for**: Simple tasks, short-horizon tasks *** ### Pattern 2: Dense Rewards **Reward every meaningful action**: ```python theme={null} def bash(self, params) -> ToolOutput: output = execute_command(params.command) # Reward progress reward = 0.0 if "answer.txt" in output: reward = 0.2 # Found relevant file if len(output) > 0: reward += 0.1 # Successful command return ToolOutput( blocks=[TextBlock(text=output)], reward=reward, finished=False ) def submit(self, params) -> ToolOutput: correct = (params.answer == self.task["answer"]) return ToolOutput( blocks=[TextBlock(text="Correct!" if correct else "Incorrect")], reward=1.0 if correct else 0.0, # Final reward finished=True ) ``` **Pros**: * Faster learning * Guides agent toward solution * Better credit assignment **Cons**: * Harder to design * Can bias agent toward suboptimal paths * Risk of reward hacking **Best for**: Complex tasks, long-horizon episodes *** ### Pattern 3: Shaped Rewards **Reward based on distance to goal**: ```python theme={null} def calculate_progress_reward(self, state) -> float: """Reward based on how close to solution""" # Example: Math problem solving # Check if agent has seen the problem if not state["viewed_problem"]: return 0.0 # Check if agent has attempted calculation if state["attempted_calculation"]: reward = 0.3 # Check if calculation is close to answer if state["last_guess"]: error = abs(state["last_guess"] - self.task["answer"]) if error < 5: reward = 0.7 # Getting close! if error < 1: reward = 0.9 # Very close! return reward return 0.1 # Viewed problem ``` **Pros**: * Strong learning signal * Guides exploration * Accelerates training **Cons**: * Requires domain knowledge * Can create reward hacking opportunities * Complex to implement **Best for**: Well-understood domains, complex navigation *** ### Pattern 4: Penalty-Based Rewards **Penalize bad actions**: ```python theme={null} def bash(self, params) -> ToolOutput: # Penalize dangerous commands dangerous = ["rm -rf", ":(){ :|:& };:", "dd if=/dev/zero"] if any(cmd in params.command for cmd in dangerous): return ToolOutput( blocks=[TextBlock(text="Error: Dangerous command blocked")], reward=-1.0, # Large penalty finished=True # End episode ) # Penalize inefficient actions if self.step_count > 20: reward = -0.1 # Penalty for taking too long else: reward = 0.0 output = execute_command(params.command) return ToolOutput( blocks=[TextBlock(text=output)], reward=reward, finished=False ) ``` **Use cases**: * Safety constraints * Efficiency requirements * Avoiding bad behaviors *** ### Pattern 5: Binary Rewards **Simple success/failure**: ```python theme={null} def submit(self, params) -> ToolOutput: correct = (params.answer == self.task["answer"]) return ToolOutput( blocks=[TextBlock(text="Correct!" if correct else "Incorrect")], reward=1.0 if correct else 0.0, # Binary: 1 or 0 finished=True ) ``` **Best for**: Classification, Q\&A, simple decisions ## Reward Scales ### Common Reward Ranges **\[0, 1] scale**: * 0 = failure * 1 = perfect success * 0.5 = partial success **\[-1, 1] scale**: * -1 = worst outcome * 0 = neutral * +1 = best outcome **Custom scales**: * Can use any range, but normalise for RL algorithms * Some RL algorithms normalise automatically; e.g. group advantage in GRPO ## Cumulative Rewards In RL, we often care about **total (undiscounted) return**: ```python theme={null} total_return = 0.0 # Episode loop while not finished: ... result = session.call_tool(...) total_return += result.reward or 0.0 finished = result.finished print(f"Total return: {total_return}") ``` **Interpretation**: * Higher total return = better performance * Compare across episodes for learning progress * Use for evaluation metrics ## Rewards for Evaluation Rewards in ORS can be used for evaluation as well as training: ```python theme={null} # Run evaluation total_rewards = [] for task in test_tasks: session = create_session(task) episode_reward = 0.0 while not finished: result = agent.step(session) episode_reward += result.reward or 0.0 finished = result.finished total_rewards.append(episode_reward) # Evaluation metrics avg_reward = sum(total_rewards) / len(total_rewards) success_rate = sum(1 for r in total_rewards if r > 0.9) / len(total_rewards) print(f"Average reward: {avg_reward:.3f}") print(f"Success rate: {success_rate:.1%}") ``` ## Next Steps Design tools that return rewards Organize tasks for RL training Understand episode lifecycle and reward accumulation Build an ORS server with reward logic *** **Key Takeaway**: Rewards are the learning signal for RL. Design them carefully to align with your true objective, provide timely feedback, and avoid unintended behaviors. Good reward design is critical for successful RL training.