
<img src="/images/ogre-lab.png" alt="Isaac Lab Training Environment" style={{ width: '70%', height: 'auto', display: 'block' }} />

## Training in Isaac Lab

This is Part 4 of the Project Ogre series. In [Part 3](/blog/project-ogre-part3-slam-mapping), we built maps using SLAM. Now we'll train a neural network to optimally control the mecanum wheels using reinforcement learning in NVIDIA Isaac Lab.

<div style={{ padding: '1rem 1.5rem', backgroundColor: '#2a2a2a', borderLeft: '4px solid #ff8c29', borderRadius: '0.375rem', margin: '1.5rem 0', color: '#e0e0e0' }}>
  <strong style={{ color: '#ff8c29' }}>Project Goal:</strong> Train an RL policy that converts velocity commands (vx, vy, vtheta) into optimal wheel velocities, learning the robot's real dynamics.
</div>

<div style={{ padding: '1rem 1.5rem', backgroundColor: '#2a2a2a', borderLeft: '4px solid #ff8c29', borderRadius: '0.375rem', margin: '1.5rem 0', color: '#e0e0e0' }}>
  <strong style={{ color: '#ff8c29' }}>📦 Full Source Code:</strong> Training environment and scripts are available at <a href="https://github.com/protomota/ogre-lab" style={{ color: '#ff8c29', textDecoration: 'underline' }}>github.com/protomota/ogre-lab</a>
</div>

The video below shows Isaac Lab training 4096 parallel instances of the robot simultaneously. Each instance learns from its own experiences, accelerating the training process dramatically compared to single-environment training.

<div style={{ position: 'relative', paddingBottom: '56.25%', height: 0, overflow: 'hidden', maxWidth: '100%', marginTop: '1.5rem' }}>
  <iframe
    style={{ position: 'absolute', top: 0, left: 0, width: '100%', height: '100%' }}
    src="https://www.youtube.com/embed/_a5k1wXDSU8"
    title="Isaac Lab Training Demonstration"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
    allowFullScreen
  />
</div>

## Why Use Reinforcement Learning?

Mecanum wheel kinematics have a clean theoretical model: given a desired body velocity (vx, vy, angular), you can compute the required wheel velocities using inverse kinematics. So why train a neural network?

**The theory doesn't match reality.**

Real robots have:
- Wheel slip on different surfaces
- Motor response delays and non-linearities
- Weight distribution asymmetries
- Mechanical imperfections in wheel alignment

An RL policy learns to compensate for these factors by experiencing thousands of trials in simulation. It discovers that "to go straight forward at 0.3 m/s, the right wheels need slightly higher velocity to compensate for weight imbalance."

The trained policy sits between Nav2's velocity commands and the actual wheel controllers, translating ideal commands into commands that achieve the desired behavior on the real robot.

## Isaac Lab Architecture

Isaac Lab is NVIDIA's GPU-accelerated robot learning framework built on Isaac Sim. Key features:

- **Parallel environments**: Train on 1024+ robots simultaneously
- **GPU physics**: PhysX runs entirely on GPU for massive speedup
- **RSL-RL integration**: Works with the RSL-RL library for PPO training

The training loop runs at ~30Hz control rate with 120Hz physics, training policies in minutes rather than hours.

## The Training Environment

The environment is defined in `ogre_navigation_env.py`. Let's break down the key components:

### Observation Space (10 dimensions)

The policy sees:

<table style={{ width: '100%', borderCollapse: 'separate', borderSpacing: '0' }}>
  <thead>
    <tr>
      <th style={{ textAlign: 'left', padding: '12px 16px', borderBottom: '2px solid #444' }}>Index</th>
      <th style={{ textAlign: 'left', padding: '12px 16px', borderBottom: '2px solid #444' }}>Description</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>0-2</td>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>Target velocity command (vx, vy, vtheta)</td>
    </tr>
    <tr>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>3-5</td>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>Current base velocity (vx, vy, vtheta)</td>
    </tr>
    <tr>
      <td style={{ padding: '12px 16px' }}>6-9</td>
      <td style={{ padding: '12px 16px' }}>Current wheel velocities (fl, fr, rl, rr)</td>
    </tr>
  </tbody>
</table>

This gives the policy enough information to understand both what it should achieve and what the robot is currently doing.

### Action Space (4 dimensions)

The policy outputs wheel velocity targets for all four wheels:
- Front-left (FL)
- Front-right (FR)
- Rear-left (RL)
- Rear-right (RR)

Raw policy outputs are in [-1, 1] and scaled by `action_scale` (8.0 rad/s in the working config) to get actual wheel velocities.

### Reward Function

The policy learns through reward signals:

```python
# Main velocity tracking reward
vel_error = target_vel - current_vel
rew_vel_tracking = 2.0 * exp(-vel_error_norm / 0.25)

# Uprightness reward (+1 upright, -1 flipped)
rew_uprightness = 1.0 * up_z

# Wheel symmetry penalty (prevents veering during straight motion)
rew_symmetry = -1.0 * (left_avg - right_avg)^2
```

The velocity tracking reward uses exponential decay - perfect tracking gives maximum reward, with diminishing returns as error increases. The symmetry penalty discourages the policy from commanding asymmetric wheel velocities when the target is straight-line motion.

### Episode Structure

- **Duration**: 10 seconds per episode
- **Termination**: Episode ends early if robot flips (base height < 0.02m)
- **Reset**: Random target velocities sampled each episode

Random targets force the policy to generalize across the full velocity space rather than memorizing a single trajectory.

## Robot Configuration

The USD robot model must match the real hardware:

```python
OGRE_MECANUM_CFG = ArticulationCfg(
    spawn=sim_utils.UsdFileCfg(
        usd_path="~/ros2_ws/src/ogre-slam/usds/ogre_robot.usd",
    ),
    init_state=ArticulationCfg.InitialStateCfg(
        pos=(0.0, 0.0, 0.04),  # Wheel axle height
    ),
    actuators={
        "wheels": ImplicitActuatorCfg(
            joint_names_expr=["fl_joint", "fr_joint", "rl_joint", "rr_joint"],
            effort_limit=10.0,
            stiffness=0.0,
            damping=10.0,
        ),
    },
)
```

Key parameters:
- `wheel_radius`: 0.040m (40mm)
- `wheelbase`: 0.095m (95mm)
- `track_width`: 0.205m (205mm)

## The Wheel Sign Challenge

One non-obvious issue: the USD model has the right wheels (FR, RR) with opposite joint axis orientation from the left wheels. Sending `[+10, +10, +10, +10]` to all wheels makes the robot spin instead of moving forward.

The solution is sign correction applied in both training and deployment:

**Training (_apply_action):**
```python
corrected_actions = self.actions.clone()
corrected_actions[:, 0] *= -1  # FR (right wheel)
corrected_actions[:, 1] *= -1  # RR (right wheel)
self.robot.set_joint_velocity_target(corrected_actions)
```

**Deployment (Isaac Sim action graph):**
The same correction must be applied when running the trained policy - multiply FR and RR velocities by -1.

This creates a normalized action space where `[+,+,+,+]` always means "all wheels forward" regardless of joint axis conventions.

## Running Training

### Prerequisites

- Isaac Sim 5.0+
- Isaac Lab installed at `~/isaac-lab/IsaacLab`
- Conda environment `env_isaaclab`

### Step 1: Sync the Environment

The training environment must be copied to Isaac Lab's task directory:

```bash
cd ~/ogre-lab
./scripts/sync_env.sh
```

This copies `ogre_navigation/` to Isaac Lab's source tree where it can be discovered and imported.

### Step 2: Run Training

```bash
conda activate env_isaaclab
cd ~/ogre-lab
./scripts/train_ogre_navigation.sh
```

Default settings:
- 4096 parallel environments
- 1000 iterations
- Headless mode (no visualization)

For debugging, run with fewer environments and visualization:

```bash
./scripts/train_ogre_navigation.sh 64 100
```

### Step 3: Monitor with TensorBoard

TensorBoard provides real-time visualization of training progress. In a separate terminal:

```bash
conda activate env_isaaclab
tensorboard --logdir ~/isaac-lab/IsaacLab/logs/rsl_rl/ogre_navigation/
```

Open http://localhost:6006 in your browser:

<img src="/images/tensorboard.png" alt="TensorBoard Training Metrics" style={{ width: '100%', height: 'auto', display: 'block', margin: '1.5rem 0' }} />

**Key metrics to watch:**

- **Train/mean_reward**: The average reward across all parallel environments. Should steadily increase as the policy learns. A healthy training run shows rewards climbing from near-zero to positive values.

- **Train/mean_episode_length**: How long episodes last before termination. Should approach the maximum (300 steps = 10 seconds at 30Hz). If episodes are short, robots are flipping or hitting termination conditions.

- **Loss/value_function**: The critic's prediction error. Should decrease over time as the value network learns to estimate expected returns. Spikes are normal early in training.

- **Loss/surrogate**: The PPO policy loss. Fluctuates during training but should generally trend downward.

TensorBoard is invaluable for diagnosing training issues. If rewards plateau, you might need to adjust reward weights. If episode lengths stay low, the action scale might be too aggressive.

### What to Expect

```
Learning iteration 50/100
  Mean reward: 1.25        # Positive and increasing = good
  Mean episode length: 299  # Near 300 = robots staying upright
```

If rewards are very negative (-20000), the velocity targets are too aggressive - reduce `max_lin_vel` and `max_ang_vel`. If episode lengths are short, robots are flipping - reduce `action_scale`.

## Exporting the Trained Policy

Once training completes, export the model for deployment:

```bash
cd ~/ogre-lab
./scripts/export_policy.sh
```

This runs Isaac Lab's `play.py` to verify the policy works, then exports:
- `policy.onnx` - For ONNX runtime inference
- `policy.pt` - PyTorch JIT format

The exported models go to the training run's `exported/` directory.

## Deploying to ROS2

Copy the trained model to the ROS2 workspace:

```bash
./scripts/deploy_model.sh --rebuild
```

This copies the latest exported model to `ogre-slam/ogre_policy_controller/models/` and rebuilds the ROS2 package.

### Testing in Isaac Sim

**Terminal 1: Isaac Sim**
- Load `ogre.usd`
- Press Play

**Terminal 2: Policy Controller**
```bash
conda deactivate  # Exit conda - ROS2 uses system Python
source ~/ros2_ws/install/setup.bash
ros2 launch ogre_policy_controller policy_controller.launch.py
```

**Terminal 3: Test Commands**
```bash
# Forward motion (0.3 m/s)
ros2 topic pub /cmd_vel_smoothed geometry_msgs/msg/Twist \
  "{linear: {x: 0.3, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}" -r 10

# Strafe left
ros2 topic pub /cmd_vel_smoothed geometry_msgs/msg/Twist \
  "{linear: {x: 0.0, y: 0.3, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}" -r 10

# Rotate
ros2 topic pub /cmd_vel_smoothed geometry_msgs/msg/Twist \
  "{linear: {x: 0.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 1.0}}" -r 10
```

Watch the robot in Isaac Sim. It should move smoothly in the commanded direction.

## The Data Flow

```
Nav2 Planner → /cmd_vel → Policy Controller → /joint_command → Isaac Sim/Robot
                               ↓
                         policy.onnx
                               ↓
                    [FL, FR, RL, RR] wheel velocities
```

The policy controller subscribes to `/cmd_vel_smoothed`, runs the velocity command through the neural network, and publishes individual wheel velocities to `/joint_command`.

## Troubleshooting Training

### Robots Exploding/Flying

**Cause**: Physics instability, usually from collision issues or excessive forces

**Solutions**:
- Check USD model has proper collision meshes
- Reduce `action_scale` (try 6.0 instead of 8.0)
- Increase physics solver iterations

### Very Negative Rewards

**Cause**: Velocity targets exceed what the robot can achieve

**Solutions**:
- Reduce `max_lin_vel` and `max_ang_vel`
- Check `action_scale` allows sufficient wheel velocity

### Short Episode Lengths

**Cause**: Robots flipping over

**Solutions**:
- Reduce `action_scale`
- Add stronger uprightness reward
- Check wheel friction parameters

### Policy Doesn't Generalize

**Cause**: Training converged to local minimum

**Solutions**:
- Increase training iterations
- Add curriculum learning (start easy, increase difficulty)
- Tune reward scales

---

## Quick Reference

### Key Files

<table style={{ width: '100%', borderCollapse: 'separate', borderSpacing: '0' }}>
  <thead>
    <tr>
      <th style={{ textAlign: 'left', padding: '12px 16px', borderBottom: '2px solid #444' }}>File</th>
      <th style={{ textAlign: 'left', padding: '12px 16px', borderBottom: '2px solid #444' }}>Location</th>
      <th style={{ textAlign: 'left', padding: '12px 16px', borderBottom: '2px solid #444' }}>Purpose</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}><code>ogre_navigation_env.py</code></td>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>ogre-lab/isaaclab_env/</td>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>Training environment</td>
    </tr>
    <tr>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}><code>rsl_rl_ppo_cfg.py</code></td>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>ogre-lab/isaaclab_env/agents/</td>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>PPO hyperparameters</td>
    </tr>
    <tr>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}><code>policy.onnx</code></td>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>ogre-slam/ogre_policy_controller/models/</td>
      <td style={{ padding: '12px 16px', borderBottom: '1px solid #333' }}>Deployed model</td>
    </tr>
    <tr>
      <td style={{ padding: '12px 16px' }}><code>policy_controller_node.py</code></td>
      <td style={{ padding: '12px 16px' }}>ogre-slam/ogre_policy_controller/</td>
      <td style={{ padding: '12px 16px' }}>ROS2 inference node</td>
    </tr>
  </tbody>
</table>

### Commands

```bash
# Train policy
./scripts/train_ogre_navigation.sh

# Export trained model
./scripts/export_policy.sh

# Deploy to ROS2
./scripts/deploy_model.sh --rebuild

# Run policy controller
ros2 launch ogre_policy_controller policy_controller.launch.py
```

## What's Next

With a trained policy deployed, [Part 5](/blog/project-ogre-part5-nav2-navigation) brings everything together with Nav2 navigation. The policy controller integrates seamlessly - Nav2 plans paths and generates velocity commands, the RL policy executes them optimally, and the robot navigates autonomously through the mapped environment.


Using reinforcement learning in NVIDIA Isaac Lab to train a neural network policy for optimal mecanum wheel velocity control.

Posts tagged "reinforcement-learning"

Project Ogre Part 4: Training a Mecanum Controller in Isaac Lab