camera-model

What this skill does

# Camera Model Reference

## Overview

Our camera system is implemented in JAX and follows OpenCV conventions. There are two layers:

1. **Core library** (`multi_camera.analysis.camera`) - Low-level functions operating in **millimeters**
2. **Camera adapter** (`multiview_biomechanical_pose.geometry.camera_adapter`) - High-level functions accepting/returning **meters**

**Source files:**
- `MultiCameraTracking/multi_camera/analysis/camera.py` - Core camera functions
- `MultiviewBiomechanicalPose/multiview_biomechanical_pose/geometry/camera_adapter.py` - Meter-safe wrappers

---

## Camera Parameters Dict

All camera functions expect a single dict with parameters for ALL cameras:

```python
camera_params = {
    "mtx":  array,  # (num_cameras, 4)  - [fx, fy, cx, cy] stored at 1/1000 scale
    "rvec": array,  # (num_cameras, 3)  - Rodrigues rotation vectors (dimensionless)
    "tvec": array,  # (num_cameras, 3)  - Translation vectors in METERS (stored at 1/1000 scale)
    "dist": array,  # (num_cameras, 5+) - Distortion coefficients [k1, k2, p1, p2, k3, ...]
}
```

The `mtx` and `tvec` fields are stored at **1/1000 scale** (effectively meters). The library functions `get_intrinsic()` and `get_extrinsic()` multiply by 1000 internally to produce pixel-scale intrinsics and mm-scale extrinsics.

**TFRecord format** uses plural names (`camera_intrinsics`, `camera_rvecs`, `camera_tvecs`, `camera_distortion`) which the dataset pipeline converts to singular names automatically.

---

## Unit Conventions (CRITICAL)

| Context | Units | Notes |
|---------|-------|-------|
| 3D points throughout the system | **meters** | World coords, camera coords, joint positions |
| `camera_params["tvec"]` storage | **meters** (1/1000 scale) | Scaled up by 1000 inside `get_extrinsic()` |
| `camera_params["mtx"]` storage | **1/1000 of pixels** | Scaled up by 1000 inside `get_intrinsic()` |
| `get_extrinsic()` returns | 4x4 matrix in **mm** | Translation component is in millimeters |
| `get_intrinsic()` returns | 3x3 K matrix in **pixels** | fx, fy, cx, cy in pixel units |
| `project_distortion()` input | 3D points in **mm** | You MUST multiply meters by 1000 |
| `project_distortion()` output | 2D points in **pixels** | |
| `triangulate_point()` input | 2D points in **pixels** | With optional confidence channel |
| `triangulate_point()` output | 3D points in **mm** | Divide by 1000 to get meters |
| `camera_adapter` functions | **meters** in/out | Handle mm conversion internally |
| 2D keypoints | **pixels** | Original image resolution |

### The Conversion Rule

When calling core library functions directly, always convert:
```python
# Meters -> mm before projection
points_2d = cam_lib.project_distortion(camera_params, cam_idx, points_3d_m * 1000.0)

# mm -> meters after triangulation
points_3d_m = triangulate_point(camera_params, points_2d) / 1000.0
```

When using `camera_adapter` functions, no conversion is needed - they handle it internally.

---

## OpenCV Extrinsic Convention

Our camera model follows the **OpenCV world-to-camera** convention:

```
P_camera = E @ P_world
```

- `get_extrinsic()` returns a **4x4 SE(3) matrix** that transforms points FROM world coordinates TO camera coordinates
- This is **NOT** the camera position in world space
- The camera position in world space would be `-R^T @ t` (or `E_inv[:3, 3]`)
- Rotation is stored as Rodrigues vectors, converted via `SO3.exp(rvec)`
- Uses `jaxlie` SE3/SO3 for all rigid body transformations

```python
# get_extrinsic returns world-to-camera transform (units: mm)
E = get_extrinsic(camera_params, i)  # 4x4 matrix

# To get camera position in world (in mm):
camera_pos_mm = jnp.linalg.inv(E)[:3, 3]

# To get camera position in meters:
camera_pos_m = camera_pos_mm / 1000.0
```

### COLMAP Conversion Warning

COLMAP stores camera-to-world transforms. When converting from COLMAP:
```python
R_w2c = R_c2w.T
t_w2c = -R_w2c @ t_colmap
```

---

## Core Library Functions (`multi_camera.analysis.camera`)

All core functions operate in **millimeters**. Import as:

```python
from multi_camera.analysis import camera as cam_lib
from multi_camera.analysis.camera import get_intrinsic, get_extrinsic
```

### `get_intrinsic(camera_params, i)` -> 3x3 K matrix

Returns the intrinsic matrix with focal lengths and principal point in **pixel units**:
```python
K = get_intrinsic(camera_params, 0)
# [[fx,  0, cx],
#  [ 0, fy, cy],
#  [ 0,  0,  1]]
```

### `get_extrinsic(camera_params, i)` -> 4x4 SE(3) matrix

Returns the **world-to-camera** transformation matrix in **millimeters**:
```python
E = get_extrinsic(camera_params, 0)
# 4x4 homogeneous transform [R|t; 0 0 0 1]
# t is in MILLIMETERS
# This is NOT the camera position - it's a world-to-camera transform
```

### `get_projection(camera_params, i)` -> 3x4 projection matrix

Returns `K @ E[:3]` - the full 3x4 projection matrix:
```python
P = get_projection(camera_params, 0)  # 3x4
```

### `project_distortion(camera_params, i, points)` -> 2D pixels

Projects 3D points to 2D image coordinates with radial distortion:
```python
# CRITICAL: points must be in MILLIMETERS
points_2d = cam_lib.project_distortion(camera_params, cam_idx, points_3d_m * 1000.0)
```

Currently uses simplified radial model (k1 only). Handles behind-camera points by shrinking them toward image center.

### `undistort_points(points, K, dist, num_iters=5)` -> undistorted 2D

Iteratively removes lens distortion from 2D image points. Supports full OpenCV model (k1-k6, p1-p2, s1-s4, tau_x, tau_y):
```python
undistorted = cam_lib.undistort_points(
    points_2d[None, ...],   # (1, N, 2) - needs batch dim
    K[None, ...],           # (1, 3, 3)
    dist_coeffs[None, :],   # (1, n_coeffs)
)[0]                        # remove batch dim
```

### `distort_3d(camera_params, i, points)` -> distorted 3D in camera coords

Computes equivalent 3D camera-frame points that would project to the same 2D coordinates under an ideal pinhole camera. Used for rendering 3D models (e.g., SMPL) on distorted images:
```python
# points in mm, returns camera-frame coords in mm
vertices_distorted = cam_lib.distort_3d(camera_params, i, vertices_mm)
```

Uses full radial+tangential model (k1, k2, k3, p1, p2).

### `triangulate_point(camera_params, points2d, return_confidence=False)` -> 3D mm

DLT triangulation from multiple 2D observations. Handles undistortion internally:
```python
# points2d: (num_cameras, T, J, 3) with [u, v, confidence]
# Returns: (T, J, 3) or (T, J, 4) with confidence, in MILLIMETERS
points_3d_mm = cam_lib.triangulate_point(camera_params, points2d, return_confidence=True)
points_3d_m = points_3d_mm[..., :3] / 1000.0
```

### `robust_triangulate_points(camera_params, points2d, sigma=150, threshold=0.5)` -> 3D mm

Robust triangulation using geometric median of pairwise triangulations (Roy et al. 2022). Automatically weights cameras by agreement:
```python
points_3d, camera_weights = cam_lib.robust_triangulate_points(
    camera_params, points2d, return_weights=True
)
# points_3d: (T, J, 4) in mm
# camera_weights: (num_cameras, T, J)
```

### `reprojection_error(camera_params, points2d, points3d)` -> residual in pixels

Computes difference between observed and projected 2D points:
```python
# points3d must be in mm (same space as projection)
residual = cam_lib.reprojection_error(camera_params, points2d, points3d_mm)
```

---

## Camera Adapter Functions (Meter-Safe API)

These functions accept and return **meters**. Prefer these over direct core library calls when possible.

```python
from multiview_biomechanical_pose.geometry.camera_adapter import (
    transform_points_world_to_camera,
    transform_points_camera_to_world,
    transform_points_between_cameras,
    project_camera_coordinates,
    unproject_from_image_with_distortion,
    unproject_and_transform_to_world,
    transform_pose_se3_world_to_camera,
    transform_pose_se3_camera_to_world,
    compute_relative_extrinsics,
)
```

### Point Transformations

```python
# World