pixi-package-manager

What this skill does

# Pixi Package Manager for Scientific Python

Master **pixi**, the modern package manager that unifies conda and PyPI ecosystems for fast, reproducible scientific Python development. Learn how to manage complex scientific dependencies, create isolated environments, and build reproducible workflows using `pyproject.toml` integration.

**Official Documentation**: https://pixi.sh
**GitHub**: https://github.com/prefix-dev/pixi

## Quick Reference Card

### Setup
```bash
# Installation must be performed separately
# On the server, load via lmod if not already in path
module load Dev/pixi

# Initialize new project with pyproject.toml
pixi init --format pyproject

# Initialize existing Python project
pixi init --format pyproject --import-environment
```

### Essential Commands
```bash
# Add dependencies
pixi add numpy scipy pandas              # conda packages
pixi add --pypi pytest-cov               # PyPI-only packages
pixi add --feature dev pytest ruff       # dev environment

# Install all dependencies
pixi install

# Run commands in environment
pixi run python script.py
pixi run pytest

# Shell with environment activated
pixi shell

# Add tasks
pixi task add test "pytest tests/"
pixi task add docs "sphinx-build docs/ docs/_build"

# Run tasks
pixi run test
pixi run docs

# Update dependencies
pixi update numpy                         # update specific
pixi update                              # update all

# List packages
pixi list
pixi tree numpy                          # show dependency tree
```

### Quick Decision Tree: Pixi vs UV vs Both

```
Need compiled scientific libraries (NumPy, SciPy, GDAL)?
├─ YES → Use pixi (conda-forge has pre-built binaries)
└─ NO → Consider uv for pure Python projects

Need multi-language support (Python + R, Julia, C++)?
├─ YES → Use pixi (supports conda ecosystem)
└─ NO → uv sufficient for Python-only

Need multiple environments (dev, test, prod, GPU, CPU)?
├─ YES → Use pixi features for environment management
└─ NO → Single environment projects work with either

Need reproducible environments across platforms?
├─ CRITICAL → Use pixi (lockfiles include all platforms)
└─ LESS CRITICAL → uv also provides lockfiles

Want to use both conda-forge AND PyPI packages?
├─ YES → Use pixi (seamless integration)
└─ ONLY PYPI → uv is simpler and faster

Legacy conda environment files (environment.yml)?
├─ YES → pixi can import and modernize
└─ NO → Start fresh with pixi or uv
```

## When to Use This Skill

- **Setting up scientific Python projects** with complex compiled dependencies (NumPy, SciPy, Pandas, scikit-learn, GDAL, netCDF4)
- **Building reproducible research environments** that work identically across different machines and platforms
- **Managing multi-language projects** that combine Python with R, Julia, C++, or Fortran
- **Creating multiple environment configurations** for different hardware (GPU/CPU), testing scenarios, or deployment targets
- **Replacing conda/mamba workflows** with faster, more reliable dependency resolution
- **Developing packages that depend on both conda-forge and PyPI** packages
- **Migrating from environment.yml or requirements.txt** to modern, reproducible workflows
- **Running automated scientific workflows** with task runners and CI/CD integration
- **Working with geospatial, climate, or astronomy packages** that require complex C/Fortran dependencies

## Core Concepts

### 1. Unified Package Management (conda + PyPI)

Pixi resolves dependencies from **both conda-forge and PyPI** in a single unified graph, ensuring compatibility:

```toml
[project]
name = "my-science-project"
dependencies = [
    "numpy>=1.24",      # from conda-forge (optimized builds)
    "pandas>=2.0",      # from conda-forge
]

[tool.pixi.pypi-dependencies]
my-custom-pkg = ">=1.0"        # PyPI-only package
```

**Why this matters for scientific Python:**
- Get optimized NumPy/SciPy builds from conda-forge (MKL, OpenBLAS)
- Use PyPI packages not available in conda
- Single lockfile ensures all dependencies are compatible

### 2. Multi-Platform Lockfiles

Pixi generates `pixi.lock` with dependency specifications for **all platforms** (Linux, macOS, Windows, different architectures):

```toml
# pixi.lock includes:
# - linux-64
# - osx-64, osx-arm64
# - win-64
```

**Benefits:**
- Commit lockfile to git → everyone gets identical environments
- Works on collaborator's different OS without changes
- CI/CD uses exact same versions as local development

### 3. Feature-Based Environments

Create multiple environments using **features** without duplicating dependencies:

```toml
[tool.pixi.feature.test.dependencies]
pytest = ">=7.0"
pytest-cov = ">=4.0"

[tool.pixi.feature.gpu.dependencies]
pytorch-cuda = "11.8.*"

[tool.pixi.environments]
test = ["test"]
gpu = ["gpu"]
gpu-test = ["gpu", "test"]  # combines features
```

### 4. Task Automation

Define reusable commands as tasks:

```toml
[tool.pixi.tasks]
test = "pytest tests/ -v"
format = "ruff format src/ tests/"
lint = "ruff check src/ tests/"
docs = "sphinx-build docs/ docs/_build"
analyse = { cmd = "python scripts/analyze.py", depends-on = ["test"] }
```

### 5. Fast Dependency Resolution

Pixi uses **rattler** (Rust-based conda resolver) for 10-100x faster resolution than conda:

- Parallel package downloads
- Efficient caching
- Smart dependency solver

### 6. pyproject.toml Integration

Pixi reads standard Python project metadata from `pyproject.toml`, enabling:
- Single source of truth for project configuration
- Compatibility with pip, uv, and other tools
- Standard Python packaging workflows

## Quick Start

### Minimal Example: Data Analysis Project

```bash
# Create new project
mkdir climate-analysis && cd climate-analysis
pixi init --format pyproject

# Add scientific stack
pixi add python=3.11 numpy pandas matplotlib xarray

# Add development tools
pixi add --feature dev pytest ipython ruff

# Create analysis script
cat > analyze.py << 'EOF'
import pandas as pd
import matplotlib.pyplot as plt

# Your analysis code
data = pd.read_csv("data.csv")
data.plot()
plt.savefig("output.png")
EOF

# Run in pixi environment
pixi run python analyze.py

# Or activate shell
pixi shell
python analyze.py
```

### Example: Machine Learning Project with GPU Support

```bash
# Initialize project
pixi init ml-project --format pyproject
cd ml-project

# Add base dependencies
pixi add python=3.11 numpy pandas scikit-learn matplotlib jupyter

# Add CPU PyTorch
pixi add --platform linux-64 --platform osx-arm64 pytorch torchvision cpuonly -c pytorch

# Create GPU feature
pixi add --feature gpu pytorch-cuda=11.8 -c pytorch -c nvidia

# Add development tools
pixi add --feature dev pytest black mypy

# Configure environments in pyproject.toml
cat >> pyproject.toml << 'EOF'

[tool.pixi.environments]
default = { solve-group = "default" }
gpu = { features = ["gpu"], solve-group = "default" }
dev = { features = ["dev"], solve-group = "default" }
EOF

# Install and run
pixi install
pixi run python train.py          # uses default (CPU)
pixi run --environment gpu python train.py  # uses GPU
```

## Patterns

### Pattern 1: Converting Existing Projects to Pixi

**Scenario**: You have an existing project with `requirements.txt` or `environment.yml`

**Solution**:

```bash
# From requirements.txt
cd existing-project
pixi init --format pyproject

# Import from requirements.txt
while IFS= read -r package; do
    # Skip comments and empty lines
    [[ "$package" =~ ^#.*$ ]] || [[ -z "$package" ]] && continue

    # Try conda first, fallback to PyPI
    pixi add "$package" 2>/dev/null || pixi add --pypi "$package"
done < requirements.txt

# From environment.yml
pixi init --format pyproject --import-environment environment.yml

# Verify installation
pixi install
pixi run python -c "import numpy, pandas, scipy; print('Success!')"
```

**Best Practice**: Review generated `pyproject.toml` and organize dependencies:
- Core runtime dependencies → `[project.dependencies]`
- PyPI-only packages → `[tool.pixi.