hpc

What this skill does

## Contents

- [When to Use What](#when-to-use-what)
- [Login Node Enforcement](#login-node-enforcement)
- [Cluster Reference](#cluster-reference)
- [Slurm Job Submission](#slurm-job-submission)
- [Array Jobs](#array-jobs)
- [SGE to Slurm Translation](#sge-to-slurm-translation)
- [Environment Variables](#environment-variables)
- [WRDS Data Access](#wrds-data-access)
- [Monitoring & Debugging](#monitoring--debugging)
- [Resource Billing](#resource-billing)

## When to Use What

Three compute environments, each with a clear role:

| Environment | Use For | Examples |
|-------------|---------|----------|
| **Local / RJDS** | Exploration, prototyping, notebooks | EDA, quick plots, marimo/Jupyter, test on small samples, iterate on code |
| **WRDS (SGE)** | Data access, SAS ETL, file parsing | SAS jobs against WRDS libraries, SEC filing parsers on `/wrds/sec/`, scan_covers, ad-hoc SQL |
| **UVA HPC (Slurm)** | Scale compute | Model estimation (PIN), large polars pipelines, anything needing >10 cores or >1 hour |

### The Workflow

```
1. EXPLORE (local/RJDS)     →  Prototype code, test on 5-10 items
2. BUILD DATA (WRDS)        →  SAS ETL or PostgreSQL queries (data lives there)
3. ESTIMATE AT SCALE (HPC)  →  sbatch when you need 100+ cores
4. ANALYZE RESULTS (local)  →  Pull results back, notebooks, regressions, tables
```

### Decision Rules

- **Does it need WRDS filesystem access?** (`/wrds/sec/`, SAS libraries) → WRDS
- **Is it CPU-intensive and embarrassingly parallel?** → HPC
- **Is it exploratory or iterative?** → Local / RJDS
- **Is it a quick SQL query?** → Either WRDS or HPC (both have PostgreSQL access)

### HPC Interactive Partition

The `interactive` partition (42 nodes, 12h max) is for **testing sbatch scripts on one chunk before submitting 176 tasks**, not for replacing local dev work:

```bash
salloc -p interactive --cpus-per-task=4 --mem=16G --time=1:00:00
# test your script, then exit and sbatch the real job
```

### Why This Split Matters

PIN estimation proved it: WRDS SGE has 10 concurrent slots and took 8+ hours without starting OWR. UVA HPC ran 70+ OWR tasks simultaneously and finished in 30 minutes. But WRDS is still the right place to build the data — the SAS libraries and SEC filings live there.

## Login Node Enforcement

### IRON LAW: NEVER RUN COMPUTE ON THE LOGIN NODE

<EXTREMELY-IMPORTANT>
The login node is shared infrastructure. Running estimation, bulk processing, or any CPU-intensive work directly via SSH will get the account flagged and the process killed.

**ALWAYS** write a Slurm submission script and submit via `sbatch`. No exceptions.

- `ssh uva-hpc 'python3 est.py owr 2020'` → **WRONG. Use sbatch.**
- `ssh uva-hpc 'nohup ./process &'` → **WRONG. Still the login node. Use sbatch.**
- `ssh uva-hpc 'for year in 2003..2024; do python3 ...; done'` → **WRONG. Use sbatch --array.**
- `sbatch run_est.sh owr` → **CORRECT.**

The login node is for: `sbatch`, `squeue`, `scancel`, `sinfo`, `scp`, `ls`, `head`, short queries.
</EXTREMELY-IMPORTANT>

### Rationalization Table

| Excuse | Reality | Do Instead |
|--------|---------|------------|
| "It's a quick test, just one stock" | One stock becomes 5,000 when you forget to change the args | Write the sbatch script first, test with `--array=1-1` |
| "nohup makes it background, so it's fine" | nohup is still the login node — same shared CPU | sbatch, not nohup |
| "I'll run the real job via sbatch later" | You'll forget. The 'test' run flags the account | sbatch from the start |
| "It only takes 30 seconds" | You don't know that until it runs | If in doubt, sbatch |

### Red Flags — STOP If You're About To

- **Write `ssh uva-hpc 'python3 ... > output'`** → STOP. Write a submit script.
- **Write `ssh uva-hpc 'nohup ... &'`** → STOP. Use sbatch.
- **Run a loop over years/permnos interactively** → STOP. Use `--array`.

## Cluster Reference

### UVA HPC (Rivanna/Afton)

- **SSH**: `ssh uva-hpc` (configured with ProxyJump through Mac via tailnet)
- **User**: `vwh7mb`
- **Home**: `/home/vwh7mb` (GPFS, 12PB shared, no per-user quota displayed)
- **Scratch**: `/scratch/vwh7mb/` (Weka, 12TB)
- **Allocation**: 10M SUs (service units ≈ weighted CPU-core-hours)

### Partitions

| Partition | Nodes | CPUs/Node | RAM/Node | MaxTime | MinNodes | MaxNodes | Use For |
|-----------|-------|-----------|----------|---------|----------|----------|---------|
| `standard` | 301 | 40+ | 384GB+ | 7 days | 0 | **1** | Single-node jobs, array tasks |
| `parallel` | 179 | 96 | 768GB | 3 days | **2** | 64 | Multi-node MPI jobs only |
| `gpu` | 44 | 36+ | 257GB+ | 3 days | — | — | GPU workloads |
| `interactive` | 42 | 32+ | 128GB+ | 12 hrs | — | — | Interactive/debugging |

### CRITICAL: Partition Selection

<EXTREMELY-IMPORTANT>
**Use `standard` for embarrassingly parallel array jobs** (PIN estimation, file processing, per-year/per-stock tasks).

The `parallel` partition requires **MinNodes=2** — it will reject single-node jobs with "Node count specification invalid". It is designed for MPI jobs that span multiple nodes.

**Wrong:** `#SBATCH --partition=parallel` for array jobs → submission fails
**Right:** `#SBATCH --partition=standard` for array jobs → 301 nodes available
</EXTREMELY-IMPORTANT>

### When to Use Each Partition

**`standard`** (default choice for most research computing):
- Embarrassingly parallel work: array jobs where each task is independent (PIN estimation, file parsing, per-stock/per-year processing)
- Single-node Python/R with `ProcessPoolExecutor`, `multiprocessing`, `mclapply`
- Any job where tasks don't communicate with each other
- MaxNodes=1, so each array element runs on exactly one node

**`parallel`** (multi-node distributed computing):
- MPI jobs where processes on different nodes exchange messages (`mpi4py`, OpenMPI, MVAPICH)
- Dask distributed or Ray clusters spanning multiple nodes
- Large linear algebra / matrix factorizations that exceed single-node RAM (ScaLAPACK, PETSc)
- Simulations with inter-process communication (CFD, molecular dynamics)
- Key requirement: your code must explicitly coordinate across nodes (MPI, Dask scheduler, etc.) — `ProcessPoolExecutor` and `multiprocessing` are single-node only
- MinNodes=2, 96 CPUs and 768GB RAM per node — use when one node isn't enough

**`gpu`** (GPU-accelerated workloads):
- Deep learning training/inference (PyTorch, TensorFlow, JAX)
- GPU-accelerated linear algebra (CuPy, RAPIDS)
- LLM inference or fine-tuning

**`interactive`** (debugging and development):
- Testing job scripts before full submission: `salloc -p interactive --cpus-per-task=4 --mem=16G --time=1:00:00`
- Debugging segfaults or data loading issues
- 12-hour max — not for production runs

### Python/R Environment

- **pixi**: Install to `$HOME/.pixi/bin/pixi` via `curl -fsSL https://pixi.sh/install.sh | bash`
- **Project envs**: `$HOME/projects/<name>/.pixi/envs/default/bin/python`
- **Modules** (alternative): `module load python` — but pixi preferred for reproducibility
- **NEVER** install Jupyter kernels globally on HPC

## Slurm Job Submission

### Basic Submit Script

```bash
#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --partition=standard
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=3:00:00
#SBATCH --output=logs/job-%A_%a.log

mkdir -p logs

export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

PYTHON=$HOME/projects/my-project/.pixi/envs/default/bin/python
$PYTHON -u my_script.py --workers ${SLURM_CPUS_PER_TASK:-8}
```

### Submission

```bash
sbatch script.sh              # submit
sbatch script.sh arg1 arg2    # args passed to script as $1, $2
```

Note: unlike SGE's `qsub run.sh <model>`, Slurm passes arguments after the script name directly. Use `${1:?Usage: sbatch script.sh <arg>}` to enforce.

## Array Jobs

### Pattern

```bash
#SBATCH --array=1-176           # tasks 1 through 176
#SBATCH --array=1-176%50        # max 50 concurrent tasks
#SBATCH --array=1,5,9,13        # specific tas