bio-hi-c-analysis-hic-data-io

What this skill does

## Version Compatibility

Reference examples tested with: cooler 0.10+, hic2cool 1.0+, hictk 1.0+, bioframe 0.7+

Before using code patterns, verify installed versions match. If versions differ:
- Python: `pip show <package>` then `help(module.function)` to check signatures
- CLI: `<tool> --version` then `<tool> --help` to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed
package and adapt the example to match the actual API rather than retrying.

Two version boundaries change BEHAVIOUR, not just signatures: hic2cool >= 0.5.0 stores Juicer norms un-inverted (divisive); < 0.5.0 inverted them to multiplicative, so two coolers made from one .hic by different hic2cool versions disagree numerically. cooler standardized creation/balance signatures around 0.8-0.10 (`balance_cooler` keyword-only after `clr`; `store=False` default). Record the converter version in provenance.

# Hi-C Data I/O

**"Load my Hi-C contact matrix, convert it, and pull out a region."** -> Open the cooler at one resolution, fetch raw or balanced pixels, and convert across .hic/.cool/.mcool while knowing what does not survive the round trip.
- Python: `cooler.Cooler('file.mcool::/resolutions/10000').matrix(balance=True).fetch('chr1')`
- CLI: `hic2cool convert in.hic out.mcool -r 0` / `hictk convert in.hic out.mcool` / `cooler cload pairs ...`

## The Single Most Important Modern Insight -- cooler Stores Observed Counts; .hic Bakes In Norms, So Conversion Is Never Lossless Both Ways

A `.cool` is a thin, open, HDF5-native *store*: three tables (`chroms`, `bins`, `pixels`) holding raw observed integer counts in an upper-triangle COO layout, plus optional cached `weight` bias columns. Balancing, expected, O/E, eigenvectors -- everything else is computed downstream on demand. Juicer's `.hic` is the opposite philosophy: a sealed binary *deliverable* with all resolutions, precomputed normalization vectors, and expected vectors welded in. Every footgun in this skill descends from that split:

- **Conversion translates between two philosophies, not two encodings.** `.hic -> .cool` loses FRAG (restriction-fragment) matrices (cooler has no FRAG concept) and Juicer's precomputed expected; a norm whose vector is missing arrives as all-NaN. `.cool -> .hic` loses asymmetric matrices, arbitrary extra pixel value columns, and non-Hi-C labeled arrays.
- **The `weight` column NAME is load-bearing.** cooler's own ICE `weight` is applied MULTIPLICATIVELY by `matrix(balance=True)`. Juicer KR/VC norms are DIVISIVE. `Cooler.matrix(divisive_weights=None)` (the default) decides by column name: weights named **KR, VC, or VC_SQRT are auto-treated as divisive**; everything else (including `weight`) is multiplicative. Import a KR vector under the name `weight` and it is applied the wrong way -- garbage, no error.
- **`.mcool` is a container of resolutions, not a matrix.** Every downstream tool wants a single-resolution URI `file.mcool::/resolutions/<bp>`, never the bare `.mcool`. Each resolution is balanced from scratch; the 100kb `weight` is NOT derivable from the 10kb `weight`.

## Format and Tool Taxonomy

| Format / Tool | Role | Mechanism | When |
|---------------|------|-----------|------|
| `.cool` | single-resolution store | HDF5 chroms/bins/pixels, raw counts + optional weight | one resolution; the analysis unit |
| `.mcool` | multi-resolution container | `/resolutions/<bp>/` each a full cooler | HiGlass tilesets; pick a resolution via URI |
| `.scool` | single-cell container | shared bins, `/cells/<name>/pixels` | scHi-C; avoids duplicating the bin table per cell |
| `.hic` (Juicer) | sealed deliverable | binary, all resolutions + baked norm/expected | Juicer/Juicebox ecosystem; BP or FRAG bins |
| `.pairs` (4DN) | upstream contact list | bgzip + pairix index; upper-triangle, flipped | input to `cooler cload`; provenance of the matrix |
| cooler (CLI+Py) | the open standard store/API | pandas/scipy selectors; `cload`/`balance`/`zoomify`/`dump` | the default; cooltools integration |
| hic2cool | .hic -> .cool/.mcool | 4DN-canonical norm handler (>=0.5.0 un-inverted) | importing Juicer norms; BP only |
| hictk | fast cross-format convert/dump | C++; reads .hic v6-9 + cooler, writes .hic v9 + cooler | large files; faster than hic2cool/straw; no FRAG, no asymmetric |

## Decision Tree by Scenario

| Scenario | Recommended | Why |
|----------|-------------|-----|
| Bare `.mcool`, tool errors / wrong scale | append `::/resolutions/<bp>` URI | the .mcool is a container; tools need one resolution |
| `.hic` -> `.mcool`, want Juicer KR/SCALE preserved | `hic2cool convert -r 0` | 4DN-canonical norm handling; keeps divisive names |
| `.hic` -> cooler, large file, speed matters | `hictk convert` | C++, order-of-magnitude faster; but no FRAG |
| `.hic` was FRAG-binned | re-bin from pairs in BP | FRAG does not survive any converter -> contact-pairs |
| Build a cooler from `.pairs` | `cooler cload pairs -c1 -p1 -c2 -p2 sizes.txt:bp` | needs flipped/deduped pairs -> contact-pairs |
| Need lower resolution | `cooler zoomify --balance` (sum RAW, re-ICE) | cannot sum balanced pixels; re-balance per resolution |
| Imported KR/VC norms | keep their original names | the KR/VC/VC_SQRT auto-divisive rule fires only on those names |
| `matrix(balance=True)` raises / all NaN | balance first; NaN rows = masked bins | unbalanced file has no weight; masked bins are expected NaN |
| `fetch('1')` returns empty on a `chr1` file | harmonize chrom naming first | chr1-vs-1 silently zeros every join, no error |
| Balance/normalize for analysis | -> matrix-operations | ICE/KR mechanics, O/E, expected live there |
| Two coolers, compare pixels | confirm bin tables byte-identical | different contigs/order shift every bin_id |
| scHi-C many cells | `cooler.create_scool` (`.scool`) -> single-cell | shared bin table; per-cell pixels |

## Load a Cooler and Select a Resolution

```python
import cooler

cooler.fileops.list_coolers('matrix.mcool')                      # e.g. ['/resolutions/1000', '/resolutions/10000', ...]
clr = cooler.Cooler('matrix.mcool::/resolutions/10000')          # single-resolution URI, never the bare .mcool
clr.binsize, clr.chromnames, clr.info['sum']                     # info is unvalidated metadata, not a guarantee
'weight' in clr.bins().columns                                   # is this resolution balanced?
```

`clr.matrix().fetch(region)` takes a UCSC region string or a bare chrom name; one arg -> symmetric square, two args -> rectangular submatrix (incl. trans). `clr.bins().fetch('chr1')`, `clr.pixels().fetch(region)` slice the tables.

## Fetch Balanced vs Raw Pixels

**Goal:** Pull a chromosome submatrix as a dense array, correctly balanced.

**Approach:** Confirm the file carries a `weight` column, then `matrix(balance=True)`; on a balanced file, all-NaN rows are masked low-coverage bins (correct), not a bug. For Juicer-imported KR/VC/VC_SQRT weights, name them correctly and the divisive auto-rule fires; force it with `divisive_weights=` only if a custom column is misnamed.

```python
balanced = clr.matrix(balance=True).fetch('chr1')                # multiplicative cooler weight
raw = clr.matrix(balance=False).fetch('chr1')                    # observed counts
kr = clr.matrix(balance='KR').fetch('chr1')                      # KR/VC/VC_SQRT auto-treated as divisive by name
sparse = clr.matrix(balance=True, sparse=True).fetch('chr1')     # scipy COO for large chromosomes
```

## Convert .hic to cooler (and Back)

```bash
hic2cool convert in.hic out.mcool -r 0                           # -r 0 = all resolutions -> .mcool; norm vectors un-inverted (>=0.5.0)
hic2cool convert in.hic out.cool -r 10000                        # single resolution -> .cool
hic2cool extract-norms in.hic out.mcool                          # add Juicer norm vectors to an existing matching cooler
hictk convert in.hic out.mcool                                   # fast C++ path; --resolutions to