aidp-hive

What this skill does

# `aidp-hive` — Apache Hive via AIDP `aidataplatform`

## When to use
- User wants to read or write a Hive table from an AIDP notebook.
- User mentions: "Hive", "HiveServer2", "HS2", "HCatalog", "Hive metastore", a Hive database/table name.

## When NOT to use
- For Oracle Big Data Service (BDS) HiveServer2 with **Kerberos** auth via Spark JDBC → use a custom skill (we removed `aidp-bds-hive` from v0.4 scope; revisit if your customer needs Kerberos specifically). The current `aidp-hive` skill covers non-Kerberos Hive (LDAP / SASL-PLAIN / NoAuth) via the `aidataplatform` format handler.
- For Iceberg-on-Hive-style metadata where data lives on `oci://` → [`aidp-iceberg`](../aidp-iceberg/SKILL.md).

## Prerequisites in the AIDP notebook
1. Helpers on `sys.path` (run `aidp-connectors-bootstrap` first).
2. Network: cluster must reach the HS2 host on the configured port (typically 10000). If your Hive lives in a customer VCN, ensure VCN peering is in place — the cluster pod CIDR has no implicit route to user VCNs.
3. Env vars / OCI Vault secrets:
   - `HIVE_HOST` (HiveServer2 hostname)
   - `HIVE_PORT` (typically `10000`)
   - `HIVE_USER`, `HIVE_PASSWORD`
   - `HIVE_SCHEMA` (Hive database name)
   - `HIVE_TABLE` (Hive table name)

## Read

```python
import os
from oracle_ai_data_platform_connectors.aidataplatform import (
    AIDP_FORMAT, aidataplatform_options,
)

opts = aidataplatform_options(
    type="HIVE",
    host=os.environ["HIVE_HOST"],
    port=int(os.environ.get("HIVE_PORT", "10000")),
    user=os.environ["HIVE_USER"],
    password=os.environ["HIVE_PASSWORD"],
    schema=os.environ["HIVE_SCHEMA"],
    table=os.environ["HIVE_TABLE"],
)
df = spark.read.format(AIDP_FORMAT).options(**opts).load()
df.show(10)
```

## Write

```python
opts = aidataplatform_options(
    type="HIVE",
    host=os.environ["HIVE_HOST"],
    port=int(os.environ.get("HIVE_PORT", "10000")),
    user=os.environ["HIVE_USER"],
    password=os.environ["HIVE_PASSWORD"],
    schema=os.environ["HIVE_SCHEMA"],
    table=os.environ["HIVE_TARGET_TABLE"],
    extra={"write.mode": "APPEND"},   # CREATE | APPEND | OVERWRITE
)
df.write.format(AIDP_FORMAT).options(**opts).save()
```

## Pushdown SQL

```python
opts = aidataplatform_options(
    type="HIVE",
    host=os.environ["HIVE_HOST"],
    port=int(os.environ.get("HIVE_PORT", "10000")),
    user=os.environ["HIVE_USER"],
    password=os.environ["HIVE_PASSWORD"],
    extra={
        "pushdown.sql": (
            "SELECT customer_id, SUM(amount) AS total "
            "FROM sales_db.transactions "
            "WHERE event_dt >= '2025-01-01' "
            "GROUP BY customer_id"
        ),
    },
)
df = spark.read.format(AIDP_FORMAT).options(**opts).load()
df.show(10)
```

## Gotchas
- **No `database.name` option** — Hive uses `schema` directly to identify the Hive database (e.g. `default`, `sales_db`). Don't pass `database_name`.
- **Network reachability is the most common failure.** Smoke-test from the cluster: `socket.create_connection((host, port), timeout=8)`. If this fails, it's a network problem, not an auth problem — VCN peering / NSG / DNS, in that order.
- **No Kerberos on this skill.** This connector handler uses LDAP / SASL-PLAIN / NoAuth. If your Hive cluster only accepts Kerberos, you need a Spark-native JDBC path with a keytab (out of scope for v0.5 — file an issue).
- **Write modes** — `CREATE` (fail if exists), `APPEND`, `OVERWRITE`. Default is `CREATE`.
- **Partitioned tables.** When writing to a partitioned Hive table, `OVERWRITE` overwrites all partitions touched by the DataFrame's distinct partition keys; existing partitions not touched are preserved.

## References
- Helper: [scripts/oracle_ai_data_platform_connectors/aidataplatform.py](../../scripts/oracle_ai_data_platform_connectors/aidataplatform.py)
- Official sample: [oracle-samples/oracle-aidp-samples → `data-engineering/ingestion/Read_Write_External_Ecosystem_Connectors/Hive.ipynb`](https://github.com/oracle-samples/oracle-aidp-samples/blob/main/data-engineering/ingestion/Read_Write_External_Ecosystem_Connectors/Hive.ipynb)