byteforge-loki-logging

What this skill does

# Loki Logging with byteforge-loki-logging

This skill helps you integrate Grafana Loki logging using the `byteforge-loki-logging` library, which handles structured JSON logging and asynchronous Loki shipping with graceful fallback.

## When to Use This Skill

Use this skill when:
- Starting a new Python/Flask application
- You want centralized logging with Loki
- You need structured JSON logs for production
- You want easy local development with console logs

## What This Skill Creates

1. **requirements.txt entry** - Adds `byteforge-loki-logging` dependency (public GitHub library)
2. **Logging initialization** - Adds `configure_logging()` call to main application file
3. **CA certificate configuration** - Docker Compose volume mount for your Loki CA certificate
4. **Environment variable documentation** - All required Loki configuration

## Step 1: Gather Project Information

**IMPORTANT**: Before making changes, ask the user these questions:

1. **"What is your application tag/name?"** (e.g., "materia-server", "trading-api")
   - This identifies your service in Loki logs
   - This becomes the `application` label in Loki — all services across the stack (Python and TypeScript) **must** use `application` as the label name, never `app` or `service`

2. **"What is your main application file?"** (e.g., "app.py", "server.py", "materia_server.py")
   - Where to add the logging configuration

3. **"What is your CA certificate filename?"** (e.g., "loki-ca.pem", "my-org-ca.pem")
   - Required for secure Loki connection over TLS
   - If the user does not have one or their Loki endpoint does not use a private CA, this step can be skipped

## Step 2: Add byteforge-loki-logging to requirements.txt

Add this line to `requirements.txt`:

```txt
# Logging configuration with Loki support (public GitHub library)
byteforge-loki-logging @ git+https://github.com/jmazzahacks/byteforge-loki-logging.git
```

Install the dependency:
```bash
pip install -r requirements.txt
```

## Step 3: Configure Logging in Application

Add to the **top** of your main application file (e.g., `{app_file}.py`):

```python
import os
from byteforge_loki_logging import configure_logging

# Configure logging with byteforge-loki-logging
# Use debug_local=True for local development, False for production with Loki
debug_mode = os.environ.get('DEBUG_LOCAL', 'true').lower() == 'true'
log_level = os.environ.get('LOG_LEVEL', 'INFO')
configure_logging(
    application_tag='{application_tag}',
    debug_local=debug_mode,
    local_level=log_level
)
```

**CRITICAL**: Replace:
- `{app_file}` -> Your main application filename (e.g., "materia_server")
- `{application_tag}` -> Your service name (e.g., "materia-server")

Place this **before** creating your Flask app or any other initialization.

### Structured JSON Logging

JSON formatting is enabled by default (`json_format=True`). Log records are formatted as:

```json
{"logger": "myapp", "level": "INFO", "message": "Request processed", "user_id": "123", "latency_ms": 42}
```

Query in Grafana: `{application="my-service"} | json | user_id="123"`

### Graceful Fallback

If the Loki connection test fails at startup, logging automatically falls back to stdout with a warning on stderr. Your application never crashes due to logging issues.

## Step 4: Configure CA Certificate

If your Loki endpoint uses a private CA certificate, mount it into the container via Docker Compose as a read-only volume. **Do not** bake the certificate into the Dockerfile with `COPY`.

In `docker-compose.yaml`:

```yaml
services:
  {app_name}:
    volumes:
      - /path/to/{ca_cert_filename}:/app/certs/loki-ca.pem:ro
    environment:
      - LOKI_CA_BUNDLE_PATH=/app/certs/loki-ca.pem
```

**CRITICAL**: Replace:
- `{app_name}` -> Your service name in docker-compose
- `{ca_cert_filename}` -> Your actual CA certificate filename from Step 1

The certificate will be available at `/app/certs/loki-ca.pem` inside the container.

If your Loki endpoint does not use a private CA (e.g., uses a publicly trusted certificate), skip this step and omit `LOKI_CA_BUNDLE_PATH`.

## Step 5: Document Environment Variables

Add to README.md or .env.example:

### Environment Variables

**Logging Configuration (Local Development):**
- `DEBUG_LOCAL` - Set to 'true' for local development (console logs), 'false' for production (Loki)
  - Default: 'true'
  - Production: 'false'
- `LOG_LEVEL` - Logging level: DEBUG, INFO, WARNING, ERROR, CRITICAL
  - Default: 'INFO'

**Loki Configuration (Production Only - required when DEBUG_LOCAL=false):**
- `LOKI_ENDPOINT` - Loki push API URL (e.g., https://loki.example.com/loki/api/v1/push)
- `LOKI_USER` - Loki username for HTTP Basic Auth
- `LOKI_PASSWORD` - Loki password for HTTP Basic Auth
- `LOKI_CA_BUNDLE_PATH` - Path to CA certificate (e.g., /app/certs/loki-ca.pem), or "false" to disable SSL verification

### Logging Behavior

**Local Development** (`DEBUG_LOCAL=true`):
- Logs output to console with human-readable formatting
- Easy to read during development
- No Loki connection required
- No need to set LOKI_* variables

**Production** (`DEBUG_LOCAL=false`):
- Logs output as structured JSON to Loki asynchronously (1-second batching)
- All LOKI_* variables must be set
- Queryable in Grafana
- Automatic fallback to stdout if Loki is unreachable at startup

## Step 6: Usage Examples

### Local Development

```bash
# In .env or shell
export DEBUG_LOCAL=true
export LOG_LEVEL=DEBUG

pip install -r requirements.txt
python {app_file}.py
```

### Production Deployment

Docker Compose example:

```yaml
services:
  {app_name}:
    build:
      context: .
    volumes:
      - /path/to/{ca_cert_filename}:/app/certs/loki-ca.pem:ro
    environment:
      - DEBUG_LOCAL=false
      - LOG_LEVEL=INFO
      - LOKI_ENDPOINT=${LOKI_ENDPOINT}
      - LOKI_USER=${LOKI_USER}
      - LOKI_PASSWORD=${LOKI_PASSWORD}
      - LOKI_CA_BUNDLE_PATH=/app/certs/loki-ca.pem
```

**NOTE**: Set these in your .env file:
```
LOKI_ENDPOINT=https://loki.example.com/loki/api/v1/push
LOKI_USER=your_loki_user
LOKI_PASSWORD=your_loki_password
```

## How It Works

The `byteforge-loki-logging` library provides:

1. **Automatic mode detection** - Console logs for local dev, Loki for production
2. **Structured JSON logging** - Consistent JSON format for Loki, enabled by default
3. **Async shipping** - Background thread with 1-second batching for minimal performance impact
4. **Secure connection** - Uses CA certificate for encrypted Loki communication
5. **Graceful fallback** - Falls back to stdout if Loki is unreachable, never crashes your app
6. **Application tagging** - Identifies your service in centralized logs via the `application` label

**You don't need to:**
- Write JSON formatters
- Configure logging handlers
- Manage Loki client setup
- Handle certificate validation
- Worry about logging crashing your application

**Just call `configure_logging()` and you're done!**

## Integration with Other Skills

### Flask API Server
If using **flask-smorest-api** skill, add logging **before** creating Flask app:

```python
import os
import logging
from flask import Flask
from byteforge_loki_logging import configure_logging

# Configure logging FIRST
debug_mode = os.environ.get('DEBUG_LOCAL', 'true').lower() == 'true'
configure_logging(application_tag='my-api', debug_local=debug_mode)

# Then create Flask app
app = Flask(__name__)

# IMPORTANT: Propagate Flask's logger to the root logger so unhandled
# exceptions in route handlers reach Loki. Without this, Flask catches
# exceptions internally and logs them via werkzeug to stdout/stderr,
# bypassing the root logger that configure_logging() set up.
app.logger.handlers.clear()
app.logger.propagate = True
app.logger.setLevel(logging.DEBUG)

# ... rest of setup
```

**Why this matters**: Flask catches exceptions in route handlers and returns a 500 response, but by default it logs the traceback through its own `app.logger` using werkzeug's error handling — not through Python