phenoml-workflow

What this skill does

# PhenoML Workflow Skill

## Instructions

This skill provides an end-to-end guided experience for creating and testing PhenoML workflows. The skill walks users through the complete workflow setup process using executable scripts.

### When to Use This Skill

Use this skill when users want to:

1. Set up a new PhenoML workflow from scratch
2. Create workflows for processing clinical notes into FHIR Condition resources
3. Set up patient registration workflows with deduplication
4. Test workflows with example data
5. Set up or verify FHIR provider connections

### Interactive Workflow Setup Flow

This skill provides a step-by-step interactive experience where the skill **gathers information from the user conversationally**, then **executes reusable Python scripts** with that information to create and test workflows.

**Step 0: Ensure Dependencies are Installed**
- Before running any scripts, ensure the required Python packages are installed:
  ```bash
  pip install python-dotenv phenoml
  ```
- If the user gets import errors when running scripts, guide them to install these packages

**Step 1: Check FHIR Provider Setup**
- First, locate and run `check_env.py` (search for it using glob `**/check_env.py`) to check credentials and detect instance type
- **If SHARED EXPERIMENT is detected** (experiment.app.pheno.ml):
  - **Skip FHIR provider setup entirely** - shared experiment uses a pre-configured Medplum sandbox
  - The system automatically uses "experiment-default" as the FHIR_PROVIDER_ID
  - No FHIR credentials (CLIENT_ID, CLIENT_SECRET, BASE_URL) are required
  - Proceed directly to Step 2 (Gather Workflow Requirements)
- **If on a DEDICATED INSTANCE** (e.g., acme.app.pheno.ml), ask the user if they have already created a FHIR provider
- If NO:
  - Run `check_env.py` to verify credentials
  - If FHIR credentials are missing, guide them to add the credentials to .env with examples:
    - **Medplum**: `FHIR_PROVIDER_BASE_URL=https://api.medplum.com/fhir/R4`
    - **Athena**: `FHIR_PROVIDER_BASE_URL=https://api.preview.platform.athenahealth.com/fhir/r4`
    - **Epic**: `FHIR_PROVIDER_BASE_URL=https://fhir.epic.com/interconnect-fhir-oauth/api/FHIR/R4`
    - **Cerner**: `FHIR_PROVIDER_BASE_URL=https://fhir-myrecord.cerner.com/r4/[tenant-id]`
  - Run `setup_fhir_provider.py` to create the provider
  - The script will save FHIR_PROVIDER_ID to .env automatically
- If YES: Run `check_env.py` to verify FHIR_PROVIDER_ID is set

**Step 2: Gather Workflow Requirements**
- Ask what type of data they want the workflow to process (e.g., clinical notes, patient demographics, lab results)
- Ask them to describe what the workflow should do in their own words
- Provide examples of common workflow types to help guide them

**Step 3: Refine Workflow Description**
- Based on their description, identify the workflow type (condition creation, patient registration, observation creation, or custom)
- Present a refined version of their workflow using best practices
- Show them the recommended workflow instructions and settings (dynamic_generation, etc.)
- Ask for confirmation or allow them to modify

**Step 4: Create the Workflow**
- Run `create_workflow.py` with CLI arguments:
  - `--name "Workflow Name"`
  - `--instructions "workflow instructions..."`
  - `--sample-data '{"key": "value"}'`
  - `--dynamic-generation true`
- The script will create the workflow and save WORKFLOW_ID to .env
- Show the user the created workflow details

**Step 5: Test the Workflow**
- Ask what patient/data they want to test with (or provide examples)
- Run `test_workflow.py` with CLI arguments:
  - `--input-data '{"patient_last_name": "Smith", ...}'`
- Show the execution results
- Offer to run additional tests with different data

### Key Principles

1. **Gather information conversationally** - Ask the user questions first to understand their needs
2. **Find and use the reusable scripts** - First locate the scripts using glob `**/phenoml-workflow/scripts/*.py`, then execute them with appropriate CLI arguments
3. **Always pass --env-file** - Since the scripts may be in a different directory than the user's project, always pass `--env-file /path/to/user/project/.env` to ensure the scripts find the correct .env file (use the user's current working directory)
4. **Prefer CLI arguments over .env** - For workflow-specific data (name, instructions, test data), pass via CLI args rather than adding to .env
5. **Use .env for credentials** - Guide users to store credentials (PHENOML_*, MEDPLUM_*, etc.) in .env for security
6. **Guide progressively** - Go step-by-step, ensuring each step completes before moving to the next
7. **Confirm before executing** - Before running each script, explain what it will do
8. **Provide context** - Explain why each step is necessary and what it accomplishes
9. **Offer examples** - When asking for user input, always provide clear examples

### Available Scripts

**Important:**
1. Before running any script, first locate it using glob `**/phenoml-workflow/scripts/*.py` to find the correct path.
2. Always pass `--env-file` pointing to the user's project .env file (their current working directory).

#### 0. check_env.py

**Purpose:** Safely verifies which environment variables are set without exposing their values

**How it works:**
- Loads .env file using python-dotenv
- Checks presence of required credentials (returns TRUE/FALSE only)
- Never exposes actual credential values
- Provides clear guidance on missing credentials

**Usage:**
```bash
# Check all credentials (pass user's .env path)
python3 /path/to/check_env.py --env-file /user/project/.env

# JSON output only
python3 /path/to/check_env.py --env-file /user/project/.env --json

# Verbose output (formatted + JSON)
python3 /path/to/check_env.py --env-file /user/project/.env --verbose
```

**Security:**
- This script is designed to prevent credential leakage in LLM conversations
- It only reports TRUE/FALSE for each credential, never the actual values
- Safe to run in any context where credentials need to be verified

**Outputs:**
- Formatted status report with ✅/❌ indicators
- Guidance on missing credentials
- Exit code 1 if core credentials are missing

#### 1. setup_fhir_provider.py

**Purpose:** Creates a FHIR provider using credentials from .env

**How it works:**
- Reads FHIR credentials from .env (FHIR_PROVIDER_BASE_URL, FHIR_PROVIDER_CLIENT_ID, FHIR_PROVIDER_CLIENT_SECRET)
- Creates the FHIR provider in PhenoML
- Saves FHIR_PROVIDER_ID to .env

**Usage:**
```bash
# Create provider using .env credentials
python3 /path/to/setup_fhir_provider.py --env-file /user/project/.env

# With custom name and provider type
python3 /path/to/setup_fhir_provider.py --env-file /user/project/.env --name "My FHIR Server" --provider athena
```

**Required .env variables:**
- PHENOML_USERNAME, PHENOML_PASSWORD, PHENOML_BASE_URL
- FHIR_PROVIDER_BASE_URL, FHIR_PROVIDER_CLIENT_ID, FHIR_PROVIDER_CLIENT_SECRET

**Example FHIR_PROVIDER_BASE_URL values:**
- **Medplum**: `https://api.medplum.com/fhir/R4`
- **Athena**: `https://api.preview.platform.athenahealth.com/fhir/r4`
- **Epic**: `https://fhir.epic.com/interconnect-fhir-oauth/api/FHIR/R4`
- **Cerner**: `https://fhir-myrecord.cerner.com/r4/[tenant-id]`

**Optional .env variables:**
- FHIR_PROVIDER_NAME (default: "FHIR Server")
- FHIR_PROVIDER_TYPE (default: "medplum")
- FHIR_AUTH_METHOD (default: "client_secret")

**Outputs:**
- FHIR_PROVIDER_ID saved to .env

#### 2. create_workflow.py

**Purpose:** Creates a PhenoML workflow from .env or CLI arguments

**How it works:**
- Reads workflow configuration from CLI args or .env
- Creates the workflow in PhenoML
- Saves WORKFLOW_ID to .env

**Usage:**
```bash
# Create workflow with CLI arguments (recommended)
python3 /path/to/create_workflow.py \
  --env-file /user/project/.env \
  --name "Extract Conditions" \
  --instructions "You are a helpful agent who..." \
  --sample-data '{"patient_last_name": "Smith", "diagnosis_text": "Patient presents