pandas-best-practices

What this skill does

# Pandas Best Practices

Expert guidelines for Pandas development, focusing on data manipulation, analysis, and efficient DataFrame operations.

## Code Style and Structure

- Write concise, technical responses with accurate Python examples
- Prioritize reproducibility in data analysis workflows
- Use functional programming; avoid unnecessary classes
- Prefer vectorized operations over explicit loops
- Use descriptive variable names reflecting data content
- Follow PEP 8 style guidelines

## DataFrame Creation and I/O

- Use `pd.read_csv()`, `pd.read_excel()`, `pd.read_json()` with appropriate parameters
- Specify `dtype` parameter to ensure correct data types on load
- Use `parse_dates` for automatic datetime parsing
- Set `index_col` when the data has a natural index column
- Use `chunksize` for reading large files incrementally

## Data Selection

- Use `.loc[]` for label-based indexing
- Use `.iloc[]` for integer position-based indexing
- Avoid chained indexing (e.g., `df['col'][0]`) - use `.loc` or `.iloc` instead
- Use boolean indexing for conditional selection: `df[df['col'] > value]`
- Use `.query()` method for complex filtering conditions

## Method Chaining

- Prefer method chaining for data transformations when possible
- Use `.pipe()` for applying custom functions in a chain
- Chain operations like `.assign()`, `.query()`, `.groupby()`, `.agg()`
- Keep chains readable by breaking across multiple lines

## Data Cleaning and Validation

### Missing Data

- Check for missing data with `.isna()` and `.info()`
- Handle missing data appropriately: `.fillna()`, `.dropna()`, or imputation
- Use `pd.NA` for nullable integer and boolean types
- Document decisions about missing data handling

### Data Quality Checks

- Implement data quality checks at the beginning of analysis
- Validate data types with `.dtypes` and convert as needed
- Check for duplicates with `.duplicated()` and handle appropriately
- Use `.describe()` for quick statistical overview

### Type Conversion

- Use `.astype()` for explicit type conversion
- Use `pd.to_datetime()` for date parsing
- Use `pd.to_numeric()` with `errors='coerce'` for safe numeric conversion
- Utilize categorical data types for low-cardinality string columns

## Grouping and Aggregation

### GroupBy Operations

- Use `.groupby()` for efficient aggregation operations
- Specify aggregation functions with `.agg()` for multiple operations
- Use named aggregation for clearer output column names
- Consider `.transform()` for broadcasting results back to original shape

### Pivot Tables and Reshaping

- Use `.pivot_table()` for multi-dimensional aggregation
- Use `.melt()` to convert wide to long format
- Use `.pivot()` to convert long to wide format
- Use `.stack()` and `.unstack()` for hierarchical index manipulation

## Performance Optimization

### Memory Efficiency

- Use categorical data types for low-cardinality strings
- Downcast numeric types when appropriate
- Use `pd.eval()` and `.eval()` for large expression evaluation

### Computation Speed

- Use vectorized operations instead of `.apply()` with row-wise functions
- Prefer built-in aggregation functions over custom ones
- Use `.values` or `.to_numpy()` for NumPy operations when faster

### Avoiding Common Pitfalls

- Avoid iterating with `.iterrows()` - use vectorized operations
- Don't modify DataFrames while iterating
- Be aware of SettingWithCopyWarning - use `.copy()` when needed
- Avoid growing DataFrames row by row - collect in list and create once

## Time Series Operations

- Use `DatetimeIndex` for time series data
- Leverage `.resample()` for time-based aggregation
- Use `.shift()` and `.diff()` for lag operations
- Use `.rolling()` and `.expanding()` for window calculations

## Merging and Joining

- Use `.merge()` for SQL-style joins
- Specify `how` parameter: 'inner', 'outer', 'left', 'right'
- Use `validate` parameter to check join cardinality
- Use `.concat()` for stacking DataFrames

## Key Conventions

- Import as `import pandas as pd`
- Use `snake_case` for column names when possible
- Document data sources and transformations
- Keep notebooks reproducible with clear cell execution order