grpc-microservices

What this skill does

# gRPC Microservices

A comprehensive skill for building high-performance, type-safe microservices using gRPC and Protocol Buffers. This skill covers service design, all streaming patterns, interceptors, load balancing, error handling, and production deployment patterns for distributed systems.

## When to Use This Skill

Use this skill when:

- Building microservices that require high-performance, low-latency communication
- Implementing real-time data streaming between services
- Designing type-safe APIs with strong contracts using Protocol Buffers
- Creating polyglot systems where services are written in different languages
- Building distributed systems requiring bidirectional streaming
- Implementing service meshes with advanced routing and observability
- Designing APIs that need to evolve with backward/forward compatibility
- Creating internal APIs where performance and type safety are critical
- Building event-driven architectures with streaming data pipelines
- Implementing client-server systems with push capabilities (server streaming)
- Designing systems requiring efficient binary serialization
- Building microservices requiring automatic code generation for multiple languages

## Core Concepts

### gRPC Fundamentals

gRPC is a modern open-source RPC framework that can run anywhere. It enables client and server applications to communicate transparently and makes it easier to build connected systems.

**Key Characteristics:**
- **HTTP/2 based**: Multiplexing, server push, header compression
- **Protocol Buffers**: Efficient binary serialization format
- **Streaming**: Bidirectional streaming support built-in
- **Code Generation**: Auto-generate client/server code in 10+ languages
- **Deadlines/Timeouts**: First-class timeout support
- **Cancellation**: Propagate cancellation across services
- **Interceptors**: Middleware pattern for cross-cutting concerns

### Protocol Buffers (protobuf)

Protocol Buffers is a language-neutral, platform-neutral extensible mechanism for serializing structured data.

**Advantages:**
- **Compact**: 3-10x smaller than JSON
- **Fast**: 20-100x faster to serialize/deserialize than JSON
- **Type-safe**: Strongly typed schema with validation
- **Backward/Forward Compatible**: Evolve schemas safely
- **Language Support**: Official support for 10+ languages
- **Self-documenting**: Schema serves as documentation

**Basic Syntax:**
```protobuf
syntax = "proto3";

message User {
  int32 id = 1;
  string name = 2;
  string email = 3;
}
```

### Service Definitions

gRPC services are defined in `.proto` files and specify available methods and their input/output types.

**Basic Service:**
```protobuf
service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
  rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
}
```

### Four Types of RPC Methods

#### 1. Unary RPC (Request-Response)
Simple request-response pattern, like a traditional REST API call.

```protobuf
rpc GetUser(GetUserRequest) returns (GetUserResponse);
```

**Use Cases:**
- CRUD operations
- Simple queries
- Synchronous operations
- Traditional request-response patterns

#### 2. Server Streaming RPC
Client sends one request, server returns a stream of responses.

```protobuf
rpc ListUsers(ListUsersRequest) returns (stream User);
```

**Use Cases:**
- Paginated results
- Real-time updates
- Server-side event push
- Large dataset downloads

#### 3. Client Streaming RPC
Client sends a stream of requests, server returns one response.

```protobuf
rpc CreateUsers(stream CreateUserRequest) returns (CreateUsersResponse);
```

**Use Cases:**
- Bulk uploads
- Batch processing
- Client-side aggregation
- File uploads in chunks

#### 4. Bidirectional Streaming RPC
Both client and server send streams of messages independently.

```protobuf
rpc Chat(stream ChatMessage) returns (stream ChatMessage);
```

**Use Cases:**
- Real-time chat applications
- Live collaboration
- Gaming (real-time state sync)
- IoT bidirectional communication

## Protobuf Schema Design

### Message Design Best Practices

**1. Use Explicit Field Numbers**

Field numbers are critical for backward compatibility and should never be reused.

```protobuf
message User {
  int32 id = 1;           // Never change this number
  string name = 2;        // Never change this number
  string email = 3;       // Never change this number
  // int32 age = 4;       // DEPRECATED - don't reuse 4
  string phone = 5;       // New field - use next available
}
```

**2. Use Enumerations for Fixed Sets**

```protobuf
enum UserRole {
  USER_ROLE_UNSPECIFIED = 0;  // Always have a zero value
  USER_ROLE_ADMIN = 1;
  USER_ROLE_MODERATOR = 2;
  USER_ROLE_MEMBER = 3;
}

message User {
  int32 id = 1;
  string name = 2;
  UserRole role = 3;
}
```

**3. Use Nested Messages for Complex Types**

```protobuf
message User {
  int32 id = 1;
  string name = 2;

  message Address {
    string street = 1;
    string city = 2;
    string state = 3;
    string zip = 4;
  }

  Address address = 3;
  repeated Address additional_addresses = 4;
}
```

**4. Use `repeated` for Arrays**

```protobuf
message UserList {
  repeated User users = 1;
}

message User {
  int32 id = 1;
  string name = 2;
  repeated string tags = 3;
}
```

**5. Use `oneof` for Union Types**

```protobuf
message SearchRequest {
  string query = 1;

  oneof filter {
    string category = 2;
    int32 user_id = 3;
    string tag = 4;
  }
}
```

**6. Use `google.protobuf` Well-Known Types**

```protobuf
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";
import "google/protobuf/empty.proto";
import "google/protobuf/wrappers.proto";

message Event {
  string id = 1;
  string name = 2;
  google.protobuf.Timestamp created_at = 3;
  google.protobuf.Duration duration = 4;
  google.protobuf.Int32Value optional_count = 5;
}
```

### Service Design Patterns

**1. Resource-Oriented Design**

Follow RESTful principles adapted for RPC:

```protobuf
service UserService {
  // Get single resource
  rpc GetUser(GetUserRequest) returns (User);

  // List resources
  rpc ListUsers(ListUsersRequest) returns (ListUsersResponse);

  // Create resource
  rpc CreateUser(CreateUserRequest) returns (User);

  // Update resource
  rpc UpdateUser(UpdateUserRequest) returns (User);

  // Delete resource
  rpc DeleteUser(DeleteUserRequest) returns (google.protobuf.Empty);
}
```

**2. Pagination Pattern**

```protobuf
message ListUsersRequest {
  int32 page_size = 1;
  string page_token = 2;
  string filter = 3;
}

message ListUsersResponse {
  repeated User users = 1;
  string next_page_token = 2;
  int32 total_count = 3;
}
```

**3. Batch Operations Pattern**

```protobuf
message BatchGetUsersRequest {
  repeated int32 user_ids = 1;
}

message BatchGetUsersResponse {
  map<int32, User> users = 1;
  repeated int32 not_found = 2;
}
```

**4. Long-Running Operations Pattern**

```protobuf
import "google/longrunning/operations.proto";

service BatchJobService {
  rpc ProcessBatch(BatchRequest) returns (google.longrunning.Operation);
  rpc GetOperation(GetOperationRequest) returns (google.longrunning.Operation);
}
```

## Streaming Patterns

### Server Streaming Patterns

**1. Pagination Streaming**

Stream large result sets efficiently:

```protobuf
service ProductService {
  rpc SearchProducts(SearchRequest) returns (stream Product);
}

message SearchRequest {
  string query = 1;
  int32 limit = 2;
}
```

**Implementation (Go):**
```go
func (s *server) SearchProducts(req *pb.SearchRequest, stream pb.ProductService_SearchProductsServer) error {
    products := s.db.Search(req.Query, req.Limit)

    for _, product := range products {
        if err := stream.Send(&product); err != nil {
            return err
        }
    }

    return nil
}
```

**2. Real-Time Updates**

Push updates to clients as they occur:

```protobuf
service EventService {
  rpc SubscribeToEvents(SubscribeRequest) returns (stream Event);
}

message SubscribeRequest