dotnet-file-io

What this skill does

# dotnet-file-io

File I/O patterns for .NET applications. Covers FileStream construction with async flags, RandomAccess API for thread-safe offset-based I/O, File convenience methods, FileSystemWatcher event handling and debouncing, MemoryMappedFile for large files and IPC, path handling security (Combine vs Join), secure temp file creation, cross-platform considerations, IOException hierarchy, and buffer sizing guidance.

**Out of scope:** PipeReader/PipeWriter and network I/O -- see [skill:dotnet-io-pipelines]. Async/await fundamentals -- see [skill:dotnet-csharp-async-patterns]. Span/Memory/ArrayPool deep patterns -- see [skill:dotnet-performance-patterns]. JSON and Protobuf serialization -- see [skill:dotnet-serialization]. BackgroundService lifecycle -- see [skill:dotnet-background-services]. Channel<T> producer/consumer -- see [skill:dotnet-channels]. GC implications of pinned/memory-mapped backing arrays -- see [skill:dotnet-gc-memory]. File upload validation (IFormFile) -- see [skill:dotnet-input-validation].

For testable file system access, consider `System.IO.Abstractions` -- see [skill:dotnet-testing-strategy] for test isolation patterns.

Cross-references: [skill:dotnet-io-pipelines] for PipeReader/PipeWriter network I/O, [skill:dotnet-gc-memory] for POH and memory-mapped backing array GC implications, [skill:dotnet-performance-patterns] for Span/Memory basics and ArrayPool usage, [skill:dotnet-csharp-async-patterns] for async/await patterns used with file streams.

---

## FileStream

### Async Flag Requirement

FileStream async methods (`ReadAsync`, `WriteAsync`) silently block the calling thread unless the stream is opened with the async flag. This is the most common file I/O mistake in .NET code.

```csharp
// CORRECT: async-capable FileStream
await using var fs = new FileStream(
    path,
    FileMode.Open,
    FileAccess.Read,
    FileShare.Read,
    bufferSize: 4096,
    useAsync: true);  // Required for true async I/O

byte[] buffer = new byte[4096];
int bytesRead = await fs.ReadAsync(buffer, cancellationToken);
```

```csharp
// ALSO CORRECT: FileOptions overload
await using var fs = new FileStream(
    path,
    FileMode.Create,
    FileAccess.Write,
    FileShare.None,
    bufferSize: 4096,
    FileOptions.Asynchronous | FileOptions.SequentialScan);
```

Without `useAsync: true` or `FileOptions.Asynchronous`, the runtime emulates async by dispatching synchronous I/O to the thread pool -- wasting a thread and adding overhead.

### FileStreamOptions (.NET 6+)

```csharp
await using var fs = new FileStream(path, new FileStreamOptions
{
    Mode = FileMode.Open,
    Access = FileAccess.Read,
    Share = FileShare.Read,
    Options = FileOptions.Asynchronous | FileOptions.SequentialScan,
    BufferSize = 4096,
    PreallocationSize = 1_048_576  // Hint for write: reduces fragmentation
});
```

`PreallocationSize` reserves disk space upfront when creating or overwriting files, reducing filesystem fragmentation on writes.

---

## RandomAccess API (.NET 6+)

`RandomAccess` provides static, offset-based, thread-safe file I/O. Unlike FileStream, it has no internal position state, so multiple threads can read/write different offsets concurrently without synchronization.

```csharp
using var handle = File.OpenHandle(
    path,
    FileMode.Open,
    FileAccess.Read,
    FileShare.Read,
    FileOptions.Asynchronous);

// Thread-safe: offset is explicit, no shared position
byte[] buffer = new byte[4096];
int bytesRead = await RandomAccess.ReadAsync(
    handle, buffer, fileOffset: 0, cancellationToken);

// Read from a different offset concurrently -- no locking needed
byte[] buffer2 = new byte[4096];
int bytesRead2 = await RandomAccess.ReadAsync(
    handle, buffer2, fileOffset: 8192, cancellationToken);
```

### Scatter/Gather I/O

```csharp
// Read into multiple buffers in a single syscall
IReadOnlyList<Memory<byte>> buffers = new[]
{
    new byte[4096].AsMemory(),
    new byte[4096].AsMemory()
};
long totalRead = await RandomAccess.ReadAsync(
    handle, buffers, fileOffset: 0, cancellationToken);
```

### When to Use RandomAccess vs FileStream

| Scenario | Use |
|----------|-----|
| Concurrent reads from different offsets | RandomAccess |
| Sequential streaming reads/writes | FileStream |
| Index files, database pages, memory-mapped alternatives | RandomAccess |
| Integration with Stream-based APIs | FileStream |

---

## File Convenience Methods

For small files where streaming is unnecessary, the `File` static methods are simpler and correct.

```csharp
// Read entire file as string (small files only)
string content = await File.ReadAllTextAsync(path, cancellationToken);

// Read all lines
string[] lines = await File.ReadAllLinesAsync(path, cancellationToken);

// Stream lines without loading entire file (.NET 8+)
await foreach (string line in File.ReadLinesAsync(path, cancellationToken))
{
    ProcessLine(line);
}

// Write text atomically (write-then-rename pattern not built-in)
await File.WriteAllTextAsync(path, content, cancellationToken);

// Read all bytes
byte[] data = await File.ReadAllBytesAsync(path, cancellationToken);
```

### When Convenience Methods Are Appropriate

| File size | Approach |
|-----------|----------|
| < 1 MB | `File.ReadAllTextAsync` / `File.ReadAllBytesAsync` |
| 1--100 MB | `File.ReadLinesAsync` or FileStream with buffered reading |
| > 100 MB | FileStream or RandomAccess with explicit buffer management |

---

## FileSystemWatcher

### Basic Setup

```csharp
using var watcher = new FileSystemWatcher(directoryPath)
{
    Filter = "*.json",
    NotifyFilter = NotifyFilters.FileName
                 | NotifyFilters.LastWrite
                 | NotifyFilters.Size,
    IncludeSubdirectories = true,
    EnableRaisingEvents = true
};

watcher.Changed += OnChanged;
watcher.Created += OnCreated;
watcher.Deleted += OnDeleted;
watcher.Renamed += OnRenamed;
watcher.Error += OnError;
```

### Debouncing Duplicate Events

FileSystemWatcher fires duplicate events for a single logical change (editors write temp file, rename, delete old). Debounce with a timer.

```csharp
public sealed class DebouncedFileWatcher : IDisposable
{
    private readonly FileSystemWatcher _watcher;
    private readonly Channel<string> _channel;
    private readonly CancellationTokenSource _cts = new();

    public DebouncedFileWatcher(string path, string filter)
    {
        _channel = Channel.CreateBounded<string>(
            new BoundedChannelOptions(100)
            {
                FullMode = BoundedChannelFullMode.DropOldest
            });

        _watcher = new FileSystemWatcher(path, filter)
        {
            EnableRaisingEvents = true
        };
        _watcher.Changed += (_, e) =>
            _channel.Writer.TryWrite(e.FullPath);
    }

    // requires: using System.Runtime.CompilerServices;
    public async IAsyncEnumerable<string> WatchAsync(
        TimeSpan debounce,
        [EnumeratorCancellation] CancellationToken ct = default)
    {
        using var linked = CancellationTokenSource
            .CreateLinkedTokenSource(ct, _cts.Token);
        var seen = new Dictionary<string, DateTime>();

        await foreach (var path in
            _channel.Reader.ReadAllAsync(linked.Token))
        {
            var now = DateTime.UtcNow;
            if (seen.TryGetValue(path, out var last)
                && now - last < debounce)
                continue;

            seen[path] = now;
            yield return path;
        }
    }

    public void Dispose()
    {
        _cts.Cancel();
        _watcher.Dispose();
        _cts.Dispose();
    }
}
```

### Buffer Overflow

The internal buffer defaults to 8 KB. When many changes occur rapidly, the buffer overflows and events are lost. Increase with `InternalBufferSize` (max 64 KB on Windows) and handle the `Error` event.

```csharp
watcher.InternalBufferSize = 65_536;  // 64 KB
watcher.Error += (_, e) =>
{
    if (e.GetException() is InternalBufferOverflowException)