Learn: System Performance - Part 3

Overview

Welcome! In this session, we’re diving deep into two essential pillars of system performance: caching mechanisms and deadlock handling, especially as they play out in large-scale and distributed systems. You'll gain practical insight into why caching is fundamental for performance, how to tackle tricky issues like cache stampedes and coherence delays, and what strategies and data structures actually work in the real world. We’ll also unravel deadlocks—what causes them, how to handle them, and how to avoid common traps in resource management. By the end, you’ll be equipped to analyze, design, and troubleshoot performant systems with confidence.

Concept-by-Concept Deep Dive

1. Caching Mechanisms and Strategies

What is Caching?

Caching is the technique of storing copies of data in a high-speed storage layer (the "cache") so that future requests for that data can be served faster. It's especially vital for dynamic or frequently accessed content, significantly reducing response times and backend load.

Core Strategies:

• Read-Through vs. Write-Through: Read-through caches fetch data from the underlying data store when a cache miss occurs, then populate the cache. Write-through caches update both the cache and the data store on writes, ensuring consistency but possibly adding latency.
• Cache Aside (Lazy Loading): The application only queries the cache first and loads from the database only on a cache miss, placing the response in the cache for subsequent requests.
• Write-Back (Write-Behind): Data is written to the cache and only periodically flushed to the data store, which can improve write performance but risks data loss on failure.

Step-by-Step Reasoning:

1. Identify Data Access Patterns: Is your workload read-heavy, write-heavy, or balanced?
2. Choose a Caching Strategy: For read-heavy workloads, cache-aside or read-through is often best. Write-heavy applications may need stronger consistency (e.g., write-through).
3. Implement Eviction Policy: Select a policy (e.g., LRU, LFU) to decide which items to evict when the cache is full.
4. Handle Consistency: Decide how to synchronize cache and source of truth (database).

Common Misconceptions:

• "Caching always improves performance": Not always true. Incorrect configuration, stale data, or poorly chosen eviction policies can hurt performance or consistency.
• "One caching strategy fits all": Workload and data volatility dictate the best approach.

---

2. Cache Stampede and Mitigation Techniques

What is a Cache Stampede?

A cache stampede occurs when many requests simultaneously miss the cache and hit the backend, often overwhelming the underlying data store. This commonly happens when a popular cached item expires or is invalidated.

Techniques to Prevent Stampedes:

• Locking (Mutex/Leases): When a cache miss occurs, the first requester acquires a lock to recompute and repopulate the cache, while others wait or use stale data.
• Request Coalescing: Aggregate multiple identical requests and return the same response once computed.
• Stale-While-Revalidate: Serve stale data while asynchronously refreshing the cache in the background.

Step-by-Step Recipe:

1. Detect a cache miss for a hot item.
2. Acquire a lock or check if a recomputation is in progress.
3. If locked, either wait or serve stale data.
4. Once recomputed, repopulate the cache and release the lock.

Common Misconceptions:

• "Cache expiration is harmless": Not if many clients request the same data at once.
• "More frequent cache invalidation is always better": It may increase the risk of stampedes.

---

3. Cache Coherence and Consistency in Distributed Systems

What is Cache Coherence?

Cache coherence ensures that multiple distributed caches reflect the same, up-to-date view of data. Coherence becomes critical in CDNs, global distributed caches, or microservices where data is cached in many locations and updated frequently.

Impact of Coherence-Related Delays:

• In Distributed Transactions: Delays can cause inconsistent reads or write conflicts, impacting transaction isolation and system correctness.
• In CDNs/Content Delivery: Slow propagation of new content can lead to users receiving stale data, or increased latency as caches synchronize.

Techniques to Manage Coherence:

• Invalidation Protocols: Push or broadcast updates to all nodes when data changes.
• Versioning and ETags: Attach version info to cached data, so stale copies can be detected and refreshed.
• Lease-based approaches: Allow caches to serve data only for a set lease period before revalidation.

Common Misconceptions:

• "CDNs always serve the latest version": They may serve stale content if propagation is delayed.
• "Consistency is easy in distributed caches": It’s notoriously hard, especially at global scale.

---

4. Cache Eviction Policies and Data Structures

What Are Eviction Policies?

Eviction policies determine which data to remove when the cache is full. The goal is to retain items most likely to be needed soon.