Coursify

Memory Hierarchy Design Principles: Access Time, Cost, and Size Trade-offs

Memory Hierarchy Design Principles: Access Time, Cost, and Size Trade-offs

Verified Sources
Nov 4, 2026

A computer’s memory hierarchy is designed to reconcile three opposing goals: (1) fast access time, (2) reasonable cost per bit, and (3) large capacity. This is done by arranging storage from small/fast to large/slower (e.g., registers → caches → DRAM → storage) and by using locality-based policies so that most accesses hit in faster levels. The hierarchy’s effectiveness is evaluated using effective access time models that combine hit time and miss penalty, which originate in cache behavior and lower-level latencies.2

The core design principles include:

  • Exploiting locality with caches sized to cover the working set.
  • Leveraging the trade-off curve between access latency and storage density across technologies (e.g., SRAM vs DRAM).
  • Choosing structure parameters (block/line size, associativity, replacement policy) that balance miss rate vs hit time/cost.2

Footnotes

  1. Computer Architecture: A Quantitative Approach (cache basics and locality). - Course notes summarizing locality, caches, and cache organization trade-offs. ↩

  2. Hennessy & Patterson AMAT / cache performance model (hit time & miss penalty). - Lecture materials commonly deriving AMAT from hit/miss behavior. ↩

  3. SRAM vs DRAM: latency and density trade-offs overview. - Describes SRAM behavior and contrasts with DRAM. ↩

  4. DRAM overview: cost/density vs latency context. - Explains DRAM’s density and latency characteristics. ↩

Cache Fundamentals and the Memory Hierarchy (Locality + Hit/Miss/AMAT)

Key terminology and the AMAT framework

To reason about performance, architectures use the hit/miss model. If a cache has hit time thitt_{hit} and miss penalty tmisst_{miss}, then the average memory access time (often called AMAT) can be expressed as:

tavg=(1−pmiss)⋅thit+pmiss⋅tmisst_{avg} = (1 - p_{miss}) \cdot t_{hit} + p_{miss} \cdot t_{miss}

where pmissp_{miss} is the miss probability.2 This formula shows why memory hierarchy design is fundamentally a trade-off optimization:

  • Making a cache faster reduces thitt_{hit} (often also reducing bandwidth/size or increasing area/power costs).
  • Making a cache “bigger” may reduce pmissp_{miss} (by capturing more of the working set), but typically increases hit time due to larger structures and wiring.2
  • Miss penalty tmisst_{miss} depends on the latency of lower levels (e.g., L2/DRAM) and the mechanism used to fetch/serve misses.2

Design implication: you typically cannot minimize tavgt_{avg} by optimizing only one parameter; memory hierarchy levels must be co-designed.

Footnotes

  1. Hennessy & Patterson AMAT / cache performance model (hit time & miss penalty). - Lecture materials commonly deriving AMAT from hit/miss behavior. ↩ ↩2

  2. Cache performance and miss penalty discussions. - Covers hit/miss concepts and performance impacts. ↩ ↩2

  3. Computer Architecture: A Quantitative Approach (cache basics and locality). - Course notes summarizing locality, caches, and cache organization trade-offs. ↩

  4. Locality of reference and its role in caching. - Defines temporal/spatial locality and explains why caches work. ↩

From access-time, cost, and size to a concrete hierarchy decision

  1. 1
    Step 1

    Determine which level dominates tavgt_{avg} by profiling: compute miss rate per level and estimate miss penalties to lower levels.

  2. 2
    Step 2

    Estimate whether the application’s working set fits within candidate cache capacities; locality principles suggest temporal/spatial reuse.2

    Footnotes

    1. Computer Architecture: A Quantitative Approach (cache basics and locality). - Course notes summarizing locality, caches, and cache organization trade-offs. ↩

    2. DRAM overview: cost/density vs latency context. - Explains DRAM’s density and latency characteristics. ↩

  3. 3
    Step 3

    Choose a cache line size that captures spatial locality: if a program accesses nearby addresses, larger blocks can reduce miss rate, but increase wasted bandwidth and can worsen hit time if structures grow.2

    Footnotes

    1. Computer Architecture: A Quantitative Approach (cache basics and locality). - Course notes summarizing locality, caches, and cache organization trade-offs. ↩

    2. Locality of reference and its role in caching. - Defines temporal/spatial locality and explains why caches work. ↩

  4. 4
    Step 4

    Set associativity to reduce conflict misses; pick replacement policies consistent with observed reuse patterns (e.g., LRU approximations vs cheaper heuristics).

  5. 5
    Step 5

    Use device-level trade-offs (SRAM vs DRAM) to set realistic access-time targets for each hierarchy level; lower levels are slower but offer larger capacity per cost.2

    Footnotes

    1. SRAM vs DRAM: latency and density trade-offs overview. - Describes SRAM behavior and contrasts with DRAM. ↩

    2. DRAM overview: cost/density vs latency context. - Explains DRAM’s density and latency characteristics. ↩

  6. 6
    Step 6

    Compute AMAT-like estimates (including miss penalties) and check whether the chosen configuration meets area/power/cost budgets.

Why access time, cost, and size pull design in different directions

1) Access time: speed increases as you move toward the top

Caches are faster because they are built with faster memory and proximity to the CPU pipeline. However, increasing cache complexity (e.g., larger arrays, more associativity, longer tag checks, wider buses) increases critical path and often hit time.

A typical hierarchy statement is: hit time grows with cache size/complexity, while miss rate shrinks with cache capacity—so designers search for the best balance using AMAT.2

2) Cost: fast memory is expensive per bit

The hierarchy exists because fast storage is not the most dense (and therefore not the cheapest per bit). For example, SRAM is faster than DRAM but has lower density and higher cost per bit, which encourages using SRAM for small, fast caches and DRAM for large main memory.2

3) Size/capacity: larger structures reduce misses via working-set coverage

If a cache is too small, it cannot retain the active data/instructions set, increasing pmissp_{miss}. If it is large enough, more accesses become hits, lowering average access time. This is the practical meaning of locality and the working set: a larger cache better covers the active region during periods of execution.3

Footnotes

  1. Hennessy & Patterson AMAT / cache performance model (hit time & miss penalty). - Lecture materials commonly deriving AMAT from hit/miss behavior. ↩

  2. Cache performance and miss penalty discussions. - Covers hit/miss concepts and performance impacts. ↩

  3. SRAM vs DRAM: latency and density trade-offs overview. - Describes SRAM behavior and contrasts with DRAM. ↩

  4. DRAM overview: cost/density vs latency context. - Explains DRAM’s density and latency characteristics. ↩ ↩2

  5. Computer Architecture: A Quantitative Approach (cache basics and locality). - Course notes summarizing locality, caches, and cache organization trade-offs. ↩

  6. Locality of reference and its role in caching. - Defines temporal/spatial locality and explains why caches work. ↩

Cache organizations: how block size, associativity, and replacement affect tavgt_{avg}

Spatial locality and cache line/block size

If programs tend to access memory locations near each other (spatial locality), fetching a whole cache line reduces misses for subsequent accesses. But larger lines may:

  • increase unused fetched data (wasted bandwidth/capacity),
  • increase miss penalty (more data transferred),
  • increase hit time (larger metadata/structures).2

Conflict misses and associativity

Even with enough total capacity, limited mapping (set index + tag) can cause different blocks to contend for the same set. Increasing associativity can reduce conflict misses by giving more candidate locations per set at the cost of more complex lookup and slightly higher thitt_{hit}.2

Replacement policy and effective locality capture

Replacement policy determines which blocks survive eviction when the working set doesn’t perfectly fit. Policies that better approximate true reuse patterns can reduce pmissp_{miss}, but hardware cost (and potential lookup time) may increase.2

Mermaid view of the interaction

Footnotes

  1. Computer Architecture: A Quantitative Approach (cache basics and locality). - Course notes summarizing locality, caches, and cache organization trade-offs. ↩ ↩2 ↩3

  2. Locality of reference and its role in caching. - Defines temporal/spatial locality and explains why caches work. ↩ ↩2

  3. Hennessy & Patterson AMAT / cache performance model (hit time & miss penalty). - Lecture materials commonly deriving AMAT from hit/miss behavior. ↩

type="tip" title="Pro Tip: Optimize AMAT, not just hit rate" content="When comparing designs, prefer tavg=(1−pmiss)thit+pmisstmisst_{avg}=(1-p_{miss})t_{hit}+p_{miss}t_{miss} over “hit rate only”. A design with a slightly higher hit time can still win if it sharply reduces pmissp_{miss} or tmisst_{miss}.2"

Footnotes

  1. Hennessy & Patterson AMAT / cache performance model (hit time & miss penalty). - Lecture materials commonly deriving AMAT from hit/miss behavior. ↩

  2. Cache performance and miss penalty discussions. - Covers hit/miss concepts and performance impacts. ↩

type="warning" title="Beware of “bigger is always better”" content="Increasing cache size reduces misses until the working set fits, but further growth can raise thitt_{hit} and cost/power. Beyond that point, AMAT can worsen because hit time increases faster than miss rate decreases.2"

Footnotes

  1. Computer Architecture: A Quantitative Approach (cache basics and locality). - Course notes summarizing locality, caches, and cache organization trade-offs. ↩

  2. Locality of reference and its role in caching. - Defines temporal/spatial locality and explains why caches work. ↩

Typical progression through memory hierarchy levels

Registers

1 (very fast)

Smallest storage; fastest access; managed by the compiler/CPU pipeline."

L1/L2 caches

2 (fast caches)

Exploit temporal/spatial locality; hit time is low; misses incur penalties."

DRAM main memory

3 (larger, slower)

Large capacity at lower cost per bit than SRAM; higher latency drives miss penalties."

SSD/HDD

4 (very large, slow)

Used via virtual memory and paging; serves as a last resort with very high latency."

Qualitative trade-off: latency vs capacity vs cost across hierarchy levels

Higher levels: lower latency but smaller capacity; lower levels: larger capacity but higher latency and typically lower cost per bit.

Design questions and edge cases

Knowledge Check

Question 1 of 4
Q1Single choice

In the AMAT model tavg=(1−pmiss)thit+pmisstmisst_{avg}=(1-p_{miss})t_{hit}+p_{miss}t_{miss}, which design change most directly reduces the second term when miss probability dominates?

Explore Related Topics

1

Hierarchical Memory Structure: Detailed Explanation with Diagram

2

Memory Allocation with First-Fit and Best-Fit

3

Cache Memory Mapping: Direct, Fully Associative, and Set-Associative Procedures

Browse all research articles