Approximate cost to access various caches and main memory?

Question

Can anyone give me the approximate time (in nanoseconds) to access L1, L2 and L3 caches, as well as main memory on Intel i7 processors?

While this isn't specifically a programming question, knowing these kinds of speed details is neccessary for some low-latency programming challenges.

Distorted

mechanical-sympathy.blogspot.com/2013/02/… – Everrs

How do I convert ns to cycles? If I simply divide 100 ns by 2.3 GHz, I get 230 cycles. Is this correct? – Impersonality

I'm curious: Under what situation is the remote L3 cache slower than the remote DRAM? The number above indicates that it can be 1.6x as slow. – Naseby

Please don't edit the question but instead post an answer with those details. Self-answering is ok on SO. – Ader

Are there any approximate values for energy consumption for memory access from each level? – Roughhew

@Nathan, you multiply by 2.3 cyc/1 ns (because 1 s = 10^9 ns, 1 Hz = 1 / 1 s and G = 10^9). – Cere

Answer 1

Here is a Performance Analysis Guide for the i7 and Xeon range of processors. I should stress, this has what you need and more (for example, check page 22 for some timings & cycles for example).

Additionally, this page has some details on clock cycles etc. The second link served the following numbers:

Core i7 Xeon 5500 Series Data Source Latency (approximate)               [Pg. 22]

local  L1 CACHE hit,                              ~4 cycles (   2.1 -  1.2 ns )
local  L2 CACHE hit,                             ~10 cycles (   5.3 -  3.0 ns )
local  L3 CACHE hit, line unshared               ~40 cycles (  21.4 - 12.0 ns )
local  L3 CACHE hit, shared line in another core ~65 cycles (  34.8 - 19.5 ns )
local  L3 CACHE hit, modified in another core    ~75 cycles (  40.2 - 22.5 ns )

remote L3 CACHE (Ref: Fig.1 [Pg. 5])        ~100-300 cycles ( 160.7 - 30.0 ns )

local  DRAM                                                   ~60 ns
remote DRAM                                                  ~100 ns

EDIT2:
The most important is the notice under the cited table, saying:

_{"NOTE: THESE VALUES ARE ROUGH APPROXIMATIONS. THEY DEPEND ON CORE AND UNCORE FREQUENCIES, MEMORY SPEEDS, BIOS SETTINGS, NUMBERS OF DIMMS, ETC,ETC..YOUR MILEAGE MAY VARY."}

EDIT: I should highlight that, as well as timing/cycle information, the above intel document addresses much more (extremely) useful details of the i7 and Xeon range of processors (from a performance point of view).

Rafaelrafaela

Shouldn't 'line unshared' have more latency than 'shared line in another core' -- a shared line (i.e. 2 core valid bits) means it can be taken directly from the LLC slice as it is guaranteed to be clean. 'Line unshared' means there is only one core valid bit and that core has to be snooped to ensure that the line is exclusive and not modified -- if it is modified then it is changed to shared; LLC now becomes dirty and it is returned to the requesting core as shared. Maybe I'm wrong -- I know the MOESI protocol is different. – Hanoverian

Certainly this is the case in SnB and Haswell. Nehalem--which this Xeon uses--was before the ring bus topology and had a unified cache but I don't see why the snoop filter would behave any differently in Nehalem. The optimisation manual section B.3.5.3 gives what I feel like is an incorrect description (it clearly pertains to Nehalem as it speaks about the Global Queue which is a Nehalem feature). This Haswell paper has a better description (top right column of page 5)(tu-dresden.de/zih/forschung/ressourcen/dateien/…) – Hanoverian

@LewisKelsey: This is surprising to me, too, because I thought half the point of inclusive L3 was that L3 could simply respond if it had a valid copy of a line. But remember, Intel uses MESIF (en.wikipedia.org/wiki/MESIF_protocol) for NUMA, AMD uses MOESI. I think within a single socket, though, MESIF isn't really a thing because data comes from L3, not core->core. So it's probably more relevant for L3 cache->cache transfers across sockets. I wonder if this "local L3 hit" is for a line shared with a core in another socket? Still doesn't make sense, valid in L3 means no core has E/M – Reeding

@PeterCordes I remembered this comment and came back and what I said just came across as immediately wrong to me. My comment is correct in the perspective of a 3rd core where it is shared between 2 other cores or just exclusive to one other core. But if you're talking about line unshared and it belongs to the core that is trying to access the line, then the benchmark is right because shared requires an RFO to get it exclusive and exclusive means there is no such RFO required. So I don't know what I was saying really. – Hanoverian

@LewisKelsey: Yeah, that's all true for writing. I thought this was for reading (Data Source Latency), which is more latency-sensitive. Reading a line never requires an RFO, just a request to share. So shouldn't a line that's already in Shared state somewhere, just hit in this socket's L3 without having to wait for coherency traffic? And thus be faster than DRAM, similar to an "unshared" L3 hit. – Reeding

Answer 2

Numbers everyone should know

           0.5 ns - CPU L1 dCACHE reference
           1   ns - speed-of-light (a photon) travel a 1 ft (30.5cm) distance
           5   ns - CPU L1 iCACHE Branch mispredict
           7   ns - CPU L2  CACHE reference
          71   ns - CPU cross-QPI/NUMA best  case on XEON E5-46*
         100   ns - MUTEX lock/unlock
         100   ns - own DDR MEMORY reference
         135   ns - CPU cross-QPI/NUMA best  case on XEON E7-*
         202   ns - CPU cross-QPI/NUMA worst case on XEON E7-*
         325   ns - CPU cross-QPI/NUMA worst case on XEON E5-46*
      10,000   ns - Compress 1K bytes with Zippy PROCESS
      20,000   ns - Send 2K bytes over 1 Gbps NETWORK
     250,000   ns - Read 1 MB sequentially from MEMORY
     500,000   ns - Round trip within a same DataCenter
  10,000,000   ns - DISK seek
  10,000,000   ns - Read 1 MB sequentially from NETWORK
  30,000,000   ns - Read 1 MB sequentially from DISK
 150,000,000   ns - Send a NETWORK packet CA -> Netherlands
|   |   |   |
|   |   | ns|
|   | us|
| ms|

From: Originally by Peter Norvig:
- http://norvig.com/21-days.html#answers
- http://surana.wordpress.com/2009/01/01/numbers-everyone-should-know/,
- http://sites.google.com/site/io/building-scalable-web-applications-with-google-app-engine

Aten

Surely these care very HUGE amounts, based on processor design, ram latency/frequency, hard disk caching(both type and size)/rpm etc etc? To Quote INTEL (for values they released for one specific CPU):"NOTE:These values are rough approximations. They depend on Core and Uncore Frequencies, Memory speeds, BIOS settings, Number of DIMMS, etc etc. YOUR MILEAGE MAY VARY..." – Rafaelrafaela

@Rafaelrafaela that's true, but this numbers show the order of magnitude – Aten

@Dave, even though type/speed/architecture of the cpu is different, I believe the relative timing should roughly remain same, so it's just a rough guideline to know when you code. More meaningful analysis should be done via profiler of course... – Tatiana

To have an idea of how much time it is, Wikipedia mentions "One nanosecond is to one second as one second is to 31.7 years." en.wikipedia.org/wiki/Nanosecond – Oversoul

Since some time passed: I'm thinking whether these numbers are still always true. For me it depends on the preconditions, say, you're running the same instruction 10k times. The speed of referencing L1 cache will significantly increase, since the processor is a pipeline and earlier computations already acquired relevant information (e.g. referencing L1). What I'm asking is: the time needed to compute one instruction comes with one overhead. But the overhead remains the same when executing the instruction multiple times. Is that correct? – Homoeo

@Homoeo Speed of referencing L1 or any other cache doesn't depend on preconditions. Preconditions can affect if reference takes place or not. – Aten

You mean a cache miss penalty? So the numbers above are still reliable, even with current processor architectures? – Homoeo

@Homoeo if there is cache miss it means that it will require access of lower level cache or even main memory. In this case it will take time according to that level access time. You can look for data for newer CPUs here sisoftware.net/?d=qa&f=ben_mem_latency – Aten

How do the main memory numbers relate to CAS latency of RAM? – Enamour

Answer 3

Cost to access various memories in a pretty page

• See this page presenting the memory latency decrease from 1990 to 2020.

Summary

1. Values having decreased but are stabilized since 2005

           1 ns        L1 cache
           3 ns        Branch mispredict
           4 ns        L2 cache
          17 ns        Mutex lock/unlock
         100 ns        Main memory (RAM)
       2 000 ns (2µs)  1KB Zippy-compress
   

2. Still some improvements, prediction for 2020

      16 000 ns (16µs) SSD random read (olibre's note: should be less)
     500 000 ns (½ms)  Round trip in datacenter
   2 000 000 ns (2ms)  HDD random read (seek)
   

See also other sources

• What every programmer should know about memory from Ulrich Drepper (2007)
  Old but still an excellent deep explanation about memory hardware and software interaction.
  • Full PDF (114 pages)
    • Comments on LWN about PDF version
    • Another ones
  • Seven posts on LWN + Comments
    • Part 1 - Introduction
    • Part 2 - Cache
    • Part 3 - Virtual Memory
    • Part 4 - NUMA support
    • Part 5 - What programmers can do
    • Part 6 - More things programmers can do
    • Part 7 - Memory performance tools
• Post The Infinite Space Between Words in codinghorror.com based on book Systems Performance: Enterprise and the Cloud
• Click to each processor listed on http://www.7-cpu.com/ to see the L1/L2/L3/RAM/... latencies (e.g. Haswell i7-4770 has L1=1ns, L2=3ns, L3=10ns, RAM=67ns, BranchMisprediction=4ns)
• http://idarkside.org/posts/numbers-you-should-know/

See also

For further understanding, I recommend the excellent presentation of modern cache architectures (June 2014) from Gerhard Wellein, Hannes Hofmann and Dietmar Fey at University Erlangen-Nürnberg.

French speaking people may appreciate an article by SpaceFox comparing a processor with a developer both waiting for information required to continue to work.

Pause

a lovely latency post. would be good to add the facts about GPU-latency masking reality ( – Iquique

Hi @Iquique Have you some sources about memory latency related to GPU? Cheers :-) – Pause

Certainly, yes, @olibre. Check the [A] posted below. – Iquique

Given how this is about latency and caching, I find it ironic that the page at your first link, with the year slider, does not cache the metric display when changing the year. In Firefox, at least, they render too slowly for dragging across the years to be smooth : / – Concatenation

Nice references, you gave titles and authors! – Aswan

Answer 4

Just for a sake of 2020's review of the predictions for 2025:

The last about 44 years of the integrated circuit technology, the classical (non-quantum) processors evolved, literally and physically "Per Aspera ad Astra". The last decade has evidenced, the classical process has got close to some hurdles, that do not have an achievable physical path forward.

Number of logical cores can and may grow, yet not more than O(n^2~3)
Frequency [MHz] has hard if not impossible to circumvent physics-based ceiling already hit
Transistor Count can and may grow, yet less than O(n^2~3) ( power, noise, "clock")
Power [W] can grow, yet problems with power distribution & heat dissipation will increase
Single Thread Perf may grow, having direct benefits from large cache-footprints and faster and wider memory-I/O & indirect benefits from less often system forced context-switching as we can have more cores to split other threads/processes among

_{( Credits go to Leonardo Suriano & Karl Rupp )}

    2022: Still some improvements, prediction for 2025+
--------------------------------------------------------------------------------
                 0.001 ns light transfer in Gemmatimonas phototrophica bacteriae
      |   |   |   |   |
      |   |   |   | ps|
      |   |   | ns|
      |   | us|        reminding us what Richard FEYNMAN told us:
      | ms|                             "There's a plenty of space
     s|                                                      down there"

-----s.-ms.-us.-ns|----------------------------------------------------------
                 0.1 ns - NOP
                 0.3 ns - XOR, ADD, SUB
                 0.5 ns - CPU L1 dCACHE reference           (1st introduced in late 80-ies )
                 0.9 ns - JMP SHORT
                 1   ns - speed-of-light (a photon) travel a 1 ft (30.5cm) distance -- will stay, throughout any foreseeable future :o)
    ?~~~~~~~~~~~ 1   ns - MUL ( i**2 = MUL i, i )~~~~~~~~~ doing this 1,000 x is 1 [us]; 1,000,000 x is 1 [ms]; 1,000,000,000 x is 1 [s] ~~~~~~~~~~~~~~~~~~~~~~~~~
               3~4   ns - CPU L2  CACHE reference           (2020/Q1)
                 5   ns - CPU L1 iCACHE Branch mispredict
                 7   ns - CPU L2  CACHE reference
                10   ns - DIV
                19   ns - CPU L3  CACHE reference           (2020/Q1 considered slow on 28c Skylake)
                71   ns - CPU cross-QPI/NUMA best  case on XEON E5-46*
               100   ns - MUTEX lock/unlock
               100   ns - own DDR MEMORY reference
               135   ns - CPU cross-QPI/NUMA best  case on XEON E7-*
               202   ns - CPU cross-QPI/NUMA worst case on XEON E7-*
               325   ns - CPU cross-QPI/NUMA worst case on XEON E5-46*
    |Q>~~~~~ 5,000   ns - QPU on-chip QUBO ( quantum annealer minimiser 1 Qop )
            10,000   ns - Compress 1K bytes with a Zippy PROCESS
            20,000   ns - Send     2K bytes over 1 Gbps  NETWORK
           250,000   ns - Read   1 MB sequentially from  MEMORY
           500,000   ns - Round trip within a same DataCenter
    ?~~~ 2,500,000   ns - Read  10 MB sequentially from  MEMORY~~(about an empty python process to copy on spawn)~~~~ x ( 1 + nProcesses ) on spawned process instantiation(s), yet an empty python interpreter is indeed not a real-world, production-grade use-case, is it?
        10,000,000   ns - DISK seek
        10,000,000   ns - Read   1 MB sequentially from  NETWORK
    ?~~ 25,000,000   ns - Read 100 MB sequentially from  MEMORY~~(somewhat light python process to copy on spawn)~~~~ x ( 1 + nProcesses ) on spawned process instantiation(s)
        30,000,000   ns - Read 1 MB sequentially from a  DISK
    ?~~ 36,000,000   ns - Pickle.dump() SER a 10 MB object for IPC-transfer and remote DES in spawned process~~~~~~~~ x ( 2 ) for a single 10MB parameter-payload SER/DES + add an IPC-transport costs thereof or NETWORK-grade transport costs, if going into [distributed-computing] model Cluster ecosystem
       150,000,000   ns - Send a NETWORK packet CA -> Netherlands
    1s:   |   |   |
      .   |   | ns|
      .   | us|
      . ms|

Just for a sake of 2015's review of the predictions for 2020:

Still some improvements, prediction for 2020 (Ref. olibre's answer below)

            16 000 ns ( 16 µs) SSD random read (olibre's note: should be less)
           500 000 ns (  ½ ms) Round trip in datacenter
         2 000 000 ns (  2 ms) HDD random read (seek)
    1s:   |   |   |
      .   |   | ns|
      .   | us|
      . ms|

In 2015 there are currently available:
======================================
               820 ns ( 0.8µs) random read from a SSD-DataPlane
             1 200 ns ( 1.2µs) Round trip in datacenter
             1 200 ns ( 1.2µs) random read from a HDD-DataPlane
    1s:   |   |   |
      .   |   | ns|
      .   | us|
      . ms|

Just for a sake of CPU and GPU latency landscape comparison:

Not an easy task to compare even the simplest CPU / cache / DRAM lineups ( even in a uniform memory access model ), where DRAM-speed is a factor in determining latency, and loaded latency (saturated system), where the latter rules and is something the enterprise applications will experience more than an idle fully unloaded system.

                    +----------------------------------- 5,6,7,8,9,..12,15,16 
                    |                               +--- 1066,1333,..2800..3300
                    v                               v
First  word = ( ( CAS latency * 2 ) + ( 1 - 1 ) ) / Data Rate  
Fourth word = ( ( CAS latency * 2 ) + ( 4 - 1 ) ) / Data Rate
Eighth word = ( ( CAS latency * 2 ) + ( 8 - 1 ) ) / Data Rate
                                        ^----------------------- 7x .. difference
******************************** 
So:
===

resulting DDR3-side latencies are between _____________
                                          3.03 ns    ^
                                                     |
                                         36.58 ns ___v_ based on DDR3 HW facts

GPU-engines have received a lot of technical marketing, while deep internal dependencies are keys to understand both the real strengths and also the real weaknesses these architectures experience in practice ( typically much different than the aggressive marketing whistled-up expectations ).

   1 ns _________ LETS SETUP A TIME/DISTANCE SCALE FIRST:
          °      ^
          |\     |a 1 ft-distance a foton travels in vacuum ( less in dark-fibre )
          | \    |
          |  \   |
        __|___\__v____________________________________________________
          |    |
          |<-->|  a 1 ns TimeDOMAIN "distance", before a foton arrived
          |    |
          ^    v 
    DATA  |    |DATA
    RQST'd|    |RECV'd ( DATA XFER/FETCH latency )

  25 ns @ 1147 MHz FERMI:  GPU Streaming Multiprocessor REGISTER access
  35 ns @ 1147 MHz FERMI:  GPU Streaming Multiprocessor    L1-onHit-[--8kB]CACHE

  70 ns @ 1147 MHz FERMI:  GPU Streaming Multiprocessor SHARED-MEM access

 230 ns @ 1147 MHz FERMI:  GPU Streaming Multiprocessor texL1-onHit-[--5kB]CACHE
 320 ns @ 1147 MHz FERMI:  GPU Streaming Multiprocessor texL2-onHit-[256kB]CACHE

 350 ns
 700 ns @ 1147 MHz FERMI:  GPU Streaming Multiprocessor GLOBAL-MEM access
 - - - - -

Understanding internalities is thus much more important, than in other fields, where architectures are published and numerous benchmarks freely available. Many thanks to GPU-micro-testers, who 've spent their time and creativity to unleash the truth of the real schemes of work inside the black-box approach tested GPU devices.

    +====================| + 11-12 [usec] XFER-LATENCY-up   HostToDevice    ~~~ same as Intel X48 / nForce 790i
    |   |||||||||||||||||| + 10-11 [usec] XFER-LATENCY-down DeviceToHost
    |   |||||||||||||||||| ~  5.5 GB/sec XFER-BW-up                         ~~~ same as DDR2/DDR3 throughput
    |   |||||||||||||||||| ~  5.2 GB/sec XFER-BW-down @8192 KB TEST-LOAD      ( immune to attempts to OverClock PCIe_BUS_CLK 100-105-110-115 [MHz] ) [D:4.9.3]
    |                       
    |              Host-side
    |                                                        cudaHostRegister(   void *ptr, size_t size, unsigned int flags )
    |                                                                                                                 | +-------------- cudaHostRegisterPortable -- marks memory as PINNED MEMORY for all CUDA Contexts, not just the one, current, when the allocation was performed
    |                        ___HostAllocWriteCombined_MEM / cudaHostFree()                                           +---------------- cudaHostRegisterMapped   -- maps  memory allocation into the CUDA address space ( the Device pointer can be obtained by a call to cudaHostGetDevicePointer( void **pDevice, void *pHost, unsigned int flags=0 ); )
    |                        ___HostRegisterPORTABLE___MEM / cudaHostUnregister( void *ptr )
    |   ||||||||||||||||||
    |   ||||||||||||||||||
    |   | PCIe-2.0 ( 4x) | ~ 4 GB/s over  4-Lanes ( PORT #2  )
    |   | PCIe-2.0 ( 8x) | ~16 GB/s over  8-Lanes
    |   | PCIe-2.0 (16x) | ~32 GB/s over 16-Lanes ( mode 16x )
    |
    |   + PCIe-3.0 25-port 97-lanes non-blocking SwitchFabric ... +over copper/fiber
    |                                                                       ~~~ The latest PCIe specification, Gen 3, runs at 8Gbps per serial lane, enabling a 48-lane switch to handle a whopping 96 GBytes/sec. of full duplex peer to peer traffic. [I:]
    |
    | ~810 [ns]    + InRam-"Network" / many-to-many parallel CPU/Memory "message" passing with less than 810 ns latency any-to-any
    |
    |   ||||||||||||||||||
    |   ||||||||||||||||||
    +====================|
    |.pci............HOST|

---

My apology for a "bigger-picture", but latency-demasking has also cardinal limits imposed from on-chip smREG/L1/L2-capacities and hit/miss-rates.

    |.pci............GPU.|
    |                    | FERMI [GPU-CLK] ~ 0.9 [ns] but THE I/O LATENCIES                                                                  PAR -- ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| <800> warps ~~ 24000 + 3200 threads ~~ 27200 threads [!!]
    |                                                                                                                                               ^^^^^^^^|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [!!]
    |                                                       smREGs________________________________________ penalty +400 ~ +800 [GPU_CLKs] latency ( maskable by 400~800 WARPs ) on <Compile-time>-designed spillover(s) to locMEM__
    |                                                                                                              +350 ~ +700 [ns] @1147 MHz FERMI ^^^^^^^^
    |                                                                                                                          |                    ^^^^^^^^
    |                                                                                                                       +5 [ns] @ 200 MHz FPGA. . . . . . Xilinx/Zync Z7020/FPGA massive-parallel streamline-computing mode ev. PicoBlazer softCPU
    |                                                                                                                          |                    ^^^^^^^^
    |                                                                                                                   ~  +20 [ns] @1147 MHz FERMI ^^^^^^^^
    |                                                             SM-REGISTERs/thread: max  63 for CC-2.x -with only about +22 [GPU_CLKs] latency ( maskable by 22-WARPs ) to hide on [REGISTER DEPENDENCY] when arithmetic result is to be served from previous [INSTR] [G]:10.4, Page-46
    |                                                                                  max  63 for CC-3.0 -          about +11 [GPU_CLKs] latency ( maskable by 44-WARPs ) [B]:5.2.3, Page-73
    |                                                                                  max 128 for CC-1.x                                    PAR -- ||||||||~~~|
    |                                                                                  max 255 for CC-3.5                                    PAR -- ||||||||||||||||||~~~~~~|
    |
    |                                                       smREGs___BW                                 ANALYZE REAL USE-PATTERNs IN PTX-creation PHASE <<  -Xptxas -v          || nvcc -maxrregcount ( w|w/o spillover(s) )
    |                                                                with about 8.0  TB/s BW            [C:Pg.46]
    |                                                                           1.3  TB/s BW shaMEM___  4B * 32banks * 15 SMs * half 1.4GHz = 1.3 TB/s only on FERMI
    |                                                                           0.1  TB/s BW gloMEM___
    |         ________________________________________________________________________________________________________________________________________________________________________________________________________________________
    +========|   DEVICE:3 PERSISTENT                          gloMEM___
    |       _|______________________________________________________________________________________________________________________________________________________________________________________________________________________
    +======|   DEVICE:2 PERSISTENT                          gloMEM___
    |     _|______________________________________________________________________________________________________________________________________________________________________________________________________________________
    +====|   DEVICE:1 PERSISTENT                          gloMEM___
    |   _|______________________________________________________________________________________________________________________________________________________________________________________________________________________
    +==|   DEVICE:0 PERSISTENT                          gloMEM_____________________________________________________________________+440 [GPU_CLKs]_________________________________________________________________________|_GB|
    !  |                                                         |\                                                                +                                                                                           |
    o  |                                                texMEM___|_\___________________________________texMEM______________________+_______________________________________________________________________________________|_MB|
       |                                                         |\ \                                 |\                           +                                               |\                                          |
       |                                              texL2cache_| \ \                               .| \_ _ _ _ _ _ _ _texL2cache +370 [GPU_CLKs] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | \                                   256_KB|
       |                                                         |  \ \                               |  \                         +                                 |\            ^  \                                        |
       |                                                         |   \ \                              |   \                        +                                 | \           ^   \                                       |
       |                                                         |    \ \                             |    \                       +                                 |  \          ^    \                                      |
       |                                              texL1cache_|     \ \                           .|     \_ _ _ _ _ _texL1cache +260 [GPU_CLKs] _ _ _ _ _ _ _ _ _ |   \_ _ _ _ _^     \                                 5_KB|
       |                                                         |      \ \                           |      \                     +                         ^\      ^    \        ^\     \                                    |
       |                                     shaMEM + conL3cache_|       \ \                          |       \ _ _ _ _ conL3cache +220 [GPU_CLKs]           ^ \     ^     \       ^ \     \                              32_KB|
       |                                                         |        \ \                         |        \       ^\          +                         ^  \    ^      \      ^  \     \                                  |
       |                                                         |         \ \                        |         \      ^ \         +                         ^   \   ^       \     ^   \     \                                 |
       |                                   ______________________|__________\_\_______________________|__________\_____^__\________+__________________________________________\_________\_____\________________________________|
       |                  +220 [GPU-CLKs]_|           |_ _ _  ___|\          \ \_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \ _ _ _ _\_ _ _ _+220 [GPU_CLKs] on re-use at some +50 GPU_CLKs _IF_ a FETCH from yet-in-shaL2cache
       | L2-on-re-use-only +80 [GPU-CLKs]_| 64 KB  L2_|_ _ _   __|\\          \ \_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \ _ _ _ _\_ _ _ + 80 [GPU_CLKs] on re-use from L1-cached (HIT) _IF_ a FETCH from yet-in-shaL1cache
       | L1-on-re-use-only +40 [GPU-CLKs]_|  8 KB  L1_|_ _ _    _|\\\          \_\__________________________________\________\_____+ 40 [GPU_CLKs]_____________________________________________________________________________|
       | L1-on-re-use-only + 8 [GPU-CLKs]_|  2 KB  L1_|__________|\\\\__________\_\__________________________________\________\____+  8 [GPU_CLKs]_________________________________________________________conL1cache      2_KB|
       |     on-chip|smREG +22 [GPU-CLKs]_|           |t[0_______^:~~~~~~~~~~~~~~~~\:________]
       |CC-  MAX    |_|_|_|_|_|_|_|_|_|_|_|           |t[1_______^                  :________]
       |2.x   63    |_|_|_|_|_|_|_|_|_|_|_|           |t[2_______^                  :________] 
       |1.x  128    |_|_|_|_|_|_|_|_|_|_|_|           |t[3_______^                  :________]
       |3.5  255 REGISTERs|_|_|_|_|_|_|_|_|           |t[4_______^                  :________]
       |         per|_|_|_|_|_|_|_|_|_|_|_|           |t[5_______^                  :________]
       |         Thread_|_|_|_|_|_|_|_|_|_|           |t[6_______^                  :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           |t[7_______^     1stHalf-WARP :________]______________
       |            |_|_|_|_|_|_|_|_|_|_|_|           |t[ 8_______^:~~~~~~~~~~~~~~~~~:________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           |t[ 9_______^                  :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           |t[ A_______^                  :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           |t[ B_______^                  :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           |t[ C_______^                  :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           |t[ D_______^                  :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           |t[ E_______^                  :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|       W0..|t[ F_______^____________WARP__:________]_____________
       |            |_|_|_|_|_|_|_|_|_|_|_|         ..............             
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[0_______^:~~~~~~~~~~~~~~~\:________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[1_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[2_______^                 :________] 
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[3_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[4_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[5_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[6_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[7_______^    1stHalf-WARP :________]______________
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[ 8_______^:~~~~~~~~~~~~~~~~:________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[ 9_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[ A_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[ B_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[ C_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[ D_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|           ............|t[ E_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|       W1..............|t[ F_______^___________WARP__:________]_____________
       |            |_|_|_|_|_|_|_|_|_|_|_|         ....................................................
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[0_______^:~~~~~~~~~~~~~~~\:________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[1_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[2_______^                 :________] 
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[3_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[4_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[5_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[6_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[7_______^    1stHalf-WARP :________]______________
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[ 8_______^:~~~~~~~~~~~~~~~~:________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[ 9_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[ A_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[ B_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[ C_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[ D_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|          ...................................................|t[ E_______^                 :________]
       |            |_|_|_|_|_|_|_|_|_|_|_|tBlock Wn....................................................|t[ F_______^___________WARP__:________]_____________
       |
       |                   ________________          °°°°°°°°°°°°°°°°°°°°°°°°°°~~~~~~~~~~°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
       |                  /                \   CC-2.0|||||||||||||||||||||||||| ~masked  ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
       |                 /                  \  1.hW  ^|^|^|^|^|^|^|^|^|^|^|^|^| <wait>-s ^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|
       |                /                    \ 2.hW  |^|^|^|^|^|^|^|^|^|^|^|^|^          |^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^|^
       |_______________/                      \______I|I|I|I|I|I|I|I|I|I|I|I|I|~~~~~~~~~~I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|I|
       |~~~~~~~~~~~~~~/ SM:0.warpScheduler    /~~~~~~~I~I~I~I~I~I~I~I~I~I~I~I~I~~~~~~~~~~~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I~I
       |              \          |           //
       |               \         RR-mode    //
       |                \    GREEDY-mode   //
       |                 \________________//
       |                   \______________/SM:0__________________________________________________________________________________
       |                                  |           |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:1__________________________________________________________________________________
       |                                  |           |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:2__________________________________________________________________________________
       |                                  |           |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:3__________________________________________________________________________________
       |                                  |           |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:4__________________________________________________________________________________
       |                                  |           |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:5__________________________________________________________________________________
       |                                  |           |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:6__________________________________________________________________________________
       |                                  |           |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:7__________________________________________________________________________________
       |                                  |           |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:8__________________________________________________________________________________
       |                                  |           |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:9__________________________________________________________________________________
       |                                ..|SM:A      |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:B      |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:C      |t[ F_______^___________WARP__:________]_______
       |                                ..|SM:D      |t[ F_______^___________WARP__:________]_______
       |                                  |_______________________________________________________________________________________
       */

---

The bottom line?

Any low-latency motivated design has to rather reverse-engineer the "I/O-hydraulics" ( as 0 1-XFERs are incompressible by the nature ) and the resulting latencies rule the performance envelope for any GPGPU solution be it computationally intensive ( read: where processing costs are forgiving a bit more a poor latency XFERs ... ) or not ( read: where ( might be to someone's surprise ) CPU-s are faster in end-to-end processing, than GPU fabrics [citations available] ).

Iquique

I have attempted to understand your answer. It seems very interesting but ASCII-graphs are not easy to read because of high/width limitations. Sorry I do not know how this could be improved... Finally I am missing a summary (at the end, I do not know what to think about CPU vs. GPU latencies). Hope you can improve your answer to provide a better eye-candy look and human understandability. Courage. Cheers :-D – Pause

Answer 5

Look at this "staircase" plot, perfectly illustrating different access times (in terms of clock tics). Notice the red CPU having an additional "step", probably because it has L4 (while others don't).

Taken from this Extremetech article.

In computer science this is called "I/O complexity".

Frons