At first glance, it does look quite strange to see Chrome spending time downloading resources even though they are coming from the cache. It's not the time spent downloading from a web server you're seeing. Rather, I believe it is the time spent downloading from a local database cache.
The retrieval of any data has some amount of cost involved. The resources are essentially stored in a database in Chrome, and to retrieve data requires a lookup, which is not instant. As well as looking up the data in a table, there is likely some processing involved to push the correct data into memory, since the data is not stored exactly how it is going to be used. It is likely to be compressed, and decompressing data can be a slow process.
You can see in the Network tab that, although it appears to take 0 ms to retrieve some resources, when you look at the Timings tab, you will see that it is actually rounded down. For example, I see both 0.08 ms stalled and 0.02 ms download in the request below, despite it showing 0 ms in the grid.
Update:
I looked further into this and found that Chrome Extensions seem to have an effect on the retrieval times from both the cache and the web, particularly ones that inject content into the page. Adblock seems to be the cause of some delay for me - explanation above still very much applies for the rest.