As Denny said, prior to SSDs a lot of the delay accessing data stored there was due to the fact that you had to physically move a mechanical apparatus to find the place on the disk where the data was stored, like moving the needle on a record player to find the right song, and then wait for the data to spin around until it was underneath the needle. This was known as "seek time" and dominated when you wanted to access a lot of little things that were spread around across the disk.
SSDs do not have such a head, which is why they are a lot faster for many purposes. However, they do have a "wear-leveling" algorithm that tries to use all the bits on the drive evenly, and can be a source of a (much tinier) amount of delay during writes. SSDs are still also a "block-based" device meaning data is fetched and written in larger batches than with RAM. Again this only slows down scattered accesses compared to reading large files in order.
Even were we to eliminate all that, there is still the fact that drives are usually on a slower, narrower bus located further away from the CPU. The RAM on the mainboard, in turn, is farther from the CPU and often on a slower bus than the RAM in the cache (which is sometimes divided into levels that each of which in turn are closer to the CPU, but smaller than one before it.) Note that electrical signals which carry the bits to the CPU are limited by the speed of light. Light is pretty fast, but even so, in one clock cycle on a 4Ghz CPU it only moves about the length of your finger. So if you read an address out of memory, and then use that address to read something else from memory, you have to make a round trip to RAM to do that, and the number of clock cycles it takes to do that will depend on how far away those bits are stored. This is why, even with RAM, these days we often read blocks at a time (smaller blocks than disk, though) and try to keep data items that get used at the same time time next to each other when we can, to increase the chances that they will be in the same block.