How to 'warm up' EC2/EBS storage?

Question

I have seen advice to 'warm up' EC2 to overcome a first write penalty:-

Warm up data partitions - There is one drawback to using disk IO in EC2: a “first write” performance hit when initially writing to new partitions. To avoid this penalty, you can “warm up” the partition by executing a sort of throw-away command that accesses it. For example, you can use the Linux dd command to write to the disk. While the penalty still occurs and cannot be avoided, at least the first write to your databases will not suffer the effects.

Source: http://answers.oreilly.com/topic/1345-getting-the-most-out-of-mysql-in-the-amazon-cloud/

...but I haven't found any further advice on best practice! Is this true of EBS storage? Can anyone recommend the 'dd' syntax that will perform this warmup and how to ensure that all blocks are 'warmed'?

Irony

Have you asked this question on the AWS forums? I'm sure the EC2/EBS teams would likely be able to help you out there. – Polemic

Answer 1

Thanks for the pointer to AWS forum. I did post the question there after posting here, and received a response from Jason@AWS. The thread is here:-

http://developer.amazonwebservices.com/connect/message.jspa?messageID=198413#198413

...and his response was:-

There is a first- read penalty for EBS volumes created from snapshots, as the volume is made available before all of the blocks have been successfully loaded. You can negate this penalty by forcing every block on the volume to be read:

$ dd if=/dev/<device> of=/dev/null

Irony

That only applies to volumes created from a snapshot. I suspect volumes may be created as sparse files which means you will want to force an allocation by writing some random data (or possibly zeros). – Hesper

dd if=/dev/zero of=/dev/<device> BS=1M (for non-snapshot volumes) – Hesper

In case of Windows snapshot volumes functioning as boot drives, dd errors right away with "Error opening input file: 32 The process cannot access the file because it is being used by another process" Any workarounds? – Mashe

Answer 2

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html indicates that snapshot volumes can be prewarmed by reading the entire disk; this speeds up reads. Snapshot and non-snapshot volumes can prewarm writes by writing the entire disk.

When you create a new EBS volume or restore a volume from a snapshot, the back-end storage blocks are allocated to you immediately. However, the first time you access a block of storage, it must be either wiped clean (for new volumes) or instantiated from its snapshot (for restored volumes) before you can access the block. This preliminary action takes time and can cause a 5 to 50 percent loss of IOPS for your volume the first time each block is accessed.

Trilley

This answer was flagged as low-quality. Could you possibly put some quotes from that link into your answer to improve it? – Kenya