In my experience, if you place an image in just some of the drawable folders, but not all, the OS will choose the "best" image for the device that you are using, even if it is not exactly the perfect fit. For example, if you put an image called arrow.png ONLY in the drawable-hdpi folder, all devices will use that image for the arrow drawable, and scale it down or up appropriately, stretching or shrinking the image.
That being said, you should be able to accomplish what you want by simply putting your image in the right folders and allowing the devices to choose the correct one. For example, if you were trying to accomplish your task with only one image, arrow.png:
drawable-xhdpi/ -> arrow.png
drawable-hdpi/ -> empty
drawable-mdpi/ -> arrow.png
drawable-ldpi/ -> empty
drawable/ -> empty
If you use the app on an ldpi device, the device will use the mdpi image, as you wanted, because it is closest to the correct resolution. On the hdpi phone, it will use the xhdpi image, because it is again the closest to the correct resolution.