> Who "have to wait" ? Apps don't have to: they get the data from
> cache and write to cache. Maybe the disk-write policy depends on the IO
> scheduler as the read policy does, but this layer is completely isolated
> from the applications.

Nope - not completely isolated. Applications that actually require
on-disk consistency invoke things such as fsync(2).

Every year or so there is a random blog post which notices that
if you truly care about your data, you'd better do fsync(2), fdatasync(2)
and/or rename(2) in the right places.

If the os has done the write already, then this is pretty much a
no-op, but if there are loads of stale write buffers your
application can block until all of those have been written out
and then performance craters.

regards

marc