[gdal-dev] Can GDAL be configured to do direct I/O? (bypass cache, control blocksize, etc)

[Even Rouault]

Selon Ethan Alpert <ealpert at digitalglobe.com>:

> I'm running in to a situation where I believe the filesystem cache
> settings are screwing things up. Based on iozone tests I'm not getting
> near the throughput I should for large 4gb+ files with multiple writing
> processes. It is a complex interaction of the number of dirty pages in
> filesystem cache in combination with small blocksize of the filesystem.
> I'm still trying to figure this out but I can now get these machines to
> tip over with just iozone.
>
>
>
>
>
> I'm actively working on how to refactor this code base to use GDAL
> WriteBlock but in absence of a solution there I'm wondering if there's
> anyway to get GDAL writing using O_DIRECT?

You can also use regularly GDALFlushCacheBlock() to force dirty GDAL cached
blocks to be written. But a few drivers also have internal caching (for example
GTiff for pixel interleaved rasters), so you might also want to combine that
with GDALFlushCache(hDS).

To use O_DIRECT, you would need to change the implementation of
cpl_vsi_unix_stdio_64.cpp to use the open(), read(), etc.. API instead of
fopen(), fread(), etc... Unless there's a way to set O_DIRECT on FILE* ?

You cannot generally control the GDAL block size, unless this is offered as a
creation option by the format driver like the TILED, BLOCKXSIZE, BLOCKYSIZE
options of the GTiff driver.

Even