[gdal-dev] Dirty block GDALRasterBand::IRasterIO error even with Libtiff > 4.0 on Windows and Mac but not Linux.

[Krishna Bhogaonker]

Hello folks.
This is my first post to the GDAL list :). So I am running into an error
using `Rasterio` 1.1.7 and `Gdal` 3.1.3 on Windows and Mac, but not Linux.
The code opens a raster GeoTIFF file and compresses the data to some
smaller format. Here is the code below and then a picture of the error
message. Note that the code is written in Python 3.8 with `rasterio` but
the error seems to come from the GDAL call.

The error message is rasterio.errors.RasterioIOError: Read or write failed.
An error occurred while writing a dirty block from
GDALRasterBand::IRasterIO . I googled around and found that I needed to
ensure Libtiff > 4.0 is installed, which I did--but the error still
persists.

One thing to note is that the GeoTIFF is rather large. I am working with
aerial drone imagery for a philanthropic group in Tanzania, so the images
can be like 2-6GB even after compressed. Here is a sample image that I use
for testing
https://github.com/00krishna-tools/nyuki/releases/download/v0.0.1/sample_image_big.tif

import rasterio


dat = 'sample_image_big.tif'
target_file = 'sample_image_big_compressed_jpeg.tif'

with rasterio.Env():

    profile.update(
        compress="JPEG",
        BIGTIFF = "YES")

    with rasterio.open(target_file, 'w', **profile) as dst:
        for ji, window in dat.block_windows(1):
            dst.write(dat.read(window=window), window=window)

I am using "windowed" reads and writes, etc., so that seems to be the
most efficient way to do it.

Here is a picture of the error with traceback:

Traceback (most recent call last):
  File "rasterio/_io.pyx", line 1369, in rasterio._io.DatasetWriterBase.write
  File "rasterio/shim_rasterioex.pxi", line 133, in rasterio._shim.io_multi_band
  File "rasterio/_err.pyx", line 182, in rasterio._err.exc_wrap_int
rasterio._err.CPLE_AppDefinedError: An error occurred while writing a
dirty block from GDALRasterBand::IRasterIODuring handling of the above
exception, another exception occurred:Traceback (most recent call
last):
  File "/Users/user/mdconda/anaconda3/envs/nyuki-env/bin/nyuki", line
10, in <module>
    sys.exit(nyuki())
  File "/Users/user/mdconda/anaconda3/envs/nyuki-env/lib/python3.7/site-packages/click/core.py",
line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/user/mdconda/anaconda3/envs/nyuki-env/lib/python3.7/site-packages/click/core.py",
line 782, in main
    rv = self.invoke(ctx)
  File "/Users/user/mdconda/anaconda3/envs/nyuki-env/lib/python3.7/site-packages/click/core.py",
line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/user/mdconda/anaconda3/envs/nyuki-env/lib/python3.7/site-packages/click/core.py",
line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/user/mdconda/anaconda3/envs/nyuki-env/lib/python3.7/site-packages/click/core.py",
line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/user/mdconda/anaconda3/envs/nyuki-env/lib/python3.7/site-packages/click/core.py",
line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/mdconda/anaconda3/envs/nyuki-env/lib/python3.7/site-packages/nyuki/nyuki.py",
line 160, in compress
    compressor(sourcetiff, target_compression, yes)
  File "/Users/user/mdconda/anaconda3/envs/nyuki-env/lib/python3.7/site-packages/nyuki/geotiff_compressor.py",
line 52, in compressor
    dst.write(dat.read(window=window), window=window)
  File "rasterio/_io.pyx", line 1371, in rasterio._io.DatasetWriterBase.write
rasterio.errors.RasterioIOError: Read or write failed. An error
occurred while writing a dirty block from GDALRasterBand::IRasterIO

So the real question is whether there is a workaround for this issue,
or is there something different

I should do with my code? Any suggestions for getting this code to
work on Mac and Windows, as it

works just fine on Linux. Thanks so much for any insights.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20201023/e4566224/attachment-0001.html>