[gdal-dev] Merge and Overviews with VSIMEM

[Sean Gillies]

Hi Brenton,

On Wed, Feb 26, 2020 at 7:13 AM Brenton Mallen <brentonmallen1 at gmail.com>
wrote:

> Hi,
>
> I'm attempting to merge 2 multi-band tiffs as well as build overviews for
> the resulting merged tiff.  I'm a bit confused when I attempt to use a
> `vsimem` object to perform the actions in memory.  For example, I have a
> couple of functions defined as
>
> def merge_bands(tiffs, out_file):
>     """Merge multi-band GeoTIFFs together.
>
>     Args:
>         tiffs (list): List of paths to TIFFs to be merged
>         out_file (str): Path to output file
>     """
>     merge_command = 'gdal_merge.py -separate -o {} '.format(out_file)
>     merge_command += ' --config GDAL_CACHEMAX 256'
>     merge_command += ' -co compress=LZW'
>     merge_command += ' -co TILED=YES'
>     for tiff in tiffs:
>         merge_command = merge_command + ' {}'.format(tiff)
>     sp.check_call(merge_command, shell=True, stdout=sp.PIPE, stderr=sp.PIPE)
>
> def build_overviews(filename, levels):
>     """Build overviews.
>     Args:
>         filename (str): Input filename
>         levels (list): Level (int) or list or levels (ints)
>             to build overviews for.
>     """
>     if not isinstance(levels, list):
>         levels = [levels]
>     command = 'gdaladdo'
>     command += ' -r average'
>     command += ' {}'.format(filename)
>     command += ' {}'.format(' '.join(str(x) for x in levels))
>     sp.check_call(command, shell=True, stdout=sp.PIPE, stderr=sp.PIPE)
>
>
> When I attempt to call it, using the following, the `vsimem` object seems
> to no longer be accessible to perform the overview creation:
>
> merged_mem = '/vsimem/merged.tiff'
>
> merge_bands(['file1.tiff', 'file2.tiff'], merged_mem)
> build_overviews(merged_mem, [2,4,8,16,32,64])
>
> I get an `exit status 2` which I believe is due to `merged_mem` being a
> NoneType after the merge is performed.
>
> I'm also confused about best practices.  Ideally, I'd like to do this all
> in the python wrapper but there doesn't seem to be an immediately obvious
> parallel for the `gdal_merge` script.
>
> I'm using python 3.8 and GDAL 3.0.2.
>
> Any guidance would be appreciated and thank you for your time,
> Brenton
>

VSIMEM files are visible within a process (
https://gdal.org/user/virtual_file_systems.html#vsimem-in-memory-files) and
Python's check_call runs your command within a *new* process. You have a
single string named merged_mem, but I think this maps to a different
in-memory file in each of the two processes in your program. Even will have
a more definitive answer, I'm sure. but I think these are your options:

1. use a physical temp file, easily shared between processes
2. use a more sophisticated shared memory (mmap?)
3. implement the merge in Python within the original process

-- 
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20200226/46a33b56/attachment.html>