[gdal-dev] VSIGetMemFileBuffer Returns NULL When Copying Vector Dataset to Vsimem

[Even Rouault]

Le 06/10/2023 à 01:11, Soren Saville Scott a écrit :
> Hi Even,
>
> Suggests something wrong with my build. Is there possibly something 
> strange happening across DLL boundaries? Do you know of any build 
> flags that could change the VSIMEM behaviour?
I'm not seeing anything suspicious here, especially because 
VSIGetMemFileBuffer() is in the C API.
>
> As a secondary question, for multi-file datasets (like shapefiles) 
> does VSIGetMemFileBuffer() encapsulate all those files - i.e. I should 
> be able to call VSIFileFromMemBuffer() on that buffer and GDALOpen() 
> and get the same information back?

You need to read each file separately. VSIGetMemFileBuffer() works at 
the file level. It has no idea of what a GDAL dataset is.

You can use GDALGetFileList() to get the list of files of a dataset. You 
might want to do that on a dataset after re-opening, since there has 
been a recent bug fix w.r.t GetFileList() on shapefiles just created 
where one of the side car file wasn't listed

>
> Thanks for your help,
> Soren
>
> On Wed, 4 Oct 2023 at 21:09, Even Rouault <even.rouault at spatialys.com> 
> wrote:
>
>     Hi,
>
>     size_t is an incorrect size for DataLength (if you run on 32 bit
>     architecture). You should use vsi_l_offset.
>
>     Besides that the following standalone inspired from your code
>     works fine for me:
>
>     #include <stdio.h>
>     #include "gdal_priv.h"
>
>     int main()
>     {
>         GDALAllRegister();
>         GDALDataset* Dataset
>     =GDALDataset::FromHandle(GDALOpenEx("poly.shp", GDAL_OF_VECTOR |
>     GDAL_OF_VERBOSE_ERROR, nullptr, nullptr, nullptr));
>         GDALDataset* Copy =
>     Dataset->GetDriver()->CreateCopy("/vsimem/foo.shp", Dataset,
>     false, nullptr, nullptr, nullptr); //appears to produce a valid
>     Dataset containing the same layers & features as its source.
>         GDALClose(Dataset);
>         GDALClose(Copy);
>         vsi_l_offset DataLength = 0;
>         GByte* Buffer = VSIGetMemFileBuffer("/vsimem/foo.shp",
>     &DataLength, true);
>         printf("%p %d\n", Buffer, int(DataLength)); // buffer not null
>     and DataLength > 0
>         CPLFree(Buffer);
>         VSIUnlink("/vsimem/foo.dbf");
>         VSIUnlink("/vsimem/foo.shx");
>         return 0;
>     }
>
>     Even
>
>     Le 04/10/2023 à 06:26, Soren Saville Scott via gdal-dev a écrit :
>>     Hi,
>>
>>     I'm integrating GDAL in C++ with a system that defines its own
>>     asset types stored on disk, and am writing a wrapper for datasets
>>     using this asset system. To 'import' a dataset I call GDALOpenEx
>>     on it, CreateCopy() it to some "/vsimem/foo.shp" path,
>>     GDALClose() the datasets to flush it, and call
>>     VSIGetMemFileBuffer with bUnlinkAndSeize=true to take ownership
>>     of the underlying memory buffer.
>>
>>     GDALDataset* Dataset = GDALOpenEx("foo.shp", GDAL_OF_VECTOR |
>>     GDAL_OF_VERBOSE_ERROR, nullptr, nullptr, nullptr);
>>     GDALDataset* Copy =
>>     Dataset->GetDriver()->CreateCopy("/vsimem/foo.shp", Dataset,
>>     false, nullptr, nullptr, nullptr); //appears to produce a valid
>>     Dataset containing the same layers & features as its source.
>>     GDALClose(Dataset);
>>     GDALClose(Copy);
>>     size_t DataLength;
>>     GByte* Buffer = VSIGetMemFileBuffer("/vsimem/foo.shp",
>>     &DataLength, true); //<-- returns NULL
>>
>>     For some reason VSIGetMemFileBuffer always fails, returning a
>>     nullptr. This is basically identical to a similar problem I had,
>>     described
>>     here: https://stackoverflow.com/questions/73644157/get-raw-buffer-for-in-memory-dataset-in-gdal-c-api
>>     <https://stackoverflow.com/questions/73644157/get-raw-buffer-for-in-memory-dataset-in-gdal-c-api> which
>>     I never really found a solution to. I have similar issues with
>>     other VSI functions, e.g. calling GDALOpenEx on a path created by
>>     VSIFileFromMemBuffer also fails.
>>
>>     Any idea as to what the problem is here?
>>
>>     Alternatively, maybe there's a better way to 'emplace' datasets
>>     inside some other file?
>>
>>     Thanks,
>>     Soren Saville Scott
>>
>>     _______________________________________________
>>     gdal-dev mailing list
>>     gdal-dev at lists.osgeo.org
>>     https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>     -- 
>     http://www.spatialys.com
>     My software is free, but my time generally not.
>
-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20231006/23b4a31c/attachment.htm>