[gdal-dev] Bug in OGR library

[Even Rouault]

Marta,

I was lucky enough to figure out from the error messages that your issue 
came when using QGIS, but it would have been appropriate to mention it 
from the start so we get the global picture.

This is in fact a QGIS bug, which is only seen with the ODS & XLSX 
driver since they use the OGR Memory driver underneath and it has 
special behavior when creating a feature with a set FID.

I've submitted a QGIS bug fix for that in 
https://github.com/qgis/QGIS/pull/46773

Even

Le 10/01/2022 à 13:34, Kopszak Marta - Partner Hurt a écrit :
>
> Hi Even,
>
> Thanks for your replay. More info about the problem:
>
> *Error:*
> /Save error:/
>
> /Export to vector file failed./
>
> /Error: Feature write errors:/
>
> /Feature creation error (OGR error: negative FID are not supported)/
>
> //
>
> The database has more than 2 bilons of records (record – I mean record 
> in the database table). The error occurs when I export a single table 
> witout the primary key to .xlsx.
>
> I suppose that when the table doesn’t have the primary key it use tid 
> – the tuple identifier, and the size of tid may exceed the size of a 
> 32-bit integer. It cause the problem to export this table to .xlsx 
> format by OGR library.
>
> Regards,
>
> Marta Kopszak
>
> *From:*gdal-dev <gdal-dev-bounces at lists.osgeo.org> *On Behalf Of *Even 
> Rouault
> *Sent:* Monday, January 10, 2022 11:42 AM
> *To:* gdal-dev at lists.osgeo.org
> *Subject:* Re: [gdal-dev] Bug in OGR library
>
> Marta,
>
> What is the bug exactly ?
>
> "2 billion records" : by record, you mean cells, like a sheet with 
> 2048 columns and 1048576 rows ? By quickly looking at the code I can't 
> see an obvious limitation, but that doesn't mean anything. Closer 
> investigation would be needed. Please file an issue at 
> https://github.com/OSGeo/gdal/issues/new 
> <https://github.com/OSGeo/gdal/issues/new> with exact details how to 
> reproduce (number of rows/features, and columns/fields in particular).
>
> But even if the writer side would work, the OGR reader side would 
> probably have issues as the whole spreadsheet will be loaded into RAM 
> and for 2 billion cells, I would expect at least 32 or more GB of RAM 
> to be needed. Perhaps spreadsheet software are able to cope with that 
> in a smarter way.
>
> Even
>
> Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :
>
>     Dear Sirs,
>
>     I ran into an error when exporting data through the OGR library to
>     the .xlsx and .ods format. The bug occurs when there are more than
>     2 billion records in the database. I know the problem was solved
>     for other formats, e.g. csv. Will the problem be solved also for
>     other formats (e.g. xlsx, .ods)? If this is not the correct
>     e-mail, please forward this e-mail to the appropriate person.
>
>     Kind regards,
>
>     Marta Kopszak
>
>
>     *Marta Kopszak*
>     Współpracownik <https://www.orange.pl/>
>
>     Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
>     Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | _RODO
>     - informacja o danych_ <https://www.orange.pl/>
>
>     <https://www.orange.pl/>
>
>
>     <https://www.orange.pl/>
>
>     _______________________________________________  <https://www.orange.pl/>
>
>     gdal-dev mailing list  <https://www.orange.pl/>
>
>     _gdal-dev at lists.osgeo.org_  <https://www.orange.pl/>
>
>     _https://lists.osgeo.org/mailman/listinfo/gdal-dev_  <https://www.orange.pl/>
>
> --   <https://www.orange.pl/>
> _http://www.spatialys.com_  <https://www.orange.pl/>
> My software is free, but my time generally not.  <https://www.orange.pl/>

-- 
http://www.spatialys.com
My software is free, but my time generally not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220110/18cd400f/attachment.html>