<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Marta,</p>
<p>I was lucky enough to figure out from the error messages that
your issue came when using QGIS, but it would have been
appropriate to mention it from the start so we get the global
picture.</p>
<p>This is in fact a QGIS bug, which is only seen with the ODS &
XLSX driver since they use the OGR Memory driver underneath and it
has special behavior when creating a feature with a set FID. <br>
</p>
<p>I've submitted a QGIS bug fix for that in
<a class="moz-txt-link-freetext" href="https://github.com/qgis/QGIS/pull/46773">https://github.com/qgis/QGIS/pull/46773</a><br>
</p>
<p>Even<br>
</p>
<div class="moz-cite-prefix">Le 10/01/2022 à 13:34, Kopszak Marta -
Partner Hurt a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:3c99b38ae23c4e858c680b10b8362b43@orange.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}pre
{mso-style-priority:99;
mso-style-link:"HTML - wstępnie sformatowany Znak";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}span.HTML-wstpniesformatowanyZnak
{mso-style-name:"HTML - wstępnie sformatowany Znak";
mso-style-priority:99;
mso-style-link:"HTML - wstępnie sformatowany";
font-family:Consolas;}p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}span.Stylwiadomocie-mail21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi
Even,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thanks
for your replay. More info about the problem:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Error:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
<br>
<i>Save error:<o:p></o:p></i></span></p>
<p class="MsoNormal"><i><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Export
to vector file failed.<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Error:
Feature write errors:<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Feature
creation error (OGR error: negative FID are not supported)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></i></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">The
database has more than 2 bilons of records (record – I mean
record in the database table). The error occurs when I
export a single table witout the primary key to .xlsx. <o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I
suppose that when the table doesn’t have the primary key it
use tid – the tuple identifier, and the size of tid may
exceed the size of a 32-bit integer. It cause the problem to
export this table to .xlsx format by OGR library.
<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Marta
Kopszak<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">
gdal-dev <a class="moz-txt-link-rfc2396E" href="mailto:gdal-dev-bounces@lists.osgeo.org"><gdal-dev-bounces@lists.osgeo.org></a>
<b>On Behalf Of </b>Even Rouault<br>
<b>Sent:</b> Monday, January 10, 2022 11:42 AM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<b>Subject:</b> Re: [gdal-dev] Bug in OGR library<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Marta,<o:p></o:p></p>
<p>What is the bug exactly ?<o:p></o:p></p>
<p>"2 billion records" : by record, you mean cells, like a sheet
with 2048 columns and 1048576 rows ? By quickly looking at the
code I can't see an obvious limitation, but that doesn't mean
anything. Closer investigation would be needed. Please file an
issue at <a href="https://github.com/OSGeo/gdal/issues/new"
moz-do-not-send="true">https://github.com/OSGeo/gdal/issues/new</a>
with exact details how to reproduce (number of rows/features,
and columns/fields in particular).<o:p></o:p></p>
<p>But even if the writer side would work, the OGR reader side
would probably have issues as the whole spreadsheet will be
loaded into RAM and for 2 billion cells, I would expect at
least 32 or more GB of RAM to be needed. Perhaps spreadsheet
software are able to cope with that in a smarter way.<o:p></o:p></p>
<p>Even<o:p></o:p></p>
<div>
<p class="MsoNormal">Le 10/01/2022 à 11:27, Kopszak Marta -
Partner Hurt a écrit :<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Dear
Sirs,<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
ran into an error when exporting data through the OGR
library to the .xlsx and .ods format. The bug occurs when
there are more than 2 billion records in the database. I
know the problem was solved for other formats, e.g. csv.
Will the problem be solved also for other formats (e.g.
xlsx, .ods)? If this is not the correct e-mail, please
forward this e-mail to the appropriate person.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kind
regards,<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Marta
Kopszak<o:p></o:p></p>
<table class="MsoNormalTable" cellspacing="3"
cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt"><br>
</td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a
href="https://www.orange.pl/" target="_blank"
moz-do-not-send="true"><b><span
style="color:windowtext;text-decoration:none">Marta
Kopszak</span></b><span
style="color:windowtext;text-decoration:none"><br>
Współpracownik</span></a><span
class="MsoHyperlink"><span
style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></p>
</td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a
href="https://www.orange.pl/" target="_blank"
moz-do-not-send="true"><span
style="color:windowtext;text-decoration:none">Rozwój
Sieci Stacjonarnej, Wydział Rozwoju Planowania<br>
Orange Polska, gen. Romualda Traugutta 55,
50-416 Wrocław | <u><span style="color:blue">RODO
- informacja o danych</span></u>
</span></a><span class="MsoHyperlink"><span
style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a
href="https://www.orange.pl/" target="_blank"
moz-do-not-send="true"><span
style="color:windowtext;text-decoration:none"> </span></a><span
class="MsoHyperlink"><span
style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></p>
</div>
<p class="MsoNormal"><a href="https://www.orange.pl/"
target="_blank" moz-do-not-send="true"><span
style="color:windowtext;text-decoration:none"><br>
</span></a><span class="MsoHyperlink"><span
style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></p>
<pre><a href="https://www.orange.pl/" target="_blank" moz-do-not-send="true"><span style="color:windowtext;text-decoration:none">_______________________________________________</span></a><span class="MsoHyperlink"><span style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></pre>
<pre><a href="https://www.orange.pl/" target="_blank" moz-do-not-send="true"><span style="color:windowtext;text-decoration:none">gdal-dev mailing list</span></a><span class="MsoHyperlink"><span style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></pre>
<pre><a href="https://www.orange.pl/" target="_blank" moz-do-not-send="true"><span style="color:windowtext;text-decoration:none"><u><span style="color:blue">gdal-dev@lists.osgeo.org</span></u></span></a><span class="MsoHyperlink"><span style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></pre>
<pre><a href="https://www.orange.pl/" target="_blank" moz-do-not-send="true"><span style="color:windowtext;text-decoration:none"><u><span style="color:blue">https://lists.osgeo.org/mailman/listinfo/gdal-dev</span></u></span></a><span class="MsoHyperlink"><span style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></pre>
</blockquote>
<pre><a href="https://www.orange.pl/" target="_blank" moz-do-not-send="true"><span style="color:windowtext;text-decoration:none">-- </span></a><span class="MsoHyperlink"><span style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></pre>
<pre><a href="https://www.orange.pl/" target="_blank" moz-do-not-send="true"><span style="color:windowtext;text-decoration:none"><u><span style="color:blue">http://www.spatialys.com</span></u></span></a><span class="MsoHyperlink"><span style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></pre>
<pre><a href="https://www.orange.pl/" target="_blank" moz-do-not-send="true"><span style="color:windowtext;text-decoration:none">My software is free, but my time generally not.</span></a><span class="MsoHyperlink"><span style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></pre>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>