[gdal-dev] [EXTERNAL] Re: Possible to issue sqlite3_limit with "memory" OGR driver?

[Meyer, Jesse R. (GSFC-618.0)[SCIENCE SYSTEMS AND APPLICATIONS INC]]

Thanks Even, that’s what I suspected.

Best,
Jesse

Lead Computer Scientist
Science Systems and Applications, Inc.
Dr Compton Tucker Team
NASA Goddard Space Flight Center

From: Even Rouault <even.rouault at spatialys.com>
Date: Thursday, July 3, 2025 at 4:23 PM
To: Meyer, Jesse R. (GSFC-618.0)[SCIENCE SYSTEMS AND APPLICATIONS INC] <jesse.r.meyer at nasa.gov>, gdal dev <gdal-dev at lists.osgeo.org>
Subject: [EXTERNAL] Re: [gdal-dev] Possible to issue sqlite3_limit with "memory" OGR driver?
CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.



Jesse,

Maybe you use SQL with the SQLite dialect? Otherwise we lack context to tell why you get SQLite kicking it.

I don't think sqlite3_limit() would really help you here, givenit can be used to grow settings beyond the compile time default which is 1 GiB I believe ("Attempts to increase a limit above its hard upper bound are silently truncated to the hard upper bound."). You would need to recompile SQLite itself.

Even
Le 03/07/2025 à 22:14, Meyer, Jesse R. (GSFC-618.0)[SCIENCE SYSTEMS AND APPLICATIONS INC] via gdal-dev a écrit :
We’re trying to gdal.Polygonize a feature that exceeds SQLite’s default blob size of 1,000,000,000 bytes.  Curiously, we’re using the memory dataset, but this appears to leverage SQLite under the hood (evidenced by the error message: ERROR 1: In GetNextRawFeature(): sqlite3_step() : string or blob too big).  SQLite docs state the that limit can be changed at runtime using the sqlite3_limit<https://www.sqlite.org/c3ref/limit.html>(db,SQLITE_LIMIT_LENGTH<https://www.sqlite.org/c3ref/c_limit_attached.html#sqlitelimitlength>,...) interface.   Maybe there could be some way to leverage PRELUDE_STATEMENTS to achieve this but we are not “opening” a dataset - we’re creating one directly from the memory driver.

What is the best way to work around this?

Best,
Jesse

Lead Computer Scientist
Science Systems and Applications, Inc.
Dr Compton Tucker Team
NASA Goddard Space Flight Center



_______________________________________________

gdal-dev mailing list

gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>

https://lists.osgeo.org/mailman/listinfo/gdal-dev

--

http://www.spatialys.com<http://www.spatialys.com/>

My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250703/a3898f56/attachment-0001.htm>