[gdal-dev] Large GeoJSONs and aborting file opening

Simon Eves simon.eves at omnisci.com
Wed Jul 28 12:22:12 PDT 2021


Dear All,

I am aware that some improvements were made in the 2.3 timeframe with
regards to dealing with large GeoJSON files, although even in 3.2, it's
still very slow and memory hungry.

Our system allows for aborting imports, but this only works reliably once
it has actually got to the stage of reading features from the file. With
the GeoJSON, it just sits in the GDALOpenEx call for ages.

My question, therefore, is whether it might be practical to run the
GDALOpenEx in a separate thread with a future to return the resulting
handle, such that it could be monitored and killed if necessary?

Mainly I would be concerned that killing the thread might trash some global
GDAL state that might then not be recoverable, or that the open relies on
some TLS for the process thread and therefore might not work properly.

We're going to try it anyway, but opinions welcomed, thanks!

Simon

-- 
<http://www.omnisci.com/>
Simon Eves
Senior Graphics Engineer, Rendering Group
100 Montgomery St (5th Floor), San Francisco, CA 94104, USA


Email: simon.eves at omnisci.com | Cell:  +1 (415) 902-1996
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210728/049f0869/attachment.html>


More information about the gdal-dev mailing list