[gdal-dev] Large GeoJSONs and aborting file opening

[Simon Eves]

Agreed. Unfortunately, we're looking for a quick solution to a customer
complaint. I'll ponder it some more.

On Wed, Jul 28, 2021 at 12:28 PM Even Rouault <even.rouault at spatialys.com>
wrote:

> Simon,
>
> I don't think that killing a thread is a safe practice in general. It
> would likely result in memory leaks and maybe in some other inconsistent
> state that could crash the whole process.  An interesting enhancement for
> such cases would be to be able to provide a progress / abort callback.
>
> Even
> Le 28/07/2021 à 21:22, Simon Eves a écrit :
>
> Dear All,
>
> I am aware that some improvements were made in the 2.3 timeframe with
> regards to dealing with large GeoJSON files, although even in 3.2, it's
> still very slow and memory hungry.
>
> Our system allows for aborting imports, but this only works reliably once
> it has actually got to the stage of reading features from the file. With
> the GeoJSON, it just sits in the GDALOpenEx call for ages.
>
> My question, therefore, is whether it might be practical to run the
> GDALOpenEx in a separate thread with a future to return the resulting
> handle, such that it could be monitored and killed if necessary?
>
> Mainly I would be concerned that killing the thread might trash some
> global GDAL state that might then not be recoverable, or that the open
> relies on some TLS for the process thread and therefore might not work
> properly.
>
> We're going to try it anyway, but opinions welcomed, thanks!
>
> Simon
>
> --
> <http://www.omnisci.com/>
> Simon Eves
> Senior Graphics Engineer, Rendering Group
> 100 Montgomery St (5th Floor), San Francisco, CA 94104, USA
>
>
> Email: simon.eves at omnisci.com | Cell:  +1 (415) 902-1996
>
>
> _______________________________________________
> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>

-- 
<http://www.omnisci.com/>
Simon Eves
Senior Graphics Engineer, Rendering Group
100 Montgomery St (5th Floor), San Francisco, CA 94104, USA


Email: simon.eves at omnisci.com | Cell:  +1 (415) 902-1996
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210728/9f44382d/attachment-0001.html>