[gdal-dev] Large GeoJSONs and aborting file opening

[Simon Eves]

We could, but it does import if you're willing to leave it long enough. I
guess we could throw an "are you sure?" if the file is large. However,
being able to kill it without bringing down the whole server is still
desirable!

On Thu, Jul 29, 2021 at 11:09 AM Andrew Bell <andrew.bell.ia at gmail.com>
wrote:

> Not exactly what you asked, but can you simply avoid opening the file at
> all if it's too large rather than start the process and then abort?
>
> On Wed, Jul 28, 2021 at 7:10 PM Simon Eves <simon.eves at omnisci.com> wrote:
>
>> Agreed. Unfortunately, we're looking for a quick solution to a customer
>> complaint. I'll ponder it some more.
>>
>> On Wed, Jul 28, 2021 at 12:28 PM Even Rouault <even.rouault at spatialys.com>
>> wrote:
>>
>>> Simon,
>>>
>>> I don't think that killing a thread is a safe practice in general. It
>>> would likely result in memory leaks and maybe in some other inconsistent
>>> state that could crash the whole process.  An interesting enhancement for
>>> such cases would be to be able to provide a progress / abort callback.
>>>
>>> Even
>>> Le 28/07/2021 à 21:22, Simon Eves a écrit :
>>>
>>> Dear All,
>>>
>>> I am aware that some improvements were made in the 2.3 timeframe with
>>> regards to dealing with large GeoJSON files, although even in 3.2, it's
>>> still very slow and memory hungry.
>>>
>>> Our system allows for aborting imports, but this only works reliably
>>> once it has actually got to the stage of reading features from the file.
>>> With the GeoJSON, it just sits in the GDALOpenEx call for ages.
>>>
>>> My question, therefore, is whether it might be practical to run the
>>> GDALOpenEx in a separate thread with a future to return the resulting
>>> handle, such that it could be monitored and killed if necessary?
>>>
>>> Mainly I would be concerned that killing the thread might trash some
>>> global GDAL state that might then not be recoverable, or that the open
>>> relies on some TLS for the process thread and therefore might not work
>>> properly.
>>>
>>> We're going to try it anyway, but opinions welcomed, thanks!
>>>
>>> Simon
>>>
>>> --
>>> <http://www.omnisci.com/>
>>> Simon Eves
>>> Senior Graphics Engineer, Rendering Group
>>> 100 Montgomery St (5th Floor), San Francisco, CA 94104, USA
>>>
>>>
>>> Email: simon.eves at omnisci.com | Cell:  +1 (415) 902-1996
>>>
>>>
>>> _______________________________________________
>>> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>>> -- http://www.spatialys.com
>>> My software is free, but my time generally not.
>>>
>>>
>>
>> --
>> <http://www.omnisci.com/>
>> Simon Eves
>> Senior Graphics Engineer, Rendering Group
>> 100 Montgomery St (5th Floor), San Francisco, CA 94104, USA
>>
>>
>> Email: simon.eves at omnisci.com | Cell:  +1 (415) 902-1996
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
> --
> Andrew Bell
> andrew.bell.ia at gmail.com
>


-- 
<http://www.omnisci.com/>
Simon Eves
Senior Graphics Engineer, Rendering Group
100 Montgomery St (5th Floor), San Francisco, CA 94104, USA


Email: simon.eves at omnisci.com | Cell:  +1 (415) 902-1996
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210729/be6ebda2/attachment.html>