[postgis-users] creating topology data is slow

[Lars Aksel Opsahl]

Yes that was the first problem we we got and we got dead locks so we where not able to finish the job at all. 

If we get a topology exception we roll back the all the transactions to avoid the problem and having invalid topology data stored in the database.

The way avoid this problem was to only work on lines that was inside each cell and gradually extend/merge the size of the cells. That way the jobs did not cause any problems for each other.

I have only run the “select * from ValidateTopology('topo_ar5_sysdata');” for parts of the data earlier and will start it for the whole layer later this week because I right now I am testing some other stuff. Thanks for reminding me about running the  ValidateTopology again. I will get back to you with the results.

Lars

________________________________________
Fra: Sandro Santilli [sandro.santilli at gmail.com] på vegne av Sandro Santilli [strk at keybit.net]
Sendt: 3. juni 2015 10:49
Til: PostGIS Users Discussion
Kopi: Lars Aksel Opsahl
Emne: Re: [postgis-users] creating topology data is slow

On Wed, Jun 03, 2015 at 05:56:24AM +0000, Lars Aksel Opsahl wrote:
> We are working quite big datasets (about 1 billion points and 20 million lines) that we convert from simple feature to topology. I ran a test yesterday and we added these 20 million lines in about 9 hours with attached attributes. This means heavy parallel pressing and grid usage on a fast database, but we use the Postgis topology package mostly as it is (picked from trunk about a moth ago) .
>
> We will give a talk about how we do this at foss4g 2015 in Como in July.

I hope a video of your talk and a copy of the slides will be made available
as I'd love to see them but will probably not come to Como.

Last time I've heard about parallel processing during topology building
it resulted in a corrupted topology.
How long does it take to ST_ValidateTopology on it ?

--strk;