[postgis-users] Topology creation performance

[Alexandre Silva]

On 23/11/20 07:39, Lars Aksel Opsahl wrote:
>
> Hi
>
>
> At NIBIO we have got Postgis Topology to perform quite ok with more
> than 25 million edges that represents land, water, roads, field types
> and more. We use topology.TopoGeo_addLinestring.
>
>
> To get this to work we had to use content based grids
> (https://github.com/larsop/content_balanced_grid) and work inside each
> grid until each single cell are done and then start to merge cells
> together and the end. The process of merging cells is more time
> consuming related to too each edge, but the number of edges are also
> limited because we only have to work with edges that cross cell borders.
>
>
> Using content based grids has advantages like
>
>  *
>
>     You can safely wOn 23/11/20 07:39, Lars Aksel Opsahl wrote:ork in
>     parallel
>
>  *
>
>     It performance good when building up new large topology datasets.
>
>
> In the case below we have more than 25 million edges that we split up
> into around 7000 cells and we see the number off cells handled pr hour
> below are not decreasing, when running 20 threads in parallel. (The
> number of edges are not equal pr cell, but limited to a max number of
> polygons pr. cell so the idea is to vary the size of cell to get the
> workload pr cell more equal)
>
>
> 852
>
> 1113
>
> 840
>
> 563
>
> 461
>
> 541
>
> 583
>
> 704
>
> 705
>
>
> Before we started to use content based grids we had the same problems
> as you describe here, the performance decreased when starting to work
> with big datasets.
>
>
> You find the code I used here
> https://github.com/larsop/resolve-overlap-and-gap
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flarsop%2Fresolve-overlap-and-gap&data=04%7C01%7Camsilva%40infoportugal.impresa.pt%7C259b72e4e3de463a867608d88f82f7ec%7Cd227b2e71c404f63b5132f3665c334e6%7C0%7C0%7C637417139959964915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=JLx0AJGgL%2FkcRCbngapI4musohbE6s32M0kOQgaUqVc%3D&reserved=0>
> if you want more info.
>
>
> Lars
>
>
>
>
> _______________________________________________
> postgis-users mailing list
> postgis-users at lists.osgeo.org
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fpostgis-users&data=04%7C01%7Camsilva%40infoportugal.impresa.pt%7C259b72e4e3de463a867608d88f82f7ec%7Cd227b2e71c404f63b5132f3665c334e6%7C0%7C0%7C637417139959984905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=giHwlsDnMLC7q4NmPZjM%2FCnbFLaW5Xl0ZPKD3BaBCtw%3D&reserved=0

Hi and thanks for your input,


I saw your foss4g presentation and your approach was one that I was
considering use, but I hadn't find the source code yet outside of the
slides.

Do the repo approach have the same ruleset as the presentation? I have
the need of processing only cells when there is no line falling outside
that cell or the ordering may not be respected, so the cell merging is
also a point that I have to ensure.


Thanks,

Alexandre Silva

[http://newsletter.impresapublishing.pt/i/barra_ip.jpg]