<div dir="auto">Hi David, <div dir="auto"><br></div><div dir="auto">That blog article is from 12 years ago, and was presenting the best solution available at that time. Several new features have been included in the meantime. For example, the usage of LATERAL JOIN would allow in your case to avoid the loop over the records in your pl/pgsql procedure, which is what is overloading your DB in the execution.</div><div dir="auto"><br></div><div dir="auto">Giuseppe.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 2 Apr 2022, 08:43 Shaozhong SHI, <<a href="mailto:shishaozhong@gmail.com">shishaozhong@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">This failed with an error message saying invalid allocation of memory.<div><br></div><div><a href="http://blog.cleverelephant.ca/2010/07/network-walking-in-postgis.html" target="_blank" rel="noreferrer">Network Walking in PostGIS · Paul Ramsey (cleverelephant.ca)</a><br></div><div><br></div><div>The following works well on small data sets (9000 rows) and do not know how long it would run for 4-5 million rows.</div><div><br></div><div><pre style="margin-top:0px;margin-bottom:0px;padding:12px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:1.30769;font-size:13px;vertical-align:baseline;box-sizing:inherit;width:auto;max-height:600px;overflow:auto;border-radius:5px"><code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">DO
$$
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">DECLARE</span>
this_id <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">bigint</span>;
this_geom geometry;
cluster_id_match <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">integer</span>;
id_a <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">bigint</span>;
id_b <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">bigint</span>;
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">BEGIN</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">DROP</span> <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">TABLE</span> IF <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">EXISTS</span> clusters;
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">CREATE</span> <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">TABLE</span> clusters (cluster_id serial, ids <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">bigint</span>[], geom geometry);
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">CREATE</span> INDEX <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">ON</span> clusters <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">USING</span> GIST(geom);
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">-- Iterate through linestrings, assigning each to a cluster (if there is an intersection)</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">-- or creating a new cluster (if there is not)</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">FOR</span> this_id, this_geom <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">IN</span> <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SELECT</span> id, geom <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">FROM</span> lines LOOP
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">-- Look for an intersecting cluster. (There may be more than one.)</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SELECT</span> cluster_id <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">FROM</span> clusters <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">WHERE</span> ST_Intersects(this_geom, clusters.geom)
LIMIT <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">1</span> <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">INTO</span> cluster_id_match;
IF cluster_id_match <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">IS</span> <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">NULL</span> <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">THEN</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">-- Create a new cluster</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">INSERT</span> <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">INTO</span> clusters (ids, geom) <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">VALUES</span> (<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">ARRAY</span>[this_id], this_geom);
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">ELSE</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">-- Append line to existing cluster</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">UPDATE</span> clusters <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SET</span> geom <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">=</span> ST_Union(this_geom, geom),
ids <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">=</span> array_prepend(this_id, ids)
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">WHERE</span> clusters.cluster_id <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">=</span> cluster_id_match;
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">END</span> IF;
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">END</span> LOOP;
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">-- Iterate through the clusters, combining clusters that intersect each other</span>
LOOP
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SELECT</span> a.cluster_id, b.cluster_id <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">FROM</span> clusters a, clusters b
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">WHERE</span> ST_Intersects(a.geom, b.geom)
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">AND</span> a.cluster_id <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit"><</span> b.cluster_id
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">INTO</span> id_a, id_b;
EXIT <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">WHEN</span> id_a <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">IS</span> <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">NULL</span>;
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">-- Merge cluster A into cluster B</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">UPDATE</span> clusters a <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SET</span> geom <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">=</span> ST_Union(a.geom, b.geom), ids <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">=</span> array_cat(a.ids, b.ids)
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">FROM</span> clusters b
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">WHERE</span> a.cluster_id <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">=</span> id_a <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">AND</span> b.cluster_id <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">=</span> id_b;
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">-- Remove cluster B</span>
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">DELETE</span> <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">FROM</span> clusters <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">WHERE</span> cluster_id <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">=</span> id_b;
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">END</span> LOOP;
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">END</span>;
$$ <span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">language</span> plpgsql;</code></pre><pre style="margin-top:0px;margin-bottom:0px;padding:12px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:1.30769;font-size:13px;vertical-align:baseline;box-sizing:inherit;width:auto;max-height:600px;overflow:auto;border-radius:5px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;padding:12px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:1.30769;font-size:13px;vertical-align:baseline;box-sizing:inherit;width:auto;max-height:600px;overflow:auto;border-radius:5px">Regards,</pre><pre style="margin-top:0px;margin-bottom:0px;padding:12px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:1.30769;font-size:13px;vertical-align:baseline;box-sizing:inherit;width:auto;max-height:600px;overflow:auto;border-radius:5px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;padding:12px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:1.30769;font-size:13px;vertical-align:baseline;box-sizing:inherit;width:auto;max-height:600px;overflow:auto;border-radius:5px">David</pre></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 2 Apr 2022 at 08:26, Alexandre Neto <<a href="mailto:senhor.neto@gmail.com" target="_blank" rel="noreferrer">senhor.neto@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><span>It really depends on many factors, but mainly of what you are trying to do. Can you explain what you are trying to do, can you share your query and data structure?</span></div></div><br><div><div>On Sat Apr 2, 2022, 01:33 AM GMT, <a href="mailto:shishaozhong@gmail.com" target="_blank" rel="noreferrer">Shaozhong SHI</a> wrote:<br></div><blockquote style="margin:0px 0px 0px 4pt;padding-left:4pt;border-left:1px solid rgb(204,204,204)"><div><div dir="ltr"><div dir="ltr">I have a script running to iterate over 4-5 million rows. It keeps showing up in red in PgAdmin. It remains active.<div><br></div><div>How long does iteration over 4-5 million rows usually take?</div><div><br></div><div>Regards,</div><div><br></div><div>David</div></div></div>
</div></blockquote></div></div>
_______________________________________________<br>
postgis-users mailing list<br>
<a href="mailto:postgis-users@lists.osgeo.org" target="_blank" rel="noreferrer">postgis-users@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/postgis-users" rel="noreferrer noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/postgis-users</a><br>
</blockquote></div>
_______________________________________________<br>
postgis-users mailing list<br>
<a href="mailto:postgis-users@lists.osgeo.org" target="_blank" rel="noreferrer">postgis-users@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/postgis-users" rel="noreferrer noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/postgis-users</a><br>
</blockquote></div>