<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:"Segoe UI Symbol";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:285746707;
mso-list-type:hybrid;
mso-list-template-ids:661823012 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt'>The main overhead I see with postgis raster are the following<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><ol style='margin-top:0in' start=1 type=1><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'><span style='font-size:11.0pt'>Limitation of raster sizes of about 1GB as I recall<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'><span style='font-size:11.0pt'>Detoasting of data which is almost always needed <a href="https://www.postgresql.org/docs/current/storage-toast.html">https://www.postgresql.org/docs/current/storage-toast.html</a><o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'><span style='font-size:11.0pt'>Mem copying of data, probably more so than using say GDAL directly<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'><span style='font-size:11.0pt'>Probably easier to do parallel operations on command line (though admittedly I haven’t tried)<o:p></o:p></span></li></ol><p class=MsoListParagraph><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Admittedly my experience with using GDAL and python directly is very low compared to doing everything in the database so I really don’t know the benefits of doing it all in GDAL.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>I still prefer to use the database hammer over having to learn yet another tool. My sense is also that the amount of SQL/plpgsql code you have to write would be much less than trying to pull your stuff out of the DB and stuff it in python/GDAL to do the same thing. This is all to say if the performance is good enough for you, I say go for that hammer even if you think you are just dealing with a nail.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Thiemo Kellner <thiemo@gelassene-pferde.biz> <br><b>Sent:</b> Friday, November 8, 2024 12:16 PM<br><b>To:</b> PostGIS Users <postgis-users@lists.osgeo.org><br><b>Subject:</b> Re: Composing raster tiles?<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks for sharing your experience. Is that saying that PostGIS has considerable overhead? Sorry, if that is naiv. I might be suffering from the hammer nail deficiency. My background is very database heavy, so too much might look like a database to me. <o:p></o:p></p><div><div><p>08.11.2024 16:15:32 Vera Green <<a href="mailto:vera.green.ca@gmail.com">vera.green.ca@gmail.com</a>>:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 2.25pt;padding:0in 0in 0in 8.0pt;margin-left:0in;margin-right:0in'><p>We use command line GDAL for all our Easter processes, if your data is large I recommend you look into that option.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Fri, Nov 8, 2024, 6:12<span style='font-family:"Arial",sans-serif'> </span>AM David Haynes <<a href="mailto:haynesd2@gmail.com">haynesd2@gmail.com</a>> wrote: <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><p class=MsoNormal>I'll try and help you with A & B <o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>a) Is it more efficient to convert the raster to vector data and calculate on the those than to calculate directly on the raster? <o:p></o:p></p></div><div><p class=MsoNormal>>> I don't think it would be faster to convert to vector because that would be dumping the raster into polygons. I think for specifically calculating slope, you are better off staying in raster. <br><br>b) To my understanding, if I calculate the slope on a raster tile, the slope,… of the borders will have accuracy problems. My I idea, was to "stitch" a tile with its direct neighbours, calculate on the composed tile, and either save a cropped calculated composed tile to its original dimension or save the calculated composed tile as is, probably the latter. <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>>> Overall correct, some small adjustments. For any raster operation, the tiles are operated on independently, so you need to "stitch" them together. I've provided a couple of ways in pseduo code <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Way 1 <o:p></o:p></p></div><div><p class=MsoNormal>1) Use ST_Union and make a big tile, <o:p></o:p></p></div><div><p class=MsoNormal>2) Do the spatial operation, <o:p></o:p></p></div><div><p class=MsoNormal>3) Break it up using ST_Tile() <br>The downside is you might run out of memory doing this. Also consider ST_MemUnion() <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>SELECT ST_Tile(ST_Slope(ST_Union(r1.ras)), 350,350) <o:p></o:p></p></div><div><p class=MsoNormal>FROM raster_table <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Way 2) <o:p></o:p></p></div><div><p class=MsoNormal>A second option is to basically create an aggregate, which is likely faster. <o:p></o:p></p></div><div><p class=MsoNormal>1) Union the tiles based on ST_Touch - make mega_tiles <o:p></o:p></p></div><div><p class=MsoNormal>2) Do the spatial operation on the mega_tiles <o:p></o:p></p></div><div><p class=MsoNormal>3) Clip the mega_tiles by the old tiles bounding box <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>WITH rtest as <br>( <br>SELECT r1.ras, ST_Union (r1.ras) as megatile <br>FROM raster_table r1 <br>LEFT JOIN raster_table r2 <br>ON ST_Touches (r1.geom, r2.geom) <br>GROUP BY r1.ras <br>) <br>SELECT ST_CLIP(ST_SLOPE(megatile), ST_Envelop(r1.rast) ) as ras <br>FROM rtest <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Maybe someone wants to make an aggregate for the raster functions? <o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Thu, Nov 7, 2024 at 3:31<span style='font-family:"Arial",sans-serif'> </span>PM <<a href="mailto:thiemo@gelassene-pferde.biz" target="_blank">thiemo@gelassene-pferde.biz</a>> wrote: <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal style='margin-bottom:12.0pt'>Hi <br><br>In my project <br><a href="https://sourceforge.net/p/treintaytres/code/HEAD/tree/trunk/code_files/data_storage/" target="_blank">https://sourceforge.net/p/treintaytres/code/HEAD/tree/trunk/code_files/data_storage/</a> I have the <br>table <br><br>TABLE_SCHEMA TABLE_NAME DATA_TYPE TYPE_NAME COLUMN_NAME <br>treintaytres topo_files<span style='font-family:"Segoe UI Symbol",sans-serif'>␟</span>t 1111 uuid id <br>treintaytres topo_files<span style='font-family:"Segoe UI Symbol",sans-serif'>␟</span>t 93 timestamptz entry_pit <br>treintaytres topo_files<span style='font-family:"Segoe UI Symbol",sans-serif'>␟</span>t 1111 uuid source_id <br>treintaytres topo_files<span style='font-family:"Segoe UI Symbol",sans-serif'>␟</span>t 12 text file_name <br>treintaytres topo_files<span style='font-family:"Segoe UI Symbol",sans-serif'>␟</span>t 1111 raster tile <br>treintaytres topo_files<span style='font-family:"Segoe UI Symbol",sans-serif'>␟</span>t 93 timestamptz file_creation_pit <br>treintaytres topo_files<span style='font-family:"Segoe UI Symbol",sans-serif'>␟</span>t 12 text file_hash <br><br>TILE contains topographical height raster data from OpenTopography. <br>They are from different regions, let's say, some tiles cover <br>Switzerland, some cover New Zealand. I want to create slope and other <br>data from the height data and I have some questions I hope you can <br>answer or point me to answers. <br><br>a) Is it more efficient to convert the raster to vector data and <br>calculate on the those than to calculate directly on the raster? <br><br>b) To my understanding, if I calculate the slope on a raster tile, the <br>slope,… of the borders will have accuracy problems. My I idea, was to <br>"stitch" a tile with its direct neighbours, calculate on the composed <br>tile, and either save a cropped calculated composed tile to its <br>original dimension or save the calculated composed tile as is, <br>probably the latter. <br>Can I compose as follows? <br>with RASTER_NEIGHBORS as ( select R1.TILE as CURRENT_TILE <br> ,R2.TILE as NEIGHBOR_TILE <br> ,<a href="http://R1.ID" target="_blank">R1.ID</a> as CURRENT_ID <br> from TOPO_FILES<span style='font-family:"Segoe UI Symbol",sans-serif'>␟</span>T R1 <br> left outer join TOPO_FILES<span style='font-family:"Segoe UI Symbol",sans-serif'>␟</span>T R2 <br> on ST_Touches(R1.TILE <br> ,R2.TILE) <br> or ST_Intersects(R1.TILE <br> ,R2.TILE) <br> where TRUE <br> --and <a href="http://R1.ID" target="_blank">R1.ID</a> = <br>'6b8ca53a-bb5f-4c2b-a9c9-94b6a706e9b0' <br> and TRUE) <br> ,NION as (select CURRENT_TILE as TILE <br> ,CURRENT_ID <br> from RASTER_NEIGHBORS <br> union <br> select NEIGHBOR_TILE as TILE <br> ,CURRENT_ID <br> from RASTER_NEIGHBORS) <br> select ST_Union(TILE) as COMPOSED_TILE <br> ,CURRENT_ID <br> from NION <br>group by CURRENT_ID; <br><br>c) Finally, I want to select all the areas where slope, TRI,… conform <br>certain criteria and have a minium surface size. Do I do it this <br>better on vector data and do I need to do this on data composed of all <br>the contiguous areas? <br><br>I would be grateful for any nudge into the right direction. Maybe URLs <br>with samples. <br><br>Kind regards <br><br>Thiemo <o:p></o:p></p></blockquote></div></blockquote></div></blockquote></div></div></div></body></html>