<html><body><div style="color:#000; background-color:#fff; font-family:arial, helvetica, sans-serif;font-size:10pt">Hi Evan,<br><br>I figure most serious users of tools like these will not limit themselves by using a MS Windows platform (& those that do are masochists anyway :-), so using tools like awk (& sed & cut & grep) for pre-processing XYZ files with missing values in any column or other issues is a ready solution for most issues that such files have.<br><div><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><span>For example, for whitespace separated XYZ files which may have missing values:</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><span><br></span></div><div style="color: rgb(0, 0, 0);
font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><span>if the missing values are just missing:<br></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><span>cat $FILE | awk '{ if(NF==3) print $0}' | ....<br></span></div><div><br></div><div>if missing values are represented by some value you want to change (eg: from -99999 to NaN )<br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;">cat $FILE | sed 's/-99999/NaN/g' | ....</div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif;
background-color: transparent; font-style: normal;">The xNIX text processing tools have allowed me do do all the pre-processing I've ever needed, for use with GMT, GDAL, GRASS & R, and unless there are very common specific use cases, I think dome guidelines about using existing tools to solve these problems are more useful than replicating the functionality in GDAL. <br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;">That said, an example of a useful (I think) case is with most GMT commands that take XYZ input, you can use the "-:" command line parameter (Expect lat/lon -YXZ- input/output rather than lon/lat) so you don't need awk... <br></div><div style="color: rgb(0, 0, 0);
font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;">Cheers,</div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;">(with grateful thanks for all your good work!)</div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style:
normal;">Brent<br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br></div> <div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;"> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div dir="ltr"> <hr size="1"> <font face="Arial" size="2"> <b><span style="font-weight:bold;">From:</span></b> Even Rouault <even.rouault@mines-paris.org><br> <b><span style="font-weight: bold;">To:</span></b> gdal-dev@lists.osgeo.org <br> <b><span style="font-weight: bold;">Sent:</span></b> Sunday, May 25, 2014 10:43 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [gdal-dev] Using GDAL with gridded XYZ that has no 'nodata' values?<br>
</font> </div> <div class="y_msg_container"><br>Le samedi 24 mai 2014 21:45:45, Sam Franklin a écrit :<br clear="none">> Greetings. This is my first post to this mailing list, so apologies if I<br clear="none">> commit a mailing-list faux-pas :)<br clear="none">> <br clear="none">> I wonder if anyone can offer a suggestion, as I'm hoping I've missed<br clear="none">> something obvious.<br clear="none">> <br clear="none">> I regularly receive bathymetric DEMs, acquired by acoustic multibeam<br clear="none">> echosounders (MBES) from offshore geophysical surveys that take the form of<br clear="none">> narrow corridors of survey or irregularly shaped areas.<br clear="none">> <br clear="none">> My goal is to have a end-to-end open source/GDAL workflow that converts the<br clear="none">> bathymetric DEM to greyscale GEOTIFF which I can then use for further<br clear="none">> processing/analysis using GDAL/OGR.<br
clear="none">> <br clear="none">> The convention in the offshore sector is to deliver bathymetric data as<br clear="none">> single 'gridded' .XYZ ASCII file, where XY coordinate pairs are evenly<br clear="none">> spaced to form a grid and the textfile row order is spatially sequential.<br clear="none">> <br clear="none">> However, crucially 'nodata' values are omitted from the XYZ. I can only<br clear="none">> assume the reasoning was to reduce the file size on disc. This is an<br clear="none">> industry convention that will not change anytime soon.<br clear="none">> <br clear="none">> gdal_translate obviously does not read these XYZ files as gdal throws an<br clear="none">> error due to the lack of true rectangular grid. I've read the XYZ info<br clear="none">> <a shape="rect" href="http://www.gdal.org/frmt_xyz.html" target="_blank">http://www.gdal.org/frmt_xyz.html </a>which is clear to me, no problem
there.<br clear="none">> <br clear="none">> I do not want to use gdal_grid on the XYZ as this will introduce a further<br clear="none">> interpolation step. The XYZs I receive are already cleansed and gridded by<br clear="none">> the surveyor which I don't wish to edit further.<br clear="none">> <br clear="none">> So, I currently use a proprietary marine geophysical package which can<br clear="none">> directly read the gridded XYZ without nodata values. Using this package, I<br clear="none">> can export to ESRI ASCII grid (.ASC) which I then use gdal_translate<br clear="none">> without issue for my further processing steps.<br clear="none">> <br clear="none">> I'd like the freedom of a non-proprietary solution and the only thing I can<br clear="none">> think of is to write a python script to pre-process my XYZ into a format<br clear="none">> that gdal will accept. I was thinking of calculating the XY
bounding box<br clear="none">> of the data and output an XYZ that blends the original values and nodata<br clear="none">> values in the correct row order.<br clear="none">> <br clear="none">> If anyone is interested, see below link to a zip of three separate<br clear="none">> examples, containing:<br clear="none">> -- 'gridded' XYZ (without nodata values)<br clear="none">> -- processed greyscale GEOTIFF (using proprietary package)<br clear="none">> -- processed color-relief RGB GEOTIFF (using proprietary package)<br clear="none">> <a shape="rect" href="https://dl.dropboxusercontent.com/u/4086367/gdal_translate_xyz_examples.zip" target="_blank">https://dl.dropboxusercontent.com/u/4086367/gdal_translate_xyz_examples.zip</a><br clear="none">> All data in these examples are referenced to SRS EPSG:27700.<br clear="none">> <br clear="none">> If I've missed anything obvious, or if anyone has any thoughts of how I can<br
clear="none">> achieve my goal using GDAL or other, or comment on my proposed solution,<br clear="none">> that would be greatly appreciated.<br clear="none"><br clear="none">What would be interesting is a link to a XYZ file that fails to be opened. In <br clear="none">GDAL 1.11, I've improved the driver to be able to handle lines with missing <br clear="none">values at their start and end. It might be possible to improve it again to <br clear="none">deal with nodata values that would be at any place within lines. If the whole <br clear="none">file is regularly gridded of course.<br clear="none"><br clear="none">> <br clear="none">> Thanks in advance.<br clear="none">> Sam<br clear="none"><br clear="none">-- <br clear="none">Geospatial professional services<br clear="none"><a shape="rect" href="http://even.rouault.free.fr/services.html" target="_blank">http://even.rouault.free.fr/services.html</a><div class="yqt6464706770"
id="yqtfd50612"><br clear="none">_______________________________________________<br clear="none">gdal-dev mailing list<br clear="none"><a shape="rect" ymailto="mailto:gdal-dev@lists.osgeo.org" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br clear="none"><a shape="rect" href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a></div><br><br></div> </div> </div> </div></body></html>