[gdal-dev] Using GDAL with gridded XYZ that has no 'nodata' values?

[Brent Wood]

Hi Evan,

I figure most serious users of tools like these will not limit themselves by using a MS Windows platform (& those that do are masochists anyway :-), so using tools like awk (& sed & cut & grep) for pre-processing XYZ files with missing values in any column or other issues is a ready solution for most issues that such files have.


For example, for whitespace separated XYZ files which may have missing values:

if the missing values are just missing:

cat $FILE | awk '{ if(NF==3) print $0}' | ....


if missing values are represented by some value you want to change (eg: from -99999 to NaN )

cat $FILE | sed 's/-99999/NaN/g' | ....

The xNIX text processing tools have allowed me do do all the pre-processing I've ever needed, for use with GMT, GDAL, GRASS & R, and unless there are very common specific use cases, I think dome guidelines about using existing tools to solve these problems are more useful than replicating the functionality in GDAL. 


That said, an example of a useful (I think) case is with most GMT commands that take XYZ input, you can use the "-:" command line parameter (Expect lat/lon -YXZ- input/output rather than lon/lat) so you don't need awk...  


Cheers,

(with grateful thanks for all your good work!)

Brent




________________________________
 From: Even Rouault <even.rouault at mines-paris.org>
To: gdal-dev at lists.osgeo.org 
Sent: Sunday, May 25, 2014 10:43 AM
Subject: Re: [gdal-dev] Using GDAL with gridded XYZ that has no 'nodata'	values?
 

Le samedi 24 mai 2014 21:45:45, Sam Franklin a écrit :
> Greetings. This is my first post to this mailing list, so apologies if I
> commit a mailing-list faux-pas :)
> 
> I wonder if anyone can offer a suggestion, as I'm hoping I've missed
> something obvious.
> 
> I regularly receive bathymetric DEMs, acquired by acoustic multibeam
> echosounders (MBES) from offshore geophysical surveys that take the form of
> narrow corridors of survey or irregularly shaped areas.
> 
> My goal is to have a end-to-end open source/GDAL workflow that converts the
> bathymetric DEM to greyscale GEOTIFF which I can then use for further
> processing/analysis using GDAL/OGR.
> 
> The convention in the offshore sector is to deliver bathymetric data as
> single 'gridded' .XYZ ASCII file, where XY coordinate pairs are evenly
> spaced to form a grid and the textfile row order is spatially sequential.
> 
> However, crucially 'nodata' values are omitted from the XYZ.  I can only
> assume the reasoning was to reduce the file size on disc. This is an
> industry convention that will not change anytime soon.
> 
> gdal_translate obviously does not read these XYZ files as gdal throws an
> error due to the lack of true rectangular grid.  I've read the XYZ info
> http://www.gdal.org/frmt_xyz.html which is clear to me, no problem there.
> 
> I do not want to use gdal_grid on the XYZ as this will introduce a further
> interpolation step. The XYZs I receive are already cleansed and gridded by
> the surveyor which I don't wish to edit further.
> 
> So, I currently use a proprietary marine geophysical package which can
> directly read the gridded XYZ without nodata values.  Using this package, I
> can export to ESRI ASCII grid (.ASC) which I then use gdal_translate
> without issue for my further processing steps.
> 
> I'd like the freedom of a non-proprietary solution and the only thing I can
> think of is to write a python script to pre-process my XYZ into a format
> that gdal will accept.  I was thinking of calculating the XY bounding box
> of the data and output an XYZ that blends the original values and nodata
> values in the correct row order.
> 
> If anyone is interested, see below link to a zip of three separate
> examples, containing:
> -- 'gridded' XYZ (without nodata values)
> -- processed greyscale GEOTIFF (using proprietary package)
> -- processed color-relief RGB GEOTIFF (using proprietary package)
> https://dl.dropboxusercontent.com/u/4086367/gdal_translate_xyz_examples.zip
> All data in these examples are referenced to SRS EPSG:27700.
> 
> If I've missed anything obvious, or if anyone has any thoughts of how I can
> achieve my goal using GDAL or other, or comment on my proposed solution,
> that would be greatly appreciated.

What would be interesting is a link to a XYZ file that fails to be opened. In 
GDAL 1.11, I've improved the driver to be able to handle lines with missing 
values at their start and end. It might be possible to improve it again to 
deal with nodata values that would be at any place within lines. If the whole 
file is regularly gridded of course.

> 
> Thanks in advance.
> Sam

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html

_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20140524/878796d7/attachment-0001.html>