[Qgis-user] Working with irregular magnetometry data points - converting them to an image

Benjamin Ducke benducke at fastmail.fm
Fri Apr 20 12:08:19 PDT 2018 

On 20/04/18 20:13, C Hamilton wrote:
> Thanks Ben,
> 
> I will give that a try. I am finding that the Interpolation algorithm is
> taking forever on a small data set.

I am not surprised. It iterates over every individual
point in your dataset, then uses slow floating point
computations to try and find other data points within
a threshold distance (probably involving square root
computations, which are among the most inefficient
things to compute). Then it needs to compute a
spatially weighted interpolator of some kind, discard
the result and start all over at the next data point.

By comparison, r.fill.stats pre-computes a set of
(less accurate, but who cares in this case of noisy
and dense data?) spatial weights in a fixed
interpolation window and then just "slides" that over
the rasterized data using nothing but (fast) basic
arithmetic operations.

> 
> Doesn't the Postgres PDAL extension handle point data like this? Would
> there be a benefit of putting the data in PostGIS?

I doubt it. That would give you the overhead of a DBMS in
addition to the sluggishness of common interpolators. If
you want to experiment more with algorithms that
interpolate from vector points, then you should make
sure to build a spatial index. Whether that happens
inside a database or in a file system should make
little difference for the effective running time.

But fastest option is to read the data sequentially
from a file, rasterizing them, then running r.fill.stats
on the rasterized result. At least that's what my
experiments with that type of data amounted to.

That's a "quick and dirty" interpolation, but "dirty"
is also an adequate description of the nature of
magnetometry data, so it's a good fit.

Cheers,

Ben

> 
> Thanks,
> 
> Calvin
> 
> On Fri, Apr 20, 2018 at 1:10 PM, Benjamin Ducke <benducke at fastmail.fm
> <mailto:benducke at fastmail.fm>> wrote:
> 
>     Hi Calvin,
> 
>     On 20/04/18 18:46, Richard McDonnell wrote:
>     > Calvin,
>     > 
>     > Under Processing Toolbox, Type interpolation into the search bar, the
>     > tool you want can be found there.
> 
>     None of the typical GIS interpolation tools will be
>     of much help here, since they have all been optimized
>     to work with relatively sparse data. They will be
>     too inefficient for high-resolution geomagnetic data.
> 
>     You data is already dense, with some gaps. So you need
>     something that is fast and efficient for small
>     interpolation radii, and will not produce too many
>     artefacts.
> 
>     If you can use the GRASS7 plug-in from within QGIS,
>     than this is a good processing path:
> 
>     1. Rasterize your data points, using a cell size
>     small enough so that you don't have too much information
>     loss (i.e. not too many occurrences of more than one data
>     point per cell). This might take a little trial and error
>     to find good settings. You should end up with a raster
>     where you have gaps of max. 1-3 empty cells between cells
>     with measurement data.
> 
>     Now you need to fill these relatively small gaps.
> 
>     2. Use r.fill.stats from current GRASS 7.4:
> 
>     https://grass.osgeo.org/grass74/manuals/r.fill.stats.html
>     <https://grass.osgeo.org/grass74/manuals/r.fill.stats.html>
> 
>     ... because that has been designed exactly for the
>     purpose. Set the interpolation radius as small as
>     possible, so that it just manages to fill the gaps.
>     With a small radius, r.fill.stats is very fast and
>     memory efficient, even on very large raster datasets.
> 
>     Note that r.fill.stats is part of GRASS 7 only since
>     version 7.4. If your QGIS ships with an earlier
>     version, then you might have to update your GRASS 7
>     binaries manually.
> 
>     Best,
> 
>     Ben
> 
>     > Regards,
>     > 
>     > Richard.
>     > 
>     > On 20/04/2018 16:42, C Hamilton wrote:
>     >> I don't know how many of you have worked with archaeological
>     >> magnetormetry data, but I have been asked to help with magnetometry
>     >> data of the ancient Mound Builders in North America. I am looking for
>     >> some suggestion on how to best turn the irregular magnetic data points
>     >> into an image.
>     >>
>     >> I see an Interpolation plugin that perhaps is the solution, but I also
>     >> see that it is not part of QGIS 3. Is this because it has not been
>     >> ported yet, or is not going to be porter, or is there a better solution?
>     >>
>     >> I don't know if any of the Lidar tools would work for this or not.
>     >> LASTools is a commercial product and the locked version only works
>     >> with small data sets and adds noise unless you purchase a license. At
>     >> least that is how it was the last time I used it.
>     >>
>     >> Thanks for any suggestions.
>     >>
>     >> Calvin
>     >>
>     >>
>     >> _______________________________________________
>     >> Qgis-user mailing list
>     >> Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
>     >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>     >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>     > 
>     > -- 
>     > *_Richard McDonnell_*
>     > *GIS Specialist PgD GIS AssocSCSI*
>     > *OPW FRM Data Management*
>     > *52 Stephens Green, Dublin 2.*
>     > TEL: 01 6476543
>     >
>     > <http://www.opw.ie/>
>     >
>     >
>     >
>     > Email Disclaimer: http://www.opw.ie/en/disclaimer/
>     <http://www.opw.ie/en/disclaimer/>
>     > 
>     > 
>     > _______________________________________________
>     > Qgis-user mailing list
>     > Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
>     > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>     > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>     > 
> 
> 
> 
>     -- 
>     Dr. Benjamin Ducke
>     Deutsches Archäologisches Institut (DAI)
>     Zentrale Berlin, IT-Referat
>     * Projekt "Stunde Null" *
>     _______________________________________________
>     Qgis-user mailing list
>     Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
>     List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>     Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     <https://lists.osgeo.org/mailman/listinfo/qgis-user>
> 
> 



-- 
Dr. Benjamin Ducke
Deutsches Archäologisches Institut (DAI)
Zentrale Berlin, IT-Referat
* Projekt "Stunde Null" *

More information about the Qgis-user mailing list