[pdal-commits] [PDAL/PDAL] daa8a5: I3S and SLPK readers (#2176)

Fri Oct 5 07:40:56 PDT 2018

  Branch: refs/heads/master
  Home:   https://github.com/PDAL/PDAL
  Commit: daa8a505ccf04be26c582a67abc394e8cc55d3a6
      https://github.com/PDAL/PDAL/commit/daa8a505ccf04be26c582a67abc394e8cc55d3a6
  Author: Kyle Mann <39525199+kylemann16 at users.noreply.github.com>
  Date:   2018-10-05 (Fri, 05 Oct 2018)

  Changed paths:
    M appveyor.yml
    M cmake/examples/hobu-config.sh
    M cmake/examples/hobu-windows.bat
    M cmake/options.cmake
    A doc/stages/readers.i3s.rst
    A doc/stages/readers.slpk.rst
    M pdal/StageExtensions.cpp
    M pdal/StageFactory.cpp
    M pdal/util/FileUtils.cpp
    M pdal/util/FileUtils.hpp
    M plugins/CMakeLists.txt
    A plugins/i3s/CMakeLists.txt
    A plugins/i3s/io/EsriReader.cpp
    A plugins/i3s/io/EsriReader.hpp
    A plugins/i3s/io/EsriUtil.cpp
    A plugins/i3s/io/EsriUtil.hpp
    A plugins/i3s/io/I3SReader.cpp
    A plugins/i3s/io/I3SReader.hpp
    A plugins/i3s/io/SlpkExtractor.cpp
    A plugins/i3s/io/SlpkExtractor.hpp
    A plugins/i3s/io/SlpkReader.cpp
    A plugins/i3s/io/SlpkReader.hpp
    A plugins/i3s/io/pool.hpp
    A plugins/i3s/lepcc/CMakeLists.txt
    A plugins/i3s/lepcc/src/BitMask.cpp
    A plugins/i3s/lepcc/src/BitMask.h
    A plugins/i3s/lepcc/src/BitStuffer2.cpp
    A plugins/i3s/lepcc/src/BitStuffer2.h
    A plugins/i3s/lepcc/src/ClusterRGB.cpp
    A plugins/i3s/lepcc/src/ClusterRGB.h
    A plugins/i3s/lepcc/src/Common.cpp
    A plugins/i3s/lepcc/src/Common.h
    A plugins/i3s/lepcc/src/FlagBytes.cpp
    A plugins/i3s/lepcc/src/FlagBytes.h
    A plugins/i3s/lepcc/src/Huffman.cpp
    A plugins/i3s/lepcc/src/Huffman.h
    A plugins/i3s/lepcc/src/Intensity.cpp
    A plugins/i3s/lepcc/src/Intensity.h
    A plugins/i3s/lepcc/src/LEPCC.cpp
    A plugins/i3s/lepcc/src/LEPCC.h
    A plugins/i3s/lepcc/src/Test_C_Api.cpp
    A plugins/i3s/lepcc/src/include/lepcc_c_api.h
    A plugins/i3s/lepcc/src/include/lepcc_tpl_api.h
    A plugins/i3s/lepcc/src/include/lepcc_types.h
    A plugins/i3s/lepcc/src/lepcc_c_api_impl.cpp
    A plugins/i3s/lepcc/src/utl_const_array.h
    A plugins/i3s/test/SlpkReaderTest.cpp
    A plugins/i3s/test/i3sReaderTest.cpp
    M scripts/appveyor/config.cmd
    M scripts/appveyor/test.cmd
    M scripts/ci/script.sh
    A test/data/i3s/SMALL_AUTZEN_LAS_All .zip
    A test/data/i3s/SMALL_AUTZEN_LAS_All.bin
    A test/data/i3s/SMALL_AUTZEN_LAS_All.slpk
    A test/data/i3s/SMALL_AUTZEN_LAS_All/3dSceneLayer.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/@specialIndexFileHASH128@
    A test/data/i3s/SMALL_AUTZEN_LAS_All/metadata.json
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodepages/0.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodes/0/attributes/1024.bin.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodes/0/attributes/128.bin.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodes/0/attributes/16.bin.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodes/0/attributes/2.bin.pccint
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodes/0/attributes/256.bin.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodes/0/attributes/32.bin.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodes/0/attributes/512.bin.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodes/0/attributes/8.bin.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/nodes/0/geometries/0.bin.pccxyz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/statistics/1.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/statistics/1024.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/statistics/128.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/statistics/16.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/statistics/2.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/statistics/256.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/statistics/32.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/statistics/512.json.gz
    A test/data/i3s/SMALL_AUTZEN_LAS_All/statistics/8.json.gz
    A test/data/i3s/plineAutzen.json
    A test/data/i3s/plineHelsinki.json
    M vendor/arbiter/arbiter.hpp

  Log Message:
  -----------
  I3S and SLPK readers (#2176)

* scans info file and adds dimensions to pointtable layout

* reads binary files from geometries url'

* added lepcc src files to i3s

* Update arbiter bundle for gzip decompression support.

* Add include.

* Include guards.

* Try bundling rapidxml before gzip.

* got decompression working and put into view. need to add gzip decomp and search children nodes

* translates all node geometries, but only xyz's

* RGB, Intensity, XYZ working. Reading only from leaf nodes currently.

* scans info file and adds dimensions to pointtable layout

* reads binary files from geometries url'

* added lepcc src files to i3s

* Update arbiter bundle for gzip decompression support.

* Add include.

* Include guards.

* Try bundling rapidxml before gzip.

* got decompression working and put into view. need to add gzip decomp and search children nodes

* translates all node geometries, but only xyz's

* updated  lepcc repo

* can't get m_arbiter to be declared in lambda scope

* Threading with 2 pools

* Threading works. Adding args

* bounds and thread counts working. tested with info and translate on provided autzen i3s file

* remove tailing / from input url

* making qol improvements

* now working with oriented bounding boxes

* added test files for i3sreader

* pre fetch commit

* Add SlpkExtractor.

* added test file

* can now read slpk as well as from http, created tests for bounding in local and remote files with autzen and small autzen

* added string include to clusterRGB

* added overrides and changed iterator to size_t to match size()

* made visual changes

* A bit of styling consistency.

* More no-op styling tweaks.

* added copyrights

* changed catch to fileExists test. Added slpk to stage extensions

* created docs

* style changes to i3s doc, added doc link to i3sReader.cpp

* added copyright

* Virtualize i3s/slpk readers.

* quat changes

* Quaternion calculations.

* Add pseudo-code for generic i3s dimension handling.

* dimension changes

* Get generic types working.

* String comparison.

* set as changes

* Fix SRS handling.

* changed dimensions for fewer elses

* added slpk reader doc and slpk from i3s doc

* changed loop in esrireader to not use errors. Throw errors on bad lepcc decompressions. Removed random if statement in infokernel. Other minor style fixes.

* removed couts

* Don't change byte order of magic number.

* Better description for createDirectories.

* Made requested changes: shortened file Utils, fixed bad copies in decompression

* adding throwErrors and minor adjustments

* changed from EsriReceiver to EsriUtil. Made some clearer variable names in parsebox

* Doc typos

* Add json check

* added else for undefined dimension

* changed json to be correct

* added dimType map. fixed json test

* elevation no longer inserts to dimmap

* added zlib check to cmake

* fixed thread problems, added await(), added size checks on data)

* switch on I3S

* I3S for appveyor

* fixed test file, fixed if else chain in addDimensions

* get windows happier

* fetch cacert so curl works for ssl, only output curl noise in debug mode

* make I3S tests an optional build/run -- the SLPK reader tests the basic software, and the I3S does remote-over-the-internet requests which take about 60s. to run

* added json checks

* added dimensions option

* added dimensions to docs and made them case insensitive

* unpacking returns correctly

* added lod option and corresponding examples in docs

* added lod tests

* changed lod to depth of node tree

* added depth test

* changed depth to min and max density searches

* updated docs

* adjusted std::max and density logic

* changed parentheses locations

* look for nodePerIndexPage in json

* works with Esri pointcloud 1.6 as long as it's lepcc encoding

* added version functionality. added pipeline jsons to test/data/i3s

* Remove much include file cruft.
Remove EsriUtil code from global pdal namespace.
Use consistent throw/catch.
Formatting stuff.
Various small changes.

* Remove comment.

* added retry block

* removed cout

* changed continue to break in retry

* Simplify ternary logic.

* changed try logic

* Don't throwError() on last line to avoid return warning.

* Fix logic bug.

* changed i3s test variable

* Build cleanup.

* Update arbiter bundle for namespacing gzip functionality.

      **NOTE:** This service has been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

      Functionality will be removed from GitHub.com on January 31st, 2019.