[Pointdown] Following with the simple start of Michael

[Rafael Gaitán]

Hi,

(Sorry for not replying, but I subscribed to the list after he sent the
message)

I'm one of the developers of http://server.pointcloudviz.com:9090,
currently is a functional but still in its early stages pointcloud
streaming server.

I'll follow the structure proposed by Michael because it's quite similar to
what we have been developing there, adding some other proposals/ideas too.

(1) DATA SET - Determined via the URL prefix, e.g.
http://www.example.com/points/serpentmound/ <
http://www.example.com/points/serpentmound/>

I agree with that, but I'll add some more parameters, since there are
multiple possible clients and different purposes, for instance our server
supports dataset selection by url, but also the format, compression, and
also the "pre-generated" tiles

For instance:
- http://server.pointcloudviz.com:9090/getTile/utah/0/0/0/0 -> Retrieves
the level 0 of the utah dataset in what I called "compact Json" format, but
other more verbose format can be developed

- http://server.pointcloudviz.com:9090/getTile/osgjs/utah/0/0/0/0 ->
Retrieves the level 0 in OSGJS format, ready to be displayed on web clients
using OSGJS

- http://server.pointcloudviz.com:9090/getTile/osgjsb/utah/0/0/0/0 -> Same
as before but retrieves the OSGJS node structure and the data is requested
later in binary format

- http://server.pointcloudviz.com:9090/getDBFile/utah/utah.plf -> Retrieves
the index file ready to be used in the PointCloudViz(
http://www.pointcloudviz.com) application.

-
http://server.pointcloudviz.com:9090/getDBFile/utah/0/0/0/utah_L0_X0_Y0_Z0.bin
-> Retrieves the data in binary format ready to be used in WEBGL and OSGJS.

The idea is a web services in which you can decide/provide what is more
convinient for your client application.


(2) METADATA - A json file at the dataset URL root describes things
including the tiling scheme, the bounding boxes of both the level-0 tile
extents and the point cloud data, and the dimensions (name, datatype,
min/mean/max).

I agree again, but following the idea of WebService and using a similar OGC
terminology we propose an API similar to:

- http://server.pointcloudviz.com:9090/getConfig -> Retrieves the
configuration/metadata of the registered datasets

- http://server.pointcloudviz.com:9090/getCapabilities/autzen -> Retrieves
the "capabilites" of a specific dataset

Additionally in our server /getConfig also sends the API, the formats and
the compressors, which has been very handy when I was working developing
the client:

"formats": [
    "osgjs",
    "osgjsb",
    "cjson"
],
"api": {
    "/getCapabilities/:db": "Retrieves the capabilities and statistics of
the selected database",
    "/getConfig": "Retrieves the configuration and API",
    "/getDBFile/:db/*": "Serves direct original files of the database",
    "*": "Other files of the web",
    "/getTile/:format/:compressor/:db/:level/:x/:y/:z": "Retrieves a
specific tile of the selected database in :format format",
    "/getTile/:db/:level/:x/:y/:z": "Retrieves a specific tile of the
selected database in json format",
    "/": "Retrieves the index.html",
    "/getTile/:format/:db/:level/:x/:y/:z": "Retrieves a specific tile of
the selected database in :format format"
},
"compressors": [
    "none",
    "gz"
]

(3) TREE STRUCTURE - Basic quadtree, where a rectangular node at level L
reduces to 4 equally-sized nodes at level L+1. The node payload includes
the point data itself plus a byte at the end that indicates which of the 4
children actually exists on disk; the root tile(s) are guaranteed to exist.
“Top-down” view only: that is, I use rectangles not cubes. The design is
adapted straight from
http://cesiumjs.org/data-and-assets/terrain/formats/heightmap-1.0.html <
http://cesiumjs.org/data-and-assets/terrain/formats/heightmap-1.0.html>.

That approach is perfectly valid for heightmaps and images, but poinclouds
could be real 3D, not only aereal, but also from ground scanners (see:
http://server.pointcloudviz.com:9090/?config=utah.osgjsb)

Our format is and adaptive Octree, which we have developed some tools to
process unlimited size datasets (I've done some tests with 3Billion point
in the pointcloudviz desktop application)

The idea is similar to the quadtree but using as designation 4 values:
Level, X, Y, Z. How this is generated it's independent of the server as
long as you can provide different levels of detail.

Our prototype uses a "lazy" approach and uses the pointcloudviz format for
serving the tiles, but as long as the API is the same, the tiles on disk
could be any format.

(4) PAYLOAD - Each tile contains a set of points covering that tile’s
spatial extent. The points are represented in raw form, in point order: X0,
Y0, Z0, T0, X1, Y1, Z1, T1, etc. Data is stored little-endian in the
datatype described in the JSON header.

This could be one of the formats, or even the default format using the
described API. Also remember that pointclouds are not only "points", they
also have a lot of other useful information as the classification, the
intensity, etc etc that need to be sent/queried to the server.

Our approach sends always all the information per tile, since the number of
requests to the server when using web clients were too many, and that gave
us poor performance, but still more work need to be done on the server side.

(5) POINT SELECTION - All the points are contained in the leaf node tiles
at level N. Each tile at level N-1 contains 1/4 of the points contained in
the level N nodes. The “decimation” process currently used just naively
takes every 4th point, but I would cal this an implementation detail: the
tiling scheme does not mandate how many points should be in a tile or how
they are chosen.

I agree, this is something that the API/Server should solve when the
request is performed. Our prototype still does not support that but it
could be implemented computing the best level for the query and extracting
the tiles that intersect that query, serving them in multiple tiles or one
dataset.


(6) COMMENTS, ASSUMPTIONS, PROBLEMS, ETC:
  - I’m aiming at the simplest approach that could reasonably work, with
the least cost to the server to generate and the client to interpret.

Either your proposal or mine are similar, so the same applies, Our
preliminary work says that the system works, but scalability is something
we need to work for.

  - I currently assume that level 0 has exactly two root tiles covering the
whole globe, but I will generalize it to be able to have a single root tile
for a given region.

In our case, we are projection-independent, so we always start with a
single root tile, which for cesium/WGS84 clients the data could be sent
reprojected.

That's something we need to work more in our server, since not always is
"easy" or standardized the way that LAS files save internally the
projection (if they have it).

  - I was using web sockets, but switched back to http because I’m (now)
assuming that number of bytes in a tile is “reasonable”, i.e. doesn’t
justify the need for sockets
  - No compression is used, although I’d like to gzip each tile. Just
because.
  - Using a “las2tiles” approach to build the tiles, based on PDAL. Should
probably make that code into a Real PDAL Driver at some point.

Using http compression (gzip) headers and a reasonable good server which
supports it, will do the trick for plain text formats, or even binary
formats. We are using that approach for the OSGJS/WebGL client, tiles are
retrieved using that header, and the server sends them gzipped but is the
browser who uncompresses it before being used by JavaScript.

As I mentioned before the tilling system could be generated with las2tiles
or whatever other technology as long as the API is preserved, same as any
other OGC format.

Best regards!
Rafa.

-- 
Rafael Gaitán <rafa.gaitan at mirage-tech.com>
Mirage Technologies S.L
http://www.mirage-tech.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pointdown/attachments/20150215/790a34f1/attachment.html>