[GeoNode-users] Huge Datase Upload

[Michael Fedak]

Echoing Francesco and Toni, I have sucessfully deployed 100 GB+ datasets on
geoserver and then used updatelayers to make them available via GeoNode.
The geotiff compression for dummies article is good, I've tried to follow
the advice given but I have to caution that I've found running gdal_retile
with those compression settings can take weeks to process your data
depending on the specifications of your machone. The issue is that it is
single-threaded and image processing is all CPU. gdal2tiles is
theoretically capable of multi-core processing and the dev of this tool has
also created maptiler: https://www.maptiler.com which is advertised as
being multi-core.

On Tue, Nov 20, 2018 at 7:22 AM Toni Schönbuchner <
toni.schoenbuchner at csgis.de> wrote:

> Hi Ana,
>
> what Francesco suggests is exactly the why I would choose.
>
> In addition before that I would check if a lossy compression
> could be an option for your imagery. gdal_translate would the
> tool I would choose. Find a useful looking post here:
> http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html
>
>
> Cheers,
>
> Toni
>
> Am 20.11.2018 um 13:08 schrieb geonode-users-request at lists.osgeo.org:
>
> Message: 1
> Date: Tue, 20 Nov 2018 08:28:33 -0300
> From: Ana Silva <anasilva.ifpb at gmail.com>
> To: geonode-users at lists.osgeo.org
> Subject: [GeoNode-users] Huge Datase Upload
> Message-ID:
> <CAJtnDSq0eb+4ShCnGHetXjQnLSL9yNn2cSeQp1a+bbC1XVfC1Q at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi everyone, I have a question. In the project I'm working on, we deal with
> a gigantic dataset (raster files larger than 18GB each, in the best case)
> and a total dataset size of more than 3TB.
> And I'm having a problem uploading this data, the uploader always shows me
> an entity too large message when I try to upload some of this data.
> I'm using Geonode 2.10.x, installed via Docker.
> Would anyone have some kind of solution to this problem?
>
>
> _______________________________________________
> geonode-users mailing list
> geonode-users at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/geonode-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/geonode-users/attachments/20181120/89ebfaf9/attachment.html>