[GeoNode-users] Huge Datase Upload

[Toni Schönbuchner]

+1

Thanks Michael for this addition!



> Am 20.11.2018 um 15:06 schrieb Michael Fedak <mikefedak at gmail.com>:
> 
> Echoing Francesco and Toni, I have sucessfully deployed 100 GB+ datasets on geoserver and then used updatelayers to make them available via GeoNode. The geotiff compression for dummies article is good, I've tried to follow the advice given but I have to caution that I've found running gdal_retile with those compression settings can take weeks to process your data depending on the specifications of your machone. The issue is that it is single-threaded and image processing is all CPU. gdal2tiles is theoretically capable of multi-core processing and the dev of this tool has also created maptiler: https://www.maptiler.com <https://www.maptiler.com/> which is advertised as being multi-core. 
> 
> On Tue, Nov 20, 2018 at 7:22 AM Toni Schönbuchner <toni.schoenbuchner at csgis.de <mailto:toni.schoenbuchner at csgis.de>> wrote:
> Hi Ana,
> 
> what Francesco suggests is exactly the why I would choose.
> 
> In addition before that I would check if a lossy compression
> could be an option for your imagery. gdal_translate would the
> tool I would choose. Find a useful looking post here:
> http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html <http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html>
> 
> 
> Cheers,
> 
> Toni
> 
>> Am 20.11.2018 um 13:08 schrieb geonode-users-request at lists.osgeo.org <mailto:geonode-users-request at lists.osgeo.org>:
>> 
>> Message: 1
>> Date: Tue, 20 Nov 2018 08:28:33 -0300
>> From: Ana Silva <anasilva.ifpb at gmail.com <mailto:anasilva.ifpb at gmail.com>>
>> To: geonode-users at lists.osgeo.org <mailto:geonode-users at lists.osgeo.org>
>> Subject: [GeoNode-users] Huge Datase Upload
>> Message-ID:
>> 	<CAJtnDSq0eb+4ShCnGHetXjQnLSL9yNn2cSeQp1a+bbC1XVfC1Q at mail.gmail.com <mailto:CAJtnDSq0eb+4ShCnGHetXjQnLSL9yNn2cSeQp1a+bbC1XVfC1Q at mail.gmail.com>>
>> Content-Type: text/plain; charset="utf-8"
>> 
>> Hi everyone, I have a question. In the project I'm working on, we deal with
>> a gigantic dataset (raster files larger than 18GB each, in the best case)
>> and a total dataset size of more than 3TB.
>> And I'm having a problem uploading this data, the uploader always shows me
>> an entity too large message when I try to upload some of this data.
>> I'm using Geonode 2.10.x, installed via Docker.
>> Would anyone have some kind of solution to this problem?
> 
> _______________________________________________
> geonode-users mailing list
> geonode-users at lists.osgeo.org <mailto:geonode-users at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/geonode-users <https://lists.osgeo.org/mailman/listinfo/geonode-users>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/geonode-users/attachments/20181120/768a1d5a/attachment.html>