[mapserver-users] Hello from Finland

Tue Mar 16 03:54:13 PDT 2021

 Hi Stefan,

When it comes to memory usage with raster files, Mapserver is reading rasters with GDAL so in this case with https://gdal.org/drivers/raster/grib.html. I have never been playing with GRIB files and I do not know how well the format is optimized for features that are needed for making fast map services: fast access to subregions and overview levels. I have no idea how well GRIB files are indexed for random access, it there is difference between png and JPEG2000 encoding, and how well the GDAL GRIB driver can utilize the possibilities that the format offer.

The most common raster format that Mapserver services are using is certainly GeoTIFF and when they are tiled, compressed, and have overviews they work pretty well. It is certainly an advantage if you can use the GRIB files directly without conversions. I see that you are using Mapcache in front of Mapserver and it will help a lot if you pre-seed the tiles. If you do not pre-seed then each time when the user is the first visitor on some site the bottle neck will be Mapserver.

The Ls/Hs idea is interesting. I do not believe there is anything ready available but I know that my colleagues at the Finnish Geodetic Institute have made quite nice things with Mapserver and shell scripts running some GDAL commands. Should they be computed on-the-fly for the area that is visible on the map?

I wonder if it could be possible to query the CONTOUR layer that you already has https://www.mapserver.org/input/vector/contour.html and search the min/max values from there. Or then to improve the contouring algorithm a bit because it is already processing all the pixels.

-Jukka Rahkonen-

-----Alkuperäinen viesti-----

Lähettäjä: mapserver-users <mapserver-users-bounces at lists.osgeo.org> Puolesta Stefan Gofferje
Lähetetty: tiistai 16. maaliskuuta 2021 12.11
Vastaanottaja: mapserver-users at lists.osgeo.org
Aihe: Re: [mapserver-users] Hello from Finland

Hi Jeff,

On 3/15/21 2:35 PM, Jeff McKenna wrote:
 > Thanks Stefan for this wonderful introduction, it's nice to hear of  > positive experiences to give us all more motivation to continue to  > helping to improve MapServer!  Welcome to both you and Iliya to the  > community.  Keep us posted on your activities!
 >
 > Oh, once your MapServer backend is plugged-in be sure to add your  > weather site to the MapServer gallery, so we can help new users see  > what can be done...

Thanks for your nice words! I now have mapserver and mapcache running successfully and talking to each other in my home lab. I have set up a number of test layers from different input sources pretty successfully.
The documentation and the configuration required some adaption of my thinking processes because it's pretty different from anything else I have ever done before but I love this kind of challenges!

Currently I have only two issues which prevent me from doing the swap right now. Thanks to NOAA using a 0-360° grid instead of (-180)-180°, there's some artefacts when reprojecting to epsg:3857. For raster maps this is already addressed in #6106. I ran into an issue which LINE maps with a CONTOUR source which probably is related and opened #6261.

Then I have a question... Although it's not strictly necessary, I would like to paint those Hs and Ls in my atmospheric pressure map. Is there a way to run scripts or something like that in mapserver. I could probably run some GDAL stuff externally over the grid files in my download script and create some point overlays which could be in a GROUP with the isoline layer but doing this inside of mapserver would kinda be cooler.

And a second question: How is mapserver handling raster files memory-wise? Does it try to load the whole file to the memory at once? 
My biggest issue during the whole 14 years I run this website always has been balancing preprocessing and map rendering. The more preprocessing I have to do, the longer the whole process of map generation takes but of course I couldn't afford to buy a $100k top-of-the-line server with terabytes of RAM and plenty of CPU cores for the rendering...
I'm thinking about my input file strategy at the moment...
1.) One GRIB file for everything (around 1.9GB)
2.) One GRIB file per parameter for all time steps
3.) One GRIB file per time step including all parameters
4.) One GRIB file per parameter for each time step

4. would be the most complex to configure and also would be fun (irony!) to make local forecasts from but probably take the least memory.
2. and 3. are a fairly good compromise with 2 being easier for calculating local forecasts and 3 easier for calculating additional stuff like my "severe weather warning level"
1. would be easiest for rendering and calculating but even at the moment with GrADS I every now and then run into memory issues (leaks?).

This project really turns out to be a fun challenge!
Stefan

-- 
  (o_   Stefan Gofferje            | SCLT, MCP, CCSA
  //\   Reg'd Linux User #247167   | VCP #2263
  V_/_  https://www.gofferje.net   | https://www.saakeskus.fi

_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users