<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:14431917;
mso-list-type:hybrid;
mso-list-template-ids:1288083432 68419607 68419609 68419611 68419599 68419609 68419611 68419599 68419609 68419611;}
@list l0:level1
{mso-level-number-format:alpha-lower;
mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="NO-BOK" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Dear all,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">On our Ubuntu server we are about to reorganize our GIS data in order to develop a more efficient and consistent solution for data storage in a mixed GIS environment.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">By “mixed GIS environment” I mean that we have people working with GRASS, QGIS, PostGIS but also many people using R and maybe the largest fraction using ESRI products, furthermore we have people using ENIV, ERDAS and
some other. Only few people (like me) actually work directly on the server…<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Until now I stored “my” data mainly in GRASS (6/7) native format which I was very happy with. But I guess our ESRI- and PostGIS-people would not accept that as a standard…<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">However, especially for time series data we cannot have several copies in different formats (tailor-made for each and every software).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">So I started thinking: what would be the most efficient and convenient solution for storing a large amount of data (e.g. high resolution raster and vector data with national extent plus time series data) in a way that
it is accessible for all (at least most) remote users (with different GIS software). As I am very fond of the temporal framework in GRASS 7 it would be a precondition that I can use these tools on the data without unreasonable performance loss. Another precondition
would be that users at remote computers in our (MS Windows) network can have access to the data.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In general, four options come into my mind:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">a)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">Stick to GRASS native format and have one copy in another format<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">b)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">Use the native formats the data come in (e.g. temperature and precipitation comes in zipped ascii-grid format)<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">c)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">Use PostGIS as a backend for data storage (raster / vector) (linked by (r./v.external.*)<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">d)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">Use another GDAL/OGR format for data storage (raster / vector) (linked by (r./v.external.*)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">My question(s) are:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">What solutions could you recommend or what solution did you choose?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Who is having experience with this kind of data management challenge?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">How do externally linked data series perform compared to GRASS native?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I searched a bit the mailing list and found this: (<a href="http://osgeo-org.1560.x6.nabble.com/GRASS7-temporal-GIS-database-questions-td5054920.html">http://osgeo-org.1560.x6.nabble.com/GRASS7-temporal-GIS-database-questions-td5054920.html</a>)
where Sören recommended “postgresql as temporal database backend”. However I am not sure if that was meant only for the temporal metadata and not the rasters themselves…<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Furthermore in the idea collection for the Temporal framework (<a href="http://grasswiki.osgeo.org/wiki/Time_series_development">http://grasswiki.osgeo.org/wiki/Time_series_development</a>, Open issues section) limitations
were mentioned regarding the number of files in a folder, which would be possibly a problem both for file based storage. The ext2 file system had “"soft" upper limit of about 10-15k files in a single directory” but theoretically many more where possible. Other
file systems may allow for more I guess… Will usage of such big directories > 10,000 files lead to performance problems?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The “Working with external data in GRASS 7” – wiki entry (http://grasswiki.osgeo.org/wiki/Working_with_external_data_in_GRASS_7) covers the technical part (and to some degree performance issues) very well. Would it be
worth adding a part on the strategic considerations / pros and cons of using external data? Or is that too much user and format dependent?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks for any feedback our thoughts around this topic…<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Cheers<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Stefan<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>