No subject
Fri Feb 8 15:06:04 EST 2008
de our data via WCS as well as WMS and WFS - but that will come down to whe=
ther our users' clients (ArcExplorer, ArcMap, and similar) are able to make=
use of it.
=20
=20
Thanks again for the info - looks like I've got some thinking to do!
=20
=20
Cheers,
=20
=20
Bonnie
=20
__________________________________
Bonnie Stewart
Research Officer
Satellite Remote Sensing Services
Information Access Division
Department of Land Information
65 Brockway Road, FLOREAT WA 6014
Ph. (08) 9387 0342 | Fx. (08) 9383 7142
bonnie.stewart at dli.wa.gov.au
www.dli.wa.gov.au
________________________________
From: Brent Fraser [mailto:bfraser at geoanalytic.com]
Sent: Mon 11/07/2005 11:09 PM
To: Bonnie Stewart; MAPSERVER-USERS at LISTS.UMN.EDU
Subject: [Maybe SPAM] Re: Organizing large temporal datasets (WAS: Increasi=
ng Maxlayers limit - anyone have spare Windows binaries? :))
Bonnie,
You may want to look into the TILEINDEX capability of MapServer. A
tileindex is a vector datasource (shapefile, PostGIS database, etc) of
outlines of other data, either raster or vector. Typically it's used to
show and entire county of mosaicked and clipped airphotos or Census
planimetric mapping vectors. In those cases MapServer does a spatial query
to select the data files to read for rendering into the output graphic file.
A tileindex could be used to represent data divided by time (as well as
geography) such as satellite imagery. A recent enhancement (v4.?) allows
the TILEINDEX to be a separate layer form the underlying data, and because
of that can handle a user-specified query on attributes stored in the
tileindex (e.g. satellite id, acquisition time, etc).
So one possibility might be to have two layers (one for the data and one
for the tileindex) per each of your 10-15 products. Then query the
tileindex based on the product name/id and the date range. I'm not sure how
to get the query from the WMS client to the server, so you may have to look
into that...
Since your data is relatively dynamic, you may want to use a PostGIS
database as the format for your tileindex to avoid any file locking problems
with shapefiles.
Brent Fraser
----- Original Message -----
From: Bonnie Stewart
To: MAPSERVER-USERS at LISTS.UMN.EDU
Sent: Monday, July 11, 2005 12:05 AM
Subject: [UMN_MAPSERVER-USERS] Increasing Maxlayers limit - anyone have
spare Windows binaries? :)
Hi all,
I'm working on a way to make a large amount of our vector and raster data
available via WMS and WFS, and I've come up against the MAXLAYERS setting
(200 in our install I think).
I'm already trying to get around the need for having so many layers in a
single mapfile by reducing the granularity available as you get to mapfiles
which cover a larger time period - let me see if I can explain what I mean
without rambling on too much.
At the lowest level of granularity, we're providing access to raster and
vector files which are derived from satellite passes - and there may be
10-40 passes per day, depending on the product. I've already set up a
process to automatically generate a mapfile containing the last 5 days worth
of data, using the GROUP parameter to group them according to the day. This
is jolly nifty for those customers who need to look at the recent info on a
pass by pass basis.
However, if I try to give people access to the entire archive (several years
worth, and 10-15 different products, at 10-40 passes per day) on a pass by
pass basis, the number of layers involved will be horrendous, so I'm trying
to reduce the granularity at which archival data is available.
My current plan is to (programmatically) create a series of weekly mapfiles,
structured in the same way as mentioned above, with individual satellite
passes grouped by day. I would have 52 of those for each year, and then to
wrap them up so that people could have an overview of what's available, I'm
planning to create some mapfiles (either yearly or monthly, maybe both),
which connect to these weekly mapfiles via WMS, and pull in each daily group
from the 52 weekly mapfiles, and define them as a single layer.
So I think I would end up with a mapfile for each year, with a group for
each week, and under each group, 7 individual layers - one for each day.
Hopefully then there are two options:
1) People's clients are able to pull apart the vector data, and maybe do a
query (for instance) to only display the lightning data from one particular
satellite pass - although having used ArcMap and ArcExplorer for testing I'm
not too hopeful about this;
2) Specific file naming conventions could allow users to find the week
they're interested in in the yearly mapfile, and then work out what the
mapfile name/URL for that week's lower level mapfile would be.
So, now I'm finally going to get to my point - even using this structure, a
MAXLAYERS setting of 200 is too low for me - I can do about 5 or 6 days of
satellite passes in 200 layers, but to get to a whole week I would need to
increase the MAXLAYERS parameter. To have a layer for each day in my yearly
mapfile, I'd need a MAXLAYERS of around 370.
I gather it's relatively easy to change the parameter and recompile
Mapserver, but as I have very little experience in this area (and I've seen
so many emails on this list seeking help with compiling mapserver!) I was
hoping somebody might possibly have already done this, and might have
Windows binaries compiled with a higher MAXLAYERS that they would be willing
to share.
Alternatively, if the structure I described above sounds silly or you can
think of something more effective, please let me know :) We do have another
approach in mind for the vector data, which is to produce an ongoing
concatenated shapefile which contains all the features for that product (for
instance, as a new pass's worth of lightning data becomes available, it is
automatically added in to one big Lightning shapefile), and then to use
classes to colour them up a bit according to the time that each event
occurred. It's a nice theory for the vector info, but I'd still need to do
something about the raster info.
Any advice (or binaries!) would be very much appreciated :)
Thanks,
Bonnie
__________________________________
Bonnie Stewart
Research Officer
Satellite Remote Sensing Services
Information Access Division
Department of Land Information
65 Brockway Road, FLOREAT WA 6014
Ph. (08) 9387 0342 | Fx. (08) 9383 7142
bonnie.stewart at dli.wa.gov.au
www.dli.wa.gov.au
This e-mail and any files transmitted with it are intended only for the use=
of the addressee(s). It may contain information that is confidential and =
privileged. If you are not an intended recipient, any use, interference wi=
th, disclosure, distribution or copying of this material is unauthorised an=
d prohibited. If you receive this in error, please notify the author by Ret=
urn email to the sender. Information in this message not relating to the of=
ficial business of DLI shall be understood as neither given nor endorsed by=
it. While every care is taken, it is recommended that you scan any attach=
ments for viruses. DLI liability is limited to re-supplying affected attach=
ments.
------_=_NextPart_001_01C5868E.29CF746E
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; charset=3Diso-8859-=
1">=0A<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">=0A<HTML>=0A<HEAD>=
=0A=0A<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version 6.5.72=
26.0">=0A<TITLE>[Maybe SPAM] Re: Organizing large temporal datasets (WAS: I=
ncreasing Maxlayers limit - anyone have spare Windows binaries? :))</TITLE>=
=0A</HEAD>=0A<BODY>=0A<DIV id=3DidOWAReplyText44821 dir=3Dltr>=0A<DIV dir=
=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Hi Brent,</FONT></DIV>=
=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A<DIV di=
r=3Dltr><FONT face=3DArial size=3D2>Thanks! I had no idea that TILEINDEXing=
=0Acould be used for time-variant data as well as spatial-variant =0Adata.=
</FONT></DIV>=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT> </D=
IV>=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2>I've been doing some googl=
ing trying to =0Afind out more information on this (no experience whatsoeve=
r with TILEINDEX thus =0Afar), and this is the best I've come up with:</FON=
T></DIV>=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2><A =0Ahref=3D"http://=
ms.gis.umn.edu/docs/howto/wcs_server">http://ms.gis.umn.edu/docs/howto/wcs_=
server</A></FONT></DIV>=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2><A =0A=
href=3D"http://mapserver.gis.umn.edu/cgi-bin/wiki.pl?WCSMapServer">http://m=
apserver.gis.umn.edu/cgi-bin/wiki.pl?WCSMapServer</A></FONT></DIV>=0A<DIV d=
ir=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A<DIV dir=3Dltr><=
FONT face=3DArial size=3D2>The section which covers "Spatio/Temporal =0AInd=
exes" in the first document is I believe the methodology you're referring t=
o, =0Aand while it's very helpful, I still have a few questions.</FONT></DI=
V>=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A<DIV =
dir=3Dltr><FONT face=3DArial size=3D2>1) So far I have only read about temp=
oral =0Atimeindexing with respect to serving WCS (Web Coverage Service) rat=
her than WMS =0Aor WFS. Is it possible to use the same methodology but serv=
e WMS or =0AWFS layers? I don't know much about WCS and I'm not sure w=
hat clients (off =0Athe shelf - we are not going to be developing any clien=
t interface for our =0Ausers) can access it.</FONT></DIV>=0A<DIV dir=3Dltr>=
<FONT face=3DArial size=3D2></FONT> </DIV>=0A<DIV dir=3Dltr><FONT face=
=3DArial color=3D#000000 size=3D2>2) Does anybody know of any =0Aadditional=
documentation/tutorials/examples using TILEINDEX for raster and =0Avector =
data which is spatially nonvariant and temporally variant? =0A:)</FONT></DI=
V>=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A<DIV =
dir=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A<DIV dir=3Dltr>=
<FONT face=3DArial size=3D2>From what I've seen of WCS, it's very =0Anifty,=
and I hope to be able to provide our data via WCS as well as WMS and WFS =
=0A- but that will come down to whether our users' clients (ArcExplorer, Ar=
cMap, =0Aand similar) are able to make use of it.</FONT></DIV>=0A<DIV dir=
=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A<DIV dir=3Dltr><FO=
NT face=3DArial size=3D2></FONT> </DIV>=0A<DIV dir=3Dltr><FONT face=3D=
Arial size=3D2>Thanks again for the info - looks like =0AI've got some=
thinking to do!</FONT></DIV>=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2>=
</FONT> </DIV>=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nb=
sp;</DIV>=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2>Cheers,</FONT></DIV>=
=0A<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A<DIV di=
r=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A<DIV dir=3Dltr><F=
ONT face=3DArial size=3D2>Bonnie</FONT></DIV>=0A<DIV dir=3Dltr><FONT face=
=3DArial size=3D2></FONT> </DIV></DIV>=0A<DIV id=3DidSignature70391 di=
r=3Dltr>=0A<DIV><FONT face=3DArial color=3D#000000 size=3D2><FONT face=3DHe=
lv size=3D2>=0A<DIV>=0A<DIV><FONT face=3DArial color=3D#000000 size=3D2><FO=
NT face=3DHelv color=3D#000000 size=3D2>=0A<DIV><FONT face=3DArial color=3D=
gray size=3D2><SPAN lang=3DEN-AU =0Astyle=3D"FONT-SIZE: 11pt; COLOR: gray; =
FONT-FAMILY: Arial">__________________________________</SPAN></FONT></DIV>=
=0A<DIV>=0A<DIV class=3DMsoNormal><FONT face=3DArial color=3Dgray size=3D2>=
<SPAN lang=3DEN-AU =0Astyle=3D"FONT-SIZE: 11pt; COLOR: gray; FONT-FAMILY: A=
rial"></SPAN></FONT><FONT =0Aface=3DArial color=3Dgray size=3D2><SPAN lang=
=3DEN-AU =0Astyle=3D"FONT-SIZE: 11pt; COLOR: gray; FONT-FAMILY: Arial">Bonn=
ie =0AStewart</SPAN></FONT></DIV>=0A<DIV class=3DMsoNormal><FONT face=3DAri=
al color=3Dgray size=3D2><SPAN lang=3DEN-AU =0Astyle=3D"FONT-SIZE: 11pt; CO=
LOR: gray; FONT-FAMILY: Arial">Research =0AOfficer<BR></SPAN></FONT><FONT f=
ace=3DArial color=3Dgray size=3D2><SPAN lang=3DEN-AU =0Astyle=3D"FONT-SIZE:=
11pt; COLOR: gray; FONT-FAMILY: Arial">Satellite Remote =0ASensing Service=
s<BR></SPAN></FONT><FONT face=3DArial color=3Dgray size=3D2><SPAN =0Alang=
=3DEN-AU style=3D"FONT-SIZE: 11pt; COLOR: gray; FONT-FAMILY: Arial">Informa=
tion =0AAccess Division<BR></SPAN></FONT><FONT face=3DArial color=3Dgray si=
ze=3D2><SPAN =0Alang=3DEN-AU style=3D"FONT-SIZE: 11pt; COLOR: gray; FONT-FA=
MILY: Arial">Department =0Aof Land Information<BR></SPAN></FONT><FONT face=
=3DArial color=3Dgray size=3D2><SPAN =0Alang=3DEN-AU style=3D"FONT-SIZE: 11=
pt; COLOR: gray; FONT-FAMILY: Arial">65 Brockway =0ARoad, FLOREAT WA </SPAN=
></FONT><FONT face=3DArial color=3Dgray size=3D2><SPAN =0Alang=3DEN-AU =0As=
tyle=3D"FONT-SIZE: 11pt; COLOR: gray; FONT-FAMILY: Arial">6014<BR></SPAN></=
FONT><FONT =0Aface=3DArial color=3Dgray size=3D2><SPAN lang=3DEN-AU =0Astyl=
e=3D"FONT-SIZE: 11pt; COLOR: gray; FONT-FAMILY: Arial">Ph. (08) 9387 0342 =
=0A| Fx. (08) 9383 7142<BR></SPAN></FONT><FONT face=3DArial color=3Dgr=
ay =0Asize=3D2><SPAN lang=3DEN-AU =0Astyle=3D"FONT-SIZE: 11pt; COLOR: gray;=
FONT-FAMILY: Arial"><A =0Ahref=3D"mailto:bonnie.stewart at dli.wa.gov.au">bon=
nie.stewart at dli.wa.gov.au</A><BR></SPAN></FONT><FONT =0Aface=3DArial color=
=3Dgray size=3D2><SPAN lang=3DEN-AU =0Astyle=3D"FONT-SIZE: 11pt; COLOR: gra=
y; FONT-FAMILY: Arial"><A =0Ahref=3D"http://www.dli.wa.gov.au">www.dli.wa.g=
ov.au</A></SPAN></FONT></DIV></DIV></FONT></FONT></DIV></DIV></FONT></FONT>=
</DIV></DIV>=0A<DIV dir=3Dltr><BR>=0A<HR tabIndex=3D-1>=0A<FONT face=3DTaho=
ma size=3D2><B>From:</B> Brent Fraser =0A[mailto:bfraser at geoanalytic.com]<B=
R><B>Sent:</B> Mon 11/07/2005 11:09 =0APM<BR><B>To:</B> Bonnie Stewart; =0A=
MAPSERVER-USERS at LISTS.UMN.EDU<BR><B>Subject:</B> [Maybe SPAM] Re: Organizin=
g =0Alarge temporal datasets (WAS: Increasing Maxlayers limit - anyone have=
spare =0AWindows binaries? :))<BR></FONT><BR></DIV>=0A<DIV>=0A<P><FONT siz=
e=3D2>Bonnie,<BR><BR> You may want to look into the TILEINDEX =0Acapa=
bility of MapServer. A<BR>tileindex is a vector datasource (shapefile=
, =0APostGIS database, etc) of<BR>outlines of other data, either raster or =
=0Avector. Typically it's used to<BR>show and entire county of mosaic=
ked and =0Aclipped airphotos or Census<BR>planimetric mapping vectors. =
; In those cases =0AMapServer does a spatial query<BR>to select the data fi=
les to read for rendering =0Ainto the output graphic file.<BR>A tileindex c=
ould be used to represent data =0Adivided by time (as well as<BR>geography)=
such as satellite imagery. A =0Arecent enhancement (v4.?) allows<BR>=
the TILEINDEX to be a separate layer form =0Athe underlying data, and becau=
se<BR>of that can handle a user-specified query on =0Aattributes stored in =
the<BR>tileindex (e.g. satellite id, acquisition time, =0Aetc).<BR><BR>&nbs=
p; So one possibility might be to have two layers (one for the =0Adata and =
one<BR>for the tileindex) per each of your 10-15 products. Then =0Aqu=
ery the<BR>tileindex based on the product name/id and the date range. =
=0AI'm not sure how<BR>to get the query from the WMS client to the server,=
so you =0Amay have to look<BR>into that...<BR><BR> Since your data i=
s relatively =0Adynamic, you may want to use a PostGIS<BR>database as the f=
ormat for your =0Atileindex to avoid any file locking problems<BR>with shap=
efiles.<BR><BR>Brent =0AFraser<BR><BR><BR>----- Original Message -----<BR>F=
rom: Bonnie Stewart<BR>To: =0AMAPSERVER-USERS at LISTS.UMN.EDU<BR>Sent: Monday=
, July 11, 2005 12:05 =0AAM<BR>Subject: [UMN_MAPSERVER-USERS] Increasing Ma=
xlayers limit - anyone =0Ahave<BR>spare Windows binaries? :)<BR><BR><BR>Hi =
all,<BR><BR>I'm working on a =0Away to make a large amount of our vector an=
d raster data<BR>available via WMS =0Aand WFS, and I've come up against the=
MAXLAYERS setting<BR>(200 in our install I =0Athink).<BR><BR>I'm already t=
rying to get around the need for having so many =0Alayers in a<BR>single ma=
pfile by reducing the granularity available as you get =0Ato mapfiles<BR>wh=
ich cover a larger time period - let me see if I can explain =0Awhat I mean=
<BR>without rambling on too much.<BR><BR>At the lowest level of =0Agranular=
ity, we're providing access to raster and<BR>vector files which are =0Aderi=
ved from satellite passes - and there may be<BR>10-40 passes per day, =0Ade=
pending on the product. I've already set up a<BR>process to automatically =
=0Agenerate a mapfile containing the last 5 days worth<BR>of data, using th=
e GROUP =0Aparameter to group them according to the day. This<BR>is jolly n=
ifty for those =0Acustomers who need to look at the recent info on a<BR>pas=
s by pass =0Abasis.<BR><BR>However, if I try to give people access to the e=
ntire archive =0A(several years<BR>worth, and 10-15 different products, at =
10-40 passes per day) =0Aon a pass by<BR>pass basis, the number of layers i=
nvolved will be horrendous, so =0AI'm trying<BR>to reduce the granularity a=
t which archival data is =0Aavailable.<BR><BR>My current plan is to (progra=
mmatically) create a series of =0Aweekly mapfiles,<BR>structured in the sam=
e way as mentioned above, with =0Aindividual satellite<BR>passes grouped by=
day. I would have 52 of those for each =0Ayear, and then to<BR>wrap them u=
p so that people could have an overview of =0Awhat's available, I'm<BR>plan=
ning to create some mapfiles (either yearly or =0Amonthly, maybe both),<BR>=
which connect to these weekly mapfiles via WMS, and =0Apull in each daily g=
roup<BR>from the 52 weekly mapfiles, and define them as a =0Asingle layer.<=
BR><BR>So I think I would end up with a mapfile for each year, =0Awith a gr=
oup for<BR>each week, and under each group, 7 individual layers - one =0Afo=
r each day.<BR>Hopefully then there are two options:<BR>1) People's clients=
=0Aare able to pull apart the vector data, and maybe do a<BR>query (for in=
stance) =0Ato only display the lightning data from one particular<BR>satell=
ite pass - =0Aalthough having used ArcMap and ArcExplorer for testing I'm<B=
R>not too hopeful =0Aabout this;<BR>2) Specific file naming conventions cou=
ld allow users to find the =0Aweek<BR>they're interested in in the yearly m=
apfile, and then work out what =0Athe<BR>mapfile name/URL for that week's l=
ower level mapfile would be.<BR><BR>So, =0Anow I'm finally going to get to =
my point - even using this structure, =0Aa<BR>MAXLAYERS setting of 200 is t=
oo low for me - I can do about 5 or 6 days =0Aof<BR>satellite passes in 200=
layers, but to get to a whole week I would need =0Ato<BR>increase the MAXL=
AYERS parameter. To have a layer for each day in my =0Ayearly<BR>mapfile, I=
'd need a MAXLAYERS of around 370.<BR><BR>I gather it's =0Arelatively easy =
to change the parameter and recompile<BR>Mapserver, but as I =0Ahave very l=
ittle experience in this area (and I've seen<BR>so many emails on =0Athis l=
ist seeking help with compiling mapserver!) I was<BR>hoping somebody might =
=0Apossibly have already done this, and might have<BR>Windows binaries comp=
iled =0Awith a higher MAXLAYERS that they would be willing<BR>to =0Ashare.<=
BR><BR>Alternatively, if the structure I described above sounds silly or =
=0Ayou can<BR>think of something more effective, please let me know :) We d=
o have =0Aanother<BR>approach in mind for the vector data, which is to prod=
uce an =0Aongoing<BR>concatenated shapefile which contains all the features=
for that =0Aproduct (for<BR>instance, as a new pass's worth of lightning d=
ata becomes =0Aavailable, it is<BR>automatically added in to one big Lightn=
ing shapefile), and =0Athen to use<BR>classes to colour them up a bit accor=
ding to the time that each =0Aevent<BR>occurred. It's a nice theory for the=
vector info, but I'd still need to =0Ado<BR>something about the raster inf=
o.<BR><BR><BR>Any advice (or binaries!) =0Awould be very much appreciated =
=0A:)<BR><BR><BR>Thanks,<BR><BR><BR>Bonnie<BR><BR>_________________________=
_________<BR>Bonnie =0AStewart<BR>Research Officer<BR>Satellite Remote Sens=
ing Services<BR>Information =0AAccess Division<BR>Department of Land Inform=
ation<BR>65 Brockway Road, FLOREAT =0AWA 6014<BR>Ph. (08) 9387 0342 | Fx. (=
08) 9383 =0A7142<BR>bonnie.stewart at dli.wa.gov.au<BR>www.dli.wa.gov.au<BR><B=
R></FONT></P></DIV>=0A=0A<FONT SIZE=3D3><BR>
<BR>
This e-mail and any files transmitted with it are intended only for the use=
of the addressee(s). It may contain information that is confidential and =
privileged. If you are not an intended recipient, any use, interference wi=
th, disclosure, distribution or copying of this material is unauthorised an=
d prohibited. If you receive this in error, please notify the author by Ret=
urn email to the sender. Information in this message not relating to the of=
ficial business of DLI shall be understood as neither given nor endorsed by=
it. While every care is taken, it is recommended that you scan any attach=
ments for viruses. DLI liability is limited to re-supplying affected attach=
ments.<BR>
</FONT>
</BODY>=0A</HTML>
------_=_NextPart_001_01C5868E.29CF746E--
More information about the mapserver-users
mailing list