[Live-demo] Provide .fix files for all Shapefiles
Stefan A. Tzeggai
tzeggai at wikisquare.de
Wed Aug 11 09:33:58 PDT 2010
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi
1. I confirm, in rc7 uDig hangs when opening any Shapefile from natural
earth. So all this is a fix and not a feature!
2. Am 11.08.2010 13:18, schrieb Cameron Shorter:
> Please go ahead and apply this patch.
I committet the patch for install_gisdata.sh
3. Jody asked:
>Can I also ask if there is a "qix" spatial index?
The script generates both .fix and .qix files.
4. Am 11.08.2010 13:18, schrieb Cameron Shorter:
> Please also run the install_gisdata.sh script, and confirm that it works
> as expected, and that the other applications like UDig work.
Due to time and troubles i was not able to test a whole build, but i did
two tests:
4.1 On my Ubuntu uDig 1.2 hangs when opening a natural_earth shapfile
from a r/o folder (error reproduced). Once i run the atlasstyler command
over the folder and chmod -w it again, uDig can load the Shapefiles.
4.2 In a virtual xubuntu 10.4 i ran install_atlasstyler.sh and
install_gisdata.sh and the resulting /usr/local/share/data directory
looked as expected.
5. Hamish wrote:
> also we worry that the /home/user dir will be kept in RAM,
> perhaps all user owned files too. (??)
When I tried to "chmod -R +w /usr/local/share/data" in the rc7, the
system was unusable afterwards and free -m used 500mb more. So the
worries are correct i guess.
Hamish wrote:
> the correct fix is not to work around the problem by polluting
> the filesystem with app-specific hacks, but rather to fix the
> app's install script.
> in this case make the data dir owned by the "users" group and
> group-writable, and add the user to the users group.
If i understand you right, than you want to only make the directoy
writable, not the files inside. That way Geotools based apps could
create new files in the directory, without all files being put into
memory. That really sounds like the cleanest solution. Sadly Jody and I
didn't come up with it earlier. But sadly my unpaid-worktime budget for
this week is running empty. ;-)
Greetings,
Steve
>
> On 11/08/10 19:24, Stefan A. Tzeggai wrote:
> Hi
>
> Jody wrote:
> JG> That is great you got a command line tool going (indeed that was
> very
> JG> quick). Updating the script (perhaps the atlas styler script) to
> JG> create the .fix files is the best option today.
>
>
> Accoring to bin/main.sh the bin/install_gisdata.sh is executed after
> bin/install_atlasstyler.sh, so the following patch is open for comments.
>
>
>
> Index: install_gisdata.sh
> ===================================================================
> --- install_gisdata.sh (revision 4603)
> +++ install_gisdata.sh (working copy)
> @@ -36,6 +36,13 @@
> exit 1
> fi
>
> +# check required tools are installed
> +if [ ! -x "`which atlasstyler`" ] ; then
> + echo "ERROR: atlasstyler is required as a tool to create .fix files
> for all shapefiles, please install it with bin/install_atlasstyler.sh
> and try again"
> + exit 1
> +fi
> +
> +
> # create tmp folders
> mkdir "$TMP"
> cd "$TMP"
> @@ -84,6 +91,13 @@
> unzip "$file" -d "$DATA_FOLDER/natural_earth"
> done
>
> +
> +# Add Geotools .fix files to all Shapefiles. Normally Geotools
> application would create these
> +# files when opeing the Shapefile, but since the data-dir is read-only,
> we do it here.
> +# This REQUIRES that install_atlasstyler.sh has been executed before
> (which is checked above)
> +find "$DATA_FOLDER/natural_earth" -iname "*.shp" -exec atlasstyler
> "addFix={}" \;
> +
> +
> chmod a+r "$DATA_FOLDER/natural_earth" ## read the data dir
> chmod 444 $DATA_FOLDER/natural_earth/* ## and all files in it
> chmod -R +X "$DATA_FOLDER/natural_earth" ## but keep x on directories
>
>
>
> Am 11.08.2010 01:47, schrieb Jody Garnett:
>
>>>> That is great you got a command line tool going (indeed that was very
>>>> quick). Updating the script (perhaps the atlas styler script) to
>>>> create the .fix files is the best option today.
>>>>
>>>> Small correction: It is especially annoying that this .fix index is
>>>> not even a spatial index - it is used to preserve row order when
>>>> editing. As a result the index files are very small.
>>>>
>>>> Can I also ask if there is a "qix" spatial index?
>>>>
>>>> (we will be getting that patch into GeoTools so the solution to
>>>> generate a fix file should be viewed as temporary and only needed to
>>>> get our live dvd release deadlines).
>>>>
>>>> Jody
>>>>
>>>> On Wed, Aug 11, 2010 at 2:38 AM, Stefan A. Tzeggai
>>>> <tzeggai at wikisquare.de> wrote:
>>>> Hi
>>>>
>>>> 1. good news: I debugged AtlasStyler and Geopublisher today a OSGeoLive
>>>> system and fixed all the problems reported.
>>>> AS and GP now have no more problems when opening a Shapefile without a
>>>> .fix spatial index from a read-only drive - it doesn't try to create the
>>>> index anymore.
>>>>
>>>> 2. The problem with read-only and .fix lies in Geotools, so it affects
>>>> uDig also.
>>>>
>>>> 3. I created a patch for Geotools
>>>> (http://jira.codehaus.org/browse/GEOT-3245) but it is not yet reviewed.
>>>>
>>>> Options are:
>>>>
>>>> A: Get the patch into Geotools and make a new release of Geotools and
>>>> uDig
>>>>
>>>> Advantage: No big changes on the OSGeo side.
>>>>
>>>> Disadvantage: Lots of work for Jody. uDig, Geopublisher, AtlasStyler and
>>>> will not profit from a spatial index when working with the demo
>>>> shapefile data.
>>>>
>>>>
>>>> B: Create .fix indexes during build after the Shapefiles are unzipped
>>>> and while the drive is still read-write.
>>>>
>>>> .fix are Geotools-specific, so ogr doesn't create these files. We need a
>>>> command line tool to add .fix indexes if we want it to be done in
>>>> 'bin/loadgisdata.sh'. Such a tool doesn't exists yet... So i added a
>>>> command line option to AtlasStyler 1.5 that does exactly that.
>>>>
>>>> So for option B, we have to do only one thing: Add a line to
>>>> 'bin/install_atlasStyler' that will add a spatial index to all
>>>> shapefiles in the $DATA_FOLDER:
>>>>
>>>>>>> find $DATA_FOLDER -iname "*.shp" -exec atlasstyler "addFix={}" \;
>>>>>>>
>>>> This REQUIRES that 'bin/loadgisdata.sh' is run before
>>>> install_AtlasStyler.sh is run.
>>>>
>>>> Actually install_Gisdata is setting the data read-only already. So in
>>>> the install_AtlasStyler.sh i will have to change the permissions twice..
>>>> something not so nice.
>>>>
>>>> Advantage: No quick re-releasing of Geotools and uDig just for one patch
>>>> two days after the last release.
>>>>
>>>> Disadvantage: Some unorthodox modifications on the install scripts.
>>>>
>>>>
>>>>
>>>> A or B ? Waiting for feedback.
>>>>
>>>> Greetings,
>>>> Steve
>>>>
>>>>
>>>> Am 10.08.2010 13:33, schrieb Cameron Shorter:
>>>>
>>>>>>> The naturalearth data shapefiles are loaded by bin/loadgisdata.sh
>>>>>>>
>>>>>>> It is loaded from the naturalearth website directly.
>>>>>>>
>>>>>>> I'm open to you adding .fix files if you think it is easy enough to
>>>>>>> achieve.
>>>>>>>
>>>>>>> On 10/08/10 21:10, Stefan A. Tzeggai wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> I have installed the rc5 to my HDD and started investigating the
>>>>>>> problems reported for AtlasStyler and Geopublisher. This is what
>>>>>>> happens:
>>>>>>>
>>>>>>> The natural earth shapes don't have .fix spatial indexes. AtlasStyler
>>>>>>> tries to create them on-the-fly and gets a permission exception.
>>>>>>>
>>>>>>> I will write a fix, so that that will not be a problem anymore, but
>>>>>>> maybe we should anyways add .fix files to all Shapefiles on the DVD.
>>>>>>>
>>>>>>> .fix files are supported by Geotools, so uDig, Geoserver,
>>>>>>> Geopublisher and AtlasStyler (and probably more) would profit
>>>>>>> performance-wise.
>>>>>>>
>>>>>>> The data dir is not in the svn.. Tell me where it is, and i can
>>>>>>> provide
>>>>>>> .fix files for all shapefiles.
>>>>>>>
>>>>>>> Greetings,
>>>>>>> Steve
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- wiki² - Softwareentwicklung
>>>>>>> Stefan A. Tzeggai, geb. Krüger
>>>>>>> Straßburger Weg 26
>>>>>>> 53113 Bonn
>>>>>>>
>>>>>>> email tzeggai at wikisquare.de
>>>>>>> phone 0228 24 000 528
>>>>>>> mobile 0176 40 38 9559
>>>>>>> webpage wikisquare.de
>>>>>>> twitter http://twitter.com/geopublishing
>>>>>>> skype alfonx
>>>>>>>
>>>>>>> reclaim your net - http://tor.eff.org
>>>>>>> enforce privacy - http://www.pgpi.org
>>>>>>> pgp key id: 51B576FD - http://pgp.mit.edu
>>>>>>>
>>>>>>> Please note that according to the German law on data retention,
>>>>>>> information on every electronic information exchange with me is
>>>>>>> retained for a period of six months.
>>>>>>>
>>>> _______________________________________________
>>>> Live-demo mailing list
>>>> Live-demo at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/live-demo
>>>> http://wiki.osgeo.org/wiki/Live_GIS_Disc
>>>>
>>>>
>>>>>
> -- wiki² - Softwareentwicklung
> Stefan A. Tzeggai, geb. Krüger
> Straßburger Weg 26
> 53113 Bonn
>
> email tzeggai at wikisquare.de
> phone 0228 24 000 528
> mobile 0176 40 38 9559
> webpage wikisquare.de
> twitter http://twitter.com/geopublishing
> skype alfonx
>
> reclaim your net - http://tor.eff.org
> enforce privacy - http://www.pgpi.org
> pgp key id: 51B576FD - http://pgp.mit.edu
>
> Please note that according to the German law on data retention,
> information on every electronic information exchange with me is
> retained for a period of six months.
_______________________________________________
Live-demo mailing list
Live-demo at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/live-demo
http://wiki.osgeo.org/wiki/Live_GIS_Disc
- --
wiki² - Softwareentwicklung
Stefan A. Tzeggai, geb. Krüger
Straßburger Weg 26
53113 Bonn
email tzeggai at wikisquare.de
phone 0228 24 000 528
mobile 0176 40 38 9559
webpage wikisquare.de
twitter http://twitter.com/geopublishing
skype alfonx
reclaim your net - http://tor.eff.org
enforce privacy - http://www.pgpi.org
pgp key id: 51B576FD - http://pgp.mit.edu
Please note that according to the German law on data retention,
information on every electronic information exchange with me is
retained for a period of six months.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkxi0PYACgkQdFDQR1G1dv1qlQCeI6yqKmv+tu8g2otq88U34D93
+pwAnR4LEBfIQDaOV94ycFgGGBNH9oKa
=pRJi
-----END PGP SIGNATURE-----
More information about the Osgeolive
mailing list