[GeoNode-users] Migration of data from 2.0 to 2.4

Paolo Corti pcorti at gmail.com
Tue Feb 9 13:14:48 PST 2016


Yes, I think that you need to fix the first error to have the
migration working properly.
Don't worry about the second error: you are getting IntegrityError as
the records were already added to the database when you ran the script
first time.
The third error is caused by the first one.
Let me know if you get more issues
p

On Tue, Feb 9, 2016 at 3:38 PM, Mana Etches <manadotcom at gmail.com> wrote:
> Hi Paolo,
>
> Firstly, I should have ran the python scripts individually within
> migrate2.sh, I noticed some errors as follows:
>
>  'create_auth_group_and_update_res.py', reveals error:
>
> Traceback (most recent call last):
>   File "create_auth_group_and_update_res.py", line 10, in <module>
>     from geonode.groups.models import GroupProfile
>   File "/usr/local/lib/python2.7/dist-packages/geonode/groups/models.py",
> line 5, in <module>
>     from django.contrib.auth.models import Group
>   File "/usr/lib/python2.7/dist-packages/django/contrib/auth/__init__.py",
> line 6, in <module>
>     from django.middleware.csrf import rotate_token
>   File "/usr/lib/python2.7/dist-packages/django/middleware/csrf.py", line
> 14, in <module>
>     from django.utils.cache import patch_vary_headers
>   File "/usr/lib/python2.7/dist-packages/django/utils/cache.py", line 26, in
> <module>
>     from django.core.cache import get_cache
>   File "/usr/lib/python2.7/dist-packages/django/core/cache/__init__.py",
> line 69, in <module>
>     if DEFAULT_CACHE_ALIAS not in settings.CACHES:
>   File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 54,
> in __getattr__
>     self._setup(name)
>   File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 49,
> in _setup
>     self._wrapped = Settings(settings_module)
>   File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 132,
> in __init__
>     % (self.SETTINGS_MODULE, e)
> ImportError: Could not import settings '/etc/genode/local_settings.py' (Is
> it on sys.path? Is there an import error in the settings file?): Import by
> filename is not supported.
>
> The second script, 'migrate_tags.py', ran errors for the first part relative
> to psycopg2.IntegrityError on my layers, eg:
>
> ('Municipality', 'municipality')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Island) already exists.
>
> ('Island', 'island')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Atoll) already exists.
>
> ('Atoll', 'atoll')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(nr) already exists.
>
> ('nr', 'nr')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(District) already exists.
>
> ('District', 'district')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Village) already exists.
>
> ('Village', 'village')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Local Level Government) already exists.
>
> ('Local Level Government', 'local-level-government')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Building Replacement Cost) already exists.
>
> ('Building Replacement Cost', 'building-replacement-cost')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(State) already exists.
>
> ('State', 'state')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Ward) already exists.
>
> ('Ward', 'ward')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Sub District) already exists.
>
> ('Sub District', 'sub-district')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Area Council) already exists.
>
> ('Area Council', 'area-council')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(hazards) already exists.
>
> ('hazards', 'hazards')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Building Exposure Aggregates) already exists.
>
> ('Building Exposure Aggregates', 'building-exposure-aggregates')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(nrj) already exists.
>
> ('nrj', 'nrj')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Land cover) already exists.
>
> ('Land cover', 'land-cover')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Land use) already exists.
>
> ('Land use', 'land-use')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Land Cover) already exists.
>
> ('Land Cover', 'land-cover_1')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Land Use) already exists.
>
> ('Land Use', 'land-use_1')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Average Annual Loss For TC & EQ) already exists.
>
> ('Average Annual Loss For TC & EQ', 'average-annual-loss-for-tc-eq')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Average Annual Loss) already exists.
>
> ('Average Annual Loss', 'average-annual-loss')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Building Replacement Costs) already exists.
>
> ('Building Replacement Costs', 'building-replacement-costs')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Exposure Cash Crops) already exists.
>
> ('Exposure Cash Crops', 'exposure-cash-crops')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(imagery) already exists.
>
> ('imagery', 'imagery')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(REQUIRED: Common-use word or phrase used to describe
> the subject of the data set.) already exists.
>
> ('REQUIRED: Common-use word or phrase used to describe the subject of the
> data set.',
> 'required-common-use-word-or-phrase-used-to-describe-the-subject-of-the-data-set')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Average Annual Loss TC & EQ) already exists.
>
> ('Average Annual Loss TC & EQ', 'average-annual-loss-tc-eq')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Average Annual Loss for TC & EQ) already exists.
>
> ('Average Annual Loss for TC & EQ', 'average-annual-loss-for-tc-eq_1')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Population) already exists.
>
> ('Population', 'population')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Average Annual Loss tc and eq) already exists.
>
> ('Average Annual Loss tc and eq', 'average-annual-loss-tc-and-eq')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Average Annual Loss for Tropical Cyclone and
> Earthquake) already exists.
>
> ('Average Annual Loss for Tropical Cyclone and Earthquake',
> 'average-annual-loss-for-tropical-cyclone-and-earthquake')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Probabilistic) already exists.
>
> ('Probabilistic', 'probabilistic')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Earthquake) already exists.
>
> ('Earthquake', 'earthquake')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Hazard) already exists.
>
> ('Hazard', 'hazard')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(cyclone) already exists.
>
> ('cyclone', 'cyclone')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(solomons) already exists.
>
> ('solomons', 'solomons')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(risk) already exists.
>
> ('risk', 'risk')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Summary Report) already exists.
>
> ('Summary Report', 'summary-report')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Risk Assessment) already exists.
>
> ('Risk Assessment', 'risk-assessment')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Loss) already exists.
>
> ('Loss', 'loss')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Topography) already exists.
>
> ('Topography', 'topography')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Document) already exists.
>
> ('Document', 'document')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Imagery) already exists.
>
> ('Imagery', 'imagery_1')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(Soil) already exists.
>
> ('Soil', 'soil')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(SOILS) already exists.
>
> ('SOILS', 'soils')
> <class 'psycopg2.IntegrityError'>
> duplicate key value violates unique constraint "taggit_tag_name_key"
> DETAIL:  Key (name)=(SOIL) already exists.
>
> And the last python script, 'migrate_group_permissions.py', revealed:
>
> Traceback (most recent call last):
>   File "migrate_group_permissions.py", line 16, in <module>
>     authenticated_group_id = dst_cur.next()[0]
> StopIteration
>
>
> Correct me if I'm wrong, it looks like the error in the first script is
> related to my settings_path within gn_migration.cfg file. I saw within the
> forums that this should be set to 'settings_path = geonode.settings'.
>
> As for errors in scripts migrate_tags.py and migrate_group_permissions.py,
> I'm hoping to get your advice on this part.
>
> Thank you immensely,
>
> Mana
>
>
>
>
> On Wed, Feb 10, 2016 at 5:09 AM, Paolo Corti <pcorti at gmail.com> wrote:
>>
>> Hi Mana
>> did you run migrate2.sh after migrating the GeoServer data directory?
>> did you see any error running that script?
>> p
>>
>> On Tue, Feb 9, 2016 at 2:00 AM, Mana Etches <manadotcom at gmail.com> wrote:
>> > Hello geonode gurus,
>> >
>> > I've installed geonode 2.4 following instructions at
>> > http://geonode.org/blog/2015/11/19/geonode-2.4-released/
>> >
>> > I've followed Capooti's migration steps at guidelines at
>> >
>> > https://github.com/capooti/geonode/blob/migration_from20_to_24/scripts/migrations/migrate20to24/index.rst
>> >
>> > I'm at a point where I can see the number of layers, maps and users on
>> > geonode GUI (but only as a registered user). I can log into geoserver
>> > admin,
>> > however I don't see my layers, stores, on the admin panel.
>> >
>> > I uploaded 2.0 GS data directory to geoserver 2.4 data directory as
>> > follows:
>> > data/default     to    data/default
>> > gwc-layers      to     gwc/layers
>> > workspaces/geonode    to      workspaces/geonode/datastore
>> >
>> > Note, I can upload layers just fine but I can only see the the layers
>> > that I
>> > uploaded in the geonode_data in postgres. I can't see the layers from
>> > 2.0
>> > database.
>> >
>> > Lastly I ran a 'geonode updatelayers' and got the following results:
>> >
>> > Finished processing 4 layers in 2.0 seconds.
>> >
>> > 0 Created layers
>> > 4 Updated layers
>> > 0 Failed layers
>> > 0.500000 seconds per layer
>> >
>> > Updating only the layers that I uploaded.
>> >
>> >
>> >
>> > For referencing purposes see my 'gn_migration.cfg' configurations below:
>> >
>> > [db20]
>> > dbname = old_db
>> > host = 192.168.137.12
>> > user = geonode
>> > password = secret (replaced)
>> >
>> > [db24]
>> > dbname = geonode
>> > host = 192.168.137.12
>> > user = geonode
>> > password = secret (replaced)
>> >
>> > [path]
>> > geonode_path = /usr/local/lib/python2.7/dist-packages/geonode
>> > settings_path = /etc/genode/local_settings.py
>> >
>> > My gut feeling is its a configuration issue.
>> >
>> > Any help or point in the right direction would be very much appreciated
>> >
>> > Mana
>> >
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > geonode-users mailing list
>> > geonode-users at lists.osgeo.org
>> > http://lists.osgeo.org/mailman/listinfo/geonode-users
>> >
>>
>>
>>
>> --
>> Paolo Corti
>> Geospatial software developer
>> web: http://www.paolocorti.net
>> twitter: @capooti
>> skype: capooti
>
>



-- 
Paolo Corti
Geospatial software developer
web: http://www.paolocorti.net
twitter: @capooti
skype: capooti


More information about the geonode-users mailing list