[Qgis-user] Using Processing for nightly ETL jobs

[Bernhard Ströbl]

Hi Björn,

yes that works. Only challenge is that you have to tell the algorithm which geometry type the data is, whereas QGIS' native PostGIS writer does check that for itself and use an appropriate one. Being not able to tell beforehand I established a geometry filter and use the multi type in the writer.

Bernhard

> pathmapper <pathmapper at posteo.de> hat am 30.05.2024 09:37 CEST geschrieben:
> 
>  
> Hi Bernhard,
> 
> not sure about the native "Export to PostgreSQL" algorithm, but there's 
> also a GDAL alg "Export to PostgreSQL (new connection)" which allows to 
> configure the DB access. Using this one in your model should do the 
> trick..
> 
> Cheers,
> Björn
> 
> On 30.05.2024 08:47, Bernhard Ströbl via QGIS-User wrote:
> > Hello everybody,
> > 
> > I did some experiments in the last couple days and your help was very
> > valuable. I succeeded in creating two models that - when run within
> > QGIS or from the console - download data of different types and write
> > it into a PostgreSQL database. For automation purposes I want to run
> > my models on a Linux server. I am wondering how I would configure the
> > database access. The algorythm "Export to PostgreSQL" takes the
> > connection name as input. The connection name is the name of a
> > pg-connection configured using the GUI (and backed by a saved
> > authentication in this case). There will be no GUI (and probably no
> > graphic server either) on the server machine. Does anybody know if
> > creating a profile with the connection (without authentication) and
> > placing that as default profile in the home directory of the user
> > running the model will do the trick? Or would I need a different
> > approach?
> > I cannot try it out because currently I still have no access to the 
> > server.
> > 
> > Any help appreciated
> > Bernhard
> > 
> > 
> >> Germán Carrillo <carrillo.german at gmail.com> hat am 22.05.2024 23:30 
> >> CEST geschrieben:
> >> 
> >> 
> >> Hi All,
> >> 
> >> have a look at:
> >> 
> >> https://github.com/gacarrillor/AppendFeaturesToLayer
> >> 
> >> It has ready-to-use models to perform transform and upsert (update or
> >> insert) operations.
> >> 
> >> You can run it via standalone processing (Python) scripts or via QGIS
> >> Process.
> >> 
> >> Regards,
> >> 
> >> Germán
> >> 
> >> 
> >> El mié, 22 may 2024 a las 12:20, Etienne Trimaille via QGIS-User (<
> >> qgis-user at lists.osgeo.org>) escribió:
> >> 
> >> > Hi,
> >> >
> >> > https://docs.qgis.org/3.34/en/docs/user_manual/processing/standalone.html
> >> >
> >> > Le mer. 22 mai 2024 à 18:52, NOSPAM via QGIS-User <
> >> > qgis-user at lists.osgeo.org> a écrit :
> >> >
> >> >> Dear List,
> >> >>
> >> >> has anybody already used QGIS' processing framework for performing
> >> >> nightly ETL jobs?
> >> >>
> >> >> I need to have nightly jobs run that
> >> >> 1) download data (e.g. from a WFS server),
> >> >> 2) perform some tasks like renaming fields or the like
> >> >> 3) and finally load them into a PostGIS database.
> >> >>
> >> >> My idea would be to design the process using the modeller. How can I
> >> >> have the thus created model run non interactively?
> >> >>
> >> >> Any ideas are welcome
> >> >>
> >> >> Bernhard
> >> >> _______________________________________________
> >> >> QGIS-User mailing list
> >> >> QGIS-User at lists.osgeo.org
> >> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >> >>
> >> > _______________________________________________
> >> > QGIS-User mailing list
> >> > QGIS-User at lists.osgeo.org
> >> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >> >
> >> 
> >> 
> >> --
> >> -----------
> >>    |\__
> >> (:>__)(
> >>    |/
> >> Soluciones Geoinformáticas Libres
> >> http://geotux.tuxfamily.org/
> >> https://twitter.com/GeoTux2 <http://twitter.com/GeoTux2>
> > _______________________________________________
> > QGIS-User mailing list
> > QGIS-User at lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user