SpatiaLite: WFS

Loading geospatial data from a WFS datasource

Starting since version 4.1.0 SpatiaLite supports the capability to load geospatial data form an external WFS datasource.
Just a very short introduction: the Web Feature Service (aka WFS) is a well known and widely used standard WEB protocol allowing to request geographic features from a remote WEB server just specifying an appropriate request URL.

The payload returned by a WFS request usually consist in some XML document.
The specific content of such XML depends on the nature of the request; the most commonly used WFS requests are:
- GetCapabilites: will return many useful informations describing in a very detailed way any other supported request. This including a full list of all Feature Types (i.e. geographic layers) made available on that server.
- DescribeFeatureType: will return a detailed Data Schema describing each Feature Type (aka layer).
- GetFeature: will return a list of Features, each one of them formatted accordingly to the corresponding declared Data Schema.
  Usually each Feature will include a GML Geometry.
The WFS protocol is a quickly evolving one, and has many different flavors; the main versions supported nowadays are the rather obsolescent 1.0.0, the widespread 1.1.0 and the recently introduced and not yet widely diffused 2.0.0

WFS is usually associated with interactive WEB mapping applications, but nothing forbids using WFS as a general purpose protocol supporting neutral data exchange in vendor-agnostic fashion.
What SpatiaLite does is simply accessing an external WFS datasource and then permanently storing the received data into a DB table. There is absolutely nothing odd in this; as you probably already know SpatiaLite supports importing data e.g. from a Shapefile. Accessing a WFS datasource is conceptually exactly the same, except that in this case the input datasource isn't a local one (file-system based) but is a remote one accessed via an Internet connection.

Useful resources for testing purposes

There are lots of WFS servers at the four corners of the world; anyway, the following ones are few URLs you can use for your first tests.

http://webgis.regione.sardegna.it/geoserver/ows?service=WFS&request=GetCapabilities

http://geomap.reteunitaria.piemonte.it/ws/gsareprot/rp-01/areeprotwfs/wfs_gsareprot_1?service=WFS&request=getCapabilities 

http://demo.opengeo.org/geoserver/wfs?service=wfs&version=1.1.0&request=getCapabilities

http://mrdata.usgs.gov/services/mt?request=getcapabilities&service=WFS&version=1.0.0&

http://mrdata.usgs.gov/services/tx?request=getcapabilities&service=WFS&version=1.0.0&

http://mrdata.usgs.gov/services/mrds?request=getcapabilities&service=WFS&version=1.0.0

http://sdi.geoportal.gov.pl/wfs_prg/wfservice.aspx?REQUEST=GetCapabilities&SERVICE=WFS&VERSION=1.1.0

Loading WFS data on spatialite_gui

You can start a WFS download session indifferently from the main menu or by pressing the corresponding toolbar button.

A dialog panel will appear; now you simply have to type (or possibly paste) the URL identifying the GetCapabities WFS service you are intending to access.
Then press the Load button.

After a little while a full list of available Feature Types (aka layers) will be shown; select one item (just click on the corresponding row).
Then fill-in the request arguments (at least the name of the target DB table is required).
And finally press the Load Data button.

The download will immediately start.
Please note: a long time could be required, depending on data size, available bandwidth and so on .... patiently wait until completion.

At the end of the data import process a diagnostic message will appear.

In this case all selected data have been successfully downloaded and permanently saved into your local DB.

Hints and tricks #1

Sometimes the Feature Types returned by a WFS datasource could contain many tenth (or even many hundredth) different layers.

In this case you can usefully apply a filter based on some specific Keyword (you can select some appropriate value from the corresponding ComboBox list if you haven't any idea about the declared Keywords).

Hints and tricks #2

You can eventually specify some Feature Attribute to be used as the Primary Key supporting the target DB table (should contain unique values).
You can select a preferred WFS protocol version; and you can choose between a monolithic download or a WFS paged download (not all servers support this option; please see below).
Sometimes the WFS server supports more than a single SRID; then you can select the most appropriate.
Other times it could be necessary swapping the axes for every GML Geometry (the WFS/GML standards are rather confused about expected axes ordering).

Horror Stories

The left figure (greenish) represents USA familiar shape; the right figure (yellowish) is exactly the same but presenting badly swapped axes.

Anyway don't despair; if you've incurred in such a mishap a single SQL query is all you need to definitely resolve your troubles:

UPDATE my_table SET geom = SwapCoords(geom);

Advanced WFS options

The WFS protocol supports many interesting advanced options.
Sometimes you could probably not be interested in downloading a whole dataset (may be, a really huge one).
You'll wish better downloading a very specific portion of the dataset, accurately selected accordingly to your specific requirements.
In this case you can eventually specify a more complex WFS request URL, so to specify some appropriate filter.
So you can eventually specify a filter based on some attribute value: e.g. you can specify a rule dictating that fips_cod='fIT', so to actually download from the worldwide dataset only the few features corresponding to Italy.
```
Filter=<Filter><PropertyIsEqualTo>
  <PropertyName>fips_code</PropertyName>
  <Literal>fIT</Literal>
</PropertyIsEqualTo></Filter>
```
Alternatively, you could specify a spatial filter based on some bounding box, e.g.
```
BBOX=10,40,15,45
```

Monolithic WFS vs WFS paging

A really interesting feature introduced by the most recent WFS 2.0.0 is the ability to support paged WFS requests; using this option you can download a whole layer by repeatedly querying just a reasonable number of Features at each time instead of requesting all the Features in single (possibly huge) monolithic block.
This one is an highly attractive option when accessing a WFS datasource:

transferring many small packages has a lower bandwidth impact.
parsing many small XML Documents has a lower impact on system resource.
last but not least: not rarely WFS servers apply a silent truncation: after transferring a predetermined number of Features the request will be considered terminated; all subsequent Features will not be sent at all.
Such a behavior is obviously intended to shield the WFS server from harmful (and possibly malicious) exceeding requests; but in many cases this will forbid to effectively download a whole layer. Using WFS paging will effectively circumvent this specific issue.

Please note: both MapServer and GeoServer (two open source WFS server implementations) do effectively support WFS paging even for WFS 1.0.0 and WFS 1.1.0 on their most recent versions. Both products are widely used, so after all is not at all difficult to encounter some WFS server effectively supporting WFS paging.

SpatiaLite will always silently check if the target server effectively supports WFS Paging; if not, an error will be returned if Paging was explicitly specified in the user request.

Loading WFS data on spatialite (command line tool)

not really different from what supported by the GUI tool (not surprisingly: the underlaying code is exactly the same for both).
the main difference is in that the CLI tool absolutely requires a well-formed (hand written) GetFeature request URL.
Anyway this offers a more flexible approach, because this way you can take full profit by the many possible options supported by WFS (e.g. filtering features by BBOX or by some attribute value and so on).

.loadwfs WFS_path_or_URL layer_name table_name [pk_column] [swap] [page_size] [with_spatial_index]

As a minimum you are required to invoke .loadwfs by specifying all mandatory arguments; you can optionally set all the other optional arguments (respecting their expected relative order).

.loadwfs http://webgis.regione.sardegna.it/geoserver/ows?service=WFS&request=GetFeature&typeName=dbu:ASL \
    dbu:ASL asl_sardegna

The above minimal request will attempt to contact the specified URL and then download all Features from the dbu:ASL WFS Feature Type aka layer.
All downloaded features will then be inserted into a DB table named asl_sardegna.

.loadwfs http://webgis.regione.sardegna.it/geoserver/ows?service=WFS&request=GetFeature&typeName=dbu:DBTCOMUNE \
    dbu:DBTCOMUNE comuni_sardegna DBTCODICEISTAT no 15 yes

This second request is fully qualified: the selected WFS Feature Type is dbu:DBTCOMUNE and the target DB table is comuni_sardegna.
This table will support a Primary Key named DBTCODICEISTAT, corresponding to the WFS attribute of the same name.
Axes are expected to already be in the correct order (no | swap | swap_axes).
In this case WFS Paging is requested by specifying a page size of 15 features for each page (a negative or zero size means disabling WFS Paging at all).
And finally (yes) the creation of a Spatial Index supporting the DB table is requested.

back