View Ticket
Not logged in
Ticket Hash: 5e5904b9e0ae86ece3e9ac31eb52d39b482b8fcf
Title: DiscardGeometryColumn() leaves orphan rows in Spatial MetaData tables
Status: Closed Type: Code_Defect
Severity: Important Priority: Immediate
Subsystem: Resolution: Overcome_By_Events
Last Modified: 2020-08-26 07:18:54
Version Found In: 4.3.0a, 4.4.0-RC1
User Comments:
anonymous added on 2017-11-21 19:19:50:
DiscardGeometryColumn() doesn't remove the column from geometry_columns_auth, geometry_columns_statistics, geometry_columns_time tables.

sqlite> SELECT load_extension('mod_spatialite');

sqlite> CREATE TABLE "test_app_city" ("id" PRIMARY KEY);
sqlite> SELECT AddGeometryColumn('test_app_city', 'point', 4326, 'POINT');
1
sqlite> SELECT DiscardGeometryColumn('test_app_city', 'point');
1
sqlite> SELECT * FROM geometry_columns_auth WHERE f_table_name='test_app_city';
test_app_city|point|0|0

mj10777 added on 2017-11-22 01:44:37:

Neither for the present developer, nor version 4.3.0 can this be confirmed.

CREATE TABLE "test_app_city" ("id" PRIMARY KEY);
SELECT AddGeometryColumn('test_app_city', 'point', 4326, 'POINT');
INSERT INTO test_app_city SELECT 1, MakePoint(1,2,4326);
SELECT UpdateLayerStatistics();
SELECT "Rows count of geometry_columns:";
SELECT count(*) FROM "geometry_columns";
SELECT * FROM "vector_layers_statistics";
SELECT "DiscardGeometryColumn";
SELECT DiscardGeometryColumn('test_app_city', 'point');

SELECT "Rows count of geometry_columns:";
SELECT count(*) FROM "geometry_columns";
SELECT "Rows count of geometry_columns_auth:";
SELECT count(*) FROM "geometry_columns_auth";
SELECT "Rows count of geometry_columns_statistics:";
SELECT count(*) FROM "geometry_columns_statistics";
SELECT "Rows count of geometry_columns_time:";
SELECT count(*) FROM "geometry_columns_time";
SELECT "Rows count of vector_layers_statistics:";
SELECT count(*) FROM "vector_layers_statistics";

returns:
SpatiaLite version ..: 4.5.0-devel	Supported Extensions:
SpatiaLite version ..: 4.3.0	Supported Extensions:

1
1
Rows count of geometry_columns:
1
SpatialTable|test_app_city|point|2017-11-22T01:42:33.986Z|1|1.0|2.0|1.0|2.0
DiscardGeometryColumn
1
Rows count of geometry_columns:
0
Rows count of geometry_columns_auth:
0
Rows count of geometry_columns_statistics:
0
Rows count of geometry_columns_time:
0
Rows count of vector_layers_statistics:


anonymous added on 2017-11-22 06:52:58:
I can confirm this on 4.3.0a too:

$ sqlite3 << EOF
SELECT load_extension('mod_spatialite');
SELECT InitSpatialMetaData(1);
SELECT "Spatialite version:";
SELECT spatialite_version();

CREATE TABLE "test_app_city" ("id" PRIMARY KEY);
SELECT AddGeometryColumn('test_app_city', 'point', 4326, 'POINT');
INSERT INTO test_app_city SELECT 1, MakePoint(1,2,4326);
SELECT UpdateLayerStatistics();
SELECT "Rows count of geometry_columns:";
SELECT count(*) FROM "geometry_columns";
SELECT * FROM "vector_layers_statistics";
SELECT "DiscardGeometryColumn";
SELECT DiscardGeometryColumn('test_app_city', 'point');

SELECT "Rows count of geometry_columns:";
SELECT count(*) FROM "geometry_columns";
SELECT "Rows count of geometry_columns_auth:";
SELECT count(*) FROM "geometry_columns_auth";
SELECT "Rows count of geometry_columns_statistics:";
SELECT count(*) FROM "geometry_columns_statistics";
SELECT "Rows count of geometry_columns_time:";
SELECT count(*) FROM "geometry_columns_time";
SELECT "Rows count of vector_layers_statistics:";
SELECT count(*) FROM "vector_layers_statistics";
EOF

1
Spatialite version:
4.3.0a
1
1
Rows count of geometry_columns:
1
SpatialTable|test_app_city|point|2017-11-22T06:52:13.939Z|1|1.0|2.0|1.0|2.0
DiscardGeometryColumn
1
Rows count of geometry_columns:
0
Rows count of geometry_columns_auth:
1
Rows count of geometry_columns_statistics:
1
Rows count of geometry_columns_time:
1
Rows count of vector_layers_statistics:
1

mj10777 added on 2017-11-22 08:34:11:

I can confirm, that when using sqlite3 with 'mod_spatialite' this does occur with all versions (including the development version).
The tests this morning were called with spatialite and not sqlite3 with 'mod_spatialite'.

sqlite3 test450.mod.db < create.mod.table.sql

1
Spatialite version:
4.5.0-devel
1
1
Rows count of geometry_columns:
1
SpatialTable|test_app_city|point|2017-11-22T08:28:13.445Z|1|1.0|2.0|1.0|2.0
DiscardGeometryColumn
1
Rows count of geometry_columns:
0
Rows count of geometry_columns_auth:
1
Rows count of geometry_columns_statistics:
1
Rows count of geometry_columns_time:
1
Rows count of vector_layers_statistics:
1

spatialite test450.db < create.table.sql

SpatiaLite version ..: 4.5.0-devel	Supported Extensions:
	- 'VirtualShape'	[direct Shapefile access]
	- 'VirtualDbf'		[direct DBF access]
	- 'VirtualXL'		[direct XLS access]
	- 'VirtualText'		[direct CSV/TXT access]
	- 'VirtualNetwork'	[Dijkstra shortest path]
	- 'RTree'		[Spatial Index - R*Tree]
	- 'MbrCache'		[Spatial Index - MBR cache]
	- 'VirtualSpatialIndex'	[R*Tree metahandler]
	- 'VirtualElementary'	[ElemGeoms metahandler]
	- 'VirtualKNN'	[K-Nearest Neighbors metahandler]
	- 'VirtualXPath'	[XML Path Language - XPath]
	- 'VirtualFDO'		[FDO-OGR interoperability]
	- 'VirtualGPKG'	[OGC GeoPackage interoperability]
	- 'VirtualBBox'		[BoundingBox tables]
	- 'SpatiaLite'		[Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.9.3, 15 August 2016
GEOS version ........: 3.5.0-CAPI-1.9.0 r4084
RTTOPO version ......: 1.1.0-dev
TARGET CPU ..........: x86_64-linux-gnu
the SPATIAL_REF_SYS table already contains some row(s)
1
1
Rows count of geometry_columns:
1
SpatialTable|test_app_city|point|2017-11-22T08:32:05.719Z|1|1.0|2.0|1.0|2.0
DiscardGeometryColumn
1
Rows count of geometry_columns:
0
Rows count of geometry_columns_auth:
0
Rows count of geometry_columns_statistics:
0
Rows count of geometry_columns_time:
0
Rows count of vector_layers_statistics:
0


mj10777 added on 2017-11-22 08:48:15:

With DropGeoTable, the same happens with: geometry_columns_time

sqlite3 drop450.mod.db < drop.mod.table.sql

1
Spatialite version:
4.5.0-devel
1
1
Rows count of geometry_columns:
1
SpatialTable|test_app_city|point|2017-11-22T08:39:52.787Z|1|1.0|2.0|1.0|2.0
DropGeoTable
1
Rows count of geometry_columns:
0
Rows count of geometry_columns_auth:
0
Rows count of geometry_columns_statistics:
0
Rows count of geometry_columns_time:
1
Rows count of vector_layers_statistics:
0


When called with spatialite, all values are 0.


mj10777 added on 2017-11-22 09:44:00:

The cause of this problem, is that the value of 'PRAGMA foreign_keys' is 0 at this point.

    In spatialite, spatialite_gui (and QGis) is is always turned on.
SELECT load_extension('mod_spatialite');
SELECT "PRAGMA foreign_keys:";
PRAGMA foreign_keys;
SELECT "Setting PRAGMA foreign_keys = ON:";
PRAGMA foreign_keys = ON;
SELECT "PRAGMA foreign_keys:";
PRAGMA foreign_keys;
SELECT InitSpatialMetaData(1);
SELECT "Spatialite version:";
SELECT spatialite_version();
CREATE TABLE "test_app_city" ("id" PRIMARY KEY);
SELECT AddGeometryColumn('test_app_city', 'point', 4326, 'POINT');
INSERT INTO test_app_city SELECT 1, MakePoint(1,2,4326);
SELECT UpdateLayerStatistics();
SELECT "Rows count of geometry_columns:";
SELECT count(*) FROM "geometry_columns";
SELECT * FROM "vector_layers_statistics";
SELECT "DiscardGeometryColumn";
SELECT DiscardGeometryColumn('test_app_city', 'point');

SELECT "Rows count of geometry_columns:";
SELECT count(*) FROM "geometry_columns";
SELECT "Rows count of geometry_columns_auth:";
SELECT count(*) FROM "geometry_columns_auth";
SELECT "Rows count of geometry_columns_statistics:";
SELECT count(*) FROM "geometry_columns_statistics";
SELECT "Rows count of geometry_columns_time:";
SELECT count(*) FROM "geometry_columns_time";
SELECT "Rows count of vector_layers_statistics:";
SELECT count(*) FROM "vector_layers_statistics";
returns:
PRAGMA foreign_keys:
0
Setting PRAGMA foreign_keys = ON:
PRAGMA foreign_keys:
1
1
Spatialite version:
4.5.0-devel
1
1
Rows count of geometry_columns:
1
SpatialTable|test_app_city|point|2017-11-22T09:25:59.616Z|1|1.0|2.0|1.0|2.0
DiscardGeometryColumn
1
Rows count of geometry_columns:
0
Rows count of geometry_columns_auth:
0
Rows count of geometry_columns_statistics:
0
Rows count of geometry_columns_time:
0
Rows count of vector_layers_statistics:
0

So the question would be: should "PRAGMA foreign_keys = 1" be also done for:
    mod_spatialite and mod_rasterlite2
or
    always check and set when triggers are effected that contain a ON DELETE CASCADE (or other nasties)


sandro added on 2017-11-25 20:11:31:
The default behavior adopted by libsqlite3 (and consequently by many other tools, this including the sqlite3 CLI front-end) is to never implicitly issue a "PRAGMA foreign_keys = ON" directive when opening a new DB connection.
The user is always expected to explicitly ask for enabling this specific feature, and there are at least two very good reasons supporting such a default choice:
1) for very long years SQLite3 was unable to really support Foreign Key; FKs were then considered just as effect-less elements.
   it was only starting since version 3.6.19 (October 2009) that full FKs support was introduced.
2) suddenly switching to a strong FK constraints implementation could easily cause lot of unexpected troubles to already existing applications.
   and consequently the optional "PRAGMA foreign_keys" directive was introduced so to leave full freedom of choice to end users.

On the other hand both libspatialite and librasterlite2 quickly started to take full profit from the powerful mechanisms allowed by a real FKs implementation.
And consequently many tools depending on these libraries (spatialite CLI, spatialite_gui, QGIS etc) started to automatically issue an appropriate PRAGMA so to always enabling FKs constraints before activating the extension modules to SQLite.

The problem is that now we have to face an unpleasant self-contradiction when dynamically loading mod_spatialite or mod_rasterlite2:
- the default behavior of all language connectors (Python, Java, PHP etc and even more important, libsqlite3 itself) is the one to never automatically enforce FKs constraints.
- but mod_spatialite and mod_rasterlite2 strictly require an effective FKs support, otherwise some nasty malfunction will surely happen.

I'm not really sure that implicitly switching to "foreign_keys=ON" during the initialization steps of the extension modules could be a really smart idea.
It could probably cause lots of unexpected problems and regressions to already existing applications, and all this will happen sneakingly and without any awareness from the users.
I suppose that a better solution should be the one to strongly highlight this issue on the documentation, making crystal clear that the expected way to correctly load both extension modules is as follows:

PRAGMA foreign_keys=ON;
SELECT load_extension('mod_spatialite');

such a solution will leave full responsibility to the user/developer, and will never silently/sneakingly affect any default setting.

sandro added on 2020-08-26 07:18:54:
too old to be still considered