{"id": "spatialite:importing-geojson-polygons-using-shapely", "page": "spatialite", "ref": "importing-geojson-polygons-using-shapely", "title": "Importing GeoJSON polygons using Shapely", "content": "Another common form of polygon data is the GeoJSON format. This can be imported into SpatiaLite directly, or by using the  Shapely  Python library. \n             Who's On First  is an excellent source of openly licensed GeoJSON polygons. Let's import the geographical polygon for Wales. First, we can use the Who's On First Spelunker tool to find the record for Wales: \n             spelunker.whosonfirst.org/id/404227475 \n             That page includes a link to the GeoJSON record, which can be accessed here: \n             data.whosonfirst.org/404/227/475/404227475.geojson \n             Here's Python code to create a SQLite database, enable SpatiaLite, create a places table and then add a record for Wales: \n             import sqlite3\n\nconn = sqlite3.connect(\"places.db\")\n# Enable SpatialLite extension\nconn.enable_load_extension(True)\nconn.load_extension(\"/usr/local/lib/mod_spatialite.dylib\")\n# Create the masic countries table\nconn.execute(\"select InitSpatialMetadata(1)\")\nconn.execute(\n    \"create table places (id integer primary key, name text);\"\n)\n# Add a MULTIPOLYGON Geometry column\nconn.execute(\n    \"SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);\"\n)\n# Add a spatial index against the new column\nconn.execute(\"SELECT CreateSpatialIndex('places', 'geom');\")\n# Now populate the table\nfrom shapely.geometry.multipolygon import MultiPolygon\nfrom shapely.geometry import shape\nimport requests\n\ngeojson = requests.get(\n    \"https://data.whosonfirst.org/404/227/475/404227475.geojson\"\n).json()\n# Convert to \"Well Known Text\" format\nwkt = shape(geojson[\"geometry\"]).wkt\n# Insert and commit the record\nconn.execute(\n    \"INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))\",\n    (\"Wales\", wkt),\n)\nconn.commit()", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[{\"href\": \"https://pypi.org/project/Shapely/\", \"label\": \"Shapely\"}, {\"href\": \"https://whosonfirst.org/\", \"label\": \"Who's On First\"}, {\"href\": \"https://spelunker.whosonfirst.org/id/404227475/\", \"label\": \"spelunker.whosonfirst.org/id/404227475\"}, {\"href\": \"https://data.whosonfirst.org/404/227/475/404227475.geojson\", \"label\": \"data.whosonfirst.org/404/227/475/404227475.geojson\"}]"}
{"id": "spatialite:importing-shapefiles-into-spatialite", "page": "spatialite", "ref": "importing-shapefiles-into-spatialite", "title": "Importing shapefiles into SpatiaLite", "content": "The  shapefile format  is a common format for distributing geospatial data. You can use the  spatialite  command-line tool to create a new database table from a shapefile. \n             Try it now with the North America shapefile available from the University of North Carolina  Global River Database  project. Download the file and unzip it (this will create files called  narivs.dbf ,  narivs.prj ,  narivs.shp  and  narivs.shx  in the current directory), then run the following: \n             spatialite rivers-database.db \n             SpatiaLite version ..: 4.3.0a       Supported Extensions:\n...\nspatialite> .loadshp narivs rivers CP1252 23032\n========\nLoading shapefile at 'narivs' into SQLite table 'rivers'\n...\nInserted 467973 rows into 'rivers' from SHAPEFILE \n             This will load the data from the  narivs  shapefile into a new database table called  rivers . \n             Exit out of  spatialite  (using  Ctrl+D ) and run Datasette against your new database like this: \n             datasette rivers-database.db \\\n    --load-extension=/usr/local/lib/mod_spatialite.dylib \n             If you browse to  http://localhost:8001/rivers-database/rivers  you will see the new table... but the  Geometry  column will contain unreadable binary data (SpatiaLite uses  a custom format based on WKB ). \n             The easiest way to turn this into semi-readable data is to use the SpatiaLite  AsGeoJSON  function. Try the following using the SQL query interface at  http://localhost:8001/rivers-database : \n             select *, AsGeoJSON(Geometry) from rivers limit 10; \n             This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at  geojson.io  to visualize it on a map. \n             To see a more interesting example, try ordering the records with the longest geometry first. Since there are 467,000 rows in the table you will first need to increase the SQL time limit imposed by Datasette: \n             datasette rivers-database.db \\\n    --load-extension=/usr/local/lib/mod_spatialite.dylib \\\n    --setting sql_time_limit_ms 10000 \n             Now try the following query: \n             select *, AsGeoJSON(Geometry) from rivers\norder by length(Geometry) desc limit 10;", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[{\"href\": \"https://en.wikipedia.org/wiki/Shapefile\", \"label\": \"shapefile format\"}, {\"href\": \"http://gaia.geosci.unc.edu/rivers/\", \"label\": \"Global River Database\"}, {\"href\": \"https://www.gaia-gis.it/gaia-sins/BLOB-Geometry.html\", \"label\": \"a custom format based on WKB\"}, {\"href\": \"https://geojson.io/\", \"label\": \"geojson.io\"}]"}
{"id": "spatialite:installing-spatialite-on-linux", "page": "spatialite", "ref": "installing-spatialite-on-linux", "title": "Installing SpatiaLite on Linux", "content": "SpatiaLite is packaged for most Linux distributions. \n                 apt install spatialite-bin libsqlite3-mod-spatialite \n                 Depending on your distribution, you should be able to run Datasette something like this: \n                 datasette --load-extension=/usr/lib/x86_64-linux-gnu/mod_spatialite.so \n                 If you are unsure of the location of the module, try running  locate mod_spatialite  and see what comes back.", "breadcrumbs": "[\"SpatiaLite\", \"Installation\"]", "references": "[]"}
{"id": "spatialite:installing-spatialite-on-os-x", "page": "spatialite", "ref": "installing-spatialite-on-os-x", "title": "Installing SpatiaLite on OS X", "content": "The easiest way to install SpatiaLite on OS X is to use  Homebrew . \n                 brew update\nbrew install spatialite-tools \n                 This will install the  spatialite  command-line tool and the  mod_spatialite  dynamic library. \n                 You can now run Datasette like so: \n                 datasette --load-extension=spatialite", "breadcrumbs": "[\"SpatiaLite\", \"Installation\"]", "references": "[{\"href\": \"https://brew.sh/\", \"label\": \"Homebrew\"}]"}
{"id": "spatialite:making-use-of-a-spatial-index", "page": "spatialite", "ref": "making-use-of-a-spatial-index", "title": "Making use of a spatial index", "content": "SpatiaLite spatial indexes are R*Trees. They allow you to run efficient bounding box queries using a sub-select, with a similar pattern to that used for  Searches using custom SQL . \n             In the above example, the resulting index will be called  idx_museums_point_geom . This takes the form of a SQLite virtual table. You can inspect its contents using the following query: \n             select * from idx_museums_point_geom limit 10; \n             Here's a live example:  timezones-api.datasette.io/timezones/idx_timezones_Geometry \n             \n                 \n                     \n                     \n                     \n                     \n                     \n                     \n                         \n                             \n                                 pkid \n                             \n                             \n                                 xmin \n                             \n                             \n                                 xmax \n                             \n                             \n                                 ymin \n                             \n                             \n                                 ymax \n                             \n                         \n                     \n                     \n                         \n                             \n                                 1 \n                             \n                             \n                                 -8.601725578308105 \n                             \n                             \n                                 -2.4930307865142822 \n                             \n                             \n                                 4.162120819091797 \n                             \n                             \n                                 10.74019718170166 \n                             \n                         \n                         \n                             \n                                 2 \n                             \n                             \n                                 -3.2607860565185547 \n                             \n                             \n                                 1.27329421043396 \n                             \n                             \n                                 4.539252281188965 \n                             \n                             \n                                 11.174856185913086 \n                             \n                         \n                         \n                             \n                                 3 \n                             \n                             \n                                 32.997581481933594 \n                             \n                             \n                                 47.98238754272461 \n                             \n                             \n                                 3.3974475860595703 \n                             \n                             \n                                 14.894054412841797 \n                             \n                         \n                         \n                             \n                                 4 \n                             \n                             \n                                 -8.66890811920166 \n                             \n                             \n                                 11.997337341308594 \n                             \n                             \n                                 18.9681453704834 \n                             \n                             \n                                 37.296207427978516 \n                             \n                         \n                         \n                             \n                                 5 \n                             \n                             \n                                 36.43336486816406 \n                             \n                             \n                                 43.300174713134766 \n                             \n                             \n                                 12.354820251464844 \n                             \n                             \n                                 18.070993423461914 \n                             \n                         \n                     \n                 \n             \n             You can now construct efficient bounding box queries that will make use of the index like this: \n             select * from museums where museums.rowid in (\n    SELECT pkid FROM idx_museums_point_geom\n    -- left-hand-edge of point > left-hand-edge of bbox (minx)\n    where xmin > :bbox_minx\n    -- right-hand-edge of point < right-hand-edge of bbox (maxx)\n    and xmax < :bbox_maxx\n    -- bottom-edge of point > bottom-edge of bbox (miny)\n    and ymin > :bbox_miny\n    -- top-edge of point < top-edge of bbox (maxy)\n    and ymax < :bbox_maxy\n); \n             Spatial indexes can be created against polygon columns as well as point columns, in which case they will represent the minimum bounding rectangle of that polygon. This is useful for accelerating  within  queries, as seen in the Timezones API example.", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[{\"href\": \"https://timezones-api.datasette.io/timezones/idx_timezones_Geometry\", \"label\": \"timezones-api.datasette.io/timezones/idx_timezones_Geometry\"}]"}
{"id": "spatialite:querying-polygons-using-within", "page": "spatialite", "ref": "querying-polygons-using-within", "title": "Querying polygons using within()", "content": "The  within()  SQL function can be used to check if a point is within a geometry: \n             select\n    name\nfrom\n    places\nwhere\n   within(GeomFromText('POINT(-3.1724366 51.4704448)'), places.geom); \n             The  GeomFromText()  function takes a string of well-known text. Note that the order used here is  longitude  then   latitude . \n             To run that same  within()  query in a way that benefits from the spatial index, use the following: \n             select\n    name\nfrom\n    places\nwhere\n    within(GeomFromText('POINT(-3.1724366 51.4704448)'), places.geom)\n    and rowid in (\n        SELECT pkid FROM idx_places_geom\n        where xmin < -3.1724366\n        and xmax > -3.1724366\n        and ymin < 51.4704448\n        and ymax > 51.4704448\n    );", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[]"}
{"id": "spatialite:spatial-indexing-latitude-longitude-columns", "page": "spatialite", "ref": "spatial-indexing-latitude-longitude-columns", "title": "Spatial indexing latitude/longitude columns", "content": "Here's a recipe for taking a table with existing latitude and longitude columns, adding a SpatiaLite POINT geometry column to that table, populating the new column and then populating a spatial index: \n             import sqlite3\n\nconn = sqlite3.connect(\"museums.db\")\n# Lead the spatialite extension:\nconn.enable_load_extension(True)\nconn.load_extension(\"/usr/local/lib/mod_spatialite.dylib\")\n# Initialize spatial metadata for this database:\nconn.execute(\"select InitSpatialMetadata(1)\")\n# Add a geometry column called point_geom to our museums table:\nconn.execute(\n    \"SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);\"\n)\n# Now update that geometry column with the lat/lon points\nconn.execute(\n    \"\"\"\n    UPDATE museums SET\n    point_geom = GeomFromText('POINT('||\"longitude\"||' '||\"latitude\"||')',4326);\n\"\"\"\n)\n# Now add a spatial index to that column\nconn.execute(\n    'select CreateSpatialIndex(\"museums\", \"point_geom\");'\n)\n# If you don't commit your changes will not be persisted:\nconn.commit()\nconn.close()", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[]"}
{"id": "spatialite:spatialite-installation", "page": "spatialite", "ref": "spatialite-installation", "title": "Installation", "content": "", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[]"}
{"id": "spatialite:spatialite-warning", "page": "spatialite", "ref": "spatialite-warning", "title": "Warning", "content": "The SpatiaLite extension adds  a large number of additional SQL functions , some of which are not be safe for untrusted users to execute: they may cause the Datasette server to crash. \n                 You should not expose a SpatiaLite-enabled Datasette instance to the public internet without taking extra measures to secure it against potentially harmful SQL queries. \n                 The following steps are recommended: \n                 \n                     \n                         Disable arbitrary SQL queries by untrusted users. See  Controlling the ability to execute arbitrary SQL  for ways to do this. The easiest is to start Datasette with the  datasette --setting default_allow_sql off  option. \n                     \n                     \n                         Define  Canned queries  with the SQL queries that use SpatiaLite functions that you want people to be able to execute. \n                     \n                 \n                 The  Datasette SpatiaLite tutorial  includes detailed instructions for running SpatiaLite safely using these techniques", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[{\"href\": \"https://www.gaia-gis.it/gaia-sins/spatialite-sql-5.0.1.html\", \"label\": \"a large number of additional SQL functions\"}, {\"href\": \"https://datasette.io/tutorials/spatialite\", \"label\": \"Datasette SpatiaLite tutorial\"}]"}