{"id": "changelog:id14", "page": "changelog", "ref": "id14", "title": "Features", "content": "Datasette is now compatible with Pyodide . This is the enabling technology behind Datasette Lite . ( #1733 ) \n \n \n Database file downloads now implement conditional GET using ETags. ( #1739 ) \n \n \n HTML for facet results and suggested results has been extracted out into new templates _facet_results.html and _suggested_facets.html . Thanks, M. Nasimul Haque. ( #1759 ) \n \n \n Datasette now runs some SQL queries in parallel. This has limited impact on performance, see this research issue for details. \n \n \n New --nolock option for ignoring file locks when opening read-only databases. ( #1744 ) \n \n \n Spaces in the database names in URLs are now encoded as + rather than ~20 . ( #1701 ) \n \n \n is now displayed as and is accompanied by tooltip showing \"2.3MB\". ( #1712 ) \n \n \n The base Docker image used by datasette publish cloudrun , datasette package and the official Datasette image has been upgraded to 3.10.6-slim-bullseye . ( #1768 ) \n \n \n Canned writable queries against immutable databases now show a warning message. ( #1728 ) \n \n \n datasette publish cloudrun has a new --timeout option which can be used to increase the time limit applied by the Google Cloud build environment. Thanks, Tim Sherratt. ( #1717 ) \n \n \n datasette publish cloudrun has new --min-instances and --max-instances options. ( #1779 )", "breadcrumbs": "[\"Changelog\", \"0.62 (2022-08-14)\"]", "references": "[{\"href\": \"https://pyodide.org/\", \"label\": \"Pyodide\"}, {\"href\": \"https://lite.datasette.io/\", \"label\": \"Datasette Lite\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1733\", \"label\": \"#1733\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1739\", \"label\": \"#1739\"}, {\"href\": \"https://github.com/simonw/datasette/pull/1759\", \"label\": \"#1759\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1727\", \"label\": \"this research issue\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1744\", \"label\": \"#1744\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1701\", \"label\": \"#1701\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1712\", \"label\": \"#1712\"}, {\"href\": \"https://hub.docker.com/datasetteproject/datasette\", \"label\": \"official Datasette image\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1768\", \"label\": \"#1768\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1728\", \"label\": \"#1728\"}, {\"href\": \"https://github.com/simonw/datasette/pull/1717\", \"label\": \"#1717\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1779\", \"label\": \"#1779\"}]"} {"id": "settings:setting-num-sql-threads", "page": "settings", "ref": "setting-num-sql-threads", "title": "num_sql_threads", "content": "Maximum number of threads in the thread pool Datasette uses to execute SQLite queries. Defaults to 3. \n datasette mydatabase.db --setting num_sql_threads 10 \n Setting this to 0 turns off threaded SQL queries entirely - useful for environments that do not support threading such as Pyodide .", "breadcrumbs": "[\"Settings\", \"Settings\"]", "references": "[{\"href\": \"https://pyodide.org/\", \"label\": \"Pyodide\"}]"} {"id": "writing_plugins:id1", "page": "writing_plugins", "ref": "id1", "title": "Writing plugins", "content": "You can write one-off plugins that apply to just one Datasette instance, or you can write plugins which can be installed using pip and can be shipped to the Python Package Index ( PyPI ) for other people to install. \n Want to start by looking at an example? The Datasette plugins directory lists more than 90 open source plugins with code you can explore. The plugin hooks page includes links to example plugins for each of the documented hooks.", "breadcrumbs": "[]", "references": "[{\"href\": \"https://pypi.org/\", \"label\": \"PyPI\"}, {\"href\": \"https://datasette.io/plugins\", \"label\": \"Datasette plugins directory\"}]"} {"id": "spatialite:importing-geojson-polygons-using-shapely", "page": "spatialite", "ref": "importing-geojson-polygons-using-shapely", "title": "Importing GeoJSON polygons using Shapely", "content": "Another common form of polygon data is the GeoJSON format. This can be imported into SpatiaLite directly, or by using the Shapely Python library. \n Who's On First is an excellent source of openly licensed GeoJSON polygons. Let's import the geographical polygon for Wales. First, we can use the Who's On First Spelunker tool to find the record for Wales: \n spelunker.whosonfirst.org/id/404227475 \n That page includes a link to the GeoJSON record, which can be accessed here: \n data.whosonfirst.org/404/227/475/404227475.geojson \n Here's Python code to create a SQLite database, enable SpatiaLite, create a places table and then add a record for Wales: \n import sqlite3\n\nconn = sqlite3.connect(\"places.db\")\n# Enable SpatialLite extension\nconn.enable_load_extension(True)\nconn.load_extension(\"/usr/local/lib/mod_spatialite.dylib\")\n# Create the masic countries table\nconn.execute(\"select InitSpatialMetadata(1)\")\nconn.execute(\n \"create table places (id integer primary key, name text);\"\n)\n# Add a MULTIPOLYGON Geometry column\nconn.execute(\n \"SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);\"\n)\n# Add a spatial index against the new column\nconn.execute(\"SELECT CreateSpatialIndex('places', 'geom');\")\n# Now populate the table\nfrom shapely.geometry.multipolygon import MultiPolygon\nfrom shapely.geometry import shape\nimport requests\n\ngeojson = requests.get(\n \"https://data.whosonfirst.org/404/227/475/404227475.geojson\"\n).json()\n# Convert to \"Well Known Text\" format\nwkt = shape(geojson[\"geometry\"]).wkt\n# Insert and commit the record\nconn.execute(\n \"INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))\",\n (\"Wales\", wkt),\n)\nconn.commit()", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[{\"href\": \"https://pypi.org/project/Shapely/\", \"label\": \"Shapely\"}, {\"href\": \"https://whosonfirst.org/\", \"label\": \"Who's On First\"}, {\"href\": \"https://spelunker.whosonfirst.org/id/404227475/\", \"label\": \"spelunker.whosonfirst.org/id/404227475\"}, {\"href\": \"https://data.whosonfirst.org/404/227/475/404227475.geojson\", \"label\": \"data.whosonfirst.org/404/227/475/404227475.geojson\"}]"} {"id": "contributing:contributing-formatting-blacken-docs", "page": "contributing", "ref": "contributing-formatting-blacken-docs", "title": "blacken-docs", "content": "The blacken-docs command applies Black formatting rules to code examples in the documentation. Run it like this: \n blacken-docs -l 60 docs/*.rst", "breadcrumbs": "[\"Contributing\", \"Code formatting\"]", "references": "[{\"href\": \"https://pypi.org/project/blacken-docs/\", \"label\": \"blacken-docs\"}]"} {"id": "index:datasette", "page": "index", "ref": "datasette", "title": "Datasette", "content": "An open source multi-tool for exploring and publishing data \n Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API. \n Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world. It is part of a wider ecosystem of tools and plugins dedicated to making working with structured data as productive as possible. \n Explore a demo , watch a presentation about the project or Try Datasette without installing anything using Glitch . \n Interested in learning Datasette? Start with the official tutorials . \n Support questions, feedback? Join the Datasette Discord .", "breadcrumbs": "[]", "references": "[{\"href\": \"https://pypi.org/project/datasette/\", \"label\": null}, {\"href\": \"https://docs.datasette.io/en/stable/changelog.html\", \"label\": null}, {\"href\": \"https://pypi.org/project/datasette/\", \"label\": null}, {\"href\": \"https://github.com/simonw/datasette/actions?query=workflow%3ATest\", \"label\": null}, {\"href\": \"https://github.com/simonw/datasette/blob/main/LICENSE\", \"label\": null}, {\"href\": \"https://hub.docker.com/r/datasetteproject/datasette\", \"label\": null}, {\"href\": \"https://datasette.io/discord\", \"label\": null}, {\"href\": \"https://pypi.org/project/datasette/\", \"label\": null}, {\"href\": \"https://docs.datasette.io/en/stable/changelog.html\", \"label\": null}, {\"href\": \"https://pypi.org/project/datasette/\", \"label\": null}, {\"href\": \"https://github.com/simonw/datasette/actions?query=workflow%3ATest\", \"label\": null}, {\"href\": \"https://github.com/simonw/datasette/blob/main/LICENSE\", \"label\": null}, {\"href\": \"https://hub.docker.com/r/datasetteproject/datasette\", \"label\": null}, {\"href\": \"https://datasette.io/discord\", \"label\": null}, {\"href\": \"https://fivethirtyeight.datasettes.com/fivethirtyeight\", \"label\": \"Explore a demo\"}, {\"href\": \"https://static.simonwillison.net/static/2018/pybay-datasette/\", \"label\": \"a presentation about the project\"}, {\"href\": \"https://datasette.io/tutorials\", \"label\": \"the official tutorials\"}, {\"href\": \"https://datasette.io/discord\", \"label\": \"Datasette Discord\"}]"} {"id": "testing_plugins:testing-plugins-pytest-httpx", "page": "testing_plugins", "ref": "testing-plugins-pytest-httpx", "title": "Testing outbound HTTP calls with pytest-httpx", "content": "If your plugin makes outbound HTTP calls - for example datasette-auth-github or datasette-import-table - you may need to mock those HTTP requests in your tests. \n The pytest-httpx package is a useful library for mocking calls. It can be tricky to use with Datasette though since it mocks all HTTPX requests, and Datasette's own testing mechanism uses HTTPX internally. \n To avoid breaking your tests, you can return [\"localhost\"] from the non_mocked_hosts() fixture. \n As an example, here's a very simple plugin which executes an HTTP response and returns the resulting content: \n from datasette import hookimpl\nfrom datasette.utils.asgi import Response\nimport httpx\n\n\n@hookimpl\ndef register_routes():\n return [\n (r\"^/-/fetch-url$\", fetch_url),\n ]\n\n\nasync def fetch_url(datasette, request):\n if request.method == \"GET\":\n return Response.html(\n \"\"\"\n
\n \n \n
\"\"\".format(\n request.scope[\"csrftoken\"]()\n )\n )\n vars = await request.post_vars()\n url = vars[\"url\"]\n return Response.text(httpx.get(url).text) \n Here's a test for that plugin that mocks the HTTPX outbound request: \n from datasette.app import Datasette\nimport pytest\n\n\n@pytest.fixture\ndef non_mocked_hosts():\n # This ensures httpx-mock will not affect Datasette's own\n # httpx calls made in the tests by datasette.client:\n return [\"localhost\"]\n\n\nasync def test_outbound_http_call(httpx_mock):\n httpx_mock.add_response(\n url=\"https://www.example.com/\",\n text=\"Hello world\",\n )\n datasette = Datasette([], memory=True)\n response = await datasette.client.post(\n \"/-/fetch-url\",\n data={\"url\": \"https://www.example.com/\"},\n )\n assert response.text == \"Hello world\"\n\n outbound_request = httpx_mock.get_request()\n assert (\n outbound_request.url == \"https://www.example.com/\"\n )", "breadcrumbs": "[\"Testing plugins\"]", "references": "[{\"href\": \"https://pypi.org/project/pytest-httpx/\", \"label\": \"pytest-httpx\"}]"} {"id": "contributing:contributing-documentation", "page": "contributing", "ref": "contributing-documentation", "title": "Editing and building the documentation", "content": "Datasette's documentation lives in the docs/ directory and is deployed automatically using Read The Docs . \n The documentation is written using reStructuredText. You may find this article on The subset of reStructuredText worth committing to memory useful. \n You can build it locally by installing sphinx and sphinx_rtd_theme in your Datasette development environment and then running make html directly in the docs/ directory: \n # You may first need to activate your virtual environment:\nsource venv/bin/activate\n\n# Install the dependencies needed to build the docs\npip install -e .[docs]\n\n# Now build the docs\ncd docs/\nmake html \n This will create the HTML version of the documentation in docs/_build/html . You can open it in your browser like so: \n open _build/html/index.html \n Any time you make changes to a .rst file you can re-run make html to update the built documents, then refresh them in your browser. \n For added productivity, you can use use sphinx-autobuild to run Sphinx in auto-build mode. This will run a local webserver serving the docs that automatically rebuilds them and refreshes the page any time you hit save in your editor. \n sphinx-autobuild will have been installed when you ran pip install -e .[docs] . In your docs/ directory you can start the server by running the following: \n make livehtml \n Now browse to http://localhost:8000/ to view the documentation. Any edits you make should be instantly reflected in your browser.", "breadcrumbs": "[\"Contributing\"]", "references": "[{\"href\": \"https://readthedocs.org/\", \"label\": \"Read The Docs\"}, {\"href\": \"https://simonwillison.net/2018/Aug/25/restructuredtext/\", \"label\": \"The subset of reStructuredText worth committing to memory\"}, {\"href\": \"https://pypi.org/project/sphinx-autobuild/\", \"label\": \"sphinx-autobuild\"}]"} {"id": "pages:tableview", "page": "pages", "ref": "tableview", "title": "Table", "content": "The table page is the heart of Datasette: it allows users to interactively explore the contents of a database table, including sorting, filtering, Full-text search and applying Facets . \n The HTML interface is worth spending some time exploring. As with other pages, you can return the JSON data by appending .json to the URL path, before any ? query string arguments. \n The query string arguments are described in more detail here: Table arguments \n You can also use the table page to interactively construct a SQL query - by applying different filters and a sort order for example - and then click the \"View and edit SQL\" link to see the SQL query that was used for the page and edit and re-submit it. \n Some examples: \n \n \n ../items lists all of the line-items registered by UK MPs as potential conflicts of interest. It demonstrates Datasette's support for Full-text search . \n \n \n ../antiquities-act%2Factions_under_antiquities_act is an interface for exploring the \"actions under the antiquities act\" data table published by FiveThirtyEight. \n \n \n ../global-power-plants?country_long=United+Kingdom&primary_fuel=Gas is a filtered table page showing every Gas power plant in the United Kingdom. It includes some default facets (configured using its metadata.json ) and uses the datasette-cluster-map plugin to show a map of the results.", "breadcrumbs": "[\"Pages and API endpoints\"]", "references": "[{\"href\": \"https://register-of-members-interests.datasettes.com/regmem/items\", \"label\": \"../items\"}, {\"href\": \"https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act\", \"label\": \"../antiquities-act%2Factions_under_antiquities_act\"}, {\"href\": \"https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=owner&_facet=country_long&country_long__exact=United+Kingdom&primary_fuel=Gas\", \"label\": \"../global-power-plants?country_long=United+Kingdom&primary_fuel=Gas\"}, {\"href\": \"https://global-power-plants.datasettes.com/-/metadata\", \"label\": \"its metadata.json\"}, {\"href\": \"https://github.com/simonw/datasette-cluster-map\", \"label\": \"datasette-cluster-map\"}]"} {"id": "pages:rowview", "page": "pages", "ref": "rowview", "title": "Row", "content": "Every row in every Datasette table has its own URL. This means individual records can be linked to directly. \n Table cells with extremely long text contents are truncated on the table view according to the truncate_cells_html setting. If a cell has been truncated the full length version of that cell will be available on the row page. \n Rows which are the targets of foreign key references from other tables will show a link to a filtered search for all records that reference that row. Here's an example from the Registers of Members Interests database: \n ../people/uk~2Eorg~2Epublicwhip~2Fperson~2F10001 \n Note that this URL includes the encoded primary key of the record. \n Here's that same page as JSON: \n ../people/uk~2Eorg~2Epublicwhip~2Fperson~2F10001.json", "breadcrumbs": "[\"Pages and API endpoints\"]", "references": "[{\"href\": \"https://register-of-members-interests.datasettes.com/regmem/people/uk~2Eorg~2Epublicwhip~2Fperson~2F10001\", \"label\": \"../people/uk~2Eorg~2Epublicwhip~2Fperson~2F10001\"}, {\"href\": \"https://register-of-members-interests.datasettes.com/regmem/people/uk~2Eorg~2Epublicwhip~2Fperson~2F10001.json\", \"label\": \"../people/uk~2Eorg~2Epublicwhip~2Fperson~2F10001.json\"}]"} {"id": "json_api:json-api-pagination", "page": "json_api", "ref": "json-api-pagination", "title": "Pagination", "content": "The default JSON representation includes a \"next_url\" key which can be used to access the next page of results. If that key is null or missing then it means you have reached the final page of results. \n Other representations include pagination information in the link HTTP header. That header will look something like this: \n link: ; rel=\"next\" \n Here is an example Python function built using requests that returns a list of all of the paginated items from one of these API endpoints: \n def paginate(url):\n items = []\n while url:\n response = requests.get(url)\n try:\n url = response.links.get(\"next\").get(\"url\")\n except AttributeError:\n url = None\n items.extend(response.json())\n return items", "breadcrumbs": "[\"JSON API\"]", "references": "[{\"href\": \"https://requests.readthedocs.io/\", \"label\": \"requests\"}]"} {"id": "introspection:jsondataview-plugins", "page": "introspection", "ref": "jsondataview-plugins", "title": "/-/plugins", "content": "Shows a list of currently installed plugins and their versions. Plugins example : \n [\n {\n \"name\": \"datasette_cluster_map\",\n \"static\": true,\n \"templates\": false,\n \"version\": \"0.10\",\n \"hooks\": [\"extra_css_urls\", \"extra_js_urls\", \"extra_body_script\"]\n }\n] \n Add ?all=1 to include details of the default plugins baked into Datasette.", "breadcrumbs": "[\"Introspection\"]", "references": "[{\"href\": \"https://san-francisco.datasettes.com/-/plugins\", \"label\": \"Plugins example\"}]"} {"id": "plugin_hooks:plugin-hook-handle-exception", "page": "plugin_hooks", "ref": "plugin-hook-handle-exception", "title": "handle_exception(datasette, request, exception)", "content": "datasette - Datasette class \n \n You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name) , or to render templates or execute SQL queries. \n \n \n \n request - Request object \n \n The current HTTP request. \n \n \n \n exception - Exception \n \n The exception that was raised. \n \n \n \n This hook is called any time an unexpected exception is raised. You can use it to record the exception. \n If your handler returns a Response object it will be returned to the client in place of the default Datasette error page. \n The handler can return a response directly, or it can return return an awaitable function that returns a response. \n This example logs an error to Sentry and then renders a custom error page: \n from datasette import hookimpl, Response\nimport sentry_sdk\n\n\n@hookimpl\ndef handle_exception(datasette, exception):\n sentry_sdk.capture_exception(exception)\n\n async def inner():\n return Response.html(\n await datasette.render_template(\n \"custom_error.html\", request=request\n )\n )\n\n return inner \n Example: datasette-sentry", "breadcrumbs": "[\"Plugin hooks\"]", "references": "[{\"href\": \"https://sentry.io/\", \"label\": \"Sentry\"}, {\"href\": \"https://datasette.io/plugins/datasette-sentry\", \"label\": \"datasette-sentry\"}]"} {"id": "changelog:id118", "page": "changelog", "ref": "id118", "title": "0.22 (2018-05-20)", "content": "The big new feature in this release is Facets . Datasette can now apply faceted browse to any column in any table. It will also suggest possible facets. See the Datasette Facets announcement post for more details. \n In addition to the work on facets: \n \n \n Added docs for introspection endpoints \n \n \n New --config option, added --help-config , closes #274 \n Removed the --page_size= argument to datasette serve in favour of: \n datasette serve --config default_page_size:50 mydb.db \n Added new help section: \n datasette --help-config \n Config options:\n default_page_size Default page size for the table view\n (default=100)\n max_returned_rows Maximum rows that can be returned from a table\n or custom query (default=1000)\n sql_time_limit_ms Time limit for a SQL query in milliseconds\n (default=1000)\n default_facet_size Number of values to return for requested facets\n (default=30)\n facet_time_limit_ms Time limit for calculating a requested facet\n (default=200)\n facet_suggest_time_limit_ms Time limit for calculating a suggested facet\n (default=50) \n \n \n Only apply responsive table styles to .rows-and-column \n Otherwise they interfere with tables in the description, e.g. on\n https://fivethirtyeight.datasettes.com/fivethirtyeight/nba-elo%2Fnbaallelo \n \n \n Refactored views into new views/ modules, refs #256 \n \n \n Documentation for SQLite full-text search support, closes #253 \n \n \n /-/versions now includes SQLite fts_versions , closes #252", "breadcrumbs": "[\"Changelog\"]", "references": "[{\"href\": \"https://simonwillison.net/2018/May/20/datasette-facets/\", \"label\": \"Datasette Facets\"}, {\"href\": \"https://docs.datasette.io/en/stable/introspection.html\", \"label\": \"docs for introspection endpoints\"}, {\"href\": \"https://github.com/simonw/datasette/issues/274\", \"label\": \"#274\"}, {\"href\": \"https://fivethirtyeight.datasettes.com/fivethirtyeight/nba-elo%2Fnbaallelo\", \"label\": \"https://fivethirtyeight.datasettes.com/fivethirtyeight/nba-elo%2Fnbaallelo\"}, {\"href\": \"https://github.com/simonw/datasette/issues/256\", \"label\": \"#256\"}, {\"href\": \"https://docs.datasette.io/en/stable/full_text_search.html\", \"label\": \"Documentation for SQLite full-text search\"}, {\"href\": \"https://github.com/simonw/datasette/issues/253\", \"label\": \"#253\"}, {\"href\": \"https://github.com/simonw/datasette/issues/252\", \"label\": \"#252\"}]"} {"id": "changelog:v0-28-databases-that-change", "page": "changelog", "ref": "v0-28-databases-that-change", "title": "Supporting databases that change", "content": "From the beginning of the project, Datasette has been designed with read-only databases in mind. If a database is guaranteed not to change it opens up all kinds of interesting opportunities - from taking advantage of SQLite immutable mode and HTTP caching to bundling static copies of the database directly in a Docker container. The interesting ideas in Datasette explores this idea in detail. \n As my goals for the project have developed, I realized that read-only databases are no longer the right default. SQLite actually supports concurrent access very well provided only one thread attempts to write to a database at a time, and I keep encountering sensible use-cases for running Datasette on top of a database that is processing inserts and updates. \n So, as-of version 0.28 Datasette no longer assumes that a database file will not change. It is now safe to point Datasette at a SQLite database which is being updated by another process. \n Making this change was a lot of work - see tracking tickets #418 , #419 and #420 . It required new thinking around how Datasette should calculate table counts (an expensive operation against a large, changing database) and also meant reconsidering the \"content hash\" URLs Datasette has used in the past to optimize the performance of HTTP caches. \n Datasette can still run against immutable files and gains numerous performance benefits from doing so, but this is no longer the default behaviour. Take a look at the new Performance and caching documentation section for details on how to make the most of Datasette against data that you know will be staying read-only and immutable.", "breadcrumbs": "[\"Changelog\", \"0.28 (2019-05-19)\"]", "references": "[{\"href\": \"https://simonwillison.net/2018/Oct/4/datasette-ideas/\", \"label\": \"The interesting ideas in Datasette\"}, {\"href\": \"https://github.com/simonw/datasette/issues/418\", \"label\": \"#418\"}, {\"href\": \"https://github.com/simonw/datasette/issues/419\", \"label\": \"#419\"}, {\"href\": \"https://github.com/simonw/datasette/issues/420\", \"label\": \"#420\"}]"} {"id": "changelog:id58", "page": "changelog", "ref": "id58", "title": "0.45 (2020-07-01)", "content": "See also Datasette 0.45: The annotated release notes . \n Magic parameters for canned queries, a log out feature, improved plugin documentation and four new plugin hooks.", "breadcrumbs": "[\"Changelog\"]", "references": "[{\"href\": \"https://simonwillison.net/2020/Jul/1/datasette-045/\", \"label\": \"Datasette 0.45: The annotated release notes\"}]"} {"id": "changelog:id60", "page": "changelog", "ref": "id60", "title": "0.44 (2020-06-11)", "content": "See also Datasette 0.44: The annotated release notes . \n Authentication and permissions, writable canned queries, flash messages, new plugin hooks and more.", "breadcrumbs": "[\"Changelog\"]", "references": "[{\"href\": \"https://simonwillison.net/2020/Jun/12/annotated-release-notes/\", \"label\": \"Datasette 0.44: The annotated release notes\"}]"} {"id": "changelog:id51", "page": "changelog", "ref": "id51", "title": "0.49 (2020-09-14)", "content": "See also Datasette 0.49: The annotated release notes . \n \n \n Writable canned queries now expose a JSON API, see JSON API for writable canned queries . ( #880 ) \n \n \n New mechanism for defining page templates with custom path parameters - a template file called pages/about/{slug}.html will be used to render any requests to /about/something . See Path parameters for pages . ( #944 ) \n \n \n register_output_renderer() render functions can now return a Response . ( #953 ) \n \n \n New --upgrade option for datasette install . ( #945 ) \n \n \n New datasette --pdb option. ( #962 ) \n \n \n datasette --get exit code now reflects the internal HTTP status code. ( #947 ) \n \n \n New raise_404() template function for returning 404 errors. ( #964 ) \n \n \n datasette publish heroku now deploys using Python 3.8.5 \n \n \n Upgraded CodeMirror to 5.57.0. ( #948 ) \n \n \n Upgraded code style to Black 20.8b1. ( #958 ) \n \n \n Fixed bug where selected facets were not correctly persisted in hidden form fields on the table page. ( #963 ) \n \n \n Renamed the default error template from 500.html to error.html . \n \n \n Custom error pages are now documented, see Custom error pages . ( #965 )", "breadcrumbs": "[\"Changelog\"]", "references": "[{\"href\": \"https://simonwillison.net/2020/Sep/15/datasette-0-49/\", \"label\": \"Datasette 0.49: The annotated release notes\"}, {\"href\": \"https://github.com/simonw/datasette/issues/880\", \"label\": \"#880\"}, {\"href\": \"https://github.com/simonw/datasette/issues/944\", \"label\": \"#944\"}, {\"href\": \"https://github.com/simonw/datasette/issues/953\", \"label\": \"#953\"}, {\"href\": \"https://github.com/simonw/datasette/issues/945\", \"label\": \"#945\"}, {\"href\": \"https://github.com/simonw/datasette/issues/962\", \"label\": \"#962\"}, {\"href\": \"https://github.com/simonw/datasette/issues/947\", \"label\": \"#947\"}, {\"href\": \"https://github.com/simonw/datasette/issues/964\", \"label\": \"#964\"}, {\"href\": \"https://codemirror.net/\", \"label\": \"CodeMirror\"}, {\"href\": \"https://github.com/simonw/datasette/issues/948\", \"label\": \"#948\"}, {\"href\": \"https://github.com/simonw/datasette/issues/958\", \"label\": \"#958\"}, {\"href\": \"https://github.com/simonw/datasette/issues/963\", \"label\": \"#963\"}, {\"href\": \"https://github.com/simonw/datasette/issues/965\", \"label\": \"#965\"}]"} {"id": "internals:internals-utils-await-me-maybe", "page": "internals", "ref": "internals-utils-await-me-maybe", "title": "await_me_maybe(value)", "content": "Utility function for calling await on a return value if it is awaitable, otherwise returning the value. This is used by Datasette to support plugin hooks that can optionally return awaitable functions. Read more about this function in The \u201cawait me maybe\u201d pattern for Python asyncio . \n \n \n async datasette.utils. await_me_maybe value : Any Any \n \n If value is callable, call it. If awaitable, await it. Otherwise return it.", "breadcrumbs": "[\"Internals for plugins\", \"The datasette.utils module\"]", "references": "[{\"href\": \"https://simonwillison.net/2020/Sep/2/await-me-maybe/\", \"label\": \"The \u201cawait me maybe\u201d pattern for Python asyncio\"}]"} {"id": "changelog:id35", "page": "changelog", "ref": "id35", "title": "0.54 (2021-01-25)", "content": "The two big new features in this release are the _internal SQLite in-memory database storing details of all connected databases and tables, and support for JavaScript modules in plugins and additional scripts. \n For additional commentary on this release, see Datasette 0.54, the annotated release notes .", "breadcrumbs": "[\"Changelog\"]", "references": "[{\"href\": \"https://simonwillison.net/2021/Jan/25/datasette/\", \"label\": \"Datasette 0.54, the annotated release notes\"}]"} {"id": "changelog:v1-0-a2", "page": "changelog", "ref": "v1-0-a2", "title": "1.0a2 (2022-12-14)", "content": "The third Datasette 1.0 alpha release adds upsert support to the JSON API, plus the ability to specify finely grained permissions when creating an API token. \n See Datasette 1.0a2: Upserts and finely grained permissions for an extended, annotated version of these release notes. \n \n \n New /db/table/-/upsert API, documented here . upsert is an update-or-insert: existing rows will have specified keys updated, but if no row matches the incoming primary key a brand new row will be inserted instead. ( #1878 ) \n \n \n New register_permissions(datasette) plugin hook. Plugins can now register named permissions, which will then be listed in various interfaces that show available permissions. ( #1940 ) \n \n \n The /db/-/create API for creating a table now accepts \"ignore\": true and \"replace\": true options when called with the \"rows\" property that creates a new table based on an example set of rows. This means the API can be called multiple times with different rows, setting rules for what should happen if a primary key collides with an existing row. ( #1927 ) \n \n \n Arbitrary permissions can now be configured at the instance, database and resource (table, SQL view or canned query) level in Datasette's Metadata JSON and YAML files. The new \"permissions\" key can be used to specify which actors should have which permissions. See Other permissions in datasette.yaml for details. ( #1636 ) \n \n \n The /-/create-token page can now be used to create API tokens which are restricted to just a subset of actions, including against specific databases or resources. See API Tokens for details. ( #1947 ) \n \n \n Likewise, the datasette create-token CLI command can now create tokens with a subset of permissions . ( #1855 ) \n \n \n New datasette.create_token() API method for programmatically creating signed API tokens. ( #1951 ) \n \n \n /db/-/create API now requires actor to have insert-row permission in order to use the \"row\" or \"rows\" properties. ( #1937 )", "breadcrumbs": "[\"Changelog\"]", "references": "[{\"href\": \"https://simonwillison.net/2022/Dec/15/datasette-1a2/\", \"label\": \"Datasette 1.0a2: Upserts and finely grained permissions\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1878\", \"label\": \"#1878\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1940\", \"label\": \"#1940\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1927\", \"label\": \"#1927\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1636\", \"label\": \"#1636\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1947\", \"label\": \"#1947\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1855\", \"label\": \"#1855\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1951\", \"label\": \"#1951\"}, {\"href\": \"https://github.com/simonw/datasette/issues/1937\", \"label\": \"#1937\"}]"} {"id": "changelog:id11", "page": "changelog", "ref": "id11", "title": "0.63 (2022-10-27)", "content": "See Datasette 0.63: The annotated release notes for more background on the changes in this release.", "breadcrumbs": "[\"Changelog\"]", "references": "[{\"href\": \"https://simonwillison.net/2022/Oct/27/datasette-0-63/\", \"label\": \"Datasette 0.63: The annotated release notes\"}]"} {"id": "changelog:v1-0-a8", "page": "changelog", "ref": "v1-0-a8", "title": "1.0a8 (2024-02-07)", "content": "This alpha release continues the migration of Datasette's configuration from metadata.yaml to the new datasette.yaml configuration file, introduces a new system for JavaScript plugins and adds several new plugin hooks. \n See Datasette 1.0a8: JavaScript plugins, new plugin hooks and plugin configuration in datasette.yaml for an annotated version of these release notes.", "breadcrumbs": "[\"Changelog\"]", "references": "[{\"href\": \"https://simonwillison.net/2024/Feb/7/datasette-1a8/\", \"label\": \"Datasette 1.0a8: JavaScript plugins, new plugin hooks and plugin configuration in datasette.yaml\"}]"} {"id": "full_text_search:configuring-fts-using-sqlite-utils", "page": "full_text_search", "ref": "configuring-fts-using-sqlite-utils", "title": "Configuring FTS using sqlite-utils", "content": "sqlite-utils is a CLI utility and Python library for manipulating SQLite databases. You can use it from Python code to configure FTS search, or you can achieve the same goal using the accompanying command-line tool . \n Here's how to use sqlite-utils to enable full-text search for an items table across the name and description columns: \n sqlite-utils enable-fts mydatabase.db items name description", "breadcrumbs": "[\"Full-text search\", \"Enabling full-text search for a SQLite table\"]", "references": "[{\"href\": \"https://sqlite-utils.datasette.io/\", \"label\": \"sqlite-utils\"}, {\"href\": \"https://sqlite-utils.datasette.io/en/latest/python-api.html#enabling-full-text-search\", \"label\": \"it from Python code\"}, {\"href\": \"https://sqlite-utils.datasette.io/en/latest/cli.html#configuring-full-text-search\", \"label\": \"using the accompanying command-line tool\"}]"} {"id": "sql_queries:sql-views", "page": "sql_queries", "ref": "sql-views", "title": "Views", "content": "If you want to bundle some pre-written SQL queries with your Datasette-hosted database you can do so in two ways. The first is to include SQL views in your database - Datasette will then list those views on your database index page. \n The quickest way to create views is with the SQLite command-line interface: \n sqlite3 sf-trees.db \n SQLite version 3.19.3 2017-06-27 16:48:08\nEnter \".help\" for usage hints.\nsqlite> CREATE VIEW demo_view AS select qSpecies from Street_Tree_List;\n \n You can also use the sqlite-utils tool to create a view : \n sqlite-utils create-view sf-trees.db demo_view \"select qSpecies from Street_Tree_List\"", "breadcrumbs": "[\"Running SQL queries\"]", "references": "[{\"href\": \"https://sqlite-utils.datasette.io/\", \"label\": \"sqlite-utils\"}, {\"href\": \"https://sqlite-utils.datasette.io/en/stable/cli.html#creating-views\", \"label\": \"create a view\"}]"} {"id": "ecosystem:sqlite-utils", "page": "ecosystem", "ref": "sqlite-utils", "title": "sqlite-utils", "content": "sqlite-utils is a key building block for the wider Datasette ecosystem. It provides a collection of utilities for manipulating SQLite databases, both as a Python library and a command-line utility. Features include: \n \n \n Insert data into a SQLite database from JSON, CSV or TSV, automatically creating tables with the correct schema or altering existing tables to add missing columns. \n \n \n Configure tables for use with SQLite full-text search, including creating triggers needed to keep the search index up-to-date. \n \n \n Modify tables in ways that are not supported by SQLite's default ALTER TABLE syntax - for example changing the types of columns or selecting a new primary key for a table. \n \n \n Adding foreign keys to existing database tables. \n \n \n Extracting columns of data into a separate lookup table.", "breadcrumbs": "[\"The Datasette Ecosystem\"]", "references": "[{\"href\": \"https://sqlite-utils.datasette.io/\", \"label\": \"sqlite-utils\"}]"} {"id": "full_text_search:configuring-fts-by-hand", "page": "full_text_search", "ref": "configuring-fts-by-hand", "title": "Configuring FTS by hand", "content": "We recommend using sqlite-utils , but if you want to hand-roll a SQLite full-text search table you can do so using the following SQL. \n To enable full-text search for a table called items that works against the name and description columns, you would run this SQL to create a new items_fts FTS virtual table: \n CREATE VIRTUAL TABLE \"items_fts\" USING FTS4 (\n name,\n description,\n content=\"items\"\n); \n This creates a set of tables to power full-text search against items . The new items_fts table will be detected by Datasette as the fts_table for the items table. \n Creating the table is not enough: you also need to populate it with a copy of the data that you wish to make searchable. You can do that using the following SQL: \n INSERT INTO \"items_fts\" (rowid, name, description)\n SELECT rowid, name, description FROM items; \n If your table has columns that are foreign key references to other tables you can include that data in your full-text search index using a join. Imagine the items table has a foreign key column called category_id which refers to a categories table - you could create a full-text search table like this: \n CREATE VIRTUAL TABLE \"items_fts\" USING FTS4 (\n name,\n description,\n category_name,\n content=\"items\"\n); \n And then populate it like this: \n INSERT INTO \"items_fts\" (rowid, name, description, category_name)\n SELECT items.rowid,\n items.name,\n items.description,\n categories.name\n FROM items JOIN categories ON items.category_id=categories.id; \n You can use this technique to populate the full-text search index from any combination of tables and joins that makes sense for your project.", "breadcrumbs": "[\"Full-text search\", \"Enabling full-text search for a SQLite table\"]", "references": "[{\"href\": \"https://sqlite-utils.datasette.io/\", \"label\": \"sqlite-utils\"}]"} {"id": "facets:speeding-up-facets-with-indexes", "page": "facets", "ref": "speeding-up-facets-with-indexes", "title": "Speeding up facets with indexes", "content": "The performance of facets can be greatly improved by adding indexes on the columns you wish to facet by.\n Adding indexes can be performed using the sqlite3 command-line utility. Here's how to add an index on the state column in a table called Food_Trucks : \n sqlite3 mydatabase.db \n SQLite version 3.19.3 2017-06-27 16:48:08\nEnter \".help\" for usage hints.\nsqlite> CREATE INDEX Food_Trucks_state ON Food_Trucks(\"state\"); \n Or using the sqlite-utils command-line utility: \n sqlite-utils create-index mydatabase.db Food_Trucks state", "breadcrumbs": "[\"Facets\"]", "references": "[{\"href\": \"https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes\", \"label\": \"sqlite-utils\"}]"} {"id": "spatialite:making-use-of-a-spatial-index", "page": "spatialite", "ref": "making-use-of-a-spatial-index", "title": "Making use of a spatial index", "content": "SpatiaLite spatial indexes are R*Trees. They allow you to run efficient bounding box queries using a sub-select, with a similar pattern to that used for Searches using custom SQL . \n In the above example, the resulting index will be called idx_museums_point_geom . This takes the form of a SQLite virtual table. You can inspect its contents using the following query: \n select * from idx_museums_point_geom limit 10; \n Here's a live example: timezones-api.datasette.io/timezones/idx_timezones_Geometry \n \n \n \n \n \n \n \n \n \n \n pkid \n \n \n xmin \n \n \n xmax \n \n \n ymin \n \n \n ymax \n \n \n \n \n \n \n 1 \n \n \n -8.601725578308105 \n \n \n -2.4930307865142822 \n \n \n 4.162120819091797 \n \n \n 10.74019718170166 \n \n \n \n \n 2 \n \n \n -3.2607860565185547 \n \n \n 1.27329421043396 \n \n \n 4.539252281188965 \n \n \n 11.174856185913086 \n \n \n \n \n 3 \n \n \n 32.997581481933594 \n \n \n 47.98238754272461 \n \n \n 3.3974475860595703 \n \n \n 14.894054412841797 \n \n \n \n \n 4 \n \n \n -8.66890811920166 \n \n \n 11.997337341308594 \n \n \n 18.9681453704834 \n \n \n 37.296207427978516 \n \n \n \n \n 5 \n \n \n 36.43336486816406 \n \n \n 43.300174713134766 \n \n \n 12.354820251464844 \n \n \n 18.070993423461914 \n \n \n \n \n \n You can now construct efficient bounding box queries that will make use of the index like this: \n select * from museums where museums.rowid in (\n SELECT pkid FROM idx_museums_point_geom\n -- left-hand-edge of point > left-hand-edge of bbox (minx)\n where xmin > :bbox_minx\n -- right-hand-edge of point < right-hand-edge of bbox (maxx)\n and xmax < :bbox_maxx\n -- bottom-edge of point > bottom-edge of bbox (miny)\n and ymin > :bbox_miny\n -- top-edge of point < top-edge of bbox (maxy)\n and ymax < :bbox_maxy\n); \n Spatial indexes can be created against polygon columns as well as point columns, in which case they will represent the minimum bounding rectangle of that polygon. This is useful for accelerating within queries, as seen in the Timezones API example.", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[{\"href\": \"https://timezones-api.datasette.io/timezones/idx_timezones_Geometry\", \"label\": \"timezones-api.datasette.io/timezones/idx_timezones_Geometry\"}]"} {"id": "changelog:new-visual-design", "page": "changelog", "ref": "new-visual-design", "title": "New visual design", "content": "Datasette is no longer white and grey with blue and purple links! Natalie Downe has been working on a visual refresh, the first iteration of which is included in this release. ( #1056 )", "breadcrumbs": "[\"Changelog\", \"0.51 (2020-10-31)\"]", "references": "[{\"href\": \"https://twitter.com/natbat\", \"label\": \"Natalie Downe\"}, {\"href\": \"https://github.com/simonw/datasette/pull/1056\", \"label\": \"#1056\"}]"} {"id": "performance:http-caching", "page": "performance", "ref": "http-caching", "title": "HTTP caching", "content": "If your database is immutable and guaranteed not to change, you can gain major performance improvements from Datasette by enabling HTTP caching. \n This can work at two different levels. First, it can tell browsers to cache the results of queries and serve future requests from the browser cache. \n More significantly, it allows you to run Datasette behind a caching proxy such as Varnish or use a cache provided by a hosted service such as Fastly or Cloudflare . This can provide incredible speed-ups since a query only needs to be executed by Datasette the first time it is accessed - all subsequent hits can then be served by the cache. \n Using a caching proxy in this way could enable a Datasette-backed visualization to serve thousands of hits a second while running Datasette itself on extremely inexpensive hosting. \n Datasette's integration with HTTP caches can be enabled using a combination of configuration options and query string arguments. \n The default_cache_ttl setting sets the default HTTP cache TTL for all Datasette pages. This is 5 seconds unless you change it - you can set it to 0 if you wish to disable HTTP caching entirely. \n You can also change the cache timeout on a per-request basis using the ?_ttl=10 query string parameter. This can be useful when you are working with the Datasette JSON API - you may decide that a specific query can be cached for a longer time, or maybe you need to set ?_ttl=0 for some requests for example if you are running a SQL order by random() query.", "breadcrumbs": "[\"Performance and caching\"]", "references": "[{\"href\": \"https://varnish-cache.org/\", \"label\": \"Varnish\"}, {\"href\": \"https://www.fastly.com/\", \"label\": \"Fastly\"}, {\"href\": \"https://www.cloudflare.com/\", \"label\": \"Cloudflare\"}]"} {"id": "publish:publish-vercel", "page": "publish", "ref": "publish-vercel", "title": "Publishing to Vercel", "content": "Vercel - previously known as Zeit Now - provides a layer over AWS Lambda to allow for quick, scale-to-zero deployment. You can deploy Datasette instances to Vercel using the datasette-publish-vercel plugin. \n pip install datasette-publish-vercel\ndatasette publish vercel mydatabase.db --project my-database-project \n Not every feature is supported: consult the datasette-publish-vercel README for more details.", "breadcrumbs": "[\"Publishing data\", \"datasette publish\"]", "references": "[{\"href\": \"https://vercel.com/\", \"label\": \"Vercel\"}, {\"href\": \"https://github.com/simonw/datasette-publish-vercel\", \"label\": \"datasette-publish-vercel\"}, {\"href\": \"https://github.com/simonw/datasette-publish-vercel/blob/main/README.md\", \"label\": \"datasette-publish-vercel README\"}]"} {"id": "deploying:deploying-openrc", "page": "deploying", "ref": "deploying-openrc", "title": "Running Datasette using OpenRC", "content": "OpenRC is the service manager on non-systemd Linux distributions like Alpine Linux and Gentoo . \n Create an init script at /etc/init.d/datasette with the following contents: \n #!/sbin/openrc-run\n\nname=\"datasette\"\ncommand=\"datasette\"\ncommand_args=\"serve -h 0.0.0.0 /path/to/db.db\"\ncommand_background=true\npidfile=\"/run/${RC_SVCNAME}.pid\" \n You then need to configure the service to run at boot and start it: \n rc-update add datasette\nrc-service datasette start", "breadcrumbs": "[\"Deploying Datasette\"]", "references": "[{\"href\": \"https://www.alpinelinux.org/\", \"label\": \"Alpine Linux\"}, {\"href\": \"https://www.gentoo.org/\", \"label\": \"Gentoo\"}]"} {"id": "publish:cli-package", "page": "publish", "ref": "cli-package", "title": "datasette package", "content": "If you have docker installed (e.g. using Docker for Mac ) you can use the datasette package command to create a new Docker image in your local repository containing the datasette app bundled together with one or more SQLite databases: \n datasette package mydatabase.db \n Here's example output for the package command: \n datasette package parlgov.db --extra-options=\"--setting sql_time_limit_ms 2500\"\nSending build context to Docker daemon 4.459MB\nStep 1/7 : FROM python:3.11.0-slim-bullseye\n ---> 79e1dc9af1c1\nStep 2/7 : COPY . /app\n ---> Using cache\n ---> cd4ec67de656\nStep 3/7 : WORKDIR /app\n ---> Using cache\n ---> 139699e91621\nStep 4/7 : RUN pip install datasette\n ---> Using cache\n ---> 340efa82bfd7\nStep 5/7 : RUN datasette inspect parlgov.db --inspect-file inspect-data.json\n ---> Using cache\n ---> 5fddbe990314\nStep 6/7 : EXPOSE 8001\n ---> Using cache\n ---> 8e83844b0fed\nStep 7/7 : CMD datasette serve parlgov.db --port 8001 --inspect-file inspect-data.json --setting sql_time_limit_ms 2500\n ---> Using cache\n ---> 1bd380ea8af3\nSuccessfully built 1bd380ea8af3 \n You can now run the resulting container like so: \n docker run -p 8081:8001 1bd380ea8af3 \n This exposes port 8001 inside the container as port 8081 on your host machine, so you can access the application at http://localhost:8081/ \n You can customize the port that is exposed by the container using the --port option: \n datasette package mydatabase.db --port 8080 \n A full list of options can be seen by running datasette package --help : \n See datasette package for the full list of options for this command.", "breadcrumbs": "[\"Publishing data\"]", "references": "[{\"href\": \"https://www.docker.com/docker-mac\", \"label\": \"Docker for Mac\"}]"} {"id": "spatialite:id1", "page": "spatialite", "ref": "id1", "title": "SpatiaLite", "content": "The SpatiaLite module for SQLite adds features for handling geographic and spatial data. For an example of what you can do with it, see the tutorial Building a location to time zone API with SpatiaLite . \n To use it with Datasette, you need to install the mod_spatialite dynamic library. This can then be loaded into Datasette using the --load-extension command-line option. \n Datasette can look for SpatiaLite in common installation locations if you run it like this: \n datasette --load-extension=spatialite --setting default_allow_sql off \n If SpatiaLite is in another location, use the full path to the extension instead: \n datasette --setting default_allow_sql off \\\n --load-extension=/usr/local/lib/mod_spatialite.dylib", "breadcrumbs": "[]", "references": "[{\"href\": \"https://www.gaia-gis.it/fossil/libspatialite/index\", \"label\": \"SpatiaLite module\"}, {\"href\": \"https://datasette.io/tutorials/spatialite\", \"label\": \"Building a location to time zone API with SpatiaLite\"}]"} {"id": "changelog:improved-support-for-spatialite", "page": "changelog", "ref": "improved-support-for-spatialite", "title": "Improved support for SpatiaLite", "content": "The SpatiaLite module \n for SQLite adds robust geospatial features to the database. \n Getting SpatiaLite working can be tricky, especially if you want to use the most\n recent alpha version (with support for K-nearest neighbor). \n Datasette now includes extensive documentation on SpatiaLite , and thanks to Ravi Kotecha our GitHub\n repo includes a Dockerfile that can build\n the latest SpatiaLite and configure it for use with Datasette. \n The datasette publish and datasette package commands now accept a new\n --spatialite argument which causes them to install and configure SpatiaLite\n as part of the container they deploy.", "breadcrumbs": "[\"Changelog\", \"0.23 (2018-06-18)\"]", "references": "[{\"href\": \"https://www.gaia-gis.it/fossil/libspatialite/index\", \"label\": \"SpatiaLite module\"}, {\"href\": \"https://github.com/r4vi\", \"label\": \"Ravi Kotecha\"}, {\"href\": \"https://github.com/simonw/datasette/blob/master/Dockerfile\", \"label\": \"Dockerfile\"}]"} {"id": "spatialite:spatialite-warning", "page": "spatialite", "ref": "spatialite-warning", "title": "Warning", "content": "The SpatiaLite extension adds a large number of additional SQL functions , some of which are not be safe for untrusted users to execute: they may cause the Datasette server to crash. \n You should not expose a SpatiaLite-enabled Datasette instance to the public internet without taking extra measures to secure it against potentially harmful SQL queries. \n The following steps are recommended: \n \n \n Disable arbitrary SQL queries by untrusted users. See Controlling the ability to execute arbitrary SQL for ways to do this. The easiest is to start Datasette with the datasette --setting default_allow_sql off option. \n \n \n Define Canned queries with the SQL queries that use SpatiaLite functions that you want people to be able to execute. \n \n \n The Datasette SpatiaLite tutorial includes detailed instructions for running SpatiaLite safely using these techniques", "breadcrumbs": "[\"SpatiaLite\"]", "references": "[{\"href\": \"https://www.gaia-gis.it/gaia-sins/spatialite-sql-5.0.1.html\", \"label\": \"a large number of additional SQL functions\"}, {\"href\": \"https://datasette.io/tutorials/spatialite\", \"label\": \"Datasette SpatiaLite tutorial\"}]"} {"id": "publish:publish-heroku", "page": "publish", "ref": "publish-heroku", "title": "Publishing to Heroku", "content": "To publish your data using Heroku , first create an account there and install and configure the Heroku CLI tool . \n You can publish one or more databases to Heroku using the following command: \n datasette publish heroku mydatabase.db \n This will output some details about the new deployment, including a URL like this one: \n https://limitless-reef-88278.herokuapp.com/ deployed to Heroku \n You can specify a custom app name by passing -n my-app-name to the publish command. This will also allow you to overwrite an existing app. \n Rather than deploying directly you can use the --generate-dir option to output the files that would be deployed to a directory: \n datasette publish heroku mydatabase.db --generate-dir=/tmp/deploy-this-to-heroku \n See datasette publish heroku for the full list of options for this command.", "breadcrumbs": "[\"Publishing data\", \"datasette publish\"]", "references": "[{\"href\": \"https://www.heroku.com/\", \"label\": \"Heroku\"}, {\"href\": \"https://devcenter.heroku.com/articles/heroku-cli\", \"label\": \"Heroku CLI tool\"}]"} {"id": "deploying:deploying-buildpacks", "page": "deploying", "ref": "deploying-buildpacks", "title": "Deploying using buildpacks", "content": "Some hosting providers such as Heroku , DigitalOcean App Platform and Scalingo support the Buildpacks standard for deploying Python web applications. \n Deploying Datasette on these platforms requires two files: requirements.txt and Procfile . \n The requirements.txt file lets the platform know which Python packages should be installed. It should contain datasette at a minimum, but can also list any Datasette plugins you wish to install - for example: \n datasette\ndatasette-vega \n The Procfile lets the hosting platform know how to run the command that serves web traffic. It should look like this: \n web: datasette . -h 0.0.0.0 -p $PORT --cors \n The $PORT environment variable is provided by the hosting platform. --cors enables CORS requests from JavaScript running on other websites to your domain - omit this if you don't want to allow CORS. You can add additional Datasette Settings options here too. \n These two files should be enough to deploy Datasette on any host that supports buildpacks. Datasette will serve any SQLite files that are included in the root directory of the application. \n If you want to build SQLite files or download them as part of the deployment process you can do so using a bin/post_compile file. For example, the following bin/post_compile will download an example database that will then be served by Datasette: \n wget https://fivethirtyeight.datasettes.com/fivethirtyeight.db \n simonw/buildpack-datasette-demo is an example GitHub repository showing a Datasette configuration that can be deployed to a buildpack-supporting host.", "breadcrumbs": "[\"Deploying Datasette\"]", "references": "[{\"href\": \"https://www.heroku.com/\", \"label\": \"Heroku\"}, {\"href\": \"https://www.digitalocean.com/docs/app-platform/\", \"label\": \"DigitalOcean App Platform\"}, {\"href\": \"https://scalingo.com/\", \"label\": \"Scalingo\"}, {\"href\": \"https://buildpacks.io/\", \"label\": \"Buildpacks standard\"}, {\"href\": \"https://github.com/simonw/buildpack-datasette-demo\", \"label\": \"simonw/buildpack-datasette-demo\"}]"} {"id": "internals:internals-datasette-client", "page": "internals", "ref": "internals-datasette-client", "title": "datasette.client", "content": "Plugins can make internal simulated HTTP requests to the Datasette instance within which they are running. This ensures that all of Datasette's external JSON APIs are also available to plugins, while avoiding the overhead of making an external HTTP call to access those APIs. \n The datasette.client object is a wrapper around the HTTPX Python library , providing an async-friendly API that is similar to the widely used Requests library . \n It offers the following methods: \n \n \n await datasette.client.get(path, **kwargs) - returns HTTPX Response \n \n Execute an internal GET request against that path. \n \n \n \n await datasette.client.post(path, **kwargs) - returns HTTPX Response \n \n Execute an internal POST request. Use data={\"name\": \"value\"} to pass form parameters. \n \n \n \n await datasette.client.options(path, **kwargs) - returns HTTPX Response \n \n Execute an internal OPTIONS request. \n \n \n \n await datasette.client.head(path, **kwargs) - returns HTTPX Response \n \n Execute an internal HEAD request. \n \n \n \n await datasette.client.put(path, **kwargs) - returns HTTPX Response \n \n Execute an internal PUT request. \n \n \n \n await datasette.client.patch(path, **kwargs) - returns HTTPX Response \n \n Execute an internal PATCH request. \n \n \n \n await datasette.client.delete(path, **kwargs) - returns HTTPX Response \n \n Execute an internal DELETE request. \n \n \n \n await datasette.client.request(method, path, **kwargs) - returns HTTPX Response \n \n Execute an internal request with the given HTTP method against that path. \n \n \n \n These methods can be used with datasette.urls - for example: \n table_json = (\n await datasette.client.get(\n datasette.urls.table(\n \"fixtures\", \"facetable\", format=\"json\"\n )\n )\n).json() \n datasette.client methods automatically take the current base_url setting into account, whether or not you use the datasette.urls family of methods to construct the path. \n For documentation on available **kwargs options and the shape of the HTTPX Response object refer to the HTTPX Async documentation .", "breadcrumbs": "[\"Internals for plugins\", \"Datasette class\"]", "references": "[{\"href\": \"https://www.python-httpx.org/\", \"label\": \"HTTPX Python library\"}, {\"href\": \"https://requests.readthedocs.io/\", \"label\": \"Requests library\"}, {\"href\": \"https://www.python-httpx.org/async/\", \"label\": \"HTTPX Async documentation\"}]"} {"id": "testing_plugins:testing-datasette-client", "page": "testing_plugins", "ref": "testing-datasette-client", "title": "Using datasette.client in tests", "content": "The datasette.client mechanism is designed for use in tests. It provides access to a pre-configured HTTPX async client instance that can make GET, POST and other HTTP requests against a Datasette instance from inside a test. \n A simple test looks like this: \n @pytest.mark.asyncio\nasync def test_homepage():\n ds = Datasette(memory=True)\n response = await ds.client.get(\"/\")\n html = response.text\n assert \"

\" in html\n \n Or for a JSON API: \n @pytest.mark.asyncio\nasync def test_actor_is_null():\n ds = Datasette(memory=True)\n response = await ds.client.get(\"/-/actor.json\")\n assert response.json() == {\"actor\": None}\n \n To make requests as an authenticated actor, create a signed ds_cookie using the datasette.client.actor_cookie() helper function and pass it in cookies= like this: \n @pytest.mark.asyncio\nasync def test_signed_cookie_actor():\n ds = Datasette(memory=True)\n cookies = {\"ds_actor\": ds.client.actor_cookie({\"id\": \"root\"})}\n response = await ds.client.get(\"/-/actor.json\", cookies=cookies)\n assert response.json() == {\"actor\": {\"id\": \"root\"}}", "breadcrumbs": "[\"Testing plugins\"]", "references": "[{\"href\": \"https://www.python-httpx.org/async/\", \"label\": \"HTTPX async client\"}]"} {"id": "installation:installation-pip", "page": "installation", "ref": "installation-pip", "title": "Using pip", "content": "Datasette requires Python 3.8 or higher. The Python.org Python For Beginners page has instructions for getting started. \n You can install Datasette and its dependencies using pip : \n pip install datasette \n You can now run Datasette like so: \n datasette", "breadcrumbs": "[\"Installation\", \"Basic installation\"]", "references": "[{\"href\": \"https://www.python.org/about/gettingstarted/\", \"label\": \"Python.org Python For Beginners\"}]"} {"id": "full_text_search:id1", "page": "full_text_search", "ref": "id1", "title": "Full-text search", "content": "SQLite includes a powerful mechanism for enabling full-text search against SQLite records. Datasette can detect if a table has had full-text search configured for it in the underlying database and display a search interface for filtering that table. \n Here's an example search : \n \n Datasette automatically detects which tables have been configured for full-text search.", "breadcrumbs": "[]", "references": "[{\"href\": \"https://www.sqlite.org/fts3.html\", \"label\": \"a powerful mechanism for enabling full-text search\"}, {\"href\": \"https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper&_sort_desc=date\", \"label\": \"an example search\"}]"} {"id": "json_api:json-api-table-arguments", "page": "json_api", "ref": "json-api-table-arguments", "title": "Special table arguments", "content": "?_col=COLUMN1&_col=COLUMN2 \n \n List specific columns to display. These will be shown along with any primary keys. \n \n \n \n ?_nocol=COLUMN1&_nocol=COLUMN2 \n \n List specific columns to hide - any column not listed will be displayed. Primary keys cannot be hidden. \n \n \n \n ?_labels=on/off \n \n Expand foreign key references for every possible column. See below. \n \n \n \n ?_label=COLUMN1&_label=COLUMN2 \n \n Expand foreign key references for one or more specified columns. \n \n \n \n ?_size=1000 or ?_size=max \n \n Sets a custom page size. This cannot exceed the max_returned_rows limit\n passed to datasette serve . Use max to get max_returned_rows . \n \n \n \n ?_sort=COLUMN \n \n Sorts the results by the specified column. \n \n \n \n ?_sort_desc=COLUMN \n \n Sorts the results by the specified column in descending order. \n \n \n \n ?_search=keywords \n \n For SQLite tables that have been configured for\n full-text search executes a search\n with the provided keywords. \n \n \n \n ?_search_COLUMN=keywords \n \n Like _search= but allows you to specify the column to be searched, as\n opposed to searching all columns that have been indexed by FTS. \n \n \n \n ?_searchmode=raw \n \n With this option, queries passed to ?_search= or ?_search_COLUMN= will\n not have special characters escaped. This means you can make use of the full\n set of advanced SQLite FTS syntax ,\n though this could potentially result in errors if the wrong syntax is used. \n \n \n \n ?_where=SQL-fragment \n \n If the execute-sql permission is enabled, this parameter\n can be used to pass one or more additional SQL fragments to be used in the\n WHERE clause of the SQL used to query the table. \n This is particularly useful if you are building a JavaScript application\n that needs to do something creative but still wants the other conveniences\n provided by the table view (such as faceting) and hence would like not to\n have to construct a completely custom SQL query. \n Some examples: \n \n \n facetable?_where=_neighborhood like \"%c%\"&_where=_city_id=3 \n \n \n facetable?_where=_city_id in (select id from facet_cities where name != \"Detroit\") \n \n \n \n \n \n ?_through={json} \n \n This can be used to filter rows via a join against another table. \n The JSON parameter must include three keys: table , column and value . \n table must be a table that the current table is related to via a foreign key relationship. \n column must be a column in that other table. \n value is the value that you want to match against. \n For example, to filter roadside_attractions to just show the attractions that have a characteristic of \"museum\", you would construct this JSON: \n {\n \"table\": \"roadside_attraction_characteristics\",\n \"column\": \"characteristic_id\",\n \"value\": \"1\"\n} \n As a URL, that looks like this: \n ?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22} \n Here's an example . \n \n \n \n ?_next=TOKEN \n \n Pagination by continuation token - pass the token that was returned in the\n \"next\" property by the previous page. \n \n \n \n ?_facet=column \n \n Facet by column. Can be applied multiple times, see Facets . Only works on the default JSON output, not on any of the custom shapes. \n \n \n \n ?_facet_size=100 \n \n Increase the number of facet results returned for each facet. Use ?_facet_size=max for the maximum available size, determined by max_returned_rows . \n \n \n \n ?_nofacet=1 \n \n Disable all facets and facet suggestions for this page, including any defined by Facets in metadata . \n \n \n \n ?_nosuggest=1 \n \n Disable facet suggestions for this page. \n \n \n \n ?_nocount=1 \n \n Disable the select count(*) query used on this page - a count of None will be returned instead.", "breadcrumbs": "[\"JSON API\", \"Table arguments\"]", "references": "[{\"href\": \"https://www.sqlite.org/fts3.html\", \"label\": \"full-text search\"}, {\"href\": \"https://www.sqlite.org/fts5.html#full_text_query_syntax\", \"label\": \"advanced SQLite FTS syntax\"}, {\"href\": \"https://latest.datasette.io/fixtures/facetable?_where=_neighborhood%20like%20%22%c%%22&_where=_city_id=3\", \"label\": \"facetable?_where=_neighborhood like \\\"%c%\\\"&_where=_city_id=3\"}, {\"href\": \"https://latest.datasette.io/fixtures/facetable?_where=_city_id%20in%20(select%20id%20from%20facet_cities%20where%20name%20!=%20%22Detroit%22)\", \"label\": \"facetable?_where=_city_id in (select id from facet_cities where name != \\\"Detroit\\\")\"}, {\"href\": \"https://latest.datasette.io/fixtures/roadside_attractions?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22}\", \"label\": \"an example\"}]"} {"id": "full_text_search:full-text-search-enabling", "page": "full_text_search", "ref": "full-text-search-enabling", "title": "Enabling full-text search for a SQLite table", "content": "Datasette takes advantage of the external content mechanism in SQLite, which allows a full-text search virtual table to be associated with the contents of another SQLite table. \n To set up full-text search for a table, you need to do two things: \n \n \n Create a new FTS virtual table associated with your table \n \n \n Populate that FTS table with the data that you would like to be able to run searches against", "breadcrumbs": "[\"Full-text search\"]", "references": "[{\"href\": \"https://www.sqlite.org/fts3.html#_external_content_fts4_tables_\", \"label\": \"external content\"}]"} {"id": "full_text_search:full-text-search-advanced-queries", "page": "full_text_search", "ref": "full-text-search-advanced-queries", "title": "Advanced SQLite search queries", "content": "SQLite full-text search includes support for a variety of advanced queries , including AND , OR , NOT and NEAR . \n By default Datasette disables these features to ensure they do not cause errors or confusion for users who are not aware of them. You can disable this escaping and use the advanced queries by adding &_searchmode=raw to the table page query string. \n If you want to enable these operators by default for a specific table, you can do so by adding \"searchmode\": \"raw\" to the metadata configuration for that table, see Configuring full-text search for a table or view . \n If that option has been specified in the table metadata but you want to over-ride it and return to the default behavior you can append &_searchmode=escaped to the query string.", "breadcrumbs": "[\"Full-text search\"]", "references": "[{\"href\": \"https://www.sqlite.org/fts5.html#full_text_query_syntax\", \"label\": \"a variety of advanced queries\"}]"} {"id": "changelog:new-configuration-settings", "page": "changelog", "ref": "new-configuration-settings", "title": "New configuration settings", "content": "Datasette's Settings now also supports boolean settings. A number of new\n configuration options have been added: \n \n \n num_sql_threads - the number of threads used to execute SQLite queries. Defaults to 3. \n \n \n allow_facet - enable or disable custom Facets using the _facet= parameter. Defaults to on. \n \n \n suggest_facets - should Datasette suggest facets? Defaults to on. \n \n \n allow_download - should users be allowed to download the entire SQLite database? Defaults to on. \n \n \n allow_sql - should users be allowed to execute custom SQL queries? Defaults to on. \n \n \n default_cache_ttl - Default HTTP caching max-age header in seconds. Defaults to 365 days - caching can be disabled entirely by settings this to 0. \n \n \n cache_size_kb - Set the amount of memory SQLite uses for its per-connection cache , in KB. \n \n \n allow_csv_stream - allow users to stream entire result sets as a single CSV file. Defaults to on. \n \n \n max_csv_mb - maximum size of a returned CSV file in MB. Defaults to 100MB, set to 0 to disable this limit.", "breadcrumbs": "[\"Changelog\", \"0.23 (2018-06-18)\"]", "references": "[{\"href\": \"https://www.sqlite.org/pragma.html#pragma_cache_size\", \"label\": \"per-connection cache\"}]"} {"id": "settings:setting-cache-size-kb", "page": "settings", "ref": "setting-cache-size-kb", "title": "cache_size_kb", "content": "Sets the amount of memory SQLite uses for its per-connection cache , in KB. \n datasette mydatabase.db --setting cache_size_kb 5000", "breadcrumbs": "[\"Settings\", \"Settings\"]", "references": "[{\"href\": \"https://www.sqlite.org/pragma.html#pragma_cache_size\", \"label\": \"per-connection cache\"}]"} {"id": "plugin_hooks:plugin-hook-extra-css-urls", "page": "plugin_hooks", "ref": "plugin-hook-extra-css-urls", "title": "extra_css_urls(template, database, table, columns, view_name, request, datasette)", "content": "This takes the same arguments as extra_template_vars(...) \n Return a list of extra CSS URLs that should be included on the page. These can\n take advantage of the CSS class hooks described in Custom pages and templates . \n This can be a list of URLs: \n from datasette import hookimpl\n\n\n@hookimpl\ndef extra_css_urls():\n return [\n \"https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css\"\n ] \n Or a list of dictionaries defining both a URL and an\n SRI hash : \n @hookimpl\ndef extra_css_urls():\n return [\n {\n \"url\": \"https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css\",\n \"sri\": \"sha384-9gVQ4dYFwwWSjIDZnLEWnxCjeSWFphJiwGPXr1jddIhOegiu1FwO5qRGvFXOdJZ4\",\n }\n ] \n This function can also return an awaitable function, useful if it needs to run any async code: \n @hookimpl\ndef extra_css_urls(datasette):\n async def inner():\n db = datasette.get_database()\n results = await db.execute(\n \"select url from css_files\"\n )\n return [r[0] for r in results]\n\n return inner \n Examples: datasette-cluster-map , datasette-vega", "breadcrumbs": "[\"Plugin hooks\", \"Page extras\"]", "references": "[{\"href\": \"https://www.srihash.org/\", \"label\": \"SRI hash\"}, {\"href\": \"https://datasette.io/plugins/datasette-cluster-map\", \"label\": \"datasette-cluster-map\"}, {\"href\": \"https://datasette.io/plugins/datasette-vega\", \"label\": \"datasette-vega\"}]"} {"id": "configuration:configuration-reference-css-js", "page": "configuration", "ref": "configuration-reference-css-js", "title": "Custom CSS and JavaScript", "content": "Datasette can load additional CSS and JavaScript files, configured in datasette.yaml like this: \n [[[cog\nfrom metadata_doc import config_example\nconfig_example(cog, \"\"\"\n extra_css_urls:\n - https://simonwillison.net/static/css/all.bf8cd891642c.css\n extra_js_urls:\n - https://code.jquery.com/jquery-3.2.1.slim.min.js\n\"\"\") \n ]]] \n [[[end]]] \n The extra CSS and JavaScript files will be linked in the of every page: \n \n \n You can also specify a SRI (subresource integrity hash) for these assets: \n [[[cog\nconfig_example(cog, \"\"\"\n extra_css_urls:\n - url: https://simonwillison.net/static/css/all.bf8cd891642c.css\n sri: sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI\n extra_js_urls:\n - url: https://code.jquery.com/jquery-3.2.1.slim.min.js\n sri: sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g=\n\"\"\") \n ]]] \n [[[end]]] \n This will produce: \n \n \n Modern browsers will only execute the stylesheet or JavaScript if the SRI hash\n matches the content served. You can generate hashes using www.srihash.org \n Items in \"extra_js_urls\" can specify \"module\": true if they reference JavaScript that uses JavaScript modules . This configuration: \n [[[cog\nconfig_example(cog, \"\"\"\n extra_js_urls:\n - url: https://example.datasette.io/module.js\n module: true\n\"\"\") \n ]]] \n [[[end]]] \n Will produce this HTML: \n ", "breadcrumbs": "[\"Configuration\", null]", "references": "[{\"href\": \"https://www.srihash.org/\", \"label\": \"www.srihash.org\"}, {\"href\": \"https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules\", \"label\": \"JavaScript modules\"}]"} {"id": "changelog:id82", "page": "changelog", "ref": "id82", "title": "0.29.2 (2019-07-13)", "content": "Bumped Uvicorn to 0.8.4, fixing a bug where the query string was not included in the server logs. ( #559 ) \n \n \n Fixed bug where the navigation breadcrumbs were not displayed correctly on the page for a custom query. ( #558 ) \n \n \n Fixed bug where custom query names containing unicode characters caused errors.", "breadcrumbs": "[\"Changelog\"]", "references": "[{\"href\": \"https://www.uvicorn.org/\", \"label\": \"Uvicorn\"}, {\"href\": \"https://github.com/simonw/datasette/issues/559\", \"label\": \"#559\"}, {\"href\": \"https://github.com/simonw/datasette/issues/558\", \"label\": \"#558\"}]"}