{"rowid": 301, "title": "New plugin hook: extra_template_vars", "content": "The extra_template_vars(template, database, table, columns, view_name, request, datasette) plugin hook allows plugins to inject their own additional variables into the Datasette template context. This can be used in conjunction with custom templates to customize the Datasette interface. datasette-auth-github uses this hook to add custom HTML to the new top navigation bar (which is designed to be modified by plugins, see #540 ).", "sections_fts": 7, "rank": null} {"rowid": 302, "title": "Secret plugin configuration options", "content": "Plugins like datasette-auth-github need a safe way to set secret configuration options. Since the default mechanism for configuring plugins exposes those settings in /-/metadata a new mechanism was needed. Secret configuration values describes how plugins can now specify that their settings should be read from a file or an environment variable: \n {\n \"plugins\": {\n \"datasette-auth-github\": {\n \"client_secret\": {\n \"$env\": \"GITHUB_CLIENT_SECRET\"\n }\n }\n }\n} \n These plugin secrets can be set directly using datasette publish . See Custom metadata and plugins for details. ( #538 and #543 )", "sections_fts": 7, "rank": null} {"rowid": 303, "title": "Facet by date", "content": "If a column contains datetime values, Datasette can now facet that column by date. ( #481 )", "sections_fts": 7, "rank": null} {"rowid": 304, "title": "Easier custom templates for table rows", "content": "If you want to customize the display of individual table rows, you can do so using a _table.html template include that looks something like this: \n {% for row in display_rows %}\n
\n

{{ row[\"title\"] }}

\n

{{ row[\"description\"] }}\n

Category: {{ row.display(\"category_id\") }}

\n
\n{% endfor %} \n This is a backwards incompatible change . If you previously had a custom template called _rows_and_columns.html you need to rename it to _table.html . \n See Custom templates for full details.", "sections_fts": 7, "rank": null} {"rowid": 305, "title": "?_through= for joins through many-to-many tables", "content": "The new ?_through={json} argument to the Table view allows records to be filtered based on a many-to-many relationship. See Special table arguments for full documentation - here's an example . ( #355 ) \n This feature was added to help support facet by many-to-many , which isn't quite ready yet but will be coming in the next Datasette release.", "sections_fts": 7, "rank": null} {"rowid": 306, "title": "Small changes", "content": "Databases published using datasette publish now open in Immutable mode . ( #469 ) \n \n \n ?col__date= now works for columns containing spaces \n \n \n Automatic label detection (for deciding which column to show when linking to a foreign key) has been improved. ( #485 ) \n \n \n Fixed bug where pagination broke when combined with an expanded foreign key. ( #489 ) \n \n \n Contributors can now run pip install -e .[docs] to get all of the dependencies needed to build the documentation, including cd docs && make livehtml support. \n \n \n Datasette's dependencies are now all specified using the ~= match operator. ( #532 ) \n \n \n white-space: pre-wrap now used for table creation SQL. ( #505 ) \n \n \n Full list of commits between 0.28 and 0.29.", "sections_fts": 7, "rank": null} {"rowid": 307, "title": "0.28 (2019-05-19)", "content": "A salmagundi of new features!", "sections_fts": 7, "rank": null} {"rowid": 308, "title": "Supporting databases that change", "content": "From the beginning of the project, Datasette has been designed with read-only databases in mind. If a database is guaranteed not to change it opens up all kinds of interesting opportunities - from taking advantage of SQLite immutable mode and HTTP caching to bundling static copies of the database directly in a Docker container. The interesting ideas in Datasette explores this idea in detail. \n As my goals for the project have developed, I realized that read-only databases are no longer the right default. SQLite actually supports concurrent access very well provided only one thread attempts to write to a database at a time, and I keep encountering sensible use-cases for running Datasette on top of a database that is processing inserts and updates. \n So, as-of version 0.28 Datasette no longer assumes that a database file will not change. It is now safe to point Datasette at a SQLite database which is being updated by another process. \n Making this change was a lot of work - see tracking tickets #418 , #419 and #420 . It required new thinking around how Datasette should calculate table counts (an expensive operation against a large, changing database) and also meant reconsidering the \"content hash\" URLs Datasette has used in the past to optimize the performance of HTTP caches. \n Datasette can still run against immutable files and gains numerous performance benefits from doing so, but this is no longer the default behaviour. Take a look at the new Performance and caching documentation section for details on how to make the most of Datasette against data that you know will be staying read-only and immutable.", "sections_fts": 7, "rank": null} {"rowid": 309, "title": "Faceting improvements, and faceting plugins", "content": "Datasette Facets provide an intuitive way to quickly summarize and interact with data. Previously the only supported faceting technique was column faceting, but 0.28 introduces two powerful new capabilities: facet-by-JSON-array and the ability to define further facet types using plugins. \n Facet by array ( #359 ) is only available if your SQLite installation provides the json1 extension. Datasette will automatically detect columns that contain JSON arrays of values and offer a faceting interface against those columns - useful for modelling things like tags without needing to break them out into a new table. See Facet by JSON array for more. \n The new register_facet_classes() plugin hook ( #445 ) can be used to register additional custom facet classes. Each facet class should provide two methods: suggest() which suggests facet selections that might be appropriate for a provided SQL query, and facet_results() which executes a facet operation and returns results. Datasette's own faceting implementations have been refactored to use the same API as these plugins.", "sections_fts": 7, "rank": null} {"rowid": 310, "title": "datasette publish cloudrun", "content": "Google Cloud Run is a brand new serverless hosting platform from Google, which allows you to build a Docker container which will run only when HTTP traffic is received and will shut down (and hence cost you nothing) the rest of the time. It's similar to Zeit's Now v1 Docker hosting platform which sadly is no longer accepting signups from new users. \n The new datasette publish cloudrun command was contributed by Romain Primet ( #434 ) and publishes selected databases to a new Datasette instance running on Google Cloud Run. \n See Publishing to Google Cloud Run for full documentation.", "sections_fts": 7, "rank": null} {"rowid": 311, "title": "register_output_renderer plugins", "content": "Russ Garrett implemented a new Datasette plugin hook called register_output_renderer ( #441 ) which allows plugins to create additional output renderers in addition to Datasette's default .json and .csv . \n Russ's in-development datasette-geo plugin includes an example of this hook being used to output .geojson automatically converted from SpatiaLite.", "sections_fts": 7, "rank": null} {"rowid": 312, "title": "Medium changes", "content": "Datasette now conforms to the Black coding style ( #449 ) - and has a unit test to enforce this in the future \n \n \n \n \n New Special table arguments : \n \n \n \n ?columnname__in=value1,value2,value3 filter for executing SQL IN queries against a table, see Table arguments ( #433 ) \n \n \n ?columnname__date=yyyy-mm-dd filter which returns rows where the spoecified datetime column falls on the specified date ( 583b22a ) \n \n \n ?tags__arraycontains=tag filter which acts against a JSON array contained in a column ( 78e45ea ) \n \n \n ?_where=sql-fragment filter for the table view ( #429 ) \n \n \n ?_fts_table=mytable and ?_fts_pk=mycolumn query string options can be used to specify which FTS table to use for a search query - see Configuring full-text search for a table or view ( #428 ) \n \n \n \n \n \n \n \n You can now pass the same table filter multiple times - for example, ?content__not=world&content__not=hello will return all rows where the content column is neither hello or world ( #288 ) \n \n \n You can now specify about and about_url metadata (in addition to source and license ) linking to further information about a project - see Source, license and about \n \n \n New ?_trace=1 parameter now adds debug information showing every SQL query that was executed while constructing the page ( #435 ) \n \n \n datasette inspect now just calculates table counts, and does not introspect other database metadata ( #462 ) \n \n \n Removed /-/inspect page entirely - this will be replaced by something similar in the future, see #465 \n \n \n Datasette can now run against an in-memory SQLite database. You can do this by starting it without passing any files or by using the new --memory option to datasette serve . This can be useful for experimenting with SQLite queries that do not access any data, such as SELECT 1+1 or SELECT sqlite_version() .", "sections_fts": 7, "rank": null} {"rowid": 313, "title": "Small changes", "content": "We now show the size of the database file next to the download link ( #172 ) \n \n \n New /-/databases introspection page shows currently connected databases ( #470 ) \n \n \n Binary data is no longer displayed on the table and row pages ( #442 - thanks, Russ Garrett) \n \n \n New show/hide SQL links on custom query pages ( #415 ) \n \n \n The extra_body_script plugin hook now accepts an optional view_name argument ( #443 - thanks, Russ Garrett) \n \n \n Bumped Jinja2 dependency to 2.10.1 ( #426 ) \n \n \n All table filters are now documented, and documentation is enforced via unit tests ( 2c19a27 ) \n \n \n New project guideline: master should stay shippable at all times! ( 31f36e1 ) \n \n \n Fixed a bug where sqlite_timelimit() occasionally failed to clean up after itself ( bac4e01 ) \n \n \n We no longer load additional plugins when executing pytest ( #438 ) \n \n \n Homepage now links to database views if there are less than five tables in a database ( #373 ) \n \n \n The --cors option is now respected by error pages ( #453 ) \n \n \n datasette publish heroku now uses the --include-vcs-ignore option, which means it works under Travis CI ( #407 ) \n \n \n datasette publish heroku now publishes using Python 3.6.8 ( 666c374 ) \n \n \n Renamed datasette publish now to datasette publish nowv1 ( #472 ) \n \n \n datasette publish nowv1 now accepts multiple --alias parameters ( 09ef305 ) \n \n \n Removed the datasette skeleton command ( #476 ) \n \n \n The documentation on how to build the documentation now recommends sphinx-autobuild", "sections_fts": 7, "rank": null} {"rowid": 314, "title": "0.27.1 (2019-05-09)", "content": "Tiny bugfix release: don't install tests/ in the wrong place. Thanks, Veit Heller.", "sections_fts": 7, "rank": null} {"rowid": 315, "title": "0.27 (2019-01-31)", "content": "New command: datasette plugins ( documentation ) shows you the currently installed list of plugins. \n \n \n Datasette can now output newline-delimited JSON using the new ?_shape=array&_nl=on query string option. \n \n \n Added documentation on The Datasette Ecosystem . \n \n \n Now using Python 3.7.2 as the base for the official Datasette Docker image .", "sections_fts": 7, "rank": null} {"rowid": 316, "title": "0.26.1 (2019-01-10)", "content": "/-/versions now includes SQLite compile_options ( #396 ) \n \n \n datasetteproject/datasette Docker image now uses SQLite 3.26.0 ( #397 ) \n \n \n Cleaned up some deprecation warnings under Python 3.7", "sections_fts": 7, "rank": null} {"rowid": 317, "title": "0.26 (2019-01-02)", "content": "datasette serve --reload now restarts Datasette if a database file changes on disk. \n \n \n datasette publish now now takes an optional --alias mysite.now.sh argument. This will attempt to set an alias after the deploy completes. \n \n \n Fixed a bug where the advanced CSV export form failed to include the currently selected filters ( #393 )", "sections_fts": 7, "rank": null} {"rowid": 318, "title": "0.25.2 (2018-12-16)", "content": "datasette publish heroku now uses the python-3.6.7 runtime \n \n \n Added documentation on how to build the documentation \n \n \n Added documentation covering our release process \n \n \n Upgraded to pytest 4.0.2", "sections_fts": 7, "rank": null} {"rowid": 319, "title": "0.25.1 (2018-11-04)", "content": "Documentation improvements plus a fix for publishing to Zeit Now. \n \n \n datasette publish now now uses Zeit's v1 platform, to work around the new 100MB image limit. Thanks, @slygent - closes #366 .", "sections_fts": 7, "rank": null} {"rowid": 320, "title": "0.25 (2018-09-19)", "content": "New plugin hooks, improved database view support and an easier way to use more recent versions of SQLite. \n \n \n New publish_subcommand plugin hook. A plugin can now add additional datasette publish publishers in addition to the default now and heroku , both of which have been refactored into default plugins. publish_subcommand documentation . Closes #349 \n \n \n New render_cell plugin hook. Plugins can now customize how values are displayed in the HTML tables produced by Datasette's browsable interface. datasette-json-html and datasette-render-images are two new plugins that use this hook. render_cell documentation . Closes #352 \n \n \n New extra_body_script plugin hook, enabling plugins to provide additional JavaScript that should be added to the page footer. extra_body_script documentation . \n \n \n extra_css_urls and extra_js_urls hooks now take additional optional parameters, allowing them to be more selective about which pages they apply to. Documentation . \n \n \n You can now use the sortable_columns metadata setting to explicitly enable sort-by-column in the interface for database views, as well as for specific tables. \n \n \n The new fts_table and fts_pk metadata settings can now be used to explicitly configure full-text search for a table or a view , even if that table is not directly coupled to the SQLite FTS feature in the database schema itself. \n \n \n Datasette will now use pysqlite3 in place of the standard library sqlite3 module if it has been installed in the current environment. This makes it much easier to run Datasette against a more recent version of SQLite, including the just-released SQLite 3.25.0 which adds window function support. More details on how to use this in #360 \n \n \n New mechanism that allows plugin configuration options to be set using metadata.json .", "sections_fts": 7, "rank": null} {"rowid": 321, "title": "0.24 (2018-07-23)", "content": "A number of small new features: \n \n \n datasette publish heroku now supports --extra-options , fixes #334 \n \n \n Custom error message if SpatiaLite is needed for specified database, closes #331 \n \n \n New config option: truncate_cells_html for truncating long cell values in HTML view - closes #330 \n \n \n Documentation for datasette publish and datasette package , closes #337 \n \n \n Fixed compatibility with Python 3.7 \n \n \n datasette publish heroku now supports app names via the -n option, which can also be used to overwrite an existing application [Russ Garrett] \n \n \n Title and description metadata can now be set for canned SQL queries , closes #342 \n \n \n New force_https_on config option, fixes https:// API URLs when deploying to Zeit Now - closes #333 \n \n \n ?_json_infinity=1 query string argument for handling Infinity/-Infinity values in JSON, closes #332 \n \n \n URLs displayed in the results of custom SQL queries are now URLified, closes #298", "sections_fts": 7, "rank": null} {"rowid": 322, "title": "0.23.2 (2018-07-07)", "content": "Minor bugfix and documentation release. \n \n \n CSV export now respects --cors , fixes #326 \n \n \n Installation instructions , including docker image - closes #328 \n \n \n Fix for row pages for tables with / in, closes #325", "sections_fts": 7, "rank": null} {"rowid": 323, "title": "0.23.1 (2018-06-21)", "content": "Minor bugfix release. \n \n \n Correctly display empty strings in HTML table, closes #314 \n \n \n Allow \".\" in database filenames, closes #302 \n \n \n 404s ending in slash redirect to remove that slash, closes #309 \n \n \n Fixed incorrect display of compound primary keys with foreign key\n references. Closes #319 \n \n \n Docs + example of canned SQL query using || concatenation. Closes #321 \n \n \n Correctly display facets with value of 0 - closes #318 \n \n \n Default 'expand labels' to checked in CSV advanced export", "sections_fts": 7, "rank": null} {"rowid": 324, "title": "0.23 (2018-06-18)", "content": "This release features CSV export, improved options for foreign key expansions,\n new configuration settings and improved support for SpatiaLite. \n See datasette/compare/0.22.1...0.23 for a full list of\n commits added since the last release.", "sections_fts": 7, "rank": null} {"rowid": 325, "title": "CSV export", "content": "Any Datasette table, view or custom SQL query can now be exported as CSV. \n \n Check out the CSV export documentation for more details, or\n try the feature out on\n https://fivethirtyeight.datasettes.com/fivethirtyeight/bechdel%2Fmovies \n If your table has more than max_returned_rows (default 1,000)\n Datasette provides the option to stream all rows . This option takes advantage\n of async Python and Datasette's efficient pagination to\n iterate through the entire matching result set and stream it back as a\n downloadable CSV file.", "sections_fts": 7, "rank": null} {"rowid": 326, "title": "Foreign key expansions", "content": "When Datasette detects a foreign key reference it attempts to resolve a label\n for that reference (automatically or using the Specifying the label column for a table metadata\n option) so it can display a link to the associated row. \n This expansion is now also available for JSON and CSV representations of the\n table, using the new _labels=on query string option. See\n Expanding foreign key references for more details.", "sections_fts": 7, "rank": null} {"rowid": 327, "title": "New configuration settings", "content": "Datasette's Settings now also supports boolean settings. A number of new\n configuration options have been added: \n \n \n num_sql_threads - the number of threads used to execute SQLite queries. Defaults to 3. \n \n \n allow_facet - enable or disable custom Facets using the _facet= parameter. Defaults to on. \n \n \n suggest_facets - should Datasette suggest facets? Defaults to on. \n \n \n allow_download - should users be allowed to download the entire SQLite database? Defaults to on. \n \n \n allow_sql - should users be allowed to execute custom SQL queries? Defaults to on. \n \n \n default_cache_ttl - Default HTTP caching max-age header in seconds. Defaults to 365 days - caching can be disabled entirely by settings this to 0. \n \n \n cache_size_kb - Set the amount of memory SQLite uses for its per-connection cache , in KB. \n \n \n allow_csv_stream - allow users to stream entire result sets as a single CSV file. Defaults to on. \n \n \n max_csv_mb - maximum size of a returned CSV file in MB. Defaults to 100MB, set to 0 to disable this limit.", "sections_fts": 7, "rank": null} {"rowid": 328, "title": "Control HTTP caching with ?_ttl=", "content": "You can now customize the HTTP max-age header that is sent on a per-URL basis, using the new ?_ttl= query string parameter. \n You can set this to any value in seconds, or you can set it to 0 to disable HTTP caching entirely. \n Consider for example this query which returns a randomly selected member of the Avengers: \n select * from [avengers/avengers] order by random() limit 1 \n If you hit the following page repeatedly you will get the same result, due to HTTP caching: \n /fivethirtyeight?sql=select+*+from+%5Bavengers%2Favengers%5D+order+by+random%28%29+limit+1 \n By adding ?_ttl=0 to the zero you can ensure the page will not be cached and get back a different super hero every time: \n /fivethirtyeight?sql=select+*+from+%5Bavengers%2Favengers%5D+order+by+random%28%29+limit+1&_ttl=0", "sections_fts": 7, "rank": null} {"rowid": 329, "title": "Improved support for SpatiaLite", "content": "The SpatiaLite module \n for SQLite adds robust geospatial features to the database. \n Getting SpatiaLite working can be tricky, especially if you want to use the most\n recent alpha version (with support for K-nearest neighbor). \n Datasette now includes extensive documentation on SpatiaLite , and thanks to Ravi Kotecha our GitHub\n repo includes a Dockerfile that can build\n the latest SpatiaLite and configure it for use with Datasette. \n The datasette publish and datasette package commands now accept a new\n --spatialite argument which causes them to install and configure SpatiaLite\n as part of the container they deploy.", "sections_fts": 7, "rank": null} {"rowid": 330, "title": "latest.datasette.io", "content": "Every commit to Datasette master is now automatically deployed by Travis CI to\n https://latest.datasette.io/ - ensuring there is always a live demo of the\n latest version of the software. \n The demo uses the fixtures from our\n unit tests, ensuring it demonstrates the same range of functionality that is\n covered by the tests. \n You can see how the deployment mechanism works in our .travis.yml file.", "sections_fts": 7, "rank": null} {"rowid": 331, "title": "Miscellaneous", "content": "Got JSON data in one of your columns? Use the new ?_json=COLNAME argument\n to tell Datasette to return that JSON value directly rather than encoding it\n as a string. \n \n \n If you just want an array of the first value of each row, use the new\n ?_shape=arrayfirst option - example .", "sections_fts": 7, "rank": null} {"rowid": 332, "title": "0.22.1 (2018-05-23)", "content": "Bugfix release, plus we now use versioneer for our version numbers. \n \n \n Faceting no longer breaks pagination, fixes #282 \n \n \n Add __version_info__ derived from __version__ [Robert Gieseke] \n This might be tuple of more than two values (major and minor\n version) if commits have been made after a release. \n \n \n Add version number support with Versioneer. [Robert Gieseke] \n Versioneer Licence:\n Public Domain (CC0-1.0) \n Closes #273 \n \n \n Refactor inspect logic [Russ Garrett]", "sections_fts": 7, "rank": null} {"rowid": 333, "title": "0.22 (2018-05-20)", "content": "The big new feature in this release is Facets . Datasette can now apply faceted browse to any column in any table. It will also suggest possible facets. See the Datasette Facets announcement post for more details. \n In addition to the work on facets: \n \n \n Added docs for introspection endpoints \n \n \n New --config option, added --help-config , closes #274 \n Removed the --page_size= argument to datasette serve in favour of: \n datasette serve --config default_page_size:50 mydb.db \n Added new help section: \n datasette --help-config \n Config options:\n default_page_size Default page size for the table view\n (default=100)\n max_returned_rows Maximum rows that can be returned from a table\n or custom query (default=1000)\n sql_time_limit_ms Time limit for a SQL query in milliseconds\n (default=1000)\n default_facet_size Number of values to return for requested facets\n (default=30)\n facet_time_limit_ms Time limit for calculating a requested facet\n (default=200)\n facet_suggest_time_limit_ms Time limit for calculating a suggested facet\n (default=50) \n \n \n Only apply responsive table styles to .rows-and-column \n Otherwise they interfere with tables in the description, e.g. on\n https://fivethirtyeight.datasettes.com/fivethirtyeight/nba-elo%2Fnbaallelo \n \n \n Refactored views into new views/ modules, refs #256 \n \n \n Documentation for SQLite full-text search support, closes #253 \n \n \n /-/versions now includes SQLite fts_versions , closes #252", "sections_fts": 7, "rank": null} {"rowid": 334, "title": "0.21 (2018-05-05)", "content": "New JSON _shape= options, the ability to set table _size= and a mechanism for searching within specific columns. \n \n \n Default tests to using a longer timelimit \n Every now and then a test will fail in Travis CI on Python 3.5 because it hit\n the default 20ms SQL time limit. \n Test fixtures now default to a 200ms time limit, and we only use the 20ms time\n limit for the specific test that tests query interruption. This should make\n our tests on Python 3.5 in Travis much more stable. \n \n \n Support _search_COLUMN=text searches, closes #237 \n \n \n Show version on /-/plugins page, closes #248 \n \n \n ?_size=max option, closes #249 \n \n \n Added /-/versions and /-/versions.json , closes #244 \n Sample output: \n {\n \"python\": {\n \"version\": \"3.6.3\",\n \"full\": \"3.6.3 (default, Oct 4 2017, 06:09:38) \\n[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]\"\n },\n \"datasette\": {\n \"version\": \"0.20\"\n },\n \"sqlite\": {\n \"version\": \"3.23.1\",\n \"extensions\": {\n \"json1\": null,\n \"spatialite\": \"4.3.0a\"\n }\n }\n} \n \n \n Renamed ?_sql_time_limit_ms= to ?_timelimit , closes #242 \n \n \n New ?_shape=array option + tweaks to _shape , closes #245 \n \n \n Default is now ?_shape=arrays (renamed from lists ) \n \n \n New ?_shape=array returns an array of objects as the root object \n \n \n Changed ?_shape=object to return the object as the root \n \n \n Updated docs \n \n \n \n \n FTS tables now detected by inspect() , closes #240 \n \n \n New ?_size=XXX query string parameter for table view, closes #229 \n Also added documentation for all of the _special arguments. \n Plus deleted some duplicate logic implementing _group_count . \n \n \n If max_returned_rows==page_size , increment max_returned_rows - fixes #230 \n \n \n New hidden: True option for table metadata, closes #239 \n \n \n Hide idx_* tables if spatialite detected, closes #228 \n \n \n Added class=rows-and-columns to custom query results table \n \n \n Added CSS class rows-and-columns to main table \n \n \n label_column option in metadata.json - closes #234", "sections_fts": 7, "rank": null} {"rowid": 335, "title": "0.20 (2018-04-20)", "content": "Mostly new work on the Plugins mechanism: plugins can now bundle static assets and custom templates, and datasette publish has a new --install=name-of-plugin option. \n \n \n Add col-X classes to HTML table on custom query page \n \n \n Fixed out-dated template in documentation \n \n \n Plugins can now bundle custom templates, #224 \n \n \n Added /-/metadata /-/plugins /-/inspect, #225 \n \n \n Documentation for --install option, refs #223 \n \n \n Datasette publish/package --install option, #223 \n \n \n Fix for plugins in Python 3.5, #222 \n \n \n New plugin hooks: extra_css_urls() and extra_js_urls(), #214 \n \n \n /-/static-plugins/PLUGIN_NAME/ now serves static/ from plugins \n \n \n now gets class=\"col-X\" - plus added col-X documentation \n \n \n Use to_css_class for table cell column classes \n This ensures that columns with spaces in the name will still\n generate usable CSS class names. Refs #209 \n \n \n Add column name classes to s, make PK bold [Russ Garrett] \n \n \n Don't duplicate simple primary keys in the link column [Russ Garrett] \n When there's a simple (single-column) primary key, it looks weird to\n duplicate it in the link column. \n This change removes the second PK column and treats the link column as\n if it were the PK column from a header/sorting perspective. \n \n \n Correct escaping for HTML display of row links [Russ Garrett] \n \n \n Longer time limit for test_paginate_compound_keys \n It was failing intermittently in Travis - see #209 \n \n \n Use application/octet-stream for downloadable databases \n \n \n Updated PyPI classifiers \n \n \n Updated PyPI link to pypi.org", "sections_fts": 7, "rank": null} {"rowid": 336, "title": "0.19 (2018-04-16)", "content": "This is the first preview of the new Datasette plugins mechanism. Only two\n plugin hooks are available so far - for custom SQL functions and custom template\n filters. There's plenty more to come - read the documentation and get involved in\n the tracking ticket if you\n have feedback on the direction so far. \n \n \n Fix for _sort_desc=sortable_with_nulls test, refs #216 \n \n \n Fixed #216 - paginate correctly when sorting by nullable column \n \n \n Initial documentation for plugins, closes #213 \n https://docs.datasette.io/en/stable/plugins.html \n \n \n New --plugins-dir=plugins/ option ( #212 ) \n New option causing Datasette to load and evaluate all of the Python files in\n the specified directory and register any plugins that are defined in those\n files. \n This new option is available for the following commands: \n datasette serve mydb.db --plugins-dir=plugins/\ndatasette publish now/heroku mydb.db --plugins-dir=plugins/\ndatasette package mydb.db --plugins-dir=plugins/ \n \n \n Start of the plugin system, based on pluggy ( #210 ) \n Uses https://pluggy.readthedocs.io/ originally created for the py.test project \n We're starting with two plugin hooks: \n prepare_connection(conn) \n This is called when a new SQLite connection is created. It can be used to register custom SQL functions. \n prepare_jinja2_environment(env) \n This is called with the Jinja2 environment. It can be used to register custom template tags and filters. \n An example plugin which uses these two hooks can be found at https://github.com/simonw/datasette-plugin-demos or installed using pip install datasette-plugin-demos \n Refs #14 \n \n \n Return HTTP 405 on InvalidUsage rather than 500. [Russ Garrett] \n This also stops it filling up the logs. This happens for HEAD requests\n at the moment - which perhaps should be handled better, but that's a\n different issue.", "sections_fts": 7, "rank": null} {"rowid": 337, "title": "0.18 (2018-04-14)", "content": "This release introduces support for units ,\n contributed by Russ Garrett ( #203 ).\n You can now optionally specify the units for specific columns using metadata.json .\n Once specified, units will be displayed in the HTML view of your table. They also become\n available for use in filters - if a column is configured with a unit of distance, you can\n request all rows where that column is less than 50 meters or more than 20 feet for example. \n \n \n Link foreign keys which don't have labels. [Russ Garrett] \n This renders unlabeled FKs as simple links. \n Also includes bonus fixes for two minor issues: \n \n \n In foreign key link hrefs the primary key was escaped using HTML\n escaping rather than URL escaping. This broke some non-integer PKs. \n \n \n Print tracebacks to console when handling 500 errors. \n \n \n \n \n Fix SQLite error when loading rows with no incoming FKs. [Russ\n Garrett] \n This fixes an error caused by an invalid query when loading incoming FKs. \n The error was ignored due to async but it still got printed to the\n console. \n \n \n Allow custom units to be registered with Pint. [Russ Garrett] \n \n \n Support units in filters. [Russ Garrett] \n \n \n Tidy up units support. [Russ Garrett] \n \n \n Add units to exported JSON \n \n \n Units key in metadata skeleton \n \n \n Docs \n \n \n \n \n Initial units support. [Russ Garrett] \n Add support for specifying units for a column in metadata.json and\n rendering them on display using\n pint", "sections_fts": 7, "rank": null} {"rowid": 338, "title": "0.17 (2018-04-13)", "content": "Release 0.17 to fix issues with PyPI", "sections_fts": 7, "rank": null} {"rowid": 339, "title": "0.16 (2018-04-13)", "content": "Better mechanism for handling errors; 404s for missing table/database \n New error mechanism closes #193 \n 404s for missing tables/databases closes #184 \n \n \n long_description in markdown for the new PyPI \n \n \n Hide SpatiaLite system tables. [Russ Garrett] \n \n \n Allow explain select / explain query plan select #201 \n \n \n Datasette inspect now finds primary_keys #195 \n \n \n Ability to sort using form fields (for mobile portrait mode) #199 \n We now display sort options as a select box plus a descending checkbox, which\n means you can apply sort orders even in portrait mode on a mobile phone where\n the column headers are hidden.", "sections_fts": 7, "rank": null} {"rowid": 340, "title": "0.15 (2018-04-09)", "content": "The biggest new feature in this release is the ability to sort by column. On the\n table page the column headers can now be clicked to apply sort (or descending\n sort), or you can specify ?_sort=column or ?_sort_desc=column directly\n in the URL. \n \n \n table_rows => table_rows_count , filtered_table_rows =>\n filtered_table_rows_count \n Renamed properties. Closes #194 \n \n \n New sortable_columns option in metadata.json to control sort options. \n You can now explicitly set which columns in a table can be used for sorting\n using the _sort and _sort_desc arguments using metadata.json : \n {\n \"databases\": {\n \"database1\": {\n \"tables\": {\n \"example_table\": {\n \"sortable_columns\": [\n \"height\",\n \"weight\"\n ]\n }\n }\n }\n }\n} \n Refs #189 \n \n \n Column headers now link to sort/desc sort - refs #189 \n \n \n _sort and _sort_desc parameters for table views \n Allows for paginated sorted results based on a specified column. \n Refs #189 \n \n \n Total row count now correct even if _next applied \n \n \n Use .custom_sql() for _group_count implementation (refs #150 ) \n \n \n Make HTML title more readable in query template ( #180 ) [Ryan Pitts] \n \n \n New ?_shape=objects/object/lists param for JSON API ( #192 ) \n New _shape= parameter replacing old .jsono extension \n Now instead of this: \n /database/table.jsono \n We use the _shape parameter like this: \n /database/table.json?_shape=objects \n Also introduced a new _shape called object which looks like this: \n /database/table.json?_shape=object \n Returning an object for the rows key: \n ...\n\"rows\": {\n \"pk1\": {\n ...\n },\n \"pk2\": {\n ...\n }\n} \n Refs #122 \n \n \n Utility for writing test database fixtures to a .db file \n python tests/fixtures.py /tmp/hello.db \n This is useful for making a SQLite database of the test fixtures for\n interactive exploration. \n \n \n Compound primary key _next= now plays well with extra filters \n Closes #190 \n \n \n Fixed bug with keyset pagination over compound primary keys \n Refs #190 \n \n \n Database/Table views inherit source/license/source_url/license_url \n metadata \n If you set the source_url/license_url/source/license fields in your root\n metadata those values will now be inherited all the way down to the database\n and table templates. \n The title/description are NOT inherited. \n Also added unit tests for the HTML generated by the metadata. \n Refs #185 \n \n \n Add metadata, if it exists, to heroku temp dir ( #178 ) [Tony Hirst] \n \n \n Initial documentation for pagination \n \n \n Broke up test_app into test_api and test_html \n \n \n Fixed bug with .json path regular expression \n I had a table called geojson and it caused an exception because the regex\n was matching .json and not \\.json \n \n \n Deploy to Heroku with Python 3.6.3", "sections_fts": 7, "rank": null} {"rowid": 341, "title": "0.14 (2017-12-09)", "content": "The theme of this release is customization: Datasette now allows every aspect\n of its presentation to be customized \n either using additional CSS or by providing entirely new templates. \n Datasette's metadata.json format \n has also been expanded, to allow per-database and per-table metadata. A new\n datasette skeleton command can be used to generate a skeleton JSON file\n ready to be filled in with per-database and per-table details. \n The metadata.json file can also be used to define\n canned queries ,\n as a more powerful alternative to SQL views. \n \n \n extra_css_urls / extra_js_urls in metadata \n A mechanism in the metadata.json format for adding custom CSS and JS urls. \n Create a metadata.json file that looks like this: \n {\n \"extra_css_urls\": [\n \"https://simonwillison.net/static/css/all.bf8cd891642c.css\"\n ],\n \"extra_js_urls\": [\n \"https://code.jquery.com/jquery-3.2.1.slim.min.js\"\n ]\n} \n Then start datasette like this: \n datasette mydb.db --metadata=metadata.json \n The CSS and JavaScript files will be linked in the of every page. \n You can also specify a SRI (subresource integrity hash) for these assets: \n {\n \"extra_css_urls\": [\n {\n \"url\": \"https://simonwillison.net/static/css/all.bf8cd891642c.css\",\n \"sri\": \"sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI\"\n }\n ],\n \"extra_js_urls\": [\n {\n \"url\": \"https://code.jquery.com/jquery-3.2.1.slim.min.js\",\n \"sri\": \"sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g=\"\n }\n ]\n} \n Modern browsers will only execute the stylesheet or JavaScript if the SRI hash\n matches the content served. You can generate hashes using https://www.srihash.org/ \n \n \n Auto-link column values that look like URLs ( #153 ) \n \n \n CSS styling hooks as classes on the body ( #153 ) \n Every template now gets CSS classes in the body designed to support custom\n styling. \n The index template (the top level page at / ) gets this: \n \n The database template ( /dbname/ ) gets this: \n \n The table template ( /dbname/tablename ) gets: \n \n The row template ( /dbname/tablename/rowid ) gets: \n \n The db-x and table-x classes use the database or table names themselves IF\n they are valid CSS identifiers. If they aren't, we strip any invalid\n characters out and append a 6 character md5 digest of the original name, in\n order to ensure that multiple tables which resolve to the same stripped\n character version still have different CSS classes. \n Some examples (extracted from the unit tests): \n \"simple\" => \"simple\"\n\"MixedCase\" => \"MixedCase\"\n\"-no-leading-hyphens\" => \"no-leading-hyphens-65bea6\"\n\"_no-leading-underscores\" => \"no-leading-underscores-b921bc\"\n\"no spaces\" => \"no-spaces-7088d7\"\n\"-\" => \"336d5e\"\n\"no $ characters\" => \"no--characters-59e024\" \n \n \n datasette --template-dir=mytemplates/ argument \n You can now pass an additional argument specifying a directory to look for\n custom templates in. \n Datasette will fall back on the default templates if a template is not\n found in that directory. \n \n \n Ability to over-ride templates for individual tables/databases. \n It is now possible to over-ride templates on a per-database / per-row or per-\n table basis. \n When you access e.g. /mydatabase/mytable Datasette will look for the following: \n - table-mydatabase-mytable.html\n- table.html \n If you provided a --template-dir argument to datasette serve it will look in\n that directory first. \n The lookup rules are as follows: \n Index page (/):\n index.html\n\nDatabase page (/mydatabase):\n database-mydatabase.html\n database.html\n\nTable page (/mydatabase/mytable):\n table-mydatabase-mytable.html\n table.html\n\nRow page (/mydatabase/mytable/id):\n row-mydatabase-mytable.html\n row.html \n If a table name has spaces or other unexpected characters in it, the template\n filename will follow the same rules as our custom CSS classes\n - for example, a table called \"Food Trucks\"\n will attempt to load the following templates: \n table-mydatabase-Food-Trucks-399138.html\ntable.html \n It is possible to extend the default templates using Jinja template\n inheritance. If you want to customize EVERY row template with some additional\n content you can do so by creating a row.html template like this: \n {% extends \"default:row.html\" %}\n\n{% block content %}\n

EXTRA HTML AT THE TOP OF THE CONTENT BLOCK

\n

This line renders the original block:

\n{{ super() }}\n{% endblock %} \n \n \n --static option for datasette serve ( #160 ) \n You can now tell Datasette to serve static files from a specific location at a\n specific mountpoint. \n For example: \n datasette serve mydb.db --static extra-css:/tmp/static/css \n Now if you visit this URL: \n http://localhost:8001/extra-css/blah.css \n The following file will be served: \n /tmp/static/css/blah.css \n \n \n Canned query support. \n Named canned queries can now be defined in metadata.json like this: \n {\n \"databases\": {\n \"timezones\": {\n \"queries\": {\n \"timezone_for_point\": \"select tzid from timezones ...\"\n }\n }\n }\n} \n These will be shown in a new \"Queries\" section beneath \"Views\" on the database page. \n \n \n New datasette skeleton command for generating metadata.json ( #164 ) \n \n \n metadata.json support for per-table/per-database metadata ( #165 ) \n Also added support for descriptions and HTML descriptions. \n Here's an example metadata.json file illustrating custom per-database and per-\n table metadata: \n {\n \"title\": \"Overall datasette title\",\n \"description_html\": \"This is a description with HTML.\",\n \"databases\": {\n \"db1\": {\n \"title\": \"First database\",\n \"description\": \"This is a string description & has no HTML\",\n \"license_url\": \"http://example.com/\",\n \"license\": \"The example license\",\n \"queries\": {\n \"canned_query\": \"select * from table1 limit 3;\"\n },\n \"tables\": {\n \"table1\": {\n \"title\": \"Custom title for table1\",\n \"description\": \"Tables can have descriptions too\",\n \"source\": \"This has a custom source\",\n \"source_url\": \"http://example.com/\"\n }\n }\n }\n }\n} \n \n \n Renamed datasette build command to datasette inspect ( #130 ) \n \n \n Upgrade to Sanic 0.7.0 ( #168 ) \n https://github.com/channelcat/sanic/releases/tag/0.7.0 \n \n \n Package and publish commands now accept --static and --template-dir \n Example usage: \n datasette package --static css:extra-css/ --static js:extra-js/ \\\n sf-trees.db --template-dir templates/ --tag sf-trees --branch master \n This creates a local Docker image that includes copies of the templates/,\n extra-css/ and extra-js/ directories. You can then run it like this: \n docker run -p 8001:8001 sf-trees \n For publishing to Zeit now: \n datasette publish now --static css:extra-css/ --static js:extra-js/ \\\n sf-trees.db --template-dir templates/ --name sf-trees --branch master \n \n \n HTML comment showing which templates were considered for a page ( #171 )", "sections_fts": 7, "rank": null} {"rowid": 342, "title": "0.13 (2017-11-24)", "content": "Search now applies to current filters. \n Combined search into the same form as filters. \n Closes #133 \n \n \n Much tidier design for table view header. \n Closes #147 \n \n \n Added ?column__not=blah filter. \n Closes #148 \n \n \n Row page now resolves foreign keys. \n Closes #132 \n \n \n Further tweaks to select/input filter styling. \n Refs #86 - thanks for the help, @natbat! \n \n \n Show linked foreign key in table cells. \n \n \n Added UI for editing table filters. \n Refs #86 \n \n \n Hide FTS-created tables on index pages. \n Closes #129 \n \n \n Add publish to heroku support [Jacob Kaplan-Moss] \n datasette publish heroku mydb.db \n Pull request #104 \n \n \n Initial implementation of ?_group_count=column . \n URL shortcut for counting rows grouped by one or more columns. \n ?_group_count=column1&_group_count=column2 works as well. \n SQL generated looks like this: \n select \"qSpecies\", count(*) as \"count\"\nfrom Street_Tree_List\ngroup by \"qSpecies\"\norder by \"count\" desc limit 100 \n Or for two columns like this: \n select \"qSpecies\", \"qSiteInfo\", count(*) as \"count\"\nfrom Street_Tree_List\ngroup by \"qSpecies\", \"qSiteInfo\"\norder by \"count\" desc limit 100 \n Refs #44 \n \n \n Added --build=master option to datasette publish and package. \n The datasette publish and datasette package commands both now accept an\n optional --build argument. If provided, this can be used to specify a branch\n published to GitHub that should be built into the container. \n This makes it easier to test code that has not yet been officially released to\n PyPI, e.g.: \n datasette publish now mydb.db --branch=master \n \n \n Implemented ?_search=XXX + UI if a FTS table is detected. \n Closes #131 \n \n \n Added datasette --version support. \n \n \n Table views now show expanded foreign key references, if possible. \n If a table has foreign key columns, and those foreign key tables have\n label_columns , the TableView will now query those other tables for the\n corresponding values and display those values as links in the corresponding\n table cells. \n label_columns are currently detected by the inspect() function, which looks\n for any table that has just two columns - an ID column and one other - and\n sets the label_column to be that second non-ID column. \n \n \n Don't prevent tabbing to \"Run SQL\" button ( #117 ) [Robert Gieseke] \n See comment in #115 \n \n \n Add keyboard shortcut to execute SQL query ( #115 ) [Robert Gieseke] \n \n \n Allow --load-extension to be set via environment variable. \n \n \n Add support for ?field__isnull=1 ( #107 ) [Ray N] \n \n \n Add spatialite, switch to debian and local build ( #114 ) [Ariel N\u00fa\u00f1ez] \n \n \n Added --load-extension argument to datasette serve. \n Allows loading of SQLite extensions. Refs #110 .", "sections_fts": 7, "rank": null} {"rowid": 343, "title": "0.12 (2017-11-16)", "content": "Added __version__ , now displayed as tooltip in page footer ( #108 ). \n \n \n Added initial docs, including a changelog ( #99 ). \n \n \n Turned on auto-escaping in Jinja. \n \n \n Added a UI for editing named parameters ( #96 ). \n You can now construct a custom SQL statement using SQLite named\n parameters (e.g. :name ) and datasette will display form fields for\n editing those parameters. Here\u2019s an example which lets you see the\n most popular names for dogs of different species registered through\n various dog registration schemes in Australia. \n \n \n \n \n \n Pin to specific Jinja version. ( #100 ). \n \n \n Default to 127.0.0.1 not 0.0.0.0. ( #98 ). \n \n \n Added extra metadata options to publish and package commands. ( #92 ). \n You can now run these commands like so: \n datasette now publish mydb.db \\\n --title=\"My Title\" \\\n --source=\"Source\" \\\n --source_url=\"http://www.example.com/\" \\\n --license=\"CC0\" \\\n --license_url=\"https://creativecommons.org/publicdomain/zero/1.0/\" \n This will write those values into the metadata.json that is packaged with the\n app. If you also pass --metadata=metadata.json that file will be updated with the extra\n values before being written into the Docker image. \n \n \n Added production-ready Dockerfile ( #94 ) [Andrew\n Cutler] \n \n \n New ?_sql_time_limit_ms=10 argument to database and table page ( #95 ) \n \n \n SQL syntax highlighting with Codemirror ( #89 ) [Tom Dyson]", "sections_fts": 7, "rank": null} {"rowid": 344, "title": "0.11 (2017-11-14)", "content": "Added datasette publish now --force option. \n This calls now with --force - useful as it means you get a fresh copy of datasette even if Now has already cached that docker layer. \n \n \n Enable --cors by default when running in a container.", "sections_fts": 7, "rank": null} {"rowid": 345, "title": "0.10 (2017-11-14)", "content": "Fixed #83 - 500 error on individual row pages. \n \n \n Stop using sqlite WITH RECURSIVE in our tests. \n The version of Python 3 running in Travis CI doesn't support this.", "sections_fts": 7, "rank": null} {"rowid": 346, "title": "0.9 (2017-11-13)", "content": "Added --sql_time_limit_ms and --extra-options . \n The serve command now accepts --sql_time_limit_ms for customizing the SQL time\n limit. \n The publish and package commands now accept --extra-options which can be used\n to specify additional options to be passed to the datasite serve command when\n it executes inside the resulting Docker containers.", "sections_fts": 7, "rank": null} {"rowid": 347, "title": "0.8 (2017-11-13)", "content": "V0.8 - added PyPI metadata, ready to ship. \n \n \n Implemented offset/limit pagination for views ( #70 ). \n \n \n Improved pagination. ( #78 ) \n \n \n Limit on max rows returned, controlled by --max_returned_rows option. ( #69 ) \n If someone executes 'select * from table' against a table with a million rows\n in it, we could run into problems: just serializing that much data as JSON is\n likely to lock up the server. \n Solution: we now have a hard limit on the maximum number of rows that can be\n returned by a query. If that limit is exceeded, the server will return a\n \"truncated\": true field in the JSON. \n This limit can be optionally controlled by the new --max_returned_rows \n option. Setting that option to 0 disables the limit entirely.", "sections_fts": 7, "rank": null} {"rowid": 348, "title": "Facets", "content": "Datasette facets can be used to add a faceted browse interface to any database table.\n With facets, tables are displayed along with a summary showing the most common values in specified columns.\n These values can be selected to further filter the table. \n Here's an example : \n \n Facets can be specified in two ways: using query string parameters, or in metadata.json configuration for the table.", "sections_fts": 7, "rank": null} {"rowid": 349, "title": "Facets in query strings", "content": "To turn on faceting for specific columns on a Datasette table view, add one or more _facet=COLUMN parameters to the URL.\n For example, if you want to turn on facets for the city_id and state columns, construct a URL that looks like this: \n /dbname/tablename?_facet=state&_facet=city_id \n This works for both the HTML interface and the .json view.\n When enabled, facets will cause a facet_results block to be added to the JSON output, looking something like this: \n {\n \"state\": {\n \"name\": \"state\",\n \"results\": [\n {\n \"value\": \"CA\",\n \"label\": \"CA\",\n \"count\": 10,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&state=CA\",\n \"selected\": false\n },\n {\n \"value\": \"MI\",\n \"label\": \"MI\",\n \"count\": 4,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&state=MI\",\n \"selected\": false\n },\n {\n \"value\": \"MC\",\n \"label\": \"MC\",\n \"count\": 1,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&state=MC\",\n \"selected\": false\n }\n ],\n \"truncated\": false\n }\n \"city_id\": {\n \"name\": \"city_id\",\n \"results\": [\n {\n \"value\": 1,\n \"label\": \"San Francisco\",\n \"count\": 6,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&city_id=1\",\n \"selected\": false\n },\n {\n \"value\": 2,\n \"label\": \"Los Angeles\",\n \"count\": 4,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&city_id=2\",\n \"selected\": false\n },\n {\n \"value\": 3,\n \"label\": \"Detroit\",\n \"count\": 4,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&city_id=3\",\n \"selected\": false\n },\n {\n \"value\": 4,\n \"label\": \"Memnonia\",\n \"count\": 1,\n \"toggle_url\": \"http://...?_facet=city_id&_facet=state&city_id=4\",\n \"selected\": false\n }\n ],\n \"truncated\": false\n }\n} \n If Datasette detects that a column is a foreign key, the \"label\" property will be automatically derived from the detected label column on the referenced table. \n The default number of facet results returned is 30, controlled by the default_facet_size setting.\n You can increase this on an individual page by adding ?_facet_size=100 to the query string, up to a maximum of max_returned_rows (which defaults to 1000).", "sections_fts": 7, "rank": null} {"rowid": 350, "title": "Facets in metadata", "content": "You can turn facets on by default for specific tables by adding them to a \"facets\" key in a Datasette Metadata file. \n Here's an example that turns on faceting by default for the qLegalStatus column in the Street_Tree_List table in the sf-trees database: \n [[[cog\nfrom metadata_doc import metadata_example\nmetadata_example(cog, {\n \"databases\": {\n \"sf-trees\": {\n \"tables\": {\n \"Street_Tree_List\": {\n \"facets\": [\"qLegalStatus\"]\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]] \n Facets defined in this way will always be shown in the interface and returned in the API, regardless of the _facet arguments passed to the view. \n You can specify array or date facets in metadata using JSON objects with a single key of array or date and a value specifying the column, like this: \n [[[cog\nmetadata_example(cog, {\n \"facets\": [\n {\"array\": \"tags\"},\n {\"date\": \"created\"}\n ]\n}) \n ]]] \n [[[end]]] \n You can change the default facet size (the number of results shown for each facet) for a table using facet_size : \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"sf-trees\": {\n \"tables\": {\n \"Street_Tree_List\": {\n \"facets\": [\"qLegalStatus\"],\n \"facet_size\": 10\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]]", "sections_fts": 7, "rank": null} {"rowid": 351, "title": "Suggested facets", "content": "Datasette's table UI will suggest facets for the user to apply, based on the following criteria: \n For the currently filtered data are there any columns which, if applied as a facet... \n \n \n Will return 30 or less unique options \n \n \n Will return more than one unique option \n \n \n Will return less unique options than the total number of filtered rows \n \n \n And the query used to evaluate this criteria can be completed in under 50ms \n \n \n That last point is particularly important: Datasette runs a query for every column that is displayed on a page, which could get expensive - so to avoid slow load times it sets a time limit of just 50ms for each of those queries.\n This means suggested facets are unlikely to appear for tables with millions of records in them.", "sections_fts": 7, "rank": null} {"rowid": 352, "title": "Speeding up facets with indexes", "content": "The performance of facets can be greatly improved by adding indexes on the columns you wish to facet by.\n Adding indexes can be performed using the sqlite3 command-line utility. Here's how to add an index on the state column in a table called Food_Trucks : \n sqlite3 mydatabase.db \n SQLite version 3.19.3 2017-06-27 16:48:08\nEnter \".help\" for usage hints.\nsqlite> CREATE INDEX Food_Trucks_state ON Food_Trucks(\"state\"); \n Or using the sqlite-utils command-line utility: \n sqlite-utils create-index mydatabase.db Food_Trucks state", "sections_fts": 7, "rank": null} {"rowid": 353, "title": "Facet by JSON array", "content": "If your SQLite installation provides the json1 extension (you can check using /-/versions ) Datasette will automatically detect columns that contain JSON arrays of values and offer a faceting interface against those columns. \n This is useful for modelling things like tags without needing to break them out into a new table. \n Example here: latest.datasette.io/fixtures/facetable?_facet_array=tags", "sections_fts": 7, "rank": null} {"rowid": 354, "title": "Facet by date", "content": "If Datasette finds any columns that contain dates in the first 100 values, it will offer a faceting interface against the dates of those values.\n This works especially well against timestamp values such as 2019-03-01 12:44:00 . \n Example here: latest.datasette.io/fixtures/facetable?_facet_date=created", "sections_fts": 7, "rank": null} {"rowid": 355, "title": "JSON API", "content": "Datasette provides a JSON API for your SQLite databases. Anything you can do\n through the Datasette user interface can also be accessed as JSON via the API. \n To access the API for a page, either click on the .json link on that page or\n edit the URL and add a .json extension to it.", "sections_fts": 7, "rank": null} {"rowid": 356, "title": "Default representation", "content": "The default JSON representation of data from a SQLite table or custom query\n looks like this: \n {\n \"ok\": true,\n \"rows\": [\n {\n \"id\": 3,\n \"name\": \"Detroit\"\n },\n {\n \"id\": 2,\n \"name\": \"Los Angeles\"\n },\n {\n \"id\": 4,\n \"name\": \"Memnonia\"\n },\n {\n \"id\": 1,\n \"name\": \"San Francisco\"\n }\n ],\n \"truncated\": false\n} \n \"ok\" is always true if an error did not occur. \n The \"rows\" key is a list of objects, each one representing a row. \n The \"truncated\" key lets you know if the query was truncated. This can happen if a SQL query returns more than 1,000 results (or the max_returned_rows setting). \n For table pages, an additional key \"next\" may be present. This indicates that the next page in the pagination set can be retrieved using ?_next=VALUE .", "sections_fts": 7, "rank": null} {"rowid": 357, "title": "Different shapes", "content": "The _shape parameter can be used to access alternative formats for the\n rows key which may be more convenient for your application. There are three\n options: \n \n \n ?_shape=objects - \"rows\" is a list of JSON key/value objects - the default \n \n \n ?_shape=arrays - \"rows\" is a list of lists, where the order of values in each list matches the order of the columns \n \n \n ?_shape=array - a JSON array of objects - effectively just the \"rows\" key from the default representation \n \n \n ?_shape=array&_nl=on - a newline-separated list of JSON objects \n \n \n ?_shape=arrayfirst - a flat JSON array containing just the first value from each row \n \n \n ?_shape=object - a JSON object keyed using the primary keys of the rows \n \n \n _shape=arrays looks like this: \n {\n \"ok\": true,\n \"next\": null,\n \"rows\": [\n [3, \"Detroit\"],\n [2, \"Los Angeles\"],\n [4, \"Memnonia\"],\n [1, \"San Francisco\"]\n ]\n} \n _shape=array looks like this: \n [\n {\n \"id\": 3,\n \"name\": \"Detroit\"\n },\n {\n \"id\": 2,\n \"name\": \"Los Angeles\"\n },\n {\n \"id\": 4,\n \"name\": \"Memnonia\"\n },\n {\n \"id\": 1,\n \"name\": \"San Francisco\"\n }\n] \n _shape=array&_nl=on looks like this: \n {\"id\": 1, \"value\": \"Myoporum laetum :: Myoporum\"}\n{\"id\": 2, \"value\": \"Metrosideros excelsa :: New Zealand Xmas Tree\"}\n{\"id\": 3, \"value\": \"Pinus radiata :: Monterey Pine\"} \n _shape=arrayfirst looks like this: \n [1, 2, 3] \n _shape=object looks like this: \n {\n \"1\": {\n \"id\": 1,\n \"value\": \"Myoporum laetum :: Myoporum\"\n },\n \"2\": {\n \"id\": 2,\n \"value\": \"Metrosideros excelsa :: New Zealand Xmas Tree\"\n },\n \"3\": {\n \"id\": 3,\n \"value\": \"Pinus radiata :: Monterey Pine\"\n }\n] \n The object shape is only available for queries against tables - custom SQL\n queries and views do not have an obvious primary key so cannot be returned using\n this format. \n The object keys are always strings. If your table has a compound primary\n key, the object keys will be a comma-separated string.", "sections_fts": 7, "rank": null} {"rowid": 358, "title": "Pagination", "content": "The default JSON representation includes a \"next_url\" key which can be used to access the next page of results. If that key is null or missing then it means you have reached the final page of results. \n Other representations include pagination information in the link HTTP header. That header will look something like this: \n link: ; rel=\"next\" \n Here is an example Python function built using requests that returns a list of all of the paginated items from one of these API endpoints: \n def paginate(url):\n items = []\n while url:\n response = requests.get(url)\n try:\n url = response.links.get(\"next\").get(\"url\")\n except AttributeError:\n url = None\n items.extend(response.json())\n return items", "sections_fts": 7, "rank": null} {"rowid": 359, "title": "Special JSON arguments", "content": "Every Datasette endpoint that can return JSON also accepts the following\n query string arguments: \n \n \n ?_shape=SHAPE \n \n The shape of the JSON to return, documented above. \n \n \n \n ?_nl=on \n \n When used with ?_shape=array produces newline-delimited JSON objects. \n \n \n \n ?_json=COLUMN1&_json=COLUMN2 \n \n If any of your SQLite columns contain JSON values, you can use one or more\n _json= parameters to request that those columns be returned as regular\n JSON. Without this argument those columns will be returned as JSON objects\n that have been double-encoded into a JSON string value. \n Compare this query without the argument to this query using the argument \n \n \n \n ?_json_infinity=on \n \n If your data contains infinity or -infinity values, Datasette will replace\n them with None when returning them as JSON. If you pass _json_infinity=1 \n Datasette will instead return them as Infinity or -Infinity which is\n invalid JSON but can be processed by some custom JSON parsers. \n \n \n \n ?_timelimit=MS \n \n Sets a custom time limit for the query in ms. You can use this for optimistic\n queries where you would like Datasette to give up if the query takes too\n long, for example if you want to implement autocomplete search but only if\n it can be executed in less than 10ms. \n \n \n \n ?_ttl=SECONDS \n \n For how many seconds should this response be cached by HTTP proxies? Use\n ?_ttl=0 to disable HTTP caching entirely for this request. \n \n \n \n ?_trace=1 \n \n Turns on tracing for this page: SQL queries executed during the request will\n be gathered and included in the response, either in a new \"_traces\" key\n for JSON responses or at the bottom of the page if the response is in HTML. \n The structure of the data returned here should be considered highly unstable\n and very likely to change. \n Only available if the trace_debug setting is enabled.", "sections_fts": 7, "rank": null} {"rowid": 360, "title": "Table arguments", "content": "The Datasette table view takes a number of special query string arguments.", "sections_fts": 7, "rank": null} {"rowid": 361, "title": "Column filter arguments", "content": "You can filter the data returned by the table based on column values using a query string argument. \n \n \n ?column__exact=value or ?_column=value \n \n Returns rows where the specified column exactly matches the value. \n \n \n \n ?column__not=value \n \n Returns rows where the column does not match the value. \n \n \n \n ?column__contains=value \n \n Rows where the string column contains the specified value ( column like \"%value%\" in SQL). \n \n \n \n ?column__notcontains=value \n \n Rows where the string column does not contain the specified value ( column not like \"%value%\" in SQL). \n \n \n \n ?column__endswith=value \n \n Rows where the string column ends with the specified value ( column like \"%value\" in SQL). \n \n \n \n ?column__startswith=value \n \n Rows where the string column starts with the specified value ( column like \"value%\" in SQL). \n \n \n \n ?column__gt=value \n \n Rows which are greater than the specified value. \n \n \n \n ?column__gte=value \n \n Rows which are greater than or equal to the specified value. \n \n \n \n ?column__lt=value \n \n Rows which are less than the specified value. \n \n \n \n ?column__lte=value \n \n Rows which are less than or equal to the specified value. \n \n \n \n ?column__like=value \n \n Match rows with a LIKE clause, case insensitive and with % as the wildcard character. \n \n \n \n ?column__notlike=value \n \n Match rows that do not match the provided LIKE clause. \n \n \n \n ?column__glob=value \n \n Similar to LIKE but uses Unix wildcard syntax and is case sensitive. \n \n \n \n ?column__in=value1,value2,value3 \n \n Rows where column matches any of the provided values. \n You can use a comma separated string, or you can use a JSON array. \n The JSON array option is useful if one of your matching values itself contains a comma: \n ?column__in=[\"value\",\"value,with,commas\"] \n \n \n \n ?column__notin=value1,value2,value3 \n \n Rows where column does not match any of the provided values. The inverse of __in= . Also supports JSON arrays. \n \n \n \n ?column__arraycontains=value \n \n Works against columns that contain JSON arrays - matches if any of the values in that array match the provided value. \n This is only available if the json1 SQLite extension is enabled. \n \n \n \n ?column__arraynotcontains=value \n \n Works against columns that contain JSON arrays - matches if none of the values in that array match the provided value. \n This is only available if the json1 SQLite extension is enabled. \n \n \n \n ?column__date=value \n \n Column is a datestamp occurring on the specified YYYY-MM-DD date, e.g. 2018-01-02 . \n \n \n \n ?column__isnull=1 \n \n Matches rows where the column is null. \n \n \n \n ?column__notnull=1 \n \n Matches rows where the column is not null. \n \n \n \n ?column__isblank=1 \n \n Matches rows where the column is blank, meaning null or the empty string. \n \n \n \n ?column__notblank=1 \n \n Matches rows where the column is not blank.", "sections_fts": 7, "rank": null} {"rowid": 362, "title": "Special table arguments", "content": "?_col=COLUMN1&_col=COLUMN2 \n \n List specific columns to display. These will be shown along with any primary keys. \n \n \n \n ?_nocol=COLUMN1&_nocol=COLUMN2 \n \n List specific columns to hide - any column not listed will be displayed. Primary keys cannot be hidden. \n \n \n \n ?_labels=on/off \n \n Expand foreign key references for every possible column. See below. \n \n \n \n ?_label=COLUMN1&_label=COLUMN2 \n \n Expand foreign key references for one or more specified columns. \n \n \n \n ?_size=1000 or ?_size=max \n \n Sets a custom page size. This cannot exceed the max_returned_rows limit\n passed to datasette serve . Use max to get max_returned_rows . \n \n \n \n ?_sort=COLUMN \n \n Sorts the results by the specified column. \n \n \n \n ?_sort_desc=COLUMN \n \n Sorts the results by the specified column in descending order. \n \n \n \n ?_search=keywords \n \n For SQLite tables that have been configured for\n full-text search executes a search\n with the provided keywords. \n \n \n \n ?_search_COLUMN=keywords \n \n Like _search= but allows you to specify the column to be searched, as\n opposed to searching all columns that have been indexed by FTS. \n \n \n \n ?_searchmode=raw \n \n With this option, queries passed to ?_search= or ?_search_COLUMN= will\n not have special characters escaped. This means you can make use of the full\n set of advanced SQLite FTS syntax ,\n though this could potentially result in errors if the wrong syntax is used. \n \n \n \n ?_where=SQL-fragment \n \n If the execute-sql permission is enabled, this parameter\n can be used to pass one or more additional SQL fragments to be used in the\n WHERE clause of the SQL used to query the table. \n This is particularly useful if you are building a JavaScript application\n that needs to do something creative but still wants the other conveniences\n provided by the table view (such as faceting) and hence would like not to\n have to construct a completely custom SQL query. \n Some examples: \n \n \n facetable?_where=_neighborhood like \"%c%\"&_where=_city_id=3 \n \n \n facetable?_where=_city_id in (select id from facet_cities where name != \"Detroit\") \n \n \n \n \n \n ?_through={json} \n \n This can be used to filter rows via a join against another table. \n The JSON parameter must include three keys: table , column and value . \n table must be a table that the current table is related to via a foreign key relationship. \n column must be a column in that other table. \n value is the value that you want to match against. \n For example, to filter roadside_attractions to just show the attractions that have a characteristic of \"museum\", you would construct this JSON: \n {\n \"table\": \"roadside_attraction_characteristics\",\n \"column\": \"characteristic_id\",\n \"value\": \"1\"\n} \n As a URL, that looks like this: \n ?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22} \n Here's an example . \n \n \n \n ?_next=TOKEN \n \n Pagination by continuation token - pass the token that was returned in the\n \"next\" property by the previous page. \n \n \n \n ?_facet=column \n \n Facet by column. Can be applied multiple times, see Facets . Only works on the default JSON output, not on any of the custom shapes. \n \n \n \n ?_facet_size=100 \n \n Increase the number of facet results returned for each facet. Use ?_facet_size=max for the maximum available size, determined by max_returned_rows . \n \n \n \n ?_nofacet=1 \n \n Disable all facets and facet suggestions for this page, including any defined by Facets in metadata . \n \n \n \n ?_nosuggest=1 \n \n Disable facet suggestions for this page. \n \n \n \n ?_nocount=1 \n \n Disable the select count(*) query used on this page - a count of None will be returned instead.", "sections_fts": 7, "rank": null} {"rowid": 363, "title": "Expanding foreign key references", "content": "Datasette can detect foreign key relationships and resolve those references into\n labels. The HTML interface does this by default for every detected foreign key\n column - you can turn that off using ?_labels=off . \n You can request foreign keys be expanded in JSON using the _labels=on or\n _label=COLUMN special query string parameters. Here's what an expanded row\n looks like: \n [\n {\n \"rowid\": 1,\n \"TreeID\": 141565,\n \"qLegalStatus\": {\n \"value\": 1,\n \"label\": \"Permitted Site\"\n },\n \"qSpecies\": {\n \"value\": 1,\n \"label\": \"Myoporum laetum :: Myoporum\"\n },\n \"qAddress\": \"501X Baker St\",\n \"SiteOrder\": 1\n }\n] \n The column in the foreign key table that is used for the label can be specified\n in metadata.json - see Specifying the label column for a table .", "sections_fts": 7, "rank": null} {"rowid": 364, "title": "Discovering the JSON for a page", "content": "Most of the HTML pages served by Datasette provide a mechanism for discovering their JSON equivalents using the HTML link mechanism. \n You can find this near the top of the source code of those pages, looking like this: \n \n The JSON URL is also made available in a Link HTTP header for the page: \n Link: https://latest.datasette.io/fixtures/sortable.json; rel=\"alternate\"; type=\"application/json+datasette\"", "sections_fts": 7, "rank": null} {"rowid": 365, "title": "Enabling CORS", "content": "If you start Datasette with the --cors option, each JSON endpoint will be\n served with the following additional HTTP headers: \n [[[cog\nfrom datasette.utils import add_cors_headers\nimport textwrap\nheaders = {}\nadd_cors_headers(headers)\noutput = \"\\n\".join(\"{}: {}\".format(k, v) for k, v in headers.items())\ncog.out(\"\\n::\\n\\n\")\ncog.out(textwrap.indent(output, ' '))\ncog.out(\"\\n\\n\") \n ]]] \n Access-Control-Allow-Origin: *\nAccess-Control-Allow-Headers: Authorization, Content-Type\nAccess-Control-Expose-Headers: Link\nAccess-Control-Allow-Methods: GET, POST, HEAD, OPTIONS\nAccess-Control-Max-Age: 3600 \n [[[end]]] \n This allows JavaScript running on any domain to make cross-origin\n requests to interact with the Datasette API. \n If you start Datasette without the --cors option only JavaScript running on\n the same domain as Datasette will be able to access the API. \n Here's how to serve data.db with CORS enabled: \n datasette data.db --cors", "sections_fts": 7, "rank": null} {"rowid": 366, "title": "The JSON write API", "content": "Datasette provides a write API for JSON data. This is a POST-only API that requires an authenticated API token, see API Tokens . The token will need to have the specified Permissions .", "sections_fts": 7, "rank": null} {"rowid": 367, "title": "Inserting rows", "content": "This requires the insert-row permission. \n A single row can be inserted using the \"row\" key: \n POST ///-/insert\nContent-Type: application/json\nAuthorization: Bearer dstok_ \n {\n \"row\": {\n \"column1\": \"value1\",\n \"column2\": \"value2\"\n }\n} \n If successful, this will return a 201 status code and the newly inserted row, for example: \n {\n \"rows\": [\n {\n \"id\": 1,\n \"column1\": \"value1\",\n \"column2\": \"value2\"\n }\n ]\n} \n To insert multiple rows at a time, use the same API method but send a list of dictionaries as the \"rows\" key: \n POST //
/-/insert\nContent-Type: application/json\nAuthorization: Bearer dstok_ \n {\n \"rows\": [\n {\n \"column1\": \"value1\",\n \"column2\": \"value2\"\n },\n {\n \"column1\": \"value3\",\n \"column2\": \"value4\"\n }\n ]\n} \n If successful, this will return a 201 status code and a {\"ok\": true} response body. \n The maximum number rows that can be submitted at once defaults to 100, but this can be changed using the max_insert_rows setting. \n To return the newly inserted rows, add the \"return\": true key to the request body: \n {\n \"rows\": [\n {\n \"column1\": \"value1\",\n \"column2\": \"value2\"\n },\n {\n \"column1\": \"value3\",\n \"column2\": \"value4\"\n }\n ],\n \"return\": true\n} \n This will return the same \"rows\" key as the single row example above. There is a small performance penalty for using this option. \n If any of your rows have a primary key that is already in use, you will get an error and none of the rows will be inserted: \n {\n \"ok\": false,\n \"errors\": [\n \"UNIQUE constraint failed: new_table.id\"\n ]\n} \n Pass \"ignore\": true to ignore these errors and insert the other rows: \n {\n \"rows\": [\n {\n \"id\": 1,\n \"column1\": \"value1\",\n \"column2\": \"value2\"\n },\n {\n \"id\": 2,\n \"column1\": \"value3\",\n \"column2\": \"value4\"\n }\n ],\n \"ignore\": true\n} \n Or you can pass \"replace\": true to replace any rows with conflicting primary keys with the new values. This requires the update-row permission. \n Pass \"alter: true to automatically add any missing columns to the table. This requires the alter-table permission.", "sections_fts": 7, "rank": null} {"rowid": 368, "title": "Upserting rows", "content": "An upsert is an insert or update operation. If a row with a matching primary key already exists it will be updated - otherwise a new row will be inserted. \n The upsert API is mostly the same shape as the insert API . It requires both the insert-row and update-row permissions. \n POST //
/-/upsert\nContent-Type: application/json\nAuthorization: Bearer dstok_ \n {\n \"rows\": [\n {\n \"id\": 1,\n \"title\": \"Updated title for 1\",\n \"description\": \"Updated description for 1\"\n },\n {\n \"id\": 2,\n \"description\": \"Updated description for 2\",\n },\n {\n \"id\": 3,\n \"title\": \"Item 3\",\n \"description\": \"Description for 3\"\n }\n ]\n} \n Imagine a table with a primary key of id and which already has rows with id values of 1 and 2 . \n The above example will: \n \n \n Update the row with id of 1 to set both title and description to the new values \n \n \n Update the row with id of 2 to set title to the new value - description will be left unchanged \n \n \n Insert a new row with id of 3 and both title and description set to the new values \n \n \n Similar to /-/insert , a row key with an object can be used instead of a rows array to upsert a single row. \n If successful, this will return a 200 status code and a {\"ok\": true} response body. \n Add \"return\": true to the request body to return full copies of the affected rows after they have been inserted or updated: \n {\n \"rows\": [\n {\n \"id\": 1,\n \"title\": \"Updated title for 1\",\n \"description\": \"Updated description for 1\"\n },\n {\n \"id\": 2,\n \"description\": \"Updated description for 2\",\n },\n {\n \"id\": 3,\n \"title\": \"Item 3\",\n \"description\": \"Description for 3\"\n }\n ],\n \"return\": true\n} \n This will return the following: \n {\n \"ok\": true,\n \"rows\": [\n {\n \"id\": 1,\n \"title\": \"Updated title for 1\",\n \"description\": \"Updated description for 1\"\n },\n {\n \"id\": 2,\n \"title\": \"Item 2\",\n \"description\": \"Updated description for 2\"\n },\n {\n \"id\": 3,\n \"title\": \"Item 3\",\n \"description\": \"Description for 3\"\n }\n ]\n} \n When using upsert you must provide the primary key column (or columns if the table has a compound primary key) for every row, or you will get a 400 error: \n {\n \"ok\": false,\n \"errors\": [\n \"Row 0 is missing primary key column(s): \\\"id\\\"\"\n ]\n} \n If your table does not have an explicit primary key you should pass the SQLite rowid key instead. \n Pass \"alter: true to automatically add any missing columns to the table. This requires the alter-table permission.", "sections_fts": 7, "rank": null} {"rowid": 369, "title": "Updating a row", "content": "To update a row, make a POST to //
//-/update . This requires the update-row permission. \n POST //
//-/update\nContent-Type: application/json\nAuthorization: Bearer dstok_ \n {\n \"update\": {\n \"text_column\": \"New text string\",\n \"integer_column\": 3,\n \"float_column\": 3.14\n }\n} \n here is the tilde-encoded primary key value of the row to update - or a comma-separated list of primary key values if the table has a composite primary key. \n You only need to pass the columns you want to update. Any other columns will be left unchanged. \n If successful, this will return a 200 status code and a {\"ok\": true} response body. \n Add \"return\": true to the request body to return the updated row: \n {\n \"update\": {\n \"title\": \"New title\"\n },\n \"return\": true\n} \n The returned JSON will look like this: \n {\n \"ok\": true,\n \"row\": {\n \"id\": 1,\n \"title\": \"New title\",\n \"other_column\": \"Will be present here too\"\n }\n} \n Any errors will return {\"errors\": [\"... descriptive message ...\"], \"ok\": false} , and a 400 status code for a bad input or a 403 status code for an authentication or permission error. \n Pass \"alter: true to automatically add any missing columns to the table. This requires the alter-table permission.", "sections_fts": 7, "rank": null} {"rowid": 370, "title": "Deleting a row", "content": "To delete a row, make a POST to //
//-/delete . This requires the delete-row permission. \n POST //
//-/delete\nContent-Type: application/json\nAuthorization: Bearer dstok_ \n here is the tilde-encoded primary key value of the row to delete - or a comma-separated list of primary key values if the table has a composite primary key. \n If successful, this will return a 200 status code and a {\"ok\": true} response body. \n Any errors will return {\"errors\": [\"... descriptive message ...\"], \"ok\": false} , and a 400 status code for a bad input or a 403 status code for an authentication or permission error.", "sections_fts": 7, "rank": null} {"rowid": 371, "title": "Creating a table", "content": "To create a table, make a POST to //-/create . This requires the create-table permission. \n POST //-/create\nContent-Type: application/json\nAuthorization: Bearer dstok_ \n {\n \"table\": \"name_of_new_table\",\n \"columns\": [\n {\n \"name\": \"id\",\n \"type\": \"integer\"\n },\n {\n \"name\": \"title\",\n \"type\": \"text\"\n }\n ],\n \"pk\": \"id\"\n} \n The JSON here describes the table that will be created: \n \n \n table is the name of the table to create. This field is required. \n \n \n columns is a list of columns to create. Each column is a dictionary with name and type keys. \n \n \n name is the name of the column. This is required. \n \n \n type is the type of the column. This is optional - if not provided, text will be assumed. The valid types are text , integer , float and blob . \n \n \n \n \n pk is the primary key for the table. This is optional - if not provided, Datasette will create a SQLite table with a hidden rowid column. \n If the primary key is an integer column, it will be configured to automatically increment for each new record. \n If you set this to id without including an id column in the list of columns , Datasette will create an auto-incrementing integer ID column for you. \n \n \n pks can be used instead of pk to create a compound primary key. It should be a JSON list of column names to use in that primary key. \n \n \n ignore can be set to true to ignore existing rows by primary key if the table already exists. \n \n \n replace can be set to true to replace existing rows by primary key if the table already exists. This requires the update-row permission. \n \n \n alter can be set to true if you want to automatically add any missing columns to the table. This requires the alter-table permission. \n \n \n If the table is successfully created this will return a 201 status code and the following response: \n {\n \"ok\": true,\n \"database\": \"data\",\n \"table\": \"name_of_new_table\",\n \"table_url\": \"http://127.0.0.1:8001/data/name_of_new_table\",\n \"table_api_url\": \"http://127.0.0.1:8001/data/name_of_new_table.json\",\n \"schema\": \"CREATE TABLE [name_of_new_table] (\\n [id] INTEGER PRIMARY KEY,\\n [title] TEXT\\n)\"\n}", "sections_fts": 7, "rank": null} {"rowid": 372, "title": "Creating a table from example data", "content": "Instead of specifying columns directly you can instead pass a single example row or a list of rows .\n Datasette will create a table with a schema that matches those rows and insert them for you: \n POST //-/create\nContent-Type: application/json\nAuthorization: Bearer dstok_ \n {\n \"table\": \"creatures\",\n \"rows\": [\n {\n \"id\": 1,\n \"name\": \"Tarantula\"\n },\n {\n \"id\": 2,\n \"name\": \"K\u0101k\u0101p\u014d\"\n }\n ],\n \"pk\": \"id\"\n} \n Doing this requires both the create-table and insert-row permissions. \n The 201 response here will be similar to the columns form, but will also include the number of rows that were inserted as row_count : \n {\n \"ok\": true,\n \"database\": \"data\",\n \"table\": \"creatures\",\n \"table_url\": \"http://127.0.0.1:8001/data/creatures\",\n \"table_api_url\": \"http://127.0.0.1:8001/data/creatures.json\",\n \"schema\": \"CREATE TABLE [creatures] (\\n [id] INTEGER PRIMARY KEY,\\n [name] TEXT\\n)\",\n \"row_count\": 2\n} \n You can call the create endpoint multiple times for the same table provided you are specifying the table using the rows or row option. New rows will be inserted into the table each time. This means you can use this API if you are unsure if the relevant table has been created yet. \n If you pass a row to the create endpoint with a primary key that already exists you will get an error that looks like this: \n {\n \"ok\": false,\n \"errors\": [\n \"UNIQUE constraint failed: creatures.id\"\n ]\n} \n You can avoid this error by passing the same \"ignore\": true or \"replace\": true options to the create endpoint as you can to the insert endpoint . \n To use the \"replace\": true option you will also need the update-row permission. \n Pass \"alter\": true to automatically add any missing columns to the existing table that are present in the rows you are submitting. This requires the alter-table permission.", "sections_fts": 7, "rank": null} {"rowid": 373, "title": "Dropping tables", "content": "To drop a table, make a POST to //
/-/drop . This requires the drop-table permission. \n POST //
/-/drop\nContent-Type: application/json\nAuthorization: Bearer dstok_ \n Without a POST body this will return a status 200 with a note about how many rows will be deleted: \n {\n \"ok\": true,\n \"database\": \"\",\n \"table\": \"
\",\n \"row_count\": 5,\n \"message\": \"Pass \\\"confirm\\\": true to confirm\"\n} \n If you pass the following POST body: \n {\n \"confirm\": true\n} \n Then the table will be dropped and a status 200 response of {\"ok\": true} will be returned. \n Any errors will return {\"errors\": [\"... descriptive message ...\"], \"ok\": false} , and a 400 status code for a bad input or a 403 status code for an authentication or permission error.", "sections_fts": 7, "rank": null} {"rowid": 374, "title": "Metadata", "content": "Data loves metadata. Any time you run Datasette you can optionally include a\n YAML or JSON file with metadata about your databases and tables. Datasette will then\n display that information in the web UI. \n Run Datasette like this: \n datasette database1.db database2.db --metadata metadata.yaml \n Your metadata.yaml file can look something like this: \n [[[cog\nfrom metadata_doc import metadata_example\nmetadata_example(cog, {\n \"title\": \"Custom title for your index page\",\n \"description\": \"Some description text can go here\",\n \"license\": \"ODbL\",\n \"license_url\": \"https://opendatacommons.org/licenses/odbl/\",\n \"source\": \"Original Data Source\",\n \"source_url\": \"http://example.com/\"\n}) \n ]]] \n [[[end]]] \n Choosing YAML over JSON adds support for multi-line strings and comments. \n The above metadata will be displayed on the index page of your Datasette-powered\n site. The source and license information will also be included in the footer of\n every page served by Datasette. \n Any special HTML characters in description will be escaped. If you want to\n include HTML in your description, you can use a description_html property\n instead.", "sections_fts": 7, "rank": null} {"rowid": 375, "title": "Per-database and per-table metadata", "content": "Metadata at the top level of the file will be shown on the index page and in the\n footer on every page of the site. The license and source is expected to apply to\n all of your data. \n You can also provide metadata at the per-database or per-table level, like this: \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"database1\": {\n \"source\": \"Alternative source\",\n \"source_url\": \"http://example.com/\",\n \"tables\": {\n \"example_table\": {\n \"description_html\": \"Custom table description\",\n \"license\": \"CC BY 3.0 US\",\n \"license_url\": \"https://creativecommons.org/licenses/by/3.0/us/\"\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]] \n Each of the top-level metadata fields can be used at the database and table level.", "sections_fts": 7, "rank": null} {"rowid": 376, "title": "Source, license and about", "content": "The three visible metadata fields you can apply to everything, specific databases or specific tables are source, license and about. All three are optional. \n source and source_url should be used to indicate where the underlying data came from. \n license and license_url should be used to indicate the license under which the data can be used. \n about and about_url can be used to link to further information about the project - an accompanying blog entry for example. \n For each of these you can provide just the *_url field and Datasette will treat that as the default link label text and display the URL directly on the page.", "sections_fts": 7, "rank": null} {"rowid": 377, "title": "Column descriptions", "content": "You can include descriptions for your columns by adding a \"columns\": {\"name-of-column\": \"description-of-column\"} block to your table metadata: \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"database1\": {\n \"tables\": {\n \"example_table\": {\n \"columns\": {\n \"column1\": \"Description of column 1\",\n \"column2\": \"Description of column 2\"\n }\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]] \n These will be displayed at the top of the table page, and will also show in the cog menu for each column. \n You can see an example of how these look at latest.datasette.io/fixtures/roadside_attractions .", "sections_fts": 7, "rank": null} {"rowid": 378, "title": "Setting a default sort order", "content": "By default Datasette tables are sorted by primary key. You can over-ride this default for a specific table using the \"sort\" or \"sort_desc\" metadata properties: \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"mydatabase\": {\n \"tables\": {\n \"example_table\": {\n \"sort\": \"created\"\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]] \n Or use \"sort_desc\" to sort in descending order: \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"mydatabase\": {\n \"tables\": {\n \"example_table\": {\n \"sort_desc\": \"created\"\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]]", "sections_fts": 7, "rank": null} {"rowid": 379, "title": "Setting a custom page size", "content": "Datasette defaults to displaying 100 rows per page, for both tables and views. You can change this default page size on a per-table or per-view basis using the \"size\" key in metadata.json : \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"mydatabase\": {\n \"tables\": {\n \"example_table\": {\n \"size\": 10\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]] \n This size can still be over-ridden by passing e.g. ?_size=50 in the query string.", "sections_fts": 7, "rank": null} {"rowid": 380, "title": "Setting which columns can be used for sorting", "content": "Datasette allows any column to be used for sorting by default. If you need to\n control which columns are available for sorting you can do so using the optional\n sortable_columns key: \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"database1\": {\n \"tables\": {\n \"example_table\": {\n \"sortable_columns\": [\n \"height\",\n \"weight\"\n ]\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]] \n This will restrict sorting of example_table to just the height and\n weight columns. \n You can also disable sorting entirely by setting \"sortable_columns\": [] \n You can use sortable_columns to enable specific sort orders for a view called name_of_view in the database my_database like so: \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"my_database\": {\n \"tables\": {\n \"name_of_view\": {\n \"sortable_columns\": [\n \"clicks\",\n \"impressions\"\n ]\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]]", "sections_fts": 7, "rank": null} {"rowid": 381, "title": "Specifying the label column for a table", "content": "Datasette's HTML interface attempts to display foreign key references as\n labelled hyperlinks. By default, it looks for referenced tables that only have\n two columns: a primary key column and one other. It assumes that the second\n column should be used as the link label. \n If your table has more than two columns you can specify which column should be\n used for the link label with the label_column property: \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"database1\": {\n \"tables\": {\n \"example_table\": {\n \"label_column\": \"title\"\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]]", "sections_fts": 7, "rank": null} {"rowid": 382, "title": "Hiding tables", "content": "You can hide tables from the database listing view (in the same way that FTS and\n SpatiaLite tables are automatically hidden) using \"hidden\": true : \n [[[cog\nmetadata_example(cog, {\n \"databases\": {\n \"database1\": {\n \"tables\": {\n \"example_table\": {\n \"hidden\": True\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]]", "sections_fts": 7, "rank": null} {"rowid": 383, "title": "Metadata reference", "content": "A full reference of every supported option in a metadata.json or metadata.yaml file.", "sections_fts": 7, "rank": null} {"rowid": 384, "title": "Top-level metadata", "content": "\"Top-level\" metadata refers to fields that can be specified at the root level of a metadata file. These attributes are meant to describe the entire Datasette instance. \n The following are the full list of allowed top-level metadata fields: \n \n \n title \n \n \n description \n \n \n description_html \n \n \n license \n \n \n license_url \n \n \n source \n \n \n source_url", "sections_fts": 7, "rank": null} {"rowid": 385, "title": "Database-level metadata", "content": "\"Database-level\" metadata refers to fields that can be specified for each database in a Datasette instance. These attributes should be listed under a database inside the \"databases\" field. \n The following are the full list of allowed database-level metadata fields: \n \n \n source \n \n \n source_url \n \n \n license \n \n \n license_url \n \n \n about \n \n \n about_url", "sections_fts": 7, "rank": null} {"rowid": 386, "title": "Table-level metadata", "content": "\"Table-level\" metadata refers to fields that can be specified for each table in a Datasette instance. These attributes should be listed under a specific table using the \"tables\" field. \n The following are the full list of allowed table-level metadata fields: \n \n \n source \n \n \n source_url \n \n \n license \n \n \n license_url \n \n \n about \n \n \n about_url \n \n \n hidden \n \n \n sort/sort_desc \n \n \n size \n \n \n sortable_columns \n \n \n label_column \n \n \n facets \n \n \n fts_table \n \n \n fts_pk \n \n \n searchmode \n \n \n columns", "sections_fts": 7, "rank": null} {"rowid": 387, "title": "Full-text search", "content": "SQLite includes a powerful mechanism for enabling full-text search against SQLite records. Datasette can detect if a table has had full-text search configured for it in the underlying database and display a search interface for filtering that table. \n Here's an example search : \n \n Datasette automatically detects which tables have been configured for full-text search.", "sections_fts": 7, "rank": null} {"rowid": 388, "title": "The table page and table view API", "content": "Table views that support full-text search can be queried using the ?_search=TERMS query string parameter. This will run the search against content from all of the columns that have been included in the index. \n Try this example: fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort \n SQLite full-text search supports wildcards. This means you can easily implement prefix auto-complete by including an asterisk at the end of the search term - for example: \n /dbname/tablename/?_search=rob* \n This will return all records containing at least one word that starts with the letters rob . \n You can also run searches against just the content of a specific named column by using _search_COLNAME=TERMS - for example, this would search for just rows where the name column in the FTS index mentions Sarah : \n /dbname/tablename/?_search_name=Sarah", "sections_fts": 7, "rank": null} {"rowid": 389, "title": "Advanced SQLite search queries", "content": "SQLite full-text search includes support for a variety of advanced queries , including AND , OR , NOT and NEAR . \n By default Datasette disables these features to ensure they do not cause errors or confusion for users who are not aware of them. You can disable this escaping and use the advanced queries by adding &_searchmode=raw to the table page query string. \n If you want to enable these operators by default for a specific table, you can do so by adding \"searchmode\": \"raw\" to the metadata configuration for that table, see Configuring full-text search for a table or view . \n If that option has been specified in the table metadata but you want to over-ride it and return to the default behavior you can append &_searchmode=escaped to the query string.", "sections_fts": 7, "rank": null} {"rowid": 390, "title": "Configuring full-text search for a table or view", "content": "If a table has a corresponding FTS table set up using the content= argument to CREATE VIRTUAL TABLE shown below, Datasette will detect it automatically and add a search interface to the table page for that table. \n You can also manually configure which table should be used for full-text search using query string parameters or Metadata . You can set the associated FTS table for a specific table and you can also set one for a view - if you do that, the page for that SQL view will offer a search option. \n Use ?_fts_table=x to over-ride the FTS table for a specific page. If the primary key was something other than rowid you can use ?_fts_pk=col to set that as well. This is particularly useful for views, for example: \n https://latest.datasette.io/fixtures/searchable_view?_fts_table=searchable_fts&_fts_pk=pk \n The fts_table metadata property can be used to specify an associated FTS table. If the primary key column in your table which was used to populate the FTS table is something other than rowid , you can specify the column to use with the fts_pk property. \n The \"searchmode\": \"raw\" property can be used to default the table to accepting SQLite advanced search operators, as described in Advanced SQLite search queries . \n Here is an example which enables full-text search (with SQLite advanced search operators) for a display_ads view which is defined against the ads table and hence needs to run FTS against the ads_fts table, using the id as the primary key: \n [[[cog\nfrom metadata_doc import metadata_example\nmetadata_example(cog, {\n \"databases\": {\n \"russian-ads\": {\n \"tables\": {\n \"display_ads\": {\n \"fts_table\": \"ads_fts\",\n \"fts_pk\": \"id\",\n \"searchmode\": \"raw\"\n }\n }\n }\n }\n}) \n ]]] \n [[[end]]]", "sections_fts": 7, "rank": null} {"rowid": 391, "title": "Searches using custom SQL", "content": "You can include full-text search results in custom SQL queries. The general pattern with SQLite search is to run the search as a sub-select that returns rowid values, then include those rowids in another part of the query. \n You can see the syntax for a basic search by running that search on a table page and then clicking \"View and edit SQL\" to see the underlying SQL. For example, consider this search for manafort is the US FARA database : \n /fara/FARA_All_ShortForms?_search=manafort \n If you click View and edit SQL you'll see that the underlying SQL looks like this: \n select\n rowid,\n Short_Form_Termination_Date,\n Short_Form_Date,\n Short_Form_Last_Name,\n Short_Form_First_Name,\n Registration_Number,\n Registration_Date,\n Registrant_Name,\n Address_1,\n Address_2,\n City,\n State,\n Zip\nfrom\n FARA_All_ShortForms\nwhere\n rowid in (\n select\n rowid\n from\n FARA_All_ShortForms_fts\n where\n FARA_All_ShortForms_fts match escape_fts(:search)\n )\norder by\n rowid\nlimit\n 101", "sections_fts": 7, "rank": null} {"rowid": 392, "title": "Enabling full-text search for a SQLite table", "content": "Datasette takes advantage of the external content mechanism in SQLite, which allows a full-text search virtual table to be associated with the contents of another SQLite table. \n To set up full-text search for a table, you need to do two things: \n \n \n Create a new FTS virtual table associated with your table \n \n \n Populate that FTS table with the data that you would like to be able to run searches against", "sections_fts": 7, "rank": null} {"rowid": 393, "title": "Configuring FTS using sqlite-utils", "content": "sqlite-utils is a CLI utility and Python library for manipulating SQLite databases. You can use it from Python code to configure FTS search, or you can achieve the same goal using the accompanying command-line tool . \n Here's how to use sqlite-utils to enable full-text search for an items table across the name and description columns: \n sqlite-utils enable-fts mydatabase.db items name description", "sections_fts": 7, "rank": null} {"rowid": 394, "title": "Configuring FTS using csvs-to-sqlite", "content": "If your data starts out in CSV files, you can use Datasette's companion tool csvs-to-sqlite to convert that file into a SQLite database and enable full-text search on specific columns. For a file called items.csv where you want full-text search to operate against the name and description columns you would run the following: \n csvs-to-sqlite items.csv items.db -f name -f description", "sections_fts": 7, "rank": null} {"rowid": 395, "title": "Configuring FTS by hand", "content": "We recommend using sqlite-utils , but if you want to hand-roll a SQLite full-text search table you can do so using the following SQL. \n To enable full-text search for a table called items that works against the name and description columns, you would run this SQL to create a new items_fts FTS virtual table: \n CREATE VIRTUAL TABLE \"items_fts\" USING FTS4 (\n name,\n description,\n content=\"items\"\n); \n This creates a set of tables to power full-text search against items . The new items_fts table will be detected by Datasette as the fts_table for the items table. \n Creating the table is not enough: you also need to populate it with a copy of the data that you wish to make searchable. You can do that using the following SQL: \n INSERT INTO \"items_fts\" (rowid, name, description)\n SELECT rowid, name, description FROM items; \n If your table has columns that are foreign key references to other tables you can include that data in your full-text search index using a join. Imagine the items table has a foreign key column called category_id which refers to a categories table - you could create a full-text search table like this: \n CREATE VIRTUAL TABLE \"items_fts\" USING FTS4 (\n name,\n description,\n category_name,\n content=\"items\"\n); \n And then populate it like this: \n INSERT INTO \"items_fts\" (rowid, name, description, category_name)\n SELECT items.rowid,\n items.name,\n items.description,\n categories.name\n FROM items JOIN categories ON items.category_id=categories.id; \n You can use this technique to populate the full-text search index from any combination of tables and joins that makes sense for your project.", "sections_fts": 7, "rank": null} {"rowid": 396, "title": "FTS versions", "content": "There are three different versions of the SQLite FTS module: FTS3, FTS4 and FTS5. You can tell which versions are supported by your instance of Datasette by checking the /-/versions page. \n FTS5 is the most advanced module but may not be available in the SQLite version that is bundled with your Python installation. Most importantly, FTS5 is the only version that has the ability to order by search relevance without needing extra code. \n If you can't be sure that FTS5 will be available, you should use FTS4.", "sections_fts": 7, "rank": null} {"rowid": 397, "title": "Settings", "content": "", "sections_fts": 7, "rank": null} {"rowid": 398, "title": "Using --setting", "content": "Datasette supports a number of settings. These can be set using the --setting name value option to datasette serve . \n You can set multiple settings at once like this: \n datasette mydatabase.db \\\n --setting default_page_size 50 \\\n --setting sql_time_limit_ms 3500 \\\n --setting max_returned_rows 2000 \n Settings can also be specified in the database.yaml configuration file .", "sections_fts": 7, "rank": null} {"rowid": 399, "title": "Configuration directory mode", "content": "Normally you configure Datasette using command-line options. For a Datasette instance with custom templates, custom plugins, a static directory and several databases this can get quite verbose: \n datasette one.db two.db \\\n --metadata=metadata.json \\\n --template-dir=templates/ \\\n --plugins-dir=plugins \\\n --static css:css \n As an alternative to this, you can run Datasette in configuration directory mode. Create a directory with the following structure: \n # In a directory called my-app:\nmy-app/one.db\nmy-app/two.db\nmy-app/datasette.yaml\nmy-app/metadata.json\nmy-app/templates/index.html\nmy-app/plugins/my_plugin.py\nmy-app/static/my.css \n Now start Datasette by providing the path to that directory: \n datasette my-app/ \n Datasette will detect the files in that directory and automatically configure itself using them. It will serve all *.db files that it finds, will load metadata.json if it exists, and will load the templates , plugins and static folders if they are present. \n The files that can be included in this directory are as follows. All are optional. \n \n \n *.db (or *.sqlite3 or *.sqlite ) - SQLite database files that will be served by Datasette \n \n \n datasette.yaml - Configuration for the Datasette instance \n \n \n metadata.json - Metadata for those databases - metadata.yaml or metadata.yml can be used as well \n \n \n inspect-data.json - the result of running datasette inspect *.db --inspect-file=inspect-data.json from the configuration directory - any database files listed here will be treated as immutable, so they should not be changed while Datasette is running \n \n \n templates/ - a directory containing Custom templates \n \n \n plugins/ - a directory containing plugins, see Writing one-off plugins \n \n \n static/ - a directory containing static files - these will be served from /static/filename.txt , see Serving static files", "sections_fts": 7, "rank": null} {"rowid": 400, "title": "Settings", "content": "The following options can be set using --setting name value , or by storing them in the settings.json file for use with Configuration directory mode .", "sections_fts": 7, "rank": null}