id,page,ref,title,content,breadcrumbs,references
internals:internals-utils-parse-metadata,internals,internals-utils-parse-metadata,parse_metadata(content),"This function accepts a string containing either JSON or YAML, expected to be of the format described in Metadata . It returns a nested Python dictionary representing the parsed data from that string.
If the metadata cannot be parsed as either JSON or YAML the function will raise a utils.BadMetadataError exception.
datasette.utils. parse_metadata content : str dict
Detects if content is JSON or YAML and parses it appropriately.","[""Internals for plugins"", ""The datasette.utils module""]",[]
internals:internals-utils-derive-named-parameters,internals,internals-utils-derive-named-parameters,"derive_named_parameters(db, sql)","Derive the list of named parameters referenced in a SQL query, using an explain query executed against the provided database.
async datasette.utils. derive_named_parameters db : Database sql : str List [ str ]
Given a SQL statement, return a list of named parameters that are used in the statement
e.g. for select * from foo where id=:id this would return [""id""]","[""Internals for plugins"", ""The datasette.utils module""]",[]
internals:internals-utils-await-me-maybe,internals,internals-utils-await-me-maybe,await_me_maybe(value),"Utility function for calling await on a return value if it is awaitable, otherwise returning the value. This is used by Datasette to support plugin hooks that can optionally return awaitable functions. Read more about this function in The “await me maybe” pattern for Python asyncio .
async datasette.utils. await_me_maybe value : Any Any
If value is callable, call it. If awaitable, await it. Otherwise return it.","[""Internals for plugins"", ""The datasette.utils module""]","[{""href"": ""https://simonwillison.net/2020/Sep/2/await-me-maybe/"", ""label"": ""The “await me maybe” pattern for Python asyncio""}]"
internals:internals-utils,internals,internals-utils,The datasette.utils module,"The datasette.utils module contains various utility functions used by Datasette. As a general rule you should consider anything in this module to be unstable - functions and classes here could change without warning or be removed entirely between Datasette releases, without being mentioned in the release notes.
The exception to this rule is anything that is documented here. If you find a need for an undocumented utility function in your own work, consider opening an issue requesting that the function you are using be upgraded to documented and supported status.","[""Internals for plugins""]","[{""href"": ""https://github.com/simonw/datasette/issues/new"", ""label"": ""opening an issue""}]"
internals:internals-tracer-trace-child-tasks,internals,internals-tracer-trace-child-tasks,Tracing child tasks,"If your code uses a mechanism such as asyncio.gather() to execute code in additional tasks you may find that some of the traces are missing from the display.
You can use the trace_child_tasks() context manager to ensure these child tasks are correctly handled.
from datasette import tracer
with tracer.trace_child_tasks():
results = await asyncio.gather(
# ... async tasks here
)
This example uses the register_routes() plugin hook to add a page at /parallel-queries which executes two SQL queries in parallel using asyncio.gather() and returns their results.
from datasette import hookimpl
from datasette import tracer
@hookimpl
def register_routes():
async def parallel_queries(datasette):
db = datasette.get_database()
with tracer.trace_child_tasks():
one, two = await asyncio.gather(
db.execute(""select 1""),
db.execute(""select 2""),
)
return Response.json(
{
""one"": one.single_value(),
""two"": two.single_value(),
}
)
return [
(r""/parallel-queries$"", parallel_queries),
]
Note that running parallel SQL queries in this way has been known to cause problems in the past , so treat this example with caution.
Adding ?_trace=1 will show that the trace covers both of those child tasks.","[""Internals for plugins"", ""datasette.tracer""]","[{""href"": ""https://github.com/simonw/datasette/issues/2189"", ""label"": ""been known to cause problems in the past""}]"
internals:internals-tracer,internals,internals-tracer,datasette.tracer,"Running Datasette with --setting trace_debug 1 enables trace debug output, which can then be viewed by adding ?_trace=1 to the query string for any page.
You can see an example of this at the bottom of latest.datasette.io/fixtures/facetable?_trace=1 . The JSON output shows full details of every SQL query that was executed to generate the page.
The datasette-pretty-traces plugin can be installed to provide a more readable display of this information. You can see a demo of that here .
You can add your own custom traces to the JSON output using the trace() context manager. This takes a string that identifies the type of trace being recorded, and records any keyword arguments as additional JSON keys on the resulting trace object.
The start and end time, duration and a traceback of where the trace was executed will be automatically attached to the JSON object.
This example uses trace to record the start, end and duration of any HTTP GET requests made using the function:
from datasette.tracer import trace
import httpx
async def fetch_url(url):
with trace(""fetch-url"", url=url):
async with httpx.AsyncClient() as client:
return await client.get(url)","[""Internals for plugins""]","[{""href"": ""https://latest.datasette.io/fixtures/facetable?_trace=1"", ""label"": ""latest.datasette.io/fixtures/facetable?_trace=1""}, {""href"": ""https://datasette.io/plugins/datasette-pretty-traces"", ""label"": ""datasette-pretty-traces""}, {""href"": ""https://latest-with-plugins.datasette.io/github/commits?_trace=1"", ""label"": ""a demo of that here""}]"
internals:internals-tilde-encoding,internals,internals-tilde-encoding,Tilde encoding,"Datasette uses a custom encoding scheme in some places, called tilde encoding . This is primarily used for table names and row primary keys, to avoid any confusion between / characters in those values and the Datasette URLs that reference them.
Tilde encoding uses the same algorithm as URL percent-encoding , but with the ~ tilde character used in place of % .
Any character other than ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz0123456789_- will be replaced by the numeric equivalent preceded by a tilde. For example:
/ becomes ~2F
. becomes ~2E
% becomes ~25
~ becomes ~7E
Space becomes +
polls/2022.primary becomes polls~2F2022~2Eprimary
Note that the space character is a special case: it will be replaced with a + symbol.
datasette.utils. tilde_encode s : str str
Returns tilde-encoded string - for example /foo/bar -> ~2Ffoo~2Fbar
datasette.utils. tilde_decode s : str str
Decodes a tilde-encoded string, so ~2Ffoo~2Fbar -> /foo/bar","[""Internals for plugins"", ""The datasette.utils module""]","[{""href"": ""https://developer.mozilla.org/en-US/docs/Glossary/percent-encoding"", ""label"": ""URL percent-encoding""}]"
internals:internals-shortcuts,internals,internals-shortcuts,Import shortcuts,"The following commonly used symbols can be imported directly from the datasette module:
from datasette import Response
from datasette import Forbidden
from datasette import NotFound
from datasette import hookimpl
from datasette import actor_matches_allow","[""Internals for plugins""]",[]
internals:internals-response-set-cookie,internals,internals-response-set-cookie,Setting cookies with response.set_cookie(),"To set cookies on the response, use the response.set_cookie(...) method. The method signature looks like this:
def set_cookie(
self,
key,
value="""",
max_age=None,
expires=None,
path=""/"",
domain=None,
secure=False,
httponly=False,
samesite=""lax"",
): ...
You can use this with datasette.sign() to set signed cookies. Here's how you would set the ds_actor cookie for use with Datasette authentication :
response = Response.redirect(""/"")
response.set_cookie(
""ds_actor"",
datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor""),
)
return response","[""Internals for plugins"", ""Response class""]",[]
internals:internals-response-asgi-send,internals,internals-response-asgi-send,Returning a response with .asgi_send(send),"In most cases you will return Response objects from your own view functions. You can also use a Response instance to respond at a lower level via ASGI, for example if you are writing code that uses the asgi_wrapper(datasette) hook.
Create a Response object and then use await response.asgi_send(send) , passing the ASGI send function. For example:
async def require_authorization(scope, receive, send):
response = Response.text(
""401 Authorization Required"",
headers={
""www-authenticate"": 'Basic realm=""Datasette"", charset=""UTF-8""'
},
status=401,
)
await response.asgi_send(send)","[""Internals for plugins"", ""Response class""]",[]
internals:internals-response,internals,internals-response,Response class,"The Response class can be returned from view functions that have been registered using the register_routes(datasette) hook.
The Response() constructor takes the following arguments:
body - string
The body of the response.
status - integer (optional)
The HTTP status - defaults to 200.
headers - dictionary (optional)
A dictionary of extra HTTP headers, e.g. {""x-hello"": ""world""} .
content_type - string (optional)
The content-type for the response. Defaults to text/plain .
For example:
from datasette.utils.asgi import Response
response = Response(
""This is XML"",
content_type=""application/xml; charset=utf-8"",
)
The quickest way to create responses is using the Response.text(...) , Response.html(...) , Response.json(...) or Response.redirect(...) helper methods:
from datasette.utils.asgi import Response
html_response = Response.html(""This is HTML"")
json_response = Response.json({""this_is"": ""json""})
text_response = Response.text(
""This will become utf-8 encoded text""
)
# Redirects are served as 302, unless you pass status=301:
redirect_response = Response.redirect(
""https://latest.datasette.io/""
)
Each of these responses will use the correct corresponding content-type - text/html; charset=utf-8 , application/json; charset=utf-8 or text/plain; charset=utf-8 respectively.
Each of the helper methods take optional status= and headers= arguments, documented above.","[""Internals for plugins""]",[]
internals:internals-request,internals,internals-request,Request object,"The request object is passed to various plugin hooks. It represents an incoming HTTP request. It has the following properties:
.scope - dictionary
The ASGI scope that was used to construct this request, described in the ASGI HTTP connection scope specification.
.method - string
The HTTP method for this request, usually GET or POST .
.url - string
The full URL for this request, e.g. https://latest.datasette.io/fixtures .
.scheme - string
The request scheme - usually https or http .
.headers - dictionary (str -> str)
A dictionary of incoming HTTP request headers. Header names have been converted to lowercase.
.cookies - dictionary (str -> str)
A dictionary of incoming cookies
.host - string
The host header from the incoming request, e.g. latest.datasette.io or localhost .
.path - string
The path of the request excluding the query string, e.g. /fixtures .
.full_path - string
The path of the request including the query string if one is present, e.g. /fixtures?sql=select+sqlite_version() .
.query_string - string
The query string component of the request, without the ? - e.g. name__contains=sam&age__gt=10 .
.args - MultiParams
An object representing the parsed query string parameters, see below.
.url_vars - dictionary (str -> str)
Variables extracted from the URL path, if that path was defined using a regular expression. See register_routes(datasette) .
.actor - dictionary (str -> Any) or None
The currently authenticated actor (see actors ), or None if the request is unauthenticated.
The object also has two awaitable methods:
await request.post_vars() - dictionary
Returns a dictionary of form variables that were submitted in the request body via POST . Don't forget to read about CSRF protection !
await request.post_body() - bytes
Returns the un-parsed body of a request submitted by POST - useful for things like incoming JSON data.
And a class method that can be used to create fake request objects for use in tests:
fake(path_with_query_string, method=""GET"", scheme=""http"", url_vars=None)
Returns a Request instance for the specified path and method. For example:
from datasette import Request
from pprint import pprint
request = Request.fake(
""/fixtures/facetable/"",
url_vars={""database"": ""fixtures"", ""table"": ""facetable""},
)
pprint(request.scope)
This outputs:
{'http_version': '1.1',
'method': 'GET',
'path': '/fixtures/facetable/',
'query_string': b'',
'raw_path': b'/fixtures/facetable/',
'scheme': 'http',
'type': 'http',
'url_route': {'kwargs': {'database': 'fixtures', 'table': 'facetable'}}}","[""Internals for plugins""]","[{""href"": ""https://asgi.readthedocs.io/en/latest/specs/www.html#connection-scope"", ""label"": ""ASGI HTTP connection scope""}]"
internals:internals-multiparams,internals,internals-multiparams,The MultiParams class,"request.args is a MultiParams object - a dictionary-like object which provides access to query string parameters that may have multiple values.
Consider the query string ?foo=1&foo=2&bar=3 - with two values for foo and one value for bar .
request.args[key] - string
Returns the first value for that key, or raises a KeyError if the key is missing. For the above example request.args[""foo""] would return ""1"" .
request.args.get(key) - string or None
Returns the first value for that key, or None if the key is missing. Pass a second argument to specify a different default, e.g. q = request.args.get(""q"", """") .
request.args.getlist(key) - list of strings
Returns the list of strings for that key. request.args.getlist(""foo"") would return [""1"", ""2""] in the above example. request.args.getlist(""bar"") would return [""3""] . If the key is missing an empty list will be returned.
request.args.keys() - list of strings
Returns the list of available keys - for the example this would be [""foo"", ""bar""] .
key in request.args - True or False
You can use if key in request.args to check if a key is present.
for key in request.args - iterator
This lets you loop through every available key.
len(request.args) - integer
Returns the number of keys.","[""Internals for plugins""]",[]
internals:internals-internal,internals,internals-internal,Datasette's internal database,"Datasette maintains an ""internal"" SQLite database used for configuration, caching, and storage. Plugins can store configuration, settings, and other data inside this database. By default, Datasette will use a temporary in-memory SQLite database as the internal database, which is created at startup and destroyed at shutdown. Users of Datasette can optionally pass in a --internal flag to specify the path to a SQLite database to use as the internal database, which will persist internal data across Datasette instances.
Datasette maintains tables called catalog_databases , catalog_tables , catalog_columns , catalog_indexes , catalog_foreign_keys with details of the attached databases and their schemas. These tables should not be considered a stable API - they may change between Datasette releases.
The internal database is not exposed in the Datasette application by default, which means private data can safely be stored without worry of accidentally leaking information through the default Datasette interface and API. However, other plugins do have full read and write access to the internal database.
Plugins can access this database by calling internal_db = datasette.get_internal_database() and then executing queries using the Database API .
Plugin authors are asked to practice good etiquette when using the internal database, as all plugins use the same database to store data. For example:
Use a unique prefix when creating tables, indices, and triggers in the internal database. If your plugin is called datasette-xyz , then prefix names with datasette_xyz_* .
Avoid long-running write statements that may stall or block other plugins that are trying to write at the same time.
Use temporary tables or shared in-memory attached databases when possible.
Avoid implementing features that could expose private data stored in the internal database by other plugins.","[""Internals for plugins""]",[]
internals:internals-datasette-urls,internals,internals-datasette-urls,datasette.urls,"The datasette.urls object contains methods for building URLs to pages within Datasette. Plugins should use this to link to pages, since these methods take into account any base_url configuration setting that might be in effect.
datasette.urls.instance(format=None)
Returns the URL to the Datasette instance root page. This is usually ""/"" .
datasette.urls.path(path, format=None)
Takes a path and returns the full path, taking base_url into account.
For example, datasette.urls.path(""-/logout"") will return the path to the logout page, which will be ""/-/logout"" by default or /prefix-path/-/logout if base_url is set to /prefix-path/
datasette.urls.logout()
Returns the URL to the logout page, usually ""/-/logout""
datasette.urls.static(path)
Returns the URL of one of Datasette's default static assets, for example ""/-/static/app.css""
datasette.urls.static_plugins(plugin_name, path)
Returns the URL of one of the static assets belonging to a plugin.
datasette.urls.static_plugins(""datasette_cluster_map"", ""datasette-cluster-map.js"") would return ""/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js""
datasette.urls.static(path)
Returns the URL of one of Datasette's default static assets, for example ""/-/static/app.css""
datasette.urls.database(database_name, format=None)
Returns the URL to a database page, for example ""/fixtures""
datasette.urls.table(database_name, table_name, format=None)
Returns the URL to a table page, for example ""/fixtures/facetable""
datasette.urls.query(database_name, query_name, format=None)
Returns the URL to a query page, for example ""/fixtures/pragma_cache_size""
These functions can be accessed via the {{ urls }} object in Datasette templates, for example:
Homepage
Fixtures database
facetable table
pragma_cache_size query
Use the format=""json"" (or ""csv"" or other formats supported by plugins) arguments to get back URLs to the JSON representation. This is the path with .json added on the end.
These methods each return a datasette.utils.PrefixedUrlString object, which is a subclass of the Python str type. This allows the logic that considers the base_url setting to detect if that prefix has already been applied to the path.","[""Internals for plugins"", ""Datasette class""]",[]
internals:internals-datasette-client,internals,internals-datasette-client,datasette.client,"Plugins can make internal simulated HTTP requests to the Datasette instance within which they are running. This ensures that all of Datasette's external JSON APIs are also available to plugins, while avoiding the overhead of making an external HTTP call to access those APIs.
The datasette.client object is a wrapper around the HTTPX Python library , providing an async-friendly API that is similar to the widely used Requests library .
It offers the following methods:
await datasette.client.get(path, **kwargs) - returns HTTPX Response
Execute an internal GET request against that path.
await datasette.client.post(path, **kwargs) - returns HTTPX Response
Execute an internal POST request. Use data={""name"": ""value""} to pass form parameters.
await datasette.client.options(path, **kwargs) - returns HTTPX Response
Execute an internal OPTIONS request.
await datasette.client.head(path, **kwargs) - returns HTTPX Response
Execute an internal HEAD request.
await datasette.client.put(path, **kwargs) - returns HTTPX Response
Execute an internal PUT request.
await datasette.client.patch(path, **kwargs) - returns HTTPX Response
Execute an internal PATCH request.
await datasette.client.delete(path, **kwargs) - returns HTTPX Response
Execute an internal DELETE request.
await datasette.client.request(method, path, **kwargs) - returns HTTPX Response
Execute an internal request with the given HTTP method against that path.
These methods can be used with datasette.urls - for example:
table_json = (
await datasette.client.get(
datasette.urls.table(
""fixtures"", ""facetable"", format=""json""
)
)
).json()
datasette.client methods automatically take the current base_url setting into account, whether or not you use the datasette.urls family of methods to construct the path.
For documentation on available **kwargs options and the shape of the HTTPX Response object refer to the HTTPX Async documentation .","[""Internals for plugins"", ""Datasette class""]","[{""href"": ""https://www.python-httpx.org/"", ""label"": ""HTTPX Python library""}, {""href"": ""https://requests.readthedocs.io/"", ""label"": ""Requests library""}, {""href"": ""https://www.python-httpx.org/async/"", ""label"": ""HTTPX Async documentation""}]"
internals:internals-datasette,internals,internals-datasette,Datasette class,"This object is an instance of the Datasette class, passed to many plugin hooks as an argument called datasette .
You can create your own instance of this - for example to help write tests for a plugin - like so:
from datasette.app import Datasette
# With no arguments a single in-memory database will be attached
datasette = Datasette()
# The files= argument can load files from disk
datasette = Datasette(files=[""/path/to/my-database.db""])
# Pass metadata as a JSON dictionary like this
datasette = Datasette(
files=[""/path/to/my-database.db""],
metadata={
""databases"": {
""my-database"": {
""description"": ""This is my database""
}
}
},
)
Constructor parameters include:
files=[...] - a list of database files to open
immutables=[...] - a list of database files to open in immutable mode
metadata={...} - a dictionary of Metadata
config_dir=... - the configuration directory to use, stored in datasette.config_dir","[""Internals for plugins""]",[]
internals:internals-database-introspection,internals,internals-database-introspection,Database introspection,"The Database class also provides properties and methods for introspecting the database.
db.name - string
The name of the database - usually the filename without the .db prefix.
db.size - integer
The size of the database file in bytes. 0 for :memory: databases.
db.mtime_ns - integer or None
The last modification time of the database file in nanoseconds since the epoch. None for :memory: databases.
db.is_mutable - boolean
Is this database mutable, and allowed to accept writes?
db.is_memory - boolean
Is this database an in-memory database?
await db.attached_databases() - list of named tuples
Returns a list of additional databases that have been connected to this database using the SQLite ATTACH command. Each named tuple has fields seq , name and file .
await db.table_exists(table) - boolean
Check if a table called table exists.
await db.view_exists(view) - boolean
Check if a view called view exists.
await db.table_names() - list of strings
List of names of tables in the database.
await db.view_names() - list of strings
List of names of views in the database.
await db.table_columns(table) - list of strings
Names of columns in a specific table.
await db.table_column_details(table) - list of named tuples
Full details of the columns in a specific table. Each column is represented by a Column named tuple with fields cid (integer representing the column position), name (string), type (string, e.g. REAL or VARCHAR(30) ), notnull (integer 1 or 0), default_value (string or None), is_pk (integer 1 or 0).
await db.primary_keys(table) - list of strings
Names of the columns that are part of the primary key for this table.
await db.fts_table(table) - string or None
The name of the FTS table associated with this table, if one exists.
await db.label_column_for_table(table) - string or None
The label column that is associated with this table - either automatically detected or using the ""label_column"" key from Metadata , see Specifying the label column for a table .
await db.foreign_keys_for_table(table) - list of dictionaries
Details of columns in this table which are foreign keys to other tables. A list of dictionaries where each dictionary is shaped like this: {""column"": string, ""other_table"": string, ""other_column"": string} .
await db.hidden_table_names() - list of strings
List of tables which Datasette ""hides"" by default - usually these are tables associated with SQLite's full-text search feature, the SpatiaLite extension or tables hidden using the Hiding tables feature.
await db.get_table_definition(table) - string
Returns the SQL definition for the table - the CREATE TABLE statement and any associated CREATE INDEX statements.
await db.get_view_definition(view) - string
Returns the SQL definition of the named view.
await db.get_all_foreign_keys() - dictionary
Dictionary representing both incoming and outgoing foreign keys for this table. It has two keys, ""incoming"" and ""outgoing"" , each of which is a list of dictionaries with keys ""column"" , ""other_table"" and ""other_column"" . For example:
{
""incoming"": [],
""outgoing"": [
{
""other_table"": ""attraction_characteristic"",
""column"": ""characteristic_id"",
""other_column"": ""pk"",
},
{
""other_table"": ""roadside_attractions"",
""column"": ""attraction_id"",
""other_column"": ""pk"",
}
]
}","[""Internals for plugins"", ""Database class""]",[]
internals:internals-database,internals,internals-database,Database class,"Instances of the Database class can be used to execute queries against attached SQLite databases, and to run introspection against their schemas.","[""Internals for plugins""]",[]
internals:internals-csrf,internals,internals-csrf,CSRF protection,"Datasette uses asgi-csrf to guard against CSRF attacks on form POST submissions. Users receive a ds_csrftoken cookie which is compared against the csrftoken form field (or x-csrftoken HTTP header) for every incoming request.
If your plugin implements a