{"id": "full_text_search:configuring-fts-by-hand", "page": "full_text_search", "ref": "configuring-fts-by-hand", "title": "Configuring FTS by hand", "content": "We recommend using  sqlite-utils , but if you want to hand-roll a SQLite full-text search table you can do so using the following SQL. \n                 To enable full-text search for a table called  items  that works against the  name  and  description  columns, you would run this SQL to create a new  items_fts  FTS virtual table: \n                 CREATE VIRTUAL TABLE \"items_fts\" USING FTS4 (\n    name,\n    description,\n    content=\"items\"\n); \n                 This creates a set of tables to power full-text search against  items . The new  items_fts  table will be detected by Datasette as the  fts_table  for the  items  table. \n                 Creating the table is not enough: you also need to populate it with a copy of the data that you wish to make searchable. You can do that using the following SQL: \n                 INSERT INTO \"items_fts\" (rowid, name, description)\n    SELECT rowid, name, description FROM items; \n                 If your table has columns that are foreign key references to other tables you can include that data in your full-text search index using a join. Imagine the  items  table has a foreign key column called  category_id  which refers to a  categories  table - you could create a full-text search table like this: \n                 CREATE VIRTUAL TABLE \"items_fts\" USING FTS4 (\n    name,\n    description,\n    category_name,\n    content=\"items\"\n); \n                 And then populate it like this: \n                 INSERT INTO \"items_fts\" (rowid, name, description, category_name)\n    SELECT items.rowid,\n    items.name,\n    items.description,\n    categories.name\n    FROM items JOIN categories ON items.category_id=categories.id; \n                 You can use this technique to populate the full-text search index from any combination of tables and joins that makes sense for your project.", "breadcrumbs": "[\"Full-text search\", \"Enabling full-text search for a SQLite table\"]", "references": "[{\"href\": \"https://sqlite-utils.datasette.io/\", \"label\": \"sqlite-utils\"}]"}
{"id": "full_text_search:configuring-fts-using-csvs-to-sqlite", "page": "full_text_search", "ref": "configuring-fts-using-csvs-to-sqlite", "title": "Configuring FTS using csvs-to-sqlite", "content": "If your data starts out in CSV files, you can use Datasette's companion tool  csvs-to-sqlite  to convert that file into a SQLite database and enable full-text search on specific columns. For a file called  items.csv  where you want full-text search to operate against the  name  and  description  columns you would run the following: \n                 csvs-to-sqlite items.csv items.db -f name -f description", "breadcrumbs": "[\"Full-text search\", \"Enabling full-text search for a SQLite table\"]", "references": "[{\"href\": \"https://github.com/simonw/csvs-to-sqlite\", \"label\": \"csvs-to-sqlite\"}]"}
{"id": "full_text_search:configuring-fts-using-sqlite-utils", "page": "full_text_search", "ref": "configuring-fts-using-sqlite-utils", "title": "Configuring FTS using sqlite-utils", "content": "sqlite-utils  is a CLI utility and Python library for manipulating SQLite databases. You can use  it from Python code  to configure FTS search, or you can achieve the same goal  using the accompanying command-line tool . \n                 Here's how to use  sqlite-utils  to enable full-text search for an  items  table across the  name  and  description  columns: \n                 sqlite-utils enable-fts mydatabase.db items name description", "breadcrumbs": "[\"Full-text search\", \"Enabling full-text search for a SQLite table\"]", "references": "[{\"href\": \"https://sqlite-utils.datasette.io/\", \"label\": \"sqlite-utils\"}, {\"href\": \"https://sqlite-utils.datasette.io/en/latest/python-api.html#enabling-full-text-search\", \"label\": \"it from Python code\"}, {\"href\": \"https://sqlite-utils.datasette.io/en/latest/cli.html#configuring-full-text-search\", \"label\": \"using the accompanying command-line tool\"}]"}
{"id": "full_text_search:full-text-search-advanced-queries", "page": "full_text_search", "ref": "full-text-search-advanced-queries", "title": "Advanced SQLite search queries", "content": "SQLite full-text search includes support for  a variety of advanced queries , including  AND ,  OR ,  NOT  and  NEAR . \n             By default Datasette disables these features to ensure they do not cause errors or confusion for users who are not aware of them. You can disable this escaping and use the advanced queries by adding  &_searchmode=raw  to the table page query string. \n             If you want to enable these operators by default for a specific table, you can do so by adding  \"searchmode\": \"raw\"  to the metadata configuration for that table, see  Configuring full-text search for a table or view . \n             If that option has been specified in the table metadata but you want to over-ride it and return to the default behavior you can append  &_searchmode=escaped  to the query string.", "breadcrumbs": "[\"Full-text search\"]", "references": "[{\"href\": \"https://www.sqlite.org/fts5.html#full_text_query_syntax\", \"label\": \"a variety of advanced queries\"}]"}
{"id": "full_text_search:full-text-search-custom-sql", "page": "full_text_search", "ref": "full-text-search-custom-sql", "title": "Searches using custom SQL", "content": "You can include full-text search results in custom SQL queries. The general pattern with SQLite search is to run the search as a sub-select that returns rowid values, then include those rowids in another part of the query. \n             You can see the syntax for a basic search by running that search on a table page and then clicking \"View and edit SQL\" to see the underlying SQL. For example, consider this search for  manafort is the US FARA database : \n             /fara/FARA_All_ShortForms?_search=manafort \n             If you click  View and edit SQL  you'll see that the underlying SQL looks like this: \n             select\n  rowid,\n  Short_Form_Termination_Date,\n  Short_Form_Date,\n  Short_Form_Last_Name,\n  Short_Form_First_Name,\n  Registration_Number,\n  Registration_Date,\n  Registrant_Name,\n  Address_1,\n  Address_2,\n  City,\n  State,\n  Zip\nfrom\n  FARA_All_ShortForms\nwhere\n  rowid in (\n    select\n      rowid\n    from\n      FARA_All_ShortForms_fts\n    where\n      FARA_All_ShortForms_fts match escape_fts(:search)\n  )\norder by\n  rowid\nlimit\n  101", "breadcrumbs": "[\"Full-text search\"]", "references": "[{\"href\": \"https://fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort\", \"label\": \"manafort is the US FARA database\"}, {\"href\": \"https://fara.datasettes.com/fara?sql=select%0D%0A++rowid%2C%0D%0A++Short_Form_Termination_Date%2C%0D%0A++Short_Form_Date%2C%0D%0A++Short_Form_Last_Name%2C%0D%0A++Short_Form_First_Name%2C%0D%0A++Registration_Number%2C%0D%0A++Registration_Date%2C%0D%0A++Registrant_Name%2C%0D%0A++Address_1%2C%0D%0A++Address_2%2C%0D%0A++City%2C%0D%0A++State%2C%0D%0A++Zip%0D%0Afrom%0D%0A++FARA_All_ShortForms%0D%0Awhere%0D%0A++rowid+in+%28%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++FARA_All_ShortForms_fts%0D%0A++++where%0D%0A++++++FARA_All_ShortForms_fts+match+escape_fts%28%3Asearch%29%0D%0A++%29%0D%0Aorder+by%0D%0A++rowid%0D%0Alimit%0D%0A++101&search=manafort\", \"label\": \"View and edit SQL\"}]"}
{"id": "full_text_search:full-text-search-enabling", "page": "full_text_search", "ref": "full-text-search-enabling", "title": "Enabling full-text search for a SQLite table", "content": "Datasette takes advantage of the  external content  mechanism in SQLite, which allows a full-text search virtual table to be associated with the contents of another SQLite table. \n             To set up full-text search for a table, you need to do two things: \n             \n                 \n                     Create a new FTS virtual table associated with your table \n                 \n                 \n                     Populate that FTS table with the data that you would like to be able to run searches against", "breadcrumbs": "[\"Full-text search\"]", "references": "[{\"href\": \"https://www.sqlite.org/fts3.html#_external_content_fts4_tables_\", \"label\": \"external content\"}]"}
{"id": "full_text_search:full-text-search-fts-versions", "page": "full_text_search", "ref": "full-text-search-fts-versions", "title": "FTS versions", "content": "There are three different versions of the SQLite FTS module: FTS3, FTS4 and FTS5. You can tell which versions are supported by your instance of Datasette by checking the  /-/versions  page. \n             FTS5 is the most advanced module but may not be available in the SQLite version that is bundled with your Python installation. Most importantly, FTS5 is the only version that has the ability to order by search relevance without needing extra code. \n             If you can't be sure that FTS5 will be available, you should use FTS4.", "breadcrumbs": "[\"Full-text search\"]", "references": "[]"}
{"id": "full_text_search:full-text-search-table-or-view", "page": "full_text_search", "ref": "full-text-search-table-or-view", "title": "Configuring full-text search for a table or view", "content": "If a table has a corresponding FTS table set up using the  content=  argument to  CREATE VIRTUAL TABLE  shown below, Datasette will detect it automatically and add a search interface to the table page for that table. \n             You can also manually configure which table should be used for full-text search using query string parameters or  Metadata . You can set the associated FTS table for a specific table and you can also set one for a view - if you do that, the page for that SQL view will offer a search option. \n             Use  ?_fts_table=x  to over-ride the FTS table for a specific page. If the primary key was something other than  rowid  you can use  ?_fts_pk=col  to set that as well. This is particularly useful for views, for example: \n             https://latest.datasette.io/fixtures/searchable_view?_fts_table=searchable_fts&_fts_pk=pk \n             The  fts_table  metadata property can be used to specify an associated FTS table. If the primary key column in your table which was used to populate the FTS table is something other than  rowid , you can specify the column to use with the  fts_pk  property. \n             The  \"searchmode\": \"raw\"  property can be used to default the table to accepting SQLite advanced search operators, as described in  Advanced SQLite search queries . \n             Here is an example which enables full-text search (with SQLite advanced search operators) for a  display_ads  view which is defined against the  ads  table and hence needs to run FTS against the  ads_fts  table, using the  id  as the primary key: \n             [[[cog\nfrom metadata_doc import metadata_example\nmetadata_example(cog, {\n    \"databases\": {\n        \"russian-ads\": {\n            \"tables\": {\n                \"display_ads\": {\n                    \"fts_table\": \"ads_fts\",\n                    \"fts_pk\": \"id\",\n                    \"searchmode\": \"raw\"\n                }\n            }\n        }\n    }\n}) \n             ]]] \n             [[[end]]]", "breadcrumbs": "[\"Full-text search\"]", "references": "[{\"href\": \"https://latest.datasette.io/fixtures/searchable_view?_fts_table=searchable_fts&_fts_pk=pk\", \"label\": \"https://latest.datasette.io/fixtures/searchable_view?_fts_table=searchable_fts&_fts_pk=pk\"}]"}
{"id": "full_text_search:full-text-search-table-view-api", "page": "full_text_search", "ref": "full-text-search-table-view-api", "title": "The table page and table view API", "content": "Table views that support full-text search can be queried using the  ?_search=TERMS  query string parameter. This will run the search against content from all of the columns that have been included in the index. \n             Try this example:  fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort \n             SQLite full-text search supports wildcards. This means you can easily implement prefix auto-complete by including an asterisk at the end of the search term - for example: \n             /dbname/tablename/?_search=rob* \n             This will return all records containing at least one word that starts with the letters  rob . \n             You can also run searches against just the content of a specific named column by using  _search_COLNAME=TERMS  - for example, this would search for just rows where the  name  column in the FTS index mentions  Sarah : \n             /dbname/tablename/?_search_name=Sarah", "breadcrumbs": "[\"Full-text search\"]", "references": "[{\"href\": \"https://fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort\", \"label\": \"fara.datasettes.com/fara/FARA_All_ShortForms?_search=manafort\"}]"}