Changes between Initial Version and Version 1 of ChangingTheDatabaseStructure


Ignore:
Timestamp:
Jan 7, 2013, 6:26:02 PM (8 years ago)
Author:
flip
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ChangingTheDatabaseStructure

    v1 v1  
     1= Changing Vespa's Database Structure =
     2
     3As of this writing (January 2013), Vespa's database format has changed seven
     4times. This document explains how to change the database in the future and
     5points to some of the old changes as examples.
     6
     7
     8== First Steps ==
     9
     10Vespa's database is very forgiving in that it's easy to recreate locally.
     11This allows for lots of experimentation before you share your changes with
     12others.
     13
     14Vespa's database is defined by
     15[browser:trunk/common/resources/create_tables.sql create_tables.sql].
     16It creates the database structure that's later populated by Python code.
     17Your first step in modifying the database will be to change that file.
     18
     19There are two other files in that same directory --
     20[browser:trunk/common/resources/create_indices.sql create_indices.sql]
     21and [browser:trunk/common/resources/create_views.sql create_views.sql] -- that
     22build on the output of `create_tables.sql`. Creating indices speeds up access
     23to frequently-accessed items at the expense of insert speed, and views are
     24sometimes useful but at present we're not using any so `create_views.sql`
     25is empty.
     26
     27You probably won't need to alter `create_indices.sql` nor `create_views.sql`.
     28
     29Once you've modified `create_tables.sql`, you can use
     30[browser:trunk/devtools/recreate_database.py recreate_database.py] to rebuild
     31the database or just delete your existing database and then run a Vespa app.
     32This is sufficient for local changes (i.e. testing on your machine) and you'll
     33probably repeat this cycle many times.
     34
     35== The Database and Vespa Init ==
     36
     37Every Vespa app calls
     38[browser:trunk/common/util/init.py init_app() in util_init.py] very early in
     39the app's invocation. This function does a number of things, including
     40upgrading or (re)creating Vespa's database if necessary.
     41
     42The file [browser:trunk/common/create_database.py create_database.py] handles
     43the work of building and populating a database from scratch. You might want
     44to read this; it's pretty straightforward and shows how the database is
     45constructed. You might also need to modify it if you're adding tables to the
     46database that need to be populated.
     47
     48Upgrading the database is handled by the
     49[browser:trunk/common/util/db_upgrader.py db_upgrader] module. This is explained
     50in detail below.
     51
     52
     53== Triggering an Update ==
     54
     55When you're ready to make your changes part of an official Vespa release,
     56you'll need to tell Vespa to upgrade the database, and you'll need to write
     57the code to do it.
     58
     59Upgrading a database occurs when `init_app()` notices that the value of
     60`DATABASE_VERSION` in [browser:trunk/common/constants.py constants.py] is
     61greater than the value of the field `database_version` in the table `vespa` in
     62the database. (This is an administrative table; it doesn't contain any user
     63data. It only contains one row and one column.) When those values are
     64different, the [browser:trunk/common/util/db_upgrader.py db_upgrader]
     65calls a function to upgrade that version.
     66
     67
     68== Upgrade Functions ==
     69
     70All of the code to upgrade a database is in the `DatabaseUpgrader` class.
     71That class knows how to upgrade based on the value of the database version
     72field.
     73
     74Upgrades are always stepwise. If `DATABASE_VERSION`
     75in constants.py is 7, and the value of `database_version` in the database is 5,
     76`DatabaseUpgrader` will call `upgrade_5()` to upgrade the database from
     77version 5 to 6, and then it will call `upgrade_6()` to upgrade it from 6 to 7.
     78
     79The method `upgrade_N()` is what you'll write, where `N` is the number of the
     80current database (i.e. the one you're updating ''FROM'', not to). Your code
     81can be simple like `_upgrade_3()` which merely added a table, or complex like
     82`_upgrade_2()` which broke the old `simulations` table into several parts and
     83replaced it with a different table of the same name.
     84
     85The code in `DatabaseUpgrader` contains comments to help you on your way,
     86and you can use the existing code as a template.
     87
     88== Summary ==
     89
     90I put the summary last so you won't be tempted to skip the details above.
     91They're important, please read them.
     92
     93 1. When working locally, there's no need to change version numbers anywhere.
     94 Just alter `create_views.sql` as you see fit, and delete your database to
     95 force Vespa to recreate it.
     96 2. Repeat step 1 as necessary.
     97
     98When you're ready to include your changes in a new Vespa version --
     99
     100 1. Increment `DATABASE_VERSION` in `constants.py`
     101 2. Write an `upgrade_N()` method for `DatabaseUpgrader` in `db_upgrader.py`.
     102
     103 
     104