I usually post about Persistence on Twitter - you can follow me there:

1. Introduction

This article describes key concepts of Flyway and how we can use this framework to continuously remodel our application’s database schema reliably and easily. At the end, we will present an example of managing a MySQL database using a Maven Flyway plugin.

Flyway updates a database from one version to a next using migrations. We can write migrations either in SQL with database specific syntax or in Java for advanced database transformations.

Migrations can either be versioned or repeatable. The former have a unique version and are applied exactly once. The latter do not have a version. Instead, they are (re-)applied every time their checksum changes.

Within a single migration run, repeatable migrations are always applied last, after pending versioned migrations have been executed. Repeatable migrations are applied in order of their description. For a single migration, all statements are run within a single database transaction.

In this article, we mainly focus on how we may use the Maven plugin to perform database migrations.

2. Flyway Maven Plugin

To install a Flyway Maven plugin, add following plugin definition to your pom.xml:

<plugin>
    <groupId>org.flywaydb</groupId>
    <artifactId>flyway-maven-plugin</artifactId>
    <version>4.0.3</version> 
</plugin>

You can check the latest version of the plugin available on Maven repository.

This Maven plugin may be configured in four different ways. Please refer to the documentation to get a list of all configurable properties.

2.1. Plugin Configuration

We may configure the plugin directly by use of the <configuration> tag in the plugin definition of the pom.xml:

<plugin>
    <groupId>org.flywaydb</groupId>
    <artifactId>flyway-maven-plugin</artifactId>
    <version>4.0.3</version>
    <configuration>
        <user>databaseUser</user>
        <password>databasePassword</password>
        <schemas>
            <schema>schemaName</schema>
        </schemas>
        ...
    </configuration>
</plugin>

2.2. Maven Properties

We may also configure the plugin by specifying configurable properties as Maven properties in pom.xml:

<project>
    ...
    <properties>
        <flyway.user>databaseUser</flyway.user>
        <flyway.password>databasePassword</flyway.password>
        <flyway.schemas>schemaName</flyway.schemas>
        ...
    </properties>
    ...
</project>

2.3. External Configuration File

We may also provide plugin configuration in a separate .properties file:

flyway.user=databaseUser
flyway.password=databasePassword
flyway.schemas=schemaName
...

Default configuration file name is flyway.properties and it should reside in the same directory as the pom.xml file. Encoding is specified by flyway.encoding (Default is UTF-8).

If you are using any other name (e.g customConfig.properties) as configuration file, then it should be specified explicitly when invoking the Maven command:

$ mvn -Dflyway.configFile=customConfig.properties

2.4. System Properties

Finally, all configuration properties may also be specified as system properties when invoking Maven on the command line:

$ mvn -Dflyway.user=databaseUser -Dflyway.password=databasePassword 
  -Dflyway.schemas=schemaName

Following is an order of precedence when a configuration is specified in more than one way:

  1. System properties
  2. External configuration file
  3. Maven properties
  4. Plugin configuration

3. Example Migration

In this section, we walk through the required steps to migrate a database schema for a MySQL database using the Maven plugin. We use an external file to configure Flyway.

This section assumes that you have already created a Maven project in a directory called $PROJECT_ROOT and MySQL database instance running on localhost:3306.

3.1. Update POM

Add an appropriate database driver dependency for the MySQL database in the pom.xml:

<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>6.0.3</version>
</dependency>

You can check the latest version of the driver available on Maven repository. Add Flyway plugin to pom.xml as explained in Section 2 above.

3.2. Configure Flyway Using External File

Create a file myFlywayConfig.properties in $PROJECT_ROOT with the following content:

flyway.user=databaseUser
flyway.password=databasePassword
flyway.schemas=app-db
flyway.url=jdbc:mysql://localhost:3306/
flyway.locations=filesystem:db/migration

The above configuration specifies that our migration scripts are located in the db/migration directory. It connects to MySQL instance available on localhost:3306 using databaseUser and databasePassword.

The application database schema is app-db. Please replace flyway.user, flyway.password, flyway.url with your database username, database password and database host/port appropriately.

3.3. Define First Migration

Flyway adheres to the following naming convention for migration scripts:

<Prefix><Version>__<Description>.sql

Where:

  • <Prefix> – Default prefix is V, which may be configured in the above configuration file using the flyway.sqlMigrationPrefix property.
  • <Version> – Migration version number. Major and minor versions may be separated by an underscore. Migration version should always start with 1.
  • <Description> – Textual description of the migration. The description needs to be separated from the version numbers with a double underscore.

Example: V1_1_0__my_first_migration.sql

Create a directory db/migration in $PROJECT_ROOT with a migration script named V1_0__create_employee_schema.sql containing SQL instructions to create e.g. an employee table:

CREATE TABLE IF NOT EXISTS `employee` (

    `id` int NOT NULL AUTO_INCREMENT PRIMARY KEY,
    `name` varchar(20),
    `email` varchar(50),
    `date_of_birth` timestamp

)ENGINE=InnoDB DEFAULT CHARSET=UTF8;

3.4. Execute Migrations

Invoke the following Maven command from $PROJECT_ROOT to execute database migrations:

$ mvn clean flyway:migrate -Dflyway.configFile=myFlywayConfig.properties

This should result in a first successful migration. The database schema may now be depicted as follows:

employee:
+----+------+-------+---------------+
| id | name | email | date_of_birth |
+----+------+-------+---------------+

Repeat the steps from subsections 3.3. and 3.4. to define and run new migrations at will.

3.5. Define And Execute Second Migration

Create a second migration file with name V2_0_create_department_schema.sql containing the following two queries:

CREATE TABLE IF NOT EXISTS `department` (

`id` int NOT NULL AUTO_INCREMENT PRIMARY KEY,
`name` varchar(20)

)ENGINE=InnoDB DEFAULT CHARSET=UTF8; 

ALTER TABLE `employee` ADD `dept_id` int AFTER `email`;

Execute a similar migration as was mentioned in section 3.4 above. The database schema looks like following after successfully executing the second migration.

employee:
+----+------+-------+---------+---------------+
| id | name | email | dept_id | date_of_birth |
+----+------+-------+---------+---------------+
department:
+----+------+
| id | name |
+----+------+

We may now verify that both migrations were indeed successful by invoking the following Maven command:

$ mvn flyway:info -Dflyway.configFile=myFlywayConfig.properties

4. How Flyway Works

To keep track of which migrations have already been applied, when and by whom, it adds a special bookkeeping table to your schema. This metadata table also tracks migration checksums and whether or not the migrations were successful.

The framework performs the following steps to accommodate evolving database schemas:

  1. It checks a database schema to locate its metadata table (SCHEMA_VERSION by default). If the metadata table does not exist, it will create one
  2. It scans an application classpath for available migrations
  3. It compares migrations against the metadata table. If a version number is lower or equal to a version marked as current, it is ignored
  4. It marks any remaining migrations as pending migrations. These are sorted based on version number and are executed in order
  5. As each migration is applied, the metadata table is updated accordingly

5. Commands

Flyway supports the following basic commands to manage database migrations.

  • Info: Prints current status/version of a database schema. It prints which migrations are pending,  which migrations have been applied, what is the status of applied migrations and when they were applied.
  • Migrate: Migrates a database schema to the current version. It scans the classpath for available migrations and applies pending migrations.
  • Baseline: Baselines an existing database, excluding all migrations, including baselineVersion. Baseline helps to start with Flyway in an existing database. Newer migrations can then be applied normally.
  • Validate: Validates current database schema against available migrations.
  • Repair: Repairs metadata table.
  • Clean: Drops all objects in a configured schema. All database objects are dropped. Of course, you should never use clean on any production database.

6. Conclusion

In this article, we’ve shown how Flyway works and how we can use this framework to remodel our application database reliably.

The code accompanying this article is available on Github.

I usually post about Persistence on Twitter - you can follow me there:


Sort by:   newest | oldest | most voted
Fosho Law
Guest

Interesting but I’m confused about how this is different than having a ddl file with “CREATE TABLE IF NOT EXISTS” statements. Could you elaborate on the differences and advantages of this? Thanks in advance. -Avid reader

Eugen Paraschiv
Guest
Hey Fosho, So – the simple option to create tables when they don’t exist is a start, but frameworks such as Flyway and Liquibase go way beyond that simple usecase. A quick example that ties to your scenario is – say you already have the table, but you need to add a new column to it. Or you need to remove one, or change the type of one. All of these kinds of changes are more about evolution, for a project that’s already deployed and in use – which is of course, most projects. So – outside of greenfield development,… Read more »
Fosho Law
Guest

Thanks for the reply Eugen. That clears up a bit but not completely. For adding/removing columns, shouldn’t there be SQL functions for checking if the column exist or not. These database migration projects sound really cool but I’m still having trouble seeing the difference between these tools and having different ddl files.

Eugen Paraschiv
Guest

Yes, of course – there is. And that’s what these frameworks use on the back-end.
It’s simply a way to manage these changes and provide a clean and consistent way of evolving (and rolling back) the structure of the schema.
If you don’t use one of these kinds of frameworks, your only other option is to manage these changes and these ddl files for deltas manually – which is a recipe for disaster (I know because I’ve tried).

Sanket Meghani
Guest
Fosho, it may not be obvious for simple scenarios as you mentioned. Now consider you have a set of SQL statements which are either dependent on each other and needs to be executed in atomic way or in particular order. If we use migration tool, such things are manages by tools and we only need to concentrate on actual migration logic. In addition, such tools allow writing migrations in Java and other languages, which allows us to take advantage of ORM and we can implement any complex business rule (pre-condition/post-condition/validation etc) if need be. Having said that, there is nothing… Read more »
Cristi Iaroi
Guest

Good to have more of these frameworks as alternatives. However I think mybatis migrations are better. http://www.mybatis.org/migrations-maven-plugin/

Basically it provides the same approach with a table to keep track of your changes but also is more customizable ( can be configured per environment and per maven profile) . Additionally it has a gradle plugin as well.

Eugen Paraschiv
Guest

That’s an interesting tool – but is it specific to MyBatis? If it is – then I’m not sure how helpful it is – because 99% of the world is not using MyBatis. If it’s generic, then besides the very unfortunate naming – it looks like an interesting alternative.

Cristi Iaroi
Guest

It isn’t mybatis specific. Just bad naming. Can be applied for any db and any db access technology. I used it with mybatis but also on projects with hibernate/spring data. In this second case it’s just sql for managing the migrations. I encourage you to try it out. It’s really easy to use.

Eugen Paraschiv
Guest

That sounds good. However, bad naming is a bit of an understatement – it’s like calling a system like ElasticSearch – “the SOLR plugin”. Nobody would even suspect it can do more.
I’d love to explore it in a separate article, sure, but unfortunately with that name, nobody’s using it anyways, so I think we’ll have to skip this one. Of course when I say nobody – I mean very very few people, not actually nobody.
Hope that makes sense. Cheers,
Eugen.

wpDiscuz