-
Notifications
You must be signed in to change notification settings - Fork 55
Database Migration
This is a strategy for migrating from one database to another using the Migration Data Adapter to allow atom entries to be written to two separate databases.
Issue beyond the scope of this plan:
- How to migrate all the existing entries from the OLD database to the NEW database
- How to synchronize entries that failed to write in the NEW database
This document will demonstrate the migration from a database used by the Hiberante Data Adapter to a database used by the Posgres Data Adapter, but other than setting up the adapters, these instructions can be used with all the existing Data Adapters.
In order to migrate from one database/schema to another, two separate Data Adapters will need to be setup in the application-context.xml. The first Data Adapter (for the existing database) will have already been setup, but we'll need to add some properties to the publisher to ensure data consistency between the entries in the two databases.
<bean name="feed-repository-bean" class="org.atomhopper.hibernate.HibernateFeedRepository">
<constructor-arg>
<map>
<entry key="hibernate.connection.driver_class" value="org.postgresql.Driver" />
<entry key="hibernate.dialect" value="org.hibernate.dialect.PostgreSQLDialect" />
<entry key="hibernate.connection.url" value="jdbc:postgresql://localhost:5432/atomhopperold" />
<entry key="hibernate.connection.username" value="username" />
<entry key="hibernate.connection.password" value="password" />
</map>
</constructor-arg>
</bean>
<bean name="hibernate-feed-publisher"
class="org.atomhopper.hibernate.adapter.HibernateFeedPublisher">
<property name="feedRepository" ref="feed-repository-bean" />
<property name="allowOverrideId">
<value>true</value>
</property>
<property name="allowOverrideDate">
<value>true</value>
</property>
</bean>
<bean name="hibernate-feed-source" class="org.atomhopper.hibernate.adapter.HibernateFeedSource">
<property name="feedRepository" ref="feed-repository-bean" />
</bean>Note: In order for the entries in both databases to have the same entry Id and created and updated dates, the allowOverrideId and allowOverrideDate properties for the hibernate-feed-pushlisher MUST be set to true.
Now the second Data Adapter will need to be configured in the application-context.xml file.
<bean id="atomHopperDataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="org.postgresql.Driver" />
<property name="url" value="jdbc:postgresql://localhost:5432/atomhoppernew" />
<property name="username" value="username" />
<property name="password" value="password" />
<property name="minIdle" value="10" />
<property name="maxIdle" value="25" />
<property name="initialSize" value="10" />
<property name="maxActive" value="50" />
</bean>
<bean name="atomHopperJdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<constructor-arg name="dataSource" ref="atomHopperDataSource"/>
</bean>
<bean name="postgres-feed-publisher" class="org.atomhopper.postgres.adapter.PostgresFeedPublisher">
<property name="jdbcTemplate" ref="atomHopperJdbcTemplate" />
<property name="allowOverrideId">
<value>true</value>
</property>
<property name="allowOverrideDate">
<value>true</value>
</property>
</bean>
<bean name="postgres-feed-source" class="org.atomhopper.postgres.adapter.PostgresFeedSource">
<property name="jdbcTemplate" ref="atomHopperJdbcTemplate" />
</bean>Note: Again, in order for the entries in both databases to have the same entry Id and created and updated dates, the allowOverrideId and allowOverrideDate properties for the postgres-feed-pushlisher MUST be set to true.
Now the Migration Data Adapter needs to be configured in the application-context.xml file. The Migration Data Adapter requires two other Data Adapters to be configured, one for the old database and one for the new database, like shown above. Then the Migration Data Adapter can be wired like this:
<bean name="migration-feed-publisher" class="org.atomhopper.migration.adapter.MigrationFeedPublisher">
<property name="oldFeedPublisher" ref="hibernate-feed-publisher" />
<property name="newFeedPublisher" ref="postgres-feed-publisher" />
<property name="writeTo">
<value>OLD</value>
</property>
<property name="readFrom">
<value>OLD</value>
</property>
</bean>
<bean name="migration-feed-source" class="org.atomhopper.migration.adapter.MigrationFeedSource">
<property name="oldFeedSource" ref="hibernate-feed-source" />
<property name="newFeedSource" ref="postgres-feed-source" />
<property name="readFrom">
<value>OLD</value>
</property>
</bean>Note: The initial configuration for the Migration Data Adapter sets both the writeTo and readFrom values to the OLD (or existing) database.
Now the final step prior to deployment is to set up the atom-server.cfg.xml to use the Migration Feed Adapter for the existing feed instead of the current Hibernate Adapter.
<?xml version="1.0" encoding="UTF-8"?>
<atom-hopper-config xmlns="http://atomhopper.org/atom/hopper-config/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://atomhopper.org/atom/hopper-config/v1.0 ./../../config/atom-hopper-config.xsd">
<defaults>
<author name="Atom Hopper" />
</defaults>
<host domain="localhost:8080" />
<workspace title="Testing Namespace" resource="/namespace/">
<categories-descriptor reference="workspace-categories-descriptor" />
<feed title="Testing Feed" resource="/feed">
<publisher reference="migration-feed-publisher" />
<feed-source reference="migration-feed-source" />
</feed>
</workspace>
</atom-hopper-config>Once both the application-context.xml and the atom-server.cfg.xml have been updated, the server can be restarted to pick up the changes. With the original settings, all entries will continue to be written to and read from only the OLD database. This should be allowed to run for a while to ensure no errors are occurring.
Now it's time to start writing to both databases, so the writeTo setting in the Migration Data Adapter will need to be changed from OLD to BOTH in the application-context.xml file.
<bean name="migration-feed-publisher" class="org.atomhopper.migration.adapter.MigrationFeedPublisher">
<property name="oldFeedPublisher" ref="hibernate-feed-publisher" />
<property name="newFeedPublisher" ref="postgres-feed-publisher" />
<property name="writeTo">
<value>BOTH</value>
</property>
<property name="readFrom">
<value>OLD</value>
</property>
</bean>Again, restart the server to pickup the change. At this point all entries are still being read from the OLD database, but new entries are being written to both the OLD and NEW databases. Here is how the Migration Data Adapter handles errors while writing to both databases with the above settings:
If an error occurs writing to the OLD database:
- The error is propagated up to the caller and no entry is written to the NEW database
If writing to the OLD database succeeds but an error occurs writing to the NEW database:
- The entry is written to the OLD database
- An error is logged in the atomhopper-logback.log file which records the feed name and entry Id of the entry that had an error
- A 201 Created message containing the entry is returned to the caller
After running with this configuration for a while and ensuring no errors have been logged writing to the NEW database, it's time to switch the Migration Data Adapter to read from the NEW database. In order to do that, the application-context.xml file will need to be updated like this:
<bean name="migration-feed-publisher" class="org.atomhopper.migration.adapter.MigrationFeedPublisher">
<property name="oldFeedPublisher" ref="hibernate-feed-publisher" />
<property name="newFeedPublisher" ref="postgres-feed-publisher" />
<property name="writeTo">
<value>BOTH</value>
</property>
<property name="readFrom">
<value>NEW</value>
</property>
</bean>Again, restart the server to pickup the change. At this point all entries are still being written to both the OLD and NEW databases, but all entries are being read from the NEW databases. Here is how the Migration Data Adapter handles errors while writing to both databases with the above settings:
If an error occurs writing to the NEW database:
- The error is propagated up to the caller and no entry is written to the NEW database
If writing to the NEW database succeeds but an error occurs writing to the OLD database:
- The entry is written to the NEW database
- An error is logged in the atomhopper-logback.log file which records the feed name and entry Id of the entry that had an error
- A 201 Created message containing the entry is returned to the caller
After running with this setting for a while and ensuring no errors occurred, the migration is completed and the Migration Data Adapter can be removed and only the new Postgres Data Adapter configured. Here are the updated application-context.xml and the atom-server.cfg.xml settings:
Remove all references to the old Hibernate Data Adapter from the configuration, specifically remove this from the application-context.xml file:
<bean name="feed-repository-bean" class="org.atomhopper.hibernate.HibernateFeedRepository">
<constructor-arg>
<map>
<entry key="hibernate.connection.driver_class" value="org.postgresql.Driver" />
<entry key="hibernate.dialect" value="org.hibernate.dialect.PostgreSQLDialect" />
<entry key="hibernate.connection.url" value="jdbc:postgresql://localhost:5432/atomhopperold" />
<entry key="hibernate.connection.username" value="username" />
<entry key="hibernate.connection.password" value="password" />
</map>
</constructor-arg>
</bean>
<bean name="hibernate-feed-publisher"
class="org.atomhopper.hibernate.adapter.HibernateFeedPublisher">
<property name="feedRepository" ref="feed-repository-bean" />
<property name="allowOverrideId">
<value>true</value>
</property>
<property name="allowOverrideDate">
<value>true</value>
</property>
</bean>
<bean name="hibernate-feed-source" class="org.atomhopper.hibernate.adapter.HibernateFeedSource">
<property name="feedRepository" ref="feed-repository-bean" />
</bean>Remove all references to the Migration Data Adapter from the configuration, specifically remove this from the application-context.xml file:
<bean name="migration-feed-publisher" class="org.atomhopper.migration.adapter.MigrationFeedPublisher">
<property name="oldFeedPublisher" ref="hibernate-feed-publisher" />
<property name="newFeedPublisher" ref="postgres-feed-publisher" />
<property name="writeTo">
<value>OLD</value>
</property>
<property name="readFrom">
<value>OLD</value>
</property>
</bean>
<bean name="migration-feed-source" class="org.atomhopper.migration.adapter.MigrationFeedSource">
<property name="oldFeedSource" ref="hibernate-feed-source" />
<property name="newFeedSource" ref="postgres-feed-source" />
<property name="readFrom">
<value>OLD</value>
</property>
</bean>Remove the allowOverrideId property and the allowOverrideData property from the postgres feed publisher, specifically remove these from the postgres-feed-publisher in the application-context.xml file:
<property name="allowOverrideId">
<value>true</value>
</property>
<property name="allowOverrideDate">
<value>true</value>
</property>Finally, update the atom-server.cfg.xml to use the Postgres Data Adapter instead of the Migration Data Adapter.
<?xml version="1.0" encoding="UTF-8"?>
<atom-hopper-config xmlns="http://atomhopper.org/atom/hopper-config/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://atomhopper.org/atom/hopper-config/v1.0 ./../../config/atom-hopper-config.xsd">
<defaults>
<author name="Atom Hopper" />
</defaults>
<host domain="localhost:8080" />
<workspace title="Testing Namespace" resource="/namespace/">
<categories-descriptor reference="workspace-categories-descriptor" />
<feed title="Testing Feed" resource="/feed">
<publisher reference="postgres-feed-publisher" />
<feed-source reference="postgres-feed-source" />
</feed>
</workspace>
</atom-hopper-config>Restart the server, and migration is complete and the OLD database can be deleted.