arangodb/Documentation/UserManual/UserManualReplication.md

Replication

@NAVIGATE_UserManualReplication @EMBEDTOC{UserManualReplicationTOC}

Introduction to Replication

Starting with ArangoDB 1.4, ArangoDB comes with optional asynchronous master-slave replication. Replication is configured on a per-database level, meaning that different databases in the same ArangoDB instance can have different replication settings. Replication must be turned on explicitly before it becomes active for a database.

In a typical master-slave replication setup, clients direct all their write operations for a specific database to the master. The master database is the only place to connect to when making any insertions/updates/deletions.

The master database will log all write operations in its so-called event log. The event log can be considered as an ordered stream of changes for the database.

Any number of slaves can then connect to the master database and fetch data from the master database's event log. The slaves then can apply all the events from the log in the same order locally. After that, they will have the same state of data as the master database.

In this setup, write operations are applied first in the master database, and applied in the slave database(s) afterwards.

For example, let's assume a write operation is applied and logged in the master database at point in time t0. To make a slave database apply the same operation, it must first fetch the write operation's data from master database's event log, then parse it and apply it locally. This will happen at some point in time after t0, let's say t1.

The difference between t1 and t0 is called the replication lag, and it is unavoidable in asychronous replication. The amount of replication lag depends on many factors, a few of which are:

• the network capacity between the slaves and the master
• the load of the master and the slaves
• the frequency in which slaves poll the master for updates

Between t0 and t1, the state of data on the master is newer than the state of data on the slave(s). At point in time t1, the state of data on the master and slave(s) is consistent again (provided no new data modifications happened on the master in between). Thus, the replication will lead to an eventually consistent state of data.

Transactions are honored in replication, i.e. transactional write operations will become visible on slaves atomically.

As all write operations will be logged to a master database's event log, the replication in ArangoDB 1.4 cannot be used for write-scaling. The main purposes of the replication in ArangoDB 1.4 are to provide read-scalability and "hot backups" for specific databases.

It is possible to connect multiple slave databaes to the same master database. Slave databases should be used as read-only instances, and no user-initiated write operations should be carried out on them. Otherwise data conflicts may occur that cannot be solved automatically, and that will make the replication stop. Master-master (or multi-master) replication is not supported in ArangoDB 1.4.

The replication in ArangoDB 1.4 is asychronous, meaning that slaves will pull changes from the master database. Slaves need to know to which master database they should connect to, but a master database is not aware of the slaves that replicate from it. When the network connection between the master database and a slave goes down, write operations on the master can continue normally. When the network is up again, slaves can reconnect to the master database and transfer the remaining changes. This will happen automatically provided slaves are configured appropriately.

The replication is turned off by default. In order to create a master-slave setup, the replication features need to be enabled on both the master database and the slave databases.

Replication is configured on a per-database level. If multiple database are to be replicated, the replication must be set up individually per database.

Components

The replication architecture in ArangoDB 1.4 consists of two main components, which can be used together or in isolation: the replication logger and the replication applier.

Both are available in ArangoDB 1.4 and can be administered via the command line or a REST API (see @ref HttpReplication).

In most cases, the replication logger will be run inside a master database, and the replication applier will be executed in a slave database.

As there can be multiple database inside an ArangoDB instance, there can be multiple replication loggers and multiple replication appliers per ArangoDB instance.

Replication Logger

###Purpose

The purpose of the replication logger is to log all changes that modify the state of data in a specific database. This includes document insertions, updates, and deletions. It also includes creating, dropping, renaming and changing collections and indexes.

When the replication logger is used, it will log all these write operations for the database in its event log, in the same order in which the operations were carried out originally.

Reading the event log sequentially provides a list of all write operations carried out in the database. Replication clients can request certain log events from the logger.

For example, ArangoDB's replication applier will permanently query the latest events from the replication logger. The applier of the slave database will then apply all changes locally to get to the same state of data as the master database. It will keep track of which parts of the event log it already synchronised, meaning that it will perform an incremental synchronisation.

Technically, the event log is a system collection named _replication inside the database the applier runs in. The event log is persisted and will still be present after a server shutdown or crash. The event log's underlying collection should not be modified by users directly. It should only be accessed using the special API methods offered by ArangoDB.

###Starting and Stopping

ArangoDB will only log changes if the replication logger is turned on for the specific database. Should there be any write operations in the database while the replication logger is turned off, these events will be lost for replication.

To turn on the replication logger for a database once, the following command can be executed:

require("org/arangodb/replication").logger.start();

Note that starting the replication logger does not necessarily mean it will be started automatically on all following server startups. This can be configured seperately (keep on reading for this).

Please note that the above command turns on the replication logger for the current database only, but not for all databases. To turn on the replication logger for multiple databases, the above command must be executed separately for each database.

To turn the replication logger off for the current database, execute the stop command:

require("org/arangodb/replication").logger.stop();

This has turned off the logger, but still the logger may be started again automatically the next time the ArangoDB server is started. This can be configured separately (again, keep on reading).

To determine the current state of the replication logger in the current database (including whether it is currently running or not), use the state command:

require("org/arangodb/replication").logger.state();

The result might look like this:

{ 
  "state" : { 
    "running" : false, 
    "lastLogTick" : "255376126918573", 
    "totalEvents" : 0, 
    "time" : "2013-08-02T11:01:28Z" 
  }, 
  "server" : { 
    "version" : "1.4.devel", 
    "serverId" : "53904504772335" 
  }, 
  "clients" : [ ] 
}

The running attribute indicates whether the logger is currently enabled and will log any events. The totalEvents attribute will indicate how many log events have been logged since the start of the ArangoDB server. The value will not be reset between multiple stops and restarts of the logger. Finally, the lastLogTick value will indicate the id of the last event that was logged. It can be used to determine whether new events were logged, and is also used by the replication applier for incremental fetching of data.

Note: the replication logger state can also be queried via the HTTP API (see @ref HttpReplication).

###Configuration

To determine whether the current database's replication logger is automatically started when the ArangoDB server is started, the logger has a separate configuration. The configuration is stored in a file REPLICATION-LOGGER-CONFIG inside the current database's directory. If it does not exist, ArangoDB will use default values.

To check and adjust the configuration of the replication logger for the current database, use the properties command. To view the current configuration, use properties without any arguments:

require("org/arangodb/replication").logger.properties();

The result might look like this:

{ 
  "autoStart" : false, 
  "logRemoteChanges" : false, 
  "maxEvents" : 1048576, 
  "maxEventsSize" : 134217728 
}

The autoStart attribute indicates whether the current database's replication logger is automatically started whenever the ArangoDB server is started. You may want to set it to true for a proper master setup. To do so, supply the updated attributes to the properties command, e.g.:

require("org/arangodb/replication").logger.properties({ 
  autoStart: true 
});

This will ensure the replication logger for the current database is automatically started on all following ArangoDB starts unless autoStart is set to false again.

Note that only those attributes will be changed that you supplied in the argument to properties. All other configuration values will remain unchanged. Also note that the replication logger can be reconfigured while it is running.

It a replication logger is turned on, the event log may be allowed to grow indefinitely. It may be sensible to set a maximum size or a maximum number of events to keep. If one of these thresholds is reached during logging, the replication logger will start removing the oldest events from the event log automatically.

The thresholds are set by adjusting the maxEvents and maxEventsSize attributes via the properties command. The maxEvents attribute is the maximum number of events to keep in the event log before starting to remove oldest events. The maxEventsSize is the maximum cumulated size of event log data (in bytes) that is kept in the log before starting to remove oldest events. If both are set to a value of 0, it means that the number and size of log events to keep is unrestricted. If both are set to a non-zero value, it means that there are restrictions on both the number and cumulated size of events to keep, and if either of the restrictions is hit, the deletion will be triggered.

The following command will set the threshold to 5000 events to keep, irrespective of the cumulated event data sizes:

require("org/arangodb/replication").logger.properties({ 
  maxEvents: 5000, 
  maxEventsSize: 0 
});

Replication Applier

###Purpose

The purpose of the replication applier is to read data from a master database's event log, and apply them locally. The applier will check the master database for new events periodically. It will perform an incremental synchronisation, i.e. only asking the master for events that occurred after the last synchronisation.

The replication applier does not get notified by the master database when there are "new" events, but uses the pull principle. It might thus take some time (the replication lag) before an event from the master database gets shipped to and applied in a slave database.

The replication applier of a database is run in a separate thread. It may encounter problems when a log event from the master cannot be applied safely, or when the connection to the master database goes down (network outage, master database is down or unavailable etc.). In this case, the database's replication applier thread might terminate itself. It is then up to the administrator to fix the problem and restart the database's replication applier.

If the replication applier cannot connect to the master database, or the communication fails at some point during the synchronisation, the replication applier will try to reconnect to the master database. It will give up reconnecting only after a configurable amount of connection attempts.

The replication applier state is queryable at any time by using the state command of the applier. This will return the state of the applier of the current database:

require("org/arangodb/replication").applier.state();

The result might look like this:

{ 
  "state" : { 
    "running" : false, 
    "lastAppliedContinuousTick" : "231848832948633", 
    "lastProcessedContinuousTick" : "231848832948633", 
    "lastAvailableContinuousTick" : null, 
    "progress" : { 
      "time" : "2013-08-02T11:40:08Z", 
      "message" : "applier created", 
      "failedConnects" : 0 
    }, 
    "totalRequests" : 0, 
    "totalFailedConnects" : 0, 
    "totalEvents" : 0, 
    "lastError" : { 
      "errorNum" : 0 
    }, 
    "time" : "2013-08-02T11:40:22Z" 
  }, 
  "server" : { 
    "version" : "1.4.devel", 
    "serverId" : "53904504772335" 
  }, 
  "endpoint" : "tcp://master.domain.org:8529" 
}

The running attribute indicates whether the replication applier of the current database is currently running and polling the server at endpoint for new events.

The failedConnects attribute shows how many failed connection attempts the replication applier currently has encountered in a row. In contrast, the totalFailedConnects attribute indicates how many failed connection attempts the applier has made in total. The totalRequest attribute shows how many requests the applier has sent to the master database in total. The totalEvents attribute shows how many log events the applier has read from the master.

The message sub-attribute of the progress sub-attribute gives a brief hint of what the applier currently does (if it is running). The lastError attribute also has an optional errorMessage sub-attribute, showing the latest error message. The errorNum sub-attribute of the lastError attribute can be used by clients to programmatically check for errors. It should be 0 if there is no error, and it should be non-zero when the applier terminated itself due to a problem.

Here is an example of the state after the replication applier terminated itself due to (repeated) connection problems:

{ 
  "state" : { 
    "running" : false, 
    "progress" : { 
      "time" : "2013-08-02T12:40:25Z", 
      "message" : "applier stopped", 
      "failedConnects" : 6 
    }, 
    "totalRequests" : 18, 
    "totalFailedConnects" : 11, 
    "totalEvents" : 0, 
    "lastError" : { 
      "time" : "2013-08-02T12:40:25Z", 
      "errorMessage" : "could not connect to master at tcp://master.example.org:8529: Could not connect to 'tcp:/...", 
      "errorNum" : 1400 
    },
    ...
  }
}

Note: the state of a database's replication applier is queryable via the HTTP API, too. Please refer to @ref HttpReplication for more details.

###Starting and Stopping

To start and stop the applier in the current database, the start and stop commands can be used:

require("org/arangodb/replication").applier.start(<tick>);
require("org/arangodb/replication").applier.stop();

Note that starting a replication applier without setting up an initial configuration will fail. The replication applier will look for its configuration in a file named REPLICATION-APPLIER-CONFIG in the current database's directory. If the file is not present, ArangoDB will use some default configuration, but it cannot guess the endpoint (the address of the master database) the applier should connect to. Thus starting the applier without configuration will fail.

Note that the first time you start the applier, you should pass 1 as value for tick. Otherwise you might see the message

"lastError" : { 
  "time" : "2014-04-16T16:59:16Z", 
  "errorMessage" : "no start tick", 
  "errorNum" : 1413 
}

Note that starting a database's replication applier via the start command will not necessarily start the applier on the next and following ArangoDB server restarts. Additionally, stopping a database's replication applier manually will not necessarily prevent the applier from being started again on the next server start. All of this is configurable seperately (hang on reading).

Please note that when starting the replication applier of database, it will resume where it stopped. This is sensible because replication log events should be applied incrementally. If the replication applier of a database has never been started before, it needs some tick value from the master database's event log from which to start fetching events.

####Configuration

To configure the replication applier of a specific database, use the properties command. Using it without any arguments will return the current configuration:

require("org/arangodb/replication").applier.properties();

The result might look like this:

{ 
  "requestTimeout" : 300, 
  "connectTimeout" : 10, 
  "ignoreErrors" : 0, 
  "maxConnectRetries" : 10, 
  "chunkSize" : 0, 
  "autoStart" : false, 
  "adaptivePolling" : true 
}

Note that there is no endpoint attribute configured yet. The endpoint attribute is required for the replication applier to be startable. You may also want to configure a username and password for the connection via the username and password attributes.

require("org/arangodb/replication").applier.properties({ 
  endpoint: "tcp://master.domain.org:8529", 
  username:  "root", 
  password: "secret"
});

This will re-configure the replication applier for the current database. The configuration will be used from the next start of the replication applier. The replication applier cannot be re-configured while it is running. It must be stopped first to be re-configured.

To make the replication applier of the current database start automatically when the ArangoDB server starts, use the autoStart attribute.

Setting the adaptivePolling attribute to true will make the replication applier poll the master database for changes with a variable frequency. The replication applier will then lower the frequency when the master is idle, and increase it when the master can provide new events). Otherwise the replication applier will poll the master database for changes with a constant frequency of 5 seconds if the master database's replication logger is turned off, and 0.5 seconds if it is turned on.

To set a timeout for connection and following request attempts, use the connectTimeout and requestTimeout values. The maxConnectRetries attribute configures after how many failed connection attempts in a row the replication applier will give up and turn itself off. You may want to set this to a high value so that temporary network outages do not lead to the replication applier stopping itself.

The chunkSize attribute can be used to control the approximate maximum size of a master's response (in bytes). Setting it to a low value may make the master respond faster (less data is assembled before the master sends the response), but may require more request-response roundtrips. Set it to 0 to use ArangoDB's built-in default value.

The following example will set most of the discussed properties for the current database's applier:

require("org/arangodb/replication").applier.properties({ 
  endpoint: "tcp://master.domain.org:8529", 
  username:  "root", 
  password: "secret",
  adaptivePolling: true,
  connectTimeout: 15,
  maxConnectRetries: 100,
  chunkSize: 262144,
  autoStart: true 
});

After the applier is now fully configured, it could theoretically be started. However, we may first need an initial sychronisation of all collections and their data from the master before we start the replication applier. The reason is that the replication logger of the master database may not contain all data we need for a full synchronisation.

The only safe method for doing a full synchronisation is thus to

• make sure the replication logger of the master database is running
• perform an intiial full sync with the master database
• note the master database's replication logger tick value and
• start the continuous replication applier using this tick value.

The initial synchronisation for the current database is executed with the sync command:

require("org/arangodb/replication").sync({
  endpoint: "tcp://master.domain.org:8529",
  username: "root",
  password: "secret
});

Warning: sync will do a full synchronisation of the collections in the current database with collections present in the master database. Any local instances of the collections and all their data are removed! Only execute this command when you are sure you want to remove the local data!

As sync does a full synchronisation, it may take a while to execute. When sync completes successfully, it show a list of collections it has synchronised in the collections attribute. It will also return the master database's replication logger tick value at the time the sync was started on the master. The tick value is contained in the lastLogTick attribute of the sync command:

{ 
  "lastLogTick" : "231848833079705", 
  "collections" : [ ... ]
}

If the master database's replication logger is turned on, you can now start the continuous synchronisation for the current database with the command

require("org/arangodb/replication").applier.start("231848833079705");

Note that the tick values should be handled as strings. Using numeric data types for tick values is unsafe because they might exceed the 32 bit value and the IEEE754 double accuracy ranges.

Example Setup

Setting up a working master-slave replication requires two ArangoDB instances:

• master: this is the instance we'll activate the replication logger on
• slave: on this instance, we'll start a replication applier, and this will fetch data from the master database's events log and apply all events locally

For the following example setup, we'll use the instance tcp://master.domain.org:8529 as the master, and the instance tcp://slave.domain.org:8530 as a slave.

The goal is to have all data from the database _system on master tcp://master.domain.org:8529 be replicated to the database _system on the slave tcp://slave.domain.org:8530.

On the master, configure the replication logger for database _system to start automatically on ArangoDB startup. Additionally, set some restrictions for the event log size:

db._useDatabase("_system");

require("org/arangodb/replication").logger.properties({
  autoStart: true,
  maxEvents: 1048576,
  maxEventsSize: 0
});

After that, start the replication logger for the _system database on the master:

db._useDatabase("_system");
require("org/arangodb/replication").logger.start();

On the slave, make sure there currently is no replication applier running in the _system database:

db._useDatabase("_system");
require("org/arangodb/replication").applier.stop();

After that, do an initial sync of the slave with data from the master. Execute the following commands on the slave:

db._useDatabase("_system");
require("org/arangodb/replication").sync({
  endpoint: "tcp://master.example.org:8529",
  username: "myuser",
  password: "mypasswd"
});

Warning: this will replace data in the slave database with data from the master database! Only execute the commands if you have verified you are on the correct server, in the correct database!

Then re-configure the slave database's replication applier to point to the master database, set the username and password for the connection, and set the autoStart attribute:

db._useDatabase("_system");
require("org/arangodb/replication").applier.properties({
  endpoint: "tcp://master.example.org:8529",
  username: "myuser",
  password: "mypasswd",
  autoStart: true,
  adaptivePolling: true
});

After that, start the applier in the slave database:

db._useDatabase("_system");
require("org/arangodb/replication").applier.start();

After that, you should be able to monitor the state and progress of the replication applier by executing the state command in the slave database:

db._useDatabase("_system");
require("org/arangodb/replication").applier.state();

You may also want to check the master and slave states via the HTTP APIs (see @ref HttpReplication) or the web interface ("Dashboard" tab).

Replication Limitations

The replication in ArangoDB 1.4-alpha has a few limitations still. Some of these limitations may be removed in later versions of ArangoDB:

• the event log of the master database is currently written directly after a write operation is carried out in the master database. In case the master crashes between having executed the write operation and having it written into the event log, the write operation may have been executed on the master, but may be lost for replication and not be applied on any slaves.
• there is no feedback from the slaves to the master. If a slave cannot apply an event it got from the master, the master will have a different state of data. In this case, the replication applier on the slave will stop and report an error. Administrators can then either "fix" the problem or re-sync the data from the master to the slave and start the applier again.
• the replication is an asynchronous master-slave replication. There is currently no way to use it as a synchronous replication, or a multi-master replication.
• at the moment it is assumed that only the replication applier executes write operations on a slave. ArangoDB currently does not prevent users from carrying out their own write operations on slaves, though this might lead to undefined behavior and the replication applier stopping.
• the replication logger will log write operations for all user-defined collections and only some system collections. Write operations for the following system collections are excluded intentionally: _trx, _replication, _users, _aal, _fishbowl, _modules and _routing. Write operations for the following system collections will be logged: _aqlfunctions, _graphs.
• Foxx applications consist of database entries and application scripts in the file system. The file system parts of Foxx applications are not tracked anywhere and thus not replicated in current versions of ArangoDB. To replicate a Foxx application, it is required to copy the application to the remote server and install it there using the foxx-manager utility.
• master servers do not know which slaves are or will be connected to them. All servers in a replication setup are currently only loosely coupled. There currently is no way for a client to query which servers are present in a replication.
• failover must currently be handled by clients or client APIs.
• there currently is one replication logger and one replication applier per ArangoDB database. It is thus not possible to have a slave apply the events logs from multiple masters.
• replication is set up on a per-database level. When using ArangoDB with multiple databases, replication must be configured individually for each database.
• the replication applier is single-threaded, but write operations on the master may be executed in parallel if they affect different collections. Thus the replication applier might not be able to catch up with a very powerful and loaded master.
• replication is only supported between ArangoDB 1.4 masters and 1.4 slaves. It is currently not possible to replicate from/to other ArangoDB versions.
• a replication applier cannot apply data from itself.
• when doing the initial synchronisation from a master to a slave using the sync method, collections are not prevented from being dropped in-between. That means that when a client fetches the data for collections C1, C2, and C3 (in this order), it will start with collection C1. Any other client may come along and drop collection C3 in the meantime. When the client reaches collection C3, it will not find this collection anymore, and the initial sync will fail. This will be fixed until the 1.4 stable release.

Replication Overhead

Running the replication logger will make all data modification operations more expensive, as the ArangoDB server will write the operations into the replication log, too.

Additionally, replication appliers that connect to an ArangoDB master will cause some extra work for the master as it needs to process incoming HTTP requests, and respond.

Overall, turning on the replication logger may reduce throughput on an ArangoDB master by some extent. If the replication feature is not required, the replication logger should be turned off.

Transactions are logged to the event log on the master as an uninterrupted sequence. While a transaction is written to the event log, the event log is blocked for other writes. Transactions should thus be as small as possible.

@BNAVIGATE_UserManualReplication