24 KiB

Raw Blame History

Upgrading to ArangoDB 1.4

@NAVIGATE_Upgrading14 @EMBEDTOC{Upgrading14TOC}

Upgrading

Please read the following sections if you upgrade from a pre-1.4 version of ArangoDB to ArangoDB 1.4.

ArangoDB 1.4 comes with a few changes, some of which are not 100% compatible to ArangoDB 1.3. The incompatibilies are mainly due to the introduction of the multiple databases feature and to some changes inside Foxx. One change also affects AQL user- defined functions, so if you use them in ArangoDB, please make sure to read @ref Upgrade14AqlFunctions "this" too.

Following is a list of incompatible changes with workarounds. Please read the list carefully and adjust any client programs or processes that work with ArangoDB appropriately. Please also make sure to create a full backup of your existing ArangoDB installation before performing an update.

The upgrading section is closed with an instruction summary and a list of potential problems and @ref Upgrading14Troubleshooting "troubleshooting options" for them. Please consult that section if you encounter any problems during or after the upgrade.

Database Directory Version Check and Upgrade

ArangoDB will perform a database version check at startup. This has not changed in ArangoDB 1.4. When ArangoDB 1.4 encounters a database created with earlier versions of ArangoDB, it will refuse to start. This is intentional. The output will then look like this:

2013-10-29T16:34:02Z [3927] INFO ArangoDB 1.4.0 -- ICU 49.1.2, V8 version 3.16.14.1, SSL engine OpenSSL 1.0.1e 11 Feb 2013
2013-10-29T16:34:02Z [3927] ERROR no databases found. Please start the server with the --upgrade option
2013-10-29T16:34:02Z [3927] ERROR unable to initialise databases: invalid database directory
2013-10-29T16:34:02Z [3927] FATAL cannot start server: invalid database directory

To make ArangoDB 1.4 start with a database directory created with an earlier ArangoDB version, you may need to invoke the upgrade procedure once. This can be done by running ArangoDB from the command line and supplying the --upgrade option:

unix> arangod data --upgrade

where data is the database directory. This will run a database version and any necessary migrations. As usual, you should create a backup of your database directory before performing the upgrade.

The output should be something like this:

....
2013-10-29T16:35:01Z [3959] INFO In database '_system': starting upgrade from version 1.3 to 1.4.0
2013-10-29T16:35:01Z [3959] INFO In database '_system': Found 18 defined task(s), 5 task(s) to run
2013-10-29T16:35:01Z [3959] INFO In database '_system': Executing task #1 (moveProductionApps): move Foxx apps into per-database directory
2013-10-29T16:35:01Z [3959] INFO In database '_system': Task successful
...
2013-10-29T16:35:01Z [3959] INFO In database '_system': Task successful
2013-10-29T16:35:01Z [3959] INFO In database '_system': upgrade successfully finished
2013-10-29T16:35:01Z [3959] INFO database upgrade passed

Please check the output the --upgrade run. It may produce errors, which need to be fixed before ArangoDB can be used properly. If no errors are present or they have been resolved, you can start ArangoDB 1.4 regularly.

Filesystem layout changes

The introduction of the "multiple databases" feature in ArangoDB 1.4 has caused some changes in how ArangoDB organises data in the filesystem. The changes may be relevant when upgrading from an earlier version of ArangoDB to 1.4.

If you upgrade from an existing ArangoDB 1.3 to 1.4, or from an older version of 1.4 to the beta or stable 1.4 version, ArangoDB will move some files and directories in the database and the applications directory around so the result matches the required new layout.

As usual when upgrading an existing ArangoDB instance, it may be necessary to force the upgrade by starting the server once with the --upgrade option.

This will automatically make the server run the upgrade tasks, which will move files around if required.

It is good practice to do a full backup of your data and applications directories before upgrading.

Please note that if you have your own scripts or procedures in place that directly work on ArangoDB files or directories, you might need to ajdust them so they use the new file and directory locations of ArangoDB 1.4.

Changes in the data directory

If you only access the database and collection files via ArangoDB commands and don't ever access the files in it separately, you may want to skip this section. You should read on if you work on/with the files in the database directory with tools other than ArangoDB (e.g. Unix commands, backup tools).

Before 1.4, there was only one database in an ArangoDB instance, and ArangoDB stored the data of all collections in directories directly underneath the main data directory.

This "old" filesystem layout looked like this:

<data directory>/
  <collection-abc>/
  <collection-xyz>/
  ...

With multiple databases in 1.4, the filesystem layout is now:

<data directory>/
  databases/
    <database-123>/
      <collection-abc>/
      <collection-xyz>/
    <database-456>/
      <collection-abc>/
      <collection-xyz>/

As you can see above, there is now an intermediate directory named databases inside the data directory, and each database resides in its own sub-directory underneath the databases directory.

Database directory names use a numeric id part, e.g. database-12345. To tell which database directory belongs to which database, you can check the file parameter.json inside a database directory (by the way: collections work the same, and you can determine the name of a collection by looking into the parameter.json file in a collection's directory).

Changes in the apps directory

Due to the multiple databases feature, there are also changes with respect to where Foxx applications need to be organised in the filesystem. If you have not yet used Foxx in ArangoDB already, you can safely skip this section.

Foxx applications in previous versions of ArangoDB resided in the application directory, with each Foxx application being contained in a directory directly underneath the main application directory like this:

apps/
  <app name>/
  <app name>/
  ...

With multiple databases in ArangoDB 1.4, Foxx applications can be installed per database. This requires Foxx applications to reside in database-specific directories.

apps/
  databases/
    <database name>/
      <app name>/
      <app name>/
      ...
    <database name>/
      <app name>/
      ...
    ...
  system/
    <app name>/
    ...

The directories of individual applications will thus be moved underneath the correct database sub-directory in the directory apps/databases. There is also the option of using so-called system applications. These are not database-specific but shared between all databases. ArangoDB is shipped with one system application named aardvark, which is ArangoDB's web-based admin interface.

Addressing specific Databases in Requests

In pre-1.4, an ArangoDB instance only managed a single database, so there was no need to specify a database name in any of the incoming HTTP requests. In ArangoDB 1.4, the server can manage multiple datbases, and it may be necessary in a request to specify which database the request should be executed in.

A specific ArangoDB database can be explicitly addressed by putting the database name into the URL. If the first part of the URL path is /_db/..., ArangoDB will interpret the ... as the database name, and strip off the database name from the URL path for further processing. This allows any existing actions that use URL paths without database names to remain fully functional.

For example, if the request URL is http://127.0.0.1:8529/_db/mydb/some-method, then ArangoDB will extract the database name mydb from the URL, and pass /some-method as the URL path to any internal or user-defined functions.

If no database name is specified in the URL path via the _db prefix, ArangoDB will use the algorithm described in @ref HttpDatabaseMapping to determine the database context for the request. If no extra endpoints are used, the algorithm will default to the _system database. A just upgraded ArangoDB instance will have all its collections and applications be mapped to the _system database too, meaning an upgraded ArangoDB instance should remain fully functional.

ArangoDB clients and drivers are not forced to supply database names as part of the ArangoDB URLs they call because of this compatibility functionaltiy. However, if clients need to access a specific database in a multiple database context, they will be required to supply the database name as part of the URLs. Most clients will use just one database most of the time, making it sufficient to set the database name when the connection to the server is established and then prefixing all requests with the database name initially set.

Changed Behavior

Changed Namespace Separator for AQL user-defined Functions

The namespace resolution operator for AQL user-defined functions has been changed from : to ::.

AQL user-defined functions were introduced in ArangoDB 1.3, and the namespace resolution perator for them has been the single colon (:) in 1.3. A call to a user-defined function in an AQL query looked like this:

RETURN mygroup:myfunc()

The single colon caused an ambiguity in the AQL grammar, making it indistinguishable from named attributes or the ternary operator in some cases, e.g.

{ mygroup:myfunc ? mygroup:myfunc }

To fix this ambiguity, the namespace resolution operator in 1.4 is changed from : to ::, so the above call will in 1.4 look like this:

RETURN mygroup::myfunc()

Names of existing user-defined AQL functions in the database will automatically be fixed when starting ArangoDB 1.4 with the --upgrade option.

Still any AQL query strings assembled on the client side must be adjusted for use with 1.4 if they refer to AQL user-defined functions. If AQL queries stored in Foxx applications or other server-side actions use the "old" function name sytanx, they must be adjusted manually, too. These change should be simple to carry out (replacing the : in names of user-defined functions with ::) but cannot be done automatically by ArangoDB.

If function names are not changed in AQL queries, referring to a function using the old (:) namespace operator is likely to cause a query parse error in 1.4.

The return value of the AQL DOCUMENT function is also changed in 1.4 when called with a single argument (a document id or key) in case the sought document cannot be found. In pre-1.4, the function returned undefined in this case. As undefined is not part of the JSON type system, 1.4 now returns null for the same case. The return value for other cases has not changed.

Changed Return Value of REST API method GET `/_api/collection/figures`

The value returned by the REST API method GET /_api/collection/<collection-name>/figures has been extended in version 1.4 to also include the following attributes:

compactors.count
compactors.fileSize
shapefiles.count
shapefiles.fileSize

These attributes were not present in the return value in ArangoDB 1.3. Clients which rely on the return value structure may need to be adjusted.

Removed Features

Removed or Renamed Configuration options

The following changes have been made in ArangoDB 1.4 with respect to configuration / command-line options:

The options --server.admin-directory and --server.disable-admin-interface have been removed.

In previous versions of ArangoDB, these options controlled where the static files of the web admin interface were located and if the web admin interface was accessible via HTTP. In ArangoDB 1.4, the web admin interface has become a Foxx application and does not require an extra location.
The option --log.filter was renamed to --log.source-filter.

This is a debugging option that should rarely be used by non-developers.

Other removed Features

The action deployment tool available in ArangoDB 1.3 has been removed in version 1.4. Installing actions can now be achieved easier by packaging them in a Foxx application and deploying them with the foxx-manager binary.

Upgrade Checklist

Here is a checklist for the required upgrade steps. For more details please consult the more detailed topics and the @ref Upgrading14Troubleshooting sections.

Upgrade ArangoDB
- Stop arangod
- Create a full backup of your database directory
- Upgrade ArangoDB to 1.4 using package manager, homebrew, git pull etc.
- Adjust configuration file arangod.conf:
  - Add option --javascript.app-path if not yet present (otherwise arangod will refuse to start)
  - Check if you want to set --server.default-api-compatibility to 10300 for "old" client drivers
  - Remove now superfluous configuration options --server.admin-directory and --server.disable-admin-interface
- If required, start arangod with the --upgrade option once and check the output/logfile for any errors
Upgrade clients/drivers
- Upgrade client drivers to a 1.4-compatible version if needed and if available
- When using AQL user-defined functions, adjust AQL function namespace separator in queries/client code
Upgrade Foxx applications
- When using Foxx application from ArangoDB 1.3, make sure to adjust the manifest and the app/controller JavaScript files
Restart arangod
Check the output of arangod and the server logfile for potential errors.

Troubleshooting

If you cannot find a solution here, please ask the Google-Group at http://groups.google.com/group/arangodb

Please make sure that you include the currently used ArangoDB version number plus the version from which you upgrade in your post. You can retrieve the current version number as follows:

unix> arangod --version

Problem: Server does not start

If the server complains about no databases being found at startup like this

2013-10-29T16:34:02Z [3927] ERROR no databases found. Please start the server with the --upgrade option
2013-10-29T16:34:02Z [3927] ERROR unable to initialise databases: invalid database directory
2013-10-29T16:34:02Z [3927] FATAL cannot start server: invalid database directory

the problem might be that an ArangoDB 1.4 server was started with database directory created with an older version of ArangoDB. In this case it is necessary to start ArangoDB with the --upgrade option once. If it still does not start, it may be necessary to specifiy the --javascript.app-path option (see next item) when running --upgrade. In case of errors, please also check ArangoDB's output or logfile for further details about the problem.

Problem: Server does not start

Contrary to previous versions, ArangoDB 1.4 requires the startup option --javascript.app-path to be set when the server is started. If the option is not set, the server will print a message like this:

2013-10-09T19:11:47Z [7121] FATAL no value has been specified for --javascript.app-path.

To fix this, you can either specify the value for --javascript.app-path on the command-line or in a configuration file. You then need to start the server with the app path specified and the --upgrade option. This will make ArangoDB create the directory automatically.

If you want to use the development mode for Foxx applications, it is necessary to also specify the parameter --javascript.dev-app-path when starting with --upgrade. This will initialise the directories for development mode apps, too.

Problem: Server does not start

The server will try to bind to all defined endpoints at startup. The list of endpoints to connect to is read from the command-line, ArangoDB's configuration file, and a file named ENDPOINTS in the database directory. The ENDPOINTS file contains all endpoints which were added at runtime.

In case a previously defined endpoint is invalid at server start, ArangoDB will fail on startup with a message like this:

2013-09-24T07:09:00Z [4318] INFO using endpoint 'tcp://127.0.0.1:8529' for non-encrypted requests
2013-09-24T07:09:00Z [4318] INFO using endpoint 'tcp://127.0.0.1:8531' for non-encrypted requests
2013-09-24T07:09:00Z [4318] INFO using endpoint 'tcp://localhost:8529' for non-encrypted requests
...
2013-09-24T07:09:01Z [4318] ERROR bind() failed with 98 (Address already in use)
2013-09-24T07:09:01Z [4318] FATAL failed to bind to endpoint 'tcp://localhost:8529'. Please review your endpoints configuration.

The problem above may be obvious: in a typical setup, 127.0.0.1 will be the same IP address, so ArangoDB will try to bind to the same address twice (which will fail). Other obvious bind problems at startup may be caused by ports being used by other programs, or IP addresses changing.

Problem: Where can Example Configuration files for ArangoDB be found?

Updating ArangoDB will likely not overwrite an existing ArangoDB configuration file. If you have a configuration file from a previous ArangoDB version in use already, you may need to adjust this file locally.

The most current configuration files for ArangoDB 1.4 can be found on Github (look for files with file extension *.conf):

for Linux & MacOS: @S_EXTREF{https://github.com/triAGENS/ArangoDB/tree/1.4/etc/relative,https://github.com/triAGENS/ArangoDB/tree/1.4/etc/relative}
for Windows: @S_EXTREF{https://github.com/triAGENS/ArangoDB/tree/1.4/VS2012/Installer,https://github.com/triAGENS/ArangoDB/tree/1.4/VS2012/Installer}

Please note that all these configuration files use relative paths that may need to be adjusted to the absolute paths on your system.

Problem: Server returns different `location` headers than in 1.3

ArangoDB 1.4 by default will return location HTTP headers that contain the database name too. This is a consequence of potentially having multiple databases in the same server instance.

For example, when creating a new document, ArangoDB 1.3 returned an HTTP response with a location header like this:

location: /_api/document/<collection name>/<document key>

Contrary, ArangoDB 1.4 will return a location header like this by default:

location: /_db/<database name>/_api/document/<collection name>/<document key>

This may not be compatible to pre-1.4 clients that rely on the old format of the location header.

Obviously one workaround is to upgrade the used client driver to the newest version. If that cannot be done or if the newest version of the client driver is not ready for ArangoDB 1.4, the server provides a startup option that can be used to increase compatibility with old clients:

--server.default-api-compatibility

Not setting this option will make ArangoDB set it to the current server version, and assume all clients are compatible. This will also make it send the new-style location headers.

Setting this value to an older version number will make the server try to keep the API compatible to older versions where possible. For example, to send the old (pre-1.4) style location headers, set the value to 10300 (1.3) as follows:

--server.default-api-compatibility 10300

The server will then return the old-style location headers.

Another way to fix the location header issue is to make the client send API compatibility information itself. This can be achieved by sending an extra HTTP header x-arango-version along with a client request. For example, sending the following header in a request will make ArangoDB return the old style location headers too:

x-arango-version: 10300

Problem: Find out the storage location of a database / collection

To tell which directory belongs to which named database and which collection, you can use a Bash script like this:

cd /path/to/data/directroy
for db in `find databases -mindepth 1 -maxdepth 1 -type d`;
  do
    echo -n "- database directory \"$db\", name ";
    grep -o -E "name\":\"[^\"]+\"" "$db"/parameter.json | cut -d ":" -f 2;

    for c in `find $db/ -mindepth 1 -maxdepth 1 -type d`;
      do
        echo -n "  - collection directory \"$c\", name ";
        grep -E -o "name\":\"[^\"]+\"" "$c/parameter.json" | cut -d ":" -f 2;
      done;

    echo;
  done

The above script should print out the names of all databases and collections with their corresponding directory names.

Problem: AQL user-functions from 1.3 do not work anymore

The namespace resolution operator for AQL user-defined functions has changed from : to ::. Names of user-defined function names need to be adjusted in AQL queries. Please refer to @ref Upgrading14ChangedBehavior for details.

Problem: Foxx Applications from 1.3 are missing in 1.4

The directory layout for Foxx applications got changed between ArangoDB 1.3 and 1.4. In ArangoDB 1.4, Foxx applications need to placed in per-database directories which were not present in 1.3. ArangoDB 1.4 will automatically move existing Foxx applications into the required new structure if you start it with the --upgrade option and the Foxx application directory --javascript.app-path. Please note that specifiying --javascript.app-path is required in 1.4 anyway, so it is good to add this option to your configuration file if not already done.

Please read more about the 1.4 Foxx application directory layout @ref Upgrading14FileSystem "here".

Problem: Foxx Applications from 1.3 do not work in 1.4

@ref UserManualFoxx "Foxx" was released in an alpha state with ArangoDB 1.3. It got more stable over time, but some its internal APIs changed during the development of ArangoDB 1.4.

Foxx applications written for ArangoDB 1.3 need some modifications in order to work properly with ArangoDB 1.4:

in the manifest file of a Foxx application, please rename the apps attribute to controllers.
the require directive for the Foxx framework components changed between 1.3 and 1.4. Whereas in 1.3 you could require Foxx.Application, in 1.4 you will need to require Foxx.Controller:

Please look out for all places like this in your 1.3 Foxx controller/application files:
```
var FoxxApplication = require("org/arangodb/foxx").Application;
var app = new FoxxApplication();
```
and adjust them as follows for 1.4:
```
var FoxxController = require("org/arangodb/foxx").Controller;
controller = new FoxxController(applicationContext);
```
You may also need to adjust the setup and teardown scripts if used. To check for any errors after adjusting your Foxx application, you can start ArangoDB 1.4 in development mode using the --javascript.dev-app-path option. This will either print or log (depending on configuration) errors occurring in Foxx applications.

@BNAVIGATE_Upgrading14

24 KiB Raw Blame History

Upgrading to ArangoDB 1.4

Upgrading

Database Directory Version Check and Upgrade

Filesystem layout changes

Changes in the data directory

Changes in the apps directory

Addressing specific Databases in Requests

Changed Behavior

Changed Namespace Separator for AQL user-defined Functions

Changed Return Value of REST API method GET /_api/collection/figures

Removed Features

Removed or Renamed Configuration options

Other removed Features

Upgrade Checklist

Troubleshooting

Problem: Server does not start

Problem: Server does not start

Problem: Server does not start

Problem: Where can Example Configuration files for ArangoDB be found?

Problem: Server returns different location headers than in 1.3

Problem: Find out the storage location of a database / collection

Problem: AQL user-functions from 1.3 do not work anymore

Problem: Foxx Applications from 1.3 are missing in 1.4

Problem: Foxx Applications from 1.3 do not work in 1.4

24 KiB

Raw Blame History

Changed Return Value of REST API method GET `/_api/collection/figures`

Problem: Server returns different `location` headers than in 1.3