Hawkin
/
arangodb
mirror of https://gitee.com/bigwinds/arangodb
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Updated scripts and queues docs.

This commit is contained in:
Alan Plum 2015-06-02 16:51:31 +02:00
parent bf6501c02b
commit ec1e9fd336
3 changed files with 160 additions and 136 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

141
Documentation/Books/Users/Foxx/Develop/Queues.mdpp
View File

@ -6,31 +6,17 @@ Foxx allows defining job queues that let you perform slow or expensive actions a
Please note that Foxx job queues are database-specific. Queues and jobs are always relative to the database in which they are created or accessed.
By default only jobs from queues inside the `_system` database will be processed automatically by the Foxx queue manager. To automatically process the queues of other databases than `_system`, it is required to start ArangoDB with the option `--server.foxx-queues-system-only false`.
By default only jobs from queues in the `_system` database will be processed automatically by the Foxx queue manager. To automatically process the queues of other databases than `_system`, you have to start ArangoDB with the option `--server.foxx-queues-system-only false`.
Restricting the Foxx queue manager to the `_system` database will lead to the queue manager having to check only the queue(s) of a single database. Contrary, checking the queues of all databases might result in more work and more CPU time used by the queue manager.
Restricting the Foxx queue manager to the `_system` database will lead to the queue manager only having to check the queues of a single database. Checking the queues of *all* databases might result in more work and more CPU time used by the queue manager.
By default the Foxx queue manager will be checking the queue(s) every second. This value can be changed by setting the startup option `--server.foxx-manager-poll-interval`. Lowering this value will result in the queue manager waking up and checking the queues more frequently, which may increase CPU usage of the server.
By default the Foxx queue manager will be checking the queues once every second. This value can be changed by setting the startup option `--server.foxx-manager-poll-interval`. Lowering this value will result in the queue manager waking up and checking the queues more frequently, which may increase CPU usage of the server.
For the low-level functionality see the section **Task Management** in the chapter **JavaScript Modules**.
For the low-level functionality see [the chapter on the **Task Management** module](../../ModuleTasks/).
**Examples**
As of ArangoDB 2.6, Foxx queue job types are defined as regular Foxx scripts. To learn more about Foxx scripts, see [the chapter on Foxx scripts](./Scripts.html).
The following Foxx route handler will enqueue a job whenever the **"/log"** route is accessed that prints "Hello World!" to the server log.
```js
var Foxx = require("org/arangodb/foxx");
var ctrl = new Foxx.Controller(applicationContext);
var queue = Foxx.queues.create("my-queue");
Foxx.queues.registerJobType("log", function (data) {
print(data);
});
ctrl.get("/log", function () {
queue.push("log", "Hello World!");
});
```
For an example of re-usable job types see the various mailer apps available in the Foxx app store.
!SECTION Creating or updating a queue
@ -105,7 +91,6 @@ When a queue is deleted, jobs on that queue will no longer be executed.
Deleting a queue will not delete any jobs on that queue.
**Parameters**
* **name**: the name of the queue to delete
@ -118,80 +103,61 @@ Foxx.queues.delete("my-queue"); // true
Foxx.queues.delete("my-queue"); // false
```
!SECTION Registering a job type
Registers a job type with the queue manager.
The job type will be registered for all queues in the context of the current database. Queues of other databases are not affected by registering a job type.
`Foxx.queues.registerJobType(name, opts)`
If **opts** is a function it will be treated as the **execute** function.
**Parameters**
* **name**: the name of the job type to register
* **opts**: an object with the following properties:
* **execute**: a function to pass the job data to when a job is executed
* **maxFailures** (optional): the number of times a job will be re-tried before it is marked as **"failed"**. A negative value or **Infinity** means that the job will be re-tried on failure indefinitely. Default: **0**
* **schema** (optional): a [Joi](https://github.com/hapijs/joi) schema to validate a job's data against before accepting it
* **preprocess** (optional): a function to pre-process a job's (validated) data before serializing it in the queue
* **backOff** (optional): either a function that takes the number of times the job has failed before as input and returns the number of milliseconds to wait before trying the job again, or the delay to be used to calculate an [exponential back-off](https://en.wikipedia.org/wiki/Exponential_backoff), or **0** for no delay. Default: **1000**
**Examples**
```js
var Foxx = require("org/arangodb/foxx");
Foxx.queues.registerJobType("log", function (data) {
print(data);
});
```
!SECTION Adding a job to a queue
Adds a job of the given type to the given queue.
Adds a job to the given queue.
The job will be added to the specified queue in the context of the current database.
`Queue::push(name, data, [opts])`
`Queue::push(script, data, [opts])`
Returns the job id.
**Parameters**
* **name**: the name of the job's job type
* **data**: the job data of the job; must be serializable to JSON
* **script**: a job type definition, consisting of an object with the following properties:
* **name**: the name of the script that will be invoked.
* **mount**: the mount path of the app that defines the script.
* **backOff** (optional): either a function that takes the number of times the job has failed before as input and returns the number of milliseconds to wait before trying the job again, or the delay to be used to calculate an [exponential back-off](https://en.wikipedia.org/wiki/Exponential_backoff), or **0** for no delay. Default: **1000**.
* **maxFailures** (optional): the number of times a job will be re-tried before it is marked as **"failed"**. A negative value or **Infinity** means that the job will be re-tried on failure indefinitely. Default: **0**.
* **schema** (optional): a [Joi](https://github.com/hapijs/joi) schema to validate a job's data against before enqueuing the job.
* **preprocess** (optional): a function to pre-process a job's (validated) data before serializing it in the queue.
* **data**: the job data of the job; must be serializable to JSON.
* **opts** (optional): an object with any of the following properties:
* **success** (optional): a function to be called after the job has been completed successfully
* **failure** (optional): a function to be called after the job has failed too many times
* **delayUntil** (optional): a timestamp in milliseconds until which the execution of the job should be delayed. Default: **Date.now()**
* **backOff** (optional): either a function that takes the number of times the job has failed before as input and returns the number of milliseconds to wait before trying the job again, or the delay to be used to calculate an [exponential back-off](https://en.wikipedia.org/wiki/Exponential_backoff), or **0** for no delay. See [Registering a job type](#registering-a-job-type)
* **allowUnknown** (optional): whether the job should be queued even if the job type name does not match a currently registered job type. Default: **false**
* **maxFailures** (optional): the number of times the job will be re-tried before it is marked as *"*failed*"*. A negative value or **Infinity** means that the job will be re-tried on failure indefinitely. See [Registering a job type](#registering-a-job-type)
Note that if you pass a function for the **backOff** calculation, **success** callback or **failure** callback options the function must not rely on any external scope or external variables.
* **success** (optional): a function to be called after the job has been completed successfully.
* **failure** (optional): a function to be called after the job has failed too many times.
* **delayUntil** (optional): a timestamp in milliseconds until which the execution of the job should be delayed. Default: **Date.now()**.
* **backOff** (optional): see **script.backOff**.
* **maxFailures** (optional): see **script.maxFailures**.
*Examples*
Note that if you pass a function for the **backOff** calculation, **success** callback or **failure** callback options the function will be serialized to the database as a string and therefore must not rely on any external scope or external variables.
Basic usage example:
**Examples**
Let's say we have an app mounted at `/mailer` that provides a script called `send-mail`:
```js
var Foxx = require("org/arangodb/foxx");
var queue = Foxx.queues.create("my-queue");
queue.push("log", "Hello World!");
queue.push(
{mount: "/mailer", name: "send-mail"},
{to: 'hello@example.com', body: 'Hello world'}
);
```
This will **not** work, because **console** was defined outside the callback function:
This will **not** work, because **log** was defined outside the callback function:
```js
var Foxx = require("org/arangodb/foxx");
var queue = Foxx.queues.create("my-queue");
var console = require("console"); // outside the callback's function scope
queue.push("log", "Hello World!", {
success: function () {
console.log("Yay!"); // throws "console is not defined"
}
});
var log = require("console").log; // outside the callback's function scope
queue.push(
{mount: "/mailer", name: "send-mail"},
{to: 'hello@example.com', body: 'Hello world'},
{success: function () {
log("Yay!"); // throws "console is not defined"
}}
);
```
!SECTION Fetching a job from the queue
@ -241,58 +207,68 @@ assertEqual(queue.complete("log").length, 1);
!SUBSECTION Fetching an array of pending jobs in a queue
`Queue::pending([type])`
`Queue::pending([script])`
Returns an array of job ids of jobs in the given queue with the status **"pending"**, optionally filtered by the given job type.
The jobs will be looked for in the specified queue in the context of the current database.
**Parameters**
* **type** (optional): the name of the job type to filter the results
* **script** (optional): an object with the following properties:
* **name**: name of the script.
* **mount**: mount path of the app defining the script.
!SUBSECTION Fetching an array of jobs that are currently in progress
`Queue::progress([type])`
`Queue::progress([script])`
Returns an array of job ids of jobs in the given queue with the status **"progress"**, optionally filtered by the given job type.
The jobs will be looked for in the specified queue in the context of the current database.
**Parameters**
* **type** (optional): the name of the job type to filter the results
* **script** (optional): an object with the following properties:
* **name**: name of the script.
* **mount**: mount path of the app defining the script.
!SUBSECTION Fetching an array of completed jobs in a queue
`Queue::complete([type])`
`Queue::complete([script])`
Returns an array of job ids of jobs in the given queue with the status **"complete"**, optionally filtered by the given job type.
The jobs will be looked for in the specified queue in the context of the current database.
**Parameters**
* **type** (optional): the name of the job type to filter the results.
* **script** (optional): an object with the following properties:
* **name**: name of the script.
* **mount**: mount path of the app defining the script.
!SUBSECTION Fetching an array of failed jobs in a queue
`Queue::failed([type])`
`Queue::failed([script])`
Returns an array of job ids of jobs in the given queue with the status **"failed"**, optionally filtered by the given job type.
The jobs will be looked for in the specified queue in the context of the current database.
**Parameters**
* **type** (optional): the name of the job type to filter the results.
* **script** (optional): an object with the following properties:
* **name**: name of the script.
* **mount**: mount path of the app defining the script.
!SUBSECTION Fetching an array of all jobs in a queue
`Queue::all([type])`
`Queue::all([script])`
Returns an array of job ids of all jobs in the given queue, optionally filtered by the given job type.
The jobs will be looked for in the specified queue in the context of the current database.
**Parameters**
* **type** (optional): the name of the job type to filter the results.
* **script** (optional): an object with the following properties:
* **name**: name of the script.
* **mount**: mount path of the app defining the script.
!SECTION Aborting a job
@ -301,4 +277,3 @@ Aborts a non-completed job.
`Job::abort()`
Sets a job's status to **"failed"** if it is not already **"complete"**, without calling the job's **onFailure** callback.

146
Documentation/Books/Users/Foxx/Develop/Scripts.mdpp
View File

@ -1,81 +1,123 @@
!CHAPTER Administrative Scripts
!CHAPTER Scripts and the Job Queue
Foxx offers hooks for administrative scripts that are executed once at a specified point.
These scripts are used to create or drop collections, create indexes or insert seed data.
Foxx allows you to define scripts that can be executed as part of the installation and removal process, invoked manually or scheduled to run at a later time using the job queue.
!SECTION Setup script
The setup script will be executed during the install process of your Foxx.
To register your script, just add a **scripts** section to your application manifest:
```json
{
...
"scripts": {
"setup": "scripts/setup.js",
"send-mail": "scripts/send-mail.js"
}
...
}
```
The scripts you define in your application manifest can be invoked from the web admin frontend or with the **foxx-manager** CLI:
```sh
unix>foxx-manager run /my-app-mount-point send-mail '{"to": "user@example.com", "body": "Hello"}'
```
You can also use the script with the Foxx job queue:
```js
var Foxx = require('org/arangodb/foxx');
var queue = Foxx.queues.get('default');
queue.push(
{mount: '/my-app-mount-point', name: 'send-mail'},
{to: 'user@example.com', body: 'Hello'}
);
```
For more information the Foxx job queue, see [the chapter about queues](./Queues.html).
!SECTION Script arguments and return values
If the script was invoked with any arguments, you can access them using the **applicationContext.argv** array.
To return data from your script, you can assign the data to **module.exports** as usual. Please note that this data will be converted to JSON.
Any errors raised by the script will be handled depending on how the script was invoked:
* if the script was invoked with the **foxx-manager** CLI, it will exit with a non-zero exit status and print the error message.
* if the script was invoked from the HTTP API (e.g. using the web admin frontend), it will return an error response using the exception's `statusCode` property if specified or 500.
* if the script was invoked fromt the Foxx job queue, the job's failure counter will be incremented and the job will be rescheduled or marked as failed if no attempts remain.
**Examples**
Let's say you want to define a script that takes two numeric values and returns the result of multiplying them:
```js
var assert = require('assert');
var argv = applicationContext.argv;
assert.equal(argv.length, 2, 'Expected exactly two arguments');
assert.equal(typeof argv[0], 'number', 'Expected first argument to be a number');
assert.equal(typeof argv[1], 'number', 'Expected second argument to be a number');
module.exports = argv[0] * argv[1];
```
!SECTION Life-Cycle Scripts
Foxx recognizes life-cycle scripts if they are defined and will invoke them during the installation, update and removal process of the app if you want.
The following scripts are currently recognized as life-cycle scripts:
!SUBSECTION Setup Script
The **setup** script will be executed without arguments during the installion of your Foxx app:
```sh
unix>foxx-manager install hello-foxx /example
```
It is typically used to setup collections, insert some seed data or create default user accounts.
As one example for a setup script we can take a look at the one given with hello-foxx:
Note that the app may not yet have been configured when the setup script is executed, so the **applicationContext.configuration** and **applicationContext.dependencies** may be unavailable.
```
var console = require("console");
var arangodb = require("org/arangodb");
var db = arangodb.db;
The setup script is typically used to create collections your app needs or insert seed data like initial administrative user accounts and so on.
var texts = applicationContext.collectionName("texts");
**Examples**
if (db._collection(texts) === null) {
var collection = db._create(texts);
```js
var db = require("org/arangodb").db;
var textsCollectionName = applicationContext.collectionName("texts");
// `textsCollectionName` is now the prefixed name of this app's "texts" collection.
// e.g. "example_texts" if the app has been mounted at `/example`
if (db._collection(textsCollectionName) === null) {
var collection = db._create(textsCollectionName);
collection.save({ text: "entry 1 from collection texts" });
collection.save({ text: "entry 2 from collection texts" });
collection.save({ text: "entry 3 from collection texts" });
}
else {
} else {
console.log("collection '%s' already exists. Leaving it untouched.", texts);
}
```
It first creates the collection texts, specific for this application and inserts three default documents.
!SUBSECTION Teardown Script
!SECTION Teardown script
The **teardown** script will be executed without arguments during the removal of your Foxx app:
The teardown script will be executed during the uninstall process of your Foxx.
```
```sh
unix>foxx-manager uninstall /example
````
This one is typically used to erase the data collected with the Foxx.
ArangoDB will never automatically delete collections attached to a Foxx unless they are dropped in the teardown script.
So if you want to keep your data simply do not drop the collections here.
As an example for a typical teardown script we can again take a look at the hello-foxx example:
It can also optionally be executed before upgrading an app.
```
var arangodb = require("org/arangodb");
var db = arangodb.db;
This script typically removes the collections and/or documents created by your app's **setup** script.
var texts = applicationContext.collectionName("texts");
var collection = db._collection(texts);
**Examples**
if (collection !== null) {
collection.drop();
}
```
```js
var db = require("org/arangodb").db;
It drops the collection unless it has not yet been dropped.
var textsCollection = applicationContext.collection("texts");
!SECTION Activate scripts
In order to activate the scripts you have to define them inside the manifest, similar to controllers.
So your manifest will look like this:
```
{
"name": "hello-foxx",
"version": "1.4.4",
"controllers": {},
"setup": "scripts/setup.js",
"teardown": "scripts/teardown.js"
if (textsCollection) {
textsCollection.drop();
}
```
Keep in mind that all scripts are optional and can be omitted.
It is only possible to define one script of each type.
```

9
js/server/modules/org/arangodb/foxx/queues.js
View File

@ -162,7 +162,14 @@ function getJobs(queue, status, type) {
}
if (type !== undefined) {
query = query.filter(qb.ref('@type').eq('job.type'));
if (typeof type === 'string') {
query = query.filter(qb.ref('@type').eq('job.type'));
} else {
query = query.filter(
qb.ref('@type.name').eq('job.type.name')
.and(qb.ref('@type.mount').eq('job.type.mount'))
);
}
vars.type = type;
}