mirror of https://gitee.com/bigwinds/arangodb
510 lines
17 KiB
Plaintext
510 lines
17 KiB
Plaintext
!CHAPTER HTTP Interface for Async Results Management
|
|
!SUBSECTION Request Execution
|
|
|
|
ArangoDB provides various methods of executing client requests. Clients can choose the appropriate method on a per-request level based on their throughput, control flow, and durability requirements.
|
|
|
|
!SUBSUBSECTION Blocking execution
|
|
|
|
ArangoDB is a multi-threaded server, allowing the processing of multiple client
|
|
requests at the same time. Communication handling and the actual work can be performed
|
|
by multiple worker threads in parallel.
|
|
|
|
Though multiple clients can connect and send their requests in parallel to ArangoDB,
|
|
clients may need to wait for their requests to be processed.
|
|
|
|
By default, the server will fully process an incoming request and then return the
|
|
result to the client. The client must wait for the server's response before it can
|
|
send additional requests over the connection. For clients that are single-threaded
|
|
or not event-driven, waiting for the full server response may be non-optimal.
|
|
|
|
Furthermore, please note that even if the client closes the HTTP
|
|
connection, the request running on the server will still continue until
|
|
it is complete and only then notice that the client no longer listens.
|
|
Thus closing the connection does not help to abort a long running query!
|
|
See below under [Async Execution and later Result Retrieval](#async_execution_and_later_result_retrieval) and [HttpJobPutCancel](../HttpAsyncResultsManagement/index.html#managing_async_results_via_http) for details.
|
|
|
|
!SUBSUBSECTION Fire and Forget
|
|
|
|
To mitigate client blocking issues, ArangoDB since version 1.4. offers a generic mechanism
|
|
for non-blocking requests: if clients add the HTTP header *x-arango-async: true* to their
|
|
requests, ArangoDB will put the request into an in-memory task queue and return an HTTP 202
|
|
(accepted) response to the client instantly. The server will execute the tasks from
|
|
the queue asynchronously, decoupling the client requests and the actual work.
|
|
|
|
This allows for much higher throughput than if clients would wait for the server's
|
|
response. The downside is that the response that is sent to the client is always the
|
|
same (a generic HTTP 202) and clients cannot make a decision based on the actual
|
|
operation's result. In fact, the operation might have not even been executed at the
|
|
time the generic response has reached the client. Clients can thus not rely on their
|
|
requests having been processed successfully.
|
|
|
|
The asynchronous task queue on the server is not persisted, meaning not-yet processed
|
|
tasks from the queue might be lost in case of a crash.
|
|
|
|
Clients should thus not send the extra header when they have strict durability
|
|
requirements or if they rely on result of the sent operation for further actions.
|
|
|
|
The maximum number of queued tasks is determined by the startup option
|
|
*-scheduler.maximal-queue-size*. If more than this number of tasks are already queued,
|
|
the server will reject the request with an HTTP 500 error.
|
|
|
|
Finally, please note that it is not possible to cancel such a
|
|
non-blocking request after the fact. If you need to cancel requests,
|
|
use [Async Execution and later Result Retrieval](#async_execution_and_later_result_retrieval)
|
|
and [HttpJobPutCancel](..#managing_async_results_via_http) below.
|
|
|
|
!SUBSUBSECTION Async Execution and later Result Retrieval
|
|
|
|
By adding the HTTP header *x-arango-async: store* to a request, clients can instruct
|
|
the ArangoDB server to execute the operation asynchronously as [above](#fire_and_forget)
|
|
"above", but also store the operation result in memory for a later retrieval. The
|
|
server will return a job id in the HTTP response header *x-arango-async-id*. The client
|
|
can use this id in conjunction with the HTTP API at */_api/job*, which is described in
|
|
detail in this manual.
|
|
|
|
Clients can ask the ArangoDB server via the async jobs API which results are
|
|
ready for retrieval, and which are not. Clients can also use the async jobs API to
|
|
retrieve the original results of an already executed async job by passing it the
|
|
originally returned job id. The server will then return the job result as if the job was
|
|
executed normally. Furthermore, clients can cancel running async jobs by
|
|
their job id, see [HttpJobPutCancel](../HttpAsyncResultsManagement/index.html#managing_async_results_via_http).
|
|
|
|
ArangoDB will keep all results of jobs initiated with the *x-arango-async: store*
|
|
header. Results are removed from the server only if a client explicitly asks the
|
|
server for a specific result.
|
|
|
|
The async jobs API also provides methods for garbage collection that clients can
|
|
use to get rid of "old" not fetched results. Clients should call this method periodically
|
|
because ArangoDB does not artificially limit the number of not-yet-fetched results.
|
|
|
|
It is thus a client responsibility to store only as many results as needed and to fetch
|
|
available results as soon as possible, or at least to clean up not fetched results
|
|
from time to time.
|
|
|
|
The job queue and the results are kept in memory only on the server, so they might be
|
|
lost in case of a crash.
|
|
|
|
!SUBSUBSECTION Canceling asynchronous jobs
|
|
|
|
As mentioned above it is possible to cancel an asynchronously running
|
|
job using its job ID. This is done with a PUT request as described in
|
|
[HttpJobPutCancel](../HttpAsyncResultsManagement/index.html#managing_async_results_via_http).
|
|
|
|
However, a few words of explanation about what happens behind the
|
|
scenes are in order. Firstly, a running async query can internally be
|
|
executed by C++ code or by JavaScript code. For example CRUD operations
|
|
are executed directly in C++, whereas AQL queries and transactions
|
|
are executed by JavaScript code. The job cancellation only works for
|
|
JavaScript code, since the mechanism used is simply to trigger an
|
|
uncatchable exception in the JavaScript thread, which will be caught
|
|
on the C++ level, which in turn leads to the cancellation of the job.
|
|
No result can be retrieved later, since all data about the request is
|
|
discarded.
|
|
|
|
If you cancel a job running on a coordinator of a cluster (Sharding),
|
|
then only the code running on the coordinator is stopped, there may
|
|
remain tasks within the cluster which have already been distributed to
|
|
the DBservers and it is currently not possible to cancel them as well.
|
|
|
|
!SUBSUBSECTION Async Execution and Authentication
|
|
|
|
If a request requires authentication, the authentication procedure is run before
|
|
queueing. The request will only be queued if it valid credentials and the authentication
|
|
succeeds. If the request does not contain valid credentials, it will not be queued but
|
|
rejected instantly in the same way as a "regular", non-queued request.
|
|
|
|
!CHAPTER Managing Async Results via HTTP
|
|
|
|
<!-- -------------------------------------------------------------------------------- -->
|
|
|
|
@RESTHEADER{PUT /_api/job/job-id, Return result of an async job}
|
|
|
|
@RESTURLPARAMS
|
|
|
|
@RESTURLPARAM{job-id,string,required}
|
|
The async job id.
|
|
|
|
@RESTDESCRIPTION
|
|
Returns the result of an async job identified by job-id. If the async job
|
|
result is present on the server, the result will be removed from the list of
|
|
result. That means this method can be called for each job-id once.
|
|
The method will return the original job result's headers and body, plus the
|
|
additional HTTP header x-arango-async-job-id. If this header is present, then
|
|
the job was found and the response contains the original job's result. If
|
|
the header is not present, the job was not found and the response contains
|
|
status information from the job manager.
|
|
|
|
@RESTRETURNCODES
|
|
|
|
@RESTRETURNCODE{204}
|
|
is returned if the job requested via job-id is still in the queue of pending
|
|
(or not yet finished) jobs. In this case, no x-arango-async-id HTTP header
|
|
will be returned.
|
|
|
|
@RESTRETURNCODE{400}
|
|
is returned if no job-id was specified in the request. In this case,
|
|
no x-arango-async-id HTTP header will be returned.
|
|
|
|
@RESTRETURNCODE{404}
|
|
is returned if the job was not found or already deleted or fetched from
|
|
the job result list. In this case, no x-arango-async-id HTTP header will
|
|
be returned.
|
|
|
|
@EXAMPLES
|
|
Not providing a job-id:
|
|
|
|
```js
|
|
unix> curl -X PUT --dump - http://localhost:8529/_api/job/
|
|
|
|
HTTP/1.1 400 Bad Request
|
|
content-type: application/json; charset=utf-8
|
|
|
|
{"error":true,"errorMessage":"bad parameter","code":400,"errorNum":400}
|
|
```
|
|
|
|
Providing a job-id for a non-existing job:
|
|
|
|
```js
|
|
unix> curl -X PUT --dump - http://localhost:8529/_api/job/foobar
|
|
|
|
HTTP/1.1 404 Not Found
|
|
content-type: application/json; charset=utf-8
|
|
|
|
{"error":true,"errorMessage":"not found","code":404,"errorNum":404}
|
|
```
|
|
|
|
Fetching the result of an HTTP GET job:
|
|
|
|
```js
|
|
unix> curl --header 'x-arango-async: store' --dump - http://localhost:8529/_api/version
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 265413601
|
|
|
|
unix> curl -X PUT --dump - http://localhost:8529/_api/job/265413601
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: application/json; charset=utf-8
|
|
x-arango-async-id: 265413601
|
|
|
|
{"server":"arango","version":"2.1.0"}
|
|
```
|
|
|
|
Fetching the result of an HTTP POST job that failed:
|
|
|
|
```js
|
|
unix> curl -X POST --header 'x-arango-async: store' --data-binary @- --dump - http://localhost:8529/_api/collection
|
|
{"name":" this name is invalid "}
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 265479137
|
|
|
|
unix> curl -X PUT --dump - http://localhost:8529/_api/job/265479137
|
|
|
|
HTTP/1.1 400 Bad Request
|
|
content-type: application/json; charset=utf-8
|
|
x-arango-async-id: 265479137
|
|
|
|
{"error":true,"code":400,"errorNum":1208,"errorMessage":"cannot create collection: illegal name"}
|
|
```
|
|
|
|
<!-- -------------------------------------------------------------------------------- -->
|
|
|
|
@RESTHEADER{PUT /_api/job/job-id/cancel, Cancel async job}
|
|
|
|
@RESTURLPARAMS
|
|
|
|
@RESTURLPARAM{job-id,string,required}
|
|
The async job id.
|
|
|
|
@RESTDESCRIPTION
|
|
Cancels the currently running job identified by job-id. Note that it still
|
|
might take some time to actually cancel the running async job.
|
|
|
|
@RESTRETURNCODES
|
|
|
|
@RESTRETURNCODE{200}
|
|
cancel has been initiated.
|
|
|
|
@RESTRETURNCODE{400}
|
|
is returned if no job-id was specified in the request. In this case,
|
|
no x-arango-async-id HTTP header will be returned.
|
|
|
|
@RESTRETURNCODE{404}
|
|
is returned if the job was not found or already deleted or fetched from
|
|
the job result list. In this case, no x-arango-async-id HTTP header will
|
|
be returned.
|
|
|
|
@EXAMPLES
|
|
|
|
```js
|
|
unix> curl -X POST --header 'x-arango-async: store' --data-binary @- --dump - http://localhost:8529/_api/cursor
|
|
{"query": "FOR i IN 1..10 FOR j IN 1..10 LET x = sleep(1.0) FILTER i == 5 && j == 5 RETURN 42"}
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 268952545
|
|
|
|
unix> curl --dump - http://localhost:8529/_api/job/pending
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: application/json; charset=utf-8
|
|
|
|
["268952545"]
|
|
|
|
unix> curl -X PUT --dump - http://localhost:8529/_api/job/268952545/cancel
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: application/json; charset=utf-8
|
|
|
|
{"result":true}
|
|
|
|
unix> curl --dump - http://localhost:8529/_api/job/pending
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: application/json; charset=utf-8
|
|
|
|
["268952545"]
|
|
```
|
|
|
|
<!-- -------------------------------------------------------------------------------- -->
|
|
|
|
@RESTHEADER{DELETE /_api/job/type, Deletes async job}
|
|
|
|
@RESTURLPARAMS
|
|
|
|
@RESTURLPARAM{type,string,required}
|
|
The type of jobs to delete. type can be:
|
|
* *all*: Deletes all jobs results. Currently executing or queued async
|
|
jobs will not be stopped by this call.
|
|
* *expired*: Deletes expired results. To determine the expiration status of a
|
|
result, pass the stamp URL parameter. stamp needs to be a UNIX timestamp,
|
|
and all async job results created at a lower timestamp will be deleted.
|
|
* *an actual job-id*: In this case, the call will remove the result of the
|
|
specified async job. If the job is currently executing or queued, it will not be aborted.
|
|
|
|
@RESTQUERYPARAMS
|
|
|
|
@RESTPARAM{stamp, number, optional}
|
|
|
|
A UNIX timestamp specifying the expiration threshold when type is expired.
|
|
|
|
@RESTDESCRIPTION
|
|
Deletes either all job results, expired job results, or the result of a specific job.
|
|
Clients can use this method to perform an eventual garbage collection of job results.
|
|
|
|
@RESTRETURNCODES
|
|
|
|
@RESTRETURNCODE{200}
|
|
is returned if the deletion operation was carried out successfully.
|
|
This code will also be returned if no results were deleted.
|
|
|
|
@RESTRETURNCODE{400}
|
|
is returned if type is not specified or has an invalid value.
|
|
|
|
@RESTRETURNCODE{404}
|
|
is returned if type is a job-id but no async job with the specified id was found.
|
|
|
|
@EXAMPLES
|
|
|
|
Deleting all jobs:
|
|
|
|
```js
|
|
unix> curl --header 'x-arango-async: store' --dump - http://localhost:8529/_api/version
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 270132193
|
|
|
|
unix> curl -X DELETE --dump - http://localhost:8529/_api/job/all
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: application/json; charset=utf-8
|
|
|
|
{
|
|
"result" : true
|
|
}
|
|
```
|
|
|
|
Deleting expired jobs:
|
|
|
|
```js
|
|
unix> curl --header 'x-arango-async: store' --dump - http://localhost:8529/_api/version
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 270197729
|
|
|
|
unix> curl -X DELETE --dump - http://localhost:8529/_api/job/expired?stamp=1401376184
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: application/json; charset=utf-8
|
|
|
|
{
|
|
"result" : true
|
|
}
|
|
```
|
|
|
|
Deleting the result of a specific job:
|
|
|
|
```js
|
|
unix> curl --header 'x-arango-async: store' --dump - http://localhost:8529/_api/version
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 270263265
|
|
|
|
unix> curl -X DELETE --dump - http://localhost:8529/_api/job/270263265
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: application/json; charset=utf-8
|
|
|
|
{
|
|
"result" : true
|
|
}
|
|
```
|
|
|
|
Deleting the result of a non-existing job:
|
|
|
|
```js
|
|
unix> curl -X DELETE --dump - http://localhost:8529/_api/job/foobar
|
|
|
|
HTTP/1.1 404 Not Found
|
|
content-type: application/json; charset=utf-8
|
|
|
|
{
|
|
"error" : true,
|
|
"errorMessage" : "not found",
|
|
"code" : 404,
|
|
"errorNum" : 404
|
|
}
|
|
```
|
|
|
|
<!-- -------------------------------------------------------------------------------- -->
|
|
|
|
@RESTHEADER{GET /_api/job/job-id, Returns async job}
|
|
|
|
@RESTURLPARAMS
|
|
|
|
@RESTURLPARAM{job-id,string,required}
|
|
The async job id.
|
|
|
|
@RESTDESCRIPTION
|
|
Returns the processing status of the specified job. The processing status can be
|
|
determined by peeking into the HTTP response code of the response.
|
|
|
|
@RESTRETURNCODES
|
|
|
|
@RESTRETURNCODE{200}
|
|
is returned if the job requested via job-id has been executed
|
|
and its result is ready to fetch.
|
|
|
|
@RESTRETURNCODE{204}
|
|
is returned if the job requested via job-id is still in the queue of pending
|
|
(or not yet finished) jobs.
|
|
|
|
@RESTRETURNCODE{404}
|
|
is returned if the job was not found or already deleted or fetched from the job result list.
|
|
|
|
@EXAMPLES
|
|
|
|
Querying the status of a done job:
|
|
|
|
```js
|
|
unix> curl --header 'x-arango-async: store' --dump - http://localhost:8529/_api/version
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 270328801
|
|
|
|
unix> curl --dump - http://localhost:8529/_api/job/270328801
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: text/plain; charset=utf-8
|
|
|
|
Querying the status of a pending job:
|
|
|
|
unix> curl --header 'x-arango-async: store' --dump - http://localhost:8529/_admin/sleep?duration=3
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 270394337
|
|
|
|
unix> curl --dump - http://localhost:8529/_api/job/270394337
|
|
|
|
HTTP/1.1 204 No Content
|
|
content-type: text/plain; charset=utf-8
|
|
```
|
|
|
|
<!-- -------------------------------------------------------------------------------- -->
|
|
|
|
@RESTHEADER{GET /_api/job/type, Returns list of async job}
|
|
|
|
@RESTURLPARAMS
|
|
|
|
@RESTURLPARAM{type,string,required}
|
|
The type of jobs to return. The type can be either done or pending. Setting
|
|
the type to done will make the method return the ids of already completed async
|
|
jobs for which results can be fetched. Setting the type to pending will return
|
|
the ids of not yet finished async jobs.
|
|
|
|
@RESTQUERYPARAMS
|
|
|
|
@RESTPARAM{count, number, optional}
|
|
|
|
The maximum number of ids to return per call. If not specified, a
|
|
server-defined maximum value will be used.
|
|
|
|
@RESTDESCRIPTION
|
|
Returns the list of ids of async jobs with a specific status (either done or pending).
|
|
The list can be used by the client to get an overview of the job system status and
|
|
to retrieve completed job results later.
|
|
|
|
@RESTRETURNCODES
|
|
|
|
@RESTRETURNCODE{200}
|
|
is returned if the list can be compiled successfully. Note: the list might be empty.
|
|
|
|
@RESTRETURNCODE{400}
|
|
is returned if type is not specified or has an invalid value.
|
|
|
|
@EXAMPLES
|
|
|
|
Fetching the list of done jobs:
|
|
|
|
```js
|
|
unix> curl --header 'x-arango-async: store' --dump - http://localhost:8529/_api/version
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 270459873
|
|
|
|
unix> curl --dump - http://localhost:8529/_api/job/done
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: application/json; charset=utf-8
|
|
|
|
[
|
|
"270459873"
|
|
]
|
|
```
|
|
|
|
Fetching the list of pending jobs:
|
|
|
|
```js
|
|
unix> curl --header 'x-arango-async: store' --dump - http://localhost:8529/_api/version
|
|
|
|
HTTP/1.1 202 Accepted
|
|
content-type: text/plain; charset=utf-8
|
|
x-arango-async-id: 270525409
|
|
|
|
unix> curl --dump - http://localhost:8529/_api/job/pending
|
|
|
|
HTTP/1.1 200 OK
|
|
content-type: application/json; charset=utf-8
|
|
|
|
[ ]
|
|
```
|