Hawkin
/
arangodb
mirror of https://gitee.com/bigwinds/arangodb
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Document RocksDB exclusive option (#7517) (#7538)

This commit is contained in:
Simon 2018-11-29 18:42:43 +01:00 committed by Jan
parent 67dc735664
commit 4132870e49
11 changed files with 142 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
Documentation/Books/AQL/Operations/Insert.md
View File

@ -100,6 +100,21 @@ FOR i IN 1..1000
} INTO users OPTIONS { overwrite: true }
```
In contrast to the MMFiles engine, the RocksDB engine does not require collection-level
locks. Different write operations on the same collection do not block each other, as
long as there are no _write-write conficts_ on the same documents. From an application
development perspective it can be desired to have exclusive write access on collections,
to simplify the development. Note that writes do not block reads in RocksDB.
Exclusive access can also speed up modification queries, because we avoid conflict checks.
Use the *exclusive* option to achieve this effect on a per query basis:
```js
FOR doc IN collection
INSERT { myval: doc.val + 1 } INTO users
OPTIONS { exclusive: true }
```
Returning the inserted documents
--------------------------------

17
Documentation/Books/AQL/Operations/Remove.md
View File

@ -118,6 +118,23 @@ FOR i IN 1..1000
REMOVE { _key: CONCAT('test', i), _rev: "1287623" } IN users OPTIONS { ignoreRevs: false }
```
In contrast to the MMFiles engine, the RocksDB engine does not require collection-level
locks. Different write operations on the same collection do not block each other, as
long as there are no _write-write conficts_ on the same documents. From an application
development perspective it can be desired to have exclusive write access on collections,
to simplify the development. Note that writes do not block reads in RocksDB.
Exclusive access can also speed up modification queries, because we avoid conflict checks.
Use the *exclusive* option to achieve this effect on a per query basis:
```js
FOR doc IN collection
REPLACE doc._key
WITH { replaced: true }
OPTIONS { exclusive: true }
```
Returning the removed documents
-------------------------------

17
Documentation/Books/AQL/Operations/Replace.md
View File

@ -110,6 +110,23 @@ FOR i IN 1..1000
REPLACE { _key: CONCAT('test', i), _rev: "1287623" } WITH { foobar: true } IN users OPTIONS { ignoreRevs: false }
```
In contrast to the MMFiles engine, the RocksDB engine does not require collection-level
locks. Different write operations on the same collection do not block each other, as
long as there are no _write-write conficts_ on the same documents. From an application
development perspective it can be desired to have exclusive write access on collections,
to simplify the development. Note that writes do not block reads in RocksDB.
Exclusive access can also speed up modification queries, because we avoid conflict checks.
Use the *exclusive* option to achieve this effect on a per query basis:
```js
FOR doc IN collection
REPLACE doc._key
WITH { replaced: true } IN collection
OPTIONS { exclusive: true }
```
Returning the modified documents
--------------------------------

17
Documentation/Books/AQL/Operations/Update.md
View File

@ -230,6 +230,23 @@ FOR i IN 1..1000
OPTIONS { ignoreRevs: false }
```
In contrast to the MMFiles engine, the RocksDB engine does not require collection-level
locks. Different write operations on the same collection do not block each other, as
long as there are no _write-write conficts_ on the same documents. From an application
development perspective it can be desired to have exclusive write access on collections,
to simplify the development. Note that writes do not block reads in RocksDB.
Exclusive access can also speed up modification queries, because we avoid conflict checks.
Use the *exclusive* option to achieve this effect on a per query basis:
```js
FOR doc IN collection
UPDATE doc
WITH { updated: true } IN collection
OPTIONS { exclusive: true }
```
Returning the modified documents
--------------------------------

17
Documentation/Books/AQL/Operations/Upsert.md
View File

@ -99,6 +99,23 @@ the searchExpression. Even worse, if you use an outdated `_rev` in the searchExp
UPSERT will trigger the INSERT path instead of the UPDATE path, because it has not found a document
exactly matching the searchExpression.
In contrast to the MMFiles engine, the RocksDB engine does not require collection-level
locks. Different write operations on the same collection do not block each other, as
long as there are no _write-write conficts_ on the same documents. From an application
development perspective it can be desired to have exclusive write access on collections,
to simplify the development. Note that writes do not block reads in RocksDB.
Exclusive access can also speed up modification queries, because we avoid conflict checks.
Use the *exclusive* option to achieve this effect on a per query basis:
```js
FOR i IN 1..1000
UPSERT { _key: CONCAT('test', i) }
INSERT { foobar: false }
UPDATE { foobar: true }
IN users OPTIONS { exclusive: true }
```
Returning documents
-------------------

47
Documentation/Books/Manual/DataModeling/OperationalFactors.md
View File

@ -232,7 +232,8 @@ Consider some ways to minimize the required amount of storage space:
store one entry. This will only be beneficial if the combined documents are
regularly retrieved together and not just subsets.
### RockDB Storage Engine
RockDB Storage Engine
---------------------
Especially for the RocksDB storage engine large documents and transactions may
negatively impact the write performance:
@ -242,10 +243,9 @@ negatively impact the write performance:
This means that transactions have to be split if they become too big, see the
[limitations section](../Transactions/Limitations.md#with-rocksdb-storage-engine).
#### Improving Update Query Perfromance
### Improving Update Query Perfromance
Use the _exclusive_ query option for modifying AQL queries on a _single collection_,
to improve the performance drastically.
You may use the _exclusive_ query option for modifying AQL queries, to improve the performance drastically.
This has the downside that no concurrent writes may occur on the collection, but ArangoDB is able
to use a special fast-path which should improve the performance by up to 50% for large collections.
@ -256,5 +256,42 @@ FOR doc IN mycollection
OPTIONS { exclusive: true }
```
The same naturally also applies for queries using _REPLACE_. Additionally you may be able to use
The same naturally also applies for queries using _REPLACE_ or _INSERT_. Additionally you may be able to use
the `intermediateCommitCount` option in the API to subdivide the AQL transaction into smaller batches.
### Read / Write Load Balance
Depending on whether your data model has a higher read- or higher write-rate you may want
to adjust some of the RocksDB specific options. Some of the most critical options to
adjust the performance and memory usage are listed below:
`--rocksdb.block-cache-size`
This is the size of the block cache in bytes. This cache is used for read operations.
Increasing the size of this may improve the performance of read heavy workloads.
You may wish to adjust this parameter to control memory usage.
`--rocksdb.write-buffer-size`
Amount of data to build up in memory before converting to a file on disk.
Larger values increase performance, especially during bulk loads.
`--rocksdb.max-write-buffer-number`
Maximum number of write buffers that built up in memory, per internal column family.
The default and the minimum number is 2, so that when 1 write buffer
is being flushed to storage, new writes can continue to the other write buffer.
`--rocksdb.total-write-buffer-size`
The total amount of data to build up in all in-memory buffers when writing into ArangoDB.
You may wish to adjust this parameter to control memory usage.
Setting this to a low value may limit the RAM that ArangoDB will use but may slow down
write heavy workloads. Setting this to 0 will not limit the size of the write-buffers.
`--rocksdb.level0-stop-trigger`
When this many files accumulate in level-0, writes will be stopped to allow compaction to catch up.
Setting this value very high may improve write throughput, but may lead to temporarily
bad read performance.

2
Documentation/Books/Manual/ReleaseNotes/NewFeatures34.md
View File

@ -162,7 +162,7 @@ per-query/per-transaction basis.
For AQL queries, all data-modification operations now support the `exclusive` option, e.g.
FOR doc IN collection
UPDATE doc WITH { updated: true } OPTIONS { exclusive: true }
UPDATE doc WITH { updated: true } IN collection OPTIONS { exclusive: true }
JavaScript-based transactions can specify which collections to lock exclusively in the
`exclusive` sub-attribute of their `collections` attribute:

2
arangod/RocksDBEngine/RocksDBCollection.cpp
View File

@ -543,7 +543,7 @@ Result RocksDBCollection::truncate(transaction::Methods* trx,
auto state = RocksDBTransactionState::toState(trx);
RocksDBMethods* mthds = state->rocksdbMethods();
if (state->isExclusiveTransactionOnSingleCollection() &&
if (state->isOnlyExclusiveTransaction() &&
state->hasHint(transaction::Hints::Hint::ALLOW_RANGE_DELETE) &&
static_cast<RocksDBEngine*>(EngineSelectorFeature::ENGINE)->canUseRangeDeleteInWal() &&
_numberDocuments >= 32 * 1024) {

5
arangod/RocksDBEngine/RocksDBTransactionState.cpp
View File

@ -87,7 +87,8 @@ Result RocksDBTransactionState::beginTransaction(transaction::Hints hints) {
<< "beginning " << AccessMode::typeString(_type) << " transaction";
TRI_ASSERT(!hasHint(transaction::Hints::Hint::NO_USAGE_LOCK) || !AccessMode::isWriteOrExclusive(_type));
TRI_ASSERT(!hasHint(transaction::Hints::Hint::NO_USAGE_LOCK) ||
!AccessMode::isWriteOrExclusive(_type));
if (_nestingLevel == 0) {
// set hints
@ -147,7 +148,7 @@ Result RocksDBTransactionState::beginTransaction(transaction::Hints hints) {
// with exlusive locking there is no chance of conflict
// with other transactions -> we can use untracked< Put/Delete methods
if (isExclusiveTransactionOnSingleCollection()) {
if (isOnlyExclusiveTransaction()) {
_rocksMethods.reset(new RocksDBTrxUntrackedMethods(this));
} else {
_rocksMethods.reset(new RocksDBTrxMethods(this));

12
arangod/StorageEngine/TransactionState.cpp
View File

@ -330,8 +330,16 @@ void TransactionState::setLockedShards(std::unordered_set<std::string> const& lo
_lockedShards = lockedShards;
}
bool TransactionState::isExclusiveTransactionOnSingleCollection() const {
return ((numCollections() == 1) && (_collections[0]->accessType() == AccessMode::Type::EXCLUSIVE));
bool TransactionState::isOnlyExclusiveTransaction() const {
if (!AccessMode::isWriteOrExclusive(_type)) {
return false;
}
for (TransactionCollection* coll : _collections) {
if (AccessMode::isWrite(coll->accessType())) {
return false;
}
}
return true;
}
int TransactionState::checkCollectionPermission(TRI_voc_cid_t cid,

4
arangod/StorageEngine/TransactionState.h
View File

@ -215,8 +215,8 @@ class TransactionState {
*/
void setLockedShards(std::unordered_set<std::string> const& lockedShards);
/// @brief whether or not a transaction is an exclusive transaction on a single collection
bool isExclusiveTransactionOnSingleCollection() const;
/// @brief whether or not a transaction only has exculsive or read accesses
bool isOnlyExclusiveTransaction() const;
protected:
/// @brief find a collection in the transaction's list of collections