1
0
Fork 0
Commit Graph

1439 Commits

Author SHA1 Message Date
Frank Celler aa3d3f8e40
Feature/cleanup ccpcheck (#9665) 2019-08-12 11:11:49 +02:00
Jan 6ad0a995b8
various replication improvements: (#9676)
- better debuggability (more log details)
- shorter minimum wait delay in active failover
- fixed too early pruning of WAL files on leaders
2019-08-12 10:53:20 +02:00
Dan Larkin-York 3d0246cb18 Decentralize includes (#9623) 2019-08-06 15:32:09 +02:00
Lars Maier ed496fe5dd Feature/hotbackup devel (#9495)
Hotbackup
2019-08-02 11:39:46 +02:00
Jan 5cc8e8ffb9
remove an optimization that is none (#9578) 2019-07-29 08:42:42 +02:00
Andrey Abramov d91fb35e39
Bug fix/iresearch trx (#9536)
* add tescase for trx and iresearch

* iresearch transaction integration

* adjust test

* reverse operation for correct index

* remove debug output

* address jsling errors

* address review comments

* forgot something

* added missing fail
2019-07-23 17:44:29 +03:00
Jan b5932c2edf
make TTL indexes behave like other indexes on creation (#9546)
* make TTL indexes behave like other indexes on creation

if a TTL index is already present on a collection, the previous behavior
was to make subsequent calls to `ensureIndex` fail unconditionally with
the error "there can only be one ttl index per collection".

now, we are comparing the attributes of the to-be-created index with the
attributes of the existing TTL index and make it only fail when the
attributes differ. if the attributes are identical, the `ensureIndex`
call succeeds and returns the existing index.
2019-07-23 13:53:58 +02:00
Tobias Gödderz 7e98f56cf5 Bug fix/clean replication api wal tracking (#9473) 2019-07-19 15:44:14 +02:00
Wilfried Goesgens ca0f2b8b86 All hail to the SI (#9445) 2019-07-19 13:52:12 +02:00
Michael Hackstein cbcf561450
Feature/min replication factor (#9433)
* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it

* Added some assertion son minReplicationFactor

* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled

* added minReplicationFactor to the user interface, preparation for the collection api changes

* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo

* added minReplicationFactor usage to tests

* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME

* minReplicationFactor now able to change via collection  properties route

* fixed wrongly assert

* added minReplicationFactor to the graph management ui

* added minReplicationFactor to the gharial api

* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.

* adjusted description of minReplicationFactor

* FollowerInfo Refactoring

* added gharial api graph creation tests with minimal replication factor

* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests

* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor

* Debug logging

* MORE Debug logging

* Included replication fast lane

* Use correct minreplicationfactor

* modified debug logging

* Fixed compileissues

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* Revert "MORE Debug logging"

This reverts commit dab5af28c0.

* Revert "MORE Debug logging"

This reverts commit 6134b664bd.

* Revert "MORE Debug logging"

This reverts commit 80160bdf3b.

* Revert "MORE Debug logging"

This reverts commit 06aabcdfe1.

* Removed debug output

* Added replication fast lane. Also refactored the commands as i cannot take it any more...

* Put some requests of RocksDBReplication onto CATCHUP Lane.

* Put some requests of MMFilesReplication onto CATCHUP Lane.

* Adjusted Fast and MED lane usage in Supervised scheduler

* Added changelog entry

* Added new features entry

* A new leader will now keep old followers in case of failover

* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Fixed JSLINT

* Unified lane handling of replication handlers

* Sorry forgotten in last commit

* replaced strings with static strings

* more use of static strings

* optimized min repl description in the ui

* decr initial loop variable

* clean up of the createWithId test

* more use of static strings

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Added some comments on condition, renamed variable as suggested in review

* Added check for min replicationFactor to be non-zero

* Added assertion

* Added function to modify min and max replication factor in one go

* added missing semicolon

* rm log devel

* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place

* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers

* check replFactor against nr dbservers

* Add lie reporting in CURRENT

* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out

* move replication checks from logical collection to rest collection handler

* added more replication tests

* Include assert only if we are not in gtest

* jslint

* set min repl factor to zero if satellite collection

* check replication attributes in v8 collection

* fixed ires tests

* fixed wrong assert

* properly check uint

* repl factor attr check

* adjusted test to be more preciese now

* Fixed race on atomics comparison

* Fixed invalid number type

* Update tests/js/common/shell/shell-cluster-collection.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review fixes

* More review fixes
2019-07-19 13:02:28 +02:00
Andrey Abramov 5bc6eac636
bug-fix/internal-issue-#609 (#9464)
* allow to access last operation tick of the transaction

* modify IResearchLink to rely on last commtted tick

* get rid of writing arangosearch markers to WAL

* further implementation

* local changes in iresearch

* add recovery states

* properly handle link creation during recovery

* adjust test cases

* properly handle nested transactions

* ungreylist recovery tests

* add more recovery tests

* do not use transaction to pass recovery tick

* do not store recoveryTick in MMFilesRecoveryState

* adjust tests

* fix mmfiles

* cleanup

* add context validity check

* fix tests

* fix more tests

* ensure subscription is not being released during commit

* address review comment

* final cleanup

* fix crash

* fix tests

* address test failures

* address review comments

* address review comments

* properly set recovery tick even if no recovery happened
2019-07-19 11:10:56 +03:00
Simon e5507d840f Feature/comm task refactor (#9426) 2019-07-16 09:43:25 +02:00
Jan c52f2a8315
refactoring (#9411) 2019-07-09 11:15:52 +02:00
Tobias Gödderz f501e00e9d Bug fix/add shard id to replication client identifier (#9366) 2019-07-08 14:03:42 +02:00
Simon 9eaeb6ce37 use a lock when calling unload (#9375) 2019-07-04 09:36:19 +02:00
Jan 1a58cc2213
add VelocyPackHelper::equal method (#9389) 2019-07-03 12:15:11 +02:00
Jan 9cb08ded92
make the comparison functions unambiguous (#9349)
* make the comparison functions unambiguous

* added @kaveh's suggestion
2019-07-01 16:35:28 +02:00
Jan fe19b8aaae
don't unfairly favor the edge index when comparing it to more specialized indexes on `_from`/`_to` plus other attributes (#9362) 2019-07-01 16:32:42 +02:00
Jan 64c8079186
Forbid indexes on _id (#9337) 2019-06-28 11:11:22 +02:00
Jan 32ce797be4
make it possible to limit the number of parallel WAL tailing invocations (#9344) 2019-06-27 18:43:53 +02:00
Simon 683552ac13 Bug fix/simplify cache setup (#9341) 2019-06-27 15:44:39 +02:00
Jan 6f0c116130
clean up usage costs (#9237) 2019-06-26 17:01:22 +02:00
Simon 739b6f273e fix windows warnings (#9326) 2019-06-26 09:24:43 +02:00
Simon cf7cf0131b Try to fix corruption error (#9258) 2019-06-25 10:18:26 +02:00
Simon cc125b377c Use GetForUpdate on unique indexes (#9234)
* Use GetForUpdate on unique indexes

* add changelog

* fix loop

* use c++11 loops

* move some code

* fix error code check
2019-06-13 16:32:24 +02:00
Jan 59d84e8301 attempt to remove expired WAL files on agents (#9231) 2019-06-09 19:46:45 +02:00
Jan Christoph Uhde 3f603f024f remove some containers from common.h (#9223)
* remove some containers from Common.h

* enterprise fixes
2019-06-07 13:27:24 +02:00
Jan eb22eac2d1
display reason for error in background listener, plus ignore shutdown (#9206) 2019-06-07 10:11:12 +02:00
Jan 3364ea6273
refactor index APIs (#9145) 2019-05-31 11:12:39 +02:00
Jan a61dd2895e
only offer indexes of type "persistent" in the web UI for the RocksDB (#9046) 2019-05-21 10:11:07 +02:00
Jan 49babbc66b
handle incomplete read errors (#9021) 2019-05-20 14:13:41 +02:00
Simon 394c070a4f Do not check isEnabled (#8997) 2019-05-17 16:33:17 +02:00
Wilfried Goesgens 1907a7211b Bug fix/cleanup system includes (#8962) 2019-05-15 15:12:59 +02:00
Jan c1d48c0fef
honor index hints in optimizer rule (#8940) 2019-05-09 18:47:36 +02:00
Simon 0502a97abb forwardport changes from 3.4 (#8894) 2019-05-08 14:34:25 +02:00
Simon 0a0cb41f04 do not acquire a snapshot for a single read op (#8916) 2019-05-07 09:50:14 +02:00
Jan 0cbdfe9289
Bug fix/vpack update (#8875) 2019-04-30 12:33:26 +02:00
Jan ca94c88a79
next hack attempt (#8802) 2019-04-29 17:09:05 +02:00
Simon 1145c7c053 Improve estimator apply algo + test (#8871) 2019-04-29 17:08:14 +02:00
jsteemann 3f4ea17b32 RocksDB returns certain statistics values as strings. honor that 2019-04-26 19:05:23 +02:00
Simon 6381659a53 fix assertion (#8860) 2019-04-26 18:51:34 +02:00
Simon 2f1cfa40e0 Fix tick overflow (#8840) 2019-04-26 11:07:04 +02:00
Simon 729e506876 Remove duplicate code (#8825) 2019-04-26 10:53:35 +02:00
Tobias Gödderz 1bf19cdc91 Read docs in EnumerateCollection in batches (#8823)
* Read docs in EnumerateCollection in batches

* Read docs in IndexExecutor in batches

* Fixed stats with batching

* Fixed a bug in RocksDBPrimaryIndexRangeIterator::nextCovering
2019-04-25 11:56:19 +02:00
Jan Christoph Uhde 677a79026c Foxx Security (#8845) 2019-04-25 09:56:29 +02:00
Simon 569198a089 Abort el-cheapo transactions if servers fail (#8799) 2019-04-22 19:31:24 +02:00
jsteemann 90f17e2ee1 ok, cannot simply make these SmallVectors :sad: 2019-04-20 11:15:06 +02:00
Dan Larkin-York 91ed0d6568 Fix issue with truncate and index selectivity estimates (#8805) 2019-04-20 10:14:15 +02:00
Jan 2669544944
Bug fix/fix skip inaccessible (#8779) 2019-04-17 15:54:15 +02:00
Jan 80b021f915
speed up projections with RocksDBPrimaryIndex if there is no FILTER condition (#8770) 2019-04-16 17:40:30 +02:00
Jan 7c0c01c401
use the primary index when full-scanning a collection (#8548) 2019-04-16 16:50:03 +02:00
Jan 1d175f446d
do not use the TTL index for queries which it does not support (#8766) 2019-04-15 18:55:19 +02:00
Dan Larkin-York 777ba1a364 Improved audit logging (#8740) 2019-04-15 14:51:32 +02:00
Jan 13b2d40aef
fix compile warnings (#8743) 2019-04-15 11:22:24 +02:00
Simon 937d743ba6 Bug fix/pregel stuff (#8733) 2019-04-11 15:58:28 +02:00
Simon 676012ca3a snafu (#8728) 2019-04-10 19:14:34 +02:00
Jan c6d3f8e052
Bug fix/pass on error messages (#8690) 2019-04-10 12:34:25 +02:00
Jan 892446ed23
keep g++ from complaining about potential nullptr dereference (#8721) 2019-04-10 11:03:41 +02:00
Simon 4e80e1b351 Simplify recovery code (#8699) 2019-04-09 21:48:04 +02:00
Jan 675b006ebc
this addresses an issue described in ES-260, and is a forward-port of… (#8696) 2019-04-09 11:53:20 +02:00
jsteemann ed1e0336cb fix memleaks 2019-04-09 11:24:03 +02:00
Simon 2b594bdab5 Reduce # of memcpy from storage engine layer (#8685) 2019-04-08 22:43:29 +02:00
Jan 4f7923a971
remove replication clients handling out of vocbase code (#8676) 2019-04-08 19:15:28 +02:00
Jan 5897baa984
added db.<collection>.getResponsibleShard() (#8683) 2019-04-08 16:10:45 +02:00
Dan Larkin-York 8f49fde98f Fix RocksDB recovery ticks (#8682) 2019-04-04 20:38:44 +02:00
Jan 5a19a0320b
remove/fix parameters in crud methods (#8675) 2019-04-03 19:55:15 +02:00
Simon 7cd84a785a Remove Obsolete code (#8657) 2019-04-03 13:40:44 +02:00
Jan ec6a52fb0e
fix smart join optimization when another optimizer rule (reduce-extraction-to-projection) turned full scans into index accesses (#8660) 2019-04-03 10:42:21 +02:00
Jan 616ea94f24
Bug fix/cleanup 31032019 (#8632) 2019-04-01 17:14:11 +02:00
Jan b6df220821
cleanup replication contexts (#8630) 2019-04-01 09:35:08 +02:00
Jan 9ab9cc7857
disambiguate internal exceptions (#8623) 2019-03-29 15:59:37 +01:00
Dan Larkin-York ffc05c05c0 Fix handling of index name conflicts during replication. (#8571) 2019-03-28 18:53:43 +01:00
KVS85 2d5bf58c94 Remove USE_IRESEARCH definition and usage (#8613) 2019-03-28 17:23:51 +01:00
Jan a5e80fa7ed
remove redundant parameter from update API (#8607) 2019-03-28 12:50:20 +01:00
Jan Christoph Uhde c3f7961b88 apply unique log ids (#8561) 2019-03-25 20:26:51 +01:00
Jan e078f35285
fixed typos, removed unneeded includes (#8547) 2019-03-25 12:09:37 +01:00
Jan 39a3f5bc4e
reintroduce smart joins after temporarily reverting them in devel (#8543) 2019-03-23 20:36:02 +01:00
Dan Larkin-York be2cccfa28 Add RocksDB background error listener. (#8535) 2019-03-22 20:25:04 +01:00
Jan cfd3418115
microoptimizations (#8519) 2019-03-22 16:53:21 +01:00
Simon f4ecf8cd9e Fix asan error in truncate (#8523) 2019-03-22 15:22:28 +01:00
Simon 3ada15fc35 The Legendary El Cheapo (#8485) 2019-03-22 11:38:33 +01:00
Jan a35f6ccf10
fix type of hidden startup option `--rocksdb.recycle-log-file-num` (#8517) 2019-03-22 11:21:53 +01:00
jsteemann dc381a99df Revert "Feature/ncc1701 (#8440)"
This reverts commit 59ad583796.
2019-03-21 19:18:46 +01:00
Jan 59ad583796
Feature/ncc1701 (#8440) 2019-03-21 15:05:36 +01:00
Jan 00ac1db850
add RocksDB options `--rocksdb.allow-fallocate` and `--rocksdb.limit-open-files-at-startup` (#8493) 2019-03-21 14:21:35 +01:00
Jan 0bef1c1098
dont attempt to remove non-existing WAL files (#8474) 2019-03-20 17:58:28 +01:00
Jan 6d107beeda
don't run compact() on a collection after a truncate() was done in th… (#8468)
* don't run compact() on a collection after a truncate() was done in the same transaction

running compact() in the same transaction will only increase the data size on disk due to RocksDB not being able to remove
any documents physically due to the snapshot we take at transaction start.
Decoupling the truncate transaction from the compact operation allows finishing the truncate transaction first, so we can
get rid of the snapshot. Running compact afterwards is then free to physically remove all the data.
As a nice side effect this change will also speed up the truncation of larger collections, because the compact will run
faster.

This change also exposes db.<collection>.compact() in the arangosh, in order to manually run a compaction on the data
range of a collection should it be needed for maintenance.

* fix documentation anchors
2019-03-20 16:27:54 +01:00
Jan 2c4accdf34
micro improvements for rocksdb options (#8433) 2019-03-19 12:47:57 +01:00
Simon 49cc3bcd1e Refactorings from cluster trx improvement branch (#8391) 2019-03-14 23:13:17 +01:00
Wilfried Goesgens d88bc376a6 When updating documents with array indices and unique values we mustn't disable indexing in rocksdb (#8404) 2019-03-14 13:35:38 +01:00
Dan Larkin-York 413e90508f Named indices (#8370) 2019-03-13 18:20:32 +01:00
Jan 9d3327c6ea
Bug fix/rearm cursors (#8363) 2019-03-12 15:28:33 +01:00
Vasiliy f5f572dd29 issue 527.1: ensure the cause of the failure is reported to the user (#8371) 2019-03-11 23:42:34 +03:00
Jan 12e11a5197
port of replication improvements from 3.4 (#8308) 2019-03-11 13:37:18 +01:00
Vasiliy e97cd8a0fb issue 526.1: initial support for vocbase-prefixed analyzer names (#8319)
* issue 526.1: initial support for vocbase-prefixed analyzer names

* backport: add tests for new functionality, add temporary workaround for legacy analyzers

* address review comments

* fix typo
2019-03-06 15:13:19 +03:00
Jan 08a6561adc
Bug fix/small issues 21022019 (#8229) 2019-03-01 17:20:10 +01:00
Vasiliy 7130ffa427 issue 525.1: ensure RocksDB CreateIndex/DropIndex WAL markers are properly written during recovery (#8282)
* issue 525.1: ensure RocksDB CreateIndex/DropIndex WAL markers are properly writen during recovery

* account for changes in devel

* skip writing DropIndex marker in recovery
2019-03-01 15:58:11 +03:00
Andrey Abramov d86fb726e6
ArangoSearch index encryption (#8214)
* update iresearch

* add index encryption support

* update iresearch

* encrypt ArangoSearch index using rocksdb encryption provider

* fix merge issue

* update iresearch

* fix cmake

* minor fixes after update

* update iresearch

* partially revert changes from commit 492d05c1f1 since they cause iresearch configuration failures

* Update arangod/IResearch/IResearchRocksDBLink.cpp

Co-Authored-By: gnusi <gnusmas@kvolitek.ru>

* partially revert changes from commit 492d05c1f1 since they cause iresearch configuration failures

* fix cmake configuration for MAX

* another attempt to fix cmake on mac

* Fix 3rdparty cmake for Windows
2019-03-01 15:56:40 +03:00
Jan 5d2ab0c901
port from 3.4 (#8275) 2019-02-28 14:36:29 +01:00
Simon dc24d7fa4e Sync Foxx Queues (#8250) 2019-02-25 17:13:14 +01:00