1
0
Fork 0
Commit Graph

45770 Commits

Author SHA1 Message Date
Michael Hackstein 19c25d1e3b Fixed merge conflict marker 2019-07-19 17:26:45 +02:00
Tobias Gödderz 7e98f56cf5 Bug fix/clean replication api wal tracking (#9473) 2019-07-19 15:44:14 +02:00
Jan 1228731251
velocypack library upgrade (#9521) 2019-07-19 15:22:14 +02:00
Jan cdbe63fa6e
Bug fix/fix races in collection creation (#9506) 2019-07-19 15:11:08 +02:00
Michael Hackstein 36b1d290a9
Bug fix/failover with min replication factor (#9486)
* Improve collection time of IResearchQueryOptimizationTest

* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it

* Added some assertion son minReplicationFactor

* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled

* added minReplicationFactor to the user interface, preparation for the collection api changes

* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo

* added minReplicationFactor usage to tests

* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME

* minReplicationFactor now able to change via collection  properties route

* fixed wrongly assert

* added minReplicationFactor to the graph management ui

* added minReplicationFactor to the gharial api

* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.

* adjusted description of minReplicationFactor

* FollowerInfo Refactoring

* added gharial api graph creation tests with minimal replication factor

* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests

* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor

* Debug logging

* MORE Debug logging

* Included replication fast lane

* Use correct minreplicationfactor

* modified debug logging

* Fixed compileissues

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* Revert "MORE Debug logging"

This reverts commit dab5af28c0.

* Revert "MORE Debug logging"

This reverts commit 6134b664bd.

* Revert "MORE Debug logging"

This reverts commit 80160bdf3b.

* Revert "MORE Debug logging"

This reverts commit 06aabcdfe1.

* Removed debug output

* Added replication fast lane. Also refactored the commands as i cannot take it any more...

* Put some requests of RocksDBReplication onto CATCHUP Lane.

* Put some requests of MMFilesReplication onto CATCHUP Lane.

* Adjusted Fast and MED lane usage in Supervised scheduler

* Added changelog entry

* Added new features entry

* A new leader will now keep old followers in case of failover

* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Fixed JSLINT

* Unified lane handling of replication handlers

* Sorry forgotten in last commit

* replaced strings with static strings

* more use of static strings

* optimized min repl description in the ui

* decr initial loop variable

* clean up of the createWithId test

* more use of static strings

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Added some comments on condition, renamed variable as suggested in review

* Added check for min replicationFactor to be non-zero

* Added assertion

* Added function to modify min and max replication factor in one go

* added missing semicolon

* rm log devel

* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place

* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers

* check replFactor against nr dbservers

* Add lie reporting in CURRENT

* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out

* move replication checks from logical collection to rest collection handler

* added more replication tests

* Include assert only if we are not in gtest

* jslint

* set min repl factor to zero if satellite collection

* check replication attributes in v8 collection

* Initial commit, old plan, does not yet work

* fixed ires tests

* Included FailoverCandidates key. Not fully implemented

* fixed wrong assert

* unified in sync follower reporting

* fixed compiler errors

* Cleanup locking, and fixed potential deadlocks

* Comments about locking order in FollowerInfo.

* properly check uint

* Keep old leader as potential failover candidate

* Transaction methods now use followerInfo to check if the leader can write, this might have the sideeffect that 'failoverCandidates' are updated

* Let agency check failoverCandidates if possible

* Initialize member variables

* Use unified follower reporting in DBServerAgencySync

* Removed obsolete variable, collecting it somewhere else

* repl factor attr check

* Reimplemented previous followers, second attempt now. PhaseOne and PhaseTwo can now synchronize on current.

* Fixed assertion, forgot an off-by-one

* adjusted test to be more preciese now

* Fixed failove candidates list

* Disable write on dropping too many followers

* Allow to run updateFailoerCandidates multiple times with same leader.

* Final fixes, resilience tests now green, crossing fingers for jenkins

* Fixed race on atomics comparison

* Fixed invalid number type

* added nullptr handling

* added nullptr handling

* Removed invalid assert

* Make takeover of leadership an atomic operation

* Update tests/js/common/shell/shell-cluster-collection.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review fixes

* Fixed creation code to use takeoverLeadership

* Update arangod/Cluster/FollowerInfo.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Applied review fixes

* There is no timeout

* Moved AQL + Pregel to INTERNAL_AQL lane, which is medium priority, to avoid deadlocks with Sync replication

* More review fixes

* Use difference if you want to compare two vectors...

* Use std::string ...

* Now check if we are in recovery mode

* Added documentation for minReplicationFactor

* Added readme update as well in documenation
2019-07-19 15:00:30 +02:00
Wilfried Goesgens c922c5f133 move deleting of ClusterComm threads to the unprepare of the ClusterFeature (#9369) 2019-07-19 13:53:00 +02:00
Wilfried Goesgens ca0f2b8b86 All hail to the SI (#9445) 2019-07-19 13:52:12 +02:00
Michael Hackstein cbcf561450
Feature/min replication factor (#9433)
* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it

* Added some assertion son minReplicationFactor

* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled

* added minReplicationFactor to the user interface, preparation for the collection api changes

* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo

* added minReplicationFactor usage to tests

* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME

* minReplicationFactor now able to change via collection  properties route

* fixed wrongly assert

* added minReplicationFactor to the graph management ui

* added minReplicationFactor to the gharial api

* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.

* adjusted description of minReplicationFactor

* FollowerInfo Refactoring

* added gharial api graph creation tests with minimal replication factor

* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests

* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor

* Debug logging

* MORE Debug logging

* Included replication fast lane

* Use correct minreplicationfactor

* modified debug logging

* Fixed compileissues

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* Revert "MORE Debug logging"

This reverts commit dab5af28c0.

* Revert "MORE Debug logging"

This reverts commit 6134b664bd.

* Revert "MORE Debug logging"

This reverts commit 80160bdf3b.

* Revert "MORE Debug logging"

This reverts commit 06aabcdfe1.

* Removed debug output

* Added replication fast lane. Also refactored the commands as i cannot take it any more...

* Put some requests of RocksDBReplication onto CATCHUP Lane.

* Put some requests of MMFilesReplication onto CATCHUP Lane.

* Adjusted Fast and MED lane usage in Supervised scheduler

* Added changelog entry

* Added new features entry

* A new leader will now keep old followers in case of failover

* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Fixed JSLINT

* Unified lane handling of replication handlers

* Sorry forgotten in last commit

* replaced strings with static strings

* more use of static strings

* optimized min repl description in the ui

* decr initial loop variable

* clean up of the createWithId test

* more use of static strings

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Added some comments on condition, renamed variable as suggested in review

* Added check for min replicationFactor to be non-zero

* Added assertion

* Added function to modify min and max replication factor in one go

* added missing semicolon

* rm log devel

* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place

* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers

* check replFactor against nr dbservers

* Add lie reporting in CURRENT

* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out

* move replication checks from logical collection to rest collection handler

* added more replication tests

* Include assert only if we are not in gtest

* jslint

* set min repl factor to zero if satellite collection

* check replication attributes in v8 collection

* fixed ires tests

* fixed wrong assert

* properly check uint

* repl factor attr check

* adjusted test to be more preciese now

* Fixed race on atomics comparison

* Fixed invalid number type

* Update tests/js/common/shell/shell-cluster-collection.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review fixes

* More review fixes
2019-07-19 13:02:28 +02:00
Simon d79ecd3870 Bug fix/comm task vst10 (#9516) 2019-07-19 11:00:49 +02:00
Andrey Abramov 5bc6eac636
bug-fix/internal-issue-#609 (#9464)
* allow to access last operation tick of the transaction

* modify IResearchLink to rely on last commtted tick

* get rid of writing arangosearch markers to WAL

* further implementation

* local changes in iresearch

* add recovery states

* properly handle link creation during recovery

* adjust test cases

* properly handle nested transactions

* ungreylist recovery tests

* add more recovery tests

* do not use transaction to pass recovery tick

* do not store recoveryTick in MMFilesRecoveryState

* adjust tests

* fix mmfiles

* cleanup

* add context validity check

* fix tests

* fix more tests

* ensure subscription is not being released during commit

* address review comment

* final cleanup

* fix crash

* fix tests

* address test failures

* address review comments

* address review comments

* properly set recovery tick even if no recovery happened
2019-07-19 11:10:56 +03:00
Tobias Gödderz 001550ada7 Bug fix/count collect should skip (#9511)
* Make CountCollectExecutor skip rather than fetch

* Cleanup, fix and test for top-level variables

* Added CHANGELOG entry and tests
2019-07-19 10:08:36 +02:00
Jan Christoph Uhde a0332dbc7d Do not create files outside of build_dir (#9063) 2019-07-19 09:45:40 +02:00
Wilfried Goesgens f713d18a1c remove unneeded information (#9514) 2019-07-19 09:29:44 +02:00
Wilfried Goesgens 2da0fb356d Feature/cleanup j client api (#9457) 2019-07-19 09:23:24 +02:00
Jan 300b8e58f4
Bug fix/fix duplicate actions (#9452) 2019-07-19 09:17:19 +02:00
Jan 6d39a48435
Bug fix/fix uninitialized value (#9517) 2019-07-19 09:16:37 +02:00
Wilfried Goesgens 1c408dc2fb connect the mocha testresults if we run it in an extra arangosh instance (#9485)
* connect the mocha testresults if we run it in an extra arangosh instance

* fix status handling, lint
2019-07-18 16:41:11 +03:00
Wilfried Goesgens 9449f6a85a migrate mochaGrep into --testCase parametrizing (#9438)
* migrate mochaGrep into --testCase parametrizing

* fix filter forwarding for spawned arangoshs - add mocha version

* sometimes we have the string undefined to ignore
2019-07-18 16:35:44 +03:00
Simran e5855ef1fc
Feature/refactor maintainers readme (#9447) 2019-07-18 15:12:29 +02:00
Matthew Von-Maszewski d823470b4c
Bugfix: Correct Windows warning (#9502)
* correct handleResult() to report transmission errors and stop.  Add reporting of line numbers impacted if CSV

* always generate import summary, even if import stops due to error

* pass line numbers to senderThread for potential error reporting.

* move line count from zero based to one based in error report.

* correct warning from Windows build about size_t to int conversion
2019-07-18 15:01:58 +02:00
Wilfried Goesgens 5f3dae977a add greylist, so gtest can find it (#9510) 2019-07-18 15:52:56 +03:00
Wilfried Goesgens 9042fed887 jslint is your friend. (#9508) 2019-07-18 14:51:34 +03:00
Wilfried Goesgens 9ea028871d Docs: improve the formatting of the batch request documentation (#9489) 2019-07-17 19:26:48 +02:00
Wilfried Goesgens 56fff5c489 Documentation/switch example generation to rocksdb (#9478) 2019-07-17 19:17:41 +02:00
Matthew Von-Maszewski 40a5c4f8b8
Bugfix: Import would not report communication errors (#9494)
* correct handleResult() to report transmission errors and stop.  Add reporting of line numbers impacted if CSV

* always generate import summary, even if import stops due to error

* pass line numbers to senderThread for potential error reporting.

* move line count from zero based to one based in error report.

* forgot to update CHANGELOG
2019-07-17 18:29:07 +02:00
Simon 680971e81a Fix Arangosh windows instability (#9439) 2019-07-17 18:20:37 +02:00
Simon 040e41854f fix the audit client testsuite (#9496) 2019-07-17 18:16:08 +02:00
Jan 16405482c8
micro optimizations (#9487) 2019-07-17 14:10:18 +02:00
jsteemann 4029e9d050 fix typo 2019-07-16 10:49:07 +02:00
jsteemann ce50950e90 add auto-generated file 2019-07-16 09:52:29 +02:00
Simon e5507d840f Feature/comm task refactor (#9426) 2019-07-16 09:43:25 +02:00
Simran 85257fff27
Add updated JS deps to changelog (#8773) 2019-07-15 17:17:39 +02:00
Max Neunhoeffer bc0951c2f0
Add hot_backup to testing black list. 2019-07-15 11:50:34 +02:00
KVS85 ba18563398
Revert macOS package naming (#9469)
For alpha, beta, milestone, preview, rc release types (as in 3.4)
2019-07-12 19:13:26 +03:00
Matthew Von-Maszewski f26403c4e3 BugFix: Some error results have messages that are not reporting (#9454)
* some error results have messages that are not reporting

* update CHANGELOG for rocksdb reporting fix

* Add mutex protection to errMsg usage per Jans code review
2019-07-11 18:29:39 +03:00
Frank Celler 4cc7b6f18e bumped to 3.6 2019-07-11 15:52:42 +02:00
Andrey Abramov f2ff365206
bug-fix/fix-arangosearch-recovery-mmfiles (#9448)
* start maintenance arangosearch threads after recovery is done

* ensure flush subscription is not being deallocated while in use

* add some tests

* properly determine storage engine

* adjust default view options

* stick to old index meta format in 3.5

* address test failures

* and cluster tests
2019-07-11 11:42:08 +03:00
Wilfried Goesgens 623048aff8 fix 6561f21536 - only abort on timeout, not regular exit failures (#9434)
* fix 6561f21536 - only abort on timeout, not regular exit failures

* make pre-check exception distinguishable, its got no ready existing test status
2019-07-10 18:48:46 +03:00
Dronplane 468a334894 Enabled and fixed disabled tests in ArangoSearch (#9450) 2019-07-10 18:19:09 +03:00
Michael Hackstein 43858dcd58 Bug fix/replication fast lane (#9373)
* Let sync replication go through FAST lane instead of SLOW lane. This should reduce the amount of drop followers under high load.

* First draft of a generic MockServer to test RestHandlers. Needs adaption later on

* Added a  test case for the RestDocumentHandler lane decission, synchronous replications should be fast-lane.

* Added CHANGELOG entry

* Applied review fixes

* Update CHANGELOG

Thanks for finding!

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Test with absurd timeout

* Removed debug timeout
2019-07-10 17:30:31 +03:00
Wilfried Goesgens 0ad76a8185 fix location - while they work with the server these tests are run in arangosh so they need to be in client/ (#9435)
* fix location - while they work with the server these tests are run in arangosh so they need to be in client/

* rename testsuite

* rename file as its testsuite

* Rename permissions_server to server_permissions
2019-07-10 16:49:12 +03:00
KVS85 8897cd8f31 Fix wrong ars test failure suppression 2019-07-10 15:08:53 +02:00
Jan 25bc8c5343 remove bogus warnings about statistics thread (#9446) 2019-07-10 14:12:11 +03:00
Andrey Abramov b3690fc24f
bug-fix/internal-issue-#595 (#9440)
* address ASAN failures in tests

* Add --isAsan for testing.js

* Fix typo
2019-07-10 01:11:22 +03:00
KVS85 e4d76e087f
Temporarily suppress failure (#9443) 2019-07-10 01:10:56 +03:00
Dronplane 3330d9bb0f Bug fix/internal issue #524 (#9436)
* Added test for inBackground arangosearch link creation

* Updated flag for index check

* Reformatted  too long line

* Fixed comment

* Fixed test run on server-side

* fixed jslint issue

* Added task unregistering in teardown

* Made transaction sync with filesystem
2019-07-10 01:06:25 +03:00
KVS85 c05288fa59
Fix recovery tmp dir 2019-07-09 18:13:25 +03:00
Tobias Gödderz 789ef28a59 Bug fix/fix snippet responsibility for views (#9416)
* Fix query snippet responsibility for views

* Select one snippet per node id, instead of parent query id
2019-07-09 16:21:43 +03:00
jsteemann e049344e15 remove debug output 2019-07-09 11:29:02 +02:00
Jan c52f2a8315
refactoring (#9411) 2019-07-09 11:15:52 +02:00