1
0
Fork 0
Commit Graph

317 Commits

Author SHA1 Message Date
Dronplane 49dfb0a96e Bug fix/issue #9612 (#9764)
* Fixed ViewExecutionNode retrieval with deleted documents present in view

* Ported solution from 3.4 branch

* Changed index store in collection from vector to set. To make reversable indexes always last to execute

* Fixed re-enter hung

* Index storage fix

* Made index order deterministic

* Fix Mac build

* Added tests for index reversal

* Fixed Mac build

* Code cleanup

* Some cleanup

* Removed some redundand copy constructor calls

* Applied review comments

* Applied review comments
2019-08-20 19:34:23 +03:00
Dronplane 2f6fb109f6 Bug fix/issue #9652 (#9697)
* Fixed analyzer properties equality check.

* Added re-normalization of stored analyzer properties

* Fixed analyzer name rules in rest and v8 handlers

* Fixed linux build

* Added tests for implicit system db name. Code cleanup.

* Analyzers access logic moved to separate function
2019-08-15 17:42:44 +03:00
Kaveh Vahedipour e91cc66b9e forward ported node corrections from 3.4 (#9700)
* forward ported node corrections from 3.4
* tests fixed
2019-08-15 15:48:49 +02:00
Andrey Abramov facc9c6435
bug-fix/issue-#9660 (#9704)
* fix issue

* Update tests/js/common/aql/aql-view-arangosearch-cluster.inc

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update tests/js/common/aql/aql-view-arangosearch-noncluster.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* fix cluster tests
2019-08-13 22:48:55 +03:00
Tobias Gödderz 9cd332b958 Feature/rebootid notice changes (#9523)
* Consolidated _servers and _serverAdvertisedEndpoints, added rebootId, prepared change notifications

* Cleanup

* Added a RebootId type

* Began implementing RebootTracker (still WIP)

* Moved RebootId operators into the class

* Removed RebootId operator<< again

* Added tests, added CallbackGuard, removed/commented old RebootTracker code

* Fix: do not try to call unset callbacks

* Split one test, added another

* Added more tests

* Renamed tests, added more tests

* Fixed missing variable declarations

* Let MockServer appear to be started

* Reorded test, fixed naming

* Implemented callMeOnChange()

* Re-implemented RebootTracker (not yet working)

* Resolved a TODO, updated a test, added comments

* Call old callbacks immediately

* Fixed tests

* Use EXPECT_* instead of ASSERT_*

* Suppress a log message

* Resolved TODOs

* Reverted changes on reading ServersRegistered

* Update RebootTracker

* Introduce `rebootId` into ServerState for Cluster

 * A server *boots* if it is started on a previously non-existing data
   directory and hence does not have a UUID yet.
 * A server *reboots* if it is started on a pre-existing data directory

We keep the rebootId in the cluster's agency under
Current/ServersKnown/$uuid/rebootId.

When rebooting (and subsequently re-joining a cluster), the server increments
its rebootId in Phase 2 of registration. This way it can be detected within the
cluster whether a server was restarted.

This information will later be used to handle cases where server restarts can
lead to problems, for example with transactions or in-progress queries.

* Move rebootId into Current/ServersKnown/

* Fixed typo

* Fixed log ids

* Add deletion of ServersKnown/UUID from agency

* Add deletion of Current/ServersKnown/UUID to removeServer

* Clean up readRebootIdFromAgency and add retry loop around it

* Bugfix

* Added nolint comments

* Fixed initialization order

* Fixed ClusterInfo-test

* Added log messages

* Revert "Fixed ClusterInfo-test"

This reverts commit d983596979.

* Disabled assertion for google tests

* Ignore windows compile warning

* Always call loadServers in loadCurrent

* Fix really subtle bug when not returning a value

* Introduce `rebootId` into ServerState for Cluster

 * A server *boots* if it is started on a previously non-existing data
   directory and hence does not have a UUID yet.
 * A server *reboots* if it is started on a pre-existing data directory

We keep the rebootId in the cluster's agency under
Current/ServersKnown/$uuid/rebootId.

When rebooting (and subsequently re-joining a cluster), the server increments
its rebootId in Phase 2 of registration. This way it can be detected within the
cluster whether a server was restarted.

This information will later be used to handle cases where server restarts can
lead to problems, for example with transactions or in-progress queries.

* Move rebootId into Current/ServersKnown/

* Add deletion of ServersKnown/UUID from agency

* Add deletion of Current/ServersKnown/UUID to removeServer

* Clean up readRebootIdFromAgency and add retry loop around it

* Fixed compile error due to forbidden implicit cast

* Fixed compile error on windows

* Fixed compile error due to devel merge

* Removed dead comment

* Removed TODO note

* Extended comment

* Removed TODO note

* Fixed using an invalidated iterator

* Copy string only if necessary

* Fixed compile error
2019-08-12 09:33:22 +02:00
Dan Larkin-York 4347f1abec Add missing includes for Windows builds. (#9657) 2019-08-08 13:03:37 +02:00
Dan Larkin-York 3d0246cb18 Decentralize includes (#9623) 2019-08-06 15:32:09 +02:00
Jan 7c98893c73
reduce wait timeouts as a mitigation for notifying waiters without holding the required mutex (#9618) 2019-08-02 15:44:38 +02:00
Dronplane 25caffe709 Bug fix/internal issue #616 (#9607)
* Added tests for analyzer removal and get with wrong db context

* Added Link ddl tests

* Added cross-base access to analyzers tests

* Fixed v8 analyzer remove/get operation. Removed redundant sorting.

* Fixed link creation with analyzer from other database

* Added check for cross-use analyzer from system db

* Fixed tests

* Fix typo in test

* Fixed comments

* Fixed indentation

* Link validation moved to LinkHelper

* Code cleanup

* Applied review comments

* Applied review comments

* Small reformatting

* added assert
2019-08-01 17:39:04 +03:00
Michael Hackstein 987ad41364
Forward Port of changes in 3.5 review (#9544)
* Bug fix 3.5/min replication factor (#9524)

* Cherry-pick minReplicationFactor

* Bug fix/failover with min replication factor (#9486)

* Improve collection time of IResearchQueryOptimizationTest

* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it

* Added some assertion son minReplicationFactor

* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled

* added minReplicationFactor to the user interface, preparation for the collection api changes

* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo

* added minReplicationFactor usage to tests

* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME

* minReplicationFactor now able to change via collection  properties route

* fixed wrongly assert

* added minReplicationFactor to the graph management ui

* added minReplicationFactor to the gharial api

* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.

* adjusted description of minReplicationFactor

* FollowerInfo Refactoring

* added gharial api graph creation tests with minimal replication factor

* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests

* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor

* Debug logging

* MORE Debug logging

* Included replication fast lane

* Use correct minreplicationfactor

* modified debug logging

* Fixed compileissues

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* Revert "MORE Debug logging"

This reverts commit dab5af28c0.

* Revert "MORE Debug logging"

This reverts commit 6134b664bd.

* Revert "MORE Debug logging"

This reverts commit 80160bdf3b.

* Revert "MORE Debug logging"

This reverts commit 06aabcdfe1.

* Removed debug output

* Added replication fast lane. Also refactored the commands as i cannot take it any more...

* Put some requests of RocksDBReplication onto CATCHUP Lane.

* Put some requests of MMFilesReplication onto CATCHUP Lane.

* Adjusted Fast and MED lane usage in Supervised scheduler

* Added changelog entry

* Added new features entry

* A new leader will now keep old followers in case of failover

* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Fixed JSLINT

* Unified lane handling of replication handlers

* Sorry forgotten in last commit

* replaced strings with static strings

* more use of static strings

* optimized min repl description in the ui

* decr initial loop variable

* clean up of the createWithId test

* more use of static strings

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Added some comments on condition, renamed variable as suggested in review

* Added check for min replicationFactor to be non-zero

* Added assertion

* Added function to modify min and max replication factor in one go

* added missing semicolon

* rm log devel

* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place

* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers

* check replFactor against nr dbservers

* Add lie reporting in CURRENT

* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out

* move replication checks from logical collection to rest collection handler

* added more replication tests

* Include assert only if we are not in gtest

* jslint

* set min repl factor to zero if satellite collection

* check replication attributes in v8 collection

* Initial commit, old plan, does not yet work

* fixed ires tests

* Included FailoverCandidates key. Not fully implemented

* fixed wrong assert

* unified in sync follower reporting

* fixed compiler errors

* Cleanup locking, and fixed potential deadlocks

* Comments about locking order in FollowerInfo.

* properly check uint

* Keep old leader as potential failover candidate

* Transaction methods now use followerInfo to check if the leader can write, this might have the sideeffect that 'failoverCandidates' are updated

* Let agency check failoverCandidates if possible

* Initialize member variables

* Use unified follower reporting in DBServerAgencySync

* Removed obsolete variable, collecting it somewhere else

* repl factor attr check

* Reimplemented previous followers, second attempt now. PhaseOne and PhaseTwo can now synchronize on current.

* Fixed assertion, forgot an off-by-one

* adjusted test to be more preciese now

* Fixed failove candidates list

* Disable write on dropping too many followers

* Allow to run updateFailoerCandidates multiple times with same leader.

* Final fixes, resilience tests now green, crossing fingers for jenkins

* Fixed race on atomics comparison

* Fixed invalid number type

* added nullptr handling

* added nullptr handling

* Removed invalid assert

* Make takeover of leadership an atomic operation

* Update tests/js/common/shell/shell-cluster-collection.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review fixes

* Fixed creation code to use takeoverLeadership

* Update arangod/Cluster/FollowerInfo.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Applied review fixes

* There is no timeout

* Moved AQL + Pregel to INTERNAL_AQL lane, which is medium priority, to avoid deadlocks with Sync replication

* More review fixes

* Use difference if you want to compare two vectors...

* Use std::string ...

* Now check if we are in recovery mode

* Added documentation for minReplicationFactor

* Added readme update as well in documenation

* Removed merge conflict leftovers 0o, i should not trust the IDE

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/Books/Manual/Architecture/Replication/README.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update CHANGELOG

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/Books/Manual/DataModeling/Collections/DatabaseMethods.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/Books/Manual/ReleaseNotes/NewFeatures35.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/DocuBlocks/Rest/Collections/1_structs.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/graphManagementView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/graphManagementView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/DocuBlocks/Rest/Graph/1_structs.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Apply suggestions from code review

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Adepted review requests, thanks for finding!

* Removed unnecessary const

* Apply suggestions from code review

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Moved initilization of variable more downwards

* Apply lock before notify_all()

* Remove documentation except DocuBlocks, covered by PR in docs repo

* Remove accidental indent

* Removed leftover merge conflict in documentation block
2019-07-23 13:14:38 +02:00
Andrey Abramov 2f3c12eccb
fix asan issue (#9540) 2019-07-22 19:19:31 +03:00
Dronplane 11ef4bf18c Bug fix/internal issue #566 (#9497)
* Implemented TOKENS with array as parameter

* Binary terms encoded as Base64

* Fixed tests for base64 encoded values

* Code cleanup

* Fixed error codes

* Fixed Mac build issue

* Changed error code in case of type error. Code cleanup.

* Reverted MSVC formatting

* added changelog message

* Applied review comments

* Applied review comments

* numeric terms moved to raw pointer.

* Fix *nix build issue
2019-07-22 14:23:35 +03:00
Michael Hackstein 36b1d290a9
Bug fix/failover with min replication factor (#9486)
* Improve collection time of IResearchQueryOptimizationTest

* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it

* Added some assertion son minReplicationFactor

* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled

* added minReplicationFactor to the user interface, preparation for the collection api changes

* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo

* added minReplicationFactor usage to tests

* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME

* minReplicationFactor now able to change via collection  properties route

* fixed wrongly assert

* added minReplicationFactor to the graph management ui

* added minReplicationFactor to the gharial api

* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.

* adjusted description of minReplicationFactor

* FollowerInfo Refactoring

* added gharial api graph creation tests with minimal replication factor

* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests

* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor

* Debug logging

* MORE Debug logging

* Included replication fast lane

* Use correct minreplicationfactor

* modified debug logging

* Fixed compileissues

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* Revert "MORE Debug logging"

This reverts commit dab5af28c0.

* Revert "MORE Debug logging"

This reverts commit 6134b664bd.

* Revert "MORE Debug logging"

This reverts commit 80160bdf3b.

* Revert "MORE Debug logging"

This reverts commit 06aabcdfe1.

* Removed debug output

* Added replication fast lane. Also refactored the commands as i cannot take it any more...

* Put some requests of RocksDBReplication onto CATCHUP Lane.

* Put some requests of MMFilesReplication onto CATCHUP Lane.

* Adjusted Fast and MED lane usage in Supervised scheduler

* Added changelog entry

* Added new features entry

* A new leader will now keep old followers in case of failover

* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Fixed JSLINT

* Unified lane handling of replication handlers

* Sorry forgotten in last commit

* replaced strings with static strings

* more use of static strings

* optimized min repl description in the ui

* decr initial loop variable

* clean up of the createWithId test

* more use of static strings

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Added some comments on condition, renamed variable as suggested in review

* Added check for min replicationFactor to be non-zero

* Added assertion

* Added function to modify min and max replication factor in one go

* added missing semicolon

* rm log devel

* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place

* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers

* check replFactor against nr dbservers

* Add lie reporting in CURRENT

* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out

* move replication checks from logical collection to rest collection handler

* added more replication tests

* Include assert only if we are not in gtest

* jslint

* set min repl factor to zero if satellite collection

* check replication attributes in v8 collection

* Initial commit, old plan, does not yet work

* fixed ires tests

* Included FailoverCandidates key. Not fully implemented

* fixed wrong assert

* unified in sync follower reporting

* fixed compiler errors

* Cleanup locking, and fixed potential deadlocks

* Comments about locking order in FollowerInfo.

* properly check uint

* Keep old leader as potential failover candidate

* Transaction methods now use followerInfo to check if the leader can write, this might have the sideeffect that 'failoverCandidates' are updated

* Let agency check failoverCandidates if possible

* Initialize member variables

* Use unified follower reporting in DBServerAgencySync

* Removed obsolete variable, collecting it somewhere else

* repl factor attr check

* Reimplemented previous followers, second attempt now. PhaseOne and PhaseTwo can now synchronize on current.

* Fixed assertion, forgot an off-by-one

* adjusted test to be more preciese now

* Fixed failove candidates list

* Disable write on dropping too many followers

* Allow to run updateFailoerCandidates multiple times with same leader.

* Final fixes, resilience tests now green, crossing fingers for jenkins

* Fixed race on atomics comparison

* Fixed invalid number type

* added nullptr handling

* added nullptr handling

* Removed invalid assert

* Make takeover of leadership an atomic operation

* Update tests/js/common/shell/shell-cluster-collection.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review fixes

* Fixed creation code to use takeoverLeadership

* Update arangod/Cluster/FollowerInfo.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Applied review fixes

* There is no timeout

* Moved AQL + Pregel to INTERNAL_AQL lane, which is medium priority, to avoid deadlocks with Sync replication

* More review fixes

* Use difference if you want to compare two vectors...

* Use std::string ...

* Now check if we are in recovery mode

* Added documentation for minReplicationFactor

* Added readme update as well in documenation
2019-07-19 15:00:30 +02:00
Michael Hackstein cbcf561450
Feature/min replication factor (#9433)
* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it

* Added some assertion son minReplicationFactor

* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled

* added minReplicationFactor to the user interface, preparation for the collection api changes

* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo

* added minReplicationFactor usage to tests

* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME

* minReplicationFactor now able to change via collection  properties route

* fixed wrongly assert

* added minReplicationFactor to the graph management ui

* added minReplicationFactor to the gharial api

* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.

* adjusted description of minReplicationFactor

* FollowerInfo Refactoring

* added gharial api graph creation tests with minimal replication factor

* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests

* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor

* Debug logging

* MORE Debug logging

* Included replication fast lane

* Use correct minreplicationfactor

* modified debug logging

* Fixed compileissues

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* Revert "MORE Debug logging"

This reverts commit dab5af28c0.

* Revert "MORE Debug logging"

This reverts commit 6134b664bd.

* Revert "MORE Debug logging"

This reverts commit 80160bdf3b.

* Revert "MORE Debug logging"

This reverts commit 06aabcdfe1.

* Removed debug output

* Added replication fast lane. Also refactored the commands as i cannot take it any more...

* Put some requests of RocksDBReplication onto CATCHUP Lane.

* Put some requests of MMFilesReplication onto CATCHUP Lane.

* Adjusted Fast and MED lane usage in Supervised scheduler

* Added changelog entry

* Added new features entry

* A new leader will now keep old followers in case of failover

* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Fixed JSLINT

* Unified lane handling of replication handlers

* Sorry forgotten in last commit

* replaced strings with static strings

* more use of static strings

* optimized min repl description in the ui

* decr initial loop variable

* clean up of the createWithId test

* more use of static strings

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Added some comments on condition, renamed variable as suggested in review

* Added check for min replicationFactor to be non-zero

* Added assertion

* Added function to modify min and max replication factor in one go

* added missing semicolon

* rm log devel

* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place

* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers

* check replFactor against nr dbservers

* Add lie reporting in CURRENT

* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out

* move replication checks from logical collection to rest collection handler

* added more replication tests

* Include assert only if we are not in gtest

* jslint

* set min repl factor to zero if satellite collection

* check replication attributes in v8 collection

* fixed ires tests

* fixed wrong assert

* properly check uint

* repl factor attr check

* adjusted test to be more preciese now

* Fixed race on atomics comparison

* Fixed invalid number type

* Update tests/js/common/shell/shell-cluster-collection.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review fixes

* More review fixes
2019-07-19 13:02:28 +02:00
Andrey Abramov 5bc6eac636
bug-fix/internal-issue-#609 (#9464)
* allow to access last operation tick of the transaction

* modify IResearchLink to rely on last commtted tick

* get rid of writing arangosearch markers to WAL

* further implementation

* local changes in iresearch

* add recovery states

* properly handle link creation during recovery

* adjust test cases

* properly handle nested transactions

* ungreylist recovery tests

* add more recovery tests

* do not use transaction to pass recovery tick

* do not store recoveryTick in MMFilesRecoveryState

* adjust tests

* fix mmfiles

* cleanup

* add context validity check

* fix tests

* fix more tests

* ensure subscription is not being released during commit

* address review comment

* final cleanup

* fix crash

* fix tests

* address test failures

* address review comments

* address review comments

* properly set recovery tick even if no recovery happened
2019-07-19 11:10:56 +03:00
Simon e5507d840f Feature/comm task refactor (#9426) 2019-07-16 09:43:25 +02:00
Andrey Abramov f2ff365206
bug-fix/fix-arangosearch-recovery-mmfiles (#9448)
* start maintenance arangosearch threads after recovery is done

* ensure flush subscription is not being deallocated while in use

* add some tests

* properly determine storage engine

* adjust default view options

* stick to old index meta format in 3.5

* address test failures

* and cluster tests
2019-07-11 11:42:08 +03:00
Dronplane 468a334894 Enabled and fixed disabled tests in ArangoSearch (#9450) 2019-07-10 18:19:09 +03:00
Andrey Abramov b3690fc24f
bug-fix/internal-issue-#595 (#9440)
* address ASAN failures in tests

* Add --isAsan for testing.js

* Fix typo
2019-07-10 01:11:22 +03:00
Dronplane 3330d9bb0f Bug fix/internal issue #524 (#9436)
* Added test for inBackground arangosearch link creation

* Updated flag for index check

* Reformatted  too long line

* Fixed comment

* Fixed test run on server-side

* fixed jslint issue

* Added task unregistering in teardown

* Made transaction sync with filesystem
2019-07-10 01:06:25 +03:00
Andrey Abramov 420326be25
Bug fix/internal issue #414 (#9428)
* add more tests

* and more...

* address msvc build issue
2019-07-08 23:17:41 +03:00
Andrey Abramov a793a4749f
Bug fix/rename iresearch analyzer feature (#9424)
* rename feature

* ensure 'MMFilesLogFileManager::SafeToUseInstance' is safe to use in multithreaded context

* address mac compilation issue
2019-07-08 15:46:57 +03:00
Andrey Abramov 174b54b036
Bug fix/internal issue #605 (#9421)
* release tick before committing changes to arangosearch

* fix tests

* address review comments

* address review comments

* add comment
2019-07-08 14:07:11 +03:00
Dronplane 22d5df16e8 Bug fix/internal issue #606 (#9413)
* Fixed stem, norm, text analyzer creation without properties. Added tests.

* Removed non-determenistic test. Same functionality is tested by gtest (IResearchAnalyzerFeatureTest.test_remove)

* Added check for analyzer type existence.

* Made error messages for analyzer creation more specific and readable

* Fixed test

* Applied review suggestions
2019-07-08 13:11:22 +03:00
Andrey Abramov 595728ec1b
Bug fix/internal issue #605 (#9405)
* fix recovery

* update iresearch

* more fixes

* address review comments
2019-07-05 13:14:37 +03:00
Dronplane b70d7372ef Bug fix/internal issue #586 (#9401)
* Removet lazy creation ob analyzers collection. Test fixes (added explicit creation of analyzers collection for test enviroments)

* Fixed test runs. Removed cluster tests for analyzer DDL

* Analyzers collection name moved to tests common

* Disbaled load_library in normalize calls

* Legacy code cleanup

* added db analyzers collection checks in analyzer creation test to cover functions removed from gtest

* Reverted analyzer properties comparsion to non-utf8 as that must be binary equal
2019-07-04 13:12:06 +03:00
Jan 7ffb0684cd fix tests that didn't properly use env variable to look for test (#9399)
resource files
2019-07-04 11:30:48 +02:00
Jan 64c8079186
Forbid indexes on _id (#9337) 2019-06-28 11:11:22 +02:00
Andrey Abramov 40cce2cee0
bug-fix/internal-issue-#565.1 (#9257)
* validate analyzer properties, do not return `null` for identity analyzer properties

* fix some tests

* Fixed tests for new analyzer parameters validation. Added explicit test for analyzer parameter validation

* update iresearch, fix analyzer tests

* fix more tests

* fix tests

* store analyzer properties as vpack

* more compilation fixes

* Updated iresearch

* Tests compilation fixed

* Test run fixes

* Test run fixes

* Fix test run

* Fixed all IresearchTests

* Fixed V8Analyzers tests

* Added ngram and delimeter analyzers vpack config

* Added ngram and stem analyzers vpack parsers. Added tests

* Fixed internal issue #593 Fixed build issue

* Fixed tests

* Fixed Gtest tests

* Changed test run to fix Mac failure

* Mac tests debugging

* Fixed jslint errors

* test tracing added

* Taken account for VPackSlice operator== false negatives
2019-06-28 10:49:01 +02:00
Lars Maier eb1aa6e024 Response compression (#9300)
* First draft of response compression.

* Cleanup.

* Removed compression from /_api/version.

* Added ruby test for response compression.
2019-06-28 10:02:48 +02:00
Andrey Abramov 33ce94379f
bug-fix/internal-issue-#597 (#9324)
* ensure immutable properties of ArangoSearch are properly handled in cluster

* address jslint warnings
2019-06-25 17:15:07 +02:00
Simon cf7cf0131b Try to fix corruption error (#9258) 2019-06-25 10:18:26 +02:00
Wilfried Goesgens 200be8ec8b Bug fix/zero mac clang warning to errors (#9279) 2019-06-18 10:28:30 +02:00
Andrey Abramov 384c4310b1
Bug fix/internal issue #589 (#9244)
* update iresearch from upstream

* adjust tfidf/bm25 to original behaviour

* update iresearch from upstream

* fix compilation issues

* reduce number of virtual calls

* upgrade iresearch from upstream

* add js test
2019-06-11 17:42:30 +03:00
Dan Larkin-York 938e5fd36c Add drop-check for index creation in cluster (#9219)
* Add drop-check for index creation in cluster.

* Move check from callback to regular read.

* Add changelog entry.

* Incorporate review suggestion

Co-Authored-By: Simon <simon@graetzer.org>

* Convert to VPackArrayIterator.
2019-06-10 10:11:17 +02:00
Jan Christoph Uhde 3f603f024f remove some containers from common.h (#9223)
* remove some containers from Common.h

* enterprise fixes
2019-06-07 13:27:24 +02:00
Wilfried Goesgens 863257f1d5 fix windows warnings / compile errors (#9215) 2019-06-06 18:08:44 +02:00
Andrey Abramov 0fb8b348db
bug-fix/internal-issue-#574 (#8991)
* restrict number of buckets to compare

* do not use more sort buckets than requested

* cleanup
2019-06-04 18:55:25 +03:00
Dronplane 1e24c79ca3 Bug fix/internal issue #565 (#9154)
* Merged latest iresearch. Removed creation of build-in analyzers from tests

* Updated tests for new analyzer parameter names

* Merged fix for format from iresearch

* Apply suggestions from code review. Corrected typo in variable name.

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* backport changes from IResearch upstream

* fix compilation issues

* backport fix from IResarch upstream

* Fixed analyzer parameter name in tests
2019-06-03 18:20:23 +03:00
Michael Hackstein d135d55d55
Bug fix/collection babies (#9124)
* Bug fix 3.4/collection babies (#9033)

* Prepare API to create multiple collections in a single request to ClusterMethods to improve speedup

* Added counter on how many collections are successfully created

* Allow multi collection creation one level higher

* CollectionMethods now allow batch createion of Collections

* Improved array size assertions

* Now a graph is createad within a single roundtrip in the agency.

* Added new header files

* Insert collections in the AGENCY with TTL and a isBuilding flag, collections with this flag should not be visisible in the coordinator

* Added forgotten C++ file

* Fixed a rare race condition, and the failing IResearch Tests

* readded callback on DONE, otherwise lists are out of sync

* Fixed assertions to let mocked tests pass...

* Fixed community cluster

* Started fixing IResearch analyzer test, catch-tests are failing ;(

* Solved missed merge-conflict

* Added helper functions in AnalyzerFeature-test

* Refactoring AnalyzerTest Section-Auth

* Refactoring AnalyzerTest Section-Emplace-Duplicates

* Refactoring AnalyzerTest Section-Emplace-Error-Cases. Recovery-Test is now red, it seemed to be green because of invalid test case before.

* Refactoring AnalyzerTest, split GET test into multiple parts, still left 'cluster simulation'.

* Attempt to extract Coordinator / DBServer tests a little bit. This commit starts to break all Coordinator tests. However i am convinced that earlier version did NOT test a cluster situation at all, but some hybrid of SingleServer with full local storage that got told to be a Coordinator from now on, but without any Coordinator setup...

* Temporarly disabled some tests in AnalyzerFeature, as discussed with @gnusi.

* Fixed include guard.

* Temporarily deactivated failing tests

* You shall save your files before you commit...

* Fixed test asserting on plan version, which is now higher than before
2019-06-03 17:11:22 +02:00
Jan 3364ea6273
refactor index APIs (#9145) 2019-05-31 11:12:39 +02:00
Andrey Abramov e8b38bfa8e
bug-fix/internal-issue-#549 (#9086)
* do not persist legacy analyzers into _analyzers table

* fix arangosearch upgrade in cluster

* get rid of Vasiliy's shit

* address review comments

* ensure link is synchronized after creation in upgrade

* fix compilation error

* minor cleanup

* fix tests

* distribute '_analyzers' collection as '_graphs'

* comment out Vasiliy's shit part 2
2019-05-30 22:00:06 +03:00
Dronplane 3d697d8584 Added filtering analyzers before adding them to view link - duplica… (#9094)
* Added filtering analyzers before adding them to view link   - duplicates are not allowed any more

* Updated test testAddDuplicateAnalyzers - now it tests analyzers with same names but in different databases
2019-05-27 13:20:51 +03:00
Andrey Abramov f5a14414e1 Bug fix/disable upgrade for views again 2 (#9084)
* temporary disable upgrade for views

* temporary comment out test for upgrade
2019-05-23 15:31:26 +02:00
Dan Larkin-York d5ecdd143a Convert unit tests to googletest framework (#9034) 2019-05-21 09:17:46 +02:00
Andrey Abramov 56696e3ea6 Bug fix/internal issue #577 (#9035)
* properly handle shorthand of system analyzer

* don't allow accessing local analyzer from within another database

* ensure local analyzer are not acessible from outside

* fix duplicate ids
2019-05-19 18:00:13 +02:00
Jan Christoph Uhde d194cfc31e bug-fix/iresearch-error-propagation (#9001)
* use result instead of bool

* WIP: use arangodb::Results in more places

* WIP: fix unittests

* fix tests
2019-05-17 19:00:29 +03:00
Jan Christoph Uhde 970f732dec add exceptions for tokens function. (#8984)
* Provide users of the TOKENS function with errors.

* remove newlines

* fix compare

* fix as asked for

* improve fix
2019-05-16 22:12:43 +03:00
Andrey Abramov b7cfa24518
bug-fix/arangosearch-upgrade (#9010)
* fix upgrade

* fix catch tests

* Update arangod/IResearch/IResearchAnalyzerFeature.cpp

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update arangod/IResearch/IResearchAnalyzerFeature.cpp

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
2019-05-16 10:38:39 +03:00
Andrey Abramov bdc2c7cdc4
fix asan error (#9005) 2019-05-15 17:16:30 +03:00