1
0
Fork 0
Commit Graph

205 Commits

Author SHA1 Message Date
Max Neunhöffer 014c3f7f53
Only load Plan and Current in ClusterInfo when actually needed. (#5649)
* Only update Plan and Current from Agency if not already done.
* Add read protection for getPlanVersion and getCurrentVersion.
* Add a further check to loadPlan and loadCurrent.
* Fix tests to new behaviour.
* Try to increase Plan/Version and Current/Version with every change.
* Add two more increments of Plan/Version
* Add missing increments in tests for Plan/Version.
* Add changelog entry.
2018-07-16 12:20:13 +02:00
Dan Larkin-York 21e16a8a24 Add load balancer awareness for cursor API (#5682) 2018-07-03 14:29:09 +02:00
Vasiliy 7aaeab50fb issue 402.1: share sync thread between IResearchView and IResearchViewDBServer (#5733) 2018-07-02 15:03:00 +03:00
Andrey Abramov 5eef6cd618
Feature/test iresearch (#5610)
* start implementing arangosearch cluster tests.

* backport: ensure view lookup is done via collectionNameResover, ensure updateProperties returns current view properties

* first attempt to fix failing tests

* refactor cluster wide view creation logic

* if view is not found in the new plan then check the old plan too

* ensure the cluster-wide view is looked up in vocbase as well on startup/recovery

* do not store cluster-wide IResearchView in vocbase

* move stale view cleanup to the shared pointer deleter, address test failures

* do not print warning

* enable arangosearch tests by default

* fix catch tests

* address icorrect return value for cluster-wide links

* address some issues with test failures due to cluster-view allocated within TRI_vocbase_t

* simplify per-cid view name, address 'catch' test failures

* ensure IResearchViewNode volatility is properly calculated in cluster

* invoke callbacks directly in AgencyMock instead of waiting for timeout

* ensure view updates via JavaScript always use the latest view definition

* pass a list of shards to `IResearchViewDBServer::snapshot`

* extend cluster aql tests

* fixes after merge

* fix class/struct inconsistencies

* comment failing tests

* remove debug logging

* add debug function

* tests cleanup

* simplify upcoming merge: pass resolver from a side

* backport: move all transaction status callback logic to Methods

* add changes missed from previous commit

* fix js and ruby tests

* more tests for IResearchViewNode

* pass transaction to IResearchViewDBServer::snapshot, address IResearchViewDBServer tests segfault

* pass transaction to IResearchView::snapshot instead of transaction state

* temporarily add trace log output to tests to try to find the cause of the core dump on Jenkins

* add more temporary debug output to trace down the segfault on Jenkins

* add even more temporary debug output to trace down the segfault on Jenkins

* ensure Vieew related maps are cleared during shutdown

* reset ClusterInfo::instance() before DatabaseFeature::unprepare()

* remove extraneous debug output

* missed line from previous commit

* uncomment required line

* add nullptr checks to RocksDBIndexFactory::prepareIndexes(...) similar to the ones in MMFilesIndexFactory::prepareIndexes(...)

* attempt to fix deadlock in tests

* add comment as per reviewer request

* fix aql test suite name

* add some debug logging

* address deadlock between ClusterInfo::loadPlan() and CollectionNameResolver::localNameLookup(...)

* eplicitly state which index definition failed in the log message

* use vocbase from shard-view isntead just in case

* explicitly state which index definition failed in the log message

* do not create shard-view instances from cluster-link instances (only register existing ones)

* add some tests
2018-06-21 20:35:04 +03:00
Vasiliy 4253dca6aa issue 381.5: ensure the LogicalView definition that is persisted to the Agency matches the definition that gets created (#5518)
* issue 381.5: ensure the LogicalView definition that is persisted to the Agency matches the definition that gets created

* backport: correct comment
2018-06-02 17:21:55 +03:00
Andrey Abramov 4649b40b96
Coordinator ArangoSearch view + Execution nodes + AgencyMock (#5160)
* add initial implementation of scatter view rule and node

* add tests for `IResearchViewNode` and `IResearchViewScatterNode`

* add missing check

* modify IResearch execution nodes to use references instead of pointers

* use view id in searialized `ExecutionNode` representation instead of the name

* add cluster mode stubs and checks

* very first attempt to distribute IResearchViewNode

* further implementation of cluster-wide arangosearch views

* fix invalid json format

* add tests for coordinator iresearch view

* allow to retrieve a list of existing views on a coordinator

* more tests for coordinator iresearch view

* some fixes to enable query explanation

* remove Collection dependency from RemoteNode

* remove unnecessary remote ArangoSearch view scatter

* fix explanation appearance

* add some assertions

* minor fixes

* implement IResearchViewCoordinator::updateProperties

* fix view DDL issues

* handle link modifications in DDL operations

* add coordinator implementation of iresearch view links

* fix tests

* further coordinator based view DDL implementation

* further IResearchViewCoordinator implementation

* add initial implementation of AgencyMock

* fix some tests

* code cleanup

* extend test + some fixes

* more tests for IResearchViewCoordinator

* fix tests for IResearchLinkCoordinator

* some fixes after merge

* fix tests

* remove declaration of nonexistent (previously removed) method

* some fixes after review

* remove string duplication

* more tests and fixes

* more fixes and tests

* more tests

* one more test

* fix 'use-after-free' asan error

* fix non-enterprise tests issues
2018-05-02 00:15:11 +03:00
Max Neunhoeffer ce8db24975
Add methods in ClusterInfo to create and drop views. 2018-03-14 23:22:44 +01:00
Max Neunhoeffer a8a307b532
Report views in ClusterInfo.
This is incomplete as it is, because we do not yet parse the views
we see in the plan.
2018-03-08 14:24:22 +01:00
Andrey Abramov a1cfb3d72b Feature iresearch (#4105) 2018-01-19 14:23:58 +01:00
Jan 25af4d7f69
try to not fail hard when a collection is dropped while the WAL is tailed (#4226) 2018-01-04 16:31:11 +01:00
Heiko 61de1b6099 Bug fix/optimize shard distribution api and ui (#3921)
* UI: document/edge editor now remembering their modes (e.g. code or tree)

* changed shardDistribution api behaviour, added PUT route to only fetch collection based shard distribution

* ui: optimized shards view, added missing cleanup function in nodes view

* broken test

* shard distribution tests not fit the new api behaviour

* variables as reference

* CHANGELOG
2018-01-02 12:42:12 +01:00
Jan 17986ebc08
return error context for "some agency operation failed" (#3760) 2017-12-06 11:16:19 +01:00
Michael Hackstein 5c633f9fae Bug fix/speedup shard distribution (#3645)
* Added a more sophisticated test for shardDistribution format

* Updated shard distribution test to use request instead of download

* Added a cxx reporter for the shard distribuation. WIP

* Added some virtual functions/classes for Mocking

* Added a unittest for the new CXX ShardDistribution Reporter.

* The ShardDsitributionReporter now reports Plan and Current correctly. However it does not dare to find a good total/current value and just returns a default. Hence these tests are still red

* Shard distribution now uses the cxx variant

* The ShardDistribution reporter now tries to execute count on the shards

* Updated changelog

* Added error case tests. If the servers time out the mechanism will stop bothering after two seconds and just report default values.
2017-11-10 15:17:08 +01:00
Simon Grätzer 7c31960cf2 Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
Jan 0561bf45ce Bug fix/isrestore (#3283)
* Make isRestore work in the cluster.

This covers sharded collections with default sharding and non-default
sharding.

* always use locally generate revision ids for storing and looking up documents
2017-10-03 11:53:49 +02:00
Jan 5165155ed1 Bug fix/fixes 0609 (#3227)
* do not use V8 variant of AQL functions in early optimization stage when a C++ variant is available

* additionally, simplify AQL function definitions and aliases

* warn when more than 90% of max mappings are in use

* added C++ variant of replication catchup

* added `--log.role` option

* updated CHANGELOG

* removed non-existing scheduler.threads option from config

* removed useless __FILE__, __LINE__ invocations

* updated CHANGELOG

* allow a priority V8 context

* remove TRI_CORE_MEM_ZONE

* try to fix Windows errors & warnings

* cleanup

* removed memory zones altogether

* exclude system collections from collection tests
2017-09-13 16:28:21 +02:00
Max Neunhöffer f3acea797b Feature/cluster inventory version (#3152)
* Get rid of a compiler warning in community edition.

* Teach /_api/replication/clusterInventory to report Plan/Version and readiness.

This is first implemented in the ClusterInfo library.
Then the clusterInventory code uses it and checks readiness.
Readiness of a collection means that it is created and all shards
and all replications have been created and are in sync.
2017-08-30 13:34:23 +02:00
Kaveh Vahedipour 1d1e0f5a50 Feature/cluster id and extended health (#3046)
* added unique id to cluster, added access to Health

* added agents to health api

* added agents to health api

* added agents to health api

* transaction information for api

* agents listed like other servers

* missing line through merge conflict
2017-08-18 11:13:23 +02:00
Jan 07abf73bd6 fix parallel access to list of failed servers (#2777) 2017-07-12 22:12:25 +02:00
Frank Celler 2807ef559c Feature/move shard fix (#2626)
Major overhaul of handling of synchronous replication.
2017-06-26 16:55:01 +02:00
Mop 619eae9be5 Revert "Squashed commit of the following:"
This reverts commit 2252088572.
2017-06-01 18:37:45 +02:00
Andreas Streichardt 2252088572 Squashed commit of the following:
commit f3d0fd6584b0e451b8c97abcb4ba8d9f2fc6f560
Author: Andreas Streichardt <andreas@arangodb.com>
Date:   Thu Jun 1 17:31:36 2017 +0200

    fix unittest

commit 7cd3544a39e1b78af9d4175cb3b978799b9bbfff
Author: Andreas Streichardt <andreas@arangodb.com>
Date:   Thu Jun 1 17:10:00 2017 +0200

    Remove debug comment

commit fb6b10dac15be49a72dbff80030a7d22abdfc3e0
Merge: 055eb1d269 6b18cc64fe
Author: Andreas Streichardt <andreas@arangodb.com>
Date:   Thu Jun 1 17:00:21 2017 +0200

    Merge branch 'devel' into shardorganizer

commit 055eb1d2693a583d21ea59ec8b6ba95ab0db57ac
Merge: 1ff7998ebf 8ea89b7677
Author: Mop <andreas@arangodb.com>
Date:   Thu Jun 1 16:56:30 2017 +0200

    Merge branch 'shardorganizer' of https://github.com/arangodb/arangodb into shardorganizer

commit 8ea89b76777c75b6a77bf695c3f074a0c4643c29
Author: Andreas Streichardt <andreas@arangodb.com>
Date:   Thu Jun 1 16:55:41 2017 +0200

    Fix shardmapping bug

commit 1ff7998ebfd691598ec5b455ca5bc2bfd7020fb4
Author: Mop <andreas@arangodb.com>
Date:   Wed May 31 17:26:08 2017 +0200

    more output

commit 68e88aa0e14316c4929d05b2c151bee6421d754d
Merge: 0978ad1d9e 44a6a78ec3
Author: Mop <andreas@arangodb.com>
Date:   Wed May 31 17:03:33 2017 +0200

    Merge branch 'shardorganizer' of https://github.com/arangodb/arangodb into shardorganizer

commit 44a6a78ec338a1e7cabb15464500d96b84c68f1d
Author: Andreas Streichardt <andreas@arangodb.com>
Date:   Wed May 31 07:42:43 2017 -0700

    Fix namespace

commit 0978ad1d9e2f01b86204990e74b66958f25eba66
Merge: f98582ccff d74e5989ad
Author: Mop <andreas@arangodb.com>
Date:   Wed May 31 16:40:35 2017 +0200

    Merge branch 'shardorganizer' of https://github.com/arangodb/arangodb into shardorganizer

commit f98582ccff3448f6c2388dab4cc2dc38034271b0
Author: Mop <andreas@arangodb.com>
Date:   Wed May 31 16:39:03 2017 +0200

    Revert "Revert "Next attempt at merging ShardOrganizer...distributeShardsLike fixed""

    This reverts commit fed45b7b10.

commit d74e5989ad478efe7d66d196715c05f4f41c9c29
Author: Andreas Streichardt <andreas@arangodb.com>
Date:   Wed May 31 16:31:31 2017 +0200

    Make it an error

commit 0a6a9ef9464df4f24ad205bbab5b9f8ded50054f
Author: Andreas Streichardt <andreas@arangodb.com>
Date:   Wed May 31 12:42:51 2017 +0200

    distributeShardsLike has to be saved as a cidString
2017-06-01 17:32:40 +02:00
Andreas Streichardt 50876d0d3f Do not look at replicationFactor while creating...just look at the plan! it is great! 2017-05-24 16:25:49 +02:00
Andreas Streichardt b5fcd15214 Fix linter 2017-05-24 14:53:06 +02:00
Andreas Streichardt 9472ab821c Fix rolling back of indices 2017-05-15 15:48:01 +02:00
jsteemann 68b4b2f393 fix shutdown order 2017-05-11 20:59:36 +02:00
Andreas Streichardt 2e4f83fc08 Invalidate current coordinators on every 2nd heartbeatrun
needed for foxx resilience stuff
2017-05-11 18:35:33 +02:00
Andreas Streichardt dad5a1429e Add waitForSyncReplication as a _create() option 2017-04-26 09:57:40 +02:00
Max Neunhoeffer dc3c380904 Fix bug found by static analysis. 2017-01-24 12:30:32 +01:00
Max Neunhoeffer f35e3a7aaf Merge branch 'devel' into schmutz-ng 2017-01-16 09:54:09 +01:00
Andreas Streichardt 191f399ce2 Move AgencyCallback stuff to cluster so it is (hopefully) clear that this
is being used within the cluster and not within the agency.
2017-01-13 18:08:27 +01:00
Kaveh Vahedipour fe48bcb982 fixed for short names in frontend shard view 2017-01-11 16:31:19 +01:00
Kaveh Vahedipour 2b9c018817 fixed resilience 2016-12-09 16:35:32 +01:00
Andreas Streichardt 82682f8d25 Wait for synchronous replication to settle 2016-12-07 18:38:15 +01:00
Andreas Streichardt 11bd9381d5 Add satellite collections 2016-12-06 16:40:50 +01:00
Frank Celler 959797c54f moved to Agency 2016-10-23 00:46:30 +02:00
Andreas Streichardt e75b8f5503 Implement cURL based inter server communication 2016-10-10 14:56:04 +02:00
Max Neunhoeffer 82e548ff8b Change setup of getResponsibleShard w.r.t. collection. 2016-10-07 21:05:18 +02:00
Max Neunhoeffer 49d7cf345f Improve version handling in ClusterInfo. 2016-09-05 16:03:28 +02:00
Michael Hackstein e444332def Removed all references to _info in TRI_collection_t. This commit does not start. Constructor is broken. 2016-08-29 10:44:31 +02:00
Michael Hackstein 5f807900d7 Implemented updated of LogicalCollectionInfo. Right now this is only used in Cluster Mode 2016-08-19 11:14:23 +02:00
Michael Hackstein bb91925903 Readded temporary logical collections in cluster. Otherwise it is entirely unclear when which collection has to be deleted. 2016-08-18 11:38:16 +02:00
Michael Hackstein 17a6da8afb Removed all references on TRI_vocbase_col_t and replaced them by LogicalCollection class. Not everything is properly moved and implemetned yet. This commit causes crashes. 2016-08-17 11:10:39 +02:00
Michael Hackstein 94ca36fdb2 Removed cluster-wide CollectionInfo and replaced it with a logical collection. The Logical collection shall be always available, in cluster and single server. 2016-08-15 09:08:24 +02:00
jsteemann 59ed0a902c refactoring 2016-07-29 17:07:16 +02:00
jsteemann 31407c3561 refactoring 2016-07-27 17:15:52 +02:00
Max Neunhoeffer 4a4472e31e Repair DBServerAgencySync in background.
Also: Notice role change follower->leader.
2016-05-25 17:27:27 +02:00
Max Neunhoeffer 79a3621430 Fix sync repl test. 2016-05-23 16:57:07 +02:00
Andreas Streichardt 4d13e8fb1b _listDatabases() => _databases() 2016-05-11 15:30:47 +02:00
jsteemann 213ade22a4 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2016-05-10 19:01:30 +02:00