1
0
Fork 0
Commit Graph

461 Commits

Author SHA1 Message Date
Andrey Abramov 5eef6cd618
Feature/test iresearch (#5610)
* start implementing arangosearch cluster tests.

* backport: ensure view lookup is done via collectionNameResover, ensure updateProperties returns current view properties

* first attempt to fix failing tests

* refactor cluster wide view creation logic

* if view is not found in the new plan then check the old plan too

* ensure the cluster-wide view is looked up in vocbase as well on startup/recovery

* do not store cluster-wide IResearchView in vocbase

* move stale view cleanup to the shared pointer deleter, address test failures

* do not print warning

* enable arangosearch tests by default

* fix catch tests

* address icorrect return value for cluster-wide links

* address some issues with test failures due to cluster-view allocated within TRI_vocbase_t

* simplify per-cid view name, address 'catch' test failures

* ensure IResearchViewNode volatility is properly calculated in cluster

* invoke callbacks directly in AgencyMock instead of waiting for timeout

* ensure view updates via JavaScript always use the latest view definition

* pass a list of shards to `IResearchViewDBServer::snapshot`

* extend cluster aql tests

* fixes after merge

* fix class/struct inconsistencies

* comment failing tests

* remove debug logging

* add debug function

* tests cleanup

* simplify upcoming merge: pass resolver from a side

* backport: move all transaction status callback logic to Methods

* add changes missed from previous commit

* fix js and ruby tests

* more tests for IResearchViewNode

* pass transaction to IResearchViewDBServer::snapshot, address IResearchViewDBServer tests segfault

* pass transaction to IResearchView::snapshot instead of transaction state

* temporarily add trace log output to tests to try to find the cause of the core dump on Jenkins

* add more temporary debug output to trace down the segfault on Jenkins

* add even more temporary debug output to trace down the segfault on Jenkins

* ensure Vieew related maps are cleared during shutdown

* reset ClusterInfo::instance() before DatabaseFeature::unprepare()

* remove extraneous debug output

* missed line from previous commit

* uncomment required line

* add nullptr checks to RocksDBIndexFactory::prepareIndexes(...) similar to the ones in MMFilesIndexFactory::prepareIndexes(...)

* attempt to fix deadlock in tests

* add comment as per reviewer request

* fix aql test suite name

* add some debug logging

* address deadlock between ClusterInfo::loadPlan() and CollectionNameResolver::localNameLookup(...)

* eplicitly state which index definition failed in the log message

* use vocbase from shard-view isntead just in case

* explicitly state which index definition failed in the log message

* do not create shard-view instances from cluster-link instances (only register existing ones)

* add some tests
2018-06-21 20:35:04 +03:00
Vasiliy a5f274234a issue 380.2.2: remove extraneous pointers from StorageEngine API (#5589) 2018-06-13 15:22:51 +03:00
Heiko 743d4430bf added replication ui (#5555) 2018-06-08 11:19:05 +02:00
Vasiliy 4253dca6aa issue 381.5: ensure the LogicalView definition that is persisted to the Agency matches the definition that gets created (#5518)
* issue 381.5: ensure the LogicalView definition that is persisted to the Agency matches the definition that gets created

* backport: correct comment
2018-06-02 17:21:55 +03:00
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00
Jan 0d98c0f3ba
Bug fix/fix codescan issues (#5371) 2018-05-16 20:40:37 +02:00
Vasiliy 5d8abce7bb issue 373.2: move toVelocyPack into LogicalDataSource (#5288)
* issue 373.2: move toVelocyPack into LogicalDataSource

* backport: move static DataSource related strings into StaticStrings, add support for registering snapshots on DBServer views
2018-05-08 17:28:39 +03:00
Vasiliy 2ce20e86d7 issue 373.1: move globally-unique id generation from collection into data-source (#5182) 2018-05-07 22:14:40 +03:00
Andrey Abramov b69b5bdfdf
Bug fix/issue #5186 (#5269)
* do not persist view on startup

* small refactoring

* ensure view is being opened after creation
2018-05-06 20:38:32 +03:00
Andrey Abramov 4649b40b96
Coordinator ArangoSearch view + Execution nodes + AgencyMock (#5160)
* add initial implementation of scatter view rule and node

* add tests for `IResearchViewNode` and `IResearchViewScatterNode`

* add missing check

* modify IResearch execution nodes to use references instead of pointers

* use view id in searialized `ExecutionNode` representation instead of the name

* add cluster mode stubs and checks

* very first attempt to distribute IResearchViewNode

* further implementation of cluster-wide arangosearch views

* fix invalid json format

* add tests for coordinator iresearch view

* allow to retrieve a list of existing views on a coordinator

* more tests for coordinator iresearch view

* some fixes to enable query explanation

* remove Collection dependency from RemoteNode

* remove unnecessary remote ArangoSearch view scatter

* fix explanation appearance

* add some assertions

* minor fixes

* implement IResearchViewCoordinator::updateProperties

* fix view DDL issues

* handle link modifications in DDL operations

* add coordinator implementation of iresearch view links

* fix tests

* further coordinator based view DDL implementation

* further IResearchViewCoordinator implementation

* add initial implementation of AgencyMock

* fix some tests

* code cleanup

* extend test + some fixes

* more tests for IResearchViewCoordinator

* fix tests for IResearchLinkCoordinator

* some fixes after merge

* fix tests

* remove declaration of nonexistent (previously removed) method

* some fixes after review

* remove string duplication

* more tests and fixes

* more fixes and tests

* more tests

* one more test

* fix 'use-after-free' asan error

* fix non-enterprise tests issues
2018-05-02 00:15:11 +03:00
Vasiliy 9062c41592 issue 383.3: implement remainder of IResearchViewDBServer tests, use the data-source id (primary key) instead of an arbitrary instance for dropCollection()/dropView(), backport from iresearch upstream: ensure block is flushed if key index is full (#5176) 2018-04-23 00:33:46 +03:00
Vasiliy f392925903 issue 374.3: use a reference to vocbase instead of a pointer in DatabaseGuard 2018-04-13 09:56:49 +03:00
Vasiliy d470371f7c issue 374.2: use a reference to vocbase instead of a pointer in StorageEngime LogicalDatasource related DDL operations, use vocbase from the found collection 2018-04-11 18:49:46 +03:00
Vasiliy d1ce3a97ef issue 355.7: ensure LogicalDataSource::vocbase() returns a reference 2018-04-09 15:38:24 +03:00
Vasiliy e4368b0991 issue 355.6: remove create() from LogicalView, remove IResearch dependency from IndexFactory, store vocbase reference in LogicalDataSource 2018-04-06 16:38:34 +03:00
Vasiliy 99b83ba8c8 issue 355.5: remove more unused methods, move view-related storage engine functionality from vocbase into DBServerLogicalView, address MSVC cmake dependency issue 2018-04-05 16:17:07 +03:00
Vasiliy 635db3b409 issue 355.4: remove redundant methods and code, use 'cp' instead of 'cmake copy_directory' where possible, use vocbase reference instead of pointer 2018-04-04 10:53:48 +03:00
Vasiliy 7c25902b27 issue 355.3: allow IResearchLink creation, as opposed to IResearch View update, to direct which CIDs get included in an IResearchView snapshot 2018-04-02 15:27:48 +03:00
Andrey Abramov 2cca7a926b remove code duplication 2018-03-30 16:01:00 +03:00
Max Neunhoeffer 5d5a44b01b
Do not store name in properties. 2018-03-28 23:29:43 +02:00
Max Neunhoeffer c7b77076a9
Merge remote-tracking branch 'origin/devel' into feature/arangosearch-cluster-views 2018-03-28 23:26:48 +02:00
Andrey Abramov e2a6e751ed do not use dynamic_pointer_cast in release 2018-03-27 21:26:21 +03:00
Andrey Abramov 65ea2bfbb9 code cleanup 2018-03-27 18:39:59 +03:00
Andrey Abramov b641cf9b13
remove view implementation part 1 2018-03-26 23:18:08 +03:00
Max Neunhoeffer fd6fef6b9b
Try to fix createView in cluster. 2018-03-26 12:57:30 +02:00
Max Neunhoeffer 2d05cc51ae
Merge remote-tracking branch 'origin/devel' into feature/arangosearch-cluster-views 2018-03-26 11:52:10 +02:00
Max Neunhoeffer 790824fd68
Merge remote-tracking branch 'origin/devel' into feature/arangosearch-cluster-views 2018-03-26 10:50:23 +02:00
Andrey Abramov 47fd5808c3 remove PhysicalView part 2 2018-03-22 22:41:28 +03:00
Andrey Abramov a874fda823 remove PhysicalView part 1 2018-03-22 20:42:42 +03:00
Max Neunhoeffer 5d5bf94000
CHange JSON of a view in Agency. 2018-03-21 17:59:54 +01:00
Max Neunhoeffer 173b5fb892
Attach ClusterInfo methods to vocbase in coordinator case. 2018-03-21 10:54:35 +01:00
Andrey Abramov 01d9baf359 remove TRI_ERROR_ARANGO_VIEW_NOT_FOUND, rename TRI_ERROR_ARANGO_COLLECTION_NOT_FOUND to TRI_ERROR_ARNANGO_DATA_SOURCE_NOT_FOUND 2018-03-17 19:36:14 +03:00
Vasiliy 06eb8ade01 issue 344.7: remove more redundant functions (#4863)
* issue 344.7: remove more redundant functions

* backport: fix missed functions under USE_ENTERPRISE
2018-03-15 17:10:28 +01:00
Vasiliy 148bdb7158 issue 344.6: remove some redundant functions (#4842) 2018-03-15 11:03:35 +01:00
Andrey Abramov b193beeabf add nullptr check 2018-03-14 18:59:50 +03:00
jsteemann 4486f89a4d fixed more obvious compile errors 2018-03-14 13:42:35 +01:00
Vasiliy c8739cd3cd manually-merge: cache data-sources in CollectionNameResolver 2018-03-14 10:11:50 +03:00
Vasiliy bf0068e7f5 Changes since last PR: use a single map in vocbase for storing LogicalDataSource instances 2018-03-08 17:28:08 +03:00
Vasiliy daa1db659c manually-merge: differentiate data-source types by category e.g. LogicalCollection/LogicalView 2018-03-08 13:40:25 +03:00
Vasiliy f9a45032cd LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups (#4732)
* Changes since last PR: use LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups
* backport: address cluster LogicalDataSource resolution failure
2018-03-06 11:41:41 +01:00
jsteemann 37e2ef461a fix tests 2018-03-05 17:30:46 +01:00
Jan 5a67a048c5
bump version number for all local DDL changes and tell agency (#4685)
this allows other listeners (e.g. for DC2DC) to get notified when
DDL operations are carried out locally and need to be applied remotely
2018-03-05 17:06:34 +01:00
Vasiliy a988cb19d7 Changes since last PR: create a common ancestor to LogicalCollection/LogicalView (#4719) 2018-03-02 09:02:09 +01:00
Wilfried Goesgens f489b7d5de fix dumping permission checks; (#4725) 2018-03-01 17:17:11 +01:00
Andrey Abramov 1b7dd672c9 Bug fix/internal issue #342: allow to bind a view name (#4554) 2018-02-13 21:53:01 +01:00
Jan 935e2790ad
forward port of replication bugfixes from 3.2 (#4572) 2018-02-12 18:13:51 +01:00
Jan 8da61a9875
Bug fix/more replication tests (#4500) 2018-02-09 13:06:47 +01:00
Andrey Abramov a1cfb3d72b Feature iresearch (#4105) 2018-01-19 14:23:58 +01:00
Jan b2b6c06cbf
Feature/efficiency (#3736) 2018-01-05 16:51:31 +01:00
Jan 2a96df5ca5
Feature iresearch (#4071) 2017-12-18 15:04:59 +01:00
Jan 9c76613e63
fix premature unlock (#3802)
* fix some deadlocks found by evil lock manager (tm)

* fix duplicate lock

* fix indentation

* ensure proper lock dependencies

* fix lock acquisition

* removed useless comment

* do not lock twice

* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not

* AQL micro optimizations

* use explicit constructor

* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context

entering and exiting the selected context is then done later on without having to hold the ConditionLocker

* remove some recursive locks

* Disable custom deadlock detection when Thread Sanitizer is enabled

* Changing ifdef's

* grr

* broke gcc

* Using atomic for ApplicationServer::_server

* fix premature unlock

* add some asserts

* honor collection locking in cluster

* yet one more lock fix

* removed assertion

* some more bugfixes

* Fixing assert

(cherry picked from commit 1155df173bfb67303077fbe04ee8d909517bfd21)
2017-12-13 13:27:42 +01:00
Jan 282be208cc
remove TRI_usleep and TRI_sleep, and use std::this_thread::sleep_for … (#3817) 2017-12-06 18:43:49 +01:00
Jan 057e87f919 fix shutdown in case no threads can be started (#3648) 2017-11-10 10:21:51 +01:00
Jan 7613bc4314 Bug fix/fixes 0211 (#3568)
* remove some non-unused V8 persistents

* do not throw that many bogus assertions

* do not rely on server role being defined

* slightly better debug output for V8 context debugging

* fix collection ids in inventory response

* simplify bootstrap a bit

* slightly better error handling

* make elapsed time a queryable value

* use less memory for stub collections

* added assertions that will always make sense

* added assertions

* do not garbage-collect while waiting

* less copying of parameters

* do not show "load indexes into memory" buttons for mmfiles engine

  as all indexes are in memory anyway

* when a collection is truncated via the web interface, flush the WAL and rotate all active journals

this will make close all open journals on leader and followers and make them subject to compaction opportunities

* fix invalid server id values being passed from web interface to backend

* introduce afterTruncate method for indexes

* added test case for issue #3447

* updated CHANGELOG

* don't warn about replicationFactor for system collections

* check that the queries actually use the geo index and not some other index

* properly report error in web interface

* fix some internals checks that made truncate fail for bigger collections in maintainer mode

* also run a compact() operation after a serious truncate

in order to make iteration over the truncated range much faster
when the collection is next accessed

* increase default maximum number of V8 contexts to at least 16
2017-11-09 12:48:15 +01:00
Simon Grätzer 64e9377c05 Replacing /_api/collection with RestHandler (#3543) 2017-11-02 14:57:17 +01:00
Simon Grätzer 7c31960cf2 Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
Simon Grätzer 88d01b89b5 Optimizations for Caches and Graph Traversals (#3169) 2017-08-31 18:33:10 +02:00
Frank Celler 6d08d4f4aa Bug fix/scheduler delete (#3077)
* removed delete call

* cleanup

* lower cpu activity of log thread too

* fix log messages

* do not enter threads into unordered_set, as it is unneeded

* do not compile in calls to disabled plan cache

* moved AQL regex cache from thread local variables to a class of its own

* more sensible thread creation and destruction
2017-08-25 12:00:17 +02:00
Jan a10832d41e do not allow the creation of volatile collections in the RocksDB stor… (#2900)
* do not allow the creation of volatile collections in the RocksDB storage engine

* only create a volatile collection for mmfiles in test

* fix tests
2017-08-03 22:45:17 +02:00
Simon Grätzer 634254b80f adding load and unload 2017-06-02 14:44:38 +02:00
jsteemann 0366fba276 try to fix thread joining for replication applier 2017-05-18 21:05:57 +02:00
Dan Larkin 037f3f38b9 Removed accidental log spam. 2017-05-04 15:30:09 -04:00
Dan Larkin d77efe38dc Added explicit WAL file lifecycle management to reduce space overhead from replication. 2017-05-04 15:16:24 -04:00
Simon Grätzer cdbea139ce added replication client garbage collection 2017-05-04 18:40:55 +02:00
jsteemann 910f5680b0 moved files 2017-05-02 12:34:28 +02:00
jsteemann fcc61115d8 try to fix feature startup order (Scheduler vs. CacheManager vs. DatabaseFeature) 2017-04-24 13:17:21 +02:00
Michael Hackstein 1087065ea5 Merge branch 'devel' of github.com:arangodb/arangodb into engine-api 2017-04-13 17:34:25 +02:00
Max Neunhoeffer 53b3a73486 Add a timeout option to <collection>.drop().
This is in particular used to allow the Schmutz to make progress
in the case of a long running transaction.
2017-04-07 15:52:19 +02:00
Dan Larkin f056aeee42 Fixed unloaded collection status reporting issue. 2017-04-06 16:21:51 -04:00
jsteemann 2aa2fd65a4 more startup 2017-03-24 14:20:51 +01:00
jsteemann 5d6dcb3bf1 re-added constructionguard 2017-03-17 15:44:22 +01:00
jsteemann e3d8ae142e some fixes for view handling 2017-03-17 11:18:46 +01:00
Dan Larkin 508aabeafe Updated schema, added LoggerView, and added support for implementation properties. 2017-03-16 19:45:48 -04:00
jsteemann 979f003538 added ViewTypesFeature 2017-03-16 12:50:22 +01:00
jsteemann 0bb2711abb Merge branch 'engine-api' of https://github.com/arangodb/arangodb into views 2017-03-16 12:16:23 +01:00
Dan Larkin aa23074efd Added view retrieval via HTTP with tests and initial work on view factory. 2017-03-15 18:44:48 -04:00
jsteemann 7e7fe8bc21 discover views on startup 2017-03-15 14:48:10 +01:00
jsteemann 49aef67cca use std::shared_ptrs for views 2017-03-15 13:07:08 +01:00
jsteemann 1a06f1e15e initial functionality for views - unstable 2017-03-14 16:35:45 +01:00
Michael Hackstein 1ebf562c4e Moved MMFiles out of vocbase.cpp 2017-03-14 10:29:17 +01:00
Michael Hackstein 1c33e77bd8 Moved unloadCollection callback into StorageEngine. 2017-03-13 17:28:08 +01:00
Michael Hackstein 83663c47b2 Deleted CollectionResult. It is identical to arangodb::Result introduced later. 2017-03-10 17:18:12 +02:00
Michael Hackstein 90b4876d3c Moved WriteMarker code Vocbase => MMFilesEngine. Also removed the decission to write a marker out of Vocbase-API 2017-03-09 11:12:39 +01:00
Michael Hackstein 489f0d12b8 Moved the decision to write a marker into the databaseFeature, it is not triggered from outside any more. Furthermore renameCollection write marker is now done in MMFiles. 2017-03-08 14:54:25 +01:00
Michael Hackstein 31df7eb527 Fixed error in recovery for duplicate collection name 2017-03-06 17:34:54 +01:00
jsteemann 236bc67216 fix crashes 2017-02-28 17:59:28 +01:00
Michael Hackstein 900b15dd3a API cleanup in LogicalCollection. Removed some unused exposed toVelocypack functions. Removed toVelocyPack with maxTick variant. That is available in the StorageEngine instead 2017-02-28 15:35:31 +01:00
Michael Hackstein df4eadac75 Replaced several specialized toVelocyPack functions of LogicalCollection by toVelocyPackIgnore(). Fixed distributeLike bug that occured through this. 2017-02-28 14:54:32 +01:00
Michael Hackstein ae14ae99a5 Merge branch 'engine-api' of github.com:arangodb/arangodb into engine-api 2017-02-27 13:01:12 +01:00
Michael Hackstein a884ee2bfb Intermediate commit. Moved many toVPack functions of LogicalCollection to a unified vpack + ignore-keys alternative. This commit is used to run all tests everywhere. May be red 2017-02-27 13:01:07 +01:00
jsteemann 90f32ed525 renamed classes 2017-02-27 11:40:15 +01:00
jsteemann d0df1952e8 Merge branch 'devel' of https://github.com/arangodb/arangodb into engine-api 2017-02-24 17:51:59 +01:00
Michael Hackstein c8f03d16e3 updateProperties now returns an CollectionResult container with errorMessage and code. Moved error reporting logic for cluster-case into this function. 2017-02-24 17:00:41 +01:00
jsteemann dc72ec3ee7 fixed a race when dropping databases 2017-02-23 15:22:50 +01:00
jsteemann 3679050a31 fixed a race when renaming collections 2017-02-23 13:41:48 +01:00
Michael Hackstein 5d452edec6 Moved ditches into MMFiles. 2017-02-23 09:02:34 +01:00
Michael Hackstein fbdbfdcb6d Moved primaryIndex knowledge out of Logical Collection. Also moved logic for close into MMFiles 2017-02-21 14:12:07 +01:00
jsteemann f5c8932744 initial version of query execution plan cache, still very rough 2017-02-21 11:58:38 +01:00
Jan Christoph Uhde 6e1f77be24 remove ditches from interface. the ditches concept needs fixing!! 2017-02-21 10:53:51 +01:00
jsteemann 3e7432ccd6 clean up write lockers a bit 2017-02-16 15:44:45 +01:00