1
0
Fork 0
Commit Graph

399 Commits

Author SHA1 Message Date
Vasiliy bf1f2eab29 issue 401.2: split IResearchViewMeta into the user configurable part and internal state (#5683)
* issue 401.2: split IResearchViewMeta into the user configurable part and internal state

* backport: update documentation and integration tests to reflect the IResearchViewMeta changes

* backport: addresss typo

* backport: address another typo
2018-06-27 16:40:51 +03:00
jsteemann a9075751bb add include file 2018-06-27 10:51:45 +02:00
Vasiliy 532b4e613e issue 401.1: move IResearchView sync thread logic into its own class (#5668)
* issue 401.1: move IResearchView sync thread logic into its own class

* backport: use arangodb::Thread for ArangoSearch Sync
2018-06-26 15:47:56 +03:00
Andrey Abramov 5eef6cd618
Feature/test iresearch (#5610)
* start implementing arangosearch cluster tests.

* backport: ensure view lookup is done via collectionNameResover, ensure updateProperties returns current view properties

* first attempt to fix failing tests

* refactor cluster wide view creation logic

* if view is not found in the new plan then check the old plan too

* ensure the cluster-wide view is looked up in vocbase as well on startup/recovery

* do not store cluster-wide IResearchView in vocbase

* move stale view cleanup to the shared pointer deleter, address test failures

* do not print warning

* enable arangosearch tests by default

* fix catch tests

* address icorrect return value for cluster-wide links

* address some issues with test failures due to cluster-view allocated within TRI_vocbase_t

* simplify per-cid view name, address 'catch' test failures

* ensure IResearchViewNode volatility is properly calculated in cluster

* invoke callbacks directly in AgencyMock instead of waiting for timeout

* ensure view updates via JavaScript always use the latest view definition

* pass a list of shards to `IResearchViewDBServer::snapshot`

* extend cluster aql tests

* fixes after merge

* fix class/struct inconsistencies

* comment failing tests

* remove debug logging

* add debug function

* tests cleanup

* simplify upcoming merge: pass resolver from a side

* backport: move all transaction status callback logic to Methods

* add changes missed from previous commit

* fix js and ruby tests

* more tests for IResearchViewNode

* pass transaction to IResearchViewDBServer::snapshot, address IResearchViewDBServer tests segfault

* pass transaction to IResearchView::snapshot instead of transaction state

* temporarily add trace log output to tests to try to find the cause of the core dump on Jenkins

* add more temporary debug output to trace down the segfault on Jenkins

* add even more temporary debug output to trace down the segfault on Jenkins

* ensure Vieew related maps are cleared during shutdown

* reset ClusterInfo::instance() before DatabaseFeature::unprepare()

* remove extraneous debug output

* missed line from previous commit

* uncomment required line

* add nullptr checks to RocksDBIndexFactory::prepareIndexes(...) similar to the ones in MMFilesIndexFactory::prepareIndexes(...)

* attempt to fix deadlock in tests

* add comment as per reviewer request

* fix aql test suite name

* add some debug logging

* address deadlock between ClusterInfo::loadPlan() and CollectionNameResolver::localNameLookup(...)

* eplicitly state which index definition failed in the log message

* use vocbase from shard-view isntead just in case

* explicitly state which index definition failed in the log message

* do not create shard-view instances from cluster-link instances (only register existing ones)

* add some tests
2018-06-21 20:35:04 +03:00
Vasiliy b7f7711d30 issue 380.4: allow transaction status change and data-source registration callbacks to be registered with a transaction (#5624) 2018-06-18 13:16:32 +02:00
Simon 3bec336aff TransactionState::addCollection refactoring (#5606) 2018-06-14 15:34:58 +02:00
Vasiliy a5f274234a issue 380.2.2: remove extraneous pointers from StorageEngine API (#5589) 2018-06-13 15:22:51 +03:00
Jan edb24deafe
cleanup collection/vocbase interface of ExecutionNodes (#5557) 2018-06-08 15:28:48 +02:00
Andrey Abramov e117a97fb4
Feature/arangosearch scatter node (#5536)
* remove collection dependency from scatter

* simplify `createBlock` API

* remove useless IResearchViewScatter node

* some fixes

* code cleanup

* scatter arangosearch view

* cleanup

* fix compilation errors

* fix test issues

* store shared_ptr rather than raw pointer in IResearchViewNode

* refactor ScatterBlock

* refactor DistributeBlock

* ensure no leaks during execution node cloning

* some fixes

* code cleanup

* remove vocbase from `DistributeNode`

* add missing ifdef

* fixes after review
2018-06-06 20:42:53 +03:00
Vasiliy 1bd4947629 issue 381.7: move per-transaction IResearchView state from the view into the TransactionState (#5530) 2018-06-06 18:35:15 +03:00
Vasiliy 3e983adb2d issue 381.6: enforce use of VIEW keyword for bound views and disallow for bound collections (#5522) 2018-06-05 14:45:53 +03:00
Vasiliy 4253dca6aa issue 381.5: ensure the LogicalView definition that is persisted to the Agency matches the definition that gets created (#5518)
* issue 381.5: ensure the LogicalView definition that is persisted to the Agency matches the definition that gets created

* backport: correct comment
2018-06-02 17:21:55 +03:00
Vasiliy ac1a6742e0 issue 381.4.1: create a separate IResearchLink implementation for use with ClusterEngine (#5505)
* issue 381.4.1: create a separate IResearchLink implementation for use with ClusterEngine

* backport: ensure StorageEngineMock assigns unique transaction ids

* delegate index normalization on the ClusterEngine to the 'actualEngine'
2018-06-01 19:05:53 +03:00
Andrey Abramov f708e46973
Feature/arangosearch gather node (#5487)
* integrate ArangoSearch into GatherNode/GatherBlock logic

* fix inconsistency after merge

* add serialization/deserealization logic for IResearchViewNode sorting

* fix optimizer rule order

* fix copy/past typo

* fix tests

* ensure `GatherNode` is properly set up for ArangoSearch views

* adjust optimizer rule order
2018-05-30 02:53:21 +03:00
Andrey Abramov 0f0235e8a9
Remove collection dependency from gather node (#5457)
* remove collection dependency

* some fixes

* fix test issues

* add tests for newly created functionality

* fix enterprise test fails

* add comment
2018-05-25 17:26:33 +03:00
Vasiliy e021085aac issue 381.3.1: remove redundant copy of CollectionNameResolver from TransactionState (#5459) 2018-05-25 16:48:43 +03:00
Vasiliy bc70486761 issue 381.2: ensure the correct index factories/normalizers are called in ClusterEngine (#5450) 2018-05-24 15:23:17 +03:00
Vasiliy c26496c44c issue 381.1: ensure view lookup is done via collectionNameResover, ensure updateProperties returns current view properties, remove redundant functions (#5439) 2018-05-23 19:29:53 +03:00
Andrey Abramov 0489322cd1
Distributed ArangoSearch query execution (part 2) (#5397)
* further implementation of arangosearch query distribution

* do not store `aql::Collection` in `IResearchViewNode`, use `Query::collections()` instead

* some fixes

* cleanup

* some fixes after review

* more fixes after review

* use assertion instead of condition
2018-05-23 19:29:18 +03:00
Jan 8e6d5df129
fixed minor several compiler complaints (#5406) 2018-05-23 11:50:00 +02:00
Vasiliy 94ddd7803d issue 389.10: refactor CollectionNameResolver to use TRI_vocbase_t& (#5424) 2018-05-23 00:59:08 +03:00
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00
Vasiliy 27f7f4a0aa issue 389.9: return TRI_vocbase_t& from Query::vocbase() (#5420)
* issue 389.9: return TRI_vocbase_t& from Query::vocbase()

* backport: use view name instead of view id in error messages
2018-05-22 19:03:14 +03:00
Vasiliy 370bde71d9 issue 389.7: expand workaround to include MSVC2015 debug code, backport from iresearch upstream: skip importing empty readers (#5391)
* issue 389.7: expand workaround to include MSVC2015 debug code, backport from iresearch upstream: skip importing empty readers

* backport: address MSVC V8/ICU link issues

* backport: fix typo
2018-05-20 22:43:37 +03:00
Vasiliy 907767f4e1 issue 389.6: ensure proper error handling in StatisticsFeature for missing feature (#5377) 2018-05-18 01:12:39 +03:00
Vasiliy 843e584746 issue 389.5: refactor StandaloneContext to be constructed with a TRI_vocbase_t& (#5370)
* issue 389.5: refactor StandaloneContext to be constructed with a TRI_vocbase_t&

* backport: address build issues
2018-05-17 01:15:50 +03:00
Vasiliy 5a66661f78 issue 389.4: implement IResearchViewDBServer::snapshot (#5358)
* issue 389.4: implement IResearchViewDBServer::snapshot
* backport: revert back namespace header changes
* backport: address review comments
2018-05-16 16:33:48 +02:00
Vasiliy 6a53154160 issue 389.2: use static strings for Index definition json attributes, use TRI_vocbase_t references instead of pointers in V8Context, use TRI_vocbase_t references instead of pointers in DatabaseInitialSyncer (#5344) 2018-05-14 19:06:24 +03:00
Andrey Abramov 671734b32f
do not scatter empty view (#5318)
* do not distribute scatter views without links

* simplify view downcasting

* refactor & add some tests

* adjust comments
2018-05-11 11:45:22 +03:00
Vasiliy d48bdf72e1 issue 373.4: address issue with MSVC picking incorrect function, minor formatting changes, backport from iresearch upstream: ensure id_to_column contains enough slots for all column ids, address another issue with _T(...) macro not working when _MBCS is defined on MSVC (#5310) 2018-05-10 20:12:37 +03:00
Vasiliy 5d8abce7bb issue 373.2: move toVelocyPack into LogicalDataSource (#5288)
* issue 373.2: move toVelocyPack into LogicalDataSource

* backport: move static DataSource related strings into StaticStrings, add support for registering snapshots on DBServer views
2018-05-08 17:28:39 +03:00
jsteemann 7f8a1cc614 Merge branch 'bug-fix/add-missing-overrides-and-final' of https://github.com/arangodb/arangodb into devel 2018-05-07 23:02:46 +02:00
Vasiliy 2ce20e86d7 issue 373.1: move globally-unique id generation from collection into data-source (#5182) 2018-05-07 22:14:40 +03:00
Andrey Abramov b69b5bdfdf
Bug fix/issue #5186 (#5269)
* do not persist view on startup

* small refactoring

* ensure view is being opened after creation
2018-05-06 20:38:32 +03:00
jsteemann 52de92d334 add missing override specifiers, add final specifiers 2018-05-04 09:01:50 +02:00
Jan 9c0460acff
Feature/aql optimizations (#5237) 2018-05-03 16:55:58 +02:00
Andrey Abramov 6b72cb1a81
fix clang-802.0.38 build issue (#5245) 2018-05-02 17:15:58 +03:00
Andrey Abramov 4649b40b96
Coordinator ArangoSearch view + Execution nodes + AgencyMock (#5160)
* add initial implementation of scatter view rule and node

* add tests for `IResearchViewNode` and `IResearchViewScatterNode`

* add missing check

* modify IResearch execution nodes to use references instead of pointers

* use view id in searialized `ExecutionNode` representation instead of the name

* add cluster mode stubs and checks

* very first attempt to distribute IResearchViewNode

* further implementation of cluster-wide arangosearch views

* fix invalid json format

* add tests for coordinator iresearch view

* allow to retrieve a list of existing views on a coordinator

* more tests for coordinator iresearch view

* some fixes to enable query explanation

* remove Collection dependency from RemoteNode

* remove unnecessary remote ArangoSearch view scatter

* fix explanation appearance

* add some assertions

* minor fixes

* implement IResearchViewCoordinator::updateProperties

* fix view DDL issues

* handle link modifications in DDL operations

* add coordinator implementation of iresearch view links

* fix tests

* further coordinator based view DDL implementation

* further IResearchViewCoordinator implementation

* add initial implementation of AgencyMock

* fix some tests

* code cleanup

* extend test + some fixes

* more tests for IResearchViewCoordinator

* fix tests for IResearchLinkCoordinator

* some fixes after merge

* fix tests

* remove declaration of nonexistent (previously removed) method

* some fixes after review

* remove string duplication

* more tests and fixes

* more fixes and tests

* more tests

* one more test

* fix 'use-after-free' asan error

* fix non-enterprise tests issues
2018-05-02 00:15:11 +03:00
jsteemann cb51e561b7 fix catch tests 2018-04-30 23:13:50 +02:00
Jan 30b12e311b
Bug fix/remove most of aql js (#5223) 2018-04-30 11:17:11 +02:00
Wilfried Goesgens 7d6e580780 Refactoring & code cleanup (#5138) (#5142) 2018-04-24 14:42:23 +02:00
Jan 2b84348b77
remove call to requiresElevatedPrivileges with default value (#5172) 2018-04-23 11:28:24 +02:00
Vasiliy 9062c41592 issue 383.3: implement remainder of IResearchViewDBServer tests, use the data-source id (primary key) instead of an arbitrary instance for dropCollection()/dropView(), backport from iresearch upstream: ensure block is flushed if key index is full (#5176) 2018-04-23 00:33:46 +03:00
Vasiliy da4519df43 issue 383.2: backport from iresearch upstream: address issue with _T(…) macro not working when _MBCS is defined on MSVC, backport from iresearch upstream: address issue with _T(...) macro not working when _MBCS is defined on MSVC (#5152)
* issue 383.2: backport from iresearch upstream: address issue with _T(...) macro not working when _MBCS is defined on MSVC, backport from iresearch upstream: address issue with _T(...) macro not working when _MBCS is defined on MSVC

* backport: add more tests for IResearchViewDBServer
2018-04-20 18:20:31 +03:00
Vasiliy 6053056b6a issue 383.1: initial stub implementation of IResearchViewDBServer (#5132) 2018-04-18 13:51:21 +03:00
Vasiliy 9a4ddee13e issue 374.4: move iresearch related log topic and type into a separate file so as not to expose it externally (#5107)
* issue 374.4: move iresearch related log topic and type into a separate file so as not to expose it externally

* remove invalid include
2018-04-16 18:30:51 +03:00
Vasiliy a69e4a63d0 backport: address MSVC build warnings 2018-04-13 11:38:43 +03:00
Vasiliy d470371f7c issue 374.2: use a reference to vocbase instead of a pointer in StorageEngime LogicalDatasource related DDL operations, use vocbase from the found collection 2018-04-11 18:49:46 +03:00
Vasiliy d1ce3a97ef issue 355.7: ensure LogicalDataSource::vocbase() returns a reference 2018-04-09 15:38:24 +03:00
Vasiliy e4368b0991 issue 355.6: remove create() from LogicalView, remove IResearch dependency from IndexFactory, store vocbase reference in LogicalDataSource 2018-04-06 16:38:34 +03:00
Simon 3ec8bcfd97
Merge branch 'devel' into bug-fix/internal-issue-#355.5 2018-04-05 17:34:55 +02:00
Simon 9980067e73 fixing various warnings (#5013) 2018-04-05 17:01:16 +02:00
Vasiliy 99b83ba8c8 issue 355.5: remove more unused methods, move view-related storage engine functionality from vocbase into DBServerLogicalView, address MSVC cmake dependency issue 2018-04-05 16:17:07 +03:00
Vasiliy 635db3b409 issue 355.4: remove redundant methods and code, use 'cp' instead of 'cmake copy_directory' where possible, use vocbase reference instead of pointer 2018-04-04 10:53:48 +03:00
Vasiliy fa13c72950 backport: address review comments 2018-04-03 10:12:38 +03:00
Vasiliy 7c25902b27 issue 355.3: allow IResearchLink creation, as opposed to IResearch View update, to direct which CIDs get included in an IResearchView snapshot 2018-04-02 15:27:48 +03:00
Andrey Abramov 9addef6419 small fixes after review 2018-03-28 18:45:14 +03:00
Andrey Abramov e2a6e751ed do not use dynamic_pointer_cast in release 2018-03-27 21:26:21 +03:00
Andrey Abramov 65ea2bfbb9 code cleanup 2018-03-27 18:39:59 +03:00
Andrey Abramov a7760cb3b7 fix tests 2018-03-27 14:44:10 +03:00
Andrey Abramov b641cf9b13
remove view implementation part 1 2018-03-26 23:18:08 +03:00
Andrey Abramov a164b25039
Merge pull request #4942 from arangodb/feature/refactor-logical-view
refactor logical view
2018-03-26 12:49:43 +03:00
Andrey Abramov f2703786f1
Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/refactor-logical-view 2018-03-23 15:17:17 +03:00
Vasiliy 0823f23764 backport: address merge conflicts 2018-03-23 12:04:25 +03:00
Vasiliy eae3d0ade8 issue 348.2: hold view lock directly instead of via an Ayncself pointer 2018-03-23 10:28:00 +03:00
Andrey Abramov 47fd5808c3 remove PhysicalView part 2 2018-03-22 22:41:28 +03:00
Andrey Abramov 614cf73ff8
Merge pull request #4922 from arangodb/bug-fix/internal-issue-#348.1
issue 348.1: use a view validity lock instead of a view member lock for snapshot readers
2018-03-22 18:48:31 +03:00
Vasiliy a9b22ad36b issue 348.1: use a view validity lock instead of a view member lock for snapshot readers 2018-03-22 12:45:29 +03:00
Andrey Abramov 15304b705b
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug-fix/internal-issue-#345 2018-03-21 14:37:07 +03:00
Jan 5105e2b3c0
Remove "atLeast" from the internal query APIs (#4848) 2018-03-21 09:10:28 +01:00
Andrey Abramov 01d9baf359 remove TRI_ERROR_ARANGO_VIEW_NOT_FOUND, rename TRI_ERROR_ARANGO_COLLECTION_NOT_FOUND to TRI_ERROR_ARNANGO_DATA_SOURCE_NOT_FOUND 2018-03-17 19:36:14 +03:00
Vasiliy 148bdb7158 issue 344.6: remove some redundant functions (#4842) 2018-03-15 11:03:35 +01:00
Jan b325d05357
Bug fix/aql auto keep on collect (#4795)
* automatically detect which variables to keep in AQL COLLECT

If a COLLECT INTO is used, then it is detected which sub-attributes
of the into variables are used later in the query, and automatic
KEEP instructions are added to the COLLECT if possible

Example query:

    FOR doc1 IN collection1
      FOR doc2 IN collection2
        COLLECT x = doc1.x INTO g
        RETURN { x, all: g[*].doc1.y }

would be turned into

    FOR doc1 IN collection1
      FOR doc2 IN collection2
        COLLECT x = doc1.x INTO g KEEP doc1
        RETURN { x, all: g[*].doc1.y }

and prevent `doc2` from being temporarily stored in the variable `g`

* fix runtime warnings

* increase default flush timeout

* added tests
2018-03-14 13:59:46 +01:00
Vasiliy c8739cd3cd manually-merge: cache data-sources in CollectionNameResolver 2018-03-14 10:11:50 +03:00
Andrey Abramov 90e0a9326b
remove ArangoSearch dependencies from StorageEngineMock 2018-03-12 13:34:57 +03:00
Vasiliy daa1db659c manually-merge: differentiate data-source types by category e.g. LogicalCollection/LogicalView 2018-03-08 13:40:25 +03:00
Vasiliy f9a45032cd LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups (#4732)
* Changes since last PR: use LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups
* backport: address cluster LogicalDataSource resolution failure
2018-03-06 11:41:41 +01:00
Andrey Abramov 9f84c95f0a Execution block mock (#4710) 2018-03-05 16:24:15 +01:00
Andrey Abramov f6c184df33 Factory method for execution block (#4729) 2018-03-05 15:02:30 +01:00
Vasiliy a988cb19d7 Changes since last PR: create a common ancestor to LogicalCollection/LogicalView (#4719) 2018-03-02 09:02:09 +01:00
Vasiliy f4dbfdbdbd Changes since last PR: partially revert commit bd86eea365737300a4e0e8d23938a88d54cfcbf4 to address test warnings, partially revert commit a1cfb3d72b to address test warnings, remove unused function (#4699) 2018-03-01 09:29:11 +01:00
Vasiliy 423ce33ed1 Changes since last PR: remove 'dataPath' IResearchView configuration property and use a hardcoded path similar to a LogicalCollection (#4683) 2018-02-27 18:10:32 +01:00
Simon cde96d97aa RocksDB replication hardening & cleanup (#4639) 2018-02-23 18:15:04 +01:00
Andrey Abramov 1b64f03bf8 Bug fix/internal issue #327 (#4649) 2018-02-22 11:11:23 +01:00
Andrey Abramov d130e0f7c6 Bug fix/internal issue #350 (#4601) 2018-02-15 18:57:08 +01:00
Vasiliy 828f8fb48c locally tested only, Changes since last PR: ensure parser.cc dependency is tracked in the build directory to avoid bison invocation on MSVC, aquire IResearchView snapshot at transaction begin() and reuse it for all queries, remove unused callback registration from Methods (#4564) 2018-02-14 12:32:29 +01:00
Dan Larkin 96dbc693db Speed up condition normalization (DNF conversion) (#4574) 2018-02-13 22:13:52 +01:00
Vasiliy cf0d575246 locally tested only, Changes since last PR: disable IResearch IQL until IResearch upstream is updated, address some warnings (#4519) 2018-02-07 09:40:06 +01:00
Andrey Abramov af5318c407 Merge branch 'devel' of https://github.com/arangodb/arangodb into buf-fix/internal-issue-#334
# Conflicts:
#	arangod/IResearch/IResearchLink.cpp
#	arangod/IResearch/IResearchView.cpp
2018-02-05 19:03:05 +03:00
Vasiliy aa041ddfb9 locally tested only (#4476) 2018-02-05 11:24:34 +01:00
Andrey Abramov 2bdc341fa0 some fixes after review 2018-02-02 20:21:42 +03:00
Andrey Abramov 630d1afacf properly handle `IndexCreate` markers in rocksdb recovery 2018-02-02 16:23:25 +03:00
Andrey Abramov ffbb00bbe2 standardize on the 'analyzer' and 'trackListPositions' naming (#4384) 2018-01-22 16:39:53 +01:00
Andrey Abramov 42cbedd9f4 fix `centos` build issue (#4379) 2018-01-22 14:34:21 +01:00
Andrey Abramov a1cfb3d72b Feature iresearch (#4105) 2018-01-19 14:23:58 +01:00
jsteemann 6f30b47853 several compile warnings fixed 2018-01-05 18:20:41 +01:00
Heiko 78213b3b4a fixed some macos ires related warnings and errors (#4196) 2018-01-05 14:51:13 +01:00
Heiko 221e3f4f93 use struct instead of class (#4194) 2018-01-03 21:39:35 +01:00
Jan 2a96df5ca5
Feature iresearch (#4071) 2017-12-18 15:04:59 +01:00