1
0
Fork 0
Commit Graph

69 Commits

Author SHA1 Message Date
Andrey Abramov a793a4749f
Bug fix/rename iresearch analyzer feature (#9424)
* rename feature

* ensure 'MMFilesLogFileManager::SafeToUseInstance' is safe to use in multithreaded context

* address mac compilation issue
2019-07-08 15:46:57 +03:00
Andrey Abramov 174b54b036
Bug fix/internal issue #605 (#9421)
* release tick before committing changes to arangosearch

* fix tests

* address review comments

* address review comments

* add comment
2019-07-08 14:07:11 +03:00
Andrey Abramov 671380b8fb
bug-fix/internal-issue-#604 (#9353)
* ensure flush subscriptions are being unsubscribed

* update tick even if no changes happened

* remove debug output

* fix test

* address review comments

* address test failures
2019-07-01 16:00:14 +02:00
Simon cf7cf0131b Try to fix corruption error (#9258) 2019-06-25 10:18:26 +02:00
Andrey Abramov 0fb8b348db
bug-fix/internal-issue-#574 (#8991)
* restrict number of buckets to compare

* do not use more sort buckets than requested

* cleanup
2019-06-04 18:55:25 +03:00
Andrey Abramov e8b38bfa8e
bug-fix/internal-issue-#549 (#9086)
* do not persist legacy analyzers into _analyzers table

* fix arangosearch upgrade in cluster

* get rid of Vasiliy's shit

* address review comments

* ensure link is synchronized after creation in upgrade

* fix compilation error

* minor cleanup

* fix tests

* distribute '_analyzers' collection as '_graphs'

* comment out Vasiliy's shit part 2
2019-05-30 22:00:06 +03:00
Simon 394c070a4f Do not check isEnabled (#8997) 2019-05-17 16:33:17 +02:00
Vasiliy 3a3c360765 issue 526.5: add analyzer retrieval functionality (#8592) 2019-03-29 13:23:36 +01:00
Andrey Abramov 930b09cd93
[3.4] bug-fix/issue-#8294 (#8430) (#8585)
* [3.4] bug-fix/issue-#8294 (#8430)

* fix invalid optimizations for multi-valued attributes

* fix broken optimizations for multivalued attributes

* adjust tests

* add `IN_RANGE` function

* add tests

* add some shallow integration tests

* fix optimization for IN operator

# Conflicts:
#	arangod/IResearch/IResearchFeature.cpp
#	arangod/IResearch/IResearchViewOptimizerRules.cpp
#	tests/IResearch/IResearchFilterBoolean-test.cpp

* fix compilation errors

* fix another compilation issue

* address compilation errors

* fix tests
2019-03-27 18:33:16 +03:00
Jan Christoph Uhde c3f7961b88 apply unique log ids (#8561) 2019-03-25 20:26:51 +01:00
Vasiliy ea48789c54 issue 466.4: implement persistence of IResearchLink WAL flush marker (#8059)
* issue 466.4: implement persistence of IResearchLink WAL flush marker

* address enterprise test failure

* address test failures

* treat missing collections/indexes as potentially removed

* change view asertios to match collection assertions

* convert assertions to exceptions since they are possible

* revert assertion removal since assertions are actually valid for coordinator

* address scenario where link is dropped twice

* check for data store state before returning error

* revert last change since it's not valid for MMFiles
2019-02-04 16:36:13 +03:00
Manuel Pöter ecf4d9d62a Fix race conditions in thread management. (#8032) 2019-01-28 15:44:46 +01:00
Andrey Abramov 360b9f456c
reformat arangosearch sources (#7941) 2019-01-11 20:32:54 +03:00
Andrey Abramov d4a010baf9
Bug fix/internal issue #316 (#7911)
* allow using scorers outside ArangoSearch view context

Signed-off-by: Andrey Abramov <andrey@arangodb.com>

* ensure query is properly optimized after replacement of scorer functions

* do not apply `handleViewsRule` to queries without views

* simplify optimization rule for ArangoSearch views

* show ArangoSearch view scorers in query explanation

* fix tests

* fix tests

* add stub for scorer related tests

* reformat

* check variable depth in `ViewExpressionContext::getVariableValue`

* add some tests

* address js test failures

* address jslint errors

* ensure `IResearchViewNode` exposes variables used in scorers

* ensure scorers with expressions are deduplicated

* fix deduplication for indexed access

* more tests

* partially address review comments

* address review comments

* simplify code

* remove irrelevant, commented out code

* ensure array comparisons are properly handled

* update changelog & loki
2019-01-10 21:04:19 +03:00
Vasiliy 5959f7f8e8 issue 511.7.3: restore single-document optimization, implement call to update data-store options, add IResearchView upgrade step (#7918)
* issue 511.7.3: restore single-document optimization, implement call to update data-store options, add IResearchView upgrade step

* address merge issue

* address another merge issue

* and another merge issue

* run ./utils/reformat.sh on the source tree

* address review comments
2019-01-09 17:16:02 +03:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Vasiliy 8e7f0df775 issue 511.4: ensure IResearchLink can be initialized without view as required for db-server, minor code fixes and cleanup (#7786)
* issue 511.4: ensure IResearchLink can be initialized without view as required for db-server, minor code fixes and cleanup

* remove unnessesary checks

* revert last change due to test failures

* fix typo in change revert

* try to address random test failures
2018-12-18 18:45:11 +03:00
Vasiliy f1bf5cc9cf issue 511.3: remove IResearchViewDBServer and use the IResearchView directly on db-server (#7748)
* issue 511.3: remove IResearchViewDBServer and use the IResearchView directly on db-server

* ensure objectId is set in RocksDBLink
2018-12-17 21:11:20 +03:00
Vasiliy e16be8f3db issue 505.1: move index json comparison into index-type-specific implementations to address issue with incorrect definition comparison (#7430)
* issue 505: move index json comparison into index-type-specific implementations to address issue with incorrect definition comparison

* remove commented-out code

* fix typo

* backport: add additional logging and fix typo
2018-12-03 16:26:08 +03:00
Andrey Abramov d6de637cb7
treat all iresearch scores as float_t for 3.4 (#7573) (#7578)
* treat all iresearch scores as float_t for 3.4

* cleanup

* attempt to fix tests

* another attempt to fix tests
2018-12-02 15:55:42 +03:00
Simon 17a95710bd Micro-Optimize AQL CXX calls (#7486) (#7489)
(cherry picked from commit 8ddb9a063b)
2018-11-29 14:00:41 +01:00
Jan bb09e357dc
Bug fix/aql micro optimizations (#7287) 2018-11-13 11:54:07 +01:00
Andrey Abramov 194bc1b254
make ArangoSearch error messages more descriptive (#7252) 2018-11-07 19:45:52 +03:00
Jan c7d76c5da4
fix some issues reported by codescan (#7195) 2018-11-02 19:04:39 +01:00
Vasiliy 8f44afb6cf issue 496.1: switch scope of responsibility between a TRI_vocbase_t and a LogicalView in respect to view creation/deletion (#7101)
* issue 496.1: switch scope of responsibility between a TRI_vocbase_t and a LogicalView in respect to view creation/deletion

* backport: address test failures

* backport: ensure arangosearch links get exported in the dump

* backport: ensure view is created during restore on the coordinator

* Updates for ArangoSearch DDL tests, IResearchView unregistration and known issues

* Add fix for internal issue 483
2018-10-30 12:50:35 +03:00
Vasiliy 78567bef09 update iresearch to codebase as of 20181011 (#6858)
* update iresearch to codebase as of 20181011

* backport: address cluster test failures

* backport: address dump test failures

* backport: address discrepency in view creation between single-server and cluster

* backport: address test failure on cluster (revert change)

* backport: address test failures

* backport: address MSVC build issues

* backport: address issue with LogicalDatasource destructing after TRI_vocbase_t

* Revert "backport: address issue with LogicalDatasource destructing after TRI_vocbase_t"

This reverts commit 4f9880bbaa22194dfbb604b5a54658de1d447ac1.
2018-10-12 21:07:12 +03:00
Andrey Abramov 647949e556
Feature/update iresearch (#6727)
* update iresearch part 1

* fix tests

* update iresearch again

* update iresearch again

* cleanup log messages

* update iresearch sources

* another update

* and another update

* add workaround for the bug in msvc2017

* fix typo

* address msvc build issues

* Fix typo
2018-10-04 23:56:17 +03:00
Dan Larkin-York 0dfabd8f04 Fix several TSan warnings (#6473) 2018-09-14 11:16:45 +02:00
Jan 6f10211f09
this moves aql::Query* out of the call parameter to any aql::Function (#6485) 2018-09-13 23:37:46 +02:00
Lars Maier e2930ccba6 Completely export views dump restore (#6466) 2018-09-12 17:21:22 +02:00
Jan 7cb1da12cf
make AQL functions construction easier (#6195) 2018-08-20 11:56:38 +02:00
Vasiliy 364360ae79 issue 435.1: allow a user to specify the number of IResearchFeature threads via command line (#6164) 2018-08-16 13:14:15 +02:00
Jan ed9d901846
fixed smaller issues (#6158) 2018-08-16 08:49:46 +02:00
Vasiliy 6fd541d110 issue 427.5: use ApplicationServer reference instead of pointer (#6145)
* issue 427.5: use ApplicationServer reference instead of pointer

* address MSVC build failure
2018-08-15 12:16:02 +03:00
Jan 95d1892f18
remove canThrow attribute from AQL functions (#6123)
Its was somehow overlapping with the isDeterministic attribute (but with the opposite meaning)
2018-08-09 16:45:54 +02:00
Andrey Abramov b8e7752787 Feature/arangosearch stabilization (#6069)
* Added authentication tests for ArangoSearch views creation (client and task)

* Added authentication tests for ArangoSearch views querying (client and task)

* Added authentication tests for ArangoSearch views update (client and task) and correct other tests

* Added authentication tests for ArangoSearch views drop (client and task) and correct other tests

* Fix tests to support additional 'properties' layer of a view

* Add data to dumped/restored collection with a link from a view to single server test and view dump/restore to cluster tests

* backport from iresearch upstream: use a bitvector instead of an acceptor lambda for consolidation policies

* ensure view access permissions are checked for REST calls

* ensure view access permissions are checked for V8 calls

* do not allow LogicalView property modification if reading the properties fails

* Fixed authentication tests for ArangoSearch views to pass regardless of errorNum and possible multiple REST codes

* add missing ldap feature for enterprise version

* do not store useless data in ArangoSearch index (see #289, #441)
2018-08-05 20:47:22 +02:00
Vasiliy eeacab3d97 issue 400.5: limit the number of async threads to be less than the number of cores (#5947) 2018-07-23 14:57:36 +03:00
jsteemann 130de56454 turn off IResearch threads on coordinator and agent 2018-07-17 22:12:39 +02:00
Jan 633362ef4e
defer creation of ArangoSearch threadpool (#5882) 2018-07-17 11:55:14 +02:00
Michael Hackstein 7a95c5e675
Feature/feature phases (#5272)
* Added feature phases

* BasicsPhase and DatabasePhase to the required files. Server now has Feature circles and does not boot. Will be sorted out later on.

* Added ClusterPhase to features

* Added V8Phase to the required features

* Added AQLPhase to the affected features

* Added ServerPhase to Features

* Added FoxxPhase to the relevant features

* Added AgencyPhase to the relevant features

* Moved registration from local variable SYS_SYSTEM_REPLICATION_FACTOR from cluster to V8 as their ordering is now vice versa

* Moved Bootstrap feature into FoxxPhase. It could be moved to ServerPhase easily if the FoxxQueue dependency would be removed

* Final movement of Startup Phases. Now solved all circles.

* Removed merge conflict

* Moved ReplicationTimeout into cluster phase and fixed cross-phase requirements

* Added greetings phase. This phase separates the Basics Phase and is the first to be run. Includes Logger and Hello/Goodbye

* Added the GreetingsPhase in the corresponding features. Now all BasicsPhase features start after greetings Phase. There is some issue in this branch which prevents the Agency from Gossipping right now. Will be fixed next

* Moved creation of the Agent into the prepare phase of the feature. THereby it is guaranteed that agents at least exists before the GeneralServer is activating endpoints

* Recovery needs to be started after the ServerID

* Moved log output of FeaturePhases to DEBUG instead of ERROR.

* Added feature phases for clients

* ClusterFeature now does not directly require AgencyFeature any more

* Added requirement of TravEngineRegistryFeature in AQL feature. Otherwise shutdown may be undefined

* The ApplicationServer can now handout the list of ordered features. Used for testing purposes

* Fixed IResearchVew Tests Setup to honor new feature ordering

* Fixed IResearchViewDBServer Tests Setup to honor new feature ordering

* Started fixing IResearchView Coordinator tests with startup ordering. Not finished yet

* Added startup phases to ViewCoordinator test

* Disabled expected logoutput in ClusterRepairsTest

* Fixed indention in test code

* LinkCoordinator now honors startup ordering

* Link meta now honors startup rdering

* Supress expected cluster logs in ViewTest

* Removed '#' accidentially added.
2018-07-16 14:09:36 +02:00
Jan 7fb985d573
Bug fix/fix codescan issues (#5872) 2018-07-16 10:58:14 +02:00
Vasiliy 42b2890887 issue 400.4: address possible deadlock due to resourceMutex reset while tasks hold and aquire read lock (#5869) 2018-07-13 17:45:30 +03:00
Vasiliy 6ba3d98e39 issue 400.3: remove redundant asynchronous execution parameter (#5852) 2018-07-12 17:46:30 +03:00
Vasiliy 64666d0edd issue 400.2.4: register IResearchView sync tasks with IResearchFeature, remove unused configuration parameter (#5813)
* issue 400.2.4: register IResearchView sync tasks with IResearchFeature, remove unused configuration parameter

* set flag before notify, fix failures in new tests from 'devel'

* revert set flag before notify, fix more failures in new tests from 'devel'

* fix even more failures in new tests from 'devel', backport: do not reset notification flag if pending tasks were seen

* fix still more failures in new tests from 'devel'

* try to work around buggy test failures
2018-07-10 18:15:01 +03:00
Andrey Abramov b767e5625e
Bug fix/internal issue #417 (#5775)
* ensure correctness of log level mapping

* fix bug

* code cleanup
2018-07-05 16:58:44 +03:00
Vasiliy 9c04a6059c issue 401.1.1: implement an asynchronous task execution pool (thread pool) in IResearchFeature (#5761) 2018-07-04 15:18:25 +03:00
Andrey Abramov 17e955d837
ArangoSearch advanced AQL functions (#5756)
* extend filter checking

* extend IResearchViewNode tests

* distinguish scorer and filter arangosearch functions

* initial implementation

* add some tests for the recently added aql functions

* more tests

* force phrase function to use context-wise analyzer

* more tests

* ensure exists function uses analyzer from context

* fix query execution tests

* allow to override context analyzer in phrase function

* modify EXISTS signature to accept optional analyzer, fix tests

* more tests

* even moar tests

* more tests

* update documentation

* add testing scorer

* fixes after review

* attempt to fix msvc compilation issues

* add extra check

* add another check
2018-07-04 02:59:31 +03:00
Andrey Abramov 5eef6cd618
Feature/test iresearch (#5610)
* start implementing arangosearch cluster tests.

* backport: ensure view lookup is done via collectionNameResover, ensure updateProperties returns current view properties

* first attempt to fix failing tests

* refactor cluster wide view creation logic

* if view is not found in the new plan then check the old plan too

* ensure the cluster-wide view is looked up in vocbase as well on startup/recovery

* do not store cluster-wide IResearchView in vocbase

* move stale view cleanup to the shared pointer deleter, address test failures

* do not print warning

* enable arangosearch tests by default

* fix catch tests

* address icorrect return value for cluster-wide links

* address some issues with test failures due to cluster-view allocated within TRI_vocbase_t

* simplify per-cid view name, address 'catch' test failures

* ensure IResearchViewNode volatility is properly calculated in cluster

* invoke callbacks directly in AgencyMock instead of waiting for timeout

* ensure view updates via JavaScript always use the latest view definition

* pass a list of shards to `IResearchViewDBServer::snapshot`

* extend cluster aql tests

* fixes after merge

* fix class/struct inconsistencies

* comment failing tests

* remove debug logging

* add debug function

* tests cleanup

* simplify upcoming merge: pass resolver from a side

* backport: move all transaction status callback logic to Methods

* add changes missed from previous commit

* fix js and ruby tests

* more tests for IResearchViewNode

* pass transaction to IResearchViewDBServer::snapshot, address IResearchViewDBServer tests segfault

* pass transaction to IResearchView::snapshot instead of transaction state

* temporarily add trace log output to tests to try to find the cause of the core dump on Jenkins

* add more temporary debug output to trace down the segfault on Jenkins

* add even more temporary debug output to trace down the segfault on Jenkins

* ensure Vieew related maps are cleared during shutdown

* reset ClusterInfo::instance() before DatabaseFeature::unprepare()

* remove extraneous debug output

* missed line from previous commit

* uncomment required line

* add nullptr checks to RocksDBIndexFactory::prepareIndexes(...) similar to the ones in MMFilesIndexFactory::prepareIndexes(...)

* attempt to fix deadlock in tests

* add comment as per reviewer request

* fix aql test suite name

* add some debug logging

* address deadlock between ClusterInfo::loadPlan() and CollectionNameResolver::localNameLookup(...)

* eplicitly state which index definition failed in the log message

* use vocbase from shard-view isntead just in case

* explicitly state which index definition failed in the log message

* do not create shard-view instances from cluster-link instances (only register existing ones)

* add some tests
2018-06-21 20:35:04 +03:00
Vasiliy b7f7711d30 issue 380.4: allow transaction status change and data-source registration callbacks to be registered with a transaction (#5624) 2018-06-18 13:16:32 +02:00
Vasiliy ac1a6742e0 issue 381.4.1: create a separate IResearchLink implementation for use with ClusterEngine (#5505)
* issue 381.4.1: create a separate IResearchLink implementation for use with ClusterEngine

* backport: ensure StorageEngineMock assigns unique transaction ids

* delegate index normalization on the ClusterEngine to the 'actualEngine'
2018-06-01 19:05:53 +03:00