1
0
Fork 0
Commit Graph

90 Commits

Author SHA1 Message Date
Jan 8f1ce5b396
fix issues found by codescan (#7198) 2018-11-02 18:53:34 +01:00
Vasiliy 850919178f issue 496.3: backport 3.4: move more coordinator-related logic out of TRI_vocbase_t, rename some arangosearch view configuration parameters, remove some consolidation policies, update iresearch to revision 6fd9760d81b136f769e277ea5b8f53996ed7a1ca (#7167)
* issue 496.3: backport 3.4: move more coordinator-related logic out of TRI_vocbase_t, rename some arangosearch view configuration parameters, remove some consolidation policies, update iresearch to revision 6fd9760d81b136f769e277ea5b8f53996ed7a1ca

* address merge issue

* backport: remove code causing nullptr access

* invalidate payload for each field in FieldIterator before setting a value

* address compilation issues
2018-11-01 23:12:39 +03:00
Andrey Abramov 8225003861
[3.4] arangosearch speedup removals (#7158)
* Feature/arangosearch speedup removals (#7134)

* speedup document removals and optimize data model

* fix invalid constexpr

* reduce number of heap allocations for removals (#7157)
2018-10-31 15:54:16 +03:00
Vasiliy 2458705607 issue 496.2: backport 3.4: allow customization of segment size limits on arangosearch view creation, minor code cleanup (#7141)
* issue 496.2: backport 3.4: allow customization of segmentsize lmts on arangosearch view creation, minor code clanup

* backport: fix incorrect condition
2018-10-30 23:41:15 +03:00
Vasiliy e6a6025818 backport: switch scope of responsibility between a TRI_vocbase_t and a LogicalView in respect to view creation/deletion (#7106)
* backport: switch scope of responsibility between a TRI_vocbase_t and a LogicalView in respect to view creation/deletion

* backport: ensure arangosearch links get exported in the dump

* backport: ensure view is created during restore on the coordinator

* Updates for ArangoSearch DDL tests, IResearchView unregistration and known issues

* Add fix for internal issue 483
2018-10-30 12:50:29 +03:00
Jan 889cee0dae
speed up query string parsing (#7116) 2018-10-29 14:42:39 +01:00
Andrey Abramov dc98852dda
Bug fix 3.4/internal issue #2921 (#7008)
* issue 494: update iresearch to changeset 6ff909851c760c2111dca5074ab17d35114bee97 (20181022)

* issue 2921: make error message more detailed (#7004)

* issue 2921: make error message more detailed

* upgrade iresearch

# Conflicts:
#	arangod/RestHandler/RestViewHandler.cpp

* fix compilation

* and another fix
2018-10-23 02:41:25 +03:00
Andrey Abramov 2b46285b3b
Feature/backport arangosearch 3.4 (#6985)
* issue 485.1: reduce scope of lock in FlushFeature (#6943)

* issue 493.1: update iresearch to changeset 15f561dee5865ead83663ef056d7bbcc38636fb2 (#6947)

* issue 493.2: trigger termination of long-running consolidation on shutdown request

* Feature/update iresearch 2 (#6984)

* update iresearch sources

* another update

* update iresearch

* fix segfault in log message

* update iresearch

* update iresearch

* update iresearch

* remove noexcept

* improve logging

* update iresearch
2018-10-20 13:24:29 +03:00
Dan Larkin-York fa6680a252 [3.4] Fix issues with ArangoSearch link permissions in cluster mode (#6726) 2018-10-16 20:43:01 +02:00
Andrey Abramov 6b6e4f6279
issue 485: ensure LogicalDataSource::drop() is called on vocbase drop (#6895) (#6923)
* issue 485: ensure LogicalDataSource::drop() is called on vocbase drop

* add missed change

* backport: address race between make(...) and async job

* add another missed change

* backport: ensure recursive lock reports itself as locked correctly

* backport: address test failure on mmfiles

* backport: remove redundant lock already held by async task

* backport: reset reader before unlinking directory
2018-10-16 21:06:47 +03:00
Andrey Abramov 4ef4e1119a
backport new arangosearch to 3.4 (#6874)
* backport new arangosearch

* backport tests

* backport another test

* backport changes in cmake
2018-10-13 01:27:31 +03:00
Dan Larkin-York 3d15b96f66 [3.4] Additional and cleaned-up logging (#6648) 2018-10-01 19:03:05 +02:00
Lars Maier 78f182ebe8 Completely export views dump restore [3.4] (#6484) 2018-09-13 12:37:37 +02:00
Vasiliy 1586b5128e issue 458.3: syntactic cleanup (#6286)
* issue 458.3: syntactic cleanup

* remove extraneous debug output
2018-08-29 17:20:00 +03:00
Andrey Abramov 615e1cf7de
Feature/add options to view (#6255)
* added OPTIONS for views

* pass options too

* validation for OPTIONS

* fix OPTIONS validation

* removed debug log message

* add forceSync options

* add forceSync view option

* fix comments

* fix some memleaks

* fix typo

* fix integration tests

* fix integration tests

* fix another integration test

* rename 'forceSync' option to 'waitForSync'

* another attempt to fix test

* ensure query caches are properly invalidated

* address integration test failures

* another attempt to fix test
2018-08-26 22:30:23 +03:00
Vasiliy 5d14775de8 issue 459.3: ensure collection permissions are checked before updating/dropping an IResearch view (#6253)
* issue 459.3: ensure collection permissions are checked before updating/dropping an IResearch view

* backport: ensure collection permissions are checked before updating/dropping an IResearch view on cluster

* backport: address test failures

* backport: address more test failures

* reuse existing classes for scoping ExecContext
2018-08-26 18:00:16 +03:00
Jan 73e44bc605
invalidate query cache after modification of views (#6198) 2018-08-24 19:54:10 +02:00
Vasiliy b55546225f issue 450.6: simplify consolidation policy typing (#6235) 2018-08-24 01:25:23 +03:00
Vasiliy e04b5318b4 issue 450.5: rename consolidation-related configuration properties and remove locale (#6232) 2018-08-23 16:04:25 +03:00
Vasiliy 61574755b6 issue 450.4: address issues with impropper comparison (#6217) 2018-08-22 15:49:56 +03:00
Vasiliy 325e61b4ed issue 450.3: use a single configuration policy differentiated by 'type' with arbitrary number of configurable parameters (#6205)
* issue 450.3: use a single configuration policy differentiated by 'type' with arbitrary number of configurable parameters

* ensure test data is persisted to the test directory

* add missed declaration

* fix typo

* add full namespace and includes
2018-08-21 14:46:28 +03:00
Vasiliy f7d7ce93fe issue 450.2: remove redundant functions from IResearchView API (#6197) 2018-08-20 16:38:17 +03:00
Vasiliy c08ab01f67 issue 450.1: remove the "commit" configuration layer from ArangoSearch properties (#6185)
* issue 450.1: remove the "commit" configuration layer from ArangoSearch properties

* fix broken tests
2018-08-20 02:18:02 +03:00
Simon 392118bd62 Use RangeDelete where possible (#6121) 2018-08-15 18:52:09 +02:00
Vasiliy 66efaa6022 issue 427.3: uncomment authentication tests, address error propagation issues during view drop (#6103) 2018-08-07 19:52:12 +03:00
Andrey Abramov b8e7752787 Feature/arangosearch stabilization (#6069)
* Added authentication tests for ArangoSearch views creation (client and task)

* Added authentication tests for ArangoSearch views querying (client and task)

* Added authentication tests for ArangoSearch views update (client and task) and correct other tests

* Added authentication tests for ArangoSearch views drop (client and task) and correct other tests

* Fix tests to support additional 'properties' layer of a view

* Add data to dumped/restored collection with a link from a view to single server test and view dump/restore to cluster tests

* backport from iresearch upstream: use a bitvector instead of an acceptor lambda for consolidation policies

* ensure view access permissions are checked for REST calls

* ensure view access permissions are checked for V8 calls

* do not allow LogicalView property modification if reading the properties fails

* Fixed authentication tests for ArangoSearch views to pass regardless of errorNum and possible multiple REST codes

* add missing ldap feature for enterprise version

* do not store useless data in ArangoSearch index (see #289, #441)
2018-08-05 20:47:22 +02:00
Vasiliy 11ba71ec27 issue 432.1: do not modify IResearchView 'locale' encoding and make it non-updatable (#6014)
* issue 432.1: do not modify IResearchView 'locale' encoding and make it non-updatable

* add a temporary workaround for MacOS build issue
2018-07-27 15:08:23 +03:00
Andrey Abramov 71f5e9325b
ensure cleanup policy is triggered (#6008) 2018-07-26 19:19:02 +03:00
Simon 2dd8593609 View Replication (#5915) 2018-07-26 10:28:46 +02:00
Vasiliy ec45ea1101 issue 430.1: add dedicated catch handlers for arangodb::basics::Exception (#5978) 2018-07-25 17:17:56 +03:00
Andrey Abramov 9205e2d9f8
allow user to explicitly track values for "exists" query (#5959)
* allow user to explicitly track values for "exists" query

* fix broken catch tests

* rename view link property

* fix tests after merge
2018-07-25 16:55:02 +03:00
Vasiliy fae47715eb issue 410.3: remove the extra neting of ArangoSearch view definition into 'properties' (#5956)
* issue 410.3: remove the extra neting of ArangoSearch view definition into 'properties'

* address merge issue

* backport: address test failures

* backport: address dump test failure
2018-07-24 16:27:28 +03:00
Vasiliy 3c384bf3e8 issue 410.1: ensure both JavaScript and Rest API for views matches API for collections (#5903)
* issue 410.1: ensure both JavaScript and Rest API for views matches API for collections
* backport: address some FIXMEs
* backport: address test failures
* backport: address more test failures
* backport: fix some Ruby tests
* temporarily comment out broken Ruby tests
* backport: fix more broken Ruby tests
* temporarily comment out failing Ruby tests
* backport: match result structure for View instances with Collection instances
* backport: return different result structure for view listing and view definition
* backport: fix even more broken Ruby tests
* backport: do not depend on unordered_set stable order
* update documentation
2018-07-19 15:07:17 +02:00
Vasiliy a401934d62 issue 399.3: remove internal persisted proerty 'collections' from externally generated IResearchView jSON (#5874)
* issue 399.3: remove internal persisted proerty 'collections' from externally generated IResearchView jSON

* backport: address test failure

* backport: address another test failure
2018-07-17 15:21:07 +03:00
Simran 34ec56d421 Feature/misc spelling corrections (#5164) 2018-07-13 13:06:20 +02:00
Vasiliy 6ba3d98e39 issue 400.3: remove redundant asynchronous execution parameter (#5852) 2018-07-12 17:46:30 +03:00
Vasiliy 57d2517bd6 issue 399.1: store only transient link information by IResearchViewCoordinator (#5834)
* issue 399.1: store only transient link information by IResearchViewCoordinator

* some fixes
2018-07-11 21:21:51 +03:00
Vasiliy 64666d0edd issue 400.2.4: register IResearchView sync tasks with IResearchFeature, remove unused configuration parameter (#5813)
* issue 400.2.4: register IResearchView sync tasks with IResearchFeature, remove unused configuration parameter

* set flag before notify, fix failures in new tests from 'devel'

* revert set flag before notify, fix more failures in new tests from 'devel'

* fix even more failures in new tests from 'devel', backport: do not reset notification flag if pending tasks were seen

* fix still more failures in new tests from 'devel'

* try to work around buggy test failures
2018-07-10 18:15:01 +03:00
Simon f47367a495 Allow dumping / restoring of views (#5774)
* allow dumping and restoring of views

* fix restore, adding some tests

* remove debug print

* fix global replication
2018-07-05 17:55:28 +02:00
Vasiliy 9c04a6059c issue 401.1.1: implement an asynchronous task execution pool (thread pool) in IResearchFeature (#5761) 2018-07-04 15:18:25 +03:00
Vasiliy 7aaeab50fb issue 402.1: share sync thread between IResearchView and IResearchViewDBServer (#5733) 2018-07-02 15:03:00 +03:00
Vasiliy bf1f2eab29 issue 401.2: split IResearchViewMeta into the user configurable part and internal state (#5683)
* issue 401.2: split IResearchViewMeta into the user configurable part and internal state

* backport: update documentation and integration tests to reflect the IResearchViewMeta changes

* backport: addresss typo

* backport: address another typo
2018-06-27 16:40:51 +03:00
Vasiliy 532b4e613e issue 401.1: move IResearchView sync thread logic into its own class (#5668)
* issue 401.1: move IResearchView sync thread logic into its own class

* backport: use arangodb::Thread for ArangoSearch Sync
2018-06-26 15:47:56 +03:00
Andrey Abramov 5eef6cd618
Feature/test iresearch (#5610)
* start implementing arangosearch cluster tests.

* backport: ensure view lookup is done via collectionNameResover, ensure updateProperties returns current view properties

* first attempt to fix failing tests

* refactor cluster wide view creation logic

* if view is not found in the new plan then check the old plan too

* ensure the cluster-wide view is looked up in vocbase as well on startup/recovery

* do not store cluster-wide IResearchView in vocbase

* move stale view cleanup to the shared pointer deleter, address test failures

* do not print warning

* enable arangosearch tests by default

* fix catch tests

* address icorrect return value for cluster-wide links

* address some issues with test failures due to cluster-view allocated within TRI_vocbase_t

* simplify per-cid view name, address 'catch' test failures

* ensure IResearchViewNode volatility is properly calculated in cluster

* invoke callbacks directly in AgencyMock instead of waiting for timeout

* ensure view updates via JavaScript always use the latest view definition

* pass a list of shards to `IResearchViewDBServer::snapshot`

* extend cluster aql tests

* fixes after merge

* fix class/struct inconsistencies

* comment failing tests

* remove debug logging

* add debug function

* tests cleanup

* simplify upcoming merge: pass resolver from a side

* backport: move all transaction status callback logic to Methods

* add changes missed from previous commit

* fix js and ruby tests

* more tests for IResearchViewNode

* pass transaction to IResearchViewDBServer::snapshot, address IResearchViewDBServer tests segfault

* pass transaction to IResearchView::snapshot instead of transaction state

* temporarily add trace log output to tests to try to find the cause of the core dump on Jenkins

* add more temporary debug output to trace down the segfault on Jenkins

* add even more temporary debug output to trace down the segfault on Jenkins

* ensure Vieew related maps are cleared during shutdown

* reset ClusterInfo::instance() before DatabaseFeature::unprepare()

* remove extraneous debug output

* missed line from previous commit

* uncomment required line

* add nullptr checks to RocksDBIndexFactory::prepareIndexes(...) similar to the ones in MMFilesIndexFactory::prepareIndexes(...)

* attempt to fix deadlock in tests

* add comment as per reviewer request

* fix aql test suite name

* add some debug logging

* address deadlock between ClusterInfo::loadPlan() and CollectionNameResolver::localNameLookup(...)

* eplicitly state which index definition failed in the log message

* use vocbase from shard-view isntead just in case

* explicitly state which index definition failed in the log message

* do not create shard-view instances from cluster-link instances (only register existing ones)

* add some tests
2018-06-21 20:35:04 +03:00
Vasiliy b7f7711d30 issue 380.4: allow transaction status change and data-source registration callbacks to be registered with a transaction (#5624) 2018-06-18 13:16:32 +02:00
Simon 3bec336aff TransactionState::addCollection refactoring (#5606) 2018-06-14 15:34:58 +02:00
Vasiliy a5f274234a issue 380.2.2: remove extraneous pointers from StorageEngine API (#5589) 2018-06-13 15:22:51 +03:00
Vasiliy 1bd4947629 issue 381.7: move per-transaction IResearchView state from the view into the TransactionState (#5530) 2018-06-06 18:35:15 +03:00
Vasiliy ac1a6742e0 issue 381.4.1: create a separate IResearchLink implementation for use with ClusterEngine (#5505)
* issue 381.4.1: create a separate IResearchLink implementation for use with ClusterEngine

* backport: ensure StorageEngineMock assigns unique transaction ids

* delegate index normalization on the ClusterEngine to the 'actualEngine'
2018-06-01 19:05:53 +03:00
Vasiliy e021085aac issue 381.3.1: remove redundant copy of CollectionNameResolver from TransactionState (#5459) 2018-05-25 16:48:43 +03:00