* integrate ArangoSearch into GatherNode/GatherBlock logic
* fix inconsistency after merge
* add serialization/deserealization logic for IResearchViewNode sorting
* fix optimizer rule order
* fix copy/past typo
* fix tests
* ensure `GatherNode` is properly set up for ArangoSearch views
* adjust optimizer rule order
* further implementation of arangosearch query distribution
* do not store `aql::Collection` in `IResearchViewNode`, use `Query::collections()` instead
* some fixes
* cleanup
* some fixes after review
* more fixes after review
* use assertion instead of condition
* issue 373.2: move toVelocyPack into LogicalDataSource
* backport: move static DataSource related strings into StaticStrings, add support for registering snapshots on DBServer views
* add initial implementation of scatter view rule and node
* add tests for `IResearchViewNode` and `IResearchViewScatterNode`
* add missing check
* modify IResearch execution nodes to use references instead of pointers
* use view id in searialized `ExecutionNode` representation instead of the name
* add cluster mode stubs and checks
* very first attempt to distribute IResearchViewNode
* further implementation of cluster-wide arangosearch views
* fix invalid json format
* add tests for coordinator iresearch view
* allow to retrieve a list of existing views on a coordinator
* more tests for coordinator iresearch view
* some fixes to enable query explanation
* remove Collection dependency from RemoteNode
* remove unnecessary remote ArangoSearch view scatter
* fix explanation appearance
* add some assertions
* minor fixes
* implement IResearchViewCoordinator::updateProperties
* fix view DDL issues
* handle link modifications in DDL operations
* add coordinator implementation of iresearch view links
* fix tests
* further coordinator based view DDL implementation
* further IResearchViewCoordinator implementation
* add initial implementation of AgencyMock
* fix some tests
* code cleanup
* extend test + some fixes
* more tests for IResearchViewCoordinator
* fix tests for IResearchLinkCoordinator
* some fixes after merge
* fix tests
* remove declaration of nonexistent (previously removed) method
* some fixes after review
* remove string duplication
* more tests and fixes
* more fixes and tests
* more tests
* one more test
* fix 'use-after-free' asan error
* fix non-enterprise tests issues
* Started Implementing the ServerBasedlocking. There now is a container that can contain multiple query snippets. It now has to setup the necessary calls to the Servers
* Added backwards linking of QueryEngines, sth. DBServers can contact their Coordinators.
* Added LogTopic AQL
* Made AccessMode::Type Hashable
* Created a Mapping Server => LockLevel => Shard and createad a JSON object containing the Lock information for a complete AQL query per server
* Added code to build coordinator engines
* Finished with first draft of Coordinator-side of new DBServer based locking.
* Added a _api/aql/setup route that creates and locks all snippets/collections for one DBServer in a single go
* Fixed some Coordinator parts
* Index node now gracefully reports if it could not find it's collection when created from vpack. Otherwise it just hardly crashed...
* Modified the Coordinator Snippet collector to be able to handle subqueries properly.
* Started adding GraphNode handling. WIP. Need to deploy engines properly. Coordinator crashes on Graph tests
* Fixed compiler errors
* WIP: EngineInfoContainer
* Separated the EngineInfoContainers for Coordinator and DBServer into different files. They diverged more than anticipated
* Added forgotten files. THe DBServer container now creates the TraverserEngine Mapping and moves it into the Infos. They are not keeping it yet and need to add it to the message as well.
* The DBServer engine infos now persist the TraverserEngine infos. Need to add them to messages though.
* The new aql exec-engine now sends out traverserEngines as well
* Formatting and adding DEBUG level output
* Made the RestAQLHandler aware of the TraverserEngineRegistry. Also created the engines now. Return format changed server-side coordinator side needs fix.
* Adapted the Coordinator side for the DBServer based Shard Locking
* The DBServer based Locking now honors restrictions to certain shards
* Fixed a strange double lock bug in the new AQL Server based locking technique. Add some DEBUG output
* Fixed usage of MAINTAINERMODE macro. The assertion was never active
* Added TestCase for ContainerCoordinatorTest to cmake
* Added -DTEST_VIRTUAL to CMAKE. This is used to define virtual functions for mocking ONLY on test-builds.
* Fixed usage of ENABLE_MAINTAINER_MODE ifdef. CLANG format
* On non-enterprise builds ENTERPRISE_VERT defaults to TEST_VIRTUAL => virtual in test else non-virtual
* Added TEST_VIRTUAL to ExecutionEngine, Query and QueryRegistry
* Added first testcase for EngineInfoContainerCoordinator not yet ready.
* Mode CreateBlock a member function of engine, we have the engine in our hands anyways no need to make it static. Included some more TEST_VIRTUAL functions.
* Fixed clang/MacOs compile error. Added some more TEST_VIRTUAL declarations
* Finally fixed the first buildEngines UnitTest \o/
* Added a unit-test for backward linking of dependencies in CoordinatorPlanner
* Added multi-snippet test for EngineInfoContainerCoordinator
* Removed QueryRegistry.h from central header files and replaced by a forward declartion.
* Added a createBlocks method on the ExecutionEngine. It should be responsible to create all those blocks at once. Adapted the UnitTests as well. Not included Tests for the new createBlocks functionality. Need to mock the options feature first
* Added another test that Coordinator Snippets of queries can be created correctly
* Fixed Coordinator-site cleanup of QueryRegistry, if any of the query creations fails with error, incl UnitTest.
* Added first test for RestAqlHandler::setup. It does only test the setup and gives prepartion for real testing.
* Added a assertion of http return code. Still no creation of queries is tested. Requires a huge amount of mocking.
* fix some deadlocks found by evil lock manager (tm)
* fix duplicate lock
* fix indentation
* ensure proper lock dependencies
* fix lock acquisition
* removed useless comment
* do not lock twice
* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not
* AQL micro optimizations
* use explicit constructor
* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context
entering and exiting the selected context is then done later on without having to hold the ConditionLocker
* remove some recursive locks
* Disable custom deadlock detection when Thread Sanitizer is enabled
* Changing ifdef's
* grr
* broke gcc
* Using atomic for ApplicationServer::_server
* fix premature unlock
* add some asserts
* honor collection locking in cluster
* yet one more lock fix
* removed assertion
* Allow the clustercomm to send nolock headers on count. This is used form within AQL
* IsLocked on transactions will now always yield true IF LOCK_NEVER is set. We simply assume someone else holds the lock for us. Also LOCK_NEVER is now set on collection/count if noLock header is send.
* Moved the flag if collections need to be locked into the TraverserEngines.
* Added enterprise-satellite hooks in EngineInfoContainerDBServer
* Removed now obsolete code
* Replaced throwing of Exception by an ResultObject
* Added some more tests and moved adding snippet to query engine more to the outside.
* Added the AQL result type
* Make the branch compile again
* Register WITH collections for Graphs in the new Collector.
* Fixed test code for failing query clone. Idea was to once clone successfully and second time to fail, we verify that first clone is cleaned up properly. However test failed on first clone...
* Removed a double builderClose
* Added Changelog entry
* Removed empty if
* Removed obsolete todo
* Properly initialize the AqlResult with nullptr on error case
* Updated comment
* Simplified Assertion
* Removed debug output object...
* Added additional catch case for std::exception to get some more error info
* Clarified evaluation order for move case
* Added Explicit
* Fixed cleanup of Coordinator if Registry fails to insert query.
* Allow to use other locks than Read/Write for AQL collections. Not yet in API.
* Updated Comments for other Locks on DBSide. Adapted Destruction CatchTests
* Fixed double builderClose and removed unnecessary double commits
* Added a comment to clarify the state
* Moved error output to trace. Leftover from debugging
* Added some tests for complex subquery patterns
* Added a 'fireAndForgetRequests' methods to cluster comm which allows to send out a bunch of messages but does not wait for their results
* Properly cleanup leftovers of queries if the instantiation step already failed
* Added code comment for fireAndForget
* Added indexes to subquery test to make the plan a bit easier
* The cleanup on DBServerEngines in error case now also cleans up traverser engines.
* Removed unnecessary includes
* Removed debug logging
* Fixed hidden merge conflict
* Use std::this_thread::sleep_for to fix windows compile, mac compile
* one more place to swap the type
* fix type forward declaration
* fix windows warnings
* fix windows warnings
* Changes since last PR: use LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups
* backport: address cluster LogicalDataSource resolution failure
* Cherry Picking LDAP changes
* Adding missing merges
* Fixing remaining mentions of FeatureCacheFeature
* Fix jslint
* Fixing some failed tests
* Fixing cluster authentication issue, red tests
* Fixing ldap testsuite, adding trace logging
* Fixint ldap tesuite setup and LDAP recognition
(cherry picked from commit 686d28a779)
* Fixing wrong assert
* Adding changelog entry, making requested changes from code review
* Fixing dump_authentication, fix typos
* improvements found during code review
* oops
* more use of sessionstorage
* fix tests
* Fixing broken handling, disallowing adding of local users when disabled
* Fixing testInvalidGrants
* Removing undefined auth level externally
* Fixing previous commit
* added tests for ldap search mode
* intentionally removed `after` methods from tests
because they are executed before the tests start
no cleanup is performed right now after the authentication tests
however, a cleanup is done at start of every test
* ldap tests all modes
* forward port changes from 3.3
* added generated files
* forward port missing changes for web UI
* added generated files
* added generated files
* fix some deadlocks found by evil lock manager (tm)
* fix duplicate lock
* fix indentation
* ensure proper lock dependencies
* fix lock acquisition
* removed useless comment
* do not lock twice
* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not
* AQL micro optimizations
* use explicit constructor
* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context
entering and exiting the selected context is then done later on without having to hold the ConditionLocker
* remove some recursive locks
* Disable custom deadlock detection when Thread Sanitizer is enabled
* Changing ifdef's
* grr
* broke gcc
* Using atomic for ApplicationServer::_server
* fix premature unlock
* add some asserts
* honor collection locking in cluster
* yet one more lock fix
* removed assertion
* some more bugfixes
* Fixing assert
(cherry picked from commit 1155df173bfb67303077fbe04ee8d909517bfd21)
* Special method to access configured access levels as opposed to only getting the effective access level
* Add integration test
* Allow grants on collection level
* Switching from ttl to supervision based failover mechanism
* Allowing canceling of ongoing actions
* refactored asyncjobmanager
* refactoring some code
* adding read-only flag
* Fixing some bugs
* Fixing tests
* Canceling ongoing operations
* removing some unused code + some asserts
* catching some exceptions to reduce log pollution, removing unnecessary code, removing tests for _changeMode
* fixing "createsANewDatabaseWithAnInvalidUser"
* Current work
* proper ifs
* Migrate resthandler to c++ and implement mode
* readonly server mode spec test
* Add changelog
* change code so it expects a full object and not just a string
* Fix jslint
* Added a more sophisticated test for shardDistribution format
* Updated shard distribution test to use request instead of download
* Added a cxx reporter for the shard distribuation. WIP
* Added some virtual functions/classes for Mocking
* Added a unittest for the new CXX ShardDistribution Reporter.
* The ShardDsitributionReporter now reports Plan and Current correctly. However it does not dare to find a good total/current value and just returns a default. Hence these tests are still red
* Shard distribution now uses the cxx variant
* The ShardDistribution reporter now tries to execute count on the shards
* Updated changelog
* Added error case tests. If the servers time out the mechanism will stop bothering after two seconds and just report default values.
* remove some non-unused V8 persistents
* do not throw that many bogus assertions
* do not rely on server role being defined
* slightly better debug output for V8 context debugging
* fix collection ids in inventory response
* simplify bootstrap a bit
* slightly better error handling
* make elapsed time a queryable value
* use less memory for stub collections
* added assertions that will always make sense
* added assertions
* do not garbage-collect while waiting
* less copying of parameters
* do not show "load indexes into memory" buttons for mmfiles engine
as all indexes are in memory anyway
* when a collection is truncated via the web interface, flush the WAL and rotate all active journals
this will make close all open journals on leader and followers and make them subject to compaction opportunities
* fix invalid server id values being passed from web interface to backend
* introduce afterTruncate method for indexes
* added test case for issue #3447
* updated CHANGELOG
* don't warn about replicationFactor for system collections
* check that the queries actually use the geo index and not some other index
* properly report error in web interface
* fix some internals checks that made truncate fail for bigger collections in maintainer mode
* also run a compact() operation after a serious truncate
in order to make iteration over the truncated range much faster
when the collection is next accessed
* increase default maximum number of V8 contexts to at least 16
* Switching from ttl to supervision based failover mechanism
* Allowing canceling of ongoing actions
* refactored asyncjobmanager
* refactoring some code
* adding read-only flag
* catching some exceptions to reduce log pollution, removing unnecessary code, removing tests for _changeMode
* fixing "createsANewDatabaseWithAnInvalidUser"
* auth = off does not longer make everyone superuser
* Fixing cluster_sync and maybe resilience
* properly initialize all properties
* use faster comparison
* properly detect and handle "method not allowed"
* code-style
* remove unused variable
* narrow variable scope
* handle non-existance of AuthenticationFeature
* remove dead code
* replace some C string handling with std::strings
* moved assertion to the correct place
* honor number of array members for IN operator
* slightly adjust error messages
* slighty adjust some error messages
* try to fix issue with lingering replication contexts on shutdown
* clean up heartbeat thread a little bit
* small fixes
* remove memory zone remainders
* remove useless calls
* attempt to fix non-deterministic CRC test
* attempt to fix non-deterministic replication tests
* fix document removal with multiple documents in cluster
* attempt to fix non-determinism of tests
* fix privilege checks for accessing collections by collection id, not by name
[arangod/GeneralServer/AuthenticationFeature.cpp:42]: (warning) Member variable 'AuthenticationFeature::_authenticationTimeout' is not initialized in the constructor.
[arangod/Utils/Authentication.h:49]: (performance) Function parameter 'roles' should be passed by reference.
[arangod/VocBase/AuthInfo.h:47]: (warning) Member variable 'AuthResult::_expiry' is not initialized in the constructor.
* Make isRestore work in the cluster.
This covers sharded collections with default sharding and non-default
sharding.
* always use locally generate revision ids for storing and looking up documents
* do not use V8 variant of AQL functions in early optimization stage when a C++ variant is available
* additionally, simplify AQL function definitions and aliases
* warn when more than 90% of max mappings are in use
* added C++ variant of replication catchup
* added `--log.role` option
* updated CHANGELOG
* removed non-existing scheduler.threads option from config
* removed useless __FILE__, __LINE__ invocations
* updated CHANGELOG
* allow a priority V8 context
* remove TRI_CORE_MEM_ZONE
* try to fix Windows errors & warnings
* cleanup
* removed memory zones altogether
* exclude system collections from collection tests
* Get rid of a compiler warning in community edition.
* Teach /_api/replication/clusterInventory to report Plan/Version and readiness.
This is first implemented in the ClusterInfo library.
Then the clusterInventory code uses it and checks readiness.
Readiness of a collection means that it is created and all shards
and all replications have been created and are in sync.
* added V8 context lifetime control options `--javascript.v8-contexts-max-invocations` and `--javascript.v8-contexts-max-age`
* make thread scheduling take into account most of the tasks dispatched via the io service
* Added virtual attributes for enterprise on Methods.cpp
* Working no access collections
* align a comment
* Documentation and test fix
* fixing community build
* removed delete call
* cleanup
* lower cpu activity of log thread too
* fix log messages
* do not enter threads into unordered_set, as it is unneeded
* do not compile in calls to disabled plan cache
* moved AQL regex cache from thread local variables to a class of its own
* more sensible thread creation and destruction
* remove API docs for endpoints that were deleted in 3.0
* remove unused variable
* set all scalar values in the ctor, so check tools do not complain
* move Result objects
* prevent potential nullptr access
* remove ambiguous boolean parameter with no clear meaning
* make code check tools happy
* don't allow copying cache Findings, allow moving Results
* the debug functionality for counting RocksDB key ranges was broken
this produced sporadic test failures
* remove superfluous iostream include
* fix range checks
* don't leak if an error happens during database removal
* try to simplify Result a bit
* try to deletion checks a bit more robust
* Count as checksum
* Make readLockId optional as well so upgrades are possible
* fix option name in startup script
* fix some replication issues with RocksDB engine
* simplify index API a bit
* fix fulltext index removal performance, simplified code
* updated CHANGELOG
* fix hanging test
* try to fix shutdown problem
* improve fulltext query performance
* fixed duplicate var
* removed obsolete code
* fix some shutdown races
* do not call ensureIndex that often
* implement transaction handler in c++
* add fake request object
* move general code into extra file
* improve error messages for RestJobHandler
* support cancel operation for async jobs with RestTransactionHandler
* fix integration test and add some optional debugging output
* fix returning of context
* fix error type
* fix code in cluster test
* remove unused variable
* ignore some return codes when closing zip files and do not report them
* hide mostly useless debug message
* clear basic authentication cache after deletion of users and after updating them
otherwise deleted/changed users can still access the database!
* adjust wording
* added notes about mmfiles-specific parameters
* updated CHANGELOG and documentation
* reduce extractions to projections
* recycle string buffers in SocketTask
* micro optimizations for mmfiles indexes
* added special lookup function for _key
* moved function into the correct file
* speed up key buffer allocations a bit
* added noexcept specifier
* correctly name variable
* explicitly move bounds
* fix and speedup from/toPersistent functions
* reuse string from ManagedDocumentResult for multiple lookups
* use move-assign
* a bit less work for single server
* speedup AQL function HASH
* single fetch optimization
* performance optimization for the case when no documents need to be returned
* make reduce-extraction-to-projection a RocksDB-only optimizer rule
* cppcheck
* try to fix compile error on MacOS
* bug fix for MacOSX
* missing namespace (in Windows compile)
* reduce extractions to projections
* recycle string buffers in SocketTask
* micro optimizations for mmfiles indexes
* added special lookup function for _key
* moved function into the correct file
* speed up key buffer allocations a bit
* added noexcept specifier
* correctly name variable
* explicitly move bounds
* fix and speedup from/toPersistent functions
* reuse string from ManagedDocumentResult for multiple lookups
* use move-assign
* a bit less work for single server
* speedup AQL function HASH
* single fetch optimization
* performance optimization for the case when no documents need to be returned
* make reduce-extraction-to-projection a RocksDB-only optimizer rule
* Added a first test for remove-filter-covered-by-traverser rule.
* initial try at adding the functions
* Initial try at fixing all cases
* Fixing optimizer rule order
* Changing rule
* excluding quantifier ANY
* Fixing assertion error
* quick fix for some failed traverser cases
* Fixing quantifier related tests
* Adding upper limit for scannedIndex in cluster
* Fixing the mmfiles asserts
* Allow the coordinator to do condition checking on edges again. Used in enterprise.
* Fix naming of optimizer rules (#2739)
* Add testcase to demonstrate #2499
- In subqueries inserts get pushed to the DB-Servers
- these create documents documents on each DB-Server that don't respect the shard-key
- subsequent deletes will fail to detect the shard correctly and will also fail.
* fix for #2499
* working documents rest handler
* fixed cluster tests
* Consolidating database APIs
* clang-format
* Fixing issue with user creation through db._createDatabase
* replaced and refactored api-index and index api
* fixed cluster
* renaming some files
* added user methods
* removed files intended for later
* Fixed CC build
* Fixed method signature
* Fixing shell_server, shell_client tests
Heiko
Vasiliy
Andrey Abramov
Jan
Jan Christoph Uhde
Simon
jsteemann
Andrey Abramov
Max Neunhoeffer
Michael Hackstein
Wilfried Goesgens
m0ppers
Frank Celler
Manuel B
Kaveh Vahedipour
Jan Christoph Uhde
Dan Larkin
Dan Larkin
baslr