* Bug fix 3.5/make arangosh reconnect (#9615)
* make arangosh reconnect
* added CHANGELOG entry
* fix lagging AgencyCallbacks (#9620)
* fix lagging AgencyCallbacks
* optimizations, discussed with @mchacki
* fix wording
* updated CHANGELOG
* fix yet another undefined behavior (#9629)
* [3.5.1] Fail the FailedLeader Job if the new leader fails. (#9628)
* Fail the FailedLeader Job if the new leader fails.
* Updated changelog.
* In case of timeout do not rollback.
* Fixed catch tests.
* Changed wording.
* DELETED rollback.
* reduce wait timeouts as a mitigation for notifying waiters without ho… (#9619)
* reduce wait timeouts as a mitigation for notifying waiters without holding the required mutex
this is a quick mitigation only, which reduces maximum wait time from 1
second to 100 milliseconds without changing other behavior.
the main problem of notifying pending writers without successfully
acquiring the required mutex still needs proper addressing.
* adjust timing-dependent test
* [3.5.1] Fast Controlled Leaderchange (#9634)
* First draft of keeping in sync during controlled leader change.
* Test if server is actually the leader in plan.
* Updated changelog.
* Added oldLeader check for set-the-leader request.
* Small fixes.
* Removed LOG_DEVEL.
* less copying, more moving! 🚚 (#9645)
* attempt to fix load_balancing tests in slow test environments (#9626)
* Bug fix/fix swagger datatype (#9045) (#9602)
* Bug fix/fix swagger datatype (#9045)
* remove http so https arangos will work
* verify that query parameters are proper swagger data types, fix offending documentation files
* return the actual type - not the list of available ones
* check formats
* there is no uint64 in swagger
* Fresh Swagger
* Port TakeoverShardLeadership from devel to 3.5.1 (#9659)
* Create TakeoverShardLeader job.
* Add TakeoverShardLeadership to Action factory.
* Add log message at level debug.
* Sort out LOG_TOPIC ids.
* Fix unit tests.
* CHANGELOG.
* Bug fix 3.5/hide mmfiles specific info in web ui (#9668)
* attempt to fix load_balancing tests in slow test environments (#9626)
* Bug fix/fix swagger datatype (#9045) (#9602)
* Bug fix/fix swagger datatype (#9045)
* remove http so https arangos will work
* verify that query parameters are proper swagger data types, fix offending documentation files
* return the actual type - not the list of available ones
* check formats
* there is no uint64 in swagger
* Fresh Swagger
* hide MMFiles-specific information when we don't need it
* Ported ResignLeadership to 3.5 (#9656)
* attempt to fix load_balancing tests in slow test environments (#9626)
* Bug fix/fix swagger datatype (#9045) (#9602)
* Bug fix/fix swagger datatype (#9045)
* remove http so https arangos will work
* verify that query parameters are proper swagger data types, fix offending documentation files
* return the actual type - not the list of available ones
* check formats
* there is no uint64 in swagger
* Fresh Swagger
* Ported ResignLeadership to 3.5
* Add the actual http route.
* Aardvark: Add k Shortest Paths example graph to UI (#9491) (#9661)
* Aardvark: Add k Shortest Paths example graph to UI (#9491)
* Add example graph to UI
* Add kShortestPathsGraph to examples.js
* Update example-graph.js
* Update aardvark.js
* Regenerate UI
* add the ability to have cluster special examples (#9613) (#9663)
* add the ability to have cluster special examples
* Update get_cluster_health.md
* fix abort condition, fix negative filtering for cluster tests
* Test if job fails with unmet assertion
* Remove cluster test example
* germanize
* better skip reasons
* removing superfluous semicolons
* Revert skip reasons, too noisy
* various replication improvements: (#9675)
* attempt to fix load_balancing tests in slow test environments (#9626)
* Bug fix/fix swagger datatype (#9045) (#9602)
* Bug fix/fix swagger datatype (#9045)
* remove http so https arangos will work
* verify that query parameters are proper swagger data types, fix offending documentation files
* return the actual type - not the list of available ones
* check formats
* there is no uint64 in swagger
* Fresh Swagger
* various replication improvements:
- better debuggability (more log details)
- shorter minimum wait delay in active failover
- fixed too early pruning of WAL files on leaders
* Bug fix 3.5/fix rocksdb return code (#9692)
* attempt to fix load_balancing tests in slow test environments (#9626)
* Bug fix/fix swagger datatype (#9045) (#9602)
* Bug fix/fix swagger datatype (#9045)
* remove http so https arangos will work
* verify that query parameters are proper swagger data types, fix offending documentation files
* return the actual type - not the list of available ones
* check formats
* there is no uint64 in swagger
* Fresh Swagger
* fix return codes for concurrent writes to same documents
* [3.5] Feature/rebootid notice changes, backport of #9523 (#9684)
* Feature/rebootid notice changes, backport of #9523
* Fixed error code to not re-use an old one
* Bug fix 3.5/issue 9679 (#9682)
* attempt to fix load_balancing tests in slow test environments (#9626)
* Bug fix/fix swagger datatype (#9045) (#9602)
* Bug fix/fix swagger datatype (#9045)
* remove http so https arangos will work
* verify that query parameters are proper swagger data types, fix offending documentation files
* return the actual type - not the list of available ones
* check formats
* there is no uint64 in swagger
* Fresh Swagger
* fixed issue #9679
* bug-fix/issue-#9660 (#9704) (#9707)
* bug-fix/issue-#9660 (#9704)
* fix issue
* Update tests/js/common/aql/aql-view-arangosearch-cluster.inc
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update tests/js/common/aql/aql-view-arangosearch-noncluster.js
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* fix cluster tests
* Update CHANGELOG
* [3.5] agency node fixes (#9698)
* node fixes port from 3.4
* fixed change log
* update rocksdb statistics to deliver sums from column family instead of single value from default family. (#9706)
* Feature 3.5/geo functions (#9710)
* Add support for WGS84 on distances (#9672)
* Add area calculations (#9693)
* Update CHANGELOG
* Use int type for server id
Change serverId to an int
Pass syncerId only for synchronous replication
Added UrlBuilder
structs to classes, reordering
Added Location class, cleanup
Fixed initialization order
Use Location class
Use string for large ints
Documentation
Added clientInfo to ReplicationClientProgressTracker and corresponding rest handlers
Pass clientInfo string in sync replication
Pass clientInfo in addFollower, too
Updated docu
Renamed UrlBuilder to UrlHelper
Updated docu
Try to fix compile error on windows
Fixed a bug and a test
* Implemented @jsteeman's comments
* Decoupled IO handling from Scheduler.
* Fixed SSL start up bug.
* Replaced Scheduler with new worker farm implementation.
* Added minimal statistics and info string for Scheduler.
* Added support for timed submissions.
* Updated delayed submission api. Updated code that used timers.
* Extracted new Scheduler into a virtual parent class. The implementation can now depend on the usecase.
* Signal handler now working.
* Changed threads names, `_stop` is atomic, check for failure during thread start + exception handling like old scheduler did.
* Commented on source code and added TODOs.
* Played around with start-stop-conditions
* Play around with start stop condition.
* start stop cond
* Sart Stop Conditions
* Removed bad cv_status check.
* Bug fix: now compare the actual objects instead of pointer values. Setup t1 and t2 depending on the thread id.
* Moved most of the stuff now unrelated to the Scheduler to GeneralServer. Got rid of JobGuard.
* Instead of waiting for a thread to terminate, put it on a clean up list and check for its termination in each supervisor run.
* Allow detaching long running threads.
* Fixed test mock.
* Updated the WorkHandle logic. Removed post functions.
* Fixed crash when obtaining shared_ptr from this in destructor.
* Added lost mutex.
* Fixed memory leak.
* Fixed merge bug.
* Changed a lot of code to optimize the scheduler.
* Fixed bug of invalidated iterator. Dont remove task on shutdown at different places. Let scheduler threads run until queue is empty.
* Only by value calls to queue.
* Added options again.
* Clean up of code.
* UI Request Lane added.
* Bug fixes in Scheduler.
* Applied reformat.
* Use sigaction.
* issue 496.3: move more coordinator-related logic out of TRI_vocbase_t, rename some arangosearch view configuration parameters, remove some consolidation policies, update iresearch to revision 6fd9760d81b136f769e277ea5b8f53996ed7a1ca
* address potential deadlock between link creation and FlushThread
* remove code causing nullptr access
* add back lock around reader reopen
* revert: address potential deadlock between link creation and FlushThread
* invalidate payload for each field in FieldIterator before setting a value
Sleeping for shorter periods increases the chances that we can continue faster...
This will speed up the initial synchronization when collections contain only few or no documents,
but there are lots of collections to sync
* Started Implementing the ServerBasedlocking. There now is a container that can contain multiple query snippets. It now has to setup the necessary calls to the Servers
* Added backwards linking of QueryEngines, sth. DBServers can contact their Coordinators.
* Added LogTopic AQL
* Made AccessMode::Type Hashable
* Created a Mapping Server => LockLevel => Shard and createad a JSON object containing the Lock information for a complete AQL query per server
* Added code to build coordinator engines
* Finished with first draft of Coordinator-side of new DBServer based locking.
* Added a _api/aql/setup route that creates and locks all snippets/collections for one DBServer in a single go
* Fixed some Coordinator parts
* Index node now gracefully reports if it could not find it's collection when created from vpack. Otherwise it just hardly crashed...
* Modified the Coordinator Snippet collector to be able to handle subqueries properly.
* Started adding GraphNode handling. WIP. Need to deploy engines properly. Coordinator crashes on Graph tests
* Fixed compiler errors
* WIP: EngineInfoContainer
* Separated the EngineInfoContainers for Coordinator and DBServer into different files. They diverged more than anticipated
* Added forgotten files. THe DBServer container now creates the TraverserEngine Mapping and moves it into the Infos. They are not keeping it yet and need to add it to the message as well.
* The DBServer engine infos now persist the TraverserEngine infos. Need to add them to messages though.
* The new aql exec-engine now sends out traverserEngines as well
* Formatting and adding DEBUG level output
* Made the RestAQLHandler aware of the TraverserEngineRegistry. Also created the engines now. Return format changed server-side coordinator side needs fix.
* Adapted the Coordinator side for the DBServer based Shard Locking
* The DBServer based Locking now honors restrictions to certain shards
* Fixed a strange double lock bug in the new AQL Server based locking technique. Add some DEBUG output
* Fixed usage of MAINTAINERMODE macro. The assertion was never active
* Added TestCase for ContainerCoordinatorTest to cmake
* Added -DTEST_VIRTUAL to CMAKE. This is used to define virtual functions for mocking ONLY on test-builds.
* Fixed usage of ENABLE_MAINTAINER_MODE ifdef. CLANG format
* On non-enterprise builds ENTERPRISE_VERT defaults to TEST_VIRTUAL => virtual in test else non-virtual
* Added TEST_VIRTUAL to ExecutionEngine, Query and QueryRegistry
* Added first testcase for EngineInfoContainerCoordinator not yet ready.
* Mode CreateBlock a member function of engine, we have the engine in our hands anyways no need to make it static. Included some more TEST_VIRTUAL functions.
* Fixed clang/MacOs compile error. Added some more TEST_VIRTUAL declarations
* Finally fixed the first buildEngines UnitTest \o/
* Added a unit-test for backward linking of dependencies in CoordinatorPlanner
* Added multi-snippet test for EngineInfoContainerCoordinator
* Removed QueryRegistry.h from central header files and replaced by a forward declartion.
* Added a createBlocks method on the ExecutionEngine. It should be responsible to create all those blocks at once. Adapted the UnitTests as well. Not included Tests for the new createBlocks functionality. Need to mock the options feature first
* Added another test that Coordinator Snippets of queries can be created correctly
* Fixed Coordinator-site cleanup of QueryRegistry, if any of the query creations fails with error, incl UnitTest.
* Added first test for RestAqlHandler::setup. It does only test the setup and gives prepartion for real testing.
* Added a assertion of http return code. Still no creation of queries is tested. Requires a huge amount of mocking.
* fix some deadlocks found by evil lock manager (tm)
* fix duplicate lock
* fix indentation
* ensure proper lock dependencies
* fix lock acquisition
* removed useless comment
* do not lock twice
* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not
* AQL micro optimizations
* use explicit constructor
* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context
entering and exiting the selected context is then done later on without having to hold the ConditionLocker
* remove some recursive locks
* Disable custom deadlock detection when Thread Sanitizer is enabled
* Changing ifdef's
* grr
* broke gcc
* Using atomic for ApplicationServer::_server
* fix premature unlock
* add some asserts
* honor collection locking in cluster
* yet one more lock fix
* removed assertion
* Allow the clustercomm to send nolock headers on count. This is used form within AQL
* IsLocked on transactions will now always yield true IF LOCK_NEVER is set. We simply assume someone else holds the lock for us. Also LOCK_NEVER is now set on collection/count if noLock header is send.
* Moved the flag if collections need to be locked into the TraverserEngines.
* Added enterprise-satellite hooks in EngineInfoContainerDBServer
* Removed now obsolete code
* Replaced throwing of Exception by an ResultObject
* Added some more tests and moved adding snippet to query engine more to the outside.
* Added the AQL result type
* Make the branch compile again
* Register WITH collections for Graphs in the new Collector.
* Fixed test code for failing query clone. Idea was to once clone successfully and second time to fail, we verify that first clone is cleaned up properly. However test failed on first clone...
* Removed a double builderClose
* Added Changelog entry
* Removed empty if
* Removed obsolete todo
* Properly initialize the AqlResult with nullptr on error case
* Updated comment
* Simplified Assertion
* Removed debug output object...
* Added additional catch case for std::exception to get some more error info
* Clarified evaluation order for move case
* Added Explicit
* Fixed cleanup of Coordinator if Registry fails to insert query.
* Allow to use other locks than Read/Write for AQL collections. Not yet in API.
* Updated Comments for other Locks on DBSide. Adapted Destruction CatchTests
* Fixed double builderClose and removed unnecessary double commits
* Added a comment to clarify the state
* Moved error output to trace. Leftover from debugging
* Added some tests for complex subquery patterns
* Added a 'fireAndForgetRequests' methods to cluster comm which allows to send out a bunch of messages but does not wait for their results
* Properly cleanup leftovers of queries if the instantiation step already failed
* Added code comment for fireAndForget
* Added indexes to subquery test to make the plan a bit easier
* The cleanup on DBServerEngines in error case now also cleans up traverser engines.
* Removed unnecessary includes
* Removed debug logging
* Fixed hidden merge conflict
Jan
KVS85
Tobias Gödderz
Dan Larkin-York
Jan Christoph Uhde
Lars Maier
Frank Celler
Simon
Vasiliy
Matthew Von-Maszewski
Frank Celler
jsteemann
Andrey Abramov
Michael Hackstein