* Squashed commit of feature-3.5/hotbackup_devel.
This puts hotbackup into 3.5.
* Port atomic-database-creation-2 to 3.5.
* Remove some wrongly ported code.
* Fix compilation.
* Fix a manual merge error.
* Remove a feature from the mocks which does not exist in 3.5.
* Add some code which was forgotten in manual merge.
* Fix a problem introduced in a manual merge.
* reuse function
* Address some whitespace issues that came up in review
* aardvark should not create the frontend collection
* create _frontend collection from c++
* recheckAndUpdate Callback in CollectionWatcher
* Wrong author ;)
* rm outdated todo
* Update lib/Basics/VelocyPackHelper.h
Co-Authored-By: Michael Hackstein <michael@arangodb.com>
* use logger unique id, use startup logger
* not needed
* optimized vector shardid method
* do not create _modules collection lazy anymre
* Formatting.
* Assert instead of if/TRI_ASSERT(false)
* Don't use exceptions as control structure
* Re-add READ_LOCKER that got lost in translation
* Fix audit log in case database creation fails early.
* legacy sharding
* Add CHANGELOG entry.
* Retry database cancellation indefinitely
* Do not use exceptions in UpgradeTask
* DropCollection is a FAST_LANE action and should not need much time or else retry.
* Remove superflous addition of LdapFeature
Proudly brought to you by ASAN tests
* Fixed check for distributShardsLike sharding on _system database
* Fixed compile issue on tests
* Removed assertion that seems to be not correct yet on devel.
* Sort out google cloud storage as remote. (#9918)
* Add successful method to ClusterCommResult.
* Improve error forwarding for cluster internal communication.
* Feature/hotbackup list retries (#9924)
* retry hot backup listing for 2 minutes in cluster before giving up
* Enable api by default.
* always create statistics collections & indices & fishbowl
* fix broken list of non existing id (#9957)
* Fix compilation after manual merge.
* Fix another compilation problem.
* Yet more fixes for compilation.
* More compilation fixes.
* Removed forgotten, illegal if.
* Removed obsolete include
* Squashed commit of feature-3.5/hotbackup_devel.
This puts hotbackup into 3.5.
* Port atomic-database-creation-2 to 3.5.
* Remove some wrongly ported code.
* Fix compilation.
* Fix a manual merge error.
* Remove a feature from the mocks which does not exist in 3.5.
* Add some code which was forgotten in manual merge.
* Fix a problem introduced in a manual merge.
* reuse function
* Address some whitespace issues that came up in review
* aardvark should not create the frontend collection
* create _frontend collection from c++
* recheckAndUpdate Callback in CollectionWatcher
* Wrong author ;)
* rm outdated todo
* Update lib/Basics/VelocyPackHelper.h
Co-Authored-By: Michael Hackstein <michael@arangodb.com>
* use logger unique id, use startup logger
* not needed
* optimized vector shardid method
* do not create _modules collection lazy anymre
* Formatting.
* Assert instead of if/TRI_ASSERT(false)
* Don't use exceptions as control structure
* Re-add READ_LOCKER that got lost in translation
* Fix audit log in case database creation fails early.
* legacy sharding
* Add CHANGELOG entry.
* Retry database cancellation indefinitely
* Do not use exceptions in UpgradeTask
* DropCollection is a FAST_LANE action and should not need much time or else retry.
* Remove superflous addition of LdapFeature
Proudly brought to you by ASAN tests
* Fixed check for distributShardsLike sharding on _system database
* Fixed compile issue on tests
* Removed assertion that seems to be not correct yet on devel.
* Sort out google cloud storage as remote. (#9918)
* Add successful method to ClusterCommResult.
* Improve error forwarding for cluster internal communication.
* Feature/hotbackup list retries (#9924)
* retry hot backup listing for 2 minutes in cluster before giving up
* Enable api by default.
* fix broken list of non existing id (#9957)
* Fix compilation after manual merge.
* Fix another compilation problem.
* Yet more fixes for compilation.
* More compilation fixes.
* Cherry-pick minReplicationFactor
* Bug fix/failover with min replication factor (#9486)
* Improve collection time of IResearchQueryOptimizationTest
* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it
* Added some assertion son minReplicationFactor
* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled
* added minReplicationFactor to the user interface, preparation for the collection api changes
* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo
* added minReplicationFactor usage to tests
* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME
* minReplicationFactor now able to change via collection properties route
* fixed wrongly assert
* added minReplicationFactor to the graph management ui
* added minReplicationFactor to the gharial api
* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.
* adjusted description of minReplicationFactor
* FollowerInfo Refactoring
* added gharial api graph creation tests with minimal replication factor
* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests
* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor
* Debug logging
* MORE Debug logging
* Included replication fast lane
* Use correct minreplicationfactor
* modified debug logging
* Fixed compileissues
* MORE Debug logging
* MORE Debug logging
* MORE Debug logging
* MORE Debug logging
* MORE Debug logging
* MORE Debug logging
* MORE Debug logging
* Revert "MORE Debug logging"
This reverts commit dab5af28c0.
* Revert "MORE Debug logging"
This reverts commit 6134b664bd.
* Revert "MORE Debug logging"
This reverts commit 80160bdf3b.
* Revert "MORE Debug logging"
This reverts commit 06aabcdfe1.
* Removed debug output
* Added replication fast lane. Also refactored the commands as i cannot take it any more...
* Put some requests of RocksDBReplication onto CATCHUP Lane.
* Put some requests of MMFilesReplication onto CATCHUP Lane.
* Adjusted Fast and MED lane usage in Supervised scheduler
* Added changelog entry
* Added new features entry
* A new leader will now keep old followers in case of failover
* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp
Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
* Fixed JSLINT
* Unified lane handling of replication handlers
* Sorry forgotten in last commit
* replaced strings with static strings
* more use of static strings
* optimized min repl description in the ui
* decr initial loop variable
* clean up of the createWithId test
* more use of static strings
* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js
Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
* Added some comments on condition, renamed variable as suggested in review
* Added check for min replicationFactor to be non-zero
* Added assertion
* Added function to modify min and max replication factor in one go
* added missing semicolon
* rm log devel
* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place
* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers
* check replFactor against nr dbservers
* Add lie reporting in CURRENT
* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out
* move replication checks from logical collection to rest collection handler
* added more replication tests
* Include assert only if we are not in gtest
* jslint
* set min repl factor to zero if satellite collection
* check replication attributes in v8 collection
* Initial commit, old plan, does not yet work
* fixed ires tests
* Included FailoverCandidates key. Not fully implemented
* fixed wrong assert
* unified in sync follower reporting
* fixed compiler errors
* Cleanup locking, and fixed potential deadlocks
* Comments about locking order in FollowerInfo.
* properly check uint
* Keep old leader as potential failover candidate
* Transaction methods now use followerInfo to check if the leader can write, this might have the sideeffect that 'failoverCandidates' are updated
* Let agency check failoverCandidates if possible
* Initialize member variables
* Use unified follower reporting in DBServerAgencySync
* Removed obsolete variable, collecting it somewhere else
* repl factor attr check
* Reimplemented previous followers, second attempt now. PhaseOne and PhaseTwo can now synchronize on current.
* Fixed assertion, forgot an off-by-one
* adjusted test to be more preciese now
* Fixed failove candidates list
* Disable write on dropping too many followers
* Allow to run updateFailoerCandidates multiple times with same leader.
* Final fixes, resilience tests now green, crossing fingers for jenkins
* Fixed race on atomics comparison
* Fixed invalid number type
* added nullptr handling
* added nullptr handling
* Removed invalid assert
* Make takeover of leadership an atomic operation
* Update tests/js/common/shell/shell-cluster-collection.js
Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
* Review fixes
* Fixed creation code to use takeoverLeadership
* Update arangod/Cluster/FollowerInfo.h
Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
* Applied review fixes
* There is no timeout
* Moved AQL + Pregel to INTERNAL_AQL lane, which is medium priority, to avoid deadlocks with Sync replication
* More review fixes
* Use difference if you want to compare two vectors...
* Use std::string ...
* Now check if we are in recovery mode
* Added documentation for minReplicationFactor
* Added readme update as well in documenation
* Removed merge conflict leftovers 0o, i should not trust the IDE
* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update Documentation/Books/Manual/Architecture/Replication/README.md
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update CHANGELOG
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update Documentation/Books/Manual/DataModeling/Collections/DatabaseMethods.md
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update Documentation/Books/Manual/ReleaseNotes/NewFeatures35.md
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update Documentation/DocuBlocks/Rest/Collections/1_structs.md
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/graphManagementView.js
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/graphManagementView.js
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Update Documentation/DocuBlocks/Rest/Graph/1_structs.md
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Apply suggestions from code review
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Adepted review requests, thanks for finding!
* Removed unnecessary const
* Apply suggestions from code review
Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
* Moved initilization of variable more downwards
* Apply lock before notify_all()
* Remove documentation except DocuBlocks, covered by PR in docs repo
* Remove accidental indent
* Bug fix 3.4/collection babies (#9033)
* Prepare API to create multiple collections in a single request to ClusterMethods to improve speedup
* Added counter on how many collections are successfully created
* Allow multi collection creation one level higher
* CollectionMethods now allow batch createion of Collections
* Improved array size assertions
* Now a graph is createad within a single roundtrip in the agency.
* Added new header files
* Insert collections in the AGENCY with TTL and a isBuilding flag, collections with this flag should not be visisible in the coordinator
* Added forgotten C++ file
* Fixed a rare race condition, and the failing IResearch Tests
* readded callback on DONE, otherwise lists are out of sync
* Fixed assertions to let mocked tests pass...
* Fixed community cluster
* Started fixing IResearch analyzer test, catch-tests are failing ;(
* Solved missed merge-conflict
* Added helper functions in AnalyzerFeature-test
* Refactoring AnalyzerTest Section-Auth
* Refactoring AnalyzerTest Section-Emplace-Duplicates
* Refactoring AnalyzerTest Section-Emplace-Error-Cases. Recovery-Test is now red, it seemed to be green because of invalid test case before.
* Refactoring AnalyzerTest, split GET test into multiple parts, still left 'cluster simulation'.
* Attempt to extract Coordinator / DBServer tests a little bit. This commit starts to break all Coordinator tests. However i am convinced that earlier version did NOT test a cluster situation at all, but some hybrid of SingleServer with full local storage that got told to be a Coordinator from now on, but without any Coordinator setup...
* Temporarly disabled some tests in AnalyzerFeature, as discussed with @gnusi.
* Fixed include guard.
* Temporarily deactivated failing tests
* You shall save your files before you commit...
* Fixed test asserting on plan version, which is now higher than before
* do not persist legacy analyzers into _analyzers table
* fix arangosearch upgrade in cluster
* get rid of Vasiliy's shit
* address review comments
* ensure link is synchronized after creation in upgrade
* fix compilation error
* minor cleanup
* fix tests
* distribute '_analyzers' collection as '_graphs'
* comment out Vasiliy's shit part 2
* move IResearchViewSort into separate unit
* ensure sort is propagated to IResearchLink
* update iresearch and extend some tests
* implement insertion of sorted values, store value comparator in link
* add tests for VPackComparer
* minor cleanup
* fix issue with sort definiton propagation
* propagate error message from index factory
* add integration tests for sorted indexes, single server
* initial implementation of IResearchViewMergeExecutor
* move iresearch::VPackComparer to separate translation unit
* store collection in IndexReadBuffer for IResearchViewMergeExecutor
* set primary sort from optimizer rule
* add some tests for sorted view case
* extend IResearchViewNode tests
* simplify primary sort definition
* pretty print order condition covered by arangosearch in query explanation
* fix compilation with g++-8
* address catch test failures
* fix tests
* adjust arangosearch optimizer rule to optimize sort in case of cluster
* make tests for sorted indexes work in cluster too
* fix tests
* extend js tests for sorted indices
* fix keyword in query explainer
* ensure sort-limit rule works with views
* backport ngram analyzer fix from upstream
* address jslint errors
* modify IResearchViewSort to support boolean direction specification format
* add some tests
* add tests
* add skip functionality
* extend tests
* add more catch tests
* inline some methods
* address jslint errors
* extend catch tests to reproduce the failure
* add some tests for check fullcount
* add more tests
* extend tests
* temporarily comment out broken tests
* extend js tests
* address review comments
* address jslint errors
* update iresearch
* Added RemoteExecutor skeleton
* Moved RemoteBlock implementations to ExecutionBlockImpl<RemoteExecutor>
* Remove unnecessary include to avoid unused function warnings
* Fixed gcc compile error
* Moved Scatter/Distribute block implementations to their new Executor versions
* Applied clang-format
* Added factory, infos and a skeleton for the unordered view executor
* Removed assert based on wrong assumption
* Added members from IResearchViewBlockBase to IResearchViewExecutor
* Moved more code into the ViewExecutor, hopefully enough to produce a working version now
* Added missing reset code, made produceRow work mostly correct
* Removed superfluous parentheses to get more useful output from Catch
* Ported fix 923b6e81ac723d1fe37f8e7bf1ab81149f3a08ef
Original commit message was:
Fixed a race condition in RemoteBlock which was triggered during
shutdown overtaking getSome.
* Applied review comments
* Inject input row instead of an item block + pos into the expression context, plus fixed some tests
* Adapted test. Search tests are now green.
* Do not ask upstream when already DONE
* Removed `limit` from next()
* Simplified code that could handle producing more than one document
* Minor readability change
* Solved two TODOs noted in the review
* Removed leftover references to DistributeNode members in the DistributeBlock
* Reverted removal of "exhausted"
* WIP: Implemented variant with scorers
* Fixed compile errors of the last commit
* Fixed some asserts and calculations
* Fixed violated assertions
* Moved files from IResearch/ to Aql/
* Replaced recursive call with a loop
* Worked on a few TODOs
* Removed IResearchViewBlock
* Set input registers correctly
* Eliminated dependency to the Node in the Executor
* Don't misuse the volatility variables for initialization
* Extended a TODO note
* Removed obsolete includes
* Removed an obsolete include from the tests
* Added missing include
* Read PKs in batches
* Fixed merge conflict
* Fixed merge conflict
* Restrict prefetching of PKs to the number of rows in the current output block
* Fixed merge
* Fix IResearch ASan errors
* Revert "Restrict prefetching of PKs to the number of rows in the current output block"
This reverts commit e0fd8698a3.
* Revert "Read PKs in batches"
This reverts commit c06c4d7a36.
* Began some small step refactoring to introduce batch-reading correctly
* Extracted method fillBuffer
* Extracted method evaluateScores
* Minor changes
* Read data from iresearch index in batches
* Replaced std::deque<IndexResult> buffer by a new class
* Solved minor TODOs
* Fixed last commit
* Fixed merge conflict
* Removed accidentally re-added view blocks
* Implemented SharedAqlItemBlockPtr
* Replaced all AqlItemBlockShell, shared_ptr<AqlItemBlock>, unique_ptr<AqlItemBlock> with SharedAqlItemBlockPtr
* Removed AqlItemBlockShell
* Bugfixes
* Added missing noexcept (used in returnBlock())
* Added nullptr constructor/operator= and noexcept specs
* Removed references to the shell
* Implemented review comments
* Fixed a compile error clang somehow ignored
* Two bugfixes and additional asserts
* Fixed ASan error
* protected AqlItemBlock destructor, and some cleanup
* added SharedAqlItemBlockPtr include in QueryCursor.h
* Added some asserts
* Fixed merge conflicts
* protected returnBlock in AqlItemBlockManager
* Port agency performance tuning for many shards to devel.
* Add more IDs to LOG_TOPIC calls.
* Even more IDs for LOG_TOPIC.
* Fix a duplicate LOG_TOPIC ID.
* Fix an old merging bug in devel.
* Don't hesitate between phases one and two for small clusters.
* Added RemoteExecutor skeleton
* Moved RemoteBlock implementations to ExecutionBlockImpl<RemoteExecutor>
* Remove unnecessary include to avoid unused function warnings
* Fixed gcc compile error
* Moved Scatter/Distribute block implementations to their new Executor versions
* Applied clang-format
* Added factory, infos and a skeleton for the unordered view executor
* Removed assert based on wrong assumption
* Added members from IResearchViewBlockBase to IResearchViewExecutor
* Moved more code into the ViewExecutor, hopefully enough to produce a working version now
* Added missing reset code, made produceRow work mostly correct
* Removed superfluous parentheses to get more useful output from Catch
* Ported fix 923b6e81ac723d1fe37f8e7bf1ab81149f3a08ef
Original commit message was:
Fixed a race condition in RemoteBlock which was triggered during
shutdown overtaking getSome.
* Applied review comments
* Inject input row instead of an item block + pos into the expression context, plus fixed some tests
* Adapted test. Search tests are now green.
* Do not ask upstream when already DONE
* Removed `limit` from next()
* Simplified code that could handle producing more than one document
* Minor readability change
* Solved two TODOs noted in the review
* Removed leftover references to DistributeNode members in the DistributeBlock
* Reverted removal of "exhausted"
* WIP: Implemented variant with scorers
* Fixed compile errors of the last commit
* Fixed some asserts and calculations
* Fixed violated assertions
* Moved files from IResearch/ to Aql/
* Replaced recursive call with a loop
* Worked on a few TODOs
* Removed IResearchViewBlock
* Set input registers correctly
* Eliminated dependency to the Node in the Executor
* Don't misuse the volatility variables for initialization
* Extended a TODO note
* Removed obsolete includes
* Removed an obsolete include from the tests
* Added missing include
* Read PKs in batches
* Fixed merge conflict
* Fixed merge conflict
* Restrict prefetching of PKs to the number of rows in the current output block
* Fixed merge
* Fix IResearch ASan errors
* Revert "Restrict prefetching of PKs to the number of rows in the current output block"
This reverts commit e0fd8698a3.
* Revert "Read PKs in batches"
This reverts commit c06c4d7a36.
* Began some small step refactoring to introduce batch-reading correctly
* Extracted method fillBuffer
* Extracted method evaluateScores
* Minor changes
* Read data from iresearch index in batches
* Replaced std::deque<IndexResult> buffer by a new class
* Solved minor TODOs
* Fixed last commit
* Fixed merge conflict
* Removed accidentally re-added view blocks
* Implemented review comments
Andrey Abramov
Jan
Max Neunhöffer
Heiko
Dan Larkin-York
Michael Hackstein
Simon
Jan Christoph Uhde
Lars Maier
Wilfried Goesgens
Tobias Gödderz