running compact() in the same transaction will only increase the data size on disk due to
RocksDB not being able to remove any documents physically due to the snapshot that is
taken at transaction start.
This change also exposes db.<collection>.compact() in the arangosh, in order to manually
run a compaction on the data range of a collection should it be needed for maintenance.
* Fix resign order
* Fixed a typo
* Get followers later, add TODOs
* Added a callback parameter to collection insert methods
* Get followers under the lock if necessary
* Extracted the replication of inserts into a separate method
* Move shortcut into replicate method
* Added callbacks for remove, replace and update
* Added missing overrides
* Extracted replication code from modifyLocal and removeLocal
* Update followers under lock also during replace, update, remove
* Fix changes from the last commit for update/replace
* Update comments, add asserts
* Remove changes for document-level locks that will be done in another PR
* Unify replication
* Adapt log messages to the devel ones
* Move common methods from its descendants to TransactionCollection, fix Mock on the way
* More IResearch test / mock fixes
* Relax asserts for nested transactions
* Reformat
* Fix non-babies remove and modify replication
* Added feature phases
* BasicsPhase and DatabasePhase to the required files. Server now has Feature circles and does not boot. Will be sorted out later on.
* Added ClusterPhase to features
* Added V8Phase to the required features
* Added AQLPhase to the affected features
* Added ServerPhase to Features
* Added FoxxPhase to the relevant features
* Added AgencyPhase to the relevant features
* Moved registration from local variable SYS_SYSTEM_REPLICATION_FACTOR from cluster to V8 as their ordering is now vice versa
* Moved Bootstrap feature into FoxxPhase. It could be moved to ServerPhase easily if the FoxxQueue dependency would be removed
* Final movement of Startup Phases. Now solved all circles.
* Removed merge conflict
* Moved ReplicationTimeout into cluster phase and fixed cross-phase requirements
* Added greetings phase. This phase separates the Basics Phase and is the first to be run. Includes Logger and Hello/Goodbye
* Added the GreetingsPhase in the corresponding features. Now all BasicsPhase features start after greetings Phase. There is some issue in this branch which prevents the Agency from Gossipping right now. Will be fixed next
* Moved creation of the Agent into the prepare phase of the feature. THereby it is guaranteed that agents at least exists before the GeneralServer is activating endpoints
* Recovery needs to be started after the ServerID
* Moved log output of FeaturePhases to DEBUG instead of ERROR.
* Added feature phases for clients
* ClusterFeature now does not directly require AgencyFeature any more
* Added requirement of TravEngineRegistryFeature in AQL feature. Otherwise shutdown may be undefined
* The ApplicationServer can now handout the list of ordered features. Used for testing purposes
* Fixed IResearchVew Tests Setup to honor new feature ordering
* Fixed IResearchViewDBServer Tests Setup to honor new feature ordering
* Started fixing IResearchView Coordinator tests with startup ordering. Not finished yet
* Added startup phases to ViewCoordinator test
* Disabled expected logoutput in ClusterRepairsTest
* Fixed indention in test code
* LinkCoordinator now honors startup ordering
* Link meta now honors startup rdering
* Supress expected cluster logs in ViewTest
* Removed '#' accidentially added.
* Modified header to new initializeCursor API
* Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet
* Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later
* First attempt to simulate thread waiting over information within the query
* Small fix to allow for isDirect handlers to go to sleep.
* Waiting in the necessary places now for the async request to be send.
* Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here?
* Include cond-var header
* Fixed mutex/cond_var usage
* Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested
* Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries.
* Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts
* Fixed endless virtual recursion
* Implemented new getOrSkip API in SortBlock
* Implemented new getOrSkip API in LimitBlock
* Initilaize all variables
* Fixed logic bug in SubqueryBlock
* getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly!
* Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime.
* Added IndexBlock::skipSome and IndexBlock::getSome
* getBlock now returns its old return value along with the state
* Switch from getBlockOld to getBlock in IndexBlock::skipSome
* Switch from getBlockOld to getBlock in IndexBlock::getSome
* ShortestPathBlock::skipSome is not implemented! Added a regression test
* Attempt to fix SubQueryResult memory management
* Fixed LIMIT Block
* Moved from ShortestPathBlock::getSomeOld to ::getSome
* Implemented ASYNC api on SingletonBlock
* Adapted EnumerateCollectionBlock to new async API
* Fixed FilterBlock and adapted return block to async API
* Adapted NORESULTS block to async AQL api.
* Adapted Modification Blocks to async API
* Fixed some initialize cursor functions to reset values required during get/skipSome
* First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still
* Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place.
* Adapted CalculationBlock to async API
* Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN
* Adapted EnumerateListBlock to Async AQL api
* Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now
* Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing
* Adapted IReasearchViewBlock to ASYNC AQL API
* Fixed SortingGatherBlock in WAITING state.
* Adapted IResearch ExecutionBlockMock to Async API
* Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍
* Implemented tonly heoretically reachable function of non void function.
* Fixed last commit
* Added inline TODO comments
* fix warning
* Fixed a clearing logic bug in RemoveNodes
* Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter.
* remove unused methods
* Fixed failure test
* implement skipping
* Moved the Query Waiting out of the ExecutionEngine.
* changed one of the collect blocks
* Removed _upstreamState from ExecutionBlockMock, that is in the base-class now
* Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api.
* do not check "hasMore" if not necessary
* Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type
(still uses getBlockOld)
* Save state to resume in DistinctCollectBlock::getOrSkipSome
* Extracted redundant code
* fixed some ops
* added one more test
* fix endless blocking
* fix compile error
* fix test
* Refactored HashedCollectBlock::getOrSkipSome
* Return blocks to the manager
* Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome
* remove unused shutdown calls, simplify ownership for expressions
* Removed superfluous variable
* Capture const variable by value
* Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome
* Added a working version of SortedCollectBlock::getOrSkipSome
Has yet to be cleaned up
* Removed isTotalAggregation special treatment
* On no input, return a group of nulls (instead of no group at all)
* Bugfixes
* Simplified code
* Move return to the end, eliminate duplicate code
* Corrected skipped count in HashedCollectBlock
* Aligned getNextRow() implementations
* Added comments
* some cleanup
* fix potential memleak
* Bugfix
* Fixed failure tests
* Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome
* Replaced hasMore with an async implementation (mostly)
* Removed getBlockOld()
* Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState)
* RemoteBlock now uses the async hasMoreState route
* remove job queue
* options
* Bugfixes in the async implementation of LimitBlock
* LimitBlock::getOrSkipSome now always skips when calculating the fullcount
* fix compile warnings
* restrict threads
* Fixed api of Waiting ExecBlockMock. Unused yet
* Made SortedGatherBlock async-capable
* Removed nonEmptyIndex hack
* Removed duplicate traceGetSome~ calls, moved all to getSome
* Added asserts before replacing getNr*Registers
* Added a TODO note and a comment
* Removed getSomeWithoutRegisterClearoutOld()
* Removed skip()
* Removed common code by using getNr*Registers()
* Use getNr*Registers() in the TraversalBlock as well
* started to add lane
* started to add lane
* added lane
* completed lane
* removed debug output
* fixed merge
* Began working on a test suite for AQL tracing/profiling
* Added more tests and asserts in aql-profiler
* Made some ExecutionBlocks final
* Added a type enum to all blocks and the per-block stats
* Add block type to stats nodes when tracing AQL on block level
* Removed initializeCursor call from instantiateFromPlan
* Avoided additional getSome calls after DONE
* Added more profiler tests
* Refactored ExecutionBlock::getOrSkipSome and fixed two bugs
- set _upstreamState also when skipping
- explicitly use xecutionBlock::getHasMoreState()
* Bugfix: update state
* Reuse parent _skipped wherever possible; rename where not (LimitBlock)
* Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code
* Implemented missing virtual function (with USE_FAILURE)
* Reset neccessary values during initializeCursor
* Simplified code in EnumerateListBlock a little
* Added a test for DistinctCollectBlock in aql-profiler
* Avoid redundant getSome calls in DistinctCollectBlock
* fix compilation
* Fixed DistinctCollectBlock profiler test
* Added a second profiler test for the DistinctCollectBlock
* Added a profiler test for EnumerateCollectionBlock
* Bugfix in EnumerateListBlock
* added --server.fifoN-size
* Simplified EnumerateCollectionBlock::getSome
* Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE
* Fix testEnumerateCollectionBlock1 for mmfiles
* do not pass by reference
* Fixed compile error
* fixed merge conflicts
* Added profiler tests for EnumerateCollectionBlock
* Test fix for mmfiles
* Fixed IResearch tests
* Bugfix in DistinctCollectBlock and a regression test
* Updated comment
* Bugfix for query statistics in cluster
* Check plan in distinct test
* Fix aql-profiler tests in cluster
* Removed unused line / bugfix for single server test runs
* This commit implements waking up of AQL queries. (#5651)
* Non-compiling intermediate commit for handover.
* Make branch compile again
* Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested!
* Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine.
* First successful query wakeup.
* The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler
* renamed finalizeExecute to shutdownExecute
* Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly
* fix audit log
* Removed callback from deleteQueryCursor. This cannot be waiting
* use CONDITION_LOCKER
* removed yet another thread-local variable
* Fixed forward declaration
* Made RestAqlHandler repeatable
* Use defer to close the query in RestAqlHandler. Now waiting will close the query as well.
* Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler
* Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case
* Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded.
* Removed reaction to existing nolockheaders in Coordinator Query Planning Phase
* Removed incorrect assertion.
* Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly.
* Fixed UserManager, now deletes nolock headers properly
* Swing to the Symphony of Destruction
* Forgot about community build...
* Fixed compiling of Catch tests
* Fixed community build
* need thread for size
* Made the restSimpleHndler repeatable
* Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor
* Reactivated StreamingCursors
* Removed debug output.
* Fixed false query continuation
* Reset thread output to non-debug
* Added missing return statements
* Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers.
* Fixed hidden merge conflict
* Bugfix in aql-profiler.js: use plan.nodes order, not stats
* Added two profiler tests for filter
* Avoid too many getBlock calls in the FilterBlock
* Removed debug output
* RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility.
* Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete
* Added another test for FILTER, and one test for the HashedCollectBlock
* Added more tests for HashedCollectBlock; avoid unneccessary getSome calls
* Added an profiler IndexBlock test
* IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls
* Added a second test profiling IndexBlock
* Added a third test for IndexBlock
* Moved general code to module
* Moved noncluster tests into a separate file
* Split aql-profiler testsuite into three files
* Added profiler tests for LimitBlock
* Added a test for NoResultsBlock
* Added profiler tests for TraversalBlock
* Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still
* Optimized TraversalBlock getSome calls due to new (nightly) test results
* Fixed std::min calls I broke
* Let shutdown calls in AQL wait, if the query is executed successfully.
* Fixed queryResult going out of scope
* fix compile error through merge conflict with devel
* Fixed compiler warning "mismatching tags"
* Removed debug log output
* Added TODO notes
* Fixed test fail due to devel merge
* Fixed some invalid sync waiting implementations
* Added a profiler test for SortBlock
* Added profiler tests for SortedCollectBlock
* Fixed bug introduced by devel merge
* Fixed Remoteblocks ignoring errors!
* Added some more continue Callbacks in used places. And removed debug log
* Removed debug log output
* Suppress clang warnings
* Bugfix: use of invalid stack pointer
* Bugfix: RemoteBlock::shutdown now sends code as int, not string
* Revert "Suppress clang warnings"
This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0.
* Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome
* Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed
* Allow copy elision
* start implementing arangosearch cluster tests.
* backport: ensure view lookup is done via collectionNameResover, ensure updateProperties returns current view properties
* first attempt to fix failing tests
* refactor cluster wide view creation logic
* if view is not found in the new plan then check the old plan too
* ensure the cluster-wide view is looked up in vocbase as well on startup/recovery
* do not store cluster-wide IResearchView in vocbase
* move stale view cleanup to the shared pointer deleter, address test failures
* do not print warning
* enable arangosearch tests by default
* fix catch tests
* address icorrect return value for cluster-wide links
* address some issues with test failures due to cluster-view allocated within TRI_vocbase_t
* simplify per-cid view name, address 'catch' test failures
* ensure IResearchViewNode volatility is properly calculated in cluster
* invoke callbacks directly in AgencyMock instead of waiting for timeout
* ensure view updates via JavaScript always use the latest view definition
* pass a list of shards to `IResearchViewDBServer::snapshot`
* extend cluster aql tests
* fixes after merge
* fix class/struct inconsistencies
* comment failing tests
* remove debug logging
* add debug function
* tests cleanup
* simplify upcoming merge: pass resolver from a side
* backport: move all transaction status callback logic to Methods
* add changes missed from previous commit
* fix js and ruby tests
* more tests for IResearchViewNode
* pass transaction to IResearchViewDBServer::snapshot, address IResearchViewDBServer tests segfault
* pass transaction to IResearchView::snapshot instead of transaction state
* temporarily add trace log output to tests to try to find the cause of the core dump on Jenkins
* add more temporary debug output to trace down the segfault on Jenkins
* add even more temporary debug output to trace down the segfault on Jenkins
* ensure Vieew related maps are cleared during shutdown
* reset ClusterInfo::instance() before DatabaseFeature::unprepare()
* remove extraneous debug output
* missed line from previous commit
* uncomment required line
* add nullptr checks to RocksDBIndexFactory::prepareIndexes(...) similar to the ones in MMFilesIndexFactory::prepareIndexes(...)
* attempt to fix deadlock in tests
* add comment as per reviewer request
* fix aql test suite name
* add some debug logging
* address deadlock between ClusterInfo::loadPlan() and CollectionNameResolver::localNameLookup(...)
* eplicitly state which index definition failed in the log message
* use vocbase from shard-view isntead just in case
* explicitly state which index definition failed in the log message
* do not create shard-view instances from cluster-link instances (only register existing ones)
* add some tests
Dan Larkin-York
Jan
Simon
Frank Celler
Tobias Gödderz
jsteemann
Vasiliy
Michael Hackstein
Andrey Abramov