* Added cleanup functions class that allows to execute functions on unexpected shutdown (like FATAL_EXIT). First use: delete the LOCK file. (#5218)
* Removed superfluous log message
* Modified header to new initializeCursor API
* Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet
* Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later
* First attempt to simulate thread waiting over information within the query
* Small fix to allow for isDirect handlers to go to sleep.
* Waiting in the necessary places now for the async request to be send.
* Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here?
* Include cond-var header
* Fixed mutex/cond_var usage
* Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested
* Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries.
* Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts
* Fixed endless virtual recursion
* Implemented new getOrSkip API in SortBlock
* Implemented new getOrSkip API in LimitBlock
* Initilaize all variables
* Fixed logic bug in SubqueryBlock
* getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly!
* Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime.
* Added IndexBlock::skipSome and IndexBlock::getSome
* getBlock now returns its old return value along with the state
* Switch from getBlockOld to getBlock in IndexBlock::skipSome
* Switch from getBlockOld to getBlock in IndexBlock::getSome
* ShortestPathBlock::skipSome is not implemented! Added a regression test
* Attempt to fix SubQueryResult memory management
* Fixed LIMIT Block
* Moved from ShortestPathBlock::getSomeOld to ::getSome
* Implemented ASYNC api on SingletonBlock
* Adapted EnumerateCollectionBlock to new async API
* Fixed FilterBlock and adapted return block to async API
* Adapted NORESULTS block to async AQL api.
* Adapted Modification Blocks to async API
* Fixed some initialize cursor functions to reset values required during get/skipSome
* First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still
* Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place.
* Adapted CalculationBlock to async API
* Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN
* Adapted EnumerateListBlock to Async AQL api
* Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now
* Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing
* Adapted IReasearchViewBlock to ASYNC AQL API
* Fixed SortingGatherBlock in WAITING state.
* Adapted IResearch ExecutionBlockMock to Async API
* Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍
* Implemented tonly heoretically reachable function of non void function.
* Fixed last commit
* Added inline TODO comments
* fix warning
* Fixed a clearing logic bug in RemoveNodes
* Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter.
* remove unused methods
* Fixed failure test
* implement skipping
* Moved the Query Waiting out of the ExecutionEngine.
* changed one of the collect blocks
* Removed _upstreamState from ExecutionBlockMock, that is in the base-class now
* Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api.
* do not check "hasMore" if not necessary
* Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type
(still uses getBlockOld)
* Save state to resume in DistinctCollectBlock::getOrSkipSome
* Extracted redundant code
* fixed some ops
* added one more test
* fix endless blocking
* fix compile error
* fix test
* Refactored HashedCollectBlock::getOrSkipSome
* Return blocks to the manager
* Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome
* remove unused shutdown calls, simplify ownership for expressions
* Removed superfluous variable
* Capture const variable by value
* Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome
* Added a working version of SortedCollectBlock::getOrSkipSome
Has yet to be cleaned up
* Removed isTotalAggregation special treatment
* On no input, return a group of nulls (instead of no group at all)
* Bugfixes
* Simplified code
* Move return to the end, eliminate duplicate code
* Corrected skipped count in HashedCollectBlock
* Aligned getNextRow() implementations
* Added comments
* some cleanup
* fix potential memleak
* Bugfix
* Fixed failure tests
* Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome
* Replaced hasMore with an async implementation (mostly)
* Removed getBlockOld()
* Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState)
* RemoteBlock now uses the async hasMoreState route
* remove job queue
* options
* Bugfixes in the async implementation of LimitBlock
* LimitBlock::getOrSkipSome now always skips when calculating the fullcount
* fix compile warnings
* restrict threads
* Fixed api of Waiting ExecBlockMock. Unused yet
* Made SortedGatherBlock async-capable
* Removed nonEmptyIndex hack
* Removed duplicate traceGetSome~ calls, moved all to getSome
* Added asserts before replacing getNr*Registers
* Added a TODO note and a comment
* Removed getSomeWithoutRegisterClearoutOld()
* Removed skip()
* Removed common code by using getNr*Registers()
* Use getNr*Registers() in the TraversalBlock as well
* started to add lane
* started to add lane
* added lane
* completed lane
* removed debug output
* fixed merge
* Began working on a test suite for AQL tracing/profiling
* Added more tests and asserts in aql-profiler
* Made some ExecutionBlocks final
* Added a type enum to all blocks and the per-block stats
* Add block type to stats nodes when tracing AQL on block level
* Removed initializeCursor call from instantiateFromPlan
* Avoided additional getSome calls after DONE
* Added more profiler tests
* Refactored ExecutionBlock::getOrSkipSome and fixed two bugs
- set _upstreamState also when skipping
- explicitly use xecutionBlock::getHasMoreState()
* Bugfix: update state
* Reuse parent _skipped wherever possible; rename where not (LimitBlock)
* Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code
* Implemented missing virtual function (with USE_FAILURE)
* Reset neccessary values during initializeCursor
* Simplified code in EnumerateListBlock a little
* Added a test for DistinctCollectBlock in aql-profiler
* Avoid redundant getSome calls in DistinctCollectBlock
* fix compilation
* Fixed DistinctCollectBlock profiler test
* Added a second profiler test for the DistinctCollectBlock
* Added a profiler test for EnumerateCollectionBlock
* Bugfix in EnumerateListBlock
* added --server.fifoN-size
* Simplified EnumerateCollectionBlock::getSome
* Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE
* Fix testEnumerateCollectionBlock1 for mmfiles
* do not pass by reference
* Fixed compile error
* fixed merge conflicts
* Added profiler tests for EnumerateCollectionBlock
* Test fix for mmfiles
* Fixed IResearch tests
* Bugfix in DistinctCollectBlock and a regression test
* Updated comment
* Bugfix for query statistics in cluster
* Check plan in distinct test
* Fix aql-profiler tests in cluster
* Removed unused line / bugfix for single server test runs
* This commit implements waking up of AQL queries. (#5651)
* Non-compiling intermediate commit for handover.
* Make branch compile again
* Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested!
* Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine.
* First successful query wakeup.
* The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler
* renamed finalizeExecute to shutdownExecute
* Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly
* fix audit log
* Removed callback from deleteQueryCursor. This cannot be waiting
* use CONDITION_LOCKER
* removed yet another thread-local variable
* Fixed forward declaration
* Made RestAqlHandler repeatable
* Use defer to close the query in RestAqlHandler. Now waiting will close the query as well.
* Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler
* Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case
* Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded.
* Removed reaction to existing nolockheaders in Coordinator Query Planning Phase
* Removed incorrect assertion.
* Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly.
* Fixed UserManager, now deletes nolock headers properly
* Swing to the Symphony of Destruction
* Forgot about community build...
* Fixed compiling of Catch tests
* Fixed community build
* need thread for size
* Made the restSimpleHndler repeatable
* Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor
* Reactivated StreamingCursors
* Removed debug output.
* Fixed false query continuation
* Reset thread output to non-debug
* Added missing return statements
* Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers.
* Fixed hidden merge conflict
* Bugfix in aql-profiler.js: use plan.nodes order, not stats
* Added two profiler tests for filter
* Avoid too many getBlock calls in the FilterBlock
* Removed debug output
* RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility.
* Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete
* Added another test for FILTER, and one test for the HashedCollectBlock
* Added more tests for HashedCollectBlock; avoid unneccessary getSome calls
* Added an profiler IndexBlock test
* IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls
* Added a second test profiling IndexBlock
* Added a third test for IndexBlock
* Moved general code to module
* Moved noncluster tests into a separate file
* Split aql-profiler testsuite into three files
* Added profiler tests for LimitBlock
* Added a test for NoResultsBlock
* Added profiler tests for TraversalBlock
* Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still
* Optimized TraversalBlock getSome calls due to new (nightly) test results
* Fixed std::min calls I broke
* Let shutdown calls in AQL wait, if the query is executed successfully.
* Fixed queryResult going out of scope
* fix compile error through merge conflict with devel
* Fixed compiler warning "mismatching tags"
* Removed debug log output
* Added TODO notes
* Fixed test fail due to devel merge
* Fixed some invalid sync waiting implementations
* Added a profiler test for SortBlock
* Added profiler tests for SortedCollectBlock
* Fixed bug introduced by devel merge
* Fixed Remoteblocks ignoring errors!
* Added some more continue Callbacks in used places. And removed debug log
* Removed debug log output
* Suppress clang warnings
* Bugfix: use of invalid stack pointer
* Bugfix: RemoteBlock::shutdown now sends code as int, not string
* Revert "Suppress clang warnings"
This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0.
* Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome
* Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed
* Allow copy elision
* issue 373.2: move toVelocyPack into LogicalDataSource
* backport: move static DataSource related strings into StaticStrings, add support for registering snapshots on DBServer views
* downgrade s2 for windows compatibility
* undefine constants if they're already there
* Fix compilation on Windows for S2 and VS 2017. (#5267)
* Fix compilation on Windows VS 2017 (s2).
* Improve compilation fix for windows.
* downgrade reference
* initial check-in of working auto block size tuning for import. Needs clean-up.
* partial fix in this branch to enable testing, awaiting better fix in different branch.
* move pacing code from findIdleSender to AutoTuneThread object.
* move pacing code from findIdleSender to AutoTuneThread object. clean up dead code. add comments about algorithm.
* edits to algorithm discussion
* correct logging level. add CHANGELOG entry.
* update comment concerning proper usage of shutdown()
* initialize new static member MaxBatchSize. Correct sendCsvBuffer() call to addPeriodByteCount() since _outputBuffer drained by sendData().
* remove redundant semicolons. add new static member MaxBatchSize.
* move MaxBatchSize to ImportHelper so both ImportFeature object and AutoTuneThread object can access it.
* minor updates to comments and statics
* add pacing discussion to import manual page
* minor edit
* add initial implementation of scatter view rule and node
* add tests for `IResearchViewNode` and `IResearchViewScatterNode`
* add missing check
* modify IResearch execution nodes to use references instead of pointers
* use view id in searialized `ExecutionNode` representation instead of the name
* add cluster mode stubs and checks
* very first attempt to distribute IResearchViewNode
* further implementation of cluster-wide arangosearch views
* fix invalid json format
* add tests for coordinator iresearch view
* allow to retrieve a list of existing views on a coordinator
* more tests for coordinator iresearch view
* some fixes to enable query explanation
* remove Collection dependency from RemoteNode
* remove unnecessary remote ArangoSearch view scatter
* fix explanation appearance
* add some assertions
* minor fixes
* implement IResearchViewCoordinator::updateProperties
* fix view DDL issues
* handle link modifications in DDL operations
* add coordinator implementation of iresearch view links
* fix tests
* further coordinator based view DDL implementation
* further IResearchViewCoordinator implementation
* add initial implementation of AgencyMock
* fix some tests
* code cleanup
* extend test + some fixes
* more tests for IResearchViewCoordinator
* fix tests for IResearchLinkCoordinator
* some fixes after merge
* fix tests
* remove declaration of nonexistent (previously removed) method
* some fixes after review
* remove string duplication
* more tests and fixes
* more fixes and tests
* more tests
* one more test
* fix 'use-after-free' asan error
* fix non-enterprise tests issues
* Initial low level interface for thread crash reporting (and management).
* Add a member version of isClusterRole()
* isolate heartbeat thread creation to new StartHeartbeatThread(). create heartbeat thread even if not a cluster or if an agent.
* update runDBServer() and runCoordinator() to shutdown more quickly by polling isStopping() at additional locations.
* copying updates from different branch / PR
* basic thread crash logging. Not yet tied into Agency arangod or have any specific threads posting crashes
* make Supervision thread a CriticalThread
* sandwich CriticalThread between Thread and other classes to create long term, repeating thread crash reporting.
* restore code lost upon branch update relating to new startHeartbeatThread() function
* add CriticalThread.cpp to build
* add new runAgentServer() function to loop for Agents. Make Heartbeat thread derive from CriticalThread.
* remove debug line
* shutdown quicker on db server and coordinator heartbeat threads
* Adjust the new condition variable usage to follow normal coding patterns. Probably was ok. But why take the chance. And simplify future maintenance.
* fix casts to use defined bit width
* adjust regex so it also works on elderly compiler infrastructure
* optimized regex
* work around MSVC bit shortage
* howto manipulate logging of the SUT
* improve regexp readability
* fix some deadlocks found by evil lock manager (tm)
* fix duplicate lock
* fix indentation
* ensure proper lock dependencies
* fix lock acquisition
* removed useless comment
* do not lock twice
* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not
* AQL micro optimizations
* use explicit constructor
* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context
entering and exiting the selected context is then done later on without having to hold the ConditionLocker
* remove some recursive locks
* Disable custom deadlock detection when Thread Sanitizer is enabled
* Changing ifdef's
* grr
* broke gcc
* Using atomic for ApplicationServer::_server
* fix premature unlock
* add some asserts
* honor collection locking in cluster
* yet one more lock fix
* removed assertion
* some more bugfixes
* Fixing assert
(cherry picked from commit 1155df173bfb67303077fbe04ee8d909517bfd21)
* remove some non-unused V8 persistents
* do not throw that many bogus assertions
* do not rely on server role being defined
* slightly better debug output for V8 context debugging
* fix collection ids in inventory response
* simplify bootstrap a bit
* slightly better error handling
* make elapsed time a queryable value
* use less memory for stub collections
* added assertions that will always make sense
* added assertions
* do not garbage-collect while waiting
* less copying of parameters
* do not show "load indexes into memory" buttons for mmfiles engine
as all indexes are in memory anyway
* when a collection is truncated via the web interface, flush the WAL and rotate all active journals
this will make close all open journals on leader and followers and make them subject to compaction opportunities
* fix invalid server id values being passed from web interface to backend
* introduce afterTruncate method for indexes
* added test case for issue #3447
* updated CHANGELOG
* don't warn about replicationFactor for system collections
* check that the queries actually use the geo index and not some other index
* properly report error in web interface
* fix some internals checks that made truncate fail for bigger collections in maintainer mode
* also run a compact() operation after a serious truncate
in order to make iteration over the truncated range much faster
when the collection is next accessed
* increase default maximum number of V8 contexts to at least 16
* Switching from ttl to supervision based failover mechanism
* Allowing canceling of ongoing actions
* refactored asyncjobmanager
* refactoring some code
* adding read-only flag
* catching some exceptions to reduce log pollution, removing unnecessary code, removing tests for _changeMode
* fixing "createsANewDatabaseWithAnInvalidUser"
* auth = off does not longer make everyone superuser
* Fixing cluster_sync and maybe resilience
* properly initialize all properties
* use faster comparison
* properly detect and handle "method not allowed"
* code-style
* remove unused variable
* narrow variable scope
* handle non-existance of AuthenticationFeature
* remove dead code
* replace some C string handling with std::strings
* moved assertion to the correct place
* honor number of array members for IN operator
* slightly adjust error messages
* slighty adjust some error messages
* try to fix issue with lingering replication contexts on shutdown
* clean up heartbeat thread a little bit
* small fixes
* remove memory zone remainders
* remove useless calls
* attempt to fix non-deterministic CRC test
* attempt to fix non-deterministic replication tests
* fix document removal with multiple documents in cluster
* attempt to fix non-determinism of tests
* fix privilege checks for accessing collections by collection id, not by name
* Send out empty heartbeats regardless of non-empty AppendEntriesRPC.
* Also improve logging:
Note if a log in the empty heartbeat sending takes > 0.01 s.
Clearly mark places where a leader resigns in logging.
Log if no empty heartbeat is sent out.
* Make leader more tolerant w.r.t. incoming AppendEntriesRPC responses.
* Add debug logging for _lastAcked and challengeLeadership.
* Remove some unused code. Do not count ourselves in challengeLeadership.
* Removal of entire activation/deactivation mechanisms in agency
* TRI_microtime up to c++11
* added term to response to sendAppendEntries.
- migrate the process handling to c++ where appropriate
- move code definition / usage closer together
- add function to portably detect external processes (not spawned by us) and whether we may
- send them signals
- add portable code to send signals to processes
- for linux match what impact a signal will have on the process we send it
- check the status of the process we sent the signal, and return if its non-deadly or the process terminated
- after 8 retries with 1 second sleep send a hard kill.
- killExternal() now returns the same structure as statusExternal()
- don't block signals on spawned processes
* Fix temporary directory handling
On linux we create a sub directory under the temp directory with our own name and a unique id.
On windows we would only return the configured system temporary directory and not create a subdirectory.
This PR makes windows behave similar to linux.
* Cleanup windows implementation of locating the base temp path
- don't try to create a temporary file. The calling function will try anyways and fail then.
- properly check for unsupported characters and bail out.
* changelog
* fix c++ string handling
* remove double directory separators, so later on JS_RemoveRecursiveDirectory() won't trip over them
* do not use V8 variant of AQL functions in early optimization stage when a C++ variant is available
* additionally, simplify AQL function definitions and aliases
* warn when more than 90% of max mappings are in use
* added C++ variant of replication catchup
* added `--log.role` option
* updated CHANGELOG
* removed non-existing scheduler.threads option from config
* removed useless __FILE__, __LINE__ invocations
* updated CHANGELOG
* allow a priority V8 context
* remove TRI_CORE_MEM_ZONE
* try to fix Windows errors & warnings
* cleanup
* removed memory zones altogether
* exclude system collections from collection tests
* some authentication optimizations...
don't query the permissions for one database three times, and don't build up tons of useless temporary strings for each request
* fix some floating point inaccuracies in AstNode value comparisons
* return different error codes for check-version
* Adds data file for exit codes that can be used by ArangoDB applications
* add some exit codes
* better return codes for centos
* add codes in arangod
* remove test code
* prevent re-downloading of starter and other small improvements
* fix some erros
* finish centos
* update urls
* add info about systemd detection
* move further towards a working systemd setup
* set service type to simple
* add arangodb-update-db script
* start work on debian packages
* a db dir that is already in place (former installation) should be reused now
* fix minor bugs
* Implement exit code translation in the windows installer.
* add autogenerated NSIS exit code mapping
* added debugging methods
* try to fix invalid access in case of error
* remove unused members
* bugfixes and comments
* all agency fixes in
* merge bug
* partially unguarded Agent::lead fixed
* all agency fixes in
* added nrBlocked to thread startup eval
* added nrBlocked to thread startup eval
* recombination of cases in State::get
* some maps replaced with unordered_maps
* optimized maps some
* removed delete call
* cleanup
* lower cpu activity of log thread too
* fix log messages
* do not enter threads into unordered_set, as it is unneeded
* do not compile in calls to disabled plan cache
* moved AQL regex cache from thread local variables to a class of its own
* more sensible thread creation and destruction
* remove API docs for endpoints that were deleted in 3.0
* remove unused variable
* set all scalar values in the ctor, so check tools do not complain
* move Result objects
* prevent potential nullptr access
* remove ambiguous boolean parameter with no clear meaning
* make code check tools happy
* don't allow copying cache Findings, allow moving Results
* the debug functionality for counting RocksDB key ranges was broken
this produced sporadic test failures
* remove superfluous iostream include
* fix range checks
* don't leak if an error happens during database removal
* try to simplify Result a bit
* try to deletion checks a bit more robust
* add "cluster selectivity estimates" to CHANGELOG
* add some documentation to RocksDBRestReplicationHandler
* fix building with relative paths
* add some more doc
* add some tests for the replication api
* fix RocksDBRestReplicationHandler and add tests
* update documentation
* remove obsolete parameter
* fix error message
* Implementing logger-first-tick, logger-tick-ranges. Fixing dump `chunkSize` documentation
* we must now ignore that datafiles are not sealed
this is because an unsealed datafile may have been produced by
renaming multiple journals to datafiles at server start
* acquire collection count after we have acquired the lock
* count the null byte as well
* fix count value acquisition
* send query fragments to the correct servers, even after failover or when a follower drops
the problem with using the previous shard-based approach is that responsibilities for shards may change at runtime
however, an AQL query must send all requests for the query to the initially used servers.
if there is a failover while the query is executing, we must still send all following requests to the same servers, and not the newly responsible servers
otherwise we potentially would try to get data from a query from server B while the query was only
instanciated on server A.
* ignore some return codes when closing zip files and do not report them
* hide mostly useless debug message
* clear basic authentication cache after deletion of users and after updating them
otherwise deleted/changed users can still access the database!
* adjust wording
* added notes about mmfiles-specific parameters
* updated CHANGELOG and documentation
* reduce extractions to projections
* recycle string buffers in SocketTask
* micro optimizations for mmfiles indexes
* added special lookup function for _key
* moved function into the correct file
* speed up key buffer allocations a bit
* added noexcept specifier
* correctly name variable
* explicitly move bounds
* fix and speedup from/toPersistent functions
* reuse string from ManagedDocumentResult for multiple lookups
* use move-assign
* a bit less work for single server
* speedup AQL function HASH
* single fetch optimization
* performance optimization for the case when no documents need to be returned
* make reduce-extraction-to-projection a RocksDB-only optimizer rule
* cppcheck
* try to fix compile error on MacOS
* bug fix for MacOSX
* missing namespace (in Windows compile)
* Take out 503 timeouts altogether.
* Overhaul of AgencyComm::sendWithFailover loop.
* Let performRequests optionally ignore 404 coll not found.
* Fix error message "database not found" when AgencyComm failed.
* Add log entries in Agency if locks are acquired too slowly.
* Reexecute the javascript cluster sync stuff even if there was no plan/current change...So failed sync jobs can retry later...
* Cover callbacks in Communicator by lock. This fixes https://github.com/arangodb/planning/issues/370
* Put in delay in waiting for leader in agency test.
* Schmutz logging to heartbeat topic.
* Add more lock time diagnostic in agent.
* Switch on agencycomm tracing in coordinator.
* reduce extractions to projections
* recycle string buffers in SocketTask
* micro optimizations for mmfiles indexes
* added special lookup function for _key
* moved function into the correct file
* speed up key buffer allocations a bit
* added noexcept specifier
* correctly name variable
* explicitly move bounds
* fix and speedup from/toPersistent functions
* reuse string from ManagedDocumentResult for multiple lookups
* use move-assign
* a bit less work for single server
* speedup AQL function HASH
* single fetch optimization
* performance optimization for the case when no documents need to be returned
* make reduce-extraction-to-projection a RocksDB-only optimizer rule
commit 3d9cf792912db1974b9ac5e00ca2b4c9245b7d34
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Fri Jun 2 15:32:43 2017 +0200
optimise for single writes in agency log
commit 65056ab9026f9b4b211dda0f17c75602b978f2bf
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri Jun 2 15:01:15 2017 +0200
More tests, taking agency log compaction interval into account.
commit 6600d707784e8fd5b62c0c75fd1826af09b8e13f
Merge: cf46882 02f00cc
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri Jun 2 14:50:38 2017 +0200
Merge branch 'agency-log-compaction-overhaul' of ssh://github.com/ArangoDB/ArangoDB into agency-log-compaction-overhaul
commit 02f00cc271d027f02b0625afb76745bfa76bf833
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Fri Jun 2 14:34:41 2017 +0200
compaction step and keep size defaults for 3.2
commit 03fc8fbff8f0ac701f7d7f94521c0c3152dd6f92
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Fri Jun 2 14:32:46 2017 +0200
Constituent fatally exists if eletion ballot cannot be persisted
commit cf4688226fc897e74bb2d9ebdfca3ce4578c3b70
Merge: c727fc4 724bd1e
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri Jun 2 13:08:15 2017 +0200
Merge branch 'agency-log-compaction-overhaul' of ssh://github.com/ArangoDB/ArangoDB into agency-log-compaction-overhaul
commit 724bd1efe19e2e9dbfc14cd819f180816b6d62d0
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Fri Jun 2 13:02:51 2017 +0200
persistence success in agency state is properly evaluated
commit c727fc48bb93e7b135b3ca929c03221c7bcaddb9
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri Jun 2 12:04:55 2017 +0200
Set default compaction step size in agency to 20000 and 10000 keep size.
commit ded16ae6945e9c1479e99bc2e7ccb4d6feca19a6
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri Jun 2 11:11:12 2017 +0200
Fix help page in startStandAloneAgency.sh for --use-persistence.
commit 13ae9f40f649a8f92eeca4b16bbb5647b540722d
Merge: 834c7c9 aa3e8c1
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu Jun 1 23:41:34 2017 +0200
Merge remote-tracking branch 'origin/devel' into agency-log-compaction-overhaul
commit 834c7c920d36db3579def66c38fb04870936f8bd
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu Jun 1 16:56:03 2017 +0200
Handle error in recvAppendEntriesRPC properly.
commit bd9c8d03b76ad25d4078740b5bf994fdba3845d0
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu Jun 1 16:55:35 2017 +0200
Handle errors in persist() and log() properly.
commit 5b4d2c3d9af078d6a1b5626af20dc9abf2546baa
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu Jun 1 16:25:22 2017 +0200
Improve error reporting.
commit d60697c5f26d6592eecefc9b9a43e9b699d1773d
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Thu Jun 1 12:16:39 2017 +0200
Agency must not responds to any requests after startup util leader has RAFT commited up to pre shutdown state
commit 92b8ede5fa022ace1596607abcf8fad1130504c8
Merge: 9340e74 d24455c
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Wed May 31 16:54:45 2017 +0200
Merge remote-tracking branch 'origin/devel' into agency-log-compaction-overhaul
commit 9340e7461130a4783c09ad8d91e5a07f9500a045
Merge: 7b7ce9d 63a9d60
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Wed May 31 12:13:55 2017 +0200
agency tests to cover compaction
commit 63a9d604c474eda4302032629dff1f0f69fa0813
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 31 11:59:11 2017 +0200
Set agency.compaction-keep-size to at least 0.
commit ef842260968a4769d9502a701b7251da32647e52
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 31 11:49:34 2017 +0200
Fix agent size 1 case (thread already gone).
commit 7b7ce9d79f6e8208c13f153b1b9a395b780d6ce1
Merge: 24e2e7e ff306bf
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Wed May 31 11:39:58 2017 +0200
Merge branch 'agency-log-compaction-overhaul' of https://github.com/arangodb/arangodb into agency-log-compaction-overhaul
commit ff306bf547bc4f528c9b66e222271ac143029508
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 31 11:11:06 2017 +0200
Move compaction into the future when we take a snapshot from leader.
commit 24e2e7e00f960928a79ce4008b8031d6b9b07fd9
Merge: 84034ac b3ea17a
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Wed May 31 11:01:13 2017 +0200
Merge branch 'agency-log-compaction-overhaul' of https://github.com/arangodb/arangodb into agency-log-compaction-overhaul
commit b3ea17a219baa2abd5892819012fb59f440cdeb8
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 31 09:42:59 2017 +0200
Get rid of double nonsense.
commit 035c8d1b34e1b73a381d5468422adf13b2ebc36a
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 31 09:25:28 2017 +0200
Sort out Agent::load sequence.
commit 84034ac2809a77145d6b1d23bf44857b3a0c4651
Merge: eb34a2e 3180a9d
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Tue May 30 17:07:20 2017 +0200
merging in
commit eb34a2e64e6ac8dc6571b92cb853c38b7022c833
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Tue May 30 16:58:05 2017 +0200
keep persistence for restarts in standalone agency
commit 3180a9d9ce4a4401a55ef02606b020316d43cbe5
Merge: 5d60524 28b9580
Author: Max Neunhoeffer <max@arangodb.com>
Date: Tue May 30 16:56:56 2017 +0200
Merge branch 'agency-log-compaction-overhaul' of ssh://github.com/ArangoDB/ArangoDB into agency-log-compaction-overhaul
commit 5d60524429d8ddda4491beecb931c3b9e3cc1d8a
Author: Max Neunhoeffer <max@arangodb.com>
Date: Tue May 30 16:56:36 2017 +0200
Implement snapshot sending in AppendEntriesRCV.
commit 28b958054f51c9cb36706df4e4345aa0f726ed15
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Tue May 30 14:20:13 2017 +0200
state machine should not advance _committed if empty
commit df18f326acea7f5bc2660a37e22f1503952e4b41
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 23:39:47 2017 +0200
Store term with compaction snapshot, recover again.
commit 2551a48b6fb513c9ea934bce755f8c364dae2f05
Author: Kaveh Vahedipour <kaveh@vahedipour.de>
Date: Mon May 29 17:45:26 2017 +0200
indices renamed to closely match RAFT documentation
commit e62dcdecf6e8650cfa5725d91b809d05591b48a4
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 16:37:43 2017 +0200
More cleanup.
commit 9f4787c46621375f0361138a8961431eb21ce5c0
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 14:50:25 2017 +0200
Revise loadLastCompactedSnapshot to return 0 without a snapshot.
commit 13285e1d70c8a4ac8c79a08de6f8fbc0f8d242bf
Merge: 3393c43 6c5f23e
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 12:06:20 2017 +0200
Merge branch 'agency-log-compaction-overhaul' of ssh://github.com/ArangoDB/ArangoDB into agency-log-compaction-overhaul
commit 3393c43c75520c74d20df09c74fbbbd8b1af5976
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 12:03:47 2017 +0200
More cleanup around Store::apply and friends.
commit 4ccb41d1839748c98e11403fa04f6a7d6af5e95b
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 09:45:41 2017 +0200
Document apply methods in Store with comments.
commit ea05c4880fedb6fe535e24761ac5cb3c26ccfc20
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 09:45:18 2017 +0200
Intentionally keep one log entry more to prevent empty log.
commit 67fb62f2259cc3c6368319917c7257ebcc177d3f
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 09:44:42 2017 +0200
Improve plausibility checks at compaction time.
commit 0bafc368785b15a94f8783c4c929f4208f87d09c
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 25 00:51:51 2017 +0200
Sort out (re-)building of agency K/V store(s) from compaction snapshots.
This is in case of (re-)start, becoming a leader and when serving
/_api/agency/store.
commit 46b0750bc6c597ec388aac0cdca32082c0cc54b8
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 25 00:50:54 2017 +0200
Set compaction interval to 200 and keep to 0.
This is way to small but tests should run with it.
Will later increase numbers again.
commit 024dc0846ae30248b464dd481a8bbc1134f56983
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 24 23:29:57 2017 +0200
Add a trivial test for agency log compaction.
commit e12fd3b46833419d7b436eeadd7246304324b891
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 24 23:26:03 2017 +0200
First part of cleanup of agency log compaction.
Now the right compaction snapshots are taken and persisted.
Furthermore, the right log entries and old snapshots are removed
after compaction, both the volatile and the persisted ones.
The readDB and spearHead stay unchanged at compaction time as it
should be.
commit d59901aea0c3ca31ef253299d2adc3353b79e664
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 24 12:18:26 2017 +0200
Remove unused member variable.
commit 6c5f23eb7b42d9f20d4dadb2932a63add99f9c76
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 09:45:41 2017 +0200
Document apply methods in Store with comments.
commit 670899f72d215e0fcc0ca0389cea9250a291e83b
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 09:45:18 2017 +0200
Intentionally keep one log entry more to prevent empty log.
commit 660f61029917bbc2ce1fae3e4fc903095b023297
Author: Max Neunhoeffer <max@arangodb.com>
Date: Mon May 29 09:44:42 2017 +0200
Improve plausibility checks at compaction time.
commit e2802e4b36d1f67d8361c1d8b0c92fbff696f439
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 25 00:51:51 2017 +0200
Sort out (re-)building of agency K/V store(s) from compaction snapshots.
This is in case of (re-)start, becoming a leader and when serving
/_api/agency/store.
commit 12b43f1b91284a1185390d6dcfbd1e838522d392
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 25 00:50:54 2017 +0200
Set compaction interval to 200 and keep to 0.
This is way to small but tests should run with it.
Will later increase numbers again.
commit c8b9a37a690b8e7e8bfa1276a3f9ba4b6b5a9c27
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 24 23:29:57 2017 +0200
Add a trivial test for agency log compaction.
commit cf0c8c1fff666f76411082f87efe685a412ecebb
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 24 23:26:03 2017 +0200
First part of cleanup of agency log compaction.
Now the right compaction snapshots are taken and persisted.
Furthermore, the right log entries and old snapshots are removed
after compaction, both the volatile and the persisted ones.
The readDB and spearHead stay unchanged at compaction time as it
should be.
commit 0a4255359a57b8686133e6014e2b82b8079f36fa
Author: Max Neunhoeffer <max@arangodb.com>
Date: Wed May 24 12:18:26 2017 +0200
Remove unused member variable.
* 'devel' of https://github.com/arangodb/arangodb: (25 commits)
Revert "Implement shardorganizer...Cluster with 1 DBServer possible again"
remove unused variable
Move FM.listJson into aardvark
Remove unused legacy FM test
Implement shardorganizer...Cluster with 1 DBServer possible again
Make sure the bundle doesn't exist before loading it from db
More meaningful object logging
Response -> IncomingResponse
Updated plain cache to properly count evictions.
More realistic binary document test
Simplified Foxx self healing (#2511)
fix superfluous re-creation attempts for collections
Made shortest path OOM test deterministic
added documentation
sync ENGINE file
adjusted ui to internal cluster api
fix usage of su
Fixing various issues
Fixing catch test
fixed vertexId getter
...
* Implement new self-heal
* Add error codes for 503, service missing/outdated
* Detect changes to service via rev
* Pretty print incoming response object in log
* Improve Foxx cluster resilience
Fixes#2083Fixes#2384Fixes#2408
Addresses #1892
* Port old Foxx API
* Implement single-file services
* Add console.errorStack/warnStack/infoStack helpers
* Simplify serviceInfo validation
* Extract github/upload logic into Aardvark and old FM API
* Move generator logic into Aardvark
* Move zip/js buffer logic into FM core
* Add Foxxmanager tests
* Send empty response when no README
* Disambiguate script arg format
Historically we allow passing an array of positional arguments or an arbitrary first argument.
This is surprising behaviour, so we should just always treat the value as a first argument.
* Rebuild bundle in development mode
* Nicer HTTP docs formatting
* Create Foxx HTTP docs
* Simplify service upload handling
* Remove inline swagger docs
* Implement public download route
* Consistency
* Rebuild aardvark
* Move bundle route into /_api/foxx/_local
* Rebuild Swagger API docs
* Add changes to CHANGELOG
* More docs
* 'engine-api' of https://github.com/arangodb/arangodb: (46 commits)
Merged RocksDBPathBasedIndex into RocksDBVPackIndex.
fix handling of externals
fix toArray()
fix returning documents
Added hash-, skip list, persistent indexes
harmonize filtering - filter on full filename.
mini modifications
standardize starting of servers for replication tests
Edge index compliant with design doc
another fix for #2404
Moved „removeLargeRange“ method to commons
Added length byte to IndexValue keys.
fixed issue #2404
Added drop to primary index
Added drop() to indexes
Fixes a coordinator crash in Clustered Traversal. In case of failover during query planning.
Fixed edge index
Expand on CORS documentation
Let ArangoDB handle the allow-credentials header
make the authentication tests run on directories
...
jsteemann
Michael Hackstein
Dan Larkin-York
Simon Grätzer
Jan
shivamdave24
Simran
Tobias Gödderz
Frank Celler
Max Neunhöffer
Andrey Abramov
Wilfried Goesgens
Jan Christoph Uhde
Vasiliy
Kaveh Vahedipour
Matthew Von-Maszewski
Heiko
Manuel B
Andrey Abramov
Frank Celler
Andreas Streichardt
m0ppers
Dan Larkin
Jan Christoph Uhde
Alan Plum
baslr