1
0
Fork 0
Commit Graph

41 Commits

Author SHA1 Message Date
Tobias Gödderz 82b0a5e745 De-inline in AQL (#10025)
* Enable IPO (LTO) via CMake

* Use separate IPO_ENABLED variable

* Enabled IPO for more binaries

* Enabled IPO for arangobackup

* Suppress an MSVC warning (though it is not unjustified...)

* Fix static builds, broken due to Policy CMP0060 introduced in cmake 3.3

* Removed policies superseded by minimum cmake version

* Disable IPO with google tests due to a g++ bug

* Disable IPO with google tests only on AUTO

* Disable warning for correct line

* Begin de-inlining Aql

* de-inlining Aql: 2nd batch

* de-inlining Aql: 3rd batch

* de-inlining Aql: 4th batch

* Moved code out of ifdef

* Enabled IPO for additional libs

* Fixed some problems found with IPO enabled

- Fixed ODR violation in Pregel
- Removed unused code in ClusterMethods
- Avoid possible uninitialized variable usage

* Set IPO globally, except for 3rdParty libs

* De-inlined AstHelper again (was undone due to a previous merge conflict)

* Added missing .cpp file

* Fixed compilation with MSVC

* Removed superfluous includes from deinlined AQL header files

* Added includes for size_t

* Deleted outdated files

* De-inlined files that were undone during the merge

* Removed includes from files that were undone during merge

* Fixed merge conflicts
2019-09-25 11:20:03 +02:00
Jan Christoph Uhde 0b8c75c7b7 one shard db - devel (#9395) 2019-09-23 15:48:37 +02:00
Jan 9635df56eb
added "peakMemoryUsage" query result figure (#7952) 2019-01-17 13:58:18 +01:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Jan 1973022d00
Bug fix/refactor find emplace (#7197) 2018-11-02 17:18:47 +01:00
Simon fc204a2b3e fix profiling code (#6190) 2018-08-20 12:59:07 +02:00
Tobias Gödderz fc3e11dbbc Async AQL (#5806)
* Modified header to new initializeCursor API

* Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet

* Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later

* First attempt to simulate thread waiting over information within the query

* Small fix to allow for isDirect handlers to go to sleep.

* Waiting in the necessary places now for the async request to be send.

* Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here?

* Include cond-var header

* Fixed mutex/cond_var usage

* Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested

* Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries.

* Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts

* Fixed endless virtual recursion

* Implemented new getOrSkip API in SortBlock

* Implemented new getOrSkip API in LimitBlock

* Initilaize all variables

* Fixed logic bug in SubqueryBlock

* getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly!

* Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime.

* Added IndexBlock::skipSome and IndexBlock::getSome

* getBlock now returns its old return value along with the state

* Switch from getBlockOld to getBlock in IndexBlock::skipSome

* Switch from getBlockOld to getBlock in IndexBlock::getSome

* ShortestPathBlock::skipSome is not implemented! Added a regression test

* Attempt to fix SubQueryResult memory management

* Fixed LIMIT Block

* Moved from ShortestPathBlock::getSomeOld to ::getSome

* Implemented ASYNC api on SingletonBlock

* Adapted EnumerateCollectionBlock to new async API

* Fixed FilterBlock and adapted return block to async API

* Adapted NORESULTS block to async AQL api.

* Adapted Modification Blocks to async API

* Fixed some initialize cursor functions to reset values required during get/skipSome

* First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still

* Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place.

* Adapted CalculationBlock to async API

* Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN

* Adapted EnumerateListBlock to Async AQL api

* Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now

* Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing

* Adapted IReasearchViewBlock to ASYNC AQL API

* Fixed SortingGatherBlock in WAITING state.

* Adapted IResearch ExecutionBlockMock to Async API

* Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍

* Implemented tonly heoretically reachable function of non void function.

* Fixed last commit

* Added inline TODO comments

* fix warning

* Fixed a clearing logic bug in RemoveNodes

* Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter.

* remove unused methods

* Fixed failure test

* implement skipping

* Moved the Query Waiting out of the ExecutionEngine.

* changed one of the collect blocks

* Removed _upstreamState from ExecutionBlockMock, that is in the base-class now

* Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api.

* do not check "hasMore" if not necessary

* Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type

(still uses getBlockOld)

* Save state to resume in DistinctCollectBlock::getOrSkipSome

* Extracted redundant code

* fixed some ops

* added one more test

* fix endless blocking

* fix compile error

* fix test

* Refactored HashedCollectBlock::getOrSkipSome

* Return blocks to the manager

* Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome

* remove unused shutdown calls, simplify ownership for expressions

* Removed superfluous variable

* Capture const variable by value

* Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome

* Added a working version of SortedCollectBlock::getOrSkipSome

Has yet to be cleaned up

* Removed isTotalAggregation special treatment

* On no input, return a group of nulls (instead of no group at all)

* Bugfixes

* Simplified code

* Move return to the end, eliminate duplicate code

* Corrected skipped count in HashedCollectBlock

* Aligned getNextRow() implementations

* Added comments

* some cleanup

* fix potential memleak

* Bugfix

* Fixed failure tests

* Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome

* Replaced hasMore with an async implementation (mostly)

* Removed getBlockOld()

* Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState)

* RemoteBlock now uses the async hasMoreState route

* remove job queue

* options

* Bugfixes in the async implementation of LimitBlock

* LimitBlock::getOrSkipSome now always skips when calculating the fullcount

* fix compile warnings

* restrict threads

* Fixed api of Waiting ExecBlockMock. Unused yet

* Made SortedGatherBlock async-capable

* Removed nonEmptyIndex hack

* Removed duplicate traceGetSome~ calls, moved all to getSome

* Added asserts before replacing getNr*Registers

* Added a TODO note and a comment

* Removed getSomeWithoutRegisterClearoutOld()

* Removed skip()

* Removed common code by using getNr*Registers()

* Use getNr*Registers() in the TraversalBlock as well

* started to add lane

* started to add lane

* added lane

* completed lane

* removed debug output

* fixed merge

* Began working on a test suite for AQL tracing/profiling

* Added more tests and asserts in aql-profiler

* Made some ExecutionBlocks final

* Added a type enum to all blocks and the per-block stats

* Add block type to stats nodes when tracing AQL on block level

* Removed initializeCursor call from instantiateFromPlan

* Avoided additional getSome calls after DONE

* Added more profiler tests

* Refactored ExecutionBlock::getOrSkipSome and fixed two bugs

- set _upstreamState also when skipping
- explicitly use xecutionBlock::getHasMoreState()

* Bugfix: update state

* Reuse parent _skipped wherever possible; rename where not (LimitBlock)

* Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code

* Implemented missing virtual function (with USE_FAILURE)

* Reset neccessary values during initializeCursor

* Simplified code in EnumerateListBlock a little

* Added a test for DistinctCollectBlock in aql-profiler

* Avoid redundant getSome calls in DistinctCollectBlock

* fix compilation

* Fixed DistinctCollectBlock profiler test

* Added a second profiler test for the DistinctCollectBlock

* Added a profiler test for EnumerateCollectionBlock

* Bugfix in EnumerateListBlock

* added --server.fifoN-size

* Simplified EnumerateCollectionBlock::getSome

* Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE

* Fix testEnumerateCollectionBlock1 for mmfiles

* do not pass by reference

* Fixed compile error

* fixed merge conflicts

* Added profiler tests for EnumerateCollectionBlock

* Test fix for mmfiles

* Fixed IResearch tests

* Bugfix in DistinctCollectBlock and a regression test

* Updated comment

* Bugfix for query statistics in cluster

* Check plan in distinct test

* Fix aql-profiler tests in cluster

* Removed unused line / bugfix for single server test runs

* This commit implements waking up of AQL queries. (#5651)

* Non-compiling intermediate commit for handover.

* Make branch compile again

* Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested!

* Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine.

* First successful query wakeup.

* The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler

* renamed finalizeExecute to shutdownExecute

* Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly

* fix audit log

* Removed callback from deleteQueryCursor. This cannot be waiting

* use CONDITION_LOCKER

* removed yet another thread-local variable

* Fixed forward declaration

* Made RestAqlHandler repeatable

* Use defer to close the query in RestAqlHandler. Now waiting will close the query as well.

* Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler

* Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case

* Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded.

* Removed reaction to existing nolockheaders in Coordinator Query Planning Phase

* Removed incorrect assertion.

* Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly.

* Fixed UserManager, now deletes nolock headers properly

* Swing to the Symphony of Destruction

* Forgot about community build...

* Fixed compiling of Catch tests

* Fixed community build

* need thread for size

* Made the restSimpleHndler repeatable

* Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor

* Reactivated StreamingCursors

* Removed debug output.

* Fixed false query continuation

* Reset thread output to non-debug

* Added missing return statements

* Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers.

* Fixed hidden merge conflict

* Bugfix in aql-profiler.js: use plan.nodes order, not stats

* Added two profiler tests for filter

* Avoid too many getBlock calls in the FilterBlock

* Removed debug output

* RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility.

* Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete

* Added another test for FILTER, and one test for the HashedCollectBlock

* Added more tests for HashedCollectBlock; avoid unneccessary getSome calls

* Added an profiler IndexBlock test

* IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls

* Added a second test profiling IndexBlock

* Added a third test for IndexBlock

* Moved general code to module

* Moved noncluster tests into a separate file

* Split aql-profiler testsuite into three files

* Added profiler tests for LimitBlock

* Added a test for NoResultsBlock

* Added profiler tests for TraversalBlock

* Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still

* Optimized TraversalBlock getSome calls due to new (nightly) test results

* Fixed std::min calls I broke

* Let shutdown calls in AQL wait, if the query is executed successfully.

* Fixed queryResult going out of scope

* fix compile error through merge conflict with devel

* Fixed compiler warning "mismatching tags"

* Removed debug log output

* Added TODO notes

* Fixed test fail due to devel merge

* Fixed some invalid sync waiting implementations

* Added a profiler test for SortBlock

* Added profiler tests for SortedCollectBlock

* Fixed bug introduced by devel merge

* Fixed Remoteblocks ignoring errors!

* Added some more continue Callbacks in used places. And removed debug log

* Removed debug log output

* Suppress clang warnings

* Bugfix: use of invalid stack pointer

* Bugfix: RemoteBlock::shutdown now sends code as int, not string

* Revert "Suppress clang warnings"

This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0.

* Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome

* Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed

* Allow copy elision
2018-07-09 14:24:10 +02:00
Jan f55df6786a
blind commit of work, as I managed to break my local git repository (#5456) 2018-05-25 13:07:28 +02:00
Jan 245c759523
fix incompatibility after fullCount change (#5399) 2018-05-22 11:28:17 +02:00
Jan 07e5750737
Feature/change fullcount behavior (#5285) 2018-05-14 15:46:07 +02:00
Simon 468231efc5 AQL Profiling code (#5165)
* initial start of profiling

* adding profiling code

* Fixing remote block tracing, fixing width and units

* Fixing some tests

* Various fixes

* adressing review comments
2018-04-24 16:17:30 +02:00
Jan 27491cc2ea
simplification of fullcount code (#5075) 2018-04-10 18:03:19 +02:00
Jan 46a56792ed
fix `fullCount` value in some cases in cluster mode (#4723) 2018-03-01 18:33:10 +01:00
jsteemann 6e0da7929b simplify stats a bit 2017-02-03 11:47:09 +01:00
jsteemann 88518f6bf8 added "httpRequests" execution stats figure 2017-02-01 18:03:26 +01:00
jsteemann b96897e296 fixed some issues detected by coverity build tool 2016-12-12 15:41:54 +01:00
Jan Steemann e88460efcb add state "loading collections" to query profile results 2016-10-13 10:31:16 +02:00
jsteemann 4a7c9f0f68 simplified bloated comments 2016-04-09 11:27:30 +02:00
Jan Steemann 6f6f7e23ba executionstats use vpack 2016-03-07 11:20:04 +01:00
Jan Steemann c6fa638b67 fixed issue #1765, issue #1770 2016-03-07 10:49:18 +01:00
Jan Steemann 9046e1831b clang-format 2016-01-27 13:43:46 +01:00
Michael Hackstein a6f58ce14d Use forward declarations for VelocyPack instead of all includes everywhere 2016-01-24 14:29:31 +01:00
jsteemann 431900f17a changed namespace from triagens to arangodb 2016-01-17 00:44:53 +01:00
Jan Steemann 3c69cbd40a proper nesting 2016-01-08 18:14:07 +01:00
Jan Steemann 3ad20c0cae auto-generated headers 2016-01-06 18:41:51 +01:00
Jan Steemann a3b026d9d1 removed useless sections from code, unified include guards 2016-01-06 14:15:22 +01:00
Jan Steemann f422576ee0 clang-format 2016-01-05 17:43:24 +01:00
Max Neunhoeffer e8e6dc8597 More VPackObjectBuilder usages. 2015-12-21 15:22:19 +01:00
Michael Hackstein 757577de42 Updated VPack addObject to openObject 2015-12-09 11:28:54 +01:00
Michael Hackstein fbee0b0859 Started to transform Parts of AQL Query to VPack. Especially the QueryResult is now using VPack internally. 2015-11-12 11:09:35 +01:00
Frank Celler 4c5d57f1b8 merged Exceptions.h and Exception.h 2015-03-23 14:05:19 +01:00
Jan Steemann db2cd7acc9 added `filtered` attribute to AQL execution statistics 2015-02-13 21:37:49 +01:00
Jan Steemann 20e7fe7a7a the great rename: array => object, list => array 2014-12-18 21:07:06 +01:00
Jan Steemann 025d73ae03 re-added support for fullCount 2014-11-01 15:20:05 +01:00
Willi Goesgens 249b89aa09 - Add statistics under all circumstances (also if the status is "exhausted")
- return statistics in all cases (empty statistics in doubt)
2014-10-23 16:39:14 +02:00
Willi Goesgens a8b8fbcf28 Properly handle statistics if multiple getSome() calls are performed. 2014-10-22 18:24:42 +02:00
Willi Goesgens 9558f5aa0a Implement passing of query statistics through cluster. 2014-10-22 17:36:23 +02:00
Jan Steemann 828fe1f1d7 enabled statistics 2014-10-22 12:59:15 +02:00
Jan Steemann 50e17b8c4f disabled optimization due to failing tests 2014-09-19 19:30:57 +02:00
Jan Steemann a58c8661b5 added some query statistics 2014-09-19 18:47:33 +02:00
Jan Steemann 1442b9ebeb pass ExecutionEngine to ExecutionBlock, added stats for ExecutionEngine 2014-08-21 12:33:13 +02:00