mirror of https://gitee.com/bigwinds/arangodb
13 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Jan
|
2f90f5ec49
|
update velocypack version (#9379) | |
Jan Christoph Uhde
|
c3f7961b88 | apply unique log ids (#8561) | |
Tobias Gödderz
|
e3f5a88762 |
Forbid ambiguous casts to and from ResultT (#8147)
* Forbid ambiguous casts to and from ResultT * Reformat * Changed enabled_if checks to check for implicit casts to Result * Added comments |
|
Dan Larkin-York
|
f4c2347fbd | Make Result final (#8157) | |
|
Jan
|
1a1f6935c6
|
🚨 Fix clang warnings (#8123) | |
Lars Maier
|
12eebb15fe |
Feature/new server infra (#7733)
* Decoupled IO handling from Scheduler. * Fixed SSL start up bug. * Replaced Scheduler with new worker farm implementation. * Added minimal statistics and info string for Scheduler. * Added support for timed submissions. * Updated delayed submission api. Updated code that used timers. * Extracted new Scheduler into a virtual parent class. The implementation can now depend on the usecase. * Signal handler now working. * Changed threads names, `_stop` is atomic, check for failure during thread start + exception handling like old scheduler did. * Commented on source code and added TODOs. * Played around with start-stop-conditions * Play around with start stop condition. * start stop cond * Sart Stop Conditions * Removed bad cv_status check. * Bug fix: now compare the actual objects instead of pointer values. Setup t1 and t2 depending on the thread id. * Moved most of the stuff now unrelated to the Scheduler to GeneralServer. Got rid of JobGuard. * Instead of waiting for a thread to terminate, put it on a clean up list and check for its termination in each supervisor run. * Allow detaching long running threads. * Fixed test mock. * Updated the WorkHandle logic. Removed post functions. * Fixed crash when obtaining shared_ptr from this in destructor. * Added lost mutex. * Fixed memory leak. * Fixed merge bug. * Changed a lot of code to optimize the scheduler. * Fixed bug of invalidated iterator. Dont remove task on shutdown at different places. Let scheduler threads run until queue is empty. * Only by value calls to queue. * Added options again. * Clean up of code. * UI Request Lane added. * Bug fixes in Scheduler. * Applied reformat. * Use sigaction. |
|
Frank Celler
|
ac9f375fb5 | big reformat | |
|
Tobias Gödderz
|
de4f5587ae |
Gharial rewrite in C++ (#5631)
* Built a C++ skeleton REST handler for gharial, with fallback to the JS handler * Moved aql::Graph to graph::Graph * Added complete edge definitions to Graph Also: - some cleanup - used forward-declarations in headers - use Graph in graph rest handler * Handle graph lookup failures according to the test suite * Added GET vertex * Bugfixes in ResultT - Added missing #include - Fixed move semantics * Move central code of readVertex to GraphOperations * ResultT fixes and complements * Implemented a graph cache * Added and used graph cache to the rest handler * Added GET edge * Added DELETE edge * Extracted some code * Added PATCH and PUT for both edge and vertex * Moved update/replace transaction code to GraphOperations * Added stub routes for POST and a TODO note * Added a test checking that deleting a vertex removes all incident edges as well * Added a test checking that deleting a vertex does not remove edges in non-graph collections * fixed compiler warnings and errors * Began work on DELETE vertex For this, added a V8Context to allow for AQL queries to use subtransactions * Continued work on DELETE vertex (still WIP) * prep for graph post routes * fixed removeVertex operation (aql) * added post vertex and post edge gharial routes * wasSynchronoues flag changed * gharial post c++ handler, naming conventions * added gharial tests * temporary disabled cache (because not completed), added graph property read functions * added c++ gharial list vertex collections * added c++ gharial graph config * added c++ gharial list graphs * added graph manager class * first implementation of create graph in c++, WIP * changed error messages * added etag to create graph api, still multiple edge definition check missing * finished POST /_api/gharial/<graph> * WIP - DELETE /_api/gharial/<graphName> * added DELETE /_api/gharial/<graphName> validation, still missing correct response * gharial delete * fixed delete gharial lock * finished DELETE /_api/gharial/<graphName> * added routes for graph based vertices and edge definitions * improved delete route * added add new edge definition to existing graph * patch edge definition in a graph, still <WIP> * finished edit edge definition route * code changes due to devel code changes * added remove edge definition route * added vertex delete function * added todo note regarding one drop collection issue * add oprhan collection to graph route implemented * Added a test * Updated a comment * Several minor changes * Minor changes during review * Changes during review * Changes during review * Bugfix: orphans may be null or omitted * Bugfix: resolve externals * minor code changes * seperated graph class to independent component classes * seperated graph class to independent component classes * removed log output * fixed create collection behaviour in a cluster environment * fixed enterprise graphs behaviour in c++ gharial api * removed log output * formatting * improved error handling, fixed a linux compile bug * more result refactoring * more result type cleanup * fixed wrongly defined test * result handling * error handling * more refactoring * Bugfix: avoid race condition in cluster when creating collections * updated graph documentation * added graph related static strings * static strings, new method to create options for gharial created collections * Some minor cleanup * more use of static strings * minor code changes, review * added missing parseint * removed gharial foxx, added js common module, added v8 general graph module * correct use of virtual method * more v8, js general graph, broken state * more v8 graph functions * fixed editEdgeDefinition, added drop function * fixed drop behaviour * added _list, _exists * added c++ rename graph collections, added v8 + graph module function * Added a regression test * added graph._deleteEdgeDefinition, v8, server * more v8g * added _removeVertexCollection * added _extendEditDefinitions * todo, need to add a helper sort method for a local defined relation * fixed test * fixed lots of tests, added more client functions, _addVertexCollection on client module is still broken * added more client graph functions, all tests green * more client functions * add del edge def route * Fix use after move * Minor changes in client general-graph.js module * Make a copy before sorting (don't touch the argument) * Minor changes and some additional asserts in graph tests * Consistently set parameter defaults * Renamed static strings * Remove superfluous function * Made comment more verbose * Minor changes in general-graph-common.js * Added missing template arguments * Fixed community build * Cleanup in editEdgeDefinition * Regression test & bugfix: comparison of edge definitions didn' order from and to * Fixed errors introduced by merge * Minor changes in v8-general-graph.cpp * Fixed test failure due to wrong error code in CE * added missing id field * Added permission checks for graph._create * Removed assertion that is no longer valid * Moved removeGraph from GraphOperations to GraphManager * Allow C++ implementation of graph._drop to handle smart graphs * Flush js client db cache after creating/dropping collections via the general graph module * Added _deleteEdgeDefinition to the general graph client module * WIP: Added permission checks for drop graph * Fixed permission checks for drop graph * Added permission checks for other graph operations * Bugfix: assert edge definitions are returned in order * Some cleanup * Removed unused method * Minor improvements in GraphManager * Fixed a type in general-graph common module * Most useful fix of all times ever: Do not auto cast from bool to int and alternate error/noerror by this * Added the initial keyword to StaticStrings * Added a new error code, used whenever a user tries to inject a documentcollection as a relation into the graph, which is invalid * Some GraphManager/Ops/Graph cleanup. Less Slice parsing, more usage of GraphObjects * Test edgeDefinitions in graphs with a defined ordering * GraphClass Layout cleanup * Do not test error messages, use codes instead * Recreated backwards compatibility of Graph Creation Permission errors * Changed error-code if edgedefinition is used twice * Added a StaticString for the GraphName * Renamed graphToVpack => graphForClient * Partly fixed graph-api test to work with better error messages. Still red: The edgeDefinitions are now sorted, the test is supposed to sort his own list, but appearently does not do so. Under investigation * Added a new error code that rejects injection of differently sharded smart collection into smartgraph. Should be more helpful to our users * graph createCollectionOptions now require an open object to be cross-called from enterprise. Made enterprise switch for creation of graph more elegant. * Updated graphs.cpp * Massive refactoring. Made Factories for graphs to make SmartGraph much more transparent. Also reduced amount of multiple implementations of the same stuff. Killed vocbase/graphs use GraphManager instead. Removed usage of GraphCache, was not completely implemented anyway and only partially used, which is bad at the moment. Option for later improvement never the less * Adapted JS code to now really use c++ variants. ALso included 3 Classes: Graph, SmartGraph and GraphModule. * Fixed undefined behaviour in Remove Vertex. Fixed smartgraph sharding if one collection already exists. * Removed DEBUG output * Removed DEBUG logs * Removed dead code * Fixed Graph EdgeDefinition test, they now have a different ordering. * Added a test when adding a vertexCollection that it is actually valid in the graph * Client Graph API now correctly sends `orphanCollections` and not `orphans` * Let GraphOperations modify the graph in-place. It should now properly handle edgeDefinitions. * Added initial cid StaticString * Included the vocbase in fromPersistence creation of Graphs. Only required to enhance 3.3 SmartGraphs on the fly. * Fixed internal error message * Fixed compiler isses originiated from merging * Removed unused imports * Regenerated generated file |
|
|
Tobias Gödderz
|
fc3e11dbbc |
Async AQL (#5806)
* Modified header to new initializeCursor API * Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet * Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later * First attempt to simulate thread waiting over information within the query * Small fix to allow for isDirect handlers to go to sleep. * Waiting in the necessary places now for the async request to be send. * Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here? * Include cond-var header * Fixed mutex/cond_var usage * Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested * Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries. * Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts * Fixed endless virtual recursion * Implemented new getOrSkip API in SortBlock * Implemented new getOrSkip API in LimitBlock * Initilaize all variables * Fixed logic bug in SubqueryBlock * getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly! * Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime. * Added IndexBlock::skipSome and IndexBlock::getSome * getBlock now returns its old return value along with the state * Switch from getBlockOld to getBlock in IndexBlock::skipSome * Switch from getBlockOld to getBlock in IndexBlock::getSome * ShortestPathBlock::skipSome is not implemented! Added a regression test * Attempt to fix SubQueryResult memory management * Fixed LIMIT Block * Moved from ShortestPathBlock::getSomeOld to ::getSome * Implemented ASYNC api on SingletonBlock * Adapted EnumerateCollectionBlock to new async API * Fixed FilterBlock and adapted return block to async API * Adapted NORESULTS block to async AQL api. * Adapted Modification Blocks to async API * Fixed some initialize cursor functions to reset values required during get/skipSome * First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still * Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place. * Adapted CalculationBlock to async API * Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN * Adapted EnumerateListBlock to Async AQL api * Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now * Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing * Adapted IReasearchViewBlock to ASYNC AQL API * Fixed SortingGatherBlock in WAITING state. * Adapted IResearch ExecutionBlockMock to Async API * Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍 * Implemented tonly heoretically reachable function of non void function. * Fixed last commit * Added inline TODO comments * fix warning * Fixed a clearing logic bug in RemoveNodes * Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter. * remove unused methods * Fixed failure test * implement skipping * Moved the Query Waiting out of the ExecutionEngine. * changed one of the collect blocks * Removed _upstreamState from ExecutionBlockMock, that is in the base-class now * Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api. * do not check "hasMore" if not necessary * Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type (still uses getBlockOld) * Save state to resume in DistinctCollectBlock::getOrSkipSome * Extracted redundant code * fixed some ops * added one more test * fix endless blocking * fix compile error * fix test * Refactored HashedCollectBlock::getOrSkipSome * Return blocks to the manager * Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome * remove unused shutdown calls, simplify ownership for expressions * Removed superfluous variable * Capture const variable by value * Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome * Added a working version of SortedCollectBlock::getOrSkipSome Has yet to be cleaned up * Removed isTotalAggregation special treatment * On no input, return a group of nulls (instead of no group at all) * Bugfixes * Simplified code * Move return to the end, eliminate duplicate code * Corrected skipped count in HashedCollectBlock * Aligned getNextRow() implementations * Added comments * some cleanup * fix potential memleak * Bugfix * Fixed failure tests * Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome * Replaced hasMore with an async implementation (mostly) * Removed getBlockOld() * Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState) * RemoteBlock now uses the async hasMoreState route * remove job queue * options * Bugfixes in the async implementation of LimitBlock * LimitBlock::getOrSkipSome now always skips when calculating the fullcount * fix compile warnings * restrict threads * Fixed api of Waiting ExecBlockMock. Unused yet * Made SortedGatherBlock async-capable * Removed nonEmptyIndex hack * Removed duplicate traceGetSome~ calls, moved all to getSome * Added asserts before replacing getNr*Registers * Added a TODO note and a comment * Removed getSomeWithoutRegisterClearoutOld() * Removed skip() * Removed common code by using getNr*Registers() * Use getNr*Registers() in the TraversalBlock as well * started to add lane * started to add lane * added lane * completed lane * removed debug output * fixed merge * Began working on a test suite for AQL tracing/profiling * Added more tests and asserts in aql-profiler * Made some ExecutionBlocks final * Added a type enum to all blocks and the per-block stats * Add block type to stats nodes when tracing AQL on block level * Removed initializeCursor call from instantiateFromPlan * Avoided additional getSome calls after DONE * Added more profiler tests * Refactored ExecutionBlock::getOrSkipSome and fixed two bugs - set _upstreamState also when skipping - explicitly use xecutionBlock::getHasMoreState() * Bugfix: update state * Reuse parent _skipped wherever possible; rename where not (LimitBlock) * Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code * Implemented missing virtual function (with USE_FAILURE) * Reset neccessary values during initializeCursor * Simplified code in EnumerateListBlock a little * Added a test for DistinctCollectBlock in aql-profiler * Avoid redundant getSome calls in DistinctCollectBlock * fix compilation * Fixed DistinctCollectBlock profiler test * Added a second profiler test for the DistinctCollectBlock * Added a profiler test for EnumerateCollectionBlock * Bugfix in EnumerateListBlock * added --server.fifoN-size * Simplified EnumerateCollectionBlock::getSome * Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE * Fix testEnumerateCollectionBlock1 for mmfiles * do not pass by reference * Fixed compile error * fixed merge conflicts * Added profiler tests for EnumerateCollectionBlock * Test fix for mmfiles * Fixed IResearch tests * Bugfix in DistinctCollectBlock and a regression test * Updated comment * Bugfix for query statistics in cluster * Check plan in distinct test * Fix aql-profiler tests in cluster * Removed unused line / bugfix for single server test runs * This commit implements waking up of AQL queries. (#5651) * Non-compiling intermediate commit for handover. * Make branch compile again * Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested! * Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine. * First successful query wakeup. * The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler * renamed finalizeExecute to shutdownExecute * Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly * fix audit log * Removed callback from deleteQueryCursor. This cannot be waiting * use CONDITION_LOCKER * removed yet another thread-local variable * Fixed forward declaration * Made RestAqlHandler repeatable * Use defer to close the query in RestAqlHandler. Now waiting will close the query as well. * Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler * Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case * Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded. * Removed reaction to existing nolockheaders in Coordinator Query Planning Phase * Removed incorrect assertion. * Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly. * Fixed UserManager, now deletes nolock headers properly * Swing to the Symphony of Destruction * Forgot about community build... * Fixed compiling of Catch tests * Fixed community build * need thread for size * Made the restSimpleHndler repeatable * Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor * Reactivated StreamingCursors * Removed debug output. * Fixed false query continuation * Reset thread output to non-debug * Added missing return statements * Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers. * Fixed hidden merge conflict * Bugfix in aql-profiler.js: use plan.nodes order, not stats * Added two profiler tests for filter * Avoid too many getBlock calls in the FilterBlock * Removed debug output * RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility. * Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete * Added another test for FILTER, and one test for the HashedCollectBlock * Added more tests for HashedCollectBlock; avoid unneccessary getSome calls * Added an profiler IndexBlock test * IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls * Added a second test profiling IndexBlock * Added a third test for IndexBlock * Moved general code to module * Moved noncluster tests into a separate file * Split aql-profiler testsuite into three files * Added profiler tests for LimitBlock * Added a test for NoResultsBlock * Added profiler tests for TraversalBlock * Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still * Optimized TraversalBlock getSome calls due to new (nightly) test results * Fixed std::min calls I broke * Let shutdown calls in AQL wait, if the query is executed successfully. * Fixed queryResult going out of scope * fix compile error through merge conflict with devel * Fixed compiler warning "mismatching tags" * Removed debug log output * Added TODO notes * Fixed test fail due to devel merge * Fixed some invalid sync waiting implementations * Added a profiler test for SortBlock * Added profiler tests for SortedCollectBlock * Fixed bug introduced by devel merge * Fixed Remoteblocks ignoring errors! * Added some more continue Callbacks in used places. And removed debug log * Removed debug log output * Suppress clang warnings * Bugfix: use of invalid stack pointer * Bugfix: RemoteBlock::shutdown now sends code as int, not string * Revert "Suppress clang warnings" This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0. * Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome * Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed * Allow copy elision |
|
|
Jan
|
e6dcff8360
|
Bug fix/cleanup 2805 (#5477) | |
Simon
|
17b1a2aafb | Rest middleware refactoring (#5332) | |
Michael Hackstein
|
4ede998186
|
Bug fix/compile issues on mac (#5280)
* Mvoved unnecessary std::move * Replaced size_t by uint64_t * Fixed usage of <array> |
|
|
Tobias Gödderz
|
8c87f51429 | Feature/fix inconsistent distribute shards like job (#4743) |