1
0
Fork 0
Commit Graph

17663 Commits

Author SHA1 Message Date
Michael Hackstein ccb2a3c62e Fixed invalid log ids 2019-10-02 10:26:38 +02:00
Markus Pfeiffer fc634a2fa0 Replace template <bool blockPassthrough> (#10127)
* Replace template <bool blockPassthrough>

by template <BlockPassthrough blockPassthrough> where BlockPassthrough is an
enum class.

This is mainly for better readability and for some type safety.

* Fixup some implicit conversions to bool
2019-10-02 09:47:09 +02:00
Jan c3423c7db0
Bug fix/disallow sharding on rev (#10123) 2019-10-01 19:42:55 +02:00
Jan 9898606223 fix logIds (#10135) 2019-10-01 20:19:38 +03:00
Dan Larkin-York c0f63856c2 Fix Windows compile error from network namespace. (#10133) 2019-10-01 18:47:00 +02:00
Kaveh Vahedipour a0be102d2c unitended multiple unlocks (#10114) 2019-10-01 16:21:44 +02:00
Max Neunhöffer 9b3f911cb5
Use execvp instead of execv and handle error. (#10102)
* Use execvp instead of execv and handle error.
* CHANGELOG.
2019-10-01 16:19:47 +02:00
Max Neunhöffer fb8bc4bc31
Fix a subtle special case of the shutdown hanger. (#10120)
* Another shutdown problem with CollectionKeysRepository.
2019-10-01 16:11:33 +02:00
Max Neunhöffer ca9affc798
Add backup sizes. (#10104)
* Single server backup size in META and create.
* More size info in backup.
* Finish size and nrfiles for create and list.
* Add nrDBServers and potentiallyInconsistent to BackupMeta.
* Fix API documentation.
* Make Windows compiler happier.
* One more fix for Windows.
* Added creation date.
2019-10-01 13:31:17 +02:00
Simon 9040f1d18a Fuerte + Pregel + Agency = 🥑 (#10110) 2019-10-01 11:19:18 +02:00
Michael Hackstein c6af396188 Make SubqueryEndExecutor compile again on all platforms again 2019-09-30 17:11:05 +02:00
Markus Pfeiffer 32884568cb Feature/subquery end node executor (#10087)
* Add Start/End subquery node skeleton

* Added a test query that triggers complex interna in the future and needs to proof that our work is side-effect free.

* Added first draft of ShadowRow Interface

* Refactored RegisterPlan and pulled it out of the ExecutionNode

* AqlItemBlock now has an additional hidden register to store subquery depth infromation. Right now no shadow rows are created, however thy could now make use of this situation

* Added API and test to insert new shadow rows into an OutputRow.

* Extrated test helper function

* Add template function to statically assert size of type

* Add Start/End subquery node skeleton

* Add Start/End subquery node creation to ExecutionNode

* Cleanup SubqueryEndNode

* Add test file

* Add isEqualTo to ExecutionNode

* Subquery nodes cleanup

* Update ExecutionNode tests

* Added API and tests to consume ShadowRows. Interface is there and compiles with templates, we might need to implement further underlying functions later on as we make progress on ShadowRows, it is only implemented in the minimal way (intentionally)

* Updates to testing code

* Added a test for nested ShadowRows and adapted OutputRow accordingly.

* More updates

* Add optimizer rule

* Fix optimizer rule

* Added additional memory include, seems to be required under GCC not udner CLANG

* it actually helps to save fies before commiting them...

* Fix optimizer rule (again)

* Fixed serialization/Deserialization of AqlItemBlock to contain the hidden subquery register now

* Added a c++ test for AqlItemBlock. It just covers the basics thus far and needs to be improved.

* Fixed toVPack of AqlInputRow for subqueries. Also added serialization / deserialization tests for AqlItemRow

* Added a c++ test for inputAqlItem row serialization => AqlItemBlock deserialization

* Add a public function to get plan from query

* Fix register planning

* Add test for splicing subquery nodes

* add static asserts for sizes of SubqueryStartNode and SubqueryEndNode

* Add comment about statically asserted size

* Fix isEqualTo virtual and override dance.

* Attempt to fix :windows: compile warning

* Fix isEqualTo, remove static_assert_size

* Remove a TODO; the plan owns the ExecutionNodes

* Only add stealPlan to Query when testing

* Remove another static_assert_size

* Fixed mac compile issues

* Added a serialization format to be backwards compatible to 3.5 whenever AQL item blocks need to be send over to remote nodes.

* Handle ShadowAqlItemRow in SingleRowFetcher

* Move some code from header to cpp

* Added more tests for serialization of shadow rows, and fixed a bug revealed by this

* Apply suggestions from code review

Thank you for review!

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review comments. Thanks for spotting

* Fixed hidden merge conflicts

* Removed unused variable

* Added first test on ShadowRows in SingleRowFetcher. Found inconvenience in original code with it.

* Added a test for multi level shadow rows

* Added a test and fixed production for multiple consecutirve shadow rows.

* Added tests for pass through blocks and shadow rows

* Attempt to explain windows on how one can actually cast numbers into other number types...

* Add SubqueryEndExecutor

* Adapted RowFetcherHelper to be able to work on ShadowRows

* Fixed minor error in RowFetcherHelper that causes LIMIT tests to error

* Temporarily disabled test suites (sorry there is only one 'test case' in this suites that does all or nothing tests ...) as it tests a feature we cannot support in this intermediate state, we need to default disable our new rule first.

* IMplemented missing function to copy over SubqueryDepth between blocks

* Seperated assert, to see faster which part is violated

* Update arangod/Aql/AqlItemBlockSerializationFormat.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Disabled subquery splice rule by default

* Forgotten merge marker

* temporarily modified test

* Some Subquery End Executor work

* Fix RowFetcherHelper to work with ShadowRows

* Add tests for SubqueryEndExecutor

* Rewrite SubqueryEndExecutor

* Update test

* Add createBlock code for SubqueryEndExecutionNode

* Some fixups adding ExecutionBlockImpl for SubqueryEndExecutor

* Fix SubqueryEndExecutor

* Update SubqueryEndExecutorTest

* Update endexecutor test

* Fix the endnode executor once more

* Fix some merge fallout

* Update test

* Add missing advanceRow to SubqueryEndExecutor

* SubqueryEndExecutor Tests

* Activate a death test

* Better handling of death test

* RowFetcherTest refactoring Part 1, still ongoing. We use this for AqlItemMatrix and SingleRowFetcher

* Update and cleanup the SubqueryEndExecutor tests

* Check that expected shadow row depths are met

* Remove commented out code in AqlItemBlock.cpp

* Remove a misplaced comment

* Sort out SubqueryEndExecutor includes

* Remove superflous include

* Consistently use NoStats{} in SubqueryEndExecutor

* Make SubqueryEndExecutor properties constexpr

* Reactivate IResearchTests

* Do not copy the accumulator in SubqueryEndNode

* Return a more sensible value for expectedNumberOfRows of SubqueryEndNodeExecutor

* Constify an argument

* Fix ownership problems with the buffer for _accumulator

* Cleanup SubqueryEndExecutor tests

Also add a test that writes to a register other than 0

* Patch RowFetcherHelper to count the number of Items returned instead of number
of calls

* Make MSVC happy
2019-09-30 15:24:13 +02:00
Dan Larkin-York 1d7225b289 Pass connection pool directly to network methods. (#10096) 2019-09-30 12:44:47 +02:00
Kaveh Vahedipour bbe7d424d5 corrected hot backup lock timings (#10076)
* corrected hot backup lock timings
* the lock timeout added to overall unlock timeout]
2019-09-30 10:22:24 +02:00
Jan 2e293b85ca
Bug fix/cppcheck issues (#10105) 2019-09-30 09:27:33 +02:00
Simon cb7bf0314b Use fuerte in RemoteExecutor (#10077) 2019-09-27 16:20:38 +02:00
Jan a088678866
fix broken LDAP tests (#10099) 2019-09-27 16:17:19 +02:00
Jan c0136b49c7
use static_cast instead of dynamic_cast (#10095)
* use static_cast instead of dynamic_cast

* remove virtual inheritance
2019-09-27 12:58:52 +02:00
Dan Larkin-York 5204e95512 Add log suppressor utility classes for cleaner test code. (#10088) 2019-09-27 12:57:38 +02:00
Tobias Gödderz 4ea18d9031 Improved query tracing, especially useful for cluster queries (#10091)
* Improved query tracing, especially useful for cluster queries

* Fix duplicate log id
2019-09-27 10:52:22 +02:00
Tobias Gödderz f120919dfe Implement a constrained sorting gather variety (#10057)
* Added TODO notes

* Pass limit of constrained sort to sorting gather

* Count output rows, and added assertions

* Extracted row-producing code in a reusable way

* Implemented SortingGatherExecutor::skipRows

* Fixed signature

* Fixed an assertion

* Added limit to (de)serialization

* Fixed an assertion

* Fixed an atMost calculation

* In the sort-limit optimization, apply limit to sorting gather as well

* Fixed optimizer rule reporting modification

* Added missing initialization

* Fix: Do not produce more rows than the limit when skipping first

* Fix: pass limit when cloning

* Updated CHANGELOG

* Fixed atMost intricacies
2019-09-27 10:50:06 +02:00
Dan Larkin-York dc23896a01 Make count, figures, revision, and index warmup use non-blocking communication (#10048) 2019-09-27 09:54:01 +02:00
Jan 176cff3eac
remove unused AQL attribute (#10094) 2019-09-26 17:16:43 +02:00
Tobias Gödderz 5e8bdcbc24 Add definitions for static variables in Executor::Properties (#10093) 2019-09-26 14:48:29 +02:00
Jan 9533d1b71c
fix skipInaccessible (#10092) 2019-09-26 14:45:58 +02:00
Jan 9da2cadf83
Bug fix/issue 10078 (#10081) 2019-09-26 12:08:48 +02:00
Wilfried Goesgens 636b2e5d5f fixed version of arangosh result evaluation refactoring - properly count objects (#10080) 2019-09-26 12:07:52 +02:00
Michael Hackstein e89d72d02c
Feature/aql subquery start executor (#10040)
* Add Start/End subquery node skeleton

* Added a test query that triggers complex interna in the future and needs to proof that our work is side-effect free.

* Added first draft of ShadowRow Interface

* Refactored RegisterPlan and pulled it out of the ExecutionNode

* AqlItemBlock now has an additional hidden register to store subquery depth infromation. Right now no shadow rows are created, however thy could now make use of this situation

* Added API and test to insert new shadow rows into an OutputRow.

* Extrated test helper function

* Add template function to statically assert size of type

* Add Start/End subquery node skeleton

* Add Start/End subquery node creation to ExecutionNode

* Cleanup SubqueryEndNode

* Add test file

* Add isEqualTo to ExecutionNode

* Subquery nodes cleanup

* Update ExecutionNode tests

* Added API and tests to consume ShadowRows. Interface is there and compiles with templates, we might need to implement further underlying functions later on as we make progress on ShadowRows, it is only implemented in the minimal way (intentionally)

* Updates to testing code

* Added a test for nested ShadowRows and adapted OutputRow accordingly.

* More updates

* Add optimizer rule

* Fix optimizer rule

* Added additional memory include, seems to be required under GCC not udner CLANG

* it actually helps to save fies before commiting them...

* Fix optimizer rule (again)

* Fixed serialization/Deserialization of AqlItemBlock to contain the hidden subquery register now

* Added a c++ test for AqlItemBlock. It just covers the basics thus far and needs to be improved.

* Fixed toVPack of AqlInputRow for subqueries. Also added serialization / deserialization tests for AqlItemRow

* Added a c++ test for inputAqlItem row serialization => AqlItemBlock deserialization

* Add a public function to get plan from query

* Fix register planning

* Add test for splicing subquery nodes

* add static asserts for sizes of SubqueryStartNode and SubqueryEndNode

* Add comment about statically asserted size

* Fix isEqualTo virtual and override dance.

* Attempt to fix :windows: compile warning

* Fix isEqualTo, remove static_assert_size

* Remove a TODO; the plan owns the ExecutionNodes

* Only add stealPlan to Query when testing

* Remove another static_assert_size

* Fixed mac compile issues

* Initial commit, added class and test stubs

* Added Executor body stub, also added a test for no input. Passes now, as no input is read yet.

* Base implementation + Test of SubqueryStart Executor. First level shadowRows are now produced. Need to add tests for multilevel, and for non-fitting output blocks.

* Added more sophisticated test about shadow row relevance

* Added a test where ShadowRow does not fit into the open block. Also added disabled test on multi level shadowrows. Requires modifications on SingleRowFetcher.

* Added a serialization format to be backwards compatible to 3.5 whenever AQL item blocks need to be send over to remote nodes.

* Handle ShadowAqlItemRow in SingleRowFetcher

* Move some code from header to cpp

* Added more tests for serialization of shadow rows, and fixed a bug revealed by this

* Apply suggestions from code review

Thank you for review!

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review comments. Thanks for spotting

* Fixed hidden merge conflicts

* Removed unused variable

* Added first test on ShadowRows in SingleRowFetcher. Found inconvenience in original code with it.

* Added a test for multi level shadow rows

* Added a test and fixed production for multiple consecutirve shadow rows.

* Added tests for pass through blocks and shadow rows

* Attempt to explain windows on how one can actually cast numbers into other number types...

* Adapted RowFetcherHelper to be able to work on ShadowRows

* Seperated assert, to see faster which part is violated

* Fixed minor error in RowFetcherHelper that causes LIMIT tests to error

* Temporarily disabled test suites (sorry there is only one 'test case' in this suites that does all or nothing tests ...) as it tests a feature we cannot support in this intermediate state, we need to default disable our new rule first.

* Improve check of ExecutionState

* Implemented shadow row handling in subquery start executor. Tests not happy yet

* IMplemented missing function to copy over SubqueryDepth between blocks

* Fixed test code of StartExecutor test. It asserted wrongly on releveance of subquery rows.

* Added instanciation of SubqueryStartExecutorTest. Code does not compile.

* Allow to create SubqueryStartExecutors.

* Update arangod/Aql/AqlItemBlockSerializationFormat.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Disabled subquery splice rule by default

* Forgotten merge marker

* temporarily modified test

* temporarily modified test

* Fixed windows compile

* Fixed unused variable warning

* Removed incorrect optimization in SubqueryStartExecutor
2019-09-26 10:01:52 +02:00
Dan Larkin-York a83c2323c9 Refactor ApplicationServer stack (#9965) 2019-09-25 17:31:59 +02:00
jsteemann d1116c1c72 fix compile error 2019-09-25 13:02:08 +02:00
Jan 8a56ed9a2c
fix sporadically failing one shard test (#10074) 2019-09-25 12:52:31 +02:00
Markus Pfeiffer 0b9dfb4b9b Subquery Start/End Nodes (#9983)
* Add Start/End subquery node skeleton

* Added a test query that triggers complex interna in the future and needs to proof that our work is side-effect free.

* Add template function to statically assert size of type

* Add Start/End subquery node skeleton

* Add Start/End subquery node creation to ExecutionNode

* Cleanup SubqueryEndNode

* Add test file

* Add isEqualTo to ExecutionNode

* Subquery nodes cleanup

* Update ExecutionNode tests

* Updates to testing code

* More updates

* Add optimizer rule

* Fix optimizer rule

* Fix optimizer rule (again)

* Add a public function to get plan from query

* Fix register planning

* Add test for splicing subquery nodes

* add static asserts for sizes of SubqueryStartNode and SubqueryEndNode

* Add comment about statically asserted size

* Fix isEqualTo virtual and override dance.

* Fix isEqualTo, remove static_assert_size

* Remove a TODO; the plan owns the ExecutionNodes

* Only add stealPlan to Query when testing

* Remove another static_assert_size

* Disabled subquery splice rule by default

* temporarily modified test

* Fixed windows compile

* Fixed unused variable warning
2019-09-25 12:35:18 +02:00
jsteemann d0324b8b08 Merge branch 'devel' of github.com:arangodb/arangodb into devel 2019-09-25 11:46:12 +02:00
jsteemann 1aafa1be28 remove unused GraphCache 2019-09-25 11:45:58 +02:00
Tobias Gödderz 82b0a5e745 De-inline in AQL (#10025)
* Enable IPO (LTO) via CMake

* Use separate IPO_ENABLED variable

* Enabled IPO for more binaries

* Enabled IPO for arangobackup

* Suppress an MSVC warning (though it is not unjustified...)

* Fix static builds, broken due to Policy CMP0060 introduced in cmake 3.3

* Removed policies superseded by minimum cmake version

* Disable IPO with google tests due to a g++ bug

* Disable IPO with google tests only on AUTO

* Disable warning for correct line

* Begin de-inlining Aql

* de-inlining Aql: 2nd batch

* de-inlining Aql: 3rd batch

* de-inlining Aql: 4th batch

* Moved code out of ifdef

* Enabled IPO for additional libs

* Fixed some problems found with IPO enabled

- Fixed ODR violation in Pregel
- Removed unused code in ClusterMethods
- Avoid possible uninitialized variable usage

* Set IPO globally, except for 3rdParty libs

* De-inlined AstHelper again (was undone due to a previous merge conflict)

* Added missing .cpp file

* Fixed compilation with MSVC

* Removed superfluous includes from deinlined AQL header files

* Added includes for size_t

* Deleted outdated files

* De-inlined files that were undone during the merge

* Removed includes from files that were undone during merge

* Fixed merge conflicts
2019-09-25 11:20:03 +02:00
jsteemann a2e0fc643b cppcheck 2019-09-24 16:59:00 +02:00
Jan 9f51c03dd7
re-open the PR, queue length counter still underflows (#10065) 2019-09-24 15:39:37 +02:00
jsteemann 02f9eb06f8 make sleeping more accurate 2019-09-24 15:18:21 +02:00
Jan 4f2e1d4054
fixed issue #10062: AQL: Could not extract custom attribute (#10068) 2019-09-24 14:36:53 +02:00
Simon ac2158ee22 Async el cheapo (#10061) 2019-09-24 12:00:13 +02:00
Michael Hackstein 265eb1549d
Feature/aql subquery opt shadow row interface (#9987)
* Added first draft of ShadowRow Interface

* Refactored RegisterPlan and pulled it out of the ExecutionNode

* AqlItemBlock now has an additional hidden register to store subquery depth infromation. Right now no shadow rows are created, however thy could now make use of this situation

* Added API and test to insert new shadow rows into an OutputRow.

* Extrated test helper function

* Added API and tests to consume ShadowRows. Interface is there and compiles with templates, we might need to implement further underlying functions later on as we make progress on ShadowRows, it is only implemented in the minimal way (intentionally)

* Added a test for nested ShadowRows and adapted OutputRow accordingly.

* Added additional memory include, seems to be required under GCC not udner CLANG

* it actually helps to save fies before commiting them...

* Fixed serialization/Deserialization of AqlItemBlock to contain the hidden subquery register now

* Added a c++ test for AqlItemBlock. It just covers the basics thus far and needs to be improved.

* Fixed toVPack of AqlInputRow for subqueries. Also added serialization / deserialization tests for AqlItemRow

* Added a c++ test for inputAqlItem row serialization => AqlItemBlock deserialization

* Attempt to fix :windows: compile warning

* Added a serialization format to be backwards compatible to 3.5 whenever AQL item blocks need to be send over to remote nodes.

* Added more tests for serialization of shadow rows, and fixed a bug revealed by this

* Apply suggestions from code review

Thank you for review!

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review comments. Thanks for spotting

* Fixed hidden merge conflicts

* Attempt to explain windows on how one can actually cast numbers into other number types...

* Update arangod/Aql/AqlItemBlockSerializationFormat.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
2019-09-24 09:09:52 +02:00
jsteemann 6372823b90 Merge branch 'devel' of github.com:arangodb/arangodb into devel 2019-09-23 19:57:20 +02:00
jsteemann 63b4df5138 Revert "Make scheduler enforce queue limits (#10027)"
This reverts commit aa532d6cad.
2019-09-23 19:57:01 +02:00
Simon 2d1c76a55a Refactor Batch API docs (#10015) 2019-09-23 19:56:04 +02:00
Jan aa532d6cad
Make scheduler enforce queue limits (#10027) 2019-09-23 18:21:04 +02:00
jsteemann 8a812ec8c0 use StaticString 2019-09-23 18:17:37 +02:00
Jan Christoph Uhde 0b8c75c7b7 one shard db - devel (#9395) 2019-09-23 15:48:37 +02:00
Tobias Gödderz fbcb4b7152 Avoid overfetch in sorting gather executor (#10047)
* Avoid overfetch in sorting gather executor

* Updated CHANGELOG

* Temporarily disabled sort-limit in cluster when fullCount is enabled, until we can fix SortingGather

* Added a cluster test for sort-limit

* Fixed non-maintainer compile

* Fixed jslint errors

* Disabled sort profiler tests until SortingGather is fixed
2019-09-20 17:00:18 +02:00
Kaveh Vahedipour dd10909dfc rebootIds instead of boot stamps (#10050)
* rebootIds instead of boot stamps
* noexcept wrong as copies are done
2019-09-20 10:26:35 +02:00
Kaveh Vahedipour 49a01e14ff Bugfix/agency lock left behind (#10021)
* fix potentially left behind agency lock
* typo
* do not silently patronise the customer
2019-09-20 10:09:21 +02:00
Jan 41b0717bc9
remove unused code (#10043) 2019-09-19 12:25:25 +02:00
Max Neunhöffer 71c8314107 Add more strong references to pthread stuff. (#10038)
* Add more strong references to pthread stuff.

* Circumvent libgcc/libmusl mistake in multi-threaded detection.

* CHANGELOG.
2019-09-19 12:36:15 +03:00
Jan 415272fe52
add assertions for Cache memory usage not underflowing its minimal usage (#10035) 2019-09-18 14:31:50 +02:00
Max Neunhöffer 51ab5a47d2
Fix a shutdown hanger because of a collection status lock. (#10033) 2019-09-18 13:36:41 +02:00
Tobias Gödderz 4b81ed33b7 Interprocedural optimizations (#9596)
* Enable IPO (LTO) via CMake

* Use separate IPO_ENABLED variable

* Enabled IPO for more binaries

* Enabled IPO for arangobackup

* Suppress an MSVC warning (though it is not unjustified...)

* Fix static builds, broken due to Policy CMP0060 introduced in cmake 3.3

* Removed policies superseded by minimum cmake version

* Disable IPO with google tests due to a g++ bug

* Disable IPO with google tests only on AUTO

* Disable warning for correct line

* Enabled IPO for additional libs

* Fixed some problems found with IPO enabled

- Fixed ODR violation in Pregel
- Removed unused code in ClusterMethods
- Avoid possible uninitialized variable usage

* Set IPO globally, except for 3rdParty libs
2019-09-18 11:29:37 +02:00
Dan Larkin-York a9df1907c3 Fix issue with cache allocation statistics. (#10028) 2019-09-18 10:45:35 +02:00
Tobias Gödderz 2f59a79435 Bug fix/sort limit rule too lax (#10014)
* Added regression test for consecutive constrained sorts

* Bugfix: sort-limit rule skipped too many node types

* Added CHANGELOG entry

* Apply sort-limit rule before single document operations
2019-09-18 09:04:43 +02:00
Tobias Gödderz 7d091523e5 Fixed some problems found with IPO enabled (#10020) 2019-09-16 17:10:13 +02:00
jsteemann 3005e1d869 fix some cppcheck warnings 2019-09-16 15:04:38 +02:00
Dan Larkin-York 8c573549b3 Make truncate use non-blocking communication. (#9980) 2019-09-16 10:46:49 +02:00
Jan 7e36a9ad9b
Bug fix/validate transaction collection (#9986) 2019-09-13 17:00:30 +02:00
Simon 002bb47264 Catch unhandled exceptions (#10009) 2019-09-13 16:03:58 +02:00
jsteemann f857153ad3 lower the log level of an uncritical message 2019-09-13 16:00:19 +02:00
Tobias Gödderz a97000cd04 Additional sort-limit tests (#10010) 2019-09-13 15:18:01 +02:00
Simon 2e91f4fe67 Non block delete (#10005) 2019-09-12 21:44:35 +02:00
Jan a93733da9d
reset _activeTrx as early as possible to release locks (#10004)
* reset _activeTrx as early as possible to release locks

* moved code into if condition
2019-09-12 19:32:17 +02:00
Tobias Gödderz e113e36138 Make AQL's constrained heap play nice with fullCount (#9981) 2019-09-12 18:56:59 +02:00
Simon 98c3b5c7b3 Improve VST Auth errors handling (#9972) 2019-09-12 16:04:25 +02:00
Jan 77034c0441
disable the creation of TTL indexes on sub-attributes (#9994) 2019-09-12 15:29:33 +02:00
jsteemann fb706c162f use steady_clock 2019-09-12 13:32:33 +02:00
Jan 2dde7e2307
Bug fix/allow not like (#9989) 2019-09-12 12:54:02 +02:00
Jan 52f188f617
make sure the input for responsibleShard is always an object (#9993) 2019-09-12 12:50:30 +02:00
Jan 84ad504a6c
corrected several wrong macro names (#9971) 2019-09-11 16:45:32 +02:00
Simon 3d2952b23a Non block modify (#9963) 2019-09-11 15:37:02 +02:00
Max Neunhöffer 7d5313611f
Fix a shutdown busy loop after main if two exceptions collide. (#9978) 2019-09-11 14:47:49 +02:00
Jan aada04e75b
don't assert/crash when using an unknown collection/shard (#9959) 2019-09-11 12:03:13 +02:00
Dronplane 311b616542 Load library flag set to false for analyzer features (#9973) 2019-09-11 12:44:02 +03:00
Michael Hackstein d251c3316d
Fixed and enabled an accidentially disabled assertion (#9969)
* Fixed and enabled an accidentially disabled assertion

* Removed debug include
2019-09-11 09:04:31 +02:00
Jan 4cb08d2a0d
fix wrong assertion in AstHelper.h (#9964) 2019-09-10 15:46:41 +02:00
Frank Celler a0852d33a2
fixed typo in define (#9945) 2019-09-10 11:34:42 +02:00
Kaveh Vahedipour 3011846025 broken hotbackup list with (#9956)
* fix broken list of non existing id
2019-09-10 10:17:24 +02:00
Markus Pfeiffer d25ea0e377 Cleanup ServerState.cpp (#9923)
* Remove unused function mkdir()

* Remove some outdated comments

* Add include guards to AgencyStrings.h

* Move check for initialized agency out of registerAtAgencyPhase1

* Whitespace cleanup

* Address two minor comments from review
2019-09-10 10:06:16 +02:00
KVS85 4fc39dd4b3
Debug segfault reimplementation (#9940)
* Changed debugSegfault to debugTerminate

* Fix *nix compilation

* More data for broken reconnect

* Remove circumventCores completely

* Fix forgotten calls
2019-09-09 23:07:45 +03:00
Dan Larkin-York 976a88c723 Make request forwarding (load-balancing) non-blocking (#9948)
* Make request forwarding (load-balancing) non-blocking.

* Apply suggestions from code review

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Address review comments.

* Use more appropriate method.
2019-09-09 20:32:37 +02:00
Jan 3a59abd1dc
various issues reported by cppcheck (#9962) 2019-09-09 20:32:04 +02:00
Tobias Gödderz 9e9fa3a4f1 Bug fix/allow agency ops in active failover (#9881)
* allow agency operations in active failover too

* Added regression test

* Allowed more calls in active failover for the health endpoint to work

* Updated CHANGELOG
2019-09-09 16:53:57 +02:00
Jan 98fea326ea
handle K_SHORTEST_PATHS where it was forgotten (#9954) 2019-09-09 14:54:31 +02:00
Tobias Gödderz 1d65a37cc8 Feature/agency paths framework (#9933)
* Added a skeleton framework for agency paths

* Added some basic tests

* Added missing header

* Move to shared_ptrs to parents

* Added a virtual base class

* Sprinkle some final specifiers

* Moved some code into class Path and simplified tests

* Added root() function

* Added assertions

* Added /arango/Supervision

* Replaced PathComponent by StaticComponent and added DynamicComponent

* Added /arango/Target

* Added /arango/Current

* Added /arango/Plan

* Added a TODO note

* Added the last missing top-level paths in /arango/

* Added aliases, cleaned up comment

* Fixed some specifiers

* s/typeof/decltype/
2019-09-09 14:04:12 +02:00
Jan 91cdcb840b
Bug fix/fix transaction errors (#9929) 2019-09-09 11:02:06 +02:00
Jan 9f078f363d
Bug fix/harden database creation against duplicate name (#9951) 2019-09-09 11:00:33 +02:00
Jan 35526deddf
improve handling of FILTERs that are known to always produce true or false (#9941) 2019-09-09 10:15:15 +02:00
Jan edfdf9e9f1
make smart graphs transaction-aware (#9912) 2019-09-06 11:21:20 +02:00
Simon 6b7fb0994e Non-Blocking reads (#9883) 2019-09-05 19:26:01 +02:00
Kaveh Vahedipour ad36adc354 Feature/hotbackup list retries (#9924)
* retry hot backup listing for 2 minutes in cluster before giving up
2019-09-05 16:44:43 +02:00
Max Neunhöffer d45757db51
Sort out google cloud storage as remote. (#9918)
* Add successful method to ClusterCommResult.
* Improve error forwarding for cluster internal communication.
2019-09-05 13:36:50 +02:00
Markus Pfeiffer 753ff4aa67 Feature/atomic database creation 2 (#9826) 2019-09-05 12:38:07 +02:00
Jan 5a433bc99c
attempt to crashes on hard shutdown of the MMFiles engine (#9900) 2019-09-05 12:27:14 +02:00
Jan 0b4d6cebb3
avoid lookup plus sequent insert/emplace for the fast path (#9903) 2019-09-05 12:25:56 +02:00
Tobias Gödderz b7f8f01a22 Feature/split libarangoserver (#9670)
* Made usage of named variables for targets a little more consistent

* Bumped minimum cmake version

- removed policies superseded by minimum cmake version
- fixed static linking due to new policy CMP0060

* Fixed library suffixes for windows

* Extracted libs arango_mmfiles and _rocksdb from arangoserver

* Replaced target variables by constants

* Extracted arango_cluster_engine from arangoserver

* Extracted llhttp from arangoserver

* First successful split of arangoserver

* Moved enterprise files to enterprise

* Again only optionally include RestTestHandler and AcceptorUnixDomain

* Cleaned source files from other libraries

* Removed old commented sources

* Split off a small third library

* Fixed boost dependency for cluster engine

* Added some missing dependencies

* Added arango_geo dep, and -J on windows

* Began to split off an arango_graph lib

* Moved more files to arango_graph

* Do not set /J globally for ATL

* Moved more files to arango_graph

* Moved graph-RestHandlers to arango_graph

* Added arango_geo dependency to _mmfiles and _rocksdb

* Updated graph dependencies

* Split off arango_pregel

* Split off arango_aql

* Added missing boost_system dependency to pregel

* Split off arango_vocbase

* Cleanup

* Added missing boost_system dependency for arango_vocbase

* Split off arango_v8server

* Split off arango_utils

* Minor cleanup

* Split of arango_storage_engine

* Split off arango_indexes and arango_cache

* Fixed some dependencies

* Split off arango_replication

* Resolved two todos

* Split off arango_agency

* Reordered some statements

* Ordered dependency definitions alphabetically

* Cleaned some deps

* Break one cycle, comment on another

* Merge the remaining arangoserver_part[123] sources

* Moved some utils to vocbase to break cycles

* Added missing backtrace dependency to iresearch-s

* Added missing boost dependency

* Added dependency arango_indexes -> arango_geo

* Added deps to arango_cluster_engine, cleaned duplicate deps

* Broke remaining dependency cycles

* Actually, missed one cycle...

* Re-added include for Mac

* arango_cache needs SharedPRNG
2019-09-05 09:37:12 +02:00
Michael Hackstein 1b1763c496
Added a stop to Network feature. There is a race condition on the gar… (#9892)
* Added a stop to Network feature. There is a race condition on the garbage collection post

* Added cleanup for analyzer test and view test

* Update tests/js/client/shell/shell-analyzer-rest-api.js

Fixed usage of wrong command
2019-09-05 08:31:11 +02:00
Jan 412e5280bc
fix audit test (#9886) 2019-09-04 17:44:00 +02:00
Jan 5572675106
Bug fix/remove base directory from include path (#9885) 2019-09-04 17:39:01 +02:00
Kaveh Vahedipour e6cb5d0f16 DropCollection is a FAST_LANE action and should not need much time or else retry. (#9893) 2019-09-04 15:34:02 +02:00
Simon 72396e6b79 Bug fix/windows network (#9895) 2019-09-03 20:22:23 +02:00
Kaveh Vahedipour d22386dbb6 agency lockup when removing 404ed callbacks and leadership preparation (#9847)
* Fix unused return value.
2019-09-03 16:59:41 +02:00
Dronplane 3bba2cd7db Fix for internal issue #633 (#9884)
* Made ANALYZER BOOST and MIN_MATCH non deterministic to prevent premature calls

* Returned Deterministic flag. Added context function implementation for cons arguments calls.

* Fixed filter volatility detection

* Added assertion

* Fixed formatting.  Removed redundant checks.

* Applied review suggestion

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
2019-09-03 15:03:49 +03:00
jsteemann d1b1b6726f Revert "Feature/resource usage (#9647)"
This reverts commit 76afb5001e.
2019-09-03 12:48:02 +02:00
Frank Celler 575f818498 fixed includes for Mac 2019-09-02 16:57:31 +02:00
Jan f1ca1d11ff
Bug fix/make graphs transaction aware (#9840) 2019-09-02 16:53:00 +02:00
Kaveh Vahedipour cced54cbc7 forceBackup is now allowInconsistent (#9850)
* forceBackup is now allowInconsistent
2019-09-02 10:32:58 +02:00
Jan 8e9f75cb5a
don't issue a warning when transaction log iterator reports more data (#9868) 2019-09-02 09:40:28 +02:00
Simon 2f9d1f8c51 Baby operations in cluster fix (#9870) 2019-09-02 09:36:37 +02:00
Jan d42490aa42
honor return values of important methods (#9859) 2019-08-30 23:11:19 +02:00
Wilfried Goesgens 76afb5001e Feature/resource usage (#9647)
* spawn an arangosh that checks cluster nodes are responsive

* attempting to kill 0 may end up bad for us, prohibit it

* don't trip over this optional stuff.

* fix killing of spectator

* only write one line per minute

* fix function name

* print sHitlist of tests, add resource usage

* fix lint

* start refactoring result processing into its own library

* /proc reading only works on linux

* new result processing library

* clean up test loading flow, remove unused blacklist implementation

* measure SUT start/stop + test time

* lint

* more non-test variables

* improve test runner naming

* finish gathering statistics about start/run/stop

* use internal stats code for external processes too

* fix status in sample testcase

* tell that procdump is gone - it seems this happenes in reality without coredumps being written

* refactor test result analyzing, add utility to work with analyzers of json dumps from CI systems

* fix testcase error handling

* also run watcher for agency

* lint

* fix default options for test added default options

* if arguments occur multiple times, update value to an array

* start implementing some test analyzers

* fix color

* write json report unconditional

* add analyzer that searches for long setup/teardown tests

* enable thread dump; log error if we fail to acquire the threads

* disable procdump for the agency

* output error if we fail to get process stats

* fix json invocation

* add debug logging

* only add buildType if that directory actually exists

* trap sleepers

* trap sleepers

* trap sleepers

* trap sleepers

* trap sleepers

* disable thread counting on the wintendo

* disable thread counting on the wintendo

* more measurements

* more measurements

* one more place

* one more place

* remove debugging code

* Update js/client/modules/@arangodb/process-utils.js

Co-Authored-By: Dan Larkin-York <danielhlarkin@users.noreply.github.com>

* Update js/client/modules/@arangodb/process-utils.js

Co-Authored-By: Dan Larkin-York <danielhlarkin@users.noreply.github.com>

* Apply suggestions from code review

Co-Authored-By: Dan Larkin-York <danielhlarkin@users.noreply.github.com>

* rename as sugested by @dan

* undo debug changes

* lint, make cluster health monitor optional per default

* fix spawning of active failover SUT, fix cluster health monitor shutdown

* use std::find_if (as @dan sugested), fix log ids

* fix scope of before-time
2019-08-30 16:20:07 +03:00
Jan 68291ce617
fixed issue #9862 (#9864) 2019-08-30 14:46:42 +02:00
Jan 7220af9602
cover more cases of "unique constraint violated" issues during replication (#9830) 2019-08-30 10:37:32 +02:00
Jan 13e1327723
fix return value when reboot id cannot be retrieved (#9857) 2019-08-30 10:34:47 +02:00
Jan ec3043dd8f
check for duplicate server endpoints on cluster startup (#9860) 2019-08-30 10:29:46 +02:00
Simon 0ee0cebb11 Non-Blocking inserts (#9823) 2019-08-30 09:17:58 +02:00
Heiko edba8f73fb added ensureCollections to orphan create method (#9833) 2019-08-29 23:01:53 +02:00
Jan 30b36a2a42
fix return value checks (#9852) 2019-08-29 20:38:53 +02:00
Simon 9a43b28f8f Improve ExecContext usability (#9806) 2019-08-28 19:05:23 +02:00
Dronplane 622d7a3edf Bug fix/issue #9795 (#9825)
* Fixed NOT IN clause in ArangoSearch

* Update CHANGELOG
2019-08-28 14:43:04 +03:00
Jan 86a496b8a3
Bug fix/fix in not in workaround (#9813) 2019-08-27 12:51:10 +02:00
Frank Celler 9aa12d7cdb
use LDAP_PATH instead of fixed one (#9812) 2019-08-27 10:13:17 +02:00
Frank Celler 6cb71f5746 Revert "use LDAP_PATH instead of fixed one"
This reverts commit b2344a6430.
2019-08-26 19:13:39 +02:00
Frank Celler b2344a6430 use LDAP_PATH instead of fixed one 2019-08-26 19:12:29 +02:00
Jan 2ee48956f7
upgrade curl to 7.65.3 (#9788) 2019-08-26 18:11:12 +02:00
Jan f3d24e0f67
remove now-obsolete warning (#9803) 2019-08-26 13:20:28 +02:00
Jan f01385e969
Bug fix/multi bugs (#9789) 2019-08-26 13:11:59 +02:00
Dan Larkin-York 1cc31e1085 Minimize unnecessary dropping of followers due to poorly set synchronous replication timeouts (#9798) 2019-08-26 11:20:02 +02:00
Jan Christoph Uhde 292be42910 count intermediate commits (#9753) 2019-08-26 11:07:57 +02:00
Simon 9ef10919c2 Improve performance of non-intersection queries (#9797) 2019-08-26 10:58:42 +02:00
Dan Larkin-York 88c9074a39 Add user restrictions for streaming transactions. (#9793)
* Add user restrictions for streaming transactions.

* Remove overzealous check.
2019-08-23 15:31:50 +02:00
Kaveh Vahedipour 737dac3f3d remove 404-ed callbacks from agency (#9709)
* remove 404-ed callbacks from agency
* revert callback documents to published api :)
* array needs be inside so that multiple unobserves to same key are possible
2019-08-23 10:13:17 +02:00
Jan 00bcc4954c
AQL date functions improvements (#9714) 2019-08-22 12:50:08 +02:00
Dronplane 9559c8a80e Bug fix/internal issue #622 (#9781)
* Added analyzer cache invalidation for dropped database

* Fixed jslint reported errors
2019-08-21 23:34:24 +03:00
Jan ba02dc4b4d
mark AQL functions FULLTEXT, NEAR, WITHIN, WITHIN_RECTANGLE as cacheable (#9771) 2019-08-21 18:10:55 +02:00
jsteemann 2c47fd9943 reduce wait timeout, use move 2019-08-21 17:32:30 +02:00
Jan df504bb67a
fix potential spurious wakeups in scheduler code (#9770) 2019-08-21 09:42:38 +02:00
Dronplane 49dfb0a96e Bug fix/issue #9612 (#9764)
* Fixed ViewExecutionNode retrieval with deleted documents present in view

* Ported solution from 3.4 branch

* Changed index store in collection from vector to set. To make reversable indexes always last to execute

* Fixed re-enter hung

* Index storage fix

* Made index order deterministic

* Fix Mac build

* Added tests for index reversal

* Fixed Mac build

* Code cleanup

* Some cleanup

* Removed some redundand copy constructor calls

* Applied review comments

* Applied review comments
2019-08-20 19:34:23 +03:00
Jan ec586667e4
downgrade WARN messages to INFO level (#9761) 2019-08-20 18:28:57 +02:00
Jan 6ffb6067fb
slightly reorder boolean members to reduce struct sizes (#9762) 2019-08-20 18:28:09 +02:00
Jan fda4eea43f
make index selection more deterministic (#9735)
* make index selection more deterministic

* serialize indexes used by traversal with their estimates

* serialize selectivity estimates for shortest path nodes too

* fix assertion that doesn't hold true in unit tests

* fix test
2019-08-20 14:45:13 +02:00
Lars Maier 75411971f5 [devel] Move Shard Bug 4567124 (#9746)
* Fixed abort in moveshard in the case new leader is not in Current.
* Fixed special case where to server is an old follower.
* Updated Changelog.
2019-08-20 11:37:56 +02:00
Jan d68b4b4da0
Bug fix/fix invalid cast (#9755) 2019-08-20 10:32:08 +02:00
Simon f88e5b56b0 Enforce stricter transaction limits (#9740) 2019-08-20 09:57:41 +02:00
Dan Larkin-York f7d41ad8fb Check scheduler queue return value (#9754) 2019-08-19 19:31:55 +02:00
Jan 6fb6fb9f2f
dont fill cache on truncate (#9721) 2019-08-19 17:53:10 +02:00
Jan 346edaec3a issue #9654: make `--rocksdb.max-write-buffer-number` work (#9750)
* issue #9654: make `--rocksdb.max-write-buffer-number` work

* fix the logic
2019-08-19 09:42:08 -04:00