1
0
Fork 0
Commit Graph

1267 Commits

Author SHA1 Message Date
Dan Larkin-York 976a88c723 Make request forwarding (load-balancing) non-blocking (#9948)
* Make request forwarding (load-balancing) non-blocking.

* Apply suggestions from code review

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Address review comments.

* Use more appropriate method.
2019-09-09 20:32:37 +02:00
Simon 6b7fb0994e Non-Blocking reads (#9883) 2019-09-05 19:26:01 +02:00
Tobias Gödderz b7f8f01a22 Feature/split libarangoserver (#9670)
* Made usage of named variables for targets a little more consistent

* Bumped minimum cmake version

- removed policies superseded by minimum cmake version
- fixed static linking due to new policy CMP0060

* Fixed library suffixes for windows

* Extracted libs arango_mmfiles and _rocksdb from arangoserver

* Replaced target variables by constants

* Extracted arango_cluster_engine from arangoserver

* Extracted llhttp from arangoserver

* First successful split of arangoserver

* Moved enterprise files to enterprise

* Again only optionally include RestTestHandler and AcceptorUnixDomain

* Cleaned source files from other libraries

* Removed old commented sources

* Split off a small third library

* Fixed boost dependency for cluster engine

* Added some missing dependencies

* Added arango_geo dep, and -J on windows

* Began to split off an arango_graph lib

* Moved more files to arango_graph

* Do not set /J globally for ATL

* Moved more files to arango_graph

* Moved graph-RestHandlers to arango_graph

* Added arango_geo dependency to _mmfiles and _rocksdb

* Updated graph dependencies

* Split off arango_pregel

* Split off arango_aql

* Added missing boost_system dependency to pregel

* Split off arango_vocbase

* Cleanup

* Added missing boost_system dependency for arango_vocbase

* Split off arango_v8server

* Split off arango_utils

* Minor cleanup

* Split of arango_storage_engine

* Split off arango_indexes and arango_cache

* Fixed some dependencies

* Split off arango_replication

* Resolved two todos

* Split off arango_agency

* Reordered some statements

* Ordered dependency definitions alphabetically

* Cleaned some deps

* Break one cycle, comment on another

* Merge the remaining arangoserver_part[123] sources

* Moved some utils to vocbase to break cycles

* Added missing backtrace dependency to iresearch-s

* Added missing boost dependency

* Added dependency arango_indexes -> arango_geo

* Added deps to arango_cluster_engine, cleaned duplicate deps

* Broke remaining dependency cycles

* Actually, missed one cycle...

* Re-added include for Mac

* arango_cache needs SharedPRNG
2019-09-05 09:37:12 +02:00
Simon 0ee0cebb11 Non-Blocking inserts (#9823) 2019-08-30 09:17:58 +02:00
KVS85 aa4c9230e8 Fix RPM package build 2019-08-29 14:34:22 +02:00
Jan 1176c38aa1
upgrade jemalloc to 5.2.1 (#9816) 2019-08-27 19:31:09 +02:00
Jan 2ee48956f7
upgrade curl to 7.65.3 (#9788) 2019-08-26 18:11:12 +02:00
Jan 00bcc4954c
AQL date functions improvements (#9714) 2019-08-22 12:50:08 +02:00
Simran 8de7bc4744
Fix some CMake warnings related to set() + docstrings (#9304)
* Fix datatypes in set()

* Typos

* Update docs weblink

* Improve program friendly strings

* Fix more bad types and add mandatory docstrings

* Adjust tool friendly strings
2019-08-16 08:52:38 +02:00
Dronplane 2f6fb109f6 Bug fix/issue #9652 (#9697)
* Fixed analyzer properties equality check.

* Added re-normalization of stored analyzer properties

* Fixed analyzer name rules in rest and v8 handlers

* Fixed linux build

* Added tests for implicit system db name. Code cleanup.

* Analyzers access logic moved to separate function
2019-08-15 17:42:44 +03:00
Simon dde1a82fce Network Code V2 (#7962) 2019-08-14 17:25:03 +02:00
Frank Celler aa3d3f8e40
Feature/cleanup ccpcheck (#9665) 2019-08-12 11:11:49 +02:00
Andrey Abramov 70de5feaa7
bug-fix/internal-issue-#591 (#9639)
* update iresearch

* update iresearch

* fix compilation errors

* update iresearch
2019-08-09 18:12:35 +03:00
Jan 7c98893c73
reduce wait timeouts as a mitigation for notifying waiters without holding the required mutex (#9618) 2019-08-02 15:44:38 +02:00
jsteemann 26e48d118e add missing includes 2019-07-29 14:06:30 +02:00
Andrey Abramov ac9f940247
properly handle removals during recovery (#9528) 2019-07-19 22:32:48 +03:00
Jan 1228731251
velocypack library upgrade (#9521) 2019-07-19 15:22:14 +02:00
Simon d79ecd3870 Bug fix/comm task vst10 (#9516) 2019-07-19 11:00:49 +02:00
Andrey Abramov 5bc6eac636
bug-fix/internal-issue-#609 (#9464)
* allow to access last operation tick of the transaction

* modify IResearchLink to rely on last commtted tick

* get rid of writing arangosearch markers to WAL

* further implementation

* local changes in iresearch

* add recovery states

* properly handle link creation during recovery

* adjust test cases

* properly handle nested transactions

* ungreylist recovery tests

* add more recovery tests

* do not use transaction to pass recovery tick

* do not store recoveryTick in MMFilesRecoveryState

* adjust tests

* fix mmfiles

* cleanup

* add context validity check

* fix tests

* fix more tests

* ensure subscription is not being released during commit

* address review comment

* final cleanup

* fix crash

* fix tests

* address test failures

* address review comments

* address review comments

* properly set recovery tick even if no recovery happened
2019-07-19 11:10:56 +03:00
Simon 680971e81a Fix Arangosh windows instability (#9439) 2019-07-17 18:20:37 +02:00
Simon e5507d840f Feature/comm task refactor (#9426) 2019-07-16 09:43:25 +02:00
Andrey Abramov f2ff365206
bug-fix/fix-arangosearch-recovery-mmfiles (#9448)
* start maintenance arangosearch threads after recovery is done

* ensure flush subscription is not being deallocated while in use

* add some tests

* properly determine storage engine

* adjust default view options

* stick to old index meta format in 3.5

* address test failures

* and cluster tests
2019-07-11 11:42:08 +03:00
Simon 241d5d19ef Fuerte changes from comm-task-refactor branch (#9422) 2019-07-08 18:02:35 +02:00
Andrey Abramov 595728ec1b
Bug fix/internal issue #605 (#9405)
* fix recovery

* update iresearch

* more fixes

* address review comments
2019-07-05 13:14:37 +03:00
Jan 5221dc98b9
@maierlars 😍 (#9394) 2019-07-04 09:20:20 +02:00
Jan 2f90f5ec49
update velocypack version (#9379) 2019-07-03 12:14:52 +02:00
Wilfried Goesgens 506a735c1b remove catch, refresh versions (#9390) 2019-07-02 18:04:46 +02:00
Jan 9cb08ded92
make the comparison functions unambiguous (#9349)
* make the comparison functions unambiguous

* added @kaveh's suggestion
2019-07-01 16:35:28 +02:00
Andrey Abramov 40cce2cee0
bug-fix/internal-issue-#565.1 (#9257)
* validate analyzer properties, do not return `null` for identity analyzer properties

* fix some tests

* Fixed tests for new analyzer parameters validation. Added explicit test for analyzer parameter validation

* update iresearch, fix analyzer tests

* fix more tests

* fix tests

* store analyzer properties as vpack

* more compilation fixes

* Updated iresearch

* Tests compilation fixed

* Test run fixes

* Test run fixes

* Fix test run

* Fixed all IresearchTests

* Fixed V8Analyzers tests

* Added ngram and delimeter analyzers vpack config

* Added ngram and stem analyzers vpack parsers. Added tests

* Fixed internal issue #593 Fixed build issue

* Fixed tests

* Fixed Gtest tests

* Changed test run to fix Mac failure

* Mac tests debugging

* Fixed jslint errors

* test tracing added

* Taken account for VPackSlice operator== false negatives
2019-06-28 10:49:01 +02:00
Jan 6354372228
attempt to fix thread-local issues with non-trivial constructions (#9321) 2019-06-26 14:40:07 +02:00
Simon cf7cf0131b Try to fix corruption error (#9258) 2019-06-25 10:18:26 +02:00
Wilfried Goesgens a714290796 use snappy target (#9312) 2019-06-24 17:04:45 +02:00
Wilfried Goesgens 42e3dcf5a1 Bug fix/wintendo compile sequence (#9295)
* move snappy up

* Fix compile warning with MSVC (#9291)

* one more warning

* use right type

* only define variable if we actually use it in the code

* only define variable if we actually use it in the code undo

* use right type
2019-06-19 12:00:48 +02:00
Dronplane 89aa67dcf4 Updated names for text and norm analyzer parameters. Added tests for new parameter names. (#9287) 2019-06-19 09:28:46 +02:00
Wilfried Goesgens 90869c087f fix snowball invocation and dependencies (#9275) 2019-06-18 11:13:13 +02:00
Wilfried Goesgens 6fd3dd024c windows warning as errors, masquerade remaining unfixeable errors (#9249) 2019-06-17 16:21:29 +02:00
Wilfried Goesgens a866f8c6fd update gtest, use json output for reporting (#9193) 2019-06-17 13:46:40 +02:00
Andrey Abramov 1982a55f54
backport bug-fix (#9255) 2019-06-12 16:20:35 +03:00
Andrey Abramov 384c4310b1
Bug fix/internal issue #589 (#9244)
* update iresearch from upstream

* adjust tfidf/bm25 to original behaviour

* update iresearch from upstream

* fix compilation issues

* reduce number of virtual calls

* upgrade iresearch from upstream

* add js test
2019-06-11 17:42:30 +03:00
Jan 091abbec77 don't make RocksDB startup fail with 0-byte log files (#9208) 2019-06-07 15:13:56 +02:00
Dronplane 8ea080355a Fixed race conditions in file_utils::mkdir (#9192) 2019-06-04 18:56:22 +03:00
Simon 1402395f1c Port fuerte changes from network pool branch (#9125) 2019-06-04 16:35:15 +02:00
Dronplane 1e24c79ca3 Bug fix/internal issue #565 (#9154)
* Merged latest iresearch. Removed creation of build-in analyzers from tests

* Updated tests for new analyzer parameter names

* Merged fix for format from iresearch

* Apply suggestions from code review. Corrected typo in variable name.

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* backport changes from IResearch upstream

* fix compilation issues

* backport fix from IResarch upstream

* Fixed analyzer parameter name in tests
2019-06-03 18:20:23 +03:00
KVS85 62d3d325e8 Add /Z7 switch to V8 gyp (#9127) 2019-05-28 17:53:27 +02:00
Jan 40ccafbded
Feature/rocksdb 6.2 (#9095) 2019-05-28 14:26:27 +02:00
Jan 033f411420
Bug fix/issues 27052019 (#9117) 2019-05-28 12:08:25 +02:00
Dronplane cce80f2553 Bug fix/internal issue #572 (#9110)
* Added helper method to DataStore in iresearch link for performing full data store reset. Fixed usage of data store reset.

* Fixed checking directory creation status on windows
2019-05-27 18:08:47 +03:00
Andrey Abramov 8ebcdfda16
bug-fix/internal-issue-#550 (#9064)
* backport fix from iresearch upstream

* move file to proper location

* minor cleanup
2019-05-21 22:46:05 +03:00
Dan Larkin-York d5ecdd143a Convert unit tests to googletest framework (#9034) 2019-05-21 09:17:46 +02:00
Jan Christoph Uhde 970f732dec add exceptions for tokens function. (#8984)
* Provide users of the TOKENS function with errors.

* remove newlines

* fix compare

* fix as asked for

* improve fix
2019-05-16 22:12:43 +03:00
Andrey Abramov 4e12fdb30b
ArangoSearch sorted indexes (#8888)
* move IResearchViewSort into separate unit

* ensure sort is propagated to IResearchLink

* update iresearch and extend some tests

* implement insertion of sorted values, store value comparator in link

* add tests for VPackComparer

* minor cleanup

* fix issue with sort definiton propagation

* propagate error message from index factory

* add integration tests for sorted indexes, single server

* initial implementation of IResearchViewMergeExecutor

* move iresearch::VPackComparer to separate translation unit

* store collection in IndexReadBuffer for IResearchViewMergeExecutor

* set primary sort from optimizer rule

* add some tests for sorted view case

* extend IResearchViewNode tests

* simplify primary sort definition

* pretty print order condition covered by arangosearch in query explanation

* fix compilation with g++-8

* address catch test failures

* fix tests

* adjust arangosearch optimizer rule to optimize sort in case of cluster

* make tests for sorted indexes work in cluster too

* fix tests

* extend js tests for sorted indices

* fix keyword in query explainer

* ensure sort-limit rule works with views

* backport ngram analyzer fix from upstream

* address jslint errors

* modify IResearchViewSort to support boolean direction specification format

* add some tests

* add tests

* add skip functionality

* extend tests

* add more catch tests

* inline some methods

* address jslint errors

* extend catch tests to reproduce the failure

* add some tests for check fullcount

* add more tests

* extend tests

* temporarily comment out broken tests

* extend js tests

* address review comments

* address jslint errors

* update iresearch
2019-05-11 17:58:56 +03:00
Wilfried Goesgens ed01582433 stfu s2 (#8960) 2019-05-10 14:37:07 +02:00
Jan 162bcd205c
Bug fix/fix velocypack error (#8954) 2019-05-09 19:38:00 +02:00
Jan 9c3fd26682
upgrade velocypack library (#8926) 2019-05-08 16:34:20 +02:00
Alan Plum 75bab782bb Upgrade Swagger UI (#8912)
* Upgrade Swagger UI

* Document update process

* Expand update process, define acceptance criteria
2019-05-08 15:28:22 +02:00
Wilfried Goesgens 1ae5f8daec split headers different, simple cast fixes (#8928) 2019-05-08 14:08:35 +02:00
Jan Christoph Uhde e0b073543b add symlinks for windows (#8846) 2019-05-02 17:32:29 +02:00
Jan 0cbdfe9289
Bug fix/vpack update (#8875) 2019-04-30 12:33:26 +02:00
Frank Celler 6abda70550 recheck if mkdir fails 2019-04-21 14:05:30 +02:00
Vasiliy 2de9510051 issue 539.1: backport from iresearch upstream: add support for text normalizing analyzer, make loading shared object ptional in the registry (#8813) 2019-04-20 23:20:15 +03:00
Vasiliy 371509000b issue 537.1: backport from iresearch upstream: ensure key is cached too (#8807) 2019-04-19 16:20:16 +03:00
Tobias Gödderz 11eb393631 Prefer make_shared over new; move buffer in move constructor and -ass… (#8746) 2019-04-12 17:33:02 +02:00
Andrey Abramov e784ded6de
fix invalid assertion (#8731) 2019-04-11 16:26:20 +03:00
Andrey Abramov b5c6af5ab1
bug-fix/internal-issue-#536 (#8726)
* backport fix from iresearch upstream

* update changelog
2019-04-11 14:12:53 +03:00
Jan 675b006ebc
this addresses an issue described in ES-260, and is a forward-port of… (#8696) 2019-04-09 11:53:20 +02:00
Simon 2b594bdab5 Reduce # of memcpy from storage engine layer (#8685) 2019-04-08 22:43:29 +02:00
Jan 5897baa984
added db.<collection>.getResponsibleShard() (#8683) 2019-04-08 16:10:45 +02:00
Jan 86458580d8
updated jemalloc version to 5.2.0 (#8684) 2019-04-05 22:01:56 +02:00
Tobias Gödderz d1416bed74 AQL by line: IResearch blocks as executors (#8387)
* Added RemoteExecutor skeleton

* Moved RemoteBlock implementations to ExecutionBlockImpl<RemoteExecutor>

* Remove unnecessary include to avoid unused function warnings

* Fixed gcc compile error

* Moved Scatter/Distribute block implementations to their new Executor versions

* Applied clang-format

* Added factory, infos and a skeleton for the unordered view executor

* Removed assert based on wrong assumption

* Added members from IResearchViewBlockBase to IResearchViewExecutor

* Moved more code into the ViewExecutor, hopefully enough to produce a working version now

* Added missing reset code, made produceRow work mostly correct

* Removed superfluous parentheses to get more useful output from Catch

* Ported fix 923b6e81ac723d1fe37f8e7bf1ab81149f3a08ef

Original commit message was:
Fixed a race condition in RemoteBlock which was triggered during
shutdown overtaking getSome.

* Applied review comments

* Inject input row instead of an item block + pos into the expression context, plus fixed some tests

* Adapted test. Search tests are now green.

* Do not ask upstream when already DONE

* Removed `limit` from next()

* Simplified code that could handle producing more than one document

* Minor readability change

* Solved two TODOs noted in the review

* Removed leftover references to DistributeNode members in the DistributeBlock

* Reverted removal of "exhausted"

* WIP: Implemented variant with scorers

* Fixed compile errors of the last commit

* Fixed some asserts and calculations

* Fixed violated assertions

* Moved files from IResearch/ to Aql/

* Replaced recursive call with a loop

* Worked on a few TODOs

* Removed IResearchViewBlock

* Set input registers correctly

* Eliminated dependency to the Node in the Executor

* Don't misuse the volatility variables for initialization

* Extended a TODO note

* Removed obsolete includes

* Removed an obsolete include from the tests

* Added missing include

* Read PKs in batches

* Fixed merge conflict

* Fixed merge conflict

* Restrict prefetching of PKs to the number of rows in the current output block

* Fixed merge

* Fix IResearch ASan errors

* Revert "Restrict prefetching of PKs to the number of rows in the current output block"

This reverts commit e0fd8698a3.

* Revert "Read PKs in batches"

This reverts commit c06c4d7a36.

* Began some small step refactoring to introduce batch-reading correctly

* Extracted method fillBuffer

* Extracted method evaluateScores

* Minor changes

* Read data from iresearch index in batches

* Replaced std::deque<IndexResult> buffer by a new class

* Solved minor TODOs

* Fixed last commit

* Fixed merge conflict

* Removed accidentally re-added view blocks

* Implemented review comments
2019-04-04 23:40:10 +03:00
KVS85 2d5bf58c94 Remove USE_IRESEARCH definition and usage (#8613) 2019-03-28 17:23:51 +01:00
Jan e0ebfefddd
added missing fix for NowNanos from upstream RocksDB (#8602) 2019-03-28 02:25:47 +01:00
Jan Christoph Uhde 523019c8d3 update rocksdb to 6.0.fb (#8576) 2019-03-26 15:52:35 +01:00
Wilfried Goesgens 6f9091878c where did the % go? (#8342) 2019-03-07 14:38:29 +01:00
Wilfried Goesgens 19abe0d63a use the specified python interpreter here too (#8330) 2019-03-06 15:41:08 +01:00
Andrey Abramov d86fb726e6
ArangoSearch index encryption (#8214)
* update iresearch

* add index encryption support

* update iresearch

* encrypt ArangoSearch index using rocksdb encryption provider

* fix merge issue

* update iresearch

* fix cmake

* minor fixes after update

* update iresearch

* partially revert changes from commit 492d05c1f1 since they cause iresearch configuration failures

* Update arangod/IResearch/IResearchRocksDBLink.cpp

Co-Authored-By: gnusi <gnusmas@kvolitek.ru>

* partially revert changes from commit 492d05c1f1 since they cause iresearch configuration failures

* fix cmake configuration for MAX

* another attempt to fix cmake on mac

* Fix 3rdparty cmake for Windows
2019-03-01 15:56:40 +03:00
Frank Celler 8e814744a5
fix buffer overflow (#8188) 2019-02-27 18:02:20 +01:00
Jan 1798036ea0
Bug fix/optimizations 18022019 (#8180) 2019-02-19 19:24:04 +01:00
Wilfried Goesgens 08c3710c29 when we DO_NOTHING tell the implementer he may stop invoking us (#8201) 2019-02-19 19:20:18 +01:00
Wilfried Goesgens 492d05c1f1 Feature/upgrade v8 7.1.302.28 (#8088) 2019-02-19 11:15:34 +01:00
Jan 820ba5dd7c
micro optimizations (#8156) 2019-02-13 16:14:28 +01:00
Vasiliy ea48789c54 issue 466.4: implement persistence of IResearchLink WAL flush marker (#8059)
* issue 466.4: implement persistence of IResearchLink WAL flush marker

* address enterprise test failure

* address test failures

* treat missing collections/indexes as potentially removed

* change view asertios to match collection assertions

* convert assertions to exceptions since they are possible

* revert assertion removal since assertions are actually valid for coordinator

* address scenario where link is dropped twice

* check for data store state before returning error

* revert last change since it's not valid for MMFiles
2019-02-04 16:36:13 +03:00
Jan 0053a72a6a update velocypack (#8076) 2019-01-31 17:31:14 +01:00
Vasiliy 22ea8bc293 issue 466.2: update iresearch to revision 9f179cea9037c702115d0884c88e742a30921cd1 (#8010)
* issue 466.2: update iresearch to revision ee15310f1450035c7d3ba664c6886ef7a39eb6bb

* update iresearch to commit 9f179cea9037c702115d0884c88e742a30921cd1
2019-01-24 16:11:04 +03:00
Jan 6944115de6
various optimizations (#7994) 2019-01-21 13:40:39 +01:00
Simon f748aee240 Added collectAll, updated fuerte (#7949) 2019-01-16 11:31:08 +01:00
Jan 2ae738792a
willi (#7948) 2019-01-15 09:41:00 +01:00
Jan fa7de56cf8
upgrade to boost 1.69.0 (#7910) 2019-01-09 17:17:33 +01:00
Jan b40b88d73b
upgrade bundled curl library to version 7.63 (#7846) 2019-01-08 11:27:59 +01:00
Jan 98ff9621bf
prevent duplicate attributes being generated by AQL queries (#7837) 2019-01-02 12:37:17 +01:00
Jan f4f61e2947
Bug fix/issues 23122018 (#7847) 2019-01-02 11:11:03 +01:00
Vasiliy db07e1ca5e issue 511.6: update iresearch to 6e870904eae5eaf891359ed06e7332666c02cb19 (#7819)
* issue 511.6: update iresearch to 6e870904eae5eaf891359ed06e7332666c02cb19

* backport: add back required MSVC command replacement

* try to address MSVC link issues

* backport: move post-build step into a separate custom target

* another attempt to address cmake build issues
2018-12-21 15:00:35 +03:00
Wilfried Goesgens c3e73ba45b upgrade rocksdb to V5.18.X (#7792) 2018-12-19 19:34:55 +01:00
Andrey Abramov 7f4740b335
update ArangoSearch consolidation policy (#7801) 2018-12-19 01:55:08 +03:00
Vasiliy 8e7f0df775 issue 511.4: ensure IResearchLink can be initialized without view as required for db-server, minor code fixes and cleanup (#7786)
* issue 511.4: ensure IResearchLink can be initialized without view as required for db-server, minor code fixes and cleanup

* remove unnessesary checks

* revert last change due to test failures

* fix typo in change revert

* try to address random test failures
2018-12-18 18:45:11 +03:00
Max Neunhöffer 2846bdc393
Fix S2 for aarch64. (#7770) 2018-12-17 09:59:33 +01:00
Andrey Abramov 43dbe065d5
fix cmake parameters coincidence (#7774) 2018-12-15 01:56:32 +03:00
jsteemann 884f085917 remove now-unused file 2018-12-12 10:28:38 +01:00
Vasiliy dff5d53c31 issue 511.1: move iresearch data-store from IResearchView to IResearchLink (#7695)
* issue 511.1: move iresearch data-store from IResearchView to IResearchLink

* backport: address build failures on some compilers

* address MacOS build issue
2018-12-10 14:00:35 +03:00
Simon 8b0e0b0b75 Upgrade S2 Library (#7519) 2018-11-29 15:49:17 +01:00
Jan 389b677161
use jemalloc for client tools as well (#7506) 2018-11-28 19:42:52 +01:00