1
0
Fork 0
Commit Graph

3066 Commits

Author SHA1 Message Date
Jan 152bc7c556 better killability of cluster AQL queries (#10360) 2019-11-06 15:33:44 +03:00
Andrey Abramov f4e6538edd Bug fix/internal issue #647 (#10292) (#10334)
* Bug fix/internal issue #647 (#10292)

* extend replication tests

* ensure proper replication order

* fix tests

* address review comments

* address test failures

* extend dump tests

* fix analyzers tests

* more fixes

* extend tests

* enhance tests

* adjust tests

* use enum instead of flags (part 1)

* cleanup

* use enum instead of flags (part 2)

* get rid of flags for views

* get rid of flags for collections

* completely get rid of collection flags

* fix replication test

* refactor index flags

* fix tests and move AnalyzerPool out of class scope

* fix tests

* adjust log levels

* add tests

* remove debug logging

* remove noexcept from `equalAanalyzer`

* extend cluster tests

* fix cluster tests

* add tests for views and smart graphs

* address jslint errors

# Conflicts:
#	arangod/Cluster/ClusterMethods.cpp
#	arangod/Cluster/v8-cluster.cpp
#	arangod/IResearch/IResearchAnalyzerFeature.cpp
#	arangod/IResearch/IResearchAnalyzerFeature.h
#	arangod/IResearch/IResearchLinkMeta.cpp
#	arangod/RestHandler/RestAnalyzerHandler.cpp
#	arangod/StorageEngine/PhysicalCollection.cpp
#	arangod/VocBase/Methods/Indexes.cpp
#	tests/IResearch/IResearchAnalyzerFeature-test.cpp
#	tests/IResearch/IResearchFeature-test.cpp
#	tests/IResearch/IResearchLinkHelper-test.cpp
#	tests/IResearch/IResearchLinkMeta-test.cpp
#	tests/IResearch/IResearchQueryOptimization-test.cpp
#	tests/IResearch/IResearchViewDBServer-test.cpp
#	tests/IResearch/IResearchViewSort-test.cpp
#	tests/V8Server/v8-analyzers-test.cpp
#	tests/VocBase/LogicalDataSource-test.cpp

* Update CHANGELOG

* Update CHANGELOG
2019-10-30 17:01:18 +03:00
Max Neunhöffer ac689fc725 Allow to ignore superuser traffic with an option. (#10289)
* Allow to ignore superuser traffic with an option.

Also: Fix some accounting bugs.

* CHANGELOG.

* Implement separate accounting for superuser and user traffic.

* Fix VST case with no authentication.

* Add description of new user-only traffic statistics.

* Add figures for user traffic.

* Simplify forwarding accounting.

* Remove brackets in metrics names.

* Finalize naming of metrics.
2019-10-24 13:11:51 +03:00
Jan 7eb53e0645 fix underflow in copying code (#10263) 2019-10-16 16:23:28 +03:00
Jan 4425c7d73a ignore files in .bin when copying js files (#10146)
* ignore files in .bin when copying js files

* updated CHANGELOG

* updated CHANGELOG
2019-10-02 19:03:43 +03:00
Jan 13dae8cace Feature 3.5/sharding config (#10111)
* added options

`--cluster.max-number-of-shards`
`--cluster.default-replication-factor`
`--cluster.min-replication-factor`
`--cluster.max-replication-factor`

* fix creating satellite collections

* added CHANGELOG entry and tests

* guard against invalid replicationFactor changes when changing properties

* fix validation

* remove stray print

* Update arangod/Cluster/ClusterFeature.h

Co-Authored-By: Michael Hackstein <michael@arangodb.com>
2019-10-01 16:43:57 +03:00
Max Neunhöffer 328f46e3d6 This merges hotbackup and atomic-db-creation into 3.5. (#9968)
* Squashed commit of feature-3.5/hotbackup_devel.

This puts hotbackup into 3.5.

* Port atomic-database-creation-2 to 3.5.

* Remove some wrongly ported code.

* Fix compilation.

* Fix a manual merge error.

* Remove a feature from the mocks which does not exist in 3.5.

* Add some code which was forgotten in manual merge.

* Fix a problem introduced in a manual merge.

* reuse function

* Address some whitespace issues that came up in review

* aardvark should not create the frontend collection

* create _frontend collection from c++

* recheckAndUpdate Callback in CollectionWatcher

* Wrong author ;)

* rm outdated todo

* Update lib/Basics/VelocyPackHelper.h

Co-Authored-By: Michael Hackstein <michael@arangodb.com>

* use logger unique id, use startup logger

* not needed

* optimized vector shardid method

* do not create _modules collection lazy anymre

* Formatting.

* Assert instead of if/TRI_ASSERT(false)

* Don't use exceptions as control structure

* Re-add READ_LOCKER that got lost in translation

* Fix audit log in case database creation fails early.

* legacy sharding

* Add CHANGELOG entry.

* Retry database cancellation indefinitely

* Do not use exceptions in UpgradeTask

* DropCollection is a FAST_LANE action and should not need much time or else retry.

* Remove superflous addition of LdapFeature

Proudly brought to you by ASAN tests

* Fixed check for distributShardsLike sharding on _system database

* Fixed compile issue on tests

* Removed assertion that seems to be not correct yet on devel.

* Sort out google cloud storage as remote. (#9918)

* Add successful method to ClusterCommResult.
* Improve error forwarding for cluster internal communication.

* Feature/hotbackup list retries (#9924)

* retry hot backup listing for 2 minutes in cluster before giving up

* Enable api by default.

* fix broken list of non existing id (#9957)

* Fix compilation after manual merge.

* Fix another compilation problem.

* Yet more fixes for compilation.

* More compilation fixes.
2019-09-11 13:13:54 +03:00
Jan f8c156a44f make graphs transaction-aware (#9855)
* make graphs transaction-aware

* simranification

* fix tests with mmfiles
2019-09-02 22:06:10 +03:00
Jan Christoph Uhde 44ab3f04b5 3.5 -- intermediate commit stats (#9780)
* count intermediate commits

add transactions statistics struct

add tests

put values in object / split test is multiple smaller tests

* fix broken condition

* fix test

* remove print

* fix windows build

* fix mmfiles

* fix cluster

* add structs in docublocks

* we do not know about intermediate commits on coordinators

* add documentation

* add isCluster function

* add serverStatistics to 3.5 client

* update documentation
2019-08-26 16:56:21 +03:00
Dronplane 5effbfd35a Porting fix for issue #9652 (#9727)
* Porting fix for issue #9652

* Update CHANGELOG
2019-08-16 13:27:51 +03:00
Wilfried Goesgens 18748882d8 Feature 3.5/add js isolate mem statistics (#9569)
* add statistics about isolate memory usage, add isolate numbers to logs

* add statistics about isolate memory usage, add isolate numbers to logs

* add documentation

* Update Documentation/DocuBlocks/Rest/Administration/get_admin_statistics.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update arangod/Statistics/Descriptions.cpp

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update arangod/Statistics/StatisticsWorker.cpp

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update lib/ApplicationFeatures/V8PlatformFeature.cpp

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update lib/V8/v8-globals.h

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update arangod/Statistics/Descriptions.cpp

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* fix time handling as sugested by @jsteemann

* cleanup

* Update Documentation/DocuBlocks/Rest/Administration/get_admin_statistics.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/DocuBlocks/Rest/Administration/get_admin_statistics.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update arangod/Statistics/StatisticsWorker.cpp

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
2019-07-26 16:36:20 +03:00
Jan 9777f53878 Bug fix 3.5/add db list transactions function (#9575)
* added missing function db._transactions(), and equivalent REST API route
GET /_api/transaction

* updated

* use performRequests

* updated CHANGELOG

* added tests, fixed segfault

* use throwNotRunning

* use read transactions for testing (mmfiles blocks with multiple write
transactions on the same collection)
2019-07-26 16:34:57 +03:00
Michael Hackstein d5840c125a Bug fix 3.5/min replication factor (#9524)
* Cherry-pick minReplicationFactor

* Bug fix/failover with min replication factor (#9486)

* Improve collection time of IResearchQueryOptimizationTest

* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it

* Added some assertion son minReplicationFactor

* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled

* added minReplicationFactor to the user interface, preparation for the collection api changes

* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo

* added minReplicationFactor usage to tests

* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME

* minReplicationFactor now able to change via collection  properties route

* fixed wrongly assert

* added minReplicationFactor to the graph management ui

* added minReplicationFactor to the gharial api

* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.

* adjusted description of minReplicationFactor

* FollowerInfo Refactoring

* added gharial api graph creation tests with minimal replication factor

* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests

* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor

* Debug logging

* MORE Debug logging

* Included replication fast lane

* Use correct minreplicationfactor

* modified debug logging

* Fixed compileissues

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* Revert "MORE Debug logging"

This reverts commit dab5af28c0.

* Revert "MORE Debug logging"

This reverts commit 6134b664bd.

* Revert "MORE Debug logging"

This reverts commit 80160bdf3b.

* Revert "MORE Debug logging"

This reverts commit 06aabcdfe1.

* Removed debug output

* Added replication fast lane. Also refactored the commands as i cannot take it any more...

* Put some requests of RocksDBReplication onto CATCHUP Lane.

* Put some requests of MMFilesReplication onto CATCHUP Lane.

* Adjusted Fast and MED lane usage in Supervised scheduler

* Added changelog entry

* Added new features entry

* A new leader will now keep old followers in case of failover

* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Fixed JSLINT

* Unified lane handling of replication handlers

* Sorry forgotten in last commit

* replaced strings with static strings

* more use of static strings

* optimized min repl description in the ui

* decr initial loop variable

* clean up of the createWithId test

* more use of static strings

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Added some comments on condition, renamed variable as suggested in review

* Added check for min replicationFactor to be non-zero

* Added assertion

* Added function to modify min and max replication factor in one go

* added missing semicolon

* rm log devel

* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place

* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers

* check replFactor against nr dbservers

* Add lie reporting in CURRENT

* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out

* move replication checks from logical collection to rest collection handler

* added more replication tests

* Include assert only if we are not in gtest

* jslint

* set min repl factor to zero if satellite collection

* check replication attributes in v8 collection

* Initial commit, old plan, does not yet work

* fixed ires tests

* Included FailoverCandidates key. Not fully implemented

* fixed wrong assert

* unified in sync follower reporting

* fixed compiler errors

* Cleanup locking, and fixed potential deadlocks

* Comments about locking order in FollowerInfo.

* properly check uint

* Keep old leader as potential failover candidate

* Transaction methods now use followerInfo to check if the leader can write, this might have the sideeffect that 'failoverCandidates' are updated

* Let agency check failoverCandidates if possible

* Initialize member variables

* Use unified follower reporting in DBServerAgencySync

* Removed obsolete variable, collecting it somewhere else

* repl factor attr check

* Reimplemented previous followers, second attempt now. PhaseOne and PhaseTwo can now synchronize on current.

* Fixed assertion, forgot an off-by-one

* adjusted test to be more preciese now

* Fixed failove candidates list

* Disable write on dropping too many followers

* Allow to run updateFailoerCandidates multiple times with same leader.

* Final fixes, resilience tests now green, crossing fingers for jenkins

* Fixed race on atomics comparison

* Fixed invalid number type

* added nullptr handling

* added nullptr handling

* Removed invalid assert

* Make takeover of leadership an atomic operation

* Update tests/js/common/shell/shell-cluster-collection.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review fixes

* Fixed creation code to use takeoverLeadership

* Update arangod/Cluster/FollowerInfo.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Applied review fixes

* There is no timeout

* Moved AQL + Pregel to INTERNAL_AQL lane, which is medium priority, to avoid deadlocks with Sync replication

* More review fixes

* Use difference if you want to compare two vectors...

* Use std::string ...

* Now check if we are in recovery mode

* Added documentation for minReplicationFactor

* Added readme update as well in documenation

* Removed merge conflict leftovers 0o, i should not trust the IDE

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/Books/Manual/Architecture/Replication/README.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update CHANGELOG

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/Books/Manual/DataModeling/Collections/DatabaseMethods.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/Books/Manual/ReleaseNotes/NewFeatures35.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/DocuBlocks/Rest/Collections/1_structs.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/graphManagementView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/graphManagementView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/DocuBlocks/Rest/Graph/1_structs.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Apply suggestions from code review

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Adepted review requests, thanks for finding!

* Removed unnecessary const

* Apply suggestions from code review

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Moved initilization of variable more downwards

* Apply lock before notify_all()

* Remove documentation except DocuBlocks, covered by PR in docs repo

* Remove accidental indent
2019-07-22 17:48:34 +03:00
Dronplane 1f8099e295 Bug fix/fix cluster tests for analyzers (#9370)
* Fixed analyzer definition in test (definition should be object). Fixed proper reporting of invalid parameter error.

* Restored ability to parse string-encoded json object in analyzers rest hadler. Test was reverted to pass string again.

* Fixed test run

* Get rid of shared ptr to simplify code.
2019-07-01 16:52:06 +02:00
Jan 1653e9698a
remove unused functionality (#9360) 2019-06-28 18:28:11 +02:00
Andrey Abramov 40cce2cee0
bug-fix/internal-issue-#565.1 (#9257)
* validate analyzer properties, do not return `null` for identity analyzer properties

* fix some tests

* Fixed tests for new analyzer parameters validation. Added explicit test for analyzer parameter validation

* update iresearch, fix analyzer tests

* fix more tests

* fix tests

* store analyzer properties as vpack

* more compilation fixes

* Updated iresearch

* Tests compilation fixed

* Test run fixes

* Test run fixes

* Fix test run

* Fixed all IresearchTests

* Fixed V8Analyzers tests

* Added ngram and delimeter analyzers vpack config

* Added ngram and stem analyzers vpack parsers. Added tests

* Fixed internal issue #593 Fixed build issue

* Fixed tests

* Fixed Gtest tests

* Changed test run to fix Mac failure

* Mac tests debugging

* Fixed jslint errors

* test tracing added

* Taken account for VPackSlice operator== false negatives
2019-06-28 10:49:01 +02:00
Simon cf7cf0131b Try to fix corruption error (#9258) 2019-06-25 10:18:26 +02:00
Jan Christoph Uhde ddb2e14e32 disable security checkes for statistics (#9282) 2019-06-24 15:07:39 +02:00
Jan Christoph Uhde 307880ffad Allow statistics for internal contexts. (#9262) 2019-06-13 19:27:29 +02:00
Jan e0c94e64d7
fix path length checks when copying js files (#9191) 2019-06-05 10:39:52 +02:00
Andrey Abramov 0fb8b348db
bug-fix/internal-issue-#574 (#8991)
* restrict number of buckets to compare

* do not use more sort buckets than requested

* cleanup
2019-06-04 18:55:25 +03:00
Andrey Abramov b2a2bd100e
Bug fix/internal issue #590 (#9176)
* disallow creation of analyzer by a full name

* fix tests
2019-06-04 00:28:20 +03:00
Tobias Gödderz 79cd45f89c Wait for replication before inserting documents (#9151)
* Wait for replication before inserting documents

Also, increased some timeouts and fixed a log message

* Fixed some log levels and a log message

* Removed repair-distribute-shards-like-spec from greylisted tests
2019-05-31 16:09:20 +03:00
Simon 93b2e64f37 Port pregel fixes (#9022) 2019-05-17 16:32:58 +02:00
Wilfried Goesgens 1907a7211b Bug fix/cleanup system includes (#8962) 2019-05-15 15:12:59 +02:00
Vasiliy 12bd4c785b issue 555.1.1: issue 555.1.1: try loading previous data store snapshots if checkpoint file unavailable, ehnace error message, address some warnings (#8975) 2019-05-13 18:33:44 +03:00
Vasiliy bbd5973337 issue 562.1: represent analyzer properties as object when possible (#8929) 2019-05-11 18:39:25 +03:00
Vasiliy 9f1340eeeb issue 563.1: include static analyzers in analyzer listing (#8925)
* issue 563.1: include static analyzers in analyzer listing

* address macos test failures
2019-05-11 14:52:48 +03:00
Vasiliy 42b830c7a9 issue 561.1: add pretty-printing to analyzers in JavaScript (#8964) 2019-05-10 21:50:15 +03:00
Wilfried Goesgens c43ec49eae make id a string so all of it can be passed on to V8. (#8933) 2019-05-08 18:52:29 +02:00
Jan Christoph Uhde f3a2eaf6d9 Bug fix/permission tests (#8890) 2019-05-07 15:07:59 +02:00
Jan Christoph Uhde 677a79026c Foxx Security (#8845) 2019-04-25 09:56:29 +02:00
Vasiliy 1a22d1360c issue 526.9.1: implement swagger interface, add documentation (#8730)
* issue 526.9.1: implement swagger interface, add documentation

* address review comments

* add ngram

* Formatting

* Move REST description to new Analyzers top chapter in HTTP book

* Missed a DocuBlock

* Add Analyzers chapter to Manual SUMMARY.md

* Move REST API description back to Manual

Headlines were broken

* Add n-gram example
2019-04-16 18:54:30 +03:00
Jan c4d8091f45
remove unused attributeExcludeHandler (#8760) 2019-04-15 18:55:59 +02:00
Dan Larkin-York 777ba1a364 Improved audit logging (#8740) 2019-04-15 14:51:32 +02:00
Simon 937d743ba6 Bug fix/pregel stuff (#8733) 2019-04-11 15:58:28 +02:00
Jan c6d3f8e052
Bug fix/pass on error messages (#8690) 2019-04-10 12:34:25 +02:00
Jan 4f7923a971
remove replication clients handling out of vocbase code (#8676) 2019-04-08 19:15:28 +02:00
Jan 5897baa984
added db.<collection>.getResponsibleShard() (#8683) 2019-04-08 16:10:45 +02:00
Vasiliy 0410588723 issue 526.7.1: remove deprecated functions, update tests to set up required analyzers (#8688)
* issue 526.7.1: remove deprecated functions, update tests to set up required analyzers

* address merge issues

* address more merge issues
2019-04-05 18:10:42 +03:00
Vasiliy f4919dc173 issue 526.6: implement REST and V8 handlers for the iresearch analyzer feature (#8626)
* issue 526.6: implement REST and V8 handlers for the iresearch analyzer feature

* address typo

* remove excess comments

* temporarily comment out tests failing on MacOS

* temporarily comment out more MacOS-only test failures
2019-04-02 19:23:28 +03:00
Jan 7bf23bf238
add shutdown protection for PregelFeature (#8628) 2019-03-29 19:36:10 +01:00
Jan 531b55f225
introduce option `--log.time-format` (#8604) 2019-03-29 17:15:33 +01:00
Jan 9ab9cc7857
disambiguate internal exceptions (#8623) 2019-03-29 15:59:37 +01:00
Jan 207c978d1e
Bug fix/move a few actions from js to cpp (#8549) 2019-03-26 10:02:21 +01:00
Jan Christoph Uhde c3f7961b88 apply unique log ids (#8561) 2019-03-25 20:26:51 +01:00
Simon bcc26926de Support Transactions in RestCursorHandler (#8539) 2019-03-25 17:04:11 +01:00
Jan e078f35285
fixed typos, removed unneeded includes (#8547) 2019-03-25 12:09:37 +01:00
Simon 3ada15fc35 The Legendary El Cheapo (#8485) 2019-03-22 11:38:33 +01:00
Jan 6d107beeda
don't run compact() on a collection after a truncate() was done in th… (#8468)
* don't run compact() on a collection after a truncate() was done in the same transaction

running compact() in the same transaction will only increase the data size on disk due to RocksDB not being able to remove
any documents physically due to the snapshot we take at transaction start.
Decoupling the truncate transaction from the compact operation allows finishing the truncate transaction first, so we can
get rid of the snapshot. Running compact afterwards is then free to physically remove all the data.
As a nice side effect this change will also speed up the truncation of larger collections, because the compact will run
faster.

This change also exposes db.<collection>.compact() in the arangosh, in order to manually run a compaction on the data
range of a collection should it be needed for maintenance.

* fix documentation anchors
2019-03-20 16:27:54 +01:00