1
0
Fork 0
Commit Graph

78 Commits

Author SHA1 Message Date
Andrey Abramov 8741d338d0 allow in-place analyzer creation via link definition (#10466) (#10481)
* allow in-place analyzer creation via link definition (#10466)

* allow in-place analyzer creation via link definition

* add special handling for _analyzers collection

* modify initial syncer

* address review commments

* fix accidentally broken test

* address compilation errors
2019-11-19 21:49:28 +01:00
Andrey Abramov f4e6538edd Bug fix/internal issue #647 (#10292) (#10334)
* Bug fix/internal issue #647 (#10292)

* extend replication tests

* ensure proper replication order

* fix tests

* address review comments

* address test failures

* extend dump tests

* fix analyzers tests

* more fixes

* extend tests

* enhance tests

* adjust tests

* use enum instead of flags (part 1)

* cleanup

* use enum instead of flags (part 2)

* get rid of flags for views

* get rid of flags for collections

* completely get rid of collection flags

* fix replication test

* refactor index flags

* fix tests and move AnalyzerPool out of class scope

* fix tests

* adjust log levels

* add tests

* remove debug logging

* remove noexcept from `equalAanalyzer`

* extend cluster tests

* fix cluster tests

* add tests for views and smart graphs

* address jslint errors

# Conflicts:
#	arangod/Cluster/ClusterMethods.cpp
#	arangod/Cluster/v8-cluster.cpp
#	arangod/IResearch/IResearchAnalyzerFeature.cpp
#	arangod/IResearch/IResearchAnalyzerFeature.h
#	arangod/IResearch/IResearchLinkMeta.cpp
#	arangod/RestHandler/RestAnalyzerHandler.cpp
#	arangod/StorageEngine/PhysicalCollection.cpp
#	arangod/VocBase/Methods/Indexes.cpp
#	tests/IResearch/IResearchAnalyzerFeature-test.cpp
#	tests/IResearch/IResearchFeature-test.cpp
#	tests/IResearch/IResearchLinkHelper-test.cpp
#	tests/IResearch/IResearchLinkMeta-test.cpp
#	tests/IResearch/IResearchQueryOptimization-test.cpp
#	tests/IResearch/IResearchViewDBServer-test.cpp
#	tests/IResearch/IResearchViewSort-test.cpp
#	tests/V8Server/v8-analyzers-test.cpp
#	tests/VocBase/LogicalDataSource-test.cpp

* Update CHANGELOG

* Update CHANGELOG
2019-10-30 17:01:18 +03:00
Max Neunhöffer 328f46e3d6 This merges hotbackup and atomic-db-creation into 3.5. (#9968)
* Squashed commit of feature-3.5/hotbackup_devel.

This puts hotbackup into 3.5.

* Port atomic-database-creation-2 to 3.5.

* Remove some wrongly ported code.

* Fix compilation.

* Fix a manual merge error.

* Remove a feature from the mocks which does not exist in 3.5.

* Add some code which was forgotten in manual merge.

* Fix a problem introduced in a manual merge.

* reuse function

* Address some whitespace issues that came up in review

* aardvark should not create the frontend collection

* create _frontend collection from c++

* recheckAndUpdate Callback in CollectionWatcher

* Wrong author ;)

* rm outdated todo

* Update lib/Basics/VelocyPackHelper.h

Co-Authored-By: Michael Hackstein <michael@arangodb.com>

* use logger unique id, use startup logger

* not needed

* optimized vector shardid method

* do not create _modules collection lazy anymre

* Formatting.

* Assert instead of if/TRI_ASSERT(false)

* Don't use exceptions as control structure

* Re-add READ_LOCKER that got lost in translation

* Fix audit log in case database creation fails early.

* legacy sharding

* Add CHANGELOG entry.

* Retry database cancellation indefinitely

* Do not use exceptions in UpgradeTask

* DropCollection is a FAST_LANE action and should not need much time or else retry.

* Remove superflous addition of LdapFeature

Proudly brought to you by ASAN tests

* Fixed check for distributShardsLike sharding on _system database

* Fixed compile issue on tests

* Removed assertion that seems to be not correct yet on devel.

* Sort out google cloud storage as remote. (#9918)

* Add successful method to ClusterCommResult.
* Improve error forwarding for cluster internal communication.

* Feature/hotbackup list retries (#9924)

* retry hot backup listing for 2 minutes in cluster before giving up

* Enable api by default.

* fix broken list of non existing id (#9957)

* Fix compilation after manual merge.

* Fix another compilation problem.

* Yet more fixes for compilation.

* More compilation fixes.
2019-09-11 13:13:54 +03:00
Jan 0fefe62d35 restore _users collection at the very end only (#9899)
* restore _users collection at the very end only,

because updating documents in _users may change the credentials
for the arangorestore user itself, which may lead to all follow-up
requests by arangorestore being rejected with "authorized"/"forbidden"

* updated CHANGELOG
2019-09-05 19:29:47 +03:00
Jan 11f5f33659
make sure all error code names are prefixed with ERROR_ @fceller @kvs85 (#9384) 2019-07-02 18:07:33 +02:00
Jan 98e9197a3f properly downgrade enterprise features when restoring them into a com… (#9161) 2019-06-07 15:45:14 +02:00
Jan 1408654d2c
fix a few issues in BackupNoAuthSysTest (#8868) 2019-04-29 19:13:39 +02:00
Matthew Von-Maszewski eef492721a compress-data option for arangodump. ported from 3.4 (#8861) 2019-04-26 18:50:58 +02:00
Vasiliy 40ebdd0b58 issue 545.1.1: output short analyzer prefixes for user definitions (#8814) 2019-04-21 15:10:28 +02:00
jsteemann ca74d7882b improve error message 2019-03-29 19:34:44 +01:00
Jan Christoph Uhde c3f7961b88 apply unique log ids (#8561) 2019-03-25 20:26:51 +01:00
Jan 5d527168d0
arangodump & arangorestore all databases (#8394) 2019-03-18 17:05:08 +01:00
Jan a841d472be
emit a warning when connecting with the client tools to a DBSERVER (#8397) 2019-03-13 13:08:41 +01:00
Jan 5502ed2a3a
vadim (#7983) 2019-01-22 14:06:17 +01:00
jsteemann ac1c3b892a fix MSVC warning 2019-01-09 10:21:49 +01:00
Lars Maier 12eebb15fe Feature/new server infra (#7733)
* Decoupled IO handling from Scheduler.

* Fixed SSL start up bug.

* Replaced Scheduler with new worker farm implementation.

* Added minimal statistics and info string for Scheduler.

* Added support for timed submissions.

* Updated delayed submission api. Updated code that used timers.

* Extracted new Scheduler into a virtual parent class. The implementation can now depend on the usecase.

* Signal handler now working.

* Changed threads names, `_stop` is atomic, check for failure during thread start + exception handling like old scheduler did.

* Commented on source code and added TODOs.

* Played around with start-stop-conditions

* Play around with start stop condition.

* start stop cond

* Sart Stop Conditions

* Removed bad cv_status check.

* Bug fix: now compare the actual objects instead of pointer values. Setup t1 and t2 depending on the thread id.

* Moved most of the stuff now unrelated to the Scheduler to GeneralServer. Got rid of JobGuard.

* Instead of waiting for a thread to terminate, put it on a clean up list and check for its termination in each supervisor run.

* Allow detaching long running threads.

* Fixed test mock.

* Updated the WorkHandle logic. Removed post functions.

* Fixed crash when obtaining shared_ptr from this in destructor.

* Added lost mutex.

* Fixed memory leak.

* Fixed merge bug.

* Changed a lot of code to optimize the scheduler.

* Fixed bug of invalidated iterator. Dont remove task on shutdown at different places. Let scheduler threads run until queue is empty.

* Only by value calls to queue.

* Added options again.

* Clean up of code.

* UI Request Lane added.

* Bug fixes in Scheduler.

* Applied reformat.

* Use sigaction.
2019-01-08 10:12:02 +01:00
Jan bd5a291d41
simran (#7881) 2019-01-04 17:09:03 +01:00
Jan 28528973e3
added arangorestore option `--cleanup-duplicate-attributes` (#7877) 2019-01-04 15:25:45 +01:00
Jan b3a3e35f3d added arangorestore options `--number-of-shards` and `--replication-f… (#7869)
* added arangorestore options `--number-of-shards` and `--replication-factor`

* updated documentation

* added version numbers
2019-01-03 11:07:02 +01:00
sleto-it 20d5e1655e
Documents that arangorestore/arangodump threads option is from v3.4.0 on (#7701) 2018-12-29 13:29:10 +01:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Jan d22997b01c
improve error messages (#7697) 2018-12-10 10:37:57 +01:00
Michael Hackstein d280205476 Improved help message for a failed restore of foxx services (#7495) 2018-11-28 10:54:37 +01:00
Heiko 3322cc5ae2 Bug fix/update foxx services after restore (#6774)
* triggers foxx queue update which leads to the internal selfHeal function of a foxx app to make foxx apps publicy visible after a foxx app is restored

* this will also add the recognition of imported foxx services

* Testing JS refactoring

* Added test case for Foxx dump -> restore cases

* Fixed check, sth- i tries to contact to the foxx service. Test is red now.

* Fixed tests for dump / restore using Foxx. Also first draft to fix the fox side. Blocked by sth. fixed in the Main Repositorx already

* Fixed typo in dump/restore testsuite

* Removed old version of the fix

* Fixed JSLint
2018-11-23 17:06:09 +01:00
Simon c584527d79 Fix restore of views, add --view option (#7425) 2018-11-22 19:24:24 +01:00
Jan 96edc0f6ff
Bug fix/fixes 121018 (#6853) 2018-10-12 12:47:30 +02:00
Jan 44448321e1
Bug fix/optimizations 210918 (#6573) 2018-09-24 20:16:16 +02:00
Lars Maier e2930ccba6 Completely export views dump restore (#6466) 2018-09-12 17:21:22 +02:00
Jan 9e8c4644a8
more detailed progress reporting for arangorestore (#6329) 2018-09-03 15:39:39 +02:00
Vasiliy 6fd541d110 issue 427.5: use ApplicationServer reference instead of pointer (#6145)
* issue 427.5: use ApplicationServer reference instead of pointer

* address MSVC build failure
2018-08-15 12:16:02 +03:00
sleto-it b95767247c
Doc - Arangodump improvements (#5881) 2018-08-07 21:54:13 +02:00
Jan b278d6874a
allow master & slave to work in parallel for RocksDB WAL tailing (#6059) 2018-08-03 13:37:53 +02:00
Simon 42cabb858a Fix dumping of views in the cluster (#6024) 2018-08-02 17:25:49 +02:00
Wilfried Goesgens 69955cb2cb improve error messages (#6003) 2018-07-26 16:02:25 +02:00
Michael Hackstein 7a95c5e675
Feature/feature phases (#5272)
* Added feature phases

* BasicsPhase and DatabasePhase to the required files. Server now has Feature circles and does not boot. Will be sorted out later on.

* Added ClusterPhase to features

* Added V8Phase to the required features

* Added AQLPhase to the affected features

* Added ServerPhase to Features

* Added FoxxPhase to the relevant features

* Added AgencyPhase to the relevant features

* Moved registration from local variable SYS_SYSTEM_REPLICATION_FACTOR from cluster to V8 as their ordering is now vice versa

* Moved Bootstrap feature into FoxxPhase. It could be moved to ServerPhase easily if the FoxxQueue dependency would be removed

* Final movement of Startup Phases. Now solved all circles.

* Removed merge conflict

* Moved ReplicationTimeout into cluster phase and fixed cross-phase requirements

* Added greetings phase. This phase separates the Basics Phase and is the first to be run. Includes Logger and Hello/Goodbye

* Added the GreetingsPhase in the corresponding features. Now all BasicsPhase features start after greetings Phase. There is some issue in this branch which prevents the Agency from Gossipping right now. Will be fixed next

* Moved creation of the Agent into the prepare phase of the feature. THereby it is guaranteed that agents at least exists before the GeneralServer is activating endpoints

* Recovery needs to be started after the ServerID

* Moved log output of FeaturePhases to DEBUG instead of ERROR.

* Added feature phases for clients

* ClusterFeature now does not directly require AgencyFeature any more

* Added requirement of TravEngineRegistryFeature in AQL feature. Otherwise shutdown may be undefined

* The ApplicationServer can now handout the list of ordered features. Used for testing purposes

* Fixed IResearchVew Tests Setup to honor new feature ordering

* Fixed IResearchViewDBServer Tests Setup to honor new feature ordering

* Started fixing IResearchView Coordinator tests with startup ordering. Not finished yet

* Added startup phases to ViewCoordinator test

* Disabled expected logoutput in ClusterRepairsTest

* Fixed indention in test code

* LinkCoordinator now honors startup ordering

* Link meta now honors startup rdering

* Supress expected cluster logs in ViewTest

* Removed '#' accidentially added.
2018-07-16 14:09:36 +02:00
jsteemann 2d9e5288ab try to fix windows compile warnings and errors 2018-07-13 11:48:05 +02:00
Simon f47367a495 Allow dumping / restoring of views (#5774)
* allow dumping and restoring of views

* fix restore, adding some tests

* remove debug print

* fix global replication
2018-07-05 17:55:28 +02:00
Dan Larkin-York 820bfee329 Refactor syncer state and make notes for future parallelization (#5742) 2018-07-03 21:32:16 +02:00
Jan 3c39337d34
make arangorestore not report errors about the target database not found (#5286)
* make arangorestore not report errors about the target database not found

when it is invoked with option `--create-database true`. In this case,
arangorestore should not print an error but simply create the target
database. This patch implements this change.

Additionally, it aborts the startup of arangorestore in case there are
any errors. Previously, the startup phase went on even when no connection
could be established, which led to useless follow-up errors being printed.

* simplify error handling a bit
2018-05-11 08:31:44 +02:00
Dan Larkin-York b9d05351fb Parallelize arangorestore using the tools developed for arangodump (#5206) 2018-05-01 10:56:11 +02:00
Dan Larkin-York 38f162e344 Parallelize arangodump (#4356) 2018-03-26 15:55:14 +02:00
Jan 5acaf9b5a3
Bug fix/carnival (#4562) 2018-02-12 09:01:57 +01:00
Jan 956902a5a9
issue #3504: export database name with dump for easier restore (#4216) 2018-01-05 14:52:56 +01:00
Jan a3d169eb5f
fixed issue #4045: fix misleading error message in arangorestore (#4049) 2017-12-16 15:18:35 +01:00
Jan db9b7189f6
proper cast to TRI_read_t (#4036) 2017-12-14 21:31:05 +01:00
Frank Celler 20c8565b25
Feature/encrypted dump (#3777) 2017-11-22 17:25:25 +01:00
Frank Celler 179ae83cbc
Feature/encrypted dump (#3768) 2017-11-22 15:59:02 +01:00
Jan fce593b724 removed `--recycle-ids` option for arangorestore (#3713) 2017-11-16 14:25:54 +01:00
Jan f6a90c4879 some cleanup (#3583) 2017-11-07 10:27:21 +01:00
Simon Grätzer 7c31960cf2 Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00