1
0
Fork 0
Commit Graph

616 Commits

Author SHA1 Message Date
Tobias Gödderz dc2e27db6c [3.4] Feature/rebootid notice changes, backport of #9523 (#9685)
* Feature/rebootid notice changes, backport of #9523

* Move to 3.4 and C++11-compatible code (except for test code)

* Backported tests/Mocks/Servers.{h,cpp}

* Rebuilt errorfiles

* Ported CallbackGuardTest from gtest to catch

* Ported RebootTrackerTest from gtest to catch

* Make sure the state method is called, so the overridden method is used during tests

* Fixed test to work with the old scheduler

* release version 3.4.8

* [3.4] fix agency lockup when removing 404-ed callbacks (#9839)

* Update arangod/Cluster/ServerState.cpp

Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com>

* Instantiate the scheduler during ::prepare()

* Fix test crash introduced during backport

* Fix a compile error on Windows (thanks, V8)
2019-09-19 15:03:39 +03:00
Jan 34481860a9 cover more cases of "unique constraint violated" issues during replication (#9828)
* release version 3.4.8

* cover more cases of "unique constraint violated" issues during
replication

* add more testing
2019-09-11 12:48:08 +03:00
Tobias Gödderz 4ada35f20c [3.4] Bug fix 3.4/add shard id to replication client identifier (#9466)
* Bug fix/add shard id to replication client identifier (#9366)

* Fixed compile (but not linker) errors

* Backported ReplicationClientProgressTracker

* Fixed compile errors, fixed hash function

* No longer use SyncerId for real asynchronous replication

* Updated docu

* Try to fix compile error on windows

* Fixed a bug

* Removed old code

* Fixed CHANGELOG entry
2019-09-11 12:45:57 +03:00
Lars Maier d8bc3685d3 [3.4] Fast Leader Change (#9642)
* Intermediate state.

* Finished.
2019-08-09 14:02:04 +02:00
Jan 65e7557f42 Bug fix 3.4/improve arangorestore (#9156)
* always import system collections first, so if importing some other collection fails later, the system collections are already in place

* properly disable enterprise features when restoring an enterprise dump into a community cluster

* disable enterprise features when restoring an enterprise dump into the community version

* fix handling of "duplicate name" issues when restoring system
collections
2019-06-09 20:21:07 +02:00
Michael Hackstein 4d4c23c302
Bug fix 3.4/collection babies (#9033)
* Prepare API to create multiple collections in a single request to ClusterMethods to improve speedup

* Added counter on how many collections are successfully created

* Allow multi collection creation one level higher

* CollectionMethods now allow batch createion of Collections

* Improved array size assertions

* Now a graph is createad within a single roundtrip in the agency.

* Added new header files

* Insert collections in the AGENCY with TTL and a isBuilding flag, collections with this flag should not be visisible in the coordinator

* Added forgotten C++ file

* Fixed a rare race condition, and the failing IResearch Tests

* readded callback on DONE, otherwise lists are out of sync

* Fixed assertions to let mocked tests pass...

* Fixed community cluster
2019-05-21 08:41:12 +02:00
Jan d29d65face
Bug fix 3.4/cleanup 31032019 (#8633) 2019-04-01 17:02:00 +02:00
Max Neunhöffer 54c533c5f6
Some more agency fixes. (#8377)
* Ignore satellite collections in shrinkCluster in agency.
* Abort RemoveFollower job if not enough in-sync followers or leader failure.
* Break quick wait loop in supervision if leadership is lost.
* In case of resigned leader, set isReady=false in clusterInventory.
* Fix catch tests.
2019-03-12 14:09:56 +01:00
Jan 30c61a5c82
improve error messages when restoring from invalid JSON data (#8211) 2019-02-20 18:34:43 +01:00
Matthew Von-Maszewski 474f0cde31 Bug fix 3.4/scheduler empty reformat (#7872)
* added check for empty scheduler

* removed log, old is 1 not 0

* require running in this thread

* test

* added isDirect to callback

* signature fixed

* added drain

* added allowDirectHandling

* disabled for testing

* Add ExecContextScope object to direct call.

* try alternate initialization of ExecContextScope

* remove ExecContextScope, no help.  try _fifoSize as part of direct decision.

* strand management to minimize reuse of same strand per listen socket

* blind attempt to address Jenkins shutdown lock up.  may remove quickly.

* add filename and line to existing error log message

* Adjust queueOperation() to stop accepting items once isStopping() becomes true.

* revert previous check-in to MMFilesCollectorThread.cpp

* big reformat

* fixed merge conflicts

* Add CHANGELOG entry.
2019-01-08 20:39:42 +01:00
Jan a4a7867451
Bug fix 3.4/arangorestore add cleanup duplicate attributes (#7876) 2019-01-04 15:26:11 +01:00
Jan a14f6dd573
prevent duplicate attributes being generated by AQL queries (#7836) 2019-01-02 12:37:47 +01:00
Frank Celler 9477af198b big reformat 2018-12-26 00:57:05 +01:00
Michael Hackstein bc65e79b02
Bug fix 3.4/fix tombstones (#7758)
* fixes a rare situation where replication could require identical lock twice

* Updated changelog
2018-12-12 22:01:27 +01:00
Simon 933ca8a775 Bug fix/restore index refactor (#7470) (#7491)
(cherry picked from commit d0efd95a37)
2018-11-29 14:08:29 +01:00
Jan ffc823e1c8
Bug fix 3.4/backport optimizations (#7434) 2018-11-26 19:16:05 +01:00
Michael Hackstein 8098bb4eed
Bug fix 3.4/syncing of followers (#7377)
* Added some DEBUG output for replication rest handler

* Some more debug logging.

* Increased the priority of the ReplicationHandler. This way we will not get stuck with locks that cannot be canceled. Also cancel the lock on the correct database.

* Added extensive log output for replication thins

* Added tombstones to RestReplicationHandler. In a very unlikely case the cancel of a lock can be executed BEFORE the code that actually registers the lock, in this case we will now write a tombstone and do not lock.

* Revert "Added extensive log output for replication thins"

This reverts commit 6d4e37ea1e59e3b3457336019cc7dbc4c979504d.

* Added extensive log output for replication things, now in ERR level instead of MAINTAINER only

* Now actually use hours for synchronization

* React to errors under soft lock if they show up.

* Added a retry loop to increase the read-lock timer.

* Added more timeing output in RocksDB collection internals to figure out why the followers are dropped

* Tweaked RocksDB options

* Revert "Tweaked RocksDB options"

This reverts commit 2bf9c43280beda4792c47d079387fe5154cdd896.

* Removed debug output

* Applied all requested changes by goedderz

* Deleted unused variable
2018-11-23 16:08:27 +01:00
Wilfried Goesgens c50d346453 add alternative to ClusterInfo::getCollection() that doesn't throw (#7413)
* add alternative to ClusterInfo::getCollection() that doesn't throw (#7339)

* handle more potential nullptrs, fix try/catch scope
2018-11-23 11:15:25 +01:00
Vasiliy d644561f1f issue 496.4.1: backport 3.4: move StorageEngine-specific flag out of the genric API and closer to the storage engine (#7213)
* issue 496.4.1: backport 3.4: move StorageEngine-specific flag out of the genric API and closer to the storage engine

* address merge issue
2018-11-04 16:52:54 +03:00
Vasiliy 850919178f issue 496.3: backport 3.4: move more coordinator-related logic out of TRI_vocbase_t, rename some arangosearch view configuration parameters, remove some consolidation policies, update iresearch to revision 6fd9760d81b136f769e277ea5b8f53996ed7a1ca (#7167)
* issue 496.3: backport 3.4: move more coordinator-related logic out of TRI_vocbase_t, rename some arangosearch view configuration parameters, remove some consolidation policies, update iresearch to revision 6fd9760d81b136f769e277ea5b8f53996ed7a1ca

* address merge issue

* backport: remove code causing nullptr access

* invalidate payload for each field in FieldIterator before setting a value

* address compilation issues
2018-11-01 23:12:39 +03:00
Max Neunhöffer 42fd0825ab
Fix timeouts for write operations from coordinator to leader. (#7081)
* Improve logging on coordinator when doing `arangorestore`.

* Return more error information in `mergeResults`.

* Longer timeout for communication coordinator -> leader for writes.

This is taking into account possible write stops from followers needed
to get in sync.

* Fix compilation.

* Get rid of numbers in exception log messages.

* Fix compilation.

* Fix indentation.
2018-10-31 14:39:48 +01:00
Vasiliy 2458705607 issue 496.2: backport 3.4: allow customization of segment size limits on arangosearch view creation, minor code cleanup (#7141)
* issue 496.2: backport 3.4: allow customization of segmentsize lmts on arangosearch view creation, minor code clanup

* backport: fix incorrect condition
2018-10-30 23:41:15 +03:00
Simon c073b9dbbe Make ensureIndexOnCoordinator more robust (#7110) (#7130) 2018-10-30 11:25:06 +01:00
Vasiliy e6a6025818 backport: switch scope of responsibility between a TRI_vocbase_t and a LogicalView in respect to view creation/deletion (#7106)
* backport: switch scope of responsibility between a TRI_vocbase_t and a LogicalView in respect to view creation/deletion

* backport: ensure arangosearch links get exported in the dump

* backport: ensure view is created during restore on the coordinator

* Updates for ArangoSearch DDL tests, IResearchView unregistration and known issues

* Add fix for internal issue 483
2018-10-30 12:50:29 +03:00
Michael Hackstein e05880895a
Bug fix 3.4/shorter foot in door (#7084)
* Implement `syncCollectionCatchup` in DatabaseTailingSyncer.

First stab, might not even compile.

* Fixed a typo.

* Fix a typo and a compilation problem.

* Further compilation fix.

* Implement two stage catchup.

* Two small corrections.

* Unified error messages in Synchronize shard job.

* Improved a code comment.

* Fixed autocasting bool->double and double->bool issue. That is truely one of the best features ever invented... </irony>

* Renamed doHardLock => toSoftLockOnly and inverted default value

* Merged soft/hard foot logic with Transaction splits

* Use scopeguards to cancel readlocks
2018-10-26 16:16:52 +02:00
jsteemann 435c014a15 Merge branch '3.4' of https://github.com/arangodb/arangodb into 3.4 2018-10-25 00:00:00 +02:00
jsteemann 5e6a267296 remove unneeded string value 2018-10-24 23:59:51 +02:00
Simon e87b42a0c3 Silence tsan warnings (#7051) 2018-10-24 23:58:47 +02:00
Vasiliy 52e2c97693 backport missed changes (#7016) 2018-10-24 15:43:45 +03:00
Michael Hackstein 9599eb9b9f
Bug fix 3.4/sync replication allow soft and hard lock (#6864)
* First attempt to not block the thread that requires the EXCLUSIVE sync-up lock

* Fixed insertion of query into registry in rest replication handler.

* Removed unnecessary / false asserts as suggested in review. Fixed code comments.

* Replaced auto with a correct type as suggested in review

* Added a helper function to validate if a query is in use in the registry

* Fixed logic bug in usage of query registry

* Fixed compile issue

* Implemented optional 'doHardLock' parameter in the replication acquire read-lock call. A hard-lock guarntees to stop all writes, a soft-lock may not.

* Fixed compile issue

* Automaticly transfrom int -> bool in initializerlist sucks...

* Inverted boolen logic bug hidden due to int->bool beeing logically inverted.

* Today it seems that bools are too complicated for my brain.

* Removed failure point, didn't write a test for it, and it is hard to write it in the current test environment. Need to find a better solution in future

* Applied chenges required by @goedderz in review

* Renamed doHardLock => toSoftLockOnly and inverted default value
2018-10-19 14:11:34 +02:00
Michael Hackstein 875185e6c1
Bug fix 3.4/sync repl release thread (#6784)
* First attempt to not block the thread that requires the EXCLUSIVE sync-up lock

* Fixed insertion of query into registry in rest replication handler.

* Removed unnecessary / false asserts as suggested in review. Fixed code comments.

* Replaced auto with a correct type as suggested in review

* Added a helper function to validate if a query is in use in the registry

* Fixed logic bug in usage of query registry

* Fixed compile issue

* Automaticly transfrom int -> bool in initializerlist sucks...

* Inverted boolen logic bug hidden due to int->bool beeing logically inverted.

* Today it seems that bools are too complicated for my brain.

* Removed failure point, didn't write a test for it, and it is hard to write it in the current test environment. Need to find a better solution in future

* Applied chenges required by @goedderz in review
2018-10-19 13:48:22 +02:00
Dan Larkin-York 3d15b96f66 [3.4] Additional and cleaned-up logging (#6648) 2018-10-01 19:03:05 +02:00
Jan a8cd72f502
Bug fix 3.4/optimizations 210918 (#6574) 2018-09-25 09:50:25 +02:00
Simon 3c965ee48a Resilience test failure points (#6545) 2018-09-20 01:04:38 +02:00
Simon 292b6312ae Properly check syncer erros, catch more exceptions (#6522) 2018-09-17 16:38:44 +02:00
Lars Maier 78f182ebe8 Completely export views dump restore [3.4] (#6484) 2018-09-13 12:37:37 +02:00
Jan 421fd68bed
use less VPackBuilders when restoring data into a collection (#6362) 2018-09-04 15:44:43 +02:00
Jan 5022ccc24d
Bug fix/fixes 2508 (#6254) 2018-08-27 21:36:39 +02:00
Jan 791eaba873 Bug fix/planning 2865 (#6252)
* potential bugfix for planning/#2865

* speed up dump tests setup

* enable authentication for backup tests

* make arangodump provide a "serverId" to the server

this allows the server to track arangodump as an active dump client
so any data required for the dumping may be retained while the
dump is ongoing

* don't log binary stuff into the logfile
2018-08-27 10:21:34 +02:00
Simon 568a09f177 Disable JS on DBServer, fix race in UserManager (#6244) 2018-08-24 22:20:49 +02:00
Jan 5f0403ed1c
Bug fix/add aql collection count cache (#6227) 2018-08-23 16:05:51 +02:00
Simon 229c09d434 Allow dirty-reads from passive (#6136) 2018-08-20 16:26:14 +02:00
Jan a5ef080a8a
attempt to make replication_sync more reliable in tests for MMFiles (#6184) 2018-08-17 14:20:40 +02:00
Simon 2d29232c59 fix restoring iresearch view in cluster (#6159) 2018-08-17 09:28:20 +02:00
Simon 392118bd62 Use RangeDelete where possible (#6121) 2018-08-15 18:52:09 +02:00
Simon 42cabb858a Fix dumping of views in the cluster (#6024) 2018-08-02 17:25:49 +02:00
Vasiliy 62ca1235ac issue 430.3: remove redundant constructor from SingleCollectionTransaction (#5996) 2018-07-26 16:54:53 +03:00
Jan 21064144c8
Bug fix/replication improvements (#5962) 2018-07-25 09:04:50 +02:00
Jan a5bb50b0bf
remove methods from VelocyPackHelper that are also in VPackSlice (#5946) 2018-07-25 09:01:29 +02:00
Vasiliy 3c384bf3e8 issue 410.1: ensure both JavaScript and Rest API for views matches API for collections (#5903)
* issue 410.1: ensure both JavaScript and Rest API for views matches API for collections
* backport: address some FIXMEs
* backport: address test failures
* backport: address more test failures
* backport: fix some Ruby tests
* temporarily comment out broken Ruby tests
* backport: fix more broken Ruby tests
* temporarily comment out failing Ruby tests
* backport: match result structure for View instances with Collection instances
* backport: return different result structure for view listing and view definition
* backport: fix even more broken Ruby tests
* backport: do not depend on unordered_set stable order
* update documentation
2018-07-19 15:07:17 +02:00