1
0
Fork 0
Commit Graph

118 Commits

Author SHA1 Message Date
Jan 39465193c2 fix replication (#3099) 2017-08-22 15:51:50 +02:00
Jan Christoph Uhde df76bbc690 Bug fix/planning issue 514 replication api (#3021)
* add "cluster selectivity estimates" to CHANGELOG

* add some documentation to RocksDBRestReplicationHandler

* fix building with relative paths

* add some more doc

* add some tests for the replication api

* fix RocksDBRestReplicationHandler and add tests

* update documentation

* remove obsolete parameter

* fix error message

* Implementing logger-first-tick, logger-tick-ranges. Fixing dump `chunkSize` documentation
2017-08-22 10:46:06 +02:00
Jan 79d76c89fb Bug fix/use count for checksum (#3033)
* Count as checksum

* Make readLockId optional as well so upgrades are possible

* fix option name in startup script

* fix some replication issues with RocksDB engine
2017-08-15 11:24:54 +02:00
Michael Hackstein 7ab117f5f8 Bug fix/dump restore auth (#2917)
* Added a backup test suite. This suite is supposed to entirely drop an ArangoDB _system and restore it into a fresh one. This also includes system collections.

* Added more test cases for backup suite. Now tests several authorization/rights cenarios

* Fixed RestReplication Handlers to restore _user collections Properly.

* Updated Changelog

* Added special handling of _users in Restore for MMFiles as well.

* Added JWT secret for cluster execution of this test, also added JWT secret to shutdown call
2017-08-09 12:59:51 +02:00
Jan da6e63c696 fixed issue #2949 (#2951) 2017-08-04 13:36:37 +02:00
m0ppers 9a0bc716d0 Do not allow replication to create/drop collections (#2898)
In the cluster case the only one who is allowed to do this is the schmutz
2017-07-28 14:24:02 +02:00
m0ppers 6fefe5ab31 In high concurrency situations double checking checksums is a must have (#2890) 2017-07-28 00:43:21 +02:00
Simon Grätzer d8c57f2cc8 Feature/rocksdb hashsearch (#2713)
* Testing hash for documents

* fixing logspam error

* added base support for kHash in the edge index

* Current version of hash enabled edge index

* removed rocksdb key type byte, dynamic prefixes for hash-index

* Fixing issues with key classes

* Fixing merge conflict

* Fixing version with semi-working hash index

* fixing various issues

* Fixing hanger in EdgeIndexIterator nextExta, when _cache is disabled. Reenable cache

* Fixing geo index bounds, and fillIndexes error handling

* Adding back failure point

* Fixing RocksDBKey::vertexId

* Added version check on startup

* Fixing version check

* added one more check

* properly detect column families at startup

* Adding additional sanity checking

* Adding missing column family

* some fixes

* Turned off geoindex sync

* fixed error

* added asserts (untested)

* Adding more asserts
2017-07-05 10:18:52 +02:00
Jan ebaeb639b2 Bug fix/incremental replication syncs too much (#2631)
* fix too much synchronization
* fix incremental sync for RocksDB engine
2017-06-21 14:56:33 +02:00
jsteemann 2d96b9ab4c fix issues found by coverity scan tool 2017-06-07 13:22:18 +02:00
jsteemann a68c95d2b7 try to honor server shutdown while locks are held 2017-06-01 02:24:00 +02:00
jsteemann 0366fba276 try to fix thread joining for replication applier 2017-05-18 21:05:57 +02:00
Max Neunhoeffer f614439416 Squashed commit of the following:
commit 19a7210119cc284af64251b202f690ab62bf615c
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 23:20:10 2017 +0200

    Allow access to /_api/user/ without database check in VST as in HTTP.

    This is a try to fix #2465

commit f973c1335652540174bf6b78df42290bded357da
Merge: 30bdc4c 365de45
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 21:32:15 2017 +0200

    Merge remote-tracking branch 'origin/devel' into vst-cleanup

commit 30bdc4ca38e028d58cebea8268683d9f94e87ad7
Merge: 14acaea 086f6a2
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 14:52:23 2017 +0200

    Merge remote-tracking branch 'origin/devel' into vst-cleanup

commit 14acaea8cb5fff2c1c8c27eee32294bfd612f629
Merge: 5bd0e79 fb088d3
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:40:29 2017 +0200

    Merge remote-tracking branch 'origin/devel' into vst-cleanup

commit 5bd0e79ab053447f94f2fd959a33b1e15302b6a8
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:36:23 2017 +0200

    Fix bug in authentication in VST

commit 9a999ce5e629aee5816a4f61b194b086d11fd169
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:24:35 2017 +0200

    Open up endpoints /_open/* in VST without being authenticated

commit 32f62db42af6d5bfc24214ebb4ca7c1fb473b55e
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:11:31 2017 +0200

    Simplify logic by not using boost::optional.

commit 925ce2f7b890c6f14205fd3e98553939d89bec61
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:11:20 2017 +0200

    Add JWT case for authentication for VST

commit f1d7f67a9fc20c91ca000e9a4dad91e0f50f1652
Merge: 040ace3 7a9ccc9
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 08:58:51 2017 +0200

    Merge remote-tracking branch 'origin/devel' into vst-cleanup

commit 040ace3e9dce47ddea5f51d29f0153a70d257c9e
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 08:23:16 2017 +0200

    Write out chunk buffer using little endian uints

commit 66ad4c0e8d3bc94091664505986b00e7cff39f2f
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 08:12:38 2017 +0200

    Move maxChunkSize query to constructor, remove dead code

commit f7b4c26cc09c6d512362340c6bf6061bbaae61a9
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 17:57:14 2017 +0200

    Do not take protocolVersion from request for VST.

commit c76d6685f9507701715abf882f0ac0e192ead59f
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:20:58 2017 +0200

    Remove dead code for compression which did never work well.

commit b823765b7c041dc5d143f7ab2e13d92f5f42d3eb
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:09:53 2017 +0200

    Fix renaming to make compiler happy.

commit b327830012d146f4f4e2d0f0467dab351fbd3bd0
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:07:36 2017 +0200

    Rename _defaultPortVpp to _defaultPortVst.

commit 5eb98a9e64d702476e8ec098b7506b1a63145d64
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:06:28 2017 +0200

    Rename header guards VPP -> VST.

commit a3e96be26d49d6151898d8d757ede6225ac6abd2
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:05:55 2017 +0200

    Rename VppRequest and VppResponse files to Vst.

commit 1d5f6f196490c3cf9312be5927b967b4abc03b67
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:04:19 2017 +0200

    Change names of header guards VPP -> VST.

commit f004b25fbef75050b34bcd4eae4eb17b8b467230
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:03:19 2017 +0200

    Rename files VppNetwork.h and VstMessage.h

commit c04d3aec19e8f0a8a3b90cd48e7736bff037359d
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 12:59:15 2017 +0200

    Implement correct chunk header for sending out VelocyStream 1.1 messages.

commit 73aeedfbbef24d5929c44bd26c16343480514270
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 01:11:23 2017 +0200

    A lot more renaming VPP -> VST.

commit 2acedc77f16a82251bc4f9e39526184e2a6f0dec
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:49:20 2017 +0200

    Rename header guard.

commit 2ade43946aa5ea0048dab81eed5ffd931ce35ea8
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:48:01 2017 +0200

    Rename files VppCommTask.* to VstCommTask.*

commit 0ffcda02a51cbad35e5ec06117d397aac37323cd
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:46:31 2017 +0200

    Rename class VppCommTask to VstCommTask.

commit 9478c6c56c995b014b7a14408544b76fa676a317
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:30:47 2017 +0200

    Rename protocol versions to VST in ProtocolVersion enum.

commit 0b69b7add6d7b5155ac71b8257def282ab4a1637
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:29:29 2017 +0200

    Recognize incoming chunk headers in VST/1.1 format.

commit f40173c4fc6dc48613ae67ac3fd810d002c9b6cd
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:11:54 2017 +0200

    Recognise and remember VST/1.0 and VST/1.1 protocol versions.

commit 29c798dc1af2d650c41eb9ed06f635786faf10f6
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:11:23 2017 +0200

    Create protocol version VST/1.1 and disable vst+ prefix for endpoints.
2017-05-12 23:22:22 +02:00
jsteemann 422b06211f fix some replication issues 2017-05-11 19:00:32 +02:00
Simon Grätzer 9950fa7245 removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
jsteemann 8c2be23d73 do not call SyncWAL() on Windows, as it does not work 2017-05-05 15:37:38 +02:00
jsteemann ac7e0a225f fixed issues found by scan tool 2017-05-05 13:51:23 +02:00
Dan Larkin d77efe38dc Added explicit WAL file lifecycle management to reduce space overhead from replication. 2017-05-04 15:16:24 -04:00
Max Neunhoeffer 1b0e976a55 Fix ignoreDistributeShardsLikeErrors behaviour. 2017-05-03 09:13:03 +02:00
jsteemann 45096dd503 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-05-02 16:09:35 +02:00
Simon Grätzer 9f40893bfd Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-05-02 15:08:13 +02:00
Simon Grätzer e335a29326 better incremental sync 2017-05-02 15:08:05 +02:00
Simon Grätzer c55ce367eb WAL: honor tick end value 2017-05-02 13:55:19 +02:00
Simon Grätzer 09de672ba1 WAL fiter after collection 2017-05-02 13:46:43 +02:00
Michael Hackstein b6bac742b1 The RestReplicationHandler now inserts new ObjectIds when replicating collections 2017-05-02 13:17:18 +02:00
jsteemann 910f5680b0 moved files 2017-05-02 12:34:28 +02:00
Dan Larkin b46ac15300 Fixed 2.8 dump compatibility and fixed a few more HTTP replication tests. 2017-04-28 16:50:39 -04:00
Simon Grätzer 84ed7ec100 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel
# Conflicts:
#	arangod/RocksDBEngine/RocksDBRestReplicationHandler.cpp
2017-04-28 14:40:17 +02:00
Simon Grätzer 0a00a7eb67 Fixed broken JSON output 2017-04-28 14:28:34 +02:00
Jan Christoph Uhde ce762fe872 add handleCommandDetermineOpenTransactions 2017-04-28 14:21:55 +02:00
Dan Larkin e2853e4b52 Enabled saving applier config and fixed some replication tests/behavior. 2017-04-27 22:39:48 -04:00
Simon Grätzer 5e8f649772 Fixed WAL tailing 2017-04-27 19:14:57 +02:00
Michael Hackstein 5c0f54a111 Copied missing RestReplicationHandlers from MMFiles to RocksDB. Preparation for ClusterSynchronous replication. Untested yet. 2017-04-27 17:16:51 +02:00
Jan Christoph Uhde 22028f14b7 add rest of applier api 2017-04-27 14:31:51 +02:00
Jan Christoph Uhde 9418144210 add applier-state 2017-04-27 13:43:57 +02:00
Andreas Streichardt dad5a1429e Add waitForSyncReplication as a _create() option 2017-04-26 09:57:40 +02:00
Michael Hackstein 344979709e Merge branch 'devel' of github.com:arangodb/arangodb into devel 2017-04-26 09:36:09 +02:00
Michael Hackstein 52b3b67479 Fixed cluster restore of System Collections. 2017-04-26 08:58:17 +02:00
Jan Christoph Uhde 26112be584 do checking before creating copy of slice 2017-04-26 08:33:23 +02:00
Dan Larkin 9dd275b497 Fixed dump/restore tests. 2017-04-25 13:54:56 -04:00
Jan Christoph Uhde fca8f01df6 create logger state just in one place 2017-04-25 14:58:32 +02:00
Simon Grätzer 8aaac6d5a2 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-04-25 12:02:04 +02:00
Simon Grätzer 5d20b79c0f Incremental Sync 2017-04-25 12:01:58 +02:00
Jan Christoph Uhde e5447d2e8c prevent crashing of replication when not using mmfiles engine 2017-04-25 11:11:58 +02:00
Simon Grätzer 91fe0f01ea formatting 2017-04-24 15:55:21 +02:00
Simon Grätzer 7176d34169 Fixed incremental synced 2017-04-24 15:40:48 +02:00
Simon Grätzer 88dc999001 Counter manager fixes 2017-04-24 12:43:44 +02:00
Simon Grätzer df57b291f8 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 2017-04-24 10:23:33 +02:00
Dan Larkin d7f56172fd Fixed a status return in replication handler. 2017-04-21 13:12:19 -04:00
Simon Grätzer 1265fb759c Merge + sync
# Conflicts:
#	arangod/RocksDBEngine/RocksDBReplicationContext.h
2017-04-21 18:11:01 +02:00
Simon Grätzer e850dd635a Incremental sync 2017-04-21 18:10:10 +02:00
Dan Larkin 557eddb912 Added batchSize support to collection dumping. 2017-04-21 11:51:56 -04:00
jsteemann 862ec7bae3 remove log garbage 2017-04-21 12:57:20 +02:00
Dan Larkin 30efe6c136 Made more progress on dump tests. 2017-04-20 23:05:56 -04:00
Dan Larkin c46d4c6a6f Merge branch 'engine-api' of github.com:arangodb/arangodb into engine-api 2017-04-20 18:04:36 -04:00
Dan Larkin 9d6aa4b297 Some more replication handler changes. 2017-04-20 18:04:33 -04:00
jsteemann c0348c2840 more fixes 2017-04-20 23:23:04 +02:00
jsteemann 493154a319 fix inventory when there is no batchId 2017-04-20 23:14:58 +02:00
Dan Larkin b7f8854666 Made some progress on dump tests. 2017-04-20 16:23:35 -04:00
Dan Larkin 57701ff766 Added (rough) logger-follow handler. 2017-04-20 14:59:49 -04:00
Jan Christoph Uhde 96b05c0f61 work on dump for rocksdb replication 2017-04-20 15:58:48 +02:00
Simon Grätzer 8bfdcaa000 Added TTL 2017-04-20 15:24:03 +02:00
Simon Grätzer a04f991aa0 Fixed warning 2017-04-20 13:34:37 +02:00
Simon Grätzer 3319a5d607 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
# Conflicts:
#	arangod/RocksDBEngine/RocksDBRestReplicationHandler.cpp
2017-04-20 12:47:22 +02:00
Simon Grätzer 388121158e Started inventory rest handler 2017-04-20 12:16:40 +02:00
Jan Christoph Uhde 325c64773d make use of ReplicationManager and ReplicationContext 2017-04-20 12:05:40 +02:00
Jan Christoph Uhde 7889291247 add contextId to replication handler 2017-04-20 10:56:17 +02:00
Dan Larkin f494f45785 Added initial, stripped RocksDBRestReplicationHandler. 2017-04-19 17:30:06 -04:00