1
0
Fork 0
Commit Graph

158 Commits

Author SHA1 Message Date
Vasiliy 635db3b409 issue 355.4: remove redundant methods and code, use 'cp' instead of 'cmake copy_directory' where possible, use vocbase reference instead of pointer 2018-04-04 10:53:48 +03:00
Jan e567392e49
improve startup resilience in case there are datafile errors (#4975) 2018-04-03 17:32:30 +02:00
Vasiliy 7c25902b27 issue 355.3: allow IResearchLink creation, as opposed to IResearch View update, to direct which CIDs get included in an IResearchView snapshot 2018-04-02 15:27:48 +03:00
Andrey Abramov 2cca7a926b remove code duplication 2018-03-30 16:01:00 +03:00
Andrey Abramov b641cf9b13
remove view implementation part 1 2018-03-26 23:18:08 +03:00
Andrey Abramov 6b9bffa343 remove PhysicalView part 3 2018-03-23 15:17:05 +03:00
Andrey Abramov 47fd5808c3 remove PhysicalView part 2 2018-03-22 22:41:28 +03:00
Andrey Abramov a874fda823 remove PhysicalView part 1 2018-03-22 20:42:42 +03:00
Vasiliy 148bdb7158 issue 344.6: remove some redundant functions (#4842) 2018-03-15 11:03:35 +01:00
Vasiliy bf0068e7f5 Changes since last PR: use a single map in vocbase for storing LogicalDataSource instances 2018-03-08 17:28:08 +03:00
Vasiliy daa1db659c manually-merge: differentiate data-source types by category e.g. LogicalCollection/LogicalView 2018-03-08 13:40:25 +03:00
Vasiliy f9a45032cd LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups (#4732)
* Changes since last PR: use LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups
* backport: address cluster LogicalDataSource resolution failure
2018-03-06 11:41:41 +01:00
Jan 47cbc52686
improved logging for MMFiles IO operations (#4366) 2018-01-19 18:37:01 +01:00
Jan 2a96df5ca5
Feature iresearch (#4071) 2017-12-18 15:04:59 +01:00
Simon Grätzer 19f68df395 Use uuid in Initial sync to ensure proper collection mapping (#3965)
* Use uuid in initalsync to ensure proper collection mapping

* fix compile error after merge
2017-12-13 22:12:47 +01:00
Jan 282be208cc
remove TRI_usleep and TRI_sleep, and use std::this_thread::sleep_for … (#3817) 2017-12-06 18:43:49 +01:00
Jan 733f27e997 Bug fix/fix compilation with gxx7 (#3637) 2017-11-10 16:00:57 +01:00
Jan 7613bc4314 Bug fix/fixes 0211 (#3568)
* remove some non-unused V8 persistents

* do not throw that many bogus assertions

* do not rely on server role being defined

* slightly better debug output for V8 context debugging

* fix collection ids in inventory response

* simplify bootstrap a bit

* slightly better error handling

* make elapsed time a queryable value

* use less memory for stub collections

* added assertions that will always make sense

* added assertions

* do not garbage-collect while waiting

* less copying of parameters

* do not show "load indexes into memory" buttons for mmfiles engine

  as all indexes are in memory anyway

* when a collection is truncated via the web interface, flush the WAL and rotate all active journals

this will make close all open journals on leader and followers and make them subject to compaction opportunities

* fix invalid server id values being passed from web interface to backend

* introduce afterTruncate method for indexes

* added test case for issue #3447

* updated CHANGELOG

* don't warn about replicationFactor for system collections

* check that the queries actually use the geo index and not some other index

* properly report error in web interface

* fix some internals checks that made truncate fail for bigger collections in maintainer mode

* also run a compact() operation after a serious truncate

in order to make iteration over the truncated range much faster
when the collection is next accessed

* increase default maximum number of V8 contexts to at least 16
2017-11-09 12:48:15 +01:00
Simon Grätzer 64e9377c05 Replacing /_api/collection with RestHandler (#3543) 2017-11-02 14:57:17 +01:00
Simon Grätzer 09840239c9 Fixed wait for sync in mmfiles (#3478) 2017-10-23 12:36:50 +02:00
Simon Grätzer 7c31960cf2 Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
Jan 0561bf45ce Bug fix/isrestore (#3283)
* Make isRestore work in the cluster.

This covers sharded collections with default sharding and non-default
sharding.

* always use locally generate revision ids for storing and looking up documents
2017-10-03 11:53:49 +02:00
Wilfried Goesgens 0faf9be695 Feature/compaction configuration (#3217)
* make the different values influencing the compaction run configurable

* Compaction statistics handling

 - we mustn't keep the number of dead objects on the compacted datafiles statistics, else it will be compacted again.
 - keep statistics of the compaction runs on the DatafileStatistics object
 - add the new statistics on DatafileStatistics to the figures api
 - implement test that assures only one compaction is run, and the statistic values are maintained

* don't mention the version number

* Implement review

 - fix documentation
 - allow 0 maxfiles to enable users to disable combined of datafiles
 - add statistic element that counts the number of combined datafiles

* Implement review

 - fix documentation
 - use locks to make statistic values consistent.
 - fix typo in variable name

* fix temporary variable unnecessary.

* changelog
2017-09-15 14:37:30 +02:00
Jan Christoph Uhde df76bbc690 Bug fix/planning issue 514 replication api (#3021)
* add "cluster selectivity estimates" to CHANGELOG

* add some documentation to RocksDBRestReplicationHandler

* fix building with relative paths

* add some more doc

* add some tests for the replication api

* fix RocksDBRestReplicationHandler and add tests

* update documentation

* remove obsolete parameter

* fix error message

* Implementing logger-first-tick, logger-tick-ranges. Fixing dump `chunkSize` documentation
2017-08-22 10:46:06 +02:00
Jan 8e65fbb539 Bug fix/issues 1708 (#3060)
* we must now ignore that datafiles are not sealed

this is because an unsealed datafile may have been produced by
renaming multiple journals to datafiles at server start

* acquire collection count after we have acquired the lock

* count the null byte as well

* fix count value acquisition

* send query fragments to the correct servers, even after failover or when a follower drops

the problem with using the previous shard-based approach is that responsibilities for shards may change at runtime
however, an AQL query must send all requests for the query to the initially used servers.
if there is a failover while the query is executing, we must still send all following requests to the same servers, and not the newly responsible servers
otherwise we potentially would try to get data from a query from server B while the query was only
instanciated on server A.
2017-08-17 21:48:27 +02:00
Jan 6180fcfdd1 Bug fix/prevent multiple journals (#3027)
* prevent multiple journals

* fix documentation

* remove _nrDesired, as it is not used anymore
2017-08-15 23:02:08 +02:00
Jan cf4368881d change init order of mmfiles logfile manager and database feature (#2697)
database feature now comes first. this ensures that the mmfiles collector thread
(owned by the mmfiles logfile manager) can always access the list of databases,
and that this list is not destroyed while the collector thread is still running.
2017-07-01 08:47:57 +02:00
Dan Larkin ee8cb749bb Added recovery tests for views and fixed a few related bugs. (#2682)
* Added recovery tests for views and fixed a few related bugs.

* Added more view recovery tests.

* Modified view recovery tests to add a waitForSync operation after.

* fixed usage of wrong view type

* fixed recovery of view change markers
2017-06-29 17:09:43 +02:00
Jan ebaeb639b2 Bug fix/incremental replication syncs too much (#2631)
* fix too much synchronization
* fix incremental sync for RocksDB engine
2017-06-21 14:56:33 +02:00
jsteemann 377431b138 abort waitForSync ops during server shutdown 2017-06-01 17:07:09 +02:00
Simon Grätzer 66f10870dd Renaming files 2017-05-31 18:12:47 +02:00
jsteemann 53b150b206 added tests for intermediate commits 2017-05-30 12:03:58 +02:00
jsteemann 8bc6b3d3a1 removed default parameters 2017-05-18 15:06:19 +02:00
Simon Grätzer 9950fa7245 removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
Jan Christoph Uhde a4a04350a3 fix replication 2017-05-08 17:43:36 +02:00
Jan Christoph Uhde acee270075 fix small bugs revealed by tests 2017-05-08 15:43:09 +02:00
Jan Christoph Uhde 8ee0406c64 move last logger creation to engines 2017-05-08 15:18:19 +02:00
Jan Christoph Uhde cf333f3b0a move creation of tickranges and first ticks into storage engines 2017-05-08 14:11:41 +02:00
Jan Christoph Uhde a8cc4c802d move logger state creation into engines 2017-05-08 12:04:31 +02:00
jsteemann d352794360 optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
Jan Christoph Uhde 3cb1cc7a52 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel
* 'devel' of https://github.com/arangodb/arangodb:
  Fixing index markers
  exclusive locks for indexes
  better incremental sync
  grunt build
  Avoid log spam.
  Removed code paths that wrote objectIds into the Agency. This did break replication.
  WAL: honor tick end value
  WAL fiter after collection
  Reactivated client-side filtering of unnecessary markers
  Added an Assert when persisting an index it's objectId is not allowed to be 0
  The RestReplicationHandler now inserts new ObjectIds when replicating collections
  moved files
  add dependencies for TransactionManager
  fix include
  move engine-specific test into engine test
2017-05-02 16:41:43 +02:00
Jan Christoph Uhde da02fd36c6 move engine specific syncer code into engines 2017-05-02 16:36:27 +02:00
jsteemann 64ec345481 add dependencies for TransactionManager 2017-05-02 12:22:13 +02:00
jsteemann f36a889709 try to fix shutdown issues 2017-04-26 09:58:19 +02:00
jsteemann 60d39f4650 replication bugfixes 2017-04-20 15:20:22 +02:00
jsteemann 78302d1766 make versionFilename() a property of the engine 2017-04-04 17:23:02 +02:00
Jan Christoph Uhde 234180c08b wip - drop database 2017-04-03 16:52:21 +02:00
jsteemann cedee8ed02 fix recovery test 2017-03-28 10:46:56 +02:00
jsteemann 17c1a6db05 fix recovery test 2017-03-28 10:04:39 +02:00
jsteemann 7bd6bd6e45 slightly adjusted index creation 2017-03-28 08:46:01 +02:00
jsteemann 3a2462532d move engine-specific includes out of standard file 2017-03-27 16:36:34 +02:00
jsteemann 55c38befbf enum databases, views and collections at startup 2017-03-23 11:52:22 +01:00
jsteemann 806c33eb1c cleanup 2017-03-23 09:01:21 +01:00
jsteemann e270ddb262 fixed failing test 2017-03-22 14:46:10 +01:00
jsteemann 71bf18dd12 renaming 2017-03-22 13:01:40 +01:00
jsteemann b8e011eee5 move attributes into engine 2017-03-21 10:31:50 +01:00
jsteemann 5d6dcb3bf1 re-added constructionguard 2017-03-17 15:44:22 +01:00
jsteemann e3d8ae142e some fixes for view handling 2017-03-17 11:18:46 +01:00
Dan Larkin 508aabeafe Updated schema, added LoggerView, and added support for implementation properties. 2017-03-16 19:45:48 -04:00
jsteemann 0bb2711abb Merge branch 'engine-api' of https://github.com/arangodb/arangodb into views 2017-03-16 12:16:23 +01:00
jsteemann 7e7fe8bc21 discover views on startup 2017-03-15 14:48:10 +01:00
jsteemann 1a06f1e15e initial functionality for views - unstable 2017-03-14 16:35:45 +01:00
Michael Hackstein bf6f785415 Merge branch 'engine-api' of github.com:arangodb/arangodb into engine-api 2017-03-13 17:31:51 +01:00
Michael Hackstein 1c33e77bd8 Moved unloadCollection callback into StorageEngine. 2017-03-13 17:28:08 +01:00
jsteemann 0a07bb3222 make RestWalHandler a handler of the MMFiles engine 2017-03-10 09:00:37 +01:00
jsteemann f1ba4874fe functionality for plugging REST handlers 2017-03-09 17:12:00 +01:00
Michael Hackstein 046793815c Merge branch 'engine-api' of github.com:arangodb/arangodb into engine-api 2017-03-09 16:37:51 +02:00
jsteemann 645889949c move engine-specific V8 functions into storage engine 2017-03-09 12:59:44 +01:00
Michael Hackstein 90b4876d3c Moved WriteMarker code Vocbase => MMFilesEngine. Also removed the decission to write a marker out of Vocbase-API 2017-03-09 11:12:39 +01:00
jsteemann 6b882ca8a2 class renaming 2017-03-08 17:54:00 +01:00
Michael Hackstein 489f0d12b8 Moved the decision to write a marker into the databaseFeature, it is not triggered from outside any more. Furthermore renameCollection write marker is now done in MMFiles. 2017-03-08 14:54:25 +01:00
jsteemann 57dd92902c move engine-specific optimizer rules into engine 2017-03-07 13:10:58 +01:00
Michael Hackstein 31df7eb527 Fixed error in recovery for duplicate collection name 2017-03-06 17:34:54 +01:00
Michael Hackstein 16c43192a0 Fixed feature startup ordering because recovery of presistent indices doid fail 2017-03-03 13:40:17 +00:00
jsteemann 42fccb69c6 remove dependency on MMFilesLogfileManager 2017-03-01 15:50:10 +01:00
Michael Hackstein cac32a4d1d Merge branch 'engine-api' of github.com:arangodb/arangodb into engine-api 2017-02-28 14:59:13 +01:00
Michael Hackstein df4eadac75 Replaced several specialized toVelocyPack functions of LogicalCollection by toVelocyPackIgnore(). Fixed distributeLike bug that occured through this. 2017-02-28 14:54:32 +01:00
jsteemann ae751e7360 create engine-specific transaction context data 2017-02-27 17:05:30 +01:00
jsteemann 90f32ed525 renamed classes 2017-02-27 11:40:15 +01:00
Michael Hackstein 5d452edec6 Moved ditches into MMFiles. 2017-02-23 09:02:34 +01:00
Jan Christoph Uhde 6e1f77be24 remove ditches from interface. the ditches concept needs fixing!! 2017-02-21 10:53:51 +01:00
Jan Christoph Uhde cf970ed230 Squashed commit of the following:
commit 2644735fe06806ec8ffb78ceb2ede04037258952
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Tue Feb 21 09:10:55 2017 +0100

    failed ammend

commit c2b8604df5a60fd00e57b4f088fe8a2624d9ac3b
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Feb 20 12:49:58 2017 +0100

    move - maxTick to Physical - RACE!
2017-02-21 09:19:49 +01:00
Jan Christoph Uhde 21cb8e3217 Merge branch 'engine-api' of github.com:arangodb/arangodb into obi-engine-api
* 'engine-api' of github.com:arangodb/arangodb:
  Removed readRevision from LogicalCollection. You only get readDocument with a token. Moved it's logic into MMFilesCollection. removed the ignoreWal parameter from readDocumentConditional. It was always true anyways...
  Moved pre-commit document modifications out of logical-collection into the physical collection. They are only relevant when actually writing to disc.
  Moved useSecondaryIndexes check from Logical to MMFiles collection
  Some cleanup in Logical collection imports
  Moved index filling into the MMFiles collection
  changelog
  grunt build
  css
  ui: queries now using cursors api - issue #2065
  remove crosscompile hack - it doesn't work anymore
  updated CHANGELOG
  add changlog entry for stacktrace log messages
  clean up write lockers a bit
  fix too early closing of sockets
  Moved restoreIndex to Physical collection.
  Fix a shell script used during build.
  Lower log level to debug in shutdown message.
  fix spelling
  Moved saveIndex logic into the physical collection
  fix potential deadlock

Conflicts:
	arangod/MMFiles/MMFilesCollection.cpp
	arangod/MMFiles/MMFilesCollection.h
	arangod/VocBase/LogicalCollection.cpp
2017-02-18 19:10:45 +01:00
Jan Christoph Uhde 2d5e74f7f6 move journaize() to PhysicalCollection
MMFilesCollection ctor might be buggy
2017-02-17 20:06:47 +01:00
jsteemann c652a313da Merge branch 'devel' of https://github.com/arangodb/arangodb into engine-api 2017-02-17 14:03:59 +01:00
jsteemann 3e7432ccd6 clean up write lockers a bit 2017-02-16 15:44:45 +01:00
Michael Hackstein a3aae2e43b Moved saveIndex logic into the physical collection 2017-02-16 12:53:05 +01:00
Michael Hackstein c3ff82f4e1 Moved version management to MMFiles collection. Logical does not need to know it. Removed PhysicalCollection inclode from logicical collection. Now we should rebuild less code whan changing sth in storage engine. 2017-02-15 15:48:33 +01:00
Jan Christoph Uhde be89cac445 move path from logical to physical 2017-02-15 11:43:38 +01:00
jsteemann 8a2e353f3d move encoding functions into their own file 2017-02-14 12:57:15 +01:00
jsteemann cbe8ee1172 renamed feature 2017-02-14 12:35:54 +01:00
Jan Christoph Uhde 163b3d45bf remove fullyCollected and friends from LogicalCollection 2017-02-13 11:47:22 +01:00
Jan Christoph Uhde 929390cee1 move index marker creation to storage engine
remove MMFilesLogfileManager from LogicalCollection

fix marker type
2017-02-10 15:14:34 +01:00
Jan Christoph Uhde 9131cde18b Merge branch 'devel' of github.com:arangodb/arangodb into devel
* 'devel' of github.com:arangodb/arangodb:
  fix another merge conflict
  fix compile error after merge
  remove logging for non-topics
  Destroy all traverser engines when AQL is shut down.
2017-02-10 10:58:07 +01:00
Jan Christoph Uhde 7a66b0772a clean up headers 2017-02-10 10:57:51 +01:00
jsteemann 23ed96b7a7 fix compile error after merge 2017-02-10 09:36:49 +01:00
jsteemann 0850e8293a Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-02-10 09:34:55 +01:00
jsteemann d024a6d00a remove logging for non-topics 2017-02-10 09:32:50 +01:00
Jan Christoph Uhde 714dd3ee3a rebase database-management 2017-02-09 18:47:20 +01:00
Michael Hackstein 7cec7a4628 Merge branch 'engine-api' of github.com:arangodb/arangodb into engine-api 2017-02-08 12:54:26 +01:00