1
0
Fork 0
arangodb/arangod/MMFiles
Max Neunhoeffer f614439416 Squashed commit of the following:
commit 19a7210119cc284af64251b202f690ab62bf615c
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 23:20:10 2017 +0200

    Allow access to /_api/user/ without database check in VST as in HTTP.

    This is a try to fix #2465

commit f973c1335652540174bf6b78df42290bded357da
Merge: 30bdc4c 365de45
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 21:32:15 2017 +0200

    Merge remote-tracking branch 'origin/devel' into vst-cleanup

commit 30bdc4ca38e028d58cebea8268683d9f94e87ad7
Merge: 14acaea 086f6a2
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 14:52:23 2017 +0200

    Merge remote-tracking branch 'origin/devel' into vst-cleanup

commit 14acaea8cb5fff2c1c8c27eee32294bfd612f629
Merge: 5bd0e79 fb088d3
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:40:29 2017 +0200

    Merge remote-tracking branch 'origin/devel' into vst-cleanup

commit 5bd0e79ab053447f94f2fd959a33b1e15302b6a8
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:36:23 2017 +0200

    Fix bug in authentication in VST

commit 9a999ce5e629aee5816a4f61b194b086d11fd169
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:24:35 2017 +0200

    Open up endpoints /_open/* in VST without being authenticated

commit 32f62db42af6d5bfc24214ebb4ca7c1fb473b55e
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:11:31 2017 +0200

    Simplify logic by not using boost::optional.

commit 925ce2f7b890c6f14205fd3e98553939d89bec61
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 12:11:20 2017 +0200

    Add JWT case for authentication for VST

commit f1d7f67a9fc20c91ca000e9a4dad91e0f50f1652
Merge: 040ace3 7a9ccc9
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 08:58:51 2017 +0200

    Merge remote-tracking branch 'origin/devel' into vst-cleanup

commit 040ace3e9dce47ddea5f51d29f0153a70d257c9e
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 08:23:16 2017 +0200

    Write out chunk buffer using little endian uints

commit 66ad4c0e8d3bc94091664505986b00e7cff39f2f
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Fri May 12 08:12:38 2017 +0200

    Move maxChunkSize query to constructor, remove dead code

commit f7b4c26cc09c6d512362340c6bf6061bbaae61a9
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 17:57:14 2017 +0200

    Do not take protocolVersion from request for VST.

commit c76d6685f9507701715abf882f0ac0e192ead59f
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:20:58 2017 +0200

    Remove dead code for compression which did never work well.

commit b823765b7c041dc5d143f7ab2e13d92f5f42d3eb
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:09:53 2017 +0200

    Fix renaming to make compiler happy.

commit b327830012d146f4f4e2d0f0467dab351fbd3bd0
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:07:36 2017 +0200

    Rename _defaultPortVpp to _defaultPortVst.

commit 5eb98a9e64d702476e8ec098b7506b1a63145d64
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:06:28 2017 +0200

    Rename header guards VPP -> VST.

commit a3e96be26d49d6151898d8d757ede6225ac6abd2
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:05:55 2017 +0200

    Rename VppRequest and VppResponse files to Vst.

commit 1d5f6f196490c3cf9312be5927b967b4abc03b67
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:04:19 2017 +0200

    Change names of header guards VPP -> VST.

commit f004b25fbef75050b34bcd4eae4eb17b8b467230
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 13:03:19 2017 +0200

    Rename files VppNetwork.h and VstMessage.h

commit c04d3aec19e8f0a8a3b90cd48e7736bff037359d
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 12:59:15 2017 +0200

    Implement correct chunk header for sending out VelocyStream 1.1 messages.

commit 73aeedfbbef24d5929c44bd26c16343480514270
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 01:11:23 2017 +0200

    A lot more renaming VPP -> VST.

commit 2acedc77f16a82251bc4f9e39526184e2a6f0dec
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:49:20 2017 +0200

    Rename header guard.

commit 2ade43946aa5ea0048dab81eed5ffd931ce35ea8
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:48:01 2017 +0200

    Rename files VppCommTask.* to VstCommTask.*

commit 0ffcda02a51cbad35e5ec06117d397aac37323cd
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:46:31 2017 +0200

    Rename class VppCommTask to VstCommTask.

commit 9478c6c56c995b014b7a14408544b76fa676a317
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:30:47 2017 +0200

    Rename protocol versions to VST in ProtocolVersion enum.

commit 0b69b7add6d7b5155ac71b8257def282ab4a1637
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:29:29 2017 +0200

    Recognize incoming chunk headers in VST/1.1 format.

commit f40173c4fc6dc48613ae67ac3fd810d002c9b6cd
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:11:54 2017 +0200

    Recognise and remember VST/1.0 and VST/1.1 protocol versions.

commit 29c798dc1af2d650c41eb9ed06f635786faf10f6
Author: Max Neunhoeffer <max@arangodb.com>
Date:   Thu May 11 00:11:23 2017 +0200

    Create protocol version VST/1.1 and disable vst+ prefix for endpoints.
2017-05-12 23:22:22 +02:00
..
CMakeLists.txt optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
MMFilesAllocatorThread.cpp
MMFilesAllocatorThread.h
MMFilesAqlFunctions.cpp fix handling of externals 2017-03-30 17:37:09 +02:00
MMFilesAqlFunctions.h functionality for plugging REST handlers 2017-03-09 17:12:00 +01:00
MMFilesCleanupThread.cpp removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesCleanupThread.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesCollection.cpp removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesCollection.h removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesCollectionExport.cpp removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesCollectionExport.h move attributes into engine 2017-03-21 10:31:50 +01:00
MMFilesCollectionKeys.cpp removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesCollectionKeys.h move collection keys into engine 2017-03-21 14:23:29 +01:00
MMFilesCollectionReadLocker.h honor exclusive locks 2017-04-25 12:56:56 +02:00
MMFilesCollectionWriteLocker.h honor exclusive locks 2017-04-25 12:56:56 +02:00
MMFilesCollectorCache.h renaming 2017-03-22 13:01:40 +01:00
MMFilesCollectorThread.cpp optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
MMFilesCollectorThread.h
MMFilesCompactionLocker.h renamed classes 2017-02-28 17:59:48 +01:00
MMFilesCompactorThread.cpp removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesCompactorThread.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesDatafile.cpp Various warnings 2017-04-26 11:26:08 +02:00
MMFilesDatafile.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesDatafileHelper.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesDatafileStatistics.cpp moved files 2017-03-16 16:45:02 +01:00
MMFilesDatafileStatistics.h moved files 2017-03-16 16:45:02 +01:00
MMFilesDatafileStatisticsContainer.cpp moved files 2017-03-16 16:45:02 +01:00
MMFilesDatafileStatisticsContainer.h moved files 2017-03-16 16:45:02 +01:00
MMFilesDitch.cpp renamed classes 2017-02-27 11:40:15 +01:00
MMFilesDitch.h renamed classes 2017-02-27 11:40:15 +01:00
MMFilesDocumentOperation.cpp
MMFilesDocumentOperation.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesDocumentPosition.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesEdgeIndex.cpp fix crashes 2017-04-25 15:47:11 +02:00
MMFilesEdgeIndex.h fix crashes 2017-04-25 15:47:11 +02:00
MMFilesEngine.cpp removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesEngine.h removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesExportCursor.cpp move attributes into engine 2017-03-21 10:31:50 +01:00
MMFilesExportCursor.h move attributes into engine 2017-03-21 10:31:50 +01:00
MMFilesFulltextIndex.cpp Geo index update, renaming 2017-05-11 13:19:51 +02:00
MMFilesFulltextIndex.h Fixing fulltext deduplication 2017-05-10 16:17:46 +02:00
MMFilesGeoIndex.cpp Changed index VPack building to include forPersistence flag. 2017-04-17 17:04:50 -04:00
MMFilesGeoIndex.h Changed index VPack building to include forPersistence flag. 2017-04-17 17:04:50 -04:00
MMFilesHashIndex.cpp fix crashes 2017-04-25 15:47:11 +02:00
MMFilesHashIndex.h fix crashes 2017-04-25 15:47:11 +02:00
MMFilesIndexElement.cpp
MMFilesIndexElement.h
MMFilesIndexFactory.cpp Removed code paths that wrote objectIds into the Agency. This did break replication. 2017-05-02 14:59:19 +02:00
MMFilesIndexFactory.h Removed code paths that wrote objectIds into the Agency. This did break replication. 2017-05-02 14:59:19 +02:00
MMFilesInitialSync.h move engine specific syncer code into engines 2017-05-02 16:36:27 +02:00
MMFilesLogfileManager.cpp optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
MMFilesLogfileManager.h optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
MMFilesOptimizerRules.cpp Working geo index + tests 2017-05-09 15:04:51 +02:00
MMFilesOptimizerRules.h Working geo index + tests 2017-05-09 15:04:51 +02:00
MMFilesPathBasedIndex.cpp added assert 2017-04-12 15:38:01 +02:00
MMFilesPathBasedIndex.h Path based enumerator 2017-03-29 17:33:02 +02:00
MMFilesPersistentIndex.cpp Merge branch 'devel' of https://github.com/arangodb/arangodb into engine-api 2017-04-18 11:33:16 +02:00
MMFilesPersistentIndex.h Changed index VPack building to include forPersistence flag. 2017-04-17 17:04:50 -04:00
MMFilesPersistentIndexFeature.cpp remove unused options 2017-05-12 12:12:09 +02:00
MMFilesPersistentIndexFeature.h change --rocksdb.enabled to be a obosolete option 2017-04-19 14:54:10 +02:00
MMFilesPersistentIndexKeyComparator.cpp class renaming 2017-03-08 17:54:00 +01:00
MMFilesPersistentIndexKeyComparator.h class renaming 2017-03-08 17:54:00 +01:00
MMFilesPrimaryIndex.cpp fixes for non-array IN lookups, added tests 2017-04-25 01:47:07 +02:00
MMFilesPrimaryIndex.h Changed index VPack building to include forPersistence flag. 2017-04-17 17:04:50 -04:00
MMFilesRemoverThread.cpp
MMFilesRemoverThread.h
MMFilesRestExportHandler.cpp moved export into engine 2017-03-21 11:41:50 +01:00
MMFilesRestExportHandler.h move attributes into engine 2017-03-21 10:31:50 +01:00
MMFilesRestHandlers.cpp Moved MMFiles replication handler. 2017-04-18 15:35:45 -04:00
MMFilesRestHandlers.h make RestWalHandler a handler of the MMFiles engine 2017-03-10 09:00:37 +01:00
MMFilesRestReplicationHandler.cpp Squashed commit of the following: 2017-05-12 23:22:22 +02:00
MMFilesRestReplicationHandler.h more dictributeShardsLike code mergedfrom 3.1 2017-04-24 15:13:40 +02:00
MMFilesRestWalHandler.cpp make RestWalHandler a handler of the MMFiles engine 2017-03-10 09:00:37 +01:00
MMFilesRestWalHandler.h make RestWalHandler a handler of the MMFiles engine 2017-03-10 09:00:37 +01:00
MMFilesRevisionsCache.cpp renaming 2017-03-22 13:01:40 +01:00
MMFilesRevisionsCache.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesSkiplist.h
MMFilesSkiplistIndex.cpp fix gcc 4.9 compilation issue 2017-04-24 11:25:10 +02:00
MMFilesSkiplistIndex.h Changed index VPack building to include forPersistence flag. 2017-04-17 17:04:50 -04:00
MMFilesSynchronizerThread.cpp
MMFilesSynchronizerThread.h
MMFilesToken.h
MMFilesTransactionCollection.cpp turn off deadlock detector in obvious cases in which it is not necessary 2017-04-27 14:47:19 +02:00
MMFilesTransactionCollection.h more startup 2017-03-24 14:20:51 +01:00
MMFilesTransactionContextData.cpp removed unused includes 2017-02-27 17:29:28 +01:00
MMFilesTransactionContextData.h removed unused includes 2017-02-27 17:29:28 +01:00
MMFilesTransactionManager.cpp optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
MMFilesTransactionManager.h optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
MMFilesTransactionState.cpp optimize query cache invalidation for MMFiles a bit 2017-04-20 10:59:37 +02:00
MMFilesTransactionState.h refactor some code to make use of arangodb::Result 2017-03-30 09:39:21 +02:00
MMFilesV8Functions.cpp fix db._engineStats() 2017-05-04 00:33:36 +02:00
MMFilesV8Functions.h fix db._engineStats() 2017-05-04 00:33:36 +02:00
MMFilesView.cpp some fixes for view handling 2017-03-17 11:18:46 +01:00
MMFilesView.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesWalLogfile.cpp
MMFilesWalLogfile.h renaming 2017-03-22 13:01:40 +01:00
MMFilesWalMarker.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesWalRecoverState.cpp fix compile warning 2017-04-19 12:31:13 +02:00
MMFilesWalRecoverState.h renaming 2017-03-22 13:01:40 +01:00
MMFilesWalRecoveryFeature.cpp try to fix startup order 2017-04-25 11:14:10 +02:00
MMFilesWalRecoveryFeature.h
MMFilesWalSlot.cpp renaming 2017-03-22 13:01:40 +01:00
MMFilesWalSlot.h
MMFilesWalSlots.cpp renaming 2017-03-22 13:01:40 +01:00
MMFilesWalSlots.h
MMFilesWalSyncRegion.h
README.md
mmfiles-fulltext-common.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-handles.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-handles.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-index.cpp cppcheck 2017-05-10 18:24:34 +02:00
mmfiles-fulltext-index.h cppcheck 2017-05-10 18:24:34 +02:00
mmfiles-fulltext-list.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-list.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-query.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-query.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-result.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-result.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-geo-index.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-geo-index.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-replication-dump.cpp Squashed commit of the following: 2017-05-12 23:22:22 +02:00
mmfiles-replication-dump.h Squashed commit of the following: 2017-05-12 23:22:22 +02:00

README.md

MMFilesEngine

How operations are stored - Overview

All operations like insert or remove are written as markers to a write ahead log (WAL). This WAL consists of files of a certain size and if such a file is full (or is manually flushed), all relevant markers are transferred (transferMarkers()) to the journals of the respective collections. During the transfer any obsolete markers will be thrown away: a sequence of insert, remove, insert on the same document will result in the last insert discarding the previous operations. When a journal file of size (journalSize()) is full, it will be sealed and renamed. By applying these operations it will become a datafile that is read-only. Datafiles will eventually be merged by a compactor thread that does about the same work as the transferMarkers function, reducing the size of the stored data.

Ditches

Ditches are used to pin objects in WAL or journal as long as they are used by other operations.