1
0
Fork 0
arangodb/arangod/MMFiles
Simon 1d57a46168 Refactor Authentication Layer (3.3) (#4588)
* added tests for revokeCollection and revokeDatabase

* optimized user permission test

* ui selection bugfix

* fixed ldap ui login

* login view

* Authentication refactoring

* localstorage fallback if user config is not available

* Fixing permission resolution test

* Adding missing import

* local storage queries now supported

* disabled collection task check for ldap

* added internal ldapEnabled function and ldap config to the ui

* more db creation tests

* removed console logs

* render fix

* Various authentication related fixes

* exec ldap test also for cluster

* Adding support to refresh user rights from external auth sources

* ldap test howto comment

* Handling roles more correctly

* jwt

* login view

* First part of rework of LDAP documentation.

* test roles in a ldap environment

* Changing role handling

* Finish revision of the LDAP chapter in the manual.

* Fixing user header

* Fixing some slight issues with LDAP users and roles

* Removing unused code

* Removing unused code

* added ldap test

* fixing a bug in restuserhandler

* more ldap tests

* ldap tests

* ldap tests

* optimized ldap testing, added cluster support, fixed some auth tests

* ldap cleanup test

* ldap tests

* auth tests

* ldap test

* Changing permission defaults

* revert change

* updated user helper test

* rm of try catch block connection

* Removing FeatureCacheFeature

* Changed permission resolution according to discussion

* updated the docs to clarify the permission resolution and the intricacies of LDAP users

* Fixing wrong permissions check in handling of PUT in ResUserHandler

* Using revision ID when replacing users

* Adding basic replace test

* tests

* Fixing some outstanding issues

* Fixing test setup, optimizing some stuff

* Fixing permission resolution rules, testsuite setup, etc

* Fix deadlock

* Adding error message for keyspace, slightly changing test setup

* Removing remaining mentions of FeatureCacheFeature

* Fix jslint

* Fixing some failed tests

* Fixing cluster authentication issue, red tests

* Fixing ldap testsuite, adding trace logging

* Fixint ldap tesuite setup and LDAP recognition

* Fixing an assert

* Cleanup, adding changelog entry

* fix typo

* Fixing dump_authentication test

* improvements found during code review

* oops

* updated CHANGELOG

* Fixing broken handling, disallowing adding of local users when disabled

* added tests for ldap search mode

* Fixing testInvalidGrants

(cherry picked from commit bc7ea2aaa29a9ed0974898f487e8a318f24912f1)

* Removing undefined auth level externally

(cherry picked from commit 70859f43ae6fd694fdbf70f669fbfdafc58e7913)

* Fixing previous commit

(cherry picked from commit 2fbcffd2ed657862ef9fb5e6d45201a6ec8ada69)

* more use of sessionstorage

* intentionally removed `after` methods from tests

because they are executed before the tests start
no cleanup is performed right now after the authentication tests
however, a cleanup is done at start of every test

* ldap tests all modes

* fix LDAP test invocation

* Added roles transformation to ldap test suite

* Fix compilation of community version.

* Imrpved the ldap testsuites by unifying their options

* fix permission problems for system collections

* Improved LDAP configuration documentation.

* Grunt.

* fixed some ro/rw display issues

* fixed some ro/rw display issues part 2

* grunt build

* bump version number

* Fixed typos in LDAP manual
2018-02-28 13:24:18 +01:00
..
CMakeLists.txt Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesAllocatorThread.cpp
MMFilesAllocatorThread.h
MMFilesAqlFunctions.cpp fixed issue #4185: On execution of FULLTEXT search / AQL query db is … (#4239) 2018-01-15 22:46:14 +01:00
MMFilesAqlFunctions.h
MMFilesCleanupThread.cpp Bug fix/fix compilation with gxx7 (#3637) 2017-11-10 16:00:57 +01:00
MMFilesCleanupThread.h fix fulltext index removal performance, simplified code (#3015) 2017-08-12 09:25:30 +02:00
MMFilesCollection.cpp properly unlock after restore-indexes (#4369) 2018-01-25 15:56:13 +01:00
MMFilesCollection.h fixed issue #4185: On execution of FULLTEXT search / AQL query db is … (#4239) 2018-01-15 22:46:14 +01:00
MMFilesCollectionExport.cpp Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesCollectionExport.h Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesCollectionKeys.cpp fix premature unlock (#3802) (#4027) 2017-12-13 18:46:14 +01:00
MMFilesCollectionKeys.h fix premature unlock (#3802) (#4027) 2017-12-13 18:46:14 +01:00
MMFilesCollectionReadLocker.h
MMFilesCollectionWriteLocker.h
MMFilesCollectorCache.h Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesCollectorThread.cpp improved logging for MMFiles IO operations (#4365) 2018-01-19 18:36:49 +01:00
MMFilesCollectorThread.h fix fulltext index removal performance, simplified code (#3015) 2017-08-12 09:25:30 +02:00
MMFilesCompactionFeature.cpp Bug fix/issues 2709 (#3333) 2017-10-01 09:43:03 +02:00
MMFilesCompactionFeature.h fix dropping of system collections and retrieving list of user databa… (#3304) 2017-09-21 10:05:46 +02:00
MMFilesCompactionLocker.h
MMFilesCompactorThread.cpp improved logging for MMFiles IO operations (#4365) 2018-01-19 18:36:49 +01:00
MMFilesCompactorThread.h Feature/compaction configuration (#3217) 2017-09-15 14:37:30 +02:00
MMFilesDatafile.cpp improved logging for MMFiles IO operations (#4365) 2018-01-19 18:36:49 +01:00
MMFilesDatafile.h potential fix for issue #3517: several "filesystem full" errors in log… (#3525) 2017-10-30 17:28:59 +01:00
MMFilesDatafileHelper.h Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesDatafileStatistics.cpp do not complain about missing datafile statistics (#3688) 2017-11-14 16:42:31 +01:00
MMFilesDatafileStatistics.h do not complain about missing datafile statistics (#3688) 2017-11-14 16:42:31 +01:00
MMFilesDatafileStatisticsContainer.cpp
MMFilesDatafileStatisticsContainer.h
MMFilesDitch.cpp
MMFilesDitch.h
MMFilesDocumentOperation.cpp Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesDocumentOperation.h Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesDocumentPosition.h Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesEdgeIndex.cpp Bug fix/fixes 1711 obi (#3730) 2017-11-17 16:34:33 +01:00
MMFilesEdgeIndex.h Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesEngine.cpp improved logging for MMFiles IO operations (#4365) 2018-01-19 18:36:49 +01:00
MMFilesEngine.h improved logging for MMFiles IO operations (#4365) 2018-01-19 18:36:49 +01:00
MMFilesExportCursor.cpp Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesExportCursor.h Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesFulltextIndex.cpp Feature 3.3/fulltext optimizations (#3925) 2017-12-19 21:40:12 +01:00
MMFilesFulltextIndex.h Feature 3.3/fulltext optimizations (#3925) 2017-12-19 21:40:12 +01:00
MMFilesGeoIndex.cpp Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesGeoIndex.h Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesHashIndex.cpp Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesHashIndex.h Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesIncrementalSync.h Active Failover for Foxx Services (3.3) (#4593) 2018-02-15 09:36:25 +01:00
MMFilesIndex.h Bug fix/fixes 0211 (#3568) 2017-11-09 12:48:15 +01:00
MMFilesIndexElement.cpp Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesIndexElement.h Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesIndexFactory.cpp Bug fix/fixes 0211 (#3568) 2017-11-09 12:48:15 +01:00
MMFilesIndexFactory.h
MMFilesLogfileManager.cpp Refactor Authentication Layer (3.3) (#4588) 2018-02-28 13:24:18 +01:00
MMFilesLogfileManager.h make truncate a bit less obstrusive (#3721) 2017-11-16 20:28:33 +01:00
MMFilesOptimizerRules.cpp Bug fix/fixes 0609 (#3227) 2017-09-13 16:28:21 +02:00
MMFilesOptimizerRules.h
MMFilesPathBasedIndex.cpp Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesPathBasedIndex.h Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesPersistentIndex.cpp Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesPersistentIndex.h Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesPersistentIndexFeature.cpp Bug fix/fixes 0609 (#3227) 2017-09-13 16:28:21 +02:00
MMFilesPersistentIndexFeature.h
MMFilesPersistentIndexKeyComparator.cpp
MMFilesPersistentIndexKeyComparator.h
MMFilesPrimaryIndex.cpp speed up collection.any() for mmfiles (#4432) 2018-02-08 11:36:27 +01:00
MMFilesPrimaryIndex.h Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesRemoverThread.cpp
MMFilesRemoverThread.h
MMFilesRestExportHandler.cpp Active Failover for Foxx Services (3.3) (#4593) 2018-02-15 09:36:25 +01:00
MMFilesRestExportHandler.h
MMFilesRestHandlers.cpp
MMFilesRestHandlers.h
MMFilesRestReplicationHandler.cpp Active Failover for Foxx Services (3.3) (#4593) 2018-02-15 09:36:25 +01:00
MMFilesRestReplicationHandler.h Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesRestWalHandler.cpp make truncate a bit less obstrusive (#3721) 2017-11-16 20:28:33 +01:00
MMFilesRestWalHandler.h
MMFilesRevisionsCache.cpp Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesRevisionsCache.h Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesSkiplist.h Bug fix/fixes 0609 (#3227) 2017-09-13 16:28:21 +02:00
MMFilesSkiplistIndex.cpp properly unlock after restore-indexes (#4369) 2018-01-25 15:56:13 +01:00
MMFilesSkiplistIndex.h Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesSynchronizerThread.cpp improved logging for MMFiles IO operations (#4365) 2018-01-19 18:36:49 +01:00
MMFilesSynchronizerThread.h
MMFilesTransactionCollection.cpp fix nullptr access (#4143) 2017-12-22 10:56:01 +01:00
MMFilesTransactionCollection.h fix premature unlock (#3802) (#4027) 2017-12-13 18:46:14 +01:00
MMFilesTransactionContextData.cpp
MMFilesTransactionContextData.h
MMFilesTransactionManager.cpp
MMFilesTransactionManager.h
MMFilesTransactionState.cpp Missing things for active / passive (#3578) 2017-11-04 20:30:23 +01:00
MMFilesTransactionState.h Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesV8Functions.cpp Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
MMFilesV8Functions.h
MMFilesView.cpp
MMFilesView.h
MMFilesWalAccess.cpp forward port replication bugfixes from 3.2 (#4575) 2018-02-13 22:14:23 +01:00
MMFilesWalAccess.h Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
MMFilesWalLogfile.cpp Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesWalLogfile.h
MMFilesWalMarker.h
MMFilesWalRecoverState.cpp fix issue #4272 (#4418) 2018-01-25 15:57:06 +01:00
MMFilesWalRecoverState.h
MMFilesWalRecoveryFeature.cpp add a test for replication applier restarts (#3750) (#3933) 2017-12-08 15:48:15 +01:00
MMFilesWalRecoveryFeature.h
MMFilesWalSlot.cpp Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
MMFilesWalSlot.h fix WAL file tick updating (happened too late before) (#2762) 2017-07-10 17:41:08 +02:00
MMFilesWalSlots.cpp Bug fix/fix compilation with gxx7 (#3637) 2017-11-10 16:00:57 +01:00
MMFilesWalSlots.h fix WAL file tick updating (happened too late before) (#2762) 2017-07-10 17:41:08 +02:00
MMFilesWalSyncRegion.h
README.md
mmfiles-fulltext-common.h
mmfiles-fulltext-index.cpp Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
mmfiles-fulltext-index.h Bug fix/isrestore (#3283) 2017-10-03 11:53:49 +02:00
mmfiles-fulltext-list.cpp Bug fix/fixes 0609 (#3227) 2017-09-13 16:28:21 +02:00
mmfiles-fulltext-list.h fix fulltext index removal performance, simplified code (#3015) 2017-08-12 09:25:30 +02:00
mmfiles-fulltext-query.cpp Bug fix/fixes 0609 (#3227) 2017-09-13 16:28:21 +02:00
mmfiles-fulltext-query.h
mmfiles-geo-index.cpp Bug fix/fixes 0609 (#3227) 2017-09-13 16:28:21 +02:00
mmfiles-geo-index.h
mmfiles-replication-common.cpp Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
mmfiles-replication-common.h Active Failover for Foxx Services (3.3) (#4593) 2018-02-15 09:36:25 +01:00
mmfiles-replication-dump.cpp forward port replication bugfixes from 3.2 (#4575) 2018-02-13 22:14:23 +01:00
mmfiles-replication-dump.h Active Failover for Foxx Services (3.3) (#4593) 2018-02-15 09:36:25 +01:00

README.md

MMFilesEngine

How operations are stored - Overview

All operations like insert or remove are written as markers to a write ahead log (WAL). This WAL consists of files of a certain size and if such a file is full (or is manually flushed), all relevant markers are transferred (transferMarkers()) to the journals of the respective collections. During the transfer any obsolete markers will be thrown away: a sequence of insert, remove, insert on the same document will result in the last insert discarding the previous operations. When a journal file of size (journalSize()) is full, it will be sealed and renamed. By applying these operations it will become a datafile that is read-only. Datafiles will eventually be merged by a compactor thread that does about the same work as the transferMarkers function, reducing the size of the stored data.

Ditches

Ditches are used to pin objects in WAL or journal as long as they are used by other operations.