arangodb/Documentation/README_maintainers.md

ArangoDB Documentation Maintainers manual

• Using Docker container
• Installing on local system
• Add / Synchronize external documentation
• Generate users documentation
• Using Gitbook
• Examples
  • Where to add new...
  • generate
• write markdown
• Include ditaa diagrams
• Read / use the documentation
• arangod Example tool
  • OUTPUT, RUN and AQL specifics
• Swagger integration
  • RESTQUERYPARAMETERS
  • RESTURLPARAMETERS
  • RESTDESCRIPTION
  • RESTRETURNCODES
  • RESTREPLYBODY
  • RESTHEADER
  • RESTURLPARAM
  • RESTALLBODYPARAM
  • RESTBODYPARAM
  • RESTSTRUCT

Using Docker container

We provide the docker container arangodb/documentation-builder which brings all neccessary dependencies to build the documentation.

The files and a description how to (re-)generate the Docker image are here:
https://github.com/arangodb-helper/build-docker-containers/tree/master/distros/debian/jessie.docu

You can automagically build it using

./scripts/generateDocumentation.sh

which will start the docker container, compile ArangoDB, generate fresh example snippets, generate swagger, and all gitbook produced output files.

You can also use proselint inside of that container to let it proof read your english ;-)

Installing on local system

Dependencies to build documentation:

• swagger 2 for the API-Documentation inside aardvark (no installation required)

• Node.js

  Make sure the option to Add to PATH is enabled. After installation, the following commands should be available everywhere:

  • node
  • npm

  If not, add the installation path to your environment variable PATH. Gitbook requires more recent node versions.

• Gitbook

  Can be installed with Node's package manager NPM:

    npm install gitbook-cli -g
  

• ditaa (DIagrams Through Ascii Art) to build the ascii art diagrams (optional)

• Calibre2 (optional, only required if you want to build the e-book version)

  http://calibre-ebook.com/download

  Run the installer and follow the instructions.

Add / Synchronize external documentation

We maintain documentation along with their respective git repositories of their component - be driver or other utilities which shouldn't be directly in sync to the ArangoDB core documentation. The maintainer of the respective component can alter the documentation, and once a good point in time is reached, it can be sync'ed over via Documentation/Scripts/fetchRefs.sh, which spiders the SUMMARY.md files of all books, creates a clone of the external resource, adds a don't edit this here note to the files, and copies them over. Use your github username as first parameter to clone using HTTP + authentification, or git if you want to use ssh+key for authentification

The syntax of the SUMMARY.md integration are special comment lines that contain git in them in a semicolon separated value list:

• The git repository - the gitrepository containing the documentation - we will clone this; If authentification is required, prepend an @ to github.com
• The directory name where to clone it under Documentation/Books/repos (so several integration points can share a working copy)
• Subdirectory - the sub-directory inside of the git repository to integrate, also used in the VERSIONS file
• Source - may be empty if the whole Subdirectory should be mapped into the book the SUMMARY.md is in, else specify source files (one per line) or directories
• Destination - may be empty if the sub-directory on the remote repo should be mapped into the book the SUMMARY.md is located in; else specify a file or directory.

If a other than the default branch should be checked out you can specify the branch|tag in the VERSIONS file. The syntax is EXTERNAL_DOC_{the-directory-name}={remote-branch-name|tag}

If private repositories with authentification need to be cloned, the integrator can specify a username/password pair to the script. He/She can also create a clone in the Documentation/Books/repos/$1 directory - where the script would clone it.

The script will reset & pull the repos.

For testing the user can checkout other remote branches in that directory.

Below the integration lines regular lines referencing the integrated .md's have to be put to add the .md's to the books summary.

Please note that the SUMMARY.md integration checks will fail if unreferenced .md's are present, or .md's are missing.

The fetched .md's should be committed along with the changes of the SUMMARY.md

An example integrating an authentificated directory structure:

#   https://@github.com/arangodb/arangosync.git;arangosync;doc-integration/Manual;;/
  * [Datacenter to datacenter replication](Scalability/DC2DC/README.md)
    * [Introduction](Scalability/DC2DC/Introduction.md)
    * [Applicability](Scalability/DC2DC/Applicability.md)
    * [Requirements](Scalability/DC2DC/Requirements.md)

Another example, integrating a single README.md from an unauthentificated repo mapping it into Drivers/js/:

* [Drivers](Drivers/README.md)
# https://github.com/arangodb/arangojs.git;arangojs;;README.md;Drivers/js/
  * [Javascript](Drivers/js/README.md)

Generate users documentation

If you've edited examples, see below how to regenerate them with ./utils/generateExamples.sh. If you've edited REST (AKA HTTP) documentation, first invoke ./utils/generateSwagger.sh. Run cd Documentation/Books && ./build.sh to generate it. The documentation will be generated in subfolders in arangodb/Documentation/Books/books - use your favorite browser to read it.

You may encounter permission problems with gitbook and its npm invocations. In that case, you need to run the command as root / Administrator.

If you see "device busy" errors on Windows, retry. Make sure you don't have intermediate files open in the ppbooks / books subfolder (i.e. browser or editor). It can also temporarily occur during phases of high HDD / SSD load.

The build-scripts contain several sanity checks, i.e. whether all examples are used, and no dead references are there. (see building examples in that case below)

If the markdown files aren't converted to html, or index.html shows a single chapter only (content of README.md), make sure Cygwin create native symlinks It does not, if SUMMARY.md in Books/ppbooks/ looks like this:

!<symlink>ÿþf o o

If sub-chapters do not show in the navigation, try another browser (Firefox). Chrome's security policies are pretty strict about localhost and file:// protocol. You may access the docs through a local web server to lift the restrictions. You can use pythons build in http server for this.

~/books$ python -m SimpleHTTPServer 8000

To only regereneate one file (faster) you may specify a filter:

make build-book NAME=Manual FILTER=Manual/Aql/Invoke.md

(regular expressions allowed)

Using Gitbook

The arangodb/Documentation/Books/build.sh script generates a website version of the manual.

If you want to generate all media ala PDF, ePUB, run arangodb/Documentation/books/build.sh build-dist-books (takes some time to complete).

If you want to generate only one of them, run below build commands in arangodb/Documentation/Books/books/[Manual|HTTP|AQL]/. Calibre's ebook-convert will be used for the conversion.

Generate a PDF:

gitbook pdf ./ppbooks/Manual ./target/path/filename.pdf

Generate an ePub:

gitbook epub ./ppbooks/Manual ./target/path/filename.epub

Examples

Where to add new...

• Documentation/DocuBlocks/* - markdown comments with execution section
• Documentation/Books/Aql|Cookbook|HTTP|Manual/SUMMARY.md - index of all sub documentations

generate

• ./utils/generateExamples.sh --onlyThisOne geoIndexSelect will only produce one example - geoIndexSelect
• ./utils/generateExamples.sh --onlyThisOne 'MOD.*' will only produce the examples matching that regex; Note that examples with enumerations in their name may base on others in their series - so you should generate the whole group.
• running onlyThisOne in conjunction with a pre-started server cuts down the execution time even more. In addition to the --onlyThisOne ... specify i.e. --server.endpoint tcp://127.0.0.1:8529 to utilize your already running arangod. Please note that examples may collide with existing collections like 'test' - you need to make sure your server is clean enough.
• you can use generateExamples like that: ./utils/generateExamples.sh \ --server.endpoint 'tcp://127.0.0.1:8529' \ --withPython C:/tools/python2/python.exe \ --onlyThisOne 'MOD.*'
• ./Documentation/Scripts/allExamples.sh generates a file where you can inspect all examples for readability.
• ./utils/generateSwagger.sh - on top level to generate the documentation interactively with the server; you may use the swagger editor to revalidate whether ../../js/apps/system/_admin/aardvark/APP/api-docs.json is accurate.
• cd Documentation/Books; make - to generate the HTML documentation

write markdown

md files are used for the structure. To join it with parts extracted from the program documentation you need to place hooks:

• @startDocuBlock <tdocuBlockName> is replaced by a Docublock extracted from source.
• @startDocuBlockInline <docuBlockName> till @endDocuBlock <docuBlockName> is replaced in with its own evaluated content - so @EXAMPLE_AQL | @EXAMPLE_ARANGOSH_[OUTPUT | RUN] sections are executed the same way as inside of source code documentation.

Include ditaa diagrams

We use the beautifull ditaa (DIagrams Through Ascii Art) to generate diagrams explaining flows etc. in our documentation.

We have i.e. Manual/Graphs/graph_user_in_group.ditaa which is transpiled by ditaa into a png file, thus you simply include a png file of the same name as image into the markdown: ![User in group example](graph_user_in_group.png) to reference it.

Read / use the documentation

• file:///Documentation/Books/books/Manual/index.html contains the generated manual
• JS-Console - Tools/API - Interactive swagger documentation which you can play with.

arangod Example tool

./utils/generateExamples.sh picks examples from the code documentation, executes them, and creates a transcript including their results.

Here is how its details work:

• all Documentation/DocuBlocks/.md* and Documentation/Books/.md* are searched.
• all lines inside of source code starting with '///' are matched, all lines in .md files.
• an example start is marked with @EXAMPLE_ARANGOSH_OUTPUT or @EXAMPLE_ARANGOSH_RUN
• the example is named by the string provided in brackets after the above key
• the output is written to Documentation/Examples/<name>.generated
• if your example depends on a storage engine, prepend <name> with _rocksdb or _mmfiles to run it against such a server
• examples end with @END_EXAMPLE_[OUTPUT|RUN|AQL]
• all code in between is executed as javascript in the arangosh while talking to a valid arangod.
• you should strive to group your examples by naming them with a common prefix per topic. You may inspect the generated js code in /tmp/arangosh.examples.js

OUTPUT, RUN and AQL specifics

By default, Examples should be self contained and thus not depend on each other. They should clean up the collections they create. Building will fail if resources aren't cleaned. However, if you intend a set of OUTPUT and RUN to demonstrate interactively and share generated ids, you have to use an alphabetical sortable naming scheme so they're executed in sequence. Using <modulename>_<sequencenumber>[a|b|c|d]_thisDoesThat seems to be a good scheme.

• EXAMPLE_ARANGOSH_OUTPUT is intended to create samples that the users can cut'n'paste into their arangosh. Its used for javascript api documentation.

  • wrapped lines: Lines starting with a pipe (/// |) are joined together with the next following line. You have to use this if you want to execute loops, functions or commands which shouldn't be torn apart by the framework.
  • Lines starting with var: The command behaves similar to the arangosh: the server reply won't be printed. However, the variable will be in the scope of the other lines - else it won't.
  • Lines starting with a Tilde (/// ~): These lines can be used for setup/teardown purposes. They are completely invisible in the generated example transcript.
  • ~removeIgnoreCollection("test") - the collection test may live across several tests.
  • ~addIgnoreCollection("test") - the collection test will again be alarmed if left over.

• it is executed line by line. If a line is intended to fail (aka throw an exception), you have to specify // xpError(ERROR_ARANGO_DOCUMENT_KEY_UNEXPECTED) so the exception will be caught; else the example is marked as broken. If you need to wrap that line, you may want to make the next line starting by a tilde to suppress an empty line:

  /// | someLongStatement()
  /// ~ // xpError(ERROR_ARANGO_DOCUMENT_KEY_UNEXPECTED)
  

• EXAMPLE_ARANGOSH_RUN is intended to be pasted into a unix shell with cURL to demonstrate how the REST-HTTP-APIs work. The whole chunk of code is executed at once, and is expected not to throw. You should use assert(condition) to ensure the result is what you've expected. The body can be a string, or a javascript object which is then represented in JSON format.

  • Send the HTTP-request: var response = logCurlRequest('POST', url, body);
  • check its response: assert(response.code === 200);
  • output a JSON server Reply: logJsonResponse(response); (will fail if its not a valid json)
  • output a JSONL server Reply: logJsonLResponse(response); (will fail if its not a valid json; use if the server responds with one json document per line; Add a note to the user that this is *(One JSON document per line)* above the example)
  • output HTML to the user: logHtmlResponse(response); (i.e. redirects have HTML documents)
  • output the plain text to dump to the user: logRawResponse(response); (don't use this if you expect a json reply)
  • dump the reply to the errorlog for testing (will mark run as failed): logErrorResponse(response);

• EXAMPLE_AQL is intended to contain AQL queries that can be pasted into arangosh or the webinterfaces query editor. Usually this query references an example dataset generator in js/common/modules/@arangodb/examples/examples.js which the users can also invoke to generate the data in their installation. This sort of example consists of three parts:

  • @DATASET{datasetName} - (optional) the name of the dataset in the above mentioned examples.js to be instanciated before executing this query.
  • @EXPLAIN{TRUE|FALSE} - (optional) print execution plan of the AQL query. The default is FALSE.
  • A following AQL query which may either end at the end of the comment block, or at the next section:
  • @BV - (optional) verbatim object containing the bind parameters to be passed into the query. Will also be put into the generated snippet.

Swagger integration

./utils/generateSwagger.sh scans the documentation, and generates swagger output. It scans for all documentationblocks containing @RESTHEADER. It is a prerequisite for integrating these blocks into the gitbook documentation.

Tokens may have curly brackets with comma separated fields. They may optionally be followed by subsequent text lines with a long descriptions.

Sections group a set of tokens; they don't have parameters.

Types Swagger has several native types referenced below: [integer|long|float|double|string|byte|binary|boolean|date|dateTime|password] - see the swagger documentation

It consists of several sections which in term have sub-parameters:

Supported Sections:

RESTQUERYPARAMETERS

Parameters to be appended to the URL in form of ?foo=bar add RESTQUERYPARAMs below

RESTURLPARAMETERS

Section of parameters placed in the URL in curly brackets, add RESTURLPARAMs below.

RESTDESCRIPTION

Long text describing this route.

RESTRETURNCODES

should consist of several RESTRETURNCODE tokens.

Supported Tokens:

RESTREPLYBODY

Similar to RESTBODYPARAM - just what the server will reply with.

RESTHEADER

Up to 3 parameters.

• [GET|POST|PUT|DELETE] url should start with a /, it may contain parameters in curly brackets, that have to be documented in subsequent RESTURLPARAM tokens in the RESTURLPARAMETERS section.
• long description
• operationId - this is a uniq string that identifies the source parameter for this rest route. It defaults to a de-spaced long description - if set once it shouldn't be changed anymore.

RESTURLPARAM

Consists of 3 values separated by ',': Attributes:

• name: name of the parameter
• type:
• [required|optionas] Optional is not supported anymore. Split the docublock into one with and one without.

Folowed by a long description.

RESTALLBODYPARAM

This API has a schemaless json body (in doubt just plain ascii)

RESTBODYPARAM

Attributes:

• name - the name of the parameter
• type - the swaggertype of the parameter
• required/optional - whether the user can omit this parameter
• subtype / format (can be empty)
  • subtype: if type is object or array, this references the enclosed variables. can be either a swaggertype, or a RESTRUCT
  • format: if type is a native swagger type, some support a format to specify them

RESTSTRUCT

Groups a set of attributes under the structure name to an object that can be referenced in other RESTSTRUCTs or RESTBODYPARAM attributes of type array or object

Attributes:

• name - the name of the parameter
• structure name - the type under which this structure can be reached (should be uniq!)
• type - the swaggertype of the parameter (or another RESTSTRUCT...)
• required/optional - whether the user can omit this parameter
• subtype / format (can be empty)
  • subtype: if type is object or array, this references the enclosed variables. can be either a swaggertype, or a RESTRUCT
  • format: if type is a native swagger type, some support a format to specify them