diff --git a/.github/workflows/diff.yaml b/.github/workflows/diff.yaml index 3cd5e7e05..efe269720 100644 --- a/.github/workflows/diff.yaml +++ b/.github/workflows/diff.yaml @@ -4,7 +4,7 @@ on: push: paths-ignore: - 'docs/**' - - '*.md' + - '**/*.md' jobs: community_build: diff --git a/README.md b/README.md index a141030e3..858c6e18d 100644 --- a/README.md +++ b/README.md @@ -6,34 +6,8 @@ data structures, multi-version concurrency control and asynchronous IO. ## Development Documentation -* [Quick Start](docs/dev/quick-start.md) -* [Workflow](docs/dev/workflow.md) -* [Storage](docs/dev/storage/v2/contents.md) -* [Query Engine](docs/dev/query/contents.md) -* [Communication](docs/dev/communication/contents.md) -* [Lisp C++ Preprocessor (LCP)](docs/dev/lcp.md) - -## Feature Specifications - -Each prominent Memgraph feature requires a feature specification. The purpose -of the feature specification is to have a base for discussing all aspects of -the feature. Elements of feature specifications should be: - -* High-level context. -* Interface. -* User stories. Usage from the end-user perspective. In the case of a library, - that should be cases on how to use the programming interface. In the case of -a shell script, that should be cases on how to use flags. -* Discussion about concurrency, memory management, error management. -* Any other essential functional or non-functional requirements. -* Test and benchmark strategy. -* Possible future changes/improvements/extensions. -* Security concerns. -* Additional and/or optional implementation details. - -It's crucial to keep feature spec up-to-date with the implementation. Take a -look at the list of [feature specifications](docs/feature_spec/contents.md) to -learn more about powerful Memgraph features. +Please continue +[here](https://www.notion.so/memgraph/memgraph-0428591638604c8385550e214ea9f3e6). ## User Documentation diff --git a/docs/dev/code-review.md b/docs/dev/code-review.md deleted file mode 100644 index c7f2659ae..000000000 --- a/docs/dev/code-review.md +++ /dev/null @@ -1,269 +0,0 @@ -# Code Review Guidelines - -This chapter describes some of the things you should be on the lookout when -reviewing someone else's code. - -## Exceptions - -Although the Google C++ Style Guide forbids exceptions, we do allow them in -our codebase. As a reviewer you should watch out for the following. - -The documentation of throwing functions needs to be in-sync with the -implementation. This must be enforced recursively. I.e. if a function A now -throws a new exception, and the function B uses A, then B needs to handle that -exception or have its documentation updated and so on. Naturally, the same -applies when an exception is removed. - -Transitive callers of the function which throws a new exception must be OK -with that. This ties into the previous point. You need to check that all users -of the new exception either handle it correctly or propagate it. - -Exceptions should not escape out of class destructors, because that will -terminate the program. The code should be changed so that such cases are not -possible. - -Exceptions being thrown in class constructors. Although this is well defined -in C++, it usually implies that a constructor is doing too much work and the -class construction & initialization should be redesigned. Usual approaches are -using the (Static) Factory Method pattern or having some sort of an -initialization method that needs to be called after the construction is done. -Prefer the Factory Method. - -Don't forget that STL functions may also throw! - -## Pointers & References - -In cases when some code passes a pointer or reference, or if a code stores a -pointer or reference you should take a careful look at the following. - - * Lifetime of the pointed to value (this includes both ownership and - multithreaded access). - * In case of a class, check validity of destructor and move/copy - constructors. - * Is the pointed to value mutated, if not it should be `const` (`const Type - *` or `const Type &`). - -## Allocators & Memory Resources - -With the introduction of polymorphic allocators (C++17 `` and -our `utils/memory.hpp`) we get a more convenient type signatures for -containers so as to keep the outward facing API nice. This convenience comes -at a cost of less static checks on the type level due to type erasure. - -For example: - - std::pmr::vector first_vec(std::pmr::null_memory_resource()); - std::pmr::vector second_vec(std::pmr::new_delete_resource()); - - second_vec = first_vec // What happens here? - - // Or with our implementation - utils::MonotonicBufferResource monotonic_memory(1024); - std::vector> first_vec(&monotonic_memory); - std::vector> second_vec(utils::NewDeleteResource()); - - second_vec = first_vec // What happens here? - -In the above, both `first_vec` and `second_vec` have the same type, but have -*different* allocators! This can lead to ambiguity when moving or copying -elements between them. - -You need to watch out for the following. - - * Swapping can lead to undefined behaviour if the allocators are not equal. - * Is the move construction done with the right allocator. - * Is the move assignment done correctly, also it may throw an exception. - * Is the copy construction done with the right allocator. - * Is the copy assignment done correctly. - * Using `auto` makes allocator propagation rules rather ambiguous. - -## Classes & Object Oriented Programming - -A common mistake is to use classes, inheritance and "OOP" when it's not -needed. This sections shows examples of encountered cases. - -### Classes without (Meaningful) Members - - class MyCoolClass { - public: - int BeCool(int a, int b) { return a + b; } - - void SaySomethingCool() { std::cout << "Hello!"; } - }; - -The above class has no members (i.e. state) which affect the behaviour of -methods. This class should need not exist, it can be easily replaced with a -more modular (and shorter) design -- top level functions. - - int BeCool(int a, int b) { return a + b; } - - void SaySomethingCool() { std::cout << "Hello!"; } - -### Classes with a Single Public Method - - clas MyAwesomeClass { - public: - MyAwesomeClass(int state) : state_(state) {} - - int GetAwesome() { return GetAwesomeImpl() + 1; } - - private: - int state_; - - int GetAwesomeImpl() { return state_; } - }; - -The above class has a `state_` and even a private method, but there's only one -public method -- `GetAwesome`. - -You should check "Does the stored state have any meaningful influence on the -public method?", similarly to the previous point. - -In the above case it doesn't, and the class should be replaced with a public -function in `.hpp` while the private method should become a private function -in `.cpp` (static or in anonymous namespace). - - // hpp - int GetAwesome(int state); - - // cpp - namespace { - int GetAwesomeImpl(int state) { return state; } - } - int GetAwesome(int state) { return GetAwesomeImpl(state) + 1; } - -A counterexample is when the state is meaningful. - - class Counter { - public: - Counter(int state) : state_(state) {} - - int Get() { return state_++; } - - private: - int state_; - }; - -But even that could be replaced with a closure. - - auto MakeCounter(int state) { - return [state]() mutable { return state++; }; - } - -### Private Methods - -Instead of private methods, top level functions should be preferred. The -reasoning is completely explained in "Effective C++" Item 23 by Scott Meyers. -In our codebase, even improvements to compilation times can be noticed if -private methods in interface (`.hpp`) files are replaced with top level -functions in implementation (`.cpp`) files. - -### Inheritance - -The rule is simple -- if there are no virtual methods (but maybe destructor), -then the class should be marked as `final` and never inherited. - -If there are virtual methods (i.e. class is meant to be inherited), make sure -that either a public virtual destructor or a protected non-virtual destructor -exist. See "Effective C++" Item 7 by Scott Meyers. Also take a look at -"Effective C++" Items 32---39 by Scott Meyers. - -An example of how inheritance with no virtual methods is replaced with -composition. - - class MyBase { - public: - virtual ~MyBase() {} - - void DoSomethingBase() { ... } - }; - - class MyDerived final : public MyBase { - public: - void DoSomethingNew() { ... DoSomethingBase(); ... } - }; - -With composition, the above becomes. - - class MyBase final { - public: - void DoSomethingBase() { ... } - }; - - class MyDerived final { - MyBase base_; - - public: - void DoSomethingNew() { ... base_.DoSomethingBase(); ... } - }; - -The composition approach is preferred as it encapsulates the fact that -`MyBase` is used for the implementation and users only interact with the -public interface of `MyDerived`. Additionally, you can easily replace `MyBase -base_;` with a C++ PIMPL idiom (`std::unique_ptr base_;`) to make the -code more modular with regards to compilation. - -More advanced C++ users will recognize that the encapsulation feature of the -non-PIMPL composition can be replaced with private inheritance. - - class MyDerived final : private MyBase { - public: - void DoSomethingNew() { ... MyBase::DoSomethingBase(); ... } - }; - -One of the common "counterexample" is the ability to store objects of -different type in a container or pass them to a function. Unfortunately, this -is not that good of a design. For example. - - class MyBase { - ... // No virtual methods (but the destructor) - }; - - class MyFirstClass final : public MyBase { ... }; - - class MySecondClass final : public MyBase { ... }; - - std::vector> first_and_second_classes; - first_and_second_classes.push_back(std::make_unique()); - first_and_second_classes.push_back(std::make_unique()); - - void FunctionOnFirstOrSecond(const MyBase &first_or_second, ...) { ... } - -With C++17, the containers for different types should be implemented with -`std::variant`, and as before the functions can be templated. - - class MyFirstClass final { ... }; - - class MySecondClass final { ... }; - - std::vector> first_and_second_classes; - // Notice no heap allocation, since we don't store a pointer - first_and_second_classes.emplace_back(MyFirstClass()); - first_and_second_classes.emplace_back(MySecondClass()); - - // You can also use `std::variant` here instead of template - template - void FunctionOnFirstOrSecond(const TFirstOrSecond &first_or_second, ...) { ... } - -Naturally, if the base class has meaningful virtual methods (i.e. other than -destructor) it maybe is OK to use inheritance but also consider alternatives. -See "Effective C++" Items 32---39 by Scott Meyers. - -### Multiple Inheritance - -Multiple inheritance should not be used unless all base classes are pure -interface classes. This decision is inherited from [Google C++ Style -Guide](https://google.github.io/styleguide/cppguide.html#Inheritance). For -example on how to design with and around multiple inheritance refer to -"Effective C++" Item 40 by Scott Meyers. - -Naturally, if there *really* is no better design, then multiple inheritance is -allowed. An example of this can be found in our codebase when inheriting -Visitor classes (though even that could be replaced with `std::variant` for -example). - -## Code Format & Style - -If something doesn't conform to our code formatting and style, just refer the -author to either [C++ Style](cpp-code-conventions.md) or [Other Code -Conventions](other-code-conventions.md). diff --git a/docs/dev/communication/contents.md b/docs/dev/communication/contents.md deleted file mode 100644 index 6b515b32a..000000000 --- a/docs/dev/communication/contents.md +++ /dev/null @@ -1,5 +0,0 @@ -# Communication - -## Bolt - -Memgraph implements [Bolt communication protocol](https://7687.org/). diff --git a/docs/dev/cpp-code-conventions.md b/docs/dev/cpp-code-conventions.md deleted file mode 100644 index 163a5c902..000000000 --- a/docs/dev/cpp-code-conventions.md +++ /dev/null @@ -1,350 +0,0 @@ -# C++ Code Conventions - -This chapter describes code conventions which should be followed when writing -C++ code. - -## Code Style - -Memgraph uses the -[Google Style Guide for C++](https://google.github.io/styleguide/cppguide.html) -in most of its code. You should follow them whenever writing new code. -Besides following the style guide, take a look at -[Code Review Guidelines](code-review.md) for common design issues and pitfalls -with C++ as well as [Required Reading](required-reading.md). - -### Often Overlooked Style Conventions - -#### Pointers & References - -References provide a shorter syntax for accessing members and better declare -the intent that a pointer *should* not be `nullptr`. They do not prevent -accessing a `nullptr` and obfuscate the client/calling code because the -reference argument is passed just like a value. Errors with such code have -been very difficult to debug. Therefore, pointers are always used. They will -not prevent bugs but will make some of them more obvious when reading code. - -The only time a reference can be used is if it is `const`. Note that this -kind of reference is not allowed if it is stored somewhere, i.e. in a class. -You should use a pointer to `const` then. The primary reason being is that -references obscure the semantics of moving an object, thus making bugs with -references pointing to invalid memory harder to track down. - -Example of this can be seen while capturing member variables by reference -inside a lambda. -Let's define a class that has two members, where one of those members is a -lambda that captures the other member by reference. - -```cpp -struct S { - std::function foo; - int bar; - - S() : foo([&]() { std::cout << bar; }) - {} -}; -``` -What would happend if we move an instance of this object? Our lambda -reference capture will point to the same location as before, i.e. it -will point to the **old** memory location of `bar`. This means we have -a dangling reference in our code! -There are multiple ways to avoid this. The simple solutions would be -capturing by value or disabling move constructors/assignments. -Still, if we capture by reference an object that is not a member -of the struct containing the lambda, we can still have a dangling -reference if we move that object somewhere in our code and there is -nothing we can do to prevent that. -So, be careful with lambda catptures, and remember that references are -still a pointer under the hood! - -[Style guide reference](https://google.github.io/styleguide/cppguide.html#Reference_Arguments) - -#### Constructors & RAII - -RAII (Resource Acquisition is Initialization) is a nice mechanism for managing -resources. It is especially useful when exceptions are used, such as in our -code. Unfortunately, they do have 2 major downsides. - - * Only exceptions can be used for to signal failure. - * Calls to virtual methods are not resolved as expected. - -For those reasons the style guide recommends minimal work that cannot fail. -Using virtual methods or doing a lot more should be delegated to some form of -`Init` method, possibly coupled with static factory methods. Similar rules -apply to destructors, which are not allowed to even throw exceptions. - -[Style guide reference](https://google.github.io/styleguide/cppguide.html#Doing_Work_in_Constructors) - -#### Constructors and member variables - -One of the most powerful tools in C++ is the move semantics. We won't go into -detail how it works, but you should know how to utilize it as much as -possible. In our example we will define a small `struct` called `S` which -contains only a single member, `text` of type `std::string`. -```cpp -struct S { - std::string text; -}; -``` -We want to define a constructor that takes a `std::string`, and saves its -value in `text`. This is a common situation, where the constructor takes -a value, and saves it in the object to be constructed. - -Our first implementation would look like this: -```cpp -S(const std::string &s) : text(s) {} -``` - -This is a valid solution but with one downside - we always copy. If we -construct an object like this: -```cpp -S s("some text"); -``` -we would create a temporary `std::string` object and then copy it to our member -variable. - -Of course, we know what to do now - we will capture temporary variables using -`&&` and move it into our `text` variable. -```cpp -S(std::string &&s) : text(std::move(s)) {} -``` - -Now let's add an extra member variable of type `std::vector` called -`words`. Our constructors accept 2 values now - `std::vector` and -`std::string`. Those arguments could be passed by value, by reference, or -as rvalues. To cover all the cases we need to define a dedicated constructor -for each case. -Fortunately, there are two simpler options, the first one is writing a -templated constructor: -```cpp -template -S(T1 &&s, T2 &&v) : text(std::forward(s), words(std::forward(v) {} -``` -But don't forget to define `requires` clause so you don't accept any type. This -solution is optimal but really hard to read AND write. The second solution is -something you should ALWAYS prefer in these simple cases where we only store -one of the arguments: -```cpp -S(std::string s, std::vector v) : text(std::move(s)), words(std::move(v)) {} -``` -This way we have an almost optimal solution. The only extra operation we have is -the extra move when we send an `lvalue`. We would copy the value to the `s`, and -then move it to the `text` variable. Before, we would copy directly to `text`. -Also, you should ALWAYS write const-correct code, meaning `s` and `v` cannot be -`const` as it's not correct here. Why is that? You CANNOT move a const object! -It would just degrade to copying the object. I would say that this is a small -price to pay for a much cleaner and more maintainable code. - -### Additional Style Conventions - -Old code may have broken Google C++ Style accidentally, but the new code -should adhere to it as close as possible. We do have some exceptions -to Google style as well as additions for unspecified conventions. - -#### Using C++ Exceptions - -Unlike Google, we do not forbid using exceptions. - -But, you should be very careful when using them and introducing new ones. They -are indeed handy, but cause problems with understanding the control flow since -exceptions are another form of `goto`. It also becomes very hard to determine -that the program is in correct state after the stack is unwound and the thrown -exception handled. Other than those issues, throwing exceptions in destructors -will terminate the program. The same will happen if a thread doesn't handle an -exception even though it is not the main thread. - -[Style guide reference](https://google.github.io/styleguide/cppguide.html#Exceptions) - -In general, when introducing a new exception, either via `throw` statement or -calling a function which throws, you must examine all transitive callers and -update their implementation and/or documentation. - -#### Assertions - -We use `CHECK` and `DCHECK` macros from glog library. You are encouraged to -use them as often as possible to both document and validate various pre and -post conditions of a function. - -`CHECK` remains even in release build and should be preferred over it's cousin -`DCHECK` which only exists in debug builds. The primary reason is that you -want to trigger assertions in release builds in case the tests didn't -completely validate all code paths. It is better to fail fast and crash the -program, than to leave it in undefined state and potentially corrupt end -user's work. In cases when profiling shows that `CHECK` is causing visible -slowdown you should switch to `DCHECK`. - -#### Template Parameter Naming - -Template parameter names should start with capital letter 'T' followed by a -short descriptive name. For example: - -```cpp -template -class KeyValueStore -``` - -## Code Formatting - -You should install `clang-format` and run it on code you change or add. The -root of Memgraph's project contains the `.clang-format` file, which specifies -how formatting should behave. Running `clang-format -style=file` in the -project's root will read the file and behave as expected. For ease of use, you -should integrate formatting with your favourite editor. - -The code formatting isn't enforced, because sometimes manual formatting may -produce better results. Though, running `clang-format` is strongly encouraged. - -## Documentation - -Besides following the comment guidelines from [Google Style -Guide](https://google.github.io/styleguide/cppguide.html#Comments), your -documentation of the public API should be -[Doxygen](https://github.com/doxygen/doxygen) compatible. For private parts of -the code or for comments accompanying the implementation, you are free to -break doxygen compatibility. In both cases, you should write your -documentation as full sentences, correctly written in English. - -## Doxygen - -To start a Doxygen compatible documentation string, you should open your -comment with either a JavaDoc style block comment (`/**`) or a line comment -containing 3 slashes (`///`). Take a look at the 2 examples below. - -### Block Comment - -```cpp -/** - * One sentence, brief description. - * - * Long form description. - */ -``` - -### Line Comment - -```cpp -/// One sentence, brief description. -/// -/// Long form description. -``` - -If you only have a brief description, you may collapse the documentation into -a single line. - -### Block Comment - -```cpp -/** Brief description. */ -``` - -### Line Comment - -```cpp -/// Brief description. -``` - -Whichever style you choose, keep it consistent across the whole file. - -Doxygen supports various commands in comments, such as `@file` and `@param`. -These help Doxygen to render specified things differently or to track them for -cross referencing. If you want to learn more, take a look at these two links: - - * http://www.stack.nl/~dimitri/doxygen/manual/docblocks.html - * http://www.stack.nl/~dimitri/doxygen/manual/commands.html - -## Examples - -Below are a few examples of documentation from the codebase. - -### Function - -```cpp -/** - * Removes whitespace characters from the start and from the end of a string. - * - * @param s String that is going to be trimmed. - * - * @return Trimmed string. - */ -inline std::string Trim(const std::string &s); -``` - -### Class - -```cpp -/** Base class for logical operators. - * - * Each operator describes an operation, which is to be performed on the - * database. Operators are iterated over using a @c Cursor. Various operators - * can serve as inputs to others and thus a sequence of operations is formed. - */ -class LogicalOperator - : public ::utils::Visitable { - public: - /** Constructs a @c Cursor which is used to run this operator. - * - * @param GraphDbAccessor Used to perform operations on the database. - */ - virtual std::unique_ptr MakeCursor(GraphDbAccessor &db) const = 0; - - /** Return @c Symbol vector where the results will be stored. - * - * Currently, outputs symbols are only generated in @c Produce operator. - * @c Skip, @c Limit and @c OrderBy propagate the symbols from @c Produce (if - * it exists as input operator). In the future, we may want this method to - * return the symbols that will be set in this operator. - * - * @param SymbolTable used to find symbols for expressions. - * @return std::vector used for results. - */ - virtual std::vector OutputSymbols(const SymbolTable &) const { - return std::vector(); - } - - virtual ~LogicalOperator() {} -}; -``` - -### File Header - -```cpp -/// @file visitor.hpp -/// -/// This file contains the generic implementation of visitor pattern. -/// -/// There are 2 approaches to the pattern: -/// -/// * classic visitor pattern using @c Accept and @c Visit methods, and -/// * hierarchical visitor which also uses @c PreVisit and @c PostVisit -/// methods. -/// -/// Classic Visitor -/// =============== -/// -/// Explanation on the classic visitor pattern can be found from many -/// sources, but here is the link to hopefully most easily accessible -/// information: https://en.wikipedia.org/wiki/Visitor_pattern -/// -/// The idea behind the generic implementation of classic visitor pattern is to -/// allow returning any type via @c Accept and @c Visit methods. Traversing the -/// class hierarchy is relegated to the visitor classes. Therefore, visitor -/// should call @c Accept on children when visiting their parents. To implement -/// such a visitor refer to @c Visitor and @c Visitable classes. -/// -/// Hierarchical Visitor -/// ==================== -/// -/// Unlike the classic visitor, the intent of this design is to allow the -/// visited structure itself to control the traversal. This way the internal -/// children structure of classes can remain private. On the other hand, -/// visitors may want to differentiate visiting composite types from leaf types. -/// Composite types are those which contain visitable children, unlike the leaf -/// nodes. Differentiation is accomplished by providing @c PreVisit and @c -/// PostVisit methods, which should be called inside @c Accept of composite -/// types. Regular @c Visit is only called inside @c Accept of leaf types. -/// To implement such a visitor refer to @c CompositeVisitor, @c LeafVisitor and -/// @c Visitable classes. -/// -/// Implementation of hierarchical visiting is modelled after: -/// http://wiki.c2.com/?HierarchicalVisitorPattern -``` - diff --git a/docs/dev/lcp.md b/docs/dev/lcp.md deleted file mode 100644 index 79f2a680b..000000000 --- a/docs/dev/lcp.md +++ /dev/null @@ -1,1349 +0,0 @@ -# Lisp C++ Preprocessor (LCP) - -In our development process we are using Common Lisp to generate some parts of -the C++ codebase. The idea behind this is supplementing C++ with better -meta-programming capabilities to automate tasks and prevent bugs due to code -duplication. Primary candidate for using more powerful meta-programming is -generating serialization code. Such code is almost always the same: go through -all `struct` or `class` members and invoke the serialization function on them. -Writing such code manually is error prone when adding members, because you may -easily forget to correctly update the serialization code. Thus, the Lisp C++ -Preprocessor was born. It is hooked in our build process as a step before -compilation. The remainder of the document describes how to use LCP and its -features. - -Contents - - * [Running LCP](#running-lcp) - * [Writing LCP](#writing-lcp) - - [Inlining C++ in Common Lisp](#inlining-cpp) - - [C++ Namespaces](#cpp-namespaces) - - [C++ Enumerations](#cpp-enums) - - [C++ Classes & Structs](#cpp-classes) - - [Defining an RPC](#defining-an-rpc) - - [Cap'n Proto Serialization](#capnp-serial) - - [SaveLoadKit Serialization](#slk-serial) - - [Object Cloning](#object-cloning) - -## Running LCP - -You can generate C++ from an LCP file by running the following command. - -`./tools/lcp ` - -The LCP will produce a `path-to-file.hpp` file and potentially a -`path-to-file.lcp.cpp` file. The `.cpp` file is generated if some parts of the -code need to be in the implementation file. This is usually the case when -generating serialization code. Note that the `.cpp` file has the extension -appended to `.lcp`, so that you are free to define your own `path-to-file.cpp` -which includes the generated `path-to-file.hpp`. - -One serialization format uses Cap'n Proto library, but to use it, you need to -provide an ID. The ID is generated by invoking `capnp id`. When you want to -generate Cap'n Proto serialization, you need to pass the generated ID to LCP. - -`./tools/lcp $(capnp id)` - -Generating Cap'n Proto serialization will produce an additional file, -`path-to-file.capnp`, which contains the serialization schema. - -You may wonder why the LCP doesn't invoke `capnp id` itself. Unfortunately, -such behaviour would be wrong when running LCP on the same file multiple -times. Each run would produce a different ID and the serialization code would -be incompatible between versions. - -### CMake - -The LCP is run in CMake using the `add_lcp` function defined in -`CMakeLists.txt`. You can take a look at the function documentation there for -information on how to add your new LCP files to the build system. - -## Writing LCP - -A LCP file should have the `.lcp` extension, but the code written is -completely valid Common Lisp code. This means that you have a complete -language at your disposal before even the C++ is compiled. You can view this -as similar to the C++ templates and macros, but they do not have access to -a complete language. - -Besides Common Lisp, you are allowed to write C++ code verbatim. This means -that C++ and Lisp code coexist in the file. How to do that, as well as other -features are described below. - -### Inlining C++ in Common Lisp {#inlining-cpp} - -To insert C++ code, you need to use a `#>cpp ... cpp<#` block. This is most -often used at the top of the file to write Doxygen documentation and put some -includes. For example: - -```cpp -#>cpp -/// @file My Doxygen style documentation about this file - -#pragma once - -#include -cpp<# -``` - -The above code will be pasted as is into the generated header file. If you -wish to have a C++ block in the `.cpp` implementation file instead, you should -use `lcp:in-impl` function. For example: - -```cpp -(lcp:in-impl -#>cpp -void MyClass::Method(int awesome_number) { - // Do something with awesome_number -} -cpp<#) -``` - -The C++ block also supports string interpolation with a syntax akin to shell -variable access, `${lisp-variable}`. At the moment, only variables are -supported and they have to be pretty printable in Common Lisp (i.e. support -the `~A` format directive). For example, we can make a precomputed sinus -function for integers from 0 to 5: - -```lisp -(let ((sin-from-0-to-5 - (format nil "~{~A~^, ~}" (loop for i from 0 below 5 collect (sin i))))) - #>cpp - static const double kSinFrom0To5[] = {${sin-from-0-to-5}}; - cpp<#) -``` - -The following will be generated. - -```cpp -static const double kSinFrom0To5[] = {0.0, 0.84147096, 0.9092974, 0.14112, -0.7568025}; -``` - -Since you have a complete language at your disposal, this is a powerful tool -to generate tables for computations which would take a very long time during -the execution of the C++ program. - -### C++ Namespaces {#cpp-namespaces} - -Although you can use inline C++ to open and close namespaces, it is -recommended to use `lcp:namespace` and `lcp:pop-namespace` functions. LCP will -report an error if you have an unclosed namespace, unlike Clang and GCC which -most of the times give strange errors due to C++ grammar ambiguity. Additional -benefit is that LCP will track the namespace stack and correctly wrap any C++ -code which should be put in the `.cpp` file. - -For example: - -```lisp -;; example.lcp -(lcp:namespace utils) - -;; Function declaration in header -#>cpp -bool StartsWith(const std::string &string, const std::string &prefix); -cpp<# - -;; Function implementation in implementation file -(lcp:in-impl -#>cpp -bool StartsWith(const std::string &string, const std::string &prefix) { - // Implementation code - return false; -} -cpp<#) - -(lcp:pop-namespace) ;; utils -``` - -The above will produce 2 files, header and implementation: - -```cpp -// example.hpp -namespace utils { - -bool StartsWith(const std::string &string, const std::string &prefix); - -} -``` - -```cpp -// example.lcp.cpp -namespace utils { - -bool StartsWith(const std::string &string, const std::string &prefix) { - // Implementation code - return false; -} - -} -``` - -### C++ Enumerations {#cpp-enums} - -LCP provides a `lcp:define-enum` macro to define a C++ `enum class` type. This -will make LCP aware of the type and all its possible values. This makes it -possible to generate the serialization code. In the future, LCP may generate -"string to enum" and "enum to string" functions. - -Example: - -```lisp -(lcp:define-enum days-in-week - (monday tuesday wednesday thursday friday saturday sunday) - ;; Optional documentation - (:documentation "Enumerates days of the week") - ;; Optional directive to generate serialization code - (:serialize)) -``` - -Produces: - -```cpp -/// Enumerates days of the week -enum class DaysInWeek { - MONDAY, - TUESDAY, - WEDNESDAY, - THURSDAY, - FRIDAY, - SATURDAY, - SUNDAY -}; - -// serialization code ... -``` - -### C++ Classes & Structs {#cpp-classes} - -For defining C++ classes, there is a `lcp:define-class` macro. Its counterpart -for structures is `lcp:define-struct`. They are exactly the same, but -`lcp:define-struct` will put members in public scope by default. Just like in -C++. - -Defining classes is a bit more involved, because they have many customization -options. They syntax follows the syntax of class definition in Common Lisp -(see `defclass`). - -Basic example: - -```lisp -(lcp:define-class my-class () - ((primitive-value :int64_t) - (stl-vector "std::vector")) - ;; Optional documentation - (:documentation "My class documentation") - ;; Define explicitly public, protected or private code. All are optional. - (:public #>cpp // some public code, e.g. methods cpp<#) - (:protected #>cpp // protected cpp<#) - (:private #>cpp //private cpp<#)) -``` - -The above will generate: - -```cpp -/// My class documentation -class MyClass { - public: - // some public code, e.g. methods - - protected: - // protected - - private: - // private - - int64_t primitive_value_; - std::vector stl_vector_; -}; -``` - -As you can see, members in LCP are followed by a type. For primitive types, a -Lisp keyword is used. E.g. `:int64_t`, `:bool`, etc. Other types, like STL -containers use a valid C++ string to specify type. - -C++ supports nesting types inside a class. You can do the same in LCP inside -any of the scoped additions. - -For example: - -```lisp -(lcp:define-class my-class () - ((member "NestedType") - (value "NestedEnum")) - (:private - (lcp:define-enum nested-enum (first-value second-value)) - - (lcp:define-class nested-type () - ((member :int64_t))) - - #>cpp - // Some other C++ code - cpp<#)) -``` - -The above should produce expected results. - -You can add a base classes after the class name. The name should be a Lisp -symbol for base classes defined through `lcp:define-class`, so that LCP tracks -the inheritance. Otherwise, it should be a string. - -For example: - -```lisp -(lcp:define-class derived (my-class "UnknownInterface") - ()) -``` - -Will generate: - -```cpp -class Derived : public MyClass, public UnknownInterface { -}; -``` - -Similarly, you can specify template parameters. Instead of giving just a name -to `define-class`, you give a list where the first element is the name of the -class, while others name the template parameters. - -```lisp -(lcp:define-class (my-map t-key t-value) () - ((underlying-map "std::unordered_map"))) -``` - -The above will generate: - -```cpp -template -class MyMap { - private: - std::unordered_map underlying_map_; -}; -``` - -Other than tweaking the class definition, you can also do additional -configuration of members. The following options are supported. - - * `:initval` -- sets the initial value of a member - * `:reader` -- generates a public getter - * `:scope` -- set the scope of a member, one of `:public`, `:private` or - `:protected` - * `:documentation` -- Doxygen documentation of a member - * various serialization options which are explained later - -For example: - -```lisp -(lcp:define-class my-class () - ((member "std::vector" :scope :protected :initval "1, 2, 3" :reader t - :documentation "Member documentation"))) -``` - -Will generate: - -```cpp -class MyClass { - public: - const auto &member() { return member_; } - - protected: - /// Member documentation - std::vector member_{1, 2, 3}; -}; -``` - -#### TypeInfo - -Defining a class or struct will also generate appropriate `utils::TypeInfo` -instance. This instance will be stored as the `static const` member named -`kType` and the class will also have a `GetTypeInfo` public member function, -for getting `TypeInfo` on an instance. If a class is part of an inheritance -hierarchy, then the `GetTypeInfo` will be `virtual`. - -This mechanism is a portable solution for Run Time Type Information in C++. -Take a look at the documentation of `TypeInfo` in the codebase for additional -information on what is provided with it. - -For example, defining a derived class: - -```lisp -(lcp:define-class derived-class (base-class) - ((member :int64_t))) -``` - -Will generate: - -```cpp -class DerivedClass : public BaseClass { - public: - static const utils::TypeInfo kType; - const utils::TypeInfo &GetTypeInfo() const override { return kType; } - - private: - int64_t member_; -}; -``` - -The generated `TypeInfo` object does not support multiple inheritance, and LCP -will report an error if it tries to generate such information. Additionally, -generating type information for template classes is not supported. In such a -case you will need to write your own description by hand. There are also cases -where you may want to tune the generation of type information. These cases -are described below. - -Sometimes, you may want to avoid having a virtual or overridden call by -treating a derived type as if it is a base type for the purposes of -`TypeInfo`. This can be done by passing the `:type-info :base t` class option. -NOTE: This will potentially break some `TypeInfo` related functions, like -`IsSubtype`. Therefore, this should only be used if the base type does not -have a correctly set `TypeInfo` interface. - -The same example, but with the option enabled: - -```lisp -(lcp:define-class derived-class (base-class) - ((member :int64_t)) - (:type-info :base t)) -``` - -Will generate: - -```cpp -class DerivedClass : public BaseClass { - public: - static const utils::TypeInfo kType; - // Has no override because we pretend this is "base". - // Is not virtual because LCP didn't detect any class inheriting this one. - const utils::TypeInfo &GetTypeInfo() const { return kType; } - - private: - int64_t member_; -}; -``` - -Similarly to `:base t`, when you have multiple inheritance but only want to -track the primary class as super class, you can use the -`:ignore-other-base-classes t` option. NOTE: Similarly to the `:base t` -option, this should only be used if you know it won't break the use cases of -`TypeInfo` for this class hierarchy. - -Again, the same example, but with `:ignore-other-base-classes t`: - -```lisp -(lcp:define-class derived-class (base-class other-base-class) - ((member :int64_t)) - (:type-info :ignore-other-base-classes t)) -``` - -Will generate: - -```cpp -class DerivedClass : public BaseClass, public OtherBaseClass { - public: - static const utils::TypeInfo kType; - // Overrides, because it's expected BaseClass has one. TypeInfo is set to - // track only BaseClass as super class. - const utils::TypeInfo &GetTypeInfo() const override { return kType; } - - private: - int64_t member_; -}; -``` - -### Defining an RPC - -In our codebase, we have implemented remote procedure calls. These are used -for communication between Memgraph instances in a distributed system. Each RPC -is registered by its type and requires serializable data structures. Writing -RPC compliant structure requires a lot of boilerplate. To ease the pain of -defining a new RPC we have a macro, `lcp:define-rpc`. - -Definition consists of 2 parts: request and response. You can specify members -of each part. Member definition is the same as in `lcp:define-class`. - -For example: - -```lisp -(lcp:define-rpc query-result - (:request - ((tx-id "tx::TransactionId") - (query-id :int64_t))) - (:response - ((values "std::vector")))) -``` - -The above will generate relatively large amount of C++ code, which is omitted -here as the details aren't important for understanding the use. Examining the -generated code is left as an exercise for the reader. - -The important detail is that in C++ you will have a `QueryResultRpc` -structure, which is used to register the behaviour of an RPC server. You need -to perform the registration manually. For example: - -```cpp -// somewhere in code you have a server instance -rpc_server.Register( - [](const auto &req_reader, auto *res_builder) { - QueryResultReq request; - Load(&request, req_reader); - // process the request and send the response - QueryResultRes response(values_for_response); - Save(response, res_builder); - }); - - -// somewhere else you have a client which sends the RPC -tx::TransactionId tx_id = ... -int64_t query_id = ... -auto response = rpc_client.template Call(tx_id, query_id); -if (response) { - const auto &values = response->getValues(); - // do something with values -} -``` - -RPC structures use Cap'n Proto for serialization. The above variables -`req_reader` and `res_builder` are used to access Cap'n Proto structures. -Obviously, the LCP will generate the Cap'n Proto schema alongside the C++ -code for serialization. - - -### Cap'n Proto Serialization {#capnp-serial} - -Primary purpose of LCP was to make serialization of types easier. Our -serialization library of choice for C++ is Cap'n Proto. LCP provides -generation and tuning of its serialization code. Previously, LCP supported -Boost.Serialization, but it was removed. - -To specify a class or structure for serialization, you may pass a -`:serialize (:capnp)` option when defining such type. (Note that -`lcp:define-enum` takes `:serialize` without any arguments). - -For example: - -```lisp -(lcp:define-struct my-struct () - ((member :int64_t)) - (:serialize (:capnp))) -``` - -`:serialize` option will generate a Cap'n Proto schema of the class and store -it in the `.capnp` file. C++ code will be generated for saving and loading -members: - -```cpp -// Top level functions -void Save(const MyStruct &self, capnp::MyStruct::Builder *builder); -void Load(MyStruct *self, const capnp::MyStruct::Reader &reader); -``` - -Since we use top level functions, the class needs to have some sort of public -access to its members. - -The schema file will be namespaced in `capnp`. To change add a prefix -namespace use `lcp:capnp-namespace` function. For example, if we use -`(lcp:capnp-namespace "my_namespace")` then the reader and builder would be in -`my_namespace::capnp`. - -Serializing a class hierarchy is also supported. The most basic case with -single inheritance works out of the box. Handling other cases is explained in -later sections. - -For example: - -```lisp -(lcp:define-class base () - ((base-member "std::vector" :scope :public)) - (:serialize (:capnp))) - -(lcp:define-class derived (base) - ((derived-member :bool :scope :public)) - (:serialize (:capnp))) -``` - -Note that all classes need to have the `:serialize` option set. Signatures of -`Save` and `Load` functions are changed to accept reader and builder to the -base class. The `Load` function now takes a `std::unique_ptr *` which is -used to take ownership of a concrete type. This approach transfers the -responsibility of type allocation and construction from the user of `Load` to -`Load` itself. - -```cpp -void Save(const Derived &self, capnp::Base::Builder *builder); -void Load(std::unique_ptr *self, const capnp::Base::Reader &reader); -``` - -#### Multiple Inheritance - -Cap'n Proto does not support any form of inheritance, instead we are -handling it manually. Single inheritance was relatively easy to add to Cap'n -Proto, we simply enumerate all derived types inside the union of a base type. - -Multiple inheritance is a different beast and as such is not directly -supported. - -One way to use multiple inheritance is only to implement the interface of pure -virtual classes without any members (i.e. interface classes). In such a case, -you do not want to serialize any other base class except the primary one. To -let LCP know that is the case, use `:ignore-other-base-classes t`. LCP will -only try to serialize the base class that is the first (leftmost) in the list -of super classes. - -```lisp -(lcp:define-class derived (primary-base some-interface other-interface) - ... - (:serialize (:capnp :ignore-other-base-classes t))) -``` - -Another form of multiple inheritance is reusing some common code. In -actuality, this is a very bad code practice and should be replaced with -composition. If it would take too long to fix such code to use composition -proper, we can tell LCP to treat such inheritance as if they are indeed -composed. This is done via `:inherit-compose` option. - -For example: - -```lisp -(lcp:define-class derived (first-base second-base) - ... - (:serialize (:capnp :inherit-compose '(second-base)))) -``` - -With `:inherit-compose` you can pass a list of parent classes which should be -encoded as composition inside the Cap'n Proto schema. LCP will complain if -there is multiple inheritance but you didn't specify `:inherit-compose`. - -The downside of this approach is that `Save` and `Load` will work only on -`FirstBase`. Serializing a pointer to `SecondBase` would be incorrect. - -#### Inheriting C++ Class Outside of LCP - -Classes defined outside of `lcp:define-class` are not visible to LCP and LCP -will not be able to generate correct serialization code. - -The cases so far have been only with classes that are pure interface and need -no serialization code. This is signaled to LCP by passing the option `:base t` -to `:capnp`. LCP will treat such classes as actually being the base class of a -hierarchy. - -For example: - -```lisp -(lcp:define-class my-class ("utils::TotalOrdering") - (...) - (:serialize (:capnp :base t))) - -(lcp:define-class derived (my-class) - (...) - (:serialize (:capnp))) -``` - -Only the base class for serialization has the `:base t` option set. Derived -classes are defined as usual. This relies on the fact that we do not expect -anyone to have a pointer to `utils::TotalOrdering` and use it for -serialization and deserialization. - -#### Template Classes - -Currently, LCP supports the most primitive form of serializing templated -classes. The template arguments must be provided to specify an explicit -instantiation. Cap'n Proto does support generics, so we may want to upgrade -LCP to use them in the future. - -To specify template arguments, pass a `:type-args` option. For example: - -```lisp -(lcp:define-class (my-container t-value) () - (...) - (:serialize (:capnp :type-args '(my-class)))) -``` - -The above will support serialization of `MyContainer` type. - -The syntax will work even if our templated class inherits from non-templated -classes. All other cases of inheritance with templates are forbidden in LCP -serialization. - -#### Cap'n Proto Schemas and Type Conversions - -You can import other serialization schemas by using `lcp:capnp-import` -function. It expects a name for the import and the path to the schema file. - -For example, to import everything from `utils/serialization.capnp` under the -name `Utils`, you can do the following: - -```lisp -(lcp:capnp-import 'utils "/utils/serialization.capnp") -``` - -To use those types, you need to register a conversion from C++ type to schema -type. There are two options, registering a whole file conversion with -`lcp:capnp-type-conversion` or converting a specific class member. - -For example, you have a class with member of type `Bound` and there is a -schema for it also named `Bound` inside the imported schema. - -You can use `lcp:capnp-type-conversion` like so: - -```lisp -(lcp:capnp-type-conversion "Bound" "Utils.Bound") - -(lcp:define-class my-class () - ((my-bound "Bound"))) -``` - -Specifying only a member conversion can be done with `:capnp-type` member -option: - -```lisp -(lcp:define-class my-class () - ((my-bound "Bound" :capnp-type "Utils.Bound"))) -``` - -#### Custom Save and Load Hooks - -Sometimes the default serialization is not adequate and you may wish to -provide your own serialization code. For those reasons, LCP provides -`:capnp-save`, `:capnp-load` and `:capnp-init` options on each class member. - -The simplest is `:capnp-init` which when set to `nil` will not generate an -`init` call on a builder. Cap'n Proto requires that compound types are -initialized before beginning to serialize its members. `:capnp-init` allows you -to delay the initialization to your custom save code. You rarely want to set -`:capnp-init nil`. - -Custom save code is added as a value of `:capnp-save`. It should be a function -which takes 3 arguments. - - 1. Name of builder variable. - 2. Name of the class (or struct) member. - 3. Name of the member in Cap'n Proto schema. - -The result of the function needs to be a C++ code block. - -You will rarely need to use the 3rd argument, so it should be ignored in most -cases. It is usually needed when you set `:capnp-init nil`, so that you can -correctly initialize the builder. - -Similarly, `:capnp-load` expects a function taking a reader, C++ member and -Cap'n Proto member, then returns a C++ block. - -Example: - -```lisp -(lcp:define-class my-class () - ((my-member "ComplexType" - :capnp-init nil - :capnp-save (lambda (builder member capnp-name) - #>cpp - auto data = ${member}.GetSaveData(); - auto my_builder = ${builder}.init${capnp-name}(); - my_builder.setData(data); - cpp<#) - :capnp-load (lambda (reader member capnp-name) - (declare (ignore capnp-name)) - #>cpp - auto data = ${reader}.getData(); - ${member}.LoadFromData(data); - cpp<#))) - (:serialize (:capnp))) -``` - -With custom serialization code, you may want to get additional details through -extra arguments to `Save` and `Load` functions. This is described in the next -section. - -There are also cases where you always need custom serialization code. LCP -provides helper functions for abstracting some common details. These functions -are listed further down in this document. - -#### Arguments for Save and Load - -Default arguments for `Save` and `Load` function are Cap'n Proto builder and -reader, respectively. In some cases you may wish to send additional arguments. -This is most commonly needed when tracking `shared_ptr` serialization, to -avoid serializing the same pointer multiple times. - -Additional arguments are specified by passing `:save-args` and `:load-args`. -You can specify either of them, but in most cases you want both. - -For example: - -```lisp -;; Class for tracking details during save -(lcp:define-class save-helper () - (...)) - -;; Class for tracking details during load -(lcp:define-class load-helper () - (...)) - -(lcp:define-class my-class () - ((member "std::shared_ptr" - :capnp-save ;; custom save - :capnp-load ;; custom load - )) - (:serialize (:capnp - :save-args '((save-helper "SaveHelper *")) - :load-args '((load-helper "LoadHelper *"))))) -``` - -The custom serialization code will now have access to `save_helper` and -`load_helper` variables in C++. You can add more arguments by expanding the -list of pairs, e.g. - -```lisp -:save-args '((first-helper "SomeType *") (second-helper "OtherType *") ...) -``` - -#### Custom Serialization Helper Functions - -##### Helper for `std::optional` - -When using `std::optional` with primitive C++ types or custom types known to -LCP, you do not need to use any helper. In the example below, things should be -serialized as expected: - -```lisp -(lcp:define-class my-class-with-primitive-optional () - ((primitive-optional "std::experimental::optional")) - (:serialize (:capnp))) - -(lcp:define-class my-class-with-known-type-optional () - ((known-type-optional "std::experimental::optional")) - (:serialize (:capnp))) -``` - -In cases when the value contained in `std::optional` needs custom -serialization code you may use `lcp:capnp-save-optional` and -`lcp:capnp-load-optional`. - -Both functions expect 3 arguments. - - 1. Cap'n Proto type in C++. - 2. C++ type of the value inside `std::optional`. - 3. Optional C++ lambda code. - -The lambda code is optional, because LCP will generate the default -serialization code which invokes `Save` and `Load` function on the value -stored inside the optional. Since most of the serialized classes follow the -convention, you will rarely need to provide this 3rd argument. - -For example: - -```lisp -(lcp:define-class my-class () - ((member "std::experimental::optional" - :capnp-save (lcp:capnp-save-optional - "capnp::SomeType" "SomeType" - "[](auto *builder, const auto &val) { ... }") - :capnp-load (lcp:capnp-load-optional - "capnp:::SomeType" "SomeType" - "[](const auto &reader) { ... return loaded_val; }")))) -``` - -##### Helper for `std::vector` - -For custom serialization of vector elements, you may use -`lcp:capnp-save-vector` and `lcp:capnp-load-vector`. They function exactly the -same as helpers for `std::optional`. - -##### Helper for enumerations - -If the enumeration is defined via `lcp:define-enum`, the default LCP -serialization should generate the correct code. - -However, if LCP cannot infer the serialization code, you can use helper -functions `lcp:capnp-save-enum` and `lcp:capnp-load-enum`. Both functions -require 3 arguments. - - 1. C++ type of equivalent Cap'n Proto enum. - 2. Original C++ enum type. - 3. List of enumeration values. - -Example: - -```lisp -(lcp:define-class my-class () - ((enum-value "SomeEnum" - :capnp-init nil ;; must be set to nil - :capnp-save (lcp:capnp-save-enum - "capnp::SomeEnum" "SomeEnum" - '(first-value second-value)) - :capnp-load (lcp:capnp-load-enum - "capnp::SomeEnum" "SomeEnum" - '(first-value second-value))))) -``` - -### SaveLoadKit Serialization {#slk-serial} - -LCP supports generating serialization code for use with our own simple -serialization framework, SaveLoadKit (SLK). - -To specify a class or structure for serialization, pass a `:serialize (:slk)` -class option. For example: - -```lisp -(lcp:define-struct my-struct () - ((member :int64_t)) - (:serialize (:slk))) -``` - -The above will generate C++ functions for saving and loading all members of -the defined type. The generated code is put inside the `slk` namespace. For -the above example, we would get the following declarations: - -```cpp -namespace slk { -void Save(const MyStruct &self, slk::Builder *builder); -void Load(MyStruct *self, slk::Reader *reader); -} -``` - -Since we use top level (i.e. non-member) functions, the class members need to -have public access. The primary reason why we use non-member functions is the -ability to have them decoupled from types. This in turn allows us to easily -compile the code with and without serialization. The obvious downside is the -requirement of public access which could potentially allow for erroneous use -of classes and its members. Therefore, the recommended way to use -serialization is with plain old data types. The programmer needs be aware of -that and use POD as an immutable type as much as possible. This -recommendation of using POD types will also help minimize the complexity of -serialization code as well as minimize required features in LCP. - -Another requirement on serialized types is that they need to be default -constructible. This keeps the serialization implementation simple and uniform. -Each type is first default constructed, potentially on stack memory. Then the -`slk::Load` function is invoked with the pointer to that instance. We could -add support for having a pointer to an uninitialized memory and perform the -construct in `slk::Load` to allow types which aren't default constructible. -At the moment, implementing this support would needlessly complicate our code -where most of the types can be and are default constructible. - -#### Single Inheritance - -The first and most common step out of the POD zone is having classes with -inheritance. LCP supports serializing classes with single inheritance. - -A minor complication appears when loading a pointer to a base class. When we -have a pointer to a base class, serializing it may save the data of some -concrete, derived type. Loading the pointer back will need to determine which -type was actually serialized. When we know the concrete type, we need to -construct it and load it. Finally, we can return a base pointer to that. For -this reason, we generate 2 loading functions: regular `Load` and -`ConstructAndLoad`. The latter function is used to do the whole process of -determining the type, constructing it and invoking regular `Load`. Since we -cannot know the type of the serialized pointer upfront, we cannot allocate the -exact required memory on the stack. For that reason, `ConstructAndLoad` will -perform a heap allocation for you. Obviously, this could be a performance -issue. In cases when we know the exact concrete type, then we can use the -regular `Load` which expects the pointer to that type. If you are using `Load` -instead of `ConstructAndLoad`, read the next paragraph carefully! - -Determining which type was serialized works by storing the `id` of -`utils::TypeInfo` when saving a class which is anywhere in the inheritance -hierarchy. This is the *first* thing the invocation to `Save` does. Later, -when we call `ConstructAndLoad` it will read that type `id` and dispatch on it -to construct the instance of that type and call the appropriate `Load` -function. Beware when invoking `Load` of polymorphic types yourself! You -*need* to read the type `id` yourself *first* and then invoke the `Load` -function. Things will not work correctly if you forget to do that, because -`Load` expects to read the serialized data members and not the type -information. - -For example: - -```lisp -(lcp:define-class base () - ... - (:serialize (:slk))) - -(lcp:define-class derived (base) - ... - (:serialize (:slk))) -``` - -We get the following declarations generated: - -```cpp -namespace slk { -// Save will correctly forward to derived class using `dynamic_cast`! -void Save(const Base &self, slk::Builder *builder); -// Load only the Base instance, does *not* forward! -void Load(Base *self, slk::Reader *reader); -// Construct the concrete type (could be Base or any derived) and call the -// correct Load. Raises `slk::SlkDecodeException` if an unknown type is -// serialized. -void ConstructAndLoad(std::unique_ptr *self, slk::Reader *reader); - -void Save(const Derived &self, slk::Builder *builder); -void Load(Derived *self, slk::Reader *reader); -// This will raise slk::SlkDecodeException, if something other than `Derived` -// was serialized. `Derived` does not have any subclassses. -void ConstructAndLoad(std::unique_ptr *self, slk::Reader *reader); -``` - -#### Multiple Inheritance - -Serializing classes with multiple inheritance is *not* supported! - -Usually, multiple inheritance is used to satisfy some interface which doesn't -carry data for serialization. In such cases, you can ignore the multiple -inheritance by specifying `:ignore-other-base-classes` option. For example: - -```lisp -(lcp:define-class derived (primary-base some-interface ...) - ... - (:serialize (:slk :ignore-other-base-classes t))) -``` - -The above will produce serialization code as if `derived` is inheriting *only* -from `primary-base`. - -#### Templated Types - -Serializing templated types is *not* supported! - -You may still write your own serialization code in C++, but LCP will not -generate it for you. - -#### Custom Save and Load Hooks - -In cases when default serialization is not adequate, you may wish to provide -your own serialization code. LCP provides `:slk-save` and `:slk-load` options -for each member. - -These hooks for custom serialization expect a function with a single argument, -`member`, representing the member currently being serialized. This allows to -have a more generic function which works with any member of some type. The -return value of the function needs to be C++ code. The generated code may -expect to have `self` and `builder` variables in scope, just like they are -found in the generated `Save` and `Load` declarations. - -For example, one of the most common use cases is saving and loading -a `std::shared_ptr`. You need to provide an argument which is used to track -which pointers were already (de)serialized. Let's take a look how this could -be done in LCP. - -```lisp -(lcp:define-struct my-struct () - ((some-ptr "std::shared_ptr" - :slk-save (lambda (member) - #>cpp - std::vector already_saved; - slk::Save(self.${member}, builder, &already_saved); - cpp<#) - :slk-load (lambda (member) - #>cpp - std::vector> already_loaded; - slk::Load(&self->${member}, reader, &already_loaded); - cpp<#))) - (:serialize (:slk))) -``` - -The above use is very artificial, because we usually have multiple shared -pointers across different members. In such cases we would like to share the -tracking data. One way to do that is explained in the next section. - -#### Additional Arguments to Generated Save and Load - -As you may have noticed, primary arguments for `Save` and `Load` are the type -instance and a `slk::Builder` or a `slk::Reader`. In some cases we would like -to accept additional arguments to help us with the serialization process. -Let's see how this is done in LCP using the `:save-args` and `:load-args` -options for `:slk` serialization. - -Both `:save-args` and `:load-args` options expect a list of pairs. Each pair -designates one argument. The first element of the pair is the argument name -and the second is the C++ type of that argument. - -As mentioned in the previous section, one of the most common cases where -default serialization doesn't cut it is when we have a `std::shared_ptr`. -Here, we would like to track already serialized pointers. Instead of having -some kind of a global variable, we could pass the tracking data as an -additional argument. Let's take the example from the previous section, and -have it take tracking data as an argument to `Save` and `Load` of `my-struct` -type. - -```lisp -(lcp:define-struct my-struct () - ((some-ptr "std::shared_ptr" - :slk-save (lambda (member) - #>cpp - slk::Save(self.${member}, builder, already_saved); - cpp<#) - :slk-load (lambda (member) - #>cpp - slk::Load(&self->${member}, reader, already_loaded); - cpp<#))) - (:serialize (:slk :save-args '((already-saved "std::vector *")) - :load-args '((already-loaded "std::vector> *"))))) -``` - -The generated declarations now look like the following: - -```cpp -void Save(const MyStruct &self, slk::Builder *builder, - std::vector *already_saved); -void Load(MyStruct *self, slk::Builder *builder, - std::vector> *already_loaded); -``` - -This can now be handy when serializing multiple instances of `my-struct`. For -example: - -```lisp -(lcp:define-struct my-array-of-struct () - ((structs "std::vector" - :slk-save (lambda (member) - #>cpp - slk::Save(self.${member}.size(), builder); - std::vector already_saved; - for (const auto &my_struct : structs) - slk::Save(my_struct, builder, &already_saved); - cpp<#) - :slk-load (lambda (member) - #>cpp - size_t size = 0; - slk::Load(&size, reader); - self->${member}.resize(size); - std::vector> already_loaded; - for (size_t i = 0; i < size; ++i) - slk::Load(&self->${member}[i], reader, &already_loaded); - cpp<#))) - (:serialize (:slk))) -``` - -### Object Cloning - -LCP supports automatic generation of cloning (deep copy) code for user-defined -classes. - -A textbook example of an object that would require a deep copy functionality is -a tree structure. The following class represents a node in the binary tree, -carrying an integer value and having pointers to its two children: - -```lisp -(lcp:define-class node () - ((value :int32_t) - (left "std::unique_ptr") - (right "std::unique_ptr")) - (:clone :return-type (lambda (typename) - #>cpp - std::unique_ptr<${typename}> - cpp<#) - :init-object (lambda (var typename) - #>cpp - auto ${var} = std::make_unique<${typename}>(); - cpp<#))) -``` - -The above will generate the following C++ class with a `Clone` function that -can be used for making a deep copy of the binary tree structure: - -```cpp -class Node { - public: - std::unique_ptr Clone() const { - auto object = std::make_unique(); - object->value_ = value_; - object->left_ = left_ ? left_->Clone() : nullptr; - object->right_ = right_ ? right_->Clone() : nullptr; - return object; - } - - private: - int32_t value_; - std::unique_ptr left_; - std::unique_ptr right_; -}; -``` - -To specify that a class is deep copyable, `:clone` class option must be passed. -We have already seen two options that `:clone` accepts: `:return-type` and -`:init-object`. - -`:return-type` expects a function that takes a single argument which is the C++ -type name of the class and produces C++ code, which is a valid C++ type -declaration. Here we used it to specify that `Clone` function should return a -`std::unique_ptr` to the newly created `Node` to override the default behavior. -When `:return-type` option is not provided and class `T` is a member of an -inheritance hierarchy, `Clone` will return `std::unique_ptr`, where -`Base` is the root of that hierarchy. If `T` is not a member of inheritance -hierarchy, `Clone` will return `T` by default. - -`:init-object` expects a function that takes two arguments, first is a variable -name, and the second one is the C++ type name of the class. It must produce C++ -code that initializes an object with the given name of the same type that -`Clone` function returns. Here we had to use it since we are overriding the -default return value of `Clone`. Unless `:init-object` argument is provided, an -object of type `T` will be instantiated with `auto object = -std::make_unique();` if `T` is a member of inheritance hierarchy, and `T -object;` if it is not. As you can see, deep copyable objects must be default -constructible. - - -#### Single Inheritance - -LCP supports deep copying of classes with single inheritance. The root class -will have a virtual `Clone` function that child classes will override. For -example: - -```lisp -(lcp:define-class base () - ((member :int32_t)) - (:clone)) - -(lcp:define-class derived (base) - ((another-member :int32_t)) - (:clone)) -``` - -We get the following code: - -```cpp -class Base { - public: - virtual std::unique_ptr Clone() const { - auto object = std::make_unique(); - object->member_ = member_; - return object; - } - - private: - int32_t member_; -}; - -class Derived : public Base { - public: - std::unique_ptr Clone() const override { - auto object = std::make_unique(); - object->member_ = member_; - object->another_member_ = another_member_; - return object; - } - - private: - int32_t another_member_; -}; -``` - -Notice that the `Clone` function of derived class also returns -`std::unique_ptr`, because C++ doesn't support return type covariance -with smart pointers. - -#### Multiple Inheritance - -Deep copying of classes with multiple inheritance is *not* supported! - -Usually, multiple inheritance is used to satisfy some interface which doesn't -carry data. In such cases, you can ignore the multiple inheritance by -specifying `:ignore-other-base-classes` option. For example: - -```lisp -(lcp:define-class derived (primary-base some-interface ...) - ... - (:clone :ignore-other-base-classes t)) -``` - -The above will produce deep copying code as if `derived` is inheriting *only* -from `primary-base`. - -#### Templated Types - -Deep copying of templated types is *not* supported! - -#### Custom Clone Hooks - -In cases when default deep copying code is not adequate, you may wish to -provide your own. LCP provides `:clone` option that can be specified for each -member. - -These hooks for custom copying expect a function with two arguments, `source` -and `dest`, representing the member location in the cloned struct and member -location in the new struct. This allows to have a more generic function which -works with any member of some type. The return value of the function needs to -be C++ code. - -It is also possible to specify that a member is cloned by copying by passing -`:copy` instead of a function as an argument to `:clone`. - -```lisp -(lcp:define-class my-class () - ((callback "std::function" - :clone :copy) - (widget "Widget" - :clone (lambda (source dest) - #>cpp - ${dest} = WidgetFactory::Create(${source}.type()); - cpp<#))) - (:clone)) -``` - -#### Additional Arguments to Generated Clone Function - -By default, `Clone` function takes no argument. In some cases we would like to -accept additional arguments necessary to create a deep copy. Let's see how this -is done in LCP using the `:args` option. - -`:args` expects a list of pairs. Each pair designates one argument. The first -element of pair is the argument name and the second is the C++ type of that -argument. - -One case where we want to pass additional arguments to `Clone` is when there is -another object that owns all objects being cloned. For example, `AstStorage` -owns all Memgraph AST nodes. For that reason, `Clone` function of all AST node -types takes an `AstStorage \*` argument. Here's a snippet from the actual AST -code: - -```lisp -(lcp:define-class tree () - ((uid :int32_t)) - (:abstractp t) - ... - (:clone :return-type (lambda (typename) - (format nil "~A*" typename)) - :args '((storage "AstStorage *")) - :init-object (lambda (var typename) - (format nil "~A* ~A = storage->Create<~A>();" - typename var typename)))) - -(lcp:define-class expression (tree) - () - (:abstractp t) - ... - (:clone)) - -(lcp:define-class where (tree) - ((expression "Expression *" :initval "nullptr" :scope :public)) - (:clone)) -``` - -`:args` option is only passed to the root class in inheritance hierarchy. By -default, the same extra arguments will be passed to all class members that are -cloned using `Clone` metehod. The generated code is: - -```cpp -class Tree { - public: - virtual Tree *Clone(AstStorage *storage) const = 0; - private: - int32_t uid_; -}; - -class Expression : public Tree { - public: - Expression *Clone(AstStorage *storage) const override = 0; -}; - -class Where : public Tree { - public: - Expression *expression_{nullptr}; - - Where *Clone(AstStorage *storage) const override { - Where *object = storage->Create(); - object->uid_ = uid_; - object->expression_ = expression_ ? expression_->Clone(storage) : nullptr; - return object; - } -}; -``` diff --git a/docs/dev/other-code-conventions.md b/docs/dev/other-code-conventions.md deleted file mode 100644 index 21d8a6f0a..000000000 --- a/docs/dev/other-code-conventions.md +++ /dev/null @@ -1,15 +0,0 @@ -# Other Code Conventions - -While we are mainly programming in C++, we do use other programming languages -when appropriate. This chapter describes conventions for such code. - -## Python - -Code written in Python should adhere to -[PEP 8](https://www.python.org/dev/peps/pep-0008/). You should run `flake8` on -your code to automatically check compliance. - -## Common Lisp - -Code written in Common Lisp should adhere to -[Google Common Lisp Style](https://google.github.io/styleguide/lispguide.xml). diff --git a/docs/dev/query/.gitignore b/docs/dev/query/.gitignore deleted file mode 100644 index 5ccff1a6b..000000000 --- a/docs/dev/query/.gitignore +++ /dev/null @@ -1 +0,0 @@ -html/ diff --git a/docs/dev/query/contents.md b/docs/dev/query/contents.md deleted file mode 100644 index ad5a8cb0b..000000000 --- a/docs/dev/query/contents.md +++ /dev/null @@ -1,34 +0,0 @@ -# Query Parsing, Planning and Execution - -This part of the documentation deals with query execution. - -Memgraph currently supports only query interpretation. Each new query is -parsed, analysed and translated into a sequence of operations which are then -executed on the main database storage. Query execution is organized into the -following phases: - - 1. [Lexical Analysis (Tokenization)](parsing.md) - 2. [Syntactic Analysis (Parsing)](parsing.md) - 3. [Semantic Analysis and Symbol Generation](semantic.md) - 4. [Logical Planning](planning.md) - 5. [Logical Plan Execution](execution.md) - -The main entry point is `Interpreter::operator()`, which takes a query text -string and produces a `Results` object. To instantiate the object, -`Interpreter` needs to perform the above steps from 1 to 4. If any of the -steps fail, a `QueryException` is thrown. The complete `LogicalPlan` is -wrapped into a `CachedPlan` and stored for reuse. This way we can skip the -whole process of analysing a query if it appears to be the same as before. - -When we have valid plan, the client code can invoke `Results::PullAll` with a -stream object. The `Results` instance will then execute the plan and fill the -stream with the obtained results. - -Since we want to optionally run Memgraph as a distributed database, we have -hooks for creating a different plan of logical operators. -`DistributedInterpreter` inherits from `Interpreter` and overrides -`MakeLogicalPlan` method. This method needs to return a concrete instance of -`LogicalPlan`, and in case of distributed database that will be -`DistributedLogicalPlan`. - -![Intepreter Class Diagram](interpreter-class.png) diff --git a/docs/dev/query/execution.md b/docs/dev/query/execution.md deleted file mode 100644 index 675bf5010..000000000 --- a/docs/dev/query/execution.md +++ /dev/null @@ -1,373 +0,0 @@ -# Logical Plan Execution - -We implement classical iterator style operators. Logical operators define -operations on database. They encapsulate the following info: what the input is -(another `LogicalOperator`), what to do with the data, and how to do it. - -Currently logical operators can have zero or more input operations, and thus a -`LogicalOperator` tree is formed. Most `LogicalOperator` types have only one -input, so we are mostly working with chains instead of full fledged trees. -You can find information on each operator in `src/query/plan/operator.lcp`. - -## Cursor - -Logical operators do not perform database work themselves. Instead they create -`Cursor` objects that do the actual work, based on the info in the operator. -Cursors expose a `Pull` method that gets called by the cursor's consumer. The -consumer keeps pulling as long as the `Pull` returns `true` (indicating it -successfully performed some work and might be eligible for another `Pull`). -Most cursors will call the `Pull` function of their input provided cursor, so -typically a cursor chain is created that is analogue to the logical operator -chain it's created from. - -## Frame - -The `Frame` object contains all the data of the current `Pull` chain. It -serves for communicating data between cursors. - -For example, in a `MATCH (n) RETURN n` query the `ScanAllCursor` places a -vertex on the `Frame` for each `Pull`. It places it on the place reserved for -the `n` symbol. Then the `ProduceCursor` can take that same value from the -`Frame` because it knows the appropriate symbol. `Frame` positions are indexed -by `Symbol` objects. - -## ExpressionEvaluator - -Expressions results are not placed on the `Frame` since they do not need to be -communicated between different `Cursors`. Instead, expressions are evaluated -using an instance of `ExpressionEvaluator`. Since generally speaking an -expression can be defined by a tree of subexpressions, the -`ExpressionEvaluator` is implemented as a tree visitor. There is a performance -sub-optimality here because a stack is used to communicate intermediary -expression results between elements of the tree. This is one of the reasons -why it's planned to use `Frame` for intermediary expression results as well. -The other reason is that it might facilitate compilation later on. - -## Cypher Execution Semantics - -Cypher query execution has *mostly* well-defined semantics. Some are -explicitly defined by openCypher and its TCK, while others are implicitly -defined by Neo4j's implementation of Cypher that we want to be generally -compatible with. - -These semantics can in short be described as follows: a Cypher query consists -of multiple clauses some of which modify it. Generally, every clause in the -query, when reading it left to right, operates on a consistent state of the -property graph, untouched by subsequent clauses. This means that a `MATCH` -clause in the beginning operates on a graph-state in which modifications by -the subsequent `SET` are not visible. - -The stated semantics feel very natural to the end-user, and Neo seems to -implement them well. For Memgraph the situation is complex because -`LogicalOperator` execution (through a `Cursor`) happens one `Pull` at a time -(generally meaning all the query clauses get executed for every top-level -`Pull`). This is not inherently consistent with Cypher semantics because a -`SET` clause can modify data, and the `MATCH` clause that precedes it might -see the modification in a subsequent `Pull`. Also, the `RETURN` clause might -want to stream results to the user before all `SET` clauses have been -executed, so the user might see some intermediate graph state. There are many -edge-cases that Memgraph does its best to avoid to stay true to Cypher -semantics, while at the same time using a high-performance streaming approach. -The edge-cases are enumerated in this document along with the implementation -details they imply. - -## Implementation Peculiarities - -### Once - -An operator that does nothing but whose `Cursor::Pull` returns `true` on the -first `Pull` and `false` on subsequent ones. This operator is used when -another operator has an optional input, because in Cypher a clause will -typically execute once for every input from the preceding clauses, or just -once if there was no preceding input. For example, consider the `CREATE` -clause. In the query `CREATE (n)` only one node is created, while in the query -`MATCH (n) CREATE (m)` a node is created for each existing node. Thus in our -`CreateNode` logical operator the input is either a `ScanAll` operator, or a -`Once` operator. - -### storage::View - -In the previous section, [Cypher Execution -Semantics](#cypher-execution-semantics), we mentioned how the preceding -clauses should not see changes made in subsequent ones. For that reason, some -operators take a `storage::View` enum value. This value determines which state of -the graph an operator sees. - -Consider the query `MATCH (n)--(m) WHERE n.x = 0 SET m.x = 1`. Naive streaming -could match a vertex `n` on the given criteria, expand to `m`, update it's -property, and in the next iteration consider the vertex previously matched to -`m` and skip it because it's newly set property value does not qualify. This -is not how Cypher works. To handle this issue properly, Memgraph designed the -`VertexAccessor` class that tracks two versions of data: one that was visible -before the current transaction+command, and the optional other that was -created in the current transaction+command. The `MATCH` clause will be planned -as `ScanAll` and `Expand` operations using `storage::View::OLD` value. This -will ensure modifications performed in the same query do not affect it. The -same applies to edges and the `EdgeAccessor` class. - -### Existing Record Detection - -It's possible that a pattern element has already been declared in the same -pattern, or a preceding pattern. For example `MATCH (n)--(m), (n)--(l)` or a -cycle-detection match `MATCH (n)-->(n) RETURN n`. Implementation-wise, -existing record detection just checks that the expanded record is equal to the -one already on the frame. - -### Why Not Use Separate Expansion Ops for Edges and Vertices? - -Expanding an edge and a vertex in separate ops is not feasible when matching a -cycle in bi-directional expansions. Consider the query `MATCH (n)--(n) RETURN -n`. Let's try to expand first the edge in one op, and vertex in the next. The -vertex expansion consumes the edge expansion input. It takes the expanded edge -from the frame. It needs to detect a cycle by comparing the vertex existing on -the frame with one of the edge vertices (`from` or `to`). But which one? It -doesn't know, and can't ensure correct cycle detection. - -### Data Visibility During and After SET - -In Cypher, setting values always works on the latest version of data (from -preceding or current clause). That means that within a `SET` clause all the -changes from previous clauses must be visible, as well as changes done by the -current `SET` clause. Also, if there is a clause after `SET` it must see *all* -the changes performed by the preceding `SET`. Both these things are best -illustrated with the following queries executed on an empty database: - - CREATE (n:A {x:0})-[:EdgeType]->(m:B {x:0}) - MATCH (n)--(m) SET m.x = n.x + 1 RETURN labels(n), n.x, labels(m), m.x - -This returns: - -+---------+---+---------+---+ -|labels(n)|n.x|labels(m)|m.x| -+:=======:+:=:+:=======:+:=:+ -|[A] |2 |[B] |1 | -+---------+---+---------+---+ -|[B] |1 |[A] |2 | -+---------+---+---------+---+ - -The obtained result implies the following operations: - - 1. In the first iteration set the value of the `B.x` to 1 - 2. In the second iteration the we observe `B.x` with the value of 1 and set - `A.x` to 2 - 3. In `RETURN` we see all the changes made in both iterations - -To implement the desired behavior Memgraph utilizes two techniques. First is -the already mentioned tracking of two versions of data in vertex accessors. -Using this approach ensures that the second iteration in the example query -sees the data modification performed by the preceding iteration. The second -technique is the `Accumulate` operation that accumulates all the iterations -from the preceding logical op before passing them to the next logical op. In -the example query, `Accumulate` ensures that the results returned to the user -reflect changes performed in all iterations of the query (naive streaming -could stream results at the end of first iteration producing inconsistent -results). Note that `Accumulate` is demanding regarding memory and slows down -query execution. For that reason it should be used only when necessary, for -example it does not have to be used in a query that has `MATCH` and `SET` but -no `RETURN`. - -### Neo4j Inconsistency on Multiple SET Clauses - -Considering the preceding example it could be expected that when a query has -multiple `SET` clauses all the changes from those preceding one are visible. -This is not the case in Neo4j's implementation. Consider the following queries -executed on an empty database: - - CREATE (n:A {x:0})-[:EdgeType]->(m:B {x:0}) - MATCH (n)--(m) SET n.x = n.x + 1 SET m.x = m.x * 2 - RETURN labels(n), n.x, labels(m), m.x - -This returns: - -+---------+---+---------+---+ -|labels(n)|n.x|labels(m)|m.x| -+:=======:+:=:+:=======:+:=:+ -|[A] |2 |[B] |1 | -+---------+---+---------+---+ -|[B] |1 |[A] |2 | -+---------+---+---------+---+ - -If all the iterations of the first `SET` clause were executed before executing -the second, all the resulting values would be 2. This not being the case, we -conclude that Neo4j does not use a barrier-like mechanism between `SET` -clauses. It is Memgraph's current vision that this is inconsistent and we -plan to reduce Neo4j compliance in favour of operation consistency. - -### Double Deletion - -It's possible to match the same graph element multiple times in a single query -and delete it. Neo supports this, and so do we. The relevant implementation -detail is in the `GraphDbAccessor` class, where the record deletion functions -reside, and not in the logical plan execution. It comes down to checking if a -record has already been deleted in the current transaction+command and not -attempting to do it again (results in a crash). - -### Set + Delete Edge-case - -It's legal for a query to combine `SET` and `DELETE` clauses. Consider the -following queries executed on an empty database: - - - CREATE ()-[:T]->() - MATCH (n)--(m) SET n.x = 42 DETACH DELETE m - -Due to the `MATCH` being undirected the second pull will attempt to set data -on a deleted vertex. This is not a legal operation in Memgraph storage -implementation. For that reason the logical operator for `SET` must check if -the record it's trying to set something on has been deleted by the current -transaction+command. If so, the modification is not executed. - -### Deletion Accumulation - -Sometimes it's necessary to accumulate deletions of all the matches before -attempting to execute them. Consider this the following. Start with an empty -database and execute queries: - - CREATE ()-[:T]->()-[:T]->() - MATCH (a)-[r1]-(b)-[r2]-(c) DELETE r1, b, c - -Note that the `DELETE` clause attempts to delete node `c`, but it does not -detach it by deleting edge `r2`. However, due to undirected edge in the -`MATCH`, both edges get pulled and deleted. - -Currently Memgraph does not support this behavior, Neo does. There are a few -ways that we could do this. - - * Accumulate on deletion (that sucks because we have to keep track of - everything that gets returned after the deletion). - * Maybe we could stream through the deletion op, but defer actual deletion - until plan-execution end. - * Ignore this because it's very edgy (this is the currently selected option). - -### Aggregation Without Input - -It is necessary to define what aggregation ops return when they receive no -input. Following is a table that shows what Neo4j's Cypher implementation and -SQL produce. - - -+-------------+------------------------+---------------------+---------------------+------------------+ -| \ | 1. Cypher, no group-by | 2. Cypher, group-by | 3. SQL, no group-by | 4. SQL, group-by | -+=============+:======================:+:===================:+:===================:+:================:+ -| Count(\*) | 0 | \ | 0 | \ | -+-------------+------------------------+---------------------+---------------------+------------------+ -| Count(prop) | 0 | \ | 0 | \ | -+-------------+------------------------+---------------------+---------------------+------------------+ -| Sum | 0 | \ | NULL | \ | -+-------------+------------------------+---------------------+---------------------+------------------+ -| Avg | NULL | \ | NULL | \ | -+-------------+------------------------+---------------------+---------------------+------------------+ -| Min | NULL | \ | NULL | \ | -+-------------+------------------------+---------------------+---------------------+------------------+ -| Max | NULL | \ | NULL | \ | -+-------------+------------------------+---------------------+---------------------+------------------+ -| Collect | [] | \ | N/A | N/A | -+-------------+------------------------+---------------------+---------------------+------------------+ - -Where: - - 1. `MATCH (n) RETURN (n.prop)` - 2. `MATCH (n) RETURN (n.prop), (n.prop2)` - 3. `SELECT (prop) FROM Table` - 4. `SELECT (prop), prop2 FROM Table GROUP BY prop2` - -Neo's Cypher implementation diverges from SQL only when performing `SUM`. -Memgraph implements SQL-like behavior. It is considered that `SUM` of -arbitrary elements should not be implicitly 0, especially in a property graph -without a strict schema (the property in question can contain values of -arbitrary types, or no values at all). - -### OrderBy - -The `OrderBy` logical operator sorts the results in the desired order. It -occurs in Cypher as part of a `WITH` or `RETURN` clause. Both the concept and -the implementation are straightforward. It's necessary for the logical op to -`Pull` everything from its input so it can be sorted. It's not necessary to -keep the whole `Frame` state of each input, it is sufficient to keep a list of -`TypedValues` on which the results will be sorted, and another list of values -that need to be remembered and recreated on the `Frame` when yielding. - -The sorting itself is made to reflect that of Neo's implementation which comes -down to these points. - - * `Null` comes last (as if it's greater than anything). - * Primitive types compare naturally, with no implicit casting except from - `int` to `double`. - * Complex types are not comparable. - * Every unsupported comparison results in an exception that gets propagated - to the end user. - -### Limit in Write Queries - -`Limit` can be used as part of a write query, in which case it will *not* -reduce the amount of performed updates. For example, consider a database that -has 10 vertices. The query `MATCH (n) SET n.x = 1 RETURN n LIMIT 3` will -result in all vertices having their property value changed, while returning -only the first to the client. This makes sense from the implementation -standpoint, because `Accumulate` is planned after `SetProperty` but before -`Produce` and `Limit` operations. Note that this behavior can be -non-deterministic in some queries, since it relies on the order of iteration -over nodes which is undefined when not explicitly specified. - -### Merge - -`MERGE` in Cypher attempts to match a pattern. If it already exists, it does -nothing and subsequent clauses like `RETURN` can use the matched pattern -elements. If the pattern can't match to any data, it creates it. For detailed -information see Neo4j's [merge -documentation.](https://neo4j.com/docs/developer-manual/current/cypher/clauses/merge/) - -An important thing about `MERGE` is visibility of modified data. `MERGE` takes -an input (typically a `MATCH`) and has two additional *phases*: the merging -part, and the subsequent set parts (`ON MATCH SET` and `ON CREATE SET`). -Analysis of Neo4j's behavior indicates that each of these three phases (input, -merge, set) does not see changes to the graph state done by subsequent phase. -The input phase does not see data created by the merge phase, nor the set -phase. This is consistent with what seems like the general Cypher philosophy -that query clause effects aren't visible in the preceding clauses. - -We define the `Merge` logical operator as a *routing* operator that uses three -logical operator branches. - - 1. The input from a preceding clause. - - For example in `MATCH (n), (m) MERGE (n)-[:T]-(m)`. This input is - optional because `MERGE` is allowed to be the first clause in a query. - - 2. The `merge_match` branch. - - This logical operator branch is `Pull`-ed from until exhausted for each - successful `Pull` from the input branch. - - 3. The `merge_create` branch. - - This branch is `Pull`ed when the `merge_match` branch does not match - anything (no successful `Pull`s) for an input `Pull`. It is `Pull`ed only - once in such a situation, since only one creation needs to occur for a - failed match. - -The `ON MATCH SET` and `ON CREATE SET` parts of the `MERGE` clause are -included in the `merge_match` and `merge_create` branches respectively. They -are placed on the end of their branches so that they execute only when those -branches succeed. - -Memgraph strives to be consistent with Neo in its `MERGE` implementation, -while at the same time keeping performance as good as possible. Consistency -with Neo w.r.t. graph state visibility is not trivial. Documentation for -`Expand` and `Set` describe how Memgraph keeps track of both the updated -version of an edge/vertex and the old one, as it was before the current -transaction+command. This technique is also used in `Merge`. The input -phase/branch of `Merge` always looks at the old data. The merge phase needs to -see the new data so it doesn't create more data then necessary. - -For example, consider the query. - - MATCH (p:Person) MERGE (c:City {name: p.lives_in}) - -This query needs to create a city node only once for each unique `p.lives_in`. -Finally the set phase of a `MERGE` clause should not affect the merge phase. -To achieve this the `merge_match` branch of the `Merge` operator should see -the latest created nodes, but filter them on their old state (if those nodes -were not created by the `create_branch`). Implementation-wise that means that -`ScanAll` and `Expand` operators in the `merge_branch` need to look at the new -graph state, while `Filter` operators the old, if available. diff --git a/docs/dev/query/interpreter-class.dot b/docs/dev/query/interpreter-class.dot deleted file mode 100644 index 0b728ccf4..000000000 --- a/docs/dev/query/interpreter-class.dot +++ /dev/null @@ -1,23 +0,0 @@ -digraph interpreter { - node [fontname="dejavusansmono"] - edge [fontname="dejavusansmono"] - node [shape=record] - edge [dir=back,arrowtail=empty,arrowsize=1.5] - Interpreter [label="{\N|+ operator(query : string, ...) : Results\l| - # MakeLogicalPlan(...) : LogicalPlan\l| - - plan_cache_ : Map(QueryHash, CachedPlan)\l}"] - Interpreter -> DistributedInterpreter - Results [label="{\N|+ PullAll(stream) : void\l|- plan_ : CachedPlan\l}"] - Interpreter -> Results - [dir=forward,style=dashed,arrowhead=open,label="<>"] - CachedPlan -> Results - [dir=forward,arrowhead=odiamond,taillabel="1",headlabel="*"] - Interpreter -> CachedPlan [arrowtail=diamond,taillabel="1",headlabel="*"] - CachedPlan -> LogicalPlan [arrowtail=diamond] - LogicalPlan [label="{\N|+ GetRoot() : LogicalOperator - \l+ GetCost() : double\l}"] - LogicalPlan -> SingleNodeLogicalPlan [style=dashed] - LogicalPlan -> DistributedLogicalPlan [style=dashed] - DistributedInterpreter -> DistributedLogicalPlan - [dir=forward,style=dashed,arrowhead=open,label="<>"] -} diff --git a/docs/dev/query/interpreter-class.png b/docs/dev/query/interpreter-class.png deleted file mode 100644 index b36d223f3..000000000 Binary files a/docs/dev/query/interpreter-class.png and /dev/null differ diff --git a/docs/dev/query/parsing.md b/docs/dev/query/parsing.md deleted file mode 100644 index 3db3c7f49..000000000 --- a/docs/dev/query/parsing.md +++ /dev/null @@ -1,62 +0,0 @@ -# Lexical and Syntactic Analysis - -## Antlr - -We use Antlr for lexical and syntax analysis of Cypher queries. Antrl uses -grammar file `Cypher.g4` downloaded from http://www.opencypher.org to generate -the parser and the visitor for the Cypher parse tree. Even though the provided -grammar is not very pleasant to work with we decided not to do any drastic -changes to it so that our transition to newly published versions of -`Cypher.g4` would be easier. Nevertheless, we had to fix some bugs and add -features, so our version is not completely the same. - -In addition to using `Cypher.g4`, we have `MemgraphCypher.g4`. This grammar -file defines Memgraph specific extensions to the original grammar. Most -notable example is the inclusion of syntax for handling authorization. At the -moment, some extensions are also found in `Cypher.g4`. For example, the syntax -for using a lambda function in relationship patterns. These extensions should -be moved out of `Cypher.g4`, so that it remains as close to the original -grammar as possible. Additionally, having `MemgraphCypher.g4` may not be -enough if we wish to split the functionality for community and enterprise -editions of Memgraph. - -## Abstract Syntax Tree (AST) - -Since Antlr generated visitor and the official openCypher grammar are not very -practical to use, we translate the Antlr's AST to our own AST. Currently there -are ~40 types of nodes in our AST. Their definitions can be found in -`src/query/frontend/ast/ast.lcp`. - -Major groups of types can be found under the following base types. - - * `Expression` --- types corresponding to Cypher expressions. - * `Clause` --- types corresponding to Cypher clauses. - * `PatternAtom` --- node or edge related information. - * `Query` --- different kinds of queries, allows extending the language with - Memgraph specific query syntax. - -Memory management of created AST nodes is done with `AstStorage`. Each type -must be created by invoking `AstStorage::Create` method. This way all of the -pointers to nodes and their children are raw pointers. The only owner of -allocated memory is the `AstStorage`. When the storage goes out of scope, the -pointers become invalid. It may be more natural to handle tree ownership via -`unique_ptr`, i.e. each node owns its children. But there are some benefits to -having a custom storage and allocation scheme. - -The primary reason we opted for not using `unique_ptr` is the requirement of -Antlr's base visitor class that the resulting values must by copyable. The -result is wrapped in `antlr::Any` so that the derived visitor classes may -return any type they wish when visiting Antlr's AST. Unfortunately, -`antlr::Any` does not work with non-copyable types. - -Another benefit of having `AstStorage` is that we can easily add a different -allocation scheme for AST nodes. The interface of node creation would not -change. - -### AST Translation - -The translation process is done via `CypherMainVisitor` class, which is -derived from Antlr generated visitor. Besides instancing our AST types, a -minimal number of syntactic checks are done on a query. These checks handle -the cases which were valid in original openCypher grammar, but may be invalid -when combined with other syntax elements. diff --git a/docs/dev/query/planning.md b/docs/dev/query/planning.md deleted file mode 100644 index a44678112..000000000 --- a/docs/dev/query/planning.md +++ /dev/null @@ -1,526 +0,0 @@ -# Logical Planning - -After the semantic analysis and symbol generation, the AST is converted to a -tree of logical operators. This conversion is called *planning* and the tree -of logical operators is called a *plan*. The whole planning process is done in -the following steps. - - 1. [AST Preprocessing](#ast-preprocessing) - - The first step is to preprocess the AST by collecting - information on filters, divide the query into parts, normalize patterns - in `MATCH` clauses, etc. - - 2. [Logical Operator Planning](#logical-operator-planning) - - After the preprocess step, the planning can be done via 2 planners: - `VariableStartPlanner` and `RuleBasedPlanner`. The first planner will - generate multiple plans where each plan has different starting points for - searching the patterns in `MATCH` clauses. The second planner produces a - single plan by mapping the query parts as they are to logical operators. - - 3. [Logical Plan Postprocessing](#logical-plan-postprocessing) - - In this stage, we perform various transformations on the generated logical - plan. Here we want to optimize the operations in order to improve - performance during the execution. Naturally, transformations need to - preserve the semantic behaviour of the original plan. - - 4. [Cost Estimation](#cost-estimation) - - After the generation, the execution cost of each plan is estimated. This - estimation is used to select the best plan which will be executed. - -The implementation can be found in the `query/plan` directory, with the public -entry point being `query/plan/planner.hpp`. - -## AST Preprocessing - -Each openCypher query consists of at least 1 **single query**. Multiple single -queries are chained together using a **query combinator**. Currently, there is -only one combinator, `UNION`. The preprocessing step starts in the -`CollectQueryParts` function. This function will take a look at each single -query and divide it into parts. Each part is separated with `RETURN` and -`WITH` clauses. For example: - - MATCH (n) CREATE (m) WITH m MATCH (l)-[]-(m) RETURN l - | | | - |------- part 1 -----------+-------- part 2 --------| - | | - |-------------------- single query -----------------| - -Each part is created by collecting all `MATCH` clauses and *normalizing* their -patterns. Pattern normalization is the process of converting an arbitrarily -long pattern chain of nodes and edges into a list of triplets `(start node, -edge, end node)`. The triplets should preserve the semantics of the match. For -example: - - MATCH (a)-[p]-(b)-[q]-(c)-[r]-(d) - -is equivalent to: - - MATCH (a)-[p]-(b), (b)-[q]-(c), (c)-[r]-(d) - -With this representation, it becomes easier to reorder the triplets and choose -different strategies for pattern matching. - -In addition to normalizing patterns, all of the filter expressions in patterns -and inside of the `WHERE` clause (of the accompanying `MATCH`) are extracted -and stored separately. During the extraction, symbols used in the filter -expression are collected. This allows for planning filters in a valid order, -as the matching for triplets is being done. Another important benefit of -having extra information on filters, is to recognize when a database index -could be used. - -After each `MATCH` is processed, they are all grouped, so that even the whole -`MATCH` clauses may be reordered. The important thing is to remember which -symbols were used to name edges in each `MATCH`. With those symbols we can -plan for *cyphermorphism*, i.e. ensure different edges in the search pattern -of a single `MATCH` map to different edges in the graph. This preserves the -semantic of the query, even though we may have reordered the matching. The -same steps are done for `OPTIONAL MATCH`. - -Another clause which needs processing is `MERGE`. Here we normalize the -pattern, since the `MERGE` is a bit like `MATCH` and `CREATE` in one. - -All the other clauses are left as is. - -In the end, each query part consists of: - - * processed and grouped `MATCH` clauses; - * processed and grouped `OPTIONAL MATCH` clauses; - * processed `MERGE` matching pattern and - * unchanged remaining clauses. - -The last stored clause is guaranteed to be either `WITH` or `RETURN`. - -## Logical Operator Planning - -### Variable Start Planner - -The `VariableStartPlanner` generates multiple plans for a single query. Each -plan is generated by selecting a different starting point for pattern -matching. - -The algorithm works as follows. - - 1. For each query part: - 1. For each node in triplets of collected `MATCH` clauses: - i. Add the node to a set of `expanded` nodes - ii. Select a triplet `(start node, edge, end node)` whose `start node` is - in the `expanded` set - iii. If no triplet was selected, choose a new starting node that isn't in - `expanded` and continue expanding - iv. Repeat steps ii. -- iii. until all triplets have been selected - and store that as a variation of the `MATCH` clauses - 2. Do step 1.1. for `OPTIONAL MATCH` and `MERGE` clauses - 3. Take all combinations of the generated `MATCH`, `OPTIONAL MATCH` and - `MERGE` and store them as variations of the query part. - 2. For each combination of query part variations: - 1. Generate a plan using the rule based planner - -### Rule Based Planner - -The `RuleBasedPlanner` generates a single plan for a single query. A plan is -generated by following hardcoded rules for producing logical operators. The -following sections are an overview on how each openCypher clause is converted -to a `LogicalOperator`. - -#### MATCH - -`MATCH` clause is used to specify which patterns need to be searched for in -the database. These patterns are normalized in the preprocess step to be -represented as triplets `(start node, edge, end node)`. When there is no edge, -then the triplet is reduced only to the `start node`. Generating the operators -is done by looping over these triplets. - -##### Searching for Nodes - -The simplest search is finding standalone nodes. For example, `MATCH (n)` -will find all the nodes in the graph. This is accomplished by generating a -`ScanAll` operator and forwarding the node symbol which should store the -results. In this case, all the nodes will be referenced by `n`. - -Multiple nodes can be specified in a single match, e.g. `MATCH (n), (m)`. -Planning is done by repeating the same steps for each sub pattern (separated -by a comma). In this case, we would get 2 `ScanAll` operators chained one -after the other. An optimization can be obtained if the node in the pattern is -already searched for. In `MATCH (n), (n)` we can drop the second `ScanAll` -operator since we have already generated it for the first node. - -##### Searching for Relationships - -A more advanced search includes finding nodes with relationships. For example, -`MATCH (n)-[r]-(m)` should find every pair of connected nodes in the database. -This means, that if a single node has multiple connections, it will be -repeated for each combination of pairs. The generation of operators starts -from the first node in the pattern. If we are referencing a new starting node, -we need to generate a `ScanAll` which finds all the nodes and stores them -into `n`. Then, we generate an `Expand` operator which reads the `n` and -traverses all the edges of that node. The edge is stored into `r`, while the -destination node is stored in `m`. - -Matching multiple relationships proceeds similarly, by repeating the same -steps. The only difference is that we need to ensure different edges in the -search pattern, map to different edges in the graph. This means that after each -`Expand` operator, we need to generate an `EdgeUniquenessFilter`. We provide -this operator with a list of symbols for the previously matched edges and the -symbol for the current edge. - -For example. - - MATCH (n)-[r1]-(m)-[r2]-(l) - -The above is preprocessed into - - MATCH (n)-[r1]-(m), (m)-[r2]-(l) - -Then we look at each triplet in order and perform the described steps. This -way, we would generate: - - ScanAll (n) > Expand (n, r1, m) > Expand (m, r2, l) > - EdgeUniquenessFilter ([r1], r2) - -Note that we don't need to make `EdgeUniquenessFilter` after the first -`Expand`, since there are no edges to compare to. This filtering needs to work -across multiple pattern, but inside a *single* `MATCH` clause. - -Let's take a look at the following. - - MATCH (n)-[r1]-(m), (m)-[r2]-(l) - -We would also generate the exact same operators. - - ScanAll (n) > Expand (n, r1, m) > Expand (m, r2, l) > - EdgeUniquenessFilter ([r1], r2) - -On the other hand, - - MATCH (n)-[r1]-(m) MATCH (m)-[r2]-(l)-[r3]-(i) - -would reset the uniqueness filtering at the start of the second match. This -would mean that we output the following: - - ScanAll (n) > Expand (n, r1, m) > Expand (m, r2, l) > Expand (l, r3, i) > - EdgeUniquenessFilter ([r2], r3) - -There is a difference in how we handle edge uniqueness compared to Neo4j. -Neo4j does not allow searching for a single edge multiple times, but we've -decided to support that. - -For example, the user can say the following. - - MATCH (n)-[r]-(m)-[r]-l - -We would ensure that both `r` variables match to the same edge. In our -terminology, we call this the *edge cycle*. For the above example, we would -generate this plan. - - ScanAll (n) > Expand (n, r, m) > Expand (m, r, l) - -We do not put an `EdgeUniquenessFilter` operator between 2 `Expand` -operators and we tell the 2nd `Expand` that it is an edge cycle. This, 2nd -`Expand` will ensure we have matched both the same edges. - -##### Filtering - -To narrow the search down, the patterns in `MATCH` can have filtered labels -and properties. A more general filtering is done using the accompanying -`WHERE` clause. During the preprocess step, all filters are collected and -extracted into expressions. Additional information on which symbols are used -is also stored. This way, each time we generate a `ScanAll` or `Expand`, we -look at all the filters to see if any of them can be used. I.e. if the symbols -they use have been bound by a newly produced operator. If a filter expression -can be used, we immediately add a `Filter` operator with that expression. - -For example. - - MATCH (n)-[r]-(m :label) WHERE n.prop = 42 - -We would produce: - - ScanAll (n) > Filter (n.prop) > Expand (n, r, m) > Filter (m :label) - -This means that the same plan is generated for the query: - - MATCH (n {prop: 42})-[r]-(m :label) - -#### OPTIONAL - -If a `MATCH` clause is preceded by `OPTIONAL`, then we need to generate a plan -such that we produce results even if we fail to match anything. This is -accomplished by generating an `Optional` operator, which takes 2 operator -trees: - - * input operation and - * optional operation. - -The input is the operation we generated for the part of the query before -`OPTIONAL MATCH`. For the optional operation, we simply generate the `OPTIONAL -MATCH` part just like we would for regular `MATCH`. In addition to operations, -we need to send the symbols which are set during optional matching to the -`Optional` operator. The operator will reset values of those symbols to -`null`, when the optional part fails to match. - -#### RETURN & WITH - -`RETURN` and `WITH` clauses are very similar to each other. The only -difference is that `WITH` separates parts of the query and can be paired with -`WHERE` clause. - -The common part is generating operators for the body of the clause. Separation -of query parts is mostly done in semantic analysis, which checks that only the -symbols exposed through `WITH` are visible in the query parts after the -clause. The minor part is done in planning. - -##### Named Results - -Both clauses contain multiple named expressions (`expr AS name`) which are -used to generate `Produce` operator. - -##### Aggregations - -If an expression contains an aggregation operator (`sum`, `avg`, ...) we need -to plan the `Aggregate` operator as input to `Produce`. This case is more -complex, because aggregation in openCypher can perform implicit grouping of -results used for aggregation. - -For example, `WITH/RETURN sum(n.x) AS s, n.y AS group` will implicitly group -by `n.y` expression. - -Another, obscure grouping can be achieved with `RETURN sum(n.a) + n.b AS s`. -Here, the `n.b` will be used for grouping, even though both the `sum` and -`n.b` are in the same named expression. - -Therefore, we need to collect all expressions which do not contain -aggregations and use them for grouping. You may have noticed that in the last -example `sum` is actually a sub-expression of `+`. `Aggregate` operator does -not see that (nor it should), so the responsibility of evaluating that falls -on `Produce`. One way is for `Aggregate` to store results of grouping -expressions on the frame in addition to aggregation results. Unfortunately, -this would require rewiring named expressions in `Produce` to reference -already evaluated expressions. In the current implementation, we opted for -`Aggregate` to store only aggregation results on the frame, while `Produce` -will re-evaluate all the other (grouping) expressions. To handle that, symbols -which are used in expressions are passed to `Aggregate`, so that they can be -remembered. `Produce` will read those symbols from the frame and use it to -re-evaluate the needed expressions. - -##### Accumulation - -After we have `Produce` and potentially `Aggregate`, we need to handle a -special case when the part of the query before `RETURN` or `WITH` performs -updates. For that, we want to run that part of the query fully, so that we get -the latest results. This is accomplished by adding `Accumulate` operator as -input to `Aggregate` or `Produce` (if there is no aggregation). Accumulation -will store all the values for all the used symbols inside `RETURN` and `WITH`, -so that they can be used in the operator which follows. This way, only parts -of the frame are copied, instead of the whole frame. Here is a minor -difference between planning `WITH`, compared to `RETURN`. Since `WITH` can -separate writing from reading, we need to advance the transaction command. -This enables the later, read parts of the query to obtain the newest changes. -This is supported by passing `advance_command` flag to `Accumulate` operator. - -In the simplest case, common to both clauses, we have `Accumulate > Aggregate -> Produce` operators, where `Accumulate` and `Aggregate` may be left out. - -##### Ordering - -Planning `ORDER BY` is simple enough. Since it may see new symbols (filled in -`Produce`), we add the `OrderBy` operator at the end. The operator will change -the order of produced results, so we pass it the ordering expressions and the -output symbols of named expressions. - -##### Filtering - -A final difference in `WITH`, is when it contains a `WHERE` clause. For that, -we simply generate the `Filter` operator, appended after `Produce` or -`OrderBy` (depending which operator is last). - -##### Skipping and Limiting - -If we have `SKIP` or `LIMIT`, we generate `Skip` or `Limit` operators, -respectively. These operators are put at the end of the clause. - -This placement may have some unexpected behaviour when combined with -operations that update the graph. For example. - - MATCH (n) SET n.x = n.x + 1 RETURN n LIMIT 1 - -The above query may be interpreted as if the `SET` will be done only once. -Since this is a write query, we need to accumulate results, so the part before -`RETURN` will execute completely. The accumulated results will be yielded up -to the given limit, and the user would get only the first `n` that was -updated. This may confuse the user because in reality, every node in the -database had been updated. - -Note that `Skip` always comes before `Limit`. In the current implementation, -they are generated directly one after the other. - -#### CREATE - -`CREATE` clause is used to create nodes and edges (relationships). - -For multiple `CREATE` clauses or multiple creation patterns in a single -clause, we perform the same, following steps. - -##### Creating a Single Node - -A node is created by simply specifying a node pattern. - -For example `CREATE (n :label {property: "value"}), ()` would create 2 nodes. -The 1st one would be created with a label and a property. This node could be -referenced later in the query, by using the variable `n`. The 2nd node cannot -be referenced and it would be created without any labels nor properties. For -node creation, we generate a `CreateNode` operator and pass it all the details -of node creation: variable symbol, labels and properties. In the mentioned -example, we would have `CreateNode > CreateNode`. - -##### Creating a Relationship - -To create a relationship, the `CREATE` clause must contain a pattern with a -directed edge. Compared to creating a single node, this case is a bit more -complicated, because either side of the edge may not exist. By exist, we mean -that the endpoint is a variable which already references a node. - -For example, `MATCH (n) CREATE (n)-[r]->(m)` would create an edge `r` and a -node `m` for each matched node `n`. If we focus on the `CREATE` part, we -generate `CreateExpand (n, r, m)` where `n` already exists (refers to matched -node) and `m` would be newly created along with edge `r`. If we had only -`CREATE (n)-[r]->(m)`, then we would need to create both nodes of the edge -`r`. This is done by generating `CreateNode (n) > CreateExpand(n, r, m)`. The -final case is when both endpoints refer to an existing node. For example, when -adding a node with a cyclical connection `CREATE (n)-[r]->(n)`. In this case, -we would generate `CreateNode (n) > CreateExpand (n, r, n)`. We would tell -`CreateExpand` to only create the edge `r` between the already created `n`. - -#### MERGE - -Although the merge operation is complex, planning turns out to be relatively -simple. The pattern inside the `MERGE` clause is used for both matching and -creating. Therefore, we create 2 operator trees, one for each action. - -For example. - - MERGE (n)-[r:r]-(m) - -We would generate a single `Merge` operator which has the following. - - * No input operation (since it is not preceded by any other clause). - - * On match operation - - `ScanAll (n) > Expand (n, r, m) > Filter (r)` - - * On create operation - - `CreateNode (n) > CreateExpand (n, r, m)` - -In cases when `MERGE` contains `ON MATCH` and `ON CREATE` parts, we simply -append their operations to the respective operator trees. - -Observe the following example. - - MERGE (n)-[r:r]-(m) ON MATCH SET n.x = 42 ON CREATE SET m :label - -The `Merge` would be generated with the following. - - * No input operation (again, since there is no clause preceding it). - - * On match operation - - `ScanAll (n) > Expand (n, r, m) > Filter (r) > SetProperty (n.x, 42)` - - * On create operation - - `CreateNode (n) > CreateExpand (n, r, m) > SetLabels (n, :label)` - -When we have preceding clauses, we simply put their operator as input to -`Merge`. - - MATCH (n) MERGE (n)-[r:r]-(m) - -The above would be generated as - - ScanAll (n) > Merge (on_match_operation, on_create_operation) - -Here we need to be careful to recognize which symbols are already declared. -But, since the `on_match_operation` uses the same algorithm for generating a -`Match`, that problem is handled there. The same should hold for -`on_create_operation`, which uses the process of generating a `Create`. So, -finally for this example, the `Merge` would have: - - * Input operation - - `ScanAll (n)` - - * On match operation - - `Expand (n, r, m) > Filter (r)` - - Note that `ScanAll` is not needed since we get the nodes from input. - - * On create operation - - `CreateExpand (n, r, m)` - - Note that `CreateNode` is dropped, since we want to expand the existing one. - -## Logical Plan Postprocessing - -Postprocessing of a logical plan is done by rewriting the original plan into -a more efficient one while preserving the original semantic of operations. -The rewriters are found in `query/plan/rewrite` directory, and currently we -only have one -- `IndexLookupRewriter`. - -### IndexLookupRewriter - -The job of this rewriter is to merge `Filter` and `ScanAll` operations into -equivalent `ScanAllBy` operations. In almost all cases using indexed -lookup will be faster than regular lookup, so `IndexLookupRewriter` simply -does the transformations whenever possible. The simplest case being the -following, assuming we have an index over `id`. - - * Original Plan - - `ScanAll (n) > Filter (id(n) == 42) > Produce (n)` - - * Rewritten Plan - - `ScanAllById (n, id=42) > Produce (n)` - -Naturally, there are some cases we need to be careful about. - - 1. Operators with Multiple Branches - - Here we may not carry `Filter` operations outside of the operator into - its branches, so the branches are rewritten as stand alone plans with a - branch new `IndexLookupRewriter`. Some of the operators with multiple - branches are `Merge`, `Optional`, `Cartesian` and `Union`. - - 2. Expand Operators - - Expand operations aren't that tricky to handle, but they have a special - case where we want to use an indexed lookup of the destination so that the - expansion is performed between known nodes. This decision may depend on - various parameters which may need further tweaking as we encounter more - use-cases of Cypher queries. - -## Cost Estimation - -Cost estimation is the final step of processing a logical plan. The -implementation can be found in `query/plan/cost_estimator.hpp`. We give each -operator a cost based on the estimated cardinality of results of that operator -and on the preset coefficient of the runtime performance of that operator. - -This scheme is rather simple and works quite well, but there are couple of -improvements we may want to do at some point. - - * Track more information about the stored graph and use that to improve the - estimates. - * Do a quick, partial run of the plan and tweak the estimation based on how - much each operator produced results. This may require us having some kind - of representative subset of the stored graph. - * Write micro benchmarks for each operator and based on the results create - sensible preset coefficients. This would replace the current coefficients - which are just assumptions on how each operator implementation performs. diff --git a/docs/dev/query/semantic.md b/docs/dev/query/semantic.md deleted file mode 100644 index ff10cbb5e..000000000 --- a/docs/dev/query/semantic.md +++ /dev/null @@ -1,134 +0,0 @@ -# Semantic Analysis and Symbol Generation - -In this phase, various semantic and variable type checks are performed. -Additionally, we generate symbols which map AST nodes to stored values -computed from evaluated expressions. - -## Symbol Generation - -Implementation can be found in `query/frontend/semantic/symbol_generator.cpp`. - -Symbols are generated for each AST node that represents data that needs to -have storage. Currently, these are: - - * `NamedExpression` - * `CypherUnion` - * `Identifier` - * `Aggregation` - -You may notice that the above AST nodes may not correspond to something named -by a user. For example, `Aggregation` can be a part of larger expression and -thus remain unnamed. The reason we still generate symbols is to have a uniform -behaviour when executing a query as well as allow for caching the results of -expression evaluation. - -AST nodes do not actually store a `Symbol` instance, instead they have a -`int32_t` index identifying the symbol in the `SymbolTable` class. This is -done to minimize the size of AST types as well as allow easier sharing of same -symbols with multiple instances of AST nodes. - -The storage for evaluated data is represented by the `Frame` class. Each -symbol determines a unique position in the frame. During interpretation, -evaluation of expressions which have a symbol will either read or store values -in the frame. For example, instance of an `Identifier` will use the symbol to -find and read the value from `Frame`. On the other hand, `NamedExpression` -will take the result of evaluating its own expression and store it in the -`Frame`. - -When a symbol is created, context of creation is used to assign a type to that -symbol. This type is used for simple type checking operations. For example, -`MATCH (n)` will create a symbol for variable `n`. Since the `MATCH (n)` -represents finding a vertex in the graph, we can set `Symbol::Type::Vertex` -for that symbol. Later, for example in `MATCH ()-[n]-()` we see that variable -`n` is used as an edge. Since we already have a symbol for that variable, we -detect this type mismatch and raise a `SemanticException`. - -Basic rule of symbol generation, is that variables inside `MATCH`, `CREATE`, -`MERGE`, `WITH ... AS` and `RETURN ... AS` clauses establish new symbols. - -### Symbols in Patterns - -Inside `MATCH`, symbols are created only if they didn't exist before. For -example, patterns in `MATCH (n {a: 5})--(m {b: 5}) RETURN n, m` will create 2 -symbols: one for `n` and one for `m`. `RETURN` clause will, in turn, reference -those symbols. Symbols established in a part of pattern are immediately bound -and visible in later parts. For example, `MATCH (n)--(n)` will create a symbol -for variable `n` for 1st `(n)`. That symbol is referenced in 2nd `(n)`. Note -that the symbol is not bound inside 1st `(n)` itself. What this means is that, -for example, `MATCH (n {a: n.b})` should raise an error, because `n` is not -yet bound when encountering `n.b`. On the other hand, -`MATCH (n)--(n {a: n.b})` is fine. - -The `CREATE` is similar to `MATCH`, but it *always* establishes symbols for -variables which create graph elements. What this means is that, for example -`MATCH (n) CREATE (n)` is not allowed. `CREATE` wants to create a new node, -for which we already have a symbol. In such a case, we need to throw an error -that the variable `n` is being redeclared. On the other hand `MATCH (n) CREATE -(n)-[r :r]->(n)` is fine, because `CREATE` will only create the edge `r`, -connecting the already existing node `n`. Remaining behaviour is the same as -in `MATCH`. This means that we can simplify `CREATE` to be like `MATCH` with 2 -special cases. - - 1. Are we creating a node, i.e. `CREATE (n)`? If yes, then the symbol for - `n` must not have been created before. Otherwise, we reference the - existing symbol. - 2. Are we creating an edge, i.e. we encounter a variable for an edge inside - `CREATE`? If yes, then that variable must not reference a symbol. - -The `MERGE` clause is treated the same as `CREATE` with regards to symbol -generation. The only difference is that we allow bidirectional edges in the -pattern. When creating such a pattern, the direction of the created edge is -arbitrarily determined. - -### Symbols in WITH and RETURN - -In addition to patterns, new symbols are established in the `WITH` clause. -This clause makes the new symbols visible *only* to the rest of the query. -For example, `MATCH (old) WITH old AS new RETURN new, old` should raise an -error that `old` is unbound inside `RETURN`. - -There is a special case with symbol visibility in `WHERE` and `ORDER BY`. They -need to see both the old and the new symbols. Therefore `MATCH (old) RETURN -old AS new ORDER BY old.prop` needs to work. On the other hand, if we perform -aggregations inside `WITH` or `RETURN`, then the old symbols should not be -visible neither in `WHERE` nor in `ORDER BY`. Since the aggregation has to go -through all the results in order to generate the final value, it makes no -sense to store old symbols and their values. A query like `MATCH (old) WITH -SUM(old.prop) AS sum WHERE old.prop = 42 RETURN sum` needs to raise an error -that `old` is unbound inside `WHERE`. - -For cases when `SKIP` and `LIMIT` appear, we disallow any identifiers from -appearing in their expressions. Basically, `SKIP` and `LIMIT` can only be -constant expressions[^1]. For example, `MATCH (old) RETURN old AS new SKIP -new.prop` needs to raise that variables are not allowed in `SKIP`. It makes no -sense to allow variables, since their values may vary on each iteration. On -the other hand, we could support variables to constant expressions, but for -simplicity we do not. For example, `MATCH (old) RETURN old, 2 AS limit_var -LIMIT limit_var` would still throw an error. - -Finally, we generate symbols for names created in `RETURN` clause. These -symbols are used for the final results of a query. - -NOTE: New symbols in `WITH` and `RETURN` should be unique. This means that -`WITH a AS same, b AS same` is not allowed, neither is a construct like -`RETURN 2, 2` - -### Symbols in Functions which Establish New Scope - -Symbols can also be created in some functions. These functions usually take an -expression, bind a single variable and run the expression inside the newly -established scope. - -The `all` function takes a list, creates a variable for list element and runs -the predicate expression. For example: - - MATCH (n) RETURN n, all(n IN n.prop_list WHERE n < 42) - -We create a new symbol for use inside `all`, this means that the `WHERE n < -42` uses the `n` which takes values from a `n.prop_list` elements. The -original `n` bound by `MATCH` is not visible inside the `all` function, but it -is visible outside. Therefore, the `RETURN n` and `n.prop_list` reference the -`n` from `MATCH`. - -[^1]: Constant expressions are expressions for which the result can be - computed at compile time. diff --git a/docs/dev/quick-start.md b/docs/dev/quick-start.md deleted file mode 100644 index 1b578fec7..000000000 --- a/docs/dev/quick-start.md +++ /dev/null @@ -1,107 +0,0 @@ -# Quick Start - -A short chapter on downloading the Memgraph source, compiling and running. - -## Obtaining the Source Code - -Memgraph uses `git` for source version control. You will need to install `git` -on your machine before you can download the source code. - -On Debian systems, you can do it inside a terminal with the following -command: - - apt install git - -After installing `git`, you are now ready to fetch your own copy of Memgraph -source code. Run the following command: - - git clone https://github.com/memgraph/memgraph.git - -The above will create a `memgraph` directory and put all source code there. - -## Compiling Memgraph - -With the source code, you are now ready to compile Memgraph. Well... Not -quite. You'll need to download Memgraph's dependencies first. - -In your terminal, position yourself in the obtained memgraph directory. - - cd memgraph - -### Installing Dependencies - -Dependencies that are required by the codebase should be checked by running the -`init` script: - - ./init - -If the script fails, dependencies installation scripts could be found under -`environment/os/`. The directory contains dependencies management script for -each supported operating system. E.g. if your system is **Debian 10**, run the -following to install all required build packages: - - ./environment/os/debian-10.sh install MEMGRAPH_BUILD_DEPS - -Once everything is installed, rerun the `init` script. - -Once the `init` script is successfully finished, issue the following commands: - - mkdir -p build - ./libs/setup.sh - -### Compiling - -Memgraph is compiled using our own custom toolchain that can be obtained from -the toolchain repository. You should read the `environment/README.txt` file -in the repository and install the apropriate toolchain for your distribution. -After you have installed the toolchain you should read the instructions for the -toolchain in the toolchain install directory (`/opt/toolchain-vXYZ/README.md`) -and install dependencies that are necessary to run the toolchain. - -When you want to compile Memgraph you should activate the toolchain using the -prepared toolchain activation script that is also described in the toolchain -`README`. - -NOTE: You **must** activate the toolchain every time you want to compile -Memgraph! - -You should now activate the toolchain in your console. - - source /opt/toolchain-vXYZ/activate - -With all of the dependencies installed and the build environment set-up, you -need to configure the build system. To do that, execute the following: - - cd build - cmake .. - -If everything went OK, you can now, finally, compile Memgraph. - - make -j$(nproc) - -### Running - -After the compilation verify that Memgraph works: - - ./memgraph --version - -To make extra sure, run the unit tests: - - ctest -R unit -j$(nproc) - -## Problems - -If you have any trouble running the above commands, contact your nearest -developer who successfully built Memgraph. Ask for help and insist on getting -this document updated with correct steps! - -## Next Steps - -Familiarise yourself with our code conventions and guidelines: - - * [C++ Code](cpp-code-conventions.md) - * [Other Code](other-code-conventions.md) - * [Code Review Guidelines](code-review.md) - -Take a look at the list of [required reading](required-reading.md) for -brushing up on technical skills. diff --git a/docs/dev/required-reading.md b/docs/dev/required-reading.md deleted file mode 100644 index 128360fac..000000000 --- a/docs/dev/required-reading.md +++ /dev/null @@ -1,129 +0,0 @@ -# Required Reading - -This chapter lists a few books that should be read by everyone working on -Memgraph. Since Memgraph is developed primarily with C++, Python and Common -Lisp, books are oriented around those languages. Of course, there are plenty -of general books which will help you improve your technical skills (such as -"The Pragmatic Programmer", "The Mythical Man-Month", etc.), but they are not -listed here. This way the list should be kept short and the *required* part in -"Required Reading" more easily honored. - -Some of these books you may find in our office, so feel free to pick them up. -If any are missing and you would like a physical copy, don't be afraid to -request the book for our office shelves. - -Besides reading, don't get stuck in a rut and be a -[Blub Programmer](http://www.paulgraham.com/avg.html). - -## Effective C++ by Scott Meyers - -Required for C++ developers. - -The book is a must-read as it explains most common gotchas of using C++. After -reading this book, you are good to write competent C++ which will pass code -reviews easily. - -## Effective Modern C++ by Scott Meyers - -Required for C++ developers. - -This is a continuation of the previous book, it covers updates to C++ which -came with C++11 and later. The book isn't as imperative as the previous one, -but it will make you aware of modern features we are using in our codebase. - -## Practical Common Lisp by Peter Siebel - -Required for Common Lisp developers. - -Free: http://www.gigamonkeys.com/book/ - -We use Common Lisp to generate C++ code and make our lives easier. -Unfortunately, not many developers are familiar with the language. This book -will make you familiar very quickly as it has tons of very practical -exercises. E.g. implementing unit testing library, serialization library and -bundling all that to create a mp3 music server. - -## Effective Python by Brett Slatkin - -(Almost) required reading for Python developers. - -Why the "almost"? Well, Python is relatively easy to pick up and you will -probably learn all the gotchas during code review from someone more -experienced. This makes the book less necessary for a newcomer to Memgraph, -but the book is not advanced enough to delegate it to -[Advanced Reading](#advanced-reading). The book is written in similar vein as -the "Effective C++" ones and will make you familiar with nifty Python features -that make everyone's lives easier. - -# Advanced Reading - -The books listed below are not required reading, but you may want to read them -at some point when you feel comfortable enough. - -## Design Patterns by Gamma et. al. - -Recommended for C++ developers. - -This book is highly divisive because it introduced a culture centered around -design patterns. The main issues is overuse of patterns which complicates the -code. This has made many Java programs to serve as examples of highly -complicated, "enterprise" code. - -Unfortunately, design patterns are pretty much missing -language features. This is most evident in dynamic languages such as Python -and Lisp, as demonstrated by -[Peter Norvig](http://www.norvig.com/design-patterns/). - -Or as [Paul Graham](http://www.paulgraham.com/icad.html) put it: - -``` -This practice is not only common, but institutionalized. For example, in the -OO world you hear a good deal about "patterns". I wonder if these patterns are -not sometimes evidence of case (c), the human compiler, at work. When I see -patterns in my programs, I consider it a sign of trouble. The shape of a -program should reflect only the problem it needs to solve. Any other -regularity in the code is a sign, to me at least, that I'm using abstractions -that aren't powerful enough-- often that I'm generating by hand the expansions -of some macro that I need to write -``` - -After presenting the book so negatively, why you should even read it then? -Well, it is good to be aware of those design patterns and use them when -appropriate. They can improve modularity and reuse of the code. You will also -find examples of such patterns in our code, primarily Strategy and Visitor -patterns. The book is also a good stepping stone to more advanced reading -about software design. - -## Modern C++ Design by Andrei Alexandrescu - -Recommended for C++ developers. - -This book can be treated as a continuation of the previous "Design Patterns" -book. It introduced "dark arts of template meta-programming" to the world. -Many of the patterns are converted to use C++ templates which makes them even -better for reuse. But, like the previous book, there are downsides if used too -much. You should approach it with a critical eye and it will help you -understand ideas that are used in some parts of our codebase. - -## Large Scale C++ Software Design by John Lakos - -Recommended for C++ developers. - -An old book, but well worth the read. Lakos presents a very pragmatic view of -writing modular software and how it affects both development time as well as -program runtime. Some things are outdated or controversial, but it will help -you understand how the whole C++ process of working in a large team, compiling -and linking affects development. - -## On Lisp by Paul Graham - -Recommended for Common Lisp developers. - -Free: http://www.paulgraham.com/onlisp.html - -An excellent continuation to "Practical Common Lisp". It starts of slow, as if -introducing the language, but very quickly picks up speed. The main meat of -the book are macros and their uses. From using macros to define cooperative -concurrency to including Prolog as if it's part of Common Lisp. The book will -help you understand more advanced macros that are occasionally used in our -Lisp C++ Preprocessor (LCP). diff --git a/docs/dev/storage/v1/accessors.md b/docs/dev/storage/v1/accessors.md deleted file mode 100644 index 2a8fac6ce..000000000 --- a/docs/dev/storage/v1/accessors.md +++ /dev/null @@ -1,110 +0,0 @@ -# DatabaseAccessor - -A `DatabaseAccessor` actually wraps a transactional access to database -data, for a single transaction. In that sense the naming is bad. It -encapsulates references to the database and the transaction object. - -It contains logic for working with database content (graph element -data) in the context of a single transaction. All CRUD operations are -performed within a single transaction (as Memgraph is a transactional -database), and therefore iteration over data, finding a specific graph -element etc are all functionalities of a `GraphDbAccessor`. - -In single-node Memgraph the database accessor also defined the lifetime -of a transaction. Even though a `Transaction` object was owned by the -transactional engine, it was `GraphDbAccessor`'s lifetime that object -was bound to (the transaction was implicitly aborted in -`GraphDbAccessor`'s destructor, if it was not explicitly ended before -that). - -# RecordAccessor - -It is important to understand data organization and access in the -storage layer. This discussion pertains to vertices and edges as graph -elements that the end client works with. - -Memgraph uses MVCC (documented on it's own page). This means that for -each graph element there could be different versions visible to -different currently executing transactions. When we talk about a -`Vertex` or `Edge` as a data structure we typically mean one of those -versions. In code this semantic is implemented so that both those classes -inherit `mvcc::Record`, which in turn inherits `mvcc::Version`. - -Handling MVCC and visibility is not in itself trivial. Next to that, -there is other book-keeping to be performed when working with data. For -that reason, Memgraph uses "accessors" to define an API of working with -data in a safe way. Most of the code in Memgraph (for example the -interpretation code) should work with accessors. There is a -`RecordAccessor` as a base class for `VertexAccessor` and -`EdgeAccessor`. Following is an enumeration of their purpose. - -### Data Access - -The client interacts with Memgraph using the Cypher query language. That -language has certain semantics which imply that multiple versions of the -data need to be visible during the execution of a single query. For -example: expansion over the graph is always done over the graph state as -it was at the beginning of the transaction. - -The `RecordAccessor` exposes functions to switch between the old and the new -versions of the same graph element (intelligently named `SwitchOld` and -`SwitchNew`) within a single transaction. In that way the client code -(mostly the interpreter) can avoid dealing with the underlying MVCC -version concepts. - -### Updates - -Data updates are also done through accessors. Meaning: there are methods -on the accessors that modify data, the client code should almost never -interact directly with `Vertex` or `Edge` objects. - -The accessor layer takes care of creating version in the MVCC layer and -performing updates on appropriate versions. - -Next, for many kinds of updates it is necessary to update the relevant -indexes. There are implicit indexes for vertex labels, as -well as user-created indexes for (label, property) pairs. The accessor -layer takes care of updating the indexes when these values are changed. - -Each update also triggers a log statement in the write-ahead log. This -is also handled by the accessor layer. - -### Distributed - -In distributed Memgraph accessors also contain a lot of the remote graph -element handling logic. More info on that is available in the -documentation for distributed. - -### Deferred MVCC Data Lookup for Edges - -Vertices and edges are versioned using MVCC. This means that for each -transaction an MVCC lookup needs to be done to determine which version -is visible to that transaction. This tends to slow things down due to -cache invalidations (version lists and versions are stored in arbitrary -locations on the heap). - -However, for edges, only the properties are mutable. The edge endpoints -and type are fixed once the edge is created. For that reason both edge -endpoints and type are available in vertex data, so that when expanding -it is not mandatory to do MVCC lookups of versioned, mutable data. This -logic is implemented in `RecordAccessor` and `EdgeAccessor`. - -### Exposure - -The original idea and implementation of graph element accessors was that -they'd prevent client code from ever interacting with raw `Vertex` or -`Edge` data. This however turned out to be impractical when implementing -distributed Memgraph and the raw data members have since been exposed -(through getters to old and new version pointers). However, refrain from -working with that data directly whenever possible! Always consider the -accessors to be the first go-to for interacting with data, especially -when in the context of a transaction. - -# Skiplist Accessor - -The term "accessor" is also used in the context of a skiplist. Every -operation on a skiplist must be performed within on an -accessor. The skiplist ensures that there will be no physical deletions -of an object during the lifetime of an accessor. This mechanism is used -to ensure deletion correctness in a highly concurrent container. -We only mention that here to avoid confusion regarding terminology. diff --git a/docs/dev/storage/v1/contents.md b/docs/dev/storage/v1/contents.md deleted file mode 100644 index e378d1379..000000000 --- a/docs/dev/storage/v1/contents.md +++ /dev/null @@ -1,6 +0,0 @@ -# Storage v1 - -* [Accessors](accessors.md) -* [Indexes](indexes.md) -* [Property Storage](property-storage.md) -* [Durability](durability.md) diff --git a/docs/dev/storage/v1/durability.md b/docs/dev/storage/v1/durability.md deleted file mode 100644 index 6648a3dc9..000000000 --- a/docs/dev/storage/v1/durability.md +++ /dev/null @@ -1,80 +0,0 @@ -# Durability - -## Write-ahead Logging - -Typically WAL denotes the process of writing a "log" of database -operations (state changes) to persistent storage before committing the -transaction, thus ensuring that the state can be recovered (in the case -of a crash) for all the transactions which the database committed. - -The WAL is a fine-grained durability format. It's purpose is to store -database changes fast. It's primary purpose is not to provide -space-efficient storage, nor to support fast recovery. For that reason -it's often used in combination with a different persistence mechanism -(in Memgraph's case the "snapshot") that has complementary -characteristics. - -### Guarantees - -Ensuring that the log is written before the transaction is committed can -slow down the database. For that reason this guarantee is most often -configurable in databases. - -Memgraph offers two options for the WAL. The default option, where the WAL is -flushed to the disk periodically and transactions do not wait for this to -complete, introduces the risk of database inconsistency because an operating -system or hardware crash might lead to missing transactions in the WAL. Memgraph -will handle this as if those transactions never happened. The second option, -called synchronous commit, will instruct Memgraph to wait for the WAL to be -flushed to the disk when a transactions completes and the transaction will wait -for this to complete. This option can be turned on with the -`--synchronous-commit` command line flag. - -### Format - -The WAL file contains a series of DB state changes called `StateDelta`s. -Each of them describes what the state change is and in which transaction -it happened. Also some kinds of meta-information needed to ensure proper -state recovery are recorded (transaction beginnings and commits/abort). - -The following is guaranteed w.r.t. `StateDelta` ordering in -a single WAL file: -- For two ops in the same transaction, if op A happened before B in the - database, that ordering is preserved in the log. -- Transaction begin/commit/abort messages also appear in exactly the - same order as they were executed in the transactional engine. - -### Recovery - -The database can recover from the WAL on startup. This works in -conjunction with snapshot recovery. The database attempts to recover from -the latest snapshot and then apply as much as possible from the WAL -files. Only those transactions that were not recovered from the snapshot -are recovered from the WAL, for speed efficiency. It is possible (but -inefficient) to recover the database from WAL only, provided all the WAL -files created from DB start are available. It is not possible to recover -partial database state (i.e. from some suffix of WAL files, without the -preceding snapshot). - -## Snapshots - -A "snapshot" is a record of the current database state stored in permanent -storage. Note that the term "snapshot" is used also in the context of -the transaction engine to denote a set of running transactions. - -A snapshot is written to the file by Memgraph periodically if so -configured. The snapshot creation process is done within a transaction created -specifically for that purpose. The transaction is needed to ensure that -the stored state is internally consistent. - -The database state can be recovered from the snapshot during startup, if -so configured. This recovery works in conjunction with write-ahead log -recovery. - -A single snapshot contains all the data needed to recover a database. In -that sense snapshots are independent of each other and old snapshots can -be deleted once the new ones are safely stored, if it is not necessary -to revert the database to some older state. - -The exact format of the snapshot file is defined inline in the snapshot -creation code. diff --git a/docs/dev/storage/v1/indexes.md b/docs/dev/storage/v1/indexes.md deleted file mode 100644 index b623d13d3..000000000 --- a/docs/dev/storage/v1/indexes.md +++ /dev/null @@ -1,116 +0,0 @@ -# Label Indexes - -These are unsorted indexes that contain all the vertices that have the label -the indexes are for (one index per label). These kinds of indexes get -automatically generated for each label used in the database. - -### Updating the Indexes - -Whenever something gets added to the record we update the index (add that -record to index). We keep an index which might contain garbage (not relevant -records, because the value got removed or something similar) but we will -filter it out when querying the index. We do it like this because we don't -have to do bookkeeping and deciding if we update the index on the end of the -transaction (commit/abort phase), moreover current interpreter advances the -command in transaction and as such assumes that the indexes now contain -objects added in the previous command inside this transaction, so we need to -update over the whole scope of transaction (whenever something is added to the -record). - -### Index Entries Label - -These kinds of indexes are internally keeping track of pair (record, vlist). -Why do we need to keep track of exactly those two things? - -Problems with two different approaches - -1) Keep track of just the record: - - - We need the `VersionList` for creating an accessor (this in itself is a - deal-breaker). - - Semantically it makes sense. An edge/vertex maps bijectionally to a - `VersionList`. - - We might try to access some members of record while the record is being - modified from another thread. - - A vertex/edge could get updated, thus expiring the record in the index. - The newly created record should be present in the index, but it's not. - Without the `VersionList` we can't reach the newly created record. - - Probably there are even more reasons... It should be obvious by now that - we need the `VersionList` in the index. - -2) Keep track of just the version list: - - - Removing from an index is a problem for two major reasons. First, if we - only have the `VersionList`, checking if it should be removed implies - checking all the reachable records, which is not thread-safe. Second, - there are issues with concurrent removal and insertion. The cleanup thread - could determine the vertex/edge should be removed from the index and - remove it, while in between those ops another thread attempts to insert - the `VersionList` into the index. The insertion does nothing because the - `VersionList` is already in, but it gets removed immediately after. - -Because of inability to keep track of just the record, or value, we need to -keep track of both of them. Resolution of problems mentioned above, in the -same order, with (record, vlist) pair - - - simple `vlist.find(current transaction)` will get us the newest visible - record - - we'll never try to access some record if it's still being written since we - will always operate on vlist.find returned record - - newest record will contain that label - - since we have (record, vlist) pair as the key in the index when we update - and delete in the same time we will never delete the same record, vlist - pair we are adding because the record, vlist pair we are deleting is - already superseded by a newer record and as such won't be inserted while - it's being deleted - -### Querying the Index - -We run through the index for the given label and do `vlist.find` operation for -the current transaction, and check if the newest return record has that -label. If it has it then we return it. By now you are probably wondering -aren't we sometimes returning duplicate vlist entries? And you are wondering -correctly, we would be returning them, but we are making sure that the entires -in the index are sorted by their `vlist*` and as such we can filter consecutive -duplicate `vlist*` to only return one of those while still being able to create -an iterator to index. - -### Cleaning the Index - -Cleaning the index is not as straightforward as it seems as a lot of garbage -can accumulate, but it's hard to know when exactly can we delete some (record, -vlist) pair. First, let's assume that we are doing the cleaning process at -some `transaction_id`, `id` such that there doesn't exist an active transaction -with an id lower than `id`. - -We scan through the whole index and for each (record, vlist) pair we first -check if it was deleted before the id (i.e. no transaction with an id >= `id` -will ever again see that record), if it was deleted before we might naively -say that it's safe to delete it, but, we must take into account that when some -new record is created from this record (update operation), that record still -contains the label but by deleting this record we won't be able to see that -vlist because that new record won't add again to index because we didn't -explicitly add that label again to it. - -Because of this we have to 'update' this index (record, vlist) pair. We have -to update the record to now point to a newer record in vlist, the one that is -not deleted yet. We can do that by querying the `version_list` for the last -record inside (oldest it has — remember that `mvcc_gc` will re-link not -visible records so the last record will be visible for the current GC id). -When updating the record inside the index, it's not okay to just update the -pointer and leave the index as it is, because with updating the `record*` we -might change the relative order of entries inside the index. We first have to -re-insert it with new `record*`, and then delete the old entry. And we need to -do insertion before the remove operation! Otherwise it could happen that the -vlist with a newer record with that label won't exist while some transaction -is querying the index. - -Records which we added as a consequence of deleting older records will be -eventually removed from the index if they don't contain label because if we -see that the record is not deleted we try to check if that record still -contains the label. We also need to be careful here because we can't check -that while the record is being potentially updated by some transaction (race -condition), so we need can check if records still contain label if it's -creation id is smaller than our `id`, as that implies that the creating -transaction either aborted or committed as our `id` is equal to the oldest -active transaction in time of starting the GC. diff --git a/docs/dev/storage/v1/property-storage.md b/docs/dev/storage/v1/property-storage.md deleted file mode 100644 index 947c4d8b3..000000000 --- a/docs/dev/storage/v1/property-storage.md +++ /dev/null @@ -1,131 +0,0 @@ -# Property Storage - -Although the reader is probably familiar with properties in *Memgraph*, let's -briefly recap. - -Both vertices and edges can store an arbitrary number of properties. Properties -are, in essence, ordered pairs of property names and property values. Each -property name within a single graph element (edge/node) can store a single -property value. Property names are represented as strings, while property values -must be one of the following types: - - Type | Description ------------|------------ - `Null` | Denotes that the property has no value. This is the same as if the property does not exist. - `String` | A character string, i.e. text. - `Boolean` | A boolean value, either `true` or `false`. - `Integer` | An integer number. - `Float` | A floating-point number, i.e. a real number. - `List` | A list containing any number of property values of any supported type. It can be used to store multiple values under a single property name. - `Map` | A mapping of string keys to values of any supported type. - -Property values are modeled in a class conveniently called `PropertyValue`. - -## Mapping Between Property Names and Property Keys. - -Although users think of property names in terms of descriptive strings -(e.g. "location" or "department"), *Memgraph* internally converts those names -into property keys which are, essentially, unsigned 16-bit integers. - -Property keys are modelled by a not-so-conveniently named class called -`Property` which can be found in `storage/types.hpp`. The actual conversion -between property names and property keys is done within the `ConcurrentIdMapper` -but the internals of that implementation are out of scope for understanding -property storage. - -## PropertyValueStore - -Both `Edge` and `Vertex` objects contain an instance of `PropertyValueStore` -object which is responsible for storing properties of a corresponding graph -element. - -An interface of `PropertyValueStore` is as follows: - - Method | Description ------------|------------ - `at` | Returns the `PropertyValue` for a given `Property` (key). - `set` | Stores a given `PropertyValue` under a given `Property` (key). - `erase` | Deletes a given `Property` (key) alongside its corresponding `PropertyValue`. - `clear` | Clears the storage. - `iterator`| Provides an extension of `std::input_iterator` that iterates over storage. - -## Storage Location - -By default, *Memgraph* is an in-memory database and all properties are therefore -stored in working memory unless specified otherwise by the user. User has an -option to specify via the command line which properties they wish to be stored -on disk. - -Storage location of each property is encapsulated within a `Property` object -which is ensured by the `ConcurrentIdMapper`. More precisely, the unsigned 16-bit -property key has the following format: - -``` -|---location--|------id------| -|-Memory|Disk-|-----2^15-----| -``` - -In other words, the most significant bit determines the location where the -property will be stored. - -### In-memory Storage - -The underlying implementation of in-memory storage for the time being is -`std::vector>`. Implementations of`at`, `set` -and `erase` are linear in time. This implementation is arguably more efficient -than `std::map` or `std::unordered_map` when the average number of properties of -a record is relatively small (up to 10) which seems to be the case. - -### On-disk Storage - -#### KVStore - -Disk storage is modeled by an abstraction of key-value storage as implemented in -`storage/kvstore.hpp'. An interface of this abstraction is as follows: - - Method | Description -----------------|------------ - `Put` | Stores the given value under the given key. - `Get` | Obtains the given value stored under the given key. - `Delete` | Deletes a given (key, value) pair from storage.. - `DeletePrefix` | Deletes all (key, value) pairs where key begins with a given prefix. - `Size` | Returns the size of the storage or, optionally, the number of stored pairs that begin with a given prefix. - `iterator` | Provides an extension of `std::input_iterator` that iterates over storage. - -Keys and values in this context are of type `std::string`. - -The actual underlying implementation of this abstraction uses -[RocksDB]{https://rocksdb.org} — a persistent key-value store for fast -storage. - -It is worthy to note that the custom iterator implementation allows the user -to iterate over a given prefix. Otherwise, the implementation follows familiar -c++ constructs and can be used as follows: - -``` -KVStore storage = ...; -for (auto it = storage.begin(); it != storage.end(); ++it) {} -for (auto kv : storage) {} -for (auto it = storage.begin("prefix"); it != storage.end("prefix"); ++it) {} -``` - -Note that it is not possible to scan over multiple prefixes. For instance, one -might assume that you can scan over all keys that fall in a certain -lexicographical range. Unfortunately, that is not the case and running the -following code will result in an infinite loop with a touch of undefined -behavior. - -``` -KVStore storage = ...; -for (auto it = storage.begin("alpha"); it != storage.end("omega"); ++it) {} -``` - -#### Data Organization on Disk - -Each `PropertyValueStore` instance can access a static `KVStore` object that can -store `(key, value)` pairs on disk. The key of each property on disk consists of -two parts — a unique identifier (unsigned 64-bit integer) of the current -record version (see mvcc docummentation for further clarification) and a -property key as described above. The actual value of the property is serialized -into a bytestring using bolt `BaseEncoder`. Similarly, deserialization is -performed by bolt `Decoder`. diff --git a/docs/dev/storage/v2/contents.md b/docs/dev/storage/v2/contents.md deleted file mode 100644 index 33496731e..000000000 --- a/docs/dev/storage/v2/contents.md +++ /dev/null @@ -1,3 +0,0 @@ -# Storage v2 - -TODO(gitbuda): Write documentation. diff --git a/docs/dev/workflow.md b/docs/dev/workflow.md deleted file mode 100644 index aef6dd158..000000000 --- a/docs/dev/workflow.md +++ /dev/null @@ -1,166 +0,0 @@ -# Memgraph Workflow - -This chapter describes the usual workflow for working on Memgraph. - -## Git - -Memgraph uses [git](https://git-scm.com/) for source version control. If you -obtained the source, you probably already have it installed. Before you can -track new changes, you need to setup some basic information. - -First, tell git your name: - - git config --global user.name "FirstName LastName" - -Then, set your Memgraph email: - - git config --global user.email "my.email@memgraph.com" - -Finally, make git aware of your favourite editor: - - git config --global core.editor "vim" - -## Github - -All of the code in Memgraph needs to go through code review before it can be -accepted in the codebase. This is done through [Github](https://github.com/). -You should already have it installed if you followed the steps in [Quick -Start](quick-start.md). - -## Working on Your Feature Branch - -Git has a concept of source code **branches**. The `master` branch contains all -of the changes which were reviewed and accepted in Memgraph's code base. The -`master` branch is selected by default. - -### Creating a Branch - -When working on a new feature or fixing a bug, you should create a new branch -out of the `master` branch. There are two branch types, **epic** and **task** -branches. The epic branch is created when introducing a new feature or any work -unit requiring more than one commit. More commits are required to split the -work into chunks to be able to easier review code or find a bug (in each -commit, there could be various problems, e.g., related to performance or -concurrency issues, which are the hardest to track down). Each commit on the -master or epic branch should be a compilable and well-documented set of -changes. Task branches should be created when a smaller work unit has to be -integrated into the codebase. The task branch could be branched out of the -master or an epic branch. We manage epics and tasks on the project management -tool called [Airtable](https://airtable.com/tblTUqycq8sHTTkBF). Each epic is -prefixed by `Exyz-MG`, on the other hand, each task has `Tabcd-MG` prefix. -Examples on how to create branches follow: - -``` -git checkout master -git checkout -b T0025-MG-fix-a-problem -... -git checkout master -git checkout -b E025-MG-huge-feature -... -git checkout E025-MG-huge-feature -git checkout -b T0123-MG-add-feature-part -``` - -Note that a branch is created from the currently selected branch. So, if you -wish to create another branch from `master` you need to switch to `master` -first. - -### Making and Committing Changes - -When you have a branch for your new addition, you can now actually start -implementing it. After some amount of time, you may have created new files, -modified others and maybe even deleted unused files. You need to tell git to -track those changes. This is accomplished with `git add` and `git rm` -commands. - - git add path-to-new-file path-to-modified-file - git rm path-to-deleted-file - -To check that everything is correctly tracked, you may use the `git status` -command. It will also print the name of the currently selected branch. - -If everything seems OK, you should commit these changes to git. - - git commit - -You will be presented with an editor where you need to type the commit -message. Writing a good commit message is an art in itself. You should take a -look at the links below. We try to follow these conventions as much as -possible. - - * [How to Write a Git Commit Message](http://chris.beams.io/posts/git-commit/) - * [A Note About Git Commit Messages](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html) - * [stopwritingramblingcommitmessages](http://stopwritingramblingcommitmessages.com/) - -### Sending Changes on a Review - -After finishing your work on your feature branch, you will want to send it on -code review. This is done by pushing the branch to Github and creating a pull -request. You can find all PRs -[here](https://github.com/memgraph/memgraph/pulls). - -### Code Integration - -When working, you have to integrate some changes to your work or push your work -to be available for others. To pull changes into a local `branch`, usually run -the following: - - git checkout {{branch}} - git pull origin {{branch}} - -To push your changes, usually run the following: - - git checkout {{branch}} - git push origin {{branch}} - -Sometimes, things could get a little bit more complicated. Diagram below shows -which git operation should be performed if a piece of code has to be integrated -from one branch to another. Note, `main_branch` is the **master** branch in our -case. - -``` - |<---------------------------| - | squash merge | - |--------------------------->| - | merge | - | | - |<-----------|<--------------| - | merge | squash merge | - | | | - |----------->|-------------->| - | rebase | merge | - | | rebase --onto | - | | | - main_branch epic_branch task_branch -``` - -There are a couple of cases: - -* If a code has to be integrated from a task branch to the main branch, use - **squash merge**. While you were working on a task, you probably committed a -couple of cleanup commits that are not relevant to the main branch. In the -other direction, while integrating the main branch to a task branch, the -**regular merge** is ok because changes from the task branch will later be -squash merged. - -* You should use **squash merge** when integrating changes from task to epic - branch (task might have irrelevant commits). On the other hand, you should -use a **regular merge** when an epic is completed and has to be integrated into -the main branch. Epic is a more significant piece of work, decoupled in -compilable and testable commits. All these commits should be preserved to be -able to find potential issues later on. - -* You should use **rebase** when integrating changes from main to an epic - branch. The epic branch has to be as clean as possible, avoid pure merge -commits. Once you rebase epic on main, all commits on the epic branch will -change the hashes. The implications are: 1) you have to force push your local -branch to the origin, 2) if you made a task branch out of the epic branch, you -would have to use **rebase --onto** (please refer to `git help rebase` for -details). In simple cases, **regular merge** should be sufficient to integrate -changes from epic to a task branch (that can even be done via GitHub web -interface). - -During any code integration, you may get reports that some files have -conflicting changes. If you need help resolving them, don't be afraid to ask -around! After you've resolved them, mark them as done with `git add` command. -You may then continue with `git {{action}} --continue`. diff --git a/docs/feature_spec/active/python-query-modules.md b/docs/feature_spec/active/python-query-modules.md deleted file mode 100644 index f70d4eac0..000000000 --- a/docs/feature_spec/active/python-query-modules.md +++ /dev/null @@ -1,185 +0,0 @@ -# Python 3 Query Modules - -## Introduction - -Memgraph exposes a C API for writing the so called Query Modules. These -modules contain definitions of procedures which can be invoked through the -query language using the `CALL ... YIELD ...` syntax. This mechanism allows -database users to extend Memgraph with their own algorithms and -functionalities. - -Using a low level language like C can be quite cumbersome for writing modules, -so it seems natural to add support for a higher level language on top of the -existing C API. - -There are languages written exactly for this purpose of extending C with high -level constructs, for example Lua and Guile. Instead of those, we have chosen -Python 3 to be the first high level language we will support. The primary reason -being that it's very popular, so more people should be able to write modules. -Another benefit of Python which comes out of its popularity is the large -ecosystem of libraries, especially graph algorithm related ones like NetworkX. -Python does have significant performance and implementation downsides compared -to Lua and Guile, but these are described in more detail later in this -document. - -## Python 3 API Overview - -The Python 3 API should be as user friendly as possible as well as look -Pythonic. This implies that some functions from the C API will not map to the -exact same functions. The most obvious case for a Pythonic approach is -registering procedures of a query module. Let's take a look at the C example -and its transformation to Python. - -```c -static void procedure(const struct mgp_list *args, - const struct mgp_graph *graph, struct mgp_result *result, - struct mgp_memory *memory); - -int mgp_init_module(struct mgp_module *module, struct mgp_memory *memory) { - struct mgp_proc *proc = - mgp_module_add_read_procedure(module, "procedure", procedure); - if (!proc) return 1; - if (!mgp_proc_add_arg(proc, "required_arg", - mgp_type_nullable(mgp_type_any()))) - return 1; - struct mgp_value *null_value = mgp_value_make_null(memory); - if (!mgp_proc_add_opt_arg(proc, "optional_arg", - mgp_type_nullable(mgp_type_any()), null_value)) { - mgp_value_destroy(null_value); - return 1; - } - mgp_value_destroy(null_value); - if (!mgp_proc_add_result(proc, "result", mgp_type_string())) return 1; - if (!mgp_proc_add_result(proc, "args", - mgp_type_list(mgp_type_nullable(mgp_type_any())))) - return 1; - return 0; -} -``` - -In Python things should be a lot simpler. - -```Python -# mgp.read_proc obtains the procedure name via __name__ attribute of a function. -@mgp.read_proc(# Arguments passed to multiple mgp_proc_add_arg calls - (('required_arg', mgp.Nullable(mgp.Any)), ('optional_arg', mgp.Nullable(mgp.Any), None)), - # Result fields passed to multiple mgp_proc_add_result calls - (('result', str), ('args', mgp.List(mgp.Nullable(mgp.Any))))) -def procedure(args, graph, result, memory): - pass -``` - -Here we have replaced `mgp_module_*` and `mgp_proc_*` C API with a much -simpler decorator function in Python -- `mgp.read_proc`. The types of -arguments and result fields can both be our types as well as Python builtin -types which can map to supported `mgp_value` types. The expected builtin types -we ought to support are: `bool`, `str`, `int`, `float` and `map`. While the -rest of the types are provided via our Python API. Optionally, we can add -convenience support for `object` type which would map to -`mgp.Nullable(mgp.Any)` and `list` which would map to -`mgp.List(mgp.Nullable(mgp.Any))`. Also, it makes sense to take a look if we -can leverage Python's `typing` module here. - -Another Pythonic change is to remove `mgp_value` C API from Python altogether. -This means that the arguments a Python procedure receives are not `mgp_value` -instances but rather `PyObject` instances. In other words, our implementation -would immediately marshal `mgp_value` to corresponding type in Python. -Obviously we would need to provide our own Python types for non-builtin -things like `mgp.Vertex` (equivalent to `mgp_vertex`) and other. - -Continuing from our example above, let's say the procedure was invoked through -Cypher using the following query. - - MATCH (n) CALL py_module.procedure(42, n) YIELD *; - -The Python procedure could then do the following and complete without throwing -neither the AssertionError nor the ValueError. - -```Python -def procedure(args, graph, result, memory): - assert isinstance(args, list) - # Unpacking throws ValueError if args does not contain exactly 2 values. - required_arg, optional_arg = args - assert isintance(required_arg, int) - assert isinstance(optional_arg, mgp.Vertex) -``` - -The rest of the C API should naturally map to either top level functions or -class methods as appropriate. - -## Loading Python Query Modules - -Our current mechanism for loading the modules is to look for `.so` files in -the directory specified by `--query-modules` flag. This is done when Memgraph -is started. We can extend this mechanism to look for `.py` files in addition -to `.so` files in the same directory and import them in the embedded Python -interpreter. The only issue is embedding the interpreter in Memgraph. There -are multiple choices: - - 1. Building Memgraph and statically linking to Python. - 2. Building Memgraph and dynamically linking to Python, and distributing - Python with Memgraph's installation. - 3. Building Memgraph and dynamically linking to Python, but without - distributing the Python library. - 4. Building Memgraph and optionally loading Python library by trying to - `dlopen` it. - -The first two options are only viable if the Python license allows, and this -will need further investigation. - -The third option adds Python as an installation dependency for Memgraph, and -without it Memgraph will not run. This is problematic for users which cannot -or do not want to install Python 3. - -The fourth option avoids all of the issues present in the first 3 options, but -comes at a higher implementation cost. We would need to try to `dlopen` the -Python library and setup function pointers. If we succeed we would import -`.py` files from the `--query-modules` directory. On the other hand, if the -user does not have Python, `dlopen` would fail and Memgraph would run without -Python support. - -After live discussion, we've decided to go with option 3. This way we don't -have to worry about mismatching Python versions we support and what the users -expect. Also, we should target Python 3.5 as that should be common between -Debian and CentOS for which we ship installation packages. - -## Performance and Implementation Problems - -As previously mentioned, embedding Python introduces usability issues compared -to other embeddable languages. - -The first, major issue is Global Interpreter Lock (GIL). Initializing Python -will start a single global interpreter and running multiple threads will -require acquiring GIL. In practice, this means that when multiple users run a -procedure written in Python in parallel the execution will not actually be -parallel. Python's interpreter will jump between executing one user's -procedure and the other's. This can be quite an issue for long running -procedures when multiple users are querying Memgraph. The solution for this -issue is Python's API for sub-interpreters. Unfortunately, the support for -them is rather poor and the API contains a lot of critical bugs when we tried -to use them. For the time being, we will have to accept GIL and its downsides. -Perhaps in the future we will gain more knowledge on how we could reduce the -acquire rate of GIL or the sub-interpreter API will get improved. - -Another major issue is memory allocation. Python's C API does not have support -for setting up a temporary allocator during execution of a single function. -It only has support for setting up a global heap allocator. This obviously -impacts our control of memory during a query procedure invocation. Besides -potential performance penalty, a procedure could allocate much more memory -than we would actually allow for execution of a single query. This means that -options controlling the memory limit during query execution are useless. On -the bright side, Python does use block style allocators and reference -counting, so the performance penalty and global memory usage should not be -that terrible. - -The final issue that isn't as major as the ones above is the global state of -the interpreter. In practice this means that any registered procedure and -imported module has access to any other procedure and module. This may pollute -the namespace for other users, but it should not be much of a problem because -Python always has things under a module scope. The other, slightly bigger -downside is that a malicious user could use this knowledge to modify other -modules and procedures. This seems like a major issue, but if we take the -bigger picture into consideration, we already have a security issue in general -by invoking `dlopen` on `.so` and potentially running arbitrary code. This was -the trade off we chose to allow users to extend Memgraph. It's up to the users -to write sane extensions and protect their servers from access. diff --git a/docs/feature_spec/active/tensorflow-op.md b/docs/feature_spec/active/tensorflow-op.md deleted file mode 100644 index ddabc34e0..000000000 --- a/docs/feature_spec/active/tensorflow-op.md +++ /dev/null @@ -1,198 +0,0 @@ -# Tensorflow Op - Technicalities - -The final result should be a shared object (".so") file that can be dynamically -loaded by the Tensorflow runtime in order to directly access the bolt client. - -## About Tensorflow - -Tensorflow is usually used with Python such that the Python code is used to -define a directed acyclic computation graph. Basically no computation is done -in Python. Instead, values from Python are copied into the graph structure as -constants to be used by other Ops. The directed acyclic graph naturally ends up -with two sets of border nodes, one for inputs, one for outputs. These are -sometimes called "feeds". - -Following the Python definition of the graph, during training, the entire data -processing graph/pipeline is called from Python as a single expression. This -leads to lazy evaluation since the called result has already been defined for a -while. - -Tensorflow internally works with tensors, i.e. n-dimensional arrays. That means -all of its inputs need to be matrices as well as its outputs. While it is -possible to feed data directly from Python's numpy matrices straight into -Tensorflow, this is less desirable than using the Tensorflow data API (which -defines data input and processing as a Tensorflow graph) because: - - 1. The data API is written in C++ and entirely avoids Python and as such is - faster - 2. The data API, unlike Python is available in "Tensorflow serving". The - default way to serve Tensorflow models in production. - -Once the entire input pipeline is defined via the tf.data API, its input is -basically a list of node IDs the model is supposed to work with. The model, -through the data API knows how to connect to Memgraph and execute openCypher -queries in order to get the remaining data it needs. (For example features of -neighbouring nodes.) - -## The Interface - -I think it's best you read the official guide... - And especially the addition -that specifies how data ops are special - - -## Compiling the TF Op - -There are two options for compiling a custom op. One of them involves pulling -the TF source, adding your code to it and compiling via bazel. This is -probably awkward to do for us and would significantly slow down compilation. - -The other method involves installing Tensorflow as a Python package and pulling -the required headers from for example: -`/usr/local/lib/python3.6/site-packages/tensorflow/include` We can then compile -our Op with our regular build system. - -This is practical since we can copy the required headers to our repo. If -necessary, we can have several versions of the headers to build several -versions of our Op for every TF version which we want to support. (But this is -unlikely to be required as the API should be stable). - -## Example for Using the Bolt Client Tensorflow Op - -### Dynamic Loading - -``` python3 -import tensorflow as tf - -mg_ops = tf.load_op_library('/usr/bin/memgraph/tensorflow_ops.so') -``` - -### Basic Usage - -``` python3 -dataset = mg_ops.OpenCypherDataset( - # This is probably unfortunate as the username and password - # get hardcoded into the graph, but for the simple case it's fine - "hostname:7687", auth=("user", "pass"), - - # Our query - ''' - MATCH (n:Train) RETURN n.id, n.features - ''', - - # Cast return values to these types - (tf.string, tf.float32)) - -# Some Tensorflow data api boilerplate -iterator = dataset.make_one_shot_iterator() -next_element = iterator.get_next() - -# Up to now we have only defined our computation graph which basically -# just connects to Memgraph -# `next_element` is not really data but a handle to a node in the Tensorflow -# graph, which we can and do evaluate -# It is a Tensorflow tensor with shape=(None, 2) -# and dtype=(tf.string, tf.float) -# shape `None` means the shape of the tensor is unknown at definition time -# and is dynamic and will only be known once the tensor has been evaluated - -with tf.Session() as sess: - node_ids = sess.run(next_element) - # `node_ids` contains IDs and features of all the nodes - # in the graph with the label "Train" - # It is a numpy.ndarray with a shape ($n_matching_nodes, 2) -``` - -### Memgraph Client as a Generic Tensorflow Op - -Other than the Tensorflow Data Op, we'll want to support a generic Tensorflow -Op which can be put anywhere in the Tensorflow computation Graph. It takes in -an arbitrary tensor and produces a tensor. This would be used in the GraphSage -algorithm to fetch the lowest level features into Tensorflow - -```python3 -requested_ids = np.array([1, 2, 3]) -ids_placeholder = tf.placeholder(tf.int32) - -model = mg_ops.OpenCypher() - "hostname:7687", auth=("user", "pass"), - """ - UNWIND $node_ids as nid - MATCH (n:Train {id: nid}) - RETURN n.features - """, - - # What to call the input tensor as an openCypher parameter - parameter_name="node_ids", - - # Type of our resulting tensor - dtype=(tf.float32) -) - -features = model(ids_placeholder) - -with tf.Session() as sess: - result = sess.run(features, - feed_dict={ids_placeholder: requested_ids}) -``` - -This is probably easier to implement than the Data Op, so it might be a good -idea to start with. - -### Production Usage - -During training, in the GraphSage algorithm at least, Memgraph is at the -beginning and at the end of the Tensorflow computation graph. At the -beginning, the Data Op provides the node IDs which are fed into the generic -Tensorflow Op to find their neighbours and their neighbours and their features. - -Production usage differs in that we don't use the Data Op. The Data Op is -effectively cut off and the initial input is fed by Tensorflow serving, with -the data found in the request. - -For example a JSON request to classify a node might look like: - -`POST http://host:port/v1/models/GraphSage/versions/v1:classify` - -With the contents: - -```json -{ - "examples": [ - {"node_id": 1}, - {"node_id": 2} - ], -} -``` - -Every element of the "examples" list is an example to be computed. Each is -represented by a dict with keys matching names of feeds in the Tensorflow graph -and values being the values we want fed in for each example. - -The REST API then replies in kind with the classification result in JSON. - -Note about adding our custom Op to Tensorflow serving. Our Ops .so can be -added into the Bazel build to link with Tensorflow serving or it can be -dynamically loaded by starting Tensorflow serving with a flag -`--custom_op_paths`. - -### Considerations - -There might be issues here that the url to connect to Memgraph is hardcoded -into the op and would thus be wrong when moved to production, requiring some -type of a hack to make work. We probably want to solve this by having the -client op take in another tf.Variable as an input which would contain a -connection url and username/password. We have to research whether this makes -it easy enough to move to production, as the connection string variable is -still a part of the graph, but maybe easier to replace. - -It is probably the best idea to utilize openCypher parameters to make our -queries flexible. The exact API as to how to declare the parameters in Python -is open to discussion. - -The Data Op might not even be necessary to implement as it is not key for -production use. It can be replaced in training mode with feed dicts and either - - 1. Getting the initial list of nodes via a Python Bolt client - 2. Creating a separate Tensorflow computation graph that gets all the relevant - node IDs into Python diff --git a/docs/feature_spec/contents.md b/docs/feature_spec/contents.md deleted file mode 100644 index 58afc346c..000000000 --- a/docs/feature_spec/contents.md +++ /dev/null @@ -1,33 +0,0 @@ -# Feature Specifications - -## Active - -* [Python Query Modules](active/python-query-modules.md) -* [Tensorflow Op](active/tensorflow-op.md) - -## Draft - -* [A-star Variable-length Expand](draft/a-star-variable-length-expand.md) -* [Cloud-native Graph Store](draft/cloud-native-graph-store.md) -* [Compile Filter Expressions](draft/compile-filter-expressions.md) -* [Database Triggers](draft/database-triggers.md) -* [Date and Time Data Types](draft/date-and-time-data-types.md) -* [Distributed Query Execution](draft/distributed-query-execution.md) -* [Edge Create or Update Queries](draft/edge-create-or-update-queries.md) -* [Extend Variable-length Filter Expressions](draft/extend-variable-length-filter-expression.md) -* [Geospatial Data Types](draft/geospatial-data-types.md) -* [Hybrid Storage Engine](draft/hybrid-storage-engine.md) -* [Load Data Queries](draft/load-data-queries.md) -* [Multitenancy](draft/multitenancy.md) -* [Query Compilation](draft/query-compilation.md) -* [Release Log Levels](draft/release-log-levels.md) -* [Rust Query Modules](draft/rust-query-modules.md) -* [Sharded Graph Store](draft/sharded-graph-store.md) -* [Storage Memory Management](draft/storage-memory-management.md) -* [Vectorized Query Execution](draft/vectorized-query-execution.md) - -## Obsolete - -* [Distributed](obsolete/distributed.md) -* [High-availability](obsolete/high-availability.md) -* [Kafka Integration](obsolete/kafka-integration.md) diff --git a/docs/feature_spec/draft/a-star-variable-length-expand.md b/docs/feature_spec/draft/a-star-variable-length-expand.md deleted file mode 100644 index 98667104b..000000000 --- a/docs/feature_spec/draft/a-star-variable-length-expand.md +++ /dev/null @@ -1,15 +0,0 @@ -# A-star Variable-length Expand - -Like DFS/BFS/WeightedShortestPath, it should be possible to support the A-star -algorithm in the format of variable length expansion. - -Syntactically, the query should look like the following one: -``` -MATCH (start)-[ - *aStar{{hops}} {{heuristic_expression} {{weight_expression}} {{aggregated_weight_variable}} {{filtering_expression}} - ]-(end) -RETURN {{aggregated_weight_variable}}; -``` - -It would be convenient to add geospatial data support before because A-star -works well with geospatial data (heuristic function might exist). diff --git a/docs/feature_spec/draft/cloud-native-graph-store.md b/docs/feature_spec/draft/cloud-native-graph-store.md deleted file mode 100644 index 038f92cb8..000000000 --- a/docs/feature_spec/draft/cloud-native-graph-store.md +++ /dev/null @@ -1,7 +0,0 @@ -# Cloud-native Graph Store - -The biggest problem with the current in-memory storage is the total cost of -ownership for large datasets non-frequently updated. An idea to solve that is a -decoupled storage and compute inside a cloud environment. E.g., on AWS, a -database instance could use EC2 machines to run the query execution against -data stored inside S3. diff --git a/docs/feature_spec/draft/compile-filter-expressions.md b/docs/feature_spec/draft/compile-filter-expressions.md deleted file mode 100644 index b3b4879f7..000000000 --- a/docs/feature_spec/draft/compile-filter-expressions.md +++ /dev/null @@ -1,40 +0,0 @@ -# Compile Filter Expressions - -Memgraph evaluates filter expression by traversing the abstract syntax tree of -the given filter. Filtering is a general operation in query execution. - -Some simple examples are: -``` -MATCH (n:Person {name: "John"}) WHERE n.age > 20 AND n.age < 40 RETURN n; -MATCH (a {id: 723})-[*bfs..10 (e, n | e.x > 12 AND n.y < 3)]-() RETURN *; -``` - -More real-world example looks like this (Ethereum network analysis): -``` -MATCH (a: Address {addr: ''})-[]->(t: Transaction)-[]->(b: Address) -RETURN DISTINCT b.addr -UNION -MATCH (a: Address {addr: ''})-[]->(t: Transaction)-[]->(b1: Address)-[]->(t2: Transaction)-[]->(b: Address) -WHERE t2.timestamp > t.timestamp -RETURN DISTINCT b.addr -UNION -MATCH (a: Address {addr: ''})-[]->(t: Transaction)-[]->(b1: Address)-[]->(t2: Transaction)-[]->(b2: Address)-[]->(t3: Transaction)-[]->(b: Address) -WHERE t2.timestamp > t.timestamp AND t3.timestamp > t2.timestamp -return distinct b.addr -UNION -MATCH (a: Address {addr: ''})-[]->(t: Transaction)-[]->(b1: Address)-[]->(t2: Transaction)-[]->(b2: Address)-[]->(t3: Transaction)-[]->(b3: Address)-[]->(t4: Transaction)-[]->(b: Address) -WHERE t2.timestamp > t.timestamp AND t3.timestamp > t2.timestamp AND t4.timestamp > t3.timestamp -RETURN DISTINCT b.addr -UNION -MATCH (a: Address {addr: ''})-[]->(t: Transaction)-[]->(b1: Address)-[]->(t2: Transaction)-[]->(b2: Address)-[]->(t3: Transaction)-[]->(b3: Address)-[]->(t4: Transaction)-[]->(b4: Address)-[]->(t5: Transaction)-[]->(b: Address) -WHERE t2.timestamp > t.timestamp AND t3.timestamp > t2.timestamp AND t4.timestamp > t3.timestamp AND t5.timestamp > t4.timestamp -RETURN DISTINCT b.addr; -``` - -Filtering may take a significant portion of query execution, which means it has -to be fast. - -The first step towards improvement might be to expose an API under which a -developer can implement its filtering logic (it's OK to support only C++ in the -beginning). Later on, we can introduce an automatic compilation of filtering -expressions. diff --git a/docs/feature_spec/draft/database-triggers.md b/docs/feature_spec/draft/database-triggers.md deleted file mode 100644 index 6ea613609..000000000 --- a/docs/feature_spec/draft/database-triggers.md +++ /dev/null @@ -1,14 +0,0 @@ -# Database Triggers - -Memgraph doesn't have any built-in notification mechanism yet. In the case a -user wants to get notified about anything happening inside Memgraph, the only -option is some pull mechanism from the client code. In many cases, that might -be suboptimal. - -A natural place to start would be put to some notification code on each update -action inside Memgraph. It's probably too early to send a notification -immediately after WAL delta gets created, but at some point after transaction -commits or after WAL deltas are written to disk might be a pretty good place. -Furthermore, Memgraph has the query module infrastructure. The first -implementation might call a user-defined query module procedure and pass -whatever gets created or updated during the query execution. diff --git a/docs/feature_spec/draft/date-and-time-data-types.md b/docs/feature_spec/draft/date-and-time-data-types.md deleted file mode 100644 index c4990c584..000000000 --- a/docs/feature_spec/draft/date-and-time-data-types.md +++ /dev/null @@ -1,13 +0,0 @@ -# Date and Time Data Types - -Neo4j offers the following functionality: - -* https://neo4j.com/docs/cypher-manual/current/syntax/temporal/ -* https://neo4j.com/docs/cypher-manual/current/functions/temporal/ - -The question is, how are we going to support equivalent capabilities? We need -something very similar because these are, in general, very well defined types. - -A note about the storage is that Memgraph has a limit on the total number of -different data types, 16 at this point. We have to be mindful of that during -the design phase. diff --git a/docs/feature_spec/draft/distributed-query-execution.md b/docs/feature_spec/draft/distributed-query-execution.md deleted file mode 100644 index 5eacdd0f8..000000000 --- a/docs/feature_spec/draft/distributed-query-execution.md +++ /dev/null @@ -1,10 +0,0 @@ -# Distributed Query Execution - -Add the ability to execute graph algorithms on a cluster of machines. The scope -of this is ONLY the query execution without changing the underlying storage -because that's much more complex. The first significant decision here is to -figure out do we implement our own distributed execution engine or deploy -something already available, like [Giraph](https://giraph.apache.org). An -important part is that Giraph by itself isn't enough because people want to -update data on the fly. The final solution needs to provide some updating -capabilities. diff --git a/docs/feature_spec/draft/edge-create-or-update-queries.md b/docs/feature_spec/draft/edge-create-or-update-queries.md deleted file mode 100644 index fb52c84c0..000000000 --- a/docs/feature_spec/draft/edge-create-or-update-queries.md +++ /dev/null @@ -1,14 +0,0 @@ -# Edge Create or Update Queries - -The old semantic of the `MERGE` clause is quite tricky. The new semantic of -`MERGE` is explained -[here](https://blog.acolyer.org/2019/09/18/updating-graph-databases-with-cypher/). - -Similar to `MERGE`, but maybe simpler is to define clauses and semantics that -apply only to a single edge. In the case an edge between two nodes doesn't -exist, it should be created. On the other hand, if it exists, it should be -updated. The syntax should look similar to the following: - -``` -MERGE EDGE (a)-[e:Type {props}]->(b) [ON CREATE SET expression ON UPDATE SET expression] ... -``` diff --git a/docs/feature_spec/draft/extend-variable-length-filter-expression.md b/docs/feature_spec/draft/extend-variable-length-filter-expression.md deleted file mode 100644 index 5c9bdcd65..000000000 --- a/docs/feature_spec/draft/extend-variable-length-filter-expression.md +++ /dev/null @@ -1,12 +0,0 @@ -# Extend Variable-length Filter Expressions - -Variable-length filtering (DFS/BFS/WeightedShortestPath) can to be arbitrarily -complex. At this point, the filtering expression only gets currently visited -node and edge: - -``` -MATCH (a {id: 723})-[*bfs..10 (e, n | e.x > 12 AND n.y < 3)]-() RETURN *; -``` - -If a user had the whole path available, he would write more complex filtering -logic. diff --git a/docs/feature_spec/draft/geospatial-data-types.md b/docs/feature_spec/draft/geospatial-data-types.md deleted file mode 100644 index 38bff5cb9..000000000 --- a/docs/feature_spec/draft/geospatial-data-types.md +++ /dev/null @@ -1,28 +0,0 @@ -# Geospatial Data Types - -Neo4j offers the following functionality: - -* https://neo4j.com/docs/cypher-manual/current/syntax/spatial/ -* https://neo4j.com/docs/cypher-manual/current/functions/spatial/ - -The question is, how are we going to support equivalent capabilities? We need -something very similar because these are, in general, very well defined types. - -The main reasons for implementing this feature are: - 1. Ease of use. At this point, users have to encode/decode time data types - manually. - 2. Memory efficiency in some cases because user defined encoding could still - be more efficient. - -The number of functionalities that could be built on top of geospatial types is -huge. Probably some C/C++ libraries should be used: - * https://github.com/OSGeo/gdal. - * http://geostarslib.sourceforge.net/ Furthermore, the query engine could use - these data types during query execution (specific for query execution). - * https://www.cgal.org. -Also, the storage engine could have specialized indices for these types of -data. - -A note about the storage is that Memgraph has a limit on the total number of -different data types, 16 at this point. We have to be mindful of that during -the design phase. diff --git a/docs/feature_spec/draft/hybrid-storage-engine.md b/docs/feature_spec/draft/hybrid-storage-engine.md deleted file mode 100644 index 0a4130393..000000000 --- a/docs/feature_spec/draft/hybrid-storage-engine.md +++ /dev/null @@ -1,20 +0,0 @@ -# Hybrid Storage Engine - -The goal here is easy to improve Memgraph storage massively! Please take a look -[here](http://cidrdb.org/cidr2020/papers/p29-neumann-cidr20.pdf) for the -reasons. - -The general idea is to store edges on disk by using an LSM like data structure. -Storing edge properties will be tricky because strict schema also has to be -introduced. Otherwise, it's impossible to store data on disk optimally (Neo4j -already has a pretty optimized implementation of that). Furthermore, we have to -introduce the paging concept. - -This is a complex feature because various aspects of the core engine have to be -considered and probably updated (memory management, garbage collection, -indexing). - -## References - -* [On Disk IO, Part 3: LSM Trees](https://medium.com/databasss/on-disk-io-part-3-lsm-trees-8b2da218496f) -* [2020-04-13 On-disk Edge Store Research](https://docs.google.com/document/d/1avoR2g9dNWa4FSFt9NVn4JrT6uOAH_ReNeUoNVsJ7J4) diff --git a/docs/feature_spec/draft/load-data-queries.md b/docs/feature_spec/draft/load-data-queries.md deleted file mode 100644 index 3206674cd..000000000 --- a/docs/feature_spec/draft/load-data-queries.md +++ /dev/null @@ -1,17 +0,0 @@ -# Load Data Queries - -Loading data into Memgraph is a challenging task. We have to implement -something equivalent to the [Neo4j LOAD -CSV](https://neo4j.com/developer/guide-import-csv/#import-load-csv). This -feature seems relatively straightforward to implement because `LoadCSV` could -be another operator that would yield row by row. By having the operator, the -operation would be composable with the rest of the `CREATE`|`MERGE` queries. -The composability is the key because users would be able to combine various -clauses to import data. - -A more general concept is [SingleStore -Pipelines](https://docs.singlestore.com/v7.1/reference/sql-reference/pipelines-commands/create-pipeline). - -We already tried with [Graph Streams](../obsolete/kafka-integration.md). An option -is to migrate that code as a standalone product -[here](https://github.com/memgraph/mgtools). diff --git a/docs/feature_spec/draft/multitenancy.md b/docs/feature_spec/draft/multitenancy.md deleted file mode 100644 index 02a7a189e..000000000 --- a/docs/feature_spec/draft/multitenancy.md +++ /dev/null @@ -1,15 +0,0 @@ -# Multitenancy - -[Multitenancy](https://en.wikipedia.org/wiki/Multitenancy) is a feature mainly -in the domain of ease of use. Neo4j made a great move by introducing -[Fabric](https://neo4j.com/developer/multi-tenancy-worked-example). - -Memgraph first step in a similar direction would be to add an abstraction layer -containing multiple `Storage` instances + the ability to specify a database -instance per client session or database transaction. - -## Replication Context - -Each transaction has to encode on top of which database it's getting executed. -Once a replica gets delta objects containing database info, the replica engine -could apply changes locally. diff --git a/docs/feature_spec/draft/query-compilation.md b/docs/feature_spec/draft/query-compilation.md deleted file mode 100644 index bc25b4745..000000000 --- a/docs/feature_spec/draft/query-compilation.md +++ /dev/null @@ -1,14 +0,0 @@ -# Query Compilation - -Memgraph supports the interpretation of queries in a pull-based way. An -advantage of interpreting queries is a fast time until the execution, which is -convenient when a user wants to test a bunch of queries in a short time. The -downside is slow runtime. The runtime could be improved by compiling query -plans. - -## Research Area 1 - -The easiest route to the query compilation might be generating [virtual -constexpr](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1064r0.html) -pull functions, making a dynamic library out of the entire compiled query plan, -and swapping query plans during the database runtime. diff --git a/docs/feature_spec/draft/release-log-levels.md b/docs/feature_spec/draft/release-log-levels.md deleted file mode 100644 index f944d3417..000000000 --- a/docs/feature_spec/draft/release-log-levels.md +++ /dev/null @@ -1,17 +0,0 @@ -# Release Log Levels - -It's impossible to control the log level in Memgraph Community. That means it's -tough to debug issues in interacting with Memgraph. At least three log levels -should be available to the user: - -* Log nothing (as it is now). -* Log each executed query. -* Log Bolt server states. - -Memgraph Enterprise has the audit log feature. The audit log provides -additional info about each query (user, source, etc.), but it's only available -in the Enterprise edition. Furthermore, the intention of audit logs isn't -debugging. - -An important note is that the logged queries should be stripped out because, in -the Memgraph cloud context, we shouldn't log sensitive data. diff --git a/docs/feature_spec/draft/rust-query-modules.md b/docs/feature_spec/draft/rust-query-modules.md deleted file mode 100644 index 160056283..000000000 --- a/docs/feature_spec/draft/rust-query-modules.md +++ /dev/null @@ -1,15 +0,0 @@ -# Rust Query Modules - -Memgraph provides the query modules infrastructure. It's possible to write -query modules in -[C/C++](https://docs.memgraph.com/memgraph/reference-overview/query-modules/c-api) -and -[Python](https://docs.memgraph.com/memgraph/reference-overview/query-modules/python-api). -The problem with C/C++ is that it's very error-prone and time-consuming. -Python's problem is that it's slow and has a bunch of other limitations listed -in the [feature spec](../active/python-query-modules.md). - -On the other hand, Rust is fast and much less error-prone compared to C. It -should be possible to use [bindgen](https://github.com/rust-lang/rust-bindgen) -to generate bindings out of the current C API and write wrapper code for Rust -developers to enjoy. diff --git a/docs/feature_spec/draft/sharded-graph-store.md b/docs/feature_spec/draft/sharded-graph-store.md deleted file mode 100644 index 72ea5e3e6..000000000 --- a/docs/feature_spec/draft/sharded-graph-store.md +++ /dev/null @@ -1,8 +0,0 @@ -# Sharded Graph Store - -Add the ability to shard graph data across machines in a cluster. The scope of -this is ONLY changing to the storage engine. - -## References - -* [Spinner: Scalable Graph Partitioning in the Cloud](https://arxiv.org/pdf/1404.3861.pdf) diff --git a/docs/feature_spec/draft/storage-memory-management.md b/docs/feature_spec/draft/storage-memory-management.md deleted file mode 100644 index 0915e76f1..000000000 --- a/docs/feature_spec/draft/storage-memory-management.md +++ /dev/null @@ -1,13 +0,0 @@ -# Storage Memory Management - -If Memgraph uses too much memory, OS will kill it. There has to be an internal -mechanism to control memory usage. - -Since C++17, polymorphic allocators are an excellent way to inject custom -memory management while having a modular code. Memgraph already uses PMR in the -query execution. Also, refer to [1] on how to start with PMR in the storage -context. - -## Resources - -[1] [PMR: Mistakes Were Made](https://www.youtube.com/watch?v=6BLlIj2QoT8) diff --git a/docs/feature_spec/draft/vectorized-query-execution.md b/docs/feature_spec/draft/vectorized-query-execution.md deleted file mode 100644 index 7cf5cdfcd..000000000 --- a/docs/feature_spec/draft/vectorized-query-execution.md +++ /dev/null @@ -1,9 +0,0 @@ -# Vectorized Query Execution - -Memgraph query engine pulls one by one record during query execution. A more -efficient way would be to pull multiple records in an array. Adding that -shouldn't be complicated, but it wouldn't be advantageous without vectorizing -fetching records from the storage. - -On the query engine level, the array could be part of the frame. In other -words, the frame and the code dealing with the frame has to be changed. diff --git a/docs/feature_spec/obsolete/distributed.md b/docs/feature_spec/obsolete/distributed.md deleted file mode 100644 index 8d91f01bd..000000000 --- a/docs/feature_spec/obsolete/distributed.md +++ /dev/null @@ -1,148 +0,0 @@ -# Distributed Memgraph specs -This document describes reasnonings behind Memgraphs distributed concepts. - -## Distributed state machine -Memgraphs distributed mode introduces two states of the cluster, recovering and -working. The change between states shouldn't happen often, but when it happens -it can take a while to make a transition from one to another. - -### Recovering -This state is the default state for Memgraph when the cluster starts with -recovery flags. If the recovery finishes successfully, the state changes to -working. If recovery fails, the user will be presented with a message that -explains what happened and what are the next steps. - -Another way to enter this state is failure. If the cluster encounters a failure, -the master will enter the Recovering mode. This time, it will wait for all -workers to respond with a message saying they are alive and well, and making -sure they all have consistent state. - -### Working -This state should be the default state of Memgraph most of the time. When in -this state, Memgraph accepts connections from Bolt clients and allows query -execution. - -If distributed execution fails for a transaction, that transaction, and all -other active transactions will be aborted and the cluster will enter the -Recovering state. - -## Durability -One of the important concepts in distributed Memgraph is durability. - -### Cluster configuration -When running Memgraph in distributed mode, the master will store cluster -metadata in a persistent store. If fore some reason the cluster shuts down, -recovering Memgraph from durability files shouldn't require any additional -flags. - -### Database ID -Each new and clean run of Memgraph should generate a new globally unique -database id. This id will associate all files that have persisted with this -run. Adding the database id to snapshots, write-ahead logs and cluster metadata -files ties them a specific Memgraph run, and it makes recovery easier to reason -about. - -When recovering, the cluster won't generate a new id, but will reuse the one -from the snapshot/wal that it was able to recover from. - -### Durability files -Memgraph uses snapshots and write-ahead logs for durability. - -When Memgraph recovers it has to make sure all machines in the cluster recover -to the same recovery point. This is done by finding a common snapshot and -finding common transactions in per-machine available write-ahead logs. - -Since we can not be sure that each machine persisted durability files, we need -to be able to negotiate a common recovery point in the cluster. Possible -durability file failures could require to start the cluster from scratch, -purging everything from storage and recovering from existing durability files. - -We need to ensure that we keep wal files containing information about -transactions between all existing snapshots. This will provide better durability -in the case of a random machine durability file failure, where the cluster can -find a common recovery point that all machines in the cluster have. - -Also, we should suggest and make clear docs that anything less than two -snapshots isn't considered safe for recovery. - -### Recovery -The recovery happens in following steps: -* Master enables worker registration. -* Master recovers cluster metadata from the persisted storage. -* Master waits all required workers to register. -* Master broadcasts a recovery request to all workers. -* Workers respond with with a set of possible recovery points. -* Master finds a common recovery point for the whole cluster. -* Master broadcasts a recovery request with the common recovery point. -* Master waits for the cluster to recover. -* After a successful cluster recovery, master can enter Working state. - -## Dynamic Graph Partitioning (abbr. DGP) - -### Implemented parameters - ---dynamic-graph-partitioner-enabled (If the dynamic graph partitioner should be - enabled.) type: bool default: false (start time) ---dgp-improvement-threshold (How much better should specific node score be - to consider a migration to another worker. This represents the minimal - difference between new score that the vertex will have when migrated - and the old one such that it's migrated.) type: int32 default: 10 - (start time) ---dgp-max-batch-size (Maximal amount of vertices which should be migrated - in one dynamic graph partitioner step.) type: int32 default: 2000 - (start time) - -### Design decisions - -* Each partitioning session has to be a new transaction. -* When and how does an instance perform the moves? - * Periodically. - * Token sharing (round robin, exactly one instance at a time has an - opportunity to perform the moves). -* On server-side serialization error (when DGP receives an error). - -> Quit partitioning and wait for the next turn. -* On client-side serialization error (when end client receives an error). - -> The client should never receive an error because of any - internal operation. - -> For the first implementation, it's good enough to wait until data becomes - available again. - -> It would be nice to achieve that DGP has lower priority than end client - operations. - -### End-user parameters - -* --dynamic-graph-partitioner-enabled (execution time) -* --dgp-improvement-threshold (execution time) -* --dgp-max-batch-size (execution time) -* --dgp-min-batch-size (execution time) - -> Minimum number of nodes that will be moved in each step. -* --dgp-fitness-threshold (execution time) - -> Do not perform moves if partitioning is good enough. -* --dgp-delta-turn-time (execution time) - -> Time between each turn. -* --dgp-delta-step-time (execution time) - -> Time between each step. -* --dgp-step-time (execution time) - -> Time limit per each step. - -### Testing - -The implementation has to provide good enough results in terms of: - * How good the partitioning is (numeric value), aka goodness. - * Workload execution time. - * Stress test correctness. - -Test cases: - * N not connected subgraphs - -> shuffle nodes to N instances - -> run partitioning - -> test perfect partitioning. - * N connected subgraph - -> shuffle nodes to N instance - -> run partitioning - -> test partitioning. - * Take realistic workload (Long Running, LDBC1, LDBC2, Card Fraud, BFS, WSP) - -> measure exec time - -> run partitioning - -> test partitioning - -> measure exec time (during and after partitioning). diff --git a/docs/feature_spec/obsolete/high-availability.md b/docs/feature_spec/obsolete/high-availability.md deleted file mode 100644 index 44cd24d8b..000000000 --- a/docs/feature_spec/obsolete/high-availability.md +++ /dev/null @@ -1,275 +0,0 @@ -# High Availability (abbr. HA) - -## High Level Context - -High availability is a characteristic of a system which aims to ensure a -certain level of operational performance for a higher-than-normal period. -Although there are multiple ways to design highly available systems, Memgraph -strives to achieve HA by elimination of single points of failure. In essence, -this implies adding redundancy to the system so that a failure of a component -does not imply the failure of the entire system. To ensure this, HA Memgraph -implements the [Raft consensus algorithm](https://raft.github.io/). - -Correct implementation of the algorithm guarantees that the cluster will be -fully functional (available) as long as any strong majority of the servers are -operational and can communicate with each other and with clients. For example, -clusters of three or four machines can tolerate the failure of a single server, -clusters of five and six machines can tolerate the failure of any two servers, -and so on. Therefore, we strongly recommend a setup of an odd-sized cluster. - -### Performance Implications - -Internally, Raft achieves high availability by keeping a consistent replicated -log on each server within the cluster. Therefore, we must successfully replicate -a transaction on the majority of servers within the cluster before we actually -commit it and report the result back to the client. This operation represents -a significant performance hit when compared with single node version of -Memgraph. - -Luckily, the algorithm can be tweaked in a way which allows read-only -transactions to perform significantly better than those which modify the -database state. That being said, the performance of read-only operations -is still not going to be on par with single node Memgraph. - -This section will be updated with exact numbers once we integrate HA with -new storage. - -With the old storage, write throughput was almost five times lower than read -throughput (~30000 reads per second vs ~6000 writes per second). - -## User Facing Setup - -### How to Setup HA Memgraph Cluster? - -First, the user needs to install `memgraph_ha` package on each machine -in their cluster. HA Memgraph should be available as a Debian package, -so its installation on each machine should be as simple as: - -```plaintext -dpkg -i /path/to/memgraph_ha_.deb -``` - -After successful installation of the `memgraph_ha` package, the user should -finish its configuration before attempting to start the cluster. - -There are two main things that need to be configured on every node in order for -the cluster to be able to run: - -1. The user has to edit the main configuration file and specify the unique node - ID to each server in the cluster -2. The user has to create a file that describes all IP addresses of all servers - that will be used in the cluster - -The `memgraph_ha` binary loads all main configuration parameters from -`/etc/memgraph/memgraph_ha.conf`. On each node of the cluster, the user should -uncomment the `--server-id=0` parameter and change its value to the `server_id` -of that node. - -The last step before starting the server is to create a `coordination` -configuration file. That file is already present as an example in -`/etc/memgraph/coordination.json.example` and you have to copy it to -`/etc/memgraph/coordination.json` and edit it according to your cluster -configuration. The file contains coordination info consisting of a list of -`server_id`, `ip_address` and `rpc_port` lists. The assumed contents of the -`coordination.json` file are: - -```plaintext -[ - [1, "192.168.0.1", 10000], - [2, "192.168.0.2", 10000], - [3, "192.168.0.3", 10000] -] -``` -Here, each line corresponds to coordination of one server. The first entry is -that server's ID, the second is its IP address and the third is the RPC port it -listens to. This port should not be confused with the port used for client -interaction via the Bolt protocol. - -The `ip_address` entered for each `server_id` *must* match the exact IP address -that belongs to that server and that will be used to communicate to other nodes -in the cluster. The coordination configuration file *must* be identical on all -nodes in the cluster. - -After the user has set the `server_id` on each node in -`/etc/memgraph/memgraph_ha.conf` and provided the same -`/etc/memgraph/coordination.json` file to each node in the cluster, they can -start the Memgraph HA service by issuing the following command on each node in -the cluster: - -```plaintext -systemctl start memgraph_ha -``` - -### How to Configure Raft Parameters? - -All Raft configuration parameters can be controlled by modifying -`/etc/memgraph/raft.json`. The assumed contents of the `raft.json` file are: - -```plaintext -{ - "election_timeout_min": 750, - "election_timeout_max": 1000, - "heartbeat_interval": 100, - "replication_timeout": 20000, - "log_size_snapshot_threshold": 50000 -} -``` - -The meaning behind each entry is demystified in the following table: - -Flag | Description -------------------------------|------------ -`election_timeout_min` | Lower bound for the randomly sampled reelection timer given in milliseconds -`election_timeout_max` | Upper bound for the randomly sampled reelection timer given in milliseconds -`heartbeat_interval` | Time interval between consecutive heartbeats given in milliseconds -`replication_timeout` | Time interval allowed for data replication given in milliseconds -`log_size_snapshot_threshold` | Allowed number of entries in Raft log before its compaction - -### How to Query HA Memgraph via Proxy? - -This chapter describes how to query HA Memgraph using our proxy server. -Note that this is not intended to be a long-term solution. Instead, we will -implement a proper Memgraph HA client which is capable of communicating with -the HA cluster. Once our own client is implemented, it will no longer be -possible to query HA Memgraph using other clients (such as neo4j client). - -The Bolt protocol that is exposed by each Memgraph HA node is an extended -version of the standard Bolt protocol. In order to be able to communicate with -the highly available cluster of Memgraph HA nodes, the client must have some -logic implemented in itself so that it can communicate correctly with all nodes -in the cluster. To facilitate a faster start with the HA cluster we will build -the Memgraph HA proxy binary that communicates with all nodes in the HA cluster -using the extended Bolt protocol and itself exposes a standard Bolt protocol to -the user. All standard Bolt clients (libraries and custom systems) can -communicate with the Memgraph HA proxy without any code modifications. - -The HA proxy should be deployed on each client machine that is used to -communicate with the cluster. It can't be deployed on the Memgraph HA nodes! - -When using the Memgraph HA proxy, the communication flow is described in the -following diagram: - -```plaintext -Memgraph HA node 1 -----+ - | -Memgraph HA node 2 -----+ Memgraph HA proxy <---> any standard Bolt client (C, Java, PHP, Python, etc.) - | -Memgraph HA node 3 -----+ -``` - -To setup the Memgraph HA proxy the user should install the `memgraph_ha_proxy` -package. - -After its successful installation, the user should enter all endpoints of the -HA Memgraph cluster servers into the configuration before attempting to start -the HA Memgraph proxy server. - -The HA Memgraph proxy server loads all of its configuration from -`/etc/memgraph/memgraph_ha_proxy.conf`. Assuming that the cluster is set up -like in the previous examples, the user should uncomment and enter the following -value into the `--endpoints` parameter: - -```plaintext ---endpoints=192.168.0.1:7687,192.168.0.2:7687,192.168.0.3:7687 -``` - -Note that the IP addresses used in the example match the individual cluster -nodes IP addresses, but the ports used are the Bolt server ports exposed by -each node (currently the default value of `7687`). - -The user can now start the proxy by using the following command: - -```plaintext -systemctl start memgraph_ha_proxy -``` - -After the proxy has been started, the user can query the HA cluster by -connecting to the HA Memgraph proxy IP address using their favorite Bolt -client. - -## Integration with Memgraph - -The first thing that should be defined is a single instruction within the -context of Raft (i.e. a single entry in a replicated log). -These instructions should be completely deterministic when applied -to the state machine. We have therefore decided that the appropriate level -of abstraction within Memgraph corresponds to `Delta`s (data structures -which describe a single change to the Memgraph state, used for durability -in WAL). Moreover, a single instruction in a replicated log will consist of a -batch of `Delta`s which correspond to a single transaction that's about -to be **committed**. - -Apart from `Delta`s, there are certain operations within the storage called -`StorageGlobalOperations` which do not conform to usual transactional workflow -(e.g. Creating indices). Since our storage engine implementation guarantees -that at the moment of their execution no other transactions are active, we can -safely replicate them as well. In other words, no additional logic needs to be -implemented because of them. - -Therefore, we will introduce a new `RaftDelta` object which can be constructed -both from storage `Delta` and `StorageGlobalOperation`. Instead of appending -these to WAL (as we do in single node), we will start to replicate them across -our cluster. Once we have replicated the corresponding Raft log entry on -majority of the cluster, we are able to safely commit the transaction or execute -a global operation. If for any reason the replication fails (leadership change, -worker failures, etc.) the transaction will be aborted. - -In the follower mode, we need to be able to apply `RaftDelta`s we got from -the leader when the protocol allows us to do so. In that case, we will use the -same concepts from durability in storage v2, i.e., applying deltas maps -completely to recovery from WAL in storage v2. - -## Test and Benchmark Strategy - -We have already implemented some integration and stress tests. These are: - -1. leader election -- Tests whether leader election works properly. -2. basic test -- Tests basic leader election and log replication. -3. term updates test -- Tests a specific corner case (which used to fail) - regarding term updates. -4. log compaction test -- Tests whether log compaction works properly. -5. large log entries -- Tests whether we can successfully replicate relatively - large log entries. -6. index test -- Tests whether index creation works in HA. -7. normal operation stress test -- Long running concurrent stress test under - normal conditions (no failures). -8. read benchmark -- Measures read throughput in HA. -9. write benchmark -- Measures write throughput in HA. - -At the moment, our main goal is to pass existing tests and have a stable version -on our stress test. We should also implement a stress test which occasionally -introduces different types of failures in our cluster (we did this kind of -testing manually thus far). Passing these tests should convince us that we have -a "stable enough" version which we can start pushing to our customers. - -Additional (proper) testing should probably involve some ideas from -[here](https://jepsen.io/analyses/dgraph-1-0-2) - -## Possible Future Changes/Improvements/Extensions - -There are two general directions in which we can alter HA Memgraph. The first -direction assumes we are going to stick with the Raft protocol. In that case -there are a few known ways to extend the basic algorithm in order to gain -better performance or achieve extra functionality. In no particular order, -these are: - -1. Improving read performance using leader leases [Section 6.4 from Raft thesis] -2. Introducing cluster membership changes [Chapter 4 from Raft thesis] -3. Introducing a [learner mode](https://etcd.io/docs/v3.3.12/learning/learner/). -4. Consider different log compaction strategies [Chapter 5 from Raft thesis] -5. Removing HA proxy and implementing our own HA Memgraph client. - -On the other hand, we might decide in the future to base our HA implementation -on a completely different protocol which might even offer different guarantees. -In that case we probably need to do a bit more of market research and weigh the -trade-offs of different solutions. -[This](https://www.postgresql.org/docs/9.5/different-replication-solutions.html) -might be a good starting point. - -## Reading materials - -1. [Raft paper](https://raft.github.io/raft.pdf) -2. [Raft thesis](https://github.com/ongardie/dissertation) (book.pdf) -3. [Raft playground](https://raft.github.io/) -4. [Leader Leases](https://blog.yugabyte.com/low-latency-reads-in-geo-distributed-sql-with-raft-leader-leases/) -5. [Improving Raft ETH](https://pub.tik.ee.ethz.ch/students/2017-FS/SA-2017-80.pdf) diff --git a/docs/feature_spec/obsolete/kafka-integration.md b/docs/feature_spec/obsolete/kafka-integration.md deleted file mode 100644 index f3b9af447..000000000 --- a/docs/feature_spec/obsolete/kafka-integration.md +++ /dev/null @@ -1,117 +0,0 @@ -# Kafka Integration - -## openCypher clause - -One must be able to specify the following when importing data from Kafka: - -* Kafka URI -* Kafka topic -* Transform [script](transform.md) URI - - -Minimum required syntax looks like: -```opencypher -CREATE STREAM stream_name AS LOAD DATA KAFKA 'URI' - WITH TOPIC 'topic' - WITH TRANSFORM 'URI'; -``` - - -The full openCypher clause for creating a stream is: -```opencypher -CREATE STREAM stream_name AS - LOAD DATA KAFKA 'URI' - WITH TOPIC 'topic' - WITH TRANSFORM 'URI' - [BATCH_INTERVAL milliseconds] - [BATCH_SIZE count] -``` -The `CREATE STREAM` clause happens in a transaction. - -`WITH TOPIC` parameter specifies the Kafka topic from which we'll stream -data. - -`WITH TRANSFORM` parameter should contain a URI of the transform script. - -`BATCH_INTERVAL` parameter defines the time interval in milliseconds -which is the time between two successive stream importing operations. - -`BATCH_SIZE` parameter defines the count of Kafka messages that will be -batched together before import. - -If both `BATCH_INTERVAL` and `BATCH_SIZE` parameters are given, the condition -that is satisfied first will trigger the batched import. - -Default value for `BATCH_INTERVAL` is 100 milliseconds, and the default value -for `BATCH_SIZE` is 10; - -The `DROP` clause deletes a stream: -```opencypher -DROP STREAM stream_name; -``` - -The `SHOW` clause enables you to see all configured streams: -```opencypher -SHOW STREAMS; -``` - -You can also start/stop streams with the `START` and `STOP` clauses: -```opencypher -START STREAM stream_name [LIMIT count BATCHES]; -STOP STREAM stream_name; -``` -A stream needs to be stopped in order to start it and it needs to be started in -order to stop it. Starting a started or stopping a stopped stream will not -affect that stream. - -There are also convenience clauses to start and stop all streams: -```opencypher -START ALL STREAMS; -STOP ALL STREAMS; -``` - -Before the actual import, you can also test the stream with the `TEST -STREAM` clause: -```opencypher -TEST STREAM stream_name [LIMIT count BATCHES]; -``` -When a stream is tested, data extraction and transformation occurs, but no -output is inserted in the graph. - -A stream needs to be stopped in order to test it. When the batch limit is -omitted, `TEST STREAM` will run for only one batch by default. - -## Data Transform - -The transform script is a user defined script written in Python. The script -should be aware of the data format in the Kafka message. - -Each Kafka message is byte length encoded, which means that the first eight -bytes of each message contain the length of the message. - -A sample code for a streaming transform script could look like this: - -```python -def create_vertex(vertex_id): - return ("CREATE (:Node {id: $id})", {"id": vertex_id}) - - -def create_edge(from_id, to_id): - return ("MATCH (n:Node {id: $from_id}), (m:Node {id: $to_id}) "\ - "CREATE (n)-[:Edge]->(m)", {"from_id": from_id, "to_id": to_id}) - - -def stream(batch): - result = [] - for item in batch: - message = item.decode('utf-8').strip().split() - if len(message) == 1: - result.append(create_vertex(message[0]))) - else: - result.append(create_edge(message[0], message[1])) - return result - -``` - -The script should output openCypher query strings based on the type of the -records. diff --git a/docs/feature_spec/replication.md b/docs/feature_spec/replication.md deleted file mode 100644 index 218897f03..000000000 --- a/docs/feature_spec/replication.md +++ /dev/null @@ -1,222 +0,0 @@ -# Replication - -## High Level Context - -Replication is a method that ensures that multiple database instances are -storing the same data. To enable replication, there must be at least two -instances of Memgraph in a cluster. Each instance has one of either two roles: -main or replica. The main instance is the instance that accepts writes to the -database and replicates its state to the replicas. In a cluster, there can only -be one main. There can be one or more replicas. None of the replicas will accept -write queries, but they will always accept read queries (there is an exception -to this rule and is described below). Replicas can also be configured to be -replicas of replicas, not necessarily replicas of the main. Each instance will -always be reachable using the standard supported communication protocols. The -replication will replicate WAL data. All data is transported through a custom -binary protocol that will try remain backward compatible, so that replication -immediately allows for zero downtime upgrades. - -Each replica can be configured to accept replicated data in one of the following -modes: - - synchronous - - asynchronous - - semi-synchronous - -### Synchronous Replication - -When the data is replicated to a replica synchronously, all of the data of a -currently pending transaction must be sent to the synchronous replica before the -transaction is able to commit its changes. - -This mode has a positive implication that all data that is committed to the -main will always be replicated to the synchronous replica. It also has a -negative performance implication because non-responsive replicas could grind all -query execution to a halt. - -This mode is good when you absolutely need to be sure that all data is always -consistent between the main and the replica. - -### Asynchronous Replication - -When the data is replicated to a replica asynchronously, all pending -transactions are immediately committed and their data is replicated to the -asynchronous replica in the background. - -This mode has a positive performance implication in which it won't slow down -query execution. It also has a negative implication that the data between the -main and the replica is almost never in a consistent state (when the data is -being changed). - -This mode is good when you don't care about consistency and only need an -eventually consistent cluster, but you care about performance. - -### Semi-synchronous Replication - -When the data is replicated to a replica semi-synchronously, the data is -replicated using both the synchronous and asynchronous methodology. The data is -always replicated synchronously, but, if the replica for any reason doesn't -respond within a preset timeout, the pending transaction is committed and the -data is replicated to the replica asynchronously. - -This mode has a positive implication that all data that is committed is -*mostly* replicated to the semi-synchronous replica. It also has a negative -performance implication as the synchronous replication mode. - -This mode is useful when you want the replication to be synchronous to ensure -that the data within the cluster is consistent, but you don't want the main -to grind to a halt when you have a non-responsive replica. - -### Addition of a New Replica - -Each replica, when added to the cluster (in any mode), will first start out as -an asynchronous replica. That will allow replicas that have fallen behind to -first catch-up to the current state of the database. When the replica is in a -state that it isn't lagging behind the main it will then be promoted (in a brief -stop-the-world operation) to a semi-synchronous or synchronous replica. Slaves -that are added as asynchronous replicas will remain asynchronous. - -## User Facing Setup - -### How to Setup a Memgraph Cluster with Replication? - -Replication configuration is done primarily through openCypher commands. This -allows the cluster to be dynamically rearranged (new leader election, addition -of a new replica, etc.). - -Each Memgraph instance when first started will be a main. You have to change -the role of all replica nodes using the following openCypher query before you -can enable replication on the main: - -```plaintext -SET REPLICATION ROLE TO (MAIN|REPLICA) WITH PORT ; -``` - -Note that the "WITH PORT " part of the query sets the replication port, -but it applies only to the replica. In other words, if you try to set the -replication port as the main, a semantic exception will be thrown. -After you have set your replica instance to the correct operating role, you can -enable replication in the main instance by issuing the following openCypher -command: -```plaintext -REGISTER REPLICA name (SYNC|ASYNC) [WITH TIMEOUT 0.5] TO ; -``` - -The socket address must be a string of the following form: - -```plaintext -"IP_ADDRESS:PORT_NUMBER" -``` - -where IP_ADDRESS is a valid IP address, and PORT_NUMBER is a valid port number, -both given in decimal notation. -Note that in this case they must be separated by a single colon. -Alternatively, one can give the socket address as: - -```plaintext -"IP_ADDRESS" -``` - -where IP_ADDRESS must be a valid IP address, and the port number will be -assumed to be the default one (we specify it to be 10000). - -Each Memgraph instance will remember what the configuration was set to and will -automatically resume with its role when restarted. - - -### How to See the Current Replication Status? - -To see the replication ROLE of the current Memgraph instance, you can issue the -following query: - -```plaintext -SHOW REPLICATION ROLE; -``` - -To see the replicas of the current Memgraph instance, you can issue the -following query: - -```plaintext -SHOW REPLICAS; -``` - -To delete a replica, issue the following query: - -```plaintext -DROP REPLICA 'name'; -``` - -### How to Promote a New Main? - -When you have an already set-up cluster, to promote a new main, just set the -replica that you want to be a main to the main role. - -```plaintext -SET REPLICATION ROLE TO MAIN; # on desired replica -``` - -After the command is issued, if the original main is still alive, it won't be -able to replicate its data to the replica (the new main) anymore and will enter -an error state. You must ensure that at any given point in time there aren't -two mains in the cluster. - -## Limitations and Potential Features - -Currently, we do not support chained replicas, i.e. a replica can't have its -own replica. When this feature becomes available, the user will be able to -configure scenarios such as the following one: - -```plaintext -main -[asynchronous]-> replica 1 -[semi-synchronous]-> replica 2 -``` - -To configure the above scenario, the user will be able to issue the following -commands: -```plaintext -SET REPLICATION ROLE TO REPLICA WITH PORT ; # on replica 1 -SET REPLICATION ROLE TO REPLICA WITH PORT ; # on replica 2 - -REGISTER REPLICA replica1 ASYNC TO "replica1_ip_address:port1"; # on main -REGISTER REPLICA replica2 SYNC WITH TIMEOUT 0.5 - TO "replica2_ip_address:port2"; # on replica 1 -``` - -In addition, we do not yet support advanced recovery mechanisms. For example, -if a main crashes, a suitable replica will take its place as the new main. If -the crashed main goes back online, it will not be able to reclaim its previous -role, but will be forced to be a replica of the new main. -In the upcoming releases, we might be adding more advanced recovery mechanisms. -However, users are able to setup their own recovery policies using the basic -recovery mechanisms we currently provide, that can cover a wide range of -real-life scenarios. - -Also, we do not yet support the replication of authentication configurations, -rendering access control replication unavailable. - -The query and authentication modules, as well as audit logs are not replicated. - -## Integration with Memgraph - -WAL `Delta`s are replicated between the replication main and replica. With -`Delta`s, all `StorageGlobalOperation`s are also replicated. Replication is -essentially the same as appending to the WAL. - -Synchronous replication will occur in `Commit` and each -`StorageGlobalOperation` handler. The storage itself guarantees that `Commit` -will be called single-threadedly and that no `StorageGlobalOperation` will be -executed during an active transaction. Asynchronous replication will load its -data from already written WAL files and transmit the data to the replica. All -data will be replicated using our RPC protocol (SLK encoded). - -For each replica the replication main (or replica) will keep track of the -replica's state. That way, it will know which operations must be transmitted to -the replica and which operations can be skipped. When a replica is very stale, -a snapshot will be transmitted to it so that it can quickly synchronize with -the current state. All following operations will transmit WAL deltas. - -## Reading materials - -1. [PostgreSQL comparison of different solutions](https://www.postgresql.org/docs/12/different-replication-solutions.html) -2. [PostgreSQL docs](https://www.postgresql.org/docs/12/runtime-config-replication.html) -3. [MySQL reference manual](https://dev.mysql.com/doc/refman/8.0/en/replication.html) -4. [MySQL docs](https://dev.mysql.com/doc/refman/8.0/en/replication-setup-slaves.html) -5. [MySQL master switch](https://dev.mysql.com/doc/refman/8.0/en/replication-solutions-switch.html) diff --git a/docs/presentation/latex-template/README.md b/docs/presentation/latex-template/README.md deleted file mode 100644 index d649be4a9..000000000 --- a/docs/presentation/latex-template/README.md +++ /dev/null @@ -1,22 +0,0 @@ -# Memgraph LaTeX Beamer Template - -This folder contains all of the needed files for creating a presentation with -Memgraph styling. You should use this style for any public presentations. - -Feel free to improve it according to style guidelines and raise issues if you -find any. - -## Usage - -Copy the contents of this folder (excluding this README file) to where you -want to write your own presentation. After copying, you can start editing the -`template.tex` with your content. - -To compile the presentation to a PDF, run `latexmk -pdf -xelatex`. Some -directives require XeLaTeX, so you need to pass `-xelatex` as the final option -of `latexmk`. You may also need to install some packages if the compilation -complains about missing packages. - -To clean up the generated files, use `latexmk -C`. This will also delete the -generated PDF. If you wish to remove generated files except the PDF, use -`latexmk -c`. diff --git a/docs/presentation/latex-template/mg-beamer.cls b/docs/presentation/latex-template/mg-beamer.cls deleted file mode 100644 index 2f4569c77..000000000 --- a/docs/presentation/latex-template/mg-beamer.cls +++ /dev/null @@ -1,82 +0,0 @@ -\NeedsTeXFormat{LaTeX2e} -\ProvidesClass{mg-beamer}[2018/03/26 Memgraph Beamer] - -\DeclareOption*{\PassOptionsToClass{\CurrentOption}{beamer}} - -\ProcessOptions \relax - -\LoadClass{beamer} - -\usetheme{Pittsburgh} - -% Memgraph color palette -\definecolor{mg-purple}{HTML}{720096} -\definecolor{mg-red}{HTML}{DD2222} -\definecolor{mg-orange}{HTML}{FB6E00} -\definecolor{mg-yellow}{HTML}{FFC500} -\definecolor{mg-gray}{HTML}{857F87} -\definecolor{mg-black}{HTML}{231F20} - -\RequirePackage{fontspec} -% Title fonts -\setbeamerfont{frametitle}{family={\fontspec[Path = ./mg-style/fonts/]{EncodeSansSemiCondensed-Regular.ttf}}} -\setbeamerfont{title}{family={\fontspec[Path = ./mg-style/fonts/]{EncodeSansSemiCondensed-Regular.ttf}}} -% Body font -\RequirePackage[sfdefault,light]{roboto} -% Roboto is pretty bad for monospace font. We will find a replacement. -% \setmonofont{RobotoMono-Regular.ttf}[Path = ./mg-style/fonts/] - -% Title slide styles -% \setbeamerfont{frametitle}{size=\huge} -% \setbeamerfont{title}{size=\huge} -% \setbeamerfont{date}{size=\tiny} - -% Other typography styles -\setbeamertemplate{frametitle}[default][center] -\setbeamercolor{frametitle}{fg=mg-black} -\setbeamercolor{title}{fg=mg-black} -\setbeamercolor{section in toc}{fg=mg-black} -\setbeamercolor{local structure}{fg=mg-orange} -\setbeamercolor{alert text}{fg=mg-red} - -% Commands -\newcommand{\mgalert}[1]{{\usebeamercolor[fg]{alert text}#1}} -\newcommand{\titleframe}{\frame[plain]{\titlepage}} -\newcommand{\mgtexttt}[1]{{\textcolor{mg-gray}{\texttt{#1}}}} - -% Title slide background -\RequirePackage{tikz,calc} -% Use title-slide-169 if aspect ration is 16:9 -\pgfdeclareimage[interpolate=true,width=\paperwidth,height=\paperheight]{logo}{mg-style/title-slide-169} -\setbeamertemplate{background}{ - \begin{tikzpicture} - \useasboundingbox (0,0) rectangle (\the\paperwidth,\the\paperheight); - \pgftext[at=\pgfpoint{0}{0},left,base]{\pgfuseimage{logo}}; - \ifnum\thepage>1\relax - \useasboundingbox (0,0) rectangle (\the\paperwidth,\the\paperheight); - \fill[white, opacity=1](0,\the\paperheight)--(\the\paperwidth,\the\paperheight)--(\the\paperwidth,0)--(0,0)--(0,\the\paperheight); - \fi - \end{tikzpicture} -} - -% Footline content -\setbeamertemplate{navigation symbols}{}%remove navigation symbols - \setbeamertemplate{footline}{ - \begin{beamercolorbox}[ht=1.6cm,wd=\paperwidth]{footlinecolor} - \vspace{0.1cm} - \hfill - \begin{minipage}[c]{3cm} - \begin{center} - \includegraphics[height=0.8cm]{mg-style/memgraph-logo.png} - \end{center} - \end{minipage} - \begin{minipage}[c]{7cm} - \insertshorttitle\ --- \insertsection - \end{minipage} - \begin{minipage}[c]{2cm} - \tiny{\insertframenumber{} of \inserttotalframenumber} - \end{minipage} - \end{beamercolorbox} -} - -\endinput diff --git a/docs/presentation/latex-template/mg-style/fonts/EncodeSansSemiCondensed-Regular.ttf b/docs/presentation/latex-template/mg-style/fonts/EncodeSansSemiCondensed-Regular.ttf deleted file mode 100644 index 4964bd2b1..000000000 Binary files a/docs/presentation/latex-template/mg-style/fonts/EncodeSansSemiCondensed-Regular.ttf and /dev/null differ diff --git a/docs/presentation/latex-template/mg-style/memgraph-logo.png b/docs/presentation/latex-template/mg-style/memgraph-logo.png deleted file mode 100644 index ed9518596..000000000 Binary files a/docs/presentation/latex-template/mg-style/memgraph-logo.png and /dev/null differ diff --git a/docs/presentation/latex-template/mg-style/title-slide-169.png b/docs/presentation/latex-template/mg-style/title-slide-169.png deleted file mode 100644 index 71a49a9a1..000000000 Binary files a/docs/presentation/latex-template/mg-style/title-slide-169.png and /dev/null differ diff --git a/docs/presentation/latex-template/mg-style/title-slide.png b/docs/presentation/latex-template/mg-style/title-slide.png deleted file mode 100644 index f889c437f..000000000 Binary files a/docs/presentation/latex-template/mg-style/title-slide.png and /dev/null differ diff --git a/docs/presentation/latex-template/template.tex b/docs/presentation/latex-template/template.tex deleted file mode 100644 index 310fe973b..000000000 --- a/docs/presentation/latex-template/template.tex +++ /dev/null @@ -1,40 +0,0 @@ -% Set 16:9 aspect ratio -\documentclass[aspectratio=169]{mg-beamer} -% Default directive sets the regular 4:3 aspect ratio -% \documentclass{mg-beamer} -\mode - -% requires xelatex -\usepackage{ccicons} - -\title{Insert Presentation Title} -\titlegraphic{\ccbyncnd} -\author{Insert Name} - -% Institute doesn't look good in our current styling class. -% \institute[Memgraph Ltd.]{\pgfimage[height=1.5cm]{mg-logo.png}} - -% Date is autogenerated on compilation, so no need to set it explicitly, -% unless you wish to override it with a different date. -% \date{March 23, 2018} - -\begin{document} - -\titleframe - -\section{Intro} - -\begin{frame}{Contents} - \tableofcontents -\end{frame} - -\begin{frame}{Memgraph Markup Test} - \begin{itemize} - \item \mgtexttt{Prefer \\mgtexttt for monospace} - \item Replace this slide with your own - \item Add even more slides in different sections - \item Make sure you spellcheck your presentation - \end{itemize} -\end{frame} - -\end{document} diff --git a/environment/README.md b/environment/README.md index 0090436d1..164de7ff2 100644 --- a/environment/README.md +++ b/environment/README.md @@ -1,44 +1,4 @@ # Memgraph Build and Run Environments -## Toolchain Installation Procedure - -1) Download the toolchain for your operating system from one of the following - links (current active toolchain is `toolchain-v2`): - -* [CentOS 7](https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/toolchain-v2/toolchain-v2-binaries-centos-7.tar.gz) -* [CentOS 8](https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/toolchain-v2/toolchain-v2-binaries-centos-8.tar.gz) -* [Debian 9](https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/toolchain-v2/toolchain-v2-binaries-debian-9.tar.gz) -* [Debian 10](https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/toolchain-v2/toolchain-v2-binaries-debian-10.tar.gz) -* [Ubuntu 18.04](https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/toolchain-v2/toolchain-v2-binaries-ubuntu-18.04.tar.gz) -* [Ubuntu 20.04](https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/toolchain-v2/toolchain-v2-binaries-ubuntu-20.04.tar.gz) - -2) Extract the toolchain with the following command: - -```bash -tar xzvf {{toolchain-archive}}.tar.gz -C /opt -``` - -3) Check and install required toolchain runtime dependencies by executing - (e.g., on **Debian 10**): - -```bash -./environment/os/debian-10.sh check TOOLCHAIN_RUN_DEPS -./environment/os/debian-10.sh install TOOLCHAIN_RUN_DEPS -``` - -4) Activate the toolchain: - -```bash -source /opt/toolchain-v2/activate -``` - -## Toolchain Upgrade Procedure - -1) Build a new toolchain for each supported OS (latest versions). -2) If the new toolchain doesn't compile on some supported OS, the last - compilable toolchain has to be used instead. In other words, the project has - to compile on the oldest active toolchain as well. Suppose some - changes/improvements were added when migrating to the latest toolchain; in - that case, the maintainer has to ensure that the project still compiles on - previous toolchains (everything from `init` script to the actual code has to - work on all supported operating systems). +Please continue +[here](https://www.notion.so/memgraph/Tools-05e0baafb78a49b386e0063b4833d23d).