Compare commits

...

39 Commits

Author SHA1 Message Date
János Benjamin Antal
24fa630279 Add patch 2022-06-10 17:04:49 +02:00
János Benjamin Antal
c651a8d15a Merge branch 'master' into T0804-MG-base-thrift-project 2022-06-08 11:21:53 +02:00
János Benjamin Antal
5b4ef8de99 Trial vmuch 2022-06-08 11:19:40 +02:00
Marko Budiselić
21ad5d4328 Fix SHOW REPLICATION ROLE and SHOW REPLICAS (#376) 2022-05-20 20:17:59 -07:00
Marko Budiselić
8e3ab1ad0f Add frequent replica ping (#380) 2022-05-20 19:29:17 -07:00
Marko Budiselić
cccf32e79d Add default commented values for Bolt SSL flags (#398) 2022-05-18 08:34:07 +02:00
Jure Bajic
22bd60c613 Fix shutdown call (#395)
* Fix shutdown not called

* Add ssl server tests
2022-05-18 07:50:06 +02:00
János Benjamin Antal
8059a3e653 Improve magic functions docs (#391)
* Improve documentation of magic functions

* Improve wording for decorators
2022-05-17 15:33:28 +02:00
Antonio Andelic
a7f4c98bea Fix module symbol loading (#335)
* Use DEEPBIND

* Add dependency on libstdc++

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-05-13 11:43:10 +02:00
Jure Bajic
483f4d04bd Update precommit hooks (#393)
* Add pre-commit

* Add clang pre-commit

* Install pre-commit in init
2022-05-11 21:07:04 +02:00
János Benjamin Antal
3e7aef432f Improve docker-compose setup for running streams e2e tests locally (#388)
* Separate pulsar and kafka docker-compose files

* Add docker-compose for redpanda

* Add small docs how to use docker-compose files
2022-05-10 13:31:45 +02:00
Jeremy B
10ea9c773e Making mgp_error enum class when compiling c++. Remains enum when com… (#389)
* Making mgp_error enum class when compiling c++. Remains enum when compiling c.
2022-05-05 17:48:17 +02:00
János Benjamin Antal
3dd90e2acb Update the storage interface 2022-05-02 09:09:31 +02:00
János Benjamin Antal
62945c3b3a Merge remote-tracking branch 'origin/master' into T0804-MG-base-thrift-project 2022-04-29 15:48:00 +02:00
János Benjamin Antal
e1704ff2d0 Revert "Use attribute instead of UNLIKELY"
This reverts commit 5780ee6c5b.
2022-04-29 15:46:49 +02:00
János Benjamin Antal
7f0c53196b Revert unvanted change 2022-04-29 15:19:23 +02:00
Jeremy B
b782271be8 Fix shared module loading
* Moving function add_query_module from CMakeLists from tests/e2e/magic_functions to tests/e2e

* Adding failing test copying behavior when loading c module (.so) into memgraph.

* Fixing issue where NO_ERROR status returned  MgpTransAddFixedResult was converted to false

* Removing unnecessary transformation

* removing incorrect parameterization of test

* re-adding parametrized transformation
2022-04-28 20:28:44 +02:00
Jure Bajic
a8ffcfa046 Update license year
* Update license year

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
2022-04-27 13:31:37 +02:00
Jure Bajic
7b78665cd8 Implement Bolt over WebSocket with asio
* Replace server implementation with asio

* Add support for bolt over WebSocket
2022-04-27 10:13:16 +02:00
János Benjamin Antal
2da28c2c87 Add meta service interface 2022-04-22 17:06:48 +02:00
János Benjamin Antal
b36cce2428 Add thrift demo 2022-03-31 09:05:52 +02:00
János Benjamin Antal
6afcfcbb89 Merge commit 'bf01c58ed9950a1b2d05a884aad9544e8193b7f3' into T0804-MG-base-thrift-project 2022-03-24 15:25:39 +01:00
János Benjamin Antal
d21798c350 Implement scanVertices 2022-03-24 08:51:37 +01:00
János Benjamin Antal
6fe2293f4a Merge remote-tracking branch 'origin/master' into T0804-MG-base-thrift-project 2022-02-28 15:34:53 +01:00
János Benjamin Antal
2f1acff7d7 Replace outdated wrapper header file 2022-02-28 13:27:44 +01:00
János Benjamin Antal
e298f11968 Add storage client demo 2022-02-10 09:07:33 +01:00
János Benjamin Antal
5a4c0f7a1b Resolve conflict of ANTLR generated function and macro from glog 2022-02-08 12:33:39 +01:00
János Benjamin Antal
52960a8877 Add base of server and client 2022-02-07 15:14:29 +01:00
János Benjamin Antal
4811024e75 Add createVertices to Thrift 2022-02-07 15:13:19 +01:00
János Benjamin Antal
9193fcade3 CMake cleanup 2022-02-07 15:12:23 +01:00
János Benjamin Antal
066cfa32e3 Fix typo 2022-02-07 15:12:13 +01:00
János Benjamin Antal
77c27482f7 Link generated thrift interface to memgraph 2022-01-28 15:17:45 +01:00
János Benjamin Antal
8f42632fd2 CMake cleanup 2022-01-28 15:14:58 +01:00
János Benjamin Antal
5780ee6c5b Use attribute instead of UNLIKELY 2022-01-28 13:42:21 +01:00
János Benjamin Antal
542928b690 Fix namespace 2022-01-28 10:17:33 +01:00
János Benjamin Antal
67d39597d7 Remove coroutines from folly 2022-01-28 10:16:11 +01:00
János Benjamin Antal
bd0efa1159 Use patch files 2022-01-28 10:15:48 +01:00
János Benjamin Antal
e60dd252d0 Use std::unordered_map 2022-01-27 08:21:03 +01:00
János Benjamin Antal
84b78ded07 Add base of thrift interface 2022-01-27 07:52:39 +01:00
106 changed files with 3789 additions and 584 deletions

View File

@@ -88,4 +88,3 @@ CheckOptions:
- key: modernize-use-nullptr.NullMacros
value: 'NULL'
...

View File

@@ -24,14 +24,6 @@ for file in $modified_files; do
git checkout-index --prefix="$tmpdir/" -- $file
echo "Running clang-format..."
$project_folder/tools/git-clang-format $tmpdir/$file
CODE=$?
if [ $CODE -ne 0 ]; then
break
fi
# Do not break header checker
echo "Running header checker..."
$project_folder/tools/header-checker.py $tmpdir/$file $file --amend-year
@@ -39,7 +31,6 @@ for file in $modified_files; do
if [ $CODE -ne 0 ]; then
FAIL=1
fi
done;
return ${FAIL}

24
.pre-commit-config.yaml Normal file
View File

@@ -0,0 +1,24 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/psf/black
rev: 22.3.0
hooks:
- id: black
args: # arguments to configure black
- --line-length=120
- --include='\.pyi?$'
# these folders wont be formatted by black
- --exclude="""\.git |
\.__pycache__|
build|
libs|
.cache"""
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: v13.0.0
hooks:
- id: clang-format

View File

@@ -184,7 +184,8 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall \
-Werror=switch -Werror=switch-bool -Werror=return-type \
-Werror=return-stack-address \
-Wno-c99-designator")
-Wno-c99-designator \
-DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT")
# Don't omit frame pointer in RelWithDebInfo, for additional callchain debug.
set(CMAKE_CXX_FLAGS_RELWITHDEBINFO

297
cmake/FindSodium.cmake Normal file
View File

@@ -0,0 +1,297 @@
# Written in 2016 by Henrik Steffen Gaßmann <henrik@gassmann.onl>
#
# To the extent possible under law, the author(s) have dedicated all
# copyright and related and neighboring rights to this software to the
# public domain worldwide. This software is distributed without any warranty.
#
# You should have received a copy of the CC0 Public Domain Dedication
# along with this software. If not, see
#
# http://creativecommons.org/publicdomain/zero/1.0/
#
########################################################################
# Tries to find the local libsodium installation.
#
# On Windows the sodium_DIR environment variable is used as a default
# hint which can be overridden by setting the corresponding cmake variable.
#
# Once done the following variables will be defined:
#
# sodium_FOUND
# sodium_INCLUDE_DIR
# sodium_LIBRARY_DEBUG
# sodium_LIBRARY_RELEASE
#
#
# Furthermore an imported "sodium" target is created.
#
if (CMAKE_C_COMPILER_ID STREQUAL "GNU"
OR CMAKE_C_COMPILER_ID STREQUAL "Clang")
set(_GCC_COMPATIBLE 1)
endif()
# static library option
if (NOT DEFINED sodium_USE_STATIC_LIBS)
option(sodium_USE_STATIC_LIBS "enable to statically link against sodium" OFF)
endif()
if(NOT (sodium_USE_STATIC_LIBS EQUAL sodium_USE_STATIC_LIBS_LAST))
unset(sodium_LIBRARY CACHE)
unset(sodium_LIBRARY_DEBUG CACHE)
unset(sodium_LIBRARY_RELEASE CACHE)
unset(sodium_DLL_DEBUG CACHE)
unset(sodium_DLL_RELEASE CACHE)
set(sodium_USE_STATIC_LIBS_LAST ${sodium_USE_STATIC_LIBS} CACHE INTERNAL "internal change tracking variable")
endif()
########################################################################
# UNIX
if (UNIX)
# import pkg-config
find_package(PkgConfig QUIET)
if (PKG_CONFIG_FOUND)
pkg_check_modules(sodium_PKG QUIET libsodium)
endif()
if(sodium_USE_STATIC_LIBS)
foreach(_libname ${sodium_PKG_STATIC_LIBRARIES})
if (NOT _libname MATCHES "^lib.*\\.a$") # ignore strings already ending with .a
list(INSERT sodium_PKG_STATIC_LIBRARIES 0 "lib${_libname}.a")
endif()
endforeach()
list(REMOVE_DUPLICATES sodium_PKG_STATIC_LIBRARIES)
# if pkgconfig for libsodium doesn't provide
# static lib info, then override PKG_STATIC here..
if (NOT sodium_PKG_STATIC_FOUND)
set(sodium_PKG_STATIC_LIBRARIES libsodium.a)
endif()
set(XPREFIX sodium_PKG_STATIC)
else()
if (NOT sodium_PKG_FOUND)
set(sodium_PKG_LIBRARIES sodium)
endif()
set(XPREFIX sodium_PKG)
endif()
find_path(sodium_INCLUDE_DIR sodium.h
HINTS ${${XPREFIX}_INCLUDE_DIRS}
)
find_library(sodium_LIBRARY_DEBUG NAMES ${${XPREFIX}_LIBRARIES}
HINTS ${${XPREFIX}_LIBRARY_DIRS}
)
find_library(sodium_LIBRARY_RELEASE NAMES ${${XPREFIX}_LIBRARIES}
HINTS ${${XPREFIX}_LIBRARY_DIRS}
)
########################################################################
# Windows
elseif (WIN32)
set(sodium_DIR "$ENV{sodium_DIR}" CACHE FILEPATH "sodium install directory")
mark_as_advanced(sodium_DIR)
find_path(sodium_INCLUDE_DIR sodium.h
HINTS ${sodium_DIR}
PATH_SUFFIXES include
)
if (MSVC)
# detect target architecture
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/arch.cpp" [=[
#if defined _M_IX86
#error ARCH_VALUE x86_32
#elif defined _M_X64
#error ARCH_VALUE x86_64
#endif
#error ARCH_VALUE unknown
]=])
try_compile(_UNUSED_VAR "${CMAKE_CURRENT_BINARY_DIR}" "${CMAKE_CURRENT_BINARY_DIR}/arch.cpp"
OUTPUT_VARIABLE _COMPILATION_LOG
)
string(REGEX REPLACE ".*ARCH_VALUE ([a-zA-Z0-9_]+).*" "\\1" _TARGET_ARCH "${_COMPILATION_LOG}")
# construct library path
if (_TARGET_ARCH STREQUAL "x86_32")
string(APPEND _PLATFORM_PATH "Win32")
elseif(_TARGET_ARCH STREQUAL "x86_64")
string(APPEND _PLATFORM_PATH "x64")
else()
message(FATAL_ERROR "the ${_TARGET_ARCH} architecture is not supported by Findsodium.cmake.")
endif()
string(APPEND _PLATFORM_PATH "/$$CONFIG$$")
if (MSVC_VERSION LESS 1900)
math(EXPR _VS_VERSION "${MSVC_VERSION} / 10 - 60")
else()
math(EXPR _VS_VERSION "${MSVC_VERSION} / 10 - 50")
endif()
string(APPEND _PLATFORM_PATH "/v${_VS_VERSION}")
if (sodium_USE_STATIC_LIBS)
string(APPEND _PLATFORM_PATH "/static")
else()
string(APPEND _PLATFORM_PATH "/dynamic")
endif()
string(REPLACE "$$CONFIG$$" "Debug" _DEBUG_PATH_SUFFIX "${_PLATFORM_PATH}")
string(REPLACE "$$CONFIG$$" "Release" _RELEASE_PATH_SUFFIX "${_PLATFORM_PATH}")
find_library(sodium_LIBRARY_DEBUG libsodium.lib
HINTS ${sodium_DIR}
PATH_SUFFIXES ${_DEBUG_PATH_SUFFIX}
)
find_library(sodium_LIBRARY_RELEASE libsodium.lib
HINTS ${sodium_DIR}
PATH_SUFFIXES ${_RELEASE_PATH_SUFFIX}
)
if (NOT sodium_USE_STATIC_LIBS)
set(CMAKE_FIND_LIBRARY_SUFFIXES_BCK ${CMAKE_FIND_LIBRARY_SUFFIXES})
set(CMAKE_FIND_LIBRARY_SUFFIXES ".dll")
find_library(sodium_DLL_DEBUG libsodium
HINTS ${sodium_DIR}
PATH_SUFFIXES ${_DEBUG_PATH_SUFFIX}
)
find_library(sodium_DLL_RELEASE libsodium
HINTS ${sodium_DIR}
PATH_SUFFIXES ${_RELEASE_PATH_SUFFIX}
)
set(CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES_BCK})
endif()
elseif(_GCC_COMPATIBLE)
if (sodium_USE_STATIC_LIBS)
find_library(sodium_LIBRARY_DEBUG libsodium.a
HINTS ${sodium_DIR}
PATH_SUFFIXES lib
)
find_library(sodium_LIBRARY_RELEASE libsodium.a
HINTS ${sodium_DIR}
PATH_SUFFIXES lib
)
else()
find_library(sodium_LIBRARY_DEBUG libsodium.dll.a
HINTS ${sodium_DIR}
PATH_SUFFIXES lib
)
find_library(sodium_LIBRARY_RELEASE libsodium.dll.a
HINTS ${sodium_DIR}
PATH_SUFFIXES lib
)
file(GLOB _DLL
LIST_DIRECTORIES false
RELATIVE "${sodium_DIR}/bin"
"${sodium_DIR}/bin/libsodium*.dll"
)
find_library(sodium_DLL_DEBUG ${_DLL} libsodium
HINTS ${sodium_DIR}
PATH_SUFFIXES bin
)
find_library(sodium_DLL_RELEASE ${_DLL} libsodium
HINTS ${sodium_DIR}
PATH_SUFFIXES bin
)
endif()
else()
message(FATAL_ERROR "this platform is not supported by FindSodium.cmake")
endif()
########################################################################
# unsupported
else()
message(FATAL_ERROR "this platform is not supported by FindSodium.cmake")
endif()
########################################################################
# common stuff
# extract sodium version
if (sodium_INCLUDE_DIR)
set(_VERSION_HEADER "${_INCLUDE_DIR}/sodium/version.h")
if (EXISTS _VERSION_HEADER)
file(READ "${_VERSION_HEADER}" _VERSION_HEADER_CONTENT)
string(REGEX REPLACE ".*#[ \t]*define[ \t]*SODIUM_VERSION_STRING[ \t]*\"([^\n]*)\".*" "\\1"
sodium_VERSION "${_VERSION_HEADER_CONTENT}")
set(sodium_VERSION "${sodium_VERSION}" PARENT_SCOPE)
endif()
endif()
# communicate results
include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(
Sodium # The name must be either uppercase or match the filename case.
REQUIRED_VARS
sodium_LIBRARY_RELEASE
sodium_LIBRARY_DEBUG
sodium_INCLUDE_DIR
VERSION_VAR
sodium_VERSION
)
if(Sodium_FOUND)
set(sodium_LIBRARIES
optimized ${sodium_LIBRARY_RELEASE} debug ${sodium_LIBRARY_DEBUG})
endif()
# mark file paths as advanced
mark_as_advanced(sodium_INCLUDE_DIR)
mark_as_advanced(sodium_LIBRARY_DEBUG)
mark_as_advanced(sodium_LIBRARY_RELEASE)
if (WIN32)
mark_as_advanced(sodium_DLL_DEBUG)
mark_as_advanced(sodium_DLL_RELEASE)
endif()
# create imported target
if(sodium_USE_STATIC_LIBS)
set(_LIB_TYPE STATIC)
else()
set(_LIB_TYPE SHARED)
endif()
if(NOT TARGET sodium)
add_library(sodium ${_LIB_TYPE} IMPORTED)
endif()
set_target_properties(sodium PROPERTIES
INTERFACE_INCLUDE_DIRECTORIES "${sodium_INCLUDE_DIR}"
IMPORTED_LINK_INTERFACE_LANGUAGES "C"
)
if (sodium_USE_STATIC_LIBS)
set_target_properties(sodium PROPERTIES
INTERFACE_COMPILE_DEFINITIONS "SODIUM_STATIC"
IMPORTED_LOCATION "${sodium_LIBRARY_RELEASE}"
IMPORTED_LOCATION_DEBUG "${sodium_LIBRARY_DEBUG}"
)
else()
if (UNIX)
set_target_properties(sodium PROPERTIES
IMPORTED_LOCATION "${sodium_LIBRARY_RELEASE}"
IMPORTED_LOCATION_DEBUG "${sodium_LIBRARY_DEBUG}"
)
elseif (WIN32)
set_target_properties(sodium PROPERTIES
IMPORTED_IMPLIB "${sodium_LIBRARY_RELEASE}"
IMPORTED_IMPLIB_DEBUG "${sodium_LIBRARY_DEBUG}"
)
if (NOT (sodium_DLL_DEBUG MATCHES ".*-NOTFOUND"))
set_target_properties(sodium PROPERTIES
IMPORTED_LOCATION_DEBUG "${sodium_DLL_DEBUG}"
)
endif()
if (NOT (sodium_DLL_RELEASE MATCHES ".*-NOTFOUND"))
set_target_properties(sodium PROPERTIES
IMPORTED_LOCATION_RELWITHDEBINFO "${sodium_DLL_RELEASE}"
IMPORTED_LOCATION_MINSIZEREL "${sodium_DLL_RELEASE}"
IMPORTED_LOCATION_RELEASE "${sodium_DLL_RELEASE}"
)
endif()
endif()
endif()

169
cmake/MgThrift.cmake Normal file
View File

@@ -0,0 +1,169 @@
find_package(Gflags)
find_package(Folly)
find_package(wangle)
find_package(fizz)
find_package(proxygen)
find_package(FBThrift)
set(THRIFTCPP2 "FBThrift::thriftcpp2")
set(THRIFT1 ${FBTHRIFT_COMPILER})
include(${FBTHRIFT_INCLUDE_DIR}/thrift/ThriftLibrary.cmake)
set(MG_INTERFACE_TARGET_NAME_PREFIX "mg-interface")
function(_mg_thrift_generate
file_name
services
language
options
file_path
output_path
include_prefix
)
cmake_parse_arguments(THRIFT_GENERATE # Prefix
"" # Options
"" # One Value args
"THRIFT_INCLUDE_DIRECTORIES" # Multi-value args
"${ARGN}")
set(thrift_include_directories)
foreach(dir ${THRIFT_GENERATE_THRIFT_INCLUDE_DIRECTORIES})
list(APPEND thrift_include_directories "-I" "${dir}")
endforeach()
set("${file_name}-${language}-HEADERS"
${output_path}/gen-${language}/${file_name}_constants.h
${output_path}/gen-${language}/${file_name}_data.h
${output_path}/gen-${language}/${file_name}_metadata.h
${output_path}/gen-${language}/${file_name}_types.h
${output_path}/gen-${language}/${file_name}_types.tcc
)
set("${file_name}-${language}-SOURCES"
${output_path}/gen-${language}/${file_name}_constants.cpp
${output_path}/gen-${language}/${file_name}_data.cpp
${output_path}/gen-${language}/${file_name}_types.cpp
)
if(NOT "${options}" MATCHES "no_metadata")
set("${file_name}-${language}-SOURCES"
${${file_name}-${language}-SOURCES}
${output_path}/gen-${language}/${file_name}_metadata.cpp
)
endif()
foreach(service ${services})
set("${file_name}-${language}-HEADERS"
${${file_name}-${language}-HEADERS}
${output_path}/gen-${language}/${service}.h
${output_path}/gen-${language}/${service}.tcc
${output_path}/gen-${language}/${service}AsyncClient.h
${output_path}/gen-${language}/${service}_custom_protocol.h
)
set("${file_name}-${language}-SOURCES"
${${file_name}-${language}-SOURCES}
${output_path}/gen-${language}/${service}.cpp
${output_path}/gen-${language}/${service}AsyncClient.cpp
)
endforeach()
if("${include_prefix}" STREQUAL "")
set(include_prefix_text "")
else()
set(include_prefix_text "include_prefix=${include_prefix}")
if(NOT "${options}" STREQUAL "")
set(include_prefix_text ",${include_prefix_text}")
endif()
endif()
set(gen_language ${language})
if("${language}" STREQUAL "cpp2")
set(gen_language "mstch_cpp2")
elseif("${language}" STREQUAL "py3")
set(gen_language "mstch_py3")
file(WRITE "${output_path}/gen-${language}/${file_name}/__init__.py")
endif()
add_custom_command(
OUTPUT ${${file_name}-${language}-HEADERS}
${${file_name}-${language}-SOURCES}
COMMAND ${THRIFT1}
--gen "${gen_language}:${options}${include_prefix_text}"
-o ${output_path}
${thrift_include_directories}
"${file_path}/${file_name}.thrift"
DEPENDS
${THRIFT1}
"${file_path}/${file_name}.thrift"
COMMENT "Generating ${MG_INTERFACE_TARGET_NAME_PREFIX}-${file_name}-${language} files. Output: ${output_path}"
)
add_custom_target(
${MG_INTERFACE_TARGET_NAME_PREFIX}-${file_name}-${language}-target ALL
DEPENDS ${${language}-${language}-HEADERS}
${${file_name}-${language}-SOURCES}
)
set("${file_name}-${language}-SOURCES" ${${file_name}-${language}-SOURCES} PARENT_SCOPE)
install(
DIRECTORY gen-${language}
DESTINATION include/${include_prefix}
FILES_MATCHING PATTERN "*.h")
install(
DIRECTORY gen-${language}
DESTINATION include/${include_prefix}
FILES_MATCHING PATTERN "*.tcc")
endfunction()
function(_mg_thrift_object
file_name
services
language
options
file_path
output_path
include_prefix
)
_mg_thrift_generate(
"${file_name}"
"${services}"
"${language}"
"${options}"
"${file_path}"
"${output_path}"
"${include_prefix}"
"${ARGN}"
)
bypass_source_check(${${file_name}-${language}-SOURCES})
add_library(
"${MG_INTERFACE_TARGET_NAME_PREFIX}-${file_name}-${language}-obj"
OBJECT
${${file_name}-${language}-SOURCES}
)
add_dependencies(
"${MG_INTERFACE_TARGET_NAME_PREFIX}-${file_name}-${language}-obj"
"${MG_INTERFACE_TARGET_NAME_PREFIX}-${file_name}-${language}-target"
)
target_include_directories(${MG_INTERFACE_TARGET_NAME_PREFIX}-${file_name}-${language}-obj PUBLIC ${output_path})
message(STATUS "MgThrift will create the Object file : ${MG_INTERFACE_TARGET_NAME_PREFIX}-${file_name}-${language}-obj")
endfunction()
function(mg_thrift_library
file_name
services
file_path
output_path
include_prefix
)
_mg_thrift_object(
"${file_name}"
"${services}"
"cpp2"
"stack_arguments" # options
"${file_path}"
"${output_path}"
"${include_prefix}"
THRIFT_INCLUDE_DIRECTORIES "${FBTHRIFT_INCLUDE_DIR}"
)
set(LIBRARY_NAME "${MG_INTERFACE_TARGET_NAME_PREFIX}-${file_name}-cpp2")
add_library(
"${LIBRARY_NAME}"
$<TARGET_OBJECTS:${LIBRARY_NAME}-obj>
)
target_link_libraries("${LIBRARY_NAME}" ${THRIFTCPP2})
message("MgThrift will create the library file : ${LIBRARY_NAME}")
endfunction()

View File

@@ -47,6 +47,14 @@ modifications:
value: ""
override: false
- name: "bolt_cert_file"
value: "/etc/memgraph/ssl/cert.pem"
override: false
- name: "bolt_key_file"
value: "/etc/memgraph/ssl/key.pem"
override: false
- name: "storage_properties_on_edges"
value: "true"
override: true

View File

@@ -0,0 +1,12 @@
diff -ur a/thrift/lib/cpp2/server/IOWorkerContext.h b/thrift/lib/cpp2/server/IOWorkerContext.h
--- a/thrift/lib/cpp2/server/IOWorkerContext.h 2022-06-08 11:50:43.043948657 +0200
+++ b/thrift/lib/cpp2/server/IOWorkerContext.h 2022-06-08 11:47:33.232695125 +0200
@@ -59,7 +59,7 @@
auto aliveLocked = alive->rlock();
if (*aliveLocked) {
// IOWorkerContext is still alive and so is replyQueue_
- queue->startConsumingInternal(&evb);
+ queue->startConsuming(&evb);
}
});
}

File diff suppressed because it is too large Load Diff

View File

@@ -1179,16 +1179,15 @@ def read_proc(func: typing.Callable[..., Record]):
"""
Register `func` as a read-only procedure of the current module.
`read_proc` is meant to be used as a decorator function to register module
procedures. The registered `func` needs to be a callable which optionally
takes `ProcCtx` as the first argument. Other arguments of `func` will be
bound to values passed in the cypherQuery. The full signature of `func`
needs to be annotated with types. The return type must be
`Record(field_name=type, ...)` and the procedure must produce either a
complete Record or None. To mark a field as deprecated, use
`Record(field_name=Deprecated(type), ...)`. Multiple records can be
produced by returning an iterable of them. Registering generator functions
is currently not supported.
The decorator `read_proc` is meant to be used to register module procedures.
The registered `func` needs to be a callable which optionally takes
`ProcCtx` as its first argument. Other arguments of `func` will be bound to
values passed in the cypherQuery. The full signature of `func` needs to be
annotated with types. The return type must be `Record(field_name=type, ...)`
and the procedure must produce either a complete Record or None. To mark a
field as deprecated, use `Record(field_name=Deprecated(type), ...)`.
Multiple records can be produced by returning an iterable of them.
Registering generator functions is currently not supported.
Example usage.
@@ -1222,16 +1221,16 @@ def write_proc(func: typing.Callable[..., Record]):
"""
Register `func` as a writeable procedure of the current module.
`write_proc` is meant to be used as a decorator function to register module
The decorator `write_proc` is meant to be used to register module
procedures. The registered `func` needs to be a callable which optionally
takes `ProcCtx` as the first argument. Other arguments of `func` will be
bound to values passed in the cypherQuery. The full signature of `func`
needs to be annotated with types. The return type must be
`Record(field_name=type, ...)` and the procedure must produce either a
complete Record or None. To mark a field as deprecated, use
`Record(field_name=Deprecated(type), ...)`. Multiple records can be
produced by returning an iterable of them. Registering generator functions
is currently not supported.
`Record(field_name=Deprecated(type), ...)`. Multiple records can be produced
by returning an iterable of them. Registering generator functions is
currently not supported.
Example usage.
@@ -1459,8 +1458,9 @@ def transformation(func: typing.Callable[..., Record]):
class FuncCtx:
"""Context of a function being executed.
Access to a FuncCtx is only valid during a single execution of a transformation.
You should not globally store a FuncCtx instance.
Access to a FuncCtx is only valid during a single execution of a function in
a query. You should not globally store a FuncCtx instance. The graph object
within the FuncCtx is not mutable.
"""
__slots__ = "_graph"
@@ -1475,6 +1475,45 @@ class FuncCtx:
def function(func: typing.Callable):
"""
Register `func` as a user-defined function in the current module.
The decorator `function` is meant to be used to register module functions.
The registered `func` needs to be a callable which optionally takes
`FuncCtx` as its first argument. Other arguments of `func` will be bound to
values passed in the Cypher query. Only the funcion arguments need to be
annotated with types. The return type doesn't need to be specified, but it
has to be supported by `mgp.Any`. Registering generator functions is
currently not supported.
Example usage.
```
import mgp
@mgp.function
def func_example(context: mgp.FuncCtx,
required_arg: str,
optional_arg: mgp.Nullable[str] = None
):
return_args = [required_arg]
if optional_arg is not None:
return_args.append(optional_arg)
# Return any kind of result supported by mgp.Any
return return_args
```
The example function above returns a list of provided arguments:
* `required_arg` is always present and its value is the first argument of
the function.
* `optional_arg` is present if the second argument of the function is not
`null`.
Any errors can be reported by raising an Exception.
The function can be invoked in Cypher using the following calls:
RETURN example.func_example("first argument", "second_argument");
RETURN example.func_example("first argument");
Naturally, you may pass in different arguments.
"""
raise_if_does_not_meet_requirements(func)
register_func = _mgp.Module.add_function
sig = inspect.signature(func)

4
init
View File

@@ -135,3 +135,7 @@ for hook in $(find $DIR/.githooks -type f -printf "%f\n"); do
ln -s -f "$DIR/.githooks/$hook" "$DIR/.git/hooks/$hook"
echo "Added $hook hook"
done;
# Install precommit hook
python3 -m pip install pre-commit
python3 -m pre_commit install

View File

@@ -36,7 +36,7 @@ ADDITIONAL USE GRANT: You may use the Licensed Work in accordance with the
3. using the Licensed Work to create a work or solution
which competes (or might reasonably be expected to
compete) with the Licensed Work.
CHANGE DATE: 2026-18-02
CHANGE DATE: 2026-27-04
CHANGE LICENSE: Apache License, Version 2.0
For information about alternative licensing arrangements, please visit: https://memgraph.com/legal.

View File

@@ -41,7 +41,7 @@ set(CPACK_DEBIAN_PACKAGE_DESCRIPTION "${CPACK_PACKAGE_DESCRIPTION_SUMMARY}
applications driver by real-time connected data.")
# Add `openssl` package to dependencies list. Used to generate SSL certificates.
# We also depend on `python3` because we embed it in Memgraph.
set(CPACK_DEBIAN_PACKAGE_DEPENDS "openssl (>= 1.1.0), python3 (>= 3.5.0)")
set(CPACK_DEBIAN_PACKAGE_DEPENDS "openssl (>= 1.1.0), python3 (>= 3.5.0), libstdc++6")
# Setting arhitecture extension for rpm packages
set(MG_ARCH_EXTENSION_RPM "noarch")
@@ -67,7 +67,7 @@ It aims to deliver developers the speed, simplicity and scale required to build
the next generation of applications driver by real-time connected data.")
# Add `openssl` package to dependencies list. Used to generate SSL certificates.
# We also depend on `python3` because we embed it in Memgraph.
set(CPACK_RPM_PACKAGE_REQUIRES "openssl >= 1.0.0, curl >= 7.29.0, python3 >= 3.5.0")
set(CPACK_RPM_PACKAGE_REQUIRES "openssl >= 1.0.0, curl >= 7.29.0, python3 >= 3.5.0, libstdc >= 6")
# All variables must be set before including.
include(CPack)

View File

@@ -1,6 +1,7 @@
# CMake configuration for the main memgraph library and executable
# add memgraph sub libraries, ordered by dependency
add_subdirectory(interface)
add_subdirectory(lisp)
add_subdirectory(utils)
add_subdirectory(requests)
@@ -37,7 +38,7 @@ set(mg_single_node_v2_sources
)
set(mg_single_node_v2_libs stdc++fs Threads::Threads
telemetry_lib mg-query mg-communication mg-memory mg-utils mg-auth mg-license mg-settings)
telemetry_lib mg-query mg-communication mg-memory mg-utils mg-auth mg-license mg-settings mg-interface-storage-cpp2)
if (MG_ENTERPRISE)
# These are enterprise subsystems
set(mg_single_node_v2_libs ${mg_single_node_v2_libs} mg-audit)

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -78,14 +78,18 @@ bool ClientContext::use_ssl() { return use_ssl_; }
ServerContext::ServerContext(const std::string &key_file, const std::string &cert_file, const std::string &ca_file,
bool verify_peer) {
ctx_.emplace(boost::asio::ssl::context::tls_server);
namespace ssl = boost::asio::ssl;
ctx_.emplace(ssl::context::tls_server);
// NOLINTNEXTLINE(hicpp-signed-bitwise)
ctx_->set_options(ssl::context::default_workarounds | ssl::context::no_sslv2 | ssl::context::no_sslv3 |
ssl::context::single_dh_use);
ctx_->set_default_verify_paths();
// TODO: add support for encrypted private keys
// TODO: add certificate revocation list (CRL)
boost::system::error_code ec;
ctx_->use_certificate_chain_file(cert_file, ec);
MG_ASSERT(!ec, "Couldn't load server certificate from file: {}", cert_file);
ctx_->use_private_key_file(key_file, boost::asio::ssl::context::pem, ec);
ctx_->use_private_key_file(key_file, ssl::context::pem, ec);
MG_ASSERT(!ec, "Couldn't load server private key from file: {}", key_file);
ctx_->set_options(SSL_OP_NO_SSLv3, ec);
@@ -100,7 +104,7 @@ ServerContext::ServerContext(const std::string &key_file, const std::string &cer
if (verify_peer) {
// Enable verification of the client certificate.
// NOLINTNEXTLINE(hicpp-signed-bitwise)
ctx_->set_verify_mode(boost::asio::ssl::verify_peer | boost::asio::ssl::verify_fail_if_no_peer_cert, ec);
ctx_->set_verify_mode(ssl::verify_peer | ssl::verify_fail_if_no_peer_cert, ec);
MG_ASSERT(!ec, "Setting SSL verification mode failed!");
}
}

View File

@@ -0,0 +1,135 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <atomic>
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <memory>
#include <string_view>
#include <thread>
#include <vector>
#include <spdlog/spdlog.h>
#include <boost/asio/io_context.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/asio/strand.hpp>
#include <boost/beast/core.hpp>
#include <boost/system/detail/error_code.hpp>
#include "communication/context.hpp"
#include "communication/v2/pool.hpp"
#include "communication/v2/session.hpp"
#include "utils/spin_lock.hpp"
#include "utils/synchronized.hpp"
namespace memgraph::communication::v2 {
template <class TSession, class TSessionData>
class Listener final : public std::enable_shared_from_this<Listener<TSession, TSessionData>> {
using tcp = boost::asio::ip::tcp;
using SessionHandler = Session<TSession, TSessionData>;
using std::enable_shared_from_this<Listener<TSession, TSessionData>>::shared_from_this;
public:
Listener(const Listener &) = delete;
Listener(Listener &&) = delete;
Listener &operator=(const Listener &) = delete;
Listener &operator=(Listener &&) = delete;
~Listener() {}
template <typename... Args>
static std::shared_ptr<Listener> Create(Args &&...args) {
return std::shared_ptr<Listener>{new Listener(std::forward<Args>(args)...)};
}
void Start() { DoAccept(); }
bool IsRunning() const noexcept { return alive_.load(std::memory_order_relaxed); }
private:
Listener(boost::asio::io_context &io_context, TSessionData *data, ServerContext *server_context,
tcp::endpoint &endpoint, const std::string_view service_name, const uint64_t inactivity_timeout_sec)
: io_context_(io_context),
data_(data),
server_context_(server_context),
acceptor_(io_context_),
endpoint_{endpoint},
service_name_{service_name},
inactivity_timeout_{inactivity_timeout_sec} {
boost::system::error_code ec;
// Open the acceptor
acceptor_.open(endpoint.protocol(), ec);
if (ec) {
OnError(ec, "open");
return;
}
// Allow address reuse
acceptor_.set_option(boost::asio::socket_base::reuse_address(true), ec);
if (ec) {
OnError(ec, "set_option");
return;
}
// Bind to the server address
acceptor_.bind(endpoint, ec);
if (ec) {
spdlog::error(
utils::MessageWithLink("Cannot bind to socket on endpoint {}.", endpoint, "https://memgr.ph/socket"));
OnError(ec, "bind");
return;
}
acceptor_.listen(boost::asio::socket_base::max_listen_connections, ec);
if (ec) {
OnError(ec, "listen");
return;
}
}
void DoAccept() {
acceptor_.async_accept(io_context_,
[shared_this = shared_from_this()](auto ec, boost::asio::ip::tcp::socket &&socket) {
shared_this->OnAccept(ec, std::move(socket));
});
}
void OnAccept(boost::system::error_code ec, tcp::socket socket) {
if (ec) {
return OnError(ec, "accept");
}
auto session = SessionHandler::Create(std::move(socket), data_, *server_context_, endpoint_, inactivity_timeout_,
service_name_);
session->Start();
DoAccept();
}
void OnError(const boost::system::error_code &ec, const std::string_view what) {
spdlog::error("Listener failed on {}: {}", what, ec.message());
alive_.store(false, std::memory_order_relaxed);
}
boost::asio::io_context &io_context_;
TSessionData *data_;
ServerContext *server_context_;
tcp::acceptor acceptor_;
tcp::endpoint endpoint_;
std::string_view service_name_;
std::chrono::seconds inactivity_timeout_;
std::atomic<bool> alive_;
};
} // namespace memgraph::communication::v2

View File

@@ -0,0 +1,68 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <cstddef>
#include <thread>
#include <vector>
#include <boost/asio/executor_work_guard.hpp>
#include <boost/asio/io_context.hpp>
#include "utils/logging.hpp"
namespace memgraph::communication::v2 {
class IOContextThreadPool final {
private:
using IOContext = boost::asio::io_context;
using IOContextGuard = boost::asio::executor_work_guard<boost::asio::io_context::executor_type>;
public:
explicit IOContextThreadPool(size_t pool_size) : guard_{io_context_.get_executor()}, pool_size_{pool_size} {
MG_ASSERT(pool_size != 0, "Pool size must be greater then 0!");
}
IOContextThreadPool(const IOContextThreadPool &) = delete;
IOContextThreadPool &operator=(const IOContextThreadPool &) = delete;
IOContextThreadPool(IOContextThreadPool &&) = delete;
IOContextThreadPool &operator=(IOContextThreadPool &&) = delete;
~IOContextThreadPool() = default;
void Run() {
background_threads_.reserve(pool_size_);
for (size_t i = 0; i < pool_size_; ++i) {
background_threads_.emplace_back([this]() { io_context_.run(); });
}
running_ = true;
}
void Shutdown() {
io_context_.stop();
running_ = false;
}
void AwaitShutdown() { background_threads_.clear(); }
bool IsRunning() const noexcept { return running_; }
IOContext &GetIOContext() noexcept { return io_context_; }
private:
/// The pool of io_context.
IOContext io_context_;
IOContextGuard guard_;
size_t pool_size_;
std::vector<std::jthread> background_threads_;
bool running_{false};
};
} // namespace memgraph::communication::v2

View File

@@ -0,0 +1,128 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <algorithm>
#include <cstdint>
#include <iostream>
#include <memory>
#include <optional>
#include <string>
#include <thread>
#include <vector>
#include <fmt/format.h>
#include <boost/asio/io_context.hpp>
#include <boost/asio/ip/address.hpp>
#include <boost/asio/ip/tcp.hpp>
#include "communication/context.hpp"
#include "communication/init.hpp"
#include "communication/v2/listener.hpp"
#include "communication/v2/pool.hpp"
#include "utils/logging.hpp"
#include "utils/message.hpp"
#include "utils/thread.hpp"
namespace memgraph::communication::v2 {
using Socket = boost::asio::ip::tcp::socket;
using ServerEndpoint = boost::asio::ip::tcp::endpoint;
/**
* Communication server.
*
* Listens for incoming connections on the server port and assigns them to the
* connection listener. The listener and session are implemented using asio
* async model. Currently the implemented model is thread per core model
* opposed to io_context per core. The reasoning for opting for the former model
* is the robustness to the multiple resource demanding queries that can be split
* across multiple threads, and then a single thread would not block io_context,
* unlike in the latter model where it is possible that thread that accepts
* request is being blocked by demanding query.
* All logic is contained within handlers that are being dispatched
* on a single strand per session. The only exception is write which is
* synchronous since the nature of the clients conenction is synchronous as
* well.
*
* Current Server architecture:
* incoming connection -> server -> listener -> session
*
* @tparam TSession the server can handle different Sessions, each session
* represents a different protocol so the same network infrastructure
* can be used for handling different protocols
* @tparam TSessionData the class with objects that will be forwarded to the
* session
*/
template <typename TSession, typename TSessionData>
class Server final {
using ServerHandler = Server<TSession, TSessionData>;
public:
/**
* Constructs and binds server to endpoint, operates on session data and
* invokes workers_count workers
*/
Server(ServerEndpoint &endpoint, TSessionData *session_data, ServerContext *server_context,
const int inactivity_timeout_sec, const std::string_view service_name,
size_t workers_count = std::thread::hardware_concurrency())
: endpoint_{endpoint},
service_name_{service_name},
context_thread_pool_{workers_count},
listener_{Listener<TSession, TSessionData>::Create(context_thread_pool_.GetIOContext(), session_data,
server_context, endpoint_, service_name_,
inactivity_timeout_sec)} {}
~Server() { MG_ASSERT(!IsRunning(), "Server wasn't shutdown properly"); }
Server(const Server &) = delete;
Server(Server &&) = delete;
Server &operator=(const Server &) = delete;
Server &operator=(Server &&) = delete;
const auto &Endpoint() const {
MG_ASSERT(IsRunning(), "You can't get the server endpoint when it's not running!");
return endpoint_;
}
bool Start() {
if (IsRunning()) {
spdlog::error("The server is already running");
return false;
}
listener_->Start();
spdlog::info("{} server is fully armed and operational", service_name_);
spdlog::info("{} listening on {}", service_name_, endpoint_.address());
context_thread_pool_.Run();
return true;
}
void Shutdown() {
context_thread_pool_.Shutdown();
spdlog::info("{} shutting down...", service_name_);
}
void AwaitShutdown() { context_thread_pool_.AwaitShutdown(); }
bool IsRunning() const noexcept { return context_thread_pool_.IsRunning() && listener_->IsRunning(); }
private:
ServerEndpoint endpoint_;
std::string service_name_;
IOContextThreadPool context_thread_pool_;
std::shared_ptr<Listener<TSession, TSessionData>> listener_;
};
} // namespace memgraph::communication::v2

View File

@@ -0,0 +1,508 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <deque>
#include <functional>
#include <memory>
#include <string>
#include <string_view>
#include <utility>
#include <variant>
#include <spdlog/spdlog.h>
#include <boost/asio/bind_executor.hpp>
#include <boost/asio/buffer.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/asio/read.hpp>
#include <boost/asio/socket_base.hpp>
#include <boost/asio/ssl/stream.hpp>
#include <boost/asio/ssl/stream_base.hpp>
#include <boost/asio/steady_timer.hpp>
#include <boost/asio/strand.hpp>
#include <boost/asio/system_context.hpp>
#include <boost/asio/write.hpp>
#include <boost/beast/core/tcp_stream.hpp>
#include <boost/beast/http.hpp>
#include <boost/beast/websocket.hpp>
#include <boost/beast/websocket/rfc6455.hpp>
#include <boost/system/detail/error_code.hpp>
#include "communication/context.hpp"
#include "communication/exceptions.hpp"
#include "utils/logging.hpp"
#include "utils/variant_helpers.hpp"
namespace memgraph::communication::v2 {
/**
* This is used to provide input to user Sessions. All Sessions used with the
* network stack should use this class as their input stream.
*/
using InputStream = communication::Buffer::ReadEnd;
using tcp = boost::asio::ip::tcp;
/**
* This is used to provide output from user Sessions. All Sessions used with the
* network stack should use this class for their output stream.
*/
class OutputStream final {
public:
explicit OutputStream(std::function<bool(const uint8_t *, size_t, bool)> write_function)
: write_function_(write_function) {}
OutputStream(const OutputStream &) = delete;
OutputStream(OutputStream &&) = delete;
OutputStream &operator=(const OutputStream &) = delete;
OutputStream &operator=(OutputStream &&) = delete;
~OutputStream() = default;
bool Write(const uint8_t *data, size_t len, bool have_more = false) { return write_function_(data, len, have_more); }
bool Write(const std::string &str, bool have_more = false) {
return Write(reinterpret_cast<const uint8_t *>(str.data()), str.size(), have_more);
}
private:
std::function<bool(const uint8_t *, size_t, bool)> write_function_;
};
/**
* This class is used internally in the communication stack to handle all user
* Websocket Sessions. It handles socket ownership, inactivity timeout and protocol
* wrapping.
*/
template <typename TSession, typename TSessionData>
class WebsocketSession : public std::enable_shared_from_this<WebsocketSession<TSession, TSessionData>> {
using WebSocket = boost::beast::websocket::stream<boost::beast::tcp_stream>;
using std::enable_shared_from_this<WebsocketSession<TSession, TSessionData>>::shared_from_this;
public:
template <typename... Args>
static std::shared_ptr<WebsocketSession> Create(Args &&...args) {
return std::shared_ptr<WebsocketSession>(new WebsocketSession(std::forward<Args>(args)...));
}
// Start the asynchronous accept operation
template <class Body, class Allocator>
void DoAccept(boost::beast::http::request<Body, boost::beast::http::basic_fields<Allocator>> req) {
execution_active_ = true;
// Set suggested timeout settings for the websocket
ws_.set_option(boost::beast::websocket::stream_base::timeout::suggested(boost::beast::role_type::server));
boost::asio::socket_base::keep_alive option(true);
// Set a decorator to change the Server of the handshake
ws_.set_option(boost::beast::websocket::stream_base::decorator([](boost::beast::websocket::response_type &res) {
res.set(boost::beast::http::field::server, std::string("Memgraph Bolt WS"));
res.set(boost::beast::http::field::sec_websocket_protocol, "binary");
}));
ws_.binary(true);
// Accept the websocket handshake
ws_.async_accept(
req, boost::asio::bind_executor(strand_, std::bind_front(&WebsocketSession::OnAccept, shared_from_this())));
}
bool Write(const uint8_t *data, size_t len) {
if (!IsConnected()) {
return false;
}
boost::system::error_code ec;
ws_.write(boost::asio::buffer(data, len), ec);
if (ec) {
OnError(ec, "write");
return false;
}
return true;
}
private:
// Take ownership of the socket
explicit WebsocketSession(tcp::socket &&socket, TSessionData *data, tcp::endpoint endpoint,
std::string_view service_name)
: ws_(std::move(socket)),
strand_{boost::asio::make_strand(ws_.get_executor())},
output_stream_([this](const uint8_t *data, size_t len, bool /*have_more*/) { return Write(data, len); }),
session_(data, endpoint, input_buffer_.read_end(), &output_stream_),
endpoint_{endpoint},
remote_endpoint_{ws_.next_layer().socket().remote_endpoint()},
service_name_{service_name} {}
void OnAccept(boost::beast::error_code ec) {
if (ec) {
return OnError(ec, "accept");
}
// Read a message
DoRead();
}
void DoRead() {
if (!IsConnected()) {
return;
}
// Read a message into our buffer
auto buffer = input_buffer_.write_end()->Allocate();
ws_.async_read_some(
boost::asio::buffer(buffer.data, buffer.len),
boost::asio::bind_executor(strand_, std::bind_front(&WebsocketSession::OnRead, shared_from_this())));
}
void OnRead(const boost::system::error_code &ec, [[maybe_unused]] const size_t bytes_transferred) {
// This indicates that the WebsocketSession was closed
if (ec == boost::beast::websocket::error::closed) {
return;
}
if (ec) {
OnError(ec, "read");
}
input_buffer_.write_end()->Written(bytes_transferred);
try {
session_.Execute();
DoRead();
} catch (const SessionClosedException &e) {
spdlog::info("{} client {}:{} closed the connection.", service_name_, remote_endpoint_.address(),
remote_endpoint_.port());
DoClose();
} catch (const std::exception &e) {
spdlog::error(
"Exception was thrown while processing event in {} session "
"associated with {}:{}",
service_name_, remote_endpoint_.address(), remote_endpoint_.port());
spdlog::debug("Exception message: {}", e.what());
DoClose();
}
}
void OnError(const boost::system::error_code &ec, const std::string_view action) {
spdlog::error("Websocket Bolt session error: {} on {}", ec.message(), action);
DoClose();
}
void DoClose() {
ws_.async_close(
boost::beast::websocket::close_code::normal,
boost::asio::bind_executor(
strand_, [shared_this = shared_from_this()](boost::beast::error_code ec) { shared_this->OnClose(ec); }));
}
void OnClose(const boost::system::error_code &ec) {
if (!IsConnected()) {
return;
}
if (ec) {
return OnError(ec, "close");
}
}
bool IsConnected() const { return ws_.is_open() && execution_active_; }
WebSocket ws_;
boost::asio::strand<WebSocket::executor_type> strand_;
communication::Buffer input_buffer_;
OutputStream output_stream_;
TSession session_;
tcp::endpoint endpoint_;
tcp::endpoint remote_endpoint_;
std::string_view service_name_;
bool execution_active_{false};
};
/**
* This class is used internally in the communication stack to handle all user
* Sessions. It handles socket ownership, inactivity timeout and protocol
* wrapping.
*/
template <typename TSession, typename TSessionData>
class Session final : public std::enable_shared_from_this<Session<TSession, TSessionData>> {
using TCPSocket = tcp::socket;
using SSLSocket = boost::asio::ssl::stream<TCPSocket>;
using std::enable_shared_from_this<Session<TSession, TSessionData>>::shared_from_this;
public:
template <typename... Args>
static std::shared_ptr<Session> Create(Args &&...args) {
return std::shared_ptr<Session>(new Session(std::forward<Args>(args)...));
}
Session(const Session &) = delete;
Session(Session &&) = delete;
Session &operator=(const Session &) = delete;
Session &operator=(Session &&) = delete;
~Session() = default;
bool Start() {
if (execution_active_) {
return false;
}
execution_active_ = true;
timeout_timer_.async_wait(boost::asio::bind_executor(strand_, std::bind(&Session::OnTimeout, shared_from_this())));
if (std::holds_alternative<SSLSocket>(socket_)) {
boost::asio::dispatch(strand_, [shared_this = shared_from_this()] { shared_this->DoHandshake(); });
} else {
boost::asio::dispatch(strand_, [shared_this = shared_from_this()] { shared_this->DoRead(); });
}
return true;
}
bool Write(const uint8_t *data, size_t len, bool have_more = false) {
if (!IsConnected()) {
return false;
}
return std::visit(
utils::Overloaded{[shared_this = shared_from_this(), data, len, have_more](TCPSocket &socket) mutable {
boost::system::error_code ec;
while (len > 0) {
const auto sent = socket.send(boost::asio::buffer(data, len),
MSG_NOSIGNAL | (have_more ? MSG_MORE : 0), ec);
if (ec) {
shared_this->OnError(ec);
return false;
}
data += sent;
len -= sent;
}
return true;
},
[shared_this = shared_from_this(), data, len](SSLSocket &socket) mutable {
boost::system::error_code ec;
while (len > 0) {
const auto sent = socket.write_some(boost::asio::buffer(data, len), ec);
if (ec) {
shared_this->OnError(ec);
return false;
}
data += sent;
len -= sent;
}
return true;
}},
socket_);
}
bool IsConnected() const {
return std::visit([this](const auto &socket) { return execution_active_ && socket.lowest_layer().is_open(); },
socket_);
}
private:
explicit Session(tcp::socket &&socket, TSessionData *data, ServerContext &server_context, tcp::endpoint endpoint,
const std::chrono::seconds inactivity_timeout_sec, std::string_view service_name)
: socket_(CreateSocket(std::move(socket), server_context)),
strand_{boost::asio::make_strand(GetExecutor())},
output_stream_([this](const uint8_t *data, size_t len, bool have_more) { return Write(data, len, have_more); }),
session_(data, endpoint, input_buffer_.read_end(), &output_stream_),
data_{data},
endpoint_{endpoint},
remote_endpoint_{GetRemoteEndpoint()},
service_name_{service_name},
timeout_seconds_(inactivity_timeout_sec),
timeout_timer_(GetExecutor()) {
ExecuteForSocket([](auto &&socket) {
socket.lowest_layer().set_option(tcp::no_delay(true)); // enable PSH
socket.lowest_layer().set_option(boost::asio::socket_base::keep_alive(true)); // enable SO_KEEPALIVE
socket.lowest_layer().non_blocking(false);
});
timeout_timer_.expires_at(boost::asio::steady_timer::time_point::max());
spdlog::info("Accepted a connection from {}:", service_name_, remote_endpoint_.address(), remote_endpoint_.port());
}
void DoRead() {
if (!IsConnected()) {
return;
}
timeout_timer_.expires_after(timeout_seconds_);
ExecuteForSocket([this](auto &&socket) {
auto buffer = input_buffer_.write_end()->Allocate();
socket.async_read_some(
boost::asio::buffer(buffer.data, buffer.len),
boost::asio::bind_executor(strand_, std::bind_front(&Session::OnRead, shared_from_this())));
});
}
bool IsWebsocketUpgrade(boost::beast::http::request_parser<boost::beast::http::string_body> &parser) {
boost::system::error_code error_code_parsing;
parser.put(boost::asio::buffer(input_buffer_.read_end()->data(), input_buffer_.read_end()->size()),
error_code_parsing);
if (error_code_parsing) {
return false;
}
return boost::beast::websocket::is_upgrade(parser.get());
}
void OnRead(const boost::system::error_code &ec, const size_t bytes_transferred) {
if (ec) {
return OnError(ec);
}
input_buffer_.write_end()->Written(bytes_transferred);
// Can be a websocket connection only on the first read, since it is not
// expected from clients to upgrade from tcp to websocket
if (!has_received_msg_) {
has_received_msg_ = true;
boost::beast::http::request_parser<boost::beast::http::string_body> parser;
if (IsWebsocketUpgrade(parser)) {
spdlog::info("Switching {} to websocket connection", remote_endpoint_);
if (std::holds_alternative<TCPSocket>(socket_)) {
auto sock = std::get<TCPSocket>(std::move(socket_));
WebsocketSession<TSession, TSessionData>::Create(std::move(sock), data_, endpoint_, service_name_)
->DoAccept(parser.release());
execution_active_ = false;
return;
}
spdlog::error("Error while upgrading connection to websocket");
DoShutdown();
}
}
try {
session_.Execute();
DoRead();
} catch (const SessionClosedException &e) {
spdlog::info("{} client {}:{} closed the connection.", service_name_, remote_endpoint_.address(),
remote_endpoint_.port());
DoShutdown();
} catch (const std::exception &e) {
spdlog::error(
"Exception was thrown while processing event in {} session "
"associated with {}:{}",
service_name_, remote_endpoint_.address(), remote_endpoint_.port());
spdlog::debug("Exception message: {}", e.what());
DoShutdown();
}
}
void OnError(const boost::system::error_code &ec) {
if (ec == boost::asio::error::operation_aborted) {
return;
}
if (ec == boost::asio::error::eof) {
spdlog::info("Session closed by peer");
} else {
spdlog::error("Session error: {}", ec.message());
}
DoShutdown();
}
void DoShutdown() {
if (!IsConnected()) {
return;
}
execution_active_ = false;
timeout_timer_.cancel();
ExecuteForSocket([](auto &socket) {
boost::system::error_code ec;
auto &lowest_layer = socket.lowest_layer();
lowest_layer.shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec);
if (ec) {
spdlog::error("Session shutdown failed: {}", ec.what());
}
lowest_layer.close();
});
}
void DoHandshake() {
if (!IsConnected()) {
return;
}
if (auto *socket = std::get_if<SSLSocket>(&socket_); socket) {
socket->async_handshake(
boost::asio::ssl::stream_base::server,
boost::asio::bind_executor(strand_, std::bind_front(&Session::OnHandshake, shared_from_this())));
}
}
void OnHandshake(const boost::system::error_code &ec) {
if (ec) {
return OnError(ec);
}
DoRead();
}
void OnClose(const boost::system::error_code &ec) {
if (ec) {
return OnError(ec);
}
}
void OnTimeout() {
if (!IsConnected()) {
return;
}
// Check whether the deadline has passed. We compare the deadline against
// the current time since a new asynchronous operation may have moved the
// deadline before this actor had a chance to run.
if (timeout_timer_.expiry() <= boost::asio::steady_timer::clock_type::now()) {
// The deadline has passed. Stop the session. The other actors will
// terminate as soon as possible.
spdlog::info("Shutting down session after {} of inactivity", timeout_seconds_);
DoShutdown();
} else {
// Put the actor back to sleep.
timeout_timer_.async_wait(
boost::asio::bind_executor(strand_, std::bind(&Session::OnTimeout, shared_from_this())));
}
}
std::variant<TCPSocket, SSLSocket> CreateSocket(tcp::socket &&socket, ServerContext &context) {
if (context.use_ssl()) {
ssl_context_.emplace(context.context_clone());
return SSLSocket{std::move(socket), *ssl_context_};
}
return TCPSocket{std::move(socket)};
}
auto GetExecutor() {
return std::visit(utils::Overloaded{[](auto &&socket) { return socket.get_executor(); }}, socket_);
}
auto GetRemoteEndpoint() const {
return std::visit(utils::Overloaded{[](const auto &socket) { return socket.lowest_layer().remote_endpoint(); }},
socket_);
}
template <typename F>
decltype(auto) ExecuteForSocket(F &&fun) {
return std::visit(utils::Overloaded{std::forward<F>(fun)}, socket_);
}
std::variant<TCPSocket, SSLSocket> socket_;
std::optional<std::reference_wrapper<boost::asio::ssl::context>> ssl_context_;
boost::asio::strand<tcp::socket::executor_type> strand_;
communication::Buffer input_buffer_;
OutputStream output_stream_;
TSession session_;
TSessionData *data_;
tcp::endpoint endpoint_;
tcp::endpoint remote_endpoint_;
std::string_view service_name_;
std::chrono::seconds timeout_seconds_;
boost::asio::steady_timer timeout_timer_;
bool execution_active_{false};
bool has_received_msg_{false};
};
} // namespace memgraph::communication::v2

View File

@@ -11,8 +11,6 @@
#pragma once
#define BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT
#include <list>
#include <memory>

View File

@@ -11,8 +11,6 @@
#pragma once
#define BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT
#include <thread>
#include <spdlog/sinks/base_sink.h>

View File

@@ -11,8 +11,6 @@
#pragma once
#define BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT
#include <deque>
#include <memory>
#include <optional>

1
src/interface/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
gen-cpp2

View File

@@ -0,0 +1,27 @@
include(MgThrift)
set(MG_INTERFACE_PATH ${CMAKE_CURRENT_SOURCE_DIR})
mg_thrift_library(
"storage" #file_name
"Storage" #services
"${MG_INTERFACE_PATH}" #file_path
"${MG_INTERFACE_PATH}" #output_path
"interface" #include_prefix
)
mg_thrift_library(
"meta" #file_name
"Meta" #services
"${MG_INTERFACE_PATH}" #file_path
"${MG_INTERFACE_PATH}" #output_path
"interface" #include_prefix
)
mg_thrift_library(
"trial" #file_name
"PingPong" #services
"${MG_INTERFACE_PATH}" #file_path
"${MG_INTERFACE_PATH}" #output_path
"interface" #include_prefix
)

74
src/interface/meta.thrift Normal file
View File

@@ -0,0 +1,74 @@
namespace cpp2 interface.meta
typedef i64 LabelId
typedef i64 IndexId
struct Result {
1: bool success;
}
struct CreatePrimaryLabelRequest {
1: binary name;
2: list<binary> primary_keys;
}
struct CreateLabelResponse {
1: Result result;
2: LabelId label_id;
}
struct GetLabelInfosRequest {
// The empty list means return all of the label infos
1: list<binary> label_names;
}
struct LabelInfo {
1: binary name;
2: LabelId label_id;
3: list<binary> primary_keys;
}
struct GetLabelInfoResponse {
1: Result result;
2: LabelInfo label_info;
}
struct GetLabelInfosResponse {
1: Result result;
2: list<LabelInfo> label_infos;
}
struct CreateIndexRequest {
1: LabelId label_id;
2: list<binary> property_names;
}
struct CreateIndexResponse {
1: Result result;
2: IndexId index_id;
}
struct IndexInfo {
1: LabelId label_id;
2: list<binary> property_names;
}
struct GetIndexInfosResponse {
1: Result result;
2: list<IndexInfo> index_infos;
}
service Meta {
CreateLabelResponse createPrimaryLabel(1: CreatePrimaryLabelRequest req);
CreateLabelResponse createSecondaryLabel(1: binary label_name);
Result dropLabel(1: binary label_name);
GetLabelInfoResponse getLabelInfo(1:binary label_name);
GetLabelInfosResponse getLabelInfos(1: list<binary> label_names);
CreateIndexResponse createIndex(1: CreateIndexRequest req);
// Don't have index names, and the label doesn't identify the index uniquely, therefore
Result dropIndex(1: IndexId index_id);
GetIndexInfosResponse getIndexInfos();
GetIndexInfosResponse getIndexInfosForLabel(1: LabelId label_id);
}

View File

@@ -0,0 +1,273 @@
namespace cpp2 interface.storage
// https://stackoverflow.com/a/34234874/6639989
cpp_include "storage/v2/view.hpp"
typedef i64 VertexId
typedef i64 Gid
// TODO(antaljanosbenjamin): Use this after introducing 128 bit vertex ids
// struct VertexId {
// 1: i64 upper_half;
// 2: i64 lower_half;
// }
struct Label {
1: i64 id;
}
struct EdgeType {
1: binary name;
}
struct EdgeId {
1: VertexId src;
// QUESTION(antaljanosbenjamin): is it okay to have vertex based (edge id = vertex id + edge id inside vertex)?
2: Gid gid;
}
struct Date {
1: i16 year;
2: byte month;
3: byte day;
}
struct LocalTime {
1: byte hour;
2: byte minute;
3: byte second;
4: i16 millisecond;
5: i16 microsecond;
}
struct LocalDateTime {
1: Date date;
2: LocalTime local_time;
}
struct Duration {
1: i64 milliseconds;
}
union Value {
1: Null null_v;
2: bool bool_v;
3: i64 int_v;
4: double double_v;
5: binary string_v;
6: list<Value> list_v;
7: map<binary, Value> (cpp.template = "std::unordered_map") map_v (cpp2.ref_type = "unique");
8: Vertex vertex_v (cpp2.ref_type = "unique");
9: Edge edge_v (cpp2.ref_type = "unique");
10: Path path_v (cpp2.ref_type = "unique");
11: Date date_v;
12: LocalTime local_time_v;
13: LocalDateTime local_date_time_v;
14: Duration duration_v;
}
struct Null {
}
struct Vertex {
1: VertexId id;
// TODO(antaljanosbenjamin): Change to sperate primary and secondary labels when schema is implemented
2: list<Label> labels;
}
struct Edge {
1: VertexId src;
2: VertexId dst;
3: EdgeType type;
}
struct PathPart {
1: Vertex dst;
2: Edge edge;
}
struct Path {
1: Vertex src;
2: list<PathPart> parts;
}
struct ValuesMap {
1: map<i64, Value> (cpp.template = "std::unordered_map") values_map;
}
struct MappedValues {
1: list<ValuesMap> properties;
}
struct ListedValues {
1: list<list<Value>> properties;
}
union Values {
// This struct is necessary because depending on the request the response
// has two different formats:
// 1. When the request specifies the returned properties, then they are
// returned in that order, therefore no extra mapping is necessary.
// 2. When the request doesn't specify the returned properties, then all
// of the properties are returned. In this case the `mapped` field is
// used. To extract the <key,value> pairs from this struct the
// mapping of i64 -> property name has to be used.
1: ListedValues listed;
2: MappedValues mapped;
}
struct Expression {
1: binary alias;
2: binary expression;
}
struct Filter {
1: binary filter_expression;
}
enum OrderingDirection {
ASCENDING = 1;
DESCENDING = 2;
}
struct OrderBy {
1: Expression expression;
2: OrderingDirection direction;
}
struct Result {
// Just placeholder data for now
1: bool success;
}
enum View {
OLD = 0,
NEW = 1
} (cpp.enum_strict, cpp.type = "memgraph::storage::View")
struct ScanVerticesRequest {
1: i64 transaction_id;
2: optional i64 start_id;
// Special values are accepted:
// * __mg__id (Vertex, but without labels)
// * __mg__labels (Vertex, but without the id)
// If both of them is specified, then it will result in a single, fully populated vertex
// QUESTION(antaljanosbenjamin): Does the `__mg__labels` is necessary? What about passing the `labels` function
// as an expression? Maybe it is an optimization. For communicating the vertex id
// the Vertex struct is really handy.
3: optional list<binary> props_to_return;
4: list<Expression> expressions;
5: optional i64 limit;
6: View view;
7: optional Filter filter;
}
struct ScanVerticesResponse {
1: Result result;
2: Values values;
3: optional map<i64, binary> (cpp.template = "std::unordered_map") property_name_map;
// contains the next start_id if there is any
4: optional VertexId next_start_id;
}
union VertexOrEdgeIds {
1: list<VertexId> vertex_ids;
2: list<EdgeId> edge_ids;
}
struct GetPropertiesRequest {
1: i64 transaction_id;
2: VertexOrEdgeIds vertex_or_edge_ids;
3: list<binary> property_names;
4: list<Expression> expressions;
5: bool only_unique = false;
6: optional list<OrderBy> order_by;
7: optional i64 limit;
8: optional Filter filter;
}
struct GetPropertiesResponse {
1: Values values;
2: optional map<i64, binary> (cpp.template = "std::unordered_map") property_name_map;
}
enum EdgeDirection {
OUT = 1;
IN = 2;
BOTH = 3;
}
struct ExpandOneRequest {
1: i64 transaction_id;
2: list<VertexId> src_vertices;
3: list<EdgeType> edge_types;
4: EdgeDirection direction;
5: bool only_unique_neighbor_rows = false;
// The empty optional means return all of the properties, while an empty
// list means do not return any properties
// TODO(antaljanosbenjamin): All of the special values should be communicated through a single vertex object
// after schema is implemented
// Special values are accepted:
// * __mg__labels
6: optional list<binary> src_vertex_properties;
// TODO(antaljanosbenjamin): All of the special values should be communicated through a single vertex object
// after schema is implemented
// Special values are accepted:
// * __mg__dst_id (Vertex, but without labels)
// * __mg__type (binary)
7: optional list<binary> edge_properties;
// QUESTION(antaljanosbenjamin): Maybe also add possibility to expressions evaluated on the source vertex?
// List of expressions evaluated on edges
8: list<Expression> expressions;
9: optional list<OrderBy> order_by;
10: optional i64 limit;
11: optional Filter filter;
}
struct ExpandOneResultRow {
// NOTE: This struct could be a single Values with columns something like this:
// src_vertex(Vertex), vertex_prop1(Value), vertex_prop2(Value), edges(list<Value>)
// where edges might be a list of:
// 1. list<Value> if only a defined list of edge properties are returned
// 2. map<binary, Value> if all of the edge properties are returned
// The drawback of this is currently the key of the map is always interpreted as a string in Value, not as an
// integer, which should be in case of mapped properties.
1: Vertex src_vertex;
2: optional Values src_vertex_properties;
3: Values edges;
}
struct ExpandOneResponse {
// This approach might not suit the expand with per shard parrallelization,
// because the property_name_map has to be accessed from multiple threads
// in order to avoid duplicated keys (two threads might map the same
// property with different numbers) and multiple passes (to unify the
// mapping amond result returned from different shards).
1: list<ExpandOneResultRow> result;
2: optional map<i64, binary> (cpp.template = "std::unordered_map") property_name_map;
}
struct NewVertex {
1: list<i64> label_ids;
2: map<i64, Value> properties;
}
struct CreateVerticesRequest {
1: required i64 transaction_id;
2: map<i64, binary> (cpp.template = "std::unordered_map") labels_name_map;
3: map<i64, binary> (cpp.template = "std::unordered_map") property_name_map;
4: list<NewVertex> new_vertices;
}
service Storage {
i64 startTransaction()
Result commitTransaction(1: i64 transaction_id)
void abortTransaction(1: i64 transaction_id)
Result createVertices(1: CreateVerticesRequest req)
ScanVerticesResponse scanVertices(1: ScanVerticesRequest req)
GetPropertiesResponse getProperties(1: GetPropertiesRequest req)
ExpandOneResponse expandOne(1: ExpandOneRequest req)
}

View File

@@ -0,0 +1,11 @@
struct Ping {
1: binary message;
}
struct Pong{
1: binary message;
}
service PingPong {
Pong ping(1: Ping req)
}

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -26,16 +26,22 @@
#include <thread>
#include <fmt/format.h>
#include <folly/init/Init.h>
#include <folly/io/async/AsyncSocket.h>
#include <folly/io/async/AsyncTransport.h>
#include <folly/io/async/EventBase.h>
#include <gflags/gflags.h>
#include <spdlog/common.h>
#include <spdlog/sinks/daily_file_sink.h>
#include <spdlog/sinks/dist_sink.h>
#include <spdlog/sinks/stdout_color_sinks.h>
#include <thrift/lib/cpp2/async/HeaderClientChannel.h>
#include "communication/bolt/v1/constants.hpp"
#include "communication/websocket/auth.hpp"
#include "communication/websocket/server.hpp"
#include "helpers.hpp"
#include "interface/gen-cpp2/Storage.h"
#include "py/py.hpp"
#include "query/auth_checker.hpp"
#include "query/discard_value_stream.hpp"
@@ -81,8 +87,8 @@
#include "communication/bolt/v1/exceptions.hpp"
#include "communication/bolt/v1/session.hpp"
#include "communication/init.hpp"
#include "communication/server.hpp"
#include "communication/session.hpp"
#include "communication/v2/server.hpp"
#include "communication/v2/session.hpp"
#include "glue/communication.hpp"
#include "auth/auth.hpp"
@@ -252,6 +258,11 @@ DEFINE_double(query_execution_timeout_sec, 600,
"Maximum allowed query execution time. Queries exceeding this "
"limit will be aborted. Value of 0 means no limit.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(replication_replica_check_frequency_sec, 1,
"The time duration between two replica checks/pings. If < 1, replicas will NOT be checked at all. NOTE: "
"The MAIN instance allocates a new thread for each REPLICA.");
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(
memory_limit, 0,
@@ -842,13 +853,14 @@ class AuthChecker final : public memgraph::query::AuthChecker {
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth_;
};
class BoltSession final : public memgraph::communication::bolt::Session<memgraph::communication::InputStream,
memgraph::communication::OutputStream> {
class BoltSession final : public memgraph::communication::bolt::Session<memgraph::communication::v2::InputStream,
memgraph::communication::v2::OutputStream> {
public:
BoltSession(SessionData *data, const memgraph::io::network::Endpoint &endpoint,
memgraph::communication::InputStream *input_stream, memgraph::communication::OutputStream *output_stream)
: memgraph::communication::bolt::Session<memgraph::communication::InputStream,
memgraph::communication::OutputStream>(input_stream, output_stream),
BoltSession(SessionData *data, const memgraph::communication::v2::ServerEndpoint &endpoint,
memgraph::communication::v2::InputStream *input_stream,
memgraph::communication::v2::OutputStream *output_stream)
: memgraph::communication::bolt::Session<memgraph::communication::v2::InputStream,
memgraph::communication::v2::OutputStream>(input_stream, output_stream),
db_(data->db),
interpreter_(data->interpreter_context),
auth_(data->auth),
@@ -858,8 +870,8 @@ class BoltSession final : public memgraph::communication::bolt::Session<memgraph
endpoint_(endpoint) {
}
using memgraph::communication::bolt::Session<memgraph::communication::InputStream,
memgraph::communication::OutputStream>::TEncoder;
using memgraph::communication::bolt::Session<memgraph::communication::v2::InputStream,
memgraph::communication::v2::OutputStream>::TEncoder;
void BeginTransaction() override { interpreter_.BeginTransaction(); }
@@ -877,7 +889,8 @@ class BoltSession final : public memgraph::communication::bolt::Session<memgraph
}
#ifdef MG_ENTERPRISE
if (memgraph::utils::license::global_license_checker.IsValidLicenseFast()) {
audit_log_->Record(endpoint_.address, user_ ? *username : "", query, memgraph::storage::PropertyValue(params_pv));
audit_log_->Record(endpoint_.address().to_string(), user_ ? *username : "", query,
memgraph::storage::PropertyValue(params_pv));
}
#endif
try {
@@ -996,10 +1009,10 @@ class BoltSession final : public memgraph::communication::bolt::Session<memgraph
#ifdef MG_ENTERPRISE
memgraph::audit::Log *audit_log_;
#endif
memgraph::io::network::Endpoint endpoint_;
memgraph::communication::v2::ServerEndpoint endpoint_;
};
using ServerT = memgraph::communication::Server<BoltSession, SessionData>;
using ServerT = memgraph::communication::v2::Server<BoltSession, SessionData>;
using memgraph::communication::ServerContext;
// Needed to correctly handle memgraph destruction from a signal handler.
@@ -1068,6 +1081,22 @@ int main(int argc, char **argv) {
if (maybe_exc) {
spdlog::error(memgraph::utils::MessageWithLink("Unable to load support for embedded Python: {}.", *maybe_exc,
"https://memgr.ph/python"));
} else {
// Change how we load dynamic libraries on Python by using RTLD_NOW and
// RTLD_DEEPBIND flags. This solves an issue with using the wrong version of
// libstd.
auto gil = memgraph::py::EnsureGIL();
// NOLINTNEXTLINE(hicpp-signed-bitwise)
auto *flag = PyLong_FromLong(RTLD_NOW | RTLD_DEEPBIND);
auto *setdl = PySys_GetObject("setdlopenflags");
MG_ASSERT(setdl);
auto *arg = PyTuple_New(1);
MG_ASSERT(arg);
MG_ASSERT(PyTuple_SetItem(arg, 0, flag) == 0);
PyObject_CallObject(setdl, arg);
Py_DECREF(flag);
Py_DECREF(setdl);
Py_DECREF(arg);
}
} else {
spdlog::error(
@@ -1198,6 +1227,7 @@ int main(int argc, char **argv) {
&db,
{.query = {.allow_load_csv = FLAGS_allow_load_csv},
.execution_timeout_sec = FLAGS_query_execution_timeout_sec,
.replication_replica_check_frequency = std::chrono::seconds(FLAGS_replication_replica_check_frequency_sec),
.default_kafka_bootstrap_servers = FLAGS_kafka_bootstrap_servers,
.default_pulsar_service_url = FLAGS_pulsar_service_url,
.stream_transaction_conflict_retries = FLAGS_stream_transaction_conflict_retries,
@@ -1241,8 +1271,10 @@ int main(int argc, char **argv) {
memgraph::utils::MessageWithLink("Using non-secure Bolt connection (without SSL).", "https://memgr.ph/ssl"));
}
ServerT server({FLAGS_bolt_address, static_cast<uint16_t>(FLAGS_bolt_port)}, &session_data, &context,
FLAGS_bolt_session_inactivity_timeout, service_name, FLAGS_bolt_num_workers);
auto server_endpoint = memgraph::communication::v2::ServerEndpoint{
boost::asio::ip::address::from_string(FLAGS_bolt_address), static_cast<uint16_t>(FLAGS_bolt_port)};
ServerT server(server_endpoint, &session_data, &context, FLAGS_bolt_session_inactivity_timeout, service_name,
FLAGS_bolt_num_workers);
// Setup telemetry
std::optional<memgraph::telemetry::Telemetry> telemetry;
@@ -1287,8 +1319,19 @@ int main(int argc, char **argv) {
MG_ASSERT(server.Start(), "Couldn't start the Bolt server!");
websocket_server.Start();
server.AwaitShutdown();
websocket_server.AwaitShutdown();
folly::init(&argc, &argv);
folly::EventBase base;
int port = 7779; // The port on which server is listening
// Create a async client socket and connect it. Change
// the ip address to where the server is listening
auto socket(folly::AsyncSocket::newSocket(&base, "127.0.0.1", port));
// Create a HeaderClientChannel object which is used in creating
// client object
auto client_channel = apache::thrift::HeaderClientChannel::newChannel(std::move(socket));
// Create a client object
interface::storage::StorageAsyncClient client(std::move(client_channel));
memgraph::query::procedure::gModuleRegistry.UnloadAllModules();

View File

@@ -21,6 +21,8 @@ struct InterpreterConfig {
// The default execution timeout is 10 minutes.
double execution_timeout_sec{600.0};
// The same as \ref memgraph::storage::replication::ReplicationClientConfig
std::chrono::seconds replication_replica_check_frequency{1};
std::string default_kafka_bootstrap_servers;
std::string default_pulsar_service_url;

View File

@@ -17,8 +17,6 @@
parser grammar Cypher;
options { tokenVocab=CypherLexer; }
cypher : statement ';'? EOF ;
statement : query ;

View File

@@ -30,7 +30,7 @@ memgraphCypherKeyword : cypherKeyword
| BATCH_SIZE
| BEFORE
| BOOTSTRAP_SERVERS
| CHECK
| CHECK_
| CLEAR
| COMMIT
| COMMITTED
@@ -361,7 +361,7 @@ stopAllStreams : STOP ALL STREAMS ;
showStreams : SHOW STREAMS ;
checkStream : CHECK STREAM streamName ( BATCH_LIMIT batchLimit=literal ) ? ( TIMEOUT timeout=literal ) ? ;
checkStream : CHECK_ STREAM streamName ( BATCH_LIMIT batchLimit=literal ) ? ( TIMEOUT timeout=literal ) ? ;
settingName : literal ;

View File

@@ -35,7 +35,8 @@ BATCH_LIMIT : B A T C H UNDERSCORE L I M I T ;
BATCH_SIZE : B A T C H UNDERSCORE S I Z E ;
BEFORE : B E F O R E ;
BOOTSTRAP_SERVERS : B O O T S T R A P UNDERSCORE S E R V E R S ;
CHECK : C H E C K ;
/* This is a workaround to make sure the generated C++ function doesn't conlift with the CHECK macro from glog */
CHECK_ : C H E C K ;
CLEAR : C L E A R ;
COMMIT : C O M M I T ;
COMMITTED : C O M M I T T E D ;

View File

@@ -160,7 +160,8 @@ class ReplQueryHandler final : public query::ReplicationQueryHandler {
/// @throw QueryRuntimeException if an error ocurred.
void RegisterReplica(const std::string &name, const std::string &socket_address,
const ReplicationQuery::SyncMode sync_mode, const std::optional<double> timeout) override {
const ReplicationQuery::SyncMode sync_mode, const std::optional<double> timeout,
const std::chrono::seconds replica_check_frequency) override {
if (db_->GetReplicationRole() == storage::ReplicationRole::REPLICA) {
// replica can't register another replica
throw QueryRuntimeException("Replica can't register another replica!");
@@ -182,8 +183,9 @@ class ReplQueryHandler final : public query::ReplicationQueryHandler {
io::network::Endpoint::ParseSocketOrIpAddress(socket_address, query::kDefaultReplicationPort);
if (maybe_ip_and_port) {
auto [ip, port] = *maybe_ip_and_port;
auto ret =
db_->RegisterReplica(name, {std::move(ip), port}, repl_mode, {.timeout = timeout, .ssl = std::nullopt});
auto ret = db_->RegisterReplica(
name, {std::move(ip), port}, repl_mode,
{.timeout = timeout, .replica_check_frequency = replica_check_frequency, .ssl = std::nullopt});
if (ret.HasError()) {
throw QueryRuntimeException(fmt::format("Couldn't register replica '{}'!", name));
}
@@ -448,7 +450,7 @@ Callback HandleReplicationQuery(ReplicationQuery *repl_query, const Parameters &
return callback;
}
case ReplicationQuery::Action::SHOW_REPLICATION_ROLE: {
callback.header = {"replication mode"};
callback.header = {"replication role"};
callback.fn = [handler = ReplQueryHandler{interpreter_context->db}] {
auto mode = handler.ShowReplicationRole();
switch (mode) {
@@ -467,6 +469,7 @@ Callback HandleReplicationQuery(ReplicationQuery *repl_query, const Parameters &
const auto &sync_mode = repl_query->sync_mode_;
auto socket_address = repl_query->socket_address_->Accept(evaluator);
auto timeout = EvaluateOptionalExpression(repl_query->timeout_, &evaluator);
const auto replica_check_frequency = interpreter_context->config.replication_replica_check_frequency;
std::optional<double> maybe_timeout;
if (timeout.IsDouble()) {
maybe_timeout = timeout.ValueDouble();
@@ -474,8 +477,9 @@ Callback HandleReplicationQuery(ReplicationQuery *repl_query, const Parameters &
maybe_timeout = static_cast<double>(timeout.ValueInt());
}
callback.fn = [handler = ReplQueryHandler{interpreter_context->db}, name, socket_address, sync_mode,
maybe_timeout]() mutable {
handler.RegisterReplica(name, std::string(socket_address.ValueString()), sync_mode, maybe_timeout);
maybe_timeout, replica_check_frequency]() mutable {
handler.RegisterReplica(name, std::string(socket_address.ValueString()), sync_mode, maybe_timeout,
replica_check_frequency);
return std::vector<std::vector<TypedValue>>();
};
notifications->emplace_back(SeverityLevel::INFO, NotificationCode::REGISTER_REPLICA,
@@ -512,7 +516,6 @@ Callback HandleReplicationQuery(ReplicationQuery *repl_query, const Parameters &
typed_replica.emplace_back(TypedValue("async"));
break;
}
typed_replica.emplace_back(TypedValue(static_cast<int64_t>(replica.sync_mode)));
if (replica.timeout) {
typed_replica.emplace_back(TypedValue(*replica.timeout));
} else {

View File

@@ -137,7 +137,8 @@ class ReplicationQueryHandler {
/// @throw QueryRuntimeException if an error ocurred.
virtual void RegisterReplica(const std::string &name, const std::string &socket_address,
const ReplicationQuery::SyncMode sync_mode, const std::optional<double> timeout) = 0;
const ReplicationQuery::SyncMode sync_mode, const std::optional<double> timeout,
const std::chrono::seconds replica_check_frequency) = 0;
/// @throw QueryRuntimeException if an error ocurred.
virtual void DropReplica(const std::string &replica_name) = 0;

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -2109,7 +2109,7 @@ template <typename T>
concept AccessorWithProperties = requires(T value, storage::PropertyId property_id,
storage::PropertyValue property_value) {
{ value.ClearProperties() } -> std::same_as<storage::Result<std::map<storage::PropertyId, storage::PropertyValue>>>;
{value.SetProperty(property_id, property_value)};
{ value.SetProperty(property_id, property_value) };
};
/// Helper function that sets the given values on either a Vertex or an Edge.

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -24,7 +24,7 @@ MgpUniquePtr<mgp_value> GetStringValueOrSetError(const char *string, mgp_memory
}
bool InsertResultOrSetError(mgp_result *result, mgp_result_record *record, const char *result_name, mgp_value *value) {
if (const auto err = mgp_result_record_insert(record, result_name, value); err != MGP_ERROR_NO_ERROR) {
if (const auto err = mgp_result_record_insert(record, result_name, value); err != mgp_error::MGP_ERROR_NO_ERROR) {
const auto error_msg = fmt::format("Unable to set the result for {}, error = {}", result_name, err);
static_cast<void>(mgp_result_set_error_msg(result, error_msg.c_str()));
return false;

View File

@@ -25,7 +25,7 @@ TResult Call(TFunc func, TArgs... args) {
static_assert(std::is_trivially_copyable_v<TFunc>);
static_assert((std::is_trivially_copyable_v<std::remove_reference_t<TArgs>> && ...));
TResult result{};
MG_ASSERT(func(args..., &result) == MGP_ERROR_NO_ERROR);
MG_ASSERT(func(args..., &result) == mgp_error::MGP_ERROR_NO_ERROR);
return result;
}
@@ -50,10 +50,10 @@ mgp_error CreateMgpObject(MgpUniquePtr<TObj> &obj, TFunc func, TArgs &&...args)
template <typename Fun>
[[nodiscard]] bool TryOrSetError(Fun &&func, mgp_result *result) {
if (const auto err = func(); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = func(); err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
static_cast<void>(mgp_result_set_error_msg(result, "Not enough memory!"));
return false;
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
const auto error_msg = fmt::format("Unexpected error ({})!", err);
static_cast<void>(mgp_result_set_error_msg(result, error_msg.c_str()));
return false;

View File

@@ -43,6 +43,17 @@
// NOLINTNEXTLINE(google-build-using-namespace)
using namespace memgraph::query::procedure;
#define MG_STATIC_ASSERT_NOEXCEPT(x) \
do { \
_Pragma("clang diagnostic push"); \
_Pragma("clang diagnostic ignored \"-Wunevaluated-expression\""); \
static_assert(noexcept(x)); \
_Pragma("clang diagnostic pop"); \
} while (false)
#define MG_EXECUTE_NOEXCEPT(x) \
MG_STATIC_ASSERT_NOEXCEPT(x); \
x
namespace {
void *MgpAlignedAllocImpl(memgraph::utils::MemoryResource &memory, const size_t size_in_bytes, const size_t alignment) {
@@ -143,48 +154,48 @@ template <typename TFunc, typename... Args>
WrapExceptionsHelper(std::forward<TFunc>(func), std::forward<Args>(args)...);
} catch (const DeletedObjectException &neoe) {
spdlog::error("Deleted object error during mg API call: {}", neoe.what());
return MGP_ERROR_DELETED_OBJECT;
return mgp_error::MGP_ERROR_DELETED_OBJECT;
} catch (const KeyAlreadyExistsException &kaee) {
spdlog::error("Key already exists error during mg API call: {}", kaee.what());
return MGP_ERROR_KEY_ALREADY_EXISTS;
return mgp_error::MGP_ERROR_KEY_ALREADY_EXISTS;
} catch (const InsufficientBufferException &ibe) {
spdlog::error("Insufficient buffer error during mg API call: {}", ibe.what());
return MGP_ERROR_INSUFFICIENT_BUFFER;
return mgp_error::MGP_ERROR_INSUFFICIENT_BUFFER;
} catch (const ImmutableObjectException &ioe) {
spdlog::error("Immutable object error during mg API call: {}", ioe.what());
return MGP_ERROR_IMMUTABLE_OBJECT;
return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT;
} catch (const ValueConversionException &vce) {
spdlog::error("Value converion error during mg API call: {}", vce.what());
return MGP_ERROR_VALUE_CONVERSION;
return mgp_error::MGP_ERROR_VALUE_CONVERSION;
} catch (const SerializationException &se) {
spdlog::error("Serialization error during mg API call: {}", se.what());
return MGP_ERROR_SERIALIZATION_ERROR;
return mgp_error::MGP_ERROR_SERIALIZATION_ERROR;
} catch (const std::bad_alloc &bae) {
spdlog::error("Memory allocation error during mg API call: {}", bae.what());
return MGP_ERROR_UNABLE_TO_ALLOCATE;
return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE;
} catch (const memgraph::utils::OutOfMemoryException &oome) {
spdlog::error("Memory limit exceeded during mg API call: {}", oome.what());
return MGP_ERROR_UNABLE_TO_ALLOCATE;
return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE;
} catch (const std::out_of_range &oore) {
spdlog::error("Out of range error during mg API call: {}", oore.what());
return MGP_ERROR_OUT_OF_RANGE;
return mgp_error::MGP_ERROR_OUT_OF_RANGE;
} catch (const std::invalid_argument &iae) {
spdlog::error("Invalid argument error during mg API call: {}", iae.what());
return MGP_ERROR_INVALID_ARGUMENT;
return mgp_error::MGP_ERROR_INVALID_ARGUMENT;
} catch (const std::logic_error &lee) {
spdlog::error("Logic error during mg API call: {}", lee.what());
return MGP_ERROR_LOGIC_ERROR;
return mgp_error::MGP_ERROR_LOGIC_ERROR;
} catch (const std::exception &e) {
spdlog::error("Unexpected error during mg API call: {}", e.what());
return MGP_ERROR_UNKNOWN_ERROR;
return mgp_error::MGP_ERROR_UNKNOWN_ERROR;
} catch (const memgraph::utils::temporal::InvalidArgumentException &e) {
spdlog::error("Invalid argument was sent to an mg API call for temporal types: {}", e.what());
return MGP_ERROR_INVALID_ARGUMENT;
return mgp_error::MGP_ERROR_INVALID_ARGUMENT;
} catch (...) {
spdlog::error("Unexpected error during mg API call");
return MGP_ERROR_UNKNOWN_ERROR;
return mgp_error::MGP_ERROR_UNKNOWN_ERROR;
}
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
// Graph mutations
@@ -823,7 +834,6 @@ DEFINE_MGP_VALUE_MAKE_WITH_MEMORY(int, int64_t);
DEFINE_MGP_VALUE_MAKE_WITH_MEMORY(double, double);
DEFINE_MGP_VALUE_MAKE_WITH_MEMORY(string, const char *);
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define DEFINE_MGP_VALUE_MAKE(type) \
mgp_error mgp_value_make_##type(mgp_##type *val, mgp_value **result) { \
return WrapExceptions([val] { return NewRawMgpObject<mgp_value>(val->GetMemoryResource(), val); }, result); \
@@ -846,7 +856,7 @@ mgp_value_type MgpValueGetType(const mgp_value &val) noexcept { return val.type;
mgp_error mgp_value_get_type(mgp_value *val, mgp_value_type *result) {
static_assert(noexcept(MgpValueGetType(*val)));
*result = MgpValueGetType(*val);
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
@@ -854,7 +864,7 @@ mgp_error mgp_value_get_type(mgp_value *val, mgp_value_type *result) {
mgp_error mgp_value_is_##type_lowercase(mgp_value *val, int *result) { \
static_assert(noexcept(MgpValueGetType(*val))); \
*result = MgpValueGetType(*val) == MGP_VALUE_TYPE_##type_uppercase; \
return MGP_ERROR_NO_ERROR; \
return mgp_error::MGP_ERROR_NO_ERROR; \
}
DEFINE_MGP_VALUE_IS(null, NULL)
@@ -874,27 +884,27 @@ DEFINE_MGP_VALUE_IS(duration, DURATION)
mgp_error mgp_value_get_bool(mgp_value *val, int *result) {
*result = val->bool_v ? 1 : 0;
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_value_get_int(mgp_value *val, int64_t *result) {
*result = val->int_v;
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_value_get_double(mgp_value *val, double *result) {
*result = val->double_v;
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_value_get_string(mgp_value *val, const char **result) {
static_assert(noexcept(val->string_v.c_str()));
*result = val->string_v.c_str();
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define DEFINE_MGP_VALUE_GET(type) \
mgp_error mgp_value_get_##type(mgp_value *val, mgp_##type **result) { \
*result = val->type##_v; \
return MGP_ERROR_NO_ERROR; \
return mgp_error::MGP_ERROR_NO_ERROR; \
}
DEFINE_MGP_VALUE_GET(list)
@@ -940,13 +950,13 @@ mgp_error mgp_list_append_extend(mgp_list *list, mgp_value *val) {
mgp_error mgp_list_size(mgp_list *list, size_t *result) {
static_assert(noexcept(list->elems.size()));
*result = list->elems.size();
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_list_capacity(mgp_list *list, size_t *result) {
static_assert(noexcept(list->elems.capacity()));
*result = list->elems.capacity();
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_list_at(mgp_list *list, size_t i, mgp_value **result) {
@@ -978,7 +988,7 @@ mgp_error mgp_map_insert(mgp_map *map, const char *key, mgp_value *value) {
mgp_error mgp_map_size(mgp_map *map, size_t *result) {
static_assert(noexcept(map->items.size()));
*result = map->items.size();
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_map_at(mgp_map *map, const char *key, mgp_value **result) {
@@ -1089,7 +1099,7 @@ size_t MgpPathSize(const mgp_path &path) noexcept { return path.edges.size(); }
mgp_error mgp_path_size(mgp_path *path, size_t *result) {
*result = MgpPathSize(*path);
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_path_vertex_at(mgp_path *path, size_t i, mgp_vertex **result) {
@@ -1687,10 +1697,8 @@ mgp_error mgp_vertex_copy(mgp_vertex *v, mgp_memory *memory, mgp_vertex **result
void mgp_vertex_destroy(mgp_vertex *v) { DeleteRawMgpObject(v); }
mgp_error mgp_vertex_equal(mgp_vertex *v1, mgp_vertex *v2, int *result) {
// NOLINTNEXTLINE(clang-diagnostic-unevaluated-expression)
static_assert(noexcept(*result = *v1 == *v2 ? 1 : 0));
*result = *v1 == *v2 ? 1 : 0;
return MGP_ERROR_NO_ERROR;
MG_EXECUTE_NOEXCEPT(*result = *v1 == *v2 ? 1 : 0);
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_vertex_labels_count(mgp_vertex *v, size_t *result) {
@@ -1947,10 +1955,8 @@ mgp_error mgp_edge_copy(mgp_edge *e, mgp_memory *memory, mgp_edge **result) {
void mgp_edge_destroy(mgp_edge *e) { DeleteRawMgpObject(e); }
mgp_error mgp_edge_equal(mgp_edge *e1, mgp_edge *e2, int *result) {
// NOLINTNEXTLINE(clang-diagnostic-unevaluated-expression)
static_assert(noexcept(*result = *e1 == *e2 ? 1 : 0));
*result = *e1 == *e2 ? 1 : 0;
return MGP_ERROR_NO_ERROR;
MG_EXECUTE_NOEXCEPT(*result = *e1 == *e2 ? 1 : 0);
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_edge_get_type(mgp_edge *e, mgp_edge_type *result) {
@@ -1967,12 +1973,12 @@ mgp_error mgp_edge_get_type(mgp_edge *e, mgp_edge_type *result) {
mgp_error mgp_edge_get_from(mgp_edge *e, mgp_vertex **result) {
*result = &e->from;
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_edge_get_to(mgp_edge *e, mgp_vertex **result) {
*result = &e->to;
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_edge_get_property(mgp_edge *e, const char *name, mgp_memory *memory, mgp_value **result) {
@@ -2082,7 +2088,7 @@ mgp_error mgp_graph_get_vertex_by_id(mgp_graph *graph, mgp_vertex_id id, mgp_mem
mgp_error mgp_graph_is_mutable(mgp_graph *graph, int *result) {
*result = MgpGraphIsMutable(*graph) ? 1 : 0;
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
};
mgp_error mgp_graph_create_vertex(struct mgp_graph *graph, mgp_memory *memory, mgp_vertex **result) {
@@ -2507,7 +2513,7 @@ mgp_error mgp_proc_add_result(mgp_proc *proc, const char *name, mgp_type *type)
mgp_error MgpTransAddFixedResult(mgp_trans *trans) noexcept {
if (const auto err = AddResultToProp(trans, "query", Call<mgp_type *>(mgp_type_string), false);
err != MGP_ERROR_NO_ERROR) {
err != mgp_error::MGP_ERROR_NO_ERROR) {
return err;
}
return AddResultToProp(trans, "parameters", Call<mgp_type *>(mgp_type_nullable, Call<mgp_type *>(mgp_type_map)),
@@ -2754,7 +2760,7 @@ mgp_error mgp_message_offset(struct mgp_message *message, int64_t *result) {
mgp_error mgp_messages_size(mgp_messages *messages, size_t *result) {
static_assert(noexcept(messages->messages.size()));
*result = messages->messages.size();
return MGP_ERROR_NO_ERROR;
return mgp_error::MGP_ERROR_NO_ERROR;
}
mgp_error mgp_messages_at(mgp_messages *messages, size_t index, mgp_message **result) {

View File

@@ -121,18 +121,18 @@ void RegisterMgLoad(ModuleRegistry *module_registry, utils::RWLock *lock, Builti
bool succ = false;
WithUpgradedLock(lock, [&]() {
const char *arg_as_string{nullptr};
if (const auto err = mgp_value_get_string(arg, &arg_as_string); err != MGP_ERROR_NO_ERROR) {
if (const auto err = mgp_value_get_string(arg, &arg_as_string); err != mgp_error::MGP_ERROR_NO_ERROR) {
succ = false;
} else {
succ = module_registry->LoadOrReloadModuleFromName(arg_as_string);
}
});
if (!succ) {
MG_ASSERT(mgp_result_set_error_msg(result, "Failed to (re)load the module.") == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_result_set_error_msg(result, "Failed to (re)load the module.") == mgp_error::MGP_ERROR_NO_ERROR);
}
};
mgp_proc load("load", load_cb, utils::NewDeleteResource());
MG_ASSERT(mgp_proc_add_arg(&load, "module_name", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&load, "module_name", Call<mgp_type *>(mgp_type_string)) == mgp_error::MGP_ERROR_NO_ERROR);
module->AddProcedure("load", std::move(load));
}
@@ -235,11 +235,16 @@ void RegisterMgProcedures(
}
};
mgp_proc procedures("procedures", procedures_cb, utils::NewDeleteResource());
MG_ASSERT(mgp_proc_add_result(&procedures, "name", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "signature", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "is_write", Call<mgp_type *>(mgp_type_bool)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "path", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "is_editable", Call<mgp_type *>(mgp_type_bool)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "name", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "signature", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "is_write", Call<mgp_type *>(mgp_type_bool)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "path", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "is_editable", Call<mgp_type *>(mgp_type_bool)) ==
mgp_error::MGP_ERROR_NO_ERROR);
module->AddProcedure("procedures", std::move(procedures));
}
@@ -298,9 +303,12 @@ void RegisterMgTransformations(const std::map<std::string, std::unique_ptr<Modul
}
};
mgp_proc procedures("transformations", transformations_cb, utils::NewDeleteResource());
MG_ASSERT(mgp_proc_add_result(&procedures, "name", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "path", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "is_editable", Call<mgp_type *>(mgp_type_bool)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "name", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "path", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&procedures, "is_editable", Call<mgp_type *>(mgp_type_bool)) ==
mgp_error::MGP_ERROR_NO_ERROR);
module->AddProcedure("transformations", std::move(procedures));
}
@@ -374,10 +382,14 @@ void RegisterMgFunctions(
}
};
mgp_proc functions("functions", functions_cb, utils::NewDeleteResource());
MG_ASSERT(mgp_proc_add_result(&functions, "name", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&functions, "signature", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&functions, "path", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&functions, "is_editable", Call<mgp_type *>(mgp_type_bool)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&functions, "name", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&functions, "signature", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&functions, "path", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&functions, "is_editable", Call<mgp_type *>(mgp_type_bool)) ==
mgp_error::MGP_ERROR_NO_ERROR);
module->AddProcedure("functions", std::move(functions));
}
namespace {
@@ -469,9 +481,10 @@ void RegisterMgGetModuleFiles(ModuleRegistry *module_registry, BuiltinModule *mo
mgp_proc get_module_files("get_module_files", get_module_files_cb, utils::NewDeleteResource(),
{.required_privilege = AuthQuery::Privilege::MODULE_READ});
MG_ASSERT(mgp_proc_add_result(&get_module_files, "path", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&get_module_files, "path", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&get_module_files, "is_editable", Call<mgp_type *>(mgp_type_bool)) ==
MGP_ERROR_NO_ERROR);
mgp_error::MGP_ERROR_NO_ERROR);
module->AddProcedure("get_module_files", std::move(get_module_files));
}
@@ -530,8 +543,10 @@ void RegisterMgGetModuleFile(ModuleRegistry *module_registry, BuiltinModule *mod
};
mgp_proc get_module_file("get_module_file", std::move(get_module_file_cb), utils::NewDeleteResource(),
{.required_privilege = AuthQuery::Privilege::MODULE_READ});
MG_ASSERT(mgp_proc_add_arg(&get_module_file, "path", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&get_module_file, "content", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&get_module_file, "path", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&get_module_file, "content", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
module->AddProcedure("get_module_file", std::move(get_module_file));
}
@@ -609,9 +624,12 @@ void RegisterMgCreateModuleFile(ModuleRegistry *module_registry, utils::RWLock *
};
mgp_proc create_module_file("create_module_file", std::move(create_module_file_cb), utils::NewDeleteResource(),
{.required_privilege = AuthQuery::Privilege::MODULE_WRITE});
MG_ASSERT(mgp_proc_add_arg(&create_module_file, "filename", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&create_module_file, "content", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&create_module_file, "path", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&create_module_file, "filename", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&create_module_file, "content", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&create_module_file, "path", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
module->AddProcedure("create_module_file", std::move(create_module_file));
}
@@ -664,8 +682,10 @@ void RegisterMgUpdateModuleFile(ModuleRegistry *module_registry, utils::RWLock *
};
mgp_proc update_module_file("update_module_file", std::move(update_module_file_cb), utils::NewDeleteResource(),
{.required_privilege = AuthQuery::Privilege::MODULE_WRITE});
MG_ASSERT(mgp_proc_add_arg(&update_module_file, "path", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&update_module_file, "content", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&update_module_file, "path", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&update_module_file, "content", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
module->AddProcedure("update_module_file", std::move(update_module_file));
}
@@ -721,7 +741,8 @@ void RegisterMgDeleteModuleFile(ModuleRegistry *module_registry, utils::RWLock *
};
mgp_proc delete_module_file("delete_module_file", std::move(delete_module_file_cb), utils::NewDeleteResource(),
{.required_privilege = AuthQuery::Privilege::MODULE_WRITE});
MG_ASSERT(mgp_proc_add_arg(&delete_module_file, "path", Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&delete_module_file, "path", Call<mgp_type *>(mgp_type_string)) ==
mgp_error::MGP_ERROR_NO_ERROR);
module->AddProcedure("delete_module_file", std::move(delete_module_file));
}
@@ -801,7 +822,8 @@ bool SharedLibraryModule::Load(const std::filesystem::path &file_path) {
spdlog::info("Loading module {}...", file_path);
file_path_ = file_path;
dlerror(); // Clear any existing error.
handle_ = dlopen(file_path.c_str(), RTLD_NOW | RTLD_LOCAL);
// NOLINTNEXTLINE(hicpp-signed-bitwise)
handle_ = dlopen(file_path.c_str(), RTLD_NOW | RTLD_LOCAL | RTLD_DEEPBIND);
if (!handle_) {
spdlog::error(
utils::MessageWithLink("Unable to load module {}; {}.", file_path, dlerror(), "https://memgr.ph/modules"));
@@ -832,8 +854,8 @@ bool SharedLibraryModule::Load(const std::filesystem::path &file_path) {
return with_error(error);
}
for (auto &trans : module_def->transformations) {
const bool was_result_added = MgpTransAddFixedResult(&trans.second);
if (!was_result_added) {
const bool success = mgp_error::MGP_ERROR_NO_ERROR == MgpTransAddFixedResult(&trans.second);
if (!success) {
const auto error =
fmt::format("Unable to add result to transformation in module {}; add result failed", file_path);
return with_error(error);
@@ -941,7 +963,7 @@ bool PythonModule::Load(const std::filesystem::path &file_path) {
auto module_cb = [&](auto *module_def, auto * /*memory*/) {
auto result = ImportPyModule(file_path.stem().c_str(), module_def);
for (auto &trans : module_def->transformations) {
succ = MgpTransAddFixedResult(&trans.second) == MGP_ERROR_NO_ERROR;
succ = MgpTransAddFixedResult(&trans.second) == mgp_error::MGP_ERROR_NO_ERROR;
if (!succ) {
return result;
}

View File

@@ -13,6 +13,7 @@
/// API for loading and registering modules providing custom oC procedures
#pragma once
#include <dlfcn.h>
#include <filesystem>
#include <functional>
#include <optional>
@@ -128,6 +129,40 @@ class ModuleRegistry final {
const std::filesystem::path &InternalModuleDir() const noexcept;
private:
class SharedLibraryHandle {
public:
SharedLibraryHandle(const std::string &shared_library, int mode) : handle_{dlopen(shared_library.c_str(), mode)} {}
SharedLibraryHandle(const SharedLibraryHandle &) = delete;
SharedLibraryHandle(SharedLibraryHandle &&) = delete;
SharedLibraryHandle operator=(const SharedLibraryHandle &) = delete;
SharedLibraryHandle operator=(SharedLibraryHandle &&) = delete;
~SharedLibraryHandle() {
if (handle_) {
dlclose(handle_);
}
}
private:
void *handle_;
};
#if __has_feature(address_sanitizer)
// This is why we need RTLD_NODELETE and we must not use RTLD_DEEPBIND with
// ASAN: https://github.com/google/sanitizers/issues/89
SharedLibraryHandle libstd_handle{"libstdc++.so.6", RTLD_NOW | RTLD_LOCAL | RTLD_NODELETE};
#else
// The reason behind opening share library during runtime is to avoid issues
// with loading symbols from stdlib. We have encounter issues with locale
// that cause std::cout not being printed and issues when python libraries
// would call stdlib (e.g. pytorch).
// The way that those issues were solved was
// by using RTLD_DEEPBIND. RTLD_DEEPBIND ensures that the lookup for the
// mentioned library will be first performed in the already existing binded
// libraries and then the global namespace.
// RTLD_DEEPBIND => https://linux.die.net/man/3/dlopen
SharedLibraryHandle libstd_handle{"libstdc++.so.6", RTLD_NOW | RTLD_LOCAL | RTLD_DEEPBIND};
#endif
std::vector<std::filesystem::path> modules_dirs_;
std::filesystem::path internal_module_dir_;
};

View File

@@ -55,49 +55,49 @@ PyObject *gMgpSerializationError{nullptr}; // NOLINT(cppcoreguidelines-avo
// Returns true if an exception is raised
bool RaiseExceptionFromErrorCode(const mgp_error error) {
switch (error) {
case MGP_ERROR_NO_ERROR:
case mgp_error::MGP_ERROR_NO_ERROR:
return false;
case MGP_ERROR_UNKNOWN_ERROR: {
case mgp_error::MGP_ERROR_UNKNOWN_ERROR: {
PyErr_SetString(gMgpUnknownError, "Unknown error happened.");
return true;
}
case MGP_ERROR_UNABLE_TO_ALLOCATE: {
case mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE: {
PyErr_SetString(gMgpUnableToAllocateError, "Unable to allocate memory.");
return true;
}
case MGP_ERROR_INSUFFICIENT_BUFFER: {
case mgp_error::MGP_ERROR_INSUFFICIENT_BUFFER: {
PyErr_SetString(gMgpInsufficientBufferError, "Insufficient buffer.");
return true;
}
case MGP_ERROR_OUT_OF_RANGE: {
case mgp_error::MGP_ERROR_OUT_OF_RANGE: {
PyErr_SetString(gMgpOutOfRangeError, "Out of range.");
return true;
}
case MGP_ERROR_LOGIC_ERROR: {
case mgp_error::MGP_ERROR_LOGIC_ERROR: {
PyErr_SetString(gMgpLogicErrorError, "Logic error.");
return true;
}
case MGP_ERROR_DELETED_OBJECT: {
case mgp_error::MGP_ERROR_DELETED_OBJECT: {
PyErr_SetString(gMgpDeletedObjectError, "Accessing deleted object.");
return true;
}
case MGP_ERROR_INVALID_ARGUMENT: {
case mgp_error::MGP_ERROR_INVALID_ARGUMENT: {
PyErr_SetString(gMgpInvalidArgumentError, "Invalid argument.");
return true;
}
case MGP_ERROR_KEY_ALREADY_EXISTS: {
case mgp_error::MGP_ERROR_KEY_ALREADY_EXISTS: {
PyErr_SetString(gMgpKeyAlreadyExistsError, "Key already exists.");
return true;
}
case MGP_ERROR_IMMUTABLE_OBJECT: {
case mgp_error::MGP_ERROR_IMMUTABLE_OBJECT: {
PyErr_SetString(gMgpImmutableObjectError, "Cannot modify immutable object.");
return true;
}
case MGP_ERROR_VALUE_CONVERSION: {
case mgp_error::MGP_ERROR_VALUE_CONVERSION: {
PyErr_SetString(gMgpValueConversionError, "Value conversion failed.");
return true;
}
case MGP_ERROR_SERIALIZATION_ERROR: {
case mgp_error::MGP_ERROR_SERIALIZATION_ERROR: {
PyErr_SetString(gMgpSerializationError, "Operation cannot be serialized.");
return true;
}
@@ -902,7 +902,7 @@ std::optional<py::ExceptionInfo> AddRecordFromPython(mgp_result *result, py::Obj
if (field_val == nullptr) {
return py::FetchError();
}
if (mgp_result_record_insert(record, field_name, field_val) != MGP_ERROR_NO_ERROR) {
if (mgp_result_record_insert(record, field_name, field_val) != mgp_error::MGP_ERROR_NO_ERROR) {
std::stringstream ss;
ss << "Unable to insert field '" << py::Object::FromBorrow(key) << "' with value: '"
<< py::Object::FromBorrow(val) << "'; did you set the correct field type?";
@@ -2281,9 +2281,10 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
auto py_seq_to_list = [memory](PyObject *seq, Py_ssize_t len, const auto &py_seq_get_item) {
static_assert(std::numeric_limits<Py_ssize_t>::max() <= std::numeric_limits<size_t>::max());
MgpUniquePtr<mgp_list> list{nullptr, &mgp_list_destroy};
if (const auto err = CreateMgpObject(list, mgp_list_make_empty, len, memory); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = CreateMgpObject(list, mgp_list_make_empty, len, memory);
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during making mgp_list"};
}
for (Py_ssize_t i = 0; i < len; ++i) {
@@ -2292,17 +2293,17 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
v = PyObjectToMgpValue(e, memory);
const auto err = mgp_list_append(list.get(), v);
mgp_value_destroy(v);
if (err != MGP_ERROR_NO_ERROR) {
if (err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (err != mgp_error::MGP_ERROR_NO_ERROR) {
if (err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
}
throw std::runtime_error{"Unexpected error during appending to mgp_list"};
}
}
mgp_value *v{nullptr};
if (const auto err = mgp_value_make_list(list.get(), &v); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = mgp_value_make_list(list.get(), &v); err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during making mgp_value"};
}
static_cast<void>(list.release());
@@ -2334,7 +2335,7 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
};
mgp_value *mgp_v{nullptr};
mgp_error last_error{MGP_ERROR_NO_ERROR};
mgp_error last_error{mgp_error::MGP_ERROR_NO_ERROR};
if (o == Py_None) {
last_error = mgp_value_make_null(memory, &mgp_v);
@@ -2360,10 +2361,10 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
MgpUniquePtr<mgp_map> map{nullptr, mgp_map_destroy};
const auto map_err = CreateMgpObject(map, mgp_map_make_empty, memory);
if (map_err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (map_err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
}
if (map_err != MGP_ERROR_NO_ERROR) {
if (map_err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during creating mgp_map"};
}
@@ -2384,16 +2385,16 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
MgpUniquePtr<mgp_value> v{PyObjectToMgpValue(value, memory), mgp_value_destroy};
if (const auto err = mgp_map_insert(map.get(), k, v.get()); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = mgp_map_insert(map.get(), k, v.get()); err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during inserting an item to mgp_map"};
}
}
if (const auto err = mgp_value_make_map(map.get(), &mgp_v); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = mgp_value_make_map(map.get(), &mgp_v); err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during creating mgp_value"};
}
static_cast<void>(map.release());
@@ -2402,14 +2403,14 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
// Copy the edge and pass the ownership to the created mgp_value.
if (const auto err = CreateMgpObject(e, mgp_edge_copy, reinterpret_cast<PyEdge *>(o)->edge, memory);
err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during copying mgp_edge"};
}
if (const auto err = mgp_value_make_edge(e.get(), &mgp_v); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = mgp_value_make_edge(e.get(), &mgp_v); err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during copying mgp_edge"};
}
static_cast<void>(e.release());
@@ -2418,14 +2419,14 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
// Copy the edge and pass the ownership to the created mgp_value.
if (const auto err = CreateMgpObject(p, mgp_path_copy, reinterpret_cast<PyPath *>(o)->path, memory);
err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during copying mgp_path"};
}
if (const auto err = mgp_value_make_path(p.get(), &mgp_v); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = mgp_value_make_path(p.get(), &mgp_v); err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during copying mgp_path"};
}
static_cast<void>(p.release());
@@ -2434,14 +2435,14 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
// Copy the edge and pass the ownership to the created mgp_value.
if (const auto err = CreateMgpObject(v, mgp_vertex_copy, reinterpret_cast<PyVertex *>(o)->vertex, memory);
err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during copying mgp_vertex"};
}
if (const auto err = mgp_value_make_vertex(v.get(), &mgp_v); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = mgp_value_make_vertex(v.get(), &mgp_v); err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error during copying mgp_vertex"};
}
static_cast<void>(v.release());
@@ -2474,14 +2475,14 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
MgpUniquePtr<mgp_date> date{nullptr, mgp_date_destroy};
if (const auto err = CreateMgpObject(date, mgp_date_from_parameters, &parameters, memory);
err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error while creating mgp_date"};
}
if (const auto err = mgp_value_make_date(date.get(), &mgp_v); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = mgp_value_make_date(date.get(), &mgp_v); err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error while creating mgp_value"};
}
static_cast<void>(date.release());
@@ -2499,14 +2500,15 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
MgpUniquePtr<mgp_local_time> local_time{nullptr, mgp_local_time_destroy};
if (const auto err = CreateMgpObject(local_time, mgp_local_time_from_parameters, &parameters, memory);
err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error while creating mgp_local_time"};
}
if (const auto err = mgp_value_make_local_time(local_time.get(), &mgp_v); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = mgp_value_make_local_time(local_time.get(), &mgp_v);
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error while creating mgp_value"};
}
static_cast<void>(local_time.release());
@@ -2531,15 +2533,15 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
MgpUniquePtr<mgp_local_date_time> local_date_time{nullptr, mgp_local_date_time_destroy};
if (const auto err = CreateMgpObject(local_date_time, mgp_local_date_time_from_parameters, &parameters, memory);
err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error while creating mgp_local_date_time"};
}
if (const auto err = mgp_value_make_local_date_time(local_date_time.get(), &mgp_v);
err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error while creating mgp_value"};
}
static_cast<void>(local_date_time.release());
@@ -2558,14 +2560,15 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
MgpUniquePtr<mgp_duration> duration{nullptr, mgp_duration_destroy};
if (const auto err = CreateMgpObject(duration, mgp_duration_from_microseconds, microseconds, memory);
err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error while creating mgp_duration"};
}
if (const auto err = mgp_value_make_duration(duration.get(), &mgp_v); err == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (const auto err = mgp_value_make_duration(duration.get(), &mgp_v);
err == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
} else if (err != MGP_ERROR_NO_ERROR) {
} else if (err != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error while creating mgp_value"};
}
static_cast<void>(duration.release());
@@ -2573,10 +2576,10 @@ mgp_value *PyObjectToMgpValue(PyObject *o, mgp_memory *memory) {
throw std::invalid_argument("Unsupported PyObject conversion");
}
if (last_error == MGP_ERROR_UNABLE_TO_ALLOCATE) {
if (last_error == mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE) {
throw std::bad_alloc{};
}
if (last_error != MGP_ERROR_NO_ERROR) {
if (last_error != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error{"Unexpected error while creating mgp_value"};
}

View File

@@ -181,25 +181,27 @@ void Streams::RegisterKafkaProcedures() {
const auto offset = procedure::Call<int64_t>(mgp_value_get_int, arg_offset);
auto lock_ptr = streams_.Lock();
auto it = GetStream(*lock_ptr, std::string(stream_name));
std::visit(utils::Overloaded{
[&](StreamData<KafkaStream> &kafka_stream) {
auto stream_source_ptr = kafka_stream.stream_source->Lock();
const auto error = stream_source_ptr->SetStreamOffset(offset);
if (error.HasError()) {
MG_ASSERT(mgp_result_set_error_msg(result, error.GetError().c_str()) == MGP_ERROR_NO_ERROR,
"Unable to set procedure error message of procedure: {}", proc_name);
}
},
[](auto && /*other*/) {
throw QueryRuntimeException("'{}' can be only used for Kafka stream sources", proc_name);
}},
std::visit(utils::Overloaded{[&](StreamData<KafkaStream> &kafka_stream) {
auto stream_source_ptr = kafka_stream.stream_source->Lock();
const auto error = stream_source_ptr->SetStreamOffset(offset);
if (error.HasError()) {
MG_ASSERT(mgp_result_set_error_msg(result, error.GetError().c_str()) ==
mgp_error::MGP_ERROR_NO_ERROR,
"Unable to set procedure error message of procedure: {}", proc_name);
}
},
[](auto && /*other*/) {
throw QueryRuntimeException("'{}' can be only used for Kafka stream sources",
proc_name);
}},
it->second);
};
mgp_proc proc(proc_name, set_stream_offset, utils::NewDeleteResource());
MG_ASSERT(mgp_proc_add_arg(&proc, "stream_name", procedure::Call<mgp_type *>(mgp_type_string)) ==
MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&proc, "offset", procedure::Call<mgp_type *>(mgp_type_int)) == MGP_ERROR_NO_ERROR);
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_arg(&proc, "offset", procedure::Call<mgp_type *>(mgp_type_int)) ==
mgp_error::MGP_ERROR_NO_ERROR);
procedure::gModuleRegistry.RegisterMgProcedure(proc_name, std::move(proc));
}
@@ -345,19 +347,19 @@ void Streams::RegisterKafkaProcedures() {
mgp_proc proc(proc_name, get_stream_info, utils::NewDeleteResource());
MG_ASSERT(mgp_proc_add_arg(&proc, "stream_name", procedure::Call<mgp_type *>(mgp_type_string)) ==
MGP_ERROR_NO_ERROR);
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&proc, consumer_group_result_name.data(),
procedure::Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
procedure::Call<mgp_type *>(mgp_type_string)) == mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(
mgp_proc_add_result(&proc, topics_result_name.data(),
procedure::Call<mgp_type *>(mgp_type_list, procedure::Call<mgp_type *>(mgp_type_string))) ==
MGP_ERROR_NO_ERROR);
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&proc, bootstrap_servers_result_name.data(),
procedure::Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
procedure::Call<mgp_type *>(mgp_type_string)) == mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&proc, configs_result_name.data(), procedure::Call<mgp_type *>(mgp_type_map)) ==
MGP_ERROR_NO_ERROR);
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&proc, credentials_result_name.data(), procedure::Call<mgp_type *>(mgp_type_map)) ==
MGP_ERROR_NO_ERROR);
mgp_error::MGP_ERROR_NO_ERROR);
procedure::gModuleRegistry.RegisterMgProcedure(proc_name, std::move(proc));
}
@@ -432,14 +434,14 @@ void Streams::RegisterPulsarProcedures() {
mgp_proc proc(proc_name, get_stream_info, utils::NewDeleteResource());
MG_ASSERT(mgp_proc_add_arg(&proc, "stream_name", procedure::Call<mgp_type *>(mgp_type_string)) ==
MGP_ERROR_NO_ERROR);
mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(mgp_proc_add_result(&proc, service_url_result_name.data(),
procedure::Call<mgp_type *>(mgp_type_string)) == MGP_ERROR_NO_ERROR);
procedure::Call<mgp_type *>(mgp_type_string)) == mgp_error::MGP_ERROR_NO_ERROR);
MG_ASSERT(
mgp_proc_add_result(&proc, topics_result_name.data(),
procedure::Call<mgp_type *>(mgp_type_list, procedure::Call<mgp_type *>(mgp_type_string))) ==
MGP_ERROR_NO_ERROR);
mgp_error::MGP_ERROR_NO_ERROR);
procedure::gModuleRegistry.RegisterMgProcedure(proc_name, std::move(proc));
}

View File

@@ -97,6 +97,10 @@ class LabelIndex {
Iterator begin() { return Iterator(this, index_accessor_.begin()); }
Iterator end() { return Iterator(this, index_accessor_.end()); }
Iterator IterateFrom(Vertex *vertex) {
return Iterator(this, index_accessor_.find_equal_or_greater(Entry{vertex, 0}));
}
private:
utils::SkipList<Entry>::Accessor index_accessor_;
LabelId label_;

View File

@@ -16,6 +16,10 @@
namespace memgraph::storage::replication {
struct ReplicationClientConfig {
std::optional<double> timeout;
// The default delay between main checking/pinging replicas is 1s because
// that seems like a reasonable timeframe in which main should notice a
// replica is down.
std::chrono::seconds replica_check_frequency{1};
struct SSL {
std::string key_file = "";

View File

@@ -41,12 +41,49 @@ Storage::ReplicationClient::ReplicationClient(std::string name, Storage *storage
}
rpc_client_.emplace(endpoint, &*rpc_context_);
TryInitializeClient();
TryInitializeClientSync();
if (config.timeout && replica_state_ != replication::ReplicaState::INVALID) {
timeout_.emplace(*config.timeout);
timeout_dispatcher_.emplace();
}
// Help the user to get the most accurate replica state possible.
if (config.replica_check_frequency > std::chrono::seconds(0)) {
replica_checker_.Run("Replica Checker", config.replica_check_frequency, [&] { FrequentCheck(); });
}
}
void Storage::ReplicationClient::TryInitializeClientAsync() {
thread_pool_.AddTask([this] {
rpc_client_->Abort();
this->TryInitializeClientSync();
});
}
void Storage::ReplicationClient::FrequentCheck() {
const auto is_success = std::invoke([this]() {
try {
auto stream{rpc_client_->Stream<replication::FrequentHeartbeatRpc>()};
const auto response = stream.AwaitResponse();
return response.success;
} catch (const rpc::RpcFailedException &) {
return false;
}
});
// States: READY, REPLICATING, RECOVERY, INVALID
// If success && ready, replicating, recovery -> stay the same because something good is going on.
// If success && INVALID -> [it's possible that replica came back to life] -> TryInitializeClient.
// If fail -> [replica is not reachable at all] -> INVALID state.
// NOTE: TryInitializeClient might return nothing if there is a branching point.
// NOTE: The early return pattern simplified the code, but the behavior should be as explained.
if (!is_success) {
replica_state_.store(replication::ReplicaState::INVALID);
return;
}
if (replica_state_.load() == replication::ReplicaState::INVALID) {
TryInitializeClientAsync();
}
}
/// @throws rpc::RpcFailedException
@@ -100,7 +137,7 @@ void Storage::ReplicationClient::InitializeClient() {
}
}
void Storage::ReplicationClient::TryInitializeClient() {
void Storage::ReplicationClient::TryInitializeClientSync() {
try {
InitializeClient();
} catch (const rpc::RpcFailedException &) {
@@ -113,10 +150,7 @@ void Storage::ReplicationClient::TryInitializeClient() {
void Storage::ReplicationClient::HandleRpcFailure() {
spdlog::error(utils::MessageWithLink("Couldn't replicate data to {}.", name_, "https://memgr.ph/replication"));
thread_pool_.AddTask([this] {
rpc_client_->Abort();
this->TryInitializeClient();
});
TryInitializeClientAsync();
}
replication::SnapshotRes Storage::ReplicationClient::TransferSnapshot(const std::filesystem::path &path) {

View File

@@ -142,16 +142,14 @@ class Storage::ReplicationClient {
std::vector<RecoveryStep> GetRecoverySteps(uint64_t replica_commit, utils::FileRetainer::FileLocker *file_locker);
void FrequentCheck();
void InitializeClient();
void TryInitializeClient();
void TryInitializeClientSync();
void TryInitializeClientAsync();
void HandleRpcFailure();
std::string name_;
Storage *storage_;
std::optional<communication::ClientContext> rpc_context_;
std::optional<rpc::Client> rpc_client_;
@@ -198,6 +196,8 @@ class Storage::ReplicationClient {
// to ignore concurrency problems inside the client.
utils::ThreadPool thread_pool_{1};
std::atomic<replication::ReplicaState> replica_state_{replication::ReplicaState::INVALID};
utils::Scheduler replica_checker_;
};
} // namespace memgraph::storage

View File

@@ -60,6 +60,10 @@ Storage::ReplicationServer::ReplicationServer(Storage *storage, io::network::End
spdlog::debug("Received HeartbeatRpc");
this->HeartbeatHandler(req_reader, res_builder);
});
rpc_server_->Register<replication::FrequentHeartbeatRpc>([](auto *req_reader, auto *res_builder) {
spdlog::debug("Received FrequentHeartbeatRpc");
FrequentHeartbeatHandler(req_reader, res_builder);
});
rpc_server_->Register<replication::AppendDeltasRpc>([this](auto *req_reader, auto *res_builder) {
spdlog::debug("Received AppendDeltasRpc");
this->AppendDeltasHandler(req_reader, res_builder);
@@ -86,6 +90,13 @@ void Storage::ReplicationServer::HeartbeatHandler(slk::Reader *req_reader, slk::
slk::Save(res, res_builder);
}
void Storage::ReplicationServer::FrequentHeartbeatHandler(slk::Reader *req_reader, slk::Builder *res_builder) {
replication::FrequentHeartbeatReq req;
slk::Load(&req, req_reader);
replication::FrequentHeartbeatRes res{true};
slk::Save(res, res_builder);
}
void Storage::ReplicationServer::AppendDeltasHandler(slk::Reader *req_reader, slk::Builder *res_builder) {
replication::AppendDeltasReq req;
slk::Load(&req, req_reader);

View File

@@ -29,6 +29,7 @@ class Storage::ReplicationServer {
private:
// RPC handlers
void HeartbeatHandler(slk::Reader *req_reader, slk::Builder *res_builder);
static void FrequentHeartbeatHandler(slk::Reader *req_reader, slk::Builder *res_builder);
void AppendDeltasHandler(slk::Reader *req_reader, slk::Builder *res_builder);
void SnapshotHandler(slk::Reader *req_reader, slk::Builder *res_builder);
void WalFilesHandler(slk::Reader *req_reader, slk::Builder *res_builder);

View File

@@ -43,6 +43,12 @@ cpp<#
(current-commit-timestamp :uint64_t)
(epoch-id "std::string"))))
;; FrequentHearthbeat is required because calling Heartbeat takes the storage lock.
;; Configured by `replication_replica_check_delay`.
(lcp:define-rpc frequent-heartbeat
(:request ())
(:response ((success :bool))))
(lcp:define-rpc snapshot
(:request ())
(:response

View File

@@ -85,6 +85,7 @@ VerticesIterable::VerticesIterable(AllVerticesIterable vertices) : type_(Type::A
}
VerticesIterable::VerticesIterable(LabelIndex::Iterable vertices) : type_(Type::BY_LABEL) {
// TODO(antaljanosbenjamin): vertices_accessor_ should be initialized here
new (&vertices_by_label_) LabelIndex::Iterable(std::move(vertices));
}
@@ -169,6 +170,22 @@ VerticesIterable::Iterator VerticesIterable::end() {
}
}
VerticesIterable::Iterator VerticesIterable::IterateFrom(const Gid vertex_id) {
switch (type_) {
case Type::ALL:
return Iterator(all_vertices_.IterateFrom(vertex_id));
case Type::BY_LABEL: {
auto vertex_it = vertices_accessor_.find_equal_or_greater(vertex_id);
if (vertex_it == vertices_accessor_.end()) {
return end();
}
return Iterator(vertices_by_label_.IterateFrom(&(*vertex_it)));
}
case Type::BY_LABEL_PROPERTY:
LOG_FATAL("IterateFrom is not supported for label property indices!");
}
}
VerticesIterable::Iterator::Iterator(AllVerticesIterable::Iterator it) : type_(Type::ALL) {
new (&all_it_) AllVerticesIterable::Iterator(std::move(it));
}

View File

@@ -94,8 +94,10 @@ class AllVerticesIterable final {
constraints_(constraints),
config_(config) {}
Iterator begin() { return Iterator(this, vertices_accessor_.begin()); }
Iterator end() { return Iterator(this, vertices_accessor_.end()); }
Iterator begin() { return Iterator{this, vertices_accessor_.begin()}; }
Iterator end() { return Iterator{this, vertices_accessor_.end()}; }
Iterator IterateFrom(Gid vertex_id) { return Iterator{this, vertices_accessor_.find_equal_or_greater(vertex_id)}; }
};
/// Generic access to different kinds of vertex iterations.
@@ -108,8 +110,13 @@ class VerticesIterable final {
Type type_;
union {
AllVerticesIterable all_vertices_;
LabelIndex::Iterable vertices_by_label_;
LabelPropertyIndex::Iterable vertices_by_label_property_;
struct {
utils::SkipList<Vertex>::Accessor vertices_accessor_;
union {
LabelIndex::Iterable vertices_by_label_;
LabelPropertyIndex::Iterable vertices_by_label_property_;
};
};
};
public:
@@ -158,6 +165,9 @@ class VerticesIterable final {
Iterator begin();
Iterator end();
// TODO(antaljanosbenjamin): write tests for this
Iterator IterateFrom(Gid vertex_id);
};
/// Structure used to return information about existing indices in the storage.

View File

@@ -15,8 +15,8 @@ namespace memgraph::storage {
/// Indicator for obtaining the state before or after a transaction & command.
enum class View {
OLD,
NEW,
OLD = 0,
NEW = 1,
};
} // namespace memgraph::storage

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2021 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -1,3 +1,11 @@
# Set up C++ functions for e2e tests
function(add_query_module target_name src)
add_library(${target_name} SHARED ${src})
SET_TARGET_PROPERTIES(${target_name} PROPERTIES PREFIX "")
target_include_directories(${target_name} PRIVATE ${CMAKE_SOURCE_DIR}/include)
endfunction()
function(copy_e2e_python_files TARGET_PREFIX FILE_NAME)
add_custom_target(memgraph__e2e__${TARGET_PREFIX}__${FILE_NAME} ALL
COMMAND ${CMAKE_COMMAND} -E copy
@@ -14,6 +22,7 @@ add_custom_target(memgraph__e2e__${TARGET_PREFIX}__${FILE_NAME} ALL
DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/${FILE_NAME})
endfunction()
add_subdirectory(server)
add_subdirectory(replication)
add_subdirectory(memory)
add_subdirectory(triggers)
@@ -23,6 +32,8 @@ add_subdirectory(temporal_types)
add_subdirectory(write_procedures)
add_subdirectory(magic_functions)
add_subdirectory(module_file_manager)
add_subdirectory(websocket)
add_subdirectory(monitoring_server)
copy_e2e_python_files(pytest_runner pytest_runner.sh "")
file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/memgraph-selfsigned.crt DESTINATION ${CMAKE_CURRENT_BINARY_DIR})
file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/memgraph-selfsigned.key DESTINATION ${CMAKE_CURRENT_BINARY_DIR})

View File

@@ -1,10 +1,3 @@
# Set up C++ functions for e2e tests
function(add_query_module target_name src)
add_library(${target_name} SHARED ${src})
SET_TARGET_PROPERTIES(${target_name} PROPERTIES PREFIX "")
target_include_directories(${target_name} PRIVATE ${CMAKE_SOURCE_DIR}/include)
endfunction()
# Set up Python functions for e2e tests
function(copy_magic_functions_e2e_python_files FILE_NAME)
copy_e2e_python_files(functions ${FILE_NAME})

View File

@@ -21,13 +21,13 @@ static void ReturnFunctionArgument(struct mgp_list *args, mgp_func_context *ctx,
struct mgp_memory *memory) {
mgp_value *value{nullptr};
auto err_code = mgp_list_at(args, 0, &value);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
mgp_func_result_set_error_msg(result, "Failed to fetch list!", memory);
return;
}
err_code = mgp_func_result_set_value(result, value, memory);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
mgp_func_result_set_error_msg(result, "Failed to construct return value!", memory);
return;
}
@@ -37,13 +37,13 @@ static void ReturnOptionalArgument(struct mgp_list *args, mgp_func_context *ctx,
struct mgp_memory *memory) {
mgp_value *value{nullptr};
auto err_code = mgp_list_at(args, 0, &value);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
mgp_func_result_set_error_msg(result, "Failed to fetch list!", memory);
return;
}
err_code = mgp_func_result_set_value(result, value, memory);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
mgp_func_result_set_error_msg(result, "Failed to construct return value!", memory);
return;
}
@@ -51,7 +51,7 @@ static void ReturnOptionalArgument(struct mgp_list *args, mgp_func_context *ctx,
double GetElementFromArg(struct mgp_list *args, int index) {
mgp_value *value{nullptr};
if (mgp_list_at(args, index, &value) != MGP_ERROR_NO_ERROR) {
if (mgp_list_at(args, index, &value) != mgp_error::MGP_ERROR_NO_ERROR) {
throw std::runtime_error("Error while argument fetching.");
}
@@ -87,7 +87,7 @@ static void AddTwoNumbers(struct mgp_list *args, mgp_func_context *ctx, mgp_func
memgraph::utils::OnScopeExit delete_summation_value([&value] { mgp_value_destroy(value); });
auto err_code = mgp_func_result_set_value(result, value, memory);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
mgp_func_result_set_error_msg(result, "Failed to construct return value!", memory);
}
}
@@ -99,7 +99,7 @@ static void ReturnNull(struct mgp_list *args, mgp_func_context *ctx, mgp_func_re
memgraph::utils::OnScopeExit delete_null([&value] { mgp_value_destroy(value); });
auto err_code = mgp_func_result_set_value(result, value, memory);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
mgp_func_result_set_error_msg(result, "Failed to fetch list!", memory);
}
}
@@ -111,14 +111,14 @@ extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *mem
{
mgp_func *func{nullptr};
auto err_code = mgp_module_add_function(module, "return_function_argument", ReturnFunctionArgument, &func);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
mgp_type *type_any{nullptr};
mgp_type_any(&type_any);
err_code = mgp_func_add_arg(func, "argument", type_any);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
}
@@ -126,7 +126,7 @@ extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *mem
{
mgp_func *func{nullptr};
auto err_code = mgp_module_add_function(module, "return_optional_argument", ReturnOptionalArgument, &func);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
@@ -137,7 +137,7 @@ extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *mem
mgp_type *type_int{nullptr};
mgp_type_int(&type_int);
err_code = mgp_func_add_opt_arg(func, "opt_argument", type_int, default_value);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
}
@@ -145,18 +145,18 @@ extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *mem
{
mgp_func *func{nullptr};
auto err_code = mgp_module_add_function(module, "add_two_numbers", AddTwoNumbers, &func);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
mgp_type *type_number{nullptr};
mgp_type_number(&type_number);
err_code = mgp_func_add_arg(func, "first", type_number);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
err_code = mgp_func_add_arg(func, "second", type_number);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
}
@@ -164,7 +164,7 @@ extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *mem
{
mgp_func *func{nullptr};
auto err_code = mgp_module_add_function(module, "return_null", ReturnNull, &func);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
}

View File

@@ -26,13 +26,13 @@ static void TryToWrite(struct mgp_list *args, mgp_func_context *ctx, mgp_func_re
// Setting a property should set an error
auto err_code = mgp_vertex_set_property(vertex, name, value);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
mgp_func_result_set_error_msg(result, "Cannot set property in the function!", memory);
return;
}
err_code = mgp_func_result_set_value(result, value, memory);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
mgp_func_result_set_error_msg(result, "Failed to construct return value!", memory);
return;
}
@@ -44,21 +44,21 @@ extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *mem
{
mgp_func *func{nullptr};
auto err_code = mgp_module_add_function(module, "try_to_write", TryToWrite, &func);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
mgp_type *type_vertex{nullptr};
mgp_type_node(&type_vertex);
err_code = mgp_func_add_arg(func, "argument", type_vertex);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
mgp_type *type_string{nullptr};
mgp_type_string(&type_string);
err_code = mgp_func_add_arg(func, "name", type_string);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
@@ -67,7 +67,7 @@ extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *mem
mgp_type *nullable_type{nullptr};
mgp_type_nullable(any_type, &nullable_type);
err_code = mgp_func_add_arg(func, "value", nullable_type);
if (err_code != MGP_ERROR_NO_ERROR) {
if (err_code != mgp_error::MGP_ERROR_NO_ERROR) {
return 1;
}
}

View File

@@ -0,0 +1,8 @@
find_package(gflags REQUIRED)
find_package(Boost REQUIRED)
add_executable(memgraph__e2e__monitoring_server monitoring.cpp)
target_link_libraries(memgraph__e2e__monitoring_server mgclient mg-utils json gflags Boost::headers)
add_executable(memgraph__e2e__monitoring_server_ssl monitoring_ssl.cpp)
target_link_libraries(memgraph__e2e__monitoring_server_ssl mgclient mg-utils json gflags Boost::headers)

View File

@@ -1,15 +1,15 @@
cert_file: &cert_file "$PROJECT_DIR/tests/e2e/websocket/memgraph-selfsigned.crt"
key_file: &key_file "$PROJECT_DIR/tests/e2e/websocket/memgraph-selfsigned.key"
cert_file: &cert_file "$PROJECT_DIR/tests/e2e/memgraph-selfsigned.crt"
key_file: &key_file "$PROJECT_DIR/tests/e2e/memgraph-selfsigned.key"
bolt_port: &bolt_port "7687"
monitoring_port: &monitoring_port "7444"
template_cluster: &template_cluster
cluster:
websocket:
monitoring:
args: ["--bolt-port=7687", "--log-level=TRACE", "--"]
log_file: "websocket-e2e.log"
log_file: "monitoring-websocket-e2e.log"
template_cluster_ssl: &template_cluster_ssl
cluster:
websocket:
monitoring:
args:
[
"--bolt-port",
@@ -23,16 +23,15 @@ template_cluster_ssl: &template_cluster_ssl
*key_file,
"--",
]
log_file: "websocket-ssl-e2e.log"
log_file: "monitoring-websocket-ssl-e2e.log"
ssl: true
workloads:
- name: "Websocket"
binary: "tests/e2e/websocket/memgraph__e2e__websocket"
- name: "Monitoring server using WebSocket"
binary: "tests/e2e/monitoring_server/memgraph__e2e__monitoring_server"
args: ["--bolt-port", *bolt_port, "--monitoring-port", *monitoring_port]
<<: *template_cluster
- name: "Websocket SSL"
binary: "tests/e2e/websocket/memgraph__e2e__websocket_ssl"
- name: "Monitoring server using WebSocket SSL"
binary: "tests/e2e/monitoring_server/memgraph__e2e__monitoring_server_ssl"
args: ["--bolt-port", *bolt_port, "--monitoring-port", *monitoring_port]
<<: *template_cluster_ssl

View File

@@ -5,3 +5,7 @@ target_link_libraries(memgraph__e2e__replication__constraints gflags mgclient mg
add_executable(memgraph__e2e__replication__read_write_benchmark read_write_benchmark.cpp)
target_link_libraries(memgraph__e2e__replication__read_write_benchmark gflags json mgclient mg-utils mg-io Threads::Threads)
copy_e2e_python_files(replication_show common.py)
copy_e2e_python_files(replication_show conftest.py)
copy_e2e_python_files(replication_show show.py)

View File

@@ -0,0 +1,26 @@
# Copyright 2022 Memgraph Ltd.
#
# Use of this software is governed by the Business Source License
# included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
# License, and you may not use this file except in compliance with the Business Source License.
#
# As of the Change Date specified in that file, in accordance with
# the Business Source License, use of this software will be governed
# by the Apache License, Version 2.0, included in the file
# licenses/APL.txt.
import mgclient
import typing
def execute_and_fetch_all(
cursor: mgclient.Cursor, query: str, params: dict = {}
) -> typing.List[tuple]:
cursor.execute(query, params)
return cursor.fetchall()
def connect(**kwargs) -> mgclient.Connection:
connection = mgclient.connect(**kwargs)
connection.autocommit = True
return connection

View File

@@ -0,0 +1,44 @@
# Copyright 2022 Memgraph Ltd.
#
# Use of this software is governed by the Business Source License
# included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
# License, and you may not use this file except in compliance with the Business Source License.
#
# As of the Change Date specified in that file, in accordance with
# the Business Source License, use of this software will be governed
# by the Apache License, Version 2.0, included in the file
# licenses/APL.txt.
import pytest
from common import execute_and_fetch_all, connect
# The fixture here is more complex because the connection has to be
# parameterized based on the test parameters (info has to be available on both
# sides).
#
# https://docs.pytest.org/en/latest/example/parametrize.html#indirect-parametrization
# is not an elegant/feasible solution here.
#
# The solution was independently developed and then I stumbled upon the same
# approach here https://stackoverflow.com/a/68286553/4888809 which I think is
# optimal.
@pytest.fixture(scope="function")
def connection():
connection_holder = None
role_holder = None
def inner_connection(port, role):
nonlocal connection_holder, role_holder
connection_holder = connect(host="localhost", port=port)
role_holder = role
return connection_holder
yield inner_connection
# Only main instance can be cleaned up because replicas do NOT accept
# writes.
if role_holder == "main":
cursor = connection_holder.cursor()
execute_and_fetch_all(cursor, "MATCH (n) DETACH DELETE n;")

46
tests/e2e/replication/show.py Executable file
View File

@@ -0,0 +1,46 @@
# Copyright 2022 Memgraph Ltd.
#
# Use of this software is governed by the Business Source License
# included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
# License, and you may not use this file except in compliance with the Business Source License.
#
# As of the Change Date specified in that file, in accordance with
# the Business Source License, use of this software will be governed
# by the Apache License, Version 2.0, included in the file
# licenses/APL.txt.
import sys
import pytest
from common import execute_and_fetch_all
@pytest.mark.parametrize(
"port, role",
[(7687, "main"), (7688, "replica"), (7689, "replica"), (7690, "replica")],
)
def test_show_replication_role(port, role, connection):
cursor = connection(port, role).cursor()
data = execute_and_fetch_all(cursor, "SHOW REPLICATION ROLE;")
assert cursor.description[0].name == "replication role"
assert data[0][0] == role
def test_show_replicas(connection):
cursor = connection(7687, "main").cursor()
actual_data = set(execute_and_fetch_all(cursor, "SHOW REPLICAS;"))
expected_column_names = {"name", "socket_address", "sync_mode", "timeout"}
actual_column_names = {x.name for x in cursor.description}
assert expected_column_names == actual_column_names
expected_data = {
("replica_1", "127.0.0.1:10001", "sync", 0),
("replica_2", "127.0.0.1:10002", "sync", 1.0),
("replica_3", "127.0.0.1:10003", "async", None),
}
assert expected_data == actual_data
if __name__ == "__main__":
sys.exit(pytest.main([__file__, "-rA"]))

View File

@@ -46,4 +46,31 @@ workloads:
args: []
<<: *template_cluster
- name: "Show"
binary: "tests/e2e/pytest_runner.sh"
args: ["replication/show.py"]
cluster:
replica_1:
args: ["--bolt-port", "7688", "--log-level=TRACE"]
log_file: "replication-e2e-replica1.log"
setup_queries: ["SET REPLICATION ROLE TO REPLICA WITH PORT 10001;"]
validation_queries: []
replica_2:
args: ["--bolt-port", "7689", "--log-level=TRACE"]
log_file: "replication-e2e-replica2.log"
setup_queries: ["SET REPLICATION ROLE TO REPLICA WITH PORT 10002;"]
validation_queries: []
replica_3:
args: ["--bolt-port", "7690", "--log-level=TRACE"]
log_file: "replication-e2e-replica3.log"
setup_queries: ["SET REPLICATION ROLE TO REPLICA WITH PORT 10003;"]
validation_queries: []
main:
args: ["--bolt-port", "7687", "--log-level=TRACE"]
log_file: "replication-e2e-main.log"
setup_queries: [
"REGISTER REPLICA replica_1 SYNC WITH TIMEOUT 0 TO '127.0.0.1:10001'",
"REGISTER REPLICA replica_2 SYNC WITH TIMEOUT 1 TO '127.0.0.1:10002'",
"REGISTER REPLICA replica_3 ASYNC TO '127.0.0.1:10003'"
]
validation_queries: []

View File

@@ -0,0 +1,8 @@
find_package(gflags REQUIRED)
find_package(Boost REQUIRED)
add_executable(memgraph__e2e__server_connection server_connection.cpp)
target_link_libraries(memgraph__e2e__server_connection mgclient mg-utils gflags)
add_executable(memgraph__e2e__server_ssl_connection server_ssl_connection.cpp)
target_link_libraries(memgraph__e2e__server_ssl_connection mgclient mg-utils gflags)

View File

@@ -0,0 +1,60 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <chrono>
#include <functional>
#include <thread>
#include <spdlog/spdlog.h>
#include <boost/asio/io_context.hpp>
#include <boost/asio/steady_timer.hpp>
#include <boost/system/detail/error_code.hpp>
#include <mgclient.hpp>
#include "utils/logging.hpp"
inline void OnTimeoutExpiration(const boost::system::error_code &ec) {
// Timer was not cancelled, take necessary action.
MG_ASSERT(!!ec, "Connection timeout");
}
inline void EstablishConnection(const uint16_t bolt_port, const bool use_ssl) {
spdlog::info("Testing successfull connection from one client");
mg::Client::Init();
boost::asio::io_context ioc;
boost::asio::steady_timer timer(ioc, std::chrono::seconds(5));
timer.async_wait(std::bind_front(&OnTimeoutExpiration));
std::jthread bg_thread([&ioc]() { ioc.run(); });
auto client = mg::Client::Connect({.host = "127.0.0.1", .port = bolt_port, .use_ssl = use_ssl});
MG_ASSERT(client, "Failed to connect!");
timer.cancel();
}
inline void EstablishMultipleConnections(const uint16_t bolt_port, const bool use_ssl) {
spdlog::info("Testing successfull connection from multiple clients");
mg::Client::Init();
boost::asio::io_context ioc;
boost::asio::steady_timer timer(ioc, std::chrono::seconds(5));
timer.async_wait(std::bind_front(&OnTimeoutExpiration));
std::jthread bg_thread([&ioc]() { ioc.run(); });
auto client1 = mg::Client::Connect({.host = "127.0.0.1", .port = bolt_port, .use_ssl = use_ssl});
auto client2 = mg::Client::Connect({.host = "127.0.0.1", .port = bolt_port, .use_ssl = use_ssl});
auto client3 = mg::Client::Connect({.host = "127.0.0.1", .port = bolt_port, .use_ssl = use_ssl});
MG_ASSERT(client1, "Failed to connect!");
MG_ASSERT(client2, "Failed to connect!");
MG_ASSERT(client3, "Failed to connect!");
timer.cancel();
}

View File

@@ -0,0 +1,56 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <unistd.h>
#include <chrono>
#include <cstddef>
#include <gflags/gflags.h>
#include <spdlog/spdlog.h>
#include <boost/asio/io_context.hpp>
#include <boost/asio/steady_timer.hpp>
#include <boost/system/detail/error_code.hpp>
#include <mgclient.hpp>
#include "common.hpp"
#include "utils/logging.hpp"
DEFINE_uint64(bolt_port, 7687, "Bolt port");
void EstablishSSLConnectionToNonSSLServer(const auto bolt_port) {
spdlog::info("Testing that connection fails when connecting to non SSL server while using SSL");
mg::Client::Init();
boost::asio::io_context ioc;
boost::asio::steady_timer timer(ioc, std::chrono::seconds(5));
timer.async_wait(std::bind_front(&OnTimeoutExpiration));
std::jthread bg_thread([&ioc]() { ioc.run(); });
auto client = mg::Client::Connect({.host = "127.0.0.1", .port = bolt_port, .use_ssl = true});
MG_ASSERT(client == nullptr, "Connection not refused when connecting with SSL turned on to a non SSL server!");
timer.cancel();
}
int main(int argc, char **argv) {
google::SetUsageMessage("Memgraph E2E server connection!");
gflags::ParseCommandLineFlags(&argc, &argv, true);
MG_ASSERT(FLAGS_bolt_port != 0);
memgraph::logging::RedirectToStderr();
const auto bolt_port = static_cast<uint16_t>(FLAGS_bolt_port);
EstablishConnection(bolt_port, false);
EstablishMultipleConnections(bolt_port, false);
EstablishSSLConnectionToNonSSLServer(bolt_port);
return 0;
}

View File

@@ -0,0 +1,57 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <unistd.h>
#include <chrono>
#include <cstddef>
#include <thread>
#include <gflags/gflags.h>
#include <spdlog/spdlog.h>
#include <boost/asio/io_context.hpp>
#include <boost/asio/steady_timer.hpp>
#include <boost/system/detail/error_code.hpp>
#include <mgclient.hpp>
#include "common.hpp"
#include "utils/logging.hpp"
DEFINE_uint64(bolt_port, 7687, "Bolt port");
void EstablishNonSSLConnectionToSSLServer(const auto bolt_port) {
spdlog::info("Testing that connection fails when connecting to SSL server without using SSL");
mg::Client::Init();
boost::asio::io_context ioc;
boost::asio::steady_timer timer(ioc, std::chrono::seconds(5));
timer.async_wait(std::bind_front(&OnTimeoutExpiration));
std::jthread bg_thread([&ioc]() { ioc.run(); });
auto client = mg::Client::Connect({.host = "127.0.0.1", .port = bolt_port, .use_ssl = false});
MG_ASSERT(client == nullptr, "Connection not refused when conneting without SSL turned on to a SSL server!");
timer.cancel();
}
int main(int argc, char **argv) {
google::SetUsageMessage("Memgraph E2E server SSL connection!");
gflags::ParseCommandLineFlags(&argc, &argv, true);
MG_ASSERT(FLAGS_bolt_port != 0);
memgraph::logging::RedirectToStderr();
const auto bolt_port = static_cast<uint16_t>(FLAGS_bolt_port);
EstablishConnection(bolt_port, true);
EstablishMultipleConnections(bolt_port, true);
EstablishNonSSLConnectionToSSLServer(bolt_port);
return 0;
}

View File

@@ -0,0 +1,34 @@
cert_file: &cert_file "$PROJECT_DIR/tests/e2e/memgraph-selfsigned.crt"
key_file: &key_file "$PROJECT_DIR/tests/e2e/memgraph-selfsigned.key"
bolt_port: &bolt_port "7687"
template_cluster: &template_cluster
cluster:
server:
args: ["--bolt-port=7687", "--log-level=TRACE", "--"]
log_file: "server-connection-e2e.log"
template_cluster_ssl: &template_cluster_ssl
cluster:
server:
args:
[
"--bolt-port",
*bolt_port,
"--log-level=TRACE",
"--bolt-cert-file",
*cert_file,
"--bolt-key-file",
*key_file,
"--",
]
log_file: "server-connection-ssl-e2e.log"
ssl: true
workloads:
- name: "Server connection"
binary: "tests/e2e/server/memgraph__e2e__server_connection"
args: ["--bolt-port", *bolt_port]
<<: *template_cluster
- name: "Server SSL connection"
binary: "tests/e2e/server/memgraph__e2e__server_ssl_connection"
args: ["--bolt-port", *bolt_port]
<<: *template_cluster_ssl

View File

@@ -0,0 +1,8 @@
There are three docker-compose files in this directory:
* [kafka.yml](kafka.yml)
* [pulsar.yml](pulsar.yml)
* [redpanda.yml](redpanda.yml)
To run one of them, use the `docker-compose -f <filename> up -V` command. Optionally you can append `-d` to detach from the started containers. You can stop the detach containers by `docker-compose -f <filename> down`.
If you experience strange errors, try to clean up the previously created containers by `docker-compose -f <filename> rm -svf`.

View File

@@ -1,13 +1,13 @@
version: "3"
version: '3.7'
services:
zookeeper:
image: 'bitnami/zookeeper:3.6.3-debian-10-r33'
image: 'bitnami/zookeeper:latest'
ports:
- '2181:2181'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: 'bitnami/kafka:2.8.0-debian-10-r49'
image: 'bitnami/kafka:latest'
ports:
- '9092:9092'
environment:
@@ -18,9 +18,3 @@ services:
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper
pulsar:
image: 'apachepulsar/pulsar:2.8.1'
ports:
- '6652:8080'
- '6650:6650'
entrypoint: ['bin/pulsar', 'standalone']

View File

@@ -18,12 +18,14 @@ import time
from multiprocessing import Process, Value
import common
TRANSFORMATIONS_TO_CHECK = [
TRANSFORMATIONS_TO_CHECK_C = [
"empty_transformation"]
TRANSFORMATIONS_TO_CHECK_PY = [
"kafka_transform.simple",
"kafka_transform.with_parameters"]
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK_PY)
def test_simple(kafka_producer, kafka_topics, connection, transformation):
assert len(kafka_topics) > 0
cursor = connection.cursor()
@@ -44,7 +46,7 @@ def test_simple(kafka_producer, kafka_topics, connection, transformation):
cursor, topic, common.SIMPLE_MSG)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK_PY)
def test_separate_consumers(
kafka_producer,
kafka_topics,
@@ -125,7 +127,7 @@ def test_start_from_last_committed_offset(
cursor, kafka_topics[0], message)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK_PY)
def test_check_stream(
kafka_producer,
kafka_topics,
@@ -307,7 +309,7 @@ def test_restart_after_error(kafka_producer, kafka_topics, connection):
cursor, "MATCH (n:VERTEX { id : 42 }) RETURN n")
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK_PY)
def test_bootstrap_server(
kafka_producer,
kafka_topics,
@@ -334,7 +336,7 @@ def test_bootstrap_server(
cursor, topic, common.SIMPLE_MSG)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK_PY)
def test_bootstrap_server_empty(
kafka_producer,
kafka_topics,
@@ -352,7 +354,7 @@ def test_bootstrap_server_empty(
)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK)
@pytest.mark.parametrize("transformation", TRANSFORMATIONS_TO_CHECK_PY)
def test_set_offset(kafka_producer, kafka_topics, connection, transformation):
assert len(kafka_topics) > 0
cursor = connection.cursor()
@@ -450,6 +452,14 @@ def test_info_procedure(kafka_topics, connection):
(local, configs, consumer_group, reducted_credentials, kafka_topics)]
common.validate_info(stream_info, expected_stream_info)
@pytest.mark.parametrize("transformation",TRANSFORMATIONS_TO_CHECK_C)
def test_load_c_transformations(connection, transformation):
cursor = connection.cursor()
query = "CALL mg.transformations() YIELD * WITH name WHERE name STARTS WITH 'c_transformations." + transformation + "' RETURN name"
result = common.execute_and_fetch_all(
cursor, query)
assert len(result) == 1
assert result[0][0] == "c_transformations." + transformation
if __name__ == "__main__":
sys.exit(pytest.main([__file__, "-rA"]))

View File

@@ -0,0 +1,8 @@
version: '3.7'
services:
pulsar:
image: 'apachepulsar/pulsar:latest'
ports:
- '6652:8080'
- '6650:6650'
entrypoint: ['bin/pulsar', 'standalone']

View File

@@ -0,0 +1,23 @@
version: '3.7'
services:
redpanda:
command:
- redpanda
- start
- --smp
- '1'
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- '0'
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
# NOTE: Please use the latest version here!
image: docker.vectorized.io/vectorized/redpanda:latest
container_name: redpanda-1
ports:
- 9092:9092
- 29092:29092

View File

@@ -1,2 +1,3 @@
copy_streams_e2e_python_files(kafka_transform.py)
copy_streams_e2e_python_files(pulsar_transform.py)
add_query_module(c_transformations c_transformations.cpp)

View File

@@ -0,0 +1,22 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "mg_procedure.h"
extern "C" int mgp_init_module(mgp_module *module, mgp_memory *memory) {
static const auto no_op_cb = [](mgp_messages *msg, mgp_graph *graph, mgp_result *result, mgp_memory *memory) {};
if (mgp_error::MGP_ERROR_NO_ERROR != mgp_module_add_transformation(module, "empty_transformation", no_op_cb)) {
return 1;
}
return 0;
}

View File

@@ -1,10 +0,0 @@
find_package(gflags REQUIRED)
find_package(Boost REQUIRED)
add_executable(memgraph__e2e__websocket websocket.cpp)
target_link_libraries(memgraph__e2e__websocket mgclient mg-utils json gflags Boost::headers)
add_executable(memgraph__e2e__websocket_ssl websocket_ssl.cpp)
target_link_libraries(memgraph__e2e__websocket_ssl mgclient mg-utils json gflags Boost::headers)
file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/memgraph-selfsigned.crt DESTINATION ${CMAKE_CURRENT_BINARY_DIR})
file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/memgraph-selfsigned.key DESTINATION ${CMAKE_CURRENT_BINARY_DIR})

View File

@@ -55,3 +55,14 @@ target_link_libraries(${test_prefix}ssl_client mg-communication)
add_manual_test(ssl_server.cpp)
target_link_libraries(${test_prefix}ssl_server mg-communication)
add_manual_test(storage_server.cpp storage_service.cpp storage_service.hpp)
target_link_libraries(${test_prefix}storage_server mg-query mg-interface-storage-cpp2)
add_manual_test(storage_client_demo.cpp)
target_link_libraries(${test_prefix}storage_client_demo mg-interface-storage-cpp2 mg-utils)
add_manual_test(trial_server.cpp)
target_link_libraries(${test_prefix}trial_server mg-interface-trial-cpp2)
add_manual_test(trial_client.cpp)
target_link_libraries(${test_prefix}trial_client mg-interface-trial-cpp2)

View File

@@ -0,0 +1,228 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <chrono>
#include <iostream>
#include <optional>
#include <ratio>
#include <thread>
#include <folly/Executor.h>
#include <folly/Unit.h>
#include <folly/executors/IOThreadPoolExecutor.h>
#include <folly/executors/ThreadedExecutor.h>
#include <folly/executors/thread_factory/NamedThreadFactory.h>
#include <folly/init/Init.h>
#include <folly/io/async/AsyncSocket.h>
#include <folly/io/async/EventBase.h>
#include <folly/io/async/ScopedEventBaseThread.h>
#include <thrift/lib/cpp2/async/ClientStreamBridge.h>
#include <thrift/lib/cpp2/async/HeaderClientChannel.h>
#include <thrift/lib/cpp2/async/RocketClientChannel.h>
#include "interface/gen-cpp2/StorageAsyncClient.h"
#include "interface/gen-cpp2/storage_types.h"
#include "storage/v2/view.hpp"
#include "utils/logging.hpp"
DEFINE_string(host, "127.0.0.1", "Storage Server host");
DEFINE_int32(port, 7779, "Storage Server port");
using interface::storage::CreateVerticesRequest;
using interface::storage::StorageAsyncClient;
using interface::storage::Value;
using interface::storage::Vertex;
void Print(std::ostream &os, const Value &value) {
switch (value.getType()) {
case Value::Type::null_v:
os << "<null>";
break;
case Value::Type::bool_v:
os << value.get_bool_v();
break;
case Value::Type::int_v:
os << value.get_int_v();
break;
case Value::Type::string_v:
os << value.get_string_v();
break;
default:
os << "UNKNOWN TYPE";
break;
}
}
std::optional<int64_t> ScanVertices(const std::shared_ptr<StorageAsyncClient> &client, int64_t transaction_id,
const std::optional<int64_t> &start_id,
const std::optional<std::vector<std::string>> &props_to_return,
const std::optional<int64_t> &limit,
const memgraph::storage::View view = memgraph::storage::View::NEW) {
interface::storage::ScanVerticesRequest req;
req.transaction_id_ref() = transaction_id;
req.view_ref() = view;
if (limit.has_value()) {
req.limit_ref() = limit.value();
}
if (start_id.has_value()) {
req.start_id_ref() = start_id.value();
}
if (props_to_return.has_value()) {
req.props_to_return_ref() = props_to_return.value();
}
return client->future_scanVertices(req)
.then(
[&props_to_return](folly::Try<interface::storage::ScanVerticesResponse> &&result) -> std::optional<int64_t> {
if (result.hasException()) {
LOG(INFO) << "FAILED: " << result.exception().get_exception()->what() << std::endl;
return std::nullopt;
}
if (!*result->result_ref()->success()) {
throw std::runtime_error("scan vertices failed");
}
const auto &response = result.value();
MG_ASSERT(props_to_return.has_value() != response.property_name_map_ref().has_value());
if (props_to_return.has_value()) {
const auto values_list_ref = response.values_ref()->listed_ref()->properties_ref();
const auto &prop_names = props_to_return.value();
if (values_list_ref->empty()) {
return std::nullopt;
}
MG_ASSERT(values_list_ref->at(0).size() == prop_names.size() + 1UL);
for (const auto &values : *values_list_ref) {
auto value_it = values.begin();
std::cout << "Vertex(";
Print(std::cout, *value_it++);
std::cout << ')';
auto name_it = prop_names.begin();
while (value_it != values.end()) {
std::cout << "\n\t" << *name_it << ": ";
Print(std::cout, *value_it);
++value_it;
++name_it;
}
std::cout << std::endl;
}
} else {
const auto values_list_ref = response.values_ref()->mapped_ref()->properties();
const auto mapped_prop_names_ref = response.property_name_map_ref();
for (const auto &values_map : *values_list_ref) {
const auto inner_values_map_ref = values_map.values_map_ref();
std::cout << "Vertex(";
Print(std::cout, inner_values_map_ref->at(1));
std ::cout << ')';
for (const auto &[prop_id, value] : *inner_values_map_ref) {
if (prop_id != 1) {
std::cout << "\n\t" << mapped_prop_names_ref->at(prop_id) << ": ";
Print(std::cout, value);
}
}
std::cout << std::endl;
}
}
if (response.next_start_id_ref().has_value()) {
return *response.get_next_start_id();
}
return std::nullopt;
})
.get();
}
void CreateVertex(const std::shared_ptr<StorageAsyncClient> &client, std::vector<std::string_view> labels,
std::vector<std::string_view> property_names) {
interface::storage::CreateVerticesRequest request {};
auto &new_vertex = request.new_vertices_ref()->emplace_back();
int prop_count = 1;
for (const auto prop : property_names) {
request.property_name_map()->emplace(prop_count, prop);
interface::storage::Value prop_value {};
prop_value.set_int_v(prop_count);
new_vertex.properties_ref()->emplace(prop_count, std::move(prop_value));
prop_count++;
}
int label_count = 1;
for (const auto label : labels) {
request.labels_name_map_ref()->emplace(label_count, label);
new_vertex.label_ids_ref()->push_back(label_count);
label_count++;
}
LOG(INFO) << "Starting transaction...";
client->future_startTransaction()
.then([client, request = std::move(request)](folly::Try<int64_t> &&result) mutable {
if (result.hasException()) {
LOG(INFO) << "FAILED1: " << result.exception().get_exception()->what() << std::endl;
return folly::makeFuture<interface::storage::Result>(std::runtime_error("failed to start transaction"));
}
const auto transaction_id = result.value();
request.transaction_id_ref() = transaction_id;
LOG(INFO) << "Sending create vertex...";
return client->future_createVertices(request) /*.via(evb)*/.then(
[transaction_id, client](folly::Try<interface::storage::Result> &&reply) {
if (reply.hasException()) {
LOG(INFO) << "FAILED2: " << reply.exception().get_exception()->what() << std::endl;
return client->future_abortTransaction(transaction_id).then([](folly::Try<void> &&reply) {
return folly::makeFuture<interface::storage::Result>(std::runtime_error("vertex creation failed"));
});
} else {
LOG(INFO) << "SUCCESS\n";
return client->future_commitTransaction(transaction_id);
}
});
})
.get();
LOG(INFO) << "TRANSACTION IS DONE\n\n\n";
}
int main(int argc, char *argv[]) {
FLAGS_logtostderr = true;
folly::init(&argc, &argv);
auto threadFactory = std::make_shared<folly::NamedThreadFactory>("io-thread");
auto ioThreadPool = std::make_shared<folly::IOThreadPoolExecutor>(8, std::move(threadFactory));
auto *evb = ioThreadPool->getEventBase();
folly::AsyncSocket::UniquePtr socket_ptr;
evb->runImmediatelyOrRunInEventBaseThreadAndWait(
[&socket_ptr, evb]() { socket_ptr = folly::AsyncSocket::newSocket(evb, FLAGS_host, FLAGS_port); });
auto headerClientChannel = apache::thrift::HeaderClientChannel::newChannel(std::move(socket_ptr));
std::shared_ptr<StorageAsyncClient> client(
new StorageAsyncClient(std::move(headerClientChannel)),
[evb](StorageAsyncClient *p) { evb->runImmediatelyOrRunInEventBaseThreadAndWait([p] { delete p; }); });
LOG(INFO) << "Start... ";
CreateVertex(client, {"label1", "label2"}, {"proop", "prooop2"});
CreateVertex(client, {"label1", "label2"}, {"proop", "prooop3"});
CreateVertex(client, {"label1", "label2"}, {"proop", "prooop4"});
CreateVertex(client, {"label1", "label3"}, {"proop", "prooop2"});
CreateVertex(client, {"label1", "label4"}, {"proop", "prooop3"});
CreateVertex(client, {"label1", "label5"}, {"proop", "prooop4"});
// const auto transaction_id = client->future_startTransaction().get();
// std::vector<std::string> props{"proop", "prooop2"};
// auto res = ScanVertices(client, transaction_id, std::nullopt, props, 3);
// std::cout << "RES:" << res.value() << std::endl;
// res = ScanVertices(client, transaction_id, *res, props, 3);
// std::cout << "RES:" << res.value_or(-1) << std::endl;
// res = ScanVertices(client, transaction_id, std::nullopt, std::nullopt, 3);
// std::cout << "RES:" << res.value() << std::endl;
// res = ScanVertices(client, transaction_id, *res, std::nullopt, 3);
// std::cout << "RES:" << res.value_or(-1) << std::endl;
return 1;
}

View File

@@ -0,0 +1,94 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <chrono>
#include <string>
#include <folly/init/Init.h>
#include <folly/portability/GFlags.h>
#include <proxygen/httpserver/HTTPServerOptions.h>
#include <spdlog/common.h>
#include <thrift/lib/cpp/concurrency/ThreadManager.h>
#include <thrift/lib/cpp/thrift_config.h>
#include <thrift/lib/cpp2/server/ThriftProcessor.h>
#include <thrift/lib/cpp2/server/ThriftServer.h>
#include <thrift/lib/cpp2/transport/rocket/server/RocketRoutingHandler.h>
#include "query/exceptions.hpp"
#include "query/frontend/opencypher/generated/MemgraphCypher.h"
#include "query/frontend/opencypher/generated/MemgraphCypherLexer.h"
#include "query/interpret/eval.hpp"
#include "storage_service.hpp"
using apache::thrift::RocketRoutingHandler;
using apache::thrift::ThriftServer;
using apache::thrift::ThriftServerAsyncProcessorFactory;
using manual::storage::StorageServiceHandler;
std::unique_ptr<RocketRoutingHandler> createRoutingHandler(std::shared_ptr<ThriftServer> server) {
return std::make_unique<RocketRoutingHandler>(*server);
}
template <typename ServiceHandler>
std::shared_ptr<ThriftServer> createServer(
memgraph::storage::Storage &db, const int32_t port,
const std::shared_ptr<folly::IOThreadPoolExecutor> &ioThreadPool,
const std::shared_ptr<apache::thrift::concurrency::ThreadManager> &threadManager) {
auto handler = std::make_shared<ServiceHandler>(db);
auto proc_factory = std::make_shared<ThriftServerAsyncProcessorFactory<ServiceHandler>>(handler);
auto server = std::make_shared<ThriftServer>();
server->setPort(port);
server->setIOThreadPool(ioThreadPool);
server->setInterface(handler);
server->setReusePort(true);
server->setIdleTimeout(std::chrono::hours(1));
server->setNumAcceptThreads(1);
server->setListenBacklog(1024);
server->setThreadManager(threadManager);
return server;
}
int main(int argc, char **argv) {
spdlog::set_level(spdlog::level::trace);
// Main storage and execution engines initialization
memgraph::storage::Config db_config{
.gc = {.type = memgraph::storage::Config::Gc::Type::PERIODIC, .interval = std::chrono::seconds(1000)},
.items = {.properties_on_edges = true},
.durability = {.storage_directory = "data",
.recover_on_startup = false,
.snapshot_wal_mode = memgraph::storage::Config::Durability::SnapshotWalMode::PERIODIC_SNAPSHOT,
.snapshot_interval = std::chrono::seconds(15),
.snapshot_retention_count = 2,
.wal_file_size_kibibytes = 20480,
.wal_file_flush_every_n_tx = 1,
.snapshot_on_exit = true},
.transaction = {.isolation_level = memgraph::storage::IsolationLevel::SNAPSHOT_ISOLATION}};
memgraph::storage::Storage db(db_config);
folly::init(&argc, &argv);
static constexpr auto kNumberOfIoThreads{4};
auto threadFactory = std::make_shared<folly::NamedThreadFactory>("io-thread");
auto ioThreadPool = std::make_shared<folly::IOThreadPoolExecutor>(kNumberOfIoThreads, std::move(threadFactory));
std::shared_ptr<apache::thrift::concurrency::ThreadManager> threadManager(
PriorityThreadManager::newPriorityThreadManager(kNumberOfIoThreads));
auto storage_server = createServer<StorageServiceHandler>(db, 7779, ioThreadPool, threadManager);
threadManager->setNamePrefix("executor");
threadManager->start();
std::cout << "alma\n";
LOG(ERROR) << "Storage Server running on port: " << 7779;
storage_server->serve();
return 0;
}

View File

@@ -0,0 +1,316 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "storage_service.hpp"
#include <algorithm>
#include <functional>
#include <iterator>
#include <stdexcept>
#include "interface/gen-cpp2/storage_types.h"
#include "spdlog/spdlog.h"
#include "storage/v2/id_types.hpp"
#include "storage/v2/property_value.hpp"
#include "storage/v2/result.hpp"
#include "storage/v2/storage.hpp"
#include "storage/v2/vertex_accessor.hpp"
#include "storage/v2/view.hpp"
#include "utils/result.hpp"
namespace {
// The response in the handler functions are const, so we cannot move out from them
memgraph::storage::PropertyValue ThriftValueToPropertyValue(const interface::storage::Value &value) {
using Type = interface::storage::Value::Type;
using PropertyValue = memgraph::storage::PropertyValue;
using ThriftValue = interface::storage::Value;
switch (value.getType()) {
case Type::null_v:
return PropertyValue{};
case Type::bool_v:
return PropertyValue{value.get_bool_v()};
case Type::int_v:
return PropertyValue{value.get_int_v()};
case Type::double_v:
return PropertyValue{value.get_double_v()};
case Type::string_v:
return PropertyValue{value.get_string_v()};
case Type::list_v: {
std::vector<PropertyValue> result;
const auto &list = value.get_list_v();
result.resize(list.size());
std::transform(list.begin(), list.end(), std::back_insert_iterator(result), ThriftValueToPropertyValue);
return PropertyValue{std::move(result)};
}
case Type::map_v: {
std::map<std::string, PropertyValue> result;
const auto &map = value.get_map_v();
std::transform(map->begin(), map->end(), std::inserter(result, result.end()),
[](const std::pair<std::string, ThriftValue> &p) -> decltype(result)::value_type {
return {std::move(p.first), ThriftValueToPropertyValue(p.second)};
});
return PropertyValue{std::move(result)};
}
case Type::date_v:
case Type::local_time_v:
case Type::local_date_time_v:
case Type::duration_v:
case Type::__EMPTY__:
case Type::vertex_v:
case Type::edge_v:
case Type::path_v:
// TODO(antaljanosbenjamin): Handle temporal types, assert on entities
return PropertyValue{};
}
}
interface::storage::Value PropertyValueToThriftValue(memgraph::storage::PropertyValue &&value) {
using Type = memgraph::storage::PropertyValue::Type;
using PropertyValue = memgraph::storage::PropertyValue;
using ThriftValue = interface::storage::Value;
interface::storage::Value thrift_value {};
switch (value.type()) {
case Type::Null:
thrift_value.set_null_v();
break;
case Type::Bool:
thrift_value.set_bool_v(value.ValueBool());
break;
case Type::Int:
thrift_value.set_int_v(value.ValueInt());
break;
case Type::Double:
thrift_value.set_double_v(value.ValueDouble());
break;
case Type::String:
thrift_value.set_string_v(std::move(value.ValueString()));
break;
case Type::List: {
std::vector<ThriftValue> result;
auto list = std::move(value.ValueList());
result.resize(list.size());
std::transform(std::make_move_iterator(list.begin()), std::make_move_iterator(list.end()),
std::back_insert_iterator(result), PropertyValueToThriftValue);
thrift_value.set_list_v(std::move(result));
break;
}
case Type::Map: {
std::unordered_map<std::string, ThriftValue> result{};
auto map = std::move(value.ValueMap());
std::transform(std::make_move_iterator(map.begin()), std::make_move_iterator(map.end()),
std::inserter(result, result.end()),
[](std::pair<std::string, PropertyValue> &&p) -> decltype(result)::value_type {
return {std::move(p.first), PropertyValueToThriftValue(std::move(p.second))};
});
thrift_value.set_map_v(std::move(result));
}
case Type::TemporalData:
// TODO(antaljanosbenjamin): Handle temporal types, assert on entities
break;
}
return thrift_value;
}
static const interface::storage::Value kNullValue = std::invoke([] {
interface::storage::Value value;
value.set_null_v();
return value;
});
} // namespace
namespace manual::storage {
int64_t StorageServiceHandler::startTransaction() {
spdlog::info("Starting transaction");
static std::atomic<int64_t> counter{0};
const auto transaction_id = ++counter;
active_transactions_.insert(transaction_id, std::make_shared<memgraph::storage::Storage::Accessor>(db_.Access()));
return transaction_id;
};
void StorageServiceHandler::commitTransaction(::interface::storage::Result &result, int64_t transaction_id) {
spdlog::info("Commiting transaction");
if (auto accessor_it = active_transactions_.find(transaction_id); accessor_it != active_transactions_.end()) {
result.success_ref() = accessor_it->second->Commit().HasError();
} else {
result.success_ref() = false;
}
};
void StorageServiceHandler::abortTransaction(int64_t transaction_id) {
spdlog::info("Aborting transaction");
if (auto accessor_it = active_transactions_.find(transaction_id); accessor_it != active_transactions_.end()) {
accessor_it->second->Abort();
}
};
void StorageServiceHandler::createVertices(::interface::storage::Result &result,
const ::interface::storage::CreateVerticesRequest &req) {
spdlog::info("Creating vertex...");
result.success_ref() = false;
auto accessor = active_transactions_.at(req.get_transaction_id());
const auto &labels_map = req.get_labels_name_map();
const auto &property_names_map = req.get_property_name_map();
// auto accessor = db_.Access();
for (auto &new_vertex : *req.new_vertices_ref()) {
auto vertex = accessor->CreateVertex();
for (const auto label_id : new_vertex.get_label_ids()) {
if (const auto result = vertex.AddLabel(accessor->NameToLabel(labels_map.at(label_id))); result.HasError()) {
return;
}
}
for (auto &[prop_id, prop] : *new_vertex.properties_ref()) {
if (const auto result = vertex.SetProperty(accessor->NameToProperty(property_names_map.at(prop_id)),
ThriftValueToPropertyValue(prop));
result.HasError()) {
return;
}
}
}
spdlog::info("Vertex creation done!");
}
static_assert(sizeof(apache::thrift::optional_field_ref<interface::storage::Values &>) > 16);
std::function<memgraph::utils::BasicResult<std::string>(memgraph::storage::VertexAccessor)> CreateVertexProcessor(
memgraph::storage::Storage::Accessor &db_accessor, interface::storage::Values &values,
apache::thrift::optional_field_ref<std::unordered_map<int64_t, std::string> &> property_name_map_ref,
const apache::thrift::optional_field_ref<const std::vector<::std::string> &> &props_to_return_ref,
const memgraph::storage::View view) {
if (!props_to_return_ref.has_value()) {
values.set_mapped();
auto &mapped_values = values.mutable_mapped();
auto result_props_ref = mapped_values.properties_ref();
property_name_map_ref.ensure();
auto &property_name_map = *property_name_map_ref;
static constexpr auto vertex_id_id = 1L;
auto internal_id_to_thrift_id = [&db_accessor, &property_name_map,
cache = std::unordered_map<memgraph::storage::PropertyId, int64_t>{},
next_id = vertex_id_id + 1l](memgraph::storage::PropertyId prop_id) mutable {
if (const auto it = cache.find(prop_id); it != cache.end()) {
return it->second;
}
const auto &name = db_accessor.PropertyToName(prop_id);
property_name_map.emplace(next_id, name);
cache.emplace(prop_id, next_id);
return next_id++;
};
return [result_props_ref = std::move(result_props_ref), view,
internal_id_to_thrift_id = std::move(internal_id_to_thrift_id)](
memgraph::storage::VertexAccessor vertex_acc) mutable -> memgraph::utils::BasicResult<std::string> {
auto props_res = vertex_acc.Properties(view);
if (props_res.HasError()) {
// TODO(antaljanosbenjamin): More fine grained error handling
return std::string{"Vertex deleted"};
}
interface::storage::ValuesMap values_map;
auto row_ref = values_map.values_map_ref();
interface::storage::Value vertex_id;
vertex_id.set_int_v(vertex_acc.Gid().AsInt());
row_ref->reserve(props_res->size() + 1);
row_ref->emplace(vertex_id_id, std::move(vertex_id));
for (auto &[prop_id, prop] : *props_res) {
const auto thrift_id = internal_id_to_thrift_id(prop_id);
row_ref->emplace(thrift_id, PropertyValueToThriftValue(std::move(prop)));
}
result_props_ref->push_back(std::move(values_map));
return {};
};
}
values.set_listed({});
auto &listed_values = values.mutable_listed();
auto result_props_ref = listed_values.properties_ref();
if (props_to_return_ref->empty()) {
// Return only the vertex ids
return [result_props_ref = std::move(result_props_ref),
view](memgraph::storage::VertexAccessor vertex_acc) mutable -> memgraph::utils::BasicResult<std::string> {
if (!vertex_acc.IsVisible(view)) {
return {};
}
auto &values_list = result_props_ref->emplace_back();
auto &value = values_list.emplace_back();
value.set_int_v(vertex_acc.Gid().AsInt());
return {};
};
}
// Return properties in order
std::vector<memgraph::storage::PropertyId> prop_ids_to_return{};
prop_ids_to_return.reserve(props_to_return_ref->size());
std::transform(props_to_return_ref->begin(), props_to_return_ref->end(), std::back_inserter(prop_ids_to_return),
std::bind_front(&memgraph::storage::Storage::Accessor::NameToProperty, &db_accessor));
return [result_props_ref = std::move(result_props_ref), prop_ids_to_return = std::move(prop_ids_to_return),
view](memgraph::storage::VertexAccessor vertex_acc) mutable -> memgraph::utils::BasicResult<std::string> {
if (!vertex_acc.IsVisible(view)) {
return {};
}
std::vector<interface::storage::Value> row{};
row.reserve(prop_ids_to_return.size() + 1);
{
interface::storage::Value vertex_id;
vertex_id.set_int_v(vertex_acc.Gid().AsInt());
row.push_back(std::move(vertex_id));
}
for (const auto &prop_id : prop_ids_to_return) {
auto property_result = vertex_acc.GetProperty(prop_id, view);
if (property_result.HasError()) {
// TODO(antaljanosbenjamin): More fine grained error handling
return std::string{"Vertex deleted"};
}
row.push_back(PropertyValueToThriftValue(std::move(property_result.GetValue())));
}
result_props_ref->push_back(std::move(row));
return {};
};
}
void StorageServiceHandler::scanVertices(::interface::storage::ScanVerticesResponse &resp,
const ::interface::storage::ScanVerticesRequest &req) {
resp.result_ref()->success_ref() = false;
auto accessor = active_transactions_.at(req.get_transaction_id());
// TODO(antaljanosbenjamin): handle filter
const auto view = req.get_view();
auto vertices = accessor->Vertices(view);
auto it = vertices.begin();
if (const auto *start_id = req.get_start_id(); start_id != nullptr) {
it = vertices.IterateFrom(memgraph::storage::Gid::FromInt(*start_id));
}
auto count = 0;
auto vertex_processor = CreateVertexProcessor(*accessor, *resp.values(), resp.property_name_map_ref(),
req.props_to_return_ref(), req.get_view());
const auto limit = std::invoke([&]() -> int64_t {
if (req.limit_ref().has_value()) {
return *req.limit_ref();
}
return 50;
});
while (count < limit && it != vertices.end()) {
if (const auto res = vertex_processor(*it); res.HasError()) {
throw std::runtime_error{res.GetError()};
};
++it;
++count;
}
if (it != vertices.end()) {
resp.next_start_id_ref() = (*it).Gid().AsInt();
}
resp.result_ref()->success_ref() = true;
}
} // namespace manual::storage

View File

@@ -0,0 +1,43 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <vector>
#include <folly/concurrency/ConcurrentHashMap.h>
#include "interface/gen-cpp2/Storage.h"
#include "interface/gen-cpp2/storage_types.h"
#include "storage/v2/storage.hpp"
namespace manual::storage {
// TODO(antaljanosbenjamin):
// - Check out different approaches of how Thrift message members can be read/write
class StorageServiceHandler final : public interface::storage::StorageSvIf {
public:
explicit StorageServiceHandler(memgraph::storage::Storage &db) : db_{db} {}
int64_t startTransaction() override;
void commitTransaction(::interface::storage::Result &result, int64_t transaction_id) override;
void abortTransaction(int64_t transaction_id) override;
void createVertices(::interface::storage::Result &result,
const ::interface::storage::CreateVerticesRequest &req) override;
void scanVertices(::interface::storage::ScanVerticesResponse &resp,
const ::interface::storage::ScanVerticesRequest &req) override;
private:
memgraph::storage::Storage &db_;
folly::ConcurrentHashMap<int64_t, std::shared_ptr<memgraph::storage::Storage::Accessor>> active_transactions_;
};
} // namespace manual::storage

View File

@@ -0,0 +1,49 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <iostream>
#include <folly/init/Init.h>
#include <thrift/lib/cpp2/async/HeaderClientChannel.h>
#include "interface/gen-cpp2/PingPong.h" // From generated code
#include "interface/gen-cpp2/PingPongAsyncClient.h"
using namespace apache::thrift;
using namespace cpp2;
using namespace folly;
int main(int argc, char **argv) {
EventBase base;
static constexpr int port = 6666; // The port on which server is listening
folly::init(&argc, &argv);
// Create a async client socket and connect it. Change
// the ip address to where the server is listening
auto socket(folly::AsyncSocket::newSocket(&base, "127.0.0.1", port));
// Create a HeaderClientChannel object which is used in creating
// client object
auto client_channel = HeaderClientChannel::newChannel(std::move(socket));
// Create a client object
PingPongAsyncClient client(std::move(client_channel));
// Invoke the add function on the server. As we are doing async
// invocation of the function, we do not immediately get
// the result. Instead we get a future object.
Ping req{};
req.message_ref() = "I am here!";
Pong result{};
LOG(ERROR) << "Sending...";
client.sync_ping(result, req);
LOG(ERROR) << result.get_message();
return 0;
}

View File

@@ -0,0 +1,56 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <iostream>
#include <folly/init/Init.h>
#include <folly/io/SocketOptionMap.h>
#include <folly/io/async/AsyncServerSocket.h>
#include <folly/net/NetworkSocket.h>
#include <thrift/lib/cpp2/server/ThriftServer.h>
#include "interface/gen-cpp2/PingPong.h" // This is included from generated code
using namespace apache::thrift;
using namespace cpp2;
const std::string prefix{"Prefix"};
// The thrift has generated service interface with name CalculatorSvIf
// At server side we have to implement this interface
class PingPongSvc : public PingPongSvIf {
public:
virtual ~PingPongSvc() {}
// We have to implement async_tm_add to implement the add function
// of the Calculator service which we defined in calculator.thrift file
void ping(Pong &res, const Ping &req) {
LOG(ERROR) << "Received\n";
res.message_ref() = prefix + req.get_message();
LOG(ERROR) << "Sent\n";
}
};
int main(int argc, char **argv) {
folly::init(&argc, &argv);
// To create a server, we need to first create server handler object.
// The server handler object contains the implementation of the service.
std::shared_ptr<PingPongSvc> ptr(new PingPongSvc());
// Create a thrift server
ThriftServer *s = new ThriftServer();
// Set server handler object
s->setInterface(ptr);
// Set the server port
s->setPort(6666);
// Start the server to serve!!
s->serve();
return 0;
}

View File

@@ -23,13 +23,13 @@ TEST(MgpTransTest, TestMgpTransApi) {
// for different string cases as these are all handled by
// IsValidIdentifier().
// Maybe add a mock instead and expect IsValidIdentifier() to be called once?
EXPECT_EQ(mgp_module_add_transformation(&module, "dash-dash", no_op_cb), MGP_ERROR_INVALID_ARGUMENT);
EXPECT_EQ(mgp_module_add_transformation(&module, "dash-dash", no_op_cb), mgp_error::MGP_ERROR_INVALID_ARGUMENT);
EXPECT_TRUE(module.transformations.empty());
EXPECT_EQ(mgp_module_add_transformation(&module, "transform", no_op_cb), MGP_ERROR_NO_ERROR);
EXPECT_EQ(mgp_module_add_transformation(&module, "transform", no_op_cb), mgp_error::MGP_ERROR_NO_ERROR);
EXPECT_NE(module.transformations.find("transform"), module.transformations.end());
// Try to register a transformation twice
EXPECT_EQ(mgp_module_add_transformation(&module, "transform", no_op_cb), MGP_ERROR_LOGIC_ERROR);
EXPECT_EQ(mgp_module_add_transformation(&module, "transform", no_op_cb), mgp_error::MGP_ERROR_LOGIC_ERROR);
EXPECT_TRUE(module.transformations.size() == 1);
}

Some files were not shown because too many files have changed in this diff Show More