Compare commits

...

62 Commits

Author SHA1 Message Date
Josip Mrden
e71b9a99ae Add line breaks 2023-09-15 13:17:49 +02:00
Josip Mrden
2aa6b9ba8f Add null value to transformations 2023-09-15 12:44:19 +02:00
Josip Mrden
6c9cdef944 Make example transformation work e2e 2023-09-15 12:39:06 +02:00
Josip Mrden
5c10fb1d51 Add building of query module default libraries without the lib prefix 2023-09-15 12:09:27 +02:00
Josip Mrden
e73b3feea3 Adjust equals operator on messages iterator 2023-09-15 10:40:16 +02:00
Josip Mrden
2ed40e9f16 Adjust name of the transformation 2023-09-15 10:24:23 +02:00
Josip Mrden
ced34bc79c Add example transformation module 2023-09-15 10:20:22 +02:00
Josip Mrden
2b3540119f Implement equals 2023-09-14 19:17:20 +02:00
Josip Mrden
3936668e13 Implemented messages interface 2023-09-14 18:07:27 +02:00
Josip Mrden
476080bace Add iterator constructors 2023-09-14 18:02:08 +02:00
Josip Mrden
8528822af1 Add blueprint for message iterator 2023-09-14 17:57:15 +02:00
Josip Mrden
7ecf970dcd Add messages methods 2023-09-14 17:47:09 +02:00
Josip Mrden
6ad5463d72 Add blueprint for classes 2023-09-14 17:01:37 +02:00
Josip Mrden
b1d560737e Add error wrappers 2023-09-14 16:45:27 +02:00
Marko Barišić
b719f0744f Update v2.11 license date (#1247) 2023-09-13 07:19:11 -04:00
Josipmrden
440838c0e9 Add dependency check for e2e tests (#1240) 2023-09-12 11:34:08 -04:00
Josipmrden
79a3c5af8e Add manual performance benchmark execution (#1239) 2023-09-12 11:33:05 -04:00
Josipmrden
bf03b38e39 Remove gqlalchemy from stress tests (#1245) 2023-09-12 11:32:16 -04:00
gvolfing
fd63944493 Add --query-callable-mappings-path package default (#1203) 2023-09-11 15:12:14 -04:00
Gareth Andrew Lloyd
6694de2dfa Fix libkrb5 TRUE and FALSE macros leakage (#1243)
Co-authored-by: Marko Budiselić <marko.budiselic@memgraph.com>
2023-09-11 12:46:40 -04:00
andrejtonev
5e5f4ffc5d Add more runtime configurable settings (#1183)
server name, query timeout settings, log.level, log.to_stderr
2023-09-11 11:30:54 -04:00
Ante Pušić
060b9d1c16 [master < T1204] Add detailed operator info to PROFILE and EXPLAIN (#1204) 2023-09-11 14:34:27 +02:00
Ante Pušić
29a505cb38 Property lookup caching (#1168) 2023-09-11 13:03:54 +02:00
Ante Pušić
d4fcd745d2 Sort SHOW INDEX INFO (#1178) 2023-09-11 10:59:41 +02:00
Josipmrden
58546a9fe1 Add detach delete in bulk (#1078) 2023-09-10 18:53:03 +02:00
Gareth Andrew Lloyd
ab56abf4ca Optimize scanning vertices (#1227) 2023-09-09 10:09:25 -04:00
Gareth Andrew Lloyd
1bd47318cd Improve PropertyStore (#1142)
Improve AnyVersionHasLabelProperty by doing less work in some instances.
Improve FindSpecificProperty.
2023-09-09 08:00:43 -04:00
Ante Pušić
0403b67073 Fix returning NULL on map projection from a null value (#1119) 2023-09-09 06:43:25 -04:00
DavIvek
9e4babcdbb Fix segfault based on issue #874 (#1175) 2023-09-09 02:04:46 -04:00
Antonio Filipovic
b094fdbadc Fix API bug on accessing deleted object (#1209) 2023-09-08 13:52:21 -04:00
Josipmrden
07dea328d8 [master < T1110] Add merge optimization to expand dynamically during runtime (#1110) 2023-09-08 17:12:25 +02:00
Gareth Andrew Lloyd
bd1852f407 Reduce flake SnapshotFallback test (#1237)
Fixed the wait period, this should ensure at least one snapshot was 
made. Also cleaned up the checking around this. And also better 
corruption.
2023-09-08 14:21:35 +01:00
imilinovic
9c51dbbb01 Implement changing from and to vertices in relationships (#1221) 2023-09-08 12:52:40 +02:00
ind1xa
c0d4f5e0bc Add traits under mgp iterator (#1210) 2023-09-08 08:57:37 +02:00
Antonio Filipovic
974a6e3027 Fix bug on mgp dispatcher guard (#1225) 2023-09-07 17:42:27 +02:00
Matija Pintarić
d9464c6ffd Add InDegree and OutDegree in O(1) (#1217) 2023-09-07 13:16:30 +02:00
Ante Javor
312d01bd0c Remove repeated log lines from TRACE log level. (#1054) 2023-09-06 23:09:51 +02:00
Antonio Filipovic
b6b32bec03 Improve performance of delta creation (#1129) 2023-09-06 11:30:21 +02:00
Antonio Filipovic
93992a275b Improve NameToId mapper on set properties (#1147) 2023-09-06 00:12:27 +02:00
Andi
b5413c6f82 Add edge import mode into the on-disk storage (#1157) 2023-09-05 19:00:53 +02:00
Josipmrden
09fd5939da Remove double scan with expand from the planner (#1085) 2023-09-05 11:02:52 +02:00
Josipmrden
02eab6ab9c Set properties C API extension (#1131)
Add SetProperties into the C++ query module API
2023-09-04 16:17:43 +02:00
Gareth Andrew Lloyd
9661c52179 Introduce a reader writer spin lock (#1187)
It is possible for multiple read only queries to be accessing the same
sequence of vertices/edges. The reader mode of the spin lock will ensure
multiple threads can make progress at the same time.
2023-09-01 14:21:15 +01:00
Gareth Andrew Lloyd
e928eed028 Replication refactor (part 4) (#1211)
More refactoring to isolate generic replication behavior. Making the 
InMemory* types even more decoupled from replication logic.
2023-08-31 16:06:44 +01:00
Josipmrden
eb5167dfef Add high write set property workload (#1172) 2023-08-31 14:46:35 +02:00
Josipmrden
b952139973 Add supernode performance workload (#1171) 2023-08-30 15:19:52 +02:00
andrejtonev
28dbcd1545 Add disk storage to e2e tests (#1202)
* Add disk storage to e2e tests

---------

Co-authored-by: Andi Skrgat <andi8647@gmail.com>
2023-08-30 13:42:11 +02:00
Matija Pintarić
d516e40841 Add ToString on C++ API mgp types(#1140) 2023-08-29 17:30:23 +02:00
Andi
a6ec81b179 Add deterministic disk vertex_count and edge_count (#1146)
* Add exact vertex_count and edge_count to disk storage
2023-08-29 13:07:23 +02:00
andrejtonev
c526ff2a8f [master < ] Remove DbAccessor from non-transactional queries (#1201)
* Decouple non-transactional queries from DbAccessor
* Invalidate auth cache after AuthQuery

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2023-08-29 11:13:42 +02:00
Aidar Samerkhanov
5f509532f2 Add timestamp to DELETE_DESERIALIZED_OBJECT delta at which this object was created. (#1179)
Add timestamp to DELETE_DESERIALIZED_OBJECT delta at which this object was created.
RocksDB currently doesn't provide timestamp() functionality in iterators of TransationDB.
Because of that we are using constant "0" timestamp for DELETE_DESERIALIZED_OBJECT.
2023-08-28 10:56:17 +04:00
Andi
4b3ba908c7 Code improvements on disk storage (#1153)
* Improvements based on a code review

---------

Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io>
2023-08-26 14:16:12 +02:00
Andi
030b554ffd Improve concurrency control for on-disk storage (#1154)
* Remove locking vertices and serialization checks
---------

Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io>
2023-08-25 14:42:52 +02:00
Gareth Andrew Lloyd
4bc5d749b2 Refactor replication, part 3 (#1177)
Changes to make replication code agnostic of the storage kind being used.

Co-authored-by: Andreja Tonev <andreja.tonev@memgraph.io>
2023-08-25 10:52:07 +01:00
imilinovic
a426ef9cc3 Add Relationship::RemoveProperty to C++ query module API (#1156) 2023-08-24 12:14:00 +02:00
Ante Pušić
60e167d676 Optimize index and constraint updates (#1159) 2023-08-23 14:52:44 +02:00
Ante Pušić
3f8befde79 Bump PyYAML version (#1174) 2023-08-23 12:48:17 +02:00
andrejtonev
9355e58e73 Decoupling replication logic from InMemoryStorage (#1169) 2023-08-22 13:29:25 +02:00
gvolfing
476968e2c8 Fix concurrent query module race condition (#1158)
Concurrent access to the same query module had a race condition on the
pointer that was used to handle the custom memory management. With this
commit, a mapping has been added to keep information about what
thread used the pointer to handle the memory resources. This should be
fine since the respected query executions are running on a dedicated
thread. Access to the mapping itself is threadsafe. A simple RAII
wrapper for the mapping container has also been added for simpler
client-side use.
2023-08-21 16:45:36 +02:00
Gareth Andrew Lloyd
97183fb9da Fix FLAGS_delta_chain_cache_threshold typo (#1181) 2023-08-21 13:16:02 +02:00
Gareth Andrew Lloyd
adb65b2fff Refactor memgraph.cpp (#1164) 2023-08-18 18:23:15 +02:00
Aidar Samerkhanov
3bf2cf65ab Optimize splitting keys inside the on-disk storage (#1155) 2023-08-17 18:09:21 +02:00
306 changed files with 16194 additions and 7543 deletions

View File

@@ -67,7 +67,11 @@ jobs:
- name: Run mgbench
run: |
cd tests/mgbench
./benchmark.py vendor-native --num-workers-for-benchmark 12 --export-results benchmark_result.json pokec/medium/*/*
./benchmark.py vendor-native --num-workers-for-benchmark 12 --export-results benchmark_pokec.json pokec/medium/*/*
./benchmark.py vendor-native --num-workers-for-benchmark 1 --export-results benchmark_supernode.json supernode
./benchmark.py vendor-native --num-workers-for-benchmark 1 --export-results benchmark_high_write_set_property.json high_write_set_property
- name: Upload mgbench results
run: |
@@ -76,7 +80,19 @@ jobs:
source ve3/bin/activate
pip install -r requirements.txt
./main.py --benchmark-name "mgbench" \
--benchmark-results-path "../../tests/mgbench/benchmark_result.json" \
--benchmark-results-path "../../tests/mgbench/benchmark_pokec.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "supernode" \
--benchmark-results-path "../../tests/mgbench/benchmark_supernode.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "high_write_set_property" \
--benchmark-results-path "../../tests/mgbench/benchmark_high_write_set_property.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"

View File

@@ -101,7 +101,7 @@ jobs:
echo ${file}
if [[ ${file} == *.py ]]; then
python3 -m black --check --diff ${file}
python3 -m isort --check-only --diff ${file}
python3 -m isort --profile black --check-only --diff ${file}
fi
done
@@ -229,6 +229,11 @@ jobs:
# branches and tags. (default: 1)
fetch-depth: 0
- name: Check e2e service dependencies
run: |
cd tests/e2e
./dependency_check.sh
- name: Build release binaries
run: |
# Activate toolchain.
@@ -275,11 +280,13 @@ jobs:
- name: Run stress test (plain)
run: |
cd tests/stress
source ve3/bin/activate
./continuous_integration
- name: Run stress test (SSL)
run: |
cd tests/stress
source ve3/bin/activate
./continuous_integration --use-ssl
- name: Run durability test

View File

@@ -0,0 +1,77 @@
name: Run performance benchmarks manually
on:
workflow_dispatch:
jobs:
performance_benchmarks:
name: "Performance benchmarks"
runs-on: [self-hosted, Linux, X64, Diff, Gen7]
env:
THREADS: 24
MEMGRAPH_ENTERPRISE_LICENSE: ${{ secrets.MEMGRAPH_ENTERPRISE_LICENSE }}
MEMGRAPH_ORGANIZATION_NAME: ${{ secrets.MEMGRAPH_ORGANIZATION_NAME }}
steps:
- name: Set up repository
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
fetch-depth: 0
- name: Build release binaries
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Initialize dependencies.
./init
# Build only memgraph release binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$THREADS
- name: Get branch name (merge)
if: github.event_name != 'pull_request'
shell: bash
run: echo "BRANCH_NAME=$(echo ${GITHUB_REF#refs/heads/} | tr / -)" >> $GITHUB_ENV
- name: Get branch name (pull request)
if: github.event_name == 'pull_request'
shell: bash
run: echo "BRANCH_NAME=$(echo ${GITHUB_HEAD_REF} | tr / -)" >> $GITHUB_ENV
- name: Run benchmarks
run: |
cd tests/mgbench
./benchmark.py vendor-native --num-workers-for-benchmark 12 --export-results benchmark_result.json pokec/medium/*/*
./benchmark.py vendor-native --num-workers-for-benchmark 1 --export-results benchmark_supernode.json supernode
./benchmark.py vendor-native --num-workers-for-benchmark 1 --export-results benchmark_high_write_set_property.json high_write_set_property
- name: Upload benchmark results
run: |
cd tools/bench-graph-client
virtualenv -p python3 ve3
source ve3/bin/activate
pip install -r requirements.txt
./main.py --benchmark-name "mgbench" \
--benchmark-results-path "../../tests/mgbench/benchmark_result.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "supernode" \
--benchmark-results-path "../../tests/mgbench/benchmark_supernode.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "high_write_set_property" \
--benchmark-results-path "../../tests/mgbench/benchmark_high_write_set_property.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"

View File

@@ -15,6 +15,7 @@ repos:
hooks:
- id: isort
name: isort (python)
args: ["--profile", "black"]
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: v13.0.0
hooks:

View File

@@ -42,6 +42,10 @@ endif()
project(memgraph LANGUAGES C CXX)
# NOTE: once in a while this needs to be toggled to check headers are
# correct and PCH isn't masking any include issues
set(CMAKE_DISABLE_PRECOMPILE_HEADERS OFF)
#TODO: upgrade to cmake 3.24 + CheckIPOSupported
#cmake_policy(SET CMP0138 NEW)
#include(CheckIPOSupported)

View File

@@ -107,6 +107,10 @@ modifications:
value: "false"
override: true
- name: "query_callable_mappings_path"
value: "/etc/memgraph/apoc_compatibility_mappings.json"
override: true
undocumented:
- "flag_file"
- "also_log_to_stderr"

27
config/mappings.json Normal file
View File

@@ -0,0 +1,27 @@
{
"dbms.components": "mgps.components",
"apoc.util.validate": "mgps.validate",
"db.schema.nodeTypeProperties":"schema.node_type_properties",
"db.schema.relTypeProperties":"schema.rel_type_properties",
"apoc.coll.contains": "collections.contains",
"apoc.coll.partition": "collections.partition",
"apoc.coll.toSet": "collections.to_set",
"apoc.coll.unionAll": "collections.unionAll",
"apoc.coll.removeAll": "collections.remove_all",
"apoc.coll.union": "collections.union",
"apoc.coll.sum": "collections.sum",
"apoc.coll.pairs": "collections.pairs",
"apoc.map.fromLists": "map.from_lists",
"apoc.map.removeKeys": "map.remove_keys",
"apoc.map.merge": "map.merge",
"apoc.create.nodes": "create.nodes",
"apoc.create.removeProperties": "create.remove_properties",
"apoc.create.node": "create.node",
"apoc.create.removeLabel": "create.remove_label",
"apoc.refactor.invert": "refactor.invert",
"apoc.refactor.cloneNode": "refactor.clone_node",
"apoc.refactor.cloneSubgraph": "refactor.clone_subgraph",
"apoc.refactor.cloneSubgraphFromPath": "refactor.clone_subgraph_from_path",
"apoc.label.exists": "label.exists"
}

View File

@@ -255,6 +255,16 @@ inline mgp_edge *graph_create_edge(mgp_graph *graph, mgp_vertex *from, mgp_verte
return MgInvoke<mgp_edge *>(mgp_graph_create_edge, graph, from, to, type, memory);
}
inline mgp_edge *graph_edge_set_from(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_vertex *new_from,
mgp_memory *memory) {
return MgInvoke<mgp_edge *>(mgp_graph_edge_set_from, graph, e, new_from, memory);
}
inline mgp_edge *graph_edge_set_to(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_vertex *new_to,
mgp_memory *memory) {
return MgInvoke<mgp_edge *>(mgp_graph_edge_set_to, graph, e, new_to, memory);
}
inline void graph_delete_edge(mgp_graph *graph, mgp_edge *edge) { MgInvokeVoid(mgp_graph_delete_edge, graph, edge); }
inline mgp_vertex *graph_get_vertex_by_id(mgp_graph *g, mgp_vertex_id id, mgp_memory *memory) {
@@ -367,6 +377,10 @@ inline mgp_map_item *map_items_iterator_next(mgp_map_items_iterator *it) {
inline mgp_vertex_id vertex_get_id(mgp_vertex *v) { return MgInvoke<mgp_vertex_id>(mgp_vertex_get_id, v); }
inline size_t vertex_get_in_degree(mgp_vertex *v) { return MgInvoke<size_t>(mgp_vertex_get_in_degree, v); }
inline size_t vertex_get_out_degree(mgp_vertex *v) { return MgInvoke<size_t>(mgp_vertex_get_out_degree, v); }
inline mgp_vertex *vertex_copy(mgp_vertex *v, mgp_memory *memory) {
return MgInvoke<mgp_vertex *>(mgp_vertex_copy, v, memory);
}
@@ -401,6 +415,10 @@ inline void vertex_set_property(mgp_vertex *v, const char *property_name, mgp_va
MgInvokeVoid(mgp_vertex_set_property, v, property_name, property_value);
}
inline void vertex_set_properties(mgp_vertex *v, struct mgp_map *properties) {
MgInvokeVoid(mgp_vertex_set_properties, v, properties);
}
inline mgp_properties_iterator *vertex_iter_properties(mgp_vertex *v, mgp_memory *memory) {
return MgInvoke<mgp_properties_iterator *>(mgp_vertex_iter_properties, v, memory);
}
@@ -437,6 +455,10 @@ inline void edge_set_property(mgp_edge *e, const char *property_name, mgp_value
MgInvokeVoid(mgp_edge_set_property, e, property_name, property_value);
}
inline void edge_set_properties(mgp_edge *e, struct mgp_map *properties) {
MgInvokeVoid(mgp_edge_set_properties, e, properties);
}
inline mgp_properties_iterator *edge_iter_properties(mgp_edge *e, mgp_memory *memory) {
return MgInvoke<mgp_properties_iterator *>(mgp_edge_iter_properties, e, memory);
}
@@ -760,4 +782,44 @@ inline void func_result_set_value(mgp_func_result *res, mgp_value *value, mgp_me
MgInvokeVoid(mgp_func_result_set_value, res, value, memory);
}
// Messages
inline mgp_source_type message_source_type(struct mgp_message *message) {
return MgInvoke<mgp_source_type>(mgp_message_source_type, message);
}
inline const char *message_payload(struct mgp_message *message) {
return MgInvoke<const char *>(mgp_message_payload, message);
}
inline size_t message_payload_size(struct mgp_message *message) {
return MgInvoke<size_t>(mgp_message_payload_size, message);
}
inline const char *message_topic_name(struct mgp_message *message) {
return MgInvoke<const char *>(mgp_message_topic_name, message);
}
inline const char *message_key(struct mgp_message *message) { return MgInvoke<const char *>(mgp_message_key, message); }
inline size_t message_key_size(struct mgp_message *message) { return MgInvoke<size_t>(mgp_message_key_size, message); }
inline int64_t message_timestamp(struct mgp_message *message) {
return MgInvoke<int64_t>(mgp_message_timestamp, message);
}
inline int64_t message_offset(struct mgp_message *message) { return MgInvoke<int64_t>(mgp_message_offset, message); }
inline size_t messages_size(struct mgp_messages *message) { return MgInvoke<size_t>(mgp_messages_size, message); }
inline mgp_message *messages_at(struct mgp_messages *message, size_t index) {
return MgInvoke<mgp_message *>(mgp_messages_at, message, index);
}
// Transformation
inline void module_add_transformation(struct mgp_module *module, const char *name, mgp_trans_cb cb) {
return MgInvokeVoid(mgp_module_add_transformation, module, name, cb);
}
} // namespace mgp

View File

@@ -647,6 +647,12 @@ struct mgp_vertex_id {
/// Get the ID of given vertex.
enum mgp_error mgp_vertex_get_id(struct mgp_vertex *v, struct mgp_vertex_id *result);
/// Get the in degree of given vertex.
enum mgp_error mgp_vertex_get_in_degree(struct mgp_vertex *v, size_t *result);
/// Get the out degree of given vertex.
enum mgp_error mgp_vertex_get_out_degree(struct mgp_vertex *v, size_t *result);
/// Result is non-zero if the vertex can be modified.
/// The mutability of the vertex is the same as the graph which it is part of. If a vertex is immutable, then edges
/// cannot be created or deleted, properties and labels cannot be set or removed and all of the returned edges will be
@@ -664,6 +670,15 @@ enum mgp_error mgp_vertex_underlying_graph_is_mutable(struct mgp_vertex *v, int
enum mgp_error mgp_vertex_set_property(struct mgp_vertex *v, const char *property_name,
struct mgp_value *property_value);
/// Set the value of properties on a vertex.
/// When the value is `null`, then the property is removed from the vertex.
/// Return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE if unable to allocate memory for storing the property.
/// Return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT if `v` is immutable.
/// Return mgp_error::MGP_ERROR_DELETED_OBJECT if `v` has been deleted.
/// Return mgp_error::MGP_ERROR_SERIALIZATION_ERROR if `v` has been modified by another transaction.
/// Return mgp_error::MGP_ERROR_VALUE_CONVERSION if `property_value` is vertex, edge or path.
enum mgp_error mgp_vertex_set_properties(struct mgp_vertex *v, struct mgp_map *properties);
/// Add the label to the vertex.
/// If the vertex already has the label, this function does nothing.
/// Return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE if unable to allocate memory for storing the label.
@@ -814,6 +829,15 @@ enum mgp_error mgp_edge_get_property(struct mgp_edge *e, const char *property_na
/// Return mgp_error::MGP_ERROR_VALUE_CONVERSION if `property_value` is vertex, edge or path.
enum mgp_error mgp_edge_set_property(struct mgp_edge *e, const char *property_name, struct mgp_value *property_value);
/// Set the value of properties on a vertex.
/// When the value is `null`, then the property is removed from the vertex.
/// Return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE if unable to allocate memory for storing the property.
/// Return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT if `v` is immutable.
/// Return mgp_error::MGP_ERROR_DELETED_OBJECT if `v` has been deleted.
/// Return mgp_error::MGP_ERROR_SERIALIZATION_ERROR if `v` has been modified by another transaction.
/// Return mgp_error::MGP_ERROR_VALUE_CONVERSION if `property_value` is vertex, edge or path.
enum mgp_error mgp_edge_set_properties(struct mgp_edge *e, struct mgp_map *properties);
/// Start iterating over properties stored in the given edge.
/// The properties of the edge are copied when the iterator is created, therefore later changes won't affect them.
/// Resulting mgp_properties_iterator needs to be deallocated with
@@ -864,6 +888,22 @@ enum mgp_error mgp_graph_detach_delete_vertex(struct mgp_graph *graph, struct mg
enum mgp_error mgp_graph_create_edge(struct mgp_graph *graph, struct mgp_vertex *from, struct mgp_vertex *to,
struct mgp_edge_type type, struct mgp_memory *memory, struct mgp_edge **result);
/// Change edge from vertex
/// Return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT if `graph` is immutable.
/// Return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE if unable to allocate a mgp_edge.
/// Return mgp_error::MGP_ERROR_DELETED_OBJECT if `from` or `to` has been deleted.
/// Return mgp_error::MGP_ERROR_SERIALIZATION_ERROR if `from` or `to` has been modified by another transaction.
enum mgp_error mgp_graph_edge_set_from(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_vertex *new_from,
struct mgp_memory *memory, struct mgp_edge **result);
/// Change edge to vertex
/// Return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT if `graph` is immutable.
/// Return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE if unable to allocate a mgp_edge.
/// Return mgp_error::MGP_ERROR_DELETED_OBJECT if `from` or `to` has been deleted.
/// Return mgp_error::MGP_ERROR_SERIALIZATION_ERROR if `from` or `to` has been modified by another transaction.
enum mgp_error mgp_graph_edge_set_to(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_vertex *new_to,
struct mgp_memory *memory, struct mgp_edge **result);
/// Delete an edge from the graph.
/// Return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT if `graph` is immutable.
/// Return mgp_error::MGP_ERROR_SERIALIZATION_ERROR if `edge`, its source or destination vertex has been modified by

File diff suppressed because it is too large Load Diff

View File

@@ -479,6 +479,12 @@ class Properties:
except KeyError:
return False
def set_properties(self, properties: dict) -> None:
if not self._vertex_or_edge.is_valid():
raise InvalidContextError()
self._vertex_or_edge.set_properties(properties)
class EdgeType:
"""Type of an Edge."""

View File

@@ -141,7 +141,7 @@ declare -A secondary_urls=(
["rocksdb"]="https://github.com/facebook/rocksdb.git"
["mgclient"]="https://github.com/memgraph/mgclient.git"
["pymgclient"]="https://github.com/memgraph/pymgclient.git"
["mgconsole"]="http://github.com/memgraph/mgconsole.git"
["mgconsole"]="https://github.com/memgraph/mgconsole.git"
["spdlog"]="https://github.com/gabime/spdlog"
["nlohmann"]="https://raw.githubusercontent.com/nlohmann/json/4f8fba14066156b73f1189a2b8bd568bde5284c5/single_include/nlohmann/json.hpp"
["neo4j"]="https://dist.neo4j.org/neo4j-community-5.6.0-unix.tar.gz"

View File

@@ -36,7 +36,7 @@ ADDITIONAL USE GRANT: You may use the Licensed Work in accordance with the
3. using the Licensed Work to create a work or solution
which competes (or might reasonably be expected to
compete) with the Licensed Work.
CHANGE DATE: 2027-02-08
CHANGE DATE: 2027-13-09
CHANGE LICENSE: Apache License, Version 2.0
For information about alternative licensing arrangements, please visit: https://memgraph.com/legal.

View File

@@ -6,6 +6,8 @@ project(memgraph_query_modules)
disallow_in_source_build()
set(CMAKE_SHARED_LIBRARY_PREFIX "")
# Everything that is installed here, should be under the "query_modules" component.
set(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME "query_modules")
string(TOLOWER ${CMAKE_BUILD_TYPE} lower_build_type)
@@ -37,9 +39,25 @@ endif()
install(PROGRAMS $<TARGET_FILE:example_cpp>
DESTINATION lib/memgraph/query_modules
RENAME example_cpp.so)
# Also install the source of the example, so user can read it.
install(FILES example.cpp DESTINATION lib/memgraph/query_modules/src)
add_library(example_cpp_transformation SHARED example_transformation.cpp)
target_include_directories(example_cpp_transformation PRIVATE ${CMAKE_SOURCE_DIR}/include)
target_compile_options(example_cpp_transformation PRIVATE -Wall)
# Strip C++ transformation example in release build.
if (lower_build_type STREQUAL "release")
add_custom_command(TARGET example_cpp_transformation POST_BUILD
COMMAND strip -s $<TARGET_FILE:example_cpp_transformation>
COMMENT "Stripping symbols and sections from the C++ transformation example module")
endif()
install(PROGRAMS $<TARGET_FILE:example_cpp_transformation>
DESTINATION lib/memgraph/query_modules)
# Also install the source of the example, so user can read it.
install(FILES example_cpp_transformation.cpp DESTINATION lib/memgraph/query_modules/src)
# Install the Python example and modules
install(FILES example.py DESTINATION lib/memgraph/query_modules RENAME py_example.py)
install(FILES graph_analyzer.py DESTINATION lib/memgraph/query_modules)

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -18,7 +18,11 @@ void ProcImpl(std::vector<mgp::Value> arguments, mgp::Graph graph, mgp::RecordFa
void SampleReadProc(mgp_list *args, mgp_graph *memgraph_graph, mgp_result *result, mgp_memory *memory) {
try {
mgp::memory = memory;
// The outcommented way of assigning the memory pointer is still
// working, but it is deprecated because of certain concurrency
// issues. Please use the guard instead.
// mgp::memory = memory;
mgp::MemoryDispatcherGuard guard(memory);
std::vector<mgp::Value> arguments;
for (size_t i = 0; i < mgp::list_size(args); i++) {
@@ -34,7 +38,11 @@ void SampleReadProc(mgp_list *args, mgp_graph *memgraph_graph, mgp_result *resul
}
void AddXNodes(mgp_list *args, mgp_graph *memgraph_graph, mgp_result *result, mgp_memory *memory) {
mgp::memory = memory;
// The outcommented way of assigning the memory pointer is still
// working, but it is deprecated because of certain concurrency
// issues. Please use the guard instead.
// mgp::memory = memory;
mgp::MemoryDispatcherGuard guard(memory);
auto graph = mgp::Graph(memgraph_graph);
std::vector<mgp::Value> arguments;
@@ -49,7 +57,11 @@ void AddXNodes(mgp_list *args, mgp_graph *memgraph_graph, mgp_result *result, mg
}
void Multiply(mgp_list *args, mgp_func_context *ctx, mgp_func_result *res, mgp_memory *memory) {
mgp::memory = memory;
// The outcommented way of assigning the memory pointer is still
// working, but it is deprecated because of certain concurrency
// issues. Please use the guard instead.
// mgp::memory = memory;
mgp::MemoryDispatcherGuard guard(memory);
std::vector<mgp::Value> arguments;
for (size_t i = 0; i < mgp::list_size(args); i++) {
@@ -67,7 +79,11 @@ void Multiply(mgp_list *args, mgp_func_context *ctx, mgp_func_result *res, mgp_m
extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *memory) {
try {
mgp::memory = memory;
// The outcommented way of assigning the memory pointer is still
// working, but it is deprecated because of certain concurrency
// issues. Please use the guard instead.
// mgp::memory = memory;
mgp::MemoryDispatcherGuard guard(memory);
AddProcedure(SampleReadProc, "return_true", mgp::ProcedureType::Read,
{mgp::Parameter("param_1", mgp::Type::Int), mgp::Parameter("param_2", mgp::Type::Double, 2.3)},
@@ -77,7 +93,11 @@ extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *mem
}
try {
mgp::memory = memory;
// The outcommented way of assigning the memory pointer is still
// working, but it is deprecated because of certain concurrency
// issues. Please use the guard instead.
// mgp::memory = memory;
mgp::MemoryDispatcherGuard guard(memory);
mgp::AddProcedure(AddXNodes, "add_x_nodes", mgp::ProcedureType::Write, {mgp::Parameter("param_1", mgp::Type::Int)},
{}, module, memory);
@@ -87,7 +107,11 @@ extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *mem
}
try {
mgp::memory = memory;
// The outcommented way of assigning the memory pointer is still
// working, but it is deprecated because of certain concurrency
// issues. Please use the guard instead.
// mgp::memory = memory;
mgp::MemoryDispatcherGuard guard(memory);
mgp::AddFunction(Multiply, "multiply",
{mgp::Parameter("int", mgp::Type::Int), mgp::Parameter("int", mgp::Type::Int, (int64_t)3)}, module,

View File

@@ -0,0 +1,64 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <exception>
#include <mgp.hpp>
static constexpr std::string_view kQuery = "query";
static constexpr std::string_view kParameters = "parameters";
std::string EscapeString(std::string s) {
std::string sign = "'";
std::string replace_sign;
size_t pos;
while ((pos = s.find(sign)) != std::string::npos) {
s.replace(pos, 1, replace_sign);
}
return s;
}
void Transformation(struct mgp_messages *messages, mgp_graph *graph, mgp_result *result, mgp_memory *memory) {
mgp::MemoryDispatcherGuard guard(memory);
auto record_factory = mgp::RecordFactory(result);
try {
auto stream_messages = mgp::Messages(messages);
for (const mgp::Message &message : stream_messages) {
auto record = record_factory.NewRecord();
auto payload = EscapeString(message.Payload());
auto query = "CREATE (:Data {payload: '" + payload + "'});";
auto query_value = mgp::Value(query.data());
record.Insert(kQuery.data(), query_value);
record.Insert(kParameters.data(), mgp::Value());
}
} catch (std::exception &ex) {
record_factory.SetErrorMessage(ex.what());
return;
}
}
extern "C" int mgp_init_module(mgp_module *module, mgp_memory *memory) {
try {
mgp::MemoryDispatcherGuard guard(memory);
mgp::AddTransformation(Transformation, "transform", module);
} catch (const std::exception &e) {
return 1;
}
return 0;
}
extern "C" int mgp_shutdown_module() { return 0; }

View File

@@ -1,3 +1,4 @@
/etc/memgraph/memgraph.conf
/etc/memgraph/apoc_compatibility_mappings.json
/etc/memgraph/auth_module/ldap.example.yaml
/etc/logrotate.d/memgraph

View File

@@ -132,6 +132,7 @@ echo "Don't forget to switch to the 'memgraph' user to use Memgraph" || exit 1
# Override CPACK_RPM_ABSOLUTE_INSTALL_FILES with our %config(noreplace), cpack
# uses plain %config.
%config(noreplace) "/etc/memgraph/memgraph.conf"
%config(noreplace) "/etc/memgraph/apoc_compatibility_mappings.json"
%config(noreplace) "/etc/memgraph/auth_module/ldap.example.yaml"
%config(noreplace) "/etc/logrotate.d/memgraph"

View File

@@ -18,6 +18,7 @@ add_subdirectory(rpc)
add_subdirectory(license)
add_subdirectory(auth)
add_subdirectory(audit)
add_subdirectory(flags)
string(TOLOWER ${CMAKE_BUILD_TYPE} lower_build_type)
@@ -33,13 +34,11 @@ set(mg_single_node_v2_sources
memgraph.cpp
)
set(mg_single_node_v2_libs stdc++fs Threads::Threads
mg-telemetry mg-query mg-communication mg-memory mg-utils mg-auth mg-license mg-settings mg-glue mg-audit)
# memgraph main executable
add_executable(memgraph ${mg_single_node_v2_sources})
target_include_directories(memgraph PUBLIC ${CMAKE_SOURCE_DIR}/include)
target_link_libraries(memgraph ${mg_single_node_v2_libs})
target_link_libraries(memgraph stdc++fs Threads::Threads
mg-telemetry mg-communication mg-memory mg-utils mg-license mg-settings mg-glue mg-flags)
# NOTE: `include/mg_procedure.syms` describes a pattern match for symbols which
# should be dynamically exported, so that `dlopen` can correctly link the
@@ -72,7 +71,7 @@ if(lower_build_type STREQUAL "release")
COMMENT "Stripping symbols and sections from memgraph")
endif()
# Generate the configuration file.
# Generate the configuration file under the build directory.
add_custom_command(TARGET memgraph POST_BUILD
COMMAND ${CMAKE_SOURCE_DIR}/config/generate.py
${CMAKE_BINARY_DIR}/memgraph
@@ -81,6 +80,11 @@ add_custom_command(TARGET memgraph POST_BUILD
${CMAKE_SOURCE_DIR}/config/flags.yaml
BYPRODUCTS ${CMAKE_BINARY_DIR}/config/memgraph.conf
COMMENT "Generating memgraph configuration file")
# Copy the mappings file to the build directory.
add_custom_command(TARGET memgraph POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy
${CMAKE_SOURCE_DIR}/config/mappings.json
${CMAKE_BINARY_DIR}/config/apoc_compatibility_mappings.json)
# Everything here is under "memgraph" install component.
set(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME "memgraph")
@@ -116,6 +120,9 @@ install(FILES ${CMAKE_SOURCE_DIR}/include/mgp.hpp
# Install the config file (must use absolute path).
install(FILES ${CMAKE_BINARY_DIR}/config/memgraph.conf
DESTINATION /etc/memgraph RENAME memgraph.conf)
# Install the mappings file (must use absolute path).
install(FILES ${CMAKE_BINARY_DIR}/config/apoc_compatibility_mappings.json
DESTINATION /etc/memgraph RENAME apoc_compatibility_mappings.json)
# Install logrotate configuration (must use absolute path).
install(FILES ${CMAKE_SOURCE_DIR}/release/logrotate.conf

View File

@@ -22,3 +22,4 @@ target_link_libraries(mg-communication Boost::headers Threads::Threads mg-utils
find_package(OpenSSL REQUIRED)
target_link_libraries(mg-communication ${OPENSSL_LIBRARIES})
target_include_directories(mg-communication SYSTEM PUBLIC ${OPENSSL_INCLUDE_DIR})
target_precompile_headers(mg-communication INTERFACE http/server.hpp <boost/beast/websocket.hpp> bolt/v1/session.hpp)

View File

@@ -159,6 +159,7 @@ class Session : public dbms::SessionInterface {
break;
case State::Idle:
case State::Result:
at_least_one_run_ = true;
state_ = StateExecutingRun(*this, state_);
break;
case State::Error:
@@ -180,6 +181,12 @@ class Session : public dbms::SessionInterface {
}
}
void HandleError() {
if (!at_least_one_run_) {
spdlog::info("Sudden connection loss. Make sure the client supports Memgraph.");
}
}
// TODO: Rethink if there is a way to hide some members. At the momement all of them are public.
TInputStream &input_stream_;
TOutputStream &output_stream_;
@@ -192,6 +199,7 @@ class Session : public dbms::SessionInterface {
bool handshake_done_{false};
State state_{State::Handshake};
bool at_least_one_run_{false};
struct Version {
uint8_t major;

View File

@@ -174,7 +174,7 @@ State SendSuccessMessage(TSession &session) {
// we send a hardcoded value for now.
std::map<std::string, Value> metadata{{"connection_id", "bolt-1"}};
if (auto server_name = session.GetServerNameForInit(); server_name) {
metadata.insert({"server", *server_name});
metadata.insert({"server", std::move(*server_name)});
}
bool success_sent = session.encoder_.MessageSuccess(metadata);
if (!success_sent) {

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -43,6 +43,7 @@ class Buffer final {
Buffer(Buffer &&) = delete;
Buffer &operator=(const Buffer &) = delete;
Buffer &operator=(Buffer &&) = delete;
~Buffer() = default;
/**
* This class provides all functions from the buffer that are needed to allow
@@ -50,12 +51,13 @@ class Buffer final {
*/
class ReadEnd {
public:
ReadEnd(Buffer *buffer);
explicit ReadEnd(Buffer *buffer);
ReadEnd(const ReadEnd &) = delete;
ReadEnd(ReadEnd &&) = delete;
ReadEnd &operator=(const ReadEnd &) = delete;
ReadEnd &operator=(ReadEnd &&) = delete;
~ReadEnd() = default;
uint8_t *data();
@@ -77,12 +79,13 @@ class Buffer final {
*/
class WriteEnd {
public:
WriteEnd(Buffer *buffer);
explicit WriteEnd(Buffer *buffer);
WriteEnd(const WriteEnd &) = delete;
WriteEnd(WriteEnd &&) = delete;
WriteEnd &operator=(const WriteEnd &) = delete;
WriteEnd &operator=(WriteEnd &&) = delete;
~WriteEnd() = default;
io::network::StreamBuffer Allocate();

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -15,6 +15,10 @@
#include <openssl/err.h>
#include <openssl/ssl.h>
// Centos 7 OpenSSL includes libkrb5 which has brings in macros TRUE and FALSE. undef to prevent issues.
#undef TRUE
#undef FALSE
#include "communication/buffer.hpp"
#include "communication/context.hpp"
#include "communication/init.hpp"

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -17,6 +17,10 @@
#include <openssl/ssl.h>
#include <boost/asio/ssl/context.hpp>
// Centos 7 OpenSSL includes libkrb5 which has brings in macros TRUE and FALSE. undef to prevent issues.
#undef TRUE
#undef FALSE
namespace memgraph::communication {
/**

View File

@@ -26,10 +26,7 @@ class Server final {
using tcp = boost::asio::ip::tcp;
public:
explicit Server(io::network::Endpoint endpoint, TSessionContext *session_context, ServerContext *context)
: listener_{Listener<TRequestHandler, TSessionContext>::Create(
ioc_, session_context, context,
tcp::endpoint{boost::asio::ip::make_address(endpoint.address), endpoint.port})} {}
explicit Server(io::network::Endpoint endpoint, TSessionContext *session_context, ServerContext *context);
Server(const Server &) = delete;
Server(Server &&) = delete;
@@ -41,11 +38,7 @@ class Server final {
"Server wasn't shutdown properly");
}
void Start() {
MG_ASSERT(!background_thread_, "The server was already started!");
listener_->Run();
background_thread_.emplace([this] { ioc_.run(); });
}
void Start();
void Shutdown() { ioc_.stop(); }
@@ -55,7 +48,7 @@ class Server final {
}
}
bool IsRunning() const { return background_thread_ && !ioc_.stopped(); }
tcp::endpoint GetEndpoint() const { return listener_->GetEndpoint(); }
tcp::endpoint GetEndpoint() const;
private:
boost::asio::io_context ioc_;
@@ -63,4 +56,22 @@ class Server final {
std::shared_ptr<Listener<TRequestHandler, TSessionContext>> listener_;
std::optional<std::thread> background_thread_;
};
template <class TRequestHandler, typename TSessionContext>
Server<TRequestHandler, TSessionContext>::Server(io::network::Endpoint endpoint, TSessionContext *session_context,
ServerContext *context)
: listener_{Listener<TRequestHandler, TSessionContext>::Create(
ioc_, session_context, context,
tcp::endpoint{boost::asio::ip::make_address(endpoint.address), endpoint.port})} {}
template <class TRequestHandler, typename TSessionContext>
void Server<TRequestHandler, TSessionContext>::Start() {
MG_ASSERT(!background_thread_, "The server was already started!");
listener_->Run();
background_thread_.emplace([this] { ioc_.run(); });
}
template <class TRequestHandler, typename TSessionContext>
boost::asio::ip::tcp::endpoint Server<TRequestHandler, TSessionContext>::GetEndpoint() const {
return listener_->GetEndpoint();
}
} // namespace memgraph::communication::http

View File

@@ -58,11 +58,11 @@ class Server final {
* Constructs and binds server to endpoint, operates on session data and
* invokes workers_count workers
*/
Server(const io::network::Endpoint &endpoint, TSessionContext *session_context, ServerContext *context,
Server(io::network::Endpoint endpoint, TSessionContext *session_context, ServerContext *context,
int inactivity_timeout_sec, const std::string &service_name,
size_t workers_count = std::thread::hardware_concurrency())
: alive_(false),
endpoint_(endpoint),
endpoint_(std::move(endpoint)),
listener_(session_context, context, inactivity_timeout_sec, service_name, workers_count),
service_name_(service_name) {}

View File

@@ -22,6 +22,10 @@
#include <openssl/err.h>
#include <openssl/ssl.h>
// Centos 7 OpenSSL includes libkrb5 which has brings in macros TRUE and FALSE. undef to prevent issues.
#undef TRUE
#undef FALSE
#include "communication/buffer.hpp"
#include "communication/context.hpp"
#include "communication/exceptions.hpp"

View File

@@ -73,40 +73,19 @@ class Server final {
* invokes workers_count workers
*/
Server(ServerEndpoint &endpoint, TSessionContext *session_context, ServerContext *server_context,
const int inactivity_timeout_sec, const std::string_view service_name,
size_t workers_count = std::thread::hardware_concurrency())
: endpoint_{endpoint},
service_name_{service_name},
context_thread_pool_{workers_count},
listener_{Listener<TSession, TSessionContext>::Create(context_thread_pool_.GetIOContext(), session_context,
server_context, endpoint_, service_name_,
inactivity_timeout_sec)} {}
int inactivity_timeout_sec, std::string_view service_name,
size_t workers_count = std::thread::hardware_concurrency());
~Server() { MG_ASSERT(!IsRunning(), "Server wasn't shutdown properly"); }
~Server();
Server(const Server &) = delete;
Server(Server &&) = delete;
Server &operator=(const Server &) = delete;
Server &operator=(Server &&) = delete;
const auto &Endpoint() const {
MG_ASSERT(IsRunning(), "You can't get the server endpoint when it's not running!");
return endpoint_;
}
const auto &Endpoint() const;
bool Start() {
if (IsRunning()) {
spdlog::error("The server is already running");
return false;
}
listener_->Start();
spdlog::info("{} server is fully armed and operational", service_name_);
spdlog::info("{} listening on {}", service_name_, endpoint_.address());
context_thread_pool_.Run();
return true;
}
bool Start();
void Shutdown() {
context_thread_pool_.Shutdown();
@@ -115,7 +94,7 @@ class Server final {
void AwaitShutdown() { context_thread_pool_.AwaitShutdown(); }
bool IsRunning() const noexcept { return context_thread_pool_.IsRunning() && listener_->IsRunning(); }
bool IsRunning() const noexcept;
private:
ServerEndpoint endpoint_;
@@ -125,4 +104,46 @@ class Server final {
std::shared_ptr<Listener<TSession, TSessionContext>> listener_;
};
template <typename TSession, typename TSessionContext>
Server<TSession, TSessionContext>::~Server() {
MG_ASSERT(!IsRunning(), "Server wasn't shutdown properly");
}
template <typename TSession, typename TSessionContext>
Server<TSession, TSessionContext>::Server(ServerEndpoint &endpoint, TSessionContext *session_context,
ServerContext *server_context, const int inactivity_timeout_sec,
const std::string_view service_name, size_t workers_count)
: endpoint_{endpoint},
service_name_{service_name},
context_thread_pool_{workers_count},
listener_{Listener<TSession, TSessionContext>::Create(context_thread_pool_.GetIOContext(), session_context,
server_context, endpoint_, service_name_,
inactivity_timeout_sec)} {}
template <typename TSession, typename TSessionContext>
bool Server<TSession, TSessionContext>::Start() {
if (IsRunning()) {
spdlog::error("The server is already running");
return false;
}
listener_->Start();
spdlog::info("{} server is fully armed and operational", service_name_);
spdlog::info("{} listening on {}", service_name_, endpoint_.address());
context_thread_pool_.Run();
return true;
}
template <typename TSession, typename TSessionContext>
const auto &Server<TSession, TSessionContext>::Endpoint() const {
MG_ASSERT(IsRunning(), "You can't get the server endpoint when it's not running!");
return endpoint_;
}
template <typename TSession, typename TSessionContext>
bool Server<TSession, TSessionContext>::IsRunning() const noexcept {
return context_thread_pool_.IsRunning() && listener_->IsRunning();
}
} // namespace memgraph::communication::v2

View File

@@ -413,6 +413,8 @@ class Session final : public std::enable_shared_from_this<Session<TSession, TSes
void OnRead(const boost::system::error_code &ec, const size_t bytes_transferred) {
if (ec) {
// TODO Check if client disconnected
session_.HandleError();
return OnError(ec);
}
input_buffer_.write_end()->Written(bytes_transferred);

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -31,7 +31,7 @@
template <typename TElement>
class RingBuffer {
public:
explicit RingBuffer(int capacity) : capacity_(capacity) { buffer_ = std::make_unique<TElement[]>(capacity_); }
explicit RingBuffer(int capacity) : capacity_(capacity), buffer_{std::make_unique<TElement[]>(capacity_)} {}
RingBuffer(const RingBuffer &) = delete;
RingBuffer(RingBuffer &&) = delete;

9
src/flags/CMakeLists.txt Normal file
View File

@@ -0,0 +1,9 @@
add_library(mg-flags STATIC audit.cpp
bolt.cpp
general.cpp
isolation_level.cpp
log_level.cpp
memory_limit.cpp
run_time_configurable.cpp)
target_include_directories(mg-flags PUBLIC ${CMAKE_SOURCE_DIR}/include)
target_link_libraries(mg-flags PUBLIC spdlog::spdlog mg-settings mg-utils)

19
src/flags/all.hpp Normal file
View File

@@ -0,0 +1,19 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "flags/audit.hpp"
#include "flags/bolt.hpp"
#include "flags/general.hpp"
#include "flags/isolation_level.hpp"
#include "flags/log_level.hpp"
#include "flags/memory_limit.hpp"
#include "flags/run_time_configurable.hpp"

28
src/flags/audit.cpp Normal file
View File

@@ -0,0 +1,28 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "flags/audit.hpp"
#include "audit/log.hpp"
#include "utils/flag_validation.hpp"
// Audit logging flags.
#ifdef MG_ENTERPRISE
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(audit_enabled, false, "Set to true to enable audit logging.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(audit_buffer_size, memgraph::audit::kBufferSizeDefault,
"Maximum number of items in the audit log buffer.", FLAG_IN_RANGE(1, INT32_MAX));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(audit_buffer_flush_interval_ms, memgraph::audit::kBufferFlushIntervalMillisDefault,
"Interval (in milliseconds) used for flushing the audit log buffer.",
FLAG_IN_RANGE(10, INT32_MAX));
#endif

23
src/flags/audit.hpp Normal file
View File

@@ -0,0 +1,23 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "gflags/gflags.h"
// Audit logging flags.
#ifdef MG_ENTERPRISE
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(audit_enabled);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_int32(audit_buffer_size);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_int32(audit_buffer_flush_interval_ms);
#endif

42
src/flags/bolt.cpp Normal file
View File

@@ -0,0 +1,42 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "flags/bolt.hpp"
#include "utils/flag_validation.hpp"
#include <limits>
#include <thread>
// Bolt server flags.
DEFINE_string(bolt_address, "0.0.0.0", "IP address on which the Bolt server should listen.");
DEFINE_VALIDATED_int32(bolt_port, 7687, "Port on which the Bolt server should listen.",
FLAG_IN_RANGE(0, std::numeric_limits<uint16_t>::max()));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(bolt_num_workers, std::max(std::thread::hardware_concurrency(), 1U),
"Number of workers used by the Bolt server. By default, this will be the "
"number of processing units available on the machine.",
FLAG_IN_RANGE(1, INT32_MAX));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(bolt_session_inactivity_timeout, 1800,
"Time in seconds after which inactive Bolt sessions will be "
"closed.",
FLAG_IN_RANGE(1, INT32_MAX));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(bolt_cert_file, "", "Certificate file which should be used for the Bolt server.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(bolt_key_file, "", "Key file which should be used for the Bolt server.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(bolt_server_name_for_init, "",
"Server name which the database should send to the client in the "
"Bolt INIT message.");

29
src/flags/bolt.hpp Normal file
View File

@@ -0,0 +1,29 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "gflags/gflags.h"
// Bolt server flags.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(bolt_address);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_int32(bolt_port);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_int32(bolt_num_workers);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_int32(bolt_session_inactivity_timeout);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(bolt_cert_file);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(bolt_key_file);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(bolt_server_name_for_init);

198
src/flags/general.cpp Normal file
View File

@@ -0,0 +1,198 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "general.hpp"
#include "glue/auth_global.hpp"
#include "storage/v2/config.hpp"
#include "utils/file.hpp"
#include "utils/flag_validation.hpp"
#include "utils/string.hpp"
#include <thread>
// Short help flag.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_bool(h, false, "Print usage and exit.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(monitoring_address, "0.0.0.0",
"IP address on which the websocket server for Memgraph monitoring should listen.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(metrics_address, "0.0.0.0",
"IP address on which the Memgraph server for exposing metrics should listen.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(monitoring_port, 7444,
"Port on which the websocket server for Memgraph monitoring should listen.",
FLAG_IN_RANGE(0, std::numeric_limits<uint16_t>::max()));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(metrics_port, 9091, "Port on which the Memgraph server for exposing metrics should listen.",
FLAG_IN_RANGE(0, std::numeric_limits<uint16_t>::max()));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(init_file, "",
"Path to cypherl file that is used for configuring users and database schema before server starts.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(init_data_file, "", "Path to cypherl file that is used for creating data after server starts.");
// General purpose flags.
// NOTE: The `data_directory` flag must be the same here and in
// `mg_import_csv`. If you change it, make sure to change it there as well.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(data_directory, "mg_data", "Path to directory in which to save all permanent data.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(data_recovery_on_startup, false, "Controls whether the database recovers persisted data on startup.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(memory_warning_threshold, 1024,
"Memory warning threshold, in MB. If Memgraph detects there is "
"less available RAM it will log a warning. Set to 0 to "
"disable.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(allow_load_csv, true, "Controls whether LOAD CSV clause is allowed in queries.");
// Storage flags.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_gc_cycle_sec, 30, "Storage garbage collector interval (in seconds).",
FLAG_IN_RANGE(1, 24 * 3600));
// NOTE: The `storage_properties_on_edges` flag must be the same here and in
// `mg_import_csv`. If you change it, make sure to change it there as well.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_properties_on_edges, false, "Controls whether edges have properties.");
// storage_recover_on_startup deprecated; use data_recovery_on_startup instead
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_bool(storage_recover_on_startup, false,
"Controls whether the storage recovers persisted data on startup.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_snapshot_interval_sec, 0,
"Storage snapshot creation interval (in seconds). Set "
"to 0 to disable periodic snapshot creation.",
FLAG_IN_RANGE(0, 7 * 24 * 3600));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_wal_enabled, false,
"Controls whether the storage uses write-ahead-logging. To enable "
"WAL periodic snapshots must be enabled.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_snapshot_retention_count, 3, "The number of snapshots that should always be kept.",
FLAG_IN_RANGE(1, 1000000));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_wal_file_size_kib, memgraph::storage::Config::Durability().wal_file_size_kibibytes,
"Minimum file size of each WAL file.",
FLAG_IN_RANGE(1, static_cast<unsigned long>(1000) * 1024));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_wal_file_flush_every_n_tx,
memgraph::storage::Config::Durability().wal_file_flush_every_n_tx,
"Issue a 'fsync' call after this amount of transactions are written to the "
"WAL file. Set to 1 for fully synchronous operation.",
FLAG_IN_RANGE(1, 1000000));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_snapshot_on_exit, false, "Controls whether the storage creates another snapshot on exit.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(storage_items_per_batch, memgraph::storage::Config::Durability().items_per_batch,
"The number of edges and vertices stored in a batch in a snapshot file.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_parallel_index_recovery, false,
"Controls whether the index creation can be done in a multithreaded fashion.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(storage_recovery_thread_count,
std::max(static_cast<uint64_t>(std::thread::hardware_concurrency()),
memgraph::storage::Config::Durability().recovery_thread_count),
"The number of threads used to recover persisted data from disk.");
#ifdef MG_ENTERPRISE
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_delete_on_drop, true,
"If set to true the query 'DROP DATABASE x' will delete the underlying storage as well.");
#endif
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(telemetry_enabled, false,
"Set to true to enable telemetry. We collect information about the "
"running system (CPU and memory information) and information about "
"the database runtime (vertex and edge counts and resource usage) "
"to allow for easier improvement of the product.");
// Streams flags
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint32(
stream_transaction_conflict_retries, 30,
"Number of times to retry when a stream transformation fails to commit because of conflicting transactions");
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint32(
stream_transaction_retry_interval, 500,
"Retry interval in milliseconds when a stream transformation fails to commit because of conflicting transactions");
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(kafka_bootstrap_servers, "",
"List of default Kafka brokers as a comma separated list of broker host or host:port.");
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(pulsar_service_url, "", "Default URL used while connecting to Pulsar brokers.");
// Query flags.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_double(query_execution_timeout_sec, -1,
"Maximum allowed query execution time. Queries exceeding this "
"limit will be aborted. Value of 0 means no limit.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(replication_replica_check_frequency_sec, 1,
"The time duration between two replica checks/pings. If < 1, replicas will NOT be checked at all. NOTE: "
"The MAIN instance allocates a new thread for each REPLICA.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(replication_restore_state_on_startup, false, "Restore replication state on startup, e.g. recover replica");
DEFINE_VALIDATED_string(query_modules_directory, "",
"Directory where modules with custom query procedures are stored. "
"NOTE: Multiple comma-separated directories can be defined.",
{
if (value.empty()) return true;
const auto directories = memgraph::utils::Split(value, ",");
for (const auto &dir : directories) {
if (!memgraph::utils::DirExists(dir)) {
std::cout << "Expected --" << flagname << " to point to directories." << std::endl;
std::cout << dir << " is not a directory." << std::endl;
return false;
}
}
return true;
});
auto memgraph::flags::ParseQueryModulesDirectory() -> std::vector<std::filesystem::path> {
const auto directories = memgraph::utils::Split(FLAGS_query_modules_directory, ",");
std::vector<std::filesystem::path> query_modules_directories;
query_modules_directories.reserve(directories.size());
std::transform(directories.begin(), directories.end(), std::back_inserter(query_modules_directories),
[](const auto &dir) { return dir; });
return query_modules_directories;
}
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(query_callable_mappings_path, "",
"The path to mappings that describes aliases to callables in cypher queries in the form of key-value "
"pairs in a json file. With this option query module procedures that do not exist in memgraph can be "
"mapped to ones that exist.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_string(license_key, "", "License key for Memgraph Enterprise.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_string(organization_name, "", "Organization name.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(auth_user_or_role_name_regex, memgraph::glue::kDefaultUserRoleRegex.data(),
"Set to the regular expression that each user or role name must fulfill.");

122
src/flags/general.hpp Normal file
View File

@@ -0,0 +1,122 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "gflags/gflags.h"
#include <filesystem>
// Short help flag.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(h);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(monitoring_address);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_int32(monitoring_port);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(metrics_address);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_int32(metrics_port);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(init_file);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(init_data_file);
// General purpose flags.
// NOTE: The `data_directory` flag must be the same here and in
// `mg_import_csv`. If you change it, make sure to change it there as well.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(data_directory);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(data_recovery_on_startup);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint64(memory_warning_threshold);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(allow_load_csv);
// Storage flags.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint64(storage_gc_cycle_sec);
// NOTE: The `storage_properties_on_edges` flag must be the same here and in
// `mg_import_csv`. If you change it, make sure to change it there as well.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(storage_properties_on_edges);
// storage_recover_on_startup deprecated; use data_recovery_on_startup instead
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(storage_recover_on_startup);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint64(storage_snapshot_interval_sec);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(storage_wal_enabled);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint64(storage_snapshot_retention_count);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint64(storage_wal_file_size_kib);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint64(storage_wal_file_flush_every_n_tx);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(storage_snapshot_on_exit);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint64(storage_items_per_batch);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(storage_parallel_index_recovery);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint64(storage_recovery_thread_count);
#ifdef MG_ENTERPRISE
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(storage_delete_on_drop);
#endif
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(telemetry_enabled);
// Streams flags
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint32(stream_transaction_conflict_retries);
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint32(stream_transaction_retry_interval);
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(kafka_bootstrap_servers);
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(pulsar_service_url);
// Query flags.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_double(query_execution_timeout_sec);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(query_modules_directory);
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(query_callable_mappings_path);
namespace memgraph::flags {
auto ParseQueryModulesDirectory() -> std::vector<std::filesystem::path>;
} // namespace memgraph::flags
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_uint64(replication_replica_check_frequency_sec);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_bool(replication_restore_state_on_startup);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(license_key);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(organization_name);
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DECLARE_string(auth_user_or_role_name_regex);

View File

@@ -0,0 +1,54 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "flags/isolation_level.hpp"
#include "utils/enum.hpp"
#include "utils/flag_validation.hpp"
#include "gflags/gflags.h"
#include <array>
#include <string_view>
inline constexpr std::array isolation_level_mappings{
std::pair{std::string_view{"SNAPSHOT_ISOLATION"}, memgraph::storage::IsolationLevel::SNAPSHOT_ISOLATION},
std::pair{std::string_view{"READ_COMMITTED"}, memgraph::storage::IsolationLevel::READ_COMMITTED},
std::pair{std::string_view{"READ_UNCOMMITTED"}, memgraph::storage::IsolationLevel::READ_UNCOMMITTED}};
const std::string isolation_level_help_string =
fmt::format("Default isolation level used for the transactions. Allowed values: {}",
memgraph::utils::GetAllowedEnumValuesString(isolation_level_mappings));
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_string(isolation_level, "SNAPSHOT_ISOLATION", isolation_level_help_string.c_str(), {
if (const auto result = memgraph::utils::IsValidEnumValueString(value, isolation_level_mappings); result.HasError()) {
switch (result.GetError()) {
case memgraph::utils::ValidationError::EmptyValue: {
std::cout << "Isolation level cannot be empty." << std::endl;
break;
}
case memgraph::utils::ValidationError::InvalidValue: {
std::cout << "Invalid value for isolation level. Allowed values: "
<< memgraph::utils::GetAllowedEnumValuesString(isolation_level_mappings) << std::endl;
break;
}
}
return false;
}
return true;
});
memgraph::storage::IsolationLevel memgraph::flags::ParseIsolationLevel() {
const auto isolation_level =
memgraph::utils::StringToEnum<memgraph::storage::IsolationLevel>(FLAGS_isolation_level, isolation_level_mappings);
MG_ASSERT(isolation_level, "Invalid isolation level");
return *isolation_level;
}

View File

@@ -0,0 +1,19 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "storage/v2/isolation_level.hpp"
namespace memgraph::flags {
memgraph::storage::IsolationLevel ParseIsolationLevel();
} // namespace memgraph::flags

122
src/flags/log_level.cpp Normal file
View File

@@ -0,0 +1,122 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "flags/log_level.hpp"
#include "utils/enum.hpp"
#include "utils/flag_validation.hpp"
#include "utils/logging.hpp"
#include "gflags/gflags.h"
#include "spdlog/common.h"
#include "spdlog/sinks/daily_file_sink.h"
#include <array>
#include <string_view>
#include <utility>
using namespace std::string_view_literals;
// Logging flags
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_bool(also_log_to_stderr, false, "Log messages go to stderr in addition to logfiles");
DEFINE_string(log_file, "", "Path to where the log should be stored.");
inline constexpr std::array log_level_mappings{
std::pair{"TRACE"sv, spdlog::level::trace}, std::pair{"DEBUG"sv, spdlog::level::debug},
std::pair{"INFO"sv, spdlog::level::info}, std::pair{"WARNING"sv, spdlog::level::warn},
std::pair{"ERROR"sv, spdlog::level::err}, std::pair{"CRITICAL"sv, spdlog::level::critical}};
const std::string log_level_help_string = fmt::format("Minimum log level. Allowed values: {}",
memgraph::utils::GetAllowedEnumValuesString(log_level_mappings));
DEFINE_VALIDATED_string(log_level, "WARNING", log_level_help_string.c_str(),
{ return memgraph::flags::ValidLogLevel(value); });
bool memgraph::flags::ValidLogLevel(std::string_view value) {
if (const auto result = memgraph::utils::IsValidEnumValueString(value, log_level_mappings); result.HasError()) {
const auto error = result.GetError();
switch (error) {
case memgraph::utils::ValidationError::EmptyValue: {
std::cout << "Log level cannot be empty." << std::endl;
break;
}
case memgraph::utils::ValidationError::InvalidValue: {
std::cout << "Invalid value for log level. Allowed values: "
<< memgraph::utils::GetAllowedEnumValuesString(log_level_mappings) << std::endl;
break;
}
}
return false;
}
return true;
}
std::optional<spdlog::level::level_enum> memgraph::flags::LogLevelToEnum(std::string_view value) {
return memgraph::utils::StringToEnum<spdlog::level::level_enum>(value, log_level_mappings);
}
spdlog::level::level_enum ParseLogLevel() {
const auto log_level = memgraph::flags::LogLevelToEnum(FLAGS_log_level);
MG_ASSERT(log_level, "Invalid log level");
return *log_level;
}
// 5 weeks * 7 days
inline constexpr auto log_retention_count = 35;
void CreateLoggerFromSink(const auto &sinks, const auto log_level) {
auto logger = std::make_shared<spdlog::logger>("memgraph_log", sinks.begin(), sinks.end());
logger->set_level(log_level);
logger->flush_on(spdlog::level::trace);
spdlog::set_default_logger(std::move(logger));
// Enable stderr sink
if (FLAGS_also_log_to_stderr) {
memgraph::flags::LogToStderr(log_level);
}
}
void memgraph::flags::InitializeLogger() {
std::vector<spdlog::sink_ptr> sinks;
// Force the stderr logger to be at the front of the sinks vector
// Will be used to disable/enable it at run-time by settings its log level
sinks.emplace_back(std::make_shared<spdlog::sinks::stderr_color_sink_mt>());
sinks.back()->set_level(spdlog::level::off);
if (!FLAGS_log_file.empty()) {
// get local time
time_t current_time{0};
struct tm *local_time{nullptr};
time(&current_time);
local_time = localtime(&current_time);
sinks.emplace_back(std::make_shared<spdlog::sinks::daily_file_sink_mt>(
FLAGS_log_file, local_time->tm_hour, local_time->tm_min, false, log_retention_count));
}
CreateLoggerFromSink(sinks, ParseLogLevel());
}
// TODO: Make sure this is used in a safe way
void memgraph::flags::AddLoggerSink(spdlog::sink_ptr new_sink) {
auto default_logger = spdlog::default_logger();
auto sinks = default_logger->sinks();
sinks.push_back(new_sink);
CreateLoggerFromSink(sinks, default_logger->level());
}
// Thread-safe because the level enum is an atomic
// NOTE: default_logger is not thread-safe and shouldn't be changed during application lifetime
void memgraph::flags::LogToStderr(spdlog::level::level_enum log_level) {
auto default_logger = spdlog::default_logger();
auto sink = default_logger->sinks().front();
sink->set_level(log_level);
}

29
src/flags/log_level.hpp Normal file
View File

@@ -0,0 +1,29 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <spdlog/sinks/sink.h>
#include <optional>
#include "gflags/gflags.h"
DECLARE_string(log_level);
DECLARE_bool(also_log_to_stderr);
namespace memgraph::flags {
bool ValidLogLevel(std::string_view value);
std::optional<spdlog::level::level_enum> LogLevelToEnum(std::string_view value);
void InitializeLogger();
void AddLoggerSink(spdlog::sink_ptr new_sink);
void LogToStderr(spdlog::level::level_enum log_level);
} // namespace memgraph::flags

View File

@@ -0,0 +1,41 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "flags/memory_limit.hpp"
#include "utils/logging.hpp"
#include "utils/sysinfo/memory.hpp"
#include "gflags/gflags.h"
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(
memory_limit, 0,
"Total memory limit in MiB. Set to 0 to use the default values which are 100\% of the phyisical memory if the swap "
"is enabled and 90\% of the physical memory otherwise.");
int64_t memgraph::flags::GetMemoryLimit() {
if (FLAGS_memory_limit == 0) {
auto maybe_total_memory = memgraph::utils::sysinfo::TotalMemory();
MG_ASSERT(maybe_total_memory, "Failed to fetch the total physical memory");
const auto maybe_swap_memory = memgraph::utils::sysinfo::SwapTotalMemory();
MG_ASSERT(maybe_swap_memory, "Failed to fetch the total swap memory");
if (*maybe_swap_memory == 0) {
// take only 90% of the total memory
*maybe_total_memory *= 9;
*maybe_total_memory /= 10;
}
return *maybe_total_memory * 1024;
}
// We parse the memory as MiB every time
return FLAGS_memory_limit * 1024 * 1024;
}

View File

@@ -0,0 +1,17 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <cstdint>
namespace memgraph::flags {
int64_t GetMemoryLimit();
} // namespace memgraph::flags

View File

@@ -0,0 +1,111 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "flags/run_time_configurable.hpp"
#include <string>
#include "flags/bolt.hpp"
#include "flags/general.hpp"
#include "flags/log_level.hpp"
#include "spdlog/cfg/helpers-inl.h"
#include "spdlog/spdlog.h"
#include "utils/exceptions.hpp"
#include "utils/settings.hpp"
#include "utils/string.hpp"
namespace {
// Bolt server name
constexpr auto kServerNameSettingKey = "server.name";
constexpr auto kDefaultServerName = "Neo4j/v5.11.0 compatible graph database server - Memgraph";
// Query timeout
constexpr auto kQueryTxSettingKey = "query.timeout";
constexpr auto kDefaultQueryTx = "600"; // seconds
// Log level
// No default value because it is not persistent
constexpr auto kLogLevelSettingKey = "log.level";
// Log to stderr
// No default value because it is not persistent
constexpr auto kLogToStderrSettingKey = "log.to_stderr";
} // namespace
namespace memgraph::flags::run_time {
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
memgraph::utils::Synchronized<std::string, memgraph::utils::SpinLock> bolt_server_name_;
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
std::atomic<double> execution_timeout_sec_;
void Initialize() {
// Register bolt server name settings
memgraph::utils::global_settings.RegisterSetting(kServerNameSettingKey, kDefaultServerName, [&] {
const auto server_name = memgraph::utils::global_settings.GetValue(kServerNameSettingKey);
MG_ASSERT(server_name, "Bolt server name is missing from the settings");
*(bolt_server_name_.Lock()) = *server_name;
});
// Update value from read settings
const auto &name = memgraph::utils::global_settings.GetValue(kServerNameSettingKey);
MG_ASSERT(name, "Failed to read server name from settings.");
*(bolt_server_name_.Lock()) = *name;
// Override server name if passed via command line argument
if (!FLAGS_bolt_server_name_for_init.empty()) {
memgraph::utils::global_settings.SetValue(kServerNameSettingKey, FLAGS_bolt_server_name_for_init);
}
// Register query timeout
memgraph::utils::global_settings.RegisterSetting(kQueryTxSettingKey, kDefaultQueryTx, [&] {
const auto query_tx = memgraph::utils::global_settings.GetValue(kQueryTxSettingKey);
MG_ASSERT(query_tx, "Query timeout is missing from the settings");
execution_timeout_sec_ = std::stod(*query_tx);
});
// Update value from read settings
const auto &tx = memgraph::utils::global_settings.GetValue(kQueryTxSettingKey);
MG_ASSERT(tx, "Failed to read query timeout from settings.");
execution_timeout_sec_ = std::stod(*tx);
// Override query timeout if passed via command line argument
if (FLAGS_query_execution_timeout_sec != -1) {
memgraph::utils::global_settings.SetValue(kQueryTxSettingKey, std::to_string(FLAGS_query_execution_timeout_sec));
}
// Register log level
auto get_global_log_level = []() {
const auto log_level = memgraph::utils::global_settings.GetValue(kLogLevelSettingKey);
MG_ASSERT(log_level, "Log level is missing from the settings");
const auto ll_enum = memgraph::flags::LogLevelToEnum(*log_level);
if (!ll_enum) {
throw utils::BasicException("Unsupported log level {}", *log_level);
}
return *ll_enum;
};
memgraph::utils::global_settings.RegisterSetting(
kLogLevelSettingKey, FLAGS_log_level, [&] { spdlog::set_level(get_global_log_level()); },
memgraph::flags::ValidLogLevel);
// Always override log level with command line argument
memgraph::utils::global_settings.SetValue(kLogLevelSettingKey, FLAGS_log_level);
// Register logging to stderr
auto bool_to_str = [](bool in) { return in ? "true" : "false"; };
const std::string log_to_stderr_s = bool_to_str(FLAGS_also_log_to_stderr);
memgraph::utils::global_settings.RegisterSetting(
kLogToStderrSettingKey, log_to_stderr_s,
[&] {
const auto enable = memgraph::utils::global_settings.GetValue(kLogToStderrSettingKey);
if (enable == "true") {
LogToStderr(get_global_log_level());
} else {
LogToStderr(spdlog::level::off);
}
},
[](std::string_view in) {
const auto lc = memgraph::utils::ToLowerCase(in);
return lc == "false" || lc == "true";
});
// Always override log to stderr with command line argument
memgraph::utils::global_settings.SetValue(kLogToStderrSettingKey, log_to_stderr_s);
}
} // namespace memgraph::flags::run_time

View File

@@ -0,0 +1,26 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "utils/spin_lock.hpp"
#include "utils/synchronized.hpp"
namespace memgraph::flags::run_time {
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
extern utils::Synchronized<std::string, utils::SpinLock> bolt_server_name_;
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
extern std::atomic<double> execution_timeout_sec_;
void Initialize();
} // namespace memgraph::flags::run_time

View File

@@ -1,4 +1,4 @@
set(mg_glue_sources auth.cpp auth_checker.cpp auth_handler.cpp communication.cpp)
add_library(mg-glue STATIC ${mg_glue_sources})
target_link_libraries(mg-glue mg-query mg-auth)
add_library(mg-glue STATIC )
target_sources(mg-glue PRIVATE auth.cpp auth_checker.cpp auth_handler.cpp communication.cpp SessionHL.cpp ServerT.cpp MonitoringServerT.cpp)
target_link_libraries(mg-glue mg-query mg-auth mg-audit)
target_precompile_headers(mg-glue INTERFACE auth_checker.hpp auth_handler.hpp)

View File

@@ -0,0 +1,14 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "glue/MonitoringServerT.hpp"
template class memgraph::communication::http::Server<
memgraph::http::MetricsRequestHandler<memgraph::dbms::SessionContext>, memgraph::dbms::SessionContext>;

View File

@@ -0,0 +1,25 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "communication/http/server.hpp"
#include "dbms/session_context.hpp"
#include "http_handlers/metrics.hpp"
extern template class memgraph::communication::http::Server<
memgraph::http::MetricsRequestHandler<memgraph::dbms::SessionContext>, memgraph::dbms::SessionContext>;
namespace memgraph::glue {
using MonitoringServerT =
memgraph::communication::http::Server<memgraph::http::MetricsRequestHandler<memgraph::dbms::SessionContext>,
memgraph::dbms::SessionContext>;
} // namespace memgraph::glue

17
src/glue/ServerT.cpp Normal file
View File

@@ -0,0 +1,17 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "glue/ServerT.hpp"
#ifdef MG_ENTERPRISE
template class memgraph::communication::v2::Server<memgraph::glue::SessionHL, memgraph::dbms::SessionContextHandler>;
#else
template class memgraph::communication::v2::Server<memgraph::glue::SessionHL, memgraph::dbms::SessionContext>;
#endif

35
src/glue/ServerT.hpp Normal file
View File

@@ -0,0 +1,35 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "communication/v2/server.hpp"
#include "glue/SessionHL.hpp"
#ifdef MG_ENTERPRISE
#include "dbms/session_context_handler.hpp"
#else
#include "dbms/session_context.hpp"
#endif
#ifdef MG_ENTERPRISE
extern template class memgraph::communication::v2::Server<memgraph::glue::SessionHL,
memgraph::dbms::SessionContextHandler>;
#else
extern template class memgraph::communication::v2::Server<memgraph::glue::SessionHL, memgraph::dbms::SessionContext>;
#endif
namespace memgraph::glue {
#ifdef MG_ENTERPRISE
using ServerT = memgraph::communication::v2::Server<memgraph::glue::SessionHL, memgraph::dbms::SessionContextHandler>;
#else
using ServerT = memgraph::communication::v2::Server<memgraph::glue::SessionHL, memgraph::dbms::SessionContext>;
#endif
} // namespace memgraph::glue

386
src/glue/SessionHL.cpp Normal file
View File

@@ -0,0 +1,386 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "glue/SessionHL.hpp"
#include <optional>
#include "audit/log.hpp"
#include "flags/run_time_configurable.hpp"
#include "glue/auth_checker.hpp"
#include "glue/communication.hpp"
#include "license/license.hpp"
#include "query/discard_value_stream.hpp"
#include "utils/spin_lock.hpp"
#include "gflags/gflags.h"
namespace memgraph::metrics {
extern const Event ActiveBoltSessions;
} // namespace memgraph::metrics
auto ToQueryExtras(const memgraph::communication::bolt::Value &extra) -> memgraph::query::QueryExtras {
auto const &as_map = extra.ValueMap();
auto metadata_pv = std::map<std::string, memgraph::storage::PropertyValue>{};
if (auto const it = as_map.find("tx_metadata"); it != as_map.cend() && it->second.IsMap()) {
for (const auto &[key, bolt_md] : it->second.ValueMap()) {
metadata_pv.emplace(key, memgraph::glue::ToPropertyValue(bolt_md));
}
}
auto tx_timeout = std::optional<int64_t>{};
if (auto const it = as_map.find("tx_timeout"); it != as_map.cend() && it->second.IsInt()) {
tx_timeout = it->second.ValueInt();
}
return memgraph::query::QueryExtras{std::move(metadata_pv), tx_timeout};
}
class TypedValueResultStreamBase {
public:
explicit TypedValueResultStreamBase(memgraph::query::InterpreterContext *interpreterContext);
std::vector<memgraph::communication::bolt::Value> DecodeValues(
const std::vector<memgraph::query::TypedValue> &values) const;
private:
// NOTE: Needed only for ToBoltValue conversions
memgraph::query::InterpreterContext *interpreter_context_;
};
/// Wrapper around TEncoder which converts TypedValue to Value
/// before forwarding the calls to original TEncoder.
template <typename TEncoder>
class TypedValueResultStream : public TypedValueResultStreamBase {
public:
TypedValueResultStream(TEncoder *encoder, memgraph::query::InterpreterContext *ic)
: TypedValueResultStreamBase{ic}, encoder_(encoder) {}
void Result(const std::vector<memgraph::query::TypedValue> &values) { encoder_->MessageRecord(DecodeValues(values)); }
private:
TEncoder *encoder_;
};
std::vector<memgraph::communication::bolt::Value> TypedValueResultStreamBase::DecodeValues(
const std::vector<memgraph::query::TypedValue> &values) const {
std::vector<memgraph::communication::bolt::Value> decoded_values;
decoded_values.reserve(values.size());
for (const auto &v : values) {
auto maybe_value = memgraph::glue::ToBoltValue(v, *interpreter_context_->db, memgraph::storage::View::NEW);
if (maybe_value.HasError()) {
switch (maybe_value.GetError()) {
case memgraph::storage::Error::DELETED_OBJECT:
throw memgraph::communication::bolt::ClientError("Returning a deleted object as a result.");
case memgraph::storage::Error::NONEXISTENT_OBJECT:
throw memgraph::communication::bolt::ClientError("Returning a nonexistent object as a result.");
case memgraph::storage::Error::VERTEX_HAS_EDGES:
case memgraph::storage::Error::SERIALIZATION_ERROR:
case memgraph::storage::Error::PROPERTIES_DISABLED:
throw memgraph::communication::bolt::ClientError("Unexpected storage error when streaming results.");
}
}
decoded_values.emplace_back(std::move(*maybe_value));
}
return decoded_values;
}
TypedValueResultStreamBase::TypedValueResultStreamBase(memgraph::query::InterpreterContext *interpreterContext)
: interpreter_context_(interpreterContext) {}
namespace memgraph::glue {
#ifdef MG_ENTERPRISE
void SessionHL::UpdateAndDefunct(const std::string &db_name) {
UpdateAndDefunct(ContextWrapper(sc_handler_.Get(db_name)));
}
void SessionHL::UpdateAndDefunct(ContextWrapper &&cntxt) {
defunct_.emplace(std::move(current_));
Update(std::forward<ContextWrapper>(cntxt));
defunct_->Defunct();
}
void SessionHL::Update(const std::string &db_name) {
ContextWrapper tmp(sc_handler_.Get(db_name));
Update(std::move(tmp));
}
void SessionHL::Update(ContextWrapper &&cntxt) {
current_ = std::move(cntxt);
interpreter_ = current_.interp();
interpreter_->in_explicit_db_ = in_explicit_db_;
interpreter_context_ = current_.interpreter_context();
}
void SessionHL::MultiDatabaseAuth(const std::string &db) {
if (user_ && !AuthChecker::IsUserAuthorized(*user_, {}, db)) {
throw memgraph::communication::bolt::ClientError(
"You are not authorized on the database \"{}\"! Please contact your database administrator.", db);
}
}
std::string SessionHL::GetDefaultDB() {
if (user_.has_value()) {
return user_->db_access().GetDefault();
}
return memgraph::dbms::kDefaultDB;
}
bool SessionHL::OnDelete(const std::string &db_name) {
MG_ASSERT(current_.interpreter_context()->db->id() != db_name && (!defunct_ || defunct_->defunct()),
"Trying to delete a database while still in use.");
return true;
}
memgraph::dbms::SetForResult SessionHL::OnChange(const std::string &db_name) {
MultiDatabaseAuth(db_name);
if (db_name != current_.interpreter_context()->db->id()) {
UpdateAndDefunct(db_name); // Done during Pull, so we cannot just replace the current db
return memgraph::dbms::SetForResult::SUCCESS;
}
return memgraph::dbms::SetForResult::ALREADY_SET;
}
#endif
std::string SessionHL::GetDatabaseName() const { return interpreter_context_->db->id(); }
std::optional<std::string> SessionHL::GetServerNameForInit() {
auto locked_name = flags::run_time::bolt_server_name_.Lock();
return locked_name->empty() ? std::nullopt : std::make_optional(*locked_name);
}
bool SessionHL::Authenticate(const std::string &username, const std::string &password) {
auto locked_auth = auth_->Lock();
if (!locked_auth->HasUsers()) {
return true;
}
user_ = locked_auth->Authenticate(username, password);
#ifdef MG_ENTERPRISE
if (user_.has_value()) {
const auto &db = user_->db_access().GetDefault();
// Check if the underlying database needs to be updated
if (db != current_.interpreter_context()->db->id()) {
const auto &res = sc_handler_.SetFor(UUID(), db);
return res == memgraph::dbms::SetForResult::SUCCESS || res == memgraph::dbms::SetForResult::ALREADY_SET;
}
}
#endif
return user_.has_value();
}
void SessionHL::Abort() { interpreter_->Abort(); }
std::map<std::string, memgraph::communication::bolt::Value> SessionHL::Discard(std::optional<int> n,
std::optional<int> qid) {
try {
memgraph::query::DiscardValueResultStream stream;
return DecodeSummary(interpreter_->Pull(&stream, n, qid));
} catch (const memgraph::query::QueryException &e) {
// Wrap QueryException into ClientError, because we want to allow the
// client to fix their query.
throw memgraph::communication::bolt::ClientError(e.what());
}
}
std::map<std::string, memgraph::communication::bolt::Value> SessionHL::Pull(SessionHL::TEncoder *encoder,
std::optional<int> n,
std::optional<int> qid) {
try {
TypedValueResultStream<TEncoder> stream(encoder, interpreter_context_);
return DecodeSummary(interpreter_->Pull(&stream, n, qid));
} catch (const memgraph::query::QueryException &e) {
// Wrap QueryException into ClientError, because we want to allow the
// client to fix their query.
throw memgraph::communication::bolt::ClientError(e.what());
}
}
std::pair<std::vector<std::string>, std::optional<int>> SessionHL::Interpret(
const std::string &query, const std::map<std::string, memgraph::communication::bolt::Value> &params,
const std::map<std::string, memgraph::communication::bolt::Value> &extra) {
std::map<std::string, memgraph::storage::PropertyValue> params_pv;
for (const auto &[key, bolt_param] : params) {
params_pv.emplace(key, ToPropertyValue(bolt_param));
}
const std::string *username{nullptr};
if (user_) {
username = &user_->username();
}
#ifdef MG_ENTERPRISE
if (memgraph::license::global_license_checker.IsEnterpriseValidFast()) {
audit_log_->Record(endpoint_.address().to_string(), user_ ? *username : "", query,
memgraph::storage::PropertyValue(params_pv), interpreter_context_->db->id());
}
#endif
try {
auto result = interpreter_->Prepare(query, params_pv, username, ToQueryExtras(extra), UUID());
const std::string db_name = result.db ? *result.db : "";
if (user_ && !AuthChecker::IsUserAuthorized(*user_, result.privileges, db_name)) {
interpreter_->Abort();
if (db_name.empty()) {
throw memgraph::communication::bolt::ClientError(
"You are not authorized to execute this query! Please contact your database administrator.");
}
throw memgraph::communication::bolt::ClientError(
"You are not authorized to execute this query on database \"{}\"! Please contact your database "
"administrator.",
db_name);
}
return {std::move(result.headers), result.qid};
} catch (const memgraph::query::QueryException &e) {
// Wrap QueryException into ClientError, because we want to allow the
// client to fix their query.
throw memgraph::communication::bolt::ClientError(e.what());
} catch (const memgraph::query::ReplicationException &e) {
throw memgraph::communication::bolt::ClientError(e.what());
}
}
void SessionHL::RollbackTransaction() { interpreter_->RollbackTransaction(); }
void SessionHL::CommitTransaction() { interpreter_->CommitTransaction(); }
void SessionHL::BeginTransaction(const std::map<std::string, memgraph::communication::bolt::Value> &extra) {
interpreter_->BeginTransaction(ToQueryExtras(extra));
}
void SessionHL::Configure(const std::map<std::string, memgraph::communication::bolt::Value> &run_time_info) {
#ifdef MG_ENTERPRISE
std::string db;
bool update = false;
// Check if user explicitly defined the database to use
if (run_time_info.contains("db")) {
const auto &db_info = run_time_info.at("db");
if (!db_info.IsString()) {
throw memgraph::communication::bolt::ClientError("Malformed database name.");
}
db = db_info.ValueString();
update = db != current_.interpreter_context()->db->id();
in_explicit_db_ = true;
// NOTE: Once in a transaction, the drivers stop explicitly sending the db and count on using it until commit
} else if (in_explicit_db_ && !interpreter_->in_explicit_transaction_) { // Just on a switch
db = GetDefaultDB();
update = db != current_.interpreter_context()->db->id();
in_explicit_db_ = false;
}
// Check if the underlying database needs to be updated
if (update) {
sc_handler_.SetInPlace(db, [this](auto new_sc) mutable {
const auto &db_name = new_sc.interpreter_context->db->id();
MultiDatabaseAuth(db_name);
try {
Update(ContextWrapper(new_sc));
return memgraph::dbms::SetForResult::SUCCESS;
} catch (memgraph::dbms::UnknownDatabaseException &e) {
throw memgraph::communication::bolt::ClientError("No database named \"{}\" found!", db_name);
}
});
}
#endif
}
SessionHL::~SessionHL() { memgraph::metrics::DecrementCounter(memgraph::metrics::ActiveBoltSessions); }
SessionHL::SessionHL(
#ifdef MG_ENTERPRISE
memgraph::dbms::SessionContextHandler &sc_handler,
#else
memgraph::dbms::SessionContext sc,
#endif
const memgraph::communication::v2::ServerEndpoint &endpoint, memgraph::communication::v2::InputStream *input_stream,
memgraph::communication::v2::OutputStream *output_stream, const std::string &default_db) // NOLINT
: Session<memgraph::communication::v2::InputStream, memgraph::communication::v2::OutputStream>(input_stream,
output_stream),
#ifdef MG_ENTERPRISE
sc_handler_(sc_handler),
current_(sc_handler_.Get(default_db)),
#else
current_(sc),
#endif
interpreter_context_(current_.interpreter_context()),
interpreter_(current_.interp()),
auth_(current_.auth()),
#ifdef MG_ENTERPRISE
audit_log_(current_.audit_log()),
#endif
endpoint_(endpoint),
run_id_(current_.run_id()) {
// Metrics update
memgraph::metrics::IncrementCounter(memgraph::metrics::ActiveBoltSessions);
}
/// ContextWrapper
ContextWrapper::ContextWrapper(memgraph::dbms::SessionContext sc)
: session_context(sc),
interpreter(std::make_unique<memgraph::query::Interpreter>(session_context.interpreter_context.get())),
defunct_(false) {
session_context.interpreter_context->interpreters.WithLock(
[this](auto &interpreters) { interpreters.insert(interpreter.get()); });
}
ContextWrapper::~ContextWrapper() { Defunct(); }
void ContextWrapper::Defunct() {
if (!defunct_) {
session_context.interpreter_context->interpreters.WithLock(
[this](auto &interpreters) { interpreters.erase(interpreter.get()); });
defunct_ = true;
}
}
ContextWrapper::ContextWrapper(ContextWrapper &&in) noexcept
: session_context(std::move(in.session_context)), interpreter(std::move(in.interpreter)), defunct_(in.defunct_) {
in.defunct_ = true;
}
ContextWrapper &ContextWrapper::operator=(ContextWrapper &&in) noexcept {
if (this != &in) {
Defunct();
session_context = std::move(in.session_context);
interpreter = std::move(in.interpreter);
defunct_ = in.defunct_;
in.defunct_ = true;
}
return *this;
}
memgraph::query::InterpreterContext *ContextWrapper::interpreter_context() {
return session_context.interpreter_context.get();
}
memgraph::query::Interpreter *ContextWrapper::interp() { return interpreter.get(); }
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *ContextWrapper::auth()
const {
return session_context.auth;
}
std::string ContextWrapper::run_id() const { return session_context.run_id; }
bool ContextWrapper::defunct() const { return defunct_; }
#ifdef MG_ENTERPRISE
memgraph::audit::Log *ContextWrapper::audit_log() const { return session_context.audit_log; }
#endif
std::map<std::string, memgraph::communication::bolt::Value> SessionHL::DecodeSummary(
const std::map<std::string, memgraph::query::TypedValue> &summary) {
std::map<std::string, memgraph::communication::bolt::Value> decoded_summary;
for (const auto &kv : summary) {
auto maybe_value = ToBoltValue(kv.second, *interpreter_context_->db, memgraph::storage::View::NEW);
if (maybe_value.HasError()) {
switch (maybe_value.GetError()) {
case memgraph::storage::Error::DELETED_OBJECT:
case memgraph::storage::Error::SERIALIZATION_ERROR:
case memgraph::storage::Error::VERTEX_HAS_EDGES:
case memgraph::storage::Error::PROPERTIES_DISABLED:
case memgraph::storage::Error::NONEXISTENT_OBJECT:
throw memgraph::communication::bolt::ClientError("Unexpected storage error when streaming summary.");
}
}
decoded_summary.emplace(kv.first, std::move(*maybe_value));
}
// Add this memgraph instance run_id, received from telemetry
// This is sent with every query, instead of only on bolt init inside
// communication/bolt/v1/states/init.hpp because neo4jdriver does not
// read the init message.
if (auto run_id = run_id_; run_id) {
decoded_summary.emplace("run_id", *run_id);
}
// Clean up previous session (session gets defunct when switching between databases)
if (defunct_) {
defunct_.reset();
}
return decoded_summary;
}
} // namespace memgraph::glue

161
src/glue/SessionHL.hpp Normal file
View File

@@ -0,0 +1,161 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "communication/v2/server.hpp"
#include "communication/v2/session.hpp"
#include "dbms/session_context.hpp"
#ifdef MG_ENTERPRISE
#include "dbms/session_context_handler.hpp"
#else
#include "dbms/session_context.hpp"
#endif
namespace memgraph::glue {
struct ContextWrapper {
explicit ContextWrapper(memgraph::dbms::SessionContext sc);
~ContextWrapper();
ContextWrapper(const ContextWrapper &) = delete;
ContextWrapper &operator=(const ContextWrapper &) = delete;
ContextWrapper(ContextWrapper &&in) noexcept;
ContextWrapper &operator=(ContextWrapper &&in) noexcept;
void Defunct();
memgraph::query::InterpreterContext *interpreter_context();
memgraph::query::Interpreter *interp();
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth() const;
std::string run_id() const;
bool defunct() const;
#ifdef MG_ENTERPRISE
memgraph::audit::Log *audit_log() const;
#endif
private:
memgraph::dbms::SessionContext session_context;
std::unique_ptr<memgraph::query::Interpreter> interpreter;
bool defunct_;
};
class SessionHL final : public memgraph::communication::bolt::Session<memgraph::communication::v2::InputStream,
memgraph::communication::v2::OutputStream> {
public:
SessionHL(
#ifdef MG_ENTERPRISE
memgraph::dbms::SessionContextHandler &sc_handler,
#else
memgraph::dbms::SessionContext sc,
#endif
const memgraph::communication::v2::ServerEndpoint &endpoint,
memgraph::communication::v2::InputStream *input_stream, memgraph::communication::v2::OutputStream *output_stream,
const std::string &default_db = memgraph::dbms::kDefaultDB);
~SessionHL() override;
SessionHL(const SessionHL &) = delete;
SessionHL &operator=(const SessionHL &) = delete;
SessionHL(SessionHL &&) = delete;
SessionHL &operator=(SessionHL &&) = delete;
void Configure(const std::map<std::string, memgraph::communication::bolt::Value> &run_time_info) override;
using TEncoder = memgraph::communication::bolt::Encoder<
memgraph::communication::bolt::ChunkedEncoderBuffer<memgraph::communication::v2::OutputStream>>;
void BeginTransaction(const std::map<std::string, memgraph::communication::bolt::Value> &extra) override;
void CommitTransaction() override;
void RollbackTransaction() override;
std::pair<std::vector<std::string>, std::optional<int>> Interpret(
const std::string &query, const std::map<std::string, memgraph::communication::bolt::Value> &params,
const std::map<std::string, memgraph::communication::bolt::Value> &extra) override;
std::map<std::string, memgraph::communication::bolt::Value> Pull(TEncoder *encoder, std::optional<int> n,
std::optional<int> qid) override;
std::map<std::string, memgraph::communication::bolt::Value> Discard(std::optional<int> n,
std::optional<int> qid) override;
void Abort() override;
// Called during Init
// During Init, the user cannot choose the landing DB (switch is done during query execution)
bool Authenticate(const std::string &username, const std::string &password) override;
#ifdef MG_ENTERPRISE
memgraph::dbms::SetForResult OnChange(const std::string &db_name) override;
bool OnDelete(const std::string &db_name) override;
#endif
std::optional<std::string> GetServerNameForInit() override;
std::string GetDatabaseName() const override;
private:
std::map<std::string, memgraph::communication::bolt::Value> DecodeSummary(
const std::map<std::string, memgraph::query::TypedValue> &summary);
#ifdef MG_ENTERPRISE
/**
* @brief Update setup to the new database.
*
* @param db_name name of the target database
* @throws UnknownDatabaseException if handler cannot get it
*/
void UpdateAndDefunct(const std::string &db_name);
void UpdateAndDefunct(ContextWrapper &&cntxt);
void Update(const std::string &db_name);
void Update(ContextWrapper &&cntxt);
/**
* @brief Authenticate user on passed database.
*
* @param db database to check against
* @throws bolt::ClientError when user is not authorized
*/
void MultiDatabaseAuth(const std::string &db);
/**
* @brief Get the user's default database
*
* @return std::string
*/
std::string GetDefaultDB();
#endif
#ifdef MG_ENTERPRISE
memgraph::dbms::SessionContextHandler &sc_handler_;
#endif
ContextWrapper current_;
std::optional<ContextWrapper> defunct_;
memgraph::query::InterpreterContext *interpreter_context_;
memgraph::query::Interpreter *interpreter_;
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth_;
std::optional<memgraph::auth::User> user_;
#ifdef MG_ENTERPRISE
memgraph::audit::Log *audit_log_;
bool in_explicit_db_{false}; //!< If true, the user has defined the database to use via metadata
#endif
memgraph::communication::v2::ServerEndpoint endpoint_;
// NOTE: run_id should be const but that complicates code a lot.
std::optional<std::string> run_id_;
};
} // namespace memgraph::glue

View File

@@ -94,20 +94,24 @@ std::unique_ptr<memgraph::query::FineGrainedAuthChecker> AuthChecker::GetFineGra
return {};
}
try {
auto locked_auth = auth_->Lock();
if (username != user_.username()) {
auto maybe_user = locked_auth->GetUser(username);
auto user = user_.Lock();
if (username != user->username()) {
auto maybe_user = auth_->ReadLock()->GetUser(username);
if (!maybe_user) {
throw memgraph::query::QueryRuntimeException("User '{}' doesn't exist .", username);
}
user_ = std::move(*maybe_user);
*user = std::move(*maybe_user);
}
return std::make_unique<memgraph::glue::FineGrainedAuthChecker>(user_, dba);
return std::make_unique<memgraph::glue::FineGrainedAuthChecker>(*user, dba);
} catch (const memgraph::auth::AuthException &e) {
throw memgraph::query::QueryRuntimeException(e.what());
}
}
void AuthChecker::ClearCache() const {
user_.WithLock([](auto &user) mutable { user = {}; });
}
#endif
bool AuthChecker::IsUserAuthorized(const memgraph::auth::User &user,

View File

@@ -16,6 +16,7 @@
#include "query/auth_checker.hpp"
#include "query/db_accessor.hpp"
#include "query/frontend/ast/ast.hpp"
#include "utils/spin_lock.hpp"
namespace memgraph::glue {
@@ -32,6 +33,8 @@ class AuthChecker : public query::AuthChecker {
std::unique_ptr<memgraph::query::FineGrainedAuthChecker> GetFineGrainedAuthChecker(
const std::string &username, const memgraph::query::DbAccessor *dba) const override;
void ClearCache() const override;
#endif
[[nodiscard]] static bool IsUserAuthorized(const memgraph::auth::User &user,
const std::vector<memgraph::query::AuthQuery::Privilege> &privileges,
@@ -39,7 +42,7 @@ class AuthChecker : public query::AuthChecker {
private:
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth_;
mutable auth::User user_;
mutable memgraph::utils::Synchronized<auth::User, memgraph::utils::SpinLock> user_; // cached user
};
#ifdef MG_ENTERPRISE
class FineGrainedAuthChecker : public query::FineGrainedAuthChecker {

16
src/glue/auth_global.hpp Normal file
View File

@@ -0,0 +1,16 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
namespace memgraph::glue {
inline constexpr std::string_view kDefaultUserRoleRegex = "[a-zA-Z0-9_.+-@]+";
} // namespace memgraph::glue

View File

@@ -14,6 +14,7 @@
#include <regex>
#include "auth/auth.hpp"
#include "auth_global.hpp"
#include "glue/auth.hpp"
#include "license/license.hpp"
#include "query/interpreter.hpp"
@@ -21,8 +22,6 @@
namespace memgraph::glue {
inline constexpr std::string_view kDefaultUserRoleRegex = "[a-zA-Z0-9_.+-@]+";
class AuthQueryHandler final : public memgraph::query::AuthQueryHandler {
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth_;
std::string name_regex_string_;

View File

@@ -151,8 +151,8 @@ storage::Result<communication::bolt::Vertex> ToBoltVertex(const storage::VertexA
properties[db.PropertyToName(prop.first)] = ToBoltValue(prop.second);
}
// Introduced in Bolt v5 (for now just send the ID)
const auto element_id = std::to_string(id.AsInt());
return communication::bolt::Vertex{id, labels, properties, element_id};
auto element_id = std::to_string(id.AsInt());
return communication::bolt::Vertex{id, std::move(labels), std::move(properties), std::move(element_id)};
}
storage::Result<communication::bolt::Edge> ToBoltEdge(const storage::EdgeAccessor &edge, const storage::Storage &db,
@@ -171,7 +171,8 @@ storage::Result<communication::bolt::Edge> ToBoltEdge(const storage::EdgeAccesso
const auto element_id = std::to_string(id.AsInt());
const auto from_element_id = std::to_string(from.AsInt());
const auto to_element_id = std::to_string(to.AsInt());
return communication::bolt::Edge{id, from, to, type, properties, element_id, from_element_id, to_element_id};
return communication::bolt::Edge{
id, from, to, std::move(type), std::move(properties), element_id, from_element_id, to_element_id};
}
storage::Result<communication::bolt::Path> ToBoltPath(const query::Path &path, const storage::Storage &db,

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -63,7 +63,7 @@ inline void LoadConfig(const std::string &product_name) {
delete[] custom_argv;
}
std::pair<std::string, std::string> LoadUsernameAndPassword(const std::string &pass_file) {
inline std::pair<std::string, std::string> LoadUsernameAndPassword(const std::string &pass_file) {
std::ifstream file(pass_file);
if (file.fail()) {
spdlog::warn("Problem with opening MG_PASSFILE, memgraph server will start without user");

View File

@@ -101,6 +101,7 @@ class MetricsService {
auto GetEventCounters() {
// NOLINTNEXTLINE(cppcoreguidelines-init-variables)
std::vector<std::tuple<std::string, std::string, uint64_t>> event_counters{};
event_counters.reserve(memgraph::metrics::CounterEnd());
for (auto i = 0; i < memgraph::metrics::CounterEnd(); i++) {
event_counters.emplace_back(memgraph::metrics::GetCounterName(i), memgraph::metrics::GetCounterType(i),
@@ -113,6 +114,7 @@ class MetricsService {
auto GetEventGauges() {
// NOLINTNEXTLINE(cppcoreguidelines-init-variables)
std::vector<std::tuple<std::string, std::string, uint64_t>> event_gauges{};
event_gauges.reserve(memgraph::metrics::GaugeEnd());
for (auto i = 0; i < memgraph::metrics::GaugeEnd(); i++) {
event_gauges.emplace_back(memgraph::metrics::GetGaugeName(i), memgraph::metrics::GetGaugeType(i),

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -91,7 +91,6 @@ std::string Endpoint::SocketAddress() const {
return ip_address + ":" + std::to_string(port);
}
Endpoint::Endpoint() {}
Endpoint::Endpoint(std::string ip_address, uint16_t port) : address(std::move(ip_address)), port(port) {
IpFamily ip_family = GetIpFamily(address);
if (ip_family == IpFamily::NONE) {

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -25,8 +25,13 @@ namespace memgraph::io::network {
* connection address.
*/
struct Endpoint {
Endpoint();
Endpoint() = default;
Endpoint(std::string ip_address, uint16_t port);
Endpoint(Endpoint const &) = default;
Endpoint(Endpoint &&) noexcept = default;
Endpoint &operator=(Endpoint const &) = default;
Endpoint &operator=(Endpoint &&) noexcept = default;
~Endpoint() = default;
enum class IpFamily : std::uint8_t { NONE, IP4, IP6 };

View File

@@ -72,7 +72,7 @@ std::optional<std::string> KVStore::Get(const std::string &key) const noexcept {
return value;
}
bool KVStore::Delete(const std::string &key) {
bool KVStore::Delete(std::string_view key) {
auto s = pimpl_->db->Delete(rocksdb::WriteOptions(), key);
return s.ok();
}

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -91,7 +91,7 @@ class KVStore final {
* true if the key doesn't exist and underlying storage
* didn't encounter any error.
*/
bool Delete(const std::string &key);
bool Delete(std::string_view key);
/**
* Deletes the keys and corresponding values from storage.

View File

@@ -9,465 +9,37 @@
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <algorithm>
#include <atomic>
#include <chrono>
#include <csignal>
#include <cstdint>
#include <exception>
#include <filesystem>
#include <functional>
#include <limits>
#include <map>
#include <memory>
#include <optional>
#include <regex>
#include <string>
#include <string_view>
#include <thread>
#include <fmt/core.h>
#include <fmt/format.h>
#include <gflags/gflags.h>
#include <spdlog/common.h>
#include <spdlog/sinks/daily_file_sink.h>
#include <spdlog/sinks/dist_sink.h>
#include <spdlog/sinks/stdout_color_sinks.h>
#include "flags/run_time_configurable.hpp"
#ifndef MG_ENTERPRISE
#include "dbms/session_context_handler.hpp"
#endif
#include "audit/log.hpp"
#include "auth/models.hpp"
#include "communication/bolt/v1/constants.hpp"
#include "communication/http/server.hpp"
#include "communication/websocket/auth.hpp"
#include "communication/websocket/server.hpp"
#include "dbms/constants.hpp"
#include "dbms/global.hpp"
#include "dbms/session_context.hpp"
#include "flags/all.hpp"
#include "glue/MonitoringServerT.hpp"
#include "glue/ServerT.hpp"
#include "glue/auth_checker.hpp"
#include "glue/auth_handler.hpp"
#include "helpers.hpp"
#include "http_handlers/metrics.hpp"
#include "license/license.hpp"
#include "license/license_sender.hpp"
#include "py/py.hpp"
#include "query/auth_checker.hpp"
#include "query/discard_value_stream.hpp"
#include "query/exceptions.hpp"
#include "query/frontend/ast/ast.hpp"
#include "query/interpreter.hpp"
#include "query/plan/operator.hpp"
#include "query/procedure/callable_alias_mapper.hpp"
#include "query/procedure/module.hpp"
#include "query/procedure/py_module.hpp"
#include "requests/requests.hpp"
#include "storage/v2/config.hpp"
#include "storage/v2/disk/storage.hpp"
#include "storage/v2/inmemory/storage.hpp"
#include "storage/v2/isolation_level.hpp"
#include "storage/v2/storage.hpp"
#include "storage/v2/view.hpp"
#include "telemetry/telemetry.hpp"
#include "utils/enum.hpp"
#include "utils/event_counter.hpp"
#include "utils/file.hpp"
#include "utils/flag_validation.hpp"
#include "utils/logging.hpp"
#include "utils/memory_tracker.hpp"
#include "utils/message.hpp"
#include "utils/readable_size.hpp"
#include "utils/rw_lock.hpp"
#include "utils/settings.hpp"
#include "utils/signals.hpp"
#include "utils/string.hpp"
#include "utils/synchronized.hpp"
#include "utils/sysinfo/memory.hpp"
#include "utils/system_info.hpp"
#include "utils/terminate_handler.hpp"
#include "version.hpp"
// Communication libraries must be included after query libraries are included.
// This is to enable compilation of the binary when linking with old OpenSSL
// libraries (as on CentOS 7).
//
// The OpenSSL library available on CentOS 7 is v1.0.0, that version includes
// `libkrb5` in its public API headers (that we include in our communication
// stack). The `libkrb5` library has `#define`s for `TRUE` and `FALSE`. Those
// defines clash with Antlr's usage of `TRUE` and `FALSE` as enumeration keys.
// Because of that the definitions of `TRUE` and `FALSE` that are inherited
// from `libkrb5` must be included after the Antlr includes. Hence,
// communication headers must be included after query headers.
#include "communication/bolt/v1/exceptions.hpp"
#include "communication/bolt/v1/session.hpp"
#include "communication/init.hpp"
#include "communication/v2/server.hpp"
#include "communication/v2/session.hpp"
#include "dbms/session_context_handler.hpp"
#include "glue/communication.hpp"
#include "auth/auth.hpp"
#include "glue/auth.hpp"
constexpr const char *kMgUser = "MEMGRAPH_USER";
constexpr const char *kMgPassword = "MEMGRAPH_PASSWORD";
constexpr const char *kMgPassfile = "MEMGRAPH_PASSFILE";
// Short help flag.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_bool(h, false, "Print usage and exit.");
// Bolt server flags.
DEFINE_string(bolt_address, "0.0.0.0", "IP address on which the Bolt server should listen.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(monitoring_address, "0.0.0.0",
"IP address on which the websocket server for Memgraph monitoring should listen.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(metrics_address, "0.0.0.0",
"IP address on which the Memgraph server for exposing metrics should listen.");
DEFINE_VALIDATED_int32(bolt_port, 7687, "Port on which the Bolt server should listen.",
FLAG_IN_RANGE(0, std::numeric_limits<uint16_t>::max()));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(monitoring_port, 7444,
"Port on which the websocket server for Memgraph monitoring should listen.",
FLAG_IN_RANGE(0, std::numeric_limits<uint16_t>::max()));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(metrics_port, 9091, "Port on which the Memgraph server for exposing metrics should listen.",
FLAG_IN_RANGE(0, std::numeric_limits<uint16_t>::max()));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(bolt_num_workers, std::max(std::thread::hardware_concurrency(), 1U),
"Number of workers used by the Bolt server. By default, this will be the "
"number of processing units available on the machine.",
FLAG_IN_RANGE(1, INT32_MAX));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(bolt_session_inactivity_timeout, 1800,
"Time in seconds after which inactive Bolt sessions will be "
"closed.",
FLAG_IN_RANGE(1, INT32_MAX));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(bolt_cert_file, "", "Certificate file which should be used for the Bolt server.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(bolt_key_file, "", "Key file which should be used for the Bolt server.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(bolt_server_name_for_init, "",
"Server name which the database should send to the client in the "
"Bolt INIT message.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(init_file, "",
"Path to cypherl file that is used for configuring users and database schema before server starts.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(init_data_file, "", "Path to cypherl file that is used for creating data after server starts.");
// General purpose flags.
// NOTE: The `data_directory` flag must be the same here and in
// `mg_import_csv`. If you change it, make sure to change it there as well.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(data_directory, "mg_data", "Path to directory in which to save all permanent data.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(data_recovery_on_startup, false, "Controls whether the database recovers persisted data on startup.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(memory_warning_threshold, 1024,
"Memory warning threshold, in MB. If Memgraph detects there is "
"less available RAM it will log a warning. Set to 0 to "
"disable.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(allow_load_csv, true, "Controls whether LOAD CSV clause is allowed in queries.");
// Storage flags.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_gc_cycle_sec, 30, "Storage garbage collector interval (in seconds).",
FLAG_IN_RANGE(1, 24 * 3600));
// NOTE: The `storage_properties_on_edges` flag must be the same here and in
// `mg_import_csv`. If you change it, make sure to change it there as well.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_properties_on_edges, false, "Controls whether edges have properties.");
// storage_recover_on_startup deprecated; use data_recovery_on_startup instead
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_bool(storage_recover_on_startup, false,
"Controls whether the storage recovers persisted data on startup.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_snapshot_interval_sec, 0,
"Storage snapshot creation interval (in seconds). Set "
"to 0 to disable periodic snapshot creation.",
FLAG_IN_RANGE(0, 7 * 24 * 3600));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_wal_enabled, false,
"Controls whether the storage uses write-ahead-logging. To enable "
"WAL periodic snapshots must be enabled.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_snapshot_retention_count, 3, "The number of snapshots that should always be kept.",
FLAG_IN_RANGE(1, 1000000));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_wal_file_size_kib, memgraph::storage::Config::Durability().wal_file_size_kibibytes,
"Minimum file size of each WAL file.",
FLAG_IN_RANGE(1, static_cast<unsigned long>(1000) * 1024));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_uint64(storage_wal_file_flush_every_n_tx,
memgraph::storage::Config::Durability().wal_file_flush_every_n_tx,
"Issue a 'fsync' call after this amount of transactions are written to the "
"WAL file. Set to 1 for fully synchronous operation.",
FLAG_IN_RANGE(1, 1000000));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_snapshot_on_exit, false, "Controls whether the storage creates another snapshot on exit.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(storage_items_per_batch, memgraph::storage::Config::Durability().items_per_batch,
"The number of edges and vertices stored in a batch in a snapshot file.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_parallel_index_recovery, false,
"Controls whether the index creation can be done in a multithreaded fashion.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(storage_recovery_thread_count,
std::max(static_cast<uint64_t>(std::thread::hardware_concurrency()),
memgraph::storage::Config::Durability().recovery_thread_count),
"The number of threads used to recover persisted data from disk.");
#ifdef MG_ENTERPRISE
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_delete_on_drop, true,
"If set to true the query 'DROP DATABASE x' will delete the underlying storage as well.");
#endif
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(telemetry_enabled, false,
"Set to true to enable telemetry. We collect information about the "
"running system (CPU and memory information) and information about "
"the database runtime (vertex and edge counts and resource usage) "
"to allow for easier improvement of the product.");
// Streams flags
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint32(
stream_transaction_conflict_retries, 30,
"Number of times to retry when a stream transformation fails to commit because of conflicting transactions");
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint32(
stream_transaction_retry_interval, 500,
"Retry interval in milliseconds when a stream transformation fails to commit because of conflicting transactions");
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(kafka_bootstrap_servers, "",
"List of default Kafka brokers as a comma separated list of broker host or host:port.");
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(pulsar_service_url, "", "Default URL used while connecting to Pulsar brokers.");
// Audit logging flags.
#ifdef MG_ENTERPRISE
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(audit_enabled, false, "Set to true to enable audit logging.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(audit_buffer_size, memgraph::audit::kBufferSizeDefault,
"Maximum number of items in the audit log buffer.", FLAG_IN_RANGE(1, INT32_MAX));
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_int32(audit_buffer_flush_interval_ms, memgraph::audit::kBufferFlushIntervalMillisDefault,
"Interval (in milliseconds) used for flushing the audit log buffer.",
FLAG_IN_RANGE(10, INT32_MAX));
#endif
// Query flags.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_double(query_execution_timeout_sec, 600,
"Maximum allowed query execution time. Queries exceeding this "
"limit will be aborted. Value of 0 means no limit.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(replication_replica_check_frequency_sec, 1,
"The time duration between two replica checks/pings. If < 1, replicas will NOT be checked at all. NOTE: "
"The MAIN instance allocates a new thread for each REPLICA.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(replication_restore_state_on_startup, false, "Restore replication state on startup, e.g. recover replica");
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(
memory_limit, 0,
"Total memory limit in MiB. Set to 0 to use the default values which are 100\% of the phyisical memory if the swap "
"is enabled and 90\% of the physical memory otherwise.");
namespace {
using namespace std::literals;
inline constexpr std::array isolation_level_mappings{
std::pair{"SNAPSHOT_ISOLATION"sv, memgraph::storage::IsolationLevel::SNAPSHOT_ISOLATION},
std::pair{"READ_COMMITTED"sv, memgraph::storage::IsolationLevel::READ_COMMITTED},
std::pair{"READ_UNCOMMITTED"sv, memgraph::storage::IsolationLevel::READ_UNCOMMITTED}};
const std::string isolation_level_help_string =
fmt::format("Default isolation level used for the transactions. Allowed values: {}",
memgraph::utils::GetAllowedEnumValuesString(isolation_level_mappings));
} // namespace
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_string(isolation_level, "SNAPSHOT_ISOLATION", isolation_level_help_string.c_str(), {
if (const auto result = memgraph::utils::IsValidEnumValueString(value, isolation_level_mappings); result.HasError()) {
const auto error = result.GetError();
switch (error) {
case memgraph::utils::ValidationError::EmptyValue: {
std::cout << "Isolation level cannot be empty." << std::endl;
break;
}
case memgraph::utils::ValidationError::InvalidValue: {
std::cout << "Invalid value for isolation level. Allowed values: "
<< memgraph::utils::GetAllowedEnumValuesString(isolation_level_mappings) << std::endl;
break;
}
}
return false;
}
return true;
});
namespace {
memgraph::storage::IsolationLevel ParseIsolationLevel() {
const auto isolation_level =
memgraph::utils::StringToEnum<memgraph::storage::IsolationLevel>(FLAGS_isolation_level, isolation_level_mappings);
MG_ASSERT(isolation_level, "Invalid isolation level");
return *isolation_level;
}
int64_t GetMemoryLimit() {
if (FLAGS_memory_limit == 0) {
auto maybe_total_memory = memgraph::utils::sysinfo::TotalMemory();
MG_ASSERT(maybe_total_memory, "Failed to fetch the total physical memory");
const auto maybe_swap_memory = memgraph::utils::sysinfo::SwapTotalMemory();
MG_ASSERT(maybe_swap_memory, "Failed to fetch the total swap memory");
if (*maybe_swap_memory == 0) {
// take only 90% of the total memory
*maybe_total_memory *= 9;
*maybe_total_memory /= 10;
}
return *maybe_total_memory * 1024;
}
// We parse the memory as MiB every time
return FLAGS_memory_limit * 1024 * 1024;
}
} // namespace
namespace {
std::vector<std::filesystem::path> query_modules_directories;
} // namespace
DEFINE_VALIDATED_string(query_modules_directory, "",
"Directory where modules with custom query procedures are stored. "
"NOTE: Multiple comma-separated directories can be defined.",
{
query_modules_directories.clear();
if (value.empty()) return true;
const auto directories = memgraph::utils::Split(value, ",");
for (const auto &dir : directories) {
if (!memgraph::utils::DirExists(dir)) {
std::cout << "Expected --" << flagname << " to point to directories." << std::endl;
std::cout << dir << " is not a directory." << std::endl;
return false;
}
}
query_modules_directories.reserve(directories.size());
std::transform(directories.begin(), directories.end(),
std::back_inserter(query_modules_directories),
[](const auto &dir) { return dir; });
return true;
});
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(query_callable_mappings_path, "",
"The path to mappings that describes aliases to callables in cypher queries in the form of key-value "
"pairs in a json file. With this option query module procedures that do not exist in memgraph can be "
"mapped to ones that exist.");
// Logging flags
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_bool(also_log_to_stderr, false, "Log messages go to stderr in addition to logfiles");
DEFINE_string(log_file, "", "Path to where the log should be stored.");
namespace {
inline constexpr std::array log_level_mappings{
std::pair{"TRACE"sv, spdlog::level::trace}, std::pair{"DEBUG"sv, spdlog::level::debug},
std::pair{"INFO"sv, spdlog::level::info}, std::pair{"WARNING"sv, spdlog::level::warn},
std::pair{"ERROR"sv, spdlog::level::err}, std::pair{"CRITICAL"sv, spdlog::level::critical}};
const std::string log_level_help_string = fmt::format("Minimum log level. Allowed values: {}",
memgraph::utils::GetAllowedEnumValuesString(log_level_mappings));
} // namespace
DEFINE_VALIDATED_string(log_level, "WARNING", log_level_help_string.c_str(), {
if (const auto result = memgraph::utils::IsValidEnumValueString(value, log_level_mappings); result.HasError()) {
const auto error = result.GetError();
switch (error) {
case memgraph::utils::ValidationError::EmptyValue: {
std::cout << "Log level cannot be empty." << std::endl;
break;
}
case memgraph::utils::ValidationError::InvalidValue: {
std::cout << "Invalid value for log level. Allowed values: "
<< memgraph::utils::GetAllowedEnumValuesString(log_level_mappings) << std::endl;
break;
}
}
return false;
}
return true;
});
namespace {
spdlog::level::level_enum ParseLogLevel() {
const auto log_level = memgraph::utils::StringToEnum<spdlog::level::level_enum>(FLAGS_log_level, log_level_mappings);
MG_ASSERT(log_level, "Invalid log level");
return *log_level;
}
// 5 weeks * 7 days
inline constexpr auto log_retention_count = 35;
void CreateLoggerFromSink(const auto &sinks, const auto log_level) {
auto logger = std::make_shared<spdlog::logger>("memgraph_log", sinks.begin(), sinks.end());
logger->set_level(log_level);
logger->flush_on(spdlog::level::trace);
spdlog::set_default_logger(std::move(logger));
}
void InitializeLogger() {
std::vector<spdlog::sink_ptr> sinks;
if (FLAGS_also_log_to_stderr) {
sinks.emplace_back(std::make_shared<spdlog::sinks::stderr_color_sink_mt>());
}
if (!FLAGS_log_file.empty()) {
// get local time
time_t current_time{0};
struct tm *local_time{nullptr};
time(&current_time);
local_time = localtime(&current_time);
sinks.emplace_back(std::make_shared<spdlog::sinks::daily_file_sink_mt>(
FLAGS_log_file, local_time->tm_hour, local_time->tm_min, false, log_retention_count));
}
CreateLoggerFromSink(sinks, ParseLogLevel());
}
void AddLoggerSink(spdlog::sink_ptr new_sink) {
auto default_logger = spdlog::default_logger();
auto sinks = default_logger->sinks();
sinks.push_back(new_sink);
CreateLoggerFromSink(sinks, default_logger->level());
}
} // namespace
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_string(license_key, "", "License key for Memgraph Enterprise.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_string(organization_name, "", "Organization name.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_string(auth_user_or_role_name_regex, memgraph::glue::kDefaultUserRoleRegex.data(),
"Set to the regular expression that each user or role name must fulfill.");
void InitFromCypherlFile(memgraph::query::InterpreterContext &ctx, std::string cypherl_file_path,
memgraph::audit::Log *audit_log = nullptr) {
memgraph::query::Interpreter interpreter(&ctx);
@@ -494,430 +66,6 @@ void InitFromCypherlFile(memgraph::query::InterpreterContext &ctx, std::string c
file.close();
}
namespace memgraph::metrics {
extern const Event ActiveBoltSessions;
} // namespace memgraph::metrics
auto ToQueryExtras(memgraph::communication::bolt::Value const &extra) -> memgraph::query::QueryExtras {
auto const &as_map = extra.ValueMap();
auto metadata_pv = std::map<std::string, memgraph::storage::PropertyValue>{};
if (auto const it = as_map.find("tx_metadata"); it != as_map.cend() && it->second.IsMap()) {
for (const auto &[key, bolt_md] : it->second.ValueMap()) {
metadata_pv.emplace(key, memgraph::glue::ToPropertyValue(bolt_md));
}
}
auto tx_timeout = std::optional<int64_t>{};
if (auto const it = as_map.find("tx_timeout"); it != as_map.cend() && it->second.IsInt()) {
tx_timeout = it->second.ValueInt();
}
return memgraph::query::QueryExtras{std::move(metadata_pv), tx_timeout};
}
class SessionHL final : public memgraph::communication::bolt::Session<memgraph::communication::v2::InputStream,
memgraph::communication::v2::OutputStream> {
public:
struct ContextWrapper {
explicit ContextWrapper(memgraph::dbms::SessionContext sc)
: session_context(sc),
interpreter(std::make_unique<memgraph::query::Interpreter>(session_context.interpreter_context.get())),
defunct_(false) {
session_context.interpreter_context->interpreters.WithLock(
[this](auto &interpreters) { interpreters.insert(interpreter.get()); });
}
~ContextWrapper() { Defunct(); }
void Defunct() {
if (!defunct_) {
session_context.interpreter_context->interpreters.WithLock(
[this](auto &interpreters) { interpreters.erase(interpreter.get()); });
defunct_ = true;
}
}
ContextWrapper(const ContextWrapper &) = delete;
ContextWrapper &operator=(const ContextWrapper &) = delete;
ContextWrapper(ContextWrapper &&in) noexcept
: session_context(std::move(in.session_context)),
interpreter(std::move(in.interpreter)),
defunct_(in.defunct_) {
in.defunct_ = true;
}
ContextWrapper &operator=(ContextWrapper &&in) noexcept {
if (this != &in) {
Defunct();
session_context = std::move(in.session_context);
interpreter = std::move(in.interpreter);
defunct_ = in.defunct_;
in.defunct_ = true;
}
return *this;
}
memgraph::query::InterpreterContext *interpreter_context() { return session_context.interpreter_context.get(); }
memgraph::query::Interpreter *interp() { return interpreter.get(); }
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth() const {
return session_context.auth;
}
#ifdef MG_ENTERPRISE
memgraph::audit::Log *audit_log() const { return session_context.audit_log; }
#endif
std::string run_id() const { return session_context.run_id; }
bool defunct() const { return defunct_; }
private:
memgraph::dbms::SessionContext session_context;
std::unique_ptr<memgraph::query::Interpreter> interpreter;
bool defunct_;
};
SessionHL(
#ifdef MG_ENTERPRISE
memgraph::dbms::SessionContextHandler &sc_handler,
#else
memgraph::dbms::SessionContext sc,
#endif
const memgraph::communication::v2::ServerEndpoint &endpoint,
memgraph::communication::v2::InputStream *input_stream, memgraph::communication::v2::OutputStream *output_stream,
const std::string &default_db = memgraph::dbms::kDefaultDB) // NOLINT
: memgraph::communication::bolt::Session<memgraph::communication::v2::InputStream,
memgraph::communication::v2::OutputStream>(input_stream, output_stream),
#ifdef MG_ENTERPRISE
sc_handler_(sc_handler),
current_(sc_handler_.Get(default_db)),
#else
current_(sc),
#endif
interpreter_context_(current_.interpreter_context()),
interpreter_(current_.interp()),
auth_(current_.auth()),
#ifdef MG_ENTERPRISE
audit_log_(current_.audit_log()),
#endif
endpoint_(endpoint),
run_id_(current_.run_id()) {
memgraph::metrics::IncrementCounter(memgraph::metrics::ActiveBoltSessions);
}
~SessionHL() override { memgraph::metrics::DecrementCounter(memgraph::metrics::ActiveBoltSessions); }
SessionHL(const SessionHL &) = delete;
SessionHL &operator=(const SessionHL &) = delete;
SessionHL(SessionHL &&) = delete;
SessionHL &operator=(SessionHL &&) = delete;
void Configure(const std::map<std::string, memgraph::communication::bolt::Value> &run_time_info) override {
#ifdef MG_ENTERPRISE
std::string db;
bool update = false;
// Check if user explicitly defined the database to use
if (run_time_info.contains("db")) {
const auto &db_info = run_time_info.at("db");
if (!db_info.IsString()) {
throw memgraph::communication::bolt::ClientError("Malformed database name.");
}
db = db_info.ValueString();
update = db != current_.interpreter_context()->db->id();
in_explicit_db_ = true;
// NOTE: Once in a transaction, the drivers stop explicitly sending the db and count on using it until commit
} else if (in_explicit_db_ && !interpreter_->in_explicit_transaction_) { // Just on a switch
db = GetDefaultDB();
update = db != current_.interpreter_context()->db->id();
in_explicit_db_ = false;
}
// Check if the underlying database needs to be updated
if (update) {
sc_handler_.SetInPlace(db, [this](auto new_sc) mutable {
const auto &db_name = new_sc.interpreter_context->db->id();
MultiDatabaseAuth(db_name);
try {
Update(ContextWrapper(new_sc));
return memgraph::dbms::SetForResult::SUCCESS;
} catch (memgraph::dbms::UnknownDatabaseException &e) {
throw memgraph::communication::bolt::ClientError("No database named \"{}\" found!", db_name);
}
});
}
#endif
}
using TEncoder = memgraph::communication::bolt::Encoder<
memgraph::communication::bolt::ChunkedEncoderBuffer<memgraph::communication::v2::OutputStream>>;
void BeginTransaction(const std::map<std::string, memgraph::communication::bolt::Value> &extra) override {
interpreter_->BeginTransaction(ToQueryExtras(extra));
}
void CommitTransaction() override { interpreter_->CommitTransaction(); }
void RollbackTransaction() override { interpreter_->RollbackTransaction(); }
std::pair<std::vector<std::string>, std::optional<int>> Interpret(
const std::string &query, const std::map<std::string, memgraph::communication::bolt::Value> &params,
const std::map<std::string, memgraph::communication::bolt::Value> &extra) override {
std::map<std::string, memgraph::storage::PropertyValue> params_pv;
for (const auto &[key, bolt_param] : params) {
params_pv.emplace(key, memgraph::glue::ToPropertyValue(bolt_param));
}
const std::string *username{nullptr};
if (user_) {
username = &user_->username();
}
#ifdef MG_ENTERPRISE
if (memgraph::license::global_license_checker.IsEnterpriseValidFast()) {
audit_log_->Record(endpoint_.address().to_string(), user_ ? *username : "", query,
memgraph::storage::PropertyValue(params_pv), interpreter_context_->db->id());
}
#endif
try {
auto result = interpreter_->Prepare(query, params_pv, username, ToQueryExtras(extra), UUID());
const std::string db_name = result.db ? *result.db : "";
if (user_ && !memgraph::glue::AuthChecker::IsUserAuthorized(*user_, result.privileges, db_name)) {
interpreter_->Abort();
if (db_name.empty()) {
throw memgraph::communication::bolt::ClientError(
"You are not authorized to execute this query! Please contact your database administrator.");
}
throw memgraph::communication::bolt::ClientError(
"You are not authorized to execute this query on database \"{}\"! Please contact your database "
"administrator.",
db_name);
}
return {result.headers, result.qid};
} catch (const memgraph::query::QueryException &e) {
// Wrap QueryException into ClientError, because we want to allow the
// client to fix their query.
throw memgraph::communication::bolt::ClientError(e.what());
} catch (const memgraph::query::ReplicationException &e) {
throw memgraph::communication::bolt::ClientError(e.what());
}
}
std::map<std::string, memgraph::communication::bolt::Value> Pull(TEncoder *encoder, std::optional<int> n,
std::optional<int> qid) override {
TypedValueResultStream stream(encoder, interpreter_context_);
return PullResults(stream, n, qid);
}
std::map<std::string, memgraph::communication::bolt::Value> Discard(std::optional<int> n,
std::optional<int> qid) override {
memgraph::query::DiscardValueResultStream stream;
return PullResults(stream, n, qid);
}
void Abort() override { interpreter_->Abort(); }
// Called during Init
// During Init, the user cannot choose the landing DB (switch is done during query execution)
bool Authenticate(const std::string &username, const std::string &password) override {
auto locked_auth = auth_->Lock();
if (!locked_auth->HasUsers()) {
return true;
}
user_ = locked_auth->Authenticate(username, password);
#ifdef MG_ENTERPRISE
if (user_.has_value()) {
const auto &db = user_->db_access().GetDefault();
// Check if the underlying database needs to be updated
if (db != current_.interpreter_context()->db->id()) {
const auto &res = sc_handler_.SetFor(UUID(), db);
return res == memgraph::dbms::SetForResult::SUCCESS || res == memgraph::dbms::SetForResult::ALREADY_SET;
}
}
#endif
return user_.has_value();
}
std::optional<std::string> GetServerNameForInit() override {
if (FLAGS_bolt_server_name_for_init.empty()) return std::nullopt;
return FLAGS_bolt_server_name_for_init;
}
#ifdef MG_ENTERPRISE
memgraph::dbms::SetForResult OnChange(const std::string &db_name) override {
MultiDatabaseAuth(db_name);
if (db_name != current_.interpreter_context()->db->id()) {
UpdateAndDefunct(db_name); // Done during Pull, so we cannot just replace the current db
return memgraph::dbms::SetForResult::SUCCESS;
}
return memgraph::dbms::SetForResult::ALREADY_SET;
}
bool OnDelete(const std::string &db_name) override {
MG_ASSERT(current_.interpreter_context()->db->id() != db_name && (!defunct_ || defunct_->defunct()),
"Trying to delete a database while still in use.");
return true;
}
#endif
std::string GetDatabaseName() const override { return interpreter_context_->db->id(); }
private:
template <typename TStream>
std::map<std::string, memgraph::communication::bolt::Value> PullResults(TStream &stream, std::optional<int> n,
std::optional<int> qid) {
try {
const auto &summary = interpreter_->Pull(&stream, n, qid);
std::map<std::string, memgraph::communication::bolt::Value> decoded_summary;
for (const auto &kv : summary) {
auto maybe_value =
memgraph::glue::ToBoltValue(kv.second, *interpreter_context_->db, memgraph::storage::View::NEW);
if (maybe_value.HasError()) {
switch (maybe_value.GetError()) {
case memgraph::storage::Error::DELETED_OBJECT:
case memgraph::storage::Error::SERIALIZATION_ERROR:
case memgraph::storage::Error::VERTEX_HAS_EDGES:
case memgraph::storage::Error::PROPERTIES_DISABLED:
case memgraph::storage::Error::NONEXISTENT_OBJECT:
throw memgraph::communication::bolt::ClientError("Unexpected storage error when streaming summary.");
}
}
decoded_summary.emplace(kv.first, std::move(*maybe_value));
}
// Add this memgraph instance run_id, received from telemetry
// This is sent with every query, instead of only on bolt init inside
// communication/bolt/v1/states/init.hpp because neo4jdriver does not
// read the init message.
if (auto run_id = run_id_; run_id) {
decoded_summary.emplace("run_id", *run_id);
}
// Clean up previous session (session gets defunct when switching between databases)
if (defunct_) {
defunct_.reset();
}
return decoded_summary;
} catch (const memgraph::query::QueryException &e) {
// Wrap QueryException into ClientError, because we want to allow the
// client to fix their query.
throw memgraph::communication::bolt::ClientError(e.what());
}
}
#ifdef MG_ENTERPRISE
/**
* @brief Update setup to the new database.
*
* @param db_name name of the target database
* @throws UnknownDatabaseException if handler cannot get it
*/
void UpdateAndDefunct(const std::string &db_name) { UpdateAndDefunct(ContextWrapper(sc_handler_.Get(db_name))); }
void UpdateAndDefunct(ContextWrapper &&cntxt) {
defunct_.emplace(std::move(current_));
Update(std::forward<ContextWrapper>(cntxt));
defunct_->Defunct();
}
void Update(const std::string &db_name) {
ContextWrapper tmp(sc_handler_.Get(db_name));
Update(std::move(tmp));
}
void Update(ContextWrapper &&cntxt) {
current_ = std::move(cntxt);
interpreter_ = current_.interp();
interpreter_->in_explicit_db_ = in_explicit_db_;
interpreter_context_ = current_.interpreter_context();
}
/**
* @brief Authenticate user on passed database.
*
* @param db database to check against
* @throws bolt::ClientError when user is not authorized
*/
void MultiDatabaseAuth(const std::string &db) {
if (user_ && !memgraph::glue::AuthChecker::IsUserAuthorized(*user_, {}, db)) {
throw memgraph::communication::bolt::ClientError(
"You are not authorized on the database \"{}\"! Please contact your database administrator.", db);
}
}
/**
* @brief Get the user's default database
*
* @return std::string
*/
std::string GetDefaultDB() {
if (user_.has_value()) {
return user_->db_access().GetDefault();
}
return memgraph::dbms::kDefaultDB;
}
#endif
/// Wrapper around TEncoder which converts TypedValue to Value
/// before forwarding the calls to original TEncoder.
class TypedValueResultStream {
public:
TypedValueResultStream(TEncoder *encoder, memgraph::query::InterpreterContext *ic)
: encoder_(encoder), interpreter_context_(ic) {}
void Result(const std::vector<memgraph::query::TypedValue> &values) {
std::vector<memgraph::communication::bolt::Value> decoded_values;
decoded_values.reserve(values.size());
for (const auto &v : values) {
auto maybe_value = memgraph::glue::ToBoltValue(v, *interpreter_context_->db, memgraph::storage::View::NEW);
if (maybe_value.HasError()) {
switch (maybe_value.GetError()) {
case memgraph::storage::Error::DELETED_OBJECT:
throw memgraph::communication::bolt::ClientError("Returning a deleted object as a result.");
case memgraph::storage::Error::NONEXISTENT_OBJECT:
throw memgraph::communication::bolt::ClientError("Returning a nonexistent object as a result.");
case memgraph::storage::Error::VERTEX_HAS_EDGES:
case memgraph::storage::Error::SERIALIZATION_ERROR:
case memgraph::storage::Error::PROPERTIES_DISABLED:
throw memgraph::communication::bolt::ClientError("Unexpected storage error when streaming results.");
}
}
decoded_values.emplace_back(std::move(*maybe_value));
}
encoder_->MessageRecord(decoded_values);
}
private:
TEncoder *encoder_;
// NOTE: Needed only for ToBoltValue conversions
memgraph::query::InterpreterContext *interpreter_context_;
};
#ifdef MG_ENTERPRISE
memgraph::dbms::SessionContextHandler &sc_handler_;
#endif
ContextWrapper current_;
std::optional<ContextWrapper> defunct_;
memgraph::query::InterpreterContext *interpreter_context_;
memgraph::query::Interpreter *interpreter_;
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth_;
std::optional<memgraph::auth::User> user_;
#ifdef MG_ENTERPRISE
memgraph::audit::Log *audit_log_;
bool in_explicit_db_{false}; //!< If true, the user has defined the database to use via metadata
#endif
memgraph::communication::v2::ServerEndpoint endpoint_;
// NOTE: run_id should be const but that complicates code a lot.
std::optional<std::string> run_id_;
};
#ifdef MG_ENTERPRISE
using ServerT = memgraph::communication::v2::Server<SessionHL, memgraph::dbms::SessionContextHandler>;
#else
using ServerT = memgraph::communication::v2::Server<SessionHL, memgraph::dbms::SessionContext>;
#endif
using MonitoringServerT =
memgraph::communication::http::Server<memgraph::http::MetricsRequestHandler<memgraph::dbms::SessionContext>,
memgraph::dbms::SessionContext>;
using memgraph::communication::ServerContext;
// Needed to correctly handle memgraph destruction from a signal handler.
@@ -925,6 +73,7 @@ using memgraph::communication::ServerContext;
// when we are exiting main, inside destructors of database::GraphDb and
// similar. The signal handler may then initiate another shutdown on memgraph
// which is in half destructed state, causing invalid memory access and crash.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
volatile sig_atomic_t is_shutting_down = 0;
void InitSignalHandlers(const std::function<void()> &shutdown_fun) {
@@ -965,7 +114,7 @@ int main(int argc, char **argv) {
exit(1);
}
InitializeLogger();
memgraph::flags::InitializeLogger();
// Unhandled exception handler init.
std::set_terminate(&memgraph::utils::TerminateHandler);
@@ -1049,7 +198,7 @@ int main(int argc, char **argv) {
auto data_directory = std::filesystem::path(FLAGS_data_directory);
const auto memory_limit = GetMemoryLimit();
const auto memory_limit = memgraph::flags::GetMemoryLimit();
// NOLINTNEXTLINE(bugprone-narrowing-conversions,cppcoreguidelines-narrowing-conversions)
spdlog::info("Memory limit in config is set to {}", memgraph::utils::GetReadableSize(memory_limit));
memgraph::utils::total_memory_tracker.SetMaximumHardLimit(memory_limit);
@@ -1061,6 +210,7 @@ int main(int argc, char **argv) {
// register all runtime settings
memgraph::license::RegisterLicenseSettings(memgraph::license::global_license_checker,
memgraph::utils::global_settings);
memgraph::flags::run_time::Initialize();
memgraph::license::global_license_checker.CheckEnvLicense();
if (!FLAGS_organization_name.empty() && !FLAGS_license_key.empty()) {
@@ -1112,7 +262,7 @@ int main(int argc, char **argv) {
.items_per_batch = FLAGS_storage_items_per_batch,
.recovery_thread_count = FLAGS_storage_recovery_thread_count,
.allow_parallel_index_creation = FLAGS_storage_parallel_index_recovery},
.transaction = {.isolation_level = ParseIsolationLevel()},
.transaction = {.isolation_level = memgraph::flags::ParseIsolationLevel()},
.disk = {.main_storage_directory = FLAGS_data_directory + "/rocksdb_main_storage",
.label_index_directory = FLAGS_data_directory + "/rocksdb_label_index",
.label_property_index_directory = FLAGS_data_directory + "/rocksdb_label_property_index",
@@ -1143,7 +293,6 @@ int main(int argc, char **argv) {
// Default interpreter configuration
memgraph::query::InterpreterConfig interp_config{
.query = {.allow_load_csv = FLAGS_allow_load_csv},
.execution_timeout_sec = FLAGS_query_execution_timeout_sec,
.replication_replica_check_frequency = std::chrono::seconds(FLAGS_replication_replica_check_frequency_sec),
.default_kafka_bootstrap_servers = FLAGS_kafka_bootstrap_servers,
.default_pulsar_service_url = FLAGS_pulsar_service_url,
@@ -1192,7 +341,8 @@ int main(int argc, char **argv) {
auto *auth = session_context.auth;
auto &interpreter_context = *session_context.interpreter_context; // TODO remove
memgraph::query::procedure::gModuleRegistry.SetModulesDirectory(query_modules_directories, FLAGS_data_directory);
memgraph::query::procedure::gModuleRegistry.SetModulesDirectory(memgraph::flags::ParseQueryModulesDirectory(),
FLAGS_data_directory);
memgraph::query::procedure::gModuleRegistry.UnloadAndLoadModulesFromDirectories();
memgraph::query::procedure::gCallableAliasMapper.LoadMapping(FLAGS_query_callable_mappings_path);
@@ -1239,11 +389,11 @@ int main(int argc, char **argv) {
auto server_endpoint = memgraph::communication::v2::ServerEndpoint{
boost::asio::ip::address::from_string(FLAGS_bolt_address), static_cast<uint16_t>(FLAGS_bolt_port)};
#ifdef MG_ENTERPRISE
ServerT server(server_endpoint, &sc_handler, &context, FLAGS_bolt_session_inactivity_timeout, service_name,
FLAGS_bolt_num_workers);
memgraph::glue::ServerT server(server_endpoint, &sc_handler, &context, FLAGS_bolt_session_inactivity_timeout,
service_name, FLAGS_bolt_num_workers);
#else
ServerT server(server_endpoint, &session_context, &context, FLAGS_bolt_session_inactivity_timeout, service_name,
FLAGS_bolt_num_workers);
memgraph::glue::ServerT server(server_endpoint, &session_context, &context, FLAGS_bolt_session_inactivity_timeout,
service_name, FLAGS_bolt_num_workers);
#endif
const auto machine_id = memgraph::utils::GetMachineId();
@@ -1283,9 +433,9 @@ int main(int argc, char **argv) {
memgraph::communication::websocket::SafeAuth websocket_auth{auth};
memgraph::communication::websocket::Server websocket_server{
{FLAGS_monitoring_address, static_cast<uint16_t>(FLAGS_monitoring_port)}, &context, websocket_auth};
AddLoggerSink(websocket_server.GetLoggingSink());
memgraph::flags::AddLoggerSink(websocket_server.GetLoggingSink());
MonitoringServerT metrics_server{
memgraph::glue::MonitoringServerT metrics_server{
{FLAGS_metrics_address, static_cast<uint16_t>(FLAGS_metrics_port)}, &session_context, &context};
#ifdef MG_ENTERPRISE

View File

@@ -20,6 +20,7 @@
#include <unordered_map>
#include "helpers.hpp"
#include "storage/v2/config.hpp"
#include "storage/v2/edge_accessor.hpp"
#include "storage/v2/inmemory/storage.hpp"
#include "utils/exceptions.hpp"
@@ -700,13 +701,14 @@ int main(int argc, char *argv[]) {
}
std::unordered_map<NodeId, memgraph::storage::Gid> node_id_map;
std::unique_ptr<memgraph::storage::Storage> store{new memgraph::storage::InMemoryStorage{{
auto store = std::make_unique<memgraph::storage::InMemoryStorage>(memgraph::storage::Config{
.items = {.properties_on_edges = FLAGS_storage_properties_on_edges},
.durability = {.storage_directory = FLAGS_data_directory,
.recover_on_startup = false,
.snapshot_wal_mode = memgraph::storage::Config::Durability::SnapshotWalMode::DISABLED,
.snapshot_on_exit = true},
}}};
});
memgraph::utils::Timer load_timer;

View File

@@ -40,7 +40,18 @@ set(mg_query_sources
add_library(mg-query STATIC ${mg_query_sources})
target_include_directories(mg-query PUBLIC ${CMAKE_SOURCE_DIR}/include)
target_link_libraries(mg-query PUBLIC dl cppitertools Python3::Python mg-integrations-pulsar mg-integrations-kafka mg-storage-v2 mg-license mg-utils mg-kvstore mg-memory mg::csv)
target_link_libraries(mg-query PUBLIC dl
cppitertools
Python3::Python
mg-integrations-pulsar
mg-integrations-kafka
mg-storage-v2
mg-license
mg-utils
mg-kvstore
mg-memory
mg::csv
mg-flags)
if(NOT "${MG_PYTHON_PATH}" STREQUAL "")
set(Python3_ROOT_DIR "${MG_PYTHON_PATH}")
endif()

View File

@@ -30,6 +30,8 @@ class AuthChecker {
#ifdef MG_ENTERPRISE
[[nodiscard]] virtual std::unique_ptr<FineGrainedAuthChecker> GetFineGrainedAuthChecker(
const std::string &username, const memgraph::query::DbAccessor *db_accessor) const = 0;
virtual void ClearCache() const = 0;
#endif
};
#ifdef MG_ENTERPRISE
@@ -103,6 +105,8 @@ class AllowEverythingAuthChecker final : public query::AuthChecker {
const query::DbAccessor * /*dba*/) const override {
return std::make_unique<AllowEverythingFineGrainedAuthChecker>();
}
void ClearCache() const override {}
#endif
}; // namespace memgraph::query

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -19,8 +19,6 @@ struct InterpreterConfig {
bool allow_load_csv{true};
} query;
// The default execution timeout is 10 minutes.
double execution_timeout_sec{600.0};
// The same as \ref memgraph::storage::replication::ReplicationClientConfig
std::chrono::seconds replication_replica_check_frequency{1};

View File

@@ -51,6 +51,8 @@ struct EvaluationContext {
/// All counters generated by `counter` function, mutable because the function
/// modifies the values
mutable std::unordered_map<std::string, int64_t> counters{};
/// Property lookup cache ({symbol: {property_id: property_value, ...}, ...})
mutable std::unordered_map<int32_t, std::map<storage::PropertyId, storage::PropertyValue>> property_lookups_cache{};
};
inline std::vector<storage::PropertyId> NamesToProperties(const std::vector<std::string> &property_names,

View File

@@ -76,7 +76,7 @@ ParsedQuery ParseQuery(const std::string &query_string, const std::map<std::stri
// Convert the ANTLR4 parse tree into an AST.
AstStorage ast_storage;
frontend::ParsingContext context{true};
frontend::ParsingContext context{.is_query_cached = true};
frontend::CypherMainVisitor visitor(context, &ast_storage);
visitor.visit(parser->tree());

View File

@@ -76,6 +76,24 @@ SubgraphDbAccessor::DetachRemoveVertex( // NOLINT(readability-convert-member-fu
"Vertex holds only partial information about edges. Cannot detach delete safely while using projected graph."};
}
storage::Result<EdgeAccessor> SubgraphDbAccessor::EdgeSetFrom(EdgeAccessor *edge, SubgraphVertexAccessor *new_from) {
VertexAccessor *new_from_impl = &new_from->impl_;
if (!this->graph_->ContainsVertex(*new_from_impl)) {
throw std::logic_error{"Projected graph must contain the new `from` vertex!"};
}
auto result = db_accessor_.EdgeSetFrom(edge, new_from_impl);
return result;
}
storage::Result<EdgeAccessor> SubgraphDbAccessor::EdgeSetTo(EdgeAccessor *edge, SubgraphVertexAccessor *new_to) {
VertexAccessor *new_to_impl = &new_to->impl_;
if (!this->graph_->ContainsVertex(*new_to_impl)) {
throw std::logic_error{"Projected graph must contain the new `to` vertex!"};
}
auto result = db_accessor_.EdgeSetTo(edge, new_to_impl);
return result;
}
storage::Result<std::optional<VertexAccessor>> SubgraphDbAccessor::RemoveVertex(
SubgraphVertexAccessor *subgraphvertex_accessor) {
VertexAccessor *vertex_accessor = &subgraphvertex_accessor->impl_;
@@ -111,10 +129,10 @@ query::Graph *SubgraphDbAccessor::getGraph() { return graph_; }
VertexAccessor SubgraphVertexAccessor::GetVertexAccessor() const { return impl_; }
auto SubgraphVertexAccessor::OutEdges(storage::View view) const -> decltype(impl_.OutEdges(view)) {
storage::Result<EdgeVertexAccessorResult> SubgraphVertexAccessor::OutEdges(storage::View view) const {
auto maybe_edges = impl_.impl_.OutEdges(view, {});
if (maybe_edges.HasError()) return maybe_edges.GetError();
auto edges = std::move(*maybe_edges);
auto edges = std::move(maybe_edges->edges);
const auto &graph_edges = graph_->edges();
std::vector<storage::EdgeAccessor> filteredOutEdges;
@@ -125,13 +143,18 @@ auto SubgraphVertexAccessor::OutEdges(storage::View view) const -> decltype(impl
}
}
return iter::imap(VertexAccessor::MakeEdgeAccessor, std::move(filteredOutEdges));
std::vector<EdgeAccessor> resulting_edges;
resulting_edges.reserve(filteredOutEdges.size());
std::ranges::transform(filteredOutEdges, std::back_inserter(resulting_edges),
[](auto const &edge) { return VertexAccessor::MakeEdgeAccessor(edge); });
return EdgeVertexAccessorResult{.edges = std::move(resulting_edges), .expanded_count = maybe_edges->expanded_count};
}
auto SubgraphVertexAccessor::InEdges(storage::View view) const -> decltype(impl_.InEdges(view)) {
storage::Result<EdgeVertexAccessorResult> SubgraphVertexAccessor::InEdges(storage::View view) const {
auto maybe_edges = impl_.impl_.InEdges(view, {});
if (maybe_edges.HasError()) return maybe_edges.GetError();
auto edges = std::move(*maybe_edges);
auto edges = std::move(maybe_edges->edges);
const auto &graph_edges = graph_->edges();
std::vector<storage::EdgeAccessor> filteredOutEdges;
@@ -142,7 +165,12 @@ auto SubgraphVertexAccessor::InEdges(storage::View view) const -> decltype(impl_
}
}
return iter::imap(VertexAccessor::MakeEdgeAccessor, std::move(filteredOutEdges));
std::vector<EdgeAccessor> resulting_edges;
resulting_edges.reserve(filteredOutEdges.size());
std::ranges::transform(filteredOutEdges, std::back_inserter(resulting_edges),
[](auto const &edge) { return VertexAccessor::MakeEdgeAccessor(edge); });
return EdgeVertexAccessorResult{.edges = std::move(resulting_edges), .expanded_count = maybe_edges->expanded_count};
}
} // namespace memgraph::query

View File

@@ -12,6 +12,7 @@
#pragma once
#include <optional>
#include <ranges>
#include <cppitertools/filter.hpp>
#include <cppitertools/imap.hpp>
@@ -21,31 +22,13 @@
#include "storage/v2/id_types.hpp"
#include "storage/v2/property_value.hpp"
#include "storage/v2/result.hpp"
#include "storage/v2/storage_mode.hpp"
#include "utils/pmr/unordered_set.hpp"
#include "utils/variant_helpers.hpp"
///////////////////////////////////////////////////////////
// Our communication layer and query engine don't mix
// very well on Centos because OpenSSL version available
// on Centos 7 include libkrb5 which has brilliant macros
// called TRUE and FALSE. For more detailed explanation go
// to memgraph.cpp.
//
// Because of the replication storage now uses some form of
// communication so we have some unwanted macros.
// This cannot be avoided by simple include orderings so we
// simply undefine those macros as we're sure that libkrb5
// won't and can't be used anywhere in the query engine.
#include "storage/v2/storage.hpp"
#undef FALSE
#undef TRUE
///////////////////////////////////////////////////////////
#include "storage/v2/storage_mode.hpp"
#include "storage/v2/view.hpp"
#include "utils/bound.hpp"
#include "utils/exceptions.hpp"
#include "utils/pmr/unordered_set.hpp"
#include "utils/variant_helpers.hpp"
namespace memgraph::query {
@@ -94,6 +77,14 @@ class EdgeAccessor final {
VertexAccessor From() const;
/// When edge is deleted and you are accessing To vertex
/// for_deleted_ flag will in this case be updated properly
VertexAccessor DeletedEdgeToVertex() const;
/// When edge is deleted and you are accessing From vertex
/// for_deleted_ flag will in this case be updated properly
VertexAccessor DeletedEdgeFromVertex() const;
bool IsCycle() const;
int64_t CypherId() const { return impl_.Gid().AsInt(); }
@@ -105,6 +96,11 @@ class EdgeAccessor final {
bool operator!=(const EdgeAccessor &e) const noexcept { return !(*this == e); }
};
struct EdgeVertexAccessorResult {
std::vector<EdgeAccessor> edges;
int64_t expanded_count;
};
class VertexAccessor final {
public:
storage::VertexAccessor impl_;
@@ -153,37 +149,62 @@ class VertexAccessor final {
return impl_.ClearProperties();
}
auto InEdges(storage::View view, const std::vector<storage::EdgeTypeId> &edge_types) const
-> storage::Result<decltype(iter::imap(MakeEdgeAccessor, *impl_.InEdges(view)))> {
auto maybe_edges = impl_.InEdges(view, edge_types);
if (maybe_edges.HasError()) return maybe_edges.GetError();
return iter::imap(MakeEdgeAccessor, std::move(*maybe_edges));
storage::Result<EdgeVertexAccessorResult> InEdges(storage::View view,
const std::vector<storage::EdgeTypeId> &edge_types) const {
auto maybe_result = impl_.InEdges(view, edge_types);
if (maybe_result.HasError()) return maybe_result.GetError();
std::vector<EdgeAccessor> edges;
edges.reserve((*maybe_result).edges.size());
std::ranges::transform((*maybe_result).edges, std::back_inserter(edges),
[](auto const &edge) { return EdgeAccessor(edge); });
return EdgeVertexAccessorResult{.edges = edges, .expanded_count = (*maybe_result).expanded_count};
}
auto InEdges(storage::View view) const { return InEdges(view, {}); }
storage::Result<EdgeVertexAccessorResult> InEdges(storage::View view) const { return InEdges(view, {}); }
auto InEdges(storage::View view, const std::vector<storage::EdgeTypeId> &edge_types, const VertexAccessor &dest) const
-> storage::Result<decltype(iter::imap(MakeEdgeAccessor, *impl_.InEdges(view)))> {
auto maybe_edges = impl_.InEdges(view, edge_types, &dest.impl_);
if (maybe_edges.HasError()) return maybe_edges.GetError();
return iter::imap(MakeEdgeAccessor, std::move(*maybe_edges));
storage::Result<EdgeVertexAccessorResult> InEdges(storage::View view,
const std::vector<storage::EdgeTypeId> &edge_types,
const VertexAccessor &dest) const {
auto maybe_result = impl_.InEdges(view, edge_types, &dest.impl_);
if (maybe_result.HasError()) return maybe_result.GetError();
std::vector<EdgeAccessor> edges;
edges.reserve((*maybe_result).edges.size());
std::ranges::transform((*maybe_result).edges, std::back_inserter(edges),
[](auto const &edge) { return EdgeAccessor(edge); });
return EdgeVertexAccessorResult{.edges = edges, .expanded_count = (*maybe_result).expanded_count};
}
auto OutEdges(storage::View view, const std::vector<storage::EdgeTypeId> &edge_types) const
-> storage::Result<decltype(iter::imap(MakeEdgeAccessor, *impl_.OutEdges(view)))> {
auto maybe_edges = impl_.OutEdges(view, edge_types);
if (maybe_edges.HasError()) return maybe_edges.GetError();
return iter::imap(MakeEdgeAccessor, std::move(*maybe_edges));
storage::Result<EdgeVertexAccessorResult> OutEdges(storage::View view,
const std::vector<storage::EdgeTypeId> &edge_types) const {
auto maybe_result = impl_.OutEdges(view, edge_types);
if (maybe_result.HasError()) return maybe_result.GetError();
std::vector<EdgeAccessor> edges;
edges.reserve((*maybe_result).edges.size());
std::ranges::transform((*maybe_result).edges, std::back_inserter(edges),
[](auto const &edge) { return EdgeAccessor(edge); });
return EdgeVertexAccessorResult{.edges = edges, .expanded_count = (*maybe_result).expanded_count};
}
auto OutEdges(storage::View view) const { return OutEdges(view, {}); }
storage::Result<EdgeVertexAccessorResult> OutEdges(storage::View view) const { return OutEdges(view, {}); }
auto OutEdges(storage::View view, const std::vector<storage::EdgeTypeId> &edge_types,
const VertexAccessor &dest) const
-> storage::Result<decltype(iter::imap(MakeEdgeAccessor, *impl_.OutEdges(view)))> {
auto maybe_edges = impl_.OutEdges(view, edge_types, &dest.impl_);
if (maybe_edges.HasError()) return maybe_edges.GetError();
return iter::imap(MakeEdgeAccessor, std::move(*maybe_edges));
storage::Result<EdgeVertexAccessorResult> OutEdges(storage::View view,
const std::vector<storage::EdgeTypeId> &edge_types,
const VertexAccessor &dest) const {
auto maybe_result = impl_.OutEdges(view, edge_types, &dest.impl_);
if (maybe_result.HasError()) return maybe_result.GetError();
std::vector<EdgeAccessor> edges;
edges.reserve((*maybe_result).edges.size());
std::ranges::transform((*maybe_result).edges, std::back_inserter(edges),
[](auto const &edge) { return EdgeAccessor(edge); });
return EdgeVertexAccessorResult{.edges = edges, .expanded_count = (*maybe_result).expanded_count};
}
storage::Result<size_t> InDegree(storage::View view) const { return impl_.InDegree(view); }
@@ -206,6 +227,12 @@ inline VertexAccessor EdgeAccessor::To() const { return VertexAccessor(impl_.ToV
inline VertexAccessor EdgeAccessor::From() const { return VertexAccessor(impl_.FromVertex()); }
inline VertexAccessor EdgeAccessor::DeletedEdgeToVertex() const { return VertexAccessor(impl_.DeletedEdgeToVertex()); }
inline VertexAccessor EdgeAccessor::DeletedEdgeFromVertex() const {
return VertexAccessor(impl_.DeletedEdgeFromVertex());
}
inline bool EdgeAccessor::IsCycle() const { return To() == From(); }
class SubgraphVertexAccessor final {
@@ -242,15 +269,24 @@ class SubgraphVertexAccessor final {
storage::Gid Gid() const noexcept { return impl_.Gid(); }
storage::Result<size_t> InDegree(storage::View view) const { return impl_.InDegree(view); }
storage::Result<size_t> OutDegree(storage::View view) const { return impl_.OutDegree(view); }
storage::Result<storage::PropertyValue> SetProperty(storage::PropertyId key, const storage::PropertyValue &value) {
return impl_.SetProperty(key, value);
}
storage::Result<std::vector<std::tuple<storage::PropertyId, storage::PropertyValue, storage::PropertyValue>>>
UpdateProperties(std::map<storage::PropertyId, storage::PropertyValue> &properties) const {
return impl_.UpdateProperties(properties);
}
VertexAccessor GetVertexAccessor() const;
};
} // namespace memgraph::query
namespace std {
template <>
struct hash<memgraph::query::VertexAccessor> {
size_t operator()(const memgraph::query::VertexAccessor &v) const { return std::hash<decltype(v.impl_)>{}(v.impl_); }
@@ -278,17 +314,17 @@ class VerticesIterable final {
it_;
public:
explicit Iterator(storage::VerticesIterable::Iterator it) : it_(it) {}
explicit Iterator(storage::VerticesIterable::Iterator it) : it_(std::move(it)) {}
explicit Iterator(std::unordered_set<VertexAccessor, std::hash<VertexAccessor>, std::equal_to<void>,
utils::Allocator<VertexAccessor>>::iterator it)
: it_(it) {}
VertexAccessor operator*() const {
return std::visit([](auto it_) { return VertexAccessor(*it_); }, it_);
return std::visit([](auto &it_) { return VertexAccessor(*it_); }, it_);
}
Iterator &operator++() {
std::visit([this](auto it_) { this->it_ = ++it_; }, it_);
std::visit([](auto &it_) { ++it_; }, it_);
return *this;
}
@@ -370,6 +406,18 @@ class DbAccessor final {
return EdgeAccessor(*maybe_edge);
}
storage::Result<EdgeAccessor> EdgeSetFrom(EdgeAccessor *edge, VertexAccessor *new_from) {
auto changed_edge = accessor_->EdgeSetFrom(&edge->impl_, &new_from->impl_);
if (changed_edge.HasError()) return storage::Result<EdgeAccessor>(changed_edge.GetError());
return EdgeAccessor(*changed_edge);
}
storage::Result<EdgeAccessor> EdgeSetTo(EdgeAccessor *edge, VertexAccessor *new_to) {
auto changed_edge = accessor_->EdgeSetTo(&edge->impl_, &new_to->impl_);
if (changed_edge.HasError()) return storage::Result<EdgeAccessor>(changed_edge.GetError());
return EdgeAccessor(*changed_edge);
}
storage::Result<std::optional<EdgeAccessor>> RemoveEdge(EdgeAccessor *edge) {
auto res = accessor_->DeleteEdge(&edge->impl_);
if (res.HasError()) {
@@ -404,8 +452,8 @@ class DbAccessor final {
std::vector<EdgeAccessor> deleted_edges;
deleted_edges.reserve(edges.size());
std::transform(edges.begin(), edges.end(), std::back_inserter(deleted_edges),
[](const auto &deleted_edge) { return EdgeAccessor{deleted_edge}; });
std::ranges::transform(edges, std::back_inserter(deleted_edges),
[](const auto &deleted_edge) { return EdgeAccessor{deleted_edge}; });
return std::make_optional<ReturnType>(vertex, std::move(deleted_edges));
}
@@ -424,6 +472,53 @@ class DbAccessor final {
return std::make_optional<VertexAccessor>(*value);
}
storage::Result<std::optional<std::pair<std::vector<VertexAccessor>, std::vector<EdgeAccessor>>>> DetachDelete(
std::vector<VertexAccessor> nodes, std::vector<EdgeAccessor> edges, bool detach) {
using ReturnType = std::pair<std::vector<VertexAccessor>, std::vector<EdgeAccessor>>;
std::vector<storage::VertexAccessor *> nodes_impl;
std::vector<storage::EdgeAccessor *> edges_impl;
nodes_impl.reserve(nodes.size());
edges_impl.reserve(edges.size());
for (auto &vertex_accessor : nodes) {
accessor_->PrefetchOutEdges(vertex_accessor.impl_);
accessor_->PrefetchInEdges(vertex_accessor.impl_);
nodes_impl.push_back(&vertex_accessor.impl_);
}
for (auto &edge_accessor : edges) {
edges_impl.push_back(&edge_accessor.impl_);
}
auto res = accessor_->DetachDelete(std::move(nodes_impl), std::move(edges_impl), detach);
if (res.HasError()) {
return res.GetError();
}
const auto &value = res.GetValue();
if (!value) {
return std::optional<ReturnType>{};
}
const auto &[val_vertices, val_edges] = *value;
std::vector<VertexAccessor> deleted_vertices;
std::vector<EdgeAccessor> deleted_edges;
deleted_vertices.reserve(val_vertices.size());
deleted_edges.reserve(val_edges.size());
std::ranges::transform(val_vertices, std::back_inserter(deleted_vertices),
[](const auto &deleted_vertex) { return VertexAccessor{deleted_vertex}; });
std::ranges::transform(val_edges, std::back_inserter(deleted_edges),
[](const auto &deleted_edge) { return EdgeAccessor{deleted_edge}; });
return std::make_optional<ReturnType>(std::move(deleted_vertices), std::move(deleted_edges));
}
storage::PropertyId NameToProperty(const std::string_view name) { return accessor_->NameToProperty(name); }
storage::LabelId NameToLabel(const std::string_view name) { return accessor_->NameToLabel(name); }
@@ -539,6 +634,10 @@ class SubgraphDbAccessor final {
storage::Result<EdgeAccessor> InsertEdge(SubgraphVertexAccessor *from, SubgraphVertexAccessor *to,
const storage::EdgeTypeId &edge_type);
storage::Result<EdgeAccessor> EdgeSetFrom(EdgeAccessor *edge, SubgraphVertexAccessor *new_from);
storage::Result<EdgeAccessor> EdgeSetTo(EdgeAccessor *edge, SubgraphVertexAccessor *new_to);
storage::Result<std::optional<std::pair<VertexAccessor, std::vector<EdgeAccessor>>>> DetachRemoveVertex(
SubgraphVertexAccessor *vertex_accessor);

View File

@@ -487,8 +487,8 @@ PullPlanDump::PullChunk PullPlanDump::CreateEdgePullChunk() {
}
auto &maybe_edges = *maybe_edge_iterable;
MG_ASSERT(maybe_edges.HasValue(), "Invalid database state!");
auto current_edge_iter = maybe_current_edge_iter ? *maybe_current_edge_iter : maybe_edges->begin();
for (; current_edge_iter != maybe_edges->end() && (!n || local_counter < *n); ++current_edge_iter) {
auto current_edge_iter = maybe_current_edge_iter ? *maybe_current_edge_iter : maybe_edges->edges.begin();
for (; current_edge_iter != maybe_edges->edges.end() && (!n || local_counter < *n); ++current_edge_iter) {
std::ostringstream os;
DumpEdge(&os, dba_, *current_edge_iter);
stream->Result({TypedValue(os.str())});
@@ -496,7 +496,7 @@ PullPlanDump::PullChunk PullPlanDump::CreateEdgePullChunk() {
++local_counter;
}
if (current_edge_iter != maybe_edges->end()) {
if (current_edge_iter != maybe_edges->edges.end()) {
maybe_current_edge_iter.emplace(current_edge_iter);
return std::nullopt;
}

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -38,7 +38,7 @@ struct PullPlanDump {
using VertexAccessorIterableIterator = decltype(std::declval<VertexAccessorIterable>().begin());
using EdgeAccessorIterable = decltype(std::declval<VertexAccessor>().OutEdges(storage::View::OLD));
using EdgeAccessorIterableIterator = decltype(std::declval<EdgeAccessorIterable>().GetValue().begin());
using EdgeAccessorIterableIterator = decltype(std::declval<EdgeAccessorIterable>().GetValue().edges.begin());
VertexAccessorIterable vertices_iterable_;
bool internal_index_created_ = false;

View File

@@ -158,9 +158,12 @@ class ExplicitTransactionUsageException : public QueryRuntimeException {
using QueryRuntimeException::QueryRuntimeException;
};
/**
* An exception for serialization error
*/
class WriteVertexOperationInEdgeImportModeException : public QueryException {
public:
WriteVertexOperationInEdgeImportModeException()
: QueryException("Write operations on vertices are forbidden while the edge import mode is active.") {}
};
class TransactionSerializationException : public QueryException {
public:
using QueryException::QueryException;
@@ -271,6 +274,12 @@ class StorageModeModificationInMulticommandTxException : public QueryException {
: QueryException("Storage mode cannot be modified in multicommand transactions.") {}
};
class EdgeImportModeModificationInMulticommandTxException : public QueryException {
public:
EdgeImportModeModificationInMulticommandTxException()
: QueryException("Edge import mode cannot be modified in multicommand transactions.") {}
};
class CreateSnapshotInMulticommandTxException final : public QueryException {
public:
CreateSnapshotInMulticommandTxException()
@@ -282,6 +291,12 @@ class CreateSnapshotDisabledOnDiskStorage final : public QueryException {
CreateSnapshotDisabledOnDiskStorage() : QueryException("In the on-disk storage mode data is already persistent.") {}
};
class EdgeImportModeQueryDisabledOnDiskStorage final : public QueryException {
public:
EdgeImportModeQueryDisabledOnDiskStorage()
: QueryException("Edge import mode is only allowed for on-disk storage mode.") {}
};
class SettingConfigInMulticommandTxException final : public QueryException {
public:
SettingConfigInMulticommandTxException()

View File

@@ -285,4 +285,8 @@ constexpr utils::TypeInfo query::MultiDatabaseQuery::kType{utils::TypeId::AST_MU
constexpr utils::TypeInfo query::ShowDatabasesQuery::kType{utils::TypeId::AST_SHOW_DATABASES, "ShowDatabasesQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::EdgeImportModeQuery::kType{utils::TypeId::AST_EDGE_IMPORT_MODE_QUERY,
"EdgeImportModeQuery", &query::Query::kType};
} // namespace memgraph

View File

@@ -1105,6 +1105,8 @@ class MapProjectionLiteral : public memgraph::query::BaseLiteral {
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
map_variable_->Accept(visitor);
for (auto pair : elements_) {
if (!pair.second) continue;
@@ -1184,6 +1186,8 @@ class PropertyLookup : public memgraph::query::Expression {
static const utils::TypeInfo kType;
const utils::TypeInfo &GetTypeInfo() const override { return kType; }
enum class EvaluationMode { GET_OWN_PROPERTY, GET_ALL_PROPERTIES };
PropertyLookup() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
@@ -1198,11 +1202,13 @@ class PropertyLookup : public memgraph::query::Expression {
memgraph::query::Expression *expression_{nullptr};
memgraph::query::PropertyIx property_;
memgraph::query::PropertyLookup::EvaluationMode evaluation_mode_{EvaluationMode::GET_OWN_PROPERTY};
PropertyLookup *Clone(AstStorage *storage) const override {
PropertyLookup *object = storage->Create<PropertyLookup>();
object->expression_ = expression_ ? expression_->Clone(storage) : nullptr;
object->property_ = storage->GetPropertyIx(property_.name);
object->evaluation_mode_ = evaluation_mode_;
return object;
}
@@ -3003,6 +3009,29 @@ class ReplicationQuery : public memgraph::query::Query {
friend class AstStorage;
};
class EdgeImportModeQuery : public memgraph::query::Query {
public:
static const utils::TypeInfo kType;
const utils::TypeInfo &GetTypeInfo() const override { return kType; }
enum class Status { ACTIVE, INACTIVE };
EdgeImportModeQuery() = default;
DEFVISITABLE(QueryVisitor<void>);
memgraph::query::EdgeImportModeQuery::Status status_;
EdgeImportModeQuery *Clone(AstStorage *storage) const override {
auto *object = storage->Create<EdgeImportModeQuery>();
object->status_ = status_;
return object;
}
private:
friend class AstStorage;
};
class LockPathQuery : public memgraph::query::Query {
public:
static const utils::TypeInfo kType;

View File

@@ -105,6 +105,7 @@ class TransactionQueueQuery;
class Exists;
class MultiDatabaseQuery;
class ShowDatabasesQuery;
class EdgeImportModeQuery;
using TreeCompositeVisitor = utils::CompositeVisitor<
SingleQuery, CypherUnion, NamedExpression, OrOperator, XorOperator, AndOperator, NotOperator, AdditionOperator,
@@ -143,6 +144,6 @@ class QueryVisitor
ConstraintQuery, DumpQuery, ReplicationQuery, LockPathQuery, FreeMemoryQuery, TriggerQuery,
IsolationLevelQuery, CreateSnapshotQuery, StreamQuery, SettingQuery, VersionQuery,
ShowConfigQuery, TransactionQueueQuery, StorageModeQuery, AnalyzeGraphQuery,
MultiDatabaseQuery, ShowDatabasesQuery> {};
MultiDatabaseQuery, ShowDatabasesQuery, EdgeImportModeQuery> {};
} // namespace memgraph::query

View File

@@ -269,6 +269,17 @@ antlrcpp::Any CypherMainVisitor::visitReplicationQuery(MemgraphCypher::Replicati
return replication_query;
}
antlrcpp::Any CypherMainVisitor::visitEdgeImportModeQuery(MemgraphCypher::EdgeImportModeQueryContext *ctx) {
auto *edge_import_mode_query = storage_->Create<EdgeImportModeQuery>();
if (ctx->ACTIVE()) {
edge_import_mode_query->status_ = EdgeImportModeQuery::Status::ACTIVE;
} else {
edge_import_mode_query->status_ = EdgeImportModeQuery::Status::INACTIVE;
}
query_ = edge_import_mode_query;
return edge_import_mode_query;
}
antlrcpp::Any CypherMainVisitor::visitSetReplicationRole(MemgraphCypher::SetReplicationRoleContext *ctx) {
auto *replication_query = storage_->Create<ReplicationQuery>();
replication_query->action_ = ReplicationQuery::Action::SET_REPLICATION_ROLE;

View File

@@ -196,6 +196,11 @@ class CypherMainVisitor : public antlropencypher::MemgraphCypherBaseVisitor {
*/
antlrcpp::Any visitReplicationQuery(MemgraphCypher::ReplicationQueryContext *ctx) override;
/**
* @return EdgeImportMode*
*/
antlrcpp::Any visitEdgeImportModeQuery(MemgraphCypher::EdgeImportModeQueryContext *ctx) override;
/**
* @return ReplicationQuery*
*/

View File

@@ -20,6 +20,7 @@ options { tokenVocab=MemgraphCypherLexer; }
import Cypher ;
memgraphCypherKeyword : cypherKeyword
| ACTIVE
| AFTER
| ALTER
| ANALYZE
@@ -48,6 +49,7 @@ memgraphCypherKeyword : cypherKeyword
| DENY
| DROP
| DUMP
| EDGE
| EDGE_TYPES
| EXECUTE
| FOR
@@ -60,9 +62,11 @@ memgraphCypherKeyword : cypherKeyword
| HEADER
| IDENTIFIED
| NULLIF
| ISOLATION
| IMPORT
| INACTIVE
| IN_MEMORY_ANALYTICAL
| IN_MEMORY_TRANSACTIONAL
| ISOLATION
| KAFKA
| LABELS
| LEVEL
@@ -143,6 +147,7 @@ query : cypherQuery
| transactionQueueQuery
| multiDatabaseQuery
| showDatabases
| edgeImportModeQuery
;
authQuery : createRole
@@ -475,3 +480,5 @@ useDatabase : USE DATABASE databaseName ;
dropDatabase : DROP DATABASE databaseName ;
showDatabases: SHOW DATABASES ;
edgeImportModeQuery : EDGE IMPORT MODE ( ACTIVE | INACTIVE ) ;

View File

@@ -23,6 +23,7 @@ lexer grammar MemgraphCypherLexer ;
import CypherLexer ;
ACTIVE : A C T I V E ;
AFTER : A F T E R ;
ALTER : A L T E R ;
ANALYZE : A N A L Y Z E ;
@@ -55,6 +56,7 @@ DIRECTORY : D I R E C T O R Y ;
DROP : D R O P ;
DUMP : D U M P ;
DURABILITY : D U R A B I L I T Y ;
EDGE : E D G E ;
EDGE_TYPES : E D G E UNDERSCORE T Y P E S ;
EXECUTE : E X E C U T E ;
FOR : F O R ;
@@ -69,9 +71,11 @@ GRANTS : G R A N T S ;
HEADER : H E A D E R ;
IDENTIFIED : I D E N T I F I E D ;
IGNORE : I G N O R E ;
ISOLATION : I S O L A T I O N ;
IMPORT : I M P O R T ;
INACTIVE : I N A C T I V E ;
IN_MEMORY_ANALYTICAL : I N UNDERSCORE M E M O R Y UNDERSCORE A N A L Y T I C A L ;
IN_MEMORY_TRANSACTIONAL : I N UNDERSCORE M E M O R Y UNDERSCORE T R A N S A C T I O N A L ;
ISOLATION : I S O L A T I O N ;
KAFKA : K A F K A ;
LABELS : L A B E L S ;
LEVEL : L E V E L ;

View File

@@ -87,6 +87,8 @@ class PrivilegeExtractor : public QueryVisitor<void>, public HierarchicalTreeVis
void Visit(TransactionQueueQuery & /*transaction_queue_query*/) override {}
void Visit(EdgeImportModeQuery & /*edge_import_mode_query*/) override {}
void Visit(VersionQuery & /*version_query*/) override { AddPrivilege(AuthQuery::Privilege::STATS); }
void Visit(MultiDatabaseQuery &query) override {

View File

@@ -400,6 +400,29 @@ SymbolGenerator::ReturnType SymbolGenerator::Visit(Identifier &ident) {
return true;
}
bool SymbolGenerator::PostVisit(MapLiteral &map_literal) {
std::unordered_map<int32_t, PropertyLookup *> property_lookups{};
for (const auto &pair : map_literal.elements_) {
if (pair.second->GetTypeInfo() != PropertyLookup::kType) continue;
auto *property_lookup = static_cast<PropertyLookup *>(pair.second);
if (property_lookup->expression_->GetTypeInfo() != Identifier::kType) continue;
auto symbol_pos = static_cast<Identifier *>(property_lookup->expression_)->symbol_pos_;
try {
auto *existing_property_lookup = property_lookups.at(symbol_pos);
// If already there (no exception), update the original and current PropertyLookups
existing_property_lookup->evaluation_mode_ = PropertyLookup::EvaluationMode::GET_ALL_PROPERTIES;
property_lookup->evaluation_mode_ = PropertyLookup::EvaluationMode::GET_ALL_PROPERTIES;
} catch (const std::out_of_range &) {
// Otherwise, add the PropertyLookup to the map
property_lookups.emplace(symbol_pos, property_lookup);
}
}
return true;
}
bool SymbolGenerator::PreVisit(Aggregation &aggr) {
auto &scope = scopes_.back();
// Check if the aggregation can be used in this context. This check should

View File

@@ -72,6 +72,8 @@ class SymbolGenerator : public HierarchicalTreeVisitor {
// Expressions
ReturnType Visit(Identifier &) override;
ReturnType Visit(PrimitiveLiteral &) override { return true; }
bool PreVisit(MapLiteral &) override { return true; }
bool PostVisit(MapLiteral &) override;
ReturnType Visit(ParameterLookup &) override { return true; }
bool PreVisit(Aggregation &) override;
bool PostVisit(Aggregation &) override;

View File

@@ -218,7 +218,7 @@ const trie::Trie kKeywords = {"union",
"data",
"directory",
"lock",
"unlock"
"unlock",
"build"};
// Unicode codepoints that are allowed at the start of the unescaped name.

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -22,9 +22,10 @@ int64_t EvaluateInt(ExpressionEvaluator *evaluator, Expression *expr, const std:
}
}
std::optional<size_t> EvaluateMemoryLimit(ExpressionEvaluator *eval, Expression *memory_limit, size_t memory_scale) {
std::optional<size_t> EvaluateMemoryLimit(ExpressionVisitor<TypedValue> &eval, Expression *memory_limit,
size_t memory_scale) {
if (!memory_limit) return std::nullopt;
auto limit_value = memory_limit->Accept(*eval);
auto limit_value = memory_limit->Accept(eval);
if (!limit_value.IsInt() || limit_value.ValueInt() <= 0)
throw QueryRuntimeException("Memory limit must be a non-negative integer.");
size_t limit = limit_value.ValueInt();

View File

@@ -99,12 +99,79 @@ class ReferenceExpressionEvaluator : public ExpressionVisitor<TypedValue *> {
UNSUCCESSFUL_VISIT(RegexMatch);
UNSUCCESSFUL_VISIT(Exists);
#undef UNSUCCESSFUL_VISIT
private:
Frame *frame_;
const SymbolTable *symbol_table_;
const EvaluationContext *ctx_;
};
class PrimitiveLiteralExpressionEvaluator : public ExpressionVisitor<TypedValue> {
public:
explicit PrimitiveLiteralExpressionEvaluator(EvaluationContext const &ctx) : ctx_(&ctx) {}
using ExpressionVisitor<TypedValue>::Visit;
TypedValue Visit(PrimitiveLiteral &literal) override {
// TODO: no need to evaluate constants, we can write it to frame in one
// of the previous phases.
return TypedValue(literal.value_, ctx_->memory);
}
TypedValue Visit(ParameterLookup &param_lookup) override {
return TypedValue(ctx_->parameters.AtTokenPosition(param_lookup.token_position_), ctx_->memory);
}
#define INVALID_VISIT(expr_name) \
TypedValue Visit(expr_name & /*expr*/) override { \
DLOG_FATAL("Invalid expression type visited with PrimitiveLiteralExpressionEvaluator."); \
return {}; \
}
INVALID_VISIT(NamedExpression)
INVALID_VISIT(OrOperator)
INVALID_VISIT(XorOperator)
INVALID_VISIT(AndOperator)
INVALID_VISIT(NotOperator)
INVALID_VISIT(AdditionOperator)
INVALID_VISIT(SubtractionOperator)
INVALID_VISIT(MultiplicationOperator)
INVALID_VISIT(DivisionOperator)
INVALID_VISIT(ModOperator)
INVALID_VISIT(NotEqualOperator)
INVALID_VISIT(EqualOperator)
INVALID_VISIT(LessOperator)
INVALID_VISIT(GreaterOperator)
INVALID_VISIT(LessEqualOperator)
INVALID_VISIT(GreaterEqualOperator)
INVALID_VISIT(InListOperator)
INVALID_VISIT(SubscriptOperator)
INVALID_VISIT(ListSlicingOperator)
INVALID_VISIT(IfOperator)
INVALID_VISIT(UnaryPlusOperator)
INVALID_VISIT(UnaryMinusOperator)
INVALID_VISIT(IsNullOperator)
INVALID_VISIT(ListLiteral)
INVALID_VISIT(MapLiteral)
INVALID_VISIT(MapProjectionLiteral)
INVALID_VISIT(PropertyLookup)
INVALID_VISIT(AllPropertiesLookup)
INVALID_VISIT(LabelsTest)
INVALID_VISIT(Aggregation)
INVALID_VISIT(Function)
INVALID_VISIT(Reduce)
INVALID_VISIT(Coalesce)
INVALID_VISIT(Extract)
INVALID_VISIT(All)
INVALID_VISIT(Single)
INVALID_VISIT(Any)
INVALID_VISIT(None)
INVALID_VISIT(Identifier)
INVALID_VISIT(RegexMatch)
INVALID_VISIT(Exists)
#undef INVALID_VISIT
private:
EvaluationContext const *ctx_;
};
class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
public:
ExpressionEvaluator(Frame *frame, const SymbolTable &symbol_table, const EvaluationContext &ctx, DbAccessor *dba,
@@ -479,9 +546,35 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
case TypedValue::Type::Null:
return TypedValue(ctx_->memory);
case TypedValue::Type::Vertex:
return TypedValue(GetProperty(expression_result_ptr->ValueVertex(), property_lookup.property_), ctx_->memory);
if (property_lookup.evaluation_mode_ == PropertyLookup::EvaluationMode::GET_ALL_PROPERTIES) {
auto symbol_pos = static_cast<Identifier *>(property_lookup.expression_)->symbol_pos_;
if (!ctx_->property_lookups_cache.contains(symbol_pos)) {
ctx_->property_lookups_cache.emplace(symbol_pos, GetAllProperties(expression_result_ptr->ValueVertex()));
}
auto property_id = ctx_->properties[property_lookup.property_.ix];
if (ctx_->property_lookups_cache[symbol_pos].contains(property_id)) {
return TypedValue(ctx_->property_lookups_cache[symbol_pos][property_id], ctx_->memory);
}
return TypedValue(ctx_->memory);
} else {
return TypedValue(GetProperty(expression_result_ptr->ValueVertex(), property_lookup.property_), ctx_->memory);
}
case TypedValue::Type::Edge:
return TypedValue(GetProperty(expression_result_ptr->ValueEdge(), property_lookup.property_), ctx_->memory);
if (property_lookup.evaluation_mode_ == PropertyLookup::EvaluationMode::GET_ALL_PROPERTIES) {
auto symbol_pos = static_cast<Identifier *>(property_lookup.expression_)->symbol_pos_;
if (!ctx_->property_lookups_cache.contains(symbol_pos)) {
ctx_->property_lookups_cache.emplace(symbol_pos, GetAllProperties(expression_result_ptr->ValueEdge()));
}
auto property_id = ctx_->properties[property_lookup.property_.ix];
if (ctx_->property_lookups_cache[symbol_pos].contains(property_id)) {
return TypedValue(ctx_->property_lookups_cache[symbol_pos][property_id], ctx_->memory);
}
return TypedValue(ctx_->memory);
} else {
return TypedValue(GetProperty(expression_result_ptr->ValueEdge(), property_lookup.property_), ctx_->memory);
}
case TypedValue::Type::Map: {
auto &map = expression_result_ptr->ValueMap();
auto found = map.find(property_lookup.property_.name.c_str());
@@ -687,7 +780,14 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
TypedValue Visit(MapLiteral &literal) override {
TypedValue::TMap result(ctx_->memory);
for (const auto &pair : literal.elements_) result.emplace(pair.first.name, pair.second->Accept(*this));
for (const auto &pair : literal.elements_) {
result.emplace(pair.first.name, pair.second->Accept(*this));
}
ctx_->property_lookups_cache.clear();
// TODO Dont clear the cache if there are remaining MapLiterals with PropertyLookups that read the same properties
// from the same variable (symbol & value)
return TypedValue(result, ctx_->memory);
}
@@ -696,20 +796,27 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
TypedValue::TMap result(ctx_->memory);
TypedValue::TMap all_properties_lookup(ctx_->memory);
auto map_variable = literal.map_variable_->Accept(*this);
if (map_variable.IsNull()) {
return TypedValue(ctx_->memory);
}
for (const auto &[property_key, property_value] : literal.elements_) {
if (property_key.name == kAllPropertiesSelector.data()) {
auto maybe_all_properties_lookup = property_value->Accept(*this);
if (maybe_all_properties_lookup.type() != TypedValue::Type::Map) {
throw QueryRuntimeException("Expected a map from AllPropertiesLookup, got {}.",
maybe_all_properties_lookup.type());
LOG_FATAL("Expected a map from AllPropertiesLookup, got {}.", maybe_all_properties_lookup.type());
}
all_properties_lookup = std::move(maybe_all_properties_lookup.ValueMap());
continue;
}
result.emplace(property_key.name, property_value->Accept(*this));
}
if (!all_properties_lookup.empty()) result.merge(all_properties_lookup);
return TypedValue(result, ctx_->memory);
@@ -974,6 +1081,33 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
}
private:
template <class TRecordAccessor>
std::map<storage::PropertyId, storage::PropertyValue> GetAllProperties(const TRecordAccessor &record_accessor) {
auto maybe_props = record_accessor.Properties(view_);
if (maybe_props.HasError() && maybe_props.GetError() == storage::Error::NONEXISTENT_OBJECT) {
// This is a very nasty and temporary hack in order to make MERGE work.
// The old storage had the following logic when returning an `OLD` view:
// `return old ? old : new`. That means that if the `OLD` view didn't
// exist, it returned the NEW view. With this hack we simulate that
// behavior.
// TODO (mferencevic, teon.banek): Remove once MERGE is reimplemented.
maybe_props = record_accessor.Properties(storage::View::NEW);
}
if (maybe_props.HasError()) {
switch (maybe_props.GetError()) {
case storage::Error::DELETED_OBJECT:
throw QueryRuntimeException("Trying to get properties from a deleted object.");
case storage::Error::NONEXISTENT_OBJECT:
throw query::QueryRuntimeException("Trying to get properties from an object that doesn't exist.");
case storage::Error::SERIALIZATION_ERROR:
case storage::Error::VERTEX_HAS_EDGES:
case storage::Error::PROPERTIES_DISABLED:
throw QueryRuntimeException("Unexpected error when getting properties.");
}
}
return *maybe_props;
}
template <class TRecordAccessor>
storage::PropertyValue GetProperty(const TRecordAccessor &record_accessor, PropertyIx prop) {
auto maybe_prop = record_accessor.GetProperty(view_, ctx_->properties[prop.ix]);
@@ -1046,6 +1180,7 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
/// @throw QueryRuntimeException if expression doesn't evaluate to an int.
int64_t EvaluateInt(ExpressionEvaluator *evaluator, Expression *expr, const std::string &what);
std::optional<size_t> EvaluateMemoryLimit(ExpressionEvaluator *eval, Expression *memory_limit, size_t memory_scale);
std::optional<size_t> EvaluateMemoryLimit(ExpressionVisitor<TypedValue> &eval, Expression *memory_limit,
size_t memory_scale);
} // namespace memgraph::query

View File

@@ -35,6 +35,7 @@
#include "csv/parsing.hpp"
#include "dbms/global.hpp"
#include "dbms/session_context_handler.hpp"
#include "flags/run_time_configurable.hpp"
#include "glue/communication.hpp"
#include "license/license.hpp"
#include "memory/memory_control.hpp"
@@ -64,6 +65,7 @@
#include "spdlog/spdlog.h"
#include "storage/v2/disk/storage.hpp"
#include "storage/v2/edge.hpp"
#include "storage/v2/edge_import_mode.hpp"
#include "storage/v2/id_types.hpp"
#include "storage/v2/inmemory/storage.hpp"
#include "storage/v2/property_value.hpp"
@@ -162,12 +164,12 @@ struct Callback {
bool should_abort_query{false};
};
TypedValue EvaluateOptionalExpression(Expression *expression, ExpressionEvaluator *eval) {
return expression ? expression->Accept(*eval) : TypedValue();
TypedValue EvaluateOptionalExpression(Expression *expression, ExpressionVisitor<TypedValue> &eval) {
return expression ? expression->Accept(eval) : TypedValue();
}
template <typename TResult>
std::optional<TResult> GetOptionalValue(query::Expression *expression, ExpressionEvaluator &evaluator) {
std::optional<TResult> GetOptionalValue(query::Expression *expression, ExpressionVisitor<TypedValue> &evaluator) {
if (expression != nullptr) {
auto int_value = expression->Accept(evaluator);
MG_ASSERT(int_value.IsNull() || int_value.IsInt());
@@ -178,7 +180,8 @@ std::optional<TResult> GetOptionalValue(query::Expression *expression, Expressio
return {};
};
std::optional<std::string> GetOptionalStringValue(query::Expression *expression, ExpressionEvaluator &evaluator) {
std::optional<std::string> GetOptionalStringValue(query::Expression *expression,
ExpressionVisitor<TypedValue> &evaluator) {
if (expression != nullptr) {
auto value = expression->Accept(evaluator);
MG_ASSERT(value.IsNull() || value.IsString());
@@ -216,17 +219,15 @@ class ReplQueryHandler final : public query::ReplicationQueryHandler {
/// @throw QueryRuntimeException if an error ocurred.
void SetReplicationRole(ReplicationQuery::ReplicationRole replication_role, std::optional<int64_t> port) override {
auto *mem_storage = static_cast<storage::InMemoryStorage *>(db_);
if (replication_role == ReplicationQuery::ReplicationRole::MAIN) {
if (!mem_storage->SetMainReplicationRole()) {
if (!db_->SetMainReplicationRole()) {
throw QueryRuntimeException("Couldn't set role to main!");
}
}
if (replication_role == ReplicationQuery::ReplicationRole::REPLICA) {
} else {
if (!port || *port < 0 || *port > std::numeric_limits<uint16_t>::max()) {
throw QueryRuntimeException("Port number invalid!");
}
if (!mem_storage->SetReplicaRole(
if (!db_->SetReplicaRole(
io::network::Endpoint(storage::replication::kDefaultReplicationServerIp, static_cast<uint16_t>(*port)),
storage::replication::ReplicationServerConfig{})) {
throw QueryRuntimeException("Couldn't set role to replica!");
@@ -236,7 +237,7 @@ class ReplQueryHandler final : public query::ReplicationQueryHandler {
/// @throw QueryRuntimeException if an error ocurred.
ReplicationQuery::ReplicationRole ShowReplicationRole() const override {
switch (static_cast<storage::InMemoryStorage *>(db_)->GetReplicationRole()) {
switch (db_->GetReplicationRole()) {
case storage::replication::ReplicationRole::MAIN:
return ReplicationQuery::ReplicationRole::MAIN;
case storage::replication::ReplicationRole::REPLICA:
@@ -249,8 +250,7 @@ class ReplQueryHandler final : public query::ReplicationQueryHandler {
void RegisterReplica(const std::string &name, const std::string &socket_address,
const ReplicationQuery::SyncMode sync_mode,
const std::chrono::seconds replica_check_frequency) override {
auto *mem_storage = static_cast<storage::InMemoryStorage *>(db_);
if (mem_storage->GetReplicationRole() == storage::replication::ReplicationRole::REPLICA) {
if (db_->GetReplicationRole() == storage::replication::ReplicationRole::REPLICA) {
// replica can't register another replica
throw QueryRuntimeException("Replica can't register another replica!");
}
@@ -275,9 +275,9 @@ class ReplQueryHandler final : public query::ReplicationQueryHandler {
io::network::Endpoint::ParseSocketOrIpAddress(socket_address, storage::replication::kDefaultReplicationPort);
if (maybe_ip_and_port) {
auto [ip, port] = *maybe_ip_and_port;
auto ret = mem_storage->RegisterReplica(
name, {std::move(ip), port}, repl_mode, storage::replication::RegistrationMode::MUST_BE_INSTANTLY_VALID,
{.replica_check_frequency = replica_check_frequency, .ssl = std::nullopt});
auto ret = db_->RegisterReplica(name, {std::move(ip), port}, repl_mode,
storage::replication::RegistrationMode::MUST_BE_INSTANTLY_VALID,
{.replica_check_frequency = replica_check_frequency, .ssl = std::nullopt});
if (ret.HasError()) {
throw QueryRuntimeException(fmt::format("Couldn't register replica '{}'!", name));
}
@@ -288,26 +288,23 @@ class ReplQueryHandler final : public query::ReplicationQueryHandler {
/// @throw QueryRuntimeException if an error ocurred.
void DropReplica(const std::string &replica_name) override {
auto *mem_storage = static_cast<storage::InMemoryStorage *>(db_);
if (mem_storage->GetReplicationRole() == storage::replication::ReplicationRole::REPLICA) {
if (db_->GetReplicationRole() == storage::replication::ReplicationRole::REPLICA) {
// replica can't unregister a replica
throw QueryRuntimeException("Replica can't unregister a replica!");
}
if (!mem_storage->UnregisterReplica(replica_name)) {
if (!db_->UnregisterReplica(replica_name)) {
throw QueryRuntimeException(fmt::format("Couldn't unregister the replica '{}'", replica_name));
}
}
using Replica = ReplicationQueryHandler::Replica;
std::vector<Replica> ShowReplicas() const override {
auto *mem_storage = static_cast<storage::InMemoryStorage *>(db_);
if (mem_storage->GetReplicationRole() == storage::replication::ReplicationRole::REPLICA) {
if (db_->GetReplicationRole() == storage::replication::ReplicationRole::REPLICA) {
// replica can't show registered replicas (it shouldn't have any)
throw QueryRuntimeException("Replica can't show registered replicas (it shouldn't have any)!");
}
auto repl_infos = mem_storage->ReplicasInfo();
auto repl_infos = db_->ReplicasInfo();
std::vector<Replica> replicas;
replicas.reserve(repl_infos.size());
@@ -357,24 +354,17 @@ class ReplQueryHandler final : public query::ReplicationQueryHandler {
/// returns false if the replication role can't be set
/// @throw QueryRuntimeException if an error ocurred.
Callback HandleAuthQuery(AuthQuery *auth_query, InterpreterContext *interpreter_context, const Parameters &parameters,
DbAccessor *db_accessor) {
Callback HandleAuthQuery(AuthQuery *auth_query, InterpreterContext *interpreter_context, const Parameters &parameters) {
AuthQueryHandler *auth = interpreter_context->auth;
#ifdef MG_ENTERPRISE
auto &sc_handler = memgraph::dbms::SessionContextHandler::ExtractSCH(interpreter_context);
#endif
// Empty frame for evaluation of password expression. This is OK since
// password should be either null or string literal and it's evaluation
// should not depend on frame.
Frame frame(0);
SymbolTable symbol_table;
EvaluationContext evaluation_context;
// TODO: MemoryResource for EvaluationContext, it should probably be passed as
// the argument to Callback.
EvaluationContext evaluation_context;
evaluation_context.timestamp = QueryTimestamp();
evaluation_context.parameters = parameters;
ExpressionEvaluator evaluator(&frame, symbol_table, evaluation_context, db_accessor, storage::View::OLD);
auto evaluator = PrimitiveLiteralExpressionEvaluator{evaluation_context};
std::string username = auth_query->user_;
std::string rolename = auth_query->role_;
@@ -387,7 +377,7 @@ Callback HandleAuthQuery(AuthQuery *auth_query, InterpreterContext *interpreter_
std::vector<std::unordered_map<AuthQuery::FineGrainedPrivilege, std::vector<std::string>>> edge_type_privileges =
auth_query->edge_type_privileges_;
#endif
auto password = EvaluateOptionalExpression(auth_query->password_, &evaluator);
auto password = EvaluateOptionalExpression(auth_query->password_, evaluator);
Callback callback;
@@ -647,21 +637,18 @@ Callback HandleAuthQuery(AuthQuery *auth_query, InterpreterContext *interpreter_
} // namespace
Callback HandleReplicationQuery(ReplicationQuery *repl_query, const Parameters &parameters,
InterpreterContext *interpreter_context, DbAccessor *db_accessor,
std::vector<Notification> *notifications) {
Frame frame(0);
SymbolTable symbol_table;
EvaluationContext evaluation_context;
InterpreterContext *interpreter_context, std::vector<Notification> *notifications) {
// TODO: MemoryResource for EvaluationContext, it should probably be passed as
// the argument to Callback.
EvaluationContext evaluation_context;
evaluation_context.timestamp = QueryTimestamp();
evaluation_context.parameters = parameters;
ExpressionEvaluator evaluator(&frame, symbol_table, evaluation_context, db_accessor, storage::View::OLD);
auto evaluator = PrimitiveLiteralExpressionEvaluator{evaluation_context};
Callback callback;
switch (repl_query->action_) {
case ReplicationQuery::Action::SET_REPLICATION_ROLE: {
auto port = EvaluateOptionalExpression(repl_query->port_, &evaluator);
auto port = EvaluateOptionalExpression(repl_query->port_, evaluator);
std::optional<int64_t> maybe_port;
if (port.IsInt()) {
maybe_port = port.ValueInt();
@@ -780,7 +767,7 @@ std::optional<std::string> StringPointerToOptional(const std::string *str) {
return str == nullptr ? std::nullopt : std::make_optional(*str);
}
stream::CommonStreamInfo GetCommonStreamInfo(StreamQuery *stream_query, ExpressionEvaluator &evaluator) {
stream::CommonStreamInfo GetCommonStreamInfo(StreamQuery *stream_query, ExpressionVisitor<TypedValue> &evaluator) {
return {
.batch_interval = GetOptionalValue<std::chrono::milliseconds>(stream_query->batch_interval_, evaluator)
.value_or(stream::kDefaultBatchInterval),
@@ -788,7 +775,7 @@ stream::CommonStreamInfo GetCommonStreamInfo(StreamQuery *stream_query, Expressi
.transformation_name = stream_query->transform_name_};
}
std::vector<std::string> EvaluateTopicNames(ExpressionEvaluator &evaluator,
std::vector<std::string> EvaluateTopicNames(ExpressionVisitor<TypedValue> &evaluator,
std::variant<Expression *, std::vector<std::string>> topic_variant) {
return std::visit(utils::Overloaded{[&](Expression *expression) {
auto topic_names = expression->Accept(evaluator);
@@ -799,7 +786,7 @@ std::vector<std::string> EvaluateTopicNames(ExpressionEvaluator &evaluator,
std::move(topic_variant));
}
Callback::CallbackFunction GetKafkaCreateCallback(StreamQuery *stream_query, ExpressionEvaluator &evaluator,
Callback::CallbackFunction GetKafkaCreateCallback(StreamQuery *stream_query, ExpressionVisitor<TypedValue> &evaluator,
InterpreterContext *interpreter_context,
const std::string *username) {
static constexpr std::string_view kDefaultConsumerGroup = "mg_consumer";
@@ -850,7 +837,7 @@ Callback::CallbackFunction GetKafkaCreateCallback(StreamQuery *stream_query, Exp
};
}
Callback::CallbackFunction GetPulsarCreateCallback(StreamQuery *stream_query, ExpressionEvaluator &evaluator,
Callback::CallbackFunction GetPulsarCreateCallback(StreamQuery *stream_query, ExpressionVisitor<TypedValue> &evaluator,
InterpreterContext *interpreter_context,
const std::string *username) {
auto service_url = GetOptionalStringValue(stream_query->service_url_, evaluator);
@@ -876,16 +863,14 @@ Callback::CallbackFunction GetPulsarCreateCallback(StreamQuery *stream_query, Ex
}
Callback HandleStreamQuery(StreamQuery *stream_query, const Parameters &parameters,
InterpreterContext *interpreter_context, DbAccessor *db_accessor,
const std::string *username, std::vector<Notification> *notifications) {
Frame frame(0);
SymbolTable symbol_table;
EvaluationContext evaluation_context;
InterpreterContext *interpreter_context, const std::string *username,
std::vector<Notification> *notifications) {
// TODO: MemoryResource for EvaluationContext, it should probably be passed as
// the argument to Callback.
EvaluationContext evaluation_context;
evaluation_context.timestamp = QueryTimestamp();
evaluation_context.parameters = parameters;
ExpressionEvaluator evaluator(&frame, symbol_table, evaluation_context, db_accessor, storage::View::OLD);
PrimitiveLiteralExpressionEvaluator evaluator{evaluation_context};
Callback callback;
switch (stream_query->action_) {
@@ -1041,27 +1026,23 @@ Callback HandleConfigQuery() {
return callback;
}
Callback HandleSettingQuery(SettingQuery *setting_query, const Parameters &parameters, DbAccessor *db_accessor) {
Frame frame(0);
SymbolTable symbol_table;
EvaluationContext evaluation_context;
Callback HandleSettingQuery(SettingQuery *setting_query, const Parameters &parameters) {
// TODO: MemoryResource for EvaluationContext, it should probably be passed as
// the argument to Callback.
evaluation_context.timestamp =
std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
.count();
EvaluationContext evaluation_context;
evaluation_context.timestamp = QueryTimestamp();
evaluation_context.parameters = parameters;
ExpressionEvaluator evaluator(&frame, symbol_table, evaluation_context, db_accessor, storage::View::OLD);
auto evaluator = PrimitiveLiteralExpressionEvaluator{evaluation_context};
Callback callback;
switch (setting_query->action_) {
case SettingQuery::Action::SET_SETTING: {
const auto setting_name = EvaluateOptionalExpression(setting_query->setting_name_, &evaluator);
const auto setting_name = EvaluateOptionalExpression(setting_query->setting_name_, evaluator);
if (!setting_name.IsString()) {
throw utils::BasicException("Setting name should be a string literal");
}
const auto setting_value = EvaluateOptionalExpression(setting_query->setting_value_, &evaluator);
const auto setting_value = EvaluateOptionalExpression(setting_query->setting_value_, evaluator);
if (!setting_value.IsString()) {
throw utils::BasicException("Setting value should be a string literal");
}
@@ -1076,7 +1057,7 @@ Callback HandleSettingQuery(SettingQuery *setting_query, const Parameters &param
return callback;
}
case SettingQuery::Action::SHOW_SETTING: {
const auto setting_name = EvaluateOptionalExpression(setting_query->setting_name_, &evaluator);
const auto setting_name = EvaluateOptionalExpression(setting_query->setting_name_, evaluator);
if (!setting_name.IsString()) {
throw utils::BasicException("Setting name should be a string literal");
}
@@ -1349,8 +1330,7 @@ bool IsWriteQueryOnMainMemoryReplica(storage::Storage *storage,
const query::plan::ReadWriteTypeChecker::RWType query_type) {
if (auto storage_mode = storage->GetStorageMode(); storage_mode == storage::StorageMode::IN_MEMORY_ANALYTICAL ||
storage_mode == storage::StorageMode::IN_MEMORY_TRANSACTIONAL) {
auto *mem_storage = static_cast<storage::InMemoryStorage *>(storage);
return (mem_storage->GetReplicationRole() == storage::replication::ReplicationRole::REPLICA) &&
return (storage->GetReplicationRole() == storage::replication::ReplicationRole::REPLICA) &&
(query_type == RWType::W || query_type == RWType::RW);
}
return false;
@@ -1359,8 +1339,7 @@ bool IsWriteQueryOnMainMemoryReplica(storage::Storage *storage,
storage::replication::ReplicationRole GetReplicaRole(storage::Storage *storage) {
if (auto storage_mode = storage->GetStorageMode(); storage_mode == storage::StorageMode::IN_MEMORY_ANALYTICAL ||
storage_mode == storage::StorageMode::IN_MEMORY_TRANSACTIONAL) {
auto *mem_storage = static_cast<storage::InMemoryStorage *>(storage);
return mem_storage->GetReplicationRole();
return storage->GetReplicationRole();
}
return storage::replication::ReplicationRole::MAIN;
}
@@ -1382,7 +1361,7 @@ InterpreterContext::InterpreterContext(const storage::Config storage_config, con
}
}
InterpreterContext::InterpreterContext(std::unique_ptr<storage::Storage> db, InterpreterConfig interpreter_config,
InterpreterContext::InterpreterContext(std::unique_ptr<storage::Storage> &&db, InterpreterConfig interpreter_config,
const std::filesystem::path &data_directory, query::AuthQueryHandler *ah,
query::AuthChecker *ac)
: db(std::move(db)),
@@ -1399,7 +1378,7 @@ Interpreter::Interpreter(InterpreterContext *interpreter_context) : interpreter_
auto DetermineTxTimeout(std::optional<int64_t> tx_timeout_ms, InterpreterConfig const &config) -> TxTimeout {
using double_seconds = std::chrono::duration<double>;
auto const global_tx_timeout = double_seconds{config.execution_timeout_sec};
auto const global_tx_timeout = double_seconds{flags::run_time::execution_timeout_sec_};
auto const valid_global_tx_timeout = global_tx_timeout > double_seconds{0};
if (tx_timeout_ms) {
@@ -1541,14 +1520,12 @@ PreparedQuery PrepareCypherQuery(ParsedQuery parsed_query, std::map<std::string,
FrameChangeCollector *frame_change_collector = nullptr) {
auto *cypher_query = utils::Downcast<CypherQuery>(parsed_query.query);
Frame frame(0);
SymbolTable symbol_table;
EvaluationContext evaluation_context;
evaluation_context.timestamp = QueryTimestamp();
evaluation_context.parameters = parsed_query.parameters;
auto evaluator = PrimitiveLiteralExpressionEvaluator{evaluation_context};
ExpressionEvaluator evaluator(&frame, symbol_table, evaluation_context, dba, storage::View::OLD);
const auto memory_limit = EvaluateMemoryLimit(&evaluator, cypher_query->memory_limit_, cypher_query->memory_scale_);
const auto memory_limit = EvaluateMemoryLimit(evaluator, cypher_query->memory_limit_, cypher_query->memory_scale_);
if (memory_limit) {
spdlog::info("Running query with memory limit of {}", utils::GetReadableSize(*memory_limit));
}
@@ -1709,13 +1686,11 @@ PreparedQuery PrepareProfileQuery(ParsedQuery parsed_query, bool in_explicit_tra
!contains_csv && !IsCallBatchedProcedureQuery(clauses) && !IsAllShortestPathsQuery(clauses);
MG_ASSERT(cypher_query, "Cypher grammar should not allow other queries in PROFILE");
Frame frame(0);
SymbolTable symbol_table;
EvaluationContext evaluation_context;
evaluation_context.timestamp = QueryTimestamp();
evaluation_context.parameters = parsed_inner_query.parameters;
ExpressionEvaluator evaluator(&frame, symbol_table, evaluation_context, dba, storage::View::OLD);
const auto memory_limit = EvaluateMemoryLimit(&evaluator, cypher_query->memory_limit_, cypher_query->memory_scale_);
auto evaluator = PrimitiveLiteralExpressionEvaluator{evaluation_context};
const auto memory_limit = EvaluateMemoryLimit(evaluator, cypher_query->memory_limit_, cypher_query->memory_scale_);
auto cypher_query_plan = CypherQueryToPlan(
parsed_inner_query.stripped_query.hash(), std::move(parsed_inner_query.ast_storage), cypher_query,
@@ -2139,38 +2114,31 @@ PreparedQuery PrepareIndexQuery(ParsedQuery parsed_query, bool in_explicit_trans
}
PreparedQuery PrepareAuthQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
std::map<std::string, TypedValue> *summary, InterpreterContext *interpreter_context,
DbAccessor *dba, utils::MemoryResource *execution_memory, const std::string *username,
std::atomic<TransactionStatus> *transaction_status,
std::shared_ptr<utils::AsyncTimer> tx_timer) {
InterpreterContext *interpreter_context) {
if (in_explicit_transaction) {
throw UserModificationInMulticommandTxException();
}
auto *auth_query = utils::Downcast<AuthQuery>(parsed_query.query);
auto callback = HandleAuthQuery(auth_query, interpreter_context, parsed_query.parameters, dba);
auto callback = HandleAuthQuery(auth_query, interpreter_context, parsed_query.parameters);
SymbolTable symbol_table;
std::vector<Symbol> output_symbols;
for (const auto &column : callback.header) {
output_symbols.emplace_back(symbol_table.CreateSymbol(column, "false"));
}
auto plan = std::make_shared<CachedPlan>(std::make_unique<SingleNodeLogicalPlan>(
std::make_unique<plan::OutputTable>(output_symbols,
[fn = callback.fn](Frame *, ExecutionContext *) { return fn(); }),
0.0, AstStorage{}, symbol_table));
auto pull_plan =
std::make_shared<PullPlan>(plan, parsed_query.parameters, false, dba, interpreter_context, execution_memory,
StringPointerToOptional(username), transaction_status, std::move(tx_timer));
return PreparedQuery{
callback.header, std::move(parsed_query.required_privileges),
[pull_plan = std::move(pull_plan), callback = std::move(callback), output_symbols = std::move(output_symbols),
summary](AnyStream *stream, std::optional<int> n) -> std::optional<QueryHandlerResult> {
if (pull_plan->Pull(stream, n, output_symbols, summary)) {
return callback.should_abort_query ? QueryHandlerResult::ABORT : QueryHandlerResult::COMMIT;
std::move(callback.header), std::move(parsed_query.required_privileges),
[handler = std::move(callback.fn), pull_plan = std::shared_ptr<PullPlanVector>(nullptr), interpreter_context](
AnyStream *stream, std::optional<int> n) mutable -> std::optional<QueryHandlerResult> {
if (!pull_plan) {
// Run the specific query
auto results = handler();
pull_plan = std::make_shared<PullPlanVector>(std::move(results));
#ifdef MG_ENTERPRISE
// Invalidate auth cache after every type of AuthQuery
interpreter_context->auth_checker->ClearCache();
#endif
}
if (pull_plan->Pull(stream, n)) {
return QueryHandlerResult::COMMIT;
}
return std::nullopt;
},
@@ -2178,8 +2146,8 @@ PreparedQuery PrepareAuthQuery(ParsedQuery parsed_query, bool in_explicit_transa
}
PreparedQuery PrepareReplicationQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
std::vector<Notification> *notifications, InterpreterContext *interpreter_context,
DbAccessor *dba) {
std::vector<Notification> *notifications,
InterpreterContext *interpreter_context) {
if (in_explicit_transaction) {
throw ReplicationModificationInMulticommandTxException();
}
@@ -2190,7 +2158,7 @@ PreparedQuery PrepareReplicationQuery(ParsedQuery parsed_query, bool in_explicit
auto *replication_query = utils::Downcast<ReplicationQuery>(parsed_query.query);
auto callback =
HandleReplicationQuery(replication_query, parsed_query.parameters, interpreter_context, dba, notifications);
HandleReplicationQuery(replication_query, parsed_query.parameters, interpreter_context, notifications);
return PreparedQuery{callback.header, std::move(parsed_query.required_privileges),
[callback_fn = std::move(callback.fn), pull_plan = std::shared_ptr<PullPlanVector>{nullptr}](
@@ -2444,8 +2412,6 @@ PreparedQuery PrepareTriggerQuery(ParsedQuery parsed_query, bool in_explicit_tra
PreparedQuery PrepareStreamQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
std::vector<Notification> *notifications, InterpreterContext *interpreter_context,
DbAccessor *dba,
const std::map<std::string, storage::PropertyValue> & /*user_parameters*/,
const std::string *username) {
if (in_explicit_transaction) {
throw StreamQueryInMulticommandTxException();
@@ -2454,7 +2420,7 @@ PreparedQuery PrepareStreamQuery(ParsedQuery parsed_query, bool in_explicit_tran
auto *stream_query = utils::Downcast<StreamQuery>(parsed_query.query);
MG_ASSERT(stream_query);
auto callback =
HandleStreamQuery(stream_query, parsed_query.parameters, interpreter_context, dba, username, notifications);
HandleStreamQuery(stream_query, parsed_query.parameters, interpreter_context, username, notifications);
return PreparedQuery{std::move(callback.header), std::move(parsed_query.required_privileges),
[callback_fn = std::move(callback.fn), pull_plan = std::shared_ptr<PullPlanVector>{nullptr}](
@@ -2495,6 +2461,13 @@ constexpr auto ToStorageMode(const StorageModeQuery::StorageMode storage_mode) n
}
}
constexpr auto ToEdgeImportMode(const EdgeImportModeQuery::Status status) noexcept {
if (status == EdgeImportModeQuery::Status::ACTIVE) {
return storage::EdgeImportMode::ACTIVE;
}
return storage::EdgeImportMode::INACTIVE;
}
bool SwitchingFromInMemoryToDisk(storage::StorageMode current_mode, storage::StorageMode next_mode) {
return (current_mode == storage::StorageMode::IN_MEMORY_TRANSACTIONAL ||
current_mode == storage::StorageMode::IN_MEMORY_ANALYTICAL) &&
@@ -2638,6 +2611,37 @@ PreparedQuery PrepareStorageModeQuery(ParsedQuery parsed_query, const bool in_ex
RWType::NONE};
}
PreparedQuery PrepareEdgeImportModeQuery(ParsedQuery parsed_query, const bool in_explicit_transaction,
InterpreterContext *interpreter_context) {
if (in_explicit_transaction) {
throw EdgeImportModeModificationInMulticommandTxException();
}
if (interpreter_context->db->GetStorageMode() != storage::StorageMode::ON_DISK_TRANSACTIONAL) {
throw EdgeImportModeQueryDisabledOnDiskStorage();
}
auto *edge_import_mode_query = utils::Downcast<EdgeImportModeQuery>(parsed_query.query);
MG_ASSERT(edge_import_mode_query);
const auto requested_status = ToEdgeImportMode(edge_import_mode_query->status_);
auto callback = [requested_status, interpreter_context]() -> std::function<void()> {
return [interpreter_context, requested_status] {
auto *disk_storage = static_cast<storage::DiskStorage *>(interpreter_context->db.get());
disk_storage->SetEdgeImportMode(requested_status);
};
}();
return PreparedQuery{{},
std::move(parsed_query.required_privileges),
[callback = std::move(callback)](AnyStream * /*stream*/,
std::optional<int> /*n*/) -> std::optional<QueryHandlerResult> {
callback();
return QueryHandlerResult::COMMIT;
},
RWType::NONE};
}
PreparedQuery PrepareCreateSnapshotQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
InterpreterContext *interpreter_context) {
if (in_explicit_transaction) {
@@ -2679,7 +2683,7 @@ PreparedQuery PrepareSettingQuery(ParsedQuery parsed_query, bool in_explicit_tra
auto *setting_query = utils::Downcast<SettingQuery>(parsed_query.query);
MG_ASSERT(setting_query);
auto callback = HandleSettingQuery(setting_query, parsed_query.parameters, dba);
auto callback = HandleSettingQuery(setting_query, parsed_query.parameters);
return PreparedQuery{std::move(callback.header), std::move(parsed_query.required_privileges),
[callback_fn = std::move(callback.fn), pull_plan = std::shared_ptr<PullPlanVector>{nullptr}](
@@ -2781,13 +2785,11 @@ std::vector<std::vector<TypedValue>> TransactionQueueQueryHandler::KillTransacti
Callback HandleTransactionQueueQuery(TransactionQueueQuery *transaction_query,
const std::optional<std::string> &username, const Parameters &parameters,
InterpreterContext *interpreter_context, DbAccessor *db_accessor) {
Frame frame(0);
SymbolTable symbol_table;
InterpreterContext *interpreter_context) {
EvaluationContext evaluation_context;
evaluation_context.timestamp = QueryTimestamp();
evaluation_context.parameters = parameters;
ExpressionEvaluator evaluator(&frame, symbol_table, evaluation_context, db_accessor, storage::View::OLD);
auto evaluator = PrimitiveLiteralExpressionEvaluator{evaluation_context};
bool hasTransactionManagementPrivilege = interpreter_context->auth_checker->IsUserAuthorized(
username, {query::AuthQuery::Privilege::TRANSACTION_MANAGEMENT}, "");
@@ -2837,7 +2839,7 @@ PreparedQuery PrepareTransactionQueueQuery(ParsedQuery parsed_query, const std::
auto *transaction_queue_query = utils::Downcast<TransactionQueueQuery>(parsed_query.query);
MG_ASSERT(transaction_queue_query);
auto callback =
HandleTransactionQueueQuery(transaction_queue_query, username, parsed_query.parameters, interpreter_context, dba);
HandleTransactionQueueQuery(transaction_queue_query, username, parsed_query.parameters, interpreter_context);
return PreparedQuery{std::move(callback.header), std::move(parsed_query.required_privileges),
[callback_fn = std::move(callback.fn), pull_plan = std::shared_ptr<PullPlanVector>{nullptr}](
@@ -2911,17 +2913,37 @@ PreparedQuery PrepareInfoQuery(ParsedQuery parsed_query, bool in_explicit_transa
case InfoQuery::InfoType::INDEX:
header = {"index type", "label", "property"};
handler = [interpreter_context] {
const std::string_view label_index_mark{"label"};
const std::string_view label_property_index_mark{"label+property"};
auto *db = interpreter_context->db.get();
auto info = db->ListAllIndices();
std::vector<std::vector<TypedValue>> results;
results.reserve(info.label.size() + info.label_property.size());
for (const auto &item : info.label) {
results.push_back({TypedValue("label"), TypedValue(db->LabelToName(item)), TypedValue()});
results.push_back({TypedValue(label_index_mark), TypedValue(db->LabelToName(item)), TypedValue()});
}
for (const auto &item : info.label_property) {
results.push_back({TypedValue("label+property"), TypedValue(db->LabelToName(item.first)),
results.push_back({TypedValue(label_property_index_mark), TypedValue(db->LabelToName(item.first)),
TypedValue(db->PropertyToName(item.second))});
}
std::sort(results.begin(), results.end(), [&label_index_mark](const auto &record_1, const auto &record_2) {
const auto type_1 = record_1[0].ValueString();
const auto type_2 = record_2[0].ValueString();
if (type_1 != type_2) {
return type_1 < type_2;
}
const auto label_1 = record_1[1].ValueString();
const auto label_2 = record_2[1].ValueString();
if (type_1 == label_index_mark || label_1 != label_2) {
return label_1 < label_2;
}
return record_1[2].ValueString() < record_2[2].ValueString();
});
return std::pair{results, QueryHandlerResult::NOTHING};
};
break;
@@ -3385,11 +3407,7 @@ PreparedQuery PrepareMultiDatabaseQuery(ParsedQuery parsed_query, bool in_explic
}
PreparedQuery PrepareShowDatabasesQuery(ParsedQuery parsed_query, InterpreterContext *interpreter_context,
const std::string &session_uuid, std::map<std::string, TypedValue> *summary,
DbAccessor *dba, utils::MemoryResource *execution_memory,
const std::optional<std::string> &username,
std::atomic<TransactionStatus> *transaction_status,
std::shared_ptr<utils::AsyncTimer> tx_timer) {
const std::string &session_uuid, const std::optional<std::string> &username) {
#ifdef MG_ENTERPRISE
if (!license::global_license_checker.IsEnterpriseValidFast()) {
throw QueryException("Trying to use enterprise feature without a valid license.");
@@ -3454,26 +3472,17 @@ PreparedQuery PrepareShowDatabasesQuery(ParsedQuery parsed_query, InterpreterCon
return status;
};
SymbolTable symbol_table;
std::vector<Symbol> output_symbols;
for (const auto &column : callback.header) {
output_symbols.emplace_back(symbol_table.CreateSymbol(column, "false"));
}
auto plan = std::make_shared<CachedPlan>(std::make_unique<SingleNodeLogicalPlan>(
std::make_unique<plan::OutputTable>(output_symbols,
[fn = callback.fn](Frame *, ExecutionContext *) { return fn(); }),
0.0, AstStorage{}, symbol_table));
auto pull_plan = std::make_shared<PullPlan>(plan, parsed_query.parameters, false, dba, interpreter_context,
execution_memory, username, transaction_status, std::move(tx_timer));
return PreparedQuery{
callback.header, std::move(parsed_query.required_privileges),
[pull_plan = std::move(pull_plan), callback = std::move(callback), output_symbols = std::move(output_symbols),
summary](AnyStream *stream, std::optional<int> n) -> std::optional<QueryHandlerResult> {
if (pull_plan->Pull(stream, n, output_symbols, summary)) {
return callback.should_abort_query ? QueryHandlerResult::ABORT : QueryHandlerResult::COMMIT;
std::move(callback.header), std::move(parsed_query.required_privileges),
[handler = std::move(callback.fn), pull_plan = std::shared_ptr<PullPlanVector>(nullptr)](
AnyStream *stream, std::optional<int> n) mutable -> std::optional<QueryHandlerResult> {
if (!pull_plan) {
auto results = handler();
pull_plan = std::make_shared<PullPlanVector>(std::move(results));
}
if (pull_plan->Pull(stream, n)) {
return QueryHandlerResult::NOTHING;
}
return std::nullopt;
},
@@ -3650,10 +3659,7 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
prepared_query = PrepareAnalyzeGraphQuery(std::move(parsed_query), in_explicit_transaction_,
&*execution_db_accessor_, interpreter_context_);
} else if (utils::Downcast<AuthQuery>(parsed_query.query)) {
prepared_query = PrepareAuthQuery(std::move(parsed_query), in_explicit_transaction_, &query_execution->summary,
interpreter_context_, &*execution_db_accessor_,
&query_execution->execution_memory_with_exception, username,
&transaction_status_, std::move(current_timer));
prepared_query = PrepareAuthQuery(std::move(parsed_query), in_explicit_transaction_, interpreter_context_);
} else if (utils::Downcast<InfoQuery>(parsed_query.query)) {
prepared_query = PrepareInfoQuery(std::move(parsed_query), in_explicit_transaction_, &query_execution->summary,
interpreter_context_, interpreter_context_->db.get(),
@@ -3663,9 +3669,8 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
prepared_query = PrepareConstraintQuery(std::move(parsed_query), in_explicit_transaction_,
&query_execution->notifications, interpreter_context_);
} else if (utils::Downcast<ReplicationQuery>(parsed_query.query)) {
prepared_query =
PrepareReplicationQuery(std::move(parsed_query), in_explicit_transaction_, &query_execution->notifications,
interpreter_context_, &*execution_db_accessor_);
prepared_query = PrepareReplicationQuery(std::move(parsed_query), in_explicit_transaction_,
&query_execution->notifications, interpreter_context_);
} else if (utils::Downcast<LockPathQuery>(parsed_query.query)) {
prepared_query = PrepareLockPathQuery(std::move(parsed_query), in_explicit_transaction_, interpreter_context_);
} else if (utils::Downcast<FreeMemoryQuery>(parsed_query.query)) {
@@ -3677,9 +3682,8 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
PrepareTriggerQuery(std::move(parsed_query), in_explicit_transaction_, &query_execution->notifications,
interpreter_context_, &*execution_db_accessor_, params, username);
} else if (utils::Downcast<StreamQuery>(parsed_query.query)) {
prepared_query =
PrepareStreamQuery(std::move(parsed_query), in_explicit_transaction_, &query_execution->notifications,
interpreter_context_, &*execution_db_accessor_, params, username);
prepared_query = PrepareStreamQuery(std::move(parsed_query), in_explicit_transaction_,
&query_execution->notifications, interpreter_context_, username);
} else if (utils::Downcast<IsolationLevelQuery>(parsed_query.query)) {
prepared_query =
PrepareIsolationLevelQuery(std::move(parsed_query), in_explicit_transaction_, interpreter_context_, this);
@@ -3699,10 +3703,11 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
prepared_query = PrepareMultiDatabaseQuery(std::move(parsed_query), in_explicit_transaction_, in_explicit_db_,
interpreter_context_, session_uuid);
} else if (utils::Downcast<ShowDatabasesQuery>(parsed_query.query)) {
prepared_query = PrepareShowDatabasesQuery(std::move(parsed_query), interpreter_context_, session_uuid,
&query_execution->summary, &*execution_db_accessor_,
&query_execution->execution_memory_with_exception, username_,
&transaction_status_, std::move(current_timer));
prepared_query =
PrepareShowDatabasesQuery(std::move(parsed_query), interpreter_context_, session_uuid, username_);
} else if (utils::Downcast<EdgeImportModeQuery>(parsed_query.query)) {
prepared_query =
PrepareEdgeImportModeQuery(std::move(parsed_query), in_explicit_transaction_, interpreter_context_);
} else {
LOG_FATAL("Should not get here -- unknown query type!");
}
@@ -3794,7 +3799,7 @@ void RunTriggersIndividually(const utils::SkipList<Trigger> &triggers, Interpret
auto trigger_context = original_trigger_context;
trigger_context.AdaptForAccessor(&db_accessor);
try {
trigger.Execute(&db_accessor, &execution_memory, interpreter_context->config.execution_timeout_sec,
trigger.Execute(&db_accessor, &execution_memory, flags::run_time::execution_timeout_sec_,
&interpreter_context->is_shutting_down, transaction_status, trigger_context,
interpreter_context->auth_checker);
} catch (const utils::BasicException &exception) {
@@ -3901,7 +3906,7 @@ void Interpreter::Commit() {
utils::MonotonicBufferResource execution_memory{kExecutionMemoryBlockSize};
AdvanceCommand();
try {
trigger.Execute(&*execution_db_accessor_, &execution_memory, interpreter_context_->config.execution_timeout_sec,
trigger.Execute(&*execution_db_accessor_, &execution_memory, flags::run_time::execution_timeout_sec_,
&interpreter_context_->is_shutting_down, &transaction_status_, *trigger_context,
interpreter_context_->auth_checker);
} catch (const utils::BasicException &e) {

View File

@@ -245,7 +245,7 @@ struct InterpreterContext {
const std::filesystem::path &data_directory, query::AuthQueryHandler *ah = nullptr,
query::AuthChecker *ac = nullptr);
InterpreterContext(std::unique_ptr<storage::Storage> db, InterpreterConfig interpreter_config,
InterpreterContext(std::unique_ptr<storage::Storage> &&db, InterpreterConfig interpreter_config,
const std::filesystem::path &data_directory, query::AuthQueryHandler *ah = nullptr,
query::AuthChecker *ac = nullptr);

View File

@@ -69,12 +69,18 @@
// macro for the default implementation of LogicalOperator::Accept
// that accepts the visitor and visits it's input_ operator
#define ACCEPT_WITH_INPUT(class_name) \
bool class_name::Accept(HierarchicalLogicalOperatorVisitor &visitor) { \
if (visitor.PreVisit(*this)) { \
input_->Accept(visitor); \
} \
return visitor.PostVisit(*this); \
// NOLINTNEXTLINE
#define ACCEPT_WITH_INPUT(class_name) \
bool class_name::Accept(HierarchicalLogicalOperatorVisitor &visitor) { \
if (visitor.PreVisit(*this)) { \
if (input_ == nullptr) { \
throw QueryRuntimeException( \
"The query couldn't be executed due to the unexpected null value in " #class_name \
" operator. To learn more about operators visit https://memgr.ph/query-operators!"); \
} \
input_->Accept(visitor); \
} \
return visitor.PostVisit(*this); \
}
#define WITHOUT_SINGLE_INPUT(class_name) \
@@ -166,7 +172,10 @@ inline void AbortCheck(ExecutionContext const &context) {
} // namespace
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define SCOPED_PROFILE_OP(name) ScopedProfile profile{ComputeProfilingKey(this), name, &context};
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define SCOPED_PROFILE_OP_BY_REF(ref) ScopedProfile profile{ComputeProfilingKey(this), ref, &context};
bool Once::OnceCursor::Pull(Frame &, ExecutionContext &context) {
SCOPED_PROFILE_OP("Once");
@@ -346,7 +355,7 @@ EdgeAccessor CreateEdge(const EdgeCreationInfo &edge_info, DbAccessor *dba, Vert
} // namespace
bool CreateExpand::CreateExpandCursor::Pull(Frame &frame, ExecutionContext &context) {
SCOPED_PROFILE_OP("CreateExpand");
SCOPED_PROFILE_OP_BY_REF(self_);
if (!input_cursor_->Pull(frame, context)) return false;
@@ -425,20 +434,21 @@ VertexAccessor &CreateExpand::CreateExpandCursor::OtherVertex(Frame &frame, Exec
template <class TVerticesFun>
class ScanAllCursor : public Cursor {
public:
explicit ScanAllCursor(Symbol output_symbol, UniqueCursorPtr input_cursor, storage::View view,
explicit ScanAllCursor(const ScanAll &self, Symbol output_symbol, UniqueCursorPtr input_cursor, storage::View view,
TVerticesFun get_vertices, const char *op_name)
: output_symbol_(output_symbol),
: self_(self),
output_symbol_(output_symbol),
input_cursor_(std::move(input_cursor)),
view_(view),
get_vertices_(std::move(get_vertices)),
op_name_(op_name) {}
bool Pull(Frame &frame, ExecutionContext &context) override {
SCOPED_PROFILE_OP(op_name_);
SCOPED_PROFILE_OP_BY_REF(self_);
AbortCheck(context);
while (!vertices_ || vertices_it_.value() == vertices_.value().end()) {
while (!vertices_ || vertices_it_.value() == vertices_end_it_.value()) {
if (!input_cursor_->Pull(frame, context)) return false;
// We need a getter function, because in case of exhausting a lazy
// iterable, we cannot simply reset it by calling begin().
@@ -449,6 +459,7 @@ class ScanAllCursor : public Cursor {
// vertices _ = get_vertices_(frame, context);
vertices_.emplace(std::move(next_vertices.value()));
vertices_it_.emplace(vertices_.value().begin());
vertices_end_it_.emplace(vertices_.value().end());
}
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker && !FindNextVertex(context)) {
@@ -463,7 +474,7 @@ class ScanAllCursor : public Cursor {
#ifdef MG_ENTERPRISE
bool FindNextVertex(const ExecutionContext &context) {
while (vertices_it_.value() != vertices_.value().end()) {
while (vertices_it_.value() != vertices_end_it_.value()) {
if (context.auth_checker->Has(*vertices_it_.value(), view_,
memgraph::query::AuthQuery::FineGrainedPrivilege::READ)) {
return true;
@@ -480,15 +491,18 @@ class ScanAllCursor : public Cursor {
input_cursor_->Reset();
vertices_ = std::nullopt;
vertices_it_ = std::nullopt;
vertices_end_it_ = std::nullopt;
}
private:
const ScanAll &self_;
const Symbol output_symbol_;
const UniqueCursorPtr input_cursor_;
storage::View view_;
TVerticesFun get_vertices_;
std::optional<typename std::result_of<TVerticesFun(Frame &, ExecutionContext &)>::type::value_type> vertices_;
std::optional<decltype(vertices_.value().begin())> vertices_it_;
std::optional<decltype(vertices_.value().end())> vertices_end_it_;
const char *op_name_;
};
@@ -504,8 +518,8 @@ UniqueCursorPtr ScanAll::MakeCursor(utils::MemoryResource *mem) const {
auto *db = context.db_accessor;
return std::make_optional(db->Vertices(view_));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAll");
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, *this, output_symbol_, input_->MakeCursor(mem),
view_, std::move(vertices), "ScanAll");
}
std::vector<Symbol> ScanAll::ModifiedSymbols(const SymbolTable &table) const {
@@ -527,8 +541,8 @@ UniqueCursorPtr ScanAllByLabel::MakeCursor(utils::MemoryResource *mem) const {
auto *db = context.db_accessor;
return std::make_optional(db->Vertices(view_, label_));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllByLabel");
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, *this, output_symbol_, input_->MakeCursor(mem),
view_, std::move(vertices), "ScanAllByLabel");
}
// TODO(buda): Implement ScanAllByLabelProperty operator to iterate over
@@ -592,8 +606,8 @@ UniqueCursorPtr ScanAllByLabelPropertyRange::MakeCursor(utils::MemoryResource *m
if (maybe_upper && maybe_upper->value().IsNull()) return std::nullopt;
return std::make_optional(db->Vertices(view_, label_, property_, maybe_lower, maybe_upper));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllByLabelPropertyRange");
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(
mem, *this, output_symbol_, input_->MakeCursor(mem), view_, std::move(vertices), "ScanAllByLabelPropertyRange");
}
ScanAllByLabelPropertyValue::ScanAllByLabelPropertyValue(const std::shared_ptr<LogicalOperator> &input,
@@ -624,8 +638,8 @@ UniqueCursorPtr ScanAllByLabelPropertyValue::MakeCursor(utils::MemoryResource *m
}
return std::make_optional(db->Vertices(view_, label_, property_, storage::PropertyValue(value)));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllByLabelPropertyValue");
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(
mem, *this, output_symbol_, input_->MakeCursor(mem), view_, std::move(vertices), "ScanAllByLabelPropertyValue");
}
ScanAllByLabelProperty::ScanAllByLabelProperty(const std::shared_ptr<LogicalOperator> &input, Symbol output_symbol,
@@ -642,8 +656,8 @@ UniqueCursorPtr ScanAllByLabelProperty::MakeCursor(utils::MemoryResource *mem) c
auto *db = context.db_accessor;
return std::make_optional(db->Vertices(view_, label_, property_));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllByLabelProperty");
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, *this, output_symbol_, input_->MakeCursor(mem),
view_, std::move(vertices), "ScanAllByLabelProperty");
}
ScanAllById::ScanAllById(const std::shared_ptr<LogicalOperator> &input, Symbol output_symbol, Expression *expression,
@@ -668,8 +682,8 @@ UniqueCursorPtr ScanAllById::MakeCursor(utils::MemoryResource *mem) const {
if (!maybe_vertex) return std::nullopt;
return std::vector<VertexAccessor>{*maybe_vertex};
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllById");
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, *this, output_symbol_, input_->MakeCursor(mem),
view_, std::move(vertices), "ScanAllById");
}
namespace {
@@ -680,8 +694,8 @@ bool CheckExistingNode(const VertexAccessor &new_node, const Symbol &existing_no
return existing_node.ValueVertex() == new_node;
}
template <class TEdges>
auto UnwrapEdgesResult(storage::Result<TEdges> &&result) {
template <class TEdgesResult>
auto UnwrapEdgesResult(storage::Result<TEdgesResult> &&result) {
if (result.HasError()) {
switch (result.GetError()) {
case storage::Error::DELETED_OBJECT:
@@ -725,8 +739,15 @@ std::vector<Symbol> Expand::ModifiedSymbols(const SymbolTable &table) const {
Expand::ExpandCursor::ExpandCursor(const Expand &self, utils::MemoryResource *mem)
: self_(self), input_cursor_(self.input_->MakeCursor(mem)) {}
Expand::ExpandCursor::ExpandCursor(const Expand &self, int64_t input_degree, int64_t existing_node_degree,
utils::MemoryResource *mem)
: self_(self),
input_cursor_(self.input_->MakeCursor(mem)),
prev_input_degree_(input_degree),
prev_existing_degree_(existing_node_degree) {}
bool Expand::ExpandCursor::Pull(Frame &frame, ExecutionContext &context) {
SCOPED_PROFILE_OP("Expand");
SCOPED_PROFILE_OP_BY_REF(self_);
// A helper function for expanding a node from an edge.
auto pull_node = [this, &frame](const EdgeAccessor &new_edge, EdgeAtom::Direction direction) {
@@ -800,58 +821,131 @@ void Expand::ExpandCursor::Reset() {
out_edges_it_ = std::nullopt;
}
ExpansionInfo Expand::ExpandCursor::GetExpansionInfo(Frame &frame) {
TypedValue &vertex_value = frame[self_.input_symbol_];
if (vertex_value.IsNull()) {
return ExpansionInfo{};
}
ExpectType(self_.input_symbol_, vertex_value, TypedValue::Type::Vertex);
auto &vertex = vertex_value.ValueVertex();
auto direction = self_.common_.direction;
if (!self_.common_.existing_node) {
return ExpansionInfo{.input_node = vertex, .direction = direction};
}
TypedValue &existing_node = frame[self_.common_.node_symbol];
if (existing_node.IsNull()) {
return ExpansionInfo{.input_node = vertex, .direction = direction};
}
ExpectType(self_.common_.node_symbol, existing_node, TypedValue::Type::Vertex);
auto &existing_vertex = existing_node.ValueVertex();
// -1 and -1 -> normal expansion
// -1 and expanded -> can't happen
// expanded and -1 -> reverse
// expanded and expanded -> see if can reverse
if ((prev_input_degree_ == -1 && prev_existing_degree_ == -1) || prev_input_degree_ < prev_existing_degree_) {
return ExpansionInfo{.input_node = vertex, .direction = direction, .existing_node = existing_vertex};
}
auto new_direction = direction;
switch (new_direction) {
case EdgeAtom::Direction::IN:
new_direction = EdgeAtom::Direction::OUT;
break;
case EdgeAtom::Direction::OUT:
new_direction = EdgeAtom::Direction::IN;
break;
default:
new_direction = EdgeAtom::Direction::BOTH;
break;
}
return ExpansionInfo{
.input_node = existing_vertex, .direction = new_direction, .existing_node = vertex, .reversed = true};
}
bool Expand::ExpandCursor::InitEdges(Frame &frame, ExecutionContext &context) {
// Input Vertex could be null if it is created by a failed optional match. In
// those cases we skip that input pull and continue with the next.
while (true) {
if (!input_cursor_->Pull(frame, context)) return false;
TypedValue &vertex_value = frame[self_.input_symbol_];
// Null check due to possible failed optional match.
if (vertex_value.IsNull()) continue;
expansion_info_ = GetExpansionInfo(frame);
ExpectType(self_.input_symbol_, vertex_value, TypedValue::Type::Vertex);
auto &vertex = vertex_value.ValueVertex();
if (!expansion_info_.input_node) {
continue;
}
auto direction = self_.common_.direction;
auto vertex = *expansion_info_.input_node;
auto direction = expansion_info_.direction;
int64_t num_expanded_first = -1;
if (direction == EdgeAtom::Direction::IN || direction == EdgeAtom::Direction::BOTH) {
if (self_.common_.existing_node) {
TypedValue &existing_node = frame[self_.common_.node_symbol];
// old_node_value may be Null when using optional matching
if (!existing_node.IsNull()) {
ExpectType(self_.common_.node_symbol, existing_node, TypedValue::Type::Vertex);
if (expansion_info_.existing_node) {
auto existing_node = *expansion_info_.existing_node;
context.db_accessor->PrefetchInEdges(vertex);
in_edges_.emplace(
UnwrapEdgesResult(vertex.InEdges(self_.view_, self_.common_.edge_types, existing_node.ValueVertex())));
auto edges_result = UnwrapEdgesResult(vertex.InEdges(self_.view_, self_.common_.edge_types, existing_node));
in_edges_.emplace(edges_result.edges);
num_expanded_first = edges_result.expanded_count;
}
} else {
context.db_accessor->PrefetchInEdges(vertex);
in_edges_.emplace(UnwrapEdgesResult(vertex.InEdges(self_.view_, self_.common_.edge_types)));
auto edges_result = UnwrapEdgesResult(vertex.InEdges(self_.view_, self_.common_.edge_types));
in_edges_.emplace(edges_result.edges);
num_expanded_first = edges_result.expanded_count;
}
if (in_edges_) {
in_edges_it_.emplace(in_edges_->begin());
}
}
int64_t num_expanded_second = -1;
if (direction == EdgeAtom::Direction::OUT || direction == EdgeAtom::Direction::BOTH) {
if (self_.common_.existing_node) {
TypedValue &existing_node = frame[self_.common_.node_symbol];
// old_node_value may be Null when using optional matching
if (!existing_node.IsNull()) {
ExpectType(self_.common_.node_symbol, existing_node, TypedValue::Type::Vertex);
if (expansion_info_.existing_node) {
auto existing_node = *expansion_info_.existing_node;
context.db_accessor->PrefetchOutEdges(vertex);
out_edges_.emplace(
UnwrapEdgesResult(vertex.OutEdges(self_.view_, self_.common_.edge_types, existing_node.ValueVertex())));
auto edges_result = UnwrapEdgesResult(vertex.OutEdges(self_.view_, self_.common_.edge_types, existing_node));
out_edges_.emplace(edges_result.edges);
num_expanded_second = edges_result.expanded_count;
}
} else {
context.db_accessor->PrefetchOutEdges(vertex);
out_edges_.emplace(UnwrapEdgesResult(vertex.OutEdges(self_.view_, self_.common_.edge_types)));
auto edges_result = UnwrapEdgesResult(vertex.OutEdges(self_.view_, self_.common_.edge_types));
out_edges_.emplace(edges_result.edges);
num_expanded_second = edges_result.expanded_count;
}
if (out_edges_) {
out_edges_it_.emplace(out_edges_->begin());
}
}
if (!expansion_info_.existing_node) {
return true;
}
num_expanded_first = num_expanded_first == -1 ? 0 : num_expanded_first;
num_expanded_second = num_expanded_second == -1 ? 0 : num_expanded_second;
int64_t total_expanded_edges = num_expanded_first + num_expanded_second;
if (!expansion_info_.reversed) {
prev_input_degree_ = total_expanded_edges;
} else {
prev_existing_degree_ = total_expanded_edges;
}
return true;
}
}
@@ -911,11 +1005,12 @@ auto ExpandFromVertex(const VertexAccessor &vertex, EdgeAtom::Direction directio
};
storage::View view = storage::View::OLD;
utils::pmr::vector<decltype(wrapper(direction, *vertex.InEdges(view, edge_types)))> chain_elements(memory);
utils::pmr::vector<decltype(wrapper(direction, vertex.InEdges(view, edge_types).GetValue().edges))> chain_elements(
memory);
if (direction != EdgeAtom::Direction::OUT) {
db_accessor->PrefetchInEdges(vertex);
auto edges = UnwrapEdgesResult(vertex.InEdges(view, edge_types));
auto edges = UnwrapEdgesResult(vertex.InEdges(view, edge_types)).edges;
if (edges.begin() != edges.end()) {
chain_elements.emplace_back(wrapper(EdgeAtom::Direction::IN, std::move(edges)));
}
@@ -923,7 +1018,7 @@ auto ExpandFromVertex(const VertexAccessor &vertex, EdgeAtom::Direction directio
if (direction != EdgeAtom::Direction::IN) {
db_accessor->PrefetchOutEdges(vertex);
auto edges = UnwrapEdgesResult(vertex.OutEdges(view, edge_types));
auto edges = UnwrapEdgesResult(vertex.OutEdges(view, edge_types)).edges;
if (edges.begin() != edges.end()) {
chain_elements.emplace_back(wrapper(EdgeAtom::Direction::OUT, std::move(edges)));
}
@@ -941,7 +1036,7 @@ class ExpandVariableCursor : public Cursor {
: self_(self), input_cursor_(self.input_->MakeCursor(mem)), edges_(mem), edges_it_(mem) {}
bool Pull(Frame &frame, ExecutionContext &context) override {
SCOPED_PROFILE_OP("ExpandVariable");
SCOPED_PROFILE_OP_BY_REF(self_);
ExpressionEvaluator evaluator(&frame, context.symbol_table, context.evaluation_context, context.db_accessor,
storage::View::OLD);
@@ -1283,7 +1378,7 @@ class STShortestPathCursor : public query::plan::Cursor {
for (const auto &vertex : source_frontier) {
if (self_.common_.direction != EdgeAtom::Direction::IN) {
context.db_accessor->PrefetchOutEdges(vertex);
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types));
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : out_edges) {
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker &&
@@ -1310,7 +1405,7 @@ class STShortestPathCursor : public query::plan::Cursor {
}
if (self_.common_.direction != EdgeAtom::Direction::OUT) {
dba.PrefetchInEdges(vertex);
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types));
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : in_edges) {
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker &&
@@ -1351,7 +1446,7 @@ class STShortestPathCursor : public query::plan::Cursor {
for (const auto &vertex : sink_frontier) {
if (self_.common_.direction != EdgeAtom::Direction::OUT) {
context.db_accessor->PrefetchOutEdges(vertex);
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types));
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : out_edges) {
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker &&
@@ -1377,7 +1472,7 @@ class STShortestPathCursor : public query::plan::Cursor {
}
if (self_.common_.direction != EdgeAtom::Direction::IN) {
dba.PrefetchInEdges(vertex);
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types));
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : in_edges) {
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker &&
@@ -1469,12 +1564,12 @@ class SingleSourceShortestPathCursor : public query::plan::Cursor {
auto expand_from_vertex = [this, &expand_pair, &context](const auto &vertex) {
if (self_.common_.direction != EdgeAtom::Direction::IN) {
context.db_accessor->PrefetchOutEdges(vertex);
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types));
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : out_edges) expand_pair(edge, edge.To());
}
if (self_.common_.direction != EdgeAtom::Direction::OUT) {
context.db_accessor->PrefetchInEdges(vertex);
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types));
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : in_edges) expand_pair(edge, edge.From());
}
};
@@ -1672,14 +1767,14 @@ class ExpandWeightedShortestPathCursor : public query::plan::Cursor {
int64_t depth) {
if (self_.common_.direction != EdgeAtom::Direction::IN) {
context.db_accessor->PrefetchOutEdges(vertex);
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types));
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : out_edges) {
expand_pair(edge, edge.To(), weight, depth);
}
}
if (self_.common_.direction != EdgeAtom::Direction::OUT) {
context.db_accessor->PrefetchInEdges(vertex);
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types));
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : in_edges) {
expand_pair(edge, edge.From(), weight, depth);
}
@@ -1938,7 +2033,7 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
int64_t depth) {
if (self_.common_.direction != EdgeAtom::Direction::IN) {
context.db_accessor->PrefetchOutEdges(vertex);
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types));
auto out_edges = UnwrapEdgesResult(vertex.OutEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : out_edges) {
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker &&
@@ -1953,7 +2048,7 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
}
if (self_.common_.direction != EdgeAtom::Direction::OUT) {
context.db_accessor->PrefetchInEdges(vertex);
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types));
auto in_edges = UnwrapEdgesResult(vertex.InEdges(storage::View::OLD, self_.common_.edge_types)).edges;
for (const auto &edge : in_edges) {
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker &&
@@ -2432,7 +2527,7 @@ Produce::ProduceCursor::ProduceCursor(const Produce &self, utils::MemoryResource
: self_(self), input_cursor_(self_.input_->MakeCursor(mem)) {}
bool Produce::ProduceCursor::Pull(Frame &frame, ExecutionContext &context) {
SCOPED_PROFILE_OP("Produce");
SCOPED_PROFILE_OP_BY_REF(self_);
if (input_cursor_->Pull(frame, context)) {
// Produce should always yield the latest results.
@@ -2470,15 +2565,12 @@ std::vector<Symbol> Delete::ModifiedSymbols(const SymbolTable &table) const { re
Delete::DeleteCursor::DeleteCursor(const Delete &self, utils::MemoryResource *mem)
: self_(self), input_cursor_(self_.input_->MakeCursor(mem)) {}
bool Delete::DeleteCursor::Pull(Frame &frame, ExecutionContext &context) {
SCOPED_PROFILE_OP("Delete");
if (!input_cursor_->Pull(frame, context)) return false;
void Delete::DeleteCursor::UpdateDeleteBuffer(Frame &frame, ExecutionContext &context) {
// Delete should get the latest information, this way it is also possible
// to delete newly added nodes and edges.
ExpressionEvaluator evaluator(&frame, context.symbol_table, context.evaluation_context, context.db_accessor,
storage::View::NEW);
auto *pull_memory = context.evaluation_context.memory;
// collect expressions results so edges can get deleted before vertices
// this is necessary because an edge that gets deleted could block vertex
@@ -2489,107 +2581,34 @@ bool Delete::DeleteCursor::Pull(Frame &frame, ExecutionContext &context) {
expression_results.emplace_back(expression->Accept(evaluator));
}
auto &dba = *context.db_accessor;
// delete edges first
for (TypedValue &expression_result : expression_results) {
AbortCheck(context);
if (expression_result.type() == TypedValue::Type::Edge) {
auto &ea = expression_result.ValueEdge();
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker &&
!(context.auth_checker->Has(ea, query::AuthQuery::FineGrainedPrivilege::CREATE_DELETE) &&
context.auth_checker->Has(ea.To(), storage::View::NEW, query::AuthQuery::FineGrainedPrivilege::UPDATE) &&
context.auth_checker->Has(ea.From(), storage::View::NEW, query::AuthQuery::FineGrainedPrivilege::UPDATE))) {
throw QueryRuntimeException("Edge not deleted due to not having enough permission!");
}
#endif
auto maybe_value = dba.RemoveEdge(&ea);
if (maybe_value.HasError()) {
switch (maybe_value.GetError()) {
case storage::Error::SERIALIZATION_ERROR:
throw TransactionSerializationException();
case storage::Error::DELETED_OBJECT:
case storage::Error::VERTEX_HAS_EDGES:
case storage::Error::PROPERTIES_DISABLED:
case storage::Error::NONEXISTENT_OBJECT:
throw QueryRuntimeException("Unexpected error when deleting an edge.");
}
}
context.execution_stats[ExecutionStats::Key::DELETED_EDGES] += 1;
if (context.trigger_context_collector && maybe_value.GetValue()) {
context.trigger_context_collector->RegisterDeletedObject(*maybe_value.GetValue());
}
}
}
// delete vertices
for (TypedValue &expression_result : expression_results) {
AbortCheck(context);
switch (expression_result.type()) {
case TypedValue::Type::Vertex: {
auto &va = expression_result.ValueVertex();
auto va = expression_result.ValueVertex();
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker &&
!context.auth_checker->Has(va, storage::View::NEW, query::AuthQuery::FineGrainedPrivilege::CREATE_DELETE)) {
throw QueryRuntimeException("Vertex not deleted due to not having enough permission!");
}
#endif
if (self_.detach_) {
auto res = dba.DetachRemoveVertex(&va);
if (res.HasError()) {
switch (res.GetError()) {
case storage::Error::SERIALIZATION_ERROR:
throw TransactionSerializationException();
case storage::Error::DELETED_OBJECT:
case storage::Error::VERTEX_HAS_EDGES:
case storage::Error::PROPERTIES_DISABLED:
case storage::Error::NONEXISTENT_OBJECT:
throw QueryRuntimeException("Unexpected error when deleting a node.");
}
}
context.execution_stats[ExecutionStats::Key::DELETED_NODES] += 1;
if (*res) {
context.execution_stats[ExecutionStats::Key::DELETED_EDGES] += static_cast<int64_t>((*res)->second.size());
}
std::invoke([&] {
if (!context.trigger_context_collector || !*res) {
return;
}
context.trigger_context_collector->RegisterDeletedObject((*res)->first);
if (!context.trigger_context_collector->ShouldRegisterDeletedObject<query::EdgeAccessor>()) {
return;
}
for (const auto &edge : (*res)->second) {
context.trigger_context_collector->RegisterDeletedObject(edge);
}
});
} else {
auto res = dba.RemoveVertex(&va);
if (res.HasError()) {
switch (res.GetError()) {
case storage::Error::SERIALIZATION_ERROR:
throw TransactionSerializationException();
case storage::Error::VERTEX_HAS_EDGES:
throw RemoveAttachedVertexException();
case storage::Error::DELETED_OBJECT:
case storage::Error::PROPERTIES_DISABLED:
case storage::Error::NONEXISTENT_OBJECT:
throw QueryRuntimeException("Unexpected error when deleting a node.");
}
}
context.execution_stats[ExecutionStats::Key::DELETED_NODES] += 1;
if (context.trigger_context_collector && res.GetValue()) {
context.trigger_context_collector->RegisterDeletedObject(*res.GetValue());
}
}
buffer_.nodes.push_back(va);
break;
}
case TypedValue::Type::Edge: {
auto ea = expression_result.ValueEdge();
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && context.auth_checker &&
!(context.auth_checker->Has(ea, query::AuthQuery::FineGrainedPrivilege::CREATE_DELETE) &&
context.auth_checker->Has(ea.To(), storage::View::NEW, query::AuthQuery::FineGrainedPrivilege::UPDATE) &&
context.auth_checker->Has(ea.From(), storage::View::NEW,
query::AuthQuery::FineGrainedPrivilege::UPDATE))) {
throw QueryRuntimeException("Edge not deleted due to not having enough permission!");
}
#endif
buffer_.edges.push_back(ea);
break;
}
// skip Edges (already deleted) and Nulls (can occur in optional
// match)
case TypedValue::Type::Edge:
case TypedValue::Type::Null:
break;
// check we're not trying to delete anything except vertices and edges
@@ -2597,13 +2616,64 @@ bool Delete::DeleteCursor::Pull(Frame &frame, ExecutionContext &context) {
throw QueryRuntimeException("Only edges and vertices can be deleted.");
}
}
}
return true;
bool Delete::DeleteCursor::Pull(Frame &frame, ExecutionContext &context) {
SCOPED_PROFILE_OP("Delete");
if (delete_executed_) {
return false;
}
if (input_cursor_->Pull(frame, context)) {
UpdateDeleteBuffer(frame, context);
return true;
}
auto &dba = *context.db_accessor;
auto res = dba.DetachDelete(std::move(buffer_.nodes), std::move(buffer_.edges), self_.detach_);
if (res.HasError()) {
switch (res.GetError()) {
case storage::Error::SERIALIZATION_ERROR:
throw TransactionSerializationException();
case storage::Error::VERTEX_HAS_EDGES:
throw RemoveAttachedVertexException();
case storage::Error::DELETED_OBJECT:
case storage::Error::PROPERTIES_DISABLED:
case storage::Error::NONEXISTENT_OBJECT:
throw QueryRuntimeException("Unexpected error when deleting a node.");
}
}
if (*res) {
context.execution_stats[ExecutionStats::Key::DELETED_NODES] += static_cast<int64_t>((*res)->first.size());
context.execution_stats[ExecutionStats::Key::DELETED_EDGES] += static_cast<int64_t>((*res)->second.size());
}
// Update deleted objects for triggers
if (context.trigger_context_collector && *res) {
for (const auto &node : (*res)->first) {
context.trigger_context_collector->RegisterDeletedObject(node);
}
if (context.trigger_context_collector->ShouldRegisterDeletedObject<query::EdgeAccessor>()) {
for (const auto &edge : (*res)->second) {
context.trigger_context_collector->RegisterDeletedObject(edge);
}
}
}
delete_executed_ = true;
return false;
}
void Delete::DeleteCursor::Shutdown() { input_cursor_->Shutdown(); }
void Delete::DeleteCursor::Reset() { input_cursor_->Reset(); }
void Delete::DeleteCursor::Reset() {
input_cursor_->Reset();
delete_executed_ = false;
}
SetProperty::SetProperty(const std::shared_ptr<LogicalOperator> &input, storage::PropertyId property,
PropertyLookup *lhs, Expression *rhs)
@@ -2724,7 +2794,8 @@ concept AccessorWithProperties = requires(T value, storage::PropertyId property_
/// RecordAccessor<Edge>
template <AccessorWithProperties TRecordAccessor>
void SetPropertiesOnRecord(TRecordAccessor *record, const TypedValue &rhs, SetProperties::Op op,
ExecutionContext *context) {
ExecutionContext *context,
std::unordered_map<std::string, storage::PropertyId> &cached_name_id) {
using PropertiesMap = std::map<storage::PropertyId, storage::PropertyValue>;
std::optional<PropertiesMap> old_values;
const bool should_register_change =
@@ -2808,9 +2879,15 @@ void SetPropertiesOnRecord(TRecordAccessor *record, const TypedValue &rhs, SetPr
}
case TypedValue::Type::Map: {
PropertiesMap new_properties;
for (const auto &[prop_id, prop_value] : rhs.ValueMap()) {
auto key = context->db_accessor->NameToProperty(prop_id);
new_properties.emplace(key, prop_value);
for (const auto &[string_key, value] : rhs.ValueMap()) {
storage::PropertyId property_id;
if (auto it = cached_name_id.find(std::string(string_key)); it != cached_name_id.end()) [[likely]] {
property_id = it->second;
} else {
property_id = context->db_accessor->NameToProperty(string_key);
cached_name_id.emplace(string_key, property_id);
}
new_properties.emplace(property_id, value);
}
update_props(new_properties);
break;
@@ -2853,7 +2930,7 @@ bool SetProperties::SetPropertiesCursor::Pull(Frame &frame, ExecutionContext &co
throw QueryRuntimeException("Vertex properties not set due to not having enough permission!");
}
#endif
SetPropertiesOnRecord(&lhs.ValueVertex(), rhs, self_.op_, &context);
SetPropertiesOnRecord(&lhs.ValueVertex(), rhs, self_.op_, &context, cached_name_id_);
break;
case TypedValue::Type::Edge:
#ifdef MG_ENTERPRISE
@@ -2862,7 +2939,7 @@ bool SetProperties::SetPropertiesCursor::Pull(Frame &frame, ExecutionContext &co
throw QueryRuntimeException("Edge properties not set due to not having enough permission!");
}
#endif
SetPropertiesOnRecord(&lhs.ValueEdge(), rhs, self_.op_, &context);
SetPropertiesOnRecord(&lhs.ValueEdge(), rhs, self_.op_, &context, cached_name_id_);
break;
case TypedValue::Type::Null:
// Skip setting properties on Null (can occur in optional match).
@@ -3331,7 +3408,7 @@ class AggregateCursor : public Cursor {
: self_(self), input_cursor_(self_.input_->MakeCursor(mem)), aggregation_(mem) {}
bool Pull(Frame &frame, ExecutionContext &context) override {
SCOPED_PROFILE_OP("Aggregate");
SCOPED_PROFILE_OP_BY_REF(self_);
if (!pulled_all_input_) {
ProcessAll(&frame, &context);
@@ -3795,7 +3872,7 @@ class OrderByCursor : public Cursor {
: self_(self), input_cursor_(self_.input_->MakeCursor(mem)), cache_(mem) {}
bool Pull(Frame &frame, ExecutionContext &context) override {
SCOPED_PROFILE_OP("OrderBy");
SCOPED_PROFILE_OP_BY_REF(self_);
if (!did_pull_all_) {
ExpressionEvaluator evaluator(&frame, context.symbol_table, context.evaluation_context, context.db_accessor,
@@ -4197,7 +4274,7 @@ Union::UnionCursor::UnionCursor(const Union &self, utils::MemoryResource *mem)
: self_(self), left_cursor_(self.left_op_->MakeCursor(mem)), right_cursor_(self.right_op_->MakeCursor(mem)) {}
bool Union::UnionCursor::Pull(Frame &frame, ExecutionContext &context) {
SCOPED_PROFILE_OP("Union");
SCOPED_PROFILE_OP_BY_REF(self_);
utils::pmr::unordered_map<std::string, TypedValue> results(context.evaluation_context.memory);
if (left_cursor_->Pull(frame, context)) {
@@ -4271,7 +4348,7 @@ class CartesianCursor : public Cursor {
}
bool Pull(Frame &frame, ExecutionContext &context) override {
SCOPED_PROFILE_OP("Cartesian");
SCOPED_PROFILE_OP_BY_REF(self_);
if (!cartesian_pull_initialized_) {
// Pull all left_op frames.
@@ -4560,7 +4637,7 @@ class CallProcedureCursor : public Cursor {
}
bool Pull(Frame &frame, ExecutionContext &context) override {
SCOPED_PROFILE_OP("CallProcedure");
SCOPED_PROFILE_OP_BY_REF(*self_);
AbortCheck(context);
@@ -4626,7 +4703,7 @@ class CallProcedureCursor : public Cursor {
// TODO: This will probably need to be changed when we add support for
// generator like procedures which yield a new result on new query calls.
auto *memory = self_->memory_resource;
auto memory_limit = EvaluateMemoryLimit(&evaluator, self_->memory_limit_, self_->memory_scale_);
auto memory_limit = EvaluateMemoryLimit(evaluator, self_->memory_limit_, self_->memory_scale_);
auto graph = mgp_graph::WritableGraph(*context.db_accessor, graph_view, context);
CallCustomProcedure(self_->procedure_name_, *proc, self_->arguments_, graph, &evaluator, memory, memory_limit,
result_, call_initializer);
@@ -4839,7 +4916,7 @@ class LoadCsvCursor : public Cursor {
: self_(self), input_cursor_(self_->input_->MakeCursor(mem)), did_pull_{false} {}
bool Pull(Frame &frame, ExecutionContext &context) override {
SCOPED_PROFILE_OP("LoadCsv");
SCOPED_PROFILE_OP_BY_REF(*self_);
AbortCheck(context);
@@ -4856,7 +4933,9 @@ class LoadCsvCursor : public Cursor {
if (input_cursor_->Pull(frame, context)) {
if (did_pull_) {
throw QueryRuntimeException(
"LOAD CSV can be executed only once, please check if the cardinality of the operator before LOAD CSV is 1");
"LOAD CSV can be executed only once, please check if the cardinality of the operator before LOAD CSV "
"is "
"1");
}
did_pull_ = true;
}

View File

@@ -152,12 +152,19 @@ class HierarchicalLogicalOperatorVisitor : public LogicalOperatorCompositeVisito
using typename LogicalOperatorLeafVisitor::ReturnType;
};
class NamedLogicalOperator {
public:
mutable const DbAccessor *dba_{nullptr};
virtual std::string ToString() const = 0;
};
/// Base class for logical operators.
///
/// Each operator describes an operation, which is to be performed on the
/// database. Operators are iterated over using a @c Cursor. Various operators
/// can serve as inputs to others and thus a sequence of operations is formed.
class LogicalOperator : public utils::Visitable<HierarchicalLogicalOperatorVisitor> {
class LogicalOperator : public utils::Visitable<HierarchicalLogicalOperatorVisitor>,
public memgraph::query::plan::NamedLogicalOperator {
public:
static const utils::TypeInfo kType;
virtual const utils::TypeInfo &GetTypeInfo() const { return kType; }
@@ -232,6 +239,8 @@ class LogicalOperator : public utils::Visitable<HierarchicalLogicalOperatorVisit
std::vector<std::shared_ptr<LogicalOperator>> loaded_ops;
};
std::string ToString() const override { return GetTypeInfo().name; }
virtual std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const = 0;
};
@@ -464,6 +473,13 @@ class CreateExpand : public memgraph::query::plan::LogicalOperator {
/// if the given node atom refers to an existing node (either matched or created)
bool existing_node_;
std::string ToString() const override {
return fmt::format("CreateExpand ({}){}[{}:{}]{}({})", input_symbol_.name(),
edge_info_.direction == query::EdgeAtom::Direction::IN ? "<-" : "-", edge_info_.symbol.name(),
dba_->EdgeTypeToName(edge_info_.edge_type),
edge_info_.direction == query::EdgeAtom::Direction::OUT ? "->" : "-", node_info_.symbol.name());
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<CreateExpand>();
object->node_info_ = node_info_.Clone(storage);
@@ -530,6 +546,8 @@ class ScanAll : public memgraph::query::plan::LogicalOperator {
/// transaction sees along with their modifications.
storage::View view_;
std::string ToString() const override { return fmt::format("ScanAll ({})", output_symbol_.name()); }
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<ScanAll>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -558,6 +576,10 @@ class ScanAllByLabel : public memgraph::query::plan::ScanAll {
storage::LabelId label_;
std::string ToString() const override {
return fmt::format("ScanAllByLabel ({} :{})", output_symbol_.name(), dba_->LabelToName(label_));
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<ScanAllByLabel>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -610,6 +632,11 @@ class ScanAllByLabelPropertyRange : public memgraph::query::plan::ScanAll {
std::optional<Bound> lower_bound_;
std::optional<Bound> upper_bound_;
std::string ToString() const override {
return fmt::format("ScanAllByLabelPropertyRange ({0} :{1} {{{2}}})", output_symbol_.name(),
dba_->LabelToName(label_), dba_->PropertyToName(property_));
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<ScanAllByLabelPropertyRange>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -668,6 +695,11 @@ class ScanAllByLabelPropertyValue : public memgraph::query::plan::ScanAll {
std::string property_name_;
Expression *expression_;
std::string ToString() const override {
return fmt::format("ScanAllByLabelPropertyValue ({0} :{1} {{{2}}})", output_symbol_.name(),
dba_->LabelToName(label_), dba_->PropertyToName(property_));
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<ScanAllByLabelPropertyValue>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -704,6 +736,11 @@ class ScanAllByLabelProperty : public memgraph::query::plan::ScanAll {
std::string property_name_;
Expression *expression_;
std::string ToString() const override {
return fmt::format("ScanAllByLabelProperty ({0} :{1} {{{2}}})", output_symbol_.name(), dba_->LabelToName(label_),
dba_->PropertyToName(property_));
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<ScanAllByLabelProperty>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -732,6 +769,8 @@ class ScanAllById : public memgraph::query::plan::ScanAll {
Expression *expression_;
std::string ToString() const override { return fmt::format("ScanAllById ({})", output_symbol_.name()); }
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<ScanAllById>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -764,6 +803,13 @@ struct ExpandCommon {
bool existing_node;
};
struct ExpansionInfo {
std::optional<VertexAccessor> input_node;
EdgeAtom::Direction direction;
std::optional<VertexAccessor> existing_node;
bool reversed{false};
};
/// Expansion operator. For a node existing in the frame it
/// expands one edge and one node and places them on the frame.
///
@@ -806,14 +852,16 @@ class Expand : public memgraph::query::plan::LogicalOperator {
class ExpandCursor : public Cursor {
public:
ExpandCursor(const Expand &, utils::MemoryResource *);
ExpandCursor(const Expand &, int64_t input_degree, int64_t existing_node_degree, utils::MemoryResource *);
bool Pull(Frame &, ExecutionContext &) override;
void Shutdown() override;
void Reset() override;
ExpansionInfo GetExpansionInfo(Frame &);
private:
using InEdgeT = std::remove_reference_t<decltype(*std::declval<VertexAccessor>().InEdges(storage::View::OLD))>;
using InEdgeT = std::vector<EdgeAccessor>;
using InEdgeIteratorT = decltype(std::declval<InEdgeT>().begin());
using OutEdgeT = std::remove_reference_t<decltype(*std::declval<VertexAccessor>().OutEdges(storage::View::OLD))>;
using OutEdgeT = std::vector<EdgeAccessor>;
using OutEdgeIteratorT = decltype(std::declval<OutEdgeT>().begin());
const Expand &self_;
@@ -826,6 +874,9 @@ class Expand : public memgraph::query::plan::LogicalOperator {
std::optional<InEdgeIteratorT> in_edges_it_;
std::optional<OutEdgeT> out_edges_;
std::optional<OutEdgeIteratorT> out_edges_it_;
ExpansionInfo expansion_info_;
int64_t prev_input_degree_{-1};
int64_t prev_existing_degree_{-1};
bool InitEdges(Frame &, ExecutionContext &);
};
@@ -836,6 +887,15 @@ class Expand : public memgraph::query::plan::LogicalOperator {
/// State from which the input node should get expanded.
storage::View view_;
std::string ToString() const override {
return fmt::format(
"Expand ({}){}[{}{}]{}({})", input_symbol_.name(),
common_.direction == query::EdgeAtom::Direction::IN ? "<-" : "-", common_.edge_symbol.name(),
utils::IterableToString(common_.edge_types, "|",
[this](const auto &edge_type) { return ":" + dba_->EdgeTypeToName(edge_type); }),
common_.direction == query::EdgeAtom::Direction::OUT ? "->" : "-", common_.node_symbol.name());
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<Expand>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -943,6 +1003,37 @@ class ExpandVariable : public memgraph::query::plan::LogicalOperator {
std::optional<memgraph::query::plan::ExpansionLambda> weight_lambda_;
std::optional<Symbol> total_weight_;
std::string OperatorName() const {
using Type = query::EdgeAtom::Type;
switch (type_) {
case Type::DEPTH_FIRST:
return "ExpandVariable";
break;
case Type::BREADTH_FIRST:
return (common_.existing_node ? "STShortestPath" : "BFSExpand");
break;
case Type::WEIGHTED_SHORTEST_PATH:
return "WeightedShortestPath";
break;
case Type::ALL_SHORTEST_PATHS:
return "AllShortestPaths";
break;
case Type::SINGLE:
LOG_FATAL("Unexpected ExpandVariable::type_");
default:
LOG_FATAL("Unexpected ExpandVariable::type_");
}
}
std::string ToString() const override {
return fmt::format(
"{} ({}){}[{}{}]{}({})", OperatorName(), input_symbol_.name(),
common_.direction == query::EdgeAtom::Direction::IN ? "<-" : "-", common_.edge_symbol.name(),
utils::IterableToString(common_.edge_types, "|",
[this](const auto &edge_type) { return ":" + dba_->EdgeTypeToName(edge_type); }),
common_.direction == query::EdgeAtom::Direction::OUT ? "->" : "-", common_.node_symbol.name());
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<ExpandVariable>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -1085,6 +1176,11 @@ class Produce : public memgraph::query::plan::LogicalOperator {
std::shared_ptr<memgraph::query::plan::LogicalOperator> input_;
std::vector<NamedExpression *> named_expressions_;
std::string ToString() const override {
return fmt::format("Produce {{{}}}", utils::IterableToString(named_expressions_, ", ",
[](const auto &nexpr) { return nexpr->name_; }));
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<Produce>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -1109,6 +1205,11 @@ class Produce : public memgraph::query::plan::LogicalOperator {
};
};
struct DeleteBuffer {
std::vector<VertexAccessor> nodes{};
std::vector<EdgeAccessor> edges{};
};
/// Operator for deleting vertices and edges.
///
/// Has a flag for using DETACH DELETE when deleting vertices.
@@ -1156,6 +1257,10 @@ class Delete : public memgraph::query::plan::LogicalOperator {
private:
const Delete &self_;
const UniqueCursorPtr input_cursor_;
DeleteBuffer buffer_;
bool delete_executed_{false};
void UpdateDeleteBuffer(Frame &, ExecutionContext &);
};
};
@@ -1263,6 +1368,7 @@ class SetProperties : public memgraph::query::plan::LogicalOperator {
private:
const SetProperties &self_;
const UniqueCursorPtr input_cursor_;
std::unordered_map<std::string, storage::PropertyId> cached_name_id_{};
};
};
@@ -1606,6 +1712,13 @@ class Aggregate : public memgraph::query::plan::LogicalOperator {
std::vector<Expression *> group_by_;
std::vector<Symbol> remember_;
std::string ToString() const override {
return fmt::format(
"Aggregate {{{0}}} {{{1}}}",
utils::IterableToString(aggregations_, ", ", [](const auto &aggr) { return aggr.output_sym.name(); }),
utils::IterableToString(remember_, ", ", [](const auto &sym) { return sym.name(); }));
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<Aggregate>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -1811,6 +1924,11 @@ class OrderBy : public memgraph::query::plan::LogicalOperator {
std::vector<Expression *> order_by_;
std::vector<Symbol> output_symbols_;
std::string ToString() const override {
return fmt::format("OrderBy {{{}}}",
utils::IterableToString(output_symbols_, ", ", [](const auto &sym) { return sym.name(); }));
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<OrderBy>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -2042,6 +2160,12 @@ class Union : public memgraph::query::plan::LogicalOperator {
std::vector<Symbol> left_symbols_;
std::vector<Symbol> right_symbols_;
std::string ToString() const override {
return fmt::format("Union {{{0} : {1}}}",
utils::IterableToString(left_symbols_, ", ", [](const auto &sym) { return sym.name(); }),
utils::IterableToString(right_symbols_, ", ", [](const auto &sym) { return sym.name(); }));
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<Union>();
object->left_op_ = left_op_ ? left_op_->Clone(storage) : nullptr;
@@ -2204,6 +2328,11 @@ class CallProcedure : public memgraph::query::plan::LogicalOperator {
mutable utils::MonotonicBufferResource monotonic_memory{1024UL * 1024UL};
utils::MemoryResource *memory_resource = &monotonic_memory;
std::string ToString() const override {
return fmt::format("CallProcedure<{0}> {{{1}}}", procedure_name_,
utils::IterableToString(result_symbols_, ", ", [](const auto &sym) { return sym.name(); }));
}
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<CallProcedure>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;
@@ -2251,6 +2380,8 @@ class LoadCsv : public memgraph::query::plan::LogicalOperator {
Expression *nullif_{nullptr};
Symbol row_var_;
std::string ToString() const override { return fmt::format("LoadCsv {{{}}}", row_var_.name()); }
std::unique_ptr<LogicalOperator> Clone(AstStorage *storage) const override {
auto object = std::make_unique<LoadCsv>();
object->input_ = input_ ? input_->Clone(storage) : nullptr;

View File

@@ -30,121 +30,68 @@ PlanPrinter::PlanPrinter(const DbAccessor *dba, std::ostream *out) : dba_(dba),
PRE_VISIT(CreateNode);
bool PlanPrinter::PreVisit(CreateExpand &op) {
WithPrintLn([&](auto &out) {
out << "* CreateExpand (" << op.input_symbol_.name() << ")"
<< (op.edge_info_.direction == query::EdgeAtom::Direction::IN ? "<-" : "-") << "["
<< op.edge_info_.symbol.name() << ":" << dba_->EdgeTypeToName(op.edge_info_.edge_type) << "]"
<< (op.edge_info_.direction == query::EdgeAtom::Direction::OUT ? "->" : "-") << "("
<< op.node_info_.symbol.name() << ")";
});
op.dba_ = dba_;
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
op.dba_ = nullptr;
return true;
}
PRE_VISIT(Delete);
bool PlanPrinter::PreVisit(query::plan::ScanAll &op) {
WithPrintLn([&](auto &out) {
out << "* ScanAll"
<< " (" << op.output_symbol_.name() << ")";
});
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
return true;
}
bool PlanPrinter::PreVisit(query::plan::ScanAllByLabel &op) {
WithPrintLn([&](auto &out) {
out << "* ScanAllByLabel"
<< " (" << op.output_symbol_.name() << " :" << dba_->LabelToName(op.label_) << ")";
});
op.dba_ = dba_;
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
op.dba_ = nullptr;
return true;
}
bool PlanPrinter::PreVisit(query::plan::ScanAllByLabelPropertyValue &op) {
WithPrintLn([&](auto &out) {
out << "* ScanAllByLabelPropertyValue"
<< " (" << op.output_symbol_.name() << " :" << dba_->LabelToName(op.label_) << " {"
<< dba_->PropertyToName(op.property_) << "})";
});
op.dba_ = dba_;
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
op.dba_ = nullptr;
return true;
}
bool PlanPrinter::PreVisit(query::plan::ScanAllByLabelPropertyRange &op) {
WithPrintLn([&](auto &out) {
out << "* ScanAllByLabelPropertyRange"
<< " (" << op.output_symbol_.name() << " :" << dba_->LabelToName(op.label_) << " {"
<< dba_->PropertyToName(op.property_) << "})";
});
op.dba_ = dba_;
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
op.dba_ = nullptr;
return true;
}
bool PlanPrinter::PreVisit(query::plan::ScanAllByLabelProperty &op) {
WithPrintLn([&](auto &out) {
out << "* ScanAllByLabelProperty"
<< " (" << op.output_symbol_.name() << " :" << dba_->LabelToName(op.label_) << " {"
<< dba_->PropertyToName(op.property_) << "})";
});
op.dba_ = dba_;
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
op.dba_ = nullptr;
return true;
}
bool PlanPrinter::PreVisit(ScanAllById &op) {
WithPrintLn([&](auto &out) {
out << "* ScanAllById"
<< " (" << op.output_symbol_.name() << ")";
});
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
return true;
}
bool PlanPrinter::PreVisit(query::plan::Expand &op) {
WithPrintLn([&](auto &out) {
*out_ << "* Expand (" << op.input_symbol_.name() << ")"
<< (op.common_.direction == query::EdgeAtom::Direction::IN ? "<-" : "-") << "["
<< op.common_.edge_symbol.name();
utils::PrintIterable(*out_, op.common_.edge_types, "|", [this](auto &stream, const auto &edge_type) {
stream << ":" << dba_->EdgeTypeToName(edge_type);
});
*out_ << "]" << (op.common_.direction == query::EdgeAtom::Direction::OUT ? "->" : "-") << "("
<< op.common_.node_symbol.name() << ")";
});
op.dba_ = dba_;
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
op.dba_ = nullptr;
return true;
}
bool PlanPrinter::PreVisit(query::plan::ExpandVariable &op) {
using Type = query::EdgeAtom::Type;
WithPrintLn([&](auto &out) {
*out_ << "* ";
switch (op.type_) {
case Type::DEPTH_FIRST:
*out_ << "ExpandVariable";
break;
case Type::BREADTH_FIRST:
*out_ << (op.common_.existing_node ? "STShortestPath" : "BFSExpand");
break;
case Type::WEIGHTED_SHORTEST_PATH:
*out_ << "WeightedShortestPath";
break;
case Type::ALL_SHORTEST_PATHS:
*out_ << "AllShortestPaths";
break;
case Type::SINGLE:
LOG_FATAL("Unexpected ExpandVariable::type_");
}
*out_ << " (" << op.input_symbol_.name() << ")"
<< (op.common_.direction == query::EdgeAtom::Direction::IN ? "<-" : "-") << "["
<< op.common_.edge_symbol.name();
utils::PrintIterable(*out_, op.common_.edge_types, "|", [this](auto &stream, const auto &edge_type) {
stream << ":" << dba_->EdgeTypeToName(edge_type);
});
*out_ << "]" << (op.common_.direction == query::EdgeAtom::Direction::OUT ? "->" : "-") << "("
<< op.common_.node_symbol.name() << ")";
});
op.dba_ = dba_;
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
op.dba_ = nullptr;
return true;
}
bool PlanPrinter::PreVisit(query::plan::Produce &op) {
WithPrintLn([&](auto &out) {
out << "* Produce {";
utils::PrintIterable(out, op.named_expressions_, ", ", [](auto &out, const auto &nexpr) { out << nexpr->name_; });
out << "}";
});
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
return true;
}
@@ -160,14 +107,7 @@ PRE_VISIT(EmptyResult);
PRE_VISIT(EvaluatePatternFilter);
bool PlanPrinter::PreVisit(query::plan::Aggregate &op) {
WithPrintLn([&](auto &out) {
out << "* Aggregate {";
utils::PrintIterable(out, op.aggregations_, ", ",
[](auto &out, const auto &aggr) { out << aggr.output_sym.name(); });
out << "} {";
utils::PrintIterable(out, op.remember_, ", ", [](auto &out, const auto &sym) { out << sym.name(); });
out << "}";
});
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
return true;
}
@@ -175,11 +115,7 @@ PRE_VISIT(Skip);
PRE_VISIT(Limit);
bool PlanPrinter::PreVisit(query::plan::OrderBy &op) {
WithPrintLn([&op](auto &out) {
out << "* OrderBy {";
utils::PrintIterable(out, op.output_symbols_, ", ", [](auto &out, const auto &sym) { out << sym.name(); });
out << "}";
});
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
return true;
}
@@ -202,29 +138,19 @@ PRE_VISIT(Unwind);
PRE_VISIT(Distinct);
bool PlanPrinter::PreVisit(query::plan::Union &op) {
WithPrintLn([&op](auto &out) {
out << "* Union {";
utils::PrintIterable(out, op.left_symbols_, ", ", [](auto &out, const auto &sym) { out << sym.name(); });
out << " : ";
utils::PrintIterable(out, op.right_symbols_, ", ", [](auto &out, const auto &sym) { out << sym.name(); });
out << "}";
});
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
Branch(*op.right_op_);
op.left_op_->Accept(*this);
return false;
}
bool PlanPrinter::PreVisit(query::plan::CallProcedure &op) {
WithPrintLn([&op](auto &out) {
out << "* CallProcedure<" << op.procedure_name_ << "> {";
utils::PrintIterable(out, op.result_symbols_, ", ", [](auto &out, const auto &sym) { out << sym.name(); });
out << "}";
});
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
return true;
}
bool PlanPrinter::PreVisit(query::plan::LoadCsv &op) {
WithPrintLn([&op](auto &out) { out << "* LoadCsv {" << op.row_var_.name() << "}"; });
WithPrintLn([&](auto &out) { out << "* " << op.ToString(); });
return true;
}

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -57,7 +57,7 @@ class ProfilingStatsToTableHelper {
auto cycles = IndividualCycles(cumulative_stats);
rows_.emplace_back(std::vector<TypedValue>{
TypedValue(FormatOperator(cumulative_stats.name)), TypedValue(cumulative_stats.actual_hits),
TypedValue(FormatOperator(cumulative_stats.name.c_str())), TypedValue(cumulative_stats.actual_hits),
TypedValue(FormatRelativeTime(cycles)), TypedValue(FormatAbsoluteTime(cycles))});
for (size_t i = 1; i < cumulative_stats.children.size(); ++i) {
@@ -137,7 +137,7 @@ class ProfilingStatsToJsonHelper {
void Output(const ProfilingStats &cumulative_stats, json *obj) {
auto cycles = IndividualCycles(cumulative_stats);
obj->emplace("name", cumulative_stats.name);
obj->emplace("name", cumulative_stats.name.c_str());
obj->emplace("actual_hits", cumulative_stats.actual_hits);
obj->emplace("relative_time", RelativeTime(cycles, total_cycles_));
obj->emplace("absolute_time", AbsoluteTime(cycles, total_cycles_, total_time_));

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -29,7 +29,7 @@ struct ProfilingStats {
int64_t actual_hits{0};
unsigned long long num_cycles{0};
uint64_t key{0};
const char *name{nullptr};
std::string name;
// TODO: This should use the allocator for query execution
std::vector<ProfilingStats> children;
};

View File

@@ -90,19 +90,8 @@ class IndexLookupRewriter final : public HierarchicalLogicalOperatorVisitor {
return true;
}
// See if it might be better to do ScanAllBy<Index> of the destination and
// then do Expand to existing.
bool PostVisit(Expand &expand) override {
bool PostVisit(Expand & /*expand*/) override {
prev_ops_.pop_back();
if (expand.common_.existing_node) {
return true;
}
ScanAll dst_scan(expand.input(), expand.common_.node_symbol, expand.view_);
auto indexed_scan = GenScanByIndex(dst_scan, FLAGS_query_vertex_count_to_expand_existing);
if (indexed_scan) {
expand.set_input(std::move(indexed_scan));
expand.common_.existing_node = true;
}
return true;
}

View File

@@ -98,6 +98,11 @@ class ReturnBodyContext : public HierarchicalTreeVisitor {
auto it = has_aggregation_.end();
auto elements_it = literal.elements_.begin();
std::advance(it, -literal.elements_.size());
if (literal.GetTypeInfo() == MapProjectionLiteral::kType) {
// Erase the map variable. Grammar-wise, its a variable and thus never has aggregations.
std::advance(it, -1);
it = has_aggregation_.erase(it);
}
while (it != has_aggregation_.end()) {
if (*it) {
has_aggr = true;
@@ -446,9 +451,9 @@ class ReturnBodyContext : public HierarchicalTreeVisitor {
std::vector<Expression *> group_by_;
std::unordered_set<Symbol> group_by_used_symbols_;
// Flag stack indicating whether an expression contains an aggregation. A
// stack is needed so that we differentiate the case where a child
// sub-expression has an aggregation, while the other child doesn't. For
// example AST, (+ (sum x) y)
// stack is needed to address the case where one child sub-expression has
// an aggregation, while the other child does not.
// For example, the AST (+ (sum x) y) is as follows:
// * (sum x) -- Has an aggregation.
// * y -- Doesn't, we need to group by this.
// * (+ (sum x) y) -- The whole expression has an aggregation, so we don't

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -28,6 +28,43 @@ namespace memgraph::query::plan {
*/
class ScopedProfile {
public:
ScopedProfile(uint64_t key, const query::plan::NamedLogicalOperator &op, query::ExecutionContext *context) noexcept
: context_(context) {
if (UNLIKELY(context_->is_profile_query)) {
root_ = context_->stats_root;
// Are we the root logical operator?
if (!root_) {
stats_ = &context_->stats;
stats_->key = key;
op.dba_ = context->db_accessor;
stats_->name = op.ToString();
op.dba_ = nullptr;
} else {
stats_ = nullptr;
// Was this logical operator already hit on one of the previous pulls?
auto it = std::find_if(root_->children.begin(), root_->children.end(),
[key](auto &stats) { return stats.key == key; });
if (it == root_->children.end()) {
root_->children.emplace_back();
stats_ = &root_->children.back();
stats_->key = key;
op.dba_ = context->db_accessor;
stats_->name = op.ToString();
op.dba_ = nullptr;
} else {
stats_ = &(*it);
}
}
context_->stats_root = stats_;
stats_->actual_hits++;
start_time_ = utils::ReadTSC();
}
}
ScopedProfile(uint64_t key, const char *name, query::ExecutionContext *context) noexcept : context_(context) {
if (UNLIKELY(context_->is_profile_query)) {
root_ = context_->stats_root;

Some files were not shown because too many files have changed in this diff Show More