Compare commits

...

121 Commits

Author SHA1 Message Date
Marko Barišić
da9d6dd8e1 Prepare for 2.12.1 patch release (#1524)
* Update license date
* Add override version
2023-11-17 08:17:43 -05:00
Deda
a65a30cfc5 Merge master into release/2.12 2023-11-17 11:30:40 +01:00
Antonio Filipovic
645568a75b Remove default memory limit on procedures (#1506)
* remove default limit on procedures
* fix bug on GraphQL also
2023-11-16 15:01:44 +01:00
Antonio Filipovic
d3f4c35362 Add OOM enabler for MG procedure (#1401) 2023-11-15 12:42:04 +01:00
Josipmrden
c037cddb0e Add granular index and constraint recovery info (#1480) 2023-11-14 17:23:06 -05:00
imilinovic
ced08fd7bc Fix GC by adding periodic jemalloc purge (#1471) 2023-11-14 15:06:21 -05:00
Marko Barišić
9cc060c4b0 Fix error in upload-to-s3 job (#1504) 2023-11-13 13:01:01 +01:00
Marko Barišić
e671a0737e Fix package specific workflow file (#1503) 2023-11-13 12:54:19 +01:00
Marko Barišić
11be3972c4 Add workflow for packaging memgraph for specific target OS (#1502) 2023-11-13 12:20:49 +01:00
DavIvek
fdab42a023 Use static linking on c++ query modules for glibcxx (#1490) 2023-11-13 12:08:48 +01:00
Andi
e5b2c19ea2 Empty Collect() returns nothing (#1482) 2023-11-13 11:45:09 +01:00
Josipmrden
e907817854 Fix for in list segmentation fault (#1494) 2023-11-13 05:17:10 +01:00
Josipmrden
0756cd6898 Add fix indexed join crash (#1478) 2023-11-12 22:12:25 -05:00
Josipmrden
38ad5e2146 Fix parallel index loading (#1479) 2023-11-12 23:51:00 +01:00
Josipmrden
3c413a7e50 Fix hash join expression matching (#1496) 2023-11-12 14:45:02 -05:00
Antonio Filipovic
17915578f8 Fix race condition and arena tracking bug (#1468) 2023-11-09 18:56:36 +01:00
Marko Barišić
4e9a036881 Fix v2.12 release pipeline (#1445) 2023-11-08 07:09:02 -05:00
DavIvek
c8fe9ee7d9 Fix accessing a variable bound to a list within BFS function (#1380) 2023-11-07 20:34:50 +01:00
Ante Javor
e4afddf518 Fix compare results in mgbench (#1319) 2023-11-07 17:04:37 +01:00
Antonio Filipovic
4d5ea03dfa Use extent hooks for memory procedure limit (#1443) 2023-11-07 16:04:29 +01:00
DavIvek
ece4b0dba8 Fix cached plan not getting invalidated (#1348) 2023-11-07 13:34:03 +01:00
Andi
66487a6dce Durability improvements (#1385) 2023-11-07 11:37:54 +01:00
Andi
f4b97fc03d Fix missing statistics for SetPropertiesCursor (#1460) 2023-11-07 09:11:20 +01:00
Antonio Filipovic
58648d1a70 Revert license sender info (#1461) 2023-11-06 16:57:09 +01:00
Antonio Filipovic
1ab7f6ac78 Add notification for user on max map count (#1408) 2023-11-06 15:44:26 +01:00
andrejtonev
dbc6054689 Replication refactor (part 5) (#1378) 2023-11-06 11:50:49 +00:00
Aidar Samerkhanov
16b8c7b27c Fix Kafka flaky unit test (#1409) 2023-11-05 20:51:56 +01:00
Antonio Filipovic
48631d1e37 Rename memory usage and memory allocated (#1426) 2023-11-03 14:40:45 +01:00
Antonio Filipovic
93e6d058d2 Remove all_shortest paths unnecessary logs (#1425) 2023-11-03 12:46:06 +01:00
Andi
3e9f25b8e4 Support creating date and localtime from localdatetime (#1381) 2023-11-03 10:54:01 +01:00
Andi
c94201621a Support deleting paths (#1383) 2023-11-02 14:07:48 +01:00
Andi
fdbc390d53 Throw when reduce inside exists (#1392) 2023-11-02 12:18:15 +01:00
Andi
5e6c5618f5 Make FrameChangeCollector owning the memory resource (#1398) 2023-11-02 09:54:39 +01:00
Andi
4aacd45640 Throw when exists() combined with CASE (#1382) 2023-11-02 08:25:34 +01:00
Gareth Andrew Lloyd
157b36162b Speedup socket unit test (#1444)
Was testing a setup that wasn't used in production, it would
unnecessarily thrash small buffers.
2023-11-01 17:24:24 +00:00
Deda
b2de962a3f Add step to run.sh which deletes /memgraph from builder before copy 2023-10-31 19:22:52 +01:00
Deda
68e79bdaf4 Fix run.sh 2023-10-31 19:18:05 +01:00
Deda
7ee44ae484 Add git clean step to run.sh 2023-10-31 19:13:55 +01:00
Deda
0e0f8fe231 Revert timeout for arm builds back to 120 minutes 2023-10-31 18:13:10 +01:00
Deda
390fc98379 Fix timeout for arm package all builds and nodejs dependency issue for amazon linux 2 2023-10-31 13:04:37 +01:00
Marko Barišić
af4fdef029 Change license date before release (#1433) 2023-10-30 13:01:14 +01:00
Josipmrden
5b9802bd7b Extend property cache to the expression evaluator (#1432)
* Add support for property cache in the produce
* Fix the previous implementation in the map literal
2023-10-28 20:32:58 -07:00
Ante Pušić
b1c3168308 Fix PROFILE infinite loop (#1431) 2023-10-28 15:34:52 +02:00
Andi
011caf3bf1 Fix mgbench daily upload (#1410) 2023-10-28 12:52:14 +02:00
Andi
b1bd977f7b Fix GQL release worklflows (#1424) 2023-10-28 12:50:16 +02:00
gvolfing
c296dc67ce Add index count to index info (#1229) 2023-10-27 18:13:05 +02:00
Ante Pušić
989bb97514 Extend Cypher queries with the index hinting feature (#1345) 2023-10-27 14:26:19 +02:00
Marko Barišić
a94588bde3 Revert CMake default build type error (#1420) 2023-10-27 08:58:37 +02:00
Marko Barišić
e9f3a5fd1b Fix default value in release workflows (#1418) 2023-10-26 16:17:16 +02:00
Marko Budiselić
80e1fba8f5 Update bug_report.md template (#1400) 2023-10-26 13:09:16 +02:00
Ante Pušić
3158a16ffd Add filtering details to EXPLAIN and PROFILE (#1265) 2023-10-25 21:36:20 +02:00
Matija Pintarić
411f8c9d56 Move essential query modules from MAGE to Memgraph (#1384)
* schema.cpp
* mgps.py
* convert.py
2023-10-25 18:27:44 +02:00
Antonio Filipovic
a84f570c6d Use extent hooks for per query memory limit (#1340) 2023-10-25 16:01:59 +02:00
Josipmrden
3d4d841753 Add constraint verification update only on necessary actions (#1341) 2023-10-25 16:01:02 +02:00
Antonio Filipovic
2426d7980d Add OOM enabler in operator tree (#1379) 2023-10-25 12:16:11 +02:00
Josipmrden
7ef10dd82a Fix gql behave dropping connection on Memgraph (#1399) 2023-10-25 10:59:02 +02:00
Gareth Andrew Lloyd
5b91f85161 Improve storage GC (#1387) 2023-10-24 23:41:21 +02:00
Antonio Filipovic
0d9bd5554c Fix potential bug on memory pool (#1299) 2023-10-24 22:31:36 +02:00
Josipmrden
e617ff9b59 Provide textual information for inefficient plans with notifications (#1343) 2023-10-24 22:20:05 +02:00
Josipmrden
be16ca7362 Add cartesian and hash join operators (#1193) 2023-10-24 21:54:42 +02:00
Josipmrden
fdf63436ab Add cartesian and hash join mgbench (#1393) 2023-10-24 19:44:11 +02:00
Josipmrden
4e8148f7d9 Add retry logic possible when conflicting transactions (#1361) 2023-10-24 19:43:23 +02:00
imilinovic
1f118e7521 Add renaming of edge types (#1364) 2023-10-24 17:12:09 +02:00
Marko Damjanić
9803f47828 Update contributing part in the README (#1224) 2023-10-24 14:38:06 +02:00
DavIvek
98680b04c9 Add DNS support for cluster replica address (#1323) 2023-10-24 13:11:36 +02:00
Josipmrden
1d45016217 Add values and keys function to map (#1246) 2023-10-24 06:19:20 +02:00
Matija Pintarić
97ed912ab6 Implement map key exists in mgp (#1336) 2023-10-23 15:29:41 +02:00
gvolfing
aec4c3dd2b Fix bug in alias mappings (#1252) 2023-10-23 13:07:46 +02:00
Antonio Filipovic
7f7f3adfcb Implement jemalloc extent hooks memory tracker (#1250)
Should improve/fix memory usage exceeds --memory-limit issues
2023-10-23 12:48:26 +02:00
andrejtonev
26e31ca06f Fix SHOW CONFIG to show the run-time flag status (#1278) 2023-10-23 10:18:07 +02:00
DavIvek
3ff2c72db9 Fix crash caused by deleting non-existing edge in DETACH DELETE (#1355) 2023-10-23 08:36:28 +02:00
Andi
af56ab6ea8 Forbid changing isolation level for disk and analytical (#1367)
Co-authored-by: Marko Budiselić <marko.budiselic@memgraph.com>
2023-10-23 06:02:56 +02:00
Marko Budiselić
945388fba6 Revert to Release under Diff Release workflow (#1390) 2023-10-22 22:31:06 +02:00
Marko Budiselić
b6693a7df0 Improve systemd config (#1288)
* Add a comment on how to restart automatically
* Add comment on how to deal with out-of-memory (OOM)
2023-10-22 18:44:24 +02:00
Aidar Samerkhanov
667e7f670e Parametrize CI workflows build type (#1324)
The Release is the default, it's also possible to run package_all with
RelWithDebInfo
2023-10-17 00:04:08 +02:00
andrejtonev
a100b900c5 Fix DB configuration error under expansion gbench (#1368) 2023-10-16 23:12:49 +02:00
Gareth Andrew Lloyd
64bf75117b Fix subtle replication bug (#1370)
When going from REPLICA to MAIN, need to ensure current WAL files are
finalised.
2023-10-16 22:56:56 +02:00
Marko Budiselić
9524a51576 Add v5 toolchain (#608)
* clang 17.0.2
* gcc 13.2
* upgrade libs
* tmp disable gpg check, tmp disable fblibs
2023-10-16 19:01:39 +02:00
Marko Budiselić
fd10d1c9f8 Fix CPU 100% usage by websocket error handling improvement (#1327) 2023-10-16 15:41:12 +02:00
andrejtonev
22d8ef75e0 Updated telemetry client-side (#1337) 2023-10-16 14:16:00 +02:00
Andi
7b0bafa21e Add human readable memory allocations in show storage info (#1366) 2023-10-16 11:35:44 +02:00
Andi
de9280b334 Refactor disk storage (#1347) 2023-10-16 09:11:07 +02:00
Kruno Golubic
766ac48261 Update banner in README (#1344) 2023-10-12 14:27:36 +02:00
Andi
06868c8be7 Run separate GQL suits for different storage modes (#1346) 2023-10-11 11:42:41 +02:00
Andi
1a3c5af797 Improve expansions on disk (#1335)
* Improve disk expansions
2023-10-11 10:18:50 +02:00
Gareth Andrew Lloyd
d278a33f31 Decouple pure replication state from storage [part 1] (#1325)
A major refactor to decouple replication state from storage.
ATM it is still owned by storage but a following part should fix that.
2023-10-10 11:44:19 +01:00
Aidar Samerkhanov
7fbf5857f2 Add GQL behave tests for on-disk storage (#1238) 2023-10-10 09:27:11 +03:00
DavIvek
0d51a20a02 Fix a crash caused by declaring a path with only one node in OPTIONAL MATCH clause (#1318) 2023-10-09 15:25:25 +02:00
DavIvek
3143c986de Fix crash caused by using exists() in a RETURN statement (#1303) 2023-10-09 11:31:49 +02:00
Andi
2fd34489af Add mgbench support for disk storage and analytical mode (#1286)
* Add mgbench support for disk storage and analytical mode
2023-10-06 10:19:29 +02:00
Gareth Andrew Lloyd
3cc2bc2791 Refactor interpreter to support multiple distributed clocks (Part 1) (#1281)
* Interpreter transaction ID decoupled from storage transaction ID
* Transactional scope for indices, statistics and constraints
* Storage::Accessor now has 2 modes (unique and shared)
* Introduced ResourceLock to fix pthread mutex problems
* Split InfoQuery in two: non-transactional SystemInfoQuery and transactional DatabaseInfoQuery
* Replicable and durable statistics
* Bumped WAL/Snapshot versions
* Initial implementation of the Lamport clock

---------

Co-authored-by: Andreja Tonev <andreja.tonev@memgraph.io>
2023-10-05 16:58:39 +02:00
Gareth Andrew Lloyd
d71b6a5007 Refactor replication client/server (#1311) 2023-09-29 11:21:42 +01:00
Andi
61ac7e1b11 Add --storage-mode flag (#1282)
* Add --storage-mode flag
2023-09-26 14:47:30 +02:00
Andi
efdf7baea0 Refactor mgbench 2023-09-22 19:05:16 +02:00
Gareth Andrew Lloyd
eb4e2b019d Fix distinct, now doesn't impacts other aggregates (#1235)
Before a distinct on one aggregate would impact distinct on another
aggregate. Fixed the logical error and at the same time did some memory
optimisations.
2023-09-20 16:45:55 +01:00
Andi
1553fcb958 Improve deserialization performance
* Change std::stoull to std::from_chars
---------

Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io>
2023-09-20 14:25:17 +02:00
andrejtonev
bce48361ca Decoupling Interpreter from Storage (#1186)
Unique/global InterpreterContext that is Storage agnostic (has a reference to the DbmsHandler instead)

* InterpreterContext is no longer the owner of Storage
* New Database structure that handles Storage, Triggers, Streams
* Renamed SessinContextHandler to DbmsHandler and simplified the multi-tenant logic
* Added Gatekeeper and updated handlers to use it

---------

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2023-09-20 13:13:54 +02:00
imilinovic
404cdf05d3 Add path pop to mgp API (#1249) 2023-09-19 12:37:55 +02:00
Marko Barišić
b719f0744f Update v2.11 license date (#1247) 2023-09-13 07:19:11 -04:00
Josipmrden
440838c0e9 Add dependency check for e2e tests (#1240) 2023-09-12 11:34:08 -04:00
Josipmrden
79a3c5af8e Add manual performance benchmark execution (#1239) 2023-09-12 11:33:05 -04:00
Josipmrden
bf03b38e39 Remove gqlalchemy from stress tests (#1245) 2023-09-12 11:32:16 -04:00
gvolfing
fd63944493 Add --query-callable-mappings-path package default (#1203) 2023-09-11 15:12:14 -04:00
Gareth Andrew Lloyd
6694de2dfa Fix libkrb5 TRUE and FALSE macros leakage (#1243)
Co-authored-by: Marko Budiselić <marko.budiselic@memgraph.com>
2023-09-11 12:46:40 -04:00
andrejtonev
5e5f4ffc5d Add more runtime configurable settings (#1183)
server name, query timeout settings, log.level, log.to_stderr
2023-09-11 11:30:54 -04:00
Ante Pušić
060b9d1c16 [master < T1204] Add detailed operator info to PROFILE and EXPLAIN (#1204) 2023-09-11 14:34:27 +02:00
Ante Pušić
29a505cb38 Property lookup caching (#1168) 2023-09-11 13:03:54 +02:00
Ante Pušić
d4fcd745d2 Sort SHOW INDEX INFO (#1178) 2023-09-11 10:59:41 +02:00
Josipmrden
58546a9fe1 Add detach delete in bulk (#1078) 2023-09-10 18:53:03 +02:00
Gareth Andrew Lloyd
ab56abf4ca Optimize scanning vertices (#1227) 2023-09-09 10:09:25 -04:00
Gareth Andrew Lloyd
1bd47318cd Improve PropertyStore (#1142)
Improve AnyVersionHasLabelProperty by doing less work in some instances.
Improve FindSpecificProperty.
2023-09-09 08:00:43 -04:00
Ante Pušić
0403b67073 Fix returning NULL on map projection from a null value (#1119) 2023-09-09 06:43:25 -04:00
DavIvek
9e4babcdbb Fix segfault based on issue #874 (#1175) 2023-09-09 02:04:46 -04:00
Antonio Filipovic
b094fdbadc Fix API bug on accessing deleted object (#1209) 2023-09-08 13:52:21 -04:00
Josipmrden
07dea328d8 [master < T1110] Add merge optimization to expand dynamically during runtime (#1110) 2023-09-08 17:12:25 +02:00
Gareth Andrew Lloyd
bd1852f407 Reduce flake SnapshotFallback test (#1237)
Fixed the wait period, this should ensure at least one snapshot was 
made. Also cleaned up the checking around this. And also better 
corruption.
2023-09-08 14:21:35 +01:00
imilinovic
9c51dbbb01 Implement changing from and to vertices in relationships (#1221) 2023-09-08 12:52:40 +02:00
ind1xa
c0d4f5e0bc Add traits under mgp iterator (#1210) 2023-09-08 08:57:37 +02:00
Antonio Filipovic
974a6e3027 Fix bug on mgp dispatcher guard (#1225) 2023-09-07 17:42:27 +02:00
Matija Pintarić
d9464c6ffd Add InDegree and OutDegree in O(1) (#1217) 2023-09-07 13:16:30 +02:00
Ante Javor
312d01bd0c Remove repeated log lines from TRACE log level. (#1054) 2023-09-06 23:09:51 +02:00
662 changed files with 43628 additions and 14765 deletions

View File

@@ -33,4 +33,4 @@ for file in $modified_files; do
fi
done;
return ${FAIL}
exit ${FAIL}

View File

@@ -1,19 +1,18 @@
---
name: Bug report
about: Create a report to help us improve
title: "[BUG] "
title: ""
labels: bug
assignees: gitbuda, antonio2368
assignees: gitbuda
---
**Memgraph version**
Which version did you use?
**Environment**
Some information about the environment you are using Memgraph on: operating
system, how do you connect, with or without docker, which driver etc.
system, architecture (ARM, x86), how do you connect, with or without docker,
which driver etc.
**Describe the bug**
A clear and concise description of what the bug is.
@@ -22,6 +21,7 @@ A clear and concise description of what the bug is.
Steps to reproduce the behavior:
1. Run the following query '...'
2. Click on '....'
3. ... IDEALLY: link to the workload info (DATASET & QUERIES) ...
**Expected behavior**
A clear and concise description of what you expected to happen.
@@ -32,3 +32,11 @@ your problem.
**Additional context**
Add any other context about the problem here.
**Verification Environment**
Once we fix it, what do you need to verify the fix?
Do you need:
* Plain memgraph package -> for which Linux?
* Plain memgraph Docker image?
* Which architecture do you use ARM | x86?
* Full Memgraph platform?

View File

@@ -59,7 +59,7 @@ jobs:
source ve3/bin/activate
pip install -r requirements.txt
./main.py --benchmark-name "macro_benchmark" \
--benchmark-results-path "../../tests/macro_benchmark/.harness_summary" \
--benchmark-results "../../tests/macro_benchmark/.harness_summary" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
@@ -73,6 +73,8 @@ jobs:
./benchmark.py vendor-native --num-workers-for-benchmark 1 --export-results benchmark_high_write_set_property.json high_write_set_property
./benchmark.py vendor-native --num-workers-for-benchmark 12 --export-results cartesian.json cartesian
- name: Upload mgbench results
run: |
cd tools/bench-graph-client
@@ -80,19 +82,25 @@ jobs:
source ve3/bin/activate
pip install -r requirements.txt
./main.py --benchmark-name "mgbench" \
--benchmark-results-path "../../tests/mgbench/benchmark_pokec.json" \
--benchmark-results "../../tests/mgbench/benchmark_pokec.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "supernode" \
--benchmark-results-path "../../tests/mgbench/benchmark_supernode.json" \
--benchmark-results "../../tests/mgbench/benchmark_supernode.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "high_write_set_property" \
--benchmark-results-path "../../tests/mgbench/benchmark_high_write_set_property.json" \
--benchmark-results "../../tests/mgbench/benchmark_high_write_set_property.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "cartesian" \
--benchmark-results "../../tests/mgbench/cartesian.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"

View File

@@ -43,7 +43,7 @@ jobs:
# Build community binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release -DMG_ENTERPRISE=OFF ..
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DMG_ENTERPRISE=OFF ..
make -j$THREADS
- name: Run unit tests
@@ -101,7 +101,7 @@ jobs:
echo ${file}
if [[ ${file} == *.py ]]; then
python3 -m black --check --diff ${file}
python3 -m isort --check-only --diff ${file}
python3 -m isort --profile black --check-only --diff ${file}
fi
done
@@ -229,6 +229,11 @@ jobs:
# branches and tags. (default: 1)
fetch-depth: 0
- name: Check e2e service dependencies
run: |
cd tests/e2e
./dependency_check.sh
- name: Build release binaries
run: |
# Activate toolchain.
@@ -239,12 +244,14 @@ jobs:
# Build release binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release ..
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$THREADS
- name: Run GQL Behave tests
run: |
cd tests/gql_behave
cd tests
./setup.sh /opt/toolchain-v4/activate
cd gql_behave
./continuous_integration
- name: Save quality assurance status
@@ -275,11 +282,13 @@ jobs:
- name: Run stress test (plain)
run: |
cd tests/stress
source ve3/bin/activate
./continuous_integration
- name: Run stress test (SSL)
run: |
cd tests/stress
source ve3/bin/activate
./continuous_integration --use-ssl
- name: Run durability test
@@ -342,7 +351,7 @@ jobs:
./init
# Build only memgraph release binarie.
cd build
cmake -DCMAKE_BUILD_TYPE=release ..
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
make -j$THREADS memgraph
- name: Run Jepsen tests
@@ -410,11 +419,12 @@ jobs:
source ve3/bin/activate
pip install -r requirements.txt
./main.py --benchmark-name "macro_benchmark" \
--benchmark-results-path "../../tests/macro_benchmark/.harness_summary" \
--benchmark-results "../../tests/macro_benchmark/.harness_summary" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
# TODO (andi) No need for path flags and for --disk-storage and --in-memory-analytical
- name: Run mgbench
run: |
cd tests/mgbench
@@ -427,7 +437,7 @@ jobs:
source ve3/bin/activate
pip install -r requirements.txt
./main.py --benchmark-name "mgbench" \
--benchmark-results-path "../../tests/mgbench/benchmark_result.json" \
--benchmark-results "../../tests/mgbench/benchmark_result.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"

View File

@@ -8,6 +8,13 @@ on:
memgraph_version:
description: "Memgraph version to upload as. If empty upload is skipped. Format: 'X.Y.Z'"
required: false
build_type:
type: choice
description: "Memgraph Build type. Default value is Release."
default: 'Release'
options:
- Release
- RelWithDebInfo
jobs:
centos-7:
@@ -20,7 +27,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package centos-7
./release/package/run.sh package centos-7 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -37,7 +44,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package centos-9
./release/package/run.sh package centos-9 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -54,7 +61,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package debian-10
./release/package/run.sh package debian-10 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -71,7 +78,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package debian-11
./release/package/run.sh package debian-11 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -89,7 +96,7 @@ jobs:
- name: "Build package"
run: |
cd release/package
./run.sh package debian-11 --for-docker
./run.sh package debian-11 ${{ github.event.inputs.build_type }} --for-docker
./run.sh docker
- name: "Upload package"
uses: actions/upload-artifact@v3
@@ -107,7 +114,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package ubuntu-18.04
./release/package/run.sh package ubuntu-18.04 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -124,7 +131,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package ubuntu-20.04
./release/package/run.sh package ubuntu-20.04 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -141,7 +148,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package ubuntu-22.04
./release/package/run.sh package ubuntu-22.04 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -158,7 +165,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package debian-11 --for-platform
./release/package/run.sh package debian-11 ${{ github.event.inputs.build_type }} --for-platform
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -175,7 +182,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package fedora-36
./release/package/run.sh package fedora-36 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -192,7 +199,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package amzn-2
./release/package/run.sh package amzn-2 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -209,7 +216,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package debian-11-arm
./release/package/run.sh package debian-11-arm ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
@@ -226,7 +233,7 @@ jobs:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package ubuntu-22.04-arm
./release/package/run.sh package ubuntu-22.04-arm ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:

294
.github/workflows/package_specific.yaml vendored Normal file
View File

@@ -0,0 +1,294 @@
name: Package Specific
# TODO(gitbuda): Cleanup docker container if GHA job was canceled.
on:
workflow_dispatch:
inputs:
memgraph_version:
description: "Memgraph version to upload as. Leave this field empty if you don't want to upload binaries to S3. Format: 'X.Y.Z'"
required: false
build_type:
type: choice
description: "Memgraph Build type. Default value is Release."
default: 'Release'
options:
- Release
- RelWithDebInfo
target_os:
type: choice
description: "Target OS for which memgraph will be packaged. Default is Ubuntu 22.04"
default: 'ubuntu-22_04'
options:
- amzn-2
- centos-7
- centos-9
- debian-10
- debian-11
- debian-11-arm
- debian-11-platform
- docker
- fedora-36
- ubuntu-18_04
- ubuntu-20_04
- ubuntu-22_04
- ubuntu-22_04-arm
jobs:
amzn-2:
if: ${{ github.event.inputs.target_os == 'amzn-2' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package amzn-2 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: amzn-2
path: build/output/amzn-2/memgraph*.rpm
centos-7:
if: ${{ github.event.inputs.target_os == 'centos-7' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package centos-7 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: centos-7
path: build/output/centos-7/memgraph*.rpm
centos-9:
if: ${{ github.event.inputs.target_os == 'centos-9' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package centos-9 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: centos-9
path: build/output/centos-9/memgraph*.rpm
debian-10:
if: ${{ github.event.inputs.target_os == 'debian-10' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package debian-10 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: debian-10
path: build/output/debian-10/memgraph*.deb
debian-11:
if: ${{ github.event.inputs.target_os == 'debian-11' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package debian-11 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: debian-11
path: build/output/debian-11/memgraph*.deb
debian-11-arm:
if: ${{ github.event.inputs.target_os == 'debian-11-arm' }}
runs-on: [self-hosted, DockerMgBuild, ARM64, strange]
timeout-minutes: 120
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package debian-11-arm ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: debian-11-aarch64
path: build/output/debian-11-arm/memgraph*.deb
debian-11-platform:
if: ${{ github.event.inputs.target_os == 'debian-11-platform' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package debian-11 ${{ github.event.inputs.build_type }} --for-platform
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: debian-11-platform
path: build/output/debian-11/memgraph*.deb
docker:
if: ${{ github.event.inputs.target_os == 'docker' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
cd release/package
./run.sh package debian-11 ${{ github.event.inputs.build_type }} --for-docker
./run.sh docker
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: docker
path: build/output/docker/memgraph*.tar.gz
fedora-36:
if: ${{ github.event.inputs.target_os == 'fedora-36' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package fedora-36 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: fedora-36
path: build/output/fedora-36/memgraph*.rpm
ubuntu-18_04:
if: ${{ github.event.inputs.target_os == 'ubuntu-18_04' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package ubuntu-18.04 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: ubuntu-18.04
path: build/output/ubuntu-18.04/memgraph*.deb
ubuntu-20_04:
if: ${{ github.event.inputs.target_os == 'ubuntu-20_04' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package ubuntu-20.04 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: ubuntu-20.04
path: build/output/ubuntu-20.04/memgraph*.deb
ubuntu-22_04:
if: ${{ github.event.inputs.target_os == 'ubuntu-22_04' }}
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package ubuntu-22.04 ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: ubuntu-22.04
path: build/output/ubuntu-22.04/memgraph*.deb
ubuntu-22_04-arm:
if: ${{ github.event.inputs.target_os == 'ubuntu-22_04-arm' }}
runs-on: [self-hosted, DockerMgBuild, ARM64, strange]
timeout-minutes: 120
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package ubuntu-22.04-arm ${{ github.event.inputs.build_type }}
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: ubuntu-22.04-aarch64
path: build/output/ubuntu-22.04-arm/memgraph*.deb
upload-to-s3:
# only run upload if we specified version. Allows for runs without upload
if: "${{ github.event.inputs.memgraph_version != '' }}"
needs: [amzn-2, centos-7, centos-9, debian-10, debian-11, debian-11-arm, debian-11-platform, docker, fedora-36, ubuntu-18_04, ubuntu-20_04, ubuntu-22_04, ubuntu-22_04-arm]
runs-on: ubuntu-latest
steps:
- name: Download artifacts
uses: actions/download-artifact@v3
with:
# name: # if name input parameter is not provided, all artifacts are downloaded
# and put in directories named after each one.
path: build/output/release
- name: Upload to S3
uses: jakejarvis/s3-sync-action@v0.5.1
env:
AWS_S3_BUCKET: "download.memgraph.com"
AWS_ACCESS_KEY_ID: ${{ secrets.S3_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.S3_AWS_SECRET_ACCESS_KEY }}
AWS_REGION: "eu-west-1"
SOURCE_DIR: "build/output/release"
DEST_DIR: "memgraph/v${{ github.event.inputs.memgraph_version }}/"

View File

@@ -0,0 +1,85 @@
name: Run performance benchmarks manually
on:
workflow_dispatch:
jobs:
performance_benchmarks:
name: "Performance benchmarks"
runs-on: [self-hosted, Linux, X64, Diff, Gen7]
env:
THREADS: 24
MEMGRAPH_ENTERPRISE_LICENSE: ${{ secrets.MEMGRAPH_ENTERPRISE_LICENSE }}
MEMGRAPH_ORGANIZATION_NAME: ${{ secrets.MEMGRAPH_ORGANIZATION_NAME }}
steps:
- name: Set up repository
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
fetch-depth: 0
- name: Build release binaries
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Initialize dependencies.
./init
# Build only memgraph release binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release ..
make -j$THREADS
- name: Get branch name (merge)
if: github.event_name != 'pull_request'
shell: bash
run: echo "BRANCH_NAME=$(echo ${GITHUB_REF#refs/heads/} | tr / -)" >> $GITHUB_ENV
- name: Get branch name (pull request)
if: github.event_name == 'pull_request'
shell: bash
run: echo "BRANCH_NAME=$(echo ${GITHUB_HEAD_REF} | tr / -)" >> $GITHUB_ENV
- name: Run benchmarks
run: |
cd tests/mgbench
./benchmark.py vendor-native --num-workers-for-benchmark 12 --export-results benchmark_result.json pokec/medium/*/*
./benchmark.py vendor-native --num-workers-for-benchmark 1 --export-results benchmark_supernode.json supernode
./benchmark.py vendor-native --num-workers-for-benchmark 1 --export-results benchmark_high_write_set_property.json high_write_set_property
./benchmark.py vendor-native --num-workers-for-benchmark 12 --export-results benchmark_cartesian.json cartesian
- name: Upload benchmark results
run: |
cd tools/bench-graph-client
virtualenv -p python3 ve3
source ve3/bin/activate
pip install -r requirements.txt
./main.py --benchmark-name "mgbench" \
--benchmark-results "../../tests/mgbench/benchmark_result.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "supernode" \
--benchmark-results "../../tests/mgbench/benchmark_supernode.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "high_write_set_property" \
--benchmark-results "../../tests/mgbench/benchmark_high_write_set_property.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
./main.py --benchmark-name "cartesian" \
--benchmark-results "../../tests/mgbench/cartesian.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"

View File

@@ -2,6 +2,15 @@ name: Release CentOS 8
on:
workflow_dispatch:
inputs:
build_type:
type: choice
description: "Memgraph Build type. Default value is Release."
default: 'Release'
options:
- Release
- RelWithDebInfo
schedule:
- cron: "0 22 * * *"
@@ -31,9 +40,13 @@ jobs:
# Initialize dependencies.
./init
# Set default build_type to Release
INPUT_BUILD_TYPE=${{ github.event.inputs.build_type }}
BUILD_TYPE=${INPUT_BUILD_TYPE:-"Release"}
# Build community binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release -DMG_ENTERPRISE=OFF ..
cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE -DMG_ENTERPRISE=OFF ..
make -j$THREADS
- name: Run unit tests
@@ -188,9 +201,13 @@ jobs:
# Initialize dependencies.
./init
# Set default build_type to Release
INPUT_BUILD_TYPE=${{ github.event.inputs.build_type }}
BUILD_TYPE=${INPUT_BUILD_TYPE:-"Release"}
# Build release binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release ..
cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE ..
make -j$THREADS
- name: Create enterprise RPM package
@@ -243,7 +260,9 @@ jobs:
- name: Run GQL Behave tests
run: |
cd tests/gql_behave
cd tests
./setup.sh /opt/toolchain-v4/activate
cd gql_behave
./continuous_integration
- name: Save quality assurance status

View File

@@ -2,6 +2,15 @@ name: Release Debian 10
on:
workflow_dispatch:
inputs:
build_type:
type: choice
description: "Memgraph Build type. Default value is Release."
default: 'Release'
options:
- Release
- RelWithDebInfo
schedule:
- cron: "0 22 * * *"
@@ -31,9 +40,13 @@ jobs:
# Initialize dependencies.
./init
# Set default build_type to Release
INPUT_BUILD_TYPE=${{ github.event.inputs.build_type }}
BUILD_TYPE=${INPUT_BUILD_TYPE:-"Release"}
# Build community binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release -DMG_ENTERPRISE=OFF ..
cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE -DMG_ENTERPRISE=OFF ..
make -j$THREADS
- name: Run unit tests
@@ -188,9 +201,13 @@ jobs:
# Initialize dependencies.
./init
# Set default build_type to Release
INPUT_BUILD_TYPE=${{ github.event.inputs.build_type }}
BUILD_TYPE=${INPUT_BUILD_TYPE:-"Release"}
# Build release binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release ..
cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE ..
make -j$THREADS
- name: Create enterprise DEB package
@@ -242,7 +259,9 @@ jobs:
- name: Run GQL Behave tests
run: |
cd tests/gql_behave
cd tests
./setup.sh /opt/toolchain-v4/activate
cd gql_behave
./continuous_integration
- name: Save quality assurance status
@@ -320,9 +339,14 @@ jobs:
source /opt/toolchain-v4/activate
# Initialize dependencies.
./init
# Set default build_type to Release
INPUT_BUILD_TYPE=${{ github.event.inputs.build_type }}
BUILD_TYPE=${INPUT_BUILD_TYPE:-"Release"}
# Build only memgraph release binary.
cd build
cmake -DCMAKE_BUILD_TYPE=release ..
cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE ..
make -j$THREADS memgraph
- name: Run Jepsen tests

View File

@@ -2,6 +2,15 @@ name: Release Ubuntu 20.04
on:
workflow_dispatch:
inputs:
build_type:
type: choice
description: "Memgraph Build type. Default value is Release."
default: 'Release'
options:
- Release
- RelWithDebInfo
schedule:
- cron: "0 22 * * *"
@@ -31,9 +40,13 @@ jobs:
# Initialize dependencies.
./init
# Set default build_type to Release
INPUT_BUILD_TYPE=${{ github.event.inputs.build_type }}
BUILD_TYPE=${INPUT_BUILD_TYPE:-"Release"}
# Build community binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release -DMG_ENTERPRISE=OFF ..
cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE -DMG_ENTERPRISE=OFF ..
make -j$THREADS
- name: Run unit tests
@@ -188,9 +201,13 @@ jobs:
# Initialize dependencies.
./init
# Set default build_type to Release
INPUT_BUILD_TYPE=${{ github.event.inputs.build_type }}
BUILD_TYPE=${INPUT_BUILD_TYPE:-"Release"}
# Build release binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release ..
cmake -DCMAKE_BUILD_TYPE=$BUILD_TYPE ..
make -j$THREADS
- name: Create enterprise DEB package
@@ -242,7 +259,9 @@ jobs:
- name: Run GQL Behave tests
run: |
cd tests/gql_behave
cd tests
./setup.sh /opt/toolchain-v4/activate
cd gql_behave
./continuous_integration
- name: Save quality assurance status

5
.gitignore vendored
View File

@@ -16,8 +16,7 @@
.ycm_extra_conf.pyc
.temp/
Testing/
build
build/
/build*/
release/examples/build
cmake-build-*
cmake/DownloadProject/
@@ -61,3 +60,5 @@ src/storage/distributed/rpc/concurrent_id_mapper_rpc_messages.hpp
src/transactions/distributed/engine_rpc_messages.hpp
/tests/manual/js/transaction_timeout/package-lock.json
/tests/manual/js/transaction_timeout/node_modules/
.vscode/
src/query/frontend/opencypher/grammar/.antlr/*

View File

@@ -15,6 +15,7 @@ repos:
hooks:
- id: isort
name: isort (python)
args: ["--profile", "black"]
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: v13.0.0
hooks:

View File

@@ -42,10 +42,6 @@ endif()
project(memgraph LANGUAGES C CXX)
# NOTE: once in a while this needs to be toggled to check headers are
# correct and PCH isn't masking any include issues
set(CMAKE_DISABLE_PRECOMPILE_HEADERS OFF)
#TODO: upgrade to cmake 3.24 + CheckIPOSupported
#cmake_policy(SET CMP0138 NEW)
#include(CheckIPOSupported)
@@ -68,7 +64,7 @@ option(MG_ENTERPRISE "Build Memgraph Enterprise Edition" ON)
# Set the current version here to override the automatic version detection. The
# version must be specified as `X.Y.Z`. Primarily used when building new patch
# versions.
set(MEMGRAPH_OVERRIDE_VERSION "")
set(MEMGRAPH_OVERRIDE_VERSION "2.12.1")
# Custom suffix that this version should have. The suffix can be any arbitrary
# string. Primarily used when building a version for a specific customer.
@@ -239,7 +235,6 @@ else()
endif()
# -----------------------------------------------------------------------------
# default build type is debug
if (NOT CMAKE_BUILD_TYPE)
set(CMAKE_BUILD_TYPE "Debug")
@@ -297,7 +292,7 @@ if (MG_ENTERPRISE)
add_definitions(-DMG_ENTERPRISE)
endif()
set(ENABLE_JEMALLOC ON)
option(ENABLE_JEMALLOC "Use jemalloc" ON)
if (ASAN)
message(WARNING "Disabling jemalloc as it doesn't work well with ASAN")

View File

@@ -1,5 +1,5 @@
<p align="center">
<img width="400px" src="https://uploads-ssl.webflow.com/5e7ceb09657a69bdab054b3a/5e7ceb09657a6937ab054bba_Black_Original%20_Logo.png">
<img src="https://public-assets.memgraph.com/github-readme-images/github-memgraph-repo-banner.png">
</p>
---
@@ -141,29 +141,20 @@ guide](https://memgraph.com/docs/memgraph/reference-guide/configuration).
## :trophy: Contributing
The main purpose of this repository is to continue evolving Memgraph, making it
faster and easier to use. Development of Memgraph happens in the open on GitHub,
and we are grateful to the community for contributing bug fixes and
improvements. Read below to learn how you can take part in improving Memgraph.
Welcome to the heart of Memgraph development! We're on a mission to supercharge Memgraph, making it faster, more user-friendly, and even more powerful. We owe a big thanks to our fantastic community of contributors who help us fix bugs and bring incredible improvements to life. If you're passionate about databases and open source, here's your chance to make a difference!
### Explore Memgraph Internals
Interested in the nuts and bolts of Memgraph? Our [internals documentation](https://memgraph.notion.site/Memgraph-Internals-12b69132d67a417898972927d6870bd2) is where you can uncover the inner workings of Memgraph's architecture, learn how to build the project from scratch, and discover the secrets of effective contributions. Dive deep into the database!
### Dive into the Contributing Guide
Ready to jump into the action? Explore our [contributing guide](CONTRIBUTING.md) to get the inside scoop on how we develop Memgraph. It's your roadmap for suggesting bug fixes and enhancements. Contribute your skills and ideas!
### Code of Conduct
Memgraph has adopted a Code of Conduct that we expect project participants to
adhere to. Please read [the full text](CODE_OF_CONDUCT.md) so that you can
understand what actions will and will not be tolerated.
Our commitment to a respectful and professional community is unwavering. Every participant in Memgraph is expected to adhere to a stringent Code of Conduct. Please carefully review [the complete text](CODE_OF_CONDUCT.md) to gain a comprehensive understanding of the behaviors that are both expected and explicitly prohibited.
### Contributing Guide
Read our [contributing guide](CONTRIBUTING.md) to learn about our development
process and how to propose bug fixes and improvements.
### Internals
Read our
[internal](https://memgraph.notion.site/Memgraph-Internals-12b69132d67a417898972927d6870bd2)
docs to learn more about Memgraph's architecture, how to build the project from
source and how to start contributing. All information related to the database,
can be found in the aforementioned docs.
We maintain a zero-tolerance policy towards any violations. Our shared commitment to this Code of Conduct ensures that Memgraph remains a place where integrity and excellence are paramount.
### :scroll: License

View File

@@ -1,55 +0,0 @@
# Try to find jemalloc library
#
# Use this module as:
# find_package(Jemalloc)
#
# or:
# find_package(Jemalloc REQUIRED)
#
# This will define the following variables:
#
# Jemalloc_FOUND True if the system has the jemalloc library.
# Jemalloc_INCLUDE_DIRS Include directories needed to use jemalloc.
# Jemalloc_LIBRARIES Libraries needed to link to jemalloc.
#
# The following cache variables may also be set:
#
# Jemalloc_INCLUDE_DIR The directory containing jemalloc/jemalloc.h.
# Jemalloc_LIBRARY The path to the jemalloc static library.
find_path(Jemalloc_INCLUDE_DIR NAMES jemalloc/jemalloc.h PATH_SUFFIXES include)
find_library(Jemalloc_LIBRARY NAMES libjemalloc.a PATH_SUFFIXES lib)
include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(Jemalloc
FOUND_VAR Jemalloc_FOUND
REQUIRED_VARS
Jemalloc_LIBRARY
Jemalloc_INCLUDE_DIR
)
if(Jemalloc_FOUND)
set(Jemalloc_LIBRARIES ${Jemalloc_LIBRARY})
set(Jemalloc_INCLUDE_DIRS ${Jemalloc_INCLUDE_DIR})
else()
if(Jemalloc_FIND_REQUIRED)
message(FATAL_ERROR "Cannot find jemalloc!")
else()
message(WARNING "jemalloc is not found!")
endif()
endif()
if(Jemalloc_FOUND AND NOT TARGET Jemalloc::Jemalloc)
add_library(Jemalloc::Jemalloc UNKNOWN IMPORTED)
set_target_properties(Jemalloc::Jemalloc
PROPERTIES
IMPORTED_LOCATION "${Jemalloc_LIBRARY}"
INTERFACE_INCLUDE_DIRECTORIES "${Jemalloc_INCLUDE_DIR}"
)
endif()
mark_as_advanced(
Jemalloc_INCLUDE_DIR
Jemalloc_LIBRARY
)

67
cmake/Findjemalloc.cmake Normal file
View File

@@ -0,0 +1,67 @@
# Try to find jemalloc library
#
# Use this module as:
# find_package(jemalloc)
#
# or:
# find_package(jemalloc REQUIRED)
#
# This will define the following variables:
#
# JEMALLOC_FOUND True if the system has the jemalloc library.
# Jemalloc_INCLUDE_DIRS Include directories needed to use jemalloc.
# Jemalloc_LIBRARIES Libraries needed to link to jemalloc.
#
# The following cache variables may also be set:
#
# Jemalloc_INCLUDE_DIR The directory containing jemalloc/jemalloc.h.
# Jemalloc_LIBRARY The path to the jemalloc static library.
include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(jemalloc
FOUND_VAR JEMALLOC_FOUND
REQUIRED_VARS
JEMALLOC_LIBRARY
JEMALLOC_INCLUDE_DIR
)
if(JEMALLOC_INCLUDE_DIR)
message(STATUS "Found jemalloc include dir: ${JEMALLOC_INCLUDE_DIR}")
else()
message(WARNING "jemalloc not found!")
endif()
if(JEMALLOC_LIBRARY)
message(STATUS "Found jemalloc library: ${JEMALLOC_LIBRARY}")
else()
message(WARNING "jemalloc library not found!")
endif()
if(JEMALLOC_FOUND)
set(Jemalloc_LIBRARIES ${JEMALLOC_LIBRARY})
set(Jemalloc_INCLUDE_DIRS ${JEMALLOC_INCLUDE_DIR})
else()
if(Jemalloc_FIND_REQUIRED)
message(FATAL_ERROR "Cannot find jemalloc!")
else()
message(WARNING "jemalloc is not found!")
endif()
endif()
if(JEMALLOC_FOUND AND NOT TARGET Jemalloc::Jemalloc)
message(STATUS "JEMALLOC NOT TARGET")
add_library(Jemalloc::Jemalloc UNKNOWN IMPORTED)
set_target_properties(Jemalloc::Jemalloc
PROPERTIES
IMPORTED_LOCATION "${JEMALLOC_LIBRARY}"
INTERFACE_INCLUDE_DIRECTORIES "${JEMALLOC_INCLUDE_DIR}"
)
endif()
mark_as_advanced(
JEMALLOC_INCLUDE_DIR
JEMALLOC_LIBRARY
)

View File

@@ -99,6 +99,10 @@ modifications:
value: "SNAPSHOT_ISOLATION"
override: true
- name: "storage_mode"
value: "IN_MEMORY_TRANSACTIONAL"
override: true
- name: "allow_load_csv"
value: "true"
override: false
@@ -107,6 +111,10 @@ modifications:
value: "false"
override: true
- name: "query_callable_mappings_path"
value: "/etc/memgraph/apoc_compatibility_mappings.json"
override: true
undocumented:
- "flag_file"
- "also_log_to_stderr"

26
config/mappings.json Normal file
View File

@@ -0,0 +1,26 @@
{
"dbms.components": "mgps.components",
"apoc.util.validate": "mgps.validate",
"db.schema.nodeTypeProperties": "schema.NodeTypeOroperties",
"db.schema.relTypeProperties": "schema.RelTypeProperties",
"apoc.coll.contains": "collections.contains",
"apoc.coll.partition": "collections.partition",
"apoc.coll.toSet": "collections.to_set",
"apoc.coll.unionAll": "collections.unionAll",
"apoc.coll.removeAll": "collections.remove_all",
"apoc.coll.union": "collections.union",
"apoc.coll.sum": "collections.sum",
"apoc.coll.pairs": "collections.pairs",
"apoc.map.fromLists": "map.from_lists",
"apoc.map.removeKeys": "map.remove_keys",
"apoc.map.merge": "map.merge",
"apoc.create.nodes": "create.nodes",
"apoc.create.removeProperties": "create.remove_properties",
"apoc.create.node": "create.node",
"apoc.create.removeLabel": "create.remove_label",
"apoc.refactor.invert": "refactor.invert",
"apoc.refactor.cloneNode": "refactor.clone_node",
"apoc.refactor.cloneSubgraph": "refactor.clone_subgraph",
"apoc.refactor.cloneSubgraphFromPath": "refactor.clone_subgraph_from_path",
"apoc.label.exists": "label.exists"
}

View File

@@ -1,2 +0,0 @@
archives
build

View File

@@ -1,5 +1,10 @@
# Memgraph Operating Environments
## Issues related to build toolchain
* GCC 11.2 (toolchain-v4) doesn't compile on Fedora 38, multiple definitions of enum issue
* spdlog 1.10/11 doesn't work with fmt 10.0.0
## os
Under the `os` directory, you can find scripts to install all required system

View File

@@ -18,7 +18,7 @@ TOOLCHAIN_BUILD_DEPS=(
curl libcurl-devel # for cmake
readline-devel # for cmake and llvm
libffi-devel libxml2-devel # for llvm
libedit-devel pcre-devel automake bison # for swig
libedit-devel pcre-devel pcre2-devel automake bison # for swig
file
openssl-devel
gmp-devel
@@ -158,9 +158,9 @@ install() {
continue
fi
if [ "$pkg" == nodejs ]; then
curl -sL https://rpm.nodesource.com/setup_16.x | bash -
if ! yum list installed nodejs >/dev/null 2>/dev/null; then
yum install -y nodejs
yum install https://rpm.nodesource.com/pub_16.x/nodistro/repo/nodesource-release-nodistro-1.noarch.rpm -y
yum install nodejs -y --setopt=nodesource-nodejs.module_hotfixes=1
fi
continue
fi
@@ -172,13 +172,6 @@ install() {
fi
continue
fi
if [ "$pkg" == nodejs ]; then
curl -sL https://rpm.nodesource.com/setup_16.x | bash -
if ! yum list installed nodejs >/dev/null 2>/dev/null; then
yum install -y nodejs
fi
continue
fi
if [ "$pkg" == java-11-openjdk ]; then
amazon-linux-extras install -y java-openjdk11
continue

View File

@@ -20,7 +20,7 @@ TOOLCHAIN_BUILD_DEPS=(
curl # snappy
readline-devel # cmake and llvm
libffi-devel libxml2-devel perl-Digest-MD5 # llvm
libedit-devel pcre-devel automake bison # swig
libedit-devel pcre-devel pcre2-devel automake bison # swig
file
openssl-devel
gmp-devel

View File

@@ -17,7 +17,7 @@ TOOLCHAIN_BUILD_DEPS=(
expat-devel xz-devel python3-devel texinfo libbabeltrace-devel # for gdb
readline-devel # for cmake and llvm
libffi-devel libxml2-devel # for llvm
libedit-devel pcre-devel automake bison # for swig
libedit-devel pcre-devel pcre2-devel automake bison # for swig
file
openssl-devel
gmp-devel

View File

@@ -24,7 +24,7 @@ TOOLCHAIN_BUILD_DEPS=(
libgmp-dev # for gdb
gperf # for proxygen
git # for fbthrift
libedit-dev libpcre3-dev automake bison # for swig
libedit-dev libpcre2-dev libpcre3-dev automake bison # for swig
)
TOOLCHAIN_RUN_DEPS=(

View File

@@ -18,7 +18,7 @@ TOOLCHAIN_BUILD_DEPS=(
libcurl4-openssl-dev # for cmake
libreadline-dev # for cmake and llvm
libffi-dev libxml2-dev # for llvm
libedit-dev libpcre3-dev automake bison # for swig
libedit-dev libpcre2-dev libpcre3-dev automake bison # for swig
curl # snappy
file # for libunwind
libssl-dev # for libevent

View File

@@ -18,7 +18,7 @@ TOOLCHAIN_BUILD_DEPS=(
libcurl4-openssl-dev # for cmake
libreadline-dev # for cmake and llvm
libffi-dev libxml2-dev # for llvm
libedit-dev libpcre3-dev automake bison # for swig
libedit-dev libpcre2-dev libpcre3-dev automake bison # for swig
curl # snappy
file # for libunwind
libssl-dev # for libevent

View File

@@ -18,7 +18,7 @@ TOOLCHAIN_BUILD_DEPS=(
curl libcurl-devel # for cmake
readline-devel # for cmake and llvm
libffi-devel libxml2-devel # for llvm
libedit-devel pcre-devel automake bison # for swig
libedit-devel pcre-devel pcre2-devel automake bison # for swig
file
openssl-devel
gmp-devel

108
environment/os/fedora-38.sh Executable file
View File

@@ -0,0 +1,108 @@
#!/bin/bash
set -Eeuo pipefail
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source "$DIR/../util.sh"
check_operating_system "fedora-38"
check_architecture "x86_64"
TOOLCHAIN_BUILD_DEPS=(
coreutils-common gcc gcc-c++ make # generic build tools
wget # used for archive download
gnupg2 # used for archive signature verification
tar gzip bzip2 xz unzip # used for archive unpacking
zlib-devel # zlib library used for all builds
expat-devel xz-devel python3-devel texinfo libbabeltrace-devel # for gdb
curl libcurl-devel # for cmake
readline-devel # for cmake and llvm
libffi-devel libxml2-devel # for llvm
libedit-devel pcre-devel pcre2-devel automake bison # for swig
file
openssl-devel
gmp-devel
gperf
diffutils
libipt libipt-devel # intel
patch
perl # for openssl
)
TOOLCHAIN_RUN_DEPS=(
make # generic build tools
tar gzip bzip2 xz # used for archive unpacking
zlib # zlib library used for all builds
expat xz-libs python3 # for gdb
readline # for cmake and llvm
libffi libxml2 # for llvm
openssl-devel
)
MEMGRAPH_BUILD_DEPS=(
git # source code control
make pkgconf-pkg-config # build system
wget # for downloading libs
libuuid-devel java-11-openjdk # required by antlr
readline-devel # for memgraph console
python3-devel # for query modules
openssl-devel
libseccomp-devel
python3 python3-pip python3-virtualenv python3-virtualenvwrapper python3-pyyaml nmap-ncat # for tests
libcurl-devel # mg-requests
rpm-build rpmlint # for RPM package building
doxygen graphviz # source documentation generators
which nodejs golang zip unzip java-11-openjdk-devel # for driver tests
sbcl # for custom Lisp C++ preprocessing
autoconf # for jemalloc code generation
libtool # for protobuf code generation
)
list() {
echo "$1"
}
check() {
if [ -v LD_LIBRARY_PATH ]; then
# On Fedora 38 yum/dnf and python11 use newer glibc which is not compatible
# with ours, so we need to momentarely disable env
local OLD_LD_LIBRARY_PATH=${LD_LIBRARY_PATH}
LD_LIBRARY_PATH=""
fi
local missing=""
for pkg in $1; do
if ! dnf list installed "$pkg" >/dev/null 2>/dev/null; then
missing="$pkg $missing"
fi
done
if [ "$missing" != "" ]; then
echo "MISSING PACKAGES: $missing"
exit 1
fi
if [ -v OLD_LD_LIBRARY_PATH ]; then
echo "Restoring LD_LIBRARY_PATH..."
LD_LIBRARY_PATH=${OLD_LD_LIBRARY_PATH}
fi
}
install() {
cd "$DIR"
if [ "$EUID" -ne 0 ]; then
echo "Please run as root."
exit 1
fi
# If GitHub Actions runner is installed, append LANG to the environment.
# Python related tests don't work without the LANG export.
if [ -d "/home/gh/actions-runner" ]; then
echo "LANG=en_US.utf8" >> /home/gh/actions-runner/.env
else
echo "NOTE: export LANG=en_US.utf8"
fi
dnf update -y
for pkg in $1; do
dnf install -y "$pkg"
done
}
deps=$2"[*]"
"$1" "${!deps}"

View File

@@ -25,7 +25,7 @@ TOOLCHAIN_BUILD_DEPS=(
libgmp-dev # for gdb
gperf # for proxygen
libssl-dev
libedit-dev libpcre3-dev automake bison # swig
libedit-dev libpcre2-dev libpcre3-dev automake bison # swig
)
TOOLCHAIN_RUN_DEPS=(

View File

@@ -24,7 +24,7 @@ TOOLCHAIN_BUILD_DEPS=(
libgmp-dev # for gdb
gperf # for proxygen
libssl-dev
libedit-dev libpcre3-dev automake bison # for swig
libedit-dev libpcre2-dev libpcre3-dev automake bison # for swig
)
TOOLCHAIN_RUN_DEPS=(

View File

@@ -24,7 +24,7 @@ TOOLCHAIN_BUILD_DEPS=(
libgmp-dev # for gdb
gperf # for proxygen
libssl-dev
libedit-dev libpcre3-dev automake bison # for swig
libedit-dev libpcre2-dev libpcre3-dev automake bison # for swig
)
TOOLCHAIN_RUN_DEPS=(

View File

@@ -24,7 +24,7 @@ TOOLCHAIN_BUILD_DEPS=(
libgmp-dev # for gdb
gperf # for proxygen
libssl-dev
libedit-dev libpcre3-dev automake bison # for swig
libedit-dev libpcre2-dev libpcre3-dev automake bison # for swig
)
TOOLCHAIN_RUN_DEPS=(

View File

@@ -1 +1,4 @@
archives
build
output
*.tar.gz

View File

@@ -7,7 +7,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
CPUS=$( grep -c processor < /proc/cpuinfo )
cd "$DIR"
source "$DIR/../util.sh"
source "$DIR/../../util.sh"
DISTRO="$(operating_system)"
# toolchain version
@@ -30,10 +30,10 @@ LLVM_VERSION=11.0.0
SWIG_VERSION=4.0.2 # used only for LLVM compilation
# Check for the dependencies.
echo "ALL BUILD PACKAGES: $($DIR/../os/$DISTRO.sh list TOOLCHAIN_BUILD_DEPS)"
$DIR/../os/$DISTRO.sh check TOOLCHAIN_BUILD_DEPS
echo "ALL RUN PACKAGES: $($DIR/../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)"
$DIR/../os/$DISTRO.sh check TOOLCHAIN_RUN_DEPS
echo "ALL BUILD PACKAGES: $($DIR/../../os/$DISTRO.sh list TOOLCHAIN_BUILD_DEPS)"
$DIR/../../os/$DISTRO.sh check TOOLCHAIN_BUILD_DEPS
echo "ALL RUN PACKAGES: $($DIR/../../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)"
$DIR/../../os/$DISTRO.sh check TOOLCHAIN_RUN_DEPS
# check installation directory
NAME=toolchain-v$TOOLCHAIN_VERSION
@@ -442,7 +442,7 @@ In order to be able to run all of these tools you should install the following
packages:
\`\`\`
$($DIR/../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)
$($DIR/../../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)
\`\`\`
## Usage

View File

@@ -7,7 +7,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
CPUS=$( grep -c processor < /proc/cpuinfo )
cd "$DIR"
source "$DIR/../util.sh"
source "$DIR/../../util.sh"
DISTRO="$(operating_system)"
# toolchain version
@@ -31,10 +31,10 @@ LLVM_VERSION_LONG=12.0.1-rc4
SWIG_VERSION=4.0.2 # used only for LLVM compilation
# Check for the dependencies.
echo "ALL BUILD PACKAGES: $($DIR/../os/$DISTRO.sh list TOOLCHAIN_BUILD_DEPS)"
$DIR/../os/$DISTRO.sh check TOOLCHAIN_BUILD_DEPS
echo "ALL RUN PACKAGES: $($DIR/../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)"
$DIR/../os/$DISTRO.sh check TOOLCHAIN_RUN_DEPS
echo "ALL BUILD PACKAGES: $($DIR/../../os/$DISTRO.sh list TOOLCHAIN_BUILD_DEPS)"
$DIR/../../os/$DISTRO.sh check TOOLCHAIN_BUILD_DEPS
echo "ALL RUN PACKAGES: $($DIR/../../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)"
$DIR/../../os/$DISTRO.sh check TOOLCHAIN_RUN_DEPS
# check installation directory
NAME=toolchain-v$TOOLCHAIN_VERSION
@@ -452,7 +452,7 @@ In order to be able to run all of these tools you should install the following
packages:
\`\`\`
$($DIR/../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)
$($DIR/../../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)
\`\`\`
## Usage

View File

@@ -7,7 +7,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
CPUS=$( grep -c processor < /proc/cpuinfo )
cd "$DIR"
source "$DIR/../util.sh"
source "$DIR/../../util.sh"
DISTRO="$(operating_system)"
function log_tool_name () {
@@ -51,17 +51,13 @@ CPPCHECK_VERSION=2.6
LLVM_VERSION=13.0.0
SWIG_VERSION=4.0.2 # used only for LLVM compilation
# Set the right env script
ENV_SCRIPT="$DIR/../os/$DISTRO.sh"
# Set the right operating system setup script.
ENV_SCRIPT="$DIR/../../os/$DISTRO.sh"
if [[ "$for_arm" = true ]]; then
ENV_SCRIPT="$DIR/../os/$DISTRO-arm.sh"
ENV_SCRIPT="$DIR/../../os/$DISTRO-arm.sh"
fi
# Check for the toolchain build dependencies.
echo "ALL BUILD PACKAGES: $(${ENV_SCRIPT} list TOOLCHAIN_BUILD_DEPS)"
${ENV_SCRIPT} check TOOLCHAIN_BUILD_DEPS
# Check for the toolchain run dependencies.
echo "ALL RUN PACKAGES: $(${ENV_SCRIPT} list TOOLCHAIN_RUN_DEPS)"
${ENV_SCRIPT} check TOOLCHAIN_RUN_DEPS
@@ -658,7 +654,7 @@ In order to be able to run all of these tools you should install the following
packages:
\`\`\`
$($DIR/../os/$ENV_SCRIPT.sh list TOOLCHAIN_RUN_DEPS)
$($DIR/../../os/$ENV_SCRIPT.sh list TOOLCHAIN_RUN_DEPS)
\`\`\`
## Usage

View File

@@ -4,7 +4,7 @@ diff -ur a/CMakeLists.txt b/CMakeLists.txt
@@ -52,9 +52,9 @@
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /EHs-c-")
add_definitions(-D_HAS_EXCEPTIONS=0)
- # Disable RTTI.
- string(REGEX REPLACE "/GR" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
- set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /GR-")
@@ -17,7 +17,7 @@ diff -ur a/CMakeLists.txt b/CMakeLists.txt
@@ -77,9 +77,9 @@
string(REGEX REPLACE "-fexceptions" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-exceptions")
- # Disable RTTI.
- string(REGEX REPLACE "-frtti" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
- set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-rtti")
@@ -25,5 +25,5 @@ diff -ur a/CMakeLists.txt b/CMakeLists.txt
+ # string(REGEX REPLACE "-frtti" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
+ # set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-rtti")
endif(CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
# BUILD_SHARED_LIBS is a standard CMake variable, but we declare it here to make

1280
environment/toolchain/v5/build.sh Executable file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,42 @@
#!/bin/bash -ex
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
PREFIX=/opt/toolchain-v5
# NOTE: Often times when versions in the build script are changes, something
# doesn't work. To avoid rebuild of the whole toolchain but rebuild specific
# lib from 0, just comment specific line under this cript and run it. Don't
# forget to comment back to avoid unnecessary deletes next time your run this
# cript.
# rm -rf "$DIR/build"
# rm -rf "$DIR/output"
# rm -rf "$PREFIX/bin/gcc"
# rm -rf "$PREFIX/bin/ld.gold"
# rm -rf "$PREFIX/bin/gdb"
# rm -rf "$PREFIX/bin/cmake"
# rm -rf "$PREFIX/bin/clang"
# rm -rf "$PREFIX/include/bzlib.h"
# rm -rf "$PREFIX/include/fmt"
# rm -rf "$PREFIX/include/lz4.h"
# rm -rf "$PREFIX/include/lzma.h"
# rm -rf "$PREFIX/include/zlib.h"
# rm -rf "$PREFIX/include/zstd.h"
# rm -rf "$PREFIX/include/jemalloc"
# rm -rf "$PREFIX/include/boost"
# rm -rf "$PREFIX/include/double-conversion"
# rm -rf "$PREFIX/include/gflags"
# rm -rf "$PREFIX/include/libunwind.h"
# rm -rf "$PREFIX/include/glog"
# rm -rf "$PREFIX/include/event2"
# rm -rf "$PREFIX/include/sodium.h"
# rm -rf "$PREFIX/include/libaio.h"
# rm -rf "$PREFIX/include/FlexLexer.h"
# rm -rf "$PREFIX/include/snappy.h"
# rm -rf "$PREFIX/include/fizz"
# rm -rf "$PREFIX/include/folly"
# rm -rf "$PREFIX/include/proxygen"
# rm -rf "$PREFIX/include/wangle"
# rm -rf "$PREFIX/include/thrift"
# rm -rf "$PREFIX"

View File

@@ -0,0 +1,41 @@
diff -ur a/folly/CMakeLists.txt b/folly/CMakeLists.txt
--- a/folly/CMakeLists.txt 2021-12-12 23:10:42.000000000 +0100
+++ b/folly/CMakeLists.txt 2022-02-03 15:19:41.349693134 +0100
@@ -28,7 +28,6 @@
)
add_subdirectory(experimental/exception_tracer)
-add_subdirectory(logging/example)
if (PYTHON_EXTENSIONS)
# Create tree of symbolic links in structure required for successful
diff -ur a/folly/experimental/exception_tracer/ExceptionTracerLib.cpp b/folly/experimental/exception_tracer/ExceptionTracerLib.cpp
--- a/folly/experimental/exception_tracer/ExceptionTracerLib.cpp 2021-12-12 23:10:42.000000000 +0100
+++ b/folly/experimental/exception_tracer/ExceptionTracerLib.cpp 2022-02-03 15:19:11.003368891 +0100
@@ -96,6 +96,7 @@
#define __builtin_unreachable()
#endif
+#if 0
namespace __cxxabiv1 {
void __cxa_throw(
@@ -154,5 +155,5 @@
}
} // namespace std
-
+#endif
#endif // defined(__GLIBCXX__)
diff -ur a/folly/Portability.h b/folly/Portability.h
--- a/folly/Portability.h 2021-12-12 23:10:42.000000000 +0100
+++ b/folly/Portability.h 2022-02-03 15:19:11.003368891 +0100
@@ -566,7 +566,7 @@
#define FOLLY_HAS_COROUTINES 0
#elif (__cpp_coroutines >= 201703L || __cpp_impl_coroutine >= 201902L) && \
(__has_include(<coroutine>) || __has_include(<experimental/coroutine>))
-#define FOLLY_HAS_COROUTINES 1
+#define FOLLY_HAS_COROUTINES 0
// This is mainly to workaround bugs triggered by LTO, when stack allocated
// variables in await_suspend end up on a coroutine frame.
#define FOLLY_CORO_AWAIT_SUSPEND_NONTRIVIAL_ATTRIBUTES FOLLY_NOINLINE

View File

@@ -0,0 +1,26 @@
diff --git a/folly/CMakeLists.txt b/folly/CMakeLists.txt
index e0e16df..471131e 100644
--- a/folly/CMakeLists.txt
+++ b/folly/CMakeLists.txt
@@ -28,7 +28,7 @@ install(
)
add_subdirectory(experimental/exception_tracer)
-add_subdirectory(logging/example)
+# add_subdirectory(logging/example)
if (PYTHON_EXTENSIONS)
# Create tree of symbolic links in structure required for successful
diff --git a/folly/Portability.h b/folly/Portability.h
index 365ef1b..42d24b8 100644
--- a/folly/Portability.h
+++ b/folly/Portability.h
@@ -560,7 +560,7 @@ constexpr auto kCpplibVer = 0;
(defined(__cpp_coroutines) && __cpp_coroutines >= 201703L) || \
(defined(__cpp_impl_coroutine) && __cpp_impl_coroutine >= 201902L)) && \
(__has_include(<coroutine>) || __has_include(<experimental/coroutine>))
-#define FOLLY_HAS_COROUTINES 1
+#define FOLLY_HAS_COROUTINES 0
// This is mainly to workaround bugs triggered by LTO, when stack allocated
// variables in await_suspend end up on a coroutine frame.
#define FOLLY_CORO_AWAIT_SUSPEND_NONTRIVIAL_ATTRIBUTES FOLLY_NOINLINE

View File

@@ -0,0 +1,29 @@
diff -ur a/CMakeLists.txt b/CMakeLists.txt
--- a/CMakeLists.txt 2021-05-05 00:53:34.000000000 +0200
+++ b/CMakeLists.txt 2022-01-27 17:18:34.758302398 +0100
@@ -52,9 +52,9 @@
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /EHs-c-")
add_definitions(-D_HAS_EXCEPTIONS=0)
- # Disable RTTI.
- string(REGEX REPLACE "/GR" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
- set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /GR-")
+ # # Disable RTTI.
+ # string(REGEX REPLACE "/GR" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
+ # set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /GR-")
else(CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
# Use -Wall for clang and gcc.
if(NOT CMAKE_CXX_FLAGS MATCHES "-Wall")
@@ -77,9 +77,9 @@
string(REGEX REPLACE "-fexceptions" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-exceptions")
- # Disable RTTI.
- string(REGEX REPLACE "-frtti" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
- set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-rtti")
+ # # Disable RTTI.
+ # string(REGEX REPLACE "-frtti" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
+ # set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-rtti")
endif(CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
# BUILD_SHARED_LIBS is a standard CMake variable, but we declare it here to make

View File

@@ -0,0 +1,75 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBEzEOZIBEACxg/IuXERlDB48JBWmF4NxNUuuup1IhJAJyFGFSKh3OGAO2Ard
sNuRLjANsFXA7m7P5eTFcG+BoHHuAVYmKnI3PPZtHVLnUt4pGItPczQZ2BE1WpcI
ayjGTBJeKItX3Npqg9D/odO9WWS1i3FQPVdrLn0YH37/BA66jeMQCRo7g7GLpaNf
IrvYGsqTbxCwsmA37rpE7oyU4Yrf74HT091WBsRIoq/MelhbxTDMR8eu/dUGZQVc
Kj3lN55RepwWwUUKyqarY0zMt4HkFJ7v7yRL+Cvzy92Ouv4Wf2FlhNtEs5LE4Tax
W0PO5AEmUoKjX87SezQK0f652018b4u6Ex52cY7p+n5TII/UyoowH6+tY8UHo9yb
fStrqgNE/mY2bhA6+AwCaOUGsFzVVPTbjtxL3HacUP/jlA1h78V8VTvTs5d55iG7
jSqR9o05wje8rwNiXXK0xtiJahyNzL97Kn/DgPSqPIi45G+8nxWSPFM5eunBKRl9
vAnsvwrdPRsR6YR3uMHTuVhQX9/CY891MHkaZJ6wydWtKt3yQwJLYqwo5d4DwnUX
CduUwSKv+6RmtWI5ZmTQYOcBRcZyGKml9X9Q8iSbm6cnpFXmLrNQwCJN+D3SiYGc
MtbltZo0ysPMa6Xj5xFaYqWk/BI4iLb2Gs+ByGo/+a0Eq4XYBMOpitNniQARAQAB
tCdMYXNzZSBDb2xsaW4gPGxhc3NlLmNvbGxpbkB0dWthYW5pLm9yZz6JAlEEEwEK
ADsCGwMCHgECF4AECwkIBwMVCggFFgIDAQAWIQQ2kMJAzlG0Zw0wrRw47nV9aRhG
IAUCYEt9dQUJFxeR4wAKCRA47nV9aRhGIBNDEACxD6vJ+enZwe3IgkJh5JtLsC9b
MWCQRlPW1EVMsg96Cb5Rtron1eN1pp1TlzENJu1/C7C/VEsr9WwOPg26Men7fNf/
O21QM9IBWd/uB0Pu333WqKh92ESS5x9ST9DrG39nVGSPkQQBMuia72VrA+crPnwT
/h/u1IN6/sff5VDIU24rUiqW2Npy733dANruj7Ny0scRXVPltnVdhqwPHt6qNjC1
t+/cCnwHgW1BR1RYXBPpB42z/m29dL9rPrG0YPGWs2Bc+EATUICfEE6eIvwfciue
IJTjKT9Y9DrogJC2AYFhjC7N04OKdCB2hFs4BjexJwr4X0GJO7LhFl03c951AsIE
GHwrucRPB5bo2vmvQ8IvZn7CmtdUJzXv9JlyU6p+MIK1pz7TK6GgSOSffQIXZn6e
nUPtm9mEwuncOfmW8/ODYPs1gCWYgyiFJx8h7eEu+M4MxHSFBs7MwXf/Ae2fSp+M
P/p198qB8fC5oVBnF95qb0Qi0uc1D+Gb+gpBF+ymMb+s/VBOR3QWiym7AzBrJ62g
UnbC9jMLGnSRI+7p7raUfMTgXr5/oQoBw7ExJVltSSRrim2YH/t4CV47mO6dR9J3
1RtsTFIRNhz+07XPsETcuCV/dgqeC8fOFLt9MY17Sufhb1DcGy4urZBOIhXcpTV7
vHVj5IYH5nYOT49NRYkCOAQTAQIAIgUCTMQ5kgIbAwYLCQgHAwIGFQgCCQoLBBYC
AwECHgECF4AACgkQOO51fWkYRiAg4A/7BXKwoRaXrMbMPOW7vuVF7c2IKB2Yqzn1
vLBCwuEHkqY237lDcXY4/5LR+1gcZ3Duw1n/BRSm0FBdvyX/JTWiWNSDUkKAO/0l
T2Tg44YLrDT3bzwu8dbU9xQt6kH+SCOHvv5Oe4k79l5mro6fF3H1M0bN63x/YoFY
ojy09D7/JptY82oR4f/VdKnfZLJcCViCb0wp8SD2NkDAudKg+K+7PD8HlTWklQQg
TZdRXxVZKIJeU42aJDqnRbAhJd64YHyClhqut9F5LUmiP5qfLfNhkKDhNOwk2Blr
BGBJkSd7wPyzcX4Mun/L6YspHjbeVMt9TD7HQlo+OOd2OjAHCx6pqwkXnzeLPEaE
cPdQ1SHgrBViAxX3DNPubLP0Knw8XwFu96EuhHZgexE1W7bB4LFsJyXAc5k1PqPD
CLsAauxmvI2OfI7opG/8wyxDvNgoPjG8fZNAgY0REqPC0JnTXChH31IxUmhNotH8
tD3DDTZOHw05n5MwwUrEE9xiETVDfFQcMLfxZ9KLz+BC2g1t5LYublRgnCMNJzFg
sNUMM02CphABzl/LCLnumr0eyQQ/weV4twEhLwSDmqLYHL0EdYW0Y3CnnU9vmYxQ
cXKbstS71sEJJYBBmSBbf9GxkOY8BRNtwVwY0kPgxv1WqdVBiAFvfB+pyAsrax9B
3UeB7ZSwRD6JAhwEEAEKAAYFAlS25GwACgkQlbYYGy0z6ew92Q//ZA9/6piQtoW4
PwP/1DtWGyKU8hwR+9FG669iPk/dAG+yoEJtFMOUpg/FUFmCX8Bc4oEHsCVyLxKt
DcCVUIRcYNSFi5hTZaBEbwsOlDT37gtlfIIu34hhHRccKaLnN/N9gNMNw8wGh9xg
Q/KtxZwcbk/bZIlDkKTJkFBRAekdEGAFDWb/AZOy+LQxS8ZAh1eWkfV0i8opmK9k
gPXtLE0WSsqtYyGs58z+BFE9NH3tEUwK6jSvtuLwQl4UrICNbKthcpb8WwH6UXzb
q3QNSYVOpf/cqRdBJA6bvb/ku/xyKVL08lGmxD9v1b137R7mafDAFPTsvH2Mt/0V
YuhtWav3r1Bl9QksDxt2DTS8wiWDUBetGqOVdcw7vBrXPEWDNBmxeJXsiJ7zJlR+
9wrJOm6RV2+l1IPxu96EaPS+kTNBijKrhxb67bww8BTEWTd0wcdJmgWRkM8SIstp
IKqd0L2TFYph2/NtrBhRg+DIEPJPpSTGsUMcCEXCZPQ+cIdlQKsWpk0tZ62DlvEl
r7E+wgUSQolRfx5KrpZifiS2zQlhzdXv28CJhsVbLyw5fUAWUKIH/dCo5NKsNLk2
Lc5DH9VWnFgxAAtW290FqeK/4ulMq7Vs1dQSwyHM2Ni3QqqeaiOrh8gbSY5CMLFN
Y3HYRwuTYPa3AobsozCzBj0Zdf/6AFe5Ag0ETMQ5kgEQAL/FwKdjxgPxtSpgq1SM
zgZtTTyLqhgGD3NZfadHWHYRIL38NDV3JeTA79Y2zj2dj7KQPDT+0aqeizTV2E3j
P3iCQ53VOT4consBaQAgKexpptnS+T1DobtICFJ0GGzf0HRj6KO2zSOuOitWPWlU
wbvX7M0LLI2+hqlx0jTPqbJFZ/Za6KTtbS6xdCPVUpUqYZQpokEZcwQmUp8Q+lGo
JD2sNYCZyap63X/aAOgCGr2RXYddOH5e8vGzGW+mwtCv+WQ9Ay35mGqI5MqkbZd1
Qbuv2b1647E/QEEucfRHVbJVKGGPpFMUJtcItyyIt5jo+r9CCL4Cs47dF/9/RNwu
NvpvHXUyqMBQdWNZRMx4k/NGD/WviPi9m6mIMui6rOQsSOaqYdcUX4Nq2Orr3Oaz
2JPQdUfeI23iot1vK8hxvUCQTV3HfJghizN6spVl0yQOKBiE8miJRgrjHilH3hTb
xoo42xDkNAq+CQo3QAm1ibDxKCDq0RcWPjcCRAN/Q5MmpcodpdKkzV0yGIS4g7s5
frVrgV/kox2r4/Yxsr8K909+4H82AjTKGX/BmsQFCTAqBk6p7I0zxjIqJ/w33TZB
Q0Pn4r3WIlUPafzY6a9/LAvN1fHRxf9SpCByJsszD03Qu5f5TB8gthsdnVmTo7jj
iordEKMtw2aEMLzdWWTQ/TNVABEBAAGJAjwEGAEKACYCGwwWIQQ2kMJAzlG0Zw0w
rRw47nV9aRhGIAUCYEt9YAUJFxeRzgAKCRA47nV9aRhGIMLtD/9HuKM4pngImcuz
YwzQmdv4j26YYyh4jVsKEmVWTiRcehEgUIlrWkCu3qzd5NK+RetS7kJ8MPnzEUfj
YbpdC6yrF6n1mSrZZ4VJMkV2ev37bIgXM+Wp1mCAGbjNxQnjn9RabT/gjIqmGuRn
AP7RsSeOSuO/gO9h2Pteciz23ussTilB+8cTooQEQQZe6Kv/zukvL+ccSehLHsZ7
qVfRUAmtt8nFkXXE+s8jfLfhqstaI2/RJu5witaPcXM8Mnz2E95aASAbZy0eQot9
0Pvf07n9yuC3tueTvzvlXx3h5U3yT44tIOmzANIQjay1TGdm+RBJ2ZYyhyLawlZ2
NVUXXSp4QZZXPA0UWbF+pb7Q9cdKDNFVuvGBljuea0Yd0T2o+ibDq43HziX9ll+l
SXk9mqvW1UcDOaxWrSsm1Gc1O9g3wqH5xHAhtY8GPh/7VgAawskPkmnlkMW6pYPy
zibbeISJL1gd1jIT63y6aoVrtNoo+wYJm280ROflh4+5QOo6QJ+jm70fkXSG/qJ5
a8/qCPTHkJc/rpkL6/TDQAJURi9RhDAC0gb40HtusbN1LZEA+i0cWTmYXap+DB4Y
R4pApilpaG87M+VUokR4xpnx7vTb2MPa7Mdenvi9FEGnKXadmT8038vlfzz5GGUT
MlVin9BQPTpdA+PpRiJvKJgVDeAFOg==
=asTC
-----END PGP PUBLIC KEY BLOCK-----

View File

@@ -255,6 +255,21 @@ inline mgp_edge *graph_create_edge(mgp_graph *graph, mgp_vertex *from, mgp_verte
return MgInvoke<mgp_edge *>(mgp_graph_create_edge, graph, from, to, type, memory);
}
inline mgp_edge *graph_edge_set_from(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_vertex *new_from,
mgp_memory *memory) {
return MgInvoke<mgp_edge *>(mgp_graph_edge_set_from, graph, e, new_from, memory);
}
inline mgp_edge *graph_edge_set_to(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_vertex *new_to,
mgp_memory *memory) {
return MgInvoke<mgp_edge *>(mgp_graph_edge_set_to, graph, e, new_to, memory);
}
inline mgp_edge *graph_edge_change_type(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_edge_type new_type,
mgp_memory *memory) {
return MgInvoke<mgp_edge *>(mgp_graph_edge_change_type, graph, e, new_type, memory);
}
inline void graph_delete_edge(mgp_graph *graph, mgp_edge *edge) { MgInvokeVoid(mgp_graph_delete_edge, graph, edge); }
inline mgp_vertex *graph_get_vertex_by_id(mgp_graph *g, mgp_vertex_id id, mgp_memory *memory) {
@@ -345,6 +360,8 @@ inline size_t map_size(mgp_map *map) { return MgInvoke<size_t>(mgp_map_size, map
inline mgp_value *map_at(mgp_map *map, const char *key) { return MgInvoke<mgp_value *>(mgp_map_at, map, key); }
inline bool key_exists(mgp_map *map, const char *key) { return MgInvoke<int>(mgp_key_exists, map, key); }
inline const char *map_item_key(mgp_map_item *item) { return MgInvoke<const char *>(mgp_map_item_key, item); }
inline mgp_value *map_item_value(mgp_map_item *item) { return MgInvoke<mgp_value *>(mgp_map_item_value, item); }
@@ -367,6 +384,10 @@ inline mgp_map_item *map_items_iterator_next(mgp_map_items_iterator *it) {
inline mgp_vertex_id vertex_get_id(mgp_vertex *v) { return MgInvoke<mgp_vertex_id>(mgp_vertex_get_id, v); }
inline size_t vertex_get_in_degree(mgp_vertex *v) { return MgInvoke<size_t>(mgp_vertex_get_in_degree, v); }
inline size_t vertex_get_out_degree(mgp_vertex *v) { return MgInvoke<size_t>(mgp_vertex_get_out_degree, v); }
inline mgp_vertex *vertex_copy(mgp_vertex *v, mgp_memory *memory) {
return MgInvoke<mgp_vertex *>(mgp_vertex_copy, v, memory);
}
@@ -463,6 +484,8 @@ inline void path_destroy(mgp_path *path) { mgp_path_destroy(path); }
inline void path_expand(mgp_path *path, mgp_edge *edge) { MgInvokeVoid(mgp_path_expand, path, edge); }
inline void path_pop(mgp_path *path) { MgInvokeVoid(mgp_path_pop, path); }
inline size_t path_size(mgp_path *path) { return MgInvoke<size_t>(mgp_path_size, path); }
inline mgp_vertex *path_vertex_at(mgp_path *path, size_t index) {

View File

@@ -333,6 +333,13 @@ class Path:
self._vertices.append(edge.end_id)
self._edges.append((edge.start_id, edge.end_id, edge.id))
def pop(self):
if not self._edges:
raise IndexError("Path contains no relationships.")
self._vertices.pop()
self._edges.pop()
def vertex_at(self, index: int) -> Vertex:
return Vertex(self._vertices[index], self._graph)

View File

@@ -111,6 +111,22 @@ enum mgp_error mgp_global_aligned_alloc(size_t size_in_bytes, size_t alignment,
/// The behavior is undefined if `ptr` is not a value returned from a prior
/// mgp_global_alloc() or mgp_global_aligned_alloc().
void mgp_global_free(void *p);
/// State of the graph database.
struct mgp_graph;
/// Allocations are tracked only for master thread. If new threads are spawned
/// inside procedure, by calling following function
/// you can start tracking allocations for current thread too. This
/// is important if you need query memory limit to work
/// for given procedure or per procedure memory limit.
enum mgp_error mgp_track_current_thread_allocations(struct mgp_graph *graph);
/// Once allocations are tracked for current thread, you need to stop tracking allocations
/// for given thread, before thread finishes with execution, or is detached.
/// Otherwise it might result in slowdown of system due to unnecessary tracking of
/// allocations.
enum mgp_error mgp_untrack_current_thread_allocations(struct mgp_graph *graph);
///@}
/// @name Operations on mgp_value
@@ -482,6 +498,9 @@ enum mgp_error mgp_map_size(struct mgp_map *map, size_t *result);
/// Result is NULL if no mapping exists.
enum mgp_error mgp_map_at(struct mgp_map *map, const char *key, struct mgp_value **result);
/// Returns true if key in map.
enum mgp_error mgp_key_exists(struct mgp_map *map, const char *key, int *result);
/// An item in the mgp_map.
struct mgp_map_item;
@@ -543,6 +562,10 @@ void mgp_path_destroy(struct mgp_path *path);
/// Return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE if unable to allocate memory for path extension.
enum mgp_error mgp_path_expand(struct mgp_path *path, struct mgp_edge *edge);
/// Remove the last node and the last relationship from the path.
/// Return mgp_error::MGP_ERROR_OUT_OF_RANGE if the path contains no relationships.
enum mgp_error mgp_path_pop(struct mgp_path *path);
/// Get the number of edges in a mgp_path.
/// Current implementation always returns without errors.
enum mgp_error mgp_path_size(struct mgp_path *path, size_t *result);
@@ -647,6 +670,12 @@ struct mgp_vertex_id {
/// Get the ID of given vertex.
enum mgp_error mgp_vertex_get_id(struct mgp_vertex *v, struct mgp_vertex_id *result);
/// Get the in degree of given vertex.
enum mgp_error mgp_vertex_get_in_degree(struct mgp_vertex *v, size_t *result);
/// Get the out degree of given vertex.
enum mgp_error mgp_vertex_get_out_degree(struct mgp_vertex *v, size_t *result);
/// Result is non-zero if the vertex can be modified.
/// The mutability of the vertex is the same as the graph which it is part of. If a vertex is immutable, then edges
/// cannot be created or deleted, properties and labels cannot be set or removed and all of the returned edges will be
@@ -841,9 +870,6 @@ enum mgp_error mgp_edge_set_properties(struct mgp_edge *e, struct mgp_map *prope
enum mgp_error mgp_edge_iter_properties(struct mgp_edge *e, struct mgp_memory *memory,
struct mgp_properties_iterator **result);
/// State of the graph database.
struct mgp_graph;
/// Get the vertex corresponding to given ID, or NULL if no such vertex exists.
/// Resulting vertex must be freed using mgp_vertex_destroy.
/// Return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE if unable to allocate the vertex.
@@ -882,6 +908,29 @@ enum mgp_error mgp_graph_detach_delete_vertex(struct mgp_graph *graph, struct mg
enum mgp_error mgp_graph_create_edge(struct mgp_graph *graph, struct mgp_vertex *from, struct mgp_vertex *to,
struct mgp_edge_type type, struct mgp_memory *memory, struct mgp_edge **result);
/// Change edge from vertex
/// Return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT if `graph` is immutable.
/// Return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE if unable to allocate a mgp_edge.
/// Return mgp_error::MGP_ERROR_DELETED_OBJECT if `from` or `to` has been deleted.
/// Return mgp_error::MGP_ERROR_SERIALIZATION_ERROR if `from` or `to` has been modified by another transaction.
enum mgp_error mgp_graph_edge_set_from(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_vertex *new_from,
struct mgp_memory *memory, struct mgp_edge **result);
/// Change edge to vertex
/// Return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT if `graph` is immutable.
/// Return mgp_error::MGP_ERROR_UNABLE_TO_ALLOCATE if unable to allocate a mgp_edge.
/// Return mgp_error::MGP_ERROR_DELETED_OBJECT if `from` or `to` has been deleted.
/// Return mgp_error::MGP_ERROR_SERIALIZATION_ERROR if `from` or `to` has been modified by another transaction.
enum mgp_error mgp_graph_edge_set_to(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_vertex *new_to,
struct mgp_memory *memory, struct mgp_edge **result);
/// Change edge type
/// Return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT if `graph` is immutable.
/// Return mgp_error::MGP_ERROR_SERIALIZATION_ERROR if `edge`, its source or destination vertex has been modified by
/// another transaction.
enum mgp_error mgp_graph_edge_change_type(struct mgp_graph *graph, struct mgp_edge *e, struct mgp_edge_type new_type,
struct mgp_memory *memory, struct mgp_edge **result);
/// Delete an edge from the graph.
/// Return mgp_error::MGP_ERROR_IMMUTABLE_OBJECT if `graph` is immutable.
/// Return mgp_error::MGP_ERROR_SERIALIZATION_ERROR if `edge`, its source or destination vertex has been modified by

View File

@@ -126,6 +126,12 @@ class MemoryDispatcher final {
map_.erase(this_id);
}
bool IsThisThreadRegistered() noexcept {
const auto this_id = std::this_thread::get_id();
std::shared_lock lock(mut_);
return map_.contains(this_id);
}
private:
std::unordered_map<std::thread::id, mgp_memory *> map_;
std::shared_mutex mut_;
@@ -136,7 +142,7 @@ class MemoryDispatcher final {
// header. The use of the 'mgp_memory *memory' pointer is deprecated
// and will be removed in upcoming releases.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
inline extern MemoryDispatcher mrd{};
inline MemoryDispatcher mrd{};
// TODO - Once we deprecate this we should remove this
// and make sure nothing relies on it anymore. This alone
@@ -164,7 +170,7 @@ class MemoryDispatcherGuard final {
// the mapping instead.
template <typename Func, typename... Args>
inline decltype(auto) MemHandlerCallback(Func &&func, Args &&...args) {
if (memory) {
if (!mrd.IsThisThreadRegistered()) {
return std::forward<Func>(func)(std::forward<Args>(args)..., memory);
}
return std::forward<Func>(func)(std::forward<Args>(args)..., mrd.GetMemoryResource());
@@ -250,6 +256,12 @@ class Graph {
void DetachDeleteNode(const Node &node);
/// @brief Creates a relationship of type `type` between nodes `from` and `to` and adds it to the graph.
Relationship CreateRelationship(const Node &from, const Node &to, const std::string_view type);
/// @brief Changes a relationship from node.
void SetFrom(Relationship &relationship, const Node &new_from);
/// @brief Changes a relationship to node.
void SetTo(Relationship &relationship, const Node &new_to);
/// @brief Changes the relationship type.
void ChangeType(Relationship &relationship, std::string_view new_type);
/// @brief Deletes a relationship from the graph.
void DeleteRelationship(const Relationship &relationship);
@@ -429,6 +441,12 @@ class Labels {
friend class Labels;
public:
using value_type = Labels;
using difference_type = std::ptrdiff_t;
using pointer = const Labels *;
using reference = const Labels &;
using iterator_category = std::forward_iterator_tag;
bool operator==(const Iterator &other) const;
bool operator!=(const Iterator &other) const;
@@ -512,6 +530,12 @@ class List {
friend class List;
public:
using value_type = List;
using difference_type = std::ptrdiff_t;
using pointer = const List *;
using reference = const List &;
using iterator_category = std::forward_iterator_tag;
bool operator==(const Iterator &other) const;
bool operator!=(const Iterator &other) const;
@@ -608,6 +632,9 @@ class Map {
/// @brief Returns the value at the given `key`.
Value const At(std::string_view key) const;
/// @brief Returns true if the given `key` exists.
bool KeyExists(std::string_view key) const;
class Iterator {
public:
friend class Map;
@@ -752,6 +779,12 @@ class Node {
/// @brief returns the string representation
const std::string ToString() const;
/// @brief returns the in degree of a node
inline size_t InDegree() const;
/// @brief returns the out degree of a node
inline size_t OutDegree() const;
private:
mgp_vertex *ptr_;
};
@@ -858,6 +891,8 @@ class Path {
/// @brief Adds a relationship continuing from the last node on the path.
void Expand(const Relationship &relationship);
/// @brief Removes the last node and the last relationship from the path.
void Pop();
/// @exception std::runtime_error Path contains element(s) with unknown value.
bool operator==(const Path &other) const;
@@ -1985,6 +2020,25 @@ inline Relationship Graph::CreateRelationship(const Node &from, const Node &to,
return relationship;
}
inline void Graph::SetFrom(Relationship &relationship, const Node &new_from) {
mgp_edge *edge = mgp::MemHandlerCallback(mgp::graph_edge_set_from, graph_, relationship.ptr_, new_from.ptr_);
relationship = Relationship(edge);
mgp::edge_destroy(edge);
}
inline void Graph::SetTo(Relationship &relationship, const Node &new_to) {
mgp_edge *edge = mgp::MemHandlerCallback(mgp::graph_edge_set_to, graph_, relationship.ptr_, new_to.ptr_);
relationship = Relationship(edge);
mgp::edge_destroy(edge);
}
inline void Graph::ChangeType(Relationship &relationship, std::string_view new_type) {
mgp_edge *edge = mgp::MemHandlerCallback(mgp::graph_edge_change_type, graph_, relationship.ptr_,
mgp_edge_type{.name = new_type.data()});
relationship = Relationship(edge);
mgp::edge_destroy(edge);
}
inline void Graph::DeleteRelationship(const Relationship &relationship) {
mgp::graph_delete_edge(graph_, relationship.ptr_);
}
@@ -2531,6 +2585,8 @@ inline const Value Map::At(std::string_view key) const {
return Value();
}
inline bool Map::KeyExists(std::string_view key) const { return mgp::key_exists(ptr_, key.data()); }
inline Map::Iterator::Iterator(mgp_map_items_iterator *map_items_iterator) : map_items_iterator_(map_items_iterator) {
if (map_items_iterator_ == nullptr) return;
if (mgp::map_items_iterator_get(map_items_iterator_) == nullptr) {
@@ -2790,6 +2846,10 @@ inline const std::string Node::ToString() const {
return "(id: " + std::to_string(Id().AsInt()) + labels + ", properties: {" + properties + "})";
}
inline size_t Node::InDegree() const { return mgp::vertex_get_in_degree(ptr_); }
inline size_t Node::OutDegree() const { return mgp::vertex_get_out_degree(ptr_); }
// Relationship:
inline Relationship::Relationship(mgp_edge *ptr) : ptr_(mgp::MemHandlerCallback(edge_copy, ptr)) {}
@@ -2951,6 +3011,8 @@ inline Relationship Path::GetRelationshipAt(size_t index) const {
inline void Path::Expand(const Relationship &relationship) { mgp::path_expand(ptr_, relationship.ptr_); }
inline void Path::Pop() { mgp::path_pop(ptr_); }
inline bool Path::operator==(const Path &other) const { return util::PathsEqual(ptr_, other.ptr_); }
inline bool Path::operator!=(const Path &other) const { return !(*this == other); }

View File

@@ -983,6 +983,24 @@ class Path:
self._vertices = None
self._edges = None
def pop(self):
"""
Remove the last node and the last relationship from the path.
Raises:
InvalidContextError: If using an invalid `Path` instance
OutOfRangeError: If the path contains no relationships.
Examples:
```path.pop()```
"""
if not self.is_valid():
raise InvalidContextError()
self._path.pop()
# Invalidate our cached tuples
self._vertices = None
self._edges = None
@property
def vertices(self) -> typing.Tuple[Vertex, ...]:
"""
@@ -1023,6 +1041,10 @@ class Path:
self._edges = tuple(Edge(self._path.edge_at(i)) for i in range(num_edges))
return self._edges
@property
def length(self) -> int:
return self._path.size()
class Record:
"""Represents a record of resulting field values."""

View File

@@ -929,6 +929,25 @@ class Path:
self._vertices = None
self._edges = None
def pop(self):
"""
Remove the last node and the last relationship from the path.
Raises:
InvalidContextError: If using an invalid `Path` instance
OutOfRangeError: If the path contains no relationships.
Examples:
```path.pop()```
"""
if not self.is_valid():
raise InvalidContextError()
self._path.pop()
# Invalidate cached tuples
self._vertices = None
self._edges = None
@property
def vertices(self) -> typing.Tuple[Vertex, ...]:
"""

View File

@@ -4,7 +4,8 @@ include(GNUInstallDirs)
include(ProcessorCount)
ProcessorCount(NPROC)
if (NPROC EQUAL 0)
if(NPROC EQUAL 0)
set(NPROC 1)
endif()
@@ -12,10 +13,10 @@ find_package(Boost 1.78 REQUIRED)
find_package(BZip2 1.0.6 REQUIRED)
find_package(Threads REQUIRED)
set(GFLAGS_NOTHREADS OFF)
# NOTE: config/generate.py depends on the gflags help XML format.
find_package(gflags REQUIRED)
find_package(fmt 8.0.1)
find_package(Jemalloc REQUIRED)
find_package(ZLIB 1.2.11 REQUIRED)
set(LIB_DIR ${CMAKE_CURRENT_SOURCE_DIR})
@@ -24,24 +25,27 @@ set(LIB_DIR ${CMAKE_CURRENT_SOURCE_DIR})
function(import_header_library name include_dir)
add_library(${name} INTERFACE IMPORTED GLOBAL)
set_property(TARGET ${name} PROPERTY
INTERFACE_INCLUDE_DIRECTORIES ${include_dir})
INTERFACE_INCLUDE_DIRECTORIES ${include_dir})
string(TOUPPER ${name} _upper_name)
set(${_upper_name}_INCLUDE_DIR ${include_dir} CACHE FILEPATH
"Path to ${name} include directory" FORCE)
"Path to ${name} include directory" FORCE)
mark_as_advanced(${_upper_name}_INCLUDE_DIR)
add_library(lib::${name} ALIAS ${name})
endfunction(import_header_library)
function(import_library name type location include_dir)
add_library(${name} ${type} IMPORTED GLOBAL)
if (${ARGN})
if(${ARGN})
# Optional argument is the name of the external project that we need to
# depend on.
add_dependencies(${name} ${ARGN0})
else()
add_dependencies(${name} ${name}-proj)
endif()
set_property(TARGET ${name} PROPERTY IMPORTED_LOCATION ${location})
# We need to create the include directory first in order to be able to add it
# as an include directory. The header files in the include directory will be
# generated later during the build process.
@@ -61,29 +65,34 @@ function(add_external_project name)
set(options NO_C_COMPILER)
set(one_value_kwargs SOURCE_DIR BUILD_IN_SOURCE)
set(multi_value_kwargs CMAKE_ARGS DEPENDS INSTALL_COMMAND BUILD_COMMAND
CONFIGURE_COMMAND)
CONFIGURE_COMMAND)
cmake_parse_arguments(KW "${options}" "${one_value_kwargs}" "${multi_value_kwargs}" ${ARGN})
set(source_dir ${CMAKE_CURRENT_SOURCE_DIR}/${name})
if (KW_SOURCE_DIR)
if(KW_SOURCE_DIR)
set(source_dir ${KW_SOURCE_DIR})
endif()
set(build_in_source 0)
if (KW_BUILD_IN_SOURCE)
if(KW_BUILD_IN_SOURCE)
set(build_in_source ${KW_BUILD_IN_SOURCE})
endif()
if (NOT KW_NO_C_COMPILER)
if(NOT KW_NO_C_COMPILER)
set(KW_CMAKE_ARGS -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} ${KW_CMAKE_ARGS})
endif()
ExternalProject_Add(${name}-proj DEPENDS ${KW_DEPENDS}
PREFIX ${source_dir} SOURCE_DIR ${source_dir}
BUILD_IN_SOURCE ${build_in_source}
CONFIGURE_COMMAND ${KW_CONFIGURE_COMMAND}
CMAKE_ARGS -DCMAKE_BUILD_TYPE=Release
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DCMAKE_INSTALL_PREFIX=${source_dir}
${KW_CMAKE_ARGS}
INSTALL_COMMAND ${KW_INSTALL_COMMAND}
BUILD_COMMAND ${KW_BUILD_COMMAND})
PREFIX ${source_dir} SOURCE_DIR ${source_dir}
BUILD_IN_SOURCE ${build_in_source}
CONFIGURE_COMMAND ${KW_CONFIGURE_COMMAND}
CMAKE_ARGS -DCMAKE_BUILD_TYPE=Release
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DCMAKE_INSTALL_PREFIX=${source_dir}
${KW_CMAKE_ARGS}
INSTALL_COMMAND ${KW_INSTALL_COMMAND}
BUILD_COMMAND ${KW_BUILD_COMMAND})
endfunction(add_external_project)
# Calls `add_external_project`, sets NAME_LIBRARY, NAME_INCLUDE_DIR variables
@@ -92,23 +101,34 @@ macro(import_external_library name type library_location include_dir)
add_external_project(${name} ${ARGN})
string(TOUPPER ${name} _upper_name)
set(${_upper_name}_LIBRARY ${library_location} CACHE FILEPATH
"Path to ${name} library" FORCE)
"Path to ${name} library" FORCE)
set(${_upper_name}_INCLUDE_DIR ${include_dir} CACHE FILEPATH
"Path to ${name} include directory" FORCE)
"Path to ${name} include directory" FORCE)
mark_as_advanced(${_upper_name}_LIBRARY ${_upper_name}_INCLUDE_DIR)
import_library(${name} ${type} ${${_upper_name}_LIBRARY} ${${_upper_name}_INCLUDE_DIR})
endmacro(import_external_library)
macro(set_path_external_library name type library_location include_dir)
string(TOUPPER ${name} _upper_name)
set(${_upper_name}_LIBRARY ${library_location} CACHE FILEPATH
"Path to ${name} library" FORCE)
set(${_upper_name}_INCLUDE_DIR ${include_dir} CACHE FILEPATH
"Path to ${name} include directory" FORCE)
mark_as_advanced(${name}_LIBRARY ${name}_INCLUDE_DIR)
endmacro(set_path_external_library)
# setup antlr
import_external_library(antlr4 STATIC
${CMAKE_CURRENT_SOURCE_DIR}/antlr4/runtime/Cpp/lib/libantlr4-runtime.a
${CMAKE_CURRENT_SOURCE_DIR}/antlr4/runtime/Cpp/include/antlr4-runtime
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/antlr4/runtime/Cpp
CMAKE_ARGS # http://stackoverflow.com/questions/37096062/get-a-basic-c-program-to-compile-using-clang-on-ubuntu-16/38385967#38385967
-DWITH_LIBCXX=OFF # because of debian bug
-DCMAKE_SKIP_INSTALL_ALL_DEPENDENCY=true
-DCMAKE_CXX_STANDARD=20
-DANTLR_BUILD_CPP_TESTS=OFF
-DWITH_LIBCXX=OFF # because of debian bug
-DCMAKE_SKIP_INSTALL_ALL_DEPENDENCY=true
-DCMAKE_CXX_STANDARD=20
-DANTLR_BUILD_CPP_TESTS=OFF
BUILD_COMMAND $(MAKE) antlr4_static
INSTALL_COMMAND $(MAKE) install)
@@ -116,6 +136,7 @@ import_external_library(antlr4 STATIC
import_external_library(benchmark STATIC
${CMAKE_CURRENT_SOURCE_DIR}/benchmark/${CMAKE_INSTALL_LIBDIR}/libbenchmark.a
${CMAKE_CURRENT_SOURCE_DIR}/benchmark/include
# Skip testing. The tests don't compile with Clang 8.
CMAKE_ARGS -DBENCHMARK_ENABLE_TESTING=OFF)
@@ -131,15 +152,15 @@ add_subdirectory(rapidcheck EXCLUDE_FROM_ALL)
# setup google test
add_external_project(gtest SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/googletest)
set(GTEST_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/googletest/include
CACHE PATH "Path to gtest and gmock include directory" FORCE)
CACHE PATH "Path to gtest and gmock include directory" FORCE)
set(GMOCK_LIBRARY ${CMAKE_CURRENT_SOURCE_DIR}/googletest/lib/libgmock.a
CACHE FILEPATH "Path to gmock library" FORCE)
CACHE FILEPATH "Path to gmock library" FORCE)
set(GMOCK_MAIN_LIBRARY ${CMAKE_CURRENT_SOURCE_DIR}/googletest/lib/libgmock_main.a
CACHE FILEPATH "Path to gmock_main library" FORCE)
CACHE FILEPATH "Path to gmock_main library" FORCE)
set(GTEST_LIBRARY ${CMAKE_CURRENT_SOURCE_DIR}/googletest/lib/libgtest.a
CACHE FILEPATH "Path to gtest library" FORCE)
CACHE FILEPATH "Path to gtest library" FORCE)
set(GTEST_MAIN_LIBRARY ${CMAKE_CURRENT_SOURCE_DIR}/googletest/lib/libgtest_main.a
CACHE FILEPATH "Path to gtest_main library" FORCE)
CACHE FILEPATH "Path to gtest_main library" FORCE)
mark_as_advanced(GTEST_INCLUDE_DIR GMOCK_LIBRARY GMOCK_MAIN_LIBRARY GTEST_LIBRARY GTEST_MAIN_LIBRARY)
import_library(gtest STATIC ${GTEST_LIBRARY} ${GTEST_INCLUDE_DIR} gtest-proj)
import_library(gtest_main STATIC ${GTEST_MAIN_LIBRARY} ${GTEST_INCLUDE_DIR} gtest-proj)
@@ -157,10 +178,10 @@ import_external_library(rocksdb STATIC
${CMAKE_CURRENT_SOURCE_DIR}/rocksdb/lib/librocksdb.a
${CMAKE_CURRENT_SOURCE_DIR}/rocksdb/include
CMAKE_ARGS -DUSE_RTTI=ON
-DWITH_TESTS=OFF
-DGFLAGS_NOTHREADS=OFF
-DCMAKE_INSTALL_LIBDIR=lib
-DCMAKE_SKIP_INSTALL_ALL_DEPENDENCY=true
-DWITH_TESTS=OFF
-DGFLAGS_NOTHREADS=OFF
-DCMAKE_INSTALL_LIBDIR=lib
-DCMAKE_SKIP_INSTALL_ALL_DEPENDENCY=true
BUILD_COMMAND $(MAKE) rocksdb)
# Setup libbcrypt
@@ -169,8 +190,8 @@ import_external_library(libbcrypt STATIC
${CMAKE_CURRENT_SOURCE_DIR}/libbcrypt
CONFIGURE_COMMAND sed s/-Wcast-align// -i ${CMAKE_CURRENT_SOURCE_DIR}/libbcrypt/crypt_blowfish/Makefile
BUILD_COMMAND make -C ${CMAKE_CURRENT_SOURCE_DIR}/libbcrypt
CC=${CMAKE_C_COMPILER}
CXX=${CMAKE_CXX_COMPILER}
CC=${CMAKE_C_COMPILER}
CXX=${CMAKE_CXX_COMPILER}
INSTALL_COMMAND true)
# Setup mgclient
@@ -178,16 +199,16 @@ import_external_library(mgclient STATIC
${CMAKE_CURRENT_SOURCE_DIR}/mgclient/lib/libmgclient.a
${CMAKE_CURRENT_SOURCE_DIR}/mgclient/include
CMAKE_ARGS -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DBUILD_TESTING=OFF
-DBUILD_CPP_BINDINGS=ON)
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DBUILD_TESTING=OFF
-DBUILD_CPP_BINDINGS=ON)
find_package(OpenSSL REQUIRED)
target_link_libraries(mgclient INTERFACE ${OPENSSL_LIBRARIES})
add_external_project(mgconsole
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/mgconsole
CMAKE_ARGS
-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_BINARY_DIR}
-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_BINARY_DIR}
BUILD_COMMAND $(MAKE) mgconsole)
add_custom_target(mgconsole DEPENDS mgconsole-proj)
@@ -204,14 +225,15 @@ import_external_library(librdkafka STATIC
${CMAKE_CURRENT_SOURCE_DIR}/librdkafka/lib/librdkafka.a
${CMAKE_CURRENT_SOURCE_DIR}/librdkafka/include
CMAKE_ARGS -DRDKAFKA_BUILD_STATIC=ON
-DRDKAFKA_BUILD_EXAMPLES=OFF
-DRDKAFKA_BUILD_TESTS=OFF
-DWITH_ZSTD=OFF
-DENABLE_LZ4_EXT=OFF
-DCMAKE_INSTALL_LIBDIR=lib
-DWITH_SSL=ON
# If we want SASL, we need to install it on build machines
-DWITH_SASL=OFF)
-DRDKAFKA_BUILD_EXAMPLES=OFF
-DRDKAFKA_BUILD_TESTS=OFF
-DWITH_ZSTD=OFF
-DENABLE_LZ4_EXT=OFF
-DCMAKE_INSTALL_LIBDIR=lib
-DWITH_SSL=ON
# If we want SASL, we need to install it on build machines
-DWITH_SASL=OFF)
target_link_libraries(librdkafka INTERFACE ${OPENSSL_LIBRARIES} ZLIB::ZLIB)
import_library(librdkafka++ STATIC
@@ -232,24 +254,24 @@ import_external_library(pulsar STATIC
${CMAKE_CURRENT_SOURCE_DIR}/pulsar/install/include
BUILD_IN_SOURCE 1
CONFIGURE_COMMAND cmake pulsar-client-cpp
-DCMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_SOURCE_DIR}/pulsar/install
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
-DBUILD_DYNAMIC_LIB=OFF
-DBUILD_STATIC_LIB=ON
-DBUILD_TESTS=OFF
-DLINK_STATIC=ON
-DPROTOC_PATH=${PROTOBUF_ROOT}/bin/protoc
-DBOOST_ROOT=${BOOST_ROOT}
-DCMAKE_PREFIX_PATH=${PROTOBUF_ROOT}
-DProtobuf_INCLUDE_DIRS=${PROTOBUF_ROOT}/include
-DBUILD_PYTHON_WRAPPER=OFF
-DBUILD_PERF_TOOLS=OFF
-DUSE_LOG4CXX=OFF
BUILD_COMMAND $(MAKE) pulsarStaticWithDeps)
-DCMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_SOURCE_DIR}/pulsar/install
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
-DBUILD_DYNAMIC_LIB=OFF
-DBUILD_STATIC_LIB=ON
-DBUILD_TESTS=OFF
-DLINK_STATIC=ON
-DPROTOC_PATH=${PROTOBUF_ROOT}/bin/protoc
-DBOOST_ROOT=${BOOST_ROOT}
-DCMAKE_PREFIX_PATH=${PROTOBUF_ROOT}
-DProtobuf_INCLUDE_DIRS=${PROTOBUF_ROOT}/include
-DBUILD_PYTHON_WRAPPER=OFF
-DBUILD_PERF_TOOLS=OFF
-DUSE_LOG4CXX=OFF
BUILD_COMMAND $(MAKE) pulsarStaticWithDeps)
add_dependencies(pulsar-proj protobuf)
if (${MG_ARCH} STREQUAL "ARM64")
if(${MG_ARCH} STREQUAL "ARM64")
set(MG_LIBRDTSC_CMAKE_ARGS -DLIBRDTSC_ARCH_x86=OFF -DLIBRDTSC_ARCH_ARM64=ON)
endif()
@@ -265,3 +287,10 @@ import_header_library(ctre ${CMAKE_CURRENT_SOURCE_DIR})
# setup absl (cmake sub_directory tolerant)
set(ABSL_PROPAGATE_CXX_STD ON)
add_subdirectory(absl EXCLUDE_FROM_ALL)
# set Jemalloc
set_path_external_library(jemalloc STATIC
${CMAKE_CURRENT_SOURCE_DIR}/jemalloc/lib/libjemalloc.a
${CMAKE_CURRENT_SOURCE_DIR}/jemalloc/include/)
import_header_library(rangev3 ${CMAKE_CURRENT_SOURCE_DIR}/rangev3/include)

View File

@@ -124,6 +124,8 @@ declare -A primary_urls=(
["librdtsc"]="http://$local_cache_host/git/librdtsc.git"
["ctre"]="http://$local_cache_host/file/hanickadot/compile-time-regular-expressions/v3.7.2/single-header/ctre.hpp"
["absl"]="https://$local_cache_host/git/abseil-cpp.git"
["jemalloc"]="https://$local_cache_host/git/jemalloc.git"
["range-v3"]="https://$local_cache_host/git/ericniebler/range-v3.git"
)
# The goal of secondary urls is to have links to the "source of truth" of
@@ -151,6 +153,8 @@ declare -A secondary_urls=(
["librdtsc"]="https://github.com/gabrieleara/librdtsc.git"
["ctre"]="https://raw.githubusercontent.com/hanickadot/compile-time-regular-expressions/v3.7.2/single-header/ctre.hpp"
["absl"]="https://github.com/abseil/abseil-cpp.git"
["jemalloc"]="https://github.com/jemalloc/jemalloc.git"
["range-v3"]="https://github.com/ericniebler/range-v3.git"
)
# antlr
@@ -252,3 +256,24 @@ cd ..
# abseil 20230125.3
absl_ref="20230125.3"
repo_clone_try_double "${primary_urls[absl]}" "${secondary_urls[absl]}" "absl" "$absl_ref"
# jemalloc ea6b3e973b477b8061e0076bb257dbd7f3faa756
JEMALLOC_COMMIT_VERSION="5.2.1"
repo_clone_try_double "${secondary_urls[jemalloc]}" "${secondary_urls[jemalloc]}" "jemalloc" "$JEMALLOC_COMMIT_VERSION"
# this is hack for cmake in libs to set path, and for FindJemalloc to use Jemalloc_INCLUDE_DIR
pushd jemalloc
./autogen.sh
MALLOC_CONF="retain:false,percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:5000,dirty_decay_ms:5000" \
./configure \
--disable-cxx \
--enable-shared=no --prefix=$working_dir \
--with-malloc-conf="retain:false,percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:5000,dirty_decay_ms:5000"
make -j$CPUS install
popd
#range-v3 release-0.12.0
range_v3_ref="release-0.12.0"
repo_clone_try_double "${primary_urls[range-v3]}" "${secondary_urls[range-v3]}" "rangev3" "$range_v3_ref"

View File

@@ -36,7 +36,7 @@ ADDITIONAL USE GRANT: You may use the Licensed Work in accordance with the
3. using the Licensed Work to create a work or solution
which competes (or might reasonably be expected to
compete) with the Licensed Work.
CHANGE DATE: 2027-02-08
CHANGE DATE: 2027-17-11
CHANGE LICENSE: Apache License, Version 2.0
For information about alternative licensing arrangements, please visit: https://memgraph.com/legal.

View File

@@ -13,6 +13,7 @@ string(TOLOWER ${CMAKE_BUILD_TYPE} lower_build_type)
add_library(example_c SHARED example.c)
target_include_directories(example_c PRIVATE ${CMAKE_SOURCE_DIR}/include)
target_compile_options(example_c PRIVATE -Wall)
target_link_libraries(example_c PRIVATE -static-libgcc -static-libstdc++)
# Strip C example in release build.
if (lower_build_type STREQUAL "release")
add_custom_command(TARGET example_c POST_BUILD
@@ -28,6 +29,7 @@ install(FILES example.c DESTINATION lib/memgraph/query_modules/src)
add_library(example_cpp SHARED example.cpp)
target_include_directories(example_cpp PRIVATE ${CMAKE_SOURCE_DIR}/include)
target_compile_options(example_cpp PRIVATE -Wall)
target_link_libraries(example_cpp PRIVATE -static-libgcc -static-libstdc++)
# Strip C++ example in release build.
if (lower_build_type STREQUAL "release")
add_custom_command(TARGET example_cpp POST_BUILD
@@ -40,9 +42,27 @@ install(PROGRAMS $<TARGET_FILE:example_cpp>
# Also install the source of the example, so user can read it.
install(FILES example.cpp DESTINATION lib/memgraph/query_modules/src)
add_library(schema SHARED schema.cpp)
target_include_directories(schema PRIVATE ${CMAKE_SOURCE_DIR}/include)
target_compile_options(schema PRIVATE -Wall)
target_link_libraries(schema PRIVATE -static-libgcc -static-libstdc++)
# Strip C++ example in release build.
if (lower_build_type STREQUAL "release")
add_custom_command(TARGET schema POST_BUILD
COMMAND strip -s $<TARGET_FILE:schema>
COMMENT "Stripping symbols and sections from the C++ schema module")
endif()
install(PROGRAMS $<TARGET_FILE:schema>
DESTINATION lib/memgraph/query_modules
RENAME schema.so)
# Also install the source of the example, so user can read it.
install(FILES schema.cpp DESTINATION lib/memgraph/query_modules/src)
# Install the Python example and modules
install(FILES example.py DESTINATION lib/memgraph/query_modules RENAME py_example.py)
install(FILES graph_analyzer.py DESTINATION lib/memgraph/query_modules)
install(FILES mgp_networkx.py DESTINATION lib/memgraph/query_modules)
install(FILES nxalg.py DESTINATION lib/memgraph/query_modules)
install(FILES wcc.py DESTINATION lib/memgraph/query_modules)
install(FILES mgps.py DESTINATION lib/memgraph/query_modules)
install(FILES convert.py DESTINATION lib/memgraph/query_modules)

10
query_modules/convert.py Normal file
View File

@@ -0,0 +1,10 @@
from json import loads
import mgp
@mgp.function
def str2object(string: str) -> mgp.Any:
if string:
return loads(string)
return None

8
query_modules/mgps.py Normal file
View File

@@ -0,0 +1,8 @@
import mgp
@mgp.read_proc
def components(
context: mgp.ProcCtx,
) -> mgp.Record(versions=list, edition=str, name=str):
return mgp.Record(versions=["5.9.0"], edition="community", name="Memgraph")

187
query_modules/schema.cpp Normal file
View File

@@ -0,0 +1,187 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <mgp.hpp>
namespace Schema {
/*NodeTypeProperties and RelTypeProperties constants*/
constexpr std::string_view kReturnNodeType = "nodeType";
constexpr std::string_view kProcedureNodeType = "node_type_properties";
constexpr std::string_view kProcedureRelType = "rel_type_properties";
constexpr std::string_view kReturnLabels = "nodeLabels";
constexpr std::string_view kReturnRelType = "relType";
constexpr std::string_view kReturnPropertyName = "propertyName";
constexpr std::string_view kReturnPropertyType = "propertyTypes";
constexpr std::string_view kReturnMandatory = "mandatory";
std::string TypeOf(const mgp::Type &type);
template <typename T>
void ProcessPropertiesNode(mgp::Record &record, const std::string &type, const mgp::List &labels,
const std::string &propertyName, const T &propertyType, const bool &mandatory);
template <typename T>
void ProcessPropertiesRel(mgp::Record &record, const std::string_view &type, const std::string &propertyName,
const T &propertyType, const bool &mandatory);
void NodeTypeProperties(mgp_list *args, mgp_graph *memgraph_graph, mgp_result *result, mgp_memory *memory);
void RelTypeProperties(mgp_list *args, mgp_graph *memgraph_graph, mgp_result *result, mgp_memory *memory);
} // namespace Schema
/*we have << operator for type in Cpp API, but in it we return somewhat different strings than I would like in this
module, so I implemented a small function here*/
std::string Schema::TypeOf(const mgp::Type &type) {
switch (type) {
case mgp::Type::Null:
return "Null";
case mgp::Type::Bool:
return "Bool";
case mgp::Type::Int:
return "Int";
case mgp::Type::Double:
return "Double";
case mgp::Type::String:
return "String";
case mgp::Type::List:
return "List[Any]";
case mgp::Type::Map:
return "Map[Any]";
case mgp::Type::Node:
return "Vertex";
case mgp::Type::Relationship:
return "Edge";
case mgp::Type::Path:
return "Path";
case mgp::Type::Date:
return "Date";
case mgp::Type::LocalTime:
return "LocalTime";
case mgp::Type::LocalDateTime:
return "LocalDateTime";
case mgp::Type::Duration:
return "Duration";
default:
throw mgp::ValueException("Unsupported type");
}
}
template <typename T>
void Schema::ProcessPropertiesNode(mgp::Record &record, const std::string &type, const mgp::List &labels,
const std::string &propertyName, const T &propertyType, const bool &mandatory) {
record.Insert(std::string(kReturnNodeType).c_str(), type);
record.Insert(std::string(kReturnLabels).c_str(), labels);
record.Insert(std::string(kReturnPropertyName).c_str(), propertyName);
record.Insert(std::string(kReturnPropertyType).c_str(), propertyType);
record.Insert(std::string(kReturnMandatory).c_str(), mandatory);
}
template <typename T>
void Schema::ProcessPropertiesRel(mgp::Record &record, const std::string_view &type, const std::string &propertyName,
const T &propertyType, const bool &mandatory) {
record.Insert(std::string(kReturnRelType).c_str(), type);
record.Insert(std::string(kReturnPropertyName).c_str(), propertyName);
record.Insert(std::string(kReturnPropertyType).c_str(), propertyType);
record.Insert(std::string(kReturnMandatory).c_str(), mandatory);
}
void Schema::NodeTypeProperties(mgp_list *args, mgp_graph *memgraph_graph, mgp_result *result, mgp_memory *memory) {
mgp::MemoryDispatcherGuard guard{memory};
;
const auto record_factory = mgp::RecordFactory(result);
try {
const mgp::Graph graph = mgp::Graph(memgraph_graph);
for (auto node : graph.Nodes()) {
std::string type = "";
mgp::List labels = mgp::List();
for (auto label : node.Labels()) {
labels.AppendExtend(mgp::Value(label));
type += ":`" + std::string(label) + "`";
}
if (node.Properties().size() == 0) {
auto record = record_factory.NewRecord();
ProcessPropertiesNode<std::string>(record, type, labels, "", "", false);
continue;
}
for (auto &[key, prop] : node.Properties()) {
auto property_type = mgp::List();
auto record = record_factory.NewRecord();
property_type.AppendExtend(mgp::Value(TypeOf(prop.Type())));
ProcessPropertiesNode<mgp::List>(record, type, labels, key, property_type, true);
}
}
} catch (const std::exception &e) {
record_factory.SetErrorMessage(e.what());
return;
}
}
void Schema::RelTypeProperties(mgp_list *args, mgp_graph *memgraph_graph, mgp_result *result, mgp_memory *memory) {
mgp::MemoryDispatcherGuard guard{memory};
;
const auto record_factory = mgp::RecordFactory(result);
try {
const mgp::Graph graph = mgp::Graph(memgraph_graph);
for (auto rel : graph.Relationships()) {
std::string type = ":`" + std::string(rel.Type()) + "`";
if (rel.Properties().size() == 0) {
auto record = record_factory.NewRecord();
ProcessPropertiesRel<std::string>(record, type, "", "", false);
continue;
}
for (auto &[key, prop] : rel.Properties()) {
auto property_type = mgp::List();
auto record = record_factory.NewRecord();
property_type.AppendExtend(mgp::Value(TypeOf(prop.Type())));
ProcessPropertiesRel<mgp::List>(record, type, key, property_type, true);
}
}
} catch (const std::exception &e) {
record_factory.SetErrorMessage(e.what());
return;
}
}
extern "C" int mgp_init_module(struct mgp_module *module, struct mgp_memory *memory) {
try {
mgp::MemoryDispatcherGuard guard{memory};
;
AddProcedure(Schema::NodeTypeProperties, std::string(Schema::kProcedureNodeType).c_str(), mgp::ProcedureType::Read,
{},
{mgp::Return(std::string(Schema::kReturnNodeType).c_str(), mgp::Type::String),
mgp::Return(std::string(Schema::kReturnLabels).c_str(), {mgp::Type::List, mgp::Type::String}),
mgp::Return(std::string(Schema::kReturnPropertyName).c_str(), mgp::Type::String),
mgp::Return(std::string(Schema::kReturnPropertyType).c_str(), mgp::Type::Any),
mgp::Return(std::string(Schema::kReturnMandatory).c_str(), mgp::Type::Bool)},
module, memory);
AddProcedure(Schema::RelTypeProperties, std::string(Schema::kProcedureRelType).c_str(), mgp::ProcedureType::Read,
{},
{mgp::Return(std::string(Schema::kReturnRelType).c_str(), mgp::Type::String),
mgp::Return(std::string(Schema::kReturnPropertyName).c_str(), mgp::Type::String),
mgp::Return(std::string(Schema::kReturnPropertyType).c_str(), mgp::Type::Any),
mgp::Return(std::string(Schema::kReturnMandatory).c_str(), mgp::Type::Bool)},
module, memory);
} catch (const std::exception &e) {
return 1;
}
return 0;
}
extern "C" int mgp_shutdown_module() { return 0; }

View File

@@ -1,3 +1,4 @@
/etc/memgraph/memgraph.conf
/etc/memgraph/apoc_compatibility_mappings.json
/etc/memgraph/auth_module/ldap.example.yaml
/etc/logrotate.d/memgraph

View File

@@ -1,12 +1,27 @@
[Unit]
Description=Memgraph: High performance, in-memory, transactional graph database
# Auto-restart config
#StartLimitIntervalSec=300
#StartLimitBurst=5
[Service]
User=memgraph
Group=memgraph
ExecStart=/usr/lib/memgraph/memgraph
# Uncomment this if Memgraph needs more time to write the snapshot on exit.
#TimeoutStopSec=5min
# Auto-restart config
#Restart=on-failure
#RestartSec=10s
[Install]
WantedBy=multi-user.target
# Checks processes' memory usage and force kills it when it is taking up enough
# memory to make the system unstable / about to crash, e.g. on Debian-based OS:
# apt install systemd-oomd
# systemctl enable systemd-oomd
# systemctl start systemd-oomd

View File

@@ -11,6 +11,12 @@ SUPPORTED_OS=(
amzn-2
)
SUPPORTED_BUILD_TYPES=(
Debug
Release
RelWithDebInfo
)
PROJECT_ROOT="$SCRIPT_DIR/../.."
TOOLCHAIN_VERSION="toolchain-v4"
ACTIVATE_TOOLCHAIN="source /opt/${TOOLCHAIN_VERSION}/activate"
@@ -18,14 +24,16 @@ HOST_OUTPUT_DIR="$PROJECT_ROOT/build/output"
print_help () {
# TODO(gitbuda): Update the release/package/run.sh help
echo "$0 init|package|docker|test {os} [--for-docker|--for-platform]"
echo "$0 init|package|docker|test {os} {build_type} [--for-docker|--for-platform]"
echo ""
echo " OSs: ${SUPPORTED_OS[*]}"
echo " Build types: ${SUPPORTED_BUILD_TYPES[*]}"
exit 1
}
make_package () {
os="$1"
build_type="$2"
build_container="mgbuild_$os"
echo "Building Memgraph for $os on $build_container..."
@@ -44,10 +52,10 @@ make_package () {
package_command=" cpack -G DEB --config ../CPackConfig.cmake "
fi
telemetry_id_override_flag=""
if [[ "$#" -gt 1 ]]; then
if [[ "$2" == "--for-docker" ]]; then
if [[ "$#" -gt 2 ]]; then
if [[ "$3" == "--for-docker" ]]; then
telemetry_id_override_flag=" -DMG_TELEMETRY_ID_OVERRIDE=DOCKER "
elif [[ "$2" == "--for-platform" ]]; then
elif [[ "$3" == "--for-platform" ]]; then
telemetry_id_override_flag=" -DMG_TELEMETRY_ID_OVERRIDE=DOCKER-PLATFORM"
else
print_help
@@ -64,6 +72,10 @@ make_package () {
if [[ "$(git rev-parse --abbrev-ref HEAD)" != "master" ]]; then
git fetch origin master:master
fi
# Ensure we have a clean build directory
docker exec "$build_container" rm -rf /memgraph
docker exec "$build_container" mkdir -p /memgraph
# TODO(gitbuda): Revisit copying the whole repo -> makese sense under CI.
docker cp "$PROJECT_ROOT/." "$build_container:/memgraph/"
@@ -89,9 +101,9 @@ make_package () {
docker exec "$build_container" bash -c "cd $container_build_dir && rm -rf ./*"
# TODO(gitbuda): cmake fails locally if remote is clone via ssh because of the key -> FIX
if [[ "$os" =~ "-arm" ]]; then
docker exec "$build_container" bash -c "cd $container_build_dir && $ACTIVATE_TOOLCHAIN && cmake -DCMAKE_BUILD_TYPE=release -DMG_ARCH="ARM64" $telemetry_id_override_flag .."
docker exec "$build_container" bash -c "cd $container_build_dir && $ACTIVATE_TOOLCHAIN && cmake -DCMAKE_BUILD_TYPE=$build_type -DMG_ARCH="ARM64" $telemetry_id_override_flag .."
else
docker exec "$build_container" bash -c "cd $container_build_dir && $ACTIVATE_TOOLCHAIN && cmake -DCMAKE_BUILD_TYPE=release $telemetry_id_override_flag .."
docker exec "$build_container" bash -c "cd $container_build_dir && $ACTIVATE_TOOLCHAIN && cmake -DCMAKE_BUILD_TYPE=$build_type $telemetry_id_override_flag .."
fi
# ' is used instead of " because we need to run make within the allowed
# container resources.
@@ -141,20 +153,34 @@ case "$1" in
package)
shift 1
if [[ "$#" -lt 1 ]]; then
if [[ "$#" -lt 2 ]]; then
print_help
fi
os="$1"
shift 1
build_type="$2"
shift 2
is_os_ok=false
for supported_os in "${SUPPORTED_OS[@]}"; do
if [[ "$supported_os" == "${os}" ]]; then
is_os_ok=true
break
fi
done
if [[ "$is_os_ok" == true ]]; then
make_package "$os" "$@"
is_build_type_ok=false
for supported_build_type in "${SUPPORTED_BUILD_TYPES[@]}"; do
if [[ "$supported_build_type" == "${build_type}" ]]; then
is_build_type_ok=true
break
fi
done
if [[ "$is_os_ok" == true && "$is_build_type_ok" == true ]]; then
make_package "$os" "$build_type" "$@"
else
if [[ "$is_os_ok" == false ]]; then
echo "Unsupported OS: $os"
elif [[ "$is_build_type_ok" == false ]]; then
echo "Unsupported build type: $build_type"
fi
print_help
fi
;;

View File

@@ -132,6 +132,7 @@ echo "Don't forget to switch to the 'memgraph' user to use Memgraph" || exit 1
# Override CPACK_RPM_ABSOLUTE_INSTALL_FILES with our %config(noreplace), cpack
# uses plain %config.
%config(noreplace) "/etc/memgraph/memgraph.conf"
%config(noreplace) "/etc/memgraph/apoc_compatibility_mappings.json"
%config(noreplace) "/etc/memgraph/auth_module/ldap.example.yaml"
%config(noreplace) "/etc/logrotate.d/memgraph"

View File

@@ -18,6 +18,10 @@ add_subdirectory(rpc)
add_subdirectory(license)
add_subdirectory(auth)
add_subdirectory(audit)
add_subdirectory(dbms)
add_subdirectory(flags)
add_subdirectory(distributed)
add_subdirectory(replication)
string(TOLOWER ${CMAKE_BUILD_TYPE} lower_build_type)
@@ -31,19 +35,13 @@ include_directories(${CMAKE_CURRENT_BINARY_DIR})
# ----------------------------------------------------------------------------
set(mg_single_node_v2_sources
memgraph.cpp
flags/isolation_level.cpp
flags/memory_limit.cpp
flags/log_level.cpp
flags/general.cpp
flags/audit.cpp
flags/bolt.cpp
)
# memgraph main executable
add_executable(memgraph ${mg_single_node_v2_sources})
target_include_directories(memgraph PUBLIC ${CMAKE_SOURCE_DIR}/include)
target_link_libraries(memgraph stdc++fs Threads::Threads
mg-telemetry mg-communication mg-memory mg-utils mg-license mg-settings mg-glue)
mg-telemetry mg-communication mg-communication-metrics mg-memory mg-utils mg-license mg-settings mg-glue mg-flags)
# NOTE: `include/mg_procedure.syms` describes a pattern match for symbols which
# should be dynamically exported, so that `dlopen` can correctly link the
@@ -76,7 +74,7 @@ if(lower_build_type STREQUAL "release")
COMMENT "Stripping symbols and sections from memgraph")
endif()
# Generate the configuration file.
# Generate the configuration file under the build directory.
add_custom_command(TARGET memgraph POST_BUILD
COMMAND ${CMAKE_SOURCE_DIR}/config/generate.py
${CMAKE_BINARY_DIR}/memgraph
@@ -85,6 +83,11 @@ add_custom_command(TARGET memgraph POST_BUILD
${CMAKE_SOURCE_DIR}/config/flags.yaml
BYPRODUCTS ${CMAKE_BINARY_DIR}/config/memgraph.conf
COMMENT "Generating memgraph configuration file")
# Copy the mappings file to the build directory.
add_custom_command(TARGET memgraph POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy
${CMAKE_SOURCE_DIR}/config/mappings.json
${CMAKE_BINARY_DIR}/config/apoc_compatibility_mappings.json)
# Everything here is under "memgraph" install component.
set(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME "memgraph")
@@ -120,6 +123,9 @@ install(FILES ${CMAKE_SOURCE_DIR}/include/mgp.hpp
# Install the config file (must use absolute path).
install(FILES ${CMAKE_BINARY_DIR}/config/memgraph.conf
DESTINATION /etc/memgraph RENAME memgraph.conf)
# Install the mappings file (must use absolute path).
install(FILES ${CMAKE_BINARY_DIR}/config/apoc_compatibility_mappings.json
DESTINATION /etc/memgraph RENAME apoc_compatibility_mappings.json)
# Install logrotate configuration (must use absolute path).
install(FILES ${CMAKE_SOURCE_DIR}/release/logrotate.conf
@@ -137,7 +143,7 @@ install(CODE "file(MAKE_DIRECTORY \$ENV{DESTDIR}/var/log/memgraph
# Memgraph CSV Import Tool Executable
# ----------------------------------------------------------------------------
add_executable(mg_import_csv mg_import_csv.cpp)
target_link_libraries(mg_import_csv mg-storage-v2)
target_link_libraries(mg_import_csv mg-storage-v2 mg-dbms)
# Strip the executable in release build.
if(lower_build_type STREQUAL "release")

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -21,5 +21,6 @@ namespace memgraph::auth {
class AuthException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
SPECIALIZE_GET_EXCEPTION_NAME(AuthException)
};
} // namespace memgraph::auth

View File

@@ -16,10 +16,12 @@ set(communication_src_files
find_package(Boost REQUIRED)
add_library(mg-communication-metrics STATIC metrics.cpp)
target_link_libraries(mg-communication-metrics json)
add_library(mg-communication STATIC ${communication_src_files})
target_link_libraries(mg-communication Boost::headers Threads::Threads mg-utils mg-io mg-auth fmt::fmt gflags)
target_link_libraries(mg-communication Boost::headers Threads::Threads mg-utils mg-io mg-auth fmt::fmt gflags mg-communication-metrics mg-events)
find_package(OpenSSL REQUIRED)
target_link_libraries(mg-communication ${OPENSSL_LIBRARIES})
target_include_directories(mg-communication SYSTEM PUBLIC ${OPENSSL_INCLUDE_DIR})
target_precompile_headers(mg-communication INTERFACE http/server.hpp <boost/beast/websocket.hpp> bolt/v1/session.hpp)

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -264,4 +264,5 @@ bool Client::ReadMessageData(Marker marker, Value &ret) {
}
return false;
}
} // namespace memgraph::communication::bolt

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -38,6 +38,7 @@ class FailureResponseException : public utils::BasicException {
: utils::BasicException{message}, code_{code} {}
const std::string &code() const { return code_; }
SPECIALIZE_GET_EXCEPTION_NAME(FailureResponseException)
private:
std::string code_;
@@ -49,6 +50,7 @@ class FailureResponseException : public utils::BasicException {
class ClientQueryException : public FailureResponseException {
public:
using FailureResponseException::FailureResponseException;
SPECIALIZE_GET_EXCEPTION_NAME(ClientQueryException)
};
/// This exception is thrown whenever a fatal error occurs during query
@@ -57,6 +59,7 @@ class ClientQueryException : public FailureResponseException {
class ClientFatalException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
SPECIALIZE_GET_EXCEPTION_NAME(ClientFatalException)
};
// Internal exception used whenever a communication error occurs. You should
@@ -64,6 +67,7 @@ class ClientFatalException : public utils::BasicException {
class ServerCommunicationException : public ClientFatalException {
public:
ServerCommunicationException() : ClientFatalException("Couldn't communicate with the server!") {}
SPECIALIZE_GET_EXCEPTION_NAME(ServerCommunicationException)
};
// Internal exception used whenever a malformed data error occurs. You should
@@ -71,6 +75,7 @@ class ServerCommunicationException : public ClientFatalException {
class ServerMalformedDataException : public ClientFatalException {
public:
ServerMalformedDataException() : ClientFatalException("The server sent malformed data!") {}
SPECIALIZE_GET_EXCEPTION_NAME(ServerMalformedDataException)
};
/// Structure that is used to return results from an executed query.
@@ -155,4 +160,5 @@ class Client final {
ChunkedEncoderBuffer<communication::ClientOutputStream> encoder_buffer_{output_stream_};
ClientEncoder encoder_{encoder_buffer_};
};
} // namespace memgraph::communication::bolt

View File

@@ -0,0 +1,55 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "communication/bolt/v1/value.hpp"
#include "communication/metrics.hpp"
namespace memgraph::communication::bolt {
template <typename TSession>
inline void RegisterNewSession(TSession &session, Value &metadata) {
auto &data = metadata.ValueMap();
session.metrics_ = bolt_metrics.Add(data.contains("user_agent") ? data["user_agent"].ValueString() : "unknown",
fmt::format("{}.{}", session.version_.major, session.version_.minor),
session.client_supported_bolt_versions_);
++session.metrics_.value()->sessions;
auto conn_type = (!data.contains("scheme") || data["scheme"].ValueString() == "none")
? BoltMetrics::ConnectionType::kAnonymous
: BoltMetrics::ConnectionType::kBasic;
++session.metrics_.value()->connection_types[(int)conn_type];
}
template <typename TSession>
inline void TouchNewSession(TSession &session, Value &metadata) {
auto &data = metadata.ValueMap();
session.metrics_ = bolt_metrics.Add(data.contains("user_agent") ? data["user_agent"].ValueString() : "unknown");
}
template <typename TSession>
inline void UpdateNewSession(TSession &session, Value &metadata) {
auto &data = metadata.ValueMap();
session.metrics_.value()->bolt_v = fmt::format("{}.{}", session.version_.major, session.version_.minor);
session.metrics_.value()->supported_bolt_v = session.client_supported_bolt_versions_;
++session.metrics_.value()->sessions;
auto conn_type = (!data.contains("scheme") || data["scheme"].ValueString() == "none")
? BoltMetrics::ConnectionType::kAnonymous
: BoltMetrics::ConnectionType::kBasic;
++session.metrics_.value()->connection_types[(int)conn_type];
}
template <typename TSession>
inline void IncrementQueryMetrics(TSession &session) {
++session.metrics_.value()->queries;
}
} // namespace memgraph::communication::bolt

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -30,6 +30,7 @@ namespace memgraph::communication::bolt {
class ClientError : public utils::BasicException {
public:
using utils::BasicException::BasicException;
SPECIALIZE_GET_EXCEPTION_NAME(ClientError)
};
/**
@@ -67,6 +68,7 @@ class VerboseError : public utils::BasicException {
code_(fmt::format("Memgraph.{}.{}.{}", ClassificationToString(classification), category, title)) {}
const std::string &code() const noexcept { return code_; }
SPECIALIZE_GET_EXCEPTION_NAME(VerboseError)
private:
std::string ClassificationToString(Classification classification) {

View File

@@ -27,6 +27,7 @@
#include "communication/bolt/v1/states/handshake.hpp"
#include "communication/bolt/v1/states/init.hpp"
#include "communication/bolt/v1/value.hpp"
#include "communication/metrics.hpp"
#include "dbms/constants.hpp"
#include "dbms/global.hpp"
#include "utils/exceptions.hpp"
@@ -43,6 +44,7 @@ namespace memgraph::communication::bolt {
class SessionException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
SPECIALIZE_GET_EXCEPTION_NAME(SessionException)
};
/**
@@ -54,7 +56,7 @@ class SessionException : public utils::BasicException {
* @tparam TOutputStream type of output stream that will be used
*/
template <typename TInputStream, typename TOutputStream>
class Session : public dbms::SessionInterface {
class Session {
public:
using TEncoder = Encoder<ChunkedEncoderBuffer<TOutputStream>>;
@@ -159,6 +161,7 @@ class Session : public dbms::SessionInterface {
break;
case State::Idle:
case State::Result:
at_least_one_run_ = true;
state_ = StateExecutingRun(*this, state_);
break;
case State::Error:
@@ -180,6 +183,12 @@ class Session : public dbms::SessionInterface {
}
}
void HandleError() {
if (!at_least_one_run_) {
spdlog::info("Sudden connection loss. Make sure the client supports Memgraph.");
}
}
// TODO: Rethink if there is a way to hide some members. At the momement all of them are public.
TInputStream &input_stream_;
TOutputStream &output_stream_;
@@ -192,6 +201,7 @@ class Session : public dbms::SessionInterface {
bool handshake_done_{false};
State state_{State::Handshake};
bool at_least_one_run_{false};
struct Version {
uint8_t major;
@@ -199,9 +209,11 @@ class Session : public dbms::SessionInterface {
};
Version version_;
std::vector<std::string> client_supported_bolt_versions_;
std::optional<BoltMetrics::Metrics> metrics_;
std::string GetDatabaseName() const override = 0;
std::string UUID() const final { return session_uuid_; }
virtual std::string GetCurrentDB() const = 0;
std::string UUID() const { return session_uuid_; }
private:
void ClientFailureInvalidData() {

View File

@@ -18,6 +18,7 @@
#include <string_view>
#include <vector>
#include "communication/bolt/metrics.hpp"
#include "communication/bolt/v1/codes.hpp"
#include "communication/bolt/v1/constants.hpp"
#include "communication/bolt/v1/exceptions.hpp"
@@ -208,7 +209,14 @@ State HandleRunV1(TSession &session, const State state, const Marker marker) {
DMG_ASSERT(!session.encoder_buffer_.HasData(), "There should be no data to write in this state");
spdlog::debug("[Run - {}] '{}'", session.GetDatabaseName(), query.ValueString());
#if MG_ENTERPRISE
spdlog::debug("[Run - {}] '{}'", session.GetCurrentDB(), query.ValueString());
#else
spdlog::debug("[Run] '{}'", query.ValueString());
#endif
// Increment number of queries in the metrics
IncrementQueryMetrics(session);
try {
// Interpret can throw.
@@ -272,7 +280,14 @@ State HandleRunV4(TSession &session, const State state, const Marker marker) {
return HandleFailure(session, e);
}
spdlog::debug("[Run - {}] '{}'", session.GetDatabaseName(), query.ValueString());
#if MG_ENTERPRISE
spdlog::debug("[Run - {}] '{}'", session.GetCurrentDB(), query.ValueString());
#else
spdlog::debug("[Run] '{}'", query.ValueString());
#endif
// Increment number of queries in the metrics
IncrementQueryMetrics(session);
try {
// Interpret can throw.

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -24,6 +24,31 @@
namespace memgraph::communication::bolt {
inline std::vector<std::string> StringifySupportedVersions(uint8_t *data) {
std::vector<std::string> res;
uint8_t range_i = 1;
uint8_t minor_i = 2;
uint8_t major_i = 3;
uint8_t chunk_size = 4;
uint8_t n_chunks = 4;
auto stringify_version = [](uint8_t major, uint8_t minor) { return fmt::format("{}.{}", major, minor); };
for (uint8_t i = 0; i < n_chunks; ++i) {
const uint32_t full_version = *((uint32_t *)data);
if (full_version == 0) break;
if (data[1] != 0) { // Supports a range of versions
uint8_t range = data[range_i];
auto max_minor = data[minor_i];
for (uint8_t r = 0; r <= range; ++r) {
res.push_back(stringify_version(data[major_i], max_minor - r));
}
} else {
res.push_back(stringify_version(data[major_i], data[minor_i]));
}
data += chunk_size;
}
return res;
}
inline bool CopyProtocolInformationIfSupported(uint16_t version, uint8_t *protocol) {
const auto *supported_version = std::find(std::begin(kSupportedVersions), std::end(kSupportedVersions), version);
if (supported_version != std::end(kSupportedVersions)) {
@@ -71,6 +96,8 @@ State StateHandshakeRun(TSession &session) {
auto dataPosition = session.input_stream_.data() + sizeof(kPreamble);
uint8_t protocol[4] = {0x00};
session.client_supported_bolt_versions_ = std::move(StringifySupportedVersions(dataPosition));
for (int i = 0; i < 4 && !protocol[3]; ++i) {
// If there is an offset defined (e.g. 0x00 0x03 0x03 0x04) the second byte
// That would enable the client to pick between 4.0 and 4.3 versions

View File

@@ -11,6 +11,7 @@
#pragma once
#include <fmt/core.h>
#include <fmt/format.h>
#include <optional>
@@ -18,6 +19,7 @@
#include "communication/bolt/v1/state.hpp"
#include "communication/bolt/v1/value.hpp"
#include "communication/exceptions.hpp"
#include "communication/metrics.hpp"
#include "spdlog/spdlog.h"
#include "utils/likely.hpp"
#include "utils/logging.hpp"
@@ -174,7 +176,7 @@ State SendSuccessMessage(TSession &session) {
// we send a hardcoded value for now.
std::map<std::string, Value> metadata{{"connection_id", "bolt-1"}};
if (auto server_name = session.GetServerNameForInit(); server_name) {
metadata.insert({"server", *server_name});
metadata.insert({"server", std::move(*server_name)});
}
bool success_sent = session.encoder_.MessageSuccess(metadata);
if (!success_sent) {
@@ -201,6 +203,9 @@ State StateInitRunV1(TSession &session, const Marker marker, const Signature sig
return result.value();
}
// Register session to metrics
RegisterNewSession(session, *maybeMetadata);
return SendSuccessMessage(session);
}
@@ -227,6 +232,9 @@ State StateInitRunV4(TSession &session, Marker marker, Signature signature) {
return result.value();
}
// Register session to metrics
RegisterNewSession(session, *maybeMetadata);
return SendSuccessMessage(session);
}
@@ -247,6 +255,10 @@ State StateInitRunV5(TSession &session, Marker marker, Signature signature) {
if (SendSuccessMessage(session) == State::Close) {
return State::Close;
}
// Register session to metrics
TouchNewSession(session, *maybeMetadata);
// Stay in Init
return State::Init;
}
@@ -274,6 +286,10 @@ State StateInitRunV5(TSession &session, Marker marker, Signature signature) {
if (SendSuccessMessage(session) == State::Close) {
return State::Close;
}
// Register session to metrics
UpdateNewSession(session, *maybeMetadata);
return State::Idle;
}

View File

@@ -276,6 +276,7 @@ class ValueException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
ValueException() : BasicException("Incompatible template param and type!") {}
SPECIALIZE_GET_EXCEPTION_NAME(ValueException)
};
/**

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -15,6 +15,10 @@
#include <openssl/err.h>
#include <openssl/ssl.h>
// Centos 7 OpenSSL includes libkrb5 which has brings in macros TRUE and FALSE. undef to prevent issues.
#undef TRUE
#undef FALSE
#include "communication/buffer.hpp"
#include "communication/context.hpp"
#include "communication/init.hpp"

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -17,6 +17,10 @@
#include <openssl/ssl.h>
#include <boost/asio/ssl/context.hpp>
// Centos 7 OpenSSL includes libkrb5 which has brings in macros TRUE and FALSE. undef to prevent issues.
#undef TRUE
#undef FALSE
namespace memgraph::communication {
/**

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -21,5 +21,6 @@ namespace memgraph::communication {
*/
class SessionClosedException : public utils::BasicException {
using utils::BasicException::BasicException;
SPECIALIZE_GET_EXCEPTION_NAME(SessionClosedException)
};
} // namespace memgraph::communication

View File

@@ -0,0 +1,42 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "metrics.hpp"
namespace {
constexpr auto kName = "name";
constexpr auto kSupportedBoltVersions = "supported_bolt_versions";
constexpr auto kBoltVersion = "bolt_version";
constexpr auto kConnectionTypes = "connection_types";
constexpr auto kSessions = "sessions";
constexpr auto kQueries = "queries";
} // namespace
namespace memgraph::communication {
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
BoltMetrics bolt_metrics;
nlohmann::json BoltMetrics::Info::ToJson() const {
nlohmann::json res;
res[kName] = name;
res[kSupportedBoltVersions] = nlohmann::json::array();
for (const auto &sbv : supported_bolt_v) {
res[kSupportedBoltVersions].push_back(sbv);
}
res[kBoltVersion] = bolt_v;
res[kConnectionTypes] = {{ConnectionTypeStr((ConnectionType)0), connection_types[0].load()},
{ConnectionTypeStr((ConnectionType)1), connection_types[1].load()}};
res[kSessions] = sessions.load();
res[kQueries] = queries.load();
return res;
}
} // namespace memgraph::communication

View File

@@ -0,0 +1,116 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <atomic>
#include <map>
#include <mutex>
#include <shared_mutex>
#include <string>
#include <vector>
#include <json/json.hpp>
namespace memgraph::communication {
class BoltMetrics {
public:
enum class ConnectionType { kAnonymous = 0, kBasic, Count };
static constexpr std::array<std::string_view, (int)ConnectionType::Count> ct_to_str = {"anonymous", "basic"};
static std::string ConnectionTypeStr(ConnectionType type) { return std::string(ct_to_str[(int)type]); }
class Metrics;
struct Info {
explicit Info(std::string name) : name(std::move(name)) {}
Info(std::string name, std::string bolt_v, std::vector<std::string> supported_bolt_v)
: name(std::move(name)), bolt_v(std::move(bolt_v)), supported_bolt_v(std::move(supported_bolt_v)) {}
const std::string name; //!< Driver name
std::string bolt_v; //!< Bolt version used
std::vector<std::string> supported_bolt_v; //!< Bolt versions supported by the driver
std::atomic_int connection_types[(int)ConnectionType::Count]; //!< Authentication used when connecting
std::atomic_int sessions; //!< Number of sessions using the same driver
std::atomic_int queries; //!< Queries executed by the driver
nlohmann::json ToJson() const;
private:
friend class Metrics;
std::atomic_int concurrent_users;
};
class Metrics {
friend class BoltMetrics;
explicit Metrics(Info &info) : info_(&info) { ++info_->concurrent_users; }
public:
~Metrics() {
if (info_) --info_->concurrent_users;
}
Metrics(const Metrics &other) : info_(other.info_) { ++info_->concurrent_users; }
Metrics(Metrics &&other) noexcept : info_(other.info_) { other.info_ = nullptr; }
Metrics &operator=(const Metrics &other) {
if (this != &other) {
if (info_) --info_->concurrent_users;
info_ = other.info_;
if (info_) ++info_->concurrent_users;
}
return *this;
}
Metrics &operator=(Metrics &&other) noexcept {
if (this != &other) {
if (info_) --info_->concurrent_users;
info_ = other.info_;
other.info_ = nullptr; // Invalidate the source object
}
return *this;
}
Info *operator->() { return info_; }
Info *info_;
};
Metrics Add(std::string name) {
std::unique_lock<std::mutex> l(mtx);
auto key = name;
auto [it, _] = info.emplace(std::piecewise_construct, std::forward_as_tuple(std::move(key)),
std::forward_as_tuple(std::move(name)));
return Metrics(it->second);
}
Metrics Add(std::string name, std::string bolt_v, std::vector<std::string> supported_bolt_v) {
std::unique_lock<std::mutex> l(mtx);
auto key = name;
auto [it, _] = info.emplace(std::piecewise_construct, std::forward_as_tuple(std::move(key)),
std::forward_as_tuple(std::move(name), std::move(bolt_v), std::move(supported_bolt_v)));
return Metrics(it->second);
}
nlohmann::json ToJson() {
std::unique_lock<std::mutex> l(mtx);
auto res = nlohmann::json::array();
for (const auto &[_, client_info] : info) {
res.push_back(client_info.ToJson());
}
return res;
}
private:
mutable std::mutex mtx;
std::map<std::string, Info> info;
};
extern BoltMetrics bolt_metrics;
} // namespace memgraph::communication

View File

@@ -22,6 +22,10 @@
#include <openssl/err.h>
#include <openssl/ssl.h>
// Centos 7 OpenSSL includes libkrb5 which has brings in macros TRUE and FALSE. undef to prevent issues.
#undef TRUE
#undef FALSE
#include "communication/buffer.hpp"
#include "communication/context.hpp"
#include "communication/exceptions.hpp"

View File

@@ -110,11 +110,7 @@ class WebsocketSession : public std::enable_shared_from_this<WebsocketSession<TS
return std::shared_ptr<WebsocketSession>(new WebsocketSession(std::forward<Args>(args)...));
}
#ifdef MG_ENTERPRISE
~WebsocketSession() { session_context_->Delete(session_); }
#else
~WebsocketSession() = default;
#endif
WebsocketSession(const WebsocketSession &) = delete;
WebsocketSession &operator=(const WebsocketSession &) = delete;
@@ -171,14 +167,15 @@ class WebsocketSession : public std::enable_shared_from_this<WebsocketSession<TS
: ws_(std::move(socket)),
strand_{boost::asio::make_strand(ws_.get_executor())},
output_stream_([this](const uint8_t *data, size_t len, bool /*have_more*/) { return Write(data, len); }),
session_{*session_context, endpoint, input_buffer_.read_end(), &output_stream_},
session_{session_context->ic, endpoint, input_buffer_.read_end(), &output_stream_, session_context->auth,
#ifdef MG_ENTERPRISE
session_context->audit_log
#endif
},
session_context_{session_context},
endpoint_{endpoint},
remote_endpoint_{ws_.next_layer().socket().remote_endpoint()},
service_name_{service_name} {
#ifdef MG_ENTERPRISE
session_context_->Register(session_);
#endif
}
void OnAccept(boost::beast::error_code ec) {
@@ -286,11 +283,7 @@ class Session final : public std::enable_shared_from_this<Session<TSession, TSes
return std::shared_ptr<Session>(new Session(std::forward<Args>(args)...));
}
#ifdef MG_ENTERPRISE
~Session() { session_context_->Delete(session_); }
#else
~Session() = default;
#endif
Session(const Session &) = delete;
Session(Session &&) = delete;
@@ -366,17 +359,17 @@ class Session final : public std::enable_shared_from_this<Session<TSession, TSes
: socket_(CreateSocket(std::move(socket), server_context)),
strand_{boost::asio::make_strand(GetExecutor())},
output_stream_([this](const uint8_t *data, size_t len, bool have_more) { return Write(data, len, have_more); }),
session_{*session_context, endpoint, input_buffer_.read_end(), &output_stream_},
session_{session_context->ic, endpoint, input_buffer_.read_end(), &output_stream_, session_context->auth,
#ifdef MG_ENTERPRISE
session_context->audit_log
#endif
},
session_context_{session_context},
endpoint_{endpoint},
remote_endpoint_{GetRemoteEndpoint()},
service_name_{service_name},
timeout_seconds_(inactivity_timeout_sec),
timeout_timer_(GetExecutor()) {
#ifdef MG_ENTERPRISE
// TODO Try to remove Register (see comment at SessionInterface declaration)
session_context_->Register(session_);
#endif
ExecuteForSocket([](auto &&socket) {
socket.lowest_layer().set_option(tcp::no_delay(true)); // enable PSH
socket.lowest_layer().set_option(boost::asio::socket_base::keep_alive(true)); // enable SO_KEEPALIVE
@@ -413,6 +406,8 @@ class Session final : public std::enable_shared_from_this<Session<TSession, TSes
void OnRead(const boost::system::error_code &ec, const size_t bytes_transferred) {
if (ec) {
// TODO Check if client disconnected
session_.HandleError();
return OnError(ec);
}
input_buffer_.write_end()->Written(bytes_transferred);

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -175,6 +175,11 @@ void Session::OnRead(const boost::beast::error_code ec, const size_t /*bytes_tra
return;
}
if (ec) {
LogError(ec, "read");
return;
}
if (!IsAuthenticated()) {
auto response = nlohmann::json();
auto auth_failed = [this, &response](const std::string &message) {

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -35,6 +35,7 @@ namespace memgraph::csv {
class CsvReadException : public utils::BasicException {
using utils::BasicException::BasicException;
SPECIALIZE_GET_EXCEPTION_NAME(CsvReadException)
};
class FileCsvSource {

3
src/dbms/CMakeLists.txt Normal file
View File

@@ -0,0 +1,3 @@
add_library(mg-dbms STATIC database.cpp replication_handler.cpp inmemory/replication_handlers.cpp)
target_link_libraries(mg-dbms mg-utils mg-storage-v2 mg-query)

41
src/dbms/database.cpp Normal file
View File

@@ -0,0 +1,41 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "dbms/database.hpp"
#include "dbms/inmemory/storage_helper.hpp"
#include "dbms/replication_handler.hpp"
#include "flags/storage_mode.hpp"
#include "storage/v2/disk/storage.hpp"
#include "storage/v2/inmemory/storage.hpp"
#include "storage/v2/storage_mode.hpp"
template struct memgraph::utils::Gatekeeper<memgraph::dbms::Database>;
namespace memgraph::dbms {
Database::Database(storage::Config config, const replication::ReplicationState &repl_state)
: trigger_store_(config.durability.storage_directory / "triggers"),
streams_{config.durability.storage_directory / "streams"},
plan_cache_{FLAGS_query_plan_cache_max_size},
repl_state_(&repl_state) {
if (config.storage_mode == memgraph::storage::StorageMode::ON_DISK_TRANSACTIONAL || config.force_on_disk ||
utils::DirExists(config.disk.main_storage_directory)) {
storage_ = std::make_unique<storage::DiskStorage>(std::move(config));
} else {
storage_ = dbms::CreateInMemoryStorage(std::move(config), repl_state);
}
}
void Database::SwitchToOnDisk() {
storage_ = std::make_unique<memgraph::storage::DiskStorage>(std::move(storage_->config_));
}
} // namespace memgraph::dbms

173
src/dbms/database.hpp Normal file
View File

@@ -0,0 +1,173 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <algorithm>
#include <filesystem>
#include <iterator>
#include <memory>
#include <optional>
#include <string_view>
#include <unordered_map>
#include "query/cypher_query_interpreter.hpp"
#include "query/stream/streams.hpp"
#include "query/trigger.hpp"
#include "storage/v2/storage.hpp"
#include "utils/gatekeeper.hpp"
#include "utils/lru_cache.hpp"
#include "utils/synchronized.hpp"
namespace memgraph::dbms {
struct DatabaseInfo {
storage::StorageInfo storage_info;
uint64_t triggers;
uint64_t streams;
};
static inline nlohmann::json ToJson(const DatabaseInfo &info) { return ToJson(info.storage_info); }
/**
* @brief Class containing everything associated with a single Database
*
*/
class Database {
public:
/**
* @brief Construct a new Database object
*
* @param config storage configuration
*/
explicit Database(storage::Config config, const replication::ReplicationState &repl_state);
/**
* @brief Returns the raw storage pointer.
* @note Ideally everybody would be using an accessor
* TODO: Remove
*
* @return storage::Storage*
*/
storage::Storage *storage() { return storage_.get(); }
storage::Storage const *storage() const { return storage_.get(); }
/**
* @brief Storage's Accessor
*
* @param override_isolation_level
* @return std::unique_ptr<storage::Storage::Accessor>
*/
std::unique_ptr<storage::Storage::Accessor> Access(
std::optional<storage::IsolationLevel> override_isolation_level = {}) {
return storage_->Access(override_isolation_level, repl_state_->IsMain());
}
std::unique_ptr<storage::Storage::Accessor> UniqueAccess(
std::optional<storage::IsolationLevel> override_isolation_level = {}) {
return storage_->UniqueAccess(override_isolation_level, repl_state_->IsMain());
}
/**
* @brief Unique storage identified (name)
*
* @return const std::string&
*/
const std::string &id() const { return storage_->id(); }
/**
* @brief Returns the storage configuration
*
* @return const storage::Config&
*/
const storage::Config &config() const { return storage_->config_; }
/**
* @brief Get the storage mode
*
* @return storage::StorageMode
*/
storage::StorageMode GetStorageMode() const { return storage_->GetStorageMode(); }
/**
* @brief Get the storage info
*
* @param force_directory Use the configured directory, do not try to decipher the multi-db version
* @return DatabaseInfo
*/
DatabaseInfo GetInfo(bool force_directory = false) const {
DatabaseInfo info;
info.storage_info = storage_->GetInfo(force_directory);
info.triggers = trigger_store_.GetTriggerInfo().size();
info.streams = streams_.GetStreamInfo().size();
return info;
}
/**
* @brief Switch storage to OnDisk
*
*/
void SwitchToOnDisk();
/**
* @brief Returns the raw TriggerStore pointer
*
* @return query::TriggerStore*
*/
query::TriggerStore *trigger_store() { return &trigger_store_; }
/**
* @brief Returns the raw Streams pointer
*
* @return query::stream::Streams*
*/
query::stream::Streams *streams() { return &streams_; }
/**
* @brief Returns the raw ThreadPool pointer (used for after commit triggers)
*
* @return utils::ThreadPool*
*/
utils::ThreadPool *thread_pool() { return &after_commit_trigger_pool_; }
/**
* @brief Add task to the after commit trigger thread pool
*
* @param new_task
*/
void AddTask(std::function<void()> new_task) { after_commit_trigger_pool_.AddTask(std::move(new_task)); }
/**
* @brief Returns the PlanCache vector raw pointer
*
* @return utils::Synchronized<utils::LRUCache<uint64_t, std::shared_ptr<PlanWrapper>>, utils::RWSpinLock>
*/
query::PlanCacheLRU *plan_cache() { return &plan_cache_; }
private:
std::unique_ptr<storage::Storage> storage_; //!< Underlying storage
query::TriggerStore trigger_store_; //!< Triggers associated with the storage
utils::ThreadPool after_commit_trigger_pool_{1}; //!< Thread pool for executing after commit triggers
query::stream::Streams streams_; //!< Streams associated with the storage
// TODO: Move to a better place
query::PlanCacheLRU plan_cache_; //!< Plan cache associated with the storage
const replication::ReplicationState *repl_state_;
};
} // namespace memgraph::dbms
extern template struct memgraph::utils::Gatekeeper<memgraph::dbms::Database>;
namespace memgraph::dbms {
using DatabaseAccess = memgraph::utils::Gatekeeper<memgraph::dbms::Database>::Accessor;
} // namespace memgraph::dbms

View File

@@ -0,0 +1,97 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#ifdef MG_ENTERPRISE
#include <algorithm>
#include <filesystem>
#include <iterator>
#include <memory>
#include <optional>
#include <string_view>
#include <unordered_map>
#include "dbms/database.hpp"
#include "handler.hpp"
namespace memgraph::dbms {
/* NOTE
* The Database object is shared. All the higher-level function calls should be protected.
* Storage function calls should already be protected; add protection where needed.
*
* Current implementation uses a handler of Database objects. It owns them and gives
* Gatekeeper::Accessor to it. These guarantee that the object won't be
* destroyed unless no one is using it.
*/
/**Config
* @brief Multi-database storage handler
*
*/
class DatabaseHandler : public Handler<Database> {
public:
using HandlerT = Handler<Database>;
/**
* @brief Generate new storage associated with the passed name.
*
* @param name Name associating the new interpreter context
* @param config Storage configuration
* @return HandlerT::NewResult
*/
HandlerT::NewResult New(std::string_view name, storage::Config config,
const replication::ReplicationState &repl_state) {
// Control that no one is using the same data directory
if (std::any_of(begin(), end(), [&](auto &elem) {
auto db_acc = elem.second.access();
MG_ASSERT(db_acc.has_value(), "Gatekeeper in invalid state");
return db_acc->get()->config().durability.storage_directory == config.durability.storage_directory;
})) {
spdlog::info("Tried to generate new storage using a claimed directory.");
return NewError::EXISTS;
}
config.name = name; // Set storage id via config
return HandlerT::New(std::piecewise_construct, name, config, repl_state);
}
/**
* @brief All currently active storage.
*
* @return std::vector<std::string>
*/
std::vector<std::string> All() const {
std::vector<std::string> res;
res.reserve(std::distance(cbegin(), cend()));
std::for_each(cbegin(), cend(), [&](const auto &elem) { res.push_back(elem.first); });
return res;
}
/**
* @brief Get the associated storage's configuration
*
* @param name
* @return std::optional<storage::Config>
*/
std::optional<storage::Config> GetConfig(std::string_view name) {
auto db = Get(name);
if (db) {
return (*db)->config();
}
return std::nullopt;
}
};
} // namespace memgraph::dbms
#endif

551
src/dbms/dbms_handler.hpp Normal file
View File

@@ -0,0 +1,551 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <algorithm>
#include <concepts>
#include <cstdint>
#include <filesystem>
#include <memory>
#include <mutex>
#include <optional>
#include <ostream>
#include <stdexcept>
#include <system_error>
#include <unordered_map>
#include "auth/auth.hpp"
#include "constants.hpp"
#include "dbms/database.hpp"
#ifdef MG_ENTERPRISE
#include "dbms/database_handler.hpp"
#endif
#include "global.hpp"
#include "query/config.hpp"
#include "query/interpreter_context.hpp"
#include "spdlog/spdlog.h"
#include "storage/v2/durability/durability.hpp"
#include "storage/v2/durability/paths.hpp"
#include "storage/v2/isolation_level.hpp"
#include "utils/exceptions.hpp"
#include "utils/file.hpp"
#include "utils/logging.hpp"
#include "utils/result.hpp"
#include "utils/rw_lock.hpp"
#include "utils/synchronized.hpp"
#include "utils/uuid.hpp"
namespace memgraph::dbms {
struct Statistics {
uint64_t num_vertex; //!< Sum of vertexes in every database
uint64_t num_edges; //!< Sum of edges in every database
uint64_t triggers; //!< Sum of triggers in every database
uint64_t streams; //!< Sum of streams in every database
uint64_t users; //!< Number of defined users
uint64_t num_databases; //!< Number of isolated databases
uint64_t indices; //!< Sum of indices in every database
uint64_t constraints; //!< Sum of constraints in every database
uint64_t storage_modes[3]; //!< Number of databases in each storage mode [IN_MEM_TX, IN_MEM_ANA, ON_DISK_TX]
uint64_t isolation_levels[3]; //!< Number of databases in each isolation level [SNAPSHOT, READ_COMM, READ_UNC]
uint64_t snapshot_enabled; //!< Number of databases with snapshots enabled
uint64_t wal_enabled; //!< Number of databases with WAL enabled
};
static inline nlohmann::json ToJson(const Statistics &stats) {
nlohmann::json res;
res["edges"] = stats.num_edges;
res["vertices"] = stats.num_vertex;
res["triggers"] = stats.triggers;
res["streams"] = stats.streams;
res["users"] = stats.users;
res["databases"] = stats.num_databases;
res["indices"] = stats.indices;
res["constraints"] = stats.constraints;
res["storage_modes"] = {{storage::StorageModeToString((storage::StorageMode)0), stats.storage_modes[0]},
{storage::StorageModeToString((storage::StorageMode)1), stats.storage_modes[1]},
{storage::StorageModeToString((storage::StorageMode)2), stats.storage_modes[2]}};
res["isolation_levels"] = {{storage::IsolationLevelToString((storage::IsolationLevel)0), stats.isolation_levels[0]},
{storage::IsolationLevelToString((storage::IsolationLevel)1), stats.isolation_levels[1]},
{storage::IsolationLevelToString((storage::IsolationLevel)2), stats.isolation_levels[2]}};
res["durability"] = {{"snapshot_enabled", stats.snapshot_enabled}, {"WAL_enabled", stats.wal_enabled}};
return res;
}
/**
* @brief Multi-database session contexts handler.
*/
class DbmsHandler {
public:
using LockT = utils::RWLock;
#ifdef MG_ENTERPRISE
using NewResultT = utils::BasicResult<NewError, DatabaseAccess>;
using DeleteResult = utils::BasicResult<DeleteError>;
/**
* @brief Initialize the handler.
*
* @param configs storage configuration
* @param auth pointer to the global authenticator
* @param recovery_on_startup restore databases (and its content) and authentication data
* @param delete_on_drop when dropping delete any associated directories on disk
*/
DbmsHandler(storage::Config config, const replication::ReplicationState &repl_state, auto *auth,
bool recovery_on_startup, bool delete_on_drop)
: lock_{utils::RWLock::Priority::READ},
default_config_{std::move(config)},
repl_state_(repl_state),
delete_on_drop_(delete_on_drop) {
// TODO: Decouple storage config from dbms config
// TODO: Save individual db configs inside the kvstore and restore from there
storage::UpdatePaths(default_config_, default_config_.durability.storage_directory / "databases");
const auto &db_dir = default_config_.durability.storage_directory;
const auto durability_dir = db_dir / ".durability";
utils::EnsureDirOrDie(db_dir);
utils::EnsureDirOrDie(durability_dir);
durability_ = std::make_unique<kvstore::KVStore>(durability_dir);
// Generate the default database
MG_ASSERT(!NewDefault_().HasError(), "Failed while creating the default DB.");
// Recover previous databases
if (recovery_on_startup) {
for (const auto &[name, _] : *durability_) {
if (name == kDefaultDB) continue; // Already set
spdlog::info("Restoring database {}.", name);
MG_ASSERT(!New_(name).HasError(), "Failed while creating database {}.", name);
spdlog::info("Database {} restored.", name);
}
} else { // Clear databases from the durability list and auth
auto locked_auth = auth->Lock();
for (const auto &[name, _] : *durability_) {
if (name == kDefaultDB) continue;
locked_auth->DeleteDatabase(name);
durability_->Delete(name);
}
}
}
#else
/**
* @brief Initialize the handler. A single database is supported in community edition.
*
* @param configs storage configuration
*/
DbmsHandler(storage::Config config, const replication::ReplicationState &repl_state)
: db_gatekeeper_{[&] {
config.name = kDefaultDB;
return std::move(config);
}(),
repl_state} {}
#endif
#ifdef MG_ENTERPRISE
/**
* @brief Create a new Database associated with the "name" database
*
* @param name name of the database
* @return NewResultT context on success, error on failure
*/
NewResultT New(const std::string &name) {
std::lock_guard<LockT> wr(lock_);
return New_(name, name);
}
/**
* @brief Get the context associated with the "name" database
*
* @param name
* @return DatabaseAccess
* @throw UnknownDatabaseException if database not found
*/
DatabaseAccess Get(std::string_view name = kDefaultDB) {
std::shared_lock<LockT> rd(lock_);
return Get_(name);
}
#else
/**
* @brief Get the context associated with the default database
*
* @return DatabaseAccess
*/
DatabaseAccess Get() {
auto acc = db_gatekeeper_.access();
MG_ASSERT(acc, "Failed to get default database!");
return *acc;
}
#endif
#ifdef MG_ENTERPRISE
/**
* @brief Delete database.
*
* @param db_name database name
* @return DeleteResult error on failure
*/
DeleteResult Delete(const std::string &db_name) {
std::lock_guard<LockT> wr(lock_);
if (db_name == kDefaultDB) {
// MSG cannot delete the default db
return DeleteError::DEFAULT_DB;
}
const auto storage_path = StorageDir_(db_name);
if (!storage_path) return DeleteError::NON_EXISTENT;
// Check if db exists
try {
// Low level handlers
if (!db_handler_.Delete(db_name)) {
return DeleteError::USING;
}
} catch (utils::BasicException &) {
return DeleteError::NON_EXISTENT;
}
// Remove from durability list
if (durability_) durability_->Delete(db_name);
// Delete disk storage
if (delete_on_drop_) {
std::error_code ec;
(void)std::filesystem::remove_all(*storage_path, ec);
if (ec) {
spdlog::error("Failed to clean disk while deleting database \"{}\".", db_name);
defunct_dbs_.emplace(db_name);
return DeleteError::DISK_FAIL;
}
}
// Delete from defunct_dbs_ (in case a second delete call was successful)
defunct_dbs_.erase(db_name);
return {}; // Success
}
#endif
/**
* @brief Return all active databases.
*
* @return std::vector<std::string>
*/
std::vector<std::string> All() const {
#ifdef MG_ENTERPRISE
std::shared_lock<LockT> rd(lock_);
return db_handler_.All();
#else
return {db_gatekeeper_.access()->get()->id()};
#endif
}
/**
* @brief Return the statistics all databases.
*
* @return Statistics
*/
Statistics Stats() {
Statistics stats{};
// TODO: Handle overflow?
#ifdef MG_ENTERPRISE
std::shared_lock<LockT> rd(lock_);
for (auto &[_, db_gk] : db_handler_) {
#else
{
auto &db_gk = db_gatekeeper_;
#endif
auto db_acc_opt = db_gk.access();
if (db_acc_opt) {
auto &db_acc = *db_acc_opt;
const auto &info = db_acc->GetInfo();
const auto &storage_info = info.storage_info;
stats.num_vertex += storage_info.vertex_count;
stats.num_edges += storage_info.edge_count;
stats.triggers += info.triggers;
stats.streams += info.streams;
++stats.num_databases;
stats.indices += storage_info.label_indices + storage_info.label_property_indices;
stats.constraints += storage_info.existence_constraints + storage_info.unique_constraints;
++stats.storage_modes[(int)storage_info.storage_mode];
++stats.isolation_levels[(int)storage_info.isolation_level];
stats.snapshot_enabled += storage_info.durability_snapshot_enabled;
stats.wal_enabled += storage_info.durability_wal_enabled;
}
}
return stats;
}
/**
* @brief Return a vector with all database info.
*
* @return std::vector<DatabaseInfo>
*/
std::vector<DatabaseInfo> Info() {
std::vector<DatabaseInfo> res;
#ifdef MG_ENTERPRISE
std::shared_lock<LockT> rd(lock_);
res.reserve(std::distance(db_handler_.cbegin(), db_handler_.cend()));
for (auto &[_, db_gk] : db_handler_) {
#else
{
auto &db_gk = db_gatekeeper_;
#endif
auto db_acc_opt = db_gk.access();
if (db_acc_opt) {
auto &db_acc = *db_acc_opt;
res.push_back(db_acc->GetInfo());
}
}
return res;
}
/**
* @brief Restore triggers for all currently defined databases.
* @note: Triggers can execute query procedures, so we need to reload the modules first and then the triggers
*
* @param ic global InterpreterContext
*/
void RestoreTriggers(query::InterpreterContext *ic) {
#ifdef MG_ENTERPRISE
std::lock_guard<LockT> wr(lock_);
for (auto &[_, db_gk] : db_handler_) {
#else
{
auto &db_gk = db_gatekeeper_;
#endif
auto db_acc_opt = db_gk.access();
if (db_acc_opt) {
auto &db_acc = *db_acc_opt;
spdlog::debug("Restoring trigger for database \"{}\"", db_acc->id());
auto storage_accessor = db_acc->Access();
auto dba = memgraph::query::DbAccessor{storage_accessor.get()};
db_acc->trigger_store()->RestoreTriggers(&ic->ast_cache, &dba, ic->config.query, ic->auth_checker);
}
}
}
/**
* @brief Restore streams of all currently defined databases.
* @note: Stream transformations are using modules, they have to be restored after the query modules are loaded.
*
* @param ic global InterpreterContext
*/
void RestoreStreams(query::InterpreterContext *ic) {
#ifdef MG_ENTERPRISE
std::lock_guard<LockT> wr(lock_);
for (auto &[_, db_gk] : db_handler_) {
#else
{
auto &db_gk = db_gatekeeper_;
#endif
auto db_acc = db_gk.access();
if (db_acc) {
auto *db = db_acc->get();
spdlog::debug("Restoring streams for database \"{}\"", db->id());
db->streams()->RestoreStreams(*db_acc, ic);
}
}
}
/**
* @brief todo
*
* @param f
*/
void ForEach(auto f) {
#ifdef MG_ENTERPRISE
std::shared_lock<LockT> rd(lock_);
for (auto &[_, db_gk] : db_handler_) {
#else
{
auto &db_gk = db_gatekeeper_;
#endif
auto db_acc = db_gk.access();
if (db_acc) { // This isn't an error, just a defunct db
f(db_acc->get());
}
}
}
/**
* @brief todo
*
* @param f
*/
void ForOne(auto f) {
#ifdef MG_ENTERPRISE
std::shared_lock<LockT> rd(lock_);
for (auto &[_, db_gk] : db_handler_) {
auto db_acc = db_gk.access();
if (db_acc) { // This isn't an error, just a defunct db
if (f(db_acc->get())) break; // Run until the first successful one
}
}
#else
{
auto db_acc = db_gatekeeper_.access();
MG_ASSERT(db_acc, "Should always have the database");
f(db_acc->get());
}
#endif
}
private:
#ifdef MG_ENTERPRISE
/**
* @brief return the storage directory of the associated database
*
* @param name Database name
* @return std::optional<std::filesystem::path>
*/
std::optional<std::filesystem::path> StorageDir_(const std::string &name) {
const auto conf = db_handler_.GetConfig(name);
if (conf) {
return conf->durability.storage_directory;
}
spdlog::debug("Failed to find storage dir for database \"{}\"", name);
return {};
}
/**
* @brief Create a new Database associated with the "name" database
*
* @param name name of the database
* @return NewResultT context on success, error on failure
*/
NewResultT New_(const std::string &name) { return New_(name, name); }
/**
* @brief Create a new Database associated with the "name" database
*
* @param name name of the database
* @param storage_subdir undelying RocksDB directory
* @return NewResultT context on success, error on failure
*/
NewResultT New_(const std::string &name, std::filesystem::path storage_subdir) {
auto config_copy = default_config_;
storage::UpdatePaths(config_copy, default_config_.durability.storage_directory / storage_subdir);
return New_(name, config_copy);
}
/**
* @brief Create a new Database associated with the "name" database
*
* @param name name of the database
* @param storage_config storage configuration
* @return NewResultT context on success, error on failure
*/
NewResultT New_(const std::string &name, storage::Config &storage_config) {
if (defunct_dbs_.contains(name)) {
spdlog::warn("Failed to generate database due to the unknown state of the previously defunct database \"{}\".",
name);
return NewError::DEFUNCT;
}
auto new_db = db_handler_.New(name, storage_config, repl_state_);
if (new_db.HasValue()) {
// Success
if (durability_) durability_->Put(name, "ok"); // TODO: Serialize the configuration?
return new_db.GetValue();
}
return new_db.GetError();
}
/**
* @brief Create a new Database associated with the default database
*
* @return NewResultT context on success, error on failure
*/
NewResultT NewDefault_() {
// Create the default DB in the root (this is how it was done pre multi-tenancy)
auto res = New_(kDefaultDB, "..");
if (res.HasValue()) {
// For back-compatibility...
// Recreate the dbms layout for the default db and symlink to the root
const auto dir = StorageDir_(kDefaultDB);
MG_ASSERT(dir, "Failed to find storage path.");
const auto main_dir = *dir / "databases" / kDefaultDB;
if (!std::filesystem::exists(main_dir)) {
std::filesystem::create_directory(main_dir);
}
// Force link on-disk directories
const auto conf = db_handler_.GetConfig(kDefaultDB);
MG_ASSERT(conf, "No configuration for the default database.");
const auto &tmp_conf = conf->disk;
std::vector<std::filesystem::path> to_link{
tmp_conf.main_storage_directory, tmp_conf.label_index_directory,
tmp_conf.label_property_index_directory, tmp_conf.unique_constraints_directory,
tmp_conf.name_id_mapper_directory, tmp_conf.id_name_mapper_directory,
tmp_conf.durability_directory, tmp_conf.wal_directory,
};
// Add in-memory paths
// Some directories are redundant (skip those)
const std::vector<std::string> skip{".lock", "audit_log", "auth", "databases", "internal_modules", "settings"};
for (auto const &item : std::filesystem::directory_iterator{*dir}) {
const auto dir_name = std::filesystem::relative(item.path(), item.path().parent_path());
if (std::find(skip.begin(), skip.end(), dir_name) != skip.end()) continue;
to_link.push_back(item.path());
}
// Symlink to root dir
for (auto const &item : to_link) {
const auto dir_name = std::filesystem::relative(item, item.parent_path());
const auto link = main_dir / dir_name;
const auto to = std::filesystem::relative(item, main_dir);
if (!std::filesystem::is_symlink(link) && !std::filesystem::exists(link)) {
std::filesystem::create_directory_symlink(to, link);
} else { // Check existing link
std::error_code ec;
const auto test_link = std::filesystem::read_symlink(link, ec);
if (ec || test_link != to) {
MG_ASSERT(false,
"Memgraph storage directory incompatible with new version.\n"
"Please use a clean directory or remove \"{}\" and try again.",
link.string());
}
}
}
}
return res;
}
/**
* @brief Get the DatabaseAccess for the database associated with the "name"
*
* @param name
* @return DatabaseAccess
* @throw UnknownDatabaseException if trying to get unknown database
*/
DatabaseAccess Get_(std::string_view name) {
auto db = db_handler_.Get(name);
if (db) {
return *db;
}
throw UnknownDatabaseException("Tried to retrieve an unknown database \"{}\".", name);
}
mutable LockT lock_; //!< protective lock
storage::Config default_config_; //!< Storage configuration used when creating new databases
const replication::ReplicationState &repl_state_; //!< Global replication state
DatabaseHandler db_handler_; //!< multi-tenancy storage handler
std::unique_ptr<kvstore::KVStore> durability_; //!< list of active dbs (pointer so we can postpone its creation)
bool delete_on_drop_; //!< Flag defining if dropping storage also deletes its directory
std::set<std::string> defunct_dbs_; //!< Databases that are in an unknown state due to various failures
#else
mutable utils::Gatekeeper<Database> db_gatekeeper_; //!< Single databases gatekeeper
#endif
};
} // namespace memgraph::dbms

View File

@@ -19,6 +19,7 @@
namespace memgraph::dbms {
#ifdef MG_ENTERPRISE
enum class DeleteError : uint8_t {
DEFAULT_DB,
USING,
@@ -34,11 +35,7 @@ enum class NewError : uint8_t {
GENERIC,
};
enum class SetForResult : uint8_t {
SUCCESS,
ALREADY_SET,
FAIL,
};
#endif
/**
* UnknownSession Exception
@@ -48,6 +45,7 @@ enum class SetForResult : uint8_t {
class UnknownSessionException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
SPECIALIZE_GET_EXCEPTION_NAME(UnknownSessionException)
};
/**
@@ -58,53 +56,7 @@ class UnknownSessionException : public utils::BasicException {
class UnknownDatabaseException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
};
/**
* @brief Session interface used by the DBMS to handle the the active sessions.
* @todo Try to remove this dependency from SessionContextHandler. OnDelete could be removed, as it only does an assert.
* OnChange could be removed if SetFor returned the pointer and the called then handled the OnChange execution.
* However, the interface is very useful to decouple the interpreter's query execution and the sessions themselves.
*/
class SessionInterface {
public:
SessionInterface() = default;
virtual ~SessionInterface() = default;
SessionInterface(const SessionInterface &) = default;
SessionInterface &operator=(const SessionInterface &) = default;
SessionInterface(SessionInterface &&) noexcept = default;
SessionInterface &operator=(SessionInterface &&) noexcept = default;
/**
* @brief Return the unique string identifying the session.
*
* @return std::string
*/
virtual std::string UUID() const = 0;
/**
* @brief Return the currently active database.
*
* @return std::string
*/
virtual std::string GetDatabaseName() const = 0;
#ifdef MG_ENTERPRISE
/**
* @brief Gets called on database change.
*
* @return SetForResult enum (SUCCESS, ALREADY_SET or FAIL)
*/
virtual dbms::SetForResult OnChange(const std::string &) = 0;
/**
* @brief Callback that gets called on database delete (drop).
*
* @return true on success
*/
virtual bool OnDelete(const std::string &) = 0;
#endif
SPECIALIZE_GET_EXCEPTION_NAME(UnknownDatabaseException)
};
} // namespace memgraph::dbms

View File

@@ -18,21 +18,21 @@
#include <unordered_map>
#include "global.hpp"
#include "utils/exceptions.hpp"
#include "utils/gatekeeper.hpp"
#include "utils/result.hpp"
#include "utils/sync_ptr.hpp"
namespace memgraph::dbms {
/**
* @brief Generic multi-database content handler.
*
* @tparam TContext
* @tparam TConfig
* @tparam T
*/
template <typename TContext, typename TConfig>
template <typename T>
class Handler {
public:
using NewResult = utils::BasicResult<NewError, std::shared_ptr<TContext>>;
using NewResult = utils::BasicResult<NewError, typename utils::Gatekeeper<T>::Accessor>;
/**
* @brief Empty Handler constructor.
@@ -43,67 +43,65 @@ class Handler {
/**
* @brief Generate a new context and corresponding configuration.
*
* @tparam T1 Variadic template of context constructor arguments
* @tparam T2 Variadic template of config constructor arguments
* @param name Name associated with the new context/config pair
* @param args1 Arguments passed (as a tuple) to the context constructor
* @param args2 Arguments passed (as a tuple) to the config constructor
* @tparam Args Variadic template of constructor arguments of T
* @param name Name associated with the new T
* @param args Arguments passed to the constructor of T
* @return NewResult
*/
template <typename... T1, typename... T2>
NewResult New(std::string name, std::tuple<T1...> args1, std::tuple<T2...> args2) {
return New_(name, args1, args2, std::make_index_sequence<sizeof...(T1)>{},
std::make_index_sequence<sizeof...(T2)>{});
template <typename... Args>
NewResult New(std::piecewise_construct_t /* marker */, std::string_view name, Args &&...args) {
// Make sure the emplace will succeed, since we don't want to create temporary objects that could break something
if (!Has(name)) {
auto [itr, _] = items_.emplace(std::piecewise_construct, std::forward_as_tuple(name),
std::forward_as_tuple(std::forward<Args>(args)...));
auto db_acc = itr->second.access();
if (db_acc) return std::move(*db_acc);
return NewError::DEFUNCT;
}
spdlog::info("Item with name \"{}\" already exists.", name);
return NewError::EXISTS;
}
/**
* @brief Get pointer to context.
*
* @param name Name associated with the wanted context
* @return std::optional<std::shared_ptr<TContext>>
* @return std::optional<typename utils::Gatekeeper<T>::Accessor>
*/
std::optional<std::shared_ptr<TContext>> Get(const std::string &name) {
std::optional<typename utils::Gatekeeper<T>::Accessor> Get(std::string_view name) {
if (auto search = items_.find(name); search != items_.end()) {
return search->second.get();
return search->second.access();
}
return {};
return std::nullopt;
}
/**
* @brief Get the config.
* @brief Delete the context associated with the name.
*
* @param name Name associated with the wanted config
* @return std::optional<TConfig>
*/
std::optional<TConfig> GetConfig(const std::string &name) const {
if (auto search = items_.find(name); search != items_.end()) {
return search->second.config();
}
return {};
}
/**
* @brief Delete the context/config pair associated with the name.
*
* @param name Name associated with the context/config pair to delete
* @param name Name associated with the context to delete
* @return true on success
* @throw BasicException
*/
bool Delete(const std::string &name) {
if (auto itr = items_.find(name); itr != items_.end()) {
itr->second.DestroyAndSync();
items_.erase(itr);
return true;
auto db_acc = itr->second.access();
if (db_acc && db_acc->try_delete()) {
db_acc->reset();
items_.erase(itr);
return true;
}
return false;
}
return false;
throw utils::BasicException("Unknown item \"{}\".", name);
}
/**
* @brief Check if a name is already used.
*
* @param name Name to check
* @return true if a context/config pair is already associated with the name
* @return true if a T is already associated with the name
*/
bool Has(const std::string &name) const { return items_.find(name) != items_.end(); }
bool Has(std::string_view name) const { return items_.find(name) != items_.end(); }
auto begin() { return items_.begin(); }
auto end() { return items_.end(); }
@@ -112,31 +110,16 @@ class Handler {
auto cbegin() const { return items_.cbegin(); }
auto cend() const { return items_.cend(); }
private:
/**
* @brief Lower level handler that hides some ugly code.
*
* @tparam T1 Variadic template of context constructor arguments
* @tparam T2 Variadic template of config constructor arguments
* @tparam I1 List of indexes associated with the first tuple
* @tparam I2 List of indexes associated with the second tuple
*/
template <typename... T1, typename... T2, std::size_t... I1, std::size_t... I2>
NewResult New_(std::string name, std::tuple<T1...> &args1, std::tuple<T2...> &args2,
std::integer_sequence<std::size_t, I1...> /*not-used*/,
std::integer_sequence<std::size_t, I2...> /*not-used*/) {
// Make sure the emplace will succeed, since we don't want to create temporary objects that could break something
if (!Has(name)) {
auto [itr, _] = items_.emplace(std::piecewise_construct, std::forward_as_tuple(name),
std::forward_as_tuple(TConfig{std::forward<T1>(std::get<I1>(args1))...},
std::forward<T2>(std::get<I2>(args2))...));
return itr->second.get();
}
spdlog::info("Item with name \"{}\" already exists.", name);
return NewError::EXISTS;
}
struct string_hash {
using is_transparent = void;
[[nodiscard]] size_t operator()(const char *s) const { return std::hash<std::string_view>{}(s); }
[[nodiscard]] size_t operator()(std::string_view s) const { return std::hash<std::string_view>{}(s); }
[[nodiscard]] size_t operator()(const std::string &s) const { return std::hash<std::string>{}(s); }
};
std::unordered_map<std::string, utils::SyncPtr<TContext, TConfig>> items_; //!< map to all active items
private:
std::unordered_map<std::string, utils::Gatekeeper<T>, string_hash, std::equal_to<>>
items_; //!< map to all active items
};
} // namespace memgraph::dbms

View File

@@ -0,0 +1,703 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "dbms/inmemory/replication_handlers.hpp"
#include "dbms/constants.hpp"
#include "dbms/dbms_handler.hpp"
#include "replication/replication_server.hpp"
#include "spdlog/spdlog.h"
#include "storage/v2/durability/durability.hpp"
#include "storage/v2/durability/snapshot.hpp"
#include "storage/v2/durability/version.hpp"
#include "storage/v2/indices/label_index_stats.hpp"
#include "storage/v2/inmemory/storage.hpp"
#include "storage/v2/inmemory/unique_constraints.hpp"
using memgraph::storage::Delta;
using memgraph::storage::EdgeAccessor;
using memgraph::storage::EdgeRef;
using memgraph::storage::EdgeTypeId;
using memgraph::storage::LabelIndexStats;
using memgraph::storage::LabelPropertyIndexStats;
using memgraph::storage::PropertyId;
using memgraph::storage::UniqueConstraints;
using memgraph::storage::View;
using memgraph::storage::durability::WalDeltaData;
namespace memgraph::dbms {
namespace {
std::pair<uint64_t, WalDeltaData> ReadDelta(storage::durability::BaseDecoder *decoder) {
try {
auto timestamp = ReadWalDeltaHeader(decoder);
SPDLOG_INFO(" Timestamp {}", timestamp);
auto delta = ReadWalDeltaData(decoder);
return {timestamp, delta};
} catch (const slk::SlkReaderException &) {
throw utils::BasicException("Missing data!");
} catch (const storage::durability::RecoveryFailure &) {
throw utils::BasicException("Invalid data!");
}
};
std::optional<DatabaseAccess> GetDatabaseAccessor(dbms::DbmsHandler *dbms_handler, std::string_view db_name) {
try {
#ifdef MG_ENTERPRISE
auto acc = dbms_handler->Get(db_name);
#else
if (db_name != dbms::kDefaultDB) {
spdlog::warn("Trying to replicate a non-default database on a community replica.");
return std::nullopt;
}
auto acc = dbms_handler->Get();
#endif
if (!acc) {
spdlog::error("Failed to get access to ", db_name);
return std::nullopt;
}
auto *inmem_storage = dynamic_cast<storage::InMemoryStorage *>(acc.get()->storage());
if (!inmem_storage || inmem_storage->storage_mode_ != storage::StorageMode::IN_MEMORY_TRANSACTIONAL) {
spdlog::error("Database \"{}\" is not IN_MEMORY_TRANSACTIONAL.", db_name);
return std::nullopt;
}
return std::optional{std::move(acc)};
} catch (const dbms::UnknownDatabaseException &e) {
spdlog::warn("No database \"{}\" on replica!", db_name);
return std::nullopt;
}
}
} // namespace
void InMemoryReplicationHandlers::Register(dbms::DbmsHandler *dbms_handler, replication::ReplicationServer &server) {
server.rpc_server_.Register<storage::replication::HeartbeatRpc>([dbms_handler](auto *req_reader, auto *res_builder) {
spdlog::debug("Received HeartbeatRpc");
InMemoryReplicationHandlers::HeartbeatHandler(dbms_handler, req_reader, res_builder);
});
server.rpc_server_.Register<storage::replication::AppendDeltasRpc>(
[dbms_handler](auto *req_reader, auto *res_builder) {
spdlog::debug("Received AppendDeltasRpc");
InMemoryReplicationHandlers::AppendDeltasHandler(dbms_handler, req_reader, res_builder);
});
server.rpc_server_.Register<storage::replication::SnapshotRpc>([dbms_handler](auto *req_reader, auto *res_builder) {
spdlog::debug("Received SnapshotRpc");
InMemoryReplicationHandlers::SnapshotHandler(dbms_handler, req_reader, res_builder);
});
server.rpc_server_.Register<storage::replication::WalFilesRpc>([dbms_handler](auto *req_reader, auto *res_builder) {
spdlog::debug("Received WalFilesRpc");
InMemoryReplicationHandlers::WalFilesHandler(dbms_handler, req_reader, res_builder);
});
server.rpc_server_.Register<storage::replication::CurrentWalRpc>([dbms_handler](auto *req_reader, auto *res_builder) {
spdlog::debug("Received CurrentWalRpc");
InMemoryReplicationHandlers::CurrentWalHandler(dbms_handler, req_reader, res_builder);
});
server.rpc_server_.Register<storage::replication::TimestampRpc>([dbms_handler](auto *req_reader, auto *res_builder) {
spdlog::debug("Received TimestampRpc");
InMemoryReplicationHandlers::TimestampHandler(dbms_handler, req_reader, res_builder);
});
}
void InMemoryReplicationHandlers::HeartbeatHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader,
slk::Builder *res_builder) {
storage::replication::HeartbeatReq req;
slk::Load(&req, req_reader);
auto const db_acc = GetDatabaseAccessor(dbms_handler, req.db_name);
if (!db_acc) return;
// TODO: this handler is agnostic of InMemory, move to be reused by on-disk
auto const *storage = db_acc->get()->storage();
storage::replication::HeartbeatRes res{storage->id(), true,
storage->repl_storage_state_.last_commit_timestamp_.load(),
std::string{storage->repl_storage_state_.epoch_.id()}};
slk::Save(res, res_builder);
}
void InMemoryReplicationHandlers::AppendDeltasHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader,
slk::Builder *res_builder) {
storage::replication::AppendDeltasReq req;
slk::Load(&req, req_reader);
auto db_acc = GetDatabaseAccessor(dbms_handler, req.db_name);
if (!db_acc) return;
storage::replication::Decoder decoder(req_reader);
auto maybe_epoch_id = decoder.ReadString();
MG_ASSERT(maybe_epoch_id, "Invalid replication message");
auto *storage = static_cast<storage::InMemoryStorage *>(db_acc->get()->storage());
auto &repl_storage_state = storage->repl_storage_state_;
if (*maybe_epoch_id != storage->repl_storage_state_.epoch_.id()) {
auto prev_epoch = storage->repl_storage_state_.epoch_.SetEpoch(*maybe_epoch_id);
repl_storage_state.AddEpochToHistoryForce(prev_epoch);
}
if (storage->wal_file_) {
if (req.seq_num > storage->wal_file_->SequenceNumber() ||
*maybe_epoch_id != storage->repl_storage_state_.epoch_.id()) {
storage->wal_file_->FinalizeWal();
storage->wal_file_.reset();
storage->wal_seq_num_ = req.seq_num;
spdlog::trace("Finalized WAL file");
} else {
MG_ASSERT(storage->wal_file_->SequenceNumber() == req.seq_num, "Invalid sequence number of current wal file");
storage->wal_seq_num_ = req.seq_num + 1;
}
} else {
storage->wal_seq_num_ = req.seq_num;
}
if (req.previous_commit_timestamp != repl_storage_state.last_commit_timestamp_.load()) {
// Empty the stream
bool transaction_complete = false;
while (!transaction_complete) {
SPDLOG_INFO("Skipping delta");
const auto [timestamp, delta] = ReadDelta(&decoder);
transaction_complete = storage::durability::IsWalDeltaDataTypeTransactionEnd(
delta.type,
storage::durability::kVersion); // TODO: Check if we are always using the latest version when replicating
}
storage::replication::AppendDeltasRes res{storage->id(), false, repl_storage_state.last_commit_timestamp_.load()};
slk::Save(res, res_builder);
return;
}
ReadAndApplyDelta(
storage, &decoder,
storage::durability::kVersion); // TODO: Check if we are always using the latest version when replicating
storage::replication::AppendDeltasRes res{storage->id(), true, repl_storage_state.last_commit_timestamp_.load()};
slk::Save(res, res_builder);
spdlog::debug("Replication recovery from append deltas finished, replica is now up to date!");
}
void InMemoryReplicationHandlers::SnapshotHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader,
slk::Builder *res_builder) {
storage::replication::SnapshotReq req;
slk::Load(&req, req_reader);
auto db_acc = GetDatabaseAccessor(dbms_handler, req.db_name);
if (!db_acc) return;
storage::replication::Decoder decoder(req_reader);
auto *storage = static_cast<storage::InMemoryStorage *>(db_acc->get()->storage());
utils::EnsureDirOrDie(storage->snapshot_directory_);
const auto maybe_snapshot_path = decoder.ReadFile(storage->snapshot_directory_);
MG_ASSERT(maybe_snapshot_path, "Failed to load snapshot!");
spdlog::info("Received snapshot saved to {}", *maybe_snapshot_path);
auto storage_guard = std::unique_lock{storage->main_lock_};
spdlog::trace("Clearing database since recovering from snapshot.");
// Clear the database
storage->vertices_.clear();
storage->edges_.clear();
storage->constraints_.existence_constraints_ = std::make_unique<storage::ExistenceConstraints>();
storage->constraints_.unique_constraints_ = std::make_unique<storage::InMemoryUniqueConstraints>();
storage->indices_.label_index_ = std::make_unique<storage::InMemoryLabelIndex>();
storage->indices_.label_property_index_ = std::make_unique<storage::InMemoryLabelPropertyIndex>();
try {
spdlog::debug("Loading snapshot");
auto recovered_snapshot = storage::durability::LoadSnapshot(
*maybe_snapshot_path, &storage->vertices_, &storage->edges_, &storage->repl_storage_state_.history,
storage->name_id_mapper_.get(), &storage->edge_count_, storage->config_);
spdlog::debug("Snapshot loaded successfully");
// If this step is present it should always be the first step of
// the recovery so we use the UUID we read from snasphost
storage->uuid_ = std::move(recovered_snapshot.snapshot_info.uuid);
storage->repl_storage_state_.epoch_.SetEpoch(std::move(recovered_snapshot.snapshot_info.epoch_id));
const auto &recovery_info = recovered_snapshot.recovery_info;
storage->vertex_id_ = recovery_info.next_vertex_id;
storage->edge_id_ = recovery_info.next_edge_id;
storage->timestamp_ = std::max(storage->timestamp_, recovery_info.next_timestamp);
spdlog::trace("Recovering indices and constraints from snapshot.");
storage::durability::RecoverIndicesAndConstraints(recovered_snapshot.indices_constraints, &storage->indices_,
&storage->constraints_, &storage->vertices_,
storage->name_id_mapper_.get());
} catch (const storage::durability::RecoveryFailure &e) {
LOG_FATAL("Couldn't load the snapshot because of: {}", e.what());
}
storage_guard.unlock();
storage::replication::SnapshotRes res{storage->id(), true,
storage->repl_storage_state_.last_commit_timestamp_.load()};
slk::Save(res, res_builder);
spdlog::trace("Deleting old snapshot files due to snapshot recovery.");
// Delete other durability files
auto snapshot_files = storage::durability::GetSnapshotFiles(storage->snapshot_directory_, storage->uuid_);
for (const auto &[path, uuid, _] : snapshot_files) {
if (path != *maybe_snapshot_path) {
spdlog::trace("Deleting snapshot file {}", path);
storage->file_retainer_.DeleteFile(path);
}
}
spdlog::trace("Deleting old WAL files due to snapshot recovery.");
auto wal_files = storage::durability::GetWalFiles(storage->wal_directory_, storage->uuid_);
if (wal_files) {
for (const auto &wal_file : *wal_files) {
spdlog::trace("Deleting WAL file {}", wal_file.path);
storage->file_retainer_.DeleteFile(wal_file.path);
}
storage->wal_file_.reset();
}
spdlog::debug("Replication recovery from snapshot finished!");
}
void InMemoryReplicationHandlers::WalFilesHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader,
slk::Builder *res_builder) {
storage::replication::WalFilesReq req;
slk::Load(&req, req_reader);
auto db_acc = GetDatabaseAccessor(dbms_handler, req.db_name);
if (!db_acc) return;
const auto wal_file_number = req.file_number;
spdlog::debug("Received WAL files: {}", wal_file_number);
storage::replication::Decoder decoder(req_reader);
auto *storage = static_cast<storage::InMemoryStorage *>(db_acc->get()->storage());
utils::EnsureDirOrDie(storage->wal_directory_);
for (auto i = 0; i < wal_file_number; ++i) {
LoadWal(storage, &decoder);
}
storage::replication::WalFilesRes res{storage->id(), true,
storage->repl_storage_state_.last_commit_timestamp_.load()};
slk::Save(res, res_builder);
spdlog::debug("Replication recovery from WAL files ended successfully, replica is now up to date!");
}
void InMemoryReplicationHandlers::CurrentWalHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader,
slk::Builder *res_builder) {
storage::replication::CurrentWalReq req;
slk::Load(&req, req_reader);
auto db_acc = GetDatabaseAccessor(dbms_handler, req.db_name);
if (!db_acc) return;
storage::replication::Decoder decoder(req_reader);
auto *storage = static_cast<storage::InMemoryStorage *>(db_acc->get()->storage());
utils::EnsureDirOrDie(storage->wal_directory_);
LoadWal(storage, &decoder);
storage::replication::CurrentWalRes res{storage->id(), true,
storage->repl_storage_state_.last_commit_timestamp_.load()};
slk::Save(res, res_builder);
spdlog::debug("Replication recovery from current WAL ended successfully, replica is now up to date!");
}
void InMemoryReplicationHandlers::LoadWal(storage::InMemoryStorage *storage, storage::replication::Decoder *decoder) {
const auto temp_wal_directory =
std::filesystem::temp_directory_path() / "memgraph" / storage::durability::kWalDirectory;
utils::EnsureDir(temp_wal_directory);
auto maybe_wal_path = decoder->ReadFile(temp_wal_directory);
MG_ASSERT(maybe_wal_path, "Failed to load WAL!");
spdlog::trace("Received WAL saved to {}", *maybe_wal_path);
try {
auto wal_info = storage::durability::ReadWalInfo(*maybe_wal_path);
if (wal_info.seq_num == 0) {
storage->uuid_ = wal_info.uuid;
}
auto &replica_epoch = storage->repl_storage_state_.epoch_;
if (wal_info.epoch_id != replica_epoch.id()) {
auto prev_epoch = replica_epoch.SetEpoch(wal_info.epoch_id);
storage->repl_storage_state_.AddEpochToHistoryForce(prev_epoch);
}
if (storage->wal_file_) {
if (storage->wal_file_->SequenceNumber() != wal_info.seq_num) {
storage->wal_file_->FinalizeWal();
storage->wal_seq_num_ = wal_info.seq_num;
storage->wal_file_.reset();
spdlog::trace("WAL file {} finalized successfully", *maybe_wal_path);
}
} else {
storage->wal_seq_num_ = wal_info.seq_num;
}
spdlog::trace("Loading WAL deltas from {}", *maybe_wal_path);
storage::durability::Decoder wal;
const auto version = wal.Initialize(*maybe_wal_path, storage::durability::kWalMagic);
spdlog::debug("WAL file {} loaded successfully", *maybe_wal_path);
if (!version) throw storage::durability::RecoveryFailure("Couldn't read WAL magic and/or version!");
if (!storage::durability::IsVersionSupported(*version))
throw storage::durability::RecoveryFailure("Invalid WAL version!");
wal.SetPosition(wal_info.offset_deltas);
for (size_t i = 0; i < wal_info.num_deltas;) {
i += ReadAndApplyDelta(storage, &wal, *version);
}
spdlog::debug("Replication from current WAL successful!");
} catch (const storage::durability::RecoveryFailure &e) {
LOG_FATAL("Couldn't recover WAL deltas from {} because of: {}", *maybe_wal_path, e.what());
}
}
void InMemoryReplicationHandlers::TimestampHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader,
slk::Builder *res_builder) {
storage::replication::TimestampReq req;
slk::Load(&req, req_reader);
auto const db_acc = GetDatabaseAccessor(dbms_handler, req.db_name);
if (!db_acc) return;
// TODO: this handler is agnostic of InMemory, move to be reused by on-disk
auto const *storage = db_acc->get()->storage();
storage::replication::TimestampRes res{storage->id(), true,
storage->repl_storage_state_.last_commit_timestamp_.load()};
slk::Save(res, res_builder);
}
uint64_t InMemoryReplicationHandlers::ReadAndApplyDelta(storage::InMemoryStorage *storage,
storage::durability::BaseDecoder *decoder,
const uint64_t version) {
auto edge_acc = storage->edges_.access();
auto vertex_acc = storage->vertices_.access();
constexpr bool kUniqueAccess = true;
constexpr bool kSharedAccess = false;
std::optional<std::pair<uint64_t, storage::InMemoryStorage::ReplicationAccessor>> commit_timestamp_and_accessor;
auto get_transaction = [storage, &commit_timestamp_and_accessor](uint64_t commit_timestamp,
bool unique = kSharedAccess) {
if (!commit_timestamp_and_accessor) {
std::unique_ptr<storage::Storage::Accessor> acc = nullptr;
if (unique) {
acc = storage->UniqueAccess(std::nullopt, false /*not main*/);
} else {
acc = storage->Access(std::nullopt, false /*not main*/);
}
auto inmem_acc = std::unique_ptr<storage::InMemoryStorage::InMemoryAccessor>(
static_cast<storage::InMemoryStorage::InMemoryAccessor *>(acc.release()));
commit_timestamp_and_accessor.emplace(commit_timestamp, std::move(*inmem_acc));
} else if (commit_timestamp_and_accessor->first != commit_timestamp) {
throw utils::BasicException("Received more than one transaction!");
}
return &commit_timestamp_and_accessor->second;
};
uint64_t applied_deltas = 0;
auto max_commit_timestamp = storage->repl_storage_state_.last_commit_timestamp_.load();
for (bool transaction_complete = false; !transaction_complete; ++applied_deltas) {
const auto [timestamp, delta] = ReadDelta(decoder);
if (timestamp > max_commit_timestamp) {
max_commit_timestamp = timestamp;
}
transaction_complete = storage::durability::IsWalDeltaDataTypeTransactionEnd(delta.type, version);
if (timestamp < storage->timestamp_) {
continue;
}
SPDLOG_INFO(" Delta {}", applied_deltas);
switch (delta.type) {
case WalDeltaData::Type::VERTEX_CREATE: {
spdlog::trace(" Create vertex {}", delta.vertex_create_delete.gid.AsUint());
auto *transaction = get_transaction(timestamp);
transaction->CreateVertexEx(delta.vertex_create_delete.gid);
break;
}
case WalDeltaData::Type::VERTEX_DELETE: {
spdlog::trace(" Delete vertex {}", delta.vertex_create_delete.gid.AsUint());
auto *transaction = get_transaction(timestamp);
auto vertex = transaction->FindVertex(delta.vertex_create_delete.gid, View::NEW);
if (!vertex) throw utils::BasicException("Invalid transaction!");
auto ret = transaction->DeleteVertex(&*vertex);
if (ret.HasError() || !ret.GetValue()) throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::VERTEX_ADD_LABEL: {
spdlog::trace(" Vertex {} add label {}", delta.vertex_add_remove_label.gid.AsUint(),
delta.vertex_add_remove_label.label);
auto *transaction = get_transaction(timestamp);
auto vertex = transaction->FindVertex(delta.vertex_add_remove_label.gid, View::NEW);
if (!vertex) throw utils::BasicException("Invalid transaction!");
auto ret = vertex->AddLabel(transaction->NameToLabel(delta.vertex_add_remove_label.label));
if (ret.HasError() || !ret.GetValue()) throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::VERTEX_REMOVE_LABEL: {
spdlog::trace(" Vertex {} remove label {}", delta.vertex_add_remove_label.gid.AsUint(),
delta.vertex_add_remove_label.label);
auto *transaction = get_transaction(timestamp);
auto vertex = transaction->FindVertex(delta.vertex_add_remove_label.gid, View::NEW);
if (!vertex) throw utils::BasicException("Invalid transaction!");
auto ret = vertex->RemoveLabel(transaction->NameToLabel(delta.vertex_add_remove_label.label));
if (ret.HasError() || !ret.GetValue()) throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::VERTEX_SET_PROPERTY: {
spdlog::trace(" Vertex {} set property {} to {}", delta.vertex_edge_set_property.gid.AsUint(),
delta.vertex_edge_set_property.property, delta.vertex_edge_set_property.value);
auto *transaction = get_transaction(timestamp);
auto vertex = transaction->FindVertex(delta.vertex_edge_set_property.gid, View::NEW);
if (!vertex) throw utils::BasicException("Invalid transaction!");
auto ret = vertex->SetProperty(transaction->NameToProperty(delta.vertex_edge_set_property.property),
delta.vertex_edge_set_property.value);
if (ret.HasError()) throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::EDGE_CREATE: {
spdlog::trace(" Create edge {} of type {} from vertex {} to vertex {}",
delta.edge_create_delete.gid.AsUint(), delta.edge_create_delete.edge_type,
delta.edge_create_delete.from_vertex.AsUint(), delta.edge_create_delete.to_vertex.AsUint());
auto *transaction = get_transaction(timestamp);
auto from_vertex = transaction->FindVertex(delta.edge_create_delete.from_vertex, View::NEW);
if (!from_vertex) throw utils::BasicException("Invalid transaction!");
auto to_vertex = transaction->FindVertex(delta.edge_create_delete.to_vertex, View::NEW);
if (!to_vertex) throw utils::BasicException("Invalid transaction!");
auto edge = transaction->CreateEdgeEx(&*from_vertex, &*to_vertex,
transaction->NameToEdgeType(delta.edge_create_delete.edge_type),
delta.edge_create_delete.gid);
if (edge.HasError()) throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::EDGE_DELETE: {
spdlog::trace(" Delete edge {} of type {} from vertex {} to vertex {}",
delta.edge_create_delete.gid.AsUint(), delta.edge_create_delete.edge_type,
delta.edge_create_delete.from_vertex.AsUint(), delta.edge_create_delete.to_vertex.AsUint());
auto *transaction = get_transaction(timestamp);
auto from_vertex = transaction->FindVertex(delta.edge_create_delete.from_vertex, View::NEW);
if (!from_vertex) throw utils::BasicException("Invalid transaction!");
auto to_vertex = transaction->FindVertex(delta.edge_create_delete.to_vertex, View::NEW);
if (!to_vertex) throw utils::BasicException("Invalid transaction!");
auto edges = from_vertex->OutEdges(View::NEW, {transaction->NameToEdgeType(delta.edge_create_delete.edge_type)},
&*to_vertex);
if (edges.HasError()) throw utils::BasicException("Invalid transaction!");
if (edges->edges.size() != 1) throw utils::BasicException("Invalid transaction!");
auto &edge = (*edges).edges[0];
auto ret = transaction->DeleteEdge(&edge);
if (ret.HasError()) throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::EDGE_SET_PROPERTY: {
spdlog::trace(" Edge {} set property {} to {}", delta.vertex_edge_set_property.gid.AsUint(),
delta.vertex_edge_set_property.property, delta.vertex_edge_set_property.value);
if (!storage->config_.items.properties_on_edges)
throw utils::BasicException(
"Can't set properties on edges because properties on edges "
"are disabled!");
auto *transaction = get_transaction(timestamp);
// The following block of code effectively implements `FindEdge` and
// yields an accessor that is only valid for managing the edge's
// properties.
auto edge = edge_acc.find(delta.vertex_edge_set_property.gid);
if (edge == edge_acc.end()) throw utils::BasicException("Invalid transaction!");
// The edge visibility check must be done here manually because we
// don't allow direct access to the edges through the public API.
{
bool is_visible = true;
Delta *delta = nullptr;
{
auto guard = std::shared_lock{edge->lock};
is_visible = !edge->deleted;
delta = edge->delta;
}
ApplyDeltasForRead(&transaction->GetTransaction(), delta, View::NEW, [&is_visible](const Delta &delta) {
switch (delta.action) {
case Delta::Action::ADD_LABEL:
case Delta::Action::REMOVE_LABEL:
case Delta::Action::SET_PROPERTY:
case Delta::Action::ADD_IN_EDGE:
case Delta::Action::ADD_OUT_EDGE:
case Delta::Action::REMOVE_IN_EDGE:
case Delta::Action::REMOVE_OUT_EDGE:
break;
case Delta::Action::RECREATE_OBJECT: {
is_visible = true;
break;
}
case Delta::Action::DELETE_DESERIALIZED_OBJECT:
case Delta::Action::DELETE_OBJECT: {
is_visible = false;
break;
}
}
});
if (!is_visible) throw utils::BasicException("Invalid transaction!");
}
EdgeRef edge_ref(&*edge);
// Here we create an edge accessor that we will use to get the
// properties of the edge. The accessor is created with an invalid
// type and invalid from/to pointers because we don't know them
// here, but that isn't an issue because we won't use that part of
// the API here.
auto ea = EdgeAccessor{edge_ref, EdgeTypeId::FromUint(0UL), nullptr, nullptr,
storage, &transaction->GetTransaction()};
auto ret = ea.SetProperty(transaction->NameToProperty(delta.vertex_edge_set_property.property),
delta.vertex_edge_set_property.value);
if (ret.HasError()) throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::TRANSACTION_END: {
spdlog::trace(" Transaction end");
if (!commit_timestamp_and_accessor || commit_timestamp_and_accessor->first != timestamp)
throw utils::BasicException("Invalid commit data!");
auto ret =
commit_timestamp_and_accessor->second.Commit(commit_timestamp_and_accessor->first, false /* not main */);
if (ret.HasError()) throw utils::BasicException("Invalid transaction!");
commit_timestamp_and_accessor = std::nullopt;
break;
}
case WalDeltaData::Type::LABEL_INDEX_CREATE: {
spdlog::trace(" Create label index on :{}", delta.operation_label.label);
// Need to send the timestamp
auto *transaction = get_transaction(timestamp, kUniqueAccess);
if (transaction->CreateIndex(storage->NameToLabel(delta.operation_label.label)).HasError())
throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::LABEL_INDEX_DROP: {
spdlog::trace(" Drop label index on :{}", delta.operation_label.label);
auto *transaction = get_transaction(timestamp, kUniqueAccess);
if (transaction->DropIndex(storage->NameToLabel(delta.operation_label.label)).HasError())
throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::LABEL_INDEX_STATS_SET: {
spdlog::trace(" Set label index statistics on :{}", delta.operation_label_stats.label);
// Need to send the timestamp
auto *transaction = get_transaction(timestamp);
const auto label = storage->NameToLabel(delta.operation_label_stats.label);
LabelIndexStats stats{};
if (!FromJson(delta.operation_label_stats.stats, stats)) {
throw utils::BasicException("Failed to read statistics!");
}
transaction->SetIndexStats(label, stats);
break;
}
case WalDeltaData::Type::LABEL_INDEX_STATS_CLEAR: {
const auto &info = delta.operation_label;
spdlog::trace(" Clear label index statistics on :{}", info.label);
// Need to send the timestamp
auto *transaction = get_transaction(timestamp);
transaction->DeleteLabelIndexStats(storage->NameToLabel(info.label));
break;
}
case WalDeltaData::Type::LABEL_PROPERTY_INDEX_CREATE: {
spdlog::trace(" Create label+property index on :{} ({})", delta.operation_label_property.label,
delta.operation_label_property.property);
auto *transaction = get_transaction(timestamp, kUniqueAccess);
if (transaction
->CreateIndex(storage->NameToLabel(delta.operation_label_property.label),
storage->NameToProperty(delta.operation_label_property.property))
.HasError())
throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::LABEL_PROPERTY_INDEX_DROP: {
spdlog::trace(" Drop label+property index on :{} ({})", delta.operation_label_property.label,
delta.operation_label_property.property);
auto *transaction = get_transaction(timestamp, kUniqueAccess);
if (transaction
->DropIndex(storage->NameToLabel(delta.operation_label_property.label),
storage->NameToProperty(delta.operation_label_property.property))
.HasError())
throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::LABEL_PROPERTY_INDEX_STATS_SET: {
const auto &info = delta.operation_label_property_stats;
spdlog::trace(" Set label-property index statistics on :{}", info.label);
// Need to send the timestamp
auto *transaction = get_transaction(timestamp);
const auto label = storage->NameToLabel(info.label);
const auto property = storage->NameToProperty(info.property);
LabelPropertyIndexStats stats{};
if (!FromJson(info.stats, stats)) {
throw utils::BasicException("Failed to read statistics!");
}
transaction->SetIndexStats(label, property, stats);
break;
}
case WalDeltaData::Type::LABEL_PROPERTY_INDEX_STATS_CLEAR: {
const auto &info = delta.operation_label;
spdlog::trace(" Clear label-property index statistics on :{}", info.label);
// Need to send the timestamp
auto *transaction = get_transaction(timestamp);
transaction->DeleteLabelPropertyIndexStats(storage->NameToLabel(info.label));
break;
}
case WalDeltaData::Type::EXISTENCE_CONSTRAINT_CREATE: {
spdlog::trace(" Create existence constraint on :{} ({})", delta.operation_label_property.label,
delta.operation_label_property.property);
auto *transaction = get_transaction(timestamp, kUniqueAccess);
auto ret =
transaction->CreateExistenceConstraint(storage->NameToLabel(delta.operation_label_property.label),
storage->NameToProperty(delta.operation_label_property.property));
if (ret.HasError()) throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::EXISTENCE_CONSTRAINT_DROP: {
spdlog::trace(" Drop existence constraint on :{} ({})", delta.operation_label_property.label,
delta.operation_label_property.property);
auto *transaction = get_transaction(timestamp, kUniqueAccess);
if (transaction
->DropExistenceConstraint(storage->NameToLabel(delta.operation_label_property.label),
storage->NameToProperty(delta.operation_label_property.property))
.HasError())
throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::UNIQUE_CONSTRAINT_CREATE: {
std::stringstream ss;
utils::PrintIterable(ss, delta.operation_label_properties.properties);
spdlog::trace(" Create unique constraint on :{} ({})", delta.operation_label_properties.label, ss.str());
std::set<PropertyId> properties;
for (const auto &prop : delta.operation_label_properties.properties) {
properties.emplace(storage->NameToProperty(prop));
}
auto *transaction = get_transaction(timestamp, kUniqueAccess);
auto ret = transaction->CreateUniqueConstraint(storage->NameToLabel(delta.operation_label_properties.label),
properties);
if (!ret.HasValue() || ret.GetValue() != UniqueConstraints::CreationStatus::SUCCESS)
throw utils::BasicException("Invalid transaction!");
break;
}
case WalDeltaData::Type::UNIQUE_CONSTRAINT_DROP: {
std::stringstream ss;
utils::PrintIterable(ss, delta.operation_label_properties.properties);
spdlog::trace(" Drop unique constraint on :{} ({})", delta.operation_label_properties.label, ss.str());
std::set<PropertyId> properties;
for (const auto &prop : delta.operation_label_properties.properties) {
properties.emplace(storage->NameToProperty(prop));
}
auto *transaction = get_transaction(timestamp, kUniqueAccess);
auto ret =
transaction->DropUniqueConstraint(storage->NameToLabel(delta.operation_label_properties.label), properties);
if (ret != UniqueConstraints::DeletionStatus::SUCCESS) {
throw utils::BasicException("Invalid transaction!");
}
break;
}
}
}
if (commit_timestamp_and_accessor) throw utils::BasicException("Did not finish the transaction!");
storage->repl_storage_state_.last_commit_timestamp_ = max_commit_timestamp;
spdlog::debug("Applied {} deltas", applied_deltas);
return applied_deltas;
}
} // namespace memgraph::dbms

View File

@@ -0,0 +1,49 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "replication/replication_server.hpp"
#include "replication/state.hpp"
#include "storage/v2/replication/serialization.hpp"
namespace memgraph::storage {
class InMemoryStorage;
}
namespace memgraph::dbms {
class DbmsHandler;
class InMemoryReplicationHandlers {
public:
static void Register(dbms::DbmsHandler *dbms_handler, replication::ReplicationServer &server);
private:
// RPC handlers
static void HeartbeatHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader, slk::Builder *res_builder);
static void AppendDeltasHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader, slk::Builder *res_builder);
static void SnapshotHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader, slk::Builder *res_builder);
static void WalFilesHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader, slk::Builder *res_builder);
static void CurrentWalHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader, slk::Builder *res_builder);
static void TimestampHandler(dbms::DbmsHandler *dbms_handler, slk::Reader *req_reader, slk::Builder *res_builder);
static void LoadWal(storage::InMemoryStorage *storage, storage::replication::Decoder *decoder);
static uint64_t ReadAndApplyDelta(storage::InMemoryStorage *storage, storage::durability::BaseDecoder *decoder,
uint64_t version);
};
} // namespace memgraph::dbms

View File

@@ -0,0 +1,67 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <variant>
#include "dbms/constants.hpp"
#include "dbms/replication_handler.hpp"
#include "replication/state.hpp"
#include "storage/v2/config.hpp"
#include "storage/v2/inmemory/storage.hpp"
#include "storage/v2/storage.hpp"
namespace memgraph::dbms {
#ifdef MG_EXPERIMENTAL_REPLICATION_MULTITENANCY
constexpr bool allow_mt_repl = true;
#else
constexpr bool allow_mt_repl = false;
#endif
inline std::unique_ptr<storage::Storage> CreateInMemoryStorage(
storage::Config config, const ::memgraph::replication::ReplicationState &repl_state) {
const auto wal_mode = config.durability.snapshot_wal_mode;
const auto name = config.name;
auto storage = std::make_unique<storage::InMemoryStorage>(std::move(config));
// Connect replication state and storage
storage->CreateSnapshotHandler(
[storage = storage.get(), &repl_state]() -> utils::BasicResult<storage::InMemoryStorage::CreateSnapshotError> {
if (repl_state.IsReplica()) {
return storage::InMemoryStorage::CreateSnapshotError::DisabledForReplica;
}
return storage->CreateSnapshot();
});
if (allow_mt_repl || name == dbms::kDefaultDB) {
// Handle global replication state
spdlog::info("Replication configuration will be stored and will be automatically restored in case of a crash.");
// RECOVER REPLICA CONNECTIONS
memgraph::dbms::RestoreReplication(repl_state, *storage);
} else if (const ::memgraph::replication::RoleMainData *data =
std::get_if<::memgraph::replication::RoleMainData>(&repl_state.ReplicationData());
data && !data->registered_replicas_.empty()) {
spdlog::warn("Multi-tenant replication is currently not supported!");
}
if (wal_mode == storage::Config::Durability::SnapshotWalMode::DISABLED && repl_state.IsMain()) {
spdlog::warn(
"The instance has the MAIN replication role, but durability logs and snapshots are disabled. Please consider "
"enabling durability by using --storage-snapshot-interval-sec and --storage-wal-enabled flags because "
"without write-ahead logs this instance is not replicating any data.");
}
return std::move(storage);
}
} // namespace memgraph::dbms

View File

@@ -1,106 +0,0 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#ifdef MG_ENTERPRISE
#include "global.hpp"
#include "query/auth_checker.hpp"
#include "query/config.hpp"
#include "query/interpreter.hpp"
#include "storage/v2/storage.hpp"
#include "handler.hpp"
namespace memgraph::dbms {
/**
* @brief Simple class that adds useful information to the query's InterpreterContext
*
* @tparam T Multi-database handler type
*/
template <typename T>
class ExpandedInterpContext : public query::InterpreterContext {
public:
template <typename... TArgs>
explicit ExpandedInterpContext(T &ref, TArgs &&...args)
: query::InterpreterContext(std::forward<TArgs>(args)...), sc_handler_(ref) {}
T &sc_handler_; //!< Multi-database/SessionContext handler (used in some queries)
};
/**
* @brief Simple structure that expands on the query's InterpreterConfig
*
*/
struct ExpandedInterpConfig {
storage::Config storage_config; //!< Storage configuration
query::InterpreterConfig interp_config; //!< Interpreter configuration
};
/**
* @brief Multi-database interpreter context handler
*
* @tparam TSCHandler High-level multi-database/SessionContext handler type
*/
template <typename TSCHandler>
class InterpContextHandler : public Handler<ExpandedInterpContext<TSCHandler>, ExpandedInterpConfig> {
public:
using InterpContextT = ExpandedInterpContext<TSCHandler>;
using HandlerT = Handler<InterpContextT, ExpandedInterpConfig>;
/**
* @brief Generate a new interpreter context associated with the passed name.
*
* @param name Name associating the new interpreter context
* @param sc_handler Multi-database/SessionContext handler used (some queries might use it)
* @param db Storage associated with the interpreter context
* @param config Interpreter's configuration
* @param dir Directory used by the interpreter
* @param auth_handler AuthQueryHandler used
* @param auth_checker AuthChecker used
* @return HandlerT::NewResult
*/
typename HandlerT::NewResult New(const std::string &name, TSCHandler &sc_handler, storage::Config storage_config,
const query::InterpreterConfig &interpreter_config,
query::AuthQueryHandler &auth_handler, query::AuthChecker &auth_checker) {
// Check if compatible with the existing interpreters
if (std::any_of(HandlerT::cbegin(), HandlerT::cend(), [&](const auto &elem) {
const auto &config = elem.second.config().storage_config;
return config.durability.storage_directory == storage_config.durability.storage_directory;
})) {
spdlog::info("Tried to generate a new context using claimed directory and/or storage.");
return NewError::EXISTS;
}
const auto dir = storage_config.durability.storage_directory;
storage_config.name = name; // Set storage id via config
return HandlerT::New(
name, std::forward_as_tuple(storage_config, interpreter_config),
std::forward_as_tuple(sc_handler, storage_config, interpreter_config, dir, &auth_handler, &auth_checker));
}
/**
* @brief All currently active storage.
*
* @return std::vector<std::string>
*/
std::vector<std::string> All() const {
std::vector<std::string> res;
res.reserve(std::distance(HandlerT::cbegin(), HandlerT::cend()));
std::for_each(HandlerT::cbegin(), HandlerT::cend(), [&](const auto &elem) { res.push_back(elem.first); });
return res;
}
};
} // namespace memgraph::dbms
#endif

View File

@@ -0,0 +1,234 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "dbms/replication_handler.hpp"
#include "dbms/constants.hpp"
#include "dbms/dbms_handler.hpp"
#include "dbms/inmemory/replication_handlers.hpp"
#include "dbms/inmemory/storage_helper.hpp"
#include "replication/state.hpp"
using memgraph::replication::ReplicationClientConfig;
using memgraph::replication::ReplicationState;
using memgraph::replication::RoleMainData;
using memgraph::replication::RoleReplicaData;
namespace memgraph::dbms {
namespace {
std::string RegisterReplicaErrorToString(RegisterReplicaError error) {
switch (error) {
using enum RegisterReplicaError;
case NAME_EXISTS:
return "NAME_EXISTS";
case END_POINT_EXISTS:
return "END_POINT_EXISTS";
case CONNECTION_FAILED:
return "CONNECTION_FAILED";
case COULD_NOT_BE_PERSISTED:
return "COULD_NOT_BE_PERSISTED";
}
}
} // namespace
bool ReplicationHandler::SetReplicationRoleMain() {
auto const main_handler = [](RoleMainData const &) {
// If we are already MAIN, we don't want to change anything
return false;
};
auto const replica_handler = [this](RoleReplicaData const &) {
// STEP 1) bring down all REPLICA servers
dbms_handler_.ForEach([](Database *db) {
auto *storage = db->storage();
// Remember old epoch + storage timestamp association
storage->PrepareForNewEpoch();
});
// STEP 2) Change to MAIN
// TODO: restore replication servers if false?
if (!repl_state_.SetReplicationRoleMain()) {
// TODO: Handle recovery on failure???
return false;
}
// STEP 3) We are now MAIN, update storage local epoch
dbms_handler_.ForEach([&](Database *db) {
auto *storage = db->storage();
storage->repl_storage_state_.epoch_ = std::get<RoleMainData>(std::as_const(repl_state_).ReplicationData()).epoch_;
});
return true;
};
// TODO: under lock
return std::visit(utils::Overloaded{main_handler, replica_handler}, repl_state_.ReplicationData());
}
bool ReplicationHandler::SetReplicationRoleReplica(const memgraph::replication::ReplicationServerConfig &config) {
// We don't want to restart the server if we're already a REPLICA
if (repl_state_.IsReplica()) {
return false;
}
// Remove registered replicas
dbms_handler_.ForEach([&](Database *db) {
auto *storage = db->storage();
storage->repl_storage_state_.replication_clients_.WithLock([](auto &clients) { clients.clear(); });
});
// Creates the server
repl_state_.SetReplicationRoleReplica(config);
// Start
const auto success =
std::visit(utils::Overloaded{[](auto) {
// ASSERT
return false;
},
[this](RoleReplicaData const &data) {
// Register handlers
InMemoryReplicationHandlers::Register(&dbms_handler_, *data.server);
if (!data.server->Start()) {
spdlog::error("Unable to start the replication server.");
return false;
}
return true;
}},
repl_state_.ReplicationData());
// TODO Handle error (restore to main?)
return success;
}
auto ReplicationHandler::RegisterReplica(const memgraph::replication::ReplicationClientConfig &config)
-> memgraph::utils::BasicResult<RegisterReplicaError> {
MG_ASSERT(repl_state_.IsMain(), "Only main instance can register a replica!");
auto res = repl_state_.RegisterReplica(config);
switch (res) {
case memgraph::replication::RegisterReplicaError::NOT_MAIN:
MG_ASSERT(false, "Only main instance can register a replica!");
return {};
case memgraph::replication::RegisterReplicaError::NAME_EXISTS:
return memgraph::dbms::RegisterReplicaError::NAME_EXISTS;
case memgraph::replication::RegisterReplicaError::END_POINT_EXISTS:
return memgraph::dbms::RegisterReplicaError::END_POINT_EXISTS;
case memgraph::replication::RegisterReplicaError::COULD_NOT_BE_PERSISTED:
return memgraph::dbms::RegisterReplicaError::COULD_NOT_BE_PERSISTED;
case memgraph::replication::RegisterReplicaError::SUCCESS:
break;
}
bool all_clients_good = true;
if (!allow_mt_repl && dbms_handler_.All().size() > 1) {
spdlog::warn("Multi-tenant replication is currently not supported!");
}
dbms_handler_.ForEach([&](Database *db) {
auto *storage = db->storage();
if (!allow_mt_repl && storage->id() != kDefaultDB) {
return;
}
// TODO: ATM only IN_MEMORY_TRANSACTIONAL, fix other modes
if (storage->storage_mode_ != storage::StorageMode::IN_MEMORY_TRANSACTIONAL) return;
all_clients_good &=
storage->repl_storage_state_.replication_clients_.WithLock([storage, &config](auto &clients) -> bool {
auto client = storage->CreateReplicationClient(config, &storage->repl_storage_state_.epoch_);
client->Start();
if (client->State() == storage::replication::ReplicaState::INVALID) {
return false;
}
clients.push_back(std::move(client));
return true;
});
});
if (!all_clients_good) return RegisterReplicaError::CONNECTION_FAILED; // TODO: this happen to 1 or many...what to do
return {};
}
auto ReplicationHandler::UnregisterReplica(std::string_view name) -> UnregisterReplicaResult {
auto const replica_handler = [](RoleReplicaData const &) -> UnregisterReplicaResult {
return UnregisterReplicaResult::NOT_MAIN;
};
auto const main_handler = [this, name](RoleMainData &mainData) -> UnregisterReplicaResult {
if (!repl_state_.TryPersistUnregisterReplica(name)) {
return UnregisterReplicaResult::COULD_NOT_BE_PERSISTED;
}
auto const n_unregistered =
std::erase_if(mainData.registered_replicas_,
[&](ReplicationClientConfig const &registered_config) { return registered_config.name == name; });
dbms_handler_.ForEach([&](Database *db) {
db->storage()->repl_storage_state_.replication_clients_.WithLock(
[&](auto &clients) { std::erase_if(clients, [&](const auto &client) { return client->Name() == name; }); });
});
return n_unregistered != 0 ? UnregisterReplicaResult::SUCCESS : UnregisterReplicaResult::CAN_NOT_UNREGISTER;
};
return std::visit(utils::Overloaded{main_handler, replica_handler}, repl_state_.ReplicationData());
}
auto ReplicationHandler::GetRole() const -> memgraph::replication::ReplicationRole { return repl_state_.GetRole(); }
bool ReplicationHandler::IsMain() const { return repl_state_.IsMain(); }
bool ReplicationHandler::IsReplica() const { return repl_state_.IsReplica(); }
void RestoreReplication(const replication::ReplicationState &repl_state, storage::Storage &storage) {
spdlog::info("Restoring replication role.");
/// MAIN
auto const recover_main = [&storage](RoleMainData const &mainData) {
for (const auto &config : mainData.registered_replicas_) {
spdlog::info("Replica {} restoration started for {}.", config.name, storage.id());
auto register_replica = [&storage](const memgraph::replication::ReplicationClientConfig &config)
-> memgraph::utils::BasicResult<RegisterReplicaError> {
return storage.repl_storage_state_.replication_clients_.WithLock(
[&storage, &config](auto &clients) -> utils::BasicResult<RegisterReplicaError> {
auto client = storage.CreateReplicationClient(config, &storage.repl_storage_state_.epoch_);
client->Start();
if (client->State() == storage::replication::ReplicaState::INVALID) {
spdlog::warn("Connection failed when registering replica {}. Replica will still be registered.",
client->Name());
}
clients.push_back(std::move(client));
return {};
});
};
auto ret = register_replica(config);
if (ret.HasError()) {
MG_ASSERT(RegisterReplicaError::CONNECTION_FAILED != ret.GetError());
LOG_FATAL("Failure when restoring replica {}: {}.", config.name, RegisterReplicaErrorToString(ret.GetError()));
}
spdlog::info("Replica {} restored for {}.", config.name, storage.id());
}
spdlog::info("Replication role restored to MAIN.");
};
/// REPLICA
auto const recover_replica = [](RoleReplicaData const &data) { /*nothing to do*/ };
std::visit(
utils::Overloaded{
recover_main,
recover_replica,
},
std::as_const(repl_state).ReplicationData());
}
} // namespace memgraph::dbms

View File

@@ -0,0 +1,69 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "replication/role.hpp"
#include "storage/v2/storage.hpp"
#include "utils/result.hpp"
// BEGIN fwd declares
namespace memgraph::replication {
struct ReplicationState;
struct ReplicationServerConfig;
struct ReplicationClientConfig;
} // namespace memgraph::replication
namespace memgraph::dbms {
class DbmsHandler;
enum class RegisterReplicaError : uint8_t { NAME_EXISTS, END_POINT_EXISTS, CONNECTION_FAILED, COULD_NOT_BE_PERSISTED };
enum class UnregisterReplicaResult : uint8_t {
NOT_MAIN,
COULD_NOT_BE_PERSISTED,
CAN_NOT_UNREGISTER,
SUCCESS,
};
/// A handler type that keep in sync current ReplicationState and the MAIN/REPLICA-ness of Storage
/// TODO: extend to do multiple storages
struct ReplicationHandler {
ReplicationHandler(memgraph::replication::ReplicationState &replState, DbmsHandler &dbms_handler)
: repl_state_(replState), dbms_handler_(dbms_handler) {}
// as REPLICA, become MAIN
bool SetReplicationRoleMain();
// as MAIN, become REPLICA
bool SetReplicationRoleReplica(const memgraph::replication::ReplicationServerConfig &config);
// as MAIN, define and connect to REPLICAs
auto RegisterReplica(const memgraph::replication::ReplicationClientConfig &config)
-> utils::BasicResult<RegisterReplicaError>;
// as MAIN, remove a REPLICA connection
auto UnregisterReplica(std::string_view name) -> UnregisterReplicaResult;
// Helper pass-through (TODO: remove)
auto GetRole() const -> memgraph::replication::ReplicationRole;
bool IsMain() const;
bool IsReplica() const;
private:
memgraph::replication::ReplicationState &repl_state_;
DbmsHandler &dbms_handler_;
};
/// A handler type that keep in sync current ReplicationState and the MAIN/REPLICA-ness of Storage
/// TODO: extend to do multiple storages
void RestoreReplication(const replication::ReplicationState &repl_state, storage::Storage &storage);
} // namespace memgraph::dbms

View File

@@ -1,61 +0,0 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "auth/auth.hpp"
#include "query/interpreter.hpp"
#include "storage/v2/storage.hpp"
#include "utils/synchronized.hpp"
#if MG_ENTERPRISE
#include "audit/log.hpp"
#endif
namespace memgraph::dbms {
/**
* @brief Structure encapsulating storage and interpreter context.
*
* @note Each session contains a copy.
*/
struct SessionContext {
// Explicit constructor here to ensure that pointers to all objects are
// supplied.
SessionContext(std::shared_ptr<memgraph::query::InterpreterContext> interpreter_context, std::string run,
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth
#ifdef MG_ENTERPRISE
,
memgraph::audit::Log *audit_log
#endif
)
: interpreter_context(interpreter_context),
run_id(run),
auth(auth)
#ifdef MG_ENTERPRISE
,
audit_log(audit_log)
#endif
{
}
std::shared_ptr<memgraph::query::InterpreterContext> interpreter_context;
std::string run_id;
// std::shared_ptr<AuthContext> auth_context;
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth;
#ifdef MG_ENTERPRISE
memgraph::audit::Log *audit_log;
#endif
};
} // namespace memgraph::dbms

Some files were not shown because too many files have changed in this diff Show More