Compare commits

...

434 Commits

Author SHA1 Message Date
Tyler Neely
204d914e8b Use cache v3 api 2022-11-29 16:45:26 +00:00
Tyler Neely
0e76a8ca01 Merge branch 'project-pineapples' into tyler_caching_experiment 2022-11-29 17:25:14 +01:00
Tyler Neely
53040c6758 Merge pull request #651 from memgraph/T1157-MG-concurrent-RsmClient-requests
Support concurrent RsmClient requests
2022-11-29 17:11:25 +01:00
Tyler Neely
8e0e6dbd86 Enable caching for libs directory 2022-11-29 16:07:55 +00:00
Tyler Neely
04124a1e9b Make AsyncRequestToken arguments const, reserve size in response vectors 2022-11-29 15:50:35 +00:00
Tyler Neely
b3605c9ab1 Fix typo in new simplified RequestRouter::CreateExpand method 2022-11-29 15:11:35 +00:00
Tyler Neely
7df3a743b9 Simplify and parallelize CreateExpand 2022-11-29 15:07:59 +00:00
Tyler Neely
f8215306e8 A variety of small code clean-ups, remove overloaded methods 2022-11-29 15:06:01 +00:00
Tyler Neely
8f08d986cb Make method names clear for RequestRouter requests, avoid unnecessary overloading 2022-11-29 14:47:32 +00:00
Tyler Neely
9144d2dccd Remove bug-prone inverted ownership of ExecutionState as a consideration of operators 2022-11-29 14:30:59 +00:00
Tyler Neely
ec529da8d2 Address clang-tidy feedback 2022-11-29 12:28:19 +00:00
Tyler Neely
1b77e029ca Use the Async methods under the hood for the blocking RsmClient::Send*Request methods for code-reuse 2022-11-29 11:33:49 +00:00
Tyler Neely
a308ee501a Unify the driving of read and write requests in the RequestRouter 2022-11-29 11:25:29 +00:00
Tyler Neely
aa7d362296 Merge branch 'project-pineapples' of github.com:memgraph/memgraph into T1157-MG-concurrent-RsmClient-requests 2022-11-29 10:13:42 +00:00
Tyler Neely
7d52eedb21 Merge pull request #674 from memgraph/tyler_rename_ShardRequestManager_to_RequestRouter
[project-pineapples <- ] Rename ShardRequestManager to RequestRouter
2022-11-29 10:29:42 +01:00
Tyler Neely
4f18fa7431 Fix LCP that broke with invisible merge conflict 2022-11-29 09:07:18 +00:00
Tyler Neely
9f10c3ea06 Merge branch 'project-pineapples' of github.com:memgraph/memgraph into tyler_rename_ShardRequestManager_to_RequestRouter 2022-11-29 08:57:28 +00:00
Kostas Kyrimis
4dc639a05a Merge pull request #644 from memgraph/T1159-MG-Add-memgraph-functions
Add mg-functions lib, expression evaluator units and e2e tests
2022-11-28 18:18:58 +02:00
Kostas Kyrimis
50df0d4d53 Fix python imports and replace const char array with constexpr 2022-11-28 17:31:41 +02:00
Kostas Kyrimis
8fd7327fbd Address GH comments 2022-11-28 15:38:12 +02:00
Tyler Neely
9fc7f9dced Standardize RequestRouter variable names as request_router 2022-11-28 13:03:07 +00:00
Tyler Neely
82db1d4ad8 Rename ShardRequestManager to RequestRouter 2022-11-28 12:38:38 +00:00
Kostas Kyrimis
ed0b67dfdb Fix compilation issues of the merge 2022-11-28 13:38:12 +02:00
Kostas Kyrimis
0f34c49e21 Merge branch 'project-pineapples' into T1159-MG-Add-memgraph-functions 2022-11-28 13:18:51 +02:00
Tyler Neely
f4d0c7769e Merge pull request #670 from memgraph/tyler_set_proper_namespace_for_ShardRequestManager
Change the namespace of ShardRequestManager to query::v2 instead of msgs
2022-11-28 11:31:09 +01:00
Tyler Neely
6d3c04bd61 Address clang-tidy feedback 2022-11-28 10:09:59 +00:00
Tyler Neely
a8dc6fd41e Change the namespace of ShardRequestManager to query::v2 instead of msgs 2022-11-28 09:43:56 +00:00
Tyler Neely
de84d4d6ea Merge branch 'project-pineapples' of github.com:memgraph/memgraph into T1157-MG-concurrent-RsmClient-requests 2022-11-28 08:53:21 +00:00
Kostas Kyrimis
6f4996de0e Fix broken merge and address GH comments 2022-11-25 16:20:38 +02:00
Kostas Kyrimis
01d5953bb6 Merge branch 'project-pineapples' into T1159-MG-Add-memgraph-functions 2022-11-25 15:47:44 +02:00
János Benjamin Antal
6b64fd5ce5 Merge pull request #664 from memgraph/MG-implicit-fallthrough-as-error
Make implicit-fallthrough a compilation error
2022-11-24 19:32:59 +01:00
János Benjamin Antal
3b798ab313 Merge branch 'project-pineapples' into MG-implicit-fallthrough-as-error 2022-11-24 11:16:57 +01:00
János Benjamin Antal
d44c1836c7 Merge pull request #649 from memgraph/T0995-MG-implement-top-error-handling-storage
Implement top level error handling in storage
2022-11-24 11:14:59 +01:00
János Benjamin Antal
c4327cfb00 Make implicit-fallthrough a compilation error 2022-11-24 11:13:55 +01:00
jbajic
d820d0a9e5 Fix clang tidy errors 2022-11-23 22:29:03 +01:00
jbajic
7aa68164f0 Merge branch 'project-pineapples' into T0995-MG-implement-top-error-handling-storage 2022-11-23 21:55:08 +01:00
jbajic
d6b444c38b Log transaction id 2022-11-23 18:02:00 +01:00
János Benjamin Antal
c647134916 Merge pull request #611 from memgraph/T1083-MG-limit-and-order-expand-one_v3
Add Limit and OrderBy to ExpandOne
2022-11-23 17:37:32 +01:00
jbajic
b3eec92525 Merge branch 'project-pineapples' into T0995-MG-implement-top-error-handling-storage 2022-11-23 16:11:58 +01:00
jbajic
ab5fc05fd7 Address review comments 2022-11-23 15:32:28 +01:00
jeremy
e77843f2ec Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-11-23 14:47:29 +01:00
János Benjamin Antal
40e145e7d0 Merge pull request #646 from memgraph/T1116-MG-update-delete-vertex-labels
Update/Delete vertex labels
2022-11-23 14:35:11 +01:00
Kostas Kyrimis
2ff81ebf04 Address missed GH comments && fix broken merge 2022-11-23 15:16:14 +02:00
jeremy
c3c68cc2ce Merge branch 'T1083-MG-limit-and-order-expand-one_v3' of github.com:memgraph/memgraph into T1083-MG-limit-and-order-expand-one_v3 2022-11-23 13:55:02 +01:00
jeremy
6d86801be0 Extract logic to convert ORderingDirection to Ordering 2022-11-23 13:54:54 +01:00
Kostas Kyrimis
407418e8f5 Merge branch 'project-pineapples' into T1159-MG-Add-memgraph-functions 2022-11-23 14:53:53 +02:00
Kostas Kyrimis
6d544e4fc0 Address GH comments 2022-11-23 14:51:38 +02:00
Jeremy B
56e2ad4546 Update src/storage/v3/shard_rsm.cpp
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
2022-11-23 13:24:35 +01:00
jbajic
9b19dd57d3 Remove storage include 2022-11-23 13:19:25 +01:00
Tyler Neely
5045323b1d Merge branch 'project-pineapples' of github.com:memgraph/memgraph into T1157-MG-concurrent-RsmClient-requests 2022-11-23 11:31:14 +00:00
Jeremy B
aa146d28f8 Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-11-23 12:26:58 +01:00
János Benjamin Antal
1cd10ab409 Merge branch 'project-pineapples' into T1116-MG-update-delete-vertex-labels 2022-11-23 12:15:34 +01:00
János Benjamin Antal
36390608aa Merge pull request #648 from memgraph/MG-remove-unused-storage-files
Remove `storage.hpp` and `storage.cpp` files
2022-11-23 12:15:09 +01:00
Jure Bajic
e565fc6e3a Merge branch 'project-pineapples' into T0995-MG-implement-top-error-handling-storage 2022-11-23 02:33:07 -08:00
jbajic
ce0f1a09f7 Remove unused methods from accessors 2022-11-23 11:32:53 +01:00
jbajic
accf015dcf Fix clang-tidy errors 2022-11-23 11:03:17 +01:00
Jeremy B
84800cff76 Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-11-23 10:57:05 +01:00
jbajic
9fade5ebac Fix tests 2022-11-23 10:37:33 +01:00
Jure Bajic
75606dfeb0 Merge branch 'project-pineapples' into T1116-MG-update-delete-vertex-labels 2022-11-23 00:34:27 -08:00
János Benjamin Antal
9c36d8928a Merge branch 'project-pineapples' into MG-remove-unused-storage-files 2022-11-23 09:19:14 +01:00
János Benjamin Antal
d926c4c4a4 Merge pull request #657 from memgraph/tyler_simulator_determinism_test
Improve simulator determinism
2022-11-23 09:14:25 +01:00
jeremy
85034ddcbe Rename variable un function def 2022-11-22 17:29:02 +01:00
Tyler Neely
ea533f43fc Print out the simulator seed when we exceed the configured abort_time 2022-11-22 16:06:57 +00:00
Tyler Neely
e0086e5666 Use spdlog::trace instead of info for simulator-related messages 2022-11-22 16:06:35 +00:00
Tyler Neely
c0a103e851 Do not advance the clock with every message, as this prevents messages of a certain request depth from ever completing 2022-11-22 16:00:06 +00:00
jeremy
3a171376d7 OrderBy in Expand has two members to differ vertices Vs edges 2022-11-22 16:47:25 +01:00
jbajic
cd7b33f23f use comparison between ErrorCode and ShardError 2022-11-22 16:22:42 +01:00
jbajic
d82cfb349e Use ShardResult 2022-11-22 15:00:32 +01:00
jeremy
6801d6ff09 Remove duplicate using statement 2022-11-22 14:49:54 +01:00
jbajic
f1e360469a Expand SHARD_ERROR with fmt format 2022-11-22 14:44:28 +01:00
jeremy
307cce9e21 Remove unused struct 2022-11-22 14:23:24 +01:00
jeremy
662fa2e6d2 Remove uneeded using statement 2022-11-22 14:22:19 +01:00
jeremy
c0cb53e156 Replace if by switch 2022-11-22 14:20:22 +01:00
jeremy
742017548f Merge branch 'T1083-MG-limit-and-order-expand-one_v3' of github.com:memgraph/memgraph into T1083-MG-limit-and-order-expand-one_v3 2022-11-22 14:11:48 +01:00
Jeremy B
bbbd722eeb Update src/storage/v3/request_helper.hpp
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-11-22 14:11:29 +01:00
jeremy
f463d9f59f Merge branch 'T1083-MG-limit-and-order-expand-one_v3' of github.com:memgraph/memgraph into T1083-MG-limit-and-order-expand-one_v3 2022-11-22 14:10:07 +01:00
Jeremy B
a6f3937692 Update src/storage/v3/request_helper.hpp
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-11-22 14:10:02 +01:00
jbajic
5717dfb165 Add ErrorCode to SHARD_ERROR macro 2022-11-22 14:05:04 +01:00
jeremy
1101c2444c Make ConvertPropertyMap expect ref and not rvalue 2022-11-22 14:01:16 +01:00
jbajic
2a6dc7bb93 Remove success 2022-11-22 13:49:12 +01:00
jbajic
d080e260e6 Remove redundant code 2022-11-22 13:27:02 +01:00
jbajic
37f5fb29ea Introduce BoltResult 2022-11-22 13:15:32 +01:00
Tyler Neely
c8c72de6ac Use duration_cast to ensure that we are retrieving milliseconds inside SimulatorStats::operator<< 2022-11-22 11:30:24 +00:00
Tyler Neely
0b19b62b12 Set the abort_time for raft tests to 1 simulated hour 2022-11-22 11:25:24 +00:00
jbajic
bffef1a653 Use experimental source location 2022-11-22 11:13:23 +01:00
Tyler Neely
66f39f2681 Add elapsed time to the SimulatorStats 2022-11-22 08:55:48 +00:00
Tyler Neely
081c3e5bed Capitalize unsigned integer literal 2022-11-21 13:16:35 +00:00
jeremy
ce8bc522d0 Clang warning 2022-11-21 12:23:04 +01:00
Tyler Neely
0f66ae31dd Use explicit unsigned integer in right shift operation 2022-11-21 11:11:39 +00:00
jeremy
e9e42a0614 add * token to variable declaration 2022-11-21 11:50:56 +01:00
jeremy
1a67dec302 Update test to use OrderBy and Limit on Expand 2022-11-21 11:45:35 +01:00
jeremy
4eb673c7b9 Add const to variable 2022-11-21 11:31:22 +01:00
Tyler Neely
71dcba331e Increment simulator time by up to 30ms in ticks 2022-11-21 10:10:45 +00:00
Tyler Neely
e43f4e2181 Sort simulator in_flight_ messages based on a stable sort of the sender address 2022-11-21 10:08:42 +00:00
jeremy
b2050d55ce Add const 2022-11-21 10:59:38 +01:00
jeremy
2fc1aeb087 Remove unneeded using statements 2022-11-21 10:59:16 +01:00
jeremy
86e5b44c1c Remove Shard::Accessor (unsued) 2022-11-21 10:42:15 +01:00
jeremy
1b0db5289d OrderByVertices only keeps OrderBy expression which corresponds to Vertices 2022-11-21 10:41:08 +01:00
Tyler Neely
45badbe21f Use unsigned integer literals for bit shifting in the NewShardUuid function 2022-11-18 17:22:50 +00:00
Tyler Neely
04420a84c7 Fix incorrect usage of IncrementServerCountAndWaitForQuiescentState in the shard_rsm.cpp simulation test 2022-11-18 16:54:38 +00:00
Tyler Neely
ce45a548c7 Significantly improve the determinism of the coordinator, UUID generation, the machine manager, the shard manager, and the cluster property test 2022-11-18 16:42:18 +00:00
Tyler Neely
3ad8489735 Run raft sim with random seeds over time, but allow a seed to be easily replayed using the RunWithSeed function 2022-11-18 10:34:21 +00:00
Tyler Neely
7115a7e75b Apply clang-tidy fixes 2022-11-18 10:24:19 +00:00
Tyler Neely
0f32407bdc Add compare header to histogram collector header 2022-11-18 10:20:45 +00:00
Tyler Neely
a37e7e4aff Add assert to ensure TestAddress will not be higher than the uchar max 2022-11-18 10:19:55 +00:00
Tyler Neely
6b9a617df0 Streamline simulator tick condition varible notification. Advance time more aggressively 2022-11-18 09:20:15 +00:00
Tyler Neely
923325b8fa Progress the simulator clock even when there are messages to deliver 2022-11-18 09:04:29 +00:00
Tyler Neely
f6017697d6 Make raft tests fully deterministic for rng_seeds between 0 and 500 at 1% message loss 2022-11-17 21:32:55 +00:00
Tyler Neely
9c3d683942 Explicitly join test threads before collecting test stats 2022-11-17 21:28:17 +00:00
Tyler Neely
cf73ed529d Block messages from being delivered upon ShutDown 2022-11-17 21:27:48 +00:00
Tyler Neely
262df5c6a2 Avoid unordered_map in Raft code for more determinism 2022-11-17 21:24:13 +00:00
Tyler Neely
098084314e Make TestAddress deterministically sortable 2022-11-17 21:22:41 +00:00
Tyler Neely
12880fc71a Don't advance the simulator handle from server threads themselves 2022-11-17 18:27:12 +00:00
Tyler Neely
80d6776210 Improve simulator determinism 2022-11-17 17:36:46 +00:00
jeremy
3840c14846 Remove nocommit comment 2022-11-17 14:33:08 +01:00
jeremy
fe03f5b206 Update include to full path
add auto To variable declaration
2022-11-17 14:11:25 +01:00
jeremy
5f88e75571 Remove double declaration 2022-11-17 14:10:49 +01:00
jeremy
27495ef43a Merge branch 'T1083-MG-limit-and-order-expand-one_v3' of github.com:memgraph/memgraph into T1083-MG-limit-and-order-expand-one_v3 2022-11-17 13:34:08 +01:00
jeremy
a499bf6dfd Rename variable 2022-11-17 13:33:11 +01:00
Jeremy B
49652d0a61 Update src/storage/v3/shard_rsm.cpp
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
2022-11-17 13:32:48 +01:00
jeremy
6e44c2295d Remove template from OrderByEdges + move to impl file 2022-11-17 13:30:39 +01:00
jeremy
d482f7da90 Merge branch 'T1083-MG-limit-and-order-expand-one_v3' of github.com:memgraph/memgraph into T1083-MG-limit-and-order-expand-one_v3 2022-11-17 12:59:37 +01:00
Jeremy B
77ab07d991 Update src/storage/v3/request_helper.hpp
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
2022-11-17 12:58:39 +01:00
jeremy
38b0b308ce Remove unnecessary reserve 2022-11-17 12:58:28 +01:00
jeremy
2f55491271 use std::SameAsAnyOf i.o. is_sale_v 2022-11-17 12:45:15 +01:00
Jeremy B
68e51e73ba Update src/storage/v3/shard_rsm.cpp
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
2022-11-17 12:42:36 +01:00
jeremy
a17a6aea5a rename variable vertice->vertex 2022-11-17 12:27:12 +01:00
jbajic
ec4804b72a Move ErrorCode to common 2022-11-17 11:03:03 +01:00
jbajic
e98ef634de Ignore warning 2022-11-16 21:03:36 +01:00
jeremy
c4e22ffde3 Remove unnecessary tag 2022-11-16 18:51:57 +01:00
jeremy
b3ef0ccd71 Moving function from shard_rsm to helper files 2022-11-16 18:50:22 +01:00
jbajic
15fc3c0834 Fix error 2022-11-16 18:12:25 +01:00
jbajic
9261fabe60 Adapt for unfound object 2022-11-16 17:41:43 +01:00
jbajic
7bdcd8f9f4 Add shard_error in response 2022-11-16 14:48:06 +01:00
jeremy
bd11225d23 Use ref instead of optional
Use ref i.o. ptr
Rename variable for clarity
2022-11-16 14:14:35 +01:00
jbajic
1f159b5a80 Fix scheme test 2022-11-16 08:58:06 +01:00
jbajic
14ddd7254d Fix communication and memgraph 2022-11-15 19:37:24 +01:00
jbajic
8629ee5ebc Fix query v2 2022-11-15 19:30:34 +01:00
jbajic
1c94c59a24 Fix tests 2022-11-15 19:29:56 +01:00
Tyler Neely
631d18465b Allow the RsmClient to store multiple in-flight requests. Update the ShardRequestManager to use the new request tokens and refactor some bug-prone aspects of it 2022-11-15 17:52:38 +00:00
jbajic
3f97a13493 Revert TError to Error 2022-11-15 18:30:25 +01:00
jbajic
5656a24c96 Remove unused exception 2022-11-15 18:25:25 +01:00
jbajic
9c05910e68 Adapt storage to new erro handling 2022-11-15 18:25:12 +01:00
jbajic
3d66bbd988 Remove shard_operation_result 2022-11-15 18:24:40 +01:00
jbajic
07032887a4 Adapt mg-expr 2022-11-15 18:24:28 +01:00
Kostas Kyrimis
9e81fe791c Fix clang-tidy warnings, remove commented out code and add HasLabel
functions in ShardRequestManager to avoid throwing bad optional on
expression evaluator
2022-11-15 19:20:01 +02:00
jeremy
cca4e97bcf Remove un-needed argument from OrderByEdges 2022-11-15 13:37:43 +01:00
Kostas Kyrimis
213583f916 Add expression evaluator && awesome_memgraph_functions unit tests 2022-11-14 20:46:20 +02:00
Kostas Kyrimis
94bc671551 Add awesome memgraph functions e2e tests 2022-11-14 20:46:19 +02:00
Kostas Kyrimis
02ef954e51 Add mg-functions lib and integrated with SE and QE 2022-11-14 20:46:16 +02:00
jeremy
131d7f2a74 OrderByElements: no longer templated over vertice/edge types. For edges, we always need to have access to the corresponding vertex_accessor (ex of sorting expr needing both : "vertex.map[edge]")
ComputeExpression: made assert instead of if check
2022-11-14 18:21:03 +01:00
jbajic
618237cc96 Adapt schema validator 2022-11-14 14:25:20 +01:00
jbajic
ef755e466c Add ShardError 2022-11-14 14:15:29 +01:00
Jure Bajic
a9c5d40721 Update src/utils/template_utils.hpp
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
2022-11-14 10:32:18 +01:00
jbajic
b4c24f4506 Remvoe storage.hpp and storage.cpp files 2022-11-14 10:23:48 +01:00
jbajic
0462b8fc8f Address review comments 2022-11-14 09:15:03 +01:00
jbajic
6eabceca4a Add unit tests for UpdateVertex 2022-11-14 09:02:55 +01:00
jeremy
2045f54577 Correct merge issue 2022-11-09 21:36:41 +01:00
Jeremy B
18009c06b6 Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-11-09 16:47:06 +01:00
Jure Bajic
db7b2aa59a Merge branch 'project-pineapples' into T1116-MG-update-delete-vertex-labels 2022-11-09 07:24:26 -08:00
János Benjamin Antal
5c0e41ed44 Merge pull request #593 from memgraph/T1079-MG-add-simple-query-to-benchmark_v2
Add new dataset for mgbench
2022-11-09 16:21:56 +01:00
Jure Bajic
12e9c1a739 Merge branch 'project-pineapples' into T1116-MG-update-delete-vertex-labels 2022-11-09 07:08:04 -08:00
jbajic
a030419565 Move LogResultError to helper 2022-11-09 16:07:27 +01:00
jeremy
6df2db0d19 Remove comment force github workflow 2022-11-09 16:02:59 +01:00
jeremy
968584a8fc Add comment force github workflow 2022-11-09 16:02:25 +01:00
jbajic
8636788ab2 Fix typos 2022-11-09 16:00:25 +01:00
jbajic
691f6af36d Remove redundant code 2022-11-09 15:52:08 +01:00
jbajic
23f1536eac Add tests 2022-11-09 15:42:09 +01:00
jeremy
33add3ecd0 Force formatting 2022-11-09 15:38:51 +01:00
János Benjamin Antal
91b5092c71 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-09 13:05:30 +01:00
János Benjamin Antal
c0f576c187 Merge pull request #645 from memgraph/MG-add-python-checks-pineapples
Add python checks
2022-11-09 13:05:15 +01:00
jbajic
a2735c8953 Add label add/remove in UpdateVertex req 2022-11-09 12:10:46 +01:00
jbajic
51fb4a6e7a Remove empty line 2022-11-09 11:32:54 +01:00
jbajic
cba183898a Update actions versions 2022-11-09 10:40:46 +01:00
jbajic
5f5fe7eb29 Add python checks 2022-11-09 10:39:29 +01:00
jeremy
2087877df2 Add more checks in test 2022-11-09 08:48:51 +01:00
Jeremy B
efa4378fb4 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-09 08:26:25 +01:00
jeremy
cad0e80d00 Update test 2022-11-08 17:42:31 +01:00
János Benjamin Antal
3c07a5dc04 Merge pull request #639 from memgraph/T1145-MG-handle-insertions-of-vertices-that-already-exist-in-a-better-way
Return an error for insertions of vertices that already exist
2022-11-08 17:11:22 +01:00
Jeremy B
e442963466 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-08 17:00:30 +01:00
János Benjamin Antal
32fe4d94d9 Merge branch 'project-pineapples' into T1145-MG-handle-insertions-of-vertices-that-already-exist-in-a-better-way 2022-11-08 16:35:38 +01:00
János Benjamin Antal
94b66a4e81 Merge pull request #638 from memgraph/T1148-MG-fix-expand-one-source-vertex
Fix setting the primary key of source vertex in ExpandOne
2022-11-08 16:35:30 +01:00
jeremy
b2f3fab693 Remove comment 2022-11-08 15:06:27 +01:00
jeremy
d7ce7cea13 Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-11-08 15:02:56 +01:00
jeremy
61b9457718 Remove split-files logic from test code 2022-11-08 11:35:54 +01:00
János Benjamin Antal
159b30ba5f Merge branch 'project-pineapples' into T1148-MG-fix-expand-one-source-vertex 2022-11-08 08:20:25 +01:00
gvolfing
b67e5b9a6c Merge branch 'project-pineapples' into T1145-MG-handle-insertions-of-vertices-that-already-exist-in-a-better-way 2022-11-08 07:53:45 +01:00
gvolfing
79756ae6fb Modify unit test
Instead of Creating an exception that is would be only used in this
file, just assert the type of the error the double vertex insertion
operation should yield.
2022-11-08 07:31:01 +01:00
jeremy
a54bcb9819 Remove un-necessary rm in workflow 2022-11-07 17:32:09 +01:00
jeremy
c16f948de9 Delete cache folder before running benchmark test 2022-11-07 13:08:35 +01:00
jeremy
2a7ed1ad82 Add single e2e benchmark test 2022-11-07 12:55:08 +01:00
jeremy
5201db46d2 Add assert for split_file 2022-11-07 12:15:57 +01:00
gvolfing
91550128a5 Conform unit test with the new error-handling 2022-11-07 11:46:24 +01:00
jeremy
d5966101ad Merge branch 'T1079-MG-add-simple-query-to-benchmark_v2' of github.com:memgraph/memgraph into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-07 10:55:48 +01:00
jbajic
88487e2513 Extract vertex creation from FillUpSourceVertexSecondaryLabels 2022-11-07 10:54:28 +01:00
jeremy
b10b1eb239 Correct shard configuration 2022-11-07 10:33:24 +01:00
jeremy
dca94f42bb Update key type in shard configuration 2022-11-07 10:14:49 +01:00
gvolfing
39b40ecf00 Extend the Error enum instead of a separate type
The error representing that a vertex is already inserted into the
skip-list was represented by the struct AlreadyInseertedElement. Instead
of using that struct, extend the memgraph::storage::v3::Error scoped
enum and use that to represent the double-insertion error.
2022-11-07 10:00:34 +01:00
jeremy
baacc52a65 Add support for split file configuration 2022-11-07 09:54:14 +01:00
jeremy
5273d319e2 Add split file for access control 2022-11-07 09:53:29 +01:00
Jeremy B
23838b50ea Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-05 14:46:33 +01:00
Tyler Neely
d85fb94bc7 Merge pull request #634 from memgraph/T1122-MG-ShardManager-ThreadPool
Add shard manager thread pool
2022-11-04 16:41:53 +01:00
jbajic
4d2036249e Fix e2e test 2022-11-04 16:37:35 +01:00
Jure Bajic
d23643d4ff Merge branch 'project-pineapples' into T1148-MG-fix-expand-one-source-vertex 2022-11-04 16:19:55 +01:00
jbajic
e52ce1e363 Add expandone test 2022-11-04 16:18:18 +01:00
Tyler Neely
1abfe28806 Correctly use a moved transport interface while constructing CoordinatorWorker 2022-11-04 15:11:32 +00:00
jeremy
d17970f6d9 Update default value for --datasets 2022-11-04 16:04:45 +01:00
jeremy
9e72c7cb54 Add extra safe check to in case we call on dataset.py 2022-11-04 15:57:26 +01:00
Tyler Neely
528e30a9be Avoid warning for not using captured this. Use std::move instead of forward where appropriate 2022-11-04 14:13:15 +00:00
Tyler Neely
bb7c7f7627 Make popped messages non-const to allow for RVO 2022-11-04 14:07:49 +00:00
jbajic
bab5e1386a Address review comments 2022-11-04 15:05:55 +01:00
gvolfing
3d954e7abc Restructure SchemaResult type and uts usage
Rename SchemaResult to ShardOperationResult move it into a separate
header and add a new type to the underlying variant that indicates that
the vertex, the user would like to insert into the skip-list already
exist.
2022-11-04 15:04:25 +01:00
Tyler Neely
fa5c9a2568 Make items popped from *worker::Queue const. Use std::move instead of std::forward in one place 2022-11-04 11:33:02 +00:00
Tyler Neely
c745f8c877 Fix build after breaking code suggestion 2022-11-04 11:32:17 +00:00
Tyler Neely
486231b1b9 Update src/machine_manager/machine_manager.hpp
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2022-11-04 12:28:01 +01:00
Tyler Neely
43ad5855c4 Update src/machine_manager/machine_manager.hpp
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2022-11-04 12:27:40 +01:00
Tyler Neely
24864ff7d2 Update src/coordinator/coordinator_worker.hpp
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2022-11-04 12:17:57 +01:00
Tyler Neely
8598f6edf4 Fix a race condition that happens when logging from a detached thread in the cluster property test. Improve the ShardManager dtor and log statements 2022-11-04 11:14:39 +00:00
jeremy
e41073bc2c Update script to need single argument for local dataset 2022-11-04 09:17:09 +01:00
jeremy
14e3e72565 Correct badly written range 2022-11-04 08:52:47 +01:00
gvolfing
1cee7ecb8a Make ShardRsm aware of trying to write the same vertex into the skip-list 2022-11-04 08:12:37 +01:00
Jeremy B
690a389563 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-03 17:43:47 +01:00
Tyler Neely
9203616283 Merge branch 'project-pineapples' into T1122-MG-ShardManager-ThreadPool 2022-11-03 17:01:07 +01:00
Tyler Neely
d198819bf1 Merge pull request #637 from memgraph/tyler_remove_storage_v2_durability_test
Remove unit tests for storage v2
2022-11-03 17:00:46 +01:00
jbajic
3a8f01af79 Fix expand one source vertex pk setting 2022-11-03 16:29:28 +01:00
Tyler Neely
7e6ec8bb26 Capture this instead of all references in scope for call to std::visit to route CoordinatorWorker messages 2022-11-03 14:38:16 +00:00
Tyler Neely
6138d76690 Remove unit tests for storage v2 2022-11-03 14:21:17 +00:00
Tyler Neely
25fdb1a1f0 Make the high_density_shard_create_scan test run much faster 2022-11-03 13:48:11 +00:00
Tyler Neely
2de1d6c359 Fix UB due to integer overflow 2022-11-03 13:27:45 +00:00
Tyler Neely
0364311dd0 Log latency histograms in the high_density_shard_create_scan test 2022-11-03 12:44:47 +00:00
Tyler Neely
9235515dab Reduce high_density_shard_create_scan shard+thread combinations. Log Raft write request demangled names 2022-11-03 12:38:12 +00:00
Tyler Neely
b83fb287ad Apply feedback from clang-tidy 2022-11-03 11:00:07 +00:00
jeremy
70dc19dfdb Mgbench: apply filtering on results from client 2022-11-03 11:03:21 +01:00
Tyler Neely
b685a21171 Merge branch 'project-pineapples' of github.com:memgraph/memgraph into T1122-MG-ShardManager-ThreadPool 2022-11-03 09:42:28 +00:00
Tyler Neely
4db83b8159 Add a few safety checks to the new concurrent Queue structures and the promise maps in the transport layer 2022-11-03 09:41:28 +00:00
Tyler Neely
dd8dd4f6c4 Restructure responsibilities for assigning request ids to the transport handles. Simplify promise tracking to avoid replier addresses, enabling eventual direct server return (DSR) 2022-11-03 09:31:06 +00:00
Tyler Neely
fa1ddfea12 Fix a bug where the MachineManager's destructor became incorrect after being moved 2022-11-02 17:45:27 +00:00
Tyler Neely
a815ec9617 Handle Coordinator work on a separate thread, unblocking the MachineManager to route additional messages to Shards 2022-11-02 17:15:52 +00:00
Tyler Neely
78528bd609 Avoid the ShutDown explicit acknowledgement due to jthread already blocking on this 2022-11-02 17:12:21 +00:00
Tyler Neely
6239f4fc3e Simplify usage of PromiseKey in LocalTransportHandle to avoid replier address 2022-11-02 17:11:26 +00:00
Jeremy B
c5eb3ff2c0 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-02 17:01:08 +01:00
János Benjamin Antal
9f71ce0f78 Merge pull request #635 from memgraph/MG-expand-fixes 2022-11-02 15:55:49 +01:00
János Benjamin Antal
589dd36bf2 Make function parameter constant 2022-11-02 14:44:54 +01:00
János Benjamin Antal
57e7169203 Eliminate dangling reference from lambda 2022-11-02 14:22:01 +01:00
János Benjamin Antal
bb3b053375 Create edge in the correct direction 2022-11-02 14:22:01 +01:00
János Benjamin Antal
a96f489756 Count the number of requests per operator 2022-11-02 14:22:01 +01:00
jeremy
e909e7d2d8 Format 2022-11-02 14:18:04 +01:00
Jeremy B
2d91a7fd1e Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-02 14:14:24 +01:00
Jeremy B
1d18f1197f Update tests/mgbench/dataset_creator.py
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-11-02 14:14:16 +01:00
Jeremy B
edeebf46ec Update tests/mgbench/benchmark.py
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-11-02 14:13:58 +01:00
Jeremy B
1148fe9aad Update tests/mgbench/datasets.py
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-11-02 14:13:40 +01:00
János Benjamin Antal
c92c795b1a Merge pull request #632 from memgraph/MG-improve-parameters 2022-11-02 14:06:30 +01:00
János Benjamin Antal
0fee412f92 Merge branch 'project-pineapples' into MG-improve-parameters 2022-11-02 13:37:35 +01:00
Tyler Neely
7596e85358 When message conversion fails to happen, demangle the concrete type name and error log a useful message 2022-11-02 12:37:34 +00:00
Jeremy B
014836ca3c Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-02 12:51:12 +01:00
Tyler Neely
68654b5a19 Merge branch 'project-pineapples' of github.com:memgraph/memgraph into T1122-MG-ShardManager-ThreadPool 2022-11-02 10:55:18 +00:00
János Benjamin Antal
4d96bf5006 Merge pull request #628 from memgraph/T1138-MG-fix-edge-id-allocator
Fix passing of edge IdAllocator not by value but as pointer
2022-11-02 10:40:35 +01:00
Marko Budiselić
9bb70eb0b9 Merge branch 'project-pineapples' into T1138-MG-fix-edge-id-allocator 2022-11-01 19:38:10 +01:00
Jeremy B
7076788dd5 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-01 18:51:57 +01:00
Tyler Neely
ee4be9aa5b Merge pull request #631 from memgraph/tyler_remove_shard_map_from_machine_manager
Remove redundant ShardMap copy from MachineManager to avoid race conditions
2022-11-01 18:33:37 +01:00
Jeremy B
96a2ad4b63 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-01 18:21:36 +01:00
Tyler Neely
84509fa477 Fix merge conflict with MachineManager constructor 2022-11-01 17:07:14 +00:00
Tyler Neely
19b5fe3caf Merge branch 'project-pineapples' into tyler_remove_shard_map_from_machine_manager 2022-11-01 17:57:03 +01:00
Tyler Neely
74db669e23 Merge pull request #622 from memgraph/tyler_1k_shard_unit_test
Add test for 1k shards, 1k create vertices, and then a scan all
2022-11-01 17:56:42 +01:00
Tyler Neely
599802033f Reformat TODO comment 2022-11-01 17:21:15 +01:00
Tyler Neely
6dd57426f8 Revert broken code suggestion 2022-11-01 15:51:37 +00:00
Tyler Neely
3c9f0c48e9 Merge branch 'project-pineapples' into tyler_1k_shard_unit_test 2022-11-01 16:31:29 +01:00
Tyler Neely
9a1258a708 Update tests/unit/1k_shards_1k_create_scanall.cpp
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2022-11-01 16:31:14 +01:00
Tyler Neely
c55ca836a4 Update tests/unit/1k_shards_1k_create_scanall.cpp
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2022-11-01 16:31:06 +01:00
Tyler Neely
5674ef4016 Update tests/unit/1k_shards_1k_create_scanall.cpp
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2022-11-01 16:30:37 +01:00
Tyler Neely
77c2afc9e7 Add TODO about ID mapping improvements 2022-11-01 15:27:35 +00:00
Tyler Neely
27292dd921 Merge branch 'project-pineapples' into tyler_remove_shard_map_from_machine_manager 2022-11-01 16:20:54 +01:00
Jeremy B
e4f1fd8647 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-11-01 16:06:44 +01:00
János Benjamin Antal
4608af9d00 Merge branch 'project-pineapples' into MG-improve-parameters 2022-11-01 15:56:16 +01:00
János Benjamin Antal
50e72a7c28 Merge branch 'project-pineapples' into T1138-MG-fix-edge-id-allocator 2022-11-01 15:56:12 +01:00
János Benjamin Antal
17a104677d Merge pull request #617 from memgraph/T1105-MG-profile-query-in-distributed 2022-11-01 15:55:52 +01:00
Tyler Neely
a6add80fc9 Use static RSM partitioning function for achieving a smooth Shard->ShardWorker distribution 2022-11-01 14:52:38 +00:00
Tyler Neely
5d3eaf6a55 Rename 1k_shards_1k_create_scanall test to high_density_shard_create_scan 2022-11-01 14:02:07 +00:00
Tyler Neely
a13f260236 Parameterize shard worker threads in the MachineConfig and simplify test output 2022-11-01 13:57:57 +00:00
jeremy
1f778ba5f3 Add possibility to give extra tests arg to MGBench 2022-11-01 14:14:55 +01:00
jeremy
787987168c Make benchmark work with any customer datasets 2022-11-01 12:51:01 +01:00
János Benjamin Antal
7b4b1ba8ed Speed up Parameters::AtTokenPosition 2022-11-01 11:51:35 +01:00
Tyler Neely
bb1e8aa164 Improve histogram output by adding a pretty table printing function 2022-11-01 10:46:25 +00:00
Tyler Neely
27a1311966 Merge branch 'tyler_1k_shard_unit_test' of github.com:memgraph/memgraph into T1122-MG-ShardManager-ThreadPool 2022-11-01 09:49:15 +00:00
János Benjamin Antal
30ff6487f2 Merge remote-tracking branch 'origin/project-pineapples' into T1105-MG-profile-query-in-distributed 2022-11-01 10:32:44 +01:00
Tyler Neely
119da2d7a7 Update CMakeLists.txt 2022-11-01 09:51:43 +01:00
Tyler Neely
fa972813d2 Update src/coordinator/shard_map.cpp
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-31 19:05:19 +01:00
Tyler Neely
951b058116 Complete migration from single-threaded ShardManager to multi-threaded ShardWorker processing 2022-10-31 18:04:30 +00:00
Tyler Neely
d0cad6e6ba Temporarily duplicate shard management logic from ShardManger in ShardWorker 2022-10-31 17:25:08 +00:00
Tyler Neely
6138277972 Merge branch 'tyler_remove_shard_map_from_machine_manager' of github.com:memgraph/memgraph into T1122-MG-ShardManager-ThreadPool 2022-10-31 17:14:01 +00:00
Tyler Neely
2f77eb96ff Remove redundant ShardMap copy from MachineManager to avoid race conditions 2022-10-31 16:49:46 +00:00
Tyler Neely
d7bc93c55f Fill Cron next time promise from each worker thread 2022-10-31 16:17:34 +00:00
Tyler Neely
039d5f51d3 Merge branch 'project-pineapples' into T1122-MG-ShardManager-ThreadPool 2022-10-31 17:06:36 +01:00
Tyler Neely
cebe6f62fa Implement skeleton worker threadpool for the ShardManager 2022-10-31 16:03:47 +00:00
jbajic
6834ce01fe Fix edge id allocator 2022-10-31 16:26:56 +01:00
jeremy
4c5cd1f847 Add possibility to have MgBench working against local file 2022-10-31 16:10:04 +01:00
János Benjamin Antal
f9e2a66961 Add const qualifier to function parameters 2022-10-31 15:53:04 +01:00
János Benjamin Antal
d5700ab5ff Use [[unlikely]] attribute 2022-10-31 15:53:04 +01:00
Tyler Neely
b0aaba6765 Merge branch 'project-pineapples' into tyler_1k_shard_unit_test 2022-10-31 15:47:38 +01:00
jeremy
e1f18f3733 Update location of Pokec datasets on aws 2022-10-31 15:19:34 +01:00
jeremy
c90b38faf0 Update aws address for datasets 2022-10-31 14:49:32 +01:00
jeremy
acbf3c764c Remove arg from __init__ 2022-10-31 13:35:41 +01:00
jeremy
37df41796f git puMerge branch 'T1079-MG-add-simple-query-to-benchmark_v2' of github.com:memgraph/memgraph into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-31 13:02:12 +01:00
jeremy
f04e1cda4b Add function 2022-10-31 13:02:05 +01:00
Jeremy B
f28ba89584 Update tests/mgbench/dataset_creator.py
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-31 13:01:42 +01:00
jeremy
60485311c8 Merge branch 'T1079-MG-add-simple-query-to-benchmark_v2' of github.com:memgraph/memgraph into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-31 12:57:46 +01:00
Jeremy B
36a1c43851 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-31 12:57:41 +01:00
Jeremy B
bae8c084b1 Update tests/mgbench/datasets.py
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-31 12:56:02 +01:00
Jeremy B
2898120eeb Update tests/mgbench/datasets.py
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-31 12:55:42 +01:00
jeremy
d62a45752a Remove unused variable 2022-10-31 12:55:38 +01:00
jeremy
c5ee6ffbc2 Update dataset_creator script 2022-10-31 12:41:28 +01:00
jeremy
5ef08f841a Update Dataset creation script 2022-10-31 11:56:20 +01:00
Kostas Kyrimis
59c7d81ae8 [🍍 < T1086-MG] Test distributed operators e2e (#607)
* Fix Explain queries
* Add Vertex/Edge accessor support for properties
* Fix projections
* Fix expansions to fetch destination vertex properties
* Fix improper use of ShardMap on bolt and replaced it with the ShardRequestManager 
* Add NameToId mappers on ShardRequestManager
* Add e2e tests for operators
* Fix OPTIONAL MATCH
2022-10-31 11:52:20 +02:00
jeremy
03c095e780 Update assert 2022-10-31 10:52:06 +01:00
Jeremy B
b542e49b3e Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-31 09:44:34 +01:00
Jeremy B
ddb30f49ea Update datasets.py 2022-10-31 09:43:22 +01:00
János Benjamin Antal
d920d7c293 Merge branch 'project-pineapples' into T1105-MG-profile-query-in-distributed 2022-10-31 09:27:05 +01:00
jeremy
b2e9717ec3 Factor HandleRead(msgs::ExpandOneRequest.. 2022-10-28 15:31:29 +02:00
jeremy
e0f6c951c1 Add possibilty to orderByElement on vector<VertexAccessor> 2022-10-28 15:31:15 +02:00
jeremy
7e35c71c58 Merge branch 'T1083-MG-limit-and-order-expand-one_v3' of github.com:memgraph/memgraph into T1083-MG-limit-and-order-expand-one_v3 2022-10-28 13:22:54 +02:00
Jeremy B
b6814b7a49 Update src/storage/v3/shard_rsm.cpp
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-28 13:22:49 +02:00
jeremy
903e29a081 Merge branch 'T1083-MG-limit-and-order-expand-one_v3' of github.com:memgraph/memgraph into T1083-MG-limit-and-order-expand-one_v3 2022-10-28 13:22:12 +02:00
Jeremy B
1c17692a26 Update src/storage/v3/shard_rsm.cpp
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-28 13:22:07 +02:00
jeremy
34fbaa0aee Merge branch 'T1083-MG-limit-and-order-expand-one_v3' of github.com:memgraph/memgraph into T1083-MG-limit-and-order-expand-one_v3 2022-10-28 13:21:52 +02:00
Jeremy B
476e2670d5 Update src/storage/v3/request_helper.cpp
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-28 13:21:48 +02:00
jeremy
be7aa55686 Add std::move 2022-10-28 13:21:43 +02:00
jeremy
46388ad35c Correct compilation 2022-10-28 13:10:00 +02:00
jeremy
0d5ee49e19 Correct test expectation 2022-10-28 12:55:56 +02:00
jeremy
74181114c2 Remove un-necessary variable 2022-10-28 12:55:47 +02:00
jeremy
009c1b4074 Replace include 2022-10-28 12:55:23 +02:00
jeremy
d0b8b27c29 Rename ordered->sorted 2022-10-28 12:52:37 +02:00
jeremy
79c2ae206f Update FillEdges usage (for compilation) 2022-10-28 11:24:41 +02:00
Tyler Neely
6b0168cb3d Add LocalTransport::ResponseLatencies 2022-10-28 08:51:12 +00:00
Tyler Neely
03cf264b76 Add test for 1k shards, 1k create vertices, and then a scan all 2022-10-28 08:29:27 +00:00
Tyler Neely
9448e23dc9 Check-in basic shard scheduler skeleton 2022-10-27 13:36:53 +00:00
Jeremy B
fa6129dc2b Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-10-26 15:59:15 +02:00
Tyler Neely
eafccaea84 Add a logarithmically bucketing histogram and record response latencies per message type (#616) 2022-10-26 15:57:11 +02:00
János Benjamin Antal
0c4f591b3c Merge branch 'project-pineapples' into T1105-MG-profile-query-in-distributed 2022-10-26 14:48:27 +02:00
Jeremy B
aeed7c32f9 Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-10-26 11:32:44 +02:00
János Benjamin Antal
534e365271 Suppress warning about exception escape for destructor 2022-10-26 11:22:27 +02:00
János Benjamin Antal
281ae158ec Make ReadTSC noexcept 2022-10-26 11:21:27 +02:00
János Benjamin Antal
39c9c215b1 Suppress clang-tidy warnings for 2022-10-26 11:14:24 +02:00
János Benjamin Antal
f6e78ce6da Fix TODOs in storage engine (#614)
Fixed various TODOs that were easy to fix to improve the code quality of the
newly implemented storage.
2022-10-26 10:40:35 +02:00
Jeremy B
8e7118efde Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-10-26 08:39:21 +02:00
János Benjamin Antal
3cf79f5bbf Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-25 21:24:20 +02:00
János Benjamin Antal
6a31c49432 Flatten nested conditional statements 2022-10-25 20:30:29 +02:00
János Benjamin Antal
fea92f4829 Merge remote-tracking branch 'origin/project-pineapples' into T1105-MG-profile-query-in-distributed 2022-10-25 20:28:54 +02:00
János Benjamin Antal
1703cd039d Populate custom data of profile query with request wait times 2022-10-25 20:15:06 +02:00
gvolfing
ca2351124b Make primary labels act as label indices (#605)
Because of the lexicographical sharding, the primary labels themselves are
acting as indexes. If a primary label is specified in a MATCH query we can
safely narrow the range of shards we have to scan through based on that label.
This PR introduces the necessary changes in order to achieve that.
2022-10-25 19:48:17 +02:00
János Benjamin Antal
5784c0d473 Return the custom data for profile queries 2022-10-25 15:26:42 +02:00
János Benjamin Antal
8ebc704819 Fix profile queries with ScanAll 2022-10-25 14:37:18 +02:00
János Benjamin Antal
5939fb2b0c Start transaction properly 2022-10-25 14:35:26 +02:00
János Benjamin Antal
ee64684b0b Add ScopedCustomProfile 2022-10-25 13:37:57 +02:00
János Benjamin Antal
1e4c02f8a5 Make profile query work 2022-10-25 13:37:27 +02:00
János Benjamin Antal
8353228ba7 Upgrade json lib to 3.11.2 2022-10-25 13:10:03 +02:00
jeremy
cdab8828e4 remove else 2022-10-25 12:30:14 +02:00
jeremy
c1d0fddaac Remove unnecessary else 2022-10-25 11:31:23 +02:00
jeremy
18423ce34d remove includes 2022-10-25 11:01:04 +02:00
jeremy
b4f68e7a60 remove includes 2022-10-25 10:56:16 +02:00
jeremy
e901c1fdb7 Refactor code 2022-10-25 10:45:33 +02:00
Jeremy B
016b3ee0d2 Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-10-25 10:30:22 +02:00
János Benjamin Antal
332afadf21 Split file parsing (#600)
Add temporary support for split files. This is only temporary solution until we
get the shard splitting implemented.
2022-10-25 10:27:13 +02:00
Tyler Neely
acc655f4fd Model-based testing of simulated full cluster (#584)
This PR adds support for generating randomized workloads that will be executed
against a simulated cluster, as well as against a correctness model. Initially
this just generates ScanAll and CreateVertex requests, and anything that it
creates, it also inserts into a `std::set`, and when we do a ScanAll, it asserts
that we get the same number of requests back. This will become much more
sophisticated over time, but it's already hitting pay-dirt.
2022-10-24 19:54:09 +02:00
jeremy
862af55266 Remove #NoCommit 2022-10-24 16:39:38 +02:00
jeremy
33c9ccee66 Adapt test 2022-10-24 16:38:28 +02:00
jeremy
8b9e7e2c65 Correct behavior of batch limit (was size_t) 2022-10-24 16:23:42 +02:00
jeremy
51e6802aa7 Safeguard in case ComputeExpression is called without opt 2022-10-24 15:44:57 +02:00
jeremy
386a0c5686 add comment 2022-10-24 15:44:37 +02:00
jeremy
994aab8774 Merge branch 'T1083-MG-limit-and-order-expand-one_v3' of github.com:memgraph/memgraph into T1083-MG-limit-and-order-expand-one_v3 2022-10-24 12:08:38 +02:00
jeremy
8112957a35 Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-10-24 12:08:02 +02:00
jeremy
b82e8748ad Attempt impl 2022-10-24 12:03:51 +02:00
Jeremy B
cb00b43ca7 Merge branch 'project-pineapples' into T1083-MG-limit-and-order-expand-one_v3 2022-10-21 18:17:34 +02:00
jeremy
55e0dbca80 Add limit to ExpandOne
Add missing pragma
Add test
Merge conflicts
2022-10-21 16:32:49 +02:00
Tyler Neely
12e7a261aa operator<< implementations for a few foundational types (#604) 2022-10-21 15:25:40 +02:00
Marko Budiselić
1c3bb969e9 Decouple interactive planning manual test (#585) 2022-10-21 13:34:13 +02:00
Jure Bajic
e5437080c5 Handle OrderBy in ScanVertices request (#594)
- Refactor shard_rsm and move function into expr.hpp/expr.cpp and request_helper.hpp/request_helper.cpp
2022-10-21 12:33:40 +02:00
Tyler Neely
17090dd8ac Load SPDLOG_LEVEL environment variable in unit tests binaries (#606)
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
2022-10-20 18:27:52 +02:00
gvolfing
281cebd386 Add filtering capabilities to ExpandOne request (#581) 2022-10-20 17:51:44 +02:00
János Benjamin Antal
f89a2bbf42 Make ExpandOne work in the query engine (#589) 2022-10-20 11:35:00 +02:00
gvolfing
85b8ce9101 Implement filtering capabilities for ScanAll (#578)
ScanVerticesrequest was not able to utilize filtering capabilities
before. With these modification it is now able to filter the scanned
vertices based on the filter_expressions field in the
ScanVerticesRequest message type.
2022-10-19 16:09:00 +02:00
Jure Bajic
6bb40a7f49 Create ExpandOne request (#580)
Refactor CreateEdge into CreateExpand
2022-10-19 13:55:46 +02:00
Jeremy B
1707ee648e Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-19 08:47:13 +02:00
gvolfing
07f34838bd Add e2e test to check ScanAll on new architecture (#603)
Add testing capabilities similar to what existed before on the older
architecture. With this test specifically, test if we can create
vertices and return them with ScanAll. There is a hard-coded 3s wait in
the test which is needed because the MachineManager need this time to
initialize itself.
2022-10-18 21:38:56 +02:00
Jeremy B
b6f8b5e130 Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-18 15:49:19 +02:00
jeremy
58243f4a26 Rename User->File
Use parser for argument i.o. simple variable in script
2022-10-18 15:47:13 +02:00
gvolfing
d06132cb33 Make ShardRequestManager work with futures (#588)
The communication between the ShardRequestManager and the RsmClient
used to be direct. In this PR this changes into a future-based
communication type. The RsmClient stores state about the currently
processed future (either read or write request) and exposes blocking
and non-blocking functionality to obtain the filled future. The
ShardRequestManager -for now- will send of the set of requests present
in the ExecutionState and block on each of them until the requests are
completed or the set of paginated responses(caused by, for example the
batch-limit in ScanAll) are ready for the next round.
2022-10-18 09:53:37 +02:00
Jeremy B
f063c1b1ad Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-17 11:41:55 +02:00
Marko Budiselić
5347c06d76 Add shutdown of LocalSystem during the shutdown process (#592) 2022-10-14 11:59:50 +02:00
Tyler Neely
4634ac484a Optimize simulator RNG usage (#590)
This causes instructions for `basic_request.cpp` to drop from 281 million to 28 million
2022-10-12 19:19:06 +03:00
jeremy
a5dc818e19 Add new dataset for mgbench 2022-10-12 16:35:47 +02:00
Jeremy B
4cb3b064c4 Add filter to scan all (#575)
Add several versions of ScanAll with filters.
Add helper function to transform an expression into string that can be parsed again once on the storage.
2022-10-12 11:46:59 +02:00
Jure Bajic
23171e76b6 Integrate bolt server (#572)
* Use query-v2 in the main executable
* Set up machine manager in memgraph
* Add `ShardRequestManager` to `Interpreter`
* Make vertex creation work
* Make scan all work
* Add edge type map in shard request manager
* Send schema over request
* Empty out DbAccessor
* Store shard mapping at creation
* Remove failing CI steps

Cooltura is the best place in Zagreb!

Co-authored-by: János Benjamin Antal <benjamin.antal@memgraph.io>
2022-10-11 16:31:46 +02:00
Jeremy B
6fd64d31f2 Update usage of Shard (#574)
Updating tests to use new constructor of Shard
Commenting test shard_request_manager
2022-10-05 11:56:36 +02:00
gvolfing
87111b2f89 Implement missing message handlers (#563)
Implement the missing message handlers with basic functionality. The
implementation does not include any capabilities to filter based on
expressions.
2022-10-03 15:31:06 +02:00
Tyler Neely
b5c7078c7d Stitch request manager with shard (#570)
Fix various bugs
2022-09-23 20:07:41 +02:00
Kostas Kyrimis
925835b080 Implement query engine client (#531)
- Add shard request manager
2022-09-22 15:05:43 +02:00
Tyler Neely
ce788f5f65 Machine manager and shard stitch (#569) 2022-09-22 13:55:16 +02:00
János Benjamin Antal
b8186bea2e Lexicographically ordered storage (#454) 2022-09-21 18:58:31 +02:00
János Benjamin Antal
3c4856dcb7 Merge remote-tracking branch 'origin/project-pineapples' into E118-MG-lexicographically-ordered-storage 2022-09-21 18:50:00 +02:00
Marko Budiselić
b4d6dc0930 Add proper transaction handling (#550) 2022-09-21 18:25:51 +02:00
Jure Bajic
817161a915 Implement expression evaluator in storage v3 (#534)
- Enable `mg-expr` in mg-storage-v3
-  Adapt id_mapper
- Change conversion function from PropertyValue to TypedValue
- Remove memgraph functions
- Enable expression tests on for storage
2022-09-21 17:26:55 +02:00
Tyler Neely
58eb2caf0f Add machine manager prototype (#533) 2022-09-21 16:57:25 +02:00
gvolfing
ecda71168c Implement message based actions (#538)
Create shard-side handlers for basic messages

Implement the handlers for CreateVertices, CreateEdges and ScanAll. Use
or modify the defined messages to interact with individual Shards and
test their behavior. Shard is currently being owned by ShardRsm
instances. The two top level dispatching functions Read() and Apply()
are responsible for read- and write operations respectively. Currently
there are a handful of messages that are defined but not utilized, these
will be used in the near future, as well as a couple of handler
functions with empty implementations.
2022-09-20 11:15:19 +02:00
János Benjamin Antal
8e1f83acc9 Reference vertices by id in edges (#532) 2022-09-14 13:11:14 +02:00
jbajic
f910cb770c Merge branch 'project-pineapples' into E118-MG-lexicographically-ordered-storage 2022-09-09 12:39:28 +02:00
Kostas Kyrimis
38d0b89b04 Implement expression evaluator library (#486) 2022-09-07 18:15:32 +03:00
Jure Bajic
a2a6a3855b Remove primary label from vertex (#529)
- Remove the primary label from vertex
- Pass vertex_validator instead of schema validator
- Fix vertex ctors
2022-09-07 13:36:10 +02:00
Jure Bajic
e9f0360fb3 Enable indices for storage v3 (#528)
- Enable indices in storage v3
- Add new test cases
- Change `CreateVertexAndValidate` to call `SetProperty` and `AddLabel`
2022-09-06 22:48:25 +02:00
János Benjamin Antal
fe4955447e Merge branch 'project-pineapples' into E118-MG-lexicographically-ordered-storage 2022-09-05 14:59:01 +02:00
Tyler Neely
1631c20df2 In memory shard map (#505) 2022-09-05 15:50:54 +03:00
János Benjamin Antal
947baedbe6 Merge remote-tracking branch 'origin/project-pineapples' into E118-MG-lexicographically-ordered-storage 2022-09-02 11:41:03 +02:00
Kostas Kyrimis
c6447eb48b Add shard requests responses (#526) 2022-09-01 18:54:47 +03:00
Jure Bajic
7e84744d07 Split storage and shards (#519)
- Rename storage to shard
- Add primary label and range for shard
- Remove id_mapper functionality from shard
- Adapt tests
2022-09-01 09:10:40 +02:00
Tyler Neely
c0d03888f4 Implement basic raft version (#498) 2022-08-30 15:07:34 +02:00
János Benjamin Antal
efb3c8d03d Remove multi-threaded related logic and variables (#460)
* Remove logic that was necessary for optimal multi-threaded performance, such
  as accumulating deleted object in local containers and appending them to a
  global one, handling overlapping locking.
* Remove background GC thread.
* Remove mutexes, locks and atomics throughout storage.
2022-08-30 13:41:53 +02:00
Jure Bajic
95dbc022c0 Integrate schema and keystore (#497)
- Integrate schema and keystore on vertex creation
- Add GC test for storage v3
- Add tests for accessors
- Fix all tests related to this except for query v2
- Fix labels not returning primary label
2022-08-29 14:38:25 +02:00
Tyler Neely
a40403e3ce Add local transport (#512)
* Create LocalTransport Io provider for sending messages to components on the same machine
* Move src/io/simulation/message_conversion.hpp to src/io/message_conversion.hpp for use in other Io providers
2022-08-29 13:49:51 +02:00
Tyler Neely
14c9e68456 Transport prototype (#466) 2022-08-12 08:24:32 +02:00
jbajic
68b26275a3 Merge branch 'project-pineapples' into E118-MG-lexicographically-ordered-storage 2022-08-08 11:34:46 +02:00
jbajic
5012824e05 Address review comments 2022-08-04 11:45:16 +02:00
Jure Bajic
a12a1ea358 Move schema to storage v3 and query v2
* Move schema to storage v3

* Remove schema from v2

* Move schema to query v2

* Remove schema from query v1

* Make glue v2

* Move schema related tests to newer versions of query and storage

* Fix typo in CMake

* Fix interpreter test

* Fix clang tidy errors

* Change temp dir name
2022-08-04 09:50:02 +02:00
János Benjamin Antal
2891041468 Make storage use KeyStore (#455) 2022-08-03 18:10:58 +02:00
János Benjamin Antal
cc5ee6a496 Merge remote-tracking branch 'origin/project-pineapples' into E118-MG-lexicographically-ordered-storage 2022-08-02 08:19:43 +02:00
jbajic
f57f30c8cf Merge branch 'project-pineapples' into E112-MG-implement-partial-schema 2022-08-01 10:46:11 +02:00
Jure Bajic
462daf3a2b Enforce schema on vertex creation
- Separating schema definition from schema validation
- Updating vertex_accessor and db_accessors with necessary methods
- Adding a primary label to Vertex
- Adding schema tests
- Updating existing tests for storage v3, and deprecating old:
  - interpreter => interpreter_v2
  - query_plan_accumulate_aggregate => storage_v3_query_plan_accumulate_aggregate
  - query_plan_create_set_remove_delete => storage_v3_query_plan_create_set_remove_delete
  - query_plan_bag_semantics => storage_v3_query_plan_bag_semantics
  - query_plan_edge_cases => storage_v3_query_plan_edge_cases
  - query_plan_v2_create_set_remove_delete => storage_v3_query_plan_v2_create_set_remove_delete
  - query_plan_match_filter_return => storage_v3_query_plan_match_filter_return
2022-07-29 13:38:17 +02:00
Marko Budiselic
2009fefc8a Merge branch 'master' into project-pineapples 2022-07-28 15:36:51 +02:00
Marko Budiselić
eb3f96d1f6 Bring changes from master to project-pineapples (#477)
* Fix aggregation functions on `null` and group-by inputs (#448)
* Upgrade Antrl to 4.10.1 and remove antlr_lock (#441)
* Update clang-tidy job (#476)
* Add parser stress test (#463)

NOTE: Doing this to have buildable comments on the project-pineapples branch

Co-authored-by: gvolfing <107616712+gvolfing@users.noreply.github.com>
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-07-28 15:36:17 +02:00
jbajic
264b233053 Merge branch 'project-pineapples' into E112-MG-implement-partial-schema 2022-07-22 11:48:45 +02:00
Jure Bajic
2ceaf59767 Create query engine v2 (#444)
Create version v2 of the query engine.
Adjust CMake and lisp files
Connect query engine v2 with storage engine v3
2022-07-19 12:28:19 +02:00
János Benjamin Antal
c0bee760bf git 2022-07-18 08:21:04 +02:00
Jure Bajic
3f4f66b57f Create schema DDL expressions
* Add initial schema implementation

* Add index to schema

* List schemas and enable multiple properties

* Implement SchemaTypes

* Apply suggestions from code review

Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

* Address review comments

* Remove Map and List

* Add schema operations in storage

* Add create and show schema queries

* Add privileges for schema

* Add missing keywords into lexer

* Add drop schema query

* Add schema visitors

* Update metadata

* Add PrepareSchemaQuery function

* Implement show schemas

* Add show schema query

* Fix schema visitor

* Add common schema type

* Fix grammar

* Temporary create ddl logic

* Fix naming for schemaproperty type to schema type

* Rename schemaproperty to schemapropertytype

* Enable Create schema ddl

* Override visitPropertyType

* Add initial schema implementation

* Add initial schema implementation

* Add index to schema

* List schemas and enable multiple properties

* Implement SchemaTypes

* Apply suggestions from code review

Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

* Address review comments

* Remove Map and List

* Apply suggestions from code review

Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>

Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>

* Add verification on creation and deletion

* Rename DeleteSchema to DropSchema

* Remove list and map from lexer

* Fix grammar with schemaTypeMap

* Add privilege and cypher visitor tests

* Catch repeating type name in schema definition

* Fix conflicting keywords

* Add notifications

* Drop float support

* Finish interpreter tests

* Fix tests

* Fix clang tidy errors

* Fix GetSchema

* Replace for with transfrom

* Add cloning og schema_property_map

* Address review comments

* Rename SchemaPropertyType to SchemaType

* Remove inline

* Assert of schema properties

Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2022-07-11 09:20:15 +02:00
Jure Bajic
2998f92595 Add initial schema implementation
* Add initial schema implementation

* Add index to schema

* List schemas and enable multiple properties

* Implement SchemaTypes

* Apply suggestions from code review

Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

* Address review comments

* Remove Map and List

* Apply suggestions from code review

Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>

Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2022-07-08 10:33:43 +02:00
János Benjamin Antal
1bdc32ba5d Copy storage v2 to create storage v3 (#416)
* Copy storage v2 to v3

* Integrate v3 to cmake

* Fix clang-tidy warnings

* Add dummy unit test for storage-v3 to trigger build for code analysis builds
2022-07-05 08:20:59 +02:00
János Benjamin Antal
21870a0e7e Merge branch 'master' into project-pineapples 2022-06-23 14:49:14 +02:00
329 changed files with 80910 additions and 16954 deletions

View File

@@ -6,6 +6,7 @@ Checks: '*,
-altera-unroll-loops,
-android-*,
-cert-err58-cpp,
-cert-str34-c,
-cppcoreguidelines-avoid-c-arrays,
-cppcoreguidelines-avoid-goto,
-cppcoreguidelines-avoid-magic-numbers,
@@ -49,6 +50,7 @@ Checks: '*,
-misc-non-private-member-variables-in-classes,
-modernize-avoid-c-arrays,
-modernize-concat-nested-namespaces,
-modernize-loop-convert,
-modernize-pass-by-value,
-modernize-use-equals-default,
-modernize-use-nodiscard,

View File

@@ -16,7 +16,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)

View File

@@ -26,12 +26,21 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
fetch-depth: 0
- name: Cache libs
uses: actions/cache@v3
env:
cache-name: cache-community-libs
with:
path: |
libs
key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('libs/setup.sh') }}
- name: Build community binaries
run: |
# Activate toolchain.
@@ -64,7 +73,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -75,14 +84,48 @@ jobs:
- name: Fetch all history for all tags and branches
run: git fetch
- name: Cache libs
uses: actions/cache@v3
env:
cache-name: cache-libs
with:
path: |
libs
key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('libs/setup.sh') }}
- name: Initialize deps
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Initialize dependencies.
./init
- name: Set base branch
if: ${{ github.event_name == 'pull_request' }}
run: |
echo "BASE_BRANCH=origin/${{ github.base_ref }}" >> $GITHUB_ENV
- name: Set base branch # if we manually dispatch or push to master
if: ${{ github.event_name != 'pull_request' }}
run: |
echo "BASE_BRANCH=origin/master" >> $GITHUB_ENV
- name: Python code analysis
run: |
CHANGED_FILES=$(git diff -U0 ${{ env.BASE_BRANCH }}... --name-only)
for file in ${CHANGED_FILES}; do
echo ${file}
if [[ ${file} == *.py ]]; then
python3 -m black --check --diff ${file}
python3 -m isort --check-only --profile "black" --diff ${file}
fi
done
- name: Build combined ASAN, UBSAN and coverage binaries
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Initialize dependencies.
./init
cd build
cmake -DTEST_COVERAGE=ON -DASAN=ON -DUBSAN=ON ..
make -j$THREADS memgraph__unit
@@ -110,7 +153,7 @@ jobs:
tar -czf code_coverage.tar.gz coverage.json html report.json summary.rmu
- name: Save code coverage
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Code coverage"
path: tools/github/generated/code_coverage.tar.gz
@@ -130,7 +173,7 @@ jobs:
source /opt/toolchain-v4/activate
# Restrict clang-tidy results only to the modified parts
git diff -U0 ${{ env.BASE_BRANCH }}... -- src | ./tools/github/clang-tidy/clang-tidy-diff.py -p 1 -j $THREADS -path build | tee ./build/clang_tidy_output.txt
git diff -U0 ${{ env.BASE_BRANCH }}... -- src | ./tools/github/clang-tidy/clang-tidy-diff.py -p 1 -j $THREADS -extra-arg="-DMG_CLANG_TIDY_CHECK" -path build | tee ./build/clang_tidy_output.txt
# Fail if any warning is reported
! cat ./build/clang_tidy_output.txt | ./tools/github/clang-tidy/grep_error_lines.sh > /dev/null
@@ -145,12 +188,21 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
fetch-depth: 0
- name: Cache libs
uses: actions/cache@v3
env:
cache-name: cache-libs
with:
path: |
libs
key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('libs/setup.sh') }}
- name: Build debug binaries
run: |
# Activate toolchain.
@@ -164,52 +216,23 @@ jobs:
cmake ..
make -j$THREADS
- name: Run leftover CTest tests
- name: Run simulation tests
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Run leftover CTest tests (all except unit and benchmark tests).
# Run simulation tests.
cd build
ctest -E "(memgraph__unit|memgraph__benchmark)" --output-on-failure
ctest -R memgraph__simulation --output-on-failure -j$THREADS
- name: Run drivers tests
run: |
./tests/drivers/run.sh
- name: Run integration tests
run: |
cd tests/integration
for name in *; do
if [ ! -d $name ]; then continue; fi
pushd $name >/dev/null
echo "Running: $name"
if [ -x prepare.sh ]; then
./prepare.sh
fi
if [ -x runner.py ]; then
./runner.py
elif [ -x runner.sh ]; then
./runner.sh
fi
echo
popd >/dev/null
done
- name: Run cppcheck and clang-format
- name: Run single benchmark test
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Run cppcheck and clang-format.
cd tools/github
./cppcheck_and_clang_format diff
- name: Save cppcheck and clang-format errors
uses: actions/upload-artifact@v2
with:
name: "Code coverage"
path: tools/github/cppcheck_and_clang_format.txt
# Run simulation tests.
cd tests/mgbench
./benchmark.py accesscontrol/small --num-workers-for-import 1 --test-system-arg "split-file splitfiles/accesscontrol_small.shard_configuration bolt-num-workers 1"
release_build:
name: "Release build"
@@ -221,12 +244,21 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
fetch-depth: 0
- name: Cache libs
uses: actions/cache@v3
env:
cache-name: cache-release-libs
with:
path: |
libs
key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('libs/setup.sh') }}
- name: Build release binaries
run: |
# Activate toolchain.
@@ -240,19 +272,6 @@ jobs:
cmake -DCMAKE_BUILD_TYPE=release ..
make -j$THREADS
- name: Run GQL Behave tests
run: |
cd tests/gql_behave
./continuous_integration
- name: Save quality assurance status
uses: actions/upload-artifact@v2
with:
name: "GQL Behave Status"
path: |
tests/gql_behave/gql_behave_status.csv
tests/gql_behave/gql_behave_status.html
- name: Run unit tests
run: |
# Activate toolchain.
@@ -262,6 +281,24 @@ jobs:
cd build
ctest -R memgraph__unit --output-on-failure -j$THREADS
- name: Run simulation tests
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Run simulation tests.
cd build
ctest -R memgraph__simulation --output-on-failure -j$THREADS
- name: Run single benchmark test
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Run simulation tests.
cd tests/mgbench
./benchmark.py accesscontrol/small --num-workers-for-import 1 --test-system-arg "split-file splitfiles/accesscontrol_small.shard_configuration bolt-num-workers 1"
- name: Run e2e tests
run: |
# TODO(gitbuda): Setup mgclient and pymgclient properly.
@@ -269,167 +306,4 @@ jobs:
./setup.sh
source ve3/bin/activate
cd e2e
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:../../libs/mgclient/lib python runner.py --workloads-root-directory .
- name: Run stress test (plain)
run: |
cd tests/stress
./continuous_integration
- name: Run stress test (SSL)
run: |
cd tests/stress
./continuous_integration --use-ssl
- name: Run durability test
run: |
cd tests/stress
source ve3/bin/activate
python3 durability --num-steps 5
- name: Create enterprise DEB package
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
cd build
# create mgconsole
# we use the -B to force the build
make -j$THREADS -B mgconsole
# Create enterprise DEB package.
mkdir output && cd output
cpack -G DEB --config ../CPackConfig.cmake
- name: Save enterprise DEB package
uses: actions/upload-artifact@v2
with:
name: "Enterprise DEB package"
path: build/output/memgraph*.deb
- name: Save test data
uses: actions/upload-artifact@v2
if: always()
with:
name: "Test data"
path: |
# multiple paths could be defined
build/logs
release_jepsen_test:
name: "Release Jepsen Test"
runs-on: [self-hosted, Linux, X64, Debian10, JepsenControl]
#continue-on-error: true
env:
THREADS: 24
MEMGRAPH_ENTERPRISE_LICENSE: ${{ secrets.MEMGRAPH_ENTERPRISE_LICENSE }}
MEMGRAPH_ORGANIZATION_NAME: ${{ secrets.MEMGRAPH_ORGANIZATION_NAME }}
steps:
- name: Set up repository
uses: actions/checkout@v2
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
fetch-depth: 0
- name: Build release binaries
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Initialize dependencies.
./init
# Build only memgraph release binarie.
cd build
cmake -DCMAKE_BUILD_TYPE=release ..
make -j$THREADS memgraph
- name: Run Jepsen tests
run: |
cd tests/jepsen
./run.sh test --binary ../../build/memgraph --run-args "test-all --node-configs resources/node-config.edn" --ignore-run-stdout-logs --ignore-run-stderr-logs
- name: Save Jepsen report
uses: actions/upload-artifact@v2
if: ${{ always() }}
with:
name: "Jepsen Report"
path: tests/jepsen/Jepsen.tar.gz
release_benchmarks:
name: "Release benchmarks"
runs-on: [self-hosted, Linux, X64, Diff, Gen7]
env:
THREADS: 24
MEMGRAPH_ENTERPRISE_LICENSE: ${{ secrets.MEMGRAPH_ENTERPRISE_LICENSE }}
MEMGRAPH_ORGANIZATION_NAME: ${{ secrets.MEMGRAPH_ORGANIZATION_NAME }}
steps:
- name: Set up repository
uses: actions/checkout@v2
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
fetch-depth: 0
- name: Build release binaries
run: |
# Activate toolchain.
source /opt/toolchain-v4/activate
# Initialize dependencies.
./init
# Build only memgraph release binaries.
cd build
cmake -DCMAKE_BUILD_TYPE=release ..
make -j$THREADS
- name: Run macro benchmarks
run: |
cd tests/macro_benchmark
./harness QuerySuite MemgraphRunner \
--groups aggregation 1000_create unwind_create dense_expand match \
--no-strict
- name: Get branch name (merge)
if: github.event_name != 'pull_request'
shell: bash
run: echo "BRANCH_NAME=$(echo ${GITHUB_REF#refs/heads/} | tr / -)" >> $GITHUB_ENV
- name: Get branch name (pull request)
if: github.event_name == 'pull_request'
shell: bash
run: echo "BRANCH_NAME=$(echo ${GITHUB_HEAD_REF} | tr / -)" >> $GITHUB_ENV
- name: Upload macro benchmark results
run: |
cd tools/bench-graph-client
virtualenv -p python3 ve3
source ve3/bin/activate
pip install -r requirements.txt
./main.py --benchmark-name "macro_benchmark" \
--benchmark-results-path "../../tests/macro_benchmark/.harness_summary" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
- name: Run mgbench
run: |
cd tests/mgbench
./benchmark.py --num-workers-for-benchmark 12 --export-results benchmark_result.json pokec/medium/*/*
- name: Upload mgbench results
run: |
cd tools/bench-graph-client
virtualenv -p python3 ve3
source ve3/bin/activate
pip install -r requirements.txt
./main.py --benchmark-name "mgbench" \
--benchmark-results-path "../../tests/mgbench/benchmark_result.json" \
--github-run-id "${{ github.run_id }}" \
--github-run-number "${{ github.run_number }}" \
--head-branch-name "${{ env.BRANCH_NAME }}"
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:../../libs/mgclient/lib python runner.py --workloads-root-directory ./distributed_queries

View File

@@ -14,7 +14,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -39,7 +39,7 @@ jobs:
source /opt/toolchain-v4/activate
# The results are also written to standard output in order to retain them in the logs
./tools/github/clang-tidy/run-clang-tidy.py -p build -j $THREADS -clang-tidy-binary=/opt/toolchain-v4/bin/clang-tidy "$PWD/src/*" |
./tools/github/clang-tidy/run-clang-tidy.py -p build -j $THREADS -extra-arg="-DMG_CLANG_TIDY_CHECK" -clang-tidy-binary=/opt/toolchain-v4/bin/clang-tidy "$PWD/src/*" |
tee ./build/full_clang_tidy_output.txt
- name: Summarize clang-tidy results

View File

@@ -17,7 +17,7 @@ jobs:
run: |
./release/package/run.sh package centos-7
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: centos-7
path: build/output/centos-7/memgraph*.rpm
@@ -34,7 +34,7 @@ jobs:
run: |
./release/package/run.sh package centos-9
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: centos-9
path: build/output/centos-9/memgraph*.rpm
@@ -51,7 +51,7 @@ jobs:
run: |
./release/package/run.sh package debian-10
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: debian-10
path: build/output/debian-10/memgraph*.deb
@@ -68,7 +68,7 @@ jobs:
run: |
./release/package/run.sh package debian-11
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: debian-11
path: build/output/debian-11/memgraph*.deb
@@ -87,7 +87,7 @@ jobs:
./run.sh package debian-11 --for-docker
./run.sh docker
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: docker
path: build/output/docker/memgraph*.tar.gz
@@ -104,7 +104,7 @@ jobs:
run: |
./release/package/run.sh package ubuntu-18.04
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: ubuntu-1804
path: build/output/ubuntu-18.04/memgraph*.deb
@@ -121,7 +121,7 @@ jobs:
run: |
./release/package/run.sh package ubuntu-20.04
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: ubuntu-2004
path: build/output/ubuntu-20.04/memgraph*.deb
@@ -138,7 +138,7 @@ jobs:
run: |
./release/package/run.sh package ubuntu-22.04
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: ubuntu-2204
path: build/output/ubuntu-22.04/memgraph*.deb
@@ -155,7 +155,7 @@ jobs:
run: |
./release/package/run.sh package debian-11 --for-platform
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: debian-11-platform
path: build/output/debian-11/memgraph*.deb
@@ -172,7 +172,7 @@ jobs:
run: |
./release/package/run.sh package debian-11-arm
- name: "Upload package"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: debian-11-arm
path: build/output/debian-11-arm/memgraph*.deb

View File

@@ -17,7 +17,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -55,7 +55,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -97,7 +97,7 @@ jobs:
tar -czf code_coverage.tar.gz coverage.json html report.json summary.rmu
- name: Save code coverage
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Code coverage"
path: tools/github/generated/code_coverage.tar.gz
@@ -112,7 +112,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -173,7 +173,7 @@ jobs:
./cppcheck_and_clang_format diff
- name: Save cppcheck and clang-format errors
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Code coverage"
path: tools/github/cppcheck_and_clang_format.txt
@@ -189,7 +189,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -225,7 +225,7 @@ jobs:
rpmlint memgraph*.rpm
- name: Save enterprise RPM package
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Enterprise RPM package"
path: build/output/memgraph*.rpm
@@ -262,7 +262,7 @@ jobs:
./continuous_integration
- name: Save quality assurance status
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "GQL Behave Status"
path: |

View File

@@ -17,7 +17,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -55,7 +55,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -97,7 +97,7 @@ jobs:
tar -czf code_coverage.tar.gz coverage.json html report.json summary.rmu
- name: Save code coverage
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Code coverage"
path: tools/github/generated/code_coverage.tar.gz
@@ -112,7 +112,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -173,7 +173,7 @@ jobs:
./cppcheck_and_clang_format diff
- name: Save cppcheck and clang-format errors
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Code coverage"
path: tools/github/cppcheck_and_clang_format.txt
@@ -189,7 +189,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -224,7 +224,7 @@ jobs:
cpack -G DEB --config ../CPackConfig.cmake
- name: Save enterprise DEB package
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Enterprise DEB package"
path: build/output/memgraph*.deb
@@ -261,7 +261,7 @@ jobs:
./continuous_integration
- name: Save quality assurance status
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "GQL Behave Status"
path: |
@@ -324,7 +324,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -349,7 +349,7 @@ jobs:
./run.sh test --binary ../../build/memgraph --run-args "test-all --node-configs resources/node-config.edn" --ignore-run-stdout-logs --ignore-run-stderr-logs
- name: Save Jepsen report
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
if: ${{ always() }}
with:
name: "Jepsen Report"

View File

@@ -19,7 +19,7 @@ jobs:
DOCKER_REPOSITORY_NAME: memgraph
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v3
- name: Set up QEMU
uses: docker/setup-qemu-action@v1

View File

@@ -17,7 +17,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -55,7 +55,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -97,7 +97,7 @@ jobs:
tar -czf code_coverage.tar.gz coverage.json html report.json summary.rmu
- name: Save code coverage
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Code coverage"
path: tools/github/generated/code_coverage.tar.gz
@@ -112,7 +112,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -173,7 +173,7 @@ jobs:
./cppcheck_and_clang_format diff
- name: Save cppcheck and clang-format errors
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Code coverage"
path: tools/github/cppcheck_and_clang_format.txt
@@ -189,7 +189,7 @@ jobs:
steps:
- name: Set up repository
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
# Number of commits to fetch. `0` indicates all history for all
# branches and tags. (default: 1)
@@ -224,7 +224,7 @@ jobs:
cpack -G DEB --config ../CPackConfig.cmake
- name: Save enterprise DEB package
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "Enterprise DEB package"
path: build/output/memgraph*.deb
@@ -261,7 +261,7 @@ jobs:
./continuous_integration
- name: Save quality assurance status
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
name: "GQL Behave Status"
path: |

12
.gitignore vendored
View File

@@ -23,6 +23,8 @@ cmake-build-*
cmake/DownloadProject/
dist/
src/query/frontend/opencypher/generated/
src/query/v2/frontend/opencypher/generated/
src/parser/opencypher/generated
tags
ve/
ve3/
@@ -50,15 +52,25 @@ src/distributed/pull_produce_rpc_messages.hpp
src/distributed/storage_gc_rpc_messages.hpp
src/distributed/token_sharing_rpc_messages.hpp
src/distributed/updates_rpc_messages.hpp
src/query/v2/frontend/ast/ast.hpp
src/query/frontend/ast/ast.hpp
src/storage/v3/bindings/ast/ast.hpp
src/query/distributed/frontend/ast/ast_serialization.hpp
src/query/v2/distributed/frontend/ast/ast_serialization.hpp
src/durability/distributed/state_delta.hpp
src/durability/single_node/state_delta.hpp
src/durability/single_node_ha/state_delta.hpp
src/query/frontend/semantic/symbol.hpp
src/query/v2/frontend/semantic/symbol.hpp
src/expr/semantic/symbol.hpp
src/query/distributed/frontend/semantic/symbol_serialization.hpp
src/query/v2/distributed/frontend/semantic/symbol_serialization.hpp
src/query/distributed/plan/ops.hpp
src/query/v2/distributed/plan/ops.hpp
src/query/plan/operator.hpp
src/query/v2/plan/operator.hpp
src/parser/opencypher/generated
src/expr/semantic/symbol.hpp
src/raft/log_entry.hpp
src/raft/raft_rpc_messages.hpp
src/raft/snapshot_metadata.hpp

View File

@@ -6,18 +6,15 @@ repos:
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/psf/black
rev: 22.3.0
rev: 22.10.0
hooks:
- id: black
args: # arguments to configure black
- --line-length=120
- --include='\.pyi?$'
# these folders wont be formatted by black
- --exclude="""\.git |
\.__pycache__|
build|
libs|
.cache"""
- repo: https://github.com/pycqa/isort
rev: 5.10.1
hooks:
- id: isort
name: isort (python)
args: ["--profile", "black"]
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: v13.0.0
hooks:

View File

@@ -182,7 +182,8 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
# c99-designator is disabled because of required mixture of designated and
# non-designated initializers in Python Query Module code (`py_module.cpp`).
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall \
-Werror=switch -Werror=switch-bool -Werror=return-type \
-Werror=switch -Werror=switch-bool -Werror=implicit-fallthrough \
-Werror=return-type \
-Werror=return-stack-address \
-Wno-c99-designator \
-DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT")

View File

@@ -83,13 +83,9 @@ modifications:
value: "true"
override: true
- name: "query_modules_directory"
value: "/usr/lib/memgraph/query_modules"
override: true
- name: "auth_module_executable"
value: "/usr/lib/memgraph/auth_module/example.py"
override: false
# - name: "query_modules_directory"
# value: "/usr/lib/memgraph/query_modules"
# override: true
- name: "memory_limit"
value: "0"

7
init
View File

@@ -140,5 +140,12 @@ done;
python3 -m pip install pre-commit
python3 -m pre_commit install
# Install py format tools
echo "Install black formatter"
python3 -m pip install black==22.10.*
echo "Install isort"
python3 -m pip install isort==5.10.*
# Link `include/mgp.py` with `release/mgp/mgp.py`
ln -v -f include/mgp.py release/mgp/mgp.py

View File

@@ -116,7 +116,7 @@ declare -A primary_urls=(
["pymgclient"]="http://$local_cache_host/git/pymgclient.git"
["mgconsole"]="http://$local_cache_host/git/mgconsole.git"
["spdlog"]="http://$local_cache_host/git/spdlog"
["nlohmann"]="http://$local_cache_host/file/nlohmann/json/4f8fba14066156b73f1189a2b8bd568bde5284c5/single_include/nlohmann/json.hpp"
["nlohmann"]="http://$local_cache_host/file/nlohmann/json/9d69186291aca4f0137b69c1dee313b391ff564c/single_include/nlohmann/json.hpp"
["neo4j"]="http://$local_cache_host/file/neo4j-community-3.2.3-unix.tar.gz"
["librdkafka"]="http://$local_cache_host/git/librdkafka.git"
["protobuf"]="http://$local_cache_host/git/protobuf.git"
@@ -141,7 +141,7 @@ declare -A secondary_urls=(
["pymgclient"]="https://github.com/memgraph/pymgclient.git"
["mgconsole"]="http://github.com/memgraph/mgconsole.git"
["spdlog"]="https://github.com/gabime/spdlog"
["nlohmann"]="https://raw.githubusercontent.com/nlohmann/json/4f8fba14066156b73f1189a2b8bd568bde5284c5/single_include/nlohmann/json.hpp"
["nlohmann"]="https://raw.githubusercontent.com/nlohmann/json/9d69186291aca4f0137b69c1dee313b391ff564c/single_include/nlohmann/json.hpp"
["neo4j"]="https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/neo4j-community-3.2.3-unix.tar.gz"
["librdkafka"]="https://github.com/edenhill/librdkafka.git"
["protobuf"]="https://github.com/protocolbuffers/protobuf.git"

12
pyproject.toml Normal file
View File

@@ -0,0 +1,12 @@
[tool.black]
line-length = 120
include = '\.pyi?$'
extend-exclude = '''
/(
| .git
| .__pycache__
| build
| libs
| .cache
)/
'''

View File

@@ -5,16 +5,23 @@ add_subdirectory(lisp)
add_subdirectory(utils)
add_subdirectory(requests)
add_subdirectory(io)
add_subdirectory(io/simulator)
add_subdirectory(kvstore)
add_subdirectory(telemetry)
add_subdirectory(communication)
add_subdirectory(memory)
add_subdirectory(storage/v2)
add_subdirectory(storage/v3)
add_subdirectory(integrations)
add_subdirectory(query)
add_subdirectory(query/v2)
add_subdirectory(slk)
add_subdirectory(rpc)
add_subdirectory(auth)
add_subdirectory(parser)
add_subdirectory(expr)
add_subdirectory(coordinator)
add_subdirectory(functions)
if (MG_ENTERPRISE)
add_subdirectory(audit)
@@ -31,13 +38,13 @@ include_directories(${CMAKE_CURRENT_BINARY_DIR})
# Memgraph Single Node v2 Executable
# ----------------------------------------------------------------------------
set(mg_single_node_v2_sources
glue/communication.cpp
memgraph.cpp
glue/auth.cpp
glue/v2/communication.cpp
memgraph.cpp
glue/v2/auth.cpp
)
set(mg_single_node_v2_libs stdc++fs Threads::Threads
telemetry_lib mg-query mg-communication mg-memory mg-utils mg-auth mg-license mg-settings)
telemetry_lib mg-query-v2 mg-communication mg-memory mg-utils mg-auth mg-license mg-settings mg-io mg-coordinator)
if (MG_ENTERPRISE)
# These are enterprise subsystems
set(mg_single_node_v2_libs ${mg_single_node_v2_libs} mg-audit)
@@ -120,19 +127,3 @@ install(CODE "file(MAKE_DIRECTORY \$ENV{DESTDIR}/var/log/memgraph
# ----------------------------------------------------------------------------
# Memgraph CSV Import Tool Executable
# ----------------------------------------------------------------------------
add_executable(mg_import_csv mg_import_csv.cpp)
target_link_libraries(mg_import_csv mg-storage-v2)
# Strip the executable in release build.
if (lower_build_type STREQUAL "release")
add_custom_command(TARGET mg_import_csv POST_BUILD
COMMAND strip -s mg_import_csv
COMMENT "Stripping symbols and sections from mg_import_csv")
endif()
install(TARGETS mg_import_csv RUNTIME DESTINATION bin)
# ----------------------------------------------------------------------------
# Memgraph CSV Import Tool Executable
# ----------------------------------------------------------------------------

View File

@@ -37,7 +37,7 @@ const std::vector<Permission> kPermissionsAll = {
Permission::CONSTRAINT, Permission::DUMP, Permission::AUTH, Permission::REPLICATION,
Permission::DURABILITY, Permission::READ_FILE, Permission::FREE_MEMORY, Permission::TRIGGER,
Permission::CONFIG, Permission::STREAM, Permission::MODULE_READ, Permission::MODULE_WRITE,
Permission::WEBSOCKET};
Permission::WEBSOCKET, Permission::SCHEMA};
} // namespace
std::string PermissionToString(Permission permission) {
@@ -84,6 +84,8 @@ std::string PermissionToString(Permission permission) {
return "MODULE_WRITE";
case Permission::WEBSOCKET:
return "WEBSOCKET";
case Permission::SCHEMA:
return "SCHEMA";
}
}

View File

@@ -38,7 +38,8 @@ enum class Permission : uint64_t {
STREAM = 1U << 17U,
MODULE_READ = 1U << 18U,
MODULE_WRITE = 1U << 19U,
WEBSOCKET = 1U << 20U
WEBSOCKET = 1U << 20U,
SCHEMA = 1U << 21U
};
// clang-format on

68
src/common/errors.hpp Normal file
View File

@@ -0,0 +1,68 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <cstdint>
#include <string_view>
namespace memgraph::common {
enum class ErrorCode : uint8_t {
SERIALIZATION_ERROR,
NONEXISTENT_OBJECT,
DELETED_OBJECT,
VERTEX_HAS_EDGES,
PROPERTIES_DISABLED,
VERTEX_ALREADY_INSERTED,
// Schema Violations
SCHEMA_NO_SCHEMA_DEFINED_FOR_LABEL,
SCHEMA_VERTEX_PROPERTY_WRONG_TYPE,
SCHEMA_VERTEX_UPDATE_PRIMARY_KEY,
SCHEMA_VERTEX_UPDATE_PRIMARY_LABEL,
SCHEMA_VERTEX_SECONDARY_LABEL_IS_PRIMARY,
SCHEMA_VERTEX_PRIMARY_PROPERTIES_UNDEFINED,
OBJECT_NOT_FOUND,
};
constexpr std::string_view ErrorCodeToString(const ErrorCode code) {
switch (code) {
case ErrorCode::SERIALIZATION_ERROR:
return "SERIALIZATION_ERROR";
case ErrorCode::NONEXISTENT_OBJECT:
return "NONEXISTENT_OBJECT";
case ErrorCode::DELETED_OBJECT:
return "DELETED_OBJECT";
case ErrorCode::VERTEX_HAS_EDGES:
return "VERTEX_HAS_EDGES";
case ErrorCode::PROPERTIES_DISABLED:
return "PROPERTIES_DISABLED";
case ErrorCode::VERTEX_ALREADY_INSERTED:
return "VERTEX_ALREADY_INSERTED";
case ErrorCode::SCHEMA_NO_SCHEMA_DEFINED_FOR_LABEL:
return "SCHEMA_NO_SCHEMA_DEFINED_FOR_LABEL";
case ErrorCode::SCHEMA_VERTEX_PROPERTY_WRONG_TYPE:
return "SCHEMA_VERTEX_PROPERTY_WRONG_TYPE";
case ErrorCode::SCHEMA_VERTEX_UPDATE_PRIMARY_KEY:
return "SCHEMA_VERTEX_UPDATE_PRIMARY_KEY";
case ErrorCode::SCHEMA_VERTEX_UPDATE_PRIMARY_LABEL:
return "SCHEMA_VERTEX_UPDATE_PRIMARY_LABEL";
case ErrorCode::SCHEMA_VERTEX_SECONDARY_LABEL_IS_PRIMARY:
return "SCHEMA_VERTEX_SECONDARY_LABEL_IS_PRIMARY";
case ErrorCode::SCHEMA_VERTEX_PRIMARY_PROPERTIES_UNDEFINED:
return "SCHEMA_VERTEX_PRIMARY_PROPERTIES_UNDEFINED";
case ErrorCode::OBJECT_NOT_FOUND:
return "OBJECT_NOT_FOUND";
}
}
} // namespace memgraph::common

19
src/common/types.hpp Normal file
View File

@@ -0,0 +1,19 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <cstdint>
namespace memgraph::common {
enum class SchemaType : uint8_t { BOOL, INT, STRING, DATE, LOCALTIME, LOCALDATETIME, DURATION };
} // namespace memgraph::common

View File

@@ -74,7 +74,7 @@ State RunHandlerV4(Signature signature, TSession &session, State state, Marker m
}
case Signature::Route: {
if constexpr (bolt_minor >= 3) {
if (signature == Signature::Route) return HandleRoute<TSession>(session);
return HandleRoute<TSession>(session);
} else {
spdlog::trace("Supported only in bolt v4.3");
return State::Close;

View File

@@ -30,6 +30,7 @@
#include "communication/context.hpp"
#include "communication/v2/pool.hpp"
#include "communication/v2/session.hpp"
#include "utils/message.hpp"
#include "utils/spin_lock.hpp"
#include "utils/synchronized.hpp"
@@ -58,10 +59,10 @@ class Listener final : public std::enable_shared_from_this<Listener<TSession, TS
bool IsRunning() const noexcept { return alive_.load(std::memory_order_relaxed); }
private:
Listener(boost::asio::io_context &io_context, TSessionData *data, ServerContext *server_context,
Listener(boost::asio::io_context &io_context, TSessionData &data, ServerContext *server_context,
tcp::endpoint &endpoint, const std::string_view service_name, const uint64_t inactivity_timeout_sec)
: io_context_(io_context),
data_(data),
data_(&data),
server_context_(server_context),
acceptor_(io_context_),
endpoint_{endpoint},
@@ -110,7 +111,7 @@ class Listener final : public std::enable_shared_from_this<Listener<TSession, TS
return OnError(ec, "accept");
}
auto session = SessionHandler::Create(std::move(socket), data_, *server_context_, endpoint_, inactivity_timeout_,
auto session = SessionHandler::Create(std::move(socket), *data_, *server_context_, endpoint_, inactivity_timeout_,
service_name_);
session->Start();
DoAccept();

View File

@@ -72,7 +72,7 @@ class Server final {
* Constructs and binds server to endpoint, operates on session data and
* invokes workers_count workers
*/
Server(ServerEndpoint &endpoint, TSessionData *session_data, ServerContext *server_context,
Server(ServerEndpoint &endpoint, TSessionData &session_data, ServerContext *server_context,
const int inactivity_timeout_sec, const std::string_view service_name,
size_t workers_count = std::thread::hardware_concurrency())
: endpoint_{endpoint},

View File

@@ -41,6 +41,7 @@
#include <boost/beast/websocket/rfc6455.hpp>
#include <boost/system/detail/error_code.hpp>
#include "communication/buffer.hpp"
#include "communication/context.hpp"
#include "communication/exceptions.hpp"
#include "utils/logging.hpp"
@@ -139,7 +140,7 @@ class WebsocketSession : public std::enable_shared_from_this<WebsocketSession<TS
private:
// Take ownership of the socket
explicit WebsocketSession(tcp::socket &&socket, TSessionData *data, tcp::endpoint endpoint,
explicit WebsocketSession(tcp::socket &&socket, TSessionData &data, tcp::endpoint endpoint,
std::string_view service_name)
: ws_(std::move(socket)),
strand_{boost::asio::make_strand(ws_.get_executor())},
@@ -311,13 +312,13 @@ class Session final : public std::enable_shared_from_this<Session<TSession, TSes
}
private:
explicit Session(tcp::socket &&socket, TSessionData *data, ServerContext &server_context, tcp::endpoint endpoint,
explicit Session(tcp::socket &&socket, TSessionData &data, ServerContext &server_context, tcp::endpoint endpoint,
const std::chrono::seconds inactivity_timeout_sec, std::string_view service_name)
: socket_(CreateSocket(std::move(socket), server_context)),
strand_{boost::asio::make_strand(GetExecutor())},
output_stream_([this](const uint8_t *data, size_t len, bool have_more) { return Write(data, len, have_more); }),
session_(data, endpoint, input_buffer_.read_end(), &output_stream_),
data_{data},
data_{&data},
endpoint_{endpoint},
remote_endpoint_{GetRemoteEndpoint()},
service_name_{service_name},
@@ -373,7 +374,7 @@ class Session final : public std::enable_shared_from_this<Session<TSession, TSes
spdlog::info("Switching {} to websocket connection", remote_endpoint_);
if (std::holds_alternative<TCPSocket>(socket_)) {
auto sock = std::get<TCPSocket>(std::move(socket_));
WebsocketSession<TSession, TSessionData>::Create(std::move(sock), data_, endpoint_, service_name_)
WebsocketSession<TSession, TSessionData>::Create(std::move(sock), *data_, endpoint_, service_name_)
->DoAccept(parser.release());
execution_active_ = false;
return;
@@ -465,7 +466,7 @@ class Session final : public std::enable_shared_from_this<Session<TSession, TSes
if (timeout_timer_.expiry() <= boost::asio::steady_timer::clock_type::now()) {
// The deadline has passed. Stop the session. The other actors will
// terminate as soon as possible.
spdlog::info("Shutting down session after {} of inactivity", timeout_seconds_);
spdlog::info("Shutting down session after {} of inactivity", timeout_seconds_.count());
DoShutdown();
} else {
// Put the actor back to sleep.

View File

@@ -0,0 +1,9 @@
set(coordinator_src_files
coordinator.cpp
shard_map.cpp)
find_package(fmt REQUIRED)
find_package(Threads REQUIRED)
add_library(mg-coordinator STATIC ${coordinator_src_files})
target_link_libraries(mg-coordinator stdc++fs Threads::Threads fmt::fmt mg-utils mg-storage-v3)

View File

@@ -0,0 +1,129 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <coordinator/coordinator.hpp>
namespace memgraph::coordinator {
CoordinatorWriteResponses Coordinator::ApplyWrite(HeartbeatRequest &&heartbeat_request) {
spdlog::info("Coordinator handling HeartbeatRequest");
// add this storage engine to any under-replicated shards that it is not already a part of
auto initializing_rsms_for_shard_manager =
shard_map_.AssignShards(heartbeat_request.from_storage_manager, heartbeat_request.initialized_rsms);
return HeartbeatResponse{
.shards_to_initialize = initializing_rsms_for_shard_manager,
};
}
CoordinatorWriteResponses Coordinator::ApplyWrite(HlcRequest &&hlc_request) {
HlcResponse res{};
auto hlc_shard_map = shard_map_.GetHlc();
MG_ASSERT(!(hlc_request.last_shard_map_version.logical_id > hlc_shard_map.logical_id));
res.new_hlc = Hlc{
.logical_id = ++highest_allocated_timestamp_,
// TODO(tyler) probably pass some more context to the Coordinator here
// so that we can use our wall clock and enforce monotonicity.
// .coordinator_wall_clock = io_.Now(),
};
// Allways return fresher shard_map for now.
res.fresher_shard_map = std::make_optional(shard_map_);
return res;
}
CoordinatorWriteResponses Coordinator::ApplyWrite(AllocateEdgeIdBatchRequest &&ahr) {
AllocateEdgeIdBatchResponse res{};
uint64_t low = highest_allocated_edge_id_;
highest_allocated_edge_id_ += ahr.batch_size;
uint64_t high = highest_allocated_edge_id_;
res.low = low;
res.high = high;
return res;
}
/// This splits the shard immediately beneath the provided
/// split key, keeping the assigned peers identical for now,
/// but letting them be gradually migrated over time.
CoordinatorWriteResponses Coordinator::ApplyWrite(SplitShardRequest &&split_shard_request) {
SplitShardResponse res{};
if (split_shard_request.previous_shard_map_version != shard_map_.shard_map_version) {
res.success = false;
} else {
res.success = shard_map_.SplitShard(split_shard_request.previous_shard_map_version, split_shard_request.label_id,
split_shard_request.split_key);
}
return res;
}
/// This adds the provided storage engine to the standby storage engine pool,
/// which can be used to rebalance storage over time.
CoordinatorWriteResponses Coordinator::ApplyWrite(
RegisterStorageEngineRequest && /* register_storage_engine_request */) {
RegisterStorageEngineResponse res{};
// TODO
return res;
}
/// This begins the process of draining the provided storage engine from all raft
/// clusters that it might be participating in.
CoordinatorWriteResponses Coordinator::ApplyWrite(
DeregisterStorageEngineRequest && /* register_storage_engine_request */) {
DeregisterStorageEngineResponse res{};
// TODO
return res;
}
CoordinatorWriteResponses Coordinator::ApplyWrite(InitializeLabelRequest &&initialize_label_request) {
InitializeLabelResponse res{};
std::optional<LabelId> new_label_id = shard_map_.InitializeNewLabel(
initialize_label_request.label_name, initialize_label_request.schema, initialize_label_request.replication_factor,
initialize_label_request.last_shard_map_version);
if (new_label_id) {
res.new_label_id = new_label_id.value();
res.fresher_shard_map = std::nullopt;
res.success = true;
} else {
res.fresher_shard_map = shard_map_;
res.success = false;
}
return res;
}
CoordinatorWriteResponses Coordinator::ApplyWrite(AllocatePropertyIdsRequest &&allocate_property_ids_request) {
AllocatePropertyIdsResponse res{};
auto property_ids = shard_map_.AllocatePropertyIds(allocate_property_ids_request.property_names);
res.property_ids = property_ids;
return res;
}
} // namespace memgraph::coordinator

View File

@@ -0,0 +1,196 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <optional>
#include <set>
#include <string>
#include <unordered_set>
#include <variant>
#include <vector>
#include <boost/uuid/uuid.hpp>
#include <coordinator/hybrid_logical_clock.hpp>
#include <coordinator/shard_map.hpp>
#include <io/simulator/simulator.hpp>
#include <io/time.hpp>
#include <io/transport.hpp>
#include <storage/v3/id_types.hpp>
#include <storage/v3/schemas.hpp>
namespace memgraph::coordinator {
using memgraph::io::Address;
using memgraph::storage::v3::LabelId;
using memgraph::storage::v3::PropertyId;
using memgraph::storage::v3::SchemaProperty;
using SimT = memgraph::io::simulator::SimulatorTransport;
using PrimaryKey = std::vector<PropertyValue>;
struct HlcRequest {
Hlc last_shard_map_version;
};
struct HlcResponse {
Hlc new_hlc;
std::optional<ShardMap> fresher_shard_map;
};
struct GetShardMapRequest {
// No state
};
struct GetShardMapResponse {
ShardMap shard_map;
};
struct AllocateHlcBatchRequest {
Hlc low;
Hlc high;
};
struct AllocateHlcBatchResponse {
bool success;
Hlc low;
Hlc high;
};
struct AllocateEdgeIdBatchRequest {
size_t batch_size;
};
struct AllocateEdgeIdBatchResponse {
uint64_t low;
uint64_t high;
};
struct AllocatePropertyIdsRequest {
std::vector<std::string> property_names;
};
struct AllocatePropertyIdsResponse {
std::map<std::string, PropertyId> property_ids;
};
struct SplitShardRequest {
Hlc previous_shard_map_version;
LabelId label_id;
PrimaryKey split_key;
};
struct SplitShardResponse {
bool success;
};
struct RegisterStorageEngineRequest {
Address address;
};
struct RegisterStorageEngineResponse {
bool success;
};
struct DeregisterStorageEngineRequest {
Address address;
};
struct DeregisterStorageEngineResponse {
bool success;
};
struct InitializeLabelRequest {
std::string label_name;
std::vector<SchemaProperty> schema;
size_t replication_factor;
Hlc last_shard_map_version;
};
struct InitializeLabelResponse {
bool success;
LabelId new_label_id;
std::optional<ShardMap> fresher_shard_map;
};
struct HeartbeatRequest {
Address from_storage_manager;
std::set<boost::uuids::uuid> initialized_rsms;
};
struct HeartbeatResponse {
std::vector<ShardToInitialize> shards_to_initialize;
};
using CoordinatorWriteRequests =
std::variant<HlcRequest, AllocateEdgeIdBatchRequest, SplitShardRequest, RegisterStorageEngineRequest,
DeregisterStorageEngineRequest, InitializeLabelRequest, AllocatePropertyIdsRequest, HeartbeatRequest>;
using CoordinatorWriteResponses = std::variant<HlcResponse, AllocateEdgeIdBatchResponse, SplitShardResponse,
RegisterStorageEngineResponse, DeregisterStorageEngineResponse,
InitializeLabelResponse, AllocatePropertyIdsResponse, HeartbeatResponse>;
using CoordinatorReadRequests = std::variant<GetShardMapRequest>;
using CoordinatorReadResponses = std::variant<GetShardMapResponse>;
class Coordinator {
public:
explicit Coordinator(ShardMap sm) : shard_map_{std::move(sm)} {}
// NOLINTNEXTLINE(readability-convert-member-functions-to-static
CoordinatorReadResponses Read(CoordinatorReadRequests requests) {
return std::visit([&](auto &&request) { return HandleRead(std::forward<decltype(request)>(request)); },
std::move(requests)); // NOLINT(hicpp-move-const-arg,performance-move-const-arg)
}
// NOLINTNEXTLINE(readability-convert-member-functions-to-static
CoordinatorWriteResponses Apply(CoordinatorWriteRequests requests) {
return std::visit([&](auto &&request) mutable { return ApplyWrite(std::forward<decltype(request)>(request)); },
std::move(requests));
}
private:
ShardMap shard_map_;
uint64_t highest_allocated_timestamp_{0};
/// Query engines need to periodically request batches of unique edge IDs.
uint64_t highest_allocated_edge_id_{0};
CoordinatorReadResponses HandleRead(GetShardMapRequest && /* get_shard_map_request */) {
GetShardMapResponse res;
res.shard_map = shard_map_;
return res;
}
CoordinatorWriteResponses ApplyWrite(HeartbeatRequest &&heartbeat_request);
CoordinatorWriteResponses ApplyWrite(HlcRequest &&hlc_request);
CoordinatorWriteResponses ApplyWrite(AllocateEdgeIdBatchRequest &&ahr);
/// This splits the shard immediately beneath the provided
/// split key, keeping the assigned peers identical for now,
/// but letting them be gradually migrated over time.
CoordinatorWriteResponses ApplyWrite(SplitShardRequest &&split_shard_request);
/// This adds the provided storage engine to the standby storage engine pool,
/// which can be used to rebalance storage over time.
static CoordinatorWriteResponses ApplyWrite(RegisterStorageEngineRequest && /* register_storage_engine_request */);
/// This begins the process of draining the provided storage engine from all raft
/// clusters that it might be participating in.
static CoordinatorWriteResponses ApplyWrite(DeregisterStorageEngineRequest && /* register_storage_engine_request */);
CoordinatorWriteResponses ApplyWrite(InitializeLabelRequest &&initialize_label_request);
CoordinatorWriteResponses ApplyWrite(AllocatePropertyIdsRequest &&allocate_property_ids_request);
};
} // namespace memgraph::coordinator

View File

@@ -0,0 +1,24 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "coordinator/coordinator.hpp"
#include "io/rsm/rsm_client.hpp"
namespace memgraph::coordinator {
using memgraph::io::rsm::RsmClient;
template <typename IoImpl>
using CoordinatorClient = RsmClient<IoImpl, CoordinatorWriteRequests, CoordinatorWriteResponses,
CoordinatorReadRequests, CoordinatorReadResponses>;
} // namespace memgraph::coordinator

View File

@@ -0,0 +1,23 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "coordinator/coordinator.hpp"
#include "io/rsm/raft.hpp"
namespace memgraph::coordinator {
template <typename IoImpl>
using CoordinatorRsm = memgraph::io::rsm::Raft<IoImpl, Coordinator, CoordinatorWriteRequests, CoordinatorWriteResponses,
CoordinatorReadRequests, CoordinatorReadResponses>;
} // namespace memgraph::coordinator

View File

@@ -0,0 +1,168 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <chrono>
#include <deque>
#include <memory>
#include <queue>
#include <variant>
#include "coordinator/coordinator.hpp"
#include "coordinator/coordinator_rsm.hpp"
#include "coordinator/shard_map.hpp"
#include "io/address.hpp"
#include "io/future.hpp"
#include "io/messages.hpp"
#include "io/rsm/raft.hpp"
#include "io/time.hpp"
#include "io/transport.hpp"
#include "query/v2/requests.hpp"
namespace memgraph::coordinator::coordinator_worker {
/// Obligations:
/// * ShutDown
/// * Cron
/// * RouteMessage
using coordinator::Coordinator;
using coordinator::CoordinatorRsm;
using io::Address;
using io::RequestId;
using io::Time;
using io::messages::CoordinatorMessages;
using msgs::ReadRequests;
using msgs::ReadResponses;
using msgs::WriteRequests;
using msgs::WriteResponses;
struct ShutDown {};
struct Cron {};
struct RouteMessage {
CoordinatorMessages message;
RequestId request_id;
Address to;
Address from;
};
using Message = std::variant<RouteMessage, Cron, ShutDown>;
struct QueueInner {
std::mutex mu{};
std::condition_variable cv;
// TODO(tyler) handle simulator communication std::shared_ptr<std::atomic<int>> blocked;
// TODO(tyler) investigate using a priority queue that prioritizes messages in a way that
// improves overall QoS. For example, maybe we want to schedule raft Append messages
// ahead of Read messages or generally writes before reads for lowering the load on the
// overall system faster etc... When we do this, we need to make sure to avoid
// starvation by sometimes randomizing priorities, rather than following a strict
// prioritization.
std::deque<Message> queue;
uint64_t submitted = 0;
uint64_t calls_to_pop = 0;
};
/// There are two reasons to implement our own Queue instead of using
/// one off-the-shelf:
/// 1. we will need to know in the simulator when all threads are waiting
/// 2. we will want to implement our own priority queue within this for QoS
class Queue {
std::shared_ptr<QueueInner> inner_ = std::make_shared<QueueInner>();
public:
void Push(Message &&message) {
{
MG_ASSERT(inner_.use_count() > 0);
std::unique_lock<std::mutex> lock(inner_->mu);
inner_->submitted++;
inner_->queue.emplace_back(std::move(message));
} // lock dropped before notifying condition variable
inner_->cv.notify_all();
}
Message Pop() {
MG_ASSERT(inner_.use_count() > 0);
std::unique_lock<std::mutex> lock(inner_->mu);
inner_->calls_to_pop++;
inner_->cv.notify_all();
while (inner_->queue.empty()) {
inner_->cv.wait(lock);
}
Message message = std::move(inner_->queue.front());
inner_->queue.pop_front();
return message;
}
void BlockOnQuiescence() const {
MG_ASSERT(inner_.use_count() > 0);
std::unique_lock<std::mutex> lock(inner_->mu);
while (inner_->calls_to_pop <= inner_->submitted) {
inner_->cv.wait(lock);
}
}
};
/// A CoordinatorWorker owns Raft<CoordinatorRsm> instances. receives messages from the MachineManager.
template <typename IoImpl>
class CoordinatorWorker {
io::Io<IoImpl> io_;
Queue queue_;
CoordinatorRsm<IoImpl> coordinator_;
bool Process(ShutDown && /*shut_down*/) { return false; }
bool Process(Cron && /* cron */) {
coordinator_.Cron();
return true;
}
bool Process(RouteMessage &&route_message) {
coordinator_.Handle(std::move(route_message.message), route_message.request_id, route_message.from);
return true;
}
public:
CoordinatorWorker(io::Io<IoImpl> io, Queue queue, Coordinator coordinator)
: io_(std::move(io)), queue_(std::move(queue)), coordinator_{std::move(io_), {}, std::move(coordinator)} {}
CoordinatorWorker(CoordinatorWorker &&) noexcept = default;
CoordinatorWorker &operator=(CoordinatorWorker &&) noexcept = default;
CoordinatorWorker(const CoordinatorWorker &) = delete;
CoordinatorWorker &operator=(const CoordinatorWorker &) = delete;
~CoordinatorWorker() = default;
void Run() {
bool should_continue = true;
while (should_continue) {
Message message = queue_.Pop();
should_continue = std::visit([this](auto &&msg) { return this->Process(std::forward<decltype(msg)>(msg)); },
std::move(message));
}
}
};
} // namespace memgraph::coordinator::coordinator_worker

View File

@@ -0,0 +1,48 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <chrono>
#include <compare>
#include <ctime>
#include <iomanip>
#include "io/time.hpp"
namespace memgraph::coordinator {
using Time = memgraph::io::Time;
/// Hybrid-logical clock
struct Hlc {
uint64_t logical_id = 0;
Time coordinator_wall_clock = Time::min();
auto operator<=>(const Hlc &other) const { return logical_id <=> other.logical_id; }
bool operator==(const Hlc &other) const = default;
bool operator<(const Hlc &other) const = default;
bool operator==(const uint64_t other) const { return logical_id == other; }
bool operator<(const uint64_t other) const { return logical_id < other; }
bool operator>=(const uint64_t other) const { return logical_id >= other; }
friend std::ostream &operator<<(std::ostream &in, const Hlc &hlc) {
auto wall_clock = std::chrono::system_clock::to_time_t(hlc.coordinator_wall_clock);
in << "Hlc { logical_id: " << hlc.logical_id;
in << ", coordinator_wall_clock: " << std::put_time(std::localtime(&wall_clock), "%F %T");
in << " }";
return in;
}
};
} // namespace memgraph::coordinator

View File

@@ -0,0 +1,576 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <optional>
#include <unordered_map>
#include <vector>
#include "common/types.hpp"
#include "coordinator/shard_map.hpp"
#include "spdlog/spdlog.h"
#include "storage/v3/schemas.hpp"
#include "storage/v3/temporal.hpp"
#include "utils/cast.hpp"
#include "utils/exceptions.hpp"
#include "utils/string.hpp"
namespace memgraph::coordinator {
using memgraph::common::SchemaType;
using memgraph::storage::v3::TemporalData;
using memgraph::storage::v3::TemporalType;
PrimaryKey SchemaToMinKey(const std::vector<SchemaProperty> &schema) {
PrimaryKey ret{};
const int64_t min_int = std::numeric_limits<int64_t>::min();
const TemporalData date{TemporalType::Date, min_int};
const TemporalData local_time{TemporalType::LocalTime, min_int};
const TemporalData local_date_time{TemporalType::LocalDateTime, min_int};
const TemporalData duration{TemporalType::Duration, min_int};
for (const auto &schema_property : schema) {
switch (schema_property.type) {
case SchemaType::BOOL:
ret.emplace_back(PropertyValue(false));
break;
case SchemaType::INT:
ret.emplace_back(PropertyValue(min_int));
break;
case SchemaType::STRING:
ret.emplace_back(PropertyValue(""));
break;
case SchemaType::DATE:
ret.emplace_back(PropertyValue(date));
break;
case SchemaType::LOCALTIME:
ret.emplace_back(PropertyValue(local_time));
break;
case SchemaType::LOCALDATETIME:
ret.emplace_back(PropertyValue(local_date_time));
break;
case SchemaType::DURATION:
ret.emplace_back(PropertyValue(duration));
break;
}
}
return ret;
}
ShardMap ShardMap::Parse(std::istream &input_stream) {
ShardMap shard_map;
const auto read_size = [&input_stream] {
size_t size{0};
input_stream >> size;
return size;
};
// Reads a string until the next whitespace
const auto read_word = [&input_stream] {
std::string word;
input_stream >> word;
return word;
};
const auto read_names = [&read_size, &read_word] {
const auto number_of_names = read_size();
spdlog::trace("Reading {} names", number_of_names);
std::vector<std::string> names;
names.reserve(number_of_names);
for (auto name_index = 0; name_index < number_of_names; ++name_index) {
names.push_back(read_word());
spdlog::trace("Read '{}'", names.back());
}
return names;
};
const auto read_line = [&input_stream] {
std::string line;
std::getline(input_stream, line);
return line;
};
const auto parse_type = [](const std::string &type) {
static const auto type_map = std::unordered_map<std::string, common::SchemaType>{
{"string", common::SchemaType::STRING}, {"int", common::SchemaType::INT}, {"bool", common::SchemaType::BOOL}};
const auto lower_case_type = utils::ToLowerCase(type);
auto it = type_map.find(lower_case_type);
MG_ASSERT(it != type_map.end(), "Invalid type in split files: {}", type);
return it->second;
};
const auto parse_property_value = [](std::string text, const common::SchemaType type) {
if (type == common::SchemaType::STRING) {
return storage::v3::PropertyValue{std::move(text)};
}
if (type == common::SchemaType::INT) {
size_t processed{0};
int64_t value = std::stoll(text, &processed);
MG_ASSERT(processed == text.size() || text[processed] == ' ', "Invalid integer format: '{}'", text);
return storage::v3::PropertyValue{value};
}
LOG_FATAL("Not supported type: {}", utils::UnderlyingCast(type));
};
spdlog::debug("Reading properties");
const auto properties = read_names();
MG_ASSERT(shard_map.AllocatePropertyIds(properties).size() == properties.size(),
"Unexpected number of properties created!");
spdlog::debug("Reading edge types");
const auto edge_types = read_names();
MG_ASSERT(shard_map.AllocateEdgeTypeIds(edge_types).size() == edge_types.size(),
"Unexpected number of properties created!");
spdlog::debug("Reading primary labels");
const auto number_of_primary_labels = read_size();
spdlog::debug("Reading {} primary labels", number_of_primary_labels);
for (auto label_index = 0; label_index < number_of_primary_labels; ++label_index) {
const auto primary_label = read_word();
spdlog::debug("Reading primary label named '{}'", primary_label);
const auto number_of_primary_properties = read_size();
spdlog::debug("Reading {} primary properties", number_of_primary_properties);
std::vector<std::string> pp_names;
std::vector<common::SchemaType> pp_types;
pp_names.reserve(number_of_primary_properties);
pp_types.reserve(number_of_primary_properties);
for (auto property_index = 0; property_index < number_of_primary_properties; ++property_index) {
pp_names.push_back(read_word());
spdlog::debug("Reading primary property named '{}'", pp_names.back());
pp_types.push_back(parse_type(read_word()));
}
auto pp_mapping = shard_map.AllocatePropertyIds(pp_names);
std::vector<SchemaProperty> schema;
schema.reserve(number_of_primary_properties);
for (auto property_index = 0; property_index < number_of_primary_properties; ++property_index) {
schema.push_back(storage::v3::SchemaProperty{pp_mapping.at(pp_names[property_index]), pp_types[property_index]});
}
const auto hlc = shard_map.GetHlc();
MG_ASSERT(shard_map.InitializeNewLabel(primary_label, schema, 1, hlc).has_value(),
"Cannot initialize new label: {}", primary_label);
const auto number_of_split_points = read_size();
spdlog::debug("Reading {} split points", number_of_split_points);
[[maybe_unused]] const auto remainder_from_last_line = read_line();
for (auto split_point_index = 0; split_point_index < number_of_split_points; ++split_point_index) {
const auto line = read_line();
spdlog::debug("Read split point '{}'", line);
MG_ASSERT(line.front() == '[', "Invalid split file format!");
MG_ASSERT(line.back() == ']', "Invalid split file format!");
std::string_view line_view{line};
line_view.remove_prefix(1);
line_view.remove_suffix(1);
static constexpr std::string_view kDelimiter{","};
auto pk_values_as_text = utils::Split(line_view, kDelimiter);
std::vector<PropertyValue> pk;
pk.reserve(number_of_primary_properties);
MG_ASSERT(pk_values_as_text.size() == number_of_primary_properties,
"Split point contains invalid number of values '{}'", line);
for (auto property_index = 0; property_index < number_of_primary_properties; ++property_index) {
pk.push_back(parse_property_value(std::move(pk_values_as_text[property_index]), schema[property_index].type));
}
shard_map.SplitShard(shard_map.GetHlc(), shard_map.labels.at(primary_label), pk);
}
}
return shard_map;
}
std::ostream &operator<<(std::ostream &in, const ShardMap &shard_map) {
using utils::print_helpers::operator<<;
in << "ShardMap { shard_map_version: " << shard_map.shard_map_version;
in << ", max_property_id: " << shard_map.max_property_id;
in << ", max_edge_type_id: " << shard_map.max_edge_type_id;
in << ", properties: " << shard_map.properties;
in << ", edge_types: " << shard_map.edge_types;
in << ", max_label_id: " << shard_map.max_label_id;
in << ", labels: " << shard_map.labels;
in << ", label_spaces: " << shard_map.label_spaces;
in << ", schemas: " << shard_map.schemas;
in << "}";
return in;
}
Shards ShardMap::GetShardsForLabel(const LabelName &label) const {
const auto id = labels.at(label);
const auto &shards = label_spaces.at(id).shards;
return shards;
}
std::vector<Shards> ShardMap::GetAllShards() const {
std::vector<Shards> all_shards;
all_shards.reserve(label_spaces.size());
std::transform(label_spaces.begin(), label_spaces.end(), std::back_inserter(all_shards),
[](const auto &label_space) { return label_space.second.shards; });
return all_shards;
}
// TODO(gabor) later we will want to update the wallclock time with
// the given Io<impl>'s time as well
Hlc ShardMap::IncrementShardMapVersion() noexcept {
++shard_map_version.logical_id;
return shard_map_version;
}
// TODO(antaljanosbenjamin) use a single map for all name id
// mapping and a single counter to maintain the next id
std::unordered_map<uint64_t, std::string> ShardMap::IdToNames() {
std::unordered_map<uint64_t, std::string> id_to_names;
const auto map_type_ids = [&id_to_names](const auto &name_to_id_type) {
for (const auto &[name, id] : name_to_id_type) {
id_to_names.emplace(id.AsUint(), name);
}
};
map_type_ids(edge_types);
map_type_ids(labels);
map_type_ids(properties);
return id_to_names;
}
Hlc ShardMap::GetHlc() const noexcept { return shard_map_version; }
boost::uuids::uuid NewShardUuid(uint64_t shard_id) {
return boost::uuids::uuid{0,
0,
0,
0,
0,
0,
0,
0,
static_cast<unsigned char>(shard_id >> 56U),
static_cast<unsigned char>(shard_id >> 48U),
static_cast<unsigned char>(shard_id >> 40U),
static_cast<unsigned char>(shard_id >> 32U),
static_cast<unsigned char>(shard_id >> 24U),
static_cast<unsigned char>(shard_id >> 16U),
static_cast<unsigned char>(shard_id >> 8U),
static_cast<unsigned char>(shard_id)};
}
std::vector<ShardToInitialize> ShardMap::AssignShards(Address storage_manager,
std::set<boost::uuids::uuid> initialized) {
std::vector<ShardToInitialize> ret{};
bool mutated = false;
for (auto &[label_id, label_space] : label_spaces) {
for (auto it = label_space.shards.begin(); it != label_space.shards.end(); it++) {
auto &[low_key, shard] = *it;
std::optional<PrimaryKey> high_key;
if (const auto next_it = std::next(it); next_it != label_space.shards.end()) {
high_key = next_it->first;
}
// TODO(tyler) avoid these triple-nested loops by having the heartbeat include better info
bool machine_contains_shard = false;
for (auto &aas : shard) {
if (initialized.contains(aas.address.unique_id)) {
machine_contains_shard = true;
if (aas.status != Status::CONSENSUS_PARTICIPANT) {
mutated = true;
spdlog::info("marking shard as full consensus participant: {}", aas.address.unique_id);
aas.status = Status::CONSENSUS_PARTICIPANT;
}
} else {
const bool same_machine = aas.address.last_known_ip == storage_manager.last_known_ip &&
aas.address.last_known_port == storage_manager.last_known_port;
if (same_machine) {
machine_contains_shard = true;
spdlog::info("reminding shard manager that they should begin participating in shard");
ret.push_back(ShardToInitialize{
.uuid = aas.address.unique_id,
.label_id = label_id,
.min_key = low_key,
.max_key = high_key,
.schema = schemas[label_id],
.config = Config{},
.id_to_names = IdToNames(),
});
}
}
}
if (!machine_contains_shard && shard.size() < label_space.replication_factor) {
// increment version for each new uuid for deterministic creation
IncrementShardMapVersion();
Address address = storage_manager;
// TODO(tyler) use deterministic UUID so that coordinators don't diverge here
address.unique_id = NewShardUuid(shard_map_version.logical_id);
spdlog::info("assigning shard manager to shard");
ret.push_back(ShardToInitialize{
.uuid = address.unique_id,
.label_id = label_id,
.min_key = low_key,
.max_key = high_key,
.schema = schemas[label_id],
.config = Config{},
.id_to_names = IdToNames(),
});
AddressAndStatus aas = {
.address = address,
.status = Status::INITIALIZING,
};
shard.emplace_back(aas);
}
}
}
if (mutated) {
IncrementShardMapVersion();
}
return ret;
}
bool ShardMap::SplitShard(Hlc previous_shard_map_version, LabelId label_id, const PrimaryKey &key) {
if (previous_shard_map_version != shard_map_version) {
return false;
}
auto &label_space = label_spaces.at(label_id);
auto &shards_in_map = label_space.shards;
MG_ASSERT(!shards_in_map.empty());
MG_ASSERT(!shards_in_map.contains(key));
MG_ASSERT(label_spaces.contains(label_id));
// Finding the Shard that the new PrimaryKey should map to.
auto prev = std::prev(shards_in_map.upper_bound(key));
Shard duplicated_shard = prev->second;
// Apply the split
shards_in_map[key] = duplicated_shard;
IncrementShardMapVersion();
return true;
}
std::optional<LabelId> ShardMap::InitializeNewLabel(std::string label_name, std::vector<SchemaProperty> schema,
size_t replication_factor, Hlc last_shard_map_version) {
if (shard_map_version != last_shard_map_version || labels.contains(label_name)) {
return std::nullopt;
}
const LabelId label_id = LabelId::FromUint(++max_label_id);
labels.emplace(std::move(label_name), label_id);
PrimaryKey initial_key = SchemaToMinKey(schema);
Shard empty_shard = {};
Shards shards = {
{initial_key, empty_shard},
};
LabelSpace label_space{
.schema = schema,
.shards = shards,
.replication_factor = replication_factor,
};
schemas[label_id] = std::move(schema);
label_spaces.emplace(label_id, label_space);
IncrementShardMapVersion();
return label_id;
}
void ShardMap::AddServer(Address server_address) {
// Find a random place for the server to plug in
}
std::optional<LabelId> ShardMap::GetLabelId(const std::string &label) const {
if (const auto it = labels.find(label); it != labels.end()) {
return it->second;
}
return std::nullopt;
}
const std::string &ShardMap::GetLabelName(const LabelId label) const {
if (const auto it =
std::ranges::find_if(labels, [label](const auto &name_id_pair) { return name_id_pair.second == label; });
it != labels.end()) {
return it->first;
}
throw utils::BasicException("GetLabelName fails on the given label id!");
}
std::optional<PropertyId> ShardMap::GetPropertyId(const std::string &property_name) const {
if (const auto it = properties.find(property_name); it != properties.end()) {
return it->second;
}
return std::nullopt;
}
const std::string &ShardMap::GetPropertyName(const PropertyId property) const {
if (const auto it = std::ranges::find_if(
properties, [property](const auto &name_id_pair) { return name_id_pair.second == property; });
it != properties.end()) {
return it->first;
}
throw utils::BasicException("PropertyId not found!");
}
std::optional<EdgeTypeId> ShardMap::GetEdgeTypeId(const std::string &edge_type) const {
if (const auto it = edge_types.find(edge_type); it != edge_types.end()) {
return it->second;
}
return std::nullopt;
}
const std::string &ShardMap::GetEdgeTypeName(const EdgeTypeId property) const {
if (const auto it = std::ranges::find_if(
edge_types, [property](const auto &name_id_pair) { return name_id_pair.second == property; });
it != edge_types.end()) {
return it->first;
}
throw utils::BasicException("EdgeTypeId not found!");
}
Shards ShardMap::GetShardsForRange(const LabelName &label_name, const PrimaryKey &start_key,
const PrimaryKey &end_key) const {
MG_ASSERT(start_key <= end_key);
MG_ASSERT(labels.contains(label_name));
LabelId label_id = labels.at(label_name);
const auto &label_space = label_spaces.at(label_id);
const auto &shards_for_label = label_space.shards;
MG_ASSERT(shards_for_label.begin()->first <= start_key,
"the ShardMap must always contain a minimal key that is less than or equal to any requested key");
auto it = std::prev(shards_for_label.upper_bound(start_key));
const auto end_it = shards_for_label.upper_bound(end_key);
Shards shards{};
std::copy(it, end_it, std::inserter(shards, shards.end()));
return shards;
}
Shard ShardMap::GetShardForKey(const LabelName &label_name, const PrimaryKey &key) const {
MG_ASSERT(labels.contains(label_name));
LabelId label_id = labels.at(label_name);
const auto &label_space = label_spaces.at(label_id);
MG_ASSERT(label_space.shards.begin()->first <= key,
"the ShardMap must always contain a minimal key that is less than or equal to any requested key");
return std::prev(label_space.shards.upper_bound(key))->second;
}
Shard ShardMap::GetShardForKey(const LabelId &label_id, const PrimaryKey &key) const {
MG_ASSERT(label_spaces.contains(label_id));
const auto &label_space = label_spaces.at(label_id);
MG_ASSERT(label_space.shards.begin()->first <= key,
"the ShardMap must always contain a minimal key that is less than or equal to any requested key");
return std::prev(label_space.shards.upper_bound(key))->second;
}
PropertyMap ShardMap::AllocatePropertyIds(const std::vector<PropertyName> &new_properties) {
PropertyMap ret{};
bool mutated = false;
for (const auto &property_name : new_properties) {
if (properties.contains(property_name)) {
auto property_id = properties.at(property_name);
ret.emplace(property_name, property_id);
} else {
mutated = true;
const PropertyId property_id = PropertyId::FromUint(++max_property_id);
ret.emplace(property_name, property_id);
properties.emplace(property_name, property_id);
}
}
if (mutated) {
IncrementShardMapVersion();
}
return ret;
}
EdgeTypeIdMap ShardMap::AllocateEdgeTypeIds(const std::vector<EdgeTypeName> &new_edge_types) {
EdgeTypeIdMap ret;
bool mutated = false;
for (const auto &edge_type_name : new_edge_types) {
if (edge_types.contains(edge_type_name)) {
auto edge_type_id = edge_types.at(edge_type_name);
ret.emplace(edge_type_name, edge_type_id);
} else {
mutated = true;
const EdgeTypeId edge_type_id = EdgeTypeId::FromUint(++max_edge_type_id);
ret.emplace(edge_type_name, edge_type_id);
edge_types.emplace(edge_type_name, edge_type_id);
}
}
if (mutated) {
IncrementShardMapVersion();
}
return ret;
}
bool ShardMap::ClusterInitialized() const {
for (const auto &[label_id, label_space] : label_spaces) {
for (const auto &[low_key, shard] : label_space.shards) {
if (shard.size() < label_space.replication_factor) {
spdlog::info("label_space below desired replication factor");
return false;
}
for (const auto &aas : shard) {
if (aas.status != Status::CONSENSUS_PARTICIPANT) {
spdlog::info("shard member not yet a CONSENSUS_PARTICIPANT");
return false;
}
}
}
}
return true;
}
} // namespace memgraph::coordinator

View File

@@ -0,0 +1,177 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <algorithm>
#include <limits>
#include <map>
#include <set>
#include <vector>
#include <boost/uuid/uuid.hpp>
#include <boost/uuid/uuid_generators.hpp>
#include "common/types.hpp"
#include "coordinator/hybrid_logical_clock.hpp"
#include "io/address.hpp"
#include "storage/v3/config.hpp"
#include "storage/v3/id_types.hpp"
#include "storage/v3/name_id_mapper.hpp"
#include "storage/v3/property_value.hpp"
#include "storage/v3/schemas.hpp"
#include "storage/v3/temporal.hpp"
#include "utils/exceptions.hpp"
#include "utils/print_helpers.hpp"
namespace memgraph::coordinator {
constexpr int64_t kNotExistingId{0};
using memgraph::io::Address;
using memgraph::storage::v3::Config;
using memgraph::storage::v3::EdgeTypeId;
using memgraph::storage::v3::LabelId;
using memgraph::storage::v3::PropertyId;
using memgraph::storage::v3::PropertyValue;
using memgraph::storage::v3::SchemaProperty;
enum class Status : uint8_t {
CONSENSUS_PARTICIPANT,
INITIALIZING,
// TODO(tyler) this will possibly have more states,
// depending on the reconfiguration protocol that we
// implement.
};
struct AddressAndStatus {
memgraph::io::Address address;
Status status;
friend bool operator<(const AddressAndStatus &lhs, const AddressAndStatus &rhs) { return lhs.address < rhs.address; }
friend std::ostream &operator<<(std::ostream &in, const AddressAndStatus &address_and_status) {
in << "AddressAndStatus { address: ";
in << address_and_status.address;
if (address_and_status.status == Status::CONSENSUS_PARTICIPANT) {
in << ", status: CONSENSUS_PARTICIPANT }";
} else {
in << ", status: INITIALIZING }";
}
return in;
}
friend bool operator==(const AddressAndStatus &lhs, const AddressAndStatus &rhs) {
return lhs.address == rhs.address;
}
};
using PrimaryKey = std::vector<PropertyValue>;
using Shard = std::vector<AddressAndStatus>;
using Shards = std::map<PrimaryKey, Shard>;
using LabelName = std::string;
using PropertyName = std::string;
using EdgeTypeName = std::string;
using PropertyMap = std::map<PropertyName, PropertyId>;
using EdgeTypeIdMap = std::map<EdgeTypeName, EdgeTypeId>;
struct ShardToInitialize {
boost::uuids::uuid uuid;
LabelId label_id;
PrimaryKey min_key;
std::optional<PrimaryKey> max_key;
std::vector<SchemaProperty> schema;
Config config;
std::unordered_map<uint64_t, std::string> id_to_names;
};
PrimaryKey SchemaToMinKey(const std::vector<SchemaProperty> &schema);
struct LabelSpace {
std::vector<SchemaProperty> schema;
// Maps between the smallest primary key stored in the shard and the shard
std::map<PrimaryKey, Shard> shards;
size_t replication_factor;
friend std::ostream &operator<<(std::ostream &in, const LabelSpace &label_space) {
using utils::print_helpers::operator<<;
in << "LabelSpace { schema: ";
in << label_space.schema;
in << ", shards: ";
in << label_space.shards;
in << ", replication_factor: " << label_space.replication_factor << "}";
return in;
}
};
struct ShardMap {
Hlc shard_map_version;
uint64_t max_property_id{kNotExistingId};
uint64_t max_edge_type_id{kNotExistingId};
std::map<PropertyName, PropertyId> properties;
std::map<EdgeTypeName, EdgeTypeId> edge_types;
uint64_t max_label_id{kNotExistingId};
std::map<LabelId, LabelSpace> label_spaces;
std::map<LabelId, std::vector<SchemaProperty>> schemas;
std::map<LabelName, LabelId> labels;
[[nodiscard]] static ShardMap Parse(std::istream &input_stream);
friend std::ostream &operator<<(std::ostream &in, const ShardMap &shard_map);
Shards GetShardsForLabel(const LabelName &label) const;
std::vector<Shards> GetAllShards() const;
// TODO(gabor) later we will want to update the wallclock time with
// the given Io<impl>'s time as well
Hlc IncrementShardMapVersion() noexcept;
Hlc GetHlc() const noexcept;
std::unordered_map<uint64_t, std::string> IdToNames();
// Returns the shard UUIDs that have been assigned but not yet acknowledged for this storage manager
std::vector<ShardToInitialize> AssignShards(Address storage_manager, std::set<boost::uuids::uuid> initialized);
bool SplitShard(Hlc previous_shard_map_version, LabelId label_id, const PrimaryKey &key);
std::optional<LabelId> InitializeNewLabel(std::string label_name, std::vector<SchemaProperty> schema,
size_t replication_factor, Hlc last_shard_map_version);
void AddServer(Address server_address);
std::optional<LabelId> GetLabelId(const std::string &label) const;
// TODO(antaljanosbenjamin): Remove this and instead use NameIdMapper
const std::string &GetLabelName(LabelId label) const;
std::optional<PropertyId> GetPropertyId(const std::string &property_name) const;
const std::string &GetPropertyName(PropertyId property) const;
std::optional<EdgeTypeId> GetEdgeTypeId(const std::string &edge_type) const;
const std::string &GetEdgeTypeName(EdgeTypeId property) const;
Shards GetShardsForRange(const LabelName &label_name, const PrimaryKey &start_key, const PrimaryKey &end_key) const;
Shard GetShardForKey(const LabelName &label_name, const PrimaryKey &key) const;
Shard GetShardForKey(const LabelId &label_id, const PrimaryKey &key) const;
PropertyMap AllocatePropertyIds(const std::vector<PropertyName> &new_properties);
EdgeTypeIdMap AllocateEdgeTypeIds(const std::vector<EdgeTypeName> &new_edge_types);
/// Returns true if all shards have the desired number of replicas and they are in
/// the CONSENSUS_PARTICIPANT state. Note that this does not necessarily mean that
/// there is also an active leader for each shard.
bool ClusterInitialized() const;
};
} // namespace memgraph::coordinator

20
src/expr/CMakeLists.txt Normal file
View File

@@ -0,0 +1,20 @@
define_add_lcp(add_lcp_expr lcp_expr_cpp_files generated_lcp_expr_files)
add_lcp_expr(semantic/symbol.lcp)
add_custom_target(generate_lcp_expr DEPENDS ${generated_lcp_expr_files})
set(mg_expr_sources
${lcp_expr_cpp_files}
parsing.cpp)
find_package(Boost REQUIRED)
add_library(mg-expr STATIC ${mg_expr_sources})
add_dependencies(mg-expr generate_lcp_expr)
target_include_directories(mg-expr PUBLIC ${CMAKE_SOURCE_DIR}/include)
target_include_directories(mg-expr PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
target_include_directories(mg-expr PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/ast)
target_include_directories(mg-expr PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/interpret)
target_include_directories(mg-expr PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/semantic)
target_link_libraries(mg-expr cppitertools Boost::headers mg-utils mg-parser mg-functions)

33
src/expr/ast.hpp Normal file
View File

@@ -0,0 +1,33 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#ifndef MG_AST_INCLUDE_PATH
#ifdef MG_CLANG_TIDY_CHECK
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#include "query/v2/bindings/bindings.hpp"
#else
#error Missing AST include path
#endif
#endif
#ifndef MG_INJECTED_NAMESPACE_NAME
#ifdef MG_CLANG_TIDY_CHECK
#else
#error Missing AST namespace
#endif
#endif
#include MG_AST_INCLUDE_PATH
namespace memgraph::expr {
using namespace MG_INJECTED_NAMESPACE_NAME; // NOLINT(google-build-using-namespace)
} // namespace memgraph::expr

View File

@@ -0,0 +1,135 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "utils/visitor.hpp"
namespace MG_INJECTED_NAMESPACE_NAME {
// Forward declares for Tree visitors.
class CypherQuery;
class SingleQuery;
class CypherUnion;
class NamedExpression;
class Identifier;
class PropertyLookup;
class LabelsTest;
class Aggregation;
class Function;
class Reduce;
class Coalesce;
class Extract;
class All;
class Single;
class Any;
class None;
class ParameterLookup;
class CallProcedure;
class Create;
class Match;
class Return;
class With;
class Pattern;
class NodeAtom;
class EdgeAtom;
class PrimitiveLiteral;
class ListLiteral;
class MapLiteral;
class OrOperator;
class XorOperator;
class AndOperator;
class NotOperator;
class AdditionOperator;
class SubtractionOperator;
class MultiplicationOperator;
class DivisionOperator;
class ModOperator;
class UnaryPlusOperator;
class UnaryMinusOperator;
class IsNullOperator;
class NotEqualOperator;
class EqualOperator;
class LessOperator;
class GreaterOperator;
class LessEqualOperator;
class GreaterEqualOperator;
class InListOperator;
class SubscriptOperator;
class ListSlicingOperator;
class IfOperator;
class Delete;
class Where;
class SetProperty;
class SetProperties;
class SetLabels;
class RemoveProperty;
class RemoveLabels;
class Merge;
class Unwind;
class AuthQuery;
class ExplainQuery;
class ProfileQuery;
class IndexQuery;
class InfoQuery;
class ConstraintQuery;
class RegexMatch;
class DumpQuery;
class ReplicationQuery;
class LockPathQuery;
class LoadCsv;
class FreeMemoryQuery;
class TriggerQuery;
class IsolationLevelQuery;
class CreateSnapshotQuery;
class StreamQuery;
class SettingQuery;
class VersionQuery;
class Foreach;
class SchemaQuery;
using TreeCompositeVisitor = memgraph::utils::CompositeVisitor<
SingleQuery, CypherUnion, NamedExpression, OrOperator, XorOperator, AndOperator, NotOperator, AdditionOperator,
SubtractionOperator, MultiplicationOperator, DivisionOperator, ModOperator, NotEqualOperator, EqualOperator,
LessOperator, GreaterOperator, LessEqualOperator, GreaterEqualOperator, InListOperator, SubscriptOperator,
ListSlicingOperator, IfOperator, UnaryPlusOperator, UnaryMinusOperator, IsNullOperator, ListLiteral, MapLiteral,
PropertyLookup, LabelsTest, Aggregation, Function, Reduce, Coalesce, Extract, All, Single, Any, None, CallProcedure,
Create, Match, Return, With, Pattern, NodeAtom, EdgeAtom, Delete, Where, SetProperty, SetProperties, SetLabels,
RemoveProperty, RemoveLabels, Merge, Unwind, RegexMatch, LoadCsv, Foreach>;
using TreeLeafVisitor = memgraph::utils::LeafVisitor<Identifier, PrimitiveLiteral, ParameterLookup>;
class HierarchicalTreeVisitor : public TreeCompositeVisitor, public TreeLeafVisitor {
public:
using TreeCompositeVisitor::PostVisit;
using TreeCompositeVisitor::PreVisit;
using TreeLeafVisitor::Visit;
using typename TreeLeafVisitor::ReturnType;
};
template <class TResult>
class ExpressionVisitor
: public memgraph::utils::Visitor<
TResult, NamedExpression, OrOperator, XorOperator, AndOperator, NotOperator, AdditionOperator,
SubtractionOperator, MultiplicationOperator, DivisionOperator, ModOperator, NotEqualOperator, EqualOperator,
LessOperator, GreaterOperator, LessEqualOperator, GreaterEqualOperator, InListOperator, SubscriptOperator,
ListSlicingOperator, IfOperator, UnaryPlusOperator, UnaryMinusOperator, IsNullOperator, ListLiteral,
MapLiteral, PropertyLookup, LabelsTest, Aggregation, Function, Reduce, Coalesce, Extract, All, Single, Any,
None, ParameterLookup, Identifier, PrimitiveLiteral, RegexMatch> {};
template <class TResult>
class QueryVisitor
: public memgraph::utils::Visitor<TResult, CypherQuery, ExplainQuery, ProfileQuery, IndexQuery, AuthQuery,
InfoQuery, ConstraintQuery, DumpQuery, ReplicationQuery, LockPathQuery,
FreeMemoryQuery, TriggerQuery, IsolationLevelQuery, CreateSnapshotQuery,
StreamQuery, SettingQuery, VersionQuery, SchemaQuery> {};
} // namespace MG_INJECTED_NAMESPACE_NAME

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,271 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <iostream>
#include <type_traits>
#include "expr/ast.hpp"
#include "expr/typed_value.hpp"
#include "utils/algorithm.hpp"
#include "utils/logging.hpp"
#include "utils/string.hpp"
namespace memgraph::expr {
namespace detail {
template <typename T>
void PrintObject(std::ostream *out, const T &arg) {
static_assert(!std::is_convertible<T, Expression *>::value,
"This overload shouldn't be called with pointers convertible "
"to Expression *. This means your other PrintObject overloads aren't "
"being called for certain AST nodes when they should (or perhaps such "
"overloads don't exist yet).");
*out << arg;
}
inline void PrintObject(std::ostream *out, const std::string &str) { *out << utils::Escape(str); }
inline void PrintObject(std::ostream *out, Aggregation::Op op) { *out << Aggregation::OpToString(op); }
inline void PrintObject(std::ostream *out, Expression *expr);
inline void PrintObject(std::ostream *out, Identifier *expr) { PrintObject(out, static_cast<Expression *>(expr)); }
template <typename T>
void PrintObject(std::ostream *out, const std::vector<T> &vec) {
*out << "[";
utils::PrintIterable(*out, vec, ", ", [](auto &stream, const auto &item) { PrintObject(&stream, item); });
*out << "]";
}
template <typename T>
void PrintObject(std::ostream *out, const std::vector<T, utils::Allocator<T>> &vec) {
*out << "[";
utils::PrintIterable(*out, vec, ", ", [](auto &stream, const auto &item) { PrintObject(&stream, item); });
*out << "]";
}
template <typename K, typename V>
void PrintObject(std::ostream *out, const std::map<K, V> &map) {
*out << "{";
utils::PrintIterable(*out, map, ", ", [](auto &stream, const auto &item) {
PrintObject(&stream, item.first);
stream << ": ";
PrintObject(&stream, item.second);
});
*out << "}";
}
template <typename T>
void PrintObject(std::ostream *out, const utils::pmr::map<utils::pmr::string, T> &map) {
*out << "{";
utils::PrintIterable(*out, map, ", ", [](auto &stream, const auto &item) {
PrintObject(&stream, item.first);
stream << ": ";
PrintObject(&stream, item.second);
});
*out << "}";
}
template <typename T1, typename T2, typename T3>
inline void PrintObject(std::ostream *out, const TypedValueT<T1, T2, T3> &value) {
using TypedValue = TypedValueT<T1, T2, T3>;
switch (value.type()) {
case TypedValue::Type::Null:
*out << "null";
break;
case TypedValue::Type::String:
PrintObject(out, value.ValueString());
break;
case TypedValue::Type::Bool:
*out << (value.ValueBool() ? "true" : "false");
break;
case TypedValue::Type::Int:
PrintObject(out, value.ValueInt());
break;
case TypedValue::Type::Double:
PrintObject(out, value.ValueDouble());
break;
case TypedValue::Type::List:
PrintObject(out, value.ValueList());
break;
case TypedValue::Type::Map:
PrintObject(out, value.ValueMap());
break;
case TypedValue::Type::Date:
PrintObject(out, value.ValueDate());
break;
case TypedValue::Type::Duration:
PrintObject(out, value.ValueDuration());
break;
case TypedValue::Type::LocalTime:
PrintObject(out, value.ValueLocalTime());
break;
case TypedValue::Type::LocalDateTime:
PrintObject(out, value.ValueLocalDateTime());
break;
default:
MG_ASSERT(false, "PrintObject(std::ostream *out, const TypedValue &value) should not reach here");
}
}
template <typename T>
void PrintOperatorArgs(std::ostream *out, const T &arg) {
*out << " ";
PrintObject(out, arg);
*out << ")";
}
template <typename T, typename... Ts>
void PrintOperatorArgs(std::ostream *out, const T &arg, const Ts &...args) {
*out << " ";
PrintObject(out, arg);
PrintOperatorArgs(out, args...);
}
template <typename... Ts>
void PrintOperator(std::ostream *out, const std::string &name, const Ts &...args) {
*out << "(" << name;
PrintOperatorArgs(out, args...);
}
} // namespace detail
class ExpressionPrettyPrinter : public ExpressionVisitor<void> {
public:
explicit ExpressionPrettyPrinter(std::ostream *out) : out_(out) {}
// Unary operators
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define UNARY_OPERATOR_VISIT(OP_NODE, OP_STR) \
/* NOLINTNEXTLINE(bugprone-macro-parentheses) */ \
void Visit(OP_NODE &op) override { detail::PrintOperator(out_, OP_STR, op.expression_); }
UNARY_OPERATOR_VISIT(NotOperator, "Not");
UNARY_OPERATOR_VISIT(UnaryPlusOperator, "+");
UNARY_OPERATOR_VISIT(UnaryMinusOperator, "-");
UNARY_OPERATOR_VISIT(IsNullOperator, "IsNull");
#undef UNARY_OPERATOR_VISIT
// Binary operators
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define BINARY_OPERATOR_VISIT(OP_NODE, OP_STR) \
/* NOLINTNEXTLINE(bugprone-macro-parentheses) */ \
void Visit(OP_NODE &op) override { detail::PrintOperator(out_, OP_STR, op.expression1_, op.expression2_); }
BINARY_OPERATOR_VISIT(OrOperator, "Or");
BINARY_OPERATOR_VISIT(XorOperator, "Xor");
BINARY_OPERATOR_VISIT(AndOperator, "And");
BINARY_OPERATOR_VISIT(AdditionOperator, "+");
BINARY_OPERATOR_VISIT(SubtractionOperator, "-");
BINARY_OPERATOR_VISIT(MultiplicationOperator, "*");
BINARY_OPERATOR_VISIT(DivisionOperator, "/");
BINARY_OPERATOR_VISIT(ModOperator, "%");
BINARY_OPERATOR_VISIT(NotEqualOperator, "!=");
BINARY_OPERATOR_VISIT(EqualOperator, "==");
BINARY_OPERATOR_VISIT(LessOperator, "<");
BINARY_OPERATOR_VISIT(GreaterOperator, ">");
BINARY_OPERATOR_VISIT(LessEqualOperator, "<=");
BINARY_OPERATOR_VISIT(GreaterEqualOperator, ">=");
BINARY_OPERATOR_VISIT(InListOperator, "In");
BINARY_OPERATOR_VISIT(SubscriptOperator, "Subscript");
#undef BINARY_OPERATOR_VISIT
// Other
void Visit(ListSlicingOperator &op) override {
detail::PrintOperator(out_, "ListSlicing", op.list_, op.lower_bound_, op.upper_bound_);
}
void Visit(IfOperator &op) override {
detail::PrintOperator(out_, "If", op.condition_, op.then_expression_, op.else_expression_);
}
void Visit(ListLiteral &op) override { detail::PrintOperator(out_, "ListLiteral", op.elements_); }
void Visit(MapLiteral &op) override {
std::map<std::string, Expression *> map;
for (const auto &kv : op.elements_) {
map[kv.first.name] = kv.second;
}
detail::PrintObject(out_, map);
}
void Visit(LabelsTest &op) override { detail::PrintOperator(out_, "LabelsTest", op.expression_); }
void Visit(Aggregation &op) override { detail::PrintOperator(out_, "Aggregation", op.op_); }
void Visit(Function &op) override { detail::PrintOperator(out_, "Function", op.function_name_, op.arguments_); }
void Visit(Reduce &op) override {
detail::PrintOperator(out_, "Reduce", op.accumulator_, op.initializer_, op.identifier_, op.list_, op.expression_);
}
void Visit(Coalesce &op) override { detail::PrintOperator(out_, "Coalesce", op.expressions_); }
void Visit(Extract &op) override { detail::PrintOperator(out_, "Extract", op.identifier_, op.list_, op.expression_); }
void Visit(All &op) override {
detail::PrintOperator(out_, "All", op.identifier_, op.list_expression_, op.where_->expression_);
}
void Visit(Single &op) override {
detail::PrintOperator(out_, "Single", op.identifier_, op.list_expression_, op.where_->expression_);
}
void Visit(Any &op) override {
detail::PrintOperator(out_, "Any", op.identifier_, op.list_expression_, op.where_->expression_);
}
void Visit(None &op) override {
detail::PrintOperator(out_, "None", op.identifier_, op.list_expression_, op.where_->expression_);
}
void Visit(Identifier &op) override { detail::PrintOperator(out_, "Identifier", op.name_); }
void Visit(PrimitiveLiteral &op) override { detail::PrintObject(out_, op.value_); }
void Visit(PropertyLookup &op) override {
detail::PrintOperator(out_, "PropertyLookup", op.expression_, op.property_.name);
}
void Visit(ParameterLookup &op) override { detail::PrintOperator(out_, "ParameterLookup", op.token_position_); }
void Visit(NamedExpression &op) override { detail::PrintOperator(out_, "NamedExpression", op.name_, op.expression_); }
void Visit(RegexMatch &op) override { detail::PrintOperator(out_, "=~", op.string_expr_, op.regex_); }
private:
std::ostream *out_;
};
namespace detail {
inline void PrintObject(std::ostream *out, Expression *expr) {
if (expr) {
ExpressionPrettyPrinter printer{out};
expr->Accept(printer);
} else {
*out << "<null>";
}
}
} // namespace detail
inline void PrintExpression(Expression *expr, std::ostream *out) {
ExpressionPrettyPrinter printer{out};
expr->Accept(printer);
}
inline void PrintExpression(NamedExpression *expr, std::ostream *out) {
ExpressionPrettyPrinter printer{out};
expr->Accept(printer);
}
} // namespace memgraph::expr

View File

@@ -0,0 +1,279 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <iostream>
#include <type_traits>
#include "expr/ast.hpp"
#include "expr/typed_value.hpp"
#include "utils/algorithm.hpp"
#include "utils/logging.hpp"
#include "utils/string.hpp"
namespace memgraph::expr {
inline constexpr const char *identifier_node_symbol = "MG_SYMBOL_NODE";
inline constexpr const char *identifier_edge_symbol = "MG_SYMBOL_EDGE";
namespace detail {
template <typename T>
void PrintObject(std::ostream *out, const T &arg) {
static_assert(!std::is_convertible<T, Expression *>::value,
"This overload shouldn't be called with pointers convertible "
"to Expression *. This means your other PrintObject overloads aren't "
"being called for certain AST nodes when they should (or perhaps such "
"overloads don't exist yet).");
*out << arg;
}
inline void PrintObject(std::ostream *out, const std::string &str) { *out << str; }
inline void PrintObject(std::ostream * /*out*/, Aggregation::Op /*op*/) {
throw utils::NotYetImplemented("PrintObject: Aggregation::Op");
}
inline void PrintObject(std::ostream * /*out*/, Expression * /*expr*/);
inline void PrintObject(std::ostream *out, Identifier *expr) { PrintObject(out, static_cast<Expression *>(expr)); }
template <typename T>
void PrintObject(std::ostream * /*out*/, const std::vector<T> & /*vec*/) {
throw utils::NotYetImplemented("PrintObject: vector<T>");
}
template <typename T>
void PrintObject(std::ostream * /*out*/, const std::vector<T, utils::Allocator<T>> & /*vec*/) {
throw utils::NotYetImplemented("PrintObject: vector<T, utils::Allocator<T>>");
}
template <typename K, typename V>
void PrintObject(std::ostream * /*out*/, const std::map<K, V> & /*map*/) {
throw utils::NotYetImplemented("PrintObject: map<K, V>");
}
template <typename T>
void PrintObject(std::ostream * /*out*/, const utils::pmr::map<utils::pmr::string, T> & /*map*/) {
throw utils::NotYetImplemented("PrintObject: map<utils::pmr::string, V>");
}
template <typename T1, typename T2, typename T3>
inline void PrintObject(std::ostream *out, const TypedValueT<T1, T2, T3> &value) {
using TypedValue = TypedValueT<T1, T2, T3>;
switch (value.type()) {
case TypedValue::Type::Null:
*out << "null";
break;
case TypedValue::Type::String:
PrintObject(out, value.ValueString());
break;
case TypedValue::Type::Bool:
*out << (value.ValueBool() ? "true" : "false");
break;
case TypedValue::Type::Int:
PrintObject(out, value.ValueInt());
break;
case TypedValue::Type::Double:
PrintObject(out, value.ValueDouble());
break;
case TypedValue::Type::List:
PrintObject(out, value.ValueList());
break;
case TypedValue::Type::Map:
PrintObject(out, value.ValueMap());
break;
case TypedValue::Type::Date:
PrintObject(out, value.ValueDate());
break;
case TypedValue::Type::Duration:
PrintObject(out, value.ValueDuration());
break;
case TypedValue::Type::LocalTime:
PrintObject(out, value.ValueLocalTime());
break;
case TypedValue::Type::LocalDateTime:
PrintObject(out, value.ValueLocalDateTime());
break;
default:
MG_ASSERT(false, "PrintObject(std::ostream *out, const TypedValue &value) should not reach here");
}
}
template <typename T>
void PrintOperatorArgs(const std::string & /*name*/, std::ostream *out, bool with_parenthesis, const T &arg) {
PrintObject(out, arg);
if (with_parenthesis) {
*out << ")";
}
}
template <typename T, typename... Ts>
void PrintOperatorArgs(const std::string &name, std::ostream *out, bool with_parenthesis, const T &arg,
const Ts &...args) {
PrintObject(out, arg);
*out << " " << name << " ";
PrintOperatorArgs(name, out, with_parenthesis, args...);
}
template <typename... Ts>
void PrintOperator(const std::string &name, std::ostream *out, bool with_parenthesis, const Ts &...args) {
if (with_parenthesis) {
*out << "(";
}
PrintOperatorArgs(name, out, with_parenthesis, args...);
}
// new
template <typename T>
void PrintOperatorArgs(std::ostream *out, const T &arg) {
PrintObject(out, arg);
}
template <typename T, typename... Ts>
void PrintOperatorArgs(std::ostream *out, const T &arg, const Ts &...args) {
PrintObject(out, arg);
PrintOperatorArgs(out, args...);
}
template <typename... Ts>
void PrintOperator(std::ostream *out, const Ts &...args) {
PrintOperatorArgs(out, args...);
}
} // namespace detail
class ExpressionPrettyPrinter : public ExpressionVisitor<void> {
public:
explicit ExpressionPrettyPrinter(std::ostream *out) : out_(out) {}
// Unary operators
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define UNARY_OPERATOR_VISIT(OP_NODE, OP_STR) \
/* NOLINTNEXTLINE(bugprone-macro-parentheses) */ \
void Visit(OP_NODE &op) override { detail::PrintOperator(OP_STR, out_, false /*with_parenthesis*/, op.expression_); }
UNARY_OPERATOR_VISIT(NotOperator, "Not");
UNARY_OPERATOR_VISIT(UnaryPlusOperator, "+");
UNARY_OPERATOR_VISIT(UnaryMinusOperator, "-");
UNARY_OPERATOR_VISIT(IsNullOperator, "IsNull");
#undef UNARY_OPERATOR_VISIT
// Binary operators
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define BINARY_OPERATOR_VISIT(OP_NODE, OP_STR) \
/* NOLINTNEXTLINE(bugprone-macro-parentheses) */ \
void Visit(OP_NODE &op) override { \
detail::PrintOperator(OP_STR, out_, true /*with_parenthesis*/, op.expression1_, op.expression2_); \
}
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define BINARY_OPERATOR_VISIT_NOT_IMPL(OP_NODE, OP_STR) \
/* NOLINTNEXTLINE(bugprone-macro-parentheses) */ \
void Visit(OP_NODE & /*op*/) override { throw utils::NotYetImplemented("OP_NODE"); }
BINARY_OPERATOR_VISIT(OrOperator, "Or");
BINARY_OPERATOR_VISIT(XorOperator, "Xor");
BINARY_OPERATOR_VISIT(AndOperator, "And");
BINARY_OPERATOR_VISIT(AdditionOperator, "+");
BINARY_OPERATOR_VISIT(SubtractionOperator, "-");
BINARY_OPERATOR_VISIT(MultiplicationOperator, "*");
BINARY_OPERATOR_VISIT(DivisionOperator, "/");
BINARY_OPERATOR_VISIT(ModOperator, "%");
BINARY_OPERATOR_VISIT(NotEqualOperator, "!=");
BINARY_OPERATOR_VISIT(EqualOperator, "=");
BINARY_OPERATOR_VISIT(LessOperator, "<");
BINARY_OPERATOR_VISIT(GreaterOperator, ">");
BINARY_OPERATOR_VISIT(LessEqualOperator, "<=");
BINARY_OPERATOR_VISIT(GreaterEqualOperator, ">=");
BINARY_OPERATOR_VISIT_NOT_IMPL(InListOperator, "In");
BINARY_OPERATOR_VISIT_NOT_IMPL(SubscriptOperator, "Subscript");
#undef BINARY_OPERATOR_VISIT
#undef BINARY_OPERATOR_VISIT_NOT_IMPL
// Other
void Visit(ListSlicingOperator & /*op*/) override { throw utils::NotYetImplemented("ListSlicingOperator"); }
void Visit(IfOperator & /*op*/) override { throw utils::NotYetImplemented("IfOperator"); }
void Visit(ListLiteral & /*op*/) override { throw utils::NotYetImplemented("ListLiteral"); }
void Visit(MapLiteral & /*op*/) override { throw utils::NotYetImplemented("MapLiteral"); }
void Visit(LabelsTest & /*op*/) override { throw utils::NotYetImplemented("LabelsTest"); }
void Visit(Aggregation & /*op*/) override { throw utils::NotYetImplemented("Aggregation"); }
void Visit(Function & /*op*/) override { throw utils::NotYetImplemented("Function"); }
void Visit(Reduce & /*op*/) override { throw utils::NotYetImplemented("Reduce"); }
void Visit(Coalesce & /*op*/) override { throw utils::NotYetImplemented("Coalesce"); }
void Visit(Extract & /*op*/) override { throw utils::NotYetImplemented("Extract"); }
void Visit(All & /*op*/) override { throw utils::NotYetImplemented("All"); }
void Visit(Single & /*op*/) override { throw utils::NotYetImplemented("Single"); }
void Visit(Any & /*op*/) override { throw utils::NotYetImplemented("Any"); }
void Visit(None & /*op*/) override { throw utils::NotYetImplemented("None"); }
void Visit(Identifier &op) override {
auto is_node = true;
auto is_edge = false;
auto is_other = false;
if (is_node) {
detail::PrintOperator(out_, identifier_node_symbol);
} else if (is_edge) {
detail::PrintOperator(out_, identifier_edge_symbol);
} else {
MG_ASSERT(is_other);
detail::PrintOperator(out_, op.name_);
}
}
void Visit(PrimitiveLiteral &op) override { detail::PrintObject(out_, op.value_); }
void Visit(PropertyLookup &op) override { detail::PrintOperator(out_, op.expression_, ".", op.property_.name); }
void Visit(ParameterLookup & /*op*/) override { throw utils::NotYetImplemented("ParameterLookup"); }
void Visit(NamedExpression & /*op*/) override { throw utils::NotYetImplemented("NamedExpression"); }
void Visit(RegexMatch & /*op*/) override { throw utils::NotYetImplemented("RegexMatch"); }
private:
std::ostream *out_;
};
namespace detail {
inline void PrintObject(std::ostream *out, Expression *expr) {
if (expr) {
ExpressionPrettyPrinter printer{out};
expr->Accept(printer);
} else {
*out << "<null>";
}
}
} // namespace detail
inline void PrintExpressionToOriginalAndReplaceNodeAndEdgeSymbols(Expression *expr, std::ostream *out) {
ExpressionPrettyPrinter printer{out};
expr->Accept(printer);
}
inline std::string ExpressiontoStringWhileReplacingNodeAndEdgeSymbols(Expression *expr) {
std::ostringstream ss;
expr::PrintExpressionToOriginalAndReplaceNodeAndEdgeSymbols(expr, &ss);
return ss.str();
}
} // namespace memgraph::expr

52
src/expr/exceptions.hpp Normal file
View File

@@ -0,0 +1,52 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "utils/exceptions.hpp"
namespace memgraph::expr {
class SyntaxException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
SyntaxException() : SyntaxException("") {}
};
class SemanticException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
SemanticException() : BasicException("") {}
};
class ExpressionRuntimeException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
};
class RedeclareVariableError : public SemanticException {
public:
explicit RedeclareVariableError(const std::string &name) : SemanticException("Redeclaring variable: " + name + ".") {}
};
class UnboundVariableError : public SemanticException {
public:
explicit UnboundVariableError(const std::string &name) : SemanticException("Unbound variable: " + name + ".") {}
};
class TypeMismatchError : public SemanticException {
public:
TypeMismatchError(const std::string &name, const std::string &datum, const std::string &expected)
: SemanticException(fmt::format("Type mismatch: {} already defined as {}, expected {}.", name, datum, expected)) {
}
};
} // namespace memgraph::expr

821
src/expr/interpret/eval.hpp Normal file
View File

@@ -0,0 +1,821 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
/// @file
#pragma once
#include <algorithm>
#include <limits>
#include <map>
#include <optional>
#include <regex>
#include <type_traits>
#include <vector>
#include "expr/ast.hpp"
#include "expr/exceptions.hpp"
#include "expr/interpret/frame.hpp"
#include "expr/semantic/symbol_table.hpp"
#include "functions/awesome_memgraph_functions.hpp"
#include "utils/exceptions.hpp"
namespace memgraph::expr {
struct StorageTag {};
struct QueryEngineTag {};
template <typename TypedValue, typename EvaluationContext, typename DbAccessor, typename StorageView, typename LabelId,
typename PropertyValue, typename ConvFunctor, typename Error, typename Tag = StorageTag>
class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
public:
ExpressionEvaluator(Frame<TypedValue> *frame, const SymbolTable &symbol_table, const EvaluationContext &ctx,
DbAccessor *dba, StorageView view)
: frame_(frame), symbol_table_(&symbol_table), ctx_(&ctx), dba_(dba), view_(view) {}
using ExpressionVisitor<TypedValue>::Visit;
utils::MemoryResource *GetMemoryResource() const { return ctx_->memory; }
TypedValue Visit(NamedExpression &named_expression) override {
const auto &symbol = symbol_table_->at(named_expression);
auto value = named_expression.expression_->Accept(*this);
frame_->at(symbol) = value;
return value;
}
TypedValue Visit(Identifier &ident) override {
return TypedValue(frame_->at(symbol_table_->at(ident)), ctx_->memory);
}
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define BINARY_OPERATOR_VISITOR(OP_NODE, CPP_OP, CYPHER_OP) \
/* NOLINTNEXTLINE(bugprone-macro-parentheses) */ \
TypedValue Visit(OP_NODE &op) override { \
auto val1 = op.expression1_->Accept(*this); \
auto val2 = op.expression2_->Accept(*this); \
try { \
return val1 CPP_OP val2; \
} catch (const TypedValueException &) { \
throw ExpressionRuntimeException("Invalid types: {} and {} for '{}'.", val1.type(), val2.type(), #CYPHER_OP); \
} \
}
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define UNARY_OPERATOR_VISITOR(OP_NODE, CPP_OP, CYPHER_OP) \
/* NOLINTNEXTLINE(bugprone-macro-parentheses) */ \
TypedValue Visit(OP_NODE &op) override { \
auto val = op.expression_->Accept(*this); \
try { \
return CPP_OP val; \
} catch (const TypedValueException &) { \
throw ExpressionRuntimeException("Invalid type {} for '{}'.", val.type(), #CYPHER_OP); \
} \
}
BINARY_OPERATOR_VISITOR(OrOperator, ||, OR);
BINARY_OPERATOR_VISITOR(XorOperator, ^, XOR);
BINARY_OPERATOR_VISITOR(AdditionOperator, +, +);
BINARY_OPERATOR_VISITOR(SubtractionOperator, -, -);
BINARY_OPERATOR_VISITOR(MultiplicationOperator, *, *);
BINARY_OPERATOR_VISITOR(DivisionOperator, /, /);
BINARY_OPERATOR_VISITOR(ModOperator, %, %);
BINARY_OPERATOR_VISITOR(NotEqualOperator, !=, <>);
BINARY_OPERATOR_VISITOR(EqualOperator, ==, =);
BINARY_OPERATOR_VISITOR(LessOperator, <, <);
BINARY_OPERATOR_VISITOR(GreaterOperator, >, >);
BINARY_OPERATOR_VISITOR(LessEqualOperator, <=, <=);
BINARY_OPERATOR_VISITOR(GreaterEqualOperator, >=, >=);
UNARY_OPERATOR_VISITOR(NotOperator, !, NOT);
UNARY_OPERATOR_VISITOR(UnaryPlusOperator, +, +);
UNARY_OPERATOR_VISITOR(UnaryMinusOperator, -, -);
#undef BINARY_OPERATOR_VISITOR
#undef UNARY_OPERATOR_VISITOR
void HandleObjectAccessError(Error &shard_error, const std::string_view accessed_object) {
switch (shard_error) {
case Error::DELETED_OBJECT:
throw ExpressionRuntimeException("Trying to access {} on a deleted object.", accessed_object);
case Error::NONEXISTENT_OBJECT:
throw ExpressionRuntimeException("Trying to access {} from a node object doesn't exist.", accessed_object);
case Error::SERIALIZATION_ERROR:
case Error::VERTEX_HAS_EDGES:
case Error::PROPERTIES_DISABLED:
case Error::VERTEX_ALREADY_INSERTED:
case Error::OBJECT_NOT_FOUND:
throw ExpressionRuntimeException("Unexpected error when accessing {}.", accessed_object);
case Error::SCHEMA_NO_SCHEMA_DEFINED_FOR_LABEL:
case Error::SCHEMA_VERTEX_PROPERTY_WRONG_TYPE:
case Error::SCHEMA_VERTEX_UPDATE_PRIMARY_KEY:
case Error::SCHEMA_VERTEX_UPDATE_PRIMARY_LABEL:
case Error::SCHEMA_VERTEX_SECONDARY_LABEL_IS_PRIMARY:
case Error::SCHEMA_VERTEX_PRIMARY_PROPERTIES_UNDEFINED:
throw ExpressionRuntimeException("Unexpected schema violation when accessing {}.", accessed_object);
}
}
TypedValue Visit(AndOperator &op) override {
auto value1 = op.expression1_->Accept(*this);
if (value1.IsBool() && !value1.ValueBool()) {
// If first expression is false, don't evaluate the second one.
return value1;
}
auto value2 = op.expression2_->Accept(*this);
try {
return value1 && value2;
} catch (const TypedValueException &) {
throw ExpressionRuntimeException("Invalid types: {} and {} for AND.", value1.type(), value2.type());
}
}
TypedValue Visit(IfOperator &if_operator) override {
auto condition = if_operator.condition_->Accept(*this);
if (condition.IsNull()) {
return if_operator.else_expression_->Accept(*this);
}
if (condition.type() != TypedValue::Type::Bool) {
// At the moment IfOperator is used only in CASE construct.
throw ExpressionRuntimeException("CASE expected boolean expression, got {}.", condition.type());
}
if (condition.ValueBool()) {
return if_operator.then_expression_->Accept(*this);
}
return if_operator.else_expression_->Accept(*this);
}
TypedValue Visit(InListOperator &in_list) override {
auto literal = in_list.expression1_->Accept(*this);
auto _list = in_list.expression2_->Accept(*this);
if (_list.IsNull()) {
return TypedValue(ctx_->memory);
}
// Exceptions have higher priority than returning nulls when list expression
// is not null.
if (_list.type() != TypedValue::Type::List) {
throw ExpressionRuntimeException("IN expected a list, got {}.", _list.type());
}
const auto &list = _list.ValueList();
// If literal is NULL there is no need to try to compare it with every
// element in the list since result of every comparison will be NULL. There
// is one special case that we must test explicitly: if list is empty then
// result is false since no comparison will be performed.
if (list.empty()) return TypedValue(false, ctx_->memory);
if (literal.IsNull()) return TypedValue(ctx_->memory);
auto has_null = false;
for (const auto &element : list) {
auto result = literal == element;
if (result.IsNull()) {
has_null = true;
} else if (result.ValueBool()) {
return TypedValue(true, ctx_->memory);
}
}
if (has_null) {
return TypedValue(ctx_->memory);
}
return TypedValue(false, ctx_->memory);
}
TypedValue Visit(SubscriptOperator &list_indexing) override {
auto lhs = list_indexing.expression1_->Accept(*this);
auto index = list_indexing.expression2_->Accept(*this);
if (!lhs.IsList() && !lhs.IsMap() && !lhs.IsVertex() && !lhs.IsEdge() && !lhs.IsNull())
throw ExpressionRuntimeException(
"Expected a list, a map, a node or an edge to index with '[]', got "
"{}.",
lhs.type());
if (lhs.IsNull() || index.IsNull()) return TypedValue(ctx_->memory);
if (lhs.IsList()) {
if (!index.IsInt())
throw ExpressionRuntimeException("Expected an integer as a list index, got {}.", index.type());
auto index_int = index.ValueInt();
// NOTE: Take non-const reference to list, so that we can move out the
// indexed element as the result.
auto &list = lhs.ValueList();
if (index_int < 0) {
index_int += static_cast<int64_t>(list.size());
}
if (index_int >= static_cast<int64_t>(list.size()) || index_int < 0) return TypedValue(ctx_->memory);
// NOTE: Explicit move is needed, so that we return the move constructed
// value and preserve the correct MemoryResource.
return std::move(list[index_int]);
}
if (lhs.IsMap()) {
if (!index.IsString())
throw ExpressionRuntimeException("Expected a string as a map index, got {}.", index.type());
// NOTE: Take non-const reference to map, so that we can move out the
// looked-up element as the result.
auto &map = lhs.ValueMap();
auto found = map.find(index.ValueString());
if (found == map.end()) return TypedValue(ctx_->memory);
// NOTE: Explicit move is needed, so that we return the move constructed
// value and preserve the correct MemoryResource.
return std::move(found->second);
}
if (lhs.IsVertex()) {
if (!index.IsString())
throw ExpressionRuntimeException("Expected a string as a property name, got {}.", index.type());
return TypedValue(GetProperty(lhs.ValueVertex(), index.ValueString()), ctx_->memory);
}
if (lhs.IsEdge()) {
if (!index.IsString())
throw ExpressionRuntimeException("Expected a string as a property name, got {}.", index.type());
return TypedValue(GetProperty(lhs.ValueEdge(), index.ValueString()), ctx_->memory);
}
// lhs is Null
return TypedValue(ctx_->memory);
}
TypedValue Visit(ListSlicingOperator &op) override {
// If some type is null we can't return null, because throwing exception
// on illegal type has higher priority.
auto is_null = false;
auto get_bound = [&](Expression *bound_expr, int64_t default_value) {
if (bound_expr) {
auto bound = bound_expr->Accept(*this);
if (bound.type() == TypedValue::Type::Null) {
is_null = true;
} else if (bound.type() != TypedValue::Type::Int) {
throw ExpressionRuntimeException("Expected an integer for a bound in list slicing, got {}.", bound.type());
}
return bound;
}
return TypedValue(default_value, ctx_->memory);
};
auto _upper_bound = get_bound(op.upper_bound_, std::numeric_limits<int64_t>::max());
auto _lower_bound = get_bound(op.lower_bound_, 0);
auto _list = op.list_->Accept(*this);
if (_list.type() == TypedValue::Type::Null) {
is_null = true;
} else if (_list.type() != TypedValue::Type::List) {
throw ExpressionRuntimeException("Expected a list to slice, got {}.", _list.type());
}
if (is_null) {
return TypedValue(ctx_->memory);
}
const auto &list = _list.ValueList();
auto normalise_bound = [&](int64_t bound) {
if (bound < 0) {
bound = static_cast<int64_t>(list.size()) + bound;
}
return std::max(static_cast<int64_t>(0), std::min(bound, static_cast<int64_t>(list.size())));
};
auto lower_bound = normalise_bound(_lower_bound.ValueInt());
auto upper_bound = normalise_bound(_upper_bound.ValueInt());
if (upper_bound <= lower_bound) {
return TypedValue(typename TypedValue::TVector(ctx_->memory), ctx_->memory);
}
return TypedValue(
typename TypedValue::TVector(list.begin() + lower_bound, list.begin() + upper_bound, ctx_->memory));
}
TypedValue Visit(IsNullOperator &is_null) override {
auto value = is_null.expression_->Accept(*this);
return TypedValue(value.IsNull(), ctx_->memory);
}
TypedValue Visit(PropertyLookup &property_lookup) override {
auto expression_result = property_lookup.expression_->Accept(*this);
auto maybe_date = [this](const auto &date, const auto &prop_name) -> std::optional<TypedValue> {
if (prop_name == "year") {
return TypedValue(date.year, ctx_->memory);
}
if (prop_name == "month") {
return TypedValue(date.month, ctx_->memory);
}
if (prop_name == "day") {
return TypedValue(date.day, ctx_->memory);
}
return std::nullopt;
};
auto maybe_local_time = [this](const auto &lt, const auto &prop_name) -> std::optional<TypedValue> {
if (prop_name == "hour") {
return TypedValue(lt.hour, ctx_->memory);
}
if (prop_name == "minute") {
return TypedValue(lt.minute, ctx_->memory);
}
if (prop_name == "second") {
return TypedValue(lt.second, ctx_->memory);
}
if (prop_name == "millisecond") {
return TypedValue(lt.millisecond, ctx_->memory);
}
if (prop_name == "microsecond") {
return TypedValue(lt.microsecond, ctx_->memory);
}
return std::nullopt;
};
auto maybe_duration = [this](const auto &dur, const auto &prop_name) -> std::optional<TypedValue> {
if (prop_name == "day") {
return TypedValue(dur.Days(), ctx_->memory);
}
if (prop_name == "hour") {
return TypedValue(dur.SubDaysAsHours(), ctx_->memory);
}
if (prop_name == "minute") {
return TypedValue(dur.SubDaysAsMinutes(), ctx_->memory);
}
if (prop_name == "second") {
return TypedValue(dur.SubDaysAsSeconds(), ctx_->memory);
}
if (prop_name == "millisecond") {
return TypedValue(dur.SubDaysAsMilliseconds(), ctx_->memory);
}
if (prop_name == "microsecond") {
return TypedValue(dur.SubDaysAsMicroseconds(), ctx_->memory);
}
if (prop_name == "nanosecond") {
return TypedValue(dur.SubDaysAsNanoseconds(), ctx_->memory);
}
return std::nullopt;
};
switch (expression_result.type()) {
case TypedValue::Type::Null:
return TypedValue(ctx_->memory);
case TypedValue::Type::Vertex:
return GetProperty(expression_result.ValueVertex(), property_lookup.property_);
case TypedValue::Type::Edge:
return GetProperty(expression_result.ValueEdge(), property_lookup.property_);
case TypedValue::Type::Map: {
// NOTE: Take non-const reference to map, so that we can move out the
// looked-up element as the result.
auto &map = expression_result.ValueMap();
auto found = map.find(property_lookup.property_.name.c_str());
if (found == map.end()) return TypedValue(ctx_->memory);
// NOTE: Explicit move is needed, so that we return the move constructed
// value and preserve the correct MemoryResource.
return std::move(found->second);
}
case TypedValue::Type::Duration: {
const auto &prop_name = property_lookup.property_.name;
const auto &dur = expression_result.ValueDuration();
if (auto dur_field = maybe_duration(dur, prop_name); dur_field) {
return std::move(*dur_field);
}
throw ExpressionRuntimeException("Invalid property name {} for Duration", prop_name);
}
case TypedValue::Type::Date: {
const auto &prop_name = property_lookup.property_.name;
const auto &date = expression_result.ValueDate();
if (auto date_field = maybe_date(date, prop_name); date_field) {
return std::move(*date_field);
}
throw ExpressionRuntimeException("Invalid property name {} for Date", prop_name);
}
case TypedValue::Type::LocalTime: {
const auto &prop_name = property_lookup.property_.name;
const auto &lt = expression_result.ValueLocalTime();
if (auto lt_field = maybe_local_time(lt, prop_name); lt_field) {
return std::move(*lt_field);
}
throw ExpressionRuntimeException("Invalid property name {} for LocalTime", prop_name);
}
case TypedValue::Type::LocalDateTime: {
const auto &prop_name = property_lookup.property_.name;
const auto &ldt = expression_result.ValueLocalDateTime();
if (auto date_field = maybe_date(ldt.date, prop_name); date_field) {
return std::move(*date_field);
}
if (auto lt_field = maybe_local_time(ldt.local_time, prop_name); lt_field) {
return std::move(*lt_field);
}
throw ExpressionRuntimeException("Invalid property name {} for LocalDateTime", prop_name);
}
default:
throw ExpressionRuntimeException("Only nodes, edges, maps and temporal types have properties to be looked-up.");
}
}
template <typename VertexAccessor, typename TTag = Tag,
typename TReturnType = std::enable_if_t<std::is_same_v<TTag, StorageTag>, bool>>
TReturnType HasLabelImpl(const VertexAccessor &vertex, const LabelIx &label, StorageTag /*tag*/) {
auto has_label = vertex.HasLabel(view_, GetLabel(label));
if (has_label.HasError() && has_label.GetError() == Error::NONEXISTENT_OBJECT) {
// This is a very nasty and temporary hack in order to make MERGE
// work. The old storage had the following logic when returning an
// `OLD` view: `return old ? old : new`. That means that if the
// `OLD` view didn't exist, it returned the NEW view. With this hack
// we simulate that behavior.
// TODO (mferencevic, teon.banek): Remove once MERGE is
// reimplemented.
has_label = vertex.HasLabel(StorageView::NEW, GetLabel(label));
}
if (has_label.HasError()) {
HandleObjectAccessError(has_label.GetError().code, "labels");
}
return *has_label;
}
template <typename VertexAccessor, typename TTag = Tag,
typename TReturnType = std::enable_if_t<std::is_same_v<TTag, QueryEngineTag>, bool>>
TReturnType HasLabelImpl(const VertexAccessor &vertex, const LabelIx &label_ix, QueryEngineTag /*tag*/) {
auto label = typename VertexAccessor::Label{LabelId::FromUint(label_ix.ix)};
return vertex.HasLabel(label);
}
TypedValue Visit(LabelsTest &labels_test) override {
auto expression_result = labels_test.expression_->Accept(*this);
switch (expression_result.type()) {
case TypedValue::Type::Null:
return TypedValue(ctx_->memory);
case TypedValue::Type::Vertex: {
const auto &vertex = expression_result.ValueVertex();
if (std::ranges::all_of(labels_test.labels_, [&vertex, this](const auto label_test) {
return this->HasLabelImpl(vertex, label_test, Tag{});
})) {
return TypedValue(true, ctx_->memory);
}
return TypedValue(false, ctx_->memory);
}
default:
throw ExpressionRuntimeException("Only nodes have labels.");
}
}
TypedValue Visit(PrimitiveLiteral &literal) override {
// TODO: no need to evaluate constants, we can write it to frame in one
// of the previous phases.
return TypedValue(literal.value_, ctx_->memory);
}
TypedValue Visit(ListLiteral &literal) override {
typename TypedValue::TVector result(ctx_->memory);
result.reserve(literal.elements_.size());
for (const auto &expression : literal.elements_) result.emplace_back(expression->Accept(*this));
return TypedValue(result, ctx_->memory);
}
TypedValue Visit(MapLiteral &literal) override {
typename TypedValue::TMap result(ctx_->memory);
for (const auto &pair : literal.elements_) result.emplace(pair.first.name, pair.second->Accept(*this));
return TypedValue(result, ctx_->memory);
}
TypedValue Visit(Aggregation &aggregation) override {
return TypedValue(frame_->at(symbol_table_->at(aggregation)), ctx_->memory);
}
TypedValue Visit(Coalesce &coalesce) override {
auto &exprs = coalesce.expressions_;
if (exprs.size() == 0) {
throw ExpressionRuntimeException("'coalesce' requires at least one argument.");
}
for (int64_t i = 0; i < exprs.size(); ++i) {
TypedValue val(exprs[i]->Accept(*this), ctx_->memory);
if (!val.IsNull()) {
return val;
}
}
return TypedValue(ctx_->memory);
}
TypedValue Visit(Function &function) override {
functions::FunctionContext<DbAccessor> function_ctx{dba_, ctx_->memory, ctx_->timestamp, &ctx_->counters, view_};
// Stack allocate evaluated arguments when there's a small number of them.
if (function.arguments_.size() <= 8) {
TypedValue arguments[8] = {TypedValue(ctx_->memory), TypedValue(ctx_->memory), TypedValue(ctx_->memory),
TypedValue(ctx_->memory), TypedValue(ctx_->memory), TypedValue(ctx_->memory),
TypedValue(ctx_->memory), TypedValue(ctx_->memory)};
for (size_t i = 0; i < function.arguments_.size(); ++i) {
arguments[i] = function.arguments_[i]->Accept(*this);
}
auto res = function.function_(arguments, function.arguments_.size(), function_ctx);
MG_ASSERT(res.GetMemoryResource() == ctx_->memory);
return res;
} else {
typename TypedValue::TVector arguments(ctx_->memory);
arguments.reserve(function.arguments_.size());
for (const auto &argument : function.arguments_) {
arguments.emplace_back(argument->Accept(*this));
}
auto res = function.function_(arguments.data(), arguments.size(), function_ctx);
MG_ASSERT(res.GetMemoryResource() == ctx_->memory);
return res;
}
}
TypedValue Visit(Reduce &reduce) override {
auto list_value = reduce.list_->Accept(*this);
if (list_value.IsNull()) {
return TypedValue(ctx_->memory);
}
if (list_value.type() != TypedValue::Type::List) {
throw ExpressionRuntimeException("REDUCE expected a list, got {}.", list_value.type());
}
const auto &list = list_value.ValueList();
const auto &element_symbol = symbol_table_->at(*reduce.identifier_);
const auto &accumulator_symbol = symbol_table_->at(*reduce.accumulator_);
auto accumulator = reduce.initializer_->Accept(*this);
for (const auto &element : list) {
frame_->at(accumulator_symbol) = accumulator;
frame_->at(element_symbol) = element;
accumulator = reduce.expression_->Accept(*this);
}
return accumulator;
}
TypedValue Visit(Extract &extract) override {
auto list_value = extract.list_->Accept(*this);
if (list_value.IsNull()) {
return TypedValue(ctx_->memory);
}
if (list_value.type() != TypedValue::Type::List) {
throw ExpressionRuntimeException("EXTRACT expected a list, got {}.", list_value.type());
}
const auto &list = list_value.ValueList();
const auto &element_symbol = symbol_table_->at(*extract.identifier_);
typename TypedValue::TVector result(ctx_->memory);
result.reserve(list.size());
for (const auto &element : list) {
if (element.IsNull()) {
result.emplace_back();
} else {
frame_->at(element_symbol) = element;
result.emplace_back(extract.expression_->Accept(*this));
}
}
return TypedValue(result, ctx_->memory);
}
TypedValue Visit(All &all) override {
auto list_value = all.list_expression_->Accept(*this);
if (list_value.IsNull()) {
return TypedValue(ctx_->memory);
}
if (list_value.type() != TypedValue::Type::List) {
throw ExpressionRuntimeException("ALL expected a list, got {}.", list_value.type());
}
const auto &list = list_value.ValueList();
const auto &symbol = symbol_table_->at(*all.identifier_);
bool has_null_elements = false;
bool has_value = false;
for (const auto &element : list) {
frame_->at(symbol) = element;
auto result = all.where_->expression_->Accept(*this);
if (!result.IsNull() && result.type() != TypedValue::Type::Bool) {
throw ExpressionRuntimeException("Predicate of ALL must evaluate to boolean, got {}.", result.type());
}
if (!result.IsNull()) {
has_value = true;
if (!result.ValueBool()) {
return TypedValue(false, ctx_->memory);
}
} else {
has_null_elements = true;
}
}
if (!has_value) {
return TypedValue(ctx_->memory);
}
if (has_null_elements) {
return TypedValue(false, ctx_->memory);
} else {
return TypedValue(true, ctx_->memory);
}
}
TypedValue Visit(Single &single) override {
auto list_value = single.list_expression_->Accept(*this);
if (list_value.IsNull()) {
return TypedValue(ctx_->memory);
}
if (list_value.type() != TypedValue::Type::List) {
throw ExpressionRuntimeException("SINGLE expected a list, got {}.", list_value.type());
}
const auto &list = list_value.ValueList();
const auto &symbol = symbol_table_->at(*single.identifier_);
bool has_value = false;
bool predicate_satisfied = false;
for (const auto &element : list) {
frame_->at(symbol) = element;
auto result = single.where_->expression_->Accept(*this);
if (!result.IsNull() && result.type() != TypedValue::Type::Bool) {
throw ExpressionRuntimeException("Predicate of SINGLE must evaluate to boolean, got {}.", result.type());
}
if (result.type() == TypedValue::Type::Bool) {
has_value = true;
}
if (result.IsNull() || !result.ValueBool()) {
continue;
}
// Return false if more than one element satisfies the predicate.
if (predicate_satisfied) {
return TypedValue(false, ctx_->memory);
} else {
predicate_satisfied = true;
}
}
if (!has_value) {
return TypedValue(ctx_->memory);
} else {
return TypedValue(predicate_satisfied, ctx_->memory);
}
}
TypedValue Visit(Any &any) override {
auto list_value = any.list_expression_->Accept(*this);
if (list_value.IsNull()) {
return TypedValue(ctx_->memory);
}
if (list_value.type() != TypedValue::Type::List) {
throw ExpressionRuntimeException("ANY expected a list, got {}.", list_value.type());
}
const auto &list = list_value.ValueList();
const auto &symbol = symbol_table_->at(*any.identifier_);
bool has_value = false;
for (const auto &element : list) {
frame_->at(symbol) = element;
auto result = any.where_->expression_->Accept(*this);
if (!result.IsNull() && result.type() != TypedValue::Type::Bool) {
throw ExpressionRuntimeException("Predicate of ANY must evaluate to boolean, got {}.", result.type());
}
if (!result.IsNull()) {
has_value = true;
if (result.ValueBool()) {
return TypedValue(true, ctx_->memory);
}
}
}
// Return Null if all elements are Null
if (!has_value) {
return TypedValue(ctx_->memory);
} else {
return TypedValue(false, ctx_->memory);
}
}
TypedValue Visit(None &none) override {
auto list_value = none.list_expression_->Accept(*this);
if (list_value.IsNull()) {
return TypedValue(ctx_->memory);
}
if (list_value.type() != TypedValue::Type::List) {
throw ExpressionRuntimeException("NONE expected a list, got {}.", list_value.type());
}
const auto &list = list_value.ValueList();
const auto &symbol = symbol_table_->at(*none.identifier_);
bool has_value = false;
for (const auto &element : list) {
frame_->at(symbol) = element;
auto result = none.where_->expression_->Accept(*this);
if (!result.IsNull() && result.type() != TypedValue::Type::Bool) {
throw ExpressionRuntimeException("Predicate of NONE must evaluate to boolean, got {}.", result.type());
}
if (!result.IsNull()) {
has_value = true;
if (result.ValueBool()) {
return TypedValue(false, ctx_->memory);
}
}
}
// Return Null if all elements are Null
if (!has_value) {
return TypedValue(ctx_->memory);
} else {
return TypedValue(true, ctx_->memory);
}
}
TypedValue Visit(ParameterLookup &param_lookup) override {
return TypedValue(conv_(ctx_->parameters.AtTokenPosition(param_lookup.token_position_)), ctx_->memory);
}
TypedValue Visit(RegexMatch &regex_match) override {
auto target_string_value = regex_match.string_expr_->Accept(*this);
auto regex_value = regex_match.regex_->Accept(*this);
if (target_string_value.IsNull() || regex_value.IsNull()) {
return TypedValue(ctx_->memory);
}
if (regex_value.type() != TypedValue::Type::String) {
throw ExpressionRuntimeException("Regular expression must evaluate to a string, got {}.", regex_value.type());
}
if (target_string_value.type() != TypedValue::Type::String) {
// Instead of error, we return Null which makes it compatible in case we
// use indexed lookup which filters out any non-string properties.
// Assuming a property lookup is the target_string_value.
return TypedValue(ctx_->memory);
}
const auto &target_string = target_string_value.ValueString();
try {
std::regex regex(regex_value.ValueString());
return TypedValue(std::regex_match(target_string, regex), ctx_->memory);
} catch (const std::regex_error &e) {
throw ExpressionRuntimeException("Regex error in '{}': {}", regex_value.ValueString(), e.what());
}
}
private:
template <class TRecordAccessor, class TTag = Tag,
class TReturnType = std::enable_if_t<std::is_same_v<TTag, QueryEngineTag>, TypedValue>>
TReturnType GetProperty(const TRecordAccessor &record_accessor, PropertyIx prop) {
auto maybe_prop = record_accessor.GetProperty(prop.name);
// Handler non existent property
return conv_(maybe_prop, dba_);
}
template <class TRecordAccessor, class TTag = Tag,
class TReturnType = std::enable_if_t<std::is_same_v<TTag, QueryEngineTag>, TypedValue>>
TReturnType GetProperty(const TRecordAccessor &record_accessor, const std::string_view name) {
auto maybe_prop = record_accessor.GetProperty(std::string(name));
// Handler non existent property
return conv_(maybe_prop, dba_);
}
template <class TRecordAccessor, class TTag = Tag,
class TReturnType = std::enable_if_t<std::is_same_v<TTag, StorageTag>, TypedValue>>
TypedValue GetProperty(const TRecordAccessor &record_accessor, PropertyIx prop) {
auto maybe_prop = record_accessor.GetProperty(view_, ctx_->properties[prop.ix]);
if (maybe_prop.HasError() && maybe_prop.GetError() == Error::NONEXISTENT_OBJECT) {
// This is a very nasty and temporary hack in order to make MERGE work.
// The old storage had the following logic when returning an `OLD` view:
// `return old ? old : new`. That means that if the `OLD` view didn't
// exist, it returned the NEW view. With this hack we simulate that
// behavior.
// TODO (mferencevic, teon.banek): Remove once MERGE is reimplemented.
maybe_prop = record_accessor.GetProperty(StorageView::NEW, ctx_->properties[prop.ix]);
}
if (maybe_prop.HasError()) {
HandleObjectAccessError(maybe_prop.GetError().code, "property");
}
return conv_(*maybe_prop, ctx_->memory);
}
template <class TRecordAccessor, class TTag = Tag,
class TReturnType = std::enable_if_t<std::is_same_v<TTag, StorageTag>, TypedValue>>
TypedValue GetProperty(const TRecordAccessor &record_accessor, const std::string_view name) {
auto maybe_prop = record_accessor.GetProperty(view_, dba_->NameToProperty(name));
if (maybe_prop.HasError() && maybe_prop.GetError() == Error::NONEXISTENT_OBJECT) {
// This is a very nasty and temporary hack in order to make MERGE work.
// The old storage had the following logic when returning an `OLD` view:
// `return old ? old : new`. That means that if the `OLD` view didn't
// exist, it returned the NEW view. With this hack we simulate that
// behavior.
// TODO (mferencevic, teon.banek): Remove once MERGE is reimplemented.
maybe_prop = record_accessor.GetProperty(view_, dba_->NameToProperty(name));
}
if (maybe_prop.HasError()) {
HandleObjectAccessError(maybe_prop.GetError().code, "property");
}
return conv_(*maybe_prop, ctx_->memory);
}
LabelId GetLabel(LabelIx label) { return ctx_->labels[label.ix]; }
Frame<TypedValue> *frame_;
const SymbolTable *symbol_table_;
const EvaluationContext *ctx_;
DbAccessor *dba_;
// which switching approach should be used when evaluating
StorageView view_;
ConvFunctor conv_;
};
/// A helper function for evaluating an expression that's an int.
///
/// @param what - Name of what's getting evaluated. Used for user feedback (via
/// exception) when the evaluated value is not an int.
/// @throw ExpressionRuntimeException if expression doesn't evaluate to an int.
template <typename ExpressionEvaluator>
int64_t EvaluateInt(ExpressionEvaluator *evaluator, Expression *expr, const std::string &what) {
TypedValue value = expr->Accept(*evaluator);
try {
return value.ValueInt();
} catch (TypedValueException &e) {
throw ExpressionRuntimeException(what + " must be an int");
}
}
template <typename ExpressionEvaluator>
std::optional<size_t> EvaluateMemoryLimit(ExpressionEvaluator *eval, Expression *memory_limit, size_t memory_scale) {
if (!memory_limit) return std::nullopt;
auto limit_value = memory_limit->Accept(*eval);
if (!limit_value.IsInt() || limit_value.ValueInt() <= 0)
throw ExpressionRuntimeException("Memory limit must be a non-negative integer.");
size_t limit = limit_value.ValueInt();
if (std::numeric_limits<size_t>::max() / memory_scale < limit)
throw ExpressionRuntimeException("Memory limit overflow.");
return limit * memory_scale;
}
} // namespace memgraph::expr

View File

@@ -0,0 +1,45 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <vector>
#include "expr/semantic/symbol_table.hpp"
#include "utils/logging.hpp"
#include "utils/memory.hpp"
#include "utils/pmr/vector.hpp"
namespace memgraph::expr {
template <typename TypedValue>
class Frame {
public:
/// Create a Frame of given size backed by a utils::NewDeleteResource()
explicit Frame(int64_t size) : elems_(size, utils::NewDeleteResource()) { MG_ASSERT(size >= 0); }
Frame(int64_t size, utils::MemoryResource *memory) : elems_(size, memory) { MG_ASSERT(size >= 0); }
TypedValue &operator[](const Symbol &symbol) { return elems_[symbol.position()]; }
const TypedValue &operator[](const Symbol &symbol) const { return elems_[symbol.position()]; }
TypedValue &at(const Symbol &symbol) { return elems_.at(symbol.position()); }
const TypedValue &at(const Symbol &symbol) const { return elems_.at(symbol.position()); }
auto &elems() { return elems_; }
utils::MemoryResource *GetMemoryResource() const { return elems_.get_allocator().GetMemoryResource(); }
private:
utils::pmr::vector<TypedValue> elems_;
};
} // namespace memgraph::expr

184
src/expr/parsing.cpp Normal file
View File

@@ -0,0 +1,184 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "expr/parsing.hpp"
#include <cctype>
#include <codecvt>
#include <locale>
#include <stdexcept>
#include "expr/exceptions.hpp"
#include "utils/logging.hpp"
#include "utils/string.hpp"
namespace memgraph::expr {
int64_t ParseIntegerLiteral(const std::string &s) {
try {
// Not really correct since long long can have a bigger range than int64_t.
return static_cast<int64_t>(std::stoll(s, 0, 0));
} catch (const std::out_of_range &) {
throw SemanticException("Integer literal exceeds 64 bits.");
}
}
std::string ParseStringLiteral(const std::string &s) {
// These functions is declared as lambda since its semantics is highly
// specific for this conxtext and shouldn't be used elsewhere.
auto EncodeEscapedUnicodeCodepointUtf32 = [](const std::string &s, int &i) {
const int kLongUnicodeLength = 8;
int j = i + 1;
while (j < static_cast<int>(s.size()) - 1 && j < i + kLongUnicodeLength + 1 && isxdigit(s[j])) {
++j;
}
if (j - i == kLongUnicodeLength + 1) {
char32_t t = stoi(s.substr(i + 1, kLongUnicodeLength), 0, 16);
i += kLongUnicodeLength;
std::wstring_convert<std::codecvt_utf8<char32_t>, char32_t> converter;
return converter.to_bytes(t);
}
throw SyntaxException(
"Expected 8 hex digits as unicode codepoint started with \\U. "
"Use \\u for 4 hex digits format.");
};
auto EncodeEscapedUnicodeCodepointUtf16 = [](const std::string &s, int &i) {
const int kShortUnicodeLength = 4;
int j = i + 1;
while (j < static_cast<int>(s.size()) - 1 && j < i + kShortUnicodeLength + 1 && isxdigit(s[j])) {
++j;
}
if (j - i >= kShortUnicodeLength + 1) {
char16_t t = stoi(s.substr(i + 1, kShortUnicodeLength), 0, 16);
if (t >= 0xD800 && t <= 0xDBFF) {
// t is high surrogate pair. Expect one more utf16 codepoint.
j = i + kShortUnicodeLength + 1;
if (j >= static_cast<int>(s.size()) - 1 || s[j] != '\\') {
throw SemanticException("Invalid UTF codepoint.");
}
++j;
if (j >= static_cast<int>(s.size()) - 1 || (s[j] != 'u' && s[j] != 'U')) {
throw SemanticException("Invalid UTF codepoint.");
}
++j;
int k = j;
while (k < static_cast<int>(s.size()) - 1 && k < j + kShortUnicodeLength && isxdigit(s[k])) {
++k;
}
if (k != j + kShortUnicodeLength) {
throw SemanticException("Invalid UTF codepoint.");
}
char16_t surrogates[3] = {t, static_cast<char16_t>(stoi(s.substr(j, kShortUnicodeLength), 0, 16)), 0};
i += kShortUnicodeLength + 2 + kShortUnicodeLength;
std::wstring_convert<std::codecvt_utf8_utf16<char16_t>, char16_t> converter;
return converter.to_bytes(surrogates);
} else {
i += kShortUnicodeLength;
std::wstring_convert<std::codecvt_utf8_utf16<char16_t>, char16_t> converter;
return converter.to_bytes(t);
}
}
throw SyntaxException(
"Expected 4 hex digits as unicode codepoint started with \\u. "
"Use \\U for 8 hex digits format.");
};
std::string unescaped;
bool escape = false;
// First and last char is quote, we don't need to look at them.
for (int i = 1; i < static_cast<int>(s.size()) - 1; ++i) {
if (escape) {
switch (s[i]) {
case '\\':
unescaped += '\\';
break;
case '\'':
unescaped += '\'';
break;
case '"':
unescaped += '"';
break;
case 'B':
case 'b':
unescaped += '\b';
break;
case 'F':
case 'f':
unescaped += '\f';
break;
case 'N':
case 'n':
unescaped += '\n';
break;
case 'R':
case 'r':
unescaped += '\r';
break;
case 'T':
case 't':
unescaped += '\t';
break;
case 'U':
try {
unescaped += EncodeEscapedUnicodeCodepointUtf32(s, i);
} catch (const std::range_error &) {
throw SemanticException("Invalid UTF codepoint.");
}
break;
case 'u':
try {
unescaped += EncodeEscapedUnicodeCodepointUtf16(s, i);
} catch (const std::range_error &) {
throw SemanticException("Invalid UTF codepoint.");
}
break;
default:
// This should never happen, except grammar changes and we don't
// notice change in this production.
DLOG_FATAL("can't happen");
throw std::exception();
}
escape = false;
} else if (s[i] == '\\') {
escape = true;
} else {
unescaped += s[i];
}
}
return unescaped;
}
double ParseDoubleLiteral(const std::string &s) {
try {
return utils::ParseDouble(s);
} catch (const utils::BasicException &) {
throw SemanticException("Couldn't parse string to double.");
}
}
std::string ParseParameter(const std::string &s) {
DMG_ASSERT(s[0] == '$', "Invalid string passed as parameter name");
if (s[1] != '`') return s.substr(1);
// If parameter name is escaped symbolic name then symbolic name should be
// unescaped and leading and trailing backquote should be removed.
DMG_ASSERT(s.size() > 3U && s.back() == '`', "Invalid string passed as parameter name");
std::string out;
for (int i = 2; i < static_cast<int>(s.size()) - 1; ++i) {
if (s[i] == '`') {
++i;
}
out.push_back(s[i]);
}
return out;
}
} // namespace memgraph::expr

27
src/expr/parsing.hpp Normal file
View File

@@ -0,0 +1,27 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
/// @file
#pragma once
#include <cstdint>
#include <string>
namespace memgraph::expr {
// These are the functions for parsing literals and parameter names from
// opencypher query.
int64_t ParseIntegerLiteral(const std::string &s);
std::string ParseStringLiteral(const std::string &s);
double ParseDoubleLiteral(const std::string &s);
std::string ParseParameter(const std::string &s);
} // namespace memgraph::expr

View File

@@ -0,0 +1,87 @@
;; Copyright 2022 Memgraph Ltd.
;;
;; Use of this software is governed by the Business Source License
;; included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
;; License, and you may not use this file except in compliance with the Business Source License.
;;
;; As of the Change Date specified in that file, in accordance with
;; the Business Source License, use of this software will be governed
;; by the Apache License, Version 2.0, included in the file
;; licenses/APL.txt.
#>cpp
#pragma once
#include <string>
#include "utils/typeinfo.hpp"
cpp<#
(lcp:namespace memgraph)
(lcp:namespace expr)
(lcp:define-class symbol ()
((name "std::string" :scope :public)
(position :int64_t :scope :public)
(user-declared :bool :initval "true" :scope :public)
(type "Type" :initval "Type::ANY" :scope :public)
(token-position :int64_t :initval "-1" :scope :public))
(:public
;; This is similar to TypedValue::Type, but this has `Any` type.
;; TODO: Make a better Type structure which can store a generic List.
(lcp:define-enum type (any vertex edge path number edge-list)
(:serialize))
#>cpp
// TODO: Generate enum to string conversion from LCP. Note, that this is
// displayed to the end user, so we may want to have a pretty name of each
// value.
static std::string TypeToString(Type type) {
const char *enum_string[] = {"Any", "Vertex", "Edge",
"Path", "Number", "EdgeList"};
return enum_string[static_cast<int>(type)];
}
Symbol() {}
Symbol(const std::string &name, int position, bool user_declared,
Type type = Type::ANY, int token_position = -1)
: name_(name),
position_(position),
user_declared_(user_declared),
type_(type),
token_position_(token_position) {}
bool operator==(const Symbol &other) const {
return position_ == other.position_ && name_ == other.name_ &&
type_ == other.type_;
}
bool operator!=(const Symbol &other) const { return !operator==(other); }
// TODO: Remove these since members are public
const auto &name() const { return name_; }
int position() const { return position_; }
Type type() const { return type_; }
bool user_declared() const { return user_declared_; }
int token_position() const { return token_position_; }
cpp<#)
(:serialize (:slk)))
(lcp:pop-namespace) ;; expr
(lcp:pop-namespace) ;; memgraph
#>cpp
namespace std {
template <>
struct hash<memgraph::expr::Symbol> {
size_t operator()(const memgraph::expr::Symbol &symbol) const {
size_t prime = 265443599u;
size_t hash = std::hash<int>{}(symbol.position());
hash ^= prime * std::hash<std::string>{}(symbol.name());
hash ^= prime * std::hash<int>{}(static_cast<int>(symbol.type()));
return hash;
}
};
} // namespace std
cpp<#

View File

@@ -0,0 +1,712 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
// Copyright 2017 Memgraph
//
// Created by Teon Banek on 11-03-2017
#pragma once
#include <algorithm>
#include <optional>
#include <ranges>
#include <unordered_map>
#include <unordered_set>
#include <variant>
#include <vector>
#include "expr/ast.hpp"
#include "expr/ast/ast_visitor.hpp"
#include "expr/exceptions.hpp"
#include "expr/semantic/symbol_table.hpp"
namespace memgraph::expr {
namespace detail {
inline std::unordered_map<std::string, Identifier *> GeneratePredefinedIdentifierMap(
const std::vector<Identifier *> &predefined_identifiers) {
std::unordered_map<std::string, Identifier *> identifier_map;
for (const auto &identifier : predefined_identifiers) {
identifier_map.emplace(identifier->name_, identifier);
}
return identifier_map;
}
} // namespace detail
/// Visits the AST and generates symbols for variables.
///
/// During the process of symbol generation, simple semantic checks are
/// performed. Such as, redeclaring a variable or conflicting expectations of
/// variable types.
class SymbolGenerator : public HierarchicalTreeVisitor {
public:
explicit SymbolGenerator(SymbolTable *symbol_table, const std::vector<Identifier *> &predefined_identifiers)
: symbol_table_(symbol_table),
predefined_identifiers_{detail::GeneratePredefinedIdentifierMap(predefined_identifiers)},
scopes_(1, Scope()) {}
using HierarchicalTreeVisitor::PostVisit;
using HierarchicalTreeVisitor::PreVisit;
using HierarchicalTreeVisitor::Visit;
using typename HierarchicalTreeVisitor::ReturnType;
// Query
bool PreVisit(SingleQuery & /*unused*/) override {
prev_return_names_ = curr_return_names_;
curr_return_names_.clear();
return true;
}
// Union
bool PreVisit(CypherUnion & /*unused*/) override {
scopes_.back() = Scope();
return true;
}
bool PostVisit(CypherUnion &cypher_union) override {
if (prev_return_names_ != curr_return_names_) {
throw SemanticException("All subqueries in an UNION must have the same column names.");
}
// create new symbols for the result of the union
for (const auto &name : curr_return_names_) {
auto symbol = CreateSymbol(name, false);
cypher_union.union_symbols_.push_back(symbol);
}
return true;
}
// Clauses
bool PreVisit(Create & /*unused*/) override {
scopes_.back().in_create = true;
return true;
}
bool PostVisit(Create & /*unused*/) override {
scopes_.back().in_create = false;
return true;
}
bool PreVisit(CallProcedure &call_proc) override {
for (auto *expr : call_proc.arguments_) {
expr->Accept(*this);
}
return false;
}
bool PostVisit(CallProcedure &call_proc) override {
for (auto *ident : call_proc.result_identifiers_) {
if (HasSymbolLocalScope(ident->name_)) {
throw RedeclareVariableError(ident->name_);
}
ident->MapTo(CreateSymbol(ident->name_, true));
}
return true;
}
bool PreVisit(LoadCsv & /*unused*/) override { return false; }
bool PostVisit(LoadCsv &load_csv) override {
if (HasSymbolLocalScope(load_csv.row_var_->name_)) {
throw RedeclareVariableError(load_csv.row_var_->name_);
}
load_csv.row_var_->MapTo(CreateSymbol(load_csv.row_var_->name_, true));
return true;
}
bool PreVisit(Return &ret) override {
auto &scope = scopes_.back();
scope.in_return = true;
VisitReturnBody(ret.body_);
scope.in_return = false;
return false; // We handled the traversal ourselves.
}
bool PostVisit(Return & /*unused*/) override {
for (const auto &name_symbol : scopes_.back().symbols) curr_return_names_.insert(name_symbol.first);
return true;
}
bool PreVisit(With &with) override {
auto &scope = scopes_.back();
scope.in_with = true;
VisitReturnBody(with.body_, with.where_);
scope.in_with = false;
return false; // We handled the traversal ourselves.
}
bool PreVisit(Where & /*unused*/) override {
scopes_.back().in_where = true;
return true;
}
bool PostVisit(Where & /*unused*/) override {
scopes_.back().in_where = false;
return true;
}
bool PreVisit(Merge & /*unused*/) override {
scopes_.back().in_merge = true;
return true;
}
bool PostVisit(Merge & /*unused*/) override {
scopes_.back().in_merge = false;
return true;
}
bool PostVisit(Unwind &unwind) override {
const auto &name = unwind.named_expression_->name_;
if (HasSymbolLocalScope(name)) {
throw RedeclareVariableError(name);
}
unwind.named_expression_->MapTo(CreateSymbol(name, true));
return true;
}
bool PreVisit(Match & /*unused*/) override {
scopes_.back().in_match = true;
return true;
}
bool PostVisit(Match & /*unused*/) override {
auto &scope = scopes_.back();
scope.in_match = false;
// Check variables in property maps after visiting Match, so that they can
// reference symbols out of bind order.
for (auto &ident : scope.identifiers_in_match) {
if (!HasSymbolLocalScope(ident->name_) && !ConsumePredefinedIdentifier(ident->name_))
throw UnboundVariableError(ident->name_);
ident->MapTo(scope.symbols[ident->name_]);
}
scope.identifiers_in_match.clear();
return true;
}
bool PreVisit(Foreach &for_each) override {
const auto &name = for_each.named_expression_->name_;
scopes_.emplace_back(Scope());
scopes_.back().in_foreach = true;
for_each.named_expression_->MapTo(
CreateSymbol(name, true, Symbol::Type::ANY, for_each.named_expression_->token_position_));
return true;
}
bool PostVisit(Foreach & /*unused*/) override {
scopes_.pop_back();
return true;
}
// Expressions
ReturnType Visit(Identifier &ident) override {
auto &scope = scopes_.back();
if (scope.in_skip || scope.in_limit) {
throw SemanticException("Variables are not allowed in {}.", scope.in_skip ? "SKIP" : "LIMIT");
}
Symbol symbol;
if (scope.in_pattern && !(scope.in_node_atom || scope.visiting_edge)) {
// If we are in the pattern, and outside of a node or an edge, the
// identifier is the pattern name.
symbol = GetOrCreateSymbolLocalScope(ident.name_, ident.user_declared_, Symbol::Type::PATH);
} else if (scope.in_pattern && scope.in_pattern_atom_identifier) {
// Patterns used to create nodes and edges cannot redeclare already
// established bindings. Declaration only happens in single node
// patterns and in edge patterns. OpenCypher example,
// `MATCH (n) CREATE (n)` should throw an error that `n` is already
// declared. While `MATCH (n) CREATE (n) -[:R]-> (n)` is allowed,
// since `n` now references the bound node instead of declaring it.
if ((scope.in_create_node || scope.in_create_edge) && HasSymbolLocalScope(ident.name_)) {
throw RedeclareVariableError(ident.name_);
}
auto type = Symbol::Type::VERTEX;
if (scope.visiting_edge) {
// Edge referencing is not allowed (like in Neo4j):
// `MATCH (n) - [r] -> (n) - [r] -> (n) RETURN r` is not allowed.
if (HasSymbolLocalScope(ident.name_)) {
throw RedeclareVariableError(ident.name_);
}
type = scope.visiting_edge->IsVariable() ? Symbol::Type::EDGE_LIST : Symbol::Type::EDGE;
}
symbol = GetOrCreateSymbolLocalScope(ident.name_, ident.user_declared_, type);
} else if (scope.in_pattern && !scope.in_pattern_atom_identifier && scope.in_match) {
if (scope.in_edge_range && scope.visiting_edge->identifier_->name_ == ident.name_) {
// Prevent variable path bounds to reference the identifier which is bound
// by the variable path itself.
throw UnboundVariableError(ident.name_);
}
// Variables in property maps or bounds of variable length path during MATCH
// can reference symbols bound later in the same MATCH. We collect them
// here, so that they can be checked after visiting Match.
scope.identifiers_in_match.emplace_back(&ident);
} else {
// Everything else references a bound symbol.
if (!HasSymbol(ident.name_) && !ConsumePredefinedIdentifier(ident.name_)) throw UnboundVariableError(ident.name_);
symbol = GetOrCreateSymbol(ident.name_, ident.user_declared_, Symbol::Type::ANY);
}
ident.MapTo(symbol);
return true;
}
ReturnType Visit(PrimitiveLiteral & /*unused*/) override { return true; }
ReturnType Visit(ParameterLookup & /*unused*/) override { return true; }
bool PreVisit(Aggregation &aggr) override {
auto &scope = scopes_.back();
// Check if the aggregation can be used in this context. This check should
// probably move to a separate phase, which checks if the query is well
// formed.
if ((!scope.in_return && !scope.in_with) || scope.in_order_by || scope.in_skip || scope.in_limit ||
scope.in_where) {
throw SemanticException("Aggregation functions are only allowed in WITH and RETURN.");
}
if (scope.in_aggregation) {
throw SemanticException(
"Using aggregation functions inside aggregation functions is not "
"allowed.");
}
if (scope.num_if_operators) {
// Neo allows aggregations here and produces very interesting behaviors.
// To simplify implementation at this moment we decided to completely
// disallow aggregations inside of the CASE.
// However, in some cases aggregation makes perfect sense, for example:
// CASE count(n) WHEN 10 THEN "YES" ELSE "NO" END.
// TODO: Rethink of allowing aggregations in some parts of the CASE
// construct.
throw SemanticException("Using aggregation functions inside of CASE is not allowed.");
}
// Create a virtual symbol for aggregation result.
// Currently, we only have aggregation operators which return numbers.
auto aggr_name = Aggregation::OpToString(aggr.op_) + std::to_string(aggr.symbol_pos_);
aggr.MapTo(CreateSymbol(aggr_name, false, Symbol::Type::NUMBER));
scope.in_aggregation = true;
scope.has_aggregation = true;
return true;
}
bool PostVisit(Aggregation & /*unused*/) override {
scopes_.back().in_aggregation = false;
return true;
}
bool PreVisit(IfOperator & /*unused*/) override {
++scopes_.back().num_if_operators;
return true;
}
bool PostVisit(IfOperator & /*unused*/) override {
--scopes_.back().num_if_operators;
return true;
}
bool PreVisit(All &all) override {
all.list_expression_->Accept(*this);
VisitWithIdentifiers(all.where_->expression_, {all.identifier_});
return false;
}
bool PreVisit(Single &single) override {
single.list_expression_->Accept(*this);
VisitWithIdentifiers(single.where_->expression_, {single.identifier_});
return false;
}
bool PreVisit(Any &any) override {
any.list_expression_->Accept(*this);
VisitWithIdentifiers(any.where_->expression_, {any.identifier_});
return false;
}
bool PreVisit(None &none) override {
none.list_expression_->Accept(*this);
VisitWithIdentifiers(none.where_->expression_, {none.identifier_});
return false;
}
bool PreVisit(Reduce &reduce) override {
reduce.initializer_->Accept(*this);
reduce.list_->Accept(*this);
VisitWithIdentifiers(reduce.expression_, {reduce.accumulator_, reduce.identifier_});
return false;
}
bool PreVisit(Extract &extract) override {
extract.list_->Accept(*this);
VisitWithIdentifiers(extract.expression_, {extract.identifier_});
return false;
}
// Pattern and its subparts.
bool PreVisit(Pattern &pattern) override {
auto &scope = scopes_.back();
scope.in_pattern = true;
if ((scope.in_create || scope.in_merge) && pattern.atoms_.size() == 1U) {
MG_ASSERT(utils::IsSubtype(*pattern.atoms_[0], NodeAtom::kType), "Expected a single NodeAtom in Pattern");
scope.in_create_node = true;
}
return true;
}
bool PostVisit(Pattern & /*unused*/) override {
auto &scope = scopes_.back();
scope.in_pattern = false;
scope.in_create_node = false;
return true;
}
bool PreVisit(NodeAtom &node_atom) override {
auto &scope = scopes_.back();
auto check_node_semantic = [&node_atom, &scope, this](const bool props_or_labels) {
const auto &node_name = node_atom.identifier_->name_;
if ((scope.in_create || scope.in_merge) && props_or_labels && HasSymbolLocalScope(node_name)) {
throw SemanticException("Cannot create node '" + node_name +
"' with labels or properties, because it is already declared.");
}
scope.in_pattern_atom_identifier = true;
node_atom.identifier_->Accept(*this);
scope.in_pattern_atom_identifier = false;
};
scope.in_node_atom = true;
if (auto *properties = std::get_if<std::unordered_map<PropertyIx, Expression *>>(&node_atom.properties_)) {
bool props_or_labels = !properties->empty() || !node_atom.labels_.empty();
check_node_semantic(props_or_labels);
for (auto kv : *properties) {
kv.second->Accept(*this);
}
return false;
}
auto &properties_parameter = std::get<ParameterLookup *>(node_atom.properties_);
bool props_or_labels = !properties_parameter || !node_atom.labels_.empty();
check_node_semantic(props_or_labels);
properties_parameter->Accept(*this);
return false;
}
bool PostVisit(NodeAtom & /*unused*/) override {
scopes_.back().in_node_atom = false;
return true;
}
bool PreVisit(EdgeAtom &edge_atom) override {
auto &scope = scopes_.back();
scope.visiting_edge = &edge_atom;
if (scope.in_create || scope.in_merge) {
scope.in_create_edge = true;
if (edge_atom.edge_types_.size() != 1U) {
throw SemanticException(
"A single relationship type must be specified "
"when creating an edge.");
}
if (scope.in_create && // Merge allows bidirectionality
edge_atom.direction_ == EdgeAtom::Direction::BOTH) {
throw SemanticException(
"Bidirectional relationship are not supported "
"when creating an edge");
}
if (edge_atom.IsVariable()) {
throw SemanticException(
"Variable length relationships are not supported when creating an "
"edge.");
}
}
if (auto *properties = std::get_if<std::unordered_map<PropertyIx, Expression *>>(&edge_atom.properties_)) {
for (auto kv : *properties) {
kv.second->Accept(*this);
}
} else {
std::get<ParameterLookup *>(edge_atom.properties_)->Accept(*this);
}
if (edge_atom.IsVariable()) {
scope.in_edge_range = true;
if (edge_atom.lower_bound_) {
edge_atom.lower_bound_->Accept(*this);
}
if (edge_atom.upper_bound_) {
edge_atom.upper_bound_->Accept(*this);
}
scope.in_edge_range = false;
scope.in_pattern = false;
if (edge_atom.filter_lambda_.expression) {
VisitWithIdentifiers(edge_atom.filter_lambda_.expression,
{edge_atom.filter_lambda_.inner_edge, edge_atom.filter_lambda_.inner_node});
} else {
// Create inner symbols, but don't bind them in scope, since they are to
// be used in the missing filter expression.
auto *inner_edge = edge_atom.filter_lambda_.inner_edge;
inner_edge->MapTo(
symbol_table_->CreateSymbol(inner_edge->name_, inner_edge->user_declared_, Symbol::Type::EDGE));
auto *inner_node = edge_atom.filter_lambda_.inner_node;
inner_node->MapTo(
symbol_table_->CreateSymbol(inner_node->name_, inner_node->user_declared_, Symbol::Type::VERTEX));
}
if (edge_atom.weight_lambda_.expression) {
VisitWithIdentifiers(edge_atom.weight_lambda_.expression,
{edge_atom.weight_lambda_.inner_edge, edge_atom.weight_lambda_.inner_node});
}
scope.in_pattern = true;
}
scope.in_pattern_atom_identifier = true;
edge_atom.identifier_->Accept(*this);
scope.in_pattern_atom_identifier = false;
if (edge_atom.total_weight_) {
if (HasSymbolLocalScope(edge_atom.total_weight_->name_)) {
throw RedeclareVariableError(edge_atom.total_weight_->name_);
}
edge_atom.total_weight_->MapTo(GetOrCreateSymbolLocalScope(
edge_atom.total_weight_->name_, edge_atom.total_weight_->user_declared_, Symbol::Type::NUMBER));
}
return false;
}
bool PostVisit(EdgeAtom & /*unused*/) override {
auto &scope = scopes_.back();
scope.visiting_edge = nullptr;
scope.in_create_edge = false;
return true;
}
private:
// Scope stores the state of where we are when visiting the AST and a map of
// names to symbols.
struct Scope {
bool in_pattern{false};
bool in_merge{false};
bool in_create{false};
// in_create_node is true if we are creating or merging *only* a node.
// Therefore, it is *not* equivalent to (in_create || in_merge) &&
// in_node_atom.
bool in_create_node{false};
// True if creating an edge;
// shortcut for (in_create || in_merge) && visiting_edge.
bool in_create_edge{false};
bool in_node_atom{false};
EdgeAtom *visiting_edge{nullptr};
bool in_aggregation{false};
bool in_return{false};
bool in_with{false};
bool in_skip{false};
bool in_limit{false};
bool in_order_by{false};
bool in_where{false};
bool in_match{false};
bool in_foreach{false};
// True when visiting a pattern atom (node or edge) identifier, which can be
// reused or created in the pattern itself.
bool in_pattern_atom_identifier{false};
// True when visiting range bounds of a variable path.
bool in_edge_range{false};
// True if the return/with contains an aggregation in any named expression.
bool has_aggregation{false};
// Map from variable names to symbols.
std::map<std::string, Symbol> symbols;
// Identifiers found in property maps of patterns or as variable length path
// bounds in a single Match clause. They need to be checked after visiting
// Match. Identifiers created by naming vertices, edges and paths are *not*
// stored in here.
std::vector<Identifier *> identifiers_in_match;
// Number of nested IfOperators.
int num_if_operators{0};
};
inline static std::optional<Symbol> FindSymbolInScope(const std::string &name, const Scope &scope,
Symbol::Type type) {
if (auto it = scope.symbols.find(name); it != scope.symbols.end()) {
const auto &symbol = it->second;
// Unless we have `ANY` type, check that types match.
if (type != Symbol::Type::ANY && symbol.type() != Symbol::Type::ANY && type != symbol.type()) {
throw TypeMismatchError(name, Symbol::TypeToString(symbol.type()), Symbol::TypeToString(type));
}
return symbol;
}
return std::nullopt;
}
bool HasSymbol(const std::string &name) const {
return std::ranges::any_of(scopes_, [&name](const auto &scope) { return scope.symbols.contains(name); });
}
bool HasSymbolLocalScope(const std::string &name) const { return scopes_.back().symbols.contains(name); }
// @return true if it added a predefined identifier with that name
bool ConsumePredefinedIdentifier(const std::string &name) {
auto it = predefined_identifiers_.find(name);
if (it == predefined_identifiers_.end()) {
return false;
}
// we can only use the predefined identifier in a single scope so we remove it after creating
// a symbol for it
auto &identifier = it->second;
MG_ASSERT(!identifier->user_declared_, "Predefined symbols cannot be user declared!");
identifier->MapTo(CreateSymbol(identifier->name_, identifier->user_declared_));
predefined_identifiers_.erase(it);
return true;
}
// Returns a freshly generated symbol. Previous mapping of the same name to a
// different symbol is replaced with the new one.
Symbol CreateSymbol(const std::string &name, bool user_declared, Symbol::Type type = Symbol::Type::ANY,
int token_position = -1) {
auto symbol = symbol_table_->CreateSymbol(name, user_declared, type, token_position);
scopes_.back().symbols[name] = symbol;
return symbol;
}
Symbol GetOrCreateSymbol(const std::string &name, bool user_declared, Symbol::Type type = Symbol::Type::ANY) {
// NOLINTNEXTLINE
for (auto scope = scopes_.rbegin(); scope != scopes_.rend(); ++scope) {
if (auto maybe_symbol = FindSymbolInScope(name, *scope, type); maybe_symbol) {
return *maybe_symbol;
}
}
return CreateSymbol(name, user_declared, type);
}
// Returns the symbol by name. If the mapping already exists, checks if the
// types match. Otherwise, returns a new symbol.
Symbol GetOrCreateSymbolLocalScope(const std::string &name, bool user_declared,
Symbol::Type type = Symbol::Type::ANY) {
auto &scope = scopes_.back();
if (auto maybe_symbol = FindSymbolInScope(name, scope, type); maybe_symbol) {
return *maybe_symbol;
}
return CreateSymbol(name, user_declared, type);
}
void VisitReturnBody(ReturnBody &body, Where *where = nullptr) {
auto &scope = scopes_.back();
for (auto &expr : body.named_expressions) {
expr->Accept(*this);
}
std::vector<Symbol> user_symbols;
if (body.all_identifiers) {
// Carry over user symbols because '*' appeared.
for (const auto &sym_pair : scope.symbols) {
if (!sym_pair.second.user_declared()) {
continue;
}
user_symbols.emplace_back(sym_pair.second);
}
if (user_symbols.empty()) {
throw SemanticException("There are no variables in scope to use for '*'.");
}
}
// WITH/RETURN clause removes declarations of all the previous variables and
// declares only those established through named expressions. New declarations
// must not be visible inside named expressions themselves.
bool removed_old_names = false;
if ((!where && body.order_by.empty()) || scope.has_aggregation) {
// WHERE and ORDER BY need to see both the old and new symbols, unless we
// have an aggregation. Therefore, we can clear the symbols immediately if
// there is neither ORDER BY nor WHERE, or we have an aggregation.
scope.symbols.clear();
removed_old_names = true;
}
// Create symbols for named expressions.
std::unordered_set<std::string> new_names;
for (const auto &user_sym : user_symbols) {
new_names.insert(user_sym.name());
scope.symbols[user_sym.name()] = user_sym;
}
for (auto &named_expr : body.named_expressions) {
const auto &name = named_expr->name_;
if (!new_names.insert(name).second) {
throw SemanticException("Multiple results with the same name '{}' are not allowed.", name);
}
// An improvement would be to infer the type of the expression, so that the
// new symbol would have a more specific type.
named_expr->MapTo(CreateSymbol(name, true, Symbol::Type::ANY, named_expr->token_position_));
}
scope.in_order_by = true;
for (const auto &order_pair : body.order_by) {
order_pair.expression->Accept(*this);
}
scope.in_order_by = false;
if (body.skip) {
scope.in_skip = true;
body.skip->Accept(*this);
scope.in_skip = false;
}
if (body.limit) {
scope.in_limit = true;
body.limit->Accept(*this);
scope.in_limit = false;
}
if (where) where->Accept(*this);
if (!removed_old_names) {
// We have an ORDER BY or WHERE, but no aggregation, which means we didn't
// clear the old symbols, so do it now. We cannot just call clear, because
// we've added new symbols.
for (auto sym_it = scope.symbols.begin(); sym_it != scope.symbols.end();) {
if (new_names.find(sym_it->first) == new_names.end()) {
sym_it = scope.symbols.erase(sym_it);
} else {
sym_it++;
}
}
}
scopes_.back().has_aggregation = false;
}
void VisitWithIdentifiers(Expression *expr, const std::vector<Identifier *> &identifiers) {
auto &scope = scopes_.back();
std::vector<std::pair<std::optional<Symbol>, Identifier *>> prev_symbols;
// Collect previous symbols if they exist.
for (const auto &identifier : identifiers) {
std::optional<Symbol> prev_symbol;
auto prev_symbol_it = scope.symbols.find(identifier->name_);
if (prev_symbol_it != scope.symbols.end()) {
prev_symbol = prev_symbol_it->second;
}
identifier->MapTo(CreateSymbol(identifier->name_, identifier->user_declared_));
prev_symbols.emplace_back(prev_symbol, identifier);
}
// Visit the expression with the new symbols bound.
expr->Accept(*this);
// Restore back to previous symbols.
for (const auto &prev : prev_symbols) {
const auto &prev_symbol = prev.first;
const auto &identifier = prev.second;
if (prev_symbol) {
scope.symbols[identifier->name_] = *prev_symbol;
} else {
scope.symbols.erase(identifier->name_);
}
}
}
SymbolTable *symbol_table_;
// Identifiers which are injected from outside the query. Each identifier
// is mapped by its name.
std::unordered_map<std::string, Identifier *> predefined_identifiers_;
std::vector<Scope> scopes_;
std::unordered_set<std::string> prev_return_names_;
std::unordered_set<std::string> curr_return_names_;
};
inline SymbolTable MakeSymbolTable(CypherQuery *query, const std::vector<Identifier *> &predefined_identifiers = {}) {
SymbolTable symbol_table;
SymbolGenerator symbol_generator(&symbol_table, predefined_identifiers);
query->single_query_->Accept(symbol_generator);
for (auto *cypher_union : query->cypher_unions_) {
cypher_union->Accept(symbol_generator);
}
return symbol_table;
}
} // namespace memgraph::expr

View File

@@ -0,0 +1,64 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <map>
#include <string>
#include "expr/ast.hpp"
#include "expr/semantic/symbol.hpp"
#include "utils/logging.hpp"
namespace memgraph::expr {
class SymbolTable final {
public:
SymbolTable() {}
const Symbol &CreateSymbol(const std::string &name, bool user_declared, Symbol::Type type = Symbol::Type::ANY,
int32_t token_position = -1) {
MG_ASSERT(table_.size() <= std::numeric_limits<int32_t>::max(),
"SymbolTable size doesn't fit into 32-bit integer!");
auto got = table_.emplace(position_, Symbol(name, position_, user_declared, type, token_position));
MG_ASSERT(got.second, "Duplicate symbol ID!");
position_++;
return got.first->second;
}
// TODO(buda): This is the same logic as in the cypher_main_visitor. During
// parsing phase symbol table doesn't exist. Figure out a better solution.
const Symbol &CreateAnonymousSymbol(Symbol::Type type = Symbol::Type::ANY) {
int id = 1;
while (true) {
static const std::string &kAnonPrefix = "anon";
std::string name_candidate = kAnonPrefix + std::to_string(id++);
if (std::find_if(std::begin(table_), std::end(table_), [&name_candidate](const auto &item) -> bool {
return item.second.name_ == name_candidate;
}) == std::end(table_)) {
return CreateSymbol(name_candidate, false, type);
}
}
}
const Symbol &at(const Identifier &ident) const { return table_.at(ident.symbol_pos_); }
const Symbol &at(const NamedExpression &nexpr) const { return table_.at(nexpr.symbol_pos_); }
const Symbol &at(const Aggregation &aggr) const { return table_.at(aggr.symbol_pos_); }
// TODO: Remove these since members are public
int32_t max_position() const { return static_cast<int32_t>(table_.size()); }
const auto &table() const { return table_; }
int32_t position_{0};
std::map<int32_t, Symbol> table_;
};
} // namespace memgraph::expr

1505
src/expr/typed_value.hpp Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,26 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "utils/exceptions.hpp"
namespace memgraph::expr {
/**
* An exception raised by the TypedValue system. Typically when
* trying to perform operations (such as addition) on TypedValues
* of incompatible Types.
*/
class TypedValueException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
};
} // namespace memgraph::expr

View File

@@ -0,0 +1 @@
add_library(mg-functions INTERFACE)

File diff suppressed because it is too large Load Diff

64
src/glue/v2/auth.cpp Normal file
View File

@@ -0,0 +1,64 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "glue/v2/auth.hpp"
namespace memgraph::glue::v2 {
auth::Permission PrivilegeToPermission(query::v2::AuthQuery::Privilege privilege) {
switch (privilege) {
case query::v2::AuthQuery::Privilege::MATCH:
return auth::Permission::MATCH;
case query::v2::AuthQuery::Privilege::CREATE:
return auth::Permission::CREATE;
case query::v2::AuthQuery::Privilege::MERGE:
return auth::Permission::MERGE;
case query::v2::AuthQuery::Privilege::DELETE:
return auth::Permission::DELETE;
case query::v2::AuthQuery::Privilege::SET:
return auth::Permission::SET;
case query::v2::AuthQuery::Privilege::REMOVE:
return auth::Permission::REMOVE;
case query::v2::AuthQuery::Privilege::INDEX:
return auth::Permission::INDEX;
case query::v2::AuthQuery::Privilege::STATS:
return auth::Permission::STATS;
case query::v2::AuthQuery::Privilege::CONSTRAINT:
return auth::Permission::CONSTRAINT;
case query::v2::AuthQuery::Privilege::DUMP:
return auth::Permission::DUMP;
case query::v2::AuthQuery::Privilege::REPLICATION:
return auth::Permission::REPLICATION;
case query::v2::AuthQuery::Privilege::DURABILITY:
return auth::Permission::DURABILITY;
case query::v2::AuthQuery::Privilege::READ_FILE:
return auth::Permission::READ_FILE;
case query::v2::AuthQuery::Privilege::FREE_MEMORY:
return auth::Permission::FREE_MEMORY;
case query::v2::AuthQuery::Privilege::TRIGGER:
return auth::Permission::TRIGGER;
case query::v2::AuthQuery::Privilege::CONFIG:
return auth::Permission::CONFIG;
case query::v2::AuthQuery::Privilege::AUTH:
return auth::Permission::AUTH;
case query::v2::AuthQuery::Privilege::STREAM:
return auth::Permission::STREAM;
case query::v2::AuthQuery::Privilege::MODULE_READ:
return auth::Permission::MODULE_READ;
case query::v2::AuthQuery::Privilege::MODULE_WRITE:
return auth::Permission::MODULE_WRITE;
case query::v2::AuthQuery::Privilege::WEBSOCKET:
return auth::Permission::WEBSOCKET;
case query::v2::AuthQuery::Privilege::SCHEMA:
return auth::Permission::SCHEMA;
}
}
} // namespace memgraph::glue::v2

23
src/glue/v2/auth.hpp Normal file
View File

@@ -0,0 +1,23 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "auth/models.hpp"
#include "query/v2/frontend/ast/ast.hpp"
namespace memgraph::glue::v2 {
/**
* This function converts query::AuthQuery::Privilege to its corresponding
* auth::Permission.
*/
auth::Permission PrivilegeToPermission(query::v2::AuthQuery::Privilege privilege);
} // namespace memgraph::glue::v2

View File

@@ -0,0 +1,297 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "glue/v2/communication.hpp"
#include <map>
#include <string>
#include <vector>
#include "common/errors.hpp"
#include "coordinator/shard_map.hpp"
#include "query/v2/accessors.hpp"
#include "query/v2/request_router.hpp"
#include "query/v2/requests.hpp"
#include "storage/v3/edge_accessor.hpp"
#include "storage/v3/id_types.hpp"
#include "storage/v3/shard.hpp"
#include "storage/v3/vertex_accessor.hpp"
#include "storage/v3/view.hpp"
#include "utils/exceptions.hpp"
#include "utils/temporal.hpp"
using memgraph::communication::bolt::Value;
namespace memgraph::glue::v2 {
query::v2::TypedValue ToTypedValue(const Value &value) {
switch (value.type()) {
case Value::Type::Null:
return {};
case Value::Type::Bool:
return query::v2::TypedValue(value.ValueBool());
case Value::Type::Int:
return query::v2::TypedValue(value.ValueInt());
case Value::Type::Double:
return query::v2::TypedValue(value.ValueDouble());
case Value::Type::String:
return query::v2::TypedValue(value.ValueString());
case Value::Type::List: {
std::vector<query::v2::TypedValue> list;
list.reserve(value.ValueList().size());
for (const auto &v : value.ValueList()) list.push_back(ToTypedValue(v));
return query::v2::TypedValue(std::move(list));
}
case Value::Type::Map: {
std::map<std::string, query::v2::TypedValue> map;
for (const auto &kv : value.ValueMap()) map.emplace(kv.first, ToTypedValue(kv.second));
return query::v2::TypedValue(std::move(map));
}
case Value::Type::Vertex:
case Value::Type::Edge:
case Value::Type::UnboundedEdge:
case Value::Type::Path:
throw communication::bolt::ValueException("Unsupported conversion from Value to TypedValue");
case Value::Type::Date:
return query::v2::TypedValue(value.ValueDate());
case Value::Type::LocalTime:
return query::v2::TypedValue(value.ValueLocalTime());
case Value::Type::LocalDateTime:
return query::v2::TypedValue(value.ValueLocalDateTime());
case Value::Type::Duration:
return query::v2::TypedValue(value.ValueDuration());
}
}
communication::bolt::Vertex ToBoltVertex(const query::v2::accessors::VertexAccessor &vertex,
const query::v2::RequestRouterInterface *request_router,
storage::v3::View /*view*/) {
auto id = communication::bolt::Id::FromUint(0);
auto labels = vertex.Labels();
std::vector<std::string> new_labels;
new_labels.reserve(labels.size());
for (const auto &label : labels) {
new_labels.push_back(request_router->LabelToName(label.id));
}
auto properties = vertex.Properties();
std::map<std::string, Value> new_properties;
for (const auto &[prop, property_value] : properties) {
new_properties[request_router->PropertyToName(prop)] = ToBoltValue(property_value);
}
return communication::bolt::Vertex{id, new_labels, new_properties};
}
communication::bolt::Edge ToBoltEdge(const query::v2::accessors::EdgeAccessor &edge,
const query::v2::RequestRouterInterface *request_router,
storage::v3::View /*view*/) {
// TODO(jbajic) Fix bolt communication
auto id = communication::bolt::Id::FromUint(0);
auto from = communication::bolt::Id::FromUint(0);
auto to = communication::bolt::Id::FromUint(0);
const auto &type = request_router->EdgeTypeToName(edge.EdgeType());
auto properties = edge.Properties();
std::map<std::string, Value> new_properties;
for (const auto &[prop, property_value] : properties) {
new_properties[request_router->PropertyToName(prop)] = ToBoltValue(property_value);
}
return communication::bolt::Edge{id, from, to, type, new_properties};
}
communication::bolt::Path ToBoltPath(const query::v2::accessors::Path & /*edge*/,
const query::v2::RequestRouterInterface * /*request_router*/,
storage::v3::View /*view*/) {
// TODO(jbajic) Fix bolt communication
MG_ASSERT(false, "Path is unimplemented!");
return {};
}
Value ToBoltValue(const query::v2::TypedValue &value, const query::v2::RequestRouterInterface *request_router,
storage::v3::View view) {
switch (value.type()) {
case query::v2::TypedValue::Type::Null:
return {};
case query::v2::TypedValue::Type::Bool:
return {value.ValueBool()};
case query::v2::TypedValue::Type::Int:
return {value.ValueInt()};
case query::v2::TypedValue::Type::Double:
return {value.ValueDouble()};
case query::v2::TypedValue::Type::String:
return {std::string(value.ValueString())};
case query::v2::TypedValue::Type::List: {
std::vector<Value> values;
values.reserve(value.ValueList().size());
for (const auto &v : value.ValueList()) {
auto value = ToBoltValue(v, request_router, view);
values.emplace_back(std::move(value));
}
return {std::move(values)};
}
case query::v2::TypedValue::Type::Map: {
std::map<std::string, Value> map;
for (const auto &kv : value.ValueMap()) {
auto value = ToBoltValue(kv.second, request_router, view);
map.emplace(kv.first, std::move(value));
}
return {std::move(map)};
}
case query::v2::TypedValue::Type::Vertex: {
auto vertex = ToBoltVertex(value.ValueVertex(), request_router, view);
return {std::move(vertex)};
}
case query::v2::TypedValue::Type::Edge: {
auto edge = ToBoltEdge(value.ValueEdge(), request_router, view);
return {std::move(edge)};
}
case query::v2::TypedValue::Type::Path: {
auto path = ToBoltPath(value.ValuePath(), request_router, view);
return {std::move(path)};
}
case query::v2::TypedValue::Type::Date:
return {value.ValueDate()};
case query::v2::TypedValue::Type::LocalTime:
return {value.ValueLocalTime()};
case query::v2::TypedValue::Type::LocalDateTime:
return {value.ValueLocalDateTime()};
case query::v2::TypedValue::Type::Duration:
return {value.ValueDuration()};
}
}
Value ToBoltValue(msgs::Value value) {
switch (value.type) {
case msgs::Value::Type::Null:
return {};
case msgs::Value::Type::Bool:
return {value.bool_v};
case msgs::Value::Type::Int64:
return {value.int_v};
case msgs::Value::Type::Double:
return {value.double_v};
case msgs::Value::Type::String:
return {std::string(value.string_v)};
case msgs::Value::Type::List: {
std::vector<Value> values;
values.reserve(value.list_v.size());
for (const auto &v : value.list_v) {
auto maybe_value = ToBoltValue(v);
values.emplace_back(std::move(maybe_value));
}
return Value{std::move(values)};
}
case msgs::Value::Type::Map: {
std::map<std::string, Value> map;
for (const auto &kv : value.map_v) {
auto maybe_value = ToBoltValue(kv.second);
map.emplace(kv.first, std::move(maybe_value));
}
return Value{std::move(map)};
}
case msgs::Value::Type::Vertex:
case msgs::Value::Type::Edge: {
throw utils::BasicException("Vertex and Edge not supported!");
}
// TODO Value to Date types not supported
}
}
storage::v3::PropertyValue ToPropertyValue(const Value &value) {
switch (value.type()) {
case Value::Type::Null:
return {};
case Value::Type::Bool:
return storage::v3::PropertyValue(value.ValueBool());
case Value::Type::Int:
return storage::v3::PropertyValue(value.ValueInt());
case Value::Type::Double:
return storage::v3::PropertyValue(value.ValueDouble());
case Value::Type::String:
return storage::v3::PropertyValue(value.ValueString());
case Value::Type::List: {
std::vector<storage::v3::PropertyValue> vec;
vec.reserve(value.ValueList().size());
for (const auto &value : value.ValueList()) vec.emplace_back(ToPropertyValue(value));
return storage::v3::PropertyValue(std::move(vec));
}
case Value::Type::Map: {
std::map<std::string, storage::v3::PropertyValue> map;
for (const auto &kv : value.ValueMap()) map.emplace(kv.first, ToPropertyValue(kv.second));
return storage::v3::PropertyValue(std::move(map));
}
case Value::Type::Vertex:
case Value::Type::Edge:
case Value::Type::UnboundedEdge:
case Value::Type::Path:
throw communication::bolt::ValueException("Unsupported conversion from Value to PropertyValue");
case Value::Type::Date:
return storage::v3::PropertyValue(
storage::v3::TemporalData(storage::v3::TemporalType::Date, value.ValueDate().MicrosecondsSinceEpoch()));
case Value::Type::LocalTime:
return storage::v3::PropertyValue(storage::v3::TemporalData(storage::v3::TemporalType::LocalTime,
value.ValueLocalTime().MicrosecondsSinceEpoch()));
case Value::Type::LocalDateTime:
return storage::v3::PropertyValue(storage::v3::TemporalData(storage::v3::TemporalType::LocalDateTime,
value.ValueLocalDateTime().MicrosecondsSinceEpoch()));
case Value::Type::Duration:
return storage::v3::PropertyValue(
storage::v3::TemporalData(storage::v3::TemporalType::Duration, value.ValueDuration().microseconds));
}
}
Value ToBoltValue(const storage::v3::PropertyValue &value) {
switch (value.type()) {
case storage::v3::PropertyValue::Type::Null:
return {};
case storage::v3::PropertyValue::Type::Bool:
return {value.ValueBool()};
case storage::v3::PropertyValue::Type::Int:
return {value.ValueInt()};
break;
case storage::v3::PropertyValue::Type::Double:
return {value.ValueDouble()};
case storage::v3::PropertyValue::Type::String:
return {value.ValueString()};
case storage::v3::PropertyValue::Type::List: {
const auto &values = value.ValueList();
std::vector<Value> vec;
vec.reserve(values.size());
for (const auto &v : values) {
vec.push_back(ToBoltValue(v));
}
return {std::move(vec)};
}
case storage::v3::PropertyValue::Type::Map: {
const auto &map = value.ValueMap();
std::map<std::string, Value> dv_map;
for (const auto &kv : map) {
dv_map.emplace(kv.first, ToBoltValue(kv.second));
}
return {std::move(dv_map)};
}
case storage::v3::PropertyValue::Type::TemporalData:
const auto &type = value.ValueTemporalData();
switch (type.type) {
case storage::v3::TemporalType::Date:
return {utils::Date(type.microseconds)};
case storage::v3::TemporalType::LocalTime:
return {utils::LocalTime(type.microseconds)};
case storage::v3::TemporalType::LocalDateTime:
return {utils::LocalDateTime(type.microseconds)};
case storage::v3::TemporalType::Duration:
return {utils::Duration(type.microseconds)};
}
}
}
} // namespace memgraph::glue::v2

View File

@@ -0,0 +1,78 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
/// @file Conversion functions between Value and other memgraph types.
#pragma once
#include "communication/bolt/v1/value.hpp"
#include "coordinator/shard_map.hpp"
#include "query/v2/bindings/typed_value.hpp"
#include "query/v2/request_router.hpp"
#include "storage/v3/property_value.hpp"
#include "storage/v3/result.hpp"
#include "storage/v3/shard.hpp"
#include "storage/v3/view.hpp"
#include "utils/result.hpp"
namespace memgraph::storage::v3 {
class EdgeAccessor;
class Storage;
class VertexAccessor;
} // namespace memgraph::storage::v3
namespace memgraph::glue::v2 {
/// @param storage::v3::VertexAccessor for converting to
/// communication::bolt::Vertex.
/// @param query::v2::RequestRouterInterface *request_router getting label and property names.
/// @param storage::v3::View for deciding which vertex attributes are visible.
///
/// @throw std::bad_alloc
communication::bolt::Vertex ToBoltVertex(const storage::v3::VertexAccessor &vertex,
const query::v2::RequestRouterInterface *request_router,
storage::v3::View view);
/// @param storage::v3::EdgeAccessor for converting to communication::bolt::Edge.
/// @param query::v2::RequestRouterInterface *request_router getting edge type and property names.
/// @param storage::v3::View for deciding which edge attributes are visible.
///
/// @throw std::bad_alloc
communication::bolt::Edge ToBoltEdge(const storage::v3::EdgeAccessor &edge,
const query::v2::RequestRouterInterface *request_router, storage::v3::View view);
/// @param query::v2::Path for converting to communication::bolt::Path.
/// @param query::v2::RequestRouterInterface *request_router ToBoltVertex and ToBoltEdge.
/// @param storage::v3::View for ToBoltVertex and ToBoltEdge.
///
/// @throw std::bad_alloc
communication::bolt::Path ToBoltPath(const query::v2::accessors::Path &path,
const query::v2::RequestRouterInterface *request_router, storage::v3::View view);
/// @param query::v2::TypedValue for converting to communication::bolt::Value.
/// @param query::v2::RequestRouterInterface *request_router ToBoltVertex and ToBoltEdge.
/// @param storage::v3::View for ToBoltVertex and ToBoltEdge.
///
/// @throw std::bad_alloc
communication::bolt::Value ToBoltValue(const query::v2::TypedValue &value,
const query::v2::RequestRouterInterface *request_router, storage::v3::View view);
query::v2::TypedValue ToTypedValue(const communication::bolt::Value &value);
communication::bolt::Value ToBoltValue(const storage::v3::PropertyValue &value);
storage::v3::PropertyValue ToPropertyValue(const communication::bolt::Value &value);
communication::bolt::Value ToBoltValue(msgs::Value value);
communication::bolt::Value ToBoltValue(msgs::Value value, const query::v2::RequestRouterInterface *request_router,
storage::v3::View view);
} // namespace memgraph::glue::v2

163
src/io/address.hpp Normal file
View File

@@ -0,0 +1,163 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <compare>
#include <fmt/format.h>
#include <boost/asio/ip/tcp.hpp>
#include <boost/functional/hash.hpp>
#include <boost/uuid/uuid.hpp>
#include <boost/uuid/uuid_generators.hpp>
#include <boost/uuid/uuid_io.hpp>
#include "utils/logging.hpp"
namespace memgraph::io {
struct PartialAddress {
boost::asio::ip::address ip;
uint16_t port;
friend bool operator==(const PartialAddress &lhs, const PartialAddress &rhs) = default;
/// unique_id is most dominant for ordering, then ip, then port
friend bool operator<(const PartialAddress &lhs, const PartialAddress &rhs) {
if (lhs.ip != rhs.ip) {
return lhs.ip < rhs.ip;
}
return lhs.port < rhs.port;
}
std::string ToString() const { return fmt::format("PartialAddress {{ ip: {}, port: {} }}", ip.to_string(), port); }
friend std::ostream &operator<<(std::ostream &in, const PartialAddress &partial_address) {
in << partial_address.ToString();
return in;
}
};
struct Address {
// It's important for all participants to have a
// unique identifier - IP and port alone are not
// enough, and may change over the lifecycle of
// the nodes. Particularly storage nodes may change
// their IP addresses over time, and the system
// should gracefully update its information
// about them.
boost::uuids::uuid unique_id;
boost::asio::ip::address last_known_ip;
uint16_t last_known_port;
static Address TestAddress(uint16_t port) {
MG_ASSERT(port <= 255);
return Address{
.unique_id = boost::uuids::uuid{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, static_cast<unsigned char>(port)},
.last_known_port = port,
};
}
// NB: don't use this in test code because it is non-deterministic
static Address UniqueLocalAddress() {
return Address{
.unique_id = boost::uuids::uuid{boost::uuids::random_generator()()},
};
}
/// `Coordinator`s have constant UUIDs because there is at most one per ip/port pair.
Address ForkLocalCoordinator() {
return Address{
.unique_id = boost::uuids::uuid{1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
.last_known_ip = last_known_ip,
.last_known_port = last_known_port,
};
}
/// `ShardManager`s have constant UUIDs because there is at most one per ip/port pair.
Address ForkLocalShardManager() {
return Address{
.unique_id = boost::uuids::uuid{2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
.last_known_ip = last_known_ip,
.last_known_port = last_known_port,
};
}
/// Returns a new ID with the same IP and port but a unique UUID.
Address ForkUniqueAddress() {
return Address{
.unique_id = boost::uuids::uuid{boost::uuids::random_generator()()},
.last_known_ip = last_known_ip,
.last_known_port = last_known_port,
};
}
PartialAddress ToPartialAddress() const {
return PartialAddress{
.ip = last_known_ip,
.port = last_known_port,
};
}
friend bool operator==(const Address &lhs, const Address &rhs) = default;
/// unique_id is most dominant for ordering, then last_known_ip, then last_known_port
friend bool operator<(const Address &lhs, const Address &rhs) {
if (lhs.unique_id != rhs.unique_id) {
return lhs.unique_id < rhs.unique_id;
}
if (lhs.last_known_ip != rhs.last_known_ip) {
return lhs.last_known_ip < rhs.last_known_ip;
}
return lhs.last_known_port < rhs.last_known_port;
}
std::string ToString() const {
return fmt::format("Address {{ unique_id: {}, last_known_ip: {}, last_known_port: {} }}",
boost::uuids::to_string(unique_id), last_known_ip.to_string(), last_known_port);
}
friend std::ostream &operator<<(std::ostream &in, const Address &address) {
in << address.ToString();
return in;
}
};
}; // namespace memgraph::io
namespace std {
template <>
struct hash<memgraph::io::PartialAddress> {
size_t operator()(const memgraph::io::PartialAddress &pa) const {
using boost::hash_combine;
using boost::hash_value;
// Start with a hash value of 0 .
std::size_t seed = 0;
if (pa.ip.is_v4()) {
auto h = std::hash<boost::asio::ip::address_v4>()(pa.ip.to_v4());
hash_combine(seed, h);
} else {
auto h = std::hash<boost::asio::ip::address_v6>()(pa.ip.to_v6());
hash_combine(seed, h);
}
hash_combine(seed, hash_value(pa.port));
// Return the result.
return seed;
}
};
} // namespace std

26
src/io/errors.hpp Normal file
View File

@@ -0,0 +1,26 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
namespace memgraph::io {
// Signifies that a retriable operation was unable to
// complete after a configured number of retries.
struct RetriesExhausted {};
// Signifies that a request was unable to receive a response
// within some configured timeout duration. It is important
// to remember that in distributed systems, a timeout does
// not signify that a request was not received or processed.
// It may be the case that the request was fully processed
// but that the response was not received.
struct TimedOut {};
}; // namespace memgraph::io

263
src/io/future.hpp Normal file
View File

@@ -0,0 +1,263 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <condition_variable>
#include <memory>
#include <mutex>
#include <optional>
#include <thread>
#include <utility>
#include "io/errors.hpp"
#include "utils/logging.hpp"
namespace memgraph::io {
// Shared is in an anonymous namespace, and the only way to
// construct a Promise or Future is to pass a Shared in. This
// ensures that Promises and Futures can only be constructed
// in this translation unit.
namespace details {
template <typename T>
class Shared {
mutable std::condition_variable cv_;
mutable std::mutex mu_;
std::optional<T> item_;
bool consumed_ = false;
bool waiting_ = false;
std::function<bool()> simulator_notifier_ = nullptr;
public:
explicit Shared(std::function<bool()> simulator_notifier) : simulator_notifier_(simulator_notifier) {}
Shared() = default;
Shared(Shared &&) = delete;
Shared &operator=(Shared &&) = delete;
Shared(const Shared &) = delete;
Shared &operator=(const Shared &) = delete;
~Shared() = default;
/// Takes the item out of our optional item_ and returns it.
T Take() {
MG_ASSERT(item_, "Take called without item_ being present");
MG_ASSERT(!consumed_, "Take called on already-consumed Future");
T ret = std::move(item_).value();
item_.reset();
consumed_ = true;
return ret;
}
T Wait() {
std::unique_lock<std::mutex> lock(mu_);
waiting_ = true;
while (!item_) {
if (simulator_notifier_) [[unlikely]] {
// We can't hold our own lock while notifying
// the simulator because notifying the simulator
// involves acquiring the simulator's mutex
// to guarantee that our notification linearizes
// with the simulator's condition variable.
// However, the simulator may acquire our
// mutex to check if we are being awaited,
// while determining system quiescence,
// so we have to get out of its way to avoid
// a cyclical deadlock.
lock.unlock();
std::invoke(simulator_notifier_);
lock.lock();
if (item_) {
// item may have been filled while we
// had dropped our mutex while notifying
// the simulator of our waiting_ status.
break;
}
} else {
cv_.wait(lock);
}
MG_ASSERT(!consumed_, "Future consumed twice!");
}
waiting_ = false;
return Take();
}
bool IsReady() const {
std::unique_lock<std::mutex> lock(mu_);
return item_.has_value();
}
std::optional<T> TryGet() {
std::unique_lock<std::mutex> lock(mu_);
if (item_) {
return Take();
}
return std::nullopt;
}
void Fill(T item) {
{
std::unique_lock<std::mutex> lock(mu_);
MG_ASSERT(!consumed_, "Promise filled after it was already consumed!");
MG_ASSERT(!item_, "Promise filled twice!");
item_ = item;
} // lock released before condition variable notification
cv_.notify_all();
}
bool IsAwaited() const {
std::unique_lock<std::mutex> lock(mu_);
return waiting_;
}
};
} // namespace details
template <typename T>
class Future {
bool consumed_or_moved_ = false;
std::shared_ptr<details::Shared<T>> shared_;
public:
explicit Future(std::shared_ptr<details::Shared<T>> shared) : shared_(shared) {}
Future() = delete;
Future(Future &&old) noexcept {
MG_ASSERT(!old.consumed_or_moved_, "Future moved from after already being moved from or consumed.");
shared_ = std::move(old.shared_);
consumed_or_moved_ = old.consumed_or_moved_;
old.consumed_or_moved_ = true;
}
Future &operator=(Future &&old) noexcept {
MG_ASSERT(!old.consumed_or_moved_, "Future moved from after already being moved from or consumed.");
shared_ = std::move(old.shared_);
consumed_or_moved_ = old.consumed_or_moved_;
old.consumed_or_moved_ = true;
return *this;
}
Future(const Future &) = delete;
Future &operator=(const Future &) = delete;
~Future() = default;
/// Returns true if the Future is ready to
/// be consumed using TryGet or Wait (prefer Wait
/// if you know it's ready, because it doesn't
/// return an optional.
bool IsReady() {
MG_ASSERT(!consumed_or_moved_, "Called IsReady after Future already consumed!");
return shared_->IsReady();
}
/// Non-blocking method that returns the inner
/// item if it's already ready, or std::nullopt
/// if it is not ready yet.
std::optional<T> TryGet() {
MG_ASSERT(!consumed_or_moved_, "Called TryGet after Future already consumed!");
std::optional<T> ret = shared_->TryGet();
if (ret) {
consumed_or_moved_ = true;
}
return ret;
}
/// Block on the corresponding promise to be filled,
/// returning the inner item when ready.
T Wait() && {
MG_ASSERT(!consumed_or_moved_, "Future should only be consumed with Wait once!");
T ret = shared_->Wait();
consumed_or_moved_ = true;
return ret;
}
/// Marks this Future as canceled.
void Cancel() {
MG_ASSERT(!consumed_or_moved_, "Future::Cancel called on a future that was already moved or consumed!");
consumed_or_moved_ = true;
}
};
template <typename T>
class Promise {
std::shared_ptr<details::Shared<T>> shared_;
bool filled_or_moved_{false};
public:
explicit Promise(std::shared_ptr<details::Shared<T>> shared) : shared_(shared) {}
Promise() = delete;
Promise(Promise &&old) noexcept {
MG_ASSERT(!old.filled_or_moved_, "Promise moved from after already being moved from or filled.");
shared_ = std::move(old.shared_);
old.filled_or_moved_ = true;
}
Promise &operator=(Promise &&old) noexcept {
MG_ASSERT(!old.filled_or_moved_, "Promise moved from after already being moved from or filled.");
shared_ = std::move(old.shared_);
old.filled_or_moved_ = true;
}
Promise(const Promise &) = delete;
Promise &operator=(const Promise &) = delete;
// NOLINTNEXTLINE(clang-analyzer-core.uninitialized.Branch)
~Promise() { MG_ASSERT(filled_or_moved_, "Promise destroyed before its associated Future was filled!"); }
// Fill the expected item into the Future.
void Fill(T item) {
MG_ASSERT(!filled_or_moved_, "Promise::Fill called on a promise that is already filled or moved!");
shared_->Fill(item);
filled_or_moved_ = true;
}
bool IsAwaited() { return shared_->IsAwaited(); }
/// Moves this Promise into a unique_ptr.
std::unique_ptr<Promise<T>> ToUnique() && {
std::unique_ptr<Promise<T>> up = std::make_unique<Promise<T>>(std::move(shared_));
filled_or_moved_ = true;
return up;
}
};
template <typename T>
std::pair<Future<T>, Promise<T>> FuturePromisePair() {
std::shared_ptr<details::Shared<T>> shared = std::make_shared<details::Shared<T>>();
Future<T> future = Future<T>(shared);
Promise<T> promise = Promise<T>(shared);
return std::make_pair(std::move(future), std::move(promise));
}
template <typename T>
std::pair<Future<T>, Promise<T>> FuturePromisePairWithNotifier(std::function<bool()> simulator_notifier) {
std::shared_ptr<details::Shared<T>> shared = std::make_shared<details::Shared<T>>(simulator_notifier);
Future<T> future = Future<T>(shared);
Promise<T> promise = Promise<T>(shared);
return std::make_pair(std::move(future), std::move(promise));
}
}; // namespace memgraph::io

View File

@@ -0,0 +1,35 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <random>
#include "io/address.hpp"
#include "io/local_transport/local_transport.hpp"
#include "io/local_transport/local_transport_handle.hpp"
#include "io/transport.hpp"
namespace memgraph::io::local_transport {
class LocalSystem {
std::shared_ptr<LocalTransportHandle> local_transport_handle_ = std::make_shared<LocalTransportHandle>();
public:
Io<LocalTransport> Register(Address address) {
LocalTransport local_transport(local_transport_handle_);
return Io{local_transport, address};
}
void ShutDown() { local_transport_handle_->ShutDown(); }
};
} // namespace memgraph::io::local_transport

View File

@@ -0,0 +1,61 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <chrono>
#include <memory>
#include <random>
#include <utility>
#include "io/address.hpp"
#include "io/local_transport/local_transport_handle.hpp"
#include "io/time.hpp"
#include "io/transport.hpp"
namespace memgraph::io::local_transport {
class LocalTransport {
std::shared_ptr<LocalTransportHandle> local_transport_handle_;
public:
explicit LocalTransport(std::shared_ptr<LocalTransportHandle> local_transport_handle)
: local_transport_handle_(std::move(local_transport_handle)) {}
template <Message RequestT, Message ResponseT>
ResponseFuture<ResponseT> Request(Address to_address, Address from_address, RequestT request, Duration timeout) {
return local_transport_handle_->template SubmitRequest<RequestT, ResponseT>(to_address, from_address,
std::move(request), timeout);
}
template <Message... Ms>
requires(sizeof...(Ms) > 0) RequestResult<Ms...> Receive(Address receiver_address, Duration timeout) {
return local_transport_handle_->template Receive<Ms...>(receiver_address, timeout);
}
template <Message M>
void Send(Address to_address, Address from_address, RequestId request_id, M &&message) {
return local_transport_handle_->template Send<M>(to_address, from_address, request_id, std::forward<M>(message));
}
Time Now() const { return local_transport_handle_->Now(); }
bool ShouldShutDown() const { return local_transport_handle_->ShouldShutDown(); }
template <class D = std::poisson_distribution<>, class Return = uint64_t>
Return Rand(D distrib) {
std::random_device rng;
return distrib(rng);
}
LatencyHistogramSummaries ResponseLatencies() { return local_transport_handle_->ResponseLatencies(); }
};
}; // namespace memgraph::io::local_transport

View File

@@ -0,0 +1,173 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <chrono>
#include <condition_variable>
#include <iostream>
#include <map>
#include <mutex>
#include "io/errors.hpp"
#include "io/message_conversion.hpp"
#include "io/message_histogram_collector.hpp"
#include "io/time.hpp"
#include "io/transport.hpp"
namespace memgraph::io::local_transport {
class LocalTransportHandle {
mutable std::mutex mu_{};
mutable std::condition_variable cv_;
bool should_shut_down_ = false;
MessageHistogramCollector histograms_;
RequestId request_id_counter_ = 0;
// the responses to requests that are being waited on
std::map<PromiseKey, DeadlineAndOpaquePromise> promises_;
// messages that are sent to servers that may later receive them
std::vector<OpaqueMessage> can_receive_;
public:
~LocalTransportHandle() {
for (auto &&[pk, promise] : promises_) {
std::move(promise.promise).TimeOut();
}
promises_.clear();
}
void ShutDown() {
std::unique_lock<std::mutex> lock(mu_);
should_shut_down_ = true;
cv_.notify_all();
}
bool ShouldShutDown() const {
std::unique_lock<std::mutex> lock(mu_);
return should_shut_down_;
}
LatencyHistogramSummaries ResponseLatencies() {
std::unique_lock<std::mutex> lock(mu_);
return histograms_.ResponseLatencies();
}
static Time Now() {
auto nano_time = std::chrono::system_clock::now();
return std::chrono::time_point_cast<std::chrono::microseconds>(nano_time);
}
template <Message... Ms>
requires(sizeof...(Ms) > 0) RequestResult<Ms...> Receive(Address /* receiver_address */, Duration timeout) {
std::unique_lock lock(mu_);
Time before = Now();
spdlog::info("can_receive_ size: {}", can_receive_.size());
while (can_receive_.empty()) {
Time now = Now();
// protection against non-monotonic timesources
auto maxed_now = std::max(now, before);
auto elapsed = maxed_now - before;
if (timeout < elapsed) {
return TimedOut{};
}
Duration relative_timeout = timeout - elapsed;
std::cv_status cv_status_value = cv_.wait_for(lock, relative_timeout);
if (cv_status_value == std::cv_status::timeout) {
return TimedOut{};
}
}
auto current_message = std::move(can_receive_.back());
can_receive_.pop_back();
auto m_opt = std::move(current_message).Take<Ms...>();
return std::move(m_opt).value();
}
template <Message M>
void Send(Address to_address, Address from_address, RequestId request_id, M &&message) {
auto type_info = TypeInfoFor(message);
std::any message_any(std::forward<M>(message));
OpaqueMessage opaque_message{.to_address = to_address,
.from_address = from_address,
.request_id = request_id,
.message = std::move(message_any),
.type_info = type_info};
PromiseKey promise_key{.requester_address = to_address, .request_id = opaque_message.request_id};
{
std::unique_lock<std::mutex> lock(mu_);
if (promises_.contains(promise_key)) {
spdlog::info("using message to fill promise");
// complete waiting promise if it's there
DeadlineAndOpaquePromise dop = std::move(promises_.at(promise_key));
promises_.erase(promise_key);
Duration response_latency = Now() - dop.requested_at;
dop.promise.Fill(std::move(opaque_message), response_latency);
histograms_.Measure(type_info, response_latency);
} else {
spdlog::info("placing message in can_receive_");
can_receive_.emplace_back(std::move(opaque_message));
}
} // lock dropped
cv_.notify_all();
}
template <Message RequestT, Message ResponseT>
ResponseFuture<ResponseT> SubmitRequest(Address to_address, Address from_address, RequestT &&request,
Duration timeout) {
auto [future, promise] = memgraph::io::FuturePromisePair<ResponseResult<ResponseT>>();
const bool port_matches = to_address.last_known_port == from_address.last_known_port;
const bool ip_matches = to_address.last_known_ip == from_address.last_known_ip;
MG_ASSERT(port_matches && ip_matches);
const auto now = Now();
const Time deadline = now + timeout;
RequestId request_id = 0;
{
std::unique_lock<std::mutex> lock(mu_);
request_id = ++request_id_counter_;
PromiseKey promise_key{.requester_address = from_address, .request_id = request_id};
OpaquePromise opaque_promise(std::move(promise).ToUnique());
DeadlineAndOpaquePromise dop{.requested_at = now, .deadline = deadline, .promise = std::move(opaque_promise)};
MG_ASSERT(!promises_.contains(promise_key));
promises_.emplace(std::move(promise_key), std::move(dop));
} // lock dropped
Send(to_address, from_address, request_id, std::forward<RequestT>(request));
return std::move(future);
}
};
} // namespace memgraph::io::local_transport

View File

@@ -0,0 +1,236 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <boost/core/demangle.hpp>
#include "io/transport.hpp"
#include "utils/type_info_ref.hpp"
namespace memgraph::io {
struct PromiseKey {
Address requester_address;
uint64_t request_id;
public:
friend bool operator<(const PromiseKey &lhs, const PromiseKey &rhs) {
if (lhs.requester_address != rhs.requester_address) {
return lhs.requester_address < rhs.requester_address;
}
return lhs.request_id < rhs.request_id;
}
};
struct OpaqueMessage {
Address to_address;
Address from_address;
uint64_t request_id;
std::any message;
utils::TypeInfoRef type_info;
/// Recursively tries to match a specific type from the outer
/// variant's parameter pack against the type of the std::any,
/// and if it matches, make it concrete and return it. Otherwise,
/// move on and compare the any with the next type from the
/// parameter pack.
///
/// Return is the full std::variant<Ts...> type that holds the
/// full parameter pack without interfering with recursive
/// narrowing expansion.
template <typename Return, Message Head, Message... Rest>
std::optional<Return> Unpack(std::any &&a) {
if (typeid(Head) == a.type()) {
Head concrete = std::any_cast<Head>(std::move(a));
return concrete;
}
if constexpr (sizeof...(Rest) > 0) {
return Unpack<Return, Rest...>(std::move(a));
} else {
return std::nullopt;
}
}
/// High level "user-facing" conversion function that lets
/// people interested in conversion only supply a single
/// parameter pack for the types that they want to compare
/// with the any and potentially include in the returned
/// variant.
template <Message... Ms>
requires(sizeof...(Ms) > 0) std::optional<std::variant<Ms...>> VariantFromAny(std::any &&a) {
return Unpack<std::variant<Ms...>, Ms...>(std::move(a));
}
template <Message... Ms>
requires(sizeof...(Ms) > 0) std::optional<RequestEnvelope<Ms...>> Take() && {
std::optional<std::variant<Ms...>> m_opt = VariantFromAny<Ms...>(std::move(message));
if (m_opt) {
return RequestEnvelope<Ms...>{
.message = std::move(*m_opt),
.request_id = request_id,
.to_address = to_address,
.from_address = from_address,
};
}
std::string demangled_name = "\"" + boost::core::demangle(message.type().name()) + "\"";
spdlog::error("failed to cast message of type {} to expected request type (probably in Receive argument types)",
demangled_name);
return std::nullopt;
}
};
class OpaquePromiseTraitBase {
public:
virtual const std::type_info *TypeInfo() const = 0;
virtual bool IsAwaited(void *ptr) const = 0;
virtual void Fill(void *ptr, OpaqueMessage &&, Duration) const = 0;
virtual void TimeOut(void *ptr) const = 0;
virtual ~OpaquePromiseTraitBase() = default;
OpaquePromiseTraitBase() = default;
OpaquePromiseTraitBase(const OpaquePromiseTraitBase &) = delete;
OpaquePromiseTraitBase &operator=(const OpaquePromiseTraitBase &) = delete;
OpaquePromiseTraitBase(OpaquePromiseTraitBase &&old) = delete;
OpaquePromiseTraitBase &operator=(OpaquePromiseTraitBase &&) = delete;
};
template <typename T>
class OpaquePromiseTrait : public OpaquePromiseTraitBase {
public:
const std::type_info *TypeInfo() const override { return &typeid(T); };
bool IsAwaited(void *ptr) const override { return static_cast<ResponsePromise<T> *>(ptr)->IsAwaited(); };
void Fill(void *ptr, OpaqueMessage &&opaque_message, Duration response_latency) const override {
T message = std::any_cast<T>(std::move(opaque_message.message));
auto response_envelope = ResponseEnvelope<T>{.message = std::move(message),
.request_id = opaque_message.request_id,
.to_address = opaque_message.to_address,
.from_address = opaque_message.from_address,
.response_latency = response_latency};
auto promise = static_cast<ResponsePromise<T> *>(ptr);
auto unique_promise = std::unique_ptr<ResponsePromise<T>>(promise);
unique_promise->Fill(std::move(response_envelope));
};
void TimeOut(void *ptr) const override {
auto promise = static_cast<ResponsePromise<T> *>(ptr);
auto unique_promise = std::unique_ptr<ResponsePromise<T>>(promise);
ResponseResult<T> result = TimedOut{};
unique_promise->Fill(std::move(result));
}
};
class OpaquePromise {
void *ptr_;
std::unique_ptr<OpaquePromiseTraitBase> trait_;
public:
OpaquePromise(OpaquePromise &&old) noexcept : ptr_(old.ptr_), trait_(std::move(old.trait_)) {
MG_ASSERT(old.ptr_ != nullptr);
old.ptr_ = nullptr;
}
OpaquePromise &operator=(OpaquePromise &&old) noexcept {
MG_ASSERT(ptr_ == nullptr);
MG_ASSERT(old.ptr_ != nullptr);
MG_ASSERT(this != &old);
ptr_ = old.ptr_;
trait_ = std::move(old.trait_);
old.ptr_ = nullptr;
return *this;
}
OpaquePromise(const OpaquePromise &) = delete;
OpaquePromise &operator=(const OpaquePromise &) = delete;
template <typename T>
std::unique_ptr<ResponsePromise<T>> Take() && {
MG_ASSERT(typeid(T) == *trait_->TypeInfo());
MG_ASSERT(ptr_ != nullptr);
auto ptr = static_cast<ResponsePromise<T> *>(ptr_);
ptr_ = nullptr;
return std::unique_ptr<T>(ptr);
}
template <typename T>
explicit OpaquePromise(std::unique_ptr<ResponsePromise<T>> promise)
: ptr_(static_cast<void *>(promise.release())), trait_(std::make_unique<OpaquePromiseTrait<T>>()) {}
bool IsAwaited() {
MG_ASSERT(ptr_ != nullptr);
return trait_->IsAwaited(ptr_);
}
void TimeOut() {
MG_ASSERT(ptr_ != nullptr);
trait_->TimeOut(ptr_);
ptr_ = nullptr;
}
void Fill(OpaqueMessage &&opaque_message, Duration response_latency) {
MG_ASSERT(ptr_ != nullptr);
trait_->Fill(ptr_, std::move(opaque_message), response_latency);
ptr_ = nullptr;
}
~OpaquePromise() {
MG_ASSERT(ptr_ == nullptr, "OpaquePromise destroyed without being explicitly timed out or filled");
}
};
struct DeadlineAndOpaquePromise {
Time requested_at;
Time deadline;
OpaquePromise promise;
};
template <class From>
std::type_info const &TypeInfoForVariant(From const &from) {
return std::visit([](auto &&x) -> decltype(auto) { return typeid(x); }, from);
}
template <class T>
utils::TypeInfoRef TypeInfoFor(const T & /* t */) {
return typeid(T);
}
template <typename From, typename Return, typename Head, typename... Rest>
std::optional<Return> ConvertVariantInner(From &&a) {
if (typeid(Head) == TypeInfoForVariant(a)) {
Head concrete = std::get<Head>(std::forward<From>(a));
return concrete;
}
if constexpr (sizeof...(Rest) > 0) {
return ConvertVariantInner<From, Return, Rest...>(std::forward<From>(a));
} else {
return std::nullopt;
}
}
/// This function converts a variant to another variant holding a subset OR superset of
/// possible types.
template <class From, class... Ms>
requires(sizeof...(Ms) > 0) std::optional<std::variant<Ms...>> ConvertVariant(From &&from) {
return ConvertVariantInner<From, std::variant<Ms...>, Ms...>(std::forward<From>(from));
}
} // namespace memgraph::io

View File

@@ -0,0 +1,132 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <chrono>
#include <cmath>
#include <compare>
#include <unordered_map>
#include <boost/core/demangle.hpp>
#include "io/time.hpp"
#include "utils/histogram.hpp"
#include "utils/logging.hpp"
#include "utils/print_helpers.hpp"
#include "utils/type_info_ref.hpp"
namespace memgraph::io {
struct LatencyHistogramSummary {
uint64_t count;
Duration p0;
Duration p50;
Duration p75;
Duration p90;
Duration p95;
Duration p975;
Duration p99;
Duration p999;
Duration p9999;
Duration p100;
Duration sum;
friend bool operator==(const LatencyHistogramSummary &lhs, const LatencyHistogramSummary &rhs) = default;
friend std::ostream &operator<<(std::ostream &in, const LatencyHistogramSummary &histo) {
in << "{ \"count\": " << histo.count;
in << ", \"p0\": " << histo.p0.count();
in << ", \"p50\": " << histo.p50.count();
in << ", \"p75\": " << histo.p75.count();
in << ", \"p90\": " << histo.p90.count();
in << ", \"p95\": " << histo.p95.count();
in << ", \"p975\": " << histo.p975.count();
in << ", \"p99\": " << histo.p99.count();
in << ", \"p999\": " << histo.p999.count();
in << ", \"p9999\": " << histo.p9999.count();
in << ", \"p100\": " << histo.p100.count();
in << ", \"sum\": " << histo.sum.count();
in << " }";
return in;
}
};
struct LatencyHistogramSummaries {
std::unordered_map<std::string, LatencyHistogramSummary> latencies;
std::string SummaryTable() {
std::string output;
const auto row = [&output](const auto &c1, const auto &c2, const auto &c3, const auto &c4, const auto &c5,
const auto &c6, const auto &c7) {
output +=
fmt::format("{: >50} | {: >8} | {: >8} | {: >8} | {: >8} | {: >8} | {: >8}\n", c1, c2, c3, c4, c5, c6, c7);
};
row("name", "count", "min (μs)", "med (μs)", "p99 (μs)", "max (μs)", "sum (ms)");
for (const auto &[name, histo] : latencies) {
row(name, histo.count, histo.p0.count(), histo.p50.count(), histo.p99.count(), histo.p100.count(),
histo.sum.count() / 1000);
}
output += "\n";
return output;
}
friend bool operator==(const LatencyHistogramSummaries &lhs, const LatencyHistogramSummaries &rhs) = default;
friend std::ostream &operator<<(std::ostream &in, const LatencyHistogramSummaries &histo) {
using memgraph::utils::print_helpers::operator<<;
in << histo.latencies;
return in;
}
};
class MessageHistogramCollector {
std::unordered_map<utils::TypeInfoRef, utils::Histogram, utils::TypeInfoHasher, utils::TypeInfoEqualTo> histograms_;
public:
void Measure(const std::type_info &type_info, const Duration &duration) {
auto &histo = histograms_[type_info];
histo.Measure(duration.count());
}
LatencyHistogramSummaries ResponseLatencies() {
std::unordered_map<std::string, LatencyHistogramSummary> ret{};
for (const auto &[type_id, histo] : histograms_) {
std::string demangled_name = "\"" + boost::core::demangle(type_id.get().name()) + "\"";
LatencyHistogramSummary latency_histogram_summary{
.count = histo.Count(),
.p0 = Duration(static_cast<int64_t>(histo.Percentile(0.0))),
.p50 = Duration(static_cast<int64_t>(histo.Percentile(50.0))),
.p75 = Duration(static_cast<int64_t>(histo.Percentile(75.0))),
.p90 = Duration(static_cast<int64_t>(histo.Percentile(90.0))),
.p95 = Duration(static_cast<int64_t>(histo.Percentile(95.0))),
.p975 = Duration(static_cast<int64_t>(histo.Percentile(97.5))),
.p99 = Duration(static_cast<int64_t>(histo.Percentile(99.0))),
.p999 = Duration(static_cast<int64_t>(histo.Percentile(99.9))),
.p9999 = Duration(static_cast<int64_t>(histo.Percentile(99.99))),
.p100 = Duration(static_cast<int64_t>(histo.Percentile(100.0))),
.sum = Duration(histo.Sum()),
};
ret.emplace(demangled_name, latency_histogram_summary);
}
return LatencyHistogramSummaries{.latencies = ret};
}
};
} // namespace memgraph::io

46
src/io/messages.hpp Normal file
View File

@@ -0,0 +1,46 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <variant>
#include <coordinator/coordinator.hpp>
#include <io/rsm/raft.hpp>
#include "query/v2/requests.hpp"
#include "utils/concepts.hpp"
namespace memgraph::io::messages {
using memgraph::coordinator::CoordinatorReadRequests;
using memgraph::coordinator::CoordinatorWriteRequests;
using memgraph::coordinator::CoordinatorWriteResponses;
using StorageReadRequest = msgs::ReadRequests;
using StorageWriteRequest = msgs::WriteRequests;
using memgraph::io::rsm::AppendRequest;
using memgraph::io::rsm::AppendResponse;
using memgraph::io::rsm::ReadRequest;
using memgraph::io::rsm::VoteRequest;
using memgraph::io::rsm::VoteResponse;
using memgraph::io::rsm::WriteRequest;
using memgraph::io::rsm::WriteResponse;
using CoordinatorMessages =
std::variant<ReadRequest<CoordinatorReadRequests>, AppendRequest<CoordinatorWriteRequests>, AppendResponse,
WriteRequest<CoordinatorWriteRequests>, VoteRequest, VoteResponse>;
using ShardMessages = std::variant<ReadRequest<StorageReadRequest>, AppendRequest<StorageWriteRequest>, AppendResponse,
WriteRequest<StorageWriteRequest>, VoteRequest, VoteResponse>;
using ShardManagerMessages = std::variant<WriteResponse<CoordinatorWriteResponses>>;
} // namespace memgraph::io::messages

960
src/io/rsm/raft.hpp Normal file
View File

@@ -0,0 +1,960 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
// TODO(tyler) buffer out-of-order Append buffers on the Followers to reassemble more quickly
// TODO(tyler) handle granular batch sizes based on simple flow control
#pragma once
#include <deque>
#include <iostream>
#include <map>
#include <set>
#include <thread>
#include <vector>
#include <boost/core/demangle.hpp>
#include "io/message_conversion.hpp"
#include "io/simulator/simulator.hpp"
#include "io/transport.hpp"
#include "utils/concepts.hpp"
namespace memgraph::io::rsm {
/// Timeout and replication tunables
using namespace std::chrono_literals;
static constexpr auto kMinimumElectionTimeout = 100ms;
static constexpr auto kMaximumElectionTimeout = 200ms;
static constexpr auto kMinimumBroadcastTimeout = 40ms;
static constexpr auto kMaximumBroadcastTimeout = 60ms;
static constexpr auto kMinimumCronInterval = 1ms;
static constexpr auto kMaximumCronInterval = 2ms;
static constexpr auto kMinimumReceiveTimeout = 40ms;
static constexpr auto kMaximumReceiveTimeout = 60ms;
static_assert(kMinimumElectionTimeout > kMaximumBroadcastTimeout,
"The broadcast timeout has to be smaller than the election timeout!");
static_assert(kMinimumElectionTimeout < kMaximumElectionTimeout,
"The minimum election timeout has to be smaller than the maximum election timeout!");
static_assert(kMinimumBroadcastTimeout < kMaximumBroadcastTimeout,
"The minimum broadcast timeout has to be smaller than the maximum broadcast timeout!");
static_assert(kMinimumCronInterval < kMaximumCronInterval,
"The minimum cron interval has to be smaller than the maximum cron interval!");
static_assert(kMinimumReceiveTimeout < kMaximumReceiveTimeout,
"The minimum receive timeout has to be smaller than the maximum receive timeout!");
static constexpr size_t kMaximumAppendBatchSize = 1024;
using Term = uint64_t;
using LogIndex = uint64_t;
using LogSize = uint64_t;
template <typename WriteOperation>
struct WriteRequest {
WriteOperation operation;
};
/// WriteResponse is returned to a client after
/// their WriteRequest was entered in to the raft
/// log and it reached consensus.
///
/// WriteReturn is the result of applying the WriteRequest to
/// ReplicatedState, and if the ReplicatedState::write
/// method is deterministic, all replicas will
/// have the same ReplicatedState after applying
/// the submitted WriteRequest.
template <typename WriteReturn>
struct WriteResponse {
bool success;
WriteReturn write_return;
std::optional<Address> retry_leader;
LogIndex raft_index;
};
template <typename ReadOperation>
struct ReadRequest {
ReadOperation operation;
};
template <typename ReadReturn>
struct ReadResponse {
bool success;
ReadReturn read_return;
std::optional<Address> retry_leader;
};
template <class... ReadReturn>
utils::TypeInfoRef TypeInfoFor(const ReadResponse<std::variant<ReadReturn...>> &read_response) {
return TypeInfoForVariant(read_response.read_return);
}
template <class ReadReturn>
utils::TypeInfoRef TypeInfoFor(const ReadResponse<ReadReturn> & /* read_response */) {
return typeid(ReadReturn);
}
template <class... WriteReturn>
utils::TypeInfoRef TypeInfoFor(const WriteResponse<std::variant<WriteReturn...>> &write_response) {
return TypeInfoForVariant(write_response.write_return);
}
template <class WriteReturn>
utils::TypeInfoRef TypeInfoFor(const WriteResponse<WriteReturn> & /* write_response */) {
return typeid(WriteReturn);
}
template <class WriteOperation>
utils::TypeInfoRef TypeInfoFor(const WriteRequest<WriteOperation> & /* write_request */) {
return typeid(WriteOperation);
}
template <class... WriteOperations>
utils::TypeInfoRef TypeInfoFor(const WriteRequest<std::variant<WriteOperations...>> &write_request) {
return TypeInfoForVariant(write_request.operation);
}
/// AppendRequest is a raft-level message that the Leader
/// periodically broadcasts to all Follower peers. This
/// serves three main roles:
/// 1. acts as a heartbeat from the Leader to the Follower
/// 2. replicates new data that the Leader has received to the Follower
/// 3. informs Follower peers when the commit index has increased,
/// signalling that it is now safe to apply log items to the
/// replicated state machine
template <typename WriteRequest>
struct AppendRequest {
Term term = 0;
LogIndex batch_start_log_index;
Term last_log_term;
std::vector<std::pair<Term, WriteRequest>> entries;
LogSize leader_commit;
};
struct AppendResponse {
bool success;
Term term;
Term last_log_term;
// a small optimization over the raft paper, tells
// the leader the offset that we are interested in
// to send log offsets from for us. This will only
// be useful at the beginning of a leader's term.
LogSize log_size;
};
struct VoteRequest {
Term term = 0;
LogSize log_size;
Term last_log_term;
};
struct VoteResponse {
Term term = 0;
LogSize committed_log_size;
bool vote_granted = false;
};
template <typename WriteRequest>
struct CommonState {
Term term = 0;
std::vector<std::pair<Term, WriteRequest>> log;
LogSize committed_log_size = 0;
LogSize applied_size = 0;
};
struct FollowerTracker {
LogIndex next_index = 0;
LogSize confirmed_log_size = 0;
};
struct PendingClientRequest {
RequestId request_id;
Address address;
Time received_at;
};
struct Leader {
std::map<Address, FollowerTracker> followers;
std::map<LogIndex, PendingClientRequest> pending_client_requests;
Time last_broadcast = Time::min();
std::string static ToString() { return "\tLeader \t"; }
};
struct Candidate {
std::map<Address, LogSize> successful_votes;
Time election_began = Time::min();
std::set<Address> outstanding_votes;
std::string static ToString() { return "\tCandidate\t"; }
};
struct Follower {
Time last_received_append_entries_timestamp;
Address leader_address;
std::string static ToString() { return "\tFollower \t"; }
};
using Role = std::variant<Candidate, Leader, Follower>;
template <typename Role>
concept AllRoles = memgraph::utils::SameAsAnyOf<Role, Leader, Follower, Candidate>;
template <typename Role>
concept LeaderOrFollower = memgraph::utils::SameAsAnyOf<Role, Leader, Follower>;
template <typename Role>
concept FollowerOrCandidate = memgraph::utils::SameAsAnyOf<Role, Follower, Candidate>;
/*
all ReplicatedState classes should have an Apply method
that returns our WriteResponseValue after consensus, and
a Read method that returns our ReadResponseValue without
requiring consensus.
ReadResponse Read(ReadOperation);
WriteResponseValue ReplicatedState::Apply(WriteRequest);
For example:
If the state is uint64_t, and WriteRequest is `struct PlusOne {};`,
and WriteResponseValue is also uint64_t (the new value), then
each call to state.Apply(PlusOne{}) will return the new value
after incrementing it. 0, 1, 2, 3... and this will be sent back
to the client that requested the mutation.
In practice, these mutations will usually be predicated on some
previous value, so that they are idempotent, functioning similarly
to a CAS operation.
*/
template <typename WriteOperation, typename ReadOperation, typename ReplicatedState, typename WriteResponseValue,
typename ReadResponseValue>
concept Rsm = requires(ReplicatedState state, WriteOperation w, ReadOperation r) {
{ state.Read(r) } -> std::same_as<ReadResponseValue>;
{ state.Apply(w) } -> std::same_as<WriteResponseValue>;
};
/// Parameter Purpose
/// --------------------------
/// IoImpl the concrete Io provider - SimulatorTransport, ThriftTransport, etc...
/// ReplicatedState the high-level data structure that is managed by the raft-backed replicated state machine
/// WriteOperation the individual operation type that is applied to the ReplicatedState in identical order
/// across each replica
/// WriteResponseValue the return value of calling ReplicatedState::Apply(WriteOperation), which is executed in
/// identical order across all replicas after an WriteOperation reaches consensus.
/// ReadOperation the type of operations that do not require consensus before executing directly
/// on a const ReplicatedState &
/// ReadResponseValue the return value of calling ReplicatedState::Read(ReadOperation), which is executed directly
/// without going through consensus first
template <typename IoImpl, typename ReplicatedState, typename WriteOperation, typename WriteResponseValue,
typename ReadOperation, typename ReadResponseValue>
requires Rsm<WriteOperation, ReadOperation, ReplicatedState, WriteResponseValue, ReadResponseValue>
class Raft {
CommonState<WriteOperation> state_;
Role role_ = Candidate{};
Io<IoImpl> io_;
std::vector<Address> peers_;
ReplicatedState replicated_state_;
Time next_cron_;
public:
Raft(Io<IoImpl> &&io, std::vector<Address> peers, ReplicatedState &&replicated_state)
: io_(std::forward<Io<IoImpl>>(io)),
peers_(peers),
replicated_state_(std::forward<ReplicatedState>(replicated_state)) {
if (peers.empty()) {
role_ = Leader{};
}
}
/// Periodic protocol maintenance. Returns the time that Cron should be called again
/// in the future.
Time Cron() {
// dispatch periodic logic based on our role to a specific Cron method.
std::optional<Role> new_role = std::visit([&](auto &role) { return Cron(role); }, role_);
if (new_role) {
role_ = std::move(new_role).value();
}
const Duration random_cron_interval = RandomTimeout(kMinimumCronInterval, kMaximumCronInterval);
return io_.Now() + random_cron_interval;
}
/// Returns the Address for our underlying Io implementation
Address GetAddress() { return io_.GetAddress(); }
using ReceiveVariant = std::variant<ReadRequest<ReadOperation>, AppendRequest<WriteOperation>, AppendResponse,
WriteRequest<WriteOperation>, VoteRequest, VoteResponse>;
void Handle(ReceiveVariant &&message_variant, RequestId request_id, Address from_address) {
// dispatch the message to a handler based on our role,
// which can be specified in the Handle first argument,
// or it can be `auto` if it's a handler for several roles
// or messages.
std::optional<Role> new_role = std::visit(
[&](auto &&msg, auto &role) mutable {
return Handle(role, std::forward<decltype(msg)>(msg), request_id, from_address);
},
std::forward<ReceiveVariant>(message_variant), role_);
// TODO(tyler) (M3) maybe replace std::visit with get_if for explicit prioritized matching, [[likely]] etc...
if (new_role) {
role_ = std::move(new_role).value();
}
}
void Run() {
while (!io_.ShouldShutDown()) {
const auto now = io_.Now();
if (now >= next_cron_) {
next_cron_ = Cron();
}
const Duration receive_timeout = RandomTimeout(kMinimumReceiveTimeout, kMaximumReceiveTimeout);
auto request_result =
io_.template ReceiveWithTimeout<ReadRequest<ReadOperation>, AppendRequest<WriteOperation>, AppendResponse,
WriteRequest<WriteOperation>, VoteRequest, VoteResponse>(receive_timeout);
if (request_result.HasError()) {
continue;
}
auto request = std::move(request_result.GetValue());
Handle(std::move(request.message), request.request_id, request.from_address);
}
}
private:
// Raft paper - 5.3
// When the entry has been safely replicated, the leader applies the
// entry to its state machine and returns the result of that
// execution to the client.
//
// "Safely replicated" is defined as being known to be present
// on at least a majority of all peers (inclusive of the Leader).
void BumpCommitIndexAndReplyToClients(Leader &leader) {
auto confirmed_log_sizes = std::vector<LogSize>{};
// We include our own log size in the calculation of the log
// confirmed log size that is present on at least a majority of all peers.
confirmed_log_sizes.push_back(state_.log.size());
for (const auto &[addr, f] : leader.followers) {
confirmed_log_sizes.push_back(f.confirmed_log_size);
Log("Follower at port ", addr.last_known_port, " has confirmed log size of: ", f.confirmed_log_size);
}
// reverse sort from highest to lowest (using std::ranges::greater)
std::ranges::sort(confirmed_log_sizes, std::ranges::greater());
// This is a particularly correctness-critical calculation because it
// determines the committed log size that will be broadcast in
// the next AppendRequest.
//
// If the following sizes are recorded for clusters of different numbers of peers,
// these are the expected sizes that are considered to have reached consensus:
//
// state | expected value | (confirmed_log_sizes.size() / 2)
// [1] 1 (1 / 2) => 0
// [2, 1] 1 (2 / 2) => 1
// [3, 2, 1] 2 (3 / 2) => 1
// [4, 3, 2, 1] 2 (4 / 2) => 2
// [5, 4, 3, 2, 1] 3 (5 / 2) => 2
const size_t majority_index = confirmed_log_sizes.size() / 2;
const LogSize new_committed_log_size = confirmed_log_sizes[majority_index];
// We never go backwards in history.
MG_ASSERT(state_.committed_log_size <= new_committed_log_size,
"as a Leader, we have previously set our committed_log_size to {}, but our Followers have a majority "
"committed_log_size of {}",
state_.committed_log_size, new_committed_log_size);
state_.committed_log_size = new_committed_log_size;
// For each size between the old size and the new one (inclusive),
// Apply that log's WriteOperation to our replicated_state_,
// and use the specific return value of the ReplicatedState::Apply
// method (WriteResponseValue) to respond to the requester.
for (; state_.applied_size < state_.committed_log_size; state_.applied_size++) {
const LogIndex apply_index = state_.applied_size;
const auto &write_request = state_.log[apply_index].second;
const WriteResponseValue write_return = replicated_state_.Apply(write_request);
if (leader.pending_client_requests.contains(apply_index)) {
const PendingClientRequest client_request = std::move(leader.pending_client_requests.at(apply_index));
leader.pending_client_requests.erase(apply_index);
const WriteResponse<WriteResponseValue> resp{
.success = true,
.write_return = std::move(write_return),
.raft_index = apply_index,
};
io_.Send(client_request.address, client_request.request_id, std::move(resp));
}
}
Log("committed_log_size is now ", state_.committed_log_size);
}
// Raft paper - 5.1
// AppendEntries RPCs are initiated by leaders to replicate log entries and to provide a form of heartbeat
void BroadcastAppendEntries(std::map<Address, FollowerTracker> &followers) {
for (auto &[address, follower] : followers) {
const LogIndex next_index = follower.next_index;
const auto missing = state_.log.size() - next_index;
const auto batch_size = std::min(missing, kMaximumAppendBatchSize);
const auto start_index = next_index;
const auto end_index = start_index + batch_size;
// advance follower's next index
follower.next_index += batch_size;
std::vector<std::pair<Term, WriteOperation>> entries;
entries.insert(entries.begin(), state_.log.begin() + start_index, state_.log.begin() + end_index);
const Term previous_term_from_index = PreviousTermFromIndex(start_index);
Log("sending ", entries.size(), " entries to Follower ", address.last_known_port,
" which are above its next_index of ", next_index);
AppendRequest<WriteOperation> ar{
.term = state_.term,
.batch_start_log_index = start_index,
.last_log_term = previous_term_from_index,
.entries = std::move(entries),
.leader_commit = state_.committed_log_size,
};
// request_id not necessary to set because it's not a Future-backed Request.
static constexpr RequestId request_id = 0;
io_.Send(address, request_id, std::move(ar));
}
}
// Raft paper - 5.2
// Raft uses randomized election timeouts to ensure that split votes are rare and that they are resolved quickly
Duration RandomTimeout(Duration min, Duration max) {
std::uniform_int_distribution time_distrib(min.count(), max.count());
const auto rand_micros = io_.Rand(time_distrib);
return Duration{rand_micros};
}
Duration RandomTimeout(int min_micros, int max_micros) {
std::uniform_int_distribution time_distrib(min_micros, max_micros);
const int rand_micros = io_.Rand(time_distrib);
return std::chrono::microseconds{rand_micros};
}
Term PreviousTermFromIndex(LogIndex index) const {
if (index == 0 || state_.log.size() + 1 <= index) {
return 0;
}
const auto &[term, data] = state_.log.at(index - 1);
return term;
}
Term CommittedLogTerm() {
MG_ASSERT(state_.log.size() >= state_.committed_log_size);
if (state_.log.empty() || state_.committed_log_size == 0) {
return 0;
}
const auto &[term, data] = state_.log.at(state_.committed_log_size - 1);
return term;
}
Term LastLogTerm() const {
if (state_.log.empty()) {
return 0;
}
const auto &[term, data] = state_.log.back();
return term;
}
template <typename... Ts>
void Log(Ts &&...args) {
const Time now = io_.Now();
const auto micros = std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count();
const Term term = state_.term;
const std::string role_string = std::visit([&](const auto &role) { return role.ToString(); }, role_);
std::ostringstream out;
out << '\t' << static_cast<int>(micros) << "\t" << term << "\t" << io_.GetAddress().last_known_port;
out << role_string;
(out << ... << args);
spdlog::info(out.str());
}
/////////////////////////////////////////////////////////////
/// Raft-related Cron methods
///
/// Cron + std::visit is how events are dispatched
/// to certain code based on Raft role.
///
/// Cron(role) takes as the first argument a reference to its
/// role, and as the second argument, the message that has
/// been received.
/////////////////////////////////////////////////////////////
// Raft paper - 5.2
// Candidates keep sending Vote to peers until:
// 1. receiving Append with a higher term (become Follower)
// 2. receiving Vote with a higher term (become a Follower)
// 3. receiving a quorum of responses to our last batch of Vote (become a Leader)
std::optional<Role> Cron(Candidate &candidate) {
const auto now = io_.Now();
const Duration election_timeout = RandomTimeout(kMinimumElectionTimeout, kMaximumElectionTimeout);
const auto election_timeout_us = std::chrono::duration_cast<std::chrono::milliseconds>(election_timeout).count();
if (now - candidate.election_began > election_timeout) {
state_.term++;
Log("becoming Candidate for term ", state_.term, " after leader timeout of ", election_timeout_us,
"ms elapsed since last election attempt");
const VoteRequest request{
.term = state_.term,
.log_size = state_.log.size(),
.last_log_term = LastLogTerm(),
};
auto outstanding_votes = std::set<Address>();
for (const auto &peer : peers_) {
// request_id not necessary to set because it's not a Future-backed Request.
static constexpr auto request_id = 0;
io_.template Send<VoteRequest>(peer, request_id, request);
outstanding_votes.insert(peer);
}
return Candidate{
.successful_votes = std::map<Address, LogIndex>(),
.election_began = now,
.outstanding_votes = outstanding_votes,
};
}
return std::nullopt;
}
// Raft paper - 5.2
// Followers become candidates if we haven't heard from the leader
// after a randomized timeout.
std::optional<Role> Cron(Follower &follower) {
const auto now = io_.Now();
const auto time_since_last_append_entries = now - follower.last_received_append_entries_timestamp;
// randomized follower timeout
const Duration election_timeout = RandomTimeout(kMinimumElectionTimeout, kMaximumElectionTimeout);
if (time_since_last_append_entries > election_timeout) {
// become a Candidate if we haven't heard from the Leader after this timeout
return Candidate{};
}
return std::nullopt;
}
// Leaders (re)send AppendRequest to followers.
std::optional<Role> Cron(Leader &leader) {
const Time now = io_.Now();
const Duration broadcast_timeout = RandomTimeout(kMinimumBroadcastTimeout, kMaximumBroadcastTimeout);
if (now > leader.last_broadcast + broadcast_timeout) {
BroadcastAppendEntries(leader.followers);
leader.last_broadcast = now;
}
return std::nullopt;
}
/////////////////////////////////////////////////////////////
/// Raft-related Handle methods
///
/// Handle + std::visit is how events are dispatched
/// to certain code based on Raft role.
///
/// Handle(role, message, ...)
/// takes as the first argument a reference
/// to its role, and as the second argument, the
/// message that has been received.
/////////////////////////////////////////////////////////////
// all roles can receive Vote and possibly become a follower
template <AllRoles ALL>
std::optional<Role> Handle(ALL & /* variable */, VoteRequest &&req, RequestId request_id, Address from_address) {
Log("received VoteRequest from ", from_address.last_known_port, " with term ", req.term);
const bool last_log_term_dominates = req.last_log_term >= LastLogTerm();
const bool term_dominates = req.term > state_.term;
const bool log_size_dominates = req.log_size >= state_.log.size();
const bool new_leader = last_log_term_dominates && term_dominates && log_size_dominates;
if (new_leader) {
MG_ASSERT(req.term > state_.term);
MG_ASSERT(std::max(req.term, state_.term) == req.term);
}
const VoteResponse res{
.term = std::max(req.term, state_.term),
.committed_log_size = state_.committed_log_size,
.vote_granted = new_leader,
};
io_.Send(from_address, request_id, res);
if (new_leader) {
// become a follower
state_.term = req.term;
return Follower{
.last_received_append_entries_timestamp = io_.Now(),
.leader_address = from_address,
};
}
if (term_dominates) {
Log("received a vote from an inferior candidate. Becoming Candidate");
state_.term = std::max(state_.term, req.term) + 1;
return Candidate{};
}
return std::nullopt;
}
std::optional<Role> Handle(Candidate &candidate, VoteResponse &&res, RequestId /* variable */, Address from_address) {
Log("received VoteResponse");
if (!res.vote_granted || res.term != state_.term) {
Log("received unsuccessful VoteResponse from term ", res.term, " when our candidacy term is ", state_.term);
// we received a delayed VoteResponse from the past, which has to do with an election that is
// no longer valid. We can simply drop this.
return std::nullopt;
}
MG_ASSERT(candidate.outstanding_votes.contains(from_address),
"Received unexpected VoteResponse from server not present in Candidate's outstanding_votes!");
candidate.outstanding_votes.erase(from_address);
MG_ASSERT(!candidate.successful_votes.contains(from_address),
"Received unexpected VoteResponse from server already in Candidate's successful_votes!");
candidate.successful_votes.insert({from_address, res.committed_log_size});
if (candidate.successful_votes.size() >= candidate.outstanding_votes.size()) {
std::map<Address, FollowerTracker> followers{};
for (const auto &[address, committed_log_size] : candidate.successful_votes) {
FollowerTracker follower{
.next_index = committed_log_size,
.confirmed_log_size = committed_log_size,
};
followers.insert({address, follower});
}
for (const auto &address : candidate.outstanding_votes) {
FollowerTracker follower{
.next_index = state_.log.size(),
.confirmed_log_size = 0,
};
followers.insert({address, follower});
}
Log("becoming Leader at term ", state_.term);
BroadcastAppendEntries(followers);
return Leader{
.followers = std::move(followers),
.pending_client_requests = std::map<LogIndex, PendingClientRequest>(),
};
}
return std::nullopt;
}
template <LeaderOrFollower LOF>
std::optional<Role> Handle(LOF & /* variable */, VoteResponse && /* variable */, RequestId /* variable */,
Address /* variable */) {
Log("non-Candidate received VoteResponse");
return std::nullopt;
}
template <AllRoles ALL>
std::optional<Role> Handle(ALL &role, AppendRequest<WriteOperation> &&req, RequestId request_id,
Address from_address) {
// log size starts out as state_.committed_log_size and only if everything is successful do we
// switch it to the log length.
AppendResponse res{
.success = false,
.term = state_.term,
.last_log_term = CommittedLogTerm(),
.log_size = state_.log.size(),
};
if constexpr (std::is_same<ALL, Leader>()) {
MG_ASSERT(req.term != state_.term, "Multiple leaders are acting under the term ", req.term);
}
const bool is_candidate = std::is_same<ALL, Candidate>();
const bool is_failed_competitor = is_candidate && req.term == state_.term;
const Time now = io_.Now();
// Raft paper - 5.2
// While waiting for votes, a candidate may receive an
// AppendEntries RPC from another server claiming to be leader. If
// the leaders term (included in its RPC) is at least as large as
// the candidates current term, then the candidate recognizes the
// leader as legitimate and returns to follower state.
if (req.term > state_.term || is_failed_competitor) {
// become follower of this leader, reply with our log status
state_.term = req.term;
io_.Send(from_address, request_id, res);
Log("becoming Follower of Leader ", from_address.last_known_port, " at term ", req.term);
return Follower{
.last_received_append_entries_timestamp = now,
.leader_address = from_address,
};
}
if (req.term < state_.term) {
// nack this request from an old leader
io_.Send(from_address, request_id, res);
return std::nullopt;
}
// at this point, we're dealing with our own leader
if constexpr (std::is_same<ALL, Follower>()) {
// small specialization for when we're already a Follower
MG_ASSERT(role.leader_address == from_address, "Multiple Leaders are acting under the same term number!");
role.last_received_append_entries_timestamp = now;
} else {
Log("Somehow entered Follower-specific logic as a non-Follower");
MG_ASSERT(false, "Somehow entered Follower-specific logic as a non-Follower");
}
// Handle steady-state conditions.
if (req.batch_start_log_index != state_.log.size()) {
Log("req.batch_start_log_index of ", req.batch_start_log_index, " does not match our log size of ",
state_.log.size());
} else if (req.last_log_term != LastLogTerm()) {
Log("req.last_log_term differs from our leader term at that slot, expected: ", LastLogTerm(), " but got ",
req.last_log_term);
} else {
// happy path - Apply log
Log("applying batch of ", req.entries.size(), " entries to our log starting at index ",
req.batch_start_log_index);
const auto resize_length = req.batch_start_log_index;
MG_ASSERT(resize_length >= state_.committed_log_size,
"Applied history from Leader which goes back in time from our commit_index");
// possibly chop-off stuff that was replaced by
// things with different terms (we got data that
// hasn't reached consensus yet, which is normal)
state_.log.resize(resize_length);
if (req.entries.size() > 0) {
auto &[first_term, op] = req.entries.at(0);
MG_ASSERT(LastLogTerm() <= first_term);
}
state_.log.insert(state_.log.end(), std::make_move_iterator(req.entries.begin()),
std::make_move_iterator(req.entries.end()));
MG_ASSERT(req.leader_commit >= state_.committed_log_size);
state_.committed_log_size = std::min(req.leader_commit, state_.log.size());
for (; state_.applied_size < state_.committed_log_size; state_.applied_size++) {
const auto &write_request = state_.log[state_.applied_size].second;
replicated_state_.Apply(write_request);
}
res.success = true;
}
res.last_log_term = LastLogTerm();
res.log_size = state_.log.size();
Log("returning log_size of ", res.log_size);
io_.Send(from_address, request_id, res);
return std::nullopt;
}
std::optional<Role> Handle(Leader &leader, AppendResponse &&res, RequestId /* variable */, Address from_address) {
if (res.term != state_.term) {
Log("received AppendResponse related to a previous term when we (presumably) were the leader");
return std::nullopt;
}
// TODO(tyler) when we have dynamic membership, this assert will become incorrect, but we should
// keep it in-place until then because it has bug finding value.
MG_ASSERT(leader.followers.contains(from_address), "received AppendResponse from unknown Follower");
// at this point, we know the term matches and we know this Follower
FollowerTracker &follower = leader.followers.at(from_address);
if (res.success) {
Log("got successful AppendResponse from ", from_address.last_known_port, " with log_size of ", res.log_size);
follower.next_index = std::max(follower.next_index, res.log_size);
} else {
Log("got unsuccessful AppendResponse from ", from_address.last_known_port, " with log_size of ", res.log_size);
follower.next_index = res.log_size;
}
follower.confirmed_log_size = std::max(follower.confirmed_log_size, res.log_size);
BumpCommitIndexAndReplyToClients(leader);
return std::nullopt;
}
template <FollowerOrCandidate FOC>
std::optional<Role> Handle(FOC & /* variable */, AppendResponse && /* variable */, RequestId /* variable */,
Address /* variable */) {
// we used to be the leader, and are getting old delayed responses
return std::nullopt;
}
/////////////////////////////////////////////////////////////
/// RSM-related handle methods
/////////////////////////////////////////////////////////////
// Leaders are able to immediately respond to the requester (with a ReadResponseValue) applied to the ReplicatedState
std::optional<Role> Handle(Leader & /* variable */, ReadRequest<ReadOperation> &&req, RequestId request_id,
Address from_address) {
Log("handling ReadOperation");
ReadOperation read_operation = req.operation;
ReadResponseValue read_return = replicated_state_.Read(read_operation);
const ReadResponse<ReadResponseValue> resp{
.success = true,
.read_return = std::move(read_return),
.retry_leader = std::nullopt,
};
io_.Send(from_address, request_id, resp);
return std::nullopt;
}
// Candidates should respond with a failure, similar to the Candidate + WriteRequest failure below
std::optional<Role> Handle(Candidate & /* variable */, ReadRequest<ReadOperation> && /* variable */,
RequestId request_id, Address from_address) {
Log("received ReadOperation - not redirecting because no Leader is known");
const ReadResponse<ReadResponseValue> res{
.success = false,
};
io_.Send(from_address, request_id, res);
Cron();
return std::nullopt;
}
// Followers should respond with a redirection, similar to the Follower + WriteRequest response below
std::optional<Role> Handle(Follower &follower, ReadRequest<ReadOperation> && /* variable */, RequestId request_id,
Address from_address) {
Log("redirecting client to known Leader with port ", follower.leader_address.last_known_port);
const ReadResponse<ReadResponseValue> res{
.success = false,
.retry_leader = follower.leader_address,
};
io_.Send(from_address, request_id, res);
return std::nullopt;
}
// Raft paper - 8
// When a client first starts up, it connects to a randomly chosen
// server. If the clients first choice is not the leader, that
// server will reject the clients request and supply information
// about the most recent leader it has heard from.
std::optional<Role> Handle(Follower &follower, WriteRequest<WriteOperation> && /* variable */, RequestId request_id,
Address from_address) {
Log("redirecting client to known Leader with port ", follower.leader_address.last_known_port);
const WriteResponse<WriteResponseValue> res{
.success = false,
.retry_leader = follower.leader_address,
};
io_.Send(from_address, request_id, res);
return std::nullopt;
}
std::optional<Role> Handle(Candidate & /* variable */, WriteRequest<WriteOperation> && /* variable */,
RequestId request_id, Address from_address) {
Log("received WriteRequest - not redirecting because no Leader is known");
const WriteResponse<WriteResponseValue> res{
.success = false,
};
io_.Send(from_address, request_id, res);
Cron();
return std::nullopt;
}
// only leaders actually handle replication requests from clients
std::optional<Role> Handle(Leader &leader, WriteRequest<WriteOperation> &&req, RequestId request_id,
Address from_address) {
auto type_info = TypeInfoFor(req);
std::string demangled_name = boost::core::demangle(type_info.get().name());
Log("handling WriteRequest<" + demangled_name + ">");
// we are the leader. add item to log and send Append to peers
MG_ASSERT(state_.term >= LastLogTerm());
state_.log.emplace_back(std::pair(state_.term, std::move(req.operation)));
LogIndex log_index = state_.log.size() - 1;
PendingClientRequest pcr{
.request_id = request_id,
.address = from_address,
.received_at = io_.Now(),
};
leader.pending_client_requests.emplace(log_index, pcr);
if (peers_.empty()) {
BumpCommitIndexAndReplyToClients(leader);
} else {
BroadcastAppendEntries(leader.followers);
}
return std::nullopt;
}
};
}; // namespace memgraph::io::rsm

264
src/io/rsm/rsm_client.hpp Normal file
View File

@@ -0,0 +1,264 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <iostream>
#include <optional>
#include <type_traits>
#include <unordered_map>
#include <vector>
#include "io/address.hpp"
#include "io/errors.hpp"
#include "io/rsm/raft.hpp"
#include "utils/result.hpp"
namespace memgraph::io::rsm {
using memgraph::io::Address;
using memgraph::io::Duration;
using memgraph::io::ResponseEnvelope;
using memgraph::io::ResponseFuture;
using memgraph::io::ResponseResult;
using memgraph::io::Time;
using memgraph::io::TimedOut;
using memgraph::io::rsm::ReadRequest;
using memgraph::io::rsm::ReadResponse;
using memgraph::io::rsm::WriteRequest;
using memgraph::io::rsm::WriteResponse;
using memgraph::utils::BasicResult;
class AsyncRequestToken {
size_t id_;
public:
explicit AsyncRequestToken(size_t id) : id_(id) {}
size_t GetId() const { return id_; }
};
template <typename RequestT, typename ResponseT>
struct AsyncRequest {
Time start_time;
RequestT request;
ResponseFuture<ResponseT> future;
};
template <typename IoImpl, typename WriteRequestT, typename WriteResponseT, typename ReadRequestT,
typename ReadResponseT>
class RsmClient {
using ServerPool = std::vector<Address>;
Io<IoImpl> io_;
Address leader_;
ServerPool server_addrs_;
/// State for single async read/write operations. In the future this could become a map
/// of async operations that can be accessed via an ID etc...
std::unordered_map<size_t, AsyncRequest<ReadRequestT, ReadResponse<ReadResponseT>>> async_reads_;
std::unordered_map<size_t, AsyncRequest<WriteRequestT, WriteResponse<WriteResponseT>>> async_writes_;
size_t async_token_generator_ = 0;
void SelectRandomLeader() {
std::uniform_int_distribution<size_t> addr_distrib(0, (server_addrs_.size() - 1));
size_t addr_index = io_.Rand(addr_distrib);
leader_ = server_addrs_[addr_index];
spdlog::debug("selecting a random leader at index {} with address {}", addr_index, leader_.ToString());
}
template <typename ResponseT>
void PossiblyRedirectLeader(const ResponseT &response) {
if (response.retry_leader) {
MG_ASSERT(!response.success, "retry_leader should never be set for successful responses");
leader_ = response.retry_leader.value();
spdlog::debug("client redirected to leader server {}", leader_.ToString());
}
if (!response.success) {
SelectRandomLeader();
}
}
public:
RsmClient(Io<IoImpl> io, Address leader, ServerPool server_addrs)
: io_{io}, leader_{leader}, server_addrs_{server_addrs} {}
RsmClient(const RsmClient &) = delete;
RsmClient &operator=(const RsmClient &) = delete;
RsmClient(RsmClient &&) noexcept = default;
RsmClient &operator=(RsmClient &&) noexcept = default;
RsmClient() = delete;
~RsmClient() = default;
BasicResult<TimedOut, WriteResponseT> SendWriteRequest(WriteRequestT req) {
auto token = SendAsyncWriteRequest(req);
auto poll_result = AwaitAsyncWriteRequest(token);
while (!poll_result) {
poll_result = AwaitAsyncWriteRequest(token);
}
return poll_result.value();
}
BasicResult<TimedOut, ReadResponseT> SendReadRequest(ReadRequestT req) {
auto token = SendAsyncReadRequest(req);
auto poll_result = AwaitAsyncReadRequest(token);
while (!poll_result) {
poll_result = AwaitAsyncReadRequest(token);
}
return poll_result.value();
}
/// AsyncRead methods
AsyncRequestToken SendAsyncReadRequest(const ReadRequestT &req) {
size_t token = async_token_generator_++;
ReadRequest<ReadRequestT> read_req = {.operation = req};
AsyncRequest<ReadRequestT, ReadResponse<ReadResponseT>> async_request{
.start_time = io_.Now(),
.request = std::move(req),
.future = io_.template Request<ReadRequest<ReadRequestT>, ReadResponse<ReadResponseT>>(leader_, read_req),
};
async_reads_.emplace(token, std::move(async_request));
return AsyncRequestToken{token};
}
void ResendAsyncReadRequest(const AsyncRequestToken &token) {
auto &async_request = async_reads_.at(token.GetId());
ReadRequest<ReadRequestT> read_req = {.operation = async_request.request};
async_request.future =
io_.template Request<ReadRequest<ReadRequestT>, ReadResponse<ReadResponseT>>(leader_, read_req);
}
std::optional<BasicResult<TimedOut, ReadResponseT>> PollAsyncReadRequest(const AsyncRequestToken &token) {
auto &async_request = async_reads_.at(token.GetId());
if (!async_request.future.IsReady()) {
return std::nullopt;
}
return AwaitAsyncReadRequest();
}
std::optional<BasicResult<TimedOut, ReadResponseT>> AwaitAsyncReadRequest(const AsyncRequestToken &token) {
auto &async_request = async_reads_.at(token.GetId());
ResponseResult<ReadResponse<ReadResponseT>> get_response_result = std::move(async_request.future).Wait();
const Duration overall_timeout = io_.GetDefaultTimeout();
const bool past_time_out = io_.Now() > async_request.start_time + overall_timeout;
const bool result_has_error = get_response_result.HasError();
if (result_has_error && past_time_out) {
// TODO static assert the exact type of error.
spdlog::debug("client timed out while trying to communicate with leader server {}", leader_.ToString());
async_reads_.erase(token.GetId());
return TimedOut{};
}
if (!result_has_error) {
ResponseEnvelope<ReadResponse<ReadResponseT>> &&get_response_envelope = std::move(get_response_result.GetValue());
ReadResponse<ReadResponseT> &&read_get_response = std::move(get_response_envelope.message);
PossiblyRedirectLeader(read_get_response);
if (read_get_response.success) {
async_reads_.erase(token.GetId());
spdlog::debug("returning read_return for RSM request");
return std::move(read_get_response.read_return);
}
} else {
SelectRandomLeader();
}
ResendAsyncReadRequest(token);
return std::nullopt;
}
/// AsyncWrite methods
AsyncRequestToken SendAsyncWriteRequest(const WriteRequestT &req) {
size_t token = async_token_generator_++;
WriteRequest<WriteRequestT> write_req = {.operation = req};
AsyncRequest<WriteRequestT, WriteResponse<WriteResponseT>> async_request{
.start_time = io_.Now(),
.request = std::move(req),
.future = io_.template Request<WriteRequest<WriteRequestT>, WriteResponse<WriteResponseT>>(leader_, write_req),
};
async_writes_.emplace(token, std::move(async_request));
return AsyncRequestToken{token};
}
void ResendAsyncWriteRequest(const AsyncRequestToken &token) {
auto &async_request = async_writes_.at(token.GetId());
WriteRequest<WriteRequestT> write_req = {.operation = async_request.request};
async_request.future =
io_.template Request<WriteRequest<WriteRequestT>, WriteResponse<WriteResponseT>>(leader_, write_req);
}
std::optional<BasicResult<TimedOut, WriteResponseT>> PollAsyncWriteRequest(const AsyncRequestToken &token) {
auto &async_request = async_writes_.at(token.GetId());
if (!async_request.future.IsReady()) {
return std::nullopt;
}
return AwaitAsyncWriteRequest();
}
std::optional<BasicResult<TimedOut, WriteResponseT>> AwaitAsyncWriteRequest(const AsyncRequestToken &token) {
auto &async_request = async_writes_.at(token.GetId());
ResponseResult<WriteResponse<WriteResponseT>> get_response_result = std::move(async_request.future).Wait();
const Duration overall_timeout = io_.GetDefaultTimeout();
const bool past_time_out = io_.Now() > async_request.start_time + overall_timeout;
const bool result_has_error = get_response_result.HasError();
if (result_has_error && past_time_out) {
// TODO static assert the exact type of error.
spdlog::debug("client timed out while trying to communicate with leader server {}", leader_.ToString());
async_writes_.erase(token.GetId());
return TimedOut{};
}
if (!result_has_error) {
ResponseEnvelope<WriteResponse<WriteResponseT>> &&get_response_envelope =
std::move(get_response_result.GetValue());
WriteResponse<WriteResponseT> &&write_get_response = std::move(get_response_envelope.message);
PossiblyRedirectLeader(write_get_response);
if (write_get_response.success) {
async_writes_.erase(token.GetId());
return std::move(write_get_response.write_return);
}
} else {
SelectRandomLeader();
}
ResendAsyncWriteRequest(token);
return std::nullopt;
}
};
} // namespace memgraph::io::rsm

155
src/io/rsm/shard_rsm.hpp Normal file
View File

@@ -0,0 +1,155 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
/// The ShardRsm is a simple in-memory raft-backed kv store that can be used for simple testing
/// and implementation of some query engine logic before storage engines are fully implemented.
///
/// To implement multiple read and write commands, change the StorageRead* and StorageWrite* requests
/// and responses to a std::variant of the different options, and route them to specific handlers in
/// the ShardRsm's Read and Apply methods. Remember that Read is called immediately when the Raft
/// leader receives the request, and does not replicate anything over Raft. Apply is called only
/// AFTER the StorageWriteRequest is replicated to a majority of Raft peers, and the result of calling
/// ShardRsm::Apply(StorageWriteRequest) is returned to the client that submitted the request.
#include <deque>
#include <iostream>
#include <map>
#include <optional>
#include <set>
#include <thread>
#include <vector>
#include "coordinator/hybrid_logical_clock.hpp"
#include "io/address.hpp"
#include "io/rsm/raft.hpp"
#include "storage/v3/id_types.hpp"
#include "storage/v3/property_value.hpp"
#include "utils/logging.hpp"
namespace memgraph::io::rsm {
using memgraph::coordinator::Hlc;
using memgraph::storage::v3::LabelId;
using memgraph::storage::v3::PropertyValue;
using ShardRsmKey = std::vector<PropertyValue>;
struct StorageWriteRequest {
LabelId label_id;
Hlc transaction_id;
ShardRsmKey key;
std::optional<int> value;
};
struct StorageWriteResponse {
bool shard_rsm_success;
std::optional<int> last_value;
// Only has a value if the given shard does not contain the requested key
std::optional<Hlc> latest_known_shard_map_version{std::nullopt};
};
struct StorageReadRequest {
LabelId label_id;
Hlc transaction_id;
ShardRsmKey key;
};
struct StorageReadResponse {
bool shard_rsm_success;
std::optional<int> value;
// Only has a value if the given shard does not contain the requested key
std::optional<Hlc> latest_known_shard_map_version{std::nullopt};
};
class ShardRsm {
std::map<ShardRsmKey, int> state_;
ShardRsmKey minimum_key_;
std::optional<ShardRsmKey> maximum_key_{std::nullopt};
Hlc shard_map_version_;
// The key is not located in this shard
bool IsKeyInRange(const ShardRsmKey &key) const {
if (maximum_key_) [[likely]] {
return (key >= minimum_key_ && key <= maximum_key_);
}
return key >= minimum_key_;
}
public:
StorageReadResponse Read(StorageReadRequest request) const {
StorageReadResponse ret;
if (!IsKeyInRange(request.key)) {
ret.latest_known_shard_map_version = shard_map_version_;
ret.shard_rsm_success = false;
} else if (state_.contains(request.key)) {
ret.value = state_.at(request.key);
ret.shard_rsm_success = true;
} else {
ret.shard_rsm_success = false;
ret.value = std::nullopt;
}
return ret;
}
StorageWriteResponse Apply(StorageWriteRequest request) {
StorageWriteResponse ret;
// Key is outside the prohibited range
if (!IsKeyInRange(request.key)) {
ret.latest_known_shard_map_version = shard_map_version_;
ret.shard_rsm_success = false;
}
// Key exist
else if (state_.contains(request.key)) {
auto &val = state_[request.key];
/*
* Delete
*/
if (!request.value) {
ret.shard_rsm_success = true;
ret.last_value = val;
state_.erase(state_.find(request.key));
}
/*
* Update
*/
// Does old_value match?
if (request.value == val) {
ret.last_value = val;
ret.shard_rsm_success = true;
val = request.value.value();
} else {
ret.last_value = val;
ret.shard_rsm_success = false;
}
}
/*
* Create
*/
else {
ret.last_value = std::nullopt;
ret.shard_rsm_success = true;
state_.emplace(request.key, request.value.value());
}
return ret;
}
};
} // namespace memgraph::io::rsm

View File

@@ -0,0 +1,8 @@
set(io_simulator_sources
simulator_handle.cpp)
find_package(fmt REQUIRED)
find_package(Threads REQUIRED)
add_library(mg-io-simulator STATIC ${io_simulator_sources})
target_link_libraries(mg-io-simulator stdc++fs Threads::Threads fmt::fmt mg-utils)

View File

@@ -0,0 +1,53 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <memory>
#include <random>
#include "io/address.hpp"
#include "io/simulator/simulator_config.hpp"
#include "io/simulator/simulator_handle.hpp"
#include "io/simulator/simulator_transport.hpp"
namespace memgraph::io::simulator {
class Simulator {
std::mt19937 rng_;
std::shared_ptr<SimulatorHandle> simulator_handle_;
uint16_t auto_port_ = 0;
public:
explicit Simulator(SimulatorConfig config)
: rng_(std::mt19937{config.rng_seed}), simulator_handle_{std::make_shared<SimulatorHandle>(config)} {}
~Simulator() { ShutDown(); }
void ShutDown() { simulator_handle_->ShutDown(); }
Io<SimulatorTransport> RegisterNew() {
Address address = Address::TestAddress(auto_port_++);
return Register(address);
}
Io<SimulatorTransport> Register(Address address) {
std::uniform_int_distribution<uint64_t> seed_distrib;
uint64_t seed = seed_distrib(rng_);
return Io{SimulatorTransport{simulator_handle_, address, seed}, address};
}
void IncrementServerCountAndWaitForQuiescentState(Address address) {
simulator_handle_->IncrementServerCountAndWaitForQuiescentState(address);
}
SimulatorStats Stats() { return simulator_handle_->Stats(); }
};
}; // namespace memgraph::io::simulator

View File

@@ -0,0 +1,30 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <chrono>
#include "io/time.hpp"
namespace memgraph::io::simulator {
using memgraph::io::Time;
struct SimulatorConfig {
uint8_t drop_percent = 0;
bool perform_timeouts = false;
bool scramble_messages = true;
uint64_t rng_seed = 0;
Time start_time = Time::min();
Time abort_time = Time::max();
};
}; // namespace memgraph::io::simulator

View File

@@ -0,0 +1,194 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "io/simulator/simulator_handle.hpp"
#include "io/address.hpp"
#include "io/errors.hpp"
#include "io/simulator/simulator_config.hpp"
#include "io/simulator/simulator_stats.hpp"
#include "io/time.hpp"
#include "io/transport.hpp"
#include "utils/exceptions.hpp"
namespace memgraph::io::simulator {
void SimulatorHandle::ShutDown() {
std::unique_lock<std::mutex> lock(mu_);
should_shut_down_ = true;
for (auto it = promises_.begin(); it != promises_.end();) {
auto &[promise_key, dop] = *it;
std::move(dop).promise.TimeOut();
it = promises_.erase(it);
}
can_receive_.clear();
cv_.notify_all();
}
bool SimulatorHandle::ShouldShutDown() const {
std::unique_lock<std::mutex> lock(mu_);
return should_shut_down_;
}
LatencyHistogramSummaries SimulatorHandle::ResponseLatencies() {
std::unique_lock<std::mutex> lock(mu_);
return histograms_.ResponseLatencies();
}
void SimulatorHandle::IncrementServerCountAndWaitForQuiescentState(Address address) {
std::unique_lock<std::mutex> lock(mu_);
server_addresses_.insert(address);
while (true) {
const size_t blocked_servers = blocked_on_receive_.size();
const bool all_servers_blocked = blocked_servers == server_addresses_.size();
if (all_servers_blocked) {
spdlog::trace("quiescent state detected - {} out of {} servers now blocked on receive", blocked_servers,
server_addresses_.size());
return;
}
spdlog::trace("not returning from quiescent because we see {} blocked out of {}", blocked_servers,
server_addresses_.size());
cv_.wait(lock);
}
}
bool SortInFlight(const std::pair<Address, OpaqueMessage> &lhs, const std::pair<Address, OpaqueMessage> &rhs) {
// NB: never sort based on the request ID etc...
// This should only be used from std::stable_sort
// because by comparing on the from_address alone,
// we expect the sender ordering to remain
// deterministic.
const auto &[addr_1, om_1] = lhs;
const auto &[addr_2, om_2] = rhs;
return om_1.from_address < om_2.from_address;
}
bool SimulatorHandle::MaybeTickSimulator() {
std::unique_lock<std::mutex> lock(mu_);
const size_t blocked_servers = blocked_on_receive_.size();
if (should_shut_down_ || blocked_servers < server_addresses_.size()) {
// we only need to advance the simulator when all
// servers have reached a quiescent state, blocked
// on their own futures or receive methods.
return false;
}
// We allow the simulator to progress the state of the system only
// after all servers are blocked on receive.
spdlog::trace("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ simulator tick ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");
stats_.simulator_ticks++;
blocked_on_receive_.clear();
cv_.notify_all();
bool timed_anything_out = TimeoutPromisesPastDeadline();
if (timed_anything_out) {
spdlog::trace("simulator progressing: timed out a request");
}
const Duration clock_advance = std::chrono::microseconds{time_distrib_(rng_)};
// We don't always want to advance the clock with every message that we deliver because
// when we advance it for every message, it causes timeouts to occur for any "request path"
// over a certain length. Alternatively, we don't want to simply deliver multiple messages
// in a single simulator tick because that would reduce the amount of concurrent message
// mixing that may naturally occur in production. This approach is to mod the random clock
// advance by a prime number (hopefully avoiding most harmonic effects that would be introduced
// by only advancing the clock by an even amount etc...) and only advancing the clock close to
// half of the time.
if (clock_advance.count() % 97 > 49) {
spdlog::trace("simulator progressing: clock advanced by {}", clock_advance.count());
cluster_wide_time_microseconds_ += clock_advance;
stats_.elapsed_time = cluster_wide_time_microseconds_ - config_.start_time;
}
if (cluster_wide_time_microseconds_ >= config_.abort_time) {
spdlog::error(
"Cluster has executed beyond its configured abort_time, and something may be failing to make progress "
"in an expected amount of time. The SimulatorConfig.rng_seed for this run is {}",
config_.rng_seed);
throw utils::BasicException{"Cluster has executed beyond its configured abort_time"};
}
if (in_flight_.empty()) {
return true;
}
std::stable_sort(in_flight_.begin(), in_flight_.end(), SortInFlight);
if (config_.scramble_messages && in_flight_.size() > 1) {
// scramble messages
std::uniform_int_distribution<size_t> swap_distrib(0, in_flight_.size() - 1);
const size_t swap_index = swap_distrib(rng_);
std::swap(in_flight_[swap_index], in_flight_.back());
}
auto [to_address, opaque_message] = std::move(in_flight_.back());
in_flight_.pop_back();
const int drop_threshold = drop_distrib_(rng_);
const bool should_drop = drop_threshold < config_.drop_percent;
if (should_drop) {
stats_.dropped_messages++;
}
PromiseKey promise_key{.requester_address = to_address, .request_id = opaque_message.request_id};
if (promises_.contains(promise_key)) {
// complete waiting promise if it's there
DeadlineAndOpaquePromise dop = std::move(promises_.at(promise_key));
promises_.erase(promise_key);
const bool normal_timeout = config_.perform_timeouts && (dop.deadline < cluster_wide_time_microseconds_);
if (should_drop || normal_timeout) {
stats_.timed_out_requests++;
dop.promise.TimeOut();
spdlog::trace("simulator timing out request ");
} else {
stats_.total_responses++;
Duration response_latency = cluster_wide_time_microseconds_ - dop.requested_at;
auto type_info = opaque_message.type_info;
dop.promise.Fill(std::move(opaque_message), response_latency);
histograms_.Measure(type_info, response_latency);
spdlog::trace("simulator replying to request");
}
} else if (should_drop) {
// don't add it anywhere, let it drop
spdlog::trace("simulator silently dropping request");
} else {
// add to can_receive_ if not
spdlog::trace("simulator adding message to can_receive_ from {} to {}", opaque_message.from_address.last_known_port,
opaque_message.to_address.last_known_port);
const auto &[om_vec, inserted] =
can_receive_.try_emplace(to_address.ToPartialAddress(), std::vector<OpaqueMessage>());
om_vec->second.emplace_back(std::move(opaque_message));
}
return true;
}
Time SimulatorHandle::Now() const {
std::unique_lock<std::mutex> lock(mu_);
return cluster_wide_time_microseconds_;
}
SimulatorStats SimulatorHandle::Stats() {
std::unique_lock<std::mutex> lock(mu_);
return stats_;
}
} // namespace memgraph::io::simulator

View File

@@ -0,0 +1,217 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <any>
#include <compare>
#include <iostream>
#include <map>
#include <memory>
#include <optional>
#include <set>
#include <utility>
#include <variant>
#include <vector>
#include "io/address.hpp"
#include "io/errors.hpp"
#include "io/message_conversion.hpp"
#include "io/message_histogram_collector.hpp"
#include "io/simulator/simulator_config.hpp"
#include "io/simulator/simulator_stats.hpp"
#include "io/time.hpp"
#include "io/transport.hpp"
namespace memgraph::io::simulator {
class SimulatorHandle {
mutable std::mutex mu_{};
mutable std::condition_variable cv_;
// messages that have not yet been scheduled or dropped
std::vector<std::pair<Address, OpaqueMessage>> in_flight_;
// the responses to requests that are being waited on
std::map<PromiseKey, DeadlineAndOpaquePromise> promises_;
// messages that are sent to servers that may later receive them
std::map<PartialAddress, std::vector<OpaqueMessage>> can_receive_;
Time cluster_wide_time_microseconds_;
bool should_shut_down_ = false;
SimulatorStats stats_;
std::set<Address> blocked_on_receive_;
std::set<Address> server_addresses_;
std::mt19937 rng_;
std::uniform_int_distribution<int> time_distrib_{0, 30000};
std::uniform_int_distribution<int> drop_distrib_{0, 99};
SimulatorConfig config_;
MessageHistogramCollector histograms_;
RequestId request_id_counter_{0};
bool TimeoutPromisesPastDeadline() {
bool timed_anything_out = false;
const Time now = cluster_wide_time_microseconds_;
for (auto it = promises_.begin(); it != promises_.end();) {
auto &[promise_key, dop] = *it;
if (dop.deadline < now && config_.perform_timeouts) {
spdlog::trace("timing out request from requester {}.", promise_key.requester_address.ToString());
std::move(dop).promise.TimeOut();
it = promises_.erase(it);
stats_.timed_out_requests++;
timed_anything_out = true;
} else {
++it;
}
}
return timed_anything_out;
}
public:
explicit SimulatorHandle(SimulatorConfig config)
: cluster_wide_time_microseconds_(config.start_time), rng_(config.rng_seed), config_(config) {}
LatencyHistogramSummaries ResponseLatencies();
~SimulatorHandle() {
for (auto it = promises_.begin(); it != promises_.end();) {
auto &[promise_key, dop] = *it;
std::move(dop).promise.TimeOut();
it = promises_.erase(it);
}
}
void IncrementServerCountAndWaitForQuiescentState(Address address);
/// This method causes most of the interesting simulation logic to happen, wrt network behavior.
/// It checks to see if all background "server" threads are blocked on new messages, and if so,
/// it will decide whether to drop, reorder, or deliver in-flight messages based on the SimulatorConfig
/// that was used to create the Simulator.
bool MaybeTickSimulator();
void ShutDown();
bool ShouldShutDown() const;
template <Message Request, Message Response>
ResponseFuture<Response> SubmitRequest(Address to_address, Address from_address, Request &&request, Duration timeout,
std::function<bool()> &&maybe_tick_simulator) {
spdlog::trace("submitting request to {}", to_address.last_known_port);
auto type_info = TypeInfoFor(request);
auto [future, promise] = memgraph::io::FuturePromisePairWithNotifier<ResponseResult<Response>>(
std::forward<std::function<bool()>>(maybe_tick_simulator));
std::unique_lock<std::mutex> lock(mu_);
RequestId request_id = ++request_id_counter_;
const Time deadline = cluster_wide_time_microseconds_ + timeout;
std::any message(request);
OpaqueMessage om{.to_address = to_address,
.from_address = from_address,
.request_id = request_id,
.message = std::move(message),
.type_info = type_info};
in_flight_.emplace_back(std::make_pair(to_address, std::move(om)));
PromiseKey promise_key{.requester_address = from_address, .request_id = request_id};
OpaquePromise opaque_promise(std::move(promise).ToUnique());
DeadlineAndOpaquePromise dop{
.requested_at = cluster_wide_time_microseconds_,
.deadline = deadline,
.promise = std::move(opaque_promise),
};
MG_ASSERT(!promises_.contains(promise_key));
promises_.emplace(std::move(promise_key), std::move(dop));
stats_.total_messages++;
stats_.total_requests++;
cv_.notify_all();
return std::move(future);
}
template <Message... Ms>
requires(sizeof...(Ms) > 0) RequestResult<Ms...> Receive(const Address &receiver, Duration timeout) {
std::unique_lock<std::mutex> lock(mu_);
const Time deadline = cluster_wide_time_microseconds_ + timeout;
auto partial_address = receiver.ToPartialAddress();
while (!should_shut_down_ && (cluster_wide_time_microseconds_ < deadline)) {
if (can_receive_.contains(partial_address)) {
std::vector<OpaqueMessage> &can_rx = can_receive_.at(partial_address);
if (!can_rx.empty()) {
OpaqueMessage message = std::move(can_rx.back());
can_rx.pop_back();
// TODO(tyler) search for item in can_receive_ that matches the desired types, rather
// than asserting that the last item in can_rx matches.
auto m_opt = std::move(message).Take<Ms...>();
MG_ASSERT(m_opt.has_value(), "Wrong message type received compared to the expected type");
return std::move(m_opt).value();
}
}
if (!should_shut_down_) {
if (!blocked_on_receive_.contains(receiver)) {
blocked_on_receive_.emplace(receiver);
spdlog::trace("blocking receiver {}", receiver.ToPartialAddress().port);
cv_.notify_all();
}
cv_.wait(lock);
}
}
spdlog::trace("timing out receiver {}", receiver.ToPartialAddress().port);
return TimedOut{};
}
template <Message M>
void Send(Address to_address, Address from_address, RequestId request_id, M message) {
spdlog::trace("sending message from {} to {}", from_address.last_known_port, to_address.last_known_port);
auto type_info = TypeInfoFor(message);
{
std::unique_lock<std::mutex> lock(mu_);
std::any message_any(std::move(message));
OpaqueMessage om{.to_address = to_address,
.from_address = from_address,
.request_id = request_id,
.message = std::move(message_any),
.type_info = type_info};
in_flight_.emplace_back(std::make_pair(std::move(to_address), std::move(om)));
stats_.total_messages++;
} // lock dropped before cv notification
cv_.notify_all();
}
Time Now() const;
template <class D = std::poisson_distribution<>, class Return = uint64_t>
Return Rand(D distrib) {
std::unique_lock<std::mutex> lock(mu_);
return distrib(rng_);
}
SimulatorStats Stats();
};
}; // namespace memgraph::io::simulator

View File

@@ -0,0 +1,46 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <cstdint>
#include <fmt/format.h>
#include "io/time.hpp"
namespace memgraph::io::simulator {
struct SimulatorStats {
uint64_t total_messages = 0;
uint64_t dropped_messages = 0;
uint64_t timed_out_requests = 0;
uint64_t total_requests = 0;
uint64_t total_responses = 0;
uint64_t simulator_ticks = 0;
Duration elapsed_time;
friend bool operator==(const SimulatorStats & /* lhs */, const SimulatorStats & /* rhs */) = default;
friend std::ostream &operator<<(std::ostream &in, const SimulatorStats &stats) {
auto elapsed_ms = std::chrono::duration_cast<std::chrono::milliseconds>(stats.elapsed_time).count();
std::string formated = fmt::format(
"SimulatorStats {{ total_messages: {}, dropped_messages: {}, timed_out_requests: {}, total_requests: {}, "
"total_responses: {}, simulator_ticks: {}, elapsed_time: {}ms }}",
stats.total_messages, stats.dropped_messages, stats.timed_out_requests, stats.total_requests,
stats.total_responses, stats.simulator_ticks, elapsed_ms);
in << formated;
return in;
}
};
}; // namespace memgraph::io::simulator

View File

@@ -0,0 +1,64 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <memory>
#include <utility>
#include "io/address.hpp"
#include "io/simulator/simulator_handle.hpp"
#include "io/time.hpp"
namespace memgraph::io::simulator {
using memgraph::io::Duration;
using memgraph::io::Time;
class SimulatorTransport {
std::shared_ptr<SimulatorHandle> simulator_handle_;
const Address address_;
std::mt19937 rng_;
public:
SimulatorTransport(std::shared_ptr<SimulatorHandle> simulator_handle, Address address, uint64_t seed)
: simulator_handle_(simulator_handle), address_(address), rng_(std::mt19937{seed}) {}
template <Message RequestT, Message ResponseT>
ResponseFuture<ResponseT> Request(Address to_address, Address from_address, RequestT request, Duration timeout) {
std::function<bool()> maybe_tick_simulator = [this] { return simulator_handle_->MaybeTickSimulator(); };
return simulator_handle_->template SubmitRequest<RequestT, ResponseT>(to_address, from_address, std::move(request),
timeout, std::move(maybe_tick_simulator));
}
template <Message... Ms>
requires(sizeof...(Ms) > 0) RequestResult<Ms...> Receive(Address receiver_address, Duration timeout) {
return simulator_handle_->template Receive<Ms...>(receiver_address, timeout);
}
template <Message M>
void Send(Address to_address, Address from_address, uint64_t request_id, M message) {
return simulator_handle_->template Send<M>(to_address, from_address, request_id, message);
}
Time Now() const { return simulator_handle_->Now(); }
bool ShouldShutDown() const { return simulator_handle_->ShouldShutDown(); }
template <class D = std::poisson_distribution<>, class Return = uint64_t>
Return Rand(D distrib) {
return distrib(rng_);
}
LatencyHistogramSummaries ResponseLatencies() { return simulator_handle_->ResponseLatencies(); }
};
}; // namespace memgraph::io::simulator

21
src/io/time.hpp Normal file
View File

@@ -0,0 +1,21 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <chrono>
namespace memgraph::io {
using Duration = std::chrono::microseconds;
using Time = std::chrono::time_point<std::chrono::system_clock, Duration>;
} // namespace memgraph::io

151
src/io/transport.hpp Normal file
View File

@@ -0,0 +1,151 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <chrono>
#include <concepts>
#include <random>
#include <variant>
#include "io/address.hpp"
#include "io/errors.hpp"
#include "io/future.hpp"
#include "io/message_histogram_collector.hpp"
#include "io/time.hpp"
#include "utils/result.hpp"
namespace memgraph::io {
using memgraph::utils::BasicResult;
// TODO(tyler) ensure that Message continues to represent
// reasonable constraints around message types over time,
// as we adapt things to use Thrift-generated message types.
template <typename T>
concept Message = std::same_as<T, std::decay_t<T>>;
using RequestId = uint64_t;
template <Message M>
struct ResponseEnvelope {
M message;
RequestId request_id;
Address to_address;
Address from_address;
Duration response_latency;
};
template <Message M>
using ResponseResult = BasicResult<TimedOut, ResponseEnvelope<M>>;
template <Message M>
using ResponseFuture = memgraph::io::Future<ResponseResult<M>>;
template <Message M>
using ResponsePromise = memgraph::io::Promise<ResponseResult<M>>;
template <Message... Ms>
struct RequestEnvelope {
std::variant<Ms...> message;
RequestId request_id;
Address to_address;
Address from_address;
};
template <Message... Ms>
using RequestResult = BasicResult<TimedOut, RequestEnvelope<Ms...>>;
template <typename I>
class Io {
I implementation_;
Address address_;
Duration default_timeout_ = std::chrono::microseconds{100000};
public:
Io(I io, Address address) : implementation_(io), address_(address) {}
/// Set the default timeout for all requests that are issued
/// without an explicit timeout set.
void SetDefaultTimeout(Duration timeout) { default_timeout_ = timeout; }
/// Returns the current default timeout for this Io instance.
Duration GetDefaultTimeout() { return default_timeout_; }
/// Issue a request with an explicit timeout in microseconds provided. This tends to be used by clients.
template <Message RequestT, Message ResponseT>
ResponseFuture<ResponseT> RequestWithTimeout(Address address, RequestT request, Duration timeout) {
const Address from_address = address_;
return implementation_.template Request<RequestT, ResponseT>(address, from_address, request, timeout);
}
/// Issue a request that times out after the default timeout. This tends
/// to be used by clients.
template <Message RequestT, Message ResponseT>
ResponseFuture<ResponseT> Request(Address to_address, RequestT request) {
const Duration timeout = default_timeout_;
const Address from_address = address_;
return implementation_.template Request<RequestT, ResponseT>(to_address, from_address, std::move(request), timeout);
}
/// Wait for an explicit number of microseconds for a request of one of the
/// provided types to arrive. This tends to be used by servers.
template <Message... Ms>
RequestResult<Ms...> ReceiveWithTimeout(Duration timeout) {
return implementation_.template Receive<Ms...>(address_, timeout);
}
/// Wait the default number of microseconds for a request of one of the
/// provided types to arrive. This tends to be used by servers.
template <Message... Ms>
requires(sizeof...(Ms) > 0) RequestResult<Ms...> Receive() {
const Duration timeout = default_timeout_;
return implementation_.template Receive<Ms...>(address_, timeout);
}
/// Send a message in a best-effort fashion. This is used for messaging where
/// responses are not necessarily expected, and for servers to respond to requests.
/// If you need reliable delivery, this must be built on-top. TCP is not enough for most use cases.
template <Message M>
void Send(Address to_address, RequestId request_id, M message) {
Address from_address = address_;
return implementation_.template Send<M>(to_address, from_address, request_id, std::move(message));
}
/// The current system time. This time source should be preferred over any other,
/// because it lets us deterministically control clocks from tests for making
/// things like timeouts deterministic.
Time Now() const { return implementation_.Now(); }
/// Returns true if the system should shut-down.
bool ShouldShutDown() const { return implementation_.ShouldShutDown(); }
/// Returns a random number within the specified distribution.
template <class D = std::poisson_distribution<>, class Return = uint64_t>
Return Rand(D distrib) {
return implementation_.template Rand<D, Return>(distrib);
}
Address GetAddress() { return address_; }
void SetAddress(Address address) { address_ = address; }
Io<I> ForkLocal(boost::uuids::uuid uuid) {
Address new_address{
.unique_id = uuid,
.last_known_ip = address_.last_known_ip,
.last_known_port = address_.last_known_port,
};
return Io(implementation_, new_address);
}
LatencyHistogramSummaries ResponseLatencies() { return implementation_.ResponseLatencies(); }
};
}; // namespace memgraph::io

View File

@@ -0,0 +1,48 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <algorithm>
#include <thread>
#include <boost/asio/ip/tcp.hpp>
#include "io/address.hpp"
#include "storage/v3/property_value.hpp"
#include "storage/v3/schemas.hpp"
namespace memgraph::machine_manager {
using memgraph::io::Address;
using memgraph::storage::v3::SchemaProperty;
using CompoundKey = std::vector<memgraph::storage::v3::PropertyValue>;
struct InitialLabelSpace {
std::string label_name;
std::vector<SchemaProperty> schema;
size_t replication_factor;
std::vector<CompoundKey> split_points;
};
struct MachineConfig {
std::vector<InitialLabelSpace> initial_label_spaces;
std::vector<Address> coordinator_addresses;
bool is_storage;
bool is_coordinator;
bool is_query_engine;
boost::asio::ip::address listen_ip;
uint16_t listen_port;
size_t shard_worker_threads = std::max(static_cast<unsigned int>(1), std::thread::hardware_concurrency());
bool sync_message_handling = false;
};
} // namespace memgraph::machine_manager

View File

@@ -0,0 +1,233 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "coordinator/coordinator_rsm.hpp"
#include "coordinator/coordinator_worker.hpp"
#include "io/message_conversion.hpp"
#include "io/messages.hpp"
#include "io/rsm/rsm_client.hpp"
#include "io/time.hpp"
#include "machine_manager/machine_config.hpp"
#include "storage/v3/shard_manager.hpp"
namespace memgraph::machine_manager {
using coordinator::Coordinator;
using coordinator::CoordinatorReadRequests;
using coordinator::CoordinatorReadResponses;
using coordinator::CoordinatorRsm;
using coordinator::CoordinatorWriteRequests;
using coordinator::CoordinatorWriteResponses;
using coordinator::coordinator_worker::CoordinatorWorker;
using CoordinatorRouteMessage = coordinator::coordinator_worker::RouteMessage;
using CoordinatorQueue = coordinator::coordinator_worker::Queue;
using io::ConvertVariant;
using io::Duration;
using io::RequestId;
using io::Time;
using io::messages::CoordinatorMessages;
using io::messages::ShardManagerMessages;
using io::messages::ShardMessages;
using io::messages::StorageReadRequest;
using io::messages::StorageWriteRequest;
using io::rsm::AppendRequest;
using io::rsm::AppendResponse;
using io::rsm::ReadRequest;
using io::rsm::VoteRequest;
using io::rsm::VoteResponse;
using io::rsm::WriteRequest;
using io::rsm::WriteResponse;
using storage::v3::ShardManager;
/// The MachineManager is responsible for:
/// * starting the entire system and ensuring that high-level
/// operational requirements continue to be met
/// * acting as a machine's single caller of Io::Receive
/// * routing incoming messages from the Io interface to the
/// appropriate Coordinator or to the StorageManager
/// (it's not necessary to route anything in this layer
/// to the query engine because the query engine only
/// communicates using higher-level Futures that will
/// be filled immediately when the response comes in
/// at the lower-level transport layer.
///
/// Every storage engine has exactly one RsmEngine.
template <typename IoImpl>
class MachineManager {
io::Io<IoImpl> io_;
MachineConfig config_;
Address coordinator_address_;
CoordinatorQueue coordinator_queue_;
std::jthread coordinator_handle_;
ShardManager<IoImpl> shard_manager_;
Time next_cron_ = Time::min();
public:
// TODO initialize ShardManager with "real" coordinator addresses instead of io.GetAddress
// which is only true for single-machine config.
MachineManager(io::Io<IoImpl> io, MachineConfig config, Coordinator coordinator)
: io_(io),
config_(config),
coordinator_address_(io.GetAddress().ForkLocalCoordinator()),
shard_manager_{io.ForkLocal(io.GetAddress().ForkLocalShardManager().unique_id), config.shard_worker_threads,
coordinator_address_} {
auto coordinator_io = io.ForkLocal(coordinator_address_.unique_id);
CoordinatorWorker coordinator_worker{coordinator_io, coordinator_queue_, coordinator};
coordinator_handle_ = std::jthread([coordinator = std::move(coordinator_worker)]() mutable { coordinator.Run(); });
}
MachineManager(MachineManager &&) noexcept = default;
MachineManager &operator=(MachineManager &&) noexcept = default;
MachineManager(const MachineManager &) = delete;
MachineManager &operator=(const MachineManager &) = delete;
~MachineManager() {
if (coordinator_handle_.joinable()) {
coordinator_queue_.Push(coordinator::coordinator_worker::ShutDown{});
coordinator_handle_.join();
}
}
Address CoordinatorAddress() { return coordinator_address_; }
void Run() {
while (true) {
MaybeBlockOnSyncHandling();
if (io_.ShouldShutDown()) {
break;
}
const auto now = io_.Now();
uint64_t now_us = now.time_since_epoch().count();
uint64_t next_us = next_cron_.time_since_epoch().count();
if (now >= next_cron_) {
spdlog::info("now {} >= next_cron_ {}", now_us, next_us);
next_cron_ = Cron();
} else {
spdlog::info("now {} < next_cron_ {}", now_us, next_us);
}
Duration receive_timeout = std::max(next_cron_, now) - now;
// Note: this parameter pack must be kept in-sync with the ReceiveWithTimeout parameter pack below
using AllMessages =
std::variant<ReadRequest<CoordinatorReadRequests>, AppendRequest<CoordinatorWriteRequests>, AppendResponse,
WriteRequest<CoordinatorWriteRequests>, VoteRequest, VoteResponse,
WriteResponse<CoordinatorWriteResponses>, ReadRequest<StorageReadRequest>,
AppendRequest<StorageWriteRequest>, WriteRequest<StorageWriteRequest>>;
spdlog::info("MM waiting on Receive on address {}", io_.GetAddress().ToString());
// Note: this parameter pack must be kept in-sync with the AllMessages parameter pack above
auto request_result = io_.template ReceiveWithTimeout<
ReadRequest<CoordinatorReadRequests>, AppendRequest<CoordinatorWriteRequests>, AppendResponse,
WriteRequest<CoordinatorWriteRequests>, VoteRequest, VoteResponse, WriteResponse<CoordinatorWriteResponses>,
ReadRequest<StorageReadRequest>, AppendRequest<StorageWriteRequest>, WriteRequest<StorageWriteRequest>>(
receive_timeout);
if (request_result.HasError()) {
// time to do Cron
continue;
}
auto &&request_envelope = std::move(request_result.GetValue());
spdlog::info("MM got message to {}", request_envelope.to_address.ToString());
// If message is for the coordinator, cast it to subset and pass it to the coordinator
bool to_coordinator = coordinator_address_ == request_envelope.to_address;
if (to_coordinator) {
std::optional<CoordinatorMessages> conversion_attempt =
ConvertVariant<AllMessages, ReadRequest<CoordinatorReadRequests>, AppendRequest<CoordinatorWriteRequests>,
AppendResponse, WriteRequest<CoordinatorWriteRequests>, VoteRequest, VoteResponse>(
std::move(request_envelope.message));
MG_ASSERT(conversion_attempt.has_value(), "coordinator message conversion failed");
spdlog::info("got coordinator message");
CoordinatorMessages &&cm = std::move(conversion_attempt.value());
CoordinatorRouteMessage route_message{
.message = std::move(cm),
.request_id = request_envelope.request_id,
.to = request_envelope.to_address,
.from = request_envelope.from_address,
};
coordinator_queue_.Push(std::move(route_message));
continue;
}
bool to_sm = shard_manager_.GetAddress() == request_envelope.to_address;
spdlog::info("smm: {}", shard_manager_.GetAddress().ToString());
if (to_sm) {
std::optional<ShardManagerMessages> conversion_attempt =
ConvertVariant<AllMessages, WriteResponse<CoordinatorWriteResponses>>(std::move(request_envelope.message));
MG_ASSERT(conversion_attempt.has_value(), "shard manager message conversion failed");
spdlog::info("got shard manager message");
ShardManagerMessages &&smm = std::move(conversion_attempt.value());
shard_manager_.Receive(std::forward<ShardManagerMessages>(smm), request_envelope.request_id,
request_envelope.from_address);
continue;
}
// treat this as a message to a specific shard rsm and cast it accordingly
std::optional<ShardMessages> conversion_attempt =
ConvertVariant<AllMessages, ReadRequest<StorageReadRequest>, AppendRequest<StorageWriteRequest>,
AppendResponse, WriteRequest<StorageWriteRequest>, VoteRequest, VoteResponse>(
std::move(request_envelope.message));
MG_ASSERT(conversion_attempt.has_value(), "shard rsm message conversion failed for {} - incorrect message type",
request_envelope.to_address.ToString());
spdlog::info("got shard rsm message");
ShardMessages &&sm = std::move(conversion_attempt.value());
shard_manager_.Route(std::forward<ShardMessages>(sm), request_envelope.request_id, request_envelope.to_address,
request_envelope.from_address);
}
}
private:
// This method exists for controlling concurrency
// during deterministic simulation testing.
void MaybeBlockOnSyncHandling() {
if (!config_.sync_message_handling) {
return;
}
// block on coordinator
coordinator_queue_.BlockOnQuiescence();
// block on shards
shard_manager_.BlockOnQuiescence();
}
Time Cron() {
spdlog::info("running MachineManager::Cron, address {}", io_.GetAddress().ToString());
coordinator_queue_.Push(coordinator::coordinator_worker::Cron{});
MaybeBlockOnSyncHandling();
Time ret = shard_manager_.Cron();
MaybeBlockOnSyncHandling();
return ret;
}
};
} // namespace memgraph::machine_manager

File diff suppressed because it is too large Load Diff

41
src/parser/CMakeLists.txt Normal file
View File

@@ -0,0 +1,41 @@
## Generate Antlr openCypher parser
set(opencypher_frontend ${CMAKE_CURRENT_SOURCE_DIR}/opencypher)
set(opencypher_generated ${opencypher_frontend}/generated)
set(opencypher_lexer_grammar ${opencypher_frontend}/grammar/MemgraphCypherLexer.g4)
set(opencypher_parser_grammar ${opencypher_frontend}/grammar/MemgraphCypher.g4)
set(antlr_opencypher_generated_src
${opencypher_generated}/MemgraphCypherLexer.cpp
${opencypher_generated}/MemgraphCypher.cpp
${opencypher_generated}/MemgraphCypherBaseVisitor.cpp
${opencypher_generated}/MemgraphCypherVisitor.cpp
)
set(antlr_opencypher_generated_include
${opencypher_generated}/MemgraphCypherLexer.h
${opencypher_generated}/MemgraphCypher.h
${opencypher_generated}/MemgraphCypherBaseVisitor.h
${opencypher_generated}/MemgraphCypherVisitor.h
)
add_custom_command(
OUTPUT ${antlr_opencypher_generated_src} ${antlr_opencypher_generated_include}
COMMAND ${CMAKE_COMMAND} -E make_directory ${opencypher_generated}
COMMAND
java -jar ${CMAKE_SOURCE_DIR}/libs/antlr-4.10.1-complete.jar
-Dlanguage=Cpp -visitor -package antlropencypher
-o ${opencypher_generated}
${opencypher_lexer_grammar} ${opencypher_parser_grammar}
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}"
DEPENDS
${opencypher_lexer_grammar} ${opencypher_parser_grammar}
${opencypher_frontend}/grammar/CypherLexer.g4
${opencypher_frontend}/grammar/Cypher.g4)
add_custom_target(generated_opencypher_parser
DEPENDS ${antlr_opencypher_generated_src} ${antlr_opencypher_generated_include})
add_library(mg-parser STATIC ${antlr_opencypher_generated_src})
add_dependencies(mg-parser generated_opencypher_parser)
target_include_directories(mg-parser PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
target_link_libraries(mg-parser antlr4)

View File

@@ -0,0 +1,391 @@
/*
* Copyright (c) 2015-2016 "Neo Technology,"
* Network Engine for Objects in Lund AB [http://neotechnology.com]
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
parser grammar Cypher;
options { tokenVocab=CypherLexer; }
cypher : statement ';'? EOF ;
statement : query ;
query : cypherQuery
| indexQuery
| explainQuery
| profileQuery
| infoQuery
| constraintQuery
;
constraintQuery : ( CREATE | DROP ) CONSTRAINT ON constraint ;
constraint : '(' nodeName=variable ':' labelName ')' ASSERT EXISTS '(' constraintPropertyList ')'
| '(' nodeName=variable ':' labelName ')' ASSERT constraintPropertyList IS UNIQUE
| '(' nodeName=variable ':' labelName ')' ASSERT '(' constraintPropertyList ')' IS NODE KEY
;
constraintPropertyList : variable propertyLookup ( ',' variable propertyLookup )* ;
storageInfo : STORAGE INFO ;
indexInfo : INDEX INFO ;
constraintInfo : CONSTRAINT INFO ;
infoQuery : SHOW ( storageInfo | indexInfo | constraintInfo ) ;
explainQuery : EXPLAIN cypherQuery ;
profileQuery : PROFILE cypherQuery ;
cypherQuery : singleQuery ( cypherUnion )* ( queryMemoryLimit )? ;
indexQuery : createIndex | dropIndex;
singleQuery : clause ( clause )* ;
cypherUnion : ( UNION ALL singleQuery )
| ( UNION singleQuery )
;
clause : cypherMatch
| unwind
| merge
| create
| set
| cypherDelete
| remove
| with
| cypherReturn
| callProcedure
;
cypherMatch : OPTIONAL? MATCH pattern where? ;
unwind : UNWIND expression AS variable ;
merge : MERGE patternPart ( mergeAction )* ;
mergeAction : ( ON MATCH set )
| ( ON CREATE set )
;
create : CREATE pattern ;
set : SET setItem ( ',' setItem )* ;
setItem : ( propertyExpression '=' expression )
| ( variable '=' expression )
| ( variable '+=' expression )
| ( variable nodeLabels )
;
cypherDelete : DETACH? DELETE expression ( ',' expression )* ;
remove : REMOVE removeItem ( ',' removeItem )* ;
removeItem : ( variable nodeLabels )
| propertyExpression
;
with : WITH ( DISTINCT )? returnBody ( where )? ;
cypherReturn : RETURN ( DISTINCT )? returnBody ;
callProcedure : CALL procedureName '(' ( expression ( ',' expression )* )? ')' ( procedureMemoryLimit )? ( yieldProcedureResults )? ;
procedureName : symbolicName ( '.' symbolicName )* ;
yieldProcedureResults : YIELD ( '*' | ( procedureResult ( ',' procedureResult )* ) ) ;
memoryLimit : MEMORY ( UNLIMITED | LIMIT literal ( MB | KB ) ) ;
queryMemoryLimit : QUERY memoryLimit ;
procedureMemoryLimit : PROCEDURE memoryLimit ;
procedureResult : ( variable AS variable ) | variable ;
returnBody : returnItems ( order )? ( skip )? ( limit )? ;
returnItems : ( '*' ( ',' returnItem )* )
| ( returnItem ( ',' returnItem )* )
;
returnItem : ( expression AS variable )
| expression
;
order : ORDER BY sortItem ( ',' sortItem )* ;
skip : L_SKIP expression ;
limit : LIMIT expression ;
sortItem : expression ( ASCENDING | ASC | DESCENDING | DESC )? ;
where : WHERE expression ;
pattern : patternPart ( ',' patternPart )* ;
patternPart : ( variable '=' anonymousPatternPart )
| anonymousPatternPart
;
anonymousPatternPart : patternElement ;
patternElement : ( nodePattern ( patternElementChain )* )
| ( '(' patternElement ')' )
;
nodePattern : '(' ( variable )? ( nodeLabels )? ( properties )? ')' ;
patternElementChain : relationshipPattern nodePattern ;
relationshipPattern : ( leftArrowHead dash ( relationshipDetail )? dash rightArrowHead )
| ( leftArrowHead dash ( relationshipDetail )? dash )
| ( dash ( relationshipDetail )? dash rightArrowHead )
| ( dash ( relationshipDetail )? dash )
;
leftArrowHead : '<' | LeftArrowHeadPart ;
rightArrowHead : '>' | RightArrowHeadPart ;
dash : '-' | DashPart ;
relationshipDetail : '[' ( name=variable )? ( relationshipTypes )? ( variableExpansion )? properties ']'
| '[' ( name=variable )? ( relationshipTypes )? ( variableExpansion )? relationshipLambda ( total_weight=variable )? (relationshipLambda )? ']'
| '[' ( name=variable )? ( relationshipTypes )? ( variableExpansion )? (properties )* ( relationshipLambda total_weight=variable )? (relationshipLambda )? ']';
relationshipLambda: '(' traversed_edge=variable ',' traversed_node=variable '|' expression ')';
variableExpansion : '*' (BFS | WSHORTEST)? ( expression )? ( '..' ( expression )? )? ;
properties : mapLiteral
| parameter
;
relationshipTypes : ':' relTypeName ( '|' ':'? relTypeName )* ;
nodeLabels : nodeLabel ( nodeLabel )* ;
nodeLabel : ':' labelName ;
labelName : symbolicName ;
relTypeName : symbolicName ;
expression : expression12 ;
expression12 : expression11 ( OR expression11 )* ;
expression11 : expression10 ( XOR expression10 )* ;
expression10 : expression9 ( AND expression9 )* ;
expression9 : ( NOT )* expression8 ;
expression8 : expression7 ( partialComparisonExpression )* ;
expression7 : expression6 ( ( '+' expression6 ) | ( '-' expression6 ) )* ;
expression6 : expression5 ( ( '*' expression5 ) | ( '/' expression5 ) | ( '%' expression5 ) )* ;
expression5 : expression4 ( '^' expression4 )* ;
expression4 : ( ( '+' | '-' ) )* expression3a ;
expression3a : expression3b ( stringAndNullOperators )* ;
stringAndNullOperators : ( ( ( ( '=~' ) | ( IN ) | ( STARTS WITH ) | ( ENDS WITH ) | ( CONTAINS ) ) expression3b) | ( IS CYPHERNULL ) | ( IS NOT CYPHERNULL ) ) ;
expression3b : expression2a ( listIndexingOrSlicing )* ;
listIndexingOrSlicing : ( '[' expression ']' )
| ( '[' lower_bound=expression? '..' upper_bound=expression? ']' )
;
expression2a : expression2b ( nodeLabels )? ;
expression2b : atom ( propertyLookup )* ;
atom : literal
| parameter
| caseExpression
| ( COUNT '(' '*' ')' )
| listComprehension
| patternComprehension
| ( FILTER '(' filterExpression ')' )
| ( EXTRACT '(' extractExpression ')' )
| ( REDUCE '(' reduceExpression ')' )
| ( COALESCE '(' expression ( ',' expression )* ')' )
| ( ALL '(' filterExpression ')' )
| ( ANY '(' filterExpression ')' )
| ( NONE '(' filterExpression ')' )
| ( SINGLE '(' filterExpression ')' )
| relationshipsPattern
| parenthesizedExpression
| functionInvocation
| variable
;
literal : numberLiteral
| StringLiteral
| booleanLiteral
| CYPHERNULL
| mapLiteral
| listLiteral
;
booleanLiteral : TRUE
| FALSE
;
listLiteral : '[' ( expression ( ',' expression )* )? ']' ;
partialComparisonExpression : ( '=' expression7 )
| ( '<>' expression7 )
| ( '!=' expression7 )
| ( '<' expression7 )
| ( '>' expression7 )
| ( '<=' expression7 )
| ( '>=' expression7 )
;
parenthesizedExpression : '(' expression ')' ;
relationshipsPattern : nodePattern ( patternElementChain )+ ;
filterExpression : idInColl ( where )? ;
reduceExpression : accumulator=variable '=' initial=expression ',' idInColl '|' expression ;
extractExpression : idInColl '|' expression ;
idInColl : variable IN expression ;
functionInvocation : functionName '(' ( DISTINCT )? ( expression ( ',' expression )* )? ')' ;
functionName : symbolicName ( '.' symbolicName )* ;
listComprehension : '[' filterExpression ( '|' expression )? ']' ;
patternComprehension : '[' ( variable '=' )? relationshipsPattern ( WHERE expression )? '|' expression ']' ;
propertyLookup : '.' ( propertyKeyName ) ;
caseExpression : ( ( CASE ( caseAlternatives )+ ) | ( CASE test=expression ( caseAlternatives )+ ) ) ( ELSE else_expression=expression )? END ;
caseAlternatives : WHEN when_expression=expression THEN then_expression=expression ;
variable : symbolicName ;
numberLiteral : doubleLiteral
| integerLiteral
;
mapLiteral : '{' ( propertyKeyName ':' expression ( ',' propertyKeyName ':' expression )* )? '}' ;
parameter : '$' ( symbolicName | DecimalLiteral ) ;
propertyExpression : atom ( propertyLookup )+ ;
propertyKeyName : symbolicName ;
integerLiteral : DecimalLiteral
| OctalLiteral
| HexadecimalLiteral
;
createIndex : CREATE INDEX ON ':' labelName ( '(' propertyKeyName ')' )? ;
dropIndex : DROP INDEX ON ':' labelName ( '(' propertyKeyName ')' )? ;
doubleLiteral : FloatingLiteral ;
cypherKeyword : ALL
| AND
| ANY
| AS
| ASC
| ASCENDING
| ASSERT
| BFS
| BY
| CALL
| CASE
| CONSTRAINT
| CONTAINS
| COUNT
| CREATE
| CYPHERNULL
| DELETE
| DESC
| DESCENDING
| DETACH
| DISTINCT
| ELSE
| END
| ENDS
| EXISTS
| EXPLAIN
| EXTRACT
| FALSE
| FILTER
| IN
| INDEX
| INFO
| IS
| KEY
| LIMIT
| L_SKIP
| MATCH
| MERGE
| NODE
| NONE
| NOT
| ON
| OPTIONAL
| OR
| ORDER
| PROCEDURE
| PROFILE
| QUERY
| REDUCE
| REMOVE
| RETURN
| SET
| SHOW
| SINGLE
| STARTS
| STORAGE
| THEN
| TRUE
| UNION
| UNIQUE
| UNWIND
| WHEN
| WHERE
| WITH
| WSHORTEST
| XOR
| YIELD
;
symbolicName : UnescapedSymbolicName
| EscapedSymbolicName
| cypherKeyword
;

View File

@@ -0,0 +1,208 @@
/*
* When changing this grammar make sure to update constants in
* src/parser/stripped_lexer_constants.hpp (kKeywords, kSpecialTokens
* and bitsets) if needed.
*/
lexer grammar CypherLexer ;
import UnicodeCategories ;
/* Skip whitespace and comments. */
Skipped : ( Whitespace | Comment ) -> skip ;
fragment Whitespace : '\u0020'
| [\u0009-\u000D]
| [\u001C-\u001F]
| '\u1680' | '\u180E'
| [\u2000-\u200A]
| '\u2028' | '\u2029'
| '\u205F'
| '\u3000'
| '\u00A0'
| '\u202F'
;
fragment Comment : '/*' .*? '*/'
| '//' ~[\r\n]*
;
/* Special symbols. */
LPAREN : '(' ;
RPAREN : ')' ;
LBRACK : '[' ;
RBRACK : ']' ;
LBRACE : '{' ;
RBRACE : '}' ;
COMMA : ',' ;
DOT : '.' ;
DOTS : '..' ;
COLON : ':' ;
SEMICOLON : ';' ;
DOLLAR : '$' ;
PIPE : '|' ;
EQ : '=' ;
LT : '<' ;
GT : '>' ;
LTE : '<=' ;
GTE : '>=' ;
NEQ1 : '<>' ;
NEQ2 : '!=' ;
SIM : '=~' ;
PLUS : '+' ;
MINUS : '-' ;
ASTERISK : '*' ;
SLASH : '/' ;
PERCENT : '%' ;
CARET : '^' ;
PLUS_EQ : '+=' ;
/* Some random unicode characters that can be used to draw arrows. */
LeftArrowHeadPart : '⟨' | '〈' | '﹤' | '' ;
RightArrowHeadPart : '⟩' | '〉' | '﹥' | '' ;
DashPart : '­' | '' | '' | '' | '' | '—' | '―'
| '' | '' | '﹣' | ''
;
/* Cypher reserved words. */
ALL : A L L ;
AND : A N D ;
ANY : A N Y ;
AS : A S ;
ASC : A S C ;
ASCENDING : A S C E N D I N G ;
ASSERT : A S S E R T ;
BFS : B F S ;
BY : B Y ;
CALL : C A L L ;
CASE : C A S E ;
COALESCE : C O A L E S C E ;
CONSTRAINT : C O N S T R A I N T ;
CONTAINS : C O N T A I N S ;
COUNT : C O U N T ;
CREATE : C R E A T E ;
CYPHERNULL : N U L L ;
DELETE : D E L E T E ;
DESC : D E S C ;
DESCENDING : D E S C E N D I N G ;
DETACH : D E T A C H ;
DISTINCT : D I S T I N C T ;
DROP : D R O P ;
ELSE : E L S E ;
END : E N D ;
ENDS : E N D S ;
EXISTS : E X I S T S ;
EXPLAIN : E X P L A I N ;
EXTRACT : E X T R A C T ;
FALSE : F A L S E ;
FILTER : F I L T E R ;
IN : I N ;
INDEX : I N D E X ;
INFO : I N F O ;
IS : I S ;
KB : K B ;
KEY : K E Y ;
LIMIT : L I M I T ;
L_SKIP : S K I P ;
MATCH : M A T C H ;
MB : M B ;
MEMORY : M E M O R Y ;
MERGE : M E R G E ;
NODE : N O D E ;
NONE : N O N E ;
NOT : N O T ;
ON : O N ;
OPTIONAL : O P T I O N A L ;
OR : O R ;
ORDER : O R D E R ;
PROCEDURE : P R O C E D U R E ;
PROFILE : P R O F I L E ;
QUERY : Q U E R Y ;
REDUCE : R E D U C E ;
REMOVE : R E M O V E ;
RETURN : R E T U R N ;
SET : S E T ;
SHOW : S H O W ;
SINGLE : S I N G L E ;
STARTS : S T A R T S ;
STORAGE : S T O R A G E ;
THEN : T H E N ;
TRUE : T R U E ;
UNION : U N I O N ;
UNIQUE : U N I Q U E ;
UNLIMITED : U N L I M I T E D ;
UNWIND : U N W I N D ;
WHEN : W H E N ;
WHERE : W H E R E ;
WITH : W I T H ;
WSHORTEST : W S H O R T E S T ;
XOR : X O R ;
YIELD : Y I E L D ;
/* Double and single quoted string literals. */
StringLiteral : '"' ( ~[\\"] | EscapeSequence )* '"'
| '\'' ( ~[\\'] | EscapeSequence )* '\''
;
fragment EscapeSequence : '\\' ( B | F | N | R | T | '\\' | '\'' | '"' )
| '\\u' HexDigit HexDigit HexDigit HexDigit
| '\\U' HexDigit HexDigit HexDigit HexDigit
HexDigit HexDigit HexDigit HexDigit
;
/* Number literals. */
DecimalLiteral : '0' | NonZeroDigit ( DecDigit )* ;
OctalLiteral : '0' ( OctDigit )+ ;
HexadecimalLiteral : '0x' ( HexDigit )+ ;
FloatingLiteral : DecDigit* '.' DecDigit+ ( E '-'? DecDigit+ )?
| DecDigit+ ( '.' DecDigit* )? ( E '-'? DecDigit+ )
| DecDigit+ ( E '-'? DecDigit+ )
;
fragment NonZeroDigit : [1-9] ;
fragment DecDigit : [0-9] ;
fragment OctDigit : [0-7] ;
fragment HexDigit : [0-9] | [a-f] | [A-F] ;
/* Symbolic names. */
UnescapedSymbolicName : IdentifierStart ( IdentifierPart )* ;
EscapedSymbolicName : ( '`' ~[`]* '`' )+ ;
/**
* Based on the unicode identifier and pattern syntax
* (http://www.unicode.org/reports/tr31/)
* and extended with a few characters.
*/
IdentifierStart : ID_Start | Pc ;
IdentifierPart : ID_Continue | Sc ;
/* Hack for case-insensitive reserved words */
fragment A : 'A' | 'a' ;
fragment B : 'B' | 'b' ;
fragment C : 'C' | 'c' ;
fragment D : 'D' | 'd' ;
fragment E : 'E' | 'e' ;
fragment F : 'F' | 'f' ;
fragment G : 'G' | 'g' ;
fragment H : 'H' | 'h' ;
fragment I : 'I' | 'i' ;
fragment J : 'J' | 'j' ;
fragment K : 'K' | 'k' ;
fragment L : 'L' | 'l' ;
fragment M : 'M' | 'm' ;
fragment N : 'N' | 'n' ;
fragment O : 'O' | 'o' ;
fragment P : 'P' | 'p' ;
fragment Q : 'Q' | 'q' ;
fragment R : 'R' | 'r' ;
fragment S : 'S' | 's' ;
fragment T : 'T' | 't' ;
fragment U : 'U' | 'u' ;
fragment V : 'V' | 'v' ;
fragment W : 'W' | 'w' ;
fragment X : 'X' | 'x' ;
fragment Y : 'Y' | 'y' ;
fragment Z : 'Z' | 'z' ;

View File

@@ -0,0 +1,399 @@
/*
* Copyright 2021 Memgraph Ltd.
*
* Use of this software is governed by the Business Source License
* included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
* License, and you may not use this file except in compliance with the Business Source License.
*
* As of the Change Date specified in that file, in accordance with
* the Business Source License, use of this software will be governed
* by the Apache License, Version 2.0, included in the file
* licenses/APL.txt.
*/
/* Memgraph specific part of Cypher grammar with enterprise features. */
parser grammar MemgraphCypher ;
options { tokenVocab=MemgraphCypherLexer; }
import Cypher ;
memgraphCypherKeyword : cypherKeyword
| AFTER
| ALTER
| ASYNC
| AUTH
| BAD
| BATCH_INTERVAL
| BATCH_LIMIT
| BATCH_SIZE
| BEFORE
| BOOTSTRAP_SERVERS
| CHECK
| CLEAR
| COMMIT
| COMMITTED
| CONFIG
| CONFIGS
| CONSUMER_GROUP
| CREDENTIALS
| CSV
| DATA
| DELIMITER
| DATABASE
| DENY
| DROP
| DUMP
| EXECUTE
| FREE
| FROM
| FOR
| FOREACH
| GLOBAL
| GRANT
| HEADER
| IDENTIFIED
| ISOLATION
| KAFKA
| LEVEL
| LOAD
| LOCK
| MAIN
| MODE
| NEXT
| NO
| PASSWORD
| PULSAR
| PORT
| PRIVILEGES
| READ
| REGISTER
| REPLICA
| REPLICAS
| REPLICATION
| REVOKE
| ROLE
| ROLES
| QUOTE
| SCHEMA
| SCHEMAS
| SESSION
| SETTING
| SETTINGS
| SNAPSHOT
| START
| STATS
| STREAM
| STREAMS
| SYNC
| TIMEOUT
| TO
| TOPICS
| TRANSACTION
| TRANSFORM
| TRIGGER
| TRIGGERS
| UNCOMMITTED
| UNLOCK
| UPDATE
| USER
| USERS
| VERSION
;
symbolicName : UnescapedSymbolicName
| EscapedSymbolicName
| memgraphCypherKeyword
;
query : cypherQuery
| indexQuery
| explainQuery
| profileQuery
| infoQuery
| constraintQuery
| authQuery
| dumpQuery
| replicationQuery
| lockPathQuery
| freeMemoryQuery
| triggerQuery
| isolationLevelQuery
| createSnapshotQuery
| streamQuery
| settingQuery
| versionQuery
| schemaQuery
;
authQuery : createRole
| dropRole
| showRoles
| createUser
| setPassword
| dropUser
| showUsers
| setRole
| clearRole
| grantPrivilege
| denyPrivilege
| revokePrivilege
| showPrivileges
| showRoleForUser
| showUsersForRole
;
replicationQuery : setReplicationRole
| showReplicationRole
| registerReplica
| dropReplica
| showReplicas
;
triggerQuery : createTrigger
| dropTrigger
| showTriggers
;
clause : cypherMatch
| unwind
| merge
| create
| set
| cypherDelete
| remove
| with
| cypherReturn
| callProcedure
| loadCsv
| foreach
;
updateClause : set
| remove
| create
| merge
| cypherDelete
| foreach
;
foreach : FOREACH '(' variable IN expression '|' updateClause+ ')' ;
streamQuery : checkStream
| createStream
| dropStream
| startStream
| startAllStreams
| stopStream
| stopAllStreams
| showStreams
;
settingQuery : setSetting
| showSetting
| showSettings
;
schemaQuery : showSchema
| showSchemas
| createSchema
| dropSchema
;
loadCsv : LOAD CSV FROM csvFile ( WITH | NO ) HEADER
( IGNORE BAD ) ?
( DELIMITER delimiter ) ?
( QUOTE quote ) ?
AS rowVar ;
csvFile : literal ;
delimiter : literal ;
quote : literal ;
rowVar : variable ;
userOrRoleName : symbolicName ;
createRole : CREATE ROLE role=userOrRoleName ;
dropRole : DROP ROLE role=userOrRoleName ;
showRoles : SHOW ROLES ;
createUser : CREATE USER user=userOrRoleName
( IDENTIFIED BY password=literal )? ;
setPassword : SET PASSWORD FOR user=userOrRoleName TO password=literal;
dropUser : DROP USER user=userOrRoleName ;
showUsers : SHOW USERS ;
setRole : SET ROLE FOR user=userOrRoleName TO role=userOrRoleName;
clearRole : CLEAR ROLE FOR user=userOrRoleName ;
grantPrivilege : GRANT ( ALL PRIVILEGES | privileges=privilegeList ) TO userOrRole=userOrRoleName ;
denyPrivilege : DENY ( ALL PRIVILEGES | privileges=privilegeList ) TO userOrRole=userOrRoleName ;
revokePrivilege : REVOKE ( ALL PRIVILEGES | privileges=privilegeList ) FROM userOrRole=userOrRoleName ;
privilege : CREATE
| DELETE
| MATCH
| MERGE
| SET
| REMOVE
| INDEX
| STATS
| AUTH
| CONSTRAINT
| DUMP
| REPLICATION
| READ_FILE
| FREE_MEMORY
| TRIGGER
| CONFIG
| DURABILITY
| STREAM
| MODULE_READ
| MODULE_WRITE
| WEBSOCKET
| SCHEMA
;
privilegeList : privilege ( ',' privilege )* ;
showPrivileges : SHOW PRIVILEGES FOR userOrRole=userOrRoleName ;
showRoleForUser : SHOW ROLE FOR user=userOrRoleName ;
showUsersForRole : SHOW USERS FOR role=userOrRoleName ;
dumpQuery: DUMP DATABASE ;
setReplicationRole : SET REPLICATION ROLE TO ( MAIN | REPLICA )
( WITH PORT port=literal ) ? ;
showReplicationRole : SHOW REPLICATION ROLE ;
replicaName : symbolicName ;
socketAddress : literal ;
registerReplica : REGISTER REPLICA replicaName ( SYNC | ASYNC )
TO socketAddress ;
dropReplica : DROP REPLICA replicaName ;
showReplicas : SHOW REPLICAS ;
lockPathQuery : ( LOCK | UNLOCK ) DATA DIRECTORY ;
freeMemoryQuery : FREE MEMORY ;
triggerName : symbolicName ;
triggerStatement : .*? ;
emptyVertex : '(' ')' ;
emptyEdge : dash dash rightArrowHead ;
createTrigger : CREATE TRIGGER triggerName ( ON ( emptyVertex | emptyEdge ) ? ( CREATE | UPDATE | DELETE ) ) ?
( AFTER | BEFORE ) COMMIT EXECUTE triggerStatement ;
dropTrigger : DROP TRIGGER triggerName ;
showTriggers : SHOW TRIGGERS ;
isolationLevel : SNAPSHOT ISOLATION | READ COMMITTED | READ UNCOMMITTED ;
isolationLevelScope : GLOBAL | SESSION | NEXT ;
isolationLevelQuery : SET isolationLevelScope TRANSACTION ISOLATION LEVEL isolationLevel ;
createSnapshotQuery : CREATE SNAPSHOT ;
streamName : symbolicName ;
symbolicNameWithMinus : symbolicName ( MINUS symbolicName )* ;
symbolicNameWithDotsAndMinus: symbolicNameWithMinus ( DOT symbolicNameWithMinus )* ;
symbolicTopicNames : symbolicNameWithDotsAndMinus ( COMMA symbolicNameWithDotsAndMinus )* ;
topicNames : symbolicTopicNames | literal ;
commonCreateStreamConfig : TRANSFORM transformationName=procedureName
| BATCH_INTERVAL batchInterval=literal
| BATCH_SIZE batchSize=literal
;
createStream : kafkaCreateStream | pulsarCreateStream ;
configKeyValuePair : literal ':' literal ;
configMap : '{' ( configKeyValuePair ( ',' configKeyValuePair )* )? '}' ;
kafkaCreateStreamConfig : TOPICS topicNames
| CONSUMER_GROUP consumerGroup=symbolicNameWithDotsAndMinus
| BOOTSTRAP_SERVERS bootstrapServers=literal
| CONFIGS configsMap=configMap
| CREDENTIALS credentialsMap=configMap
| commonCreateStreamConfig
;
kafkaCreateStream : CREATE KAFKA STREAM streamName ( kafkaCreateStreamConfig ) * ;
pulsarCreateStreamConfig : TOPICS topicNames
| SERVICE_URL serviceUrl=literal
| commonCreateStreamConfig
;
pulsarCreateStream : CREATE PULSAR STREAM streamName ( pulsarCreateStreamConfig ) * ;
dropStream : DROP STREAM streamName ;
startStream : START STREAM streamName ( BATCH_LIMIT batchLimit=literal ) ? ( TIMEOUT timeout=literal ) ? ;
startAllStreams : START ALL STREAMS ;
stopStream : STOP STREAM streamName ;
stopAllStreams : STOP ALL STREAMS ;
showStreams : SHOW STREAMS ;
checkStream : CHECK STREAM streamName ( BATCH_LIMIT batchLimit=literal ) ? ( TIMEOUT timeout=literal ) ? ;
settingName : literal ;
settingValue : literal ;
setSetting : SET DATABASE SETTING settingName TO settingValue ;
showSetting : SHOW DATABASE SETTING settingName ;
showSettings : SHOW DATABASE SETTINGS ;
versionQuery : SHOW VERSION ;
showSchema : SHOW SCHEMA ON ':' labelName ;
showSchemas : SHOW SCHEMAS ;
propertyType : symbolicName ;
propertyKeyTypePair : propertyKeyName propertyType ;
schemaPropertyMap : '(' propertyKeyTypePair ( ',' propertyKeyTypePair )* ')' ;
createSchema : CREATE SCHEMA ON ':' labelName schemaPropertyMap ;
dropSchema : DROP SCHEMA ON ':' labelName ;

View File

@@ -0,0 +1,118 @@
/*
* Copyright 2021 Memgraph Ltd.
*
* Use of this software is governed by the Business Source License
* included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
* License, and you may not use this file except in compliance with the Business Source License.
*
* As of the Change Date specified in that file, in accordance with
* the Business Source License, use of this software will be governed
* by the Apache License, Version 2.0, included in the file
* licenses/APL.txt.
*/
/* Memgraph specific Cypher reserved words used for enterprise features. */
/*
* When changing this grammar make sure to update constants in
* src/parser/stripped_lexer_constants.hpp (kKeywords, kSpecialTokens
* and bitsets) if needed.
*/
lexer grammar MemgraphCypherLexer ;
import CypherLexer ;
UNDERSCORE : '_' ;
AFTER : A F T E R ;
ALTER : A L T E R ;
ASYNC : A S Y N C ;
AUTH : A U T H ;
BAD : B A D ;
BATCH_INTERVAL : B A T C H UNDERSCORE I N T E R V A L ;
BATCH_LIMIT : B A T C H UNDERSCORE L I M I T ;
BATCH_SIZE : B A T C H UNDERSCORE S I Z E ;
BEFORE : B E F O R E ;
BOOTSTRAP_SERVERS : B O O T S T R A P UNDERSCORE S E R V E R S ;
CHECK : C H E C K ;
CLEAR : C L E A R ;
COMMIT : C O M M I T ;
COMMITTED : C O M M I T T E D ;
CONFIG : C O N F I G ;
CONFIGS : C O N F I G S;
CONSUMER_GROUP : C O N S U M E R UNDERSCORE G R O U P ;
CREDENTIALS : C R E D E N T I A L S ;
CSV : C S V ;
DATA : D A T A ;
DELIMITER : D E L I M I T E R ;
DATABASE : D A T A B A S E ;
DENY : D E N Y ;
DIRECTORY : D I R E C T O R Y ;
DROP : D R O P ;
DUMP : D U M P ;
DURABILITY : D U R A B I L I T Y ;
EXECUTE : E X E C U T E ;
FOR : F O R ;
FOREACH : F O R E A C H;
FREE : F R E E ;
FREE_MEMORY : F R E E UNDERSCORE M E M O R Y ;
FROM : F R O M ;
GLOBAL : G L O B A L ;
GRANT : G R A N T ;
GRANTS : G R A N T S ;
HEADER : H E A D E R ;
IDENTIFIED : I D E N T I F I E D ;
IGNORE : I G N O R E ;
ISOLATION : I S O L A T I O N ;
KAFKA : K A F K A ;
LEVEL : L E V E L ;
LOAD : L O A D ;
LOCK : L O C K ;
MAIN : M A I N ;
MODE : M O D E ;
MODULE_READ : M O D U L E UNDERSCORE R E A D ;
MODULE_WRITE : M O D U L E UNDERSCORE W R I T E ;
NEXT : N E X T ;
NO : N O ;
PASSWORD : P A S S W O R D ;
PORT : P O R T ;
PRIVILEGES : P R I V I L E G E S ;
PULSAR : P U L S A R ;
READ : R E A D ;
READ_FILE : R E A D UNDERSCORE F I L E ;
REGISTER : R E G I S T E R ;
REPLICA : R E P L I C A ;
REPLICAS : R E P L I C A S ;
REPLICATION : R E P L I C A T I O N ;
REVOKE : R E V O K E ;
ROLE : R O L E ;
ROLES : R O L E S ;
QUOTE : Q U O T E ;
SCHEMA : S C H E M A ;
SCHEMAS : S C H E M A S ;
SERVICE_URL : S E R V I C E UNDERSCORE U R L ;
SESSION : S E S S I O N ;
SETTING : S E T T I N G ;
SETTINGS : S E T T I N G S ;
SNAPSHOT : S N A P S H O T ;
START : S T A R T ;
STATS : S T A T S ;
STOP : S T O P ;
STREAM : S T R E A M ;
STREAMS : S T R E A M S ;
SYNC : S Y N C ;
TIMEOUT : T I M E O U T ;
TO : T O ;
TOPICS : T O P I C S;
TRANSACTION : T R A N S A C T I O N ;
TRANSFORM : T R A N S F O R M ;
TRIGGER : T R I G G E R ;
TRIGGERS : T R I G G E R S ;
UNCOMMITTED : U N C O M M I T T E D ;
UNLOCK : U N L O C K ;
UPDATE : U P D A T E ;
USER : U S E R ;
USERS : U S E R S ;
VERSION : V E R S I O N ;
WEBSOCKET : W E B S O C K E T ;

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,85 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <string>
#include "antlr4-runtime.h"
#include "utils/exceptions.hpp"
#include "parser/opencypher/generated/MemgraphCypher.h"
#include "parser/opencypher/generated/MemgraphCypherLexer.h"
#include "utils/concepts.hpp"
namespace memgraph::frontend::opencypher {
class SyntaxException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
SyntaxException() : SyntaxException("") {}
};
/**
* Generates openCypher AST
* This thing must me a class since parser.cypher() returns pointer and there is
* no way for us to get ownership over the object.
*/
enum class ParserOpTag : uint8_t {
CYPHER, EXPRESSION
};
template<ParserOpTag Tag = ParserOpTag::CYPHER>
class Parser {
public:
/**
* @param query incoming query that has to be compiled into query plan
* the first step is to generate AST
*/
Parser(const std::string query) : query_(std::move(query)) {
parser_.removeErrorListeners();
parser_.addErrorListener(&error_listener_);
if constexpr(Tag == ParserOpTag::CYPHER) {
tree_ = parser_.cypher();
}
else {
tree_ = parser_.expression();
}
if (parser_.getNumberOfSyntaxErrors()) {
throw SyntaxException(error_listener_.error_);
}
}
auto tree() { return tree_; }
private:
class FirstMessageErrorListener : public antlr4::BaseErrorListener {
void syntaxError(antlr4::Recognizer *, antlr4::Token *, size_t line, size_t position, const std::string &message,
std::exception_ptr) override {
if (error_.empty()) {
error_ = "line " + std::to_string(line) + ":" + std::to_string(position + 1) + " " + message;
}
}
public:
std::string error_;
};
FirstMessageErrorListener error_listener_;
std::string query_;
antlr4::ANTLRInputStream input_{query_};
antlropencypher::MemgraphCypherLexer lexer_{&input_};
antlr4::CommonTokenStream tokens_{&lexer_};
// generate ast
antlropencypher::MemgraphCypher parser_{&tokens_};
antlr4::tree::ParseTree *tree_ = nullptr;
};
} // namespace memgraph::frontend::opencypher

File diff suppressed because it is too large Load Diff

View File

@@ -51,7 +51,6 @@ class EdgeAccessor final {
public:
storage::EdgeAccessor impl_;
public:
explicit EdgeAccessor(storage::EdgeAccessor impl) : impl_(std::move(impl)) {}
bool IsVisible(storage::View view) const { return impl_.IsVisible(view); }
@@ -97,7 +96,6 @@ class VertexAccessor final {
static EdgeAccessor MakeEdgeAccessor(const storage::EdgeAccessor impl) { return EdgeAccessor(impl); }
public:
explicit VertexAccessor(storage::VertexAccessor impl) : impl_(impl) {}
bool IsVisible(storage::View view) const { return impl_.IsVisible(view); }

View File

@@ -204,7 +204,7 @@ const trie::Trie kKeywords = {"union",
"pulsar",
"service_url",
"version",
"websocket"
"websocket",
"foreach"};
// Unicode codepoints that are allowed at the start of the unescaped name.

View File

@@ -114,4 +114,4 @@ std::string ExecutionStatsKeyToString(const ExecutionStats::Key key) {
}
}
} // namespace memgraph::query
} // namespace memgraph::query

View File

@@ -368,13 +368,12 @@ VertexAccessor &CreateExpand::CreateExpandCursor::OtherVertex(Frame &frame, Exec
TypedValue &dest_node_value = frame[self_.node_info_.symbol];
ExpectType(self_.node_info_.symbol, dest_node_value, TypedValue::Type::Vertex);
return dest_node_value.ValueVertex();
} else {
auto &created_vertex = CreateLocalVertex(self_.node_info_, &frame, context);
if (context.trigger_context_collector) {
context.trigger_context_collector->RegisterCreatedObject(created_vertex);
}
return created_vertex;
}
auto &created_vertex = CreateLocalVertex(self_.node_info_, &frame, context);
if (context.trigger_context_collector) {
context.trigger_context_collector->RegisterCreatedObject(created_vertex);
}
return created_vertex;
}
template <class TVerticesFun>

View File

@@ -0,0 +1,48 @@
define_add_lcp(add_lcp_query lcp_query_v2_cpp_files generated_lcp_query_v2_files)
add_lcp_query(frontend/ast/ast.lcp)
add_lcp_query(plan/operator.lcp)
add_custom_target(generate_lcp_query_v2 DEPENDS ${generated_lcp_query_v2_files})
set(mg_query_v2_sources
${lcp_query_v2_cpp_files}
common.cpp
cypher_query_interpreter.cpp
frontend/semantic/required_privileges.cpp
frontend/stripped.cpp
interpreter.cpp
metadata.cpp
plan/operator.cpp
plan/preprocess.cpp
plan/pretty_print.cpp
plan/profile.cpp
plan/read_write_type_checker.cpp
plan/rewrite/index_lookup.cpp
plan/rule_based_planner.cpp
plan/variable_start_planner.cpp
serialization/property_value.cpp
bindings/typed_value.cpp
accessors.cpp)
find_package(Boost REQUIRED)
add_library(mg-query-v2 STATIC ${mg_query_v2_sources})
add_dependencies(mg-query-v2 generate_lcp_query_v2)
target_include_directories(mg-query-v2 PUBLIC ${CMAKE_SOURCE_DIR}/include)
target_include_directories(mg-query-v2 PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/bindings)
target_link_libraries(mg-query-v2 dl cppitertools Boost::headers)
target_link_libraries(mg-query-v2 mg-integrations-pulsar mg-integrations-kafka mg-storage-v3 mg-license mg-utils mg-kvstore mg-memory mg-coordinator)
target_link_libraries(mg-query-v2 mg-expr mg-functions)
if(NOT "${MG_PYTHON_PATH}" STREQUAL "")
set(Python3_ROOT_DIR "${MG_PYTHON_PATH}")
endif()
if("${MG_PYTHON_VERSION}" STREQUAL "")
find_package(Python3 3.5 REQUIRED COMPONENTS Development)
else()
find_package(Python3 "${MG_PYTHON_VERSION}" EXACT REQUIRED COMPONENTS Development)
endif()
target_link_libraries(mg-query-v2 Python3::Python)

103
src/query/v2/accessors.cpp Normal file
View File

@@ -0,0 +1,103 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "query/v2/accessors.hpp"
#include "query/v2/request_router.hpp"
#include "query/v2/requests.hpp"
#include "storage/v3/id_types.hpp"
namespace memgraph::query::v2::accessors {
EdgeAccessor::EdgeAccessor(Edge edge, const RequestRouterInterface *request_router)
: edge(std::move(edge)), request_router_(request_router) {}
EdgeTypeId EdgeAccessor::EdgeType() const { return edge.type.id; }
const std::vector<std::pair<PropertyId, Value>> &EdgeAccessor::Properties() const { return edge.properties; }
Value EdgeAccessor::GetProperty(const std::string &prop_name) const {
auto maybe_prop = request_router_->MaybeNameToProperty(prop_name);
if (!maybe_prop) {
return {};
}
const auto prop_id = *maybe_prop;
auto it = std::find_if(edge.properties.begin(), edge.properties.end(), [&](auto &pr) { return prop_id == pr.first; });
return it->second;
}
const Edge &EdgeAccessor::GetEdge() const { return edge; }
bool EdgeAccessor::IsCycle() const { return edge.src == edge.dst; };
size_t EdgeAccessor::CypherId() const { return edge.id.gid; }
VertexAccessor EdgeAccessor::To() const {
return VertexAccessor(Vertex{edge.dst}, std::vector<std::pair<PropertyId, msgs::Value>>{}, request_router_);
}
VertexAccessor EdgeAccessor::From() const {
return VertexAccessor(Vertex{edge.src}, std::vector<std::pair<PropertyId, msgs::Value>>{}, request_router_);
}
VertexAccessor::VertexAccessor(Vertex v, std::vector<std::pair<PropertyId, Value>> props,
const RequestRouterInterface *request_router)
: vertex(std::move(v)), properties(std::move(props)), request_router_(request_router) {}
VertexAccessor::VertexAccessor(Vertex v, std::map<PropertyId, Value> &&props,
const RequestRouterInterface *request_router)
: vertex(std::move(v)), request_router_(request_router) {
properties.reserve(props.size());
for (auto &[id, value] : props) {
properties.emplace_back(std::make_pair(id, std::move(value)));
}
}
VertexAccessor::VertexAccessor(Vertex v, const std::map<PropertyId, Value> &props,
const RequestRouterInterface *request_router)
: vertex(std::move(v)), request_router_(request_router) {
properties.reserve(props.size());
for (const auto &[id, value] : props) {
properties.emplace_back(std::make_pair(id, value));
}
}
Label VertexAccessor::PrimaryLabel() const { return vertex.id.first; }
const msgs::VertexId &VertexAccessor::Id() const { return vertex.id; }
std::vector<Label> VertexAccessor::Labels() const { return vertex.labels; }
bool VertexAccessor::HasLabel(Label &label) const {
return std::find_if(vertex.labels.begin(), vertex.labels.end(),
[label](const auto &l) { return l.id == label.id; }) != vertex.labels.end();
}
const std::vector<std::pair<PropertyId, Value>> &VertexAccessor::Properties() const { return properties; }
Value VertexAccessor::GetProperty(PropertyId prop_id) const {
auto it = std::find_if(properties.begin(), properties.end(), [&](auto &pr) { return prop_id == pr.first; });
if (it == properties.end()) {
return {};
}
return it->second;
}
// NOLINTNEXTLINE(readability-convert-member-functions-to-static)
Value VertexAccessor::GetProperty(const std::string &prop_name) const {
auto maybe_prop = request_router_->MaybeNameToProperty(prop_name);
if (!maybe_prop) {
return {};
}
return GetProperty(*maybe_prop);
}
msgs::Vertex VertexAccessor::GetVertex() const { return vertex; }
} // namespace memgraph::query::v2::accessors

164
src/query/v2/accessors.hpp Normal file
View File

@@ -0,0 +1,164 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <map>
#include <optional>
#include <utility>
#include <vector>
#include "query/exceptions.hpp"
#include "query/v2/requests.hpp"
#include "storage/v3/view.hpp"
#include "utils/bound.hpp"
#include "utils/exceptions.hpp"
#include "utils/memory.hpp"
#include "utils/memory_tracker.hpp"
namespace memgraph::query::v2 {
class RequestRouterInterface;
} // namespace memgraph::query::v2
namespace memgraph::query::v2::accessors {
using Value = msgs::Value;
using Edge = msgs::Edge;
using Vertex = msgs::Vertex;
using Label = msgs::Label;
using PropertyId = msgs::PropertyId;
using EdgeTypeId = msgs::EdgeTypeId;
class VertexAccessor;
class EdgeAccessor final {
public:
explicit EdgeAccessor(Edge edge, const RequestRouterInterface *request_router);
[[nodiscard]] EdgeTypeId EdgeType() const;
[[nodiscard]] const std::vector<std::pair<PropertyId, Value>> &Properties() const;
[[nodiscard]] Value GetProperty(const std::string &prop_name) const;
[[nodiscard]] const Edge &GetEdge() const;
[[nodiscard]] bool IsCycle() const;
[[nodiscard]] size_t CypherId() const;
[[nodiscard]] VertexAccessor To() const;
[[nodiscard]] VertexAccessor From() const;
friend bool operator==(const EdgeAccessor &lhs, const EdgeAccessor &rhs) { return lhs.edge == rhs.edge; }
friend bool operator!=(const EdgeAccessor &lhs, const EdgeAccessor &rhs) { return !(lhs == rhs); }
private:
Edge edge;
const RequestRouterInterface *request_router_;
};
class VertexAccessor final {
public:
using PropertyId = msgs::PropertyId;
using Label = msgs::Label;
using VertexId = msgs::VertexId;
VertexAccessor(Vertex v, std::vector<std::pair<PropertyId, Value>> props,
const RequestRouterInterface *request_router);
VertexAccessor(Vertex v, std::map<PropertyId, Value> &&props, const RequestRouterInterface *request_router);
VertexAccessor(Vertex v, const std::map<PropertyId, Value> &props, const RequestRouterInterface *request_router);
[[nodiscard]] Label PrimaryLabel() const;
[[nodiscard]] const msgs::VertexId &Id() const;
[[nodiscard]] std::vector<Label> Labels() const;
[[nodiscard]] bool HasLabel(Label &label) const;
[[nodiscard]] const std::vector<std::pair<PropertyId, Value>> &Properties() const;
[[nodiscard]] Value GetProperty(PropertyId prop_id) const;
[[nodiscard]] Value GetProperty(const std::string &prop_name) const;
[[nodiscard]] msgs::Vertex GetVertex() const;
// NOLINTNEXTLINE(readability-convert-member-functions-to-static)
[[nodiscard]] size_t InDegree() const { throw utils::NotYetImplemented("InDegree() not yet implemented"); }
// NOLINTNEXTLINE(readability-convert-member-functions-to-static)
[[nodiscard]] size_t OutDegree() const { throw utils::NotYetImplemented("OutDegree() not yet implemented"); }
friend bool operator==(const VertexAccessor &lhs, const VertexAccessor &rhs) {
return lhs.vertex == rhs.vertex && lhs.properties == rhs.properties;
}
friend bool operator!=(const VertexAccessor &lhs, const VertexAccessor &rhs) { return !(lhs == rhs); }
private:
Vertex vertex;
std::vector<std::pair<PropertyId, Value>> properties;
const RequestRouterInterface *request_router_;
};
// Highly mocked interface. Won't work if used.
class Path {
public:
// Empty for now
explicit Path(const VertexAccessor & /*vertex*/, utils::MemoryResource *memory = utils::NewDeleteResource())
: mem(memory) {}
template <typename... TOthers>
explicit Path(const VertexAccessor &vertex, const TOthers &...others) {}
template <typename... TOthers>
Path(std::allocator_arg_t /*unused*/, utils::MemoryResource *memory, const VertexAccessor &vertex,
const TOthers &...others) {}
Path(const Path & /*other*/) {}
Path(const Path & /*other*/, utils::MemoryResource *memory) : mem(memory) {}
Path(Path && /*other*/) noexcept {}
Path(Path && /*other*/, utils::MemoryResource *memory) : mem(memory) {}
Path &operator=(const Path &path) {
if (this == &path) {
return *this;
}
return *this;
}
Path &operator=(Path &&path) noexcept {
if (this == &path) {
return *this;
}
return *this;
}
~Path() {}
friend bool operator==(const Path & /*lhs*/, const Path & /*rhs*/) { return true; };
utils::MemoryResource *GetMemoryResource() { return mem; }
auto &vertices() { return vertices_; }
auto &edges() { return edges_; }
const auto &vertices() const { return vertices_; }
const auto &edges() const { return edges_; }
private:
std::vector<VertexAccessor> vertices_;
std::vector<EdgeAccessor> edges_;
utils::MemoryResource *mem = utils::NewDeleteResource();
};
} // namespace memgraph::query::v2::accessors

View File

@@ -0,0 +1,29 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "query/v2/frontend/ast/ast.hpp"
namespace memgraph::query::v2 {
class AuthChecker {
public:
virtual bool IsUserAuthorized(const std::optional<std::string> &username,
const std::vector<query::v2::AuthQuery::Privilege> &privileges) const = 0;
};
class AllowEverythingAuthChecker final : public query::v2::AuthChecker {
bool IsUserAuthorized(const std::optional<std::string> & /*username*/,
const std::vector<query::v2::AuthQuery::Privilege> & /*privileges*/) const override {
return true;
}
};
} // namespace memgraph::query::v2

View File

@@ -0,0 +1,16 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "query/v2/bindings/bindings.hpp"
#include "expr/ast/ast_visitor.hpp"

View File

@@ -0,0 +1,19 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#ifdef MG_AST_INCLUDE_PATH
#error You are probably trying to include files from expr from both the storage and query engines! You will have a rough time kid!
#endif
#define MG_AST_INCLUDE_PATH "query/v2/frontend/ast/ast.hpp" // NOLINT(cppcoreguidelines-macro-usage)
#define MG_INJECTED_NAMESPACE_NAME memgraph::query::v2 // NOLINT(cppcoreguidelines-macro-usage)

View File

@@ -0,0 +1,20 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "query/v2/bindings/bindings.hpp"
#include "expr/ast/cypher_main_visitor.hpp"
namespace memgraph::query::v2 {
using CypherMainVisitor = memgraph::expr::CypherMainVisitor;
} // namespace memgraph::query::v2

View File

@@ -0,0 +1,47 @@
// Copyright 2022 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include "query/v2/bindings/bindings.hpp"
#include "expr/interpret/eval.hpp"
#include "query/v2/bindings/typed_value.hpp"
#include "query/v2/context.hpp"
#include "query/v2/conversions.hpp"
#include "query/v2/db_accessor.hpp"
#include "query/v2/requests.hpp"
#include "storage/v3/conversions.hpp"
#include "storage/v3/property_value.hpp"
#include "storage/v3/result.hpp"
#include "storage/v3/view.hpp"
namespace memgraph::query::v2 {
class RequestRouterInterface;
namespace detail {
class Callable {
public:
auto operator()(const storage::v3::PropertyValue &val) const {
return storage::v3::PropertyToTypedValue<TypedValue>(val);
};
auto operator()(const msgs::Value &val, RequestRouterInterface *request_router) const {
return ValueToTypedValue(val, request_router);
};
};
} // namespace detail
using ExpressionEvaluator = expr::ExpressionEvaluator<TypedValue, query::v2::EvaluationContext, RequestRouterInterface,
storage::v3::View, storage::v3::LabelId, msgs::Value,
detail::Callable, common::ErrorCode, expr::QueryEngineTag>;
} // namespace memgraph::query::v2

Some files were not shown because too many files have changed in this diff Show More