Compare commits

...

65 Commits

Author SHA1 Message Date
gvolfing
23474fd6cc initial changes 2023-05-04 08:31:08 +02:00
Antonio Filipovic
91017b7f36 Update profile query to use PoolResource for LOAD CSV (#885) 2023-04-26 18:04:13 +02:00
gvolfing
00f8d54249 Parallelize index creation (#882) 2023-04-26 16:28:02 +02:00
János Benjamin Antal
4fcdd52f88 Use correct memory resource (#900) 2023-04-26 10:02:55 +02:00
János Benjamin Antal
6c947947eb Parallelize recovery (#868)
* Parallelize edge recovery

* Load vertex labels and properties parallel

* Add parallel connectivity loading

* Add batches information to snapshot

* Introduce `items_per_batch` and `recovery_thread_count` flags

* Make possible to load snapshots with old version

* Add vertex batches to `RecoveryInfo`

* Extend durability integration tests with v15 test cases

* Add `std::vector` based `InitProperties`

* Use `InitProperties` in snapshot loading
2023-04-25 16:25:25 +02:00
Ante Javor
64fd281b2e Update benchgraph methodology (#899) 2023-04-25 09:45:25 +02:00
János Benjamin Antal
97e250129e Change AccumulateCursor to use utils::pmr::deque (#888)
* Increase performance by eliminating unnecessary `TypedValue` copies
2023-04-24 16:22:22 +02:00
Marko Budiselić
b02b201129 Improve Jepsen setup (#893) 2023-04-23 16:16:49 +02:00
Antonio Filipovic
2c6a55775d Fix max block size bug on LOAD CSV(#877) 2023-04-19 16:10:20 +02:00
Ante Javor
940bf6722c Add mgbench tutorial (#836)
* Add Docker runner
* Add Docker client
* Add benchgraph.sh script
* Add package script
2023-04-19 08:21:55 +02:00
Bruno Sačarić
49b5343238 Update license year (#867) 2023-04-05 11:06:32 +02:00
Andi
26a0866938 Fix index optimization bug (#860) 2023-04-04 23:43:13 +02:00
Bruno Sačarić
6545283dac Add upload to S3 job to Package All workflow (#845)
* add to package_all

* add separate workflow

* make reduced jobs version for testing

* typo

* exclude amzn-2 in init script, because isort 5.12. fails to install

* change dir name

* move env var for release version

* bugfix

* Revert "make reduced jobs version for testing"

This reverts commit 7bb75f34a4.

* remove releaes folder

* extend timeout for arm builds

* increase timeout limit to dangerous levels

* revert timeouts, fix upload naming

* remove untested workflow
2023-04-04 21:54:25 +02:00
Bruno Sačarić
69c735934c Update docker credentials (#853) 2023-04-04 21:47:04 +02:00
Antonio Filipovic
64e837b355 Introduce analytics mode (#772) 2023-04-04 18:46:26 +02:00
Antonio Filipovic
a586f2f98d Change EvalContext and QueryExecution to use PoolResource on LOAD CSV (#825)
* Change PullPlan to use specific PoolResource for LOAD CSV
2023-04-04 16:54:08 +02:00
Josipmrden
9fc51f74a0 Skip label based auth on user with global visibility on graph (#837) 2023-04-04 11:13:25 +02:00
Josipmrden
128771a6ec Add SHA-256 password encryption (#839) 2023-04-03 16:29:21 +02:00
Josipmrden
f5a49ed29f Add Cypher subqueries (#794) (#851)
Co-authored-by: Bruno Sačarić <bruno.sacaric@gmail.com>
2023-03-31 13:49:10 +00:00
Josipmrden
398503da7a Add index statistics for better query planning (#812) 2023-03-30 15:34:34 +02:00
Bruno Sačarić
0819b40202 Fix bug on AllShortest with multiple edges between nodes (#832) 2023-03-29 16:39:41 +02:00
Andi
029be10f1d Add queries to show or terminate active transactions (#790) 2023-03-27 15:46:00 +02:00
Aidar Samerkhanov
a9dc344b49 Add automatic CPU architecture detection to CMake (#838) 2023-03-27 11:26:10 +02:00
Marko Budiselić
8b0dca9eab Upgrade pre-commit hook to use isort 5.12 (#840) 2023-03-26 17:34:51 +02:00
Ante Javor
cb813c3070 Add bigger LDBC dataset to mgbench (#747) 2023-03-21 21:44:11 +01:00
Ante Javor
6349fc9501 Add time-depended execution to the mgbench client (#805) 2023-03-18 20:18:58 +01:00
Jure Bajic
c4167bafdd Add support for Amazon Linux 2 and stop generating C++ using Lisp/LCP (#814) 2023-03-14 19:24:55 +01:00
Bruno Sačarić
6f51141148 Update license year(#821) 2023-03-08 10:46:21 +01:00
Ante Pušić
97d45ab1d8 Add Python query module API mock (#757) 2023-03-07 15:41:19 +01:00
Josipmrden
6abd356d01 [master < E214] WHERE Exists feature (#818)
Add WHERE exists() to filter based on neighbouring pattern expressions
2023-03-07 00:28:41 +01:00
Vlasta
99a6c72bba Change message on incompatible epoch_id error (#786) 2023-03-06 20:01:02 +01:00
Ante Pušić
173f5430aa Remove noexcept from functions that may throw (#819) 2023-03-06 17:34:34 +01:00
Jure Bajic
362dc95e27 Add support for Ubuntu 22.04 ARM (#810) 2023-03-01 18:44:56 +01:00
Kruno Golubic
eead24a562 Add Lima badge to README (#802) 2023-02-24 17:08:07 +01:00
Antonio Filipovic
d79dd69607 Improve performance with props init on node|edge creation (#788) 2023-02-24 15:40:35 +01:00
Katarina Supe
024bf0c578 Merge pull request #806 from memgraph/vpavicic-patch-1
Update README.md
2023-02-23 15:06:40 +01:00
Vlasta
d7443a5558 Update README.md 2023-02-23 15:05:03 +01:00
Kruno Golubic
2df357b012 Add RedHat badge to README (#796) 2023-02-20 18:18:29 +01:00
Kruno Golubic
b2b5a6e2a0 Add Fedora badge to README (#795) 2023-02-20 17:59:57 +01:00
Ante Javor
5e2ee6c817 Improve mgbench C++ client (#760) 2023-02-17 17:54:05 +01:00
Antonio Filipovic
862a1afdf1 Improve Visit performance (#774) 2023-02-17 13:09:25 +01:00
Antonio Filipovic
bbce21e78f Update pull request template (#775) 2023-02-17 11:50:17 +01:00
Jure Bajic
15c8662023 Add support for Fedora 36 (#787) 2023-02-17 10:47:36 +01:00
Katarina Supe
beaba0fc16 Update README with Cloud, Lab and import info (#768) 2023-02-07 15:43:41 +01:00
Josipmrden
8f70c5f2a5 Fix label-based auth using OLD view instead of NEW when merging nodes (#755) 2023-02-01 13:20:26 +01:00
Bruno Sačarić
14c651d3ba edit workflows (#756) 2023-01-31 23:20:49 +01:00
Andi
04efc7a4a6 Remove torch and igraph from sys cache (#720) 2023-01-31 13:11:59 +01:00
Bruno Sačarić
19832b5838 Merge pull request #749 from memgraph/update-license-year-2.5.2
[master < MG] Update license year
2023-01-26 13:08:47 +01:00
Bruno Sačarić
e63fae2d5b Update license year 2023-01-26 13:02:33 +01:00
Bruno Sačarić
34dd47ef07 Fix nested FOREACH shadowing bug (#725) 2023-01-25 20:06:05 +01:00
Jure Bajic
d3f275b231 Exclude license dir from CI trigger (#740) 2023-01-25 18:49:02 +01:00
Ante Pušić
aad4bcb7a0 Fix C++ API memory leak on Relationships() (#743) 2023-01-25 17:23:46 +01:00
Bruno Sačarić
034b54cb72 Fix bug on all shortest paths with an upper bound (#737) 2023-01-25 15:32:00 +01:00
Josipmrden
8cf51d9f68 Fix bug in query plan to use indexes on optional match and foreach (#736)
* Add fix in query plan to use indexes on optional match and foreach
2023-01-25 12:53:33 +01:00
Antonio Filipovic
1cd1da84fd Fix bug on (vertex|edge) properties in C++ API (#732) 2023-01-23 12:57:17 +01:00
Jure Bajic
128a6cd522 Update license year (#739) 2023-01-19 12:31:59 +01:00
niko4299
d9eeedb9ee Adding qid in bolt (#721) 2023-01-18 16:33:03 +01:00
Andi
156e2cd095 On delete triggers invalid edge reference (#717)
* Added check if there is invalid reference to the underlying edge

* Added fix and e2e tests

* Isolation levels tracking based on from_vertex_

* Added explicit transaction test + edge accessor changes based on the vertex_edge

* Autocommit on tests, initialize deleted by checking out_edges

Co-authored-by: Marko Budiselić <marko.budiselic@memgraph.com>
2023-01-18 15:05:10 +01:00
Ante Javor
8b834c702c Update mgbench to run Diff workflow under 30mins (#730) 2023-01-14 16:11:49 +01:00
Katarina Supe
eda5213d95 Release pypi mgp 1.1.1 (#727) 2022-12-24 09:33:53 +01:00
Bruno Sačarić
1f2a15e7c8 Fix MATCH not allowed on replica (#709) 2022-12-23 14:47:12 +01:00
Ante Javor
d72e7fa38d Fix mgp.py create edge type hint and comment (#724) 2022-12-23 10:08:52 +01:00
Antonio Filipovic
e5e37bc14a Fix LOAD CSV large memory usage (#712) 2022-12-22 19:38:48 +01:00
Katarina Supe
3ee068bbf9 Update build badge (#722) 2022-12-20 21:56:05 +01:00
Jure Bajic
68e846b182 Update license year (#711) 2022-12-13 13:54:09 +01:00
274 changed files with 26731 additions and 2993 deletions

View File

@@ -1,11 +1,14 @@
[master < Epic] PR
- [ ] Check, and update documentation if necessary
- [ ] Update [changelog](https://docs.memgraph.com/memgraph/changelog)
- [ ] Write E2E tests
- [ ] Compare the [benchmarking results](https://bench-graph.memgraph.com/) between the master branch and the Epic branch
- [ ] Provide the full content or a guide for the final git message
[master < Task] PR
- [ ] Check, and update documentation if necessary
- [ ] Update [changelog](https://docs.memgraph.com/memgraph/changelog)
- [ ] Provide the full content or a guide for the final git message
To keep docs changelog up to date, one more thing to do:
- [ ] Write a release note here
- [ ] Tag someone from docs team in the comments

View File

@@ -3,7 +3,7 @@ name: Daily Benchmark
on:
workflow_dispatch:
schedule:
- cron: "0 1 * * *"
- cron: "0 22 * * *"
jobs:
release_benchmarks:

View File

@@ -14,6 +14,7 @@ on:
- "**/*.md"
- ".clang-format"
- "CODEOWNERS"
- "licenses/*"
jobs:
community_build:
@@ -436,7 +437,7 @@ jobs:
- name: Run mgbench
run: |
cd tests/mgbench
./benchmark.py --num-workers-for-benchmark 12 --export-results benchmark_result.json pokec/medium/*/*
./benchmark.py vendor-native --num-workers-for-benchmark 12 --export-results benchmark_result.json pokec/medium/*/*
- name: Upload mgbench results
run: |

View File

@@ -2,7 +2,12 @@ name: Package All
# TODO(gitbuda): Cleanup docker container if GHA job was canceled.
on: workflow_dispatch
on:
workflow_dispatch:
inputs:
memgraph_version:
description: "Memgraph version to upload as. If empty upload is skipped. Format: 'vX.Y.Z'"
required: false
jobs:
centos-7:
@@ -106,7 +111,7 @@ jobs:
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: ubuntu-1804
name: ubuntu-18.04
path: build/output/ubuntu-18.04/memgraph*.deb
ubuntu-2004:
@@ -123,7 +128,7 @@ jobs:
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: ubuntu-2004
name: ubuntu-20.04
path: build/output/ubuntu-20.04/memgraph*.deb
ubuntu-2204:
@@ -140,7 +145,7 @@ jobs:
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: ubuntu-2204
name: ubuntu-22.04
path: build/output/ubuntu-22.04/memgraph*.deb
debian-11-platform:
@@ -160,6 +165,40 @@ jobs:
name: debian-11-platform
path: build/output/debian-11/memgraph*.deb
fedora-36:
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package fedora-36
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: fedora-36
path: build/output/fedora-36/memgraph*.rpm
amzn-2:
runs-on: [self-hosted, DockerMgBuild, X64]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package amzn-2
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: amzn-2
path: build/output/amzn-2/memgraph*.rpm
debian-11-arm:
runs-on: [self-hosted, DockerMgBuild, ARM64, strange]
timeout-minutes: 60
@@ -174,5 +213,44 @@ jobs:
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: debian-11-arm
name: debian-11-aarch64
path: build/output/debian-11-arm/memgraph*.deb
ubuntu-2204-arm:
runs-on: [self-hosted, DockerMgBuild, ARM64, strange]
timeout-minutes: 60
steps:
- name: "Set up repository"
uses: actions/checkout@v3
with:
fetch-depth: 0 # Required because of release/get_version.py
- name: "Build package"
run: |
./release/package/run.sh package ubuntu-22.04-arm
- name: "Upload package"
uses: actions/upload-artifact@v3
with:
name: ubuntu-22.04-aarch64
path: build/output/ubuntu-22.04-arm/memgraph*.deb
upload-to-s3:
# only run upload if we specified version. Allows for runs without upload
if: "${{ github.event.inputs.memgraph_version != '' }}"
needs: [centos-7, centos-9, debian-10, debian-11, docker, ubuntu-1804, ubuntu-2004, ubuntu-2204, debian-11-platform, fedora-36, amzn-2, debian-11-arm, ubuntu-2204-arm]
runs-on: ubuntu-latest
steps:
- name: Download artifacts
uses: actions/download-artifact@v3
with:
# name: # if name input parameter is not provided, all artifacts are downloaded
# and put in directories named after each one.
path: build/output/release
- name: Upload to S3
uses: jakejarvis/s3-sync-action@v0.5.1
env:
AWS_S3_BUCKET: "download.memgraph.com"
AWS_ACCESS_KEY_ID: ${{ secrets.S3_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.S3_AWS_SECRET_ACCESS_KEY }}
AWS_REGION: "eu-west-1"
SOURCE_DIR: "build/output/release"
DEST_DIR: "memgraph/${{ github.event.inputs.memgraph_version }}/"

View File

@@ -3,7 +3,7 @@ name: Release CentOS 8
on:
workflow_dispatch:
schedule:
- cron: "0 1 * * *"
- cron: "0 22 * * *"
jobs:
community_build:

View File

@@ -3,7 +3,7 @@ name: Release Debian 10
on:
workflow_dispatch:
schedule:
- cron: "0 1 * * *"
- cron: "0 22 * * *"
jobs:
community_build:

View File

@@ -31,8 +31,8 @@ jobs:
- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Download memgraph binary
run: |

View File

@@ -0,0 +1,63 @@
name: "Mgbench Bolt Client Publish Docker Image"
on:
workflow_dispatch:
inputs:
version:
description: "Mgbench bolt client version to publish on Dockerhub."
required: true
force_release:
type: boolean
required: false
default: false
jobs:
mgbench_docker_publish:
runs-on: ubuntu-latest
env:
DOCKER_ORGANIZATION_NAME: memgraph
DOCKER_REPOSITORY_NAME: mgbench-client
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Check if specified version is already pushed
run: |
EXISTS=$(docker manifest inspect $DOCKER_ORGANIZATION_NAME/$DOCKER_REPOSITORY_NAME:${{ github.event.inputs.version }} > /dev/null; echo $?)
echo $EXISTS
if [[ ${EXISTS} -eq 0 ]]; then
echo 'The specified version has been already released to DockerHub.'
if [[ ${{ github.event.inputs.force_release }} = true ]]; then
echo 'Forcing the release!'
else
echo 'Stopping the release!'
exit 1
fi
else
echo 'All good the specified version has not been release to DockerHub.'
fi
- name: Build & push docker images
run: |
cd tests/mgbench
docker buildx build \
--build-arg TOOLCHAIN_VERSION=toolchain-v4 \
--platform linux/amd64,linux/arm64 \
--tag $DOCKER_ORGANIZATION_NAME/$DOCKER_REPOSITORY_NAME:${{ github.event.inputs.version }} \
--tag $DOCKER_ORGANIZATION_NAME/$DOCKER_REPOSITORY_NAME:latest \
--file Dockerfile.mgbench_client \
--push .

View File

@@ -3,7 +3,7 @@ name: Release Ubuntu 20.04
on:
workflow_dispatch:
schedule:
- cron: "0 1 * * *"
- cron: "0 22 * * *"
jobs:
community_build:

7
.gitignore vendored
View File

@@ -34,9 +34,6 @@ TAGS
*.fas
*.fasl
# LCP generated C++ files
*.lcp.cpp
src/database/distributed/serialization.hpp
src/database/single_node_ha/serialization.hpp
src/distributed/bfs_rpc_messages.hpp
@@ -50,15 +47,11 @@ src/distributed/pull_produce_rpc_messages.hpp
src/distributed/storage_gc_rpc_messages.hpp
src/distributed/token_sharing_rpc_messages.hpp
src/distributed/updates_rpc_messages.hpp
src/query/frontend/ast/ast.hpp
src/query/distributed/frontend/ast/ast_serialization.hpp
src/durability/distributed/state_delta.hpp
src/durability/single_node/state_delta.hpp
src/durability/single_node_ha/state_delta.hpp
src/query/frontend/semantic/symbol.hpp
src/query/distributed/frontend/semantic/symbol_serialization.hpp
src/query/distributed/plan/ops.hpp
src/query/plan/operator.hpp
src/raft/log_entry.hpp
src/raft/raft_rpc_messages.hpp
src/raft/snapshot_metadata.hpp

View File

@@ -1,16 +1,16 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
rev: v4.4.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/psf/black
rev: 22.10.0
rev: 23.1.0
hooks:
- id: black
- repo: https://github.com/pycqa/isort
rev: 5.10.1
rev: 5.12.0
hooks:
- id: isort
name: isort (python)

View File

@@ -231,7 +231,15 @@ endif()
message(STATUS "CMake build type: ${CMAKE_BUILD_TYPE}")
# -----------------------------------------------------------------------------
set(MG_ARCH "x86_64" CACHE STRING "Host architecture to build Memgraph on. Supported values are x86_64 (default), ARM64.")
if (NOT MG_ARCH)
set(MG_ARCH_DESCR "Host architecture to build Memgraph on. Supported values are x86_64, ARM64.")
if (${CMAKE_HOST_SYSTEM_PROCESSOR} MATCHES "aarch64")
set(MG_ARCH "ARM64" CACHE STRING ${MG_ARCH_DESCR})
else()
set(MG_ARCH "x86_64" CACHE STRING ${MG_ARCH_DESCR})
endif()
endif()
message(STATUS "MG_ARCH: ${MG_ARCH}")
# setup external dependencies -------------------------------------------------

View File

@@ -4,10 +4,6 @@
---
<p align="center">
Build modern, graph-based applications on top of your streaming data in minutes.
</p>
<p align="center">
<a href="https://github.com/memgraph/memgraph/blob/master/licenses/APL.txt">
<img src="https://img.shields.io/badge/license-APL-green" alt="license" title="license"/>
@@ -22,7 +18,7 @@ Build modern, graph-based applications on top of your streaming data in minutes.
<p align="center">
<a href="https://github.com/memgraph/memgraph">
<img src="https://img.shields.io/github/workflow/status/memgraph/memgraph/Release%20Ubuntu%2020.04/master" alt="build" title="build"/>
<img src="https://img.shields.io/github/actions/workflow/status/memgraph/memgraph/release_debian10.yaml?branch=master&label=build%20and%20test&logo=github"/>
</a>
<a href="https://memgraph.com/docs/" alt="Documentation">
<img src="https://img.shields.io/badge/documentation-Memgraph-orange" />
@@ -56,6 +52,17 @@ to ensure that youre getting the [best possible
performance](http://memgraph.com/benchgraph) consistently and without surprises.
Its also ACID-compliant and highly available.
## :zap: Features
- Run Python, Rust, and C/C++ code natively, check out the
[MAGE](https://github.com/memgraph/mage) graph algorithm library
- Native support for machine learning
- Streaming support
- Replication
- Authentication and authorization
- ACID compliance
## :video_game: Memgraph Playground
You don't need to install anything to try out Memgraph. Check out
@@ -78,28 +85,49 @@ your browser.
### macOS
[![macOS](https://img.shields.io/badge/macOS-Docker-000000?style=for-the-badge&logo=macos&logoColor=F0F0F0)](https://memgraph.com/docs/memgraph/install-memgraph-on-macos-docker)
[![macOS](https://img.shields.io/badge/lima-AACF41?style=for-the-badge&logo=macos&logoColor=F0F0F0)](https://memgraph.com/docs/memgraph/install-memgraph-on-ubuntu)
### Linux
[![Linux](https://img.shields.io/badge/Linux-Docker-FCC624?style=for-the-badge&logo=linux&logoColor=black)](https://memgraph.com/docs/memgraph/install-memgraph-on-linux-docker)
[![Debian](https://img.shields.io/badge/Debian-D70A53?style=for-the-badge&logo=debian&logoColor=white)](https://memgraph.com/docs/memgraph/install-memgraph-on-debian)
[![Ubuntu](https://img.shields.io/badge/Ubuntu-E95420?style=for-the-badge&logo=ubuntu&logoColor=white)](https://memgraph.com/docs/memgraph/install-memgraph-on-ubuntu)
[![Cent
OS](https://img.shields.io/badge/cent%20os-002260?style=for-the-badge&logo=centos&logoColor=F0F0F0)](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
[![Cent OS](https://img.shields.io/badge/cent%20os-002260?style=for-the-badge&logo=centos&logoColor=F0F0F0)](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
[![Fedora](https://img.shields.io/badge/fedora-0B57A4?style=for-the-badge&logo=fedora&logoColor=F0F0F0)](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
[![RedHat](https://img.shields.io/badge/redhat-EE0000?style=for-the-badge&logo=redhat&logoColor=F0F0F0)](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
You can find the binaries and Docker images on the [Download
Hub](https://memgraph.com/download) and the installation instructions in the
[official documentation](https://memgraph.com/docs/memgraph/installation).
## :zap: Features
- Run Python, Rust, and C/C++ code natively, check out the
[MAGE](https://github.com/memgraph/mage) graph algorithm library
- Native support for machine learning
- Streaming support
- Replication
- Authentication and authorization
- ACID compliance
## :cloud: Memgraph Cloud
Check out [Memgraph Cloud](https://memgraph.com/docs/memgraph-cloud) - a cloud service fully managed on AWS and available in 6 geographic regions around the world. Memgraph Cloud allows you to create projects with Enterprise instances of MemgraphDB from your browser.
<p align="left">
<a href="https://memgraph.com/docs/memgraph-cloud">
<img width="450px" alt="Memgraph Cloud" src="https://public-assets.memgraph.com/memgraph-gifs%2Fcloud.gif">
</a>
</p>
## :link: Connect to Memgraph
[Connect to the database](https://memgraph.com/docs/memgraph/connect-to-memgraph) using Memgraph Lab, mgconsole, various drivers (Python, C/C++ and others) and WebSocket.
### :microscope: Memgraph Lab
Visualize graphs and play with queries to understand your data. [Memgraph Lab](https://memgraph.com/docs/memgraph-lab) is a user interface that helps you explore and manipulate the data stored in Memgraph. Visualize graphs, execute ad hoc queries, and optimize their performance.
<p align="left">
<a href="https://memgraph.com/docs/memgraph-lab">
<img width="450px" alt="Memgraph Cloud" src="https://public-assets.memgraph.com/memgraph-gifs%2Flab.gif">
</a>
</p>
## :file_folder: Import data
[Import data](https://memgraph.com/docs/memgraph/import-data) into Memgraph using Kafka, RedPanda or Pulsar streams, CSV and JSON files, or Cypher commands.
## :bookmark_tabs: Documentation
@@ -143,17 +171,17 @@ Memgraph Community is available under the [BSL
license](./licenses/BSL.txt).</br> Memgraph Enterprise is available under the
[MEL license](./licenses/MEL.txt).
## 🙋 Community
## :busts_in_silhouette: Community
- :purple_heart: [**Discord**](https://discord.gg/memgraph)
- :ocean: [**Stack Overflow**](https://stackoverflow.com/questions/tagged/memgraphdb)
- :busts_in_silhouette: [**Discourse forum**](https://discourse.memgraph.com/)
- :open_file_folder: [**Memgraph GitHub**](https://github.com/memgraph)
- :bird: [**Twitter**](https://twitter.com/memgraphdb)
- :movie_camera:
[**YouTube**](https://www.youtube.com/channel/UCZ3HOJvHGxtQ_JHxOselBYg)
<p align="center">
<a href="#">
<img src="https://img.shields.io/badge/⬆back_to_top_⬆-white" alt="Back to top" title="Back to top"/>
<img src="https://img.shields.io/badge/⬆️ back_to_top_⬆-white" alt="Back to top" title="Back to top"/>
</a>
</p>

View File

@@ -103,6 +103,10 @@ modifications:
value: "true"
override: false
- name: "storage_parallel_index_recovery"
value: "false"
override: true
undocumented:
- "flag_file"
- "also_log_to_stderr"

156
environment/os/amzn-2.sh Executable file
View File

@@ -0,0 +1,156 @@
#!/bin/bash
set -Eeuo pipefail
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source "$DIR/../util.sh"
check_operating_system "amzn-2"
check_architecture "x86_64"
TOOLCHAIN_BUILD_DEPS=(
gcc gcc-c++ make # generic build tools
wget # used for archive download
gnupg2 # used for archive signature verification
tar gzip bzip2 xz unzip # used for archive unpacking
zlib-devel # zlib library used for all builds
expat-devel xz-devel python3-devel texinfo
curl libcurl-devel # for cmake
readline-devel # for cmake and llvm
libffi-devel libxml2-devel # for llvm
libedit-devel pcre-devel automake bison # for swig
file
openssl-devel
gmp-devel
gperf
diffutils
patch
libipt libipt-devel # intel
perl # for openssl
)
TOOLCHAIN_RUN_DEPS=(
make # generic build tools
tar gzip bzip2 xz # used for archive unpacking
zlib # zlib library used for all builds
expat xz-libs python3 # for gdb
readline # for cmake and llvm
libffi libxml2 # for llvm
openssl-devel
)
MEMGRAPH_BUILD_DEPS=(
git # source code control
make # build system
wget # for downloading libs
libuuid-devel java-11-openjdk # required by antlr
readline-devel # for memgraph console
python3-devel # for query modules
openssl-devel
libseccomp-devel
python3 python3-pip nmap-ncat # for tests
#
# IMPORTANT: python3-yaml does NOT exist on CentOS
# Install it using `pip3 install PyYAML`
#
PyYAML # Package name here does not correspond to the yum package!
libcurl-devel # mg-requests
rpm-build rpmlint # for RPM package building
doxygen graphviz # source documentation generators
which nodejs golang zip unzip java-11-openjdk-devel # for driver tests
autoconf # for jemalloc code generation
libtool # for protobuf code generation
)
list() {
echo "$1"
}
check() {
local missing=""
# On Fedora yum/dnf and python10 use newer glibc which is not compatible
# with ours, so we need to momentarely disable env
local OLD_LD_LIBRARY_PATH=${LD_LIBRARY_PATH:-""}
LD_LIBRARY_PATH=""
for pkg in $1; do
if [ "$pkg" == "PyYAML" ]; then
if ! python3 -c "import yaml" >/dev/null 2>/dev/null; then
missing="$pkg $missing"
fi
continue
fi
if ! yum list installed "$pkg" >/dev/null 2>/dev/null; then
missing="$pkg $missing"
fi
done
if [ "$missing" != "" ]; then
echo "MISSING PACKAGES: $missing"
exit 1
fi
LD_LIBRARY_PATH=${OLD_LD_LIBRARY_PATH}
}
install() {
cd "$DIR"
if [ "$EUID" -ne 0 ]; then
echo "Please run as root."
exit 1
fi
# If GitHub Actions runner is installed, append LANG to the environment.
# Python related tests don't work without the LANG export.
if [ -d "/home/gh/actions-runner" ]; then
echo "LANG=en_US.utf8" >> /home/gh/actions-runner/.env
else
echo "NOTE: export LANG=en_US.utf8"
fi
yum update -y
for pkg in $1; do
if [ "$pkg" == libipt ]; then
if ! yum list installed libipt >/dev/null 2>/dev/null; then
yum install -y http://repo.okay.com.mx/centos/8/x86_64/release/libipt-1.6.1-8.el8.x86_64.rpm
fi
continue
fi
if [ "$pkg" == libipt-devel ]; then
if ! yum list installed libipt-devel >/dev/null 2>/dev/null; then
yum install -y http://repo.okay.com.mx/centos/8/x86_64/release/libipt-devel-1.6.1-8.el8.x86_64.rpm
fi
continue
fi
if [ "$pkg" == nodejs ]; then
curl -sL https://rpm.nodesource.com/setup_16.x | bash -
if ! yum list installed nodejs >/dev/null 2>/dev/null; then
yum install -y nodejs
fi
continue
fi
if [ "$pkg" == PyYAML ]; then
if [ -z ${SUDO_USER+x} ]; then # Running as root (e.g. Docker).
pip3 install --user PyYAML
else # Running using sudo.
sudo -H -u "$SUDO_USER" bash -c "pip3 install --user PyYAML"
fi
continue
fi
if [ "$pkg" == nodejs ]; then
curl -sL https://rpm.nodesource.com/setup_16.x | bash -
if ! yum list installed nodejs >/dev/null 2>/dev/null; then
yum install -y nodejs
fi
continue
fi
if [ "$pkg" == java-11-openjdk ]; then
amazon-linux-extras install -y java-openjdk11
continue
fi
if [ "$pkg" == java-11-openjdk-devel ]; then
amazon-linux-extras install -y java-openjdk11
yum install -y java-11-openjdk-devel
continue
fi
yum install -y "$pkg"
done
}
deps=$2"[*]"
"$1" "${!deps}"

View File

@@ -26,6 +26,7 @@ TOOLCHAIN_BUILD_DEPS=(
diffutils
libipt libipt-devel # intel
patch
perl # for openssl
)
TOOLCHAIN_RUN_DEPS=(
@@ -36,7 +37,6 @@ TOOLCHAIN_RUN_DEPS=(
readline # for cmake and llvm
libffi libxml2 # for llvm
openssl-devel
perl # for openssl
)
MEMGRAPH_BUILD_DEPS=(
@@ -64,6 +64,10 @@ list() {
check() {
local missing=""
# On Fedora yum/dnf and python10 use newer glibc which is not compatible
# with ours, so we need to momentarely disable env
local OLD_LD_LIBRARY_PATH=${LD_LIBRARY_PATH}
LD_LIBRARY_PATH=""
for pkg in $1; do
if ! dnf list installed "$pkg" >/dev/null 2>/dev/null; then
missing="$pkg $missing"
@@ -73,6 +77,7 @@ check() {
echo "MISSING PACKAGES: $missing"
exit 1
fi
LD_LIBRARY_PATH=${OLD_LD_LIBRARY_PATH}
}
install() {

View File

@@ -0,0 +1,96 @@
#!/bin/bash
set -Eeuo pipefail
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source "$DIR/../util.sh"
check_operating_system "ubuntu-22.04"
check_architecture "arm64" "aarch64"
TOOLCHAIN_BUILD_DEPS=(
coreutils gcc g++ build-essential make # generic build tools
wget # used for archive download
gnupg # used for archive signature verification
tar gzip bzip2 xz-utils unzip # used for archive unpacking
zlib1g-dev # zlib library used for all builds
libexpat1-dev libbabeltrace-dev liblzma-dev python3-dev texinfo # for gdb
libcurl4-openssl-dev # for cmake
libreadline-dev # for cmake and llvm
libffi-dev libxml2-dev # for llvm
curl # snappy
file
git # for thrift
libgmp-dev # for gdb
gperf # for proxygen
libssl-dev
libedit-dev libpcre3-dev automake bison # for swig
)
TOOLCHAIN_RUN_DEPS=(
make # generic build tools
tar gzip bzip2 xz-utils # used for archive unpacking
zlib1g # zlib library used for all builds
libexpat1 libbabeltrace1 liblzma5 python3 # for gdb
libcurl4 # for cmake
libreadline8 # for cmake and llvm
libffi7 libxml2 # for llvm
libssl-dev # for libevent
)
MEMGRAPH_BUILD_DEPS=(
git # source code control
make pkg-config # build system
curl wget # for downloading libs
uuid-dev default-jre-headless # required by antlr
libreadline-dev # for memgraph console
libpython3-dev python3-dev # for query modules
libssl-dev
libseccomp-dev
netcat # tests are using nc to wait for memgraph
python3 python3-virtualenv python3-pip # for qa, macro_benchmark and stress tests
python3-yaml # for the configuration generator
libcurl4-openssl-dev # mg-requests
sbcl # for custom Lisp C++ preprocessing
doxygen graphviz # source documentation generators
mono-runtime mono-mcs zip unzip default-jdk-headless # for driver tests
dotnet-sdk-6.0 golang nodejs npm
autoconf # for jemalloc code generation
libtool # for protobuf code generation
)
list() {
echo "$1"
}
check() {
check_all_dpkg "$1"
}
install() {
cd "$DIR"
apt update
# If GitHub Actions runner is installed, append LANG to the environment.
# Python related tests doesn't work the LANG export.
if [ -d "/home/gh/actions-runner" ]; then
echo "LANG=en_US.utf8" >> /home/gh/actions-runner/.env
else
echo "NOTE: export LANG=en_US.utf8"
fi
apt install -y wget
for pkg in $1; do
if [ "$pkg" == dotnet-sdk-6.0 ]; then
if ! dpkg -s dotnet-sdk-6.0 2>/dev/null >/dev/null; then
wget -nv https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
dpkg -i packages-microsoft-prod.deb
apt-get update
apt-get install -y apt-transport-https dotnet-sdk-6.0
fi
continue
fi
apt install -y "$pkg"
done
}
deps=$2"[*]"
"$1" "${!deps}"

View File

@@ -51,11 +51,19 @@ CPPCHECK_VERSION=2.6
LLVM_VERSION=13.0.0
SWIG_VERSION=4.0.2 # used only for LLVM compilation
# Check for the dependencies.
echo "ALL BUILD PACKAGES: $($DIR/../os/$DISTRO.sh list TOOLCHAIN_BUILD_DEPS)"
$DIR/../os/$DISTRO.sh check TOOLCHAIN_BUILD_DEPS
echo "ALL RUN PACKAGES: $($DIR/../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)"
$DIR/../os/$DISTRO.sh check TOOLCHAIN_RUN_DEPS
# Set the right env script
ENV_SCRIPT="$DIR/../os/$DISTRO.sh"
if [[ "$for_arm" = true ]]; then
ENV_SCRIPT="$DIR/../os/$DISTRO-arm.sh"
fi
# Check for the toolchain build dependencies.
echo "ALL BUILD PACKAGES: $(${ENV_SCRIPT} list TOOLCHAIN_BUILD_DEPS)"
${ENV_SCRIPT} check TOOLCHAIN_BUILD_DEPS
# Check for the toolchain run dependencies.
echo "ALL RUN PACKAGES: $(${ENV_SCRIPT} list TOOLCHAIN_RUN_DEPS)"
${ENV_SCRIPT} check TOOLCHAIN_RUN_DEPS
# check installation directory
NAME=toolchain-v$TOOLCHAIN_VERSION
@@ -379,6 +387,62 @@ if [ ! -f $PREFIX/bin/gdb ]; then
--without-babeltrace \
--enable-tui \
--with-python=python3
elif [[ "${DISTRO}" == fedora* ]]; then
# Remove readline, gdb does not compile
env \
CC=gcc \
CXX=g++ \
CFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" \
CXXFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" \
CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2 -fPIC" \
LDFLAGS="-Wl,-z,relro" \
PYTHON="" \
../configure \
--build=x86_64-linux-gnu \
--host=x86_64-linux-gnu \
--prefix=$PREFIX \
--disable-maintainer-mode \
--disable-dependency-tracking \
--disable-silent-rules \
--disable-gdbtk \
--disable-shared \
--without-guile \
--with-system-gdbinit=$PREFIX/etc/gdb/gdbinit \
--with-expat \
--with-system-zlib \
--with-lzma \
--with-babeltrace \
--with-intel-pt \
--enable-tui \
--with-python=python3
elif [[ "${DISTRO}" == "amzn-2" ]]; then
# Remove readline, gdb does not compile
env \
CC=gcc \
CXX=g++ \
CFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" \
CXXFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" \
CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2 -fPIC" \
LDFLAGS="-Wl,-z,relro" \
PYTHON="" \
../configure \
--build=x86_64-linux-gnu \
--host=x86_64-linux-gnu \
--prefix=$PREFIX \
--disable-maintainer-mode \
--disable-dependency-tracking \
--disable-silent-rules \
--disable-gdbtk \
--disable-shared \
--without-guile \
--with-system-gdbinit=$PREFIX/etc/gdb/gdbinit \
--with-expat \
--with-system-zlib \
--with-lzma \
--with-babeltrace \
--with-intel-pt \
--enable-tui \
--with-python=python3
else
# https://buildd.debian.org/status/fetch.php?pkg=gdb&arch=amd64&ver=8.2.1-2&stamp=1550831554&raw=0
env \
@@ -594,7 +658,7 @@ In order to be able to run all of these tools you should install the following
packages:
\`\`\`
$($DIR/../os/$DISTRO.sh list TOOLCHAIN_RUN_DEPS)
$($DIR/../os/$ENV_SCRIPT.sh list TOOLCHAIN_RUN_DEPS)
\`\`\`
## Usage
@@ -651,6 +715,7 @@ export PS1="($NAME) \$PS1"
export LD_LIBRARY_PATH=$PREFIX/lib:$PREFIX/lib64
export CXXFLAGS=-isystem\ $PREFIX/include\ \$CXXFLAGS
export CFLAGS=-isystem\ $PREFIX/include\ \$CFLAGS
export VENV=$PREFIX
# disable root
function su () {
@@ -702,7 +767,7 @@ PROXYGEN_SHA256=5360a8ccdfb2f5a6c7b3eed331ec7ab0e2c792d579c6fff499c85c516c11fe14
SNAPPY_SHA256=75c1fbb3d618dd3a0483bff0e26d0a92b495bbe5059c8b4f1c962b478b6e06e7
SNAPPY_VERSION=1.1.9
XZ_VERSION=5.2.5 # for LZMA
ZLIB_VERSION=1.2.12
ZLIB_VERSION=1.2.13
ZSTD_VERSION=1.5.0
WANGLE_SHA256=1002e9c32b6f4837f6a760016e3b3e22f3509880ef3eaad191c80dc92655f23f
@@ -1106,119 +1171,121 @@ if [ ! -f $PREFIX/include/libaio.h ]; then
popd
fi
log_tool_name "folly $FBLIBS_VERSION"
if [ ! -d $PREFIX/include/folly ]; then
if [ -d folly-$FBLIBS_VERSION ]; then
rm -rf folly-$FBLIBS_VERSION
if [[ "${DISTRO}" != "amzn-2" ]]; then
log_tool_name "folly $FBLIBS_VERSION"
if [ ! -d $PREFIX/include/folly ]; then
if [ -d folly-$FBLIBS_VERSION ]; then
rm -rf folly-$FBLIBS_VERSION
fi
mkdir folly-$FBLIBS_VERSION
tar -xzf ../archives/folly-$FBLIBS_VERSION.tar.gz -C folly-$FBLIBS_VERSION
pushd folly-$FBLIBS_VERSION
patch -p1 < ../../folly.patch
# build is used by facebook builder
mkdir _build
pushd _build
cmake .. $COMMON_CMAKE_FLAGS \
-DBOOST_LINK_STATIC=ON \
-DBUILD_TESTS=OFF \
-DGFLAGS_NOTHREADS=OFF \
-DCXX_STD="c++20"
make -j$CPUS install
popd && popd
fi
mkdir folly-$FBLIBS_VERSION
tar -xzf ../archives/folly-$FBLIBS_VERSION.tar.gz -C folly-$FBLIBS_VERSION
pushd folly-$FBLIBS_VERSION
patch -p1 < ../../folly.patch
# build is used by facebook builder
mkdir _build
pushd _build
cmake .. $COMMON_CMAKE_FLAGS \
-DBOOST_LINK_STATIC=ON \
-DBUILD_TESTS=OFF \
-DGFLAGS_NOTHREADS=OFF \
-DCXX_STD="c++20"
make -j$CPUS install
popd && popd
fi
log_tool_name "fizz $FBLIBS_VERSION"
if [ ! -d $PREFIX/include/fizz ]; then
if [ -d fizz-$FBLIBS_VERSION ]; then
rm -rf fizz-$FBLIBS_VERSION
log_tool_name "fizz $FBLIBS_VERSION"
if [ ! -d $PREFIX/include/fizz ]; then
if [ -d fizz-$FBLIBS_VERSION ]; then
rm -rf fizz-$FBLIBS_VERSION
fi
mkdir fizz-$FBLIBS_VERSION
tar -xzf ../archives/fizz-$FBLIBS_VERSION.tar.gz -C fizz-$FBLIBS_VERSION
pushd fizz-$FBLIBS_VERSION
# build is used by facebook builder
mkdir _build
pushd _build
cmake ../fizz $COMMON_CMAKE_FLAGS \
-DBUILD_TESTS=OFF \
-DBUILD_EXAMPLES=OFF \
-DGFLAGS_NOTHREADS=OFF
make -j$CPUS install
popd && popd
fi
mkdir fizz-$FBLIBS_VERSION
tar -xzf ../archives/fizz-$FBLIBS_VERSION.tar.gz -C fizz-$FBLIBS_VERSION
pushd fizz-$FBLIBS_VERSION
# build is used by facebook builder
mkdir _build
pushd _build
cmake ../fizz $COMMON_CMAKE_FLAGS \
-DBUILD_TESTS=OFF \
-DBUILD_EXAMPLES=OFF \
-DGFLAGS_NOTHREADS=OFF
make -j$CPUS install
popd && popd
fi
log_tool_name "wangle FBLIBS_VERSION"
if [ ! -d $PREFIX/include/wangle ]; then
if [ -d wangle-$FBLIBS_VERSION ]; then
rm -rf wangle-$FBLIBS_VERSION
log_tool_name "wangle FBLIBS_VERSION"
if [ ! -d $PREFIX/include/wangle ]; then
if [ -d wangle-$FBLIBS_VERSION ]; then
rm -rf wangle-$FBLIBS_VERSION
fi
mkdir wangle-$FBLIBS_VERSION
tar -xzf ../archives/wangle-$FBLIBS_VERSION.tar.gz -C wangle-$FBLIBS_VERSION
pushd wangle-$FBLIBS_VERSION
# build is used by facebook builder
mkdir _build
pushd _build
cmake ../wangle $COMMON_CMAKE_FLAGS \
-DBUILD_TESTS=OFF \
-DBUILD_EXAMPLES=OFF \
-DGFLAGS_NOTHREADS=OFF
make -j$CPUS install
popd && popd
fi
mkdir wangle-$FBLIBS_VERSION
tar -xzf ../archives/wangle-$FBLIBS_VERSION.tar.gz -C wangle-$FBLIBS_VERSION
pushd wangle-$FBLIBS_VERSION
# build is used by facebook builder
mkdir _build
pushd _build
cmake ../wangle $COMMON_CMAKE_FLAGS \
-DBUILD_TESTS=OFF \
-DBUILD_EXAMPLES=OFF \
-DGFLAGS_NOTHREADS=OFF
make -j$CPUS install
popd && popd
fi
log_tool_name "proxygen $FBLIBS_VERSION"
if [ ! -d $PREFIX/include/proxygen ]; then
if [ -d proxygen-$FBLIBS_VERSION ]; then
rm -rf proxygen-$FBLIBS_VERSION
log_tool_name "proxygen $FBLIBS_VERSION"
if [ ! -d $PREFIX/include/proxygen ]; then
if [ -d proxygen-$FBLIBS_VERSION ]; then
rm -rf proxygen-$FBLIBS_VERSION
fi
mkdir proxygen-$FBLIBS_VERSION
tar -xzf ../archives/proxygen-$FBLIBS_VERSION.tar.gz -C proxygen-$FBLIBS_VERSION
pushd proxygen-$FBLIBS_VERSION
patch -p1 < ../../proxygen.patch
# build is used by facebook builder
mkdir _build
pushd _build
cmake .. $COMMON_CMAKE_FLAGS \
-DBUILD_TESTS=OFF \
-DBUILD_SAMPLES=OFF \
-DGFLAGS_NOTHREADS=OFF \
-DBUILD_QUIC=OFF
make -j$CPUS install
popd && popd
fi
mkdir proxygen-$FBLIBS_VERSION
tar -xzf ../archives/proxygen-$FBLIBS_VERSION.tar.gz -C proxygen-$FBLIBS_VERSION
pushd proxygen-$FBLIBS_VERSION
patch -p1 < ../../proxygen.patch
# build is used by facebook builder
mkdir _build
pushd _build
cmake .. $COMMON_CMAKE_FLAGS \
-DBUILD_TESTS=OFF \
-DBUILD_SAMPLES=OFF \
-DGFLAGS_NOTHREADS=OFF \
-DBUILD_QUIC=OFF
make -j$CPUS install
popd && popd
fi
log_tool_name "flex $FBLIBS_VERSION"
if [ ! -f $PREFIX/include/FlexLexer.h ]; then
if [ -d flex-$FLEX_VERSION ]; then
rm -rf flex-$FLEX_VERSION
log_tool_name "flex $FBLIBS_VERSION"
if [ ! -f $PREFIX/include/FlexLexer.h ]; then
if [ -d flex-$FLEX_VERSION ]; then
rm -rf flex-$FLEX_VERSION
fi
tar -xzf ../archives/flex-$FLEX_VERSION.tar.gz
pushd flex-$FLEX_VERSION
./configure $COMMON_CONFIGURE_FLAGS
make -j$CPUS install
popd
fi
tar -xzf ../archives/flex-$FLEX_VERSION.tar.gz
pushd flex-$FLEX_VERSION
./configure $COMMON_CONFIGURE_FLAGS
make -j$CPUS install
popd
fi
log_tool_name "fbthrift $FBLIBS_VERSION"
if [ ! -d $PREFIX/include/thrift ]; then
if [ -d fbthrift-$FBLIBS_VERSION ]; then
rm -rf fbthrift-$FBLIBS_VERSION
log_tool_name "fbthrift $FBLIBS_VERSION"
if [ ! -d $PREFIX/include/thrift ]; then
if [ -d fbthrift-$FBLIBS_VERSION ]; then
rm -rf fbthrift-$FBLIBS_VERSION
fi
git clone --depth 1 --branch v$FBLIBS_VERSION https://github.com/facebook/fbthrift.git fbthrift-$FBLIBS_VERSION
pushd fbthrift-$FBLIBS_VERSION
# build is used by facebook builder
mkdir _build
pushd _build
if [ "$TOOLCHAIN_STDCXX" = "libstdc++" ]; then
CMAKE_CXX_FLAGS="-fsized-deallocation"
else
CMAKE_CXX_FLAGS="-fsized-deallocation -stdlib=libc++"
fi
cmake .. $COMMON_CMAKE_FLAGS \
-Denable_tests=OFF \
-DGFLAGS_NOTHREADS=OFF \
-DCMAKE_CXX_FLAGS="$CMAKE_CXX_FLAGS"
make -j$CPUS install
popd
fi
git clone --depth 1 --branch v$FBLIBS_VERSION https://github.com/facebook/fbthrift.git fbthrift-$FBLIBS_VERSION
pushd fbthrift-$FBLIBS_VERSION
# build is used by facebook builder
mkdir _build
pushd _build
if [ "$TOOLCHAIN_STDCXX" = "libstdc++" ]; then
CMAKE_CXX_FLAGS="-fsized-deallocation"
else
CMAKE_CXX_FLAGS="-fsized-deallocation -stdlib=libc++"
fi
cmake .. $COMMON_CMAKE_FLAGS \
-Denable_tests=OFF \
-DGFLAGS_NOTHREADS=OFF \
-DCMAKE_CXX_FLAGS="$CMAKE_CXX_FLAGS"
make -j$CPUS install
popd
fi
popd
@@ -1226,7 +1293,7 @@ popd
# create toolchain archive
if [ ! -f $NAME-binaries-$DISTRO.tar.gz ]; then
DISTRO_FULL_NAME=${DISTRO}
if [[ "${DISTRO}" == centos* ]]; then
if [[ "${DISTRO}" == centos* ]] || [[ "${DISTRO}" == fedora* ]]; then
if [[ "$for_arm" = "true" ]]; then
DISTRO_FULL_NAME="$DISTRO_FULL_NAME-aarch64"
else

View File

@@ -19,13 +19,16 @@ function architecture() {
}
check_architecture() {
local ARCH=$(architecture)
for arch in "$@"; do
if [ "$(architecture)" = "$arch" ]; then
if [ "${ARCH}" = "$arch" ]; then
echo "The right architecture!"
return 0
fi
done
echo "Not the right architecture!"
echo "Expected: $@"
echo "Actual: ${ARCH}"
exit 1
}

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -378,6 +378,10 @@ inline mgp_value *vertex_get_property(mgp_vertex *v, const char *property_name,
return MgInvoke<mgp_value *>(mgp_vertex_get_property, v, property_name, memory);
}
inline void vertex_set_property(mgp_vertex *v, const char *property_name, mgp_value *property_value) {
MgInvokeVoid(mgp_vertex_set_property, v, property_name, property_value);
}
inline mgp_properties_iterator *vertex_iter_properties(mgp_vertex *v, mgp_memory *memory) {
return MgInvoke<mgp_properties_iterator *>(mgp_vertex_iter_properties, v, memory);
}
@@ -410,6 +414,10 @@ inline mgp_value *edge_get_property(mgp_edge *e, const char *property_name, mgp_
return MgInvoke<mgp_value *>(mgp_edge_get_property, e, property_name, memory);
}
inline void edge_set_property(mgp_edge *e, const char *property_name, mgp_value *property_value) {
MgInvokeVoid(mgp_edge_set_property, e, property_name, property_value);
}
inline mgp_properties_iterator *edge_iter_properties(mgp_edge *e, mgp_memory *memory) {
return MgInvoke<mgp_properties_iterator *>(mgp_edge_iter_properties, e, memory);
}

343
include/_mgp_mock.py Normal file
View File

@@ -0,0 +1,343 @@
import typing
from enum import Enum
import networkx as nx
NX_LABEL_ATTR = "labels"
NX_TYPE_ATTR = "type"
SOURCE_TYPE_KAFKA = "SOURCE_TYPE_KAFKA"
SOURCE_TYPE_PULSAR = "SOURCE_TYPE_PULSAR"
"""
This module provides helpers for the mock Python API, much like _mgp.py does for mgp.py.
"""
class InvalidArgumentError(Exception):
"""
Signals that some of the arguments have invalid values.
"""
pass
class ImmutableObjectError(Exception):
pass
class LogicErrorError(Exception):
pass
class DeletedObjectError(Exception):
pass
class EdgeConstants(Enum):
I_START = 0
I_END = 1
I_KEY = 2
class Graph:
"""Wrapper around a NetworkX MultiDiGraph instance."""
__slots__ = ("nx", "_highest_vertex_id", "_highest_edge_id", "_valid")
def __init__(self, graph: nx.MultiDiGraph) -> None:
if not isinstance(graph, nx.MultiDiGraph):
raise TypeError(f"Expected 'networkx.classes.multidigraph.MultiDiGraph', got '{type(graph)}'")
self.nx = graph
self._highest_vertex_id = None
self._highest_edge_id = None
self._valid = True
@property
def vertex_ids(self):
return self.nx.nodes
def vertex_is_isolate(self, vertex_id: int) -> bool:
return nx.is_isolate(self.nx, vertex_id)
@property
def vertices(self):
return (Vertex(node_id, self) for node_id in self.nx.nodes)
def has_node(self, node_id):
return self.nx.has_node(node_id)
@property
def edges(self):
return self.nx.edges
def is_valid(self) -> bool:
return self._valid
def get_vertex_by_id(self, vertex_id: int) -> "Vertex":
return Vertex(vertex_id, self)
def invalidate(self):
self._valid = False
def is_immutable(self) -> bool:
return nx.is_frozen(self.nx)
def make_immutable(self):
self.nx = nx.freeze(self.nx)
def _new_vertex_id(self):
if self._highest_vertex_id is None:
self._highest_vertex_id = max(vertex_id for vertex_id in self.nx.nodes)
return self._highest_vertex_id + 1
def _new_edge_id(self):
if self._highest_edge_id is None:
self._highest_edge_id = max(edge[EdgeConstants.I_KEY.value] for edge in self.nx.edges(keys=True))
return self._highest_edge_id + 1
def create_vertex(self) -> "Vertex":
vertex_id = self._new_vertex_id()
self.nx.add_node(vertex_id)
self._highest_vertex_id = vertex_id
return Vertex(vertex_id, self)
def create_edge(self, from_vertex: "Vertex", to_vertex: "Vertex", edge_type: str) -> "Edge":
if from_vertex.is_deleted() or to_vertex.is_deleted():
raise DeletedObjectError("Accessing deleted object.")
edge_id = self._new_edge_id()
from_id = from_vertex.id
to_id = to_vertex.id
self.nx.add_edge(from_id, to_id, key=edge_id, type=edge_type)
self._highest_edge_id = edge_id
return Edge((from_id, to_id, edge_id), self)
def delete_vertex(self, vertex_id: int):
self.nx.remove_node(vertex_id)
def delete_edge(self, from_vertex_id: int, to_vertex_id: int, edge_id: int):
self.nx.remove_edge(from_vertex_id, to_vertex_id, edge_id)
@property
def highest_vertex_id(self) -> int:
if self._highest_vertex_id is None:
self._highest_vertex_id = max(vertex_id for vertex_id in self.nx.nodes) + 1
return self._highest_vertex_id
@property
def highest_edge_id(self) -> int:
if self._highest_edge_id is None:
self._highest_edge_id = max(edge[EdgeConstants.I_KEY.value] for edge in self.nx.edges(keys=True))
return self._highest_edge_id + 1
class Vertex:
"""Represents a graph vertex."""
__slots__ = ("_id", "_graph")
def __init__(self, id: int, graph: Graph) -> None:
if not isinstance(id, int):
raise TypeError(f"Expected 'int', got '{type(id)}'")
if not isinstance(graph, Graph):
raise TypeError(f"Expected '_mgp_mock.Graph', got '{type(graph)}'")
if not graph.nx.has_node(id):
raise IndexError(f"Unable to find vertex with ID {id}.")
self._id = id
self._graph = graph
def is_valid(self) -> bool:
return self._graph.is_valid()
def is_deleted(self) -> bool:
return not self._graph.nx.has_node(self._id) and self._id <= self._graph.highest_vertex_id
@property
def underlying_graph(self) -> Graph:
return self._graph
def underlying_graph_is_mutable(self) -> bool:
return not nx.is_frozen(self._graph.nx)
@property
def labels(self) -> typing.List[int]:
return self._graph.nx.nodes[self._id][NX_LABEL_ATTR].split(":")
def add_label(self, label: str) -> None:
if nx.is_frozen(self._graph.nx):
raise ImmutableObjectError("Cannot modify immutable object.")
self._graph.nx.nodes[self._id][NX_LABEL_ATTR] += f":{label}"
def remove_label(self, label: str) -> None:
if nx.is_frozen(self._graph.nx):
raise ImmutableObjectError("Cannot modify immutable object.")
labels = self._graph.nx.nodes[self._id][NX_LABEL_ATTR]
if labels.startswith(f"{label}:"):
labels = "\n" + labels # pseudo-string starter
self._graph.nx.nodes[self._id][NX_LABEL_ATTR] = labels.replace(f"\n{label}:", "")
elif labels.endswith(f":{label}"):
labels += "\n" # pseudo-string terminator
self._graph.nx.nodes[self._id][NX_LABEL_ATTR] = labels.replace(f":{label}\n", "")
else:
self._graph.nx.nodes[self._id][NX_LABEL_ATTR] = labels.replace(f":{label}:", ":")
@property
def id(self) -> int:
return self._id
@property
def properties(self):
return (
(key, value)
for key, value in self._graph.nx.nodes[self._id].items()
if key not in (NX_LABEL_ATTR, NX_TYPE_ATTR)
)
def get_property(self, property_name: str):
return self._graph.nx.nodes[self._id][property_name]
def set_property(self, property_name: str, value: object):
self._graph.nx.nodes[self._id][property_name] = value
@property
def in_edges(self) -> typing.Iterable["Edge"]:
return [Edge(edge, self._graph) for edge in self._graph.nx.in_edges(self._id, keys=True)]
@property
def out_edges(self) -> typing.Iterable["Edge"]:
return [Edge(edge, self._graph) for edge in self._graph.nx.out_edges(self._id, keys=True)]
class Edge:
"""Represents a graph edge."""
__slots__ = ("_edge", "_graph")
def __init__(self, edge: typing.Tuple[int, int, int], graph: Graph) -> None:
if not isinstance(edge, typing.Tuple):
raise TypeError(f"Expected 'Tuple', got '{type(edge)}'")
if not isinstance(graph, Graph):
raise TypeError(f"Expected '_mgp_mock.Graph', got '{type(graph)}'")
if not graph.nx.has_edge(*edge):
raise IndexError(f"Unable to find edge with ID {edge[EdgeConstants.I_KEY.value]}.")
self._edge = edge
self._graph = graph
def is_valid(self) -> bool:
return self._graph.is_valid()
def is_deleted(self) -> bool:
return (
not self._graph.nx.has_edge(*self._edge)
and self._edge[EdgeConstants.I_KEY.value] <= self._graph.highest_edge_id
)
def underlying_graph_is_mutable(self) -> bool:
return not nx.is_frozen(self._graph.nx)
@property
def id(self) -> int:
return self._edge[EdgeConstants.I_KEY.value]
@property
def edge(self) -> typing.Tuple[int, int, int]:
return self._edge
@property
def start_id(self) -> int:
return self._edge[EdgeConstants.I_START.value]
@property
def end_id(self) -> int:
return self._edge[EdgeConstants.I_END.value]
def get_type_name(self):
return self._graph.nx.get_edge_data(*self._edge)[NX_TYPE_ATTR]
def from_vertex(self) -> Vertex:
return Vertex(self.start_id, self._graph)
def to_vertex(self) -> Vertex:
return Vertex(self.end_id, self._graph)
@property
def properties(self):
return (
(key, value)
for key, value in self._graph.nx.edges[self._edge].items()
if key not in (NX_LABEL_ATTR, NX_TYPE_ATTR)
)
def get_property(self, property_name: str):
return self._graph.nx.edges[self._edge][property_name]
def set_property(self, property_name: str, value: object):
self._graph.nx.edges[self._edge][property_name] = value
class Path:
"""Represents a path comprised of `Vertex` and `Edge` instances."""
__slots__ = ("_vertices", "_edges", "_graph")
__create_key = object()
def __init__(self, create_key, vertex_id: int, graph: Graph) -> None:
assert create_key == Path.__create_key, "Path objects must be created using Path.make_with_start"
self._vertices = [vertex_id]
self._edges = []
self._graph = graph
@classmethod
def make_with_start(cls, vertex: Vertex) -> "Path":
if not isinstance(vertex, Vertex):
raise TypeError(f"Expected 'Vertex', got '{type(vertex)}'")
if not isinstance(vertex.underlying_graph, Graph):
raise TypeError(f"Expected '_mgp_mock.Graph', got '{type(vertex.underlying_graph)}'")
if not vertex.underlying_graph.nx.has_node(vertex._id):
raise IndexError(f"Unable to find vertex with ID {vertex._id}.")
return Path(cls.__create_key, vertex._id, vertex.underlying_graph)
def is_valid(self) -> bool:
return self._graph.is_valid()
def underlying_graph_is_mutable(self) -> bool:
return not nx.is_frozen(self._graph.nx)
def expand(self, edge: Edge):
if edge.start_id != self._vertices[-1]:
raise LogicErrorError("Logic error.")
self._vertices.append(edge.end_id)
self._edges.append((edge.start_id, edge.end_id, edge.id))
def vertex_at(self, index: int) -> Vertex:
return Vertex(self._vertices[index], self._graph)
def edge_at(self, index: int) -> Edge:
return Edge(self._edges[index], self._graph)
def size(self) -> int:
return len(self._edges)

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -55,6 +55,15 @@ class NotEnoughMemoryException : public std::exception {
const char *what() const throw() { return "Not enough memory!"; }
};
class MustAbortException : public std::exception {
public:
explicit MustAbortException(const std::string &message) : message_(message) {}
const char *what() const noexcept override { return message_.c_str(); }
private:
std::string message_;
};
// Forward declarations
class Nodes;
using GraphNodes = Nodes;
@@ -66,9 +75,12 @@ struct MapItem;
class Duration;
class Value;
struct StealType {};
inline constexpr StealType steal{};
inline mgp_memory *memory{nullptr};
/* #region Graph (Id, Graph, Nodes, GraphRelationships, Relationships, Properties & Labels) */
/* #region Graph (Id, Graph, Nodes, GraphRelationships, Relationships & Labels) */
/// Wrapper for int64_t IDs to prevent dangerous implicit conversions.
class Id {
@@ -138,6 +150,10 @@ class Graph {
/// @brief Deletes a relationship from the graph.
void DeleteRelationship(const Relationship &relationship);
bool MustAbort() const;
void CheckMustAbort() const;
private:
mgp_graph *graph_;
};
@@ -159,7 +175,7 @@ class Nodes {
explicit Iterator(mgp_vertices_iterator *nodes_iterator);
Iterator(const Iterator &other);
Iterator(const Iterator &other) noexcept;
Iterator &operator=(const Iterator &other) = delete;
~Iterator();
@@ -207,7 +223,7 @@ class GraphRelationships {
explicit Iterator(mgp_vertices_iterator *nodes_iterator);
Iterator(const Iterator &other);
Iterator(const Iterator &other) noexcept;
Iterator &operator=(const Iterator &other) = delete;
~Iterator();
@@ -253,7 +269,7 @@ class Relationships {
explicit Iterator(mgp_edges_iterator *relationships_iterator);
Iterator(const Iterator &other);
Iterator(const Iterator &other) noexcept;
Iterator &operator=(const Iterator &other) = delete;
~Iterator();
@@ -281,46 +297,12 @@ class Relationships {
mgp_edges_iterator *relationships_iterator_ = nullptr;
};
/// @brief View of node properties.
class Properties {
public:
explicit Properties(mgp_properties_iterator *properties_iterator);
/// @brief Returns the size of the properties map.
size_t Size() const;
/// @brief Returns whether the properties map is empty.
bool Empty() const;
/// @brief Returns the value associated with the given `key`. If theres no such value, the behavior is undefined.
/// @note Each key-value pair needs to be checked, ensuing O(n) time complexity.
Value operator[](const std::string_view key) const;
std::map<std::string_view, Value>::const_iterator begin() const;
std::map<std::string_view, Value>::const_iterator end() const;
std::map<std::string_view, Value>::const_iterator cbegin() const;
std::map<std::string_view, Value>::const_iterator cend() const;
/// @brief Returns the key-value iterator for the given `key`. If theres no such pair, returns the end of the
/// iterator.
/// @note Each key-value pair needs to be checked, ensuing O(n) time complexity.
std::map<std::string_view, Value>::const_iterator find(const std::string_view key) const;
/// @exception std::runtime_error Map contains value(s) of unknown type.
bool operator==(const Properties &other) const;
/// @exception std::runtime_error Map contains value(s) of unknown type.
bool operator!=(const Properties &other) const;
private:
std::map<const std::string_view, Value> property_map_;
};
/// @brief View of node labels.
class Labels {
public:
explicit Labels(mgp_vertex *node_ptr);
Labels(const Labels &other);
Labels(const Labels &other) noexcept;
Labels(Labels &&other) noexcept;
Labels &operator=(const Labels &other) noexcept;
@@ -363,6 +345,7 @@ class Labels {
private:
mgp_vertex *node_ptr_;
};
/* #endregion */
/* #region Types */
@@ -397,7 +380,7 @@ class List {
/// @brief Creates a List from the given initializer_list.
explicit List(const std::initializer_list<Value> list);
List(const List &other);
List(const List &other) noexcept;
List(List &&other) noexcept;
List &operator=(const List &other) noexcept;
@@ -489,7 +472,7 @@ class Map {
/// @brief Creates a Map from the given initializer_list (map items correspond to initializer list pairs).
Map(const std::initializer_list<std::pair<std::string_view, Value>> items);
Map(const Map &other);
Map(const Map &other) noexcept;
Map(Map &&other) noexcept;
Map &operator=(const Map &other) noexcept;
@@ -519,7 +502,7 @@ class Map {
explicit Iterator(mgp_map_items_iterator *map_items_iterator);
Iterator(const Iterator &other);
Iterator(const Iterator &other) noexcept;
Iterator &operator=(const Iterator &other) = delete;
~Iterator();
@@ -578,7 +561,7 @@ class Node {
/// @brief Creates a Node from the copy of the given @ref mgp_vertex.
explicit Node(const mgp_vertex *const_ptr);
Node(const Node &other);
Node(const Node &other) noexcept;
Node(Node &&other) noexcept;
Node &operator=(const Node &other) noexcept;
@@ -590,16 +573,19 @@ class Node {
mgp::Id Id() const;
/// @brief Returns an iterable & indexable structure of the nodes labels.
class Labels Labels() const;
mgp::Labels Labels() const;
/// @brief Returns whether the node has the given `label`.
bool HasLabel(std::string_view label) const;
/// @brief Returns an iterable & indexable structure of the nodes properties.
class Properties Properties() const;
/// @brief Returns an std::map of the nodes properties.
std::map<std::string, Value> Properties() const;
/// @brief Returns the value of the nodes `property_name` property.
Value operator[](const std::string_view property_name) const;
/// @brief Sets the chosen property to the given value.
void SetProperty(std::string property, Value value);
/// @brief Retrieves the value of the chosen property.
Value GetProperty(const std::string &property) const;
/// @brief Returns an iterable structure of the nodes inbound relationships.
Relationships InRelationships() const;
@@ -635,7 +621,7 @@ class Relationship {
/// @brief Creates a Relationship from the copy of the given @ref mgp_edge.
explicit Relationship(const mgp_edge *const_ptr);
Relationship(const Relationship &other);
Relationship(const Relationship &other) noexcept;
Relationship(Relationship &&other) noexcept;
Relationship &operator=(const Relationship &other) noexcept;
@@ -649,11 +635,14 @@ class Relationship {
/// @brief Returns the relationships type.
std::string_view Type() const;
/// @brief Returns an iterable & indexable structure of the relationships properties.
class Properties Properties() const;
/// @brief Returns an std::map of the relationships properties.
std::map<std::string, Value> Properties() const;
/// @brief Returns the value of the relationships `property_name` property.
Value operator[](const std::string_view property_name) const;
/// @brief Sets the chosen property to the given value.
void SetProperty(std::string property, Value value);
/// @brief Retrieves the value of the chosen property.
Value GetProperty(const std::string &property) const;
/// @brief Returns the relationships source node.
Node From() const;
@@ -688,7 +677,7 @@ class Path {
/// @brief Creates a Path starting with the given `start_node`.
explicit Path(const Node &start_node);
Path(const Path &other);
Path(const Path &other) noexcept;
Path(Path &&other) noexcept;
Path &operator=(const Path &other) noexcept;
@@ -744,7 +733,7 @@ class Date {
/// @brief Creates a Date object with the given `year`, `month`, and `day` properties.
Date(int year, int month, int day);
Date(const Date &other);
Date(const Date &other) noexcept;
Date(Date &&other) noexcept;
Date &operator=(const Date &other) noexcept;
@@ -799,7 +788,7 @@ class LocalTime {
/// properties.
LocalTime(int hour, int minute, int second, int millisecond, int microsecond);
LocalTime(const LocalTime &other);
LocalTime(const LocalTime &other) noexcept;
LocalTime(LocalTime &&other) noexcept;
LocalTime &operator=(const LocalTime &other) noexcept;
@@ -858,7 +847,7 @@ class LocalDateTime {
/// `millisecond`, and `microsecond` properties.
LocalDateTime(int year, int month, int day, int hour, int minute, int second, int millisecond, int microsecond);
LocalDateTime(const LocalDateTime &other);
LocalDateTime(const LocalDateTime &other) noexcept;
LocalDateTime(LocalDateTime &&other) noexcept;
LocalDateTime &operator=(const LocalDateTime &other) noexcept;
@@ -929,7 +918,7 @@ class Duration {
/// `microsecond` properties.
Duration(double day, double hour, double minute, double second, double millisecond, double microsecond);
Duration(const Duration &other);
Duration(const Duration &other) noexcept;
Duration(Duration &&other) noexcept;
Duration &operator=(const Duration &other) noexcept;
@@ -986,6 +975,8 @@ class Value {
explicit Value(mgp_value *ptr);
explicit Value(StealType /*steal*/, mgp_value *ptr);
// Null constructor:
explicit Value();
@@ -1056,7 +1047,7 @@ class Value {
/// @note The behavior of accessing `duration` after performing this operation is undefined.
explicit Value(Duration &&duration);
Value(const Value &other);
Value(const Value &other) noexcept;
Value(Value &&other) noexcept;
Value &operator=(const Value &other) noexcept;
@@ -1594,6 +1585,14 @@ inline Id::Id(int64_t id) : id_(id) {}
inline Graph::Graph(mgp_graph *graph) : graph_(graph) {}
inline bool Graph::MustAbort() const { return must_abort(graph_); }
inline void Graph::CheckMustAbort() const {
if (MustAbort()) {
throw MustAbortException("Query was asked to abort.");
}
}
inline int64_t Graph::Order() const {
int64_t i = 0;
for (const auto _ : Nodes()) {
@@ -1704,7 +1703,7 @@ inline Nodes::Iterator::Iterator(mgp_vertices_iterator *nodes_iterator) : nodes_
}
}
inline Nodes::Iterator::Iterator(const Iterator &other) : Iterator(other.nodes_iterator_) {}
inline Nodes::Iterator::Iterator(const Iterator &other) noexcept : Iterator(other.nodes_iterator_) {}
inline Nodes::Iterator::~Iterator() {
if (nodes_iterator_ != nullptr) {
@@ -1795,7 +1794,7 @@ inline GraphRelationships::Iterator::Iterator(mgp_vertices_iterator *nodes_itera
}
}
inline GraphRelationships::Iterator::Iterator(const Iterator &other) : Iterator(other.nodes_iterator_) {}
inline GraphRelationships::Iterator::Iterator(const Iterator &other) noexcept : Iterator(other.nodes_iterator_) {}
inline GraphRelationships::Iterator::~Iterator() {
if (nodes_iterator_ != nullptr) {
@@ -1814,10 +1813,12 @@ inline GraphRelationships::Iterator &GraphRelationships::Iterator::operator++()
if (out_relationships_iterator_ != nullptr) {
auto next = mgp::edges_iterator_next(out_relationships_iterator_);
if (next == nullptr) {
mgp::edges_iterator_destroy(out_relationships_iterator_);
out_relationships_iterator_ = nullptr;
if (next != nullptr) {
return *this;
}
mgp::edges_iterator_destroy(out_relationships_iterator_);
out_relationships_iterator_ = nullptr;
}
// 2. Move onto the next nodes
@@ -1904,7 +1905,7 @@ inline Relationships::Iterator::Iterator(mgp_edges_iterator *relationships_itera
}
}
inline Relationships::Iterator::Iterator(const Iterator &other) : Iterator(other.relationships_iterator_) {}
inline Relationships::Iterator::Iterator(const Iterator &other) noexcept : Iterator(other.relationships_iterator_) {}
inline Relationships::Iterator::~Iterator() {
if (relationships_iterator_ != nullptr) {
@@ -1963,40 +1964,11 @@ inline Relationships::Iterator Relationships::cbegin() const { return Iterator(r
inline Relationships::Iterator Relationships::cend() const { return Iterator(nullptr); }
// Properties:
inline Properties::Properties(mgp_properties_iterator *properties_iterator) {
for (auto property = mgp::properties_iterator_get(properties_iterator); property;
property = mgp::properties_iterator_next(properties_iterator)) {
auto value = Value(property->value);
property_map_.emplace(property->name, value);
}
mgp::properties_iterator_destroy(properties_iterator);
}
inline size_t Properties::Size() const { return property_map_.size(); }
inline bool Properties::Empty() const { return Size() == 0; }
inline Value Properties::operator[](const std::string_view key) const { return property_map_.at(key); }
inline std::map<std::string_view, Value>::const_iterator Properties::begin() const { return property_map_.begin(); }
inline std::map<std::string_view, Value>::const_iterator Properties::end() const { return property_map_.end(); }
inline std::map<std::string_view, Value>::const_iterator Properties::cbegin() const { return property_map_.cbegin(); }
inline std::map<std::string_view, Value>::const_iterator Properties::cend() const { return property_map_.cend(); }
inline bool Properties::operator==(const Properties &other) const { return property_map_ == other.property_map_; }
inline bool Properties::operator!=(const Properties &other) const { return !(*this == other); }
// Labels:
inline Labels::Labels(mgp_vertex *node_ptr) : node_ptr_(mgp::vertex_copy(node_ptr, memory)) {}
inline Labels::Labels(const Labels &other) : Labels(other.node_ptr_) {}
inline Labels::Labels(const Labels &other) noexcept : Labels(other.node_ptr_) {}
inline Labels::Labels(Labels &&other) noexcept : node_ptr_(other.node_ptr_) { other.node_ptr_ = nullptr; }
@@ -2086,7 +2058,7 @@ inline List::List(const std::initializer_list<Value> values) : ptr_(mgp::list_ma
}
}
inline List::List(const List &other) : List(other.ptr_) {}
inline List::List(const List &other) noexcept : List(other.ptr_) {}
inline List::List(List &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; }
@@ -2194,7 +2166,7 @@ inline Map::Map(const std::initializer_list<std::pair<std::string_view, Value>>
}
}
inline Map::Map(const Map &other) : Map(other.ptr_) {}
inline Map::Map(const Map &other) noexcept : Map(other.ptr_) {}
inline Map::Map(Map &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; }
@@ -2239,7 +2211,7 @@ inline Map::Iterator::Iterator(mgp_map_items_iterator *map_items_iterator) : map
}
}
inline Map::Iterator::Iterator(const Iterator &other) : Iterator(other.map_items_iterator_) {}
inline Map::Iterator::Iterator(const Iterator &other) noexcept : Iterator(other.map_items_iterator_) {}
inline Map::Iterator::~Iterator() {
if (map_items_iterator_ != nullptr) {
@@ -2306,10 +2278,6 @@ inline void Map::Insert(std::string_view key, Value &&value) {
value.ptr_ = nullptr;
}
inline std::map<std::string_view, Value>::const_iterator Properties::find(const std::string_view key) const {
return property_map_.find(key);
}
inline bool Map::operator==(const Map &other) const { return util::MapsEqual(ptr_, other.ptr_); }
inline bool Map::operator!=(const Map &other) const { return !(*this == other); }
@@ -2324,7 +2292,7 @@ inline Node::Node(mgp_vertex *ptr) : ptr_(mgp::vertex_copy(ptr, memory)) {}
inline Node::Node(const mgp_vertex *const_ptr) : ptr_(mgp::vertex_copy(const_cast<mgp_vertex *>(const_ptr), memory)) {}
inline Node::Node(const Node &other) : Node(other.ptr_) {}
inline Node::Node(const Node &other) noexcept : Node(other.ptr_) {}
inline Node::Node(Node &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; }
@@ -2355,7 +2323,7 @@ inline Node::~Node() {
inline mgp::Id Node::Id() const { return Id::FromInt(mgp::vertex_get_id(ptr_).as_int); }
inline class Labels Node::Labels() const { return mgp::Labels(ptr_); }
inline mgp::Labels Node::Labels() const { return mgp::Labels(ptr_); }
inline bool Node::HasLabel(std::string_view label) const {
for (const auto node_label : Labels()) {
@@ -2366,10 +2334,6 @@ inline bool Node::HasLabel(std::string_view label) const {
return false;
}
inline class Properties Node::Properties() const { return mgp::Properties(mgp::vertex_iter_properties(ptr_, memory)); }
inline Value Node::operator[](const std::string_view property_name) const { return Properties()[property_name]; }
inline Relationships Node::InRelationships() const {
auto relationship_iterator = mgp::vertex_iter_in_edges(ptr_, memory);
if (relationship_iterator == nullptr) {
@@ -2390,6 +2354,26 @@ inline void Node::AddLabel(const std::string_view label) {
mgp::vertex_add_label(this->ptr_, mgp_label{.name = label.data()});
}
inline std::map<std::string, Value> Node::Properties() const {
mgp_properties_iterator *properties_iterator = mgp::vertex_iter_properties(ptr_, memory);
std::map<std::string, Value> property_map;
for (auto *property = mgp::properties_iterator_get(properties_iterator); property;
property = mgp::properties_iterator_next(properties_iterator)) {
property_map.emplace(std::string(property->name), Value(property->value));
}
mgp::properties_iterator_destroy(properties_iterator);
return property_map;
}
inline void Node::SetProperty(std::string property, Value value) {
mgp::vertex_set_property(ptr_, property.data(), value.ptr());
}
inline Value Node::GetProperty(const std::string &property) const {
mgp_value *vertex_prop = mgp::vertex_get_property(ptr_, property.data(), memory);
return Value(steal, vertex_prop);
}
inline bool Node::operator<(const Node &other) const { return Id() < other.Id(); }
inline bool Node::operator==(const Node &other) const { return util::NodesEqual(ptr_, other.ptr_); }
@@ -2403,7 +2387,7 @@ inline Relationship::Relationship(mgp_edge *ptr) : ptr_(mgp::edge_copy(ptr, memo
inline Relationship::Relationship(const mgp_edge *const_ptr)
: ptr_(mgp::edge_copy(const_cast<mgp_edge *>(const_ptr), memory)) {}
inline Relationship::Relationship(const Relationship &other) : Relationship(other.ptr_) {}
inline Relationship::Relationship(const Relationship &other) noexcept : Relationship(other.ptr_) {}
inline Relationship::Relationship(Relationship &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; }
@@ -2436,12 +2420,24 @@ inline mgp::Id Relationship::Id() const { return Id::FromInt(mgp::edge_get_id(pt
inline std::string_view Relationship::Type() const { return mgp::edge_get_type(ptr_).name; }
inline class Properties Relationship::Properties() const {
return mgp::Properties(mgp::edge_iter_properties(ptr_, memory));
inline std::map<std::string, Value> Relationship::Properties() const {
mgp_properties_iterator *properties_iterator = mgp::edge_iter_properties(ptr_, memory);
std::map<std::string, Value> property_map;
for (mgp_property *property = mgp::properties_iterator_get(properties_iterator); property;
property = mgp::properties_iterator_next(properties_iterator)) {
property_map.emplace(property->name, Value(property->value));
}
mgp::properties_iterator_destroy(properties_iterator);
return property_map;
}
inline Value Relationship::operator[](const std::string_view property_name) const {
return Properties()[property_name];
inline void Relationship::SetProperty(std::string property, Value value) {
mgp::edge_set_property(ptr_, property.data(), value.ptr());
}
inline Value Relationship::GetProperty(const std::string &property) const {
mgp_value *edge_prop = mgp::edge_get_property(ptr_, property.data(), memory);
return Value(steal, edge_prop);
}
inline Node Relationship::From() const { return Node(mgp::edge_get_from(ptr_)); }
@@ -2464,7 +2460,7 @@ inline Path::Path(const mgp_path *const_ptr) : ptr_(mgp::path_copy(const_cast<mg
inline Path::Path(const Node &start_node) : ptr_(mgp::path_make_with_start(start_node.ptr_, memory)) {}
inline Path::Path(const Path &other) : Path(other.ptr_) {}
inline Path::Path(const Path &other) noexcept : Path(other.ptr_) {}
inline Path::Path(Path &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; }
@@ -2533,7 +2529,7 @@ inline Date::Date(int year, int month, int day) {
ptr_ = mgp::date_from_parameters(&params, memory);
}
inline Date::Date(const Date &other) : Date(other.ptr_) {}
inline Date::Date(const Date &other) noexcept : Date(other.ptr_) {}
inline Date::Date(Date &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; }
@@ -2627,7 +2623,7 @@ inline LocalTime::LocalTime(int hour, int minute, int second, int millisecond, i
ptr_ = mgp::local_time_from_parameters(&params, memory);
}
inline LocalTime::LocalTime(const LocalTime &other) : LocalTime(other.ptr_) {}
inline LocalTime::LocalTime(const LocalTime &other) noexcept : LocalTime(other.ptr_) {}
inline LocalTime::LocalTime(LocalTime &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; };
@@ -2732,7 +2728,7 @@ inline LocalDateTime::LocalDateTime(int year, int month, int day, int hour, int
ptr_ = mgp::local_date_time_from_parameters(&params, memory);
}
inline LocalDateTime::LocalDateTime(const LocalDateTime &other) : LocalDateTime(other.ptr_) {}
inline LocalDateTime::LocalDateTime(const LocalDateTime &other) noexcept : LocalDateTime(other.ptr_) {}
inline LocalDateTime::LocalDateTime(LocalDateTime &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; };
@@ -2845,7 +2841,7 @@ inline Duration::Duration(double day, double hour, double minute, double second,
ptr_ = mgp::duration_from_parameters(&params, memory);
}
inline Duration::Duration(const Duration &other) : Duration(other.ptr_) {}
inline Duration::Duration(const Duration &other) noexcept : Duration(other.ptr_) {}
inline Duration::Duration(Duration &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; };
@@ -2917,6 +2913,7 @@ inline bool Duration::operator<(const Duration &other) const {
/* #region Value */
inline Value::Value(mgp_value *ptr) : ptr_(mgp::value_copy(ptr, memory)) {}
inline Value::Value(StealType /*steal*/, mgp_value *ptr) : ptr_{ptr} {}
inline Value::Value() : ptr_(mgp::value_make_null(memory)) {}
@@ -2997,7 +2994,7 @@ inline Value::Value(Duration &&duration) {
duration.ptr_ = nullptr;
}
inline Value::Value(const Value &other) : Value(other.ptr_) {}
inline Value::Value(const Value &other) noexcept : Value(other.ptr_) {}
inline Value::Value(Value &&other) noexcept : ptr_(other.ptr_) { other.ptr_ = nullptr; }

View File

@@ -1283,7 +1283,7 @@ class Graph:
raise InvalidContextError()
self._graph.detach_delete_vertex(vertex._vertex)
def create_edge(self, from_vertex: Vertex, to_vertex: Vertex, edge_type: EdgeType) -> None:
def create_edge(self, from_vertex: Vertex, to_vertex: Vertex, edge_type: EdgeType) -> Edge:
"""
Create an edge.
@@ -1292,13 +1292,16 @@ class Graph:
to_vertex: `Vertex' to where edge is directed.
edge_type: `EdgeType` defines the type of edge.
Returns:
Created `Edge`.
Raises:
ImmutableObjectError: If `graph` is immutable.
UnableToAllocateError: If unable to allocate an edge.
DeletedObjectError: If `from_vertex` or `to_vertex` has been deleted.
SerializationError: If `from_vertex` or `to_vertex` has been modified by another transaction.
Examples:
```graph.create_edge(from_vertex, vertex, edge_type)```
```edge = graph.create_edge(from_vertex, vertex, edge_type)```
"""
if not self.is_valid():
raise InvalidContextError()

1655
include/mgp_mock.py Normal file

File diff suppressed because it is too large Load Diff

72
init
View File

@@ -14,7 +14,6 @@ function print_help () {
echo "Optional arguments:"
echo -e " -h\tdisplay this help and exit"
echo -e " --without-libs-setup\tskip the step for setting up libs"
echo -e " --wsl-quicklisp-proxy \"host:port\"\tquicklist HTTP proxy (this flag + HTTP proxy are required on WSL)"
}
function setup_virtualenv () {
@@ -35,7 +34,6 @@ function setup_virtualenv () {
popd > /dev/null
}
wsl_quicklisp_proxy=""
setup_libs=true
if [[ $# -eq 1 && "$1" == "-h" ]]; then
print_help
@@ -43,16 +41,6 @@ if [[ $# -eq 1 && "$1" == "-h" ]]; then
else
while(($#)); do
case "$1" in
--wsl-quicklisp-proxy)
shift
if [[ $# -eq 0 ]]; then
echo "Missing proxy URL"
print_help
exit 1
fi
wsl_quicklisp_proxy=":proxy \"http://$1/\""
shift
;;
--without-libs-setup)
shift
setup_libs=false
@@ -79,42 +67,17 @@ echo "All packages are in-place..."
# create a default build directory
mkdir -p ./build
# quicklisp package manager for Common Lisp
quicklisp_install_dir="$HOME/quicklisp"
if [[ -v QUICKLISP_HOME ]]; then
quicklisp_install_dir="${QUICKLISP_HOME}"
fi
if [[ ! -f "${quicklisp_install_dir}/setup.lisp" ]]; then
wget -nv https://beta.quicklisp.org/quicklisp.lisp -O quicklisp.lisp || exit 1
echo \
"
(load \"${DIR}/quicklisp.lisp\")
(quicklisp-quickstart:install $wsl_quicklisp_proxy :path \"${quicklisp_install_dir}\")
" | sbcl --script || exit 1
rm -rf quicklisp.lisp || exit 1
fi
ln -Tfs "$DIR/src/lisp" "${quicklisp_install_dir}/local-projects/lcp"
# Install LCP dependencies
# TODO: We should at some point cache or have a mirror of packages we use.
# TODO: move the installation of LCP's dependencies into ./setup.sh
echo \
"
(load \"${quicklisp_install_dir}/setup.lisp\")
(ql:quickload '(:lcp :lcp/test) :silent t)
" | sbcl --script
if [[ "$setup_libs" == "true" ]]; then
# Setup libs (download).
cd libs
./cleanup.sh
./setup.sh
cd ..
# Setup libs (download).
cd libs
./cleanup.sh
./setup.sh
cd ..
fi
# Fix for centos 7 during release
if [ "${ARCHITECTURE}" = "centos-7" ]; then
python3 -m pip uninstall virtualenv
if [ "${DISTRO}" = "centos-7" ] || [ "${DISTRO}" = "debian-11" ] || [ "${DISTRO}" = "amzn-2" ]; then
python3 -m pip uninstall -y virtualenv
python3 -m pip install virtualenv
fi
@@ -143,15 +106,18 @@ for hook in $(find $DIR/.githooks -type f -printf "%f\n"); do
echo "Added $hook hook"
done;
# Install precommit hook
python3 -m pip install pre-commit
python3 -m pre_commit install
# Install py format tools
echo "Install black formatter"
python3 -m pip install black==22.10.*
echo "Install isort"
python3 -m pip install isort==5.10.*
# Install precommit hook except on old operating systems because we don't
# develop on them -> pre-commit hook not required -> we can use latest
# packages.
if [ "${DISTRO}" != "centos-7" ] && [ "$DISTRO" != "debian-10" ] && [ "${DISTRO}" != "ubuntu-18.04" ] && [ "${DISTRO}" != "amzn-2" ]; then
python3 -m pip install pre-commit
python3 -m pre_commit install
# Install py format tools for usage during the development.
echo "Install black formatter"
python3 -m pip install black==23.1.*
echo "Install isort"
python3 -m pip install isort==5.12.*
fi
# Link `include/mgp.py` with `release/mgp/mgp.py`
ln -v -f include/mgp.py release/mgp/mgp.py

View File

@@ -117,7 +117,7 @@ declare -A primary_urls=(
["mgconsole"]="http://$local_cache_host/git/mgconsole.git"
["spdlog"]="http://$local_cache_host/git/spdlog"
["nlohmann"]="http://$local_cache_host/file/nlohmann/json/4f8fba14066156b73f1189a2b8bd568bde5284c5/single_include/nlohmann/json.hpp"
["neo4j"]="http://$local_cache_host/file/neo4j-community-3.2.3-unix.tar.gz"
["neo4j"]="http://$local_cache_host/file/neo4j-community-5.6.0-unix.tar.gz"
["librdkafka"]="http://$local_cache_host/git/librdkafka.git"
["protobuf"]="http://$local_cache_host/git/protobuf.git"
["pulsar"]="http://$local_cache_host/git/pulsar.git"
@@ -142,7 +142,7 @@ declare -A secondary_urls=(
["mgconsole"]="http://github.com/memgraph/mgconsole.git"
["spdlog"]="https://github.com/gabime/spdlog"
["nlohmann"]="https://raw.githubusercontent.com/nlohmann/json/4f8fba14066156b73f1189a2b8bd568bde5284c5/single_include/nlohmann/json.hpp"
["neo4j"]="https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/neo4j-community-3.2.3-unix.tar.gz"
["neo4j"]="https://dist.neo4j.org/neo4j-community-5.6.0-unix.tar.gz"
["librdkafka"]="https://github.com/edenhill/librdkafka.git"
["protobuf"]="https://github.com/protocolbuffers/protobuf.git"
["pulsar"]="https://github.com/apache/pulsar.git"
@@ -180,9 +180,9 @@ repo_clone_try_double "${primary_urls[libbcrypt]}" "${secondary_urls[libbcrypt]}
# neo4j
file_get_try_double "${primary_urls[neo4j]}" "${secondary_urls[neo4j]}"
tar -xzf neo4j-community-3.2.3-unix.tar.gz
mv neo4j-community-3.2.3 neo4j
rm neo4j-community-3.2.3-unix.tar.gz
tar -xzf neo4j-community-5.6.0-unix.tar.gz
mv neo4j-community-5.6.0 neo4j
rm neo4j-community-5.6.0-unix.tar.gz
# nlohmann json
# We wget header instead of cloning repo since repo is huge (lots of test data).

View File

@@ -36,7 +36,7 @@ ADDITIONAL USE GRANT: You may use the Licensed Work in accordance with the
3. using the Licensed Work to create a work or solution
which competes (or might reasonably be expected to
compete) with the Licensed Work.
CHANGE DATE: 2026-07-11
CHANGE DATE: 2027-05-04
CHANGE LICENSE: Apache License, Version 2.0
For information about alternative licensing arrangements, please visit: https://memgraph.com/legal.

202
licenses/third-party/ldbc/LICENSE vendored Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

View File

@@ -1,6 +1,10 @@
# Install systemd service (must use absolute path).
install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/memgraph.service
DESTINATION /lib/systemd/system)
DESTINATION /lib/systemd/system)
# Set parameters to recognize the host distro
cmake_host_system_information(RESULT DISTRO QUERY DISTRIB_NAME)
cmake_host_system_information(RESULT DISTRO_VERSION QUERY DISTRIB_VERSION)
# ---- Setup CPack --------
@@ -12,10 +16,11 @@ set(CPACK_PACKAGE_DESCRIPTION_SUMMARY
# Setting arhitecture extension for deb packages
set(MG_ARCH_EXTENSION_DEB "all")
if (${MG_ARCH} STREQUAL "x86_64")
if(${MG_ARCH} STREQUAL "x86_64")
set(MG_ARCH_EXTENSION_DEB "amd64")
elseif (${MG_ARCH} STREQUAL "ARM64")
set(MG_ARCH_EXTENSION_DEB "arm64")
elseif(${MG_ARCH} STREQUAL "ARM64")
set(MG_ARCH_EXTENSION_DEB "arm64")
endif()
# DEB specific
@@ -34,21 +39,24 @@ set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA
"${CMAKE_CURRENT_SOURCE_DIR}/debian/postrm;"
"${CMAKE_CURRENT_SOURCE_DIR}/debian/postinst;")
set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)
# Description formatting is important, summary must be followed with a newline and 1 space.
set(CPACK_DEBIAN_PACKAGE_DESCRIPTION "${CPACK_PACKAGE_DESCRIPTION_SUMMARY}
Contains Memgraph, the graph database. It aims to deliver developers the
speed, simplicity and scale required to build the next generation of
applications driver by real-time connected data.")
# Add `openssl` package to dependencies list. Used to generate SSL certificates.
# We also depend on `python3` because we embed it in Memgraph.
set(CPACK_DEBIAN_PACKAGE_DEPENDS "openssl (>= 1.1.0), python3 (>= 3.5.0), libstdc++6")
# Setting arhitecture extension for rpm packages
set(MG_ARCH_EXTENSION_RPM "noarch")
if (${MG_ARCH} STREQUAL "x86_64")
if(${MG_ARCH} STREQUAL "x86_64")
set(MG_ARCH_EXTENSION_RPM "x86_64")
elseif (${MG_ARCH} STREQUAL "ARM64")
set(MG_ARCH_EXTENSION_RPM "aarch64")
elseif(${MG_ARCH} STREQUAL "ARM64")
set(MG_ARCH_EXTENSION_RPM "aarch64")
endif()
# RPM specific
@@ -56,18 +64,26 @@ set(CPACK_RPM_PACKAGE_URL https://memgraph.com)
set(CPACK_RPM_PACKAGE_VERSION "${MEMGRAPH_VERSION_RPM}")
set(CPACK_RPM_FILE_NAME "memgraph-${MEMGRAPH_VERSION_RPM}-1.${MG_ARCH_EXTENSION_RPM}.rpm")
set(CPACK_RPM_EXCLUDE_FROM_AUTO_FILELIST_ADDITION
/var /var/lib /var/log /etc/logrotate.d
/lib /lib/systemd /lib/systemd/system /lib/systemd/system/memgraph.service)
/var /var/lib /var/log /etc/logrotate.d
/lib /lib/systemd /lib/systemd/system /lib/systemd/system/memgraph.service)
set(CPACK_RPM_PACKAGE_REQUIRES_PRE "shadow-utils")
set(CPACK_RPM_USER_BINARY_SPECFILE "${CMAKE_CURRENT_SOURCE_DIR}/rpm/memgraph.spec.in")
set(CPACK_RPM_PACKAGE_LICENSE "Memgraph License")
# Description formatting is important, no line must be greater than 80 characters.
set(CPACK_RPM_PACKAGE_DESCRIPTION "Contains Memgraph, the graph database.
It aims to deliver developers the speed, simplicity and scale required to build
the next generation of applications driver by real-time connected data.")
# Add `openssl` package to dependencies list. Used to generate SSL certificates.
# We also depend on `python3` because we embed it in Memgraph.
set(CPACK_RPM_PACKAGE_REQUIRES "openssl >= 1.0.0, curl >= 7.29.0, python3 >= 3.5.0, libstdc >= 6, logrotate")
set(CPACK_RPM_PACKAGE_REQUIRES "openssl >= 1.0.0, curl >= 7.29.0, python3 >= 3.5.0, libstdc++ >= 3.4.29, logrotate")
# If amzn-2
if(DISTRO STREQUAL "Amazon Linux" AND DISTRO_VERSION STREQUAL "2")
# It causes issues with glibcxx 2.4
set(CPACK_RPM_PACKAGE_AUTOREQ " no")
endif()
# All variables must be set before including.
include(CPack)

View File

@@ -1,3 +1,4 @@
.venv
dist
mgp.py
poetry.lock

View File

@@ -1,6 +1,6 @@
[tool.poetry]
name = "mgp"
version = "1.1.0"
version = "1.1.1"
description = "Memgraph's module for developing MAGE modules. Used only for type hinting!"
authors = [
"katarinasupe <katarina.supe@memgraph.io>",

View File

@@ -0,0 +1,14 @@
FROM amazonlinux:2
ARG TOOLCHAIN_VERSION
RUN yum -y update \
&& yum install -y wget git tar
# Do NOT be smart here and clean the cache because the container is used in the
# stateful context.
RUN wget -q https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/${TOOLCHAIN_VERSION}/${TOOLCHAIN_VERSION}-binaries-amzn-2-x86_64.tar.gz \
-O ${TOOLCHAIN_VERSION}-binaries-amzn-2-x86_64.tar.gz \
&& tar xzvf ${TOOLCHAIN_VERSION}-binaries-amzn-2-x86_64.tar.gz -C /opt
ENTRYPOINT ["sleep", "infinity"]

View File

@@ -0,0 +1,11 @@
version: "3"
services:
debian-11-arm:
build:
context: debian-11-arm
container_name: "mgbuild_debian-11-arm"
ubuntu-2204-arm:
build:
context: ubuntu-22.04-arm
container_name: "mgbuild_ubuntu-22.04-arm"

View File

@@ -28,3 +28,11 @@ services:
build:
context: ubuntu-22.04
container_name: "mgbuild_ubuntu-22.04"
mgbuild_fedora-36:
build:
context: fedora-36
container_name: "mgbuild_fedora-36"
mgbuild_amzn-2:
build:
context: amzn-2
container_name: "mgbuild_amzn-2"

View File

@@ -0,0 +1,15 @@
FROM fedora:36
ARG TOOLCHAIN_VERSION
# Stops tzdata interactive configuration.
RUN yum -y update \
&& yum install -y wget git
# Do NOT be smart here and clean the cache because the container is used in the
# stateful context.
RUN wget -q https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/${TOOLCHAIN_VERSION}/${TOOLCHAIN_VERSION}-binaries-fedora-36-x86_64.tar.gz \
-O ${TOOLCHAIN_VERSION}-binaries-fedora-36-x86_64.tar.gz \
&& tar xzvf ${TOOLCHAIN_VERSION}-binaries-fedora-36-x86_64.tar.gz -C /opt
ENTRYPOINT ["sleep", "infinity"]

View File

@@ -3,7 +3,14 @@
set -Eeuo pipefail
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
SUPPORTED_OS=(centos-7 centos-9 debian-10 debian-11 ubuntu-18.04 ubuntu-20.04 ubuntu-22.04 debian-11-arm)
SUPPORTED_OS=(
centos-7 centos-9
debian-10 debian-11 debian-11-arm
ubuntu-18.04 ubuntu-20.04 ubuntu-22.04 ubuntu-22.04-arm
fedora-36
amzn-2
)
PROJECT_ROOT="$SCRIPT_DIR/../.."
TOOLCHAIN_VERSION="toolchain-v4"
ACTIVATE_TOOLCHAIN="source /opt/${TOOLCHAIN_VERSION}/activate"
@@ -23,9 +30,9 @@ make_package () {
echo "Building Memgraph for $os on $build_container..."
package_command=""
if [[ "$os" =~ ^"centos".* ]]; then
if [[ "$os" =~ ^"centos".* ]] || [[ "$os" =~ ^"fedora".* ]] || [[ "$os" =~ ^"amzn".* ]]; then
docker exec "$build_container" bash -c "yum -y update"
package_command=" cpack -G RPM --config ../CPackConfig.cmake && rpmlint memgraph*.rpm "
package_command=" cpack -G RPM --config ../CPackConfig.cmake && rpmlint --file='../../release/rpm/rpmlintrc' memgraph*.rpm "
fi
if [[ "$os" =~ ^"debian".* ]]; then
docker exec "$build_container" bash -c "apt update"
@@ -76,7 +83,7 @@ make_package () {
docker exec "$build_container" bash -c "cd /memgraph && git config --global --add safe.directory '*'"
docker exec "$build_container" bash -c "cd /memgraph && $ACTIVATE_TOOLCHAIN && ./init"
docker exec "$build_container" bash -c "cd $container_build_dir && rm -rf ./*"
if [[ "$os" == "debian-11-arm" ]]; then
if [[ "$os" =~ "-arm" ]]; then
docker exec "$build_container" bash -c "cd $container_build_dir && $ACTIVATE_TOOLCHAIN && cmake -DCMAKE_BUILD_TYPE=release -DMG_ARCH="ARM64" $telemetry_id_override_flag .."
else
docker exec "$build_container" bash -c "cd $container_build_dir && $ACTIVATE_TOOLCHAIN && cmake -DCMAKE_BUILD_TYPE=release $telemetry_id_override_flag .."

View File

@@ -0,0 +1,17 @@
FROM ubuntu:22.04
ARG TOOLCHAIN_VERSION
# Stops tzdata interactive configuration.
ENV DEBIAN_FRONTEND=noninteractive
RUN apt update && apt install -y \
ca-certificates wget git
# Do NOT be smart here and clean the cache because the container is used in the
# stateful context.
RUN wget -q https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/${TOOLCHAIN_VERSION}/${TOOLCHAIN_VERSION}-binaries-ubuntu-22.04-arm64.tar.gz \
-O ${TOOLCHAIN_VERSION}-binaries-ubuntu-22.04-arm64.tar.gz \
&& tar xzvf ${TOOLCHAIN_VERSION}-binaries-ubuntu-22.04-arm64.tar.gz -C /opt
ENTRYPOINT ["sleep", "infinity"]

View File

@@ -30,7 +30,7 @@ BuildRequires: systemd
# This is needed to prevent Python compilation errors when building the RPM
# package
# https://github.com/scylladb/scylla/issues/2235
%if 0%{?rhel} < 8
%if 0%{?rhel} && 0%{?rhel} < 8
%global __os_install_post \
/usr/lib/rpm/redhat/brp-compress \
%{!?__debug_package:\
@@ -40,7 +40,9 @@ BuildRequires: systemd
/usr/lib/rpm/redhat/brp-strip-static-archive %{__strip} \
%{!?__jar_repack:/usr/lib/rpm/redhat/brp-java-repack-jars} \
%{nil}
%else
%endif
%if 0%{?fedora} && 0%{?fedora} < 35
%global __os_install_post \
/usr/lib/rpm/brp-compress \
%{!?__debug_package:\

4
release/rpm/rpmlintrc Normal file
View File

@@ -0,0 +1,4 @@
# from https://github.com/google/earthenterprise/blob/master/earth_enterprise/rpmlintrc
# We are not packaging log dir
addFilter("E: logrotate-log-dir-not-packaged")

View File

@@ -1,7 +1,6 @@
# CMake configuration for the main memgraph library and executable
# add memgraph sub libraries, ordered by dependency
add_subdirectory(lisp)
add_subdirectory(utils)
add_subdirectory(requests)
add_subdirectory(io)
@@ -106,6 +105,10 @@ install(PROGRAMS $<TARGET_FILE:memgraph>
# Install Python source for supporting our embedded Python.
install(FILES ${CMAKE_SOURCE_DIR}/include/mgp.py
DESTINATION lib/memgraph/python_support)
install(FILES ${CMAKE_SOURCE_DIR}/include/mgp_mock.py
DESTINATION lib/memgraph/python_support)
install(FILES ${CMAKE_SOURCE_DIR}/include/_mgp_mock.py
DESTINATION lib/memgraph/python_support)
# Install the includes file for writing custom procedures in C and C++>
install(FILES ${CMAKE_SOURCE_DIR}/include/mg_procedure.h

View File

@@ -16,6 +16,10 @@ target_link_libraries(mg-auth mg-utils mg-kvstore mg-license )
target_link_libraries(mg-auth ${Seccomp_LIBRARIES})
target_include_directories(mg-auth SYSTEM PRIVATE ${Seccomp_INCLUDE_DIRS})
find_package(OpenSSL REQUIRED)
target_link_libraries(mg-auth ${OPENSSL_LIBRARIES})
target_include_directories(mg-auth SYSTEM PUBLIC ${OPENSSL_INCLUDE_DIR})
# Install reference auth modules and their configuration files.
install(PROGRAMS ${CMAKE_CURRENT_SOURCE_DIR}/reference_modules/example.py
DESTINATION lib/memgraph/auth_module)

View File

@@ -1,19 +1,64 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Licensed as a Memgraph Enterprise file under the Memgraph Enterprise
// License (the "License"); by using this file, you agree to be bound by the terms of the License, and you may not use
// this file except in compliance with the License. You may obtain a copy of the License at https://memgraph.com/legal.
//
//
#include "auth/crypto.hpp"
#include <iomanip>
#include <sstream>
#include <gflags/gflags.h>
#include <libbcrypt/bcrypt.h>
#include <openssl/evp.h>
#include <openssl/opensslv.h>
#include <openssl/sha.h>
#include "auth/exceptions.hpp"
#include "utils/enum.hpp"
#include "utils/flag_validation.hpp"
namespace {
using namespace std::literals;
inline constexpr std::array password_encryption_mappings{
std::pair{"bcrypt"sv, memgraph::auth::PasswordEncryptionAlgorithm::BCRYPT},
std::pair{"sha256"sv, memgraph::auth::PasswordEncryptionAlgorithm::SHA256},
std::pair{"sha256-multiple"sv, memgraph::auth::PasswordEncryptionAlgorithm::SHA256_MULTIPLE}};
inline constexpr uint64_t ONE_SHA_ITERATION = 1;
inline constexpr uint64_t MULTIPLE_SHA_ITERATIONS = 1024;
} // namespace
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables,misc-unused-parameters)
DEFINE_VALIDATED_string(password_encryption_algorithm, "bcrypt",
"The password encryption algorithm used for authentication.", {
if (const auto result =
memgraph::utils::IsValidEnumValueString(value, password_encryption_mappings);
result.HasError()) {
const auto error = result.GetError();
switch (error) {
case memgraph::utils::ValidationError::EmptyValue: {
std::cout << "Password encryption algorithm cannot be empty." << std::endl;
break;
}
case memgraph::utils::ValidationError::InvalidValue: {
std::cout << "Invalid value for password encryption algorithm. Allowed values: "
<< memgraph::utils::GetAllowedEnumValuesString(password_encryption_mappings)
<< std::endl;
break;
}
}
return false;
}
return true;
});
namespace memgraph::auth {
const std::string EncryptPassword(const std::string &password) {
namespace BCrypt {
std::string EncryptPassword(const std::string &password) {
char salt[BCRYPT_HASHSIZE];
char hash[BCRYPT_HASHSIZE];
@@ -28,7 +73,7 @@ const std::string EncryptPassword(const std::string &password) {
throw AuthException("Couldn't hash password!");
}
return std::string(hash);
return {hash};
}
bool VerifyPassword(const std::string &password, const std::string &hash) {
@@ -38,5 +83,102 @@ bool VerifyPassword(const std::string &password, const std::string &hash) {
}
return ret == 0;
}
} // namespace BCrypt
namespace SHA {
#if OPENSSL_VERSION_MAJOR >= 3
std::string EncryptPasswordOpenSSL3(const std::string &password, const uint64_t number_of_iterations) {
unsigned char hash[SHA256_DIGEST_LENGTH];
EVP_MD_CTX *ctx = EVP_MD_CTX_new();
EVP_MD *md = EVP_MD_fetch(nullptr, "SHA2-256", nullptr);
EVP_DigestInit_ex(ctx, md, nullptr);
for (auto i = 0; i < number_of_iterations; i++) {
EVP_DigestUpdate(ctx, password.c_str(), password.size());
}
EVP_DigestFinal_ex(ctx, hash, nullptr);
EVP_MD_free(md);
EVP_MD_CTX_free(ctx);
std::stringstream result_stream;
for (auto hash_char : hash) {
result_stream << std::hex << std::setw(2) << std::setfill('0') << (int)hash_char;
}
return result_stream.str();
}
#else
std::string EncryptPasswordOpenSSL1_1(const std::string &password, const uint64_t number_of_iterations) {
unsigned char hash[SHA256_DIGEST_LENGTH];
SHA256_CTX sha256;
SHA256_Init(&sha256);
for (auto i = 0; i < number_of_iterations; i++) {
SHA256_Update(&sha256, password.c_str(), password.size());
}
SHA256_Final(hash, &sha256);
std::stringstream ss;
for (auto hash_char : hash) {
ss << std::hex << std::setw(2) << std::setfill('0') << (int)hash_char;
}
return ss.str();
}
#endif
std::string EncryptPassword(const std::string &password, const uint64_t number_of_iterations) {
#if OPENSSL_VERSION_MAJOR >= 3
return EncryptPasswordOpenSSL3(password, number_of_iterations);
#else
return EncryptPasswordOpenSSL1_1(password, number_of_iterations);
#endif
}
bool VerifyPassword(const std::string &password, const std::string &hash, const uint64_t number_of_iterations) {
auto password_hash = EncryptPassword(password, number_of_iterations);
return password_hash == hash;
}
} // namespace SHA
bool VerifyPassword(const std::string &password, const std::string &hash) {
const auto password_encryption_algorithm = utils::StringToEnum<PasswordEncryptionAlgorithm>(
FLAGS_password_encryption_algorithm, password_encryption_mappings);
if (!password_encryption_algorithm.has_value()) {
throw AuthException("Invalid password encryption flag '{}'!", FLAGS_password_encryption_algorithm);
}
switch (password_encryption_algorithm.value()) {
case PasswordEncryptionAlgorithm::BCRYPT:
return BCrypt::VerifyPassword(password, hash);
case PasswordEncryptionAlgorithm::SHA256:
return SHA::VerifyPassword(password, hash, ONE_SHA_ITERATION);
case PasswordEncryptionAlgorithm::SHA256_MULTIPLE:
return SHA::VerifyPassword(password, hash, MULTIPLE_SHA_ITERATIONS);
}
throw AuthException("Invalid password encryption flag '{}'!", FLAGS_password_encryption_algorithm);
}
std::string EncryptPassword(const std::string &password) {
const auto password_encryption_algorithm = utils::StringToEnum<PasswordEncryptionAlgorithm>(
FLAGS_password_encryption_algorithm, password_encryption_mappings);
if (!password_encryption_algorithm.has_value()) {
throw AuthException("Invalid password encryption flag '{}'!", FLAGS_password_encryption_algorithm);
}
switch (password_encryption_algorithm.value()) {
case PasswordEncryptionAlgorithm::BCRYPT:
return BCrypt::EncryptPassword(password);
case PasswordEncryptionAlgorithm::SHA256:
return SHA::EncryptPassword(password, ONE_SHA_ITERATION);
case PasswordEncryptionAlgorithm::SHA256_MULTIPLE:
return SHA::EncryptPassword(password, MULTIPLE_SHA_ITERATIONS);
}
}
} // namespace memgraph::auth

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Licensed as a Memgraph Enterprise file under the Memgraph Enterprise
// License (the "License"); by using this file, you agree to be bound by the terms of the License, and you may not use
@@ -11,10 +11,11 @@
#include <string>
namespace memgraph::auth {
enum class PasswordEncryptionAlgorithm : uint8_t { BCRYPT, SHA256, SHA256_MULTIPLE };
/// @throw AuthException if unable to encrypt the password.
const std::string EncryptPassword(const std::string &password);
std::string EncryptPassword(const std::string &password);
/// @throw AuthException if unable to verify the password.
bool VerifyPassword(const std::string &password, const std::string &hash);
} // namespace memgraph::auth

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Licensed as a Memgraph Enterprise file under the Memgraph Enterprise
// License (the "License"); by using this file, you agree to be bound by the terms of the License, and you may not use
@@ -16,6 +16,7 @@
#include "auth/crypto.hpp"
#include "auth/exceptions.hpp"
#include "license/license.hpp"
#include "query/constants.hpp"
#include "utils/cast.hpp"
#include "utils/logging.hpp"
#include "utils/settings.hpp"
@@ -34,13 +35,19 @@ namespace memgraph::auth {
namespace {
// Constant list of all available permissions.
const std::vector<Permission> kPermissionsAll = {
Permission::MATCH, Permission::CREATE, Permission::MERGE, Permission::DELETE,
Permission::SET, Permission::REMOVE, Permission::INDEX, Permission::STATS,
Permission::CONSTRAINT, Permission::DUMP, Permission::AUTH, Permission::REPLICATION,
Permission::DURABILITY, Permission::READ_FILE, Permission::FREE_MEMORY, Permission::TRIGGER,
Permission::CONFIG, Permission::STREAM, Permission::MODULE_READ, Permission::MODULE_WRITE,
Permission::WEBSOCKET};
const std::vector<Permission> kPermissionsAll = {Permission::MATCH, Permission::CREATE,
Permission::MERGE, Permission::DELETE,
Permission::SET, Permission::REMOVE,
Permission::INDEX, Permission::STATS,
Permission::CONSTRAINT, Permission::DUMP,
Permission::AUTH, Permission::REPLICATION,
Permission::DURABILITY, Permission::READ_FILE,
Permission::FREE_MEMORY, Permission::TRIGGER,
Permission::CONFIG, Permission::STREAM,
Permission::MODULE_READ, Permission::MODULE_WRITE,
Permission::WEBSOCKET, Permission::TRANSACTION_MANAGEMENT,
Permission::STORAGE_MODE};
} // namespace
std::string PermissionToString(Permission permission) {
@@ -87,6 +94,10 @@ std::string PermissionToString(Permission permission) {
return "MODULE_WRITE";
case Permission::WEBSOCKET:
return "WEBSOCKET";
case Permission::TRANSACTION_MANAGEMENT:
return "TRANSACTION_MANAGEMENT";
case Permission::STORAGE_MODE:
return "STORAGE_MODE";
}
}
@@ -264,7 +275,7 @@ PermissionLevel FineGrainedAccessPermissions::Has(const std::string &permission,
void FineGrainedAccessPermissions::Grant(const std::string &permission,
const FineGrainedPermission fine_grained_permission) {
if (permission == kAsterisk) {
if (permission == query::kAsterisk) {
global_permission_ = CalculateGrant(fine_grained_permission);
} else {
permissions_[permission] = CalculateGrant(fine_grained_permission);
@@ -272,7 +283,7 @@ void FineGrainedAccessPermissions::Grant(const std::string &permission,
}
void FineGrainedAccessPermissions::Revoke(const std::string &permission) {
if (permission == kAsterisk) {
if (permission == query::kAsterisk) {
permissions_.clear();
global_permission_ = std::nullopt;
} else {

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Licensed as a Memgraph Enterprise file under the Memgraph Enterprise
// License (the "License"); by using this file, you agree to be bound by the terms of the License, and you may not use
@@ -15,7 +15,6 @@
#include <json/json.hpp>
namespace memgraph::auth {
const std::string kAsterisk = "*";
// These permissions must have values that are applicable for usage in a
// bitmask.
// clang-format off
@@ -40,7 +39,9 @@ enum class Permission : uint64_t {
STREAM = 1U << 17U,
MODULE_READ = 1U << 18U,
MODULE_WRITE = 1U << 19U,
WEBSOCKET = 1U << 20U
WEBSOCKET = 1U << 20U,
TRANSACTION_MANAGEMENT = 1U << 21U,
STORAGE_MODE = 1U << 22U
};
// clang-format on

View File

@@ -146,9 +146,10 @@ QueryData Client::Execute(const std::string &query, const std::map<std::string,
throw ServerMalformedDataException();
}
auto &header = fields.ValueMap();
QueryData ret{{}, std::move(records), std::move(metadata.ValueMap())};
auto &header = fields.ValueMap();
if (header.find("fields") == header.end()) {
throw ServerMalformedDataException();
}
@@ -164,6 +165,10 @@ QueryData Client::Execute(const std::string &query, const std::map<std::string,
ret.fields.emplace_back(std::move(field_item.ValueString()));
}
if (header.contains("qid")) {
ret.metadata["qid"] = header["qid"];
}
return ret;
}

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -73,40 +73,6 @@ inline std::pair<std::string, std::string> ExceptionToErrorMessage(const std::ex
namespace details {
template <typename TSession>
State HandleRun(TSession &session, const State state, const Value &query, const Value &params) {
if (state != State::Idle) {
// Client could potentially recover if we move to error state, but there is
// no legitimate situation in which well working client would end up in this
// situation.
spdlog::trace("Unexpected RUN command!");
return State::Close;
}
DMG_ASSERT(!session.encoder_buffer_.HasData(), "There should be no data to write in this state");
spdlog::debug("[Run] '{}'", query.ValueString());
try {
// Interpret can throw.
const auto [header, qid] = session.Interpret(query.ValueString(), params.ValueMap());
// Convert std::string to Value
std::vector<Value> vec;
std::map<std::string, Value> data;
vec.reserve(header.size());
for (auto &i : header) vec.emplace_back(std::move(i));
data.emplace("fields", std::move(vec));
// Send the header.
if (!session.encoder_.MessageSuccess(data)) {
spdlog::trace("Couldn't send query header!");
return State::Close;
}
return State::Result;
} catch (const std::exception &e) {
return HandleFailure(session, e);
}
}
template <bool is_pull, typename TSession>
State HandlePullDiscard(TSession &session, std::optional<int> n, std::optional<int> qid) {
try {
@@ -229,7 +195,36 @@ State HandleRunV1(TSession &session, const State state, const Marker marker) {
return State::Close;
}
return details::HandleRun(session, state, query, params);
if (state != State::Idle) {
// Client could potentially recover if we move to error state, but there is
// no legitimate situation in which well working client would end up in this
// situation.
spdlog::trace("Unexpected RUN command!");
return State::Close;
}
DMG_ASSERT(!session.encoder_buffer_.HasData(), "There should be no data to write in this state");
spdlog::debug("[Run] '{}'", query.ValueString());
try {
// Interpret can throw.
const auto [header, qid] = session.Interpret(query.ValueString(), params.ValueMap());
// Convert std::string to Value
std::vector<Value> vec;
std::map<std::string, Value> data;
vec.reserve(header.size());
for (auto &i : header) vec.emplace_back(std::move(i));
data.emplace("fields", std::move(vec));
// Send the header.
if (!session.encoder_.MessageSuccess(data)) {
spdlog::trace("Couldn't send query header!");
return State::Close;
}
return State::Result;
} catch (const std::exception &e) {
return HandleFailure(session, e);
}
}
template <typename TSession>
@@ -257,7 +252,40 @@ State HandleRunV4(TSession &session, const State state, const Marker marker) {
spdlog::trace("Couldn't read extra field!");
}
return details::HandleRun(session, state, query, params);
if (state != State::Idle) {
// Client could potentially recover if we move to error state, but there is
// no legitimate situation in which well working client would end up in this
// situation.
spdlog::trace("Unexpected RUN command!");
return State::Close;
}
DMG_ASSERT(!session.encoder_buffer_.HasData(), "There should be no data to write in this state");
spdlog::debug("[Run] '{}'", query.ValueString());
try {
// Interpret can throw.
const auto [header, qid] = session.Interpret(query.ValueString(), params.ValueMap());
// Convert std::string to Value
std::vector<Value> vec;
std::map<std::string, Value> data;
vec.reserve(header.size());
for (auto &i : header) vec.emplace_back(std::move(i));
data.emplace("fields", std::move(vec));
if (qid.has_value()) {
data.emplace("qid", Value{*qid});
}
// Send the header.
if (!session.encoder_.MessageSuccess(data)) {
spdlog::trace("Couldn't send query header!");
return State::Close;
}
return State::Result;
} catch (const std::exception &e) {
return HandleFailure(session, e);
}
}
template <typename TSession>

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -58,6 +58,10 @@ auth::Permission PrivilegeToPermission(query::AuthQuery::Privilege privilege) {
return auth::Permission::MODULE_WRITE;
case query::AuthQuery::Privilege::WEBSOCKET:
return auth::Permission::WEBSOCKET;
case query::AuthQuery::Privilege::STORAGE_MODE:
return auth::Permission::STORAGE_MODE;
case query::AuthQuery::Privilege::TRANSACTION_MANAGEMENT:
return auth::Permission::TRANSACTION_MANAGEMENT;
}
}

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -15,6 +15,7 @@
#include "auth/models.hpp"
#include "glue/auth.hpp"
#include "license/license.hpp"
#include "query/constants.hpp"
#include "query/frontend/ast/ast.hpp"
#include "utils/synchronized.hpp"
@@ -38,7 +39,7 @@ bool IsUserAuthorizedGloballyLabels(const memgraph::auth::User &user,
if (!memgraph::license::global_license_checker.IsEnterpriseValidFast()) {
return true;
}
return user.GetFineGrainedAccessLabelPermissions().Has(memgraph::auth::kAsterisk, fine_grained_permission) ==
return user.GetFineGrainedAccessLabelPermissions().Has(memgraph::query::kAsterisk, fine_grained_permission) ==
memgraph::auth::PermissionLevel::GRANT;
}
@@ -47,7 +48,7 @@ bool IsUserAuthorizedGloballyEdges(const memgraph::auth::User &user,
if (!memgraph::license::global_license_checker.IsEnterpriseValidFast()) {
return true;
}
return user.GetFineGrainedAccessEdgeTypePermissions().Has(memgraph::auth::kAsterisk, fine_grained_permission) ==
return user.GetFineGrainedAccessEdgeTypePermissions().Has(memgraph::query::kAsterisk, fine_grained_permission) ==
memgraph::auth::PermissionLevel::GRANT;
}
@@ -84,6 +85,7 @@ bool AuthChecker::IsUserAuthorized(const std::optional<std::string> &username,
return maybe_user.has_value() && IsUserAuthorized(*maybe_user, privileges);
}
#ifdef MG_ENTERPRISE
std::unique_ptr<memgraph::query::FineGrainedAuthChecker> AuthChecker::GetFineGrainedAuthChecker(
const std::string &username, const memgraph::query::DbAccessor *dba) const {

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -26,9 +26,11 @@ class AuthChecker : public query::AuthChecker {
bool IsUserAuthorized(const std::optional<std::string> &username,
const std::vector<query::AuthQuery::Privilege> &privileges) const override;
#ifdef MG_ENTERPRISE
std::unique_ptr<memgraph::query::FineGrainedAuthChecker> GetFineGrainedAuthChecker(
const std::string &username, const memgraph::query::DbAccessor *dba) const override;
#endif
[[nodiscard]] static bool IsUserAuthorized(const memgraph::auth::User &user,
const std::vector<memgraph::query::AuthQuery::Privilege> &privileges);

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -18,6 +18,7 @@
#include "auth/models.hpp"
#include "glue/auth.hpp"
#include "license/license.hpp"
#include "query/constants.hpp"
namespace {
@@ -253,19 +254,18 @@ bool AuthQueryHandler::CreateUser(const std::string &username, const std::option
if (first_user) {
spdlog::info("{} is first created user. Granting all privileges.", username);
GrantPrivilege(username, memgraph::query::kPrivilegesAll
GrantPrivilege(
username, memgraph::query::kPrivilegesAll
#ifdef MG_ENTERPRISE
,
{{{memgraph::query::AuthQuery::FineGrainedPrivilege::CREATE_DELETE, {memgraph::auth::kAsterisk}}}},
{
{
{
memgraph::query::AuthQuery::FineGrainedPrivilege::CREATE_DELETE, {
memgraph::auth::kAsterisk
}
}
}
}
,
{{{memgraph::query::AuthQuery::FineGrainedPrivilege::CREATE_DELETE, {memgraph::query::kAsterisk}}}},
{
{
{
memgraph::query::AuthQuery::FineGrainedPrivilege::CREATE_DELETE, { memgraph::query::kAsterisk }
}
}
}
#endif
);
}

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -57,6 +57,7 @@
#include "storage/v2/storage.hpp"
#include "storage/v2/view.hpp"
#include "telemetry/telemetry.hpp"
#include "utils/enum.hpp"
#include "utils/event_counter.hpp"
#include "utils/file.hpp"
#include "utils/flag_validation.hpp"
@@ -103,42 +104,6 @@ constexpr const char *kMgUser = "MEMGRAPH_USER";
constexpr const char *kMgPassword = "MEMGRAPH_PASSWORD";
constexpr const char *kMgPassfile = "MEMGRAPH_PASSFILE";
namespace {
std::string GetAllowedEnumValuesString(const auto &mappings) {
std::vector<std::string> allowed_values;
allowed_values.reserve(mappings.size());
std::transform(mappings.begin(), mappings.end(), std::back_inserter(allowed_values),
[](const auto &mapping) { return std::string(mapping.first); });
return memgraph::utils::Join(allowed_values, ", ");
}
enum class ValidationError : uint8_t { EmptyValue, InvalidValue };
memgraph::utils::BasicResult<ValidationError> IsValidEnumValueString(const auto &value, const auto &mappings) {
if (value.empty()) {
return ValidationError::EmptyValue;
}
if (std::find_if(mappings.begin(), mappings.end(), [&](const auto &mapping) { return mapping.first == value; }) ==
mappings.cend()) {
return ValidationError::InvalidValue;
}
return {};
}
template <typename Enum>
std::optional<Enum> StringToEnum(const auto &value, const auto &mappings) {
const auto mapping_iter =
std::find_if(mappings.begin(), mappings.end(), [&](const auto &mapping) { return mapping.first == value; });
if (mapping_iter == mappings.cend()) {
return std::nullopt;
}
return mapping_iter->second;
}
} // namespace
// Short help flag.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_HIDDEN_bool(h, false, "Print usage and exit.");
@@ -227,6 +192,20 @@ DEFINE_VALIDATED_uint64(storage_wal_file_flush_every_n_tx,
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_snapshot_on_exit, false, "Controls whether the storage creates another snapshot on exit.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(storage_items_per_batch, memgraph::storage::Config::Durability().items_per_batch,
"The number of edges and vertices stored in a batch in a snapshot file.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(storage_parallel_index_recovery, false,
"Controls whether the index creation can be done in a multithreaded fashion.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_uint64(storage_recovery_thread_count,
std::max(static_cast<uint64_t>(std::thread::hardware_concurrency()),
memgraph::storage::Config::Durability().recovery_thread_count),
"The number of threads used to recover persisted data from disk.");
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_bool(telemetry_enabled, false,
"Set to true to enable telemetry. We collect information about the "
@@ -290,21 +269,21 @@ inline constexpr std::array isolation_level_mappings{
const std::string isolation_level_help_string =
fmt::format("Default isolation level used for the transactions. Allowed values: {}",
GetAllowedEnumValuesString(isolation_level_mappings));
memgraph::utils::GetAllowedEnumValuesString(isolation_level_mappings));
} // namespace
// NOLINTNEXTLINE (cppcoreguidelines-avoid-non-const-global-variables)
DEFINE_VALIDATED_string(isolation_level, "SNAPSHOT_ISOLATION", isolation_level_help_string.c_str(), {
if (const auto result = IsValidEnumValueString(value, isolation_level_mappings); result.HasError()) {
if (const auto result = memgraph::utils::IsValidEnumValueString(value, isolation_level_mappings); result.HasError()) {
const auto error = result.GetError();
switch (error) {
case ValidationError::EmptyValue: {
case memgraph::utils::ValidationError::EmptyValue: {
std::cout << "Isolation level cannot be empty." << std::endl;
break;
}
case ValidationError::InvalidValue: {
case memgraph::utils::ValidationError::InvalidValue: {
std::cout << "Invalid value for isolation level. Allowed values: "
<< GetAllowedEnumValuesString(isolation_level_mappings) << std::endl;
<< memgraph::utils::GetAllowedEnumValuesString(isolation_level_mappings) << std::endl;
break;
}
}
@@ -317,7 +296,7 @@ DEFINE_VALIDATED_string(isolation_level, "SNAPSHOT_ISOLATION", isolation_level_h
namespace {
memgraph::storage::IsolationLevel ParseIsolationLevel() {
const auto isolation_level =
StringToEnum<memgraph::storage::IsolationLevel>(FLAGS_isolation_level, isolation_level_mappings);
memgraph::utils::StringToEnum<memgraph::storage::IsolationLevel>(FLAGS_isolation_level, isolation_level_mappings);
MG_ASSERT(isolation_level, "Invalid isolation level");
return *isolation_level;
}
@@ -377,21 +356,21 @@ inline constexpr std::array log_level_mappings{
std::pair{"INFO"sv, spdlog::level::info}, std::pair{"WARNING"sv, spdlog::level::warn},
std::pair{"ERROR"sv, spdlog::level::err}, std::pair{"CRITICAL"sv, spdlog::level::critical}};
const std::string log_level_help_string =
fmt::format("Minimum log level. Allowed values: {}", GetAllowedEnumValuesString(log_level_mappings));
const std::string log_level_help_string = fmt::format("Minimum log level. Allowed values: {}",
memgraph::utils::GetAllowedEnumValuesString(log_level_mappings));
} // namespace
DEFINE_VALIDATED_string(log_level, "WARNING", log_level_help_string.c_str(), {
if (const auto result = IsValidEnumValueString(value, log_level_mappings); result.HasError()) {
if (const auto result = memgraph::utils::IsValidEnumValueString(value, log_level_mappings); result.HasError()) {
const auto error = result.GetError();
switch (error) {
case ValidationError::EmptyValue: {
case memgraph::utils::ValidationError::EmptyValue: {
std::cout << "Log level cannot be empty." << std::endl;
break;
}
case ValidationError::InvalidValue: {
std::cout << "Invalid value for log level. Allowed values: " << GetAllowedEnumValuesString(log_level_mappings)
<< std::endl;
case memgraph::utils::ValidationError::InvalidValue: {
std::cout << "Invalid value for log level. Allowed values: "
<< memgraph::utils::GetAllowedEnumValuesString(log_level_mappings) << std::endl;
break;
}
}
@@ -403,7 +382,7 @@ DEFINE_VALIDATED_string(log_level, "WARNING", log_level_help_string.c_str(), {
namespace {
spdlog::level::level_enum ParseLogLevel() {
const auto log_level = StringToEnum<spdlog::level::level_enum>(FLAGS_log_level, log_level_mappings);
const auto log_level = memgraph::utils::StringToEnum<spdlog::level::level_enum>(FLAGS_log_level, log_level_mappings);
MG_ASSERT(log_level, "Invalid log level");
return *log_level;
}
@@ -522,6 +501,7 @@ class BoltSession final : public memgraph::communication::bolt::Session<memgraph
: memgraph::communication::bolt::Session<memgraph::communication::v2::InputStream,
memgraph::communication::v2::OutputStream>(input_stream, output_stream),
db_(data->db),
interpreter_context_(data->interpreter_context),
interpreter_(data->interpreter_context),
auth_(data->auth),
#if MG_ENTERPRISE
@@ -529,6 +509,11 @@ class BoltSession final : public memgraph::communication::bolt::Session<memgraph
#endif
endpoint_(endpoint),
run_id_(data->run_id) {
interpreter_context_->interpreters.WithLock([this](auto &interpreters) { interpreters.insert(&interpreter_); });
}
~BoltSession() override {
interpreter_context_->interpreters.WithLock([this](auto &interpreters) { interpreters.erase(&interpreter_); });
}
using memgraph::communication::bolt::Session<memgraph::communication::v2::InputStream,
@@ -674,6 +659,7 @@ class BoltSession final : public memgraph::communication::bolt::Session<memgraph
// NOTE: Needed only for ToBoltValue conversions
const memgraph::storage::Storage *db_;
memgraph::query::InterpreterContext *interpreter_context_;
memgraph::query::Interpreter interpreter_;
memgraph::utils::Synchronized<memgraph::auth::Auth, memgraph::utils::WritePrioritizedRWLock> *auth_;
std::optional<memgraph::auth::User> user_;
@@ -880,7 +866,10 @@ int main(int argc, char **argv) {
.wal_file_size_kibibytes = FLAGS_storage_wal_file_size_kib,
.wal_file_flush_every_n_tx = FLAGS_storage_wal_file_flush_every_n_tx,
.snapshot_on_exit = FLAGS_storage_snapshot_on_exit,
.restore_replicas_on_startup = true},
.restore_replicas_on_startup = true,
.items_per_batch = FLAGS_storage_items_per_batch,
.recovery_thread_count = FLAGS_storage_recovery_thread_count,
.allow_parallel_index_creation = FLAGS_storage_parallel_index_recovery},
.transaction = {.isolation_level = ParseIsolationLevel()}};
if (FLAGS_storage_snapshot_interval_sec == 0) {
if (FLAGS_storage_wal_enabled) {

View File

@@ -1,13 +1,7 @@
define_add_lcp(add_lcp_query lcp_query_cpp_files generated_lcp_query_files)
add_lcp_query(frontend/ast/ast.lcp)
add_lcp_query(frontend/semantic/symbol.lcp)
add_lcp_query(plan/operator.lcp)
add_custom_target(generate_lcp_query DEPENDS ${generated_lcp_query_files})
set(mg_query_sources
${lcp_query_cpp_files}
frontend/ast/ast.cpp
frontend/semantic/symbol.cpp
plan/operator_type_info.cpp
common.cpp
cypher_query_interpreter.cpp
dump.cpp
@@ -46,7 +40,6 @@ set(mg_query_sources
find_package(Boost REQUIRED)
add_library(mg-query STATIC ${mg_query_sources})
add_dependencies(mg-query generate_lcp_query)
target_include_directories(mg-query PUBLIC ${CMAKE_SOURCE_DIR}/include)
target_link_libraries(mg-query dl cppitertools Boost::headers)
target_link_libraries(mg-query mg-integrations-pulsar mg-integrations-kafka mg-storage-v2 mg-license mg-utils mg-kvstore mg-memory)

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -107,5 +107,37 @@ storage::PropertyValue PropsSetChecked(T *record, const storage::PropertyId &key
}
}
template <typename T>
concept AccessorWithInitProperties = requires(T accessor,
const std::map<storage::PropertyId, storage::PropertyValue> &properties) {
{ accessor.InitProperties(properties) } -> std::same_as<storage::Result<bool>>;
};
/// Set property `values` mapped with given `key` on a `record`.
///
/// @throw QueryRuntimeException if value cannot be set as a property value
template <AccessorWithInitProperties T>
bool MultiPropsInitChecked(T *record, std::map<storage::PropertyId, storage::PropertyValue> &properties) {
try {
auto maybe_values = record->InitProperties(properties);
if (maybe_values.HasError()) {
switch (maybe_values.GetError()) {
case storage::Error::SERIALIZATION_ERROR:
throw TransactionSerializationException();
case storage::Error::DELETED_OBJECT:
throw QueryRuntimeException("Trying to set properties on a deleted object.");
case storage::Error::PROPERTIES_DISABLED:
throw QueryRuntimeException("Can't set property because properties on edges are disabled.");
case storage::Error::VERTEX_HAS_EDGES:
case storage::Error::NONEXISTENT_OBJECT:
throw QueryRuntimeException("Unexpected error when setting a property.");
}
}
return std::move(*maybe_values);
} catch (const TypedValueException &) {
throw QueryRuntimeException("Cannot set properties.");
}
}
int64_t QueryTimestamp();
} // namespace memgraph::query

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -16,4 +16,6 @@
namespace memgraph::query {
inline constexpr uint16_t kDefaultReplicationPort = 10000;
inline constexpr auto *kDefaultReplicationServerIp = "0.0.0.0";
inline const std::string kAsterisk = "*";
inline constexpr uint16_t kDeleteStatisticsNumResults = 6;
} // namespace memgraph::query

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -24,6 +24,15 @@
namespace memgraph::query {
enum class TransactionStatus {
IDLE,
ACTIVE,
VERIFYING,
TERMINATED,
STARTED_COMMITTING,
STARTED_ROLLBACK,
};
struct EvaluationContext {
/// Memory for allocations during evaluation of a *single* Pull call.
///
@@ -66,6 +75,7 @@ struct ExecutionContext {
SymbolTable symbol_table;
EvaluationContext evaluation_context;
std::atomic<bool> *is_shutting_down{nullptr};
std::atomic<TransactionStatus> *transaction_status{nullptr};
bool is_profile_query{false};
std::chrono::duration<double> profile_execution_time;
plan::ProfilingStats stats;
@@ -82,7 +92,9 @@ static_assert(std::is_move_assignable_v<ExecutionContext>, "ExecutionContext mus
static_assert(std::is_move_constructible_v<ExecutionContext>, "ExecutionContext must be move constructible!");
inline bool MustAbort(const ExecutionContext &context) noexcept {
return (context.is_shutting_down != nullptr && context.is_shutting_down->load(std::memory_order_acquire)) ||
return (context.transaction_status != nullptr &&
context.transaction_status->load(std::memory_order_acquire) == TransactionStatus::TERMINATED) ||
(context.is_shutting_down != nullptr && context.is_shutting_down->load(std::memory_order_acquire)) ||
context.timer.IsExpired();
}

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -71,6 +71,10 @@ class EdgeAccessor final {
return impl_.SetProperty(key, value);
}
storage::Result<bool> InitProperties(const std::map<storage::PropertyId, storage::PropertyValue> &properties) {
return impl_.InitProperties(properties);
}
storage::Result<storage::PropertyValue> RemoveProperty(storage::PropertyId key) {
return SetProperty(key, storage::PropertyValue());
}
@@ -125,6 +129,10 @@ class VertexAccessor final {
return impl_.SetProperty(key, value);
}
storage::Result<bool> InitProperties(const std::map<storage::PropertyId, storage::PropertyValue> &properties) {
return impl_.InitProperties(properties);
}
storage::Result<storage::PropertyValue> RemoveProperty(storage::PropertyId key) {
return SetProperty(key, storage::PropertyValue());
}
@@ -422,6 +430,25 @@ class DbAccessor final {
return accessor_->LabelPropertyIndexExists(label, prop);
}
std::optional<storage::IndexStats> GetIndexStats(const storage::LabelId &label,
const storage::PropertyId &property) const {
return accessor_->GetIndexStats(label, property);
}
std::vector<std::pair<storage::LabelId, storage::PropertyId>> ClearIndexStats() {
return accessor_->ClearIndexStats();
}
std::vector<std::pair<storage::LabelId, storage::PropertyId>> DeleteIndexStatsForLabels(
const std::span<std::string> labels) {
return accessor_->DeleteIndexStatsForLabels(labels);
}
void SetIndexStats(const storage::LabelId &label, const storage::PropertyId &property,
const storage::IndexStats &stats) {
accessor_->SetIndexStats(label, property, stats);
}
int64_t VerticesCount() const { return accessor_->ApproximateVertexCount(); }
int64_t VerticesCount(storage::LabelId label) const { return accessor_->ApproximateVertexCount(label); }

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -120,9 +120,8 @@ class HintedAbortError : public utils::BasicException {
using utils::BasicException::BasicException;
HintedAbortError()
: utils::BasicException(
"Transaction was asked to abort, most likely because it was "
"executing longer than time specified by "
"--query-execution-timeout-sec flag.") {}
"Transaction was asked to abort either because it was executing longer than time specified or another user "
"asked it to abort.") {}
};
class ExplicitTransactionUsageException : public QueryRuntimeException {
@@ -212,6 +211,21 @@ class IsolationLevelModificationInMulticommandTxException : public QueryExceptio
: QueryException("Isolation level cannot be modified in multicommand transactions.") {}
};
class IsolationLevelModificationInAnalyticsException : public QueryException {
public:
IsolationLevelModificationInAnalyticsException()
: QueryException(
"Isolation level cannot be modified when storage mode is set to IN_MEMORY_ANALYTICAL."
"IN_MEMORY_ANALYTICAL mode doesn't provide any isolation guarantees, "
"you can think about it as an equivalent to READ_UNCOMMITED.") {}
};
class StorageModeModificationInMulticommandTxException : public QueryException {
public:
StorageModeModificationInMulticommandTxException()
: QueryException("Storage mode cannot be modified in multicommand transactions.") {}
};
class CreateSnapshotInMulticommandTxException final : public QueryException {
public:
CreateSnapshotInMulticommandTxException()
@@ -230,6 +244,12 @@ class VersionInfoInMulticommandTxException : public QueryException {
: QueryException("Version info query not allowed in multicommand transactions.") {}
};
class AnalyzeGraphInMulticommandTxException : public QueryException {
public:
AnalyzeGraphInMulticommandTxException()
: QueryException("Analyze graph query not allowed in multicommand transactions.") {}
};
class ReplicationException : public utils::BasicException {
public:
using utils::BasicException::BasicException;
@@ -237,4 +257,11 @@ class ReplicationException : public utils::BasicException {
: utils::BasicException("Replication Exception: {} Check the status of the replicas using 'SHOW REPLICA' query.",
message) {}
};
class TransactionQueueInMulticommandTxException : public QueryException {
public:
TransactionQueueInMulticommandTxException()
: QueryException("Transaction queue queries not allowed in multicommand transactions.") {}
};
} // namespace memgraph::query

View File

@@ -0,0 +1,276 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "query/frontend/ast/ast.hpp"
#include "query/frontend/ast/ast_visitor.hpp"
#include "utils/typeinfo.hpp"
namespace memgraph {
constexpr utils::TypeInfo query::LabelIx::kType{utils::TypeId::AST_LABELIX, "LabelIx", nullptr};
constexpr utils::TypeInfo query::PropertyIx::kType{utils::TypeId::AST_PROPERTYIX, "PropertyIx", nullptr};
constexpr utils::TypeInfo query::EdgeTypeIx::kType{utils::TypeId::AST_EDGETYPEIX, "EdgeTypeIx", nullptr};
constexpr utils::TypeInfo query::Tree::kType{utils::TypeId::AST_TREE, "Tree", nullptr};
constexpr utils::TypeInfo query::Expression::kType{utils::TypeId::AST_EXPRESSION, "Expression", &query::Tree::kType};
constexpr utils::TypeInfo query::Where::kType{utils::TypeId::AST_WHERE, "Where", &query::Tree::kType};
constexpr utils::TypeInfo query::BinaryOperator::kType{utils::TypeId::AST_BINARY_OPERATOR, "BinaryOperator",
&query::Expression::kType};
constexpr utils::TypeInfo query::UnaryOperator::kType{utils::TypeId::AST_UNARY_OPERATOR, "UnaryOperator",
&query::Expression::kType};
constexpr utils::TypeInfo query::OrOperator::kType{utils::TypeId::AST_OR_OPERATOR, "OrOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::XorOperator::kType{utils::TypeId::AST_XOR_OPERATOR, "XorOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::AndOperator::kType{utils::TypeId::AST_AND_OPERATOR, "AndOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::AdditionOperator::kType{utils::TypeId::AST_ADDITION_OPERATOR, "AdditionOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::SubtractionOperator::kType{utils::TypeId::AST_SUBTRACTION_OPERATOR,
"SubtractionOperator", &query::BinaryOperator::kType};
constexpr utils::TypeInfo query::MultiplicationOperator::kType{utils::TypeId::AST_MULTIPLICATION_OPERATOR,
"MultiplicationOperator", &query::BinaryOperator::kType};
constexpr utils::TypeInfo query::DivisionOperator::kType{utils::TypeId::AST_DIVISION_OPERATOR, "DivisionOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::ModOperator::kType{utils::TypeId::AST_MOD_OPERATOR, "ModOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::NotEqualOperator::kType{utils::TypeId::AST_NOT_EQUAL_OPERATOR, "NotEqualOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::EqualOperator::kType{utils::TypeId::AST_EQUAL_OPERATOR, "EqualOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::LessOperator::kType{utils::TypeId::AST_LESS_OPERATOR, "LessOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::GreaterOperator::kType{utils::TypeId::AST_GREATER_OPERATOR, "GreaterOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::LessEqualOperator::kType{utils::TypeId::AST_LESS_EQUAL_OPERATOR, "LessEqualOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::GreaterEqualOperator::kType{utils::TypeId::AST_GREATER_EQUAL_OPERATOR,
"GreaterEqualOperator", &query::BinaryOperator::kType};
constexpr utils::TypeInfo query::InListOperator::kType{utils::TypeId::AST_IN_LIST_OPERATOR, "InListOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::SubscriptOperator::kType{utils::TypeId::AST_SUBSCRIPT_OPERATOR, "SubscriptOperator",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::NotOperator::kType{utils::TypeId::AST_NOT_OPERATOR, "NotOperator",
&query::UnaryOperator::kType};
constexpr utils::TypeInfo query::UnaryPlusOperator::kType{utils::TypeId::AST_UNARY_PLUS_OPERATOR, "UnaryPlusOperator",
&query::UnaryOperator::kType};
constexpr utils::TypeInfo query::UnaryMinusOperator::kType{utils::TypeId::AST_UNARY_MINUS_OPERATOR,
"UnaryMinusOperator", &query::UnaryOperator::kType};
constexpr utils::TypeInfo query::IsNullOperator::kType{utils::TypeId::AST_IS_NULL_OPERATOR, "IsNullOperator",
&query::UnaryOperator::kType};
constexpr utils::TypeInfo query::Aggregation::kType{utils::TypeId::AST_AGGREGATION, "Aggregation",
&query::BinaryOperator::kType};
constexpr utils::TypeInfo query::ListSlicingOperator::kType{utils::TypeId::AST_LIST_SLICING_OPERATOR,
"ListSlicingOperator", &query::Expression::kType};
constexpr utils::TypeInfo query::IfOperator::kType{utils::TypeId::AST_IF_OPERATOR, "IfOperator",
&query::Expression::kType};
constexpr utils::TypeInfo query::BaseLiteral::kType{utils::TypeId::AST_BASE_LITERAL, "BaseLiteral",
&query::Expression::kType};
constexpr utils::TypeInfo query::PrimitiveLiteral::kType{utils::TypeId::AST_PRIMITIVE_LITERAL, "PrimitiveLiteral",
&query::BaseLiteral::kType};
constexpr utils::TypeInfo query::ListLiteral::kType{utils::TypeId::AST_LIST_LITERAL, "ListLiteral",
&query::BaseLiteral::kType};
constexpr utils::TypeInfo query::MapLiteral::kType{utils::TypeId::AST_MAP_LITERAL, "MapLiteral",
&query::BaseLiteral::kType};
constexpr utils::TypeInfo query::Identifier::kType{utils::TypeId::AST_IDENTIFIER, "Identifier",
&query::Expression::kType};
constexpr utils::TypeInfo query::PropertyLookup::kType{utils::TypeId::AST_PROPERTY_LOOKUP, "PropertyLookup",
&query::Expression::kType};
constexpr utils::TypeInfo query::LabelsTest::kType{utils::TypeId::AST_LABELS_TEST, "LabelsTest",
&query::Expression::kType};
constexpr utils::TypeInfo query::Function::kType{utils::TypeId::AST_FUNCTION, "Function", &query::Expression::kType};
constexpr utils::TypeInfo query::Reduce::kType{utils::TypeId::AST_REDUCE, "Reduce", &query::Expression::kType};
constexpr utils::TypeInfo query::Coalesce::kType{utils::TypeId::AST_COALESCE, "Coalesce", &query::Expression::kType};
constexpr utils::TypeInfo query::Extract::kType{utils::TypeId::AST_EXTRACT, "Extract", &query::Expression::kType};
constexpr utils::TypeInfo query::All::kType{utils::TypeId::AST_ALL, "All", &query::Expression::kType};
constexpr utils::TypeInfo query::Single::kType{utils::TypeId::AST_SINGLE, "Single", &query::Expression::kType};
constexpr utils::TypeInfo query::Any::kType{utils::TypeId::AST_ANY, "Any", &query::Expression::kType};
constexpr utils::TypeInfo query::None::kType{utils::TypeId::AST_NONE, "None", &query::Expression::kType};
constexpr utils::TypeInfo query::ParameterLookup::kType{utils::TypeId::AST_PARAMETER_LOOKUP, "ParameterLookup",
&query::Expression::kType};
constexpr utils::TypeInfo query::RegexMatch::kType{utils::TypeId::AST_REGEX_MATCH, "RegexMatch",
&query::Expression::kType};
constexpr utils::TypeInfo query::NamedExpression::kType{utils::TypeId::AST_NAMED_EXPRESSION, "NamedExpression",
&query::Tree::kType};
constexpr utils::TypeInfo query::PatternAtom::kType{utils::TypeId::AST_PATTERN_ATOM, "PatternAtom",
&query::Tree::kType};
constexpr utils::TypeInfo query::NodeAtom::kType{utils::TypeId::AST_NODE_ATOM, "NodeAtom", &query::PatternAtom::kType};
constexpr utils::TypeInfo query::EdgeAtom::Lambda::kType{utils::TypeId::AST_EDGE_ATOM_LAMBDA, "Lambda", nullptr};
constexpr utils::TypeInfo query::EdgeAtom::kType{utils::TypeId::AST_EDGE_ATOM, "EdgeAtom", &query::PatternAtom::kType};
constexpr utils::TypeInfo query::Pattern::kType{utils::TypeId::AST_PATTERN, "Pattern", &query::Tree::kType};
constexpr utils::TypeInfo query::Clause::kType{utils::TypeId::AST_CLAUSE, "Clause", &query::Tree::kType};
constexpr utils::TypeInfo query::SingleQuery::kType{utils::TypeId::AST_SINGLE_QUERY, "SingleQuery",
&query::Tree::kType};
constexpr utils::TypeInfo query::CypherUnion::kType{utils::TypeId::AST_CYPHER_UNION, "CypherUnion",
&query::Tree::kType};
constexpr utils::TypeInfo query::Query::kType{utils::TypeId::AST_QUERY, "Query", &query::Tree::kType};
constexpr utils::TypeInfo query::CypherQuery::kType{utils::TypeId::AST_CYPHER_QUERY, "CypherQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::ExplainQuery::kType{utils::TypeId::AST_EXPLAIN_QUERY, "ExplainQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::ProfileQuery::kType{utils::TypeId::AST_PROFILE_QUERY, "ProfileQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::IndexQuery::kType{utils::TypeId::AST_INDEX_QUERY, "IndexQuery", &query::Query::kType};
constexpr utils::TypeInfo query::Create::kType{utils::TypeId::AST_CREATE, "Create", &query::Clause::kType};
constexpr utils::TypeInfo query::CallProcedure::kType{utils::TypeId::AST_CALL_PROCEDURE, "CallProcedure",
&query::Clause::kType};
constexpr utils::TypeInfo query::Match::kType{utils::TypeId::AST_MATCH, "Match", &query::Clause::kType};
constexpr utils::TypeInfo query::SortItem::kType{utils::TypeId::AST_SORT_ITEM, "SortItem", nullptr};
constexpr utils::TypeInfo query::ReturnBody::kType{utils::TypeId::AST_RETURN_BODY, "ReturnBody", nullptr};
constexpr utils::TypeInfo query::Return::kType{utils::TypeId::AST_RETURN, "Return", &query::Clause::kType};
constexpr utils::TypeInfo query::With::kType{utils::TypeId::AST_WITH, "With", &query::Clause::kType};
constexpr utils::TypeInfo query::Delete::kType{utils::TypeId::AST_DELETE, "Delete", &query::Clause::kType};
constexpr utils::TypeInfo query::SetProperty::kType{utils::TypeId::AST_SET_PROPERTY, "SetProperty",
&query::Clause::kType};
constexpr utils::TypeInfo query::SetProperties::kType{utils::TypeId::AST_SET_PROPERTIES, "SetProperties",
&query::Clause::kType};
constexpr utils::TypeInfo query::SetLabels::kType{utils::TypeId::AST_SET_LABELS, "SetLabels", &query::Clause::kType};
constexpr utils::TypeInfo query::RemoveProperty::kType{utils::TypeId::AST_REMOVE_PROPERTY, "RemoveProperty",
&query::Clause::kType};
constexpr utils::TypeInfo query::RemoveLabels::kType{utils::TypeId::AST_REMOVE_LABELS, "RemoveLabels",
&query::Clause::kType};
constexpr utils::TypeInfo query::Merge::kType{utils::TypeId::AST_MERGE, "Merge", &query::Clause::kType};
constexpr utils::TypeInfo query::Unwind::kType{utils::TypeId::AST_UNWIND, "Unwind", &query::Clause::kType};
constexpr utils::TypeInfo query::AuthQuery::kType{utils::TypeId::AST_AUTH_QUERY, "AuthQuery", &query::Query::kType};
constexpr utils::TypeInfo query::InfoQuery::kType{utils::TypeId::AST_INFO_QUERY, "InfoQuery", &query::Query::kType};
constexpr utils::TypeInfo query::Constraint::kType{utils::TypeId::AST_CONSTRAINT, "Constraint", nullptr};
constexpr utils::TypeInfo query::ConstraintQuery::kType{utils::TypeId::AST_CONSTRAINT_QUERY, "ConstraintQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::DumpQuery::kType{utils::TypeId::AST_DUMP_QUERY, "DumpQuery", &query::Query::kType};
constexpr utils::TypeInfo query::ReplicationQuery::kType{utils::TypeId::AST_REPLICATION_QUERY, "ReplicationQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::LockPathQuery::kType{utils::TypeId::AST_LOCK_PATH_QUERY, "LockPathQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::LoadCsv::kType{utils::TypeId::AST_LOAD_CSV, "LoadCsv", &query::Clause::kType};
constexpr utils::TypeInfo query::FreeMemoryQuery::kType{utils::TypeId::AST_FREE_MEMORY_QUERY, "FreeMemoryQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::TriggerQuery::kType{utils::TypeId::AST_TRIGGER_QUERY, "TriggerQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::IsolationLevelQuery::kType{utils::TypeId::AST_ISOLATION_LEVEL_QUERY,
"IsolationLevelQuery", &query::Query::kType};
constexpr utils::TypeInfo query::StorageModeQuery::kType{utils::TypeId::AST_STORAGE_MODE_QUERY, "StorageModeQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::CreateSnapshotQuery::kType{utils::TypeId::AST_CREATE_SNAPSHOT_QUERY,
"CreateSnapshotQuery", &query::Query::kType};
constexpr utils::TypeInfo query::StreamQuery::kType{utils::TypeId::AST_STREAM_QUERY, "StreamQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::SettingQuery::kType{utils::TypeId::AST_SETTING_QUERY, "SettingQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::VersionQuery::kType{utils::TypeId::AST_VERSION_QUERY, "VersionQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::Foreach::kType{utils::TypeId::AST_FOREACH, "Foreach", &query::Clause::kType};
constexpr utils::TypeInfo query::ShowConfigQuery::kType{utils::TypeId::AST_SHOW_CONFIG_QUERY, "ShowConfigQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::AnalyzeGraphQuery::kType{utils::TypeId::AST_ANALYZE_GRAPH_QUERY, "AnalyzeGraphQuery",
&query::Query::kType};
constexpr utils::TypeInfo query::TransactionQueueQuery::kType{utils::TypeId::AST_TRANSACTION_QUEUE_QUERY,
"TransactionQueueQuery", &query::Query::kType};
constexpr utils::TypeInfo query::Exists::kType{utils::TypeId::AST_EXISTS, "Exists", &query::Expression::kType};
constexpr utils::TypeInfo query::CallSubquery::kType{utils::TypeId::AST_CALL_SUBQUERY, "CallSubquery",
&query::Clause::kType};
} // namespace memgraph

File diff suppressed because it is too large Load Diff

View File

@@ -301,6 +301,7 @@ cpp<#
(lcp:define-class expression (tree "::utils::Visitable<HierarchicalTreeVisitor>"
"::utils::Visitable<ExpressionVisitor<TypedValue>>"
"::utils::Visitable<ExpressionVisitor<TypedValue*>>"
"::utils::Visitable<ExpressionVisitor<void>>")
()
(:abstractp t)
@@ -308,6 +309,7 @@ cpp<#
#>cpp
using utils::Visitable<HierarchicalTreeVisitor>::Accept;
using utils::Visitable<ExpressionVisitor<TypedValue>>::Accept;
using utils::Visitable<ExpressionVisitor<TypedValue*>>::Accept;
using utils::Visitable<ExpressionVisitor<void>>::Accept;
Expression() = default;
@@ -407,6 +409,7 @@ cpp<#
(:public
#>cpp
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -438,6 +441,7 @@ cpp<#
(:public
#>cpp
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -485,6 +489,7 @@ cpp<#
}
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -538,6 +543,7 @@ cpp<#
ListSlicingOperator() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -581,6 +587,7 @@ cpp<#
IfOperator() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -628,6 +635,7 @@ cpp<#
PrimitiveLiteral() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
DEFVISITABLE(HierarchicalTreeVisitor);
cpp<#)
@@ -656,6 +664,7 @@ cpp<#
ListLiteral() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -688,6 +697,7 @@ cpp<#
MapLiteral() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -720,6 +730,7 @@ cpp<#
Identifier() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
DEFVISITABLE(HierarchicalTreeVisitor);
@@ -757,6 +768,7 @@ cpp<#
PropertyLookup() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -797,6 +809,7 @@ cpp<#
LabelsTest() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -837,6 +850,7 @@ cpp<#
Function() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -892,6 +906,7 @@ cpp<#
Reduce() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -930,6 +945,7 @@ cpp<#
Coalesce() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -969,6 +985,7 @@ cpp<#
Extract() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1005,6 +1022,7 @@ cpp<#
All() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1046,6 +1064,7 @@ cpp<#
Single() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1087,6 +1106,7 @@ cpp<#
Any() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1128,6 +1148,7 @@ cpp<#
None() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1159,6 +1180,7 @@ cpp<#
ParameterLookup() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
DEFVISITABLE(HierarchicalTreeVisitor);
cpp<#)
@@ -1186,6 +1208,7 @@ cpp<#
RegexMatch() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1205,6 +1228,7 @@ cpp<#
(lcp:define-class named-expression (tree "::utils::Visitable<HierarchicalTreeVisitor>"
"::utils::Visitable<ExpressionVisitor<TypedValue>>"
"::utils::Visitable<ExpressionVisitor<TypedValue*>>"
"::utils::Visitable<ExpressionVisitor<void>>")
((name "std::string" :scope :public)
(expression "Expression *" :initval "nullptr" :scope :public
@@ -1213,7 +1237,9 @@ cpp<#
(token-position :int32_t :initval -1 :scope :public
:documentation "This field contains token position of first token in named expression used to create name_. If NamedExpression object is not created from query or it is aliased leave this value at -1.")
(symbol-pos :int32_t :initval -1 :scope :public
:documentation "Symbol table position of the symbol this NamedExpression is mapped to."))
:documentation "Symbol table position of the symbol this NamedExpression is mapped to.")
(is-aliased :bool :initval "false" :scope :public
:documentation "True if the variable is aliased"))
(:public
#>cpp
using utils::Visitable<ExpressionVisitor<TypedValue>>::Accept;
@@ -1223,6 +1249,7 @@ cpp<#
NamedExpression() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1595,7 +1622,7 @@ cpp<#
(:clone :ignore-other-base-classes t)
(:type-info :ignore-other-base-classes t))
(lcp:define-class cypher-query (query)
(lcp:define-class cypher-query (query "::utils::Visitable<HierarchicalTreeVisitor>")
((single-query "SingleQuery *" :initval "nullptr" :scope :public
:slk-save #'slk-save-ast-pointer
:slk-load (slk-load-ast-pointer "SingleQuery")
@@ -1614,13 +1641,25 @@ cpp<#
CypherQuery() = default;
DEFVISITABLE(QueryVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
single_query_->Accept(visitor);
for (auto *cypher_union : cypher_unions_) {
cypher_union->Accept(visitor);
}
}
return visitor.PostVisit(*this);
}
cpp<#)
(:private
#>cpp
friend class AstStorage;
cpp<#)
(:serialize (:slk))
(:clone))
(:serialize (:slk :ignore-other-base-classes t))
(:clone :ignore-other-base-classes t)
(:type-info :ignore-other-base-classes t))
(lcp:define-class explain-query (query)
((cypher-query "CypherQuery *" :initval "nullptr" :scope :public
@@ -2259,7 +2298,7 @@ cpp<#
(lcp:define-enum privilege
(create delete match merge set remove index stats auth constraint
dump replication durability read_file free_memory trigger config stream module_read module_write
websocket)
websocket transaction_management)
(:serialize))
(lcp:define-enum fine-grained-privilege
(nothing read update create_delete)
@@ -2308,7 +2347,7 @@ const std::vector<AuthQuery::Privilege> kPrivilegesAll = {
AuthQuery::Privilege::FREE_MEMORY, AuthQuery::Privilege::TRIGGER,
AuthQuery::Privilege::CONFIG, AuthQuery::Privilege::STREAM,
AuthQuery::Privilege::MODULE_READ, AuthQuery::Privilege::MODULE_WRITE,
AuthQuery::Privilege::WEBSOCKET};
AuthQuery::Privilege::WEBSOCKET, AuthQuery::Privilege::TRANSACTION_MANAGEMENT};
cpp<#
(lcp:define-class info-query (query)
@@ -2378,6 +2417,26 @@ cpp<#
(:serialize (:slk))
(:clone))
(lcp:define-class analyze-graph-query (query)
((action "Action" :scope :public)
(labels "std::vector<std::string>" :scope :public))
(:public
(lcp:define-enum action
(analyze delete)
(:serialize))
#>cpp
AnalyzeGraphQuery() = default;
DEFVISITABLE(QueryVisitor<void>);
cpp<#)
(:private
#>cpp
friend class AstStorage;
cpp<#)
(:serialize (:slk))
(:clone))
(lcp:define-class replication-query (query)
((action "Action" :scope :public)
(role "ReplicationRole" :scope :public)
@@ -2636,6 +2695,26 @@ cpp<#
(:serialize (:slk))
(:clone))
(lcp:define-class transaction-queue-query (query)
((action "Action" :scope :public)
(transaction_id_list "std::vector<Expression*>" :scope :public))
(:public
(lcp:define-enum action
(show-transactions terminate-transactions)
(:serialize))
#>cpp
TransactionQueueQuery() = default;
DEFVISITABLE(QueryVisitor<void>);
cpp<#)
(:private
#>cpp
friend class AstStorage;
cpp<#)
(:serialize (:slk))
(:clone))
(lcp:define-class version-query (query) ()
(:public
#>cpp
@@ -2686,5 +2765,64 @@ cpp<#
(:serialize (:slk))
(:clone))
(lcp:define-class exists (expression)
((pattern "Pattern *" :initval "nullptr" :scope :public
:slk-save #'slk-save-ast-pointer
:slk-load (slk-load-ast-pointer "Pattern"))
(symbol-pos :int32_t :initval -1 :scope :public
:documentation "Symbol table position of the symbol this Aggregation is mapped to."))
(:public
#>cpp
Exists() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
pattern_->Accept(visitor);
}
return visitor.PostVisit(*this);
}
Exists *MapTo(const Symbol &symbol) {
symbol_pos_ = symbol.position();
return this;
}
cpp<#)
(:protected
#>cpp
Exists(Pattern * pattern) : pattern_(pattern) {}
cpp<#)
(:private
#>cpp
friend class AstStorage;
cpp<#)
(:serialize (:slk))
(:clone))
(lcp:define-class call-subquery (clause)
((cypher-query "CypherQuery *" :scope :public
:slk-save #'slk-save-ast-pointer
:slk-load (slk-load-ast-pointer "CypherQuery")))
(:public
#>cpp
CallSubquery() = default;
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
cypher_query_->Accept(visitor);
}
return visitor.PostVisit(*this);
}
cpp<#)
(:private
#>cpp
friend class AstStorage;
cpp<#)
(:serialize (:slk))
(:clone))
(lcp:pop-namespace) ;; namespace query
(lcp:pop-namespace) ;; namespace memgraph

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -89,12 +89,17 @@ class LoadCsv;
class FreeMemoryQuery;
class TriggerQuery;
class IsolationLevelQuery;
class StorageModeQuery;
class CreateSnapshotQuery;
class StreamQuery;
class SettingQuery;
class VersionQuery;
class Foreach;
class ShowConfigQuery;
class CallSubquery;
class AnalyzeGraphQuery;
class TransactionQueueQuery;
class Exists;
using TreeCompositeVisitor = utils::CompositeVisitor<
SingleQuery, CypherUnion, NamedExpression, OrOperator, XorOperator, AndOperator, NotOperator, AdditionOperator,
@@ -103,7 +108,7 @@ using TreeCompositeVisitor = utils::CompositeVisitor<
ListSlicingOperator, IfOperator, UnaryPlusOperator, UnaryMinusOperator, IsNullOperator, ListLiteral, MapLiteral,
PropertyLookup, LabelsTest, Aggregation, Function, Reduce, Coalesce, Extract, All, Single, Any, None, CallProcedure,
Create, Match, Return, With, Pattern, NodeAtom, EdgeAtom, Delete, Where, SetProperty, SetProperties, SetLabels,
RemoveProperty, RemoveLabels, Merge, Unwind, RegexMatch, LoadCsv, Foreach>;
RemoveProperty, RemoveLabels, Merge, Unwind, RegexMatch, LoadCsv, Foreach, Exists, CallSubquery, CypherQuery>;
using TreeLeafVisitor = utils::LeafVisitor<Identifier, PrimitiveLiteral, ParameterLookup>;
@@ -123,12 +128,13 @@ class ExpressionVisitor
LessOperator, GreaterOperator, LessEqualOperator, GreaterEqualOperator, InListOperator, SubscriptOperator,
ListSlicingOperator, IfOperator, UnaryPlusOperator, UnaryMinusOperator, IsNullOperator, ListLiteral,
MapLiteral, PropertyLookup, LabelsTest, Aggregation, Function, Reduce, Coalesce, Extract, All, Single, Any,
None, ParameterLookup, Identifier, PrimitiveLiteral, RegexMatch> {};
None, ParameterLookup, Identifier, PrimitiveLiteral, RegexMatch, Exists> {};
template <class TResult>
class QueryVisitor : public utils::Visitor<TResult, CypherQuery, ExplainQuery, ProfileQuery, IndexQuery, AuthQuery,
InfoQuery, ConstraintQuery, DumpQuery, ReplicationQuery, LockPathQuery,
FreeMemoryQuery, TriggerQuery, IsolationLevelQuery, CreateSnapshotQuery,
StreamQuery, SettingQuery, VersionQuery, ShowConfigQuery> {};
class QueryVisitor
: public utils::Visitor<TResult, CypherQuery, ExplainQuery, ProfileQuery, IndexQuery, AuthQuery, InfoQuery,
ConstraintQuery, DumpQuery, ReplicationQuery, LockPathQuery, FreeMemoryQuery, TriggerQuery,
IsolationLevelQuery, CreateSnapshotQuery, StreamQuery, SettingQuery, VersionQuery,
ShowConfigQuery, TransactionQueueQuery, StorageModeQuery, AnalyzeGraphQuery> {};
} // namespace memgraph::query

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -11,8 +11,10 @@
#include "query/frontend/ast/cypher_main_visitor.hpp"
#include <support/Any.h>
#include <tree/ParseTreeVisitor.h>
#include <algorithm>
#include <any>
#include <climits>
#include <codecvt>
#include <cstring>
@@ -239,6 +241,23 @@ antlrcpp::Any CypherMainVisitor::visitDumpQuery(MemgraphCypher::DumpQueryContext
return dump_query;
}
antlrcpp::Any CypherMainVisitor::visitAnalyzeGraphQuery(MemgraphCypher::AnalyzeGraphQueryContext *ctx) {
auto *analyze_graph_query = storage_->Create<AnalyzeGraphQuery>();
if (ctx->listOfColonSymbolicNames()) {
analyze_graph_query->labels_ =
std::any_cast<std::vector<std::string>>(ctx->listOfColonSymbolicNames()->accept(this));
} else {
analyze_graph_query->labels_.emplace_back("*");
}
if (ctx->DELETE()) {
analyze_graph_query->action_ = AnalyzeGraphQuery::Action::DELETE;
} else {
analyze_graph_query->action_ = AnalyzeGraphQuery::Action::ANALYZE;
}
query_ = analyze_graph_query;
return analyze_graph_query;
}
antlrcpp::Any CypherMainVisitor::visitReplicationQuery(MemgraphCypher::ReplicationQueryContext *ctx) {
MG_ASSERT(ctx->children.size() == 1, "ReplicationQuery should have exactly one child!");
auto *replication_query = std::any_cast<ReplicationQuery *>(ctx->children[0]->accept(this));
@@ -468,6 +487,20 @@ antlrcpp::Any CypherMainVisitor::visitIsolationLevelQuery(MemgraphCypher::Isolat
return isolation_level_query;
}
antlrcpp::Any CypherMainVisitor::visitStorageModeQuery(MemgraphCypher::StorageModeQueryContext *ctx) {
auto *storage_mode_query = storage_->Create<StorageModeQuery>();
storage_mode_query->storage_mode_ = std::invoke([mode = ctx->storageMode()]() {
if (mode->IN_MEMORY_ANALYTICAL()) {
return StorageModeQuery::StorageMode::IN_MEMORY_ANALYTICAL;
}
return StorageModeQuery::StorageMode::IN_MEMORY_TRANSACTIONAL;
});
query_ = storage_mode_query;
return storage_mode_query;
}
antlrcpp::Any CypherMainVisitor::visitCreateSnapshotQuery(MemgraphCypher::CreateSnapshotQueryContext *ctx) {
query_ = storage_->Create<CreateSnapshotQuery>();
return query_;
@@ -631,6 +664,7 @@ void GetTopicNames(auto &destination, MemgraphCypher::TopicNamesContext *topic_n
destination = std::any_cast<Expression *>(topic_names_ctx->accept(&visitor));
}
}
} // namespace
antlrcpp::Any CypherMainVisitor::visitKafkaCreateStreamConfig(MemgraphCypher::KafkaCreateStreamConfigContext *ctx) {
@@ -883,6 +917,34 @@ antlrcpp::Any CypherMainVisitor::visitShowSettings(MemgraphCypher::ShowSettingsC
return setting_query;
}
antlrcpp::Any CypherMainVisitor::visitTransactionQueueQuery(MemgraphCypher::TransactionQueueQueryContext *ctx) {
MG_ASSERT(ctx->children.size() == 1, "TransactionQueueQuery should have exactly one child!");
auto *transaction_queue_query = std::any_cast<TransactionQueueQuery *>(ctx->children[0]->accept(this));
query_ = transaction_queue_query;
return transaction_queue_query;
}
antlrcpp::Any CypherMainVisitor::visitShowTransactions(MemgraphCypher::ShowTransactionsContext * /*ctx*/) {
auto *transaction_shower = storage_->Create<TransactionQueueQuery>();
transaction_shower->action_ = TransactionQueueQuery::Action::SHOW_TRANSACTIONS;
return transaction_shower;
}
antlrcpp::Any CypherMainVisitor::visitTerminateTransactions(MemgraphCypher::TerminateTransactionsContext *ctx) {
auto *terminator = storage_->Create<TransactionQueueQuery>();
terminator->action_ = TransactionQueueQuery::Action::TERMINATE_TRANSACTIONS;
terminator->transaction_id_list_ = std::any_cast<std::vector<Expression *>>(ctx->transactionIdList()->accept(this));
return terminator;
}
antlrcpp::Any CypherMainVisitor::visitTransactionIdList(MemgraphCypher::TransactionIdListContext *ctx) {
std::vector<Expression *> transaction_ids;
for (auto *transaction_id : ctx->transactionId()) {
transaction_ids.push_back(std::any_cast<Expression *>(transaction_id->accept(this)));
}
return transaction_ids;
}
antlrcpp::Any CypherMainVisitor::visitVersionQuery(MemgraphCypher::VersionQueryContext * /*ctx*/) {
auto *version_query = storage_->Create<VersionQuery>();
query_ = version_query;
@@ -941,6 +1003,10 @@ antlrcpp::Any CypherMainVisitor::visitSingleQuery(MemgraphCypher::SingleQueryCon
calls_write_procedure = true;
has_update = true;
}
} else if (const auto *call_subquery = utils::Downcast<CallSubquery>(clause); call_subquery != nullptr) {
if (has_return) {
throw SemanticException("CALL can't be put after RETURN clause.");
}
} else if (utils::IsSubtype(clause_type, Unwind::kType)) {
check_write_procedure("UNWIND");
if (has_update || has_return) {
@@ -1053,6 +1119,9 @@ antlrcpp::Any CypherMainVisitor::visitClause(MemgraphCypher::ClauseContext *ctx)
if (ctx->foreach ()) {
return static_cast<Clause *>(std::any_cast<Foreach *>(ctx->foreach ()->accept(this)));
}
if (ctx->callSubquery()) {
return static_cast<Clause *>(std::any_cast<CallSubquery *>(ctx->callSubquery()->accept(this)));
}
// TODO: implement other clauses.
throw utils::NotYetImplemented("clause '{}'", ctx->getText());
return 0;
@@ -1410,19 +1479,23 @@ antlrcpp::Any CypherMainVisitor::visitEntityPrivilegeList(MemgraphCypher::Entity
return result;
}
antlrcpp::Any CypherMainVisitor::visitListOfColonSymbolicNames(MemgraphCypher::ListOfColonSymbolicNamesContext *ctx) {
std::vector<std::string> symbolic_names;
for (auto *symbolic_name : ctx->colonSymbolicName()) {
symbolic_names.push_back(std::any_cast<std::string>(symbolic_name->symbolicName()->accept(this)));
}
return symbolic_names;
}
/**
* @return std::vector<std::string>
*/
antlrcpp::Any CypherMainVisitor::visitEntitiesList(MemgraphCypher::EntitiesListContext *ctx) {
std::vector<std::string> entities;
if (ctx->listOfEntities()) {
for (auto *entity : ctx->listOfEntities()->entity()) {
entities.push_back(std::any_cast<std::string>(entity->symbolicName()->accept(this)));
}
} else {
entities.emplace_back("*");
if (ctx->listOfColonSymbolicNames()) {
return ctx->listOfColonSymbolicNames()->accept(this);
}
entities.emplace_back("*");
return entities;
}
@@ -1451,6 +1524,8 @@ antlrcpp::Any CypherMainVisitor::visitPrivilege(MemgraphCypher::PrivilegeContext
if (ctx->MODULE_READ()) return AuthQuery::Privilege::MODULE_READ;
if (ctx->MODULE_WRITE()) return AuthQuery::Privilege::MODULE_WRITE;
if (ctx->WEBSOCKET()) return AuthQuery::Privilege::WEBSOCKET;
if (ctx->TRANSACTION_MANAGEMENT()) return AuthQuery::Privilege::TRANSACTION_MANAGEMENT;
if (ctx->STORAGE_MODE()) return AuthQuery::Privilege::STORAGE_MODE;
LOG_FATAL("Should not get here - unknown privilege!");
}
@@ -1543,6 +1618,7 @@ antlrcpp::Any CypherMainVisitor::visitReturnItem(MemgraphCypher::ReturnItemConte
named_expr->expression_ = std::any_cast<Expression *>(ctx->expression()->accept(this));
MG_ASSERT(named_expr->expression_);
if (ctx->variable()) {
named_expr->is_aliased_ = true;
named_expr->name_ = std::string(std::any_cast<std::string>(ctx->variable()->accept(this)));
users_identifiers.insert(named_expr->name_);
} else {
@@ -2160,7 +2236,10 @@ antlrcpp::Any CypherMainVisitor::visitAtom(MemgraphCypher::AtomContext *ctx) {
auto *list = std::any_cast<Expression *>(ctx->extractExpression()->idInColl()->expression()->accept(this));
auto *expr = std::any_cast<Expression *>(ctx->extractExpression()->expression()->accept(this));
return static_cast<Expression *>(storage_->Create<Extract>(ident, list, expr));
} else if (ctx->existsExpression()) {
return std::any_cast<Expression *>(ctx->existsExpression()->accept(this));
}
// TODO: Implement this. We don't support comprehensions, filtering... at
// the moment.
throw utils::NotYetImplemented("atom expression '{}'", ctx->getText());
@@ -2204,6 +2283,17 @@ antlrcpp::Any CypherMainVisitor::visitLiteral(MemgraphCypher::LiteralContext *ct
return visitChildren(ctx);
}
antlrcpp::Any CypherMainVisitor::visitExistsExpression(MemgraphCypher::ExistsExpressionContext *ctx) {
auto *exists = storage_->Create<Exists>();
exists->pattern_ = std::any_cast<Pattern *>(ctx->patternPart()->accept(this));
if (exists->pattern_->identifier_) {
throw SyntaxException("Identifiers are not supported in exists(...).");
}
return static_cast<Expression *>(exists);
}
antlrcpp::Any CypherMainVisitor::visitParenthesizedExpression(MemgraphCypher::ParenthesizedExpressionContext *ctx) {
return std::any_cast<Expression *>(ctx->expression()->accept(this));
}
@@ -2498,6 +2588,20 @@ antlrcpp::Any CypherMainVisitor::visitShowConfigQuery(MemgraphCypher::ShowConfig
return query_;
}
antlrcpp::Any CypherMainVisitor::visitCallSubquery(MemgraphCypher::CallSubqueryContext *ctx) {
auto *call_subquery = storage_->Create<CallSubquery>();
MG_ASSERT(ctx->cypherQuery(), "Expected query inside subquery clause");
if (ctx->cypherQuery()->queryMemoryLimit()) {
throw SyntaxException("Memory limit cannot be set on subqueries!");
}
call_subquery->cypher_query_ = std::any_cast<CypherQuery *>(ctx->cypherQuery()->accept(this));
return call_subquery;
}
LabelIx CypherMainVisitor::AddLabel(const std::string &name) { return storage_->GetLabelIx(name); }
PropertyIx CypherMainVisitor::AddProperty(const std::string &name) { return storage_->GetPropertyIx(name); }

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -183,6 +183,16 @@ class CypherMainVisitor : public antlropencypher::MemgraphCypherBaseVisitor {
*/
antlrcpp::Any visitDumpQuery(MemgraphCypher::DumpQueryContext *ctx) override;
/**
@return std::vector<std::string>
*/
antlrcpp::Any visitListOfColonSymbolicNames(MemgraphCypher::ListOfColonSymbolicNamesContext *ctx) override;
/**
* @return AnalyzeGraphQuery*
*/
antlrcpp::Any visitAnalyzeGraphQuery(MemgraphCypher::AnalyzeGraphQueryContext *ctx) override;
/**
* @return ReplicationQuery*
*/
@@ -253,6 +263,11 @@ class CypherMainVisitor : public antlropencypher::MemgraphCypherBaseVisitor {
*/
antlrcpp::Any visitIsolationLevelQuery(MemgraphCypher::IsolationLevelQueryContext *ctx) override;
/**
* @return StorageModeQuery*
*/
antlrcpp::Any visitStorageModeQuery(MemgraphCypher::StorageModeQueryContext *ctx) override;
/**
* @return CreateSnapshotQuery*
*/
@@ -358,6 +373,26 @@ class CypherMainVisitor : public antlropencypher::MemgraphCypherBaseVisitor {
*/
antlrcpp::Any visitShowSettings(MemgraphCypher::ShowSettingsContext *ctx) override;
/**
* @return TransactionQueueQuery*
*/
antlrcpp::Any visitTransactionQueueQuery(MemgraphCypher::TransactionQueueQueryContext *ctx) override;
/**
* @return ShowTransactions*
*/
antlrcpp::Any visitShowTransactions(MemgraphCypher::ShowTransactionsContext *ctx) override;
/**
* @return TerminateTransactions*
*/
antlrcpp::Any visitTerminateTransactions(MemgraphCypher::TerminateTransactionsContext *ctx) override;
/**
* @return TransactionIdList*
*/
antlrcpp::Any visitTransactionIdList(MemgraphCypher::TransactionIdListContext *ctx) override;
/**
* @return VersionQuery*
*/
@@ -763,6 +798,11 @@ class CypherMainVisitor : public antlropencypher::MemgraphCypherBaseVisitor {
*/
antlrcpp::Any visitParameter(MemgraphCypher::ParameterContext *ctx) override;
/**
* @return Exists* (Expression)
*/
antlrcpp::Any visitExistsExpression(MemgraphCypher::ExistsExpressionContext *ctx) override;
/**
* @return Expression*
*/
@@ -887,6 +927,11 @@ class CypherMainVisitor : public antlropencypher::MemgraphCypherBaseVisitor {
*/
antlrcpp::Any visitShowConfigQuery(MemgraphCypher::ShowConfigQueryContext *ctx) override;
/**
* @return CallSubquery*
*/
antlrcpp::Any visitCallSubquery(MemgraphCypher::CallSubqueryContext *ctx) override;
public:
Query *query() { return query_; }
const static std::string kAnonPrefix;

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -60,6 +60,7 @@ class ExpressionPrettyPrinter : public ExpressionVisitor<void> {
void Visit(Reduce &op) override;
void Visit(Coalesce &op) override;
void Visit(Extract &op) override;
void Visit(Exists &op) override;
void Visit(All &op) override;
void Visit(Single &op) override;
void Visit(Any &op) override;
@@ -264,6 +265,8 @@ void ExpressionPrettyPrinter::Visit(Extract &op) {
PrintOperator(out_, "Extract", op.identifier_, op.list_, op.expression_);
}
void ExpressionPrettyPrinter::Visit(Exists & /*op*/) { PrintOperator(out_, "Exists", "expression"); }
void ExpressionPrettyPrinter::Visit(All &op) {
PrintOperator(out_, "All", op.identifier_, op.list_expression_, op.where_->expression_);
}

View File

@@ -236,6 +236,7 @@ atom : literal
| ( ANY '(' filterExpression ')' )
| ( NONE '(' filterExpression ')' )
| ( SINGLE '(' filterExpression ')' )
| ( EXISTS '(' existsExpression ')' )
| relationshipsPattern
| parenthesizedExpression
| functionInvocation
@@ -275,6 +276,8 @@ reduceExpression : accumulator=variable '=' initial=expression ',' idInColl '|'
extractExpression : idInColl '|' expression ;
existsExpression : patternPart ;
idInColl : variable IN expression ;
functionInvocation : functionName '(' ( DISTINCT )? ( expression ( ',' expression )* )? ')' ;

View File

@@ -22,6 +22,7 @@ import Cypher ;
memgraphCypherKeyword : cypherKeyword
| AFTER
| ALTER
| ANALYZE
| ASYNC
| AUTH
| BAD
@@ -53,10 +54,13 @@ memgraphCypherKeyword : cypherKeyword
| FREE
| FROM
| GLOBAL
| GRAPH
| GRANT
| HEADER
| IDENTIFIED
| ISOLATION
| IN_MEMORY_ANALYTICAL
| IN_MEMORY_TRANSACTIONAL
| KAFKA
| LABELS
| LEVEL
@@ -86,6 +90,7 @@ memgraphCypherKeyword : cypherKeyword
| SNAPSHOT
| START
| STATS
| STORAGE
| STREAM
| STREAMS
| SYNC
@@ -102,6 +107,8 @@ memgraphCypherKeyword : cypherKeyword
| USER
| USERS
| VERSION
| TERMINATE
| TRANSACTIONS
;
symbolicName : UnescapedSymbolicName
@@ -117,16 +124,19 @@ query : cypherQuery
| constraintQuery
| authQuery
| dumpQuery
| analyzeGraphQuery
| replicationQuery
| lockPathQuery
| freeMemoryQuery
| triggerQuery
| isolationLevelQuery
| storageModeQuery
| createSnapshotQuery
| streamQuery
| settingQuery
| versionQuery
| showConfigQuery
| transactionQueueQuery
;
authQuery : createRole
@@ -170,6 +180,7 @@ clause : cypherMatch
| callProcedure
| loadCsv
| foreach
| callSubquery
;
updateClause : set
@@ -182,6 +193,8 @@ updateClause : set
foreach : FOREACH '(' variable IN expression '|' updateClause+ ')' ;
callSubquery : CALL '{' cypherQuery '}' ;
streamQuery : checkStream
| createStream
| dropStream
@@ -197,6 +210,14 @@ settingQuery : setSetting
| showSettings
;
transactionQueueQuery : showTransactions
| terminateTransactions
;
showTransactions : SHOW TRANSACTIONS ;
terminateTransactions : TERMINATE TRANSACTIONS transactionIdList;
loadCsv : LOAD CSV FROM csvFile ( WITH | NO ) HEADER
( IGNORE BAD ) ?
( DELIMITER delimiter ) ?
@@ -259,6 +280,8 @@ privilege : CREATE
| MODULE_READ
| MODULE_WRITE
| WEBSOCKET
| TRANSACTION_MANAGEMENT
| STORAGE_MODE
;
granularPrivilege : NOTHING | READ | UPDATE | CREATE_DELETE ;
@@ -279,11 +302,11 @@ revokePrivilegesList : privilegeOrEntities ( ',' privilegeOrEntities )* ;
privilegesList : privilege ( ',' privilege )* ;
entitiesList : ASTERISK | listOfEntities ;
entitiesList : ASTERISK | listOfColonSymbolicNames ;
listOfEntities : entity ( ',' entity )* ;
listOfColonSymbolicNames : colonSymbolicName ( ',' colonSymbolicName )* ;
entity : COLON symbolicName ;
colonSymbolicName : COLON symbolicName ;
showPrivileges : SHOW PRIVILEGES FOR userOrRole=userOrRoleName ;
@@ -293,6 +316,8 @@ showUsersForRole : SHOW USERS FOR role=userOrRoleName ;
dumpQuery: DUMP DATABASE ;
analyzeGraphQuery: ANALYZE GRAPH ( ON LABELS ( listOfColonSymbolicNames | ASTERISK ) ) ? ( DELETE STATISTICS ) ? ;
setReplicationRole : SET REPLICATION ROLE TO ( MAIN | REPLICA )
( WITH PORT port=literal ) ? ;
@@ -334,6 +359,10 @@ isolationLevelScope : GLOBAL | SESSION | NEXT ;
isolationLevelQuery : SET isolationLevelScope TRANSACTION ISOLATION LEVEL isolationLevel ;
storageMode : IN_MEMORY_ANALYTICAL | IN_MEMORY_TRANSACTIONAL ;
storageModeQuery : STORAGE MODE storageMode ;
createSnapshotQuery : CREATE SNAPSHOT ;
streamName : symbolicName ;
@@ -402,3 +431,7 @@ showSettings : SHOW DATABASE SETTINGS ;
showConfigQuery : SHOW CONFIG ;
versionQuery : SHOW VERSION ;
transactionIdList : transactionId ( ',' transactionId )* ;
transactionId : literal ;

View File

@@ -25,96 +25,107 @@ import CypherLexer ;
UNDERSCORE : '_' ;
AFTER : A F T E R ;
ALTER : A L T E R ;
ASYNC : A S Y N C ;
AUTH : A U T H ;
BAD : B A D ;
BATCH_INTERVAL : B A T C H UNDERSCORE I N T E R V A L ;
BATCH_LIMIT : B A T C H UNDERSCORE L I M I T ;
BATCH_SIZE : B A T C H UNDERSCORE S I Z E ;
BEFORE : B E F O R E ;
BOOTSTRAP_SERVERS : B O O T S T R A P UNDERSCORE S E R V E R S ;
CHECK : C H E C K ;
CLEAR : C L E A R ;
COMMIT : C O M M I T ;
COMMITTED : C O M M I T T E D ;
CONFIG : C O N F I G ;
CONFIGS : C O N F I G S;
CONSUMER_GROUP : C O N S U M E R UNDERSCORE G R O U P ;
CREATE_DELETE : C R E A T E UNDERSCORE D E L E T E ;
CREDENTIALS : C R E D E N T I A L S ;
CSV : C S V ;
DATA : D A T A ;
DELIMITER : D E L I M I T E R ;
DATABASE : D A T A B A S E ;
DENY : D E N Y ;
DIRECTORY : D I R E C T O R Y ;
DROP : D R O P ;
DUMP : D U M P ;
DURABILITY : D U R A B I L I T Y ;
EXECUTE : E X E C U T E ;
FOR : F O R ;
FOREACH : F O R E A C H;
FREE : F R E E ;
FREE_MEMORY : F R E E UNDERSCORE M E M O R Y ;
FROM : F R O M ;
GLOBAL : G L O B A L ;
GRANT : G R A N T ;
GRANTS : G R A N T S ;
HEADER : H E A D E R ;
IDENTIFIED : I D E N T I F I E D ;
IGNORE : I G N O R E ;
ISOLATION : I S O L A T I O N ;
KAFKA : K A F K A ;
LABELS : L A B E L S ;
LEVEL : L E V E L ;
LOAD : L O A D ;
LOCK : L O C K ;
MAIN : M A I N ;
MODE : M O D E ;
MODULE_READ : M O D U L E UNDERSCORE R E A D ;
MODULE_WRITE : M O D U L E UNDERSCORE W R I T E ;
NEXT : N E X T ;
NO : N O ;
NOTHING : N O T H I N G ;
PASSWORD : P A S S W O R D ;
PORT : P O R T ;
PRIVILEGES : P R I V I L E G E S ;
PULSAR : P U L S A R ;
READ : R E A D ;
READ_FILE : R E A D UNDERSCORE F I L E ;
REGISTER : R E G I S T E R ;
REPLICA : R E P L I C A ;
REPLICAS : R E P L I C A S ;
REPLICATION : R E P L I C A T I O N ;
REVOKE : R E V O K E ;
ROLE : R O L E ;
ROLES : R O L E S ;
QUOTE : Q U O T E ;
SERVICE_URL : S E R V I C E UNDERSCORE U R L ;
SESSION : S E S S I O N ;
SETTING : S E T T I N G ;
SETTINGS : S E T T I N G S ;
SNAPSHOT : S N A P S H O T ;
START : S T A R T ;
STATS : S T A T S ;
STOP : S T O P ;
STREAM : S T R E A M ;
STREAMS : S T R E A M S ;
SYNC : S Y N C ;
TIMEOUT : T I M E O U T ;
TO : T O ;
TOPICS : T O P I C S;
TRANSACTION : T R A N S A C T I O N ;
TRANSFORM : T R A N S F O R M ;
TRIGGER : T R I G G E R ;
TRIGGERS : T R I G G E R S ;
UNCOMMITTED : U N C O M M I T T E D ;
UNLOCK : U N L O C K ;
UPDATE : U P D A T E ;
USER : U S E R ;
USERS : U S E R S ;
VERSION : V E R S I O N ;
WEBSOCKET : W E B S O C K E T ;
EDGE_TYPES : E D G E UNDERSCORE T Y P E S ;
AFTER : A F T E R ;
ALTER : A L T E R ;
ANALYZE : A N A L Y Z E ;
ASYNC : A S Y N C ;
AUTH : A U T H ;
BAD : B A D ;
BATCH_INTERVAL : B A T C H UNDERSCORE I N T E R V A L ;
BATCH_LIMIT : B A T C H UNDERSCORE L I M I T ;
BATCH_SIZE : B A T C H UNDERSCORE S I Z E ;
BEFORE : B E F O R E ;
BOOTSTRAP_SERVERS : B O O T S T R A P UNDERSCORE S E R V E R S ;
CALL : C A L L ;
CHECK : C H E C K ;
CLEAR : C L E A R ;
COMMIT : C O M M I T ;
COMMITTED : C O M M I T T E D ;
CONFIG : C O N F I G ;
CONFIGS : C O N F I G S;
CONSUMER_GROUP : C O N S U M E R UNDERSCORE G R O U P ;
CREATE_DELETE : C R E A T E UNDERSCORE D E L E T E ;
CREDENTIALS : C R E D E N T I A L S ;
CSV : C S V ;
DATA : D A T A ;
DELIMITER : D E L I M I T E R ;
DATABASE : D A T A B A S E ;
DENY : D E N Y ;
DIRECTORY : D I R E C T O R Y ;
DROP : D R O P ;
DUMP : D U M P ;
DURABILITY : D U R A B I L I T Y ;
EDGE_TYPES : E D G E UNDERSCORE T Y P E S ;
EXECUTE : E X E C U T E ;
FOR : F O R ;
FOREACH : F O R E A C H;
FREE : F R E E ;
FREE_MEMORY : F R E E UNDERSCORE M E M O R Y ;
FROM : F R O M ;
GLOBAL : G L O B A L ;
GRANT : G R A N T ;
GRAPH : G R A P H ;
GRANTS : G R A N T S ;
HEADER : H E A D E R ;
IDENTIFIED : I D E N T I F I E D ;
IGNORE : I G N O R E ;
ISOLATION : I S O L A T I O N ;
IN_MEMORY_ANALYTICAL : I N UNDERSCORE M E M O R Y UNDERSCORE A N A L Y T I C A L ;
IN_MEMORY_TRANSACTIONAL : I N UNDERSCORE M E M O R Y UNDERSCORE T R A N S A C T I O N A L ;
KAFKA : K A F K A ;
LABELS : L A B E L S ;
LEVEL : L E V E L ;
LOAD : L O A D ;
LOCK : L O C K ;
MAIN : M A I N ;
MODE : M O D E ;
MODULE_READ : M O D U L E UNDERSCORE R E A D ;
MODULE_WRITE : M O D U L E UNDERSCORE W R I T E ;
NEXT : N E X T ;
NO : N O ;
NOTHING : N O T H I N G ;
PASSWORD : P A S S W O R D ;
PORT : P O R T ;
PRIVILEGES : P R I V I L E G E S ;
PULSAR : P U L S A R ;
READ : R E A D ;
READ_FILE : R E A D UNDERSCORE F I L E ;
REGISTER : R E G I S T E R ;
REPLICA : R E P L I C A ;
REPLICAS : R E P L I C A S ;
REPLICATION : R E P L I C A T I O N ;
REVOKE : R E V O K E ;
ROLE : R O L E ;
ROLES : R O L E S ;
QUOTE : Q U O T E ;
SERVICE_URL : S E R V I C E UNDERSCORE U R L ;
SESSION : S E S S I O N ;
SETTING : S E T T I N G ;
SETTINGS : S E T T I N G S ;
SNAPSHOT : S N A P S H O T ;
START : S T A R T ;
STATISTICS : S T A T I S T I C S ;
STATS : S T A T S ;
STOP : S T O P ;
STORAGE : S T O R A G E;
STORAGE_MODE : S T O R A G E UNDERSCORE MODE;
STREAM : S T R E A M ;
STREAMS : S T R E A M S ;
SYNC : S Y N C ;
TERMINATE : T E R M I N A T E ;
TIMEOUT : T I M E O U T ;
TO : T O ;
TOPICS : T O P I C S;
TRANSACTION : T R A N S A C T I O N ;
TRANSACTION_MANAGEMENT : T R A N S A C T I O N UNDERSCORE M A N A G E M E N T ;
TRANSACTIONS : T R A N S A C T I O N S ;
TRANSFORM : T R A N S F O R M ;
TRIGGER : T R I G G E R ;
TRIGGERS : T R I G G E R S ;
UNCOMMITTED : U N C O M M I T T E D ;
UNLOCK : U N L O C K ;
UPDATE : U P D A T E ;
USER : U S E R ;
USERS : U S E R S ;
VERSION : V E R S I O N ;
WEBSOCKET : W E B S O C K E T ;

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -25,13 +25,15 @@ class PrivilegeExtractor : public QueryVisitor<void>, public HierarchicalTreeVis
std::vector<AuthQuery::Privilege> privileges() { return privileges_; }
void Visit(IndexQuery &) override { AddPrivilege(AuthQuery::Privilege::INDEX); }
void Visit(IndexQuery & /*unused*/) override { AddPrivilege(AuthQuery::Privilege::INDEX); }
void Visit(AuthQuery &) override { AddPrivilege(AuthQuery::Privilege::AUTH); }
void Visit(AnalyzeGraphQuery & /*unused*/) override { AddPrivilege(AuthQuery::Privilege::INDEX); }
void Visit(ExplainQuery &query) override { query.cypher_query_->Accept(*this); }
void Visit(AuthQuery & /*unused*/) override { AddPrivilege(AuthQuery::Privilege::AUTH); }
void Visit(ProfileQuery &query) override { query.cypher_query_->Accept(*this); }
void Visit(ExplainQuery &query) override { query.cypher_query_->Accept(dynamic_cast<QueryVisitor &>(*this)); }
void Visit(ProfileQuery &query) override { query.cypher_query_->Accept(dynamic_cast<QueryVisitor &>(*this)); }
void Visit(InfoQuery &info_query) override {
switch (info_query.info_type_) {
@@ -76,10 +78,14 @@ class PrivilegeExtractor : public QueryVisitor<void>, public HierarchicalTreeVis
void Visit(IsolationLevelQuery &isolation_level_query) override { AddPrivilege(AuthQuery::Privilege::CONFIG); }
void Visit(StorageModeQuery & /*storage_mode_query*/) override { AddPrivilege(AuthQuery::Privilege::STORAGE_MODE); }
void Visit(CreateSnapshotQuery &create_snapshot_query) override { AddPrivilege(AuthQuery::Privilege::DURABILITY); }
void Visit(SettingQuery & /*setting_query*/) override { AddPrivilege(AuthQuery::Privilege::CONFIG); }
void Visit(TransactionQueueQuery & /*transaction_queue_query*/) override {}
void Visit(VersionQuery & /*version_query*/) override { AddPrivilege(AuthQuery::Privilege::STATS); }
bool PreVisit(Create & /*unused*/) override {

View File

@@ -0,0 +1,18 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include "query/frontend/semantic/symbol.hpp"
#include "utils/typeinfo.hpp"
namespace memgraph {
constexpr utils::TypeInfo query::Symbol::kType{utils::TypeId::SYMBOL, "Symbol", nullptr};
} // namespace memgraph

View File

@@ -0,0 +1,75 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#pragma once
#include <string>
#include "utils/typeinfo.hpp"
namespace memgraph {
namespace query {
class Symbol {
public:
static const utils::TypeInfo kType;
static const utils::TypeInfo &GetTypeInfo() { return kType; }
enum class Type { ANY, VERTEX, EDGE, PATH, NUMBER, EDGE_LIST };
// TODO: Generate enum to string conversion from LCP. Note, that this is
// displayed to the end user, so we may want to have a pretty name of each
// value.
static std::string TypeToString(Type type) {
const char *enum_string[] = {"Any", "Vertex", "Edge", "Path", "Number", "EdgeList"};
return enum_string[static_cast<int>(type)];
}
Symbol() {}
Symbol(const std::string &name, int position, bool user_declared, Type type = Type::ANY, int token_position = -1)
: name_(name), position_(position), user_declared_(user_declared), type_(type), token_position_(token_position) {}
bool operator==(const Symbol &other) const {
return position_ == other.position_ && name_ == other.name_ && type_ == other.type_;
}
bool operator!=(const Symbol &other) const { return !operator==(other); }
// TODO: Remove these since members are public
const auto &name() const { return name_; }
int position() const { return position_; }
Type type() const { return type_; }
bool user_declared() const { return user_declared_; }
int token_position() const { return token_position_; }
std::string name_;
int64_t position_;
bool user_declared_{true};
memgraph::query::Symbol::Type type_{Type::ANY};
int64_t token_position_{-1};
};
} // namespace query
} // namespace memgraph
namespace std {
template <>
struct hash<memgraph::query::Symbol> {
size_t operator()(const memgraph::query::Symbol &symbol) const {
size_t prime = 265443599u;
size_t hash = std::hash<int>{}(symbol.position());
hash ^= prime * std::hash<std::string>{}(symbol.name());
hash ^= prime * std::hash<int>{}(static_cast<int>(symbol.type()));
return hash;
}
};
} // namespace std

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -64,12 +64,9 @@ auto SymbolGenerator::CreateSymbol(const std::string &name, bool user_declared,
return symbol;
}
auto SymbolGenerator::GetOrCreateSymbolLocalScope(const std::string &name, bool user_declared, Symbol::Type type) {
auto &scope = scopes_.back();
if (auto maybe_symbol = FindSymbolInScope(name, scope, type); maybe_symbol) {
return *maybe_symbol;
}
return CreateSymbol(name, user_declared, type);
auto SymbolGenerator::CreateAnonymousSymbol(Symbol::Type /*type*/) {
auto symbol = symbol_table_->CreateAnonymousSymbol();
return symbol;
}
auto SymbolGenerator::GetOrCreateSymbol(const std::string &name, bool user_declared, Symbol::Type type) {
@@ -206,7 +203,7 @@ bool SymbolGenerator::PreVisit(CallProcedure &call_proc) {
bool SymbolGenerator::PostVisit(CallProcedure &call_proc) {
for (auto *ident : call_proc.result_identifiers_) {
if (HasSymbolLocalScope(ident->name_)) {
if (HasSymbol(ident->name_)) {
throw RedeclareVariableError(ident->name_);
}
ident->MapTo(CreateSymbol(ident->name_, true));
@@ -214,10 +211,37 @@ bool SymbolGenerator::PostVisit(CallProcedure &call_proc) {
return true;
}
bool SymbolGenerator::PreVisit(CallSubquery & /*call_sub*/) {
scopes_.emplace_back(Scope{.in_call_subquery = true});
return true;
}
bool SymbolGenerator::PostVisit(CallSubquery & /*call_sub*/) {
// no need to set the flag to true as we are popping the scope
auto subquery_scope = scopes_.back();
scopes_.pop_back();
auto &main_query_scope = scopes_.back();
if (!subquery_scope.has_return) {
return true;
}
// append symbols returned in from subquery to outer scope
for (const auto &[symbol_name, symbol] : subquery_scope.symbols) {
if (main_query_scope.symbols.find(symbol_name) != main_query_scope.symbols.end()) {
throw SemanticException("Variable in subquery already declared in outer scope!");
}
main_query_scope.symbols[symbol_name] = symbol;
}
return true;
}
bool SymbolGenerator::PreVisit(LoadCsv &load_csv) { return false; }
bool SymbolGenerator::PostVisit(LoadCsv &load_csv) {
if (HasSymbolLocalScope(load_csv.row_var_->name_)) {
if (HasSymbol(load_csv.row_var_->name_)) {
throw RedeclareVariableError(load_csv.row_var_->name_);
}
load_csv.row_var_->MapTo(CreateSymbol(load_csv.row_var_->name_, true));
@@ -227,6 +251,8 @@ bool SymbolGenerator::PostVisit(LoadCsv &load_csv) {
bool SymbolGenerator::PreVisit(Return &ret) {
auto &scope = scopes_.back();
scope.in_return = true;
scope.has_return = true;
VisitReturnBody(ret.body_);
scope.in_return = false;
return false; // We handled the traversal ourselves.
@@ -265,7 +291,7 @@ bool SymbolGenerator::PostVisit(Merge &) {
bool SymbolGenerator::PostVisit(Unwind &unwind) {
const auto &name = unwind.named_expression_->name_;
if (HasSymbolLocalScope(name)) {
if (HasSymbol(name)) {
throw RedeclareVariableError(name);
}
unwind.named_expression_->MapTo(CreateSymbol(name, true));
@@ -282,7 +308,7 @@ bool SymbolGenerator::PostVisit(Match &) {
// Check variables in property maps after visiting Match, so that they can
// reference symbols out of bind order.
for (auto &ident : scope.identifiers_in_match) {
if (!HasSymbolLocalScope(ident->name_) && !ConsumePredefinedIdentifier(ident->name_))
if (!HasSymbol(ident->name_) && !ConsumePredefinedIdentifier(ident->name_))
throw UnboundVariableError(ident->name_);
ident->MapTo(scope.symbols[ident->name_]);
}
@@ -310,11 +336,19 @@ SymbolGenerator::ReturnType SymbolGenerator::Visit(Identifier &ident) {
if (scope.in_skip || scope.in_limit) {
throw SemanticException("Variables are not allowed in {}.", scope.in_skip ? "SKIP" : "LIMIT");
}
if (scope.in_exists && (scope.visiting_edge || scope.in_node_atom)) {
auto has_symbol = HasSymbol(ident.name_);
if (!has_symbol && !ConsumePredefinedIdentifier(ident.name_) && ident.user_declared_) {
throw SemanticException("Unbounded variables are not allowed in exists!");
}
}
Symbol symbol;
if (scope.in_pattern && !(scope.in_node_atom || scope.visiting_edge)) {
// If we are in the pattern, and outside of a node or an edge, the
// identifier is the pattern name.
symbol = GetOrCreateSymbolLocalScope(ident.name_, ident.user_declared_, Symbol::Type::PATH);
symbol = GetOrCreateSymbol(ident.name_, ident.user_declared_, Symbol::Type::PATH);
} else if (scope.in_pattern && scope.in_pattern_atom_identifier) {
// Patterns used to create nodes and edges cannot redeclare already
// established bindings. Declaration only happens in single node
@@ -322,21 +356,22 @@ SymbolGenerator::ReturnType SymbolGenerator::Visit(Identifier &ident) {
// `MATCH (n) CREATE (n)` should throw an error that `n` is already
// declared. While `MATCH (n) CREATE (n) -[:R]-> (n)` is allowed,
// since `n` now references the bound node instead of declaring it.
if ((scope.in_create_node || scope.in_create_edge) && HasSymbolLocalScope(ident.name_)) {
if ((scope.in_create_node || scope.in_create_edge) && HasSymbol(ident.name_)) {
throw RedeclareVariableError(ident.name_);
}
auto type = Symbol::Type::VERTEX;
if (scope.visiting_edge) {
// Edge referencing is not allowed (like in Neo4j):
// `MATCH (n) - [r] -> (n) - [r] -> (n) RETURN r` is not allowed.
if (HasSymbolLocalScope(ident.name_)) {
if (HasSymbol(ident.name_)) {
throw RedeclareVariableError(ident.name_);
}
type = scope.visiting_edge->IsVariable() ? Symbol::Type::EDGE_LIST : Symbol::Type::EDGE;
}
symbol = GetOrCreateSymbolLocalScope(ident.name_, ident.user_declared_, type);
symbol = GetOrCreateSymbol(ident.name_, ident.user_declared_, type);
} else if (scope.in_pattern && !scope.in_pattern_atom_identifier && scope.in_match) {
if (scope.in_edge_range && scope.visiting_edge->identifier_->name_ == ident.name_) {
if (scope.in_edge_range && scope.visiting_edge && scope.visiting_edge->identifier_ &&
scope.visiting_edge->identifier_->name_ == ident.name_) {
// Prevent variable path bounds to reference the identifier which is bound
// by the variable path itself.
throw UnboundVariableError(ident.name_);
@@ -438,6 +473,55 @@ bool SymbolGenerator::PreVisit(Extract &extract) {
return false;
}
bool SymbolGenerator::PreVisit(Exists &exists) {
auto &scope = scopes_.back();
if (scope.in_set_property) {
throw utils::NotYetImplemented("Set property can not be used with exists, but only during matching!");
}
if (scope.in_with) {
throw utils::NotYetImplemented("WITH can not be used with exists, but only during matching!");
}
scope.in_exists = true;
const auto &symbol = CreateAnonymousSymbol();
exists.MapTo(symbol);
return true;
}
bool SymbolGenerator::PostVisit(Exists & /*exists*/) {
auto &scope = scopes_.back();
scope.in_exists = false;
return true;
}
bool SymbolGenerator::PreVisit(NamedExpression &named_expression) {
if (auto &scope = scopes_.back(); scope.in_call_subquery && scope.in_return &&
!utils::Downcast<Identifier>(named_expression.expression_) &&
!named_expression.is_aliased_) {
throw SemanticException("Expression returned from subquery must be aliased (use AS)!");
}
return true;
}
bool SymbolGenerator::PreVisit(SetProperty & /*set_property*/) {
auto &scope = scopes_.back();
scope.in_set_property = true;
return true;
}
bool SymbolGenerator::PostVisit(SetProperty & /*set_property*/) {
auto &scope = scopes_.back();
scope.in_set_property = false;
return true;
}
// Pattern and its subparts.
bool SymbolGenerator::PreVisit(Pattern &pattern) {
@@ -447,6 +531,7 @@ bool SymbolGenerator::PreVisit(Pattern &pattern) {
MG_ASSERT(utils::IsSubtype(*pattern.atoms_[0], NodeAtom::kType), "Expected a single NodeAtom in Pattern");
scope.in_create_node = true;
}
return true;
}
@@ -461,7 +546,7 @@ bool SymbolGenerator::PreVisit(NodeAtom &node_atom) {
auto &scope = scopes_.back();
auto check_node_semantic = [&node_atom, &scope, this](const bool props_or_labels) {
const auto &node_name = node_atom.identifier_->name_;
if ((scope.in_create || scope.in_merge) && props_or_labels && HasSymbolLocalScope(node_name)) {
if ((scope.in_create || scope.in_merge) && props_or_labels && HasSymbol(node_name)) {
throw SemanticException("Cannot create node '" + node_name +
"' with labels or properties, because it is already declared.");
}
@@ -555,11 +640,11 @@ bool SymbolGenerator::PreVisit(EdgeAtom &edge_atom) {
edge_atom.identifier_->Accept(*this);
scope.in_pattern_atom_identifier = false;
if (edge_atom.total_weight_) {
if (HasSymbolLocalScope(edge_atom.total_weight_->name_)) {
if (HasSymbol(edge_atom.total_weight_->name_)) {
throw RedeclareVariableError(edge_atom.total_weight_->name_);
}
edge_atom.total_weight_->MapTo(GetOrCreateSymbolLocalScope(
edge_atom.total_weight_->name_, edge_atom.total_weight_->user_declared_, Symbol::Type::NUMBER));
edge_atom.total_weight_->MapTo(GetOrCreateSymbol(edge_atom.total_weight_->name_,
edge_atom.total_weight_->user_declared_, Symbol::Type::NUMBER));
}
return false;
}
@@ -602,10 +687,6 @@ bool SymbolGenerator::HasSymbol(const std::string &name) const {
return std::ranges::any_of(scopes_, [&name](const auto &scope) { return scope.symbols.contains(name); });
}
bool SymbolGenerator::HasSymbolLocalScope(const std::string &name) const {
return scopes_.back().symbols.contains(name);
}
bool SymbolGenerator::ConsumePredefinedIdentifier(const std::string &name) {
auto it = predefined_identifiers_.find(name);

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -50,6 +50,8 @@ class SymbolGenerator : public HierarchicalTreeVisitor {
bool PostVisit(Create &) override;
bool PreVisit(CallProcedure &) override;
bool PostVisit(CallProcedure &) override;
bool PreVisit(CallSubquery & /*unused*/) override;
bool PostVisit(CallSubquery & /*unused*/) override;
bool PreVisit(LoadCsv &) override;
bool PostVisit(LoadCsv &) override;
bool PreVisit(Return &) override;
@@ -64,6 +66,8 @@ class SymbolGenerator : public HierarchicalTreeVisitor {
bool PostVisit(Match &) override;
bool PreVisit(Foreach &) override;
bool PostVisit(Foreach &) override;
bool PreVisit(SetProperty & /*set_property*/) override;
bool PostVisit(SetProperty & /*set_property*/) override;
// Expressions
ReturnType Visit(Identifier &) override;
@@ -79,6 +83,9 @@ class SymbolGenerator : public HierarchicalTreeVisitor {
bool PreVisit(None &) override;
bool PreVisit(Reduce &) override;
bool PreVisit(Extract &) override;
bool PreVisit(Exists & /*exists*/) override;
bool PostVisit(Exists & /*exists*/) override;
bool PreVisit(NamedExpression & /*unused*/) override;
// Pattern and its subparts.
bool PreVisit(Pattern &) override;
@@ -113,6 +120,10 @@ class SymbolGenerator : public HierarchicalTreeVisitor {
bool in_where{false};
bool in_match{false};
bool in_foreach{false};
bool in_exists{false};
bool in_set_property{false};
bool in_call_subquery{false};
bool has_return{false};
// True when visiting a pattern atom (node or edge) identifier, which can be
// reused or created in the pattern itself.
bool in_pattern_atom_identifier{false};
@@ -134,7 +145,6 @@ class SymbolGenerator : public HierarchicalTreeVisitor {
static std::optional<Symbol> FindSymbolInScope(const std::string &name, const Scope &scope, Symbol::Type type);
bool HasSymbol(const std::string &name) const;
bool HasSymbolLocalScope(const std::string &name) const;
// @return true if it added a predefined identifier with that name
bool ConsumePredefinedIdentifier(const std::string &name);
@@ -144,10 +154,12 @@ class SymbolGenerator : public HierarchicalTreeVisitor {
auto CreateSymbol(const std::string &name, bool user_declared, Symbol::Type type = Symbol::Type::ANY,
int token_position = -1);
// Returns a freshly generated anonymous symbol.
auto CreateAnonymousSymbol(Symbol::Type type = Symbol::Type::ANY);
auto GetOrCreateSymbol(const std::string &name, bool user_declared, Symbol::Type type = Symbol::Type::ANY);
// Returns the symbol by name. If the mapping already exists, checks if the
// types match. Otherwise, returns a new symbol.
auto GetOrCreateSymbolLocalScope(const std::string &name, bool user_declared, Symbol::Type type = Symbol::Type::ANY);
void VisitReturnBody(ReturnBody &body, Where *where = nullptr);

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -51,6 +51,7 @@ class SymbolTable final {
const Symbol &at(const Identifier &ident) const { return table_.at(ident.symbol_pos_); }
const Symbol &at(const NamedExpression &nexpr) const { return table_.at(nexpr.symbol_pos_); }
const Symbol &at(const Aggregation &aggr) const { return table_.at(aggr.symbol_pos_); }
const Symbol &at(const Exists &exists) const { return table_.at(exists.symbol_pos_); }
// TODO: Remove these since members are public
int32_t max_position() const { return static_cast<int32_t>(table_.size()); }

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -208,7 +208,10 @@ const trie::Trie kKeywords = {"union",
"websocket",
"foreach",
"labels",
"edge_types"};
"edge_types",
"off",
"in_memory_transactional",
"in_memory_analytical"};
// Unicode codepoints that are allowed at the start of the unescaped name.
const std::bitset<kBitsetSize> kUnescapedNameAllowedStarts(

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -31,6 +31,72 @@
namespace memgraph::query {
class ReferenceExpressionEvaluator : public ExpressionVisitor<TypedValue *> {
public:
ReferenceExpressionEvaluator(Frame *frame, const SymbolTable *symbol_table, const EvaluationContext *ctx)
: frame_(frame), symbol_table_(symbol_table), ctx_(ctx) {}
using ExpressionVisitor<TypedValue *>::Visit;
utils::MemoryResource *GetMemoryResource() const { return ctx_->memory; }
#define UNSUCCESSFUL_VISIT(expr_name) \
TypedValue *Visit(expr_name &expr) override { return nullptr; }
TypedValue *Visit(Identifier &ident) override { return &frame_->at(symbol_table_->at(ident)); }
UNSUCCESSFUL_VISIT(NamedExpression);
UNSUCCESSFUL_VISIT(OrOperator);
UNSUCCESSFUL_VISIT(XorOperator);
UNSUCCESSFUL_VISIT(AdditionOperator);
UNSUCCESSFUL_VISIT(SubtractionOperator);
UNSUCCESSFUL_VISIT(MultiplicationOperator);
UNSUCCESSFUL_VISIT(DivisionOperator);
UNSUCCESSFUL_VISIT(ModOperator);
UNSUCCESSFUL_VISIT(NotEqualOperator);
UNSUCCESSFUL_VISIT(EqualOperator);
UNSUCCESSFUL_VISIT(LessOperator);
UNSUCCESSFUL_VISIT(GreaterOperator);
UNSUCCESSFUL_VISIT(LessEqualOperator);
UNSUCCESSFUL_VISIT(GreaterEqualOperator);
UNSUCCESSFUL_VISIT(NotOperator);
UNSUCCESSFUL_VISIT(UnaryPlusOperator);
UNSUCCESSFUL_VISIT(UnaryMinusOperator);
UNSUCCESSFUL_VISIT(AndOperator);
UNSUCCESSFUL_VISIT(IfOperator);
UNSUCCESSFUL_VISIT(InListOperator);
UNSUCCESSFUL_VISIT(SubscriptOperator);
UNSUCCESSFUL_VISIT(ListSlicingOperator);
UNSUCCESSFUL_VISIT(IsNullOperator);
UNSUCCESSFUL_VISIT(PropertyLookup);
UNSUCCESSFUL_VISIT(LabelsTest);
UNSUCCESSFUL_VISIT(PrimitiveLiteral);
UNSUCCESSFUL_VISIT(ListLiteral);
UNSUCCESSFUL_VISIT(MapLiteral);
UNSUCCESSFUL_VISIT(Aggregation);
UNSUCCESSFUL_VISIT(Coalesce);
UNSUCCESSFUL_VISIT(Function);
UNSUCCESSFUL_VISIT(Reduce);
UNSUCCESSFUL_VISIT(Extract);
UNSUCCESSFUL_VISIT(All);
UNSUCCESSFUL_VISIT(Single);
UNSUCCESSFUL_VISIT(Any);
UNSUCCESSFUL_VISIT(None);
UNSUCCESSFUL_VISIT(ParameterLookup);
UNSUCCESSFUL_VISIT(RegexMatch);
UNSUCCESSFUL_VISIT(Exists);
private:
Frame *frame_;
const SymbolTable *symbol_table_;
const EvaluationContext *ctx_;
};
class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
public:
ExpressionEvaluator(Frame *frame, const SymbolTable &symbol_table, const EvaluationContext &ctx, DbAccessor *dba,
@@ -159,50 +225,53 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
}
TypedValue Visit(SubscriptOperator &list_indexing) override {
auto lhs = list_indexing.expression1_->Accept(*this);
ReferenceExpressionEvaluator referenceExpressionEvaluator(frame_, symbol_table_, ctx_);
TypedValue *lhs_ptr = list_indexing.expression1_->Accept(referenceExpressionEvaluator);
TypedValue lhs;
const auto referenced = nullptr != lhs_ptr;
if (!referenced) {
lhs = list_indexing.expression1_->Accept(*this);
lhs_ptr = &lhs;
}
auto index = list_indexing.expression2_->Accept(*this);
if (!lhs.IsList() && !lhs.IsMap() && !lhs.IsVertex() && !lhs.IsEdge() && !lhs.IsNull())
if (!lhs_ptr->IsList() && !lhs_ptr->IsMap() && !lhs_ptr->IsVertex() && !lhs_ptr->IsEdge() && !lhs_ptr->IsNull())
throw QueryRuntimeException(
"Expected a list, a map, a node or an edge to index with '[]', got "
"{}.",
lhs.type());
if (lhs.IsNull() || index.IsNull()) return TypedValue(ctx_->memory);
if (lhs.IsList()) {
lhs_ptr->type());
if (lhs_ptr->IsNull() || index.IsNull()) return TypedValue(ctx_->memory);
if (lhs_ptr->IsList()) {
if (!index.IsInt()) throw QueryRuntimeException("Expected an integer as a list index, got {}.", index.type());
auto index_int = index.ValueInt();
// NOTE: Take non-const reference to list, so that we can move out the
// indexed element as the result.
auto &list = lhs.ValueList();
auto &list = lhs_ptr->ValueList();
if (index_int < 0) {
index_int += static_cast<int64_t>(list.size());
}
if (index_int >= static_cast<int64_t>(list.size()) || index_int < 0) return TypedValue(ctx_->memory);
// NOTE: Explicit move is needed, so that we return the move constructed
// value and preserve the correct MemoryResource.
return std::move(list[index_int]);
return referenced ? TypedValue(list[index_int], ctx_->memory)
: TypedValue(std::move(list[index_int]), ctx_->memory);
}
if (lhs.IsMap()) {
if (lhs_ptr->IsMap()) {
if (!index.IsString()) throw QueryRuntimeException("Expected a string as a map index, got {}.", index.type());
// NOTE: Take non-const reference to map, so that we can move out the
// looked-up element as the result.
auto &map = lhs.ValueMap();
auto &map = lhs_ptr->ValueMap();
auto found = map.find(index.ValueString());
if (found == map.end()) return TypedValue(ctx_->memory);
// NOTE: Explicit move is needed, so that we return the move constructed
// value and preserve the correct MemoryResource.
return std::move(found->second);
return referenced ? TypedValue(found->second, ctx_->memory) : TypedValue(std::move(found->second), ctx_->memory);
}
if (lhs.IsVertex()) {
if (lhs_ptr->IsVertex()) {
if (!index.IsString()) throw QueryRuntimeException("Expected a string as a property name, got {}.", index.type());
return TypedValue(GetProperty(lhs.ValueVertex(), index.ValueString()), ctx_->memory);
return {GetProperty(lhs_ptr->ValueVertex(), index.ValueString()), ctx_->memory};
}
if (lhs.IsEdge()) {
if (lhs_ptr->IsEdge()) {
if (!index.IsString()) throw QueryRuntimeException("Expected a string as a property name, got {}.", index.type());
return TypedValue(GetProperty(lhs.ValueEdge(), index.ValueString()), ctx_->memory);
}
return {GetProperty(lhs_ptr->ValueEdge(), index.ValueString()), ctx_->memory};
};
// lhs is Null
return TypedValue(ctx_->memory);
@@ -258,7 +327,15 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
}
TypedValue Visit(PropertyLookup &property_lookup) override {
auto expression_result = property_lookup.expression_->Accept(*this);
ReferenceExpressionEvaluator referenceExpressionEvaluator(frame_, symbol_table_, ctx_);
TypedValue *expression_result_ptr = property_lookup.expression_->Accept(referenceExpressionEvaluator);
TypedValue expression_result;
if (nullptr == expression_result_ptr) {
expression_result = property_lookup.expression_->Accept(*this);
expression_result_ptr = &expression_result;
}
auto maybe_date = [this](const auto &date, const auto &prop_name) -> std::optional<TypedValue> {
if (prop_name == "year") {
return TypedValue(date.year, ctx_->memory);
@@ -332,42 +409,38 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
}
return std::nullopt;
};
switch (expression_result.type()) {
switch (expression_result_ptr->type()) {
case TypedValue::Type::Null:
return TypedValue(ctx_->memory);
case TypedValue::Type::Vertex:
return TypedValue(GetProperty(expression_result.ValueVertex(), property_lookup.property_), ctx_->memory);
return TypedValue(GetProperty(expression_result_ptr->ValueVertex(), property_lookup.property_), ctx_->memory);
case TypedValue::Type::Edge:
return TypedValue(GetProperty(expression_result.ValueEdge(), property_lookup.property_), ctx_->memory);
return TypedValue(GetProperty(expression_result_ptr->ValueEdge(), property_lookup.property_), ctx_->memory);
case TypedValue::Type::Map: {
// NOTE: Take non-const reference to map, so that we can move out the
// looked-up element as the result.
auto &map = expression_result.ValueMap();
auto &map = expression_result_ptr->ValueMap();
auto found = map.find(property_lookup.property_.name.c_str());
if (found == map.end()) return TypedValue(ctx_->memory);
// NOTE: Explicit move is needed, so that we return the move constructed
// value and preserve the correct MemoryResource.
return std::move(found->second);
return TypedValue(found->second, ctx_->memory);
}
case TypedValue::Type::Duration: {
const auto &prop_name = property_lookup.property_.name;
const auto &dur = expression_result.ValueDuration();
const auto &dur = expression_result_ptr->ValueDuration();
if (auto dur_field = maybe_duration(dur, prop_name); dur_field) {
return std::move(*dur_field);
return TypedValue(*dur_field, ctx_->memory);
}
throw QueryRuntimeException("Invalid property name {} for Duration", prop_name);
}
case TypedValue::Type::Date: {
const auto &prop_name = property_lookup.property_.name;
const auto &date = expression_result.ValueDate();
const auto &date = expression_result_ptr->ValueDate();
if (auto date_field = maybe_date(date, prop_name); date_field) {
return std::move(*date_field);
return TypedValue(*date_field, ctx_->memory);
}
throw QueryRuntimeException("Invalid property name {} for Date", prop_name);
}
case TypedValue::Type::LocalTime: {
const auto &prop_name = property_lookup.property_.name;
const auto &lt = expression_result.ValueLocalTime();
const auto &lt = expression_result_ptr->ValueLocalTime();
if (auto lt_field = maybe_local_time(lt, prop_name); lt_field) {
return std::move(*lt_field);
}
@@ -375,20 +448,20 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
}
case TypedValue::Type::LocalDateTime: {
const auto &prop_name = property_lookup.property_.name;
const auto &ldt = expression_result.ValueLocalDateTime();
const auto &ldt = expression_result_ptr->ValueLocalDateTime();
if (auto date_field = maybe_date(ldt.date, prop_name); date_field) {
return std::move(*date_field);
}
if (auto lt_field = maybe_local_time(ldt.local_time, prop_name); lt_field) {
return std::move(*lt_field);
return TypedValue(*lt_field, ctx_->memory);
}
throw QueryRuntimeException("Invalid property name {} for LocalDateTime", prop_name);
}
case TypedValue::Type::Graph: {
const auto &prop_name = property_lookup.property_.name;
const auto &graph = expression_result.ValueGraph();
const auto &graph = expression_result_ptr->ValueGraph();
if (auto graph_field = maybe_graph(graph, prop_name); graph_field) {
return std::move(*graph_field);
return TypedValue(*graph_field, ctx_->memory);
}
throw QueryRuntimeException("Invalid property name {} for Graph", prop_name);
}
@@ -547,6 +620,8 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
return TypedValue(result, ctx_->memory);
}
TypedValue Visit(Exists &exists) override { return TypedValue{frame_->at(symbol_table_->at(exists)), ctx_->memory}; }
TypedValue Visit(All &all) override {
auto list_value = all.list_expression_->Accept(*this);
if (list_value.IsNull()) {

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -18,9 +18,13 @@
#include <cstddef>
#include <cstdint>
#include <functional>
#include <iterator>
#include <limits>
#include <memory>
#include <optional>
#include <thread>
#include <unordered_map>
#include <utility>
#include <variant>
#include "auth/models.hpp"
@@ -59,10 +63,12 @@
#include "utils/logging.hpp"
#include "utils/memory.hpp"
#include "utils/memory_tracker.hpp"
#include "utils/on_scope_exit.hpp"
#include "utils/readable_size.hpp"
#include "utils/settings.hpp"
#include "utils/string.hpp"
#include "utils/tsc.hpp"
#include "utils/typeinfo.hpp"
#include "utils/variant_helpers.hpp"
namespace EventCounter {
@@ -327,11 +333,11 @@ Callback HandleAuthQuery(AuthQuery *auth_query, AuthQueryHandler *auth, const Pa
auth->GrantPrivilege(username, kPrivilegesAll
#ifdef MG_ENTERPRISE
,
{{{AuthQuery::FineGrainedPrivilege::CREATE_DELETE, {auth::kAsterisk}}}},
{{{AuthQuery::FineGrainedPrivilege::CREATE_DELETE, {query::kAsterisk}}}},
{
{
{
AuthQuery::FineGrainedPrivilege::CREATE_DELETE, { auth::kAsterisk }
AuthQuery::FineGrainedPrivilege::CREATE_DELETE, { query::kAsterisk }
}
}
}
@@ -975,8 +981,10 @@ struct PullPlanVector {
struct PullPlan {
explicit PullPlan(std::shared_ptr<CachedPlan> plan, const Parameters &parameters, bool is_profile_query,
DbAccessor *dba, InterpreterContext *interpreter_context, utils::MemoryResource *execution_memory,
std::optional<std::string> username, TriggerContextCollector *trigger_context_collector = nullptr,
std::optional<size_t> memory_limit = {});
std::optional<std::string> username, std::atomic<TransactionStatus> *transaction_status,
TriggerContextCollector *trigger_context_collector = nullptr,
std::optional<size_t> memory_limit = {}, bool use_monotonic_memory = true);
std::optional<plan::ProfilingStatsWithTotalTime> Pull(AnyStream *stream, std::optional<int> n,
const std::vector<Symbol> &output_symbols,
std::map<std::string, TypedValue> *summary);
@@ -1000,16 +1008,25 @@ struct PullPlan {
// we have to keep track of any unsent results from previous `PullPlan::Pull`
// manually by using this flag.
bool has_unsent_results_ = false;
// In the case of LOAD CSV, we want to use only PoolResource without MonotonicMemoryResource
// to reuse allocated memory. As LOAD CSV is processing row by row
// it is possible to reduce memory usage significantly if MemoryResource deals with memory allocation
// can reuse memory that was allocated on processing the first row on all subsequent rows.
// This flag signals to `PullPlan::Pull` which MemoryResource to use
bool use_monotonic_memory_;
};
PullPlan::PullPlan(const std::shared_ptr<CachedPlan> plan, const Parameters &parameters, const bool is_profile_query,
DbAccessor *dba, InterpreterContext *interpreter_context, utils::MemoryResource *execution_memory,
std::optional<std::string> username, TriggerContextCollector *trigger_context_collector,
const std::optional<size_t> memory_limit)
std::optional<std::string> username, std::atomic<TransactionStatus> *transaction_status,
TriggerContextCollector *trigger_context_collector, const std::optional<size_t> memory_limit,
bool use_monotonic_memory)
: plan_(plan),
cursor_(plan->plan().MakeCursor(execution_memory)),
frame_(plan->symbol_table().max_position(), execution_memory),
memory_limit_(memory_limit) {
memory_limit_(memory_limit),
use_monotonic_memory_(use_monotonic_memory) {
ctx_.db_accessor = dba;
ctx_.symbol_table = plan->symbol_table();
ctx_.evaluation_context.timestamp = QueryTimestamp();
@@ -1018,13 +1035,21 @@ PullPlan::PullPlan(const std::shared_ptr<CachedPlan> plan, const Parameters &par
ctx_.evaluation_context.labels = NamesToLabels(plan->ast_storage().labels_, dba);
#ifdef MG_ENTERPRISE
if (license::global_license_checker.IsEnterpriseValidFast() && username.has_value() && dba) {
ctx_.auth_checker = interpreter_context->auth_checker->GetFineGrainedAuthChecker(*username, dba);
auto auth_checker = interpreter_context->auth_checker->GetFineGrainedAuthChecker(*username, dba);
// if the user has global privileges to read, edit and write anything, we don't need to perform authorization
// otherwise, we do assign the auth checker to check for label access control
if (!auth_checker->HasGlobalPrivilegeOnVertices(AuthQuery::FineGrainedPrivilege::CREATE_DELETE) ||
!auth_checker->HasGlobalPrivilegeOnEdges(AuthQuery::FineGrainedPrivilege::CREATE_DELETE)) {
ctx_.auth_checker = std::move(auth_checker);
}
}
#endif
if (interpreter_context->config.execution_timeout_sec > 0) {
ctx_.timer = utils::AsyncTimer{interpreter_context->config.execution_timeout_sec};
}
ctx_.is_shutting_down = &interpreter_context->is_shutting_down;
ctx_.transaction_status = transaction_status;
ctx_.is_profile_query = is_profile_query;
ctx_.trigger_context_collector = trigger_context_collector;
}
@@ -1037,21 +1062,28 @@ std::optional<plan::ProfilingStatsWithTotalTime> PullPlan::Pull(AnyStream *strea
// single `Pull`.
static constexpr size_t stack_size = 256UL * 1024UL;
char stack_data[stack_size];
utils::ResourceWithOutOfMemoryException resource_with_exception;
utils::MonotonicBufferResource monotonic_memory(&stack_data[0], stack_size, &resource_with_exception);
// We can throw on every query because a simple queries for deleting will use only
// the stack allocated buffer.
// Also, we want to throw only when the query engine requests more memory and not the storage
// so we add the exception to the allocator.
// TODO (mferencevic): Tune the parameters accordingly.
utils::PoolResource pool_memory(128, 1024, &monotonic_memory);
std::optional<utils::LimitedMemoryResource> maybe_limited_resource;
utils::ResourceWithOutOfMemoryException resource_with_exception;
utils::MonotonicBufferResource monotonic_memory{&stack_data[0], stack_size, &resource_with_exception};
std::optional<utils::PoolResource> pool_memory;
if (!use_monotonic_memory_) {
pool_memory.emplace(8, kExecutionPoolMaxBlockSize, &resource_with_exception, &resource_with_exception);
} else {
// We can throw on every query because a simple queries for deleting will use only
// the stack allocated buffer.
// Also, we want to throw only when the query engine requests more memory and not the storage
// so we add the exception to the allocator.
// TODO (mferencevic): Tune the parameters accordingly.
pool_memory.emplace(128, 1024, &monotonic_memory, &resource_with_exception);
}
std::optional<utils::LimitedMemoryResource> maybe_limited_resource;
if (memory_limit_) {
maybe_limited_resource.emplace(&pool_memory, *memory_limit_);
maybe_limited_resource.emplace(&*pool_memory, *memory_limit_);
ctx_.evaluation_context.memory = &*maybe_limited_resource;
} else {
ctx_.evaluation_context.memory = &pool_memory;
ctx_.evaluation_context.memory = &*pool_memory;
}
// Returns true if a result was pulled.
@@ -1143,6 +1175,7 @@ PreparedQuery Interpreter::PrepareTransactionQuery(std::string_view query_upper)
db_accessor_ =
std::make_unique<storage::Storage::Accessor>(interpreter_context_->db->Access(GetIsolationLevelOverride()));
execution_db_accessor_.emplace(db_accessor_.get());
transaction_status_.store(TransactionStatus::ACTIVE, std::memory_order_release);
if (interpreter_context_->trigger_store.HasTriggers()) {
trigger_context_collector_.emplace(interpreter_context_->trigger_store.GetEventTypes());
@@ -1194,7 +1227,7 @@ PreparedQuery Interpreter::PrepareTransactionQuery(std::string_view query_upper)
PreparedQuery PrepareCypherQuery(ParsedQuery parsed_query, std::map<std::string, TypedValue> *summary,
InterpreterContext *interpreter_context, DbAccessor *dba,
utils::MemoryResource *execution_memory, std::vector<Notification> *notifications,
const std::string *username,
const std::string *username, std::atomic<TransactionStatus> *transaction_status,
TriggerContextCollector *trigger_context_collector = nullptr) {
auto *cypher_query = utils::Downcast<CypherQuery>(parsed_query.query);
@@ -1209,16 +1242,19 @@ PreparedQuery PrepareCypherQuery(ParsedQuery parsed_query, std::map<std::string,
if (memory_limit) {
spdlog::info("Running query with memory limit of {}", utils::GetReadableSize(*memory_limit));
}
if (const auto &clauses = cypher_query->single_query_->clauses_; std::any_of(
clauses.begin(), clauses.end(), [](const auto *clause) { return clause->GetTypeInfo() == LoadCsv::kType; })) {
auto clauses = cypher_query->single_query_->clauses_;
bool contains_csv = false;
if (std::any_of(clauses.begin(), clauses.end(),
[](const auto *clause) { return clause->GetTypeInfo() == LoadCsv::kType; })) {
notifications->emplace_back(
SeverityLevel::INFO, NotificationCode::LOAD_CSV_TIP,
"It's important to note that the parser parses the values as strings. It's up to the user to "
"convert the parsed row values to the appropriate type. This can be done using the built-in "
"conversion functions such as ToInteger, ToFloat, ToBoolean etc.");
contains_csv = true;
}
// If this is LOAD CSV query, use PoolResource without MonotonicMemoryResource as we want to reuse allocated memory
auto use_monotonic_memory = !contains_csv;
auto plan = CypherQueryToPlan(parsed_query.stripped_query.hash(), std::move(parsed_query.ast_storage), cypher_query,
parsed_query.parameters,
parsed_query.is_cacheable ? &interpreter_context->plan_cache : nullptr, dba);
@@ -1239,9 +1275,9 @@ PreparedQuery PrepareCypherQuery(ParsedQuery parsed_query, std::map<std::string,
header.push_back(
utils::FindOr(parsed_query.stripped_query.named_expressions(), symbol.token_position(), symbol.name()).first);
}
auto pull_plan =
std::make_shared<PullPlan>(plan, parsed_query.parameters, false, dba, interpreter_context, execution_memory,
StringPointerToOptional(username), trigger_context_collector, memory_limit);
auto pull_plan = std::make_shared<PullPlan>(plan, parsed_query.parameters, false, dba, interpreter_context,
execution_memory, StringPointerToOptional(username), transaction_status,
trigger_context_collector, memory_limit, use_monotonic_memory);
return PreparedQuery{std::move(header), std::move(parsed_query.required_privileges),
[pull_plan = std::move(pull_plan), output_symbols = std::move(output_symbols), summary](
AnyStream *stream, std::optional<int> n) -> std::optional<QueryHandlerResult> {
@@ -1301,8 +1337,8 @@ PreparedQuery PrepareExplainQuery(ParsedQuery parsed_query, std::map<std::string
PreparedQuery PrepareProfileQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
std::map<std::string, TypedValue> *summary, InterpreterContext *interpreter_context,
DbAccessor *dba, utils::MemoryResource *execution_memory,
const std::string *username) {
DbAccessor *dba, utils::MemoryResource *execution_memory, const std::string *username,
std::atomic<TransactionStatus> *transaction_status) {
const std::string kProfileQueryStart = "profile ";
MG_ASSERT(utils::StartsWith(utils::ToLowerCase(parsed_query.stripped_query.query()), kProfileQueryStart),
@@ -1339,6 +1375,16 @@ PreparedQuery PrepareProfileQuery(ParsedQuery parsed_query, bool in_explicit_tra
&interpreter_context->ast_cache, interpreter_context->config.query);
auto *cypher_query = utils::Downcast<CypherQuery>(parsed_inner_query.query);
bool contains_csv = false;
auto clauses = cypher_query->single_query_->clauses_;
if (std::any_of(clauses.begin(), clauses.end(),
[](const auto *clause) { return clause->GetTypeInfo() == LoadCsv::kType; })) {
contains_csv = true;
}
// If this is LOAD CSV query, use PoolResource without MonotonicMemoryResource as we want to reuse allocated memory
auto use_monotonic_memory = !contains_csv;
MG_ASSERT(cypher_query, "Cypher grammar should not allow other queries in PROFILE");
Frame frame(0);
SymbolTable symbol_table;
@@ -1363,12 +1409,13 @@ PreparedQuery PrepareProfileQuery(ParsedQuery parsed_query, bool in_explicit_tra
// We want to execute the query we are profiling lazily, so we delay
// the construction of the corresponding context.
stats_and_total_time = std::optional<plan::ProfilingStatsWithTotalTime>{},
pull_plan = std::shared_ptr<PullPlanVector>(nullptr)](
pull_plan = std::shared_ptr<PullPlanVector>(nullptr), transaction_status, use_monotonic_memory](
AnyStream *stream, std::optional<int> n) mutable -> std::optional<QueryHandlerResult> {
// No output symbols are given so that nothing is streamed.
if (!stats_and_total_time) {
stats_and_total_time = PullPlan(plan, parameters, true, dba, interpreter_context,
execution_memory, optional_username, nullptr, memory_limit)
execution_memory, optional_username, transaction_status,
nullptr, memory_limit, use_monotonic_memory)
.Pull(stream, {}, {}, summary);
pull_plan = std::make_shared<PullPlanVector>(ProfilingStatsToTable(*stats_and_total_time));
}
@@ -1399,6 +1446,139 @@ PreparedQuery PrepareDumpQuery(ParsedQuery parsed_query, std::map<std::string, T
RWType::R};
}
std::vector<std::vector<TypedValue>> AnalyzeGraphQueryHandler::AnalyzeGraphCreateStatistics(
const std::span<std::string> labels, DbAccessor *execution_db_accessor) {
using LPIndex = std::pair<storage::LabelId, storage::PropertyId>;
std::vector<std::vector<TypedValue>> results;
std::map<LPIndex, std::map<storage::PropertyValue, int64_t>> counter;
// Preprocess labels to avoid later checks
std::vector<LPIndex> indices_info = execution_db_accessor->ListAllIndices().label_property;
if (labels[0] != kAsterisk) {
for (auto it = indices_info.cbegin(); it != indices_info.cend();) {
if (std::find(labels.begin(), labels.end(), execution_db_accessor->LabelToName(it->first)) == labels.end()) {
it = indices_info.erase(it);
} else {
++it;
}
}
}
// Iterate over all indexed vertices
std::for_each(indices_info.begin(), indices_info.end(), [execution_db_accessor, &counter](const LPIndex &index_info) {
auto vertices = execution_db_accessor->Vertices(storage::View::OLD, index_info.first, index_info.second);
std::for_each(vertices.begin(), vertices.end(), [&index_info, &counter](const auto &vertex) {
counter[index_info][*vertex.GetProperty(storage::View::OLD, index_info.second)]++;
});
});
results.reserve(counter.size());
std::for_each(counter.begin(), counter.end(), [&results, execution_db_accessor](const auto &counter_entry) {
const auto &[label_property, values_map] = counter_entry;
std::vector<TypedValue> result;
result.reserve(kDeleteStatisticsNumResults);
// Extract info
int64_t count_property_value = std::accumulate(
values_map.begin(), values_map.end(), 0,
[](int64_t prev_value, const auto &prop_value_count) { return prev_value + prop_value_count.second; });
// num_distinc_values will never be 0
double avg_group_size = static_cast<double>(count_property_value) / static_cast<double>(values_map.size());
double chi_squared_stat = std::accumulate(
values_map.begin(), values_map.end(), 0.0, [avg_group_size](double prev_result, const auto &value_entry) {
return prev_result + utils::ChiSquaredValue(value_entry.second, avg_group_size);
});
execution_db_accessor->SetIndexStats(
label_property.first, label_property.second,
storage::IndexStats{.statistic = chi_squared_stat, .avg_group_size = avg_group_size});
// Save result
result.emplace_back(execution_db_accessor->LabelToName(label_property.first));
result.emplace_back(execution_db_accessor->PropertyToName(label_property.second));
result.emplace_back(count_property_value);
result.emplace_back(static_cast<int64_t>(values_map.size()));
result.emplace_back(avg_group_size);
result.emplace_back(chi_squared_stat);
results.push_back(std::move(result));
});
return results;
}
std::vector<std::vector<TypedValue>> AnalyzeGraphQueryHandler::AnalyzeGraphDeleteStatistics(
const std::span<std::string> labels, DbAccessor *execution_db_accessor) {
std::vector<std::pair<storage::LabelId, storage::PropertyId>> loc_results;
if (labels[0] == kAsterisk) {
loc_results = execution_db_accessor->ClearIndexStats();
} else {
loc_results = execution_db_accessor->DeleteIndexStatsForLabels(labels);
}
std::vector<std::vector<TypedValue>> results;
std::transform(loc_results.begin(), loc_results.end(), std::back_inserter(results),
[execution_db_accessor](const auto &label_property_index) {
return std::vector<TypedValue>{
TypedValue(execution_db_accessor->LabelToName(label_property_index.first)),
TypedValue(execution_db_accessor->PropertyToName(label_property_index.second))};
});
return results;
}
Callback HandleAnalyzeGraphQuery(AnalyzeGraphQuery *analyze_graph_query, DbAccessor *execution_db_accessor) {
Callback callback;
switch (analyze_graph_query->action_) {
case AnalyzeGraphQuery::Action::ANALYZE: {
callback.header = {"label", "property", "num estimation nodes",
"num groups", "avg group size", "chi-squared value"};
callback.fn = [handler = AnalyzeGraphQueryHandler(), labels = analyze_graph_query->labels_,
execution_db_accessor]() mutable {
return handler.AnalyzeGraphCreateStatistics(labels, execution_db_accessor);
};
break;
}
case AnalyzeGraphQuery::Action::DELETE: {
callback.header = {"label", "property"};
callback.fn = [handler = AnalyzeGraphQueryHandler(), labels = analyze_graph_query->labels_,
execution_db_accessor]() mutable {
return handler.AnalyzeGraphDeleteStatistics(labels, execution_db_accessor);
};
break;
}
}
return callback;
}
PreparedQuery PrepareAnalyzeGraphQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
DbAccessor *execution_db_accessor, InterpreterContext *interpreter_context) {
if (in_explicit_transaction) {
throw AnalyzeGraphInMulticommandTxException();
}
// Creating an index influences computed plan costs.
auto invalidate_plan_cache = [plan_cache = &interpreter_context->plan_cache] {
auto access = plan_cache->access();
for (auto &kv : access) {
access.remove(kv.first);
}
};
utils::OnScopeExit cache_invalidator(invalidate_plan_cache);
auto *analyze_graph_query = utils::Downcast<AnalyzeGraphQuery>(parsed_query.query);
MG_ASSERT(analyze_graph_query);
auto callback = HandleAnalyzeGraphQuery(analyze_graph_query, execution_db_accessor);
return PreparedQuery{std::move(callback.header), std::move(parsed_query.required_privileges),
[callback_fn = std::move(callback.fn), pull_plan = std::shared_ptr<PullPlanVector>{nullptr}](
AnyStream *stream, std::optional<int> n) mutable -> std::optional<QueryHandlerResult> {
if (UNLIKELY(!pull_plan)) {
pull_plan = std::make_shared<PullPlanVector>(callback_fn());
}
if (pull_plan->Pull(stream, n)) {
return QueryHandlerResult::COMMIT;
}
return std::nullopt;
},
RWType::NONE};
}
PreparedQuery PrepareIndexQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
std::vector<Notification> *notifications, InterpreterContext *interpreter_context) {
if (in_explicit_transaction) {
@@ -1524,7 +1704,8 @@ PreparedQuery PrepareIndexQuery(ParsedQuery parsed_query, bool in_explicit_trans
PreparedQuery PrepareAuthQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
std::map<std::string, TypedValue> *summary, InterpreterContext *interpreter_context,
DbAccessor *dba, utils::MemoryResource *execution_memory, const std::string *username) {
DbAccessor *dba, utils::MemoryResource *execution_memory, const std::string *username,
std::atomic<TransactionStatus> *transaction_status) {
if (in_explicit_transaction) {
throw UserModificationInMulticommandTxException();
}
@@ -1545,7 +1726,7 @@ PreparedQuery PrepareAuthQuery(ParsedQuery parsed_query, bool in_explicit_transa
0.0, AstStorage{}, symbol_table));
auto pull_plan = std::make_shared<PullPlan>(plan, parsed_query.parameters, false, dba, interpreter_context,
execution_memory, StringPointerToOptional(username));
execution_memory, StringPointerToOptional(username), transaction_status);
return PreparedQuery{
callback.header, std::move(parsed_query.required_privileges),
[pull_plan = std::move(pull_plan), callback = std::move(callback), output_symbols = std::move(output_symbols),
@@ -1558,7 +1739,7 @@ PreparedQuery PrepareAuthQuery(ParsedQuery parsed_query, bool in_explicit_transa
RWType::NONE};
}
PreparedQuery PrepareReplicationQuery(ParsedQuery parsed_query, const bool in_explicit_transaction,
PreparedQuery PrepareReplicationQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
std::vector<Notification> *notifications, InterpreterContext *interpreter_context,
DbAccessor *dba) {
if (in_explicit_transaction) {
@@ -1586,7 +1767,7 @@ PreparedQuery PrepareReplicationQuery(ParsedQuery parsed_query, const bool in_ex
// NOLINTNEXTLINE(clang-analyzer-cplusplus.NewDeleteLeaks)
}
PreparedQuery PrepareLockPathQuery(ParsedQuery parsed_query, const bool in_explicit_transaction,
PreparedQuery PrepareLockPathQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
InterpreterContext *interpreter_context, DbAccessor *dba) {
if (in_explicit_transaction) {
throw LockPathModificationInMulticommandTxException();
@@ -1615,7 +1796,7 @@ PreparedQuery PrepareLockPathQuery(ParsedQuery parsed_query, const bool in_expli
RWType::NONE};
}
PreparedQuery PrepareFreeMemoryQuery(ParsedQuery parsed_query, const bool in_explicit_transaction,
PreparedQuery PrepareFreeMemoryQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
InterpreterContext *interpreter_context) {
if (in_explicit_transaction) {
throw FreeMemoryModificationInMulticommandTxException();
@@ -1632,7 +1813,7 @@ PreparedQuery PrepareFreeMemoryQuery(ParsedQuery parsed_query, const bool in_exp
RWType::NONE};
}
PreparedQuery PrepareShowConfigQuery(ParsedQuery parsed_query, const bool in_explicit_transaction) {
PreparedQuery PrepareShowConfigQuery(ParsedQuery parsed_query, bool in_explicit_transaction) {
if (in_explicit_transaction) {
throw ShowConfigModificationInMulticommandTxException();
}
@@ -1736,7 +1917,7 @@ Callback ShowTriggers(InterpreterContext *interpreter_context) {
}};
}
PreparedQuery PrepareTriggerQuery(ParsedQuery parsed_query, const bool in_explicit_transaction,
PreparedQuery PrepareTriggerQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
std::vector<Notification> *notifications, InterpreterContext *interpreter_context,
DbAccessor *dba, const std::map<std::string, storage::PropertyValue> &user_parameters,
const std::string *username) {
@@ -1786,7 +1967,7 @@ PreparedQuery PrepareTriggerQuery(ParsedQuery parsed_query, const bool in_explic
// NOLINTNEXTLINE(clang-analyzer-cplusplus.NewDeleteLeaks)
}
PreparedQuery PrepareStreamQuery(ParsedQuery parsed_query, const bool in_explicit_transaction,
PreparedQuery PrepareStreamQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
std::vector<Notification> *notifications, InterpreterContext *interpreter_context,
DbAccessor *dba,
const std::map<std::string, storage::PropertyValue> & /*user_parameters*/,
@@ -1828,6 +2009,15 @@ constexpr auto ToStorageIsolationLevel(const IsolationLevelQuery::IsolationLevel
}
}
constexpr auto ToStorageMode(const StorageModeQuery::StorageMode storage_mode) noexcept {
switch (storage_mode) {
case StorageModeQuery::StorageMode::IN_MEMORY_TRANSACTIONAL:
return storage::StorageMode::IN_MEMORY_TRANSACTIONAL;
case StorageModeQuery::StorageMode::IN_MEMORY_ANALYTICAL:
return storage::StorageMode::IN_MEMORY_ANALYTICAL;
}
}
PreparedQuery PrepareIsolationLevelQuery(ParsedQuery parsed_query, const bool in_explicit_transaction,
InterpreterContext *interpreter_context, Interpreter *interpreter) {
if (in_explicit_transaction) {
@@ -1843,7 +2033,15 @@ PreparedQuery PrepareIsolationLevelQuery(ParsedQuery parsed_query, const bool in
interpreter]() -> std::function<void()> {
switch (isolation_level_query->isolation_level_scope_) {
case IsolationLevelQuery::IsolationLevelScope::GLOBAL:
return [interpreter_context, isolation_level] { interpreter_context->db->SetIsolationLevel(isolation_level); };
return [interpreter_context, isolation_level] {
if (auto maybe_error = interpreter_context->db->SetIsolationLevel(isolation_level); maybe_error.HasError()) {
switch (maybe_error.GetError()) {
case storage::Storage::SetIsolationLevelError::DisabledForAnalyticalMode:
throw IsolationLevelModificationInAnalyticsException();
break;
}
}
};
case IsolationLevelQuery::IsolationLevelScope::SESSION:
return [interpreter, isolation_level] { interpreter->SetSessionIsolationLevel(isolation_level); };
case IsolationLevelQuery::IsolationLevelScope::NEXT:
@@ -1861,6 +2059,41 @@ PreparedQuery PrepareIsolationLevelQuery(ParsedQuery parsed_query, const bool in
RWType::NONE};
}
PreparedQuery PrepareStorageModeQuery(ParsedQuery parsed_query, const bool in_explicit_transaction,
InterpreterContext *interpreter_context) {
if (in_explicit_transaction) {
throw StorageModeModificationInMulticommandTxException();
}
auto *storage_mode_query = utils::Downcast<StorageModeQuery>(parsed_query.query);
MG_ASSERT(storage_mode_query);
const auto storage_mode = ToStorageMode(storage_mode_query->storage_mode_);
auto exists_active_transaction = interpreter_context->interpreters.WithLock([](const auto &interpreters_) {
return std::any_of(interpreters_.begin(), interpreters_.end(), [](const auto &interpreter) {
return interpreter->transaction_status_.load() != TransactionStatus::IDLE;
});
});
if (exists_active_transaction) {
spdlog::info(
"Storage mode will be modified when there are no other active transactions. Check the status of the "
"transactions using 'SHOW TRANSACTIONS' query and ensure no other transactions are active.");
}
auto callback = [storage_mode, interpreter_context]() -> std::function<void()> {
return [interpreter_context, storage_mode] { interpreter_context->db->SetStorageMode(storage_mode); };
}();
return PreparedQuery{{},
std::move(parsed_query.required_privileges),
[callback = std::move(callback)](AnyStream * /*stream*/,
std::optional<int> /*n*/) -> std::optional<QueryHandlerResult> {
callback();
return QueryHandlerResult::COMMIT;
},
RWType::NONE};
}
PreparedQuery PrepareCreateSnapshotQuery(ParsedQuery parsed_query, bool in_explicit_transaction,
InterpreterContext *interpreter_context) {
if (in_explicit_transaction) {
@@ -1871,11 +2104,18 @@ PreparedQuery PrepareCreateSnapshotQuery(ParsedQuery parsed_query, bool in_expli
{},
std::move(parsed_query.required_privileges),
[interpreter_context](AnyStream *stream, std::optional<int> n) -> std::optional<QueryHandlerResult> {
if (auto maybe_error = interpreter_context->db->CreateSnapshot(); maybe_error.HasError()) {
if (auto maybe_error = interpreter_context->db->CreateSnapshot({}); maybe_error.HasError()) {
switch (maybe_error.GetError()) {
case storage::Storage::CreateSnapshotError::DisabledForReplica:
throw utils::BasicException(
"Failed to create a snapshot. Replica instances are not allowed to create them.");
case storage::Storage::CreateSnapshotError::DisabledForAnalyticsPeriodicCommit:
spdlog::warn(utils::MessageWithLink("Periodic snapshots are disabled for analytical mode.",
"https://memgr.ph/replication"));
break;
case storage::Storage::CreateSnapshotError::ReachedMaxNumTries:
spdlog::warn("Failed to create snapshot. Reached max number of tries. Please contact support");
break;
}
}
return QueryHandlerResult::COMMIT;
@@ -1883,7 +2123,7 @@ PreparedQuery PrepareCreateSnapshotQuery(ParsedQuery parsed_query, bool in_expli
RWType::NONE};
}
PreparedQuery PrepareSettingQuery(ParsedQuery parsed_query, const bool in_explicit_transaction, DbAccessor *dba) {
PreparedQuery PrepareSettingQuery(ParsedQuery parsed_query, bool in_explicit_transaction, DbAccessor *dba) {
if (in_explicit_transaction) {
throw SettingConfigInMulticommandTxException{};
}
@@ -1909,7 +2149,155 @@ PreparedQuery PrepareSettingQuery(ParsedQuery parsed_query, const bool in_explic
// NOLINTNEXTLINE(clang-analyzer-cplusplus.NewDeleteLeaks)
}
PreparedQuery PrepareVersionQuery(ParsedQuery parsed_query, const bool in_explicit_transaction) {
std::vector<std::vector<TypedValue>> TransactionQueueQueryHandler::ShowTransactions(
const std::unordered_set<Interpreter *> &interpreters, const std::optional<std::string> &username,
bool hasTransactionManagementPrivilege) {
std::vector<std::vector<TypedValue>> results;
results.reserve(interpreters.size());
for (Interpreter *interpreter : interpreters) {
TransactionStatus alive_status = TransactionStatus::ACTIVE;
// if it is just checking status, commit and abort should wait for the end of the check
// ignore interpreters that already started committing or rollback
if (!interpreter->transaction_status_.compare_exchange_strong(alive_status, TransactionStatus::VERIFYING)) {
continue;
}
utils::OnScopeExit clean_status([interpreter]() {
interpreter->transaction_status_.store(TransactionStatus::ACTIVE, std::memory_order_release);
});
std::optional<uint64_t> transaction_id = interpreter->GetTransactionId();
if (transaction_id.has_value() && (interpreter->username_ == username || hasTransactionManagementPrivilege)) {
const auto &typed_queries = interpreter->GetQueries();
results.push_back({TypedValue(interpreter->username_.value_or("")),
TypedValue(std::to_string(transaction_id.value())), TypedValue(typed_queries)});
}
}
return results;
}
std::vector<std::vector<TypedValue>> TransactionQueueQueryHandler::KillTransactions(
InterpreterContext *interpreter_context, const std::vector<std::string> &maybe_kill_transaction_ids,
const std::optional<std::string> &username, bool hasTransactionManagementPrivilege) {
std::vector<std::vector<TypedValue>> results;
for (const std::string &transaction_id : maybe_kill_transaction_ids) {
bool killed = false;
bool transaction_found = false;
// Multiple simultaneous TERMINATE TRANSACTIONS aren't allowed
// TERMINATE and SHOW TRANSACTIONS are mutually exclusive
interpreter_context->interpreters.WithLock([&transaction_id, &killed, &transaction_found, username,
hasTransactionManagementPrivilege](const auto &interpreters) {
for (Interpreter *interpreter : interpreters) {
TransactionStatus alive_status = TransactionStatus::ACTIVE;
// if it is just checking kill, commit and abort should wait for the end of the check
// The only way to start checking if the transaction will get killed is if the transaction_status is
// active
if (!interpreter->transaction_status_.compare_exchange_strong(alive_status, TransactionStatus::VERIFYING)) {
continue;
}
utils::OnScopeExit clean_status([interpreter, &killed]() {
if (killed) {
interpreter->transaction_status_.store(TransactionStatus::TERMINATED, std::memory_order_release);
} else {
interpreter->transaction_status_.store(TransactionStatus::ACTIVE, std::memory_order_release);
}
});
std::optional<uint64_t> intr_trans = interpreter->GetTransactionId();
if (intr_trans.has_value() && std::to_string(intr_trans.value()) == transaction_id) {
transaction_found = true;
if (interpreter->username_ == username || hasTransactionManagementPrivilege) {
killed = true;
spdlog::warn("Transaction {} successfully killed", transaction_id);
} else {
spdlog::warn("Not enough rights to kill the transaction");
}
break;
}
}
});
if (!transaction_found) {
spdlog::warn("Transaction {} not found", transaction_id);
}
results.push_back({TypedValue(transaction_id), TypedValue(killed)});
}
return results;
}
Callback HandleTransactionQueueQuery(TransactionQueueQuery *transaction_query,
const std::optional<std::string> &username, const Parameters &parameters,
InterpreterContext *interpreter_context, DbAccessor *db_accessor) {
Frame frame(0);
SymbolTable symbol_table;
EvaluationContext evaluation_context;
evaluation_context.timestamp = QueryTimestamp();
evaluation_context.parameters = parameters;
ExpressionEvaluator evaluator(&frame, symbol_table, evaluation_context, db_accessor, storage::View::OLD);
bool hasTransactionManagementPrivilege = interpreter_context->auth_checker->IsUserAuthorized(
username, {query::AuthQuery::Privilege::TRANSACTION_MANAGEMENT});
Callback callback;
switch (transaction_query->action_) {
case TransactionQueueQuery::Action::SHOW_TRANSACTIONS: {
callback.header = {"username", "transaction_id", "query"};
callback.fn = [handler = TransactionQueueQueryHandler(), interpreter_context, username,
hasTransactionManagementPrivilege]() mutable {
std::vector<std::vector<TypedValue>> results;
// Multiple simultaneous SHOW TRANSACTIONS aren't allowed
interpreter_context->interpreters.WithLock(
[&results, handler, username, hasTransactionManagementPrivilege](const auto &interpreters) {
results = handler.ShowTransactions(interpreters, username, hasTransactionManagementPrivilege);
});
return results;
};
break;
}
case TransactionQueueQuery::Action::TERMINATE_TRANSACTIONS: {
std::vector<std::string> maybe_kill_transaction_ids;
std::transform(transaction_query->transaction_id_list_.begin(), transaction_query->transaction_id_list_.end(),
std::back_inserter(maybe_kill_transaction_ids), [&evaluator](Expression *expression) {
return std::string(expression->Accept(evaluator).ValueString());
});
callback.header = {"transaction_id", "killed"};
callback.fn = [handler = TransactionQueueQueryHandler(), interpreter_context, maybe_kill_transaction_ids,
username, hasTransactionManagementPrivilege]() mutable {
return handler.KillTransactions(interpreter_context, maybe_kill_transaction_ids, username,
hasTransactionManagementPrivilege);
};
break;
}
}
return callback;
}
PreparedQuery PrepareTransactionQueueQuery(ParsedQuery parsed_query, const std::optional<std::string> &username,
bool in_explicit_transaction, InterpreterContext *interpreter_context,
DbAccessor *dba) {
if (in_explicit_transaction) {
throw TransactionQueueInMulticommandTxException();
}
auto *transaction_queue_query = utils::Downcast<TransactionQueueQuery>(parsed_query.query);
MG_ASSERT(transaction_queue_query);
auto callback =
HandleTransactionQueueQuery(transaction_queue_query, username, parsed_query.parameters, interpreter_context, dba);
return PreparedQuery{std::move(callback.header), std::move(parsed_query.required_privileges),
[callback_fn = std::move(callback.fn), pull_plan = std::shared_ptr<PullPlanVector>{nullptr}](
AnyStream *stream, std::optional<int> n) mutable -> std::optional<QueryHandlerResult> {
if (UNLIKELY(!pull_plan)) {
pull_plan = std::make_shared<PullPlanVector>(callback_fn());
}
if (pull_plan->Pull(stream, n)) {
return QueryHandlerResult::COMMIT;
}
return std::nullopt;
},
RWType::NONE};
}
PreparedQuery PrepareVersionQuery(ParsedQuery parsed_query, bool in_explicit_transaction) {
if (in_explicit_transaction) {
throw VersionInfoInMulticommandTxException();
}
@@ -2263,6 +2651,13 @@ PreparedQuery PrepareConstraintQuery(ParsedQuery parsed_query, bool in_explicit_
RWType::NONE};
}
std::optional<uint64_t> Interpreter::GetTransactionId() const {
if (db_accessor_) {
return db_accessor_->GetTransactionId();
}
return {};
}
void Interpreter::BeginTransaction() {
const auto prepared_query = PrepareTransactionQuery("BEGIN");
prepared_query.query_handler(nullptr, {});
@@ -2272,12 +2667,14 @@ void Interpreter::CommitTransaction() {
const auto prepared_query = PrepareTransactionQuery("COMMIT");
prepared_query.query_handler(nullptr, {});
query_executions_.clear();
transaction_queries_->clear();
}
void Interpreter::RollbackTransaction() {
const auto prepared_query = PrepareTransactionQuery("ROLLBACK");
prepared_query.query_handler(nullptr, {});
query_executions_.clear();
transaction_queries_->clear();
}
Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
@@ -2285,12 +2682,13 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
const std::string *username) {
if (!in_explicit_transaction_) {
query_executions_.clear();
transaction_queries_->clear();
}
query_executions_.emplace_back(std::make_unique<QueryExecution>());
auto &query_execution = query_executions_.back();
std::optional<int> qid =
in_explicit_transaction_ ? static_cast<int>(query_executions_.size() - 1) : std::optional<int>{};
// This will be done in the handle transaction query. Our handler can save username and then send it to the kill and
// show transactions.
std::optional<std::string> user = StringPointerToOptional(username);
username_ = user;
// Handle transaction control queries.
@@ -2298,10 +2696,19 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
const auto trimmed_query = utils::Trim(upper_case_query);
if (trimmed_query == "BEGIN" || trimmed_query == "COMMIT" || trimmed_query == "ROLLBACK") {
query_executions_.emplace_back(
std::make_unique<QueryExecution>(utils::MonotonicBufferResource(kExecutionMemoryBlockSize)));
auto &query_execution = query_executions_.back();
std::optional<int> qid =
in_explicit_transaction_ ? static_cast<int>(query_executions_.size() - 1) : std::optional<int>{};
query_execution->prepared_query.emplace(PrepareTransactionQuery(trimmed_query));
return {query_execution->prepared_query->header, query_execution->prepared_query->privileges, qid};
}
// Don't save BEGIN, COMMIT or ROLLBACK
transaction_queries_->push_back(query_string);
// All queries other than transaction control queries advance the command in
// an explicit transaction block.
if (in_explicit_transaction_) {
@@ -2310,27 +2717,61 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
// If we're not in an explicit transaction block and we have an open
// transaction, abort it since we're about to prepare a new query.
else if (db_accessor_) {
AbortCommand(&query_execution);
query_executions_.emplace_back(
std::make_unique<QueryExecution>(utils::MonotonicBufferResource(kExecutionMemoryBlockSize)));
AbortCommand(&query_executions_.back());
}
std::unique_ptr<QueryExecution> *query_execution_ptr = nullptr;
try {
// Set a default cost estimate of 0. Individual queries can overwrite this
// field with an improved estimate.
query_execution->summary["cost_estimate"] = 0.0;
query_executions_.emplace_back(
std::make_unique<QueryExecution>(utils::MonotonicBufferResource(kExecutionMemoryBlockSize)));
query_execution_ptr = &query_executions_.back();
utils::Timer parsing_timer;
ParsedQuery parsed_query =
ParseQuery(query_string, params, &interpreter_context_->ast_cache, interpreter_context_->config.query);
query_execution->summary["parsing_time"] = parsing_timer.Elapsed().count();
TypedValue parsing_time{parsing_timer.Elapsed().count()};
if ((utils::Downcast<CypherQuery>(parsed_query.query) || utils::Downcast<ProfileQuery>(parsed_query.query))) {
CypherQuery *cypher_query = nullptr;
if (utils::Downcast<CypherQuery>(parsed_query.query)) {
cypher_query = utils::Downcast<CypherQuery>(parsed_query.query);
} else {
auto *profile_query = utils::Downcast<ProfileQuery>(parsed_query.query);
cypher_query = profile_query->cypher_query_;
}
if (const auto &clauses = cypher_query->single_query_->clauses_;
std::any_of(clauses.begin(), clauses.end(),
[](const auto *clause) { return clause->GetTypeInfo() == LoadCsv::kType; })) {
// Using PoolResource without MonotonicMemoryResouce for LOAD CSV reduces memory usage.
// QueryExecution MemoryResource is mostly used for allocations done on Frame and storing `row`s
query_executions_[query_executions_.size() - 1] = std::make_unique<QueryExecution>(
utils::PoolResource(8, kExecutionPoolMaxBlockSize, utils::NewDeleteResource(), utils::NewDeleteResource()));
query_execution_ptr = &query_executions_.back();
}
}
auto &query_execution = query_executions_.back();
std::optional<int> qid =
in_explicit_transaction_ ? static_cast<int>(query_executions_.size() - 1) : std::optional<int>{};
query_execution->summary["parsing_time"] = std::move(parsing_time);
// Set a default cost estimate of 0. Individual queries can overwrite this
// field with an improved estimate.
query_execution->summary["cost_estimate"] = 0.0;
// Some queries require an active transaction in order to be prepared.
if (!in_explicit_transaction_ &&
(utils::Downcast<CypherQuery>(parsed_query.query) || utils::Downcast<ExplainQuery>(parsed_query.query) ||
utils::Downcast<ProfileQuery>(parsed_query.query) || utils::Downcast<DumpQuery>(parsed_query.query) ||
utils::Downcast<TriggerQuery>(parsed_query.query))) {
utils::Downcast<TriggerQuery>(parsed_query.query) || utils::Downcast<AnalyzeGraphQuery>(parsed_query.query) ||
utils::Downcast<TransactionQueueQuery>(parsed_query.query))) {
db_accessor_ =
std::make_unique<storage::Storage::Accessor>(interpreter_context_->db->Access(GetIsolationLevelOverride()));
execution_db_accessor_.emplace(db_accessor_.get());
transaction_status_.store(TransactionStatus::ACTIVE, std::memory_order_release);
if (utils::Downcast<CypherQuery>(parsed_query.query) && interpreter_context_->trigger_store.HasTriggers()) {
trigger_context_collector_.emplace(interpreter_context_->trigger_store.GetEventTypes());
@@ -2339,29 +2780,34 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
utils::Timer planning_timer;
PreparedQuery prepared_query;
utils::MemoryResource *memory_resource =
std::visit([](auto &execution_memory) -> utils::MemoryResource * { return &execution_memory; },
query_execution->execution_memory);
if (utils::Downcast<CypherQuery>(parsed_query.query)) {
prepared_query = PrepareCypherQuery(std::move(parsed_query), &query_execution->summary, interpreter_context_,
&*execution_db_accessor_, &query_execution->execution_memory,
&query_execution->notifications, username,
trigger_context_collector_ ? &*trigger_context_collector_ : nullptr);
prepared_query =
PrepareCypherQuery(std::move(parsed_query), &query_execution->summary, interpreter_context_,
&*execution_db_accessor_, memory_resource, &query_execution->notifications, username,
&transaction_status_, trigger_context_collector_ ? &*trigger_context_collector_ : nullptr);
} else if (utils::Downcast<ExplainQuery>(parsed_query.query)) {
prepared_query = PrepareExplainQuery(std::move(parsed_query), &query_execution->summary, interpreter_context_,
&*execution_db_accessor_, &query_execution->execution_memory_with_exception);
} else if (utils::Downcast<ProfileQuery>(parsed_query.query)) {
prepared_query = PrepareProfileQuery(std::move(parsed_query), in_explicit_transaction_, &query_execution->summary,
interpreter_context_, &*execution_db_accessor_,
&query_execution->execution_memory_with_exception, username);
prepared_query = PrepareProfileQuery(
std::move(parsed_query), in_explicit_transaction_, &query_execution->summary, interpreter_context_,
&*execution_db_accessor_, &query_execution->execution_memory_with_exception, username, &transaction_status_);
} else if (utils::Downcast<DumpQuery>(parsed_query.query)) {
prepared_query = PrepareDumpQuery(std::move(parsed_query), &query_execution->summary, &*execution_db_accessor_,
&query_execution->execution_memory);
memory_resource);
} else if (utils::Downcast<IndexQuery>(parsed_query.query)) {
prepared_query = PrepareIndexQuery(std::move(parsed_query), in_explicit_transaction_,
&query_execution->notifications, interpreter_context_);
} else if (utils::Downcast<AnalyzeGraphQuery>(parsed_query.query)) {
prepared_query = PrepareAnalyzeGraphQuery(std::move(parsed_query), in_explicit_transaction_,
&*execution_db_accessor_, interpreter_context_);
} else if (utils::Downcast<AuthQuery>(parsed_query.query)) {
prepared_query = PrepareAuthQuery(std::move(parsed_query), in_explicit_transaction_, &query_execution->summary,
interpreter_context_, &*execution_db_accessor_,
&query_execution->execution_memory_with_exception, username);
prepared_query = PrepareAuthQuery(
std::move(parsed_query), in_explicit_transaction_, &query_execution->summary, interpreter_context_,
&*execution_db_accessor_, &query_execution->execution_memory_with_exception, username, &transaction_status_);
} else if (utils::Downcast<InfoQuery>(parsed_query.query)) {
prepared_query = PrepareInfoQuery(std::move(parsed_query), in_explicit_transaction_, &query_execution->summary,
interpreter_context_, interpreter_context_->db,
@@ -2398,6 +2844,11 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
prepared_query = PrepareSettingQuery(std::move(parsed_query), in_explicit_transaction_, &*execution_db_accessor_);
} else if (utils::Downcast<VersionQuery>(parsed_query.query)) {
prepared_query = PrepareVersionQuery(std::move(parsed_query), in_explicit_transaction_);
} else if (utils::Downcast<StorageModeQuery>(parsed_query.query)) {
prepared_query = PrepareStorageModeQuery(std::move(parsed_query), in_explicit_transaction_, interpreter_context_);
} else if (utils::Downcast<TransactionQueueQuery>(parsed_query.query)) {
prepared_query = PrepareTransactionQueueQuery(std::move(parsed_query), username_, in_explicit_transaction_,
interpreter_context_, &*execution_db_accessor_);
} else {
LOG_FATAL("Should not get here -- unknown query type!");
}
@@ -2420,12 +2871,34 @@ Interpreter::PrepareResult Interpreter::Prepare(const std::string &query_string,
return {query_execution->prepared_query->header, query_execution->prepared_query->privileges, qid};
} catch (const utils::BasicException &) {
EventCounter::IncrementCounter(EventCounter::FailedQuery);
AbortCommand(&query_execution);
AbortCommand(query_execution_ptr);
throw;
}
}
std::vector<TypedValue> Interpreter::GetQueries() {
auto typed_queries = std::vector<TypedValue>();
transaction_queries_.WithLock([&typed_queries](const auto &transaction_queries) {
std::for_each(transaction_queries.begin(), transaction_queries.end(),
[&typed_queries](const auto &query) { typed_queries.emplace_back(query); });
});
return typed_queries;
}
void Interpreter::Abort() {
auto expected = TransactionStatus::ACTIVE;
while (!transaction_status_.compare_exchange_weak(expected, TransactionStatus::STARTED_ROLLBACK)) {
if (expected == TransactionStatus::TERMINATED || expected == TransactionStatus::IDLE) {
transaction_status_.store(TransactionStatus::STARTED_ROLLBACK);
break;
}
expected = TransactionStatus::ACTIVE;
std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
utils::OnScopeExit clean_status(
[this]() { transaction_status_.store(TransactionStatus::IDLE, std::memory_order_release); });
expect_rollback_ = false;
in_explicit_transaction_ = false;
if (!db_accessor_) return;
@@ -2437,7 +2910,7 @@ void Interpreter::Abort() {
namespace {
void RunTriggersIndividually(const utils::SkipList<Trigger> &triggers, InterpreterContext *interpreter_context,
TriggerContext trigger_context) {
TriggerContext trigger_context, std::atomic<TransactionStatus> *transaction_status) {
// Run the triggers
for (const auto &trigger : triggers.access()) {
utils::MonotonicBufferResource execution_memory{kExecutionMemoryBlockSize};
@@ -2449,7 +2922,8 @@ void RunTriggersIndividually(const utils::SkipList<Trigger> &triggers, Interpret
trigger_context.AdaptForAccessor(&db_accessor);
try {
trigger.Execute(&db_accessor, &execution_memory, interpreter_context->config.execution_timeout_sec,
&interpreter_context->is_shutting_down, trigger_context, interpreter_context->auth_checker);
&interpreter_context->is_shutting_down, transaction_status, trigger_context,
interpreter_context->auth_checker);
} catch (const utils::BasicException &exception) {
spdlog::warn("Trigger '{}' failed with exception:\n{}", trigger.Name(), exception.what());
db_accessor.Abort();
@@ -2504,6 +2978,25 @@ void Interpreter::Commit() {
// a query.
if (!db_accessor_) return;
/*
At this point we must check that the transaction is alive to start committing. The only other possible state is
verifying and in that case we must check if the transaction was terminated and if yes abort committing. Exception
should suffice.
*/
auto expected = TransactionStatus::ACTIVE;
while (!transaction_status_.compare_exchange_weak(expected, TransactionStatus::STARTED_COMMITTING)) {
if (expected == TransactionStatus::TERMINATED) {
throw memgraph::utils::BasicException(
"Aborting transaction commit because the transaction was requested to stop from other session. ");
}
expected = TransactionStatus::ACTIVE;
std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
// Clean transaction status if something went wrong
utils::OnScopeExit clean_status(
[this]() { transaction_status_.store(TransactionStatus::IDLE, std::memory_order_release); });
std::optional<TriggerContext> trigger_context = std::nullopt;
if (trigger_context_collector_) {
trigger_context.emplace(std::move(*trigger_context_collector_).TransformToTriggerContext());
@@ -2517,7 +3010,8 @@ void Interpreter::Commit() {
AdvanceCommand();
try {
trigger.Execute(&*execution_db_accessor_, &execution_memory, interpreter_context_->config.execution_timeout_sec,
&interpreter_context_->is_shutting_down, *trigger_context, interpreter_context_->auth_checker);
&interpreter_context_->is_shutting_down, &transaction_status_, *trigger_context,
interpreter_context_->auth_checker);
} catch (const utils::BasicException &e) {
throw utils::BasicException(
fmt::format("Trigger '{}' caused the transaction to fail.\nException: {}", trigger.Name(), e.what()));
@@ -2579,10 +3073,10 @@ void Interpreter::Commit() {
// This means the ordered execution of after commit triggers are not guaranteed.
if (trigger_context && interpreter_context_->trigger_store.AfterCommitTriggers().size() > 0) {
interpreter_context_->after_commit_trigger_pool.AddTask(
[trigger_context = std::move(*trigger_context), interpreter_context = this->interpreter_context_,
[this, trigger_context = std::move(*trigger_context),
user_transaction = std::shared_ptr(std::move(db_accessor_))]() mutable {
RunTriggersIndividually(interpreter_context->trigger_store.AfterCommitTriggers(), interpreter_context,
std::move(trigger_context));
RunTriggersIndividually(this->interpreter_context_->trigger_store.AfterCommitTriggers(),
this->interpreter_context_, std::move(trigger_context), &this->transaction_status_);
user_transaction->FinalizeTransaction();
SPDLOG_DEBUG("Finished executing after commit triggers"); // NOLINT(bugprone-lambda-function-name)
});

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -11,6 +11,8 @@
#pragma once
#include <unordered_set>
#include <gflags/gflags.h>
#include "query/auth_checker.hpp"
@@ -37,6 +39,7 @@
#include "utils/settings.hpp"
#include "utils/skip_list.hpp"
#include "utils/spin_lock.hpp"
#include "utils/synchronized.hpp"
#include "utils/thread_pool.hpp"
#include "utils/timer.hpp"
#include "utils/tsc.hpp"
@@ -48,6 +51,7 @@ extern const Event FailedQuery;
namespace memgraph::query {
inline constexpr size_t kExecutionMemoryBlockSize = 1UL * 1024UL * 1024UL;
inline constexpr size_t kExecutionPoolMaxBlockSize = 2048UL; // 2 ^ 11
class AuthQueryHandler {
public:
@@ -169,6 +173,24 @@ class ReplicationQueryHandler {
virtual std::vector<Replica> ShowReplicas() const = 0;
};
class AnalyzeGraphQueryHandler {
public:
AnalyzeGraphQueryHandler() = default;
virtual ~AnalyzeGraphQueryHandler() = default;
AnalyzeGraphQueryHandler(const AnalyzeGraphQueryHandler &) = default;
AnalyzeGraphQueryHandler &operator=(const AnalyzeGraphQueryHandler &) = default;
AnalyzeGraphQueryHandler(AnalyzeGraphQueryHandler &&) = default;
AnalyzeGraphQueryHandler &operator=(AnalyzeGraphQueryHandler &&) = default;
static std::vector<std::vector<TypedValue>> AnalyzeGraphCreateStatistics(const std::span<std::string> labels,
DbAccessor *execution_db_accessor);
static std::vector<std::vector<TypedValue>> AnalyzeGraphDeleteStatistics(const std::span<std::string> labels,
DbAccessor *execution_db_accessor);
};
/**
* A container for data related to the preparation of a query.
*/
@@ -179,12 +201,12 @@ struct PreparedQuery {
plan::ReadWriteTypeChecker::RWType rw_type;
};
class Interpreter;
/**
* Holds data shared between multiple `Interpreter` instances (which might be
* running concurrently).
*
* Users should initialize the context but should not modify it after it has
* been passed to an `Interpreter` instance.
*/
struct InterpreterContext {
explicit InterpreterContext(storage::Storage *db, InterpreterConfig config,
@@ -214,6 +236,7 @@ struct InterpreterContext {
const InterpreterConfig config;
query::stream::Streams streams;
utils::Synchronized<std::unordered_set<Interpreter *>, utils::SpinLock> interpreters;
};
/// Function that is used to tell all active interpreters that they should stop
@@ -235,6 +258,10 @@ class Interpreter final {
std::optional<int> qid;
};
std::optional<std::string> username_;
bool in_explicit_transaction_{false};
bool expect_rollback_{false};
/**
* Prepare a query for execution.
*
@@ -290,6 +317,11 @@ class Interpreter final {
void BeginTransaction();
/*
Returns transaction id or empty if the db_accessor is not initialized.
*/
std::optional<uint64_t> GetTransactionId() const;
void CommitTransaction();
void RollbackTransaction();
@@ -297,21 +329,41 @@ class Interpreter final {
void SetNextTransactionIsolationLevel(storage::IsolationLevel isolation_level);
void SetSessionIsolationLevel(storage::IsolationLevel isolation_level);
std::vector<TypedValue> GetQueries();
/**
* Abort the current multicommand transaction.
*/
void Abort();
std::atomic<TransactionStatus> transaction_status_{TransactionStatus::IDLE};
private:
struct QueryExecution {
std::optional<PreparedQuery> prepared_query;
utils::MonotonicBufferResource execution_memory{kExecutionMemoryBlockSize};
utils::ResourceWithOutOfMemoryException execution_memory_with_exception{&execution_memory};
std::variant<utils::MonotonicBufferResource, utils::PoolResource> execution_memory;
utils::ResourceWithOutOfMemoryException execution_memory_with_exception;
std::map<std::string, TypedValue> summary;
std::vector<Notification> notifications;
explicit QueryExecution() = default;
explicit QueryExecution(utils::MonotonicBufferResource monotonic_memory)
: execution_memory(std::move(monotonic_memory)) {
std::visit(
[&](auto &memory_resource) {
execution_memory_with_exception = utils::ResourceWithOutOfMemoryException(&memory_resource);
},
execution_memory);
};
explicit QueryExecution(utils::PoolResource pool_resource) : execution_memory(std::move(pool_resource)) {
std::visit(
[&](auto &memory_resource) {
execution_memory_with_exception = utils::ResourceWithOutOfMemoryException(&memory_resource);
},
execution_memory);
};
QueryExecution(const QueryExecution &) = delete;
QueryExecution(QueryExecution &&) = default;
QueryExecution &operator=(const QueryExecution &) = delete;
@@ -322,7 +374,7 @@ class Interpreter final {
// destroy the prepared query which is using that instance
// of execution memory.
prepared_query.reset();
execution_memory.Release();
std::visit([](auto &memory_resource) { memory_resource.Release(); }, execution_memory);
}
};
@@ -338,6 +390,8 @@ class Interpreter final {
// and deletion of a single query execution, i.e. when a query finishes,
// we reset the corresponding unique_ptr.
std::vector<std::unique_ptr<QueryExecution>> query_executions_;
// all queries that are run as part of the current transaction
utils::Synchronized<std::vector<std::string>, utils::SpinLock> transaction_queries_;
InterpreterContext *interpreter_context_;
@@ -347,8 +401,6 @@ class Interpreter final {
std::unique_ptr<storage::Storage::Accessor> db_accessor_;
std::optional<DbAccessor> execution_db_accessor_;
std::optional<TriggerContextCollector> trigger_context_collector_;
bool in_explicit_transaction_{false};
bool expect_rollback_{false};
std::optional<storage::IsolationLevel> interpreter_isolation_level;
std::optional<storage::IsolationLevel> next_transaction_isolation_level;
@@ -365,12 +417,32 @@ class Interpreter final {
}
};
class TransactionQueueQueryHandler {
public:
TransactionQueueQueryHandler() = default;
virtual ~TransactionQueueQueryHandler() = default;
TransactionQueueQueryHandler(const TransactionQueueQueryHandler &) = default;
TransactionQueueQueryHandler &operator=(const TransactionQueueQueryHandler &) = default;
TransactionQueueQueryHandler(TransactionQueueQueryHandler &&) = default;
TransactionQueueQueryHandler &operator=(TransactionQueueQueryHandler &&) = default;
static std::vector<std::vector<TypedValue>> ShowTransactions(const std::unordered_set<Interpreter *> &interpreters,
const std::optional<std::string> &username,
bool hasTransactionManagementPrivilege);
static std::vector<std::vector<TypedValue>> KillTransactions(
InterpreterContext *interpreter_context, const std::vector<std::string> &maybe_kill_transaction_ids,
const std::optional<std::string> &username, bool hasTransactionManagementPrivilege);
};
template <typename TStream>
std::map<std::string, TypedValue> Interpreter::Pull(TStream *result_stream, std::optional<int> n,
std::optional<int> qid) {
MG_ASSERT(in_explicit_transaction_ || !qid, "qid can be only used in explicit transaction!");
const int qid_value = qid ? *qid : static_cast<int>(query_executions_.size() - 1);
const int qid_value = qid ? *qid : static_cast<int>(query_executions_.size() - 1);
if (qid_value < 0 || qid_value >= query_executions_.size()) {
throw InvalidArgumentsException("qid", "Query with specified ID does not exist!");
}
@@ -390,7 +462,9 @@ std::map<std::string, TypedValue> Interpreter::Pull(TStream *result_stream, std:
try {
// Wrap the (statically polymorphic) stream type into a common type which
// the handler knows.
AnyStream stream{result_stream, &query_execution->execution_memory};
AnyStream stream{result_stream,
std::visit([](auto &execution_memory) -> utils::MemoryResource * { return &execution_memory; },
query_execution->execution_memory)};
const auto maybe_res = query_execution->prepared_query->query_handler(&stream, n);
// Stream is using execution memory of the query_execution which
// can be deleted after its execution so the stream should be cleared
@@ -430,6 +504,7 @@ std::map<std::string, TypedValue> Interpreter::Pull(TStream *result_stream, std:
// methods as we will delete summary contained in them which we need
// after our query finished executing.
query_executions_.clear();
transaction_queries_->clear();
} else {
// We can only clear this execution as some of the queries
// in the transaction can be in unfinished state

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -62,6 +62,8 @@ class CostEstimator : public HierarchicalLogicalOperatorVisitor {
static constexpr double kEdgeUniquenessFilter{1.5};
static constexpr double kUnwind{1.3};
static constexpr double kForeach{1.0};
static constexpr double kUnion{1.0};
static constexpr double kSubquery{1.0};
};
struct CardParam {
@@ -212,6 +214,31 @@ class CostEstimator : public HierarchicalLogicalOperatorVisitor {
return true;
}
bool PreVisit(Union &op) override {
double left_cost = EstimateCostOnBranch(&op.left_op_);
double right_cost = EstimateCostOnBranch(&op.right_op_);
// the number of hits in the previous operator should be the joined number of results of both parts of the union
cardinality_ *= (left_cost + right_cost);
IncrementCost(CostParam::kUnion);
return false;
}
bool PreVisit(Apply &op) override {
double input_cost = EstimateCostOnBranch(&op.input_);
double subquery_cost = EstimateCostOnBranch(&op.subquery_);
// if the query is a unit subquery, we don't want the cost to be zero but 1xN
input_cost = input_cost == 0 ? 1 : input_cost;
subquery_cost = subquery_cost == 0 ? 1 : subquery_cost;
cardinality_ *= input_cost * subquery_cost;
IncrementCost(CostParam::kSubquery);
return false;
}
bool Visit(Once &) override { return true; }
auto cost() const { return cost_; }
@@ -232,6 +259,12 @@ class CostEstimator : public HierarchicalLogicalOperatorVisitor {
void IncrementCost(double param) { cost_ += param * cardinality_; }
double EstimateCostOnBranch(std::shared_ptr<LogicalOperator> *branch) {
CostEstimator<TDbAccessor> cost_estimator(db_accessor_, parameters);
(*branch)->Accept(cost_estimator);
return cost_estimator.cost();
}
// converts an optional ScanAll range bound into a property value
// if the bound is present and is a constant expression convertible to
// a property value. otherwise returns nullopt

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -25,6 +25,7 @@
#include <cppitertools/chain.hpp>
#include <cppitertools/imap.hpp>
#include "query/common.hpp"
#include "spdlog/spdlog.h"
#include "license/license.hpp"
@@ -52,6 +53,7 @@
#include "utils/likely.hpp"
#include "utils/logging.hpp"
#include "utils/memory.hpp"
#include "utils/pmr/deque.hpp"
#include "utils/pmr/list.hpp"
#include "utils/pmr/unordered_map.hpp"
#include "utils/pmr/unordered_set.hpp"
@@ -115,6 +117,8 @@ extern const Event CartesianOperator;
extern const Event CallProcedureOperator;
extern const Event ForeachOperator;
extern const Event EmptyResultOperator;
extern const Event EvaluatePatternFilterOperator;
extern const Event ApplyOperator;
} // namespace EventCounter
namespace memgraph::query::plan {
@@ -208,17 +212,18 @@ VertexAccessor &CreateLocalVertex(const NodeCreationInfo &node_info, Frame *fram
storage::View::NEW);
// TODO: PropsSetChecked allocates a PropertyValue, make it use context.memory
// when we update PropertyValue with custom allocator.
std::map<storage::PropertyId, storage::PropertyValue> properties;
if (const auto *node_info_properties = std::get_if<PropertiesMapList>(&node_info.properties)) {
for (const auto &[key, value_expression] : *node_info_properties) {
PropsSetChecked(&new_node, key, value_expression->Accept(evaluator));
properties.emplace(key, value_expression->Accept(evaluator));
}
} else {
auto property_map = evaluator.Visit(*std::get<ParameterLookup *>(node_info.properties));
for (const auto &[key, value] : property_map.ValueMap()) {
auto property_id = dba.NameToProperty(key);
PropsSetChecked(&new_node, property_id, value);
properties.emplace(dba.NameToProperty(key), value);
}
}
MultiPropsInitChecked(&new_node, properties);
(*frame)[node_info.symbol] = new_node;
return (*frame)[node_info.symbol].ValueVertex();
@@ -299,17 +304,18 @@ EdgeAccessor CreateEdge(const EdgeCreationInfo &edge_info, DbAccessor *dba, Vert
auto maybe_edge = dba->InsertEdge(from, to, edge_info.edge_type);
if (maybe_edge.HasValue()) {
auto &edge = *maybe_edge;
if (const auto *properties = std::get_if<PropertiesMapList>(&edge_info.properties)) {
for (const auto &[key, value_expression] : *properties) {
PropsSetChecked(&edge, key, value_expression->Accept(*evaluator));
std::map<storage::PropertyId, storage::PropertyValue> properties;
if (const auto *edge_info_properties = std::get_if<PropertiesMapList>(&edge_info.properties)) {
for (const auto &[key, value_expression] : *edge_info_properties) {
properties.emplace(key, value_expression->Accept(*evaluator));
}
} else {
auto property_map = evaluator->Visit(*std::get<ParameterLookup *>(edge_info.properties));
for (const auto &[key, value] : property_map.ValueMap()) {
auto property_id = dba->NameToProperty(key);
PropsSetChecked(&edge, property_id, value);
properties.emplace(dba->NameToProperty(key), value);
}
}
if (!properties.empty()) MultiPropsInitChecked(&edge, properties);
(*frame)[edge_info.symbol] = edge;
} else {
@@ -410,10 +416,11 @@ VertexAccessor &CreateExpand::CreateExpandCursor::OtherVertex(Frame &frame, Exec
template <class TVerticesFun>
class ScanAllCursor : public Cursor {
public:
explicit ScanAllCursor(Symbol output_symbol, UniqueCursorPtr input_cursor, TVerticesFun get_vertices,
const char *op_name)
explicit ScanAllCursor(Symbol output_symbol, UniqueCursorPtr input_cursor, storage::View view,
TVerticesFun get_vertices, const char *op_name)
: output_symbol_(output_symbol),
input_cursor_(std::move(input_cursor)),
view_(view),
get_vertices_(std::move(get_vertices)),
op_name_(op_name) {}
@@ -448,7 +455,7 @@ class ScanAllCursor : public Cursor {
#ifdef MG_ENTERPRISE
bool FindNextVertex(const ExecutionContext &context) {
while (vertices_it_.value() != vertices_.value().end()) {
if (context.auth_checker->Has(*vertices_it_.value(), memgraph::storage::View::OLD,
if (context.auth_checker->Has(*vertices_it_.value(), view_,
memgraph::query::AuthQuery::FineGrainedPrivilege::READ)) {
return true;
}
@@ -469,6 +476,7 @@ class ScanAllCursor : public Cursor {
private:
const Symbol output_symbol_;
const UniqueCursorPtr input_cursor_;
storage::View view_;
TVerticesFun get_vertices_;
std::optional<typename std::result_of<TVerticesFun(Frame &, ExecutionContext &)>::type::value_type> vertices_;
std::optional<decltype(vertices_.value().begin())> vertices_it_;
@@ -487,7 +495,7 @@ UniqueCursorPtr ScanAll::MakeCursor(utils::MemoryResource *mem) const {
auto *db = context.db_accessor;
return std::make_optional(db->Vertices(view_));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem),
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAll");
}
@@ -510,7 +518,7 @@ UniqueCursorPtr ScanAllByLabel::MakeCursor(utils::MemoryResource *mem) const {
auto *db = context.db_accessor;
return std::make_optional(db->Vertices(view_, label_));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem),
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllByLabel");
}
@@ -575,7 +583,7 @@ UniqueCursorPtr ScanAllByLabelPropertyRange::MakeCursor(utils::MemoryResource *m
if (maybe_upper && maybe_upper->value().IsNull()) return std::nullopt;
return std::make_optional(db->Vertices(view_, label_, property_, maybe_lower, maybe_upper));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem),
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllByLabelPropertyRange");
}
@@ -607,7 +615,7 @@ UniqueCursorPtr ScanAllByLabelPropertyValue::MakeCursor(utils::MemoryResource *m
}
return std::make_optional(db->Vertices(view_, label_, property_, storage::PropertyValue(value)));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem),
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllByLabelPropertyValue");
}
@@ -625,7 +633,7 @@ UniqueCursorPtr ScanAllByLabelProperty::MakeCursor(utils::MemoryResource *mem) c
auto *db = context.db_accessor;
return std::make_optional(db->Vertices(view_, label_, property_));
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem),
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllByLabelProperty");
}
@@ -651,7 +659,7 @@ UniqueCursorPtr ScanAllById::MakeCursor(utils::MemoryResource *mem) const {
if (!maybe_vertex) return std::nullopt;
return std::vector<VertexAccessor>{*maybe_vertex};
};
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem),
return MakeUniqueCursorPtr<ScanAllCursor<decltype(vertices)>>(mem, output_symbol_, input_->MakeCursor(mem), view_,
std::move(vertices), "ScanAllById");
}
@@ -1828,7 +1836,7 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
: self_(self),
input_cursor_(self_.input_->MakeCursor(mem)),
visited_cost_(mem),
expanded_(mem),
total_cost_(mem),
next_edges_(mem),
traversal_stack_(mem),
pq_(mem) {}
@@ -1838,6 +1846,9 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
ExpressionEvaluator evaluator(&frame, context.symbol_table, context.evaluation_context, context.db_accessor,
storage::View::OLD);
auto create_state = [this](const VertexAccessor &vertex, int64_t depth) {
return std::make_pair(vertex, upper_bound_set_ ? depth : 0);
};
// For the given (edge, direction, weight, depth) tuple checks if they
// satisfy the "where" condition. if so, places them in the priority
@@ -1929,73 +1940,118 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
}
};
// Check if upper bound exists
upper_bound_ = self_.upper_bound_
? EvaluateInt(&evaluator, self_.upper_bound_, "Max depth in all shortest paths expansion")
: std::numeric_limits<int64_t>::max();
std::optional<VertexAccessor> start_vertex;
auto *memory = context.evaluation_context.memory;
auto create_path = [this, &frame, &memory]() {
auto &current_level = traversal_stack_.back();
auto &edges_on_frame = frame[self_.common_.edge_symbol].ValueList();
// Clean out the current stack
if (current_level.empty()) {
if (!edges_on_frame.empty()) {
if (!self_.is_reverse_)
edges_on_frame.erase(edges_on_frame.end());
else
edges_on_frame.erase(edges_on_frame.begin());
}
traversal_stack_.pop_back();
return false;
}
auto [current_edge, current_edge_direction, current_weight] = current_level.back();
current_level.pop_back();
// Edges order depends on direction of expansion
if (!self_.is_reverse_)
edges_on_frame.emplace_back(current_edge);
else
edges_on_frame.emplace(edges_on_frame.begin(), current_edge);
auto next_vertex = current_edge_direction == EdgeAtom::Direction::IN ? current_edge.From() : current_edge.To();
frame[self_.total_weight_.value()] = current_weight;
if (next_edges_.find({next_vertex, traversal_stack_.size()}) != next_edges_.end()) {
auto next_vertex_edges = next_edges_[{next_vertex, traversal_stack_.size()}];
traversal_stack_.emplace_back(std::move(next_vertex_edges));
} else {
// Signal the end of iteration
utils::pmr::list<DirectedEdge> empty(memory);
traversal_stack_.emplace_back(std::move(empty));
}
if ((current_weight > visited_cost_.at(next_vertex)).ValueBool()) return false;
// Place destination node on the frame, handle existence flag
if (self_.common_.existing_node) {
const auto &node = frame[self_.common_.node_symbol];
ExpectType(self_.common_.node_symbol, node, TypedValue::Type::Vertex);
if (node.ValueVertex() != next_vertex) return false;
} else {
frame[self_.common_.node_symbol] = next_vertex;
}
return true;
};
auto create_DFS_traversal_tree = [this, &context, &memory, &create_state, &expand_from_vertex]() {
while (!pq_.empty()) {
if (MustAbort(context)) throw HintedAbortError();
const auto [current_weight, current_depth, current_vertex, directed_edge] = pq_.top();
pq_.pop();
const auto &[current_edge, direction, weight] = directed_edge;
auto current_state = create_state(current_vertex, current_depth);
auto position = total_cost_.find(current_state);
if (position != total_cost_.end()) {
if ((position->second < current_weight).ValueBool()) continue;
} else {
total_cost_.emplace(current_state, current_weight);
if (current_depth < upper_bound_) {
expand_from_vertex(current_vertex, current_weight, current_depth);
}
}
// Searching for a previous vertex in the expansion
auto prev_vertex = direction == EdgeAtom::Direction::IN ? current_edge.To() : current_edge.From();
// Update the parent
if (next_edges_.find({prev_vertex, current_depth - 1}) == next_edges_.end()) {
utils::pmr::list<DirectedEdge> empty(memory);
next_edges_[{prev_vertex, current_depth - 1}] = std::move(empty);
}
next_edges_.at({prev_vertex, current_depth - 1}).emplace_back(directed_edge);
}
};
// upper_bound_set is used when storing visited edges, because with an upper bound we also consider suboptimal paths
// if they are shorter in depth
if (self_.upper_bound_) {
upper_bound_ = EvaluateInt(&evaluator, self_.upper_bound_, "Max depth in all shortest path expansion");
upper_bound_set_ = true;
} else {
upper_bound_ = std::numeric_limits<int64_t>::max();
upper_bound_set_ = false;
}
// Check if upper bound is valid
if (upper_bound_ < 1) {
throw QueryRuntimeException("Maximum depth in all shortest paths expansion must be at least 1.");
}
std::optional<VertexAccessor> start_vertex;
auto *memory = context.evaluation_context.memory;
// On first Pull run, traversal stack and priority queue are empty, so we start a pulling stream
// and create a DFS traversal tree (main part of algorithm). Then we return the first path
// created from the DFS traversal tree (basically a DFS algorithm).
// On each subsequent Pull run, paths are created from the traversal stack and returned.
while (true) {
// Check if there is an external error.
if (MustAbort(context)) throw HintedAbortError();
// If traversal stack if filled, the DFS traversal tree is created. Traverse the tree iteratively by
// preserving the traversal state on stack.
// The algorithm is run all at once by create_DFS_traversal_tree, after which we
// traverse the tree iteratively by preserving the traversal state on stack.
while (!traversal_stack_.empty()) {
auto &current_level = traversal_stack_.back();
auto &edges_on_frame = frame[self_.common_.edge_symbol].ValueList();
// Clean out the current stack
if (current_level.empty()) {
if (!edges_on_frame.empty()) {
if (!self_.is_reverse_)
edges_on_frame.erase(edges_on_frame.end());
else
edges_on_frame.erase(edges_on_frame.begin());
}
traversal_stack_.pop_back();
continue;
}
auto [current_edge, current_edge_direction, current_weight] = current_level.back();
current_level.pop_back();
// Edges order depends on direction of expansion
if (!self_.is_reverse_)
edges_on_frame.emplace_back(current_edge);
else
edges_on_frame.emplace(edges_on_frame.begin(), current_edge);
auto next_vertex = current_edge_direction == EdgeAtom::Direction::IN ? current_edge.From() : current_edge.To();
frame[self_.total_weight_.value()] = current_weight;
if (next_edges_.find({next_vertex, traversal_stack_.size()}) != next_edges_.end()) {
auto next_vertex_edges = next_edges_[{next_vertex, traversal_stack_.size()}];
traversal_stack_.emplace_back(std::move(next_vertex_edges));
} else {
// Signal the end of iteration
utils::pmr::list<DirectedEdge> empty(memory);
traversal_stack_.emplace_back(std::move(empty));
}
if ((current_weight > visited_cost_.at(next_vertex)).ValueBool()) continue;
// Place destination node on the frame, handle existence flag
if (self_.common_.existing_node) {
const auto &node = frame[self_.common_.node_symbol];
ExpectType(self_.common_.node_symbol, node, TypedValue::Type::Vertex);
if (node.ValueVertex() != next_vertex) continue;
} else {
frame[self_.common_.node_symbol] = next_vertex;
}
return true;
if (create_path()) return true;
}
// If priority queue is empty start new pulling stream.
@@ -2016,48 +2072,19 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
// Clear existing data structures.
visited_cost_.clear();
expanded_.clear();
next_edges_.clear();
traversal_stack_.clear();
total_cost_.clear();
pq_.push({TypedValue(), 0, *start_vertex, std::nullopt});
expand_from_vertex(*start_vertex, TypedValue(), 0);
visited_cost_.emplace(*start_vertex, 0);
frame[self_.common_.edge_symbol] = TypedValue::TVector(memory);
}
// Create a DFS traversal tree from the start node
while (!pq_.empty()) {
if (MustAbort(context)) throw HintedAbortError();
auto [current_weight, current_depth, current_vertex, maybe_directed_edge] = pq_.top();
pq_.pop();
// Expand only if what we've just expanded is less than max depth.
if (current_depth < upper_bound_) {
if (maybe_directed_edge) {
auto &[current_edge, direction, weight] = *maybe_directed_edge;
if (expanded_.find(current_edge) != expanded_.end()) continue;
expanded_.emplace(current_edge);
}
expand_from_vertex(current_vertex, current_weight, current_depth);
}
// if current vertex is not starting vertex, maybe_directed_edge will not be nullopt
if (maybe_directed_edge) {
auto &[current_edge, direction, weight] = *maybe_directed_edge;
// Searching for a previous vertex in the expansion
auto prev_vertex = direction == EdgeAtom::Direction::IN ? current_edge.To() : current_edge.From();
// Update the parent
if (next_edges_.find({prev_vertex, current_depth - 1}) == next_edges_.end()) {
utils::pmr::list<DirectedEdge> empty(memory);
next_edges_[{prev_vertex, current_depth - 1}] = std::move(empty);
}
next_edges_.at({prev_vertex, current_depth - 1}).emplace_back(*maybe_directed_edge);
}
}
create_DFS_traversal_tree();
// DFS traversal tree is create,
if (start_vertex && next_edges_.find({*start_vertex, 0}) != next_edges_.end()) {
auto start_vertex_edges = next_edges_[{*start_vertex, 0}];
traversal_stack_.emplace_back(std::move(start_vertex_edges));
@@ -2070,9 +2097,9 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
void Reset() override {
input_cursor_->Reset();
visited_cost_.clear();
expanded_.clear();
next_edges_.clear();
traversal_stack_.clear();
total_cost_.clear();
ClearQueue();
}
@@ -2082,6 +2109,7 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
// Upper bound on the path length.
int64_t upper_bound_{-1};
bool upper_bound_set_{false};
struct AspStateHash {
size_t operator()(const std::pair<VertexAccessor, int64_t> &key) const {
@@ -2093,8 +2121,8 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
using NextEdgesState = std::pair<VertexAccessor, int64_t>;
// Maps vertices to minimum weights they got in expansion.
utils::pmr::unordered_map<VertexAccessor, TypedValue> visited_cost_;
// Marking the expanded edges to prevent multiple visits.
utils::pmr::unordered_set<EdgeAccessor> expanded_;
// Maps vertices to weights they got in expansion.
utils::pmr::unordered_map<NextEdgesState, TypedValue, AspStateHash> total_cost_;
// Maps the vertex with the potential expansion edge.
utils::pmr::unordered_map<NextEdgesState, utils::pmr::list<DirectedEdge>, AspStateHash> next_edges_;
// Stack indicating the traversal level.
@@ -2112,8 +2140,8 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
// Priority queue comparator. Keep lowest weight on top of the queue.
class PriorityQueueComparator {
public:
bool operator()(const std::tuple<TypedValue, int64_t, VertexAccessor, std::optional<DirectedEdge>> &lhs,
const std::tuple<TypedValue, int64_t, VertexAccessor, std::optional<DirectedEdge>> &rhs) {
bool operator()(const std::tuple<TypedValue, int64_t, VertexAccessor, DirectedEdge> &lhs,
const std::tuple<TypedValue, int64_t, VertexAccessor, DirectedEdge> &rhs) {
const auto &lhs_weight = std::get<0>(lhs);
const auto &rhs_weight = std::get<0>(rhs);
// Null defines minimum value for all types
@@ -2132,8 +2160,8 @@ class ExpandAllShortestPathsCursor : public query::plan::Cursor {
// Priority queue - core element of the algorithm.
// Stores: {weight, depth, next vertex, edge and direction}
std::priority_queue<std::tuple<TypedValue, int64_t, VertexAccessor, std::optional<DirectedEdge>>,
utils::pmr::vector<std::tuple<TypedValue, int64_t, VertexAccessor, std::optional<DirectedEdge>>>,
std::priority_queue<std::tuple<TypedValue, int64_t, VertexAccessor, DirectedEdge>,
utils::pmr::vector<std::tuple<TypedValue, int64_t, VertexAccessor, DirectedEdge>>,
PriorityQueueComparator>
pq_;
@@ -2257,10 +2285,19 @@ std::vector<Symbol> ConstructNamedPath::ModifiedSymbols(const SymbolTable &table
return symbols;
}
Filter::Filter(const std::shared_ptr<LogicalOperator> &input, Expression *expression)
: input_(input ? input : std::make_shared<Once>()), expression_(expression) {}
Filter::Filter(const std::shared_ptr<LogicalOperator> &input,
const std::vector<std::shared_ptr<LogicalOperator>> &pattern_filters, Expression *expression)
: input_(input ? input : std::make_shared<Once>()), pattern_filters_(pattern_filters), expression_(expression) {}
ACCEPT_WITH_INPUT(Filter)
bool Filter::Accept(HierarchicalLogicalOperatorVisitor &visitor) {
if (visitor.PreVisit(*this)) {
input_->Accept(visitor);
for (const auto &pattern_filter : pattern_filters_) {
pattern_filter->Accept(visitor);
}
}
return visitor.PostVisit(*this);
}
UniqueCursorPtr Filter::MakeCursor(utils::MemoryResource *mem) const {
EventCounter::IncrementCounter(EventCounter::FilterOperator);
@@ -2270,8 +2307,24 @@ UniqueCursorPtr Filter::MakeCursor(utils::MemoryResource *mem) const {
std::vector<Symbol> Filter::ModifiedSymbols(const SymbolTable &table) const { return input_->ModifiedSymbols(table); }
static std::vector<UniqueCursorPtr> MakeCursorVector(const std::vector<std::shared_ptr<LogicalOperator>> &ops,
utils::MemoryResource *mem) {
std::vector<UniqueCursorPtr> cursors;
cursors.reserve(ops.size());
if (!ops.empty()) {
for (const auto &op : ops) {
cursors.push_back(op->MakeCursor(mem));
}
}
return cursors;
}
Filter::FilterCursor::FilterCursor(const Filter &self, utils::MemoryResource *mem)
: self_(self), input_cursor_(self_.input_->MakeCursor(mem)) {}
: self_(self),
input_cursor_(self_.input_->MakeCursor(mem)),
pattern_filter_cursors_(MakeCursorVector(self_.pattern_filters_, mem)) {}
bool Filter::FilterCursor::Pull(Frame &frame, ExecutionContext &context) {
SCOPED_PROFILE_OP("Filter");
@@ -2281,6 +2334,10 @@ bool Filter::FilterCursor::Pull(Frame &frame, ExecutionContext &context) {
ExpressionEvaluator evaluator(&frame, context.symbol_table, context.evaluation_context, context.db_accessor,
storage::View::OLD);
while (input_cursor_->Pull(frame, context)) {
for (const auto &pattern_filter_cursor : pattern_filter_cursors_) {
pattern_filter_cursor->Pull(frame, context);
}
if (EvaluateFilter(evaluator, self_.expression_)) return true;
}
return false;
@@ -2290,6 +2347,39 @@ void Filter::FilterCursor::Shutdown() { input_cursor_->Shutdown(); }
void Filter::FilterCursor::Reset() { input_cursor_->Reset(); }
EvaluatePatternFilter::EvaluatePatternFilter(const std::shared_ptr<LogicalOperator> &input, Symbol output_symbol)
: input_(input), output_symbol_(output_symbol) {}
ACCEPT_WITH_INPUT(EvaluatePatternFilter);
UniqueCursorPtr EvaluatePatternFilter::MakeCursor(utils::MemoryResource *mem) const {
EventCounter::IncrementCounter(EventCounter::EvaluatePatternFilterOperator);
return MakeUniqueCursorPtr<EvaluatePatternFilterCursor>(mem, *this, mem);
}
EvaluatePatternFilter::EvaluatePatternFilterCursor::EvaluatePatternFilterCursor(const EvaluatePatternFilter &self,
utils::MemoryResource *mem)
: self_(self), input_cursor_(self_.input_->MakeCursor(mem)) {}
std::vector<Symbol> EvaluatePatternFilter::ModifiedSymbols(const SymbolTable &table) const {
return input_->ModifiedSymbols(table);
}
bool EvaluatePatternFilter::EvaluatePatternFilterCursor::Pull(Frame &frame, ExecutionContext &context) {
SCOPED_PROFILE_OP("EvaluatePatternFilter");
input_cursor_->Reset();
frame[self_.output_symbol_] = TypedValue(input_cursor_->Pull(frame, context), context.evaluation_context.memory);
return true;
}
void EvaluatePatternFilter::EvaluatePatternFilterCursor::Shutdown() { input_cursor_->Shutdown(); }
void EvaluatePatternFilter::EvaluatePatternFilterCursor::Reset() { input_cursor_->Reset(); }
Produce::Produce(const std::shared_ptr<LogicalOperator> &input, const std::vector<NamedExpression *> &named_expressions)
: input_(input ? input : std::make_shared<Once>()), named_expressions_(named_expressions) {}
@@ -3159,7 +3249,7 @@ class AccumulateCursor : public Cursor {
private:
const Accumulate &self_;
const UniqueCursorPtr input_cursor_;
utils::pmr::vector<utils::pmr::vector<TypedValue>> cache_;
utils::pmr::deque<utils::pmr::vector<TypedValue>> cache_;
decltype(cache_.begin()) cache_it_ = cache_.begin();
bool pulled_all_input_{false};
};
@@ -3982,8 +4072,14 @@ class DistinctCursor : public Cursor {
utils::pmr::vector<TypedValue> row(seen_rows_.get_allocator().GetMemoryResource());
row.reserve(self_.value_symbols_.size());
for (const auto &symbol : self_.value_symbols_) row.emplace_back(frame[symbol]);
if (seen_rows_.insert(std::move(row)).second) return true;
for (const auto &symbol : self_.value_symbols_) {
row.emplace_back(frame.at(symbol));
}
if (seen_rows_.insert(std::move(row)).second) {
return true;
}
}
}
@@ -4528,24 +4624,24 @@ auto ToOptionalString(ExpressionEvaluator *evaluator, Expression *expression) ->
return std::nullopt;
};
TypedValue CsvRowToTypedList(csv::Reader::Row row) {
TypedValue CsvRowToTypedList(csv::Reader::Row &row) {
auto *mem = row.get_allocator().GetMemoryResource();
auto typed_columns = utils::pmr::vector<TypedValue>(mem);
typed_columns.reserve(row.size());
for (auto &column : row) {
typed_columns.emplace_back(std::move(column));
}
return TypedValue(typed_columns, mem);
return {std::move(typed_columns), mem};
}
TypedValue CsvRowToTypedMap(csv::Reader::Row row, csv::Reader::Header header) {
TypedValue CsvRowToTypedMap(csv::Reader::Row &row, csv::Reader::Header header) {
// a valid row has the same number of elements as the header
auto *mem = row.get_allocator().GetMemoryResource();
utils::pmr::map<utils::pmr::string, TypedValue> m(mem);
for (auto i = 0; i < row.size(); ++i) {
m.emplace(std::move(header[i]), std::move(row[i]));
}
return TypedValue(m, mem);
return {std::move(m), mem};
}
} // namespace
@@ -4584,18 +4680,17 @@ class LoadCsvCursor : public Cursor {
// have to read at most cardinality(n) rows (but we can read less and stop
// pulling MATCH).
if (!input_is_once_ && !input_pulled) return false;
if (auto row = reader_->GetNextRow(context.evaluation_context.memory)) {
if (!reader_->HasHeader()) {
frame[self_->row_var_] = CsvRowToTypedList(std::move(*row));
} else {
frame[self_->row_var_] = CsvRowToTypedMap(
std::move(*row), csv::Reader::Header(reader_->GetHeader(), context.evaluation_context.memory));
}
return true;
auto row = reader_->GetNextRow(context.evaluation_context.memory);
if (!row) {
return false;
}
return false;
if (!reader_->HasHeader()) {
frame[self_->row_var_] = CsvRowToTypedList(*row);
} else {
frame[self_->row_var_] =
CsvRowToTypedMap(*row, csv::Reader::Header(reader_->GetHeader(), context.evaluation_context.memory));
}
return true;
}
void Reset() override { input_cursor_->Reset(); }
@@ -4709,4 +4804,72 @@ bool Foreach::Accept(HierarchicalLogicalOperatorVisitor &visitor) {
return visitor.PostVisit(*this);
}
Apply::Apply(const std::shared_ptr<LogicalOperator> input, const std::shared_ptr<LogicalOperator> subquery,
bool subquery_has_return)
: input_(input ? input : std::make_shared<Once>()),
subquery_(subquery),
subquery_has_return_(subquery_has_return) {}
bool Apply::Accept(HierarchicalLogicalOperatorVisitor &visitor) {
if (visitor.PreVisit(*this)) {
input_->Accept(visitor) && subquery_->Accept(visitor);
}
return visitor.PostVisit(*this);
}
UniqueCursorPtr Apply::MakeCursor(utils::MemoryResource *mem) const {
EventCounter::IncrementCounter(EventCounter::ApplyOperator);
return MakeUniqueCursorPtr<ApplyCursor>(mem, *this, mem);
}
Apply::ApplyCursor::ApplyCursor(const Apply &self, utils::MemoryResource *mem)
: self_(self),
input_(self.input_->MakeCursor(mem)),
subquery_(self.subquery_->MakeCursor(mem)),
subquery_has_return_(self.subquery_has_return_) {}
std::vector<Symbol> Apply::ModifiedSymbols(const SymbolTable &table) const {
// Since Apply is the Cartesian product, modified symbols are combined from
// both execution branches.
auto symbols = input_->ModifiedSymbols(table);
auto subquery_symbols = subquery_->ModifiedSymbols(table);
symbols.insert(symbols.end(), subquery_symbols.begin(), subquery_symbols.end());
return symbols;
}
bool Apply::ApplyCursor::Pull(Frame &frame, ExecutionContext &context) {
SCOPED_PROFILE_OP("Apply");
while (true) {
if (pull_input_ && !input_->Pull(frame, context)) {
return false;
};
if (subquery_->Pull(frame, context)) {
// if successful, next Pull from this should not pull_input_
pull_input_ = false;
return true;
}
// failed to pull from subquery cursor
// skip that row
pull_input_ = true;
subquery_->Reset();
// don't skip row if no rows are returned from subquery, return input_ rows
if (!subquery_has_return_) return true;
}
}
void Apply::ApplyCursor::Shutdown() {
input_->Shutdown();
subquery_->Shutdown();
}
void Apply::ApplyCursor::Reset() {
input_->Reset();
subquery_->Reset();
pull_input_ = true;
}
} // namespace memgraph::query::plan

2344
src/query/plan/operator.hpp Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -133,6 +133,8 @@ class CallProcedure;
class LoadCsv;
class Foreach;
class EmptyResult;
class EvaluatePatternFilter;
class Apply;
using LogicalOperatorCompositeVisitor = utils::CompositeVisitor<
Once, CreateNode, CreateExpand, ScanAll, ScanAllByLabel,
@@ -141,7 +143,8 @@ using LogicalOperatorCompositeVisitor = utils::CompositeVisitor<
Expand, ExpandVariable, ConstructNamedPath, Filter, Produce, Delete,
SetProperty, SetProperties, SetLabels, RemoveProperty, RemoveLabels,
EdgeUniquenessFilter, Accumulate, Aggregate, Skip, Limit, OrderBy, Merge,
Optional, Unwind, Distinct, Union, Cartesian, CallProcedure, LoadCsv, Foreach, EmptyResult>;
Optional, Unwind, Distinct, Union, Cartesian, CallProcedure, LoadCsv, Foreach, EmptyResult,
EvaluatePatternFilter, Apply>;
using LogicalOperatorLeafVisitor = utils::LeafVisitor<Once>;
@@ -1122,6 +1125,9 @@ pulled.")
((input "std::shared_ptr<LogicalOperator>" :scope :public
:slk-save #'slk-save-operator-pointer
:slk-load #'slk-load-operator-pointer)
(pattern_filters "std::vector<std::shared_ptr<LogicalOperator>>" :scope :public
:slk-save #'slk-save-ast-vector
:slk-load (slk-load-ast-vector "std::shared_ptr<LogicalOperator>"))
(expression "Expression *" :scope :public
:slk-save #'slk-save-ast-pointer
:slk-load (slk-load-ast-pointer "Expression")))
@@ -1136,6 +1142,7 @@ a boolean value.")
Filter() {}
Filter(const std::shared_ptr<LogicalOperator> &input_,
const std::vector<std::shared_ptr<LogicalOperator>> &pattern_filters_,
Expression *expression_);
bool Accept(HierarchicalLogicalOperatorVisitor &visitor) override;
UniqueCursorPtr MakeCursor(utils::MemoryResource *) const override;
@@ -1159,6 +1166,7 @@ a boolean value.")
private:
const Filter &self_;
const UniqueCursorPtr input_cursor_;
const std::vector<UniqueCursorPtr> pattern_filter_cursors_;
};
cpp<#)
(:serialize (:slk))
@@ -1777,6 +1785,45 @@ operator's implementation does not expect this.")
(:serialize (:slk))
(:clone))
(lcp:define-class evaluate-pattern-filter (logical-operator)
((input "std::shared_ptr<LogicalOperator>" :scope :public
:slk-save #'slk-save-operator-pointer
:slk-load #'slk-load-operator-pointer)
(output-symbol "Symbol" :scope :public))
(:documentation "Applies the pattern filter by putting the value of the input cursor to the frame.")
(:public
#>cpp
EvaluatePatternFilter() {}
EvaluatePatternFilter(const std::shared_ptr<LogicalOperator> &input, Symbol output_symbol);
bool Accept(HierarchicalLogicalOperatorVisitor &visitor) override;
UniqueCursorPtr MakeCursor(utils::MemoryResource *) const override;
std::vector<Symbol> ModifiedSymbols(const SymbolTable &) const override;
bool HasSingleInput() const override { return true; }
std::shared_ptr<LogicalOperator> input() const override { return input_; }
void set_input(std::shared_ptr<LogicalOperator> input) override {
input_ = input;
}
cpp<#)
(:private
#>cpp
class EvaluatePatternFilterCursor : public Cursor {
public:
EvaluatePatternFilterCursor(const EvaluatePatternFilter &, utils::MemoryResource *);
bool Pull(Frame &, ExecutionContext &) override;
void Shutdown() override;
void Reset() override;
private:
const EvaluatePatternFilter &self_;
UniqueCursorPtr input_cursor_;
};
cpp<#)
(:serialize (:slk))
(:clone))
(lcp:define-class limit (logical-operator)
((input "std::shared_ptr<LogicalOperator>" :scope :public
:slk-save #'slk-save-operator-pointer
@@ -2336,6 +2383,52 @@ clauses.
(:serialize (:slk))
(:clone))
(lcp:define-class apply (logical-operator)
((input "std::shared_ptr<LogicalOperator>" :scope :public
:slk-save #'slk-save-operator-pointer
:slk-load #'slk-load-operator-pointer)
(subquery "std::shared_ptr<LogicalOperator>" :scope :public
:slk-save #'slk-save-operator-pointer
:slk-load #'slk-load-operator-pointer)
(subquery-has-return "bool" :scope :public))
(:documentation "Applies symbols from both output branches.")
(:public
#>cpp
Apply() {}
Apply(const std::shared_ptr<LogicalOperator> input, const std::shared_ptr<LogicalOperator> subquery, bool subquery_has_return);
bool Accept(HierarchicalLogicalOperatorVisitor &visitor) override;
UniqueCursorPtr MakeCursor(utils::MemoryResource *) const override;
std::vector<Symbol> ModifiedSymbols(const SymbolTable &) const override;
bool HasSingleInput() const override { return true; }
std::shared_ptr<LogicalOperator> input() const override { return input_; }
void set_input(std::shared_ptr<LogicalOperator> input) override {
input_ = input;
}
cpp<#)
(:private
#>cpp
class ApplyCursor : public Cursor {
public:
ApplyCursor(const Apply &, utils::MemoryResource *);
bool Pull(Frame &, ExecutionContext &) override;
void Shutdown() override;
void Reset() override;
private:
const Apply &self_;
UniqueCursorPtr input_;
UniqueCursorPtr subquery_;
bool pull_input_{true};
bool subquery_has_return_{true};
};
cpp<#)
(:serialize (:slk))
(:clone))
(lcp:pop-namespace) ;; plan
(lcp:pop-namespace) ;; query
(lcp:pop-namespace) ;; memgraph

View File

@@ -0,0 +1,151 @@
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
// License, and you may not use this file except in compliance with the Business Source License.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.
#include <cstdint>
#include "query/plan/operator.hpp"
namespace memgraph {
constexpr utils::TypeInfo query::plan::LogicalOperator::kType{utils::TypeId::LOGICAL_OPERATOR, "LogicalOperator",
nullptr};
constexpr utils::TypeInfo query::plan::Once::kType{utils::TypeId::ONCE, "Once", &query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::NodeCreationInfo::kType{utils::TypeId::NODE_CREATION_INFO, "NodeCreationInfo",
nullptr};
constexpr utils::TypeInfo query::plan::CreateNode::kType{utils::TypeId::CREATE_NODE, "CreateNode",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::EdgeCreationInfo::kType{utils::TypeId::EDGE_CREATION_INFO, "EdgeCreationInfo",
nullptr};
constexpr utils::TypeInfo query::plan::CreateExpand::kType{utils::TypeId::CREATE_EXPAND, "CreateExpand",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::ScanAll::kType{utils::TypeId::SCAN_ALL, "ScanAll",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::ScanAllByLabel::kType{utils::TypeId::SCAN_ALL_BY_LABEL, "ScanAllByLabel",
&query::plan::ScanAll::kType};
constexpr utils::TypeInfo query::plan::ScanAllByLabelPropertyRange::kType{
utils::TypeId::SCAN_ALL_BY_LABEL_PROPERTY_RANGE, "ScanAllByLabelPropertyRange", &query::plan::ScanAll::kType};
constexpr utils::TypeInfo query::plan::ScanAllByLabelPropertyValue::kType{
utils::TypeId::SCAN_ALL_BY_LABEL_PROPERTY_VALUE, "ScanAllByLabelPropertyValue", &query::plan::ScanAll::kType};
constexpr utils::TypeInfo query::plan::ScanAllByLabelProperty::kType{
utils::TypeId::SCAN_ALL_BY_LABEL_PROPERTY, "ScanAllByLabelProperty", &query::plan::ScanAll::kType};
constexpr utils::TypeInfo query::plan::ScanAllById::kType{utils::TypeId::SCAN_ALL_BY_ID, "ScanAllById",
&query::plan::ScanAll::kType};
constexpr utils::TypeInfo query::plan::ExpandCommon::kType{utils::TypeId::EXPAND_COMMON, "ExpandCommon", nullptr};
constexpr utils::TypeInfo query::plan::Expand::kType{utils::TypeId::EXPAND, "Expand",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::ExpansionLambda::kType{utils::TypeId::EXPANSION_LAMBDA, "ExpansionLambda",
nullptr};
constexpr utils::TypeInfo query::plan::ExpandVariable::kType{utils::TypeId::EXPAND_VARIABLE, "ExpandVariable",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::ConstructNamedPath::kType{
utils::TypeId::CONSTRUCT_NAMED_PATH, "ConstructNamedPath", &query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Filter::kType{utils::TypeId::FILTER, "Filter",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Produce::kType{utils::TypeId::PRODUCE, "Produce",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Delete::kType{utils::TypeId::DELETE, "Delete",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::SetProperty::kType{utils::TypeId::SET_PROPERTY, "SetProperty",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::SetProperties::kType{utils::TypeId::SET_PROPERTIES, "SetProperties",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::SetLabels::kType{utils::TypeId::SET_LABELS, "SetLabels",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::RemoveProperty::kType{utils::TypeId::REMOVE_PROPERTY, "RemoveProperty",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::RemoveLabels::kType{utils::TypeId::REMOVE_LABELS, "RemoveLabels",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::EdgeUniquenessFilter::kType{
utils::TypeId::EDGE_UNIQUENESS_FILTER, "EdgeUniquenessFilter", &query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::EmptyResult::kType{utils::TypeId::EMPTY_RESULT, "EmptyResult",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Accumulate::kType{utils::TypeId::ACCUMULATE, "Accumulate",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Aggregate::Element::kType{utils::TypeId::AGGREGATE_ELEMENT, "Element", nullptr};
constexpr utils::TypeInfo query::plan::Aggregate::kType{utils::TypeId::AGGREGATE, "Aggregate",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Skip::kType{utils::TypeId::SKIP, "Skip", &query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::EvaluatePatternFilter::kType{
utils::TypeId::EVALUATE_PATTERN_FILTER, "EvaluatePatternFilter", &query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Limit::kType{utils::TypeId::LIMIT, "Limit",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::OrderBy::kType{utils::TypeId::ORDERBY, "OrderBy",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Merge::kType{utils::TypeId::MERGE, "Merge",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Optional::kType{utils::TypeId::OPTIONAL, "Optional",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Unwind::kType{utils::TypeId::UNWIND, "Unwind",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Distinct::kType{utils::TypeId::DISTINCT, "Distinct",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Union::kType{utils::TypeId::UNION, "Union",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Cartesian::kType{utils::TypeId::CARTESIAN, "Cartesian",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::OutputTable::kType{utils::TypeId::OUTPUT_TABLE, "OutputTable",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::OutputTableStream::kType{utils::TypeId::OUTPUT_TABLE_STREAM, "OutputTableStream",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::CallProcedure::kType{utils::TypeId::CALL_PROCEDURE, "CallProcedure",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::LoadCsv::kType{utils::TypeId::LOAD_CSV, "LoadCsv",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Foreach::kType{utils::TypeId::FOREACH, "Foreach",
&query::plan::LogicalOperator::kType};
constexpr utils::TypeInfo query::plan::Apply::kType{utils::TypeId::APPLY, "Apply",
&query::plan::LogicalOperator::kType};
} // namespace memgraph

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -50,23 +50,6 @@ class PostProcessor final {
double EstimatePlanCost(const std::unique_ptr<LogicalOperator> &plan, TVertexCounts *vertex_counts) {
return query::plan::EstimatePlanCost(vertex_counts, parameters_, *plan);
}
template <class TPlanningContext>
std::unique_ptr<LogicalOperator> MergeWithCombinator(std::unique_ptr<LogicalOperator> curr_op,
std::unique_ptr<LogicalOperator> last_op, const Tree &combinator,
TPlanningContext *context) {
if (const auto *union_ = utils::Downcast<const CypherUnion>(&combinator)) {
return std::unique_ptr<LogicalOperator>(
impl::GenUnion(*union_, std::move(last_op), std::move(curr_op), *context->symbol_table));
}
throw utils::NotYetImplemented("query combinator");
}
template <class TPlanningContext>
std::unique_ptr<LogicalOperator> MakeDistinct(std::unique_ptr<LogicalOperator> last_op, TPlanningContext *context) {
auto output_symbols = last_op->OutputSymbols(*context->symbol_table);
return std::make_unique<Distinct>(std::move(last_op), output_symbols);
}
};
/// @brief Generates the LogicalOperator tree for a single query and returns the
@@ -82,10 +65,9 @@ class PostProcessor final {
/// @sa RuleBasedPlanner
/// @sa VariableStartPlanner
template <template <class> class TPlanner, class TDbAccessor>
auto MakeLogicalPlanForSingleQuery(std::vector<SingleQueryPart> single_query_parts,
PlanningContext<TDbAccessor> *context) {
auto MakeLogicalPlanForSingleQuery(QueryParts query_parts, PlanningContext<TDbAccessor> *context) {
context->bound_symbols.clear();
return TPlanner<PlanningContext<TDbAccessor>>(context).Plan(single_query_parts);
return TPlanner<PlanningContext<TDbAccessor>>(context).Plan(query_parts);
}
/// Generates the LogicalOperator tree and returns the resulting plan.
@@ -103,48 +85,34 @@ template <class TPlanningContext, class TPlanPostProcess>
auto MakeLogicalPlan(TPlanningContext *context, TPlanPostProcess *post_process, bool use_variable_planner) {
auto query_parts = CollectQueryParts(*context->symbol_table, *context->ast_storage, context->query);
auto &vertex_counts = *context->db;
double total_cost = 0;
double total_cost = std::numeric_limits<double>::max();
using ProcessedPlan = typename TPlanPostProcess::ProcessedPlan;
ProcessedPlan last_plan;
ProcessedPlan plan_with_least_cost;
for (const auto &query_part : query_parts.query_parts) {
std::optional<ProcessedPlan> curr_plan;
double min_cost = std::numeric_limits<double>::max();
if (use_variable_planner) {
auto plans = MakeLogicalPlanForSingleQuery<VariableStartPlanner>(query_part.single_query_parts, context);
for (auto plan : plans) {
// Plans are generated lazily and the current plan will disappear, so
// it's ok to move it.
auto rewritten_plan = post_process->Rewrite(std::move(plan), context);
double cost = post_process->EstimatePlanCost(rewritten_plan, &vertex_counts);
if (!curr_plan || cost < min_cost) {
curr_plan.emplace(std::move(rewritten_plan));
min_cost = cost;
}
}
} else {
auto plan = MakeLogicalPlanForSingleQuery<RuleBasedPlanner>(query_part.single_query_parts, context);
std::optional<ProcessedPlan> curr_plan;
if (use_variable_planner) {
auto plans = MakeLogicalPlanForSingleQuery<VariableStartPlanner>(query_parts, context);
for (auto plan : plans) {
// Plans are generated lazily and the current plan will disappear, so
// it's ok to move it.
auto rewritten_plan = post_process->Rewrite(std::move(plan), context);
min_cost = post_process->EstimatePlanCost(rewritten_plan, &vertex_counts);
curr_plan.emplace(std::move(rewritten_plan));
}
total_cost += min_cost;
if (query_part.query_combinator) {
last_plan = post_process->MergeWithCombinator(std::move(*curr_plan), std::move(last_plan),
*query_part.query_combinator, context);
} else {
last_plan = std::move(*curr_plan);
double cost = post_process->EstimatePlanCost(rewritten_plan, &vertex_counts);
if (!curr_plan || cost < total_cost) {
curr_plan.emplace(std::move(rewritten_plan));
total_cost = cost;
}
}
} else {
auto plan = MakeLogicalPlanForSingleQuery<RuleBasedPlanner>(query_parts, context);
auto rewritten_plan = post_process->Rewrite(std::move(plan), context);
total_cost = post_process->EstimatePlanCost(rewritten_plan, &vertex_counts);
curr_plan.emplace(std::move(rewritten_plan));
}
if (query_parts.distinct) {
last_plan = post_process->MakeDistinct(std::move(last_plan), context);
}
plan_with_least_cost = std::move(*curr_plan);
return std::make_pair(std::move(last_plan), total_cost);
return std::make_pair(std::move(plan_with_least_cost), total_cost);
}
template <class TPlanningContext>

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -84,56 +84,6 @@ std::vector<Expansion> NormalizePatterns(const SymbolTable &symbol_table, const
return expansions;
}
// Fills the given Matching, by converting the Match patterns to normalized
// representation as Expansions. Filters used in the Match are also collected,
// as well as edge symbols which determine Cyphermorphism. Collecting filters
// will lift them out of a pattern and generate new expressions (just like they
// were in a Where clause).
void AddMatching(const std::vector<Pattern *> &patterns, Where *where, SymbolTable &symbol_table, AstStorage &storage,
Matching &matching) {
auto expansions = NormalizePatterns(symbol_table, patterns);
std::unordered_set<Symbol> edge_symbols;
for (const auto &expansion : expansions) {
// Matching may already have some expansions, so offset our index.
const size_t expansion_ix = matching.expansions.size();
// Map node1 symbol to expansion
const auto &node1_sym = symbol_table.at(*expansion.node1->identifier_);
matching.node_symbol_to_expansions[node1_sym].insert(expansion_ix);
// Add node1 to all symbols.
matching.expansion_symbols.insert(node1_sym);
if (expansion.edge) {
const auto &edge_sym = symbol_table.at(*expansion.edge->identifier_);
// Fill edge symbols for Cyphermorphism.
edge_symbols.insert(edge_sym);
// Map node2 symbol to expansion
const auto &node2_sym = symbol_table.at(*expansion.node2->identifier_);
matching.node_symbol_to_expansions[node2_sym].insert(expansion_ix);
// Add edge and node2 to all symbols
matching.expansion_symbols.insert(edge_sym);
matching.expansion_symbols.insert(node2_sym);
}
matching.expansions.push_back(expansion);
}
if (!edge_symbols.empty()) {
matching.edge_symbols.emplace_back(edge_symbols);
}
for (auto *pattern : patterns) {
matching.filters.CollectPatternFilters(*pattern, symbol_table, storage);
if (pattern->identifier_->user_declared_) {
std::vector<Symbol> path_elements;
for (auto *pattern_atom : pattern->atoms_)
path_elements.emplace_back(symbol_table.at(*pattern_atom->identifier_));
matching.named_paths.emplace(symbol_table.at(*pattern->identifier_), std::move(path_elements));
}
}
if (where) {
matching.filters.CollectWhereFilter(*where, symbol_table);
}
}
void AddMatching(const Match &match, SymbolTable &symbol_table, AstStorage &storage, Matching &matching) {
return AddMatching(match.patterns_, match.where_, symbol_table, storage, matching);
}
auto SplitExpressionOnAnd(Expression *expression) {
// TODO: Think about converting all filtering expression into CNF to improve
// the granularity of filters which can be stand alone.
@@ -519,6 +469,8 @@ void Filters::AnalyzeAndStoreFilter(Expression *expr, const SymbolTable &symbol_
if (!add_prop_is_not_null_check(is_not_null)) {
all_filters_.emplace_back(make_filter(FilterInfo::Type::Generic));
}
} else if (auto *exists = utils::Downcast<Exists>(expr)) {
all_filters_.emplace_back(make_filter(FilterInfo::Type::Pattern));
} else {
all_filters_.emplace_back(make_filter(FilterInfo::Type::Generic));
}
@@ -528,6 +480,78 @@ void Filters::AnalyzeAndStoreFilter(Expression *expr, const SymbolTable &symbol_
// as `expr1 < n.prop AND n.prop < expr2`.
}
// Fills the given Matching, by converting the Match patterns to normalized
// representation as Expansions. Filters used in the Match are also collected,
// as well as edge symbols which determine Cyphermorphism. Collecting filters
// will lift them out of a pattern and generate new expressions (just like they
// were in a Where clause).
void AddMatching(const std::vector<Pattern *> &patterns, Where *where, SymbolTable &symbol_table, AstStorage &storage,
Matching &matching) {
auto expansions = NormalizePatterns(symbol_table, patterns);
std::unordered_set<Symbol> edge_symbols;
for (const auto &expansion : expansions) {
// Matching may already have some expansions, so offset our index.
const size_t expansion_ix = matching.expansions.size();
// Map node1 symbol to expansion
const auto &node1_sym = symbol_table.at(*expansion.node1->identifier_);
matching.node_symbol_to_expansions[node1_sym].insert(expansion_ix);
// Add node1 to all symbols.
matching.expansion_symbols.insert(node1_sym);
if (expansion.edge) {
const auto &edge_sym = symbol_table.at(*expansion.edge->identifier_);
// Fill edge symbols for Cyphermorphism.
edge_symbols.insert(edge_sym);
// Map node2 symbol to expansion
const auto &node2_sym = symbol_table.at(*expansion.node2->identifier_);
matching.node_symbol_to_expansions[node2_sym].insert(expansion_ix);
// Add edge and node2 to all symbols
matching.expansion_symbols.insert(edge_sym);
matching.expansion_symbols.insert(node2_sym);
}
matching.expansions.push_back(expansion);
}
if (!edge_symbols.empty()) {
matching.edge_symbols.emplace_back(edge_symbols);
}
for (auto *const pattern : patterns) {
matching.filters.CollectPatternFilters(*pattern, symbol_table, storage);
if (pattern->identifier_->user_declared_) {
std::vector<Symbol> path_elements;
for (auto *const pattern_atom : pattern->atoms_)
path_elements.push_back(symbol_table.at(*pattern_atom->identifier_));
matching.named_paths.emplace(symbol_table.at(*pattern->identifier_), std::move(path_elements));
}
}
if (where) {
matching.filters.CollectWhereFilter(*where, symbol_table);
}
}
void AddMatching(const Match &match, SymbolTable &symbol_table, AstStorage &storage, Matching &matching) {
AddMatching(match.patterns_, match.where_, symbol_table, storage, matching);
// If there are any pattern filters, we add those as well
for (auto &filter : matching.filters) {
PatternFilterVisitor visitor(symbol_table, storage);
filter.expression->Accept(visitor);
filter.matchings = visitor.getMatchings();
}
}
void PatternFilterVisitor::Visit(Exists &op) {
std::vector<Pattern *> patterns;
patterns.push_back(op.pattern_);
FilterMatching filter_matching;
AddMatching(patterns, nullptr, symbol_table_, storage_, filter_matching);
filter_matching.type = PatternFilterType::EXISTS;
filter_matching.symbol = std::make_optional<Symbol>(symbol_table_.at(op));
matchings_.push_back(std::move(filter_matching));
}
static void ParseForeach(query::Foreach &foreach, SingleQueryPart &query_part, AstStorage &storage,
SymbolTable &symbol_table) {
for (auto *clause : foreach.clauses_) {
@@ -560,6 +584,9 @@ std::vector<SingleQueryPart> CollectSingleQueryParts(SymbolTable &symbol_table,
if (auto *merge = utils::Downcast<query::Merge>(clause)) {
query_part->merge_matching.emplace_back(Matching{});
AddMatching({merge->pattern_}, nullptr, symbol_table, storage, query_part->merge_matching.back());
} else if (auto *call_subquery = utils::Downcast<query::CallSubquery>(clause)) {
query_part->subqueries.emplace_back(
std::make_shared<QueryParts>(CollectQueryParts(symbol_table, storage, call_subquery->cypher_query_)));
} else if (auto *foreach = utils::Downcast<query::Foreach>(clause)) {
ParseForeach(*foreach, *query_part, storage, symbol_table);
} else if (utils::IsSubtype(*clause, With::kType) || utils::IsSubtype(*clause, query::Unwind::kType) ||

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -70,7 +70,26 @@ class UsedSymbolsCollector : public HierarchicalTreeVisitor {
}
bool Visit(Identifier &ident) override {
symbols_.insert(symbol_table_.at(ident));
if (!in_exists || ident.user_declared_) {
symbols_.insert(symbol_table_.at(ident));
}
return true;
}
bool PreVisit(Exists &exists) override {
in_exists = true;
// We do not visit pattern identifier since we're in exists filter pattern
for (auto &atom : exists.pattern_->atoms_) {
atom->Accept(*this);
}
return false;
}
bool PostVisit(Exists & /*exists*/) override {
in_exists = false;
return true;
}
@@ -79,6 +98,9 @@ class UsedSymbolsCollector : public HierarchicalTreeVisitor {
std::unordered_set<Symbol> symbols_;
const SymbolTable &symbol_table_;
private:
bool in_exists{false};
};
/// Normalized representation of a pattern that needs to be matched.
@@ -99,6 +121,93 @@ struct Expansion {
NodeAtom *node2 = nullptr;
};
struct FilterMatching;
enum class PatternFilterType { EXISTS };
/// Collects matchings from filters that include patterns
class PatternFilterVisitor : public ExpressionVisitor<void> {
public:
explicit PatternFilterVisitor(SymbolTable &symbol_table, AstStorage &storage)
: symbol_table_(symbol_table), storage_(storage) {}
using ExpressionVisitor<void>::Visit;
// Unary operators
void Visit(NotOperator &op) override { op.expression_->Accept(*this); }
void Visit(IsNullOperator &op) override { op.expression_->Accept(*this); };
void Visit(UnaryPlusOperator &op) override{};
void Visit(UnaryMinusOperator &op) override{};
// Binary operators
void Visit(OrOperator &op) override {
op.expression1_->Accept(*this);
op.expression2_->Accept(*this);
}
void Visit(XorOperator &op) override {
op.expression1_->Accept(*this);
op.expression2_->Accept(*this);
}
void Visit(AndOperator &op) override {
op.expression1_->Accept(*this);
op.expression2_->Accept(*this);
}
void Visit(NotEqualOperator &op) override {
op.expression1_->Accept(*this);
op.expression2_->Accept(*this);
};
void Visit(EqualOperator &op) override {
op.expression1_->Accept(*this);
op.expression2_->Accept(*this);
};
void Visit(InListOperator &op) override {
op.expression1_->Accept(*this);
op.expression2_->Accept(*this);
};
void Visit(AdditionOperator &op) override{};
void Visit(SubtractionOperator &op) override{};
void Visit(MultiplicationOperator &op) override{};
void Visit(DivisionOperator &op) override{};
void Visit(ModOperator &op) override{};
void Visit(LessOperator &op) override{};
void Visit(GreaterOperator &op) override{};
void Visit(LessEqualOperator &op) override{};
void Visit(GreaterEqualOperator &op) override{};
void Visit(SubscriptOperator &op) override{};
// Other
void Visit(ListSlicingOperator &op) override{};
void Visit(IfOperator &op) override{};
void Visit(ListLiteral &op) override{};
void Visit(MapLiteral &op) override{};
void Visit(LabelsTest &op) override{};
void Visit(Aggregation &op) override{};
void Visit(Function &op) override{};
void Visit(Reduce &op) override{};
void Visit(Coalesce &op) override{};
void Visit(Extract &op) override{};
void Visit(Exists &op) override;
void Visit(All &op) override{};
void Visit(Single &op) override{};
void Visit(Any &op) override{};
void Visit(None &op) override{};
void Visit(Identifier &op) override{};
void Visit(PrimitiveLiteral &op) override{};
void Visit(PropertyLookup &op) override{};
void Visit(ParameterLookup &op) override{};
void Visit(NamedExpression &op) override{};
void Visit(RegexMatch &op) override{};
std::vector<FilterMatching> getMatchings() { return matchings_; }
SymbolTable &symbol_table_;
AstStorage &storage_;
private:
/// Collection of matchings in the filter expression being analyzed.
std::vector<FilterMatching> matchings_;
};
/// Stores the symbols and expression used to filter a property.
class PropertyFilter {
public:
@@ -153,7 +262,7 @@ struct FilterInfo {
/// applied for labels or a property. Non generic types contain extra
/// information which can be used to produce indexed scans of graph
/// elements.
enum class Type { Generic, Label, Property, Id };
enum class Type { Generic, Label, Property, Id, Pattern };
Type type;
/// The original filter expression which must be satisfied.
@@ -166,6 +275,8 @@ struct FilterInfo {
std::optional<PropertyFilter> property_filter;
/// Information for Type::Id filtering.
std::optional<IdFilter> id_filter;
/// Matchings for filters that include patterns
std::vector<FilterMatching> matchings;
};
/// Stores information on filters used inside the @c Matching of a @c QueryPart.
@@ -287,6 +398,16 @@ struct Matching {
std::unordered_set<Symbol> expansion_symbols{};
};
// TODO clumsy to need to declare it before, usually only the struct definition would be in header
struct QueryParts;
struct FilterMatching : Matching {
/// Type of pattern filter
PatternFilterType type;
/// Symbol for the filter expression
std::optional<Symbol> symbol;
};
/// @brief Represents a read (+ write) part of a query. Parts are split on
/// `WITH` clauses.
///
@@ -331,6 +452,9 @@ struct SingleQueryPart {
std::vector<Matching> merge_matching{};
/// @brief All the remaining clauses (without @c Match).
std::vector<Clause *> remaining_clauses{};
/// The subqueries vector are all the subqueries in this query part ordered in a list by
/// the order of calling.
std::vector<std::shared_ptr<QueryParts>> subqueries{};
};
/// Holds query parts of a single query together with the optional information

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -14,6 +14,7 @@
#include "query/db_accessor.hpp"
#include "query/frontend/ast/pretty_print.hpp"
#include "query/plan/operator.hpp"
#include "utils/string.hpp"
namespace memgraph::query::plan {
@@ -148,7 +149,6 @@ bool PlanPrinter::PreVisit(query::plan::Produce &op) {
}
PRE_VISIT(ConstructNamedPath);
PRE_VISIT(Filter);
PRE_VISIT(SetProperty);
PRE_VISIT(SetProperties);
PRE_VISIT(SetLabels);
@@ -157,6 +157,7 @@ PRE_VISIT(RemoveLabels);
PRE_VISIT(EdgeUniquenessFilter);
PRE_VISIT(Accumulate);
PRE_VISIT(EmptyResult);
PRE_VISIT(EvaluatePatternFilter);
bool PlanPrinter::PreVisit(query::plan::Aggregate &op) {
WithPrintLn([&](auto &out) {
@@ -251,6 +252,22 @@ bool PlanPrinter::PreVisit(query::plan::Foreach &op) {
op.input_->Accept(*this);
return false;
}
bool PlanPrinter::PreVisit(query::plan::Filter &op) {
WithPrintLn([](auto &out) { out << "* Filter"; });
for (const auto &pattern_filter : op.pattern_filters_) {
Branch(*pattern_filter);
}
op.input_->Accept(*this);
return false;
}
bool PlanPrinter::PreVisit(query::plan::Apply &op) {
WithPrintLn([](auto &out) { out << "* Apply"; });
Branch(*op.subquery_);
op.input_->Accept(*this);
return false;
}
#undef PRE_VISIT
bool PlanPrinter::DefaultPreVisit() {
@@ -589,6 +606,13 @@ bool PlanToJsonVisitor::PreVisit(Filter &op) {
op.input_->Accept(*this);
self["input"] = PopOutput();
for (auto pattern_idx = 0; pattern_idx < op.pattern_filters_.size(); pattern_idx++) {
auto pattern_filter_key = "pattern_filter" + std::to_string(pattern_idx + 1);
op.pattern_filters_[pattern_idx]->Accept(*this);
self[pattern_filter_key] = PopOutput();
}
output_ = std::move(self);
return false;
}
@@ -908,6 +932,7 @@ bool PlanToJsonVisitor::PreVisit(Cartesian &op) {
output_ = std::move(self);
return false;
}
bool PlanToJsonVisitor::PreVisit(Foreach &op) {
json self;
self["name"] = "Foreach";
@@ -924,6 +949,32 @@ bool PlanToJsonVisitor::PreVisit(Foreach &op) {
return false;
}
bool PlanToJsonVisitor::PreVisit(EvaluatePatternFilter &op) {
json self;
self["name"] = "EvaluatePatternFilter";
self["output_symbol"] = ToJson(op.output_symbol_);
op.input_->Accept(*this);
self["input"] = PopOutput();
output_ = std::move(self);
return false;
}
bool PlanToJsonVisitor::PreVisit(Apply &op) {
json self;
self["name"] = "Apply";
op.input_->Accept(*this);
self["input"] = PopOutput();
op.subquery_->Accept(*this);
self["subquery"] = PopOutput();
output_ = std::move(self);
return false;
}
} // namespace impl
} // namespace memgraph::query::plan

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -74,6 +74,7 @@ class PlanPrinter : public virtual HierarchicalLogicalOperatorVisitor {
bool PreVisit(ConstructNamedPath &) override;
bool PreVisit(Filter &) override;
bool PreVisit(EvaluatePatternFilter & /*unused*/) override;
bool PreVisit(EdgeUniquenessFilter &) override;
bool PreVisit(Merge &) override;
@@ -94,6 +95,7 @@ class PlanPrinter : public virtual HierarchicalLogicalOperatorVisitor {
bool PreVisit(CallProcedure &) override;
bool PreVisit(LoadCsv &) override;
bool PreVisit(Foreach &) override;
bool PreVisit(Apply & /*unused*/) override;
bool Visit(Once &) override;
@@ -186,8 +188,10 @@ class PlanToJsonVisitor : public virtual HierarchicalLogicalOperatorVisitor {
bool PreVisit(Optional &) override;
bool PreVisit(Filter &) override;
bool PreVisit(EvaluatePatternFilter & /*op*/) override;
bool PreVisit(EdgeUniquenessFilter &) override;
bool PreVisit(Cartesian &) override;
bool PreVisit(Apply & /*unused*/) override;
bool PreVisit(ScanAll &) override;
bool PreVisit(ScanAllByLabel &) override;

View File

@@ -91,18 +91,25 @@ bool ReadWriteTypeChecker::PreVisit([[maybe_unused]] Foreach &op) {
bool ReadWriteTypeChecker::Visit(Once &) { return false; } // NOLINT(hicpp-named-parameter)
void ReadWriteTypeChecker::UpdateType(RWType op_type) {
// Update type only if it's not the NONE type and the current operator's type
// is different than the one that's currently inferred.
if (type != RWType::NONE && type != op_type) {
type = RWType::RW;
}
// Stop inference because RW is the most "dominant" type, i.e. it isn't
// affected by the type of nodes in the plan appearing after the node for
// which the type is set to RW.
if (type == RWType::RW) {
return;
}
if (type == RWType::NONE && op_type != RWType::NONE) {
// if op_type is NONE, type doesn't change.
if (op_type == RWType::NONE) {
return;
}
// Update type only if it's not the NONE type and the current operator's type
// is different than the one that's currently inferred.
if (type != RWType::NONE && type != op_type) {
type = RWType::RW;
}
if (type == RWType::NONE) {
type = op_type;
}
}

View File

@@ -15,7 +15,7 @@
namespace memgraph::query::plan {
class ReadWriteTypeChecker : public virtual HierarchicalLogicalOperatorVisitor {
struct ReadWriteTypeChecker : public virtual HierarchicalLogicalOperatorVisitor {
public:
ReadWriteTypeChecker() = default;
@@ -89,7 +89,6 @@ class ReadWriteTypeChecker : public virtual HierarchicalLogicalOperatorVisitor {
bool Visit(Once &) override;
private:
void UpdateType(RWType op_type);
};

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -27,6 +27,7 @@
#include "query/plan/operator.hpp"
#include "query/plan/preprocess.hpp"
#include "storage/v2/indices.hpp"
DECLARE_int64(query_vertex_count_to_expand_existing);
@@ -444,6 +445,8 @@ class IndexLookupRewriter final : public HierarchicalLogicalOperatorVisitor {
bool PreVisit(Foreach &op) override {
prev_ops_.push_back(&op);
op.input()->Accept(*this);
RewriteBranch(&op.update_clauses_);
return false;
}
@@ -452,6 +455,28 @@ class IndexLookupRewriter final : public HierarchicalLogicalOperatorVisitor {
return true;
}
bool PreVisit(EvaluatePatternFilter &op) override {
prev_ops_.push_back(&op);
return true;
}
bool PostVisit(EvaluatePatternFilter & /*op*/) override {
prev_ops_.pop_back();
return true;
}
bool PreVisit(Apply &op) override {
prev_ops_.push_back(&op);
op.input()->Accept(*this);
RewriteBranch(&op.subquery_);
return false;
}
bool PostVisit(Apply & /*op*/) override {
prev_ops_.pop_back();
return true;
}
std::shared_ptr<LogicalOperator> new_root_;
private:
@@ -470,6 +495,7 @@ class IndexLookupRewriter final : public HierarchicalLogicalOperatorVisitor {
// FilterInfo with PropertyFilter.
FilterInfo filter;
int64_t vertex_count;
std::optional<storage::IndexStats> index_stats;
};
bool DefaultPreVisit() override { throw utils::NotYetImplemented("optimizing index lookup"); }
@@ -510,8 +536,11 @@ class IndexLookupRewriter final : public HierarchicalLogicalOperatorVisitor {
return best_label;
}
// Finds the label-property combination which has indexed the lowest amount of
// vertices. If the index cannot be found, nullopt is returned.
// Finds the label-property combination. The first criteria based on number of vertices indexed -> if one index has
// 10x less than the other one, always choose the smaller one. Otherwise, choose the index with smallest average group
// size based on key distribution. If average group size is equal, choose the index that has distribution closer to
// uniform distribution. Conditions based on average group size and key distribution can be only taken into account if
// the user has run `ANALYZE GRAPH` query before If the index cannot be found, nullopt is returned.
std::optional<LabelPropertyIndex> FindBestLabelPropertyIndex(const Symbol &symbol,
const std::unordered_set<Symbol> &bound_symbols) {
auto are_bound = [&bound_symbols](const auto &used_symbols) {
@@ -522,6 +551,31 @@ class IndexLookupRewriter final : public HierarchicalLogicalOperatorVisitor {
}
return true;
};
/*
* Comparator function between two indices. If new index has >= 10x vertices than the existing, it cannot be better.
* If it is <= 10x in number of vertices, check average group size of property values. The index with smaller
* average group size is better. If the average group size is the same, choose the one closer to the uniform
* distribution
* @param found: Current best label-property index.
* @param new_stats: Label-property index candidate.
* @param vertex_count: New index's number of vertices.
* @return -1 if the new index is better, 0 if they are equal and 1 if the existing one is better.
*/
auto compare_indices = [](std::optional<LabelPropertyIndex> &found, std::optional<storage::IndexStats> &new_stats,
int vertex_count) {
if (!new_stats.has_value()) {
return 0;
}
if (vertex_count / 10.0 > found->vertex_count) {
return 1;
}
int cmp_avg_group = utils::CompareDecimal(new_stats->avg_group_size, found->index_stats->avg_group_size);
if (cmp_avg_group != 0) return cmp_avg_group;
return utils::CompareDecimal(new_stats->statistic, found->index_stats->statistic);
};
std::optional<LabelPropertyIndex> found;
for (const auto &label : filters_.FilteredLabels(symbol)) {
for (const auto &filter : filters_.PropertyFilters(symbol)) {
@@ -536,7 +590,6 @@ class IndexLookupRewriter final : public HierarchicalLogicalOperatorVisitor {
if (!db_->LabelPropertyIndexExists(GetLabel(label), GetProperty(property))) {
continue;
}
int64_t vertex_count = db_->VerticesCount(GetLabel(label), GetProperty(property));
auto is_better_type = [&found](PropertyFilter::Type type) {
// Order the types by the most preferred index lookup type.
static const PropertyFilter::Type kFilterTypeOrder[] = {
@@ -545,17 +598,32 @@ class IndexLookupRewriter final : public HierarchicalLogicalOperatorVisitor {
auto *type_sort_ix = std::find(kFilterTypeOrder, kFilterTypeOrder + 3, type);
return type_sort_ix < found_sort_ix;
};
if (!found || vertex_count < found->vertex_count ||
(vertex_count == found->vertex_count && is_better_type(filter.property_filter->type_))) {
found = LabelPropertyIndex{label, filter, vertex_count};
int64_t vertex_count = db_->VerticesCount(GetLabel(label), GetProperty(property));
std::optional<storage::IndexStats> new_stats = db_->GetIndexStats(GetLabel(label), GetProperty(property));
// Conditions, from more to less important:
// the index with 10x less vertices is better.
// the index with smaller average group size is better.
// the index with equal avg group size and distribution closer to the uniform is better.
// the index with less vertices is better.
// the index with same number of vertices but more optimized filter is better.
if (!found || vertex_count * 10 < found->vertex_count) {
found = LabelPropertyIndex{label, filter, vertex_count, new_stats};
continue;
}
if (int cmp_res = compare_indices(found, new_stats, vertex_count);
cmp_res == -1 ||
cmp_res == 0 && (found->vertex_count > vertex_count ||
found->vertex_count == vertex_count && is_better_type(filter.property_filter->type_))) {
found = LabelPropertyIndex{label, filter, vertex_count, new_stats};
}
}
}
return found;
}
// Creates a ScanAll by the best possible index for the `node_symbol`. Best
// index is defined as the index with least number of vertices. If the node
// Creates a ScanAll by the best possible index for the `node_symbol`. If the node
// does not have at least a label, no indexed lookup can be created and
// `nullptr` is returned. The operator is chained after `input`. Optional
// `max_vertex_count` controls, whether no operator should be created if the

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -25,15 +25,6 @@ namespace memgraph::query::plan {
namespace {
bool HasBoundFilterSymbols(const std::unordered_set<Symbol> &bound_symbols, const FilterInfo &filter) {
for (const auto &symbol : filter.used_symbols) {
if (bound_symbols.find(symbol) == bound_symbols.end()) {
return false;
}
}
return true;
}
// Ast tree visitor which collects the context for a return body.
// The return body of WITH and RETURN clauses consists of:
//
@@ -495,7 +486,8 @@ std::unique_ptr<LogicalOperator> GenReturnBody(std::unique_ptr<LogicalOperator>
// Where may see new symbols so it comes after we generate Produce and in
// general, comes after any OrderBy, Skip or Limit.
if (body.where()) {
last_op = std::make_unique<Filter>(std::move(last_op), body.where()->expression_);
last_op = std::make_unique<Filter>(std::move(last_op), std::vector<std::shared_ptr<LogicalOperator>>{},
body.where()->expression_);
}
return last_op;
}
@@ -504,6 +496,12 @@ std::unique_ptr<LogicalOperator> GenReturnBody(std::unique_ptr<LogicalOperator>
namespace impl {
bool HasBoundFilterSymbols(const std::unordered_set<Symbol> &bound_symbols, const FilterInfo &filter) {
return std::ranges::all_of(
filter.used_symbols.begin(), filter.used_symbols.end(),
[&bound_symbols](const auto &symbol) { return bound_symbols.find(symbol) != bound_symbols.end(); });
}
Expression *ExtractFilters(const std::unordered_set<Symbol> &bound_symbols, Filters &filters, AstStorage &storage) {
Expression *filter_expr = nullptr;
for (auto filters_it = filters.begin(); filters_it != filters.end();) {
@@ -517,14 +515,20 @@ Expression *ExtractFilters(const std::unordered_set<Symbol> &bound_symbols, Filt
return filter_expr;
}
std::unique_ptr<LogicalOperator> GenFilters(std::unique_ptr<LogicalOperator> last_op,
const std::unordered_set<Symbol> &bound_symbols, Filters &filters,
AstStorage &storage) {
auto *filter_expr = ExtractFilters(bound_symbols, filters, storage);
if (filter_expr) {
last_op = std::make_unique<Filter>(std::move(last_op), filter_expr);
std::unordered_set<Symbol> GetSubqueryBoundSymbols(const std::vector<SingleQueryPart> &single_query_parts,
SymbolTable &symbol_table, AstStorage &storage) {
const auto &query = single_query_parts[0];
if (!query.matching.expansions.empty() || query.remaining_clauses.empty()) {
return {};
}
return last_op;
if (std::unordered_set<Symbol> bound_symbols; auto *with = utils::Downcast<query::With>(query.remaining_clauses[0])) {
auto input_op = impl::GenWith(*with, nullptr, symbol_table, false, bound_symbols, storage);
return bound_symbols;
}
return {};
}
std::unique_ptr<LogicalOperator> GenNamedPaths(std::unique_ptr<LogicalOperator> last_op,

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -49,6 +49,7 @@ struct PlanningContext {
/// write) the first `n`, but the latter `n` would only read the already
/// written information.
std::unordered_set<Symbol> bound_symbols{};
bool is_write_query{false};
};
template <class TDbAccessor>
@@ -81,8 +82,13 @@ namespace impl {
// removed from `Filters`.
Expression *ExtractFilters(const std::unordered_set<Symbol> &, Filters &, AstStorage &);
std::unique_ptr<LogicalOperator> GenFilters(std::unique_ptr<LogicalOperator>, const std::unordered_set<Symbol> &,
Filters &, AstStorage &);
/// Checks if the filters has all the bound symbols to be included in the current part of the query
bool HasBoundFilterSymbols(const std::unordered_set<Symbol> &bound_symbols, const FilterInfo &filter);
// Returns the set of symbols for the subquery that are actually referenced from the outer scope and
// used in the subquery.
std::unordered_set<Symbol> GetSubqueryBoundSymbols(const std::vector<SingleQueryPart> &single_query_parts,
SymbolTable &symbol_table, AstStorage &storage);
/// Utility function for iterating pattern atoms and accumulating a result.
///
@@ -164,82 +170,96 @@ class RuleBasedPlanner {
/// tree.
using PlanResult = std::unique_ptr<LogicalOperator>;
/// @brief Generates the operator tree based on explicitly set rules.
PlanResult Plan(const std::vector<SingleQueryPart> &query_parts) {
PlanResult Plan(const QueryParts &query_parts) {
auto &context = *context_;
std::unique_ptr<LogicalOperator> input_op;
// Set to true if a query command writes to the database.
bool is_write = false;
for (const auto &query_part : query_parts) {
MatchContext match_ctx{query_part.matching, *context.symbol_table, context.bound_symbols};
input_op = PlanMatching(match_ctx, std::move(input_op));
for (const auto &matching : query_part.optional_matching) {
MatchContext opt_ctx{matching, *context.symbol_table, context.bound_symbols};
auto match_op = PlanMatching(opt_ctx, nullptr);
if (match_op) {
input_op = std::make_unique<Optional>(std::move(input_op), std::move(match_op), opt_ctx.new_symbols);
}
}
uint64_t merge_id = 0;
for (const auto &clause : query_part.remaining_clauses) {
MG_ASSERT(!utils::IsSubtype(*clause, Match::kType), "Unexpected Match in remaining clauses");
if (auto *ret = utils::Downcast<Return>(clause)) {
input_op = impl::GenReturn(*ret, std::move(input_op), *context.symbol_table, is_write, context.bound_symbols,
*context.ast_storage);
} else if (auto *merge = utils::Downcast<query::Merge>(clause)) {
input_op = GenMerge(*merge, std::move(input_op), query_part.merge_matching[merge_id++]);
// Treat MERGE clause as write, because we do not know if it will
// create anything.
is_write = true;
} else if (auto *with = utils::Downcast<query::With>(clause)) {
input_op = impl::GenWith(*with, std::move(input_op), *context.symbol_table, is_write, context.bound_symbols,
*context.ast_storage);
// WITH clause advances the command, so reset the flag.
is_write = false;
} else if (auto op = HandleWriteClause(clause, input_op, *context.symbol_table, context.bound_symbols)) {
is_write = true;
input_op = std::move(op);
} else if (auto *unwind = utils::Downcast<query::Unwind>(clause)) {
const auto &symbol = context.symbol_table->at(*unwind->named_expression_);
context.bound_symbols.insert(symbol);
input_op =
std::make_unique<plan::Unwind>(std::move(input_op), unwind->named_expression_->expression_, symbol);
std::unique_ptr<LogicalOperator> final_plan;
} else if (auto *call_proc = utils::Downcast<query::CallProcedure>(clause)) {
std::vector<Symbol> result_symbols;
result_symbols.reserve(call_proc->result_identifiers_.size());
for (const auto *ident : call_proc->result_identifiers_) {
const auto &sym = context.symbol_table->at(*ident);
context.bound_symbols.insert(sym);
result_symbols.push_back(sym);
for (const auto &query_part : query_parts.query_parts) {
std::unique_ptr<LogicalOperator> input_op;
context.is_write_query = false;
for (const auto &single_query_part : query_part.single_query_parts) {
input_op = HandleMatching(std::move(input_op), single_query_part, *context.symbol_table, context.bound_symbols);
uint64_t merge_id = 0;
uint64_t subquery_id = 0;
for (const auto &clause : single_query_part.remaining_clauses) {
MG_ASSERT(!utils::IsSubtype(*clause, Match::kType), "Unexpected Match in remaining clauses");
if (auto *ret = utils::Downcast<Return>(clause)) {
input_op = impl::GenReturn(*ret, std::move(input_op), *context.symbol_table, context.is_write_query,
context.bound_symbols, *context.ast_storage);
} else if (auto *merge = utils::Downcast<query::Merge>(clause)) {
input_op = GenMerge(*merge, std::move(input_op), single_query_part.merge_matching[merge_id++]);
// Treat MERGE clause as write, because we do not know if it will
// create anything.
context.is_write_query = true;
} else if (auto *with = utils::Downcast<query::With>(clause)) {
input_op = impl::GenWith(*with, std::move(input_op), *context.symbol_table, context.is_write_query,
context.bound_symbols, *context.ast_storage);
// WITH clause advances the command, so reset the flag.
context.is_write_query = false;
} else if (auto op = HandleWriteClause(clause, input_op, *context.symbol_table, context.bound_symbols)) {
context.is_write_query = true;
input_op = std::move(op);
} else if (auto *unwind = utils::Downcast<query::Unwind>(clause)) {
const auto &symbol = context.symbol_table->at(*unwind->named_expression_);
context.bound_symbols.insert(symbol);
input_op =
std::make_unique<plan::Unwind>(std::move(input_op), unwind->named_expression_->expression_, symbol);
} else if (auto *call_proc = utils::Downcast<query::CallProcedure>(clause)) {
std::vector<Symbol> result_symbols;
result_symbols.reserve(call_proc->result_identifiers_.size());
for (const auto *ident : call_proc->result_identifiers_) {
const auto &sym = context.symbol_table->at(*ident);
context.bound_symbols.insert(sym);
result_symbols.push_back(sym);
}
// TODO: When we add support for write and eager procedures, we will
// need to plan this operator with Accumulate and pass in
// storage::View::NEW.
input_op = std::make_unique<plan::CallProcedure>(
std::move(input_op), call_proc->procedure_name_, call_proc->arguments_, call_proc->result_fields_,
result_symbols, call_proc->memory_limit_, call_proc->memory_scale_, call_proc->is_write_);
} else if (auto *load_csv = utils::Downcast<query::LoadCsv>(clause)) {
const auto &row_sym = context.symbol_table->at(*load_csv->row_var_);
context.bound_symbols.insert(row_sym);
input_op =
std::make_unique<plan::LoadCsv>(std::move(input_op), load_csv->file_, load_csv->with_header_,
load_csv->ignore_bad_, load_csv->delimiter_, load_csv->quote_, row_sym);
} else if (auto *foreach = utils::Downcast<query::Foreach>(clause)) {
context.is_write_query = true;
input_op = HandleForeachClause(foreach, std::move(input_op), *context.symbol_table, context.bound_symbols,
single_query_part, merge_id);
} else if (auto *call_sub = utils::Downcast<query::CallSubquery>(clause)) {
input_op = HandleSubquery(std::move(input_op), single_query_part.subqueries[subquery_id++],
*context.symbol_table, *context_->ast_storage);
} else {
throw utils::NotYetImplemented("clause '{}' conversion to operator(s)", clause->GetTypeInfo().name);
}
// TODO: When we add support for write and eager procedures, we will
// need to plan this operator with Accumulate and pass in
// storage::View::NEW.
input_op = std::make_unique<plan::CallProcedure>(
std::move(input_op), call_proc->procedure_name_, call_proc->arguments_, call_proc->result_fields_,
result_symbols, call_proc->memory_limit_, call_proc->memory_scale_, call_proc->is_write_);
} else if (auto *load_csv = utils::Downcast<query::LoadCsv>(clause)) {
const auto &row_sym = context.symbol_table->at(*load_csv->row_var_);
context.bound_symbols.insert(row_sym);
input_op =
std::make_unique<plan::LoadCsv>(std::move(input_op), load_csv->file_, load_csv->with_header_,
load_csv->ignore_bad_, load_csv->delimiter_, load_csv->quote_, row_sym);
} else if (auto *foreach = utils::Downcast<query::Foreach>(clause)) {
is_write = true;
input_op = HandleForeachClause(foreach, std::move(input_op), *context.symbol_table, context.bound_symbols,
query_part, merge_id);
} else {
throw utils::NotYetImplemented("clause '{}' conversion to operator(s)", clause->GetTypeInfo().name);
}
}
// Is this the only situation that should be covered
if (input_op->OutputSymbols(*context.symbol_table).empty()) {
input_op = std::make_unique<EmptyResult>(std::move(input_op));
}
if (query_part.query_combinator) {
final_plan = MergeWithCombinator(std::move(input_op), std::move(final_plan), *query_part.query_combinator);
} else {
final_plan = std::move(input_op);
}
}
// Is this the only situation that should be covered
if (input_op->OutputSymbols(*context.symbol_table).empty()) {
input_op = std::make_unique<EmptyResult>(std::move(input_op));
if (query_parts.distinct) {
final_plan = MakeDistinct(std::move(final_plan));
}
return input_op;
return final_plan;
}
private:
@@ -251,6 +271,26 @@ class RuleBasedPlanner {
storage::EdgeTypeId GetEdgeType(EdgeTypeIx edge_type) { return context_->db->NameToEdgeType(edge_type.name); }
std::unique_ptr<LogicalOperator> HandleMatching(std::unique_ptr<LogicalOperator> last_op,
const SingleQueryPart &single_query_part, SymbolTable &symbol_table,
std::unordered_set<Symbol> &bound_symbols) {
MatchContext match_ctx{single_query_part.matching, symbol_table, bound_symbols};
last_op = PlanMatching(match_ctx, std::move(last_op));
for (const auto &matching : single_query_part.optional_matching) {
MatchContext opt_ctx{matching, symbol_table, bound_symbols};
std::vector<Symbol> bound_symbols(context_->bound_symbols.begin(), context_->bound_symbols.end());
auto once_with_symbols = std::make_unique<Once>(bound_symbols);
auto match_op = PlanMatching(opt_ctx, std::move(once_with_symbols));
if (match_op) {
last_op = std::make_unique<Optional>(std::move(last_op), std::move(match_op), opt_ctx.new_symbols);
}
}
return last_op;
}
std::unique_ptr<LogicalOperator> GenCreate(Create &create, std::unique_ptr<LogicalOperator> input_op,
const SymbolTable &symbol_table,
std::unordered_set<Symbol> &bound_symbols) {
@@ -394,119 +434,11 @@ class RuleBasedPlanner {
// Try to generate any filters even before the 1st match operator. This
// optimizes the optional match which filters only on symbols bound in
// regular match.
auto last_op = impl::GenFilters(std::move(input_op), bound_symbols, filters, storage);
for (const auto &expansion : matching.expansions) {
const auto &node1_symbol = symbol_table.at(*expansion.node1->identifier_);
if (bound_symbols.insert(node1_symbol).second) {
// We have just bound this symbol, so generate ScanAll which fills it.
last_op = std::make_unique<ScanAll>(std::move(last_op), node1_symbol, match_context.view);
match_context.new_symbols.emplace_back(node1_symbol);
last_op = impl::GenFilters(std::move(last_op), bound_symbols, filters, storage);
last_op = impl::GenNamedPaths(std::move(last_op), bound_symbols, named_paths);
last_op = impl::GenFilters(std::move(last_op), bound_symbols, filters, storage);
}
// We have an edge, so generate Expand.
if (expansion.edge) {
auto *edge = expansion.edge;
// If the expand symbols were already bound, then we need to indicate
// that they exist. The Expand will then check whether the pattern holds
// instead of writing the expansion to symbols.
const auto &node_symbol = symbol_table.at(*expansion.node2->identifier_);
auto existing_node = utils::Contains(bound_symbols, node_symbol);
const auto &edge_symbol = symbol_table.at(*edge->identifier_);
MG_ASSERT(!utils::Contains(bound_symbols, edge_symbol), "Existing edges are not supported");
std::vector<storage::EdgeTypeId> edge_types;
edge_types.reserve(edge->edge_types_.size());
for (const auto &type : edge->edge_types_) {
edge_types.push_back(GetEdgeType(type));
}
if (edge->IsVariable()) {
std::optional<ExpansionLambda> weight_lambda;
std::optional<Symbol> total_weight;
auto last_op = GenFilters(std::move(input_op), bound_symbols, filters, storage, symbol_table);
if (edge->type_ == EdgeAtom::Type::WEIGHTED_SHORTEST_PATH ||
edge->type_ == EdgeAtom::Type::ALL_SHORTEST_PATHS) {
weight_lambda.emplace(ExpansionLambda{symbol_table.at(*edge->weight_lambda_.inner_edge),
symbol_table.at(*edge->weight_lambda_.inner_node),
edge->weight_lambda_.expression});
last_op = HandleExpansion(std::move(last_op), matching, symbol_table, storage, bound_symbols,
match_context.new_symbols, named_paths, filters, match_context.view);
total_weight.emplace(symbol_table.at(*edge->total_weight_));
}
ExpansionLambda filter_lambda;
filter_lambda.inner_edge_symbol = symbol_table.at(*edge->filter_lambda_.inner_edge);
filter_lambda.inner_node_symbol = symbol_table.at(*edge->filter_lambda_.inner_node);
{
// Bind the inner edge and node symbols so they're available for
// inline filtering in ExpandVariable.
bool inner_edge_bound = bound_symbols.insert(filter_lambda.inner_edge_symbol).second;
bool inner_node_bound = bound_symbols.insert(filter_lambda.inner_node_symbol).second;
MG_ASSERT(inner_edge_bound && inner_node_bound, "An inner edge and node can't be bound from before");
}
// Join regular filters with lambda filter expression, so that they
// are done inline together. Semantic analysis should guarantee that
// lambda filtering uses bound symbols.
filter_lambda.expression = impl::BoolJoin<AndOperator>(
storage, impl::ExtractFilters(bound_symbols, filters, storage), edge->filter_lambda_.expression);
// At this point it's possible we have leftover filters for inline
// filtering (they use the inner symbols. If they were not collected,
// we have to remove them manually because no other filter-extraction
// will ever bind them again.
filters.erase(std::remove_if(
filters.begin(), filters.end(),
[e = filter_lambda.inner_edge_symbol, n = filter_lambda.inner_node_symbol](FilterInfo &fi) {
return utils::Contains(fi.used_symbols, e) || utils::Contains(fi.used_symbols, n);
}),
filters.end());
// Unbind the temporarily bound inner symbols for filtering.
bound_symbols.erase(filter_lambda.inner_edge_symbol);
bound_symbols.erase(filter_lambda.inner_node_symbol);
if (total_weight) {
bound_symbols.insert(*total_weight);
}
// TODO: Pass weight lambda.
MG_ASSERT(match_context.view == storage::View::OLD,
"ExpandVariable should only be planned with storage::View::OLD");
last_op = std::make_unique<ExpandVariable>(std::move(last_op), node1_symbol, node_symbol, edge_symbol,
edge->type_, expansion.direction, edge_types, expansion.is_flipped,
edge->lower_bound_, edge->upper_bound_, existing_node,
filter_lambda, weight_lambda, total_weight);
} else {
last_op = std::make_unique<Expand>(std::move(last_op), node1_symbol, node_symbol, edge_symbol,
expansion.direction, edge_types, existing_node, match_context.view);
}
// Bind the expanded edge and node.
bound_symbols.insert(edge_symbol);
match_context.new_symbols.emplace_back(edge_symbol);
if (bound_symbols.insert(node_symbol).second) {
match_context.new_symbols.emplace_back(node_symbol);
}
// Ensure Cyphermorphism (different edge symbols always map to
// different edges).
for (const auto &edge_symbols : matching.edge_symbols) {
if (edge_symbols.find(edge_symbol) == edge_symbols.end()) {
continue;
}
std::vector<Symbol> other_symbols;
for (const auto &symbol : edge_symbols) {
if (symbol == edge_symbol || bound_symbols.find(symbol) == bound_symbols.end()) {
continue;
}
other_symbols.push_back(symbol);
}
if (!other_symbols.empty()) {
last_op = std::make_unique<EdgeUniquenessFilter>(std::move(last_op), edge_symbol, other_symbols);
}
}
last_op = impl::GenFilters(std::move(last_op), bound_symbols, filters, storage);
last_op = impl::GenNamedPaths(std::move(last_op), bound_symbols, named_paths);
last_op = impl::GenFilters(std::move(last_op), bound_symbols, filters, storage);
}
}
MG_ASSERT(named_paths.empty(), "Expected to generate all named paths");
// We bound all named path symbols, so just add them to new_symbols.
for (const auto &named_path : matching.named_paths) {
@@ -543,6 +475,143 @@ class RuleBasedPlanner {
return std::make_unique<plan::Merge>(std::move(input_op), std::move(on_match), std::move(on_create));
}
std::unique_ptr<LogicalOperator> HandleExpansion(std::unique_ptr<LogicalOperator> last_op, const Matching &matching,
const SymbolTable &symbol_table, AstStorage &storage,
std::unordered_set<Symbol> &bound_symbols,
std::vector<Symbol> &new_symbols,
std::unordered_map<Symbol, std::vector<Symbol>> &named_paths,
Filters &filters, storage::View view) {
for (const auto &expansion : matching.expansions) {
const auto &node1_symbol = symbol_table.at(*expansion.node1->identifier_);
if (bound_symbols.insert(node1_symbol).second) {
// We have just bound this symbol, so generate ScanAll which fills it.
last_op = std::make_unique<ScanAll>(std::move(last_op), node1_symbol, view);
new_symbols.emplace_back(node1_symbol);
last_op = GenFilters(std::move(last_op), bound_symbols, filters, storage, symbol_table);
last_op = impl::GenNamedPaths(std::move(last_op), bound_symbols, named_paths);
last_op = GenFilters(std::move(last_op), bound_symbols, filters, storage, symbol_table);
}
if (expansion.edge) {
last_op = GenExpand(std::move(last_op), expansion, symbol_table, bound_symbols, matching, storage, filters,
named_paths, new_symbols, view);
}
}
return last_op;
}
std::unique_ptr<LogicalOperator> GenExpand(std::unique_ptr<LogicalOperator> last_op, const Expansion &expansion,
const SymbolTable &symbol_table, std::unordered_set<Symbol> &bound_symbols,
const Matching &matching, AstStorage &storage, Filters &filters,
std::unordered_map<Symbol, std::vector<Symbol>> &named_paths,
std::vector<Symbol> &new_symbols, storage::View view) {
// If the expand symbols were already bound, then we need to indicate
// that they exist. The Expand will then check whether the pattern holds
// instead of writing the expansion to symbols.
const auto &node1_symbol = symbol_table.at(*expansion.node1->identifier_);
bound_symbols.insert(node1_symbol);
const auto &node_symbol = symbol_table.at(*expansion.node2->identifier_);
auto *edge = expansion.edge;
auto existing_node = utils::Contains(bound_symbols, node_symbol);
const auto &edge_symbol = symbol_table.at(*edge->identifier_);
MG_ASSERT(!utils::Contains(bound_symbols, edge_symbol), "Existing edges are not supported");
std::vector<storage::EdgeTypeId> edge_types;
edge_types.reserve(edge->edge_types_.size());
for (const auto &type : edge->edge_types_) {
edge_types.push_back(GetEdgeType(type));
}
if (edge->IsVariable()) {
std::optional<ExpansionLambda> weight_lambda;
std::optional<Symbol> total_weight;
if (edge->type_ == EdgeAtom::Type::WEIGHTED_SHORTEST_PATH || edge->type_ == EdgeAtom::Type::ALL_SHORTEST_PATHS) {
weight_lambda.emplace(ExpansionLambda{symbol_table.at(*edge->weight_lambda_.inner_edge),
symbol_table.at(*edge->weight_lambda_.inner_node),
edge->weight_lambda_.expression});
total_weight.emplace(symbol_table.at(*edge->total_weight_));
}
ExpansionLambda filter_lambda;
filter_lambda.inner_edge_symbol = symbol_table.at(*edge->filter_lambda_.inner_edge);
filter_lambda.inner_node_symbol = symbol_table.at(*edge->filter_lambda_.inner_node);
{
// Bind the inner edge and node symbols so they're available for
// inline filtering in ExpandVariable.
bool inner_edge_bound = bound_symbols.insert(filter_lambda.inner_edge_symbol).second;
bool inner_node_bound = bound_symbols.insert(filter_lambda.inner_node_symbol).second;
MG_ASSERT(inner_edge_bound && inner_node_bound, "An inner edge and node can't be bound from before");
}
// Join regular filters with lambda filter expression, so that they
// are done inline together. Semantic analysis should guarantee that
// lambda filtering uses bound symbols.
filter_lambda.expression = impl::BoolJoin<AndOperator>(
storage, impl::ExtractFilters(bound_symbols, filters, storage), edge->filter_lambda_.expression);
// At this point it's possible we have leftover filters for inline
// filtering (they use the inner symbols. If they were not collected,
// we have to remove them manually because no other filter-extraction
// will ever bind them again.
filters.erase(
std::remove_if(filters.begin(), filters.end(),
[e = filter_lambda.inner_edge_symbol, n = filter_lambda.inner_node_symbol](FilterInfo &fi) {
return utils::Contains(fi.used_symbols, e) || utils::Contains(fi.used_symbols, n);
}),
filters.end());
// Unbind the temporarily bound inner symbols for filtering.
bound_symbols.erase(filter_lambda.inner_edge_symbol);
bound_symbols.erase(filter_lambda.inner_node_symbol);
if (total_weight) {
bound_symbols.insert(*total_weight);
}
// TODO: Pass weight lambda.
MG_ASSERT(view == storage::View::OLD, "ExpandVariable should only be planned with storage::View::OLD");
last_op = std::make_unique<ExpandVariable>(std::move(last_op), node1_symbol, node_symbol, edge_symbol,
edge->type_, expansion.direction, edge_types, expansion.is_flipped,
edge->lower_bound_, edge->upper_bound_, existing_node, filter_lambda,
weight_lambda, total_weight);
} else {
last_op = std::make_unique<Expand>(std::move(last_op), node1_symbol, node_symbol, edge_symbol,
expansion.direction, edge_types, existing_node, view);
}
// Bind the expanded edge and node.
bound_symbols.insert(edge_symbol);
new_symbols.emplace_back(edge_symbol);
if (bound_symbols.insert(node_symbol).second) {
new_symbols.emplace_back(node_symbol);
}
// Ensure Cyphermorphism (different edge symbols always map to
// different edges).
for (const auto &edge_symbols : matching.edge_symbols) {
if (edge_symbols.find(edge_symbol) == edge_symbols.end()) {
continue;
}
std::vector<Symbol> other_symbols;
for (const auto &symbol : edge_symbols) {
if (symbol == edge_symbol || bound_symbols.find(symbol) == bound_symbols.end()) {
continue;
}
other_symbols.push_back(symbol);
}
if (!other_symbols.empty()) {
last_op = std::make_unique<EdgeUniquenessFilter>(std::move(last_op), edge_symbol, other_symbols);
}
}
last_op = GenFilters(std::move(last_op), bound_symbols, filters, storage, symbol_table);
last_op = impl::GenNamedPaths(std::move(last_op), bound_symbols, named_paths);
last_op = GenFilters(std::move(last_op), bound_symbols, filters, storage, symbol_table);
return last_op;
}
std::unique_ptr<LogicalOperator> HandleForeachClause(query::Foreach *foreach,
std::unique_ptr<LogicalOperator> input_op,
const SymbolTable &symbol_table,
@@ -563,6 +632,108 @@ class RuleBasedPlanner {
return std::make_unique<plan::Foreach>(std::move(input_op), std::move(op), foreach->named_expression_->expression_,
symbol);
}
std::unique_ptr<LogicalOperator> HandleSubquery(std::unique_ptr<LogicalOperator> last_op,
std::shared_ptr<QueryParts> subquery, SymbolTable &symbol_table,
AstStorage &storage) {
std::unordered_set<Symbol> outer_scope_bound_symbols;
outer_scope_bound_symbols.insert(std::make_move_iterator(context_->bound_symbols.begin()),
std::make_move_iterator(context_->bound_symbols.end()));
context_->bound_symbols =
impl::GetSubqueryBoundSymbols(subquery->query_parts[0].single_query_parts, symbol_table, storage);
auto subquery_op = Plan(*subquery);
context_->bound_symbols.clear();
context_->bound_symbols.insert(std::make_move_iterator(outer_scope_bound_symbols.begin()),
std::make_move_iterator(outer_scope_bound_symbols.end()));
auto subquery_has_return = true;
if (subquery_op->GetTypeInfo() == EmptyResult::kType) {
subquery_has_return = false;
}
last_op = std::make_unique<Apply>(std::move(last_op), std::move(subquery_op), subquery_has_return);
if (context_->is_write_query) {
last_op = std::make_unique<Accumulate>(std::move(last_op), last_op->ModifiedSymbols(symbol_table), true);
}
return last_op;
}
std::unique_ptr<LogicalOperator> GenFilters(std::unique_ptr<LogicalOperator> last_op,
const std::unordered_set<Symbol> &bound_symbols, Filters &filters,
AstStorage &storage, const SymbolTable &symbol_table) {
auto pattern_filters = ExtractPatternFilters(filters, symbol_table, storage, bound_symbols);
auto *filter_expr = impl::ExtractFilters(bound_symbols, filters, storage);
if (filter_expr) {
last_op = std::make_unique<Filter>(std::move(last_op), std::move(pattern_filters), filter_expr);
}
return last_op;
}
std::unique_ptr<LogicalOperator> MakeExistsFilter(const FilterMatching &matching, const SymbolTable &symbol_table,
AstStorage &storage,
const std::unordered_set<Symbol> &bound_symbols) {
std::vector<Symbol> once_symbols(bound_symbols.begin(), bound_symbols.end());
std::unique_ptr<LogicalOperator> last_op = std::make_unique<Once>(once_symbols);
std::vector<Symbol> new_symbols;
std::unordered_set<Symbol> expand_symbols(bound_symbols.begin(), bound_symbols.end());
auto filters = matching.filters;
std::unordered_map<Symbol, std::vector<Symbol>> named_paths;
last_op = HandleExpansion(std::move(last_op), matching, symbol_table, storage, expand_symbols, new_symbols,
named_paths, filters, storage::View::OLD);
last_op = std::make_unique<Limit>(std::move(last_op), storage.Create<PrimitiveLiteral>(1));
last_op = std::make_unique<EvaluatePatternFilter>(std::move(last_op), matching.symbol.value());
return last_op;
}
std::vector<std::shared_ptr<LogicalOperator>> ExtractPatternFilters(Filters &filters, const SymbolTable &symbol_table,
AstStorage &storage,
const std::unordered_set<Symbol> &bound_symbols) {
std::vector<std::shared_ptr<LogicalOperator>> operators;
for (const auto &filter : filters) {
for (const auto &matching : filter.matchings) {
if (!impl::HasBoundFilterSymbols(bound_symbols, filter)) {
continue;
}
switch (matching.type) {
case PatternFilterType::EXISTS: {
operators.push_back(MakeExistsFilter(matching, symbol_table, storage, bound_symbols));
break;
}
}
}
}
return operators;
}
std::unique_ptr<LogicalOperator> MergeWithCombinator(std::unique_ptr<LogicalOperator> curr_op,
std::unique_ptr<LogicalOperator> last_op,
const Tree &combinator) {
if (const auto *union_ = utils::Downcast<const CypherUnion>(&combinator)) {
return impl::GenUnion(*union_, std::move(last_op), std::move(curr_op), *context_->symbol_table);
}
throw utils::NotYetImplemented("This type of merging queries is not yet implemented!");
}
std::unique_ptr<LogicalOperator> MakeDistinct(std::unique_ptr<LogicalOperator> last_op) {
return std::make_unique<Distinct>(std::move(last_op), last_op->OutputSymbols(*context_->symbol_table));
}
};
} // namespace memgraph::query::plan

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -155,6 +155,18 @@ auto ExpansionNodes(const std::vector<Expansion> &expansions, const SymbolTable
return nodes;
}
FilterMatching ToFilterMatching(Matching &matching) {
FilterMatching filter_matching;
filter_matching.expansions = matching.expansions;
filter_matching.edge_symbols = matching.edge_symbols;
filter_matching.filters = matching.filters;
filter_matching.node_symbol_to_expansions = matching.node_symbol_to_expansions;
filter_matching.named_paths = matching.named_paths;
filter_matching.expansion_symbols = matching.expansion_symbols;
return filter_matching;
}
} // namespace
VaryMatchingStart::VaryMatchingStart(Matching matching, const SymbolTable &symbol_table)
@@ -209,11 +221,31 @@ CartesianProduct<VaryMatchingStart> VaryMultiMatchingStarts(const std::vector<Ma
return MakeCartesianProduct(std::move(variants));
}
CartesianProduct<VaryMatchingStart> VaryFilterMatchingStarts(const Matching &matching,
const SymbolTable &symbol_table) {
auto filter_matchings_cnt = 0;
for (const auto &filter : matching.filters) {
filter_matchings_cnt += static_cast<int>(filter.matchings.size());
}
std::vector<VaryMatchingStart> variants;
variants.reserve(filter_matchings_cnt);
for (const auto &filter : matching.filters) {
for (const auto &filter_matching : filter.matchings) {
variants.emplace_back(filter_matching, symbol_table);
}
}
return MakeCartesianProduct(std::move(variants));
}
VaryQueryPartMatching::VaryQueryPartMatching(SingleQueryPart query_part, const SymbolTable &symbol_table)
: query_part_(std::move(query_part)),
matchings_(VaryMatchingStart(query_part_.matching, symbol_table)),
optional_matchings_(VaryMultiMatchingStarts(query_part_.optional_matching, symbol_table)),
merge_matchings_(VaryMultiMatchingStarts(query_part_.merge_matching, symbol_table)) {}
merge_matchings_(VaryMultiMatchingStarts(query_part_.merge_matching, symbol_table)),
filter_matchings_(VaryFilterMatchingStarts(query_part_.matching, symbol_table)) {}
VaryQueryPartMatching::iterator::iterator(const SingleQueryPart &query_part,
VaryMatchingStart::iterator matchings_begin,
@@ -221,7 +253,9 @@ VaryQueryPartMatching::iterator::iterator(const SingleQueryPart &query_part,
CartesianProduct<VaryMatchingStart>::iterator optional_begin,
CartesianProduct<VaryMatchingStart>::iterator optional_end,
CartesianProduct<VaryMatchingStart>::iterator merge_begin,
CartesianProduct<VaryMatchingStart>::iterator merge_end)
CartesianProduct<VaryMatchingStart>::iterator merge_end,
CartesianProduct<VaryMatchingStart>::iterator filter_begin,
CartesianProduct<VaryMatchingStart>::iterator filter_end)
: current_query_part_(query_part),
matchings_it_(matchings_begin),
matchings_end_(matchings_end),
@@ -230,7 +264,10 @@ VaryQueryPartMatching::iterator::iterator(const SingleQueryPart &query_part,
optional_end_(optional_end),
merge_it_(merge_begin),
merge_begin_(merge_begin),
merge_end_(merge_end) {
merge_end_(merge_end),
filter_it_(filter_begin),
filter_begin_(filter_begin),
filter_end_(filter_end) {
if (matchings_it_ != matchings_end_) {
// Fill the query part with the first variation of matchings
SetCurrentQueryPart();
@@ -242,26 +279,37 @@ VaryQueryPartMatching::iterator &VaryQueryPartMatching::iterator::operator++() {
// * matchings (m1) and (m2)
// * optional matchings (o1) and (o2)
// * merge matching (g1)
// * filter matching (f1) and (f2)
// We want to produce parts for:
// * (m1), (o1), (g1)
// * (m1), (o2), (g1)
// * (m2), (o1), (g1)
// * (m2), (o2), (g1)
// Create variations by changing the merge part first.
if (merge_it_ != merge_end_) ++merge_it_;
// If all merge variations are done, start them from beginning and move to the
// next optional matching variation.
if (merge_it_ == merge_end_) {
// * (m1), (o1), (g1), (f1)
// * (m1), (o1), (g1), (f2)
// * (m1), (o2), (g1), (f1)
// * (m1), (o2), (g1), (f2)
// * (m2), (o1), (g1), (f1)
// * (m2), (o1), (g1), (f2)
// * (m2), (o2), (g1), (f1)
// * (m2), (o2), (g1), (f2)
// Create variations by changing the filter part first.
if (filter_it_ != filter_end_) ++filter_it_;
// Create variations by changing the merge part.
if (filter_it_ == filter_end_) {
filter_it_ = filter_begin_;
if (merge_it_ != merge_end_) ++merge_it_;
}
// Create variations by changing the optional part.
if (merge_it_ == merge_end_ && filter_it_ == filter_begin_) {
merge_it_ = merge_begin_;
if (optional_it_ != optional_end_) ++optional_it_;
}
// If all optional matching variations are done (after exhausting merge
// variations), start them from beginning and move to the next regular
// matching variation.
if (optional_it_ == optional_end_ && merge_it_ == merge_begin_) {
if (optional_it_ == optional_end_ && merge_it_ == merge_begin_ && filter_it_ == filter_begin_) {
optional_it_ = optional_begin_;
if (matchings_it_ != matchings_end_) ++matchings_it_;
}
// We have reached the end, so return;
if (matchings_it_ == matchings_end_) return *this;
// Fill the query part with the new variation of matchings.
@@ -283,6 +331,28 @@ void VaryQueryPartMatching::iterator::SetCurrentQueryPart() {
if (merge_it_ != merge_end_) {
current_query_part_.merge_matching = *merge_it_;
}
DMG_ASSERT(filter_it_ != filter_end_ || filter_begin_ == filter_end_,
"Either there are no filter matchings or we can always generate"
"a variation");
auto all_filter_matchings = *filter_it_;
auto all_filter_matchings_idx = 0;
for (auto &filter : current_query_part_.matching.filters) {
auto matchings_size = filter.matchings.size();
std::vector<FilterMatching> new_matchings;
new_matchings.reserve(matchings_size);
for (auto i = 0; i < matchings_size; i++) {
new_matchings.push_back(ToFilterMatching(all_filter_matchings[all_filter_matchings_idx]));
new_matchings[i].symbol = filter.matchings[i].symbol;
new_matchings[i].type = filter.matchings[i].type;
all_filter_matchings_idx++;
}
filter.matchings = std::move(new_matchings);
}
}
bool VaryQueryPartMatching::iterator::operator==(const iterator &other) const {
@@ -291,7 +361,8 @@ bool VaryQueryPartMatching::iterator::operator==(const iterator &other) const {
// iterators can be at any position.
return true;
}
return matchings_it_ == other.matchings_it_ && optional_it_ == other.optional_it_ && merge_it_ == other.merge_it_;
return matchings_it_ == other.matchings_it_ && optional_it_ == other.optional_it_ && merge_it_ == other.merge_it_ &&
filter_it_ == other.filter_it_;
}
} // namespace memgraph::query::plan::impl

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -230,6 +230,8 @@ class VaryMatchingStart {
// Cartesian product of all of them is returned.
CartesianProduct<VaryMatchingStart> VaryMultiMatchingStarts(const std::vector<Matching> &, const SymbolTable &);
CartesianProduct<VaryMatchingStart> VaryFilterMatchingStarts(const Matching &matching, const SymbolTable &symbol_table);
// Produces alternative query parts out of a single part by varying how each
// graph matching is done.
class VaryQueryPartMatching {
@@ -245,6 +247,7 @@ class VaryQueryPartMatching {
typedef const SingleQueryPart *pointer;
iterator(const SingleQueryPart &, VaryMatchingStart::iterator, VaryMatchingStart::iterator,
CartesianProduct<VaryMatchingStart>::iterator, CartesianProduct<VaryMatchingStart>::iterator,
CartesianProduct<VaryMatchingStart>::iterator, CartesianProduct<VaryMatchingStart>::iterator,
CartesianProduct<VaryMatchingStart>::iterator, CartesianProduct<VaryMatchingStart>::iterator);
@@ -266,15 +269,20 @@ class VaryQueryPartMatching {
CartesianProduct<VaryMatchingStart>::iterator merge_it_;
CartesianProduct<VaryMatchingStart>::iterator merge_begin_;
CartesianProduct<VaryMatchingStart>::iterator merge_end_;
CartesianProduct<VaryMatchingStart>::iterator filter_it_;
CartesianProduct<VaryMatchingStart>::iterator filter_begin_;
CartesianProduct<VaryMatchingStart>::iterator filter_end_;
};
auto begin() {
return iterator(query_part_, matchings_.begin(), matchings_.end(), optional_matchings_.begin(),
optional_matchings_.end(), merge_matchings_.begin(), merge_matchings_.end());
optional_matchings_.end(), merge_matchings_.begin(), merge_matchings_.end(),
filter_matchings_.begin(), filter_matchings_.end());
}
auto end() {
return iterator(query_part_, matchings_.end(), matchings_.end(), optional_matchings_.end(),
optional_matchings_.end(), merge_matchings_.end(), merge_matchings_.end());
optional_matchings_.end(), merge_matchings_.end(), merge_matchings_.end(), filter_matchings_.end(),
filter_matchings_.end());
}
private:
@@ -286,6 +294,7 @@ class VaryQueryPartMatching {
CartesianProduct<VaryMatchingStart> optional_matchings_;
// Like optional matching, but for merge matchings.
CartesianProduct<VaryMatchingStart> merge_matchings_;
CartesianProduct<VaryMatchingStart> filter_matchings_;
};
} // namespace impl
@@ -304,25 +313,70 @@ class VariableStartPlanner {
// Generates different, equivalent query parts by taking different graph
// matching routes for each query part.
auto VaryQueryMatching(const std::vector<SingleQueryPart> &query_parts, const SymbolTable &symbol_table) {
std::vector<impl::VaryQueryPartMatching> alternative_query_parts;
alternative_query_parts.reserve(query_parts.size());
for (const auto &query_part : query_parts) {
alternative_query_parts.emplace_back(impl::VaryQueryPartMatching(query_part, symbol_table));
auto VaryQueryMatching(const QueryParts &query_parts, const SymbolTable &symbol_table) {
std::vector<impl::VaryQueryPartMatching> varying_query_matchings;
auto single_query_parts = ExtractSingleQueryParts(std::make_unique<QueryParts>(query_parts));
for (const auto &single_query_part : single_query_parts) {
varying_query_matchings.emplace_back(single_query_part, symbol_table);
}
return iter::slice(MakeCartesianProduct(std::move(alternative_query_parts)), 0UL, FLAGS_query_max_plans);
return iter::slice(MakeCartesianProduct(std::move(varying_query_matchings)), 0UL, FLAGS_query_max_plans);
}
std::vector<SingleQueryPart> ExtractSingleQueryParts(const std::shared_ptr<QueryParts> query_parts) {
std::vector<SingleQueryPart> results;
for (const auto &query_part : query_parts->query_parts) {
for (const auto &single_query_part : query_part.single_query_parts) {
results.push_back(single_query_part);
for (const auto &subquery : single_query_part.subqueries) {
const auto subquery_results = ExtractSingleQueryParts(subquery);
results.insert(results.end(), std::make_move_iterator(subquery_results.begin()),
std::make_move_iterator(subquery_results.end()));
}
}
}
return results;
}
QueryParts ReconstructQueryParts(const QueryParts &old_query_parts,
const std::vector<SingleQueryPart> &single_query_parts_variation, uint64_t &index) {
auto reconstructed_query_parts = old_query_parts;
for (auto i = 0; i < old_query_parts.query_parts.size(); i++) {
const auto &old_query_part = old_query_parts.query_parts[i];
for (auto j = 0; j < old_query_part.single_query_parts.size(); j++) {
const auto &old_single_query_part = old_query_part.single_query_parts[j];
reconstructed_query_parts.query_parts[i].single_query_parts[j] = single_query_parts_variation[index++];
for (auto k = 0; k < old_single_query_part.subqueries.size(); k++) {
const auto &subquery = old_single_query_part.subqueries[k];
reconstructed_query_parts.query_parts[i].single_query_parts[j].subqueries[k] =
std::make_shared<QueryParts>(ReconstructQueryParts(*subquery, single_query_parts_variation, index));
}
}
}
return reconstructed_query_parts;
}
public:
explicit VariableStartPlanner(TPlanningContext *context) : context_(context) {}
/// @brief Generate multiple plans by varying the order of graph traversal.
auto Plan(const std::vector<SingleQueryPart> &query_parts) {
auto Plan(const QueryParts &query_parts) {
return iter::imap(
[context = context_](const auto &alternative_query_parts) {
[context = context_, old_query_parts = query_parts, this](const auto &alternative_query_parts) {
uint64_t index = 0;
auto reconstructed_query_parts = ReconstructQueryParts(old_query_parts, alternative_query_parts, index);
RuleBasedPlanner<TPlanningContext> rule_planner(context);
context->bound_symbols.clear();
return rule_planner.Plan(alternative_query_parts);
return rule_planner.Plan(reconstructed_query_parts);
},
VaryQueryMatching(query_parts, *context_->symbol_table));
}
@@ -330,7 +384,7 @@ class VariableStartPlanner {
/// @brief The result of plan generation is an iterable of roots to multiple
/// generated operator trees.
using PlanResult = typename std::result_of<decltype (&VariableStartPlanner<TPlanningContext>::Plan)(
VariableStartPlanner<TPlanningContext>, std::vector<SingleQueryPart> &)>::type;
VariableStartPlanner<TPlanningContext>, QueryParts &)>::type;
};
} // namespace memgraph::query::plan

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -78,6 +78,11 @@ class VertexCountCache {
return db_->LabelPropertyIndexExists(label, property);
}
std::optional<storage::IndexStats> GetIndexStats(const storage::LabelId &label,
const storage::PropertyId &property) const {
return db_->GetIndexStats(label, property);
}
private:
typedef std::pair<storage::LabelId, storage::PropertyId> LabelPropertyKey;

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -146,10 +146,10 @@ struct mgp_date {
mgp_date(const memgraph::utils::Date &date, memgraph::utils::MemoryResource *memory) noexcept
: memory(memory), date(date) {}
mgp_date(const std::string_view string, memgraph::utils::MemoryResource *memory) noexcept
mgp_date(const std::string_view string, memgraph::utils::MemoryResource *memory)
: memory(memory), date(memgraph::utils::ParseDateParameters(string).first) {}
mgp_date(const mgp_date_parameters *parameters, memgraph::utils::MemoryResource *memory) noexcept
mgp_date(const mgp_date_parameters *parameters, memgraph::utils::MemoryResource *memory)
: memory(memory), date(MapDateParameters(parameters)) {}
mgp_date(const int64_t microseconds, memgraph::utils::MemoryResource *memory) noexcept
@@ -194,10 +194,10 @@ struct mgp_local_time {
// have everything noexcept here.
static_assert(std::is_nothrow_copy_constructible_v<memgraph::utils::LocalTime>);
mgp_local_time(const std::string_view string, memgraph::utils::MemoryResource *memory) noexcept
mgp_local_time(const std::string_view string, memgraph::utils::MemoryResource *memory)
: memory(memory), local_time(memgraph::utils::ParseLocalTimeParameters(string).first) {}
mgp_local_time(const mgp_local_time_parameters *parameters, memgraph::utils::MemoryResource *memory) noexcept
mgp_local_time(const mgp_local_time_parameters *parameters, memgraph::utils::MemoryResource *memory)
: memory(memory), local_time(MapLocalTimeParameters(parameters)) {}
mgp_local_time(const memgraph::utils::LocalTime &local_time, memgraph::utils::MemoryResource *memory) noexcept
@@ -250,8 +250,7 @@ struct mgp_local_date_time {
mgp_local_date_time(const std::string_view string, memgraph::utils::MemoryResource *memory) noexcept
: memory(memory), local_date_time(CreateLocalDateTimeFromString(string)) {}
mgp_local_date_time(const mgp_local_date_time_parameters *parameters,
memgraph::utils::MemoryResource *memory) noexcept
mgp_local_date_time(const mgp_local_date_time_parameters *parameters, memgraph::utils::MemoryResource *memory)
: memory(memory),
local_date_time(MapDateParameters(parameters->date_parameters),
MapLocalTimeParameters(parameters->local_time_parameters)) {}
@@ -301,7 +300,7 @@ struct mgp_duration {
// have everything noexcept here.
static_assert(std::is_nothrow_copy_constructible_v<memgraph::utils::Duration>);
mgp_duration(const std::string_view string, memgraph::utils::MemoryResource *memory) noexcept
mgp_duration(const std::string_view string, memgraph::utils::MemoryResource *memory)
: memory(memory), duration(memgraph::utils::ParseDurationParameters(string)) {}
mgp_duration(const mgp_duration_parameters *parameters, memgraph::utils::MemoryResource *memory) noexcept

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -35,7 +35,7 @@ namespace memgraph::query::procedure {
constexpr const char *func_code =
"import ast\n\n"
"no_removals = ['collections', 'abc', 'sys']\n"
"no_removals = ['collections', 'abc', 'sys', 'torch', 'torch_geometric', 'igraph']\n"
"modules = set()\n\n"
"def visit_Import(node):\n"
" for name in node.names:\n"

View File

@@ -1,4 +1,4 @@
// Copyright 2022 Memgraph Ltd.
// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -490,6 +490,11 @@ Streams::StreamsMap::iterator Streams::CreateConsumer(StreamsMap &map, const std
retry_interval = interpreter_context_->config.stream_transaction_retry_interval](
const std::vector<typename TStream::Message> &messages) mutable {
auto accessor = interpreter_context->db->Access();
// register new interpreter into interpreter_context_
interpreter_context->interpreters->insert(interpreter.get());
utils::OnScopeExit interpreter_cleanup{
[interpreter_context, interpreter]() { interpreter_context->interpreters->erase(interpreter.get()); }};
EventCounter::IncrementCounter(EventCounter::MessagesConsumed, messages.size());
CallCustomTransformation(transformation_name, messages, result, accessor, *memory_resource, stream_name);

Some files were not shown because too many files have changed in this diff Show More