Commit Graph

279 Commits

Author SHA1 Message Date
antoniofilipovic
ca20b78479 split between try register and register replica 2024-02-13 11:22:41 +01:00
Andi
4a7c7f0898 Distributed coordinators (#1693) 2024-02-13 08:49:28 +00:00
Andi
cf80687d1d HA: Organize Raft coordinator group (#1687) 2024-02-08 09:11:33 +00:00
Antonio Filipovic
c15b62a88d HA: Disable replication from old main (#1674) 2024-02-07 11:20:47 +01:00
andrejtonev
7ead00f23e Adding authentication data replication (#1666)
* Add AUTH system tx deltas
* Add auth data RPC and handlers
* Support multiple system deltas in a single transaction
* Added e2e test
* Bugfix: KVStore segfault after move

---------

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2024-02-05 10:37:00 +00:00
Andi
cb7b88ad92 HA: Support restart of instances (#1672) 2024-02-01 11:55:48 +01:00
Andi
78a88737f8 HA: Add automatic failover (#1646)
Co-authored-by: antoniofilipovic <filipovicantonio1998@gmail.com>
2024-01-29 15:34:00 +01:00
Andi
38ade99652 HA: Add coordinator to replication cluster (#1608) 2024-01-24 13:07:51 +01:00
andrejtonev
071df2f439 Replication refactor part 7 (#1550)
* Split queries into system and data queries
* System queries are sequentially executed and generate separate transaction deltas
* System transaction try locks for 100ms
* last_commited_system_ts saved to DBMS durability
* Replicating CREATE/DROP DATABASE
* Sending a system snapshot if REPLICA behind
* Passing a copy of the gatekeeper::access as std::any to all functions that could call an async execution
* Removed delete_on_drop flag (we now always delete on drop)
* Using UUID as the directory name for databases
* DBMS durability update (added versioning and salient information)
* Automatic migration from previous version
* Interpreter can run some queries without a target database
* SHOW REPLICA returns the status of the currently active DB
* Returning UUID instead of db name in the RPC responses
* Using UUIDs for database specification in RPC (not name)
* FrequentCheck forces update on reconnect
* TimestampRpc will detect if a replica is behind, and will update client's state
* Safer SLK reads
* Split SHOW DATABASES in two SHOW DATABASES (list of current databases) and SHOW DATABASE a single string naming the current database

---------

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2024-01-23 12:06:10 +01:00
DavIvek
4bb9238679 Add support for triggers in database dump (#1610) 2024-01-09 13:05:54 +01:00
Gareth Andrew Lloyd
0fb8e4116f Fix REPLICA timestamps (#1615)
* Fix up REPLICA GetInfo and CreateSnapshot

Subtle bug where these actions were using the incorrect transactional
access while in REPLICA role. This casued timestamp to be incorrectly
bumped, breaking REPLICA from doing replication.

* Delay DNS resolution

Rather than resolve at endpoint creation, we will instread resolve only
on Socket connect. This allows k8s deployments to change their IP during
pod restarts.

* Minor sonarsource fixes

---------
Co-authored-by: Andreja <andreja.tonev@memgraph.io>
Co-authored-by: DavIvek <david.ivekovic@memgraph.io>
2024-01-05 16:42:54 +00:00
gvolfing
ac0c4193b0 Remove comment 2023-11-28 11:46:32 +01:00
gvolfing
9868dee73b Merge branch 'master' into Implement-constant-time-label-and-edge-type-retrieval 2023-11-28 10:21:59 +01:00
Andi
7f5a55f1b2 Fix restarts when using init-file flag (#1465) 2023-11-24 13:11:47 +01:00
gvolfing
08acde3973 Merge branch 'master' into Implement-constant-time-label-and-edge-type-retrieval 2023-11-24 12:24:20 +01:00
andrejtonev
8b9e1fa08b Replication refactor part 6 (#1484)
Single (instance level) connection to a replica (messages from all databases get multiplexed through it)
ReplicationClient split in two: ReplicationClient and ReplicationStorageClient
New ReplicationClient, moved under replication, handles the raw connection, owned by MainRoleData
ReplicationStorageClient handles the storage <-> replica state machine and holds to a stream
Removed epoch and storage from *Clients
rpc::Stream proactively aborts on error and sets itself to a defunct state
Removed HandleRpcFailure, instead we simply log the error and let the FrequentCheck handle re-connection
replica_state is now a synced variable
ReplicaStorageClient state machine bugfixes
Single FrequentCheck that goes through DBMS
Moved ReplicationState under DbmsHandler
Moved some replication startup logic under the DbmsHandler's constructor
Removed InMemoryReplicationClient
CreateReplicationClient has been removed from Storage
Simplified GetRecoverySteps and made safer

---------

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2023-11-23 11:02:35 +01:00
Gareth Andrew Lloyd
e4f94c15c6 Fixes for clang-tidy / sonar issues (#1536) 2023-11-22 13:05:02 +00:00
Andi
1d90b60f56 Add schema.assert (#1485) 2023-11-21 09:19:50 +01:00
gvolfing
a370b09d12 Merge branch 'master' into Implement-constant-time-label-and-edge-type-retrieval 2023-11-14 13:13:16 +01:00
gvolfing
1527bdf435 Make metadata collection setable with flag
There might be a performance impect of updating the metadata store on
bulk operations. Hence this flag which is disabling the collection by
default. If the queries to obtain the information are called with this
flag disabled, the database will throw an exception.
2023-11-14 13:10:08 +01:00
Josipmrden
e907817854 Fix for in list segmentation fault (#1494) 2023-11-13 05:17:10 +01:00
gvolfing
2946d74fdd Merge branch 'master' into Implement-constant-time-label-and-edge-type-retrieval 2023-11-08 07:48:15 +01:00
DavIvek
ece4b0dba8 Fix cached plan not getting invalidated (#1348) 2023-11-07 13:34:03 +01:00
gvolfing
eb4ebab438 Merge branch 'master' into Implement-constant-time-label-and-edge-type-retrieval 2023-11-07 12:29:22 +01:00
gvolfing
260d60451d Modify retrieval function signatures
Before the functions that are retrieving the data from the metadata
holding datastructures were returning a std::string, and that was
propagated outward all the way through. To keep this functions
consistent with the rest of the storage/dbaccessor functions the LabelId
and EdgeTypeId will be propagated respectively and the conversion into
string will only happen at the interpreter level.
2023-11-07 12:07:52 +01:00
Andi
66487a6dce Durability improvements (#1385) 2023-11-07 11:37:54 +01:00
gvolfing
c4d9116c9c Add queries to obtain the labels and edge types
Add two queries to be able to retrieve the labels and edge types this is
done through additions to the DatabaseInfoQuery query types.
2023-11-07 09:35:28 +01:00
Antonio Filipovic
1ab7f6ac78 Add notification for user on max map count (#1408) 2023-11-06 15:44:26 +01:00
andrejtonev
dbc6054689 Replication refactor (part 5) (#1378) 2023-11-06 11:50:49 +00:00
Antonio Filipovic
48631d1e37 Rename memory usage and memory allocated (#1426) 2023-11-03 14:40:45 +01:00
Antonio Filipovic
93e6d058d2 Remove all_shortest paths unnecessary logs (#1425) 2023-11-03 12:46:06 +01:00
Andi
5e6c5618f5 Make FrameChangeCollector owning the memory resource (#1398) 2023-11-02 09:54:39 +01:00
gvolfing
c296dc67ce Add index count to index info (#1229) 2023-10-27 18:13:05 +02:00
Antonio Filipovic
a84f570c6d Use extent hooks for per query memory limit (#1340) 2023-10-25 16:01:59 +02:00
Antonio Filipovic
2426d7980d Add OOM enabler in operator tree (#1379) 2023-10-25 12:16:11 +02:00
Josipmrden
e617ff9b59 Provide textual information for inefficient plans with notifications (#1343) 2023-10-24 22:20:05 +02:00
DavIvek
98680b04c9 Add DNS support for cluster replica address (#1323) 2023-10-24 13:11:36 +02:00
andrejtonev
26e31ca06f Fix SHOW CONFIG to show the run-time flag status (#1278) 2023-10-23 10:18:07 +02:00
Andi
af56ab6ea8 Forbid changing isolation level for disk and analytical (#1367)
Co-authored-by: Marko Budiselić <marko.budiselic@memgraph.com>
2023-10-23 06:02:56 +02:00
andrejtonev
22d8ef75e0 Updated telemetry client-side (#1337) 2023-10-16 14:16:00 +02:00
Andi
7b0bafa21e Add human readable memory allocations in show storage info (#1366) 2023-10-16 11:35:44 +02:00
Gareth Andrew Lloyd
d278a33f31 Decouple pure replication state from storage [part 1] (#1325)
A major refactor to decouple replication state from storage.
ATM it is still owned by storage but a following part should fix that.
2023-10-10 11:44:19 +01:00
Gareth Andrew Lloyd
3cc2bc2791 Refactor interpreter to support multiple distributed clocks (Part 1) (#1281)
* Interpreter transaction ID decoupled from storage transaction ID
* Transactional scope for indices, statistics and constraints
* Storage::Accessor now has 2 modes (unique and shared)
* Introduced ResourceLock to fix pthread mutex problems
* Split InfoQuery in two: non-transactional SystemInfoQuery and transactional DatabaseInfoQuery
* Replicable and durable statistics
* Bumped WAL/Snapshot versions
* Initial implementation of the Lamport clock

---------

Co-authored-by: Andreja Tonev <andreja.tonev@memgraph.io>
2023-10-05 16:58:39 +02:00
Gareth Andrew Lloyd
d71b6a5007 Refactor replication client/server (#1311) 2023-09-29 11:21:42 +01:00
andrejtonev
bce48361ca Decoupling Interpreter from Storage (#1186)
Unique/global InterpreterContext that is Storage agnostic (has a reference to the DbmsHandler instead)

* InterpreterContext is no longer the owner of Storage
* New Database structure that handles Storage, Triggers, Streams
* Renamed SessinContextHandler to DbmsHandler and simplified the multi-tenant logic
* Added Gatekeeper and updated handlers to use it

---------

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2023-09-20 13:13:54 +02:00
andrejtonev
5e5f4ffc5d Add more runtime configurable settings (#1183)
server name, query timeout settings, log.level, log.to_stderr
2023-09-11 11:30:54 -04:00
Ante Pušić
d4fcd745d2 Sort SHOW INDEX INFO (#1178) 2023-09-11 10:59:41 +02:00
Andi
b5413c6f82 Add edge import mode into the on-disk storage (#1157) 2023-09-05 19:00:53 +02:00
Gareth Andrew Lloyd
e928eed028 Replication refactor (part 4) (#1211)
More refactoring to isolate generic replication behavior. Making the 
InMemory* types even more decoupled from replication logic.
2023-08-31 16:06:44 +01:00
andrejtonev
c526ff2a8f [master < ] Remove DbAccessor from non-transactional queries (#1201)
* Decouple non-transactional queries from DbAccessor
* Invalidate auth cache after AuthQuery

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2023-08-29 11:13:42 +02:00