In previous post we Setup Multi node Cassandra Cluster on Google Compute Engine. In this post we analyse Cassandra startup logs and visualise how handshakes happens when new node joins cluster. Later we will see how data is persisted in file system after flushing, after deleting record we will look into how tombstone information is stored and how deleted record is maintained in Cassandra.
Observation: From above log we can see Messaging service is started on /10.128.0.2:7000 (eth0) and IP address of node is /10.128.0.2 and after starting its state is Normal (last highlighted line of log)
Since it is very first node started in data centre so only topology of this node is updated.
Start Cassandra on Instance-2 and capture log: IP address of Instance-2 is 10.128.0.3
Observation: From above log we can see Messaging service is started on /10.128.0.3:7000 (eth0) and IP address of node is /10.128.0.3 and after starting its state is Normal .
Since Instance-1 is already running when Cassandra starts on this (10.128.0.3) node, it does handshake with already running node (10.128.0.2) and topology of both Instance-1 (10.128.0.2) and Instance-2 (10.128.0.3) is updated.
Start Cassandra on Instance-3 and capture log: IP address of Instance-3 is 10.128.0.4
Observation: From above log we can see Messaging service is started on /10.128.0.4:7000 (eth0) and IP address of node is /10.128.0.4 and after starting its state is Normal .
Since Instance-1 & Instance-2 is already running when Cassandra starts on this(10.128.0.4) node, it does handshakes with already running node (10.128.0.2 and 10.128.0.3) and topology of all three Instance-1 (10.128.0.2), Instance-2 (10.128.0.3) and Instance-3 (10.128.0.4) is updated.
We have highlighted token range for all running nodes in green. When a new node joins Virtually tokens are assigned.
Start Cassandra on Instance-4 and capture log: IP address of Instance-4 is 10.128.0.5
Observation: From above log we can see IP address of node is /10.128.0.4 and after starting its state is Normal .
Since Instance-1, Instance-2 & Instance-3 is already running when Cassandra starts on this(10.128.0.5) node, it does handshakes with already running node (10.128.0.2 , 10.128.0.3 and 10.128.0.4) and topology of all four Instance-1 (10.128.0.2), Instance-2 (10.128.0.3), Instance-3 (10.128.0.4) and Instance-3 (10.128.0.5) is updated.
All four instance of Cassandra is up and running. Run below command and validate data centre(DC) configuration : two node in rack1 and 2 node in rack2.
4 instance of Cassandra is up and running([rack-1: I1 & I2] [rack-2 : I3 & I4] )
Keyspace and Table(column family) creation : Run cqlsh utility on one of the terminal .
Create Keyspace with replication factor 3: We have four instance of Cassandra running so we can have 3 copy of data on different instances in given DC.
On keyspace creation success prompt does not gives any success message, describe keyspaces and see it has been created.
Select and Use a keyspace : Since column families (like table in RDMS) are created in context of keyspace so select keyspace we want to work with.
Create Table(Column Family) in selected Keyspace:
Insert data in user Table(Column Family) and select table to display data:
Replicated of data on Cassandra nodes: Which node stores data for username: zytham ?
Below command shows one copy is stored on instance-1(where we are running query) and 2 copy is stored on Instance-3 and Instance-4 (rack2)
Run for some other user, it is not necessary that one copy is always stored on instance where query ie being run.
Data directory and its storage : Default data storage directory is /opt/apache-cassandra-3.11.3/data/
We have OCProc keyspace created as directory. Run below command and find persisted data.
Where is our data, no data is persisted yet :
Answer: memtable has all our user data, it not yet persisted in disk (sstable)
How does data comes from memtable to disk ?
Answer: When memtable flush data then it is stored on disk. Let use nodetool to forceful flush data and see what changes it brings to directory "user-f82692c0bcff11e8a9c80961902fe681".
On flush memtable flushes data to disk(Intermediate stage is sstable). File "mc-1-big-Data.db" contains our data and along with flush also creates files for index and filter.
View User data in "mc-1-big-Data.db" : Using sstabledump on instance-1 we can visualise user data in file "mc-1-big-Data.db".
Instance-1 stores only one record with username = zytham. We can see that each cell is stored with timestamp and column value.
Where are other user records ? : Apply node flush on Instance-3 and display data stored in file first generation data file "mc-1-big-Data.db". Below we can see one copy of each user record is stored on Instance-3.
Display JOSN form of data visualisation: Execute above command without -d switch.
Delete record from user table: delete row form user where username = 'mishra' and city='Torento';
See data file again, no change. Where did chnage happend till now ? :
Answer: memtable.
Flush using nodetool : New generation files are persisted along with old one, old files are not deleted.
When old (old generation) files be deleted ? :
Answer: During comapction
Apply compaction: Delete old generation file and merged old & new generation files.
Deleted and compacted: Now visualise merged data file. It has not been removed, it has been marked as deleted (Why?) : If some failed node joins cluster after recovery and has old data, then this marked deleted help to update that node and remove stalled data which has been deleted already.
Here is a nice from the last pikcle post which explains in details - Deletes and Tombstones in Cassandra
Cassandra node running status and Handshake visualisation
On starting Cassandra on any node we are interested in three things.- Messaging service starting on which node (what is Ip address and interface) and what is Ip address of node where Cassandra started and what's final status (Highlighted in pink)
- Which (starting) node does handshaking with which (already running) node (Highlighted in blue)
- Which all topology is updated on starting Cassandra on given node (Highlighted in Yellow)
INFO [main] 2018-09-20 16:15:57,812 StorageService.java:618 - Cassandra version: 3.11.3 INFO [main] 2018-09-20 16:15:57,813 StorageService.java:619 - Thrift API version: 20.1.0 INFO [main] 2018-09-20 16:15:57,813 StorageService.java:620 - CQL supported versions: 3.4.4 (default: 3.4.4) INFO [main] 2018-09-20 16:15:57,813 StorageService.java:622 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4) INFO [main] 2018-09-20 16:15:57,893 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 50 MB and a resize interval of 60 minutes INFO [main] 2018-09-20 16:15:57,914 MessagingService.java:761 - Starting Messaging Service on /10.128.0.2:7000 (eth0) INFO [main] 2018-09-20 16:16:03,012 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED INFO [main] 2018-09-20 16:16:29,114 StorageService.java:704 - Loading persisted ring state INFO [main] 2018-09-20 16:16:29,137 StorageService.java:822 - Starting up server gossip INFO [main] 2018-09-20 16:16:29,250 TokenMetadata.java:479 - Updating topology for /10.128.0.2 INFO [main] 2018-09-20 16:16:29,251 TokenMetadata.java:479 - Updating topology for /10.128.0.2 INFO [main] 2018-09-20 16:16:29,370 StorageService.java:1446 - JOINING: Finish joining ring INFO [main] 2018-09-20 16:16:29,426 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='keyspace1', ColumnFamily='standard1') INFO [main] 2018-09-20 16:16:29,432 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='keyspace1', ColumnFamily='counter1') INFO [main] 2018-09-20 16:16:29,439 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='stockdb', ColumnFamily='user') INFO [main] 2018-09-20 16:16:29,519 StorageService.java:2289 - Node /10.128.0.2 state jump to NORMAL
Observation: From above log we can see Messaging service is started on /10.128.0.2:7000 (eth0) and IP address of node is /10.128.0.2 and after starting its state is Normal (last highlighted line of log)
Since it is very first node started in data centre so only topology of this node is updated.
Start Cassandra on Instance-2 and capture log: IP address of Instance-2 is 10.128.0.3
INFO [main] 2018-09-20 16:18:26,317 QueryProcessor.java:163 - Preloaded 1 prepared statements INFO [main] 2018-09-20 16:18:26,318 StorageService.java:618 - Cassandra version: 3.11.3 INFO [main] 2018-09-20 16:18:26,318 StorageService.java:619 - Thrift API version: 20.1.0 INFO [main] 2018-09-20 16:18:26,318 StorageService.java:620 - CQL supported versions: 3.4.4 (default: 3.4.4) INFO [main] 2018-09-20 16:18:26,319 StorageService.java:622 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4) INFO [main] 2018-09-20 16:18:26,403 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 50 MB and a resize interval of 60 minutes INFO [main] 2018-09-20 16:18:26,423 MessagingService.java:761 - Starting Messaging Service on /10.128.0.3:7000 (eth0) INFO [main] 2018-09-20 16:18:26,545 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED INFO [HANDSHAKE-/10.128.0.2] 2018-09-20 16:18:26,582 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.2 INFO [main] 2018-09-20 16:18:27,582 StorageService.java:704 - Loading persisted ring state INFO [main] 2018-09-20 16:18:27,607 StorageService.java:822 - Starting up server gossip INFO [main] 2018-09-20 16:18:27,715 TokenMetadata.java:479 - Updating topology for /10.128.0.3 INFO [main] 2018-09-20 16:18:27,716 TokenMetadata.java:479 - Updating topology for /10.128.0.3 INFO [main] 2018-09-20 16:18:27,845 StorageService.java:1446 - JOINING: Finish joining ring INFO [main] 2018-09-20 16:18:27,896 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='keyspace1', ColumnFamily='standard1') INFO [main] 2018-09-20 16:18:27,900 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='keyspace1', ColumnFamily='counter1') INFO [main] 2018-09-20 16:18:27,901 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='stockdb', ColumnFamily='user') INFO [main] 2018-09-20 16:18:27,991 StorageService.java:2289 - Node /10.128.0.3 state jump to NORMAL INFO [main] 2018-09-20 16:18:28,033 AuthCache.java:172 - (Re)initializing CredentialsCache (validity period/update interval/max entries) (2000/2000/1000) INFO [main] 2018-09-20 16:18:28,043 Gossiper.java:1692 - Waiting for gossip to settle... INFO [GossipStage:1] 2018-09-20 16:18:28,731 Gossiper.java:1053 - Node /10.128.0.2 has restarted, now UP INFO [GossipStage:1] 2018-09-20 16:18:28,748 StorageService.java:2289 - Node /10.128.0.2 state jump to NORMAL INFO [GossipStage:1] 2018-09-20 16:18:28,775 TokenMetadata.java:479 - Updating topology for /10.128.0.2 INFO [GossipStage:1] 2018-09-20 16:18:28,776 TokenMetadata.java:479 - Updating topology for /10.128.0.2 INFO [RequestResponseStage-5] 2018-09-20 16:18:28,803 Gossiper.java:1019 - InetAddress /10.128.0.2 is now UP INFO [HANDSHAKE-/10.128.0.2] 2018-09-20 16:18:28,822 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.2
Observation: From above log we can see Messaging service is started on /10.128.0.3:7000 (eth0) and IP address of node is /10.128.0.3 and after starting its state is Normal .
Since Instance-1 is already running when Cassandra starts on this (10.128.0.3) node, it does handshake with already running node (10.128.0.2) and topology of both Instance-1 (10.128.0.2) and Instance-2 (10.128.0.3) is updated.
Start Cassandra on Instance-3 and capture log: IP address of Instance-3 is 10.128.0.4
INFO [ScheduledTasks:1] 2018-09-20 17:12:45,013 TokenMetadata.java:498 - Updating topology for all endpoints that have changed INFO [main] 2018-09-20 17:12:45,213 StorageService.java:600 - Populating token metadata from system tables INFO [main] 2018-09-20 17:12:45,370 StorageService.java:607 - Token metadata: Normal Tokens: /10.128.0.2:[......... ] /10.128.0.3:[..........] /10.128.0.4:[..........] NFO [main] 2018-09-20 17:12:46,849 MessagingService.java:761 - Starting Messaging Service on /10.128.0.4:7000 (eth0) INFO [main] 2018-09-20 17:12:46,960 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED INFO [HANDSHAKE-/10.128.0.2] 2018-09-20 17:12:47,004 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.2 INFO [main] 2018-09-20 17:12:48,000 StorageService.java:704 - Loading persisted ring state INFO [main] 2018-09-20 17:12:48,026 StorageService.java:822 - Starting up server gossip INFO [main] 2018-09-20 17:12:48,125 TokenMetadata.java:479 - Updating topology for /10.128.0.4 INFO [main] 2018-09-20 17:12:48,125 TokenMetadata.java:479 - Updating topology for /10.128.0.4 INFO [main] 2018-09-20 17:12:48,228 StorageService.java:1446 - JOINING: Finish joining ring INFO [main] 2018-09-20 17:12:48,274 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='keyspace1', ColumnFamily='standard1') INFO [main] 2018-09-20 17:12:48,280 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='keyspace1', ColumnFamily='counter1') INFO [main] 2018-09-20 17:12:48,280 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='stockdb', ColumnFamily='user') INFO [main] 2018-09-20 17:12:48,361 StorageService.java:2289 - Node /10.128.0.4 state jump to NORMAL INFO [main] 2018-09-20 17:12:48,394 AuthCache.java:172 - (Re)initializing CredentialsCache (validity period/update interval/max entries) (2000/2000/1000) INFO [main] 2018-09-20 17:12:48,401 Gossiper.java:1692 - Waiting for gossip to settle... INFO [HANDSHAKE-/10.128.0.3] 2018-09-20 17:12:48,979 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.3 INFO [HANDSHAKE-/10.128.0.3] 2018-09-20 17:12:48,993 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.3 INFO [GossipStage:1] 2018-09-20 17:12:49,015 Gossiper.java:1053 - Node /10.128.0.2 has restarted, now UP INFO [GossipStage:1] 2018-09-20 17:12:49,030 StorageService.java:2289 - Node /10.128.0.2 state jump to NORMAL INFO [GossipStage:1] 2018-09-20 17:12:49,045 TokenMetadata.java:479 - Updating topology for /10.128.0.2 INFO [GossipStage:1] 2018-09-20 17:12:49,047 TokenMetadata.java:479 - Updating topology for /10.128.0.2 INFO [GossipStage:1] 2018-09-20 17:12:49,050 Gossiper.java:1053 - Node /10.128.0.3 has restarted, now UP INFO [GossipStage:1] 2018-09-20 17:12:49,063 StorageService.java:2289 - Node /10.128.0.3 state jump to NORMAL INFO [RequestResponseStage-3] 2018-09-20 17:12:49,073 Gossiper.java:1019 - InetAddress /10.128.0.3 is now UP INFO [RequestResponseStage-3] 2018-09-20 17:12:49,074 Gossiper.java:1019 - InetAddress /10.128.0.2 is now UP INFO [GossipStage:1] 2018-09-20 17:12:49,078 TokenMetadata.java:479 - Updating topology for /10.128.0.3 INFO [GossipStage:1] 2018-09-20 17:12:49,079 TokenMetadata.java:479 - Updating topology for /10.128.0.3 INFO [HANDSHAKE-/10.128.0.2] 2018-09-20 17:12:49,304 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.2
Observation: From above log we can see Messaging service is started on /10.128.0.4:7000 (eth0) and IP address of node is /10.128.0.4 and after starting its state is Normal .
Since Instance-1 & Instance-2 is already running when Cassandra starts on this(10.128.0.4) node, it does handshakes with already running node (10.128.0.2 and 10.128.0.3) and topology of all three Instance-1 (10.128.0.2), Instance-2 (10.128.0.3) and Instance-3 (10.128.0.4) is updated.
We have highlighted token range for all running nodes in green. When a new node joins Virtually tokens are assigned.
Start Cassandra on Instance-4 and capture log: IP address of Instance-4 is 10.128.0.5
INFO [HANDSHAKE-/10.128.0.2] 2018-09-20 16:08:40,370 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.2 INFO [GossipStage:1] 2018-09-20 16:08:40,373 Gossiper.java:1053 - Node /10.128.0.4 has restarted, now UP INFO [main] 2018-09-20 16:08:40,399 StorageService.java:1446 - JOINING: Finish joining ring INFO [GossipStage:1] 2018-09-20 16:08:40,402 StorageService.java:2289 - Node /10.128.0.4 state jump to NORMAL INFO [HANDSHAKE-/10.128.0.4] 2018-09-20 16:08:40,403 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.4 INFO [GossipStage:1] 2018-09-20 16:08:40,444 TokenMetadata.java:479 - Updating topology for /10.128.0.4 INFO [GossipStage:1] 2018-09-20 16:08:40,447 TokenMetadata.java:479 - Updating topology for /10.128.0.4 INFO [GossipStage:1] 2018-09-20 16:08:40,447 Gossiper.java:1053 - Node /10.128.0.2 has restarted, now UP INFO [RequestResponseStage-3] 2018-09-20 16:08:40,468 Gossiper.java:1019 - InetAddress /10.128.0.4 is now UP INFO [GossipStage:1] 2018-09-20 16:08:40,472 StorageService.java:2289 - Node /10.128.0.2 state jump to NORMAL INFO [RequestResponseStage-2] 2018-09-20 16:08:40,483 Gossiper.java:1019 - InetAddress /10.128.0.2 is now UP INFO [GossipStage:1] 2018-09-20 16:08:40,485 TokenMetadata.java:479 - Updating topology for /10.128.0.2 INFO [GossipStage:1] 2018-09-20 16:08:40,486 TokenMetadata.java:479 - Updating topology for /10.128.0.2 INFO [GossipStage:1] 2018-09-20 16:08:40,491 Gossiper.java:1053 - Node /10.128.0.3 has restarted, now UP INFO [GossipStage:1] 2018-09-20 16:08:40,518 StorageService.java:2289 - Node /10.128.0.3 state jump to NORMAL INFO [HANDSHAKE-/10.128.0.3] 2018-09-20 16:08:40,536 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.3 INFO [main] 2018-09-20 16:08:40,540 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='keyspace1', ColumnFamily='standard1') INFO [GossipStage:1] 2018-09-20 16:08:40,537 TokenMetadata.java:479 - Updating topology for /10.128.0.3 INFO [GossipStage:1] 2018-09-20 16:08:40,550 TokenMetadata.java:479 - Updating topology for /10.128.0.3 INFO [main] 2018-09-20 16:08:40,551 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='keyspace1', ColumnFamily='counter1') INFO [main] 2018-09-20 16:08:40,556 SecondaryIndexManager.java:509 - Executing pre-join tasks for: CFS(Keyspace='stockdb', ColumnFamily='user') INFO [RequestResponseStage-1] 2018-09-20 16:08:40,599 Gossiper.java:1019 - InetAddress /10.128.0.3 is now UP INFO [main] 2018-09-20 16:08:40,643 StorageService.java:2289 - Node /10.128.0.5 state jump to NORMAL INFO [main] 2018-09-20 16:08:40,668 AuthCache.java:172 - (Re)initializing CredentialsCache (validity period/update interval/max entries) (2000/2000/1000) INFO [main] 2018-09-20 16:08:40,676 Gossiper.java:1692 - Waiting for gossip to settle... INFO [HANDSHAKE-/10.128.0.3] 2018-09-20 16:08:41,284 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.3 INFO [HANDSHAKE-/10.128.0.4] 2018-09-20 16:08:41,297 OutboundTcpConnection.java:561 - Handshaking version with /10.128.0.4
Observation: From above log we can see IP address of node is /10.128.0.4 and after starting its state is Normal .
Since Instance-1, Instance-2 & Instance-3 is already running when Cassandra starts on this(10.128.0.5) node, it does handshakes with already running node (10.128.0.2 , 10.128.0.3 and 10.128.0.4) and topology of all four Instance-1 (10.128.0.2), Instance-2 (10.128.0.3), Instance-3 (10.128.0.4) and Instance-3 (10.128.0.5) is updated.
Cassandra Data storage visualisation
All four instance of Cassandra is up and running. Run below command and validate data centre(DC) configuration : two node in rack1 and 2 node in rack2.nikhilranjan234@instance-1:~$ nodetool status Datacenter: dc1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.128.0.2 38.99 MiB 256 ? 02b41029-cacc-47d8-91ca-44a579071529 r1 UN 10.128.0.3 44.05 MiB 256 ? 94b6296c-f1d2-4817-af32-8ae8e7ea07fc r1 UN 10.128.0.4 61.11 MiB 256 ? 0ec021b0-0ae9-47fc-bd5b-894287d78a0b r2 UN 10.128.0.5 85.07 MiB 256 ? 0828fce5-715c-4482-a909-e9e1fd40e26a r2
4 instance of Cassandra is up and running([rack-1: I1 & I2] [rack-2 : I3 & I4] )
Keyspace and Table(column family) creation : Run cqlsh utility on one of the terminal .
nikhilranjan234@instance-1:~$ cqlsh `hostname -I` -u cassandra -p cassandra Connected to wm-cluster at 10.128.0.2:9042. [cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4] Use HELP for help. cassandra@cqlsh>
Create Keyspace with replication factor 3: We have four instance of Cassandra running so we can have 3 copy of data on different instances in given DC.
On keyspace creation success prompt does not gives any success message, describe keyspaces and see it has been created.
cassandra@cqlsh> CREATE KEYSPACE IF NOT EXISTS "OCProc" WITH REPLICATION = {'class':'NetworkTopologyStrategy', 'dc1' : 3}; cassandra@cqlsh> describe keyspaces; system_schema system "OCProc" system_traces system_auth keyspace1 system_distributed stockdb
Select and Use a keyspace : Since column families (like table in RDMS) are created in context of keyspace so select keyspace we want to work with.
cassandra@cqlsh> use "OCProc" ; cassandra@cqlsh:OCProc>
Create Table(Column Family) in selected Keyspace:
cassandra@cqlsh:OCProc> CREATE TABLE user ( ... username text, ... email text, ... city text, ... phone varint, ... encrypted_password blob, ... PRIMARY KEY (username, city) ... )WITH comment = 'Creating USERS Tabel to store users details'; cassandra@cqlsh:OCProc> describe tables; user
Insert data in user Table(Column Family) and select table to display data:
cassandra@cqlsh:OCProc> INSERT INTO user ... ( username , email , city , phone , encrypted_password ) ... VALUES ( ... 'zytham', ... 'zytham@gmail.com', ... 'Patna', ... 9999888800, ... 0x9792977ed729792e403da53024c6069a9158b8c4 ... ); cassandra@cqlsh:OCProc> INSERT INTO user ... ( username , email , city , phone , encrypted_password ) ... VALUES( ... 'ranjan', ... 'ranjan@gmail.com', ... 'Bangalore', ... 678998800, ... 0x8914977ed729792e403da53024c6069a9158b8c4 ... ); cassandra@cqlsh:OCProc> INSERT INTO user ... ( username , email , city , phone , encrypted_password ) ... VALUES( ... 'mishra', ... 'zytham@gmail.com', ... 'Torento', ... 00980099766, ... 0x891497745729792e403da53024c6069a9158b8c4 ... ); cassandra@cqlsh:OCProc> SELECT * FROM user; username | city | email | encrypted_password | phone ----------+-----------+------------------+--------------------------------------------+------------ zytham | Patna | zytham@gmail.com | 0x9792977ed729792e403da53024c6069a9158b8c4 | 9999888800 ranjan | Bangalore | ranjan@gmail.com | 0x8914977ed729792e403da53024c6069a9158b8c4 | 678998800 mishra | Torento | zytham@gmail.com | 0x891497745729792e403da53024c6069a9158b8c4 | 980099766
Replicated of data on Cassandra nodes: Which node stores data for username: zytham ?
Below command shows one copy is stored on instance-1(where we are running query) and 2 copy is stored on Instance-3 and Instance-4 (rack2)
nikhilranjan234@instance-1:~$ nodetool getendpoints OCProc user zytham 10.128.0.2 10.128.0.4 10.128.0.5
Data directory and its storage : Default data storage directory is /opt/apache-cassandra-3.11.3/data/
Data storage hierarchy : <KEYSPACE>/<TABLE_UNIQUE_ID>/<GENERATION_DATA>
nikhilranjan234@instance-1:/$ cd /opt/apache-cassandra-3.11.3/data/data nikhilranjan234@instance-1:/opt/apache-cassandra-3.11.3/data/data$ ls -l total 32 drwxr-xr-x 4 nikhilranjan234 nikhilranjan234 4096 Sep 18 10:22 keyspace1 drwxr-xr-x 3 nikhilranjan234 nikhilranjan234 4096 Sep 20 18:07 OCProc drwxr-xr-x 3 nikhilranjan234 nikhilranjan234 4096 Sep 18 07:03 stockdb drwxr-xr-x 26 nikhilranjan234 nikhilranjan234 4096 Sep 18 06:36 system drwxr-xr-x 6 nikhilranjan234 nikhilranjan234 4096 Sep 18 06:36 system_auth drwxr-xr-x 5 nikhilranjan234 nikhilranjan234 4096 Sep 18 06:36 system_distributed drwxr-xr-x 12 nikhilranjan234 nikhilranjan234 4096 Sep 18 06:36 system_schema drwxr-xr-x 4 nikhilranjan234 nikhilranjan234 4096 Sep 18 06:36 system_traces
We have OCProc keyspace created as directory. Run below command and find persisted data.
nikhilranjan234@instance-1:/opt/apache-cassandra-3.11.3/data/data$ cd OCProc/ nikhilranjan234@instance-1:/opt/apache-cassandra-3.11.3/data/data/OCProc$ cd user-f82692c0bcff11e8a9c80961902fe681/ nikhilranjan234@instance-1:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ ls backups
Where is our data, no data is persisted yet :
Answer: memtable has all our user data, it not yet persisted in disk (sstable)
How does data comes from memtable to disk ?
Answer: When memtable flush data then it is stored on disk. Let use nodetool to forceful flush data and see what changes it brings to directory "user-f82692c0bcff11e8a9c80961902fe681".
nikhilranjan234@instance-1:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ ls backups nikhilranjan234@instance-1:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ nodetool flush nikhilranjan234@instance-1:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ ls -l total 40 drwxr-xr-x 2 nikhilranjan234 nikhilranjan234 4096 Sep 20 18:07 backups -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 43 Sep 20 18:44 mc-1-big-CompressionInfo.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 83 Sep 20 18:44 mc-1-big-Data.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 10 Sep 20 18:44 mc-1-big-Digest.crc32 -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 16 Sep 20 18:44 mc-1-big-Filter.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 10 Sep 20 18:44 mc-1-big-Index.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 4772 Sep 20 18:44 mc-1-big-Statistics.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 62 Sep 20 18:44 mc-1-big-Summary.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 92 Sep 20 18:44 mc-1-big-TOC.txt
On flush memtable flushes data to disk(Intermediate stage is sstable). File "mc-1-big-Data.db" contains our data and along with flush also creates files for index and filter.
View User data in "mc-1-big-Data.db" : Using sstabledump on instance-1 we can visualise user data in file "mc-1-big-Data.db".
nikhilranjan234@instance-1:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ sstabledump -d mc-1-big-Data.db WARN 18:55:20,083 Small commitlog volume detected at /opt/apache-cassandra-3.11.3/data/commitlog; setting commitlog_total_space_in_mb to 2503. You can override this in cassandra.yaml WARN 18:55:20,100 Small cdc volume detected at /opt/apache-cassandra-3.11.3/data/cdc_raw; setting cdc_total_space_in_mb to 1251. You can override this in cassandra.yaml WARN 18:55:20,267 Only 7.951GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots [zytham]@0 Row[info=[ts=1537467165740069] ]: Patna | [email=zytham@gmail.com ts=1537467165740069], [encrypted_password=9792977ed729792e403da53024c6069a9158b8c4 ts=1537467165740069], [p hone=9999888800 ts=1537467165740069]
Where are other user records ? : Apply node flush on Instance-3 and display data stored in file first generation data file "mc-1-big-Data.db". Below we can see one copy of each user record is stored on Instance-3.
nikhilranjan234@instance-3:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ sstabledump -d mc-1-big-Data.db WARN 18:55:50,970 Small commitlog volume detected at /opt/apache-cassandra-3.11.3/data/commitlog; setting commitlog_total_space_in_mb to 2503. You can override this in cassandra.yaml WARN 18:55:50,977 Small cdc volume detected at /opt/apache-cassandra-3.11.3/data/cdc_raw; setting cdc_total_space_in_mb to 1251. You can override this in cassandra.yaml WARN 18:55:51,146 Only 7.946GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots [zytham]@0 Row[info=[ts=1537467165740069] ]: Patna | [email=zytham@gmail.com ts=1537467165740069], [encrypted_password=9792977ed729792e403da53024c6069a9158b8c4 ts=1537467165740069], [p hone=9999888800 ts=1537467165740069] [ranjan]@79 Row[info=[ts=1537467165763097] ]: Bangalore | [email=ranjan@gmail.com ts=1537467165763097], [encrypted_password=8914977ed729792e403da53024c6069a9158b8c4 ts=1537467165763097 ], [phone=678998800 ts=1537467165763097] [mishra]@163 Row[info=[ts=1537467168375384] ]: Torento | [email=zytham@gmail.com ts=1537467168375384], [encrypted_password=891497745729792e403da53024c6069a9158b8c4 ts=1537467168375384] , [phone=980099766 ts=1537467168375384]
Display JOSN form of data visualisation: Execute above command without -d switch.
nikhilranjan234@instance-3:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ sstabledump mc-1-big-Data.db [ { "partition" : { "key" : [ "zytham" ], "position" : 0 }, "rows" : [ { "type" : "row", "position" : 78, "clustering" : [ "Patna" ], "liveness_info" : { "tstamp" : "2018-09-20T18:12:45.740069Z" }, "cells" : [ { "name" : "email", "value" : "zytham@gmail.com" }, { "name" : "encrypted_password", "value" : "0x9792977ed729792e403da53024c6069a9158b8c4" }, { "name" : "phone", "value" : 9999888800 } ] } ] }, { "partition" : { "key" : [ "ranjan" ], "position" : 79 }, "rows" : [ { "type" : "row", "position" : 162, "clustering" : [ "Bangalore" ], "liveness_info" : { "tstamp" : "2018-09-20T18:12:45.763097Z" }, "cells" : [ { "name" : "email", "value" : "ranjan@gmail.com" }, { "name" : "encrypted_password", "value" : "0x8914977ed729792e403da53024c6069a9158b8c4" }, { "name" : "phone", "value" : 678998800 } ] } ] }, { "partition" : { "key" : [ "mishra" ], "position" : 163 }, "rows" : [ { "type" : "row", "position" : 245, "clustering" : [ "Torento" ], "liveness_info" : { "tstamp" : "2018-09-20T18:12:48.375384Z" }, "cells" : [ { "name" : "email", "value" : "zytham@gmail.com" }, { "name" : "encrypted_password", "value" : "0x891497745729792e403da53024c6069a9158b8c4" }, { "name" : "phone", "value" : 980099766 } ] } ] } ]
Delete record from user table: delete row form user where username = 'mishra' and city='Torento';
nikhilranjan234@instance-3:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ cqlsh `hostname -I` -u cassandra -p cassandra Connected to wm-cluster at 10.128.0.4:9042. [cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4] Use HELP for help. cassandra@cqlsh> cassandra@cqlsh> cassandra@cqlsh> cassandra@cqlsh> cassandra@cqlsh> describe keyspaces system_schema system "OCProc" system_traces system_auth keyspace1 system_distributed stockdb cassandra@cqlsh> use "OCProc"; cassandra@cqlsh:OCProc> delete from user where username = 'mishra' and city='Torento'; cassandra@cqlsh:OCProc> exit
See data file again, no change. Where did chnage happend till now ? :
Answer: memtable.
nikhilranjan234@instance-3:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ sstabledump -d mc-1-big-Data.db WARN 19:13:18,553 Small commitlog volume detected at /opt/apache-cassandra-3.11.3/data/commitlog; setting commitlog_total_space_in_mb to 2503. You can override this in cassandra.yaml WARN 19:13:18,559 Small cdc volume detected at /opt/apache-cassandra-3.11.3/data/cdc_raw; setting cdc_total_space_in_mb to 1251. You can override this in cassandra. yaml WARN 19:13:18,729 Only 7.945GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots [zytham]@0 Row[info=[ts=1537467165740069] ]: Patna | [email=zytham@gmail.com ts=1537467165740069], [encrypted_password=9792977ed729792e403da53024c6069a9158b8c4 ts=153 7467165740069], [phone=9999888800 ts=1537467165740069] [ranjan]@79 Row[info=[ts=1537467165763097] ]: Bangalore | [email=ranjan@gmail.com ts=1537467165763097], [encrypted_password=8914977ed729792e403da53024c6069a9158b8c4 t s=1537467165763097], [phone=678998800 ts=1537467165763097] [mishra]@163 Row[info=[ts=1537467168375384] ]: Torento | [email=zytham@gmail.com ts=1537467168375384], [encrypted_password=891497745729792e403da53024c6069a9158b8c4 ts =1537467168375384], [phone=980099766 ts=1537467168375384]
Flush using nodetool : New generation files are persisted along with old one, old files are not deleted.
nikhilranjan234@instance-3:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ nodetool flush nikhilranjan234@instance-3:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ ls -l total 76 drwxr-xr-x 2 nikhilranjan234 nikhilranjan234 4096 Sep 20 18:07 backups -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 43 Sep 20 18:54 mc-1-big-CompressionInfo.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 173 Sep 20 18:54 mc-1-big-Data.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 10 Sep 20 18:54 mc-1-big-Digest.crc32 -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 16 Sep 20 18:54 mc-1-big-Filter.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 31 Sep 20 18:54 mc-1-big-Index.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 4786 Sep 20 18:54 mc-1-big-Statistics.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 62 Sep 20 18:54 mc-1-big-Summary.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 92 Sep 20 18:54 mc-1-big-TOC.txt -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 43 Sep 20 19:15 mc-2-big-CompressionInfo.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 42 Sep 20 19:15 mc-2-big-Data.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 10 Sep 20 19:15 mc-2-big-Digest.crc32 -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 16 Sep 20 19:15 mc-2-big-Filter.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 10 Sep 20 19:15 mc-2-big-Index.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 4637 Sep 20 19:15 mc-2-big-Statistics.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 62 Sep 20 19:15 mc-2-big-Summary.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 92 Sep 20 19:15 mc-2-big-TOC.txt
When old (old generation) files be deleted ? :
Answer: During comapction
Apply compaction: Delete old generation file and merged old & new generation files.
nikhilranjan234@instance-3:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ nodetool compact nikhilranjan234@instance-3:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ ls -l total 40 drwxr-xr-x 2 nikhilranjan234 nikhilranjan234 4096 Sep 20 18:07 backups -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 51 Sep 20 19:17 mc-3-big-CompressionInfo.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 171 Sep 20 19:17 mc-3-big-Data.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 10 Sep 20 19:17 mc-3-big-Digest.crc32 -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 16 Sep 20 19:17 mc-3-big-Filter.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 31 Sep 20 19:17 mc-3-big-Index.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 4805 Sep 20 19:17 mc-3-big-Statistics.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 62 Sep 20 19:17 mc-3-big-Summary.db -rw-r--r-- 1 nikhilranjan234 nikhilranjan234 92 Sep 20 19:17 mc-3-big-TOC.txt
Deleted and compacted: Now visualise merged data file. It has not been removed, it has been marked as deleted (Why?) : If some failed node joins cluster after recovery and has old data, then this marked deleted help to update that node and remove stalled data which has been deleted already.
nikhilranjan234@instance-3:/opt/apache-cassandra-3.11.3/data/data/OCProc/user-f82692c0bcff11e8a9c80961902fe681$ sstabledump -d mc-3-big-Data.db WARN 19:26:45,958 Small commitlog volume detected at /opt/apache-cassandra-3.11.3/data/commitlog; setting commitlog_total_space_in_mb to 2503. You can override this in cassandra.yaml WARN 19:26:45,968 Small cdc volume detected at /opt/apache-cassandra-3.11.3/data/cdc_raw; setting cdc_total_space_in_mb to 1251. You can override this in cassandra. yaml WARN 19:26:46,151 Only 7.966GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots [zytham]@0 Row[info=[ts=1537467165740069] ]: Patna | [email=zytham@gmail.com ts=1537467165740069], [encrypted_password=9792977ed729792e403da53024c6069a9158b8c4 ts=153 7467165740069], [phone=9999888800 ts=1537467165740069] [ranjan]@79 Row[info=[ts=1537467165763097] ]: Bangalore | [email=ranjan@gmail.com ts=1537467165763097], [encrypted_password=8914977ed729792e403da53024c6069a9158b8c4 t s=1537467165763097], [phone=678998800 ts=1537467165763097] [mishra]@163 Row[info=[ts=-9223372036854775808] del=deletedAt=1537470698672680, localDeletion=1537470698 ]: Torento |
Here is a nice from the last pikcle post which explains in details - Deletes and Tombstones in Cassandra
Tags:
Cassandra
Liên hệ Aivivu, đặt vé máy bay tham khảo
ReplyDeletevé máy bay đi Mỹ giá rẻ 2021
có chuyến bay từ mỹ về việt nam
vé máy bay từ đức về việt nam giá rẻ
vé máy bay từ nga về việt nam bao nhiêu
các chuyến bay từ anh về việt nam
lịch bay từ pháp về việt nam
các khách sạn cách ly ở hà nội
Whether it's cookies, sweets, cakes, noodles, sandwiches, or more, you've likely seen them with China Bopet film suppliers packaged in plastic in a supermarket or convenience shop. These containers sell spices and rice and wheat flour, two essential foods. Bottles and tubes for cosmetics, culinary oils, containers, and jars make extensive use of plastic and plastic packing bags. There was an incredible uptake of PET bottles filled with cold beverages shortly after their introduction. Plastic packaging items have flooded the market due to their adaptability and efficacy, even though they are not totally recyclable.
ReplyDelete