一、什么是水平分庫
將一張表水平切分到多個庫中
1.1分片原則
- 1.需要分片的表是少數的
- 2.能不切分盡量不要切分
- 3.日志表可以采取歸檔方式
- 4.選擇合適的切分規則和分片建,確保數據分片均勻,否則依然存在性能瓶頸
- 5.盡量避免跨分片join操作,保證關聯操作表在同一分片
1.2分片后如何處理查詢
- 1、根據簡單分片規則,對分片鍵進行路由到正確的后端物理數據庫
-
2、如果不是使用分片鍵的話,將會遍歷后端數據庫,極大消耗性能
二、水平切分步驟
2.1根據業務狀態確定要進行水平切分的表
讀寫頻繁,訪問量非常大的表才需要切分,一般是訂單表
如何選擇分片鍵
- 1、盡可能的比較均勻分布數據到各個節點上,自增長主鍵并不是很好的選擇,因為不會用于查詢
- 2、業務字段是最頻繁的或者最重要的查詢條件
2.2分析業務模型選擇分片鍵及分片算法
一般分片鍵選擇的是頻繁作為查詢的字段,關鍵是能保證分片后的數據分布均勻,常用簡單取模算法
- 對訂單相關表進行水平切分
- 不僅僅是訂單表,經常與訂單表關聯查詢的表也需要一并分片,避免跨分片查詢,大表不適合作為全局表
- 訂單號,可以保證分片均勻,但是實際業務很少根據訂單號來查詢
- 下單人ID,業務查詢更多,更適合,但是分片并不均勻,但不嚴重,值得考慮
- 采用簡單取模分片算法,可以保證數據盡可能均勻
2.3適用mycat部署分片集群
- 1、使用schema.xml配置邏輯庫及邏輯表
<!-- <table name="order_master" primaryKey="order_id" dataNode="dn_orderdb" /> -->
<table name="order_master" primaryKey="order_id" dataNode="dn_orderdb01,dn_orderdb02,dn_orderdb03,dn_orderdb04" rule="order_master"/>
<!-- 只需要增加表所在的物理節點schema名稱和指定規則rule -->
<dataNode name="dn_orderdb01" dataHost="mysql10143" database="orderdb01" />
<dataNode name="dn_orderdb02" dataHost="mysql10143" database="orderdb02" />
<dataNode name="dn_orderdb03" dataHost="mysql10144" database="orderdb03" />
<dataNode name="dn_orderdb04" dataHost="mysql10144" database="orderdb04" />
- 2.使用rule.xml配置分片表的分片規則
<tableRule name="order_master">
<rule>
<columns>customer_id</columns>
<algorithm>mod-long</algorithm>
</rule>
</tableRule>
<function name="mod-long" class="org.opencloudb.route.function.PartitionByMod">
<!-- 分片算法,唯一;JAVA類路徑,1.6是"io.mycat.route.function.PartitionByMod"-->
<property name="count">4</property>
</function>
- 3、驗證
# 現在邏輯庫上進行查看,此時是沒有數據的
# 需要提前建庫建表,否則報錯表不存在
app_imooc@172.16.10.142 00:26: [imooc_db]> select * from order_master;
Empty set (0.08 sec)
# 在邏輯庫imooc_db上插入5條數據,正常是orderdb02 2條數據(節點索引順序為1),其他各1條數據
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70005,90005,'yzw5',5,5,5,'address1',1,20);
# truncate table報錯
app_imooc@172.16.10.142 10:51: [imooc_db]> truncate table order_master;
ERROR 1105 (HY000): DROP command denied to user 'bm_mycat'@'172.16.10.142' for table 'order_master'
# 邏輯庫查看插入結果
app_imooc@172.16.10.142 10:50: [imooc_db]> select order_id,order_sn,customer_id from order_master;
+----------+----------+-------------+
| order_id | order_sn | customer_id |
+----------+----------+-------------+
| 80005 | 70002 | 90002 |
| 80005 | 70003 | 90003 |
| 80005 | 70004 | 90004 |
| 80006 | 70001 | 90001 |
| 80007 | 70005 | 90005 |
+----------+----------+-------------+
5 rows in set (0.00 sec)
# 在物理庫查看是否有這4條數據,驗證成功
root@localhost 10:52: [orderdb01]> select order_id,order_sn,customer_id from order_master;
+----------+----------+-------------+
| order_id | order_sn | customer_id |
+----------+----------+-------------+
| 80005 | 70004 | 90004 |
+----------+----------+-------------+
1 row in set (0.00 sec)
root@localhost 10:52: [orderdb02]> select order_id,order_sn,customer_id from order_master;
+----------+----------+-------------+
| order_id | order_sn | customer_id |
+----------+----------+-------------+
| 80006 | 70001 | 90001 |
| 80007 | 70005 | 90005 |
+----------+----------+-------------+
2 rows in set (0.00 sec)
root@localhost 10:05: [orderdb03]> select order_id,order_sn,customer_id from order_master;
+----------+----------+-------------+
| order_id | order_sn | customer_id |
+----------+----------+-------------+
| 80005 | 70002 | 90002 |
+----------+----------+-------------+
1 row in set (0.00 sec)
root@localhost 10:52: [orderdb04]> select order_id,order_sn,customer_id from order_master;
+----------+----------+-------------+
| order_id | order_sn | customer_id |
+----------+----------+-------------+
| 80005 | 70003 | 90003 |
+----------+----------+-------------+
1 row in set (0.00 sec)
- 4、使用server.xml配置訪問權限
#配置用戶登錄
2.4測試分片集群,采用應用端雙寫方式進行
2.5業務及數據遷移
三、全局自增ID
- 分片表中的自增ID在邏輯表中有重復
- 第三方給ID或者使用mycat自增ID
3.1全局自增ID方法
- 1、本地文件方式:適用服務器本地磁盤的方式
- 2、數據庫方式:適用數據庫存儲的方式(自增主鍵方式)
- 3、本地時間戳方式:適用時間戳
- 4、分布式zookeeper生成ID
3.2本地文件全局ID
- 優點:本地加載,讀取速度快,配置簡單
- 缺點:集群部署無法使用,不同的mycat無法保證id唯一,使mycat變成了有狀態的中間件
配置方法
- 1.server.xml增加屬性
<property name="sequnceHandlerType>0</property>
- 2.sequence_conf.properties配置
#default global sequence
GLOBAL.HISIDS=
GLOBAL.MINID=10001
GLOBAL.MAXID=20000
GLOBAL.CURID=10000
# self define sequence
ORDER_MASTER.HISIDS=
ORDER_MASTER.MINID=1001
ORDER_MASTER.MAXID=2000
ORDER_MASTER.CURID=1000
# 以上配置文件中,自定義表名必須大寫書寫HISIDS:表示使用過的歷史分段(一般無特殊需要可不配置)
# MINID :最小ID 值
# MAXID :表示最大ID 值
# CURID :表示當前ID 值。
# 當 sequence_conf.properties的配置名字與 表名一致的時候sql可以不包含ID字段(此處表名為id_local_file),邏輯表需要增加屬性autoIncrement="true"
- 3.插入數據
# mycat restart
# delete from order_master;
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70005,90005,'yzw5',5,5,5,'address1',1,20);
# 此時邏輯表中的order_id唯一
app_imooc@172.16.10.142 11:35: [imooc_db]> select order_id,order_sn,customer_id from order_master;
+----------+----------+-------------+
| order_id | order_sn | customer_id |
+----------+----------+-------------+
| 10004 | 70004 | 90004 |
| 10003 | 70003 | 90003 |
| 10001 | 70001 | 90001 |
| 10005 | 70005 | 90005 |
| 10002 | 70002 | 90002 |
+----------+----------+-------------+
5 rows in set (0.01 sec)
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70005,90005,'yzw5',5,5,5,'address1',1,20);
# 此時邏輯表中的order_id依然是唯一,但是ID值使用的是自定義表自增屬性的值
app_imooc@172.16.10.142 11:48: [imooc_db]> select order_id,order_sn,customer_id from order_master;
+----------+----------+-------------+
| order_id | order_sn | customer_id |
+----------+----------+-------------+
| 1009 | 70003 | 90003 |
| 1008 | 70002 | 90002 |
| 1007 | 70001 | 90001 |
| 1011 | 70005 | 90005 |
| 1010 | 70004 | 90004 |
+----------+----------+-------------+
5 rows in set (0.00 sec)
# 不指定自增字段插入
# 邏輯表必須增加自增屬性,否則不指定自增字段插入則使用分片表本身自增id
#<table name="order_master" primaryKey="order_id" autoIncrement="true" dataNode="dn_orderdb01,dn_orderdb02,dn_orderdb03,dn_orderdb04" rule="order_master"/>
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70005,90005,'yzw5',5,5,5,'address1',1,20);
app_imooc@172.16.10.142 11:58: [imooc_db]> select order_id,order_sn,customer_id from order_master;
+----------+----------+-------------+
| order_id | order_sn | customer_id |
+----------+----------+-------------+
| 1001 | 70001 | 90001 |
| 1005 | 70005 | 90005 |
| 1003 | 70003 | 90003 |
| 1004 | 70004 | 90004 |
| 1002 | 70002 | 90002 |
+----------+----------+-------------+
5 rows in set (0.01 sec)
3.3數據庫方式
- 優點:mycat重啟后,sequence值不會被初始化,因為從數據庫中取,每次取完按自增步長增加
- 缺點:sequence數據庫主從切換后,存在mycat適用緩存序列號,可能存在ID重復風險,可以手動增加步長避免
配置方法
- 1.server.xml增加屬性
<property name="sequnceHandlerType">1</property>
- 2.sequence_db_conf.properties
#sequence stored in datanode
GLOBAL=dn_test
ORDER_MASTER=dn_test
datanode必須要mycat能訪問的數據庫,并不一定需要在分片表的數據庫中
3.schema.xml增加主機節點和數據節點,讓mycat能訪問上面配置的數據節點
4.在對應節點的數據庫中增加函數和表,用戶必須授予表、函數、存儲過程的訪問權限
# 需要全局ID的表明需要寫入表中,大寫
INSERT INTO mycat_sequence values('ORDER_MASTER', 1, 100);
root@master 11:48: [test]> select * from mycat_sequence;
+--------------+---------------+-----------+
| NAME | current_value | increment |
+--------------+---------------+-----------+
| GLOBAL | 100000 | 100 |
| ORDER_MASTER | 1 | 100 |
+--------------+---------------+-----------+
2 rows in set (0.00 sec)
- 5.schema.xml增加配置,適用全局自增IDautoIncrement="true"
<table name="order_master" primaryKey="order_id" autoIncrement="true" dataNode="dn_orderdb01,dn_orderdb02,dn_orderdb03,dn_orderdb04" rule="order_master"/>
- 6.在邏輯庫中插入數據
# 適用ID表里面的全局ID插入
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70005,90005,'yzw5',5,5,5,'address1',1,20);
# 使用ID表里面的對應表ID插入
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70005,90005,'yzw5',5,5,5,'address1',1,20);
# 不指定id列,效果同上
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70005,90005,'yzw5',5,5,5,'address1',1,20);
3.4本地時間戳方式
ID= 64 位二進制 (42(毫秒)+5(機器 ID)+5(業務編碼)+12(重復累加)
- 優點:與mycat無關,跟時間有關系
- 缺點:字段最大長度必須大于等于18,int無法滿足
配置方法
- 1.server.xml增加屬性
<property name="sequnceHandlerType">2</property>
- 2.sequence_time_conf.properties
#sequence depend on TIME
WORKID=01
DATAACENTERID=01
# 0-31任意整數
- 3.在對應節點的數據庫中增加函數和表,用戶必須授予表、函數、存儲過程的訪問權限
# 需要全局ID的表明需要寫入表中,大寫
INSERT INTO mycat_sequence values('ORDER_MASTER', 1, 100);
root@master 11:48: [test]> select * from mycat_sequence;
+--------------+---------------+-----------+
| NAME | current_value | increment |
+--------------+---------------+-----------+
| GLOBAL | 100000 | 100 |
| ORDER_MASTER | 1 | 100 |
+--------------+---------------+-----------+
2 rows in set (0.00 sec)
- 4.schema.xml增加配置,適用全局自增IDautoIncrement="true"
<table name="order_master" primaryKey="order_id" autoIncrement="true" dataNode="dn_orderdb01,dn_orderdb02,dn_orderdb03,dn_orderdb04" rule="order_master"/>
- 5.在邏輯庫中插入數據
# int無法滿足
ERROR 1264 (22003): Out of range value for column 'order_id' at row
app_imooc@172.16.10.142 12:32: [imooc_db]> select next value for MYCATSEQ_ORDER_MASTER;
+---------------------+
| 1025962766269288448 |
+---------------------+
| 1025962766269288448 |
+---------------------+
1 row in set (0.00 sec)
app_imooc@172.16.10.142 12:32: [imooc_db]> select next value for MYCATSEQ_GLOBAL;
+---------------------+
| 1025962827724230656 |
+---------------------+
| 1025962827724230656 |
+---------------------+
1 row in set (0.00 sec)
1
# 適用ID表里面的全局ID插入
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_GLOBAL,70005,90005,'yzw5',5,5,5,'address1',1,20);
# 使用ID表里面的對應表ID插入
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_id,order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(next value for MYCATSEQ_ORDER_MASTER,70005,90005,'yzw5',5,5,5,'address1',1,20);
# 不指定id列,效果同上
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70005,90005,'yzw5',5,5,5,'address1',1,20);
select order_id,order_sn,customer_id from order_master;
3.5分布式zookeeper生成ID 未測試,1.5沒有默認配置文件
基于ZK 與本地配置的分布式ID 生成器(可以通過ZK 獲取集群(機房)唯一InstanceID,也可以通過配置文件配置InstanceID)ID 結構:long 64 位,ID 最大可占63 位
current time millis(微秒時間戳38 位,可以使用17 年)
instanceId(實例ID,可以通過ZK 或者配置文件獲取,5 位,也就是十進制0-31)
threadId(線程ID,9 位)
increment(自增,6 位)
一共63 位,可以承受單機房單機器單線程1000*(2^6)=640000 的并發。
- 優點:無悲觀鎖,無強競爭,吞吐量更高
- 缺點:對zookeeper集群的要求增加。
配置方法
- 1.server.xml增加屬性
<property name="sequnceHandlerType">3</property>
- 2.sequence_distributed_conf.properties
INSTANCEID=01
CLUSTERID=01
- 3.在對應節點的數據庫中增加函數和表,用戶必須授予表、函數、存儲過程的訪問權限
# 需要全局ID的表明需要寫入表中,大寫
INSERT INTO mycat_sequence values('ORDER_MASTER', 1, 100);
root@master 11:48: [test]> select * from mycat_sequence;
+--------------+---------------+-----------+
| NAME | current_value | increment |
+--------------+---------------+-----------+
| GLOBAL | 100000 | 100 |
| ORDER_MASTER | 1 | 100 |
+--------------+---------------+-----------+
2 rows in set (0.00 sec)
- 5.schema.xml增加配置,適用全局自增IDautoIncrement="true"
<table name="order_master" primaryKey="order_id" autoIncrement="true" dataNode="dn_orderdb01,dn_orderdb02,dn_orderdb03,dn_orderdb04" rule="order_master"/>
- 6.在邏輯庫中插入數據
四.ER關系
- 跨分片查詢存在跨節點問題
app_imooc@172.16.10.142 13:55: [imooc_db]> select * from order_master m join order_detail o on m.order_id=o.order_id;
ERROR 1064 (HY000): invalid route in sql, multi tables found but datanode has no intersection sql:select * from order_master m join order_detail o on m.order_id=o.order_id
- 根據ER關系,也需要對order_detail這張表進行分片
4.1配置步驟
- 1.在原先order_master所在4個分配數據庫(orderdb01/02/03/04)中建立order_detail表結構
CREATE TABLE `order_detail` (
`order_detail_id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增主鍵ID,訂單詳情表ID',
`order_id` int(10) unsigned NOT NULL COMMENT '訂單表ID',
`product_id` int(10) unsigned NOT NULL COMMENT '訂單商品ID',
`product_name` varchar(50) NOT NULL COMMENT '商品名稱',
`product_cnt` int(11) NOT NULL DEFAULT '1' COMMENT '購買商品數量',
`product_price` decimal(8,2) NOT NULL COMMENT '購買商品單價',
`average_cost` decimal(8,2) NOT NULL DEFAULT '0.00' COMMENT '平均成本價格',
`weight` float DEFAULT NULL COMMENT '商品重量',
`fee_money` decimal(8,2) NOT NULL DEFAULT '0.00' COMMENT '優惠分攤金額',
`w_id` int(10) unsigned NOT NULL COMMENT '倉庫ID',
`modified_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '最后修改時間',
PRIMARY KEY (`order_detail_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='訂單詳情表';
- 2.schema.xml將全局分片表進行ER分片,在父表中增加子表信息
<table name="order_master" primaryKey="order_id" autoIncrement="true" dataNode="dn_orderdb01,dn_orderdb02,dn_orderdb03,dn_orderdb04" rule="order_master">
<childTable name="order_detail" primaryKey="order_detail_id" joinKey="order_id" parentKey="order_id" autoIncrement="true" />
<!-- 指定關聯子表的表名、主鍵,及父表關聯key、父表的主鍵,同時也需要指定全局ID -->
</table>
- 3.配置表order_detail全局自增ID(使用數據庫方式)
# server.xml
# sequence_db_conf.properties增加order_detail自增ID
#sequence stored in datanode
GLOBAL=dn_test
ORDER_MASTER=dn_test
ORDER_DETAIL=dn_test
# 數據庫中增加分片表信息
insert into mycat_sequence values('ORDER_DETAIL',1,1);
- 4.插入數據
# 先插入order_master
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70001,90001,'yzw1',1,1,1,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70002,90002,'yzw2',2,2,2,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70003,90003,'yzw3',3,3,3,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70004,90004,'yzw4',4,4,4,'address1',1,20);
insert into order_master(order_sn,customer_id,shipping_user,province,city,district,address,payment_method,order_money) values(70005,90005,'yzw5',5,5,5,'address1',1,20);
app_imooc@172.16.10.142 14:46: [imooc_db]> select order_sn,customer_id,order_id from order_master;
+----------+-------------+----------+
| order_sn | customer_id | order_id |
+----------+-------------+----------+
| 70005 | 90005 | 5 |
| 70001 | 90001 | 6 |
| 70005 | 90005 | 10 |
| 70004 | 90004 | 4 |
| 70004 | 90004 | 9 |
| 70003 | 90003 | 3 |
| 70003 | 90003 | 8 |
| 70002 | 90002 | 2 |
| 70002 | 90002 | 7 |
+----------+-------------+----------+
9 rows in set (0.00 sec)
# 根據order_id插入order_detail
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,2,12,'商品2',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,3,13,'商品3',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,4,14,'商品4',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,5,15,'商品5',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,6,16,'商品6',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,7,17,'商品7',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,8,18,'商品8',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,9,19,'商品9',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,10,20,'商品10',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,5,21,'商品11',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,6,12,'商品2',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,8,12,'商品2',1,22.2,12,1,10,2);
insert into order_detail(order_detail_id,order_id,product_id,product_name,product_cnt,product_price,average_cost,weight,fee_money,w_id) values(next value for MYCATSEQ_ORDER_DETAIL,8,12,'商品2',1,22.2,12,1,10,2);
- 5.此時再進行分片關聯查詢驗證數據
select m.order_id,o.order_detail_id,o.product_id,o.product_name from order_master m join order_detail o on m.order_id=o.order_id;
+----------+-----------------+------------+--------------+
| order_id | order_detail_id | product_id | product_name |
+----------+-----------------+------------+--------------+
| 3 | 4 | 13 | 商品3 |
| 8 | 9 | 18 | 商品8 |
| 8 | 14 | 12 | 商品2 |
| 8 | 15 | 12 | 商品2 |
| 2 | 3 | 12 | 商品2 |
| 7 | 8 | 17 | 商品7 |
| 4 | 5 | 14 | 商品4 |
| 9 | 10 | 19 | 商品9 |
| 5 | 6 | 15 | 商品5 |
| 6 | 7 | 16 | 商品6 |
| 10 | 11 | 20 | 商品10 |
| 5 | 12 | 21 | 商品11 |
| 6 | 13 | 12 | 商品2 |
+----------+-----------------+------------+--------------+
13 rows in set (0.00 sec)
- 6.在其中一個分片上可以驗證,order_id一致的表落在同一個分片
root@localhost 15:05: [orderdb02]> select order_sn,customer_id,order_id,mod(customer_id,4) from order_master;
+----------+-------------+----------+
| order_sn | customer_id | order_id |
+----------+-------------+----------+
| 70005 | 90005 | 5 |
| 70001 | 90001 | 6 |
| 70005 | 90005 | 10 |
+----------+-------------+----------+
3 rows in set (0.00 sec)
root@localhost 15:06: [orderdb02]> select order_detail_id,product_id,order_id from order_detail;
+-----------------+------------+----------+
| order_detail_id | product_id | order_id |
+-----------------+------------+----------+
| 6 | 15 | 5 |
| 7 | 16 | 6 |
| 11 | 20 | 10 |
| 12 | 21 | 5 |
| 13 | 12 | 6 |
+-----------------+------------+----------+
5 rows in set (0.00 sec)
五、MyCat的其他常用功能
1、MyCat的SQL攔截器
SQL攔截是一個比較有用的高級技巧,用戶可以寫一個java類,將傳入MyCAT的SQL進行改寫然后交給Mycat去執行,此技巧可以完成如下一些特殊功能:
- 捕獲和記錄某些特殊的SQL
- 記錄sql查找異常
- 出于性能優化的考慮,改寫SQL,比如改變查詢條件的順序或增加分頁限制
- 將某些Select SQL強制設置為Read 模式,走讀寫分離(很多事務框架很難剝離事務中的Select SQL
- 后期Mycat智能優化,攔截所有sql 做智能分析,自動監控節點負載,自動優化路由,提供數據庫優化建議
SQL攔截的原理是在路由之前攔截SQL,然后做其他處理,完了之后再做路由,執行,如下圖所示:
默認的攔截器實現了Mysql轉義字符的過濾轉換,非默認攔截器只有一個攔截記錄sql的攔截器。
默認SQL攔截器:
配置:
<system>
<property name="sqlInterceptor">org.opencloudb.interceptor.impl.DefaultSqlInterceptor</property>
</system>
源碼:
/**
* escape mysql escape letter
*/
@Override
public String interceptSQL(String sql, int sqlType) {
if (sqlType == ServerParse.UPDATE || sqlType == ServerParse.INSERT||
sqlType == ServerParse.SELECT||sqlType == ServerParse.DELETE) {
return sql.replace("\\'", "''"); }
else {
return sql;
}
}
}
配置server.xml
# 1.5可用
<system>
<property name="sqlInterceptor">org.opencloudb.interceptor.impl.StatisticsSqlInterceptor</property>
<property name="sqlInterceptorType">UPDATE,DELETE,INSERT,SELECT</property>
<property name="sqlInterceptorFile">/tmp/sql.txt</property>
</system>
# 1.6可用
<system>
<property name="sqlInterceptor">io.mycat.server.interceptor.impl.StatisticsSqlInterceptor</property>
<!-- 1.6 io.mycat.server.interceptor.impl.StatisticsSqlInterceptor -->
<property name="sqlInterceptorType">UPDATE,DELETE,INSERT,SELECT</property>
<property name="sqlInterceptorFile">/tmp/sql.txt</property>
</system>
sqlInterceptorType: 攔截sql類型
sqlInterceptorFile: sql保存文件路徑
注意:捕獲記錄sql攔截器的配置只有1.4及其以后可用,1.3無本攔截。
如果需要實現自己的sql攔截,只需要將配置類改為自己配置即可:
- 1、定義自定義類 implements SQLInterceptor ,然后改寫sql后返回。
- 2、將自己實現的類放入catlet 目錄,可以為class或jar。
- 3、配置配置文件:
<system>
<property name="sqlInterceptor">org.opencloudb.interceptor.impl.自定義class</property>
<!--其他配置-->
</system>
測試
# mycat restart
app_imooc@172.16.10.142 15:39: [imooc_db]> select order_sn,order_id,customer_id from order_master limit 1;
+----------+----------+-------------+
| order_sn | order_id | customer_id |
+----------+----------+-------------+
| 70005 | 5 | 90005 |
+----------+----------+-------------+
1 row in set (0.01 sec)
app_imooc@172.16.10.142 15:40: [imooc_db]> delete from order_master where order_id=5;
Query OK, 1 row affected (0.03 sec)
文件名會帶日期
sql2020-01-19.txt
2、mycat sql防火墻
- 統一控制哪些用戶可以通過哪些主機訪問后端數據庫
- 統一屏蔽一些SQL語句,加強安全控制(如沒有條件的delete語句等)
server.xml文件
- firewall標簽用來定義防火墻
- firewall下whitehost標簽用來定義 IP白名單
- blacklist用來定義 SQL黑名單
<firewall>
<whitehost>
<host user="mycat" host="127.0.0.1"></host> #ip 白名單 用戶對應的可以訪問的 ip 地址
</whitehost>
<blacklist check="true">
<property name="selelctAllow">false</property> #黑名單允許的 權限 后面為默認
</blacklist>
</firewall>
黑名單攔截明細配置
配置項 | 缺省值 | 描述 | |
---|---|---|---|
selelctAllow | true | 是否允許執行 SELECT 語句 | |
selectAllColumnAllow | true | 是否允許執行 SELECT * FROM T 這樣的語句。如果設置為 false,不允許執行 select * from t,但可以select * from (select id, name from t) a。這個選項是防御程序通過調用 select * 獲得數據表的結構信息。 | |
selectIntoAllow | true | SELECT 查詢中是否允許 INTO 字句 | |
deleteAllow | true | 是否允許執行 DELETE 語句 | |
updateAllow | true | 是否允許執行 UPDATE 語句 | |
insertAllow | true | 是否允許執行 INSERT 語句 | |
replaceAllow | true | 是否允許執行 REPLACE 語句 | |
mergeAllow | true | 是否允許執行 MERGE 語句,這個只在 Oracle 中有用 | |
callAllow | true | 是否允許通過 jdbc 的 call 語法調用存儲過程 | |
setAllow | true | 是否允許使用 SET 語法 | |
truncateAllow | true | truncate 語句是危險,缺省打開,若需要自行關閉 | |
createTableAllow | true | 是否允許創建表 | |
alterTableAllow | true | 是否允許執行 Alter Table 語句 | |
dropTableAllow | true | 是否允許修改表 | |
commentAllow | false | 是否允許語句中存在注釋,Oracle 的用戶不用擔心,Wall 能夠識別 hints和注釋的區別 | |
noneBaseStatementAllow | false | 是否允許非以上基本語句的其他語句,缺省關閉,通過這個選項就能夠屏蔽 DDL | |
multiStatementAllow | false | 是否允許一次執行多條語句,缺省關閉 | |
useAllow | true | 是否允許執行 mysql 的 use 語句,缺省打開 | |
describeAllow | true | 是否允許執行 mysql 的 describe 語句,缺省打開 | |
showAllow | true | 是否允許執行 mysql 的 show 語句,缺省打開 | |
commitAllow | true | 是否允許執行 commit 操作 | |
rollbackAllow | true | 是否允許執行 roll back 操作 | |
##如果把 selectIntoAllow、deleteAllow、updateAllow、insertAllow、mergeAllow 都設置為 false,這就是一個只讀數據源了。## | |||
攔截配置-永真條件 | |||
selectWhereAlwayTrueCheck | true | 檢查 SELECT 語句的 WHERE 子句是否是一個永真條件 | |
selectHavingAlwayTrueCheck | true | 檢查 SELECT 語句的 HAVING 子句是否是一個永真條件 | |
deleteWhereAlwayTrueCheck | true | 檢查 DELETE 語句的 WHERE 子句是否是一個永真條件 | |
deleteWhereNoneCheck | false | 檢查 DELETE 語句是否無 where 條件,這是有風險的,但不是 SQL 注入類型的風險 | |
updateWhereAlayTrueCheck | true | 檢查 UPDATE 語句的 WHERE 子句是否是一個永真條件 | |
updateWhereNoneCheck | false | 檢查 UPDATE 語句是否無 where 條件,這是有風險的,但不是SQL 注入類型的風險 | |
conditionAndAlwayTrueAllow | false | 檢查查詢條件(WHERE/HAVING 子句)中是否包含 AND 永真條件 | |
conditionAndAlwayFalseAllow | false | 檢查查詢條件(WHERE/HAVING 子句)中是否包含 AND 永假條件 | |
conditionLikeTrueAllow | true | 檢查查詢條件(WHERE/HAVING 子句)中是否包含 LIKE 永真條件 | |
其他攔截配置 | |||
selectIntoOutfileAllow | false | SELECT ... INTO OUTFILE 是否允許,這個是 mysql 注入攻擊的常見手段,缺省是禁止的 | |
selectUnionCheck | true | 檢測 SELECT UNION | |
selectMinusCheck | true | 檢測 SELECT MINUS | |
selectExceptCheck | true | 檢測 SELECT EXCEPT | |
selectIntersectCheck | true | 檢測 SELECT INTERSECT | |
mustParameterized | false | 是否必須參數化,如果為 True,則不允許類似 WHERE ID = 1 這種不參數化的 SQL | |
strictSyntaxCheck | true | 是否進行嚴格的語法檢測,Druid SQL Parser 在某些場景不能覆蓋所有的SQL 語法,出現解析 SQL 出錯,可以臨時把這個選項設置為 false,同時把 SQL 反饋給 Druid 的開發者。 | |
conditionOpXorAllow | false | 查詢條件中是否允許有 XOR 條件。XOR 不常用,很難判斷永真或者永假,缺省不允許。 | |
conditionOpBitwseAllow | true | 查詢條件中是否允許有"&"、"~"、" | "、"^"運算符。 |
conditionDoubleConstAllow | false | 查詢條件中是否允許連續兩個常量運算表達式 | |
minusAllow | true | 是否允許 SELECT * FROM A MINUS SELECT * FROM B 這樣的語句 | |
intersectAllow | true | 是否允許 SELECT * FROM A INTERSECT SELECT * FROM B 這樣的語句 | |
constArithmeticAllow | true | 攔截常量運算的條件,比如說 WHERE FID = 3 - 1,其中"3 - 1"是常量運算表達式。 | |
limitZeroAllow | false | 是否允許 limit 0 這樣的語句 | |
禁用對象檢測配置 | |||
tableCheck | true | 檢測是否使用了禁用的表 | |
schemaCheck | true | 檢測是否使用了禁用的 Schema | |
functionCheck | true | 檢測是否使用了禁用的函數 | |
objectCheck | true | 檢測是否使用了“禁用對對象” | |
variantCheck | true | 檢測是否使用了“禁用的變量” | |
readOnlyTables | 空 | 指定的表只讀,不能夠在 SELECT INTO、DELETE、UPDATE、INSERT、MERGE 中作為"被修改表"出現 |
參考:
https://www.cnblogs.com/jenvid/p/10180461.html
https://blog.51cto.com/5660061/2391986?source=dra