近期在linux上搭建了用于分布式存儲(chǔ)的----GlusterFS和Ceph這兩個(gè)開源的分布式文件系統(tǒng)。
前言----大家可以去github上搜索一下，看源碼或者官方文檔介紹，更多的去了解Ceph，在這里我就不一一的去介紹原理以及抽象技術(shù)層面的基礎(chǔ)知識(shí)。下面我就搭建部署過(guò)程中遇到的問(wèn)題，向大家做一個(gè)介紹及部署過(guò)程的詳細(xì)流程。同時(shí)，也希望研究或者喜好這方面的同學(xué)，帶有生產(chǎn)環(huán)境成熟方案或者意見的，也請(qǐng)?jiān)谶@里互相討論，我們一起共同進(jìn)步，感興趣的同學(xué)可以加我微信(請(qǐng)備注請(qǐng)求與來(lái)源)，互相溝通交流。
下面開始Ceph的詳細(xì)介紹：遇到的問(wèn)題也會(huì)在過(guò)程中解決，并且特別說(shuō)明的是，安裝中沒有特別說(shuō)明的，都是安裝在同一臺(tái)admin-node上的配置。
1.首先要查看你的網(wǎng)絡(luò)是否與外網(wǎng)連通，好多都是虛擬機(jī)，需要配置代理，代理一定要配置在
/etc/environment 中，否則不起作用。或者在yum.conf中配置代理也可以。我選在/etc/environment中配置。
2.查看host及hostname名稱
[root@vm-10-112-178-135 gadmin]# vi /etc/hosts
[root@vm-10-112-178-135 gadmin]# vi /etc/hostname
3.安裝epel
[root@vm-10-112-178-135 gadmin]# yum install -y epel-release
已加載插件：fastestmirrorLoading mirror speeds from cached hostfile正在解決依賴關(guān)系--> 正在檢查事務(wù)---> 軟件包 epel-release.noarch.0.7-9 將被 安裝--> 解決依賴關(guān)系完成依賴關(guān)系解決============================================================================================================================================================================================================================================ Package? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 架構(gòu)? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 版本? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 源? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 大小============================================================================================================================================================================================================================================正在安裝: epel-release? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? noarch? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 7-9? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? epel? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 14 k事務(wù)概要============================================================================================================================================================================================================================================安裝? 1 軟件包總下載量：14 k安裝大小：24 kDownloading packages:警告：/var/cache/yum/x86_64/7/epel/packages/epel-release-7-9.noarch.rpm: 頭V3 RSA/SHA256 Signature, 密鑰 ID 352c64e5: NOKEYepel-release-7-9.noarch.rpm 的公鑰尚未安裝epel-release-7-9.noarch.rpm? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? |? 14 kB? 00:00:00從 http://file.idc.pub/os/epel/RPM-GPG-KEY-EPEL-7 檢索密鑰導(dǎo)入 GPG key 0x352C64E5: 用戶ID? ? : "Fedora EPEL (7)"
指紋? ? ? : 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
來(lái)自? ? ? : http://file.idc.pub/os/epel/RPM-GPG-KEY-EPEL-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安裝? ? : epel-release-7-9.noarch? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 驗(yàn)證中? ? ? : epel-release-7-9.noarch? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1/1
已安裝:
epel-release.noarch 0:7-9
完畢！
[root@vm-10-112-178-135 gadmin]# yum info epel-release
已加載插件：fastestmirror
Repository epel-testing is listed more than once in the configuration
Repository epel-testing-debuginfo is listed more than once in the configuration
Repository epel-testing-source is listed more than once in the configuration
Repository epel is listed more than once in the configuration
Repository epel-debuginfo is listed more than once in the configuration
Repository epel-source is listed more than once in the configuration
Loading mirror speeds from cached hostfile
已安裝的軟件包
名稱? ? ：epel-release
架構(gòu)? ? ：noarch
版本? ? ：7
發(fā)布? ? ：9
大小? ? ：24 k
源? ? ：installed
來(lái)自源：epel
簡(jiǎn)介? ? ： Extra Packages for Enterprise Linux repository configuration
網(wǎng)址? ? ：http://download.fedoraproject.org/pub/epel
協(xié)議? ? ： GPLv2
描述? ? ： This package contains the Extra Packages for Enterprise Linux (EPEL) repository
: GPG key as well as configuration for yum.
4.手動(dòng)添加ceph.repo 的 yum資源庫(kù)文件，下載ceph包文件從這里面下載
[root@vm-10-112-178-135 gadmin]# vi /etc/yum.repos.d/ceph.repo
文件內(nèi)容，將以下文件內(nèi)容復(fù)制到文件中：
[ceph]
name=Ceph packages for $basearch
baseurl=http://download.ceph.com/rpm-jewel/el7/$basearch
enabled=1
priority=1
gpgcheck=1
type=rpm-md
gpgkey=http://download.ceph.com/keys/release.asc
[ceph-noarch]
name=Ceph noarch packages
baseurl=http://download.ceph.com/rpm-jewel/el7/noarch
enabled=1
priority=1
gpgcheck=1
type=rpm-md
gpgkey=http://download.ceph.com/keys/release.asc
[ceph-x86_64]
name=Ceph x86_64 packages
baseurl=http://download.ceph.com/rpm-jewel/el7/x86_64
enabled=0
priority=1
gpgcheck=1
type=rpm-md
gpgkey=http://download.ceph.com/keys/release.asc
[ceph-aarch64]
name=Ceph source packages
baseurl=http://download.ceph.com/rpm-jewel/el7/aarch64
enabled=0
priority=1
gpgcheck=1
type=rpm-md
gpgkey=http://download.ceph.com/keys/release.asc
[ceph-source]
name=Ceph source packages
baseurl=http://download.ceph.com/rpm-jewel/el7/SRPMS
enabled=0
priority=1
gpgcheck=1
type=rpm-md
gpgkey=http://download.ceph.com/keys/release.asc
[apache2-ceph-noarch]
name=Apache noarch packages for Ceph
baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/master/SRPMS
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=http://download.ceph.com/keys/autobuild.asc
[apache2-ceph-source]
name=Apache source packages for Ceph
baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/master/SRPMS
enabled=0
priority=2
gpgcheck=1
type=rpm-md
gpgkey=http://download.ceph.com/keys/autobuild.asc
5.安裝ceph-deploy工具
該工具是python寫的一個(gè)官方的ceph集群安裝工具。
[root@vm-10-112-178-135 gadmin]#sudo yum update && sudo yum install ceph-deploy
6.查看是否安裝了NTP服務(wù)，用來(lái)校準(zhǔn)時(shí)間，因?yàn)槊颗_(tái)機(jī)器通信，保證時(shí)間統(tǒng)一
7.安裝openssh-server，配置ssh免密碼連接。注意是在所有節(jié)點(diǎn)上。
在所有的節(jié)點(diǎn)上安裝openssh-server
[root@vm-10-112-178-135 gadmin]# sudo yum install openssh-server
[root@vm-10-112-178-135 gadmin]#
添加用戶名
[root@vm-10-112-178-135 gadmin]# sudo useradd -d /home/cephadmin -m cephadmin
[root@vm-10-112-178-135 gadmin]# passwd cephadmin
設(shè)置用戶擁有root權(quán)限的免密碼權(quán)限
[root@vm-10-112-178-135 gadmin]# echo "cephadmin ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadmin
cephadmin ALL = (root) NOPASSWD:ALL
[root@vm-10-112-178-135 gadmin]# sudo chmod 0440 /etc/sudoers.d/cephadmin
[cephadmin@vm-10-112-178-135 ~]$ ssh-keygen
[cephadmin@vm-10-112-178-135 .ssh]$ cd ~/.ssh/
[cephadmin@vm-10-112-178-135 .ssh]$ ll
總用量 8
-rw------- 1 cephadmin cephadmin 1679 7月? 5 20:19 id_rsa
-rw-r--r-- 1 cephadmin cephadmin? 409 7月? 5 20:19 id_rsa.pub
[cephadmin@vm-10-112-178-135 .ssh]$ vi config
Copy the key to each Ceph Node, replacing {username} with the user name you created with Create a Ceph Deploy User.
ssh-copy-id {username}@node1
ssh-copy-id {username}@node2
ssh-copy-id {username}@node3
(Recommended) Modify the ~/.ssh/config file of your ceph-deploy admin node so that ceph-deploy can log in to Ceph nodes as the user you created without requiring you to specify --username {username} each time you execute ceph-deploy. This has the added benefit of streamlining ssh and scp usage. Replace {username} with the user name you created:
Host node1
Hostname node1
User {username}
Host node2
Hostname node2
User {username}
Host node3
Hostname node3
User {username}
Enable Networking On Bootup
Ceph OSDs peer with each other and report to Ceph Monitors over the network. If networking is off by default, the Ceph cluster cannot come online during bootup until you enable networking.
The default configuration on some distributions (e.g., CentOS) has the networking interface(s) off by default. Ensure that, during boot up, your network interface(s) turn(s) on so that your Ceph daemons can communicate over the network. For example, on Red Hat and CentOS, navigate to /etc/sysconfig/network-scripts and ensure that the ifcfg-{iface} file has ONBOOT set to yes.
[cephadmin@vm-10-112-178-135 my-cluster]$ cd ~/.ssh/
[cephadmin@vm-10-112-178-135 .ssh]$ ll
總用量 12
-rw-rw-r-- 1 cephadmin cephadmin? 171 7月? 5 20:26 config
-rw------- 1 cephadmin cephadmin 1679 7月? 5 20:19 id_rsa
-rw-r--r-- 1 cephadmin cephadmin? 409 7月? 5 20:19 id_rsa.pub
[cephadmin@vm-10-112-178-135 .ssh]$ vi config
[cephadmin@vm-10-112-178-135 .ssh]$ cat config
Host 10.112.178.141
Hostname vm-10-112-178-141
User cephadmin
Host 10.112.178.142
Hostname vm-10-112-178-142
User cephadmin
Host 10.112.178.143
Hostname vm-10-112-178-143
User cephadmin
[cephadmin@vm-10-112-178-135 .ssh]$
[cephadmin@vm-10-112-178-135 .ssh]$ sudo ssh-copy-id -i ~/.ssh/id_rsa.pub? cephadmin@vm-10-112-178-141
The authenticity of host 'vm-10-112-178-141 (10.112.178.141)' can't be established.
ECDSA key fingerprint is af:7a:19:1d:76:b7:9b:51:0c:88:3a:4c:33:d0:f8:a5.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cephadmin@vm-10-112-178-141's password:
Number of key(s) added: 1
Now try logging into the machine, with:? "ssh 'cephadmin@vm-10-112-178-141'"
and check to make sure that only the key(s) you wanted were added.
[cephadmin@vm-10-112-178-135 .ssh]$ sudo ssh-copy-id -i ~/.ssh/id_rsa.pub? cephadmin@vm-10-112-178-142
The authenticity of host 'vm-10-112-178-142 (10.112.178.142)' can't be established.
ECDSA key fingerprint is af:7a:19:1d:76:b7:9b:51:0c:88:3a:4c:33:d0:f8:a5.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cephadmin@vm-10-112-178-142's password:
Number of key(s) added: 1
Now try logging into the machine, with:? "ssh 'cephadmin@vm-10-112-178-142'"
and check to make sure that only the key(s) you wanted were added.
[cephadmin@vm-10-112-178-135 .ssh]$ sudo ssh-copy-id -i ~/.ssh/id_rsa.pub? cephadmin@vm-10-112-178-143
The authenticity of host 'vm-10-112-178-143 (10.112.178.143)' can't be established.
ECDSA key fingerprint is af:7a:19:1d:76:b7:9b:51:0c:88:3a:4c:33:d0:f8:a5.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cephadmin@vm-10-112-178-143's password:
Number of key(s) added: 1
Now try logging into the machine, with:? "ssh 'cephadmin@vm-10-112-178-143'"
and check to make sure that only the key(s) you wanted were added.
[cephadmin@vm-10-112-178-135 .ssh]$ ssh cephadmin@10.112.178.141
Bad owner or permissions on /home/cephadmin/.ssh/config
[cephadmin@vm-10-112-178-135 .ssh]$ sudo firewall-cmd --zone=public --add-service=ceph-mon --permanent
FirewallD is not running
說(shuō)明防火墻已經(jīng)關(guān)閉了，這樣就不用考率防火墻了。
[cephadmin@vm-10-112-178-135 ~]$ mkdir my-cluster
[cephadmin@vm-10-112-178-135 ~]$ cd my-cluster/
[cephadmin@vm-10-112-178-135 my-cluster]$
[cephadmin@vm-10-112-178-135 my-cluster]$ ceph-deploy new vm-10-112-178-135
[vm-10-112-178-135][DEBUG ] 總下載量：59 M[vm-10-112-178-135][DEBUG ] 安裝大小：218 M[vm-10-112-178-135][DEBUG ] Downloading packages:[vm-10-112-178-135][WARNIN] No data was received after 300 seconds, disconnecting...[vm-10-112-178-135][INFO? ] Running command: sudo ceph --version[vm-10-112-178-135][ERROR ] Traceback (most recent call last):[vm-10-112-178-135][ERROR ]? File "/usr/lib/python2.7/site-packages/ceph_deploy/lib/vendor/remoto/process.py", line 119, in run[vm-10-112-178-135][ERROR ]? ? reporting(conn, result, timeout)[vm-10-112-178-135][ERROR ]? File "/usr/lib/python2.7/site-packages/ceph_deploy/lib/vendor/remoto/log.py", line 13, in reporting[vm-10-112-178-135][ERROR ]? ? received = result.receive(timeout)[vm-10-112-178-135][ERROR ]? File "/usr/lib/python2.7/site-packages/ceph_deploy/lib/vendor/remoto/lib/vendor/execnet/gateway_base.py", line 704, in receive[vm-10-112-178-135][ERROR ]? ? raise self._getremoteerror() or EOFError()[vm-10-112-178-135][ERROR ] RemoteError: Traceback (most recent call last):[vm-10-112-178-135][ERROR ]? File "/usr/lib/python2.7/site-packages/ceph_deploy/lib/vendor/remoto/lib/vendor/execnet/gateway_base.py", line 1036, in executetask[vm-10-112-178-135][ERROR ]? ? function(channel, **kwargs)[vm-10-112-178-135][ERROR ]? File "", line 12, in _remote_run
[vm-10-112-178-135][ERROR ]? File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
[vm-10-112-178-135][ERROR ]? ? errread, errwrite)
[vm-10-112-178-135][ERROR ]? File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
[vm-10-112-178-135][ERROR ]? ? raise child_exception
[vm-10-112-178-135][ERROR ] OSError: [Errno 2] No such file or directory
[vm-10-112-178-135][ERROR ]
[vm-10-112-178-135][ERROR ]
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph --version
原因是網(wǎng)絡(luò)比較慢,達(dá)到5分鐘超時(shí)
解決方案:1.可以在每個(gè)節(jié)點(diǎn)上先行安裝sudo yum -y install ceph
2.數(shù)量比較多的話多執(zhí)行幾次此命令
3.最佳方案是搭建本地源
[cephadmin@vm-10-112-178-135 my-cluster]$ sudo yum install -y ceph
[cephadmin@vm-10-112-178-135 my-cluster]$ sudo yum -y install ceph-radosgw
[ceph_deploy.cli][INFO? ]? dev? ? ? ? ? ? ? ? ? ? ? ? ? : master
[ceph_deploy.cli][INFO? ]? nogpgcheck? ? ? ? ? ? ? ? ? ? : False
[ceph_deploy.cli][INFO? ]? local_mirror? ? ? ? ? ? ? ? ? : None
[ceph_deploy.cli][INFO? ]? release? ? ? ? ? ? ? ? ? ? ? : None
[ceph_deploy.cli][INFO? ]? install_mon? ? ? ? ? ? ? ? ? : False
[ceph_deploy.cli][INFO? ]? gpg_url? ? ? ? ? ? ? ? ? ? ? : None
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts vm-10-112-178-141
[ceph_deploy.install][DEBUG ] Detecting platform for host vm-10-112-178-141 ...
[vm-10-112-178-141][DEBUG ] connection detected need for sudo
sudo：抱歉，您必須擁有一個(gè)終端來(lái)執(zhí)行 sudo
[ceph_deploy][ERROR ] RuntimeError: connecting to host: vm-10-112-178-141 resulted in errors: IOError cannot send (already closed?)
[cephadmin@vm-10-112-178-135 my-cluster]$
解決方案：
[摘要] Linux ssh執(zhí)行遠(yuǎn)端服務(wù)器sudo命令時(shí)有如下報(bào)錯(cuò)：
sudo: sorry, you must have a tty to run sudo
sudo：抱歉，您必須擁有一個(gè)終端來(lái)執(zhí)行 sudo
網(wǎng)上搜了一下，解決辦法是編輯 /etc/sudoers 文件，將Default requiretty注釋掉。
sudo vi /etc/sudoers
#Default requiretty #注釋掉 Default requiretty 一行
具體操作：
sudo sed -i 's/Defaults? ? requiretty/#Defaults? ? requiretty/g' /etc/sudoers
sudo cat /etc/sudoers | grep requiretty
-----由于http代理網(wǎng)速太慢，只能選擇分別再每臺(tái)虛擬機(jī)節(jié)點(diǎn)上 手動(dòng)安裝 ceph，安裝指令如下----
sudo yum -y install ceph
sudo yum -y install ceph-radosgw
[cephadmin@vm-10-112-178-135 my-cluster]$ ceph-deploy install vm-10-112-178-135 vm-10-112-178-141 vm-10-112-178-142 vm-10-112-178-143
ceph-deploy install {ceph-node}[{ceph-node} ...] --local-mirror=/opt/ceph-repo --no-adjust-repos --release=jewel
[cephadmin@vm-10-112-178-135 my-cluster]$ ceph-deploy osd prepare vm-10-112-178-142:/var/local/osd0 vm-10-112-178-143:/var/local/osd1
[vm-10-112-178-142][WARNIN] 2017-07-06 16:19:33.928816 7f2940a11800 -1? ** ERROR: error creating empty object store in /var/local/osd0: (13) Permission denied
[vm-10-112-178-142][WARNIN]
[vm-10-112-178-142][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /var/local/osd0
解決方案：
因?yàn)闄C(jī)器上的 /var/local/osd0 與 /var/local/osd1 這兩個(gè)文件夾沒有權(quán)限，使用 chmod 777 對(duì)文件夾 進(jìn)行賦予 權(quán)限。
[cephadmin@vm-10-112-178-135 my-cluster]$ ceph-deploy osd activate vm-10-112-178-142:/var/local/osd0 vm-10-112-178-143:/var/local/osd1
