Centos7部署etcd静态集群

因为etcd新版本和旧版本在使用的命令上有写不同，为了验证etcd的功能，我这里就先安装旧版本的etcd，在下一个章节的etcd界面可视化的时候会针对etcd新版本的使用，做进一步的说明。先安装etcd，每个节点上都同样的部署，这里就值写一个节点的部署方法，其他两个节点的部署方法一样：

安装节点

下载etcd文件，并且解压后方到指定的路径下。

[root@www.wulaoer.org ~]#  wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
[root@www.wulaoer.org ~]# tar -zxf etcd-v3.3.10-linux-amd64.tar.gz 
[root@www.wulaoer.org ~]# mv etcd-v3.3.10-linux-amd64 /usr/local/etcd
[root@www.wulaoer.org ~]# cd /usr/local/etcd/

配置etcd信息

这里配置etcd的配置信息，以及etcd的数据存储位置

[root@www.wulaoer.org etcd]# cp etcd* /usr/local/bin/
[root@www.wulaoer.org ~]# mkdir /usr/local/etcd/{data,config}
[root@www.wulaoer.org ~]# cat <<EOF | tee /usr/local/etcd/config/etcd.conf
> #节点名称
> ETCD_NAME=$(hostname -s)
> #数据存放位置
> ETCD_DATA_DIR=/usr/local/etcd/data
> EOF

设置etcd的开机自启动信息，使用systemctl启动etcd服务

[root@www.wulaoer.org ~]# cat <<EOF | tee /etc/systemd/system/etcd.service
> [Unit]
> Description=Etcd Server
> Documention=https://github.com/coreos/etcd
> After=network.target
> [Service]
> User=root
> Type=notify
> EnvironmentFile=/usr/local/etcd/config/etcd.conf
> ExecStart=/usr/local/etcd/data
> RestartSec=10s
> LimitNOFILE=40000
> [Install]
> WantedBy=multi-user.target
> EOF
[Unit]
Description=Etcd Server
Documention=https://github.com/coreos/etcd
After=network.target
[Service]
User=root
Type=notify
EnvironmentFile=/usr/local/etcd/config/etcd.conf
ExecStart=/usr/local/etcd/etcd
RestartSec=10s
LimitNOFILE=40000
[Install]
WantedBy=multi-user.target

按照以上方法，在另外两个节点上安装，不需要修改任何地方，配置后，按照下面的方法进行重启。

启动etcd服务

使用systemctl启动etcd服务，没有报错。不过注意，如果配置好集群之后需要把数据删除，如果不删除会出现所有节点都是集群。

[root@www.wulaoer.org ~]# systemctl daemon-reload && systemctl enable etcd && systemctl start etcd
[root@wulaoer etcd]# systemctl daemon-reload && systemctl enable etcd && systemctl start etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /etc/systemd/system/etcd.service.
[root@wolf etcd]# systemctl daemon-reload && systemctl enable etcd && systemctl start etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /etc/systemd/system/etcd.service.

所有节点已经配置好，并且已经启动没有错误，如果启动出现问题可以根据下面的查看日志的方法，解决问题。

查看etcd日志

[root@www.wulaoer.org ~]# journalctl -u etcd

我们可以根据日志查看etcd启动失败的原因，然后解决问题。

配置静态集群

所有节点已经配置好etcd，并且启动了服务，下面开始配置etcd配置信息和启动etcd信息，节点www.wulaoer.org的配置。

[root@www.wulaoer.org ~]# vim /usr/local/etcd/config/etcd.conf 
#节点名称
ETCD_NAME=www.wulaoer.org
#数据存放位置
ETCD_DATA_DIR=/usr/local/etcd/data
#集群信息
ETCD_LISTEN_PEER_URLS="http://192.168.111.128:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.111.128:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.111.128:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.111.128:2379"
ETCD_INITIAL_CLUSTER="www.wulaoer.org=http://192.168.111.128:2380,wulaoer=http://192.168.111.129:2380,wolf=http://192.168.111.130:2380"   #这里的www.wulaoer.org，wulaoer，wolf是节点名称需要和上面的匹配
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
[root@www.wulaoer.org ~]# vim /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
Documentation=https://github.com/coreos/etcd
After=network.target
[Service]
User=root
Type=notify
WorkingDirectory=/usr/local/etcd/
EnvironmentFile=-/usr/local/etcd/config/etcd.conf
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/etcd/etcd \
--name=\"${ETCD_NAME}\" \
--data-dir=\"${ETCD_DATA_DIR}\" \
--listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \
--listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" \
--initial-advertise-peer-urls=\"${ETCD_INITIAL_ADVERTISE_PEER_URLS}\" \
--advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \
--initial-cluster=\"${ETCD_INITIAL_CLUSTER}\"  \
--initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \
--initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\""
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000
[Install]
WantedBy=multi-user.target

节点wulaoer的配置信息和启动信息

[root@wulaoer ~]# cat /usr/local/etcd/config/etcd.conf 
#节点名称
ETCD_NAME=wulaoer
#数据存放位置
ETCD_DATA_DIR=/usr/local/etcd/data
#集群信息
ETCD_LISTEN_PEER_URLS="http://192.168.111.129:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.111.129:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.111.129:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.111.129:2379"
ETCD_INITIAL_CLUSTER="www.wulaoer.org=http://192.168.111.128:2380,wulaoer=http://192.168.111.129:2380,wolf=http://192.168.111.130:2380"  #这里的www.wulaoer.org，wulaoer，wolf是节点名称需要和上面的匹配
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
[root@wulaoer ~]# cat /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
Documentation=https://github.com/coreos/etcd
After=network.target
[Service]
User=root
Type=notify
WorkingDirectory=/usr/local/etcd/
EnvironmentFile=-/usr/local/etcd/config/etcd.conf
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \
--name=\"${ETCD_NAME}\" \
--data-dir=\"${ETCD_DATA_DIR}\" \
--listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \
--listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" \
--initial-advertise-peer-urls=\"${ETCD_INITIAL_ADVERTISE_PEER_URLS}\" \
--advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \
--initial-cluster=\"${ETCD_INITIAL_CLUSTER}\"  \
--initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \
--initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\""
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000
[Install]
WantedBy=multi-user.target

节点wolf的配置信息和启动信息

[root@wolf ~]# cat /usr/local/etcd/config/etcd.conf 
#节点名称
ETCD_NAME=wolf
#数据存放位置
ETCD_DATA_DIR=/usr/local/etcd/data
#集群信息
ETCD_LISTEN_PEER_URLS="http://192.168.111.130:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.111.130:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.111.130:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.111.130:2379"
ETCD_INITIAL_CLUSTER="www.wulaoer.org=http://192.168.111.128:2380,wulaoer=http://192.168.111.129:2380,wolf=http://192.168.111.130:2380"  #这里的www.wulaoer.org，wulaoer，wolf是节点名称需要和上面的匹配
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

[root@wolf ~]# cat /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
Documentation=https://github.com/coreos/etcd
After=network.target
[Service]
User=root
Type=notify
WorkingDirectory=/usr/local/etcd/
EnvironmentFile=-/usr/local/etcd/config/etcd.conf
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \
--name=\"${ETCD_NAME}\" \
--data-dir=\"${ETCD_DATA_DIR}\" \
--listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \
--listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" \
--initial-advertise-peer-urls=\"${ETCD_INITIAL_ADVERTISE_PEER_URLS}\" \
--advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \
--initial-cluster=\"${ETCD_INITIAL_CLUSTER}\"  \
--initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \
--initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\""
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000
[Install]
WantedBy=multi-user.target

所有节点信息配置之后需要启动一下所有节点，安装下面的顺序，先关闭防火墙在启动etcd服务

[root@www.wulaoer.org ~]# systemctl stop firewalld
[root@www.wulaoer.org ~]# systemctl disable firewalld
[root@www.wulaoer.org ~]# systemctl daemon-reload
[root@www.wulaoer.org ~]# systemctl restart etcd.service
[root@www.wulaoer.org ~]# systemctl enable etcd.service

启动之后，为了避免节点时间统一，所以我们需要做一个定时任务，统一时间

[root@www.wulaoer.org ~]# crontab -l
0 1 * * * /usr/sbin/ntpdate ntp.sjtu.edu.cn >> /var/log/ntpdate.log 2>&1 &
[root@www.wulaoer.org ~]# systemctl reload crond.service

验证etcd集群

所有节点的etcd服务已经启动了，也做了时间同步，我们看一下我们的etcd集群是否创建成功，先检查一下etcd的状态。

[root@www.wulaoer.org ~]# etcdctl cluster-health
member 25f2724acdf5b013 is healthy: got healthy result from http://192.168.111.128:2379
member d3ddf28b4a611036 is healthy: got healthy result from http://192.168.111.130:2379
member ebecc1c831d00be2 is healthy: got healthy result from http://192.168.111.129:2379
cluster is healthy

也可以这样

[root@www.wulaoer.org ~]#  ETCDCTL_API=3 etcdctl --endpoints=http://192.168.111.128:2379,http://192.168.111.130:2379,http://192.168.111.129:2379 endpoint health
http://192.168.111.130:2379 is healthy: successfully committed proposal: took = 1.307688ms
http://192.168.111.129:2379 is healthy: successfully committed proposal: took = 2.050772ms
http://192.168.111.128:2379 is healthy: successfully committed proposal: took = 2.464973ms

或者

[root@www.wulaoer.org ~]# etcdctl --endpoints=http://192.168.111.128:2379,http://192.168.111.130:2379,http://192.168.111.129:2379 cluster-health
member 25f2724acdf5b013 is healthy: got healthy result from http://192.168.111.128:2379
member d3ddf28b4a611036 is healthy: got healthy result from http://192.168.111.130:2379
member ebecc1c831d00be2 is healthy: got healthy result from http://192.168.111.129:2379
cluster is healthy

查看一下集群中的leader选举是否已经产出

[root@www.wulaoer.org ~]# etcdctl member list
25f2724acdf5b013: name=www.wulaoer.org peerURLs=http://192.168.111.128:2380 clientURLs=http://192.168.111.128:2379 isLeader=true
d3ddf28b4a611036: name=wolf peerURLs=http://192.168.111.130:2380 clientURLs=http://192.168.111.130:2379 isLeader=false
ebecc1c831d00be2: name=wulaoer peerURLs=http://192.168.111.129:2380 clientURLs=http://192.168.111.129:2379 isLeader=false

第一个已经是leader了，我们做一个验证把www.wulaoer.org服务停掉，看一下集群是否重新选举leader。

[root@www.wulaoer.org ~]# systemctl stop etcd
[root@www.wulaoer.org ~]# systemctl start etcd
[root@www.wulaoer.org ~]# etcdctl member list
25f2724acdf5b013: name=www.wulaoer.org peerURLs=http://192.168.111.128:2380 clientURLs=http://192.168.111.128:2379 isLeader=false
d3ddf28b4a611036: name=wolf peerURLs=http://192.168.111.130:2380 clientURLs=http://192.168.111.130:2379 isLeader=false
ebecc1c831d00be2: name=wulaoer peerURLs=http://192.168.111.129:2380 clientURLs=http://192.168.111.129:2379 isLeader=true

停掉www.wulaoer.org节点的etcd服务之后，立刻有wulaoer节点的etcd担任了leader，服务管理其他服务的数据。在www.wulaoer.org停掉服务的过程中，在其他节点上检查服务的状态就会出现错误。

[root@wulaoer ~]# etcdctl cluster-health
failed to check the health of member 25f2724acdf5b013 on http://192.168.111.128:2379: Get http://192.168.111.128:2379/health: dial tcp 192.168.111.128:2379: connect: connection refused
member 25f2724acdf5b013 is unreachable: [http://192.168.111.128:2379] are all unreachable
member d3ddf28b4a611036 is healthy: got healthy result from http://192.168.111.130:2379
member ebecc1c831d00be2 is healthy: got healthy result from http://192.168.111.129:2379
cluster is degraded

我们测试一下数据的安全问题，我现在www.wulaoer.org节点上创建一个键值，然后把www.wulaoer.org节点stop掉，在wulaoer节点上去获取创建的键值，看是否成功。

[root@www.wulaoer.org ~]# etcdctl set --ttl '0' wulaoer "www.wulaoer.org"
www.wulaoer.org
[root@wulaoer ~]# etcdctl get wulaoer
www.wulaoer.org
[root@wolf ~]# etcdctl get wulaoer
www.wulaoer.org

集群中的数据存储不会丢失，这时我们可以在etcd的集群前加一个负载均衡，根据etcd的2379端口做一个自动检测，如果服务不可用自动从负载均衡上去掉，例如nginx的反向代理就可以，或者lvs以及proxy。