Configure a cluster network with Cephadm? [closed]

Question

I configured a simple Ceph cluster with cephadm, like this:

cephadm bootstrap --mon-ip 192.168.0.4
ssh-copy-id -f -i /etc/ceph/ceph.pub root@host2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@host3
ceph orch host add host2
ceph orch host add host3
ceph orch apply osd --all-available-devices

And it works fine. Now I would like to set up a cluster network on 192.168.1.0/24.

I tried ceph config set global cluster_network 192.168.1.0/24 but it doesn't have any effect. I am not seeing any traffic on the secondary NICs.

How can I troubleshoot this? What is the recommended way to make cephadm notice the change?

Appetizer

Answer 1

It turns out this is easy. You set the new configs:

ceph config set global cluster_network 192.168.1.0/24

Then have the orchestrator reconfigure the daemons:

ceph orch daemon reconfig mon.host1
ceph orch daemon reconfig mon.host2
ceph orch daemon reconfig mon.host3
ceph orch daemon reconfig osd.1
ceph orch daemon reconfig osd.2
ceph orch daemon reconfig osd.3
...

Appetizer

how do you ensure that network_cluster is really running? – Valerlan

It didn't work for me, i am running octopus release. – Anlace

Answer 2

Here's a link

you can specify a customized ceph.conf before the 'cephadm bootstrap' command [1] to add a dedicated cluster network (if you really need that, it has been discussed extensively on the list):

octo1:~ # cat <<EOF > /root/ceph.conf
[global]
public network = 192.168.124.0/24
cluster network = 192.168.127.0/24
EOF


octo1:~ # cephadm bootstrap -c /root/ceph.conf --mon-ip 192.168.124.5

After you deployed the OSDs they should be attached to the cluster network:

octo1:~ # ceph osd metadata 0 | grep addr
"back_addr": "[v2:192.168.127.11:6800/4272275002,v1:192.168.127.11:6801/4272275002]", "front_addr": "[v2:192.168.124.5:6802/4272275002,v1:192.168.124.5:6803/4272275002]", "hb_back_addr": "[v2:192.168.127.11:6802/4272275002,v1:192.168.127.11:6803/4272275002]", "hb_front_addr": "[v2:192.168.124.5:6804/4272275002,v1:192.168.124.5:6805/4272275002]",

You should see the replication traffic on the respective interfaces.

Jumpy