Diagnostics and Troubleshooting

Ceph

To get a quick Ceph status check, run ceph health on a controller. You should get this response:

# ceph health
HEALTH_OK

If the Ceph noout option has been set, you’ll get this response instead:

# ceph health
HEALTH_WARN noout flag(s) set

More details are available with ceph -s:

# ceph -s
    cluster 8c213314-82f6-44ea-bcdc-f8d44a93186b
     health HEALTH_WARN
            noout flag(s) set
     monmap e1: 4 mons at {cc1-d=10.10.4.1:6789/0,vmhost1-d=10.10.4.3:6789/0,vmhost2-d=10.10.4.4:6789/0,vmhost3-d=10.10.4.5:6789/0}
            election epoch 76, quorum 0,1,2,3 cc1-d,vmhost1-d,vmhost2-d,vmhost3-d
     osdmap e42: 4 osds: 4 up, 4 in
            flags noout
      pgmap v19117: 384 pgs, 3 pools, 13272 MB data, 1674 objects
            26745 MB used, 3678 GB / 3704 GB avail
                 384 active+clean
  client io 11374 kB/s rd, 9661 B/s wr, 188 op/s

Controller Services

This command performs a quick check for the OpenStack nova services on the controller(s) and vmhosts:

# nova service-list
+----+------------------+-----------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host      | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+-----------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-cert        | cc1       | internal | enabled | up    | 2015-09-04T00:23:14.000000 | -               |
| 2  | nova-conductor   | cc1       | internal | enabled | up    | 2015-09-04T00:23:14.000000 | -               |
| 3  | nova-consoleauth | cc1       | internal | enabled | up    | 2015-09-04T00:23:14.000000 | -               |
| 4  | nova-scheduler   | cc1       | internal | enabled | up    | 2015-09-04T00:23:14.000000 | -               |
| 5  | nova-network     | cc1       | internal | enabled | up    | 2015-09-04T00:23:16.000000 | -               |
| 6  | nova-compute     | vmhost2   | nova     | enabled | up    | 2015-09-04T00:23:08.000000 | -               |
| 7  | nova-compute     | vmhost1   | nova     | enabled | up    | 2015-09-04T00:23:12.000000 | -               |
| 8  | nova-compute     | vmhost3   | nova     | enabled | up    | 2015-09-04T00:23:14.000000 | -               |
+----+------------------+-----------+----------+---------+-------+----------------------------+-----------------+

All of the following services should be running on the SCM controller node with the given ports, and writing to the given logs. The Non HA Port is used by services in single-controller setups. In dual-controller setups, HAProxy listens on the service’s normal (non HA) port and forwards to the service listening at the HA port.

You can check if a port is listening with Linux commands like:

# lsof -i:port

Example:

# lsof -i:8774
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nova-api 7071 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7129 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7130 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7131 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7132 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7140 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7141 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7142 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7143 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7144 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7145 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7146 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7147 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7148 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7149 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7150 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)
nova-api 7151 nova    6u  IPv4  17822      0t0  TCP *:8774 (LISTEN)

or:

# nc -vzw2 ip port

Example:

# nc -vzw2 localhost 8774
Connection to localhost 8774 port [tcp/*] succeeded!
Service Description Non HA Port HA Port Log
Scyld Cloud Portal SCM web dashboard 443 7690 /var/log/apache2/error.log /var/www/wsgi/cloudportal/...
Scyld Cloud Controller SCM-OpenStack interface 7703 7693 /var/log/apache2/cloudcontroller/error.log /var/www/wsgi/cloudcontroller/scyld-cloud-controller/scyld-cloud-controller/cloudcontroller.log
Scyld Cloud Auth SCM authentication and authorization 7701 7691 /var/log/apache2/cloudauth/error.log /var/www/wsgi/cloudauth/scyld-cloud-auth/scyld-cloud-auth/cloudauth.log
Scyld Cloud Accountant SCM accounting 7702 7692 /var/www/wsgi/cloudacct/scyld-cloud-acct/scyld-cloud-acct/cloudacct.log
MySQL relational database 3306 2996 /var/log/mysql/
RabbitMQ message broker 5672   /var/log/rabbitmq/
sshd secure shell 22    
Redis in-memory database 6379   /var/log/redis/redis-server.log
Keystone OpenStack authentication, authorization, service discovery public 5000, admin 35357 public 4990, admin 35347 /var/log/keystone/keystone.log
nova-api OpenStack compute     /var/log/nova/nova-api.log
nova-cert OpenStack compute     /var/log/nova/nova-cert.log
nova-conductor OpenStack compute     /var/log/nova/nova-conductor.log
nova-scheduler OpenStack compute     /var/log/nova/nova-scheduler.log
nova-consoleauth OpenStack compute     /var/log/nova/consoleauth.log
cinder-api OpenStack block     /var/log/cinder/cinder-api.log
cinder-scheduler OpenStack block     /var/log/cinder/cinder-scheduler.log
cinder-volume OpenStack block 8776 8766 /var/log/cinder/cinder-volume.log
glance-api OpenStack image 9292 9282 /var/log/glance/api.log
glance-registry OpenStack image 9191 9181 /var/log/glance/registry.log
ceph-mon Ceph monitor 6789   /var/log/ceph/ceph-mon.log
aptcacher Linux software installer 3142 3132  
haproxy HA proxy service stats 8080    

VMHost Services

Service Description Non HA Port HA Port Log
nova-api-metadata OpenStack compute 8775   /var/log/nova/nova-api-metadata.log
nova-network OpenStack compute     /var/log/nova/nova-network.log
nova-compute OpenStack compute 8774 8764 /var/log/nova/nova-compute.log
libvirt virtualization 16514