Diagnostics and Troubleshooting¶
SCM Controller Services¶
All of the following services should be running on the SCM controller with the given ports, and writing to the given logs.
The Non HA Port is used by services in single-controller setups. In dual-controller setups, HAProxy listens on the service’s normal (non HA) port and forwards to the service listening at the HA port.
You can check if a port is listening with Linux commands like:
# lsof -i :port
| Service | Description | Non HA Port | HA Port | Log |
|---|---|---|---|---|
| Scyld Cloud Portal | SCM Web Dashboard | 443 | 7690 | /var/log/httpd/cloudportal/ /var/www/wsgi/cloudportal/cloudportal.log |
| Scyld Cloud Auth | SCM Authentication and Authorization | 443 | 7691 | /var/log/httpd/cloudauth/ /var/www/wsgi/cloudauth/cloudauth.log |
| Scyld Cloud Accountant | SCM Accounting | 443 | 7692 | /var/log/httpd/cloudaccountant/ /var/www/wsgi/cloudaccountant/cloudaccountant.log |
| Scyld Cloud Controller | SCM-OpenStack Interface | 443 | 7693 | /var/log/httpd/cloudcontroller/ /var/www/wsgi/cloudcontroller/cloudcontroller.log |
| Redis | In-memory database | 6379 | /var/log/redis/redis-server.log | |
| Haproxy (if present) | HA proxy service | 8080 (Stats) | /var/log/haproxy.log |
OpenStack Services¶
OpenStack services are run inside Docker containers running on each OpenStack controller.
To check the health of the Docker containers on a controller:
# docker ps -a
[root@cc1-d ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9e97dbdf37a9 k4e.scm:4000/kolla/centos-binary-grafana:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks grafana
3d0d128f4b32 k4e.scm:4000/kolla/centos-binary-gnocchi-statsd:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks gnocchi_statsd
16e1bbb9f94d k4e.scm:4000/kolla/centos-binary-gnocchi-metricd:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks gnocchi_metricd
2f137db7e998 k4e.scm:4000/kolla/centos-binary-gnocchi-api:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks gnocchi_api
1fc44027a3eb k4e.scm:4000/kolla/centos-binary-horizon:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks horizon
ee806258eeca k4e.scm:4000/kolla/centos-binary-heat-engine:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks heat_engine
8c48feaa647a k4e.scm:4000/kolla/centos-binary-heat-api-cfn:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
b03f7cf87248 k4e.scm:4000/kolla/centos-binary-heat-api:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks heat_api
ee130fc0fbf9 k4e.scm:4000/kolla/centos-binary-cinder-backup:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks cinder_backup
1152a577e92a k4e.scm:4000/kolla/centos-binary-cinder-volume:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks cinder_volume
cf6a9c0e015d k4e.scm:4000/kolla/centos-binary-cinder-scheduler:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks cinder_scheduler
10c43d513d5b k4e.scm:4000/kolla/centos-binary-cinder-api:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks cinder_api
5c4a5573e3a2 k4e.scm:4000/kolla/centos-binary-neutron-metadata-agent:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
a97881bd17f2 k4e.scm:4000/kolla/centos-binary-neutron-lbaas-agent:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks neutron_lbaas_agent
d106b0e253ca k4e.scm:4000/kolla/centos-binary-neutron-l3-agent:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
7530545a141f k4e.scm:4000/kolla/centos-binary-neutron-dhcp-agent:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
a8d52deb9ed2 k4e.scm:4000/kolla/centos-binary-neutron-openvswitch-agent:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
0e3077090730 k4e.scm:4000/kolla/centos-binary-neutron-server:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
0f3007c497a1 k4e.scm:4000/kolla/centos-binary-nova-conductor:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks nova_conductor
9e5461eab29c k4e.scm:4000/kolla/centos-binary-nova-spicehtml5proxy:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks nova_spicehtml5proxy
a7e1e000587d k4e.scm:4000/kolla/centos-binary-nova-scheduler:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks nova_scheduler
fe20eb20c8d5 k4e.scm:4000/kolla/centos-binary-nova-consoleauth:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks nova_consoleauth
da68b9764cd4 k4e.scm:4000/kolla/centos-binary-nova-api:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks nova_api
aac9176c907e k4e.scm:4000/kolla/centos-binary-glance-api:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks glance_api
0a096c67003e k4e.scm:4000/kolla/centos-binary-glance-registry:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks glance_registry
37202d5529f7 k4e.scm:4000/kolla/centos-binary-keystone:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks keystone
11dd5af66a86 k4e.scm:4000/kolla/centos-binary-rabbitmq:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
7e65e2a8825b k4e.scm:4000/kolla/centos-binary-mariadb:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks mariadb
cca6cdc74245 k4e.scm:4000/kolla/centos-binary-memcached:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks memcached
796247e2a783 k4e.scm:4000/kolla/centos-binary-kibana:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks kibana
e0d6f0a0c25e k4e.scm:4000/kolla/centos-binary-keepalived:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks keepalived
9abb6ca64f2d k4e.scm:4000/kolla/centos-binary-haproxy:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks haproxy
cb4f91e02914 k4e.scm:4000/kolla/centos-binary-elasticsearch:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks elasticsearch
4cf2280d2854 k4e.scm:4000/kolla/centos-binary-ceph-rgw:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks ceph_rgw
c34e2b58edba k4e.scm:4000/kolla/centos-binary-ceph-osd:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks ceph_osd_0
c0349b4d3650 k4e.scm:4000/kolla/centos-binary-ceph-mon:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks ceph_mon
afa384731c7b k4e.scm:4000/kolla/centos-binary-cron:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks cron
e3003c071f19 k4e.scm:4000/kolla/centos-binary-kolla-toolbox:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
c7eee2e64766 k4e.scm:4000/kolla/centos-binary-heka:3.0.2 "kolla_start" 2 weeks ago Up 2 weeks heka
Log files for each service are available from Docker by running docker logs SERVICE_NAME or under /var/lib/docker/volumes/kolla_logs/_data/SERVICE_NAME.
A web dashboard for logs, Kibana, is also available at the same hostname that provides the Horizon dashboard at port 5601.
Ceph¶
To get a quick Ceph status check,
run ceph health on a controller.
You should get this response:
# ceph health
HEALTH_OK
If the Ceph noout option has been set,
you’ll get this response instead:
# ceph health
HEALTH_WARN noout flag(s) set
More details are available with ceph -s:
# ceph -s
cluster 8c213314-82f6-44ea-bcdc-f8d44a93186b
health HEALTH_WARN
noout flag(s) set
monmap e1: 4 mons at {cc1-d=10.10.4.1:6789/0,vmhost1-d=10.10.4.3:6789/0,vmhost2-d=10.10.4.4:6789/0,vmhost3-d=10.10.4.5:6789/0}
election epoch 76, quorum 0,1,2,3 cc1-d,vmhost1-d,vmhost2-d,vmhost3-d
osdmap e42: 4 osds: 4 up, 4 in
flags noout
pgmap v19117: 384 pgs, 3 pools, 13272 MB data, 1674 objects
26745 MB used, 3678 GB / 3704 GB avail
384 active+clean
client io 11374 kB/s rd, 9661 B/s wr, 188 op/s