Troubeshooting abcdesktop core services
Troubeshooting nginx errors
Read pod's status
kubectl get pods -n abcdesktop
NAME READY STATUS RESTARTS AGE
memcached-od-78578c879-bb8qq 1/1 Running 0 164m
mongodb-od-5b4dd4765f-ptw2j 1/1 Running 0 164m
nginx-od-788c97cdc9-b4gbq 0/1 CrashLoopBackOff 36 (57s ago) 164m
openldap-od-65759b74dc-tbvfg 1/1 Running 0 164m
pyos-od-7d5d9457cf-jw6nk 1/1 Running 0 164m
speedtest-od-c94b56c88-48cvq 1/1 Running 0 164m
The pod nginx-od-788c97cdc9-b4gbq has CrashLoopBackOff status. This is wrong.
Read the pod's log
kubectl logs -l run=nginx-od -n abcdesktop
Issue with an error in nginx configuration file
running standart configuration file
starting nginx web server in foreground
nginx: [emerg] unexpected "s" in /etc/nginx/sites-enabled/default:10
Nginx has failed to start. There is an error in the configuration file.
We need to fix the nginx-config ConfigMap in the yaml file.
Start the pod by hands
If the kubectl logs command doesn't return usable information. You can update the pod default command and then start the service by hands.
Update the container description to replace the default command by a sleep command
- name: nginx
imagePullPolicy: Always
image: abcdesktopio/oc.nginx:3.0
command: [ "/usr/bin/sleep" ]
args: ["1d"]
The container will start the command /usr/bin/sleep for 1d (one day).
A default nginx debug pods is available on https://github.com/abcdesktopio/conf/tree/main/kubernetes/debug
kubectl apply -f https://raw.githubusercontent.com/abcdesktopio/conf/main/kubernetes/debug/nginx-3.0.yaml
deployment.apps/nginx-od configured
Check that nginx pod has been updated and that the status is Running
kubectl get pods -l run=nginx-od -n abcdesktop
NAME READY STATUS RESTARTS AGE
nginx-od-666df64f4-whtng 1/1 Running 0 2m30s
Nginx web service is not started inside the container, only the pod is started. We need to get a shell inside the container to start the nginx web service by hands.
Run the command /usr/local/openresty/nginx/sbin/nginx -p /etc/nginx -c nginx.conf -e /var/log/nginx/error.log
kubectl exec -n abcdesktop -it deployment/nginx-od -- bash
root@nginx-od-666df64f4-whtng:/# /usr/local/openresty/nginx/sbin/nginx -p /etc/nginx -c nginx.conf -e /var/log/nginx/error.log
Nginx returns an explicit error, the /etc/nginx/sites-enabled/default file is wrong.
nginx: [emerg] unexpected "s" in /etc/nginx/sites-enabled/default:10
root@nginx-od-666df64f4-whtng:/#
It's time to fix the nginx-config ConfigMap in the yaml file.
Troubeshooting pyos errors
Read pod's status
kubectl get pods -n abcdesktop
NAME READY STATUS RESTARTS AGE
memcached-od-5ff8844d56-sw9n5 1/1 Running 0 90m
mongodb-od-77c945467d-c47nl 1/1 Running 0 90m
nginx-od-666df64f4-wf99b 1/1 Running 0 22m
openldap-od-5bbdd75864-m6qmh 1/1 Running 0 90m
pyos-od-57946b67c4-m5zc9 0/1 CrashLoopBackOff 5 (17s ago) 3m18s
speedtest-od-7f5484966f-kxkw4 1/1 Running 0 90m
The pod pyos-od-57946b67c4-m5zc9 has CrashLoopBackOff status. This is wrong.
Read the pod's log
kubectl logs -l run=pyos-od -n abcdesktop
2023-10-15 14:53:00,136 [INFO ] oc.logging.init_logging: Initializing logging subsystem
2023-10-15 14:53:00,136 [INFO ] oc.logging.load_config: Reading cherrypy configuration section 'global/logging': path = od.config
2023-10-15 14:53:00,138 [CRITICAL] oc.logging.configure: Failed to configure logging: config_or_path = 'od.config'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 179, in as_dict
value = unrepr(value)
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 367, in unrepr
obj = b.astnode(s)
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 229, in astnode
p = ast.parse('__tempvalue__ = ' + s)
File "/usr/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
File "<unknown>", line 1
__tempvalue__ = 'abcdesktop
^
SyntaxError: EOL while scanning string literal
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/pyos/oc/logging.py", line 89, in configure
init_logging(config_or_path, is_cp_file)
File "/var/pyos/oc/logging.py", line 80, in init_logging
cfg = config_or_path if isinstance(config_or_path, dict) else load_config(config_or_path, is_cp_file)
File "/var/pyos/oc/logging.py", line 66, in load_config
cfg = Config(path)['global']['logging']
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 119, in __init__
self.update(file)
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 130, in update
self._apply(Parser.load(config))
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 205, in load
return Parser().dict_from_file(input) if is_file else input.copy()
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 194, in dict_from_file
return self.as_dict()
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 185, in as_dict
raise ValueError(msg, x.__class__.__name__, x.args)
ValueError: ('Config error in section: \'global\', option: \'namespace\', value: "\'abcdesktop". Config values must be valid Python.', 'SyntaxError', ('EOL while scanning string literal', ('<unknown>', 1, 28, "__tempvalue__ = 'abcdesktop\n")))
Failed to load configuration file od.config ('Config error in section: \'global\', option: \'namespace\', value: "\'abcdesktop". Config values must be valid Python.', 'SyntaxError', ('EOL while scanning string literal', ('<unknown>', 1, 28, "__tempvalue__ = 'abcdesktop\n")))
It's time to fix the abcdesktop-config ConfigMap.
Start the pod by hands
If the kubectl logs command doesn't return usable information. You can update the pod default command and then start the service by hands.
Update the container description to replace the default command by a sleep command
- name : pyos
imagePullPolicy: Always
image: abcdesktopio/oc.pyos:3.0
command: [ "/usr/bin/sleep" ]
args: ["1d"]
The container will start the command /usr/bin/sleep for 1d (one day).
A default nginx debug pods is available on https://github.com/abcdesktopio/conf/tree/main/kubernetes/debug
kubectl apply -f https://raw.githubusercontent.com/abcdesktopio/conf/main/kubernetes/debug/pyos-3.0.yaml
deployment.apps/pyos-od configured
Check that pyos pod is Running
kubectl get pods -l run=pyos-od -n abcdesktop
NAME READY STATUS RESTARTS AGE
pyos-od-6cd679d6b8-css9q 1/1 Running 0 5s
Pyos service is not started inside the container, only the pod is started. We need to get a shell inside the container to start the pyos service by hands.
Run the command cd /var/pyos && ./od.py
kubectl exec -n abcdesktop -it deployment/pyos-od -- bash
root@pyos-od-6cd679d6b8-css9q:/var/pyos# cd /var/pyos && ./od.py
od.py command returns the same explicit error, the od.config file is wrong.
2023-10-15 14:53:00,136 [INFO ] oc.logging.init_logging: Initializing logging subsystem
2023-10-15 14:53:00,136 [INFO ] oc.logging.load_config: Reading cherrypy configuration section 'global/logging': path = od.config
2023-10-15 14:53:00,138 [CRITICAL] oc.logging.configure: Failed to configure logging: config_or_path = 'od.config'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 179, in as_dict
value = unrepr(value)
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 367, in unrepr
obj = b.astnode(s)
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 229, in astnode
p = ast.parse('__tempvalue__ = ' + s)
File "/usr/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
File "<unknown>", line 1
__tempvalue__ = 'abcdesktop
^
SyntaxError: EOL while scanning string literal
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/pyos/oc/logging.py", line 89, in configure
init_logging(config_or_path, is_cp_file)
File "/var/pyos/oc/logging.py", line 80, in init_logging
cfg = config_or_path if isinstance(config_or_path, dict) else load_config(config_or_path, is_cp_file)
File "/var/pyos/oc/logging.py", line 66, in load_config
cfg = Config(path)['global']['logging']
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 119, in __init__
self.update(file)
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 130, in update
self._apply(Parser.load(config))
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 205, in load
return Parser().dict_from_file(input) if is_file else input.copy()
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 194, in dict_from_file
return self.as_dict()
File "/usr/local/lib/python3.8/dist-packages/cherrypy/lib/reprconf.py", line 185, in as_dict
raise ValueError(msg, x.__class__.__name__, x.args)
ValueError: ('Config error in section: \'global\', option: \'namespace\', value: "\'abcdesktop". Config values must be valid Python.', 'SyntaxError', ('EOL while scanning string literal', ('<unknown>', 1, 28, "__tempvalue__ = 'abcdesktop\n")))
Failed to load configuration file od.config ('Config error in section: \'global\', option: \'namespace\', value: "\'abcdesktop". Config values must be valid Python.', 'SyntaxError', ('EOL while scanning string literal', ('<unknown>', 1, 28, "__tempvalue__ = 'abcdesktop\n")))
We need to fix the abcdesktop-config ConfigMap in the yaml file.
kubectl create -n abcdesktop configmap abcdesktop-config --from-file=od.config -o yaml --dry-run=client | kubectl replace -n abcdesktop -f -