k8s删除名称空间报错,一直处于Terminating状态中

技术文章 1年前 (2020) 完美者
435 0

标签:名称   period   sed   namespace   阶段   dde   try   continue   mat   

  • 删除ns,一直处于Terminating状态中
    强制删除也是出现报错

  • 原因:因为ingress controller的镜像 pull 失败,一直在 retry ,所以我就把 ingress-controller delete 掉,但是一直卡住在删除 namespace 阶段 Ctrl + c

    [root@master1 ingress]# kubectl delete -f mandatory.yaml
    namespace "ingress-nginx" deleted
    configmap "nginx-configuration" deleted
    configmap "tcp-services" deleted
    configmap "udp-services" deleted
    serviceaccount "nginx-ingress-serviceaccount" deleted
    clusterrole.rbac.authorization.k8s.io "nginx-ingress-clusterrole" deleted
    role.rbac.authorization.k8s.io "nginx-ingress-role" deleted
    rolebinding.rbac.authorization.k8s.io "nginx-ingress-role-nisa-binding" deleted
    clusterrolebinding.rbac.authorization.k8s.io "nginx-ingress-clusterrole-nisa-binding" deleted
    daemonset.apps "nginx-ingress-controller" deleted
    ^C
  • 强制结束以后发现 pod 已经被删掉,导入容器以后重新 create 报错,报错内容就是 ingress-nginx 名称空间 处于 Terminating 状态
  • [root@master1 ingress]# kubectl create -f mandatory.yaml
    clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
    clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
    Error from server (AlreadyExists): error when creating "mandatory.yaml": object is being deleted: namespaces "ingress-ngin                                                               x" already exists
    Error from server (Forbidden): error when creating "mandatory.yaml": configmaps "nginx-configuration" is forbidden: unable                                                                to create new content in namespace ingress-nginx because it is being terminated
    Error from server (Forbidden): error when creating "mandatory.yaml": configmaps "tcp-services" is forbidden: unable to cre                                                               ate new content in namespace ingress-nginx because it is being terminated
    Error from server (Forbidden): error when creating "mandatory.yaml": configmaps "udp-services" is forbidden: unable to cre                                                               ate new content in namespace ingress-nginx because it is being terminated
    Error from server (Forbidden): error when creating "mandatory.yaml": serviceaccounts "nginx-ingress-serviceaccount" is for                                                               bidden: unable to create new content in namespace ingress-nginx because it is being terminated
    Error from server (Forbidden): error when creating "mandatory.yaml": roles.rbac.authorization.k8s.io "nginx-ingress-role"                                                                is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
    Error from server (Forbidden): error when creating "mandatory.yaml": rolebindings.rbac.authorization.k8s.io "nginx-ingress                                                               -role-nisa-binding" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
    Error from server (Forbidden): error when creating "mandatory.yaml": daemonsets.apps "nginx-ingress-controller" is forbidd                                                               en: unable to create new content in namespace ingress-nginx because it is being terminated
    
    • 强制删除名称空间报错
    [root@master1 ingress]#  kubectl delete ns ingress-nginx --force --grace-period=0
    warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may                                                                continue to run on the cluster indefinitely.
    Error from server (Conflict): Operation cannot be fulfilled on namespaces "ingress-nginx": The system is ensuring all cont                                                               ent is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

    解决步骤

    • 导出运行的名称空间至json文件,删掉其中的spec字段内容,因为k8s集群是携带认证的
    [root@master1 ingress]# kubectl get ns ingress-nginx -o json > tmp.json
    [root@master1 ingress]# vim tmp.json
    [root@master1 ingress]# cat tmp.json
    {
        "apiVersion": "v1",
        "kind": "Namespace",
        "metadata": {
            "annotations": {
                "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"ingress-nginx\"}}\n"
            },
            "creationTimestamp": "2020-09-11T02:16:47Z",
            "deletionTimestamp": "2020-09-11T02:33:30Z",
            "name": "ingress-nginx",
            "resourceVersion": "175097",
            "selfLink": "/api/v1/namespaces/ingress-nginx",
            "uid": "9f748136-88c5-4627-a531-81547e191073"
        },
        "status": {
            "conditions": [
                {
                    "lastTransitionTime": "2020-09-11T02:33:35Z",
                    "message": "Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request",
                    "reason": "DiscoveryFailed",
                    "status": "True",
                    "type": "NamespaceDeletionDiscoveryFailure"
                },
                {
                    "lastTransitionTime": "2020-09-11T02:33:35Z",
                    "message": "All legacy kube types successfully parsed",
                    "reason": "ParsedGroupVersions",
                    "status": "False",
                    "type": "NamespaceDeletionGroupVersionParsingFailure"
                },
                {
                    "lastTransitionTime": "2020-09-11T02:33:35Z",
                    "message": "All content successfully deleted",
                    "reason": "ContentDeleted",
                    "status": "False",
                    "type": "NamespaceDeletionContentFailure"
                }
            ],
            "phase": "Terminating"
        }
    }
    • 执行以下curl命令,使用kube-apiserver的8080端口,执行删除操作

      curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8080/api/v1/namespaces/ingress-nginx/finalize
    • 如果kube-apiserver未开非安全端口,可以手动创建一个代理会话
    [root@master1 ingress]# kubectl proxy --port=8081
    # 新开一个shell终端执行curl命令
    [root@master1 ~]# curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8081/api/v1/namespaces/ingress-nginx/finalize
    • 然后就会删掉此ingress-nginx名称空间
    [root@master1 ingress]# kubectl get ns
    NAME                   STATUS        AGE
    default                Active        7d20h
    kube-node-lease        Active        7d20h
    kube-public            Active        7d20h
    kube-system            Active        7d20h
    kubernetes-dashboard   Terminating   7d14h

    k8s删除名称空间报错,一直处于Terminating状态中

    标签:名称   period   sed   namespace   阶段   dde   try   continue   mat   

    原文地址:https://blog.51cto.com/liujingyu/2531898

    版权声明:完美者 发表于 2020-09-17 22:47:37。
    转载请注明:k8s删除名称空间报错,一直处于Terminating状态中 | 完美导航

    暂无评论

    暂无评论...