继续遇到“超出其进度极限"的情况.尽管进度有所变化

本文介绍了继续遇到“超出其进度极限"的情况.尽管进度有所变化的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我是新的AKS，ACR和DevOps管道，我正在尝试设置CI/CD管道.

I'm new AKS, ACR, and DevOps Pipelines and I'm trying to setup a CI/CD pipeline.

我有一个同时具有AKS和ACR的资源组设置.由于我只是在玩耍，因此AKS目前仅使用Standard_B2s，并且仅使用一个节点.

I have a resource group setup that has both AKS and ACR in it. AKS is using Standard_B2s and only one node at this point since I'm just playing around.

图像是在提交给主服务器后自动部署到ACR的(尚未弄清楚如何设置测试)，但是当部署到AKS时，我一直在获取:

Images are being deployed to ACR automatically on a commit to master--haven't figured out how to setup testing yet--but when it comes to deploying to AKS, I just keep getting a:

##[error]error: deployment "client-deployment" exceeded its progress deadline

我将client.yaml更改为包括一个小时的progressDeadlineSeconds，因为10、15和20分钟不起作用:

I've changed my client.yaml to include a progressDeadlineSeconds of like an hour as 10, 15, and 20 minutes didn't work:

apiVersion: apps/v1 kind: Deployment metadata: name: client-deployment spec: progressDeadlineSeconds: 3600 replicas: 1 selector: matchLabels: component: client template: metadata: labels: component: client spec: containers: - name: client image: testappcontainers.azurecr.io/testapp-client ports: - containerPort: 3000 --- apiVersion: v1 kind: Service metadata: name: client-cluster-ip-service spec: type: ClusterIP selector: component: client ports: - port: 3000 targetPort: 3000

我刚刚正在修改管道为我生成的azure-pipelines.yml，目前，我将其修改为以下内容:

I've just been modifying the azure-pipelines.yml that Pipelines generated for me, which I currently have as the following:

# Docker # Build and push an image to Azure Container Registry # docs.microsoft/azure/devops/pipelines/languages/docker trigger: - master resources: - repo: self variables: # Container registry service connection established during pipeline creation dockerRegistryServiceConnection: <dockerRegistryServiceConnection_key> imageRepository: 'testapp' containerRegistry: 'testappcontainers.azurecr.io' dockerfilePath: '$(Build.SourcesDirectory)' tag: '$(Build.BuildId)' imagePullSecret: <imagePullSecret_key> # Agent VM image name vmImageName: 'ubuntu-latest' stages: - stage: Build displayName: Build and push stage jobs: - job: Build displayName: Build pool: vmImage: $(vmImageName) steps: - task: Docker@2 displayName: Build and push client image to container registry inputs: command: buildAndPush repository: $(imageRepository)-client dockerfile: $(dockerfilePath)/client/Dockerfile containerRegistry: $(dockerRegistryServiceConnection) tags: | $(tag) - upload: manifests artifact: manifests - stage: Deploy displayName: Deploy stage dependsOn: Build jobs: - deployment: Deploy displayName: Deploy job pool: vmImage: $(vmImageName) environment: 'testapp.default' strategy: runOnce: deploy: steps: - task: KubernetesManifest@0 displayName: Create imagePullSecret inputs: action: createSecret secretName: $(imagePullSecret) dockerRegistryEndpoint: $(dockerRegistryServiceConnection) - task: KubernetesManifest@0 displayName: Deploy to Kubernetes cluster inputs: action: deploy manifests: | $(Pipeline.Workspace)/manifests/client.yaml imagePullSecrets: | $(imagePullSecret) containers: | $(containerRegistry)/$(imageRepository):$(tag)

这也是失败的任务的日志:

Here is the log too for the Task that fails:

##[debug]Evaluating condition for step: 'Deploy to Kubernetes cluster' ##[debug]Evaluating: SucceededNode() ##[debug]Evaluating SucceededNode: ##[debug]=> True ##[debug]Result: True ##[section]Starting: Deploy to Kubernetes cluster ============================================================================== Task : Deploy to Kubernetes Description : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts Version : 0.162.1 Author : Microsoft Corporation Help : docs.microsoft/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest ============================================================================== ##[debug]agent.TempDirectory=/home/vsts/work/_temp ##[debug]loading inputs and endpoints ##[debug]loading INPUT_ACTION ##[debug]loading INPUT_KUBERNETESSERVICECONNECTION ##[debug]loading INPUT_STRATEGY ##[debug]loading INPUT_TRAFFICSPLITMETHOD ##[debug]loading INPUT_PERCENTAGE ##[debug]loading INPUT_BASELINEANDCANARYREPLICAS ##[debug]loading INPUT_MANIFESTS ##[debug]loading INPUT_CONTAINERS ##[debug]loading INPUT_IMAGEPULLSECRETS ##[debug]loading INPUT_RENDERTYPE ##[debug]loading INPUT_DOCKERCOMPOSEFILE ##[debug]loading INPUT_HELMCHART ##[debug]loading INPUT_KUSTOMIZATIONPATH ##[debug]loading INPUT_RESOURCETOPATCH ##[debug]loading INPUT_RESOURCEFILETOPATCH ##[debug]loading INPUT_MERGESTRATEGY ##[debug]loading INPUT_SECRETTYPE ##[debug]loading ENDPOINT_AUTH_<token> ##[debug]loading ENDPOINT_AUTH_SCHEME_<token> ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZUREENVIRONMENT ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZURETENANTID ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTNAME ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_ROLEBINDINGNAME ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SECRETNAME ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_APITOKEN ##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTCERTIFICATE ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN ##[debug]loading SECRET_CONTAINER_PASSWORD ##[debug]loading SECRET_CONTAINER_USERNAME ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN ##[debug]loaded 32 ##[debug]Agent.ProxyUrl=undefined ##[debug]Agent.CAInfo=undefined ##[debug]Agent.ClientCert=undefined ##[debug]Agent.SkipCertValidation=undefined ##[debug]SYSTEM_HOSTTYPE=build ##[debug]System.TeamFoundationCollectionUri=dev.azure/thetestcompany/ ##[debug]Build.BuildNumber=20191231.5 ##[debug]Build.DefinitionName=test-app ##[debug]System.DefinitionId=4 ##[debug]Agent.JobName=Deploy job ##[debug]System.TeamProject=test-app ##[debug]Build.BuildId=41 ##[debug]System.TeamProject=test-app ##[debug]namespace=null ##[debug]containers=***/testapp:41 ##[debug]imagePullSecrets=testappcontainers<key>-auth ##[debug]manifests=/home/vsts/work/1/manifests/client.yaml ##[debug]percentage=0 ##[debug]strategy=none ##[debug]trafficSplitMethod=pod ##[debug]baselineAndCanaryReplicas=0 ##[debug]arguments=null ##[debug]secretArguments=null ##[debug]secretType=dockerRegistry ##[debug]secretName=null ##[debug]dockerRegistryEndpoint=null ##[debug]kubernetesServiceConnection=<token> ##[debug]<token> data namespace = default ##[debug]System.TeamFoundationCollectionUri=dev.azure/thetestcompany/ ##[debug]System.HostType=build ##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s ##[debug]Build.SourceBranchName=master ##[debug]Build.Repository.Provider=TfsGit ##[debug]Build.Repository.Uri=thetestcompany@dev.azure/thetestcompany/test-app/_git/test-app ##[debug]agent.proxyurl=undefined ##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined ##[debug]AZURE_HTTP_USER_AGENT=VSTS_<hash>_build_4_0 ##[debug]Agent.ProxyUrl=undefined ##[debug]Agent.CAInfo=undefined ##[debug]Agent.ClientCert=undefined ##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json ##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json ##[debug]system.culture=en-US ##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json ##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json ##[debug]system.culture=en-US ##[debug]action=deploy ##[debug]kubernetesServiceConnection=<token> ##[debug]agent.tempDirectory=/home/vsts/work/_temp ##[debug]<token> data authorizationType = AzureSubscription ##[debug]<token>=testappk8s-dns-<key>.hcp.westus.azmk8s.io/ ##[debug]<token> auth param serviceAccountCertificate = *** ##[debug]<token> auth param apiToken = *** ##[debug]set KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config ##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]/home/vsts/work/_temp/kubectlTask/1577816701759/config ##[debug]<token> data acceptUntrustedCerts = undefined ##[debug]which 'kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]which 'kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s ##[debug]defaultRoot: '/home/vsts/work/1/s' ##[debug]findOptions.allowBrokenSymbolicLinks: 'false' ##[debug]findOptions.followSpecifiedSymbolicLink: 'true' ##[debug]findOptions.followSymbolicLinks: 'true' ##[debug]matchOptions.debug: 'false' ##[debug]matchOptions.nobrace: 'true' ##[debug]matchOptions.noglobstar: 'false' ##[debug]matchOptions.dot: 'true' ##[debug]matchOptions.noext: 'false' ##[debug]matchOptions.nocase: 'false' ##[debug]matchOptions.nonull: 'false' ##[debug]matchOptions.matchBase: 'false' ##[debug]matchOptions.nocomment: 'false' ##[debug]matchOptions.nonegate: 'false' ##[debug]matchOptions.flipNegate: 'false' ##[debug]pattern: '/home/vsts/work/1/manifests/client.yaml' ##[debug]findPath: '/home/vsts/work/1/manifests/client.yaml' ##[debug]statOnly: 'true' ##[debug]found 1 paths ##[debug]applying include pattern ##[debug]1 matches ##[debug]1 final results ##[debug]agent.tempDirectory=/home/vsts/work/_temp ##[debug]New K8s objects after addin imagePullSecrets are :[{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"client-deployment"},"spec":{"progressDeadlineSeconds":3600,"replicas":1,"selector":{"matchLabels":{"component":"client"}},"template":{"metadata":{"labels":{"component":"client"}},"spec":{"containers":[{"name":"client","image":"***/testapp-client","ports":[{"containerPort":3000}]}],"imagePullSecrets":[{"name":"testappcontainers1741032e-auth"}]}}}},{"apiVersion":"v1","kind":"Service","metadata":{"name":"client-cluster-ip-service"},"spec":{"type":"ClusterIP","selector":{"component":"client"},"ports":[{"port":3000,"targetPort":3000}]}}] ##[debug]agent.tempDirectory=/home/vsts/work/_temp ##[debug]agent.tempDirectory=/home/vsts/work/_temp ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]/usr/bin/kubectl arg: apply ##[debug]/usr/bin/kubectl arg: ["-f","/home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782"] ##[debug]/usr/bin/kubectl arg: ["--namespace","default"] ##[debug]exec tool: /usr/bin/kubectl ##[debug]arguments: ##[debug] apply ##[debug] -f ##[debug] /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 ##[debug] --namespace ##[debug] default [command]/usr/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 --namespace default deployment.apps/client-deployment unchanged service/client-cluster-ip-service unchanged ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]/usr/bin/kubectl arg: ["rollout","status"] ##[debug]/usr/bin/kubectl arg: Deployment/client-deployment ##[debug]/usr/bin/kubectl arg: ["--namespace","default"] ##[debug]exec tool: /usr/bin/kubectl ##[debug]arguments: ##[debug] rollout ##[debug] status ##[debug] Deployment/client-deployment ##[debug] --namespace ##[debug] default [command]/usr/bin/kubectl rollout status Deployment/client-deployment --namespace default error: deployment "client-deployment" exceeded its progress deadline ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]which '/usr/bin/kubectl' ##[debug]found: '/usr/bin/kubectl' ##[debug]/usr/bin/kubectl arg: get ##[debug]/usr/bin/kubectl arg: service/client-cluster-ip-service ##[debug]/usr/bin/kubectl arg: ["-o","json"] ##[debug]/usr/bin/kubectl arg: ["--namespace","default"] ##[debug]exec tool: /usr/bin/kubectl ##[debug]arguments: ##[debug] get ##[debug] service/client-cluster-ip-service ##[debug] -o ##[debug] json ##[debug] --namespace ##[debug] default [command]/usr/bin/kubectl get service/client-cluster-ip-service -o json --namespace default { "apiVersion": "v1", "kind": "Service", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"client-cluster-ip-service\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":3000,\"targetPort\":3000}],\"selector\":{\"component\":\"client\"},\"type\":\"ClusterIP\"}}\n" }, "creationTimestamp": "name": "client-cluster-ip-service", "namespace": "default", "resourceVersion": "1234045", "selfLink": "/api/v1/namespaces/default/services/client-cluster-ip-service", "uid": "5f077159-2bdd-11ea-af20-3eaa105eb2b3" }, "spec": { "clusterIP": "10.0.181.220", "ports": [ { "port": 3000, "protocol": "TCP", "targetPort": 3000 } ], "selector": { "component": "client" }, "sessionAffinity": "None", "type": "ClusterIP" }, "status": { "loadBalancer": {} } } ##[debug]KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config ##[debug]set KUBECONFIG= ##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;] ##[debug]task result: Failed ##[error]error: deployment "client-deployment" exceeded its progress deadline ##[debug]Processed: ##vso[task.issue type=error;]error: deployment "client-deployment" exceeded its progress deadline ##[debug]Processed: ##vso[taskplete result=Failed;]error: deployment "client-deployment" exceeded its progress deadline ##[section]Finishing: Deploy to Kubernetes cluster

然后在Azure CLI中，它显示部署已在其中，但是没有可用的pod:

Then in Azure CLI, it shows the deployment is there, but with no available pods:

eox-dev@Azure:~$ kubectl get deployments NAME READY UP-TO-DATE AVAILABLE AGE client-deployment 0/1 1 0 3h47m eox-dev@Azure:~$ kubectl describe deployment client-deployment Name: client-deployment Namespace: default CreationTimestamp: Tue, 31 Dec 2019 15:50:30 +0000 Labels: <none> Annotations: deployment.kubernetes.io/revision: 1 kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"client-deployment","namespace":"default"},"spec":{"progre... Selector: component=client Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: component=client Containers: client: Image: testappcontainers.azurecr.io/testapp-client Port: 3000/TCP Host Port: 0/TCP Environment: <none> Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available False MinimumReplicasUnavailable Progressing False ProgressDeadlineExceeded OldReplicaSets: <none> NewReplicaSet: client-deployment-5688bdc69c (1/1 replicas created) Events: <none>

那我在做什么错了?

推荐答案

来自服务器的错误(BadRequest):容器中的容器客户端" "client-deployment-5688bdc69c-hxlcf"正在等待启动:尝试并 无法提取图片

Error from server (BadRequest): container "client" in pod "client-deployment-5688bdc69c-hxlcf" is waiting to start: trying and failing to pull image

根据我的经验，这与imagePullSecrets和Kubernetes namespace有关.

Based on my experience, this is more relative with imagePullSecrets and the Kubernetes namespace.

在您的Create imagePullSecret和Deploy to Kubernetes cluster任务中，我看到您没有将值提供给任务参数:namespace.这将导致创建一个新的名称空间，名称为default，因为您未指定该名称空间.

In your Create imagePullSecret and Deploy to Kubernetes cluster task, I saw that you did not provide the value to task parameter: namespace. This will lead to a new namespace which name is default will be created, since you unspecified the namespace.

然后， kubernetes机密对于每个名称空间，由createSecret操作生成的操作是分开的.总之，不同的名称空间具有不同的秘密值:

And, the kubernetes secret which generated by createSecret action is seperated for each namespace. In one word, different namespace has different secret value:

秘密存储在给定的名称空间中，并且只能被访问 通过同一名称空间中的Pod.

Secrets are stored within a given namespace and can only be accessed by pods within the same namespace.

现在，让我们回到构建编译过程.

Now, let’s back to your build compile process.

在yml定义中，Create imagePullSecret将为新名称空间default创建一个secret，该名称空间由task自动创建，因为您没有提供给定的名称空间值.

In your yml definition, Create imagePullSecret will create a secret for new namespace default which created by task automatically as you did not provide the given namespace value.

然后，由于相同的原因，在下一个任务Deploy to Kubernetes cluster中，此处的任务将重新创建另一个新的命名空间default(注意:这与上一个不同).另外，您可以从日志中看到此进度:

Then, in next task Deploy to Kubernetes cluster, because of the same reason, here the task will re-created a another new namespace default(Note: this is not same with the previous one). Also, you could see this progress from the log:

这时，从上一个任务生成的secret对于当前名称空间将不可用.但是，您知道ACR是一个私有容器注册表，我们的系统必须验证kubernetes secret是否可用.

At this time, the secret that generated from the previous task will not available for the current namespace. BUT, as you know, the ACR is a private container registry which our system must verify whether the kubernetes secret is available.

此外，在部署到Kubernetes集群"任务中，您将存储库指定为$(imageRepository)，这与将映像推送到 $(imageRepository)-client 的存储库不同.

In addition, in your Deploy to Kubernetes cluster task, you were specifying the repository as $(imageRepository) which does not same with the repository you push the image to $(imageRepository)-client.

这也可以在您的日志中检查:

This can also be checked in your log:

这就是为什么您的kubernetes中没有可用节点，并且您也无法拉取图像的原因.

That's why there's no available node in your kubernetes, and you failed to pull the image also.

为避免此问题，请确保在KubernetesManifest@0任务中提供了namespace值.

To avoid the issue, please ensure you provide the namespace value in KubernetesManifest@0 task.

- task: KubernetesManifest@0 displayName: Create imagePullSecret inputs: action: createSecret secretName: $(imagePullSecret) namespace: $(k8sNamespace) dockerRegistryEndpoint: $(DRServiceConnection) - task: KubernetesManifest@0 displayName: Deploy to Kubernetes cluster inputs: action: deploy namespace: $(k8sNamespace) manifests: | $(System.ArtifactsDirectory)/manifests/deployment.yml imagePullSecrets: | $(imagePullSecret) containers: | $(containerRegistry)/$(imageRepository)-client:$(tag)

每个命名空间的imagePullSecrets的秘密

secret to imagePullSecrets of each namespace