Sync the pipelinerun status from the informers #2573

afrittoli · 2020-05-07T14:48:03Z

Changes

When we reconcile a pipelinerun, we should ensure that the
pipelinerun status is always in sync with the actual list of taskruns
that can be provided by the taskrun informer.

The only way to filter taskruns is by labels tekton.dev/pipelinerun.
In case an orphaned taskrun is found, we can use the other labels
on the taskrun to reconstruct the missing entry in the pipelinerun
status.

Fixes #2558

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes tests (if functionality changed/added)
Commit messages follow commit message best practices

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

afrittoli · 2020-05-07T14:51:33Z

@n3wscott @mattmoor any feedback on this would be really welcome :)

tekton-robot · 2020-05-07T14:51:37Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	72.4%	72.3%	-0.1

tekton-robot · 2020-05-11T20:59:13Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	72.0%	73.5%	1.4

tekton-robot · 2020-05-12T08:35:41Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	72.0%	73.5%	1.4

afrittoli · 2020-05-12T11:35:35Z

/test pull-tekton-pipeline-integration-tests

vdemeester

Looks good to me
/meow

pkg/reconciler/pipelinerun/pipelinerun_test.go

tekton-robot · 2020-05-12T12:51:20Z

@vdemeester:

In response to this:

Looks good to me
/meow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tekton-robot · 2020-05-12T12:51:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tekton-robot · 2020-05-12T16:14:14Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	73.3%	74.6%	1.4

tekton-robot · 2020-05-12T16:52:15Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	73.3%	74.6%	1.4

afrittoli · 2020-05-12T18:12:07Z

>> Waiting for tests to finish for pipelinerun
ERROR: tests timed out

@vdemeester perhaps we are at the time limit with alpha + beta + the new test? Shall I increase the timeout a bit?

afrittoli · 2020-05-12T20:35:21Z

/test pull-tekton-pipeline-integration-tests

afrittoli · 2020-05-12T20:40:43Z

I wonder if the new pool could be related to the timeout @bobcatfish?
I couldn't find a way to see if there is any variation in test duration... or even to see for sure how the last test run took :(

vdemeester · 2020-05-13T09:33:30Z

/test pull-tekton-pipeline-integration-tests

tekton-robot · 2020-05-14T09:28:57Z

This PR cannot be merged: expecting exactly one kind/ label

Available kind/ labels are:

kind/bug: Categorizes issue or PR as related to a bug.
kind/flake: Categorizes issue or PR as related to a flakey test
kind/cleanup: Categorizes issue or PR as related to cleaning up code, process, or technical debt.
kind/design: Categorizes issue or PR as related to design.
kind/documentation: Categorizes issue or PR as related to documentation.
kind/feature: Categorizes issue or PR as related to a new feature.
kind/misc: Categorizes issue or PR as a miscellaneuous one.

afrittoli · 2020-05-14T09:30:20Z

/test pull-tekton-pipeline-integration-tests

afrittoli · 2020-05-14T13:10:45Z

The integration tests are failing way too often on this PR, perhaps it's better to hold this one until I get a better understanding of the reason for that.
/hold

pritidesai · 2020-05-15T00:07:57Z

pkg/reconciler/pipelinerun/pipelinerun.go

+		}
+	}
+	// Then loop by pipelinetask name over all the TaskRuns associated to Conditions
+	for pipelineTaskName, actualConditionTaskRuns := range conditionTaskRuns {


@afrittoli pr.Status.TaskRuns holds an entry for the pipelineTask associated to a condition (could be more than one condition as well) with nil TaskRunStatus and one or more checks under ConditionChecks as in pr.Status.TaskRuns[PipelineTaskRunName].ConditionChecks.

Condition containers are always under ConditionChecks and not part of the map pr.Status.TaskRuns.

For example, for a pipelineTask with two conditions, conditionTaskRuns contains two containers, one for each condition and after these condition containers are created, this query returns them as valid taskRuns associated with the current pipeline:
c.taskRunLister.TaskRuns(pr.Namespace).List(labels.SelectorFromSet(pipelineRunLabels))

so in the looping over those tasks, as they are conditionalChecks, line 974 is not executed and assumes the pipelineTask was not found in line 987 and creates a new taskRun name.

Hope its making sense 😉

Thank you so much for looking into this!

L974 is designed explicitly to only track TaskRun not associated to conditions, but I agree the logic is broken there. It works fine in case of orphaned conditions, but it doesn't in case of non-orphaned ones.

If the conditions are created and still running, there will be TaskRuns for them in the list, they can be identified as condition ones from the labels. The main TaskRun does not exists yet, however its name, under normal conditions, will already be in the PipelineRun status so that it may contain the status of the conditions:

taskrun-name (not yet created) PipelineTaskName: pipelineTaskName, Status: nil, ConditionChecks: [array of condition checks],

Since my loops are based on what I get from the TaskRun list, I do not discover the existing TaskRunName with no real TaskRun under which conditions are hosted. I create a new one, and I end up having two TaskRuns with the same conditions - I think.

I will work on fixing this. I now separated my function in two parts, one which doesn't need the controller and takes the list of taskruns as input, so that it should be easier to write extra tests.

I fixed the issue now by initializing taskRunByPipelineTask from the known PR status, and adding to it as orphaned TaskRuns are recovered.
I added much more test coverage, hopefully it will work ok this time.

tekton-robot · 2020-05-15T15:29:53Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	73.5%	75.3%	1.8

afrittoli · 2020-05-15T15:30:25Z

/hold cancel

When we reconcile a pipelinerun, we should ensure that the pipelinerun status is always in sync with the actual list of taskruns that can be provided by the taskrun informer. The only way to filter taskruns is by labels tekton.dev/pipelinerun. In case an orphaned taskrun is found, we can use the other labels on the taskrun to reconstruct the missing entry in the pipelinerun status, whether it's a missing taskrun or a missing condition check.

tekton-robot · 2020-05-15T15:42:21Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	73.5%	75.3%	1.8

afrittoli · 2020-05-15T15:52:47Z

/kind feature

tekton-robot · 2020-05-15T15:53:12Z

This PR cannot be merged: expecting exactly one kind/ label

Available kind/ labels are:

kind/bug: Categorizes issue or PR as related to a bug.
kind/flake: Categorizes issue or PR as related to a flakey test
kind/cleanup: Categorizes issue or PR as related to cleaning up code, process, or technical debt.
kind/design: Categorizes issue or PR as related to design.
kind/documentation: Categorizes issue or PR as related to documentation.
kind/feature: Categorizes issue or PR as related to a new feature.
kind/misc: Categorizes issue or PR as a miscellaneuous one.

afrittoli · 2020-05-15T16:15:14Z

/test pull-tekton-pipeline-integration-tests

afrittoli · 2020-05-15T17:09:18Z

@pritidesai @bobcatfish this should be ready now - CI is green and test coverage much improved now.

pritidesai · 2020-05-19T06:21:57Z

sorry @afrittoli havent got chance to review it, its in my todo list for tomorrow :)

pritidesai · 2020-05-20T23:12:41Z

thanks @afrittoli
/lgtm

tekton-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 7, 2020

tekton-robot requested review from bobcatfish and dibyom May 7, 2020 14:48

tekton-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 7, 2020

afrittoli force-pushed the issues/2558 branch from c0ef193 to 045723f Compare May 7, 2020 19:57

afrittoli added the kind/bug Categorizes issue or PR as related to a bug. label May 8, 2020

afrittoli force-pushed the issues/2558 branch 2 times, most recently from 66628f8 to 55216e6 Compare May 11, 2020 20:55

afrittoli changed the title ~~WIP Sync the pipelinerun status from the informers~~ Sync the pipelinerun status from the informers May 11, 2020

tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 11, 2020

afrittoli force-pushed the issues/2558 branch from 55216e6 to 028ae9e Compare May 12, 2020 08:28

vdemeester approved these changes May 12, 2020

View reviewed changes

pkg/reconciler/pipelinerun/pipelinerun_test.go Outdated Show resolved Hide resolved

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 12, 2020

afrittoli force-pushed the issues/2558 branch 3 times, most recently from 32df491 to fd37ed4 Compare May 12, 2020 16:10

afrittoli force-pushed the issues/2558 branch from fd37ed4 to c704b90 Compare May 12, 2020 16:48

afrittoli added kind/bug Categorizes issue or PR as related to a bug. and removed kind/bug Categorizes issue or PR as related to a bug. labels May 14, 2020

tekton-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 14, 2020

afrittoli mentioned this pull request May 14, 2020

DNM Force cluster dump on a succesful run #2621

Closed

3 tasks

pritidesai reviewed May 15, 2020

View reviewed changes

afrittoli force-pushed the issues/2558 branch from 3dbde91 to 9bff381 Compare May 15, 2020 15:26

tekton-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 15, 2020

tekton-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 15, 2020

afrittoli force-pushed the issues/2558 branch from 9bff381 to b273801 Compare May 15, 2020 15:39

tekton-robot added the kind/feature Categorizes issue or PR as related to a new feature. label May 15, 2020

afrittoli removed the kind/bug Categorizes issue or PR as related to a bug. label May 15, 2020

afrittoli mentioned this pull request May 19, 2020

Include TaskRun Created by Condition in List of TaskRuns Under PipelineRun Status #2438

Closed

tekton-robot assigned pritidesai May 20, 2020

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label May 20, 2020

tekton-robot merged commit 1805671 into tektoncd:master May 21, 2020

imjasonh mentioned this pull request Sep 1, 2020

Create Run resources from PipelineTask references #3115

Closed

Sync the pipelinerun status from the informers #2573

Sync the pipelinerun status from the informers #2573

Uh oh!

Conversation

afrittoli commented May 7, 2020

Changes

Submitter Checklist

Reviewer Notes

Uh oh!

afrittoli commented May 7, 2020

Uh oh!

tekton-robot commented May 7, 2020

Uh oh!

tekton-robot commented May 11, 2020

Uh oh!

tekton-robot commented May 12, 2020

Uh oh!

afrittoli commented May 12, 2020

Uh oh!

vdemeester left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tekton-robot commented May 12, 2020

Uh oh!

tekton-robot commented May 12, 2020

Uh oh!

tekton-robot commented May 12, 2020

Uh oh!

tekton-robot commented May 12, 2020

Uh oh!

afrittoli commented May 12, 2020

Uh oh!

afrittoli commented May 12, 2020

Uh oh!

afrittoli commented May 12, 2020

Uh oh!

vdemeester commented May 13, 2020

Uh oh!

tekton-robot commented May 14, 2020

Uh oh!

afrittoli commented May 14, 2020

Uh oh!

afrittoli commented May 14, 2020

Uh oh!

pritidesai May 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afrittoli May 15, 2020

Choose a reason for hiding this comment

Uh oh!

afrittoli May 15, 2020

Choose a reason for hiding this comment

Uh oh!

tekton-robot commented May 15, 2020

Uh oh!

afrittoli commented May 15, 2020

Uh oh!

tekton-robot commented May 15, 2020

Uh oh!

afrittoli commented May 15, 2020

Uh oh!

tekton-robot commented May 15, 2020

Uh oh!

afrittoli commented May 15, 2020

Uh oh!

afrittoli commented May 15, 2020

Uh oh!

pritidesai commented May 19, 2020

Uh oh!

pritidesai commented May 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

pritidesai May 15, 2020 •

edited

Loading