runtimetest: add validateSeccomp #514

zhouhao3 · 2017-11-09T07:11:18Z

Signed-off-by: Zhou Hao [email protected]

wking · 2017-11-09T17:02:48Z

cmd/runtimetest/main.go

+				if name == "getcwd" {
+					_, err := os.Getwd()
+					if err == nil {
+						return fmt.Errorf("when action is %v, return value should being passed to user space as the errno value without executing the system call", sys.Action)


I don't see any runtime-oriented RFC 2119 requirements for linux.seccomp, which means we only have the config-oriented SeccSyscallsNamesRequired in specerror, which means we have no grounds for making this a fatal error in compliance testing. I think the best we can do (at least for 1.0.x configs) is warn the caller that the runtime is skipping some not-strictly-required-but-probably-expected functionality.

Before spec adds RFC requirements, we'd better just give a warn instead of error

Mashimiao · 2017-11-10T01:55:48Z

validation/validation_test.go

+	syscallArgs := seccomp.SyscallOpts{
+		Action:   "errno",
+		Syscall:  "getcwd",
+		Index:    "0",


what are Index, Value used for in this case?

According to the code in parse_action.go:

action, _ := parseAction(arguments[0]) if action == config.DefaultAction && args.argsAreEmpty() { // default already set, no need to make changes return nil }

If args is empty, you can't add syscall.

If you really want to test the Action, you should change the defautAction not equal to the specified action.
Then, empty args will not be a problem.

Mashimiao · 2017-11-16T03:41:55Z

cmd/runtimetest/main.go

+				if name == "getcwd" {
+					_, err := os.Getwd()
+					if err == nil {
+						logrus.Warnf("when action is %v, return value should being passed to user space as the errno value without executing the system call", sys.Action)


I think the warning is not suitable, that's not what we really want to tell users.
We want to tell them even seccomp is configured but this runtime can not apply it or similar warning

Mashimiao · 2017-11-17T08:15:40Z

LGTM

wking · 2017-12-13T19:16:47Z

cmd/runtimetest/main.go

+	for _, sys := range spec.Linux.Seccomp.Syscalls {
+		if sys.Action == "SCMP_ACT_ERRON" {
+			for _, name := range sys.Names {
+				if name == "getcwd" {


Can we write t.Skips for other names and actions? I don't want to give users the false impression that we're testing them by silently ignoring other values.

Can we write t.Skips…

This is probably blocking on passing the TAP instance down into the validators, which I do in #308. I don't see a clean way to handle this case before #308 lands; if maintainers want a stub in the meantime that's fine, and I don't really care if/how this gets handled in the stub.

Does that mean that all the action and name combinations will be exported as skipped information? Maybe I misunderstood.

Does that mean that all the action and name combinations will be exported as skipped information?

Yes, but only if a configuration uses them. That would make it easy for folks writing tests in validation to see that runtimetest was ignoring something which the configuration had requested.

wking · 2017-12-13T19:29:23Z

cmd/runtimetest/main.go

+				if name == "getcwd" {
+					_, err := os.Getwd()
+					if err == nil {
+						logrus.Warnf("Syscall action %v can not be properly applied in the runtime", sys.Action)


This is a non-TAP warning that will be gobbled by RuntimeInsideValidate. I think you want something like:

t.Skip(1, fmt.Sprintf("%s syscall returns errno", sys.Name)) if err == nil { t.Diagnostic("did not return an error") } else { t.Diagnosticf("returned %d (%s)", err.WhateverOtherErrno(), err.Error()) }

I tried to modify it, what do you think?

wking · 2017-12-13T19:30:43Z

cmd/runtimetest/main.go

+		return nil
+	}
+	for _, sys := range spec.Linux.Seccomp.Syscalls {
+		if sys.Action == "SCMP_ACT_ERRON" {


SCMP_ACT_ERRON → SCMP_ACT_ERRNO.

wking · 2017-12-27T17:51:25Z

cmd/runtimetest/main.go

+	if spec.Linux == nil || spec.Linux.Seccomp == nil {
+		return nil
+	}
+	t := tap.New()


This TAP-within-a-TAP approach is a bit strange, but I think it might work… We can drop this once #308 lands and the t instance is getting passed around.

Some other modifications can wait until #308 land. Thank you for your advice.

wking · 2017-12-27T17:55:53Z

cmd/runtimetest/main.go

+				if name == "getcwd" {
+					_, err := os.Getwd()
+					if err == nil {
+						t.Diagnostic("Syscall action ERRNO can not be properly applied in the runtime")


Maybe the diagnostic could be:

getcwd did not return an error

Once #308 lands with a local test, that test will be the %s syscall returns errno you have in the skip below, so the total output will be:

not ok … - getcwd syscall returns errno # getcwd did not return an error

wking · 2017-12-27T17:57:44Z

cmd/runtimetest/main.go

+						t.Diagnostic("Syscall action ERRNO can not be properly applied in the runtime")
+					}
+				} else {
+					t.Skip(i, fmt.Sprintf("%s syscall returns errno", name))


This should be t.Skip(1, …), since we're only skipping one test.

wking · 2017-12-27T17:58:26Z

cmd/runtimetest/main.go

+					t.Skip(i, fmt.Sprintf("%s syscall returns errno", name))
+				}
+			}
+		}


And here we want an else case saying that we're skipping the action:

} else { t.Skip(1, fmt.Sprintf("syscall action %s", sys.Action)) }

or similar.

Signed-off-by: Zhou Hao <[email protected]>

zhouhao3 · 2018-01-02T03:28:04Z

@liangchenye @Mashimiao PTAL

Mashimiao · 2018-01-02T14:30:46Z

LGTM

zhouhao3 · 2018-01-03T02:22:30Z

ping @liangchenye

liangchenye · 2018-01-03T02:25:14Z

LGTM

zhouhao3 force-pushed the runtimetest-seccomp branch 3 times, most recently from 0abf4c1 to 3a9ec3a Compare November 9, 2017 07:34

wking reviewed Nov 9, 2017

View reviewed changes

Mashimiao reviewed Nov 10, 2017

View reviewed changes

zhouhao3 force-pushed the runtimetest-seccomp branch 4 times, most recently from 4b330b9 to 59c6c2d Compare November 10, 2017 07:14

Mashimiao added this to the v0.4.0 milestone Nov 15, 2017

zhouhao3 force-pushed the runtimetest-seccomp branch from 59c6c2d to 2138ac8 Compare November 16, 2017 03:27

Mashimiao reviewed Nov 16, 2017

View reviewed changes

zhouhao3 force-pushed the runtimetest-seccomp branch 2 times, most recently from 3f1f3ca to f363ae3 Compare November 17, 2017 02:57

zhouhao3 force-pushed the runtimetest-seccomp branch 2 times, most recently from 0a6e9ae to bd222f9 Compare December 13, 2017 07:58

wking reviewed Dec 13, 2017

View reviewed changes

zhouhao3 force-pushed the runtimetest-seccomp branch 2 times, most recently from 87a2e04 to 4907259 Compare December 27, 2017 06:45

wking reviewed Dec 27, 2017

View reviewed changes

runtimetest: add validateSeccomp

5bb8754

Signed-off-by: Zhou Hao <[email protected]>

zhouhao3 force-pushed the runtimetest-seccomp branch from 4907259 to 5bb8754 Compare December 28, 2017 05:42

liangchenye merged commit a002fc8 into opencontainers:master Jan 3, 2018

zhouhao3 deleted the runtimetest-seccomp branch January 3, 2018 02:28

runtimetest: add validateSeccomp #514

runtimetest: add validateSeccomp #514

Uh oh!

Conversation

zhouhao3 commented Nov 9, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mashimiao commented Nov 17, 2017 • edited by caniszczyk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wking Dec 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhouhao3 commented Jan 2, 2018

Uh oh!

Mashimiao commented Jan 2, 2018 • edited by caniszczyk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhouhao3 commented Jan 3, 2018

Uh oh!

liangchenye commented Jan 3, 2018 • edited by caniszczyk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Mashimiao commented Nov 17, 2017 •

edited by caniszczyk

Loading

wking Dec 26, 2017 •

edited

Loading

Mashimiao commented Jan 2, 2018 •

edited by caniszczyk

Loading

liangchenye commented Jan 3, 2018 •

edited by caniszczyk

Loading