custom-http-errors breaks modsecurity transaction info

[michalschott]

NGINX Ingress controller version: 0.30, 0.32

Kubernetes version (use kubectl version): 1.16.9 / 1.16.10

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): flatcar stable
• Kernel (e.g. uname -a): 4.19.124-flatcar
• Install tools: kops
• Others:

What happened:

Modsecurity does not print out transaction info into auditlog if custom-http-error annotation / configmap is being set.

What you expected to happen:

Transaction info should be printed out.

How to reproduce it:

1. Install nginx-ingress, assume this is added as arg: - --configmap=NAMESPACE/nginx-ingress-controller
2. enable modsecurity plugin (I just add this into nginx-ingress-controller configmap)

enable-modsecurity: "true"
enable-owasp-modsecurity-crs: "true"

3. I'm mounting modsecurity.conf file as a volume because I can not make it work with configmap/annotation snippets

4. ensure this is present in your modsecurity.conf file:

SecRuleEngine On
# -- Audit log configuration -------------------------------------------------

# Log the transactions that are marked by a rule, as well as those that
# trigger a server error (determined by a 5xx or 4xx, excluding 404,
# level response status codes).
#
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"

# Log everything we know about a transaction.
SecAuditLogParts ABIJDEFHZ

# Use a single file for logging. This is much easier to look at, but
# assumes that you will use the audit log only ocassionally.
#
SecAuditLogType Serial
#SecAuditLog /var/log/modsec_audit.log
SecAuditLog /dev/stdout

# log format
SecAuditLogFormat JSON

# Specify the path for concurrent audit logging.
#SecAuditLogStorageDir /opt/modsecurity/var/audit/

# SecDebugLogLevel 9
# SecDebugLog /var/log/modsec-debug.log

This should give details about blocked requests, but as soon as custom-http-error is being added to either configmap or as annotation, you won't get any details.

Other approach I've tried is to use server-snippet annotation on ingress:

      error_page 404 = @custom_404;
      location @custom_404 {
        internal;

        proxy_intercept_errors off;

        proxy_set_header       X-Code             404;
        proxy_set_header       X-Format           $http_accept;
        proxy_set_header       X-Original-URI     $request_uri;
        proxy_set_header       X-Namespace        $namespace;
        proxy_set_header       X-Ingress-Name     $ingress_name;
        proxy_set_header       X-Service-Name     $service_name;
        proxy_set_header       X-Service-Port     $service_port;
        add_header             Referrer-Policy    "no-referrer";
        add_header             X-Content-Type-Options nosniff;
        add_header             X-Frame-Options    SAMEORIGIN;
        add_header             X-XSS-Protection   "1; mode=block";

        set $proxy_upstream_name "upstream-default-backend";

        more_set_headers       "Strict-Transport-Security: max-age=31536000; includeSubDomains";

        rewrite                (.*) / break;

        proxy_pass            http://upstream_balancer;

      }

Anything else we need to know:

I think that modsecurity documentation could have an update about various scenarios how to set this up properly ie difference between enabling this on global/server level vs ingress only, or how to disable checks on localhost:

nginx-ingress-controller-5cbc45cc96-h65s9 nginx-ingress-controller {"transaction":{"client_ip":"127.0.0.1","time_stamp":"Mon Jun  8 12:37:44 2020","server_id":"3155bd55f0b49853acd59a77664121da939ce1c2","client_port":54262,"host_ip":"127.0.0.1","host_port":10246,"unique_id":"159161986417.546039","request":{"method":"GET","http_version":1.1,"uri":"/is-dynamic-lb-initialized","headers":{"Host":"127.0.0.1:10246","User-Agent":"Go-http-client/1.1","Accept-Encoding":"gzip"}},"response":{"body":"OK\n","http_code":200,"headers":{"Server":"","Server":"","Date":"Mon, 08 Jun 2020 12:37:44 GMT","Content-Type":"text/html","Connection":"close","Feature-Policy":"geolocation 'none';midi 'none';notifications 'none';push 'none';sync-xhr 'none';microphone 'none';camera 'none';magnetometer 'none';gyroscope 'none';speaker 'self';vibrate 'none';fullscreen 'self';payment 'none';"}},"producer":{"modsecurity":"ModSecurity v3.0.3 (Linux)","connector":"ModSecurity-nginx v1.0.1","secrules_engine":"DetectionOnly","components":["OWASP_CRS/3.2.0\""]},"messages":[{"message":"Host header is a numeric IP address","details":{"match":"Matched \"Operator `Rx' with parameter `^[\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `127.0.0.1:10246' )","reference":"o0,15v46,15","ruleId":"920350","file":"/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf","lineNumber":"678","data":"127.0.0.1:10246","severity":"4","ver":"OWASP_CRS/3.2.0","rev":"","tags":["application-multi","language-multi","platform-multi","attack-protocol","OWASP_CRS","OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST","WASCTC/WASC-21","OWASP_TOP_10/A7","PCI/6.5.10"],"maturity":"0","accuracy":"0"}}]}}

/kind bug
/kind documentation

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ecktom]

I had the same issue and was able to get it working using the following setup:

nginx-conf:

  custom-http-errors: "404,403,503"
  enable-modsecurity: "true"
  modsecurity-snippet: |
    # Log the transactions that are marked by a rule, as well as those that
    # trigger a server error (determined by a 5xx or 4xx, excluding 404,
    # level response status codes).
    SecAuditEngine RelevantOnly
    SecAuditLogRelevantStatus "^(?:5|4(?!04))"
    # Log to stdout to allow forwarding logs
    SecAuditLogParts ABIJDEFHZ
    SecAuditLogType Serial
    SecAuditLog /dev/stdout
    SecAuditLogFormat JSON

annotations on ingress:

    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_intercept_errors off;
    nginx.ingress.kubernetes.io/server-snippet: |
      modsecurity_rules_file /etc/nginx/modsecurity/custom/nginx-modsecurity.conf;

mounted nginx-modsecurity.conf:

    SecRuleEngine On
    ... some Rules...
    Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

I've not tested how to set this up globally but would assume it works with the server-wide snippets as well. Putting modsecurity_rules_file to the configuration-snippet like shown below did not work and resulted in the missing audit log.
Also note you might not need proxy_intercept_errors if you're fine with upstream errors getting intercepted.

    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_intercept_errors off;
      modsecurity_rules_file /etc/nginx/modsecurity/custom/nginx-modsecurity.conf;

[kolesaev]

My Modsecurity was configured and working very well with JSON log format, like

{"transaction":{"client_ip":"15.237.69.154","time_stamp":"Tue Aug 22 12:23:18 2023","server_id":"d70d8c6bad2613fdec0277ef5dc4c823ef823aa6","client_port":16620,"host_ip":"10.0.0.170","host_port":443,"unique_id":"16927069987.326001","request":{"method":"POST","http_version":2.0,"uri":"/api/v1/path-example","headers":{"host":"payment-widget.example.com","content-type":"application/json","user-agent":"Blackbox Exporter/0.24.0","content-length":"15"}},"response":{"body":"","http_code":401,"headers":{"Strict-Transport-Security":"max-age=15724800; includeSubDomains","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","X-Content-Type-Options":"nosniff","Expires":"0","WWW-Authenticate":"Bearer","traceId":"f89e21bb-6ef2-4b79-93c6-f2fc10b13c61","Server":"","Server":"","Pragma":"no-cache","X-XSS-Protection":"1; mode=block","Connection":"close","Date":"Tue, 22 Aug 2023 12:23:18 GMT","X-Frame-Options":"DENY"}},"producer":{"modsecurity":"ModSecurity v3.0.8 (Linux)","connector":"ModSecurity-nginx v1.0.3","secrules_engine":"Enabled","components":["OWASP_CRS/3.3.4\""]},"messages":[]}}

, but when we have enabled custom-error-pages, we got nice error pages, but modsecurity moved from JSON log format to the following format

2023/08/22 13:19:27 [error] 101#101: *1490 [client 95.25.161.104] ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `5' ) [file "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "81"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.3.4"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "10.0.37.108"] [uri "/.env"] [unique_id "169271036775.026488"] [ref ""], client: 95.25.161.104, server: ui.dev.tech.calypso.cash, request: "GET /.env HTTP/2.0", host: "example.com"

Our modsecurity audit log configs were

# -- Audit log configuration -------------------------------------------------
# Log the transactions that are marked by a rule, as well as those that
# trigger a server error (determined by a 5xx or 4xx, excluding 404,  
# level response status codes).
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"

# Log everything we know about a transaction.
SecAuditLogParts ABIJDEFHZ

# Use a single file for logging. This is much easier to look at, but
# assumes that you will use the audit log only ocassionally.
SecAuditLogType Serial
SecAuditLog /dev/stdout
SecAuditLogFormat JSON

# Specify the path for concurrent audit logging.
SecAuditLogStorageDir /var/log/audit/

I have tried SecAuditLogType as Parallel and as Concurrent to try getting JSON audit logs into files, but doesn't help, there are no any audit log files.

Also with configs below, but files are created, but there are no logs written to them

SecAuditLog /var/log/audit/modsec_audit.log
SecAuditLog2 /var/log/audit/modsec_audit_2.log

@ecktom 's Fixes didn't help

@michalschott have you found any fix here?

Please anyone who knows how to fix it let me know

Upd. Manual edit nginx config with reload nginx helps.

sed -i "s|modsecurity off;|#modsecurity off;|g" nginx.conf
nginx -t && nginx -s reload

As i understood, it happening because there is modsecurity off; in error locations. I'm researching how to fix

[kolesaev]

Hi

I created a new nginx.tmpl as a configmap and mounted it to /etc/nginx/template/nginx.tmpl as a volume

Default config contained modsecurity off; and was looking like below

{{ define "CUSTOM_ERRORS" }}
        {{ $enableMetrics := .EnableMetrics }}
        {{ $modsecurityEnabled := .ModsecurityEnabled }}
        {{ $upstreamName := .UpstreamName }}
        {{ range $errCode := .ErrorCodes }}
        location @custom_{{ $upstreamName }}_{{ $errCode }} {
            internal;

            # Ensure that modsecurity will not run on custom error pages or they might be blocked
            {{ if $modsecurityEnabled }}
            modsecurity off;
            {{ end }}

            proxy_intercept_errors off;

            proxy_set_header       X-Code             {{ $errCode }};
            proxy_set_header       X-Format           $http_accept;

I added exception for code 403

{{ define "CUSTOM_ERRORS" }}
        {{ $enableMetrics := .EnableMetrics }}
        {{ $modsecurityEnabled := .ModsecurityEnabled }}
        {{ $upstreamName := .UpstreamName }}
        {{ range $errCode := .ErrorCodes }}
        location @custom_{{ $upstreamName }}_{{ $errCode }} {
            internal;

            # Ensure that modsecurity will not run on custom error pages or they might be blocked
            {{ if and $modsecurityEnabled (ne $errCode 403) }} # here
            modsecurity off;
            {{ end }}

            proxy_intercept_errors off;

            proxy_set_header       X-Code             {{ $errCode }};
            proxy_set_header       X-Format           $http_accept;

Hope it helps you