gsbLookup does not perform canonicalization and lookups correctly

[rcbarnett-zz]

MODSEC-227: The gsbLookup operator does not perform canonicalization correctly. The example from the Safe Browsing manual (http://a.b.c/1/2.html?param=1) results in two lookups:

GSB: Successfully extracted url: a.B.c/1/2.Html?param=1
GSB: Canonicalize url #2: a.B.c/

whereas the manual specifies many more:

a.b.c/1/2.html?param=1
a.b.c/1/2.html
a.b.c/
a.b.c/1/
b.c/1/2.html?param=1
b.c/1/2.html
b.c/
b.c/1/

Also notice that gsbLookup does not currently transform input to lowercase (but it should).

[rcbarnett-zz]

Original reporter: ivanr

[rcbarnett-zz]

bpinto: Ivan,

We are doing now the most common lookups (full url and base urls). We will improve it in minor versions of 2.6.x.
Also.. transformation must be used when users want to use gsblookup.

[rcbarnett-zz]

ivanr: I disagree with your assessment and the change of the type to "Improvement". The current implementation of gsbLookup does not follow the specification and is thus faulty. Even if you accept that only 2 lookups are acceptable, the canonicalization issue still remains. For example, when sent (notice the dot at the end of the hostname):

Target value: "http://a....b.c/1/2.html?param=1"
GSB: Successfully extracted url: a....b.c/1/2.html?param=1
GSB: Canonicalize url #2: a....b.c/

The operator will completely miss the attack.

[rcbarnett-zz]

bpinto: For now we are doing the lookups when we see a url a.b.c/a/b/index.html

1 - a.b.c/a/b/index.html
2 - a.b.c/

We will implement the full transformations soon

[rcbarnett-zz]

bpinto: Added more improvements:

1 - reduce .. to .
2 - parser supports . in the end of domain
3 - better domain parser

• a.b.c.d../
  a.b.c.d/
  b.c.d/
  c.d/

[rcbarnett-zz]

bpinto: Added reduce multiple slashes in a single slash

[rcbarnett-zz]

bpinto: This is what we will have for 2.6.0

Example : a.b.c/1/2.html?param=1
1- a.b.c/1/2.html?param=1
2- a.b.c/1/2.html
3- a.b.c/
4- a.b.c/1/
5- b.c/1/2.html?param=1
6- b.c/1/2.html
7- b.c/
8- b.c/1/

Also :

• multiple slashes in a single one
• Consecutive dots in a single dot