BBC: Blead breaks BIGJ/Lingua-DE-ASCII-0.14.tar.gz

[andk]

Description

BIGJ/Lingua-DE-ASCII-0.14.tar.gz test suite has started to fail with 5.35.10. Test summary output:

Test Summary Report
-------------------
t/special_characters.t         (Wstat: 256 Tests: 204 Failed: 1)
  Failed test:  65
  Non-zero exit status: 1
t/to_latin1.t                  (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=6, Tests=209,  5 wallclock secs ( 0.04 usr  0.01 sys +  3.67 cusr  0.11 csys =  3.83 CPU)
Result: FAIL
Failed 2/6 test programs. 2/209 subtests failed.

Bisect leads to:

271c3af797311222713e1573330fe45bb5609e6a is the first bad commit
commit 271c3af797311222713e1573330fe45bb5609e6a
Author: Yves Orton <demerphq@gmail.com>
Date:   Sun Feb 20 10:38:09 2022 +0100

    regex engine: Issue #19168 - Fix variable length lookbehind matches

Steps to Reproduce

cpan -i BIGJ/Lingua-DE-ASCII-0.14.tar.gz

Expected behavior

Should test the distro successfully and install it

Perl configuration

# perl -V output goes here
Summary of my perl5 (revision 5 version 35 subversion 10) configuration:
  Commit id: 271c3af797311222713e1573330fe45bb5609e6a
  Platform:
    osname=linux
    osvers=5.16.0-5-amd64
    archname=x86_64-linux
    uname='linux k93msid 5.16.0-5-amd64 #1 smp preempt debian 5.16.14-1 (2022-03-15) x86_64 gnulinux '
    config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.35.9-11-g271c3af797/da1c -Dmyhostname=k93msid -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Dlibswanted=cl pthread socket inet nsl gdbm dbm malloc dl ld sun m crypt sec util c cposix posix ucb BSD gdbm_compat -Uuseithreads -Uuselongdouble -DEBUGGING=-g'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='cc'
    ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O2 -g'
    cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='11.2.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib
    libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.33.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.33'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_TIMES
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_DEVEL
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
  Built under linux
  Compiled at Mar 23 2022 06:30:34
  @INC:
    /home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.35.9-11-g271c3af797/da1c/lib/site_perl/5.35.10/x86_64-linux
    /home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.35.9-11-g271c3af797/da1c/lib/site_perl/5.35.10
    /home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.35.9-11-g271c3af797/da1c/lib/5.35.10/x86_64-linux
    /home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.35.9-11-g271c3af797/da1c/lib/5.35.10

[hvds]

The first test fail reduces to a difference in matching of "abcdef" =~ m{(?<=(?<!yz)b)c}: after 271c3af this no longer matches, when it should. Reducing the nested negative lookbehind from length-2 to length-1 allows it to match. Not yet checked, but this may be a special optimization kicking in when a fixed-length lookbehind would need to match before start of string.

debug output before and after 271c3af

Before:

Compiling REx "(?<=(?<!yz)b)c"
Final program:
   1: IFMATCH[-1] (13)
   3:   UNLESSM[-2] (9)
   5:     EXACT <yz> (7)
   7:     SUCCEED (0)
   8:   TAIL (9)
   9:   EXACT <b> (11)
  11:   SUCCEED (0)
  12: TAIL (13)
  13: EXACT <c> (15)
  15: END (0)
anchored "c" at 0..0 (checking anchored) minlen 1 
Enabling $` $& $' support (0x7).

EXECUTING...

Matching REx "(?<=(?<!yz)b)c" against "abcdef"
Intuit: trying to determine minimum start position...
  doing 'check' fbm scan, [0..6] gave 2
  Found anchored substr "c" at offset 2 (rx_origin now 2)...
  (multiline anchor test skipped)
  try at offset...
Intuit: Successfully guessed: match at offset 2
   2 <ab> <cdef>             |   0| 1:IFMATCH[-1](13)
   1 <a> <bcdef>             |   1|  3:UNLESSM[-2](9)
   1 <a> <bcdef>             |   1|  9:EXACT <b>(11)
   2 <ab> <cdef>             |   1|  11:SUCCEED(0)
                             |   1|  SUCCEED: subpattern success...
   2 <ab> <cdef>             |   0| 13:EXACT <c>(15)
   3 <abc> <def>             |   0| 15:END(0)
Match successful!
Freeing REx: "(?<=(?<!yz)b)c"

After:

Compiling REx "(?<=(?<!yz)b)c"
Final program:
   1: IFMATCH[-1] (13)
   3:   UNLESSM[-2] (9)
   5:     EXACT <yz> (7)
   7:     LOOKBEHIND_END (0)
   8:   TAIL (9)
   9:   EXACT <b> (11)
  11:   LOOKBEHIND_END (0)
  12: TAIL (13)
  13: EXACT <c> (15)
  15: END (0)
anchored "c" at 0..0 (checking anchored) minlen 1 
Enabling $` $& $' support (0x7).

EXECUTING...

Matching REx "(?<=(?<!yz)b)c" against "abcdef"
Intuit: trying to determine minimum start position...
  doing 'check' fbm scan, [0..6] gave 2
  Found anchored substr "c" at offset 2 (rx_origin now 2)...
  (multiline anchor test skipped)
  try at offset...
Intuit: Successfully guessed: match at offset 2
   2 <ab> <cdef>             |   0| 1:IFMATCH[-1](13)
   1 <a> <bcdef>             |   1|  3:UNLESSM[-2](9)
   1 <a> <bcdef>             |   1|  9:EXACT <b>(11)
   2 <ab> <cdef>             |   1|  11:LOOKBEHIND_END(0)
                             |   1|  LOOKBEHIND_END: subpattern failed...
                             |   1|  failed...
                             |   0| failed...
Match failed
Freeing REx: "(?<=(?<!yz)b)c"

I'll take a look, but I'm sure @demerphq will be able to solve this much quicker if he gets a chance to look at it.

[hvds]

Not yet checked, but this may be a special optimization kicking in when a fixed-length lookbehind would need to match before start of string.

Confirmed this: "aabcdef" does match, for example.

It seems that match_end is getting set to pos 2 for the outer lookbehind, then to pos 1 for the inner lookbehind, then (due to the early bailout) not getting restored, so that we end up checking locinput against the wrong end point for the outer lookbehind. In particular, with "aabcdef" we end up reaching line 9534 (case IFMATCH_A):

            match_end = ST.prev_match_end;

.. but for "abcdef" we never reach that line.

I think the fix is as below, will do some tests on that.

--- a/regexec.c
+++ b/regexec.c
@@ -9484,6 +9484,7 @@ NULL
 
                 /* If the lookbehind doesn't start in the actual string, is a
                  * trivial match failure */
+                match_end = ST.prev_match_end;
                 if (logical) {
                     logical = 0;
                     sw = 1 - cBOOL(ST.wanted);

[hvds]

I've created PR #19558 with the above fix, and confirmed that Lingua-DE-ASCII passes its tests with this.

[hvds]

#19558 now merged, which fixes this.

[demerphq]

On Wed, 23 Mar 2022 at 14:15, Hugo van der Sanden ***@***.***> wrote: Not yet checked, but this may be a special optimization kicking in when a fixed-length lookbehind would need to match before start of string. Confirmed this: "aabcdef" does match, for example. It seems that match_end is getting set to pos 2 for the outer lookbehind, then to pos 1 for the inner lookbehind, then (due to the early bailout) not getting restored,

The patch you did makes sense, but i dont follow what you mean by early bailout.

so that we end up checking locinput against the wrong end point for the outer lookbehind. In particular, with "aabcdef" we end up reaching line 9534 (case IFMATCH_A): match_end = ST.prev_match_end; .. but for "abcdef" we never reach that line. I think the fix is as below, will do some tests on that. --- a/regexec.c +++ b/regexec.c @@ -9484,6 +9484,7 @@ NULL /* If the lookbehind doesn't start in the actual string, is a * trivial match failure */ + match_end = ST.prev_match_end; if (logical) { logical = 0; sw = 1 - cBOOL(ST.wanted); This makes sense to me, ill review the logic in more detail later.

Yves

…

-- perl -Mre=debug -e "/just|another|perl|hacker/"