Rule organization for binaries and scripts

[mr-tz]

Per the work in #1080 capa will support source code/scripts. Similar to the discussion on ELF/PE files (#701) we should decide how to organize and write new rules. This could include the creation of new features and/or changes to the rule syntax.

While the concerns and points from the ELF/PE still hold, the analysis of source code/scripts and compiled binaries differs enough to discuss new solutions here. To reiterate, the concerns are:

• avoiding cross-OS matches
• performance
• intuitive capability documentation for humans
• duplication of meta data and file directory structure
• possibilities to leverage higher-level rules
• backwards compatibility and/or breaking changes requiring a new major release

[mr-tz]

binary and script rules in one file

This is how we ended up organizing ELF/PE (for the most part).

An example rule may look like this (see addition in the last line, using function-name here per Adam's current work [topic for another discussion]):

rule:
  meta:
    name: encode data using Base64
    namespace: data-manipulation/encoding/base64
    ...
  features:
    - or:
      - and:
        - mnemonic: shl
        - mnemonic: shr
        - number: 0x3F = modulo 64
        - or:
          - number: 0x3D = '='
          - number: 0x3D3D = '=='
        - match: contain loop
        - optional:
          - number: 2
          - number: 3
          - number: 4
          - number: 6
          - number: 0xF
          - string: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
      - api: System.Convert::ToBase64String
      - function-name: System.Convert.ToBase64String

This looks nice and comes with all the previously shown pros such as easy documentation and re-use of meta data. For rules that require more logic, files could be split up.

Do we still want to use this approach when we support multiple scripting languages? Do we want to add a new key qualifier?

      - function-name/c#: System.Convert.ToBase64String
      - function-name/python: base64.b64encode
      - ...

[adamstorek]

In my opinion, the second approach would make for clearer rules, but it would also require changes to the parser. The second approach is easier to parse and more consistent, however, is it a good approach to mix source code/binary constructs? What if the scope differs for the two?

[mr-tz]

is it a good approach to mix source code/binary constructs?

That's a great question. From the rules I've seen so far I think it could work. However, if we end up with much more supported languages and concepts it will become messy and we'll have to create a new file organization or concept.

[williballenthin]

when a rule matches multiple "formats", i'd typically like to see guards like:

  - and:
    - format: pe
    - os: windows
    - api: CreateFileA
  - and:
    - format: python
    - api: os.open

we haven't been consistent about this today, because when there's just one or two possible formats its pretty obvious which is which. but with the proliferation of .NET and scripting, we should try to be much better at this.

(furthermore, we can imagine an optimization pass done by the rule matcher to prune subtrees that will never match when there's a format restriction.)

[williballenthin]

style-wise, i think when we're looking for single APIs to match functionality, its reasonable to mix formats within a rule. but once there's complicated logic, we should split rules up for the benefit of human readers.