os, arch, and format features

[williballenthin]

Implementation of ELF-related features, ref #699

adds:

• OS features: os: windows, os: linux, os: macos, etc. at global scope (insn, bb, function, AND file scopes) closes detect OS from ELF file #724
• file format features: format: pe, format: elf at file scope
• arch features: arch: amd64 and arch: i386 at global scope
• tests demonstrating feature extraction on ELF files

extractors are now expected to extract each of the above. implements these for the viv, smda, pefile, and ida extractors. documentation for these features is found in mandiant/capa-rules#443

breaking changes:

• rename legacy term arch to bitness to make space for arch: feature

this means we'll need to do a major release (v3.0.0); however, the new features os, format, and arch also are breaking changes, so there's no getting around this.

does not attempt:

• the meta.os key. this can be added subsequently in another PR.

demo

[github-actions]

Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the master (unreleased) section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: [x] No CHANGELOG update needed

CHANGELOG updated or no update needed, thanks! 😄

[williballenthin]

requesting initial review to ensure i'm on the right track and making sense.

[williballenthin]

do we need to add a is_supported_arch function?
we should verify supported file types (and arch, see above) before firing the SMDA extractor
we will need to update output for the standalone tool

04cc94a

[williballenthin]

we should add OS to collected metadata and I would also vote we separate arch, bitness, and file format from the format key

1b9a6c3

[williballenthin]

eventually we will want to implement file-scoped limitation checks for all supported file types similar to what we currently do with PE files

agree, though at the moment, i don't think we would have any file limitation rules that would fire on ELF binaries. i'd propose to implement the limitation check only once we have those rules, since we'll have to write even more new code to do the lightweight ELF feature extraction (and this PR is already big).

[williballenthin]

@mike-hunhoff items identified above are addressed. would you review those few commits? anything further you'd want done before merging?

[mr-tz]

impressive, thanks for the great work here!

[mike-hunhoff]

updates look good, thank you!

[Ana06]

Amazing work!!! 🥇

[williballenthin]

ugh viv ELF support seems like it might have been broken in v1.0.4?

https://github.com/fireeye/capa/runs/3404104636

#735

reported here vivisect/vivisect#443 fix here vivisect/vivisect#444

[williballenthin]

this is ready to go