vm: fix modmul operation

[Slava0135]

Problem

Resolves #3598

Solution

Use Rem operation instead of Mod, same as in MOD opcode

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.19%. Comparing base (fb4a4f0) to head (9aca090).
Report is 311 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3599      +/-   ##
==========================================
- Coverage   86.10%   85.19%   -0.92%     
==========================================
  Files         331      333       +2     
  Lines       38107    39007     +900     
==========================================
+ Hits        32813    33231     +418     
- Misses       3780     4206     +426     
- Partials     1514     1570      +56     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[AnnaShaleva]

Yep, it shouldn't be a Euclidian division, a really nice catch!