Test failures from the Convex Problem Depot

[ericphanson]

These don't seem like upstream issues, because they are just using Float64 on Convex.jl tests which are passed by other solvers.

From https://travis-ci.com/ericphanson/SDPA_GMP.jl/jobs/232679443, which is using the MathOptInterface2 branch of my Convex fork.

• affine_Partial_transpose. This one is extra strange to me because it only solves trivial problems in the partial transpose tests.

affine_Partial_transpose: Test Failed at /home/travis/.julia/packages/Convex/Mnzj6/src/problem_depot/problems/affine.jl:620
  Expression: ≈(partialtranspose(S, 2, dims), evaluate(Rt2), atol=atol, rtol=rtol)
   Evaluated: Complex{Float64}[0.383852+0.528893im 0.742664+0.24492im … 0.308696+0.210021im 0.364071+0.951709im; 0.690416+0.583909im 0.510558+0.988567im … 0.651434+0.32428im 0.327982+0.737351im; … ; 0.545792+0.570356im 0.00173464+0.360559im … 0.157738+0.138702im 0.806053+0.956032im; 0.813276+0.481045im 0.867989+0.432329im … 0.920658+0.69532im 0.422239+0.321757im] ≈ Complex{Float64}[0.0+0.0im 0.0+0.0im … 0.0+0.0im 0.0+0.0im; 0.0+0.0im 0.0+0.0im … 0.0+0.0im 0.0+0.0im; … ; 0.0+0.0im 0.0+0.0im … 0.0+0.0im 0.0+0.0im; 0.0+0.0im 0.0+0.0im … 0.0+0.0im 0.0+0.0im] (atol=0.001, rtol=0.0)
Stacktrace:
 [1] affine_Partial_transpose(::getfield(Main, Symbol("##12#14")){String}, ::Val{true}, ::Float64, ::Float64, ::Type{Float64}) at /home/travis/.julia/packages/Convex/Mnzj6/src/problem_depot/problems/affine.jl:620
 [2] macro expansion at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:11 [inlined]
 [3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [4] (::getfield(Main, Symbol("##11#13")))(::String, ::typeof(Convex.ProblemDepot.affine_Partial_transpose)) at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:11

• affine_Diagonal_atom test

affine_Diagonal_atom: Test Failed at /home/travis/.julia/packages/Convex/Mnzj6/src/problem_depot/problems/affine.jl:466
  Expression: all(abs.(evaluate(Diagonal(x)) - Diagonal([1, 2, 3, 4])) .<= atol)
Stacktrace:
 [1] affine_Diagonal_atom(::getfield(Main, Symbol("##12#14")){String}, ::Val{true}, ::Float64, ::Float64, ::Type{Float64}) at /home/travis/.julia/packages/Convex/Mnzj6/src/problem_depot/problems/affine.jl:466
 [2] macro expansion at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:11 [inlined]
 [3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [4] (::getfield(Main, Symbol("##11#13")))(::String, ::typeof(Convex.ProblemDepot.affine_Diagonal_atom)) at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:11

• affine_Diagonal_atom presolve

affine_Diagonal_atom: Error During Test at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:10
  Got exception outside of a @test
  SDPA_GMP.PresolveError("Inconsistency at constraint index 2. Problem is dual infeasible.")
  Stacktrace:
   [1] presolve(::SDPA_GMP.Optimizer{Float64}) at /home/travis/build/ericphanson/SDPA_GMP.jl/src/presolve.jl:36
   [2] initializeSolve(::SDPA_GMP.Optimizer{Float64}) at /home/travis/build/ericphanson/SDPA_GMP.jl/src/file_io.jl:184
   [3] optimize!(::SDPA_GMP.Optimizer{Float64}) at /home/travis/build/ericphanson/SDPA_GMP.jl/src/MOI_wrapper.jl:301
   [4] optimize!(::MathOptInterface.Utilities.CachingOptimizer{SDPA_GMP.Optimizer{Float64},MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}) at /home/travis/.julia/packages/MathOptInterface/WXD0G/src/Utilities/cachingoptimizer.jl:189
   [5] optimize!(::MathOptInterface.Bridges.LazyBridgeOptimizer{MathOptInterface.Utilities.CachingOptimizer{SDPA_GMP.Optimizer{Float64},MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}}) at /home/travis/.julia/packages/MathOptInterface/WXD0G/src/Bridges/bridge_optimizer.jl:184
   [6] optimize!(::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.AbstractOptimizer,MathOptInterface.Utilities.Model{Float64}}) at /home/travis/.julia/packages/MathOptInterface/WXD0G/src/Utilities/cachingoptimizer.jl:189
   [7] #solve!#5(::Bool, ::Bool, ::Function, ::Problem{Float64}, ::SDPA_GMP.Optimizer{Float64}) at /home/travis/.julia/packages/Convex/Mnzj6/src/MOI_solve.jl:192
   [8] solve! at /home/travis/.julia/packages/Convex/Mnzj6/src/MOI_solve.jl:179 [inlined]
   [9] (::getfield(Main, Symbol("##12#14")){String})(::Problem{Float64}) at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:13
   [10] affine_Diagonal_atom(::getfield(Main, Symbol("##12#14")){String}, ::Val{true}, ::Float64, ::Float64, ::Type{Float64}) at /home/travis/.julia/packages/Convex/Mnzj6/src/problem_depot/problems/affine.jl:482
   [11] macro expansion at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:11 [inlined]
   [12] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
   [13] (::getfield(Main, Symbol("##11#13")))(::String, ::typeof(Convex.ProblemDepot.affine_Diagonal_atom)) at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:11
   [14] macro expansion at /home/travis/.julia/packages/Convex/Mnzj6/src/problem_depot/problem_depot.jl:103 [inlined]
   [15] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
   [16] #foreach_problem#4(::Array{Regex,1}, ::Function, ::getfield(Main, Symbol("##11#13"))) at /home/travis/.julia/packages/Convex/Mnzj6/src/problem_depot/problem_depot.jl:101
   [17] (::getfield(Convex.ProblemDepot, Symbol("#kw##foreach_problem")))(::NamedTuple{(:exclude,),Tuple{Array{Regex,1}}}, ::typeof(foreach_problem), ::Function) at ./none:0
   [18] macro expansion at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:6 [inlined]
   [19] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
   [20] top-level scope at /home/travis/build/ericphanson/SDPA_GMP.jl/test/Convex.jl:6
   [21] include at ./boot.jl:317 [inlined]
   [22] include_relative(::Module, ::String) at ./loading.jl:1044
   [23] include(::Module, ::String) at ./sysimg.jl:29
   [24] include(::String) at ./client.jl:392
   [25] macro expansion at /home/travis/build/ericphanson/SDPA_GMP.jl/test/runtests.jl:17 [inlined]
   [26] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
   [27] top-level scope at /home/travis/build/ericphanson/SDPA_GMP.jl/test/runtests.jl:15
   [28] include at ./boot.jl:317 [inlined]
   [29] include_relative(::Module, ::String) at ./loading.jl:1044
   [30] include(::Module, ::String) at ./sysimg.jl:29
   [31] include(::String) at ./client.jl:392
   [32] top-level scope at none:0
   [33] eval(::Module, ::Any) at ./boot.jl:319
   [34] exec_options(::Base.JLOptions) at ./client.jl:243
   [35] _start() at ./client.jl:425

• lp_dotsort_atom: only for sdpa. Under the default parameters the solution is not precise enough.

lp_dotsort_atom: Test Failed at /Users/travis/.julia/packages/Convex/gaI3W/src/problem_depot/problems/lp.jl:201
  Expression: ≈(p.optval, 19, atol=atol, rtol=rtol)
   Evaluated: 18.998915046180628 ≈ 19 (atol=0.001, rtol=0.0)
Stacktrace:
 [1] lp_dotsort_atom(::getfield(Main, Symbol("##12#14")){String}, ::Val{true}, ::Float64, ::Float64, ::Type{Float64}) at /Users/travis/.julia/packages/Convex/gaI3W/src/problem_depot/problems/lp.jl:201
 [2] macro expansion at /Users/travis/build/ericphanson/SDPAFamily.jl/test/Convex.jl:31 [inlined]
 [3] macro expansion at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [4] (::getfield(Main, Symbol("##11#13")))(::String, ::typeof(Convex.ProblemDepot.lp_dotsort_atom)) at /Users/travis/build/ericphanson/SDPAFamily.jl/test/Convex.jl:31

• lp_pos_atom: only for sdpa. Under the default parameters the solution is not precise enough.

lp_pos_atom: Test Failed at C:\Users\zhuji\.julia\packages\Convex\LTy7f\src\problem_depot\problems\lp.jl:115
  Expression: ≈(p.optval, 3, atol=atol, rtol=rtol)
   Evaluated: 2.9879415445029736 ≈ 3 (atol=0.001, rtol=0.0)
Stacktrace:
 [1] lp_pos_atom(::getfield(Main, Symbol("##14#16")){String}, ::Val{true}, ::Float64, ::Float64, ::Type{Float64}) at C:\Users\zhuji\.julia\packages\Convex\LTy7f\src\problem_depot\problems\lp.jl:115
 [2] macro expansion at C:\Users\zhuji\.julia\dev\SDPAFamily\test\Convex.jl:35 [inlined]
 [3] (::getfield(Main, Symbol("##13#15")))(::String, ::typeof(Convex.ProblemDepot.lp_pos_atom)) at C:\Users\zhuji\.julia\dev\SDPAFamily\test\Convex.jl:35
 [4] #foreach_problem#4(::Array{Regex,1}, ::typeof(foreach_problem), ::getfield(Main, Symbol("##13#15"))) at C:\Users\zhuji\.julia\packages\Convex\LTy7f\src\problem_depot\problem_depot.jl:103
 [5] (::getfield(Convex.ProblemDepot, Symbol("#kw##foreach_problem")))(::NamedTuple{(:exclude,),Tuple{Array{Regex,1}}}, ::typeof(foreach_problem), ::Function) at .\none:0
 [6] top-level scope at C:\Users\zhuji\.julia\dev\SDPAFamily\test\Convex.jl:33
 [7] top-level scope at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Test\src\Test.jl:1113
 [8] top-level scope at C:\Users\zhuji\.julia\dev\SDPAFamily\test\Convex.jl:33

It looks like we pass the SOCP tests though, which is pretty cool since they were giving us so much trouble. Then we get a hang on lp_min_atom. Is it just too big?

I will rerun CI excluding that test to see what we get with tests after that one.

[JiazhengZhu]

With affine_Diagonal_atom, the problem is again the underflowing issue when using Float64. (:dd variant passes the test.) Also the error is due to presolve detecting the inconsistency before calling the solver. Should we change this behaviour?

[JiazhengZhu]

Okay I can confirm the affine_Partial_transpose failed because the size is a bit large for presolve to handle (~1500 variables) and there is also underflowing issue.

[ericphanson]

For the affine diagonal atom, I don’t quite understand where the underflowing happens and how it causes an inconsistency. Could you give me more details?

For the partial transpose, where does the presolve break down? I’d expect it to be slow but still working for large problems.

[ericphanson]

Just a quick thought: are the errors coming from lines like this?

SDPAFamily.jl/src/file_io.jl

Line 71 in de5ac88

append!(xMatvec, parse.(T, split(xMatstring[1:end-1], ",")))

Maybe if sdpa-gmp is printing many digits and we try to parse it into a Float64, we get incorrect results? In which case, we could parse the strings into BigFloats, then convert them to type T before populating the optimizer (with convert). That way we get all the digits into Julia and it can do the conversion, which hopefully will be more accurate. (I haven't tested this though...)

[JiazhengZhu]

When using :gmp variant, for affine diagonal atom the printout (Ymat) looks like
+1.282403116520467692512586027314051267839098235615783947181830679413211601854832e+26,+1.282403116520467692512586032456271108500823977315855888436669581930732405006457e+26...
So when varmap instructs Julia to, say, take the difference these two large numbers cancels to exactly zero, rather than giving -0.5142219840661725741700071941254838902517520803151625.

[JiazhengZhu]

We could just use Problem{BigFloat} but with Float64 entries? Everything just gets promoted and the readback will be correct.

[ericphanson]

Doesn’t that run into the problems with internal Float64 assumptions in MOI? Or maybe I don’t quite get what you’re proposing

[ericphanson]

When using :gmp variant, for affine diagonal atom the printout (Ymat) looks like
+1.282403116520467692512586027314051267839098235615783947181830679413211601854832e+26,+1.282403116520467692512586032456271108500823977315855888436669581930732405006457e+26...
So when varmap instructs Julia to, say, take the difference these two large numbers cancels to exactly zero, rather than giving -0.5142219840661725741700071941254838902517520803151625.

It seems strange that we're getting huge numbers like that from these problems. Can you look into it a bit more and see where they are coming from? The other Float64-only solvers can't handle those numbers either.

[JiazhengZhu]

Yes but I'm assuming it will get fixed relatively soon? But indeed what you suggested would definitely work, it's just that I am a bit hesitant to use BigFloat for all problems. Reading back the results for larger problem like this is taking quite long using BigFloat: on my laptop it's around 10 sec while using Float64 it's less than a second. So I thought maybe it's something we should use sparingly. I can open up a PR for MOI if it speeds things up, otherwise I'm equally happy to do what you suggested. I guess this is also necessary for MPB to work since it can only be in Float64. Would you like me to do this anyways?

I just updated SDPAFamily to allow for custom binaries for all variants. But Convex.jl tests failed on travis: could you perhaps fix the most recent Convex.jl update? I think you removed solver_info.jl but are still trying to include it in the main Convex.jl.

[ericphanson]

Uhhhh I broke Convex.jl?? Let me see

[ericphanson]

Ah ok, the main repo is fine (I was scared because I just tagged a release; CI passed though so I probably didn't need to worry). I probably pushed a broken commit to the MathOptInterface branch though? Sorry about that. I'm gonna open a PR against Convex.jl with that branch soon to get CI going on it also. I just pushed a commit with passing tests to that branch so we should be OK again. Sorry about that.

[ericphanson]

Yes but I'm assuming it will get fixed relatively soon?

Sorry, I didn't quite follow: what will get fixed soon?

But indeed what you suggested would definitely work, it's just that I am a bit hesitant to use BigFloat for all problems. Reading back the results for larger problem like this is taking quite long using BigFloat: on my laptop it's around 10 sec while using Float64 it's less than a second.

Ah really? The parse calls, not the solve?

So I thought maybe it's something we should use sparingly. I can open up a PR for MOI if it speeds things up, otherwise I'm equally happy to do what you suggested. I guess this is also necessary for MPB to work since it can only be in Float64. Would you like me to do this anyways?

To be clear, do you mean switch parsing to use Bigfloat, then convert it to Float64?

I just updated SDPAFamily to allow for custom binaries for all variants.
Nice! For WSL?

[JiazhengZhu]

Apologies for the ambiguity...

I'm assuming the float64 sqrt(2) in MOI will get fixed soon?

And yes switching the problem type to {Float64} saves those 10 sec. The solve time returned by SDPA is the same.

Yes we could just return the solution as bigfloats. I have this impression that the post processing bit do not handle type information...

It was done for all environments. So you just need to specify the path accordingly and set the wsl flag. I just updated my repo with a new wsl built plain sdpa binary. It has all dependencies statically linked so no need to install anything on wsl.