Bug: Improper use of `order` causes R session crash

```
DT = data.table(x=rep(c("b","a","c"),each=3), y=c(1,3,6), v=1:9)
DT[order(-x, "D")] #KERPLOOEY
```

Obviously this is improper use of `data.table` -- I was just trying to figure out what code is called when `DT[order(x)]` is run.

But an implosion as opposed to an error / warning seems a bit drastic.
