Non-ASCII characters in revert strings are mangled in responses
So, a warning in advance -- I'm not sure whether this is actually a ganache-core bug, or whether it's perhaps a web3 bug. But, I'm filing it here, and I'll let you deal with the rest!
## Expected Behavior
Revert strings should not get mangled, obviously. OK, that doesn't say very much. Let's move on and I can describe how they're currently getting mangled.
## Current Behavior
So, currently, if I send a transaction or a call and it reverts, I find that in the response from Ganache the revert string is mangled. Specifically, it gets interpreted as ISO-8859-1 instead of as UTF-8.
...except that's not quite true. Let's say my revert string contains invalid UTF-8; for instance, maybe it consists of a lone `0xa1` byte. In this case, the `0xa1` is not interpreted as ISO-8859-1, which would yield `¡`. Instead, we get the three-character string `ï¿½`. What's going on here?
Well, what's going on is that actually, the revert string *is* getting interpreted as UTF-8, and invalid UTF-8 is being replaced by the Unicode replacement character, U+FFFD... and then it is, somehow, at some point, being *re-encoded* into bytes, and *that* is being interpreted as ISO-8859-1. Crazy, huh?
Note that this only seems to occur in the response itself; for instance, the printout of the revert string in `ganache-cli`'s running terminal output is *not* mangled in this way. It's only mangled in the response I get externally.
## Possible Solution
So, what's going on here? What could be causing this?
Well, first let's rule out what *isn't* causing it. The obvious culprit is the decoding of revert strings in `runtimeerror.js`, right? Except that's working fine. The `ethereumjs-abi` decoder correctly decodes UTF-8.
And like I said -- at least part of `ganache-core` seems to have it decoded just fine, since it's printed out to the command line just fine. Additionally, note the point about replacement characters above; it's not being read as ISO-8859-1 *initially*, it's being read as UTF-8 initially. But somehow, at some *later* step, it's being re-encoded to bytes, and that's then being read back as ISO-8859-1.
The fact that the error only seems to occur in external responses gives me a suspicion as to what's going on. Now, all this time, I've been using `web3` to make my requests. But those requests go over HTTP, right? (Or WebSockets, I suppose, but I haven't been using that and don't know anything about that.) Which requires encoding. My suspicion, then, is that problem lies at the interface between `ganache-core` and `web3`.
For some reason -- this is my guess -- `web3` is reading Ganache's responses as ISO-8859-1, and not UTF-8. Perhaps Ganache is providing the wrong `charset` in its `Content-Type` header. Or perhaps it isn't specifying a charset at all, and for some reason `web3` defaults to using ISO-8859-1 instead of UTF-8. That seems ridiculous, seeing as UTF-8 has been the default on the web for years now, but, IDK, it's possible, I guess.
So, like I said, maybe this is actually `web3`'s fault... but maybe it can be fixed by just specifying the correct `charset`. I hope so, that'd be nice!
Of course, maybe I'm entirely wrong about the cause, but I gotta say this sure looks like the most likely possibility.
## Steps to Reproduce (for bugs)
Run a transaction that reverts with a reason that includes non-ASCII characters. For examples, you can check out my `returndata-test` project in our `solidity-test-cases` repository; in particular, take a look at `ErrorError.sol`. Try running the functions `utf8` (to see the simplest case of the mangling), `badUtf8` (to see the mangling with replacement characters), and, if you really want, `isoVsCpDistinguisher` (to confirm, FWIW, that it's ISO-8859-1 rather than CP-1252, although the particular incorrect encoding used isn't really that relevant here).
(Like, strictly speaking, based on what I tested, it could actually be ISO-8859-9 rather than ISO-8859-1; I didn't run tests that would distinguish those two. But c'mon, how likely is that? And it's not relevant one way or another.)
## Your Environment
* Version used: Ganache CLI v6.4.3 (ganache-core: 2.5.5) [which I know is a little out of date but I'm having some trouble updating it]
* Version of `web3` used to make the calls: 1.0.0-beta37, of course (if this is fixed in newer verisons of `web3`, well... you know we can't use those!)
* Operating System and version: Linux Mint 19.1 Tessa 64-bit
* Link to your project: https://github.com/trufflesuite/solidity-test-cases/tree/master/returndata-test