How do you read and store a UTF-8 value in Michelson

Question

The Michelson document from the Tezos Developers Resources page states the following about the string type:

Strings, with some usual escape sequences: \n, \, ". Unescaped line-breaks (both \n and \r) cannot appear in a Michelson string. Moreover, the current version of Michelson restricts strings to be the printable subset of 7-bit ASCII, namely characters with codes from within [32, 126] range, plus the escaped characters mentioned above.

It sounds like that is not a good type to hold a UTF-8 value. It looks like there is another type bytes. The documentation says the following.

Bytes are used for serializing data, in order to check signatures and compute hashes on them. They can also be used to incorporate data from the wild and untyped outside world.

This might be a good type to store UTF-8 value, but how do I use it?

For example, let's say I want to store a Russian phrase Привет, мир in a Michelson contract that has an entrypoint store_message with a parameter bytes that updates the storage to that value. If I am using the tezos-client, what operations do I need to perform on that string before sending it to the entrypoint. Then when the storage is updated and I read the storage from the contract, what operation do I need to perform on bytes to convert it back into the original message?

Answer 1

Byte sequences in Michelson are written in hexadecimal with a leading 0x. To convert between this representation and reguar UTF8 strings, you can use any tool that is dumb enough to iterate over UTF8 strings byte-per-byte; here is a solution using bash and tezos-codec:

# Encoding an UTF8 string as Michelson bytes
$ MSG='"Привет, мир"'
$ ENCODED_STRING=$(tezos-codec encode ground.string from "$MSG")
$ ENCODED_BYTES=$(tezos-codec decode ground.bytes from "$ENCODED_STRING")
$ MICHELSON_BYTES="0x$(echo $ENCODED_BYTES | cut -d \" -f 2)"
$ echo $MICHELSON_BYTES
0xd09fd180d0b8d0b2d0b5d1822c20d0bcd0b8d180

# Decoding Michelson bytes as an UTF8 string
$ MICHELSON_BYTES=0xd09fd180d0b8d0b2d0b5d1822c20d0bcd0b8d180
$ ENCODED_BYTES="\"$(echo $MICHELSON_BYTES | cut -d x -f 2)\""
$ ENCODED_STRING=$(tezos-codec encode ground.bytes from "$ENCODED_BYTES")
$ MSG=$(tezos-codec decode ground.string from "$ENCODED_STRING")
$ echo $MSG
"Привет, мир"

This relies on the fact that the ground binary encodings for strings and byte sequences are identical but the JSON encodings are different.