6

In the Operations on bytes section of the Michelson white document, it is mentioned that:

"PACK: Serializes a piece of data to its optimized binary representation."

I am curious to know more details of how PACK and UNPACK are implemented and what the binary representation is. I have looked through the source code and there are not any many comments in the implementation.

Edit:

The details that user catsigma have given are very useful, but I would still like to be able to follow how serialization works in the source code and where the operation to binary code mappings come from.

I found the interp function here and the implementation of Pack:

src/proto_alpha/lib_protocol/src/script_interpreter.ml#L697

let rec interp =
...
| Pack t, Item (value, rest) ->
    Script_ir_translator.pack_data ctxt t value >>=? fun (bytes, ctxt) ->
    logged_return (Item (bytes, rest), ctxt)

pack_data is defined here:

src/proto_alpha/lib_protocol/src/script_ir_translator.ml#L3162

let pack_data ctxt typ data =
  unparse_data ctxt Optimized typ data >>=? fun (data, ctxt) ->
  let unparsed = strip_annotations @@ data in
  let bytes = Data_encoding.Binary.to_bytes_exn expr_encoding (Micheline.strip_locations unparsed) in
  Lwt.return @@ Gas.consume ctxt (Script.serialized_cost bytes) >>=? fun ctxt ->
  let bytes = MBytes.concat "" [ MBytes.of_string "\005" ; bytes ] in
  Lwt.return @@ Gas.consume ctxt (Script.serialized_cost bytes) >>=? fun ctxt ->
  return (bytes, ctxt)

Data_encoding.Binary.to_bytes_exn looks like the key function.

src/lib_data_encoding/binary_writer.ml#L338

let to_bytes_exn e v =
  match Encoding.classify e with
  | `Fixed n -> begin
      (* Preallocate the complete buffer *)
      let state = { buffer = MBytes.create n ;
                    offset = 0 ; allowed_bytes = Some n } in
      write_rec e state v ;
      state.buffer
    end
  | `Dynamic | `Variable ->
      (* Preallocate a minimal buffer and let's not hardcode a
         limit to its extension. *)
      let state = { buffer = MBytes.create 4096 ;
                    offset = 0 ; allowed_bytes = None } in
      write_rec e state v ;
      MBytes.sub state.buffer 0 state.offset

Now let's try to find what the encoding of False would be. Here is some more code reorganized src/lib_data_encoding/binary_writer.ml:

let rec write_rec : type a. a Encoding.t -> state -> a -> unit =
  fun e state value ->
    let open Encoding in
    match e.encoding with
    | Bool -> Atom.bool state value
    ...

module Atom = struct
  let bool state v = uint8 state (if v then 255 else 0)
  let uint8 = int `Uint8
  let int kind state v =
    check_int_range (Binary_size.min_int kind) v (Binary_size.max_int kind) ;
    let ofs = state.offset in
    may_resize state (Binary_size.integer_to_size kind) ;
    set_int kind state.buffer ofs v

  let check_int_range min v max =
    if (v < min || max < v) then
      raise (Invalid_int { min ; v ; max })

  let set_int kind buffer ofs v =
    match kind with
    ...
    | `Uint8 -> MBytes.set_int8 buffer ofs v
end

Now we need to find MBytes.set_int8. I have only found the type signature file.

src/lib_protocol_environment/sigs/v1/mBytes.mli

mBytes looks like it is an external library but I haven't found the name. Also, I am not seeing how this matches up with the op_mapping as mentioned in tezbridge-crypto.

3

I have implemented the PACK and UNPACK in tezbridge-crypto project.

You can checkout the source code here: codec.js#L426

Here are some test cases: index.js#L35

In short words, there are several mappings(codec.js#L166, codec.js#L288) between the Michline value and the binary representation. The conversion is just a one way walk though the data, and each data type(like int, string..) has its specific encoding/decoding method.

| improve this answer | |
  • I was grepping around the source code from the tezos repository for the op_mapping you have. I am curious where these mappings are defined in there, but I have not found them. Looking through tezos source code has led me to a function write_rec in binary_writer.ml. – MCH Mar 4 '19 at 16:15
  • These are also good resources: The data_encoding library and Module Tezos_data_encoding.Data_encoding – MCH Mar 4 '19 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.