Implementation details for PACK and UNPACK

Question

In the Operations on bytes section of the Michelson white document, it is mentioned that:

"PACK: Serializes a piece of data to its optimized binary representation."

I am curious to know more details of how PACK and UNPACK are implemented and what the binary representation is. I have looked through the source code and there are not any many comments in the implementation.

Edit:

The details that user catsigma have given are very useful, but I would still like to be able to follow how serialization works in the source code and where the operation to binary code mappings come from.

I found the interp function here and the implementation of Pack:

src/proto_alpha/lib_protocol/src/script_interpreter.ml#L697

let rec interp =
...
| Pack t, Item (value, rest) ->
    Script_ir_translator.pack_data ctxt t value >>=? fun (bytes, ctxt) ->
    logged_return (Item (bytes, rest), ctxt)

pack_data is defined here:

src/proto_alpha/lib_protocol/src/script_ir_translator.ml#L3162

let pack_data ctxt typ data =
  unparse_data ctxt Optimized typ data >>=? fun (data, ctxt) ->
  let unparsed = strip_annotations @@ data in
  let bytes = Data_encoding.Binary.to_bytes_exn expr_encoding (Micheline.strip_locations unparsed) in
  Lwt.return @@ Gas.consume ctxt (Script.serialized_cost bytes) >>=? fun ctxt ->
  let bytes = MBytes.concat "" [ MBytes.of_string "\005" ; bytes ] in
  Lwt.return @@ Gas.consume ctxt (Script.serialized_cost bytes) >>=? fun ctxt ->
  return (bytes, ctxt)

Data_encoding.Binary.to_bytes_exn looks like the key function.

src/lib_data_encoding/binary_writer.ml#L338

let to_bytes_exn e v =
  match Encoding.classify e with
  | `Fixed n -> begin
      (* Preallocate the complete buffer *)
      let state = { buffer = MBytes.create n ;
                    offset = 0 ; allowed_bytes = Some n } in
      write_rec e state v ;
      state.buffer
    end
  | `Dynamic | `Variable ->
      (* Preallocate a minimal buffer and let's not hardcode a
         limit to its extension. *)
      let state = { buffer = MBytes.create 4096 ;
                    offset = 0 ; allowed_bytes = None } in
      write_rec e state v ;
      MBytes.sub state.buffer 0 state.offset

Now let's try to find what the encoding of False would be. Here is some more code reorganized src/lib_data_encoding/binary_writer.ml:

let rec write_rec : type a. a Encoding.t -> state -> a -> unit =
  fun e state value ->
    let open Encoding in
    match e.encoding with
    | Bool -> Atom.bool state value
    ...

module Atom = struct
  let bool state v = uint8 state (if v then 255 else 0)
  let uint8 = int `Uint8
  let int kind state v =
    check_int_range (Binary_size.min_int kind) v (Binary_size.max_int kind) ;
    let ofs = state.offset in
    may_resize state (Binary_size.integer_to_size kind) ;
    set_int kind state.buffer ofs v

  let check_int_range min v max =
    if (v < min || max < v) then
      raise (Invalid_int { min ; v ; max })

  let set_int kind buffer ofs v =
    match kind with
    ...
    | `Uint8 -> MBytes.set_int8 buffer ofs v
end

Now we need to find MBytes.set_int8. I have only found the type signature file.

src/lib_protocol_environment/sigs/v1/mBytes.mli

mBytes looks like it is an external library but I haven't found the name. Also, I am not seeing how this matches up with the op_mapping as mentioned in tezbridge-crypto.

Answer 1

I have implemented the PACK and UNPACK in tezbridge-crypto project.

You can checkout the source code here: codec.js#L426

Here are some test cases: index.js#L35

In short words, there are several mappings(codec.js#L166, codec.js#L288) between the Michline value and the binary representation. The conversion is just a one way walk though the data, and each data type(like int, string..) has its specific encoding/decoding method.