Implementing Ethereum personal_sign (EIP-191) from go-ethereum gives different signature from ethers.js

Question

I am attempting to generate a personal_sign in Golang like its implemented in ethers.js. Similar question but that ended up using the regular sign over the personal sign_implementation.

Ethers

// keccak256 hash of the data
let dataHash = ethers.utils.keccak256(
  ethers.utils.toUtf8Bytes(JSON.stringify(dataToSign))
);

//0x8d218fc37d2fd952b2d115046b786b787e44d105cccf156882a2e74ad993ee13

let signature = await wallet.signMessage(dataHash); // 0x469b07327fc41a2d85b7e69bcf4a9184098835c47cc7575375e3a306c3718ae35702af84f3a62aafeb8aab6a455d761274263d79e7fc99fbedfeaf759d8dc9361c

Golang:

func signHash(data []byte) common.Hash {
    msg := fmt.Sprintf("\x19Ethereum Signed Message:\n%d%s", len(data), data)

    return crypto.Keccak256Hash([]byte(msg))
}

privateKey, err := crypto.HexToECDSA(hexPrivateKey)
if err != nil {
    log.Fatal(err)
}

dataHash := crypto.Keccak256Hash(dataToSign) //0x8d218fc37d2fd952b2d115046b786b787e44d105cccf156882a2e74ad993ee13

signHash := signHash(dataHash.Bytes())

signatureBytes, err := crypto.Sign(signHash.Bytes(), privateKey)
if err != nil {
    log.Fatal(err)
}


// signatureBytes 0xec56178d3dca77c3cee7aed83cdca2ffa2bec8ef1685ce5103cfa72c27beb61313d91b9ad9b9a644b0edf6352cb69f2f8acd25297e3c64cd060646242e0455ea00

As you can see the hash is the same, but the signature is different:

0x469b07327fc41a2d85b7e69bcf4a9184098835c47cc7575375e3a306c3718ae35702af84f3a62aafeb8aab6a455d761274263d79e7fc99fbedfeaf759d8dc9361c Ethers

0xec56178d3dca77c3cee7aed83cdca2ffa2bec8ef1685ce5103cfa72c27beb61313d91b9ad9b9a644b0edf6352cb69f2f8acd25297e3c64cd060646242e0455ea00 Golang

Looking at the source code of Ethers.js I can't find anything different aside how the padding is managed.

Edit Check the approved answer

signHash(data []byte) common.Hash {
    hexData := hexutil.Encode(data)
    msg := fmt.Sprintf("\x19Ethereum Signed Message:\n%d%s", len(hexData), hexData)

    return crypto.Keccak256Hash([]byte(msg))
}

Serpens

Answer 1

There is a bug in the JavaScript code.

From the documentation of signer.signMessage() (see the Note section), it appears that a string is UTF8 encoded and binary data must be passed as TypedArray or Array.
The Keccak hash is returned hex encoded, i.e. as string, and is therefore UTF8 encoded, which is incorrect. Instead, it must be converted to a TypedArray. For this purpose the library provides the function ethers.utils.arrayify().

The following JavaScript is based on the posted code, but performs the required hex decoding:

(async () => {
    let privateKey = "0x8da4ef21b864d2cc526dbdb2a120bd2874c36c9d0a1fb7f8c63d7f7a8b41de8f";
    let dataToSign = {"data1":"value1","data2":"value2"};

    let dataHash = ethers.utils.keccak256(
      ethers.utils.toUtf8Bytes(JSON.stringify(dataToSign))
    );
    dataHashBin = ethers.utils.arrayify(dataHash)
    
    let wallet = new ethers.Wallet(privateKey);
    let signature = await wallet.signMessage(dataHashBin); 
    
    document.getElementById("signature").innerHTML = signature; // 0xfcc3e9431c139b5f943591af78c280b939595ce9df66210b7b8bb69565bdd2af7081a8acc0cbb5ea55bd0d673b176797966a5180c11ac297b7e6344c5822e66d1c
})();

<script src="https://cdn.ethers.io/lib/ethers-5.0.umd.min.js" type="text/javascript"></script>
<p style="font-family:'Courier New', monospace;" id="signature"></p>

which produces the following signature:

0xfcc3e9431c139b5f943591af78c280b939595ce9df66210b7b8bb69565bdd2af7081a8acc0cbb5ea55bd0d673b176797966a5180c11ac297b7e6344c5822e66d1c

---

The Go code below is based on the unmodified posted Go code, but using key and data from the JavaScript code for a comparison:

package main

import (
    "fmt"
    "github.com/ethereum/go-ethereum/common"
    "github.com/ethereum/go-ethereum/crypto"
    "encoding/hex"
    "encoding/json"
    "log"
)

func signHash(data []byte) common.Hash {
    msg := fmt.Sprintf("\x19Ethereum Signed Message:\n%d%s", len(data), data)
    return crypto.Keccak256Hash([]byte(msg))
}

func main() {

    hexPrivateKey := "8da4ef21b864d2cc526dbdb2a120bd2874c36c9d0a1fb7f8c63d7f7a8b41de8f"
    dataMap := map[string]string{"data1":"value1","data2":"value2"}
    dataToSign, _ := json.Marshal(dataMap)

    privateKey, err := crypto.HexToECDSA(hexPrivateKey)
    if err != nil {
            log.Fatal(err)
    }

    dataHash := crypto.Keccak256Hash(dataToSign) //0x8d218fc37d2fd952b2d115046b786b787e44d105cccf156882a2e74ad993ee13

    signHash := signHash(dataHash.Bytes())

    signatureBytes, err := crypto.Sign(signHash.Bytes(), privateKey)
    if err != nil {
            log.Fatal(err)
    }
    
    fmt.Println("0x" + hex.EncodeToString(signatureBytes))
}

The Go Code gives the following signature:

0xfcc3e9431c139b5f943591af78c280b939595ce9df66210b7b8bb69565bdd2af7081a8acc0cbb5ea55bd0d673b176797966a5180c11ac297b7e6344c5822e66d01

---

Both signatures match except for the last byte.

The JavaScript code returns the signature in the format r|s|v (see here). v is one byte in size and is just the value in which both signatures differ.

It is v = 27 + rid where rid is the recovery ID. The recovery ID has values between 0 and 3, so v has values between 27 and 30 or 0x1b and 0x1e (see here).

The Go code, on the other hand, returns the recovery ID in the last byte instead of v. So that the signature of the Go code matches that of the JavaScript code in the last byte as well, the recovery ID must be replaced by v:

signatureBytes[64] += 27
fmt.Println("0x" + hex.EncodeToString(signatureBytes))

Ransdell

Thank you, this was the correct answer. Because I have to match the JS implementation, I modified the go signHash function to encode the Byte Array into a Hex, after fixing the last byte it returns the same signature. Will post the code on my original answer as an update for future reference. – Serpens