Do the git repository data structures use a canonical encoding?

Question

I'm using dulwich (a Python library) to access a git repository. When I use get_object to retrieve a commit, it has a number of attributes. One of those is author. When I retrieve this attribute, I get bytes and so the attribute is an an unknown encoding.

Is there an encoding I can safely assume? Does git translate all the metadata to utf-8 before storing it? If it doesn't, how do I know which encoding to use to decode the bytes?

Babby

Presumably the current i18n.commitEncoding value holds the correct value (default is UTF-8). – Adamo

@MartijnPieters - Is the value of this config option propagated when you clone a repository, or do you have to remember to set it everywhere when you use a repository that has chosen an unusual encoding? If it's the latter, I'm going to start encoding everything I send to github as UTF-16 and watch the hilarity ensue. – Babby

I see that the commit object actually has a encoding header, copied from the .git/config setting. This is part of the objects data, so no hilarity, sorry. – Adamo

@MartijnPieters - I was noticing that, but it was None so I thought maybe it was meaningless. I guess maybe that means I can assume the default utf-8 for that particular commit. – Babby

@MartijnPieters - Anyway, it looks like you've probably answered my question. So maybe you should format your comment as an answer and I can accept it. :-) – Babby

I'm doing a few additional tests; I've not been able to actually get my commit messages recoded to UTF-16 just yet.. – Adamo

@MartijnPieters - How about EBCDIC? ;-) – Babby

Answer 1

Metadata is supposed to be encoded with the value set by the i18n.commitEncoding config value; whenever a commit is created the current value is copied into the 'encoding' header on the object, if set; the default value is UTF-8.

That encoding value is available on Dulwitch objects as the '.encoding' attribute; if it is None then i18n.commitEncoding was not explicitly set and you can use UTF-8 as the default.

However! The actual data stored simply follows whatever bytes where handed to git and no re-coding takes place. The configuration value is purely informational. So you need to take into account that an incorrect codec was used, if you are going to use object.encoding or 'utf8' as the codec, use a sensible error handler or fallback strategy.

Adamo

Thank you for doing the research necessary to figure this out. It's really disappointing to learn that no recoding is done. I should still try uploading an EBCDIC encoded repo that's honest with its encoding flag and see what GitHub does with it. ;-) – Babby