Rune vs byte ranging over string

Question

According to https://blog.golang.org/strings and my testings, it looks like while we range a string, the characters we get are rune type, but if we get it by str[index], they will be byte type, why is it?

Goar

The conclusion of the blog explains why str[index] is a byte: Strings are built from bytes so indexing them yields bytes, not characters. – Dragrope

Related: https://mcmap.net/q/583061/-is-there-any-difference-between-range-over-string-and-range-over-rune-slice/13860 – Etruria

Answer 1

To the first level, the why is because that's how the language is defined. The String type tells us that:

A string value is a (possibly empty) sequence of bytes. The number of bytes is called the length of the string and is never negative. Strings are immutable: once created, it is impossible to change the contents of a string.

and:

A string's bytes can be accessed by integer indices 0 through len(s)-1.

Meanwhile, range is a clause you can insert into a for statement, and the specification says:

The expression on the right in the "range" clause is called the range expression, which may be ... [a] string ...

and:

2. For a string value, the "range" clause iterates over the Unicode code points in the string starting at byte index 0. On successive iterations, the index value will be the index of the first byte of successive UTF-8-encoded code points in the string, and the second value, of type rune, will be the value of the corresponding code point. If the iteration encounters an invalid UTF-8 sequence, the second value will be 0xFFFD, the Unicode replacement character, and the next iteration will advance a single byte in the string.

If you want to know why the language is defined that way, you really have to ask the definers themselves. However, note that if for ranged only over the bytes, you'd need to construct your own fancier loops to range over the runes. Given that for ... range does work through the runes, if you want to work through the bytes in string s instead, you can write:

for i := 0; i < len(s); i++ {
    ...
}

and easily access s[i] inside the loop. You can also write:

for i, b := range []byte(s) {
}

and access both index i and byte b inside the loop. (Conversion from string to []byte, or vice versa, can require a copy since []byte can be modified. In this case, though, the range does not modify it and the compiler can optimize away the copy. See icza's comment below or this answer to golang: []byte(string) vs []byte(*string).) So you have not lost any ability, just perhaps a smidgen of concision.

Olympe

Conversion from string to []byte generally does make a copy, but ranging over the bytes of a string (converted to []byte) is an exception and the compiler optimizes away the copy. For details, see golang: []byte(string) vs []byte(*string) – Exquisite

@icza: ah, I was only thinking of cases like this, where it is clear that the result does not (or cannot) get modified. Obviously converting to a modifiable []byte would need a copy in general. Will fix the above. – Olympe

Answer 2

Just a quick and simple answer on why the language is defined this way.

Think about what a rune is. A rune represents a Unicode code point, which can be composed of multiple bytes and also have different representations depending on the encoding.

Now think what doing mystring[i] would mean if that returned a rune and not a byte. Since you cannot know the length of each rune without scanning the string, that operation would require scanning the whole string every single time, thus making array-like access take O(n) instead of O(1).

It would be very counter-intuitive for the users of the language if mystring[i] scanned the whole string every time, and also more complex for the language developers. This is why most programming languages (like Go, Rust, Python) differentiate between Unicode characters and bytes, and sometimes only support indexing on bytes.

Accessing a string one rune at a time is instead much simpler when iterating from the beginning of it, like for example using range. Consecutive bytes can be scanned and grouped together until they form a valid Unicode character that can be returned as a rune, moving on to the next one.

Wolfsbane

quick follow-up question, if I use mystring[i] to scan through a string, time complexity will be O(n^2), is it true? – Goar

If that returned a rune, yes it would be O(n^2). Since it returns a byte, complexity is O(n) where n is the number of bytes. – Wolfsbane

Was about to ask a slightly silly question and then figured it out; leaving it here in case others have the same thought. 1st glance it seems like you wouldn't have to scan the entire string, because Unicode only allows up to 4 bytes to be strung together; you could start scanning at i-3 to decide if byte i is part of a multi-byte rune. The problem there is that you don't know how many bytes the runes before i are; so you can find the rune of byte i in constant time, but you don't know if that rune is the ith; It could be anywhere in [i/4, i]. – Catercornered

Answer 3

Just letting you know. If you want to iterate with a classic for loop over a string and using operator [] to get the rune, you can do:

{
  rstr := []rune(MyString)
  for idx := 0; idx < len(rstr); idx++ {
    // code before...
    currentRune := rstr[idx]
    _ = currentRune // to avoid unused error
    // code after...
  }
}

Felton