ruby-core

Issue #18579 has been updated by duerst (Martin D端rst).


tenderlovemaking (Aaron Patterson) wrote in #note-3:
> naruse (Yui NARUSE) wrote in #note-2:
> > The encoding of the resulted string depends "ascii only or not" and "ascii compatibility".
> > The principle of the resulted encoding is
> > * if LHS is ascii only
> >   * if RHS is ascii only
> >      * LHS's encoding
> >   * else if RHS is ascii compatible
> >      * RHS's encoding
> >   * else (RHS is ascii incompatible)
> >      * exception.
> > * else if LHS is ascii compatible
> >   * if RHS is ascii only
> >      * LHS's encoding
> >   * else if RHS is ascii compatible
> >      * if LHS's encoding equals to RHS's encoding
> >          * LHS's encoding
> >      * else
> >          * exception
> >   * else (RHS is ascii incompatible)
> >      * exception.
> > * else if LHS is ascii compatible
> >     * if LHS's encoding equals to RHS's encoding
> >       * LHS's encoding
> >     * else
> >       * exception
> 
> Is there anything we can do to simplify these rules?  It's way too complicated to remember.

First, it should be "* else if LHS is ascii incompatible" in the third bullet on the top level.

Second, it's not really that difficult to understand what happens. An encoding defines a mapping from a sequence of bytes to a sequence of characters. What concatenation does is that it concatenates the sequence of bytes and the sequence of characters at the same time, if it can easily know that that's possible (*). If not, then an error is produced.

(*) The reason for writing "if it can easily know" is because there are cases such as the following:
```
"r辿sum辿".encode('iso-8859-1') + "r辿sum辿".encode('iso-8859-2')
```
Because "辿" is encoded with the same byte in both iso-8859-1 and iso-8859-2, this concatenation would in theory work with the above rules, but it's too much to teach the implementation about which 8-bit bytes,... code for the same character in which encoding.

----------------------------------------
Feature #18579: Concatenation of ASCII-8BIT strings shouldn't behave differently depending on string contents
https://bugs.ruby-lang.org/issues/18579#change-96457

* Author: tenderlovemaking (Aaron Patterson)
* Status: Rejected
* Priority: Normal
----------------------------------------
Currently strings tagged with ASCII-8BIT will behave differently when concatenating depending on the string contents.

When concatenating strings the resulting string has the encoding of the LHS.  For example:

```
z = a + b
```

`z` will have the encoding of `a` (if the encodings are compatible).


However `ASCII-8BIT` behaves differently.  If `b` has "ASCII-8BIT" encoding, then the encoding of `z` will sometimes be the encoding of `a`, sometimes it will be the encoding of `b`, and sometimes it will be an exception.

Here is an example program:

```ruby
def concat a, b
  str = a + b
  str
end

concat "bar",                     "foo".encode("US-ASCII")    # Return value encoding is LHS, UTF-8
concat "bar".encode("US-ASCII"),  "foo".b                     # Return value encoding is LHS, US-ASCII
concat "ほげ",                    "foo".b                     # Return value encoding is LHS, UTF-8
concat "bar",                     "bad\376\377str".b          # Return value encoding is RHS, ASCII-8BIT.  Why?
concat "ほげ",                    "bad\376\377str".b          # Exception
```

This behavior is too hard to understand.  Usually we think LHS encoding will win, or there will be an exception. Even worse is that string concatenation can "infect" strings.  For example:


```ruby
def concat a, b
  str = a + b
  str
end

str = concat "bar", "bad\376\377str".b # this worked
p str
str = concat "ほげ", str               # exception
p str
```

The first concatenation succeeded, but the second one failed.  As a developer it is difficult to find where the "bad string" was introduced.  In the above example, the string may have been read from the network, but by the time an exception is raised it is far from where the "bad string" originated.  In the above example, the bad data came from like 6, but the exception was raised on line 8.

I propose that ASCII-8BIT strings raise an exception if they cannot be converted in to the LHS encoding.  So the above program would become like this:

```ruby
def concat a, b
  str = a + b
  str
end

concat "bar",                     "foo".encode("US-ASCII")    # Return value encoding is LHS, UTF-8
concat "bar".encode("US-ASCII"),  "foo".b                     # Return value encoding is LHS, US-ASCII
concat "ほげ",                    "foo".b                     # Return value encoding is LHS, UTF-8
concat "bar",                     "bad\376\377str".b          # Exception <--- NEW!!
concat "ほげ",                    "bad\376\377str".b          # Exception
```


I'm open to other solutions, but the underlying issue is that concatenating an ASCII-8BIT string with a non-ASCII-8BIT string is usually a bug and by the time an exception is raised, it is very far from the origin of the string.



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>

Thread

Prev Next

In This Thread

Prev Next