[ruby-core:93365] [Ruby trunk Feature#15940] Coerce symbols internal fstrings in UTF8 rather than ASCII to better share memory with string literals
From:
jean.boussier@...
Date:
2019-06-25 22:06:42 UTC
List:
ruby-core #93365
Issue #15940 has been updated by byroot (Jean Boussier).
@naruse Interesting, I actually had no idea you could `String#<<(integer)`.
In my humble opinion, integer shifting on string returned by `Symbol#to_s` is quite specific, and is unlikely to be common in the wild.
Additionally UTF8 strings accept everything ASCII string would, so to break existing code it would need to expect `String#<<(127+)` to blow up.
```ruby
>> 'a' << 234324
=> "a\u{39354}"
>> 'a'.force_encoding(Encoding::BINARY) << 234324
RangeError: 234324 out of char range
```
That being said, it's not my role to judge the backward compatibility impact, and I'm likely to overlook parts of it.
More generally, let me know if there's anything I can do to push this forward. I think I'll update the test suite in the pull request to acknowledge the encoding change and get a green test suite.
----------------------------------------
Feature #15940: Coerce symbols internal fstrings in UTF8 rather than ASCII to better share memory with string literals
https://bugs.ruby-lang.org/issues/15940#change-78880
* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee:
* Target version:
----------------------------------------
Patch: https://github.com/ruby/ruby/pull/2242
It's not uncommon for symbols to have literal string counterparts, e.g.
```ruby
class User
attr_accessor :name
def as_json
{ 'name' => name }
end
end
```
Since the default source encoding is UTF-8, and that symbols coerce their internal fstring to ASCII when possible, the above snippet will actually keep two instances of `"name"` in the fstring registry. One in ASCII, the other in UTF-8.
Considering that UTF-8 is a strict superset of ASCII, storing the symbols fstrings as UTF-8 instead makes no significant difference, but allows in most cases to reuse the equivalent string literals.
The only notable behavioral change is `Symbol#to_s`.
Previously `:name.to_s.encoding` would be `#<Encoding:US-ASCII>`.
After this patch it's `#<Encoding:UTF-8>`. I can't foresee any significant compatibility impact of this change on existing code.
However, there are several ruby specs asserting this behavior, but I don't know if they can be changed or not: https://github.com/ruby/spec/commit/a73a1c11f13590dccb975ba4348a04423c009453
If this specification is impossible to change, then we could consider changing the encoding of the String returned by `Symbol#to_s`, e.g in ruby pseudo code:
```ruby
def to_s
str = fstr.dup
str.force_encoding(Encoding::ASCII) if str.ascii_only?
str
end
```
--
https://bugs.ruby-lang.org/
Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>