ruby-core

Issue #11625 has been updated by steved (Steve Dierker).

File digest.patch added

Hi,

I had a similar problem like Aaron, but with MD5 instead of SHA1. I stumbled upon this thread and completed the patch for all digest algorithms.
Hope it helps!

best,
Steve

 

----------------------------------------
Feature #11625: Unlock GVL for SHA1 calculations
https://bugs.ruby-lang.org/issues/11625#change-74191

* Author: tenderlovemaking (Aaron Patterson)
* Status: Assigned
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
I'm trying to calculate many sha1 checksums, but the current sha1 implementation doesn't unlock the GVL, so I can't do it in parallel.  I've attached a patch that unlocks the GVL when calculating sha1sums so that I can do them in parallel.

The good point about this patch is that I can calculate sha1's in parallel.  Here is the test code I'm using:

~~~
require 'digest/sha1'
require 'thread'

Thread.abort_on_exception = true

THREADS = (ENV['THREADS'] || 1).to_i

store = 'x' * (ENV['SIZE'] || 1024).to_i

queue = Queue.new

600000.times do
  queue << store
end

THREADS.times { queue << nil }

ts = THREADS.times.map {
  Thread.new {
    while work = queue.pop
      Digest::SHA1.hexdigest(work)
    end
  }
}

ts.each(&:join)
~~~

Here is what the output looks like after I've applied the patch:

~~~
[aaron@TC ruby (trunk)]$ THREADS=1 SIZE=4096 time ./ruby test.rb 
       22.62 real        21.78 user         0.66 sys
[aaron@TC ruby (trunk)]$ THREADS=4 SIZE=4096 time ./ruby test.rb 
       15.87 real        34.53 user         8.27 sys
[aaron@TC ruby (trunk)]$ 
~~~

The digests that I'm calculating are for fairly large strings, so this patch works well for me.  The downsides are that it seems slightly slower (though I'm not sure that it's significant) with a single thread:

Test code:

~~~
require 'benchmark/ips'
require 'digest/sha1'

Benchmark.ips do |x|
  x.report('sha1') { Digest::SHA1.hexdigest('x' * 4096) }
end
~~~

Before my patch (higher numbers are better):

~~~
[aaron@TC ruby (trunk)]$ ./ruby shaips.rb 
Calculating -------------------------------------
                sha1     2.604k i/100ms
-------------------------------------------------
                sha1     27.441k (賊 3.9%) i/s -    138.012k
~~~

After my patch:

~~~
[aaron@TC ruby (trunk)]$ ./ruby shaips.rb 
Calculating -------------------------------------
                sha1     2.419k i/100ms
-------------------------------------------------
                sha1     25.848k (賊 2.8%) i/s -    130.626k
~~~

Other downside is that I changed the `update` method to dup strings so that the GVL can be safely released.

This patch pays off for me because of the size of the strings I'm working with, but I'm not sure if it's fine for the general case.

---Files--------------------------------
sha1gvl.diff (3.79 KB)
digest.patch (7.96 KB)


-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>