Segment7

RDoc 4.0.1

2013-03-27T15:50:24-07:00

home: https://github.com/rdoc/rdoc
rdoc: http://docs.seattlerb.org/rdoc
bugs: https://github.com/rdoc/rdoc/issues

RDoc produces HTML and command-line documentation for Ruby projects. RDoc includes the +rdoc+ and +ri+ tools for generating and displaying documentation from the command-line.

Changes:

Bug fixes
- RDoc::Options parser should rescue from OptionParser::ParseError.
- Updated example of RDoc::Options to include reopening RDoc::Options. Pointed out by Michael Granger
- Moved RubyGems documentation installed message into RDoc hook. For RubyGems bug #469 by Jeff Sandberg
- An Error is now raised when a heredoc is not terminated. Fixes exceptions when processing comment blocks. Reported by darix
- rdoc --quiet --no-ignore-invalid now exits for invalid options. Pull request #192 by Jeremy Evans
- RDoc::Parser::C no longer ignores a (METHOD) cast in rbdefinemethod. Pull request #184 by Carlos Agarie
- RDoc::Servlet no longer ignores extra directories from -d. Pull request #173 by Thomas Leitner
- Fixed rdoc --ri-site. Bug #193 by Michal Papis.
- RDoc no longer attempts to parse binary files. Bug #189 by postmodern, Bug #190 by Christoffer Lervåg, Bug #195 by Aaron Patterson
- rdoc --pipe output now contains for markdown compliance.
- RDoc no longer leaves emacs-style modelines in .txt, .md or .rd files. Bug #178 by Zachary Scott
- RDoc no longer puts raw markup in HTML output for markdown input. Bug #204 by Erik Hollensbe
- Code objects with nodoc are no longer included in the ri store. Bug #177 by Thomas Leitner.
- Text#snippet now creates a RDoc::Markup::ToHtmlSnippet correctly.
- The C parser now de-duplicates call-seq if the same C function is used for multiple method names. Bug #203 by Pete Higgins

rdoc 4.0.0

2013-02-24T09:32:18-08:00

RDoc produces HTML and command-line documentation for Ruby projects. RDoc includes the +rdoc+ and +ri+ tools for generating and displaying documentation from the command-line.

Changes

RDoc 4.0 includes several new features and several breaking changes. The changes should not affect users of rdoc or ri.

Notable feature additions are markdown support and an WEBrick servlet that can serve HTML from an ri store. (This means that RubyGems 2.0+ no longer needs to build HTML documentation when installing gems.)

Changes since RDoc 3.12.1:

Breaking changes
- The default output encoding for RDoc is now UTF-8. Previously RDoc used the default external encoding which was determined from your locale. Issue #106 by Justin Baker.
- RDoc::RI::Store is now RDoc::Store so ri data generated by RDoc 4 cannot be read by earlier versions of RDoc. RDoc::RI::Store exists as an alias of RDoc::Store so ri data from older versions can still be read. RDoc::RI::Store will be removed in RDoc 5.
  
  Tests that create RDoc::CodeObjects on the fly without wiring them into the documentation tree (did not use add_class, add_method, etc.) must be updated to use these methods. The documentation tree automatically attaches them to the store instance which allows lookups to work correctly. Additionally, a new method RDoc::Store#add_file must be used instead of RDoc::TopLevel.new. The latter will not be attached to the documentation tree.
- RDoc generators must accept an RDoc::Store and an RDoc::Options in initialize. RDoc no longer passes an Array of RDoc::TopLevel objects to #generate. Use RDoc::Store#all_files instead.
- Some markup formatters (RDoc::Markup::To*) now accept an RDoc::Options instance as the first argument. Notably, the base class Formatter and ToHtml*. (This is not universal due to the difficult at accessing the user's options instance deep inside RDoc. A future major release may remedy this.)
- Added new markup nodes and specials that RDoc::Markup::Formatter subclasses must handle. If you're using RDoc::Markup::FormatterTestCase the new methods you need to add should be readily apparent.
- Removed RDoc::RI::Paths::SYSDIR and ::SITEDIR. These were hidden constants so no breakage is expected. Use RDoc::RI::Paths::systemdir and ::sitedir instead.
- RDoc::RI::Store#modules has been renamed to RDoc::Store#module_names to avoid confusion with RDoc::Store#all_modules imported from RDoc::TopLevel.
- RDoc::RDocError has been removed. It was deprecated throughout RDoc 3.
- ri -f html is no longer supported.
- Comment definitions in C comments are now only discovered from the first line. A colon on a subsequent line won't trigger definition extraction. Issue #103, see also http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/42942
- Fixed :stopdoc: for class A::B where A has not been seen. Issue #95 by Ryan Davis
- RDoc::ClassModule#each_ancestor no longer yields itself if there is circular ancestry
Major enhancements
- ri can now show pages (README, etc.)
  
  ri rdoc:README
  
  Will show the README for the latest version of RDoc. You can specify exact gem versions such as "rdoc-4.0:README" or view pages from the standard library documentation with "ruby:README".
  
  RDoc 3 did not save pages in ri data so you will need to regenerate documentation from your gems to use this feature.
- Added Markdown as a supported format. The markdown format can be set on a per-file or per-comment basis with the +:markdown:+ directive like the rd and tomdoc formats and on a per-project basis with rdoc --markup markdown --write-options
- Removed global state from RDoc. RDoc::Store holds the documentation tree and connects the driver to the parsers and generator. This also allows documentation parsing and generation for multiple instances, but the rdoc command-line tool does not support this.
  
  Due to this change RDoc::RDoc.current and RDoc::RDoc.reset no longer exist.
Minor enhancements
- Added --page-dir option to give pretty names for a FAQ, guides, or other documentation you write that is not stored in the project root. For example, with the following layout:
  
  README.txt guides/syntax.txt guides/conversion.txt
  
  Running rdoc --page-dir guides will make the files in "guides" appear to be at the top level of the project. This means they will appear to exist at the top level in HTML output and you can access them with ri your\_gem:syntax and ri your\_gem:conversion.
- Added --root for building documentation from outside the source dir.
- Added current heading and page-top links to HTML headings.
- Added a ChangeLog parser. It automatically parses files that begin with 'ChangeLog'
- Added a table of contents to the sidebar.
- RDoc markup format merges adjacent labels in a label or note list into a single definition list item for output.
- RDoc now tracks use of extend. Pull request #118 by Michael Granger.
- RDoc now tracks methods that use super. Pull request #116 by Erik Hollensbe.
- Added methods ::systemdir, ::sitedir, ::home_dir and ::gem_dir to fetch the components of RDoc::RI::Paths.path individually.
- Added support for rb_file_const.
- RDoc now processes files in sorted order. Issue #71 by Vít Ondruch
- RDoc now warns with --verbose when methods are duplicated. Issue #71 by Vít Ondruch
- ri will display documentation for all methods in a class if -a is given. Issue #57 by casper
- The RDoc coverage report will report line information for attributes, constants and methods missing documentation. Issue #121 by Zachary Scott
- RDoc now reports a warning for files that are unreadable due to permissions problems.
- RDoc controls documentation generation for RubyGems 2.0+
Bug fixes
- Fixed parsing of multibyte files with incomplete characters at byte 1024. Ruby bug #6393 by nobu, patch by Nobuyoshi Nakada and Yui NARUSE.
- Fixed rdoc -E. Ruby Bug #6392 and (modified) patch by Nobuyoshi Nakada
- Added link handling to Markdown output. Bug #160 by burningTyger.
- Fixed HEREDOC output for the limited case of a heredoc followed by a line end. When a HEREDOC is not followed by a line end RDoc is not currently smart enough to restore the source correctly. Bug #162 by Zachary Scott.
- Fixed parsing of executables with shebang and encoding comments. Bug #161 by Marcus Stollsteimer
- RDoc now ignores methods defined on constants instead of creating a fake module. Bug #163 by Zachary Scott.
- Fixed ChangeLog parsing for FFI gem. Bug #165 by Zachary Scott.
- RDoc now links #=== methods. Bug #164 by Zachary Scott.
- Allow [] following argument names for TomDoc. Bug #167 by Ellis Berner.
- Fixed the RDoc servlet for home and site directories. Bug #170 by Thomas Leitner.
- Fixed references to methods in the RDoc servlet. Bug #171 by Thomas Leitner.
- Fixed debug message when generating the darkfish root page. Pull Request #174 by Thomas Leitner.
- Fixed deletion of attribute ri data when a class was loaded then saved. Issue #171 by Thomas Leitner.
- Fully qualified names for constants declared from the top level are now attached to their class or module properly.
- Fixed table of contents display in HTML output for classes and modules.
- Incremental ri builds of C files now work. C variable names from previous runs are now saved between runs.
- A word that is directly followed by a multi-word tidy link label no longer disappears. (Like text{link}[http://example])
- Fixed legacy template support. Pull Request #107 by Justin Baker.
- An HTML class in a verbatim section no longer triggers ruby parsing. Issue #92 by Vijay Dev
- Improved documentation for setting the default documentation format for your ruby project. Issue #94 by Henrik Hodne
- Fixed handling of LANG in the RDoc::Options tests. Issue #99 by Vít Ondruch
- RDoc no longer quits when given an entry that is not a file or directory. Issue #101 by Charles Nutter
- Fixed bug in syntax-highlighting that would corrupt regular expressions. Ruby Bug #6488 by Benny Lyne Amorsen.
- "class Object" no longer appears in the coverage report if all its methods are documented. This suppresses a false positive for libraries that add toplevel methods. Pull Request #128 by Zachary Scott.
- Fixed testgenurl test name in TestRDocMarkupToHtml. Pull Request #130 by Zachary Scott.
- Comment-defined methods ahead of define_method are now discovered. Issue #133 by eclectic923
- Fixed detection of define_method documentation. Issue #138 by Marvin Gülker.
- Fixed lexing of character syntax (?z). Reported by Xavier Noria.
- Add license to gem spec. Issue #144 by pivotalcommon
- Fixed comment selection for classes. Pull request #146 by pioz
- Fixed parsing of def self.&() end. Issue #148 by Michael Lucy
- Generated RD parser files are now included in the gem. Issue #145 by Marvin Gülker
- Class and module aliases now create new classes to avoid duplicate names in the class list. Issue #143 by Richard Schneeman, Rails Issue #2839
- RDoc::Markup::Parser now correctly matches indentation of lists when multibyte characters are used in the list labels. Issue #140 by burningTyger
- Fixed mangling of email addresses that look like labels. Issue #129 by Tobias Koch
- Classes and modules in a C file may now be created in any order. Issue #124 by Su Zhang
- A metaprogrammed method supports the :args: directive. Issue #100
- A metaprogrammed method supports the :yields: directive.
- RDoc will now look for directives up to the end of the line. For example, class B < A; end # :nodoc: will now hide documentation of B. Issue #125 by Zachary Scott
- Fixed tokenization of % when it is not followed by a $-string type
- Fixed display of _END_ in documentation examples in HTML output
- Fixed tokenization of reserved words used as new-style hash keys
- RDoc now handles class << $gvar by ignoring the body
- Fixed parsing of class A:: B.
- Worked around bug in RDoc::RubyLex where tokens won't be reinterpreted after unget_tk.
- Fixed class << ::Foo writing documentation to /Foo.html
- Fixed class ::A referencing itself from inside its own namespace.

Changes since RDoc 4.0.0.rc.2:

Bug fix
- Templates now use the correct encoding when generating pages. Issue #183 by Vít Ondruch

RDoc 4.0.0.rc.2.1

2013-02-08T15:02:34-08:00

As a preview release, please file bugs for any problems you have with rdoc at https://github.com/rdoc/rdoc/issues

RDoc 4.0 includes several new features and several breaking changes. The changes should not affect users of rdoc or ri.

Changes since RDoc 3.12.1:

Breaking changes
- The default output encoding for RDoc is now UTF-8. Previously RDoc used the default external encoding which was determined from your locale. Issue #106 by Justin Baker.
- RDoc::RI::Store is now RDoc::Store so ri data generated by RDoc 4 cannot be read by earlier versions of RDoc. RDoc::RI::Store exists as an alias of RDoc::Store so ri data from older versions can still be read. RDoc::RI::Store will be removed in RDoc 5.
  
  Tests that create RDoc::CodeObjects on the fly without wiring them into the documentation tree (did not use addclass, addmethod, etc.) must be updated to use these methods. The documentation tree automatically attaches them to the store instance which allows lookups to work correctly. Additionally, a new method RDoc::Store#add_file must be used instead of RDoc::TopLevel.new. The latter will not be attached to the documentation tree.
- RDoc generators must accept an RDoc::Store and an RDoc::Options in initialize. RDoc no longer passes an Array of RDoc::TopLevel objects to #generate. Use RDoc::Store#all_files instead.
- Some markup formatters (RDoc::Markup::To) now accept an RDoc::Options instance as the first argument. Notably, the base class Formatter and ToHtml. (This is not universal due to the difficult at accessing the user's options instance deep inside RDoc. A future major release may remedy this.)
- Added new markup nodes and specials that RDoc::Markup::Formatter subclasses must handle. If you're using RDoc::Markup::FormatterTestCase the new methods you need to add should be readily apparent.
- Removed RDoc::RI::Paths::SYSDIR and ::SITEDIR. These were hidden constants so no breakage is expected. Use RDoc::RI::Paths::systemdir and ::sitedir instead.
- RDoc::RI::Store#modules has been renamed to RDoc::Store#modulenames to avoid confusion with RDoc::Store#allmodules imported from RDoc::TopLevel.
- RDoc::RDocError has been removed. It was deprecated throughout RDoc 3.
- ri -f html is no longer supported.
- Comment definitions in C comments are now only discovered from the first line. A colon on a subsequent line won't trigger definition extraction. Issue #103, see also http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/42942
- Fixed :stopdoc: for class A::B where A has not been seen. Issue #95 by Ryan Davis
- RDoc::ClassModule#each_ancestor no longer yields itself if there is circular ancestry
Major enhancements
- ri can now show pages (README, etc.)
  
  ri rdoc:README
  
  Will show the README for the latest version of RDoc. You can specify exact gem versions such as "rdoc-4.0:README" or view pages from the standard library documentation with "ruby:README".
  
  RDoc 3 did not save pages in ri data so you will need to regenerate documentation from your gems to use this feature.
- Added Markdown as a supported format. The markdown format can be set on a per-file or per-comment basis with the +:markdown:+ directive like the rd and tomdoc formats and on a per-project basis with rdoc --markup markdown --write-options
- Removed global state from RDoc. RDoc::Store holds the documentation tree and connects the driver to the parsers and generator. This also allows documentation parsing and generation for multiple instances, but the rdoc command-line tool does not support this.
  
  Due to this change RDoc::RDoc.current and RDoc::RDoc.reset no longer exist.
Minor enhancements
- Added --page-dir option to give pretty names for a FAQ, guides, or other documentation you write that is not stored in the project root. For example, with the following layout:
  
  README.txt guides/syntax.txt guides/conversion.txt
  
  Running rdoc --page-dir guides will make the files in "guides" appear to be at the top level of the project. This means they will appear to exist at the top level in HTML output and you can access them with ri your_gem:syntax and ri your_gem:conversion.
- Added --root for building documentation from outside the source dir.
- Added current heading and page-top links to HTML headings.
- Added a ChangeLog parser. It automatically parses files that begin with 'ChangeLog'
- Added a table of contents to the sidebar.
- RDoc markup format merges adjacent labels in a label or note list into a single definition list item for output.
- RDoc now tracks use of extend. Pull request #118 by Michael Granger.
- RDoc now tracks methods that use super. Pull request #116 by Erik Hollensbe.
- Added methods ::systemdir, ::sitedir, ::homedir and ::gemdir to fetch the components of RDoc::RI::Paths.path individually.
- Added support for rbfileconst.
- RDoc now processes files in sorted order. Issue #71 by Vít Ondruch
- RDoc now warns with --verbose when methods are duplicated. Issue #71 by Vít Ondruch
- ri will display documentation for all methods in a class if -a is given. Issue #57 by casper
- The RDoc coverage report will report line information for attributes, constants and methods missing documentation. Issue #121 by Zachary Scott
- RDoc now reports a warning for files that are unreadable due to permissions problems.
- RDoc controls documentation generation for RubyGems 2.0+
Bug fixes
- Fixed parsing of multibyte files with incomplete characters at byte 1024. Ruby bug #6393 by nobu, patch by Nobuyoshi Nakada and Yui NARUSE.
- Fixed rdoc -E. Ruby Bug #6392 and (modified) patch by Nobuyoshi Nakada
- Added link handling to Markdown output. Bug #160 by burningTyger.
- Fixed HEREDOC output for the limited case of a heredoc followed by a line end. When a HEREDOC is not followed by a line end RDoc is not currently smart enough to restore the source correctly. Bug #162 by Zachary Scott.
- Fixed parsing of executables with shebang and encoding comments. Bug #161 by Marcus Stollsteimer
- RDoc now ignores methods defined on constants instead of creating a fake module. Bug #163 by Zachary Scott.
- Fixed ChangeLog parsing for FFI gem. Bug #165 by Zachary Scott.
- RDoc now links #=== methods. Bug #164 by Zachary Scott.
- Allow [] following argument names for TomDoc. Bug #167 by Ellis Berner.
- Fixed the RDoc servlet for home and site directories. Bug #170 by Thomas Leitner.
- Fixed references to methods in the RDoc servlet. Bug #171 by Thomas Leitner.
- Fixed debug message when generating the darkfish root page. Pull Request #174 by Thomas Leitner.
- Fixed deletion of attribute ri data when a class was loaded then saved. Issue #171 by Thomas Leitner.
- Fully qualified names for constants declared from the top level are now attached to their class or module properly.
- Fixed table of contents display in HTML output for classes and modules.
- Incremental ri builds of C files now work. C variable names from previous runs are now saved between runs.
- A word that is directly followed by a multi-word tidy link label no longer disappears. (Like text{link}[http://example])
- Fixed legacy template support. Pull Request #107 by Justin Baker.
- An HTML class in a verbatim section no longer triggers ruby parsing. Issue #92 by Vijay Dev
- Improved documentation for setting the default documentation format for your ruby project. Issue #94 by Henrik Hodne
- Fixed handling of LANG in the RDoc::Options tests. Issue #99 by Vít Ondruch
- RDoc no longer quits when given an entry that is not a file or directory. Issue #101 by Charles Nutter
- Fixed bug in syntax-highlighting that would corrupt regular expressions. Ruby Bug #6488 by Benny Lyne Amorsen.
- "class Object" no longer appears in the coverage report if all its methods are documented. This suppresses a false positive for libraries that add toplevel methods. Pull Request #128 by Zachary Scott.
- Fixed testgenurl test name in TestRDocMarkupToHtml. Pull Request #130 by Zachary Scott.
- Comment-defined methods ahead of define_method are now discovered. Issue #133 by eclectic923
- Fixed detection of define_method documentation. Issue #138 by Marvin Gülker.
- Fixed lexing of character syntax (?z). Reported by Xavier Noria.
- Add license to gem spec. Issue #144 by pivotalcommon
- Fixed comment selection for classes. Pull request #146 by pioz
- Fixed parsing of def self.&() end. Issue #148 by Michael Lucy
- Generated RD parser files are now included in the gem. Issue #145 by Marvin Gülker
- Class and module aliases now create new classes to avoid duplicate names in the class list. Issue #143 by Richard Schneeman, Rails Issue #2839
- RDoc::Markup::Parser now correctly matches indentation of lists when multibyte characters are used in the list labels. Issue #140 by burningTyger
- Fixed mangling of email addresses that look like labels. Issue #129 by Tobias Koch
- Classes and modules in a C file may now be created in any order. Issue #124 by Su Zhang
- A metaprogrammed method supports the :args: directive. Issue #100
- A metaprogrammed method supports the :yields: directive.
- RDoc will now look for directives up to the end of the line. For example, class B < A; end # :nodoc: will now hide documentation of B. Issue #125 by Zachary Scott
- Fixed tokenization of % when it is not followed by a $-string type
- Fixed display of END in documentation examples in HTML output
- Fixed tokenization of reserved words used as new-style hash keys
- RDoc now handles class << $gvar by ignoring the body
- Fixed parsing of class A:: B.
- Worked around bug in RDoc::RubyLex where tokens won't be reinterpreted after unget_tk.
- Fixed class << ::Foo writing documentation to /Foo.html
- Fixed class ::A referencing itself from inside its own namespace.

Changes since RDoc 4.0.0.rc.2:

Bug fix
- Templates now use the correct encoding when generating pages. Issue #183 by Vít Ondruch

RDoc XSS vulnerability CVE-2013-0256 releases 3.9.5, 3.12.1, 4.0.0.rc.2

2013-02-06T10:36:20-08:00

RDoc versions 3.9.5, 3.12.1 and 4.0.0.rc.2 have been released!

RDoc produces HTML and command-line documentation for Ruby projects. RDoc includes the rdoc and ri tools for generating and displaying documentation from the command-line.

Vulnerability Description

RDoc documentation generated by rdoc 2.3.0 through rdoc 3.12 and prereleases up to rdoc 4.0.0.preview2.1 are vulnerable to an XSS exploit. This exploit may lead to cookie disclosure to third parties.

The exploit exists in darkfish.js which is copied from the RDoc install location to the generated documentation.

RDoc is a static documentation generation tool. Patching the library itself is insufficient to correct this exploit. Those hosting rdoc documentation will need to apply the following patch. If applied while ignoring whitespace, this patch will correct all affected versions:

diff --git darkfish.js darkfish.js
index 4be722f..f26fd45 100644
--- darkfish.js
+++ darkfish.js
@@ -109,13 +109,15 @@ function hookSearch() {
 function highlightTarget( anchor ) {
   console.debug( "Highlighting target '%s'.", anchor );

-  $("a[name=" + anchor + "]").each( function() {
-    if ( !$(this).parent().parent().hasClass('target-section') ) {
-      console.debug( "Wrapping the target-section" );
-      $('div.method-detail').unwrap( 'div.target-section' );
-      $(this).parent().wrap( '<div class="target-section"></div>' );
-    } else {
-      console.debug( "Already wrapped." );
+  $("a[name]").each( function() {
+    if ( $(this).attr("name") == anchor ) {
+      if ( !$(this).parent().parent().hasClass('target-section') ) {
+        console.debug( "Wrapping the target-section" );
+        $('div.method-detail').unwrap( 'div.target-section' );
+        $(this).parent().wrap( '<div class="target-section"></div>' );
+      } else {
+        console.debug( "Already wrapped." );
+      }
     }
   });
 };

RDoc 3.9.5, 3.12.1 and RDoc 4.0.0.rc.2 and newer are not vulnerable to this exploit.

This exploit was discovered by Evgeny Ermakov corwmh@gmail.com.

This vulnerability has been assigned the CVE identifier CVE-2013-0256.

RDoc 3.9.5

RDoc 3.9.5 was released to match the RDoc in ruby 1.9.3-p385.

Bug fixes
- Fixed an XSS exploit in darkfish.js. This could lead to cookie disclosure to third parties. See CVE-2013-0256.rdoc for full details including a patch you can apply to generated RDoc documentation.

RDoc 3.12.1

RDoc 3.12.1 was updated as the latest stable release as RDoc 4 is not yet ready.

Bug fixes
- Fixed an XSS exploit in darkfish.js. This could lead to cookie disclosure to third parties. See CVE-2013-0256.rdoc for full details including a patch you can apply to generated RDoc documentation.
- Ensured that rd parser files are generated before checking the manifest.

RDoc 4.0.0.rc.2

Minor enhancements
- Added current heading and page-top links to HTML headings.
Bug fixes
- Fixed an XSS exploit in darkfish.js. This could lead to cookie disclosure to third parties. See CVE-2013-0256.rdoc for full details including a patch you can apply to generated RDoc documentation.
- Fixed parsing of multibyte files with incomplete characters at byte 1024. Ruby bug #6393 by nobu, patch by Nobuyoshi Nakada and Yui NARUSE.
- Fixed rdoc -E. Ruby Bug #6392 and (modified) patch by Nobuyoshi Nakada
- Added link handling to Markdown output. Bug #160 by burningTyger.
- Fixed HEREDOC output for the limited case of a heredoc followed by a line end. When a HEREDOC is not followed by a line end RDoc is not currently smart enough to restore the source correctly. Bug #162 by Zachary Scott.
- Fixed parsing of executables with shebang and encoding comments. Bug #161 by Marcus Stollsteimer
- RDoc now ignores methods defined on constants instead of creating a fake module. Bug #163 by Zachary Scott.
- Fixed ChangeLog parsing for FFI gem. Bug #165 by Zachary Scott.
- RDoc now links #=== methods. Bug #164 by Zachary Scott.
- Allow [] following argument names for TomDoc. Bug #167 by Ellis Berner.
- Fixed the RDoc servlet for home and site directories. Bug #170 by Thomas Leitner.
- Fixed references to methods in the RDoc servlet. Bug #171 by Thomas Leitner.
- Fixed debug message when generating the darkfish root page. Pull Request #174 by Thomas Leitner.
- Fixed deletion of attribute ri data when a class was loaded then saved. Issue #171 by Thomas Leitner.

RDoc 4.0.0.preview2

2012-12-01T12:51:06-08:00

rdoc version 4.0.0.preview2 has been released!

RDoc produces HTML and command-line documentation for Ruby projects. RDoc includes the rdoc and ri tools for generating and displaying documentation from the command-line.

Changes:

4.0.0.preview2

As a preview release, please file bugs for any problems you have with rdoc at github.com/rdoc/rdoc/issues

RDoc 4.0 includes several new features and several breaking changes. The changes should not affect users of `rdoc` or `ri`.

Breaking changes
- The default output encoding for RDoc is now UTF-8. Previously RDoc used the default external encoding which was determined from your locale. Issue #106 by Justin Baker.
- RDoc::RI::Store is now RDoc::Store so ri data generated by RDoc 4 cannot be read by earlier versions of RDoc. RDoc::RI::Store exists as an alias of RDoc::Store so ri data from older versions can still be read. RDoc::RI::Store will be removed in RDoc 5.
  
  Tests that create RDoc::CodeObjects on the fly without wiring them into the documentation tree (did not use add_class, add_method, etc.) must be updated to use these methods. The documentation tree automatically attaches them to the store instance which allows lookups to work correctly. Additionally, a new method RDoc::Store#add_file must be used instead of RDoc::TopLevel.new. The latter will not be attached to the documentation tree.
- RDoc generators must accept an RDoc::Store and an RDoc::Options in initialize. RDoc no longer passes an Array of RDoc::TopLevel objects to #generate. Use RDoc::Store#all_files instead.
- Some markup formatters (RDoc::Markup::To*) now accept an RDoc::Options instance as the first argument. Notably, the base class Formatter and ToHtml*. (This is not universal due to the difficult at accessing the user's options instance deep inside RDoc. A future major release may remedy this.)
- Added new markup nodes and specials that RDoc::Markup::Formatter subclasses must handle. If you're using RDoc::Markup::FormatterTestCase the new methods you need to add should be readily apparent.
- Removed RDoc::RI::Paths::SYSDIR and ::SITEDIR. These were hidden constants so no breakage is expected. Use RDoc::RI::Paths::system_dir and ::site_dir instead.
- RDoc::RI::Store#modules has been renamed to RDoc::Store#module_names to avoid confusion with RDoc::Store#all_modules imported from RDoc::TopLevel.
- RDoc::RDocError has been removed. It was deprecated throughout RDoc 3.
- ri -f html is no longer supported.
- Comment definitions in C comments are now only discovered from the first line. A colon on a subsequent line won't trigger definition extraction. Issue #103, see also blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/42942
- Fixed :stopdoc: for class A::B where A has not been seen. Issue #95 by Ryan Davis
- RDoc::ClassModule#each_ancestor no longer yields itself if there is circular ancestry
Major enhancements
- ri can now show pages (README, etc.)
```
ri rdoc:README
```
  Will show the README for the latest version of RDoc. You can specify exact gem versions such as “rdoc-4.0:README” or view pages from the standard library documentation with “ruby:README”.
  
  RDoc 3 did not save pages in ri data so you will need to regenerate documentation from your gems to use this feature.
- Added Markdown as a supported format. The markdown format can be set on a per-file or per-comment basis with the :markdown: directive like the rd and tomdoc formats and on a per-project basis with rdoc --markup markdown --write-options
- Removed global state from RDoc. RDoc::Store holds the documentation tree and connects the driver to the parsers and generator. This also allows documentation parsing and generation for multiple instances, but the rdoc command-line tool does not support this.
  
  Due to this change RDoc::RDoc.current and RDoc::RDoc.reset no longer exist.
Minor enhancements
- Added –root for building documentation from outside the source dir.
- Added a table of contents to the sidebar.
- RDoc markup format merges adjacent labels in a label or note list into a single definition list item for output.
- RDoc now tracks use of extend. Pull request #118 by Michael Granger.
- RDoc now tracks methods that use super. Pull request #116 by Erik Hollensbe.
- Added methods ::system_dir, ::site_dir, ::home_dir and ::gem_dir to fetch the components of RDoc::RI::Paths.path individually.
- Added support for rb_file_const.
- RDoc now processes files in sorted order. Issue #71 by Vít Ondruch
- RDoc now warns with –verbose when methods are duplicated. Issue #71 by Vít Ondruch
- ri will display documentation for all methods in a class if -a is given. Issue #57 by casper
- The RDoc coverage report will report line information for attributes, constants and methods missing documentation. Issue #121 by Zachary Scott
- RDoc now reports a warning for files that are unreadable due to permissions problems.
- RDoc controls documentation generation for RubyGems 2.0+
Bug fixes
- A word that is directly followed by a multi-word tidy link label no longer disappears. (Like text{link}[http://example])
- Fixed legacy template support. Pull Request #107 by Justin Baker.
- An HTML class in a verbatim section no longer triggers ruby parsing. Issue #92 by Vijay Dev
- Improved documentation for setting the default documentation format for your ruby project. Issue #94 by Henrik Hodne
- Fixed handling of LANG in the RDoc::Options tests. Issue #99 by Vít Ondruch
- RDoc no longer quits when given an entry that is not a file or directory. Issue #101 by Charles Nutter
- Fixed bug in syntax-highlighting that would corrupt regular expressions. Ruby Bug #6488 by Benny Lyne Amorsen.
- “class Object” no longer appears in the coverage report if all its methods are documented. This suppresses a false positive for libraries that add toplevel methods. Pull Request #128 by Zachary Scott.
- Fixed test_gen_url test name in TestRDocMarkupToHtml. Pull Request #130 by Zachary Scott.
- Comment-defined methods ahead of define_method are now discovered. Issue #133 by eclectic923
- Fixed detection of define_method documentation. Issue #138 by Marvin Gülker.
- Fixed lexing of character syntax (?z). Reported by Xavier Noria.
- Add license to gem spec. Issue #144 by pivotalcommon
- Fixed comment selection for classes. Pull request #146 by pioz
- Fixed parsing of def self.&() end. Issue #148 by Michael Lucy
- Generated RD parser files are now included in the gem. Issue #145 by Marvin Gülker
- Class and module aliases now create new classes to avoid duplicate names in the class list. Issue #143 by Richard Schneeman, Rails Issue #2839
- RDoc::Markup::Parser now correctly matches indentation of lists when multibyte characters are used in the list labels. Issue #140 by burningTyger
- Fixed mangling of email addresses that look like labels. Issue #129 by Tobias Koch
- Classes and modules in a C file may now be created in any order. Issue #124 by Su Zhang
- A metaprogrammed method supports the :args: directive. Issue #100
- A metaprogrammed method supports the :yields: directive.
- RDoc will now look for directives up to the end of the line. For example,
```
class B < A; end # :nodoc:
```
  will now hide documentation of B. Issue #125 by Zachary Scott
- Fixed tokenization of % when it is not followed by a $-string type
- Fixed display of __END__ in documentation examples in HTML output
- Fixed tokenization of reserved words used as new-style hash keys
- RDoc now handles class << $gvar by ignoring the body
- Fixed parsing of class A
  
  B.
- Worked around bug in RDoc::RubyLex where tokens won't be reinterpreted after unget_tk.
- Fixed class << ::Foo writing documentation to /Foo.html
- Fixed class ::A referencing itself from inside its own namespace.

Streaming zlib processing for Ruby

2012-07-10T12:52:18-07:00

Earlier today I checked in a patch that adds streaming zlib processing to Ruby. This allows you to process a stream without needing to allocate space to hold the entire result. To add the streaming support I changed #inflate and #deflate to accept a block. A handful of other methods (such as #finish) accept a block as well due to the internals of Zlib. Here's an example:

require 'zlib'

# dd if=/dev/zero of=/dev/stdout bs=1m count=1024 | gzip -c > 1G.gz
gzipped = File.read '1G.gz'

# auto-detect gzip
z = Zlib::Inflate.new Zlib::MAX_WBITS + 32

# This prints NULs
z.inflate gzipped do |chunk|
  puts chunk
end

# Don't forget to finish the stream, there may be bytes leftover.
puts z.finish

If the input was a zlib stream instead of a gzip stream, this would work:

Zlib.inflate deflated do |chunk|
  puts chunk
end

But Zlib.inflate doesn't allow specification of window bits for gzip auto-detection.

The biggest difference is in memory used. Here's 1GB of gzip-compressed NULs:

$ ll 1G.gz
-rw-r--r--  1 drbrain  staff  1042069 Jul  9 17:09 1G.gz

Here's the memory used inflating this stream:

$ /usr/bin/time -l make -s runruby > /dev/null
        3.56 real         3.25 user         0.29 sys
  46682112  maximum resident set size
         0  average shared memory size
         0  average unshared data size
         0  average unshared stack size
     12644  page reclaims
         0  page faults
         0  swaps
         0  block input operations
         0  block output operations
         0  messages sent
         0  messages received
         1  signals received
         3  voluntary context switches
        14  involuntary context switches

Only 46MB used!

Here's without streaming:

$ /usr/bin/time -l make -s runruby > /dev/null
        3.82 real         3.21 user         0.60 sys
1079824384  maximum resident set size
         0  average shared memory size
         0  average unshared data size
         0  average unshared stack size
    264889  page reclaims
         0  page faults
         0  swaps
         0  block input operations
         0  block output operations
         0  messages sent
         0  messages received
         1  signals received
         3  voluntary context switches
        18  involuntary context switches

Over 1GB used!

As you may have noticed, despite all the extra calls back to ruby to execute the block, the streaming output was 250ms faster, but does that hold up?

Benchmarks

For all these benchmarks I chose files of NULs as the inflate to large files as this exercises the growth of the output buffer the most, which is what this new feature changes. In general, these benchmarks show that in single-thread use streaming improves inflate speed especially as the output size increases while in multi-thread use streaming reduces inflate speed unless the output size grows very large.

I chose the input sizes of 10KB, 100KB and 1GB as tests of the internals of the zlib extension. At 10KB buffer expansion will only happen once for streaming mode, but several times for non-streaming. 100KB is in the realm of compressed HTTP responses and has a reasonable balance between the way streaming and non-streaming work are handled. 1GB is extremely large and when multi-threaded would be taxing on my laptop's memory capacity for non-streaming and taxing on the GVL while streaming.

Using files that are less compressible would be a better real-world example and may change the speedups shown. For single-thread tests, the speedup would likely increase while for multi-thread tests the slowdown would likely decrease.

With a 1GB file of NULs, ministat shows a 6% speedup:

x stream
+ no-stream
+------------------------------------------------------------------------------+
|                                                                   +          |
|                                                                   +          |
|                                                                   ++         |
|                                                                 ++++         |
|              x                                                 +++++         |
|         x  xxx                                                +++++++        |
|  x x xxxx  xxx x                                              +++++++        |
|  x xxxxxxx xxxxx          x                                  ++++++++   +    |
|x x xxxxxxx xxxxxx  x      x   x       x     x            x   ++++++++++ +++ +|
|  |_________MA__________|                                                     |
|                                                                |__A__|       |
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  50     3.4484119      3.690871      3.497622     3.5034132   0.045816556
+  50     3.7066622     3.7709568     3.7281418     3.7290156   0.013754228
Difference at 95.0% confidence
	0.225602 +/- 0.013422
	6.4395% +/- 0.383111%
	(Student's t, pooled s = 0.0338255)

I believe this is due to the non-streaming execution needing to realloc() the output buffer during expansion. The streaming version yields the buffer to the block and creates a new buffer which avoids the realloc() overhead for large strings.

For 100KB of NULs inflated 100 times, no speedup is shown:

x stream
+ no-stream
+------------------------------------------------------------------------------+
|     +   xxxxx                                                                |
|    +++ +xxxxx                                                                |
|    +++++*xxxxx x                                                             |
|  +++++++*xxxxxxx                  + +  +                +                    |
|+ +++++++**xxxxx* * xxx   +       +*++ ++   +  + +       ++ +  x    x     +  +|
|    |_______M__A__________|                                                   |
||________M____________A____________________|                                  |
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  50   0.036626816   0.045912027   0.037170887   0.037605586  0.0017813789
+  50   0.035252094   0.047364235   0.036613226   0.038649669  0.0033843024
No difference proven at 95.0% confidence

At 10KB of NULs inflated 1000 times each run, the improvement drops to 6%:

x stream
+ no-stream
+------------------------------------------------------------------------------+
|              ++                                                              |
|    xxx       ++                                                              |
|    xxx       +++                                                             |
|    xxx       +++                                                             |
|   xxxx x     +++                                                             |
|   xxxx x    ++++                                                             |
|   xxxx x    ++++                                                             |
|   xxxxxx    ++++ +                                                           |
|  xxxxxxx   +*+++++  +                                                        |
|  xxxxxxxx  **++++++ ++  *                + +  *   +                         +|
||____M_A______|                                                               |
|       |_______M___A___________|                                              |
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  50   0.046477079    0.05746603   0.047408819   0.047745066  0.0016545295
+  50   0.049071074   0.064721823   0.049803019   0.050768676   0.002927008
Difference at 95.0% confidence
	0.00302361 +/- 0.000943385
	6.33282% +/- 1.97588%
	(Student's t, pooled s = 0.00237748)

In this case the improvement is likely due to the way the buffer is expanded. Streaming mode jumps from the initial buffer size (1KB) to the maximum buffer size (16KB) immediately while non-streaming mode increments the buffer in steps. I'll need further performance tests to see if eliminating stepping output buffer growth is worthwhile for both modes.

When multiple zlib streams are being processed in parallel streaming is 20% slower when expanding a 10KB streams of NULs 1000 times in four threads:

x stream
+ no-stream
+------------------------------------------------------------------------------+
|              +                                         xx                    |
|           +  +                                       x xx                    |
|           ++ +                                       x xx                    |
|          +++ +                                       x xx x                  |
|          ++++++                                      x xx x                  |
|        ++++++++                                     xxxxx x x                |
|   +    ++++++++ +                                   xxxxxxx x x              |
|++ +  + ++++++++ ++    +         +                  xxxxxxxxxxxx xxx  x      x|
|                                                     |___MA____|              |
|       |____A____|                                                            |
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  50    0.19220901    0.21440268    0.19658995     0.1977196  0.0041590721
+  50    0.14678812    0.17618799    0.15705991    0.15724883  0.0045392638
Difference at 95.0% confidence
	-0.0404708 +/- 0.0017274
	-20.4688% +/- 0.87366%
	(Student's t, pooled s = 0.00435332)

With 100KB of NULs expanded 100 times in four threads there's a 14% slowdown:

x stream
+ no-stream
+------------------------------------------------------------------------------+
|        + +++ +                                      xxxx x                   |
|        + +++ +                                      xxxxxx  x                |
| + ++ +++++++ ++  +    +                   +       xxxxxxxx  x +            x |
|++ ++++++++++ ++  +  + +   + +++          ++    xxxxxxxxxxxxxx *xxx   x    xxx|
|                                                   |____M_A______|            |
|  |________M____A_____________|                                               |
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  50   0.069380999   0.076375008   0.071210146   0.071699142  0.0016866267
+  50   0.057549715   0.073060989   0.060353994   0.061419401  0.0034865206
Difference at 95.0% confidence
	-0.0102797 +/- 0.0010867
	-14.3373% +/- 1.51564%
	(Student's t, pooled s = 0.00273866)

These slowdowns are likely due to increased GVL contention. As the output size grows the slowdown decreases.

With 1G of NULs expanded once in four thread's there's a 1% speedup:

x stream
+ no-stream
+------------------------------------------------------------------------------+
|    x x +                                                                     |
|    x x++x++                                                                  |
|    x ***x++                                                                  |
|    xx******                                                                  |
|    xx******                                                                  |
|  xx********+++                                                               |
|  x**********++   +x  x +   +                 +                              +|
|    |__A___|                                                                  |
||________M__A__________|                                                      |
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  50     5.1492789     5.3822649     5.2116029     5.2147137   0.043439149
+  50     5.1704421     6.0051818     5.2339363     5.2634319     0.1322479
Difference at 95.0% confidence
	0.0487181 +/- 0.0390566
	0.934244% +/- 0.748968%
	(Student's t, pooled s = 0.0984288)

The speedup is due to realloc() needing to copy the string multiple times.

Verifying benchmarks with ministat

2012-06-27T16:48:50-07:00

Most people know to benchmark to verify performance improvements, but comparing a handful of results isn't enough. You need to be sure your results are statistically significant. ministat is a tool written by Poul-Henning Kamp that determines if two result sets are statistically significant and display the difference in performance between the two.

I recently used ministat to measure performance changes for my GVL release patch for Ruby's Zlib extension. The patch will allow multiple zlib or gzip streams to be deflated or inflated in parallel. I was asked to show any performance difference because the GVL release/acquire can be expensive depending upon the work done without the GVL.

I posted a video showing the difference between inflating very large files, but what about single-threaded use of Zlib? What about the worst case? While parallel processing of large streams is great, it shouldn't overly penalize single-thread or small-stream processing. I ran several benchmarks of single-threaded use to show minimal difference, but in the worst-case benchmarks I couldn't be sure there was any significant performance change.

Zlib background

For simplicity I'll only be mentioning inflate from here on, but the same loop handles inflate and deflate of gzip, zlib and raw deflate streams identically.

While the Zlib extension is written in C, I've written this in roughly equivalent ruby while hand-waving over some of the details of how libz works since they're unimportant to the performance comparison. Before my patch run loop looked like this:

setup_input_buffer stream
setup_output_buffer stream

loop do
  status = inflate stream

  schedule_another_thread

  return output_buffer if status == STREAM_END

  if status == BUF_ERROR then
    expand_output_buffer stream
    next
  else
    # handle other errors
  end
end

First the input and output buffers are set up. The input buffer contains the deflated string to be inflated and the output buffer will contain the inflated string. The default output buffer size is 1024 bytes.

Inside the loop inflate is called which will expand as much of the input stream as will fit in the output buffer, then ruby is given the chance to schedule another thread to provide fairness, then errors are checked.

A BUF_ERROR indicates there isn't enough room in the output buffer and more space needs to be added. The expand_output_buffer function adds up to another 16KB to the output buffer. Combined with scheduling another thread each time inflate returned, fairness is provided.

With my patch the run loop looks something like this:

setup_input_buffer stream
setup_output_buffer stream

release_GVL do
  until interrupted do
    status = inflate stream

    return output_buffer if status == STREAM_END

    if status == BUF_ERROR then
      expand_output_buffer_without_GVL stream
      next
    else
      # …
    end
  end
end

The differences are releasing the GVL, the interrupted flag and the change in expanding the output buffer.

To reduce copying the Zlib extension uses a ruby String for the output buffer, so expanding it requires GC interaction. The new function uses realloc(3) directly to avoid GVL contention and mirrors rb_str_resize().

The interrupted flag allows processing of timeouts, a killed thread or clean shutdown to proceed smoothly. Like the no-GVL version the buffer is only expanded by 16KB to allow interruptions to be serviced quickly. Without this flag a large stream would need to complete before the thread it is running in could be shut down which may take several seconds. (In the C version of release_GVL, rb_thread_blocking_region(), you provide an unblocking function that sets the interrupted flag when ruby wants to regain control of the thread.)

Releasing the GVL allow other threads to run the Ruby VM so while a stream is being inflated other threads can run on other CPUs. For large streams this can result in a performance increase when other Ruby threads are running.

Worst Case Benchmark

After inspecting the two implementations we have enough information to create a worst-case scenario for testing the performance of the patch.

Since GVL release/acquire can be expensive we want to do that as much as possible which means inflate should do as little as possible. The buffer should not be expanded since that allows more than one trip through inflate() per GVL release/acquire, so inflated strings must fit in the 1024 byte output buffer.

I wrote a benchmark that inflates many strings with an inflated size of 1000 bytes using multiple threads, but not so many strings as to prevent a quick return from the benchmark section. (While inflating strings of zero bytes would probably be worse, that seems unrealistic.)

Here's the benchmark program I wrote:

require 'zlib'
require 'benchmark'

r = Random.new 0

file_count = 10_000

deflated = (0..file_count).map do
  input = r.bytes 1000
  Zlib::Deflate.deflate input
end

times = Benchmark.measure do
  (0..3).map do
    Thread.new do
      deflated.each do |input|
        Zlib::Inflate.inflate input
      end
    end
  end.each do |t|
    t.join
  end
end

puts times.real

Instead of using time, Benchmark.measure is used, but only around the inflate part. This reduces the amount of noise in the benchmark. By choosing only 10,000 files multiple benchmarks can be run (on my machine the inflate portion takes a little over ½ second). By using the same random seed the benchmark is repeatable.

Before using ministat I ran the same benchmark but with 100,000 files I had the following results without the patch:

$ for f in `jot 5`; do ruby20 test.rb; done
5.420000   5.970000  11.390000 (  8.162893)
5.400000   6.270000  11.670000 (  8.263046)
5.460000   5.920000  11.380000 (  8.133742)
5.410000   6.290000  11.700000 (  8.289913)
5.500000   6.620000  12.120000 (  8.478085)

and with the patch:

$ for f in `jot 5`; do make -s runruby; done
5.120000   6.240000  11.360000 (  8.039715)
5.240000   6.260000  11.500000 (  8.097961)
5.280000   5.940000  11.220000 (  8.004246)
5.210000   6.360000  11.570000 (  8.171124)
5.240000   6.200000  11.440000 (  8.054929)

So while running with the patch seemed faster, the results overlap by a small margin. I wanted to be reasonably sure there was an improvement which is where ministat comes in. In order to get useful data out of ministat I would need more data (this is easy to verify by running ministat on these results and seeing that it says "No difference").

To capture the data I used a shell for loop to run the benchmark 50 times and redirect the output to a file: for f in `jot 50`; do make -s runruby; done > with, and again without the patch applied directed to a "without" file.

ministat

To determine if the results were statistically significant I ran the data through ministat with a confidence interval of 99%. From the ministat man page:

The ministat command was written by Poul-Henning Kamp out of frustration over all the bogus benchmark claims made by people with no understanding of the importance of uncertainty and statistics.

So through ministat we will be able to determine if the suspected difference from the previous benchmark is an actual difference, and what the certainty of that difference is.

$ ministat -c 99 -s without with
x without
+ with
+------------------------------------------------------------------------------+
|                           +     + +  x    *  xx      x                       |
|         +     x +   +     +  +  + +  x   **+xx*xx    x * x +               x |
|+        ++++  x + x*+ * + ++x+x++x*++x* ***+x**xx * +x**** * + x   x       xx|
|                                |_____________A_____________|                 |
|                    |______________A_______________|                          |
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  50    0.61660767    0.71641994    0.66669798    0.66647618   0.022548872
+  50    0.59242272    0.69294882    0.64941692    0.64917859    0.02509235
Difference at 99.0% confidence
	-0.0172976 +/- 0.0125332
	-2.59538% +/- 1.88051%
	(Student's t, pooled s = 0.0238545)

Looking at the last four lines, you can see there is a statistically significant difference in the benchmark results at 99% confidence. Running the benchmark with the patch shows a 17ms decrease in time ± 13ms or a 2.6% decrease in time ± 1.9% for this sample set, so I can say "minor improvement".

The box shows a plot of the with and without values along with the average (and median) values, and standard deviation bars. For full details see the ministat man page.

There are a couple of likely reasons for the slight increase in performance such as the GVL release/acquire being inexpensive enough that inflate takes longer allowing parallelism or that GVL release/acquire isn't as expensive as thought, but without an easy way to measure GVL contention it's only speculation.

Getting ministat

To my knowledge ministat only ships with FreeBSD, but fortunately it is really easy to add to nearly any system with a C compiler. You can find a copy of ministat in this repository (which, ironically, uses autotools) or you can grab the C file directly and run cc ministat.c -lm -o ministat and move the ministat executable into your PATH like I did.

Mechanize 2.5

2012-05-09T13:48:00-07:00

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, and can follow links and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

Changes

Minor enhancement
- Added Mechanize#ignore_bad_chunking for working around servers that don’t terminate chunked transfer-encoding properly. Enabling this may cause data loss. Issue #116
- Removed content-type check from Mechanize::Page allowing forced parsing of incorrect or missing content-types. Issue #221 by GarthSnyder
Bug fixes
- Fixed typos in EXAMPLES and GUIDES. Pull Request #213 by Erkan Yilmaz.
- Fixed handling of a quoted content-disposition size. Pull Request #220 by Jason Rust
- Mechanize now ignores a missing gzip footer like browsers do. Issue #224 by afhbl
- Mechanize handles saving of files with the same name better now. Pull Request #223 by Godfrey Chan, Issue #219 by Jon Hart
- Mechanize now sends headers across redirects. Issue #215 by Chris Gahan
- Mechanize now raises Mechanize::ResponseReadError when the server does not terminate chunked transfer-encoding properly. Issue #116
- Mechanize no longer raises an exception when multiple identical radiobuttons are checked. Issue #214 by Matthias Guenther
- Fixed documentation for pre_connect_hooks and post_connect_hooks. Issue #226 by Robert Poor
- Worked around ruby 1.8 run with -Ku and ISO-8859-1 encoded characters in URIs. Issue #228 by Stanislav O.Pogrebnyak

Mechanics of Programming

2012-05-03T17:57:28-07:00

I sat down with Elise Worthy at RailsConf to pair with her on a Rails application that used the Wunderground API to retrieve the weather for multiple cities. If you don't know Elise, she's a Hungry Academy student formerly of Seattle. You can read about how she built her app on the Hungry Academy Blog. Since Elise is new to programming, pairing with her made me think about how I work on small tasks when implementing a larger feature.

When I first sat down, Elise had a class to retrieve the weather for Washington D.C., a controller and a view to display it. Our task was to add a search box so you could show the weather for any ZIP code in addition to Washington D.C.

By implementing weather for a fixed city first Elise had followed the most difficult lesson for me to learn, only implement one idea, or story, at a time. When I try to add too much I can't fit all the details in my head. By making the weather work for one fixed city she had a solid base of work she could revert to to try again if she got stuck.

I mostly helped her with the question of "what should I do next?" Elise knew which steps she needed to perform but didn't know what the best order was. I've found that working on the easiest task first is best. By picking the easiest task you have time to think about the hard tasks in the back of your mind. By the time you get around to them you may have made them easier or you may have come up with an easier way to implement them.

In the course of doing the easiest thing next we switched back and forth between the API wrapper, the controller and the view. Each change we made was very small. For example, first add the search form, next extract the ZIP code from params, then replace showing Washington D.C. with the user's choice.

Each step only changed a couple lines that we verified in the browser so we didn't get ahead of ourselves. By first changing from a fixed city to an arbitrary city we advanced towards our goal without confusing ourselves with extra details. While this broke one feature of the app, it was obvious that it would be easy to restore in the future when we were ready to address it.

I also find it useful to add TODO comments when I know I need to change or fix something but don't want to do it yet. The TODO item may be a distraction at this time or something difficult that a future step will make easier. I find it easy to forget some necessary cleanup without them. We added a few when Elise would think of something she needed to change but I didn't want to get us distracted.

There was one other tip I had for Elise, as she was adding support for both Washington D.C. and the user's ZIP her first idea was to use two instance variables to provide the data to the view, one for Washington D.C. and the other for the user's city. She also asked if I thought this was the best way.

I told her that it is easier for me to think of handling just one item or handling many items, but not just two items. By having only these two categories of items to display it makes it easier to maintain your code by preventing duplication.

The best thing that came out of pairing with Elise was that I got to think about how I program at the lowest level. While I got to show Elise some tips I've learned over the years, she showed me the details of my own internal process that I barely think about anymore.

Mechanize 2.4 Security Fix

2012-04-20T17:53:57-07:00

Changes

Security fix:

Mechanize#auth and Mechanize#basic_auth allowed disclosure of passwords to malicious servers and have been deprecated.

In prior versions of mechanize only one set of HTTP authentication credentials were allowed for all connections. If a mechanize instance connected to more than one server then a malicious server detecting mechanize could ask for HTTP Basic authentication. This would expose the username and password intended only for one server.

Mechanize#auth and Mechanize#basic_auth now warn when used.

To fix the warning switch to Mechanize#add_auth which requires the URI the credentials are intended for, the username and the password. Optionally an HTTP authentication realm or NTLM domain may be provided.
Minor enhancement
- Improved exception messages for 401 Unauthorized responses. Mechanize now tells you if you were missing credentials, had an incorrect password, etc.