Segment7

Mechanize 2.5

2012-05-09T13:48:00-07:00

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, and can follow links and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

Changes

Minor enhancement
- Added Mechanize#ignore_bad_chunking for working around servers that don’t terminate chunked transfer-encoding properly. Enabling this may cause data loss. Issue #116
- Removed content-type check from Mechanize::Page allowing forced parsing of incorrect or missing content-types. Issue #221 by GarthSnyder
Bug fixes
- Fixed typos in EXAMPLES and GUIDES. Pull Request #213 by Erkan Yilmaz.
- Fixed handling of a quoted content-disposition size. Pull Request #220 by Jason Rust
- Mechanize now ignores a missing gzip footer like browsers do. Issue #224 by afhbl
- Mechanize handles saving of files with the same name better now. Pull Request #223 by Godfrey Chan, Issue #219 by Jon Hart
- Mechanize now sends headers across redirects. Issue #215 by Chris Gahan
- Mechanize now raises Mechanize::ResponseReadError when the server does not terminate chunked transfer-encoding properly. Issue #116
- Mechanize no longer raises an exception when multiple identical radiobuttons are checked. Issue #214 by Matthias Guenther
- Fixed documentation for pre_connect_hooks and post_connect_hooks. Issue #226 by Robert Poor
- Worked around ruby 1.8 run with -Ku and ISO-8859-1 encoded characters in URIs. Issue #228 by Stanislav O.Pogrebnyak

Mechanics of Programming

2012-05-03T17:57:28-07:00

I sat down with Elise Worthy at RailsConf to pair with her on a Rails application that used the Wunderground API to retrieve the weather for multiple cities. If you don't know Elise, she's a Hungry Academy student formerly of Seattle. You can read about how she built her app on the Hungry Academy Blog. Since Elise is new to programming, pairing with her made me think about how I work on small tasks when implementing a larger feature.

When I first sat down, Elise had a class to retrieve the weather for Washington D.C., a controller and a view to display it. Our task was to add a search box so you could show the weather for any ZIP code in addition to Washington D.C.

By implementing weather for a fixed city first Elise had followed the most difficult lesson for me to learn, only implement one idea, or story, at a time. When I try to add too much I can't fit all the details in my head. By making the weather work for one fixed city she had a solid base of work she could revert to to try again if she got stuck.

I mostly helped her with the question of "what should I do next?" Elise knew which steps she needed to perform but didn't know what the best order was. I've found that working on the easiest task first is best. By picking the easiest task you have time to think about the hard tasks in the back of your mind. By the time you get around to them you may have made them easier or you may have come up with an easier way to implement them.

In the course of doing the easiest thing next we switched back and forth between the API wrapper, the controller and the view. Each change we made was very small. For example, first add the search form, next extract the ZIP code from params, then replace showing Washington D.C. with the user's choice.

Each step only changed a couple lines that we verified in the browser so we didn't get ahead of ourselves. By first changing from a fixed city to an arbitrary city we advanced towards our goal without confusing ourselves with extra details. While this broke one feature of the app, it was obvious that it would be easy to restore in the future when we were ready to address it.

I also find it useful to add TODO comments when I know I need to change or fix something but don't want to do it yet. The TODO item may be a distraction at this time or something difficult that a future step will make easier. I find it easy to forget some necessary cleanup without them. We added a few when Elise would think of something she needed to change but I didn't want to get us distracted.

There was one other tip I had for Elise, as she was adding support for both Washington D.C. and the user's ZIP her first idea was to use two instance variables to provide the data to the view, one for Washington D.C. and the other for the user's city. She also asked if I thought this was the best way.

I told her that it is easier for me to think of handling just one item or handling many items, but not just two items. By having only these two categories of items to display it makes it easier to maintain your code by preventing duplication.

The best thing that came out of pairing with Elise was that I got to think about how I program at the lowest level. While I got to show Elise some tips I've learned over the years, she showed me the details of my own internal process that I barely think about anymore.

Mechanize 2.4 Security Fix

2012-04-20T17:53:57-07:00

Changes

Security fix:

Mechanize#auth and Mechanize#basic_auth allowed disclosure of passwords to malicious servers and have been deprecated.

In prior versions of mechanize only one set of HTTP authentication credentials were allowed for all connections. If a mechanize instance connected to more than one server then a malicious server detecting mechanize could ask for HTTP Basic authentication. This would expose the username and password intended only for one server.

Mechanize#auth and Mechanize#basic_auth now warn when used.

To fix the warning switch to Mechanize#add_auth which requires the URI the credentials are intended for, the username and the password. Optionally an HTTP authentication realm or NTLM domain may be provided.
Minor enhancement
- Improved exception messages for 401 Unauthorized responses. Mechanize now tells you if you were missing credentials, had an incorrect password, etc.

rdoc-spellcheck 1.0

2012-04-09T16:34:30-07:00

rdoc-spellcheck checks your documentation for spelling errors. File, class, and method names are automatically excluded from the results and you can add your own words to the default word list.

Features

Checks documentation for pages, classes, modules, methods, constants, etc.
Prints a report showing each misspelled word and correction suggestions

Synopsis

$ rdoc -f spellcheck lib *.rdoc

Requirements

rdoc
libaspell – See github.com/evan/raspell/blob/master/README.rdoc for installation instructions for aspell

Install

sudo gem install rdoc-spellcheck

ruby-growl 4.0

2012-04-04T17:36:32-07:00

ruby-growl version 4.0 has been released!

Documentation

A pure-ruby growl notifier for UDP and GNTP growl protocols. ruby-growl allows you to perform Growl notifications from machines without growl installed (for example, non-OSX machines).

What is growl? Growl is a really cool “global notification system originally for Mac OS X”.

You can receive Growl notifications on various platforms and send them from any machine that runs Ruby.

OS X: growl.info Windows: www.growlforwindows.com/gfw/ Linux: github.com/mattn/growl-for-linux

ruby-growl also contains a command-line notification tool named ‘growl’. It is almost completely option-compatible with growlnotify. (All except for -p is supported, use –priority instead.)

Changes:

API changes:
- Growl is now a wrapper for Growl::UDP (1.2 and older) and Growl::GNTP (1.3 and newer). The main difference is that notifications need to be registered with Growl#add_notification instead of via Growl#initialize.
- Ruby 1.9.2 or newer is required to use ruby-growl
Major enhancements
- Added GNTP protocol support for registration and notification including application and notification icons and callbacks.
- Moved UDP protocol support to Growl::UDP
- Growl automatically determines if the growl server supports GNTP or UDP and uses the best protocol.

return_bang 1.1

2012-04-01T13:10:29-07:00

home: github.com/drbrain/return_bang
rdoc: docs.seattlerb.org/return_bang
bugs: github.com/drbrain/return_bang/issues

return_bang implements non-local exits for methods. As a bonus, you also get exception handling that ignores standard Ruby’s inflexible begin; rescue; ensure; end syntax.

Use return_bang to exit back to a processing loop from deeply nested code, or just to confound your enemies and your friends! What could possibly go wrong?

1.1 / 2012-04-01

Minor enhancements
- Added raise! to raise exceptions
- Added rescue! to rescue exceptions raised
- Added ensure! to always execute a block of code to perform cleanup

On Community Funding of Open Source

2012-03-29T19:49:42-07:00

The other day Yehuda Katz announced a kickstarter for creating Rails.app, an OS X application that makes it easy to bring new programmers to Rails and Ruby.

I think the idea is fantastic. When I first started learning Ruby I had to do battle with compilers, RubyGems didn't exist, the RAA had but a handful of libraries, and there weren't really even any tutorials for learning Ruby around. I had either fortitude or stubbornness on my side to get through the much steeper learning curve than many of you had to deal with to get where I am today, and it sucked.

I think the idea of soliciting community for funding is fantastic. It's not the first time a Rubyist has solicited money to work on open source. I funded Gregory Brown's work on Prawn a few years ago and was happy to do so even though I've never used Prawn (although I'd like to someday) I was incredibly happy to support it. I'll probably never use Rails.app, either, but that doesn't mean I'm not happy to support it, nor that Rubyists are wrong to support it.

Now, to get to my point, I don't understand why I keep seeing such negative feedback around Yehuda's choice of soliciting money to bring us Rails.app. Sure, it may just be a few of you, but if you don't want to give Yehuda money, fine, just don't. If you think Yehuda is asking for too much money, fine, just don't donate. If you just don't like Yehuda, fine, just don't donate.

But stop with the personal attacks, even the snarky ones and the jokes at Yehuda's expense.

You would not find it so funny if people were ganging up on you.

This is a fantastic idea Yehuda has presented and if he had the ability he would bring it to us for free. Right now he doesn't have that ability so he's asking for our help.

Being a major contributor to open source is incredibly difficult. While thousands or even millions of people may be happily using your software, you mostly hear from the people who are having problems with it, especially after they're extremely frustrated. Throwing insults atop this is incredibly demotivating and depressing, so cut it out.

If want to continue to complain, whinge or make cruel jokes, towards Yehuda or any other Rubyist, why not take that effort and put it towards something positive. Do something that is welcoming to new Rubyists like improving some ruby documentation, submitting a bug report or submitting a patch to fix a bug.

Stop making our community look like a bunch of jerks. It's fine if you disagree with another Rubyist, especially one who has contributed so heavily to Ruby, but you should have enough respect for them to be polite.

net-http-persistent 2.6

2012-03-26T15:17:50-07:00

Manages persistent connections using Net::HTTP plus a speed fix for Ruby 1.8. It’s thread-safe too!

Using persistent HTTP connections can dramatically increase the speed of HTTP. Creating a new HTTP connection for every request involves an extra TCP round-trip and causes TCP congestion avoidance negotiation to start over.

Net::HTTP supports persistent connections with some API methods but does not handle reconnection gracefully. Net::HTTP::Persistent supports reconnection and retry according to RFC 2616.

Changes:

Minor enhancement
- Net::HTTP::Persistent#idle_timeout may be set to nil to disable expiration of connections. Pull Request #21 by Aaron Stone

A use of Enumerable#chunk

2012-03-16T16:19:07-07:00

In Ruby 1.9, Enumerable has a few new methods including Enumerable#chunk (which was added for 1.9.2). The #chunk method walks your Enumerable and divides it into chunks based on a selecting block. Unlike Enumerable#partition, the chunks are returned in-order. Here's an example from the documentation:

[3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5].chunk { |n|
  n.even?
}.each { |even, ary|
  p [even, ary]
}
#=> [false, [3, 1]]
#   [true, [4]]
#   [false, [1, 5, 9]]
#   [true, [2, 6]]
#   [false, [5, 3, 5]]

When I first saw this method I thought, "this looks like a useful method… but how?"

I'm working on bringing Markdown support to RDoc and the last remaining base Markdown feature I need to support is a hard break due to two spaces at the end of a line in a paragraph.

For background, RDoc parses various formats into a common syntax tree which is can then be transformed for any supported output (such as HTML, colored ANSI text, etc.). In this syntax tree a paragraph can contain one or more strings which are joined at output time into the paragraph you see.

To add hard line breaks, I decided to create a new HardBreak object and inject it into the paragraph where two trailing spaces are encountered in the source document. The formatters can then be updated to insert the appropriate line break character when emitting a paragraph.

Enumerable#chunk comes in because the Markdown parser doesn't join strings as it's parsing (since the grammar rules get re-used) and is instead performed as a post-processing step. (String joining as a post-processing step also makes the parser cleaner by hiding the ugliness in one spot rather than spreading it across multiple grammar rules.) Before inserting HardBreak objects this was sufficient:

parts = paragraph.parts.join.rstrip
paragraph.parts.replace [parts]

But now I need to join String chunks and include HardBreaks as-is which is a perfect use of Enumerable#chunk:

parts = paragraph.parts.chunk do |part|
  String === part
end.map do |string, chunk|
  string ? chunk.join.rstrip : chunk
end.flatten

paragraph.parts.replace parts

The 1.8-compatible implementation is much uglier since I have to track whether I'm in a String chunk or not in addition to performing the processing. I'm too embarrassed to post it, but you'll be able to find it in the rdoc source once I commit and push it.

hoe-travis

2012-03-01T16:46:15-08:00

hoe-travis is a Hoe plugin that allows your gem to gain maximum benefit from travis-ci.org. The plugin contains a .travis.yml generator and a pre-defined rake task which runs the tests and ensures your manifest file is correct.

With hoe-travis it is easy to add additional checks. Custom checks can be easily verified locally by simply running a rake task instead of committing and pushing a change, waiting for travis to run your tests, then trying a new commit if you didn’t fix the problem.

Features

.travis.yml generation task
Pre-defined rake tasks which are run by travis-ci
Easy to hook up rake tasks for additional travis-ci setup or checks

Getting Started

If you’re not already using Hoe with your project, see: docs.seattlerb.org/hoe/Hoe.pdf

To get started with hoe-travis, first install it:

sudo gem install hoe-travis

Then add hoe-travis as a plugin to your Rakefile:

Hoe.plugin :travis

Then generate a .travis.yml

$ rake travis:generate

This will bring up your EDITOR with your travis.yml for any desired tweaks. Save the file when you’re done, then check in your .travis.yml. For further details of how the configuration is generated see Setup at Hoe::Travis and Configuration at Hoe::Travis.

(If you don’t have the EDITOR environment variable set to your favorite editor, please do so. Note that some editors may need extra flags to wait for files to be edited. For MacVIM, export EDITOR="mvim --remote-wait" will wait for the file to be closed before returning.)

If you would like to make future changes to your .travis.yml you can run:

$ rake travis:edit

Which, like travis:generate, will bring up your EDITOR with your .travis.yml. When you’ve saved the file the changes will be checked by travis-lint before writing back to .travis.yml and give you a chance to correct them.

If you’ve edited your .travis.yml by hand you can run:

$ rake travis:check

to check it.

Testing your travis-ci setup is easy with hoe-travis. You can run:

$ rake travis

to run the same checks travis-ci will. By default this includes running the tests and ensuring the Manifest.txt file is complete. There is also the before script:

$ rake travis:before

Which will run the setup tasks needed for your project.

You can also enable and disable travis-ci using rake travis:enable and rake travis:disable. See Setup at Hoe::Travis for details.