paydrotalks

Unix `cat` For File Creation

2012-04-26T12:56:16-07:00

I spend copious amounts of time on the servers at 8tracks.com. On occasion, I upload a ruby/base/etc script to the server to test. In the past I normally fire up vi on the server, paste the code in, save it, and execute it. When the script fails because of a bug, I fix the file on my local editor, copy the code, reopen the file on the server with vi, save it, and execute it. This is slow. Here’s a faster way using cat and the CTRL+d keyboard shortcut in a terminal.


$ cat > my_new_file
Hello, world!
<hit CTRL+d here>
$ cat my_new_file
Hello, world!

This removes vi from the equation. This works for my previous tip about executing arbitrary ruby code on the shell.

Punctuation In Quotes

2012-04-19T09:41:28-07:00

I’m always confused by punctuations marks with quotations. For instance, where do you place the period when the sentence ends with quotes? Does the period go inside the quotes or outside? I know the rules (inside), but it still bothers me and I couldn’t figure out why … until I found this (emphasis mine)!

When it comes to commas and periods, though, logic doesn’t enter into the equation, at least not in the United States. Universal American usage places commas and periods inside the quotation marks, regardless of logic.

I was doing it all wrong with logic! It’s like measuring distance in America. Obviously, one mile is 5,280 feet. And one foot is 12 inches! DUH.

Load mysqldump CSVs into Ruby FasterCSV

2012-04-11T14:55:16-07:00

Working with mysqldump CSV files using the FasterCSV is a pain. mysqldump converts NULL fields into \N and it uses backslashes (‘\’) to escape characters. The only character in a well formed CSV that you need to escape is the double quote (‘"’). You can change the mysqldump escape characters and force double quotes around the fields, but FasterCSV::MalformedCsvError errors still occur.

After multiple tries I looked to shell scripting to get this working. Modifying large files this way wont be fast, but I do this with scripts that run in the background so I don’t mind the time it takes.

First thing is to dump your table(s) into CSV formats. The following mysqldump command outputs tab separated fields with no surrounding quotes.

  # Creates a "/path/to/store/csvs/your_table_here.txt" file
  $ mysqldump -u<user> \
  > -p<password> \
  > -t --tab=/path/to/store/csvs/ your_db_here your_table_here

Since the only double quote in the file belongs to the content, I can escape them with sed. At the same time, I like to change \N to NULL. This sed command does the trick.

  # Edit in place
  $ sed -i \
  > -e 's/"/""/g' \
  > -e 's/\t[[:space:]]*\\N/\tNULL/g' \
  > /path/to/store/csvs/your_table_here.txt

Now that everything is properly escaped I double quote the fields and replace the tabs with commas.

  # Edit in place
  $ sed -i \
  > -e 's/\t/","/g' \
  > -e 's/^\(.*\)$/"\1"/g' \
  > /path/to/store/csvs/your_table_here.txt

You can combine both of those steps into one sed command, but I like having them separate in my scripts. It’s easier to digest.

Now you’re ready to use that file with FasterCSV (or CSV if you’re on ruby 1.9) with the default settings!

  FasterCSV.open('/path/to/store/csvs/your_table_here.txt') do |row|
    puts row[0], row[1] # ...
  end

I like to use a converter to convert those NULL fields to nils.

  FasterCSV.open(
    '/path/to/store/csvs/your_table_here.txt', 
    :converters => lambda{|f| f == "NULL" ? nil : f}
  ) do |row|
    puts row[0], row[1] # ...
  end

Ruby Scripting with Shell Here Documents

2012-04-11T12:49:32-07:00

Ever need to run a command you know how to do very well in Ruby but don’t know off hand with bash? It’s pretty easy if you have a single line handy for ruby like:

$ ruby -e 'puts "hello!"'

But, what happens if the ruby code is more complex and has multiple lines? You can obviously use semi-colons, but that’s less elegant. Let me introduce you to bash heredocs. Here’s an example of uploading a file to S3 with the right_aws gem.

$ cat <<RUBIES | ruby
> # Write my rubyies here!
> require 'right_aws'
> s3 = RightAws::S3.new('key', 'secret_key')
> bucket = s3.bucket("my_awesome_bucket")
> bucket.put("path/in/s3/my_large_file.csv", open('/path/to/local/file.csv', 'r'))
> RUBIES

Let's rock out

2012-04-05T09:34:48-07:00

I Love Rock ‘n’ Roll from paydro on 8tracks.

Chrome Browser Just Made My Day

2011-08-17T14:38:52-07:00

Chrome added the ability to view :hover or :focus styles in the web inspector.

This is amazing.

If only they added :before and :after as well.

Racc

2011-04-01T12:54:23-07:00

I just came across Racc – a ruby implementation of Yacc. This is bringing back memories of my compiler course in college. I’m geeking out!

In college, I built a compiler for Pascal with Lex and Yacc. The project converted Pascal code to Lisp code and ran on Sun machines. It was pretty awesome. Sadly, I don’t have the code anymore, but I loved building it!

I started building a Mysql client in the web (think phpmyadmin but without the suck) and I have a need to parse SQL. Racc + SQL Parsing? Can there be anymore joy? Also, Racc’s runtime modules come with the Ruby standard library. How did I miss this?

Time to build a simple SQL parser in Ruby.