Murray's Journal

HistogramTools 0.3

2013-12-17T11:00:00.000-08:00

A new version 0.3 of HistogramTools is now on CRAN. HistogramTools provides a number of methods for manipulating histograms, measuring the distance between histograms, calculating the information loss due to binning aggregate data sets, and other tools useful for statistical analysis of binned/histogram data. It also uses RProtoBuf to provide a protocol buffer representations of the default R histogram class to allow histograms over large data sets to be computed and manipulated in a MapReduce environment with tools written in other languages.

The full list of updates includes :

Moved 'Hmisc' from Depends to Imports.
Improved introduction vignette significantly.
Added ScaleHistograms function.
Added PlotRelativeFrequency function to plot relative frequency histograms.
Added minkowski.dist, intersect.dist, kl.divergence, jeffrey.divergence measures for two histograms.
Added PreBinnedHistogram for creating histogram objects from an already binned dataset (e.g. just a vector of bins and counts).

Dirk's CRANberries service provides a diff to the previous release 0.2. More information is at the HistogramTools page on CRAN which includes the 18-page package vignette and 1-page Quick Reference Guide. Please mail me directly with any questions or suggestions about this package.

New Work on Flash Provisioning at USENIX ATC on June 26

2013-06-07T23:21:00.000-07:00

Later this month I'll be at the USENIX Annual Technical Conference in San Jose with some coauthors on the Storage Analytics and Colossus teams at Google to present some of our recent work on optimizing flash provisioning for cloud storage workloads. Our paper is titled "Janus: Optimal Flash Provisioning for Cloud Storage Workloads", and a pre-print is available from Google Research.

This work is about using statistical samples of I/O patterns from a large distributed filesystem to formulate and solve an optimization problem that helps us allocate flash better in our datacenters. I'm looking forward to returning to USENIX ATC as it has been nearly 10 years since I've been to this conference. Shoot me a mail if you will be there and want to meet up.

Two Recent Short Papers

2012-10-14T17:33:00.000-07:00

My group at Google continues to grow, and we had the opportunity to publish a few short workshop papers this year about some of the areas we've investigated this year.

Projecting Disk Usage Based on Historical Trends in a Cloud Environment, Proceedings of the 3rd international workshop on Scientific cloud computing, ACM (2012)
Uncertainty in Aggregate Estimates from Sampled Distributed Traces, 2012 Workshop on Managing Systems Automatically and Dynamically, USENIX

The first paper describes some of the work we've done on forecasting storage growth in datacenters for capacity planning purposes using ensemble forecasting methods and trend-change detection. It builds on some of the earlier work we did for websearch traffic forecasting and, to a lesser extent, building a market economy for datacenter resources.

The second paper, to which I made only minor contributions, is a more mathematical description of a method of quantifying the uncertainty in aggregate metrics from a sampled RPC tracing system for large-scale distributed systems (e.g. Dapper).

Both papers are addressing problems that usually come up in very large-scale distributed systems, and the applicability is somewhat limited in smaller contexts, but I would be very interested in feedback regardless.

Cycling 320+ Miles Next Week for Charity

2012-09-06T23:34:00.000-07:00

Next week I will be cycling from Eureka to San Francisco for the California Climate Ride. I'll be mostly out of touch for the week, but will try to post pictures and check in via email and mobile phone when possible. Please consider donating towards my fundraising goal to support the San Francisco Bicycle Coalition. Make a Donation!

What I've been up to..

2010-10-28T23:20:00.000-07:00

It's been nearly a year since I posted here and much has changed. The obvious and most important change is a second new addition to our family which I've been blogging about elsewhere. On the work front I was able to publish a paper about some of my work studying the Availability in Globally Distributed Storage Systems at Google last year. This is an exciting space given the growth of cloud based storage services and sophisticated distributed storage software.

I've been blogging a little more regularly about work-related topics on Google company blogs, with four posts so far this year :

As you can see I've been working on data analysis, distributed cloud storage, and open source, along with some other projects I'm not yet able to talk about. I'll try to post more updates about some of my interests and side projects in the remainder of the year.

Fun with Amazon Web Services

2010-01-10T17:25:00.000-08:00

Amazon has been doing a really great job at selling excess compute capacity in their datacenters through products such as Amazon Elastic Compute Cloud (EC2), Elastic MapReduce, and their simple and structured distributed storage products. The economics of this kind of model, as represented in the two graphs here are clearly compelling. Instead of buying large numbers of computer to mostly sit idle, new start-up companies, researchers, and individuals can rent the excess capacity from Amazon instead. Last year I worked on some related ideas for internal pricing and provisioning of resources at work. This was my first direct experience with the Amazon consumer offerings however, and I was impressed. It took less than half an hour last night to sign up, start a few basic Linux instances, copy some application code over, compile it, and begin running it on the Linux Xen instances.

Not everything is so easily scaled to run on more computers. Some tasks are more feasibly done with human involvement, and I've also been experimenting with Amazon Mechanical Turk as well. This service is named after the 18th century fake chess-playing machine that actually used a hidden human operator to control the device. I have used this service recently to improve the captions for FreeBSD technical conference videos that I am involved with and the results have been stunning.

The results of cheap on-demand distributed computer clusters and a global english language work force that can be paid by the task almost engender too many business ideas to contemplate.. If only there were more hours in the day..

Support Simon Singh and Scientific Debate

2009-06-07T23:12:00.000-07:00

Simon Singh has been sued for libel by the British Chiropractic Association. Simon is an author, journalist, and TV producer who works to popularize math and science. I had the opportunity to hear Simon speak about an earlier book on the Big Bang at Keble College, Oxford. Simon wrote a more recent book on alternative medicine and suggests that there is no evidence for the efficacy of chiropractic treatments for asthma, ear infections, and other infant conditions. British Libel laws are more strict than those in the U.S. and this scientific debate has unbelievably been construed as a form of libel. Read more about the dispute and sign the petition here.

Erdős Number

2009-06-01T01:10:00.000-07:00

My current Erdős number is 4. There are several paths of length 3 from my M.Sc. advisor, Joël Ouaknine, to Paul Erdős. The path currently returned by the AMS Collaboration Distance Calculator is:

Murray Stokely coauthored with Joël Ouaknine
Joël Ouaknine coauthored with A. W. Roscoe
A. W. Roscoe coauthored with Mary Ellen Rudin
Mary Ellen Rudin coauthored with Paul Erdős

Dice Wars

2009-05-09T17:06:00.000-07:00

I was almost completely unaware of the phenomenon of Flash Games until late last year when a friend of mine started working at a company that makes ads for them. I recently discovered the Dice Wars game which is a typical example of the genre. It is simple yet addictive. The basic premise is similar to the classic board game Risk. Unlike the board game however, the board is smaller, you can not transit armies, army placement is random, and games are much much quicker.

Although the dice layout is completely random, it is addicting because much of the game involved strategy of placement. New armies are awarded after each turned based on the largest connected set of game territories your armies control. As with Risk, you roll one die for each army and the sum of the faces for all N attacking dice is compared to the sum of the faces for all M defending dice with defenders winning the tie. Some quick R code can be used to compute the probabilities of winning a given attack given N attacking armies and M defending armies. The left column represents the number of attacking dice and the first row represents the number of defending dice. Each cell represents the probability of a successful attack given M vs N fair dice.

Dice	1	2	3	4	5	6
1	.4167	.0926	.0116	.0008	2e-05	0
2	.8380	.4437	.1520	.0358	.0061	.0077
3	.9730	.7786	.4536	.1917	.0607	.0149
4	.9973	.9392	.7427	.4595	.2204	.0834
5	.9998	.9879	.9093	.7181	.4637	.2424
6	.999997	.9982	.9753	0.884	.6996	.4667

Back after a 4 year hiatus...

2009-04-28T23:02:00.000-07:00

I've imported a number of short posts I made on a previous personal blog from 2004-2005 and relaunched this as blog.stokely.org. My posts in the past tended to be short updates about travel plans or links to pictures. Those type of updates now go to something like Flickr, Facebook, or Twitter, so I'll be using this blog to post less frequent but longer musings about technology, math, computer science, travel, and life in the bay area. I try to keep things partitioned across three separate blogs so that friends and family are not inundanted with posts about which they are uninterested:

blog.stokely.org - This blog: general posts about my life, math, technology, and more.

freebsd.stokely.org - Posts about my involvement with the open-source FreeBSD Operating System.

ava.stokely.org - Posts about Ava and our life with a new baby girl.

Returning to Industry

2005-09-21T23:38:00.000-07:00

I've decided to accept a position at Google and postpone the Ph.D. for now. We are moving to Mountain View next week. Our new apartment is 1 mile from downtown and 2 miles from the Googleplex. Pictures are available here.

Summer Events - Garden Party, Ball, etc.

2005-06-24T23:37:00.000-07:00

I've posted a few pictures of summer events from the last few weeks of Trinity Term at Wadham. These include the Finalists/Graduates Garden Party with the Warden, and the Burlesque Ball.

Camping in Lakes District

2005-05-30T23:36:00.000-07:00

Christian and I went camping in the Lakes District for an extended weekend. The pictures are posted here and here.

Back from Ottawa

2005-05-20T23:35:00.000-07:00

I have returned from BSDCan in Ottawa and my friend of 16 years, Christian, is visiting for 2 weeks. The pictures from Canada are here.

Pictures from Moscow

2005-05-03T23:34:00.000-07:00

I have returned from Moscow and posted the pictures here.

Back to Moscow

2005-04-23T23:33:00.000-07:00

I'm flying to Moscow on Tuesday to give FreeBSD talks at Open Source Forum Russia and at Moscow State University. I will return on 2 May.

Back from Hiking Trip to Snowdonia

2005-04-16T23:32:00.000-07:00

I've just returned from a great week in Snowdonia with other Wadham College MCR members on the annual reading/hiking retreat. Pictures are here.

Cambridge Exchange Dinner

2005-04-06T23:31:00.000-07:00

I just returned after a nice exchange dinner at Christ College Cambridge. We spent the night and had a nice dinner and visit with their MCR. The next morning I stayed later to have lunch with some friends. Pictures are posted here.

End of Coursework

2005-03-22T23:30:00.000-08:00

I submitted my final projects and thus the Hilary Term has ended for me, hurray! No more coursework, only a dissertation to write.

MCR Guest Dinner

2005-03-11T23:30:00.000-08:00

Nik and his girlfriend Lisa came up to Oxford for another MCR Guest Dinner. A few pictures have been posted here.

Seigo visits Oxford

2005-02-03T23:28:00.000-08:00

Seigo Tanimura arrived in Oxford today and we walked around town together. A few pictures (mostly from Christ Church) are here. This is the third continent we've met on after originally meeting at a FreeBSD meeting in Tokyo.

Burns Night

2005-01-21T23:27:00.000-08:00

We had haggis, whisky, poetry, and bagpipes in hall to celebrate Burns Night in Wadham. The pictures are here.

Moved to Summertown (Finished)

2005-01-19T23:26:00.000-08:00

We've moved from Iffley to Summertown, which is a little closer to the Oxford city centre and a much better place to live for various reasons.

Hilary Term

2005-01-17T23:25:00.000-08:00

Hilary term has begun in Oxford so I won't have time to post again until early Spring. My
academic page contains some information about lectures and activities this term.

Moving to Summertown

2005-01-16T23:24:00.000-08:00

DeDe and I are back from our 10 day trip to Morocco. The pictures have been posted here. We had a great time. Term starts tomorrow and we are in the middle of moving from Iffley to Summertown in Oxford.