About Nothing

Setting Up a MongoDB Cluster

2014-10-10T16:16:00.005-05:00

I recently setup a MongoDB cluster on my workstation. While the Mongo documentation is very good, the proper setup is scattered between the Replication and Sharding documentation. I figured this might be helpful to others working with Mongo for the first time. Obviously this is not a valid production setup! For this, you'd need to place the replication sets, shard servers and config servers on separate machines.

The commands rs.help() and sh.help() come in handy along the way. Additional useful commands include rs.status(), rs.printReplicationInfo(), rs.printSlaveReplicationInfo(), sh.status(), and db.{$collection-name}.getShardDistribution().

Config Server Setup for 3 Nodes
1. Create a data directory for each of the three config server nodes.
mkdir c:\data\cluster\config\node1
mkdir c:\data\cluster\config\node2
mkdir c:\data\cluster\config\node3

2. Start the 3 config server nodes.
mongod --configsvr --dbpath c:\data\cluster\config\node1 --bind_ip ${machine-name} --port 27020
mongod --configsvr --dbpath c:\data\cluster\config\node2 --bind_ip ${machine-name} --port 27021
mongod --configsvr --dbpath c:\data\cluster\config\node3 --bind_ip ${machine-name} --port 27022

Start 2 MongoDB Shard Servers (mongos) Used By Client Apps
1. Start the 2 mongos instances.
mongos --configdb ${machine-name}:27020,${machine-name}:27021,${machine-name}:27022 --port 27017
mongos --configdb ${machine-name}:27020,${machine-name}:27021,${machine-name}:27022 --port 27018

Replication Set (RS) and Shard Setup for Shard 1
1. Create the data directory for each node.
mkdir c:\data\cluster\node1
mkdir c:\data\cluster\node2
mkdir c:\data\cluster\node3

2. Start the 3 nodes.
mongod --dbpath c:\data\cluster\node1 --bind_ip ${machine-name} --port 27000 --replSet "rs1"
mongod --dbpath c:\data\cluster\node2 --bind_ip ${machine-name} --port 27001 --replSet "rs1"
mongod --dbpath c:\data\cluster\node3 --bind_ip ${machine-name} --port 27002 --replSet "rs1"

3. Connect to one of the nodes.
mongo --host ${machine-name} --port 27000

4. Create the replication set.
rs.initiate()

5. Add the second node to the replication set.
rs.add("${machine-name}:27001")

6. Add the third node to the replication set.
rs.add("${machine-name}:27002")

7. Validate your replication setup. You should see JSON output with a single PRIMARY and two SECONDARY records.
rs1:PRIMARY> rs.status()
{
"set" : "rs1",
"date" : ISODate("2014-10-10T20:44:52Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "l7a973:27000",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 1989,
"optime" : Timestamp(1412972013, 1),
"optimeDate" : ISODate("2014-10-10T20:13:33Z"),
"electionTime" : Timestamp(1412971998, 2),
"electionDate" : ISODate("2014-10-10T20:13:18Z"),
"self" : true
},
{
"_id" : 1,
"name" : "l7a973:27001",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 1882,
"optime" : Timestamp(1412972013, 1),
"optimeDate" : ISODate("2014-10-10T20:13:33Z"),
"lastHeartbeat" : ISODate("2014-10-10T20:44:50Z"),
"lastHeartbeatRecv" : ISODate("2014-10-10T20:44:51Z"),
"pingMs" : 0,
"syncingTo" : "l7a973:27000"
},
{
"_id" : 2,
"name" : "l7a973:27002",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 1879,
"optime" : Timestamp(1412972013, 1),
"optimeDate" : ISODate("2014-10-10T20:13:33Z"),
"lastHeartbeat" : ISODate("2014-10-10T20:44:51Z"),
"lastHeartbeatRecv" : ISODate("2014-10-10T20:44:50Z"),
"pingMs" : 0,
"syncingTo" : "l7a973:27000"
}
],
"ok" : 1
}

8. Connect to one of the shard servers (mongos).
mongo --host ${machine-name} --port 27017

9. Add the replication set to the shard.
sh.addShard("rs1/${machine-name}:27000")

10. Enable sharding for the database (I'm using "ngi" for my database name).
sh.enableSharding("${database-name}")

Replication Set (RS) and Shard Setup for Shard 2
1. Create the data directory for each node.
mkdir c:\data\cluster\node4
mkdir c:\data\cluster\node5
mkdir c:\data\cluster\node6

2. Start the 3 nodes.
mongod --dbpath c:\data\cluster\node4 --bind_ip ${machine-name} --port 27010 --replSet "rs2"
mongod --dbpath c:\data\cluster\node5 --bind_ip ${machine-name} --port 27011 --replSet "rs2"
mongod --dbpath c:\data\cluster\node6 --bind_ip ${machine-name} --port 27012 --replSet "rs2"

3. Connect to one of the nodes.
mongo --host ${machine-name} --port 27010

4. Create the replication set.
rs.initiate()

5. Add the second node to the replication set.
rs.add("${machine-name}:27011")

6. Add the third node to the replication set.
rs.add("${machine-name}:27012")

7. Validate your replication setup. You should see JSON output with a single PRIMARY and two SECONDARY records.
rs2:PRIMARY> rs.status()
{
"set" : "rs2",
"date" : ISODate("2014-10-10T20:46:15Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "l7a973:27010",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 1642,
"optime" : Timestamp(1412973972, 1),
"optimeDate" : ISODate("2014-10-10T20:46:12Z"),
"electionTime" : Timestamp(1412972423, 2),
"electionDate" : ISODate("2014-10-10T20:20:23Z"),
"self" : true
},
{
"_id" : 1,
"name" : "l7a973:27011",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 7,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2014-10-10T20:46:14Z"),
"lastHeartbeatRecv" : ISODate("2014-10-10T20:46:13Z"),
"pingMs" : 0
},
{
"_id" : 2,
"name" : "l7a973:27012",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 3,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2014-10-10T20:46:14Z"),
"lastHeartbeatRecv" : ISODate("2014-10-10T20:46:14Z"),
"pingMs" : 0
}
],
"ok" : 1
}

8. Connect to one of the shard servers (mongos)
mongo --host ${machine-name} --port 27017

9. Add the replication set to the shard.
mongos>sh.addShard("rs2/${machine-name}:27010")

10. Validate your setup. You should see JSON output with a single PRIMARY and two SECONDARY records.
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"version" : 4,
"minCompatibleVersion" : 4,
"currentVersion" : 5,
"clusterId" : ObjectId("54383c1cc3b7bde946a478cf")
}
shards:
{ "_id" : "rs1", "host" : "rs1/l7a973:27000,l7a973:27001,l7a973:27002" }
{ "_id" : "rs2", "host" : "rs2/l7a973:27010,l7a973:27011,l7a973:27012" }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "ngi", "partitioned" : true, "primary" : "rs1" }

Add Some Test Data and Validate the Distribution
1. Connect to one of the shard servers (mongos)
mongo --host ${machine-name} --port 27017

2. Switch to your database.
mongos>use ${database-name}

3. Add 100,000 test data records.
mongos> for(var i=1; i<=100000; i++) { db.mycollection.insert({x:i}) }

4. Shard the collection based on the id. Remember that "ngi" is my database name.
mongos> sh.shardCollection("ngi.mycollection", {_id: 1})
{ "collectionsharded" : "ngi.mycollection", "ok" : 1 }

5. Validate that the collection is sharded
mongos> db.mycollection.getShardDistribution()

Shard rs1 at rs1/l7a973:27000,l7a973:27001,l7a973:27002
data : 602KiB docs : 12860 chunks : 1
estimated data per chunk : 602KiB
estimated docs per chunk : 12860

Shard rs2 at rs2/l7a973:27010,l7a973:27011,l7a973:27012
data : 3.98MiB docs : 87140 chunks : 2
estimated data per chunk : 1.99MiB
estimated docs per chunk : 43570

Totals
data : 4.57MiB docs : 100000 chunks : 3
Shard rs1 contains 12.86% data, 12.86% docs in cluster, avg obj size on shard : 48B
Shard rs2 contains 87.13% data, 87.14% docs in cluster, avg obj size on shard : 48B

The Future of Analytics and Business Intelligence

2012-10-30T16:28:00.001-05:00

Been excitedly waiting to see what comes from Numenta since ~2006. Can't wait to see what's next. If I could go work for them tomorrow I would. Can't wait to see what Hawkins is talking about 20 years from now.

Clojure!

2012-10-30T16:14:00.001-05:00

Spent some time looking at Clojure today. Definitely mind-bending given I've never worked with a Lisp dialect. Downloaded and walked trough the examples on the site. I will definitely be taking a deeper dive if nothing more than to help diversify my perspective. I'm interested in learning more including ClojureScript.

Habitat for Humanity

2012-07-18T22:45:00.003-05:00

I spent the day yesterday with 15 co-workers helping to build a Habitat for Humanity house. It was fun, felt good and was less stressful than work. We spent the day putting siding on the house. I enjoyed challenging my fear of heights standing on scaffolding at the peak of the roof while using a nail gun.

HTML Parsing With Groovy and TagSoup

2009-12-29T08:43:00.003-06:00

I'm working on an app where I need to parse some HTML. This is the first time I've had to do screen-scraping with Groovy. After a bit of trial and error I think I'm getting the hang of it. The HTML I'm working with isn't well-formed, so the default Groovy XmlSlurper and XmlParser puke. After some digging I found TagSoup. It "parses HTML as it is found in the wild: poor, nasty and brutish, though quite often far from short".

It made my parsing much easier. Thanks John Cowan!

Groovy XmlSlurper and HTTP 503 Response Code

2009-12-29T05:38:00.008-06:00

I struggled a bit when trying to parse some XHTML with Groovy's XmlSlurper (and XmlParser). I was receiving the following:

Caught: java.io.IOException: Server returned HTTP response code: 503 for URL: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd

It turns out that the guys from W3C got sick of dealing with the excessive traffic for their DTDs. So now they return a Service Unavailable (HTTP 503) if they detect parser requests.

To solve the problem I had to set the loading of external DTDs to false. Here's the code.

def slurper = new XmlSlurper()
slurper.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false)
def results = slurper.parseText(htmlResponse)

Googling for the answer wasn't extremely helpful. This blog post helped (I think it's in Japanese). This post also helped. Thanks guys!

I decided to re-post the solution since it took me awhile googling for the answer.

The Science of Avatar

2009-12-27T06:30:00.004-06:00

An interesting read on the science of Avatar. I still haven't seen the movie; just too much going on with the holidays.

(Near) Real-Time Analytics

2009-12-08T06:32:00.004-06:00

At my new gig, I've been asking whether the team has considered the possibility of using map/reduce or a similar grid-based solution to conduct our analytics in (near) real-time. Interestingly enough, I ran across Nati Shalom's post on real-time analytics yesterday. This should help give me some ammunition to convince everyone that we need to move in this direction for the solution we're building. Thanks Nati!

A Feast for Crows

2009-12-08T06:16:00.003-06:00

I just finished re-reading George R. R. Martin's A Feast for Crows. I enjoyed it more than the first time I read it. My favorite still continues to be A Storm of Swords. Now if he'd just publish A Dance with Dragons!

The Definitive Guide to Grails

2009-05-13T06:45:00.003-05:00

I finished up The Definitive Guide to Grails close to a month ago, but I forgot to blog about it (I'm using my blog to help keep track of which books I've read). It was an excellent read. I'm sold on Groovy and Grails, particularly for Java shops. Given the recent SpringSource purchase of G2One, I expect Groovy and Grails to gain much wider adoption in the enterprise.

The Productive Programmer

2009-03-22T05:58:00.003-05:00

I just finished up The Productive Programmer by Neal Ford. It was so good I decided to buy my own copy. It definitely made me realize how much more efficient I can make myself. There are a ton of tips for both Mac and Windows. One of the major themes was automate everything you can. Thanks Neal!

No Fluff Just Stuff

2009-03-13T15:15:00.007-05:00

Today is the first day of the Twin Cities Software Symposium. The first talk I attended was REST: Information Driven Architectures for the 21st Century by Brian Sletten. Very informative. Definitely not an introductory REST talk. I'm curious to hear Brian's Semantic SOA talk later today. A couple things that stuck included Jon Postel's "Be liberal in what you accept, and conservative in what you send."

A couple of things Brian talked about that sound like they're worth investigating include Sinatra, a DSL for building web applications in Ruby and retrievr, which lets you find Flickr photos by creating a sketch of what you're after.

Brian's new gig sounds pretty cool: League of Legends.

Einstein

2009-03-01T06:05:00.002-06:00

I recently finished reading Einstein: His Life and Universe, a biography by Walter Isaacson. I enjoyed it very much. It gave excellent insight into the man. The most interesting thing to me was that even though he was brilliant, he struggled with things in his everyday life just like everyone else. His family life wasn't perfect and neither were many other aspects of his life. I love the fact he was a non-conformist not only in science, but in political affairs as well.

Crystal Clear

2009-02-14T17:04:00.008-06:00

I just finished up Crystal Clear by Alistair Cockburn. A very good book, but it was a stretch to get it to 300 pages. The first chapter threw me for a loop the way it was structured and the last chapter, a case study, was a dud. While the team size for Crystal should be 8 or less, a case study with 1.5 developers doesn't sound like a very good case study. However, the chapters in between were excellent. Cockburn admittedly structured each chapter differently attempting to cater to different readers. It gave me some insights into how a successful team should interact and was very complementary to the other Agile documentation that I've seen. I definitely liked his guidance on Walking Skeleton and Incremental Re-architecture. It helped me reinforce the concept of an Architectural Slice that I've been conveying to the folks on the large Java project that I'm currently working on.

Kudos to Andy Miller for recommending this book!

Shipped It!

2009-02-14T09:13:00.003-06:00

Jared Richardson of Ship It! fame spoke at the TCJUG on Monday. His topic was your career. He started out a bit slow, but the pace picked up as his presentation progressed. I think most folks got quite a bit out of it, but it reminded me of several presentations I had seen before. Particularly there was some overlap with a presentation I went to several years ago by Dave Thomas at NFJS Denver. Dave's theme was about investing in your career. That was the first time I heard Dave's infamous "Herding Racehorses and Racing Sheep".

I did learn some new stuff at Jared's presentation. In particular I learned about qik, which looks pretty cool. It allows you to share a live video feed from your phone; Jared had someone in the audience do the live feed to qik using Jared's iPhone. I liked Jared's acronym for public speaking (L)ock eyes, (I)ntonation, (P)ause as well. This will definitely come in handy for me in the future.

The part that surprised me the most was how few people in the audience new about Blogs and Feed Readers. Another shocker was how few people had heard of The Pragmatic Programmer. Maybe people were just too lazy to raise their hands. If not, c'mon TCJUG attendees!

Thanks for flying all the way to Minnesota to enlighten us Jared!

Groovy Encapsulation - Say What?

2009-01-31T06:19:00.007-06:00

I'm reading Groovy Recipes by Scott Davis and find this troubling:

class Book3{
private String title
private String getTitle(){}
private void setTitle(title){}
}
def b3 = new Book3()
b3.@title = "Groovy Recipes"
println b3.@title
===> Groovy Recipes

In Groovy, private attributes can be modified, even if you use private setters. That's not cool. I'm hoping there's some way to enforce encapsulation, but it's not looking good right now.

Groovy!

2009-01-25T06:52:00.001-06:00

I just finished up Getting Started with Grails the free book from InfoQ by Jason Rudolph. An excellent book. Thanks Jason! I thought it was so good I decided to pay for it (even though it's free) to help support the author and InfoQ. There's a few discrepancies because the book is almost two years old and quite a bit has changed in Grails since then, but I had very few problems working through the examples with the latest version of Grails.

I'm definitely on the Groovy and Grails learning train. Both still seem very promising to me (being a Java guy) and I'm going to continue investigating both. While I set out to learn more about Ruby and Rails this year, the winds have shifted and I'm now focused on Groovy and Grails. I purchased the PDF version of Scott Davis' book Groovy Recipes and The Definitive Guide to Grails.

Hockey Day Minnesota

2009-01-17T06:38:00.001-06:00

Today is Hockey Day Minnesota. I'm going to try to honor that by getting my daughters out on the neighbor's pond followed up by taking them to the University of Minnesota Women's game against Bemidji State. It'll then be on to termite practice where the kids will get to play pond hockey instead of practicing. We'll wind it down with watching some of the Gopher and Wild games. It should be a ton of fun!

Code Freeze

2009-01-17T06:14:00.001-06:00

I attended the Code Freeze conference at the University of Minnesota on Thursday. It was an excellent local event, especially considering it was an all-day event for only $90. This was the first year I've attended Code Freeze; this year's theme was Maximizing Developer Value. Neal Ford kicked things off and as usual he knocked it out of the park. His topic was On the Lamb from the Furniture Police. It covered the fact that as programmers we're hired to concentrate for long periods of time, yet corporate environments provide the exact opposite affect.

Other speakers included Luke Francl, Nate Schutta, Susan Standiford, Andy Miller and Tomo Lennox. I was very impressed with Nate's presentation, it seemed to directly pick-up where Neil left off. I was particularly interested in Nate's comments about the working of the human brain, as it is an area of interest for me.

I was also intrigued by Andy's presentation entitled "Why I don’t estimate with "points" (and how you too can be delivered from the tedium of repetitive estimation)". Andy and I are currently at the same client working together on a large re-engineering project. I haven't work with Andy for long, but I was very impressed with his presentation and was impressed with his pragmatic approach to estimating. It definitely opened my eyes to new ideas.

Hackers and Painters

2009-01-10T07:38:00.001-06:00

One of my 2009 resolutions is to read more. I just finished up Hackers and Painters. I had made it half-way through a couple of years ago and decided to start over and I made it all the way through. A quick and insightful read. Paul Graham is fairly opinionated, which makes for a good read. I've never seen Lisp, but I am very curious if it as good as he claims. Based on my experiences, I definitely agree with his thoughts on development productivity. It makes me that much more interested in learning Ruby and Rails. The last time I worked seriously with a dynamic language was with Perl in college when we were building the Heil X6 SMT's kernel, simulator and assembler. The Heil X6 was the computer we (a team of 6 graduate and undergraduates) built in our ECE 554 Digital Engineering Lab using FPGAs that we had to hand-wire. What a trip!

Groovy and Grails

2009-01-10T07:13:00.002-06:00

At the DevJam Jam Session on Wednesday, Mike Hugo gave a presentation on Groovy and Grails. I had heard Scott Davis speak about Groovy and Grails over a year ago and thought it was pretty cool, but back then it didn't feel real. Since then Grails has produced a 1.0 release; put simply I was extremely impressed. Mike is a very good presenter, his slides had very few words and some great pictures. He kept it lively with some humorous slides plus some succinct demos. I've been looking at Ruby on Rails off and on over the last couple of years. One of my New Year's resolutions for 2009 was to really dig into Rails and write a decent sized application. However, after Mike's presentation I began to think that a better investment of my time may be in Grails given that it runs on the JVM and plays very nicely with the Java language. I downloaded Grails yesterday and played around with it a bit. Very nice. Just like with Rails, it's amazingly easy to get a site up and running quickly. Grails is still a 1.0 release, so I have some concerns about it's maturity. However, it runs on the JVM and uses Spring, Hibernate and SiteMesh under the covers. Obviously all of them are very mature.

After the presentation by Mike, we entered the Fish Bowl round table discussion, which I believe is the best part of the Jam Sessions. I'm amazed by the brain power in the room at these sessions and I definitely like the Fight Club mentality. The conversation touched on a number of topics including the advantages / disadvantages of statically vs dynamically typed languages. The consensus seemed to be we're definitely seeing a movement toward dynamically typed languages. Someone even dropped the Lisp quote along the lines of Hackers and Painters ("...we could do that in Lisp back in 1968..."), which was interesting given that I just finished the book. One person, who seemed to know a ton about language theory (he even dropped F#), mentioned that he thinks that the gulf between the two is getting bigger and bigger: static languages are becoming more strongly typed and dynamic languages are/will become more dynamic. It makes sense to me given where I see the Java language headed and makes my desire to learn Ruby and Groovy that much stronger.

Building Scalable Web Sites

2008-01-05T09:31:00.000-06:00

I just finished reading the book Building Scalable Web Sites by Cal Henderson of Flickr fame. It was a very interesting read and covered quite a few things I wasn't expecting. Most of the code was PHP, which I've never worked with, so that in itself was cool to see for the first time. He covered a ton of topics, including walking through how they scaled Flickr. I got the most out of chapter 9: Scaling Web Applications. He also gives a decent overview of Web Services in the last chapter, which I wasn't expecting. The book was easy to read and some of the difficult topics he explained clearly and concisely.

Overall it was an excellent book and I would recommend it to anyone building web sites regardless of whether or not they're using PHP.

Spring 2.5 Upgrade

2007-12-21T08:44:00.000-06:00

I upgraded out platform infrastructure to Spring 2.5 this morning. It was pretty straight-forward, although I had one painful issue. For any of our tests that use Hibernate, I was getting the following error:

Caused by: org.hibernate.HibernateException: saveOrUpdate is not valid without active transaction
at org.hibernate.context.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:297)
at $Proxy39.saveOrUpdate(Unknown Source)

After a bit of digging, I realized we had the following Hibernate property set:

<prop key="hibernate.current_session_context_class">org.hibernate.context.ThreadLocalSessionContext</prop>

Basically, this was preventing Spring from managing the transaction context correctly. Once I removed it, all of our tests passed. I wonder if this is the reason why we've had some problems with NOT_SUPPORTED and SUPPORTS previously. This will require some additional testing...

Spore

2007-12-20T10:57:00.000-06:00

I stumbled across a new PC game today entitled Spore. It's not GA yet, but it looks extremely interesting. It looks like Civilization meets The Sims. Spore was created by Will Wright, the creator of The Sims. Here's the video I watched. You start as a single-celled organism and you evolve your species all the way up to an advanced spaced-based civilization.

Desktop Matters (TSE Day 4)

2007-12-15T10:08:00.000-06:00

In the second session of the morning, I decided to attend the talk entitled Desktop Matters by Jim Moore. I didn't learn much in this presentation. He walked through the history of computers from a UI perpsective: mainframe to client server to web. He then went into UI technologies that are available today to build rich applications: Flex, Apollo, Silverlight, GWT, WebStart, etc. He had a couple of bullets points for each, but he didn't get into much detail in terms of what each does. I would've liked to have seen more in the way of the positives vs. negatives of each and some recommendations.