Jon's Blog

Lego Set Database

Sat, 16 Feb 2013 08:51:41 -0600

As part of my Lego hobby, I've been looking for a way to create a Lego part/color histogram. In order to do this, I needed a data set containing inventories for the majority of sets that Lego has produced. A number of sites around the net have this information:

Lego.com - Only has part replacement lists for newer sets.
peeron.com
bricklink.com
rebrickable.com

None of these sites have this information freely available for bulk download. I contacted a few of the site admins, and Nathan from Rebrickable was happy to give me access to his database content and gave consent for me to release it as a download under a CC 3.0 by-sa license.

I was asked to mention that Rebrickable's part information is somewhat normalized, so this data may not line up exactly with what is found on other sites. More information about this process can be found in a blog post on Rebrickable.

The download that I've put together contains a spreadsheet for each of the 7,783 sets that I was given an inventory for. I've tried to make them as readable as possible so that you can just open up a spreadsheet and view the inventory for a set.

Here's the download. You'll need 7zip to extract it. Enjoy!

Lego Set Database

A simple vector library in C++

Wed, 31 Aug 2011 14:41:36 -0500

I should start off by saying that if you are planning to do any serious work which requires vector mathematics, you should probably just use one of the many open source libraries or packages:

Vectors are really a solved problem. They are relatively simple, and there are many libraries which can handle any operation that you can think of efficiently. However, there is certainly value in rolling your own, especially if you are interested in understanding what is going on behind the scenes.

Purpose

Vectors can be used for all sorts of things. They are particularly useful for maintaining state in physics and particle simulations. It is logical to break up the world into an x, y, and z component, and then manipulate these components separately. These vectors can then be used to represent location, velocity, acceleration, or other multidimensional values that might be interesting.

Important operations

What do we need to be able to do with vectors? Here's a short list of what I would like out of a basic vector library:

Vector addition, subtraction, multiplication, and division
Scalar addition, subtraction, multiplication, and division
Equivalence operator
Dot product
Cross product
Length

Depending on your needs, there are any number of operations that you may want to add, but these cover the basics.

Basic class

The first thing that we need to do is define the basic structure of the vector class which will house the vector components and the operations listed above. I've chosen to name this class vec3 (as in GLSL), and generally follow other naming conventions from GLSL wherever possible. I've chosen to do this in order to make switching between shader code and c++ code less jarring, even though it pains me to use a lowercase class name.

Anyhow, here's the class definition:

class vec3 {
    public:
        vec3();
        vec3(float, float, float);
        int operator==(vec3 rhs);
        vec3 operator+(vec3 rhs);
        vec3 operator-(vec3 rhs);
        vec3 operator*(vec3 rhs);
        vec3 operator/(vec3 rhs);
        vec3 operator+(float scalar);
        vec3 operator-(float scalar);
        vec3 operator*(float scalar);
        vec3 operator/(float scalar);
        vec3 cross(vec3 rhs);
        float dot(vec3 rhs);
        float length();

        float x;
        float y;
        float z;
};

Constructors

Starting from the top, we have the default constructor which will just set x, y, and z to zero. The second constructor will set the values to the provided floats. These are both trivial, so I won't insult you with the actual implementation code.

Equivalence Operator

From there, we move on to the implementation of the equivalence operation. This is quite simple. It is simple the result of each component equality combined using the and operation. It's a simple one liner:

bool vec3::operator==(vec3 rhs) {
    return(x == rhs.x && y == rhs.y && z == rhs.z);
}

Vector Arithmetic

Basic vector arithmetic is straight forward. The desired operation is simply carried out per component. So, this:

vec3 a = b + c;

Is the same as this:

vec3 a = vec3(b.x +c.x, b.y + c.y, b.z + c.z);

For simplicity, I'll just include the code for vector addition. The other operations are identical aside from using the appropriate operation:

vec3 vec3::operator+(vec3 rhs) {
    return vec3( x + rhs.x, 
                 y + rhs.y, 
                 z + rhs.z);
}

Scalar arithmetic

It can sometimes be useful to apply the same scalar operation to each component of a vector. This is what the overloaded operators will do when passed a single float.

Here's an example with division. The other operations again follow the same form.

vec3 vec3::operator/(float scalar) {
    return vec3(x / scalar, 
                      y / scalar,
                      z / scalar);
}

Dot Product

Now that we have the basics covered, it's time to dig in to the slightly more complex operations. The vector dot product can be defined as the product of the vector magnitudes multiplied by the cosine of the angle between the vectors. More simply, it is the sum of the product of the each vector component. This can be easily implemented as follows:

float vec3::dot(vec3 rhs) {
    return (x * rhs.x + 
            y * rhs.y + 
            z * rhs.z);
}

Note that the dot product always returns a single scalar value, so it is often also referred to as a scalar product.

Cross product

The magnitude of the cross product will be the area of the parallelogram which has the two vectors as sides. The direction of the cross product will be perpendicular to both vectors. The actual calculation of the cross product performed by the following routine:

vec3 vec3::cross(vec3 rhs) {
    return vec3( y * rhs.z - z * rhs.y,
                 z * rhs.x - x * rhs.z,
                 x * rhs.y - y * rhs.x);
}

Length

To calculate the length of the vector, we can use the distance formula. For our three component vector, that is sqrt(x^2 + y^2 + z^2). Here's the code:

float vec3::length() {
    return float(sqrt( x*x + y*y + z*z ));
}

Optimization

This is a post about how to create a basic vector library which can be easily read and understood. Therefore, even the most basic optimizations are left out. It would be quite beneficial to implement many of the operations using SIMD instruction which can handle processing all of the floats at once. This code probably isn't helpful in production, but sometimes it is great to just see how things work.

Taylor University Wikipedia Vandalism

Fri, 06 May 2011 10:40:53 -0500

As part of a recent project to better understand the culture of the university that I'm studying and working at, I decided to look at the instances of anonymous Wikipedia vandalism coming from school-owned IP addresses. While a lot of the vandalism was simply boring and offensive, there were a number of interesting themes which emerged.

Residence Life Focus

Taylor University is known among Christian colleges for its strong focus on residence life. This comes across clearly in student vandalism on Wikipedia. Well over half of all the intelligible vandalism had something to do with residence halls.

Many of the campus dorm stereotypes also emerged. Wengatz Hall often gets a bad rap for being overly preppy. This showed up quite directly, but was also evident in more subtle ways. One revert of vandalism stated that "most of campus doesn't care about a Wengatz open house".

Ironically, a fair amount of the vandalism also appeared to come from residents of Wengatz hall stating their superiority. This included one comment which regarded Wengatz as "the main male dormitory on campus." There were also numerous attempts to publicize and hype up a Wengatz event.

The women's halls were not left out, as Olson Hall was labelled the dorm with "the girls you date", Bergwall was noted for housing "the girls you study with", and it was suggested that "the girls you marry" come from English.

Life Together Covenant

The Life Together Covenant (LTC) is the community life agreement that Taylor students are required to adhere to. Students are given a copy of the LTC at the beginning of each year and are asked to sign it. There is a good deal of misunderstanding surrounding the LTC. It was created to provide shared standards to facilitate Christian community on campus, but many students view it simply as a list of rules. This came across in much of the vandalism about rules and discipline on the Wikipedia page.

One student mentioned that there are certain punishments for "those found to be in violation of the LTC". This clearly exemplifies the feeling that this document represents a set of boundaries which must not be crossed. The phrasing makes it sound more like a regulation coming from the EPA than a document designed to create a positive sense of community.

This viewpoint is further supported by students who come out of the discipline process feeling let down. One former student indicated what sounds like a complete failure of the intended process:

I didn't know I was doing anything wrong. No one confronted me about it and after three and a half years and nearly eighty thousand dollars in debt, I was removed from school. The Dean of Students told me that many students had been reporting to him about my unChristian behaviors for that past three and a half years, and he was trying to give me a opportunity change. I was really confused. Taylor seemed so strongly behind Biblical principles that I expected to be confronted before a beauracracy was notified. I guess I'll just have to work two jobs now.

Other Vandalism

The bell tower on the campus is one of the structural icons which symbolizes the university. It's twin towers represent the goal of integrating student learning with with the Christian faith. However, it is not universally admired. At least one individual felt that the tower was a negative part of the campus:

The term "bell tower" is a misnomer, as the tower does not contain any actual bells. The structure, which in reality is actually quite ugly, instead contains loudspeakers in a cage-like structure near the top of the towers which sound every 15 minutes. Many students wish that it was demolished and replaced by a more classic-looking bell tower that contains some actual bells.

Several students also voiced their support for the academic faculty via the Wikipedia page. Most notably, the entire page was replaced by the phrase, "Jeff Cramer Rocks!" more than once.

There was also some commentary on the change to the new university tagline, "beyond the mind", including one individual who remarked that, "it sucks if you ask me, but what can you do?"

Conclusion

While Wikipedia is not generally a good source of scholarly information, it can shed light on voices which are not otherwise heard. By examining vandalism on Wikipedia, several important aspects of the campus rose to the surface. Examining what students and others do to deface a Wikipedia article provides a unique way of looking at the institutional ethos.

Google Books Citation Bookmarklet

Fri, 10 Sep 2010 22:29:47 -0500

I've been working on a paper for class and I've found several great books on Google Books. Google Books is a great resource, unfortunately, I wasn't able to find a super simple way to cite books from there.

Google does provide a link to WorldCat, which in turn has built in citations, but that's a bit of a hassle. So, I whipped a up a quick Google Books Citation bookmarklet. Just drag it to your bookmarks bar and click it to open a new window with the citations for the book you are looking at. You will need popups allowed for books.google.com.

Basic Server Security

Fri, 06 Aug 2010 21:05:53 -0500

I stumbled across a presentation on HN today which detailed a vulnerability on several sites which allowed the contents of memcaches to be viewed.

It's a pretty interesting scenario. Sites use caches to speed up all kinds of things from profile information, tweets, and evidently even login credentials. These caches are often based on memcached because it works all in memory and never blocks for I/O.

Memcached has a very simple interface. It is simple ASCII over a TCP/IP connection. There is no authentication required to access or modify the cache. This means that if the cache is is open to outside network traffic, all data that gets cached is publicly accessible. There isn't generally a reason for the cache to be publicly available, but it is important to run some basic security checks when configuring a server so that problems like this one don't occur.

When I get done configuring a server or adding new packages, I like to check a few things.

Open Ports

In general, I want as few processes listening for connections to the outside world as possible. All software has bugs, and the more software that you have accessible to the outside world, the more likely it is that your server can be compromised by exploiting one of these bugs.

Port scanning

As I set things up, I try not to open more ports or even run more processes than I need, but it's important to make sure that I didn't miss anything. The way I like to do this is by port scanning my box.

Port scanning involves attempting to connect to my server on each port via some remote machine. If the connection is established, then the port is open to the outside world. If not, it either means that your server isn't listening on that port or a firewall is blocking it somewhere along the way. I suppose this could be done one port at a time using netcat or telnet, but nmap automates this task.

:~# nmap joncraton.org

Starting Nmap 4.62 ( http://nmap.org ) at 2010-08-06 23:41 UTC
Not shown: 1713 closed ports
Interesting ports on joncraton.org (127.0.0.1):
PORT      STATE SERVICE

80/tcp    open  http
443/tcp   open  https

Nmap done: 1 IP address (1 host up) scanned in 0.152 seconds

You can also get similar information by running netstat from the host machine:

:~# netstat -lnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:9001          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:9002          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN

This show that there are several processes listening on the loopback interfaces. These are my Django instances and memcached. It also shows that ports 80 and 443 are open to the public as they should be. So, the external ports look good.

SSH

For a single server, you can't easily get away from having SSH open on a port. In order to prevent random bots from continuously attempting to login to my server, I like to run SSH on a different port. This is easy to do in the SSH configuration. It is also highly recommended to disable root login from SSH and perhaps even disable password based logins altogether and instead use SSH keys for authentication.

Web Servers

Web servers generally have default configurations that are fairly secure, but it is always good to read up on what you can do to tighten things up a bit.

Conclusion

Poor operation procedures can lead to very large problems. Running through some basic checks when setting up a server can protect data and save a lot headache later on.