Marginally Interesting by Mikio L. Braun

What it means to do a Ph.D. - psychologically

2012-01-24T10:23:00+01:00

Most people who decide to do a Ph.D. are well aware that it will mean a lot of work. You have to learn a lot of new stuff, possibly also outside of the topics you have studied so far. Taking machine learning as an example, you probably need to learn much more math than you’ve already been exposed to, including a mix of linear algebra, optimization theory, probability theory, statistics, and so on. But you also need to learn something about the area where you apply your methods, for example, bioinformatics, linguistics, and so on.

But at the same time, doing a Ph.D. also poses some psychological challenges and from my experience I can say that many students are quite surprised by the level of problems they face. In contrast to a Bachelor or a Master, which requires you to learn some topic and be able to apply what you’ve learned to new similar problems, doing a Ph.D. means doing something which hasn’t been done before. You need to solve a problem which hasn’t been solved before.

Now this may sound not that surprising because that’s what research is all about: exploring questions, solving problems, advancing the state-of-the-art. But you only realize what this really means when you’re one or two years into your graduate studies, you have learned quite a lot and come to understand the nature of the problem, and you realize that you have no idea how to solve the problem.

There is of course a lot you can do to hedge the risk of failing. For example, you can start with simpler subproblems and work yourself up towards the full problem. You can work on a number of smaller problems such that you build up a collection of work done. But at some point you will invariable find yourself in a situation when you have to admit that you really cannot know whether you’ll be able to solve the problem, or whether any of your usual strategies will help.

And this doesn’t even include the social aspects of doing a Ph.D., of getting published, getting cited, building up some form of reputation in the community.

I found myself in exactly this situation towards the end of my studies. I had to switch topics inbetween because the original idea didn’t quite turn out as expected. I wrote my thesis about convergence of eigenvalues and eigenvectors of the kernel matrix. But till the very end, a central proof was missing. I had run extensive numerical simulations so I was quite sure about what I wanted to prove, but only in the very end I managed to put the proof together. So here I was, with a few month left before my position ended, trying to solve that problem every day but not knowing whether I would be able to do that in the end or not. To illustrate my state of mind, when I moved to a different town, I couldn’t rent the truck of the size I had reserved but only one which was about a meter shorter. All my friends told me “Mikio, forget it, we’ll never get all your stuff in there”, but I was just like “ah, impossible, well, yes… .” In the end, everything except for one cupboard went in which was ok, and showed that we both had been wrong.

Actually, I have come to believe that this experience is part of what it means to do a Ph.D.. Eventually, you will succeed in one way or another, and you will have learned a very valuable lesson. You will see how the problem slowly sinks into your mind until your understanding of the problem will lead you to a solution, or uncover that it is not possible, but you will also have understood why.

In the end, doing a Ph.D. is exactly about this: Learning to do what no one has done before and be confident even when there is only a limited amount of time and you have no idea whether you will be able to solve the problem. And that is an important part of what science is about.

Click here for the full article

Fast Cross Validation

2011-12-20T14:58:00+01:00

Fast Cross Validation Via Sequential Analysis - Talk

These are the slides to our talk (joint work with Tammo Krüger and Danny Panknin) at the BigLearning workshop at NIPS 2011. You can also have a look at the paper and the appendix.

So what is it about? In a nutshell, we try to speed up cross-validation by starting with subsamples of the data and identifying quickly parameter configurations which are clearly suboptimal. Learning on subsets is of course much faster so ideally you’ll save a lot of time because you will only have a handfull of parameter candidates left on the full data set.

The method is based on the sequential analysis framework which deals with the problem of statistical hypothesis testing when the sample size isn’t fixed.

The main problem one faces is that the performance of parameter configurations changes significantly as the sample size increases. For a fixed parameter configuration (say a kernel width and a regularization parameter for an SVM), it is clear that the error converges, and usually becomes smaller as the number of samples increases. However, if one compares two configurations, one can often observe that one configuration is better for small sample sizes, while the other becomes better later on. This phenomenon is linked to the complexity of the model associated with a parameter choice. General speaking, more complex models require more data to fit correctly and will overfit on too few data points.

Our method accounts for this effect by adjusting the statistical tests to maximize the number of failures before a configuration is removed from the set of active configurations. Nevertheless, fast cross-validation is faster by a factor of 50-100 on our benchmark data sets.

Click here for the full article

Scala discussion heating up?

2011-11-30T11:18:00+01:00

Apparently, the discussions about “Scala being too complex” are heating up, mostly due to a leaked email from one of Yammer’s programmers to the Scala people where he discusses some of his experiences he’s had with using Scala in a production environment, and the other being a post on HN comparing Scala to Perl in the sense that both languages have too much flexibility in solving a specific task leading to a mix of different programming paradigms and styles which will make you code harder to read and maintain.

Now we’ve been using Scala as our main programming language for the last two and half years for TWIMPACT, so I know what people are talking about. And the truth is, it is all true, sadly. On the one hand, Scala is a pretty awesome programming language which is very nicely designed. I’ve said this before, but normally you will eventually come across some feature of a programming language which is not designed well and you have to code your way around it, but I’ve yet to come about something like it in Scala.

On the other hand, it is also true that some of the libraries are not as fast as they should be. Although I like the idea of immutable collections a lot, every time I need performance, I’d rather put in a Java collection. Also, it’s true that the collection library is pretty complex. It all kind of makes sense to get a clean design of the classes, but it’s pretty complicated with all those classes like Seq, SeqLike, Traversable, TraversableOnce, etc. However, you’ll probably only need to know all the details if you want to write your own collections which integrate seamlessly with the existing collection classes.

It’s also true that upgrading to a new version is hard. For some reason, many libraries seem to be quite deeply interlocked with the Scala version. While our own code never had to be changed if Scala went to a new version, this wasn’t true for most libraries, unfortunately, meaning that you have to wait till all the libraries have been upgraded to the new version before you can do the update yourself. And frankly, I don’t see why this is necessary.

We’ve never bothered with sbt, but directly went for maven due to it’s better integration in most IDE’s. We’re using IntelliJ IDEA whose Scala plugin has come a long way and gives pretty good support. There is also a lot to be improved in the basic tools like the compiler or the shell in terms of startup time. Scala seems to preload several megabytes of jar files on startup, probably in an attempt at optimization, but in the end, it only means that starting Scala takes anywhere between 5 - 10 seconds which is really a lot if you’re working on the shell (and every other language starts up almost immediately) The guys behind JRuby have invested a lot of time to cut down on the startup time, and that was time well spent.

People are also often attacking Scala for it’s complexity. While it’s certainly true that it’s easier to hire some Java expert than someone who knows Scala, IMHO Scala is a big improvement in many ways over Java, which feels overly verbose once you’ve learned Scala. As with every language, there are more basic concepts and more advanced concepts and usually, you don’t have to master them all from the start. Also, people often argue as if the complexity about learning a programming language is all in the programming language, but you also have to consider the standard libraries and tools. For example, while the Java programming language is relatively simple in terms of concepts, the standard tools and frameworks are pretty intimidating to learn (all that XML, Maven, Spring, etc.)

Then people are also complaining about the community, which is supposedly not helpful enough, or too fragmented, or only consists of crazy people who are just thinking about how to implement everything in terms of category theory. I don’t think that is true. Scala is still young, and the community can still grow. We’ve uncovered a number of bugs (mostly Leo who has a knack for finding bugs in libraries) and people were mostly as responsive as you’d expect them to be. One of the strengths of Scala is also that it is quite painless to reuse existing Java projects (as any other programming language for the JVM). I never found it that repulsive as some seem to use a Java library from Scala. The integration is quite painless, and if you really have to, you can add a bit of syntactic sugar on your side for the stuff you need most.

Finally, I really don’t get the argument of people who are saying “Scala is too complex, I switched to Python (or some other scripting language)”. To me, these are completely different sets of programming languages. While it’s true that there are some applications like writing medium sized web sites which you can nowadays do in either a scripting language or a compiled language, there are many applications where Python (or any other scripting language) just can’t compete. In scripting languages, it’s hard to add primitive data types which are really fast unless someone else already took care of implementing the most computing-intensive routines in C.

So in summary, Scala is both awesome and awful, just like almost every piece of sufficiently advanced technology. You can work with Scala, and it’s a lot of fun, or you can reject it for a number of reasons, just acknowledge the complexity and don’t give in to hypes and marketing.

Updates: Coda Hale’s comment on the leaked email, Yammer’s official statement on Scala, Typesafe’s post on their committment to the industry.

Click here for the full article

Analyzing Social Media Data

2011-11-01T22:20:00+01:00

Analyzing social media has become quite popular. People have been predicting box office openings based on Twitter chatter, studied information diffusion patterns, information flows between classes of users, how real-world events like earthquakes are reflected in Twitter.

This is all pretty exciting and interesting, but there are also a few things where there is still room for improvement.

There is very little stuff on real-time analysis. Many papers boast with the hundreds of millions of tweets (and the access to Twitter’s firehose necessary to get that amount of data) which have formed the basis for the paper. However, many papers later introduce some more or less arbitrary ways of truncating the data, for example by taking a number of “most active users”. This is both true for Jure Leskovec’s paper as well as the Yahoo research’s paper.

However, I think that getting to real-time is extremely important, because you cannot just wait for days or longer to get your analysis. By that time, more data will have been streaming in, and when are you going to analyze that data?

Another problem with many of the analyses is that they focus on the positive cases only. Meaning that they develop some method to detect bursts or trends and then use some famous real-world example (like Japan winning the women’s soccer championship) to show that the method is triggered by the data. However, few publications go so far as to validate their method on negative examples as well, showing that the method not only detect trends well, but also does so robustly with few false positives.

A classical example is the highly cited 2003 paper by Jon Kleinberg ”Bursty and Hierarchical Structure from Streams” which explains how to detect areas of higher than usual activity, for example, from email streams. But then, the paper shows how the detected structure coincides with real deadlines for two examples without discussing negative examples in depth.

Many methods also seem to believe that an analysis which is based on hundreds of millions of data points is automatically true in general. While this is certainly true for simple statistics which you can estimate well, there are other methods which can overfit. And for those, as many other disciplines like bioinformatics have had to learn the hard way, as you get more data, the probability that you find some evidence for your hypothesis increases drastically.

To get reliable results, you need to follow the same rules as when validating the performance of a machine learning algorithm: Test on data which is disjoint from training data. If your method detects trends, check it on data which you believe has no structure. If you aggregate topics, check it on days when nothing special was happening. If you analyze the structure of the data, check on an independent sample (ideally from a period of time which is a bit removed from the original sample).

That way you might have less data available, but your results will improve a lot in terms of reliability.

Click here for the full article

One does not simply scale into real-time

2011-10-10T22:15:00+02:00

Real-time seems to be the next big thing in big data. Map-Reduced has shown how to perform big analyses on huge data sets in parallel, and the next challenge seems to be to find a similar kind of approach to real-time.

When you look around the web, there are two major approaches out there which try to building something which can scale to deal with Twitter-firehose-scale amounts of data. One is starting with a MapReduce framework like Hadoop and somehow finagle real-time or at least streaming capabilities on it. The other approach starts with some event-driven “streaming” computing architecture and makes it scale on cluster.

These are interesting and very cool projects, however from our own experience with retweet analysis at TWIMPACT, I get the feeling that both approaches fall short of providing a definitive answer.

In short: One does not simply scale into real-time.

Real-Time Stream Analysis

So what is real-time stream analysis? (Apart from the fact that I seem to be unable to decide whether to write it with a hyphen or as one word like Donaudampfschifffahrtskapitänsmütze).

Basically, the idea is that you have some sort of event stream like Twitter messages, or Apache log data, or URL expansion requests at bit.ly, which comes in at a high volume of several hundreds or even thousands of events per second. Let’s just focus on Twitter stream analysis for now.

Typical applications are to compute some statistics from the stream which summarizes it in a nice fashion. For example

what is the most frequently retweeted tweet?
what is the most frequently mentioned URL?
what are the most influental users (in terms of mentions/retweets)

These are pretty basic counting tasks. Very closely linked are questions like what the score of an arbitrary tweet or URL is, or what the first few hundred most influental users are, and so on. You can also compute more complex scores based on more of these numbers. For example, our own TWIMPACT score is based on all the counts of a user’s retweets, and something like the Klout score is also based on a number of statistics (I’m only assuming here, of course).

Since the data arrives in real-time, you typically also want to get the results in real-time (instead of a daily report which is already hours old when you get it). Ideally, you get updated scores with each event, also I’d say everything is ok if a query always takes less than a second.

Finally, you’d probably also want to be able to look at historical data to see how a user or retweet has performed in the past and to see whether its activity is going up or going down.

Just to give you an idea of the amount of data involved: Each tweet corresponds to about 1k of data (including all metadata). If we assume that we have about 1000 tweets per second (actually it’s probably more), then we get about 86.4 million tweets per day, or about 82.4GB of new data per day (about 30TB per year).

Now let’s discuss how you would approach this problem in a database centric fashion and using a stream processing framework.

Databases Approach

With “database approach” I try to cover a whole range including traditional relational databases, NoSQL databases and MapReduce frameworks. The common denominator is that you basically pipe your data into the database and use the built-in queries of the database to compute your statistics. Depending on the type of NoSQL database you are using you’ll probably have to do the analysis online to precompute all your statistics because you the database doesn’t have sufficient query capabilities.

As I see it, there are two main problems with this approach: First of all, the size of your database grows at a non-trivial rate. Unless you’re Google and you’ve planned for exponential growths of your data centers anyway, you will eventually run out of space and performance. Even if you assume that the reponse time will increase only logarithmically in your data size, your cluster will eventually become quite slow to deal with real-time data.

This will directly affect the time it will take for your queries to complete. However, as queries also put quite some load on your disks, this problem will only get worse over time.

At the same time, most of the data will likely be irrelevant for your current analysis. Tweets stop being retweeted, URLs fall out of fashion. In other word, there is a huge amount of historical baggage clogging up your servers. It is true that you still need this data in order to compute historical statistics, but note that the data doesn’t change anymore, so that it would make much more sense to analyse the data once and only put the results in some read-only storage.

You could of course try to keep the database size constant by periodically discarding old data. I think this is the right approach and I’ll discuss it in more detail below. Note, however that many databases cannot really deal well with huge amount of deletions, as they need some form of more or less disruptive garbage collection (vacuum, compaction, etc.) to actually free up the space.

Finally, one should also not forget that MapReduce doesn’t work with all kinds of problems, but only with problems which are already inherently easy to parallelize. Actually, if you look into research in parallel algorithms, you will see that almost no problems scale linearly in the number of available processors (and you will also see that many of the efficient algorithms work with shared mutable state! Ugh!)

Stream Processing

So if storing all your data on slow disks to process it later is the wrong approach, you should probably try to process the data as it comes in through some form of pipeline. This is more or less the idea behind stream processing. Two example frameworks are Storm, originally developed by BackType (recently acquired by Twitter), and S4 developed by Yahoo.

If you look closer, these are basically quite sophisticated frameworks to scale an actor based approach to concurrency. If you’re not familiar with it, it’s the idea to structure some computation in terms of independent small pieces of code which do not share state and communicate with one another through messages.

Frameworks like the ones above let you define a computation in terms of a number of processing nodes which may also run on different servers, with the ability to add more parallel workers of a certain kind on the fly to scale up processing resources where necessary.

This approach is essentially orthogonal to the database approach. In fact, stream processing frameworks usually don’t even deal with persistence, they only focus on the computation. Therefore, there is also nothing specific to real-time stream processing in these frameworks. In essence, they deal with the question of how to split up a computation into small independent parts and how to scale such a network on a cluster.

To me, the basic problem with this approach (and with actor based concurrency) is that it doesn’t deal well with peak volumes which surpass the computation bandwith. In fact, it’s even not so simple to say given such a network what the maxium throughput is. Conceptually, it is the throughput of the slowest component, but you also have to take the message routing topology into account.

Now, once more messages need to be processed than possible, somewhere in the system messages queues are starting to fill up. This is a general problem with such systems, not specific to actor based concurrency. The book Release It! contains some very entertaining and also frightening real-world war stories of what can go wrong with systems where you plug together components with quite different capacities.

Another problem is actor based concurrency tends to break a simple function which may fit on a single screen into dozens of classes, but that is another problem.

In any case, the question is how you can guarantee that your system will run stably even for high peak volumes, apart from just adding more nodes? The easiest thing would be to randomly drop events (resulting in a more or less consistent subsample which at least has the same distribution as the original data stream), but is there something different you could do?

Stream Distillation

So to summarize: Putting all your data into a database is problematic because the data steadily grows and computing statistics based on the data is too slow. You also don’t really need to keep all your data at hand to have an analysis of the current state of the stream.

Stream processing, on the other hand, is a nice tool to scale your computations, but it doesn’t deal well with peak volumes, and depending on how you persist your data, you run into the same scaling issues as the database centric approach.

In other words, what we need is a way to separate the data we currently need for our analysis from the historic data and analyses, and we also need a way to limit the bandwidth of events while having as little distortion as possible on the statistics we’re interested in.

As it turns out, these kinds of problems are closely related to “frequent itemset” problems in data mining. The idea is to identify the top most frequent items in a data stream with limited resources, both in terms of memory and computation. Such methods are discussed, for example, in Chapter 4 of Rajaraman and Ullman’s upcoming book Mining of Massive Datasets.

Most of these methods work by keeping a fixed amount of counts and then replacing the least frequent ones when a new type of event occurs. They come with some sort of theoretical guarantee to identify the correct set of items, or at least some bound on the differences in the count. However, when analysing retweets you get quite good results because only a fraction of the retweets is retweeted more than once, so that most of the slots are occupied by retweets which do not reoccur and discarding them doesn’t hurt.

Summary

As you might have expected, this is exactly the approach we took with TWIMPACT. All the demos you can see at our site are powered by our new trending backend (codename “Trevor”), which is basically an in-memory trend database built using ideas from stream mining, together with read-only snapshots on disk for later historical analyses.

These ideas can of course also be combined with databases and stream processing, but already without a huge amount of parallelism, we’re able to process a few thousand tweets per second.

In summary: it’s not enough to just scale your database needs and your computational algorithms, to make your analysis framework stable against peak volumes and sustainable in terms of data growth, you also need to separate your analysis database from the historic data and add some ideas from stream mining to extract a statistically approximate subsample with capped bandwidth from your data.

Click here for the full article

Peer Review and NoSQL

2011-09-20T10:03:00+02:00

Disclaimer: This post definitely falls into the tl;dr categories of posts. I’ve been collecting these ideas for quite some time now, and somehow this post got longer and longer. Anyway, it is a complex topic.

Ever since I started to work in an academic environment back in ‘96 (working as a student in Rolf Eckmiller’s neuroinformatics group), the peer review process has always been a big topic. There were always some people complaining, discussing possible ways to improve it, or dismissing the whole idea of peer review at all.

The interesting thing is that very little has changed since then. If we look not only at peer review but the whole scientific publication landscape, you can see a few significant changes. For example, open access is much more real than it used to be, and there are more scientific journals which let authors keep their full copyright and the right to republish the papers on their webpages, etc.

These changes are all important, but what I find curious is that the general peer review process hasn’t changed at all. The process to get published or accepted at a conference is still the same: You submit your paper to some board where it gets handed to two or more reviewers whose identity is not revealed to you. Based on the verdict of the reviewers the action editor/program chair decides what to do with your paper.

I won’t repeat all the things which people perceive as being broken with this system. Let’s just say that the process has a high error rate, both false positives and false negatives (a.k.a. the “bad review problem”), it can take very long for a paper to get published, and the workload on the reviewers is pretty high.

Still, nothing has changed. Is this only because we sort of bring this problem upon ourselves (as opposed to the system being forced upon us by some external agency)? Or is there a deeper reason?

A Closer Look at Peer Review

I think the main reason why peer review is so resilient is that it is already a quite elegant partial solution to a number of interwoven requirements. In other words, just like democracy, it is an imperfect system but the best we have discovered so far.

In this whole scientific publication business, there are a number of stakeholders involved:

Science as a whole wants to progress to solve the smaller and bigger mysteries of life, the universe and all the rest. Science needs the publication system to be efficient, fair, and open, such that information can be distributed quickly and without bias.
Researchers want to have fun researching, but also need to build up a reputation to keep doing so. For this, they need the publication system to build up a track record of their work.
Researchers also need to have access to the works of others, to know what has already been done, which problems have been solved, and so on. The publication system is basically like an enormous, ever expanding library of knowledge.
Universities and funding agencies need the publication system to asses the scientific output of researchers for hiring decisions and to explain to tax payers how and why the money has been spent. Peer review is a very handy way of assessing scientific output. You can use it to basically just say “I don’t know exactly what they have been doing (and it’s probably not even practically relevant for another decade or so), but at least these other researchers said that it’s good.”
Publishers mainly want/need to make money (and probably also have a name in the whole scientific endeavor).

What’s important to understand, is however how peer review addresses all of these problems at least partially:

Exchange of information is more or less efficient, fair, and open. It could be more efficient, but the publication lag is still on the same order of magnitude as the actual work. It’s not like science is already five years ahead of a huge backlog of publications (at least I hope that’s not the case…) It is fair, because a good reviewer is bound by a scientific code of ethic to be a fair and unbiased, and it is open because everyone can submit something (as opposed to a closed club where you first have to become a member to get a chance to publish.)
Researchers get an excellent standardized measure of scientific output. A published journal paper is something nobody can take away from you. Not only is a published paper an important step towards tenure, it is also something everyone agrees on. On the other hand, peer review gives you a level of filtering such that the amount of new results is just large enough to be manageable.
Universities and funding agencies are also happy, because they have a solid, generally accepted measure of scientific productivity, which even laymen can understand.
Publishers can get their share by building a strong brand, becoming a journal with a high impact (while having researchers doing the actual peer review work for free, but that is another problem).

In summary, peer review is an okay solution to a complex problem, and whatever solution you propose to replace it has to cover all of these aspects as well.

You can’t ignore the complexity of the problem

You’re probably wonder when I’ll come to the NoSQL bit of this post, but before we get there, let’s briefly discuss how common alternatives fail because they do not address all of the above aspects.

For example, a common approach is to say that we should replace this whole process with a social media site around publications. Let’s just call it SciNet for now (and I know that already exists, but it’s really hard to find something in that namespace which isn’t already taken). “Likes” or “Recommendations” would work as filtering, connections between users give people structure to navigate, or to form “Web of Trusts”, and so on.

This idea has some appeal, but it neglects the aspect of building a track record and giving an objective measure of scientific output, because you’ll have a hard time explaining to your funding agencies that that non-peer reviewed paper of yours is a solid piece of work because it got 1.5M “likes” on SciNet. I’m not saying that this probably cannot be solved, but you can’t just copy existing concepts, and you would also need to invest quite an amount of lobbying to convince the universities which hire professors and the funding agencies which pay for your research to accept these measures.

Other approaches focus mainly turn-around times and open access, proposing some central server which is a mixture of a preprint server and a perpetual archive. Such systems don’t really address the filtering aspect, and also don’t deal with the main problem of how to improve peer review.

Finally getting to the NoSQL part

So from a higher-level, we have a situation which is pretty common in engineering: We have a well-tested and established piece of “technology” for a complex problem. It’s been around for quite some time now, and it shows. Somehow, it hasn’t kept up with the acceleration of communication which the Internet brought about. We’ve seen how fast information can be exchanged, and we’d like to have that kind of quality for our professional scientific exchange as well.

Of course, there is still room for improvement. People could just work harder to write better reviews on time, action editors could press reviewers harder to give good reviews. Already communities have found ways around the long turn-around times by moving to conferences (like computer science) or preprint servers (like physics). Conferences are actually an interesting example, because they play a quite different role in computer science and mathematics. In CS, conferences have become as important as journals, which is problematic because the review process is quite different (as there is really no way for a revision). In mathematics, conferences are much more informal. Often you can apply with just an abstract. That way, conferences function mostly as a platform for exchange, and less as an outlet for publications.

But in order to change the problem, you either have to find a solution which is uniformly better than the current system on all the aspects I’ve talked about earlier, or you have to put an equal amount of work into marketing to convince people that some of the aspects are not important anymore.

All of this reminds me of the NoSQL movement. Classical relational database systems were the standard till a few years ago. Like peer review they address a very sophisticated set of requirements, and have been around for quite some time. However, it also became more and more apparent that they aren’t good for certain applications.

The main contribution of the NoSQL movement was to understand that some of the requirements could be weakened because they really weren’t that important for certain kinds of applications, and to see how that changed set of requirements could be used to produce systems which scale more easily.

What does this mean for the scientific publication system? I think to find an alternative process, we need to be fully aware of all the requirements the current system addresses, but we also need to question these requirements and be ready to fight hard to make people do the same. Because otherwise we’re stuck with finding a system which is better in all the aspects than the current system.

Note that this approach is also different from just focusing on one aspect and ignoring the rest as some of the approaches I’ve discussed above. It’s really something different to say “we’ve considered these requirements, but we think they aren’t important anymore” than “we just considered half of the problem for now.”

Rethinking why we have peer review

So the question is which parts of the problem can go? I think generally there is the tendency to believe that we don’t really need the publishers anymore. The Internet has made it very easy to publish something even in a permanent fashion, and most of the actual work has already been done by us anyway.

There is really no way around an efficient exchange of information and being able to find the information you look for. These are probably the core requirements.

Track records and objective measures of scientific output are of course important, but I think we might be able to find something new here eventually (and the current system also doesn’t really work well anyway. Daniel Lemire has a number of posts how papers as units of scientific work don’t make sense).

I think peer review is still very valuable, but its role probably needs to change. If we find more effective ways of filtering and measuring the impact, we no longer need peer review to be the first threshold to publication, and we no longer suffer from its errors or long turn-around times.

What can we do for now

So what can we do for now. Actually, I think you can do a lot. Don’t forget that we’re running this system ourselves. So whenever you are a reviewer, work hard to be an unbiased and fair reviewer. Never recommend to reject a paper just because you somehow missed the point and didn’t like the overall approach. NEVER reject a paper simply because it hasn’t compared itself against method X (there are thousands of methods out there), unless there is a very good reason to do so. NEVER reject a paper because you believe it is similar to method Y, unless you are very certain that they are very similar. In all the cases I got reviews like this, it never was true.

If you are an action editor or area chair, don’t accept bad reviews. If you organize a workshop, think about alternative ways to accept and review papers. Turn your blog into an informal journal, invite people to submit their work if they want to get the word out.

If you are in a position to discuss with decision makers in funding agencies, talk to them about alternative ways to measure scientific output. If you are in a committee to hire new faculty members, don’t just rely on impact factors to assess the scientific output of a member, but encourage the others to also look at the contributions of a candidate to the community besides peer reviewed publications.

And if you want to develop something new, always be aware of the full complexity of the problem, and be ready to explain why you neglect some of its aspects.

For further reading, Marcio von Muhlen has an interesting post called “We Need a Github of Science” which covers a lot of ground, and also tries to take into account the whole problem.

A last piece of advice: First get tenure or some other kind of permanent position, then work on improving the system. Always remember that others are publishing papers in the old system while you fantasize about a better world.

Click here for the full article

Short Review: Visualize This by Nathan Yau

2011-08-26T12:20:00+02:00

I can’t really remember how I came across this book. I think it was recommended by Amazon. The price was ok (about 31€), at least for a Wiley book, so I just went ahead and bought it. You probably know Nathan Yau from his blog FlowingData where he frequently posts visualizations and interesting infographics.

Overall, the book is quite nice. It starts with some basic discussion about on visualizations in general, stressing the fact that visualizations are an excellent tool to tell the stories behind statistical data. It then goes through some tools, starting with Excel, and then covering tools like R, as well as JavaScript plotting libraries like protovis (now abandoned in favor of D3.js), and several other more specialized libraries, for example, for maps.

The remainder of the book goes through different kinds of visualizations in detail, from timeseries data, scatter plots, maps, etc. Each of the chapters focuses on one tool and show how to get the final plot in great detail.

What I found particularly interesting is that his workflow almost always includes refining the plot in Illustrator to make the graphic more appealing and to add explanations and further labels. This might be nice if you create static visualizations, but if you want to generate dynamic visualizations automatically from data, you’ll have to keep tweaking the original plot until it looks nice enough.

The book is probably too entry level if you already have some experience with data analysis or programming. It tries to require no prior knowledge in programming, although I wonder whether you can really learn how to use R from the examples alone. On the other hand, if you want to do some visualization with maps, for example, it’s nice to have almost complete examples in there.

I also particularly liked the introduction and the final chapter which give a lot of interesting insight on the business of creating visualizations.

Click here for the full article

Short Review of Edward R. Tufte's "The Visual Display of Quantitative Information"

2011-08-15T17:21:00+02:00

On the bottom line, I found the book quite interesting to read, although you probably would have managed to fit the material into three to five blog posts (yes, that’s how we measure document lengths today). The book spends about a third of the time reviewing the history of statistical plots. While it might be quite fascinating that William Playfair produced pretty modern looking plots already in the 18th century, I’m not sure that this is the best way to approach such a field.

Another third is spent on common mistakes and lies in statistics plots, including all kinds of exaggerations, visual noise (cross hatching, moire effects and friends), and downright stupid plots, for example “executive summary”-style plots consisting of only three bars (one of which is the sum of the other two).

The strongest part of the book IMHO was the third part which develops a number of design principles. Tufte’s main points are to use as much “ink” (read “toner” or “pixels”) as possible to show data, reducing gridlines and axes as much as possible. Tufte is also an advocate for data rich plots, arguing that our visual system is quite capable of dealing with high information densities.

Like most machine learners, I’ve done most of my plots with MATLAB and more recently matplotlib, and I’m sort of used to the style their provide. Tufte’s approach is somewhat different, and more clean, which is a nice change. JavaScript plotting libraries like protovis or D3.js follow the aesthetics of Tufte more.

What I particularily liked about his approach was the idea that visualizations can really help to understand data using our visual system. As he says, “Above all, show data”, meaning that you shouldn’t hesitate to put as much data as possible before your eyes (within reason) so that you can really start exploring the structure in your data visually.

Click here for the full article

Cross-post: Hey, Google+ is not world peace

2011-07-19T10:11:00+02:00

Sorry for the lack of posts lately. Somehow I’ve become a bit spread out too thin across tumblr, the TWIMPACT Dev Blog, and now of course Google+.

I originally set up this blog to be the ideal place for everything with jsMath to render some latex, markdown for the editing, and disqus for comments, but sometimes I find it easier to post something on tumblr. No idea whether this will eventually converge on some platform.

In any case, I posted the following originally on the TWIMPACT Dev Blog:

Some people start behaving like Google+ is the second coming of Christ, a cure against cancer and world peace all rolled into one. Mike Elgan has gone on a “Google+ diet” and is redirecting all his communication (including email!) to Google+. Other bloggers (e.g. Kevin Rose) have shut down their blog completely, redirecting their site to their Google+ profile. Others state that everything else has started to become boring once your exposed to Google+.

Now admittedly, Google+ is very nice and it’s definitely a big step forward, but there is still a lot that needs to be done. I’m not dismissing it in any way. It’s has a lot of potential, but as every complex system out there, there are a lot of details which need further attention.

So here is the list:

RSS feeds. A lot of people are still using RSS readers for blogs. Google+ doesn’t have this feature yet. Switching your blog to Google+ currently means that all those people won’t get your updates anymore. Doesn’t sound like a nice move to me. The closest thing you can get right now is an (unofficial) hack at http://plusfeed.appspot.com/
Some form of bookmarking. Just as in Twitter, it is currently very hard to find interesting stuff in your stream again. +1 would be a nice way to bookmark posts, but currently they don’t show up under the +1 tab.
Private messages. Sometimes you want to have a small private exchange. You can always go back to email, but that would be quite disruptive. I know that you can have a privat conversation if you share a post to just one person, but that not very obvious. What you need is a button or a menu entry next to the person’s icon or on their profile page.
Real-time search. Now that has me really baffled. On all other Google products (email, calendar, docs, etc.), search is an integral part of the experience. In fact, putting full text search on Gmail was one of the game changers back when it came out. Still, no real-time search on Google+, neither for all public posts nor on your private streams. IMHO, this is probably the most important feature that is missing right now.
Public API and third party integration. To cross post your stuff to Facebook and make everyone there crazy.

Click here for the full article

Resist Holy Tech Wars

2011-06-26T20:30:00+02:00

Only a Sith deals in absolutes.
Obi-Wan Kenobi, Star Wars Episode III: Revenge of the Sith

It might probably be due to my age (recently turned thirty-six), but I’m recently observing a certain aversion in me against tech holy wars. It’s something between ”C’mon, you can’t be serious” and “Not again”. I know that they have a decades old tradition in computer science, but still, I find them somewhat irritating.

Discussion about programming languages are full of these. For example, you have the dynamic vs. static typing discussion, functional vs. object-oriented, what is the right abstraction for concurrency: locks, actors, STM, or something entirely different, immutable vs. mutable data structures, etc. Of course, most of this discussion is led in a reasonably rational tone, but every now and then, you meet people who categorically reject anything which isn’t X.
In database systems, there is the NoSQL vs. SQL databases, which break down to consistency vs. being eventually consistent, “scaling up” vs. “scaling out”, and so on.
In Machine Learning we have the Frequentism vs. Bayesians divide. I actually often forget that people are taking this serious, but then again I end up with people rejecting ideas because they are from the other group.

Note that most of these questions are pretty big ones: Programming languages, databases, notions of probability and inference under uncertainty. There are little holy wars on smaller things.

I had an interesting exchange on Twitter today with David MacIver on this, and we agreed that in many cases, it is clear that there are arguments in favor of each side depending on the context. Different programming languages are fit for different things, and there is no language or programming paradigm which is universally superior. At the same time, the costs for mastering both alternatives are often quite large. You tend to grow attached to the programming paradigm you do most of your work in, and also to the tools, editors, libraries, the community, etc., and might over time become relucant to switch.

David added that in many cases, however, it’s actually not that difficult to do both but people suffer from sunk cost fallacies. What this means is that people are somewhat averse to writing off investments they have already made. Basically, if you spent a lot of time learning a certain technology, you would feel like you’ve wasted all that time if you switched to a different technology.

Of course, and this is the reason why it’s called a fallacy, this has usually nothing to do with which technology is actually better for the problem at hand.

As I said, holy wars are often about complex, high-level stuff. And unfortunately, things are never that easy. It’s usually quite complicated. Both sides have areas where they shine, and others where they fail. At the same time, problems usually also have rather complex requirements which seldom align with the available solutions.

I think that for many, holy wars are also rooted in a wish to simplify life, to get easy answers. “Which programming language should I learn?” For which problem? Number crunching? Distributed programming? Building a web site? “Which database technology is the best one?” Which will scale with your demands for the next ten years? There are no simple answers. But that’s life, basically.

I think what irritates me most about this is when smart people, who otherwise seem to be able to take in a huge amount of detail, give in to holy wars. In particular when they’re scientists, because our professionality demands that we’re open to new things, and always suspicious of what we think we already know.

To close this rant, and to counter the argument that I’m basically having a holy war against holy wars, I admit that holy wars (at least in tech) have their merit. Often, it forces both sides to focus on what’s special, and possibly also grow in the process. Given the huge number of technologies to learn today, it’s also a good thing to just focus on one thing for some time, and nothing helps like believing that you’ve found the greatest thing there is. At the end of the day, however, you have to put it all into perspective and admit that perfect solutions only exist in fairy tales.

Click here for the full article