thinkberg http://thinkberg.com/space/start start en Text 2012-10-01T11:10:14+01:00 http://thinkberg.com/space/start arte Blogging? http://feedproxy.google.com/~r/thinkberg/~3/JR8rSad2SqM/1 I wonder if I should discontinue the blog as I do not regularly write much. Usually it's just some random thing that comes to mind. Most of the interesting stuff happens on other networks, even though I do value having my ramblings under control. That is not so true on Facebook or Twitter. The integration into the blog is rather difficult with ever-changing rules and API's … http://thinkberg.com/space/start/2012-09-06/1#Blogging? >>Facebook or >>Twitter. The integration into the blog is rather difficult with ever-changing rules and API's …
]]> Blogging? en Text 2012-09-06T12:57:24+01:00 http://thinkberg.com/space/start/2012-09-06/1#Blogging? arte http://thinkberg.com/comments/start/2012-09-06/1#post http://thinkberg.com/space/start/2012-09-06/1#Blogging? DataScience to the rescue http://feedproxy.google.com/~r/thinkberg/~3/C5qJbcOKb3I/1 My colleague Mikio Braun just gave a talk on our history doing real-time analysis on the Twitter data stream. It's quite interesting how we started with jruby and rails and a normal RDBMS and worked it until it broke, then moved on to use scala (actually, I was pressing Mikio to get away from ruby and start using a JVM based solution) and the shiny new Cassandra NOSQL database. Again after some fine tuning and clustering it broke again for our use case.That's how we ended up, still using scala, to create our own storage and management platform that does exactly what we need and is fast enough to process the stream in real-time.Of course such a history of a technology fight is always interesting to other developers as we can tell them what to avoid or how to tackle certain problems, be it simple programming stuff or mathematical modeling. We ended up with an idea to bring the experience to the public by creating datascience-berlin.de, an offer of a number of seminars from basic DataScience to advanced issues and in future even more focussed seminars for specific industries.http://datascience-berlin.de/ | Twitter: @datasciencebln http://thinkberg.com/space/start/2012-04-20/1#DataScience_to_the_rescue >>gave a talk on our history doing real-time analysis on the >>Twitter data stream. It's quite interesting how we started with >>jruby and >>rails and a normal >>RDBMS and worked it until it broke, then moved on to use scala (actually, I was pressing Mikio to get away from ruby and start using a JVM based solution) and the shiny new >>Cassandra NOSQL database. Again after some fine tuning and clustering it broke again for our use case.

That's how we ended up, still using scala, to create our own storage and management platform that does exactly what we need and is fast enough to process the stream in real-time.

Of course such a history of a technology fight is always interesting to other developers as we can tell them what to avoid or how to tackle certain problems, be it simple programming stuff or mathematical modeling. We ended up with an idea to bring the experience to the public by creating >>datascience-berlin.de, an offer of a number of seminars from basic DataScience to advanced issues and in future even more focussed seminars for specific industries.

>>http://datascience-berlin.de/ | Twitter: >>@datasciencebln

]]> DataScience to the rescue en Text 2012-04-20T10:05:33+01:00 http://thinkberg.com/space/start/2012-04-20/1#DataScience_to_the_rescue arte http://thinkberg.com/comments/start/2012-04-20/1#post http://thinkberg.com/space/start/2012-04-20/1#DataScience_to_the_rescue Twitter Crowd Attention http://feedproxy.google.com/~r/thinkberg/~3/YQ8n6g5td_w/1 It is an easy task to map where users are, but what are the places people talk about? I have set myself the task to find out and a nice example is now visible in this youtube video shown mentioned locations during March 2011 when the earthquake hit japan :( http://thinkberg.com/space/start/2012-02-25/1#Twitter_Crowd_Attention >>this youtube video shown mentioned locations during March 2011 when the earthquake hit japan :(

2011-03-10@23:10

]]> Twitter Crowd Attention en Text 2012-02-25T16:26:07+01:00 http://thinkberg.com/space/start/2012-02-25/1#Twitter_Crowd_Attention arte http://thinkberg.com/comments/start/2012-02-25/1#post http://thinkberg.com/space/start/2012-02-25/1#Twitter_Crowd_Attention TWIMPACT NIPS preparations http://feedproxy.google.com/~r/thinkberg/~3/lDkwLgdYEuw/1 Finally, as Mikio wrote in our dev blog the analysis is running smoothly. Next we are going to put the non-serial analyzable stuff on the cluster to roll the whole of 2011 into our snapshots. It will be fun to see what are the hotspots. And as a side effect we are looking at a comparison of our own little language detector to the chrome language detector.What Mikio doesn't know, i am already planning on using more memory for even more fun stuff to analyze. http://thinkberg.com/space/start/2011-11-29/1#TWIMPACT_NIPS_preparations >>as Mikio wrote in our dev blog the analysis is running smoothly. Next we are going to put the non-serial analyzable stuff on the cluster to roll the whole of 2011 into our snapshots. It will be fun to see what are the hotspots. And as a side effect we are looking at a comparison of our own little language detector to the chrome language detector.

What Mikio doesn't know, i am already planning on using more memory for even more fun stuff to analyze.

]]> TWIMPACT NIPS preparations en Text 2011-11-29T19:31:51+01:00 http://thinkberg.com/space/start/2011-11-29/1#TWIMPACT_NIPS_preparations arte http://thinkberg.com/comments/start/2011-11-29/1#post http://thinkberg.com/space/start/2011-11-29/1#TWIMPACT_NIPS_preparations Warum der Bundestrojaner so schlecht ist ... http://feedproxy.google.com/~r/thinkberg/~3/dDhFvATAns0/1 Siehe Google+, wo ich in Ansätzen versuche zu beleuchten wieso Software des Bundes so schlecht ist ...https://plus.google.com/117122816629542437147/posts/f5WpLAYGqFj http://thinkberg.com/space/start/2011-10-09/1#Warum_der_Bundestrojaner_so_schlecht_ist_... >>https://plus.google.com/117122816629542437147/posts/f5WpLAYGqFj
]]> Warum der Bundestrojaner so schlecht ist ... en Text 2011-10-09T10:53:31+01:00 http://thinkberg.com/space/start/2011-10-09/1#Warum_der_Bundestrojaner_so_schlecht_ist_... arte http://thinkberg.com/comments/start/2011-10-09/1#post http://thinkberg.com/space/start/2011-10-09/1#Warum_der_Bundestrojaner_so_schlecht_ist_... The curse of the bit dump ... http://feedproxy.google.com/~r/thinkberg/~3/DvILzT4Zutw/1 I wonder where it will end. Everyone is speaking of big data and even I take part in it. At first I thought we might end up something like in "Muell", a comic book by Juan Gimenez picturing a world where only a very small part is still habitable and all the rest of it looks like a big dump. Funnily, that is not going to happen. At least I think that we are going in a different direction that has to do with our fascination for big data, which is closely related to "messyism": Stupidly collecting things we no longer need. A not so long time ago everything would at some point simply disintegrate because the material would not last forever. With the industrial revolution we invented more and more durable materials and now we have this volatile bit which is going to haunt us even longer, because we make it stick in our data centers. Already most of Europe and parts of the rest of our world are covered with data centers. We will need even more if we don't look back at how our ancestors handled information.Partly due to lack of storage, but also for the sake of simplicity and understanding information was boiled down, filtered and then came part where it was stored (drawn or written down mostly). That's the point where I think we need to think again and focus on the part where understanding and filtering reduces the amount of information.Still, there is this wish to keep up with the speed of our time and having it all available. At the speed we produce data, the analytics tools that evolve will lose in the end if we try to catch up the bit dumps already there. We have a chance though to keep the pace by shifting away from big data storage analytics to real-time data analytics that simply keeps our level of knowledge, the insights from our continuous data streams, up-to-date at all times.The resulting data will be much less and much more informative than all the bit muell it came from and it will in the end protect us from ending up to look like Coruscant, a planet of data centers. http://thinkberg.com/space/start/2011-06-27/1#The_curse_of_the_bit_dump_... >>I take part in it.

muellschwermetall At first I thought we might end up something like in "Muell", a comic book by Juan Gimenez picturing a world where only a very small part is still habitable and all the rest of it looks like a big dump. Funnily, that is not going to happen. At least I think that we are going in a different direction that has to do with our fascination for big data, which is closely related to "messyism": Stupidly collecting things we no longer need.

datacenters A not so long time ago everything would at some point simply disintegrate because the material would not last forever. With the industrial revolution we invented more and more durable materials and now we have this volatile bit which is going to haunt us even longer, because we make it stick in our data centers. Already most of Europe and parts of the rest of our world are covered with data centers. We will need even more if we don't look back at how our ancestors handled information.

Partly due to lack of storage, but also for the sake of simplicity and understanding information was boiled down, filtered and then came part where it was stored (drawn or written down mostly). That's the point where I think we need to think again and focus on the part where understanding and filtering reduces the amount of information.

Still, there is this wish to keep up with the speed of our time and having it all available. At the speed we produce data, the analytics tools that evolve will lose in the end if we try to catch up the bit dumps already there.

coruscant We have a chance though to keep the pace by shifting away from big data storage analytics to real-time data analytics that simply keeps our level of knowledge, the insights from our continuous data streams, up-to-date at all times.

The resulting data will be much less and much more informative than all the bit muell it came from and it will in the end protect us from ending up to look like Coruscant, a planet of data centers.

]]> The curse of the bit dump ... en Text 2011-06-27T09:39:20+01:00 http://thinkberg.com/space/start/2011-06-27/1#The_curse_of_the_bit_dump_... arte http://thinkberg.com/comments/start/2011-06-27/1#post http://thinkberg.com/space/start/2011-06-27/1#The_curse_of_the_bit_dump_... updated arab revolt trends with location detection http://feedproxy.google.com/~r/thinkberg/~3/2KFCSlPIGv4/1 IMPORTANT: To really update the application, give it a few reloads in the browser until you see a map below the keywords and the trending graph.I have updated our trend detection for the arabic world in two ways: I have separated libya and iran into their own trackers. The trends now also contain retweet information. An experimental features shows an approximate coordinate or what is being talked about in the tweet More to come, the data stream already transmits mention trends, retweet trends, hashtag trends and link trends in addition to what is already there. I just did not have enough time to put it into the interface. You can now select the tracker in the menu top right:Enjoy! http://thinkberg.com/space/start/2011-03-06/1#updated_arab_revolt_trends_with_location_detection IMPORTANT: To really update the application, give it a few reloads in the browser until you see a map below the keywords and the trending graph.

I have updated our trend detection for the arabic world in two ways:

  1. I have separated >>libya and >>iran into their own trackers.
  2. The trends now also contain retweet information.
  3. An experimental features shows an approximate coordinate or what is being talked about in the tweet
More to come, the data stream already transmits mention trends, retweet trends, hashtag trends and link trends in addition to what is already there. I just did not have enough time to put it into the interface.

screeshot16small

You can now select the tracker in the menu top right:

menu

Enjoy!

]]> updated arab revolt trends with location detection en Text 2011-03-06T22:25:47+01:00 http://thinkberg.com/space/start/2011-03-06/1#updated_arab_revolt_trends_with_location_detection arte http://thinkberg.com/comments/start/2011-03-06/1#post http://thinkberg.com/space/start/2011-03-06/1#updated_arab_revolt_trends_with_location_detection How to use egypt.twimpact.com http://feedproxy.google.com/~r/thinkberg/~3/Bkbfv-HTWl4/1 http://thinkberg.com/space/start/2011-02-17/1#How_to_use_egypt.twimpact.com link=http://egypt.twimpact.com
]]> How to use egypt.twimpact.com en Text 2011-02-17T22:43:27+01:00 http://thinkberg.com/space/start/2011-02-17/1#How_to_use_egypt.twimpact.com arte http://thinkberg.com/comments/start/2011-02-17/1#post http://thinkberg.com/space/start/2011-02-17/1#How_to_use_egypt.twimpact.com Spike on arrest of Al-Jazeera Journalists http://feedproxy.google.com/~r/thinkberg/~3/axlHLUtw1FE/1 Here is the spike our trender collected after Al-Jazeera journalists have been taken into custody. http://thinkberg.com/space/start/2011-01-31/1#Spike_on_arrest_of_Al-Jazeera_Journalists link=space/start/2011-01-31/1/wike.png
]]> Spike on arrest of Al-Jazeera Journalists en Text 2011-01-31T14:16:08+01:00 http://thinkberg.com/space/start/2011-01-31/1#Spike_on_arrest_of_Al-Jazeera_Journalists arte http://thinkberg.com/comments/start/2011-01-31/1#post http://thinkberg.com/space/start/2011-01-31/1#Spike_on_arrest_of_Al-Jazeera_Journalists Following #egypt on twitter http://feedproxy.google.com/~r/thinkberg/~3/d3EtHcVcYo8/1 Trying to follow all the keywords relevant to the egypt crisis can be very difficult. The updates sometimes come in so quickly that it is impossible to read. I've tried it first, tracking all keywords directly. The opportunity to actually provide a useful service is great, so I have installed a special egypt events tracker onhttp://egypt.twimpact.com/It tracks retweets that contain the keywords and ranks the keywords over time. All this is done using Mikios new "squid farm" trending system. This is not even starting to cratch the surface of its possibilities; but who cares. We will install it to track a few million topics on twimpact.com later on.It's actually streaming the latest tweets directly to all subscribers. The latest updates are there, a tag cloud showing the relevance of the keywords over time and the trending rate within the last hour.What's next? I am going to install a link trender to track all links and if I get it working a picture stream from the twitter data. http://thinkberg.com/space/start/2011-01-30/1#Following_#egypt_on_twitter >>http://egypt.twimpact.com/

It tracks retweets that contain the keywords and ranks the keywords over time. All this is done using >>Mikios new "squid farm" trending system. This is not even starting to cratch the surface of its possibilities; but who cares. We will install it to track a few million topics on >>twimpact.com later on.

It's actually streaming the latest tweets directly to all subscribers. The latest updates are there, a tag cloud showing the relevance of the keywords over time and the trending rate within the last hour.

What's next? I am going to install a link trender to track all links and if I get it working a picture stream from the twitter data.

]]> Following #egypt on twitter en Text 2011-01-30T12:56:45+01:00 http://thinkberg.com/space/start/2011-01-30/1#Following_#egypt_on_twitter arte http://thinkberg.com/comments/start/2011-01-30/1#post http://thinkberg.com/space/start/2011-01-30/1#Following_#egypt_on_twitter