O'Reilly Radar - Insight, analysis, and research about emerging technologies.

Developer Week in Review: Oracle's big bet fails to pay off

2012-05-25T13:00:00Z

I've been taking the opportunity this week to do some spring office cleaning. Unfortunately, I clean my home office infrequently enough that at a certain point, cleaning it becomes more an exercise in archeology than organization. There's nothing like finding a six-month-old check you never deposited to encourage more frequent cleaning.

The same can be said for code, of course. It's far too easy to let crufty code build up in an application, and then be faced with the mother of all refactoring efforts six months down the road, when your code finally reaches a critical mass of flaky behavior. It's worth the effort to continually refactor and improve your code, assuming you can convince your product management that quality is as important as new features.

Android is almost out of the woods

It wouldn't be a Week in Review without the latest in ~~Godzilla vs. Gamera~~ Oracle vs. Google. Things aren't looking all too sunny for Oracle at the moment, as the jury in the case just threw out all the patent-related claims in the lawsuit. This doesn't leave Oracle with much left on the plate, as the case now boils down to the question of whether the Java APIs are copyrightable. That's a matter the jury is deadlocked on.

Like all things legal, this is going to drag on for years as there are appeals and retrials and the like. But for the moment, it appears that Android is out of the woods, at least as far as the use of Java is concerned. Of course, there's still all those pesky International Trade Commission issues keeping many Android handsets waiting at the border, but that's a battle for another day ...

Scripters of the world, rejoice!

For Perl developers, a point release of the language is a major event, as it only occurs roughly once a year. This year's edition has just been released, and Perl 5.16 packs a ton of improvements (nearly 600,000 lines' worth!).

Since Perl is such a mature language, most of the changes are incremental. Probably the most significant is further enhancements in Unicode support. Nonetheless, there should be something useful for the serious Perl developer.

FreeBSD bids GCC farewell

As the licensing on the GCC compiler has become increasingly restrictive, some of us have been wondering when the fallout would start. Wait no longer: The FreeBSD team has ditched GCC for the more BSD-friendly licensing of Clang.

GCC has spent decades as the compiler of choice for just about everything, but recent changes in the GPL have made it less attractive to use, especially in commercial development. With the Apple-sponsored Clang compiler now seen as a viable (and perhaps even superior) alternative, with a much less restrictive license, the Free Software Foundation may need to decide if it would rather stand on principle, or avoid becoming marginalized.

Got news?

Please send tips and leads here.

OSCON 2012 — Join the world's open source pioneers, builders, and innovators July 16-20 in Portland, Oregon. Learn about open development, challenge your assumptions, and fire up your brain.

Save 20% on registration with the code RADAR20

Related:

Intellectual property gone mad

Mobile's embedded irony

More Developer Week in Review coverage

Four short links: 25 May 2012

2012-05-25T10:00:00Z

Meet The New Boss, Worse Than The Old Boss -- transcript of a thoughtful music industry insider considering the effect of the net on the business. The other problem? I’ve been expecting for years now to see aggregate revenue flowing to artist increase. Disintermediation promised us this. It hasn’t happened. Everywhere I look artists seem to be working more for less money. And every time I come across aggregate data that is positive it turns out to have a black cloud inside. Example: Touring revenues up since 1999. Because more bands are touring, staying on the road longer and playing for fewer people. Surely you all can see Malthusian trajectory?
Kottke on Quarterly -- I eyed TED's book club and thought "hmm, interesting business model: you like my taste, sign up and I'll send you things". Quarterly is a "my taste as a service" service. (via Sacha Judd)
Pipe Viewer -- clever little command-line utility to show progress of pipes.
Sheryl Sandberg's HBS Class Day Speech -- two things stood out, beyond the honesty of the talk: If you’re offered a seat on a rocket ship, don’t ask what seat (that's her quoting Eric Schmidt) and [careers] are not a ladder; they’re a jungle gym (her quoting Facebook's head of HR). (via Sacha Judd)

Strata Week: Visualizing a better life

2012-05-24T15:30:00Z

Here are a few of the data stories that caught my attention this week:

Visualizing a better life

How do you compare the quality of life in different countries? As The Guardian's Simon Rogers points out, GDP has commonly been the indicator used to show a country's economic strength, but it's insufficient for comparing the quality of life and happiness of people.

To help build a better picture of what quality of life means to people, the Organization for Economic Cooperation and Development OECD built the Your Better Life Index. The index lets people select the things that matter to them: housing, income, jobs, community, education, environment, governance, health, life satisfaction, safety and work-life balance. The OECD launched the tool last year and offered an update this week, adding data on gender and inequality.

Screenshot from OECD's Your Better Life Index.

"It's counted as a major success by the OECD," writes Rogers, "particularly as users consistently rank quality of life indicators such as education, environment, governance, health, life satisfaction, safety and work-life balance above more traditional ones. Designed by Moritz Stefaner and Raureif, it's also rather beautiful."

The countries that come out on top most often based on users' rankings: "Denmark (life satisfaction and work-life balance), Switzerland (health and jobs), Finland (education), Japan (safety), Sweden (environment), and the USA (income)."

Researchers' access to data

The New York Times' John Markoff examines social science research and the growing problem of datasets that are not made available to other scholars. Opening data helps make sure that research results can be verified. But Markoff suggests that in many cases, data is being kept private and proprietary.

Much of the data he's talking about here is:

"... gathered by researchers at companies like Facebook, Google and Microsoft from patterns of cellphone calls, text messages and Internet clicks by millions of users around the world. Companies often refuse to make such information public, sometimes for competitive reasons and sometimes to protect customers' privacy. But to many scientists, the practice is an invitation to bad science, secrecy and even potential fraud."

"The debate will only intensify as large companies with deep pockets do more research about their users," Markoff predicts.

Updates to Hadoop

Apache has released the alpha version of Hadoop 2.0.0. We should stress "alpha" here, and as Hortonworks' Arun Murthy notes, it's "not ready to run in production." However, he adds the update "is still an important step forward, as it represents the very first release that delivers new and important capabilities," including: HDFS HA (manual failover) and next generation MapReduce.

In other Hadoop news, MapR has unveiled a series of new features and initiatives for its Hadoop distribution, including release of a fully compliant ODBC 3.52 driver, support for the Linux Pluggable Authentication Modules (PAM), and the availability of the source code for several of its components.

Have data news to share?

Feel free to email me.

Related:

Visualization of the Week series

Data science is a pipeline between academic disciplines

Hadoop: What it is, how it works, and what it can do

Knight Foundation grants $2 million for data journalism research

2012-05-24T14:00:00Z

Every day, the public hears more about technology and media entrepreneurs, from when they started in the garages and the dorm rooms, all the way up until when they go public, get acquired or go spectacularly bust. The way that the world mourned the passing of Steve Jobs last year and that young people now look to Mark Zuckerberg as a model for what's possible offer some insight into that dynamic.

For those who want to follow in their footsteps, the most interesting elements of those stories will be the muddy details of who came up with the idea, who wrote the first lines of code, who funded them, how they were mentored and then how the startup executed upon their ideas.

Today, foundations and institutions alike are getting involved in the startup ecosystem, but with a different hook than the venture capitalists on Sand Hill Road in California or Y Combinator: They're looking for smart, ambitious social entrepreneurs who want to start civic startups and increase the social capital of the world. From the Code for America Civic Accelerator to the Omidyar Foundation to Google.org to the Knight Foundation's News Challenge, there's more access to seed capital than ever before.

There are many reasons to watch what the Knight Foundation is doing, in particular, as it shifts how it funds digital journalism projects. The foundation's grants are going toward supporting many elements of the broader open government movement, from civic media to government transparency projects to data journalism platforms.

Many of these projects — or elements and code from them — have a chance at becoming part of the plumbing of digital democracy in the 21st century, although we're still on the first steps of the long road of that development.

This model for catalyzing civic innovation in the public interest is, in the broader sweep of history, still relatively new. (Then again, so is the medium you're reading this post on.) One barrier that the Internet has helped lower is in the process of discovering and selecting good ideas to fund and letting bad ideas fall to the wayside. Another is changing how ideas are capitalized through microfunding approaches or how distributing opportunities for participation in helping products or services go to market now can happen though crowdfunding platforms like Kickstarter.

When the Pebble smartwatch received $10 million through Kickstarter this year, it offered a notable data point into how this model could work. We'll see how others follow.

These models could contribute to the development of small pieces of civic architecture around the world, loosely joining networks in civil society with mobile technology, lightweight programming languages and open data.

After years of watching how the winners of the Knight News Challenges have — or have not — contributed to this potential future, its architects are looking at big questions: How should resources be allocated in newsrooms? What should be measured? Are governments more transparent and accountable due to the use of public data by journalists? What data is available? What isn't? What's useful and relevant to the lives of citizens? How can data visualization, news applications and interactive maps inform and engage readers?

In the context of these questions, the fact that the next Knight News Challenge will focus on data will create important new opportunities to augment the practice of journalism and accelerate the pace of open government. John Bracken (@jsb), the Knight Foundation's program director for journalism and media innovation, offered an explanation for this focus on the foundation's blog:

"Knight News Challenge: Data is a call for making sense of this onslaught of information. 'As data sits teetering between opportunity and crisis, we need people who can shift the scales and transform data into real assets,' wrote Roger Ehrenberg earlier this year.

"Or, as danah boyd has put it, 'Data is cheap, but making sense of it is not.'

"The CIA, the NBA's Houston Rockets, startups like BrightTag and Personal ('every detail of your life is data') — they're all trying to make sense out of data. We hope that this News Challenge will uncover similar innovators discovering ways for applying data towards informing citizens and communities."

Regardless of what happens with this News Challenge, some of those big data questions stand a much better chance of being answered because of the Knight Foundation's $2 million grant to Columbia University to research and distribute best practices for digital reporting, data visualizations and measuring impact.

Earlier this spring, I spoke with Emily Bell, the director of the Tow Center for Digital Journalism, about how this data journalism research at Columbia will close the data science "skills gap" in newsrooms. Bell is now entrusted with creating the architecture for learning that will teach the next generation of data journalists at Columbia University.

In search of the reasoning behind the grant, I talked to Michael Maness (@MichaelManess), vice president of journalism and media innovations at the Knight Foundation. Our interview, lightly edited for content and clarity, follows.

The last time I checked, you're in charge of funding ideas that will make the world better through journalism and technology. Is that about right?

Michael Maness: That's the hope. What we're trying to do is make sure that we're accelerating innovation in the journalism and media space that continues to help inform and engage communities. We think that's vital for democracy. What I do is work on those issues and fund ideas around that to not only make it easier for journalists to do their work, but citizens to engage in that same practice.

The Knight News Challenge has changed a bit over the last couple of years. How has the new process been going?

Michael Maness: I've been in the job a little bit more than a year. I came in at the tail end of 2011 and the News Challenge of 2011. We had some great winners, but we noticed that in the amount of time from when you applied in the News Challenge to when you were funded could be up to 10 months, by the time everything was done, and certainly eight months in terms of the process. So we reduced that to about 10 weeks. It's intense for the judges to do that, but we wanted to move more quickly, recognizing the speed of disruption and the energy of innovation and how fast it's moving.

We've also switched to a thematic theme. We're going to do three [themes] this year. The point of it is to fund as fast as possible those ideas that we think are interesting and that we think will have a big impact.

This last round was around networks. The reason we focused on networks is the apparent rise of network power. The second reason is we get people, for example, that say, "This is the new Twitter for X" or "This is the new Facebook for journalists." Our point is actually, you should be using and leveraging existing things for that.

We found when we looked back at the last five years of the News Challenge that people who came in with networks or built networks in accordance with what they're doing had a higher and faster scaling rate. We want to start targeting areas to do that, too.

We hear a lot about entrepreneurs, young people and the technology itself, but schools and libraries seem really important to me. How will existing institutions be part of the future that you're funding and building?

Michael Maness: One of the things that we're doing is moving into more "prototyping" types of grants and then finding ways of scaling those out, helping get ideas into a proof-of-concept phase so users kick the tires and look for scaling afterward.

In terms of the institutions, one of the things that we've seen that's been a bit of a frustration point is making sure that when we have innovations, [we're] finding the best ways to parlay those into absorption in these kinds of institutions.

A really good standout for that, from a couple years ago as a News Challenge winner, is DocumentCloud, which has been adopted by a lot of the larger legacy media institutions. From a university standpoint, we know one of the things that is key is getting involvement with students as practitioners. They're trying these things out and they're doing the two kinds of modeling that we're talking about. They're using the newest tools in the curriculum.

That's one of the reasons we made the grant [to Columbia.] They have a good track record. The other reason is that you have a real practitioner there with Emily Bell, doing all of her digital work from The Guardian and really knowing how to implement understandings and new ways of reporting. She's been vital. We see her as someone who has lived in an actual newsroom, pulling in those digital projects and finding new ways for journalists to implement them.

The other aspect is that there are just a lot of unknowns in this space. As we move forward, using these new tools for data visualization, for database reporting, what are the things that work? What are the things that are hard to do? What are the ideas that make the most impact? What efficiencies can we find to help newsrooms do it? We didn't really have a great body of knowledge around that, and that's one of the things that's really exciting about the project at Columbia.

How will you make sure the results of the research go beyond Columbia's ivy-covered walls?

Michael Maness: That was a big thing that we talked about, too, because it's not in us to do a lot of white papers around something like this. It doesn't really disseminate. A lot of this grant is around making sure that there are convocations.

We talk a lot about the creation of content objects. If you're studying data visualization, we should be making sure that we're producing that as well. This will be something that's ongoing and emerging. Definitely, a part of it is that some of these resources will go to hold gatherings, to send people out from Columbia to disseminate [research] and also to produce findings in a way that can be moved very easily around a digital ecosystem.

We want to make sure that you're running into this work a lot. This is something that we've baked into the grant, and we're going to be experimenting with, I think, as it moves forward. But I hear you, that if we did all of this — and it got captured behind ivy walls — it's not beneficial to the industry.

Related:

Commerce Weekly: Facebook continues its mobile acquisition spree

2012-05-24T13:00:00Z

Here are the commerce stories that caught my attention this week.

Facebook acquires social gift-giving app Karma

The media focus on Facebook's IPO might be missing the point — as Michael V. Copeland points out in a post at Wired, the IPO is really about the cash Facebook now has on hand to continue its acquisition spree. And that's just what it did. On the heels of the market's closing bell last Friday (May 18), Facebook announced its acquisition of Karma, a mobile ecommerce app that facilitates social gift giving — what some analysts are calling the next big mobile commerce boom.

IDG News' Cameron Scott points out in a post at PCWorld that mobile is where Facebook needs to focus now, as "it currently generates less revenue per user [in mobile] than it does on the desktop." He notes that Karma already is integrated with Facebook and describes how the app works:

"Karma allows users to buy gifts from its catalogue from their mobile phones. Recipients receive a text message notifying them of the gift and directing them to a website where they can exchange it if they want to and enter their shipping address."

Chris Preimesberger over at eWeek.com reiterates Facebook's need to "monetize interaction with its subscribers in better fashion on mobile devices" and provides a nice roundup of Facebook's mobile-related acquisitions to date.

X.commerce harnesses the technologies of eBay, PayPal and Magento to create the first end-to-end multi-channel commerce technology platform. Our vision is to enable merchants of every size, service providers and developers to thrive in a marketplace where in-store, online, mobile and social selling are all mission critical to business success. Learn more at x.com.

If you can't beat "showrooming" ...

Raise your hand if you've showroomed. I'm certainly guilty, and much to retailers' dismay, this type of shopping behavior is becoming more common. Mobile Commerce Daily reports this week on a new study from the Interactive Advertising Bureau (IAB) that shows "53 percent of mobile commerce users have stopped an in-store purchase as a result of using their mobile phone." Joe Lazlo, senior director of the Mobile Marketing Center of Excellence at the IAB told Mobile Commerce Daily the practice of showrooming isn't likely to go away:

"Mobile shoppers today use their mobiles to save them time or money or both — mobile shopping helps make them more efficient. And since we all love to make the best use of our time and money, these kinds of behaviors are likely to spread and become mainstream really quickly.

"Fighting this trend is certainly going to be a losing strategy, so I predict that savvy marketers will learn to live with, and even court, these newly empowered consumers, who can and do leave a store if they discover a better deal somewhere else."

Staff contributors from McKinsey & Company call this situation a "retail store apocalypse" in a post over at Forbes. The group says, however, that "showrooming shouldn't be a show stopper. These digital shoppers are ready to buy. Excelling at multichannel sales is today's must-have capability, and retailers must adapt if they want to survive." They offer seven practical strategies to avoid succumbing to the apocalypse. Some highlights include establishing your store as an authority on your products, making use of data — or learning to — to better target consumers on an individual level, and re-imagining the role of your retail store as more of a service hub. You can read more on these ideas and their other suggestions here.

Mobile disrupts the grocery store checkout line

More and more, consumers are incorporating mobile into retail shopping, but what about grocery shopping? According to Kunur Patel in a post at AdAge Digital, the focus of mobile in grocery stores has more to do with in-store shopping behavior and convenience rather than online grocery competition. She points to loyalty program company Catalina Mobile and describes how it's testing the mobile grocery shopping waters, in part by targeting the big grocery store dice roll — choosing the fastest checkout line. Patel writes:

"[Catalina Mobile] is behind mobile apps in 110 of Ahold USA's Stop & Shop stores in Massachusetts, Rhode Island and Connecticut. Here's how it works: A shopper walks into the store, opens the store-branded app and receives offers based on their shopping history. To skip the line at checkout, shoppers can scan barcodes of items they put into the cart to buy the haul right on the phone. Catalina says the app will be available in 200 more Stop & Shops this summer."

Patel also highlights Coupons.com's app Grocery iQ:

"Shoppers create lists in the app by scanning barcodes or typing in entries, then the app organizes those items by aisle. The most recent [app] update ports coupons into the search tool so that, say, a shopper may add one brand of yogurt to the list over another because Coupons.com offers 50¢ off."

Unless the consumer is shopping at a participating store with a loyalty card that syncs with the app, the coupons will need to be emailed and printed.

Tip us off

News tips and suggestions are always welcome, so please send them along.

Related:

Facebook finds a mobile commerce partner

Facebook's shopping spree continues

Target doesn't want to be the showroom for online retailers

More Commerce Weekly coverage

Four short links: 24 May 2012

2012-05-24T10:00:00Z

Last Saturday My Son Found His People at the Maker Faire -- aww to the power of INFINITY.
Dictionaries Linking Words to Concepts (Google Research) -- Wikipedia entries for concepts, text strings from searches and the oppressed workers down the Text Mines, and a count indicating how often the two were related.
Magic Wand (Kickstarter) -- I don't want the game, I want a Bluetooth magic wand. I don't want to click the OK button, I want to wave a wand and make it so! (via Pete Warden)
E-Commerce Performance (Luke Wroblewski) -- If a page load takes more than two seconds, 40% are likely to abandon that site. This is why you should follow Steve Souders like a hawk: if your site is slower than it could be, you're leaving money on the table.

Jon Loeliger offers some practices to use with Git

2012-05-24T08:00:00Z

After finishing the second edition of "Version Control with Git," author Jon Loeliger talked to me about some of the advice he offers and how to use Git effectively as changes to code pile up.

Highlights from the full video interview include:

What's new in Git since the first edition of the book? [Discussed at the 0:38 mark]

Importance of understanding concepts behind Git [Discussed at the 2:40 mark]

How to manage complicated branching [Discussed at the 3:33 mark]

Aspects of Github beyond storage [Discussed at the 6:22 mark]

You can view the entire conversation in the following video:

Related:

Clojure's advantage: Immediate feedback with REPL

2012-05-23T15:00:00Z

Chas Emerick (@cemerick) is the co-author of "Clojure Programming" along with Brian Carper and Christophe Grand. He maintains a busy blog at cemerick.com and he also produces "Mostly Lazy … a Clojure podcast".

I asked Chas to enumerate some of the topics that would make a difference to developers: something that would attract attention to Clojure as a language, so we wouldn't spend time talking about yet another syntax. One of the first things he immediately mentioned was REPL. Writing code in Clojure is often about making changes and immediately seeing your results. Clojure has emphasized this shell-like approach to development and created an environment that allows for immediate evaluation of code and incorporating changes into a running process.

Clojure differs from other languages in that this interactive shell isn't an afterthought. Ruby's IRB or Java's beanshell are similar attempts at interactivity, but they are not primary features of each language. With Clojure, REPL is built in, and you can connect to any running Clojure process and modify and execute code. In this interview we discuss some of the possibilities that this introduces for Clojure developers.

The full interview is embedded below and available here. For the entire interview transcript, click here.

Highlights from interview:

On what is unique about REPL

"...what's really unique about Clojure is that most people's workflow when developing and using Clojure is tightly tied to the REPL, using the dynamic interactive development capabilities that the REPL provides to really boost your productivity and give you a very immediate sense of control over both the Clojure runtime, the application, programming or service that you're building." [Discussed 00:52]

How does REPL affect your development?

"Generally what you do in Clojure is you start up a JVM instance that is running the Clojure runtime and starts a REPL that you connect to ... then you stay connected to that running Clojure runtime for hours, days. I've had Clojure environments running for weeks in an interactive development setting, where you gradually massage the code that's running within that runtime that corresponds to files you have on disc.

"You can choose what code to load into that environment at any time, massage the data that's being processed by your application. It's a great feedback mechanism and gives you an immediate, fine‑grain sense of control over what you're doing [Discussed 01:18]

On using REPL to deploy production patches

"It's true, you can start up a REPL running on Heroku right now, connect to it and modify your application. Everything will work as you would expect if you happen to be running the application using Foreman locally.

"You don't want to be, in general, modifying your production environments from an interactive standpoint. You want to have a repeatable process ... Depending on the circumstances, it can be reasonable in a 'fire drill' situation to push a critical, time‑sensitive patch out to production ... 99% of the time you probably shouldn't reach for it, but it's very good to know that it's there." [Discussed 05:44]

On using REPL for direct access to production statistics

"JMX is great in terms of providing a structured interface for doing monitoring. But you need to plan ahead of time for the things you're going to monitor for and make sure you have the right extensions to monitor the things that you care about.

"Having a REPL available to connect to in every environment — whether it's development, user acceptance, functional testing or production — means that when you need to, you can get in there and write a one‑off function. Pull some data from this database, see what's going on with this function, capture some data that you wouldn't normally be capturing, stuff that may be far too large to log or to practically get access through JMX. It's a great tool." [Discussed 05:44]

Related:

White House launches new digital government strategy

2012-05-23T14:33:40Z

There's a long history of people who have tried to transform the United States federal government through better use of information technology and data. It extends back to the early days of Alexander Hamilton's ledgers of financial transaction, continues through information transmitted through telegraph, radio, telephone, and comes up to the introduction of the Internet, which has been driving dreams of better e-government for decades.

Vivek Kundra, the first U.S. chief information officer, and Aneesh Chopra, the nation's first chief technology officer, were chosen by President Barack Obama to try to bring the federal government's IT infrastructure and process into the 21st century, closing the IT gap that had opened between the private sector and public sector.

Today, President Obama issued a presidential memorandum on building a 21st century digital government.

In this memorandum, the president directs each major federal agency in the United States to make two key services that American citizens depend upon available on mobile devices within the next 12 months and to make "applicable" government information open and machine-readable by default. President Obama directed federal agencies to do two specific things: comply with the elements of the strategy by May 23, 2013 and to create a "/developer" page on ever major federal agency's website. Here's an excerpt from President Obama's memo:

The innovative use of technology is fundamentally transforming how the American people do business and live their daily lives. Exponential increases in computing power, the rise of high-speed networks, and the growing mobile revolution have put the Internet at our fingertips, encouraging innovations that are giving rise to new industries and reshaping existing ones.

Innovators in the private sector and the Federal Government have used these technological advances to fundamentally change how they serve their customers. However, it is time for the Federal Government to do more. For far too long, the American people have been forced to navigate a labyrinth of information across different Government programs in order to find the services they need. In addition, at a time when Americans increasingly pay bills and buy tickets on mobile devices, Government services often are not optimized for smartphones or tablets, assuming the services are even available online.

On April 27, 2011, I issued Executive Order 13571 (Streamlining Service Delivery and Improving Customer Service), requiring executive departments and agencies (agencies) to, among other things, identify ways to use innovative technologies to streamline their delivery of services to lower costs, decrease service delivery times, and improve the customer experience. As the next step toward modernizing the way Government works, I charged my Federal Chief Information Officer (CIO) with developing a comprehensive Government-wide strategy to build a 21st century digital Government that delivers better digital services to the American people.

Today, the CIO is releasing that strategy, entitled "Digital Government: Building a 21st Century Platform to Better Serve the American People" (Strategy), which provides agencies with a 12-month roadmap that focuses on several priority areas.The Strategy will enable more efficient and coordinated digital service delivery by requiring agencies to establish specific, measurable goals for delivering better digital services; encouraging agencies to deliver information in new ways that fully utilize the power and potential of mobile and web-based technologies; ensuring the safe and secure delivery and use of digital services to protect information and privacy; requiring agencies to establish central online resources for outside developers and to adopt new standards for making applicable Government information open and machine-readable by default;

aggregating agencies' online resource pages for developers in a centralized catalogue on www.Data.gov; and requiring agencies to use web performance analytics and customer satisfaction measurement tools on all ".gov" websites.

Ultimately, this Strategy will ensure that agencies use emerging technologies to serve the public as effectively as possible. As a Government, and as a trusted provider of services, we must never forget who our customers are — the American people.

While Kundra and Chopra set in a motion of series of reforms, from more transparency on IT spending and waste to an ambitious open government data program to adoption of cloud computing to improved IT security and a modern approach to open innovation, they left an immense portfolio and set of challenges for Steven VanRoekel and Todd Park to take on and implement against. Many of those challenges remain the same, including attracting talent, reforming procurement, data quality, and the reality of agency mainframes that are still running on COBOL.

"There are many things the federal government should do to improve IT performance and efficiency," said Darrell West, vice president for government studies and director of the Center for Tech Innovation at the Brookings Institute, when asked for comment. "It can quit adopting expensive legacy systems that are obsolete from the moment they are purchased and move towards more nimble strategies. There are many new apps that are available through the federal Apps Store and agencies can use them to improve performance and cut costs. In the medical area, there are 40,000 mobile health applications. Technology should be a money saver, not a money waster. The key problem federal officials face is overcoming economic interests that are vested in the past, not the future of technology innovation."

Big visions matter, in terms of inspiring the country to action or a historic course, from building transcontinental railroads to sending men to the Moon to starting up a new government agency. Implementing against that vision, however, in a time of great budget pressure, increased demands for government services online and falling trust in institutions, is just as important.

Today, VanRoekel and Park have put their own stamp on the future of digital government in the United States with the introduction of a digital strategy for 21st century government, which went online this morning. (In a notable upgrade to the way such policies have been released, this digital strategy was coded in HTML5.)

They introduced it to the nation in an decidedly non-Washingtonian sort of way, traveling north to New York City, a city that has become one the global epicenters for data-centric digital government under Mayor Michael Bloomberg, to directly engage Gotham City's community of tech entrepreneurs, venture capitalists and (civic) developers at TechCrunch's Disrupt Conference. Video of their announcement at Disruptis embedded below:

Whether their "pitch" for smart government gets funded and supported is an open question. One measure will be whether this trip results in new recruits: Park and VanRoekel urged the developers and entrepreneurs in the room to "join a startup called the U.S. government" as one of 15 "Presidential Innovation Fellows." Park said in New York that the fellowships will focus on five areas: a USAID campaign, open data across the federal government, releasing more health information, an online system they're calling "MyGov," a request for proposals program to help startups with procurement.

You can watch Park and VanRoekel answer follow up questions from writer Greg Ferenstein (@ferenstein) on procurement, the role of the private sector and emerging technology in the video below:

Park and VanRoekel took questions from the press on a call following their presentation. Molly Walker published audio of their discussion of the digital strategy, which you can listen to in the player below:

What's next for open data?

There have been longstanding questions about what constitute "high quality datasets" for years. VanRoekel, in past conversation, has identified an activity that many Americans engage in — looking for real estate. Buying a house is the single biggest purchase many citizens will ever make.

Now, VanRoekel and Park are talking about releasing more data that could actually inform that purchase. The challenge is that the third parties that they want to have use it need to have it be high quality, regularly updated, standardized and accessible. For instance, developers making energy apps don't want data published every quarter: they want it every week, every day or, if possible, every hour.

To learn more about how they're thinking about these challenges, I interviewed Park and VanRoekel yesterday about the components and thinking behind this strategy. My discussion with VanRoekel follows. Look for an extended discussion with Park and video from TechCrunch Disrupt later today.

This digital strategy has been coming for a long time. What's in it?

VanRoekel: Over the last 10 months or so we've been hard at work on a bunch of divergent strategies, thinking about our web presence as a federal government, managing the 1,800 .gov domain names we have out there, and tens of thousands of websites and millions of pages. We've been focused on mobile and the consumerization of technology and that impact on government. And as you know, I come storming in with a passion around open data, making data machine-readable and accessible, unlocking the potential of that data, as you and I talked about last time, and many times at the FCC. We had a moment this winter when we realized these divergent strategies were really converging into one thing, where they were all parts of a bigger strategy, which breaks down into several areas.

This first is an information-centric approach. That pillar is about making data openly available as the new default across government.

The second is around a shared platform, where we can share resources across government, removing duplication and leveraging existing projects that we're doing.

The third layer is all about customer centric-government. Typically in the government, we very tightly couple the presentation of stuff with the stuff itself. Finding that stuff is nearly impossible, if you have to navigate a quagmire. The nature of this is embracing the new application delivery model, looking at how we deliver content and data more effectively to Americans. Also, how do we create, across the customer-centric platform, a way of creating government as a platform, where we have a common approach to building applications on top of government data. Our goal is to bring government to Americans where they live, versus expect Americans to come to government, where we are. We saw this in the past, with weather and GPS. Combined, there's over a $100=billion dollar industry, if not more. We now, as Americans, probably take that for granted, in the ability to just connect in and use that data from the government. We have unending examples of places where this government data can be unleashed to the private sector and to American citizens to provide better e-services.

The last pillar, which I won't go into depth now but that it's very important that we're doing, is that we're building consistent methodology around security and privacy. We're calling on a bunch of groups in this strategy to help us on that front.

How will government become more data-centric?

VanRoekel: Yesterday was the three-year anniversary of Data.gov. It was an important first step to raise the awareness of people around data. It largely was a bulk upload, bulk download system. It didn't manifest in data being digital by default through the entire lifecycle of that data. It was sort of an add-on, to take the step of getting it to Data.gov. The key difference in what we're doing here and evolving Data.gov — which will be involved, as part of this — is architecting data for openness.

There are some very specific deliverables in the strategy itself, including a government-wide open data content and web API policy and then work with NIST and others to identify standards and best practices for interoperability to scale that across government. We will have, from its creation to its dissemination, open data as the new normal. We're going to, as part of this, inspire investing in IT systems that respect open data and think about open data.

This also bridges into big data and that roadmap as well. The next step of that is thinking about web APIs, how we build for modularity. It's about how we deliver our content and manifest systems in a way that allow APIs to do that. Part of this deliverable will also be about expanding Data.gov to also be the metadata catalog of all data in the federal government. It won't be a place where you use data but it will be a place where you find data, discover data or discover the relatedness of data. We also intend to take Data.gov and add some API key management and some other technology there, so it is the developer platform of all this stuff.

As you remember from the Open Government Directive, where we directed all agencies to create a "/open" page. Part of this deliverable will be all agencies creating a "/developer" page, to start to build the catalog of citizen-accessible web APIs for their systems.

This may all be a bit abstract to a non-technical citizen. Why is open data important to the American people? Why is this a strategy that's important enough to elevate to this level?

VanRoekel: For citizen-facing discussions and the conversation I'll have with consumer publications, I start with the customer-centric element and think about what are those applications that will be delivered by government and go drive a new set of normals and value systems. I always harken back to examples like GPS, like weather, but we can also look at early agencies that have done a better job, in the realm of health, tax, or travel, including that MyTSA app on your smartphone giving you wait times and security gates at the airport nearest you, or travel tips. Those are possible from unlocking government data and making that government data available to application providers.

To the TechCrunch crowd, I'm going to talk about real estate. When you're buying a home, why doesn't it manifest to you the myriad of data that the government has locked up about school quality, healthcare quality, infrastructure investments, broadband, everything else that people really care about when they're picking a place to live? We don't do that — we do roof composition and the number of bathrooms, and that's typically the extent of it. Some services are doing a better job with other government data but largely it's pretty silo'ed and not very specific to what Americans really care about.

How will this strategy result in those releases happening in the format and timeliness that making that kind of data useful would require? How do you drive the timely release of data in a consumable format from agencies, given the technical challenges they have?

VanRoekel: That's why we're launching the strategy now. It couldn't more important to manifest those things across all of the opportunities we have. From declaring that open data is now the new default — that you have to, when you create datasets, do so in an open way — when we say that when you buy IT systems that interface with data, they need to be purchased in a way that respect the aspects of open data, when you talk about dissemination of data, we're going to inspire in the strategy different aspects of agencies building and delivering open data solutions. The first tranche of this is that each agency has to do two of them. That's a "crawl, walk, run" approach. You have to have the magic mix of all of these elements: great policy, great technology, and great people leading the way. That's what we're trying to bring to bear to get there. We've been kind of haphazard. There's been no one at the top saying "thou shalt do this." Now we have that with this strategy.

President Obama has talked about information technology with some frequency, in the context of the importance of American innovation, improved government services and job creation. How much have you talked with the president about this strategy? How closely has he been involved in helping to shape it? How much support do you have from him?

VanRoekel: We have an incredible amount of support, both from the president and his staff. He was excited enough about this strategy that he insisted in actually issuing a memo as a cover page for it. This presidential memorandum is basically telling agencies "go do this stuff. It's important," talking about the necessity of unlocking this government technology on behalf of citizens. That doesn't happen very often, with him weighing in on the importance of what's happening here.

If you look at how well agencies have complied with the "/open" requirements, in terms of simply updating their open government plans, you'll see some laggards. What policy hooks will you have, with respect to making sure the deliverables in this strategy come through?

VanRoekel: The important part of this is that we have one policy out there. Policy doesn't go all the way. Policy should be a place that gives framing and direction, and inspires people to act. It explains the art of the possible. Part of this is just education on what is the possible and how we do this. I think the important part that will make this stick long term includes three things.

One is oversight. Part of this strategy will be working with agencies to create reporting mechanisms — and those reporting mechanisms will be built in open, standard ways that allow people to create easily accessible dashboards that track agency progress on these deliverables, so we can put some pressure on people from an oversight standpoint. The new normal for dashboards needs to be open data as the feeds, and allowing people to build on top of them.

The second part is regular accountability in meetings with us. We've got other mechanisms to touch base with people, too. I care a lot about this. At the FCC, I mandated this stuff and watched and saw it through to some really successful endpoints.

The third — and this is probably one of the most important elements — is injecting this into the different parts of doing business within government agencies. It's getting the permit team to do new things. It's getting the IT team to buy and deploy things in new ways.

Luckily, we're not starting from scratch. I think agencies generally —- and IT professional in those agencies — want to deliver against mission. They also care a lot about this stuff. They know this is the new way of doing things. They just need to be empowered in a way that gets them to that end point. We're not going to get a huge tide of resistance. It's going to bring them measurable value.

Part of this strategy is around mobile consolidation, including contracts and other things which are going to save them a lot of money in this fiscal environment. We've got a nice forcing function there.

How much is the "bring your own device" (BYOD) wave in government and enterprise driving this data-centricism of this digital strategy? CIOs are stressed about not being able to get mobile app development talent to create multiple applications for multiple platforms.

VanRoekel: It definitely is part of it. We have to walk that careful walk of being device and vendor-agnostic, in terms of the strategies we lay out. We don't want to be creating a marketplace winner through policy. We also have to be cognizant that we really have to open up the doors here for us to utilize these government systems in new ways. I know that's not going to be a sweeping phenomenon right away. We need to get stuff moving in the right direction. I think this is the way we do it. We do it by getting agencies to deliver solutions that are decoupled from the underlying data and ride government like a platform. We rapidly prototype and create single websites presence for the government. We get the private sector to step up and deliver solutions as well.

So you want agencies to be "dogfooding" the same data and web services that will be consumed by third parties?

VanRoekel: I think there have to be places where, when we unlock the data, it makes sense for us to actively deliver solutions against that data. That's either proof-of-concept work or actually deliver solutions against it.

This post has been updated to add audio and video when it became available online.

Schlomo Schapiro on continuous delivery platforms

2012-05-23T13:05:14Z

In this new Velocity Podcast, I had a conversation with Schlomo Schapiro (@schlomoschapiro), a Systems Architect at ImmobilienScout24, and a Velocity Europe committee member. This conversation centers mostly on building a continuous delivery platform. Schlomo has some interesting insights into how to deploy a platform that delivers 24x7.

Our conversation lasted 00:11:15 and if you want to pinpoint any particular topic, you can find the specific timing below. Schlomo talks a little about ImmobilienScout24, the tech stack they use, and his role at about 40 seconds into the conversation. The rest of the conversation is outlined below.

Can you explain what a continuous delivery platform embodies? 00:01:37

What role can DevOps play with a continuous delivery platform? 00:03:29

How do you demonstrate the business value in building a continuous delivery platform and that it is working? 00:04:40

What does a typical stack for a continuous delivery platform look like? 00:05:42

What kind of of organizational changes are needed for a continuous delivery platform? 00:06:46

What does the technical stack look like for a continuous delivery platform? 00:07:49

So an open source stack is the way you would build a continuous delivery platform? 00:08:52

Are we going to see you at Velocity in Santa Clara in June and London in October? 00:10:20

Velocity 2012: Web Operations & Performance — The smartest minds in web operations and performance are coming together for the Velocity Conference, being held June 25-27 in Santa Clara, Calif.

Save 20% on registration with the code RADAR20

Related:

Subscribe to the free Velocity podcast through iTunes

See more Velocity podcasts

Four short links: 23 May 2012

2012-05-23T10:00:00Z

Tale of Two Pwnies (Chromium Blog) -- So, how does one get full remote code execution in Chrome? In the case of Pinkie Pie’s exploit, it took a chain of six different bugs in order to successfully break out of the Chrome sandbox. Lest you think all attacks come from mouth-breathing script kiddies, this is how the pros do it. (via Bryan O'Sullivan)
The Future is Specific (Chris Granger) -- In traditional web-MVC, the code necessary to serve a single route is spread across many files in many different folders. In a normal editor this means you need to do a lot of context switching to get a sense for everything going on. Instead, this mode replaces the file picker with a route picker, as routes seem like the best logical unit for a website. There's a revolution coming in web dev tools: we've had the programmer adapting to the frameworks with little but textual assistance from the IDE. I am loving this flood of creativity because it has the promise to reduce bugs and increase the speed by which we generate good code.
Best Online Editors For Teaching HTML/CSS/JS (Pamela Fox) -- Over the past few months, I've been teaching in-person classes on HTML, CSS, and JavaScript, as part of GirlDevelopIt San Francisco. Along the way, I've experimented with various online consoles and editors, and I thought I'd share my experience with using them for teaching.
Makie -- design a doll online, they'll 3d-print and ship it to you. Hello, future of manufacturing, fancy seeing you in a dollhouse!

A gaming revolution, minus the hype

2012-05-22T14:00:00Z

In the following interview, "Playful Design" author John Ferrara (@PlayfulDesign) explains what he sees as the real gaming revolution — not "gamification," or the application of gaming characteristics to existing applications and processes, but how games themselves can and will be a "force of cultural transformation." Ferrera also reveals five universal principles of good game design.

Our interview follows.

How are mobile and social technologies affecting game design and the evolution of gaming technology?

John Ferrara: One of the really surprising things about modern smartphones and tablets is that the've turned out to be such credible gaming platforms. They open doors to new ways of experiencing games by giving designers access to touchscreens, accelerometers, cameras, microphones, GPS, and Internet connectivity through a single device. They also allow games to be experienced in new contexts, enjoyed on the train to work, in the minutes between meetings, and while you're out with friends. The traditional gaming model, where players sit passively in one place in the home and stare at a fixed screen, seems stodgy and limiting by comparison.

The funny thing about social technology is that before we had video games, gaming was almost always a social activity. You needed to have multiple people to play most board games, card games, and sports — in fact, the game was often just a pretense for people to get together. But then video games made solitary experiences more of the norm. Now social technology is bringing gaming back to its multiplayer roots, but it's also going beyond what was ever possible before by enabling hyper-social experiences where you're playing with dozens of friends and family at once. Even though you may be separated from these people in space and time, you have an intimate sense of shared presence and community when you're playing. That's revolutionary.

How do you see the social media aspects of gaming seeping into day-to-day life?

John Ferrara: Games certainly can transform the workplace, though I want to caution that it's very easy to make the mistake of dressing up everyday work activities as games by just tacking on some points and badges. That's not game design, and people will recognize that it's not. In the process of failing, approaches like this generate cynicism toward the effort. Games need to be designed to be games first and foremost. They must be intrinsically rewarding, enjoyed for their own sake.

That said, I absolutely believe that games can work at work. As you suggest, for example, they have great strengths for training. Games create a safe space for people to test out their mastery of a set of skills in ways that aren't possible or practical in the real world. They can also help people figure out how best to handle different situations. Say, for example, that you created a game to develop management skills. You might allow players to assign values to their in-game avatars like "nurturing," "autocratic," or "optimistic," which lead to different behavior paths. Players could then examine how these traits play out in a situation filled with characters who have different values like "dependability," "autonomy," and "efficiency." A structure like this could not only impart insight about management styles, but also invite introspection about how an individual's own personality traits may lead to success and failure in the real world.

In your book's introduction, you say, "I hope to start moving toward a post-hype discussion of how games can most effectively achieve great things in the real world." Who is leading the way — or at least moving in the right direction — and what are they doing?

John Ferrara: You know, there's so much really inventive work being done right now. Recently, I've been playing a lot of "Zombies, Run!," and I think it's great. This is a game for smartphones that overlays a narrative about survivors in a zombie apocalypse onto your daily run. As you're out getting your exercise, you're listening to the game events as they unfold, and you can hear the zombies closing in. It's a great use of fantasy, and it plays as a true game with meaningful choices and conflict.

There's also a great group at the University of Wisconsin-Madison that's developed a smartphone app called ARIS, which builds game scenarios into physical locations, and they've developed dozens of applications for it. One of them is being developed as a museum tour for the Minnesota Historical Center, giving people quests to complete by scanning objects in the exhibit and then using them to complete objectives in a story line. The museum is actually changing the way the exhibit is laid out to better accommodate the gameplay, moving away from the traditional snaking path to more of an open layout that allows players to move more freely between the interacting displays to solve the game's challenges.

Some of the thought leaders who I really admire include Eric Klopfer and Scot Osterweil at MIT, Ian Bogost at the Georgia Institute of Technology, and Jane McGonigal. A common current among these thinkers is their emphasis on games themselves as a force of cultural transformation, rather than simplistic "gamification" of software applications that lead to little or no meaningful change.

What about engineering games like "Foldit" — with improved UX, could this type of crowdsourced gaming become a viable research tool?

John Ferrara: This is what's been called "human computation," where a group of people work together to solve some complex problem as a by-product of some other action, like playing a game. Luis Von Ahn at Carnegie Mellon describes games as algorithms that are executed by people rather than machines, and I think that's a really fascinating idea. Foldit is a great example. This is a puzzle game where players try to figure out how to fold chains of proteins. This is a problem that's very well suited to human computation because it requires a type of intuitive reasoning that's very difficult for actual computers. Foldit made a big news last fall when the people playing it decoded the structure of a protein related to a virus that causes AIDS in monkeys, which had eluded researchers for years.

This is a wonderful demonstration of how this type of game can be really valuable to researchers. At the same time, I'm very critical of Foldit because I think its gameplay experience is kind of awful. It's very difficult to figure out which actions lead to the results you see on-screen — like why you're awarded points the way you are — and there's not a strong sense of objectives or conflict. These design issues place limits on the appeal of Foldit, and that's a big problem because human computation works better the more people you have playing. If the gameplay were really compelling and fun, then the sky would be the limit.

How do you see the collection and use of gaming data evolving?

John Ferrara: Games can produce enormous volumes of data because it's really simple to gather every little interaction the player has in the game and report it all back to a central server. This has immediate applications for game design itself. Zynga, for example, uses data to determine which design choices create greater tendencies for players to stay engaged longer, involve more friends, or pay to enhance the game experience. I expect this kind of data collection and analysis to become the norm because companies will be more successful the better they can do it.

I would suggest that financial services could be one of the biggest secondary beneficiaries of such data because there's so much to learn about how people make financial decisions under different circumstances. Staying with the Zynga theme, suppose players have the option of investing in any of a variety of different farm crops, each of which has different strengths and vulnerabilities to environmental conditions. How do players choose which ones they should purchase? How do they appraise risk and reward? Which presentations of information lead to a better understanding of a crop's attributes? Which lead people to make more appropriate choices for their goals? All of these questions can be examined quantitatively through games and can lead to greater insights into the innate qualities of human psychology that drive investor behavior and decision making.

What are some emerging best practices for game technology?

John Ferrara: Best practices vary widely depending on the game and the type of player motivations to which it appeals. For example, games meant to promote a sense of immersion like "Red Dead Redemption" remove as much of the user interface elements from immediate view as possible. Data-intensive games like "Tiny Tower" benefit by compressing as much information and as many functional controls as they can into the smallest possible space.

With that in mind, there are some clear universal principles for the design of all games:

Skip the manual and embed as much instruction into the gameplay as you can.
Fit the game into the player's lifestyle so that he or she can play when and where it's convenient.
Don't cheat — people recognize when a game unfairly stacks the odds against them and they resent it.
Make sure players always have a clear sense of cause and effect, and that they understand what actions are available to them.
Above all, playtest, playtest, playtest. It's impossible to fully anticipate how people will react to a game short of actually watching them play it.

In the book, you argue that games should be used as instruments of persuasion. Why is this?

John Ferrara: To be clear, it's not that all games should be persuasive but that people who want to persuade should look at games very seriously; I believe they present an ideal way to convince people to adopt a particular point of view or to move them to action in the real world. Ian Bogost describes games as a form of "procedural rhetoric," meaning that they communicate messages through participation in the experience. This creates a lot of advantages for persuasion. For example, it allows a kind of self-directed discovery where people adopt the designer's message as a working hypothesis and then test its truthfulness through the gameplay. That's a really powerful way to get your point across. Furthermore, it builds a sense of personal ownership of the insight the player has uncovered.

Are there ethical concerns related to persuasion in gaming environments?

John Ferrara: As there are for any medium, certainly. Film, television, books, billboards, oratory, and posters have all been appropriated for less-than-above-board purposes. Whether it's propaganda, demagoguery, misleading advertising, or dirty politics, you'd expect that games would be subject to the same kinds of unethical practices. It's especially important to be aware of this in the case of games, considering how compelling a procedural rhetoric can be. Rather than casting a negative light on games, however, I think that speaks to their power to effect meaningful change in the real world. I believe that games can achieve great things, and I expect that over the next decade we'll see them doing a lot of good.

This interview was edited and condensed.

Related:

Gamification has issues, but they aren't the ones everyone focuses on

7 areas beyond gaming where Kinect could play a role

Three game characteristics that can be applied to education

The purpose of gamification

In defense of games in the workplace

Data journalism research at Columbia aims to close data science skills gap

2012-05-22T13:07:03Z

Successfully applying data science to the practice of journalism requires more than providing context and finding clarity in vasts amount of unstructured data: it will require media organizations to think differently about how they work and who they venerate. It will mean evolving towards a multidisciplinary approach to delivering stories, where reporters, videographers, news application developers, interactive designers, editors and community moderators collaborate on storytelling, instead of being segregated by departments or buildings.

The role models for this emerging practice of data journalism won't be found on broadcast television or on the lists of the top journalists over the past century. They're drawn from the increasing pool of people who are building new breeds of newsrooms and extending the practice of computational journalism. They see the reporting that provisions their journalism as data, a body of work that can itself can be collected, analyzed, shared and used to create longitudinal insights about the ways that society, industry or government are changing. (Or not, as the case may be.)

In a recent interview, Emily Bell (@EmilyBell), director of the Tow Center for Digital Journalism at the Columbia University School of Journalism, offered her perspective about what's needed to train the data journalists of the future and the changes that still need to occur in media organizations to maximize their potential. In this context, while the role of institutions and "journalism education are themselves evolving, they both will still fundamentally matter for "what's next," as practitioners adapt to changing newsonomics.

Our discussion took place in the context of a notable investment in the future of data journalism: a $2 million research grant to Columbia University from the Knight Foundation to research and distribute best practices for digital reportage, data visualizations and measuring impact. Bell explained more about what how the research effort will help newsrooms determine what's next on the Knight Foundation's blog:

The knowledge gap that exists between the cutting edge of data science, how information spreads, its effects on people who consume information and the average newsroom is wide. We want to encourage those with the skills in these fields and an interest and knowledge in journalism to produce research projects and ideas that will both help explain this world and also provide guidance for journalism in the tricky area of ‘what next’. It is an aim to produce work which is widely accessible and immediately relevant to both those producing journalism and also those learning the skills of journalism.

We are focusing on funding research projects which relate to the transparency of public information and its intersection with journalism, research into what might broadly be termed data journalism, and the third area of ‘impact’ or, more simply put, what works and what doesn’t.

Our interview, lightly edited for content and clarity, follows.

What did you do before you became director of the Tow Center for Digital Journalism?

I spent ten years where I was editor-in-chief of The Guardian website. During the last four of those, I was also overall director of digital content for all The Guardian properties. That included things like mobile applications, et cetera, but from the editorial side.

Over the course of that decade, you saw one or two things change online, in terms of what journalists could do, the tools available to them and the news consumption habits of people. You also saw the media industry change, in terms of the business models and institutions that support journalism as we think of it. What are the biggest challenges and opportunities for the future journalism?

For newspapers, there was an early warning system: that newspaper circulation has not really consistently risen since the early 1980s. We had a long trajectory of increased production and actually, an overall systemic decline which has been masked by a very, very healthy advertising market, which really went on an incredible bull run with a more static pictures, and just "widen the pipe," which I think fooled a lot of journalism outlets and publishers into thinking that that was the real disruption.

And, of course, it wasn’t.

The real disruption was the ability of anybody anywhere to upload multimedia content and share it with anybody else who was on a connected device. That was the thing that really hit hard, when you look at 2004 onwards.

What journalism has to do is reinvent its processes, its business models and its skillsets to function in a world where human capital does not scale well, in terms of sifting, presenting and explaining all of this information. That’s really the key to it.

The skills that journalists need to do that -- including identifying a story, knowing why something is important and putting it in context -- are incredibly important. But how you do that, which particular elements you now use to tell that story are changing.

Those now include the skills of understanding the platform that you’re operating on and the technologies which are shaping your audiences’ behaviors and the world of data.

By data, I don’t just mean large caches of numbers you might be given or might be released by institutions: I mean that the data thrown off by all of our activity, all the time, is simply transforming the speed and the scope of what can be explained and reported on and identified as stories at a really astonishing speed. If you don’t have the fundamental tools to understand why that change is important and you don’t have the tools to help you interpret and get those stories out to a wide public, then you’re going to struggle to be a sustainable journalist.

The challenge for sustainable journalism going forward is not so different from what exists in other industries: there's a skills gap. Data scientists and data journalists use almost the exact same tools. What are the tools and skills that are needed to make sense of all of this data that you talked about? What will you do to catalog and educate students about them?

It's interesting when you say that the skills of these clients are very similar, which is absolutely right. First of all, you have a basic level of numeracy needed - and maybe not just a basic level, but a more sophisticated understanding of statistical analysis. That’s not something which is routinely taught in journalism schools but that I think will increasingly have to be.

The second thing is having some coding skills or some computer science understanding to help with identifying the best, most efficient tools and the various ways that data is manipulated.

The third thing is that when you’re talking about 'data scientists,' it’s really a combination of those skills. Adding data doesn’t mean you don't have to have other journalism skills which do not change: understanding context, understanding what the story might be, and knowing how to derive that from the data that you’re given or the data that exists. If it’s straightforward, how do you collect it? How do you analyze it? How do you interpret them and present it?

It’s easy to say, but it’s difficult to do. It’s particularly difficult to reorient the skillsets of an industry which have very much resided around the idea of a written story and an ability with editing. Even in the places where I would say there’s sophisticated use of data in journalism, it’s still a minority sport.

I’ve talked to several heads of data in large news organizations and they’ve said, “We have this huge skills gap because we can find plenty of people who can do the math; we can find plenty of people who are data scientists; we can’t find enough people who have those skills but also have a passion or an interest in telling stories in a journalistic context and making those relatable.”

You need a mindset which is about putting this in the context of the story and spotting stories, as well having creative and interesting ideas about how you can actually collect this material for your own stories. It’s not a passive kind of processing function if you’re a data journalist: it’s an active speaking, inquiring and discovery process. I think that that’s something which is actually available to all journalists.

Think about just local information and how local reporters go out and speak to people every day on the beat, collect information, et cetera. At the moment, most get from those entities don’t structure the information in a way that will help them find patterns and build new stories in the future.

This is not just about an amazing graphic that the New York Times does with census data over the past 150 years. This is about almost every story. Almost every story has some component of reusability or a component where you can collect the data in a way that helps your reporting in the future.

To do that requires a level of knowledge about the tools that you’re using, like coding, Google Refine or Fusion Tables. There are lots of freely available tools out there that are making this easier. But, if you don’t have the mindset that approaches, understands and knows why this is going to help you and make you a better reporter, then it’s sometimes hard to motivate journalists to see why they might want to grab on.

The other thing to say, which is really important, is there is currently a lack of both jobs and role models for people to point to and say, “I want to be that person.”

I think the final thing I would say to the industry is we’re getting a lot of smart journalists now. We are one of the schools where all of our digital concentrations from students this year include a basic grounding in data journalism. Every single one of them. We have an advanced course taught by Susan McGregor in data visualization. But we’re producing people from the school now, who are being hired to do these jobs, and the people who are hiring them are saying, “Write your own job description because we know we want you to do something, we just don’t quite know what it is. Can you tell us?”

You can’t cookie-cutter these people out of schools and drop them into existing roles in news trends because those are still developing. What we’re seeing are some very smart reporters with data-centric mindsets and also the ability to do these stories -- but they want to be out reporting. They don’t want to be confined to a desk and a spreadsheet. Some editors usually find that very hard to understand, “Well, what does that job look like?”

I think that this is where working with the industry, we can start to figure some of these things out, produce some experimental work or stories, and do some of the thinking in the classroom that helps people figure out what this whole new world is going to look like.

What do journalism schools need to do to close this 'skills gap?' How do they need to respond to changing business models? What combination of education, training and hands-on experience must they provide?

One of the first things they need to do is identify the problem clearly and be honest about it. I like to think that we’ve done that at Columbia, although I’m not a data journalist. I don’t have a background in it. I’m a writer. I am, if you like, completely the old school.

But one of the things I did do at The Guardian was helped people who early on said to me, “Some of this transformation means that we have to think about data as being a core part of what we do.” Because of the political context and the position I was in, I was able to recognize that that was an important thing that they were saying and we could push through changes and adoption in those areas of the newsroom.

That’s how The Guardian became interested in data. It’s the same in journalism school. One of the early things that we talked about [at Columbia] was how we needed to shift some of what the school did on its axis and acknowledge that this was going to be key part of what we do in the future. Once we acknowledged that that is something we had to work towards, [we hired] Susan McGregor from the Wall Street Journal’s Interactive Team. She’s an expert in data journalism and has an MA in technology in education.

If you say to me, “Well, what’s the ground vision here?” I would say the same thing I would say to anybody: over time, and hopefully not too long a course of time, we want to attract a type of student that is interested and capable in this approach. That means getting out and motivating and talking to people. It means producing attractive examples which high school children and undergraduate programs think about [in their studies]. It means talking to the CS [computer science] programs -- and, in fact, more about talking to those programs and math majors than you would be talking to the liberal arts professors or the historians or the lawyers or the people who have traditionally been involved.

I think that has an effect: it starts to show people who are oriented towards storytelling but have capabilities which are align more with data science skill sets that there’s a real task for them. We can’t message that early enough as an industry. We can’t message it early enough as an educator to get people into those tracks. We have to really make sure that the teaching is high quality and that we’re not just carried away with the idea of the new thing, we need to think pretty deeply about how we get those skills.

What sort of basic sort of statistical teaching do you need? What are the skills you need for data visualization? How do you need to introduce design as well as computer science skills into the classroom, in a way which makes sense for stories? How do you tier that understanding?

You're always going to produce superstars. Hopefully, we’ll be producing superstars in this arena soon as well.

We need to take the mission seriously. Then we need to build resources around it. And that’s difficult for educational organizations because it takes time to introduce new courses. It takes time to signal that this is something you think is important.

I think we’ve done a reasonable job of that so far at Columbia, but we’ve got a lot further to go. It's important that institutions like Columbia do take the lead and demonstrate that we think this is something that has to be a core curriculum component.

That’s hard, because journalism schools are known for producing writers. They’re known for different types of narratives. They are not necessarily lauded for producing math or computer science majors. That has to change.

Related:

Quantified me

2012-05-22T13:00:00Z

For some reason I have an aversion to the quantified self terminology. I guess I'm suspicious of excessive overt tracking of stuff that I hope to make into unconscious habit. It probably goes back to when I used to be a runner. I ran a couple of marathons and I would of course log every run and used upcoming races to motivate my training. I ran with a pulse monitor and used the real-time feedback to adjust my pace to the intention of each training session.

I was incredibly disciplined about my training right up until I stopped improving. Once I plateaued I just couldn't stick with it. I experienced a similar pattern with biking, rowing, yoga, and everything else I tried. Train hard, track everything, plateau, quit.

Then a few years ago I read about a study that looked at motivation and it made the point that sometimes leaving things open ended actually improves our ability to stick with it. I've been looking for that study for two years but can't find it again. It has stuck in my head though and fundamentally changed how I think about things. It's made much more skeptical of the value of competitions and other goals in achieving long-term fitness. And something is different for me now because I've been doing CrossFit for three years without quitting. Of course, it might just be that I haven't plateaued yet. But I also think nurturing an open-ended mindset has helped.

Having plateaued and quit so many times I guess I'm just skeptical of the value of tracking the minutia of my exercise life. I wouldn't have known I plateaued if I hadn't tracked the data after all.

So not too long ago when Sara Winge forwarded me a link to an article on the "datasexual" with the subject line "You've been memed" I was taken aback. "Me? I don't track stuff. I don't own a Fitbit. In fact, I'm a huge skeptic of the value of all this stuff. To me it seems too much like putting the cart of technology before the horse of just doing the work." But then I thought about it honestly and I had to admit it. Who am I kidding? I'm an obsessive tracker.

I track every Crossfit workout on Beyond The Whiteboard. I started a paleo / ancestral health diet in December and I use a kitchen scale to measure portions. I kept a journal of every meal for three months and when that got cumbersome I started taking a picture of them with my phone. I do it to encourage consciousness of what I'm eating and to make sure I'm keeping my macronutrient balance where it should be. I weigh myself at least three times each week and log weight, waist, and neck measurements each time to estimate body fat.

Not too long ago after I rowed what felt like a fast 2k during a crossfit workout I dug up my old logs from the '90s to see how it compared to the twenty-something me (slower of course, but not awful). I still had those logs and knew where to find them.

From there it gets more obsessive. Once I changed my eating habits I started getting a full lipid panel and other tests every three months to assess the impact of my new high fat / low carb diet (I get over 2/3 of calories from fats now). The next time around I plan to add tests for inflammation markers and a few other things.

I wasn't happy with my doctor only being able to order fasting blood sugar though, so I bought a glucometer and started monitoring my own real-time blood sugar. I measure fasting and +1, +2, and +3 hour postprandial glucose levels after various meals to evaluate my insulin response and to better tune my diet. I also occasionally measure pre- and post-workout glucose levels to optimize when to workout relative to mealtime.

Periodic at home A1c tests verify that my long-term glucose levels are in keeping with what I'm measuring in real time — as a correlation to verify test accuracy and to help me interpret the short-term results. Oh, and I ordered a 23andMe test kit to see (among other things) if I have any genetic disposition to diabetes.

So, I guess I have to admit it. Quantifying the self isn't just something other people do, it's something I do. Yet I remain a skeptic.

The line I'm trying to walk is between obsessive tracking that results in post-plateau burnout and using tracking to maintain awareness and intention while trying to remain open ended. "Maybe I'll work out today." "Maybe I'll lose a few pounds, or maybe I'll gain a few." But at the same time I want to take advantage of the awareness that comes from tracking. More importantly, I want to know what the data says about how healthy I am. A degradation in insulin response wouldn't just be a problem with a plateauing exercise program after all, it would have major long-term health impact.

Related:

Four short links: 22 May 2012

2012-05-22T10:00:00Z

New Zealand Government Budget App -- when the NZ budget is announced, it'll go live on iOS and Android apps. Tablet users get details, mobile users get talking points and speeches. Half-political, but an interesting approach to reaching out to voters with political actions.
Health Care Data Dump (Washington Post) -- 5B health insurance claims (attempted anonymized) to be released. Researchers will be able to access that data, largely using it to probe a critical question: What makes health care so expensive?
Perl 5.16.0 Out -- two epic things here: 590k lines of changes, and announcement quote from Auden. Auden is my favourite poet, Perl my favourite programming language.
WYSIHTML5 (GitHub) -- wysihtml5 is an open source rich text editor based on HTML5 technology and the progressive-enhancement approach. It uses a sophisticated security concept and aims to generate fully valid HTML5 markup by preventing unmaintainable tag soups and inline styles.