Then there the issue of self-discipline, writing every day, prioritising a long term project over all the short term possibilities for gratification, resisting the mood swings which make me want to tell everyone when I think I have a good idea and to give up the whole bloody thing when a concept seems to fall apart in my hands. Karen Blixen’s beautiful injunction to authors is ‘write every day, without hope, without despair’. Yes, but how?

I’m paid to run the RSA, not write books, and as it doesn’t come easy to me, I need to muster extra motivation. Being able to say my book got published is important but given the personal barriers, it’s not enough. I need to believe that the book will make one or both of two types of difference; benignly influencing public discourse (even if only at the margins and for a while) or getting some people interested in some stuff they might not have engaged with otherwise.

The intellectual problem with which I am wrestling goes to both types of impact. The argument that there are three fundamental drivers of social action, and that understanding them and the way they interact is critical to assessing the strengths and weaknesses of organisations and strategies, must both hold water with people who think about this kind of stuff but also resonate with a thoughtful general reader.

Imagine piles of plates. The plates in each pile are of different sizes, some fit neatly together, others less so and some plates cross over into two piles. But, still, if you stand back it is clear that there are three different piles; if you were describing them or thinking about how you might build them up and break them down it would be the ‘threeness’ of the arrangement that would be the most useful starting point, especially if you wanted somehow to explore building a platform across them all. This is what I am trying to prove to myself.

The potential book itself is not simply, or mainly, a description of the conceptual piles (which is by the way a painful condition from which I frequently suffer). It is about how the best solutions are built across all three, about how and why in most circumstances this isn’t possible because the piles are different sizes or they are leaning away from each other, and about what we need to do right now in Britain to grow and shape those piles so that there is the basis for a platform across them. But unless the piles themselves make sense none of the rest of it works.

Sorry to labour the metaphor but if you were to build a pile of plates you would presumably start with the widest and toughest ones as the base. My hunch is that most robust starting point is three voices in our head: ‘I’ll do what I’m told’. ‘I’ll do what I want to do’ and ‘I’ll do what the group needs me to do’.

The next layer might be to trace the evolutionary foundation of these voices in leadership and followership, survival and reciprocity and communality (particularly group parenting).  From there we might a layer relating to a behavioural disposition; obedience, ambition, and responsibility. Moving to the social level, the next layer might comprise the social expression of each pillar of ideas; respectively, hierarchy, market, and community. Then there are the forms of rationality which can be added in turn; bureaucratic, individualist and solidaristic.

In my last post I overlaid the core values of the enlightenment, which can be seen as powerful ways of both legitimating and shaping core human drives; the belief in progress aligning with hierarchy, autonomy with individualism and universalism with group identity. Even more tentatively, how about the Freudian personality triptych; ego as the conventional responses we learn as children through the authority of adults, id as the primal drive of self-gratification and super-ego as the call of social responsibility?

But as the piles rise, so they start to wobble. The principle of universalism, for example, is on the one hand, pursued in the modern world primarily thought the expansion of individual rights and, on the other, implies a common bond of humanity in contrast to ‘solidarity’ which usually expressed in terms of bounded group membership.

The higher I build the piles of conceptual plates the more comprehensive the theory and the more it links big social trends to individual predispositions which any reader can recognise, but also the more prone to collapse is the whole schema.

Should I keep it simple, should I be ambitious or should I give up the whole endeavour?

A few weeks ago I was talking to a talented young MP who was telling me of his plans to leave politics. ‘I’m bored’ he said ‘I don’t believe in politics anymore and, truth be told,’ he said, as he cast his eyes around the atrium at Portcullis House ’nor do most of us’. A Conservative MP said to me that his Party’s obsession with Europe is partly based on genuine problems with the EU, partly on the divisive Thatcher legacy but also ‘because at last people have got something in politics to get passionate about’.

Is there anything that can be done? Regardless of party affiliation is there a message or strategy that could lift politics out of the doldrums and offer a credible policy platform. As a purely academic exercise I’ve had a go.

The rather obscure starting point is this table

In the columns I have merged two frames of analysis. The first is from cultural theory and its three ubiquitous ways of thinking about and seeing change: the individualistic, the hierarchical and the solidaristic (NB there is also another passive form – the fatalistic).

The second uses the three core ideas of the Enlightenment as identified by Tzvetan Todorov: universalism (which I translate as fairness), humanism (the belief in progress defined by greater human flourishing) and autonomy (freedom).

These two sets of concepts (which have been the focus respectively of two of my annual lectures), one about ideas the other about frames for seeing and acting, don’t map neatly onto each other. ‘Universalism’ asserts the equal rights of all people (although in the Enlightenment this actually meant well-off white men) whereas ‘solidarity’ is about the interests of those defined as in the group while ‘fairness’ stands somewhere between and also brings in notions of just desserts. Also, hierarchies can be ambivalent about progress (think of the Roman Catholic Church). It is also important not to assume that individualism and solidarism line up on a right left axis. Nationalism and even racism can be as much expressions of in-group solidarity as trade unionism and – as the Commons debate yesterday emphasises – social conservatives don’t think individual rights trump all other considerations.

Nevertheless, for the purposes of my argument, there is sufficient overlap based on clustering of ideas and feelings around three fundamental human instincts (individual survival, desire and ambition; kinship and social belonging; and leadership/followership).

In each box there is a number. The numbers represent a view of the relative power in contemporary society of each way of thinking in relation to, first, their account of the ends to which change should be aiming, second, the means which should be used to achieve that change, third, the cultural milieu in which change takes place.

The argument is that hierarchy is most powerful when it comes to the task of describing the ends of change. To put it simply, we need leaders to tell us where to march towards. Solidarity/fairness is less powerful when to comes to ends partly because it is often a rather nostalgic or defensive expression, and also because its definition of the good society (whether it’s socialist utopianism or nationalist nostalgia) tends to sound rather pie in the sky. The perspective of individualism leads to a suspicion of the whole idea of purposive ends – where we get will depend on our choices with the hidden hand of the market ensuring progress.

When it comes to means – the method of change – individualism has the strongest hand; a position which is even more pronounced in today’s complex world. If we could achieve change simply by people doing what they feel to be in their own interests things would be pretty simple, which is why markets can be so fast moving and innovative in comparison to bureaucracies (hierarchy) or collectives (solidarity).

Finally, in terms of cultural mores, while individualism offers only the shallowness of consumerism and hierarchies tend to be fractured both horizontally and vertically, it is the domain of solidarity that provides meaning and belonging for us social beings.

Therefore, in modern society, the most effective model of change will involve leaders defining ends which inspire, using means which achieve those ends through alignment with individual choices, with processes of change being humanised by a culture resonant with solidarity and mutual obligation. To use a mechanical metaphor, hierarchy sets the direction, individualism provides the engine and solidarity is the oil.

How does this translate into a political programme? First, we need a new type of leadership (I have described this here). Second, we need both to draw on individual aspiration as the most effective means to change while also helping people to aspire to what will genuinely provide fulfilment not simply what advertisers tell them they should want: freedom to produce, create and connect rather than freedom merely to consume and possess. Finally, we need to find ways of generating new forms of solidarity and mutual obligation in a diverse, fast changing world.

Politicians are often told by their advisors to try to ‘own the future’. This means rather than simply hypothesising a better world, describe the things already happening that prefigure that world and then show how you are the person to liberate the forces of progress and remove the barriers to change.

As I said in my last post commenting on the possible end of the social recession, there is enough material to start to describe what we might mean by a responsible modern society. Also, with the growth of interest and engagement in enterprise (traditional, social and micro) we can begin to see the outlines of a new venturesome economy.

Sadly, when it comes to new type of leadership, at least in the political sphere, I can only refer back to my opening sentence.

Finlayson’s piece rests on a cast iron assumption that there is a set of social problems which are getting worse, the important question therefore is ‘why?’. He quotes a 2007 Joseph Rowntree Foundation project in which the RSA and I were involved called ‘social evils’ which explored the way society was deteriorating even while the economy was thriving. Finlayson concludes that with the economic collapse the debate over what lies behind social deterioration will intensify.

The posssibility the article didn’t consider, and is rarely discussed, is that in certain important respects the social recession might be coming to an end. In this post I don’t have time to gather all the references but, take my word for it, there is reasonable evidence for significant improvements in all the following areas:

Crime rates

Violent crime rates

Teenage pregnancy

Obesity

Binge drinking and drug consumption among young people

Levels of volunteering and feelings of neighbourliness

Children’s overall wellbeing

Why aren’t we discussing this more? The idea that society is improving because we are choosing to behave more wisely and responsibly is uncomfortable for parts of both the left and the right. For the former it shouldn’t be happening in an unequal society still dominated by individualistic values. For the conservative right it shouldn’t be taking place in a society coarsened by moral relativism and weakened by diversity and multiculturalism.

It is possible to counter the improving data by saying it is short term, inconclusive or outweighed by things going in the wrong direction. But if we were to accept that society is getting less broken, what explanations might be on offer?

There was recently a spate of articles based on epidemiological research speculating that falling levels of lead in the atmosphere are the best explanation for plummeting crime rates. A factor endogenous to social trends may indeed be implicated. But I am more drawn to a more structural explanation.

In  policy, economic and social analysis the single ability of greatest value is to be able to distinguish a cycle from a trend. It is hard analytically and challenging psychologically. Not only are human beings inherently short-termist in their outlook but we are drawn to things that are more visible. Cycles are like the second hand on a clock – we can see them moving which gives us a sense of time passing, while trends are like the hour hand.

Here is a bold thesis. Since the dawn of the enlightenment in the late mediaeval period – what Kant described as man entering into adulthood – human progress has accelerated guided by the core principles of that revolution – universalism (justice), autonomy (freedom) and humanism (progress itself). These principles underlie the long trend of human advance which makes citizens in the developed world richer, healthier, more intelligent, more tolerant and more peaceful than ever before. But within that trend the misapplication of those same principles has also led to terrible cycles, most obviously the cycle of colonial exploitation which culminated in the nightmare of the First World War and the cycle of totalitarian ideology which led to the horror of Nazi Germany, Maoist China and Stalinist Soviet Union. At the heart of these terrible events was a hubristic perversion of the idea of progress.

The recent cycle, which began in the sixties and may now be startng to end, also involved a misapplication of an enlightenment principle  – this time freedom – and was much milder in comparison. But the idea that society can flourish relying on no more than individuals pursuing a policy of possessive individualism is at last starting to lose favour. Is this what lies behind the evidence of a receding social recession?

If, in the end, society learns and improves should we focus less on cycles of deterioration and more on long trends of progress? If so, it shifts the debate in a subtle but, to my mind, crucial way.

Instead of asking what we have to do to make progress possible we should instead ask what the barriers are to allowing the further natural development of the human spirit into a higher, healthier, more fulfilled and rounded form. The answers might be similar but the framing of the question has the scope to increase significantly our sense of possibility and agency.

The obvious charges against this thesis are that it is determinist and complacent. But I don’t think society improves automatically. It happens through struggle and debate. And the other key domains of our lives – economics and politics – sometimes accelerate social progress and sometimes delay or reverse it. Right now I feel more confident about society than about economics and very worried about politics – for reasons I will explore in a future post if this one doesn’t get ripped to shreds.

There are several reasons to question whether PBR for probation will deliver:

All PBR systems are subject to the inherent tendency for providers to ‘skim’ and ‘park’. The former process involves identifying those in the client group most likely to achieve an outcome (in this case not reoffending) with very little support. The provider then gets paid for something which would have happened anyway without having to add any value (economists call this ‘rent seeking’) and the taxpayer picks up the dead weight cost.

Parking involves limiting the service provided to people who are highly unlikely to meet the outcome and in whom it is therefore cost inefficient for providers to invest resources. The latter phenomenon leads to one of the classic debates in the design of PBR – one which has been rolling around the probation debate – should payment simply be for the result (not re-offending) or should there also be payments for what is often called ‘direction of travel’ – things like attending life skills courses or entering drug treatment. The case for direction of travel payments is that it incentivises providers to do something for the hardest to help; the case against is that it provides scope for gaming and the proliferation of ineffective interventions, such as payments for members of troubled families to attend parenting classes of questionable efficacy.

A particular risk for probation arises from the private/third sector PBR system running in parallel with the state probation service, which will continue to be responsible for more dangerous ex-offenders. The temptation for new providers to dump harder to help clients back on to the probation service will be strong. And while collaboration between providers is essential ,regulating that boundary is likely to be a continual source of tension on the ground.

Given the weak results so far from the Work Programme it is a bold claim that PBR will deliver better outcomes and provide a wholly new service for short stay ex-offenders all within a declining real terms budget.  Some organisations are being very bullish about the possibilities for them to provide great services in the new system, but similar big ambitions articulated by putative Work Programme providers went largely by the board when, under pressure from the Treasury, the contracts were ultimately awarded almost entirely on the basis of price.

Third sector organisations looking to be PBR probation providers should heed the five (yes, five) harsh lessons of the Work Programme.

* Charities, lacking the contract planning and negotiation capacity of the private sector, generally failed to get on the original framework of prime providers. Many wasted tens of thousands of pounds in failed bids.

* Many charities which subsequently entered subcontracting arrangements with primes then felt a lot of pain – in many cases terminal – as the expected and promised client referral rate failed to materialise.

* Relatedly, the PBR mechanism means the contracts have what is sometimes called a ‘smile’ profile. That is, providers get some money up front as an attachment fee but then have to pay for services and make a revenue loss before later in the contract starting to get payments for meeting outcomes. Lacking reserves or access to finance many third sector providers are simply unable to ride out the downswing of the smile.

* The PBR margins on offer to the third sector are anyway very modest on contracts which were already tight even before the prime providers had taken their cut. Many third sector providers are in effect subsidising their provision with charitable income (something which raises difficult issues of propriety and transparency).

* Many third sector providers who thought they were doing a good job found that they were unable to cope with the monitoring and performance requirements of the  various PBR employment  programmes (indeed across the sector third sector providers are having regularly to hand back contacts). Whether this means the system is badly and unfairly designed or that lots of third sector provision is not actually any good is, of course, bitterly contested.

A third problem with PBR as a mechanism for providing public services is the flip side of a claimed advantage. When advocating the Work Programme PBR ministers were apt to talk positively about the ‘black box’ of service delivery. Rather than the over regulated and over managed state services providers would be free to use whatever mechanism works best to achieve the outcome. As one advisor put it to me ‘if getting people to stand on their head and read poetry gets them back into work then what’s the problem’? Fair enough; but the black box in combination with competitive pressure means there is also little incentive for providers to share information, compare practice and improve. ERSA (the trade body for employment programme providers) has been working with Nesta and the RSA to encourage greater information sharing and innovation in the sector but it is not easy to convince providers they should share trade secrets, or even agree a common framework of professional standards.

Most of these issues are generic to PBR as a mechanism. Indeed I have had some rather worrying conversations with civil servants who seem unaware that issues like this are not random risks but core characteristics of such systems, whatever their other advantages. But in probation there is one other major issue.

If a work programme provider fails, someone continues to be unemployed but if probation supervision fails, an ex offender may commit a serious crime. The Work Programme and other employment programmes such as Mandatory Work Activity have been subject to much critical media scrutiny but this is as nothing to what will happen if a badly supervised private sector probation client murders someone. I simply have no idea how a private provider could factor in such massive reputational risks to the cost benefit equation of bidding.

The basic idea of payment by results whereby taxpayers pay for outcomes rather than processes is powerful, especially in a time of austerity, and for a number of reasons PBR is likely to grow. As the chair of the board of a smallish employment service company, I can see both how good practice can deliver powerful results for disadvantaged people but also how tough it is to make money (something which should reassure the public).

The potential upside of PBR makes it all the more important that we understand its inherent pitfalls.

Our goal must be to create a new care covenant between citizens and the state, involving all parts of our society and economy.  This covenant should be based on the principle that care is a shared responsibility – its risks and benefits are mutual and should not be left to individuals and families to shoulder alone.  It must be grounded in values of dignity and respect for those who need and provide care, ensuring older and disabled people and their families have the same choices and chances to live the lives they want as any other section of society.

And it must seek to strengthen mutual care and support within our families and communities, as well as through care services

Fine words, as my grandmother used to say, butter no parsnips. But it is good to see an emphasis being put on values, mutual care and shared responsibility as well as Government policy. As I have argued here before, we need a ‘whole system’ reform of care – something which involves attitudes, behaviours and norms as well as policies.

Yesterday I interviewed the philosopher Michael Sandel for the next edition of the RSA Journal. We discussed his book ‘what money can’t buy’ which has just come out in paperback. In it he bemoans the ‘marketisation’ of more and more areas of life evidenced, for example, in poor children being bribed to study or sports memorabilia being turned from personal objects collected by fans to a billion dollar mass market. His objection to such practices lies in the way they exacerbate inequality.

The more things that are paid for, the more difference having money makes to people’s lives. Take the example of ticketing. If the criterion for access is a willingness to queue early or long, the tickets will tend to go to the most passionate, but if the rich can buy tickets whenever they want from on-line sites, wealth rather than passion becomes crucial.

Sandel also worries about the way financial incentives ‘corrupt’ motives and meanings. If we bribe children to read, do we reinforce the idea that reading is onerous? If we can pay for a footballer’s autograph, it lacks the intrinsic value which we would attach to something we had collected ourselves in a memorable encounter or just standing in the rain for hours.

Bringing together Michael’s analysis and my interests, we had a fascinating conversation about care. This is surely the most important example of the way the value of something changes when it is marketised. The very idea of care contains within it a notion of emotional commitment. Yet, however much we might teach nurses how to be compassionate, we cannot get inside the heart of someone who is paid to care and force them to experience the feelings that we have when we care for a loved one and which we want to believe lead loved ones to care for us.

Michael reminded me of a famous article from 2001 by Arlie Hochschild called ‘The Nanny Chain’ which ponders what we are doing to people and relationships when immigrant carers are paid to look after middle class children and use their remittances to pay a different stranger to look after their own children back home.

In terms of Sandel’s idea of corruption, consider the contrast between the value we place on care in the private and public domain. In the former we say that nothing matters more than the welfare of our loved ones. We give and receive care as part of a web of reciprocity which goes towards defining who we are as social beings. But in the market, care is low status and low paid. Might it be that the low value ascribed to paid care might corrupt the status of voluntary and familial care, adding a loss of self-worth and status (and influence) to the other challenges of being a carer?

To argue that the socialisation and marketisation of care may be problematic is difficult. It might sound like a suggestion that there was a golden past of unlimited compassion, or a failure to appreciate the radically different conditions created by population ageing and female employment, or – worst of all – an attempt to force care back into the home and onto the shoulders of unpaid women.

Politicians are under pressure to come up with solutions. Labour is putting its emphasis on the need to join up social and health care commissioning and perhaps underestimating the reasons why this is very hard to do. But no reconfiguration of spending or services is going to be enough to resolve the care crisis. Liz Kendall is right: we must explore care as part of our lives and our society as well as a particularly intractable set of funding and policy problems.