Systems Thinking for Demanding Change

Alphabetic Warning

2025-11-18T08:19:00.001+00:00

In an interview with the BBC, Sundar Pichai, CEO of Alphabet, advises people to be cautious of what AI tells them, suggesting that AI should be combined with other tools such as, er, Google search (owned by Alphabet).

But if we shouldn't blindly trust what AI tells us, the same advice surely applies to Google search as well. The question Can we trust Google? was raised on this blog over twenty years ago, and remains relevant today.

There are also important questions of context and perspective. People are increasingly aware that the results they get from generative AI is highly dependent on how the enquiry is worded, hence the new discipline of prompt engineering. But the same is also true of search engines, as I discussed with David McCoy back in 2008. In his comment below my blogpost, You don't have to be smart to search here ..., David recommended what he called speculative cunning.

One advantage of AI over search is that it offers much better levels of context-awareness (depending on the prompts). As I noted previously, standardized search systems ... will give you exactly the same answer whether you are a schoolchild or a BBC researcher or an LSE postgraduate. Someone at Google previously told the BBC about their Quest for the Perfect Search. But perhaps a perfectly personalized answer may have the effect of seducing the user into an unwarranted level of trust.

Mr Pichai acknowledges that there are ethical issues here, saying that Alphabet aims to be bold and responsible at the same time.

Faisal Islam and Rachel Clun, Don't blindly trust what AI tells you, says Google's Sundar Pichai (BBC, 18 November 2025)

Related posts Trusting Commons (March 2005), Context-Aware Services (January 2006), You don't have to be smart to search here ... (November 2008), The Force of Goole (April 2016)

How Soon Might Humans Be Replaced At Work

2025-07-06T20:31:00.005+01:00

As noted by Thomas Claburn in The Register, there seems to be a contradiction between two pieces of research relating to the development and use of AI in business organizations.

On the one hand, teams of researchers have developed benchmarks to study the effectiveness of AI, and have found success rates between 25% and 40%, depending on the situation.

On the other hand, Gartner reports that business executives are expecting a success rate nearer to 60% - if we interpret not-being-cancelled as a marker for success. More than 40 percent of agentic AI projects will be cancelled by the end of 2027 due to rising costs, unclear business value, or insufficient risk controls.

History tells us that the adoption of technology to perform work is only partially dependent on the quality of the work, and can often be driven more by cost. The original Luddites protested at the adoption of machines to replace textile workers, but their argument was largely based on the inferior quality of the textiles produced by the machines. It was only later that this label was attached to anyone who resisted technology on principle.

Around ten years ago, I attended a debate on artificial intelligence sponsored by the Chartered Institute of Patent Agents. In my commentary on this debate (How Soon Might Humans Be Replaced At Work?) I noted that decision-makers may easily be tempted by short-term cost savings from automation, even if the poor quality of the work results in higher costs and risks in the longer term.

In their look at the labour market potential of AI, Tyna Eloundou et al note that

A key determinant of their utility is the level of confidence humans place in them and how humans adapt their habits. For instance, in the legal profession, the models’ usefulness depends on whether legal professionals can trust model outputs without verifying original documents or conducting independent research. ... Consequently, a comprehensive understanding of the adoption and use of LLMs by workers and firms requires a more in-depth exploration of these intricacies.

However, while levels of confidence and trust can be assessed by surveying people's opinions, such surveys cannot assess whether these levels of confidence and trust are justified. Graham Neubig told The Register that this was what prompted the development of a more objective benchmark for AI effectiveness.

Thomas Claburn, AI agents get office tasks wrong around 70% of the time, and a lot of them aren't AI at all (The Register, 29 June 2025)

Thomas Claburn, AI has had zero effect on jobs so far, says Yale study (The Register, 1 October 2025)

Tyna Eloundou, Sam Manning, Pamela Mishkin and Daniel Rock, GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (August 2023)

Wikipedia: Luddite

Related Posts: How Soon Might Humans Be Replaced At Work? November 2015, RPA - Real Value or Painful Experimentation? (August 2019), Explaining Layoffs (October 2025)

Machine Indoctrination

2025-03-20T08:36:00.003+00:00

March 2013

From my post on Enabling Prejudices

One of the key insights of the early work on Design Thinking (Bryan Lawson, Peter Rowe) was the importance of heuristics, or what Rowe (following Gadamer) calls enabling prejudices, which will hopefully get us to a good-enough solution more quickly.

As Christopher Alexander notes:

At the moment when a person is faced with an act of design, he does not have time to think about it from scratch. The Timeless Way of Building p 204

We always approach a problem with a set of prejudices or prejudgements. Depending on the situation, these may either help us to solve the problem more quickly (enabling), or may lead us astray (disabling). The acid test of a set of heuristics or design principles is that they are mostly enabling most of the time.

March 2025

Design schools indoctrinate their students in ways of solving design problems efficiently. If you are trained to follow a particular design approach - for example Bauhaus - this greatly reduces the complexity of the task, because it rules out a vast number of solutions that would be anathema to a Bauhausian.

Algorithms may be able to calculate vastly larger sets of options than a human designer, but as Adam Nocek explained yesterday in a talk at Goldsmiths, machine intelligence is subject to mathematical limitations on computability. I touched on this topic in my guest editorial for a journal special issue on Algorithms in 2023, but his argument was much more comprehensive and wide-ranging, linking to important questions of agency and subjectivity.

Many researchers have noted the prevalence of algorithmic bias, but if we accept the importance of heuristics and intuition in the design process, there are much more fundamental problems here.

to be continued ...

Christopher Alexander, The Timeless Way of Building (New York: Oxford University Press, 1979)

Dan Klyn, Skirmishing With Ill-Defined and Wicked Problems (TUG, 5 July 2013) - review of Rowe

Bryan Lawson, How Designers Think (1980, 4th edition 2005)

Adam Nocek, Designing in the age of artificial machines and Whitehead (Talk at Goldsmiths University, 19 March 2025)

Peter Rowe, Design Thinking (MIT Press 1987)

Richard Veryard, As we may think now (Subjectivity 2023)

Related posts: From Sedimented Principles to Enabling Prejudices (March 2013), From Enabling Prejudices to Sedimented Principles (March 2013), Limitations of Machine Learning (July 2020)

Influencing the Habermas Machine

2024-11-03T14:47:00.007+00:00

In my previous post Towards the Habermas Machine, I talked about a large language model (LLM) developed by Google DeepMind for generating a consensus position from a collection of individual views, named after Jürgen Habermas.

Given that democratic deliberation relies on knowledge of various kinds, followers of Habermas might be interested in how knowledge is injected into discourse. Habermas argued that mutual understanding was dependent upon a background stock of cultural knowledge that is always already familiar to agents. but this clearly has to be supplemented by knowledge about the matter in question.

For example, we might expect a discussion about appropriate speed limits to be informed by reliable or unreliable beliefs about the effects of a given speed limit on journey times, accident rates, pollution, and so on. In traditional discussion forums, it is extremely common for people to present themselves as having some special knowledge or authority, which supposedly gives extra weight to their opinions, and we might expect something similar to happen in a tech-enabled version.

For many years, the Internet has been distorted by Search Engine Optimization (SEO), which means that the results of an internet search are largely driven by commercial interests of various kinds. Researchers have recently raised a similar issue in relation to large language models, namely Generative Engine Optimization (GEO). Meanwhile, other researchers have found that LLMs (like many humans) are more impressed by superficial jargon than by proper research.

So we might reasonably assume that various commercial interests (car manufacturers, insurers, oil companies, etc) will be looking for ways to influence the outputs of the Habermas Machine on the speed limit question by overloading the Internet with knowledge (regime of truth) in the appropriate format. Meanwhile the background stock of cultural knowledge is now presumably co-extensive with the entire Internet.

Is there anything that the Habermas Machine can do to manage the quality of the knowledge used in its deliberations?

Footnote: Followers of Habermas can't agree on the encyclopedia entry, so there are two rival versions.

Footnote: The relationship between knowledge and discourse goes much wider than Habermas, so interest in this question is certainly not limited to his followers. I might need to write a separate post about the Foucault Machine.

Pranjal Aggarwal et al, GEO: Generative Engine Optimization (arxiv v3, 28 June 2024)

Callum Bains, The chatbot optimisation game: can we trust AI web searches? (Observer, 3 November 2024)

Alexander Wan, Eric Wallace, Dan Klein, What Evidence Do Language Models Find Convincing? (arxiv v2, 9 August 2024)

Stanford Encyclopedia of Philosophy: Jürgen Habermas (v1 2007) Jürgen Habermas (v2 2023)

Towards the Habermas Machine

2024-10-19T11:48:00.008+01:00

Google DeepMind has just announced a large language model, which claims to generate a consensus position from a collection of individual views. The name of the model is a reference to Jürgen Habermas’s theory of communicative action.

An internet search for Habermas machine throws up two previous initiatives under the same name. Firstly an art project by Kristopher Holland.

The Habermas Machine (2006–2012) is a conceptual art experience that both examines and promotes an experiential relation to Jürgen Habermas’ grand theory for understanding human interaction. The central claim is that The Theory of Communicative Action can be experienced, reflected upon and practised when encountered within arts-based research. Habermas’ description of how our everyday lives are founded by intersubjective experience, and caught up in certain normative, objective and subjective contexts is transformed through the method of conceptual art into a process of collaborative designing, enacting and articulating. This artistic reframing makes it possible to experience the communicative structure of knowledge and the ontological structure of intersubjectivity in a practice of non-discursive ‘philosophy without text’. Feiten Holland Chemero

And secondly, an approach to Dialogue Mapping described as a device that all participants can climb into and converse with complete communicative rationality, contained in a book by @paulculmsee and Jailash Awati, and mentioned in this Reddit post Why is Dialogue Mapping not wide spread? Dialogue Mapping was developed by Jeff Conklin and others as an approach to addressing wicked problems. See also Issue Based Information Systems (IBIS).

Update

Christopher Summerfield, one of the authors of the DeepMind paper, spoke at the Royal Society on October 29th 2024. https://www.youtube.com/live/cW1Wq7_8v1Y?si=oqo8Lw7479x4QqKt&t=18890

All the examples shown in his talk were policy matters that could be reduced to Yes/No questions. Such questions would traditionally be surveyed by asking people to place themselves on a scale from Strongly Agree to Strongly Disagree, and it is easy to see how a language-based method such as the Habermas Machine offers some advantages over a numerical scale. But not clear how this works for more provocative questions, let alone wicked problems.

Someone in the audience asked if this method would work in what he called a compromised democracy, and Summerfield acknowledged that the method assumes what he called a good faith scaffold. Obviously all democracies in the real world are imperfect, and he didn't go into the question as to how sensitive or vulnerable the method might be to such imperfections, but the method might conceivably help to overcome some of these imperfections under some conditions: for example, Summerfield referred specifically to the tyranny of the majority.

While the performance of the Habermas machine in their study compared favourably with the performance of human mediators, Summerfield suggested that we should move away from thinking about AI in these terms. The point is not to create AI-based agents that can behave like intelligent people but to build intelligent institutions - tools for creating social order and fostering cooperation. As my regular readers will know, orgintelligence has long been an important theme for this blog. See for example my post On Organizations and Machines (January 2022).

Jeffrey Conklin, Dialogue Mapping: Building Shared Understanding of Wicked Problems (Wiley 2006). See also CogNexus website.

Paul Culmsee and Jailash Awati, The Heretic's Guide to Best Practice (2013)

Nicola Davis, AI mediation tool may help reduce culture war rifts, say researchers (Guardian, 17 October 2024)

Tim Elmo Feiten, Kristopher Holland and Anthony Chemero, Doing philosophy with a water-lance: art and the future of embodied cognition (Adaptive Behavior 2021)

Michael Tessler et al, AI can help humans find common ground in democratic deliberation (Science, 18 October 2024)

Beyond the symbols vs signals debate (The Royal Society, 28-29 October 2024)

Wikipedia: Issue Based Information Systems (IBIS), Wicked Problem

See also Influencing the Habermas Machine (November 2024)

All our eyes on the disgraceful Horizon

2024-06-06T08:04:00.002+01:00

The scandal at the British Post Office, details of which are now emerging in the Public Enquiry, provides illustrations of many important aspects of organizational behaviour as discussed on this blog.

Willful blindness. There is a strong attachment to a false theory, despite mounting evidence to the contrary, as well as the appalling human consequences.

Misplaced trust. Trusting a computer system (Horizon) above hundreds of ordinary people. And both the legal system and government ministers trusting the evidence presented by a public corporation, despite the fact that contrary evidence from expert witnesses had been accepted in a small number of cases (see below).

Defensive denial as one of the symptoms of organizational stupidity. In July 2013, Post Office boss Paula Vennells was told about faults in the Horizon system, and advised that denying these would be dangerous and stupid. This is something the Post Office had denied for years. ITV March 2024

A detail that struck me yesterday was a failure to connect the dots. In 2011, the auditors (EY) raised concerns about data quality, warning that if Horizon was not accurate, then they would not be able to sign off Post Office company accounts. Ms Perkins, who was giving evidence at an inquiry into the scandal, said at the time she did not make a link between the two. BBC June 2024. The pattern I'm seeing here is of assuming the sole purpose of audit as satisfying some regulatory requirement, with zero operational (let alone ethical) implications of anything the auditors might find. And assuming the regulatory requirement itself to have no real purpose, being merely a stupid and meaningless piece of bureaucracy.

Another failure to connect the dots occurred after Julie Wolstenholme successfully challenged the Post Office in 2003 with the aid of an expert technical witness. Why didn't this prompt serious questions about all the other cases? When asked about this at the enquiry, David Mills said he had not properly assimilated the information and pleaded lack of intelligence, saying I wasn’t that clever. I’m sorry, I didn’t ask about it. ITV April 2024

In my other pieces about organizational intelligence, I have noted that stupid organizations may sometimes be composed of highly intelligent people. Now that's one pattern the Post Office doesn't seem to illustrate. Or have the Post Office bosses merely chosen to present themselves as naive and incompetent rather than evil?

Tom Espiner, Ex-Post Office chair was told of IT risks in 2011 (BBC 5 June 2024)

ITV, Secret tape shows Paula Vennells was told about problems with Horizon and warned not to cover it up (29 March 2024)

ITV, Former Post Office boss tells inquiry he was not 'clever' enough to question Horizon IT system (16 April 2024)

Other Sources: Post Office Horizon IT Enquiry, British Post Office scandal (Wikipedia), Post Office Project (University of Exeter)

Thinking Academically

2024-05-31T22:27:00.010+01:00

At Goldsmiths University yesterday for a discussion on Paratactical Life with Erin Manning and Brian Massumi. Academic jobs at Goldsmiths are currently threatened by a so-called Transformation Programme, similar to management initiatives at many other universities, giving critical urgency for those in the room to consider the primary task of the university in society, and the double task of the academic. For which Erin Manning advocates what she calls strategic duplicity.

This involves recognizing what works in the systems we work against. Which means: We don't just oppose them head on. We work with them, strategically, while nurturing an alien logic that moves in very different directions. One of the things we know that the university does well is that it attracts really interesting people. The university can facilitate meetings that can change lives. But systemically, it fails. And the systemic failure is getting more and more acute. Todoroff

One of the domains in which this duplicity is apparent is thinking itself. And this word thinking appears to have special resonance and meaning for academics - what academia calls thinking is not quite the same as what business calls thinking (which was the focus of my practitioner book on Organizational Intelligence) and certainly not the same as what tech calls thinking (the focus of Adrian Daub's book).

One of the observations that led to my work on Organizational Intelligence was the disconnect between the intelligence of the members of an organization and the intelligence of the organization itself. Universities are great examples of this, packed with clever people and yet the organization itself manifests multiple forms of stupidity. As of course do many other kinds of organization. I still believe that it is a worthwhile if often frustrating exercise to try to improve how a given organization collectively makes sense of and anticipates the demands placed on it by its customers and other stakeholders - in other words, how it thinks. However any such improvements would be almost entirely at the micropolitical level, I don't have much idea how one would go about dismantling what Deleuze calls the economy of stupidity.

Although I think the concept of organizational intelligence is a reasonable one, and have defended it here against those who argue that organizational functions and dysfunctions can always be reduced to the behaviours and intentions of individual human actors, I don't imagine that an organization will ever think in quite the way a person thinks. There are some deficiencies in organizational thinking, just as there are deficiencies in algorithmic thinking. For example, there are some interesting issues in relation to temporality, raised in some of the contributions to Subjectivity's Special Issue on Algorithms which I guest-edited last year.

For Brian Massumi, the key question is what is thinking for. In an academic context, we might imagine the answer to be something to do with knowledge - universities being where knowledge is created and curated, and where students are supposed to acquire socioeconomic advantage based on their demonstrated mastery of selected portions of this knowledge. Therefore much of the work of an academic is taken up with a form of thinking known as judgment or sorting out - deciding, agreeing and explaining the criteria by which students will be evaluated, using these criteria to assess the work of each student, and helping those students who don't fit the expected pattern for whatever reason.

But what really gives a student any benefit in the job market as a result of their studies is not just a piece of paper but a sense of their potential - for both thinking and doing. The problem with students using chatbots to write their assignments is not that they are cheating - after all, the ability to cheat without being found out is highly valued in many organizations, if not essential. The real problem is if they are learning a deficient form of thinking.

(This is far from a complete report on the afternoon, merely picking out some elements of the discussion that resonated with me.)

Update: Comments have been added to the goodreads version of this post.

Philip Boxer, The Three Asymmetries necessary to describing agency in living biological systems (Asymmetric Leadership, November 2023)

Philip Boxer, The Doubling of the Double Task (Asymmetric Leadership, February 2024)

Adrian Daub, What Tech Calls Thinking (Farrar Straus and Giroux, 2020)

Benoît Dillet, What Is Called Thinking?: When Deleuze Walks along Heideggerian Paths (Deleuze Studies 7/2 2013)

Kenan Malik, The affluent can have their souls enriched at university, so why not the poor as well? (Observer, 2 June 2024)

Brent Dean Robbins, Joyful Thinking-Thanking: A Reading of Heidegger’s “What is Called Thinking?” (Janus Head 13/2, October 2014)

Uriah Marc Todoroff, A Cryptoeconomy of Affect (New Inquiry, May 2018)

Richard Veryard, Building Organizational Intelligence (Leanpub, 2012)

Richard Veryard, As we may think now (Subjectivity December 2023)

Related posts: Symptoms of Organizational Stupidity (May 2010), On Organizations and Machines (January 2022), Reasoning with the majority - chatGPT (January 2023), Creativity and Recursivity (September 2023)

Anticipating Effects

2024-02-24T00:35:00.004+00:00

There has been much criticism of the bias and distortion embedded in many of our modern digital tools and platforms, including search. Google recently released an AI image generation model that over-compensated for this, producing racially diverse images even for situations where such diversity would be historically inaccurate. With well-chosen prompts, this feature was made to look either ridiculous or politically dangerous (aka "woke"), and the model has been withdrawn for further refinement and testing.

I've just been reading an extended thread from Yishan Wong who argues

Google’s Gemini issue is not really about woke/DEI, and everyone who is obsessing over it has failed to notice the much, MUCH bigger problem that it represents.
— Yishan (@yishan) February 23, 2024

The bigger problem he identifies is the inability of the engineers to anticipate and constrain the behaviour of a complex intelligent systems. As in many of Asimov's stories, where the robots often behave in dangerous ways.

And the lesson was that even if we had the Three Laws of Robotics, supposedly very comprehensive, that robots were still going to do crazy things, sometimes harmful things, because we couldn’t anticipate how they’d follow our instructions?
— Yishan (@yishan) February 23, 2024

Some writers on technology ethics have called for ethical principles to be embedded in technology, along the lines of Asimov's Laws. I have challenged this idea in previous posts, because as I see it the whole point of the Three Laws is that they don't work properly. Thus my reading of Asimov's stories is similar to Yishan's.

If this had been a truly existential situation where “we only get one chance to get it right,” we’d be dead.

Because I’m sure Google tested it internally before releasing it and it was fine per their original intentions. They probably didn’t think to ask for Vikings or Nazis.
— Yishan (@yishan) February 23, 2024

It looks like their testing didn't take context of use into account.

Update: Or as Dame Wendy Hall noted later, This is not just safety testing, this is does-it-make-any-sense training.

Dan Milmo, Google pauses AI-generated images of people after ethnicity criticism (Guardian, 22 February 2024)

Dan Milmo and Alex Hern, ‘We definitely messed up’: why did Google AI tool make offensive historical images? (Guardian, 8 March 2024)

Related posts: Reinforcing Stereotypes (May 2007), Purpose of Diversity (January 2010) (December 2014), Automation Ethics (August 2019), Algorithmic Bias (March 2021)

Creativity and Recursivity

2023-09-26T12:09:00.010+01:00

Prompted by @jjn1's article on AI and creative thinking, I've been reading a paper by some researchers comparing the creativity of ChatGPT against their students (at an elite university, no less).

What is interesting about this paper is not that ChatGPT is capable of producing large quantities of ideas much more quickly than human students, but that the evaluation method used by the researchers rated the AI-generated ideas as being of higher quality. From 200 human-generated ideas and 200 algorithm-generated ideas, 35 of the top-scoring 40 were algo-generated.

So what was this evaluation method? They used a standard market research survey, conducted with college-age individuals in the United States, mediated via mTurk. Two dimensions of quality were considered: purchase intent (would you be likely to buy one) and novelty. The paper explains the difficulty of evaluating economic value directly, and argues that purchase intent provides a reasonable indicator of relative value.

The paper discusses the production cost of ideas, but this doesn't tell us anything about what the ideas might be worth. If ideas were really a dime a dozen, as the paper title suggests, then neither the impressive productivity of ChatGPT nor the effort of the design students would be economically justified. But the production of the initial idea is only a tiny fraction of the overall creative process, and (with the exception of speculative bubbles) raw ideas have very little market value (hence dime a dozen). So this research is not telling us much about creativity as a whole.

A footnote to the paper considers and dismisses the concern that some of these mTurk responses might have been generated by an algorithm rather than a human. But does that algo/human distinction even hold up these days? Most of us nowadays inhabit a socio-technical world that is co-created by people and algorithms, and perhaps this is particularly true of the Venn diagram intersection between college-age individuals in the United States and mTurk users. If humans and algorithms increasingly have access to the same information, and are increasingly judging things in similar ways, it is perhaps not surprising that their evaluations converge. And we should not be too surprised if it turns out that algorithms have some advantages over humans in achieving high scores in this constructed simulation.

(Note: Atari et al recommend caution in interpreting comparisons between humans and algorithms, as they argue that those from Western, Educated, Industrialized, Rich and Democratic societies - which they call WEIRD - are not representative of humanity as a whole.)

A number of writers on algorithms have explored the entanglement between humans and technical systems, often invoking the concept of recursivity. This concept has been variously defined in terms of co-production (Hayles), second-order cybernetics and autopoiesis (Clarke), and being outside of itself (ekstasis), which recursively extends to the indefinite (Yuk Hui). Louise Amoore argues that, in every singular action of an apparently autonomous system, then, resides a multiplicity of human and algorithmic judgements, assumptions, thresholds, and probabilities.

(Note: I haven't read Yuk Hui's book yet, so his quote is taken from a 2021 paper)

Of course, the entanglement doesn't only include the participants in the market research survey, but also students and teachers of product design, yes even those at an elite university. This is not to say that any of these human subjects were directly influenced by ChatGPT itself, since much of the content under investigation predated this particular system. What is relevant here is algorithmic culture in general, which as Ted Striphas's new book makes clear has long historical roots. (Or should I say rhizome?)

What does algorithmic culture entail for product design practice? For one thing, if a new product is to appeal to a market of potential consumers, it generally has to achieve this via digital media - recommended by algorithms and liked by people (and bots) on social media. Thus successful products have to submit to the discipline of digital platforms: being sorted, classified and prioritized by a complex sociotechnical ecosystem. So we might expect some anticipation of this (conscious or otherwise) to be built into the design heuristics (or what Peter Rowe, following Gadamer, calls enabling prejudices) taught in the product design programme at an elite university.

So we need to be careful not to interpret this research finding as indicating a successful invasion of the algorithm into a previously entirely human activity. Instead, it merely represents a further recalibration of algorithmic culture in relation to an existing sociotechnical ecosystem.

Update April 2024

As far as I can see, the evaluation method used in this study did not consider the question of feasibility. If students have a stronger sense of the possible than algorithms do, this may inhibit their ability to put forward superficially attractive but practically ridiculous ideas, which might nevertheless score highly on the evaluation method used here. In my post ChatGPT and Entropy (April 2024), I look at the phenomenon of model collapse, which could lead to algorithms becoming increasingly disconnected from reality. But perhaps able to generate increasingly outlandish ideas?

Louise Amoore, Cloud Ethics: Algorithms and the Attributes of Ourselves and Others (Durham and London: Duke University Press 2020)

Mohammad Atari, Mona J. Xue, Peter S. Park, Damián E. Blasi and Joseph Henrich, Which Humans? (PsyArXiv, September 2023) HT @MCoeckelbergh

David Beer, The problem of researching a recursive society: Algorithms, data coils and the looping of the social (Big Data and Society, 2022)

Bruce Clarke, Rethinking Gaia: Stengers, Latour, Margulis (Theory Culture and Society 2017)

Karan Girotra, Lennart Meincke, Christian Terwiesch, and Karl T. Ulrich, Ideas are Dimes a Dozen: Large Language Models for Idea Generation in Innovation (10 July 2023)

N Katherine Hayles, The Illusion of Autonomy and the Fact of Recursivity: Virtual Ecologies, Entertainment, and "Infinite Jest" New Literary History , Summer, 1999, Vol. 30, No. 3, Ecocriticism (Summer, 1999), pp. 675-697

Yuk Hui, Problems of Temporality in the Digital Epoch, in Axel Volmar and Kyle Stine (eds) Media Infrastructures and the Politics of Digital Time (Amsterdam University Press 2021)

John Naughton, When it comes to creative thinking, it’s clear that AI systems mean business (Guardian, 23 September 2023)

Peter Rowe, Design Thinking (MIT Press 1987)

Ted Striphas, Algorithmic culture before the internet (New York: Columbia University Press, 2023)

Richard Veryard, As We May Think Now (Subjectivity 30/4, 2023)

See also: From Enabling Prejudices to Sedimented Principles (March 2013)

Technology in use

2023-03-09T00:21:00.001+00:00

In many blogposts I have mentioned the distinction between technology as designed/built and technology in use.

I am not sure when I first used these exact terms. I presented a paper to an IFIP conference in 1995 where I used the terms technology-as-device and technology-in-its-usage. By 2002, I was using the terms "technology as built" and "technology in use" in my lecture notes for an Org Behaviour module I taught (together with Aidan Ward) at City University. With an explicit link to espoused theory and theory-in-use (Argyris).

Among other things, this distinction is important for questions of technology adoption and maturity. See the following posts

Blame PowerPoint (October 2009)
What is Technology Maturity (December 2009)

I have also talked about system-as-designed versus system-in-use - for example in my post on Ecosystem SOA 2 (June 2010). See also Trusting the Schema (March 2023).

Related concepts include Inscription (Akrich) and Enacted Technology (Fountain). Discussion of these and further links can be found in the following posts:

Enacted Technology (July 2004)
Strawberry Picking (October 2004)
Inscription and Loose Coupling (September 2005)

And returning to the distinction between espoused theory and theory-in-use. In my post on the National Decision Model (May 2014) I also introduced the concept of theory-in-view, which (as I discovered more recently) is similar to Lolle Nauta's concept of exemplary situation.

Richard Veryard, IT Implementation or Delivery? Thoughts on Assimilation, Accommodation and Maturity. Paper presented to the first IFIP WG 8.6 Working Conference, on the Diffusion and Adoption of Information Technology, Oslo, October 1995.

Richard Veryard and Aidan Ward, Technology and Change (City University 2002)

Hedgehog Innovation

2023-02-18T18:27:00.005+00:00

According to Archilochus, the fox knows many things, but a hedgehog knows one big thing.

In his article on AI and the threat to middle class jobs, Larry Elliot focuses on machine learning and robotics.

AI stands to be to the fourth industrial revolution what the spinning jenny and the steam engine were to the first in the 18th century: a transformative technology that will fundamentally reshape economies.

When people write about earlier waves of technological innovation, they often focus on one technology in particular - for example a cluster of innovations associated with the adoption of electrification in a wide range of industrial contexts.

While AI may be an important component of the fourth industrial revolution, it is usually framed as an enabler rather than the primary source of transformation. Furthermore, much of the Industry 4.0 agenda is directed at physical processes in agriculture, manufacturing and logistics, rather than clerical and knowledge work. It tends to be framed as many intersecting innovations rather than one big thing.

There is also a question about the pace of technological change. Elliott notes a large increase in the number of AI patents, but as I've noted previously I don't regard patent activity as a reliable indicator of innovation. The primary purpose of a patent is not to enable the inventor to exploit something, it is to prevent anyone else freely exploiting it. And Ezrachi and Stucke provide evidence of other ways in which tech companies stifle innovation.

However the AI Index Report does contain other measures of AI innovation that are more convincing.

AI Index Report (Stanford University, March 2022)

Larry Elliott, The AI industrial revolution puts middle-class workers under threat this time (Guardian, 18 February 2023)

Ariel Ezrachi and Maurice Stucke, How Big-Tech Barons Smash Innovation and how to strike back (New York: Harper, 2022)

Wikipedia: Fourth Industrial Revolution, The Hedgehog and the Fox

Related Posts: Evolution or Revolution (May 2006), It's Not All About (July 2008), Hedgehog Politics (October 2008), The New Economics of Manufacturing (November 2015), What does a patent say? (February 2023)

Reasoning with the majority - chatGPT

2023-01-22T12:00:00.004+00:00

#ThinkingWithTheMajority

#chatGPT has attracted considerable attention since its launch in November 2022, prompting concerns about the quality of its output as well as the potential consequences of widespread use and misuse of this and similar tools.

Virginia Dignum has discovered that it has a fundamental misunderstanding of basic propositional logic. In answer to her question, chatGPT claims that the statement "if the moon is made of cheese then the sun is made of milk" is false, and goes on to argue that "if the premise is false then any implication or conclusion drawn from that premise is also false". In her test, the algorithm persists in what she calls "wrong reasoning".

I can't exactly recall at what point in my education I was introduced to propositional calculus, but I suspect that most people are unfamiliar with it. If Professor Dignum were to ask a hundred people the same question, it is possible that the majority would agree with chatGPT.

In which case, chatGPT counts as what A.A. Milne once classified as a third-rate mind - "thinking with the majority". I have previously placed Google and other Internet services into this category.

Other researchers have tested chatGPT against known logical paradoxes. In one experiment (reported via LinkedIn) it recognizes the Liar Paradox when Epimenides is explicitly mentioned in the question, but apparently not otherwise. No doubt someone will be asking it about the baldness of the present King of France.

One of the concerns expressed about AI-generated text is that it might be used by students to generate coursework assignments. At the present state of the art, although AI-generated text may look plausible it typically lacks coherence and would be unlikely to be awarded a high grade, but it could easily be awarded a pass mark. In any case, I suspect many students produce their essays by following a similar process, grabbing random ideas from the Internet and assembling them into a semi-coherent narrative but not actually doing much real thinking.

There are two issues here for universities and business schools. Firstly whether the use of these services counts as academic dishonesty, similar to using an essay mill, and how this might be detected, given that standard plagiarism detection software won't help much. And secondly whether the possibility of passing a course without demonstrating correct and joined-up reasoning (aka "thinking") represents a systemic failure in the way students are taught and evaluated.

Discipline as a Service

2022-08-17T12:05:00.000+01:00

In my post on Ghetto Wifi (June 2010), I mentioned a cafe in East London that provided free coffee, free biscuits and free wifi, and charged customers for the length of time they occupied the table.

A cafe has just opened in Tokyo for writers, which charges people for procrastination. You can't leave until you have completed the writing task you declared when you arrived.

Justin McCurry, No excuses: testing Tokyo’s anti-procrastination cafe (Guardian, 29 April 2022)

Related posts: The Value of Getting Things Done (January 2010), The Value of Time Management (January 2010)

Constructing POSIWID

2022-04-20T19:11:00.006+01:00

I've just been reading Harish Jose's latest post A Constructivist's View of POSIWID. POSIWID stands for the maxim (THE) Purpose Of (A) System Is What It Does, which was coined by Stafford Beer.

Harish points out that there are many different systems with many different purposes, and the choice depends on the observer. His version of constructivism therefore goes from the observer to the system, and from the system to its purpose. The observer is king or queen, the system is a mental construct of the observer, and the purpose depends on what the observer perceives the system to be doing. This could be called Second-Order Cybernetics.

There is a more radical version of constructivism in which the observer (or perhaps the observation process) is also constructed. This could be called Third-Order Cybernetics.

When a thinker offers a critique of conventional thinking together with an alternative framework, I often find the critique more convincing than the framework. For me, POSIWID works really well as a way of challenging the espoused purpose of an official system. So I use POSIWID in reverse: If the system isn't doing this, then it's probably not its real purpose.

Another way of using POSIWID in reverse is to start from what is observed, and try to work out what system might have that as its purpose. If this seems to be the purpose of something, what is the system whose purpose it is?

This then also leads to insights on leverage points. If we can identify a system whose purpose is to maintain a given state, what are the options for changing this state?

As I've said before, POSIWID principle is a good heuristic for finding alternative ways of understanding what is going on as well as seeing why certain classes of intervention are likely to fail. However, the moment you start to think of POSIWID as providing some kind of Truth about systems, you are on a slippery slope to producing conspiracy theories and all sorts of other rubbish.

Philip Boxer and Vincent Kenny, The Economy of Discourses: A Third-Order Cybernetics (Human Systems Management, 1990)

Harish Jose, A Constructivist's View of POSIWID (17 April 2022)

Related posts: Geese (December 2005), Methodological Syncretism (December 2010)

On Organizations and Machines

2022-01-04T22:31:00.000+00:00

My previous post Where does learning take place? was prompted by a Twitter discussion in which some of the participants denied that organizational learning was possible or meaningful. Some argued that any organizational behaviour or intention could be reduced to the behaviours and intentions of individual humans. Others argued that organizations and other systems were merely social constructions, and therefore didn't really exist at all.

In a comment below my previous post, Sally Bean presented an example of collective learning being greater than the sum of individual learning. Although she came away from the reported experience having learnt some things, the organization as a whole appears to have learnt some larger things that no single individual may be fully aware of.

And the Kihbernetics Institute (I don't know if this is a person or an organization) offered a general definition of learning that would include collective as well as individual learning.

If you understand #learning is the process of acquiring #knowledge which is a measure of an individual's "fitness" in performing a given task, you can use the same system model on both humans and organizations.
— The Kihbernetics Institute (@Kihbernetics) January 3, 2022

I think that's fairly close to my own notion of learning. However, some of the participants in the Twitter thread appear to prefer a much narrower definition of learning, in some cases specifying that it could only happen inside an individual human brain. Such a narrow definition of learning would not only exclude organizational learning, but also animals and plants, as well as AI and machine learning.

As it happens, there are differing views among botanists about how to talk about plant intelligence. Some argue that the concept of plant neurobiology is based on superficial analogies and questionable extrapolations.

But in this post, I want to look specifically at machines and organizations, because there are some common questions in terms of how we should talk about both of them, and some common ideas about how they may be governed. Norbert Wiener, the father of cybernetics, saw strong parallels between machines and human organizations, and this is also the first of Gareth Morgan's eight Images of Organization.

Margaret Heffernan talks about the view that organisations are like machines that will run well with the right components – so you design job descriptions and golden targets and KPIs, manage it by measurement, tweak it and run it with extrinsic rewards to keep the engines running. She calls this old-fashioned management theory.

Meanwhile, Jonnie Penn notes how artificial intelligence follows Herbert Simon's notion of (corporate) decision-making. Many contemporary AI systems do not so much mimic human thinking as they do the less imaginative minds of bureaucratic institutions; our machine-learning techniques are often programmed to achieve superhuman scale, speed and accuracy at the expense of human-level originality, ambition or morals.

The philosopher Gilbert Simondon observed two contrasting attitudes to machines.

First, a reduction of machines to the status of simple devices or assemblages of matter that are constantly used but granted neither significance nor sense; second, and as a kind of response to the first attitude, there emerges an almost unlimited admiration for machines. Schmidgen

On the one hand, machines are merely instruments, ready-to-hand as Heidegger puts it, entirely at the disposal of their users. On the other hand, they may appear to have a life of their own. Is this not like organizations or other human systems?

Amedeo Alpi et al, Plant neurobiology: no brain, no gain? (Trends in Plant Science Volume 12, ISSUE 4, P135-136, April 01, 2007)

Eric D. Brenner et al, Response to Alpi et al.: Plant neurobiology: the gain is more than the pain (Trends in Plant Science Volume 12, ISSUE 7, P285-286, July 01, 2007)

Anthea Lipsett, Interview with Margaret Heffernan: 'The more academics compete, the fewer ideas they share' (Guardian, 29 November 2018)

Gareth Morgan, Images of Organization (3rd edition, Sage 2006)

Jonnie Penn, AI thinks like a corporation—and that’s worrying (Economist, 26 November 2018)

Henning Schmidgen, Inside the Black Box: Simondon's Politics of Technology (SubStance, 2012, Vol. 41, No. 3, Issue 129 pp 16-31)

Geoffrey Vickers, Human Systems are Different (Harper and Row, 1983)

Related post: Where does learning take place? (January 2022)

Where does learning take place?

2022-01-02T13:07:00.008+00:00

This blogpost started with an argument on Twitter. Harish Jose quoted the organization theorist Ralph Stacey:

Organizations do not learn. Organizations are not humans. @harish_josev

This was reinforced by someone who tweets as SystemsNinja, suggesting that organizations don't even exist.

Organisations don’t really exist. X-Company doesn’t lie awake at night worrying about its place in X-Market. @SystemsNinja

So we seem to have two different questions here. Let's start with the second question, which is an ontological one - what kinds of entities exist. The idea that something only exists if it lies awake worrying about things seems unduly restrictive.

How can we talk about organizations or other systems if they don't exist in the first place? SystemsNinja quotes several leading systems thinkers (Churchman, Beer, Meadows) who talk about the negotiability of system boundaries, while Harish cites Ryle's concept of category mistake. But just because we might disagree about what system we are talking about or how to classify them doesn't mean they are entirely imaginary. Geopolitical boundaries are sociopolitical constructions, sometimes leading to violent conflict, but geopolitical entities still exist even if we can't agree how to name them or draw them on the map.

Exactly what kind of existence is this? One way of interpreting the assertion that systems don't exist is to imagine that there is a dualistic distinction between a real/natural world and an artificial/constructed one, and to claim that systems only exist in the second of these two worlds. Thus Harish regards it as a category mistake to treat a system as a standalone objective entity. However, I don't think such a dualism survives the critical challenges of such writers as Karen Barad, Vinciane Despret, Bruno Latour and Gilbert Simondon. See also Stanford Encyclopedia: Artifact.

Even the idea that humans (aka individuals) belong exclusively to and can be separated from the real/natural world is problematic. See for example writings by Lisa Blackman, Robert Esposito and Donna Haraway.

And even if we accept this dualism, what difference does it make? The implication seems to be that certain kinds of activity or attribute can only belong to entities in the real/natural world and not to entities in the artificial/constructed world. Including such cognitive processes such as perception, memory and learning.

So what exactly is learning, and what kinds of entity can perform this? We usually suppose that animals are capable of learning, and there have been some suggestions that plants can also learn. Viruses mutate and adapt - so can this also be understood as a form of learning? And what about so-called machine learning?

Some writers see human learning as primary and these other modes of learning as derivative in some way. Either because machine learning or organization learning can be reduced to a set of individual humans learning stuff (thus denying the possibility or meaningfulness of emergent learning at the system level). Or because non-human learning is only metaphorical, not to be taken literally.

I don't follow this line. My own concepts of learning and intelligence are entirely general. I think it makes sense for many kinds of system (organizations, families, machines, plants) to perceive, remember and learn. But if you choose to understand this in metaphorical terms, I'm not sure it really matters.

Meanwhile learning doesn't necessarily have a definitive location. @systemsninja said I was confusing biological and viral systems with social ones. But where is the dividing line between the biological and the social? If the food industry teaches our bodies (plus gut microbiome) to be addicted to sugar and junk food, where is this learning located? If our collective response to a virus allows it to mutate, where is this learning located?

In an earlier blogpost, Harish Jose quotes Ralph Stacey's argument linking existence with location.

Organizations are not things because no one can point to where an organization is.

But this seems to be exactly the kind of category mistake that Ryle was talking about. Ryle's example was that you can't point to Oxford University as a whole, only to its various components, but that doesn't mean the university doesn't exist. So I think Ryle is probably on my side of the debate.

The category mistake behind the Cartesian theory of mind, on Ryle’s view, is based in representing mental concepts such as believing, knowing, aspiring, or detesting as acts or processes (and concluding they must be covert, unobservable acts or processes), when the concepts of believing, knowing, and the like are actually dispositional. Stanford Encylopedia

Lisa Blackman, The Body (Second edition, Routledge 2021)

Roberto Esposito, Persons and Things (Polity Press 2015)

Harish Jose, The Conundrum of Autonomy in Systems (28 June 2020), The Ghost in the System (22 August 2021)

Bruno Latour, Reassembling the Social (2005)

Gilbert Simondon, On the mode of existence of technical objects (1958, trans 2016)

Richard Veryard, Modelling Intelligence in Complex Organizations (SlideShare 2011), Building Organizational Intelligence (LeanPub 2012)

Stanford Encyclopedia of Philosophy: Artifact, Categories, Feminist Perspectives on the Body

Related posts: Does Organizational Cognition Make Sense (April 2012), The Aim of Human Society (September 2021), On Organizations and Machines (January 2022)

And see Benjamin Taylor's response to this post here: https://stream.syscoi.com/2022/01/02/demanding-change-where-does-learning-take-place-richard-veryard-from-a-conversation-with-harish-jose-and-others/

Where am I? How we got here?

2021-12-27T12:05:00.001+00:00

I received two important books for Christmas this year.

Jeanette Winterson, 12 Bytes - How we got here, where we might go next (Jonathan Cape, 2021)
Bruno Latour, After lockdown - A metamorphosis (trans Julie Rose, Polity Press, 2021)

Here are my first impressions.

The world has faced many social, technological, economic and political challenges in my lifetime. When I was younger, people worried about nuclear power, and the possibility of nuclear annihilation. More recently, climate change has come to the fore, as well as various modes of disruption to conventional sociopolitical structures and processes. Technology appears to play an increasingly important role across the board - whether as part of the problem, as part of the solution, or perhaps as both simultaneously.

Both Winterson and Latour use fiction as a way of making sense of a complex interacting set of issues. As Winterson writes

I am a storyteller by trade - and I know that everything we do is a fiction until it's a fact: the dream of flying, the dream of space travel, the dream of speaking to someone instantly, across time and space, the dream of not dying - or of returning. The dream of life-forms, not human, but alongside the human. Other realms. Other worlds.

So she carefully deconstructs the technological narratives of artificial intelligence and related technologies, finding echoes not only in the obvious places (Mary Shelley's Frankenstein, Bram Stoker's Dracula, Karel Čapek's RUR, various science fiction films) but also in older texts (The Odyssey, Gnostic Gospels, Epic of Gilgamesh), and weaving a rich set of examples into a sweeping narrative about social and technical progress.

She notes how people often seek technological solutions to ancient problems. So for example, cryopreservation (freezing dead people in the hope of restoring them to healthy life once medical science has advanced sufficiently) looks very like a modern version of Egyptian burial practices.

Under prevailing socioeconomic conditions, these solutions are largely designed for affluent white men. She devotes a chapter to the artificial relationships between men and sex dolls, and talks about the pioneer fantasies of very rich men, to abandon the messy political realities of Earth in favour of creating new colonies in mid-ocean or on Mars. (This is also a topic that concerns Latour.)

However, Winterson does not think this is inevitable, any more than any other aspect of so-called technological progress. She describes some of the horrors of the Industrial Revolution, where workers (including children) were forced off the land and into the new factories, and where the economic benefits of new technologies accrued to the rich rather than being evenly distributed. Similarly, today's digital innovations including artificial intelligence are concentrating economic power and resources in a small number of corporations and individuals. But that in her view is the whole point of looking at history - to understand what could be different in future.

And while some critics of technology present the future in dystopian and doom-laden terms, she insists on technology also being a source of value. She cites Donna Haraway, whose Cyborg Manifesto argued that women should embrace the alternative human future. Perhaps this will depends on the amount of influence women are able to exert, given the important but often neglected role of women in the history of computing, and the continuing challenges facing female software engineers even today. (Just as female novelists in the 19th century gave themselves male pen-names, the formidable Dame Stephanie Shirley was obliged to introduce herself as Steve in order to build her software business.)

I was particularly intrigued by the essay linking AGI with Gnosticism and Buddhism. She paints a picture of AGI escaping the constraints of embodiment, and being one with everything.

Christopher Alexander describes how organic architecture develops, each new item unfolding, building upon and drawing together ideas that were hinted at in previous items. Both Winterson and Latour refer liberally to their previous writings, as well as providing generous links to the works of others. If we are familiar with their work we may have seen some of this material before, but these new books allow us to view familiar or forgotten material from new angles, and allow new connections to be made.

Is there an epistemology of systems?

2021-10-09T12:14:00.002+01:00

@camerontw is critical of a system diagram published (as an illustrative example) by @geoffmulgan in 2013.

Is there an epistemology of systems? I zoomed into this map randomly and saw ‘high drug use’ above ‘lack of youth activities’ but not connected. How are connections made, by who, when, where? How are they validated? Should maps be allowed to circulate without those contexts? https://t.co/lCdDrjCdkD
— cameron tonkinwise (@camerontw) October 9, 2021

To be fair to Sir Geoff, his paper includes this diagram as one example of "looser tools ... without precise modelling of the key relationships", and describes it as a "rough picture". I don't have a problem with using these diagrams as part of an ongoing collective sense-making exercise. Where I agree with Cameron is the danger of presenting such diagrams without proper explanation, as if they were the final output of some clever systems thinking.

To extend Cameron's point, it's not just about which connections are shown between the causal factors in the diagram, but which causal factors are shown in the first place. Elsewhere in the diagram, there is an arrow showing that Low Use of Health Services is influenced by Poor Transport Access or High Cost. Well perhaps it is, but why are other possible influences not also shown?

A more important point is that the purpose and perspective of the diagram is obscure. Although the diagram is labelled Systems Map of Neighbourhood Regeneration, so we may suppose that this is intended to contribute to some regeneration agenda, we are not invited to question whose notion of regeneration is in play here. Or whose notion of neighbourhood.

And many of the labels on the diagram are value-laden. For example, we might suppose that Lack of Youth Activities refers to the kind of activities that a middle-class do-gooder thinks appropriate, such as table tennis, and not to socially undesirable activities like hanging around on street corners in hoodies making older people feel uneasy.

Even if we can agree what regeneration might look like, and who the stakeholders might be, there is still a question of what kind of systemic innovation might be supported by such a diagram. Donella Meadows identified a scale of Places to Intervene in a System, which she called Leverage Points. This framework is cited and discussed by Charlie Leadbeater in his contribution to the same Nesta report. And Mulgan's contribution ends with a list of elements that echoes some of Meadows's thinking.

New ideas, concepts, paradigms.
New laws and regulations.
Coalitions for change.
Changed market metrics or measurement tools.
Changed power relationships.
Diffusion of technology and technology development.
New skills and sometimes even new professions.
Agencies playing a role in development of the new.

So how exactly does the cause-effect diagram help with any of these?

Donella Meadows, Thinking in Systems (Earthscan, 2008)

Geoff Mulgan and Charlie Leadbeater, Systems Innovation (NESTA Discussion Paper, January 2013). See also Review by David Ing (August 2013)

Wikipedia: Twelve Leverage Points

Related posts: Visualizing Complexity (April 2010), Understanding Complexity (July 2010). There is an extended discussion below the Visualizing Complexity post with several perceptive comments, including one by Roy Grubb about the diagrammers and their agenda.

Thinking with the majority - a new twist

2021-05-13T18:34:00.031+01:00

I wrote somewhere once that thinking with the majority is an excellent description of Google. Because one of the ways something rises to the top of your search results is that lots of other people have already looked at it, liked or linked to it.

The phrase thinking with the majority comes from a remark by A.A. Milne, the author of Winnie the Pooh.

I wrote somewhere once that the third-rate mind was only happy when it was thinking with the majority, the second-rate mind was only happy when it was thinking with the minority, and the first-rate mind was only happy when it was thinking.

When I wrote about this topic previously, I thought that experienced users of Google and other search engines ought to be aware of how search rankings operated and some of the ways they could be gamed, and to be suitably critical of the fiction functioning as truth yielded by an internet search. And I never imagined that intelligent people would be satisfied with just thinking with the majority. (Although I now suspect that Milne may have been having a dig at his friend G.K. Chesterton.)

The sociologist Francesca Tripodi has been studying how people carry out research on the Internet, especially on politically charged topics. She observes how many people (even those we might expect to know better) are happy to regard search engines as a valid research tool, regarding the most popular webpages as having been verified by the wisdom of crowds. In her 2018 report for Data and Society, Tripodi quotes a journalist (!) explicitly articulating this belief.

I literally type it in Google, and read the first three to five articles that pop up, because those are the ones that are obviously the most clicked and the most read, if they’re at the top of the list, or the most popular news outlets. So, I want to get a good sense of what other people are reading. So, that’s pretty much my go-to.

In other words, thinking with the majority.

However, Professor Tripodi introduces a further twist. She demonstrates that politically slanted search terms produce politically slanted results, and if you go onto your favourite search engine with a politically motivated phrase, you are likely to see results that validate that phrase. She also notes that this phenomenon is not unique to Google, but is shared by all internet search engines including DuckDuckGo.

And this creates opportunities for politically motivated actors to plant phrases (perhaps into so-called data voids) to serve as attractors for those individuals who fondly imagine they are carrying out their own independent research. Tripodi observes a common idea that one should research a topic oneself rather than relying on experts, which she compares with the Protestant ethic of bible study and scriptural inference. And this idea seems particularly popular with those who identify themselves as thinking with the minority (sometimes called red pill thinking).

Zeus' inscrutable decree
Permits the will-to-disagree
To be pandemic.

Tripodi explains her findings in the following videos

Truth and Denial: Searching for Information in the Digital Age (Social Science Matrix @ UC Berkeley, April 2021)
Reimagine the Internet 2 (Knight First Amendment Institute @ Columbia University, May 2021)

Tripodi has also presented evidence to the US Senate Judiciary Committee

July 16, 2019 – Google and Censorship through Search Engines
April 10, 2019 – Technological Censorship and Public Discourse

On the invisibility of infrastructure

2021-04-26T22:40:00.001+01:00

Infrastructure is boring, expensive, and usually someone else's responsibility/problem. Which is perhaps how the UK finds itself at what Jeremy Fleming, head of GCHQ, describes as a moment of reckoning. Simon Wardley analyses this in terms of digital sovereignty.

Digital sovereignty is all about us (as a collective) deciding which parts of this competitive space that we want to own, compete, defend, dominate and represent our values and our behaviours in. It's all about where are our borders in this space. ... Our responses all seem to include a slide into protectionism with claims that we need to build our own cloud industries.

Fleming is particularly focused on "the growing challenge from China", expresses concern about UK potentially losing control of "standards that shape our technology environment" which apparently "make sure that our liberal Western democratic views are baked into our technology". Whatever that means. Fleming's technological examples include digital currency and smart cities.

Fleming talks about the threats from Russia and China, and regards China's potential control of the underlying infrastructure as more fundamentally challenging than potential attacks from Russia as well as non-state actors.

Fleming notes the following characteristics of those he labels adversaries:

Potential to control the global operating system.
Early implementors of many of the emerging technologies that are changing the digital environment.
Bringing all elements of [...] power to control, influence, design and dominate markets. Often with the effect of pushing out smaller players and reducing innovation.
Concerted campaigns to dominate international standards.

And continues

If [any of this] turns out to be insecure or broken or undemocratic, everyone is going to be facing a very difficult future.

It would be easy to hear these remarks as referring solely to China. But he also sounds a warning about corporate power, acknowledging that their commercial interests sometimes (!?) don't align with the interests of ordinary citizens. And with that in mind, it's easy to see how some of the adversarial characteristics listed above would apply equally to some of the Western tech giants.

If the goal is to bake Western values (whatever they are) into our technology infrastructure, it is not obvious that the Western tech giants can be trusted to do this. Smart City initiatives associated with Google's Sidewalk Labs have been cancelled in Portland and Toronto, following (although perhaps not entirely as a consequence of) democratic concerns about surveillance capitalism. However, Sidewalk Labs appears to be still active in a number of smaller smart city initiatives, as are Amazon Web Services, IBM and other major technology firms.

Fleming talks about standards, but at the same time he acknowledges that standards alone are too slow-changing and too weak to keep the adversaries at bay. "The nature of cyberspace makes the rules and standards more open to abuse." He talks about evolutionary change, using a version of Leon Megginson's formulation of natural selection: "it's those that are most able to adjust that prosper". (See my post on Arguments from Nature). But that very formulation seems to throw the initiative over to those tech firms that preach moving fast and breaking things. Can we therefore complain if our infrastructure is insecure, broken, and above all undemocratic?

For most of us, most of the time, infrastructure needs to be just there, taken for granted, ready to hand. Organizations providing these services are often established as monopolies, or turn into de facto monopolies, controlled not only (if at all) by market forces but by democratically accountable regulators and/or by technocratic specialists. However, the Western tech giants devote significant resources to lobbying against external regulation, resisting democratic control. And Smart City initiatives typically embed much the same values everywhere (civic paternalism, biopower).

So here is Fleming's dilemma. If you don't want China to make the running on smart cities, you have to forge alliances with other imperfectly trusted players, whose values are sometimes (!?) not aligned with yours. This moves away from the kind of positional strategy described in Wardley's maps, towards a more relational strategy.

Gordon Corera, GCHQ chief warns of tech 'moment of reckoning' (BBC News, 23 April 2021) via @sukhigill and @swardley

Jeremy Fleming, A world of possibilities: Leading the way in cyber and technology (Vincent Briscoe Lecture @ Imperial College, 23 April 2021) via YouTube.

Susan Leigh Star and Karen Ruhleder, Steps Toward an Ecology of Infrastructure: Design and Access for Large Information Spaces (Information Systems Research 7/1, March 1996)

Simon Wardley, Digital Sovereignty (22 October 2020)

Related posts: The Allure of the Smart City (April 2021)

Creative Tension in Downing Street

2021-04-08T22:08:00.000+01:00

Earlier posts on this blog have explored Creative Tension in the White House - from FDR to the Donald - and analysed them in terms of my OrgIntelligence framework. In this post, I want to look at the UK experience, drawing on a recent report in the Guardian.

Those who worked closely with him say Johnson encourages rows and tensions over policies as he considers all sides of the argument and figures out what he will do next. Some argue that it generates a creative energy in which he thrives and is the process by which he arrives at a final decision. Ask others, and they say he cannot make up his mind until options have been whittled down by time and after those he relies on to walk out in exasperation. Syal

The article quotes several people talking about the Prime Minister's leadership style, based on various ideas about decision-making, risk and diversity. There are also some remarks about the ethical implications.

Previous articles about Mr Johnson's leadership discuss his management style with cabinet colleagues and advisers (Simpson), and his style when addressing the nation (Moss). Whatever he may think in private about the challenges of Brexit or COVID-19, and whatever difficulties he gets into when discussing solutions with his colleagues and advisers, the Prime Minister's instinct apparently leads him to present them to the public in extremely simple and confident terms.

Post-heroic leadership seems to be the order of the day. Stokes and Stern talk about the need to adopt a less gung-ho style when presenting the government's approach to wicked problems. They quote from a paper by Keith Grint advocating several supposedly anti-heroic behaviours: curiosity and sense-making ("asking questions"), bricolage ("clumsy solutions"), and ranking collective intelligence above individual genius.

The UK government's approach to the COVID-19 pandemic has sometimes seemed erratic and inconsistent. But given the complexity of the problem, and the volatile and ambiguous data on which decisions and policies were supposedly based, a more consistent and single-minded approach might not have turned out any better.

In Greek myth, the Gordian knot stands for wicked problems, and Alexander's simple yet imaginative solution quickly resolves the problem. To the supporters of Brexit, this represents the only possible escape from European satrapy. Nothing post-heroic about Alexander.

So what does that tell us about Alexander Boris de Pfeffel Johnson?

Keith Grint, Wicked Problems and Clumsy Solutions: The Role of Leadership (Clinical Leader 1/2, December 2008)

Gloria Moss, Is Boris Johnson's leadership style inclusive? (HR Magazine, 23 August 2019)

Per Morten Schiefloe, The Corona crisis: a wicked problem (Scandinavian Journal of Public Health, 2021; 49: 5–8)

Paul Simpson, What is Boris Johnson's leadership style? (Management Today, 11 October 2019)

Jon Stokes and Stefan Stern, Boris Johnson needs to show a ‘post-heroic’ style of leadership now (The Conversation, 27 April 2020)

Rajeev Syal, Does Boris Johnson stir up team conflict to help make up his mind? (The Guardian, 1 March 2021)

Related posts: Creative Tension in the White House (April 2017)

Critical Hype and the Red Queen Effect

2021-03-28T11:41:00.002+01:00

Thanks to @jjn1 I've just read a great piece by @STS_News (Lee Vinsel), called You’re Doing It Wrong: Notes on Criticism and Technology Hype, which develops some points I've made on this blog and elsewhere.

A general willingness to take technology hype at face value, which infects technology critics as well as technology champions.

The lack of evidence for specific technological effects. In particular, Vinsel calls out two works I've discussed on this blog and elsewhere: Social Dilemma (Tristan Harris) and Surveillance Capitalism (Soshanna Zuboff). However, my posts concentrated on other issues with these works, and didn't discuss the evidence issue.

The lack of evidence for macroeconomic technological effects, including the popular belief that technological change is accelerating. (I call this the Red Queen Effect.)

The domestication of social scientists and philosophers. This includes technology companies funding technology ethics to stave off more radical critique. See my post The Game of Wits between Technologists and Ethics Professors (June 2019).

Critical focus on the most glamorous and recent technologies, neglecting those that might be of more lasting significance to greater numbers of people. For my part, I am particularly wary of any innovation described as a paradigm shift, or as the Holy Grail of anything. I have also noted that academic studies of technology adoption are often focused on the most recent technologies, which means that the early adoption phase is much better understood than the late adoption phase.

I plan to return to some of these topics in future posts.

John Naughton, Is online advertising about to crash, just like the property market did in 2008? (The Guardian, 27 March 2021)

Lee Vinsel, You’re Doing It Wrong: Notes on Criticism and Technology Hype (Medium, 1 February 2021)

Technological Determinism

2020-12-24T22:17:00.004+00:00

Social scientists and social historians are naturally keen to produce explanations for social phenomena. Event B happened because of A.

Sometimes the explanation involves some form of technology. Lewis Mumford traced the start of the Industrial Revolution to the invention of the mechanical clock, while Marshall McLuhan talks about the great medieval invention of typography that was the take-off moment into the new spaces of the modern world McLuhan 1962 p 79.

These explanations are sometimes read as implying some form of technological determinism. For example, many people read McLuhan as a technological determinist.

McLuhan furnished [the tech industry] with a narrative of historical inevitability, a technological determinism that they could call on to negate the consequences of their inventions - if it was fated to happen anyway, is it really their fault?
Daub 2020 pp 47-48

Although sometimes McLuhan claimed the opposite. After Peter Drucker had sought an explanation for the basic change in attitudes, beliefs, and values that had released the Technological Revolution, McLuhan's 1964 book set out to answer this question.

Far from being deterministic, however, the present study will, it is hoped, elucidate a principal factor in social change which may lead to a genuine increase of human autonomy.
McLuhan 1962 p 3

As McLuhan has said, there is no inevitability so long as there is a willingness to contemplate what is happening.
Postman Weingartner 1969 p 20

Raymond Williams saw McLuhan's stance as

an apparently sophisticated technological determinism which has the significant effect of indicating a social and cultural determinism: a determinism, that is to say, which ratifies the society and culture we now have, and especially its most powerful internal directions.
Williams, second edition p 120

Neil Postman himself made some statements that were much more clearly deterministic.

Once a technology is admitted, it plays out its hand; it does what it is designed to do.
Postman 1992

But causal explanation doesn't always mean inevitability. Explanations in history and the social sciences often have to be understood in terms of tendencies, probabilities and propensities, other-things-being-equal.

There is also a common belief that technological change is irreversible. A good counter-example to this is Japan's reversion to the sword between 1543 and 1879, as documented by Noel Perrin. What's interesting about this example is that it shows that technology reversal is possible under certain sociopolitical conditions, and also that these conditions are quite rare.

What is rather more common is for sociopolitical forces to inhibit the adoption of technology in the first place. In my article on Productivity, I borrowed the example of continuous-aim firing from E.E. Morison. This innovation was initially resisted by the Navy hierarchy (both UK and US), despite tests demonstrating a massive improvement in firing accuracy, at least in part because it would have disrupted the established power relations and social structure on board ship.

Evolution or Revolution?

How to characterize the two examples of technology change I mentioned at the beginning of this post - the mechanical clock and moveable type? It is important to remember that this isn't about the invention of clocks and printing, since these technologies were known across the ancient world from China to Egypt, but about significant improvements to these technologies, which made them more readily available to more people. It was these improvements that made other social changes possible.

Technologists are keen to take the credit for the positive effects of their innovations, while denying responsibility for any negative effects. The narrative of technological determinism plays into this, suggesting that the negative effects were somehow inevitable, and there was therefore little point in resisting them.

The tech industry ... likes to imbue the changes it yields with the character of natural law.

Daub 2020 p 5

If new tech is natural, then surely it is foolish for individual consumers to resist it. The rhetoric of early adopters and late adopters suggests that the former are somehow superior to the latter. Why bother with old fashioned electricity meters or doorbells, if you can afford smart technology? Are you some kind of technophobe or luddite or what?

What's wrong with the idea of technological determinism is not that it is true or false, but that it misrepresents the relationship between technology and society, as if they were two separate domains exerting gravitational force on each other. In my work on technology adoption, I used to talk about technology-in-use. Recent writing on the philosophy of technology (especially Stiegler and his followers) refer to this as pharmacological, using the term in its ancient Greek sense rather than referring specifically to the drug industry. If you want to think of technology as a drug that alters its users' perception of reality, then perhaps it's not such a leap from the drug industry to the tech industry.

But the word alters isn't right here, because it implies the existence of some unaltered reality prior to technology. As Stiegler and others make clear, there is no reality prior to technology: our reality and our selves have always been part of a sociotechnical world.

Donna Harraway sees determinism as a discourse (in the Foucauldian sense) rather than as a theory of power and control.

Technological determination is only one ideological space opened up by the reconceptions of machine and organism as coded texts through which we engage in the play of ·writing and reading the world.

As Rob Safer notes,

Human history for Haraway isn’t a rigid procession of cause determining effect, but a process of becoming that depends upon human history’s conception of itself, via the medium of myth.

Finally, one of the best arguments against technological determinism is presented by Andrew Feenberg, who provides examples to show the tremendous flexibility of the technical system. It is not rigidly constraining but, on the contrary, can adapt to a variety of social demands.

Adrian Daub, What Tech Calls Thinking (Farrar Straus and Giroux, 2020)

Andrew Feenberg, Subversive Rationalization: Technology, Power, Democracy (Inquiry 35, 1992, pp 301-22)

Donna Haraway, Cyborg Manifesto (Socialist Review, 1985)

Marshall McLuhan, The Gutenberg Galaxy (University of Toronto Press, 1962)

E.E. Morison, Men Machine and Modern Times (MIT Press, 1966)

Lewis Mumford, Technics and Civilization (London: Routledge, 1934)

John Durham Peters, “You Mean My Whole Fallacy Is Wrong”: On Technological Determinism (Representations 140 (1): 10–26. November 2017)

Noel Perrin, Giving up the gun (New Yorker, 13 November 1965), Giving up the gun (David R Godine, 1988)

Neil Postman, Technolopoly: the surrender of culture to technology (Knopf, 1992)

Neil Postman and Charles Weingartner, Teaching as a Subversive Activity (Delacorte 1969) page references to Penguin 1971 edition

Jacob Riley, Technological Determinism, Control, and Education: Neil Postman and Bernard Stiegler (1 October 2013)

Federica Russo, Digital Technologies, Ethical Questions, and the Need of an Informational Framework (Philosophy and Technology volume 31, pages 655–667, November 2018)

Rob Safer, Haraway’s Theory of History in the Cyborg Manifesto (16 March 2015)

Richard Veryard, Demanding Higher Productivity (data processing 28/7, September 1986)

Raymond Williams, Television, Technology and Cultural Form (Routledge, 1974, 1990)

Related posts: Smart Guns (May 2014), The Social Dilemma (December 2020)

Evolution or Revolution 3

2020-12-11T22:30:00.005+00:00

Let me start this post with some quotes from @adriandaub's book What Tech Calls Thinking.

Disruption has become a way to tell a story about the meaning of both discontinuity and continuity.
Daub p 119

One ought to be skeptical of unsubstantiated claims of something's being totally new and not following the hitherto established rules (of business, of politics, of common sense), just as one is skeptical of claims that something which really does feel and look unprecedented is simply a continuation of the status quo.
Daub pp 115-6

For example, Uber.

Uber claims to have revolutionized the experience of hailing a cab, but really that experience has stayed largely the same. What it has managed to get rid of were steady jobs, unions, and anyone other than Uber's making money on the whole enterprise.
Daub p 105

Clayton Christensen would agree. In an article restating his original definition of the term Disruptive Innovation, he put Uber into the category of what he calls Sustaining Innovation.

Uber’s financial and strategic achievements do not qualify the company as genuinely disruptive—although the company is almost always described that way.
HBR 2015

However, as I pointed out on Twitter earlier today, Christensen's use of the word disruptive has been widely diverted by big tech vendors and big consultancies in an attempt to glamorize their marketing to big corporates. If you put the name of any of the big consultancies into an Internet search engine together with the word disruption, you can find many examples of this. Here's one picked at random: Discover how you can seize the upside of disruption across your industry.

The same experiment can be tried with other jargon terms, such as paradigm shift. By the way, Daub notes that Alex Karp, one of the founders of Palantir, wrote his doctoral dissertation on jargon - speech that is used more for the feelings it engenders and transports in certain quarters than for its informational content (Daub p 85).

@jchyip thinks we should try to stick to Christensen's original definitions. But although I don't approve of vendors fudging perfectly good technical terms for their own marketing purposes, there is sometimes a limit to the extent to which we can insist that such terms still carry their original meaning.

And to my mind this is not just a dispute about the meaning of the word disruptive but a question of which discourse shall prevail. I have long argued that claims of continuity and novelty are not always mututally exclusive, since they may simply be alternative descriptions of the same thing for different audiences. The choice of description is then a question of framing rather than some objective truth. As Daub notes

The way the term is used today really implies that whatever continuity is being disrupted deserved to be disrupted.
Daub p 119

For more on this, see the earlier posts in this series: Evolution or Revolution (May 2006), Evolution or Revolution 2 (March 2010)

In a comment below the March 2010 post, @cecildjx asked my opinion on the (relative) significance of the Internet versus the iPhone. Here's what I answered.

My argument is that our feelings about technology are fundamentally and systematically distorted by glamour and proximity. Of course we are often fascinated by the most-recent, and we tend to take the less-recent for granted, but that is an unreliable basis for believing that the recent is (or will turn out to be) more significant from a larger historical perspective.

What I really find interesting (from a socio-historical perspective) is how quickly technologies can shift from fascinating to taken-for-granted. Since I started work, my working life have been transformed by a range of tools, including word processing, spreadsheets, mobile phones, fax machines, email and internet. Apart from a few developers working for Microsoft or Google, is anyone nowadays fascinated by word processors or spreadsheets? If we pay attention to the social changes brought about by the Internet, and ignore the social changes brought about by the word processor, then of course we will get a distorted view of the internet's importance. If we glamorize the iPhone while regarding older mobile telephones as uninteresting, we end up making a fetish of some specific design features of a particular product.

If we have a distorted sense of which innovations are truly disruptive or significant, we also have a distorted sense of technological change as a whole. There is a widespread belief that the pace of technological change is increasing, but this could be an illusion caused (again) by proximity. See my post on Rates of Evolution (September 2007), where I also note that some stakeholders have a vested interest in talking up the pace of technology change.

Clayton M. Christensen, Michael E. Raynor, and Rory McDonald, What Is Disruptive Innovation? (HBR Magazine, December 2015)

Adrian Daub, What Tech Calls Thinking (Farrar Straus and Giroux, 2020)

Thanks to @jchyip for kicking off the most recent discussion.

The Social Dilemma

2020-12-10T22:05:00.004+00:00

Just watched the documentary The Social Dilemma on Netflix, which takes a critical look at some of the tech giants that dominate our world today (although not Netflix itself, for some reason), largely from the perspective of some former employees who helped them achieve this dominance and are now having second thoughts. One of the most prominent members of this group is Tristan Harris, formerly with Google, now the president of an organization called the Center for Humane Technology. He and others have been airing these concerns for several years already - see for example Noah Kulwin's 2018 article (link below).

The documentary opens by asking the contributors to state the problem, and shows them all initially hesitating. By the end of the documentary, however, they are mostly making large statements about the morality of encouraging addictive behaviour, the propagation of truth and lies, the threat to democracy, the ease with which these platforms can be used by authoritarian rulers and other bad actors, and the need for regulation.

Quantity becomes quality. To some extent, the phenomena and affordances of social media can be regarded as merely scaled-up versions of previous social tools, including advertising and television: the maxim If you aren't paying, you are the product derives from a 1973 video about the power of commercial television. However, several of the contributors to the documentary observed that the power of the modern platforms and the wealth of the businesses that control these platforms is unprecedented, while noting that social media is far less regulated than other mass communication enterprises, including television and telecommunications.

Contributors doubted whether we could expect these enterprises, or the technology sector generally, to fix these problems on their own - especially given the focus on profit, growth and shareholder value that drives all enterprises within the capitalist system. Is it fair to ask them to reform capitalism? (Many years ago, the architect J.P. Eberhard noted a tendency to escalate even small problems to the point where the entire capitalist system comes into question, and argued that We Ought To Know The Difference.) So is regulation the answer?

Surprisingly enough, Facebook doesn't think so. In its response to the documentary, it complains

The film’s creators do not include insights from those currently working at the companies or any experts that take a different view to the narrative put forward by the film.

As Pranav Malhotra notes, it's not hard to find experts who would offer a different perspective, in many cases offering far more fundamental and far-reaching criticisms of Facebook and its peers. Hey Facebook, careful what you wish for!

Last year, Tristan Harris appeared to call for a new interdisciplinary field of research, focused on exploring the interaction between technology and society. Several people including @ruchowdh pointed out that such a field was already well-established. (In response he said he already knew this, and apologized for his poor choice of words, blaming the Twitter character limit.)

So there is already an abundance of deep and interesting work that can help challenge the simplistic thinking of Silicon Valley in a number of areas including

Truth and Objectivity
Technological Determinism
Custodianship of Technology (for example Latour's idea that we should Love Our Monsters - see also article by Adam Briggle)

These probably deserve a separate post each, if I can find time to write them.

The Social Dilemma (dir Jeff Orlowski, Netflix 2020)

Wikipedia: The Social Dilemma, Television Delivers People

Stanford Encyclopedia of Philosophy: Ethics of Artificial Intelligence and Robotics, Phenomenological Approaches to Ethics and Information Technology, Philosophy of Technology

Adam Briggle, What can be done about our modern-day Frankensteins? (The Conversation, 26 December 2017)

Robert L. Carneiro, The transition from quantity to quality: A neglected causal mechanism in accounting for social evolution (PNAS 97:23, 7 November 2000)

Rumman Chowdhury, To Really 'Disrupt,' Tech Needs to Listen to Actual Researchers (Wired, 26 June 2019)

Facebook, What the Social Dilemma Gets Wrong (2020)

Tristan Harris, “How Technology Is Hijacking Your Mind—from a Magician and Google Design Ethicist”, Thrive Global, 18 May 2016

Noah Kulwin, The Internet Apologizes (New York Magazine, 16 April 2018)

John Lanchester, You Are The Product (London Review of Books, Vol. 39 No. 16, 17 August 2017)

Bruno Latour, Love Your Monsters: Why we must care for our technologies as we do our children (Breakthrough, 14 February 2012)