• 2014 How are conscientiousness and cognitive ability related to one another? A re-examination of the intelligence compensation hypothesis – “true association between these constructs may be zero or positive at the population level but that the use of selected research samples has sometimes resulted in the appearance of a negative association.”
• 2006 Is conscientiousness positively or negatively related to intelligence? Insights from the national level – “Self-report conscientiousness scores are negatively associated with IQ” while “Observer-rated conscientiousness scores tend to positively correlate with IQ.”
• 2015 Why is Conscientiousness negatively correlated with intelligence? – “the negative relationship between intelligence and Conscientiousness is due to fluid intelligence affecting the development of Conscientiousness, in an educated and need-achieving population”
• 2014 The social influences on trait Conscientiousness: Findings from a nationally representative sample – “a significant, but very small positive association between intelligence, social factors and Conscientiousness.”
• 2023 Conscientiousness and Cognitive Abilities: a Meta-Analysis – “Several random-effects meta-analyses were conducted, obtaining meta-analytical correlations of .02, -.06 and -.06 between Conscientiousness (global) and cognitive abilities (global), crystallized intelligence and fluid intelligence, respectively.” Unclear why they exclude studies that do NOT “report any link between Conscientiousness and cognitive ability”, that seems like a “fail to reject the null hypothesis” result that should be very relevant?

The goal of this book seems to be twofold:

1. to expose the lay reader to the world of complexity theory and science
2. to argue that the true nature of reality is, in some sense, “consciousness” and that this is somehow realted to complexity theory

It achieves the first goal, and comes nowhere close the second.

Overall I found the book to be engaging and well-written, if poorly argued at some of its most important points. I am bothering to write up these reflections because I think it is a good representative of the late 20th, early 21st century flavour of “let’s show how some surprising discoveries from Western science like quantum mechanics actually jive in a postmodern way with a pastiche of religious and pseudo-religious wisdom”. I am using wisdom not in quotes here and non-ironically, in that I do think religious traditions can be sources of wisdom. But the rest of that quote gives an accurate sense as to how generally non-rigorous I think these approaches are.

I am all for attempting to bridge religion and sicence in a vaguely CP Snow’s “Two Cultures” kind of way, but I can’t presently recall ever having encountered the fruits of such an effort that didn’t fall almost completely flat.

The book starts out well, providing a lucid introduction to complexity theory. The part I found most interesting was the idea that some amount of randomness or unpredictability was key to complex systems. Theise introduces the term “quenched disorder” as a level of randomness that is sufficiently in between extremes that it permits for adaptibilty through random discovery of new paths/behaviours, while not collapsing into chaos due to lack of structure or predictabiltiy.

Next Theise brings us into the world of cells, which as a pathologist is his area of expertise. He asks us to consider looking at a body, then zooming into its cells; or looking at an ant colony from afar, then zooming into a single ant, and the futher still to its cells. At each level we encounter a complex self-organizing system, and there is no real answer to the question of which is the “right” level. Whether you’re looking at a single thing or a phenomenon is a matter of perspective, it is both at once.

This doubling Niese argues is related to what physicists call complementarity. This is most familiar to us from the wave-partical duality demonstrated by the slit experiment.

Bohr, however, went further, asserting that complentarity was fundamental not just for describing existence at the incredibly minute scales of the quantum realm but for describing living beings at our normal everyday scale as well.

At this point in the text a footnote references an article by Theise in Nature vol 435. Curiously the single-page essay (not a peer-reviewd article) doesn’t mention Bohr at all. Not clear if this is an editing error or what, but a brief foray into some secondary literature on the web (most of it admittedly not high quality) suggests that Bohr was indeed taken with the idea that the principle of complementarity was not only relevant at the quantum scale. At this point this isn’t a direct “appeal to authority” fallacy, though it is sort of skirting close to that.

Theise elaborates from this

The realization that one’s body is a unitary entity by itself, but also, equally, not, has important implications. One is that the boundary of one’s body starts to become indistinct. At the everyday scale, my boundary is my skin and your boundary is your skin. Close your eyes and feel how your fingers meet this book or the device on which you are reading it. You feel a sharp, distinct edge between you and the not-you where skin meets object. But at the microscopic level, just how sharp and distinct is the surface of your skin? Not very distinct at all. Cells from the top layer are constantly shedding as they die. Much of the dust of our homes, in fact, consists of sloughed off cells from our skin… At the microscopic level, we are thus not actually bounded by the top of our skins. Our boundaries are at least as broad as the spaces we inhabit. (p 48)

Here is where I begin to take issue with Theise’s argument.

Boundaries, boundaries everywhere, as far as the eye can see… or not at all

Simply because our skin cells slough off and coagulate into dust bunnies, how does that mean the boundary of my body has somehow extended to behind my dresser? This is absurd. It’s about as cogent as when my daughter says that because she has touched her finger to her tongue and then her finger to the floor, her tongue is therefore on the floor.

Maybe Theise could have argued that the microorganisms living on our skin and hair can have physical interactions with the world beyond us that then somehow influence our own bodies’ perceptive ability or homeostatic state–this may well be the case, and would be a stronger point in his favour, but still results in the boundary of our body basically being exactly what we thought it was, plus a couple micrometers here and there to accommodate the volume of these microorganisms.

There may well be other more compelling arguments for how the boundaries of our bodies are less rigid than we usually think (and I am generally open to arguments from the extended mind/extended cognition world that our minds may be fruitfully considered to have a fuzzier and wider boundary than we think), but that is not the argument Thesis makes here. The argument he does make is entirely unconvincing so far.

Theise attempts to expand this argument via the bacterial microbiomes that inhabit our guts and reside on the surface of our skin.

The microbiome doesn’t only colonize us; it is integral to our living, healthy bodies… These organisms… travel from us whenever we touch something. Doornobs, cell phones, counteropts, pens, each other. Every time you shake hands with, kiss, or hug someone else, some of ‘you’ gets left behind and some of the person you touched travels away with you. This process of bacterial exchange is so pronounced that the microbiomes of people (and pets) who live together become one giant shared microbiome, an continuous multcellular entity enveloping the human (and dog and cat) islands… if we keep this fact about or shared microbiomes in mind, the boundaries between ourselves and what’s outside our skin start to look even blurrier

Does one giant, shared, boundary-less microbiome really emerge in this case? Going back to Theise’s earlier arguments about perspective, I would agree that it is possible to imagine a perspective in which that is the case. From a certain, distant point of view, one might look at some cohabitating humans, dogs, etc and say “wow that’s a bunch of mixed up bacteria”. But note that this doesn’t fit into the “zooming out” metaphor Theise used earlier: we can’t both “zoom out” to see humans, pets and bacteria all at the same time, since they are only perceptible on vastly different scales. If we “zoom in” close enough that we see bacteria, we would in fact not see a shared, unitary, boundary-less microbiome at all, but rather one full of fairly sharp discontinuities. E coli in the intestines, but basically nowhere else, Staphylococcus on the skin but basically nowhere else, minimal bacteria in the air between bodies, and so on. We would see boundaries everywhere.

Is there a scientific perspective according to which it is particularly useful to see this situation as a single, boundaryless microbiome? Theise has certainly not given us any reason to think so. Even if he could, this doesn’t mean that the boundaries of our skin or bodies have suddenly evaporated, it simply means there are ways of exchanging parts of our bodies across those boundaries. Just because a medieval fortification could permit egress and ingress through gates in its walls does’t mean those walls don’t exist all of a sudden. If attackers discovered that every block in the wall could be passed through simply as if it did not exist, then it would make sense to say there is no boundary. And if biologists discovered that our skin permitted every particle and microorganism to move through it, then we’d say “wow there really is no boundary” but of course that is not remotely the case.

Everything is everything

Theise attempts a similar argument across time: how can we draw a boundary between our selves now, and yesterday, and back to an embryo, and then back to our parents as embryos, ad infinitum. Hegel rightly criticized this sort of muddled thiking about 2 centuries ago as the “night in which all cows are black”. In this vein of thought, I guess we’d say that desks and spoons and wooden jewelry carved from the lumber from a single tree really just are that tree which is a fun way of thinking when you’re high but if you try eating your soup with a toothpick or sitting down to study with your books on your ladle you will find that accepting the reality of these boundaries is a much more useful way of thinking. It is of course possible to think in the other “everything is one” way, and doing so can provide valuable ethical motivation and contemplative satisfaction, but in most everyday cases is is not a useful way of looking at the world, and is certainly not “truer” than any other way.

In the next chapter, Theise attempts to provide some scientific motivation for complementarity from his own area of expertise. He imagines a world in which rather than microscopes having first revealed to us cells with their walls separating them from each other, some alternate microscope technology in which we first see the nucleii floating in an undivided fluid, and only later discover cell walls.

In this alternate history, the fluid doctrine would have been born as the foundational paradigm of Western medicine and biology. In the years that followed, after applying special stains and seeing cell membranes for the first time, these scientists wouldn’t have backtracked, saying, “Oh, we were wrong, the body is made of cells.” They would have instead said something like, “Oh look, there is a semipermeable membrande partitioning of the body’s fluid continuum.”

So which is it? Is the body made of discrete cells or is it a fluid continuum? Yet again, we find a complementarity. The two different views illumiante different truths of our bodies. Each view captures aspects that are hidden by the other, while at the same time concealing details the other view reveals. Once more, a complete understanding of the whole depends on both views, equally, even though they seem contradictory.

First of all, it is entirely possible that scientists would have backtracked in this hypothetical scenario. Scientists once held to various fluid theories of heat, which were later rejected in favour of mechanical theories that better explained experimental observations.

Theise here makes the claim that the fluid view captures something hidden by the other view, but doesn’t tell us what this is, nor why whatever insights it reveals couldn’t be equally adequately expressed in the cell model. This is of course a book for a lay audience and Theise is an expert, so we can grant him some benefit of the doubt that there are such facts or insights.

But we should at least expect an argument that these views are incompatible in the same way that the wave/particle duality shows an incompatibility: the wave behaviour is literally impossible if the light were behaving as a particle, and yet it is both. In this case though, there is no assertion that some observed behaviour or property is completely inexplicable and physically mystifying from one point of view and only explicable from the other point of view.

As some evidence in support something like a fluid model, Theise offers us acupuncture which has effects that are “testable and reproducible” but have not been sufficiently explained by standard anatomy or by the cell doctrine.

Anatomically, those acupoints do not correspond to any nerves, blood vessels, lymphatics, or any other apparent anatomic structure… The cell doctrine seems unable to explain acupuncture then, but thinking of the body as a fluid might provide useful insights.

Contrariwise, there is a surfeit of researching attempting to, and claming some success with, explaining the effectiveness of acpuncture on the basis of Western biology, e.g. Understandings of acupuncture application and mechanisms.

Nothing is anything

Somewhat subtly, but without sufficient justification, Theise eventually moves from a discussion of complementarity and alternate perspectives to absolute claims like:

We have searched deep into the smallest recesses of existence–down into the quantum foam and space-time itself–and nowhere could we find an object with inherent existence in and of itself.

What would it mean for something to exist “in and of itself”? Theise doesn’t tell us, and I’d rather not blame him for not being a philosopher, except that he has now transitioned firmly into the realm of metaphysics, making sweeping claims about the nature of reality. His brand of metaphysics is, perhaps surprisingly, not particularly post-modern or new agey, it is the old claim that only the fundamental entities (perhaps he would rather say phenomena) of physics are real, and everything else is a sort of mirage based upon them. From this point of view it is also not surprising that he finds sufficient overlap between his theses and ancient religious wisdom, which also sees everyday objects and concerns as unreal shadows of an underlying, fundamental (usually mystical) reality.

The book is not done at this point, but I am done arguing with it.

I suppose my main gripes with this book could be condensed to

• I believe with George Box that “all models are wrong, some are useful”, and that this should be the stance of the scientist and generally is unless they are waylaid by the (reasonable, and probably unavoidable) temptation to make claims for the absolute truth of their preferred models
• Theise believes instead that he has discovered the true model of reality
• But all of his arguments along the way to this conclusion fall apart quickly under any sort of scrutiny

It also has a sandbox mode, which is briefly mentioned in the official rails docs, and is well-described here The Power of Rails Console with the –sandbox Option.

However, it comes with a big gotcha – one that is so problematic that the feature should in my opinion not even exist. (DHH’s philosophy is to “provide sharp knives” even if if the developer may cut themselves — fine but if we want to use that metaphor, this is a sharp knife that will eventually give you tetanus.)

The way it provides safety against accidentally modifying data is by wrapping everything you do in a database transaction, then just rolling that transaction back at the end of your session. This will work, and isn’t an unreasonable thing to do, but depending on what database you’re using, it will increase the likelihood of other queries and database operations failing or deadlocking.

Postgres table locks

With postgres, whenever you read from a table in a transaction, it will automatically take an ACCESS SHARE lock on that table (and its indexes, as far as I can tell), as per the docs on table-level locks. This is the lowest level of lock (as in, the one that conflicts with the fewest other lock modes), so it won’t prevent other transactions in other processes from normal querying, but it will prevent other processes from doing anything that requires an ACCESS EXCLUSIVE lock, for example many table and index operations. These will all fail to execute due to your sandboxed rails console, and will be taken if are you doing pretty much anything at all involving the database in your console session.

Postgres row locks

If, in your sandboxed rails console session, you accidentally or intentionally update some rows, this will (in addition to table level locks) take a row-level lock on the rows you are modifying. The precise lock level will depend on the nature of the field you are modifying (mostly, whether it is indexed or not) but either way, you can still run into a situation where your sandbox session causes other transactions to either hang or deadlock.

Consider the this case:

• your console session starts, and you modify user 1’s name to “foo”, acquiring lock 1a on this row
• an application session starts a transaction, and modifies user 2’s name to “bar”, acquiring lock 2a on this row
• your console session then modifies user 2’s name to “foo2”, attempting to acquire lock 2b on the same row; this has to wait on lock 2a to be released
• the same application transaction attempts to modify user 1’s name to “bar2”, attempting to acquire lock 1b on that row
• we now have a deadlock, as there is a cycle in the lock dependency graph. postgres will detect this and abort one of the transactions, with no guarantee as to which one will be selected for termination

Other reasons

Even if you aren’t concerned about interfering with DDL statements modifying the database schema, or you “know” you won’t be modifying data, so don’t have to worry about deadlocks, long running transactions are problematic for various reasons, including preventing vacuum and potentially introducing replication lag.

What about mysql and other databases?

My familiarity is primarily with postgres, and the mysql docs are not as clear on this point, but Transaction Isolation Levels does state that at the READ COMMITTED (and above, presumably) isolation level, innodb will hold “locks only for rows that it updates or deletes”, so I suspect rails console sandbox on mysql will have most of the same problems as postgres.

What to do instead?

Instead of this, you can create a read only postgres (or mysql) user, and have the rails console session use that instead. This could be configured by an environment variable, or an interactive startup script you run before enabling a console session.

This way, you get the safety of preventing any accidental data modification, while avoiding the most serious pitfalls: causing your application queries to deadlock or hang. However, your console queries will still take table level locks which will block table modification and related operations.

You can reduce that risk by keeping your rails console sessions short, but the best solution is just to not use the rails console in production at all.

• Ensure you have a staging (or similar) environment that includes a copy (or subset) of your production data, scrubbed of any sensitive info like credentials, access tokens, and PII.
• And/or, set up replication to another database and do all your queries on it, or even extract your data to a warehouse for analytical querying.

I’m not sure if the Spotify algorithm thinks I like the Alan Watts stuff, or if it just crops up so frequently in the type of music I do want to listen to that it is unavoiable.

When I click the button to tell Spotify “I don’t like this song” and it says “Okay, we won’t recommend things like this again” does it understand that by “things like this” I just mean “songs with Alan Watts”, literally just that?

You see my dilemma.

I was reading PostgreSQL Internals: 3 Things to Know About UPDATE Statements which begins with a nice, succinct discussion of how Postgres’ MVCC design results in write amplification. Basically, updates to rows are not made “in place” but rather by writing complete new versions of the row. Depending on the columns of a given table, and what is being updated, modifying a single byte (e.g. for a boolean column) can lead to a write amplification factor of several orders of magnitude, if the row also contains > 100-byte varchar columns, since a completely new copy of the row is written, along with the changes, and the old row is vacuumed later.

This reminded me of the article from Uber several years ago on why they switched from Postgres to MySQL, one of the main reasons being write amplification. But Postgres is not alone in using MVCC to provide ACID guarantees, MySQL uses MVCC as well, albeit a fairly different implementation. But then, if that’s the case, how does MySQL avoid write amplification.

MySQL write amplification?

According to MySQL’s docs on InnoDB Multi-Versioning, MySQL like Postgres uses multiple versions of the same row to service concurrent queries. But rather than writing entirely new row copies, MySQL uses undo logs which are maintained in a separate undo tablespace, used both for transaction rollbacks and to “build earlier versions of a row for a consistent read.”

This latter statement suggests that rather than storing an entire copy of old data, something like a changelog is maintained, which can be used to reconstruct the old version of the row at read time. The docs link to Consistent Nonlocking Reads, but this covers just the API/expected use rather than the implementation details. We also read that “The physical size of an undo log record in the rollback segment is typically smaller than the corresponding inserted or updated row” which further suggests that undo logs don’t contain full copies, but also that they aren’t just a changelog or ordered list of diffs either.

For a more in-depth view of InnoDB, we can turn to An In-Depth Analysis of UNDO Logs in InnoDB. Summarizing the relevant parts:

• InnoDB has 3 different types of undo record, and the “update” type appears to support a dynamic number of “update fields” which indeed do seem to function like a changelog. So if we are changing just one column on a row, rather than copying the whole row, InnoDB just creates a new undo log entry with the changed data.
• As explained in section 7 of the article, “Undo for MVCC”, as we suspected a transaction will (using its own transaction ID and the set of visible transaction IDs it is assigned at start time) walk back through a row’s undo log to find relevant versions and reconstruct them to return the correct result (according to the isolation level).

Performance implications

So we can surmise from this, with some educated guessing:

• Given that the MySQL undo log only writes the changed fields rather than copying whole rows, it is plausible that, depending on the nature of writes and table structure, this will indeed result in less write amplification than Postgres’ MVCC approach.
• But note that the MySQL undo log entries also contain a fair amount of header (and footer) metadata. For a small Postgres table (e.g. only a couple int/byte columns) it’s plausible that Postgres’ MVCC approach would actually use less space by copying the whole row.
• For situations with several open transactions, each of which have modified different columns of a row, a reader will probably have to do more work to reconstruct the result with MySQL vs Postgres.
  • On the other hand, the actual performance impacts of this will probably largely depend on how the data is stored. If old Postgres tuples are stored scattered across multiple pages, compared to MySQL which seems to store these undo log entries generally close together in a dedicated tablespace, then Postgres’ algorithm might be faster from a Big O notation point of view, but could perform worse in practice due to poor data locality, in other words, read amplification.
• Some other aspects of the MySQL approach seem like they could lead to problems, like limitations to the number of undo log entries which presumably limit the number and range of open transactions, but I’m not going to dig into that now.

But thinking back to the Uber article, the write amplification they are focused on is a very different type, having nothing explicitly to do with MVCC. Instead, it’s all about how Postgres must update all indexes when a row is modified, even if you are only modifying non-indexed columns. Postres has since introduced (or improved) support for something called Heap Only Tuple (HOT) updates, which are covered well here Fighting PostgreSQL write amplification with HOT updates.

My main takeaway from looking into this is that write amplification can and does happen in various places. “Solving” it will often be a matter of shuffling it around or increasing read amplification, rather than just making it go away. This reminds me of fan-out vs fan-in approaches to implementing “news feed” style features: do you make reads fast for everyone, by having every new post write to all the followers’ feeds, thereby making writes slow (fanout, write amplification)? Or do you make write fast, but reads slow by having them query all the followee’s post list (fan in, read amplification)? In practice I remember reading that Twitter did some of both, doing fanouts for most users, but doing fan-ins for high-follower-count users, so that Justin Bieber didn’t cause downtime whenever he tweeted.

I was browsing the web this morning and visited this page about the CHERI project. That page is hosted on the University of Cambridge website and should be completely safe to visit.

However, when I loaded it up on my iPhone, I saw a banner from Optimum advertising their free upgrade to faster internet. (I believe this is known as a DOM injection attack.) I had a literal WTF moment as this should be impossible. The website is served over HTTPS which should mean that, unless someone has broken the crypto underling modern TLS, even my ISP, which sees all of my internet traffic, should be incapable of decrypting and modifying any of that traffic.

But what if that website loads some resources not via HTTPS? I opened up the debugger’s networking tab on a Firefox desktop browser window, loaded that site, and sure enough, there is a plain HTTP request to a single resource. (Oddly I am not able to reproduce this consistently, nor at all with Chrome.)

So what probably happened is

• Optimum saw an unencrypted HTTP request
• fingerprinted my request by IP address and perhaps some other mechanisms
• determined that it had not already sent me that advertisement
• injected some Javascript into the content of that request
• the Javascript payload added a little banner at the top of the page I was viewing notifying me of the newer, faster internet speeds

When I got past the initial shock, I recalled that I had read about something like this before, and was able to track down a post from hackernews Xfinity is Man-in-the-Middle (MITM) Attacking my Internet from 2019. See also Comcast still uses MITM javascript injection to serve unwanted ads and messages from 2016.

This commenter on the Xfinity post purports to have knowledge of the details of how Comcast’s “User Messaging” feature works, via a network appliance that is operated by other ISPs as well. I’m unable to find any more information on the internet about what this appliance might be, who makes it, etc.

It isn’t hard to find software that does this sort of thing at the org/corporate level, e.g. Ivanti Traffic Manager can “decrypt and re-encrypt SSL traffic dynamically”, and there is probably a raft of Cisco hardware and software that can do the same things. But I haven’t been able to find anything specifically for ISPs that would support this functionality. The closest thing is this document on Cisco SD WAN technology that mentions that

An accepted packet is eligible to be modified by the additional parameters configured in the action portion of the policy configuration.

I would guess, however, that this is too “low level” an operation for the sort of functionality we’re talking about, and there is likely some kind of appliance dedicated to intercepting and modifying specifically HTTP traffic without worrying about modifying and reassembling individual packets.

But it seems equally plausible that it is just software running on the modem or router itself, which could periodically “phone home” to the ISP to check whether there are any “user messaging” campaign the ISP wants to run.

Unfortunately there probably are no ISPs (maybe Google Fiber?) whom you could trust to not engage in this sort of shady activity.

In Crafting Interpreters, this arises in the context of an optimization called NaN boxing that I won’t get into (though you can read more about it in the book here, or what is the purpose of NaN boxing?), but basically the idea is:

• the basic Value type of the Lox language from the book takes up 16 bytes: 8 bytes for the largest possible values (doubles and Obj pointers), plus 4 bytes for the type tag (since a dynamic language needs to keep track of this at runtime), and 4 bytes of padding.
• can we make this smaller somehow?
• well certain IEEE 754 NaN layouts have a bunch of unused bits, maybe we can use those, since a IEEE 754 float is also 64 bits, enough for floats and pointers, and since most 64-bit architectures actually only use 48 bits for a pointer, we still have 16 bits to store type information.

To do this, we’ll need to convince the C compiler (since this particular Lox implementation is written in C) to treat this 64 bit value as variously a float or uint64_t, and this is the essence of type punning.

However, in a side note Nystrom mentions

Spec authors don’t like type punning because it makes optimization harder. A key optimization technique is reordering instructions to fill the CPU’s execution pipelines. A compiler can reorder code only when doing so doesn’t have a user-visible effect, obviously.

Pointers make that harder. If two pointers point to the same value, then a write through one and a read through the other cannot be reordered. But what about two pointers of different types? If those could point to the same object, then basically any two pointers could be aliases to the same value. That drastically limits the amount of code the compiler is free to rearrange.

To avoid that, compilers want to assume strict aliasing—pointers of incompatible types cannot point to the same value. Type punning, by nature, breaks that assumption.

https://craftinginterpreters.com/optimization.html#numbers

Unstated here is that “cannot point to the same value” means that if they do, this will invoke dreaded undefined behaviour (there are many thorough references to UB, see John Regehr’s Guide to undefined behaviour in C and C++ for a particularly good one, also Garbage In, Garbage Out: Arguing about UB (youtube presentation), and also Falsehoods Programmers Believe about UB).

This reminded me of having worked through the classic Beej’s Guide to Network Programming, which I still think is a great (if perhaps outdated) intro to sockets and programming in C. (It was also the basis for my attempt to write a basic tcp server in Fortran). An essential element of the magic of BSD sockets you are introduced to in that guide is the addrinfo struct, which looks like this:

struct addrinfo {
    int              ai_flags;     // AI_PASSIVE, AI_CANONNAME, etc.
    int              ai_family;    // AF_INET, AF_INET6, AF_UNSPEC
    int              ai_socktype;  // SOCK_STREAM, SOCK_DGRAM
    int              ai_protocol;  // use 0 for "any"
    size_t           ai_addrlen;   // size of ai_addr in bytes
    struct sockaddr *ai_addr;      // struct sockaddr_in or _in6
    char            *ai_canonname; // full canonical hostname

    struct addrinfo *ai_next;      // linked list, next node
};

Note that the ai_addr pointer to struct sockaddr is said to be either sockaddr_in or sockaddr_in6 – this allows the code to work with either ipv4 or ipv6.

When we are setting up a server to listen on a port, we first need to call bind and pass it, among other things, a pointer to a struct sockaddr – but we’ll actually have either a sockaddr_in for ipv4 or sockaddr_in6 for ipv6, which will have to be cast first to a generic sockaddr.

Beej notes that the “old” way (maybe this is what the guide had when I first went through it?) was like this:

bind(sockfd, (struct sockaddr *)&my_addr, sizeof my_addr);

But this seems to violate the rule of strict aliasing outlined above: we now have two pointers to the same memory of different types, one is a struct sockaddr_in (assuming we’re using ipv4) and the one passed to bind is a cast to a struct sockaddr. How is this not introducing undefined behaviour?

Am I alone in my confusion?

Turns out some other folks on the internet had the same questions:

• Berkeley sockets, breaking aliasing rules?
• Am I crazy or is it impossible to work with BSD sockets in C++ without UB?
• How to cast sockaddr_storage and avoid breaking strict-aliasing rules (this one specifically mention’s Beej’s guide)

One Stack Overflow response states:

The cast itself in that line does not break the strict aliasing rule. The rule is only broken if the implementation of bind() dereferences that pointer without converting it back to the right type.

Any strict aliasing problems there are problems for the implementer of bind(), not the user.

https://stackoverflow.com/a/17988634

So this means perhaps our code isn’t causing UB, but the bind implementation itself may be? (Looking at __inet_bind, which seems to be what ultimately does the heavy lifting for ipv4 sockets, the code is definitely dereferencing and reading from the pointer.) That still seems like a very undesirable situation.

Responses to the reddit post above (“Am I crazy…”) echo a similar idea:

And bind() gets a free pass for either one of two reasons: it’s written in C and thus is only bound by the rules of C, and/or it’s provided by your implementation, and those don’t have to abide by the same rules as us mere mortals.

If *you* try to read or write it through the aliased type, yes, it is UB.

Fortunately, you’re not, bind() is a black box to the compiler.

I didn’t find either of these responses entirely satisfying.

• Just because bind is a syscall provided by the kernel doesn’t seem like it should mean it is exempt from C’s aliasing rules.
• Again, even though bind is in this case a “black box” to the compiler, are we to just trust that the compiled bind code will not be subject to UB?

But it turns out that there is some truth to these responses, as per this rant from Linus on aliasing, in which he says the kernel uses the GCC flag -fno-strict-aliasing precisely to avoid issues like this. (Of course, “us mere mortals” are free to use this flag too.)

Linus also comes out strongly in support of using C unions to do type punning, which has apparently been the approved way to do this sort of thing in GCC for some time.

So does this mean that it is not possible to write bind (and likely related kernel networking code) without using potentially-standard-violating compiler extensions? It seems like the answer would have been “yes”, prior to the C99 and C11 standards, which added official support for type punning through unions (though there appears to still be disagreement by some over how official the support is).

Additional reading

• What is the strict aliasing rule and why do we care?
• Type punning, strict aliasing, and optimization

An oft-cited benefit of paying down technical debt is that it will allow you to move faster. This is typically characterized in terms of increased velocity.

But if you are measuring velocity by factoring in “accidental complexity”, then when you remove the source of that accidental complexity, your velocity will not actually appear to improve. This point is well put by Moneyball for software teams: an imperfect heuristic for quantifying dev performance. This runs contrary to what we’d generally expect from a measure of developer velocity, and also contrary to the business case that is usually made for paying down technical debt: to simplify, reduced debt = increased velocity.

The aforementioned article presents a reasonable way out of the impasse of a) needing to factor tech debt into your estimates, b) wanting tech debt reductions to be visible as increased velocity i.e. more “story points” per cycle. (Whether it is necessary or desirable to adopt this approach is debatable, and personally I’d still bias towards avoiding measurement of velocity at all, and having developers do PERT estimates only as necessary. This seems to be less common outside small, high trust teams.)

But even if you are measuring velocity this way, it is entirely possible that you could pay down all the tech debt and still not have your velocity  (as a measurement of essential complexity) increase:

• new sources of accidental complexity can emerge that are not at all under your control, whether from vendors, dependencies that are suddenly EOLd or have hard to patch security flaws, and so on
• eliminating a given set of sources of accidental complexity may enable your team to take on other types of projects that are subject to additional kinds of accidental complexity. These new tasks or projects may have higher business value, but that won’t be reflected on your velocity (or probably reflected anywhere because We can’t measure developer productivity). This is reminiscent of the way in which improving tail latency can actually cause median latency to increase: previously, requests from slow clients would time out or never be initiated in the first place, but after improving performance for those clients, their requests will now succeed, albeit at latencies greater than your current median.
• even if your velocity does go up, will the signal be strong enough to be distinguishable from statistical noise (maybe one of your senior developers finally got their kid into an after school program and now they are back to “full performance” level)? do you have tools sophisticated enough to discern this?

Returning to the moneyball comparison from above: sport performance is generally easily and unambiguously measurable (did the ball go in the net?), and what it means to do well can be reduced to “win competitions”. Software development is inherently harder to measure, and has many axes along which something can be called “a success”. Moreover, whether a software project turns out to have been successful may not be discernible for months or years down the road, for reasons business or market reasons that have nothing to do with the development project “in itself”. For these reasons, it is at best a waste of time, if not actively harmful, to foist such methods into software management.

That said, my thoughts on this topic are always challenged by the ideas of Hubbard (e.g. How to Measure Anything) whose main argument boils down to, if it matters you can notice it, and if you can notice it, you can (at least sort of) measure it.

Much of it is well-argued and compelling. I need to consider it further before I can say how much I agree or disagree.

But one part is so poorly argued it merits mentioning. Henderson writes:

There are other examples of luxury beliefs as well, such as the downplaying of individual agency in shaping life outcomes.

A 2019 study led by Joseph Daniels at Marquette University was published in the journal of Applied Economics Letters.

They found that individuals with higher income or a higher social status were the most likely to say that success results from luck and connections rather than hard work, while low-income individuals were more likely to say success comes from hard work and individual effort.

Well, which belief is more likely to be true?

Plenty of research indicates that compared with an external locus of control, an internal locus of control is associated with better academic, economic, health, and relationship outcomes. Believing you are responsible for your life’s direction rather than external forces appears to be beneficial.

Here’s the late Stanford psychology professor Albert Bandura. His vast body of research showed that belief in personal agency, or what he described as “self-efficacy,” has powerful positive effects on life outcomes.

Undermining self-efficacy will have little effect on the rich and educated, but will have pronounced effects for the less fortunate.

According to Henderson this is evidence that this belief is a “luxury belief”.  I have a few problems with this section

1. The framing

Henderson asks “which belief is more likely to be true?” and then goes on to discuss the benefits of believing one vs the other, which patently has nothing to do with a belief being true. Was this just bad editing? This is such a non sequitur, I don’t know what to make of it.

There is plenty of research (though what I was able to find in a brief internet search generally seemed low quality) on whether hard work vs luck vs other factors are contributors to success. Henderson does not consider any of that here.

Additionally, there is a large body of research, as Henderson states, on the benefits of internal locus of control (e.g. The happy personality) and it does support the idea that internal locus of control is correlated with happiness and achievement. But it seems like a bit of a jump to go from “people emphasize the important of luck and connections in having success” to “people have weak/less internal locus of control and therefore are more likely to be unhappy and achieve less”, which is the conclusion that Henderson implies. The data used in the paper under discussion come from the World Values Survey, specifically the questions here are:

An answer of “in the long run, hard work usually brings a better life’ carries a value of one (1) in the survey and an answer of ‘hard work doesn’t generally bring success – it’s
more a matter of luck and connections’ carries a value
of ten (10).

To me this seems more about achieving societal markers of success (income, savings, housing, cars), while internal vs external locus of control is more about an individual believing they are responsible for and in control of what they do.

I am obviously not an expert on these terms, nor deeply familiar with the relevant theory and research, so I may be wrong here, but it seems like a big analytical leap that Henderson is making here. The relevance of this research to Henderson’s claim needs to be argued but he assumes it is obvious and uncontroversial.

2. The research

Reading the original research paper, we see the following correlations between belief in luck/connections (there are some others as well, but these are the ones that stand out as most problematic for Henderson’s case, in my opinion):

• with social class: 0.0834
• with gender: 0.0678
• with country’s GDP: 0.0844
• with income: 0.0077

Henderson seems to latch onto the correlation with social class, but this  leaves us with a few head scratchers, given his conclusions:

• if this is a luxury belief, why is the correlation with income so much weaker than the correlation with social class?
• why is there a correlation with GDP? are entire populations of wealthier countries more likely to adopt this view in order to distinguish themselves from other countries?
• what’s with women? why are they so more drawn to this luxury belief than men?

It seems like Henderson had an idea, went around looking for evidence, found something resembling evidence if you squint right and ignore some things, and then just ran with it.

I understand that the piece of writing I’m discussing is not academic per se, but given that he frames it as a transcript of a talk he delivered at a “behavioural science festival” I found the lack of rigor in this part disappointing. It casts a shadow over the rest of the argument.

To be honest, if I had noticed the handy wiring guide Crutchfield mentioned in my purchase email, I probably wouldn’t have had several of these difficulties, but I somehow missed it until just now.

First of all, you are going to have to solder some wires

If you’ve ever done a project like this, that fact is probably obvious, but I naively assumed that the Crux SWRMZ-64C Wiring Interface that came with my package would mean the whole thing was just plug n play. I got all the dashboard panels popped off and the old radio removed before I realized this, then ran out of time and had to put everything back together and wait until the next weekend when I had time to tackle the soldering.

Fortunately I had recently acquired a soldering gun and been getting some practice. You could maybe use crimp connectors or some other method instead of soldering, but the soldering was not that difficult, provides a better connection, and is more fun.

Removing the panels to get at the guts

This part was actually kind of interesting, turns out most of the panels on your car just pop off with the help of some trim panel tools and a philips head screwdriver. Again, something that is probably obvious to many folks.

One bit of advice I got from somewhere on the internet for this step was: after you’ve unscrewed the gear shift nob and popped off the gear shift trim panel, you can press down a little white lever with your finger or screwdriver to shift the car into neutral, which makes it easier to remove the climate control assembly. You could probably just do this at the outset. Obviously make sure your parking break is on.

Soldering the stereo wires to the wiring interface

Basically you can just follow the wiring guide from Crutchfield here, and match similarly-colored wires to each other. However

1. The manufacturer’s manual that comes with the wiring interface incorrectly identifies one of the wires as blue, when it is actually (in my case) brown. This is the Steering Wheel Control (SWC) Phone wire, which won’t be used for anything if you have the same year/model Mazda and this stereo. I soldered it to the brown wire on the stereo which turned out to be unnecessary, but not actively harmful.
2. The blue wire of the power antennae is similarly useless for this car, as the antennae doesn’t require power. You can just plug the antennae jack into the back of the radio, and top the blue wire off with electrical tape. Easy.
3. Because Steering Wheel Controls are simply incompatible with this stereo, flipping the dip switches on the interface module is unnecessary. Additionally, connecting the two wires that come out the other side of this module is also unnecessary, you can just toss those away, or keep them in your electronic junk box forever until you die because you will never use them for anything so just throw them away.
4. There will be an extra module just hanging out that connected to your old stereo, but has nowhere to go with the new one. I’m not sure what this is for, possibly SWC, but either way you can just ignore it and leave it hanging around. This little white guy here, no home for you any more buddy.

Mounting the stereo

I mostly ignored the mounting instructions that came with the stereo itself as they were confusing and seemingly incompatible with the Metra 99-7521B Dash Kit that was part of the package, and necessary to ensure you don’t have a gaping hole remaining in your dashboard.

I don’t have any good advice for this part, other than “just do what seems like it will work, ensuring that the unit is securely attached, without putting undue stress on the wires”.

Hearing this thing actually play a CD was probably the most satisfying feeling of accomplishment I’ve had in years.