The Nonlinear Library: LessWrong Weekly The Nonlinear Fund

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Golden Age of Building? Excerpts and lessons from Empire State, Pentagon, Skunk Works and SpaceX, published by jacobjacob on September 1, 2023 on LessWrong.
Patrick Collison has a fantastic list of examples of people quickly accomplishing ambitious things together since the 19th Century. It does make you yearn for a time that feels... different, when the lethargic behemoths of government departments could move at the speed of a racing startup:
[...] last century, [the Department of Defense] innovated at a speed that puts modern Silicon Valley startups to shame: the Pentagon was built in only 16 months (1941-1943), the Manhattan Project ran for just over 3 years (1942-1946), and the Apollo Program put a man on the moon in under a decade (1961-1969). In the 1950s alone, the United States built five generations of fighter jets, three generations of manned bombers, two classes of aircraft carriers, submarine-launched ballistic missiles, and nuclear-powered attack submarines.
[Note: that paragraph is from a different post.]
Inspired by partly by Patrick's list, I spent some of my vacation reading and learning about various projects from this Lost Age. I then wrote up a memo to share highlights and excerpts with my colleagues at Lightcone.
After that, some people encouraged me to share the memo more widely -- and I do think it's of interest to anyone who harbors an ambition for greatness and a curiosity about operating effectively.
How do you build the world's tallest building in only a year? The world's largest building in the same amount of time? Or America's first fighter jet in just 6 months?
How??
Writing this post felt like it helped me gain at least some pieces of this puzzle. If anyone has additional pieces, I'd love to hear them in the comments.
Empire State Building
The Empire State was the tallest building in the world upon completion in April 1931. Over my vacation I read a rediscovered 1930s notebook, written by the general contractors themselves. It details the construction process and the organisation of the project.
I will share some excerpts, but to contextualize them, consider first some other skyscrapers built more recently:
Design startConstruction endTotal timeBurj Khalifa200420106 yearsShanghai Tower200820157 yearsAbraj Al-Balt2002201210 yearsOne World Trade Center200520149 yearsNordstrom Tower2010202010 yearsTaipei 101199720047 years
(list from skyscrapercenter.com)
Now, from the Empire State book's foreword:
The most astonishing statistics of the Empire State was the extraordinary speed with which it was planned and constructed. [...] There are different ways to describe this feat. Six months after the setting of the first structural columns on April 7, 1930, the steel frame topped off on the eighty-sixth floor. The fully enclosed building, including the mooring mast that raised its height to the equivalent of 102 stories, was finished in eleven months, in March 1931. Most amazing though, is the fact that within just twenty months -- from the first signed contractors with the architects in September 1929 to opening-day ceremonies on May 1, 1931 -- the Empire State was designed, engineered, erected, and ready for tenants.
Within this time, the architectural drawings and plans were prepared, the Vicitorian pile of the Waldorf-Astoria hotel was demolished [demolition started only two days after the initial agreement was signed], the foundations and grillages were dug and set, the steel columns and beams, some 57,000 tons, were fabricated and milled to precise specifications, ten million common bricks were laid, more than 62,000 cubic yards of concrete were poured, 6,400 windows were set, and sixty-seven elevators were installed in seven miles of shafts. At peak activity, 3,500 workers were employed on site, and the frame ros

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against Almost Every Theory of Impact of Interpretability, published by Charbel-Raphaël on August 17, 2023 on LessWrong.
Epistemic Status: I believe I am well-versed in this subject. I erred on the side of making claims that were too strong and allowing readers to disagree and start a discussion about precise points rather than trying to edge-case every statement. I also think that using memes is important because safety ideas are boring and anti-memetic. So let's go!
Many thanks to @scasper, @Sid Black , @Neel Nanda , @Fabien Roger , @Bogdan Ionut Cirstea, @WCargo, @Alexandre Variengien, @Jonathan Claybrough, @Edoardo Pona, @Andrea_Miotti, Diego Dorn, Angélina Gentaz, Clement Dumas, and Enzo Marsot for useful feedback and discussions.
When I started this post, I began by critiquing the article A Long List of Theories of Impact for Interpretability, from Neel Nanda, but I later expanded the scope of my critique. Some ideas which are presented are not supported by anyone, but to explain the difficulties, I still need to 1. explain them and 2. criticize them. It gives an adversarial vibe to this post. I'm sorry about that, and I think that doing research into interpretability, even if it's no longer what I consider a priority, is still commendable.
How to read this document? Most of this document is not technical, except for the section "What does the end story of interpretability look like?" which can be mostly skipped at first. I expect this document to also be useful for people not doing interpretability research. The different sections are mostly independent, and I've added a lot of bookmarks to help modularize this post.
If you have very little time, just read (this is also the part where I'm most confident):
Auditing deception with Interp is out of reach (4 min)
Enumerative safety critique (2 min)
Technical Agendas with better Theories of Impact (1 min)
Here is the list of claims that I will defend:
(bolded sections are the most important ones)
The overall Theory of Impact is quite poor
Interp is not a good predictor of future systems
Auditing deception with interp is out of reach
What does the end story of interpretability look like? That's not clear at all.
Enumerative safety?
Reverse engineering?
Olah's Interpretability dream?
Retargeting the search?
Relaxed adversarial training?
Microscope AI?
Preventive measures against Deception seem much more workable
Steering the world towards transparency
Cognitive Emulations - Explainability By design
Interpretability May Be Overall Harmful
Outside view: The proportion of junior researchers doing Interp rather than other technical work is too high
So far my best ToI for interp: Nerd Sniping?
Even if we completely solve interp, we are still in danger
Technical Agendas with better Theories of Impact
Conclusion
Note: The purpose of this post is to criticize the Theory of Impact (ToI) of interpretability for deep learning models such as GPT-like models, and not the explainability and interpretability of small models.
The emperor has no clothes?
I gave a talk about the different risk models, followed by an interpretability presentation, then I got a problematic question, "I don't understand, what's the point of doing this?" Hum.
Feature viz? (left image) Um, it's pretty but is this useful? Is this reliable?
GradCam (a pixel attribution technique, like on the above right figure), it's pretty. But I've never seen anybody use it in industry. Pixel attribution seems useful, but accuracy remains the king.
Induction heads? Ok, we are maybe on track to retro engineer the mechanism of regex in LLMs. Cool.
The considerations in the last bullet points are based on feeling and are not real arguments. Furthermore, most mechanistic interpretability isn't even aimed at being useful right now. But in the rest of

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My current LK99 questions, published by Eliezer Yudkowsky on August 1, 2023 on LessWrong.
So this morning I thought to myself, "Okay, now I will actually try to study the LK99 question, instead of betting based on nontechnical priors and market sentiment reckoning." (My initial entry into the affray, having been driven by people online presenting as confidently YES when the prediction markets were not confidently YES.) And then I thought to myself, "This LK99 issue seems complicated enough that it'd be worth doing an actual Bayesian calculation on it"--a rare thought; I don't think I've done an actual explicit numerical Bayesian update in at least a year.
In the process of trying to set up an explicit calculation, I realized I felt very unsure about some critically important quantities, to the point where it no longer seemed worth trying to do the calculation with numbers. This is the System Working As Intended.
On July 30th, Danielle Fong said of this temperature-current-voltage graph,
'Normally as current increases, voltage drop across a material increases. in a superconductor, voltage stays nearly constant, 0. that appears to be what's happening here -- up to a critical current. with higher currents available at lower temperatures deeply in the "fraud or superconduct" territory, imo. like you don't get this by accident -- you either faked it, or really found something.'
The graph Fong is talking about only appears in the initial paper put forth by Young-Wan Kwon, allegedly without authorization. A different graph, though similar, appears in Fig. 6 on p. 12 of the 6-author LK-endorsed paper rushed out in response.
Is it currently widely held by expert opinion, that this diagram has no obvious or likely explanation except "superconductivity" or "fraud"? If the authors discovered something weird that wasn't a superconductor, or if they just hopefully measured over and over until they started getting some sort of measurement error, is there any known, any obvious way they could have gotten the same graph?
One person alleges an online rumor that poorly connected electrical leads can produce the same graph. Is that a conventional view?
Alternatively: If this material is a superconductor, have we seen what we expected to see? Is the diminishing current capacity with increased temperature usual? How does this alleged direct measurement of superconductivity square up with the current-story-as-I-understood-it that the material is only being very poorly synthesized, probably only in granules or gaps, and hence only detectable by looking for magnetic resistance / pinning?
This is my number-one question. Call it question 1-NO, because it's the question of "How does the NO story explain this graph, and how prior-improbable or prior-likely was that story?", with respect to my number one question.
Though I'd also like to know the 1-YES details: whether this looks like a high-prior-probability superconductivity graph; or a graph that requires a new kind of superconductivity, but one that's theoretically straightforward given a central story; or if it looks like unspecified weird superconductivity, with there being no known theory that predicts a graph looking roughly like this.
What's up with all the partial levitation videos? Possibilities I'm currently tracking:
2-NO-A: There's something called "diamagnetism" which exists in other materials. The videos by LK and attempted replicators show the putative superconductor being repelled from the magnet, but not being locked in space relative to the magnet. Superconductors are supposed to exhibit Meissner pinning, and the failure of the material to be pinned to the magnet indicates that this isn't a superconductor. (Sabine Hossenfelder seems to talk this way here. "I lost hope when I saw this video; this doesn

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Yes, It's Subjective, But Why All The Crabs?, published by johnswentworth on July 28, 2023 on LessWrong.
Crabs
Nature really loves to evolve crabs.
Some early biologist, equipped with knowledge of evolution but not much else, might see all these crabs and expect a common ancestral lineage. That's the obvious explanation of the similarity, after all: if the crabs descended from a common ancestor, then of course we'd expect them to be pretty similar.
. but then our hypothetical biologist might start to notice surprisingly deep differences between all these crabs. The smoking gun, of course, would come with genetic sequencing: if the crabs' physiological similarity is achieved by totally different genetic means, or if functionally-irrelevant mutations differ across crab-species by more than mutational noise would induce over the hypothesized evolutionary timescale, then we'd have to conclude that the crabs had different lineages. (In fact, historically, people apparently figured out that crabs have different lineages long before sequencing came along.)
Now, having accepted that the crabs have very different lineages, the differences are basically explained. If the crabs all descended from very different lineages, then of course we'd expect them to be very different.
. but then our hypothetical biologist returns to the original empirical fact: all these crabs sure are very similar in form. If the crabs all descended from totally different lineages, then the convergent form is a huge empirical surprise! The differences between the crab have ceased to be an interesting puzzle - they're explained - but now the similarities are the interesting puzzle. What caused the convergence?
To summarize: if we imagine that the crabs are all closely related, then any deep differences are a surprising empirical fact, and are the main remaining thing our model needs to explain. But once we accept that the crabs are not closely related, then any convergence/similarity is a surprising empirical fact, and is the main remaining thing our model needs to explain.
Agents
A common starting point for thinking about "What are agents?" is Dennett's intentional stance:
Here is how it works: first you decide to treat the object whose behavior is to be predicted as a rational agent; then you figure out what beliefs that agent ought to have, given its place in the world and its purpose. Then you figure out what desires it ought to have, on the same considerations, and finally you predict that this rational agent will act to further its goals in the light of its beliefs. A little practical reasoning from the chosen set of beliefs and desires will in most instances yield a decision about what the agent ought to do; that is what you predict the agent will do.
Daniel Dennett, The Intentional Stance, p. 17
One of the main interesting features of the intentional stance is that it hypothesizes subjective agency: I model a system as agentic, and you and I might model different systems as agentic.
Compared to a starting point which treats agency as objective, the intentional stance neatly explains many empirical facts - e.g. different people model different things as agents at different times. Sometimes I model other people as planning to achieve goals in the world, sometimes I model them as following set scripts, and you and I might differ in which way we're modeling any given person at any given time. If agency is subjective, then the differences are basically explained.
. but then we're faced with a surprising empirical fact: there's a remarkable degree of convergence among which things people do-or-don't model as agentic at which times. Humans yes, rocks no. Even among cases where people disagree, there are certain kinds of arguments/evidence which people generally agree update in a

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Grantmaking is Funding-Limited Right Now, published by johnswentworth on July 19, 2023 on LessWrong.
For the past few years, I've generally mostly heard from alignment grantmakers that they're bottlenecked by projects/people they want to fund, not by amount of money. Grantmakers generally had no trouble funding the projects/people they found object-level promising, with money left over. In that environment, figuring out how to turn marginal dollars into new promising researchers/projects - e.g. by finding useful recruitment channels or designing useful training programs - was a major problem.
Within the past month or two, that situation has reversed. My understanding is that alignment grantmaking is now mostly funding-bottlenecked. This is mostly based on word-of-mouth, but for instance, I heard that the recent lightspeed grants round received far more applications than they could fund which passed the bar for basic promising-ness. I've also heard that the Long-Term Future Fund (which funded my current grant) now has insufficient money for all the grants they'd like to fund.
I don't know whether this is a temporary phenomenon, or longer-term. Alignment research has gone mainstream, so we should expect both more researchers interested and more funders interested. It may be that the researchers pivot a bit faster, but funders will catch up later. Or, it may be that the funding bottleneck becomes the new normal. Regardless, it seems like grantmaking is at least funding-bottlenecked right now.
Some takeaways:
If you have a big pile of money and would like to help, but haven't been donating much to alignment because the field wasn't money constrained, now is your time!
If this situation is the new normal, then earning-to-give for alignment may look like a more useful option again. That said, at this point committing to an earning-to-give path would be a bet on this situation being the new normal.
Grants for upskilling, training junior people, and recruitment make a lot less sense right now from grantmakers' perspective.
For those applying for grants, asking for less money might make you more likely to be funded. (Historically, grantmakers consistently tell me that most people ask for less money than they should; I don't know whether that will change going forward, but now is an unusually probable time for it to change.)
Note that I am not a grantmaker, I'm just passing on what I hear from grantmakers in casual conversation. If anyone with more knowledge wants to chime in, I'd appreciate it.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Accidentally Load Bearing, published by jefftk on July 13, 2023 on LessWrong.
Sometimes people will talk about Chesterton's Fence, the idea that if you want to change something - removing an apparently useless fence - you should first determine why it was set up that way:
The gate or fence did not grow there. It was not set up by somnambulists who built it in their sleep. It is highly improbable that it was put there by escaped lunatics who were for some reason loose in the street. Some person had some reason for thinking it would be a good thing for somebody. And until we know what the reason was, we really cannot judge whether the reason was reasonable. It is extremely probable that we have overlooked some whole aspect of the question, if something set up by human beings like ourselves seems to be entirely meaningless and mysterious. - G. K. Chesterton, The
Drift From Domesticity
Figuring out something's designed purpose can be helpful in evaluating changes, but a risk is that it puts you in a frame of mind where what matters is the role the original builders intended.
A few years ago I was rebuilding a bathroom in our house, and there was a vertical stud that was in the way. I could easily tell why it was there: it was part of a partition for a closet. And since I knew its designed purpose and no longer needed it for that anymore, the Chesterton's Fence framing would suggest that it was fine to remove it. Except that over time it had become accidentally load bearing: through other (ill conceived) changes to the structure this stud was now helping hold up the second floor of the house. In addition to considering why something was created, you also need to consider what additional purposes it may have since come to serve.
This is a concept I've run into a lot when making changes to complex computer systems. It's useful to look back through the change history, read original design documents, and understand why a component was built the way it was. But you also need to look closely at how the component integrates into the system today, where it can easily have taken on additional roles.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.