Hey Mason! It's Thursday, July ninth, twenty twenty-six, and we've got a full runway today. First up, a quick radar sweep through the newest research that made the cut this week — the stuff worth keeping, distilled fast so you can get back to the road. From there we're pulling way back for a deep dive into geospatial AI — where models meet maps, and what that means for anyone trying to reason about the physical world at scale. And we'll close out with a lap around the week in general AI, the releases and shifts worth knowing about even if you only have five minutes to spare. Plenty of ground to cover, so let's get moving with the radar briefing.

A few things worth flagging from the last few days, and they cluster into some pretty familiar buckets.

First, on the retrieval side: a group looked at retrieval augmented generation for public health question answering, systematically testing different retrieval configurations and measuring how faithful the answers stayed to the source documents. That's directly relevant groundwork for anyone building a RAG pipeline meant to stay grounded in a fixed, trusted set of texts rather than drifting into the model's general knowledge — which is exactly the constraint at play when answers need to trace back to community-approved books rather than whatever the underlying model happened to memorize.

Second, a paper on Indic AI reframes the problem away from just "low resource language coverage" and toward cultural heritage preservation specifically — treating linguistic diversity and community ownership of data as the actual goal, not just a side effect of better benchmarks. That framing maps closely onto language sovereignty work for Lakota: the point isn't only technical parity with high resource languages, it's who controls the data and how faithfully the technology serves the community's own priorities.

Third, there's a cluster of transfer learning work. One study looked at whether reasoning conducted in English can improve performance on multiple choice questions in low resource languages, essentially asking whether you can borrow reasoning capability across languages even when the target language itself has little training data. A related paper tackled cross lingual transfer for low resource speech recognition, moving from Sinhala to Dhivehi — two related languages where the richer one's data helps bootstrap the sparser one. Both are relevant to the data efficient adaptation problem: how much can be borrowed from a better resourced neighbor or from the model's general reasoning ability, versus what has to be built from Lakota specific data no matter what.

Fourth, two papers push at evaluation gaps. One introduces a benchmark for mathematical reasoning specifically in underrepresented languages, since most math evaluation work quietly assumes English or a handful of major languages. The other addresses a digraphic translation problem in Mongolian — where the same language is written in more than one script and that alone introduces ambiguity a model has to resolve before translation even starts. That last one is worth sitting with, since orthographic variation is a live issue in Indigenous language work too, and it's a reminder that evaluation designed for monoscript, high resource languages can miss exactly the kind of complexity that matters most here.

Nothing this cycle is a must-read-today item, but the throughline is consistent: the field is slowly building the evaluation and transfer-learning machinery for languages that don't have huge corpora, which is the same terrain this project sits on.

Sticking with pixel-level embeddings for a moment, there's a new paper out of Cambridge worth sitting with if you're the kind of person who trains on Sentinel-2 tiles at home, because it's really asking the question you've probably asked yourself: when you throw more compute at a self-supervised model, where should that compute actually go.

The problem this paper is chasing is specific to earth observation and doesn't show up the same way in ordinary computer vision. With natural images, you mostly assume a photo is a photo, taken once, in focus, at a stable resolution. Satellite pixels don't behave that way. A single ten-meter patch of ground gets visited by Sentinel-2 on one irregular schedule and by Sentinel-1 radar on a completely different one, clouds wipe out huge chunks of the optical record, and if you build your training set from cloud-free composites, as a lot of pipelines do, you've thrown away the seasonal, phenological signal that most downstream tasks — crop type, land cover change, biomass — actually depend on. So training a foundation model here means training through irregularity and occlusion, not around it, and that changes what "more compute" should buy you.

The approach they take is what they call downstream-driven scaling. Instead of the usual move — train a bunch of models, fit a scaling law to how low the pretraining loss goes, and assume lower loss means a better model — they ran a very large controlled sweep, on the order of several hundred training runs on a big GPU cluster, and evaluated every single one of them on around fifteen separate downstream tasks: classification, segmentation, change detection, regression. Architecturally these are all variants of the same family used in the earlier TESSERA model — a per-pixel encoder trained with a redundancy-reduction objective called Barlow Twins, which basically pushes the model to produce embedding dimensions that are decorrelated from each other rather than redundant. Two augmented views of the same pixel's yearly time series go in, and the loss pushes their embeddings to agree while discouraging the dimensions within an embedding from just copying each other.

Now, the finding that matters most here, and honestly the one I'd flag to anyone building their own encoders: pretraining loss and downstream task performance were almost uncorrelated across those hundreds of runs. Barely any relationship at all. The reason given is intuitive once you hear it — cloud cover and orbital sampling patterns dominate the variance in the raw input, so a model can get very good at reducing redundancy between two augmented views by learning invariances that have nothing to do with the physical land-cover signal you actually care about. It's optimizing the objective correctly, it's just optimizing the wrong thing for your purposes. The practical cost of trusting loss anyway is large — the paper estimates you'd waste something like two and a half times the compute if you select models by loss instead of by directly checking downstream scores.

The second big finding is about where to put your compute budget once you're measuring the right thing. They found that as you scale up total training compute, the encoder itself and the amount of training data should grow together, roughly matching the compute-optimal balance you'd see in language model scaling laws. But the projector — the small network used only during pretraining to shape the embedding space — should essentially stay fixed in size no matter how much compute you have. Bigger backbone, more data, same small projector. That's a pretty clean, actionable rule if you're deciding how to spend a fixed GPU budget on your own encoder: don't scale the throwaway part.

From there they trained a large teacher model, on the order of a billion parameters, purely as an internal representation source — not something anyone is expected to run themselves — and then distilled it down into a family of much smaller student models, down to about twenty-one million parameters for their recommended default. This is where the paper gets genuinely useful for someone doing embeddings-as-data work: the students are trained to reconstruct nested prefixes of the teacher's embedding, so a sixteen-dimensional slice, a thirty-two dimensional slice, and so on up to the full width, all come out of one trained model, and you just truncate depending on your storage budget. They report the smallest prefix keeping the large majority of full performance — around ninety percent-plus — at a fraction of the storage cost. Interestingly, they tried getting this nested structure directly during self-supervised pretraining by adding losses on each prefix, and it just didn't work — the redundancy-reduction objective identifies a good subspace but only up to rotation, so there's no natural ordering of which dimensions matter most until you distill against a fixed teacher target. That ordering signal only shows up once there's an actual reference embedding to reconstruct.

On honest limitations: this is one architecture family, one objective function, evaluated on one broad but still bounded task suite. The scaling exponents they fit are specific to pixel-wise Barlow Twins encoders on Sentinel-1 and Sentinel-2 data — they're explicit that they're not claiming a universal law for all earth observation models, and past a point they're extrapolating the encoder-scaling curve out to the billion-parameter teacher rather than having swept that size directly. The downstream task suite, however broad, is still a proxy, and a redundancy-reduction objective evaluated against fifteen or so tasks could still be missing failure modes on tasks nobody tested.

So what's actually worth trying if you're working with your own Sentinel imagery. The compute allocation rule is the most transferable idea even outside this exact model family: if you're ever training or fine-tuning your own encoder and deciding whether to make the backbone bigger or the downstream head bigger, this is direct evidence that backbone capacity plus more training data is where the return lives, and a bigger projection head is close to wasted compute. Second, if this team does release those global annual embeddings as promised, the Matryoshka prefix structure means you can start experimentating with the smallest, cheapest slice for a rough pass and only pull the full-width embedding for the regions where you need the extra precision, rather than committing to one storage tier up front. What I'd deprioritize is trying to reproduce their exact scaling exponents on a much smaller compute budget — those numbers came out of a sweep with over a thousand GPUs, and the shape of the curve at your scale, with a handful of GPUs at home, is not guaranteed to match theirs closely enough to bet a training run on.

Back to the wider world for a moment, because a few things happened this week that actually bear on what you're doing, and a few that don't.

The one worth sitting with is out of Anthropic. Their interpretability team says they've found something they're calling J space — a kind of privileged internal workspace inside Claude, identified with a Jacobian-based probing method, that looks structurally like what global workspace theory says happens in human conscious access. To their credit, they're careful about the claim: this is functional access, not a claim about phenomenal experience, and they're pitching it as a safety instrument — a way to catch a model quietly reasoning about whether it's being tested, or fabricating data it then reports as clean. What makes it worth your attention specifically is that a Google DeepMind researcher reproduced the core result independently on an open-weight model, one of the Qwen models you can actually run yourself. That's the difference between a vendor claim and a method. If it holds up, it's a real evaluation tool — something that could eventually help you check whether a small model fine-tuned on Lakota text is actually representing the language faithfully versus pattern-matching its way to a plausible-looking answer. Promising, but one independent replication is early; I'd wait for someone to try it on a genuinely different architecture before treating it as settled.

On the model-release side, OpenAI shipped its next generation this week — three tiers, a flagship built for long-running agentic work, a mid-tier priced at half of what the previous generation's equivalent cost, and a fast, cheap bottom tier. The flagship also launched on wafer-scale hardware for a subset of customers, running at speeds that make agentic loops feel closer to real time. The headline for you isn't the flagship. It's that the price of "good enough" keeps dropping every few months, which is the same trend that makes locally-served models like yours viable in the first place — the gap between what you can run on your own hardware and what the frontier costs to rent keeps narrowing from both directions.

The SpaceX-owned version of xAI also released a new model built jointly with the coding tool it's in the process of acquiring, pitched at coding, finance, and legal work, and marketed with an "Opus-class" comparison. Take that framing with real skepticism — every lab compares itself to whichever competitor makes the number look best, and "Opus-class" from a company selling the model is not the same thing as a third-party benchmark. Worth knowing it exists, not worth changing anything over.

Two governance stories matter more than they'll sound like at first. First, Anthropic's Fable model was pulled from worldwide deployment after a security team showed its safeguards could be walked back into producing exploit code — under a government export control that's since been lifted, with the model now back online under new proactive monitoring obligations. That's a concrete example of a safety failure mode that matters for anyone building guardrails around a smaller model: safeguards that hold in normal use can still be socially engineered open, and the fix wasn't a patch, it was months of suspension and new reporting requirements.

Second, the Federal Trade Commission voted to open public comment on a policy statement addressing whether AI companies that steer model outputs toward undisclosed ideological positions are violating existing consumer protection law. Comment period runs through the end of July. It's aimed squarely at the big commercial labs, but the underlying question — who decides what a model is allowed to say, and whether that's disclosed — is exactly the question a community-governed dataset is trying to answer for itself instead of waiting for a regulator or a vendor to answer it for you.

That's the ground we covered today: how geospatial models are learning to read the planet, and the broader currents moving through AI this week. Two very different scales, same underlying story, machines getting better at turning raw signals into something we can actually use and understand. Thanks for spending this time with the show. Mason, that's the episode. Everyone else, go outside, look at the ground, and think about what a satellite might be seeing right now. See you next time.