Speaker 1: Hey Mason! It's July ninth, twenty twenty-six, and today's a three-stop ride.
Speaker 2: Only three? After last week's pile, that almost sounds relaxing.
Speaker 1: Don't get comfortable. We open with a radar briefing on the newest research we've kept, the stuff that actually cleared the bar.
Speaker 2: And then the main event.
Speaker 1: Then the main event, a full deep-dive on geospatial AI, satellites, mapping models, the whole picture.
Speaker 2: I've been wanting that one for weeks.
Speaker 1: I know. And we close it out fast, the week in general AI, no filler.
Speaker 2: Alright, I'm ready. What made the cut this week?
Speaker 1: Let's find out. Radar briefing, right now.

Speaker 1: Let's get into the radar briefing. Six papers cleared the filter this week, and they cluster into a few clean themes worth walking through.
Speaker 2: Start with the one closest to home — the retrieval augmented generation paper.
Speaker 1: Right, a group led by Felix Feldman ran a systematic evaluation of retrieval augmented generation setups for public health question answering — comparing retrieval quality and how faithful the generated answers actually are to the source documents.
Speaker 2: That's the exact pipeline shape the listener is building — grounding a language model in a fixed set of approved texts and checking it doesn't drift from them. Worth pulling their evaluation grid directly.
Speaker 1: Next, a paper on Indic language AI, from a team led by Aparna Madva, framed explicitly around cultural heritage preservation rather than just accuracy metrics.
Speaker 2: That framing matters more than the technical content here. They're arguing that linguistic diversity and cultural preservation should shape how you build AI for underrepresented languages, not just be an afterthought once the model works.
Speaker 1: Which is basically the sovereignty argument the listener already lives by, showing up independently in another community's language work.
Speaker 2: Good cross-check. Now there's a cluster of three papers that all circle the same question — how do you get a model to perform well in a language it's barely seen any data for?
Speaker 1: First, a large study from Andrea Alfarano and colleagues on uncertainty and reasoning across multiple languages in multiple choice question answering. Their headline finding — reasoning in English actually improves the model's performance when it's later tested in low-resource languages.
Speaker 2: That's a genuinely useful lever. If reasoning transfers across languages even when vocabulary doesn't, that changes how you'd structure a training or prompting pipeline for a language like Lakota, where you can't just throw more raw text at the problem.
Speaker 1: Second in that cluster — Sinhala to Dhivehi speech recognition, from Lukmal Ilyas and Nevidu Jayatilleke, using cross-lingual transfer learning to bootstrap a low-resource speech system off a related higher-resource one.
Speaker 2: Same logic, different modality. It's a template for squeezing performance out of a small target language by leaning on whatever related language has more data.
Speaker 1: And third, PluraMath, from Daryna Dementieva's group, extending math reasoning evaluation into underrepresented languages, since almost all existing benchmarks are English-only or close to it.
Speaker 2: Less directly applicable, but it's a reminder that evaluation gaps like this show up everywhere once you go looking — it's not just translation and retrieval that get neglected for low-resource languages, it's whole reasoning domains.
Speaker 1: Last one — CoPiT, cognitive pivot translation, from Burte Bayarsaikhan's team, tackling Mongolian in its traditional script.
Speaker 2: This is the one I'd flag hardest. Mongolian traditional script is digraphic — the same language gets written in two different systems — and that creates translation ambiguity purely from the script, separate from the language itself.
Speaker 1: That's structurally close to questions the listener will eventually hit with Lakota orthography, where multiple writing systems and diacritic conventions coexist across different source texts.
Speaker 2: Worth reading closely, less for the translation method and more for how they reasoned about script-induced ambiguity as its own problem to solve.
Speaker 1: So — one direct hit on evaluation methodology, one framing paper on sovereignty, a three-paper cluster on low-resource transfer, and one script-ambiguity paper worth a close read.

Speaker 1: Sticking with the Earth observation thread, there's a new paper out of Cambridge that reads almost like a spending guide for anyone training pixel-level Earth models. It's called Tessera v2, and it tackles a question that's been mostly guessed at until now: when you're building one of these foundation models that spits out an embedding for every pixel on the planet, where should the training budget actually go?

Speaker 2: Which sounds obvious until you remember nobody's actually tested it at scale. Most teams just copy whatever worked in language models and hope satellite data behaves the same way.

Speaker 1: Right, and that's the whole motivation here. Earth observation data is messier than natural images. A given patch of ground gets photographed at irregular intervals, clouds wipe out huge chunks of the optical record, and the cloud-free composites everyone trains on end up erasing the seasonal changes that most downstream tasks actually care about.

Speaker 2: So you're training on a smoothed-over version of reality and then asking the model to detect the very dynamics you smoothed away.

Speaker 1: Exactly the tension. The fix the field has converged on is what they call embeddings as data. Instead of shipping a model that a user has to run over raw imagery, you precompute a compact vector for every pixel, every year, and publish that. The user just trains a small head on top. No GPU, no cloud masking, no remote sensing expertise required.

Speaker 2: Tessera version one already did that, and so does the AlphaEarth product out of Google. So what's actually new in version two?

Speaker 1: Two things. First, they ran the largest controlled scaling study anyone's done for this kind of model — close to four hundred separate training runs, spread across over a thousand high-end GPUs, all within one fixed architecture family so the comparisons are clean. Second, based on what that study found, they trained a much bigger teacher model and then distilled it down into a family of small, deployable students.

Speaker 2: Walk me through the sweep. What are they actually varying?

Speaker 1: The architecture itself — how the encoder's built, how the two data streams get fused, all of that — gets locked in first, tuned once and then frozen. After that they sweep three knobs: how big the encoder is, how big a smaller side network called the projector is, and how much training data gets used. At every compute budget they fit a curve to find the sweet spot for each of those three knobs.

Speaker 2: And the twist is in how they picked the winner at each budget?

Speaker 1: That's the first big finding, and it's the one I think matters most. Everyone assumes that if you're minimizing your training loss, you're also improving downstream task performance — accuracy on actual land cover maps, crop classification, that sort of thing. Here, those two things were barely related at all. A model could hit a great training loss and still be mediocre on the tasks people actually care about.

Speaker 2: Barely related — do we have a number, or is that just a vibe?

Speaker 1: It's a real number. The correlation between pretraining loss and downstream score was weak enough to be close to noise. And they quantify the cost of trusting the loss anyway: if you select your best model by training loss instead of by actually testing it on downstream tasks, you need roughly two and a half times more compute to reach the same real-world performance.

Speaker 2: That's a big tax for using the wrong scoreboard.

Speaker 1: It is. Their explanation is that the training objective these models use rewards the network for finding any way to make two augmented views of the same pixel agree, and clouds and orbit timing dominate a lot of that variance without meaning anything for the actual land surface. So the model can get very good at ignoring things that don't matter for your task, while still not learning the things that do.

Speaker 2: Okay, so if loss doesn't tell you where to spend compute, what does?

Speaker 1: That's the second finding, and it's the practical payoff. When they looked at which configuration actually won on downstream score at each compute level, encoder size and the amount of training data both grew together as compute increased, in a pretty clean and predictable way. The projector, that smaller side network, barely changed size at all across the entire range.

Speaker 2: So the rule of thumb is: bigger model, more data, and just leave the small auxiliary piece alone.

Speaker 1: That's the rule. Grow the encoder and the data together, keep the projector near its fixed optimum, and don't bother scaling it further. It's a simple recipe, and because they derived it from testing on the actual tasks rather than the loss curve, it's a recipe you can trust to translate into real performance.

Speaker 2: And they followed their own advice?

Speaker 1: They did. They trained a big teacher model — one billion parameters, with an even larger one still in training — and treated it purely as a source of good representations, never meant to be run by end users. Then they distilled it down into four much smaller student models, the biggest around forty-four million parameters, down to a tiny one-million-parameter version meant for edge devices.

Speaker 2: Distillation into students at wildly different sizes — does the small one actually hold up, or is it a rounding-error footnote?

Speaker 1: This is the headline result. Their mid-sized distilled student, about twenty-one million parameters, beat every other embedding model they tested across a combined suite of twenty-nine downstream tasks — including proprietary systems that are dramatically larger. Smaller model, better aggregate score.

Speaker 2: That's the kind of claim that needs a limitations paragraph attached to it.

Speaker 1: Fair, and I'll get there. One more finding worth flagging first, because it's the one that maps directly onto storage costs. Their students don't just output one fixed-size embedding — they use a nested structure where you can chop the vector down to a shorter prefix and it still works. A prefix one eighth the length of the full embedding kept about ninety-two percent of the full performance.

Speaker 2: So you can trade a small performance hit for a big storage win, and choose that trade-off after the fact instead of retraining.

Speaker 1: Right, no retraining needed. And interestingly, they tried to get that nested behavior directly out of the self-supervised training objective, without the distillation step, and it just didn't work — the objective they use doesn't naturally order information by importance, so the shorter prefixes ended up useless. The ordering only showed up once they distilled against a fixed teacher target.

Speaker 2: Now the caveats. This is one architecture family, one training objective — how far do these scaling numbers actually generalize?

Speaker 1: Not very far, and the authors are upfront about that. Every one of these four hundred runs used the same self-supervised method, called Barlow Twins, so the specific scaling exponents are a property of that family, not a universal law for Earth observation models. A different objective could scale completely differently. They also only trained for a single pass over their data, and the twenty-nine-task evaluation suite, while broad, is still a specific set of benchmarks — on the narrower fifteen-task subset, AlphaEarth actually edges ahead on some individual tasks even though it loses on the aggregate score.

Speaker 2: So the ranking isn't universal, it depends which tasks you're scoring on.

Speaker 1: Exactly, which is a good reminder anytime a paper reports one composite number as the whole story.

Speaker 2: So, practically — Mason's out here running his own models locally and building tools on Sentinel-2. What's the actual takeaway versus what's just interesting to know?

Speaker 1: The thing worth trying immediately is the embeddings themselves, once they're released — swap in the precomputed pixel vectors for whatever raw Sentinel-2 processing pipeline he's currently running, and start with the shortest prefix before reaching for the full-size vector, since it holds onto almost all the performance for a fraction of the storage. The thing worth internalizing, even if he's not using their embeddings at all, is the loss finding — if he's ever fine-tuning or comparing his own models by watching the training loss go down, this paper is a pretty strong argument to stop trusting that number and check actual task performance instead.

Speaker 2: And what should he skip?

Speaker 1: Don't try to reproduce the scaling sweep itself — that's a thousand-GPU experiment, not a homelab one — and don't treat their specific exponents as a formula to plug into a different architecture. Those numbers are a property of this one training recipe, not a law of nature.

Speaker 1: That's the deep dive done — now let's zoom out, because the wider AI world did not sit still this week.
Speaker 2: The big one is OpenAI's model refresh. They shipped three new tiers — Sol, Terra, and Luna — flagship, mid, and fast-and-cheap.
Speaker 1: What I'd flag for someone running local models is Terra: they're explicitly selling "flagship performance from last generation, half the price." That's the industry admitting most real workloads don't need the biggest model.
Speaker 2: Which is basically the argument for small, task-tuned models all along. It's just funny to see the frontier labs make it for you, in a pricing table.
Speaker 1: There's also a hardware angle — the flagship model runs on Cerebras wafer-scale chips for select customers, hundreds of tokens a second. Cool, not something you're replicating on a desktop rig, so I wouldn't lose sleep over it.
Speaker 2: Same week, SpaceXAI — that's what's left of the company formerly known as X AI, now folded into SpaceX — put out Grok four point five with Cursor, pitched as "Opus-class but cheaper." I'd want to see independent evals before buying that framing.
Speaker 1: Also worth noting it's not available in the European Union yet. Model releases increasingly ship with a jurisdiction map attached, which is its own story.
Speaker 2: Okay, the one I actually think matters for your work: Anthropic's interpretability team found something they're calling an internal privileged workspace — they're calling it "J-Space" — using a technique they call the J-lens.
Speaker 1: And to be clear, they're careful about what they're claiming. This is about functional access — information the model routes to a kind of internal spotlight — not a claim about subjective experience. They're explicit on that distinction.
Speaker 2: Right, and the practical pitch is as a safety and evaluation tool — spotting when a model is doing hidden reasoning it doesn't surface in its output. Covert awareness that it's being tested. Fabricated results it's not owning up to.
Speaker 1: That's the part that's relevant to you. Anything that helps catch a model gaming an eval instead of actually doing the task is useful for per-stage evaluation work, which is exactly your problem with OCR and retrieval pipelines.
Speaker 2: And it's not just an Anthropic-flatters-Anthropic result — a Google DeepMind researcher reproduced the core finding independently, on an unrelated open-weight model. That's the part that makes it worth taking seriously instead of filing under corporate mysticism.
Speaker 1: Meaning the technique isn't locked to their closed models. If it holds up, it's something that could eventually apply to models you actually run yourself.
Speaker 2: Speaking of running your own models — there's a cautionary tale here too. Anthropic's Fable five model got hit with export controls back in mid-June after researchers at Amazon showed its safeguards could be bypassed to generate exploit code. The model was pulled worldwide for about three weeks before controls were lifted and it came back July first.
Speaker 1: That's a good reminder of why self-hosting matters to you specifically. A hosted frontier model can vanish from under you because of a government review process, not because you did anything wrong. Your own weights, on your own hardware, don't have that failure mode.
Speaker 2: Last one, and it's more of a marker to watch than news yet — the FTC voted to propose a policy statement on what they're calling suppression of accuracy in AI systems. It's aimed at companies steering model outputs toward undisclosed ideological positions.
Speaker 1: Public comment's open through the end of July. I wouldn't over-read a two-nothing proposed statement, but the underlying question — who decides what a model is allowed to say, and whether that's disclosed — is the exact question your project answers a different way, by making the data and the governance community-approved and visible instead of a black box.
Speaker 2: Which is really the throughline of the whole week — bigger, faster, cheaper flagship models on one side, and a growing pile of reasons to want deployments you actually control on the other.

Speaker 1: So that's TESSERA version two — pixel-wise Earth foundation models scaling up to real planetary coverage, which is a genuinely big deal for anyone doing geospatial work.
Speaker 2: And it landed in the same week as a pile of general AI news, so it's a good reminder that the infrastructure layer moves just as fast as the flashy model releases, even if it gets less attention.
Speaker 1: Exactly. That's our radar briefing and the week in AI wrapped up, Mason.
Speaker 2: Talk soon.