Mike Ravkine PRO

@dipankarsarkar I would love to hear any feedback you have from interacting with the R12 explorer app that I've been sharing screenshots of - it is able to answer your question directly, we have data for 1 and 8 confounders under low-memory-cell-count as well as high-memory-cell-count to complete the picture. Paradoxically the more cells you have the "easier" this task actually gets along the memory dimension, since each swap overwrites a smaller % of the cell space. But at the same time more cells means more unique swaps so parsing becomes more difficult. Generally: It's an entropic dance across many coupled dimensions simultaneously, the wall is it's shadow. If you're looking for a simple model, there isn't one here.

For deeper data analysis use you can grab the repo from my github, pull the r12-leaderboard data (instructions in the README.md) and run a full local copy of the processing stack.

@dipankarsarkar Comparing degradation across tasks is somewhat difficult, each problem has a specific SNR shape in the entropy space and its unusual to find two tasks failing "the same way". The Letters operation itself is a really simple one: count of counts. This is a clean top-to-bottom pass, and thats why we get such a nice waterfall on the plots.

The Shuffle operation itself is more complex - it requires both 1) state-tracking under sustained overwrite pressure, each pass modifies 2 elements 2) parsing a stream of similar-looking instructions (there are not many unique shuffles, so they repeat but their order is important!) and optionally 3) ignoring things that look like shuffles but are not shuffles: "Bob trades with Alice" vs "Bob thinks Alice is nice." If you peek the Shuffle results even without confounders (no #3, just pure #1 and #2) you can see the more complex signature of the task

Effort is almost perfectly linear, but Success is not: there's a capability wall here sitting at exactly 20 swaps, past that TB degrades rapidly.

If we introduce 2 confounding statements, the wall moves in to 12 swaps:

@dipankarsarkar You are absolutely spot on - the Letters task, as an example, is almost flawlessly monotonic with regard to number of steps (words to count letters across):

If you wish to deep-dive explore the ReasonScape results for this model further, the R12 explorer has been updated with Ternary-Bonsai results this morning. The Shuffle results are a little spicier this task has the concept of a "confounder" which is text that looks like a step but isn't we are checking if the model is actually following instructions or blind fuzzy pattern matching and this makes TB wobble a little bit.

Some clarity is emerging:

The distribution of response lengths has shifted considerably in 3.6 and 2 of my tasks are no longer fitting into 16k, the ignorance zone blows up.

Re-running at 32k then we'll see if that extra thinking pays off or nah.

An interesting outlier here is the word-sort task where 3.6 thinks ~half as much and this costs it about 10pp of performance.

You're very much on to something here, and this is why I think it matters if this behavior is intentional or latent.

If they've taught it to recognize benchmarks specifically, that's benchmaxxing and is not going to help real world performance when your real tasks don't trigger the maxxxed paths. This is a genuine concern.

If they've taught it to "reach beyond the prompt" in the general sense, to understand the context and user intent behind the query, that's a genuinely useful capability and would explain why this model feels a little different.

Some stats: some version of this reasoning path happened in 39 out of 1070 test configurations, across 4 of my 12 tasks. In the most common occurrence, responsible for 30 of 39 hits, it recognized the task as being from BigBenchHard specifically and uses it's knowledge of the BBH category sets - which unfortunately suggests benchmaxxing.

🔥 Got any pics of this rig? Would love to see how it's managing thermals.