Mike Ravkine PRO

@dipankarsarkar I would love to hear any feedback you have from interacting with the R12 explorer app that I've been sharing screenshots of - it is able to answer your question directly, we have data for 1 and 8 confounders under low-memory-cell-count as well as high-memory-cell-count to complete the picture. Paradoxically the more cells you have the "easier" this task actually gets along the memory dimension, since each swap overwrites a smaller % of the cell space. But at the same time more cells means more unique swaps so parsing becomes more difficult. Generally: It's an entropic dance across many coupled dimensions simultaneously, the wall is it's shadow. If you're looking for a simple model, there isn't one here.

For deeper data analysis use you can grab the repo from my github, pull the r12-leaderboard data (instructions in the README.md) and run a full local copy of the processing stack.

@dipankarsarkar Comparing degradation across tasks is somewhat difficult, each problem has a specific SNR shape in the entropy space and its unusual to find two tasks failing "the same way". The Letters operation itself is a really simple one: count of counts. This is a clean top-to-bottom pass, and thats why we get such a nice waterfall on the plots.

The Shuffle operation itself is more complex - it requires both 1) state-tracking under sustained overwrite pressure, each pass modifies 2 elements 2) parsing a stream of similar-looking instructions (there are not many unique shuffles, so they repeat but their order is important!) and optionally 3) ignoring things that look like shuffles but are not shuffles: "Bob trades with Alice" vs "Bob thinks Alice is nice." If you peek the Shuffle results even without confounders (no #3, just pure #1 and #2) you can see the more complex signature of the task

Effort is almost perfectly linear, but Success is not: there's a capability wall here sitting at exactly 20 swaps, past that TB degrades rapidly.

If we introduce 2 confounding statements, the wall moves in to 12 swaps:

@dipankarsarkar You are absolutely spot on - the Letters task, as an example, is almost flawlessly monotonic with regard to number of steps (words to count letters across):

If you wish to deep-dive explore the ReasonScape results for this model further, the R12 explorer has been updated with Ternary-Bonsai results this morning. The Shuffle results are a little spicier this task has the concept of a "confounder" which is text that looks like a step but isn't we are checking if the model is actually following instructions or blind fuzzy pattern matching and this makes TB wobble a little bit.