Link to Podcast

What we covered:
– Why "problem solving" is often just a vague label people use when they haven't explicitly enumerated the underlying skills, and how those skills can in fact be exhaustively mapped in a knowledge graph.
– How to approach research problems: Alex's PhD journey, top-down familiarity vs bottom-up mastery.
– If you have natural talent, use it, but not as a crutch, otherwise you'll stunt your long-term development. Don't turn your blessing into a curse.
– The story behind building our SAT prep curriculum: realizing that the standard school curriculum leaves a massive "missing middle" unaddressed; identifying 115+ missing topics to bridge the gap between textbook math and the hardest SAT questions.
– Watching the manifold hypothesis play out in test prep: the SAT may appear to allow an astronomical space of possible problem types, but in reality the actual problems live on a compact, highly structured manifold that can be fully enumerated and scaffolded in a knowledge graph

Timestamps:
00:00:00 - Intro: "problem solving" is what you call it when you don't really know what it is (i.e. you haven't explicitly enumerated the skills)
00:04:11 - How to approach research problems: Alex's PhD journey, top-down familiarity vs bottom-up mastery
00:20:28 - If you have natural talent, don't use it as a crutch. Don't turn your blessing into a curse.
00:29:06 - SAT prep, iteration 1: Realizing that the standard school curriculum leaves a massive "missing middle" unaddressed
00:33:45 - SAT prep, iteration 2: Covering the "missing middle" problems
00:53:38 - SAT prep, iteration 3: Building the "missing middle" knowledge graph
01:08:11 - Watching the manifold hypothesis play out in SAT prep
01:16:42 - The unreasonable effectiveness of the knowledge graph

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Link to Podcast

The raw transcript is provided below. Please understand that there may be typos.

Jason (00:00) All right, so what do we got on deck for the day for the show?

Justin (00:04) ⁓ yeah, so I figured, ⁓ I mean, one thing that we’ve been talking about for a while is the unreasonable effectiveness of the knowledge graph, especially in the SAT course, this idea of, of problem solving, problem solving is like this, this nebulous thing that everyone says like, it’s on the SAT. Like you got to get the kids to, to think critically, to solve problems, solve their way. And like, once you actually

like drill down deep into that. I mean, it’s, easy to think like, like, okay, we’re just going to teach the skill of problem solving. ⁓ or how do we teach that? But once you, once you start drilling down into like just a high level of accountability of like, okay, what does this mean? What do the kids have to do? You realize that like, well, it all comes down to these, these skills that you can enumerate explicitly. They’re explicit skills. And what’s more, you can arrange them in a knowledge graph.

It’s really, it’s a course. It’s a body of well-connected knowledge, hierarchical, just like anything else. And so when people look at problem solving and refer to like, ⁓ the kid needs to learn problem solving, need to learn to think critically. There’s this whole body of knowledge that they just haven’t gotten, ⁓ but it can be enumerated exhaustively. And yeah, I mean,

Jason (01:30) I we should, I think we should go through what our, sort of journey. Yeah. Because it’s funny. mean, problem solving has always been this, just this, uh, it’s like this meta skill or something, right? It’s like, you know, kids get to get better at problem solving, right? And we think of Poya’s problem solving techniques of draw a diagram, make a list, make a table, you know, solve a simpler problem trial and error, you know, I mean, there’s a few more, but.

You know, and I, I tried teaching this to kids. And when we were doing, ⁓ you know, like, they’re like fifth grade, it kind of stuff, fourth, fifth, sixth grade for, for, for math competitions. And it’s, I mean, trial and error works pretty well, right? Cause everything I do just tries some stuff, you know, ⁓ sometimes making lists, organizing information. I mean, these things can be helpful, but.

There’s a, like, it’s like a, it’s like a limited level of effectiveness, you know, to solving actual problems. It’s sort of like when you, there’s no hope. There’s no hope of solving the problem, right? Like, I have no idea. Like you sort of like, let me just try and figure out the contours of this problem and these drawing diagrams and making tables and things are ways of maybe getting your head around it. But when you’re actually talking about sort of solving concrete problems in the format of like a math test.

standardized test or competition test that’s not really gonna work for you most of the time. think you really what you have to do is you have to be familiar with these this class of problems that you’re trying to solve because if you don’t have the time typically and even if you had the time there’s just so much contextual information about that specific kind of problem and what type of methods will unlock it.

You’re not going to invent them on the spot in a minute or two minutes or five minutes,

Justin (03:34) Yeah, you don’t have the time to approach it like a like a research problem or like a like a even like a take home assignment or like you just you if you don’t if you can’t figure it out in a couple minutes, you’re you’re you’re toast. I mean, even on like the putnam or something. I mean, like it extends from a couple minutes to like, I forget exactly, but you definitely don’t have more than an hour per problem. It’s like if you’re if you’re taking hours to figure this out, like you’re just you’re toast.

Jason (04:02) Right. mean, obviously, mathematicians, I mean, they spend months or years on problems. you know, I mean, Alex, you’re a resident mathematician. mean, so it seems to me that like, you mean it’s it’s basically when you’re research is a there’s a sort of a family of problems is a much bigger problem. You’re trying to breaking it down across and solving sub problems on the way to some bigger result. So no one problem takes years, but

You these individual problems can take days or weeks or months, I guess, as you accumulate tools and techniques and insight. mean, let’s talk a little bit about how that mean, just just before we get into how we approach it or how we’re thinking about it now, Alex, I’d like to hear your thoughts from a mathematician. How do mathematicians solve research problems? I what do we do? What are we doing? What are you doing?

Alex (04:54) I can certainly talk about from my own personal experience as someone that’s done a research project, a PhD. so I came in thinking I had pretty good fundamentals, know, first class, honors degree, had a master’s degree in fluid dynamics. went into a PhD program, which was a research project in fluid dynamics. I think I’m good. I can pretty much just go straight in, start sort of understanding, you know, there’s a particular couple.

one or two particular papers that we wanted to kind of extend, if you like, that was kind of our first, my first goal. And I thought, yeah, I’m good to go. But it was a, it was a real struggle. It turned out despite having, you know, all those degrees, etc. I need to spend a lot of time at least six months, like really, really kind of familiarizing yourself with what are the, what are the foundations that have led to this particular research paper?

And you know them really, really well. You know, I mean, there’s a lot of stuff on the periphery that, perhaps you don’t necessarily need to know. I mean, that’s actually part of a skill. what’s relevant and what isn’t, like discerning which is relevant, what’s relevant and what isn’t. But I made the mistake of kind of leaping in and just trying to kind of like figure stuff out when I didn’t really have a strong enough knowledge of my particular research area. Even though thought I had enough fundamentals to do it.

So yeah, so it really is about learning the tools and techniques of your chosen field. It’s going to vary a lot between the different fields, really understanding the paper, talking to other mathematicians, running experiments, re-deriving most results you’re going to end up using. ⁓ And then once you’ve got to a point where you really, really kind of understand the fundamentals on the research level, that’s when you’re ready to actually start tackling new problems.

Jason (06:50) So that was interesting. You said you rederived the results that you’re to use. So you have a series of tools. So the results are essentially your tools, right? Yes. And in order to be really good with these tools, really skilled with these tools, you have to go through the derivation process, which is based at the research level is sort of a way of practicing and really understanding the essence of that tool, the extent, the limitations, the constraints and the techniques.

Right? Because the techniques that went into finding that result are techniques that you’re probably going to pull in later. Right? I mean, I’m just.

Alex (07:25) Yeah, mean, absolutely. mean, when I mean, you know, research papers aren’t really designed really with pedagogy in mind. It’s just like, you know, it’s like this, here’s, here’s a result for this kind of wave number in my, in my situation, you know. And it’s like, and I remember just sitting down with my supervisor, we’re like, where does that come from kind of thing. So you really have to dig very hard.

to go back and make sure you really understand when that result’s coming. No one’s explaining it to you. This is not like math academy for research. It doesn’t really exist. ⁓ So you’ve really got to sort of dig hard and make sure that you really understand where that result is coming from, what its limitations are, when it can be applied, when it cannot be applied. ⁓ So ⁓ interestingly enough, actually, it’s one of those things. I think you have to be quite mature as ⁓ a person.

to really be able to kind of have the discipline to go back and do all that kind of intermediate grunt work before you actually kind of set out to do your own research.

Justin (08:28) this

sounds like this sounds a lot like in software, like the difference between just importing a solution from the library and saying that like, oh, I do machine learning, I imported this model, and I ran it now I’m a machine learning researcher engine, like, no, no, no, you have to actually, you can’t just use the off the shelf, like you can’t just use the theorem, and wield the theorem and say like, now I am all powerful, you actually have to go like right.

code from scratch, re-derive the result, whatever, from the bottom up to really understand the mechanics of what is it. It’s not enough to just take it off the shelf and use it. You need to know what went into building this thing. Is that accurate?

Alex (09:13) Yeah, that’s a good analogy. ⁓ I mean, you I mean, you could could could sort of approach it from that kind of top down point of view, I suppose. But, know, in broad strokes, I mean, we often talk about this when we’re discussing, you know, what’s it’s a debate that comes up quite a lot in that kind of community, you know, what’s the best way to learn these kind of machine learning ⁓ algorithms and ⁓ be able to kind of build stuff.

as I say top down or bottom up, you know, I mean, I think you hit the nail on the head, Justin, when you say that, know, down kind of in broad strokes, like, yeah, fine, you know, grab it, you know, download a few libraries, get some stuff working so you can’t get your arms around the actual things you’re trying to solve. But if you really, really want to know this stuff fundamentally so that…

when you hit an unexpected problem, you’re not part of the crew that’s sitting around scratching their heads being, what are they about to do? You actually got the fundamentals in place to actually break new ground, solve problems that no one else can solve. So yeah, absolutely.

Justin (10:13) You know, I remember one post that you made a while back about how when you were starting out on your research project, didn’t you write something about how you started out like trying to approach it like top down and later on you realized that you actually had to build all this knowledge bottom up. And there was like a lot of time that maybe wasn’t used as efficiently as you could have used it, just spent top down and you realized you needed to like go back.

to the references of the paper, go back to their references or something and just like build up your knowledge, something like that.

Alex (10:47) Yeah, I mean, ⁓ I might my supervisors were were brilliant. ⁓ But I think like all supervisors, they’re quite keen to sort of like see progress quite quickly. And I think that if you’re if you’re approaching it, a PhD project or any kind of research project, as as you’re probably supposed to, you know, it isn’t necessarily that quick. It’s kind of like short term pain for long term gain, you know, you go sort of ⁓ go through the fundamentals before you can attempt anything anything new and

Um, yeah, I started out, yeah, basically writing, writing code from day one, basically writing code from day one to solve this eigenvalue problem, which I could then feed into a model, which kind of gave me all these kind of acoustic modes. And, it was, it was cool. I kind of felt like I was getting, it was, was kind of broad strokes getting my arm around the problem. Like we spoke to, but I did find myself hitting a little bit when we were, when I was really trying to break new ground, like really new grounds, I kind of felt that, uh, something’s missing here. I really need to go back and kind of like,

almost like see, understand the historical journey of where we’re at now, you know, because the top down approach, got so far got some results, great, can put that in the paper, great. it has a breaking really new ground. I thought I was hitting a wall until I really, really went back and did what I probably should have done in the first six months, which was, ⁓ you know, studied, studied the really important papers ⁓ and read, derive all the results and really have a very, very clear understanding. Okay, this is why we’re here.

at this moment and really understand the problems that the field was facing.

Jason (12:19) Well, you know, it’s, um, it’s kind of interesting. One trap that the, that are, guess, one sort of temptation of the top down is, is it gives you enough, uh, uh, uh, almost like a perception of expertise, a perception of, of, of skill, of familiarity to problem that you can talk a good game. Right.

And, you know, I, I’ve noticed that in technology that I get full. I’ve been fooled a number of times by people because like, want to believe that they’re good. Cause I don’t want to want to spend all this time digging in and testing them. And I was like, Oh yeah. So, you know, this and they, they use all the right buzzwords and they talk about the right methodology and the right libraries and the right frameworks. And that they talk knowingly and as if they have battle scars or whatever about, you know, how you develop complex systems and quality software. And, and it’s, really like talking to good game.

Right? Because you can, you you read, you read some thought pieces by people in the area, right? So you get this sort of vibe, you get this sort of vibe or the sort of the language of the domain. Who’s who and what’s what and what’s cool and what’s not. What are people excited about? What are the important results? And, and then you’re like, you could talk to somebody who’s an expert for 20, 30 minutes. I’m like, oh, this

guy knows his stuff. Yeah, we should talk to him. It turns out it was an inch deep. But it gives you a quick payoff. It gives you the dopamine hit. It impresses people around you. It might get you job. It might get you grant maybe or a research support or whatever environment you’re in. But then again, you run out gas. You start being able to make progress.

you as you said, like the building stuff for the fundamentals can be an arduous thing and it could be, it could give people the sense that you’re making no progress. And I guess a lot of these, these PhD advisors are like, man, I’ve seen, I’ve had people in PhD programs who were just spinning their wheels and not making any progress and that’s not a good sign. And I, the people who end up completing their PhD and getting their dissertation done, I see some perception.

there’s some perception of progress pretty consistently as opposed to someone who just kind of so they’re always like, because they’re they’re they’re heuristic is like, is their perception of progress, right? And it’s hard to tell well, this guy’s he’s out back doing push ups and deadlifts. And he’s working really hard. Like, I don’t know, I don’t see any of that. I just see like, any they really doesn’t seem to know anything, right? So it’s, it’s sort of interesting, it’s easy to get pulled in because you get some quick wins, right? And everybody’s like, well, that’s amazing, right? But the hard, hard work of building up

those skills or those techniques or those tools or those foundations. It’s so, it can feel unrewarding. It can feel demoralizing because the people around you who are sort of giving you, acknowledging your progress or not acknowledging anything is they’re like, I don’t know. Are you getting better? Are you getting, so I can see how you get sucked into that. But you know, it’s, ⁓ so I would like to, one thing I’d like to ask you about, you know, it’s kind of funny because we’ve never really talked too much about your PhD.

I mean, you’ve made some references to it here and there. There’s some stories. when you were like getting in your when you got stuck, like is there a story where you first got your first big problem and you’re just like, I have no idea what to do. Am I am I even going to be able to do this? I mean, what was your do you have anything that you can tell us about?

Alex (15:59) Oh yes, remember there was so the the the big um the big result I managed to kind of come up with was this phenomenon called modal scattering. It had been done it had been shown to occur using something called the finite element methods which is kind of more computational. Methods that I kind of specialized in were more analytic called asymptotic methods and

So this phenomenon was known to occur. And the question I wanted to ask was, can this phenomenon be shown to take place using these asymptotic methods? And I remember there was one extremely prolific ⁓ researcher in the field. And he was respected by everybody. And I met him. And he was a really great guy. And ⁓ he made a lot of progress in the field. But there was one particular result which I relied on, which turned out to be not

quite correct or it was correct, but not quite in the context he envisaged or there was a special case. It’s been a long time since I did it. I can’t quite remember. So I was kind of chasing shadows a little bit. And then one day I just had like one of these kind of extremely intense sessions where I just kind of went through the math and like, huh, I’m not getting omega equals negative two. I’m getting omega equals negative a half. And I kind of like went to my

Supervisor I think this result here isn’t isn’t right. Can you know am I going crazy or is that right and they sort of?

Jason (17:24) Crazy

pills? Yes.

Alex (17:27) And

they said, I actually don’t. looks like you’re right. said, OK. Well, that was good. yeah.

Jason (17:34) You must have been kind of impressed by that, right? Your PhD thesis. He’s like, huh, like that’s a good find, right? That’s like finding a bug in a major library, right?

Alex (17:44) Yeah, I can’t remember how they how they reacted. I think it was just more like, yeah, he’s right. So it was kind of like, you know, like the Yuri’s jumping like, well done.

Justin (17:55) Is that?

Jason (17:56) I’m getting the

Justin (17:58) That’s such a mathematician’s thought.

Jason (18:01) It’s

like a…

Alex (18:08) So, so I remember that helped. I remember it was just lots of because the equations that we got, I I just remember that there was sort of taking up like, pages and pages just for like, just just to give like, just to describe one function that it was just it was it it was I remember it was extremely, it was extremely arduous and difficult. And it was just it was it was was tough to push, push through it.

But yeah, I definitely question my own ability. Am I capable of doing this? Is there some unknown unknown here that is just preventing this from kind of coming out in the wash? ⁓

Jason (18:47) I’m out your skill your own individual talent or

Alex (18:52) I mean,

a little bit. think so. mean, when you’re when you’re a place like UCL, you’re surrounded by like brilliant people who’ve done amazing things. it’s like, you know, I mean, what imposter syndrome, I think everybody gets that at some point. I think I definitely felt that in the time I was there. mean, I think the fact that, yeah, had really good supervisors. And the fact that, yeah, I mean, the imposter syndrome didn’t set in quite so much.

⁓

I was humble pretty quickly, I think. Why must that hit and break walls?

Jason (19:37) You know,

it’s funny. I was watching these YouTube videos of these NBA players talking about these NBA stars. They’re welcome to the league moment where they were this hot shot college player, first round draft pick, you know, just this. Louded rookie and they came in and then some really top-notch, you know, player just put them in their place. is the kid. got a lot to learn. Yeah.

I mean, I think it happens. It happens to everybody. Almost everyone comes in. Rarely do they come in and does somebody come in as a rookie and then just start torching the veterans. Now, sometimes you have this just like once in a lifetime type, once in an era type talents that can do that. But that’s really rare. Most of the superstars get humbled a number of times.

Alex (20:28) Yeah, I mean, natural talent sort of, you know, in some ways, having like so called natural talent or being having having some, I don’t know, what the word is that gift or something in a particular thing. I mean, it could be somewhat a blessing and a curse in a way, because you can take it for granted. So, you know, as sort of a high school math, that’s easy. A level is pretty easy. Even undergraduate. Yes, it wasn’t too bad. And ⁓ but when you get to a certain level, it’s like everybody has natural talent.

But some of those people also work extremely hard. That’s level you’re at now, buddy, kind of thing. So that is a completely different story. Natural talent is not necessarily going to help you to go all the way. You’ve got to work extremely hard, just like the people around you are doing, who are also extremely talented. you’re like, yeah. I guess I’ve always been reasonably humble, I think.

Justin (21:25) You don’t want to let your natural gift or talent turn into a crutch. The moment you use it as a crutch, like you just rely on it and stop developing other parts of your game. That’s the moment when you start hitting like a really sharp asymptote, right? It reminds me of, ⁓ you know, in the original ⁓ Pasadena program, how you’d get some sixth graders who are like incredibly sharp. ⁓ oddly enough, sometimes it it was the sharpest

sixth graders who could do the most work in their head, who would struggle the most ⁓ a year later because they were so resistant to writing anything down. They were like, no, I don’t need pencil and paper. Like what? No, I just do all this in my head. and ⁓ I mean, there were others who were like that, but also like wrote things down and they, things went smoothly for them. They went really far very quickly. It was great. But the ones who were very stubborn and used their

their outsides working memory capacity, it became into a crutch. They can only solve problems that they can immediately, like entirely fit in their working memory without usage of paper and pencil. That came back to bite them really fast.

Jason (22:36) And in sports, you’d see this a lot of times the guys who would who are blessed with speed, right? Like they have a higher gear and then they could just run around people, whether it was basketball or football or soccer or whatever, they could they were just faster, a lot faster. And so they were a weapon. That speed was a weapon. Just like height can be saying basketball or size and football, just having a lot of strength and size or being

you it’s really tall is obviously huge, can be a huge advantage. But speed was one of these, it doesn’t matter what sport you’re in, if you’re really fast, like tennis, I mean, anything, pick just about, other than maybe golf, but even golfing, how fast you can swing a club, you can hit it a lot further than other people. But, but, which is great because you get this speed and you have this inherent advantage that it’s a weapon that can be leveraged. But the problem is if you have a lot of speed,

you can use that as a crutch instead of learning the skills. Right? So where there are other guys would be like, well, I got to learn a lot more ball handling skills or, you know, technical skilled soccer with his foot skills. And I got to, I got to, I could just run past you. Like, I don’t have to worry about from the time I was like 11 or 10, I just go and well, guess what? Now you’re 15, 16 and

you you still haven’t developed the skills with a guy, guys are a 13, 14, they’re working it. Cause like, can’t get past people by just running past them because I’m not faster. So I have to like fake them out, which means I got to really work on this. And in that would happen. And what happens with a of these guys is they would get to say college and, or like, like Alex, you said, everybody’s a good athlete, right? You went from.

you know, or the differential is much smaller. It’s okay. Well, you’re fast still, but you’re not that much faster. Right. And there are some guys who are pretty close to as fast, or maybe guys who are faster, but it doesn’t work anymore. And then all of a sudden you had, should have been spending years developing these fundamental skills. too, it’s often too late. Right. It’s really hard to do that. So it’s like, it’s interesting, you know, I mean, I think it’s in everything when you’re

having a natural gift that you can also call it a natural curse, right? now you have do special and you don’t have to do things that other people have to do, which are the things you need to be doing anyway.

Justin (24:58) Yeah, it’s like this.

duality of these pure skills. It’s like on one hand, like we can up your game by getting you faster, getting you stronger, just like in a pure sense, in the weight room or just sprinting. ⁓ But if that’s the only thing that you lean on, like if that’s your entire game, then you’re limiting yourself, right? It’s like the…

Jason (25:26) Well, it was so tough because you and I both taught and worked with these six, seventh, sixth graders in the first learning algebra. And, you know, my son Colby was the all time worst.

Justin (25:38) my God. Yeah. is true.

Jason (25:41) And

the reason is because he could get away with it. was so hard to convince. And I had this other kid, Nathan, who was a year younger than Colby. And I was like, dude, I’m like, I know you kid. Like, I’m telling you, like, I know you can do all these steps in your head, but you will eventually run into a point where it’s not going to work so well. And the problem is for them, they’re like, well, it works. And I remember this in, okay, in, in, basketball. So, and I use them, I got anybody’s listeners. do.

Ton of sports analogies, I’m sorry, but it’s just how I think about things. But I would, I was not a, you know, I’m an average height guard and I was used my speed to go around people and I would have a couple of moves. And I remember people say, Oh, you just always do this or that. And I’m like, well, stop me. And I was like, right. Like they, would complain cause I could just do this left or this right every time. And they couldn’t stop it.

But I didn’t develop an outside shot like I should have because I could just use my speed to go around people. So I know this personally. I messed up because, and I think Justin, talked about this story in the past where a coach said, Jason, if you want to play college basketball at the college level at your height, you need a really, really good three-point shot. So you need to start working on it. I’m like, oh yeah. So the thing is it’s really, it’s.

Justin (26:58) Yeah.

Jason (27:04) the more of an advantage this inherent, this gift gives you, the harder it is to break them usually. I sometimes it’s just a matter of what their coachability is, right? But you know, as with kids, if something works, you say, don’t, no, no, no, don’t do that, do this. And they’re like, it works. Like, why would I not do that? Right? It’s just, this doesn’t make any sense. Like you’re telling me,

Like I see the world as it is and this happens and it works every time. Why would I not do that? Cause it actually, it’s easy, right? And I’m comfortable doing it, but it’s a curse. It locks that in and it’s so hard to break for an adult who’s like, look, kid, I’m trying to help you, man. Like I’m trying to help you, please.

Justin (27:40) actually.

And conventional knowledge too is like lean into what works, right? You hear that in marketing all the time. You hear that like everywhere, lean into what works. So it’s, it’s hard to kind of untangle that sometimes between like the difference between leaning into what works versus using what is currently working as a crutch. It’s really subtle. Sometimes it’s hard to tell the difference, but the, and in the longterm, these are two completely different trajectories.

Jason (28:17) Yeah, it’s fascinating. well, Alex, if you have anything more on the story, we can go on it, but I wanna, we can probably pull it back to what we originally supposed to talk about. think we get kind of unsurprised. Yeah, let’s do that.

Alex (28:31) Let’s go back to where we were talking about the SAT. I suppose we wanted to talk, there’s a number of things we wanted to talk about. So yeah, so we just spent some time fitting in what’s called the so-called missing middle that we’ve talked a bit about on X about the missing middle. And I think that’d probably be kind of a fun thing to talk about.

Justin (28:52) We to go through the history of the SAT course, which is what we were originally going to talk about before Jason just grabbed the wheel. But yeah, let’s get back to that, the history. ⁓

Jason (29:06) So, okay, so let’s just, let me just frame it a little bit. then, ⁓ we initially cobbled together an SAT course for, ⁓ it was Alpha School, which I can say publicly, because they always, mentioned Math Academy a lot. So Alpha School’s a very forward leaning ⁓ private school, I guess, in Texas and other places. But, ⁓

They were like, Hey, you know, we’re trying to prepare students to take the SAT, can you put together a course? And I was like, yeah, yeah. And said, Alex, can you just take a look at the, you know, review what is described, the, the, the, the described skills that are covered in the SAT and take what we have and put it into a single course. Cause that way it’s spread across geometry and algebra and algebra two and pre-algebra. it’s a sort of like, or integrated math. It’s like, there’s no way of like, how close are they? Like, I don’t know. know, it’s, um,

So now at the time, Sandy warned me about it. She’s like, look, I, you, this really going to be a good SAT course? I mean, any of you guys? And I said, well, I mean, I think we have all the fundamentals there should be fine. And she’s like, I don’t, she warned, she was going to ingest and you, think I mentioned, I asked you about it too. I said, what do you think? Cause you tutored kids for NLSAT for years and you, you were like,

It’s not totally comprehensive. Yeah, these are the constituent skills that a student is going to need for the SAT that exists in the standard curriculum, whether it’s traditional or integrated.

Justin (30:49) Yeah, but I don’t like the standard.

Jason (30:52) You echoed Sandy’s warning, like I-

Justin (30:54) This is not nearly sufficient. Yeah. Like you can’t just come out of that. ⁓ our original course, I was just based on the standard curriculum stuff. A student can’t just take that and then come out like scoring like an 800 on the SAT math. ⁓ right. had to do there’s, there’s a lot of stuff outside of the standard curriculum that, that we were going to have to build in.

Jason (31:15) And at the time, so Alex, as usual, had his, he had like multiple courses going. So I like to think, Alex, can you just, can you put this together? And you’re like, okay, what, what do you want me to do? I’m like, look, can you just, you’re like, fine. And within like a week you just analyze the stuff. You’re like, all right, this is, this is what it is. And you’re like, yeah, I think it looks good. Cause you were just looking at, weren’t analyzing tests. You were just looking at the descriptions of the topics and like, I think we got all this stuff and I think it’s good. And then.

Maybe it six months later, maybe not even that long. I got a call with Andy Montgomery, who’s like head of curriculum or something for AlphaSchool. Really, really nice guy, really bright guy. he’s like a PhD in computer science from like Stanford or something. And he’s always been like really impressed with Math Academy and was very, he was a supporter, so he’s easy to talk to. And he’s like, listen, Jason.

⁓ you guys have a great reputation, you know, especially on X and things like that. One way to ruin it is to say, have a course that prepares you for something and doesn’t. And I was like, well, you know, what do you mean? He’s like, look, there are, there are, you know, we had our students go through the test and they were taking practice tests and we had our team analyze it. And there’s just a lot of, a lot of, ⁓ there are a number of topics or questions that they just really weren’t prepared for. And I was like,

And I like, well, I don’t love hearing that, but I appreciate the feedback. And then I remember that, you know, Justin, your warning and Sandy’s warning, because then I got off the call and I talked to Sandy. She’s like, I told you. Right. She’s like, and I said, well, they wanted something we had to give them the best thing we had. And she’s, but by making it a public course that that exposed us a little bit.

And so then I was like, okay, we gotta, we gotta fix this. This is, this is not cool. Right. And it’s just one of those things. I just was not paying attention to it. We had too many balls in the air, too many different courses we’re working on, too many different tech projects. I just was like, yeah, yeah, we’re, they’re good. They’re, know, and then, um, and I’m sorry, this windup is long, but there’s just, it’s a little bit of a, yeah, a little backwards, a little complicated story, but, um, and then we,

Justin (33:40) Good background.

Jason (33:45) And then Alex was like, all right, we got to sort this out. think after that call, I jumped on both you guys and said, okay, we got to fix this now. This is a problem. Alex, we started doing some research into it. we had a guy we had hired, a mathematician who was really analyzing past tests and sort of trying to figure out what was missing. What were topics that questions on these exams that…

students really would not be adequately prepared for. was sort of like a groping run in a dark room trying to find this stuff. And we went through a couple of two guys, really. There’s a lot of work and was, and again, in Alex you couldn’t put your full attention on it because you had, know, think probably the statistics course, machine learning course, discrete math course, you know, all this other things. So you’re just, this is just like,

course number four five on your list. And it was just sort of this research project was going on. But at a certain point it was like, okay, we need to get systematic about this. And we took all 1500 problems on the public database that the SAT, what is it, the college board publishes. They, these are the 1500 problems or whatever they are.

that are representative of the type of problems that are on the exam. And we got systematic about it. And then Alex, that was when the time we started realizing, okay, like what are these topics? Like here are the problems, here are the harder questions on the SAT. Here are the questions that are in the material we have. What’s in that gap? And we we started referring to that as the missing middle. And so Alex,

you and your team started really pushing on that, right?

you know, how do we close that gap? mean, talk a little bit about that process, that, you know, finding the missing middle.

Alex (35:50) Yeah, so it was kind of like a double pronged approach. ⁓ when we had all the kind of the question bank questions, we knew we needed a test prep course, a course that basically had ⁓ every single type of problem that could come up on the SAT. So we said, OK, we need those questions and we need solutions to those questions. So sort of set about the process of ⁓ populating.

the database with those questions and solutions that are done to our standard and things like that and of like fitting out the missing middle from there, from that end. ⁓

Jason (36:25) But

just to be clear, we didn’t use those questions. We just use them as a model for the kind of questions, for a class of questions that we would need, right?

Alex (36:34) Correct, yeah, that’s right. it’s just like, OK, yeah, exactly. And the solutions were completely ours. We looked at the question stems. It’s like, OK, we would make them our own. The solutions were basically completely written by our team. ⁓ But yeah, so we started sort of populating the system with those kind of test prep questions and kind of fitting out the missing middle that way.

But then Justin needed, he needed to have his arms around like the practice test because as part of the test prep course, we need high fidelity exams. So he started looking at the, he started looking at the practice tests and you know, and did a really, really, really thorough job of identifying, okay, this is a missing middle topic, that’s a missing middle topic. so over the course of about, was it about a week or so, two weeks, you know, just add,

did like a missing middle analysis from the point of view looking at the tests, the exams, gave that to me and said, okay, this needs to be in the curriculum.

Jason (37:41) Phase two, because you went through phase one because you were just looking at the 1500 problems. You’re like, think we’re, and I would kick your out of classroom. Like, are we close? I how many are you like, well, I think we’re maybe 10 more, 10, 20 topics away or whatever. Like we’re getting in. And I kind of mentioned it offhand to Justin and he’s like, I don’t know. Right. then.

Justin (38:05) It was one like you had mentioned to me like, it sounds like the SAT course is going to be done next week. And I was just like my conception of like what this course needs to cover versus the actual exam just based on tutoring experience. was like, for one week away, that doesn’t something does not, it sounds like there should, there should be more on it. So, ⁓

Jason (38:25) got

you to looking at it. I why, okay, well, go take a look at some practice exams and see. And then you went and looked at all the practice exams. And then you dug in and you went and looked for published exams or old exams that had been cobbled together by people who remembered certain questions and then just kind of, right? Or something like that.

Justin (38:47) Well, have ⁓ official, the SAT had released official exams for years past that you can pass.

Jason (38:53) So it was that plus the practice exam. So give a huge range of questions, which sort of was a different emphasis or different types of problems that were on the ⁓ question bank,

Justin (39:04) Yeah, turns out the question bake was incomplete in coverage.

Jason (39:10) Right, right. So, Alex, how many topics do you say missing middle topics? And when I say missing middle topics, it was not just like one topic bridge. Sometimes it would be a whole sequence of topics to bridge the gap because it wasn’t, it was obviously, so if you had the questions on the SAT, the hardest questions, it was sort of like we were closer and further on certain topics. And so you had to bridge that. Sometimes it was one or two topics. Sometimes it was a whole sequence of a cluster of scaffolding that had to be built up towards,

for a student would be able to have a reasonable chance of solving a problem in the time allotted, right?

Alex (39:44) Yeah, I mean, I think in the end, it ended up being about 115 new topics is what we added. A lot is a lot. there’s a lot. The biggest area was like the descriptive statistics stuff. It’s quite a lot. So it’s not, know, on the SAT, there’s quite a lot of emphasis on, you know, sort of analyzing different types of data sets. And some of the problems are quite challenging.

And there was a lot of missing middle stuff that we needed to add for that. I don’t think we really had enough coverage, just even in even just considering the regular curriculum in let alone the the the

Jason (40:23) across the board. was just our weakest area. Scriptural statistics was a clear weak point, but it was everywhere. Analyzing quadratic systems, lead to geometry.

Justin (40:34) For listeners, 115 topics, compare that to our algebra one course has what, like 200 ish topics? So this is about half of like a full length, full school year course. That’s a lot of stuff. Yeah. So, right. So you’re saying there’s descriptive statistics. What else, what else was in

Alex (40:41) I see something like that.

That was so what I tended to find is that it was a lot of stuff. It was kind of on the on the periphery. mean, this is this is this description statistics stands out because it was just such a huge amount of stuff. And that’s kind of what we so when we said we were kind of done with the question bank, it was like that’s as far as we kind of got. We need a ton of this stuff. And because it’s graphs and tables and things like that, we need lots and lots of topics. Kind of the kind of a lot of the attention was kind of focused on that for a little while, just kind of getting that.

in place. Aside from that, there was was kind of just stuff like almost like one or two topic extensions of things we already already had. That was the vast majority of the cases. Okay, you cover this, but you don’t cover it to the point where someone who is using this to prepare for the SAT is going to see that question. yeah, I know how to do this. You know, I got some examples, which are kind of written, which are written down, which I thought were quite interesting. so things like

like we’ve got a lot of stuff in it had a lot of stuff in the system on like exponential models, but things like sort of shifting units of time, you know, going from hours to days and things like how does that affect the model and percentage increase and decrease that are kind of related. But that was that was like, that was one area that I thought was, yeah, that was one.

Justin (42:11) And that’s not a straight up unit conversion. That’s like, cause the time is in the exponent of the exponential model. Right? So you have to, it’s not just a, just multiply this by like 365 days in a year. No, there’s, there’s, you have to use exponent rules and stuff like that to actually make this conversion. Cause it’s in a kind of complicated setting.

Alex (42:30) Yes, exactly. And, you know, I think there’s one which there was one type of topic which was quite exclusive to the SAT. I’ve never seen it anywhere else before. Like, and again, exponential functions like you call it riddles when you first did your

Justin (42:45) Yeah, because I didn’t have a good name for it. I was like, what the hell is this? Like, why would anyone… Okay, fine. All right. This is just, this is basically a riddle. It’s just worded intentionally, confusingly to try to throw somebody off. But yeah.

Alex (42:58) Yes, I really wanted to keep that name because I thought it was great. But I thought when we got down to the crux of it, it’s OK. What this really is, is taking an exponential model and rewriting it so that you subtract one in the exponent, but that affects the constant. So the constant can reveal different parts about the function itself. so yeah, don’t really have that, like flipping between one type and exponential model and another and how that is related to the very

the constants are related to the to model itself what it reveals so that yeah that’s that’s quite that’s kind of that’s quite niche and there’s a really nasty questions on the exam so ⁓ so again it was just kind of like it’s just one or two topics that had to be added onto all the exponential function stuff to make sure they can do that ⁓ but it needed to be there because if you see that for the first time in the exam or

You start doing practice exams, you’re hit with these all the time. It’s quite a frequent type of question. It’s going to be pretty brutal if you haven’t had explicit instructions to what the hell is actually going on. ⁓ Some of those quadratic ones, I mean, the curriculum had all the stuff about finding roots of quadratics, finding the vertex, and all those kinds of things. thinking about, almost again, riddle-type questions.

these properties are true, what does that tell you about the relationship between the coefficients? so that all had to be ⁓ included. ⁓ Some things like, know, some questions, actually probably one of my favorites, I think that we did add was, ⁓ again, I saw on the the SAT quite a lot, which was like, they give you quite a verbose description of some sort of physical phenomena like electric flux density or something like that. the bottom line is that the kids taking this

aren’t necessarily physicists, they might not know, have the faintest idea what the hell you’re talking about. it’s like, okay,

Justin (44:53) They may not have even taken a single physics course. Flex, they might have no idea what that even is.

Alex (44:59) Exactly. So it’s like, okay, so we need we need a topic that basically explains all this physics stuff to non physicists. Not the way to explain what electric flux density is, it’s getting to the crux of what what’s this trying to ⁓ ascertain the student is able to do is like, take this complex description, sort of highlight the really important parts, use that to write down a formula.

⁓ Check, you do like a unit check, almost like a sanity check. Okay, the units balance. That means that the formulas looks like it’s good to go and then kind of do some kind of manipulation. So we’re not trying to create like mini physics experts here. It’s just that one skill. Take an unfamiliar concept that’s presented in that very, very wordy kind of form, extract out the useful information and work with it.

Jason (45:47) Well, you what’s interesting about this is because, because, right, a lot of students don’t take physics, right? They take biology and chemistry and maybe take two years of chemistry, take basic chemistry, and they take AP chemistry. They don’t even take physics. And so it’s sort of an unfair, it’s unreasonable questions to ask in lot of ways, because it disadvantage anyone who hasn’t happened to have taken physics. And, but I feel like it’s sort of like a, it’s like a…

Assessment by correlation. So students who’ve taken physics and can remember the physics are probably stronger students and they do better at this and they score. You know what mean? It’s just kind of bullshit, you know, advantage that you’re giving, you know.

Justin (46:26) Yeah, it’s enough. It’s like it’s, you can hide behind the idea that like, ⁓ well, we’re not they don’t have to know physics. It’s about parsing ⁓ a unfamiliar word out of a problem statement and like spinning like, just based on the description, ⁓ figuring out what’s relevant on the fly. But but yeah, the truth is, if you have seen this word before, if you have

just in a couple of problems, you don’t have to be like an expert on it, but if you have exposure to it before, then you’re at a big, big advantage. you know that flux, it really just comes down to the amount of ⁓ force field lines going through a geometric surface area. You already have in your head, like, this is really about surface area, right? Whereas somebody who’s never ever seen that before, they’re like,

Flux, that like, what does that have to do with length, surface area, They don’t even know what’s important. So seeing it before kind of tips you off, it helps you leverage this strategy of knowing what’s important because your priors and your head are all shifted. ⁓

Jason (47:35) Yeah, they’re going to be, it’s going to, if you’re not familiar with really going to throw you off, you’re to be like, Oh crap. Right. Can freak out a little bit. Like, I supposed to know what flex is? Well, I don’t know what flex is. It’s not, it’s not like, and I don’t think most people say, well, they’re, they’re going to use some physics stuff. You’re not going to know. And just what we want you to do is just add, go with a description and don’t think about it. But that’s, don’t think what most students are expecting. It’s just, it’s just unfair. So Alex, you had to build up a whole curriculum.

mini curriculum within the math fundamentals course, SAT math fundamentals course, that gave a range of these types of problems. Because there are lots of different versions of this, right? So it wasn’t just like, well, it’s just density or just flux. There were just different. So you had to kind of do enough range so they could get comfortable with a lot of different words and terminology and constants and stuff like that, right?

Alex (48:33) Yeah, yeah, absolutely. mean, again, very, very, very finely scaffolded. So, ⁓ you know, just just the ability to kind of like given a wide range of context, just just to do that, you know, read the description and write down appropriate formula. And then the next step will be OK, do that. But then you have to manipulate the formula and then the next step will be OK. Then we’ve got this one, which is kind of like a rate of change. You have to kind of spot that, you know, hooks law. It’s not about the extension. It’s not about the the length of the.

of the spring is the extension of the spring shift to recognize that that is referring to like some change in the in the length of the spring. So yeah, so we had a few topics that that basically, I think did a pretty good job at kind of like closing the gap. And I think that’s what the missing missing middle is designed to do exactly that is to close the gap between a regular curriculum and like what was actually needed for this for the for the test prep to be successful on the on the ⁓ on the exam. ⁓

I gonna say something about that now. Yeah, so it’s a little bit like, yeah, oh, there’s two things I was gonna say. So the difference between the fundamentals course and the test prep course, I kind of imagine it a little bit like it. Imagine you’re like a professional boxer. And you know, you haven’t got a fight lined up, but you’re a professional boxer, this is what you do for a living. You know, so you need to be sort of showing up to the gym every single day.

training your ass off, you making sure you’re working on your footwork, on your strength, on your endurance, on your jab, on your, all your punches and stuff like that. So just general all round conditioning and fitness. That’s what the, that’s what the fundamentals course does. It just kind of gets you to the point where you’re.

Jason (50:09) And the fundamental boxing skills, right? fundamental. How to move and, you know, eat and punch and whatever, that stuff. These are lot of skills that they’re working on, but they’re not, they’re not, what you call, sparring.

Alex (50:12) Exactly, absolutely. The fundamental box…

Yeah, they’re not exactly training for a specific opponent. it’s like, okay, so okay, so then then then then like, next day you go is okay, we got fight lined up. There’s a it’s a guy who’s from from X place. He’s a Southpaw. He’s tall. ⁓ You know, he’s he’s he’s fit. He’s strong. So that’s the point where you can’t got all your fundamentals nailed. And you’re then preparing for that specific opponent. So there’s certain things you want to sort of like lean into focus more on just to make sure you’re as prepared as you can be for that opponent.

Jason (50:56) It’s like a fight strategy, right?

Alex (50:58) Exactly.

Exactly.

Justin (51:00) If you’re going up against Mike Tyson, are you just gonna not look at his like previous fight videos? You’re just gonna do your normal workout routine and then just show up like like it could be anyone steps in the ring like no, you’re gonna actually like look, what does he do? How does he lean? Where when he throws punches? Are there any like weak points exposed more often than you would see in other opponents? You’re kind of studying him before you step into the ring with him because otherwise you’re just gonna get your ass kicked.

And you know that he’s doing that to you,

Jason (51:30) Right. This is true of all sports. whether you’re, you know, like you, especially at college and professional level, they break down tape. They spend hours and hours and hours a week watching tape of upcoming opponents. Right. And so, you know, when, when, you know, if they’re playing in the NFL and you’re playing on another football team, American football, ⁓ then you’re trying to figure out, what we, okay, they have

they’re a run first offense or they’re a pass first offense. So they have these kind of, you know, secondary and they have strengths and weaknesses and things that they want to do. And then teams that have put up a, either beat them or, or, or taken away some things. How did they do it? What can we learn from that? So you really, right. You start preparing for that team. Now the, the, the, best athletes would do that on their own. Like there’s, think a Dennis Rodman who played for Detroit back many, many years ago.

I don’t know, we did play for Detroit, then play for the Bulls. And he was like this incredible rebounder in basketball, right? And he played the NBA. And I think I read something where he would spend hours and hours and hours watching how these other big men from these other teams would rebound. And he would also, where were certain shooters, when they would shoot, where their ball would tend to go off the glass. So he would position himself under the basket because there’s a bigger tendency

when this guy shoots, it goes short and it tends to fall short. So I want to be in that or it tends to go long. It’s a bounce at the side. he would do all these things. He’d learn how the other big men and the other team would position themselves. So he would be in and have a more leveraged position. And then he would understand also the tendency of whoever’s shooting where the ball would tend to go. And she would add up all these little advantages. And then he was like this unbelievably good rebounder, but anyway, so same goes in all sports. It’s like, okay, you get your fundamentals in place.

Now it’s time to game plan, strategize, come up with and start doing the things you need to do to maximize your performance on the day of the event.

Yeah. So, well, you know, it’s interesting. One thing I want to say, though, so I think it’s a really cool way of thinking about I hadn’t really thought about it in that way. But I remember one analogy I used that I sometimes I think of analogies, I always think of almost like stories. I actually watched a movie. Right. And so I was talking about the missing middle. And I said, imagine because we’re first talking about this. I said, OK, imagine that you hired this really high priced SAT tutor. Two hundred dollars an hour, some crazy amount.

And this guy delivers results. And people say, you know, if you can hire this guy, work with my son or with my daughter and have this incredible results, you know, and so you, you meet this guy and you’re like, so what, what do you do that’s so special? And he’s like, well, you know, I’ve done this for a long time. I’ve analyzed all the tests. I analyze where these students, I first I give these kids, students have really in-depth assessments. I really understand their strengths and weaknesses. And so the first thing we do is remediate all of those things.

That’s obviously what they would do. And then what I do is I’ve developed, there’s typically a lot of missing skills between the harder questions on the SAT and then what the students have covered in school. And I’ve developed my own materials over the past 10 years, you know, where I have, you know, that I have teaching students, okay, do I need to do this and I gotta do this and I’m gonna build on that. And so I call my golden topics. So I have this whole thing, you know, and you’re sitting there as a parent, go, wow, the golden topics.

Like, what are those golden topics? And he’s like, he pulls out his like a little, and he has like a some booklet that he’s put together and said, okay, we see we cover all this exponential stuff and the script, you know, like just backs what you’re talking about. And so we got to come up with this, these golden topics that, that is sort of this, this area, these things are not covered, they’re not covered in the normal curriculum. They’re not even covering the SAT prep books, because there’s not enough room. They get mostly practice tests and fundamentals.

And, somebody has taken the time to really do this so that he, that they can systematically level up his clients, you know, the students, so they have incredible results, which means he can command a huge, you know, tutoring fee and get incredible word of mouth because there was the results are really going to dictate how he is, how successful he, his, his tutoring businesses. And so he spent years fine tuning these.

these golden topics, which is so that’s essentially what this process of identifying the missing mill or identifying the golden, the golden topics, right? So it’s like, because I was like, and my wife, when I was kind of, I kind of mentioned a version of the story to you guys in passing this, just imagine with this goal, what what would those topics be? What would they be? You know, and you’re like, you start thinking like, you know, right, and they kind of gave you a little visualization, like, Oh, I think I know, like, what would give you an unfair advantage?

Right. And so that was another thing I always, I tend to do with Justin. I’ve done this a few times with you. played this little game where, and I’m like, you know, should we do X or Y or whatever? And you’re like, well, I don’t know. It makes a difference. Okay. Let me, okay. me put this way. What if I could do this and then you, couldn’t, would that be unfair? And you’re like, I’m like, well, okay, then let’s do that. Right. Like doing this, would you be pissed if we had a competition and I get a million dollars?

and you don’t because I get to do this. You’re like, well, yeah, okay. Well, then let’s do that. ⁓

Justin (56:57) I think the way you frame it sometimes that is the most punchy is when you say something like, know, okay, if you, if there were a competition and you want to like a billion dollars, a billion dollars, if you win this competition, what would you do? Would you just like, is this a strategy that you would bring to the competition or is there anything else that you would do to maximize your chance of winning? And then when you think of it like that, or I mean, you can also flip it on the other hand, the…

tutoring the bloodthirsty King’s kid, where the King is unimpressed, he’s gonna have your head chopped off. Like, do you take that to the SAT setting of like, okay, if this King’s kid doesn’t get a sky high score on the SAT, like your life is done. Is there anything else that you would do to potentially save your life here? I mean, if there’s not, then okay, but like, is there anything else? Is this your…

infinite accountability, infinite reward strategy. Because if it’s not, then we need to go figure out what that strategy is and do that instead. ⁓

Jason (58:04) Exactly.

Alex (58:05) It’s funny, actually, was a process of identifying the missing middle, I suppose, is one of those things that took a while for us to kind of really hone in on, okay, this is how you kind of this is how you do it. ⁓ There was a good analogy with when we were talking a little while ago about I’m almost like the Admiral of the Navy and Justin is kind of like special ops.

Jason (58:28) It’s funny, I was gonna tell that story, yeah.

Alex (58:31) is a brilliant is a brilliant analogy. Because we did we did actually have a few people that basically I their task was to go and analyze find out what the missing middle was. I mean, this was like months ago. And the thing is, is that unless you’ve got a very extremely comprehensive idea of what the hell it is we’re actually doing here, it’s like it’s very hard to kind of explain that to someone in like a couple of meetings, obviously, like just someone and it’s hard for me to because I’ve got I’m sort of I’m I’m

I remember this Navy fleet. ⁓

Jason (59:01) You have a huge team of, I don’t know, 25, 30, but whatever many it is, it’s always a little bit flexible. You have a big team of people, of editors and reviewers and question writers and all these different skill sets and levels of responsibility that you have to manage. You have to keep this entire fleet moving together with some level of synchronicity to make it efficient. So when I come in and say, Hey, Alex, I think we need to actually just.

What? You know, right. And I do that probably a lot more than you’d like, but sometimes like, got stop, we got to do this other thing. And so you get to turn around the whole thing. We have whole battle plan. We got Navy. got this. We got these certain ships are going into repair and then we got to pull in and you don’t understand. There’s a whole orchestration, a whole coordination thing. so which which happens when you have lots of lots of people who are distributed all around the world who have very particular skill sets.

and very particular sets of responsibilities. And, you know, I come in and I’m like, Hey, I was thinking about something this morning in the shower and like, God.

Justin (1:00:10) Well, the funny part is like, sometimes these things are like, they’re not immediately obvious that they’re huge gain, right? They’re just something that’s like, huh, that’s a little weird. We should look into that. And it’s one of those things you said this before, when you were telling the story, which I think you should tell the story. But it’s like you came to Alex and you’re like, Hey, I think there’s a giant squid at the bottom of the ocean. It’s just like, he’s like, really, you want me to hold just for this giant squid? Okay, you tell the story because this

Jason (1:00:32) Yeah,

So this

was about the problems. So once we got to the test prep course, the idea was that we would bin certain question categories of questions. So you have like, you know, 40, 50, 80 bins or something. So we have we have, you know, 20 bins of geometry questions, and we have different types of questions within each bin that are in a certain type of category. And we have levels of difficulty. And it was like, OK, and I was like, look,

we can categorize this and we can put them in some related category and then we can kind of calibrate the student strength in each one and then they can work on the level they need to. that was like, you know, cause there’s a certain problem solving ability that they’ll just develop as they get more exposure to a certain class of problems within a category. And then I don’t know why I started thinking, was like, you know, is there a hierarchy? I mean,

Is there a hierarchy more than just like they go up? It’s like they play and there’s like a, just like a knowledge, kind of like, is there a knowledge graph there? And I remember, Alex, I said, I asked something to you and you’re like, I, you know, because, you know, cause Justin, thought the same thing for a year. Cause I’d be like, tell me about the SAT. Could we scaffold that whole thing? You’re like, Jason, I don’t know. There’d be.

That probably three, 400 topics. It’s so sparse. There’s such huge variance. just cannot scaffold like you can algebra one or.

Justin (1:02:12) There were two questions. The first question is, can it be scaffolded? Is it in theory, like, is there a structure here? Question number two is can we enumerate it and not have it blow up into like hundreds or like 500, 1000 top? Like how big is this? ⁓ And so yeah, I was very much in the camp of like, you know, I know there’s structure here, but I just, I don’t know if it’s feasible.

for us to enumerate every single little thing.

Jason (1:02:43) I asked this multiple times. mean, I have a habit of asking the same question many times to explore it. like, so Justin, I know I’ve asked you this before and I’ll say it all day. Like, okay, let’s go with me on this. Is this and well, would tend to get the same answer. you know, you’d be like, Jason, I hear what you’re saying. I just think there’s too much. It’s just too sparse. It’s not feasible to scaffold it. Just not in the structure. Anyway, so that was where we all kind of sat with it.

And I can’t remember why I started, was like, you know, I just, I feel like there’s more structure there. Cause if you create structure, you get efficiency, right? And there’s, it allows you to compress the information. Therefore the learning efficiency is higher because you’re not wasting time on things that you don’t need to, and you’re focused more on things that you need to learn. And then you can pull that forward. so, and Alex, I think I brought up to you initially and you were just like,

What? and so you were extremely skeptical and I think probably annoyed, although you probably hit it. just probably went to Kremme and you’re just like, I fucking Jason. And so I go and so here’s the analogy I use. said, like, it’s like, like, you know, I think there might be I’ve been raised to stuff. I think giant squids are probably real. And you’re like,

Alex (1:03:46) You’re skeptical.

Jason (1:04:13) What? Like, no, I’m serious. think, look, I know it sounds kind of out there, but I think there’s a lot of evidence that giant squids are a real species or family of, you know, of animals that are down. I think really giant squids and you’re just like, okay, whatever. You know, you basically, I just, don’t, you know, I’m the admiral Navy. I don’t have time to play around with you. I don’t like we’re going to go look for giant squids.

And I go, Tessa, give me a favor. like, I think, I think there’s some giant squids down there. Take your team, take your, take your boat, go out there. I want you to just, just humor me and go down. And so you went down, you did your, you went to your suburbs, put your guys down and you come back. I’m like, so? And you’re like, dude, like what? And he’s like, well.

One guy has the bends, he’s unconscious. The other guy is in shock and is talking and they won’t even say it. And then we have a picture of like a giant eye and like, my goodness, yeah. Like really? Like, yeah, yeah, yeah. So that guy’s like, there’s a giant squid down there, man. And so, cause what I’d ask you to do is I said, could you just, said, just, there was like a set of topics or something. There’s some category where that would be in a bin. said, is there structure here?

Right. And then I can’t remember what the, what the topics were. And then you went and found it and you said, holy crap. Like it was just, there was just perfect structure. Right.

Justin (1:05:47) Yeah, yeah. There’s a, it’s one of those things where it’s like, ⁓ I mean, the first time you, go through these topics, maybe the structure doesn’t jump out at you, but once that’s kind of consolidated in your brain, you take a second look, it jumps out more. It’s like rereading an essay sort of like you write the essay. It’s two in the morning. You think this essay is great. And you read it again at like 10 AM the next morning. You’re like, wow, okay. There there’s improvements that I can make. And then.

Now you think it’s perfect and then you do it again the next day and then more stuff jumps out at you. ⁓ And eventually it gets to the right level, but it’s one of those things where sometimes in order to really see the structure, you have to try your best, let that sit in your brain, consolidate overnight and look at it again the next day. It’s not always easy to see the structure that jumps out.

Jason (1:06:37) What were the topics? you remember? Do you have to have your head? I mean, it’s been a while. Maybe.

Justin (1:06:42) It was was something in algebra, think. ⁓ Alex, did you remember?

Alex (1:06:47) Yeah, I think it was it was some like linear equations, you know, but we’re progressing levels of difficulty, which I think I just sort of dumped into one topic with some kind of had like sort of like 18 knowledge points or something like that. And yeah, I think I’ll give that to you and use our we could split this. I think I might recognize some structure and you recognize more and then we kind of inferred that there’s just structure everywhere. Yeah.

Jason (1:07:12) Yeah. the thing was, like, I was just skeptical that there wasn’t structure. I just didn’t believe it. Now it was just an instinct. It was just an intuition. Well, obviously I didn’t have any hard evidence as it’s like, it just, this doesn’t seem right to me. I just don’t believe it. And, you know, Justin, you not having an entire fleet on your command, you’re just…

you, you just like, I, so you’re like, you see, you want me to just spend the weekend? Let me get just to just go do this. Like put everything on hold. I want, I want to answer this question because if I’m wrong, I’m wrong. Right. If there’s no giant squid, you know, if there’s no way to scaffold this stuff, then screw it. Right. At least I’ll, is I quit having this nagging question in the back of my head. But if there, if I’m, if my instinct is right and there is structure, then well, we got a whole lot of opportunity here.

And so you came back, we’re like, yeah, hell, holy crap. And then you looked at it more and you’re like, my God. Like it was like a multiple phase thing where you’re just like, my God, right?

Justin (1:08:11) Yeah,

well, was, you know, ⁓ it was really coming face to face with the manifold hypothesis playing out again, the manifold hypothesis being that like all of life is just in this very high dimensional space, right? There’s like tons and tons of possibilities of things that in theory could potentially happen. ⁓ But the things that actually happen lie in a much lower dimensional subspace within that space. And so it’s one of those things where it’s like, okay,

Well, based on the SAT, based on all these foundational skills, they can be pulled together in so many different ways. I mean, you, can do the calculation. Let’s say you have like a hundred subskills, 200 subskills, and you’re computing how many different combinations of, uh, of three or four of those subskills. I mean, you get an astronomically large number and it’s just something where it’s like, okay, well, there’s no way that we’re going to hit all of these combinations. But, but once you actually look at the exam.

And you see these combinations that show up over and over again. It’s a much, much smaller space. It’s not this astronomically large number. It’s a large number. mean, it’s like, what, like we 115, missing middle topics. mean, it’s. It’s, not, it’s, it’s not a trivially small number, but like you can do it. If you’re willing to put in some, some elbow grease and, and, and do the work, you can turn this into a legitimate.

highly scaffolded course that can take a student, even a student who’s not a particularly mathematically ⁓ gifted, ⁓ you can get them to fill in a lot of these gaps that a lot of ⁓ genius students are just kind of inferring on the fly. You can make that explicit for them and scaffold them through that process and get them to at least in the subset of knowledge that is covered on the SAT, you can get them

highly trained on that.

Jason (1:10:13) Yeah. Yeah. Yeah. So, so that was interesting. so, so when you, when you came back and you figure this stuff out, and then we had a conversation with Alex and Alex was, you were like, okay, right. It was, you saw what the research, the little proof of concept stuff that Justin had done and you’re like, that’s pretty, that’s pretty ⁓ convincing. Right. And so that’s, and so that’s when you said, all right, well, let me start looking at this. And you started going down that.

rabbit hole yourself, right? And trying to find structure in these things, then having the team help you with.

Alex (1:10:48) Yeah,

yeah, that’s right. So yes, I was convinced after Justin’s research. And ⁓ I thought, yeah, I think we can I think we can make this work. I can also see advantage. Obviously, we’ve got a really, really efficient system. So it’s, it’s, it’s, it’s beneficial to try and work with the existing infrastructure that we have, if we can get it to work. So was like, okay, that’s, that’s, that’s, that’s, that’s really motivating. ⁓ And yeah, the research that Justin did made me

convince me. yeah, so that was the point where I got like the rest of the team involved in terms of organizing all of these, you know, I mean, what was that sort of 1500 1700 questions by this point, they actually had like 10 of each type. So it’s kind of like 17,000 questions.

Justin (1:11:33) It’s like an archaeological expedition. I found the head of the T-Rex. Of course there’s the rest of the T-Rex here. Look at the whole team excavating this. We can bet on the rest of the fossil team.

Alex (1:11:43) Yes, exactly. So it was kind of that. was just, OK, we’re going to make this work as best as we possibly can. So I got someone like the top guys on the team sort of helping me with the actual structuring. So basically categorizing all these different questions into hierarchies and individual topics. Do what we usually do, sort of three or four knowledge points each. So all grouped ⁓ with hierarchy into topic hierarchy too. So increasing difficulty as you go through the topic.

And yeah, and it worked. We managed to map out the entire ⁓ bank of questions that we had into a hierarchy complete with proper topics, all the prerequisite relationships between them. The Knowledge Graph is still pretty flat, as you would kind of expect. But there is a lot of structure there. mean, what’s this, five or six layers deep?

And bearing in mind, of course, is that the whole point of the fundamentals course is to make sure, is to get people to the point where they’re, they can really answer most of these questions already. So once they complete the, cause what we do is we don’t allow, we’re not gonna allow anyone to sort of take the fundamentals, to take the test prep course until they complete the fundamentals course. So they can-

Jason (1:12:57) You can only be promoted into it after you’ve completed it. You can’t just sign up.

Alex (1:13:01) Yes, exactly. So that means they’re in pretty good shape by the time they’re hitting the test prep course. So they should place out of a large part of it, which means although we have something like 250, 300.

topics in our test prep course, they’re going to be kind of hovering around the leaf nose where all the sort of the this kind of tricky, the harder problems are where there’s sort of the the middle, the distance between the where the fundamentals course finishes and the test prep course sort of starts, if you like, which isn’t which is kind of a thin layer relative to the rest of the course. So yeah, it’s a we got it to got it to work. that was that that was the point where me as sort of

Navy commander had to go kind of like in communicado for like three weeks, weave all these topics together. In terms of like, you get getting the knowledge graph structure in place, it’s one thing to have the topics, but then you actually got to get all the kind of the knowledge graph and all the data correct for that, the team helps a lot with. But yeah, a lot of it is stuff only I can do.

Jason (1:14:06) Yeah.

Justin (1:14:07) I you messaged me that weekend where you were like, man, I just finished setting all these prerequisites and stuff. I think I’m just going to lie down in a dark room for a little bit. I think that’s something like that. was like, yeah, sounds about right. That must have been a ton of work.

Jason (1:14:22) part of what we kind of came to realize as we identified this structure, even as we’re getting, because I think I’d asked Justin at some point, even after we realized that there were structure there, I was like, how much

space being once we scaffold this stuff all the way, how much problem solving is left? Like is it we’re to get ourselves nine? Because I said maybe we only get ourselves 75 % or 80%. 85 that’s still a huge win if we can scaffold, you know, in the missing middle and in the test prep, but maybe there’s just 10 15 % that’s just problem solving that, you know, it’s just going to be sort of a

inefficient, but you’re just going to have to do a lot of problems and in a certain category and hope that you sort of pull together some skills. And I said, what percent? And you’re like, I don’t think there’s anything left. I mean, I we’ve closed the gap. It’s like air. It’s like an airlock. There’s no, there’s no problem solving required. We totally full.

Justin (1:15:24) Yeah, we’ve enumerated it. How well did we enumerate it? 100%. Like every there’s no like, what do you what do mean we have to like, think about like, getting to the off manifold stuff? No, we got the manifold. That’s it. We excavated it. The full thing. All the bones. We got all the ribs of the t-rex, everything. There’s nothing to fill in.

Jason (1:15:43) All

relation required it’s left. so which, which brought up it, which raised an interesting question, which was like, okay, well, if we could do this with the SAT and then I’m sure you do the ACT, I mean, can you do with this AMC questions? Can you do, you mean it is every, or is all of these things, it’s like, wow, it’s problem solving. And it’s like, you know, and I started to thinking maybe problem solving was just a word for, don’t know.

I have not enumerated. I don’t really know. It’s just it’s problem solving. Right. Whereas like, OK, again, and I think, Justin, this is where we had the discussion was, OK, you can only do problem solving. get to enumerate. Who wins? And you’re like, well, enumerating the not getting disgefolded approach. OK. So the question is, how how well can you enumerate it? Like, is it like, OK, yeah, you can enumerate it, but it’s infeasibly large. So that’s not possible. But.

That was kind of where started the question of, or sort of the idea of the unreasonable effectiveness of a knowledge graph. Because you can enumerate a knowledge space, a domain with a knowledge graph, then it removes this dark areas. It’s like you totally flood the room with light. They’re no longer dark areas. It’s like problem solving, problem solving, problem solving. It’s gone.

Justin (1:17:03) You know, this is almost like the thermodynamics of knowledge acquisition in the sense that, okay, the work has to be done for the student to acquire knowledge, but the knowledge graph, it’s this trick that allows you to trade off student work for instructor work. Instead of having the student just like go through this, this hodgepodge of problems and hopefully they can infer all the structure.

⁓ and just have them grind through a large enough volume that they can infer what you do is you say like, okay, well, we’ll just have the instructor, the content, the system, we’re going to work on the system, figure this all out behind the scenes. This is not going to take up your time as a student, where you have this all ready for you. Get the structure, present you with the structure. ⁓ it doesn’t mean that there’s less work that has to happen. It’s, but there’s less work for the student. We have to do more of the work, but that’s, that’s the trick. ⁓

And we can do our work at scale versus the students.

Jason (1:18:06) Imagine it

like you can almost imagine it like this. Like you’re in a building and there’s no there are no stairs or elevators and the only people who can get to the top have to do be on belay and they have to kind of go up and you have to be pretty athletic and pretty good shape and it’s a lot of damn work to to to get to the top you just using like a guy’s you’re on a kind of rock climber and you’re like oh we’re just gonna build upstairs.

It’s like you still got to walk up the stairs and maybe take a very hard, but there is each step is the same distance and just step, step, step, platform, step, step, step. And so it’s like, yeah, you don’t have, I mean, you you can’t be like 90 years old and get to the top of this 80 story building probably, but as long as you’re in reasonable shape and you’re willing to walk upstairs, anyone can do it. And that’s, think the sort of unreasonable effect of the knowledge graph. So Alex, I know you had some things to say about it.

Alex (1:18:59) So one thing I think that’s something that occurred to me quite recently really is that, I mean, the great thing about having a knowledge graph is that it’s a very strict implementation of mastery learning. So we don’t let you do topic X unless you have mastered or at least proficient in all of the prerequisites for topic X. So you have to have demonstrated that. ⁓ Now, as curriculum designers, especially when we’re looking at some of these kind of missing middle topics or test prep topics,

One thing it’s really, where I think is really powerful is it allows you to kind of include all those really subtle prerequisites that could easily slip under the radar. But if you’re just analyzing these problems really carefully, it’s like, that’s a really important prerequisite. I’m going to make sure that that is a prerequisite of this topic, which means you know the student is prepared with all the necessary prerequisites before they even see the topic.

I there was one particular type of question on the SAT, was to do with finding that the, know, the, the, the, the, think it was like the smallest, smallest difference between the mean of one data set and the media of another data set. And as a whole bunch of like really quite subtle prerequisites that are required to answer that question, like, you know, how do you find the, the smallest value of, of a difference? It’s like something you students need implicit, explicit practice on and you need to explain that.

⁓ And it’s even a function composition turned out to be a prerequisite. Again, it’s like you, by having these very, very highly scaffolded, highly enumerated areas of knowledge, each with their own sort of like what we call key prerequisites, you can capture the essence of every single problem that you want the student to face. And it’s like, they’re not going to see that problem unless they’ve mastered the prerequisites. Even those ones that are kind of very likely to slip through the radar.

in a regular setting or whatever. I think that’s the further we go into competition math and things like that, think that kind of encapsulating those really subtle key prerequisites is going to be really, really helpful thing for us ⁓ in terms of getting the student to the point where they can answer these questions. And we’re confident they have the prerequisite knowledge. ⁓ Another thing, of course, just by simply enumerating all the prerequisites, you can say to yourself, well,

Hang on, this topic’s got 15 prerequisites. That’s way too much. That’s cognitive overload, huge issue right there. This needs to be split up, organized in a different way. So having that information just in front of you allows you to plan, to take all the things we know about the science of learning into account, incorporate all those sort of subtle prerequisites so that the student is not struggling at any point.

Now, how far we can push that with competition map, AMC, and all that stuff remains to be seen. I think we’ve got bigger tests ahead ⁓ than the SAT. But I think the SAT has given us a very, very solid base to see how far we can really push this idea.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.