Watch on YouTube or Listen on Spotify

What we covered:
– The benefits of short problems. Math Academy problems typically take only a minute or two. This way, students can stay on the rails with lots of reps, successfully building up complexity instead of getting crushed by it from the start.
– What goes wrong in college math classes: they tend not to scaffold content very well, forcing students to build their own bridges across knowledge & skill gaps. Weekly problem sets often consist of a handful of hour-long problems that instructors hope students will “self-scaffold” up to. In reality, what happens more often is that students fall off the rails.
– Founders of growing start-ups cannot be hands-off. “Things falling off the rails” is the most realistic and most dangerous failure mode, not micromanaging. Founders of small, scaling companies need to be in “founder mode,” not the “manager mode” that CEOs of huge, well-established companies are in.
– Within teams, it’s important to let conversations flow out of scope. Every innovation, every solved problem, requires relevant background context, and you often don't know what the full context is beforehand. It's easy to let conversations flow out of scope when you like who you're working with and what you're working on.

0:00 - Introduction
1:32 - Why Math Academy problems are short by design
9:48 - Long problems dilute reps on the skill that actually matters
11:00 - Isolate the new skill first, then recombine into full problems
14:10 - Typical undergrad math classes: too few problems, too complex from the start
18:07 - The proof skills gap: often assumed and not taught
29:32 - Alignment decay: teams naturally drift out of sync unless continually aligned
35:04 - Small misalignments compound fast
38:28 - Founder mode: stay in the weeds to stay in sync
49:07 - Early, frequent parent communication avoids end-of-term blowups
50:48 - High-trust collaboration requires relentless communication
57:42 - Out-of-scope conversation enables context sharing
59:14 - Over-scoping kills context sharing
1:00:51 - Enjoyment & trust fuel context sharing
1:06:13 - Missing context produces confidently wrong outcomes
1:10:01 - LLMs fail when context is missing
1:11:38 - Humans fail when context is missing
1:14:19 - Online discourse fails when context is missing

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Watch on YouTube or Listen on Spotify

The raw transcript is provided below. Please understand that there may be typos.

Justin (00:00) Welcome to the Math Academy podcast. I’m Justin Skysack, Chief Quant and Director of Analytics at Math Academy. And I’m here with our founder, Jason Roberts, to talk about one of the biggest dangers in both learning and startups, falling off the rails. This unfortunately happens all the time in college math classes, where each weekly problem set might have just a handful of really long problems, each taking like an hour or more.

The hope is that students will somehow self-scaffold across the large gaps in knowledge. But the reality is that many of these students can’t build those bridges on their own. So they just fall off the rails. And the same failure mode even shows up in startups. Founders of growing companies can’t be hands off. The real danger isn’t micromanaging, it’s letting things fall off the rails. Early stage companies need founders in founder mode. In the weeds, actively keeping everybody aligned.

not manager mode, which you might see at big stable organizations. But here’s the interesting part. Yeah, keeping things on the rails requires relentless communication, but it’s also important to often let that communication flow out of scope. Every innovation, every solved problem requires relevant background context, and you often don’t know what the full context is beforehand. Staying on the rails requires context sharing and overscoping limits context sharing.

So how do you force things to stay on the rails, yet allow conversations to flow out of scope? Let’s get into it.

Why don’t we talk about ⁓ the inspiration for various features on the system? Like, why are lessons the way they are? Why are the problems, why don’t we do like a math problem set, like 20 minute problems? Why are our problems like generally like a minute, two minutes? Or in the really, really advanced courses, maybe sometimes you have a three minute problem, it’s not very common, but generally it’s.

Like why, why, why do we scope them down so much? ⁓ yeah. Why do we measure answer time on the diagnostic? Why not just let people spend as much time as they want on the problem? There’s things like that. Yeah. I’m sure some listeners are, are familiar who have, who have like, you know, read the, math academy way and, ⁓ are just like really following our stuff lot, but just, just these are, these are kind of common, common questions that I think we have kind of interesting answers to. And.

Not just interesting answers like why, also like failure modes. Like I can tell you like five different ways that the learning experience can fail if you don’t do a lesson the way that we do it.

Jason (02:44) I mean, there’s more than one way for something to succeed, but there’s an infinite number of ways for them to fail, right? So it’s sort of like, you know, it’s not there’s not one way, but there’s, that’s, it’s, there’s a limited number of ways to get a really good outcome. There are a lot of ways to get a good, a good to great outcome, right? And just starts expanding to get a pretty good or acceptably good. Okay. There’s a lot of ways you can do it. And like, okay, it’s an okay. They learn some stuff is pretty good. It’s a lot of ways to do that.

But then we’re just failure like infinite failures. There’s just so many ways to make things just go wrong. And a lot of instincts that people have are just kind of backwards. They just don’t work. There’s a lot of sort of delusions or fantasies about what they think should or shouldn’t happen. And it isn’t until if you were held accountable, that you go, guess it doesn’t work so well or isn’t so efficient or whatever. let’s talk about problem time.

I think that’s an interesting one that you brought that up. What’s the optimal length that a problem should be? Now, I think you’d have to think, OK, per level of math. So the optimal problem length for a fourth grader is not going to be the same thing for optimal problem length for a student in multivariable calculus. Another 20-year-old might be doing if they were going as part of their standard education. First of all, the

There’s this the difficulty of the math. So if I’m adding seven plus four or something, like, you know, it just doesn’t take that long. And but it’s you’re just, they’re simple. are basic fundamental building block skills that are just that are in themselves simple. And so you just want to get a lot of reps on those things. But of course these the attention span of a kid that age, right?

And as they get older, can have a little more, a little bit of time, which you’d expect out of a seventh or eighth grader, it’d be more than a fourth grader, but not like a 10th grader and so on. So the older you get, the more ability you have to sort of control, more self-control, I think. And there’s a sort of academic term for it, escaping me at the moment. you, right? What were you guys saying?

Justin (05:03) have that to contend with.

Oh, we think of executive functioning.

Jason (05:08) Yeah, executive function. Yeah, executive function would be one. It wasn’t the exact term, but you’re right. You know, what their, where their executive function is, what I, which kind of determine what should I be doing? You know, like, you know, as far as the task selection or what part of the task would I be doing? It’s sort of the organizing or planner, you know, of your brain. ⁓ so an executive function, it’s not a part of your brain, but part of the cognitive process. ⁓ so. ⁓

The thing is like, okay, if you think about it like this, okay, so if I’m, if I’m running, I don’t know, let’s call it soccer practice, coaching soccer. And I say, well, there’s a lot of things these kids gotta learn. They gotta learn how to, you know, depending on what age group they’re at, right? Well, let’s just say I’m dealing with some, you know, 12 year olds or something. It’s like, well, a lot of them don’t know how to tackle that well. A lot of don’t know how to, you know, lot of them can’t strike the ball very well. Some don’t need to work on their dribbling. We need to be able work on.

how to mark up defense, we need to work how to cross the ball. But there’s all these things we gotta work on, right? And I can think of 10 really, really important things that just will move the needle, like now. And then I got another 20 or 30 that we can, that are also important, but I need to get to, okay. So you’re like, okay, well, how about I spend all practice on one of those skills? You’d probably go that, so they’re gonna be really good at tackling, but they can’t dribble and they can’t shoot the ball and they can’t.

do anything else. You’re like, yeah, but they can tackle like nobody’s business. That isn’t, not only is that not going to be an optimal outcome if you’re like competing against other teams, you know, cause we have like a, you’re held accountable and on, on, Saturday’s soccer game, right? If you get to lose 10 to nothing and the parent, kids are crying, the parents are like, what are you doing? These are kids seem like they can dribble and stuff. Our kids can’t do anything. You’re like, well really just want to spend all the time.

So you want to apportion it out over all of the fundamental skills that are really important to this overall effort. And so you think, okay, if I’m going to have them work, I can only work on, I don’t know, an individual practice, I can probably work on five of them because practice isn’t three hours long or you like an hour and a half at best.

You know, kids don’t listen. I gotta get them organized. There’s some friction in there to just getting everybody doing what they’re supposed to be doing. And so you think, okay, okay, so in each individual skill, like how much they need multiple reps, how can I get enough reps on each one so that it sticks? Because if I say, okay, you do one rep.

You get to strike the ball one time and then we’re doing something else. The kid like, that’s not enough. Can you, do have to stick it, take strike at 200 times? No, that’s, that’s excessive. Like what is it? I don’t know. Depending on it, 10, 20, 30 times, if it’s something, but if it’s something that I can do, it takes 10 seconds to do your strike. Your two, two players are, you separate them by 30 yards and you’re striking it back and forth. And maybe you can get, you know, a strike every, you know, 30 seconds.

You know, you kick it back and forth, the guy traps it, steps it on, okay, step around, strike it again. Okay, sometimes you miss and you have to go get it. Okay, so five minutes, you get, let’s call it 10 strikes. You’re like, okay, I’d like 30 strikes, but I got to work on throw-ins. I got to work on it. So you’re like, what’s, so you’re thinking, so you’re trying to set up the drill so that it’s as efficient as possible. Because if I only had two guys kicking it and the rest of the players in the line watching them.

then everybody gets like one or two strikes and the drill with 20 minutes. I say, what if I had every we have, you have a partner, everybody spaces out, everybody’s, you know, 20 yards apart or 30 yards apart. You strike it back and forth. This looks a little chaotic, but everybody’s getting lots of strikes and I’m walking along and make sure everybody’s, you Hey, hey, now get the ball set. Come on. You gotta plant your foot. Okay. Good. Good. Hit it. Strike the follow through. Okay.

look, aim it to the guy, right? So he doesn’t go chase every time, right? Okay. And I got, you know, I see you kind of do the correction. So again, again, you’re like, okay, I got everybody got 10, maybe 15 solid strikes. You’re kind of like, okay, that’s good. Now, if I had done it the other way and they got like, and they sat in a line and I got one strike, you’d be like, you know, an experienced coach would be like, you know, Jason, I think you get more out of your practice time. If you like organize structured things a little different and I’d like, oh yeah, that’s a good point. Right. You know, so

So you think of it from math, you’re like, okay, well, if I have a math problem, takes 10 minutes or 20 minutes, then how many reps can I get? And there are, in any particular problem, there are very specific component subtasks that you really want them focused in on. There’s other stuff in there that’s like lot of arithmetic and stuff. Well, this turns into, okay, well, after I do this,

integration or differentiation. got these fractions and I get those fraction arithmetic and I got a, you know, it’s like, I know, you know, fractions. Like, I don’t want you spending seven minutes on the arithmetic.

Justin (10:26) To

concentrate your practice on the stuff that is new, that you’re struggling most with, that you don’t know yet. Concentrate on that. Pull that out of the skill. mean, we’ll practice putting it back in, doing the whole skill altogether. And yes, you do need to keep doing stuff with fractions, but that’ll happen naturally. Like, well, we’ll have some problems like that. We don’t need to go crazy on it every single time. We need to really…

hammer down the new movement that we’re trying to get you to learn.

Jason (11:00) Exactly. I think one of the, if I recall, the course that was the most difficult to do that with was multivariable calculus. In linear algebra, maybe two, both of those can blow up into the massive computations. I remember Alex

really work because I would, because I remember like some P of team, would create these things, you know, and then we have a problem, it takes like seven minutes. And I remember early conversations with Alex, like we got it, we got to try and get these things to a few minutes. And then he was like, yeah, that’s a good point. And then he really went back and was like, well, how can we frame these problems so that maybe the setup part, maybe it’s the setting up of the integral or calculating some of the integrals important where we’re asking for is I don’t need you to complete the entire computation. That’s going to take.

five minutes, I just need you to recognize the context, the situation, the parameters. I need to parameterize integral. need you to figure out what the function is that you were actually integrating. I need you to figure out some, you know, whatever the bounds or the things, and then maybe do a couple of steps. And like you’ve gotten, you squeezed 90%, 90, 95 % of the problem and you did so in 90 seconds. But if I made you do the whole thing top to bottom, it would take you 10 minutes.

Justin (12:20) Yeah, not only would it take you 10 minutes, but there’s a likelihood that you’re gonna, like, you’re gonna fall off the rail somewhere in there. Especially if this is your first encounter and you know this problem is gonna take you a while. And now you’re five minutes in, you’ve kind of drifted off and you’re kind of doing the wrong thing. You spend the next five, 10 minutes doing the wrong thing and then you get the problem wrong. Now like 15 minutes have passed and you’ve just realized like, ⁓ that’s what I got wrong. That I need to do.

this differently. It’s better to just like, let’s let’s just start by focusing on the new thing that you’re most likely to mess up, right? It’s getting strong on that you get like maybe you get the first problem wrong because you’re because you because you have a subtle misinterpretation of what it is you’re supposed to do and then that draws your attention to like, ⁓ I get it. I should have done this. And then only only one minute has passed. Now you you fix that on the next minute, two minutes have passed, you get a few reps doing that correctly. And now we do that again on another part of

of the problem and now after that we kind of pull it all together.

Jason (13:23) Right. You’re you’re you’re you’re doing you’re making them do the ugly parts some of the time, but on different problems. Right. Sometimes you have to do the finishing part. Sometimes you do the setup part. Sometimes, you know, whatever. But we’re going to we’re going to focus our fire on the on the part that really matters, the tricky, the new part, the part that you really the student doesn’t get, you know.

I know you can differentiate this function or integrate it or do this, but I don’t know if you understand how to set up an integral for the spherical, you know, polar equation or something. know, which that’s when we need reps and I’m going to show it. You’re going to do one, you know, like you get it anymore. If you strike the ball one time, you can, you’re a great ball striker. Like, I’m sorry, you got, there’s a lot of reps on this thing. So, what’s interesting is I think like a lot of universities, you know, they would have to the whole problem. So if you’ve got a problem set.

You would fill up pages of stuff doing your multivariable calculus problem sets, right? Your homework. Now, I remember you kind of placed out a lot of that stuff because you did it on your own when you were in high school and then you were more proof-oriented level stuff. I remember your now wife, Sanjana, when she was at Caltech.

Justin (14:46) I saw a lot of the problem sets and stuff like that. It was enough, like it was very clear. It’s like, okay, you have a very small number of problems and these problems are like really, really hard. And this is kind of like the standard. I I did help Sanjana’s sister with some of her math. Really? Right.

Jason (15:09) analysis, which

is super, which is known to be a very, very difficult class for even talented math majors at at really, you know, selective schools. This is this is always a hard class.

Justin (15:26) Yeah. Yeah. And so the types of problems that she was getting is like, well, this is typically how it goes, but you get like, I don’t know, like five, 10 problems on a weekly set or something. And these problems are all like really advanced. Like you’re not getting reps on just your standard definitions, ⁓ proving that something simple is a closed set. No, we’re going to give you some really

crazy ⁓ situations, some really crazy construct this crazy mathematical object. And you’re supposed to infer that it’s a closed set or an open set or whatever. and, and we’re gonna then, and because of that, you’re also supposed to infer that, well, like you can use these theorems for closed or open sets that are going to take you into it. Like, I mean, yeah, you got to build up to that, of course, but like as, kind of your first

It’s your first like, no, like just strike the ball. We’re not, we’re not practicing like your buddy kicks it to you. You do like a twirl and like, like you’re upside down and then you kick it into the top corner of the goal. Like we’re just, you just got to kick the ball first. That’s what we’re going to, that’s we should do first, but that’s often not what happens. And so of course it’s like, it was just kind of like, unless you, what really used to happen, what,

What happens in that sort of situation, the students who succeed in that either have learned this in a better practice conditions elsewhere, or they have figured out how to kind of scaffold themselves through the reps that need to happen. Like, I think like, okay, I know this has something to do with closed sets or open sets. ⁓

I forget exactly how the proofs work in that. Let me spin up on that. then you like go on, you take it upon yourself to go through a few examples of that. And then you return back to your problem. It’s like, they’re trying to force you into doing some like self, self scaffolding, like, ⁓ for you to teach yourself to this level.

But they’re not telling you how to do that. Everyone just assumes, like, oh, well, of course, students should know how to scaffold themselves up to this level. And if they don’t, then they’re just lazy, which is not true. This often doesn’t occur to students unless they work with somebody who shows them, like, OK, all right, this is how you approach the problem. These are the heuristics for getting yourself from A to B.

And like one of the things that, I mean, probably the best example of something that was not ⁓ explicitly taught in the class, but is like the number one technique for all sorts of proofs is that it’s kind of like you’re, often are trying to just, you’re trying to transform some input assumptions into an output result. And you’ve got these two ends of the rope. Now that the closer, the closer you can move these two ends to each other, the easier.

your time you’re going to have just like you having the insight of how to bridge the gap. So what you want to do, like if you don’t see how to approach it immediately, then it’s like, well, why don’t we just take the input assumptions and sort of like, are there any theorems that we can apply to them to sort of get in the direction of the output result? ⁓ And then you do some work on that and you’re like, well,

Okay, we got a few things that we can do here. And then you start at the opposite end. You’re like, okay, here’s the output result. Let’s work backwards from that. What are some states that if we got here, then we could then conclude the result. Let’s just figure out what those could be. And then you kind of work backwards a bit. And then eventually you realize, ⁓ hey, if you take the input in this direction,

And you take the output, bring it back in this direction. Then those two quantities, those two expressions, whatever they start to sort of look a little similar to each other. You don’t see it fully off the bat, but it kind of brings your focus into like, okay. So really I bet what we’re going to have to do is, turn this thing into that thing. Now you’re, you’re gap. Now you just recurse, right? It’s same problem over and over again, but now you’re, you’re, you’re two ends of the rope or closer.

And you just keep doing this until you complete it. any, so this was like one of those meta principles that was like, with the types of proofs that, she was doing that you really have to be good at this. ⁓ but it’s something, it was not scaffolded up and it was not, this meta heuristic was not taught explicitly.

Jason (20:17) Yeah, so that’s, that’s known as the forwards backwards method, right? So it’s ⁓ for anyone who’s unfamiliar. ⁓ and what, whenever, when I’ve proof methods approved to my sixth graders. So this is normally a freshmen math major classes for students who’ve completed calculus, ideally before they take real analysis or abstract algebra. they used to not do this in my time, but over, I think in the 2000s and 2010s that was became more more common. They’re like, okay, we need this.

teach, we need to insert a course on proof techniques, methods of proof, introduction, abstract mathematics, it goes by different names. And, you know, they teach you all these sort of principles and these rules of logic and, you know, working with sets and things like that. And, and then once you have those, you’re in a much better position to tackle something like real analysis, which is applying these methods of proof to real number line and into theorems based on the real number line. And so

Or an abstract algebra, you’re like algebraic, applying it, method of proof to algebraic ⁓ objects. ⁓ So I remember that what I would do is I would say, OK, guys, so here’s what ⁓ we’re proving, and here’s what we know. And it would usually be, and oftentimes, is written either in partially in English,

or partial or like pseudo English. wasn’t like, and then I would say, let’s, let’s pick a series of steps that would convert this sort of lingual English description of this mathematical statement into an algebraic and to an algebraic statement. It’s purely like, y equals this or whatever and equals that. And then I do the reverse is written in and then I put that in algebra. So then you have algebra algebra, right? So it’s like, it’s like they’re both written in Japanese.

I don’t really go Japanese, I’m gonna convert both to English.

Justin (22:17) Yeah. I work with them. Right. Yeah. So you’re just trying to make them more, like more, more familiar and more similar to each other. Yeah. More concrete. Yeah.

Jason (22:19) Right?

You can

reason about algebraic equations, reason about equations, can redundant algebraic equation reason about equations with the rules of algebra. But when you just have English statements, it’s really hard to like, how do you reason about now? A mathematician could say, well, we hand wavy. Well, this is a closed set, so we know that and it’s just a bunch of everybody kind of knows what the other math is just talking about the other mathematician because they’re just compacting.

You know, pages of algebra into a few statements and they’re all willing to accept it because they all agree that that’s true. But when you’re dealing with younger students or you’re really trying to get them to understand rigorous. Careful reasoning, then you have to you have to go. You’re supposed to go to the algebra steps and then you just apply algebra. So we take English to algebra, algebra, that, that, that, that, that, that, you know, a bunch of algebraic, you know, steps, then convert that to English and you’re done. Right. So, OK, so let’s.

It was funny. was, was, but going, then you doing a lot of reps on that was absolutely key for the student. Absolutely. yeah. tons of reps on that stuff.

Justin (23:34) Yeah, yeah, exactly. And it was like, that was really, was after we did a bunch of reps and that sort of thing, and a couple other things, or there’s other kind of ⁓ tricks, like, like if the statement, if you’re asked to prove a statement that feels obvious, then you should use proof by contradiction. That’s another sort of thing. Or if you, if you’re doing, you’re bringing the ends of the rope together and you, you’re like, well, isn’t it kind of obvious? Then just make that step of proof by contradiction.

Jason (24:02) It’s like in movies, like foreshadowing, like they show the gun on the table. It’s like, I don’t know, do you think they’re gonna first get killed by the gun? My guess. Yes. Why are they showing the gun sitting on the table? Right.

Justin (24:11) Yeah, exactly.

Jason (24:13) So they’re kind kind of, they’re giving away the game a little.

Justin (24:17) these were like, I mean, these are just like these these kind of golden prerequisites. You had mentioned like the golden topics for like the SAT stuff that like you realize not taught in the school curriculum, but you really need to know it if you’re going to do well. Yeah, exactly. Exactly. And so there’s the same kind of stuff when it comes to proof writing in particular. ⁓ And I mean, she

Jason (24:32) You do, you have massive band.

Justin (24:43) just, she didn’t have those prerequisites in place and that was causing some struggle. Of course, no, of no fault of her own. was just, this was never taught explicitly. And you probably, you don’t really figure this out much. And unless you’re really like, you just look into this stuff, solve problems for, for fun. Like you just, you learn, you pick it up elsewhere. Cause you watched some math video or you’re really into like the hearing math, summer stuff. Yeah. Yeah. Exactly.

And but once we filled those in, it’s like, well, okay, she doesn’t need my help anymore. Like she was, she’s good. I mean, it it took quite a bit of work to fill those in. ⁓ it wasn’t just like a one day or two day sort of thing. Like this was sort of an ongoing thing for, for, for a few months, but, once,

Jason (25:27) bright student who is in a very good elite school. They worked very hard. So these are not, this is not a half-assed student who low aptitude, not indifferent level of motivation. is no, everything was set to go in this situation, but it sounds like she didn’t, she hadn’t taken, or they didn’t offer a methods approved course after calculus and before real analysis. did, they just kind of did the.

Justin (25:51) I can’t remember

if there was, but I

Jason (25:55) If they did, she was still lacking those skills.

Justin (25:57) Yeah, exactly. So I actually took yet. ⁓

two methods of proof courses in my background. One was at a community college, one was at Notre Dame. ⁓ But funny enough, I took the one at ⁓ Notre Dame after I had taken all my advanced math courses. I needed another credit for that. But anyway, but it was kind of interesting just to see what was covered. like, mean, this sort of stuff wasn’t really explicitly gone over.

It was one of those things. Yeah. mean, the thing was it’s like they would give you a lot of problems to do. was much more of a rapid fire problem setting. Not okay. I should say it was not anything near, but like, oh, this is a two minute problem. This is a three minute problem. It was more like maybe instead of like five to 10 homework problems for the week, maybe you had like 20 or something like that. little better. A little more scope down.

Jason (26:35) point of that course.

Yeah.

Justin (27:03) But ⁓ it just, yeah, I don’t know. It’s kind of like the, there were some examples of how to do these problems, but there wasn’t just a.

I guess the examples were not exactly narrated with a, like, what’s the overall procedure of what we’re, what’s the overall idea of what we’re trying to do?

Jason (27:30) why are we doing this? When is it gonna be useful? How does it apply? get some practice and not just an explanation, you need practice applying it in a context where it all links together and you’re like, I get what’s going on here. So.

Justin (27:44) Yeah.

Yeah. So it’s just, it doesn’t seem to be, it’s like proof, proof, it’s proof writing course, even, when there’s an explicit, like introduction to proof course that has meant to bridge this gap. I mean, I’m sure it helps more students cross the chasm, but it’s still, there’s still large steps that you got a long jump over.

you can’t do that, then you’re kind of screwed unless you have somebody else to learn it from.

Jason (28:14) like in terms of accountability you do okay either one you’d have to have like someone senior in the math department really defining what’s done in the course in terms of like we’re using this book which they probably do you’re using this book but really emphasizing these are the these are the specific things they need to be able to do by the end of the

quarter because when they start real analysis or abstract algebra, whichever one they do first, they’re going to have to do proof by induction and proof by contradiction, and they’re going have to do proofs with sets, and they’re going have to do proofs, know, whatever. so with real numbers, and they have to do all these things. And so that the person teaching the course say, can they do these things at the end of the course? Right. And another way to do it is say, okay, well, anybody’s teaching real analysis abstract algebra, they have to even have to prove the quarter before.

So you gotta eat your own dog food. So if these kids can’t prove, you know, statements about open and closed sets because you didn’t properly scaffold them the previous quarter with the techniques and stuff, then well, that’s how you’re gonna suffer. And then you’re have a watered down course or frustrated course, or you’re gonna have this massive curve where everybody gets like a 30 % on the midterm and final because nobody knows anything and you’re gonna pretend that.

you know, a 37 % is an A.

Justin (29:32) Yeah, you know, it kind of reminds me of, I think there’s been some times previously when you’ve talked about what happens when you’ve got two people building different parts of the same system that are meant to integrate to each other, but they’re not talking to each other and they’re not like, they’re just not synchronized and coordinated.

Jason (29:51) I’ve done this myself.

Justin (29:53) Yeah, you’ve experienced the pain that happens.

Jason (29:55) Okay,

so let me you a So my very first startup, I was like 20, you know, I started a year after I graduated high school and I did it with my good friend, Phil, who we were math majors at University of until he bailed for sculpture and film because he’s, he’s weak of soul. No, seriously. He just, he’s also, he’s like a, he’s like a math guy who was also like an actor and artist. So he kind of spanned that. So he did math for a while and then he decided he wanted to do.

stuff. so we did the startup. He had been a coder like me. He had been writing code in high school and he worked for the psychology department writing psychology tests on the Mac. So he was a paid programmer while an undergraduate. So we started this company together and we’re building ⁓

So this is a Windows app that had this big, elaborate UI, lots of flash, know, like grids of flashing numbers and things. was like a, you know, it was very cool. And this is before you could just like buy an off the shelf grid of something, you know, you had to build everything was like core graphics objects, you know, and rendering texts and placing them on the screen, you know, stuff. And I’m building the whole engine.

this elaborate financial simulation engine that simulated these equities and futures and derivatives behaviors and all sorts of really complex yield curves and things and option, these whole options, families of options and it was really complex. And we were in Vermont. We’d sit down and we’re working next to each other talking every day and he’s building the grizzle stuff. And then I was like, he had all this kind of fake numbers.

to for his testing. And we had not been building kind of like we had not established ahead of time, sort of like ⁓ a user interface. This is the this is the sort of the API that you’re going to use, Phil, to get this data. And so then we’re getting it. And this is this is back before the day we even had like a network. I had like a flop. We had like a three point edge floppy disk that we would fling across the room to back and forth, which are like a frisbee.

usually would land and get stuck, hit the wall or knock over somebody’s books or something. So we would, and so, and I remember, so we did it and we sat down together and it was like, we both in, I don’t know what we were thinking, cause we were like, no, I was like, well, you do that. And then it was this whole thing that took us, I think it took a few days of kind of like figuring out how to get.

his all his UI code to talk to my engine code because we were so green and we were 22. Like, mean, I worked for summers for an engineering from writing some stuff and he, you know, this is back in the early night. So we didn’t know what the hell we were doing. And then we learned the hard way. Like, okay, you need to be the front of the backend need to be talking to each other from the day one. You need to be like writing a system and getting everything to compile.

And then in, in, in, shaping and melding the, the front and the back ends to work together from day one in defining a user and interface. Well, so these are going to be my, my API calls. This is how you put information. It’s like information out, you know, this is what it looks like. And then you can build this sort of a fake, like a mocking thing that maybe he could just use that has some, some hard coded numbers that he’s knows what to expect. then I, whatever.

But I learned that lesson early off. mean, like we’ve been talking about stuff constantly for, and we were working on, I think for months. And that was all, it was all just mismatch. And it was, it was just a hard, like, it was a hard thing to overcome. And I that’s just the happens with, with, with a human nature. If you don’t, if you don’t get systems to integrate from day one and they, and so that kind of has built in accountability. If he’s like, well, Jason, like,

You said there was going to be these functions to get this information. You don’t have it like, yeah, my bad. Like let me write. Sorry. Wait, where’s the third parameter? I don’t know. I’m testing in. I’m like, yeah, good point. So, okay, let me fix this or whatever. Like, and I’m like, well, why aren’t you showing the the time to expiration for these things? He’s like, ⁓ you got to show that, you know, and so this is. It’s kind of like we’re talking about with like teachers is like is a little transgression. You got to suffer. That’s wrong. Fix it.

Right early, find the problems early and fix them. So I’m doing this. You’re expecting that. Let’s see if we’re on the same page. Teacher and student, two courses going together back in front end.

Justin (34:50) Yes.

Yeah, because otherwise, regardless of whether it’s a student who’s drifting off course or whether it’s

Jason (35:12) developers

writing parts of a system.

Justin (35:15) developers. Yeah, right. These these these it’s like these these ⁓ impedance mismatch these defects just to pound over and over and over until you you look up and you expect like, expect to be able to reach out and grab the other thing and he turns out you’re like 100 miles.

Jason (35:34) That’s right. That’s right. And I think in that situation, it sounds like there was like a lot of decoupling between who’s designing and teaching the methods of proof course and what the real skills that have to be mastered by the students who complete the course and what’s expected coming into the courses that really follow on. And if they’re not teaching the same course, they’re going to be to some, you know, and see, it’s kind of a pain, you know, it’s like

It was sort of interesting. was talking to a friend of mine at lunch yesterday and he was talking about how he hired this guy, this marketing guy. So he has a, my buddy has a big like a fulfillment company and it’s, have ⁓ like pick and pack warehouses all over the country. And he’s been, he, he and his wife founded this, you know, 20 years ago and you know, he’s been running for.

for many years and they have a big technology arm to this thing that does, that handles all the infrastructure, that handles all the tech stuff that they built up over years because when they started this, the stuff didn’t exist. You couldn’t buy like a license something off the shelf thing. so they hire, he was telling me this story at lunch yesterday, he hired a marketing guy and he’s like the guy had like really good.

you know, conversations. yeah, totally get we’re on the same page. And it was really. ⁓ He had a lot of confidence that this guy knew what he was talking about. What did they mark? What the new website needed to be, the branding, the all this whatever marketing strategy was going to be the whatever. And he says, wife is very much like. Well, because he was sort of really he was like, I don’t know, I’m not getting the right.

that I’m getting some signal that maybe he’s on their eyes off track. That maybe, and his wife is like, you know, Tom, just let him do his thing. You got to hire really good people, give them the context, the goals and information and let them do their thing. And so he was feeling guilty. He’s like, I don’t want to be a micromanager, right? Because micromanaging is bad. ⁓ Lo and behold, six months show up and it’s incredibly underwhelming. It’s terrible.

Terrell. He’s just like, my God, how much money did I spend on this expense? And then he’s he’s asking, OK, so wait, wait, why this? And the guy’s like, well, let me see. I’m trying to explain this to you like, know, like, hey, you’re this out of touch boomer or something. You know what mean? And and he’s like, don’t. In this, you know, the conversation, he’s it sounded so good. They sounded like he understood everything he told it with and he was like so excited.

And I say, well, know, Tom, there’s a really good article came out like a year ago. There was a talk given at startup school by Brian Chesky, who’s one of the founders of Airbnb called Founder Mode, which Paul Graham then wrote a you know, widely circulated essay by the same name Founder Mode. And Brian Chesky’s given a number of talks on or interviews on what is Founder Mode, right? And some people say, Founder Mode is burning forever. Like people who are older, like that’s of course.

But the gist of it was that a lot of well-meaning people would tell young founders who have growing scaling companies, look, the way you scale a company beyond just 10 or 5, 10, 15, 20 people is you can hire great people, empower them, give them the direction, give them the resource and get out of the way. he’s like, and then Brian is like, that almost killed Airbnb, almost utterly destroyed it. And when he gave that talk,

And then we talk about how I fixed it, which I’ll go into a second, but all these other founders learned the same lesson. They got all the same advice from all these people, hire great people, give them direction, the other way, don’t micromanage, and then you’re good. And it was a failure. And they all had to learn the same lesson that Brian learned is that you have to get into the details. You gotta roll your sleeves up and you gotta meet.

You don’t just have your direct reports. You’ve got to meet down with the people in the engineering part of the marketing department, the sales, the fulfill whatever. And you’ve got to get in there and you need to be seeing what they’re doing. You need to be talking to them. You need to be, you don’t just wait till the end product and go yes or no. Cause they just waste huge amount of money and programs sort of his takeaway was that that that advice everybody was giving really applied to that. When

when you have a skilled company, the founder’s gone and we hired some, know, MBA type CEO takes over and they hire people who are supposedly experts in their area and they do it. But he’s like that also allows people in executive are just really good at bullshit. You know, everybody’s, you know, but these companies are often so big and successful just because they have so much momentum and so much market, you know, ⁓

so much scale that they’re going to almost go to see no matter what, smaller scaling companies can just totally run off the road. And it was sort of the same thing. A Tom fell victim to the don’t be a micromanager. Right. To wait to the end. You know, and is like, no, if I say I’m I said, I’m I send you this essay, I’m sending you for interviews. You got to watch this because he was like, because he felt really bad. He’s like, he’s like having this cognitive dissonance. Like I.

My instinct is telling me to get involved, but I know micromanagement is bad and it’s a disaster. And I was like, Tom, Tom, no, no, this is Fondremont. That’s totally, that was wrong. You got to be involved. You need like every, I’d be talking to this guy at least ⁓ once a week, if not more, show me what you’re researching. What are you thinking? Give me some stuff. And so one thing Brian Chesky says in this, I was looking for some really short interviews to share with them. Cause I want a time to, to get comfortable and understand this and not have this sort of.

shame that he’s a micromanaging because he’s asking, he’s trying to get more involved in the process. And Brian Teske says that he was meeting with one of, you know, one of people who, in this sense, and then, know, the person goes, well, I was like, is this my decision or your decision? He’s like, that’s exactly the problem. It’s our decision.

Right. And we’re working together. I’m helping us. Steve Jobs did this like you would, you know, he would continually when people I remember this great book I read about the development of the. Well, it was was about the. iPhone, but it was specifically about the guy who wrote the you would the touch the touch, keep keep had because back in the day, the only thing the only way would type stuff in was like, you know, really rudimentary, like.

on a number pad, but he would like the Blackberry, right? People are like, what’s the Blackberry thing? actually look it up.

Justin (42:37) Yeah.

to Blackberry for a while. was really, I was really young, but.

Jason (42:45) Yeah,

it was huge in the 2000s and in the business world, all business people had like Blackberries. Had a little screen and you had all your little keypad below it that was actually physical keys. Well, this idea that you could type on a screen because they didn’t have like touch sensitive screens really. I they had maybe something that was like you could like you could go to a bank and like a teller, like a big thing, but not like a little screen. And that’s a few read Steve Jobs biography by Walter Isaacson. He talks all about that.

⁓ the Gorilla Glass and I mean, it was 3M who made the screens and development that because that became a technology that you could actually use, which made something like I found possible. Anyway, so this book about the development of this keypad is like, everybody would be working on their stuff. And then every few days or a week at the least, they would meet as a team. And everybody show off what they’re doing to their the lead of their group.

And then once it was good enough, they bring it in to show Steve and you get ushered in and Steve would have a couple of lieutenants there and then his boss would come in with him and he’d go through the whole thing. And this is like when it’s done, it’s like it’s far enough that we have something to show. And Steve would say, well, I see there’s two ways to do this. And the guy’s like, yeah. And he’s like, why are there? Why isn’t there one? He’s like, well, there’s two. He’s like, which one is better? He’s like, well, let’s do that. You know, what he did, the whole Steve isms of simplifying, simplifying, but.

Anyway, the point was it was like a continuous iteration. He’s like, Oh, no. And the example that Brian Chesky was give about Steve Jobs, he’s like, he’s like, I don’t think Johnny Ive would tell you Johnny Ive, who was the renowned industrial designer for Apple. So if you think of all the cool, you know, iPads, iPhones, Mac books, airs, anything you can think of, he designed. And

And he’s like, I don’t think Johnny Ive would ever have accused Steve Jobs of micromanaging. But in Walter Isaacson book, which I read that in like 2010 or 11 when it first came out, right after Steve Jobs passed away. the… One thing Steve Isaacson said is that… Walter Isaacson. Sorry. What Isaacson said about Steve Jobs is he would…

stopped by Johnny Ive’s workshop like pretty much every single day. What’s going on? Just because he loved, he loved the design process. He loved talking about with Johnny, like what he’s thinking, what are the problems he’s trying to solve? What are some models, you know, and he did, you know, sometimes he’d have, they’d have stuff to show off and sometimes they wouldn’t. As they were working, they would just talk and they would just walk. And so he was, Steve was integral to the design process, which is why

In part, the stuff was so great. mean, yeah, you had a brilliant designer in Johnny Ive, but you had a CEO who, I mean, he had an incredible aesthetic taste. mean, there was this saying, I can’t remember who, I think it was John Gruber, said something like, the design of a product will never exceed the taste of the CEO. Actually, I I told you, I said that to you like a couple of days ago. We were talking about something else.

⁓

And so that so there’s that I mean there’s a lot of things that made the successful but one thing that was successful Steve was deeply involved. And and and a very collaborative positive collaborative relationship with Johnny I. And I’m.

Justin (46:27) Sounds

like, it sounds like another one of just another instance of if you just stay on the rails with your buddy, like from the beginning, keep that like communication loop tight, just stay together in this then, then it is good. Like it’s like, if, if, if the, if the founder comes in and has been, um,

staying on top of like what’s happening and, actually is invested in the details and they are giving feedback on that. Like, this is all good. Like it’s, it becomes the failure mode that happens is when the founder has just kind of like has no, has gone out of touch for a long time. And something has a ⁓ lot of momentum has been built ⁓ into building something that is not aligned with what the founder wants to see.

Or the founder has not been keeping on top of the details, doesn’t really understand the process. And there’s a communication barrier between like, you know, like the, the maker is, is like building their thing. And then the founder comes in and then says some stuff and the maker’s like, ah, they don’t really understand like how, difficult this is or like what that’s what they ask for is impossible. just. But if you, if you let it get bad, if you let it get so out of touch, cause you’re not syncing up.

frequently, then yeah, I can spiral out of control. But if you just start from the beginning, just stay tight together, that’s the way.

Jason (48:01) Yeah, it’s like whenever you have two processes that need to be in sync and it could be just two human beings working on something, it could be two departments, could be backend front end developers, could be whatever it is. Whenever those things are supposed to be in sync, well, the more frequently they sync up with accountability, like, that actually doesn’t connect.

Johnny designed this thing, the engineering department says they’re gonna need an extra half an inch to get that thing. great, find that early. And the third day that Johnny’s planning on the model is like, well, the head of engineering said, well, you need at least nine and a half inches, not nine. you’re, you know, it’s like, okay, well, the casting process and the molding process, got it early, right? If you waste time doing the wrong thing, this is not gonna work. ⁓ And that happens at department, so, you and you almost can’t…

The error is rarely over communication. The error is usually under communication. It’s like you can almost can’t over communication. There was one thing I used say to you when you first were teaching math academy, I say communicate with parents early and often. Get a positive line of communication. Tell them your plans for the year, tell them you’re excited, tell them you’re just going to be a great year. And then first couple of things that students do well, immediately tell the parents that they’ve done a great job after a great start. I’m really excited. think, you know, she.

great year. think she has a lot of thing. And as soon as they start to go off on rails, say, you know, I’m a little concerned she hasn’t done homework the last couple of days. And then as long folks in classes really need to get things early and often. And the parents are synced up, you guys are synced up. But if you wait, and you end up a C minus at the end of the quarter, and he’s like, what happened?

Justin (49:50) Yeah, that’s right. Yeah, right. Because like, it built building some kind of like positive relationship with the parents that you can break bad news later on if it happens and have them on your side, they trust you and they’re going to try and help rectify the situation. And also, even if it’s bad news, like pretty much all the way out, like, at least they hear about it early and paper

Jason (50:13) trail.

Like I’ve been telling you, even we’ve been over the days, not doing the homework, not trying not focusing. I mean, you know, and there’s a whole and so when it goes bad, and you have to talk to the principal or whatever, and you’re like, Look, you know, she’s just not I don’t think this is this is the right class for her because she’s not doing the work and she’s not trying and not whatever. And, and that’s where they go, Yeah, let’s find the right better placement where the student can be successful.

whatever. yeah, anyway, that’s just kind of thing is over communication. I think like one of the things that I think why I think we’ve been successful at what we’ve been trying to do is that you and I talk a lot. Right? I mean, I, yeah, I mean, we

Justin (50:59) Every day, sometimes twice a day, yeah.

Jason (51:02) Least, I think on average we talk twice a day. Probably usually there’s a zoom call when we call. I’ll be working on something like, hey, can we jump on a call? Let’s talk about this. How about that? And a of times I’ll call you when I’m on my drive to lunch. And I’ll call you on walking the dog. If I didn’t call you while I was walking my dog, which is like five o’clock, I’ll call you when I head to the gym. You know, so I would say more than at least two to two and a half.

Justin (51:15) Yeah.

Jason (51:29) If I average it, would say two to two, two to two and a half times a day would be the average. But that, I think, and there’s probably some times just like, well, you know, maybe, you know, maybe you guys are wasting time. Jason, you probably shouldn’t have called him and probably let Justin just kind of cook on what he was doing and ask him for the seventh time, you know, how should we think about this thing? But.

Justin (51:54) On average, those pay off though, big time. It’s like the, repeated, whatever there’s a problem or a question that we don’t really understand or have a satisfying answer to. It’s like a spaced repetition sort of thing. It’s like, just think about it. Yeah. Question today, we kind of brainstorm a little bit, just talk about it a little bit. That’s consolidated overnight. Day later, ask the same question.

We are not there to a solution yet, but we’ve gotten further than the previous discussion. it’s like, eventually it’s like, no matter what the hard problem is, eventually it kind of cracks and we figure out some way to it. Yeah. Yeah, exactly.

Jason (52:35) We see our way.

Yeah, you know, don’t, I don’t talk to Alex as much. Alex, we once, twice a week. And Alex is in, Alex is our director curriculum and he’s in the UK. Although we talk every day on Slack. ⁓ There’s less work necessary communication, you know, because it’s more like he’s running a team, they’re developing these courses, you know.

I reserve feedback, especially when it’s a new kind of a course and there’s a more unknowns and things about how technology is coming into play or what the expectations are for certain things. it’s less, it’s less hard to get out of sync. I mean, there’s less to talk about the differential equations course he’s working on. It’s like, how’s it coming? Oh, this one does this and this, we’re covering this, cool. What about that? I mean, that’s like this, Let’s keep it up. Stuff’s like, here, I’ll buy it here. Here’s some screenshots. Here’s some.

Justin (53:30) Hey, you’re great.

Jason (53:35) Tim, this is so and so is working on this. This is looking really good. I’m wrestling with some of these topics, trying to get this module. I think we need to consolidate some stuff. I think we might need to do this, that. I’m like, okay, that sounds great. know, sometimes earlier years ago when courses are sort of methodology or sort of standards were set a certain way was less clear and I’d like, okay, listen, you know, like we’re talking multivariable calculus. We can’t have seven minute problems. You know, and conversations about how do we scope these things down and make them.

really efficient so they’re learning things they need learned but they learn they do learn start to finish and all these things but you know there’s a lot of discussions about that kind of stuff but once you figure that stuff out there’s less so that so we talk a little less frequently although we do typically do the three of us do a call every week an hour and a couple hour calls sometimes yeah and then sometimes we’ll do two a lot of slack stuff but he the thing about alex is he’s got a kid and

They’re in the UK. And so when we’re coming online and get our days going, he’s kind of trying to finish things up and dinner and homework. And, you know, it’s like ⁓ I feel a little worse bugging him. Yeah, then I don’t I don’t feel as bad. I always tell you, like, dude, like if I call you and you and and son are having dinner or you’re doing something, you’re watching it, we just like don’t answer. I’m fine. I’m like, I will just I will take that as.

Justin (54:52) Thanks

Jason (55:05) You’re busy. I’m not worried about, but I’m like, if you’re working anyway, and I’m like walking the dog and I’m thinking about something then like that’s that I think would be helpful if we could brainstorm or think about them. Let’s talk. So anyway, that’s kind of how freeways curious like we so so it’s interesting about this whole syncing up. It’s like if like we are a fully distributed company. You’re in Boston, Sandy and I are in Pasadena. Alex is in the UK. I can’t remember what cities and it’s somewhere.

hour from London or something and and then the rest of the content team is spread throughout the world. Am I forgetting anybody? Well we’ve had other people but you know we other contractors and stuff but we’re

Justin (55:50) Yeah, everyone’s spread out. There’s no hub or anything. It’s just everyone’s is distributed. Yeah.

Jason (55:57) We’ve learned to be very, I think, efficient and effective with this model. think, know, it’s funny, they talk about like, um, they talk about, you know, companies that struggle with this. And I think some of them who have, who are like, can’t either fully distributed or you’re not. Like it’s tough if you have a hub of people who are working together and then other people are kind of off on their own. Because there’s a lot of communication that happens between these people.

Justin (56:22) Yeah.

Jason (56:26) that’s not written down.

and therefore it’s not easily shareable.

But if there’s anything that you and I are talking about that I’m thinking, Alex needs to know about this, I’ll just do it in our shared channel. Or if Sandy needs to know about it, and I’ll put it in the four of us, put it in the director’s channel. Or whatever. that we kind of, and then of course we have lots of channels for different specialized topics, but it just seems to work really well. I mean, how do you feel about the…

I don’t know how things have evolved, the structure, the evolution of our distributed work.

Justin (57:08) Yeah, you know, honestly, it, um, well, you and I worked together closely in Pasadena for several years and the way we’re doing it now distributed, feels, it feels like there’s, there’s really no loss of information transmission. It feels just, just as weird to me. Yeah, I know.

Jason (57:26) It’s yeah.

It’s

weird how well it works because we had lots of like impromptu conversations in the evenings or even daytime where we just get to…

Justin (57:42) Well, OK, I think one thing in particular that works out well for us is allowing conversations to flow out of scope. So when I think of a dinosaur corporate company who tries to go remote and it doesn’t work very well, ⁓ actually I have an uncle who ⁓

He works for a big company and his team was remote for a while and it was not working so good. And he’s trying to get people back into the office. But one of the problems he cited is, that there’s just like a lack of like spontaneous idea generation and stuff like, know, like the, you just like talk to your buddy at the other desk about some thing that just crossed your mind. Then you come up with an idea, you know, ⁓ but what we

do a lot is we allow conversations to flow out of scope. We don’t put a scope on the conversation.

Jason (58:39) Does anybody

listening to the show can attest?

Justin (58:51) Yeah,

maybe. That was dope, but that should be. But it’s like, it’s, one of those things that like initially you kind of, maybe if you haven’t allowed conversations to flow out of scope before, then you kind of feel like, what are we talking about? Like, no, you gotta like, let’s bring things back in. What’s the to-do list, check everything off. Like, but it’s like, when you focus your thoughts too much,

Jason (58:55) Hahaha

Justin (59:18) You talked about this on a previous podcast where it’s like, just, you lose, once you focus down too much, you lose all the kind of background brainstorming that you were doing. It’s like the balloons kind of fly away.

Jason (59:33) Right. Oh yeah. I just, can’t hold onto multiple. Once I start focusing on one idea, I lose the other two that are kind of just in the periphery of my brain. And they just think I lose them. Like, what was I thinking about? I can’t, it’s gone.

Justin (59:44) I think that kind of happens at a meta level, like, and it’s just addition to within your own brain, but just like also just at a with within an organization, there are these kind of like spontaneous ⁓ thoughts or actions or problems or noticing things. that’s weird. I wonder why this like just things like that, that are not like, they’re not drilled down fully, but they’re just kind of happening in the background. And if, if a conversation

ends up, if it comes up in a conversation and you drill down deep into it, come away with something valuable. ⁓ But if every conversation is scoped down really hard and like every meeting has a name on it. This is the meeting to discuss that thing X, Y, Z. We’re not discussing A, B, C. We’re trying very hard to stay on task. Then you run

into a problem where you’re just exploiting what you already know how to do and you’re not really exploring at all the space. ⁓

Jason (1:00:51) I think a couple of key characteristics that make it work is, one, you gotta like the person you’re talking to. I really enjoy talking with you and Alex. It’s So I sound like, I’m what’s up? And you’re like, I’m just getting on this label. I pretend that I have some time, a lot times I have nothing to talk about.

Justin (1:01:09) likewise.

Yeah, you know, there was one time that you called me and I was right next to Sanjana and I picked up the phone and then you said like, what’s up dog? And then she was just like fell over laughing.

Jason (1:01:32) It’s like, how did we get here?

Justin (1:01:35) I know. But yeah, that’s the kind of vibe. It’s a very much friendly sort of… Not corporate, not stodgy, not…

Jason (1:01:48) Yeah. Yeah. Yes. Because if you don’t, if you’re not enjoying speaking with the other person, then you’re going to scope it down because you want to get it done and get it out of here. What do we got to talk about? Okay. So we’re to talk about the SAT math course again, or we’re going to talk about gravity, or we’re going to talk about the whatever. You know, my boss wants to go over this stuff for me. Or once we have a team, three people, we’ll go to another meeting, right? And we’re just going to grind through this stuff.

Anyway, any ideas? What are their shortcomings? What are their failure modes? mean, I don’t know. Well, get back to me if you… It’s just, it’s not fun. Nobody really wants to be there. Some people may not hate it, but most people it’s not like on their top 10 list of things they’d be like to be doing. Where I’m like excited to talk to you, I’m like, what’s up? How’s it coming? How’s the thing going? A lot of times you’re running some experiment, you’re trying something. How’s that? And being really excited about what you’re working on.

Right? Because you’re excited about the thing you’re doing and you enjoy speaking with the person you’re working on it with. Then it that sets the stage to have like fun, more meandering out of scope conversations. Right. You’re not you’re not doing everything you can to end the conversation as quickly as possible. It’s like, OK, you know, it’s like, you know, the.

I get a call from the bank and there’s something and she’s like, okay, that sounds like, okay, yeah, okay. Yep. That’s what I think. Yep. Or no. Okay. Bye. It’s like, I’m just first a nice person, but I don’t really want to talk about, you know, banking services is like, what, what do need from my end to get this thing done that I’m done? And then like, I got it. got it. So, but if you do that, and that’s why I think, you know, there’s something like talk about culture, you know,

Justin (1:03:36) Yeah.

Jason (1:03:43) What’s the culture of your company? I still don’t understand what half that means, but, I don’t know if most people do, but it’s like, the end of the day, you gotta work with people who you really like and can work with and can vibe with to use how the kids say today, you know? And if you do that and everybody’s really excited about what they’re working on, then you’re set up to succeed. But I think…

The other thing for us, and this is probably just because I’m so, ⁓ Equatious, right? And it’s like, I want to talk all the time. I think better when I’m talking things out loud. think that’s one thing I’ve mentioned to you a few times. It’s like when I, know, it’s like, we need to write to know what you think. It’s like, I kind need to talk to know what I think.

Thank you. And I is kind of like a, it’s kind of like my white mental whiteboard is just talking. And so like when you and I talk, can kind of, I can be like, okay, so, okay, so let’s, let’s, let’s, let’s, let’s think about this again. Okay. So this happens, right. And then, okay, then this, like I’m writing it down to whiteboard. Okay. Right. And then this, and then this, you’ll go, well, actually, da, da, da. Okay. Right, right, right. Okay. Right. Okay. So then scratch it up with it. So we’re kind of like, we kind of have this shared whiteboard.

That’s part of our conversation that allows me to kind of think through things more, you know, and I think that that helps. So for me personally, the conversation is is is an aid in clarifying my thinking. ⁓ But. But I think also it’s just sort of the. You know, we’ve kind of set the. The the habit of just.

the norm of having lots of conversations. So it’s like, okay, Jason’s gonna call me and as long as, you know, I can get off, you know, for dinner or whatever, then you want, I’ve got whatever. And then we do. And a of times we solve really good problems and come to lots more concrete solutions, better understanding things. it’s, there are some times where it’s not as productive, but it’s never really painful.

And if I feel like we’re not really getting anywhere, I’m like, okay, well, I think that’s it. Then I’ll just kind of like let it go. I’m like, all right, well, I’ll let you get back to it.

Justin (1:06:07) One thing about the conversations is that it.

It really, mean, one of the things that’s that, that, that we have found over and over, that’s incredibly important to getting good results out of anything, whether it’s like, ⁓ somebody who’s working on something or even an LLM that’s trying to do something for you is context. And if you, if you’re scoping down your conversations all the time, you’re only giving somebody the context that you think they need to solve the

the thing that you’re asking them to do, which is often not nearly enough context. We’ve learned this over and over, right? You ask an alum to do something or even you ask another person to do something and you think that you’ve given them plenty of background information, but once you really dig down deeply into it, maybe they come back with their first attempt and then it just dawns on you that like, my God, they have.

There’s a lot of foundational things that I forgot to say that they’re not spun up on. And it’s kind of like the glass shatters. You have that happen enough and then you just kind of like come to accept that if you are, unless you have a very tight loop with somebody and you are constantly feeding them just overall context, even if it doesn’t seem like a hundred percent ⁓ on the nose relevant to the thing they’re doing at the moment.

Like they need that. That’s the only way that they’re gonna have enough background context to pull from when needed.

Jason (1:07:49) That’s a really good point. think, and I think people really underestimate the amount of context that they have relative to what little bit that they’re sharing. Cause you’d be saying things to somebody, but you’re selecting the context that you think is relevant. And a lot of times your assumptions are wrong. They need more. And they may understand the words that are coming out of your mouth and may they have some understanding of it, but it’s still.

doesn’t have the level of concreteness or connection to other things or whatever to really work for them. And so then they’re kind of cargo-culting in a way, right? They’re kind of, you said these things, I’m doing these things, I don’t really know the why and where this came from in the history, so I don’t have this intuitive sense of all these things. So I’m just gonna go off what these words mean in English or something.

Justin (1:08:31) Yeah.

Yeah, that’s often not enough information.

Jason (1:08:48) Right. Well, I think when you were doing some experience with some LLM stuff, you kept going and have to put more and more and more and more context and really fine tune. And I mean, like this like, oh, I wrote like a one second, one or two sentence prompt. And then it turned out I went to five and 10. It’s like, went to tens of tens, 20 pages.

Justin (1:09:10) Yeah,

I broke it down into like, originally I thought this was just a, basically one paragraph worth of information that I have to give. And then it turns out after I spent a couple of days on it, just like drilling down on what are its failure modes, try to scaffold it better through the process. turns into this five step thing and each step is itself like some paragraph of information. And I have to give, like, I have to enumerate all the, the structure of each step and like.

For instance, this qualifies as blah, blah. This other thing does not. have to give some examples of that because it’s not enough to write. And it’s not, often you think of words in a slightly different way than they might be used just generally like the dictionary definition. And yeah.

Jason (1:10:01) Well, you know, it’s ⁓ my friend of mine from college, Peter Stone, who is a ⁓ machine learning professor, was at UTS. I think he’s now the ⁓ chief scientist at Sony ⁓ and really, really smart guy. And I remember we were working on a, we had this little startup we were kind of doing with another guy and back in the day, intelligent trading machines, think it was called ATM.

And he said, one thing he said to me, goes, you know, he said, when you’re building a machine learning system, and this is back in like 1999, and he goes, the machine learning algorithm has to have all the information that a human has about that.

And if you withhold anything, don’t expect it to operate at the level that a human can operate. And so we are dealing with a machine learning very quantitative, concrete thing. It’s a little easier to figure out what the human knows or doesn’t know. When you’re talking about like an LM type of situation where it’s a more sort of a more nebulous. It’s really hard to figure out.

what the human knows that may be relevant. you try and make some hard cuts, you’re like, and it turns out, maybe I’ll include this or maybe I won’t. I it turns out later that a lot of information was important that the LM didn’t know. And the same goes to talk with humans. we’ve been recently trying to hire someone for a position and we interviewed tons of people and it was just.

very clear that a lot of them lacked sufficient context in a lot of areas.

Justin (1:11:58) It’s like you can talk to them at a high level about these things. You can get faked out because they’re, I don’t know if they’re cargo culting or they’re talking from their own experience, which kind of maps to similar high level principles, but doesn’t actually come back down into our specific context. Yeah. But you run into a similar problem of.

You’re conversing in abstraction land. And then it turns out there’s a lot of information in the, in the fine details of things that you only get really by doing the reps. Yeah.

Jason (1:12:35) And sharing the concrete product

or shared work product or this thing that you’re doing. It’s like, like I can say blue. And you may be thinking of the blue of the shirt, which is kind of a dark blue purple, but I may thinking like a like a lighter blue, a light with no purple in it or something. And you’d be like, are no red in that I should say. And a much lighter blue or greenish. And you’d be like, they’re both true. Of course, this is blue. What you talking about? This is blue. But is this

Justin (1:12:54) Yeah.

Jason (1:13:04) It’s a totally different blue and has a totally different effect. And that’s just something as simple as a color, you know, and the same as with everything, the English language is so general and unspecific, which makes it, you know, we talk to each other and humans talk to each which I think a lot of times in your philosophers really, I think it was Wittgenstein said that he’s got, it was this course, one of these incredible, the most

impressive philosophers of the 1900s or something. he said, I think he basically bailed from philosophy because of the semantic mismatch. It’s like, we can never agree on what the hell we’re even talking about. I’m not a philosopher, so someone who’s a PhD in philosophy, they’re like, well, Jason, that’s a silver simplification. But nevertheless, there was a frustration with

the inability for us to agree on what these terms actually mean in a way that we can argue about them in a productive manner and come to a shared agreement understanding. And it’s just this endless talking, which is

Justin (1:14:12) make logically tight, precise arguments with fuzzy terms. ⁓

Jason (1:14:18) Yeah.

Which is why, like when you see on any comment thread anywhere on the Internet, everybody’s arguing about all this stuff. And I think a large percentage of the things they’re arguing about, if they were sitting in the same room together, they may not be arguing because they go back and forth clarifying really what they’re talking about. this is your brother. I mean, mean in this. ⁓ OK. You know, mean, and again, it’s that.

They’re not synced up. So if you can’t have this real time inter thing where I’m kind of reading, you know, you’re saying something, but I can kind of see the tone of your voice, your face, like how concrete this thing is you’re saying, or it’s just kind of off the cuff or you’re like, this is definitive statement. And I’m like, ⁓ so, you know, it’s just not an affirmation. You gotta share that information, share that context so much.

Justin (1:15:07) Yeah. And the thing is it’s like, it’s not just one conversation of a context dump. Like this is many conversations because often the context is it’s so big. It does not fit in your working memory. You have to spread it out. You have to consolidate context and like to make room for more. I mean, that’s even an LLM usage. That’s like one of the big frustrations, right? Is that like context, everyone knows context is important.

And LLMs now have, larger context windows than they used to, but it’s still not, it’s compared to the amount of context that is often required to really perform a task to the level that you need. Like the context window is still tiny, like a hundred thousand tokens, a million tokens. Like that’s still like, I don’t know, maybe you can dump a part of a book in there, but like, you really need like a library’s worth of information to complete the task.

I mean, humans do this better. Like they can actually consolidate all this into long-term memory.

Jason (1:16:09) For all on error, I did little piece from this book, a little piece from that book, and back and forth and getting, I’m filling out what are you saying and your understanding. I’m like, I feel like he’s thinking about this way, or you’re correcting my understanding of something. And it’s just this constant feedback and error correction about what we’re dealing with. So we can pull context from lots, as opposed to just saying, I’m going to try ahead of time to get everything you need to know and put it in here in a way that’s not going to flood the channel or whelm the LM with information, but try and get, but I don’t know.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.