00:00:09.409 - 00:00:48.509

All right. Welcome to everyone who is just coming in now. Um, I see folks coming in from the waiting room, so welcome. We're gonna get started in just a moment. OK, excellent. Hi, um, good afternoon to, to most folks who are joining

00:00:48.509 - 00:01:02.750

us. Welcome. Um, thank you so much for joining us for this, the latest event in our Biddle speaker series, which is co-sponsored by the Biddle Law Library here at the University of Pennsylvania Carey Law School and the Society of Academic Law Librarians. My name is Genevieve

00:01:02.750 - 00:01:16.399

Tang. I'm gonna be your host for today's session. Uh, if you've joined us before, you know that the Speaker series exists to showcase emerging and impactful scholarship on all aspects of legal information and law librarianship, and it is my great pleasure today to introduce our speaker,

00:01:16.488 - 00:01:33.029

Professor Stephanie Davidson, who will be speaking on her new paper, Meeting the Wolf at the Gate Teaching Lawing Skills in the Age of AI. Stephanie has been a law library leader and teaching in law and librarianship programs for many years. She's served as the library director

00:01:33.029 - 00:01:53.389

and associate professor at Willamette since 2022. Her scholarship focuses on many different facets of librarianship, including user needs in an ever-changing information environment, support for faculty scholarship, and law library reference as an access to justice service. This session, uh, I feel confident in saying is one

00:01:53.389 - 00:02:06.790

of our most eagerly anticipated to date based solely on our advanced registration numbers. And I think, uh, the incredible title is an important part of that. I love the title of this paper. Um, if you've read the abstract that we shared in the program description, you

00:02:06.790 - 00:02:25.970

know that this paper promises to offer something highly sought after, very important. Um, in our contemporary conversation around generative AI in law teaching, uh, which is less focused on hype and more focused on practicality and integration with other lawyering skills. Uh, if you're anything like me,

00:02:26.089 - 00:02:42.210

uh, I'm very interested in entry points to discussing these tools in a way that's really key to experiential teaching and critical engagement. Um, so Stephanie, I think I can say we're all very eager to learn from you today. Um, on a practical note, uh, throughout today's

00:02:42.210 - 00:02:55.389

talk, please feel free to add questions at any time in our Q&A, um, link below. I'm going to be monitoring the Q&A and the chat so that Stephanie can focus on her remarks, um, and then we're going to save time at the end for your questions,

00:02:55.410 - 00:03:10.479

um, and I will read those out into the records so that our recording reflects, um, everything we're discussing here today, and a recording of the session will be available on the Pen Carry Law scholarship repository after the session. So thank you all so much for joining

00:03:10.479 - 00:03:28.160

us. We're very happy you're here. And with that, I am going to turn it over to Stephanie. Great, thank you. Hope you can hear me. Uh, thumbs up. Awesome. Uh, so thank you very much, uh, to, uh, to everybody, Penn, uh, everyone for having me talk

00:03:28.160 - 00:03:43.809

about this very early research. Uh, this is very much a work in progress. I am deep in the weeds of the thinking and the writing, so, uh, I'm excited to kind of test out some of these. Ideas with you all, and I think because most of

00:03:43.809 - 00:04:04.050

you, I suspect, are also working at these intersections of research and writing instruction and pedagogy, rethinking pedagogy and student support, um, I, I, I think your, your comments and questions at the end, I'm really looking forward to. So the paper grew out of reflections on teaching

00:04:04.050 - 00:04:28.850

first year lawyering over the course of the past several years. So, after about two decades of teaching legal research, um, I moved into a lawyering course here that teaches, uh, research, analysis, writing, professional identity formation, etc. And I was observing and quietly. Um, inarticulately uncomfortable about

00:04:28.850 - 00:04:47.329

some gaps in the learning process for some students. Coincidentally, at the start of, of the GI AI boom, um, but the gaps in the learning process were what really kind of prompted the, uh, the initial project and then the AI, uh, as in some ways as

00:04:47.329 - 00:05:06.390

both, uh, restrictor and, um, maybe enhancer. So, the talk and the underlying paper are about AI but not primarily. Um, it's really about, it's not about tools or even whether AI is a good thing or a bad thing, uh, for legal education or, or teaching LRW.

00:05:07.250 - 00:05:29.429

It's about the learning and professional formation that happens, uh, largely in our lawyering classes, um, and the students who are most at risk in that, in that process, in an environment where AI is pervasive and new and very disruptive. So, law student success is something that

00:05:29.429 - 00:05:49.269

matters immensely to our institutions. Uh, law schools operate under a strong presumption that the students that we admit are capable of success. Um, we have this ABA standard 501B that requires we admit only applicants who appear capable of completing our program and being admitted to the

00:05:49.269 - 00:06:12.320

bar. So, we presume. We have to presume capability on day one of orientation. And yet, as we know, academic attrition persists, licensure failures persist. And if students are capable, then I want to try to better understand some of the gaps between potential and performance. Um, and

00:06:12.320 - 00:06:35.475

from my kind of current recent seat, uh, as a lawyering professor, I'm particularly ly interested in the levers that we have across the institution, but especially among lawyering, LRW academic success, faculty, and legal research faculty in the library to, to help narrow that gap. Because if

00:06:35.475 - 00:06:57.369

we admit capable students, then acknowledging struggle means going beyond, I think, attributing that to aptitude or attitude or the work that they're doing. Um, and I think we have a responsibility to future lawyers to continually monitor the learning environment, as the ABA puts it, um, but

00:06:57.369 - 00:07:19.049

also our students' learning experience, especially in the context of generative AI and the potential, as we know, to short circuit learning. So, when I started teaching lawyering, I was observing a variety of struggling students, certainly all of them unique to some degree, and not necessarily any

00:07:19.049 - 00:07:36.089

different than when I was teaching legal research, but where my research courses were generally 1 to 2 credits, lawyering is 4 in the fall and 2 in the spring. And so, I've had a lot more contact with the same small group to observe them over the

00:07:36.089 - 00:07:58.730

course of the year. And so I'll give you three short composites of students to illustrate some of what I observed of, of struggle. So, first, we've got Jenna. Jenna completes most assignments on time, but, uh, they don't follow instructions carefully. They don't respond to feedback, even

00:07:58.730 - 00:08:21.410

requests for uh check-ins or seek clarification of feedback or otherwise follow up for help. They attend class regularly, but participation is fairly low. Um, their final assignment replicates, uh, problems with both not following instructions and with the same substance of the work that had been noted

00:08:21.589 - 00:08:41.369

in earlier work. And at the end of the first semester course, they receive a failing grade, and they're confused, they reach out for the first time and say they're confused because they turned in everything that was assigned. Um, so that's composite one. Then we've got Alex,

00:08:41.450 - 00:09:00.859

and all these are composites of multiple students that I've had in the last few years. So, Alex is engaged in class, shows some good insight with, uh, with analysis, but tends to do the minimum on assignments that require careful work that he perceives as tedious. Um,

00:09:00.890 - 00:09:20.849

he complains that the difficulty of, of of analysis in particular seems unnecessary, and that there must be a faster and better way to do this, that this can't be what lawyers really do. And then finally, uh, Carsten. Carsten was a paralegal before coming to law school,

00:09:21.099 - 00:09:44.869

which she mentions in class regularly in the first semester. Um, her observations in class sometimes demonstrate a good comprehension. Of the concepts, um, but are often non-responsive to the prompt, leaning instead on previous experience, overconfidence about her, uh, her knowledge, uh, and kind of missing the

00:09:44.869 - 00:10:07.419

question that's, that's being asked. So, none of these students are Present as being incapable of success, but each has a variety of expectations that are kind of misaligned with expectations of the 1L environment. And while some will take feedback and adapt, you know, earlier or later

00:10:07.640 - 00:10:25.539

in the course, whether it's the grade on the assignment or the grade on the course, um, others will really dig in or fail to understand where they're, where they're going wrong. And I'm interested in, as a lawyering professor, and how to help my students develop the

00:10:25.539 - 00:10:43.900

skills that they need for summer work, but also that they need for the bar, um, because my instinct is that many of the things that cause them problems in the first year persist, um, and, and end up showing up in their bar study as well. So

00:10:43.900 - 00:11:04.390

regardless of entering metrics, it seems some students are still failing in their execution rather than capacity, if we assume capacity. I don't think that it's as simple as not doing the work, um. And, and again, what they learn in the first year and are able to

00:11:04.570 - 00:11:29.500

uh succeed at in the first year is preparing them uh for academic success as well as future practice success. So if we leave our trusty LSAT and undergrad GPA metrics aside, I would have said, again, just instinctively that all three of these students need to understand

00:11:29.500 - 00:11:55.630

something fundamental about law school that they're not understanding or not accepting. Not conceptually, but practically, that there are like fundamental practical mismatches between the expectations of law school and the expectations of these students. And the academic success literature has written about many aspects of these things,

00:11:55.849 - 00:12:18.869

um, and about the ways that students can fall behind academically and why. And my, my kind of observation of that literature, um, and the way that that struggling students are often discussed, um, was, it was interesting. There's a subtle, uh, difference in the way that I

00:12:18.869 - 00:12:37.460

and maybe colleagues will talk about, uh, how to help these students and the way that the, that the ASP literature kind of conceives of them. And, and I, it, I think the shift is important. It's, it's instead of, of talking about their deficits, talking about at

00:12:37.460 - 00:13:02.849

least deficits kind of on a general sense, talking about gaps or misalignments between, uh, between expectations. I think we can't put blame entirely on students, especially when Um, I think most of us know there are a lot of problematic, uh, traditional long-term, um, things, uh, you

00:13:02.849 - 00:13:21.080

know, inherent in law school education. So, so what I kept coming back to was this idea of, of gaps or mismatches. My last paper talked a lot about gaps, and I wasn't liking that metaphor, and I came back to friction, um, for, for. lack of a

00:13:21.080 - 00:13:42.479

better word at this point. Um, and metaphorically, the idea of a force that impedes movement or slows it down, makes it harder to start once you've stopped. Um, and I'm just gonna focus on 4 here that I've roughly divided into two, what I'm calling external or

00:13:42.479 - 00:14:06.960

social frictions, and that's the hidden curriculum of law school, and, uh, threats to belonging. And then two internal or cognitive frictions, and that's uh self-regulated learning and discomfort intolerance. Um, because if students are capable and their early performance falters in ways that can be at least

00:14:06.960 - 00:14:27.780

partially explained by these forces, then I wanna understand the forces better. Because I think again, instinctively we know that generative AI intersects with all of these, all of the things that um that keep students from, from succeeding either uh they can interact positively or they can

00:14:27.840 - 00:14:46.590

uh bypass learning. So that's been my, my interest. Not the only forces working against law students' success, certainly, but I think it gives us plenty to work with, uh, in terms of understanding and problem solving our, uh, institutional responses. So, just a little bit of unpacking

00:14:46.590 - 00:15:11.210

of, of these four. So these first two strike me as related to law school culture and socialization that students experience, uh, so externally or socially, uh, in, in the law school environment. So the hidden curriculum of law school refers to all of the norms and expectations

00:15:11.210 - 00:15:30.159

that are embedded in our institutions, but not written into syllabi or our learning objectives. It is worth noting, and I want to spend time on, on this in the paper, that the existence of the hidden curriculum, its nature is contested, uh, the values and norms that

00:15:30.159 - 00:15:50.989

we are implicitly passing along to students, preference, tradition, and hierarchy, um, They are essentially white upper-class values, um, all the while painting this discourse with a gloss of neutrality, which we also expect students to produce, um, and I think that's going to require, uh, quite a

00:15:50.989 - 00:16:14.059

bit of, of, of attention. So the hidden curriculum is, is frictiony, problematic in its hiddenness, but also in its, in its bias. The socialization of students into law as a profession is not neutral, um, and it's not entirely explicit. We say and, and believe that law

00:16:14.059 - 00:16:36.380

school teaches students new ways of producing knowledge, teaches specific practices of intellectual humility and careful analytical work at the same time, um, new kinds of engagement with colleagues and professors and attorneys, um, but with little formal instruction about how to do all that. Some students come

00:16:36.380 - 00:16:56.380

to us primed with knowledge of a lot of these, uh, the norms of law school and law practice, but anybody without that knowledge is automatically playing at a higher level of difficulty, and failure to decode those norms or delay, uh, can certainly lead to academic or

00:16:56.380 - 00:17:20.390

professional missteps or even, even failures. So then the second one is belonging or belongingness, and this is about the feeling of being connected to and valued by your community individually and authentically. It's a subjective determination that an individual has, um, based on their perception, but certainly

00:17:20.390 - 00:17:41.989

is, is affected and altered by, uh, by objective influences. So, in law school, I think belongingness is relevant both in and outside of the classroom, um, can certainly be affected positively and negatively by, uh, by things like the class profile, the diversity of the faculty and

00:17:41.989 - 00:18:06.280

student and administration, uh, population, by the extent to which they see themselves reflected in, uh, leadership within the student population. Uh, the, the extent to which they see that, uh, authenticity and support for, uh, for that authenticity. And a student's feeling of belonging affects behavior and

00:18:06.280 - 00:18:27.619

academic performance, um, again, both positively and negatively. So it's, uh, it's of concern for students who, who are at risk of, of struggling. When it's, when belongings threatened, you know, there are pieces about threats to belonging that um that are, are, I, I will have to

00:18:27.619 - 00:18:47.979

take more time to dig into, but this idea that when belonging is threatened, it's not just a student's mood that suffers. This is something much um. Sort of much more expansive. It's their overall well-being, uh, their mental health, their class attendance, uh, engagement in classes, engagement

00:18:47.979 - 00:19:09.859

in all of law school can suffer and ultimately academic performance. And if we're only focused on the academics, uh, but these bigger, sort of bigger, uh, frictions, bigger missing pieces are there, uh, we're, we're missing something. It's worth noting that belonging also affects a student's expectations

00:19:09.859 - 00:19:27.979

for their own success, that, uh, students with a greater sense of belonging report having greater confidence in their ability to succeed. They will hold themselves to higher, uh, kind of higher bars. Um, and on the converse, when students lack a sense of belonging, they are less

00:19:27.979 - 00:19:52.425

likely to seek out help. They're more likely to interpret their poor performance as a confirmation of their lack of ability or at risk of opting for private AI type assistance over what they might perceive as more public and risky faculty and staff supports, which adds more

00:19:52.425 - 00:20:14.219

risk instead of instead of help. One particular risk to belonging is stereotype threat, um, when members of a group fear being reduced to the stereotype of their group. And this can arise in classroom discussion, in peer to peer conflict, uh, even in interactions with faculty and

00:20:14.219 - 00:20:38.540

staff, um, and students can internalize this as lowered expectation again for their own, uh, their own success. When that threat of stereotyping is there, is kind of constantly present. Um, it affects students' attention and focus, um, pulling it from, from class content. So diverting cognitive resources

00:20:38.540 - 00:21:03.089

from the program, um, and, and potentially altering their, uh, interpretation even of, of professor feedback. It's not only underrepresented students who experience threats to belonging though, um, sort of a, a troubling statistic that, uh, the, the nationwide lessee survey survey, the student engagement survey found that

00:21:03.469 - 00:21:25.849

across all demographic groups that were Uh, that were answered in, in response. Fewer than 1/3 of any group reported strong agreement that they felt they belonged, uh, in the law school environment. And this was a, a survey of, of more than 10,000 students. So, the concern

00:21:25.849 - 00:21:49.170

is across, kind of across, uh, lines. So I think these two social sort of frictions maybe illustrate how students experience and interpret challenges in social contexts in and outside of the classroom that affect whether they engage, seek help, or ultimately bypass learning through the use of

00:21:49.170 - 00:22:14.655

AI or other means. And then these next two, I'm calling internal or cognitive frictions. The first one, self-regulated learning, refers to, uh, having control over one's learning process. Self-regulated learners learn from their successes and their mistakes. They plan ahead, they set goals. They reflect after receiving,

00:22:14.935 - 00:22:37.229

especially after receiving critical feedback, um, and they can, they can attribute their successes and mistakes to factors that are correctable. Um, they don't attribute them to a personal failure, but rather, uh, um, performance or a technique or skill that they're able to make improvements to. Our

00:22:37.229 - 00:22:56.300

first year students generally, I think, you know, with for the most part across across institutions tend to be high achievers who excelled even if they haven't mastered these learning skills, and students who come to law school not fully understanding how they learn. And they don't know

00:22:56.300 - 00:23:19.685

that they don't know that, um, and are unaccustomed to deep reflection on their learning process, are less well equipped to adapt to the law school environment as a result. That past. Success then can lead to some overconfidence in their skills, confusing familiarity with mastery, um, mistaking

00:23:19.685 - 00:23:43.349

comprehension of cases for fluency, uh, in legal argument and discourse, for example. The skill that most supports self-regulated learning, uh, is metacognitive skills. Um, this is the awareness and reflection on one's own thinking, and law students don't necessarily tend to have strong metacognitive skills at the

00:23:43.349 - 00:24:00.229

start of the program. We see a lot of students who expect their first year experience to be more dependent on memorization and recall of legal principles as it may have been if they had an undergrad course in law, but not prepared for the difficulty of legal

00:24:00.229 - 00:24:18.800

reasoning and writing. So building their metacognitive skills, uh, will enable them to push through that difficult uh period where they realize how much they don't understand. Uh, and then to reflect on their learning so that they can improve, uh, improve in the future. The second one,

00:24:19.060 - 00:24:43.050

in this category is discomfort tolerance or frustration tolerance, somewhat similar to what we might call grit, um, and has a connection also to mindsets. This is a description of how students respond to those difficult moments in learning when they're uncomfortably challenged by new concepts, by new

00:24:43.050 - 00:25:03.410

techniques that don't fit into their existing mental frameworks. We expect students to push through a lot of difficult moments in law school. It's a necessary part of learning the law. Some students have a lower tolerance for that kind of discomfort, um, and they internalize that discomfort

00:25:03.410 - 00:25:26.630

even as a threat to their own ability or, or perception of themselves. Some differences in, um, in how students respond to that frustration, I think can be explained by mindset theory. Students with a fixed mindset will tend to attribute frustration to a lack of ability, uh,

00:25:26.709 - 00:25:45.890

which leads to more negative feelings, withdrawal, potentially bypass of the learning process through cheating or, or giving up. Students with a growth mindset tend to accept frustration as being part of the learning process, um, and can continue working through it without the negative effects on their,

00:25:46.010 - 00:26:07.469

on their perception of themselves. If frustration and discomfort with learning the law leads to a negative perception about one's ability, the risks of students turning to AI or other tools to bypass learning, I think, is really high. Mindset can be shifted, uh, and fostered towards growth

00:26:07.469 - 00:26:29.400

with some effort, but a lot of institutional cues in law school are pointing towards a fixed mindset that we have to counteract. The first year class profiles that we tout and share um are compiled based on LSAT and undergrad GPA metrics, um, the profile then standing

00:26:29.400 - 00:26:50.010

for, uh, being kind of an accepted measure of the class's ability. And then those first year classes are curved against the standard mean based primarily on a single final exam. So, students are ranked and measured and admitted in ways that reinforce their fixed mindsets about themselves.

00:26:51.209 - 00:27:15.089

Legal education is difficult, we all know, um, especially in the first year when they're so new to what legal discourse is. Learning legal analysis requires a sustained effort, heavy, fast-paced workload with a lot of ambiguity and periodic public critique via cold calls in class, um, and

00:27:15.089 - 00:27:34.569

most of that experience is new and difficult, um, and unpleasant for a lot of one else. But the difficulty, the reason I, I include so much of this is that the difficulty that students are experiencing in that first year is in large part desirable difficulty that

00:27:34.569 - 00:27:57.479

strengthens their thinking and builds their skills and frameworks for long term use. So, building tolerance for that discomfort is essential, um. And But when students don't have the meta-cognitive skills to improve their learning, to reflect, and when they also have a low tolerance for those frustration

00:27:57.479 - 00:28:18.910

points, um, they're more likely to exit, more likely to use AI, um, at the exact moment when the learning process, uh, requires their, their persistence. I don't think that the greatest risk of generative AI in the one bowel curriculum is cheating. It's a risk, but I

00:28:18.910 - 00:28:40.599

don't think that it's the greatest risk. I think the greatest risk is false mastery, bypassing this desirable difficulty of moving through, uh, that early learning process, and then cumulative effects of reliance on AI. Looking or sounding like a lawyer is a lot easier than doing the

00:28:40.599 - 00:29:10.140

work, and AI really exacerbates that. So in lawyering, um, competence, even just, uh, objective legal analysis in that first, uh, big assignment requires a really repetitive, effortful, iterative construction, um, of, of, of material, um, and feedback that they can take to reshape their thinking. Um. Generative

00:29:10.140 - 00:29:32.709

AI can simulate the product of that process without requiring the structure building that produces it. So they may be able to produce a passable product without any understanding of, of how it was created. Um, and I, for a time was less concerned about that moment of

00:29:32.709 - 00:29:53.040

producing the product. Um, the, the change to my thinking has been in the idea of transfer, that their ability to turn in a product that's passable does not mean that they would be able to replicate that without turning to the same, uh, bypass tool that they

00:29:53.040 - 00:30:11.800

used before. So, if students draft with AI assistance early in their law school career, they can produce writing that may be good enough to escape notice. Uh, risk is probably higher the bigger the class is. They may receive passing grades, believe themselves to be competent in

00:30:11.800 - 00:30:32.280

the course objectives. Others will certainly believe our students to be competent in those objectives. But they will likely not have internalized how to do the things that we mean to assess, um, nor can they transfer those skills across contexts, across genres. Um, and most importantly, I

00:30:32.280 - 00:30:50.449

think they won't have learned how to detect and fix the faulty reasoning and analysis that can pop, pop up in, uh, in AI. So without going through the process of learning to create that first legal memo, painful as it is, they won't be able to assess

00:30:50.670 - 00:31:11.770

the quality of the memo, uh, that's presented by a protege or co-counsel or, or another service. So the risk I think is generative AI makes it possible, even somewhat easy to fake competence. All of the frictions, um, on the, on the previous slides, I think, make

00:31:11.770 - 00:31:34.510

struggling students particularly vulnerable to false mastery and bypass by using generative AI. The tools offer a path of very little resistance if students want to exit the struggle. Turning to AI early in their learning process actually increases their reliance on it, which is uh intuitive, but

00:31:34.510 - 00:31:52.750

that can be especially harmful if it's, if it's used to bypass, uh, bypass learning. We also know that increased reliance on AI actually lowers a student's ability to recognize flaws in the output. They're less able to do it and they're less willing to spend the time

00:31:53.239 - 00:32:13.410

to do it, to critique that output. Which is especially dangerous in the legal context, and I'm not just talking about hallucinations. We're talking about the core reasoning that that that is going on in the, in the documents that you might ask AI to produce. So, any

00:32:13.410 - 00:32:31.089

user of generative AI is at risk of missing errors in the output, certainly, but expert learners, so not just practicing attorneys, but expert learners in law school and those who've mastered legal reasoning are at lower risk because they know how to replicate their analysis, uh, and

00:32:31.089 - 00:32:49.400

compare it to uh what the AI is, is generating. But law students, especially in the first year, especially when struggling, are at the greatest risk of missing those errors because the AI's output is going beyond what the student has mastered. They're not qualified to assess the,

00:32:49.500 - 00:33:10.650

the output. So I think the frictions experienced by struggling students make any off-ramp look desirable. Um, AI is a particularly attractive off-ramp, um, and one that just compounds the, the bypass risks. Over time, it's a snowball effect. The student learns less, thinks they've learned more, turns

00:33:10.650 - 00:33:36.050

to AI to fill out, fill in the gaps, and then can't assess its accuracy. So, pedagogy and assessment, I think, are the answer, and that means we have to design assignments that require retrieval and construction before AI engagement and explicit instruction on why that is. Part

00:33:36.050 - 00:33:57.040

of building their meta-cognitive skills is modeling that process of, of, of our, in our teaching. I think we have to structure reflection. Students aren't always good at reflection. If they can't self-critique and haven't been through that process, um, we have to lead them through that, through

00:33:57.040 - 00:34:20.020

it better with, uh, sort of better guided questions, um, and really help them kind of expose. gaps between what they understand and what the output says. So, anything where the student is engaging with the output as a critical consumer of it, whether that's legal reasoning, or,

00:34:20.040 - 00:34:41.360

um, or other, other types of, of, of writing or, or knowledge. We need to separate the assessment of process from the assessment of product, we do a lot of product assessment, um, Assessing process is difficult, it's really time consuming, um, but it, it, I think it's

00:34:41.360 - 00:35:06.189

necessary so that passable product, uh, doesn't stand in for, uh, for facility with, with process. We need to make iteration visible, iteration of research, of reasoning, of selection of best cases, counterarguments, um, writing, all of that visible to graders and to students. Um, and I think

00:35:06.189 - 00:35:26.159

we need to explicitly teach students about fluency illusions. Um, this thing that we're doing here is, is at this skill level, and now we're gonna move up to this skill level, and that's something different. Um, but I think we have to be really explicit about, um,

00:35:26.725 - 00:35:50.320

About what it is that we're doing rather than expecting that their, their work in other doctrinal classes is, is shoring up their ability to replicate the reasoning from the cases, students are not able to do that solely from, from reading and discussion. And then just some

00:35:50.320 - 00:36:11.449

kind of particular ideas about desirable difficulty um and generative learning from from learning science. Students actually do learn more when they attempt to solve problems on their own before being shown exactly how. As long as the problem is solvable, um, even with feedback, so it doesn't

00:36:11.449 - 00:36:30.310

have to be solvable on their own, uh, but if they can solve it with help, um, the difficulty that they experience, uh, is productive. It actually helps retain, uh, long-term memory. But if they prompt first and think later or expect to think later, uh, inverting that

00:36:30.310 - 00:36:51.969

sequence, uh, bypasses learning. So they have to try to create a complete problems on their own before AI exposure or any other uh short, short circuiting. We also need to teach meta-cognitive skills. This is not a new, uh, suggestion, but we need them from 10 all

00:36:51.969 - 00:37:13.830

the way through the practice of law. First year students are novice learners, uh, in the law, and many are novice learners, uh. Period. They're not able to reliably reflect and report on their skills. They need feedback and guidance throughout that reflection. Um, they can reflect on

00:37:13.969 - 00:37:35.030

perception and experience, but they're less, uh, able to reflect on, on skill and performance. So teaching meta-cognitive skills will help them push through uh difficulty and believe that they can, they can succeed, that there's something on the other side of that difficulty. Some tools for building

00:37:35.030 - 00:37:56.044

metacognitive skills include use of rubrics. There's a lot of work on sort of private rubrics versus public rubrics. The extent to which a rubric is sort of holistic versus, um, you know, chunked in a way that Students may look to just check the boxes. Um, so,

00:37:56.135 - 00:38:18.270

uh, uh, lots of nuance in how those are, are used. Uh, checklists at, at multiple stages, not just checklists of content, but checklists of thinking, um, and having to respond to prompts. And then modeling meta-cognition through our teaching, demonstrating the way that we plan and reflect

00:38:18.270 - 00:38:37.219

and make changes to course coverage, uh, helps support students and make them feel like they're not the only ones who, uh, we expect to, to reflect. We also need to uh hold on to structure building for retention of skills. Um, again, it's the process, not the

00:38:37.219 - 00:38:58.520

product. Um, this is really especially critical in the first semester. Some students are able to create structure out of the material more easily. They can filter out irrelevant information, extract the most important ideas. Others cannot naturally, but they can be supported. Uh, with the use of

00:38:58.520 - 00:39:21.000

reflection and more feedback, um, during and after that task. So, early work on structure building, um, everything from, uh, case briefs through, um, thinking about really beginning analysis that's going to feel, uh, too simplistic to them, uh, but actually isn't in what we're asking them to

00:39:21.000 - 00:39:41.570

do. There's also a difference in um rule learners and example learners. Uh, law in law school, example, learners may have a harder time applying the rule again, so they take a case and they try to mimic the same, uh, example rather than applying the rule. Um,

00:39:41.610 - 00:40:02.459

we see this throughout the first semester with, with, uh, with first year students, not able to separate the rule from the, from the example. AI tools interact directly with um with all of these mechanisms that that um are are connected to and necessary for long term

00:40:02.459 - 00:40:23.159

retention, either to support or to thwart it. Um, and I think these have to be embedded throughout the curriculum, much like the call for legal writing to be embedded throughout the curriculum. The more classes that include practices that build metacognitive skills, structure building skills, the better,

00:40:23.439 - 00:40:41.929

um, in part just because retention is better when the practice is varied and spaced. Um, so bringing the skills into multiple classes helps to reinforce that. Um, there's an interesting recent piece from the UK that I've been Um, spending more time with on the use of

00:40:41.929 - 00:41:00.959

AI as a sort of reflection point, kind of a dialectical, um, examination. It's called Gen AI as Whetstone. I have a list of references on the last slide. Um, but the idea that I'm finding quite a bit of promise in is this, um, You know, the

00:41:00.969 - 00:41:22.000

thesis antithesis of uncritical acceptance of AI and reactionary resistance of AI and that the synthesis is um both efficiency and depth. Lovely idea. They certainly acknowledge the, the kind of things on both, on both sides of that, but frame all of that as a pedagogical resource.

00:41:22.750 - 00:41:46.060

That AI, they position generative AI as a kind of mechanical provocateur, um, that, that is the thing that students then are, um, are engaging with, that in the process of, um, engaging with the AI output, not just critiquing it kind of separately, um, but that that

00:41:46.070 - 00:42:10.659

becomes sort of mini Socratic. I don't think that they use that framing, but Um, in the similar way as professors student dialogue builds knowledge, um, this idea that pushing back against the authority of anything outside, um, but especially AI can, can build their Awareness, um, kind

00:42:10.659 - 00:42:27.669

of challenge assumptions that they have about, uh, about authority and how knowledge is produced, uh, produced in the law. Um, I, I, I, I don't know what this looks like exactly in practical terms, um, but I like the focus on the very difficulty of the work,

00:42:27.939 - 00:42:47.860

bringing it and the AI tools kind of front and center in the classroom, um, sort of welcoming the productive friction. Uh, welcoming the wolf, if, if we continue the metaphor, um, because we have to preserve the productive struggle and eliminate wasted struggle, um, and that's an

00:42:47.860 - 00:43:08.870

instructional design challenge. Um, and again, the central question isn't should they use AI or even when should they use AI or even what type of AI. It's does this particular use of AI in this context preserve or bypass the cognitive work that builds competence in this

00:43:09.250 - 00:43:32.260

skill? And then with respect to struggling students, if we believe our students are capable, then we have an obligation to design instruction so that use of AI strengthens rather than replaces that cognitive development. So I wanna just end by kind of asking you all about whether

00:43:32.260 - 00:43:53.540

the frictions resonate with you in terms of your teaching, um, about which ones maybe feel most salient at your institutions, um, among your students. About where the use of AI seems to intersect with certain kinds of anxiety or fears of failure, um, both timeline wise and

00:43:53.540 - 00:44:14.429

maybe also, uh, differences in kind of anxiety. And then what struggles can we, can I more specifically define the struggles that we need to protect and preserve versus the ones that we ought to redesign. Um, and then how can the individuals in the institutions, librarians, um,

00:44:14.719 - 00:44:36.840

in terms of information literacy and research, LRW, broader faculty as experts in building legal reasoning and argument, and academic success faculty, if you have them as experts in, uh, sort of all things learning. How can all of us partner in Preserving cognitive rigor in this new

00:44:36.840 - 00:44:55.600

environment where AI is not going away. Um, you know, again, the wolf at the gate is not going to be turned away. Uh, the question is, is, I think, whether we can understand what parts of learning have to remain effortful and undisturbed, and why, uh, and

00:44:55.600 - 00:45:17.830

where. Where we go from there, and I think that's, yeah, that's all. And so these will be posted, um, and I'm happy to share slides separately if, if anybody would like, um, but some articles here and paper is forthcoming. Um, thank you so much. I feel

00:45:17.830 - 00:45:33.570

like if we were in a meeting, we would all be able to send up the emojis that reflect, um, uh, um, delight. Uh, thank you so much. Um, we've got activity in the Q&A, um, which I will read out. Um, uh, I have my own questions,

00:45:33.709 - 00:45:47.280

which I'm hoping I'll get a chance to get to, but I promised, uh, our audience that they could have first dibs. So, um, I'm gonna start, uh, if I may, with our first question. Um, In talking about belonging, you mentioned that students may turn to private

00:45:47.280 - 00:46:01.719

AI use instead of seeking faculty or staff help. Have you experienced this with students, and how do you think this use of AI affects students both personally and in terms of their law school performance? I ask this in part because I've seen AI exhibiting biases and

00:46:01.719 - 00:46:18.620

stereotypes in ways that I have found disturbing. Yeah, I, I have not had experience of being aware of that happening, but I've, I've certainly heard from other people who've had that happen. I have had the kind of private remedy experience of the student who you reach

00:46:18.620 - 00:46:36.145

out to. You say, this is not satisfactory. You're not getting this. Come see me. They don't see. You instead turn to AI, turn in hallucinated cases, and then you push, you know, we engage usually in a dialogue of, tell me what your process was like, and,

00:46:36.264 - 00:46:51.969

and then it becomes a, a wild tale of, of how they found things. Um, and I think there is a saving face kind of component of, I'm gonna do this thing where nobody can see me and nobody can see my vulnerability. Um, and I think there's

00:46:51.969 - 00:47:11.939

definitely a big piece of normalizing the help and normalizing that everybody's struggling, even though law students do tend to put on a pretty sleek shell. Um, yeah, I, I think the, the potential for bias is off the charts, um, and, and more probably needs to be

00:47:12.080 - 00:47:29.580

written about that in terms of that dialogue, what that dialogue looks like. Thank you. Um, our next question, uh, I, I, I'm gonna sense that this is actually nestled inside a comment, but there is, uh, a question here. Um, I understand that mindsets and law student

00:47:29.580 - 00:47:43.459

challenges may create reliance on AI, but don't we have problems that are more fundamental? We've experienced a generation that thinks that Googling is a sole research skill, and soon we'll have students who are reliant primarily on AI in secondary and undergraduate education, if they don't have

00:47:43.459 - 00:47:56.889

so already. They will use AI because that is all they have done in the past. Your, your thoughts? Yeah, I think they, I think they're the same challenges though. I think any students, I mean, I am not the, the learning science expert. I am, I am

00:47:56.889 - 00:48:17.340

really, you know, new to engaging with all of this, but the, the pedagogy research has all kinds of things that predate the existence of AI and, and even in the law research and, and, uh, law library research. Every time there's a new technology, there's the promise,

00:48:17.379 - 00:48:37.469

and then there's the, the letdown, there's the, you know, promises democratization, but turns out to be, you know, wildly different in how it's implemented and reinforces further gaps. Um, I think to call it a Google generation problem, or call it a COVID generation problem, or purely

00:48:37.469 - 00:48:56.719

call it a, you know, they, they think this is research when it's not, it doesn't help me understand how to improve it, um, and I think Cause I get that. I absolutely get that, and I've made many of those comments myself, um. But I think we

00:48:56.719 - 00:49:10.520

have to bring them in, which is why I think it's the idea of the wolf just kept coming to me as, as the, the wild thing, the bad thing, the not bad thing, the thing that we have to engage with. Um, I think we have to

00:49:10.520 - 00:49:27.510

show them why Google is, is. Can produce bad results and thin results, and the consequences of thinness and lack of rigor, rather than expecting them to just believe us, cause I don't actually want my students to just believe what I tell them as the authority either.

00:49:28.050 - 00:49:42.610

Um, and we say that we don't want students to come in as passive learners. So I think if we say that, then I think we have to show rather than just tell. Oh, thank you. Um, I have to admit during the the as you were concluding,

00:49:42.879 - 00:49:57.750

um, and speaking about the wolf at the gate, um, I had a, uh, another metaphor came to mind, which is, um, the vampire that can't come in unless you permit it to. Um, and I'm gonna sneak in my own kind of question here, which is, um,

00:49:58.120 - 00:50:20.620

thinking about internal and external frictions. Um, when our law schools make a big To do about their decision to partner with a vendor or bring in a tool with kind of explicit promises about using it as a thinking partner and its role in supporting. Pedagogy and

00:50:20.620 - 00:50:37.179

learning in the first year, um, but without structures around that to actually fulfill that promise, it feels like we are inviting, uh, new norms around what it means to be successful in law school and inviting in, uh, insecurities around the future of law practice and, and

00:50:37.179 - 00:50:49.489

what we ought to be doing in order to set ourselves up for success, um, and how that might actually create a friction between what. Our institutions are putting out publicly and what we're trying to accomplish in the classroom in that kind of critical space. I'm wondering

00:50:49.489 - 00:51:05.459

if that resonates with you. Absolutely, um, and you know, we very recently, just in the last year, became a all things, well, we were an all things Google, but we got the Gemini full turned on, um, across the, across campus, and yeah, there's a huge disconnect

00:51:05.459 - 00:51:20.290

between What the prime kind of prime movers, first movers in an area would like because they want to be able to test it around, but the expectation that that partnership then puts on faculty on the ground without, no, you, I mean, you said it very well

00:51:20.290 - 00:51:39.340

that there we uncritically accept a lot of technology. With vague promises, um, in a way that we wouldn't uncritically accept a lot of other things. Um, and that certainly bothers me, uh, in the academy. I don't think that it's necessarily a reason to restrict it. I

00:51:39.340 - 00:51:55.489

just think we have to be talking about it all the time. My students have been Initially unwilling to talk much about AI, they were like, it's bad, it's bad, we know it's bad, um, but I think there's probably a shift age-wise in what their exposure has

00:51:55.489 - 00:52:11.679

been, um, to the point where hopefully they'll be a little bit more willing to engage and see it used, and, um, but we also have an obligation, I think, to bring in ethical arguments and treat them seriously, um, and, and treat opposition to using it as,

00:52:11.770 - 00:52:29.040

as, um, as a valid. Position as well. So it's a lot. There's so much and uh institutions seem really, they're, they're just slow. I think it's difficult. They're having the same problems thinking about all this that, that we are. Um, I want to make sure I

00:52:29.040 - 00:52:39.919

give time to the rest of our questioners. Um, with employers wanting to start their hiring at the end of the first semester, as well as their desire to hire law students that are exposed to AI, do you see a way to balance the wolf at the

00:52:39.919 - 00:52:56.989

gate with these concerns and preserving their cognitive processes? Yeah, I, I think it probably requires a lot more understanding of what your career services folks are seeing and what the students are reporting they're seeing, and then what you're doing in the classroom. And I think we

00:52:56.989 - 00:53:13.399

haven't had to have such rapid iteration of that process before, you know, before we were more concerned about, are they using Westlawn next or are they still on, you know, or do they have access to Hain or Bloomberg, um, I think this has to be kind

00:53:13.399 - 00:53:32.544

of a constant, um, you know, cycle of, of learning. I, I don't know the answer to sort of building the skills in engaging. I don't, I'm uncomfortable with the idea of. Bringing it into 1st semester, 1st year, because it's, it, I think it's too early, but

00:53:32.544 - 00:53:49.064

I also think it's too early to do very good research. Um, I think the more time I spend with 1st years, the more I want to push most research to the 2nd semester because they just, they don't understand legal argument enough to know what to do

00:53:49.064 - 00:54:06.510

with these things that they're finding. Uh, so, to the extent that what firms are wanting is facility with the, the AI tools that are built into the legal vendors, um, I think that's a great opportunity for co-curricular instruction or even one credit, um, you know, intersession

00:54:06.510 - 00:54:26.300

options for first years that maybe aren't available to others, if that's even possible. Um, I, I think it has to be opt-in rather than, rather than opt out. Thank you. Um, another questioner asks, I wonder whether you think pandemic-related changes tie into these ideas. I don't

00:54:26.300 - 00:54:46.110

know. Um, I have not really fully engaged with, with that literature from education. Um, I would like to. I I'm sure that there is something to the, the coinciding of when sort of when AI came online and as it became more and more uh available and

00:54:46.110 - 00:55:07.560

better quality at the free level with the at-home emergency remote instruction instruction, um, but I, yeah, I, I don't know yet what that, what that is. Um, I think in law school we're not quite seeing those folks. We are, but they were in college rather than

00:55:07.560 - 00:55:25.979

high school. Something exciting to look forward to. Um, uh, do you think that an inability to focus or maintain attention plays into some of the frictions, like not being able to follow instructions or finding work tedious or issues with self-regulated learning because, or, or in light

00:55:25.979 - 00:55:41.290

of the fact that AI takes away the friction of having to focus? Oh yeah, absolutely. I think that as a Um, uh, uh, you know, adult ADHD person who, who was diagnosed late in life. I also think that there probably are students who just have never

00:55:41.290 - 00:55:56.979

gotten, uh, evaluated, and we do, you know, as, as everybody, I think it does, I think, you know, refer students out to accessible services to make sure that they're, um, that they're getting evaluated if they're feeling like that's a struggle. For them in a way that

00:55:56.979 - 00:56:14.219

it wasn't a struggle before. But yeah, I think we have so many intrusions on our attention these days. Um, they're everywhere from, you know, we all know the devices, the, the notifications, the, you know, the expectation that all of these things will be on all the

00:56:14.219 - 00:56:31.510

time, uh, makes it difficult to, to detach, and it is shrinking our attention span. Um, and yeah, and AI doesn't help that because then, then maybe the anxiety rises and you have an earlier need to bypass rather than rather than later. Yeah, I cannot imagine being

00:56:31.510 - 00:56:45.330

a law student in this day and age. It was so different than when I was a student. Thank you. I will say that also as you were speaking, I was thinking about myself not only as a Educator in law school, but as a parent of a

00:56:45.330 - 00:57:00.989

person who is also grappling at a at an earlier stage of um the insecurities and anxieties that that brings to the forefront, um. Another questioner asks or says this was wonderful, and then asks, law librarians know how hard it can be to exist in law schools.

00:57:01.100 - 00:57:15.149

In my experience, the hidden curriculum, strict hierarchy, and belonging often places law librarians as others along with some of our students. How do we help ourselves to help the students? Do we discuss these issues in class? Does it work well when a group that is lower

00:57:15.149 - 00:57:32.669

in the hierarchy, the pink ghetto, is the one discussing the issues? That is an excellent question, and I support everybody who is doing work in that area. It's just not been me so far, so I'm less well versed in um what the strategies are. I think,

00:57:33.000 - 00:57:45.370

here's just my instinct kind of off the cuff is that we have to be less nice about it. Um, and I think we have to take a cue from the legal writing folks that have Done a pretty darn good job of raising the profile um in

00:57:45.370 - 00:58:07.649

the institution. Libraries in general are, are threatened. Um, we were on a small college campus and that's the Northwest Colleges something that accredits the, the university. Uh. And their new proposed changes drastically restrict the, the emphasis on the library and uh sort of vague up what

00:58:07.649 - 00:58:24.340

you, what it is that you actually have to have in terms of facilities and expertise. So, I think, I think this is a good time for librarians to be partnering with people who know how to do advocacy and, and figuring that out. These are such great

00:58:24.340 - 00:58:38.790

questions. I'm really happy. Yeah, I wanna, um, in, in our last moments, I do want to say thank you to all of our folks in the audience who have, um, uh, been so engaged and in tune with us. Um, I feel like, uh, this, this conversation

00:58:38.790 - 00:58:51.629

came at a perfect moment for, for me and I hope for many of the rest of you as well. Um, and I can't wait to read the paper when, um, when you're ready to share it. Um, so now that we're approaching the top of the hour,

00:58:51.709 - 00:59:07.000

um, I just want to say thank you to Stephanie so much for, um, for sharing this work with us and taking the time. Um, we're going to be back with another Biddle Speaker series webinar next month, so please be on the lookout for information about that,

00:59:07.300 - 00:59:23.979

and I encourage everyone to continue these conversations because we won't, we won't be able to make any of these positive changes, um, without, uh, without continuing that, that work amongst ourselves. Um, couple we have comments coming in on the Q&A. Thank you, thank you, thank you.

00:59:24.219 - 00:59:34.360

Um, so I'll leave it there with that, um. With, with that in mind, um, and thanks to everyone and have a wonderful afternoon. Thank you.