00:00:14.500 - 00:00:32.700

All right, welcome everyone. I know we've got some folks actively entering the zoom now as I speak. So, um, I want to say welcome to everyone. Thank you for joining us. Um, I am so pleased to uh to kick off this year's Biddle Speaker series in

00:00:32.700 - 00:00:52.479

partnership with the Society for American Academic Law Librarians, Saul, formerly known as the New Org. Um, my name's Genevieve Tung. I'm going to be your host for today's session. This speaker series exists to showcase emerging and impactful scholarship on all aspects of law librarianship and legal

00:00:52.479 - 00:01:11.399

information. So it's my pleasure to introduce you to today's speaker, Jacob Sayward, who will speak about his new work in progress, citation analysis as supplemental collections analysis in law school libraries. Collections is, I will be honest, not my primary focus, but even from outside of uh

00:01:11.399 - 00:01:28.004

of that space I sense that research and scholarship in this area is really having a moment. I think it's it's the in the zeitgeist, so I'm really pleased that we're able to feature work on this topic um. Kick off this year's speaker series. Jacob's paper touches

00:01:28.004 - 00:01:44.699

on a lot of foundational questions about what we're trying to capture when we make collections decisions and what that means and how it relates to our institutional goals, the way our faculty and our journals are using scholarship and information to produce their own work. So with

00:01:44.699 - 00:01:59.860

that, I'll say that Jacob Sayward is the associate director for collection development at the University of Minnesota and serves as a member of the library's management team. He has chaired ALL's Committee on Relations with Information vendors, where he met regularly with legal publishers to resolve member

00:01:59.860 - 00:02:19.029

complaints regarding unethical pricing practices and nondisclosure terms and licenses. Hate those. He also chaired the Academic Law Library Special Interest Section committees on both faculty services and collection development. He currently co-chairs NELO's Print to Electronic Standing Committee advocating for libraries on issues related to the transition

00:02:19.029 - 00:02:36.020

of legal information materials from print to electronic format. Throughout today's talk, please feel free to add your questions to the Q&A to reach out via chat if there is a technical or legal issue. I will be monitoring the chat and the Q&A throughout so that Jacob

00:02:36.020 - 00:02:49.190

can focus on his remarks, and then we will have time at the end for him to address your questions at the end of the session. I will read those out for our uh recorded audience. Um, so thanks again so much for joining us and I'm going

00:02:49.190 - 00:03:05.509

to turn it over to Jacob. Hello everybody. I want to start by thanking everybody at Penn's Carey Law School and the Biddle Law Library for virtually hosting me today, and I want to especially thank Genevieve Tun for all the arrangement and help for me. I want

00:03:05.509 - 00:03:18.809

to thank all of you attending today for taking the time out of your work to listen to me talk about this project. This is a real work in progress for me, so I'm sincerely happy for your feedback. In addition to the questions at the end of

00:03:18.809 - 00:03:36.399

the presentation, I want to invite any of you to email me later with any suggestions, questions, or other thoughts. I could really use them. As Genevieve said, my name is Jacob Sayward, and today I'm presenting on a research project tentatively entitled Citation Analysis, a Supplemental Collections

00:03:36.399 - 00:03:55.589

Analysis in Law School Libraries. I welcome suggestions for a better title too. I'm gonna start off by talking about traditional approaches to what we sometimes call collection analysis, or collection evaluation, or collection assessment. Then I'll get into some issues with those, how they play out at

00:03:55.589 - 00:04:14.679

other libraries, and why we don't do a lot of it in academic law libraries. Then I'll get into some recent efforts we've undertaken at the University of Minnesota Law Library. The measurement and evaluation of library services, is this great old book, actually two editions of this

00:04:14.679 - 00:04:32.959

great old book, that breaks down different ways to assess a library's work. It includes, of course, multiple chapters on evaluating a library's collection. It's from a different time and describes libraries that operated differently than today. So some of the assessment ideas will seem far-fetched for a

00:04:32.959 - 00:04:48.079

library in 2025. But I think this type of blast from the past is a decent thing to consider once in a while as an outside check on how we do things now. Especially when so many of us are focused on the immediate tasks necessary to just

00:04:48.079 - 00:05:04.989

help our library get through one more day. Similarly, the fact that this book does not focus on law libraries may keep it from being of more immediate relevance to some of us, but that also means it's essentially offering us an outside perspective, an outside perspective that's

00:05:04.989 - 00:05:21.809

worth stopping and considering, at least once in a blue moon. This quote from the book may seem wonderful to you, or maybe it'll just seem pedestrian and obvious, but it gets to some of the issues that have come up in this research of mine. The taxonomy

00:05:21.809 - 00:05:43.220

adopted in this book breaks down collection evaluation options into two broad categories, material-centered approaches and use-centered approaches. Material centered approaches ask what is in the collection and what value do those materials have. The use-centered approaches ask how the collection materials are actually used. If libraries should

00:05:43.220 - 00:06:01.140

consider combining approaches, mixing and matching, then I think we should keep in mind the different elements of both of these approaches. One of the big examples of the material centered approach is the comparison of a collection with some core list or bibliography of what should be

00:06:01.140 - 00:06:18.690

in a research collection. These are especially common for reference collections, to the point where historically, we've had entire reference books that were just lists of other reference books that should be in a collection, depending on its size, focus, etc. The many new law school libraries springing

00:06:18.690 - 00:06:36.959

up in the 20th century, often consulted lists of core materials that every such collection supposedly needed to have. Another approach is the conspectus approach, brought to us by the Research Libraries Group. If you've ever looked at a collection development policy and seen a long list of

00:06:36.959 - 00:06:53.799

subjects with numerical grades assigned to each, indicating the depth of collecting a library supposedly adopted for each of those subjects, then you've seen the conspectus approach in action. Now, this effort was too widely adopted to be considered a failure, but there are good reasons the folks

00:06:53.799 - 00:07:12.970

behind it ended up abandoning it after several tries. And the trend in collection policies has long been away from such an approach. In my career in law libraries, the times I've heard librarians actually use the phrase collection analysis, have actually been, you know, pretty minimal. But

00:07:12.970 - 00:07:32.209

when they have, they've largely been occasions describing peer to peer comparisons facilitated by OCLC services. A library would compare some of its collection, often a single subject or jurisdiction, to one or more other peer libraries via OCLC. Now, this is a neat tool. It's interesting to

00:07:32.209 - 00:07:50.489

know what other libraries collect, but it certainly requires some follow-up analysis beyond what the tool can provide. For instance, If you discover other libraries in your area, all have a monograph that your library lacks, what is your conclusion? Should you also purchase the monograph, considering its

00:07:50.489 - 00:08:08.950

ubiquity, a sign of its necessity in every collection? Or do you see its wide adoption in other libraries as an indication of easy availability by in library alone, and instead conclude that your finite collection funds should be devoted to something more distinctive for your library. There

00:08:08.950 - 00:08:28.200

are some real limitations to these material centered approaches, incentivizing comparisons to some ideal collection as conceived of by some individual or group at some point in the past. A good check on this approach would take into account some other things. Use-centered approaches, look at what's actually

00:08:28.200 - 00:08:45.989

happening with the collection in a library. The first example would be statistics from circulation. Now, charge out data is limited when so much use of a library's collection happens in the building without actually charging a book out. So libraries often found themselves developing ways of measuring

00:08:45.989 - 00:09:01.900

in-house use of collections as well. Now, the data from interlibrary loan requests can be very illustrative. For one thing, they tell you about patrons' interest in materials that they currently cannot access via the collection, whether it's because the materials were never in the collection in the

00:09:01.900 - 00:09:17.059

first place, or because the materials were simply currently in use by somebody else when the patron wanted them. But I also think ILL requests, requiring at least a little extra effort on the part of a patron, like filling out a form and or waiting for the

00:09:17.059 - 00:09:37.000

material to arrive, indicate an especially motivated interest in a particular type, not just a passing interest. Not all interest in legal materials is the same, and there may be reasons to sometimes distinguish between different uses. Finally, as more of our resources are available online, the usage

00:09:37.000 - 00:09:55.840

data from online research databases can be very informative about how those aspects of the collection are being used, and depending on their availability and reliability, they can inform many collection decisions about whether to, say, renew or cancel a given module of some online database. These are,

00:09:55.890 - 00:10:17.080

in many ways superior to the material centered approach, though they have their own limitations. A lot of usage probably predicts future usage, but for how long? Usage patterns change. A collection analysis focused on use will always help inform a collection designed for posterity. In addition, use-based

00:10:17.080 - 00:10:30.679

measures like circulation data will tell you how the books in your collection are being used, but they won't tell you how the books that aren't in your collection could or would be used if they were present. That's something ILL request data is useful for. Usage data

00:10:30.890 - 00:10:50.190

also doesn't exist in a vacuum. And there can be a circular relationship between your collection, its use by patrons, and patrons' research interests. For instance, let's say a particular book in your collection turns out to have a high amount of circulation. You could conclude that you

00:10:50.190 - 00:11:06.299

just did a really great job selecting such a useful book. But what if it has high circulation because of your library's outreach efforts? Perhaps a new acquisitions list or recommendations from your reference librarians convinced your patrons that the book was very useful. Are you really assessing

00:11:06.299 - 00:11:26.640

your collection building, or simply observing that your patrons follow your library's recommendations about what's useful? Some of these causation issues arise of citation-based approaches to collection analysis, but we're going to spend some time considering these approaches anyway. Now, most of the time in law, as well

00:11:26.640 - 00:11:47.270

as other fields, we talk about citations and scholarship, we're talking about using them to assess scholarly impact. The data becomes a proxy for success and prestige of an individual scholar or institution. But we also sometimes use citations to measure the prestige of publications, specifically journals. When

00:11:47.270 - 00:12:03.070

you go to the Washington and Lee Journal website and sort journals by their citation rates, you're assessing those journals relative prestige. And law libraries have, from time to time, use these tools when deciding which of their dwindling print journal subscriptions they'll continue to renew, and which

00:12:03.070 - 00:12:25.130

they'll decide to instead cancel. Now, for decades, non-law academic libraries have been doing larger, more complex versions of that type of citation analysis for the purposes of collection evaluation. Let's spend a minute talking about what those look like. These large research libraries usually undertook these types

00:12:25.130 - 00:12:43.739

of collection analysis when they wanted to ensure they were allocating their finite collection budget for a particular discipline in a good way. They would focus, say, on all mathematics journals, or all biology journals, and examine which were most cited in that discipline. The result was essentially

00:12:43.739 - 00:13:00.260

another recommended title list for the discipline, and then they would proceed to compare their collection to that new core list and adjust their collection, just like in a material centered approach. Now there are a few things I want to know. The libraries engaging in these analyses

00:13:00.630 - 00:13:16.760

usually looked at how an entire discipline cited the articles in these journals, not a particular subset of scholars. Whether this was because of the citation data available, or because it was assumed a larger corpus of citations would be more predictive, it's not necessarily obvious, but this

00:13:16.760 - 00:13:34.000

is definitely a different model than what I'll be discussing later. Another thing I want to note, here the focus is on on how articles and journals are cited. Almost none of the studies focused on citations to materials that were not journals. Again, whether this was because

00:13:34.000 - 00:13:51.890

of the data available, or simply because individual journal renewal decisions were the most important thing to study is not clear, but I bet a lot of money that it's surely a result of both of those factors. Is such a deep analysis important for law libraries? Well,

00:13:52.090 - 00:14:07.289

maybe not. So much of the use of our journals is through uh is first of all, it's online and it's through either free resources or a few of our most common research databases. And those databases do not offer much in the way of title by title

00:14:07.289 - 00:14:28.059

customization of which journals are made available to our researchers. This type of analysis may be more useful for, again, those dwindling number of print subscriptions our libraries still have, or for those journals unavailable in these large non-customizable packages. But what if we turned our attention to

00:14:28.059 - 00:14:46.510

another part of our collection, where there may be more title by title level selection occurring? Monographs. Let's acknowledge that individual monograph selection is still a bit of a privilege, and that more libraries are shifting to just in time collecting of of monographs instead of just in

00:14:46.510 - 00:15:07.770

case collecting. On top of that, we also have libraries acquiring sub monographs in large e-book packages from publishers, or accessing them via demand-driven programs. But many of the monographs are still coming to us by individual firm orders, being selected individually. Shouldn't we be curious about how

00:15:07.770 - 00:15:26.140

these monographs are used, and by whom, and for what? On that question of how they're used and why, I want to refer back to a category of library assessment from that Baker and Lancaster book I mentioned at the beginning of this presentation, a type of assessment

00:15:26.140 - 00:15:44.320

that is not specifically about collections. User studies was the name of the category assigned to library assessments, usually occurring in the form of surveys, interviews, and focus groups that try to get a sense of how all types of patrons use all aspects of a library. Now

00:15:44.320 - 00:16:01.000

that could could include any aspect of using a library, but some of those, a fraction of those, were about how different patron groups went about their research, which can be very informative about a library's collection. For just one example of one of the studies, one library

00:16:01.000 - 00:16:17.479

observed uh through a user study, that its undergrads mostly stuck to to basic reference materials, while its graduate students were the primary researchers for monographs and journals. And that was very useful information to this uh library. It helped inform future decisions that the library made about

00:16:17.479 - 00:16:35.919

its collections. Now, with this idea of user studies in mind, we should ask ourselves about our own patrons, the types of research they engage in, and when they actually use monographs. And a few important examples of frequent monograph use jumped out at us at the University

00:16:35.919 - 00:16:56.179

of Minnesota Law Library. When our faculty publish their scholarship, whether in a journal or a monograph, they will cite to many sources to back up their assertions. Some of these citations will be to monographs, and each citation supposedly indicates that the faculty author actually reviewed the

00:16:56.179 - 00:17:10.680

monograph in question. We're all gonna pretend to believe that for just a few minutes. That's a pretty important type of use. This is some of the scholarly output of the law school, the stuff we memorialize in our institutional repositories, and supporting it is a core mission

00:17:10.680 - 00:17:31.038

for a law school library. How good a job our monograph collection does supporting this mission is an important question when evaluating our collections. Similarly, we also have the journals published by our respective schools. These journals are edited by students who undertake painstaking site checking and source

00:17:31.038 - 00:17:49.109

pulling processes, where they try to review every work cited in an article excepted for publication, to ensure the source backs up the assertion the author is making. And when the author cites monographs, the students need to get their hands on these monographs. Supporting these school published

00:17:49.109 - 00:18:07.589

journals is an important mission for our libraries, and assessing how we do this is worthwhile. So that means gathering data on the monographs being cited in our faculty scholarship, and in the scholarship published in our school's journals. Now there are surely a lot of different ways

00:18:07.589 - 00:18:21.930

to go about doing this, uh, but one important, uh, kind of distinction I think we should keep in mind is going to be, uh, basically the different levels of difficulty and the different methods of trying to get this data. It could take a lot of time

00:18:21.930 - 00:18:36.219

and labor to gather this data manually. It would be easier if all the data existed in some format that would allow us to expedite things. Now, for better or worse, at the University of Minnesota Law Library, we now have experiences with both difficult and less difficult

00:18:36.219 - 00:18:56.750

approaches. About a year ago, part-time student workers in our acquisitions department began reviewing citations and journal articles, authored by our law school's faculty, and copying information about all the monographs cited into a list. Then they would they would review the cited monographs in this list against

00:18:56.750 - 00:19:13.219

our holdings to see which were available to our patrons from our own collection. They were trained to do this by our serial's assistant, Kelly Schwartz, who also designed the workflows they would follow. These student workers had some experience with legal materials through their other library responsibilities,

00:19:13.589 - 00:19:30.000

but there was no expectation that they would be perfect when it comes to reviewing citations of legal scholarship. When reviewing journal articles, they were greatly helped by the fact that so many had blue book formatted citations. Monographs would jump out at them in all caps. If

00:19:30.000 - 00:19:46.550

I cited sources status as a monograph, was at all ambiguous? We asked the student workers to err on the side of considering a monograph. This means the resulting lists did include a number of sources we weren't so interested in. Alongside all the scholarly scholarly monographs we

00:19:46.550 - 00:20:03.910

wanted to know about, there were plenty of white papers published for free online by NGOs, or even centuries-old legal texts that had that had been published and republished in countless countless iterations over the years. This meant the resulting lists were a little longer than preferred, and

00:20:03.910 - 00:20:22.150

that students were spending time reviewing some sources we didn't care as much about for holdings in our collection. We were not conducting a scientific study that we intended to publish in a bibliometrics journal. We were merely reviewing cited sources to inform our collection practices. When we

00:20:22.150 - 00:20:39.130

set our student reviewers on faculty authored publications, we chose to have them go through articles published in our school's SSRN research paper series. This had the benefit of being a single chronological list of articles published by our law faculty, with no duplication and at least a

00:20:39.130 - 00:20:58.560

certain amount of version control. Having one's articles included in the series is widely enough desired by our faculty to result in most of their journal articles being included, if not complete 100% comprehensive coverage. What the SSRN research paper series is missing, of course, are the monographs

00:20:58.560 - 00:21:13.680

authored by our faculty. We have not yet come up with a great workflow for reviewing all the citations found within monographs authored by our faculty. We do keep track of our faculty's publications, including monographs, but we could stand to be a little more systematic about it.

00:21:14.540 - 00:21:29.010

It'd be great if we had a single chronological all encompassing list for our students to work through. Another problem that would come up would be for any monogas for which we do not have ebook access. For these print monographs, citations would have to be transcribed instead

00:21:29.010 - 00:21:47.790

of merely copied and pasted. That would inevitably slow down the process. There's also the fact that citation formats will vary more greatly across monographs than they do across journals. Also, it could just be more daunting and demoralizing as a student worker, to be tasked with going

00:21:47.790 - 00:22:02.689

through the seemingly endless citations of a monograph compared to an individual journal article. The light at the end of the tunnel is more visible. The workflows need to be fleshed out, but we just haven't done this yet. But I do think this could yield some interesting

00:22:02.689 - 00:22:20.199

results. We haven't started reviewing every article in every issue of our school's journals for citations, like we did with our school's SSRN research paper series, but we have tried to get a jumpstar in this process by getting a hold of these source pulling spreadsheets our journals

00:22:20.199 - 00:22:34.239

student editors use to facilitate and track the citations for each of the articles published in their journals. If you've ever helped out one of these students at the reference desk, you may have seen one of these spreadsheets. They already have all the cited sources listed, and

00:22:34.239 - 00:22:51.540

they may even have these sources already cited, already sorted into categories like journal articles versus monographs, versus case opinions, versus statutes, etc. You can imagine how valuable it is to our process, to have all the monograph citations already listed and easily filtered for in an existing

00:22:51.540 - 00:23:08.579

spreadsheet. Each of our school's journals has a reference librarian assigned as a library liaison for them, and our reference librarians have a lot of contact with the students working on the journals. These liaison librarians have helpfully requested source blend spreadsheets for recent articles to aid us

00:23:08.579 - 00:23:27.800

with the citation analysis. We haven't received anywhere near 100% of the source blend spreadsheets for recent journal articles, but each one we do receive saves us time in gathering citations and sifting for the monograph citations among us. Maybe at this point, if you fancy yourself a

00:23:27.800 - 00:23:40.400

data person at all, you're ready to scream at me and at all the work that's going into gathering this citation data. You may be screaming, we have tools for this already, Jacob. You may be yelling at your screen, don't rely on students or even humans when

00:23:40.400 - 00:23:57.589

this data exists already. Well, there are citators and there are citators, and a lot of the favored citators in legal research platforms designed for cases citing other cases, don't do as good a job when it comes to journals citing monographs. And there's another big legal research

00:23:57.589 - 00:24:15.880

database that's a lot more journal focused, but still has really underdeveloped citator functionality, despite years of promised improvements. I won't name names, but the best citator platforms for this type of analysis are not those focused on legal materials. Now you may have used Web of Science

00:24:15.880 - 00:24:34.130

scope as before for interdisciplinary research. I found myself using them primarily on occasions when they had a particular data point coded that Westlaw didn't, or because they had export functionality that Hanline lacked. Their coverage of legal scholarship leaves a lot to be desired, but we haven't

00:24:34.130 - 00:24:52.540

been aiming for perfection with our citation analysis. We're just, we just wanted to review as many of the monographs cited by our faculty and journals as possible, so a large bulk would be helpful, even if it were far from comprehensive. Scope has helped with that. A

00:24:52.540 - 00:25:07.800

key functionality for Scoopus is that it allowed us to export all the monographs cited in all our faculty authored works in one large CSV file, instead of having to export the citations from each article individually. Perhaps this can be done with web of science as well,

00:25:07.969 - 00:25:23.579

if we get into the weeds with its API, but we have not gone to that stage yet. Even Scopus's basic export function has a cap on the total amount it can handle, which we were thankfully just under. If our school's data had exceeded this cap, we

00:25:23.579 - 00:25:37.770

might have had to arbitrarily break it up into multiple export attempts by, say, a year. Though the hardcore Scopus users would point out that we should simply just start working with the Scopus API to get larger amounts of data out of its platform. That's not a

00:25:37.770 - 00:25:53.910

priority for us now, figuring that out, figuring out how to do that is not a priority, priority for us now. But if we ever decided to look at monograph citation data for more than one school at once, it would probably be a necessity. Scopus also has

00:25:53.910 - 00:26:08.569

a handful of monographs authored by our faculty in its database, which means that we could get lists of all the monographs cited within those monographs, quickly and easily out of Scopus for our review. One can only hope that Scobase is going to grow its coverage of

00:26:08.569 - 00:26:30.859

monographs in the future. These different methods all yield lists of monographs which our student workers compare to our library's holdings. Our library's ILS and catalog are integrated into the larger University of Minnesota library system. So when I say our holdings, at least sometimes I'm talking about

00:26:30.859 - 00:26:47.449

all the print holdings of all these University of Minnesota libraries, along with all the ebooks available in our system. Again, we did not expect perfection on the part of our students during this part of the process. They inevitably missed some holdings, misinterpreted additions or distinctions between

00:26:47.449 - 00:27:04.209

monographs, but the results still allowed us to review large amounts of the monographs cited by our faculty and within our journals, and determine if we currently held them. Is there a way to automate this part of the process? Probably, but we haven't nailed it yet. The

00:27:04.209 - 00:27:22.079

lists of sighted monographs don't contain unique identifiers like OCLT numbers, or seldom contain DOIs because those aren't in the citations. If we did have some unique, more unique identifiers, we could use some tools to more automatically compare them to our holdings, instead of having our student

00:27:22.079 - 00:27:38.489

workers conduct individual searches for titles and authors. If I consult more with some experts on our ILS, perhaps we'll find a way to streamline some part of this process. Or perhaps someone will come up with a great Python script or AI based solution. For now, we

00:27:38.489 - 00:27:55.479

have the results we have. The results did not, of course, definitively tell us the fact that the author or source pulling student used our library's copy of the monograph when they needed it. Perhaps that copy was charged at the time, or perhaps it hadn't even arrived

00:27:55.479 - 00:28:10.959

at our library. Perhaps the faculty member had their own personal copy of the monograph, or the student editor relied on in a library alone to review the part of the monograph they needed. Perhaps, horror of horrors, a faculty member cited a monograph without actually reviewing it.

00:28:11.290 - 00:28:28.599

I've heard tale of such a thing. But given our even more limited abilities to screen for such things, we are happy to consider these citations a relatively reliable proxy for not only actual use of these monographs, but significant valuable scholarly use of the type the library

00:28:28.599 - 00:28:45.189

can mention to the dean when justifying its collection budget. I'll get into the results of our review against our holdings in a bit. But for now, I'll just say that most of the legal monographs cited were in our collection. Or perhaps you're wondering if that's even

00:28:45.189 - 00:28:57.869

necessary. Maybe you're thinking, so what, Jacob? You bought a bunch of monographs that ended up getting cited. If you hadn't bought some of them, couldn't you just have interlibrary loan to them? After all, student editors are pretty used to using interlibrary loan as part of their

00:28:57.869 - 00:29:12.430

source pulling process. Well, For any given monograph, I think that's true enough. When you purchase a lot of scholarly monographs, the value of any given one is going to be relatively minimal. But there are costs to enter library alone, so we really shouldn't think about it

00:29:12.430 - 00:29:29.689

as a completely seamless replacement, especially for a whole swath of monographs. And while student editors And while student editors relying on in library alone a bunch is one thing, it's a good thing for faculty to find most of the monographs they need in the library, especially

00:29:29.689 - 00:29:45.910

if those monographs meet some nebulous shifting criteria in that faculty member's mind of what a library should have. Our faculty are inevitably going to use ILL some, but wouldn't it be great for the library's reputation in the law school, if this was something they encountered as

00:29:45.910 - 00:30:03.449

an exception, primarily when they wanted something pretty far afield, subject of law. It's also possible that the very reason faculties take the monographs in our collection is because those monographs happen to be in our collection. Faculty outreach varies across libraries, but maybe the faculty member is

00:30:03.449 - 00:30:18.500

setting a monograph because their library liaison flagged it for them when it arrived, or because they noticed it on the library's new acquisitions list. If this is the case, I think your library library's outreach efforts deserve applause, because many libraries are cutting services like new acquisitions,

00:30:18.569 - 00:30:35.500

due to poor adoption by faculty. But it's true that this data, such as it is, could be measuring things other than how good we are at collecting, and I think that's important to. What we've been doing here is not a scientific study, leading to publishable results

00:30:35.500 - 00:30:49.619

from which grand conclusions can be drawn about our collection. What we've been doing is an internal review. We started with some instances of monograph use we really cared about, that fit with some of the core missions of our library, and we look for some ways to

00:30:49.619 - 00:31:04.089

see how our collection measured. That's why I haven't been focusing on the end results or giving you any numbers yet. It's why the title of the presentation includes a hedge word like supplemental in it. There's plenty more we could do to increase the reliability and validity

00:31:04.089 - 00:31:20.430

if we were going to treat this as a scientific study. So far, the primary use of the resulting data is the facilitation of second chance acquisitions. I look at the sighted monographs that are not in our holdings, and consider whether they should be purchased at this

00:31:20.430 - 00:31:37.540

point. This is still a nuanced-filled judgment call approach. But I have more information than the person in my position had when these monographs were first published, because all these monographs were consulted at least once by stakeholders at our own institution to further our institutions scholarly up.

00:31:38.349 - 00:31:52.560

That moves the needle quite a bit on acquisition decisions in my mind, even if it doesn't make an acquisition decision, a slam dunk. If the monograph was cited once, but is still extremely far out of scope for us, I might pass on it again. But an

00:31:52.560 - 00:32:07.890

interdisciplinary resource a bit closer to law, if still available to purchase and relatively cheap, I'm deciding to purchase some of those now. And the odd monograph that was cited and is firmly in our scope as a legal monograph, but somehow never made it into our collection.

00:32:08.140 - 00:32:23.680

I'm using this citation as a great excuse to purchase it in retrospect. Again, we're not assuming that a single citation is going to predict lots of future citations, but a single citation is one more use than many volumes in our collection will ever receive. Maybe it's

00:32:23.680 - 00:32:40.219

at least a little predictive. Because most of our citations don't include publisher information, it's going to take more work to track how often monographs from different publishers are being cited. But that could be one example of a future avenue of exploration. What if, for instance, we

00:32:40.219 - 00:32:53.869

found out that Publisher X is cited a lot more than we thought and publisher Y a lot less? That could definitely affect our future acquisition decisions. There are a lot more patterns we could try to examine in the list of sighted monographs that aren't currently in

00:32:53.869 - 00:33:12.760

our library holdings. We just have to devote the time to figure them out, and then use them to inform future monograph purchase decisions. One distinction between this review and the citation analysis conducted by so many non-law libraries is that we've been exclusively focused on local citations.

00:33:13.250 - 00:33:26.959

That's because we're most concerned with how our monographs have been used, and there's no reason to think that a citation by someone with no connection to the University of Minnesota would have used the monograph in our law library. An added element is that the manual approaches

00:33:26.959 - 00:33:43.119

to collecting this data take a lot of time. So focusing on a single institution may be all we have time to explore. But if we somehow perfected the automated approach to this citation-based collection evaluation, it may behoove us to undergo a more discipline wide review of

00:33:43.119 - 00:33:59.099

all citations to legal monographs. They would not tell us much about how our existing collection has been used. Maybe even how it might have been used had we made different monograph purchase decisions in the past, but if we ultimately, even if we ultimately only cared about

00:33:59.099 - 00:34:18.179

the future of our library's collection and the use of its monographs by our own school stakeholders, there's always the possibility that citations to monographs by all legal scholars everywhere, better predict our faculty's future citation patterns than our faculty's past citation patterns do. First, the input may

00:34:18.179 - 00:34:35.340

gain reliability simply from being much larger. Most of the monographs cited in our review were only cited in one article. Those cited in more than one article may largely be cited by the same faculty member in successive articles. But a review of all legal scholarship could

00:34:35.340 - 00:34:51.198

give us a lot more granular impressions of monographs cited a large number of times versus a medium number of times versus a small number of times or not at all. A monograph cited once by our faculty could be a real outlier, or maybe our current faculty

00:34:51.198 - 00:35:07.310

is itself the outlier compared to other faculty. And our faculty, like all school faculty, will evolve over time. Perhaps their citation patterns will change drastically. But as long as we're looking at purely local sites by our own school stakeholders, I think we're conducting more of a

00:35:07.310 - 00:35:24.830

youth centered analysis than a material centered analysis, even as it borrows from some material centered approaches. We're definitely not using our faculty citations to create some new universal core lists against which we think all uh collections should compare themselves. We are merely trying to get a

00:35:24.830 - 00:35:41.020

better understanding of our collection's use. So, uh, I kind of want to de-emphasize the numbers as much as possible because they're not really the point of this, but I figure if you're curious, you deserve to know a little bit about them. So, uh, again, first I'll

00:35:41.020 - 00:35:57.419

just start with some very rough numbers from that kind of manual effort of reviewing our faculty authored journals, journal articles for citations to monographs. Uh, our student workers have gotten through a couple of years of faculty authored journals, uh, journal articles so far. Uh, one thing

00:35:57.419 - 00:36:12.939

to note is that the rate of monograph citation varies greatly across journal articles. I'm sure it varies greatly by faculty author as well, but definitely by articles. Some will have no monograph citations at all. It doesn't take much time for students to get through those. Others

00:36:12.939 - 00:36:31.860

will have a few, and others will have quite a bunch. Um, so far, our, our big list includes almost 3000 sighted monographs, though I, I want to stress this includes some duplicates, and deduplication is a big issue because of how inconsistently monographs are cited. Uh, we

00:36:31.860 - 00:36:46.100

still have a lot of deduplication to do, so that number could really be a lot smaller. Uh, more than 500 of those, uh, have been reviewed as potentially being absent from our holdings. Again, we're not perfect about how we compare these against our holdings, uh, but

00:36:46.100 - 00:37:02.909

we've definitely gotten through a few 100 of these so far. Uh, between 100 and 200 of those absent from our holdings, we've decided to purchase at least contingently. Uh, it's it's between 100 and 200 because some of the decisions are really also going to be a

00:37:02.909 - 00:37:18.919

function of is the monograph still available to purchase and what is the price. But assuming it's available and cheap, you know, that's, that's more than 100 extra monographs we're going to purchase as a result of this graph citation review. Now, if we compare this to kind

00:37:18.919 - 00:37:36.300

of the more automated process yielded by Scopus, we got a much smaller list, maybe 1/3 to 5 as long as our monograph citation list from the uh manual review. But these are very incomparable lists. Uh, Scopus is basically everything it's in this database from all time,

00:37:36.550 - 00:37:51.409

right? Uh, so it's reviewing across a much larger range of publication dates, whereas the manually reviewed list of journal articles, we only really have gotten to the past couple of years worth of journal articles from our faculty. So we really don't at this point, have a

00:37:51.409 - 00:38:05.370

great sense of how much overlap there is between the scoopist list and the manual list. There's surely some, but it's gonna take a lot of effort to really figure out which are in both and how much of an effect versus uh the overlap has versus a

00:38:05.370 - 00:38:24.270

more uh distinct sense of these two lists. Uh, one other thing I want to note, um, usually when we talk about citation, you know, citation metrics, citation rigging, citation analysis, uh, we're talking about them in terms of impact, and there's a lot written and spoken about

00:38:24.270 - 00:38:40.520

how the many ways in which this is problematic. The, the way things are cited is problematic, the ana analysis is problematic, the conclusions drawn from them are problematic. Um, we are kind of staying away from judgments of prestige and value of these citations, uh. As a

00:38:40.520 - 00:38:56.469

part of overall legal scholarship. So we're telling ourselves that, you know, this study is really just about these citations as evidence of these monographs being used. Um, so we're kind of sidestepping all the problematic aspects of, you know, citation metrics, um, but I want to acknowledge

00:38:56.469 - 00:39:13.979

that that is kind of a cop out because assuming we actually keep going forward with this and make future collection decisions based on what we find in these, uh, you know, citation analysis, we are kind of perpetuating some of the problematic aspects of citation metrics. So

00:39:13.979 - 00:39:26.889

that's where I'm at with this research project. I, I truly think law libraries could stand to do more analysis of their collections, uh, but there are good reasons for there being no one good form of collection analysis that we've all adopted or that we all use.

00:39:27.739 - 00:39:41.959

I do think we could benefit from spending a little more time thinking about first our law libraries missions, and then from there why we collect the materials we collect and then from there distilling that down to the most important uses of some of our collection materials

00:39:42.290 - 00:39:58.129

and then working backwards from there on how we might measure them. But in no way do I think every library should immediately dive into laborious or time-consuming approaches to measuring how their collection ends up getting used. I wish for all our sakes that these more automated

00:39:58.129 - 00:40:13.409

ways of measuring the stuff reveal themselves and get better, because I would like to see more libraries playing around with this, experimenting with this, testing these out, and using them to get a better sense of their monograph collections, their collections overall, and how their collections are

00:40:13.409 - 00:40:30.070

being used. But however we end up approaching it, I think there's a lot to learn about how well our collections serve these goals. With that, I'm happy to hear some questions, comments, suggestions, and thank you one more time for your time. Thank you so much, Jacob.

00:40:30.110 - 00:40:47.429

That was fantastic. Um, I'm, I'm gonna jump into uh some of the Q&A that has populated thus far. People are very free to add more questions as we're chatting. Um, uh, our first question, uh, and I think this goes to, um, I'm gonna jump in with

00:40:47.429 - 00:40:59.800

my own thought first, uh, which is as you were speaking I was thinking about how, uh how much I would love to be able to say to a new faculty member who we're recruiting. Oh, we, we, we've already, we've already analyzed all of your monograph citations

00:40:59.800 - 00:41:12.919

and we have all these things in the collection. It's going to be such a great draw for you. We, we, we're, we're trying to read your mind. Um, so our questioner asks, have you ever surveyed faculty as to why they cited certain works and in order

00:41:12.919 - 00:41:30.310

to investigate citation quality as opposed to or perhaps in addition to quantity. Uh, the short answer is no, I am not doing that, done that. That's very interesting. Um, there, there are times in my career where I've had more of a Scarley Services responsibilities, and I

00:41:30.310 - 00:41:43.350

wish I had done that. Um, I definitely encourage anyone to do that because I think we can all learn more about that, um, and I think that'd be of interest, of course, not just to me with my kind of niche way of looking at collections demonstrated

00:41:43.350 - 00:42:00.899

here, but to scholarly services overall and services overall. Thank you. Next question. Given that most scholars read many, many more sources than they actually cite, are you concerned at all about the reliance on this method may skew the perception of the collection in the same way

00:42:00.899 - 00:42:16.840

that reliance on circulation statistics skews analysis by ignoring in library use? Uh, yeah, yes, I think, I think we should be concerned in the sense that if we, we only relied on these, if we over relied on these, uh, yeah, things would get pretty skewed. Um,

00:42:17.169 - 00:42:33.419

I do not have optimism about anything I'm talking about being widely adopted or having a huge impact, so maybe that's why my concern is minimal. Um, but also I guess I would, I would love to be here, hear some theories about um. The citation to overall

00:42:33.419 - 00:42:46.830

monograph review ratio and why it would be consistently skewed in one direction over the other. I'm sure there are some persuasive examples, but once we know that, then maybe we could correct for that or take it into account. Um, I certainly don't think all of our

00:42:46.830 - 00:43:02.689

monograph collecting should be based on a citation analysis. Uh, you know, the examples I've talked about are a little bit more about course correction. Uh, and I'm, I'm a little less concerned about that kind of running amuck. Um, but certainly the maximalist version of this or

00:43:02.689 - 00:43:20.959

anything could really skew things, and that's, that's completely true. That makes sense. We should keep in mind. Um, another question, uh, citations are field dependent, so with this kind of study lead to some collections being potentially lopsided in terms of highly and low cited fields, i.e.,

00:43:21.570 - 00:43:41.649

constitutional law versus tax law. Yeah, I think this is definitely true, um. But I also think field representation in a law school are, I mean, are not balanced either. Um, if, if we tried to serve, say, all of our current faculty, we'd find that certain fields

00:43:41.649 - 00:43:56.649

are more presented than others, and yet if we were being completely our collection building we're completely driven by that fact by the interests of our current faculty, they would also be skewed. So I think that's true and in some ways, I guess the short version is

00:43:56.649 - 00:44:08.310

in some ways it'd be true and it'd be skewed, but it would like make sense for it to be skewed because it's meeting the interest of our current faculty. But in other ways you never want to be too skewed, and the interest of your faculty will

00:44:08.310 - 00:44:28.020

change over time, and you don't want to overindex for that. So, keeping that in mind is completely correct. Um, next question, um, did your data find journals or monographs that were cited more than once or significantly more than once? If so, I would imagine that being

00:44:28.020 - 00:44:47.100

too valuable information confirming for journals you subscribe to that are frequently cited or are important to keep. So I guess this is about the uh the the. Amplitude of of those duplications. Yeah, so we didn't look at um citations of journals. We looked at journals, uh,

00:44:47.139 - 00:45:01.860

citing monographs and we looked at, you know, yeah, basically journals, journal articles citing monographs and a tiny bit of monographs citing monographs. Um, most of the monographs were cited, uh, you know, that we found were cited once and only once. Um, you know, we're we're one

00:45:01.860 - 00:45:17.310

school, only so many monographs are gonna be cited, and then I, I hesitant to say this because I haven't systematically reviewed this, but just anecdotally my review, my sense is that when there were monographs that were cited repeatedly, they were cited repeatedly by the same faculty

00:45:17.310 - 00:45:27.469

member. So they might remember writing the same subject for like 3 or 4 articles in a row and keep bringing up the same monograph 3 or 4 articles in a row. Now, I'm really glad we had that monograph because apparently this faculty member really likes it.

00:45:27.860 - 00:45:43.340

Um, but those are the instances. Uh, we haven't noticed, you know, any monograph with like a runaway amount of citations, or at least not what I would consider like a scholarly monograph. But I do think if we did like, um, in a discipline wide across many

00:45:43.340 - 00:46:01.129

schools, we would start seeing some monographs separate themselves out, again, especially by discipline and affected by things like skewed discipline citation rates. Um, so yeah, I think that's definitely a possibility, but nothing has quite revealed itself yet. Thank you. Uh, I, I, I'm gonna jump in

00:46:01.129 - 00:46:15.719

because I want to make sure I get time to ask my question as well. Um, you had mentioned that you have a shared catalog with your larger university collection. Um, we were just talking about interdisciplinarity. I'm curious if you recall or if you could generalize, um,

00:46:16.250 - 00:46:33.389

that, I mean, oftentimes when I imagine what my faculty are citing that we don't hold, it's often interdisciplinary uh titles that we would not collect uh. Because they were out of scope or something like that, I'm curious about how you approach, um, did the results of

00:46:33.389 - 00:46:52.889

your research show a bent towards titles that would be through your institution's main library versus your law collection, and if this and also informs kind of how you think about the scope of what should be in the law collection. So Can I, I It wasn't a

00:46:52.889 - 00:47:08.810

perfect enough study to be really quote exact numbers, but most of the monographs cited were law monographs. Most of those, overwhelming amount of those, were already in our collection, but most of the ones that were cited that we didn't have were not long. Uh, so, um,

00:47:09.209 - 00:47:23.010

they were things that we would hope to rely on from the rest of the university, and we're lucky enough to be attached to a large research university, and they did provide many interdisciplinary titles, um, but they didn't provide all of them. And as far as how

00:47:23.010 - 00:47:39.800

that informed our future collection, um, there's a few that we bought if say university had a copy but it was in like a real far flung like satellite campus, um, but we're pretty content to rely on ones if they're, you know, in our parent university libraries

00:47:39.800 - 00:47:54.929

and those libraries are not far away, they'll be delivered the same day. Um, beyond that, uh, I don't know how much the parent university part of it really played into the played as a factor, but Genevieve, is there some aspect of the question I'm not really

00:47:54.929 - 00:48:10.830

getting at? No, you you're answering me perfectly. I, I, I, um, we talk a lot about interdisciplinarity at my institution, and I'm always like, how do we shift our lines? So this is, this is very, um, what you're saying makes perfect sense. Um, our next uh

00:48:10.830 - 00:48:28.649

questioner asks about using AI to scrape for monograph citations, thoughts on that. Yeah, I mean, uh, if I was starting it again from scratch, uh, maybe that's how I do that. But, um, the little bit I saw from just pecking away AI was, uh, it wanted

00:48:28.649 - 00:48:42.100

to give me a lot of like Python scripts, and then I would have to remind myself everything I've already forgotten about Python, which is everything I've ever known about Python, um, and implementation to it. So if you have like a data librarian, uh, available to you.

00:48:42.320 - 00:48:52.350

Uh, I would encourage you to go wild with that stuff, but right now I didn't have a great way of doing it. Um, I was, I I certainly I was hoping that, yeah, I could pull a rabbit out of a hat when it comes to like

00:48:52.800 - 00:49:07.070

searching our library's holdings or something else, uh, but it, it didn't quite do it. But I think, you know, certainly better people will have better ideas to uh how to use AI to make some of this more automated and less laborious and time consuming. Thank you.

00:49:07.820 - 00:49:22.260

Next questioner says thank you so much. It seems to me that citations are primarily a function of discoverability, i.e., the thing that gets cited is the thing that showed up high in the results for a user search. While this can be relevant, do you have concerns

00:49:22.260 - 00:49:39.540

that citations are really just a measure of reach or search engine optimization rather than content value? Yes, it concerns me before I get to this project as a concern for our field scholarship, like legal scholarship. This is a problem for legal scholarship, and it's one of

00:49:39.540 - 00:49:54.340

those, you know, uh, problems with recitation practices that people point out, and I hope the people complaining about it win, and I hope they persuade people to change how they do it, um. But until they change how they do it, I think our faculty are gonna

00:49:54.340 - 00:50:05.899

cite what they're gonna cite, and we're gonna have to provide the monographs for that one way or the other, and it's easier to provide them from our collection. And I don't mean to be fatalistic fatalistic or resigned. I applaud folks who are fighting more than I

00:50:05.899 - 00:50:21.280

am to fix that fundamental problem, but I do think it's a fun, fundamental problem about how we do uh scholarship in law, and our library's collection is just like a little bit downstream of that. Uh, now, to make sure we don't Too much a victim of

00:50:21.280 - 00:50:33.719

that problem, we can make sure we don't only ever buy monographs that are cited, and I think we do. Most biographs we buy are brand new, so we don't know whether they'll be cited or not, but I don't think shifting a little bit of money to

00:50:33.719 - 00:50:51.199

ones we know are cited is gonna be too much of a detriment, um, but people are right to be concerned about citation practices in the field of law. Thank you. Um, I have one comment and then another question. Um, our commenter says, I think John asked

00:50:51.199 - 00:51:06.280

about the, the was one of our earlier questioners. I think John's point is important, particularly when you look at faculty scholarship and non-law journals. One of my faculty recently told me her publication in a medical slash health journal is limited to 7 citations, so there isn't

00:51:06.280 - 00:51:23.239

room for her to cite everything she used or finds valuable, which is a fascinating practice that is news to me. So thank you for that comment, Pam. And next question, have you thought of any ways to account for citation biases that might perpetuate problematic aspects of

00:51:23.239 - 00:51:42.419

citation analysis, such as accounting for self citations or citing to only notable names in the field? Yeah, I mean, um. Uh, again, I think, I think we're, we're downstream of those fundamental problems with citation practices, but I don't want it to just throw up our hands.

00:51:42.790 - 00:51:58.909

So if I had a great idea for Compensating for that, I, I would implement it. But right now my great idea is just not to overindex on sighted stuff. Uh, sighted stuff is still, I'd say just a supplemental aspect of our collection analysis and how we

00:51:58.909 - 00:52:16.270

go about our future collection building. I would never suggest that it should drive everything we do. Um, I think just when thinking about my own school, because so many of the monographs cited are only cited once. I don't know if there's a runaway problem of this

00:52:16.270 - 00:52:34.050

obnoxiously oversighted legal scholar just as being cited over and over again by our faculty. Um, we're still dealing with monographs where the difference, the most important difference is, are they cited 0 times or 1 time. And that's a, that's kind of a dangerous level of subtlety

00:52:34.050 - 00:52:51.310

to draw strong con strong conclusions from, and that's why I'm kind of encouraging us not to draw too strong conclusions from them, but I also think it mitigates the runaway oversighted faculty scholar problem a little bit. Uh, well, thank you. Uh, this is the last, uh

00:52:51.310 - 00:53:09.229

last call for Q&A, um, but I'm I'm, I feel like you've you've definitely tapped into something that, uh, is, is front of mind for a lot of folks, um, in our profession. So I am very grateful that you are undertaking this work, even if it's not

00:53:09.229 - 00:53:28.729

a scientific survey, um, I think it's incredibly helpful um to think about. Different ways of uh slicing, slicing this up, um, and applying different facets um to our collection analysis. So, um, thank you all so much. Oh, we have one. Last question. Oh, thank you. Uh,

00:53:29.060 - 00:53:43.260

I talked to fill up the time so the question can come in. Um, John asks, do you take into account the time of publication? A book cited on AI, for example, might be obsolete even if it came out a few years ago. Your thoughts? Yeah, so,

00:53:43.459 - 00:53:56.889

so there is a little bit. You know, scholarship out there on citation rates, mostly outside law, uh, you know, is it, is it, are most of the things getting cited really new stuff or are they mostly the same old stuff getting cited forever and there's this

00:53:56.889 - 00:54:11.090

kind of, my understanding is it's kind of bimodal distribution, but I bet if I had time to look into what we found on our monograph citations, most of the things are being cited within a few years of publication, and then there's probably like a long die

00:54:11.090 - 00:54:28.399

off except for a unrepresentative few monographs that get cited forever. Uh, I think that's true, and it would be a good reason to keep in mind that you don't wanna build your austerity aspect of your collection around citation practices. You only want to use them to

00:54:28.399 - 00:54:44.750

inform kind of the near-term collection decisions. Thank you so much. Um, we have a last last last word in the chat is thank you, Jacob, very interesting. I'm sure you see that. Um, I would like to just thank you again on behalf of everybody here for

00:54:44.750 - 00:55:02.219

sharing your work, um, and we will, uh, just for folks who are watching, we will be posting a recording of this conversation. In Pen carry Law's institutional repository, um, and we will have a link to Jacob's paper there as well when it is ready for all

00:55:02.219 - 00:55:17.139

of you to read it. Um, so please check that out. Jacob, thank you so much again for joining us this afternoon. Um, it has been a pleasure and uh thanks to all of you for attending and please look forward to our next convening of the Bid

00:55:17.139 - 00:55:39.979

Speakeries, which will be on. Uh, December 4th, right after Thanksgiving, um, where we will, uh, pick up the conversation. Um, so, yes, have a great, um, afternoon, um, or yeah, it's afternoon for most of us here, um, and, uh, thank you again for joining us. Yeah