I presented this past Saturday at SAA 2016, as part of Panel 704: Assessing Archives – Case Studies in Using Data as an Advocacy Tool. It was my first time attending SAA, and it was pretty amazing to see just how many archivists could be in one place at one time. The week+ in Atlanta heat was an issue but I did make it out alive
It’ll be obvious that this talk is a revision/update of one I gave at ACA in 2014. I haven’t done much with the data analysis since that time, but have reflected a fair bit on what it means. The discoveries from that work have impacted my strategy in outreach projects as well — particularly in thinking about the value of comprehensive web exhibits.
Evaluating Outreach Activities at a Municipal Archives
This presentation is based largely on research that I did while I was the lead person for outreach at a municipal archives. Since the proposal went through, I switched jobs, so I’m now considering how to apply my findings in a role where I’m starting outreach work completely from scratch.
I’m going to talk today about the data I gathered, and the conclusions I was able draw from that data, about how well outreach projects were actually working. Then I’ll reflect a little at the end on how I’ve been able to respond to those findings, and what things I have been able to actually do differently in my work.
For a bit of context: Thunder Bay has a population of just over 100,000 people, and we’re on the northwest shore of Lake Superior. Our land has been inhabited by and a meeting place among the Anishinaabe people for thousands of years. It became an important site for the fur trade in the early 19th century, and then was settled along with industrial booms in mining, then logging, then shipping. Although we are a small city, we also serve as a de facto regional capital for Northwestern Ontario, an area of over 200,000 square miles, smaller than Texas but bigger than California.
Thunder Bay’s municipal and settled history is less than 150 years old. Archival records of this history are held by a handful of institutions, including the City Archives, where I did this research; and my new employer Lakehead University.
I started work at the City Archives in 2008. As an institutional archives with responsibility for records management, most of the focus had been serving internal rather than external customers. We wanted to shift that over time, grow and strengthen community connections, and provide more service to the public, of course without neglecting the needs of City administration.
Over the years we started doing a lot of new things for the first time: building web exhibits, holding public tours, bringing our materials out to other public spaces, using social media, and posting large numbers of photos online.
All of this was done with the sort of basic assumption that I think a lot of us make in outreach. We will engage in a variety of projects, including public events, educational programs, and exhibits, which will presumably lead to an increase in traffic on our website, which will presumably lead to an increase in inperson visits and substantive research, which will lead to increased public support, which will lead to increased funding, which allows us to do even more of all of the above.
Once we get started on this cycle it will all keep building into a massive success.
But there are a lot of assumptions being made in this model. That one thing reliably leads to another, and that if the input is sufficient, the output will be rewarding. This idea that you’ll get results back proportionally to how hard you worked. I assumed these things at first, but each time I looked more closely at what was going on, I realized they weren’t accurate.
We need to look more closely at the connections between each of those steps, and I only cover some of them today, but each of those connections should be questioned and looked at critically.
The analysis I’ll be presenting is based on:
● In-person visitor statistics and retrieval statistics going back to 1996
● Website visits, going back to 2009
● Flickr views, starting November 2012
● Varied data from other institutions, as far back as 1985, most mid2000s
I won’t be talking methods today but I’m happy to discuss data gathering and analysis, and the results that didn’t make it into this presentation, with anyone afterwards.
As a side note, I was pretty excited to read the work done on new standards and metrics for archival access. I plan to incorporate that work into my assessment work in the future.
Okay, so what are some of the trends we are seeing? What do these numbers actually say?
In-person visits are declining somewhat over time, though this decline is gradual. This is monthly data from 1996 to the end of 2015.
This is supported by what’s going in other archives: most frequently but not always a slight decrease in in-person visits over the years.
On the other hand, there is an overall increase in distance research. While fewer people are attending in person, more are doing research over the phone or by email or on the web.
Website traffic shows an increase over time. That trend is small compared to the weekly variation, though, and some of that variation is seasonal, and some is due to spikes in interest resulting from outreach efforts. This graph shows seven years of traffic for the City Archives website. That big spike in the middle was from an exhibit launch that did very well.
Can the increase in web traffic over time be said to be due to increased outreach efforts, or is it just the side effect of a general increase in use of web resources by people in our community?
Well, here’s the Archives web traffic as a percentage of total City website traffic, 2009 to 2014. It’s pretty constant. The absolute numbers of people viewing archival documents online is increasing, no doubt. But this indicates that it might be driven by the web habits of the public, rather than being a triumph of archival outreach.
How about people viewing digitized photos on Flickr? The most obvious result is that the bulk of views are coming on days when the traffic just absolutely spikes. Those kinds of spikes are definitely event-driven: either a bunch of new photos are added, and all the new photos get checked out by our followers; or a link gets shared by someone influential, for example the Public Library or a local reporter, and that gets us new viewers and new followers.
Outside of those busy days there is a baseline, which has been slowly increasing over time as more photos are added. That baseline is mostly made up of people searching for particular types of images, much more so than people drawn to look at City Archives photographs specifically.
In fact, we’ve found that there’s very, very little cross-traffic between our website and our Flickr account. Few referrals, and busy days on Flickr don’t match the busy days on our website. Most people who visit our website don’t visit in person, and many of our in-person visitors haven’t looked at our website. These are mostly separate sets of people engaging with us in separate ways.
Web exhibit launches, especially when they are supported by media or public attention, lead to significant increases in web visitors. We find this is not just due to people viewing the exhibit itself, but then also viewing other exhibits, looking at the landing page, basically everything else. People who come to see the exhibit stay for other things. (However, they aren’t then spending time on Flickr, or coming in to visit us that week.)
The impact of all this, though, is very short term. We’re looking at one week, two weeks, of popularity before things go back to normal.
This type of pattern lets us determine a baseline, for what “normal” traffic would be. If outside of the initial launch, and outside of any special events, there’s a fairly stable rate of access, then you can measure the impact of special events on top of it. So in this case, this exhibit about former Mayors, has a baseline of between 75 and 150 views per week. So the launch of the exhibit, with over 3500 views, represents that much additional traffic, caused by novelty and by media attention.
One more interesting finding from things I’ve been working on: the average inperson visits for an archival institution are much more dependent on the population of the municipality in which they’re located than on the size of their collections. This slide has got the data from the other institutions I heard from.
It seems that: the more people who can get to the reading room without expense, the more people who will do so. This sounds obvious. But this is another way of finding a baseline against
which to measure success.
There are factors affecting your usage numbers that no amount of outreach activities can shift. How large a community you are working in. The materials you have in your collection: some documents will always attract more popular interest than others.
When you’re judging the success of a project, it’s important to judge it against your baseline or recent trends, and say “Are these results significant, for us? Did this project make a difference in how people found or used our archives?”
Once you can determine what the actual results of that outreach project were, what sorts of responses you got that you wouldn’t have otherwise, you can also contrast that with the amount of time, money, and effort that went into the project.
Evaluating the results of outreach allows us to look at projects and programs through the lens of cost versus results. And that’s kind of a scary thought in some ways, because it seems really wrong to base archival work primarily on financial priorities. But: there are costs to doing any outreach projects. Staff time and staff energy are the biggest asset we have, and should be spent wisely.
While nowhere I’ve worked has gone so far as to assign monetary value to visitors, we do pay attention to the amount of time it takes to, say, post 10 pictures on Flickr, or prepare a major exhibit, including supervision, editing, and dealing with systems. Online exhibits are not cheap to put together!
If that exhibit is seen by 400 people in the first month, but it cost $5000 in total staff time to prepare, then we’ve got a cost of $12.50 per viewer. If we reach 1000 people, then it’s $5 per viewer. So the larger the impact, the more cost effective we can say it was.
Providing access to historical records is our mandate, and we don’t ever expect to be a profit centre, so this seems fair. But what if it worked out to a cost of $50 per viewer, or even higher? Would you be comfortable investing in a project at that sort of cost? Would it be the best use of resources, when there’s so much else to do? At some point, you have to decide that the chosen strategy isn’t working, and try something else.
On the other hand, using Flickr to share images is averaging less than one penny per view. No infrastructure development, minimal staff time. Maybe the total impact is less, and some context is lost, but it’s hard to argue against that sort of efficiency.
So, what does this all mean for developing a good outreach strategy? There are implications for that “Master Plan” discussed earlier, mostly that it’s an incomplete way of looking at things. Doing more projects online doesn’t mean we’re necessarily increasing how much reference service we provide. Flickr traffic doesn’t drive web traffic, or vice versa, and neither of those is a driver for in-person visits, which are actually declining over time. And so far we have no way of knowing whether an increased public profile results in more public support.
The problem with what I was expecting when I started looking at all these results, was in assuming that outreach and access are one single, massive pipeline delivering records to people.
Thinking of everything as “access” oversimplifies what’s really going on. We’re actually dealing with multiple, very separate channels. Very different ways of approaching the archives, of learning about the archives, of accessing archival material.
There is next to no overlap between in-person visitors (many will check hours online but will not view either finding aids or web exhibits); many distance researchers (again, they find contact information but generally don’t check anything else); the people who spend time on the City Archives’ website (whoever they are, they are not the people visiting!); and the people viewing photographs on Flickr (they’re mostly coming from within Flickr, occasionally from social media, almost never from the website.)
It’s not that useful to think of outreach as increasing the total homogenous volume of people accessing archival records, even though if you add each of those separate numbers together that’s what it could look like. Instead we’re adding new channels, new types of access, that are bringing in different people, and the nature of those interactions is different.
We can take on projects that lead to more interactions with members of the public. We can do work that creates more diverse interactions with members of the public. We can dedicate our efforts to developing high-quality and meaningful interactions with members of the public. But outreach is not one big homogenous program which is leading us constantly to bigger and better things.
So, since I did a lot of this analysis in 2014, how has it been put to use?
Well, the cost-result calculation got some attention. That led us to focus less on complex and comprehensive exhibit building, and more on providing straightforward online access to commonly used records, like maps and bylaws.
Also, it was a great demonstration that even in a small institution, where our research visits number in the hundreds per year rather than month, keeping and understanding user data is still worthwhile.
And now I’ve been starting all this over again in a new job, one where there’s again no recent history of outreach work, and in this case very little measurement of use as well. We’ve started measuring and tracking access now, and we’ll be able to use metrics to evaluate projects going forward. I’m also approaching work with exhibits and social media with the same pragmatic perspective.
Most importantly, I’m keeping in mind that outreach speaks to different audiences through different projects, and I’m very aware that we can’t expect a one-size-fits-all approach to work.
I’ve spent a lot of time playing with numbers and graphs. That isn’t for everyone. If we look too closely at the numbers, get caught up in the ups and downs and final scores, there’s also a risk of losing track of what it is we really want to achieve.
Is outreach a means to an end, or an end in itself? Access to archival records is an end in itself. Outreach as a means of improving and increasing that access is a fundamental good. Outreach projects, especially tangible things that are created and have been seen by a measurable number of people, are a demonstration that we are present in the community and doing our work with a focus towards public service.
But we have to focus on the end result, increased access to archival records, and not on the projects themselves. Pointing to a display or a brochure or a Facebook page as a success in itself is not a good strategy. Instead, we need to point to the results of those projects, the increase in access, the quantity and quality of interactions. The evaluation we do tells us what’s working and what’s not. And we owe it to ourselves to look critically at our own work, our successes and failures, in order to learn and be better at all this in the future.
Slides (PDF): SAA2016Powerpoint_SJ
Using Google Analytics Data to Expand Discovery and Use of Digital Archival Content
Practical Technology for Archives
Talks about using Google Analytics to see how digital content is being discovered, and thereby get a better impression of who is actually using your site, and what they are interested in.
Considering Outreach Assessment: Strategies, Sample Scenarios, and a Call to Action
Shannon L. Farrell and Kristen Mastel
In The Library With The Lead Pipe
Discusses strategies for evaluating in-person events, focusing on goals and not just how many people showed up.