Electronic Publication and Academic Credentialing
Jeffrey Rydberg-Cox

Introduction

As we were forming our initial plans for the panel on Electronic Publication and Classics for the 2004 meeting of the APA, I was in the middle of my third year review at the University of Missouri at Kansas City, and I found it very interesting to see what pieces of my academic output were counting in my review and what pieces were not. This process sparked a desire to gain a more general understanding of what kinds of electronic publication counted and what kinds did not in our field. To try to gain a better understanding, we asked members of the APA to complete a short survey about whether or not electronic publication counted at their institution and their personal thoughts about the issue. This paper discusses some of the survey results and also briefly discusses the reasons that some things counted and some things did not.

At the outset, I would like to offer a few general thoughts about the promotion and tenure process -- at least as I have experienced it in my own department. I think that it is important to point out the vague nature of the entire promotion and tenure process even without considering the question of electronic publication. When we formed the survey, I think we were expecting firm and fixed guidelines in an area where specificity is hard to come by. My university does not offer a definitive statement of what types of publications or how many publications are required for tenure. What I have is Chancellor's memorandum #35 that says we are supposed to contribute substantially to scholarship in our field and develop both a national and international reputation. To try to determine what exactly this might mean, I have turned to my tenured colleagues who actually vote whether or not to tenure me, and I have received conflicting advice; one colleague tells me that a book is required while another tells me that a substantial number of articles is enough. Added to this, is the fact that during the past four years in my department, four people have gone up for tenure -- two have been promoted and two have been denied tenure. This means that the six untenured members of my department spend what can only be described as an unhealthy amount of time considering the records of those who have gone before us.

I do happen to know of a few colleagues who have firm and fixed guidelines. I have one acquaintance who has been told he needs two books, and he has been given a list of acceptable publishers and a second list of venues where his book must receive a favorable review in order to get tenure. I have another friend at an institution who has been told that six high-quality articles or a book will get tenure. These sorts of specific rules are definitely the exception. In our survey, 75% of respondents report that their department does not have specifically detailed guidelines about the quantity or type of publication that is required for promotion or tenure.

In place of firm guidelines, we have oral tradition. In conversations over the past three years, I have been told several pieces of information that I would love to have had spelled out for me formally as I planned my research agenda as a beginning assistant professor. For example, an edited collection of essays is not thought to have much value because the editor isn't personally contributing to scholarship, and his/her own contribution is not thought to have undergone any sort of peer review. Likewise, whole classes of publication such as encyclopedia articles, dictionary entries and book reviews simply aren't valued very highly. This is particularly interesting in our field because our respondents to our survey regularly cited the Bryn Mawr Classical Review as the type of electronic journal that should count because of its strong reputation and editorial practices. In my case, however, publication in BMCR would be a waste of time because that class of writing simply doesn't count whether in print or electronic form.

In summary, then, we are looking at an area that is vague and ill defined at best. Of course, I don't mean this to imply that electronic publication isn't viewed with more suspicion than other types of publication, but electronic publication is one type of publication among many whose value must be negotiated as part of the promotion and tenure process. In this negotiation, however, electronic publication can be made to count to the extent that it can situate itself in relationship to a few essential practices and buzzwords. In particular, I think we need to show that the work has been subject to peer review, that it contributes to knowledge, that it has had an impact on our field, and that it has the ability to attract extramural funding.

The Survey

With those preliminary thoughts and that thesis in place, we should turn to the survey itself. A request to fill out this survey was sent via e-mail to all members of the APA and a request was also sent out to the members of the classics e-mail discussion list. We accepted responses to the survey from late September until mid-December of 2003. We asked respondents to tell us if they taught at a public or private institution, the highest classics degree offered at their institution, the type of department they taught in, the size of their department, if electronic publication counted at their institution and if they personally believed it should count.

Figure 1: Survey Used to Gather Data For This Study

Overall, we had 116 responses to this survey. Responses mostly came from people in tenured or tenure track positions with only a few responses from adjunct or visiting faculty. Just under half of our respondents had served as chair of their department and a similar number had served on promotion and tenure committees.

We had a skew towards respondents from public institutions and a very strong skew towards members of Classics departments

Likewise, our responses were almost equally split between departments with either MA or PhD graduate programs and programs that only offered BAs, with only a few responses from programs that don't offer majors.

The responses to the questions about whether or not electronic publication counted at their institutions and if they personally believed that electronic publication should count were very surprising. A full 75% of respondents said that electronic publication would count for promotion and tenure at their institution, and only one person said that he/she did not personally believe that it should count. Lest that one person be cast as a Luddite, I should point out that the comments from that person were very insightful and eloquently discussed many of the issues I will discuss in a few minutes. I also found it quite surprising that there was no real distinction between public and private institutions or the types of degrees offered at their institutions.

As you can see, Ph.D-granting institutions are slightly less likely to accept electronic publication, but the difference is not significant. Indeed, this raises the question of whether we are looking at a debate that has already ended. I do not think this is the case; many of the people who said that electronic publication counted at their institution gave only a qualified ‘yes.’ Our largest response group said that it could count in certain qualified circumstances.

When Electronic Publication Can Count: Peer Review, Contributions to Knowledge, and Impact

I think that we see two phenomena at work in the fact that many people think that electronic publication can count only under certain conditions. One is a distinction between theory and practice. Electronic work has a certain popular appeal and also the appeal of being the wave of the future. People don't want to say that it shouldn't or doesn't count -- but at the same time they don't entirely believe it yet. One respondent points out that it counts at his institution, but they wouldn't advise a junior colleague to bank his/her career on this fact. The respondent wrote, “We would be happy to count peer-reviewed electronic publications, but we would not advise junior colleagues to assume that electronic publications would be fairly evaluated by college level committees composed of non-classicists.” I should point out that this tracks with advice I received as a graduate student when I was considering a dissertation project that would have involved statistical analysis of Greek. My advisor -- who himself was very favorably disposed to electronic projects -- told me that he thought an electronic project might hinder my chances at a long-term tenure-track position. Another respondent gave and took away in the same breath -- he or she said that it would count if it was published in a high-quality venue but that no such venues existed. This person wrote, “Electronic publications count if they are peer-reviewed, but no venue for electronic publications in my field carries as much weight as even mid-level print publication venue.” Finally, one person took the time-honored course of academics when faced with this sort of conflict; he/she blamed the provost and the dean, writing, “They are sometimes included in tenure dossiers but the perception is that many faculty and deans disdain them.”

Another very interesting fact emerged from the survey. While we would like to think that academic standards can keep up with changes in technology, the fact is that these issues of electronic publication in promotion and tenure decisions simply don't come up very often in many departments. There are two reasons for this. First, in the absence of clear statements and precedents of electronic publications' counting towards tenure, junior faculty shy away from presenting this sort of work in their dossiers. Second, many departments have simply been tenured in for a long time and there is little occasion for it to come up. As one respondent said, “We are all tenured, so it has not formally been an issue.” This more general phenomenon emerges clearly in other responses to the survey. One person wrote, “Frankly, it hasn't come up in any of the tenure deliberations I've been involved in since I joined the college's T and P committee.” Another respondent wrote, “The matter has not yet come up in dept, college, or university T/P committees -- I've served on all of these.”

I want to return now to the broad swath of responses that argued that electronic publication would count at their institution if certain conditions were met. Four themes were consistently mentioned in narrative responses to our survey. 60% of respondents said that only peer reviewed publication would count, 30% said that it needed to be high quality, and 15% said it needed to have some sort of impact on the field. In addition, a few people brought up important issues of longevity and continued accessibility.

I found it interesting as I was reading through the detailed responses to realize that all three of these key areas are themselves the subject of debate and negotiation. For example, in the area of peer review, there seems to be a spectrum where some people want the peer-review process to stand as a surrogate for their own judgment while others argue that peer review is one factor among many and that P&T committees should form their own detailed judgments about the quality of their work. At one end of this spectrum, we had comments such as:

“No publication should count unless it was subjected to a rigorous, objective process of peer review. This has to be a process that actually weeds out things that should not be published, and such weeding-out capabilities should be proven by the existence of a substantial number of submissions to that publication process that have been rejected. I know of no electronic publishing process that meets these criteria, and therefore I do not really think that any electronic publications should count for tenure and promotion at present; however I think the possibility should be theoretically available in case a properly rigorous process (i.e. one that really does reject things that are not good) emerges.”

At the other end of this spectrum, we found the following claim:

“Review committees should be bound by appropriate scholarly standards based on content, not form. If a committee is not sufficiently confident to form an independent judgment of a dossier regardless of medium, what right does it have to pass a verdict on a colleague's career?”

Likewise, the question of what scholarship is raises its head in this venue as in many others. One piece of my oral tradition that I did not mention earlier states that textbooks and pedagogical materials are of less value than other types of scholarship, and one of our respondents echoes this tradition when she or he claims that electronic publication should be evaluated as a subset of teaching rather than research. For example, one respondent wrote quite simply, “Websites should be considered as a part of teaching, not scholarship.” At the other end of the spectrum, we find arguments that teaching is a large part of what we do and consideration of innovative pedagogies should very much be considered as part of our research. As another respondent reported:

“It depends on what the purpose of what the publication is, and even what it means to be a Classicist. If the purpose is to say ‘more and more about less and less,’ and to limit and/or exclude scholars for providing a helpful service to students and others in perhaps allied academic disciplines, then electronic publication shouldn't ‘count.’ . . . Of course a really good ‘culture site’ on the WWW should not count ‘as much’ as an article in a refereed journal; but why shouldn't such helpfulness count for HALF an article, or even ONE THIRD of an article?”

Impact is to my mind one of the most nebulous criteria in the negotiations about the relative value of any publications. Is the number of citations of an article a valuable measure of its impact? If so, over what time period? If you assume it takes an article two years to be published, it will be difficult or impossible to measure the impact of anything I published after my first year. There are certainly many other ways to measure impact, but all of them are problematic; awards might be one measure, but only one or two books a year can win such awards and these certainly aren't the only books that have impact. Something like the TLG has obviously had an enormous impact on the way that we work, but the TLG itself is almost never cited. How do you interpret the number of hits that a web site generates? Perseus, for example, delivers eight million pages a month, but is this itself a measure of impact and does it form a standard that other electronic projects must match? More generally, how many hits are required before an electronic project is thought to have impact.

What emerges from the survey, then, is a portrait of the negotiation process, but few firm answers. We know that we want our scholarship to be high quality, reviewed by our peers, and to have an impact on the field, but we have widely varying definitions of what all of these qualities mean or how we should be measuring them. So the question of how to build a successful career in the midst of this is unclear. Electronic projects are certainly risky because they open up these questions in ways that they might not otherwise be addressed with print. The stakes involved in project selection are similarly high -- misunderstanding the rules of the game can derail or end an academic career.

Electronic Publications that ‘Counted’ and Those that Did Not

In my time as a post-doctoral researcher at Perseus, I engaged in two projects that were both integrated into the Perseus digital library as part of the electronic Greek and Latin lexica. The first project was a calculation of word collocations or words that are likely to appear together in Greek and Latin texts. Briefly, this is the sort of information that tells us in English that we are more likely to say “strong coffee” than “powerful coffee” or in Greek that the words angellos and pempo are very likely to appear in the same sentences. The second project involved a cluster analysis of the English definitions in the Greek and Latin lexicon in an attempt to locate words with similar definition in both the same language and in the opposite languages.

Figure 2: Similar Definitions and Word Collocation Tool in the Perseus Digital Library

The example in Figure 2 shows that readers who are interested in the word muthos might also want to look at other Greek words like phatis or epos, while in Latin they might want to investigate the word fabula. These tools are presented side by side in every definition in the Greek and Latin lexia in Perseus and viewed exactly the same number of times. However, the collocation tool counted in my third-year review while the similar definition tool did not. The primary reason for this was that that algorithm for the collocations was published in the journal Literary and Linguistic Computing -- a peer-reviewed journal with a strong reputation from Oxford University Press, while I never published the techniques underlying the similar definition tool. Although in many ways the similar definition tool is a more interesting and innovative application of computer technology, it is the collocation tool that carried weight in my evaluation.

Let me turn now to two other projects that look like more traditional journal articles that disappeared in surprising ways. The first is an article entitled “Knowledge Management in the Perseus Digital Library” that appeared in the on-line journal Ariadne. This is a journal sponsored by the Joint Information Services Commission in England and an important venue for describing digital library and digital humanities projects. At the same time, it is an entirely electronic journal, published only on-line. The second is an article that appeared in the proceedings of a national conference on digital library research entitled “Automatic Disambiguation of Latin Abbreviations in Early Modern Texts for Humanities Digital Libraries.” This conference requires submission of a full paper -- not just an abstract -- for peer review before it is accepted. All papers are published in print in conjunction with the conference, and in my mind this is an extremely important venue in my field. However, I have had papers such as this one disappear from summaries of research activity generated by the dean's office. Even though this paper that was subject to full review, it did not count because it appeared in “conference proceedings.” At the same time, the Ariadne article counted and was given quite a bit of weight because citations of this article appear in the Arts and Humanities Citation Index.

Finally, I should point out that in my career I have marked up somewhere around fifty texts, commentaries, and lexica for electronic publication in the Perseus digital library and on a project web site for my Cultural Heritage Language Technologies grant. Marking up these texts did not count at all -- but I should mention that the fact that I obtained extramural funding to mark up these texts counts quite heavily.

Charting A Path for the Future

In conclusion, my experience corresponds generally with the situation we found in our survey. Electronic publication can count if it can situate itself advantageously in relationship to practices and buzzwords like peer review, quality scholarship, impact, and extramural funding with the caveat that these practices are themselves subject to a great deal of negotiation. A final question still remains: What can we do to improve the general reputation of electronic work? First, we need to understand that we are playing with the electronic equivalent of incunabula and that negotiation is part of the game for the foreseeable future. It's hard to argue for the lasting value of a work when you are trying to explain that the P&T group will need Acrobat 6 instead of 5 and a browser with CSS support. Second, we need to be mindful of the practices associated with print publication in the field. All too often, we are infected with an ‘irrational exuberance’ and argue that anything electronic should count simply because it is electronic, and I think this reduces our credibility. Third, we need to be mindful of other print analogues in our institutional culture. If printed pedagogical materials don't count at our home institutions, we shouldn't argue that electronic ones do. Likewise, if we are editing texts, we should be mindful of the rewards, credits, and impact measures offered to editors of print journals.

This focus on the standards and practices of print is not, however, without problems. The need to assimilate our work to print practices runs the risk of stifling the innovative and creative possibilities that make work in an electronic environment so exciting -- at least for me. Looking just to the models of print leaves us without a way to evaluate large and important projects such as Perseus, the TLG, VRoma, The Stoa, or Sunoikisis. Editorship of a journal isn't an exact match, and extramural funding is a surrogate that dodges the important questions.

So what do I think should count? I want electronic projects that ask and answer interesting research questions, push the boundaries of what we can do, or change expectations of how we work together. I realize that this might be just as vague as the other standards we have been discussing, but perhaps they might be a way to evaluate new kinds of projects. The survey suggested another few starting points that are worth further consideration, including the APA's establishing a forum for peer- review of electronic projects, the APA's creating a policy statement about electronic publication, and a large journal such as TAPA simply deciding to go all electronic to pave the way for other journals to follow suit.

In conclusion, let me point out a final irony. I was very glad for the invitation to speak on this APA panel in January and even happier to learn that our talks will be included on the APA web site. I am, however, most pleased that our talks will also appear in a printed journal -- my own tenure file is due in June and I can always use one more publication!

Jeffrey Rydberg-Cox
Director, Classical Studies Program
Assistant Professor, Departments of English and Religious Studies,University of Missouri at Kansas City
Principal Investigator, Cultural Heritage Language Technologies Project
rydbergcoxj@umkc.edu