"Implementing the New ERIC: What the Future Holds for Indexed Access to Research"

A panel presented April 12, 2005
American Educational Research Association Annual Meeting
Montreal Quebec
This is the text of the presentation by Kate Corby

The other speakers were Luna Levinson, Director of ERIC; Lawrence Rudner, former Director on the Clearinghouse on Assessment and Evaluation and current member of the ERIC Steering Committee, and Adrianna Kezar, former Director of the ERIC Clearinghouse on Higher Education, now at the University of Southern California. A summary of the other presentations is available at: http://www.lib.msu.edu/corby/education/eric/openletters.htm

I have been asked to give a library practitioner’s view of the current ERIC situation. As you might expect, library professionals have been very concerned about changes to ERIC. Since other panelists will bring a bigger picture view of the situation,  I’ve decided to focus my remarks on the database product.

The nitty-gritty of how a database is constructed is significant.  Finding preceding scholarship reliably is transparently important to all researchers.  Education Index, now called Education Abstracts began indexing journals in education in 1929.  By the time ERIC came along in 1966 Education Index was the established standard in journal indexing for education.  Yet less than a decade later, ERIC’s Current Index to Journals in Education (CIJE) was the premier journal index in the field.  Why?  Abstracts were one factor, CIJE had them and Education Index did not, but it is important to remember that in the days of print indexes, abstracts meant users had to look everything up in two places.   ERIC also led the way in electronic searching, which revolutionized literature searching.  Backfiles may also have been an issue, until this year, the electronic version of Education Index, now called Education Abstracts, went back only to 1984. 

I believe the biggest reason ERIC moved so quickly to the fore was the consistent specific indexing. Because searchers got so much better results from an ERIC search, many thought ERIC indexed more journal articles than Education Index. That is not true, ERIC scanned more journals, and also included research reports – the ERIC documents – but ERIC indexed articles selectively.  For most journals covered by ERIC, the database didn’t index every article in a journal issue, only selected articles were included. Right up to December of 2003, Education Abstracts was indexing more articles per year than ERIC.  For much of their 35 years, ERIC was also several months slower to index materials than Education Abstracts.  Yet nearly everyone preferred ERIC.  It was the quality of the indexing.

Here’s an illustrative example. I found an item published by the Association for Supervision and Curriculum Development that was indexed in both Education Abstracts and ERIC.  It is not a journal article, it is a book.  It was recently issued in a second edition and was once again picked up by both indexes.  All four include the basics, title, author, and publication details.  The details of producer added indexing are revealing:

Notice ERIC’s indexing of the 1996 edition, a significantly longer abstract and 13 subject headings.  That’s a lot, but this is a book and the indexing covers all the subjects it discusses.  Specific indexing, not just two or three broad headings.  It is dangerous to extrapolate too heavily from one instance, but from long experience I can assure you these differences are typical.  ERIC indexing was superior to Education Abstracts, making it easier to find relevant items in the ERIC database. 

So what about the second edition indexed in the final few months of the former ERIC’s life?  I talked with a contact on the contractor side of the ERIC program.  She theorized that this, and several other poorly constructed entries I pointed out, was the result of rushed work by the depleted clearinghouse staffs in the final days of their contracts.   This citation is at least accurate; I found several from the same time period that were not.  The contractor has promised to look at the issue but told me frankly that it was unlikely to be fixed until they are on schedule for the new product.

Since no new content has been added to the database since July of 2004, we are left to speculate about the features of the forthcoming product from the language of the Statement of Work.  In doing so I, and my colleagues have been fortunate to have several opportunities meet with  representatives of the ERIC Program,.  Both Dr. Levinson and others at the Department of Education and representatives of the contractor, Computer Sciences Corporation have been very accessible to us.  They don’t always have answers but they do attempt to respond.  The Contractor is sponsoring a booth in the AERA exhibits area, introducing the new beta version of an online paper submission program.   They also came in force to the January, American Library Association meeting where we held the first ERIC User Group meeting.  The User Group is not strictly a library effort. A representative of AERA has contacted us to get involved.  We hope to provide an ongoing mechanism for all ERIC users to offer ideas and input to the development process.

The Statement of Work, outlining the new product contained a detailed.  At the time the Draft Statement was issued many of us argued that the timetable was unrealistically short, but the delay in awarding the contract caused the timetable to be shortened even more.  We should have had a list of journals to be indexed by mid November 2004.  I am told that is now in the final approval process and should be out within days.  New indexing was to have started being available in December; it has yet to begin. The 2004 gap, caused by the changeover was to have been eliminated by the end of January 2005.  Both the contractor and the ERIC office have been frank about the missed deadlines, but have explicitly decided not to offer a revised timetable.  

Many people were concerned that these changes were coming in tandem with a move to emphasize “scientifically based research” by the Department of Education.  Some of the information that circulated, especially in the spring and summer of 2003 when this plan was still in draft form, indicated that officials in the Department were concerned about the fact that ERIC contained non-peer reviewed materials.  Librarians are great believers in having information available.  We jumped on this concern with enthusiasm.  The Department has consistently allayed our fears.  Dr. Levinson assures us that the gray literature will remain core to ERIC’s mission.  One of the few changes between the draft and final version of the Statement of Work expanded the coverage of the database by mandating that it include “international materials written in English. "  Representatives of the contractor have also said that they are hoping to expand coverage not limit it.  One change -- ERIC will index education journals cover-to-cover, no more selective coverage. 

The response to Federal concerns about the quality of items indexed in ERIC has so far focused on labeling.  The What Works Clearinghouse started the ball rolling in this direction. Its whole purpose is to review completed research and assess its quality.  ERIC developers plan changes to the database to indicate quality.  Journal article records will indicate whether the journal is peer reviewed.  Adding such labels is not straightforward.  I spoke with a representative of ProQuest a major provider of journal indexing software.  ProQuest labels materials it indexes for its core product and includes scholarly peer reviewed journals as one type.  They freely admit that it has caused problems, publishers have different definitions of what constitutes a peer-reviewed publication and users have objected to the designations for individual titles.  But overall, the system has been a big plus, particularly useful to newer users who are not yet familiar with the publications in a discipline.  The ERIC Statement of Work also raised the possibility of using indicators of quality like acceptance rate, circulation, etc.  As long as the database stays inclusive and does not force users to pick a particular category, the use of such labels should not be a detriment for most users.

The quality of future indexing, as illustrated by the example I gave above, is also a big concern.  The Statement of Work talked of author-supplied abstracts and indexing and automatically generated indexing.   Since the quality of the index is a large part of what makes ERIC so valuable, librarian’s response to the proposals included concerns about too great an emphasis on timesavings and too little emphasis on quality.  This was another area where the language changed between draft and final statement.  The final version included more detail about index quality, specifying accuracy and consistency. 

The Statement of Work also discussed reviewing the one of the lynchpins of index quality, the Thesaurus.  The consistent use of descriptors to provide specific indexing in the ERIC database has been a huge plus.  Last fall changes were made to the ERIC Thesaurus that had the effect of flattening some of the subject hierarchies. Most of the changes were in subjects that are only peripherally related to education or are related to subjects of instruction within schools.  One aspect of these changes that I found troubling is that they increased the Thesaurus’ complexity, and violated its internal logic. 

All good thesauri list accepted terms and unaccepted terms that refer users to the proper place.  Think of the yellow pages -- if you look up lawyers it says see attorneys.  Then at the spot you’ve been referred to you will get a list of related suggestions some more specific some more general. Most of the changes to the ERIC Thesaurus eliminate a term as a descriptor and direct you to use a different term, but often of the terms they refer users to are only peripherally related to the original term.  As if in recognition of the logical error, the listing in the Thesaurus keeps all the related terms that had been associated with the former descriptor.  For example:  Condom which tells users to Use Term: Sexually Transmitted Diseases; clearly not a synonym for Condom then goes on to suggest Related Terms: Acquired Immunodeficiency Syndrome (2004); Contraception; Disease Control; Health Promotion; Pregnancy; Sex Education; Sexual Abstinence(2004). 

Condoms (2004)

using Sexually Transmitted Diseases as a search criteria

It turns out that a number of these “Related Terms” were also made “Synonyms” with these changes, so if you try to pursue using Sexual Abstinence it will say Use Term: Sexuality, and give you another list of related terms, some of them also dead ends.  Even with out the circular references it is simply confusing to call words “Synonyms” when they are not. 

 This referral was not done consistently.   Adverbs now instructs users to "Use Term: Form Classes (Languages)" with no linkage to the previously related terms of Syntax, Sentence Structure, or Vocabulary.

Adverbs (2004)

using Form Classes (Languages) as a search criteria

Also introduced was the concept of “USEAND.”  Athletic Equipment used to be a descriptor, now searchers are instructed to “USEAND.” That is, to construct the search “Athletics and Equipment,” combining two descriptors. 

Athletic Equipment (2004)

using Athletic Equipment (2004) as a search criteria

This new “USEAND” designation has also been used inconsistently. Bilingual Special Education now refers users to Special Education.  Why is it not a “USEAND” to construct the search “Bilingualism and Special Education?”  Other Thesaurus entries allow users to click on a button to execute the search, but that does not work with “USEAND” even though the button is present.

These are major changes to the Thesaurus. Today’s powerful technologies may facilitate a more robust Thesaurus structure than print allowed, but certainly such a significant structural change should have been heralded and explained

I pointed out one of the circular dead end situations to the Contractor and was told it was a mistake, but it hasn’t been corrected and the response “Let me know if you find more” was not particularly comforting.  I have found several others, but there were almost 3,000 changes, so it is not reasonable to expect users to report them all. 

Online full text is one of the reasons ERIC was changed.  One paper I read suggested that eliminating the Clearinghouses freed up resources so that more online full text of ERIC materials could be supplied to all.  Full text what everyone wants, even librarians, although librarians are perhaps a bit skeptical about providing full text inexpensively because we have been on the front lines of trying to keep up with online journal access for several years.  The situation has been chaotic with vendor changes, disappearing content, and licensing restrictions.

While keeping up with electronic content can be a real challenge, what is more of a concern is the risk of going to only one source.   In the old ERIC model there were microfiche collections everywhere.  If your copy got lost we’d lend you ours.  No one likes to use microfiche, but technology is catching up.  MSU recently bought a machine that reads microfiche and converts it to an electronic file.  Meanwhile schools that cancelled their microfiche order and relied on the old ERIC online product called E*Subscribe recently lost access to about 3,900 ERIC documents that, because of licensing agreement limitations, could not be brought over as electronic documents into the new ERIC system.  Libraries have learned this lesson repeatedly on a number of fronts.  Vendors pull content for many reasons: copyright disputes, retractions, age, lack of use…  If we have not archived or backed up our access with other formats, it is lost. 

Federal law protects access to older online federal government material even when it is no longer supported by administration policy.  But ERIC documents are by-and-large not government produced material.  So this improvement of free full text for all on the web comes at a significant price.  There is only one archive.  If that archive fails in any way, we all lose.

Summing up, I’d say the preliminary message from the new ERIC product, is “just wait.”  There’s going to be lots of new content, we’re just a little behind.  We made a few mistakes with the Thesaurus, we’ll clean them up later.  There seem to be some problems with the last few entries to the database, we’ll look at them when we get caught up.  Everyone who’s ever been part of a major project has probably said the same thing -- we can’t do everything at once; we’re prioritizing.  I can be understanding about that and still be a little bit concerned that thus far, in every step that impacts database content, they have not taken the time to get it right

The library community has also been concerned about the loss of information resources that resulted from the closings of the Clearinghouses.  In January I compiled a table listing the current status of the former Clearinghouse resources.  It is on the web at http://www.lib.msu.edu/corby/education/eric/January2005Clearinghouse.doc   I’ve met with several different groups of librarians in the past two years.  We are trying to find ways to work together to replace the centralized structure that ERIC provided and that was so effective in nurturing innovation over the last 35+ years.   We’ve done some work with the former AskERIC, now called Educators Reference Desk. http://www.eduref.org/  So far we have identified volunteers to maintain the web guides,  but will need to find funding before we can bring back the question answering service.  We are also hoping to put together coalitions that will work on new projects, but everything seems to be moving forward very slowly.