Month: January 2011

scholarship

Alternative metrics at ScienceOnline2011 and beyond

After a very interesting session on alternative metrics at ScienceOnline2011, I have been trying to figure out if I should write anything about it here.  My thoughts are rather scattered, incomplete and probably lacking in originality, but I decided to put something up anyway to help me sort through things.

100 Feet of tape measure
CC image courtesy of Flickr user karindalziel

The first thing I’m thinking about related to alternative metrics is “What problems are we trying to solve?”  I am very familiar with the criticisms of the impact factor, but I’m interested in returning to the basic questions.

What are researchers doing that some kind of metrics could help them with?

  • Find a job
  • Get tenure
  • Get promoted
  • Have their research read by a lot of folks
  • Publish in places that make them look smart
  • Quickly evaluate how smart other people are
  • Convince people to give them money for their research
  • Do more research

At the moment, the impact factor can affect all of these things, although it wasn’t developed to do that (see this article for a history of its development and some defense of the number).

What kind of quantitative information do we need to help researchers accomplish the things they want to do?

Various metrics have been proposed to help researchers:

  • rate journals (average quality)
  • rate individual articles
  • rate authors

I think the fundamental question comes back to how do these metrics help the researcher make decisions about their research and publication.

The second big question I have is whether or not the problem is all about the limitations of the impact factor or if the problem is with the academic culture that misuses existing metrics?  New metrics being proposed are not perfect, and I would argue that any quantitative measure of scholarship will have flaws (though perhaps not as many as the singular reliance on IF).

As more scholars do work outside of traditional peer review journals, I think tenure and promotion committees will feel more pressure to expand beyond simply looking at the impact factor.  We have some interesting examples of scholarship beyond the peer reviewed literature, and several new journals making the case for assessing individual articles rather than the journal as a whole.  These developments will also put

But those of us who are very connected to the online scholarly community of blogs, twitter feeds and social networks need to remember that most faculty are still not aware of or interested in what this community has to offer.  As a result, we can develop all of the alternative metrics we like, but until there is greater acceptance of these “alternative” forms of scholarship, I doubt that any alternative metric will be able to gain a prominent foothold.

At the ScienceOnline2011 conference, Jason Hoyt, Chief Scientists at Mendeley made the argument that we should put our energies into whatever alternative metric will bring down the impact factor.  I wonder if that is putting the cart before the horse.  Perhaps we need to get faculty (even those who have no interest in the blogosphere) to acknowledge the limitations of the impact factor first.

Of course, it is much easier to develop a quantitative measure of scientific impact than to change the culture of scholarly disciplines.

communication

Discovering the scientific conversation

I often like to think of science as a conversation.  It is a conversation that other folks need to be able to hear, so it needs to be discoverable.

We’ve come a long way since da Vinci wrote his notes in code.  Research results are regularly published as journal articles, and references and citations attempt to credit previous work.  The conversation of science could (at one point) be seen as the steady progression of peer-reviewed journal articles and technical comments, with some conference proceedings thrown in for good measure.

Conveniently, this was fairly easy (if expensive and time consuming) to access and preserve.  Publishers originally worked with print index makers and eventually digital database folks.  Conference abstracts were often preserved, even if the actual presentation wasn’t.  And each discipline typically had one primary source to find this information: GeoRef for the geologists or Chemical Abstracts for the chemists.

Things are changing.  And the ScienceOnline2011 conference provided a lot of examples of this new conversation in action.

The peer-reviewed journal article is no longer the only place where this conversation is taking place.  Scientists are commenting on and rating papers on publisher websites.  Scholars are making comments via twitter and friendfeed.  Bloggers are providing detailed (and informed) commentary on published papers, making suggestions for further research and trying to re-create published experiments.  Scientists are citing and archiving data that is stored all over the place.

So, how can researchers and student follow this conversation?

Just a few of problems:

  • Comments, ratings and supplemental material are usually not indexed in the traditional research databases we point students to.
  • Google is great at uncovering conference presentations posted on SlideShare or Google Docs, but not so great at making the connection between the presentation and the conference abstract.
  • If researchers access a journal article via an aggregator (not through the publishers website) they probably won’t have access to the supplemental material
  • Will the non-article material be preserved?
  • Will a published journal article link back to the Open Notebook that was used during the course of the experiment?  Will that notebook be preserved?
  • Most research databases and publisher websites don’t provide links to blog posts commenting about the article.

Is this a problem for researchers, or just for librarians and science historians?

I spend a lot of time in classrooms teaching students how to track citations forward and backward in time using tools such as Scopus and Google Scholar.  But if Scopus is stripping out citations to archived data, and if there is no connection to the blog post that sparked a whole new research direction, they aren’t seeing the whole story.

Is there a need for a more complicated discovery system that searches everything and makes the appropriate connections?  Is the semantic web a solution to these problems?

While I don’t know the answer, I will continue to look for ways to expose undergraduates to this exciting conversation of science.

citations

Citations, Data and ScienceOnline2011

Despite the vast array of challenges and problems with creating and tracking citations to journal articles, the scholarly publishing realm has developed (over the past 350 years) standards to deal with these things.  New concepts such as DOIs, an increase in the number of providers who track citations (Web of Knowledge, Scopus, Google Scholar), and tools to easily format citations have made all of this a bit easier.

Scholars are now facing new challenges in creating and tracking citations.  The types of material being cited are probably more varied than ever.  Scholars are citing archived data sets, websites that may not exist in few months (or years), multimedia, and perhaps even blog posts and tweets in addition to the traditional journal articles, books and technical reports.

At the Science Online 2011 conference, several speakers lead discussions that focused on the challenges and possible solutions to some of these new issues.

Jason Hoyt, Chief Scientist at Mendeley, discussed some of their new initiatives to track citations based on user libraries.  Since I don’t want to spread misinformation about the nature of these initiatives and I’m not entirely clear about them, you’ll just have to stay tuned for more information.

Martin Fenner discussed his work with project ORCID, which will be a publisher-independant tool to help with author disambiguation.

Overall, there was an interesting discussion about the nature of citation itself.  The way the metrics count it, a citation is a citation.  You get ‘credit’ for a citation even if the folks who cite you say that you are completely wrong.  Is there a way to use the semantic web to indicate how a citation is being used?  For example, Scopus indicates that Andrew Wakefield’s retracted paper about autism and vaccines has been cited 714 times since its publication, including almost 65 citations since the paper was retracted at the beginning of 2010.  Could there be a way to easily say how many of these citations say that Wakefield was wrong?

With all of these interesting advances, there are a lot of challenges.  Can the same set of metadata used to describe genetic data be used to describe high energy physics data?  Are we moving toward a future where scholarly metadata is exponentially more fuzzy than it is now?  Will standard procedures develop – is there an incentive for standard procedures develop?  Who will develop them?

I don’t know enough to even hazard a guess at the answer to these questions.  For a least a little while, before scientists, publishers and librarians work out the details, undergraduate students are going to be even more frustrated at citing material for their projects, especially due to varying faculty expectations.  The “How do you cite this?” questions at the reference desk will get much more complicated before they get any easier.

culture, workshop

Managing your scholarly identity

When someone Googles your name, do you know what they will find?  When a colleague, student or potential employer go searching for your scholarly record, will they find accurate information?  When you are looking for a collaborator, a reviewer or a potential hire what sources do you trust for reliable and up-to-date information about that scholar?

Have you Googled yourself lately?

Unfortunately, faculty websites and college faculty profiles can often be absent, out-of-date, or impossible to find.

Enter the database of scholars.  There are several types out there – those that require registration and constant maintenance by individual scholars, those that automatically pull data from other sources, and those that do a bit of both.

My college has recently acquired access to one of the latter, Scholar Universe.  SUNY has negotiated with Scholar Universe (normally a subscription database) to provide open searching of SUNY scholar profiles.  Check out my SUNY colleagues and especially my SUNY Geneseo colleagues.

Faculty at my institution are now confronted with their public profiles, and a renewed interest in making sure that the information available about them is accurate and complete.  Yesterday, in collaboration with the Office of Sponsored Research, we held a workshop for faculty on editing their Scholar Universe profiles and otherwise managing their scholarly identity.

So, what can an individual researcher do to take control of their scholarly identity?  Here are some of my thoughts:

First, know how others see you.  Google yourself.  Do vanity searches in the databases used in your discipline.  Are you happy with the results?  While a database might not list all of your publications (because of which journals they choose to include), is a list of your publications available online?

Second, if you see wrong information – correct it.  Is your webpage 8 years old?  Make a few updates.  Remove time sensitive stuff like office hours and course schedules so that it doesn’t get so easily out of date.  Add stuff that won’t get out of date like publications, current and prior affiliations, and expertise.  If you see wrong information in a database or on another website, try to correct it by contacting the editor of the site (of course, sometimes this just isn’t possible.)

Third, add to the body of scholarly information available about you.  Create profiles on Nature Network or Mendeley and include your list of publications.  Post a copy of your CV (if you don’t know how to post a document online, try using Google Docs to upload a copy to the web). Assuming you have permission to do so, upload a pre-print of your publications to your website, an institutional repository (ask your librarian) or a disciplinary repository.

Fourth, do what you can do help scholars find all of your publications in one place, especially if you have a common name.  Register with ResearcherID.com to collect all of your publications in one place, and make sure that you only have one identity on Scopus.

What else can a researcher do?  How do you manage your scholarly identity?

personal

Tools of the trade – How I get stuff done

For Christmas this year, I received a leather cover for the notebook I use at work, so I thought I’d share a little bit about the tools I use to get things done.

First, the analog stuff.

The Notabilia notebook with the leather cover I got for Christmas
The Notabilia notebook with the leather cover I got for Christmas
  • I use a Notabilia composition book from Levenger to keep all of my meeting notes and general ideas.  I’ve tried various online and software solutions to keeping notes, but I like having everything in one place, and the ability to (occasionally) go without a computer.  I like the fact that the pages in a composition book are bound, and the paper quality in this one is excellent.  I received a beautiful leather cover for this notebook for Christmas (Thanks, Mom and Dad!)

  • I get pretty picky about my pens.  I use black uniball vision elite pens for all of my note taking.

  • I keep a small moleskine notebook to help me organize my day.  Now, all of my meetings and classes are recorded in my online schedule, but I need some way to structure my “unscheduled” time (when I have some).  So in my small notebook I  just jot down which projects I will spend my time on that day.  Low tech, but useful.

Then, the digital stuff.

  • Oracle Calendar.  This is perhaps my least favorite online calendar, but it is the one my organization uses, so I live with it.  My colleagues can add meetings to my schedule and easily see if I’m busy (and I can do likewise).  Through some clunky third party software I can get this on my iPhone, but I really can’t wait until my organization drops this in favor of Google Calendar, or just about anything else.
  • TaskPaper.  My to-do list.  Lots of folks swear by the more complex personal project management software, but I really like the simplicity of this project, and I like the fact that I can sync it with my iPhone.
  • Google Docs.  Essential for working on documents on multiple computers and sharing stuff with other folks.  I get really annoyed now when folks just want to send .doc files back and forth via email.
  • Microsoft Word.  Having said that, I still use MS Word for a lot of my lesson planning.  I like the “Notebook” template that allows me to keep my library instruction lesson plans
  • Coda.  Great program for editing code.  I don’t do this as much as I used to, but it is a great program.
  • Adium.  Useful for getting stuff done with colleagues and students.  Also useful to avoid getting stuff done.
  • Tweetdeck.  Best way to keep up with folks on twitter.

What analog and digital tools allow you to get your work done?

Uncategorized

Strategies to help students find a project topic

Sometimes, one of the hardest parts of writing a term paper is just starting out: What on earth do I write about?

Faculty sometimes provide a very narrow set of topic choices, but students are often given wide latitude to select a topic of interest to them.  Then they just need to decide (a) What are they interested in? and (b) How can that be a term paper topic for my upper level science class?  Easy enough, right?  Not for many students.  I actually remember a mild panic setting in when I was an undergraduate given unlimited options about what to write about.

Thick arrow made from jigsaw puzzle pieces
Thick arrow made from jigsaw puzzle pieces. CC image courtesy of flickr user Horia Varlan

In a couple of disciplines, I have done informal sessions outlining strategies to help students find a topic.  I present some resources and give students time to poke around.  I’ve done this as a modified jigsaw activity with good results.  With the professor and the librarian present, the students can ask questions and get clarification about their topic or resource choices.  End of semester survey results indicate that this seems to be helpful for students.

All of the strategies I present basically show students a variety of resources to help spark their ideas – they don’t have to think of a topic off the top of their heads.

Examine Science Blogs and News Sites

Advantages: Articles are written in easy to understand prose and their brevity make it easy to scan multiple topics quickly.

Disadvantages: You will have to translate the topic idea from the news article/blog into the primary literature.

Look at Relevant Journal Table of Contents

Advantages: You are going directly to the primary literature, and once you find an article, expanding your search can be very easy by tracking down citations.

Disadvantages:  Article titles are notoriously difficult to comprehend for undergraduates.  Something that might be particularly interesting might be hidden behind overly complex scientific language.

Searching Relevant Databases

Advantages:  If you have a general idea of what you might want to do, this might be very useful for helping you narrow down your topic.  You also make a very quick leap to the scholarly literature.

Disadvantages:  A search for “the evolution of fish” might turn up so many results that you can be overwhelmed.  And scanning journal article titles can sometimes lead to more confusion, not less.

personal

Books I read in 2010

I read 31 books in 2010, just one more than last year.  As usual, the list is a mix of lighthearted fiction and some (slightly) more serious non-fiction.  I tend to prefer humorous books of all sorts, and the lovely combination of science, technology, personality and humor made Mary Roach’s Packing for Mars : the curious science of life in the void my favorite book this year.

The year started out rather juvenile with a quick read of the popular Twilight series.  I was feeding my daughter in a chair in my 12 year old niece’s room, and the only books within reach were Junie B Jones, Kindergartner, or Twilight.  I’m not convinced I made the right choice.  However, my knowledge of the books has allowed me to engage in some interesting conversations with my niece, which have been fun.

My final book of the year was the new ‘biography’ of cancer Emperor of All Maladies by Siddhartha Mukherjee.  The book was very engaging and thought provoking.  As a book that discusses some history of science, I appreciated the focus on scientific methodologies and the important role personality plays in scientific discovery.

As usual, I wish some of my favorite authors could write more – I eagerly await the next books by Christopher Moore, Jasper Fforde and Bill Bryson.

Here is the list.  Items marked by an asterisk (*) were read on the Kindle app for iPhone.