San Francisco Declaration on Research Assessment

An impact factor of an academic journal is the average number of citations to their recent published articles. For example, a journal with an impact factor of 10 receives, on average, 10 citations per each article it publishes. Of course, very highly cited articles greatly boost the factor, and by it’s nature, the impact factor cannot tell anything about the merits of individual articles published in the journal.

However, this is exactly what the impact factor is mostly used for. Time-stressed tenure and funding committees routinely use the impact factors of the journals (via their names) that an applicant has published in as a fast way of gauging the publication record of the applicant. Hence the self-perpetuating competition to publish in the “top” journals that is the mainstay of academic life. Incidentally, this practice has obvious  implications for the attractiveness of new open access journals.

As the editorial of Science puts it:

The impact factor, a number calculated annually for each scientific journal based on the average number of times its articles have been referenced in other articles, was never intended to be used to evaluate individual scientists, but rather as a measure of journal quality. However, it has been increasingly misused in this way, with scientists now being ranked by weighting each of their publications according to the impact factor of the journal in which it appeared.

Much (virtual) ink has been spilled to decry this rampant misuse of the impact factor. However, concrete actions to remedy the situation have been few and far between. Thus it may not be a surprise that the San Francisco Declaration on Research Assessment (DORA) has  gotten so much attention. As the description on their site states:

The San Francisco Declaration on Research Assessment (DORA), initiated by the American Society for Cell Biology (ASCB) together with a group of editors and publishers of scholarly journals, recognizes the need to improve the ways in which the outputs of scientific research are evaluated. The group met in December 2012 during the ASCB Annual Meeting in San Francisco and subsequently circulated a draft declaration among various stakeholders. DORA as it now stands has benefited from input by many of the original signers listed below. It is a worldwide initiative covering all scholarly disciplines. We encourage individuals and organizations who are concerned about the appropriate assessment of scientific research to sign DORA.

The Nature News report includes comments from the DORA chairman:

“We, the scientific community, are to blame — we created this mess, this perception that if you don’t publish inCellNature or Science, you won’t get a job,” says Stefano Bertuzzi, executive director of the American Society for Cell Biology (ACSB), who coordinated DORA after talks at the ACSB’s annual meeting last year. “The time is right for the scientific community to take control of this issue,” he says.

Their first and main recommendation is clear and striking – impact factors are declared unfit for duty:

Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.

Interestingly, Nature is not amongst the signatories:

Nature Publishing Group, which publishes this blog, has not signed DORA: Nature’s editor-in-chief, Philip Campbell, said that the group’s journals had published many editorials critical of excesses in the use of JIFs, “but the draft statement contained many specific elements, some of which were too sweeping for me or my colleagues to sign up to”.)

If journal impact factors are not to be used anymore, the way forward is seen to lie with more modern article-level metrics. Altmetric indicators are developing rapidly, and new research on their relationships to traditional metrics appears at an increasing pace.


Space Oddity… in space

So fricking awesome: a revised version of David Bowie’s Space Oddity, recorded by Commander Chris Hadfield on board the International Space Station.


Moving to Vienna for a postdoc (VP1)

From: http://www.traveldiscounters.ca/Travel-Vacations/Vienna-Austria-Travel.php

After finishing one’s PhD, it is typical that budding scientists spend a period doing research abroad. Cue Wikipedia:

Postdoctoral research is scholarly research conducted by a person who has completed doctoral studies. It is intended to further deepen expertise in a specialist subject, including integrating a team, acquiring novel skills and methods. Postdoctoral research is often considered essential while advancing the scholarly mission of the host institution; it is expected to produce relevant publications.

A single such period spent in one place is colloquially called “a postdoc”, and typically is one to two years in duration. It is common to spend several of these, often moving from place to place, accruing experience, publications, and applying for tenure track positions in ones native country (or anywhere). In practice, having postdoctoral experience from abroad is a de facto prerequisite for tenure positions in Finland, and in many other countries as well.

Well, I finished my PhD already in 2011, and have been doing a postdoc in the same university ever since. Mine was a exceptional case: I was involved in managing a couple of research projects, and responsible for a large international conference being held this June. After that is done, I’ll take a well-deserved summer holiday, and the move to Vienna, Austria for a two year postdoc. Luckily I have also gotten some research done, but I really look forward to being able to concentrate on it more.

I thought it would be interesting to share my experiences of the process. I’ve lived in Helsinki my whole life, and despite having traveled quite a bit, moving abroad is not an insignificant step. Especially to a country whose language I don’t speak (I had Russian instead of the typical French or German in school)! On top of that come considerations typical for my career stage: where to get funding, how to choose where to go (both professionally and personally), what to study (what is interesting to you vs. what gets funded), and so on. Luckily I was already able to secure a postdoc grant for the entire period, which will be the topic of my next post.

I am planning on writing a series of blog posts on all aspects of this transition, hopefully offering some useful advice for people who are pondering the same issues. I’ll mark the posts by a running “(VP#)” numbering in the titles for easy reference. If you have any particular aspect you would like to read about, please let me know!


Science Europe’s open access statement

The Science Europe foundation is an association of European research organisations whose objective is to strengthen collaboration between national research organisations throughout Europe. The most important Finnish research funder, Academy of Finland which currently only “recommends” open access, is a member. I recently personally heard from the Academy director Heikki Mannila that the they will closely follow the Science Europe’s decisions in future Academy policy-making.

Thus the recently announced “Science Europe Position Statement: Principles on the Transition to Open Access to Research Publications (April 2013)” is especially important for Finnish researchers. To quote from the document:

The benefits of Open Access are clear; furthermore, the technology available allows for a decisive move towards making Open Access a reality. The ultimate goal is to move to a new and sustainable system of scholarly communication of Open Access that guarantees the highest quality of publications and maximises the impact of research results. Science Europe Member Organisations acknowledge that the transition towards such a system presents challenges and that a common understanding of these challenges, and a collective approach to tackle them, is the most efficient way forward to accomplish the transition.

Open access advocate Ross Mounce warmly welcomed the statement on the The Open Knowledge Foundation blog, highlighting it’s rejection of hybrid open access – a counterpoint to some arguments that have been put forward. It will be very interesting to see if the Academy will interpret the position as a nudge towards implementing an actual mandate for Academy-funded research.


A few further bits and pieces:

  • I only just now ran into Cites & Insights: Crawford at Large, which is a web-based journal that is covering open access developments in great detail. The latest issue includes recent arguments for and against CC-BY licensing, a very nuanced discussion on various facets of the green vs. gold debate, and several items on article processing charges, repositories, the latest policy developments, etc, giving a very frank evaluation of the positions of various open access advocates.
  • Ars Technica published a rundown of the latest open access news.
  • A recent Scientific American blog post argues that Elite journals are going to hell in a handbasket since their proportion of the most highly cited articles is diminishing. Although the data is certainly interesting (and, coincidentally, showing that the plethora of Nature-branded journals is working well for NPG), I’m not quite convinced this is a huge trend at least yet.
  • Finally, as interviewed for Deutche Welle, the editor-in-chief of Nature Medicine raises the doubt that scientists are less inclined to do tough experiments for open access journals. Although some such effect might be real (scientists being human and swayed also by non-scientific motivations as is well known), it’s a bit doubtful whether this distinction has anything to do with open access per se. Furthermore, new mechanisms such as open peer review would likely have a big impact in this.

Update on open access in Finland

I’ve been recently involved in a few initiatives related to open access in Finland and our university, Aalto.

First, there is an initiative to build a common Aalto Current Research Information System (ACRIS), where all Aalto researchers will input their published research. The system will directly feed into Aaltodoc, our university’s publication archive. I was graciously given a chance for offering input to working groups for both projects, and in general the planned systems sound very sensible to me. However, the self-archiving workflow isn’t as streamlined as it should be before the ACRIS system is adopted in 2015, and in the meanwhile, it is expected that significant self-archiving will start taking place in 2014 due to funder mandates.  As we recently wrote with my colleague Jani Kotakoski recently in Signum, the University of Helsinki open access mandate has not worked as well as it should had, and the technically overly complicated self-archiving process is surely partly to blame. Hopefully some of the suggestions that I made (such as metadata autofilling by doi) can be implemented before the system comes to widespread use, and Aalto will have better levels of compliance (and awareness).

I’ve also heard that our university is finalizing a draft of it’s upcoming open access policy – I’ve been trying to get involved with the process, but so far the people making the decisions have decreed that the policy preparation should be done behind closed doors. I’m a bit peeved with this, and it certainly doesn’t help with the criticism that Aalto administration is too closed to grass-roots participation. As we also wrote in Signum, Aalto stands in a very good position to learn from the experiences of others. Furthermore, the timing is very opportune since the international political situation has really started to take shape in the past year. I’m personally strongly in favor of Aalto further adopting a strict mandate for open access self-archiving of all Aalto-funded research, and believe I have the arguments to back why this would beneficial for the University. Let’s hope they will open the process for comment in the draft stage.

Second, on March 19, there was the second meeting of the Tutkimuksen Tietoaineistot (TTA) -project looking at issues of open and linked data, metadata formats, long term data preservation etc. The second meeting was a bit technical for me, but it was good to hear what is being done on this front. Finally, the founding of an Open Science Finland working group was announced under the auspices of the Open Knowledge Finland association. There will be a joint meeting with the Finnish Open Access working group (FinnOA) on April 17, which I am going to attend.

Third, Bo-Christer Björk published an article on the rise of open access in the Finnish-language Tieteessä Tapahtuu magazine (loose translatable as “Happening in Science”). The article is a recap of the results of the peer-reviewed article Björk and his colleagues published in BMC Medicine last year, which made big impact when it came out (see e.g. the Guardian story I’ve linked to before). Nice to see more articles in Finnish on the topic that seems to be very much on the table nationally at the moment.

I’ll keep following the situation and will be sure to keep the blog updated on new developments.


There are also a couple of interesting links that I’d like to add here.

Michael Eisen, one of the more radical open access advocates, gives an overview of The Past, Present and Future of Scholarly Publishing on his blog. Randal Olson draws attention to a potentially ugly side of the open science movement, and Heather Morrison decries Elsevier’s licensing policies.

A couple of interesting podcasts recently came out. First, we have a pair of publishers justifying the costs incurred in publishing scientific research on The Chronicle of Higher Education. Then, we have an interview with Ross Mounce for the Journal of Ecology podcast. I had the pleasure of meeting Ross, who is a Panton Fellow at the Open Knowledge Foundation, at the OKFest in Helsinki. He’s very active in advocating open data and open access, and I warmly recommend subscribing to his blog at http://rossmounce.co.uk.

Finally, the Nature Publishing Group announced the upcoming launch of a data journal Scientific Data. While on the face of it the initiative is laudable and in line with open science ideals, some of its specifics (e.g. the article fees and licensing terms) have already generated a lot of critical discussion on the Open Science mailing list.

It will be interesting to see how the debate develops when we move closer to the launch.


Nature’s special issue on the future of publishing

Nature Magazine special issue: The Future of Publishing. Image credit: Brendan Monroe.

Nature Magazine’s latest issue is a special on the future of publishing. There’s a ton of great content, starting with the introduction and the Editorial:

New technologies allow a much greater and faster transition to a digital future, and this week’s special issue reveals that scientists are finding a multitude of ways to publish and access their research results. As this journal has noted before, the future of research literature will ideally be an amalgam of papers, data and software that interlinks with tools for analysis, annotation, visualization and citation. The need for common standards is as great as ever.

But it is demand, not supply, that will shape how scientists and publishers grasp these opportunities. For instance, a key reason that online open-access journals are now accepted as a mainstream (if still minority) method of publishing research is because of the mandates steadily introduced since 2001 by institutions and by research funders.

The issue features a story on the rise of predatory publishing, a truly outrageous case of using journal identity theft to scam authors, on the changing roles of libraries and open data, on licensing issues (paywalled, alas), on the future of scholarly communication (see also this comment), and many more.

However, my favorite article by far is Richard Van Noorden’s meticulously researched look at the true cost of science publishing. His article covers almost all the important emerging data on the many aspects of the issue, and addresses most of the recent debates surrounding the cost of article processing fees, true publication costs, the debate on the added value of traditional publishers, and licensing issues. Richard was also a guest on the latest Nature podcast.

I’ll highlight a few of the passages that most caught my eye, but I really recommend reading the whole thing. On the current debate:

The variance in prices is leading everyone involved to question the academic publishing establishment as never before. For researchers and funders, the issue is how much of their scant resources need to be spent on publishing, and what form that publishing will take. For publishers, it is whether their current business models are sustainable — and whether highly selective, expensive journals can survive and prosper in an open-access world.

On the current wide variety in the cost of publishing:

Data from the consulting firm Outsell in Burlingame, California, suggest that the science-publishing industry generated $9.4 billion in revenue in 2011 and published around 1.8 million English-language articles — an average revenue per article of roughly $5,000. Analysts estimate profit margins at 20–30% for the industry, so the average cost to the publisher of producing an article is likely to be around $3,500–4,000….
Outsell estimates that the average per-article charge for open-access publishers in 2011 was $660.

But Philip Campbell, editor-in-chief of Nature, estimates his journal’s internal costs at £20,000–30,000 ($30,000–40,000) per paper.

On publisher profits:

Elsevier’s reported margins are 37%, but financial analysts estimate them at 40–50% for the STM publishing division before tax. (Nature says that it will not disclose information on margins.) Profits can be made on the open-access side too: Hindawi made 50% profit on the articles it published last year, says Peters….
Commercial publishers are widely acknowledged to make larger profits than organizations run by academic institutions. A 2008 study by London-based Cambridge Economic Policy Associates estimated margins at 20% for society publishers, 25% for university publishers and 35% for commercial publishers.

On added value:

The key question is whether the extra effort adds useful value, says Timothy Gowers, a mathematician at the University of Cambridge, UK, who last year led a revolt against Elsevier. Would scientists’ appreciation for subscription journals hold up if costs were paid for by the authors, rather than spread among subscribers? “If you see it from the perspective of the publisher, you may feel quite hurt,” says Gowers. “You may feel that a lot of work you put in is not really appreciated by scientists. The real question is whether that work is needed, and that’s much less obvious.”

A more-expensive, more-selective journal should, in principle, generate greater prestige and impact. Yet in the open-access world, the higher-charging journals don’t reliably command the greatest citation-based influence, argues Jevin West, a biologist at the University of Washington in Seattle. Earlier this year, West released a free tool that researchers can use to evaluate the cost-effectiveness of open-access journals (see Nature http://doi.org/kwh; 2013).

On the path forward:

More than 60% of journals already allow authors to self-archive content that has been peer-reviewed and accepted for publication, says Stevan Harnad, a veteran open-access campaigner and cognitive scientist at the University of Quebec in Montreal, Canada. Most of the others ask authors to wait for a time (say, a year), before they archive their papers. However, the vast majority of authors don’t self-archive their manuscripts unless prompted by university or funder mandates.

As I said, really excellent reporting, and is one of those things that goes a long way to justify for me the current cost structure of the Nature Publishing Group. Go read it all.


Planck results on the CMB, and latest on the Higgs

Planck CMB. Copyright ESA and the Planck Collaboration.

The Planck Collaboration released the most detailed map yet of the Cosmic Microwave Background (CMB), as measured by the European Space Agency’s (ESA) Planck satellite, along with an arXiv release of 30 submitted research articles analyzing the data. What boggles my mind is that as far as I’ve understood, these measurement represent pretty much the best possible data that can be obtained from the CMB, limited not by instrumentation but by fundamental quantum effects.

The Starts with a Bang blog gave a very good primer on what was expected from the data on the eve of the release, which I recommend reading before checking out the results. (See also Sean Carroll’s anticipatory post on his blog.) For the results, you can read the official press release by ESA, the story by Nature News, or head back to Starts with a Bang for an excellent recap of the results:

So yes to inflation, no to gravitational waves from it.

Yes to three very light, standard-model neutrinos, no to any extras.

Yes to a slightly slower-expanding, older Universe, no to spatial curvature.

Yes to more dark matter and normal matter, yes also to a little less dark energy.

It thus seems the results were not very unexpected, although the corrections to the energy balance and age of the universe were perhaps a bit more significant than was expected. See also Peter Woit’s take on the implications for string theory (spoiler: no support whatsover).


The Moriond CMS update of the Higgs→γγ search. Tantalizing hints of disagreement with the Standard model predictions are significantly reduced with the newer, larger dataset. From Résonaances.

The other breaking recent news came from the Moriond particle physics conference (it seems physicists have a good thing going with these skiing conferences, as I personally also know :), which saw the release of the latest data from teams working on the Higgs boson at the Large Hadron Collider. The Quantum Diaries presents the results thus:

No more Higgs-like, Higgs-ish or even Higgsy boson. The CMS and ATLAS collaborations, the two large experiments operating at the Large Hadron Collider (LHC) at CERN, have now gathered sufficient evidence to say that the new boson discovered last summer is almost certainly “a” Higgs boson. Note that we are going to call it “a” Higgs boson and not “the” Higgs boson since we still need more data to determine what type of Higgs boson we have found. But all the analysis conducted so far strongly indicates that we are indeed dealing with a type of Higgs boson.

In a nutshell: it’s a Higgs boson (as opposed to anything else), and in all likelyhood the Higgs boson (fully as predicted by the Standard Model of particle physics), with no anomalous properties or new physics still in sight. However, as stated in the press release, more data needs to be collected and analyzed before this is conclusive. Unfortunately due to the LHC shut-down, we will have to wait two years for new more stringent data.

For the most important plots of the data, see the viXra log, or the slides from the relevant talk at Moriond. For analysis of the agreement with the Standard model, see the Resonaances blog. Although many theoreticians are dismayed by the lack of any evidence for supersymmetry or other popular theories beyond the Standard model, there are some who think this is a good sanity check and will direct research to more fruitful directions.


Follow

Get every new post delivered to your Inbox.

Join 752 other followers