Jul
22
The Science of Measurement
Filed Under Big Data, Blog, Education, healthcare, Industry Analysis, internet, Publishing, Reed Elsevier, Search, semantic web, social media, STM, Thomson, Uncategorized, Workflow | 1 Comment
This is the third attempt in a week to try to record the thinking that initiated the first one and pervaded the second . So here goes : ” Science is based upon measurement and evaluation , yet the activities of scientists have themselves been less measured and evaluated than the subjects of their research . ” In a society that now seeks the ROI of everything , this is becoming an important consideration . In the past we have been happy to measure secondary effects , like the shadow on the wall called “impact factor ” , when it came to measuring and evaluating ” good science ” . Now we can see clearly who is reading what and the way they rate it ( look only at Mendeley and ReadCube) , and what they say about it on Twitter and the blogs , we have a much broader measure of what is influential and effective . The market leaders in measurement a decade ago were Thomson (ISI ) , using the outstanding heritage of Eugene Garfield . They were followed by Elsevier , who today, by way of Scopus and its offshoots , probably match them in some ways and exceed them in others . Today , these players find themselves in a very competitive market space , and one in which pressure is mounting . Science will be deluged by data unless someone can signpost high quality quickly , and use it in filters to protect users from the unnecessary , while enabling everything to stay available to allow some people to search totality .
I started to get interested in this last year , when the word “alt-metrics ” first showed up . A PLoS blog by Jan Leloup in November 2011 asked for data :
“We seek high quality submissions that advance the understanding of the efficacy of altmetrics, addressing research areas including:
- Validated new metrics based on social media.
- Tracking science communication on the Web.
- Relation between traditional metrics and altmetrics including validation and correlation.
- The relationship between peer review and altmetrics.
- Evaluated tools for gathering, analyzing, or disseminating altmetrics “
So a wide range of new measuring points is required alongside new techniques for evaluating data about measurement gathered from a very wide variety of sources. And what is “altmetrics ” ? Simply the growing business of using social media data collection as a new evaluation point in order to triangulate measurements that point to the relative importance of various scientific inputs . Here the founders make the point at www.altmetrics.org:
“altmetrics is the creation and study of new metrics based on the Social Web for analyzing, and informing scholarship.
Our vision is summarized in:
“Altmetrics expand our view of what impact looks like, but also of what’s making the impact. This matters because expressions of scholarship are becoming more diverse. Articles are increasingly joined by:
- The sharing of “raw science” like datasets, code, and experimental designs
- Semantic publishing or “nanopublication,” where the citeable unit is an argument or passage rather than entire article.
- Widespread self-publishing via blogging, microblogging, and comments or annotations on existing work.
Because altmetrics are themselves diverse, they’re great for measuring impact in this diverse scholarly ecosystem. In fact, altmetrics will be essential to sift these new forms, since they’re outside the scope of traditional filters. This diversity can also help in measuring the aggregate impact of the research enterprise itself. ”
So a new science of measurement and evaluation is being born , and , as it emerges , others begin to see ways of commercialising it . And rightly so , since without some competition here progress will be slow . The leader at present is a young London start-up called , wisely , Altmetric . It has created an algorithm, encased it in a brightly coloured “doughnut” with at-a-glance scoring, and its first implementation is on PLoS articles . I almost hesitate to write that it is a recent investment of Macmillan Global Science and Education’s Digital Science subsidiary , since they seem to crop up so often in these pages . But it is also certainly true that if this observant management has noted the trend then others have as well . Watch out for a crop of start-ups here , and the rapid evolution of new algorithms .
Which really brings me back to the conclusion already written in my previous pieces but not fully drawn out . Measurement and Evaluation – M&E – is the content layer above metadata in our content stack . It has the potential to stop us from drowning in our own productivity . It will have high value in every content vertical , not just in science . Readers will increasingly expect signposts and scores so that they do not waste their time . And more importantly than anything else, those who buy data in any form will need to predict the return that they are likely to get in order to buy with security . They will not get their buying decisions right , of course , but the altmetrics will enable them to defend themselves to budget officers , taxpayers , and you and I wen we complain that so much funding is wasted !
Jul
18
Totally New Directions for Science Publishing
Filed Under B2B, Big Data, Blog, eLearning, healthcare, Industry Analysis, internet, Publishing, Reed Elsevier, Search, semantic web, STM, Workflow | 1 Comment
It was inevitable that some readers of my piece on Gold OA earlier this week would come back and say that I have grown too fond of defining what won’t work and should become more proactive about stating the new directions. Fair cop. Here then are two “assertions” about that future for science journal publishers which include areas in which “traditional” text-based “publishing” has only the slightest current experience and skills base, yet which will be vitally significant for the industry five years out. Both fit into a vision of scholarly communication, and the evolution of the publisher’s role away from primary publishing (which will become the prerogative of librarians and repositories) and into workflow management solutions and the quality of content within process.
My two candidates for step-change status are:
1. The evolution of video into an accompanying feature and then into the vital core reporting medium for scientific research reporting.
2. The development of robust and auditable techniques for evaluating the impacts of content on research, creating measures for ROI in content purchasing, and fresh, searchable data derived from the evaluation of usage. This data, along with content metadata, will be more valuable to players in this market than the underlying content on which it rests.
Lets start with the first issue. I am fed up with being told that science and scientists are too dull or too complex for video. Too dull? Just go and play these two minutes of an interview with John Maynard Smith, the great biologist, on Vitek Tracz’s pioneering site Web of Stories (http://www.webofstories.com/play/7277?o=MS) and try to maintain that view. And this site has excellent metadata, as does the video-based Journal of Visual Experimentation (JoVE) which announces its extension this month to covering experimental reporting in physics and engineering as well as the life sciences (www.jove.com/about/press-releases). Note that both of these sites set a premium upon narrative, and recall the narrative argument in my recent piece on next generation learning (After the Textbook is over... 3 June 2012) which was demonstrated in some wonderful transmedia software (http://www.inthetelling.com/tellit.html). Once again this demonstrates that video is quite capable of forming the narrative stem onto which links, citation, indexation, abstracts and other aids to discovery and navigation can be attached. Indeed, the text can be attached, along with demos and lectures and evidential data. Video file sharing is notoriously easy in the broadband world. Some academics will complain that they lack video story-telling skills, and this in turn may be something that publishers can add to the APC – as long as they acquire those skills themselves in time!
And then there is data. I have thundered on about evidential data and the importance of using the article as the routefinder that leads researchers to the statistics, or the findings, or the software used in the analysis. And we have all talked endlessly about metadata searching, about applications of Big Data in science and about data analytics. But I am moving to the view that we are crucially underplaying the importance of another sort of data, which we used to characterize as “usage data” and wonder whether it was going to become significantly exploitable. The CIBER team have long warned about the underuse of usage logs, but the force of the point has been increasingly brought home to me by an appreciation of what excellent data output can be derived from interfaces like Mendeley or ReadCube. We now begin to appreciate almost for the first time what usage patterns can be mapped – and what they mean. This is important for researchers, and vital for publishers. Users will rightly demand this form of data analysis, and will become increasingly interested in what, of the mass of data that they buy access to, is effective and cost-effective. This will start at the sharp end, in areas like drug discovery, but will grow into a habit of mind as data overload becomes ever more daunting. Concentrating purchasing policies on data that can be demonstrated to support improved decision making or better compliance or increased productivity will drive us to collect and analyse our user data to demonstrate that what we have makes a difference. And as we are driven this way we will get deeper and deeper into examining what users do with our data, and we will be surprised by how much we can track and know. And that knowledge will form another layer in our content stack, alongside metadata itself.
This game is already afoot in the biopharma sector. Eight weeks ago Relay Technology Management (http://relaytm.com) launched a “real-time Business Intelligence and Data Visualization Solution” for life sciences. Building on their RVI (Relative Value Index) formula, this new BD Live! (Business Development Live!) construction demonstrates some of the ways in which scientists and researchers in the future will want to have their data assets assessed – and the ROI of their purchases demonstrated. It is probably no accident then that Nature Publishing Group made an investment in Relay at the end of last year.
« go back — keep looking »