About Viziometrics

Scientific results are communicated visually in the literature through diagrams, visualizations, and photographs. These information-dense objects have been largely ignored in bibliometrics and scientometrics studies when compared to citations and text. In this project, we use techniques from computer vision and machine learning to classify more than 8 million figures from PubMed into 5 figure types and study the resulting patterns of visual information as they relate to impact. We find that the distribution of figures and figure types in the literature has remained relatively constant over time, but can vary widely across field and topic. We find a significant correlation between scientific impact and the use of visual information, where higher impact papers tend to include more diagrams, and to a lesser extent more plots and photographs. To explore these results and other ways of extracting this visual information, we have built a visual browser to illustrate the concept and explore design alternatives for supporting viziometric analysis and organizing visual information. We use these results to articulate a new research agenda – viziometrics – to study the organization and presentation of visual information in the scientific literature.

Press and Recognition

• June 2016: The Economist has written a nice print piece on our arXiv paper.
• June 2016: Featured on LabWorm, a discovery platform that exposes top research tools with the goal of promoting a more open, collaborative and cutting edge science.
• June 2016: MIT Technology Review wrote a nice piece on our project: The First Visual Search Engine for Scientific Diagrams

Methods

We originally used patch-based machine vision techniques to classify figures by visualization type, achieving 91% accuracy on a test set with 5 categories – equations (394), photos (782), tables (436), visualizations (890), and diagrams (769). More recently, we have begun using deep learning to achieve higher quality results at the expense of training time. For the task of classifying millions of images that we extracted from source papers, we found approximate 35% of them contains multiple sub-figures. A dismantling algorithm we proposed in ICPRAM 2015 resolves this issue by parsing each composite figure into multiple sub-figures. The algorithm splits each composite figure into visual “tokens” recursively, classifies each token as either auxiliary (e.g., text fragments) or standalone figures, then merges the tokens recursively to reconstruct whole figures. The algorithm terminates when the reonstructed figure achieve a certain “completeness” score based on their types and positions. Using the results of the dismantler, we can more precisely classify the sub-figures.

Data

Our data for this research project comes from several sources. Currently, the prototype includes more than 8 million images from PubMed Central. We plan to add other data sets as they become available.

People

Students:

• Sean Yang, PhD Student, University of Washington, Electrical and Computer Engineering (lead)

Faculty:

• Bill Howe, Associate Professor, University of Washington, Information School
• Jevin D. West, Assistant Professor, University of Washington, Information School

Collaborators:

• Po-shen Lee, ExtraHop
• Maxim Grechkin, Facebook
• Hoifung Poon, Microsoft Research
• Chris Oh, HBC Digital
• Lia Kazakova, ExtraHop

Papers

• GraviTIE: Exploratory Analysis of Large-Scale Heterogeneous Image Collections
  The World Wide Web Conference 2019
  proceeding | video | bibtex

  @article{Yang2019GraviTIE,
                      title={GraviTIE: Exploratory Analysis of Large-Scale Heterogeneous Image Collections},
                      author={Yang, Sean and Rodriguez, Luke and West, Jevin D. and Howe, Bill},
                      journal={The Web Conf 2019},
                      year={2019}
                    }
              

• Central Figure Identification
  (In prep)
• MultiDEC: Multi-Modal Clustering of Image-Caption Pairs
  (In prep)
  arXiv | bibtex

  @article{yang2019multidec,
                    title={MultiDEC: Multi-Modal Clustering of Image-Caption Pairs},
                    author={Yang, Sean and Huang, Kuan-Hao and Howe, Bill},
                    journal={arXiv preprint arXiv:1901.01860},
                    year={2019}
                  }
                

• Viziometrics: Analyzing Visual Information in the Scientific Literature (2017)
  IEEE Transactions on Big Data
  pdf | bibtex

  @article{lee2016viziometrics,
                    author = {Lee, Poshen and West, Jevin and Howe, Bill},
                    title = {Viziometrics: Analyzing Visual Patterns in the Scientific Literature},
                    journal = {IEEE Transactions on Big Data},
                    year = {2017}
                  }
                

• EZLearn: Exploiting Organic Supervision in Large-Scale Data Annotation (2017)
  IJCAI-ECAI 2018
  pdf | bibtex

  @article{grechkin2017ezlearn,
                      title={Ezlearn: Exploiting organic supervision in large-scale data annotation},
                      author={Grechkin, Maxim and Poon, Hoifung and Howe, Bill},
                      journal={IJCAI-ECAI 2018},
                      year={2018}
                    }
                

• Deep Mapping of the Visual Literature (2017)
  Proceedings of the 26th International Conference on World Wide Web, BigScholar Workshop
  pdf | bibtex

  @article{howe2017deepmap,
                    author = {Howe, Bill and Lee, Poshen and Grechkin, Maxim and Yang, T. Sean and West, Jevin},
                    title = {Deep Mapping of the Visual Literature},
                    booktitle = {BigScholar Workshop (co-located at WWW)},
                    year = {2017}
                  }
                

• The Impact of Figure Type in The Biomedical Literature (2017)
  PLoS Biology (in prep)
  pdf | bibtex

  @article{lee2017plosbio,
                    author = {Lee, Poshen and West, Jevin and Howe, Bill},
                    title = {The impact of figure type in the biomedical literature},
                    booktitle = {},
                    year = {2017}
                  }
                

• PhyloParser: A Hybrid Algorithm for Extracting Phylogenies from Dendrograms (2017)
  The 14th IAPR International Conference on Document Analysis and Recognition
  pdf | bibtex

  @article{lee2017phyloparser,
                    author = {Lee, Poshen and Yang, T. Sean and West, Jevin and Howe, Bill},
                    title = {PhyloParser: A Hybrid Algorithm for Extracting Phylogenies from Dendrograms},
                    booktitle = {Document Analysis and Recognition (ICDAR), 2017 14th International Conference on},
                    year = {2017}
                  }
                

• Viziometrics: Identifying Central Figures in Scientific Papers (2017)
  IEEE Information Visualization conference (InfoVis) 2017
  pdf | video | bibtex

  @article{Kazakova2017,
                    author = {Kazakova, Olga and Lee, Poshen Lee and Oh, Bum Mook and Yang, T. Sean and West, Jevin and Howe, Bill},
                    title = {Viziometrics: Identifying Central Figures in Scientific Papers},
                    booktitle = {IEEE Information Visualization conference 2017},
                    year = {2017}
                  }
                

• VizioMetrix: A Platform for Analyzing the Visual Information in Big Scholarly Data (2016)
  Proceedings of the 25th International Conference on World Wide Web, BigScholar Workshop
  pdf | bibtex

  @inproceedings{lee2016viziometrix,
                  author = {Lee, Poshen and West, Jevin and Howe, Bill},
                  title = {VizioMetrix: A Platform for Analyzing the Visual Information in Big Scholarly Data},
                  booktitle = {BigScholar Workshop (co-located at WWW)},
                  year = {2016}
                }
                

• Dismantling Composite Visualizations in the Scientific Literature (2015)
  4th International Conference on Pattern Recognition Applications and Methods
  pdf | bibtex

  @inproceedings{lee2015dismantling,
                  author = {Lee, Poshen and Howe, Bill},
                  title = {Dismantling Composite Visualizations in the Scientific Literature},
                  booktitle = {4th International Conference on Pattern Recognition Applications and Methods (ICPRAM)},
                  year = {2015}
                }
                

Contact

If you have questions, please email

Sean Yang at tyyang38@uw.edu

Acknowledgements

This work supported in part by the Gordon and Betty Moore Foundation, the Alfred P. Sloan Foundation, the UW eScience Institute, and the University of Washington iSchool.