1. The HMO Research Network (HMORN) is a well established alliance of 18 research departments in the United States and Israel. Since 1994, the HMORN has conducted numerous multi-institutional studies encompassing a wide range of methodologies and scientific disciplines. As part of these studies, HMORN-based scientists have published papers that describe their approaches to solving many of the problems facing multi-institutional research. We present here a review of these papers in order to provide guidance to newer networks and to identify the challenges that continue to confront multi-institutional research. Careful attention to such organizational themes can help the HMORN and newer multi-institutional consortia develop into sustainable “learning research networks”. BACKGROUND Data model, quality and validity Range of data elements available, benefits of a research data warehouse, examples of studies supported by a data warehouse. HMORN virtual data warehouse (VDW) and its approach to standardizing variable names, labels, definitions and coding. Approaches adopted by HMORN studies and networks to standardize assessment of data quality. Proposed conceptual model for multisite data quality assessment. Site-level variation in variables; validity of critical data elements; performance of alternative algorithms for establishing diagnoses. Issues in research design (e.g., calculating associations; sample attrition; date accuracy; data completeness; representativeness). Governance and access to the network and its data Governance structure of the HMORN. Governance structure of HMORN scientific networks. Distributed research network data governance models. Progressively more sophisticated distributed query software. Protection of the privacy and safety of human subjects Delays due to variability in IRB review for multisite studies. Desire of investigators for facilitated multisite IRB review. HMORN strategy for IRB review of multisite project protocols. Administrative processes in research Need to continually improve the efficiency of administrative processes (e.g., contracting and financial closeout). Need to develop standardized approaches for establishment of data use and business associate agreements. Cancer Research Network assessment of administrative processes, and staff and PI satisfaction with those processes. Management of knowledge Needs for standardized tools to help conduct studies efficiently. Description of www.researchtoolkit.org resource compendium.www.researchtoolkit.org Relationships with host organizations Interest of HMO leaders, clinicians, and members in participating in traditional or cluster randomized trials. Faster translation of research into practice as a potential advantage of HMORN-based research. Describing and Optimizing Research Collaborations: A Review of the HMORN Literature John F. Steiner, MD, MPH and Andrea Paolino, MA (Kaiser Permanente Institute for Health Research - Denver, CO) Eric B. Larson, MD, MPH and Ella E. Thompson, BS (Group Health Research Institute - Seattle, WA) Four papers were purely conceptual, 22 were descriptive, and 26 included an analytic component to assess the performance of some aspect of the HMORN. Of the 52 items, the primary themes were organization, quality, or the validity of data elements (30), data and network governance (11), human subjects privacy and safety (4), the relationship between the research enterprise and the “host” health care system (4), knowledge management (2), and business processes (1). RESULTS COMMON THEMES Commissioned by the AcademyHealth Electronic Data Methods (EDM) Forum, a project supported by the Agency for Healthcare Research and Quality (AHRQ) through the American Recovery & Reinvestment Act of 2009, Grant U13 HS19564-01 ACKNOWLEDGEMENTS We reviewed 52 publications by HMORN investigators through December, 2012 that described general issues conducting research within the Network. Two reviewers independently reviewed and assigned to each paper a single purpose (conceptual, descriptive, or analytic), and one or more of six primary and secondary themes: (a) organization, quality, or the validity of data elements; (b) data and network governance; (c) human subjects privacy and safety; (d) the relationship between the research enterprise and the “host” health care system; (e) knowledge management; and (f) business processes. Reviewers resolved differences in their ratings by consensus. METHODS The HMORN must evaluate its processes by developing standard metrics to compare performance across sites, and demonstrate the ability to conduct research better, cheaper and faster than competitors. Work already underway to address gaps in the literature and document HMORN research efficiencies include: Developing targeted publications to close key gaps. Developing and capturing IRB and contracting efficiency metrics. Improving knowledge management tools and approaches. Collecting operational benchmarks for research center productivity and efficiency to support site-level quality improvement efforts. A LEARNING RESEARCH NETWORK THE STATE OF THE LITERATURE Our review identified numerous gaps in the published literature. Such gaps in the literature exist for numerous reasons. Incentive is to publish scientific content, not research processes. Few forums or opportunities exist to publish research efficiencies. Increasingly limited funding exists for infrastructure related work. HMORN’s informal nature and site variation make standardization challenging. GAPS IN THE LITERATURE ANALYTIC PAPERS DESCRIPTIVE PAPERS CONCEPTUAL PAPERS The HMORN has made major contributions to the literature on the development of common data models and strategies for data governance. In some areas the HMORN has conducted substantial but unpublished work (e.g., contracting and IRB efficiencies, costs to build and develop a site VDW, data quality improvement processes). In other areas little has been done (e.g., management of knowledge about successful strategies for conducting research, documentation of successful research translation locally or across sites). No multi- institutional networks have systematically described their culture or assessed the complex effects of culture on scientific productivity or efficiency.