0
Original Contribution |

Privacy and Confidentiality in the Publication of Pedigrees:  A Survey of Investigators and Biomedical Journals FREE

Jeffrey R. Botkin, MD, MPH; William M. McMahon, MD; Ken R. Smith, PhD; Jean E. Nash
[+] Author Affiliations

From the Department of Pediatrics and the Division of Medical Ethics (Dr Botkin), Department of Psychiatry (Dr McMahon), Department of Family and Consumer Studies (Dr Smith), and the Department of Human Genetics (Ms Nash), University of Utah, Salt Lake City.


JAMA. 1998;279(22):1808-1812. doi:10.1001/jama.279.22.1808.
Text Size: A A A
Published online

Context.— Pedigree diagrams efficiently communicate family information to genetics investigators; however, the publication of pedigrees poses a risk to the privacy and confidentiality of individuals depicted in the diagrams. Two sets of authoritative guidelines have been published to protect the privacy and confidentiality of subjects, but the influence of these guidelines on publication practices for pedigrees is unknown.

Objective.— To determine the attitudes, practices, and experiences of investigators and journals with respect to privacy and confidentiality concerns in the publication of pedigrees.

Design.— Investigators who have published pedigrees and editors of 26 biomedical journals were surveyed. Journals were reviewed for content in their "information for authors" sections and for documentation of informed consent in articles containing pedigrees.

Outcome Measures.— Practices regarding confidentiality and privacy reported by investigators and editors.

Results.— Of 226 surveys sent to investigators, 177 were returned (78% response rate). Sixty-one investigators (36%) stated that family members were not informed that their pedigree would be published; 131 (78%) do not obtain informed consent specifically for pedigree publication and only 12 (28%) of the 43 who obtained consent obtained consent from all family members depicted. Thirty-two individuals (19%) reported having altered published pedigrees and 14 (45%) of 31 who had altered pedigrees stated that alterations were not disclosed to journals. Of the 14 journals that responded (54% response rate), only 3 reported written policies for managing potentially identifying information. Two journals reported having asked authors to alter pedigrees and 3 stated they had permitted alterations. A review of 5 genetics journals over a 2-year period revealed no documentation of consent for pedigree publication.

Conclusions.— Current practices in the publication of pedigrees do not conform with established recommendations and risk the privacy and confidentiality of subjects, often without informed consent. Attempts to address this problem through the alteration of data are being used, although this practice impairs the integrity of scientific communication.

MEDICAL case descriptions may reveal personal information about subjects through verbal descriptions, photographs, and pedigree diagrams. Since biomedical journals are publically available, published case descriptions create a potential confict between the desire to share scientific information and the need to protect subject privacy and confidentiality. Because of the familial nature of genetic information and the large number of subjects involved, pedigree publication poses a particularly difficult set of problems for the protection of subject privacy and confidentiality.

Pedigree diagrams permit clinicians and investigators to visualize patterns of inheritance of medical conditions in families and are one of the most commonly used tools in medical genetics.1 Pedigree diagrams and the associated text often contain personal information from several family generations, including medical conditions or genetic carrier status for medical conditions, behavior or psychiatric history, causes of death, reproduction history, marriage history, adoption history, and paternity status of family members, including instances of misattributed paternity and incest. The inadvertent disclosure of personal information within the family and to interested third parties is a serious concern. Further, when genetic testing is being used to define carrier status or susceptibility to disease, subjects themselves may learn new information about their own genetic status from published reports. Obtaining this complex genetic or other personal information without the benefit of genetic counseling or clinical consultation increases the possibilities for misinterpretation of the data, particularly since research findings are initially tentative and open to reinterpretation.

The breach of subject privacy and confidentiality risks adverse psychosocial effects, but it is also an infringement of the rights and expectations of family members if they have not given informed consent for the disclosure of such information.2,3 Confidentiality is an issue because research participants may not be aware that personal information will be published in widely available sources. Privacy is an issue because individuals may learn unwanted information about themselves. In addition, some individuals will be included on published pedigrees from information provided by other family members. In these circumstances, personal information may be published without informed consent for any participation in the research. However, the challenges to obtaining informed consent for publication of personally identifiable information are considerable and, in some circumstances, informed refusals may impede scientific progress or public health interventions.4 Balancing these competing values is a complex task.57

Two organizations have addressed privacy and confidentiality in the publication of identifying information in the scientific and editorial communities. The Office for Protection from Research Risks of the National Institutes of Health addressed the issue of publication of pedigrees specifically in its 1993 guidebook in the section on human genetic research.8 The section makes the following statement: "Where a risk of identification exists, participants must consent, in writing, to the release of personal information." The guidebook suggests a series of questions to be addressed by institutional review boards: (1) Is the pedigree essential to the publication? (2) Can identifying data be omitted? (Here, parenthetically, the guidebook adds, "The practice of altering data—such as changing the birth order and gender is controversial, and no clear professional consensus yet exists as to whether this is an appropriate practice.") (3) If an identifying pedigree is to be published, have subjects given consent for publication?

The International Committee of Medical Journal Editors (ICMJE) issued guidelines in 1995 for the protection of patient anonymity in research publications.9 The following is the full statement published by the ICMJE:

Patients have rights to privacy that should not be infringed without informed consent. Identifying information should not be published in written descriptions, photographs, or pedigrees unless the information is essential for scientific purposes and the patient (or parent or guardian) gives written informed consent for publication. Informed consent for this purpose requires that the patient be shown the manuscript to be published. Identifying details should be omitted if they are not essential, but patient data should never be altered or falsified in an attempt to attain anonymity. Complete anonymity is difficult to achieve, and informed consent should be obtained if there is any doubt. For example, masking of the eye region in photographs of patients is inadequate protection of anonymity. The requirement of informed consent should be included in the journal's instructions for authors. When informed consent has been obtained it should be indicated in the published article.

Our study was conducted to evaluate the current practices and attitudes of investigators involved in genetic research and of editors of selected biomedical journals with respect to privacy and confidentiality concerns in the publication of pedigrees and other potentially identifying information.

This study has 3 components: (1) a survey of journal editors with respect to their policies and practices for maintaining confidentiality in the publication of potentially identifying information; (2) a review of prominent journals that publish human genetic research to evaluate documentation of informed consent and the information provided for authors; and (3) a survey of investigators actively involved in human genetic research on their attitudes, experiences, and practices with respect to privacy and confidentiality issues in the publication of pedigrees, including the use of masking and altering data. By masking we mean the use of symbols (such as diamonds) in a way that is obvious to the reader that information is being withheld. By alteration we mean changing a feature or features of a subject in ways that the alteration may not be evident to readers—changing gender for example.

This project was approved by the institutional review board at the University of Utah Health Science Center.

Journal Editor Survey

The 26 journals listed in Table 1 were surveyed to assess their policies and practices with respect to confidentiality and informed consent in the publication of pedigrees and other potentially identifying information. The questionnaire was a 19-item instrument that assessed the types of articles published in the journal and their policies and practices. No remuneration was provided to journal editors for their response. Journals were not offered anonymity for their reports of journal policies and practices.

Table Graphic Jump LocationTable 1.—Editor Survey Respondents and Journals Reviewed for Information for Authors
Review of Journal Articles and Information for Authors

We reviewed all articles from the 1995 and 1996 volumes of the American Journal of Medical Genetics (including its quarterly publication Neuropsychiatric Genetics), the American Journal of Human Genetics, Nature Genetics, and Psychiatric Genetics for human pedigree diagrams. When pedigree diagrams were used, the text was read for documentation of informed consent. In addition, the information for authors sections were reviewed from 1997 volumes of the 20 journals designated in Table 1 to identify instructions for the publication of potentially identifying information. The primary targets of this review and the journal survey were general medical journals, general human genetics journals, and journals that publish neuropsychiatric research. Neuropsychiatric research was a focus since (1) the genetics of neuropsychiatric disorders is an active area of research, (2) the clinical status of individuals with neuropsychiatric conditions may not be evident to others, and (3) these conditions often carry burdens of stigma that raise particular concerns for subject privacy and confidentiality.

Investigator Survey

Our objective was to identify a sample of investigators who had recently published pedigrees. To do so, the 1996 volumes of the American Journal of Medical Genetics (including its quarterly Neuropsychiatric Genetics), the American Journal of Human Genetics, Nature Genetics, and Psychiatric Genetics were reviewed for pedigree diagrams. A total of 226 different corresponding authors were identified who had published pedigree diagrams in this time period. A 32-item questionnaire was mailed to each corresponding author with a $5 remuneration. A second identical survey was sent to nonresponders. The instrument was designed to assess demographic information and the investigator's attitudes and experiences in all projects conducted over the previous 5 years. The responses of investigators were rendered anonymous. Statistical analysis of the data relied on simple univariate statistical methods. χ2 tests were used for testing associations of categorical variables.

Journal Editor Survey

Responses were obtained from 14 (54%) of 26 editors of the journals contacted. The responding journal editors are indicated in Table 1. Two journals report publishing more than 50 articles per year with pedigrees and 1 journal publishes 20 to 50 articles with pedigrees per year. Half of the respondents (7) publish fewer than 5 pedigrees per year. In response to the question, "Does your journal have a written policy that establishes manuscript requirements for maintaining subject confidentiality in the publication of potentially identifying information, such as case descriptions, photographs or pedigree diagrams?" 3 editors responded "yes" and 11 responded "no." When asked, "In practice, does your journal require informed consent from subjects specifically for the publication of potentially identifying information, such as case descriptions, photographs or pedigree diagrams?" 5 editors responded "yes, routinely," 2 responded "yes, sometimes," and 6 responded "no." We asked whether peer reviewers were asked to comment specifically on confidentiality concerns when potentially identifying information was present—12 (86%) responded "no."

With respect to the masking and altering of pedigrees (as defined in the "Methods" section), 2 editors reported that they had asked authors to alter a diagram for confidentiality concerns and 3 editors stated they had permitted authors to alter diagrams. One editor reported that the journal had asked authors to mask a pedigree and 4 editors stated they had permitted authors to mask pedigrees because of confidentiality concerns. When asked, "To your knowledge, has a research subject or subject family member ever contacted your journal due to a (perceived) breach in subject privacy or confidentiality?" 4 editors reported "yes." Finally we asked, "Overall, do you think there are any problems with subject confidentiality related to the publication of pedigrees in biomedical journals?" In response, 8 indicated "yes," 2 responded "no," and 4 stated "don't know."

Review of Journal Articles and Information for Authors

Our initial question was whether informed consent for pedigree publication was documented in articles containing pedigrees. The results are shown in Table 2. While 18% of the articles contain an indication that informed consent for participation in research was obtained, none of the articles in 5 journals over a 2-year period had explicit documentation of informed consent for pedigree publication.

Table Graphic Jump LocationTable 2.—Results of Journal Review for Informed Consent for Pedigree Publication

Our review of the information for authors sections in 20 journals indicates that 12 journals include some information about the publication of potentially identifying information and 2 address pedigrees specifically. JAMA's information for authors states: "Include a signed statement of informed consent to publish patient descriptions, photographs, and pedigrees from all persons (parents or legal guardians for minors) who can be identified in such written descriptions, photographs, or pedigrees. Such persons should be shown the manuscript before submission." The statement of editorial policies of the American Journal of Medical Genetics states: "If photographs or any other identifiable data are included, a copy of the signed consent must accompany the manuscript." With respect to pedigree diagrams, the instructions for contributors in the American Journal of Medical Genetics includes the following statement: "Caution should be exercised in using the unmarried/illegitimate dashed lines and in identifying such offspring in pedigrees." The American Journal of Human Genetics states that it requires copies of signed consent forms for photographs. The American Journal of Psychiatry emphasizes the need for anonymity and states: "Identifying information such as names, initials, hospital numbers, and dates must be avoided. Also, authors should disguise identifying information when discussing patients' characteristics and personal history." In contrast to this position on disguising data, the British Medical Journal states: "Black bands across the eyes are wholly ineffective in disguising the patient, and changing details of patients to try to disguise them is bad scientific practice." The British Medical Journal and Lancet require written consent for the publication of case reports and photographs. The Archives of General Psychiatry, the Archives of Neurology, the Journal of Clinical Investigation, and the Journal of Nervous and Mental Disease include statements about the need to protect patient anonymity. Neurology states that it adheres to the guidelines of the ICMJE with respect to subject confidentiality. The New England Journal of Medicine requires written permission for photographs of identifiable individuals. The other 8 journals reviewed do not include information about managing potentially identifying information.

Investigator Survey

Of 226 surveys sent, 177 were returned, for a response rate of 78%. Respondents were predominantly male (69%), in mid career, and relatively experienced in medical genetics, with 42% having published 20 or more articles in the field and 43% having published 6 or more articles with pedigree diagrams. Six percent of the respondents had published 1 pedigree diagram. The majority of the corresponding investigators contacted (62%) had a non-US mailing address and 60% of the responding investigators had a non-US mailing address. The attitudes and practices reported below are not significantly correlated with the professional degree, gender, age, or with foreign vs US mailing address.

When asked the question, "What has been your personal level of concern over subject confidentiality in the publication of pedigrees?" 49% (82/169) responded "little or no concern," 40% (68/169) stated "somewhat concerned," and 11% (19/169) responded "very concerned." A series of questions addressed the experiences of investigators with respect to family interest in, and responses to, the publication of their pedigree. Of our respondents, 37% (63/170) stated that they knew that family members had read articles published with their pedigrees. In response to an open-ended question about how investigators knew whether family members were reading the articles, many investigators indicated that they often or routinely sent copies of the articles to family members. Of those responding that family members had read their articles, 57% (34/60) indicated that family members had (correctly or incorrectly) identified themselves in published pedigrees. Of the investigators who were aware of family members reading articles containing their pedigree, 26% (13/51) believed that family members had experienced benefits from identifying themselves or their family and 6% (3/50) believed that some had experienced harm.

A series of questions addressed the practices of investigators with respect to informed consent for pedigree publication. We asked, "Were the family members depicted in your published pedigrees aware that the pedigree(s) would be published?" In response, 16% (27/172) indicated "yes, all family members," 31% (53/172) responded "yes, some family members," 11% (19/172) selected "yes, very few family members," and 36% (61/172) responded "no, probably not." When asked whether they consult with family members about including their pedigree in a publication, 52% (88/172) responded "no." We asked whether investigators had ever shown the manuscript to the family prior to submitting it for publication. In response, 70% (104/149) stated "no." When asked whether written informed consent was obtained specifically for publication of pedigrees, 78% (131/167) stated "no." Of those investigators who had obtained consent for pedigree publication, 28% (12/43) obtained consent from all family members depicted, while 72% (31/43) obtained consent from selected family members. When asked whether family members are included on published pedigrees when the individuals had not consented to participate in the research, 41% (67/162) responded "yes" and 44% (72/162) stated "no," with the remainder responding "don't know."

A series of questions addressed the practices of masking and altering pedigrees. The results are included in Table 3. Of note, 19% of the respondents report altering pedigrees for their publications in the last 5 years. When those who had altered pedigrees were asked whether the alterations were disclosed to the journal, 45% (14/31) responded "no." When alterations were disclosed to the journal, 48% (10/21) responded that the alterations were always disclosed to the journal readers, while 52% responded that readers were sometimes (4/21), rarely (2/21), or not (5/21) notified of alterations. From an additional set of questions, it is clear that decisions to mask or alter pedigrees are almost always initiated by the investigators rather than by the journals. Finally, we asked, "Overall, do you think there are any problems with subject confidentiality related to the publication of pedigrees in biomedical journals?" In response, 42% (70/166) selected "yes," 42% (69/166) selected "no," and the remainder (27/166) responded "don't know."

Table Graphic Jump LocationTable 3.—Investigators' Use of Masked and Altered Pedigrees

Our results are relevant both to the human subject protections in biomedical research and to the integrity of scientific communication. The investigator survey indicates that it is common for research subjects to read published articles containing their pedigree information. In many circumstances, investigators are actively sharing this information with research participants. This collaboration between investigators and family members is laudatory and beneficial in many respects. However, the risk of an inadvertent disclosure of personal information through the publication and/or sharing of the pedigree appears significant.

Our surveys indicate that there is a moderate level of concern and a wide array of practices in the editorial community and among investigators with respect to privacy and confidentiality issues in the publication of pedigrees. We did not survey subjects involved in genetic research so we do not have direct evidence of concerns or harms from research participants. However, it is notable that 6% of the investigators believe harm has occurred to research participants from the publication of pedigrees and 4 of the responding journals indicated that they have experienced communication from research participants with concerns about breaches of privacy or confidentiality. In the experience of one of us (W.M.M.), subjects involved in neuropsychiatric genetic research have expressed serious concerns about publication of clinical diagnoses in pedigree diagrams. Further, there have been successful lawsuits based on the identification of individuals in published material,10 although, to our knowledge, no cases have developed from the publication of pedigrees.

Our review of the journals for documentation of informed consent for pedigree publication and for instructions for authors concerning guidelines for potentially identifying information indicates that many leading journals are not conforming with the recommendations of the 1995 statement by the ICMJE. As noted, the ICMJE recommends that informed consent be obtained for publishing identifiable information, documentation of informed consent be included in the published article, and the requirements in this respect be detailed in the information for authors section. Our results indicate that there is poor adherence to all of these recommendations in most of the journals reviewed and surveyed. Our survey is, in part, an update of a journal survey performed by Nylenna and Riis in 1991.11 These authors surveyed 116 medical journals and received 90 responses. In their survey, only 7 journals of the 75 that publish case reports required informed consent for publication and only 13 had written rules for protecting confidentiality. Five of the journals had experienced complaints concerning breaches of confidentiality.

Our findings from the review of published articles and instructions for authors are supported by the reported practices of investigators. Many investigators (36%) report that families are not even aware that pedigrees would be published and 78% report that specific informed consent is not obtained for the publication of pedigrees. We found no documentation of informed consent for pedigree publication in 5 key journals over a 2-year period. When consent is sought, it is not usually sought from all of the individuals depicted in the diagram. These practices are not in conformity with the Office for Protection from Research Risks guidelines or the recommendations of the ICMJE.

While current practices do not conform with recommendations for human subject protection, they also do not consistently maintain the integrity of scientific communication. Most investigators and editors are sensitive to privacy and confidentiality concerns. To address these concerns, masking and altering pedigrees appears to be prevalent. The masking of information may be an acceptable practice because of its explicit nature; however, the ICMJE has taken a strong stand against altering data. Of particular concern is our finding that 45% of those who report altering pedigrees do not consistently reveal the alterations to journals. It also appears common that even when alterations are disclosed to journals, the fact may not be revealed in the published article. Our journal survey confirms that journal staff members are aware of the practice of data alteration and sometimes encourage it, and that they occasionally publish altered data without notifying readers. Therefore, current practices can impair scientific communication in the attempt to protect subject privacy and confidentiality.

While a full discussion of potential avenues to ameliorate these problems is beyond the scope of this article, there are some points to consider raised by the data. Journals and institutional review boards should take measures to bring themselves in conformity with the current recommendations of the ICMJE and the Office for Protection from Research Risks. Journals should consider the following: (1) requesting that peer reviewers and the editorial board formally assess risks to subject privacy and confidentiality, (2) requiring explicit informed consent for the publication of potentially identifying information, (3) documenting informed consent for publication in the published articles, (4) taking an active stand against the alteration of data, and (5) detailing requirements for managing identifying information in the instructions for authors. Institutional review boards should carefully review how investigators plan to protect privacy and confidentiality in the publication phase of projects and they should require explicit informed consent for publication of potentially identifying information. Broad efforts to educate the biomedical community on these issues also is warranted.

However, not all of the ICMJE recommendations are appropriate specifically for pedigree publication. Showing subjects the manuscript prior to publication may not be advisable in this context since this practice itself risks breaching the privacy and confidentiality of family members and risks disclosing genetic information to subjects without counseling and, perhaps, prior to validation. The practice of showing the manuscript to the individual depicted would be appropriate for photographs or individual case descriptions. Additional work needs to be done to define when pedigrees are identifiable and to better understand how informed consent can be accomplished. Masking of information is an effective approach in some circumstances to maintain adequate scientific communication while protecting subject privacy and confidentiality. More discussion needs to focus on how masking can be done in an appropriate and consistent manner.

Our methods were not designed to ensure a random or statistically representative sample of investigators involved in pedigree research or an exhaustive survey of biomedical journals. Further, an attempt to evaluate the "identifiability" of pedigrees, and to assess investigator and journal responses when pedigrees are deemed identifiable, might provide a finer-grained picture of attitudes and practices. It is possible that the most identifiable pedigrees were those for which consent was obtained or in which masking was used. The lack of data on the attitudes and experiences of research participants also is a significant deficit in achieving a full understanding of these issues. Nevertheless, our large sample of published investigators and our data from a number of major journals in the field provide a clear indication that problems exist in the protection of subject privacy and confidentiality in pedigree research and that solutions should be actively pursued.

Jorde LB, Carey JC, White RL. Medical Genetics.  Baltimore, Md: Mosby-Year Book Inc; 1995:61.
Clever LH. Obtain informed consent before publishing information about patients.  JAMA.1997;278:628-629.
Powers M. Publication-related risks to privacy: ethical implications of pedigree studies.  IRB.1993;15:7-11.
Snider DE. Patient consent for publication and the health of the public.  JAMA.1997;278:624-626.
Murray JC, Pagon RA. Informed consent for research publication of patient-related data.  Clin Res.1984;32:404-408.
Frankel MS, Teich AH. Ethical and Legal Issues in Pedigree Research.  Washington, DC: American Association for the Advancement of Science; 1993:33.
Fontanarosa PB, Glass RM. Informed consent for publication.  JAMA.1997;278:682-683.
Office for Protection from Research Risks.  Protecting Human Research Subjects: Institutional Review Board Guidebook.  Washington, DC: Office for Protection from Research Risks; 1993:5-42–5-56.
International Committee of Medical Journal Editors.  Protection of patients' rights to privacy.  BMJ.1995;311:1272.
 Anderson v Strong Memorial Hospital,  140 Misc 2d 770 (1988).
Nylenna M, Riis P. Identification of patients in medical publications: need for informed consent.  BMJ.1991;302:1182.

Figures

Tables

Table Graphic Jump LocationTable 1.—Editor Survey Respondents and Journals Reviewed for Information for Authors
Table Graphic Jump LocationTable 2.—Results of Journal Review for Informed Consent for Pedigree Publication
Table Graphic Jump LocationTable 3.—Investigators' Use of Masked and Altered Pedigrees

References

Jorde LB, Carey JC, White RL. Medical Genetics.  Baltimore, Md: Mosby-Year Book Inc; 1995:61.
Clever LH. Obtain informed consent before publishing information about patients.  JAMA.1997;278:628-629.
Powers M. Publication-related risks to privacy: ethical implications of pedigree studies.  IRB.1993;15:7-11.
Snider DE. Patient consent for publication and the health of the public.  JAMA.1997;278:624-626.
Murray JC, Pagon RA. Informed consent for research publication of patient-related data.  Clin Res.1984;32:404-408.
Frankel MS, Teich AH. Ethical and Legal Issues in Pedigree Research.  Washington, DC: American Association for the Advancement of Science; 1993:33.
Fontanarosa PB, Glass RM. Informed consent for publication.  JAMA.1997;278:682-683.
Office for Protection from Research Risks.  Protecting Human Research Subjects: Institutional Review Board Guidebook.  Washington, DC: Office for Protection from Research Risks; 1993:5-42–5-56.
International Committee of Medical Journal Editors.  Protection of patients' rights to privacy.  BMJ.1995;311:1272.
 Anderson v Strong Memorial Hospital,  140 Misc 2d 770 (1988).
Nylenna M, Riis P. Identification of patients in medical publications: need for informed consent.  BMJ.1991;302:1182.
CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 23

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Topics
PubMed Articles