Young girls’ need for information after starting the combined oral contraceptive pill
This is a scientific research article. It has been reviewed and approved by the editor in chief of the Norwegian Journal of Nursing Research and independent peer reviewers. It has been translated from Norwegian. The original Norwegian version can be accessed by clicking 'Norsk' in the upper right corner.
Bakgrunn: P-piller er det mest brukte prevensjonsmiddelet i Norge, og helsesøster er en betydelig forskriver av prevensjon til unge jenter. Konkrete ferdigheter i tillegg til kunnskap er avgjørende for unge jenters handlingskompetanse i seksuelle situasjoner og relasjoner.
Hensikt: Studiens hensikt var å kartlegge og belyse unge jenters informasjonsbehov om p-piller og p-pillebruk og å sammenlikne informasjonsbehovet for de yngste og de eldste i gruppen av jenter som helsesøster frem til nå har forskrevet p-piller til.
Metode: Studien hadde et beskrivende design. Data ble samlet retrospektivt fra en internettbasert database og analysert ved hjelp av kvantitativ innholdsanalyse. Totalt 500 henvendelser fra jenter på 16 og 19 år ble samlet inn fra nettsiden Klara Klok. Kjikvadrat-test ble benyttet for å sammenlikne de to aldersgruppenes informasjonsbehov.
Resultater: Resultatene fra denne studien antyder at unge jenter har størst informasjonsbehov om blødning og blødningsforstyrrelser i tillegg til prevensjonseffekt. Det ble funnet signifikante forskjeller mellom 16- og 19-åringenes informasjonsbehov om prevensjonseffekt og ufarlige bivirkninger.
Konklusjon: Unge jenter kan ha et sammensatt informasjonsbehov ved bruk av p-piller. Helsesøstre har en viktig rolle i å informere og følge opp unge p-pillebrukere og bør ha kunnskap om de ulike informasjonsbehov som kan oppstå ved bruk av prevensjonsmiddelet.
The combined oral contraceptive pill is the most frequently used contraceptive device in Norway (1), and several studies show that the majority (between 70 and 80 per cent) of Norwegian teenage girls use or have used oral contraceptives (2-4). Insufficient knowledge on oral contraceptives may lead to discontinuation of use and unintended pregnancies in young girls (5). Young girls who use oral contraceptives have a right to information on the general use of the medication and possible risks and side effects (6). As of 2002 public health nurses and midwives with relevant accredited education have had the right to prescribe contraceptives in accordance with a specified list worked out by the Norwegian Medicines Agency, to girls aged 16-19 years (7). After this authorisation was given to public health nurses and midwives, the public health nurses in particular have become important prescribers of contraceptives to young women (7). Young girls have reported receiving less information than what doctors, public health nurses, and midwives have reported giving them (3). In order to gain knowledge on what information the public health nurse should supply when prescribing oral contraceptives, it may be useful to investigate what information the girls ask for, which again may determine young girls’ action competence in sexual situations and relations (8).
About one half of all women between 20-24 years of age report in a survey from 2010 that they do not know enough about hormonal contraceptives (9). The same tendency is seen in international surveys (5,10,11). A standardised user information pamphlet on oral contraceptives has been published (12); it may, however, be difficult to know how much information to give to a young girl who is starting to use oral contraceptives (3). A placebo-controlled study has shown that expectations regarding side effects have a substantial effect on the number of side effects experienced (13). A survey of inquiries sent to the SUSS telephone (Centre for young health, sexual relations, and sexuality) shows that bleeding was the most frequently reported side effect among oral contraceptive users in the survey sample (14). Other surveys show that adolescents make good use of free and easily accessible health information and counselling on telephone and Internet (7,8,15,16). The anonymity of such inquiries very likely makes it easier to address difficult and taboo questions.
In order to implement knowledge-based strategies and measures it is necessary to know something about adolescents’ knowledge on and attitudes to sexual health (8). According to Skjeldestad (7), there is rather scant knowledge on young women’s use of contraceptives, and they are rarely represented in surveys on contraceptive habits (7). Skjeldestad refers to studies from 1983, 1994 and 2004, and we have been unable to find any more recent relevant studies. A literature study from 2010, surveying research on oral contraceptives, showed that the existing studies are inadequate and that the field is in great need of further research (5).
The purpose of this study was twofold: 1) to map and investigate young girls’ need of information on oral contraceptives and their use, and 2) to compare the need for information on oral contraceptives and their use in the youngest and oldest in the group of girls to which public health nurses have prescribed oral contraceptives since 2002.
The study had a descriptive design and data were collected retrospectively from an internet-based database and analysed using quantitative content analysis (17,18).
The web site Klara Klok [Klara the Wise] has as its main target group children and young people aged 10 to 25 (19). The purpose of the Klara Klok service is to offer information and counselling on a.o. body, health, and sexuality (19,20). Klara Klok is a national and open question and answer service used by young people nationwide, and has a broadly composed professional panel (19). In 2014 the website received 56 198 inquiries (19), and according to unpublished statistics from Klara Klok, 2898 of these inquiries were about contraception. The questions were as a rule answered within three to four workdays, and users have reported trusting the answers given (20).
The data in this study consist of written, anonymous inquiries from girls aged 16 to 19, sent to the website Klara Klok (21). There are no given criteria regarding the size of the sample for content analysis (18), but a sample size of 400 to 600 would be sufficient, to ensure both reasonably good precision and manageability of the data (22).
To collect data on which questions this user group poses on oral contraceptives and their use, we decided to collect questions from 250 inquiries from 16 year olds and 19 year olds, a total of 500 inquiries. All inquiries on contraceptives from the two age groups were identified and read. Only inquiries on use of oral contraceptives were included in the data material. The inquiries were collected retrospectively by starting with the latest inquiries on contraception in January 2015 and ending when reaching 250 inquiries from each age group. According to figures from Klara Klok, significantly more 16 year olds have questions on contraceptives, and we had to go back in time to collect 250 inquiries from the 19 year olds. The inquiries from the 16 year olds were made between February 2014 and January 2015, and the inquiries from the 19 year olds were made between December 2012 and January 2015. The inquiries were accessed by going to the tab “latest response” and choosing the subject “contraceptives” (see figure 1).
We excluded questions on mini-pills/progesterone pills as these differ from the combined oral contraceptive pill in containing only the hormone progesterone. Mini-pills may thus give a different bleeding pattern than the combined pills (23). The inquiries were first copied to a Word-document, sorted by age, and numbered. The data collection took place in January and February 2015.
The data analysis was performed in three stages: 1) reading through the text material and establishing question categories, 2) pilot testing, 3) statistical analyses and reliability test.
The data were first read through repeatedly in order to gain an overview of the material. The work with establishing categories had an a priori design as its point of departure (18) and a template was prepared before coding the material. In collaboration with employees at Klara Klok, who read and respond to the questions posed by the users of the service, five working categories were initially established (as shown in figure 2). These categories built mainly on what the Klara Klok employees considered to be the most frequently posed questions, based on their own experience. An external public health nurse, with no knowledge of the study but with experience from working with young people and sexuality and prescription of oral contraceptives, participated in a pilot test where she identified and categorised 50 randomly chosen inquiries (10 per cent of the material) related to oral contraceptives and their use according to the suggested categories. Through a process entailing adding, discussing, and combining categories, the external public health nurse and the researchers agreed on six main categories that would cover the questions asked (see figure 2).
We started from descriptive analyses and assumed that the youngest and oldest groups of oral contraceptive users, to whom public health nurses may prescribe oral contraceptives, would be in need of different information due to age and different experiences with oral contraceptives. A Chi-square test was used to compare the groups with regard to the types of questions asked (17). The results from the Chi-square test are presented in table 1. P-values less than or equal to .05 were considered statistically significant. Statistical analyses were performed in SPSS Statistics version 22 (22,25).
Reliability and validity
In order to evaluate the reliability of the results, we drew, in accordance with Neuendorf’s recommendations (18), a new random sample of 10 per cent of the material (50 inquiries) by using SPSS. This again was coded by the same external public health nurse who participated in the pilot test. All categories were represented in this sample. The reliability coefficient Cohen’s kappa (26), which estimates degree of agreement between two or more coders, was calculated to ensure that the categorisation of the material was done with accuracy and reliability. The degree of agreement between first author and external public health nurse was assessed as very good, with an average k = .86 (26). According to Altman (27), the kappa value may be interpreted in the following way: < .20 is poor, .21 - .40 is fair, and .41 - .60 is considered moderate, .61 - .80 is good and .81 – 1.0 is very good.
To strengthen the validity of the study the question categories were developed in collaboration with employees at Klara Klok. We decided to start with the most recent inquiries to get an updated representation of the need for information. As described above, the material was read repeatedly and the categories reassessed until considered mutually exclusive by professionals and researchers (see figure 2) (18).
Anything available on the Internet is in principle considered public information unless it is protected by access control (28). In this study the sum of questions in the categories mentioned above were subject to analysis, not the individual inquiry. It is therefore considered unlikely that young users of Klara Klok will experience any unwanted exposure as a result of the study. The data in this study were anonymous and impossible to trace back to sender.
The total 500 inquiries consisted of 307 questions from 16 year olds and 321 questions from 19 year olds, a total of 628 questions. Where several questions were included in one inquiry, these were coded as separate units. The majority of the questions were related to the categories “Bleeding and bleeding disturbances” and “Contraceptive effectiveness”. The 16 year olds focused mostly on contraceptive effectiveness, while the 19 year olds focused mostly on bleeding and bleeding disturbances. Both groups had the least number of questions in the category “Serious side effects”. All results and the distribution between categories and age groups are shown in figure 3 and table 1.
A statistically significant difference was found between the questions from 16 and 19 year olds in the categories Contraceptive effectiveness (p= .007) and Harmless side effects (p= .026). The Klara Klok data show that 16 year olds to a greater extent than 19 year olds have a need for information relating to the very effectiveness of using oral contraceptives. The results also show that the 19 year olds ask more questions relating to harmless side effects of using oral contraceptives. However, the number of questions in this category were few in comparison to the total number of questions.
The study suggests that the greatest information need of a total 500 inquiries and 628 questions to Klara Klok were related to the categories Bleeding and bleeding disturbances and Contraceptive effectiveness. Irregular bleeding is a known side effect among users of oral contraceptives (29). A global survey from 2012 shows that one third of the women reported that monthly bleeding had negative consequences for daily life, in particular sexual relations and exercise (30). This study suggests that bleeding and bleeding disturbances concern the girls to a great extent. The results of 2420 inquiries to the SUSS telephone showed that bleeding was the most frequent side effect (14). Knowing how much information to give to young girls about side effects and bleeding may be difficult. Irregular bleeding may worry young girls and cause a discontinuation of oral contraceptives use, resulting in unintended pregnancies (29). Young girls need knowledge about this issue to be prepared for such a side effect (31). This study shows that the 19 year olds have more questions related to bleeding than the 16 year olds.
In line with this study earlier studies have found that six out of ten adolescents were uncertain as to the effectiveness of the oral contraceptive (32). A large number of the inquiries to Klara Klok were about the oral contraceptive’s effectiveness as protection against pregnancy. This shows great awareness of the risk of pregnancy, and that there is a need for emphasising the contraceptive effectiveness at the time of prescription. If the oral contraceptive is used properly, the protection against pregnancy is 99 per cent (12). The study shows that the 16 year olds have more questions than the 19 year olds on this subject. One may thus assume that the younger group has less experience with oral contraceptives and are more uncertain as to the contraceptive effectiveness. It is possible that the 19 year olds have learnt to trust the oral contraceptive after having used it. This again might suggest that the younger the girl is, the more important it is to give information on the safety.
The study shows a significant difference between the two age groups in the category Harmless side effects. Here as well the result points to that the older girls experience more unwanted side effects from the oral contraceptive than the younger ones. As the data material did not contain any information on how long the girls had used the oral contraceptive, it is, however, difficult to conclude that unwanted side effects and bleeding disorders increase with long-term use.
Girls who use oral contraceptives have increased risk of developing blot clots (33), and there is clear advice on which oral contraceptive pill gives the least risk for this and should therefore be an obvious first choice when the public health nurse prescribes oral contraceptives (23). An important part of the information given at the time of prescription is about the somewhat increased risk of developing blood clots (12,34), and according to earlier studies the public health nurse is aware of the need to inform about this (2,3). Inquiries to Klara Klok on blood clots and other serious side effects make up a relatively small part of the material for this study. This may suggest that there is a gap between the public health nurse’s emphasis on information on the risk of developing blood clots and the girls’ need for such information. One may wonder whether the girls do not ask about this because they have already been thoroughly informed by the prescriber, or whether they are not all that familiar with this risk. Earlier surveys show that in excess of 60 per cent reported having been informed of the risk of developing blood clots at the time of prescription of the oral contraceptive (9). In line with earlier research this study shows that there are more inquiries from 19 year olds on this subject, which may suggest that awareness regarding this risk increases with age (2).
According to unpublished figures from Klara Klok, in excess of 5000 inquiries were made concerning contraceptives by 16 year olds between 2008 and 2014, against only 1600 inquiries from 19 year olds on the same subject. This suggests that younger girls generally have more questions and a greater need for information than older girls. Young girls who use oral contraceptives have a right to information on the general use of medication and possible risks and side effects from using oral contraceptives (6). Prescriber has a duty to inform the girls based on this right (35). This information may enable young people to enjoy the pleasures of sexuality, and also be prepared to handle any risk situations that may arise (8). The public health nurse has become a significant contraceptive counsellor (7), and there has been a significant reduction in the number of teenage abortions since public health nurses were given the right to prescribe contraceptives and subsidised contraception to girls between 16 and 19 was introduced (36). The public health nurse’s role in this effort appears to be of great importance. The school health services and the health stations for young people are services that give young people access to health personnel that can offer good and adapted information on contraceptives. An extended right to prescribe for midwives and public health nurses is implemented as of January 1 2016. Public health nurses and midwives may now prescribe all hormonal contraceptives, including long-acting reversible contraception (LARC) such as intrauterine devices and implants, to all girls and women over 16 years of age (37). Long-acting hormonal contraceptives such as intrauterine devices or implants are now free or less expensive for girls between 16 and 19 years of age. A high degree of protection against pregnancy, low risk of user error, and continuity in use, makes LARC a clear first choice in prescription of hormonal contraceptives (36). It is, however, known that also these devices may have side effects in the form of bleeding disturbances (12), and at the time of prescription the consultation should include exhaustive information on this. This study has shown that side effects such as bleeding disturbances, in addition to the uncertainty related to contraceptive effectiveness, concern young users of contraceptives to a great extent. The findings may contribute to the public health nurse being able to give adapted information when prescribing any hormonal contraceptives.
Strengths and limitations
A sample does not give a perfect image of a population (17). This is also the case for this study’s sample. Klara Klok is a well-known and free national question and answer service, and inquiries to the service are ensured total anonymity (21). Users of Klara Klok are asked to give age, gender, and county, but the users decide whether they want to give correct information. This is not possible to check and it is also not possible to ask follow-up questions in order to gain more insight into the users’ situation and information need. Knowledge on who sends in questions is thus limited and we do know whether they make up a representative part of the adolescent population. The questions sent to Klara Klok on oral contraceptives are asked by girls who experience problems or uncertainties related to the contraceptive device. Therefore this does not give any indication of how great a proportion of the oral contraceptive users in general have the same experiences. Quantitative content analysis is a method used to describe the manifest content in data material in a systematic and fairly objective way (18). With this approach one will only relate to what is manifestly present in the text. No subjective interpretations of implicit meanings or perspectives are made, and one will thus miss diversity and variation in the material (38). An advantage with using quantitative content analysis is that it is a systematic, re-testable method for breaking up large quantities of text into categories based on explicit rules for coding (39). Another strength of this study is that the website from which the data originate is much visited, and that the feedback is mainly positive (20). The service can give useful insight into young girls’ experiences with using oral contraceptives. It is also possible that young girls who make inquiries on the Internet will find it easier to ask anonymously through a screen than face to face with the public health nurse for instance. The material may thus yield a more uncensored view of the information need. The Klara Klok website is used by youngsters with varied backgrounds, from the whole country (20). This, in addition to full anonymity, may support a claim that the inquiries credibly represent young girls’ need for information on the use of oral contraceptives.
The study has contributed to surveying and illuminating young girls’ need for information related to oral contraceptives and their use, and has compared the information need related to oral contraceptives and their use, for the youngest and the oldest in the group of girls to whom the public health nurse has prescribed oral contraceptives since 2002.
The results of this study show that young girls have the greatest need for information related to the categories Bleeding and Bleeding disturbances and Contraceptive effectiveness. Significant differences were found in the categories Contraceptive effectiveness, where the 16 year olds had a greater need for information than the 19 year olds, and in the category Harmless side effects, where the 19 year olds had a greater need for information than the 16 year olds. The public health nurse’s role as contraception counsellor is now more important than ever. The public health nurse has a vital role in informing and following up young users of oral contraceptives and should have knowledge on the various information needs that may arise in relation to the use of contraceptive devices. The results of this study may thus contribute to that the public health nurse can offer complete and adapted contraceptive information when prescribing oral contraceptives and other hormonal contraceptives to the various age groups.
1. Helsedirektoratet. Statistikk om seksuell helse. Available from: https://helsedirektoratet.no/folkehelse/seksuell-helse/statistikk-om-seksuell-helse-i-norge(Downloaded 08.05.2015).
2. Hansen T, Skjeldestad FE. Adolescents: Is there an association between knowledge of oral contraceptives and profession of provider? European Journal of Contraception and Reproductive Health Care 2007;12(4):303–8.
3. Sandø AD, Aanes SG, Slørdal L, Spigset O. Rutiner ved førstegangsforskrivning av p-piller. Tidsskrift for Den norsk legeforening 2010;130:23(23):44–8. Available from: http://tidsskriftet.no/article/2046342 (Downloaded 08.06.2016).
4. Bratlie M, Aarvold T, Skarn E, Lundekvam J, Nesheim BI, Askevold E. Long-acting reversible contraception for adolescents and young adults-A crosssectional study of women and general practitioners in Oslo, Norway. European Journal of Contraception and Reproductive Health Care 2014;19:S189–S90.
5. Hall KS, Castano PM, Stone PW, Westhoff C. Measuring oral contraceptive knowledge: a review of research findings and limitations. Patient Educ Couns. 2010;81(3):388–94.
6. Lovdata. Lov om pasient og brukerrettigheter. 2. juli 1999 [Pasient- og brukerrettighetsloven]. Available from: http://lovdata.no/lov/1999-07-02-63(Downloaded 01.05.2015).
7. Skjeldestad FE. Forskrivning av prevensjon til unge kvinner. Tidsskrift for Den norske legeforening 2012;132(3):292–4. Available from: http://tidsskriftet.no/article/2210166/(Downloaded 08.06.2016).
8. Helsedirektoratet. Forebygging av uønsket svangerskap og abort 2010-2015- strategier for bedre seksuell helse. Oslo: Helsedirektoratet, avdeling for miljø og helse; 2010.
9. Øren A, Leistad L, Haugan T. Endres prevensjonsvaner og abortrate hos kvinner 20–24 år ved tilbud om gratis hormonell prevensjon? Trondheim: Sintef Teknologi og samfunn, avd., forebyggende helsearbeid, 2010.
10. Hamani Y, Sciaki-Tamir Y, Deri-Hasid R, Miller-Pogrund T, Milwidsky A, Haimov-Kochman R. Misconceptions about oral contraception pills among adolescents and physicians. Human Reproduction. 2007;22(12):3078–83.
11. Sereno S, Correia MJ, Antunes I, Martins I. Adolescence and contraception: Myths and misconceptions. European Journal of Contraception and Reproductive Health Care 2014;19:S129.
12. Gamnes, S., Johansen, M., Sex og Samfunn, senter for ung seksualitet. Metodebok. 7 rev. ed. Oslo 2015.
13. Redmond G, Godwin AJ, Olson W, Lippman JS. Use of placebo controls in an oral contraceptive trial: Methodological issues and adverse event incidence. Contraception 1999;60(2):81–5.
14.Bøe AB, Lillelien L. P-pillebruk, bivirkninger og prevensjonssvikt blant brukere av SUSS-telefonen (Prosjektoppgave i medisin). Det medisinske fakultet, Universitetet i Oslo; 2011.
15. Gray NJ, Klein JD, Noyce PR, Sesselberg TS, Cantrill JA. Health information-seeking behaviour in adolescence: The place of the internet. Social Science and Medicine. 2005;60(7):1467–78.
16. Sokkary N, Mansouri R, Yoost J, Focseneanu M, Dumont T, Nathwani M, et al. A Multicenter Survey of Contraceptive Knowledge among Adolescents in North America. Journal of Pediatric and Adolescent Gynecology. 2013;26(5):274–6.
17. Polit DF, Beck CT. Essentials of Nursing Research, Appraising Evidence for Nursing Practise. 8 ed. Philadelphia: Wolters Kluwers Health, Lippincott Williams & Wilkins; 2014.
18. Neuendorf KA. Content analysis. Thousand Oaks, California: Sage Publications, Inc.; 2002.
19. Klara Klok. Årsrapport 2014. Available from: https://www.nfk.no/Filnedlasting.aspx?MId1=15352&FilId=33241(Downloaded 01.05.2015).
20. Gundersen T, Windsvold A. Klara Klok, en helsenettjeneste for ungdom. Norsk institutt for forskning om oppvekst, velferd og aldring 2011.
21. Klara Klok. Forsiden. Available from: http://www.klara-klok.no/(Downloaded 05.01.15).
22. Jacobsen DI. Forståelse, beskrivelse og forklaring Innføring i metode for hele- og sosialfagene. Høyskoleforlaget, Kristiansand. 2010.
23. Legemiddelverket. Anbefalte hormonelle prevensjonsmidler. Available from: http://www.legemiddelverket.no/Bruk_og_raad/p-piller/prevensjonsguide-hormonell/Sider/default.aspx(Downloaded08.05.2015).
24. IBM. SPSS software. Available from: http://www-01.ibm.com/software/analytics/spss/(Downloaded 01.03.2015).
25. Field A. Discovering statistics using SPSS. 3. utg. SAGE Publications, London. 2009.
26. Cohen J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement. 1960;20(1):37–46.
27. Altman, D. Practical statistics for medical research. CRC press, 1990.
28. Den nasjonale forskningsetiske komité for samfunnsvitenskap og humaniora (NESH). Available from: https://www.etikkom.no/forskningsetiske-retningslinjer/etiskeretningslinjer-for-forskning-pa-internett/(Downloaded 01.12.2015).
29. Godfrey EM, Whiteman MK, Curtis KM. Treatment of unscheduled bleeding in women using extended- or continuous-use combined hormonal contraception: a systematic review. Contraception. 2013;87(5):567–75.
30. Szarewski A, von Stenglin A, Rybowski S. Women›s attitudes towards monthly bleeding: Results of a global population-based survey. The European Journal of Contraception and Reproductive Health Care. 2012;17(4):270–83.
31. Weisberg E, Merki-Feld GS, McGeechan K, Fraser IS. Randomized comparison of bleeding patterns in women using a combined contraceptive vaginal ring or a low-dose combined oral contraceptive on a menstrually signaled regimen. Contraception. 2015;91(2):121–6.
32. Frost JJ, Lindberg LD, Finer LB. Young Adults› Contraceptive Knowledge, Norms and Attitudes: Associations with Risk Of Unintended Pregnancy. Perspectives on Sexual and Reproductive Health. 2012;44(2):107-16.
33. Lidegaard O, Lokkegaard E, Svendsen AL, Agger C. Hormonal contraception and risk of venous thromboembolism: national follow-up study. BMJ (Clinical research ed). 2009;339:b2890.
34. Tepper NK, Curtis KM, Steenland MW, Marchbanks PA. Blood pressure measurement prior to initiating hormonal contraception: a systematic review. Contraception. 2013;87(5):631-8.
35. Lovdata. Lov om helsepersonell. 2 juli 1999; nr. 4. [Helsepersonelloven]. Available from: https://lovdata.no/dokument/NL/lov/1999-07-02-64#KAPITTEL_2(Downloaded 01.05.2015).
36. Helse og omsorgsdepartementet. Høring – utvidet rekvireringsrett for helsesøstre og jordmødre, forslag til endring i forskrift om rekvirering og utlevering av legemidler fra apotek. Available from: http://www.regjeringen.no/nb/dep/hod/dok/hoeringer/hoeringsdok/2014/Horing---utvidet--rekvireringsrett-for-helsesostre-og-jordmodre-forslag-til-endring-i-forskrift-om-rekvirering-og-utlevering-av-legemidler-fra-apotek/Horingsbrev.html?id=767295(Downloaded 01.02.2015).
37. Helse- og omsorgsdepartementet. Forskrift om endring i forskrift av 27. april 1998 nr. 455 om rekvirering og utlevering av legemidler fra apotek § 2–5 2015.
38. Ryghaug M. Å bringe tekster i tale- mulige metodiske innfallsvinkler til tekstanalyse i statsvitenskap. Norsk vitenskapelig tidsskrift. 2002, 18 ER(04).
39. Stemler S. An overview of content analysis. Practical assessment, Research & Evaluation. 2001, 7 (17).