Semantic Contours in Tracks Based on Emotional Tags

Transcription

1 Dowloaded from orbit.dtu.dk o: Dec, Sematic Cotours i Tracks Based o Emotioal Tags Peterse, ichael Kai; Hase, Lars Kai; Butkus, Adrius Published i: Computer usic odelig ad Retrieval. Lik to article, DOI:./----_ Publicatio date: Documet Versio Early versio, also kow as pre-prit Lik back to DTU Orbit Citatio (APA): Peterse,. K., Hase, L. K., & Butkus, A. (). Sematic Cotours i Tracks Based o Emotioal Tags. I Computer usic odelig ad Retrieval.: Geesis of eaig i Soud ad usic (st ed., Vol. Volume /, pp. -). Spriger. (Lecture Notes i Computer Sciece, Vol. ). DOI:./--- -_ Geeral rights Copyright ad moral rights for the publicatios made accessible i the public portal are retaied by the authors ad/or other copyright owers ad it is a coditio of accessig publicatios that users recogise ad abide by the legal requiremets associated with these rights. Users may dowload ad prit oe copy of ay publicatio from the public portal for the purpose of private study or research. You may ot further distribute the material or use it for ay profit-makig activity or commercial gai You may freely distribute the URL idetifyig the publicatio i the public portal If you believe that this documet breaches copyright please cotact us providig details, ad we will remove access to the work immediately ad ivestigate your claim.

2 Sematic cotours i tracks based o emotioal tags ichael Kai Peterse, Lars Kai Hase, ad Adrius Butkus Techical Uiversity of Demark, DTU Iformatics, Buildig, DK., Kgs.Lygby, Demark {mkp,lkh,ab}@imm.dtu.dk Abstract. Outliig a high level cogitive approach to how we select media based o affective user prefereces, we model the latet sematics of lyrics as patters of emotioal compoets. Usig a selectio of affective last.fm tags as top-dow emotioal buoys, we apply LSA latet sematic aalysis to bottom-up represet the correlatio of terms ad sog lyrics i a vector space that reflects the emotioal cotext. Aalyzig the resultig patters of affective compoets, by comparig them agaist last.fm tag clouds describig the correspodig sogs, we propose that it might be feasible to automatically geerate affective user prefereces based o sog lyrics. Key words: Patter recogitio, emotios, text processig Itroductio Both words ad music move i time, ad as T.S. Elliot phrased it oly by the form, the patter, ca words or music reach. A paoply of sesatios ad emotios are elicited whe we liste to a sog, which i tur reflect cogitive aspects of the uderlyig structure i both soud ad lyrics. Over the past half cetury these aspects of musical affect have bee the focus of a wide field of research ragig from how emotios arise based o the uderlyig harmoic ad rhythmical structures formig our expectatios [-], to how we cosciously experiece these patters empathetically as cotours of tesios ad release []. Basic feeligs of happiess, beig or gettig are ot just perceived but materialize as chages i heart rate, ski coductace, respiratio or blood pressure, as has bee documeted i umerous cogitive studies of music ad emotios []. Applyig biosesors to measure the features that uderlie the various affective states, the resultig patters appear sufficietly cosistet to determie what emotios are beig triggered based o the physiological chages aloe []. But listeig to sogs ivolves ot oly basic elemets of affect, but also higher level structures reflected i the lyrics which provide the basis for a sog. To a large extet laguage allows us to share ad describe distict affective aspects that we extract from the cotiuous affective ebb ad flow of emotios shapig our frame of mid. Despite the ofte idiosycratic character of tags defied by

3 .K. Peterse, L.K. Hase, ad A. Butkus hudred thousads of users i social etworks like last.fm, a umber of studies withi the music iformatio retrieval commuity idicate that users ofte ted to agree o the affective terms they attach to music, which ca be iterpreted as a simplified mood groud-truth reflectig the perceived emotioal cotext of the music [-]. Durig the past decade advaces i euroimagig techologies eablig studies of brai activity have established that musical structure to a larger extet tha previously thought is beig processed i laguage areas of the brai [], ad specifically related to lyrical music some fudametal aspects appear essetially idetical to those of laguage []. Neural resources betwee music ad laguage appear to be shared both i sytactic sequecig ad also sematic processig of patters reflectig tesio ad resolutio [-], addig support for fidigs of liguistic ad melodic compoets of sogs beig processed i iteractio []. Similarly there appears to be a overlap betwee laguage regios i the brai ad so-called mirror euros, which trasfer sesory iformatio of what we perceive by re-eactig them o a motor level. ediatig the iputs across audiovisual modalities, the resultig sesory-motor itegratios are represeted i a similar form, whether they origiate from actios we observe i others, oly imagie or actually eact ourselves [-]. This has led to the suggestio that our empathetic comprehesio of uderlyig itetios behid actios, or the emotioal states reflected i seteces ad melodic phrases, are based o a imitative re-eactmet of the perceived motio []. So if both low-level features of media ad our emotioal resposes ca be ecoded i words, we hypothesize that this might allow us to defie a high level cogitive model emulatig how we select media based o affective user prefereces. I such a model the bottom-up part would resemble cogitive compoet aalysis []. Coied as a term to describe aspects of usupervised clusterig of data, the uderlyig algorithms approximate how our brai discovers self-orgaizig patters whe assemblig images from lies ad edges of visual objects [], recostructs words from the statistical regularities of phoemes i speech [] or lear the meaig of words based o their co-occurrece withi multiple cotexts [-]. But equally importat: cogitive processes ivolve a large amout of top-dow feedback which sculpts the receptive resposes of euros o every level ad vastly outumbers the sesory iputs [-]. That is, the brai applies a aalysis-by-sythesis approach, which combies a top-dow capability to ifer structure from bottom-up processig of statistical regularities i what we perceive. Our emotios are i this sese essetial for maitaiig a balace betwee cogitio ad perceptio, as core affect is a itegral elemet i what attracts us to objects ad tur what we sese ito meaigful represetatios that ca be categorized i words [-]. A way to emulate this approach of the huma brai i relatio to search of media, could be to apply usupervised learig of features based o latet sematics, extracted from lyrics associated with sogs. Ad combie the bottomup extracted represetatio with top-dow aspects of attetio reflectig preferred emotioal structures, similar to the combiatios of user geerated affec-

4 Sematic cotours i tracks based o emotioal tags tive terms foud i tag clouds i social etworks like last.fm. Selectig a umber of frequetly used emotioal last.fm tags as buoys to defie a sematic plae of psychological valece ad arousal dimesios, we project a umber of sog lyrics ito this space ad apply LSA latet sematic aalysis [-], to model the correlatio of texts ad affective terms as vectors reflectig the emotioal cotext of the sogs. We outlie i the followig sectios: the affective plae used for modelig emotioal structure, the extractio of latet sematics from texts associated with media, a aalysis of the emotioal patters, followed by a discussio of the potetial i combiig latet sematics ad emotioal compoets to eable persoalized search of media. Affective plae Drawig o stadard psychological parameters for emotioal assessmet, affective terms are ofte mapped out alog the two psychological dimesios of valece ad arousal [-]. Withi this D emotioal plae the dimesio of valece describes how pleasat somethig is alog a axis goig from positive to egative associated with words like or, whereas arousal captures the amout of ivolvemet ragig from passive states like ad to active aspects of excitatio as reflected i terms like or. This approach to represet emotios withi a affective space framed by valece ad arousal dimesios goes beyod earlier attempts to defie distict categories like Hever s circle of adjectives (). Based o resposes from participats listeig to musical excerpts, clusters of words were grouped ito eight segmets of similar adjectives coverig emotios like, lyrical, calm, dreamy,, serious, tragic, ad excitig []. How may differet parameters are required to capture the various compoets i a affective space has sice the bee the subject of a umber of studies. The results idicate that a model which provides a good fit to how people describe emotioal states, ca be defied based o five uderlyig latet variables: ager, ess, disgust, fear ad happiess []. I such a model these factors are ot ecessarily correlated with whether they are perceived as pleasat or upleasat, as opposig aspects might ofte occur together eve if they represet cotrastig positive ad egative aspects of valece. Empirical results for ratig of emotioal words, also idicate that certai terms e.g. syoyms for or ager seem to be based o oe category oly ad are defied as either positive or egative alog a sigle dimesio. Whereas other affective terms appear more complex ad appear to be combiatios of more emotioal categories, like despair beig perceived as a mixture of ess ad axiety, or excitemet ivolvig aspects of both happiess ad surprise []. I ay liguistic descriptio we perceive ot oly the lexical meaig of words but ifuse them with feeligs of positive or egative valece [], which might serve to filter what is essetial ad determie what becomes part of our memories []. Experimets usig DS multidimesioal scalig to group musical excerpts accordig to similarity istead of word categories, idicated that the model which provides the best fit to the emotioal resposes seems agai to be based o the

5 .K. Peterse, L.K. Hase, ad A. Butkus two psychological dimesios of valece ad arousal, but here combied with a additioal third dimesio capturig aspects of shape related to the degree of melodic cotiuity or rhythmical fragmetatio i the music []. I a further cluster aalysis of the data, the participats were foud to separate the resposes ito two segmets alog the arousal dimesio, meaig how eergetic or laid back the music was perceived as beig. The similarity judgemets of musical excerpts withi the two clusters i tur appeared to be divided alog the fault lies of valece, essetially separatig the emotioal reactios ito a affective plae cosistig of,, ad quadrats. If we attempt to model top-dow cogitive attetioal aspects reflectig affective structure, tag-clouds i music social etworks like last.fm provide a iterestig case. The affective terms which are frequetly chose as tags by users to describe music seem to form clusters aroud primary moods like,, or more agitated feeligs like ad. This correlatio betwee social etwork tags ad the specific music tracks they are associated with, has bee used i the music iformatio retrieval commuity to defie a simplified mood groud-truth, reflectig ot just the words people frequetly use whe describig the perceived emotioal cotext, but also which tracks they agree o attachig these tags to [-]. Selectig twelve of these frequetly used tags:,,,,,,,, makes it possible to defie a affective plae reflectig the above cogitive models of emotioal resposes to music, as a basis for extractig latet sematics. Sematic space To geerate the bottom-up part of how we cogitively extract meaig from strigs of texts, LSA latet sematic aalysis models comprehesio from word occurreces i multiple cotexts, aalogous to huma laguage acquisitio [- ]. Words rarely come shrik-wrapped with a defiitive meaig but are cotiuously modified by the cotext i which they are set. No matter how may examples of word usage for a verb are listed i a dictioary they remai just that: case stories which illustrate how a predicate will map oto a certai value give a specific argumet. Replacig ay of the surroudig words i the setece will create yet aother istatiatio of the propositio, which we might agai iterpret differetly depedig o what phrases come before or after i the text. Istead of attemptig to defie the specific meaig of a word based o how it fits withi a particular grammatical phrase structure, LSA latet sematic aalysis, models the plethora of meaigs a word might have by cocateatig all the situatios i which it appears ad represet them as a sigle vector withi

6 Sematic cotours i tracks based o emotioal tags i a high dimesioal sematic space. Squeezig as may of the sytactic relatios ad seses of word usage ito a sigle vector, makes it possible to extract statistical properties based o how ofte a term appears i a large umber of paragraphs. Ad subsequetly codese this represetatio ito meaigful sematic relatios costructed from a average of the differet cotexts i which the word is used. Iitially a text corpus is costructed which allows for modelig terms as liear combiatios of the multiple paragraphs ad the seteces i which they occur, assembled from tes of thousads of pages of literature, poetry, wikipedia ad ews articles. The uderlyig text corpora ca be thought of as resemblig huma memory where umerous episodes combied with lexical kowledge are ecoded ito strigs of text. Spaed by rows of words ad colums of documets, the cells of this huge term-documet matrix up how frequetly each word appears i a correspodig paragraph of text. However i a simple co-occurrece matrix ay similarities betwee words like car ad vehicle will be lost as each idividual term appears oly withi its ow horizotal row. Nor will it be obvious that a word like rock might mea somethig completely differet depedig o which of the cotextual colums it appears i. The raw matrix couts of how may times a word occurs i differet cotexts does therefore ot by itself provide a model of comprehesio, as we would ormally expect texts that describe the same topic to share may of the terms that are used, or imagie that words that resemble each other are also applied i a similar fashio. ost of these relatios remai hidde withi the matrix, because there are tes of thousads of redudat variables i the origial term-documet matrix obscurig the uderlyig sematic structure. Reducig the dimesioality of the origial matrix usig SVD sigular value decompositio [], the umber of parameters ca be dimiished so we ca fit syoymous words or group similar documets ito a much smaller umber of factors that ca be represeted withi a sematic space. Geometrically speakig, the terms ad documets i the codesed matrix derived from the SVD dimesioality reductio, ca be iterpreted as poits i a k dimesioal subspace, which eables us to calculate the degree of similarity betwee texts based o the dot product of their correspodig vectors. But before comparig terms or documets, the etries i the cells of the matrix eed to be adjusted so they reflect how we cogitively perceive associative processes. First by replacig the raw cout of how ofte a word appears i a text by the logarithm of that umber. This will smooth the word frequecy so it resembles the shape of learig curves typically foud i empirical psychological coditioig experimets. Likewise the degree of associatio of two words both occurrig i two documets will be higher tha if they each appear twice separately i a text. Here a local weightig fuctio defies how saliet the word occurrece is i the correspodig documet, ad a global weightig fuctio how sigificat its appearace is amog all the cotexts []. As a ext step the word cout is divided by the etropy of the term, to esure that the term frequecy will be modified by how much iformatio the word actually adds about the cotext it

7 .K. Peterse, L.K. Hase, ad A. Butkus appears i. This log-etropy weightig sigificatly improves the results whe compared to a raw word frequecy cout []. Aother way to iterpret the relatios betwee words formig a sematic eighborhood would be to thik of them as odes costitutig a eural etwork. I such a etwork of odes, resemblig populatios of euros i our brais, LSA could model the stregth of the liks coectig oe word to aother. Whe we come across a word like i a phrase, it will create a ode i our short term episodic memory, which will i tur trigger eighborig odes represetig words or evets that ivoke similar cootatios i our past memories. The stregth of the coectios iitially based o word co-occurrece are gradually trasformed ito sematic relatios as the liks betwee odes are beig costraied by the limitatios of our memory. As a result oly those odes which remai sufficietly activated whe our attetio shifts towards the ext phrase will be itegrated ito the patters formig our workig memory. Ad whether these coectios grow sufficietly strog for the odes to reach a threshold level of activatio ecessary for beig itegrated i workig memory, ca be see as a fuctio of the cosie betwee the word vectors []. Whe we compare two terms i the LSA sematic space based o the the cosie of the agle betwee their vectors, values i-betwee. ad will idicate icreasigly sigificat degrees of similarity betwee the words, while a egative or low value aroud will idicate a radom lack of correlatio. If we for istace select the affective term ad calculate the cosie betwee the agle of its vector represetatio ad ay other word i the text corpus, we ca determie which other term vectors are sematically close, ad i decreasig order list to what degree they share aspects reflectig the meaig of that word:.. grief. sorrow. mour. sigh. weep. tear. griev. piti. ala Lookig at these earest eighbors it would seem that istead of iterpretig isolated as a sigle vector made from the various documets i which it appears, we might rather thik of the meaig of that word as a sematic eighborhood of vectors. I this part of our LSA sematic space these earest eighbors form a etwork of odes, where each word add differet aspects to the meaig depedig o the stregth of their associative liks to. So if we imagie text comprehesio as a process that combies the words which shape a setece with the associatios they trigger we ca model this as a bottomup spreadig activatio process. I this etwork the stregth of liks betwee

8 Sematic cotours i tracks based o emotioal tags odes will be defied by their weights ad cosequetly the coectios amog all odes ca be mapped out i a coectivity matrix. Beig exposed to a icomig word the stimulus will spread from the ode geerated i episodic memory, to its sematically earest eighbors stored i log term workig memory. How may of these coectios grow sufficietly strog for the odes to be itegrated i log term workig memory, determies whether our comprehesio is reduced to a assembly lie where separate words are merely glued together based o the icomig text aloe. Or it will istead provide a blueprit for recostructig a situatio model, resemblig a aimated pi-ball machie where the associatios triggered by the words bouce off walls formig a itricate maze of memories. Ad oce reality kicks i, i terms of the costraits posed by the limited capacity of our workig memory, what odes will remai activated could be uderstood as proportioal to the LSA cosie similarity of vectors, triggered by the words beig parsed ad their earest eighbors already residig i our memories []. Results I the ext subsectios we outlie the structure of the LSA patters i regards to the distributio of emotioal compoets, compare the LSA aalyses of lyrics agaist their correspodig last.fm tag clouds, ad fially explore correlatios betwee LSA patters ad the uderlyig musical structure of the sogs.. Distributio of LSA compoets Projectig the lyrics of twety-four sogs selected from the weekly top track charts at last.fm, we compute the correlatio betwee lyrics ad the previously selected twelve affective tags used as markers i the LSA space, while discardig cosie values below a threshold of.. Whereas the user-defied tags at last.fm describe a sog as a whole, we aim to model the shiftig cotours of tesio ad release which evoke emotios, ad therefore project each of the idividual lies of the lyrics ito the sematic space. Aalyzig idividual lies o a timescale of secods also reflects the cogitive temporal costraits applied by our brais i geeral whe we bid successive evets ito perceptual uits []. We perceive words as successive phoemes ad vowels o a scale of roughly millisecods, which are i tur itegrated ito larger segmets with a legth of approximately secods. We thus ase that lies of lyrics cosistig of a few words each correspod to oe of these high level perceptual uits. The outputs are matrixes cosistig of colums of LSA values triggered by each lie i the lyrics i respose to the twelve emotioal tags makig up the rows. Similar to a emotioal space, the colums of the matrix reflect a vertical spa from positive to egative valece. The upper rows i the colums correspod to active positive emotios like ad followed by more passive aspects like ad towards the ceter of the colums. Further dow the values i the colums correspod to active egative aspects like

9 .K. Peterse, L.K. Hase, ad A. Butkus while the bottom rows i the matrix reflect passive egative emotios such as melacholic ad. Whe the idividual lies of lyrics are projected agaist the selected last.fm tags, it results i twelve dimesioal vector values sigifyig affective compoets that are activated simultaeously rather tha discrete emotios. Iitially addig up the LSA values i the matrices alog each row, we ca plot the activatio of each emotioal compoet over the etire lyrics. Aalyzig a small sample of twety-four sogs, the med up values of LSA correlatio betwee the idividual ad the last.fm affective terms appears to divide the lyrics ito roughly three groups: Balaced distributio of emotios where the lyrics simultaeously trigger affective compoets from the outer extremes of both ad. Combied with more passive positive aspects like it results i types of patters as foud i the sogs: Thigs i wat i a lover (Alais orissette), Bleedig love (Leoa Lewis), The Scietist (Coldplay), ad world (Gary Jules), Nothig else matters (etallica), Starlight (use) ad Come away with me (Norah Joes). (Fig.) - or alteratively the patters juxtapose active positive ad egative elemets of ad versus agaist each other, with relatively less cotributio from passive positive aspects like, as i the sogs: Everybody hurts (R.E.), Iris (Goo Goo Dolls), Woderwall (Oasis), Time to preted Rehab (Amy Wiehouse). (Fig.) Cetered distributio of emotioal compoets emphasizig passive positive aspects like or combied with passive egative emotios close to, with relatively less sigificat cotributios from active positive affective compoets such as as i the sogs: Now at last (Feist), y immortal (Evaescece), Creep (Radiohead) ad Colorblid (Coutig Crows). (Fig.) Uiform distributio of emotioal compoets activated across the etire affective spectrum of valece ad arousal as i the sogs: Always where i eed to be (The Kooks), Sa Queti (Johy Cash), Clocks (Coldplay), What I ve doe (Liki Park), Fallig slowly (Gle Hasard), Stairway to heave (Led Zeppeli), Smells like tee spirit (Nirvaa) ad Such great heights (The Postal Service)(Fig.).. LSA emotios versus last.fm tags The tag clouds at last.fm describe a sog as a whole, so i order to assess to what degree the retrieved LSA correlatio values of lyrics ad affective terms approximate the user-defied tags, we use the accumulated LSA values med up over the etire lyrics as outlied i previous sectio. To facilitate a compariso betwee lyrics ad the tag clouds, which my oly cotai a few of the selected last.fm affective terms used i the LSA aalysis, we subsequetly group the LSA values of closely related tags ito a emotioal space cosistig of four

10 The Scietist avg > > u <. - - love bleedig The Scietist avg... - ad World Gary Jules avg Gary Jules ad World u No h o etallica C m Aw Nothig Else atters W AX PRESENCE AX AX PRESENCE..... PRESENCE PRESENCE avg.... < PRESENCE PRESENCE... avg < AX < avg > AX AX W No h o C m Aw > E E < < < < < avg avg presence > > AX PRESENCE > >. - - avg AX AX > E < bleedig love avg Sematic cotours i tracks based o emotioal tags > Bleedig. - -.Love Thigs I Wat I A Lover Coldplay- Nothig Else atters Leoa Lewis Alais orissette etallica E E Coldplay... PRESENCE F g Accumu ated LSA va ues o emot oa compoets tr ggered by the yr cs ad the r correspod g ast m tag c ouds - rom top e t ad across Th gs wat a over (A a s or ssette) B eed g ove (Leoa Lew s) The Sc et st (Co dp ay) ad wor d (Gary Ju es) Noth g e se matters (eta ca) Star ght (use) ad Come away w th me (Norah Joes)

11 Oasis E E Woderwall Wiehouse Amy - - Rehab < GT avg > < avg > E Feist R d oh d N wa mm.... E Cou C.ow.... C C AX PRESENCE Cou C. ow C >.. avg < PRESENCE PRESENCE PRESENCE PRESENCE AX PRESENCE.. Fig.. Accumulated LSA values of emotioal compoets triggered by the lyrics ad their correspodig last.fm tag clouds - from top left ad across: Everybody hurts (R.E.), Iris (Goo Goo Dolls), Woderwall (Oasis), Time to preted Rehab (Amy Wiehouse) E R d oh d E- - -.E AX PRESENCE < C mm Now At Last < avg..... avg < E N an N an PRESENCE > > N an E N an AX AX N an N an N N. an an an. -. an..... N N an - - an.... N N E < < avg PRESENCE > AX AX AX avg AX <. avg k. Peterse, L.K. Hase, ad A. Butkus < Rehab Amy Wiehouse E avg PRESENCE > > AX > < Time To Preted E E AX F g Accumu ated LSA va ues o emot oa compoets tr ggered by the yr cs ad the r correspod g ast m tag c ouds - rom top e t ad across Now at ast (Fe st) y mmorta (Evaescece) Creep (Rad ohead) ad Co orb d (Coutg Crows) PRESENCE

12 Stairway To Heave avg < > avg Led Zeppeli Gle Hasard Coldplay Fallig Slowly Sa Queti > Clocks Always Where I Need To Be Liki Park What I ve Doe Johy Cash - Coldplay The Kooks Park SematicLiki cotours i tracks based o emotioal tags What I ve Doe < avg avg > > avg - - < avg PRESENCE < > > E PRESENCE E avg AX AX > E-.E-.E-.E < < avg avg < PRESENCE > > PRESENCE E E AX E E AX E E- - -.E-.-.E E < < Clocks m N Th Po L d Z pp Gle Hasard w Fallig Slowly G H H E-. N m Th Po G. H avg > AX AX < PRESENCE PRESENCE.. PRESENCE. PRESENCE.. F g Accumu ated LSA va ues o emot oa compoets tr ggered by the yr cs ad the r correspod g ast m tag c ouds - rom top e t ad across A ways where eed to be (The Kooks) Sa Quet (Johy Cash) C ocks (Co dp ay) What I ve doe (L k Park) Fa g s ow y (G e Hasard) Sta rway to heave (Led Zeppe ) Sme s ke tee sp r t (N rvaa) ad Such great he ghts (The Posta Serv ce).. PRESENCE AX < AX AX PRESENCE avg ax > PRESENCE AX