Question prediction language model

 Ökologie

 3 views
of 8

Please download to get full document.

View again

All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Description
Abstract This paper proposes the use of a language representation that specifies the relationship between terms of a sentence using question words. The proposed representation is tailored to help the search for documents containing an answer for a
Share
Transcript
  Question Prediction Language Model Luiz Augusto Pizzato and Diego Moll´a Centre for Language TechnologyMacquarie UniversitySydney, Australia { pizzato, diego } @ics.mq.edu.au Abstract This paper proposes the use of a languagerepresentation that specifies the relationshipbetween terms of a sentence using questionwords. The proposed representation is tai-lored to help the search for documents con-taining an answer for a natural languagequestion. This study presents the construc-tion of this language model, the framework where it is used, and its evaluation. 1 Introduction Although Information Retrieval (IR) can be helpedby NLP techniques such as named entity (NE)recognition, phrase extraction and syntax parsing(Strzalkowski, 1999), they are not generally useddueto their high complexity. One such task that peo-ple can perform somewhat easily whilst still beinghard for computers is the answering of factoid ques-tions based on textual content. The Question An-swering (QA) Track of TREC (Voorhees, 2005) fo-cuses on answering questions using the AQUAINTcorpus (Graff, 2002), which contains 375 millionwordsfrom three different sources of newswire data:Xinhua News Service (XIE) from People’s RepublicofChina, theNewYorkTimesNewsService(NYT),and the Associated Press Worldstream News Service(APW).For the QA task, not only it is important to find ananswerinadocument, butalsotofindthedocumentsthat might contain the answer in the first place. MostQA systems take the approach of using off-the-shelf IR systems to return a list of documents that maycontain an answer, and then processing the list of documents to look for the required answer. Nor-mally the processing time for every question in thesesystems is long because of the sheer amount of work that is required after the list of document is returned.Many QA systems focus on the input and outputof IR systems. For example, Dumais et al. (2002)perform a passive-to-active voice transformation of the question, in an attempt to bring the IR querycloser to the document it is expected to retrieve.Some IR work focuses on improving QA by pas-sage retrieval re-ranking using word overlap mea-sures. For instance, Tellex et al. (2003) compare agroup of passage retrieval techniques and concludethat those that apply density-based metrics 1 are themost suitable to be used for QA.Some work has been done on IR models thatspecifically aid the QA task. The work of Monz (2004) defines a weighting scheme that takesinto consideration the distance of the query terms.Murdock and Croft (2004) propose a translation lan-guage model that defines the likelihood of the ques-tion being the translation of a certain document.Tiedemann (2005) uses a multi-layer index contain-ing more linguistic oriented information and a ge-netic learning algorithm to determine the best pa-rameters for querying those indexes when appliedfor the QA task. Tiedemann argues that since ques-tion answering is an all-natural language task, lin-guistic oriented IR will help finding better docu-ments for QA.In this paper we propose a language representa- 1 Ranking of passages based on the number of query wordsand the proximity between them. Proceedings of the Australasian Language Technology Workshop 2007, pages 92-9992  tion that when used in the IR stage of a questionanswering system improves its results. As a conse-quence it helps to reduce the processing time due toa better retrieval set and because it has the capacityof giving answer cues.This paper is divided into five sections. The nextsection presents the Question Predication LanguageModel and some of its features. Section 3 introduceshow the the model is used and how the necessary re-sources for its usage were built. Section 4 describessome experiments and present some preliminary re-sults. Section 5 presents the concluding remarks andfuture work. 2 Question Prediction Language Model We describe a language model that focuses onextracting a simple semantic representation of anEnglish text that can be easily stored in digitaldatabases and processed by Information Retrieval(IR) tools. We focus on extracting a particular kindof semantic that help us to find the location of a textthat has some likelihood of answering a question.The model and its semantic are defined as QuestionPrediction (QP).The Question Prediction Language Model(QPLM) represents sentences by specifying thesemantic relationship among its components usingquestion words. In this way, we focus on dividingthe problem of representing a large sentence intosmall questions that could be asked about itscomponents. In other words, we represent therelationship among key words of a sentence as shortquestions. For instance, the sentence “  Jack eatsham ” could be represented by the following twotriples: Who ( eat,Jack ) and What ( eat,ham ) .Using this model it is possible to answer shortquestions that focus on relations existent inside asentence context, such as “ Who eats ham? ” and“ What does Jack eat? ”.The QPLM represents sentences as semantic rela-tions expressed by triples q  ( w,a ) where q  is a ques-tion word, w is the word that concerns the questionword q  and a is the word that answers the relation q  about w . For instance the relation Who ( eat,Jack ) tells us that the person who eats is Jack. The repre-sentationof oursemanticrelationsastriples Q ( w,a ) is important because it allows the representation of  John askedplaced schooleveryflag who what what which where  Figure1: Graph Representationsentences as directed graphs of semantic relations.This representation has the capacity of generatingquestions about the sentence being analysed. Fig-ure 1 shows such a representation of the sentence:“  John asked that a flag be placed in every school ”.Having the sentence of Figure 1 and removinga possible answer a from any relation triple, it ispossible to formulate a complete question about thissentence that would require a as an answer. For in-stance, wecanobservethatremovingthenode John we obtain the question “ Who asked for a flag to be placed in every school? ” where Who was extractedfrom the triple Who ( ask,John ) . The same is validfor other relations, such as removing word school toobtain question “ Where did John asked for a flag tobe placed? ”. The name Question Prediction for thismodel is due to its capability of generating questionsregarding the sentence that has been modeled.In this section, we have shown how our modelrepresents the semantic information. In the next sec-tion we focus on the implementation of QPLM andits usage. 3 Building and using QPLM As observed in Figure 2, a training set of QPLMtriples was created using mapping rules from a cor-pus of semantic role labels. Using a syntactic parserand a NE recognizer with our training set, we wereable to learn pattern rules that we further applied inthe processing of the AQUAINT corpus.PropBank (Palmer et al., 2005) is a corpus withannotated predicate-argument relations from thesame newswired source of information as the PennTreebank  2 . We used PropBank as our starting point 2 http://www.cis.upenn.edu/ treebank 93  handcraftedmapping rulesPropBankQPLMtrainingdataautomatedlearningPropBankwith parse treesand named entitiesQPLMpattern rulesConnexorLingPipeAQUAINTQPLM annotatedAQUAINT Figure 2: Creation and usage of pattern rules.because it comprises the same textual style, and thepredicate-argument relations (also referred to as se-mantic roles) can be mapped to QPLM triples.We studied the possibility of using semantic rolelabeling tools to perform the semantic annotation,however our experiments using these tools showedus that they have not yet achieved a reasonable speedperformance. For instance, the SwiRL semantic rolelabeling system 3 would take a couple of years tofully process the AQUAINT corpus. In contrast, oursystem takes a couple of days if all the necessaryinformation is already at hand; adding the time re-quired for syntactic parsing and NE recognition, thetotal processing period is not longer than two weeks. 3.1 Training corpus PropBank is processed through a set of map-ping rules from the predicate-argument relations toQPLM. Using a PropBank map as our training datagives us the benefit of a large training set, but atthe same time it will only create relations that arepresent in PropBank, therefore excluding some rela-tions that we wish to include. For instance, relationsthat do not involve any action, such as the ownershiprelation in ( Whose ( car,Maria ) ) and the quan-tity relation in ( HowMany ( country,twenty ) )),among others.PropBank defines relations between predicate andarguments without properly defining their meaning.On the other hand, it does keep a format where theargument number 0 represents the agent acting uponsomething and argument number 1 represents pa-tients or themes. PropBank was manually anno-tated according to the PropBank Marking Guide- 3 http://swirl-parser.sourceforge.net/ lines (Babko-Malaya, October 2006). The guide-lines represent an effort to build a consistent set of relations, however a closer look at the corpus showsthat consistency is a hard task to achieve, particu-larly with the vaguely defined arguments number 3onwards. For those cases the inclusion of a functiontag proved to be useful 4 .Observing how arguments and predicates relate toeach other, we created a set of rules mapping fromargument-predicate relations to the QPLM. The ba-sic differences between both models is that theQPLM triple contains a label representing a morespecific semantic relation, and that it associates onlythe head of the linked phrases. For instance, thesentence “ The retired professor received a lifetimeachievement award  ” is represented as: (1) Semantic Roles : [The retired professor] ARG 0 [received] pred [a lifetime achievement award] ARG 1 .(2) QPLM : Who(receive, professor), What(receive, award) As can be observed in (1), semantic role label-ing does not provide information about which is themain term (normally the head of a phrase) of eachargument, while in (2), QPLM represents relationsbetween the phrase heads. In order to find the phrasehead, we applied a syntactic parser (Connexor 5 ) toPropBank sentences. However, the phrase headsare not always clearly defined (particularly when thesyntactic parse tree is broken due to problems in theparser) creating an extra difficulty for the mappingprocess. When a syntactic path cannot be found be-tween predicates and any of the words from the ar-gument, we then try to find the head of the phrase 4 Afunctiontagisinformationattachedtotheargumentsrep-resenting relations such as negation, location, time and direc-tion. 5 http://www.connexor.com 94  by syntactically parsing the phrase by itself. If thisalso fails to provide us with a head, we simply usethe first available non-stopword if possible.The stage of finding the related phrases headsshowed to be quite important, not only because wewould be defining which words relate to each other,but also because if a broken parse tree is found, norulescould belearnt fromtheresulting QPLM triple.An analysis of the data showed us that 68% of theQPLM triples derived from PropBank were gener-ated from an unbroken parse, while the rest usedsome of the other methods.We understand that even though our model hassimilarities with Semantic Role Labeling, we aretaking a step further in the sense of semantic repre-sentation. QPLM has a finer semantic representationmeaning that a predicate argument relation in Prop-Bank might have different representations in QPLM.Our mapping rules takes into consideration not onlythe number of the argument but also the predicateinvolved and the POS or NE of the related words.Even though we cover different aspects of Prop-Bank in our mapping, we observed that many pred-icates hold different meanings for the same argu-ments which creates a problem for our mappingstrategy. This problem was not fixed because of the prohibitive amount of work needed to manuallymark all the different meanings for the same pred-icate in different sentences. In these cases, wherethe same predicates and the same argument repre-sent different semantics according to the QPLM, wechose the one most representative for the set of sen-tences using that predicate and argument. For in-stance, the argument number 3 of predicate spend  for the majority of the cases represents a quantityof money that was spent (a HowMuch label), how-ever we have one case where the argument is cash (aWhat label). This type of mapping compromises theaccuracy of our conversion, however a randomly se-lected set of 40 documents was manually evaluatedshowing that nearly 90% of the QPLM triples werecorrectly converted.After the mapping was finalized we obtained atraining set of rules with 60,636 rules, and 39 typesof semantic relations (Table 1). aboutwhat do outofwhatadv forwhat overwhatafterwhat fromwhat subjagainstwhat how towhataroundwhat howlong towhomaswhat howmuch underwhatatwhat howold whatbehindwhat intowhat whenbelowwhat inwhat wherebeneathwhat likewhat whobetweenwhat obj whombeyondwhat ofwhat whybywhat onwhat withwhat Table 1: QPLM Semantic Relations Original : John kicked the ball bought by Susan. QPLM  : Who(kick, John), What(kick, ball), What(buy, ball),Who(buy, Susan) Parse Tree : John npsubj → kick va ← obj ball nn ← det the det ball nn ← mod buy vp ← agt by prep ← pcomp Susan np  Named Entities : < ENAMEX Type=NAME > John <  /ENAMEX > kicked the ball bought by < ENAMEXType=NAME > Susan <  /ENAMEX > . Table 2: Training Files 3.2 Rule learning The PropBank corpus, after being automaticallyconverted to QPLM triples, is used to learn the rulesthat are used to find the QPLM information of plaintext. The QPLM annotation relies on the output of a syntactic parser and of a named-entity recognizerfor its annotation and for the rule learning process.We are currently using Connexor for syntax pars-ing and LingPipe 6 to recognize NEs. Our seman-tic model uses pattern rules (PRules) created fromthe representation of the same sentence as syntacticparse trees, MUC style named entity, and a list of QPLM triples. Table 2 presents the different infor-mation that we use for training.Having these representations at hand, a set of rules is learned using the following process (see Fig-ure 3 for an example):1. replace the part of speech information with therespective named entity category in the syntac-tic parse tree; 6 http://www.alias-i.com/lingpipe/ 95  2. identify leaf-to-root links along the combinedsyntactic and named-entity (S+NE) path be-tween w and a for every triple Q ( w,a ) ;3. for the existing S+NE paths, replace w and a by a marker in both the triples and the paths,registering those as pattern rules (PRule);repeat steps 2 to 3 for all triples and documents;4. combine all PRules found, calculate their fre-quency of occurrence and group them by com-mon triples. It is important to note that if wehave a sentence such as “  Jack eats ”, we wouldhave a frequency of two ( 2 × ) for the pattern a  personsubj → w va . 1. John person subj → kick va 2. Who ( kick,John ) : John personsubj → kick va 3. Who ( w,a ) : a personsubj → w va 4. Who ( w,a ) : • 1 × : a personsubj → w va • 1 × : a personpcomp → by prepagt → w vp Figure 3: Process exampleAfter computing all the training files we wouldhave a resulting PRule file containing all possibleS+NE paths that can generate the manually definedtriples. If an S+NE path could not be found then aPRule cannot be generated and the current trainingtriple is skipped. 3.3 Applying QPLM Using the training corpus described above, we foundall the PRules needed in order to generate the se-mantic triples when having an S+NE representa-tion. The rules are grouped by QPLM triples, havingtheir S+NE paths attached with a frequency value.This frequency value represents how many times anS+NE path was used to generated a PRule in thetraining corpus.To convert S+NE files into QPLM, we start by ap-plying those PRules that have the highest frequencyvalues. ThesePRulesarebelievedtobethemostsig-nificant ones. Also it is important to observe that if an S+NE path generates different QPLM triples, weonlyneedtoapplytheonewiththehigherfrequency.For instance, if the pattern w  personsubj → a va is as-sociated with the triple Who ( w,a ) with frequencyof 8 and with the triple Where ( w,a ) with a fre-quency of 2, the S+NE path will only generate the Who triple. Because frequency is the decisive fac-tor, in the previous example we have 20% of chanceof wrongly assigning an incorrect semantic label.We observed that more precise PRules could becreated taking into account that some verbs con-stantlygenerateadifferentQPLMtripleforthesameS+NE path. These new PRules (which we refer to asFW) are defined with a fixed w becoming less fre-quent but at the same time more precise. The pre-cision of FW rules combined with the generality of the previous ones (which we refer to as GN) assureus that we have a correct analysis of a known verbas well as fair guess of an unseen one. To ensurethat known verbs are evaluated first by the more pre-cise FW rules, we assign a much higher weight tothose rules than GN ones. An evaluation using thecombination of both types of rules has shown us thatassigning a weight 800 times higher to FW than toGN gives us the best results.We also observed that due to the large amount of learnt PRules, the process for creating the QPLMwas slow. In order to improve the speed perfor-manceoftheprocess, wedecidedtocompromiseoursystem precision and recall by removing the leastimportant rules, i.e. those with a frequency equalto one. The lower number of PRules caused a de-crease of recall which is more noticeable when tak-ing into account the FW rules. Even though weexperienced a decrease of precision, removing lowfrequent PRules causes the removal of abnormalPRules that were generated by parsing errors.In the next section we describe the environmentwhere QPLM was applied, followed by some exper-imental results. 4 Evaluation It is possible to evaluate our model implementationon how well it performs the task of assigning thecorrect semantic labels to a certain text. Howeverbecausethemodelwasdesignedsoitwouldimprovethe IR stages of a QA system, we believe that themost significant evaluation at this point is in terms 96
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks