Description of the jsRealB generator for the E2E Challenge

Guy Lapalme, RALI-DIRO, Université de Montréal

Note: this work was done in October 2018, so these results were not part of the E2E Challenge official competition.

Context

The E2E Challenge dataset was designed to provide data for developing algorithms to generate text from a dialogue act-based meaning representation (MR). The underlying assumption was that this problem should be dealt with a machine-learning (ML) approach. The organizers even provided an ML baseline system.

Although we have great respect for ML, we thought it might be interesting to compare the results of these systems with a much simpler symbolic system, using jsRealB, a bilingual English/French realizer written in Javascript that we have developed over the last few years. We describe here only the English realization, but we also developed a French version having the similar structure for which we mostly translated the words in French and adapted some formulations. Because of the independent random variation between the English and French realization, these sentences are not literal translations of each other, although they convey the same meaning being generated from the same MR. When used in the demonstration page, the realization is performed by the browser, but for the batch tests shown below corresponding to the shared task, we use the same code as a node.js module.

After examining a few sentences of the development set (we never looked at the test set during the development), we designed a template-based text structure, filled with the values of the attributes of MR. In jsRealB, we first build a constituency structure for the sentence with constructors corresponding to the conventional constituent symbols (e.g. S for a sentence, NP for a Noun Phrase, N for a noun, etc...). When the string value structure is needed, most of the details for the final realization (capitalization, gender and number agreement, etc.) are dealt by jsRealB. For more details, see the tutorial and the documentation.

Text organization

The description
A sentence that starts with the name of the place (given by the attribute name), the verb be, the nearby place (attribute near), the region (attribute area), the type of food served (attribute food) and the range of prices (attribute priceRange). Default values or an empty string are generated for missing attributes. This is written as follows in Javascript:
    var name="name" in fields?Q(fields["name"]):NP(D("the"),N(oneOf("restaurant","establishment")));
    var allPlaces=["place","arena","venue","establishment","restaurant"];
    var eatType=NP(D("a"),N("eatType" in fields?fields["eatType"]:oneOf(allPlaces)));
    var near="near" in fields?PP(P("near"),Q(fields["near"])):"";
    var area="";
    if ("area" in fields){
        area=fields["area"]=="riverside"?oneOf(()=>PP(P("on"),NP(D("the"),N("riverside"))),
                                               ()=>PP(P("in"),NP(D("the"),N("riverside"),N("area"))),
                                              )
                                        :PP(P("in"),NP(D("the"),N("city"),N("centre")))
    }
    var advice=S(name,VP(V("be"),eatType,near,area));
    var food="";
    if ("food" in fields){
        var fo=fields["food"];
        if (fo=="Fast food")fo="fast";
        food=A(fo)
        var npFood=NP(food,N("food"));
        var serve=V(oneOf("serve","provide","have","offer"))
        advice.add(oneOf(SP(Pro("that"),serve,npFood),
                         VP(serve.clone().t("pr"),npFood)
                         )
                  )
    };
    var priceRange="";
    if ("priceRange" in fields){
        var pr=fields["priceRange"];
        priceRange=pr.indexOf("£")>=0?PP(P("with"),N("price").n("p"),Q(pr))
                                     :PP(P("with"),NP(A(pr),N("price").n("p")));
        advice.add(priceRange);
    }
The customer rating (attribute customerRating) written in Javascript as follows:
    var customerRating="";
    if ("customer rating" in fields){
        var cr=fields["customer rating"];
        customerRating=S(Pro("I").g("n"),
                         VP(V("have"),
                            oneOf(()=>NP(D("a"),Q(cr),oneOf(N("customer"),Q("")),N("rating")),
                                  ()=>NP(D("a"),oneOf(N("customer"),Q("")),N("rating"),P("of"),Q(cr))))
                         );
    }
The family friendliness (attribute familyFriendly) written in Javascript as follows. jsRealB can realize the negation of a sentence by a simple flag.
    var familyFriendly=""
    if ("familyFriendly" in fields){
        familyFriendly=S(oneOf(()=>Pro("I").g("n"),
                               ()=>NP(D("the"),N(oneOf(allPlaces))),
                               ()=>Pro("I").n("p")
                         ),
                         VP(V("be"),
                            NP(oneOf(N("family").lier(),N("kid")),A("friendly"))))
                       .typ({"neg":fields["familyFriendly"]=="no"})
    }

These three structures are then simply concatenated with a space which forces the English realization.

    return advice+" "+customerRating+" "+familyFriendly;

This simple solution was developed in a few hours (50 lines of Javascript), but it gave good enough results with instantaneous response time all in the browser. The javascript code of the realizer for both French and English.

Evaluation

We then ran the evaluation scripts of the E2E competition on the test set and we obtained the following results (we could not manage to get the METEOR scoring to work on our machine):

ScoreValue
BLEU0.4980
NIST7.3038
ROUGE_L0.5878
CIDEr1.8387

These scores are not the best, but they are competitive. Given the fact, that all sentence are well formed, with good syntax and capitalization, they would probably get good human evaluation scores!

Conclusion

We found this exercise very interesting and we thank the organizers to accept to show an alternative to the usual ML approach for the E2E challenge. It would probably be interesting to see if an ML approach between the attribute values and the constituency structure of jsRealB would be feasible and if it would improve the results.