admin管理员组文章数量:1565250
Good applications for crummy machine translation
— Kenneth W. Church & Eduard H. Hovy, 1993
- There is a risk that eval can devolve into mindless metrics.
- The success of the eval often depends very strongly on the selection of an appropriate application.
- It is wise to identify the niche application first (where strengths of machines are valued) and then we will be in a much better position to address evaluation questions, and then steer them towards high-payoff niches of functionality.
- Agree with ALPAC that this basic research could not be justified in terms of short-term return on investiment. When compared with human capabilities, MT systems of the time were not deemed a success, and might never be.
Has anything changed since ALPAC?
→ Increasing commercial value
The application venue of MT has been shifted from government to industry, so as the punding providers. One must choose an application that exploits the strengths of the machine and does not compete with the strengths of human. This point is well put in the following: The question now is not whether MT is feasible, but in what domains it is most likely to be effective... The object of an evaluation is, to determine whether a system permits an adequate response to given needs and constraints. --- Lehrberger and Bourbeau, 1988
The blame is to be laid on the desire for generality
In spite of all the literature on MT, the general evaluation measures often fail to pinpoint the strengths of systems. They seem to confound important and less important aspects. Unfortunately, this failure seems to be characteristic of many of the task-independent evaluation metrics. We propose that MT eval metrics should be sensitive to the intended use of the system. And it becomes crucial to the success of an MT effort to identify high-payoff niche application so that the MT system will stand up well to the eval, even though the system might produce crummy translations. By and large 这有一点grants-oriented的嫌疑但是世界又需要这样的effort
Traditional Eval Metrics
- System-based
Tied to a particular system, can’t be used effectively f
本文标签: FormulasuccessModestypapersummary
版权声明:本文标题:[Paper Summary: Modesty is the Formula for Success] Good applications for crummy machine translation 内容由热心网友自发贡献,该文观点仅代表作者本人, 转载请联系作者并注明出处:https://www.elefans.com/dongtai/1726617600a1078370.html, 本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容,一经查实,本站将立刻删除。
发表评论