In recent years, significant progress has been made in developing reference-free metrics, which aim to assess the quality of machine translation outputs without relying on reference translations.
The Quality Estimation (QE) shared task also adopts the Multidimensional Quality Metrics (MQM) framework for evaluation. Due to this shared framework and evaluation approach, we highly encourage participants from the QE shared task to simultaneously participate in the metrics task and vice-versa!
While the QE shared task focuses on segment-level annotations, gathering human assessments for various “unique” segments, the metrics task takes a different approach. In the metrics task, the main objective is to compare different systems and identify the best system performance on the given test data.
In essence, the metrics task aims to answer the research question: “Which system performs the best on this test data?” On the other hand, the QE task seeks to address the research question: “Is this translation of sufficient quality, or does it require human revision?” It emphasizes assessing the translation quality rather than comparing different systems.
Although the test sets and analysis differ between the two tasks, we believe that a well-tuned QE system can also serve as an effective metric. By participating in both the QE shared task and the metrics task, you can gain valuable insights into the performance of your system from different perspectives.
Furthermore, both tasks use the Codalab submission system and leverage the MQM framework for evaluation, ensuring a seamless and consistent experience for participants.
Join us in exploring the fascinating intersection of quality estimation and metric development by participating in both the QE shared task and the metrics task this year!
The dates for the QE as a metric subtask are exactly the same as the main Metrics task.
| Date | |
|---|---|
| System outputs ready to download | 10th August, 2023 |
| Submission deadline for metrics task | 17th August, 2023 ❗ |
| Paper submission deadline to WMT | 5th September, 2023 |
| WMT Notification of acceptance | 6th October, 2023 |
| WMT Camera-ready deadline | 18th October, 2023 |
| Conference | 6th - 7th December, 2023 |