Thoughts on Machine Translation and Post-Editing Analysis

Post analysis is a fairly new concept which I imagine will become one of the primary selling points for agency licenses for the new generation of  CAT tools such as MemSource because it allows agencies to pay translators based on the work actually performed rather than on a snapshot of what the files looked like when they were assigned. For example, if translator A and B are assigned files on Monday, and then translator A translates some common or similar text on Tuesday, and then translator B leverages those TUs through a cloud-based TM on Wednesday, the corresponding repetition discounts would be applied to those segments for translator B. That’s all fine and well, and while we translators may not enjoy seeing our “estimated fees” diminished, it makes good sense, and I think most translators would find the concept to be acceptable.

We translators may not enjoy seeing our rates discounted, but as long as such discounts accurately reflect the actual effort made, we are likely to accept them.
However, when MT is thrown into the mix, the issue of post analysis suddenly turns into a controversial one. Consider this scenario: a translator uses the MT feature within a CAT tool, making the appropriate edits, and then the post analysis calculates the similarity of the confirmed TU against what was provided by MT and presents the results in the post analysis report. In the words of the MemSource help file, “[the post-editing analysis] analyzes the MT post-editing effort for each segment and compares the machine translation output with the final post-edited translation. Therefore, if the machine translation output was accepted without further editing (the linguist did not need to change it at all), it would come up as a 100% match in the analysis.”  Source for quote.

My concern is that including MT in the post analysis seems to suggest that post-editing of MT should be dealt with the same as TM leverage, which, of course, is absolutely absurd.

Let’s look at this logically: when I leverage a human TU, Trados, MemoQ, etc. is nice enough to highlight the differences for me, so I might change a date, a name, a verb conjugation, etc. and only quickly glance at the rest of the segment because I know that I, or a trusted colleague, has already checked it. This, in a nutshell, is the justification for repetition discounts: when TUs have already been checked by a human being, less effort is required to translate a similar segment, and so the new segment is discounted. Again, we translators may not want to charge less, but most would agree that this concept is fair and in everyone’s interest, and so it has become generally accepted practice in the industry.

On the other hand, let’s say I am using the MT feature within a CAT tool. If I perform just a few changes to a MT segment, the post-analysis may deem my efforts to be 90% similar to the original MT. However, despite the apparent similarity, the amount of effort required to edit MT is entirely different than that required to recycle my own or a colleague’s work in the form of a match extracted from a specially assigned, checked, and subject-matter and client-specific TM which has presumably already been checked by one human being and possibly even proofread by a second. In contrast, when leveraging MT, there is obviously no presumption that the results will be relevant, consistent, accurate, etc. and so a translator must focus his or her efforts on the entire segment, not just the changed or new bits. If you are still not convinced that TM and MT leverage are two entirely different, just consider what happens when “the machine” gets lucky and a segment requires no change at all. In this case, the post analysis would deem the segment a 100% match, despite that fact that confirming the accuracy and suitability of the segment may still require significant effort by the translator. After all, we are not going to start taking Google’s word for it, are we?

As things stand today, translators should be wary of enabling the MT feature within a CAT tool for jobs to be paid on the basis of post-analysis, but what is the long-term solution? Obviously, it would not be in anyone’s interest to turn our backs on a tool that has the potential to drastically increase output and efficiency, but at the same, it is simply unreasonable to bill MT leverage according to established repetition rates based on human-confirmed TUs. The most obvious solution would be for translators and agencies to establish a completely separate rate structure for calculating MT in post analyses which recognizes the effort required by translators to use MT properly while still providing agencies the discounts they need to remain competitive. The table below offers a traditional discount scheme for human TM leverage on the left as well as a proposed discounts scheme for leveraging MT in the post-analysis on the right.

Traditional repetition discounts Proposed MT post-analysis discounts
% Similarity % of Rate to be Billed % Similarity % of Rate to be Billed
100% 0-30% 100% 80%
95-99% 50% 95-99% 85%
75-94% 75% 75-94% 90%
0-74% 100% 0-74% 100%

Obviously, the issue of translation rates, whether or not to discount them, and if so, the percentage of such a discount is something that must be carefully considered and negotiated between translators and agencies in each case. However, the mere potential to establish a discount scheme for leveraging MT such as the one above demonstrates two points:

  • First, leveraging MT is entirely different from leveraging TM, and therefore, any discounts applied for MT leverage must be significant less than those applied for traditional TM matches.
  • Second, it is in fact possible to use MT within a CAT tool in a way that is transparent, in accordance with professional ethics, and in the interest of all of the stakeholders in a translation project.

We translators may not enjoy seeing our rates discounted, but as long as such discounts accurately reflect the actual effort made, we are likely to accept them. Besides, given that cloud-based TM is rapidly being adopted as standard practice for most translation agencies, the concept of post-analysis is likely to be adopted as a natural consequence of working in “the cloud”.

This article originally appeared as a guest posting on Emma Goldsmith’s blog Signs and Symptoms of Translation. As the issue of post-analysis and how LSP’s are likely to apply it is something that I have become concerned about, I wrote this in response to Emma’s November 2012 posting entitled MemSource and Studio 2011: a side-by-side comparison.

The image that appears on this page is courtesy of Stuart Miles / FreeDigitalPhotos.net

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.