Identifying and Avoiding AI Translation Risks

Law firms are switching from conventional machine translation to AI translation for first-pass understanding, and the data clearly support this move. In Japanese-to-English patent translation, frontier large language models produce one or two minor errors per sentence and one or two major errors per paragraph, similar to first drafts produced by well-trained human translators. This is a major advancement over MT output, in which we would typically see major errors in essentially every sentence. In even better news, LLMs sometimes allow attorneys who do not speak Japanese to dig deeper into ambiguities in important sentences.

For attorneys trying to understand what is and is not disclosed in a Japanese publication, LLM-based AI offers significant advantages over conventional machine translation. The Japanese-to-English pair has always been difficult for MT. There are differences, not only in the structures of the languages, but also in the content of the communication itself. For example, Japanese does not ordinarily use the plural or singular and does not have definite and indefinite articles. From an English language point of view, this information is simply missing. Because old machine translation could not consider the context of the entire publication, it simply defaulted to whichever was statistically more common. LLMs can consider context. It is also possible to interrogate an LLM and ask questions such as, "Does the context also allow for a plural reading of this noun?" In short, AI makes better (or at least better informed) guesses and is capable of discussing them.

This is an improvement, but also a double-edged sword, as it can be more difficult to tell when an LLM is guessing. Conventional MT tended to fail loudly by producing the occasional nonsensical sentence and outputting untranslated words in the original language, making it obvious that human attention was necessary. LLMs are models that guess what a human would most likely say next, on the basis of everything that has been said so far. This propensity to output exactly what you would expect masks failures and hallucinations, so that the overall sentence sounds reasonable even if the translation is wrong.

While context awareness is an upgrade over MT systems that looked at one sentence at a time, it carries its own risk. AI tends to give considerable weight to whatever is in its "context window," including the output that it has generated so far. That is to say, it treats its own writing as a source of truth simply because it is there. As an example of how this can create risk, consider the Japanese word "軸," which can be translated as "shaft," "axle," or "axis." If the word appears six times in your document and the LLM has translated it as "shaft" in the first five instances, it is very likely to translate it as "shaft" in the sixth, even if "axis" is actually the correct choice in that sentence. It may actually go so far as to anchor its translation of that sentence around the word "shaft," and remold the rest of the sentence to fit. What is more, the LLM is likely to defend its choice when queried simply because the previous context predisposes it to do so (autoregressive bias). Because confirmation bias also makes human readers more likely to accept what we have seen in the text so far, this sort of failure is rarely noticed.

AI translation has inherited some of the problems of earlier neural network machine translation, such as Google Translate, which burst onto the scene ten years ago. At the time, I wrote about the specific risks to patent attorneys posed by assumption, smoothing, replacement, omission, false inclusion, and scrambling. LLMs keep all of these and add drift, inconsistency, false consistency, gullibility, and tokenization failure.

This list may make it sound as if LLMs are uniquely problematic, but human translators actually fail in similar ways. At the level of information theory, translation is inherently "lossy." The encoding and decoding involved when a person reads a sentence written in one language and then writes it in another language is subject to mathematically inherent loss, known as equivocation, as well as unavoidable noise and distortion. In professional human translation, these problems are identified and corrected through understanding the technology described, training, supervision, rule sets, and multiple-linguist review.

Attorneys can bring some of these techniques to bear on initial LLM output. For example, you can ask another AI model, or another instance of the same model with a different prompt, to review an LLM translation. Keep in mind that, if you are not fluent in the source text language, this technique will only be useful for surfacing errors and will not let you arrive at a translation that will stand up to scrutiny by an expert translator. It can tell you that something might be wrong, but it cannot tell you that something is definitely right. We have tested these techniques and integrated some of them into our own workflow for expert-certified translations produced by a human team. The next post in this series looks at specific techniques and results.

One thing that we learned early is that it is best to avoid extended conversations about a translation with the same AI. In a long enough exchange, you can get an AI to translate almost any way that you want, and to confidently give hallucinated reasons as to why the translation is correct. These creative endeavors will not withstand scrutiny by a human expert.

The propensity toward agreeableness is perhaps the greatest weakness of LLMs when compared to non-interactive MT. An examiner will sometimes accept a machine translation of a foreign reference, particularly if it is on a public site such as the EPO Espacenet service. But they generally know that an LLM translation provided by an attorney or an applicant will depend entirely on undiscoverable prompts, parameter settings, and context. Unverifiable and unduplicatable, LLM translation is neither a matter of record nor an opinion.

None of this reduces the real advantages of a tool that can provide instantaneous insights into what Japanese documents disclose, as long as we remember that it is a tool and not an authority. The question is not whether to use AI translation but when to rely on the output and when you need a human eye. In the next piece in this series, we look at specific techniques for catching the failures described above, including results from testing current frontier models against a sentence that was ruled on by the Southern District of New York.