How Do You Evaluate the Quality of Your Translations, and Why You Should Not Trust a Friend to Give You an Opinion on Translation Quality?

By August 22, 2017General, Translation
translation-quality-evaluate-definition

The question of “quality” is a surprisingly difficult issue to pin down with certainty in the world of business translation. The problem arises partly because it seems that every LSP with a website claims to have wonderful quality, and often because the translation buyer does not understand the target language (the product of the service) well enough to make a competent assessment on what they receive as translation product.

So how does one tell when the quality story is real or not? There are also no independent evaluation agencies available to provide translation buyers with meaningful and objective clues on the quality of the service offerings of different LSPs. Imagine buying a car with no knowledge of key specifications, reliability ratings, crash-safety data, or any idea of how much ongoing maintenance is required on the car. Unfortunately, this is often the situation a translation buyer finds themselves in. We at MasterWord believe this is an important issue for buyers to understand, as educated and knowledgeable customers tend to also be the most satisfied customers. This post is an attempt to clarify this issue for all existing and future customers, and also raise the quality of the customer experience of all our customers.

The difficulty arises because of the following factors:

  1. Subjective Evaluation of translation product: Sometimes this can be as arbitrary as asking a friend who spent a summer in Spain to assess a Spanish translation!
  2. Buyers have limited or no knowledge of the target language: While some companies may have in-country staff to review translations, many companies may not have access to these resources and thus have to depend on the LSP to deliver a quality product.
  3. Lack of agreement on what quality means: Translation in the business context needs to be suited to purpose and some content e.g. legal liability related content, and core marketing messaging, needs more attention than technical support content that may only have a shelf life of a month. In a business translation situation the price and timeliness of the work has to be balanced with the value of the translation in the overall business mission.
  4. Expectations that are not aligned with budget and time reality. The best quality requires multiple eyes and levels of review which can be done at a higher budget and with more time.

The driving force behind translation of business content is to find success in an international market. The translated content needs to hit the mark in each country and region in terms of suitability, appropriateness, usefulness, and accuracy in the information about your products or services. Translation errors can be embarrassing or even seriously damaging when they result into culturally inappropriate or completely wrong messages. At best, translation errors and awkward syntax will only damage your brand reputation and image; at worst, they may cause financial and legal problems that threaten your company’s future in an international market.

So, what is quality? Is it possible to have a basic definition of this to enable better communication on the subject? While the following description in not perfect or all-encompassing, it facilitates us getting a basic understanding of the scope of the subject. It is important for a buyer to understand how their LSP measures quality and know that this quality assurance mentality is properly integrated into their standard production process. Agencies that don’t have a de facto quality focused culture and production process are not likely to be able to produce a quality service.

Here are the basic criteria that a Translation Service Provider offering a quality service should fulfill:

A. Translation

  • Correct transfer of information from the source text to the target text.
  • Appropriate choice of terminology, vocabulary, idiom, and register in the target language.
  • Appropriate use of grammar, spelling, punctuation, and syntax, as well as the accurate transfer of dates, names, figures, etc. in the target language.
  • Appropriate style for the purpose of the text

B. Work process

  • Certification in accordance with national and/or international quality standards.

 Thus, we see that there are two clear dimensions to the issue:

  1. The actual translation work itself described by a) above, and
  2. The translation production process which can be substantiated by certifications from standards bodies described b) above. This assures a buyer that a strict and certified production process is in place and thus provides assurance that a quality deliverable is the likely outcome on a regular and consistent basis. These certifications have a strong focus is on administrativedocumentationreview and revision processes not just the quality assessment of the final translation.

However, the translation quality could mean different things as Common Sense Advisory points out and it is important to ensure that the same reference point is being used.

5 Definitions of Translation Quality


Perfection

The translation meets ideals for accuracy and language


Compliance with Specifications

The translation adheres to specifications within a reasonable tolerance


User Satisfaction

The translation satisfies users, is fit for use, and meets their needs


Process-Based

The translation is produced according to process


Value

The translation delivers the best value for the money

Figure 1 CSA Translation Quality Categorization

Common Sense Advisory has proposed the following definition:

“A quality translation demonstrates accuracy and fluency required for the audience and purpose and complies with all other specifications negotiated between the requester and provider, taking into account end-user needs.”

LSPs that have considered the quality issue deeply will have clearly documented procedures and processes for ensuring quality along both of these dimensions. The best ones will have very clear communications with clients and translators continuously along these dimensions that are triggered by well-established practice and experience. This ensures that all the people involved in a translation project have a clear idea of the work expectations and have an understanding of how their work will be further processed and evaluated before delivery to a final customer.

In addition to possessing the necessary certifications, described by ISO 9001:2008ISO 13611:2014EN 15038, which focus on a disciplined and certifiable overall production process, your language service provider (LSP) should also conduct regular Translation Quality Assurance (TQA) evaluations.

Translation Quality Assurance is the process of evaluating a translated text to ensure that it meets your company’s quality standards. If your LSP is not performing regular TQAs—and performing them correctly—it is likely that your translations will contain grammar errors, inappropriate style, and poor content messaging. The key to having an effective TQA process is that the process be objective.

Core-2015-05-27

Figure 2 MQM Error Classification Scheme

So how do you know if the quality that you’re receiving is the quality that you should be receiving? One strategy is to request TQA reports from your translation service provider to learn about their quality assurance process and to find out how often they perform TQA assessments. LSPs that perform these assessments regularly develop a very well calibrated understanding of the quality of translation deliverable. They can quickly identify problems in the supply-chain and thus maintain long-term consistency. Since there can be varied and disparate opinion on what constitutes quality it is easier to agree on what constitutes an error rather than on what constitutes “quality” in the abstract, and that an important factor in quality is the absence of errors. Thus we see that the EU MQM schema shown above and the TAUS DQF quality measurement focus very specifically on a range of different error types to establish quality scores and ratings.

dqf_subset-2015-06-12

Figure 3 TAUS DQF Data Quality Framework

Let’s examine three components that should be part of your LSP’s Translation Quality Assurance process:

Classifying Errors

A TQA should do more than simply determine whether a translation “passes” or “fails;” it should identify the underlying cause of errors in order to improve the translation process. At MasterWord, we use three error classifications—Errors of Meaning, Errors of Form/Grammar, and Errors of Compliance—and we grade each error based on three levels of severity—Critical, Major, and Minor.

Errors of Meaning: Examples include using the wrong term, forgetting to include a word, or adding an unnecessary word.

Errors of Form/Grammar: These include syntax, spelling, and punctuation errors.

Errors of Compliance: Examples include glossary compliance errors, register/style guide compliance errors, and literal translation/fluency errors.

Grading the Severity of Errors

Some errors are minor and do not affect the understandability or usability of the content. Others are more severe and could damage your brand or even cause financial or legal problems.

At MasterWord, we use the following categories to grade the severity of errors:

Critical Errors: These errors render the content unfit for use. They may carry legal, safety, health, or financial consequences, or result in potentially offensive statements. Critical errors may also prevent the reader from understanding the content as intended or present incorrect information that, if acted upon, could result in harm to the reader. A particularly bad grammatical error that changes the meaning of the text would be considered a critical error.

Major Errors: These errors result in a significant change in meaning that impacts the understandability or usability of the content but does not render the content unusable. For instance, a misspelled word can make it difficult for the reader to comprehend the intended meaning and, therefore, it would be considered a major error.

Minor Errors: These errors do not change the meaning or impact the understandability or usability of the content. If an extra space appears after a full stop, for example, this is considered an error, but it would not compromise the understandability or usability of the text.

Grading Translated Documents

Each translated document that is part of your TQA should be individually graded. The grade will determine whether the translation will be published, edited, or rejected entirely. It will also help your Project Manager identify the translators who have the knowledge and skills to translate your documents and those who are not qualified to do so.

At MasterWord, we grade translated documents based on three criteria:

  • The number of translated words;
  • The number of errors; and
  • The severity of errors.

Translations that score between 86% and 100% are considered satisfactory and acceptable. They typically require either light editing or no editing at all. This is an embedded process so we always know how we are doing on every project that we undertake.

Translations that score between 80% and 85% require moderate editing or proofreading, and documents that score between 75% and 79% need extensive editing. Translations that score below 75% are rejected and require re-translation.

Regular TQAs are necessary to ensure that your translators are operating at their highest level of excellence and only within their strongest area of subject matter expertise. They improve the quality and consistency of your translated documents, strengthen your brand, and prevent legal and financial problems.

The existence of such well-established QA processes assures that MWS produces high quality translations consistently and on an ongoing basis. The objective nature of this process ensures that stakeholders always have very specific issues to address, and avoids situations where you might have team members at a client who say they “don’t like a translation”. However, it is often said, “You cannot manage what you cannot measure.” The more objective a TQA process, the easier it is to manage and adjust for very specific client requirements. Some projects may require less rigor as the quality requirements are lower and others may require more careful QA with multiple steps of review to ensure it is of the highest quality. Without objective ways to measure the quality of our work, we are left at the mercy of fickle evaluations by lay people who can be highly subjective and not entirely fair.

Over the years MasterWord has learned that the following steps ensure that expected quality levels are reached rapidly with minimal re-iteration and maximum satisfaction.

  1. Define The Quality Expectations: This is most easily done by stating the kinds of errors that are “acceptable” or noon-critical and linked to an objective measurement like a TQA score.
  2. Communicate Quality Measurement Scheme: Like our TQA for example together with samples so that customers understand clearly what they will see.
  3. Provide Adequate Context & Guidance: A style guide, terminology and other reference materials help greatly to achieving better quality.
  4. Provide Support for Queries During Translation: Translators often have questions on how key aspects of the translation should be handled. Timely responses to this ensure the project will have better results.
  5. Provide Terminology Guidance
  6. Provide Effective Feedback Mechanisms – this can with regard to the source material and quality and also to key required stylistic and terminology needs.
  7. Use collaboration software to enable in situ

 For more information on improving the quality of your translations, reducing costs, and saving time, please call us at +1 281-589-0810 or +1 866-716-4999 (option 3 for translation) or email us at masterword@masterword.com. We are available 24 hours a day, seven days a week.

References:

Translation process, specifications guidance:

Leave a Reply