Our Analysts' Insights

The "yes" vote on Brexit and the election of Donald Trump surprised many media outlets, pollsters, analysts, and political experts. Those that predicted a different result emerged as the biggest losers of public confidence in their forecasting ability. Meanwhile, the U.S. election added the memes of fake news, fauxtos, and foreign interference to the discourse. Commentators across the political spectrum now debate a "post-fact" or "post-truth" world that would be familiar to the fictional character Winston Smith in George Orwell's master work.

What went wrong? News reports cited a variety of problems, including data sampling errors, flawed survey methodologies and designs, and voters who either wouldn't reveal their choices or lied about their decision to pollsters. Predicting the U.S. presidential race was further complicated by the Electoral College, an indirect election process in which voters in each state actually elect representatives who in turn vote for the president. This indirection means that the victor isn't necessarily the winner of the popular vote.

What really struck us in the field of market research and a lot of other people elsewhere – was the failure of the polls to predict the margins that gave Trump his Electoral College victory. Since we began tracking the language sector in 2002, CSA Research has used the same tried-and-true market research tools such as surveys and interviews that political pollsters and analysts employ to call the election results. 

Because we rely on these quantitative and qualitative methods to calculate market size and identify behaviors, preferences, and trends, we reviewed the Brexit and presidential campaigns to determine what effect their outcomes might have on this market research model. Our Annual Survey of the language services market shows what we do differently:

• The process. We directly survey suppliers of language services and technology about company demographics, revenue and where it originates, product and service offerings, business plans including M&A, staffing, clients, and other business details. We use this data to size the overall market, including specific service and technology sectors, and establish benchmarks for a variety of business functions. We also compare the results to the other research that we conduct throughout the year, for both the supply and demand sides of the market on a wide range of topics such as best practices, technology, and trends. Our work for the next year's survey begins immediately upon the publication of the results of our annual report so it's an active, continuing process. 
   
• The methodology. We've conducted our comprehensive Annual Survey of the language sector since 2010, and a predecessor study beginning in 2005. It is based on our market-sizing algorithm, which we developed with our statistician, analysts, and input from across the market. The survey itself is the result of similar attention to market detail, where each question helps us define the state of the market. We design the data collection methods, refine question ordering and wording, define rules to ensure accuracy, and run countless tests. This year-over-year analysis allows us to continuously assess the state of the market on various axes, as well as review historical data and trends that affect business trajectories. 
   
• The database. To qualify for inclusion in the database that we've been developing since 2002, companies must have two or more full-time employees, a minimum level of revenue that varies by country, and visible market activity. These three filters combine to yield more than 18,500 qualifying suppliers. We invite all these companies to participate – then we spend countless hours soliciting the right companies to achieve a representative sample.
   
• A representative sample. Working with our statistician, we calculate how many survey responses from our database we need to reach confidence levels and intervals that ensure data validity and reliability. Because CSA Research has extensive knowledge of the supply chain demographics, we take a targeted approach with LSPs that yields the most representative sample in terms of size, geography, and services. If the data we collect for any sector or region doesn't meet our criteria, we do not make data cuts or correlations. For example, if we have insufficient responses from China or India, we do not break out data for those countries. For other projects, we have suspended or terminated surveys because we did not have the representative sample.
   
• The data collection process. CSA Research confidentially surveys suppliers for their financial and business data. Because more than 99% of suppliers are privately owned, few publish such information unless required to do so by their local authorities. We analyze all answers to eliminate data that is out of bounds for their individual company profile. We disqualify respondents that do not fit our criteria, duplicate data already provided by the company, do not answer all the questions, or answer them falsely (as flagged by our algorithm). For example, in 2016, we rejected more than 200 responses for one or more of these reasons.
   
• Related research. Our Annual Survey collects data that we use in a variety of other ways. For example, we also rank the largest suppliers globally, by size in each of nine U.N.-defined regions, by the rate of growth, for specialties such as over-the-phone interpreting, and CEO characteristics. Participants who want to be considered for ranking electronically confirm that the data they provide is accurate.  

The methodology, supplier database, representative sample, and the data collection process are the foundation of our market research – and show that our approach is strong and reliable. We do not assume correct results, but instead apply tested methods to deliver valid and reliable data-driven research that our clients can trust.