Machine Learning: The Future Of Data Insights. The Time To Start Is Now (Part 1 of 2)

Leveraging data to enhance the ability to make business decisions is an essential priority for most businesses. Traditionally, however, the investment needed for purchasing, implementing, and maintaining the required tools and training employees on using them has been very high. The cloud and advanced machine learning technologies enable a new generation of business intelligence and data analytics services that are very affordable and easy to use, and therefore suitable for both large and small enterprises.

Learning to leverage modern business intelligence and machine learning technologies and services is an urgent priority for all companies that want to further enhance their decision-making and business performance, and be a leader in their industry vertical.

The ability of the cloud to significantly reduce IT hardware and software costs is nowadays well understood. However, business executives outside of the technology industry are usually less familiar with how business intelligence (BI) and data analytics tools powered by machine learning (ML) and natural language processing (NLP) technologies can significantly ease adopting a data culture, generating better insights from data, and enhancing decision-making to drive better business performance.

This article is for you if:

• You want to learn how the cloud, ML, and NLP technologies can help large and small companies use data to enhance business decision-making and performance (Part 1).
• You need to identify some BI and data analytics tools powered by ML and NLP that are affordable, easy to use, and require no prior experience in ML to get started (Part 2).
• You must understand how to quickly start leveraging a modern cloud-based BI and data analytics solution powered by ML and NLP (Part 2).

After reading this article, you'll walk away with a better understanding of:

• Four powerful data analytics capabilities powered by ML and NLP designed to help businesses generate better data insights (Part 1).
• Amazon QuickSight, a new cloud-based BI service by Amazon Web Services (AWS), currently offering the lowest total cost of ownership (TCO) and the most advanced set of ML-powered functionalities in a BI tool (Part 2).
• Amazon Forecast, a new fully managed and affordable service by AWS to deliver highly accurate time-series forecasts that is based on the same ML technology used at Amazon.com and requires no prior ML expertise to get started (Part 2).
• An agile approach to adopting BI and data analytics tools powered by ML and NLP suitable for any company that wants to generate better data insights and enhance decision-making processes across the entire organization (Part 2).

Machine learning: the future of data insights

Leveraging data to enhance the ability to make business decisions and improve business performance across all functional domains and at all levels of an organization is today an essential priority for more and more businesses.

Also, the amount of relevant data continue to grow exponentially and includes:

• Traditional structured data generated by IT systems (i.e., Financial Planning and HR Systems, ERPs, CRMs, Marketing Automation Platforms, etc.).
• Unstructured data generated by customers (i.e., customer service messages, product reviews, social media posts, etc.).
• Real-time streaming data generated by digital touch points and devices (i.e., events data generated by web and mobile applications, video feeds generated by video cameras, data generated by industrial sensors and other IoT devices, etc.).

Unfortunately, the tools that many companies still use to generate and consume data insights like spreadsheets and charts, traditional BI tools, and small teams of business and data analysts result in processes that often are too expensive, not responsive enough, require data skills too advanced for most employees, and do not leverage all the potentially available data. Hence, many companies have not yet been able to leverage the power of data to improve business performance fully.

Given the scale of today's data, finding the most valuable insights is like looking for a needle in a haystack.

But affordable and easy to use tools are now available. To automate the generation of data insights, facilitate more conversations about what matters, and give anyone - regardless of level or skill set - the ability to understand and use data in their jobs every day, business executives need to understand and adopt four essential data analytics capabilities powered by ML and NLP technologies:

Auto-narratives

Data storytelling, the practice of building narratives around data sets and visualizations to help convey the meaning of the data (i.e., the what, the why, and the so what), is widely recognized as the fastest and best way to empower business teams to understand and act on data through the power of stories (i.e., Forbes, Jan. 2020 - Why data storytellers will define the next decade of data).

Data storytelling helps companies build a data-driven and transparent culture, create conversations about what matters, and free people to do what they were hired to do, without expecting every employee to become a data whiz spending hours combing through spreadsheets and charts.

However, having teams of business and data analysts writing narratives to enrich reports and dashboards is an expensive proposition that many companies cannot afford. To solve this problem, new software tools powered by ML and NLP have recently emerged that can automate generating summaries of dashboards in plain language and interpreting and describing data insights so that everyone can have a consistent and shared understanding.

An auto-narrative is a natural-language summary widget that displays descriptive text instead of charts. Data analysts can embed these widgets throughout their analysis to highlight key insights and callouts when creating their dashboards.

With auto-narratives, dashboard users don't have to sift through the visual, drilling down, comparing values, and rechecking ideas to extract a conclusion. The conclusions that can be extrapolated from the data are already stated in plain language, and everyone can share a single interpretation.

Typically, auto-narrative software generates automated interpretations of the charts and tables included in a dashboard and provides several pre-defined insights in natural language (i.e., period over period change, minimum and maximum values, top and bottom ranking values, etc.). In most cases, dashboard authors can also use an ad-hoc narrative editor to create completely custom narratives.

Using pre-defined and custom narratives, dashboard authors can effectively tell the story of the data in plain language, and the story will be automatically updated whenever the data in the dashboards will change.

Some BI tools come with embedded auto-narrative functionalities (i.e., Amazon QuickSight and Microsoft PowerBI). Standalone software applications are also available (i.e., Narrative Science and Arria for BI) to add auto-narratives to BI tools lacking this capability.

The best tools are affordable and easy to use, and large and small companies should adopt them now.

Natural language query powered by ML

ML-powered natural language query (NLQ) capabilities empower business users to ask questions about their data using everyday business language and get answers in seconds.

For example, users can simply type "show me last year weekly sales in california versus new york" or "what are the best selling categories in california" and get an instant answer in a visualization.

Without NLQ capabilities, when a new business question arises and no answer is found in the existing data dashboards, business users must submit a data request to their BI teams and often wait several weeks for their question to be answered and added to the dashboard.

Using a BI tool with NLQ capabilities, business users can get answers to their questions instantly and reduce the burden on their thinly-staffed BI teams. All they need to do is to type in their question in a search bar using plain language. As they type, the best tools provide auto-complete suggestions, spell check, and acronym and synonym matching, so users don't have to worry about typos or remember the exact business terms in the data.

Using deep learning and machine learning (to power natural language processing and schema understanding), trained over multiple data points and business areas, the most advanced tools can extract business terms (such as revenue, growth, allocation, etc.) and intent from a user's question, surface the related data from the source, and return the answer in the form of numbers and graphs.

The best come pre-trained on data from various domains and industries like sales, marketing, operations, retails, human resources, pharmaceuticals, insurance, energy, and more, to understand complex business language without requiring users to learn any specific syntax to ask questions. They also continuously improve accuracy over time by learning from user interactions from within an organization and provide users with the ability to share feedback regarding the correctness of an answer.

Datasets enabled for natural language querying need to be correctly setup. The best tools leveraging ML require only a minimum set of activities during the preparation phase. Other tools offering NLQ capabilities with less advanced ML functionalities require significantly more time to correctly set up each dataset and offer a more constrained user experience.

Currently, among BI tools Amazon QuickSight provides the most powerful implementation of this capability and Microsoft PowerBI provide a similar, although less powerful, functionality. A few dedicated software tools are also available (i.e., Arria Answer or Knowi) that can be used to add this capability to other BI tools.

The best tools are affordable and easy to use, and large and small companies should adopt them now.

ML Anomaly Detection

What happened that doesn't usually happen, and why did it happen?

Identifying unanticipated issues and opportunities early on, together with their critical contributing factors, is an essential task for any business executive.

For example, a VP of Sales might want to monitor weekly sales data to identify sudden drops in sales potentially caused by unanticipated competitive actions, and a VP of Operations might want to monitor hourly vibration and temperature data to identify equipment requiring maintenance.

A VP of Sales responsible for 2,000 product families sold at 200 retail locations, for example, would need to ask their staff to monitor about 1.60 million business metrics each month (i.e., 2,000 x 200 x 4 = 1.60 million). A VP of Operations responsible for 5,000 pieces of equipment located in several manufacturing locations and operated for 16 hours each day would need to ask their staff to monitor 3.36 million metrics each month (i.e., 5,000 x 2 x 21 x 16 = 3.36 million). Having business and data analysts perform this work manually would be very expensive, and in many cases not possible.

ML-powered anomaly detection software tools can continuously analyze millions of metrics to discover hidden trends and outliers in data. They provide business users with deep insights that are often not visible in aggregate metrics and not scalable with manual analysis.

With ML-powered anomaly detection, business users can find outliers in their data without the need for manual analysis, custom development, or ML domain expertise. Some outliers will represent significant events, and others will represent random occurrences. Analyzing the factors that contributed to an outlier event will uncover which data is worth investigating.

ML-powered anomaly detection it currently used across several industries (i.e., retail, manufacturing, healthcare, fintech, etc.) and common use cases include the following:

• Detecting anomalies in business metrics to identify potential issues and opportunities (i.e., revenues, costs, customer experiences, partners' performance, etc.).
• Detecting anomalies in IT software and hardware performance.
• Detecting unexpected drops in retail store traffic using computer vision.
• Spotting product defects and automating quality inspections using computer vision.
• Improving operations with computer visions at the edge: finding bottlenecks in industrial processes and assessing workers safety within facilities (i.e., people not wearing the required protective equipment or stepping into unsafe zones).
• Detecting abnormal behavior in industrial equipment or fleet vehicles to enable preventive maintenance using data from IoT sensors.
• Preventing fraud in several activities (i.e., payments, insurance claims, credit applications, drug prescriptions, etc.).

Currently, among BI tools Amazon QuickSight provides the most powerful implementation of this capability and Microsoft PowerBI provide a similar, although less powerful, functionality. A few dedicated software tools are also available (i.e., Amazon Lookout for Metrics, Vision, and Equipment, Anodot, Crunch Metrics, and Outlier) that can be used to add this capability to other BI tools.

The best tools are affordable and easy to use, and large and small companies should adopt them now.

ML Forecasting

If everything continues to happen as expected, what will happen in the future?

Companies today use everything from simple spreadsheets to complex financial planning software in a bid to accurately forecast future business outcomes such as product demand, resource needs, and financial performance. Forecasts for product supply and demand can be used for optimal inventory management, staff scheduling and topology planning, and are more generally a crucial technology for most aspects of supply chain optimization. But high accuracy remains elusive for two main reasons:

• First, traditional forecasts struggle to incorporate large volumes of historical data, missing important signals from the past that are lost in the noise.
• Second, traditional forecasts rarely incorporate related but independent data, which can offer important context (such as prices, holidays/events, locations, marketing promotions, etc.). Without the full history and the broader context, most forecasts fail to predict the future accurately.

Adopting a forecasting tool that uses machine learning to combine time series data with additional variables to build forecasts, companies can significantly increase the accuracy of their predictions.

Historically, only large enterprises with significant financial and technical resources could take advantage of this technology given the need to build, train, and deploy custom forecasting applications powered by machine learning.

With the recent introduction of fully managed cloud-based ML-powered forecasting services, however, there are no servers to provision, and no machine learning models to build, train, or deploy, and this technology is finally accessible to small and mid-sized companies as well.

Companies only need to provide historical data, plus any additional data that they believe may impact their forecasts. For example, the demand for a shirt's particular color may change with the seasons and store location. This complicated relationship is hard to determine on its own, but machine learning is ideally suited to recognize it. Once a company provides its data, these tools will automatically examine it, identify what is meaningful, and produce a forecasting model capable of making predictions that are significantly more accurate in as little as a few hours.

As a consequence of the ubiquity of time series data, it is easy to find forecasting problems everywhere. However, a key question is whether there truly is a need for solving a forecasting problem or whether an alternative approach would provide a better solution (i.e., as in the case of identifying the best product recommendations for any given customer). Posing this question is important because, scientifically speaking, forecasting is among the hardest problems in machine learning.

The next important question to ask is why does a company need to solve the forecasting problem? In many business settings, forecasting is usually just a means to an end. For example, for demand forecasting in a retail context, the forecast may be used to make inventory management decisions. The forecast problem is typically an input to a decision problem, which in turn may be modeled as an optimization problem. Examples of such decision problems include the number of units to be purchased or the best approach to deal with the existing inventory.

Currently, among BI tools Amazon QuickSight is the only service that provides a basic ML-powered forecasting capability (i.e., the tool only uses one ML algorithm and cannot incorporate related independent data). However, a few dedicated and more powerful fully managed services are available that can be used for very demanding applications and that can be integrated with most BI tools (i.e., Amazon Forecast, Anodot, and PredictHQ).

The best tools are affordable and easy to use, and large and small companies should adopt them now.

In the first part of this article, we've discussed four new data analytics capabilities powered by ML and NLP technologies that large and small companies can use to enhance their ability to generate data insights. In the second part, we'll introduce a couple of powerful yet affordable tools offering these capabilities and an agile approach that any company can adopt to quickly start leveraging modern BI and ML technologies to enhance decision-making and business performance. The time to start is now and Augeo Partners can help.