Category: Data Analytics

With multiple big data frameworks available on the market, choosing the right one is a challenge. A classic approach of comparing the pros and cons of each platform is unlikely to help, as businesses should consider each framework from the perspective of their particular needs. Facing multiple Hadoop MapReduce vs. Apache Spark requests, our big data consulting practitioners compare two leading frameworks to answer a burning question: which option to choose – Hadoop MapReduce or Spark. 

A quick glance at the market situation

Both Hadoop and Spark are open source projects by Apache Software Foundation and both are the flagship products in big data analytics. Hadoop has been leading the big data market for more than 5 years. According to our recent market research, Hadoop’s installed base amounts to 50,000+ customers, while Spark boasts 10,000+ installations only. However, Spark’s popularity skyrocketed in 2013 to overcome Hadoop in only a year. A new installation growth rate (2016/2017) shows that the trend is still ongoing. Spark is outperforming Hadoop with 47% vs. 14% correspondingly.

To make the comparison fair, we will contrast Spark with Hadoop MapReduce, as both are responsible for data processing.

The key difference between Hadoop MapReduce and Spark

In fact, the key difference between Hadoop MapReduce and Spark lies in the approach to processing: Spark can do it in-memory, while Hadoop MapReduce has to read from and write to a disk. As a result, the speed of processing differs significantly – Spark may be up to 100 times faster. However, the volume of data processed also differs: Hadoop MapReduce is able to work with far larger data sets than Spark.

Now, let’s take a closer look at the tasks each framework is good for.

Tasks Hadoop MapReduce is good for:

• Linear processing of huge data sets. Hadoop MapReduce allows parallel processing of huge amounts of data. It breaks a large chunk into smaller ones to be processed separately on different data nodes and automatically gathers the results across the multiple nodes to return a single result. In case the resulting dataset is larger than available RAM, Hadoop MapReduce may outperform Spark.
• Economical solution, if no immediate results are expected. Our Hadoop team considers MapReduce a good solution if the speed of processing is not critical. For instance, if data processing can be done during night hours, it makes sense to consider using Hadoop MapReduce.

 

Tasks Spark is good for:

• Fast data processing. In-memory processing makes Spark faster than Hadoop MapReduce – up to 100 times for data in RAM and up to 10 times for data in storage.
• Iterative processing. If the task is to process data again and again – Spark defeats Hadoop MapReduce. Spark’s Resilient Distributed Datasets (RDDs) enable multiple map operations in memory, while Hadoop MapReduce has to write interim results to a disk.
• Near real-time processing. If a business needs immediate insights, then they should opt for Spark and its in-memory processing.
• Graph processing. Spark’s computational model is good for iterative computations that are typical in graph processing. And Apache Spark has GraphX – an API for graph computation.
• Machine learning. Spark has MLlib – a built-in machine learning library, while Hadoop needs a third-party to provide it. MLlib has out-of-the-box algorithms that also run in memory. But if required, our Spark specialists will tune and adjust them to tailor to your needs.
• Joining datasets. Due to its speed, Spark can create all combinations faster, though Hadoop may be better if joining of very large data sets that requires a lot of shuffling and sorting is needed.

 

Examples of practical applications

We analyzed several examples of practical applications and made a conclusion that Spark is likely to outperform MapReduce in all applications below, thanks to fast or even near real-time processing. Let’s look at the examples.

• Customer segmentation. Analyzing customer behavior and identifying segments of customers that demonstrate similar behavior patterns will help businesses to understand customer preferences and create a unique customer experience.
• Risk management. Forecasting different possible scenarios can help managers to make right decisions by choosing non-risky options.
• Real-time fraud detection. After the system is trained on historical data with the help of machine-learning algorithms, it can use these findings to identify or predict an anomaly in real time that may signal of a possible fraud.
• Industrial big data analysis. It’s also about detecting and predicting anomalies, but in this case, these anomalies are related to machinery breakdowns. A properly configured system collects the data from sensors to detect pre-failure conditions.

Which framework to choose?

It’s your particular business needs that should determine the choice of a framework. Linear processing of huge datasets is the advantage of Hadoop MapReduce, while Spark delivers fast performance, iterative processing, real-time analytics, graph processing, machine learning and more. In many cases Spark may outperform Hadoop MapReduce. The great news is the Spark is fully compatible with the Hadoop eco-system and works smoothly with Hadoop Distributed File System, Apache Hive, etc.

Apache Cassandra and Apache Hadoop are members of the same Apache Software Foundation family. We could have contrasted these two frameworks, but that comparison would not be fair because Apache Hadoop is the ecosystem that encompasses several components. As Cassandra is responsible for big data storage, we have chosen its equivalent from the Hadoop’s ecosystem, which is Hadoop Distributed File System (HDFS). Here, we’ll try to find out if Cassandra and HDFS are like twins who are identical in appearance and just bear different names, or they are rather a brother and a sister who may look similar, but still are very different.

Master/slave vs. masterless architecture

Before we dwell on the features that distinguish HDFS and Cassandra, we should understand the peculiarities of their architectures, as they are the reason for many differences in functionality. If you look at the picture below, you’ll see two contrasting concepts. HDFS’s architecture is hierarchical. It contains a master node, as well as numerous slave nodes. On the contrary, Cassandra’s architecture consists of multiple peer-to-peer nodes and resembles a ring.

5 Key functional differences

1. Dealing with massive data sets

Both HDFS and Cassandra are designed to store and process massive data sets. However, you would need to make a choice between these two, depending on the data sets you have to deal with. HDFS is a perfect choice for writing large files to it. HDFS is designed to take one big file, split it into multiple smaller files and distribute them across the nodes. In fact, if you need to read some files from HDFS, the operation is reverse: HDFS has to collect multiple files from different nodes and deliver some result that corresponds to your query. By contrast, Cassandra is the perfect choice for writing and reading multiple small records. Its masterless architecture enables fast writes and reads from any node. This makes IT solution architects opt for Cassandra if it is required to work with time series data, which is usually the basis for the Internet of Things.

While in theory HDFS and Cassandra look mutually exclusive, in real life they may coexist. If we continue with the IoT big data, we can come up with a scenario where HDFS is used for a data lake. In this case, new readings will be added to Hadoop files (say, there will be a separate file per each sensor). At the same time, a data warehouse may be built on Cassandra.

2. Resisting to failures

Both HDFS and Cassandra are considered reliable and failure resistant. To ensure this, both apply replication. Simply put, when you need to store a data set, HDFS and Cassandra distribute it to some node and create the copies of the data set to store on several other nodes. So, the principle of failure resistance is simple: if some node fails, the data sets that it contained are not irretrievably lost – their copies are still available on other nodes. For example, by default, HDFS will create three copies, though you are free to set any other number of replicas. Just don’t forget that more copies mean more storage space and longer time to perform the operation. Cassandra also allows choosing the required replication parameters.

However, with its masterless architecture, Cassandra is more reliable. If HDFS’s master node and secondary node fail, all the data sets will be lost without the possibility of recovery. Of course, the case is not frequent, but still this can happen.

3. Ensuring data consistency

Data consistency level determines how many nodes should confirm that they have stored a replica so that the whole write operation is considered a success. In case of read operations, data consistency level determines how many nodes should respond before the data is returned to a user.

In terms of data consistency, HDFS and Cassandra behave quite differently. Let’s say, you ask HDFS to write a file and create two replicas. In this case, the system will refer to Node 5 first, then Node 5 will ask Node 12 to store a replica and finally Node 12 will ask Node 20 to do the same. Only after that, the write operation is acknowledged.

Cassandra does not use HDFS’s sequential approach, so there is no queue. Besides, Cassandra allows you to declare the number of nodes you want to confirm the success of operation (it can range from any node to all nodes responding). One more advantage of Cassandra is that it allows varying data consistency levels for each write and read operation. By the way, if a read operation reveals inconsistency among replicas, Cassandra initiates a read repair to update the inconsistent data.

4. Indexing

As both systems work with enormous data volumes, scanning only a certain part of big data instead of a full scan would increase the system’s speed. Indexing is exactly the feature that allows doing that.

Both Cassandra and HDFS support indexing, but in different ways. While Cassandra has many special techniques to retrieve data faster and even allows creating multiple indexes, HDFS’s capabilities go only to a certain level – to the files the initial data set was split into. However, record-level indexing can be achieved with Apache Hive.

5. Delivering analytics

Both designed for big data storage, Cassandra and HDFS still have to do with analytics. Not by themselves, but in combination with specialized big data processing frameworks such as Hadoop MapReduce or Apache Spark.

Apache Hadoop’s ecosystem already includes MapReduce and Apache Hive (a query engine) along with HDFS. As we described above, Apache Hive helps overcome the lack of record-level indexing, which enables to speed up an intensive analysis where the access to records is required. However, if you need Apache Spark’s functionality, you can opt for this framework, as it is also compatible with HDFS.

Cassandra also runs smoothly together with either Hadoop MapReduce or Apache Spark that can run on top of this data storage.

HDFS and Cassandra in the framework of CAP theorem

According to the CAP theorem, a distributed data store can only support two of the following three features:  

• Consistency: a guarantee that the data is always up-to-date and synchronized, which means that at any given moment any user will get the same response to their read query, no matter which node returns it.
• Availability: a guarantee that a user will always get a response from the system within a reasonable time.
• Partition tolerance: a guarantee that the system will continue operation even if some of its components are down.

In a nutshell

If you are to make a choice between Apache Cassandra and HDFS, the first thing to take into account is the nature of your raw data. If you have to store and process large data sets, you can consider HDSF, if multiple small records – Cassandra may be a better option. Besides, you should form your requirements towards data consistency, availability and partition tolerance. To make a final decision, it’s critical to understand the exact use of big data storage.

Even if Cassandra seems to outperform HDFS in most cases described, this does not mean that HDFS is weak. Based on your business needs, a professional Hadoop consulting team may suggest a combination of frameworks and technologies with HDFS and Hive or HBase at core that would enable great and seamless performance.

Editor’s note: If, despite all your efforts, your decision-making is still gut-feeling-based rather than informed, check whether you use the right mix of data analytics types. Read on and turn to our data analytics consultants for tailored recommendations.

Back in the 17th century, John Dryden wrote, “He who would search for pearls must dive below.” Although the author did not have advanced data analytics in mind, the quote perfectly describes its essence. Together with ScienceSoft, let’s find out how deep one should go into data in search of much-needed and fact-based insights.

Types of data analytics

There are 4 different types of analytics. Here, we start with the simplest one and go further to the more sophisticated types. As it happens, the more complex an analysis is, the more value it brings.

Descriptive analytics

Descriptive analytics answers the question of what happened. Let us bring an example from ScienceSoft’s practice: having analyzed monthly revenue and income per product group, and the total quantity of metal parts produced per month, a manufacturer was able to answer a series of ‘what happened’ questions and decide on focus product categories.

Descriptive analytics juggles raw data from multiple data sources to give valuable insights into the past. However, these findings simply signal that something is wrong or right, without explaining why. For this reason, our data consultants don’t recommend highly data-driven companies to settle for descriptive analytics only, they’d rather combine it with other types of data analytics.

Diagnostic analytics

At this stage, historical data can be measured against other data to answer the question of why something happened. For example, you can check ScienceSoft’s BI demo to see how a retailer can drill the sales and gross profit down to categories to find out why they missed their net profit target. Another flashback to our data analytics projects: in the healthcare industry, customer segmentation coupled with several filters applied (like diagnoses and prescribed medications) allowed identifying the influence of medications.

Diagnostic analytics gives in-depth insights into a particular problem. At the same time, a company should have detailed information at their disposal, otherwise, data collection may turn out to be individual for every issue and time-consuming.

Looking for Tailored Recommendations on Data Analytics?

Get a clear picture of your data analytics needs after a free 30-minute consultation with ScienceSoft’s experts.

Predictive analytics

Predictive analytics tells what is likely to happen. It uses the findings of descriptive and diagnostic analytics to detect clusters and exceptions, and to predict future trends, which makes it a valuable tool for forecasting. Check ScienceSoft’s case study to get details on how advanced data analytics allowed a leading FMCG company to predict what they could expect after changing brand positioning.

Predictive analytics belongs to advanced analytics types and brings many advantages like sophisticated analysis based on machine or deep learning and proactive approach that predictions enable. However, our data consultants state it clearly: forecasting is just an estimate, the accuracy of which highly depends on data quality and stability of the situation, so it requires careful treatment and continuous optimization.

Prescriptive analytics

The purpose of prescriptive analytics is to literally prescribe what action to take to eliminate a future problem or take full advantage of a promising trend. An example of prescriptive analytics from our project portfolio: a multinational company was able to identify opportunities for repeat purchases based on customer analytics and sales history.

Prescriptive analytics uses advanced tools and technologies, like machine learning, business rules and algorithms, which makes it sophisticated to implement and manage. Besides, this state-of-the-art type of data analytics requires not only historical internal data but also external information due to the nature of algorithms it’s based on. That is why, before deciding to adopt prescriptive analytics, ScienceSoft strongly recommends weighing the required efforts against an expected added value.

What types of data analytics do companies choose?

To identify if there is a prevailing type of data analytics, let’s turn to different surveys on the topic for the period 2016-2019.

For the 2016 Global Data and Analytics Survey: Big Decisions, more than 2,000 executives were asked to choose a category that described their company’s decision-making process best. Further, C-suite was questioned with what type of analytics they relied on most. The results were the following: descriptive analytics dominated (58%) in the “Rarely data-driven decision-making” category; diagnostic analytics topped the list (34%) in the “Somewhat data-driven” category; predictive analytics (36%) led in the “Highly data-driven” category.

The survey findings are in line with ScienceSoft’s hands-on experience as they show the need for one or the other type of analytics at different stages of a company’s development. For example, the companies that strived for informed decision-making found descriptive analytics insufficient and added up diagnostics analytics or even went as far as predictive one.

For another survey, BARC’s BI Trend Monitor 2017, 2,800 executives shared their opinion on the growing importance of advanced analytics. The term advanced analytics was the umbrella term for predictive and prescriptive analytics types.

According to the 2018 Advanced and Predictive Analytics Market Research, advanced analytics was for the first time considered “critical” or “very important” by a majority of respondents.

Within the BARC’s BI Trend Monitor 2019 survey, C-suite still named advanced analytics among the most important business intelligence trends.

What types of data analytics does your business need?

To define the right mix of data analytics types for your organization, we recommend answering the following questions:

• What’s the current state of data analytics in my company?
• How deep do I need to dive into the data? Are the answers to my problems obvious?
• How far are my current data insights from the insights I need?

The answers to these questions will help you settle on a data analytics strategy. Ideally, the strategy should allow incrementally implementing the analytics types, from the simplest to more advanced. The next step would be to design the data analytics solution with the optimal technology stack, and a detailed roadmap to implement and launch it successfully.

You may try to complete all these tasks with the efforts of an in-house team. In this case, you’ll need to find and train highly qualified data analytics specialists, which will most probably turn lengthy and pricey. To maximize the ROI from implementing data analytics in your organization, we advise you to turn to an experienced data analytics provider with a background in your industry. A mature vendor will share the best practices and take care of everything, from the analysis of your current data analytics state and selection of the right mix of data analytics to bringing the technical solution to life. If the described approach resonates with you, our data analytics services are at your disposal.

Editor’s note: In the article, Alex Bekker, Head of Data Analytics Department at ScienceSoft, explains how big data analytics can help a company drive revenue and reduce operational costs. Read on to learn how to start your big data journey and be welcome to explore ScienceSoft’s offer in big data services to learn what approach we follow to help our clients embrace big data potential.

As Head of Data Analytics, I enjoy studying the experiences of renowned companies who drive great value from big data initiatives, so that my team can offer our customers similar and even better results. Here, I’ve selected impressive big data use cases from the manufacturing industry, including, from ScienceSoft’s practice, that I hope will inspire you to embark on a big data journey.

#1. Production optimization

Extracting process improvement

A vertically integrated precious-metal manufacturer’s ore grade declined. The only logical way to avoid loss was to improve metal extracting and refining processes. Using sensor data, the manufacturer’s big data solution identified what factors influenced output the most. And the dominant parameter turned out to be oxygen level. With this insight, the team slightly changed the leaching process and increased the yield by 3.7%. Thanks to big data analysis, the manufacturer now earns $10-20 million additionally every year. Quite a gain, considering the ore grade deterioration rate was 20%.

Chemical yield perfection

A leading European chemicals manufacturer sought to improve yield. Using sensors, their big data solution analyzed how each input factor influenced production output. It analyzed temperatures, quantities, carbon dioxide flow and coolant pressures and compared their influence rates to one another. As a result, they revealed that carbon dioxide flow rates hugely affect the yield. And by slightly changing the parameters, they achieved a significant decrease in raw materials waste (by 20%) and energy costs (by 15%), and impressively improved the yield.

Vaccine yield improvement

A huge pharmaceutical company needed to find a way to improve the yield of their vaccines. To do that, the company’s big data solution analyzed their equipment sensor data, revealed interdependencies between various production parameters and compared how each of them affected the yield. Then, 9 most crucial parameters were identified, reviewed and adjusted to optimize the manufacturing process. It improved vaccines’ yield by 50%. Now, the company additionally makes $5-10 million a year per one substance.

Sugar-sweet optimization

High humidity levels and low-quality raw materials badly affected the taste of sugar of a large sugar manufacturer. To fight it, data science came in use to analyze sensor data and find correlations between the parameters contributing to the best sugar quality. Fortunately, with this insight the manufacturer managed to find a way to quickly influence product quality and achieve a unified sugar standard regardless of external factors. It allowed them to reduce production costs, increase customer satisfaction and simplify workloads.

#2. Quality assurance

Early-stage vehicle quality assurance

As early as 2014, BMW used big data to detect vulnerabilities in their new car prototypes. Data was collected from sensors on the tested prototypes and cars already in use. Due to big data analysis, BMW’s solution (probably integrated with their vehicle design and modelling software) spotted weaknesses and error patterns in the prototypes and in cars already in use. It enabled engineers to remove uncovered vulnerabilities before the prototypes actually went into production and helped reduce recalls of cars already in use. As a result, BMW can not only ensure higher quality at early stages, but also reduce warranty costs, boost brand reputation and probably save lives.

Jet engine design

Rolls-Royce uses big data extensively. And one of their most interesting manufacturing big data experiences is connected with modelling new aircraft engines. At the design stage, their software (integrated with a big data tool) creates simulations of new jet engines and analyzes terabytes of big data to see whether the new models are any good. This allows the company to find weaknesses before the model gets to production, which reduces defect-related costs and helps design the product of a much higher quality. Very smart, don’t you think?

#3. Enterprise management

Data-driven enterprise growth

Using big data analytics in manufacturing, companies can tackle global development challenges, such as transferring production to other countries or opening new factories in new locations. Companies’ historical and external data analysis can establish whether it’s still profitable to run factories in current locations or at current scopes by building predictive models and what-if scenarios. Besides, in the right hands, big data can help explore oceans of unseen opportunity, such as offering new products or even conquering new markets.

Accessible raw materials

To avoid costs connected with supply chain failures, an enterprise needed a better way to manage raw materials delivery. They decided to use their suppliers’ route details as well as weather and traffic data provided by trustworthy external sources to identify the probability of delivery delays. To do that, their big data tool (quite possibly integrated with their MRP) used predictive analytics and calculated possible delays and raw materials shortages. Based on these calculations, the enterprise worked out a supply-related emergency plan and is now able to run their production uninterrupted and avoid excessive downtime costs.

Predictive maintenance

Intel’s factory equipment live-streams IoT-generated data into their big data solution (probably integrated with MES). The analytics solution uses this data for pattern recognition, fault detection and visualization. It allows engineers to see what tendencies require their immediate attention and what actions are needed to prevent serious breakdowns on the shop floor. Such predictive maintenance reduces reaction time from 4 hours to 30 seconds and cuts costs. In 2017, thanks to big data and IoT, Intel predicted saving $100 million. This doesn’t look surprising at all: according to the research, predictive maintenance has appeared on companies’ radars exactly in 2017 and has got straight to top 3 big data use cases.

#4. After sales

Connected car vehicles

One of ScienceSoft’s customers from the connected car industry uses big data to provide after-sales support to their clients and ensure continuous improvement. The customer’s operational centers analyze in real time tons of data fed from car sensors (diagnostics data, mileage, geolocation, etc.) and generate insights into the product’s performance. The analysis of this data allows the company to monitor the product’s state, note and even predict some malfunctions and offer maintenance service early enough to avoid serious breakdowns. Such approach allows the customer to increase the product quality and enhance customer experience.

Hull cleaning

As a standard after-sales procedure, Caterpillar Marine was requested by one if their clients to do an analysis of how hull cleaning impacts fleet performance. Caterpillar’s big data solution (integrated with their Asset Intelligence platform) analyzed data from sensors on ships running with and without cleaned hulls. Then, it found correlations between the client’s hull-cleaning investments and fleet performance. Soon, Caterpillar concluded that their client needed to clean hulls more often (every 6.2 months, not 2 years) and that related investments paid off. As to the manufacturer, big data allowed them to ensure the most efficient exploitation of their products and improve the company’s image.

Wind farm optimization

As a proponent of after-sales with a personalized approach to customers in manufacturing, General Electric helps power producers use big data at 4 levels.

Level 1. Wind turbine’s sensor data analytics enables power producers to optimize turbine’s blade pitch and energy conversion automatically.

Level 2. Wind farm monitoring software compares sensor data to predicted values and recognizes performance patterns, which helps power producers perform preventive maintenance at the farms.

Level 3. Power producers use historical and real-time data to build predictive models, find correlations, detect faults and recognize patterns to optimize the farm’s work.

Level 4. The data is visualized and presented to top management for global-scale informed decision making.

Are you inspired to start leveraging big data potential?

ScienceSoft’s team of big data experts is ready to design, implement or support your big data project to ensure considerable ROI on your big data investments.

A guide on how to start

If the examples of successful big data initiatives triggered your interest, I’ll gladly share a roadmap my colleagues at ScienceSoft and I devised for our customers to set off on a big data journey safely and effectively.

Ready… Set…

To prepare for a big data adoption project, the first thing crucial for success is finding the right approach. Rather than getting obsessed with the idea of big data, dashing to get the budget and then failing to extract value from it, first, you should lay the groundwork for the possible future ‘novelty.’ Let me show you the steps that will help you achieve business-IT alignment:

Step 1. I always advise big data project sponsors to start with reading about the possibilities of big data, then look at the business strategy and define what goals in it can be achieved with big data’s help.

Step 2. You should get more details on your company’s manufacturing problems and needs. The best way to do it is talking to the engineering management at your enterprise and asking them how the quality improvement process is going. Chances are, the process is problematic and no solution has yet been found, which is where you explain that such challenges can be solved with a thing called big data analytics.

Step 3. Try to get the consent of the engineering management to prove (if needed) to the company’s top management that they do need big data. And also warn them that their involvement will be necessary later to help data analysts understand the needed details of the manufacturing process.

Step 4. Determine a certain range of how much a particular big data project costs and talk to your top management about big data adoption and big data benefits. In our other article, my colleague, Olga Baturina, provided some telling statistics of big data gains.

If you need more details on how to ensure business IT-alignment, you can have a look at the guide written by my colleague, Boris Shiklo, CTO of ScienceSoft.

Go!

I always warn big data project sponsors against applying big data capabilities to complex tasks right from the start. Just like you can’t go to space a few days after deciding to become an astronaut. So, my advice to manufacturing companies is to start out with a simple project (for example, trying to achieve a stable output quality at a vaccine factory). A simple starting project allows you to see how big data can solve your problems with low risks and investments. Which, in its turn, is likely to positively affect your top management’s opinion on big data and encourage them to plan further big data investments (for more serious analytical projects).

At ScienceSoft, we usually break a big data project down into ‘digestible’ phases that are to be approached separately. Here are the sample phases of a big data project for manufacturing:

• Aggregating data.
• Using simple analytical algorithms.
• Turning to more sophisticated analytical methods.
• Incrementally automating your production management.

Aggregating data

Before any analysis can happen, you have to start aggregating data. In some cases, it’s not a problem at all: you just deploy/add sensors on your manufacturing equipment, prepare data storing facilities and enjoy the flow of ‘freshly-cut’ data.

In other cases, such as if your production cycle is months- or even years-long, it can prove difficult because you may lack the info on how your production process parameters influence output. And without knowing it, it’s all really a shot in the dark. But don’t get upset: there are ways to fight it. For example, in ScienceSoft’s projects, we recommend our customers to focus on one part of their manufacturing process, rather than on the entire process. That way they can improve the overall process by analyzing and adjusting its constituent parts.

Making analytical baby steps and advancing to big data strides

At ScienceSoft, we usually define the next stages of revealing big data insights:

1. At first, you can perform relatively simple big data analysis to make targeted changes in your manufacturing processes (to improve product quality, for instance).
2. Then, you can dig your data deeper to find ways to change your business processes. For example, you used to perform reactive maintenance and, with big data, you start preventive maintenance.
3. When the time comes, you can even transform your business model, finding a better way to do it through big data analysis (say, you decide to get closer to the customer by making the cars you produce a smart connected product; you deploy sensors on them, analyze data from cars in use and provide after sales services).

The situation, I most commonly encounter, is that at early stages, customers only need the most usual analytical methods, such as correlations and regression analysis. As their big data competences and needs grow, analytical methods become more elaborate, and they start employing predictive analytics and machine learning in search of new business opportunities.

Production management automation

Automation of your production management is probably the most sophisticated way of using big data in manufacturing processes. The concept of automated production management is fairly simple: your historical and incoming sensor data is analyzed in real time and the control apps send targeted commands to actuators on your equipment.

A good example of production management automation is the case with General Electric’s wind turbines. Sensors provide data on energy generation and wind direction, according to which the blade pitch is changed to optimize the wind turbine’s efficiency.

An example to make it clear

Let me share an example of a generalized customer from my practice – a company who produces baby food and decides to go big data. They start with data aggregation (deploy/add data sensors on the production floor and prepare data storage).

For the sake of the example, let’s imagine that systematically, a few times a month, the baby food batches substantially drop in quality. Now, the big data team (together with the engineering team, R&D, product control managers) can find out what causes these quality drops. And together we realize that the manufacturing process doesn’t allow for the variations in the quality of raw material (baby food ingredients). If the ingredients’ quality is lower, the machinery isn’t ‘tuned’ to get a better quality output (say, you don’t adjust temperature and cooking times). And besides that, we also find a way to cut the production cycle duration. This big data application (better quality assurance) can be a good first project.

Getting valuable insights quickly and cheaply makes the company more interested in further big data capabilities and more complex analytical algorithms. And in a while, the enterprise starts running predictive analytics, equipment wear-out analysis and machine learning. Among other things, it allows them to perform predictive maintenance, which enables the staff to react to alarming trends on the manufacturing floor before any real damage is caused.

And as the company expands globally, we help the company to use big data powers to assure and control baby food quality across all the franchisees.

Now, survive

Yes, while starting big-data-adoption action, there are always impediments. Big data project sponsors I talk to commonly voice the following concerns:

1. Lacking in-house technical skills.

I believe, not every business needs complete outsourcing. For example, at early stages, when you’ll need to experiment a lot, it’s simply easier, if your ‘domestic’ people are involved, thus it’s natural to hire new skilled tech employees or retrain old ones.

2. Lacking the understanding of big data potential

In case of outsourcing a big data project, your vendor will need to work closely with your team (the engineering team, R&D, product control managers, etc.) to ensure the deep understanding of manufacturing processes. Together with a vendor who has a solid approach to cooperation, you’ll be able to see elaborate ways to improve production and its management with big data potential.

3. Resisting the new technologies.

Some employees – let’s hope the lesser part – will probably resist big data. And there’s nothing personal about it: for creatures of habit, it’s just more convenient to use the old technologies. Training your staff as well as controlling their usage of the new solution can help deal with this challenge.

You are now ready

The manufacturing use cases show that big data can bring big money and big value. They also show that big data is most widely used for production optimization. And it’s quite logical: big data solutions are really good at finding correlations. While production changes based on sensibly selected correlations can improve yield enormously.

To reap the benefits that big data offers and start using big data in your manufacturing organization, you need to carefully plan your actions. So, let’s rehearse them. You should:

– Find the right approach to your big data. Carefully analyze your business needs, find a way to fulfill them with big data.

– Prudently plan your big data adoption. Don’t jump to the most difficult part right off the start. Find a small-scale project to test big data on. Aggregate data, test simple algorithms and then try more daring ones.

– Watch for management challenges. Gain a thorough big data understanding, don’t rush into outsourcing the project completely and engage a needed number of engineering technologists.

But before starting some real action, I advise you to turn to big data consulting, since it can ease the hardships of big data projects and contribute to big data understanding. If you want to know more about our big data consulting services, reach out to me.

In the business world, initiatives are usually locked unless they promise a decent return on investment. According to this scenario, BI initiatives have a high chance of being implemented. Judge for yourselves: ‘BI initiatives drive an average return on investment of 11% and are seen to generate revenue growth and cost savings from every business function’.^[1]

With a ‘bring value’ principle in their DNA, ScienceSoft’s business intelligence implementation team puts best efforts for your BI project to generate an above-the-average ROI. In this article, we’ve compiled business intelligence examples so that you could get a real feel of BI capabilities in 3 industries. For each industry, you’ll find:

• Typical insights that a BI solution helps to get.
• Examples of dashboards implemented on Microsoft Power BI, a self-service analysis and visualization tool.
• Real-life examples of BI solutions that ScienceSoft developed for the retail, healthcare, and real estate development industries.

Business intelligence for retail: Focus on customers and performance

Retail-specific insights

With the help of business intelligence software, retail companies can:

• Understand customer segments, their preferences and behavior across the channels (i.e., store, web, mobile, social media).
• Analyze sales from different angles, i.e., like-for-like sales, sales by geography, by channel, by store, by category.
• Measure the performance of retail stores, product categories, product lines, brands, SKUs, private label products, and identify best practices.
• Create data-driven forecasts (i.e., for sales, for new product lines).
• Analyze the performance of an online store: traffic, conversion rates, cart abandonment, wish list, promotion efficiency.
• Map the number of employees to the customer traffic flow, and the inventory level to sales.

An example of a retail dashboard

Being that concise, the dashboard can show the retailer’s sales from different perspectives as it allows applying multiple filters. For example with a few clicks, a business user can choose a specific country, a product category and a sales channel, and see the graphs and values altered in line with their selection. To see how it works in practice, watch our BI demo.

A real-life business case for retail

For one of our customers, ScienceSoft developed a data analytics solution that enabled robust retail analytics. Aggregating data from CRM, Magento (ecommerce), and Google Analytics, the solution provided an undistorted picture of traffic and conversion rates, as well as allowed measuring the engagement of online store visitors.

Thanks to real-time analytics capabilities embedded in the solution, the customer was able to track the actual stock level both at the warehouse and in the stores, which was a huge step forward compared to the shared document previously used for that purpose.

Besides, the solution allowed for goal management and KPI tracking, which helped the customer to assess their employees’ quality of work (i.e., how much they cross-sell and up-sell).

Read also:

Business intelligence for healthcare: Focus on patients, control of financials and improvement

Healthcare-specific insights

With BI solutions, healthcare providers can aggregate electronic health records, patient-generated health data, and internal data on costs, staffing, and utilization (i.e., of beds, equipment, operating rooms) to:

• Monitor patients’ health status, evaluate the treatment progress and reduce the rate of readmissions.
• Analyze and improve patient outcomes (i.e., identify trends in disease development, find the dependencies between health risk factors and diagnoses, medications and outcomes).
• Measure and control the financials, such as revenue cycle management, costs, return on investments and cash flow.
• Improve internal processes based on analyzing medication use, equipment and bed utilization efficiency.

An example of a healthcare dashboard

A real-life business case for healthcare

ScienceSoft implemented a BI solution for a health information exchange vendor. The solution aggregated data retrieved from the systems of multiple healthcare providers and allowed its comprehensive analysis. Our BI team also designed and implemented a set of dashboards to let the customer:

• Check the patients’ profiles.
• Analyze health problems the patients encountered, classify these problems (i.e., by type, time range, patients’ age), and identify the most frequent ones.
• Track the medications used.
• Find out dependencies, i.e., those existing between diagnoses, medications and health problems.

Business intelligence for real estate development: Focus on property and tenant management

Real estate-specific insights

With BI applications in place, real estate developers can:

• Efficiently manage the property by analyzing property metrics, use, tenure, occupancy status.
• Assess the location and availability of the property.
• Improve space utilization by understanding occupancy (i.e., by geography, property type) and payable rent.
• Monitor the financial health of the business (by tracking such KPIs as a net income, cash flow, profitability per square foot, market share growth).
• Understand the tenants’ profiles and manage them.
• Manage risks.

An example of a dashboard for a real estate developer

A real-life business case for real estate development

For a commercial and residential real estate developer, ScienceSoft implemented a BI solution that enabled comprehensive financial reporting based on the data aggregated from 40 diverse data sources. The reporting embraced main financial indicators like income, operating expenses, cash at the end of the period, assets and liabilities, and more, for both the entire company and its separate branches. 17 financial reports of different levels of detail provided the developer with a multifaceted view of their finances.

Though different, these BI examples have something in common

Despite a BI solution is always tailored to a certain business and its needs, we can still name two features shared by the three examples we considered:

• Data aggregation is power: making use of data retrieved from multiple sources is a foundation for comprehensive and insightful reporting.
• It’s convenient when a reporting tool allows for self-service business analytics with drill-down and filtering opportunities, as well as intuitive dashboards – this speeds up the path to insights.

[1] Source: Dresner Advisory Services LLC, accessed on 10/22/2019.

The extensive list of Microsoft certifications includes the ones in business intelligence and data science domains. As data science and business intelligence consultants, we can easily explain why these certificates are so popular: these directions are actively developing and the relevant workforce is highly demanded on the market.

Here, we take a closer look at the current situation with Microsoft certifications that prove the mastery of SQL Server, Power BI, Excel, and Azure services that relate to AI and machine learning (we’ll call them business intelligence and data science certificates) and explain what skills they verify, how to get them and what benefits they can bring.

Benefits of having Microsoft certifications

For companies, having a Microsoft certified specialist on board means having skills that the company can make use of in real-life projects. According to the research, Microsoft certified developers are 90% more productive and nearly 60% more efficient compared to their non-certified peers.

For a vast number of BI and data science-related jobs like BI developers, data scientists and data analysts, Microsoft certifications serve as a quality mark. That’s the first-hand experience that ScienceSoft’s analytics team can share. Being Microsoft certified turns a competitive advantage and increases the holders’ chances for hiring and having a higher salary than their uncertified peers. Plus, this demonstrates their desire and ability to learn and develop.

Available Microsoft BI and data science certifications

At Inspire 2019, Microsoft announced significant changes in their certification program and the adoption of role-based certifications. For example, they are moving away from Microsoft Certified Solutions Associates and Microsoft Certified Solutions Experts (or MCSAs and MCSEs, which are still available) and introducing the ‘Microsoft Certified’ status that goes along with the job role and the level achieved within their certification framework (Fundamentals, Associate, or Expert). Besides, Microsoft will extend their list of certifications with 10+ roles in January – June 2020 with some of them possibly being from BI or data science domains.

As of September 2019, the list of available BI and data science certifications looks as follows:

How to get Microsoft BI and data science certifications?

To get a certification, a candidate should make a certain path suggested by Microsoft. Mainly, this path consists of two corresponding exams. However, there are cases when a candidate should satisfy some prerequisites first. For example, to get MCSE: Data Management and Analytics, a candidate should first attain one of the relevant MSCAs in the business intelligence domain.

To show various BI and data science paths that Microsoft offers, we’ve compiled two convenient schemes:

Microsoft business intelligence certification paths

Microsoft data science certification paths

For each exam, Microsoft shares clear guidelines. They describe what skills will be tested, indicate sample tasks and the share of this kind of tasks in the overall test. Besides, Microsofts provides rich learning materials and a candidate can choose the preferred preparation option, be it instructor-led or self-paced training, studying relevant books or taking official practice tests.

How much do certifications cost?

The certification cost depends on the path that a candidate is to follow. As the cost of one exam is $165, the final cost will depend on the number of exams in a path that a candidate should take. Let’s consider the longest (and therefore the most expensive path) – earning the MCSE Data Management and Analytics certification by a candidate following the MCSA SQL Server 2012/2014 path. As the candidate has to pass four exams, the total cost of getting the MCSE is $660 (in case, a candidate passes all the exams on their first attempt). By the way, a candidate is allowed to take a certain exam no more than 5 times within 12 months.

Still, we strongly recommend clarifying the exact pricing before registering to take an exam, as Microsoft grants discounts to some candidates, for example, Microsoft Partner Network members, as well as makes promotional offers.

Do Microsoft certifications expire?

Technically, Microsoft BI certifications don’t expire. However, each certification is always bound to a specific date when it’s earned. As the technologies evolve and their fresher versions replace their predecessors, it’s natural to expect that a certification earned in 2012 may fail to create a competitive advantage in 2019. Perfectly understanding this, Microsoft regularly updates their exams. Plus, they usually offer transitional exams to help renew the certification at a discount.

To go or not to go for Microsoft BI and data science certifications?

We, at ScienceSoft, see strong arguments for obtaining Microsoft BI and data science certifications:

• Microsoft is a world-known tech giant, and their certifications serve as a quality mark for BI and data science specialists worldwide.
• The list of Microsoft business intelligence tools is vast, and they are widely used for data warehouses, OLAP cubes and reporting components of analytics solutions, which makes the specialists that have mastered such tools highly demanded.
• Microsoft Azure is among the top three cloud providers in 2019, so the mastery of Azure technologies is a true competitive advantage for the specialists working in data science and big data domains.

Similar to a constellation where you can spot different stars, artificial intelligence (AI) can be brought down into different types. To help you decide what AI type will shine brightest and contribute to your business’ stellar performance, our data science consultants will define each. However, let’s first dispel the clouds to have a clear look at AI as a whole.

AI defined

Artificial intelligence enables a computer system to be trained and apply the gained knowledge to new inputs. This ability rests upon math and algorithms and is applicable only to the tasks that the system has been trained to perform.

To visualize the AI concept clearer, you may think of a chatbot whose task is to help restaurant visitors to book a table. By nature, this chatbot is a computer program that is trained on tons of booking-a-table questions and relevant answers. That’s how it learns of a typical conversation flow on the topic. After the training, the chatbot can dialogue with customers. However, if a customer deviates from the main topic of booking a table and asks for food recommendations, the chatbot in question is of little help as it is not trained to perform this task.

Types of artificial intelligence

To help you decide how exactly AI can be put into practical use, we offer our classification based on our 34 years of experience in data science. For each AI type, we’ll give a brief explanation, name the most prominent business use cases, as well as share real-life examples from our artificial intelligence consulting practice.

Analytic AI

Powered with machine learning (including its most advanced deep learning techniques), analytic AI scans tons of data for dependencies and patterns to ultimately produce recommendations or provide a business with insights, thus contributing to data-driven decision-making.

Sentiment analysis and supplier risk assessment are just a few examples of analytic AI in action. If you’d like to get a complete picture of how such a solution works, our experts have summarized the insights gained from their experience with two of the use cases – inventory optimization and demand forecasting.

Functional AI

Functional AI is very similar to analytic AI – it also scans huge amounts of data and searches for patterns and dependencies in it. However, instead of giving recommendations, functional AI takes actions. For instance, being the part of the IoT cloud, it can spot a machine-breakdown pattern in the sensor data received from a certain machine, and trigger a command to turn this machine off. Another example: robots that Amazon uses to bring the shelves with the goods to the pickers, thus speeding up the picking process.

Interactive AI

This type of AI allows businesses to automate communication without compromising on interactivity. To envisage this type of AI, think of chatbots and smart personal assistants whose abilities can vary from answering pre-built questions to understanding the conversation context.

Interactive AI can serve another purpose – improving a company’s internal processes. For example, one of our projects was dedicated to creating a chatbot to facilitate the corporate process of vacation booking.

Text AI

Businesses that use text AI can enjoy text recognition, speech-to-text conversion, machine translation, and content generation capabilities. Even if a company is not Google or Amazon, or any other giant company that provides text AI as a service, it can still take advantage of this AI type. For example, the company can use text AI to power an internal corporate knowledge base.

Contrary to a traditional knowledge base that rests upon a search by keywords, an AI-powered one can find the document containing the most relevant answer even if the document doesn’t have full keywords. This is possible thanks to semantic search and natural language processing, which allow AI to build semantic maps and recognize synonyms to understand the context of the user’s question.

Visual AI

With visual AI, businesses can identify, recognize, classify and sort objects or convert images and videos into insights. A computer system that helps an insurer to estimate damage based on damaged car photos or a machine that grades apples based on their color and size are the examples of visual AI. This type of AI covers computer vision or augmented reality fields.

To get the real feel of the value that visual AI can bring, you can read in more detail about a face recognition solution that we developed to help a retailer enhance and personalize their customer service; or about an application for automated inspections that allowed a manufacturer to immediately control the quality of the produced details.

In the finale

AI has become a trend that cannot be disregarded: hardly any report or survey doesn’t mention its growing importance. Gartner names AI a top priority for CIOs. Accenture Technology Vision 2019 Report cites that 89% of businesses either have already adopted or are experimenting with AI. And PWC 2019 AI Predictions Survey of 1,000 executives reveals that they expect the following value from AI investments: increased revenue and profits (48%), better customer experience (46%) and improved decision-making (40%). However, to take advantage of AI, companies have to overcome some obstacles. For example, a survey by McKinsey&Company names top 3 barriers: the lack of a clear strategy for AI (43%), the lack of the talent with appropriate skills (42%) and functional silos that constrain end-to-end AI solutions (30%).

And what about your business? We have shown the multifaceted nature of AI for you to find your way to get value from it. You can opt for any of 5 AI types – analytic, interactive, text, visual, and functional – or wisely combine several ones.

Editor’s note: In this article, Maryna explains how to gain vision into your business processes with cloud business intelligence (BI). Check out the top 3 cloud BI tools according to ScienceSoft, and if you need thorough expert engagement in your BI project, consider our business intelligence services.

According to Boardroom Stats from Google Cloud, more than 40% of survey respondents plan to increase their investment in cloud-based services and products while 30% are going to migrate legacy enterprise software to the cloud. So, if right now you’re trying to figure out what makes the cloud so popular and whether it is the right place for your BI solution, keep on reading. Below, I dwell on the idea of cloud business intelligence and share my thoughts about why cloud business intelligence solutions are worth considering.

Cloud business analytics and its deployment models

Cloud-based business intelligence is the process of transforming data into meaningful insights with the help of technology, which is partially or fully conducted in the cloud environment.

Cloud BI solutions allow companies to:

• Eliminate all hardware-related BI costs.
• Deploy a BI solution fast as well as scale it up and down in storage and compute resources time- and cost-effectively.
• Access data via any web browser or mobile device.
• Ensure BI solution’s high availability and fault-tolerance.
• Securely share actionable insights among colleagues.

A web-based BI solution or its part (e.g., data storage) can function within 3 cloud types:

Public cloud

A public cloud implies that the infrastructure costs are split among cloud tenants, which makes it the most affordable option for cloud BI. It’s a great option for small and mid-sized businesses with limited budget calls or companies dealing with big data workloads.

For example, in one of ScienceSoft’s projects, we hosted a big data analytics solution in Azure and AWS clouds to let a market research company accommodate and effectively process the continuously growing amount of data thanks to storage and compute cloud resources. At the same time, they were able to meet their budget requirements due to reasonable public cloud pricing.

Private cloud

If you worry about regulatory compliance or data security, I advise deploying a BI solution in a private cloud. Being the most expensive cloud option, the private cloud provides dedicated storage and compute resources exclusively at your company’s disposal.

Hybrid cloud

If you cannot afford locating the whole BI solution in the private cloud but have to comply with strict regulations (HIPAA, GLBA, GDPR, etc.), a hybrid cloud is your way to go. This computing environment combines the traits of a public cloud and a private cloud. By choosing this option, you can, say, store and analyze your sensitive data in the private cloud and experiment with big data in the public one.

Need Help with Defining Where to Host Your BI Solution?

ScienceSoft is ready to advise on the most suitable location of your BI solution so that you can gain greater visibility into your business processes and uplift your decision-making.

Cloud BI software capabilities to look for

When deciding on placing BI in the cloud, many companies face the dilemma of what software to choose to structure their BI solution with. Modern BI software offers a wide range of capabilities, and, as a BI consultant, I advise you to prioritize the following ones:

• Built-in data management capabilities – to allow extracting data from multiple sources, transforming it into the required format suitable for analysis as well as cleansing data to ensure its high quality.
• Advanced analytics capabilities – to facilitate data mining and predictive analytics for effective root-cause analysis and accurate forecasting.

• Rich cloud reporting and visualization capabilities – to enable every user to obtain the required analytics insights in the comprehendible format.
• Collaborative capabilities – to share analytics with colleagues in real time for quicker decision-making and data democratization.

In case you are a bit overwhelmed with the whole variety of cloud business intelligence software, you can always resort to professional consultancy for assistance or fix the software-related issues once and for all by opting for data analytics as a service.

Top cloud business intelligence tools

With the above capabilities taken into account and drawing on our experience with cloud-based BI solutions, ScienceSoft presents you with its list of top web-based business intelligence tools.

Power BI

Power BI is a recognized leader in analytics and business intelligence platforms. Affordable pricing, intuitive interface, and the ability to address analytical challenges of different complexity make Microsoft Power BI top our list.

Services: Power BI Pro, Power BI Premium, Power BI Mobile, Power BI Embedded, Power BI Report Server.

Pricing: Power BI Mobile – free, Power BI Pro – $9.99/user/month, Power BI Premium – starting from $4,995/dedicated resources/month.

Demo: Power BI

Qlik Sense

The tool is praised for its unique associative analytics engine in combination with drag-and-drop visualization and data storytelling functionality.

Pricing: Qlik Sense Business edition – $30/user/month, Qlik Sense Enterprise: Professional User – $70/user/month, Analyzer User – $40/ user/month.

Tableau

With its user-friendly interface, Tableau offers such capabilities as vast data integration, augmented data preparation, advanced data analysis, rich visualization, secure collaboration, and much more.

Services: Tableau Prep, Tableau Server, Tableau Online, Tableau Data Management, Tableau Server Management, Tableau Mobile, Embedded Analytics.

Pricing: Tableau Creator – $70/user/month, Tableau Explorer – $35- $42/user/month (depending on the deployment model), Tableau Viewer – $15/user/month.

It’s time to benefit from the synergy of BI and the cloud!

Cloud computing is already a business requirement rather than an option because of scalability, cost-effectiveness, and high service availability it offers. So, if you want to reap the above cloud benefits with your BI solution, let your first step be choosing a reliable vendor that will help your develop your BI implementation strategy and guide you towards your project’s success. In case you are looking for competent assistance, feel free to consider ScienceSoft for this task.

In a world where consultancies offer a hefty list of big data services, businesses still struggle to understand what value big data actually brings and what its most efficient use can be. Before committing to big data initiatives, companies tend to search for their competitors’ real-life examples and evaluate the success of their endeavors. So, our data consultants decided to save a mile on the investigation path for those interested in big data usage and conducted secondary research based on 11 dedicated studies and reports published between 2015 and 2019. We also spiced our research up with the voices of well-known companies that shared their experience in big data adoption.

How companies of different sizes use big data

In this section, we’ll refer to the following segments: small, mid-sized, large and very large organizations. This categorization is based on the number of employees in a business or an institution:

• Small organizations (1-100 employees).
• Mid-sized organizations (101-1,000 employees).
• Large organizations (1,001- 5,000 employees).
• Very large organizations (more than 5,000 employees).

 

How different industries use big data

Three industries most active in big data usage are telecommunications, healthcare, and financial services. ^[2]

Telecommunications

Healthcare

Financial services

Education

Insurance

What use cases prevail for each big data technology

Hadoop use cases

Spark use cases

What value big data brings

The history of big data usage in numbers

Big data stories of big companies

Check what Walmart, PepsiCo, JPMorgan Chase, Rolls-Royce, and Uber have to say about their big data experience.

“Over time, the need for more insights has resulted in over 100 petabytes of analytical data that needs to be cleaned, stored, and served with minimum latency through our Hadoop-based big data platform. Since 2014, we have worked to develop a big data solution that ensures data reliability, scalability, and ease-of-use, and are now focusing on increasing our platform’s speed and efficiency.”

Reza Shiftehfar, Hadoop Platform Team Leader at Uber

“Walmart relies on big data to get a real-time view of the workflow in the pharmacy, distribution centers and throughout our stores and e-commerce.”

Walmart Staff

“[About their big data platform Pep Worx] We were able to launch the product [Quaker Overnight Oats] using very targeted media, all the way through targeted in-store support, to engage those most valuable shoppers and bring the product to life at retail in a unique way. These priority customers drove 80% of the product’s sales growth in the first 12 weeks after launch.”

Jeff Swearingen, Senior Vice President of Marketing at PepsiCo

“Artificial intelligence, big data and machine learning are helping us reduce risk and fraud, upgrade service, improve underwriting and enhance marketing across the firm.”

Jamie Dimon, Chairman and Chief Executive Officer at JPMorgan Chase

“We have huge clusters of high-power computing which are used in the design process. We generate tens of terabytes of data on each simulation of one of our jet engines. We then have to use some pretty sophisticated computer techniques to look into that massive dataset and visualize whether that particular product we’ve designed is good or bad. Visualizing big data is just as important as the techniques we use for manipulating it.”

Paul Stein, Chief Scientific Officer at Rolls-Royce

A quick final look at big data

The findings of our secondary research are in line with our hands-on experience: businesses increasingly adopt big data, and, overall, they are highly satisfied with the results of their initiatives. Though the majority of big data use cases are about data storage and processing, they cover multiple business aspects, such as customer analytics, risk assessment and fraud detection. So, each business can find the relevant use case to satisfy their particular needs.

References

[1] 2017 Big Data Analytics Market Study by Dresner Advisory Services

[2] IDC/Dell EMC, Big Data: Turning Promise Into Reality

[3] Survey Report 2018: Big Data Analytics for Financial Services

[4] 2016 Predictive Modeling Benchmark Survey (U.S.) by Willis Towers Watson

[5] Business Application Research Center, Why Companies Use Big Data Analytics

[6] TDWI, Hadoop for the Enterprise

[7] Databricks, Apache Spark Survey 2016 Report

[8] Apache Spark Market Survey by Taneja Group

[9] McKinsey, Analytics comes of age

[10] 2017 Big Data Executive Survey by NewVantage Partners

[11] 2018 Big Data Executive Survey by NewVantage Partners

Introduction

Teradata is an integrated platform that provides functionality to store, access, and analyze organizational data on the Cloud as well as On-Premise infrastructure. Teradata Database provides an information repository system, along with support for various tools and utilities, making it a complete and active relational database management system.

Teradata is based on a parallel Data Warehouse with shared-nothing architecture.

• Data is stored in a row-based format.
• It supports a hybrid storage model in which frequently accessed data is stored in SSD whereas rarely accessed data is stored on HDD.
• The platform supports a Table partitioning feature and enforces Primary and Secondary Indexes.
• The Data Warehouse can scale up to 2048 nodes, thus offering data storage ability up to 94 petabytes.
• The Data Model is designed to be fault-tolerant and be scalable with redundant network connectivity to ensure reliability for critical use case

Teradata Storage Approach and Challenges

The sizes of the Teradata systems managing relational databases range up to 2,048 CPUs and 2,048 GB of memory supporting 128 TB or larger databases. The BYNET interconnect supports up to 512 nodes.

However, the major challenges with Teradata are:

• Huge data warehouse cost
• Not being an agile cloud data warehouse

Teradata is on the higher end of the pricing spectrum, and so capacity management is its biggest challenge. And when no solution is presented to optimize storage, customers decide to move away from Teradata to other alternatives. Managing storage capacity effectively is the only solution to overcome this challenge

Optimization of Teradata Storage

While working with Teradata, the following database sizing considerations must be taken into account:

• Capacity planning of system and data disks
• Data disk space allocation
• Determination of usable data space

Capacity Planning of System and Data Disks

• Warm and hot data typically constitute what is often called the operational data store. One of the Teradata data warehouse’s features is its hybrid storage, where frequently used hot data is stored on very high-performance solid state drives while less frequently used cold data is placed on traditional hard disk drives. The placement and migration of data based on data temperature is fully automatic and ongoing with the Teradata Virtual Storage feature. By paying attention to data temperature, Teradata can deliver higher query throughput and more consistent response times.
• Data compression includes both value-compression and algorithmic compression of data. When describing the compression of hash and join indexes, compression generally refers to row compression. While algorithmic compression can be either lossless or lossy, depending on the algorithm used. Compression has two principal applications: Reducing storage costs and Enhancing system performance.

Data Disk Space Allocation

• Disk or spool space is allocated based on system needs and tends to run out quickly unless proactively planned and reserved. Some guidelines for allocating disk space optimally include:
  • Create a special database to act as a spool space reservoir.
  • Allocate 2% of the total user space in the system for this database.
  • Assign roughly 0.25% of the total space to each user as an upper limit, ensuring that each receives at least as much space as the size of the largest table they access concurrently.
• Apart from the actual allocation of the disk space, there are other operational considerations that can optimize disk allocation:
  • Limit query size – the smaller the query, the less spool space required. If a particular user only performs small queries, then allocate less spool to that user. If a user performs many large queries, then allocate more spool to that user.
  • Optimize database size – the more AMPs in the configuration, the more thinly spread the data, so the more spool required per AMP.
  • Optimize the other factors such as average spool use per user, number of concurrent users, number of concurrent queries permitted for any one user.
• Plan for system and table space. Consider database sizing issues such as allocating permanent space and estimating database size requirements

Determination of Usable Data Space

This looks into system and hardware-oriented considerations undertaken for capacity planning.

Here are some recommended tasks in this regard:

• Use Teradata Viewpoint to monitor and manage the workload.
• Collect user resource usage detail data. Heavy resource consumers over time, skewed work, and application usage trends can be identified.
• Collect ResUsage data. The ResUsage tables report on the aggregate effect of all user requests on various system components over time, and can identify bottlenecks and capacity issues. See Resource Usage Macros and Tables for details on all ResUsage data and macros.
• Use the Locking Logger utility. Locking logger is essential for identifying locking conflicts.
• Use Database query logging. The Database Query Log records details on queries run, including arrival rates, response times, objects accessed, and SQL statements performed
• Use the Priority Scheduler utility. Priority Scheduler monitor output information shows comparative CPU usage across Priority Scheduler groups. This monitor output should be collected daily, minimally at a 5 or 10-minute interval, then summarized or charted.
• Additionally, use the following best practices:
  • Reserve 25% to 35% of total space for spool space and spool growth buffer. When you create user SysAdmin, you can leave the SPOOL parameter unspecified, so it defaults to the maximum allocation of the owning user, user DBC.
  • Allow an extra 5% of PERM space in user DBC.
  • Each time a new user or database is created, specify the maximum amount of spool space that a query submitted by that user can consume

Conclusion

Teradata is a complete and active relational database management system with a data repository as well as analytical system. However, Teradata is priced higher than its competitors and that urges customers to consider alternatives and move away from Teradata. One of the ways to combat this issue is to look at ways to optimize and provide better storage options. Managing storage capacity effectively would give Teradata the advantage it needs to retain and increase customer engagement.