Category: Data Analytics

Editor’s note: In the article, Irene reveals some tips on how a company can measure and improve the quality of their data. If you want to organize your data management process promptly and correctly, we at ScienceSoft are ready to share and implement our best practices. For more information, check our data management services.

One of the crucial rules of using data for business purposes is as simple as this: the quality of your decisions strongly depends on the quality of your data. However, simply knowing it isn’t extremely helpful. To get tangible results, you should measure the quality of your data and act on these measurements to improve it. Here, we throw some light on complicated data quality issues and share tips on how to excel in resolving them.

How to define data quality: attributes, measures and metrics

It would be right to start this section with a universally recognized definition of data quality. But here comes the first trouble: there is none. In this respect, we can rely on our 34-year experience in data analytics and take the liberty to offer our own definition: data quality is the state of data, which is tightly connected with its ability (or inability) to solve business tasks. This state can be either “good” or “bad”, depending on to what extent data corresponds to the following attributes:

• Consistency
• Accuracy
• Completeness
• Auditability
• Orderliness
• Uniqueness
• Timeliness.

To reveal what’s behind each attribute, our data management team put together this table and filled it with illustrative examples based on customer data. We also mentioned sample metrics that can be chosen to get quantifiable results while measuring these data quality attributes. 

An important remark: for big data, not all the characteristics are 100% achievable. So, if you are a big data company, you may be interested in checking the specifics of big data quality management.

Why low data quality is a problem

Do you think that the whole problem of poor data quality is exaggerated and the attributes considered above are not worth the attention they’ve been given? We’re going to provide real-life examples of what impact low-quality data can have on business processes.

Unreliable info

A manufacturer thinks that they know the exact location of the truck transporting their finished products from the production site to the distribution center. They optimize routing, estimate delivery time, etc. And it turns out that the location data is wrong. The truck arrives later, which disrupts the normal workflow at the distribution center. Not to mention routing recommendations that turned out useless.

Incomplete data

Say, you are working to optimize your supply chain management. To assess suppliers and understand which ones are disciplined and trustworthy and which ones are not, you track the delivery time. But unlike scheduled delivery time, the actual delivery time field is not mandatory in your system. Naturally, your warehouse employees usually forget to key it in. Not knowing this critical information (having incomplete data), you fail to understand how your suppliers perform.

Ambiguous data interpretation

A machinery maintenance system may have a field called “Breakdown reason” intended to help identify what caused the failure. Usually, it takes the form of a drop-down menu and includes the “Other” option. As a result, a weekly report may say that in 80% of cases the machinery failure was caused by the “Other” reason. Thus, a manufacturer can experience low overall equipment efficiency without being able to learn how to improve it.

Duplicated data

At a first glance, duplicated data may not pose a challenge. But in fact, it can become a serious issue. For example, if a customer appears more than once in your CRM, it not only takes up additional storage but also leads to a wrong customer count. Additionally, duplicated data weakens marketing analysis: it disintegrates a customer’s purchasing history and, consequently, makes the company unable to understand customer needs and segment customers properly.

Outdated information

Imagine that a customer once completed a retailer’s questionnaire and stated that they did not have children. However, time passed – and now they have a newborn baby. The happy parents are ready to spend their budget on diapers, baby food and clothes, but is our retailer aware of that? Is this customer included in “Customers with babies” segment? No to both. This is how obsolete data may result in wrong customer segmentation, poor knowledge of the market and lost profit.

Late data entry/update

Late data entries and updates may negatively affect data analysis and reporting, as well as your business processes. An invoice sent to the wrong address is a typical example to illustrate the case. And to spice the story up even more, here’s another example on asset tracking. The system can state that the cement mixer is unavailable at the moment only because the responsible employee is several hours late with updating its status. 

Want to avoid the consequences of poor data quality?

ScienceSoft offers services ranging from consulting to implementation to help you tune your data quality management process and ensure your decision-making won’t suffer from low data quality.

Best practices of data quality management

As the consequences of poor data quality can appear disruptive, it’s critical to learn what the remedies are. Here, we share best practices that can help you improve the quality of your data.

• Making data quality a priority

The first step is to make data quality improvement a high priority and ensure that every employee understands the problems that low data quality brings. Sounds quite simple. However, incorporating data quality management into business processes requires multiple serious steps:

1. Designing an enterprise-wide data strategy.
2. Creating clear user roles with rights and accountability.
3. Setting up a data quality management process (we’ll explain it in detail later in the article).
4. Having a dashboard to monitor the status quo.

A typical root cause for poor data quality is manual data entries: by employees, by customers or even by multiple users. Thus, companies should think how to automate data entry processes in order to reduce human error. Whenever the system can do something automatically (for example, autocompletes, call or e-mail logs), it is worth implementing.

• Preventing duplicates, not just curing them

A well-known truth is that it is easier to prevent a disease than cure it. You can treat duplicates in the same way! On the one hand, you can just regularly clean them. On the other hand, you can create duplicate detection rules. They allow identifying that a similar entry already exists in the database and forbid creating another one or suggest merging the entries.

• Taking care of both master and metadata

Nursing your master data is extremely important, but you shouldn’t forget about your metadata either. For example, without time stamps that metadata reveals, companies won’t be able to control data versions. As a result, they could extract obsolete values for their reports, instead of updated ones.

Data quality management: process stages described

Data quality management is a setup process, which is aimed at achieving and maintaining high data quality. Its main stages involve the definition of data quality thresholds and rules, data quality assessment, data quality issues resolution, data monitoring and control.

To provide as clear an explanation as possible, we’ll go beyond theory and explain each stage with an example based on customer data. Here is a sample snippet from a database:

1. Define data quality thresholds and rules

If you think there’s only one option – perfect data that is 100% compliant with all data quality attributes (in other words, 100% consistent, 100% accurate, and so on) – you may be surprised to know that there are more scenarios than that. First, reaching 100% everywhere is an extremely cost- and effort-intensive endeavor, so normally companies decide what data is critical and focus on several data quality attributes that are most applicable to this data. Second, a company not always needs 100% perfect data quality, sometimes they can do with the level that is ‘good enough.’ Third, if you need various levels of quality for various data, you may set various thresholds for different fields. Now, you may have a question: how to measure if the data meets these thresholds or not? For that, you should set data quality rules.

Now, when the theory part is over, we’re switching to a practical example.

Say, you decide that the customer full name field is critical for you, and you set a 98% quality threshold for it, while the date of birth field is of lesser importance, and you’ll be satisfied with 80% threshold. As a next step, you decide that customer full name must be complete and accurate, and the date of birth must be valid (that is to say, it should comply with the orderliness attribute). As you’ve chosen several data quality attributes for the customer full name, all of them should hit a 98% quality threshold.

Now you set data quality rules that you think will cover all the chosen data quality attributes. In our case, these are the following:

• Customer full name must not be N/A (to check completeness).
• Customer full name must include at least one space (to check accuracy).
• Customer name must consist only of letters, no figures allowed (to check accuracy).
• Only first letters in customer name, middle name (if any) and surname must be capitalized (to check accuracy).
• Date of birth must be a valid date that falls into the interval from 01/01/1900 to 01/01/2010.

2. Assess the quality of data

Now, it’s time to have a look at our data and check whether it meets the rules we set. So, we start profiling data or, in other words, getting statistical information about it. That’s how it works: we have 8 individual records (although your real data set is certainly much bigger than that) that we check against our first rule Customer full name must not be N/A. All the records comply with the rule, which means that data is 100% complete.

To measure data accuracy, we have 3 rules:

• Customer full name must include at least one space.
• Customer name must consist only of letters, no figures allowed.
• Only first letters in customer name, middle name (if any) and surname must be capitalized.

Again, we do data profiling, for each of the rules, and we get the following results: 100%, 88% and 88% (below, we’ve highlighted the records non-compliant to the data accuracy rule). In total, we have only 92%, which is also under our 98% threshold.

As for the date of birth field, we’ve identified two data records that don’t comply with the rule we set. So, data quality for this field is as high as 75%, which is also below the threshold.

3. Resolve data quality issues

At this stage, we should think what caused the issues to eliminate their root cause. In our example, we identified several problems for the customer full name field that can be solved by introducing clear standards for manual data entries, as well as data quality-related key performance indicators for the employees responsible for keying data into a CRM system.

In the example with the date of birth field, the data entered was not validated against the date format or range. As a temporary measure, we clean and standardize the data. But to avoid such mistakes in the future, we should set a validation rule in the system that will not accept a date unless it complies with the format and range.

4. Monitor and control data

Data quality management is not a one-time effort, rather a non-stop process. You need to regularly review data quality policies and rules with the intent to continuously improve them. This is a must, as the business environment is constantly changing. Say, one day a company may opt for enriching their customer data by purchasing and integrating an external data set that contains demographic data. So, probably, they’ll have to come up with new data quality rules, as an external data set can contain the data they haven’t dealt with so far.

Categories of data quality tools

To address various data quality issues, companies should consider not one tool but a combination of them. For example, Gartner names the following categories:

• Parsing and standardization tools break the data into components and bring them to a unified format.
• Cleaning tools remove incorrect or duplicated data entries or modify the values to meet certain rules and standards.
• Matching tools integrate or merge closely related data records.
• Profiling tools gather stats about data and later use it for data quality assessment.
• Monitoring tools control the status-quo of data quality.
• Enrichment tools bring in external data and integrate it into the existing data.

Currently, the market can boast a long list of data quality management tools. The trick is that some of them focus on a certain category of data quality issues, while others cover several aspects. To pick the right tools, you should either dedicate significant time to research or let professional consultants do this job for you.

Boundless data quality management squeezed into one paragraph

Data quality management guards you from low-quality data that can totally discredit your data analytics efforts. However, to do data quality management right, you should keep in mind many aspects. Choosing the metrics to assess data quality, selecting the tools, and describing data quality rules and thresholds are just several important steps. Hopefully, this complicated task can be fulfilled with professional assistance. At ScienceSoft, we are happy to back up your data quality management project at any stage, just let us know.

To prevent losing customers through customer attrition, companies turn to churn analytics. This type of analytics helps them measure, monitor and reduce the churn rate. The need for customer churn analytics is one of the reasons our clients turn to our BI implementation services. In this article, our BI experts summarize the main benefits customer churn analysis can bring and explain how to conduct it.

Why analyze customer churn?

To boost profit

As churn analysis provides you with meaningful insights into how to retain your customers, there appears an opportunity for additional profit. Just look at these numbers: increasing customer retention rates by 5% can boost your profits by as much as 25% and even more. We believe that it is already convincing enough to start analyzing customer churn.

To create a better customer experience

Effective churn analysis contributes to a deeper understanding of customer journeys. Considering the point where customers are likely to leave, companies can develop a set of retention activities to create a more comfortable customer experience and fulfill customer needs much better. This creates the conditions for growing a community of loyal customers who will share their positive experiences and become brand advocates.

To optimize products and services proactively

Customer churn analysis gives companies a quite accurate prediction about customer preferences: key attributes they are looking for in products/services, features they are dissatisfied with, triggers that make customers more likely to churn, etc. Empowered with such insights, companies have valuable data, which contributes to optimizing the existing product or creating one anew.

How to calculate customer churn?

Calculating customer (=subscription) churn alone is not informative enough for most businesses, as the percentage of all customers who choose to cease the relationship with your company does not reflect its impact on your bottom line.

To learn how customer churn affects business, you also need to calculate gross revenue churn (the percentage of revenue that is lost during a targeted period)

or employ more complex calculating methods.

How does customer churn analytics work?

Once you’ve rated your customer churn, customer data analytics and BI tools empower you to analyze it. To define triggers that cause customers to quit, you need to segment the leaving customers (through cohort analysis, analyses of churn rates by customer life cycle stages and behavior). The triggers empower you to define the likeliness of churn for every customer and set thresholds for defining at-risk customers. This way you can step in and take remedial actions for the sake of churn prevention. To create a predictive customer churn model, we recommend adding big data technologies into the analytical mix.

Stop your customers from turning their backs on you

By 2020, great customer experience is predicted to become the primary brand differentiator. And customer churn analysis allows businesses to continually improve customer experience and the overall brand image. Do you want to be among those companies? Drop a line to our BI implementation experts to stop giving out your revenue to the competition.

While the snowball of big data is rushing down a mountain gaining speed and volume, companies are trying to keep up with it. And down they go, completely forgetting to put on masks, helmets, gloves and sometimes even skis. Without these, it’s terribly easy to never make it down in one piece. And putting on all the precaution measures at a high speed can be too late or too difficult.

Prioritizing big data security low and putting it off till later stages of big data adoption projects isn’t always a smart move. People don’t say “Security’s first” for no reason. At the same time, we admit that ensuring big data security comes with its concerns and challenges, which is why it is more than helpful to get acquainted with them.

And as ‘surprising’ as it is, almost all security challenges of big data stem from the fact that it is big. Very big.

Short overview

Problems with security pose serious threats to any system, which is why it’s crucial to know your gaps. Here, our big data experts cover the most vicious security challenges that big data has in stock:

1. Vulnerability to fake data generation
2. Potential presence of untrusted mappers
3. Troubles of cryptographic protection
4. Possibility of sensitive information mining
5. Struggles of granular access control
6. Data provenance difficulties
7. High speed of NoSQL databases’ evolution and lack of security focus
8. Absent security audits

Now that we’ve outlined the basic problem areas of big data security, let’s look at each of them a bit closer.

#1. Vulnerability to fake data generation

Before proceeding to all the operational security challenges of big data, we should mention the concerns of fake data generation. To deliberately undermine the quality of your big data analysis, cybercriminals can fabricate data and ‘pour’ it into your data lake. For instance, if your manufacturing company uses sensor data to detect malfunctioning production processes, cybercriminals can penetrate your system and make your sensors show fake results, say, wrong temperatures. This way, you can fail to notice alarming trends and miss the opportunity to solve problems before serious damage is caused. Such challenges can be solved through applying fraud detection approach.

#2. Potential presence of untrusted mappers

Once your big data is collected, it undergoes parallel processing. One of the methods used here is MapReduce paradigm. When the data is split into numerous bulks, a mapper processes them and allocates to particular storage options. If an outsider has access to your mappers’ code, they can change the settings of the existing mappers or add ‘alien’ ones. This way, your data processing can be effectively ruined: cybercriminals can make mappers produce inadequate lists of key/value pairs. Which is why the results brought up by the Reduce process will be faulty. Besides, outsiders can get access to sensitive information.

The problem here is that getting such access may not be too difficult since generally big data technologies don’t provide an additional security layer to protect data. They usually tend to rely on perimeter security systems. But if those are faulty, your big data becomes a low hanging fruit.

#3. Troubles of cryptographic protection

Although encryption is a well-known way of protecting sensitive information, it is further on our list of big data security issues. Despite the possibility to encrypt big data and the essentiality of doing so, this security measure is often ignored. Sensitive data is generally stored in the cloud without any encrypted protection. And the reason for acting so recklessly is simple: constant encryptions and decryptions of huge data chunks slow things down, which entails the loss of big data’s initial advantage – speed.

#4. Possibility of sensitive information mining

Perimeter-based security is typically used for big data protection. It means that all ‘points of entry and exit’ are secured. But what IT specialists do inside your system remains a mystery.

Such a lack of control within your big data solution may let your corrupt IT specialists or evil business rivals mine unprotected data and sell it for their own benefit. Your company, in its turn, can incur huge losses, if such information is connected with new product/service launch, company’s financial operations or users’ personal information.

Here, data can be better protected by adding extra perimeters. Also, your system’s security could benefit from anonymization. If somebody gets personal data of your users with absent names, addresses and telephones, they can do practically no harm.

#5. Struggles of granular access control

Sometimes, data items fall under restrictions and practically no users can see the secret info in them, like, personal information in medical records (name, email, blood sugar, etc.). But some parts of such items (free of ‘harsh’ restrictions) could theoretically be helpful for users with no access to the secret parts, say, for medical researchers. Nevertheless, all the useful contents are hidden from them. And this is where talk of granular access starts. Using that, people can access needed data sets but can view only the info they are allowed to see.

The trick is that in big data such access is difficult to grant and control simply because big data technologies aren’t initially designed to do so. Generally, as a way out, the parts of needed data sets, that users have right to see, are copied to a separate big data warehouse and provided to particular user groups as a new ‘whole’. For a medical research, for instance, only the medical info (without the names, addresses and so on) gets copied. Though, the volumes of your big data grow even faster this way. Other complex solutions of granular access issues can also adversely affect the system’s performance and maintenance.

#6. Data provenance difficulties

Data provenance – or historical records about your data – complicates matters even more. Since its job is to document the source of data and all manipulations performed with it, we can only image what a gigantic collection of metadata that can be. Big data isn’t small in volume itself. And now picture that every data item it contains has detailed information about its origin and the ways it was influenced (which is difficult to get in the first place).

For now, data provenance is a broad big data concern. From security perspective, it is crucial because:

1. Unauthorized changes in metadata can lead you to the wrong data sets, which will make it difficult to find needed information.
2. Untraceable data sources can be a huge impediment to finding the roots of security breaches and fake data generation cases.

#7. High speed of NoSQL databases’ evolution and lack of security focus

This point may seem as a positive one, while it actually is a serious concern. Now NoSQL databases are a popular trend in big data science. And its popularity is exactly what causes problems.

Technically, NoSQL databases are continuously being honed with new features. And just like we said in the beginning of this article, security is being mistreated and left in the background. It is universally hoped that the security of big data solutions will be provided externally. But rather often it is ignored even on that level.

#8. Absent security audits

Big data security audits help companies gain awareness of their security gaps. And although it is advised to perform them on a regular basis, this recommendation is rarely met in reality. Working with big data has enough challenges and concerns as it is, and an audit would only add to the list. Besides, the lack of time, resources, qualified personnel or clarity in business-side security requirements makes such audits even more unrealistic.

But don’t be scared: they are all solvable

Yes, there are lots of big data security issues and concerns. And yes, they can be quite crucial. But it doesn’t mean that you should immediately curse big data as a concept and never cross paths with it again. No. The thing you should do is carefully design your big data adoption plan remembering to put security to the place it deserves – first. This may be a tricky thing to do, but you can always resort to professional big data consulting to create the solution you need.

Is your company determined to implement a data lake for your big data? That’s definitely exciting and great news! However, you have challenging times ahead, as you have so many fundamental issues to clarify and decide upon. At this stage, you are most likely interested in a data lake architecture and the required technology stack. To make your journey smooth and comfortable, our big data consultants have prepared the overview of alternative implementation approaches.

Zones in a data lake

A data lake is a repository intended for storing huge amounts of data in its native format. Data lake implementation will allow you to derive value out of raw data of various types. Unlike a data warehouse, a data lake has no constraints in terms of data type – it can be structured, unstructured, as well as semi-structured. In terms of architecture, a data lake may consist of several zones: a landing zone (also known as a transient zone), a staging zone and an analytics sandbox. Of all the zones mentioned, only staging is the obligatory one, while all the others are optional. To find out what each zone is for, let’s take a closer look at them.

1. Landing zone

Here comes the data (structured, unstructured and semi-structured) that undergoes preliminary cleaning and/or filtering. For example, you collect IoT data from sensors. If one of the sensors is sending abnormally high values while the other sensors that measure the same parameter have not registered anything unusual, a processing engine deployed within this zone will mark the values as erroneous.

2. Staging zone

There are two ways for data to appear in the staging zone. First, it can come from the landing zone (if any), like the sensor data from our previous example. Secondly, we can get data, which does not require any preprocessing, from other internal or external data sources. Customer comments in social networks will be a good example to illustrate this case.

3. Analytics sandbox

This is the zone for data experiments driven by data analysts. It is different from the analytics as we know it, as its findings (if any) are not directly used by business. By the way, we deliberately specified this if any. It happens quite often that analysts apply some models or algorithms to raw data (which may also be coupled with the data from a big data warehouse or from other internal or external data sources) and get no valuable findings. For exploratory data analytics, this is normal.

4. And one more zone under question – curated data zone

By now, our list should have been over, if there weren’t one slight hitch. In some sources, you may come upon one more component of a data lake – the curated data zone. This is the zone with organized data ready for data analysis.

There exist different opinions about whether the curated data zone should be considered a part of a data lake or not. While both approaches are reasonable, we think that it should rather not. However, prior to providing the arguments to support our point of view, let’s put the terminology to order.

Have another look at the description of the curated data zone. Doesn’t it look very similar to a good old traditional data warehouse? It absolutely does! The only difference is that a traditional data warehouse deals with traditional data only, while the curated data zone – with both traditional and big data. To neutralize the influence of data types, let’s extend the name to a big data warehouse.

Now, after we clarified that the curated data zone can as well be called a big data warehouse, let’s discuss why we consider that it’s outside a data lake. The data stored in a big data warehouse is fundamentally different from the data in any zone of a data lake – it is more organized, and it is already the source of insights for business users.

Besides, at this stage of data journey, the differentiation between traditional and big data becomes uncritical. Both types peacefully coexist and complement each other to fulfill its purpose – to provide business users with insights. For example, to segment customers, you can analyze a lot of data among which there will be big data such as surfing history on the website and the activities in customer mobile apps. Later you can run reports on sales or profit per customer segment, which is pure traditional business intelligence.

If you wonder why then a big data warehouse is sometimes considered a part of a data lake, we have an explanation for this as well. Most businesses that decide to take the advantage of big data already have a traditional data warehouse in place. So, they usually choose to extend their analytical solution by building a data lake around it. In this case, a traditional data warehouse remains a habitual important element and all new elements are associated with a data lake.

Technological alternatives for implementing a data lake

The list of technologies for big data storage includes a myriad of names: Hadoop Distributed File System, Apache Cassandra, Apache HBase, Amazon S3, MongoDB are just a few most popular ones. Undoubtedly, while selecting a technology stack for a data lake, one will think first of the technologies that enable big data storage. The foundation is the right one, though you need to think about processing as well. So, the list of technologies should be further extended with Apache Storm, Apache Spark, Hadoop MapReduce, etc. No wonder if you are puzzled what combination is the best choice for your data lake!

1. Defining factors to choose a technology stack

Despite each case is individual, we’ve summed up five important factors that will become a starting point of your discussion with your big data consultants:

• Data to be stored and processed: IoT big data, texts, video, etc.
• Required architecture of a data lake
• Scalability
• In-cloud or on-premises solution
• Integration with the existing components of IT architecture.

Is there a leading technology?

According to general big data consulting practice, Hadoop Distributed File System (HDFS) is the most popular among the multitude of possible technologies for a big data lake. The reasons are as follows:

• HDFS is extremely good at handling the diversity of data in a big data lake. IoT big data, video and audio files and text records – with HDFS you can store every data type. If we compare, Apache Cassandra is good for storing IoT big data, while MongoDB – texts.
• HDFS supports a wide range of processing techniques. HDFS is one of the elements of Apache Hadoop ecosystem that includes multiple other components such as Hadoop MapReduce, Hadoop YARN, Apache Hive, Apache HBase, etc. As they belong to the same family, it’s natural that each of them is highly compatible with HDFS. Besides, HDFS has proved to be highly compatible with Apache Spark which gives an opportunity to perform big data processing quickly.

Of course, you can also consider other technologies to implement a data lake. An important criterion is to know how to bypass their limitations. For example, after comparing HDFS and Cassandra, you can decide to run a data lake on the latter. Why not, if you are planning a data lake exclusively as a staging area for IoT data, and you know how to compensate Cassandra’s lack of joins?

2. Data lake as a service

Amazon Web Services, Microsoft Azure, Google Cloud Platform have a relevant offer – a data lake as a service. In fact, it would be difficult for a newbie to spot the differences among these three offers. In essence, they are quite similar: you need an AWS/Azure/GCP account, your data and willingness to pay for the service. In return, you get a predefined set of technologies deployed in the cloud and get rid of a maintenance headache. Under-the-hood technology stack is, of course, different, though the functions they perform are habitual ones: storage, processing, streaming and analytics. We are planning to write a separate blog post revealing the pros and cons of these three offers. So, stay tuned.

Let’s briefly recap

What are the main factors that influence the choice of technologies for a data lake?

• The types of data to be stored and processed
• The zones of a data lake (only a staging zone or a landing zone and an analytics sandbox)
• Scalability
• In-cloud or on-premises solution
• Integration with the existing components of IT architecture.

In the end, should we opt for one technology only?
No, you shouldn’t. Our practice shows that data lake solutions are implemented based on several technologies. To solve a business task, big data consultants can choose a separate technology for each zone of a data lake.

Is there a preferred technology for a data lake?
Hadoop Distributed File System is the most popular, yet not the only technology available. However, be careful and rely on your business goals and, correspondingly, requirements to your future analytical solution rather than on a framework’s popularity.

If I do not want to implement a data lake from scratch, can I opt for a ready-to-use solution?
Yes, you can. Amazon Web Services, Microsoft Azure and Google Cloud Platform offer a data lake as a service. What is needed from you – your data and your subscription and service fees. And you get a data lake that it easy and fast to deploy.

Editor’s note: When you implement a big data solution, choosing the right storage is the first order of business. Read on to learn about the big data solution options and don’t hesitate to explore our approach delivering to big data services, if you need to back up your big data project.

When ScienceSoft’s clients need to design their big data solution, we offer them to structure it with two storage elements: a data lake and a big data warehouse, which we distinguish from a traditional enterprise data warehouse. Here, looking ahead, we should say that a big data warehouse, unlike a data lake, is an obligatory element of a full-scale big data analytical solution. But, first things first, let us show you how data lakes and big data warehouses are different from each other in terms of architecture and their functional purpose.

The differences between a data lake and a data warehouse

Data state

ScienceSoft’s big data experts employ data lakes for storing all kinds of data – structured, unstructured and semi-structured. As for a big data warehouse, we use it as a storage for structured data.

The approach to storing data

A big data warehouse stores data according to the schema-on-write approach: before loading into the big data warehouse, data needs to be transformed into a unified structure to be fit for the big data warehouse.

A data lake stores data according to the schema-on-read approach: raw data is loaded into the data lake as it is and applied to the schema only when it is read. Thus, storing data in a data lake requires less effort.

Architecture

When speaking of a data lake, its flexible architecture may involve three elements:

• A landing zone – a transient area, where data undergoes preliminary filtering.
• A staging zone – a storage repository.
• An analytics sandbox – the area where data analysts perform experiments for exploratory data analytics.

When developing a data lake solution, our experts consider the staging zone the only obligatory element. If you want to learn more about the data lake zones and why we consider the landing zone and the analytical sandbox optional, study this article, here our data analytics researcher, Irene Mikhailouskaya, dwells on the data lake architecture.

As concerns the big data warehouse, it has a rigid architecture. Its elements are highly structured and obligatory as they are tied to business processes for the big data warehouse to correctly analyze and report data.

Storage costs

Drawing on our experience in rendering big data services, we have to admit that storing data in a big data warehouse is costly as you cannot load data unless it is of the required structure. And such a preparatory process is rather time- and resource-consuming. Thereby, we usually recommend our clients to consider integrating a data lake into the big data warehouse architecture as a cost-effective alternative: storing data in the data lake involves minimum or no data structuring before being loaded.

Users

Big data warehouses cater to the needs of business users and data analysts who use big data strategically to improve the decision-making process. Data lakes are mainly used as temporary storage of big data and the zone for data scientists and analysts to drive experiments.

Technologies

As both the big data warehouse and the data lake deal with big data, there is no difference in the technology stack to employ for storing, streaming and processing data:

Security

The use of big data is associated with certain security challenges. When developing big data solutions, ScienceSoft’s experts pay special attention to the high granularity of access control, when users’ access is limited depending on their roles. This measure prevents sensitive data leakage.

As opposed to big data warehouses, data lakes lack security focus due to the nature of stored data and its functional purpose. As only a limited number of users are granted the access, a data lake is protected as a whole, following the “all-or-nothing” approach.

Don’t know how your big data solution should look?

ScienceSoft’s team is ready to advise you on how to leverage big data potential with a tailor-made solution.

The synergy of the data lake and the big data warehouse

Many big data project sponsors we talk to wonder if they can use a data lake or a big data warehouse alone in a data analytics solution. Our answer is it’s not an either-or choice: a data lake alone is never enough to design a full-scale big data analytics solution. We often recommend having the synergy of both. This is the case of businesses who need to both store large amounts of raw data to conduct experiments, and deliver intelligence to decision-makers. One of the telling examples when both elements function in sync within the one big data solution is an IoT solution, where the initial sensor data is stored in its raw format in the data lake, and then it undergoes the ETL\ELT process to be stored in the big data warehouse for further analysis. Such an alliance allows leveraging big data potential time- and cost-effectively.

How to start your big data journey?

Now, that you know your options, you need to decide whether your big data solution’s architecture will involve a big data warehouse and a data lake, or just a big data warehouse. To choose which way to go, you need to define:

• For what purposes your data will be used.
• What your requirements to data quality, speed of the data flow and the need for analytical experiments are.
• Who will use the data.

There are many factors to take into account and balance when deciding on the high-level big data architecture. We saw how long-drawn-out architectural decisions delayed actual big data implementation for years. And, unfortunately, we’ve witnessed how a wrong decision may result in massive rework later. ScienceSoft’s big data team would be happy to help with consulting or architecture design.

Use cases

• Drilling processes optimization.
• Predictive and preventive maintenance.
• Equipment maintenance planning.
• Remote equipment monitoring and control.
• Inventory management optimization.

How it works: Sensors installed on the drilling equipment send temperature, pressure, vibration, flow, position, torque, and other readings. Gathered and analyzed by the big data solution, this data powers real-time insights into drilling processes (e.g., drilling direction, drilling fluid composition and pressure, drilling bit position) and intelligent software-to-equipment commands (e.g., adjusting drilling bit position to target specific formations or avoid obstruction).

Big data software also gathers equipment operational data (e.g., the rotation speed of the drilling bit, drilling fluid temperature) and equipment metadata (e.g., model, operational settings). This data is used to build accurate AI/ML models that help generate alerts on abnormal events and identify failure-causing equipment usage patterns. This allows O&G companies to minimize NPT (non-productive time), optimize inventory management processes and equipment maintenance schedules, extend equipment lifespan, and more.

Drilling and equipment data can also be coupled with real-time and historical geological data (e.g., rock formation evaluation, mud properties) to build and adjust drilling models, predict anomalies, and prevent unwanted events like kicks and blowouts.

Imagine that in 2017 someone is still actually posting letters because they think email is difficult and costly. Each time they have to bring their letters to a post office, where later a postman collects them to deliver. Delivery then takes at least a few days, while emails could be exchanged instantly. Inefficient and time-consuming, isn’t it? It’s surprising, but this is what some companies do when they overlook software for cash flow analysis and choose to carry on with Excel-based manual data processing.

BI consulting practitioners break the stereotype that only big companies need tech-based cash flow analysis and forecasting. In fact, midsized businesses need them too to keep track of their cash, and do it efficiently. Long gone the time when companies had no other alternatives than to go through the effort of using several Excel files, matching and filtering them manually to get a comprehensive picture. Now, the technology is convenient and affordable.

What can be analyzed?

A cash flow analysis answers a range of questions. For example, a real estate developer can check if there is enough cash to invest in a new project; a manufacturer whether external funding is needed to revamp the plant machinery; a retailer how much money is buried in stock; a bank if the cash flow is adequate to meet the liquidity coverage ratio.

These are just practicalities. The main question is whether a cash flow is sustainable.

To enable a comprehensive cash flow analysis and forecasting, a company can aggregate data from numerous sources:

• Cash flow history
• Planned and actual operating expenses and capital expenditures
• Accounts receivables/payable balances
• General ledger data

Besides, all the figures can be taken directly from the ERP modules (finance, accounting, sales, human resources, etc.). This means that all the values are kept up-to-date and can automatically turn from forecasted to actual, whenever confirmed.

The challenges solved with technology

Cash flow planning and forecasting

Cash flow analysis software will equip managers with timely, accurate and easy-to-use reports and charts giving an overview by geography, branches, bank account, etc. Cash forecasts can be generated automatically as frequent as necessary (daily, weekly, quarterly or monthly), as well as on demand. With such forecasts, financial managers will get an advance warning of cash shortage or surplus and will have time to take some actions (for instance, invest in company’s growth if there is free cash).

Project-level view

Big projects, both internal and for customers beyond the organization, may influence cash flow dramatically. Financial managers cannot treat big projects as black boxes, they need to look inside. For any project, it is necessary to know its duration, cost of each stage, terms of payment (prepayment or payment determent; a lump sum or by installments). Data analysis contributes to both an accurate cash plan and successful project implementation.

Risk management

With cash flow analysis software, a company has a reliable tool to manage risks. For example, a manufacturing company operates at a profit (its P&L says so) and wants to increase its production volume. However, the company’s cash flow forecast shows that there is not enough cash. Additionally, the software makes a quick projection with increased cost of goods produced, which shows that the company will only be losing money in a long-term perspective. A quick liquidity analysis helps to make the right decision.

A point of truth from different angles – instantly

Another advantage of a tech-based cash flow analysis is that the data from different sources is aggregated at the data warehouse level. For end users, this means fast response and a quick glance from different perspectives, as the system already has an answer to any question and just waits for a query. For example, financial managers can switch between the cash flow from operations to the cash flow from investing and then to the cash flow from financing all in a few clicks.

To sum up

Cash flow management is crucial to ensure that a business is healthy. A company that leaves its cash flow uncontrolled, risks to end up with insolvency and to ruin its reputation. Business intelligence services is there to help companies adopt the cash flow analysis technology that brings value, eliminates a big chunk of manual work, accelerates decision-making and solves such challenges as cash flow planning and forecasting, project-level view, risk management, and analysis from different perspectives in a few clicks.

Apache Cassandra obviously can’t tell the future. It can only enable you to organize data storage (or at least make it as organized as it can get in a distributed system). But how good is Cassandra at it? Find all the needed details below so that Cassandra performance is not all Greek to you anymore.

Terms you may not know yet

Down below, our Cassandra specialists use quite a lot of specific terms that you may encounter for the first time. Here, you may find all these terms briefly explained.

Token is a somewhat abstract number assigned to every node of the cluster in an ascending manner. All the nodes form a token ring.

Partitioner is the algorithm that decides what nodes in the cluster are going to store data.

Replication factor determines the number of data replicas.

Keyspace is the global storage space that contains all column families of one application.

Column family is a set of Cassandra’s minimal units of data storage (columns). Columns consist of a column name (key), a value and a timestamp.

Memtable is a cache memory structure.

SSTable is an unchangeable data structure created as soon as a memtable is flushed onto a disk.

Primary index is a part of the SSTable that has a set of this table’s row keys and points to the keys’ location in the given SSTable.

Primary key in Cassandra consists of a partition key and a number of clustering columns (if any). The partition key helps to understand what node stores the data, while the clustering columns organize data in the table in ascending alphabetical order (usually).

Bloom filters are data structures used to quickly find which SSTables are likely to have the needed data.

Secondary index can locate data within a single node by its non-primary-key columns. SASI (SSTable Attached Secondary Index) is an improved version of a secondary index ‘affixed’ to SSTables.

Materialized view is a means of ‘cluster-wide’ indexing that creates another variant of the base table but includes the queried columns into the partition key (while with a secondary index, they are left out of it). This way, it’s possible to search for indexed data across the whole cluster without looking into every node.

Data modeling in Cassandra

Cassandra’s performance is highly dependent on the way the data model is designed. So, before you dive into it, make sure that you understand Cassandra’s three data modeling ‘dogmas’:

1. Disk space is cheap.
2. Writes are cheap.
3. Network communication is expensive.

These three statements reveal the true sense behind all Cassandra’s peculiarities described in the article.

And as to the most important rules to follow while designing a Cassandra data model, here they are:

• Do spread data evenly in the cluster, which means having a good primary key.
• Do reduce the number of partition reads, which means first thinking about the future queries’ composition before modeling the data.

Data partitioning and denormalization

To assess Cassandra performance, it’s logical to start in the beginning of data’s path and first look at its efficiency while distributing and duplicating data.

Partitioning and denormalization: The process

While distributing data, Cassandra uses consistent hashing and practices data replication and partitioning. Imagine that we have a cluster of 10 nodes with tokens 10, 20, 30, 40, etc. A partitioner converts the data’s primary key into a certain hash value (say, 15) and then looks at the token ring. The first node whose token is bigger than the hash value is the first choice to store the data. And if we have the replication factor of 3 (usually it is 3, but it’s tunable for each keyspace), the next two tokens’ nodes (or the ones that are physically closer to the first node) also store the data. This is how we get data replicas on three separate nodes nice and easy. But besides that, Cassandra also practices denormalization and encourages data duplication: creating numerous versions of one and the same table optimized for different read requests. Imagine how much data it is, if we have the same huge denormalized table with repeating data on 3 nodes and each of the nodes also has at least 3 versions of this table.

Partitioning and denormalization: The downside

The fact that data is denormalized in Cassandra may seem weird, if you come from a relational-database background. When any non-big-data system scales up, you need to do things like read replication, sharding and index optimization. But at some point, your system becomes almost inoperable, and you realize that the amazing relational model with all its joins and normalization is the exact reason for performance issues.

To solve this, Cassandra has denormalization as well as creates several versions of one table optimized for different reads. But this ‘aid’ does not come without consequence. When you decide to increase your read performance by creating data replicas and duplicated table versions, write performance suffers a bit because you can’t just write once anymore. You need to write the same thing n times. Besides, you need a good mechanism of choosing which node to write to, which Cassandra provides, so no blames here. And although these losses to the write performance in Cassandra are scanty and often neglected, you still need the resources for multiple writes.

Partitioning and denormalization: The upside

Consistent hashing is very efficient for data partitioning. Why? Because the token ring covers the whole array of possible keys and the data is distributed evenly among them with each of the nodes getting loaded roughly the same. But the most pleasant thing about it is that your cluster’s performance is almost linearly scalable. It sounds too good to be true, but it is in fact so. If you double the number of nodes, the distance between their tokens will decrease by half and, consequently, the system will be able to handle almost twice as many reads and writes. The extra bonus here: with doubled nodes, your system becomes even more fault-tolerant.

The write

Write: The process

After being directed to a specific node, a write request first gets to the commit log (it stores all the info about in-cache writes). At the same time, the data gets stored in the memtable. At some point (for instance, when the memtable is full), Cassandra flushes the data from cache onto the disk – into SSTables. At the same moment, the commit log purges all its data, since it no longer has to watch out for the corresponding data in cache. After a node writes the data, it notifies the coordinator node about the successfully completed operation. And the number of such success notifications depends on the data consistency level for writes set by your Cassandra specialists.

Such a process happens on all nodes that get to write a partition. But what if one of them is down? There’s an elegant solution for it – hinted handoff. When the coordinator sees that a replica node is not responding, it stores the missed write. Then, Cassandra temporarily creates in the local keyspace a hint that will later remind the ‘derailed’ node to write certain data after it goes back up. If the node doesn’t recover within 3 hours, the coordinator stores the write permanently.

Write: The downside

Still, the write is not perfect. Here’re some upsetting things:

• Append operations work just fine, while updates are conceptually missing in Cassandra (although it’s not entirely right to say so, since such a command exists). When you need to update a certain value, you just add an entry with the same primary key but a new value and a younger timestamp. Just imagine how many updates you may need and how much space that will take up. Moreover, it can affect read performance, since Cassandra will need to look through lots of data on a single key and check whichever the newest one is. However, once in a while, compaction is enacted to merge such data and free up space.
• The hinted handoff process can overload the coordinator node. If this happens, the coordinator will refuse writes, which can result in the loss of some data replicas.

Write: The upside

Cassandra’s write performance is still pretty good, though. Here’s why:

• Cassandra avoids random data input having a clear scenario for how things go, which contributes to the write performance.
• To make sure that all the chosen nodes do write the data, even if some of them are down, there’s the above-mentioned hinted handoff process. However, you should note that hinted handoff only works when your consistency level is met.
• The design of the write operation involves the commit log, which is nice. Why? If a node goes down, replaying the commit log after it’s up again will restore all the lost in-cache writes to the memtable.

The read

Read: The process

When a read request starts its journey, the data’s partition key is used to find what nodes have the data. After that, the request is sent to a number of nodes set by the tunable consistency level for reads. Then, on each node, in a certain order, Cassandra checks different places that can have the data. The first one is the memtable. If the data is not there, it checks the row key cache (if enabled), then the bloom filter and then the partition key cache (also if enabled). If the partition key cache has the needed partition key, Cassandra goes straight to the compression offsets, and after that it finally fetches the needed data out of a certain SSTable. If the partition key wasn’t found in partition key cache, Cassandra checks the partition summary and then the primary index before going to the compression offsets and extracting the data from the SSTable.

After the data with the latest timestamp is located, it is fetched to the coordinator. Here, another stage of the read occurs. As we’ve stated here, Cassandra has issues with data consistency. The thing is that you write many data replicas, and you may read their old versions instead of the newer ones. But Cassandra doesn’t ignore these consistency-related problems: it tries to solve them with a read repair process. The nodes that are involved in the read return results. Then, Cassandra compares these results based on the “last write wins” policy. Hence, the new data version is the main candidate to be returned to the user, while the older versions are rewritten to their nodes. But that’s not all. In the background, Cassandra checks the rest of the nodes that have the requested data (because the replication factor is often bigger than consistency level). When these nodes return results, the DB also compares them and the older ones get rewritten. Only after this, the user actually gets the result.

Read: The downside

Cassandra read performance does enjoy a lot of glory, but it’s still not entirely flawless.

• All is fine as long as you only query your data by the partition key. If you want to do it by an out-of-the-partition-key column (use a secondary index or a SASI), things can go downhill. The problem is that secondary indexes and SASIs don’t contain the partition key, which means there’s no way to know what node stores the indexed data. It leads to searching for the data on all nodes in the cluster, which is neither cheap nor quick.
• Both the secondary index and the SASI aren’t good for high cardinality columns (as well as for counter and static columns). Using these indexes on the ‘rare’ data can significantly decrease read performance.
• Bloom filters are based on probabilistic algorithms and are meant to bring up results very fast. This often leads to false positives, which is another way to waste time and resources while searching in the wrong places.
• Apart from the read, secondary indexes, SASIs and materialized views can adversely affect the write. In case with SASI and secondary index, every time data is written to the table with an indexed column, the column families that contain indexes and their values will have to be updated. And in case with materialized views, if anything new is written to the base table, the materialized view itself will have to be changed.
• If you need to read a table with thousands of columns, you may have problems. Cassandra has limitations when it comes to the partition size and number of values: 100 MB and 2 billion respectively. So if your table contains too many columns, values or is too big in size, you won’t be able to read it quickly. Or even won’t be able to read it at all. And this is something to keep in mind. If the task doesn’t strictly require reading this number of columns, it’s always better to split such tables into multiple pieces. Besides, you should remember that the more columns the table has, the more RAM you’ll need to read it.

Read: The upside

Fear not, there are strong sides to the read performance as well.

• Cassandra provides excitingly steady data availability. It doesn’t have a single point of failure, plus, it has data stored on numerous nodes and in numerous places. So, if multiple nodes are down (up to half the cluster), you will read your data anyway (provided that your replication factor is tuned accordingly).
• The consistency problems can be solved in Cassandra through the clever and fast read repair process. It is quite efficient and very helpful, but still we can’t say it works perfectly all the time.
• You may think that the read process is too long and that it checks too many places, which is inefficient when it comes to querying frequently accessed data. But Cassandra has an additional shortened read process for the often-needed data. For such cases, the data itself can be stored in a row cache. Or its ‘address’ can be in the key cache, which facilitates the process a lot.
• Secondary indexes can still be useful, if we’re speaking about analytical queries, when you need to access all or almost all nodes anyway.
• SASIs can be an extremely good tool for conducting full text searches.
• The mere existence of materialized views can be seen as an advantage, since they allow you to easily find needed indexed columns in the cluster. Although creating additional variants of tables will take up space.

Cassandra performance: Conclusion

Summarizing Cassandra performance, let’s look at its main upside and downside points. Upside: Cassandra distributes data efficiently, allows almost linear scalability, writes data fast and provides almost constant data availability. Downside: data consistency issues aren’t a rarity and indexing is far from perfect.

Obviously, nobody’s without sin, and Cassandra is not an exception. Some issues can indeed influence write or read performance greatly. So, you will need to think about Cassandra performance tuning if you encounter write or read inefficiencies, and that can involve anything from slightly tweaking your replication factors or consistency levels to an entire data model redesign. But this in no way means that Cassandra is a low-performance product. If compared with MongoDB and HBase on its performance under mixed operational and analytical workload, Cassandra – with all its stumbling blocks – is by far the best out of the three (which only proves that the NoSQL world is a really long way from perfect). However, Cassandra’s high performance depends a lot on the expertise of the staff that deals with your Cassandra clusters. So, if you choose Cassandra, nice job! Now, choose the right people to work with it.

So why is business intelligence important? Business analysts often say that the main advantage of a BI solution is “eliminating guesswork from your business processes”. This becomes possible because business intelligence is used to analyze data and present actionable insights to stimulate informed decision making within an enterprise.

We employ our experience in BI implementation services to explain what advantages a company can get leveraging business intelligence. To add practicality, we’ll show how these major benefits of business intelligence are rocked by Starbucks, an early BI adopter and savvy user of data-driven business analytics.

Benefit 1. Understanding customers and tuning the company’s offering accordingly

BI tools enable companies to process customer data from multiple sources and create a 360-degree customer profile. As one of the goals of business intelligence is to present business-critical data in an easy-to-understand manner, companies can clearly understand their customers’ needs and behavior. As a result, they are empowered to tune their offering accordingly and deliver top-notch products and services.

Launching new product lines

Starbucks used BI to analyze industry reports about at-home beverage consumption and data about how customers order products while in a Starbucks shop. The company employed this info to create K-Cups and bottled beverages to sell in grocery stores. This let the company prevent customers from using other coffee brands at home.

Defining store locations

Since 2008, Starbucks has used Atlas, BI mapping software, to locate their new stores. The platform allows the company to estimate the economic viability of a new store location by evaluating massive amounts of data, such as area population density, average income of the residents, traffic patterns and proximity to other Starbucks locations. Thus, the company opens stores exactly where customers need them, which boosts sales without hurting the business in other locations.

Benefit 2. Boosting sales and marketing activities

With BI solutions, companies take a closer look at multidimensional retail data (from transactions to social media) to forecast customer needs and define sales and marketing activities to meet the demand.

Anticipating customer demand

Using predictive analytics, Starbucks successfully beefs up sales with analytics-powered in-store digital boards. The boards display items based on time of the day, weather, social trends and more: for example, breakfast items in the morning, hot drinks in colder weather, holiday specialties, dairy-free alternatives, etc. Such a data-driven approach allows Starbucks to entice customers with a more appealing offering.

Customizing order suggestions

Starbucks’ mobile app with more than 16 million active users presents the wealth of data on customers’ purchasing habits. To make use of that data correctly, the company uses a reinforcement learning platform to offer customers tailor-made order suggestions based on their popular selections, order history and the inventory of a local store. Through providing such personalized experience for customers, the company boosts both customer loyalty and sales.

Watch BI in action!

ScienceSoft shows how a couple of customized dashboards can tell you the whole story about your company’s health and performance.

Benefit 3. Optimizing back-end operations

A BI solution can analyze data to advance a company’s internal business processes, such as order management, scheduling, staffing, inventory management, and supply chain management.

Staffing and staff scheduling in a smart way

Due to staff scheduling optimization, it takes about three minutes from the time when a Starbucks customer gets in line until the order is delivered – regardless of the part of the day they come in. BI tools help store managers constantly monitor store performance against labor efforts to identify how well the store is doing with the current staff. With these tools, managers can optimize the work by growing or downsizing staff and rescheduling some duties (cleaning work areas, coffee machines, etc.) to perform them in quiet periods or after hours. That way, baristas are not overloaded and have the time for small talk with a customer or, for example, drawing a balloon on a cup for a birthday person. Such personalized approach increases customer satisfaction, which lies in the heart of the company’s customer-centric policy.

Optimizing supply chain management

Starbucks’ supply chain has no room for waste and inefficiency due to the centralized BI practice. BI software enables on-demand access to constantly updated information on stock inventory, transport scheduling and storage capacity. Based on real-time reports, the company manages to react with agility to, for example, poor quality of raw products by finding alternative suppliers while maintaining competitive prices.

Benefit 4. Keeping a close eye on the competition

As a part of a BI solution, benchmarking delivers practical insights on how to outperform competitors. Competitive analysis allows continuously improving a company’s performance.

Finding new ways of service delivery

Competitive benchmarking empowered Starbucks to uncover the need to reconfigure their third place strategy, which presupposed stores that provide both take-away and eat-in options. Following in the footsteps of McDonald’s, Luckin Coffee and other competitors, Starbucks launched food delivery service and opened pick-up only stores. The company’s efforts seem to be working: they managed to reach customers beyond those who already include Starbucks as part of their morning or afternoon routine.

And this all leads to – Benefit 5. Revenue increase and cost reduction

Businesses effectively employing BI software earn more by analyzing customers and their demands, boosting marketing and sales activities, optimizing business-supporting operations and benchmarking.

As for Starbucks, successfully employed BI and data analytics practices empower them not only to attract customers, consequently boosting the volume of sales, but also to follow the ever-changing consumers’ behavior to the extent that few companies have yet been able to accomplish.

How to gain from BI?

ScienceSoft can help you leverage the BI benefits that Starbucks and many other companies already enjoy. Turn to our BI consultants as a first step to introducing a BI solution to your business.

Start your BI implementation journey

While real-time analytics and big data are both trending, it seems that real-time big data analytics, which is their combination, should be a very promising initiative, and many businesses should be desirous of it. Let’s find out if this is really so.

You will find this article richly supplied with the examples of real-time customer big data analytics. We’ve done so for the reasons of ease and consistency. Though there are more areas where real-time data analytics can be applied.

Let’s start from defining the term

If you are going to skip this section because you think there can’t be two definitions of real-time, please don’t be surprised – there are. In fact, the definition of real-time is extremely vague, and it differs a lot from company to company or, to be more exact, from business task to business task.

Our big data consulting team has come up with the following definition:

Real-time big data analytics means that big data is processed as it arrives and either a business user gets consumable insights without exceeding a time period allocated for decision-making or an analytical system triggers an action or a notification.

As real-time is often confused with instantaneous, let’s clarify the time frames for data input and response. As far as data input is concerned, the real-time processing engine can be designed to either push or pull data. The most widespread example is a push option with an incessantly flowing high-volume data (also known as streaming). However, the real-time processing engine is not always capable of ingesting streaming data. Alternatively, it can be designed to pull data by asking if any new data has arrived. The time between such queries depends on business needs and can vary from milliseconds to hours.

Correspondingly, the response time also varies. For instance, a self-driving car requires a very fast response time – just several milliseconds. If we deal with sensors installed, say, on a wind turbine, and they communicate a slowly growing gearbox oil temperature, which is still below the critical level but higher than normal, we need one-minute response time to change blade pitch, thus offloading the turbine and preventing machine breakdown or even fire. However, a bank’s analytical system would allow several minutes to assess the creditworthiness of an applicant; and a retailer’s dynamic pricing can take up to an hour to update. Still, all these examples are considered real-time.

Real-time big data analytics as a competitive advantage

Although in general organizations value managing data in real time, not all the companies go for real-time big data analytics. The reasons could be different: the lack of expertise or insufficient funds, the fear of the associated challenges or overall management team’s reluctance. However, those companies who implement real-time analytics can gain a competitive advantage.

Let’s say you are a fashion retailer who would like to take the advantage by delivering a top-notch customer service. Analyzing big data in real time can help bring this great initiative into life. For example, once a customer is passing by a retailer’s store, they get a push notification on their smartphones that serves to incentivize them to enter. Usually, it’s a personalized promo offer that is based on the customer’s purchasing or even surfing history on the website. Once a customer is in the store, the staff gets a notification in their mobile apps. This makes them aware of the customer’s latest purchases, overall style preferences, interest in promotions, a typical spend, etc. It looks like a win-win situation for both customers and retailers, doesn’t it?

An ecommerce retailer can also achieve better performance by analyzing big data in real time. For instance, they can reduce the number of abandoned carts.  Say, a customer has gone that far, but for some reason, they’ve decided not to finalize their purchase. Still, there are good chances to incentivize them to change their mind. The system is turning to the customer’s profile data, as well as the purchasing and surfing history to compare the customer’s behavior with the conduct of other customers from the same segment and their response to different actions in a similar situation. Based on the analysis results, the system chooses the most suitable of all the possible actions – for example, offers a discount.

A typical architecture for real-time big data analytics

Let’s have a look at how a typical real-time big data analytics solution works. To make the explanation more vivid, we will accompany it with an example that is illustrative for everybody, as, now and again, we all assume the role of a customer.

Imagine a retailer that is aiming to deliver a personalized customer experience. The first step on this long road is to recognize a customer, once they are in the store. A retailer can achieve this in multiple ways, for example, by implementing face recognition.

With this only data source, the retailer can do a simple analysis, like calculate how many male and female customers are currently in the store. However, the retailer will not satisfy themselves with one data source only. Even to know how many of the customers have come for the first time and how many are regulars, another data source is needed, for example, CRM. The general context will also be helpful, for instance, the information about the store’s opening hours.

After processing, real-time data finds its way to a real-time dashboard or turns into either a notification or a system’s action. We’ve already provided the example for the first case, when the retailer can understand how many customers are in the store at the moment. Let’s look at another option in detail. Say, a customer has formed a shopping list in the mobile app and is moving around the store. Based on the customer’s current location data (gathered by beacons and processed by the same real-time analytics), the app can prompt the most optimal way along the sales floor so that they can grab everything that is on their list.

Let’s continue with the above mentioned example to explain the contribution of machine learning. By the way, machine learning itself does not happen in real time. It’s an elaborate process, and the system requires significant time to analyze an enormous volume of data, which usually covers the period of 1+ year, from different angles to come up with valuable models and patterns. These models help the system to make real-time decisions. Now, to the example: the system has already analyzed customer profiles and segments they belong to, their behavior model, the purchasing history, the response to marketing campaigns, etc. and built a model that enables personalized recommendations. And while the customer is walking in the aisles, the system can notify them about promo offers or related products that the customer will find interesting.

The concept of machine learning also requires model verification applications, as they enable a constant improvement of the models’ accuracy. Additionally, they improve the quality of the input data by allowing a basic filtering from erroneous or noisy data.

Now let’s turn our eyes to data storage. It consists of two components: a data lake and a data warehouse. The former is the place to store all the raw data or the data that has undergone a very simple processing. A data warehouse allows making big data 2-10 times smaller by extracting, transforming and loading only some data from the data lake.

In a word, a retailer cannot live by real-time analytics alone. You can see some other important components of the scheme that fall out of real-time. Still, they are critical if the retailer wants to get valuable and deep insights. For example, a data analytics module, which we haven’t mentioned yet, is responsible for running complex analysis by applying elaborate algorithms and statistical models driven by data analysts. Indeed, this process can take hours or more, but the results are worth waiting. Correspondingly, the retailer’s analytical dashboards will always contain not only real-time but also historical data.

To sum it up

If thoroughly planned and properly implemented, real-time big data analytics definitely can become a competitive advantage. Taking into account how different the interpretations of real-time can be, it’s important to have a clear understanding of the company’s requirements to the analytical system.

In the article, we’ve described a typical architecture for real-time data analytics solution. Before taking it as an example, check whether it will cover your short-term and long-term business needs. If for some reasons, it does not, you may always turn for professional advice on how to tailor it.