Traditional data architectures are not able to deal with these data sets. Actually, the term Big Data seems to imply that other data is somehow small (it isn’t) or that the key issue to deal with it is the massive size. However, the characteristics of Big Data that require new architectures go beyond just volume (the size of the digital universe, i.E., all digital data created in the world, is estimated to be around 40 zettabytes – 40 trillion gigabytes – in 2020):
Variety (i.E., data from multiple repositories, domains, or types). Structured data is that which can be organized neatly within the columns of a database. This type of data is relatively easy to enter, store, query, and analyze. Unstructured data is more difficult to sort and extract value from. Examples of unstructured data include emails, social media posts, word-processing documents; audio, video and photo files; and web pages.
Velocity (i.E., rate of flow). Every second, Google receives almost 100,000 searches; the same amount of YouTube videos is watched; more than 1000 photos are uploaded to Instagram, almost 10,000 tweets are posted, and more than 3 million emails sent (Source).
Variability (i.E., the change in other characteristics). Data’s meaning is constantly changing. For example, language processing by computers is exceedingly difficult because words often have several meanings. Data scientists must account for this variability by creating sophisticated programs that understand context and meaning.
These characteristics – volume, variety, velocity, and variability – are known colloquially as the Four Vs of Big Data.
In addition, Big-Data practitioners are proposing additional Vs such as:
Veracity (i.E., the quality of data). If source data is not correct, analyses will be worthless.
Visualization (i.E., the meaning of the data). Data must be understandable to nontechnical stakeholders and decision makers. Visualization is the creation of complex graphs that tell the data scientist’s story, transforming the data into information, information into insight, insight into knowledge, and knowledge into advantage. A great example of this are the various graphics that often accompany stories on climate change: They are pictures that are worth more than a billion data points:
Value (i.E., opportunities and savings). Ultimately, the entire point of Big Data is to improve decision-making by organizations.
Big Data Definitions
Several definitions of Big Data have been proposed, including ‘extremely large data sets’; ‘extensive datasets require a scalable architecture for efficient storage, manipulation, and analysis’; and the exponential increase and availability of data in our world’.
The term Big Data describes the massive amounts of data being collected in today’s networked, digitized, sensor-populated, and information-driven world and the tools used to analyze and extract information from these large and complex data sets.
The growth rates for data volumes, speeds, and complexity of Big Data overwhelms conventional data processing software. That’s why improved, and entirely new analytical techniques and processes are in the process of being developed and continuously refined. This includes the areas of data capture, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy, and data source.
In this context, the term Big Data Analytics describes the process of applying serious computing power – the latest in machine learning and artificial intelligence – to massive and highly complex sets of information.
One important concept to Big Data is metadata, which is often described as ‘data that describes other data’, for instance how and when data was collected, how it has been processed, or how it is linked to other data.
Although developed in the early days of computers, it still models many concepts used in data science and machine learning.
(Source: Mind Map created by G. Wagenmaker)
Data usually is just a collection of raw facts, often collected from various sources and in multiple formats, that are quite useless unless they are analyzed and organized. For example, images and videos can hold a lot of data that requires interpretation to extract information from them.
Information is gained from data by consistently organizing, structuring and contextualizing raw data according to users’ requirements. This makes information more valuable than raw data.
One key aspect of knowledge is the application of information to answer a question or solve a problem. Combined with past experience and insights, know-how, and skills, contextualized information is key to gaining knowledge. Knowledge is the most valuable distillation of data, and although knowledge gives you the means to solve a problem, it doesn’t necessarily show you the best way to do so.
The ability to pick the best way to reach the desired outcome comes from experience gained in earlier attempts to reach a successful solution. The DIKW model describes this ability as wisdom. People gain wisdom through experience and knowledge, some of which comes from developing an understanding of problem-solving methods and gathering intelligence from other people solving the same problems.
Examples of Big Data Analytics
There are many areas and industries where Big Data Analytics already play a role and the examples are too numerous to be covered here. So we just showcase a few to give you an idea of what kind of impact Big Data already has.
Big Data and Climate Simulations
One of the most data-intensive scientific disciplines involves planetary climate simulations. As scientists are refining these models in order to describe the intricacies of the Earth’s climate system as detailed and precise as possible, the amount and complexity of the associated data is growing exponentially.
Climate models, also known as earth system models, work by representing the physics, chemistry, and biology of the climate system as mathematical equations.
Most earth system models run on supercomputers, but they require even more computing power scientists have available. This limits the size of the cells in their 3D grid (see image above). In current models, a cell typically measures 80-100 km on each side, with one value per cell representing a single variable like temperature, cloud cover, or rainfall.
The figure below shows the projected increase in global climate data holdings for climate models, remotely sensed data, and in situ instrumental/proxy data.
(Source: 10.1126/science.1197869)
Climate models are based on well-documented physical processes to simulate the transfer of energy and materials through the climate system. These models use mathematical equations to characterize how energy and matter interact in different parts of the ocean, atmosphere, and land.
Building and running a climate model is complex process of identifying and quantifying Earth system processes, representing them with mathematical equations, setting variables to represent initial conditions and subsequent changes in climate forcing, and repeatedly solving the equations using powerful supercomputers.
Framework of big data in climate change studies. (click on image to enlarge) (Source doi:10.3390/bdcc3010012)
Big Data and Transportation
Big data and the IoT work in conjunction. Huge amounts of unstructured data extracted from the sensors embedded in IoT devices provide the basis for sophisticated DIKW-type problem-solving and decision-making processes in order to improve products and services across many industries. Example
Citizens also benefit from open data through real-time access to traffic information so that they can better plan their journeys and avoid congestion. Real-time navigation alerts drivers to delays and helps them choose the fastest route. Smart parking apps point them directly to available spots, eliminating time spent fruitlessly circling city blocks. For instance, by optimizing emergency call dispatching and synchronizing traffic lights for emergency vehicles, cities can cut emergency response times by 20–35 percent.
A key challenge of smart cities is the need to process extremely large amounts of complex and geographically distributed sources of data (citizens, traffic, vehicles, city infrastructures, IoT devices, etc.), combined with the additional need to deal with this information in real time.
These systems require new approaches to Big Data management. For instance, the European CLASS project developed a novel software architecture framework to design, deploy and execute distributed big data analytics with real-time constraints for smart cities, connected cars and future autonomous vehicles.
Aircraft Safety and Maintenance. d. For instance, the latest Airbus A350 has 50,000 sensors on board collecting 2.5 terabytes of data every day it operates. Engine data is amongst the most complex and the thousands of sensors in each modern aircraft engine feeds data into AI-embedded maintenance and engineering systems that allow operators to act and solve problems immediately.
These systems are able to harvest data from aircraft operations automatically and then update maintenance programs.
Big Data in the Financial Services Sector
Financial services have always been a data intensive industry, from the vast amount of credit card transactions to credit scoring and fraud detection. To give you an idea of the scope, global credit cards generated about 441 billion purchase transactions in 2019.
The main areas where financial services companies apply Big Data is in:
Risk management: Analysis of in-house credit card data freely accessible for banks enables credit scoring and credit granting which form part of the most popular tools for risk management and investment evaluation.
These help institutions get a better understanding of their customers, predict customer behavior, accurately target potential customers and further improve customer satisfaction with a strategic service design.
Big Data and Materials Science
Materials innovation is the key to the most pressing challenges from global climate change to future energy sources. However, trial-and-error and the lack of systematic data have significantly hampered breakthrough discovery in materials research.
In an effort to overcome this, in 2011 the U.S. Unexpected variability in shape can have detrimental effects in the nanoparticle behavior and their functional properties. This represents a tremendous challenge because the selection of experimentally significant samples becomes increasingly difficult and requires knowledge of the relevant sizes, shapes, and structural complexity a priori.
Big Data combined with data mining and statistical methods can tackle this problem.
For instance, researchers at Osaka University employed machine learning to design new polymers for use in photovoltaic devices. More birds live in this biodiversity hotspot in than anywhere else on the African continent—a veritable teeming feathered metropolis.
“We want to understand how species—in this case birds—coexist without driving each other to extinction,” Ayebare said. “To protect a species, you must first understand where they are and why.”
Past methods to understand how animals, birds or insects used space relied on experiments in laboratories or on small plots of land. Create a desirable space, then see what creature comes or stays.
Scientists strategically selected points of land across huge elevation and environmental gradients and recorded all birds seen or heard over a fixed period of time. That led to the identification of over 6,000 individuals across 129 species.
“We’re interested in the circumstances that allow biodiversity to flourish—what makes species co-existence possible?” Zipkin said. “There’s is a lot of pressure on biodiversity in the modern age.
Among their findings were that birds partition their habitat use along environmental gradients: temperature, precipitation, and forest vegetation types. The data revealed a sense of the different strategies the birds adopt to survive.
“Species have organized themselves over millions of years,” Ayebare said. “We want to develop ways figure out what they will do next to survive.”
Translating big data into big insights demands tenacity, Zipkin said.
To successfully compete in today’s marketplace, small businesses need the tools larger companies use. .
Editor’s note: Looking for CRM software for your business? .
Small and midsize businesses (SMBs) face many of the same challenges as large enterprises. SAS’s easy-to-use analytics, automated forecasting and data mining enable businesses without a lot of resources to accomplish more with less. These analytics help companies overcome challenges to grow and compete. SAS’s message to SMBs is simple: “Identify what’s working and fix what isn’t. Make more intelligent decisions.” Contact SAS for details, pricing and to learn about its free software trials.
2. Alteryx
Analyzing complex business intelligence doesn’t have to be rocket science. Alteryx offers advanced data mining and analytics tools that present information in a simple, understandable way.
Alteryx combines your business’s internal data with publicly available information to help you make better business decisions. These insights allow you to create graphs, storylines and interactive visuals from the dashboard. Contact the company for pricing information.
3. Kissmetrics
Looking to increase your marketing ROI? Kissmetrics enables you to understand, segment and engage your customers based on their behavior.
Plans start at $300 per month.
InsightSquared
. InsightSquared offers a free trial, and its service plans are modular and scalable. Contact InsightSquared for pricing.
6. IBM Cognos Analytics
Contact IBM for pricing information.
It’s a turnkey program, meaning no installation or programming is necessary. All you need to do is log into your merchant portal. Contact Tranzlogic for pricing information.
8. Qualtrics
If you don’t currently have any rich sources for data, research may be the answer. Qualtrics lets you conduct a wide variety of studies and surveys to gain quality insights for data-driven decisions.
