Is Data Lake and Big Data the same?

Question

I am trying to understand all if there is a real difference between data lake and Big data if you check the concepts both are like a Big repository which saves the information until it becomes necessary, so, When can we say that we are using big data or data lake?

Nebulous

They're both buzzwords with nebulous technical meanings anyway. If the nuances matter more than marketing value, why not specifically describe what you're actually doing? – Maladapted

Hi, I have to present an architecture which be able to sync with multiple repositories, all these data should be store in a big repository, but I dunno if I am using the correct terms for that. – Nebulous

Answer 1

I can't say I've come across the term 'big repository' before, but to answer the original question, no, data lake and big data are not the same, although in fairness they are both thrown around a lot and the definitions vary depending who you ask, but I'll try to give it a shot:

---

Big Data

Is used to describe both the technology ecosystem around, and to some extent the industry that deals with, data that is in some way too big or too complex to be conveniently stored and/or processed by traditional means.

Sometimes this can be a matter of sheer data volume: Once you get into the 100s of terabytes or petabytes, your good old fashioned RDBMS databases tend to throw in the towel, and we are forced to spread our data across many disks, not just one large one. And at those volumes we'll want to parallellize our workloads, leading to things like MPP databases, the Hadoop ecosystem, and DAG-based processing.

However, volume alone does not tell the whole story. A popular definition of Big Data is described by the so-called '4 Vs': Volume, Variety, Velocity, and Veracity. In a nutshell:

• Volume - as mentioned above, refers to the difficulty caused by the size of the data

• Variety - refers to the inherent complexity of dealing with disparate types of data; some of your data will be structured (think SQL data tables), while other data might be either semi-structured (XML documents) or unstructured (raw image files), and the technology to deal with this variety is nontrivial

• Velocity - refers to the velocity with which new data may be generated; when collecting real time events like IoT data, or web traffic, or financial transactions, or database changes, or anything else that happens in real time, the 'velocity' of data flowing into (and in many cases, out of) your systems, can easily exceed the capabilities of traditional database technologies, necessitating some sort of scalable message bus (Kafka) and possibly a Complex Event Processing framework (such as Spark Streaming or Apache Flink)

• Veracity - the final 'V', refers to the added complexity of dealing with data which often comes from sources outside of your control, and which may contain data which is invalid, erroneous, malicious, malformed, or all of the above. This adds a need for data validation, data quality checking, data normalization, and more.

In this definition, 'big data' is data which, due to the particular challenges associated with the 4 V's, is unfit for processing with traditional database technologies; while 'big data tools' are tools which are specifically designed to deal with those challenges.

---

Data Lake

In contrast, Data Lake is generally used as a term to describe a certain type of file or blob storage layer that allows storage of practically unlimited amounts of structured and unstructured data as needed in a big data architecture.

Some companies will use the term 'Data Lake' to mean not just the storage layer, but also all the associated tools, from ingestion, ETL, wrangling, machine learning, analytics, all the way to datawarehouse stacks and possibly even BI and visualization tools. As a big data architect however, I find that use of the term confusing and prefer to talk about the data lake and the tooling around it as separate components with separate capabilities and responsibilities. As such, the responsibility of the Data Lake is to be the central, high-durability store for any type of data that you might want to store at rest.

By most accounts, the term 'data lake' was coined by James Dixon, Founder and CTO of Pentaho, who describes it thus:

“If you think of a datamart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.”

Amazon Web Services defines it on their page 'What Is A Data Lake':

A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.

From Wikipedia:

A data lake is a system or repository of data stored in its natural format, usually object blobs or files. A data lake is usually a single store of all enterprise data including raw copies of source system data and transformed data used for tasks such as reporting, visualization, analytics and machine learning.

And finally Gartner:

A data lake is a collection of storage instances of various data assets additional to the originating data sources. These assets are stored in a near-exact, or even exact, copy of the source format. The purpose of a data lake is to present an unrefined view of data to only the most highly skilled analysts, to help them explore their data refinement and analysis techniques independent of any of the system-of-record compromises that may exist in a traditional analytic data store (such as a data mart or data warehouse).

On on-premises clusters, the data lake usually refers to the main storage on the cluster, in the distributed file system, usually HDFS, though other file systems exist, such as GFS used at Google or the MapR File system on MapR clusters.

In the cloud, data lakes are generally not stored on clusters, since it's just not cost effective to keep a cluster running at all times, but rather on durable cloud storage, such as Amazon S3, Azure ADLS, or Google Cloud Storage. Compute clusters can then be launched on demand and connect seamlessly to the cloud storage to run transformations, machine learning, analytical jobs, etc.

Interpol

Just to sum, usually Data lake is like as big repository that many people use for a lot of things and Big Data is an complete ecosystem, am I right ? – Nebulous

user3342209: that sums it up pretty well, yes – Interpol

Answer 2

Big Data is just a term to encapsulate the massive amounts of data that is now being generated. It doesn't refer to anything specific or any specific amount of data.

Data Lake to me = Schema on Read. Data that is unstructured and dumped to object storage or similar without an associated schema.

Rattail

That may be so, but what about the terms Business Data Lake and Raw Data Lake? Combining data is often the clue for added value, your def is correct but how will that help? The wiki def above is better imho. – Chalk

Those terms are not always used -- I've certainly heard them used in an enterprise context, but 'raw data lake' just sounds like 'data swamp under construction' to me, while 'business data lake' sounds a lot like "exactly the same data and structure as in the old datawarehouse, but now it looks like we're cool with big data, let's all go to a conference and tell everyone how we're 'disrupting with AI and blockchain'" – Interpol

Answer 3

Big Data and Data Lake are two interrelated terms but have completely different meaning and this is the main reason why people often get confused between the two terms. So let's have a brief understanding of the differences between the two.

BIG DATA As the name itself says it all, Big Data is simply the data that is humongous in size. The data that is in the order of petabytes and more is considered as Big Data. Not only the size, but there are a few more parameters that define Big Data. The sources that are generating this data, the different formats of it, and the speed with which it is generated, all these factors when combined define Big Data. Big Data in the simplest of words is huge amounts of DATA. That's it.

DATA LAKE A data lake is a repository for Big Data. It stores data of all types i.e. structured, unstructured, and semi-structured, that has been generated from different sources. It stores data in its rawest form. A data lake is different from the data warehouse. Data warehouses store data in a well-structured form. Data present in a data lake may or may not be utilized in the future but the data in a data warehouse is meant for utilization since all the irrelevant has already been disposed of.

Big Data is huge data and data lake is the storehouse for it.

Endosteum

Answer 4

Big data is a field that deals with methods that analyze, extract information from it systematically, or otherwise deal with large or complex data sets that cannot be handled by traditional data processing applications.

Data lake is a data system or repository stored in a natural/raw format, usually an object blob or file. A data lake is usually a single data store, including original copies of source system data, sensor data, social data, etc. as well as transformed data used for tasks such as reporting, visualization, advanced analysis, and machine learning.

Tuck

Answer 5

big data refers to the large and diverse sets of data, while data lakes are the storage infrastructure designed to hold this data in its raw form for later analysis and processing. Data lakes play a crucial role in big data analytics by providing a scalable and flexible storage solution for vast amounts of information

Peddling

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Warfourd