Big Data Research

Everyday, we create 2.5 quintillion bytes of data–so much that 90% of the data in the world today has been created in the last two years alone. - IBM

Where Big Data Comes from...

Massive datasets that have outgrown the capabilities of relational databases can be considered Big Data. The growth of technology use per person and the increasing number of connected devices and analytics gathered creates a need for massive storage and the ability to manipulate it in a useful way.

According to IBM Big Data is comprised of Volume, Velocity, and Variety.  Volume being the fact there is a huge amount of data, Velocity, the speed at which we want the data (i.e. streaming, always available, instantaneous, point of sale), and Variety, like video, audio, images, apps, text etc.

NoSQL positions itself as a way to manage Big Data.  It does not rely on the relational model.  They tend to be superior in scalability and performance.  It does not require a fixed table schema with relationships.  Companies like Google use it over traditional databases because the MySQL structure could not keep up with the Volume or Velocity of data.

Hadoop by Apache is a software project that helps with Big Data Management.  It allows for the distributed processing of large data sets.  It is designed with the ability to scale from one computer to clusters of servers with built in software to detect hardware failure for maximum uptime.  Hive is an extension of Hadoop that provides data summary features, ad-hoc queries and analysis of large data sets. It has functions similar to that of Google MapReduce but can also work with data stored in SQL.


Big Data is not going away and solutions like NoSQL and Hadoop to help manage it will become increasingly important.

Business Process Management according to MIchael Hammer

Business process management is defined as "an integrated system for managing business performance by managing end-to-end business processes.  The figure below is a representation of the essential process management cycle.

The Essential Process Management Cycle

This cycle is derived from Deming's PDCA cycle (Plan Do Check Act) and represents a shift from how enterprises are typically managed.  It is based on the premise that the correct way to manage an organization is to control and have a well-defined end-to-end business process through which customer value is created. This is a departure from the typical management mindset of trial and error, pushing people harder, and financial manipulation.  Therefore, BPM is essentially a customer-centered approach to organizational management.  Customers care about one thing only, results.  The results that customers expect are the outputs of interrelated business processes. An organization must focus on customers, results, and business processes to be successful.

With BPM, an organization can enable high-performance processes that operate with lower costs and with greater efficiency, accuracy, and precision.  Integrating all of a companies process improvement goals under the integrated environment of BPM leverages a wide range of targeted tools to specific problems. With the process design insight that BPM provides it is possible to achieve goals that are apparently incompatible like reducing inventory while at the same time reducing out-of-stocks.

In order to effectively implement BPM on a sustained basis, five critical enablers are required:  Process Design, the specification of what tasks are to be performed and by whom, Process Metrics, the targets that will be measured, Process performers, the people who work in the process, Process Infrastructure, the supporters of the process like IT and HR, and finally Process Owners. the person(s) responsible for the end-to-end process.

Lean and the Toyota Way 2011

Toyota is the lean thinking pioneer.  A common misconception is that is a management philosophy when that couldn't be farther from the truth.  Lean thinking is a way of thinking that creates a culture of leadership, teaching, and continual endless improvement also known as kaizen.  The title of my blog post "Toyota Way 2011" even represents the kaizen way of thinking.  The Toyota Way is an internal booklet that describes the lean thinking. Toyota was very specific to add a date to the booklet so show that there is always room for improvement and no process is set in stone and that even the foundation of lean thinking will and should be subject to experimentation and improvement.

The lean thinking house is used by Toyota to summarize the Toyota Way.  It's six major sections are

1. Goal (roof)
2. Foundation
3. Pillar - Respect for People
4. Pillar - Continuous Improvement
5. 14 Principles
6. Lean Product Development 

I am going to briefly go over each section of the Toyota Way and provide an analysis along the way. I will seldom use the terms employee or business partners because Lean thinking is encouraged, fostered, and expected of all people that are stakeholders in the organization.

The Lean Thinking House

The Lean goal is to provide "Sustainable shortest lead time, best quality and value (to people and society), most customer delight, lowest cost, high morale, safety."  In a nutshell, the system goal of lean thinking is to deliver value to both the customer and society in shorter and shorter time increments while still achieving the highest quality possible and morale levels.  This is referred to as creating a flow of value to the customer without delay.  It is important to note that reducing cycle times does not mean cutting corners or reducing product quality.  Cycle times are reduced by never ending continuous improvement, that requires a culture of respect for people in which people feel they are safe to experiment and change the status quo. From the bottom up people are encouraged and expected to out-learn and out-improve the competition.

The foundation of Lean thinking focuses on Manager-Teachers.  Any new employee at Toyota goes through several months or training under one of their managers to learn the foundations of lean thinking and how to properly identify waste and spot improvements.  Their internal Motto is Good Thinking, Good Products. Toyota achieves this Good Thinking, and Good Products Culture with a culture of mentoring.  Managers are expected to be leaders, teachers, and masters of their domain of work.  Managers are meant to be less of a director and more of a mentor.  A Toyota saying is "my manager can do my job better than me".  The expectation of managers' ability to perform any 'apprentice's' job better relates to a core principle of Lean called Gemba.

Gemba is a a core principle that Lean thinking expects of Managers.  Gemba means that people are expected to "see with their own eyes" at the front line. The truth cannot be learned from reports or numbers and going to the front-line of value work spot waste or improvement possibilities.    

The first pillar of Lean thinking is Respect for People.  Lean thinking and Toyota promote a culture of respect and sensitivity to peoples morale.  A saying is "Develop People and Then Build Products", this goes back to managers acting as teachers rather than directors and teaching people to analyze the root cause of a problem relating again back to Gemba.  Managers are expected to "Walk the Talk" in their own goals of eliminating waste and continuous improvement.   People are to develop long lasting relationships with partners based on trust and to help each other improve and stay profitable.  Teams do team-work, not group work and are encouraged to decide how to improve their own teams and to implement them rather than relying on a set of standards for all teams.  This is an interesting contrast to many organizations way of thinking.  Economies of scale and standardization is often encouraged in many cultures but lean is almost the complete opposite focusing on short cycle times and people deciding for themselves what their own standards should be.

Pillar two of Lean is Continuous Improvement.  Continuous Improvement is based on several ideas: 

1. Go See
2. Kaizen
3. Perfection Challenge
4. Work Toward Flow

Go See is the principle that is described as critical and fundamental and is highlighted as the first factor of success in continuous improvement in the Toyota Way 2011. Go See relates back to the term Gemba.  All level managers should "frequently go to the place of real work and see and understand themselves". It literally means to go to the physical location of front-line work and to get as close to it as possible with the aim of understanding problems and opportunities to improve.

Kaizen is a mindset and a practice.  It suggests "My work is to do my work and to improve my work" and "continuously improve for its own sake."  Kaizen implies that people should choose and practice techniques that a team has agreed to try until they are masters.  Once a baseline of work has been created and mastered is is time to experiment until a better way can be found.  This process never ends.  It is important to note that standardized work is not a part of Kaizen, the idea of coming up with an initial standard of work is to have a baseline of comparison for future improvements.  People are told "they aren't earning their pay if they leave standardized work unchanged for a whole month."  Understanding gained through Kaizen is meant to be spread laterally so that other groups can learn from the knowledge gain and implement their new understanding if the team thinks it will benefit them.  There is always a Perfection Challenge with "high expectations to challenge oneself, team members, and partners to levels of skill, master, waste reduction, and vision beyond the status quo."

Work Toward Flow suggests optimizing a process that creates a flow without delay to the customer by reducing waste that the customer does not pay for and continuously improvement the process and reducing cycle times.

The next section of the Lean Thinking House is the 14 principles.  The 14 principles are: base management decisions on a long-term philosophy, move toward flow, use pull systems, level the work, stopping and fixing problems, master norms, use simple visual management, use well-tested technology, grow leaders from within, develop exceptional people, respect and help partners to improve,  go see for yourself at the real place of work, make decisions slowly by consensus, and relentless reflection and kaizen.

I am not going to go over each individual principle but a quote from J. Liker in the Toyota Way sums up the value of the principles. 

Many good American companies have respect for individuals, and practice kaizen and other [Toyota] tools.  But what is important is having all the elements together as a system.  It must be practiced every day in a very consistent manner.

The last major section of the Lean Thinking House is Lean Product Development. Lean thinkers execute two key processes well, product development and production.  Lean product developers strive to create useful knowledge and to learn better than their competition without wasting effort by placing an emphasis on high value learning.  Two types of high-value information include uncertain things and a focus on early testing and feedback.  Lean product development increases new knowledge, value of info, the amount of valuable feedback, valuable data, and reusable knowledge while at the same time reducing the cost of information.

Lean thinking is a broad subject matter and culture would take anyone years to fully understand.   As businesses learn from Toyota to focus less on the instant bottom-line and more on long term solutions with continuous improvement and a respect for people we will see a much more energized and responsive marketplace.