Internship Project.1
Internship Project.1
We have seen Machine Learning as a buzzword for the past few years, the reason for this might
be the high amount of data production by applications, the increase of computation power in the
past few years and the development of better algorithms.Machine Learning is used anywhere from
automating mundane tasks to offering intelligent insights, industries in every sector try to benefit
from it. You may already be using a device that utilizes it. For example, a wearable fitness tracker
like Fitbit, or an intelligent home assistant like Google Home. But there are much more examples
of ML in use.
Machine learning (ML) is a category of an algorithm that allows software applications to become
more accurate in predicting outcomes without being explicitly programmed. The basic premise of
machine learning is to build algorithms that can receive input data and use statistical analysis to
predict an output while updating outputs as new data becomes available.
1
2. Machine Learning – What Today’s AI Can Do?
When you tag a face in a Facebook photo, it is AI that is running behind the scenes and
identifying faces in a picture. Face tagging is now omnipresent in several applications that
display pictures with human faces. Why just human faces? There are several applications that
detect objects such as cats, dogs, bottles, cars, etc. We have autonomous cars running on our
roads that detect objects in real time to steer the car. When you travel, you use Google Directions
to learn the real-time traffic situations and follow the best path suggested by Google at that point
of time. This is yet another implementation of object detection technique in real time.
Let us consider the example of Google Translate application that we typically use while visiting
foreign countries. Google’s online translator app on your mobile helps you communicate with
the local people speaking a language that is foreign to you.
There are several applications of AI that we use practically today. In fact, each one of us use AI
in many parts of our lives, even without our knowledge. Today’s AI can perform extremely
complex jobs with a great accuracy and speed. Let us discuss an example of complex task to
understand what capabilities are expected in an AI application that you would be developing
today for your clients.
Example:
We all use Google Directions during our trip anywhere in the city for a daily commute or even
for inter-city travels. Google Directions application suggests the fastest path to our destination at
that time instance. When we follow this path, we have observed that Google is almost 100%
right in its suggestions and we save our valuable time on the trip.
You can imagine the complexity involved in developing this kind of application considering that
there are multiple paths to your destination and the application has to judge the traffic situation in
every possible path to give you a travel time estimate for each such path. Besides, consider the
fact that Google Directions covers the entire globe. Undoubtedly, lots of AI and Machine
Learning techniques are in-use under the hoods of such applications.
Considering the continuous demand for the development of such applications, you will now
appreciate why there is a sudden demand for IT professionals with AI skills.
2
3. Machine Learning – Traditional AI
The journey of AI began in the 1950's when the computing power was a fraction of what it is
today. AI started out with the predictions made by the machine in a fashion a statistician does
predictions using his calculator. Thus, the initial entire AI development was based mainly on
statistical techniques.
In this chapter, let us discuss in detail what these statistical techniques are.
Statistical Techniques
The development of today’s AI applications started with using the age-old traditional statistical
techniques. You must have used straight-line interpolation in schools to predict a future value.
There are several other such statistical techniques which are successfully applied in developing
so-called AI programs. We say “so-called” because the AI programs that we have today are
much more complex and use techniques far beyond the statistical techniques used by the early AI
programs.
Some of the examples of statistical techniques that are used for developing AI applications in
those days and are still in practice are listed here:
Regression
Classification
Clustering
Probability Theories
Decision Trees
Here we have listed only some primary techniques that are enough to get you started on AI
without scaring you of the vastness that AI demands. If you are developing AI applications based
on limited data, you would be using these statistical techniques.
3
However, today the data is abundant. To analyze the kind of huge data that we possess statistical
techniques are of not much help as they have some limitations of their own. More advanced
methods such as deep learning are hence developed to solve many complex problems.
As we move ahead in this tutorial, we will understand what Machine Learning is and how it is
used for developing such complex AI applications.
4
4 . Machine Learning – Categories of Machine Learning
Machine learning evolved from left to right as shown in the above diagram.
Initially, researchers started out with Supervised Learning. This is the case of housing
price prediction discussed earlier.
This was followed by unsupervised learning, where the machine is made to learn on its
own without any supervision.
Scientists discovered further that it may be a good idea to reward the machine when it
does the job the expected way and there came the Reinforcement Learning.
Very soon, the data that is available these days has become so humongous that the
conventional techniques developed so far failed to analyze the big data and provide us the
predictions.
Thus, came the deep learning where the human brain is simulated in the Artificial Neural
Networks (ANN) created in our binary computers.
The machine now learns on its own using the high computing power and huge memory
resources that are available today.
It is now observed that Deep Learning has solved many of the previously unsolvable
problems.
5
Let us now study each of these categories in more detail.
6
Supervised Learning is where the AI really began its journey. This technique was applied
successfully in several cases. You have used this model while doing the hand-written recognition
on your machine. Several algorithms have been developed for supervised learning. You will
learn about them in the following chapters.
7
4.2 Unsupervised Learning
In unsupervised learning, we do not specify a target variable to the machine, rather we ask
machine “What can you tell me about X?”. More specifically, we may ask questions such as
given a huge data set X, “What are the five best groups we can make out of X?” or “What
features occur together most frequently in X?”. To arrive at the answers to such questions, you
can understand that the number of data points that the machine would require to deduce a
strategy would be very large. In case of supervised learning, the machine can be trained with
even about few thousands of data points. However, in case of unsupervised learning, the number
of data points that is reasonably accepted for learning starts in a few millions. These days, the
data is generally abundantly available. The data ideally requires curating. However, the amount
of data that is continuously flowing in a social area network, in most cases data curation is an
impossible task.
The following figure shows the boundary between the yellow and red dots as determined by
unsupervised machine learning. You can see it clearly that the machine would be able to
determine the class of each of the black dots with a fairly good accuracy.
The unsupervised learning has shown a great success in many modern AI applications, such as
face detection, object detection, and so on.
8
4.3 Reinforcement Learning
Consider training a pet dog, we train our pet to bring a ball to us. We throw the ball at a certain
distance and ask the dog to fetch it back to us. Every time the dog does this right, we reward the
dog. Slowly, the dog learns that doing the job rightly gives him a reward and then the dog starts
doing the job right way every time in future. Exactly, this concept is applied in “Reinforcement”
type of learning. The technique was initially developed for machines to play games. The machine
is given an algorithm to analyze all possible moves at each stage of the game. The machine may
select one of the moves at random. If the move is right, the machine is rewarded, otherwise it
may be penalized. Slowly, the machine will start differentiating between right and wrong moves
and after several iterations would learn to solve the game puzzle with a better accuracy. The
accuracy of winning the game would improve as the machine plays more and more games.
This technique of machine learning differs from the supervised learning in that you need not
supply the labelled input/output pairs. The focus is on finding the balance between exploring the
new solutions versus exploiting the learned solutions.
9
4.4 Deep Learning
The deep learning is a model based on Artificial Neural Networks (ANN), more specifically
Convolutional Neural Networks (CNN)s. There are several architectures used in deep learning
such as deep neural networks, deep belief networks, recurrent neural networks, and
convolutional neural networks.
These networks have been successfully applied in solving the problems of computer vision,
speech recognition, natural language processing, bioinformatics, drug design, medical image
analysis, and games. There are several other fields in which deep learning is proactively applied.
The deep learning requires huge processing power and humongous data, which is generally
easily available these days.
We will talk about deep learning more in detail in the coming chapters.
10
5. Machine Learning – Supervised Learning
Supervised learning is one of the important models of learning involved in training machines.
This chapter talks in detail about the same.
There are several algorithms available for supervised learning. Some of the widely used
algorithms of supervised learning are as shown below:
k-Nearest Neighbours
Decision Trees
Naive Bayes
Logistic Regression
Support Vector Machines
As we move ahead in this chapter, let us discuss in detail about each of the algorithms.
The diagram shows three types of objects, marked in red, blue and green colors. When you run
the kNN classifier on the above dataset, the boundaries for each type of object will be marked as
shown below:
11
Now, consider a new unknown object that you want to classify as red, green or blue. This is
depicted in the figure below:
As you see it visually, the unknown data point belongs to a class of blue objects. Mathematically,
this can be concluded by measuring the distance of this unknown point with every other point in
the data set. When you do so, you will know that most of its neighbours are of blue color. The
12
average distance to red and green objects would be definitely more than the average distance to
blue objects. Thus, this unknown object can be classified as belonging to blue class.
The kNN algorithm can also be used for regression problems. The kNN algorithm is available as
ready-to-use in most of the ML libraries.
You would write a code to classify your input data based on this flowchart. The flowchart is self-
explanatory and trivial. In this scenario, you are trying to classify an incoming email to decide
when to read it.
13
In reality, the decision trees can be large and complex. There are several algorithms available to
create and traverse these trees. As a Machine Learning enthusiast, you need to understand and
master these techniques of creating and traversing decision trees.
From the diagram, we can visually inspect the separation of red dots from green dots. You may
draw a boundary line to separate out these dots. Now, to classify a new data point, you will just
need to determine on which side of the line the point lies.
14
5.1.5 Support Vector Machines
Look at the following distribution of data. Here the three classes of data cannot be linearly
separated. The boundary curves are non-linear. In such a case, finding the equation of the curve
becomes a complex job.
The Support Vector Machines (SVM) comes handy in determining the separation boundaries in
such situations.
15
6. Machine Learning – Unsupervised Learning
So far what you have seen is making the machine learn to find out the solution to our target. In
regression, we train the machine to predict a future value. In classification, we train the machine
to classify an unknown object in one of the categories defined by us. In short, we have been
training machines so that it can predict Y for our data X. Given a huge data set and not
estimating the categories, it would be difficult for us to train the machine using supervised
learning. What if the machine can look up and analyze the big data running into several
Gigabytes and Terabytes and tell us that this data contains so many distinct categories?
As an example, consider the voter’s data. By considering some inputs from each voter (these are
called features in AI terminology), let the machine predict that there are so many voters who
would vote for X political party and so many would vote for Y, and so on. Thus, in general, we
are asking the machine given a huge set of data points X, “What can you tell me about X?”. Or it
may be a question like “What are the five best groups we can make out of X?”. Or it could be
even like “What three features occur together most frequently in X?”.
Let us now discuss one of the widely used algorithms for classification in unsupervised machine
learning.
The 2000 and 2004 Presidential elections in the United States were close — very close. The
largest percentage of the popular vote that any candidate received was 50.7% and the lowest was
47.9%. If a percentage of the voters were to have switched sides, the outcome of the election
would have been different. There are small groups of voters who, when properly appealed to,
will switch sides. These groups may not be huge, but with such close races, they may be big
16
enough to change the outcome of the election. How do you find these groups of people? How do
you appeal to them with a limited budget? The answer is clustering.
First, you collect information on people either with or without their consent: any sort of
information that might give some clue about what is important to them and what will
influence how they vote.
Then you put this information into some sort of clustering algorithm.
Next, for each cluster (it would be smart to choose the largest one first) you craft a message that
will appeal to these voters.
Finally, you deliver the campaign and measure to see if it’s working.
Clustering is a type of unsupervised learning that automatically forms clusters of similar
things. It is like automatic classification. You can cluster almost anything, and the more
similar the items are in the cluster, the better the clusters are. In this chapter, we are going
to study one type of clustering algorithm called k-means. It is called k-means because it
finds ‘k’ unique clusters, and the center of each cluster is the mean of the values in that
cluster.
Clustering is sometimes called unsupervised classification because it produces the same result as
classification does but without having predefined classes. Now, we are comfortable with both
supervised and unsupervised learning.
17
7.Machine Learning – Skills for Machine Learning
Advanced analytics techniques, big data infrastructure and powerful algorithms are providing
organizations with the ability to utilize their data for significant business value. The challenge,
however, is finding the right skill sets and talent to make better use of the data.
The growth of data science requires a deeper set of skills and Capabilities from data science
practitioners. While the fundamentals of data science have been around for decades, only
recently have the tools and techniques matured to provide the capabilities necessary to
accomplish more advanced data analytics, AI and machine learning (ML) goals.
As such, there are many skills for machine learning and AI that data scientists, IT professionals
and ML engineers should hone before embarking on AI and ML projects. Some of these skills
are general math and statistics skills, while others are more specific to the technology and
platforms being used.
18
Data scientists require a firm grasp on concepts such as the following:
p-value;
19
focus on predictive applications achieve their goals by minimizing an objective function and
learning the weights applied to the testing data to obtain final predictions.
Other optimizations include the needs for cost and error functions, methods for rapidly
determining values from big data and iterating for better performance and accuracy. Key areas
data scientists should have knowledge in include cost function and objective functions,
likelihood and error functions, gradient descent algorithms and their variants.
Data scientists should have experience with a variety of data plotting and charting approaches
including the following:
Histograms;
Bar and area charts, pie and line charts, waterfall charts, thermometer and candlestick
charts;
20
Scatter plots and bubble charts;
One of the requirements for data wrangling is data access and collection. Data scientists should
have some experience in big data access through state-of-the-art and widely accepted data
platforms including Hadoop and Spark, as well as more traditional data access approaches
including SQL and NoSQL methods. They should also have experience with common databases
such as MongoDB and Postgres. In addition, data scientists should be familiar with manipulating
the data with data selection, data extraction and methods by which huge data sets are filtered to
the relevant portions.
Equally as important as data selection is dealing with questionable quality data. Collecting data
from multiple data sources can lead to many issues including missing, incorrect, conflicting or
potentially biased data. With data programming and collection approaches, data wrangling and
21
preparation requires addressing all of these forms of data imperfections and applying
transformations, manipulations, formatting changes and augmentation to improve overall data
quality.
Data scientists will need to know how to identify missing or erroneous data, methods of data
imputation, approaches for augmenting or enhancing data sets, methods for data transformation
and multiplication as needed, identification and treatment for outliers, correction of data types,
data scaling and normalization, detection of potential bias in data, data deduplication and data
anonymization.
These models are generally relevant to the line of business and to organizational needs ranging
from recommending products to customers to projecting sales and inventory, from understanding
trends in customer or patient data to classifying a wide range of data into categories.
Data analysis is done using a wide range of tools including Excel, big data analysis tools like
Hadoop and Spark, commercial analytics offerings such as SAS and MATLAB and open source
offerings using R, Python, Java, Julia and other languages. Knowledge of these tools and using
them to achieve data analysis objectives is incredibly important to a data scientist's success.
22
7.7 ML algorithms, modeling and feature engineering
ML has become the most visible aspect of the modern data scientist's job as it requires them to
build models from data using their skills for machine learning methods and algorithms. Data
scientists need to understand the vast range of ML algorithms including:
Bayesian Methods;
K-Nearest Neighbors;
Ensemble Methods;
Data scientists must ensure they are up to speed with the innovation taking place in ML
algorithms.
Data scientists aiming to have skills for machine learning will also need to know how to use
third-party models for their own needs to shorten overall model development time. This means
having an understanding of transfer learning and how to enhance models.
23
7.8 Programming skills
Much of the technology around data science has evolved over the past few decades. Open source
as well as commercial offerings provide a plethora of tools, libraries, frameworks and support
functionality across the full lifecycle of the data scientist's job responsibilities.
Data scientists need proficiency in a range of languages including Python, R, Julia and Java-
based languages. Python in particular has been the star of the data science world. In 2018, 66%
of data scientists reported using Python every day, overtaking R as the most popular language for
data science. Julia and other languages are helpful for high-speed and big data processing, and
even the use of commercial offerings from SAS and MATLAB are helpful to accomplish a range
of data science and analytics tasks, particularly in enterprise settings where the ability to scale up
projects is important.
24
7.11 Data engineering and manipulation tools
To accomplish many of those data preparation and manipulation tasks, data scientists should
have experience using tools for big data manipulation, including open source offerings such as
Pandas, as well as those provided by commercial or cloud-based providers. Data scientists
should also have experience working with unstructured data such as images, videos, emails and
documents that come from different channels and sources.
25
8. Machine Learning Applications
Let’s explore other real-world machine learning applications that are sweeping the world! I’ll be
discussing the following Applications of Machine Learning one by one:
Product Recommendations
Image Recognition
Sentiment Analysis
Banking Domain
Language Translation
Social media platforms use machine learning algorithms and approaches to create some attractive
and excellent features. For instance, Facebook notices and records your activities, your chats,
likes, and comments, and the time you spend on specific kinds of posts. Machine learning learns
from your own experience and makes friends and page suggestions for your profile.
26
2. Product Recommendations
One of the most popular and known applications of machine learning is Product
Recommendation. Product recommendation is one of the stark features of almost every e-
commerce website today, which is an advanced application of machine learning techniques.
Using machine learning and AI, websites track your behavior based on your previous purchase,
your searching pattern, your cart history, and make product recommendations.
27
3. Image Recognition
Image Recognition is one of the most significant and notable Machine Learning and AI
techniques: an approach for cataloging and detecting a feature or an object in the digital image.
This technique is being adopted for further analysis, such as pattern recognition, face detection,
or face recognition.
4. Sentiment Analysis
Sentiment analysis is a real-time machine learning application that determines the emotion or
opinion of the speaker or the writer. For instance, if someone has written a review or email (or
any form of a document), a sentiment analyzer will instantly find out the actual thought and tone
of the text. This sentiment analysis application can are used to analyze a review based website,
decision-making applications, etc.
28
5. Automating Employee Access Control
Organizations are actively implementing machine learning algorithms to determine the level of
access employees would need in various areas, depending on their job profiles.
Machine learning algorithms are used to develop behavior models for endangered cetaceans and
other marine species, helping scientists regulate and monitor their populations.
Significant healthcare sectors are actively looking at using Machine Learning algorithms to
manage better. They predict the waiting times of patients in the emergency waiting rooms across
various departments of hospitals. The models use vital factors that help define the algorithm,
details of staff at various times of day, records of patients, and complete logs of department chats
and the layout of emergency rooms. Machine learning algorithms also come to play when
detecting a disease, therapy planning, and prediction of the disease situation.
29
8. Predict Potential Heart Failure
An algorithm designed to scan a doctor’s free-form e-notes and identify patterns in a patient’s
cardiovascular history is making waves in medicine. Instead of a physician digging through
multiple health records to arrive at a sound diagnosis, redundancy is now reduced with
computers making an analysis based on available information.
9. Banking Domain
Banks are now using the latest advanced technology machine learning has to offer to help
prevent fraud & protect accounts from hackers. The algorithms determine what factors to
consider to create a filter to keep harm at bay. Various sites that are unauthentic will be
automatically filtered out and restricted from initiating transactions.
Machine learning plays a significant role in the translation of one language to another. We are
amazed at how the websites can translate from one language to another effortlessly and gives
contextual meaning as well. The technology behind the translation tool is called ‘machine
translation.’ It has enabled the world to interact with people from all corners of the world;
without it, life would not be as easy as it is now. It has provided a sort of confidence to travelers
and business associates to safely venture into foreign lands with the conviction that language will
no longer be a barrier.
30
9. Advantages and Disadvantages of Machine Learning
3. Continuous Improvement
5. Wide Applications
1. Data Acquisition
3. Interpretation of Results
4. High error-susceptibility
31
10. Future Scope and Conclusion
10.2 Conclusion
Getting to know some of the popular applications of machine learning along with technology
evolving at a rapid pace, we are excited about the possibilities which the Machine Learning
Course has to offer in the days to come. And as the demand for AI and machine learning has
increased, organizations require professionals with in-and-out knowledge of these growing
technologies and hands-on experience.
32