Programming – Harini Jayaraman

K-Means Clustering

harini-jayaraman — Fri, 11 Mar 2022 08:30:00 +0000

In my previous blog, I introduced Supervised and Unsupervised learning methods. We will learn about one of the unsupervised learning methods called Clustering in this blog. Clustering involves dividing a population or set of data points into groups so that the data points in the same group are more similar than those in other groups. The goal is to group similar traits into clusters and assign them to groups. There are several approaches to the task of clustering: Hierarchical clustering (agglomerative, divisive) – Define a set of nested clusters organized as a tree. Model-based clustering (mixture models) – A statistical modelling framework based on probability distributions. Partitional clustering (K-means, K-medoids): Division of the set of data points into non-overlapping subsets (the clusters) such that each data point is in exactly one subset. In this blog post, we will focus on K-means clustering. What is K-Means Clustering? The K-means clustering algorithm depends on two parameters, the number of clusters and Cluster centroids. For a particular choice of the number of clusters K, the objective is to minimize the squared distance of the points within each cluster from its center (Referred to as a centroid). The algorithm is initialized by randomly picking the centers of each cluster and then assigning each data point to the closest centroid. After all the data points have been assigned to clusters, the centroids are updated to be the mean of all the data points for each cluster. Having updated the centroid values, we can assign each data point to its closest updated centroid, and then recompute the centroid values. The process is repeated until the centroid positions stop changing between K clusters. For a more detailed understanding, please check this blog. How to determine what value to use for K? The number of clusters are unknown in most cases. In such cases, measures of internal validation like, Calinski – Harabasz index and Silhouette can be useful to select the number of clusters. Calinski – Harabasz index The K-means algorithm returns the clustering minimizing within the sum of squares (WSS). The WSS measures the variability “within”, that is the variability between the data points assigned to cluster K and the corresponding centroid. The between clusters sum of squares (BSS), measures the variability “between”, that is the variability between the clusters accounting for the center of the data. Because the total variation in the data, the total sum of squares (TSS) is fixed, minimization of WSS corresponds to the maximization of BSS. The Calinski-Harabasz index is given by: with CH = 0 when K = 1 The Calinski-Harabasz index compares the relative magnitudes of BSS and WSS, accounting for K and N.A large value of this index for a given value of K is an indication of a clustering solution with low within variability and large between cluster variability. Silhouette The silhouette is a measure of how close each point in one cluster is to points in the neighboring clusters. Silhouette analysis provides a way to visually assess the number of clusters. The silhouette takes values in the range [−1, 1]. Silhouette coefficients near 1 indicate that the observation is far away from the neighboring clusters. A value of 0 indicates that the observation is on or very close to the decision boundary between two neighboring clusters. Negative values indicate that the observation might have been assigned to the wrong cluster. For each observation in the data compute: , the average dissimilarity (distance) from observation xi to the other members of the same cluster k. The average dissimilarity from observation to the members of the other clusters. Let the minimum of such dissimilarity computed. Hence b(i) can be seen as the average dissimilarity between and the nearest cluster to which it does not belong. The silhouette for observation is: To understand the method better, I have performed a cluster analysis of the audio features data from Spotify using k-means. The dataset contains data about audio features for a collection of songs extracted from the music streaming platform Spotify with 231 observations and 13 variables. The songs are classified into three genres: acoustic, pop, rock. For each song, measurements about audio features and popularity are recorded; more information about the audio features is available here. I have used the Cluster package in R. Storing within the sum of squares and the between the sum of squares, with the output we can calculate and plot the Calinski-Harabasz index. We perform K-means on Spotify data for different values of K. Here we take the maximum value to be K = 10 The “appropriate” K corresponds to the largest value of CH. This shows that an optimal number of clusters is K = 2. The silhouette plot is used to assess cluster coherence. Observations with high silhouette values are much closer to their own cluster members than the members of other clusters. Observations with low (or even negative) silhouettes are close to other cluster members and thus may have ambiguous cluster membership. Using the silhouette function in the cluster package, we find the silhouette width by inputting two arguments, clustering data and distance matrix computed using Euclidean distance. The average Silhouette for K=2 is larger than the average Silhouette for K =3. Hence, Clustering K = 2 is better. By comparing the clustering obtained to the classification into genres, we could observe that the clustering is almost similar to the genre of the given data. We could assess the accuracy by measures of external validations like the RAND index and Adjusted RAND index. However, I have not covered that in this blog. You can have a look at that here. The Rand index has a value between 0 and 1, with 0 indicating that the two data clusterings do not agree on any pair of points and 1 indicating that the data clustering is exactly the same. Here we have a rand index of 0.72 for K = 2 and 0.75 for K = 3. Both these values are almost near 1 hence suggesting K-means clustering using audio features of data is similar to the real clustering of data according to its genres. The Adjusted Rand Index is used to measure the similarity of data points presented in the clusters i.e., how similar the instances that are present in the cluster. So, this measure should be high as possible else we can assume that the data points are randomly assigned in the clusters. Here ARI = 0.46 for K=2 and 0.451 for K=3, which is a relatively good measure. Summary To summarise, the K-means clustering algorithm: aims to group similar data points into clusters requires a choice for the number of clusters iterates between two steps, allocation and updating. Further Reading I found this blog useful for learning about the implementation of the K-means algorithm in R. To read about how to implement the K-means algorithm and other clustering methods in python you can click here. Also, to understand K-means clustering in a simpler way, this video can be helpful.

Source

Supervised Vs. Unsupervised Learning

harini-jayaraman — Sat, 05 Mar 2022 18:54:00 +0000

As the world is getting ‘smarter’ every day, most of the world is moving towards Machine learning and artificial intelligence to keep up with the expectations. Statistical machine learning encompasses automatic learning and data analysis procedures, which learn a task from a series of examples with the goal of identifying patterns and performing predictions under uncertainty. We will explore and understand the differences between the two main machine learning methods in this blog What is Supervised Learning? Supervised learning, as the name indicates, has the presence of a supervisor as a teacher. Basically, Supervised learning is when we teach or train the machine to use the data that is well defined and labelled. Just as a child, how we aren’t able to identify the fruits or vegetables right until we get to see, feel, and learn from the inputs, the same logic is applied here to the machines. In supervised learning, models need to find the mapping function to map the input variable (X) with the output variable (Y). The goal of supervised learning is to train the model so that it can predict the output when it is given new data. For example, let’s assume we have images of different types of fruits. The task of our supervised learning model is to identify the fruits and classify them accordingly. So to identify the image in supervised learning, we will give the input data as well as output for that, which means we will train the model by the shape, size, color, and taste of each fruit. Once the training is completed, we will test the model by giving the new set of fruit. The model will identify the fruit and predict the output using a suitable algorithm. Supervised learning can be separated into two types of problems, classification and regression. Classification problems use an algorithm to accurately assign test data into specific categories, such as separating apples from oranges, or disease or no disease etc. Regression is another type of supervised learning method that uses an algorithm to understand the relationship between dependent and independent variables. For example, study relationship between two variables like height and weight. By measuring and labeling inputs and outputs, the model can learn over time. What is Unsupervised Learning? In unsupervised learning, information that is neither classified nor labeled is used to train a machine, allowing the algorithm to act based on that information without guidance. The objective of the algorithm is to group unsorted data based on similarity, pattern, and difference, without any prior training. The machine does not receive any training, as opposed to supervised learning. Therefore, the machine is limited to finding the hidden structure in unlabeled data on its own. For instance, if we consider the above example as fruits, it will identify the patterns like long, round or curvy, and similarities and categorize them. If we have apples and bananas, we might have two different clusters of bananas and apples after running the algorithm. Unsupervised learning models are used for three main tasks: clustering, association and dimensionality reduction: Clustering is a data mining technique for grouping unlabeled data based on their similarities or differences. For instance, Can we try to find groups of patients with similar symptoms and pain characteristics? Association is another type of unsupervised learning method that uses different rules to find relationships between variables in a given dataset. For example, a person buying a product X also tend to buy product Y. Dimensionality reduction is a learning technique used when the number of features (or dimensions) in a given dataset is too high. Summary Supervised Learning Unsupervised Learning Goals Predict outcome of New data Insight into a large volume of data Complexity Simple Computationally complex Accuracy Better accuracy Less accurate We can choose our methods depending on our input data (labelled or unlabelled) and by defining our goals. Among other things, supervised learning models are ideal for spam detection, sentiment analysis, weather forecasting, and price prediction. Unsupervised learning is better suited for anomaly detection, recommendation engines, customer persona, and medical imaging. Further reading: I found these blogs and youtube videos useful for basic understanding and exploring various methods in these categories: https://www.technologynetworks.com/informatics/articles/supervised-vs-unsupervised-learning-352077 https://towardsdatascience.com/supervised-vs-unsupervised-learning-14f68e32ea8d https://link.springer.com/book/10.1007/978-0-387-84858-7 I hope you found the blog insightful. Will discuss one of the unsupervised method, clustering in my next blog. References : Elements of statistical learning – Trevor Hastie, Robert Tibshirani , Jerome Friedman https://www.ibm.com/cloud/blog/supervised-vs-unsupervised-learning

Source