Let's assume we have a data set that contains the following information:

Each sample was measured with 10 replicates and the mean values are given in the table below.

                
                  ┌────────┬───────┐
                  │ Sample │ [Fe]  │
                  ├────────┼───────┤
                  │    1   │  2.3  │
                  │    2   │  2.7  │
                  │    3   │  1.9  │
                  │    4   │  2.1  │
                  │    5   │  2.5  │
                  └────────┴───────┘
                                    
            

Now, let's assume we have a data set that contains the following information:

Each sample was measured with 10 replicates and the mean values are given in the table below.

                
                  ┌────────┬───────┬───────┬───────┬───────┐
                  │ Sample │ [Fe]  │ [Cu]  │ [Zn]  │ [Mn]  │
                  ├────────┼───────┼───────┼───────┼───────┤
                  │    1   │  2.3  │  1.2  │  0.5  │  0.2  │
                  │    2   │  2.7  │  1.1  │  0.4  │  0.3  │
                  │    3   │  1.9  │  1.3  │  0.6  │  0.1  │
                  │    4   │  2.1  │  1.0  │  0.3  │  0.4  │
                  │    5   │  2.5  │  1.4  │  0.7  │  0.5  │
                  └────────┴───────┴───────┴───────┴───────┘
                                    
            

There are several metrics available to calculate the similarity between samples with multiple variables (multivariate data).

The Minkowski distance is a generalized difference-based metric: \[ d = \left( \sum_{i=1}^{n} |x_i - y_i|^p \right)^{1/p} \] where \( p \) is the order of the distance. Adjust \( p \) with the slider to see how the distance calculation changes.

Correlation coefficients quantify the similarity between variables by measuring linear relationships.

Pearson Correlation: Measures linear relationships between two variables, ranging from -1 to +1. \[ r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}} \] where \(\bar{x}\) and \(\bar{y}\) are the means of the variables.

\[d = 1 - r\] where \(r\) is the Pearson correlation coefficient and \(d\) is the distance.

Correlation coefficients quantify the similarity between variables by measuring linear relationships.

Spearman Correlation: Ranks* data to assess monotonic relationships, useful for non-linear data. \[ r_s = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)} \] where \(d_i\) is the difference between ranks*.

\[d = 1 - r_s\] where \(r_s\) is the Spearman correlation coefficient and \(d\) is the distance.

Cosine similarity measures the similarity between two vectors by comparing their direction, normalized by their magnitudes.

Cosine Similarity: Defined as the dot product of two vectors divided by the product of their norms. \[ \text{similarity} = \frac{\sum_{i=1}^{n} x_i \cdot y_i}{||X|| \cdot ||Y||} \] where \( ||X|| = \sqrt{\sum_{i=1}^{n} x_i^2} \) and \( ||Y|| = \sqrt{\sum_{i=1}^{n} y_i^2} \).

Minkowski Distance

General distance metric for multivariate data.

Adjustable: Manhattan (p=1), Euclidean (p=2).

Works best with continuous numeric data.

Pearson Correlation

Measures linear relationships.

Good for continuous, normally distributed data.

Sensitive to outliers.

HCA groups similar items into clusters based on their similarity or distance.

The result is a dendrogram, a tree-like structure showing the nested clusters.

Types of HCA:

Agglomerative (bottom-up): Starts with individual items and merges them into clusters.

Divisive (top-down): Starts with one cluster of all items and splits them into smaller clusters.

HCA starts by calculating distances or similarities between all pairs of items.

The result is a distance matrix, showing how similar or different each pair of items is.

Common distance measures include:

Euclidean distance: Measures straight-line distance.

Manhattan distance: Measures distance by summing absolute differences.

The two items with the smallest distance are merged to form a cluster.

This cluster is represented as a node in the dendrogram, showing the first grouping of similar items.

Each node’s height represents the distance at which items or clusters are joined.

After each merge, recalculate distances between the new cluster and other items or clusters.

Different linkage methods can be used to update distances:

Single linkage: Uses the smallest distance between items in two clusters.

Complete linkage: Uses the largest distance between items in two clusters.

Average linkage: Uses the average distance between all items in two clusters.

The process continues until all items are merged into a single cluster.

The dendrogram visually represents this nested grouping structure.

The height of each node indicates the distance at which clusters are merged.

The process continues until all items are merged into a single cluster.

The dendrogram visually represents this nested grouping structure.

The height of each node indicates the distance at which clusters are merged.

A dendrogram helps identify clusters and understand the relationship between items:

Each leaf represents an individual item in the dataset.

Clusters are formed by cutting the dendrogram at a certain height.

The height of each node indicates the distance or similarity level at which clusters are joined.

To identify clusters, look for branches that join at lower heights for more similar items.

Advantages

Does not require a predefined number of clusters; HCA provides a complete hierarchy.

Generates a dendrogram that visually represents cluster relationships at multiple levels.

Effective for small to medium datasets, especially when clusters are well-separated.

Ideal for discovering nested or hierarchical data structures, useful in fields like taxonomy and genomics.

k-means is a popular clustering algorithm for partitioning data into \( k \) clusters.

The algorithm aims to minimize within-cluster variance, making clusters as compact as possible.

Each cluster is defined by its centroid, which is the mean of all points within that cluster.

The number of clusters \( k \) must be chosen before running k-means.

Selecting \( k \) can be challenging, as it affects the algorithm’s results significantly.

A common approach is to start with k=2 and increase until the clustering quality stabilizes.

k-means starts by randomly selecting \( k \) initial centroids in the data space.

These centroids act as the starting points for forming clusters.

The choice of initial centroids can affect the final clustering results, so multiple initializations may be tested.

Each data point is assigned to the nearest centroid, forming initial clusters.

This assignment is based on minimizing the distance between points and centroids (usually Euclidean distance).

Points closest to a centroid belong to that centroid’s cluster.

Once points are assigned, the centroids are recalculated as the mean of all points in each cluster.

This step shifts each centroid to a new position, centered within its assigned points.

After updating, points are reassigned to the nearest centroid, and the process repeats.

k-means alternates between assigning points and updating centroids until convergence.

Convergence is reached when centroid positions stop changing significantly.

The final clusters minimize within-cluster variance.

To assess clustering quality, metrics such as the following are commonly used:

Inertia (within-cluster variance): Measures compactness within clusters; lower is better.

Silhouette Score: Measures cohesion and separation; ranges from -1 to 1.

ANOVA: Evaluates the separation between clusters by comparing between-cluster and within-cluster variance.

Advantages

Simple and efficient for large datasets with low computational complexity, especially when clusters are well-separated.

Can be easily adapted for different use cases, such as image compression and document clustering.

Works well with spherical-shaped clusters where data points are evenly distributed around a centroid.

Principal Component Analysis (PCA) is a dimensionality reduction technique often used in data science and machine learning to simplify datasets by reducing the number of variables while preserving essential information.

PCA works by identifying directions (principal components) along which the data varies the most.

Applications include data visualization, noise reduction, and feature extraction for machine learning.

The coordinate axes of a space can be represented as a matrix. In its simplest form, this is the identity matrix, where all axes are perpendicular.

The identity matrix (or unity matrix) has values of 1 along its diagonal and 0 elsewhere, representing standard perpendicular axes.

By modifying the angles between the axes, we can adjust this matrix to define new coordinate systems.

PCA reduces dimensions by transforming coordinate axes. It finds new axes, principal components, where data varies most.

This is a mathematical axis transformation, projecting original data onto principal component axes.

Example: With two features, x1 and x2, PCA rotates the axes to find PC1 (highest variance), possibly discarding PC2 for dimensional reduction.

The covariance matrix reveals directions in the data with maximum variation, guiding PCA to choose the best transformation.

Each matrix element represents covariance between two features. Diagonal elements show variance within each feature, while off-diagonal elements show relationships between features.

The covariance matrix shows how dimensions of the data correlate and spread. Each value describes the joint variance between two dimensions.

The eigenvectors of the covariance matrix represent directions of the greatest variance in the data. These define the new axes, known as principal components.

Suppose we have a dataset with four features: x1, x2, x3, and x4. The covariance matrix reveals relationships between these features:

              
                         ┌                    ┐
                         │ 2.5  0.8  0.6  1.2 │
                   Cov = │ 0.8  1.9  0.4  0.9 │
                         │ 0.6  0.4  2.3  1.1 │
                         │ 1.2  0.9  1.1  3.2 │
                         └                    ┘
              
          

Diagonal elements (e.g., 2.5 for x1) show variance for each feature. Off-diagonal values indicate covariance between features (e.g., 0.8 between x1 and x2).

An eigenvector of a matrix is a special vector that, when multiplied by the matrix, changes only in scale, not direction.

For a matrix A and an eigenvector v, we have: \[ A \cdot v = \lambda \cdot v \] where λ is the eigenvalue that scales v.

In PCA, eigenvectors of the covariance matrix give directions for the principal components, with the eigenvalues showing their importance (variance captured).

The covariance matrix of the data:

                    
                           ┌                    ┐
                           │ 2.5  0.8  0.6  1.2 │
                     Cov = │ 0.8  1.9  0.4  0.9 │
                           │ 0.6  0.4  2.3  1.1 │
                           │ 1.2  0.9  1.1  3.2 │
                           └                    ┘
                    
                

Eigenvectors represent the new axes in PCA, also known as principal components.

The individual values within an eigenvector describe how much each original feature contributes to the principal component.

A larger value indicates that a specific feature plays a more significant role in this component.

To transform the data, we use the eigenvectors of the covariance matrix. These eigenvectors form a new basis for the data.

Each data point is projected onto this new basis, creating two key components:

Scores: Represent the data points in the new coordinate system defined by the principal components.

Loadings: Show how the original variables contribute to each principal component.

Different Fruits as Samples.

Apple, Banana, Lime, Grape, Pineapple

Different measurements methods as Features.

Sweetness, Weight, Price

Sourness, Color(Red), Color(Blue), Color(Green)

The PCA bi-plot shows the relationship between samples and features in the transformed space.

Samples are represented by points, while features are shown as vectors.

The length and direction of feature vectors indicate their importance and relationship to samples.

Advantages of PCA

Dimensionality Reduction: Simplifies datasets by reducing variables, preserving core information.

Noise Reduction: Filters noise by focusing on components with the highest variance.

Improved Visualization: Enables visualization of high-dimensional data in 2D or 3D.

Feature Extraction: Highlights uncorrelated variables, making dominant features clear.

Linear Discriminant Analysis (LDA) is similar to PCA, but with a focus on distinguishing between predefined groups.

Instead of maximizing overall variance, LDA aims to maximize separation between groups.

LDA uses the within-group covariance matrix inverse, multiplied by the between-group covariance matrix, to find new axes. \[ C = \Sigma_w^{-1} \cdot \Sigma_b \]

We will have a look on a dataset of 178 different Wines.

Multiple measurements were taken for each wine.

Using PCA, we will try to reduce the dimensionality of the dataset.

Using K-Means, we will try to cluster the wines into different groups.