Linear Regression

Linear Regression seeks to establish a connection between independent and dependent variables by determining a “best-fit line” that minimizes the distance from all data points through the least squares method. This method aims to find a linear equation that minimizes the sum of squared residuals (SSR).

For instance, the green line depicted below represents a better fit than the blue line due to its minimal distance from all data points.

Lasso Regression (L1)

Lasso Regression is a regularization technique aimed at curbing overfitting by incorporating a degree of bias into the model. It minimizes the squared difference of residuals while introducing a penalty, where the penalty is proportional to the absolute value of the slope, scaled by a parameter known as lambda. This lambda serves as a hyperparameter that can be adjusted to enhance model fitting.

L1 Regularization is particularly advantageous when dealing with many features, as it tends to ignore variables with minimal slope values.

Ridge Regression (L2)

Ridge Regression functions similarly to Lasso Regression, with the primary distinction being in the calculation of the penalty term. It incorporates a penalty that is the square of the magnitude multiplied by lambda.

L2 Regularization is optimal when facing multicollinearity, where independent variables show strong correlation, as it shrinks all coefficients towards zero.

Elastic Net Regression

Elastic Net Regression merges the penalties from both Lasso and Ridge Regression, offering a more regularized model. This method balances both penalties, typically resulting in superior performance compared to using either L1 or L2 in isolation.

Polynomial Regression

Polynomial Regression models the relationship between independent and dependent variables as an n-degree polynomial. The polynomial expressions are sums of terms in the format of ( k.x^n ), where ( n ) is a non-negative integer, ( k ) is a constant, and ( x ) is the independent variable. This approach is particularly suited for non-linear datasets.

Logistic Regression

Logistic Regression is a classification method that determines the best-fit curve for a dataset. It employs the sigmoid function to map outputs to a range between 0 and 1. Unlike linear regression, which uses the least squares method, logistic regression utilizes Maximum Likelihood Estimation (MLE) to ascertain the optimal curve.

K-Nearest Neighbors (KNN)

KNN is a classification algorithm that categorizes new data points by evaluating their proximity to the nearest classified points. It operates on the assumption that closely situated data points are likely to be similar.

This algorithm is often referred to as a lazy learner, as it retains the training data and only classifies when a new data point requires prediction.

Typically, KNN employs Euclidean distance to identify the nearest classified points, and the mode of these closest classes is selected to determine the predicted class for the new point.

If the value of K is too low, a new data point may be misclassified as an outlier; conversely, if K is too high, it may dilute the impact of classes with fewer samples.

Naive Bayes

Naive Bayes is a classification technique rooted in Bayes Theorem, predominantly applied in text classification tasks. Bayes Theorem outlines the probability of an event based on pre-existing knowledge of related conditions.

The theorem can be summarized as follows:

The term "Naive" refers to the assumption that the presence of a specific feature is independent of the presence of other features.

Support Vector Machines

The objective of Support Vector Machines (SVM) is to identify a hyperplane in an n-dimensional space (where n represents the number of features) that effectively separates data points into distinct classes. This hyperplane is determined by maximizing the margin between classes.

Support vectors are the data points closest to the hyperplane, which can influence its position and orientation, thereby helping to maximize the margin between different classes.

Decision Tree

A Decision Tree is a classifier structured like a tree, containing a sequence of conditional statements that guide a sample to a conclusion.

The internal nodes of a decision tree represent features, branches signify decision rules, and leaf nodes indicate outcomes. The decision nodes function as if-else statements, while leaf nodes contain the results of those decisions.

The process begins by selecting an attribute for the root node using an attribute selection measure (such as ID3 or CART), and recursively compares subsequent attributes with their parent node to generate child nodes until reaching the leaf nodes.

Random Forest

Random Forest is an ensemble learning method that comprises multiple decision trees. It employs bagging and feature randomness during the construction of each tree to develop an uncorrelated forest of decision trees.

Each tree within a random forest is trained on a different subset of data to predict outcomes. The final prediction is determined by the majority vote among the trees.

For example, if a single decision tree predicts class 0 while the ensemble predicts class 1, this illustrates the strength of the random forest approach.

Extra Trees

Extra Trees closely resembles the Random Forest classifier, with the key difference lying in root node selection. While Random Forest utilizes the optimal feature for splitting, Extra Trees selects a feature randomly, enhancing randomness and reducing feature correlation.

Additionally, Random Forest employs bootstrap replicas to generate subsets of size N for training, whereas Extra Trees utilize the entire original dataset.

Due to its unique approach, the Extra Trees algorithm is typically faster in computation compared to Random Forest.

AdaBoost

AdaBoost is a boosting algorithm that differs from Random Forest in several ways:

1. Rather than creating a forest of decision trees, AdaBoost constructs a forest of decision stumps (a stump is a decision tree with a single node and two leaves).
2. Each decision stump is allocated distinct weights in the final decision-making process.
3. It assigns higher weights to misclassified data points, emphasizing their significance in the development of subsequent models.
4. The process merges multiple "weak classifiers" into a robust classifier.

Gradient Boosting

Gradient Boosting constructs multiple decision trees, where each subsequent tree learns from the errors made by its predecessors. It leverages residual errors to enhance predictive accuracy, aiming to minimize these errors as much as possible.

Similar to AdaBoost, the key difference is that AdaBoost builds decision stumps, whereas Gradient Boosting creates decision trees with multiple leaves.

The process commences with the creation of an initial decision tree that provides average predictions, followed by a new tree that uses the initial features and residual errors as dependent variables. Predictions are iteratively refined until reaching minimal error.

XGBoost

XGBoost is an advanced and regularized version of Gradient Boosting. It incorporates sophisticated regularization techniques (L1 & L2) to enhance the model's ability to generalize.

XGBoost utilizes similarity scores between leaves and their parent nodes to determine the appropriate root and child nodes.

K-Means Clustering

K-Means Clustering is an unsupervised machine learning algorithm that categorizes unlabeled data into K distinct clusters, where K is predetermined by the user.

This iterative algorithm employs cluster centroids to partition unlabeled data into K clusters, ensuring that data points with similar characteristics are grouped together.

1. Define K and create K clusters.
2. Calculate the Euclidean distance of each data point from the K centroids.
3. Assign the closest data point to each centroid to form a cluster.
4. Recalculate centroids by averaging the assigned data points.

Hierarchical Clustering

Hierarchical Clustering is another clustering method that organizes data into a hierarchy of clusters, represented in a tree structure. This method autonomously identifies relationships within the data and separates them into n clusters, where n corresponds to the dataset size.

There are two primary approaches to hierarchical clustering: agglomerative and divisive.

Agglomerative clustering starts with each data point as an individual cluster, gradually merging them until only one cluster remains. Conversely, divisive hierarchical clustering begins with the entire dataset as one cluster, progressively splitting it into smaller, less similar clusters.

DBSCAN Clustering

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) operates under the assumption that a data point belongs to a cluster if it is close to multiple points within that cluster rather than relying on any single point.

Two vital parameters in DBSCAN are epsilon and min_points. Epsilon defines the proximity required for points to be considered part of a cluster, while min_points establishes the minimum number of points necessary to form a cluster.

Apriori Algorithm

The Apriori algorithm is an association rule mining technique that correlates data items based on their interdependencies.

Key steps for generating an association rule using the Apriori algorithm include:

1. Calculate support for each item set of size 1, where support indicates item frequency within the dataset.
2. Eliminate item sets below the minimum support threshold, as determined by the user.
3. Construct item sets of size n+1 (where n is the size of the previous item set) and repeat steps 1 and 2 until all item sets exceed the support threshold.
4. Create rules using confidence, which measures how often x and y co-occur given that x is already present.

Principal Component Analysis (PCA)

PCA is a linear dimensionality reduction technique that transforms correlated features into a smaller number of uncorrelated features known as principal components.

Although implementing PCA results in some information loss, it offers numerous advantages, such as enhancing model performance, decreasing hardware requirements, and improving data visualization opportunities.

Thanks for Reading!

If you enjoyed this content and wish to support my work, consider following me on Medium and my publication tailored for Python developers and AI enthusiasts. Connect with me on LinkedIn, and if you're interested, join Medium through my referral link—part of your membership fee will support me.

Stay updated by subscribing to my email list so you won’t miss future articles!