primer basics machine learning factorization
Dec. 5, 2019

A Google research scientist, Steffen Rendle, introduced Factorization Machines in one of his papers in 2010. As a result, the machine learning community quickly took notice. Today, factorization machines have become a built-in algorithm in Amazon SageMaker. For many reasons, it has therefore become a popular and impactful method for making predictions and recommendations.

Understanding Factorization Machines

Factorization machines are a type of supervised learning algorithm. They are used for classification and regression tasks. They accomplish this by measuring interactions between variables within large data sets.

FM’s are extensions of linear models which model the interactions of variables. They map and plot their interactions to a lower dimension. As a result, the number of parameters extends linearly through the dimensions.

Factorization Machines are comparable to Support Vector Machines (SVM). Now they are considered a cross between SVMs and matrix factorization. Support Vector Machines are popular prediction algorithms in machine learning that are liked for their simplicity. However, SVMs have significant weaknesses. For example, they can’t learn reliable parameters in non-linear dimensions. SVMs use dense parametrization and their computation of a prediction relies on the training data, or support vectors.

Advantages of Factorization Machines

Performance and accuracy are state-of-the-art. Similar to SVMs, Factorization Machines are general prediction models. However, this algorithm is based on interactions of variables with factorized parameters. That’s why Factorization Machines are capable of handling problems where data is hugely sparse. In sum, the advantages of Factorization Machines include:

  • They can estimate parameters under very sparse data and therefore scale to fit large datasets.
  • They depends on a linear number of parameters
  • FM doesn’t rely on training data, resulting in more compact models.
  • FM’s s are generally able to work with any real-valued feature vector. Other factorization models require special input data.

Other Types of Factorization Models vs. Factorization Machines

These models did not always exist. A few earlier models includes matrix factorization (MF), parallel factor analysis, and other specialized models like SVD++, PITF, and FPMC.

The downfall of more specialized models is they aren’t as ideally suited for making general predictions.

For FM, by defining the features of the training data, they can perform the same functions as other factorization models. Therefore, Factorization Machines can serve a dual purpose.

Factorization Machines are therefore a great way to quickly make a prediction with little data preparation. It is one of the best tools for a fast, generalized outcome.

Today, they hold great promise in the world of machine learning. As a result, it’s a must-have algorithm in any data scientist’s back pocket.