MBM algorithm

By Ward Dehairs, January 11, 2024

Working for Spentys, I was tasked with upgrading a core element of the Forma application: the ability to change the pose of an arbitrary scan of a body part such as the hand. While superficially an easy task, doing this at a high quality is very difficult. The standard practice for creating well rigged characters in 3D animation is to have artists spend days polishing and perfecting a 3D model, yet we want software to do this process automaticaly...

The MBM model algorithm is a general solution for this problem. The basic idea is to have a reference or base mesh that is morphed to conform to the scan.

Morphing means to change the shape or more specifically the position of the vertices without changing faces or edges. In other words the topology of the base mesh is conserved. Using this method we can predefine a rig; a detailed 3D model of the body part that is fully equipped with a skeleton and weight painting and/or other means of posing such as blend shapes. This model can be polished and refined as much as needed, while the algorithm in essence just modifies this idealised model to conform to the scan. Thus we arrive at a rigged version of the input scan that the end-user can pose as they please.

The starting point: an idealized rigged mesh and an arbitrary scan.

We start by shaping a copy of the idealized mesh into the form of the scan. This is not a simple step, but it is relatively common practice so pre-existing algorithms can be used to achieve this.

With the morphed mesh available, the next step is to match the skeleton of the rig to the morph result. This can be done rather accurately using the Kabsch algorithm, although this actually doesn't lead to great results in terms of rigging and posing later on.

A better approach is to use a series of pre-defined pivot points in the original base mesh. These points can be mapped onto the morphed mesh by defining them relative to some points on the mesh itself. This can be done via 3D bary-centric coordinates, while the reference points in term are mapped via 2D bary-centric coordinates defined on the mesh.

After getting the bones in a satisfactory position, the morphed mesh is inserted into it. This requires the skinning effect to be inverted, which can be done exactly since a skinning deformation is just a linear transform.

In post-processing we can use the determined transform of the bones to determine physical angles such as the hand abduction here.

This was eventually incorporated into the forma application.