Motion graphics videos are widely used in Web design, digital advertising, animated logos and film title sequences, to capture a viewer’s attention.
But editing such video is challenging because the video provides a low-level sequence of pixels and frames rather than higher-level structure such as the objects in the video with their corresponding motions and occlusions.
We present a motion vectorization pipeline for converting motion graphics video into an SVG motion program that provides such structure.
The resulting SVG program can be rendered using any SVG renderer (e.g. most Web browsers) and edited using any SVG editor.
We also introduce a program transformation API that facilitates editing of a SVG motion program to create variations that adjust the timing, motions and/or appearances of objects.
We show how the API can be used to create a variety of effects including retiming object motion to match a music beat, adding motion textures to objects, and collision preserving appearance changes.
background
A more formal way of describing the problem above is to consider our data as observations of some signal which maps latent variables
However, even with this information we know nothing about the geometry of the individual
The answer is yes! But to do so, we need some theoretical results first.
the laplacian over product spaces
The main theoretical result we can leverage is the way the Laplacian operator decomposes over product spaces. Recall that the Laplacian
with Neumann boundary conditions
We call these
where
i.e. the eigenvalue of the eigenfunction is precisely the sum of the eigenvalues of the eigenfunctions which form the product. Lastly, these eigenvalues are all nonnegative and monotonically increasing.
To illustrate with an example, consider the simplest possible product space—a rectangle, which is the product of two closed line segments. Figure 2 shows the part of the spectrum of the rectangle, broken down into a sort of infinite multiplication table of the spectrum of each line segment.
Borrowing some fundamental terminology, we call the eigenfunctions along the top row and left column “factor eigenfunctions” and the rest of the eigenfunctions “product eigenfunctions.”
our method
Suppose we are just in the two manifold factor case right now, i.e.
-
Compute the spectrum of the Laplacian. Since we don’t have infinite data samples, we need to approximate the eigenfunctions. We can do so by computing
eigenvectors of the graph Laplacian instead. For more information on the asymptotics of the eigenvectors, see Stefan Lafon’s PhD dissertation. -
Factorize the eigenvectors. From step 1, we obtain a set of eigenvectors sorted by increasing eigenvalue. Using our result that product eigenvectors factorize into factor eigenvectors, for each eigenvector
, we find the factorization which is closest to . This “closeness” is measured by the absolute cosine similarity,2This gives us a list
of triplets , where each triplet contains our best approximation . -
Group factor eigenvectors by manifold factor. Define a weighted graph
where is the set of unique factor eigenvectors in . The edge weights are given by the highest similarity scoreOur problem now becomes one of separating the vertices into two groups, with higher priority given to separating vertices with high edge weights. Thus, this becomes formulated as a MAX-CUT problem. By using a MAX-CUT SDP solver with
as the input, we can sort our factor eigenvectors into two bins that correspond to and .
some fun results!
An interesting application of this method is in structural biology—specifically, applied to cryo-electron miscroscopy (cryo-EM) data. Cryo-EM is a microscopy technique that won the Nobel Prize in chemistry and also played a role in many of the COVID-19 studies done over the past year.
The main problem in cryo-EM is a reconstruction problem: given all these images, can we recover information about the 3D structure of the molecule being imaged? In particular, a macromolecule may be composed of several different independently moving subunits. If each of these subunits moves continuously, then the shape space of the entire molecule can be modeled as a product manifold!
In our experiments, we use a model of a molecule with two independently moving subunits (see Figure 4, right). The red subunit rotates freely around the z-axis (latent space = circle), and the blue subunit stretches linearly along a limited range of the x-axis (latent space = line segment). Applying our algorithm gives two eigenvector groups, shown in Figure 5. We can see that the group on the left corresponds to the spectrum of the blue subunit (eigenvectors of a line) and the group on the right corresponds to the spectrum of the red subunit (eigenvectors of a circle).
So now we have eigenvector groups corresponding to each manifold factor. What’s next? Well, one thing we can do is use these eigenvectors to visualize the structure of each manifold factor. To do so, we can plot a diffusion map embedding of each eigenspace. Figure 6 shows visualizations using our method compared to standard diffusion maps and linear ICA. Without the factorization, it’s hard to get a good understanding of the individual structures within the data.
future work
A big question for future research is what this looks like in the case of
On a different note, I also had a great presenting at AISTATS! Unfortunately everything was virtual this year, but despite this we actually had a good amount of visitors come to our poster and engage in some nice discussions with us. It was great to experience a conference poster session for the first time (I was definitely a little nervous, even behind a screen), and I’m looking forward to more conferences in the future.
This work was done in collaboration with Amit Moscovich and Amit Singer.