Formula-Driven Supervised Learning (FDSL) pre-trains vision models on images rendered from mathematical formulas, so no real images, annotations, or privacy concerns are involved. Existing FDSL methods, however, treat pre-training as classification: the continuous generation parameters are discretized into class labels, and everything that happens inside a class is thrown away.

Fractals are generated by an Iterated Function System (IFS), and their shapes change gradually as the affine parameters change. We argue that this continuity is the supervision signal itself. Instead of asking a model to recognize categories, we ask it to respect the relative similarity orderings induced by the generative parameters.

FDSML keeps the standard triplet objective and changes only how positives and negatives are chosen.

Each image $\boldsymbol{x}_i$ is generated by an IFS $\{(\boldsymbol{\psi}_k, \pi_k)\}_{k=1}^{N}$, where $\boldsymbol{\psi}_k \in \mathbb{R}^6$ are the affine parameters of map $k$ and $\pi_k$ its selection probability. We summarize the generative process with a compact descriptor and a distance on it:

$$\boldsymbol{\Psi}_i = \sum_{k=1}^{N} \pi_k \, \boldsymbol{\psi}_k \in \mathbb{R}^6, \qquad d_{\boldsymbol{\Psi}}(i, j) = \lVert \boldsymbol{\Psi}_i - \boldsymbol{\Psi}_j \rVert_2 .$$

Given an anchor and two candidates, $\boldsymbol{x}_1$ is the positive and $\boldsymbol{x}_2$ the negative whenever $d_{\boldsymbol{\Psi}}(a, 1) \le d_{\boldsymbol{\Psi}}(a, 2)$, and the triplet is fed to the usual margin loss:

$$\mathcal{L}_{\mathrm{triplet}} = \max\Bigl(0,\; \lVert g_\theta(\boldsymbol{x}_a) - g_\theta(\boldsymbol{x}_p) \rVert_2 - \lVert g_\theta(\boldsymbol{x}_a) - g_\theta(\boldsymbol{x}_n) \rVert_2 + \alpha \Bigr).$$

Note that $d_{\boldsymbol{\Psi}}$ is used only ordinally, to decide which candidate is closer. It is not treated as a perceptual distance, so the embedding is trained to respect relative orderings rather than categorical separability.

Two fractals drawn at random from the whole dataset are usually so different that the comparison is trivial. We therefore use the discrete class labels as a coarse prior that restricts the candidate pool; the final positive/negative assignment is still made by $d_{\boldsymbol{\Psi}}$.

Does the parameter ordering actually match what is rendered? We measured how often the candidate closer in $d_{\boldsymbol{\Psi}}$ is also closer under the symmetric Chamfer distance between the rendered shapes.

Under each, the descriptor essentially reproduces the class label. Under same, no label rule exists at all, yet the descriptor still reaches 80.6 %: this is supervision that class labels cannot provide even in principle, and it is the regime where FDSML works best.

FractalDB-1k creates its 1,000 instances per class from only 25 parameter perturbations combined with flips and rendering variations, and its affine maps are not forced to be contractive. The result is low intra-class diversity (left, blue box) and collapsed, uninformative samples (right, red box), neither of which is good for fine-grained relational supervision.

FineFractalDB keeps the 1,000 classes × 1,000 images of FractalDB-1k but refines the generation process:

• 1,000 independent parameter perturbations per class (instead of 25), so every instance is a distinct point in parameter space.
• Strict contraction constraints on every affine map, which keep the attractor stable.
• Rejection sampling on pixel occupancy (< 0.05 % discarded), which removes nearly empty renderings.

Drag the slider to compare random samples from the two datasets. Each row is one class, each column a different instance.

Retrieval depends on the geometry of the embedding space rather than on decision boundaries, which makes it the most direct test of our claim. With a ResNet-50 backbone fine-tuned with the Proxy-Anchor loss, FDSML pre-training on FineFractalDB consistently beats both training from scratch and classification-based FDSL on FractalDB-1k.

Two observations stand out. First, on SOP the FDSL model is actually worse than training from scratch (65.6 vs. 70.4 Recall@1): classification-based synthetic supervision does not automatically provide a useful inductive bias for embedding-based tasks. Second, FDSML closes a large part of the gap to real-image pre-training while never seeing a natural image. The full table with all filtering strategies and Recall@K values is in the paper.

FDSML is designed for relational tasks, but it does not sacrifice category recognition. Top-1 accuracy after fine-tuning (%):

On ViT-Tiny, FDSML lifts ImageNet-100 from 84.40 % to 88.90 % and even edges out supervised ImageNet-1k pre-training on CIFAR-10. The same trend holds for ViT-Base (see the paper).

The objective and the dataset are complementary

Ablation with ViT-Tiny (Top-1 %). Swapping the objective alone already helps; swapping the dataset helps only under FDSML.

Cross-entropy assigns the same label to every instance in a class, so it cannot exploit the extra intra-class variation of FineFractalDB and even suffers from it. PA-Triplet turns exactly that variation into supervision.

Robustness to corruptions

Classification error on ImageNet-C (lower is better), averaged within each corruption group:

For ResNet-50, FDSML improves every corruption group. For ViT-Tiny the picture is mixed: structural and environmental corruptions (blur, weather, geometric) improve, while high-frequency noise and digital corruptions get worse. We report this trade-off as is; understanding its frequency-domain cause is future work.

Where does the model look?

Grad-CAM for ResNet-50 models fine-tuned on ImageNet-1k. FDSML produces activation maps that are concentrated on the object, qualitatively closer to the ImageNet pre-trained model, whereas FDSL responses are often diffuse or drift to the background.

• Show more Grad-CAM examples

Attention inside the Transformer

Layer-wise CLS-to-patch attention of ViT-Tiny on a fractal input, from shallow (left) to deep (right). FDSL attends almost exclusively to the silhouette boundary; FDSML also attends to textured interior regions, which suggests it captures fine-grained appearance in addition to coarse structure.

Structure of the embedding space

UMAP projection of ResNet-50 features for ten FineFractalDB test classes. FDSML (each) organizes each class into structured trajectories that follow the generative parameters, with the images (A)–(D) sampled along one trajectory shown below each panel.

• Only IFS fractals so far. PA-Triplet could be instantiated for other formula-driven generators (e.g. VisualAtom, TileDB), but each needs its own similarity definition over its parameters and its own validation.
• One-million-image scale. Whether the gains from parameter-aware supervision persist at larger scales is an open question.
• Extra overhead. Descriptor computation and triplet construction add cost over cross-entropy FDSL; we have not analyzed this cost in detail.
• Architecture-dependent robustness. The ViT noise/digital trade-off above remains unexplained.

The FDSML source code is released under CC BY-NC-SA 4.0 (see LICENSE). Parts of the code are derived from the third-party projects below, which keep their original licenses (see NOTICE.md for details).

This project page is built on the academic-project-template by denkiwakame, provided under CC BY-SA 4.0.

This work was supported by JSPS KAKENHI Grant Numbers JP24H00742 and JP24H00748, JST SPRING Grant Number JPMJSP2128, and computational resources of ABCI 3.0 provided by AIST and AIST Solutions.