1 Introduction↩︎

A cube held before a lamp casts a two-dimensional shadow. The shadow is genuinely 2D, every point of it lies in the plane, yet it is not a generic 2D shape: as the cube rotates, the shadow deforms in ways no rigid flat object could, and even a single frame carries statistical traces of the projection that produced it. This paper asks the analogous question one dimension up, computationally: given only a 3D point cloud, or a short sequence of them, can any method decide whether the data is the projection of a higher-dimensional rigid object?¹

Three observations motivate the benchmark.

1.0.0.1 (1) The task is not intrinsic-dimension estimation.

A rich literature estimates the dimension of the manifold underlying observed data [1]–[3]. But a projection reduces dimension: the orthographic shadow of the 3-sphere \(S^3 \subset \mathbb{R}^4\) is a solid 3-ball; the shadow of the 2-dimensional Clifford torus remains 2-dimensional. Intrinsic dimension of the observed data therefore cannot, even in principle, fully identify shadows. What distinguishes them is how mass, density, and topology are arranged, the signatures of the projection map itself. Section 6.1 quantifies this gap: classical estimators reach 73% on HyperShadow where a small learned model reaches 96%.

1.0.0.2 (2) “4D” datasets in machine learning are space + time.

Existing 4D benchmarks, 4D world models and dynamic point-cloud suites such as MSR-Action3D, DeformingThings4D, and HOI4D [4], [5], model 3D geometry evolving over time. To our knowledge no public dataset contains projections of objects with four or more spatial dimensions, despite the pedagogical ubiquity of the tesseract.

1.0.0.3 (3) Humans can do a version of this task.

[6] showed in psychophysical experiments that human observers discriminate rigid from non-rigid motion of a hypercube’s projection at accuracy comparable to the 3D case, suggesting the perceptual system exploits geometric regularities that do not require four-dimensional experience. There is, however, no machine counterpart: no dataset, no baseline, no algorithmic detector. HyperShadow provides all three, and our rigidity witness (Section 5) is the explicit algorithmic analogue of the human judgement studied there.

1.0.0.4 Contributions.

1. HyperShadow, a reproducible benchmark of 10,800 static point clouds and 1,800 temporal sequences: native 3D shapes vs.3D projections of eleven object families in \(\mathbb{R}^4\)–\(\mathbb{R}^6\), under two projection models and four cumulative corruption tiers, with fairness rules that remove every shortcut we could identify (Section 3).

2. A characterization of the failure of intrinsic-dimension estimation on projected data, establishing that shadow detection is a distinct problem (Section 6.1).

3. Baselines spanning three methodological families, hand-crafted geometric features, persistent homology, and a compact learned point network, including leave-one-family-out generalization tests (Sections 4, 6).

4. The rigidity witness: a zero-parameter, closed-form statistic for temporal data that certifies incompatibility with rigid 3D motion, achieving AUROC 0.982 (Section 5). We propose dimensional witnesses, statistics whose value is provably bounded under any 3D explanation, as a general template, analogous in logical structure to Bell inequalities: one does not observe the hidden structure, one rules out the class of explanations that lack it.

1.0.0.5 Scope.

All results hold under the stated simulation assumptions. Nothing in this paper is evidence for or against higher spatial dimensions in physical reality; the benchmark is an instrument for studying detectability, validated where ground truth is known.

2 Related work↩︎

2.0.0.1 Intrinsic-dimension estimation.

Estimators from Levina–Bickel MLE [1] to TwoNN [2] and their successors, surveyed by [3], infer the dimension of the data manifold, typically benchmarked on manifolds embedded in higher-dimensional ambient spaces [7]. HyperShadow inverts the setting: the generating object has higher dimension than the observation space. Learned bottleneck approaches such as IDEA [8] estimate dimension with autoencoders; they too target the data manifold, not the generator.

2.0.0.2 Topological data analysis.

Persistent homology summarizes multi-scale topology of point clouds [9], with recent work making \(H_1\)/\(H_2\) computation robust in high ambient dimension [10] and enriching standard 3D benchmarks with topological features [11]. Projection alters topology (shadows self-intersect and fill voids), which motivates our TDA baseline.

2.0.0.3 Point-cloud learning.

PointNet-style set networks [12] and successors dominate 3D point classification on ModelNet40 and ScanObjectNN [13]. Our PointNet-lite is a deliberately small member of this family, sized to demonstrate that the task does not require large models or compute.

2.0.0.4 Four spatial dimensions.

Computer-graphics work renders 4D scenes for visualization [14]; vision science shows humans judge 4D rigidity from 3D projections [6]. Existing “4D” ML datasets are dynamic 3D [4], [5]. None provide a benchmark of higher-spatial-dimension projections.

3 The HyperShadow benchmark↩︎

Figure 1 summarizes generation. All code is seeded NumPy; every sample is reproducible from a seed and a metadata record.

3.1 Object families↩︎

Native 3D (label 0), 7 families: sphere, solid ball, torus (area-uniform via rejection sampling), cube surface, capped cylinder, random ellipsoids, and a tube around a trefoil knot (topologically non-trivial). Sampling is uniform with respect to surface area or volume.

Higher-dimensional (label 1), 11 families: unit hyperspheres \(S^3\) and \(S^4\); the solid 4-ball; tesseract and 5-cube boundaries (uniform over cells); the Clifford torus and duocylinder ridge in \(\mathbb{R}^4\) (flat tori, uniform in angles); generalized hypertori in \(\mathbb{R}^4\) and \(\mathbb{R}^5\) (rejection-corrected); and random smooth compact manifolds embedded in \(\mathbb{R}^5\)/\(\mathbb{R}^6\) by random trigonometric maps, providing unbounded family diversity.

3.2 Projection and corruption↩︎

Each \(N\)-D object receives a Haar-uniform rotation in \(\mathrm{SO}(N)\) (QR decomposition of a Gaussian matrix with sign correction), then is projected to \(\mathbb{R}^3\) orthographically (drop coordinates, the mathematical shadow) or perspectively (successive division through a virtual viewpoint along each extra axis, the 4D pinhole camera). Four cumulative corruption tiers follow: tier 0 clean; tier 1 Gaussian jitter (\(\sigma = 2\%\) of cloud scale); tier 2 \(+\) removal of a random 20–40% half-space slab (self-occlusion); tier 3 \(+\) heavier jitter and distance-dependent dropout from a random viewpoint (LiDAR-like sparsity).

3.3 Fairness rules↩︎

Benchmarks of synthetic classes are vulnerable to shortcuts. We remove every one we identified:

• Identical treatment: native shapes receive the same Haar-random (3D) rotations and identical corruption tiers.

• Normalization: every cloud is centred, scaled to unit mean radius, and resampled to exactly 1,024 points, no information in position, scale, or cardinality.

• Volume is not the answer: the solid ball sits in the native class, so “fills a volume” cannot solve the task; conversely the Clifford torus shadow remains a surface, so “is a surface” fails too.

• Class balance by tier: corruption tiers are distributed identically across classes.

3.4 Tracks↩︎

Static: 10,800 clouds of 1,024 points (600 per family), split 70/10/20 into train/validation/test. Temporal: 1,800 sequences of 16 frames \(\times\) 512 points. A single object rotates rigidly at constant angular velocity along a random one-parameter subgroup of \(\mathrm{SO}(d)\) (matrix exponential of a random antisymmetric generator); each frame is projected and normalized independently, with point identity preserved across frames. For label 0 the rigid rotation is in \(\mathbb{R}^3\); for label 1, in \(\mathbb{R}^N\).

4 Baselines↩︎

4.0.0.1 Intrinsic dimension (0 parameters).

TwoNN [2] and Levina–Bickel MLE [1] computed per cloud; a single decision threshold (direction and value fit on half the data, evaluated on the other half).

4.0.0.2 Geometric features + GBT.

39 rotation-invariant features per cloud, the D2 pairwise-distance histogram, radial mass profile, PCA eigenvalue spectrum, \(k\)-NN distance-ratio statistics (a local dimension proxy), and nearest-neighbour density statistics, classified by gradient-boosted trees under 5-fold cross-validation.

4.0.0.3 Persistent homology + GBT.

Ripser [9] persistence diagrams up to \(H_2\) on 256-point subsamples; per-dimension lifetime statistics and top-5 \(H_1\)/\(H_2\) lifetimes; same classifier protocol.

4.0.0.4 PointNet-lite.

A 190,914-parameter set network (Figure 2): shared per-point MLP (64–128–256 channels, batch normalization), concatenated max- and mean-pooling, and a small dropout-regularized head. Training: AdamW, cosine schedule, 60 epochs, random-rotation and jitter augmentation; approximately 4 minutes on a GTX 1650 Ti (4 GB). The deliberate smallness is a claim: the signal is strong enough that no scale is needed.

5 The rigidity witness↩︎

Let \(X_t \in \mathbb{R}^{n\times 3}\) be corresponding points across frames. For any rigid 3D motion there exist \(R \in \mathrm{SO}(3)\), translation \(t\), and scale \(s\) (absorbing per-frame normalization) with \(X_{t+1} = s X_t R^\top + t\) exactly; the optimal \((s, R, t)\) is closed-form (Kabsch/Procrustes). Define the witness \[w(X) \;=\; \frac{1}{T-1}\sum_{t=1}^{T-1} \operatorname{RMS}\bigl(X_{t+1} - \hat{s}_t X_t \hat{R}_t^\top - \hat{t}_t\bigr),\] the mean residual of the optimal rigid alignment between consecutive frames.

For native sequences \(w\) is bounded by the noise floor. For the shadow of a rigid rotation in \(\mathbb{R}^N\), the apparent 3D motion composes visible rotation with a re-weighting of hidden coordinates; no rigid 3D map explains it, and \(w\) stays bounded away from zero. Crucially, \(w\) is a certificate: a large value does not merely correlate with higher dimensionality, it rules out the entire class of rigid-3D explanations, up to noise. This is the logical structure of a Bell inequality, transplanted to kinematics.

Empirically (Figure 3): native sequences concentrate at \(w \approx 0\) (clean tier) and \(w \approx 0.05\) (the noise tier’s floor); shadows concentrate at \(w \approx 0.10\), four times the clean-motion residual. A single threshold fit on half the sequences achieves held-out accuracy \(0.978 \pm 0.004\) over five random splits and AUROC 0.982, with zero learned parameters. The hardest families (Clifford torus 0.92, tesseract 0.93) are those whose random rotation plane most often lies almost entirely within the visible subspace, making the shadow nearly rigid, a geometric, not statistical, limitation.

6 Results↩︎

6.1 Static track↩︎

Table 1 and Figure 4 give the headline comparison. Mean estimated intrinsic dimension is 2.5 (native) vs. (projected) under TwoNN, a real but small gap, confounded exactly as predicted: the solid ball (native, ID \(\approx 3\)) and projected 2-manifolds (ID \(\approx 2\)) sit on the wrong sides of any threshold. Dimensionality is the wrong observable; the projection map, not the manifold dimension, is what leaves evidence.

Learned and feature-based methods degrade gracefully with corruption (PointNet: 0.984 clean \(\rightarrow\) 0.939 sensor tier). The per-shape error profile is theoretically coherent: the residual confusions are \(S^4\)-shadows vs.balls (0.54) and the ball class generally (0.82), pairs that differ only in radial density profile, since the shadow of a hypersphere is a ball with mass pushed toward the rim.

6.2 Generalization to unseen families↩︎

Table 2 shows two consistent patterns: (i) detection transfers well above chance to never-seen generators, so the model learns projection signatures rather than a shape catalog; (ii) 5D objects are uniformly easier than 4D, each collapsed dimension compounds the density-fold signature. The tesseract is the hardest object in the benchmark: its flat cells project to polyhedron-like shadows that mimic native geometry.

6.3 Temporal track↩︎

The rigidity witness (Section 5) dominates: AUROC 0.982 with zero parameters, against 96.2% for the best learned static method. Motion reveals dimensionality that still frames hide, consistent with, and now quantifying, the human results of [6].

7 Limitations and ethics↩︎

7.0.0.1 Simulation scope.

Results certify detectability under our generative assumptions (uniform sampling, two projection models, our noise models). Real sensors differ; Table 2 measures generalization across object families but not across projection physics.

7.0.0.2 The witness needs correspondence.

The Kabsch witness assumes tracked points across frames. Untracked variants (e.g., distributional rigidity via optimal transport) are open.

7.0.0.3 Coverage.

Eleven higher-dimensional families cannot exhaust “higher-dimensional objects”; the random-manifold family mitigates but does not eliminate this.

7.0.0.4 Interpretation discipline.

The benchmark will inevitably invite speculative application to anomalous real-world data. We state plainly: a positive detection on real data would establish only that the data is inconsistent with the rigid-3D model class considered, never the existence of extra dimensions. Every mundane explanation (tracking error, non-rigid objects, atmospherics, optics) dominates a priori and must be eliminated independently.

8 Conclusion↩︎

HyperShadow turns a century-old thought experiment, Plato’s cave, Abbott’s Flatland, one dimension up, into a measurable machine-learning problem with controlled ground truth. Shadows of higher-dimensional objects are detectably different from native 3D geometry; the difference is invisible to intrinsic-dimension estimation, learnable from single frames, and certifiable from motion by a closed-form witness. We release the dataset, models, and code, and propose the systematic study of dimensional witnesses, observable statistics bounded under low-dimensional explanations, as the path by which this line of work could eventually meet physical data.

8.0.0.1 Acknowledgements.

AI assistance was used in developing the synthetic data generation pipeline and baseline code, and in drafting and editing the manuscript. All experiments were designed and executed by the author, and all results, claims, and references were verified by the author.

9 Reproducibility↩︎

All data are generated from seeded NumPy; the following commands reproduce every number in this paper. Thirty-four unit tests verify geometric correctness (Haar orthogonality, measure-uniform sampling, projection identities).

python -m hypershadow.generate --out data --per-class 600 --seed 0
python -m hypershadow.generate --out data --temporal --per-class 100 --seed 1
python -m baselines.features      --data data/static.npz
python -m baselines.id_estimators --data data/static.npz
python -m baselines.tda           --data data/static.npz --max-samples 1200
python -m baselines.pointnet      --data data/static.npz --epochs 60
python -m baselines.pointnet      --data data/static.npz --epochs 40 \;
    --holdout-shapes hypertorus_4d hypertorus_5d
python -m baselines.rigidity      --data data/temporal.npz

10 Shape gallery↩︎

References↩︎