1 Conclusions↩︎

Our approach of permutation-invariance through Janossy pooling unifies a number of existing approaches, and opens up avenues to develop both new methodological extensions, as well as better theory. Our paper focused on two main approaches: \(k\)-ary interactions and random permutations. The former involves exact Janossy pooling for a restricted class of functions \(\mathstrut\mkern 2.5muf\mkern-11mu\raise 1.6ex \scriptscriptstyle\rightharpoonup\). Adding an additional neural network \(\rho\) can recover lost model capacity and capture additional higher-order interactions, but hurts tractability and identifiability. Placing restrictions on \(\rho\) (convexity, Lipschitz continuity etc.) can allow a more refined control of this trade-off, allowing theoretical and empirical work to shed light on the compromises involved. The second was a random permutation approach which conversely involves no clear trade-offs between model capacity and computation when \(\rho\) is made more complex, instead it modifies the relationship between the tractable approximate loss \(\overline{\overline{\lower 0.2exJ}}\) and the original Janossy loss \(\overline{\overline{\lower 0.2exL}}\). While there is a difference between \(\overline{\overline{\lower 0.2exJ}}\) and \(\overline{\overline{\lower 0.2exL}}\), we saw the strongest empirical performance coming from this approach in our experiments (shown in the last row of Table [tab:accuracy]); future work is required to identify which problems \(\pi\)-SGD is best suited for and when its convergence criteria are satisfied. Further, a better understanding how the loss-functions \(\overline{\overline{\lower 0.2exL}}\) and \(\overline{\overline{\lower 0.2exJ}}\) relate to each other can shed light on the slightly black-box nature of this procedure. It is also important to understand the relationship between the random permutation optimization to canonical ordering and how one might be used to improve the other. Finally, it is important to apply our methodology to a wider range of applications. Two immediate domains are more challenging tasks involving graphs and tasks involving non-Poisson point processes.