SoK: Certified Robustness for Deep Neural Networks

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Expressive Losses for Verified Robustness via Convex Combinations	98.43%	Deterministic	ICLR 2024	Approach name: CC-IBP
2	Certified Training: Small Boxes are All You Need	98.22%	Deterministic	ICLR 2023	Approach name: SABR
3	Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective	98.14%	Deterministic	NeurIPS 2022	SortNet
4	Fast certified robust training with short warmup	97.95%	Deterministic	NeurIPS 2021
5	Boosting the certified robustness of L-infinity distance nets	97.95%	Deterministic	ICLR 2022
6	Robustra: training provable robust neural networks over reference adversarial space	97.91%	Deterministic	IJCAI 2019
7	Scalable verified training for provably robust image classification	97.77%	Deterministic	ICCV 2019	Train on TPU
8	Towards stable and efficient training of verifiably robust neural networks	97.76%	Deterministic	ICLR 2020
9	Towards certifying L-infinity robustness using neural networks with L-inf-dist neurons	97.70%	Deterministic	ICML 2021	Reported by this paper
10	MixTrain: Scalable Training of Verifiably Robust Neural Networks	97.1%	Deterministic	*preprint
11	Scaling provable adversarial defenses	96.33%	Deterministic	NeurIPS 2018
12	Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope	94.18%	Deterministic	ICML 2018

Showing 1 to 12 of 12 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Towards Evaluating and Training Verifiably Robust Neural Networks	94.02%	Deterministic	CVPR 2021
2	Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective	93.40%	Deterministic	NeurIPS 2022	SortNet
3	Certified Training: Small Boxes are All You Need	93.40%	Deterministic	ICLR 2023	Approach name: SABR
4	Boosting the certified robustness of L-infinity distance nets	93.20%	Deterministic	ICLR 2022
5	Fast certified robust training with short warmup	93.10%	Deterministic	NeurIPS 2021
6	Towards certifying L-infinity robustness using neural networks with L-inf-dist neurons	93.09%	Deterministic	ICML 2021
7	Towards stable and efficient training of verifiably robust neural networks	92.98%	Deterministic	ICLR 2020
8	Scalable verified training for provably robust image classification	91.95%	Deterministic	ICCV 2019	Train on TPU
9	Provably robust boosted decision stumps and trees against adversarial attacks	87.54%	Deterministic	NeurIPS 2019	Non-neural-network approach
10	Adversarial training and provable defenses: bridging the gap	85.7%	Deterministic	ICLR 2020
11	Robustra: training provable robust neural networks over reference adversarial space	83.09%	Deterministic	IJCAI 2019
12	Training for faster adversarial robustness verification via inducing ReLU stability	80.68%	Deterministic	ICLR 2019
13	MixTrain: Scalable Training of Verifiably Robust Neural Networks	58.4%	Deterministic	*preprint
14	Scaling provable adversarial defenses	56.90%	Deterministic	NeurIPS 2018

Showing 1 to 14 of 14 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Provably robust deep learning via adversarially trained smoothed classifiers	68.2%	Probabilistic	NeurIPS 2019	combine adversarial training, self-training, and unlabeled data
2	Unlabeled data improves adversarial robustness	63.8%	Probabilistic	NeurIPS 2019	use unlabeled data
3	Expressive Losses for Verified Robustness via Convex Combinations	63.78%	Deterministic	ICLR 2024	Approach name: CC-IBP
4	Certified Training: Small Boxes are All You Need	62.84%	Deterministic	ICLR 2023	Approach name: SABR
5	IBP Regularization for Verified Adversarial Robustness via Branch-and-Bound	61.97%	Deterministic	ICML 2022 Workshop on Formal Verification of Machine Learning
6	Adversarial training and provable defenses: bridging the gap	60.5%	Deterministic	ICLR 2020
7	Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective	56.94%	Deterministic	NeurIPS 2022	SortNet + MLP
8	Towards Evaluating and Training Verifiably Robust Neural Networks	56.63%	Deterministic	CVPR 2021
9	Robustra: training provable robust neural networks over reference adversarial space	56.32%	Deterministic	IJCAI 2019
10	Boosting the certified robustness of L-infinity distance nets	54.12%	Deterministic	ICLR 2022
11	Towards stable and efficient training of verifiably robust neural networks	53.97%	Deterministic	ICLR 2020
12	Scaling provable adversarial defenses	53.89%	Deterministic	NeurIPS 2018
13	Fast certified robust training with short warmup	52.85%	Deterministic	NeurIPS 2021
14	Scalable verified training for provably robust image classification	50.02%	Deterministic	ICCV 2019	Train on TPU

Showing 1 to 14 of 14 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Raising the Bar for Certified Adversarial Robustness with Diffusion Models	41.78%	Deterministic	*preprint	SortNet with EDM-generated data
2	Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective	40.39%	Deterministic	NeurIPS 2022	SortNet
3	Boosting the certified robustness of L-infinity distance nets	40.06%	Deterministic	ICLR 2022
4	Expressive Losses for Verified Robustness via Convex Combinations	35.44%	Deterministic	ICLR 2024	Approach name: MTL-IBP
5	Towards certifying L-infinity robustness using neural networks with L-inf-dist neurons	35.42%	Deterministic	ICML 2021
6	Certified Training: Small Boxes are All You Need	35.13%	Deterministic	ICLR 2023	Approach name: SABR
7	Fast certified robust training with short warmup	34.97%	Deterministic	NeurIPS 2021
8	Towards Evaluating and Training Verifiably Robust Neural Networks	34.92%	Deterministic	CVPR 2021
9	Automatic perturbation analysis for scalable certified robustness and beyond	33.38%	Deterministic	NeurIPS 2020
10	Towards stable and efficient training of verifiably robust neural networks	33.06%	Deterministic	ICLR 2020
11	Scalable verified training for provably robust image classification	32.04%	Deterministic	ICCV 2019	Train on TPU
12	Robustra: training provable robust neural networks over reference adversarial space	25.13%	Deterministic	IJCAI 2019
13	Scaling provable adversarial defenses	21.78%	Deterministic	NeurIPS 2018

Showing 1 to 13 of 13 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Expressive Losses for Verified Robustness via Convex Combinations	26.39%	Deterministic	ICLR 2024	Approach name: CC-IBP
2	Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective	18.18%	Deterministic	NeurIPS 2022	SortNet+MLP (2x larger)
3	Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond	15.86%	Deterministic	NeurIPS 2020
4	Scalable verified training for provably robust image classification	14.85%	Deterministic	ICCV 2019	WideResNet, reported by Auto-LiRPA paper

Showing 1 to 4 of 4 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Expressive Losses for Verified Robustness via Convex Combinations	13.30%	Deterministic	ICLR 2024	Approach name: Exp-IBP
2	Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective	9.54%	Deterministic	NeurIPS 2022	SortNet+MLP (2x larger)
3	Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond	8.73%	Deterministic	NeurIPS 2020
4	Scalable verified training for provably robust image classification	6.13%	Deterministic	ICCV 2019	based on arXiv version (1810.12715)

Showing 1 to 4 of 4 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Provably robust deep learning via adversarially trained smoothed classifiers	38.2%	Probabilistic	NeurIPS 2019	translated from $ℓ_{2}$ certification, combine adversarial training, self-training, and unlabeled data
2	Certified adversarial robustness via randomized smoothing	28.6%	Probabilistic	ICML 2019	translated from $ℓ_{2}$ certification

Showing 1 to 2 of 2 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Provably Adversarially Robust Nearest Prototype Classifiers	73.0%	Deterministic	ICML 2022	Non-neural-network approach
2	SmoothMix: training confidence-calibrated smoothed classifiers for certified robustness	70.7%	Probabilistic	NeurIPS 2021	under larger attack radius $ϵ = 1.75$
3	Consistency regularization for certified robustness of smoothed classifiers	70.5%	Probabilistic	NeurIPS 2020	under larger attack radius $ϵ = 1.75$
4	Second-Order Provable Defenses against Adversarial Attacks	69.79%	Deterministic	ICML 2020	! a loose certification --- certified accuracy against only the class with second largest logit, but not all non-ground-truth classes; non-relu neural networks
5	Certified adversarial robustness with additive noise	69.0%	Probabilistic	NeurIPS 2019
6	Globally-robust neural networks	62.8%	Deterministic	ICML 2021
7	Lipschitz-certifiable training with a tight outer bound	47.95%	Deterministic	NeurIPS 2020
8	Scaling provable adversarial defenses	44.53%	Deterministic	NeurIPS 2018

Showing 1 to 8 of 8 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	A Recipe for Improved Certifiable Robustness: Capacity and Data	78.1%	Deterministic	*preprint
2	Scaling in Depth: Unlocking Robustness Certification on ImageNet	70.1%	Deterministic	NeurIPS 2023	With data-agumentation using a DDPM model. Without the DDPM augmentation for the dataset, it reports 66.9%.
3	Raising the Bar for Certified Adversarial Robustness with Diffusion Models	69.05%	Deterministic	*preprint	LOT with EDM-generated data
4	Certified adversarial robustness with additive noise	65.6%	Probabilistic	NeurIPS 2019
5	LOT: Layer-wise Orthogonal Training on Improving l2 Certified Robustness	64.49%	Deterministic	NeurIPS 2022
6	Householder activations for provable robustness against adversarial attacks	62.96%	Deterministic	ICLR 2022
7	Orthogonalizing convolutional layers with the cayley transform	59.16%	Deterministic	ICLR 2021
8	Globally-robust neural networks	58.4%	Deterministic	ICML 2021	"Scaling in Depth: Unlocking Robustness Certification on ImageNet" reproduced 60.0%
9	Scaling provable adversarial defenses	51.96%	Deterministic	NeurIPS 2018
10	Lipschitz-certifiable training with a tight outer bound	51.30%	Deterministic	NeurIPS 2020

Showing 1 to 10 of 10 entries

Many records from CIFAR-10,

ℓ_{2}

,

ϵ = 0.25

imply stronger baselines in this setting. To avoid duplication, these stronger baselines are not listed here.

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Provably robust deep learning via adversarially trained smoothed classifiers	81%	Probabilistic	NeurIPS 2019	combine adversarial training, self-training, and unlabeled data
2	(Certified!!) Adversarial Robustness for Free!	79.3%	Probabilistic	ICLR 2023	based on denoising with large diffusion models
3	DensePure: Understanding Diffusion Models towards Adversarial Robustness	76.6%	Probabilistic	ICLR 2023	based on denoising with large diffusion models
4	Unlabeled data improves adversarial robustness	72%	Probabilistic	NeurIPS 2019	parse from Figure 1(a)
5	Macer: attack-free and scalable robust training via maximizing certified radius	71%	Probabilistic	ICLR 2020
6	Boosting Randomized Smoothing with Variance Reduced Classifiers	70.6%	Probabilistic	ICLR 2022	From Table 8
7	On the certified robustness for ensemble models and beyond	70.4%	Probabilistic	ICLR 2022
8	A Recipe for Improved Certifiable Robustness: Capacity and Data	69.5%	Deterministic	*preprint
9	Robust and Accurate -- Compositional Architectures for Randomized Smoothing	69.0%	Probabilistic	*preprint	parse from Table 8
10	Consistency regularization for certified robustness of smoothed classifiers	68.8%	Probabilistic	NeurIPS 2020
11	SmoothMix: training confidence-calibrated smoothed classifiers for certified robustness	67.9%	Probabilistic	NeurIPS 2021
12	Certified adversarial robustness via randomized smoothing	60%	Probabilistic	ICML 2019	parse from Figure 6 (top)

Showing 1 to 12 of 12 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Provably robust deep learning via adversarially trained smoothed classifiers	40%	Probabilistic	NeurIPS 2019	combine adversarial training, self-training, and unlabeled data
2	On the certified robustness for ensemble models and beyond	39.5%	Probabilistic	ICLR 2022
3	Boosting Randomized Smoothing with Variance Reduced Classifiers	38.8%	Probabilistic	ICLR 2022	From Table 8
4	Macer: attack-free and scalable robust training via maximizing certified radius	38%	Probabilistic	ICLR 2020
5	Consistency regularization for certified robustness of smoothed classifiers	37.8%	Probabilistic	NeurIPS 2020
6	Robust and Accurate -- Compositional Architectures for Randomized Smoothing	37.8%	Probabilistic	*preprint	parse from Table 10
7	DensePure: Understanding Diffusion Models towards Adversarial Robustness	37.4%	Probabilistic	ICLR 2023	based on denoising with large diffusion models
8	SmoothMix: training confidence-calibrated smoothed classifiers for certified robustness	37.2%	Probabilistic	NeurIPS 2021
9	(Certified!!) Adversarial Robustness for Free!	35.5%	Probabilistic	ICLR 2023	based on denoising with large diffusion models
10	A Recipe for Improved Certifiable Robustness: Capacity and Data	35.1%	Deterministic	*preprint
11	Certified adversarial robustness via randomized smoothing	22%	Probabilistic	ICML 2019	parse from Figure 6 (top)

Showing 1 to 11 of 11 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	On the certified robustness for ensemble models and beyond	20.3%	Probabilistic	ICLR 2022
2	Boosting Randomized Smoothing with Variance Reduced Classifiers	19.8%	Probabilistic	ICLR 2022	From Table 8
3	Consistency regularization for certified robustness of smoothed classifiers	19.5%	Probabilistic	NeurIPS 2020
4	Provably robust deep learning via adversarially trained smoothed classifiers	19%	Probabilistic	NeurIPS 2019	combine adversarial training, self-training, and unlabeled data
5	Macer: attack-free and scalable robust training via maximizing certified radius	19%	Probabilistic	ICLR 2020

Showing 1 to 5 of 5 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	DensePure: Understanding Diffusion Models towards Adversarial Robustness	67.0%	Probabilistic	ICLR 2023	based on denoising with large diffusion models
2	(Certified!!) Adversarial Robustness for Free!	54.3%	Probabilistic	ICLR 2023	based on denoising with large diffusion models
3	Provably robust deep learning via adversarially trained smoothed classifiers	45%	Probabilistic	NeurIPS 2019	combine adversarial training, self-training, and unlabeled data
4	Boosting Randomized Smoothing with Variance Reduced Classifiers	44.6%	Probabilistic	ICLR 2022	From Table 8
5	On the certified robustness for ensemble models and beyond	44.4%	Probabilistic	ICLR 2022
6	Consistency regularization for certified robustness of smoothed classifiers	44%	Probabilistic	NeurIPS 2020
7	SmoothMix: training confidence-calibrated smoothed classifiers for certified robustness	43%	Probabilistic	NeurIPS 2021
8	Macer: attack-free and scalable robust training via maximizing certified radius	43%	Probabilistic	ICLR 2020
9	Robust and Accurate -- Compositional Architectures for Randomized Smoothing	42.2%	Probabilistic	*preprint	parse from Table 1
10	Certified adversarial robustness via randomized smoothing	39%	Probabilistic	ICML 2019	parse from Figure 6 (bottom)
11	Black-box certification with randomized smoothing: A functional optimization based framework	39%	Probabilistic	NeurIPS 2020
12	Scaling in Depth: Unlocking Robustness Certification on ImageNet	14.20%	Deterministic	NeurIPS 2023

Showing 1 to 12 of 12 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	DensePure: Understanding Diffusion Models towards Adversarial Robustness	42.2%	Probabilistic	ICLR 2023	based on denoising with large diffusion models
2	On the certified robustness for ensemble models and beyond	30.4%	Probabilistic	ICLR 2022
3	(Certified!!) Adversarial Robustness for Free!	29.5%	Probabilistic	ICLR 2023	based on denoising with large diffusion models
4	Boosting Randomized Smoothing with Variance Reduced Classifiers	28.6%	Probabilistic	ICLR 2022	From Table 8
5	Provably robust deep learning via adversarially trained smoothed classifiers	28%	Probabilistic	NeurIPS 2019	combine adversarial training, self-training, and unlabeled data
6	Macer: attack-free and scalable robust training via maximizing certified radius	27%	Probabilistic	ICLR 2020
7	SmoothMix: training confidence-calibrated smoothed classifiers for certified robustness	26%	Probabilistic	NeurIPS 2021
8	Consistency regularization for certified robustness of smoothed classifiers	24%	Probabilistic	NeurIPS 2020
9	Black-box certification with randomized smoothing: A functional optimization based framework	21%	Probabilistic	NeurIPS 2020
10	Certified adversarial robustness via randomized smoothing	19%	Probabilistic	ICML 2019	parse from Figure 6 (bottom)

Showing 1 to 10 of 10 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Improved, deterministic smoothing for l1 certified robustness	63.07%	Deterministic	ICML 2021
2	Randomized smoothing of all shapes and sizes	63%	Probabilistic	ICML 2020	combine adversarial training, self-training, and unlabeled data
3	L1 adversarial robustness certificates: a randomized smoothing approach	39%	Probabilistic	*preprint	parse from Figure 6
4	Black-box certification with randomized smoothing: A functional optimization based framework	34%	Probabilistic	NeurIPS 2020
5	Certified robustness to adversarial examples with differential privacy	18%	Probabilistic	S&P 2019	parse from Figures 7 and 8 of this paper since the original paper does not provide certified accuray under this setting

Showing 1 to 5 of 5 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Improved, deterministic smoothing for l1 certified robustness	51.33%	Deterministic	ICML 2021
2	Randomized smoothing of all shapes and sizes	48%	Probabilistic	ICML 2020	combine adversarial training, self-training, and unlabeled data
3	Black-box certification with randomized smoothing: A functional optimization based framework	17%	Probabilistic	NeurIPS 2020
4	L1 adversarial robustness certificates: a randomized smoothing approach	16%	Probabilistic	*preprint	parse from Figure 6
5	Certified robustness to adversarial examples with differential privacy	5%	Probabilistic	S&P 2019	parse from Figures 7 and 8 of this paper since the original paper does not provide certified accuray under this setting

Showing 1 to 5 of 5 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Randomized smoothing of all shapes and sizes	55%	Probabilistic	ICML 2020	combine adversarial training, self-training, and unlabeled data
2	Improved, deterministic smoothing for l1 certified robustness	49%	Deterministic	ICML 2021
3	Black-box certification with randomized smoothing: A functional optimization based framework	42%	Probabilistic	NeurIPS 2020
4	L1 adversarial robustness certificates: a randomized smoothing approach	40%	Probabilistic	*preprint	parse from Figure 6
5	Certified robustness to adversarial examples with differential privacy	25%	Probabilistic	S&P 2019	parse from Figures 7 and 8 of this paper since the original paper does not provide certified accuray under this setting

Showing 1 to 5 of 5 entries

Search:

Rank	Paper Name	Reported Certified Accuracy	Certification Type	Venue	Comment
1	Randomized smoothing of all shapes and sizes	48%	Probabilistic	ICML 2020	combine adversarial training, self-training, and unlabeled data
2	Improved, deterministic smoothing for l1 certified robustness	45%	Deterministic	ICML 2021
3	Black-box certification with randomized smoothing: A functional optimization based framework	30%	Probabilistic	NeurIPS 2020
4	L1 adversarial robustness certificates: a randomized smoothing approach	26%	Probabilistic	*preprint	parse from Figure 6
5	Certified robustness to adversarial examples with differential privacy	16%	Probabilistic	S&P 2019	parse from Figures 7 and 8 of this paper since the original paper does not provide certified accuray under this setting

Showing 1 to 5 of 5 entries

MNIST, ℓ∞, ϵ=0.1

MNIST, ℓ∞, ϵ=0.3

CIFAR-10, ℓ∞, ϵ=2/255

CIFAR-10, ℓ∞, ϵ=8/255

TinyImageNet, ℓ∞, ϵ=1/255

ImageNet 64x64, ℓ∞, ϵ=1/255

ImageNet, ℓ∞, ϵ=1/255

MNIST, ℓ2, ϵ=1.58

CIFAR-10, ℓ2, ϵ=36/255

CIFAR-10, ℓ2, ϵ=0.25

CIFAR-10, ℓ2, ϵ=1.0

CIFAR-10, ℓ2, ϵ=2.0

ImageNet, ℓ2, ϵ=1.0

ImageNet, ℓ2, ϵ=2.0

CIFAR-10, ℓ1, ϵ=1.0

CIFAR-10, ℓ1, ϵ=2.0

ImageNet, ℓ1, ϵ=1.0

ImageNet, ℓ1, ϵ=2.0