# Attention is All You Need? Good Embeddings with Statistics ARE enough

Audio Understanding WITHOUT Convolutions/Transformers/BERTs/Mixers/Attention/RNNs or ….

###### Abstract

This paper presents a way of doing large scale audio understanding without traditional state of the art neural architectures. Ever since the introduction of deep learning for understanding audio signals in the past decade, convolutional architectures have been able to achieve state of the art results surpassing traditional hand-crafted features. In the recent past, there has been a similar shift away from traditional convolutional and recurrent neural networks towards purely end-to-end Transformer architectures. We, in this work, explore an approach, based on Bag-of-Words model. Our approach does not have any convolutions, recurrence, attention, transformers or other approaches such as BERT. We utilize micro and macro level clustered vanilla embeddings, and use a MLP head for classification. We only use feed-forward encoder-decoder models to get the bottlenecks of spectral envelops, spectral patches and slices as well as multi-resolution spectra. A classification head (a feed-forward layer), similar to the approach in SimCLR is trained on a learned representation. Using simple codes learned on latent representations, we show how we surpass traditional convolutional neural network architectures, and come strikingly close to outperforming powerful Transformer architectures. This work hopefully would pave way for exciting advancements in the field of representation learning without massive, end-to-end neural architectures.^{1}^{1}1The author would like to thank Prof. J. K. Verma for help in proof reading the manuscript and the Department of Mathematics, IIT Bombay for providing facilities, and Prof. Stephen Boyd’s lectures for motivating this work.

Prateek Verma
\addressStanford University

450 Jane Stanford Way, Stanford CA, 94305

Representation learning, acoustic scene analysis, multi-scale representations, attention, k-means clustering, bag-of-words, envelop codes, fully connected networks.

## 1 Introduction

Audio Understanding is a widely studied problem in domains such as signal processing, machine learning and perception. It is to teach computers to hear like humans do, and the goal to help develop human level perception in machines. With the advancement in machine learning, there has been rapid progress in making these system reach almost human level performance. For the purpose of this paper, we would pose the problem as to understand the categories of sound present in a input signal. CNN architectures have become a standard way of learning these mappings, as shown in [1]. Further, with the recent success of Transformer architectures in computer vision [2], nlp [3], and audio [4, 5], there has been a pivot recently on improvements on the core architecture, and adapting it for acoustic scene understanding. There have been in the past approaches inspired from natural language processing such as computing a bag-of-words model on features such as MFCC [6] yielding surprising strong results. We in this work take inspiration from traditional bag-of-words model along with powerfulness of neural net models, taking in some ways the best of two worlds. The current work is also based on the idea of learning code book and doing some statistics on these learned code words as show in [7], [8], [9]. In problems such as unsupervised learning/one-shot learning, the goal is to learn a latent representation for an input signal, and use this latent representation for a variety of application using a classification head. [10]. Audio embeddings have been powerful to aid in a variety of applications such as speech recognition and audio understanding [11], [12], conditional audio synthesis [12] as well as transformation [13], [14]. This work also devises a way of coming up with coming up with a latent vector summarizing the contents of the audio signal, and then using a classification head similar to [10] for classification purposes. This work also draws inspiration of idea of learning mapping from a signal of interest to a clustered latent space as shown in [15]. Unlike their work which utilized UMAP [16], we use coded book statistics as a representation of input mel-spectogram.

The contributions of the paper are as follows: i) We propose a framework purely based on learned embedding and statistics based on it and achieve significant performance as compared to traditional convolutional and transformer architectures. We achieve this without having any convolutional, transformer, mixer, attention or recurrent blocks. ii) We compute statistics (bag-of-words [6]) over dictionary learned over various latent representations of mel-spectogram from vanilla autoencoders. We capture various facets of audio signals, and learn a concatenated dictionary of spectral patches, spectral envelop, frequency band envelop and overall statistics. iii) We show how we can improve the performance of our model by randomly masking the input signal, making them robust by drawing inspiration from approaches such as BERT [17].

## 2 Dataset

For evaluation of our approach, we use Free-Sound 50K dataset [18] instead of widely used AudioSet [19]. It is an open dataset having over 100 hours of manually labelled samples drawn from AudioSet ontology of 200 classes. This was done, as the audio samples are available as opposed to YouTube links, and the readers are advised to read [18] for advantages of it over AudioSet [19]. One of the significant advantage is that it contains twice the same number of training examples in the balanced setup as AudioSet. We used the training, validation and test splits as as given. The dataset has in total of about 51,197 clips. We have down-sampled all the clips to be of 16kHz. We may also mention that we have been consistent in the training and the validation setup while reporting the mean-average precision (mAP) scores as reported in [18]. Training on longer context will result in improved performance as well as augmentation etc. The training was done on 1s of audio, with the label of the clip assigned to all of the 1s patches in case the clip is longer. Once the model is trained, the mAP scores are reported at clip level, with the probability scores averaged on 1s chunks to predict the contents (possible classes out of 200). Thus we can compare our performance with the baseline convolutional (DenseNet, Res-Net, VGGish) [18] and Transformer architectures [4].

## 3 Methodology

This section describes the methodology of how we obtained a feature vector from a mel-spectogram input. Briefly, for every input representation, the system learns a latent representation by fully-connected vanilla auto-encoders with varying degrees of compression factor . Once we obtain these codes, we cluster them according to a fixed number of dictionary elements having the vocabulary , assigning each input representation a discrete value from to , according to the cluster to which it belongs. We obtain overall statistics of the counts of each of the dictionary elements similar to a bag-of-words (BOW) model. Multiple representations are concatenated, to obtain a final feature code. This code is used with MLP classification head similar to [10].

### 3.1 Learning Latent Representations

We choose to work with mel-spectogram input having a total of 96 bins encompassing a total range from 0 to 8kHz, as our input signals are 16 kHz. We choose a hop size of 10ms, with a window of 30ms with FFT size to be 2048, with a hanning window. For learning a latent representation given an input, we deploy a three layer MLP encoder with a bottleneck layer of size to be reduced by a factor of from the input dimension. For this paper, we experiment with compression factor and . This bottleneck is then again passed onto a 3 layer MLP decoder with the goal to reconstruct back the same input. We used mean-square error criteria for training this auto-encoder block, with 2048 neurons of encoder and decoder with a dropout factor of 0.5. We call different encoder representations learned at a particular compression factor , to be , , , for an encoding learned for a spectral patch, spectral envelop, frequency band energy across time, and scaled down-sampled mel-spectra.

#### 3.1.1 Representation of Spectral Patches

To extract embeddings from spectral patches, we draw inspiration from recent works in Audio Transformer [4], and Vision Transformer [2]. We take patches of size 8 frequency bins along the frequency axis, with 10 bins along time, randomly across the mel-spectogram which has input dimensions of 96 x 100 for each of the 1s inputs, to obtain a patch level embedding . This would capture the overall local variations present in an input mel-spectogram. For a input spectra, we divide it in a grid form and obtain 120 patch level embeddings (96*100 /(8*10))

#### 3.1.2 Representation of Frequency Band Envelop

We randomly take patches of 8 mel-bins across time to model energy modulations at different frequency bands across time. This can capture variations that can be thought of as behaviour of energy in a frequency band across time. For every input spectogram, 8 such patches are chosen having input dimension of 8 x 100 and obtain a bottleneck feature . For any input spectra, we thus obtain a total of 12 input bottleneck features, obtained by equally dividing the frequency axis.

#### 3.1.3 Representation of Spectral Envelop

Similar to frequency band envelope, we learn embeddings to characteristics of spectral envelop across time. For this we take patches across 100ms encompassing the entire frequency spread of 96 mel-bins, thus having the dimension of 96 x 10. Again these patches are randomly chosen and we obtain a bottleneck feature . We get a total of 10 bottleneck vectors for any given input spectra.

#### 3.1.4 Representation of the Whole Spectogram

Finally, given an input mel-spectogram, we obtain its global characteristics. This is done by down-sampling and resizing the input spectogram from 96x100 to 8x10, done by using bi-cubic interpolation. This is mainly done to capture at a very coarse level how the spectogram ”looks”. We again follow a similar recipe of obtaining a bottleneck feature, and call it . We obtain only 1 bottleneck vector for every input spectra, corresponding to its corresponding code-word out of . Thus, in total we obtain 143 latent vectors (120+12+10+1) that describes input spectogram at various levels.

### 3.2 Clustering and Code Book Generation, and getting embeddings

For each family of the bottleneck features obtained, we do k-means clustering independently with number of cluster fixed as D, which is a hyper-parameter. This can be easily implemented using popularly available libraries e.g. sklearn. For any given spectogram, the goal is to obtain a discrete feature representation that can capture all of the features that we have explained above. Given any input spectra, we first obtain a total of 143 latent codes as described above at a chosen bottleneck factor . These embeddings are clustered using k-means separately using each of the family to which they belong, i.e. we cluster separately from . For each of the learned k-means mapping we obtain a feature vector of dimension , similar to a bag-of-words representation i.e. each feature vector contains the counts of how many times the embeddings falls into the bucket of the cluster centroids. Thus we obtain a dimension vector for each of the four categories mentioned thus obtaining, , , and . Each of these features , contains counts of how often the code-word of embedding from the audio spectogram is present in a given 1s input. As expected the sum of the feature vector would be 143 equal to the total number of embeddings we have from the four family of features.

### 3.3 Classification Setup & Input Code Masking

We concatenate all of the feature vectors , , and to obtain a feature vector for a particular input spectogram. This is a dimensional input, and is used along with the label present in the dataset. As proposed in work by [10], we employ just MLP classification heads. Since we have a multi-label classification problem, use sigmoid as our final activation function, with Huber loss as our error criterion between the actual and the predicted samples. We explore to see how we can improve the performance, and improve the robustness further by randomly masking input features by a chosen probability . This is as simple as replacing the actual count of the feature by 0. This makes our predictions more robust to outliers and as we will see improves the performance of our models.

## 4 Results

Given a feature vector of dimension equal to 4*dimension of the number of code words chosen , we have as our input of dimension . As described in the previous section is concatenation of individual feature vectors obtained from all of the four features. We deploy same strategy as what proposed for classification head given a feature vector in [10]. We experiment with several combinations of hyper-parameters namely the size of the feed-forward classification head i.e. small namely 2-layer 256 dimensional fully connected architecture and large namely 2-layer 4096 dimensional model. Note that [10] used a 2-layer 2048 neuron MLP architecture. This was carried out for all 4 different sizes of code-book : 16, 64, 256, 1024. For all of the proposed models as described above, we tweak the dropout rate as a form of regularization with and . The results are as shown in the figure below.

We see from the plot above that the best mAP score is for learned 256 code words, with the classification head having 512 neurons in each layer with dropout rate of 0.4 and a compression factor of 10. The choice of intermediate number of code words makes sense: Too few of the code-words may not be able to capture all of the variations present in the data, and too large of code words yields to over-fitting and learning finer unnecessary details. The same holds true as expected for the size of classification head, smaller models yielding much better performance as opposed to a large model possibly over-fitting the dataset.

Now with the best performing architecture keeping it fixed, we choose to improve the performance of the models even further by making models robust to errors/outliers in various input codes. Here, we randomly drop input features by a probability and see the performance, in terms of mean average precision. Masking of inputs is not new, and has been used in various contexts. As shown in BERT [17], it adds robustness of the learned embeddings. We see that there exists an optimum value of the number of tokens one should drop: too many tokens being dropped and we loose valuable information. Too few tokens dropped and we do not build the model agnostic to outliers. We are perhaps the first ones to introduce this idea in a bag-of-words setup. We obtain the best overall performance of about 0.38 mAP score with 35% randomly dropped tokens. We also would like to note that there can exist many more values/parameters that can be tuned. Finally, in Table 1 we report clip level scores which are obtained by averaging the probability scores as reported in [18]. We see that we come strikingly close to state of the art Transformer models.

We report the best performing model with that of traditional convolutional neural architectures and Transformer models. We see that we achieve comparable performance with our setup which is fascinating in itself. However the goal of this work is to show how we can obtain significant comparable performance without utilizing any convolutional, transformer, attention, rnns or mixer blocks [20]. In future it will be exciting to see similar techniques being deployed in various applications where there is some sort of conditioning needed for controlled synthesis and end to end audio transforms [21], [12].

Neural Model Architecture | mAP |
---|---|

CRNN [18] | 0.41 |

VGG-like [18] | 0.43 |

ResNet-18 [18] | 0.37 |

DenseNet-121 [18] | 0.42 |

Small Transformer [4] | 0.46 |

Ours | 0. 44 |

## 5 Conclusion and Future Work

We have shown the power of vanilla embeddings followed by code-book learning to yield representations in clustered space of the signal of interest. We achieve this by modelling the envelops across different frequency bins, patches as well as the overall spectogram at coarser scale. This idea can be further strengthened in future by deploying more sophisticated clustering algorithms such as UMAP [16]. Due to limited amount of computing resources, we could not achieve large scale hyper-parameter tuning beyond what is reported, although it would certainly help. It will also be interesting to see if embeddings derived from approach like neural discrete representation learning [13] or sparse coding perform better. Further, achieving good clustering performance in unbalanced dataset remains an active area of research.

## References

- [1] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al., “Cnn architectures for large-scale audio classification,” in 2017 ieee international conference on acoustics, speech and signal processing (icassp). IEEE, 2017, pp. 131–135.
- [2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
- [3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” arXiv preprint arXiv:1706.03762, 2017.
- [4] Prateek Verma and Jonathan Berger, “Audio transformers: Transformer architectures for large scale audio understanding. adieu convolutions,” arXiv preprint arXiv:2105.00335, 2021.
- [5] Prateek Verma and Chris Chafe, “A generative model for raw audio using transformer architectures,” arXiv preprint arXiv:2106.16036, 2021.
- [6] Stephanie Pancoast and Murat Akbacak, “Bag-of-audio-words approach for multimedia event classification,” Tech. Rep., SRI International Menlo Park United States, 2012.
- [7] Prateek Verma and Julius Smith, “A framework for contrastive and generative learning of audio representations,” arXiv e-prints, pp. arXiv–2010, 2020.
- [8] Alexei Baevski, Steffen Schneider, and Michael Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” arXiv preprint arXiv:1910.05453, 2019.
- [9] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv preprint arXiv:2006.11477, 2020.
- [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton, “A simple framework for contrastive learning of visual representations,” arXiv preprint arXiv:2002.05709, 2020.
- [11] Prateek Verma, Chris Chafe, and Jonathan Berger, “Neuralogram: A deep neural network based representation for audio signals,” arXiv preprint arXiv:1904.05073, 2019.
- [12] Albert Haque, Michelle Guo, Prateek Verma, and Fei-Fei Li, “Audio-linguistic embeddings for spoken sentences,” in 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 7355–7359.
- [13] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu, “Neural discrete representation learning,” arXiv preprint arXiv:1711.00937, 2017.
- [14] Prateek Verma and Julius O Smith, “Neural style transfer for audio spectograms,” arXiv preprint arXiv:1801.01589, 2018.
- [15] Prateek Verma and Kenneth Salisbury, “Unsupervised learning of audio perception for robotics applications: Learning to project data to t-sne/umap space,” arXiv preprint arXiv:2002.04076, 2020.
- [16] Leland McInnes, John Healy, and James Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018.
- [17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
- [18] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra, “Fsd50k: an open dataset of human-labeled sound events,” arXiv preprint arXiv:2010.00475, 2020.
- [19] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 776–780.
- [20] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al., “Mlp-mixer: An all-mlp architecture for vision,” arXiv preprint arXiv:2105.01601, 2021.
- [21] Albert Haque, Michelle Guo, and Prateek Verma, “Conditional end-to-end audio transforms,” arXiv preprint arXiv:1804.00047, 2018.