A) Categorization on deep neural networks

Deep leaning techniques refer to a class of machine learning techniques, where many layers of information processing stages in hierarchial architectures are exploited for unsupervised feature learning and for supervised pattern analysis/classification. The core procedure of deep learning is to compute hierarchical features or representations from the observed data, where the higher-level features or factors are defined from lower-level data structure. Deep learning techniques can be generally divided into three dimensions, i.e., generative deep architectures, discriminative deep architectures and hybrid deep architectures [Reference Deng30]. Typical architectures include Deep Belief Network (DBN) [Reference Hinton, Osindero and Teh31], Boltzmann Machine (BM) [Reference Ackley, Hinton and Sejnowski32], Restricted Boltzmann Machine (RBM) [Reference Smolensky33], Deep Auto-Encoder (DAE) [Reference Bengio, Lamblin, Popovici and Larochelle34], CNN [Reference Krizhevsky, Sutskever and Hinton13], and so on. Among these architectures, CNNs have shown superior performance on the learning of discriminative image feature representation, and thus are widely-used in image classification and annotation tasks. In the following, we will specify the network structure of a typical CNN.

B) Convolutional neural networks

A typical CNN often consists of several convolutional and fully-connected layers. The exact number of layers generally depends on the requirement of network capacity and memory cost for a specific classification task (e.g., eight layers for AlexNet, and 22 layers for GoogleNet). In particular, let ${\bf X}=[{\bf x}_{1},\ldots, {\bf x}_{N}]$ be the matrix of image training data, where x _i is the feature vector of the ith image. N is the total number of images. Denote ${\bf Y} = [{\bf y}_{1},\ldots,{\bf y}_{N}]^{T} \in \{0,1\}^{N \times K}$ , where ${\bf y}_{i} \in \{0,1\}^{K \times 1}$ is the category indicator vector for x _i . K is the number of categories. Suppose there are M layers in total and ${\bf W} = \{{\bf W}^{(1)},\ldots,{\bf W}^{(M)}\}$ are the model parameters. In each layer, we absorb the bias term into the weights and denote them as a whole. ${\bf W}^{(m)}=[{\bf w}_{1}^{(m)},\ldots,{\bf w}_{d_{m}}^{(m)}]^{T}\in R^{d_{m}\times d_{m-1}}$ , where ${\bf w}_{i}^{(m)} \in R^{d_{m-1}}$ , d _m−1 is the dimension of the (m−1)th feature map. ${\bf Z}^{(m)}({\bf X}) = [{\bf z}^{(m)}({\bf x}_{i}),\ldots,{\bf z}^{(m)}({\bf x}_{N})]^{T} \in R^{N \times d_{m}}$ denotes the feature map produced by the mth layer.

Given an image x _i , convolutional layers first task the image as input. The extracted deep representations are denoted as ${\bf W}\ast{\bf x}_{i}$ , where * denotes a set of operations of convolution, polling, and activation, and W denotes the overall parameters for convolutional and pooling operations. The pooling layer in a CNN is designed to summarize the outputs of neighboring groups of neurons in the same kernel map, which can be conducted by mean-pooling or max-pooling. Some earlier works adopt the neighborhood summarization strategy by adjacently pooling without overlaps. To make it more clear, a pooling layer can be considered as consisting of a grid of pooling units spaced s pixels apart. Each pooling unit summarizes a neighborhood of size r×r centered at the location in a CNN. If s=r, we can obtain the non-overlapping polling. If s < r, we can obtain overlapping pooling. Extensive studies have shown that the overlapping pooling are difficult to overfit [Reference Krizhevsky, Sutskever and Hinton13].

For the activation function, the standard way to activate a neuron's output is through tan or sigmoid function, which are considered as saturating nonlinear functions. However, for the training optimization with gradient descent, these saturating nonlinear functions show much slower convergence than the non-saturating nonlinear function, such as max(0, x). The nonlinear function is referred to Rectified Linear Units (ReLUs). It has been demonstrated that deep CNNs with ReLUs can be trained several times faster than their equivalents with tan unites, which is crucial for the training of large models on large datasets.

Given the output from convolutional layers, we further feed it into a series of fully-connected layers. The output of the last fully-connected layer is considered as an input to a softmax classifier, which can generate a distribution over the final category labels, which is given by:

(1) $${\bf p}({\bf x}_i) = f({\bf W}\ast{\bf x}_i),$$

where f(·) represents fully-connected layers to map convolutional features to a feature vector that could be matched with the categories entries, as well as includes a softmax layer to further transform the feature vector to probabilities. The goal is to minimize the following objective function in the form of a softmax regression with weight decay, which is given by:

(2) $$\eqalign{ {\cal L}({\bf W}) = &-{1 \over N} \left[\sum_{i=1}^N\sum_{j=1}^K {\bf 1}_{Y_{ij}}(j)\log p(Y_{ij}=1|{\bf x}_i;{\bf W})\right] \cr &+ {\beta \over 2}||{\bf W}||_F, }$$

where Y _ij is the (i, j)th entry of ${\bf Y}$ . ${\bf 1}_{Y_{ij}}(j)$ is the indicator function such that ${\bf 1}_{Y_{ij}}(j)=1$ if Y _ij =1, otherwise zero. β is the coefficient of weight decay, which is designed to reduce model complexity and thus to prohibit overfitting. The probability p can be calculated by a softmax function, which is shown as:

(3) $$p(Y_{ij}=1|{\bf x}_i;{\bf W}) = {\exp({\bf z}^{(M-1)}_j) \over \sum_{k=1}^K \exp({\bf z}^{(M-1)}_k)}.$$

To optimize the above objective function, the derivatives to ${\bf w}^{(M)}_{j}$ in the output layer can be calculated as:

(4) $$\eqalign{ &{\partial {\cal L}({\bf W}) \over \partial {\bf w}^{(M)}_j} = -{1 \over N} \sum_{i=1}^N {\bf z}^{(M-1)}({\bf x}_i) \left[ {\bf 1}_{Y_{ij}}(j)\right. \cr &- p \left. \lpar Y_{ij}=1|{\bf z}^{(M-1)}({\bf x}_i);{\bf w}_j^{(M)}\rpar \right] + \beta {\bf w}^{(M)}_j. }$$

Parameters in other layers can be calculated by the back propagation algorithm (BP) [Reference Rumelhart, Hintont and Williams35]. Different CNN architectures have different numbers of convolutional layers and fully-connected layers. Detailed performance comparisons for different CNN model architectures on ImageNet challenges are shown in Table 1.

Table 1. The comparison of different CNN architectures on model size, classification error rate, and model depth.

“conv” and “fc” indicates convolutional and fully-connected layers, respectively.

A) Model-free image tagging with deep representation

The state-of-the-art model-free image tagging approaches adopt the powerful CNN features as image representations, which specifically replace traditional hand-crafted features with features that are automatically learned with a CNN. The CNN feature can be either extracted from pre-trained deep neural networks (e.g., GoogleNet [Reference Szegedy42], VGG [Reference Simonyan and Zisserman43], or ResNet [Reference He, Zhang, Ren and Sun44]) on ImageNet, or fine-tuned on the target tag vocabularies from target domain. Specifically, the feature vector in the last fully-connected layer, which is activated by a rectified activation function (e.g., ReLU), usually constitutes the final compact representations (e.g., 4096 dimensions) for an image.

Based on the deep representation, most of works adopt the idea of “collective intelligence,” which constitutes the tagging hypotheses by neighbor voting from the training instances. The specific procedure generally refers to tag propagation from visual-similar images. The instance voting-based models are non-parametric and even do not involve any training process. The complexity of this type of approaches depends on the amount of training data instances. A typical image tag refinement framework by instance neighbor voting is shown in Fig. 2. For example, Li et al. [Reference Li, Snoek and Worring40,Reference Li, Snoek and Worring45], Wang et al. [Reference Wang, Zhang, Jing and Ma41] and Guillaumin et al. [Reference Guillaumin, Mensink, Verbeek and Schmid46] propose to accumulate the neighbor votes received from the visual similar images of the input image for each tag. Later, to estimate the relevance between a tag and an input image, the approaches of content-based tag refinement [Reference Wang, Jing, Zhang and Zhang47] and tag ranking [Reference Liu, Hua, Yang, Wang and Zhang37] propose to use Gaussian function to measure the similarity between the input image and training images, which have already been annotated by the tag. This type of approaches also has a very intuitive explanation that tags of the input image can be determined by the soft voting from its visual neighbors from training set. Image retagging [Reference Liu, Hua, Wang and Zhang48] and low-rank-based image tag refinement approaches [Reference Zhu, Yan and Ma49], though effective by imposing some reasonable constraints (e.g., low rank, error sparsity, etc.), still rely on the idea of the instance neighbor voting.

Fig. 2. An example of model-free image tag refinement framework. For an input image on the left, we first find its semantic-related images in training set by searching with its initial image tagging results. Second, we build a star-graph from the semantic-related images based on visual similarity on the right. Both the semantic-related and visual-similar nearest neighbor images for the input image are marked by yellow rectangles in the left image list and the right star graph. The final tagging list for the input image is generated from the voting of those nearest-neighbor images marked by yellow rectangles. [Best viewed in color].

In summary, based on the strong representation ability of CNN features, the model-free based algorithms can outperform those model-based approaches if the number of training instances is large enough. However, since the training data are often collected from user-contributed image-tag pairs (e.g., Flickr), the assumption of model-free approaches that the tags provided by users are reasonably accurate is not justifiable. Previous research has reported that half of the user-contributed tags from Flickr are noises to image content [Reference Chua, Tang, Hong, Li, Luo and Zheng50]. Therefore, due to the limited size and heavy noises of the training set in real applications, incomplete and imprecise tags still hinder user from accessing to the images after the tagging/tag refinement process by these model-free approaches.

B) Model-based image tagging by deep neural network

Since the model-free approaches can only achieve limited performance on the limited training instances in real applications. To enhance the generalization ability of image tagging models, the second paradigm of image tagging approaches proposes to learn parameterized models from training data. One of the representative works from the early research of model-based concept representation is visual synset [Reference Tsai, Jing, Liu, Rowley, Ioffe and Rehg38], which shares similar motivation to the work of view-dependent concept representations [Reference Fu, Wang, Rui, Wang, Mei and Lu10]. Visual synset applies multi-class one-versus-all linear SVM [Reference Cortes and Vapnik51], which is learned from the automatically generated visual synsets from a large collection of web images. Compared with the visual synset, view-dependent approach [Reference Fu, Wang, Rui, Wang, Mei and Lu10] can discriminate different views and represent them with proper view-dependent representations. Specifically, model-based components for generic views are designed to enhance generalization ability, while model-free components for specific views act as complements since they are too sparse and diverse. Besides, view-dependent approach [Reference Fu, Wang, Rui, Wang, Mei and Lu10] is easily paralleled as the model-based and model-free components are learned per concept, with no dependency on each other. The view-dependent approach is more scalable, which can add new concept representations into the existing vocabulary rather than re-training the visual models on the whole vocabulary again compared to the methods using discriminative models, such as SVM. Moreover, the view-dependent approach collects larger vocabularies for image tagging with nearly 30% increases compared to visual synset.

To further relieve the problem of limited training instances, some pioneer works propose to leverage the integration of transfer learning schemes and deep learning techniques. Transfer deep learning targets at the transfer of knowledge from a source domain to a target domain using deep learning algorithms. A typical model-based image tagging framework by transfer deep learning is shown in Fig. 3. In [Reference Maxime, Leno, Ivan and Josef52], transfer learning problems are divided into two categories. One class tries to improve the classification performance of categories that have deficit training examples by discovering other similar categories and transferring knowledge among them [Reference Srivastava and Salakhutdinov53]. The second class of works aims to solve the different data distribution problems, so as to effectively transfer the knowledge learned from the source domain to the target [Reference Fu, Mei, Yang, Lu and Rui17,Reference Maxime, Leno, Ivan and Josef52].

Fig. 3. An example of model-based image tagging framework. (a) The training set contains the labeled source images and the unlabeled target images. (b) The network of the transfer deep learning with ontology priors. It is first trained on both ImageNet (the source domain) and personal photos (the target domain) by pre-training and fine-tuning for discovering shared middle-level feature abstractions across domains. Once the shared feature abstractions are learned, the top layer with ontology priors is further trained. In the testing stage, the resultant parameters W and B can be transferred to the target domain to obtain the middle-level feature representations (a bottom-up transfer) and high-level confidence scores (a top-down transfer). (c) An illustration of the ontology collecting scheme. (d) The input, in the testing stage, is highly flexible which can either be a single photo or a photo collection. (e) The tagging result.

However, although promising performance has been achieved from the earlier model-based approaches, the data noise problems existed in training instances are not considered for further improvement. To simultaneously learn better generalization model and noise-robust model, there are two schemes to adapt deep learning algorithms to noisy data. One type of approaches proposes to conduct preprocessing procedure to remove the noisy data before training. Other works propose to enable the deep learning network itself to identify the authenticity of labels and thus a robust model can be learned. The preprocessing methods can be implemented either by the conventional outlier detection, or by the pre-training strategy in deep learning frameworks. First, the specific methods in outlier detection include Principle Component Analysis (PCA), Robust PCA [Reference Candès, Li, Ma and Wright54], Robust Kernel PCA [Reference Xu, Caramanis and Mannor55], probabilistic modeling [Reference Kim and Scott56] and one-class SVM [Reference Liu, Hua and Smith57], and so on. These methods regard the outliers as those “few and different” samples, which usually locates far away from the sample centers in some learned embedding subspaces. Specifically, the samples which have a large discrepancy with other samples in the same category is considered to be noises. However, the challenge of these methods is to distinguish “hard samples” from the truly noisy samples. Those “hard samples” are correct image samples under a category, yet with large visual variances with the majority. Second, recovering the clean training samples by a layer-wise auto-encoder or denoising auto-encoder [Reference Vincent, Larochelle, Lajoie, Bengio and Manzagol58] in pre-training and then initializing a deep network by the pre-trained model parameters is an effective method to remove global noises, which has been used and extensively evaluated in face parsing [Reference Luo, Wang and Tang59]. However, these methods are mainly designed for the cases where noises are contained from parts of images (e.g., background noises), while web images (e.g., from Flickr) are often completely mislabeled.

To train a robust deep learning model on noisy training data, Larsen et al. propose one of the pioneer works which adds noise modeling into the neural networks [Reference Larsen, Andersen, Hintz-madsen and Hansen60]. However, they make a symmetric label noise assumption, which is often not true in real applications. Mnih et al. propose to label aerial images from noisy data, where only a binary classification is considered [Reference Mnih and Hinton61]. One of the promising works is proposed by Sukhbaatar et al. who introduce an extra noise layer as a part of training process in multi-class image classification [Reference Sukhbaatar and Fergus62]. They first train a base model on noisy training data for several iterations, then activate the extra noise layer to absorb the noise from the learned base model. Recently, a similar work to [Reference Sukhbaatar and Fergus62] has been proposed to handle the noise problems [Reference Fu, Wu, Mei, Wang, Lu and Rui21]. This work starts from embedding the feature map of the last deep layer into a new affinity representation. Second, by adopting the “few and different” assumption about the noises, this work minimizes the discrepancy between the affinity representation and its low-rank approximation. Third, this discrepancy is further transformed into the objective function to give those “few and different” noisy samples low-level authorities in training. Extensive experiments have shown superior performance than previous research.

Unlike above model-based learning, which learns concept representations for image tagging by cleaning image samples from noisy training instances, zero-shot learning takes one step further, which aims to learn tagging model from seen categories and tag an image from unseen categories [Reference Akata, Perronnin, Harchaoui and Schmid63–Reference Socher, Ganjoo, Manning and Ng68]. Attributes sharing and word embedding are the two main streams in zero-shot learning approaches. One of the typical approaches is from [Reference Fu, Yang, Hospedales, Xiang and Gong69], where the authors consider pre-fixed unseen tags with the number of U and learn a multi-label model to annotate the 2 ^U possible combinations between them. The limitation of this approach derives from the small number of unseen tags. Fast0Tag is recently proposed to enrich the family of zero-shot learning by zero-shot multi-label classification [Reference Zhang, Gong and Shah70]. They directly model the labels and propose to assign or rank many unseen tags for unseen images. To achieve better performance, deep learning features which are extracted from CNN model are indispensable in image modeling.

A) MODEL-FREE IMAGE TAGGING

We first select three typical model-free image tagging models for comparison, which include:

• – KNN [Reference Makadia, Pavlovic and Kumar74]: K - N earest- N eighbor based approaches generate a tag list by calculating the relevance of a tag to a given image. The procedure is conducted by first searching the top-K nearest neighbor images which have been annotated by the tag, and further ranking a tag list by tag frequency.

• – TagVoting [Reference Li, Snoek and Worring40]: Compared with KNN, Tag Voting based approaches leverage user-side information, and constrain each user has at most one image in the neighbor set. Besides, this approach considers tag prior frequency to limit the occurrence of frequent tags.

• – TagProp [Reference Guillaumin, Mensink, Verbeek and Schmid46]: Tag Prop agation based approaches employ neighbor voting with distance metric learning scheme into image tagging.

Table 3 shows that the tagging performance of different model-free image tagging approaches, by using either traditional hand-crafted features with bag-of-word (BoW) representations [Reference Sivic and Zisserman8] or deep learning features. We can observe from the table that deep learning based representation can achieve consistently better performance than hand-crafted features for the three typical model-free approaches, which demonstrates the powerful representation ability of deep learning techniques to image tagging. For different model-free tagging models, we can observe that KNN [Reference Makadia, Pavlovic and Kumar74] and TagVoting [Reference Li, Snoek and Worring40] can achieve comparable performance. TagProp [Reference Guillaumin, Mensink, Verbeek and Schmid46] can achieve better results, due to the proposed distance metric learning scheme. Note that the deep features used here are extracted from the pre-trained VGG-19 network [Reference Simonyan and Zisserman43] on ImageNet. Specifically fine-tuned features for this image tagging application can achieve further improvement.

Table 3. Image tagging performance (measured by MAP) on MIRFlickr-25K and NUS-WIDE-270K for different comparison approaches.

B) Model-based image tagging

We select the following typical model-based image tagging approaches for comparison:

• – CNN: C onvolutional N eural N etwork adopts the state-of-the-art network architecture with several convolutional layers and fully-connected layers [Reference Krizhevsky, Sutskever and Hinton13].

• – RPCA+CNN: R obust P rinciple C omponent A nalysis + CNN first removes samples with large reconstruction errors by RPCA [Reference Candès, Li, Ma and Wright54], and conducts CNN training on the cleaned samples.

• – CAE+CNN: C onvolutional A uto- E ncoder + CNN proposes to reduce the noise effect in CNN training by the layer-wise pre-training and fine-tuning strategy [Reference Luo, Wang and Tang59].

• – NA+CNN: N oise A daption layer + CNN proposes to add an additional bottom-up noise-adaption layer into the traditional CNN architecture for noise removal [Reference Sukhbaatar and Fergus62].

• – NR+CNN: N oise R obust layer + CNN designs an objective function to assign those “few and different” noisy samples with low-level authorities in training, and thus the noise effect can be reduced [Reference Fu, Wu, Mei, Wang, Lu and Rui21].

Each model is trained and evaluated on NUS-WIDE-270K [Reference Chua, Tang, Hong, Li, Luo and Zheng50], since the dataset contains the largest vocabulary size among different image tagging datasets.

Since the annotated tags for each image is limited (only 81 tags have ground truth), a randomly-selected 1000 photos from NUS-WIDE-270K [Reference Chua, Tang, Hong, Li, Luo and Zheng50] are used as the test set for extensive evaluation. Each method produces top five categories with the highest prediction scores as a tagging list. 25 human-labelers are employed to evaluate each tag with three levels: 2–Highly Relevant; 1–Relevant; 0–Non Relevant. We adopt both the mean average precision (MAP) and the Normalized Discounted Cumulative Gain (NDCG) as the metric to evaluate the tagging performance. Table 4 shows the tagging performance. We can observe that simple image classification architecture CNN [Reference Krizhevsky, Sutskever and Hinton13] cannot achieve good performance, since there are usually noises in image tagging datasets. For different noise-robust CNN architectures, CAE+CNN [Reference Luo, Wang and Tang59] proposes to reduce the noise effect by auto-encoder, which can achieve a slightly better result than simple CNN. Compared with CAE+CNN [Reference Luo, Wang and Tang59], RPCA+CNN [Reference Candès, Li, Ma and Wright54], and NA+CNN [Reference Sukhbaatar and Fergus62] can achieve significant improvement with a large margin in terms of NDCG. NR+CNN [Reference Fu, Wu, Mei, Wang, Lu and Rui21] achieves the best performance due to the effective noise robust layer proposed in their paper. Exemplary tagging results of NR+CNN [Reference Fu, Wu, Mei, Wang, Lu and Rui21] are shown in Fig. 4. We can also observe the vocabulary difference between image classification and image tagging, where the underlined tags in Fig. 4 are even missing from ImageNet categories, yet important to reflect the diverse visual content in image tagging.

Fig. 4. Image tagging results by a typical model-based approach [21]. Note that the underlined tags are missing from ImageNet categories, which shows the vocabulary difference between image classification and image tagging.

Table 4. Tagging performance in terms of both mean average precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) for the 1000 testing photos from NUS-WIDE-270K dataset.

A) PhotoTime

PhotoTime gives the first attempt to provide the features of sorting and organizing personal photos by image tagging techniques. The app is only available for iOS users now, with the total app size of 33 MB. With the help of cutting-edge deep neural networks, PhotoTime can automatically tag and categorize photos and conduct face grouping, hence users can effortlessly search photos by the provided features. Besides, PhotoTime can organize photos not only on the phone, but also those from social media platforms (e.g., iCloud, Facebook, Instagram, Twitter, Dropbox, Flickr, and Google+), without additional storage space needed. The versions released after June 2016 have made Amazon Cloud embedded for more flexible storage.

PhotoTime sends photo thumbnails, which are much smaller than the original photos, to cloud servers for image tagging, and keep the amount of data transmission minimal. When the tagging procedure is finished, PhotoTime sends back the auto-generated tags instead of photos or thumbnails. The specific tagging speed depends on network latency, with the highest efficiency of 500 photos per minute. Users can add personal tags in the detailed photo view, and all the new tags can become searchable tags.

B) Google Photos

Google Photos is a photo sharing and storage service developed by Google. Mobile apps for Android and iOS allow users to upload and access pictures on mobile. There are also desktop tools for Windows and MacOS. Google Photos grows very fast, with 200 million active users in the first month after its release, and it also allows users to back up an unlimited number of photos and videos. Google Photos organizes photo library using smart groups based on time and location. When users upload photos, Google Photos can automatically analyze them and add them to specific categories for quick access. Hence, users are able to conduct keyword-based search to find a photo. The keyword could be object, time point, or location.

Google Photos makes use of a CNN architecture, which is similar to the architecture proposed by Geoffrey Hinton and used in the ImageNet Large Scale Visual Recognition Competition. In contrast to the 1000 visual classes used in the competition, Google uses more visual classes based on the popular labels from users. Google Photos also makes use of free base entities, which are the basis for knowledge graph in Google search. Besides, users can browse into any image albums and display it on screen as slideshow. In addition, the images could be remixed to create videos for sharing on social media. Users are also allowed to label persons with their names, which can be used for searching by name in the future.

C) Microsoft OneDrive

OneDrive provides a file-hosting service developed by Microsoft, which is a part of Microsoft suite of online services. OneDrive allows users to store files as well as other personal data like Windows settings or BitLocker recovery keys in the cloud. When mobile users from Android, iOS, and Windows Phone take a photo by their phones, the photo will be automatically uploaded to OneDrive. OneDrive organizes photo libraries based on date and several tags, like “mountain,” “outside,” “hand,” etc. It can also automatically recognizes objects in a photo and creates tags for them by the state-of-the-art deep neural networks with over 6 K tag categories, so users are able to search photos with the tags provided. Furthermore, users can also remove or edit the tags for a photo.

D) Photo app on iOS 10

The photo app embedded on iOS 10 provides a client-based way to organize photos. Photos are categorized by date and location, and two new album views are provided, which includes “People” and “Places.” Each album contains photos and videos organized by faces or location information. Apple now uses deep neural networks to recognize faces, and then organizes these photos into mini albums inside the “People” album. Users can add names to them, which allows users to organize all the photos by person. The “Places” album records a map of user photos, which helps users to browse exactly where they recorded a video or snapped a picture on their trips. The photo app embedded on iOS 10 allows users to search for photos using keyword-based natural language which can be people, objects, locations, and other characteristics. However, the system usually suffers from low recall due to the limited vocabulary of queries that can be searched. Therefore, designing a system with the capability of instantly finding arbitrary photos of what users want is still worthy to explore in the future.