现有进展：IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 69, 2021 2663 Deep Learning Enabled Semantic Communication Systems Huiqiang Xie , Graduate Student Member, IEEE, Zhijin Qin , Member, IEEE, Geoffrey Ye Li , Fellow, IEEE, and Biing-Hwang Juang , Life Fellow, IEEE Abstract—Recently, deep learned enabled end-to-end commu- nication systems have been developed to merge all physical layer blocks in the traditional communication systems, which make joint transceiver optimization possible. PDF p.1
仍然存在的问题：Besides, these applications need to support massiveconnectivityoverlimitedspectrumresourcesbutrequire lower latency, which poses critical challenges to traditional source-channel coding. PDF p.1
本文提出的方案：Particularly, we propose a deep learning based semantic communication system, named DeepSC, for text transmission. PDF p.1
方案起作用的机制：In the proposed DeepSC, a joint semantic-channel coding is designed to cope with channel noise and semantic distortion, which addresses aforementioned Question 3. r The transceiver of the DeepSC is composed of semantic encoder, channel encoder, channel decoder, and semantic decoder. PDF p.2
作者希望证明的结论：Among the traditional baselines in Fig. 6(a), Brotli coding outperforms the Huffman and fixed-length encoding over AWGN channels when the turbo coding is adopted for channel coding. PDF p.8
Particularly, we propose a deep learning based semantic communication system, named DeepSC, for text transmission. PDF p.1
Specifically, we propose a DL enabled semantic communication system (DeepSC) to address the aforementioned challenges. PDF p.2
In the proposed DeepSC, a joint semantic-channel coding is designed to cope with channel noise and semantic distortion, which addresses aforementioned Question 3. r The transceiver of the DeepSC is composed of semantic encoder, channel encoder, channel decoder, and semantic decoder. PDF p.2
By adopting the structure of autoencoder in DL and removing block structure, the transmitter and receiver in the E2E system are optimized jointly as an E2E reconstruc- tion task. PDF p.2
中间语义表示是什么
Semantic Representation in Natural Language Processing NLP makes machines understand human languages, with the main goal to understand the syntax and text. PDF p.3
Particularly, the transmitter consists of a semantic encoder to extract the semantic features from the texts to be transmitted and a channel encoder to generate symbols to facilitate the transmission subsequently. PDF p.5
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
As the number of symbols per word grows, the BLEU scores increase significantly due to the increasing distance between constellations gradually. PDF p.10
Notable accomplish- ments during the period include development of vec- tor quantization for voice applications, voice coders at extremely low bit rates, 800 bps and around 300 bps, and robust vocoders for use in satellite communica- tions. PDF p.13
bit / token / channel-use / CBR 证据
Notable accomplish- ments during the period include development of vec- tor quantization for voice applications, voice coders at extremely low bit rates, 800 bps and around 300 bps, and robust vocoders for use in satellite communica- tions. PDF p.13
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
In the proposed DeepSC, a joint semantic-channel coding is designed to cope with channel noise and semantic distortion, which addresses aforementioned Question 3. r The transceiver of the DeepSC is composed of semantic encoder, channel encoder, channel decoder, and semantic decoder. PDF p.2
Definition 2: Physical channel noise is caused by the physical channel impairment, such as, additive white Gaussian noise (AWGN), fading channel, and multiple path, which incurs the signal attenuation and distortion. PDF p.3
If x is sent, the signal received at the receiver will be y = hx + n, (2) wherey ∈CM×1,hrepresentstheRayleighfadingchannelwith CN(0, 1) and n ∼CN(0, σ2 n). PDF p.4
For example, simple neural networks could be used to model the AWGN channel, multiplicative Gaussian noise channel, and the erasure channel [22]. PDF p.4
In this paper, we mainly consider the AWGN channels and Rayleigh fading channels for simplicity while focus on semantic coding and decoding. PDF p.4
Compared with BLEU score, BERT has been fed by billions of sentences. PDF p.5
The second one is the loss function for mutual information, which maximize the achieved data rate during the transmitter training. PDF p.5
After initializing the weights, W, bias, b, and using embedding vector to represent the input words, the first phase is to train the mutual information model by unsupervised learning to estimate the achieved data rate for the second phase. PDF p.6
Among the traditional baselines in Fig. 6(a), Brotli coding outperforms the Huffman and fixed-length encoding over AWGN channels when the turbo coding is adopted for channel coding. PDF p.8
In Fig. 6(b), the DL enabled approaches outperform all tra- ditional approaches over the Rayleigh fading channels, where RS coding is better than turbo coding in terms of 2-grams to 4-grams. PDF p.8
现有进展：Particularly, DL has shown its great potentials to solve the existing technical problems in both physical layer communications [4]–[6] and wireless resource allocations [7], [8]. PDF p.1
仍然存在的问题：However, even if the DL-enabled communication systems yield better performance and/or lower complexity for some scenarios and conditions, their state-of-the-art models mainly focus on performance improvement at the bit or symbol level, which usually takes bit-error rate (BER) or symbol-error rate (SER) as the performance metric. PDF p.1
本文提出的方案：Particularly, we design a deep learning (DL)- enabled semantic communication system for speech signals, named DeepSC-S. PDF p.1
方案起作用的机制：In order to improve the recovery accuracy of speech signals, especially for the essential information, DeepSC- S is developed based on an attention mechanism by utilizing a squeeze-and-excitation (SE) network. PDF p.1
作者希望证明的结论：Accordingly, the total FLOPs required by DeepSC-S are 9.36 × 109, including 4.65×109 FLOPs at the transmitter and 4.71×109 FLOPs at the receiver, which achieves a 4.82% increase over the CNN- based system. PDF p.8
Typically, a DL-based communication system is designed to reduce the complexity and/or improve the system per- formance, by merging one or multiple communication mod- ules in the traditional block-wise architecture and using a neural network (NN) to represent the intelligent transceiver. PDF p.1
In this paper, we propose a DL-enabled semantic communi- cation system for speech signals, named DeepSC-S, by learn- ing and extracting speech signals, and then recovering them at the receiver from the received features directly. PDF p.2
Powered by the Transformer [32], the semantic encoder and the channel encoder are co-designed to minimize the semantic error and to improve the system capacity. PDF p.3
As shown in Fig. 2, the transmitter consists of two individual components: the semantic encoder and the channel encoder, each component is implemented by an independent NN. PDF p.3
中间语义表示是什么
The semantic information refers to the information relevant to the transmission goal at the receiver, however, even the most cutting-edge work cannot define the semantic information or semantic features by a precise mathematical formula. PDF p.1
More- over, the semantic information varies for different transmission purposes, which could be in various formats, e.g., age of information [11], or more complicated semantic features. PDF p.1
A semantic communication system with different inputs is shown in Fig. 1, which only transmits the semantic features highly relevant to the transmission task at the receiver. PDF p.2
In this case, the extracted semantic features only contain the text characteristics while the other features will not be transmitted by the transmitter. PDF p.2
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
However, most DL algorithms pre-process speech signals to obtain magnitude, spectra, or Mel-Frequency Cepstrum by various operations, such as discrete cosine transform (DCT), before feeding into a learning system. PDF p.2
Inspired by this, a lite distributed semantic communication system for text transmis- sion, named L-DeepSC [16], has been further proposed to address the challenge of IoT devices by pruning and quantizing NN parameters. PDF p.3
According to ITU-T G.711 standard, 64 Kbps pulse code modulation (PCM) is rec- ommended for speech source coding in telephone systems with 28 = 256 quantization levels [36]. PDF p.7
Moreover, 16-bits PCM is adopted in our work for speech transmission in multimedia transmission systems with 216 = 65, 536 quantization levels. PDF p.7
Moreover, it achieves SDR scores over 80 in high SNRs due to the high PCM quantization accuracy. PDF p.9
bit / token / channel-use / CBR 证据
论文未以可检索文本完整报告输入尺寸、latent/token 数、每个 index 的 bit 数与总开销；本报告不在缺少形状和码本参数时伪造压缩率。可按 $R=N_s b_s/N_{src}$ 或 $\rho=n/k$ 在取得参数后推导。
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
4 Physical Channel 𝑝ℎ(𝒚|𝒙) … … Semantic Encoder Channel Encoder Transmitter … … Channel Decoder Semantic Decoder Receiver 𝒔 ො𝒔 𝒙 𝒚 𝒘 Fig. 2: The model structure of DL-enabled speech semantic communication system. PDF p.4
Receiver Similar to the transmitter, the receiver also consists of two cascaded parts, including the channel decoder and the semantic decoder. PDF p.4
The channel decoder is to mitigate the channel distortion and attenuation, and the semantic decoder recovers speech signals based on the learned and extracted speech semantic features. PDF p.4
Denote the NN parameters of the channel decoder and the semantic decoder as χ and δ, respectively. PDF p.4
As depicted in Fig. 2, the decoded signal, bs, can be obtained from the received signal, y, by the following operation: bs = RS δ (RC χ(y)), (3) where RC χ(·) and RS δ (·) indicate the channel decoder and the semantic decoder w.r.t. parameters χ and δ, respectively. PDF p.4
For the traditional communications, the advanced channel coding techniques are achieved at the bit level to target a low BER. PDF p.4
With more SE-ResNet modules, the performance of feature learning and extracting to the essential information will improve, however, it also in- creases the computational cost. PDF p.5
By comparing the testing results for models trained under various channel conditions, we adopt one of these models as the robust model, which could achieve good performance when coping with different channel environments. PDF p.6
For each SE-ResNet module, the number of blocks in the split layer is 2, which is achieved by 2 CNN modules with 16 filters in each module, and the transition layer is implemented by a CNN module with 32 filters. PDF p.7
Accordingly, the total FLOPs required by DeepSC-S are 9.36 × 109, including 4.65×109 FLOPs at the transmitter and 4.71×109 FLOPs at the receiver, which achieves a 4.82% increase over the CNN- based system. PDF p.8
仍然存在的问题：INTRODUCTION The continuously increasing number of connected-mobile devices and enriched intelligent demands cause the explo- sion of wireless data traffic, which brings new challenges to communication systems, including providing the cornerstone for various intelligent tasks, exploiting the limited frequency resource, and dealing with the huge volumes of data. PDF p.1
本文提出的方案：To exploit the correlation among the multimodal data from mul- tiple users, we propose a deep neural network enabled semantic communication system, named MU-DeepSC, to execute the visual question answering (VQA) task as an example. PDF p.1
方案起作用的机制：The Proposed MU-DeepSC As shown in Fig. 1, the proposed MU-DeepSC consists of an image transmitter, a text transmitter, and a receiver. 1) Image Transmitter: For the image transmitter in Fig. 1, which includes a semantic encoder and channel encoder. PDF p.2
作者希望证明的结论：Among the methods in Fig. 3, the proposed MU-DeepSC outperforms other baselines, especially in the low SNR regime, and is about to approach the upper bound at high SNR regime. PDF p.4
To exploit the correlation among the multimodal data from mul- tiple users, we propose a deep neural network enabled semantic communication system, named MU-DeepSC, to execute the visual question answering (VQA) task as an example. PDF p.1
The recent successful approaches for multimodal data fusion are mostly based on neural networks, and representative techniques include Deep Belief Net (DBN), Stacked Autoencoder (SAE), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) [11]. PDF p.1
2 ResNet-101 Conv Layers Bi-LSTM & Embedding Dense Layers Reshape Layer & Normalization Reshape Layer & Normalization Physical Channels Signal Detection Reshape Layer Reshape Layer Conv Layers Dense Layers MAC Network Image Semantic Encoder Image Channel Encoder Text Semantic Encoder Text Channel Encoder Image Channel Decoder Text Channel Decoder Semantic Decoder I S Ts I M T M I X T X Ix T x Y ˆ Ix ˆ T x ˆ I X ˆ T X ˆ I M ˆ T M Answer Image Transmitter Text Transmitter Receiver Fig. 1. PDF p.2
The Proposed MU-DeepSC As shown in Fig. 1, the proposed MU-DeepSC consists of an image transmitter, a text transmitter, and a receiver. 1) Image Transmitter: For the image transmitter in Fig. 1, which includes a semantic encoder and channel encoder. PDF p.2
中间语义表示是什么
1 Task-Oriented Multi-User Semantic Communications for VQA Task Huiqiang Xie, Student Member, IEEE, Zhijin Qin, Senior Member, IEEE, and Geoffrey Ye Li, Fellow, IEEE, Abstract—Semantic communications focus on the transmission of semantic features. PDF p.1
Then, the semantic image information can be extracted by the image semantic encoder, denoted as MI = SEI (SI; αI) , (1) where MI ∈R1×C1×14×14, where C1 is the number of feature maps, SI is the input resized image and αI is the trainable parameters. PDF p.2
We employ one layer Bi-LSTM to extract the semantic representations of the input sentence. PDF p.2
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
论文未以可检索文本完整报告输入尺寸、latent/token 数、每个 index 的 bit 数与总开销；本报告不在缺少形状和码本参数时伪造压缩率。可按 $R=N_s b_s/N_{src}$ 或 $\rho=n/k$ 在取得参数后推导。
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
2 ResNet-101 Conv Layers Bi-LSTM & Embedding Dense Layers Reshape Layer & Normalization Reshape Layer & Normalization Physical Channels Signal Detection Reshape Layer Reshape Layer Conv Layers Dense Layers MAC Network Image Semantic Encoder Image Channel Encoder Text Semantic Encoder Text Channel Encoder Image Channel Decoder Text Channel Decoder Semantic Decoder I S Ts I M T M I X T X Ix T x Y ˆ Ix ˆ T x ˆ I X ˆ T X ˆ I M ˆ T M Answer Image Transmitter Text Transmitter Receiver Fig. 1. PDF p.2
3 Similar to the image transmitter, the transmitted signal, XT , will be reshaped into the complex signal, xT ∈C1× K2L 2 , firstly and normalized by lnorm(xT ) = xT E(∥xT ∥2). (6) 3) Receiver: The receiver is shown in Fig. 1(c), where convolution layers with different units are used for the image channel decoder, dense layers with different units for the text channel decoder, and the MAC network is adopted for the semantic decoder. PDF p.3
The received symbols are detected firstly, then various semantic information is recovered through different channel decoders, and is finally merged the various semantic information to get answers. PDF p.3
Then, the signals are semantically recovered information by the channel decoders for text and image, denoted as ˆ MI = CDI ˆXI; γI , (9) and ˆ MT = CDT ˆXT ; γT , (10) respectively, where ˆ MI ∈R1×C1×14×14, ˆ MT ∈R1×L×K1, γI and γT are the corresponding trainable parameters. PDF p.3
Similar to the channel encoders, the image and text channel decoder consists of CNN layers and dense layers to decompress and recover semantic information. PDF p.3
实验设置与证据
数据集：ImageNet、VQA
Baseline：JPEG、LDPC、DeepSC
信道/链路：AWGN、Rayleigh、Rician、QAM
指标：accuracy
SNR 条件：18dB
主要实验结论（带全文页码）
Here, we employ the additional domain knowledge, i.e., channel estimation, to improve the training speed and enhance the final decision accuracy. PDF p.3
In order to improve the accuracy of answers, the cross-entropy (CE) is used as the loss function to measure the difference between the correct answer, a, and the predicted answer, ˆa, which can be formulated as LCE(a, ˆa; α, β, γ, ϕ) = −p (a) log (p (ˆa)), (12) where p(a) is the real probability of the answer, and p(ˆa) is the probability of the predicted answer. PDF p.3
Among the methods in Fig. 3, the proposed MU-DeepSC outperforms other baselines, especially in the low SNR regime, and is about to approach the upper bound at high SNR regime. PDF p.4
Besides, compared with the separate source-channel coding in traditional communications, the proposed MU-DeepSC is jointly optimized to achieve better performance at the answer accuracy. PDF p.4
The simulation results have demonstrated that the MU-DeepSC outperforms various benchmarks, especially in the low SNR regime. PDF p.5
Task-Oriented Multi-User Semantic Communications for VQA，原 PDF 第 2 页（架构/方法页）。Task-Oriented Multi-User Semantic Communications for VQA，原 PDF 第 3 页（关键结果页）。
Task-Oriented Multi-User Semantic Communications
2022 · IEEE Journal on Selected Areas in Communications · 多用户接入与广播
作者：Huiqiang Xie; Zhijin Qin; Xiaoming Tao; Khaled B. Letaief
团队归属：DeepSC 稳定作者链：Huiqiang Xie, Zhijin Qin；机构记录：Queen Mary University of London; Tsinghua University; Hong Kong University of Science and Technology; Peng Cheng Laboratory
现有进展：Moreover, we will focus on semantic communications, the new emerging communication paradigm, which has shown its superiority in handling the massive volume of data. PDF p.1
仍然存在的问题：However, in practice, we must gather multimodal data from different users/devices, transmit over the air, and process/fuse multimodal data at the receiver. PDF p.2
本文提出的方案：For the multimodal multi-user system, we develop a Transformer enabled model, named, DeepSC-VQA, for the VQA task by extracting text-image information at the transmitters and fusing it at the receiver. PDF p.1
方案起作用的机制：Transformer network consists of the encoder layers and decoder layers. PDF p.3
作者希望证明的结论：Even when using imperfect CSI, the DeepSC-IR still outperforms the benchmarks with slight performance degradation at Recall@1. PDF p.8
In particular, a novel layer-wise Transformer is designed to help fuse multimodal data by adding connection between each of the encoder and decoder layers. PDF p.1
The main contributions of this paper are summarized as follows: • We propose a Transformer [24] based transmitter struc- ture, which is applicable for both text and image trans- mission by effectively extracting semantic information for different tasks. PDF p.2
This addresses the aforementioned Q2. • Based on the proposed structure, we propose three dif- ferent deep learning (DL) enabled multiuser semantic communication frameworks, named DeepSC-IR for im- age retrieval, DeepSC-MT for machine translation, and DeepSC-VQA for VQA. PDF p.2
Specially, we propose a novel layer-wise Transformer, which can exploit more text in- formation to guide image information, to fuse the text and image information. PDF p.2
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
Preliminaries The text preprocessing includes two parts: tokenize and embedding. PDF p.3
The input sentence is first splitted into scalar- wise tokens, each representing one word or one sub-word. PDF p.3
These scalar-wise tokens are then mapped into vector-shaped tokens with learnable word vectors and used as the input to the Transformer. PDF p.3
Each patch is linearly projected into vector-shaped tokens and used as an input to the Transformer. PDF p.3
An extra learnable <CLS> token is added to the input sequence such that its corresponding output token serves as a global representation for the input sequence. PDF p.3
bit / token / channel-use / CBR 证据
论文未以可检索文本完整报告输入尺寸、latent/token 数、每个 index 的 bit 数与总开销；本报告不在缺少形状和码本参数时伪造压缩率。可按 $R=N_s b_s/N_{src}$ 或 $\rho=n/k$ 在取得参数后推导。
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
For the Rayleigh fading channel, the channel coefficient follows CN(0, 1); for the Rician fading channel, it follows CN(µ, σ2) with µ = p r/(r + 1) and σ = p 1/(r + 1), where r is the Rician coefficient. PDF p.3
We set r = 2 for Rician channels and H = I for AWGN channels. PDF p.8
Single-Modal Multi-User Semantic Communication The Recall@1 performance comparison for different chan- nels on CUB-200-2011 and for different datasets over Rician channels are shown in Fig. 4 and Fig. 5, respectively. PDF p.8
From Fig. 4, for different channels on CUB-200-2011, the proposed DeepSC-IR provides a significant gain at the low SNR regimes and approaches to the upper bound at the high SNR regimes among the reported methods, outperforming the JPEG-LDPC with 8-QAM by a margin of more than 24dB gain for 0.4 Recall@1 over fading channels. PDF p.8
From Fig. 5, for different datasets over Rician channels, the DeepSC-IR also outperforms the JPEG-LDPC with 8-QAM in the three popular datasets at Recall@1 with more than 24 dB gain, respectively. PDF p.8
SNR 条件：24dB、24 dB、18dB、9dB、24dB、18 dB、18 dB、18 dB、18 DB、18 dB
主要实验结论（带全文页码）
From Fig. 4, for different channels on CUB-200-2011, the proposed DeepSC-IR provides a significant gain at the low SNR regimes and approaches to the upper bound at the high SNR regimes among the reported methods, outperforming the JPEG-LDPC with 8-QAM by a margin of more than 24dB gain for 0.4 Recall@1 over fading channels. PDF p.8
Even when using imperfect CSI, the DeepSC-IR still outperforms the benchmarks with slight performance degradation at Recall@1. PDF p.8
From Fig. 5, for different datasets over Rician channels, the DeepSC-IR also outperforms the JPEG-LDPC with 8-QAM in the three popular datasets at Recall@1 with more than 24 dB gain, respectively. PDF p.8
From Fig. 6, on English-to-Chinese over different channels, the DeepSC-MT outperforms the UTF-8-Turbo with QPSK at the low SNR regimes over AWGN, as well as at all SNR regimes over fading channels. PDF p.9
More inaccurate CSI decreases BLEU score for both systems, in which the DeepSC-MT outperforms the benchmark and retains its high robustness to imperfect CSI. PDF p.9
方案起作用的机制：In [6], an innovative semantic com- munication system was developed by introducing Transformer in natural language processing yield and successfully used for text transmission. PDF p.1
作者希望证明的结论：Obviously, the proposed DDPG scheme achieves the highest reward value and relatively fast convergence speed. PDF p.10
The QoE specifically consists of two components, semantic transmission rate score and semantic similarity score, corresponding to user quality of service and target task performance, respectively. PDF p.2
The architecture of task-oriented semantic communication system. PDF p.3
The semantic encoder performs image feature extraction and semantic compression, where the feature extractor consists of the convolutional layers of 18-layer deep residual nets (ResNet18) [27]. PDF p.3
An image classifier composed of the fully connected (FC) layer acts as the semantic decoder at the receiver. PDF p.3
中间语义表示是什么
The authors in [10] provided a novel compression method for image features, which could decrease the number of feature maps transmitted from smart devices to edge servers and ensure the task success probability of downstream inference. PDF p.2
Based on these, a semantic-aware resource allocation method was inves- tigated that maximizes the QoE in TOSCN by optimizing the number of transmitted semantic symbols, channel allocation, and user power. PDF p.2
Inspired by [19], each user employs the adaptive semantic feature compression approach to control the size of data packets to be delivered to the edge server within a slot. PDF p.2
This greedy transmission mode increases the degree of compression of semantic features by users, resulting in a decrease in intelligent task accuracy. PDF p.2
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
Compared with the value-based Deep Q Network algorithm, DDPG operates over continu- ous action spaces and directly outputs the optimal alloca- tion strategy without traversing the value function of each action policy, avoiding the problems of excessive quantization error or soaring computational complexity caused by naive discretization [26]. PDF p.2
The DDPG Framework A standard DRL setup involving an agent that observes the noisy environment in discrete time slots is considered. PDF p.7
The system state at slot t can be denoted as the following tuple st = {n1, . . . , nJ, h1 t, . . . , hD t , ˆv1 t , . . . , ˆvD t }, (35) where J is the number of classification task categories, and nj is the number of devices to perform task j. n1, . . . , nJ are discrete variables that depend on the task assignment. h1 t, . . . , hD t denote the channel gains from users to the base station at slot t. ˆv1 t , . . . , ˆvD t denote the queue length of users at the beginning of current slot. 2) Action Space: The agent directly maps the current state st to an action at which includes the compression ratio, bandwidth proportion, and power proportion of each user. PDF p.8
bit / token / channel-use / CBR 证据
Then, a joint optimization problem of the semantic compression ratio, transmit power, and bandwidth of each user is formulated. PDF p.1
By flexibly control- ling compression ratio, a resource allocation mechanism is proposed in [19] to optimize the task success probability. PDF p.2
This paper achieves the maximum transmission efficiency of tasks over a period of time by jointly optimizing the compression ratio and wireless resource allocation strategy of semantic communication users. PDF p.2
The detailed contributions of this paper are summarized as follows: • This paper presents a construction method of the back- ground knowledge base (BKB), which stores relation- ships between semantic compression ratios and AI task performance under various channel states. PDF p.3
To achieve the preferential occupation of wireless resources by data with richer semantic information, a joint optimization problem of the semantic compression ratio, transmit power, and bandwidth of each intelligent device is formulated. • With the ultimate goal of maximizing a long-term trans- mission efficiency of tasks, this paper exploits DRL to tackle the wireless resource management problem in TOSCN. PDF p.3
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
Then the output ˆ M after channel decoding at the edge server is given by ˆ M = E−1 SC(Y, χ), (8) where E−1 SC(·) is the channel decoder network with trainable parameters χ. PDF p.4
The semantic decoder is responsible for converting the data output by the channel decoder into a series of probability val- ues, and infers the result of image classification according to the maximum probability value. PDF p.4
Denoting the noise power per unit bandwidth as N0, the received noise power from user ui,j can be expressed as σi,j t 2 = N0Bi,j t . (14) The channel gains are represented as independent random variables while considering both large-scale fading as well as small-scale Rayleigh fading. PDF p.5
At this point, more communication resources must be used to improve the visibility of reconstructed data, such as image clarity. PDF p.6
In the above problem, the loss of short-term gain may promote the whole network to achieve higher long-term gains. PDF p.7
The final objective of training the DDPG framework is to seek an opti- mal action network to maximize Jψ, which can be expressed as µ(s|ϑµ) = arg max Jψ. (30) The update of action network can be achieved by applying the chain rule to the sampled performance objective function as follows ∇ϑµJψ = 1 N N X t=1 ∇aC(s, a|ϑC)|s=st,a=µ(st|ϑµ)∇ϑµµ(s|ϑµ)|s=st. (31) Consequently, the process of updating ϑµ by gradient descent can be expressed as ϑµ = ϑµ −αactor∇ϑµJψ, (32) where αactor denotes the learning rate of action network. PDF p.8
To further cut down the training overhead of the DDPG agent and improve the performance of the proposed scheme, the agent will be punished when constraint (23e) is not satisfied. PDF p.8
Similar to the conclusions in [8], the communication mode that transmits feature maps can achieve better task performance when the actual SNR is around the training SNR. PDF p.10
现有进展：Partic- ularly, the speech recognition systems combining frequency-domain CNN with long short-term mem- ory (LSTM) have been developed [28], [29], which PDF p.3
仍然存在的问题：Joint Institute. intelligent applications require extremely high trans- mission efficiency and impose enormous challenges on conventional communication systems. PDF p.1
本文提出的方案：1 Deep Learning Enabled Semantic Communications with Speech Recognition and Synthesis Zhenzi Weng, Zhijin Qin, Xiaoming Tao, Chengkang Pan, Guangyi Liu, and Geoffrey Ye Li Abstract—In this paper, we develop a deep learning based semantic communication system for speech trans- mission, named DeepSC-ST. PDF p.1
方案起作用的机制：But to serve the intelligent tasks, the extracted semantic information only consists of the task-related seman- tic features and the other irrelative features can be ignored to minimize the data to be transmitted. PDF p.1
作者希望证明的结论：Moreover, the DeepSC-ST significantly outperforms the benchmarks when the SNR ranges from -12 dB to 4 dB for the AWGN channels, and -12 dB to 8 dB for the Rayleigh channels and the Rician channels. PDF p.11
First, the speech recognition-related semantic features are extracted for transmission by a joint semantic-channel encoder and the text is recovered at the receiver based on the received semantic features, which significantly reduces the required amount of data transmission without performance degra- dation. PDF p.1
But to serve the intelligent tasks, the extracted semantic information only consists of the task-related seman- tic features and the other irrelative features can be ignored to minimize the data to be transmitted. PDF p.1
Tong et al. [14] developed a multi-user audio semantic communica- tion system to collaboratively train the convolution neural network (CNN)-based autoencoder by imple- menting federated learning over multiple devices and a server. PDF p.2
Moreover, Shi et al. [15] designed an understanding and transmission architecture for semantic communications and verified its effective- ness by deploying the architecture into the speech transmission system, which converts speech signals into semantic symbols to ensure high semantic fidelity and decodes the received semantic sym- bols into a speech waveform. PDF p.2
中间语义表示是什么
First, the speech recognition-related semantic features are extracted for transmission by a joint semantic-channel encoder and the text is recovered at the receiver based on the received semantic features, which significantly reduces the required amount of data transmission without performance degra- dation. PDF p.1
Moreover, Shi et al. [15] designed an understanding and transmission architecture for semantic communications and verified its effective- ness by deploying the architecture into the speech transmission system, which converts speech signals into semantic symbols to ensure high semantic fidelity and decodes the received semantic sym- bols into a speech waveform. PDF p.2
At the receiver, the text sequence is esti- mated based on the received semantic features. PDF p.2
The main contributions of this paper are summarized as follows: • A novel semantic communication system, named DeepSC-ST, is proposed for the com- munication scenarios with speech input, in which a joint semantic-channel coding scheme is developed. • The text-related semantic features are extracted from the input speech by leveraging CNN and recurrent neural network (RNN)-based seman- tic transmitter, which significantly reduces the transmission data and the required communi- cation resources without performance degrada- tion. • We develop speech recognition and speech synthesis tasks to achieve diverse system out- put. PDF p.2
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
Fur- thermore, a robust semantic communication system to combat semantic error has been proposed in [23], which incorporates the image with the generated semantic noise and utilizes the masked autoencoder to mitigate the effect of semantic noise with the aid of the discrete codebook shared by the transmitter and the receiver. PDF p.3
Recently, a revolution- ary transformation from hybrid modeling to end-to- end (E2E) modeling has been witnessed to directly recognize the token sequence from an input speech by leveraging a single integrated neural network, which simplifies the speech recognition pipeline and brings significant performance gains. PDF p.3
Denote t = [t1, t2, . . . , tK], where tk is a token from the token set, t, that could be a character in the alphabet or a word boundary. PDF p.4
Then there are 29 tokens if including apostrophe, space, and blank as word boundaries, that is, t = [a, b, c, . . . , z, apostrophe, space, blank]. PDF p.4
First, the received symbols, y, is mapped into the text-related semantic features, bp, by the channel decoder, where bp = [bp1, bp2, . . . , bpL] denotes a probability matrix and probability vector bpl = [bp1 l , bp2 l , . . . , bp29 l ] comprises 29 probabilities corresponding to 29 tokens in t. PDF p.5
bit / token / channel-use / CBR 证据
论文未以可检索文本完整报告输入尺寸、latent/token 数、每个 index 的 bit 数与总开销；本报告不在缺少形状和码本参数时伪造压缩率。可按 $R=N_s b_s/N_{src}$ 或 $\rho=n/k$ 在取得参数后推导。
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
Receiver As in Fig. 1, the receiver includes the channel decoder and the feature decoder to recover the text-related semantic features and recognize the final text transcription as close to the raw text se- quence as possible. PDF p.5
First, the received symbols, y, is mapped into the text-related semantic features, bp, by the channel decoder, where bp = [bp1, bp2, . . . , bpL] denotes a probability matrix and probability vector bpl = [bp1 l , bp2 l , . . . , bp29 l ] comprises 29 probabilities corresponding to 29 tokens in t. PDF p.5
Denote the neural network parameters of the channel decoder as θR, then the recovered features, bp, can be obtained from the received symbols, y, by bp = RS θR(y), (3) where RS θR(·) indicates the channel decoder w.r.t. parameters θR. PDF p.5
6 Semantic Encoder Channel Encoder Channels 𝒉 𝒘 Feature Decoder Text-related semantic features 𝒑 Channel Decoder 𝒙 𝒚 ෝ𝒑 Input speech 𝒔 Transmitter Joint Semantic-Channel Coding I want text User ID Receiver I want speech “How are you?” “How are you?” speech synthesis Text 𝒕 ො𝒔 Fig. 1: The proposed DL-enabled semantic communication system for speech recognition and speech synthesis (DeepSC-ST). where θ denotes the neural network parameters of the transmitter and the receiver, θ = (θT , θR). PDF p.6
The received symbols, Y , are reshaped into V before feeding into the channel decoder, represented by three dense layers. PDF p.7
Particularly, an encoder maps the input token sequence into the internal feature repre- sentation after a 512-dimensional token embedding, which is achieved by a stack of three convolutional layers with 512 filters followed by batch normaliza- tion and ReLU activation, as well as a bidirectional long short-term memory (LSTM) layer containing 512 units. PDF p.9
Moreover, the DeepSC-ST significantly outperforms the benchmarks when the SNR ranges from -12 dB to 4 dB for the AWGN channels, and -12 dB to 8 dB for the Rayleigh channels and the Rician channels. PDF p.11
From the figure, the proposed DeepSC-ST provides lower WER and outperforms the speech transceiver under various channel conditions, as well as the text transceiver and the feature transceiver when SNR<8 dB. PDF p.11
According to the simulation results, the DeepSC-ST is able to achieve PDF p.11
From the figure, the DeepSC-ST obtains lower FDSD and KDSD scores than the SR+DeepSC under all tested channel conditions, besides, it achieves better speech recovery, i.e., lower FDSD and KDSD scores, than the speech transceiver, the feature transceiver, and the text transceiver from - 8 dB to 2 dB under the AWGN channels, as well nearly -10 dB to 10 dB under the Rayleigh channels and the Rician channels. PDF p.13
Deep Learning Enabled Semantic Communications With Speech Recognition and Synthesis，原 PDF 第 4 页（架构/方法页）。Deep Learning Enabled Semantic Communications With Speech Recognition and Synthesis，原 PDF 第 9 页（关键结果页）。
Semantic Communication With Memory
2023 · IEEE Journal on Selected Areas in Communications · 多用户接入与广播
作者：Huiqiang Xie; Zhijin Qin; Geoffrey Ye Li
团队归属：DeepSC 稳定作者链：Huiqiang Xie, Zhijin Qin；机构记录：Queen Mary University of London; Tsinghua University; Imperial College London
现有进展：Semantic communication has shown a great potential to increase the reliability in performing intelligent tasks, reducing the network traffic, and thus alleviating spec- trum shortage. PDF p.1
仍然存在的问题：INTRODUCTION The seamlessly connected world fosters unique services, like virtual reality (VR), mobile immersive eXtended reality (XR), or autonomous driving, and brings new challenges to communication systems, such as the scarcity of resources, the congestion of network traffic, and the scalable connectivity for edge intelligence [1]. PDF p.1
本文提出的方案：In this paper, we introduce an essential component, memory, into semantic communications to mimic human communications. PDF p.1
方案起作用的机制：Huang et al. [12] have designed the image semantic coding method by introducing the framework of rate-distortion, which can save the number of bits as well as keep the good quality of the reconstructed image. PDF p.1
作者希望证明的结论：Besides, the Mem-DeepSC outperforms the separate Mem-DeepSC in low SNR regimes, which means that the three stage training algorithm can help improve the robustness to channel noise. PDF p.8
Specially, we propose two dynamic transmission methods to enhance the transmission reliability as well as to reduce the communication overheads by masking some unessential elements, which are recognized through training the model with mutual information. PDF p.1
The transceiver has three modules, a semantic codec to extract the semantic features of the source and perform the task, a joint source-channel (JSC) codec to compress and recover the semantic features, and the memory module to store the received context in multiple time slots and aid semantic decoder in performing the task. PDF p.2
Subsequently, with the semantic encoder and channel encoder, the k-th context sentence over the k-th time slot can be encoded as xc(k) = C (S (sc(k); α) ; β) , (1) where xc(k) ∈CL×1 is the transmitted signals after the power normalization, S (·; α) and C (·; β) are denoted as the semantic encoder with parameter α and channel encoder with parameter β, respectively. PDF p.3
中间语义表示是什么
In order to reduce the commu- nication overheads, Yang et al. [8] have developed bandwidth- limited semantic communication by removing the redundancy of semantic features while keeping similar classification ac- curacy. PDF p.1
Shao et al. [9] have proposed a dynamic semantic communication system to adaptively adjust the number of the active semantic features under different signal-to-noise ratios (SNRs) with a graceful classification accuracy degradation. PDF p.1
In the proposed Mem-DeepSC, the transmitter can extract the semantic features at the sentence level effectively and the receiver can process received semantic features from the previous time-slots by employing the memory module, which addresses the aforementioned Q1. • To make the Mem-DeepSC applicable to dynamic trans- mission environment, the relationship between the length of semantic signal and the channel noise is derived. PDF p.2
Espe- cially, two dynamic transmission methods are proposed to preserve semantic features from distortion and reduce the communication resources. PDF p.2
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
The reason that noise spheres spread the signal sphere is that the latent semantic codewords have different constellation points. PDF p.5
bit / token / channel-use / CBR 证据
In order to reduce the transmission overheads, the summation operation is taken here, in which these semantic features at the word level are merged to get one semantic feature at the sentence level. PDF p.4
The Relationship Between the Length of Semantic Signal and Channel Noise Adaptive modulation has been developed for conventional communications [30], where the modulation order and code rate change according to SNRs. PDF p.5
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
Moreover, we derive the relationship between the length of semantic signal and the channel noise to validate the possibility of dynamic transmis- sion. PDF p.1
In the proposed Mem-DeepSC, the transmitter can extract the semantic features at the sentence level effectively and the receiver can process received semantic features from the previous time-slots by employing the memory module, which addresses the aforementioned Q1. • To make the Mem-DeepSC applicable to dynamic trans- mission environment, the relationship between the length of semantic signal and the channel noise is derived. PDF p.2
Transmitting the signals over the channels, the received signal can be presented as yc(k) = h(k) ⊙xc(k) + n(k), (2) where h(k) is the channel coefficients and n is the ad- ditive white Gaussian noise (AWGN), in which n(k) ∼ CN 0, σ2 nIL . PDF p.3
For the Rayleigh fading channel, the chan- nel coefficient follows h(k) ∼ CN (0, IL); for the Ri- cian fading channel, it follows h(k) ∼CN µhIL×1, σ2 hIL with µh = p r/(r + 1) and σh = p 1/(r + 1), where r is the Rician coefficient. PDF p.3
With the estimated channel state information (CSI), ˆh, the transmitted signals, ˆx(k), can be detected by ˆxc(k) = ˆhH(k) ⊙yc(k) ⊘ ˆh(k) ⊙ˆhH(k) . (3) After signal detection, the semantic features can be recovered by ˆzc(k) = C−1 (ˆxc(k); γ) , (4) where ˆzc(k) ∈RN×1 and C−1 (·; γ) is denoted as the channel decoder with parameter γ. PDF p.3
实验设置与证据
数据集：论文文本未明确命中，需查看原表格或附录
Baseline：DeepJSCC、Turbo、BERT、DeepSC
信道/链路：AWGN、Rayleigh、Rician、fading channel、QAM
指标：accuracy
SNR 条件：-6dB、0dB、6dB、12dB、18dB
主要实验结论（带全文页码）
Finally, the third step is to optimize the entire system jointly to achieve the global optimization. PDF p.5
In this situation, we can achieve such adaptive operation by masking some elements, i.e., masking less at low SNR regimes to ensure the reliability of performing tasks and masking more elements at high SNR regimes to achieve a higher transmission rate. PDF p.5
With the relationship, it is possible to achieve dynamic transmission. PDF p.6
In order to achieve stable optimization, an approximate bound-optimization (or Majorize-Minimize) algorithm is employed. PDF p.7
SIMULATION RESULTS In this section, we compare the proposed semantic com- munication systems with memory with the traditional source coding and channel coding method over various channels, in which the proposed mask methods are compared with different benchmarks. PDF p.8
本文提出的方案：In this work, we propose a novel task-oriented semantic communication framework based on scene graph, named DeepSC-SG. PDF p.1
方案起作用的机制：As shown in Fig. 1, the whole framework consists of the semantic transmitter and semantic receiver. PDF p.2
作者希望证明的结论：Among the methods in Fig. 3, the proposed DeepSC-SG achieves considerable performance at the low SNR regimes and approaches the upper bound at the high SNR regimes. PDF p.5
In this work, we propose a novel task-oriented semantic communication framework based on scene graph, named DeepSC-SG. PDF p.1
Specifically, we first devise a scene graph based semantic encoder, which extracts the explainable scene graph semantics from the input images and encodes the semantics into informative graph embeddings. PDF p.1
After receiving the semantics, a semantic decoder is devised to achieve the downstream image retrieval task by computing scene graph similarities. PDF p.1
For text transmission, a semantic communication system named DeepSC was proposed in [5] to effectively encode texts into sentence semantics by a Transformer-based architecture. PDF p.1
In contrast with the data reconstruction semantic systems, the task execution systems show more potential in the scenarios with limited wireless resources owing to the reduced transmission overhead. PDF p.1
Communication System Average Number of Transmitted Symbols for One Image Compression Ratio Performance of NDCG@5 DeepSC-SG 192 1465:1 0.662 JPEG-LDPC with 8-QAM 10854 26:1 0.662 channel noise. PDF p.6
Here we compute the compression ratio by comparing the image compression result with the uncoded image. PDF p.6
信道处理机制：decoder 实际收到什么
分类：有限星座/调制符号直接过噪声信道 接收端从有噪星座或 soft symbol 恢复语义；需区分是否有显式 bit/index 译码。
The received signal y ∈Cdc/2 can be expressed as: y = hx′ + n, (13) where h represents the channel coefficients and the additive white Gaussian noise (AWGN) n ∼CN(0, σ2 n). PDF p.4
After separately training the semantic encoder and JSC codec, we jointly train the whole system by minimizing the loss function (17) between ˆXi and Xj so as to improve the robustness to channel noise. PDF p.4
−18 −12 −6 0 6 12 18 SNR (dB) 0.0 0.2 0.4 0.6 0.8 1.0 NDCG@5 Error-free Transmisson DeepSC-SG Separate DeepSC-SG JPEG-LDPC (a) AWGN Channels −18 −12 −6 0 6 12 18 SNR (dB) 0.0 0.2 0.4 0.6 0.8 1.0 NDCG@5 Error-free Transmisson DeepSC-SG Separate DeepSC-SG JPEG-LDPC (b) Rayleigh Channels −18 −12 −6 0 6 12 18 SNR (dB) 0.0 0.2 0.4 0.6 0.8 1.0 NDCG@5 Error-free Transmisson DeepSC-SG Separate DeepSC-SG JPEG-LDPC (c) Rician Channels Fig. 3: NDCG@K comparison between DeepSC-SG and JPEG-LDPC with 8-QAM over different channels. PDF p.5
Query Image (a) DeepSC-SG (b) JPEG-LDPC Fig. 4: Top-4 similar image retrieval results at 18dB over Rayleigh Channels. which produces 2048-dim object and predicate features. PDF p.5
The second benchmark performs source and channel coding separately, including joint photographic experts group (JPEG) for image source coding, low-density parity-check code (LDPC) with a coding rate of 1/3 for image channel coding and 8 quadrature amplitude modulation (8-QAM) method. PDF p.5
实验设置与证据
数据集：Flickr30k、VQA
Baseline：JPEG、LDPC、BERT、DeepSC
信道/链路：AWGN、Rayleigh、Rician、QAM
指标：accuracy
SNR 条件：18dB、36dB、10dB、10dB
主要实验结论（带全文页码）
So far, we have achieved the image semantic encoder based on scene graphs, which captures the hidden structured se- mantic information and encodes an image into an informative embedding vector X. PDF p.3
After separately training the semantic encoder and JSC codec, we jointly train the whole system by minimizing the loss function (17) between ˆXi and Xj so as to improve the robustness to channel noise. PDF p.4
Among the methods in Fig. 3, the proposed DeepSC-SG achieves considerable performance at the low SNR regimes and approaches the upper bound at the high SNR regimes. PDF p.5
The experimental results demonstrate the proposed DeepSC-SG can achieve considerable performance at the low SNR regimes and approach the upper bound at the high SNR regimes in the downstream image retrieval work. PDF p.6
Task-Oriented Explainable Semantic Communications Based on Structured Scene Graphs，原 PDF 第 2 页（架构/方法页）。Task-Oriented Explainable Semantic Communications Based on Structured Scene Graphs，原 PDF 第 4 页（关键结果页）。
A Robust Semantic Communication System for Image Transmission
仍然存在的问题：By embracing this in- novative optimization objective, semantic communications can effectively reduce the volume of data to be transmitted, thereby successfully mitigating the challenges arising from the rapid data growth within communication networks [2]. PDF p.1
本文提出的方案：Specifically, we propose a novel metric for quantifying the intensity of semantic impairment and develop a semantic impairment dataset. PDF p.1
方案起作用的机制：Lu et al. [4] designed a confidence-based distillation mechanism for efficient semantic encoding and proposed a semantic text communication system by utilizing reinforce- ment learning to address the semantic gap. PDF p.1
作者希望证明的结论：Furthermore, it is remarkable that although the semantic fidelity of all systems diminishes as semantic impairment escalates, our proposed DeepSC-RI distinctly achieves supe- rior performance, especially in classification accuracy, which further validate the robustness of the proposed system. PDF p.5
Specifically, we propose a novel metric for quantifying the intensity of semantic impairment and develop a semantic impairment dataset. PDF p.1
Furthermore, we introduce a deep learning enabled semantic communication system, termed as DeepSC- RI, to enhance the robustness of image transmission, which incorporates a multi-scale semantic extractor with a dual-branch architecture for extracting semantics with varying granularity, thereby improving the robustness of the system. PDF p.1
The distinctive contributions of our work are further detailed in a point-wise manner. • We propose a novel metric termed as image semantic impairment intensity for quantifying the intensity of se- mantic impairments. • We construct an image semantic impairment dataset with varying levels of semantic impairments for assessing the robustness of systems. • Moreover, we propose a deep learning enabled semantic communication system for robust image transmission, namely DeepSC-RI, which leverages the mutli-scale se- mantic information to substantially mitigate semantic impairments and enhance semantic fidelity. PDF p.2
Tao, “Vector quantized semantic communication system,” IEEE Wireless Commun. PDF p.6
bit / token / channel-use / CBR 证据
论文未以可检索文本完整报告输入尺寸、latent/token 数、每个 index 的 bit 数与总开销；本报告不在缺少形状和码本参数时伪造压缩率。可按 $R=N_s b_s/N_{src}$ 或 $\rho=n/k$ 在取得参数后推导。
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
Distinct from the extensively investigated physical channel noise and fading effects, we elaborate on the mechanism of semantic impairments and establish a robust semantic commu- nication system to fight against semantic impairments. PDF p.1
In this paper, we consider the physical impairments imposed by AWGN and Rician fading channels. PDF p.2
Receiver The received multi-scale semantic information, ˆSm, is re- covered after passing through the channel decoder, which can be represented as ˆSm = f −1 c (Rx; γ), (4) where f −1 c (·; γ) is the channel decoder having the trainable parameter set γ. PDF p.2
The received signal, Rx, undergoes processing by the chan- nel decoder to recover the multi-scale semantic information, which is expressed as ˆSm = fγ(Rx), (24) where fγ(·) represents the channel decoder which consists of linear layers and the trainable parameter set γ. PDF p.4
实验设置与证据
数据集：论文文本未明确命中，需查看原表格或附录
Baseline：BPG、LDPC、DeepSC
信道/链路：AWGN、Rician、QAM
指标：PSNR、LPIPS、accuracy
SNR 条件：18 dB、18 dB
主要实验结论（带全文页码）
Baseline Models and Simulation Settings The proposed method is compared with a series of existing methods. PDF p.4
As illustrated in Fig. 5(c), at an ISII of 0.3, the semantic communication systems employing the ViT and the UNet achieve classification accuracy of around 80%. PDF p.5
Furthermore, it is remarkable that although the semantic fidelity of all systems diminishes as semantic impairment escalates, our proposed DeepSC-RI distinctly achieves supe- rior performance, especially in classification accuracy, which further validate the robustness of the proposed system. PDF p.5
The experimental results demonstrate that the propsoed architecture PDF p.5
System performance under Rician fading channels versus ISII with CIFAR10. can improve the semantic fidelity of the system by eliminating semantic impairments. PDF p.6
A Robust Semantic Communication System for Image Transmission，原 PDF 第 2 页（架构/方法页）。A Robust Semantic Communication System for Image Transmission，原 PDF 第 4 页（关键结果页）。
A Robust Semantic Text Communication System
2024 · IEEE Transactions on Wireless Communications · 通信安全与隐私
作者：Xiang Peng; Zhijin Qin; Xiaoming Tao; Jianhua Lü; Lajos Hanzo
团队归属：DeepSC 稳定作者链：Zhijin Qin；机构记录：Tsinghua University; National Engineering Research Center for Information Technology in Agriculture; University of Southampton
仍然存在的问题：However, semantic communications are susceptible not only to physical channel impairments, but also to semantic impairments, which degrade semantic understanding at the receiver and disrupt the associated downstream tasks. PDF p.1
本文提出的方案：Then, we propose a robust deep learning enabled semantic communication system (R-DeepSC) by introducing a semantic corrector for robust semantic encoding so as to facilitate semantic transmission. PDF p.1
方案起作用的机制：Then, we propose a robust deep learning enabled semantic communication system (R-DeepSC) by introducing a semantic corrector for robust semantic encoding so as to facilitate semantic transmission. PDF p.1
作者希望证明的结论：The results shown in Fig. 7 exhibit similar trends to those under AWGN channels. PDF p.9
Then, we propose a robust deep learning enabled semantic communication system (R-DeepSC) by introducing a semantic corrector for robust semantic encoding so as to facilitate semantic transmission. PDF p.1
Moreover, we develop a non- autoregressive version of R-DeepSC, namely NA-RDeepSC, which offers improved inference speed by relying on a non- autoregressive architecture and an adaptive generator embedded into the semantic decoder. PDF p.1
Our experimental results demonstrate the superior robustness of the proposed R-DeepSC and NA- RDeepSC architectures in eliminating semantic impairments, hence highlighting the significance of this work in advancing the development of robust semantic communications. PDF p.1
DeepSC [3] is a pioneering example of deep learning aided semantic communications, presenting an efficient joint semantic-channel coding architecture conceived for semantic transmission. PDF p.1
中间语义表示是什么
Hu et al. [20] proposed a robust semantic communication system relying on shared codebooks to tackle both sample-dependent and sample-independent semantic contamination. PDF p.3
Before tranmission, the robust semantic communication system of Fig. 2 must carry out semantic encoding to extract the pertinent semantic features, followed by semantic correction to refine the semantics, and deep learning enabled channel encoding to guard against physical channel impairments, including impairments caused by AWGN and Rician fading channels. PDF p.3
By applying these scores to the semantic representations of all tokens, the semantics of the sentence may be obtained. PDF p.5
However, if a token is incorrect, its corrupted semantic representation may interfere with the semantics of other tokens, leading to corrupted semantic information. PDF p.5
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
Hu et al. [20] proposed a robust semantic communication system relying on shared codebooks to tackle both sample-dependent and sample-independent semantic contamination. PDF p.3
Denote the input text as S, which is broken into tokens based on the tokenization rules. PDF p.3
For instance, when tokenizing the sentence “this is predefined”, the resulting tokenized sequence could be [‘this’, ‘is’, ‘predefined’] or [‘this’, ‘is’, ‘pre’, ‘defined’], depending on the specific tokenization rules used. PDF p.3
During the process of tokenization, the collection of all tokens is referred to as the dictionary, denoted as ν. PDF p.3
The tokenized sequence can be represented as S = {s1, s2, · · · , sL}, where si is the i-th token. PDF p.3
bit / token / channel-use / CBR 证据
Zhang et al. [10] proposed a semantic communication system for flexible code rate optimization to achieve bandwidth efficiency while maintaining transmission quality. PDF p.1
The remaining systems use Huffman and low-density parity-check (LDPC) codes with a 0.5 code rate for channel coding, and adaptive modulation [46] techniques for transmission. PDF p.8
Consequently, the optimization objective of semantic communications is no longer the classic bit error rate or symbol error rate, but the fidelity of the semantic information at the receiver. PDF p.1
Before tranmission, the robust semantic communication system of Fig. 2 must carry out semantic encoding to extract the pertinent semantic features, followed by semantic correction to refine the semantics, and deep learning enabled channel encoding to guard against physical channel impairments, including impairments caused by AWGN and Rician fading channels. PDF p.3
By utilizing a channel decoder, adaptive generator, and semantic decoder, the received text, ˆS, can be represented as ˆS = gζ[gµ[gδ[Y]], ν], (4) where gδ[·] is the channel decoder having the trainable parameter set δ, gµ[·] is the adaptive generator associated with having the trainable parameter set µ, and gζ[·] is the semantic decoder having the trainable parameter set ζ. PDF p.4
Specifically, the trainable parameters of our system, including the channel encoder and channel decoder, are optimized and obtained by joint training in an end-to-end manner. PDF p.4
Moreover, when transmitting over physical channels, the transmitted signal will be subject to the effects of channel noise and fading, as seen in Eq. (3). PDF p.5
The objective of the proposed system is to eliminate semantic impairments in the transmitted text and achieve high- fidelity end-to-end semantic communications, which can be represented as max D E(U, ˆS), (6) where E(·) quantifies the semantic similarity between the uncorrupted text and the received text, and D is the semantic impairment dataset. PDF p.5
This process can help eliminate the interference of corrupted text and improve the accuracy and performance of the Transformer model. PDF p.5
Non-Autoregressive Decoder for Inference Acceleration Although the Transformer of [35] has achieved remarkable performace, the inference time of this autoregressive form has increased substantially due to the complete dependence between tokens. PDF p.6
Although these solutions achieve excellent performance, their premise is that the decoder has access to the source text, S, which can then be utilized to build the independent conditional sequence, I, for the semantic decoder. PDF p.6
Nonetheless, our solutions still achieve significant improve- ments in correcting ASR errors, hence they are eminently suitable for practical real-world applications, such as speech recognition. PDF p.9
通信审稿价值与 Codex 判断
价值在于说明语义特征并非天然安全，并给出可靠性、隐私和资源开销之间可测量的权衡。
局限：证据主要来自数据集与仿真信道，缺少真实射频链路/原型验证。
A Robust Semantic Text Communication System，原 PDF 第 2 页（架构/方法页）。A Robust Semantic Text Communication System，原 PDF 第 12 页（关键结果页）。
A Unified Multi-Task Semantic Communication System for Multimodal Data
2024 · IEEE Transactions on Communications · 物理层调制、波形与 MIMO
团队归属：DeepSC 稳定作者链：Zhijin Qin；机构记录：Zhejiang University; National Engineering Research Center for Information Technology in Agriculture; Tsinghua University
仍然存在的问题：However, the employed deep neural networks in semantic communications have to be updated when the task is changed or multiple models need to be stored for performing different tasks. PDF p.1
本文提出的方案：To address this issue, we develop a unified deep learning-enabled semantic communication system (U-DeepSC), where a unified end-to-end framework can serve many different tasks with multiple modalities of data. PDF p.1
方案起作用的机制：The U-DeepSC consists of the semantic/channel encoders for each modality, and the unified semantic/channel decoder with light- weight task-specific heads. PDF p.3
作者希望证明的结论：More- over, the proposed U-DeepSC achieves close performance to the T-DeepSC in all considered tasks. PDF p.12
As the number of required features varies from task to task, we propose a vector-wise dynamic scheme that can adjust the number of transmitted symbols for different tasks. PDF p.1
In [23], a vector quantization-variational autoencoder (VQ-VAE) based robust semantic communication systems has been developed for image classification. PDF p.2
However, the model in [24] still needs to be retrained separately for different tasks and the transceiver architecture has not been unified for different tasks yet. PDF p.2
Therefore, in this paper, we propose a unified DL-based semantic communication system (U-DeepSC). PDF p.2
中间语义表示是什么
To reduce the transmission overhead, we then design a unified codebook for feature representation to serve multiple tasks, where only the indices of these task-specific features in the codebook are transmitted. PDF p.1
The semantic communication system in [12] converts speech signals into semantic features and decodes the received features into a reconstructed speech waveform. PDF p.1
To reduce the transmission overhead as well as enable digital transmission, we adopt the codebook design in [23], where a discrete codebook shared by the transmitter and receiver is designed for encoded feature representation and only the indices of these encoded features in the codebook are transmitted. PDF p.2
Different from [23] where the codebook is for a specific task with single modality, we design a unified codebook for multimodal data. PDF p.2
To reduce the transmission overhead, we then design a unified codebook for feature representation to serve multiple tasks, where only the indices of these task-specific features in the codebook are transmitted. PDF p.1
In [23], a vector quantization-variational autoencoder (VQ-VAE) based robust semantic communication systems has been developed for image classification. PDF p.2
To reduce the transmission overhead as well as enable digital transmission, we adopt the codebook design in [23], where a discrete codebook shared by the transmitter and receiver is designed for encoded feature representation and only the indices of these encoded features in the codebook are transmitted. PDF p.2
Different from [23] where the codebook is for a specific task with single modality, we design a unified codebook for multimodal data. PDF p.2
Since FSM can hierarchically prune redundant feature vectors, the computation complexity is reduced and inference speed can be accelerated. • We develop a unified codebook for multi-task services to support digital communication and reduce transmission overhead. PDF p.3
bit / token / channel-use / CBR 证据
To reduce the transmission overhead, we then design a unified codebook for feature representation to serve multiple tasks, where only the indices of these task-specific features in the codebook are transmitted. PDF p.1
According to the simulation results, the proposed U-DeepSC achieves comparable performance to the task-oriented semantic communication system designed for a specific task but with significant reduction in both transmission overhead and model size. PDF p.1
While transmitting more encoded features can enhance performance against noise by capitalizing on feature redundancy, this also introduces higher transmission overhead. PDF p.2
To determine an appropriate transmission overhead tailored to each task, we develop a dynamic channel encoder for U-DeepSC. PDF p.2
To reduce the transmission overhead as well as enable digital transmission, we adopt the codebook design in [23], where a discrete codebook shared by the transmitter and receiver is designed for encoded feature representation and only the indices of these encoded features in the codebook are transmitted. PDF p.2
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
The U-DeepSC consists of the semantic/channel encoders for each modality, and the unified semantic/channel decoder with light- weight task-specific heads. PDF p.3
The unified receiver consists of the channel decoder and the semantic decoder. PDF p.3
Specifically, the received vector at the receiver is given by yi = hizi + ni, i ∈{v, t, s}, (2) where hi ∈C represents the channel gain coefficient and ni ∼ CN(0, σ2) is the additive white Gaussian noise (AWGN). PDF p.4
The channel noise also affects probability of keeping the feature vectors, we employ another MLP layer to extract the noise feature f n = MLP(σ2) ∈REl, where σ2 is the noise variance. PDF p.7
To optimize dynamic transmission overhead across varying channel conditions, the ratio of transmitted feature vectors is governed by a predetermined variable value δ = fn σ2; ψ , where ψ denotes the trainable parameters and σ2 denotes the variable channel noise. PDF p.8
This is achieved by generating a set of random trainable parameter vectors, as illustrated in Fig. 2, and jointly learning them across the entire network. PDF p.6
To achieve this goal, we design a channel encoder to dynamically adjust the number of output feature vectors for different tasks under different channel conditions in U- DeepSC, which is able to dynamically achieve satisfactory performance by transmitting the least number of features. PDF p.6
After dropping a small part of features each time, the model can adaptively adjust the next selection according to the existing unmasked features, to achieve a higher tolerance for error selection than the single selection performed at the end of the transmitter. PDF p.7
However, if we apply loss (19) directly, more feature vectors will be selected as the training goes on, since the model tends to keep more feature vectors to improve the performance. PDF p.8
To balance the performance and the number of transmitted symbols, we add the l1-norm of δ to loss (19), as given by LFSM = 1 Nq Nq X q=1 δq −1 Nd Nd X i=1 mq i !2 + ∥δ∥. (20) It makes the model keep fewer feature vectors with the decrement of δ, and enables the model to achieve a good performance by only transmitting a part of feature vectors. PDF p.8
A Unified Multi-Task Semantic Communication System for Multimodal Data，原 PDF 第 3 页（架构/方法页）。A Unified Multi-Task Semantic Communication System for Multimodal Data，原 PDF 第 11 页（关键结果页）。
现有进展：In this direction, the importance and meaning behind the transmitted data are taken into account in the system design, and the concept of semantic communication has attracted increasing attention [8], [9], [10], [11]. PDF p.1
仍然存在的问题：However, existing learning-based approaches still face limitations in defining semantic level loss and often struggle to find a good trade-off between preserving semantic information and preserving intricate details. PDF p.1
本文提出的方案：First, inspired by practical observations, we introduce the concept of semantic contrastive loss and propose a semantic contrastive coding (SemCC) approach that treats data corruption during transmission as a form of data augmentation within the CL framework. PDF p.1
方案起作用的机制：Guo et al. [49] explored the ability of pre-trained large language model (LLM) such as ChatGPT to extract semantic infor- mation by introducing a cross-layer manager, thus achieving lower semantic loss under limited bandwidth. PDF p.3
作者希望证明的结论：This figure clearly shows that the proposed SemCC consistently outperforms the compared ones in terms of accuracy. PDF p.9
In addi- tion, the existing semantic communication approaches cannot effectively train semantic encoders and decoders without the support of downstream models. PDF p.1
Moreover, we propose a semantic re-encoding (SemRE) operation, which uses a duplicate of the semantic encoder deployed at the receiver to guide the entire training process when the downstream model is inaccessible. PDF p.1
While recent efforts used advanced deep learning technologies in semantic communication systems, there are still some issues that need to be addressed, which are discussed below. 1) How to Evaluate the Loss of Semantic Level During the Training Process: While the performance of a semantic communication system can be effectively evaluated by the downstream task, it is crucial to note that the direct use of the loss functions of the downstream task, such as the cross-entropy loss [12], may not fully match the intrinsic characteristics of semantic information, and may not guide the training of semantic encoder and decoder well, which could lead to a decreased effectiveness and robustness. PDF p.2
Therefore, it is necessary to integrate the inherent properties of semantic information into the semantic level loss. 2) How to Train Semantic Encoders and Decoders Without the Help of a Pre-Trained Downstream Network: The semantic communication system faces significant challenges in scenar- ios where the receiver is prohibited from accessing not only the weights but also the architecture of the downstream model (i.e. the pre-trained downstream is black-box). PDF p.2
中间语义表示是什么
Subsequently, ˆx will be used to exert the downstream task and obtain the inference results through the following process f x = Fb ϕ1(ˆx), (6) where Fb ϕ1(·) characterized by parameter ϕ1 denotes the feature extraction operation performed by the back- bone of the pretrained downstream model, and f x = {f (1), f (2), · · · f (C)} is the output feature map with C chan- nels. PDF p.4
The backbone of pretrained downstream mode Fb ϕ1(·) is applied to x and ˆx, which generates the feature maps f x = Fb ϕ1(x) and f ˆx = Fb ϕ1(ˆx), respectively. PDF p.5
Specifically, we can obtain the feature map f m = Fb(m) and f ˆ m = Fb( ˆm) by feeding m and ˆm into the backbone of pretrained downstream model respectively. PDF p.5
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
Gündüz, “DeepJSCC- Q: Constellation constrained deep joint source-channel coding,” IEEE J. PDF p.15
Tao, “Learning based joint coding- modulation for digital semantic communication systems,” in Proc. PDF p.15
bit / token / channel-use / CBR 证据
Digital Object Identifier 10.1109/TCOMM.2024.3400912 significant accuracy improvement of up to 53% on the CIFAR-10 dataset with a bandwidth compression ratio of 1/24, and also obtain comparable image reconstruction quality as the bandwidth compression ratio is improved. PDF p.1
Simulation results show that the proposed approaches can achieve leading accuracy performance in the downstream task under a range of bandwidth compression ratios, and demonstrate remarkable adaptability to both AWGN and Rayleigh fading channels with different noise levels, and also make a good trade-off between the image reconstruction quality and inference performance. PDF p.3
In particular, our approaches achieve a significant accuracy improvement of up to 53% on the CIFAR-10 dataset with a bandwidth compression ratio of 1/24. PDF p.3
Typ- ically, k < n should be satisfied to the bandwidth constraint, where k/n is referred to as the bandwidth compression ratio. PDF p.4
In particular, a large bandwidth compression ratio indicates a favorable communication condition, while a small one indicates a limited use of bandwidth. PDF p.4
Researchers have made considerable efforts to approximate the channel capacity by developing the advanced channel coding techniques such as low-density parity check (LDPC) [3] and polar code [4] in the 5G New Radio (NR). PDF p.1
In this context, we no longer pay attention to the typical metric of the technical level of communication such as bit error ratio (BER) and symbol error rate (SER). PDF p.3
Simulation results show that the proposed approaches can achieve leading accuracy performance in the downstream task under a range of bandwidth compression ratios, and demonstrate remarkable adaptability to both AWGN and Rayleigh fading channels with different noise levels, and also make a good trade-off between the image reconstruction quality and inference performance. PDF p.3
Next, s is transmitted over the noisy channel, where both the additive Gaussian white noise (AWGN) channel and Rayleigh fading channels are considered in this paper. PDF p.4
Specifically, for the AWGN channel, the received signals can be expressed as ˆs = s + ϵ, (3) where ˆs is the received signals, and ϵ ∈CN(0, σ2I) denotes the additional noise sample. PDF p.4
In the second training stage, we aim to further optimize the performance of the semantic communication system by jointly fine-tuning the encoder and decoder with a small learning rate to achieve significant inference performance and reconstructed image quality, especially when the bandwidth compression ratio is low. PDF p.6
In particular, θr 1 is updated based on θ1 and we will introduce how this update is achieved. PDF p.7
Instead, its weights are updated periodically to achieve better training stability. PDF p.8
DeepSC trains the semantic encoder and decoder with both semantic loss provided by the whole pre-trained ResNet-20 and observation loss in (11) to achieve efficient semantic information transmission. PDF p.8
Hence, practical digital transmission schemes incorporating channel coding and modulation can not outperform this upper bound. PDF p.9
仍然存在的问题：However, few of them addressed the training approach of DNN-based semantic models, while the effectiveness of task- oriented SemCom relies heavily on semantic models deployed on each transceiver, which requires continuous update along with the changing channel environment and datasets [1]. PDF p.1
Our FedCL enables collaborative train- ing of local semantic encoders across multiple clients and a global semantic decoder owned by the base station. PDF p.1
This framework supports heterogeneous semantic encoders since it does not require client-side model aggregation. PDF p.1
However, in practice, client-side semantic transmitters should accommo- date personalized encoders, which can adapt to diverse data distributions and varying model structures due to local devices’ different computation and storage capabilities. PDF p.1
In this work, we propose a federated contrastive learning (FedCL) framework for task-oriented SemCom, where person- alized semantic encoders and a global semantic decoder are trained collaboratively between the clients and the base station (BS). PDF p.1
中间语义表示是什么
Consequently, it provides superior supervision for learning discriminative local semantic features. PDF p.1
This network leverages contrastive learning to generate global semantic centroids, which are updated in each round to provide a unified semantic space for supervised local semantic feature learning. PDF p.1
After the client-side forward propagation is completed, the encoded feature fk is reshaped into semantic symbols sk and transmitted to the BS over the wireless channel along with the label yk. PDF p.2
Subsequently, the noised semantic symbols received from all participating clients are reshaped into noised semantic feature ˆfk, which is utilized as the input for forward propagation of the semantic decoder hosted on the BS. PDF p.2
Then, the feature received at the BS can be expressed as ˆsk = hksk + nk, ∀k ∈K, (2) where hk denotes the channel coefficient and nk is the IID channel noise vector, which follows symmetric complex Gaussian distribution CN(0, δ2I) with zero mean and variance δ2 [10]. PDF p.2
Different from FedProto and FedAvg which suffer significant degrada- tion at low SNR, the FedCL only degrades by approximately 10%, revealing that trainable global semantic centroids based on contrastive learning provide a better regularization for noised semantic features, which improve the robustness against the interference of wireless channel noise. PDF p.5
It preserves clearer semantic boundary against significant channel noise compared to benchmarks without trainable global semantic centroids, suggesting its superior separation capability based on the contrastive learning method. PDF p.5
实验设置与证据
数据集：CIFAR-10
Baseline：论文文本未明确命中，需查看原表格或附录
信道/链路：OFDM
指标：accuracy、latency
SNR 条件：9 dB、10 dB、5 dB
主要实验结论（带全文页码）
Simulation results demonstrate that FedCL surpasses benchmark approaches in task performance, particularly in scenarios with low signal-to-noise ratio (SNR) and significant data heterogeneity. PDF p.2
Therefore, dislike other FL frameworks with centroid regularization that achieve the global centroids by simply aggregating the local centroids [11], [12], we design the SCG to generate trainable global semantic centroids F = {F c}C c=1 via contrastive learning. PDF p.3
With 20 clients, FedCL outperforms FedProto and FedAvg by 12.75% and 38.55%, respectively. PDF p.4
At SNR = 9 dB, FedCL achieves 25.09% and 66.24% higher accuracy than FedProto and FedAvg, respectively. PDF p.5
Different from FedProto and FedAvg which suffer significant degrada- tion at low SNR, the FedCL only degrades by approximately 10%, revealing that trainable global semantic centroids based on contrastive learning provide a better regularization for noised semantic features, which improve the robustness against the interference of wireless channel noise. PDF p.5
Federated Contrastive Learning for Personalized Semantic Communication，原 PDF 第 2 页（架构/方法页）。Federated Contrastive Learning for Personalized Semantic Communication，原 PDF 第 4 页（关键结果页）。
Hybrid Digital-Analog Semantic Communications
2025 · IEEE Journal on Selected Areas in Communications · 物理层调制、波形与 MIMO
作者：Huiqiang Xie; Zhijin Qin; Zhu Han; Khaled B. Letaief
团队归属：DeepSC 稳定作者链：Huiqiang Xie, Zhijin Qin；机构记录：Jinan University; Tsinghua University; University of Houston; Hong Kong University of Science and Technology
仍然存在的问题：Letaief, Fellow, IEEE Abstract—Digital and analog semantic communications (Sem- Com) face inherent limitations such as data security concerns in analog SemCom, as well as leveling-off and cliff-edge effects in digital SemCom. PDF p.1
本文提出的方案：In order to overcome these challenges, we propose a novel SemCom framework and a corresponding system called HDA-DeepSC, which leverages a hybrid digital- analog approach for multimedia transmission. PDF p.1
方案起作用的机制：The proposed HDA SemCom framework consists of the HDA transmitter, the wireless channel model, and the HDA receiver, which employs both digital semantic transmis- sion and analog semantic transmission. PDF p.3
作者希望证明的结论：Compared with the previous de-noise frameworks, e.g., DnCNN, that predict the noise with only one step, the de-noising diffusion framework can predict the noise with multiple steps, such that matches the distributions of noise and achieves better performance of de-noise. PDF p.8
In order to overcome these challenges, we propose a novel SemCom framework and a corresponding system called HDA-DeepSC, which leverages a hybrid digital- analog approach for multimedia transmission. PDF p.1
Addi- tionally, we propose denoising diffusion-based signal detection techniques, which involve carefully designed variance schedules and sampling algorithms to refine transmitted signals. PDF p.1
Inspired by the concept of HDA codes, we propose a novel framework called DL-based HDA semantic communication. PDF p.2
The main contributions are summarized as follows: • A novel HDA semantic communication framework is proposed, which takes advantage of analog and digital semantic communications and addresses the limitations inherent in each. • Based on the HDA semantic communication framework, we propose an HDA semantic communication system, named HDA-DeepSC, for multimedia transmission, in which the new analog-digital allocation and fusion mod- ules are proposed to generate the analog and digital components. PDF p.2
中间语义表示是什么
Similarly, the images are converted into continuous semantic features and adaptively assigned to different subchannels based on the channel state information (CSI). PDF p.1
Wang et al. [11] have proposed a video semantic communication system, in which the semantic features of frames are extracted into continuous signals and transmitted using analog communication methods. PDF p.1
4 the semantic features are recovered by the analog and digital receivers, respectively. 1) Analog Receiver: The semantic features transmitted by analog communications are estimated by ˆzA = C−1 A (ˆxA; βr), (8) where ˆzA is the estimated semantic features and C−1 A (·; βr) is denoted as the analog channel decoder with parameter βr. 2) Digital Receiver: For digital semantic transmission, the transmitted bit streams are recovered firstly by ˆb = C−1 D M−1 (ˆxD) , (9) where C−1 D (·) represents the digital channel decoder and M−1(·) is denoted as the fixed-size demodulation. PDF p.4
Then, the semantic features transmitted with digital semantic transmis- sion are recovered by ˆzD = Q−1(E−1(ˆb)), (10) where E−1(·) and Q−1(·) are denoted as the entropy decoder and dequantizer, respectively. PDF p.4
Letaief is with the Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China (email: eekhaled@ust.hk).). be categorized into two categories: analog semantic commu- nications and digital semantic communications. PDF p.1
Analog semantic communications [3]–[11] convey the se- mantic information using continuous signals, which takes advantage of deep learning (DL) to design end-to-end sys- tems and maps the source to the non-fixed-size constellations directly. PDF p.1
The speech signals are processed by the DeepSC-ST and output the continuous constellation points at the transmitter. PDF p.1
The commer- cial encryption algorithms are designed for discrete signals, e.g., bit streams, raising concerns about the data security of continuous signal-based systems. PDF p.1
Therefore, digital semantic communications have attracted the attention of researchers. PDF p.1
bit / token / channel-use / CBR 证据
The bandwidth compression ratio is defined as η = L 3×H×W . PDF p.3
11 0.05 0.1 0.15 Bandwidth Compression Ratio 28 30 32 34 36 38 PSNR (dB) HDA-DeepSC Analog DeepSC with DiffSDNet DeepJSCC-Q BPG+1/2 LDPC+16QAM (a) PSNR 0.05 0.1 0.15 Bandwidth Compression Ratio 10 15 20 25 MS-SSIM (dB) HDA-DeepSC Analog DeepSC with DiffSDNet DeepJSCC-Q BPG+1/2 LDPC+16QAM (b) MS-SSIM Fig. 6. PDF p.11
PSNR and MS-SSIM performance for different bandwidth compression ratios on the Kodak dataset over AWGN channels. 0.5 1 1.5 2 2.5 3 Digital-Analog Ratio 32 34 36 38 40 PSNR (dB) SNR = 5dB SNR = 10dB SNR = 15dB (a) PSNR 0.5 1 1.5 2 2.5 3 Digital-Analog Ratio 18 20 22 24 MS-SSIM (dB) SNR = 5dB SNR = 10dB SNR = 15dB (b) MS-SSIM Fig. 7. PDF p.11
Bandwidth Compression Ratio Comparisons Fig. 6 demonstrates the comparisons for different band- width compression ratios over AWGN channels at SNR=10dB. PDF p.11
For example, the HDA-DeepSC achieves the same PSNR as separate codings (the BPG with 1/2 LDPC and 16QAM) with a 33% improvement on bandwidth compression ratio. PDF p.11
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
Quantizer & Entropy Encoder Digital Channel Encoder & Modulation Semantic Decoder Analog Channel Decoder Entropy Decoder & Dequantizer Demodulation & Digital Channel Decoder Signal Detection Physical Channels Analog-Digital Allocation Analog-Digital Fusion z A z D z b A x D x x y ˆ A x ˆ D x ˆD z ˆA z ˆz ˆ ˆb Fig. 1. PDF p.3
Then, b is encoded with digital channel encoders (e.g., LDPC codes) and fixed-size constellations (e.g., 16-QAM) by xD = M (CD(b)) , (5) where xD ∈CLD×1 is the encoded symbols, M(·) represents the fixed-size modulation, and CD(·) is denoted as the digital channel encoder. PDF p.3
Wireless Channel Model When x is transmitted over the block fading channels, the received signal can be given by y = hx + n, (6) where h is the channel coefficient that remains constant within a channel coherence time, n is the additive white Gaussian noise (AWGN), in which n ∼CN 0, σ2 nIL . PDF p.3
For the Rayleigh fading channel, the channel coefficient follows h ∼CN (0, 1); for the Rician fading channel, it follows h ∼CN µh, σ2 h with µh = p r/(r + 1) and σh = p 1/(r + 1), where r is the Rician coefficient. PDF p.3
The Hybrid Digital-Analog Receiver The receiver comprises signal detection that estimates the transmitted symbols, a analog-digital fusion module that fuses the digital and analog semantic information, channel decoders that alleviate the distortions from the wireless channels, and a semantic decoder that recovers the images with the received semantic information. PDF p.3
Compared with the convolutional neural network (CNN) layer to capture the local information, the dense layer is good at capturing global information and preserving the entire attributes, which follows the target of the analog channel codec. PDF p.5
Finally, we train the whole network with LSD + λrLRate to improve the quality of the recovered image and reduce the number of bit streams in an end-to-end manner, which converges to the global optimization. PDF p.6
Compared with the previous de-noise frameworks, e.g., DnCNN, that predict the noise with only one step, the de-noising diffusion framework can predict the noise with multiple steps, such that matches the distributions of noise and achieves better performance of de-noise. PDF p.8
The reasons behind the designed variance schedule can be summarized as • Compared with the conventional diffusion-based frame- work with 1,000 steps for generative tasks, we empirically find that the de-noise task does not need too many steps due to the low complexity of the de-noise task. • We design a monotonic function of γ(t) to achieve coarse- to-fine de-noise processing, which has an unequal interval SNR, e.g., a small interval in high SNR regions and a large interval in low SNR regions. PDF p.8
For the small noise level at the high SNR regimes, the analog DeepSC is capable of restoring the signals therefore all methods achieve a similar PSNR as the SNR increases. PDF p.9
仍然存在的问题：However, existing image DeepSC methods, frequently encounter challenges in balancing rate- distortion performance and computational complexity, and often exhibit inferior performance compared to traditional schemes, especially on high-resolution datasets. PDF p.1
本文提出的方案：To address these limita- tions, we propose a novel image DeepSC system, using quadtree partition-based joint semantic-channel coding, named Quad- DeepSC, which maintains low complexity while achieving state- of-the-art transmission performance. PDF p.1
方案起作用的机制：As illustrated in Fig. 1 (a), the LIC framework consists of a neural analyzer ga(·), a neural synthesizer gs(·), and an entropy estimator pˆy(·). PDF p.3
作者希望证明的结论：The experimental results indicate that the proposed Quad-DeepSC significantly outperforms current DeepSC systems. PDF p.10
To address these limita- tions, we propose a novel image DeepSC system, using quadtree partition-based joint semantic-channel coding, named Quad- DeepSC, which maintains low complexity while achieving state- of-the-art transmission performance. PDF p.1
The proposed Quad-DeepSC integrates quadtree partition-based entropy estimation and fea- ture coding modules with lightweight feature extraction and re- construction networks to form an end-to-end architecture. PDF p.1
In contrast, [13] adopted a different DeepSC framework by incorporating a LIC into the JSCC architecture for semantic feature extraction, entropy estimation, and reconstruction. [14] arXiv:2512.05395v1 [eess.IV] 5 Dec 2025 PDF p.1
2 enhanced this architecture by integrating a LIC with checker- board partition-based entropy model [15], achieving superior performance that surpassed the image communication scheme, which employs VTM [16] (state-of-the-art engineered image codec of the VVC standard) for source coding and adopt the optimal modulation and scheme (MCS) index under the 3GPP standards [17] for channel coding and digital modulation, on medium-resolution image datasets. [18] introduced digital modulation into the framework proposed in [13], further boost- ing performance improvements. PDF p.2
中间语义表示是什么
In contrast, [13] adopted a different DeepSC framework by incorporating a LIC into the JSCC architecture for semantic feature extraction, entropy estimation, and reconstruction. [14] arXiv:2512.05395v1 [eess.IV] 5 Dec 2025 PDF p.1
In entropy estimation, latent features are hierarchically partitioned into groups, where each group is conditioned on the estimation of previous ones to model the probability distribution of the current group. PDF p.2
In this design, the feature coding module maps latent features to transmitted symbols following the same conditional and hierarchical structure used in entropy estimation. PDF p.2
Given x, the ga(·) maps it to a semantic feature y: y = ga(x; θga). (1) The y is then quantized to ˆy = Q(y). PDF p.3
The latter has undergone substantial development, as it employ neural networks to optimize the entire semantic coding process, including nonlinear transforms, latent distributions estimation, and quantization. PDF p.1
When LICs are adopted in conventional image com- munication systems, channel impairments and floating-point arithmetic inaccuracies across heterogeneous hardware [8] can severely degrade reconstruction quality, as LICs rely on probability estimates with extremely high precision for entropy coding, thereby limiting their applicability. PDF p.1
Additionally, Cm×n and Rm×n represent the space of m×n complex and real matrices, respectively. px is the probability density function for the continuous random variable x, and P¯x is the probability mass function for the discrete random variable ¯x. PDF p.2
AE, AD, and Q represent arithmetic encoding, arithmetic decoding, and quantization, respectively. (b) The proposed DeepSC system extends the LIC framework by adding a feature encoder ftx(·), a feature decoder frx(·), and a fixed, non-trainable channel model W(·). PDF p.3
Given x, the ga(·) maps it to a semantic feature y: y = ga(x; θga). (1) The y is then quantized to ˆy = Q(y). PDF p.3
bit / token / channel-use / CBR 证据
In particular, hk i.i.d. ∼CN(0, 1) for the k-th channel use. PDF p.4
The channel band- width ratio (CBR) is defined as CBR ≜l n = l HW C , measuring bandwidth efficiency as the number of channel uses per source pixel. PDF p.4
The estimation results of QP-SCCTX effectively guide feature coding to eliminate these redundancies while linking the output codewords of DeepSC to minimize CBR. y is partitioned into four groups: [y0, y1, y2, y3] through quadtree partition. PDF p.7
Consequently, ki,j constitutes a side information tensor k ∈RHy×Wy, whose transmission cost is given by: CBRside info = b(k)/Ck H × W × 3 = Hy × Wy × ⌈log2(Cy)⌉ H × W × 3 × Ck , (25) where Ck denotes the spectral efficiency of the side link to transmit k. PDF p.8
Additionally, we restrict the range of ki,j using a quantization function Qk(·) that maps each ki,j to its nearest value in a rate predefined set [k0, . . . , kn−1], and transmit only the corresponding indices: CBRside info = b PNG(Qk(k)) H × W × 3 × Ck , (26) where Qk(·) ∈[0, .., n −1] (e.g., Qk(ki) = i). PDF p.8
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
The wireless channel is modeled by a transfer function W(s) = h ⊙s + n, where h ∈Cl is the complex channel coefficient sequence and n ∼CN(0, σ2Il) is additive white gaussian noise (AWGN). PDF p.4
The estimation quality is measured by the normalized mean-squared error (NMSE) [28]: NMSEdB = 10 log10(E ∥e∥2 2 E [∥h∥2 2] ). (15) We consider AWGN and fading channel scenarios. PDF p.4
In the AWGN case, the channel coefficient h remains constant. PDF p.4
For rayleigh fading, it is assumed that h varies independently for 0 3 0 3 2 1 2 1 0 3 0 3 2 1 2 1 3 0 3 0 1 2 1 2 3 0 3 0 1 2 1 2 1 2 1 2 3 0 3 0 1 2 1 2 3 0 3 0 2 1 2 1 0 3 0 3 2 1 2 1 0 3 0 3 𝐲 Spatial Partition Index Channel Partition Index:0 Index:1 Index:2 Index:3 add cat or add cat or add cat or add cat or add Fig. 2: The detailed procedure of the quadtree partition, the inverse process is called quadtree fusion. each transmission. PDF p.4
Random deep fading due to intermittent blockages is modeled by a markov switch on the channel itself: line-of- sight (LOS) segments use a rician model hk = µ + σswk, where wk ∼CN(0, 1), µ = q K K+1 and σs = q 1 K+1, while non-line-of-sight (NLOS) segments are rayleigh but attenuated in amplitude by 10−A(dB) blk /20 and multiplied by a log-normal shadowing factor 10Xk/20, Xk ∼N(0, σ2 sh). PDF p.4
The quadtree partition- based coding scheme, which improves image transmission efficiency, is then detailed. PDF p.5
Integrated LIC-DeepSC Strategy The DeepSC system, built on LIC, can achieve excellent transmission performance. PDF p.5
This enhancement improves the quality of source decoding and enhances robustness to channel impairments when the pre-trained Quad-LIC is em- bedded within the DeepSC system. PDF p.6
The transformation of symbol dimensions is achieved using fully connected (FC) layers. PDF p.6
Compared with the two-step checkerboard entropy model [15], QP-SCCTX provides finer granularity and more effectively exploits spatial contexts. PDF p.7
Image Semantic Communication With Quadtree Partition-Based Coding，原 PDF 第 2 页（架构/方法页）。Image Semantic Communication With Quadtree Partition-Based Coding，原 PDF 第 9 页（关键结果页）。
Lightweight Digital Semantic Communication Based on DeepReceiver
2025 · IEEE Transactions on Vehicular Technology · 物理层调制、波形与 MIMO
作者：Yunqi Feng; Jianping Yu; Weidang Lu; Nan Zhao; Arumugam Nallanathan
仍然存在的问题：However, existing research mainly focus on analog SemCom that directly maps the semantic features extracted from the source data into analog symbols, which restricts the applicability of SemCom in real-world scenarios. PDF p.1
本文提出的方案：In this correspondence, we propose a novel digital SemCom with lightweight semantic vector quantization auto-encoder/decoder (L-VQAE) to align with the digital communicationprotocols.Moreover,consideringtheeffectofchannelnoise, a deep learning (DL) based receiver (DeepReceiver) module is further in- troduced for symbol recovery. PDF p.1
方案起作用的机制：The proposed L-VQAE consists of the semantic encoder/decoder (Codec) module, VQ module and a DL based receiver (DeepReceiver) module, each designed to address distinct challenges in the SemCom process. PDF p.1
作者希望证明的结论：As the results indicate, the DeepReceiver consistently achieves a lower BER across a wide range of SNR levels, demonstrating its capability in mitigating channel-induced distortions compared to traditional decision schemes. PDF p.4
In this correspondence, we propose a novel digital SemCom with lightweight semantic vector quantization auto-encoder/decoder (L-VQAE) to align with the digital communicationprotocols.Moreover,consideringtheeffectofchannelnoise, a deep learning (DL) based receiver (DeepReceiver) module is further in- troduced for symbol recovery. PDF p.1
The digital modulation process leads to non-differentiable mapping from the continuous output of the semantic encoder into discrete constellation symbols, posing a significant challenge to the realization of digital SemCom. PDF p.1
In [11], a robust VQ Sem- Com was designed for multi-scale fusion with the U-net architecture, which achieved high performance in multi-scale structural similarity (MS-SSIM) while necessitating the transmission of a considerable amount of data at each scale. PDF p.1
Inspired by the above discussion, we propose a novel digi- tal SemCom with lightweight semantic vector quantization auto- encoder/decoder (L-VQAE) for image transmission. PDF p.1
中间语义表示是什么
However, existing research mainly focus on analog SemCom that directly maps the semantic features extracted from the source data into analog symbols, which restricts the applicability of SemCom in real-world scenarios. PDF p.1
However, the current SemCom systems directly map the semantic features into analog channel input symbols [4], [5]. PDF p.1
The authors in [13] leveraged generative adversarial networks for image feature extraction and incorporates multi-channel indexing along with a spatially conditional normalization module to enhance semantic representation and reconstruction quality. PDF p.1
The VQ module leverages a cosine similarity-based codebook map- ping strategy to improve the alignment between semantic features and quantized representations. PDF p.1
1666 IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, VOL. 75, NO. 1, JANUARY 2026 Lightweight Digital Semantic Communication Based on DeepReceiver Yunqi Feng , Member, IEEE, Jianping Yu , Weidang Lu , Senior Member, IEEE, Nan Zhao , Senior Member, IEEE, and Arumugam Nallanathan , Fellow, IEEE Abstract—Semantic communication (SemCom) has garnered significant interest due to its potential to minimize data transmission volume without compromising communication effectiveness. PDF p.1
In this correspondence, we propose a novel digital SemCom with lightweight semantic vector quantization auto-encoder/decoder (L-VQAE) to align with the digital communicationprotocols.Moreover,consideringtheeffectofchannelnoise, a deep learning (DL) based receiver (DeepReceiver) module is further in- troduced for symbol recovery. PDF p.1
Index Terms—Digital semantic communication, DeepReceiver, vector quantization (VQ), lightweight. PDF p.1
The digital modulation process leads to non-differentiable mapping from the continuous output of the semantic encoder into discrete constellation symbols, posing a significant challenge to the realization of digital SemCom. PDF p.1
A joint coding-modulation strategy was proposed in [6] based on binary phase shift keying (BPSK) for digital SemCom. PDF p.1
bit / token / channel-use / CBR 证据
The PSNR performance of the L-VQAE compared to the baselines at a compression ratio of 1/12. PDF p.4
In Fig. 5, the MS-SSIM performance of the L-VQAE is verified across a range of compression ratios. PDF p.4
The MS-SSIM performance of the L-VQAE on ImageNet dataset with different compression ratios. PDF p.5
The PSNR performance of the L-VQAE with different datasets at a compression ratio of 1/12. ratios. PDF p.5
We further evaluate the PSNR performance of the L-VQAE with different datasets at a compression ratio of 1/12. PDF p.5
Moreover, the intrinsic presence of channel noise, which inevitably degrades the system performance, underscores the necessity for incorporating information recovery strate- gies within the framework design [14]. PDF p.1
The DeepReceiver, integrating CNN and long-short term memory (LSTM) architectures, enables learn- able bitstream refinement to effectively mitigate channel impairments and reduce bit error rate (BER). PDF p.1
Moreover, n ∼CN(0, σ2 n) is the zero mean additive white Gaussian noise (AWGN) vector. PDF p.2
Network architecture of the semantic codec module. detection is demodulated into bitstreams and the bitstreams are then decoded via the channel decoder. PDF p.2
However, traditional receivers suffer from error propagation, as the performance of subsequent modules can beimpactedbyerrorsfromearlierstages.Moreover,traditionaldecision schemes are more susceptible to channel noise in low SNR conditions, deteriorating the quality of the recovered bitstreams. PDF p.2
实验设置与证据
数据集：CIFAR-10、ImageNet、Kodak
Baseline：DeepJSCC、BPG、LDPC、DeepSC
信道/链路：AWGN、MIMO、OFDM
指标：PSNR、SSIM、MS-SSIM、accuracy、latency、BER
SNR 条件：0 dB、8 dB、1 dB
主要实验结论（带全文页码）
Therefore, in the L-VQAE, the quantization errorsareminimizedbyoptimizingthesynergybetweentheauto-Codec and the codebook, which can achieve an effective balance between data compression and transmission. PDF p.3
This decoupled training strategy ensures that each component can be optimized for its specific task, leading to improved training stability and system performance. PDF p.4
As the results indicate, the DeepReceiver consistently achieves a lower BER across a wide range of SNR levels, demonstrating its capability in mitigating channel-induced distortions compared to traditional decision schemes. PDF p.4
As a result, the DeepReceiver-based system achieves better PSNR performance and more reliable end-to-end image reconstruction. PDF p.4
Notably, compared to the traditional BPG scheme with low-density parity-check (LDPC) codes, the L-VQAE can avoid the cliff effect and outperformstheBPG+LDPC,demonstratingitssuperiorrobustnessand image reconstruction capabilities. PDF p.4
Lightweight Digital Semantic Communication Based on DeepReceiver，原 PDF 第 2 页（架构/方法页）。Lightweight Digital Semantic Communication Based on DeepReceiver，原 PDF 第 4 页（关键结果页）。
Multitask Semantic Communication: A Mutual Information-Aided Semi-Supervised Approach
2025 · IEEE Internet of Things Journal · 物理层调制、波形与 MIMO
仍然存在的问题：However, the preliminary semantic communication systems are independently designed according to a single target, including reconstruction [5], [6], [7] and task execution [8], [9], [10], [11], [12], which fail to support the IoT scenarios with multiple tasks occurring simultaneously based on the same received content. PDF p.1
本文提出的方案：48388 IEEE INTERNET OF THINGS JOURNAL, VOL. 12, NO. 22, 15 NOVEMBER 2025 Multitask Semantic Communication: A Mutual Information-Aided Semi-Supervised Approach Yining Wang , Wenqiang Yi , Member, IEEE, Shujun Han , Member, IEEE, Xiaodong Xu , Senior Member, IEEE, Ping Zhang , Fellow, IEEE, and Arumugam Nallanathan , Fellow, IEEE Abstract—In this article, we design an end-to-end digital semantic communication system to transmit semantic sym- bols that simultaneously facilitate image classification tasks and reconstruction tasks. PDF p.1
方案起作用的机制：Therefore, the MI maximization problem can be formulated as max α,θ ˆI(JS)(S; Z). (14) Hence, by jointly optimizing the MIJSCC encoder Eα(·) and the discriminator Tθ(·), the learned semantic representation summarizes the important information from the input image. PDF p.6
作者希望证明的结论：Additionally, the SSCC suffers from the cliff effect compared with our proposed MIJSCC framework, although it achieves a comparable performance at 21 dB. PDF p.10
When these objectives change, the semantic encoder should be retrained to match the new task, making it inflexible and difficult to adapt to the diverse and evolving requirements of the receiver. PDF p.1
Tian et al. [18] proposed an asynchronous multitask semantic communication framework with contrastive-based encoder and task-related decoders, which can accomplish multiple tasks in a single transmission. PDF p.2
Furthermore, with the assistance of mean-square error (MSE) minimization, the MIJSCC encoder and decoder can be jointly trained to accomplish reconstruction-style objectives, thereby supporting image recovery at the receiver side. 2) To implement the proposed MIJSCC framework in practical communication systems with limited RF capa- bilities, we integrate a standard asymmetric quantizer, which adapts the learned semantic representation for practical digital transmission. PDF p.2
As these two tasks share a similar latent space to describe different levels of visual information for the same object, we proposed a unified framework to train them simultaneously. 1) Transmitter Model: The source information is an image s ∈RL×H×C, where L, H, C denote the length, height, and color dimensions, respectively. PDF p.3
中间语义表示是什么
By training a mutual information- assisted joint source-channel coding (MIJSCC) framework, the learned semantic representation can incorporate both pixel- level generative information for reconstruction and structural discriminative information for classification, which are obtained label-free via global and local mutual information estima- tion and maximization, as well as mean-square error (MSE) minimization. PDF p.1
Considering dynamic channel conditions in practical communication systems, we further design an adaptive MIJSCC (A-MIJSCC) framework with attention-based semantic enhancement (A-MIJSCC), which allows for the sequential activation of varying dimensions of the semantic representation according to channel signal-to-noise ratio. PDF p.1
The deep learning (DL) enabled semantic communication [2] framework proposed in recent years has provided the future communication systems with the ability of knowledge perception and task com- prehension [3], which can achieve prompt system reactions and dependable, efficient information exchange in Internet of Things (IoT) scenarios [4]. PDF p.1
In this context, introducing semi-supervised learning helps alleviate the dependence on labeled data and improves the flexibil- ity of the system in adapting to diverse downstream tasks using shared semantic representations. PDF p.1
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
48388 IEEE INTERNET OF THINGS JOURNAL, VOL. 12, NO. 22, 15 NOVEMBER 2025 Multitask Semantic Communication: A Mutual Information-Aided Semi-Supervised Approach Yining Wang , Wenqiang Yi , Member, IEEE, Shujun Han , Member, IEEE, Xiaodong Xu , Senior Member, IEEE, Ping Zhang , Fellow, IEEE, and Arumugam Nallanathan , Fellow, IEEE Abstract—In this article, we design an end-to-end digital semantic communication system to transmit semantic sym- bols that simultaneously facilitate image classification tasks and reconstruction tasks. PDF p.1
Then, the high-resolution semantic representa- tion is quantized into finite constellation symbols to satisfy the hardware constraint on discrete control in practical radio frequency systems. PDF p.1
Furthermore, with the assistance of mean-square error (MSE) minimization, the MIJSCC encoder and decoder can be jointly trained to accomplish reconstruction-style objectives, thereby supporting image recovery at the receiver side. 2) To implement the proposed MIJSCC framework in practical communication systems with limited RF capa- bilities, we integrate a standard asymmetric quantizer, which adapts the learned semantic representation for practical digital transmission. PDF p.2
Specifically, we quantize the 32-bit float-number semantic representation into inte- gers with fewer bits. PDF p.2
Then the reshaped symbols can be mapped to discrete constellation points exhibiting larger point-distances, identifiable amplitudes and phases, thus can be seamlessly applied to existing communication systems. 3) To further reduce the semantic transmission over- head, we design an adaptive MIJSCC (A-MIJSCC) framework with attention-based semantic enhancement (A-MIJSCC), which consecutively deactivates different numbers of dimensions in the semantic representation Authorized licensed use limited to: Peng Cheng Laboratory. PDF p.2
bit / token / channel-use / CBR 证据
It is also demonstrated that the A-MIJSCC method facilitates the adaptive semantic transmission under varying channel conditions, which effectively reduces the transmission overhead while preserving task performance. PDF p.1
Moreover, to compensate for the potential performance loss caused by deactivation, we introduce a semantic enhance- ment module to reinforce the important dimensions in the masked semantic representations, thereby maintain- ing the task performance while reducing transmission overhead. 4) To confirm the viability and superiority of our proposed MIJSCC framework, we conduct comprehen- sive experiments on CIFAR10 dataset. PDF p.3
Moreover, the number of quantization bits q can be controlled for balancing the transmission overhead and task performance. PDF p.5
Otherwise, fewer dimensions will be activated to reduce the transmission overhead. PDF p.8
Additionally, for BPG image compression, there exists a maximum achievable compression ratio RBPG max . PDF p.10
信道处理机制：decoder 实际收到什么
分类：有限星座/调制符号直接过噪声信道 接收端从有噪星座或 soft symbol 恢复语义；需区分是否有显式 bit/index 译码。
In [29], Xie et al. analyzed the relationship between the semantic signal length and the channel noise and proposed a packing sphere theory-based method to dynamically map the semantic signal into latent semantic codewords without noise overlap. PDF p.2
Under Assumption 1, the received signal y can be expressed as follows: y = hx + n (3) where h denotes the channel coefficient and n represents the independent identically distributed (IID) channel noise vector, which follows the symmetric complex Gaussian distribution CN(0, δ2) with zero mean and variance δ2. PDF p.3
Note that due to the existing mature channel estimation techniques, the performance in this work can be straightforwardly extended to different channel fading models, e.g., Rayleigh and Rician channels. PDF p.4
ADAPTIVE MIJSCC WITH ATTENTION-BASED SEMANTIC ENHANCEMENT To enhance the robustness of the proposed MIJSCC frame- work against varying channel noise and further reduce the Fig. 3. PDF p.7
To further verify the robustness of the MIJSCC framework, we conducted additional experiments under Rayleigh fading channel incorporating LS channel estimation. PDF p.11
实验设置与证据
数据集：论文文本未明确命中，需查看原表格或附录
Baseline：BPG、JPEG
信道/链路：Rayleigh、Rician、fading channel
指标：PSNR、SSIM、accuracy、latency、BER
SNR 条件：21 dB、9 dB、12 dB、21 dB
主要实验结论（带全文页码）
Stage 2 is supervised to achieve the lightweight classifier according to personalized local tasks. PDF p.7
In order to achieve a MIJSCC codec that is robust under various channel conditions, we train the above models under SNR ∈{6, 21} dB, which is chosen randomly at each update. PDF p.7
Here, we achieve the classifier by freezing the parameters of the MIJSCC encoder and training a 3-layer fully connected network using the semantic representation as input. PDF p.7
According to the channel mask generation module, the noise variance is first input into the noise encoder Nenc(·) to achieve the noise feature g, which is multiplied with an unit upper triangular matrix. PDF p.8
Inspired by this, our study integrates the con- cept of dynamic neural networks [37] into the proposed MIJSCC framework to achieve adaptive channel coding. PDF p.8
仍然存在的问题：Nonetheless, ensuring accurate and reli- able transmission of multi-modal streams remains a critical challenge in dynamic and resource-limited environments. PDF p.1
本文提出的方案：The main contributions include: • Cross-Modal Semantic Resource Allocation Architecture: We propose an architecture that leverages the advantages of cross-modal semantic correlations and utilizes modali- ty relevance to reduce the transmission of redundant information in the scenarios of spectrum scarcity. • Task-Oriented Semantic Importance Eval- uation: We develop a transmission scheme that utilizes semantic importance to miti- gate the issue of tactile streams preempting video stream channels within a perforated architecture. PDF p.2
方案起作用的机制：By leveraging cross-modal semantic communication, the infer- ence and prediction capabilities can enhance interaction between modalities and reduce redun- dancy. PDF p.2
作者希望证明的结论：The proposed architecture outperforms the bench- mark models, demonstrating its ability to balance semantic fidelity and transmission efficiency while also improving overall throughput. PDF p.7
To tackle this challenge, this article proposes a semantic-aware cross-modal resource allocation architecture that capitalizes on the inher- ent relations and semantic correlations between multi-modal streams to facilitate efficient semantic symbol processing and stream transmission. PDF p.1
Experimental results demonstrate that the proposed resource allocation architecture exhibits exceptional reliabil- ity and efficiency. PDF p.1
For instance, under a perforated architecture, tactile data streams, which require low latency and high reliability [2], can occupy spectrum resources intended for other ser- vices, e.g., audio and video streams. PDF p.1
The main contributions include: • Cross-Modal Semantic Resource Allocation Architecture: We propose an architecture that leverages the advantages of cross-modal semantic correlations and utilizes modali- ty relevance to reduce the transmission of redundant information in the scenarios of spectrum scarcity. • Task-Oriented Semantic Importance Eval- uation: We develop a transmission scheme that utilizes semantic importance to miti- gate the issue of tactile streams preempting video stream channels within a perforated architecture. PDF p.2
中间语义表示是什么
To tackle this challenge, this article proposes a semantic-aware cross-modal resource allocation architecture that capitalizes on the inher- ent relations and semantic correlations between multi-modal streams to facilitate efficient semantic symbol processing and stream transmission. PDF p.1
However, the research into the co-optimization of semantic symbols and cross-modal stream transmission remains limited. PDF p.2
First, the task relevance module obtains semantic and task results by calculating the contribution of each semantic feature to the results. PDF p.3
Then, the inner relevance module considers the cor- relations between semantic features. PDF p.3
This approach helps prevent incomplete video stream transmission and enhances semantic reliability. • Joint Deep Reinforcement Learning Frame- work: We design a framework that per- forms joint resource allocation for discrete semantic information with dueling double deep Q network (D3QN) and continuous cross-modal streams with actor-critic algo- rithm, accommodating diverse demands of different stream types in data transmission. PDF p.2
In particular, our proposed method effectively processes both discrete semantic information and continuous cross-modal streams simultaneously. PDF p.2
Actor-critic network can handle continuous data, while D3QN can pro- cess discrete semantic data. PDF p.5
The solid line represents the continuous multi-modal streams, and the dashed line represents the discrete semantic information. PDF p.5
Subsequently, we introduced the JDRL algorithm processes discrete semantic information through centralized training and decentralized execution, enhancing resource allocation efficiency. PDF p.7
bit / token / channel-use / CBR 证据
论文未以可检索文本完整报告输入尺寸、latent/token 数、每个 index 的 bit 数与总开销；本报告不在缺少形状和码本参数时伪造压缩率。可按 $R=N_s b_s/N_{src}$ 或 $\rho=n/k$ 在取得参数后推导。
IEEE Network • xx/xx xxxx 4 multi-modal services, reduce transmission latency of tactile data, and improve network resource uti- lization. PDF p.4
Joint Deep Reinforcement Learning Based Resource Allocation In cross-modal communications, it is significant to improve the transmission rate and reliability under FIGURE 2. PDF p.4
Compared with other one-modal communication architectures, the proposed cross-modal semantic communication archi- tecture shows its advantage. PDF p.6
Polyester achieved the lowest MAE when evaluated using the temporal importance metric. PDF p.6
The semantic correlation method may significantly improve the accuracy of semantic transmission, especially in cross-modal communication. PDF p.6
本文提出的方案：Motivated by these gaps, we introduce a novel semantic-aware resource allocation framework for AAV-based NOMA networks, inspired by the theory of active inference from cognitive neuroscience [44], [45], [46]. PDF p.2
方案起作用的机制：全文自动定位未找到可靠句子，需回到 PDF 人工核查。
作者希望证明的结论：Notably, the proposed approach achieves the same sum rate as the optimizer and outperforms the other methods. PDF p.13
Zheng. (Corresponding author: Ali Krayani.) Ali Krayani, Lucio Marcenaro, and Carlo Regazzoni are with the Depart- ment of Electrical, Electronic, Telecommunications Engineering and Naval Architecture, University of Genoa, 16145 Genoa, Italy, and also with the Italian National Inter-University Consortium for Telecommunications (CNIT), 43124 Parma, Italy (e-mail: ali.krayani@ieee.org; lucio.marcenaro@unige.it; carlo.regazzoni@unige.it). PDF p.1
Felix Obite is with the Department of Electrical, Electronic, Telecommu- nications Engineering and Naval Architecture, University of Genoa, 16145 Genoa, Italy, and also with the School of Electronic Engineering and Com- puter Science, Queen Mary University of London, E1 4NS London, U.K. (e-mail: felix.obite@edu.unige.it). PDF p.1
The proposed framework, named Active Semantic Generalized Dynamic Bayesian Network (Active-SGDBN), integrates semantic representation, structured causal reasoning, and Bayesian active inference within a unified architecture. PDF p.2
Results show that our method matches the performance of exhaustive expert optimiza- tion in terms of sum-rate and bit error rate (BER), while significantly outperforming conventional baselines such as random allocation and Q-learning, especially in previously unseen environments. PDF p.3
中间语义表示是什么
The proposed framework, named Active Semantic Generalized Dynamic Bayesian Network (Active-SGDBN), integrates semantic representation, structured causal reasoning, and Bayesian active inference within a unified architecture. PDF p.2
By embedding semantic understanding into the communication loop, the frame- work goes beyond traditional optimization and learning methods, enabling AAVs to make context-aware deci- sions in real time. 2) World Model Construction with Semantic Tokens: We develop a generative world model by cluster- ing expert-optimized trajectories and power allocations using a Growing Neural Gas (GNG) algorithm. PDF p.3
This process captures the joint dynamics of AAV mobility and user association, and encodes them into a structured dic- tionary of symbolic semantic tokens. PDF p.3
By comparing predicted semantic symbols with per- ceived ones, the AAV continuously updates its actions to minimize discrepancies, allowing it to generalize well to new scenarios even with limited training data. 4) Expert-Level Performance and Generalization: We validate the effectiveness of Active-SGDBN through extensive simulations. PDF p.3
In such scenarios, the AAV’s total flight time is divided into a series of discrete time intervals to simplify the computation. PDF p.2
By embedding semantic understanding into the communication loop, the frame- work goes beyond traditional optimization and learning methods, enabling AAVs to make context-aware deci- sions in real time. 2) World Model Construction with Semantic Tokens: We develop a generative world model by cluster- ing expert-optimized trajectories and power allocations using a Growing Neural Gas (GNG) algorithm. PDF p.3
This process captures the joint dynamics of AAV mobility and user association, and encodes them into a structured dic- tionary of symbolic semantic tokens. PDF p.3
These tokens serve as interpretable representations that bridge the physical (distance) and digital (power) domains, facilitating more transparent and modular decision-making. 3) Bayesian Active Inference for Online Policy Opti- mization: We formulate a semantic-level active infer- ence mechanism that enables the AAV to infer and refine power allocation strategies during online operation. PDF p.3
The proposed approach comprises two stages: 1) Learning the World Model during offline training, a perceptual learning process where the AAV learns generative models of word tokens representing both the radio and the physical environ- ments, capturing the complex relationships and dependencies TABLE I NOTATION SUMMARY of the joint objective; and 2) an online Active-SGDBN stage that employs a Bayesian active inference for optimization of the power allocation based on AAV mobility. PDF p.5
bit / token / channel-use / CBR 证据
论文未以可检索文本完整报告输入尺寸、latent/token 数、每个 index 的 bit 数与总开销；本报告不在缺少形状和码本参数时伪造压缩率。可按 $R=N_s b_s/N_{src}$ 或 $\rho=n/k$ 在取得参数后推导。
Simulation results demonstrate that Active-SGDBN achieves near-optimal performance and surpasses benchmark schemes in both sum-rate and bit error rate (BER), validating its effectiveness and generalization capability. PDF p.1
Results show that our method matches the performance of exhaustive expert optimiza- tion in terms of sum-rate and bit error rate (BER), while significantly outperforming conventional baselines such as random allocation and Q-learning, especially in previously unseen environments. PDF p.3
Following that, we assess the effectiveness of the proposed method in adapting to new environmental conditions, with the goal of maximizing the sum-rate and minimizing bit error rates, supported by addi- tional numerical results. PDF p.10
These examples were used as input for an optimizer (exhaustive search) in an offline process to determine the best power allocation strategies, which ensure a high sum-rate and minimum bit error rates. PDF p.10
Consequently, the AAV continues this adjustment process online after each movement and action, aiming to improve its performance in terms of sum-rate and bit error rate (BER) to match the optimizer’s performance. PDF p.12
实验设置与证据
数据集：论文文本未明确命中，需查看原表格或附录
Baseline：论文文本未明确命中，需查看原表格或附录
信道/链路：论文文本未明确命中，需查看原表格或附录
指标：latency、throughput、BER、SER
SNR 条件：论文文本未明确命中，需查看原表格或附录
主要实验结论（带全文页码）
This is achieved by learning an interactive transition matrix that encodes the probabilistic relationships between the AAV’s actions and the world’s tokens. PDF p.8
Such predictions facilitate the action selection process, allowing for sub-optimal power allocation and enabling adjustments to these strategies using active inference to achieve desired performance through a sub-ideal digital configuration. PDF p.8
By anticipating these tokens, the AAV can forecast the corresponding distance word related to its physical configuration and also predict the preferred power word connected to the digital configuration it is likely to achieve. PDF p.8
Top-down predictive messages are compared with bottom-up diagnostic messages to calculate the differences between them. PDF p.9
Finally, we highlight the performance gains attained through active decision-making compared to various benchmarks. PDF p.10
通信审稿价值与 Codex 判断
价值在于把语义质量转化为可优化的网络效用，并回答有限功率、带宽、时延应分给谁。
局限：证据主要来自数据集与仿真信道，缺少真实射频链路/原型验证。
A Semantic-Aware Resource Allocation for Emerging AAV-NOMA Networks Empowered by Bayesian Active Inference，原 PDF 第 3 页（架构/方法页）。A Semantic-Aware Resource Allocation for Emerging AAV-NOMA Networks Empowered by Bayesian Active Inference，原 PDF 第 10 页（关键结果页）。
Composable Multimodal Semantic Communication: A Lightweight Large AI Model Approach
2026 · IEEE Transactions on Communications · 任务导向边缘推理
作者：Tantan Zhao; Fan Li; Xinyu Huang; Yiqun Liu; Arumugam Nallanathan
仍然存在的问题：Qin et al. [6] surveyed the emerging multimodal semantic communication direction and indicated that existing studies still lacked uni- fied system-level solutions for flexible modality compositions. PDF p.1
本文提出的方案：The main contributions of this paper are summarized as follows: 1) Communication-Oriented Composable Multimodal Semantic Representation: We propose a communication-oriented semantic representation that enables arbitrary combinations of image, audio, and video modalities through a flexible and learnable modality composition weighting mechanism. PDF p.2
方案起作用的机制：To enhance semantic robustness, a bottleneck-aware lightweight semantic knowledge base is constructed by leveraging a frozen large language model with visual prompts, where raw images or middle video frames are jointly used with textual semantics to mitigate semantic ambiguity and compensate for semantic degra- dation caused by wireless channel impairments. PDF p.1
作者希望证明的结论：CMSC consistently outperforms LAM-MSC in all cases, achieving the highest similarity of 0.8750 in the image+audio+video setting, compared with 0.7719 for LAM-MSC. PDF p.9
The main contributions of this paper are summarized as follows: 1) Communication-Oriented Composable Multimodal Semantic Representation: We propose a communication-oriented semantic representation that enables arbitrary combinations of image, audio, and video modalities through a flexible and learnable modality composition weighting mechanism. PDF p.2
Multimodal data including image, audio, and video are firstly encoded by image encoder, audio encoder, and video encoder, respectively, to obtain multimodal semantic features. PDF p.2
Then, the conditional encoder is employed to align and fuse multimodal semantic features to obtain unified semantic feature containing multimodal information. • Stage 2 This article has been accepted for publication in IEEE Transactions on Communications. PDF p.2
The middle frame enhanced text is fed into the text encoder to obtain the semantic feature, which is then fed into the conditional encoder to obtain the conditional fea- tures. PDF p.3
中间语义表示是什么
Text is adopted as a semantic bridge to unify heterogeneous modalities, while a flexible and learnable modality composition weighting mechanism enables arbitrary combinations of image, audio, and video inputs to be aggre- gated into a communication-oriented semantic representation. PDF p.1
To enhance semantic robustness, a bottleneck-aware lightweight semantic knowledge base is constructed by leveraging a frozen large language model with visual prompts, where raw images or middle video frames are jointly used with textual semantics to mitigate semantic ambiguity and compensate for semantic degra- dation caused by wireless channel impairments. PDF p.1
Index Terms—Multimodal semantic communication, compos- able multimodal representation, lightweight large AI model, semantic knowledge base with visual prompts, 6G ubiquitous intelligence. PDF p.1
In particular, the absence of an explicit, lightweight SKB augmented with visual prompts limits their ability to resolve semantic ambiguity introduced by multimodal conversion and This article has been accepted for publication in IEEE Transactions on Communications. PDF p.1
More recently, MLLMs have increasingly adopted unified token representations. PDF p.2
Qwen3-VL unifies images, text, and videos into a shared token space to support multimodal reasoning and generation [21]. PDF p.2
This memory capability of SKB for context tokens is guaranteed by the Transformer with self-attention and an autoregressive model of LwGPT-4. • Stage 5 Composable multimodal reconstruction is completed in this stage. PDF p.3
For U-DeepSC, we reproduce the baseline with modality-specific semantic front-ends, a 2-layer 8-head fusion Transformer, and an 8-bit quantization bottleneck. PDF p.6
Moreover, both CMSC variants significantly outperform the U-DeepSC baseline across all SNRs and channel models, indicating the advantage of the proposed text-centric semantic unification over token-level multimodal semantic transmission. PDF p.14
bit / token / channel-use / CBR 证据
论文未以可检索文本完整报告输入尺寸、latent/token 数、每个 index 的 bit 数与总开销；本报告不在缺少形状和码本参数时伪造压缩率。可按 $R=N_s b_s/N_{src}$ 或 $\rho=n/k$ 在取得参数后推导。
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
Then, Xsc is transmitted by the wireless channels, and the received semantic feature can be expressed by Y = HXsc + N, (8) where H is the channel gain which follows a Rayleigh or Rician fading distribution depending on the propagation scenario, and N is the complex Gaussian noise with mean zero and variance σ2 n. PDF p.5
The newly added Opus+LDPC baseline also achieves higher accuracy than CMSC, but with a lower compression rate. PDF p.10
Although LAM-MSC, Fairseq, and Opus+LDPC achieve higher accu- racy in the single-modality setting, CMSC is designed for composable multimodal semantic communication with empha- sis on semantic compactness and robustness across diverse modality combinations. PDF p.10
In addition, the newly added BPG+LDPC baseline achieves competitive accuracy but still provides a lower compression rate than CMSC. PDF p.10
As LAM-MSC does not report video results, we select DVST [42] and U-DeepSC as semantic com- munication baselines, and include VVC+LDPC as a traditional SSCC baseline. PDF p.10
实验设置与证据
数据集：LibriSpeech
Baseline：DeepJSCC、BPG、JPEG、LDPC、DeepSC
信道/链路：AWGN、Rayleigh、Rician、fading channel
指标：accuracy、semantic similarity、latency、CLIP
SNR 条件：论文文本未明确命中，需查看原表格或附录
主要实验结论（带全文页码）
For the subsequent ablation study, only the lightweight modality fusion weights are fine-tuned and compared with fixed weights, while all pre-trained backbones remain frozen. PDF p.6
The results show that LwGPT-4 maintains a highly parameter-efficient design, with only 3.15 M trainable parameters, corresponding to a trainable ratio of 0.0402%. PDF p.7
The results show that CMSC with LwGPT- 4 achieves the lowest E2E delay among the compared settings, which further confirms the deployment-oriented efficiency of the proposed lightweight SKB design. PDF p.7
As shown in Fig. 6, CMSC achieves the most semantically consistent and visually coherent results among all considered cases. PDF p.8
CMSC consistently outperforms LAM-MSC in all cases, achieving the highest similarity of 0.8750 in the image+audio+video setting, compared with 0.7719 for LAM-MSC. PDF p.9
Composable Multimodal Semantic Communication: A Lightweight Large AI Model Approach，原 PDF 第 2 页（架构/方法页）。Composable Multimodal Semantic Communication: A Lightweight Large AI Model Approach，原 PDF 第 16 页（关键结果页）。
Generative AI-Enabled Cooperative Jamming for Secure Semantic Communication
2026 · IEEE Transactions on Communications · 多用户接入与广播
仍然存在的问题：However, these secure solutions require retraining the semantic encoder and decoder, which limits the flexibility and practical deployment. PDF p.1
本文提出的方案：To address this issue, we propose a secure SemCom framework for AE-SemCom and GEN-SemCom without altering the parameters of the semantic encoder and decoder. PDF p.1
作者希望证明的结论：Specifically, for the eavesdropper, the proposed cooperative jammer even outperforms the Gaussian jammer in terms of the PSNR by about 10dB when the jammer transmitting power Pj is 0.3. PDF p.8
In eavesdropping environments, the semantic information may be overheard by the eavesdropper even under poor channel con- ditions with a powerful semantic decoder. PDF p.1
Some advanced secure communication deployments such as autoencoder-enhanced Sem- Com (AE-SemCom) and generative model-enhanced SemCom (GEN-SemCom) have been introduced to efficiently transmit the semantic information. PDF p.1
However, these secure solutions require retraining the semantic encoder and decoder, which limits the flexibility and practical deployment. PDF p.1
To address this issue, we propose a secure SemCom framework for AE-SemCom and GEN-SemCom without altering the parameters of the semantic encoder and decoder. PDF p.1
中间语义表示是什么
In this direction, GANs can effectively generate high-quality images from low- dimensional latent vectors through their adversarial frame- work, which significantly enhances the semantic representation of images in SemCom. PDF p.2
The authors in [19] designed a novel channel-aware semantic encoder for image transmission, and introduced a dynamic codebook update mechanism for GEN-SemCom to optimize transmission efficiency. PDF p.2
For example, for AE-SemCom, DNNs used in the encoding and decoding process pose privacy risks due to the potential leakage of sensitive information through learned latent representations [20]. PDF p.2
These models rely on low-dimensional, distinct latent vectors as input, which are essential for generating high-quality output. PDF p.2
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
The authors in [19] designed a novel channel-aware semantic encoder for image transmission, and introduced a dynamic codebook update mechanism for GEN-SemCom to optimize transmission efficiency. PDF p.2
For video transmission, we need some moderate change on the design of the cooperative jammer and the RRM to capture spatiotemporal dependencies across frames, and for text transmission, to perturb critical logic or entity dimensions in the continuous token embedding space. PDF p.8
bit / token / channel-use / CBR 证据
We use a compression ratio of 1/48 for AE-SemCom and 1/110 for GEN-SemCom. PDF p.8
信道处理机制：decoder 实际收到什么
分类：连续 latent/信道符号联合训练 接收端拿到带噪连续特征或均衡后的复符号，而不是出错的 VQ index；能抗模拟噪声但不等价于解决数字 index error。
Moreover, to improve the transmission efficiency of SemCom systems, the authors in [13] proposed a semantic communication system based on contrastive learning (CL) to built upon a residual network (ResNet) backbone [14], which utilizes skip connec- tions to simplify gradient propagation and enhance feature preservation against channel noise, thereby achieving a bet- ter balance between the semantic information and complex details. PDF p.1
The received signal at Bob is given by ˆzb = hb ⊙z + nb, (2) where ⊙is Hadamard product, and nb ∼CN(0, σ2I) is the additive white Gaussian noise (AWGN). PDF p.3
To obtain channel-aware latent variables, the inversion can optimize the latent variables by considering the transmission power constraints and channel noise. PDF p.4
Specifically, the latent variable z is derived by z = arg min u∈U ∥Gs(PN(u) + n; γ) −x∥2 2 , (5) where U = RNc×L denotes the latent space, γ denotes the parameter of the semantic StyleGAN2 generator, PN(·) denotes the transmission power constraints to ensure the average power constraint of the signal, and n represents the channel noise. PDF p.4
Mathematically, the signal received by Eve over the fading channel can be expressed as ˆze = he ⊙z + ne, (7) where he is the wireless channel for the Alice-to-Eve link assumed independent of hb, ne ∼ CN(0, σ2 eI) denotes AWGN at the eavesdropper, and σ2 e represents the noise variance. PDF p.4
To achieve this, we design the secure loss functions for the jamming and the RRM module, respectively. PDF p.6
This can be achieved by reducing the uncertainty of the reconstructed image, thereby making ˆxe converge toward a deterministic black or white image [34]. PDF p.7
To achieve this equilibrium, we integrate the loss functions of these modules into a unified objective: Ltotal = λ1Ljam + λ2LRRM + λ3Ltriplet, (25) where λ1, λ2, λ3 represent the hyperparameters controlling the relative importance of each term. PDF p.7
We can observe that coop- erative jamming techniques have the potential to significantly improve the security of semantic communication. PDF p.8
Moreover, compared with the Gaussian jammer, the interference ability of the cooperative jammer is significantly improved in all metrics. PDF p.8
现有进展：Although numerous methods have been developed for se- cure data transmission, the potential of exploiting semantic noise in wireless communications has not been thoroughly investigated, particularly with respect to leveraging the char- acteristics of the wireless channel environment. PDF p.2
仍然存在的问题：Du et al. [15] discussed the security challenges of wireless com- munications for the semantic Internet of Things and exam- ined the applicability of conventional security techniques to semantic communications, including physical-layer security, covert transmission, and encryption. PDF p.2
本文提出的方案：We propose a novel secure image semantic communication (SISC) framework over multiple- input multiple-output (MIMO) fading channels. PDF p.1
方案起作用的机制：Physical layer security (PLS) is an efficient technology for achieving secure communications in wireless networks by exploiting the inherent randomness and dynamics of wireless channels to ensure data confidentiality [4]. PDF p.1
作者希望证明的结论：The results indicate that as the transmit power increases, the image recovery performance gradually improves and tends to stabilize after reaching a certain value. PDF p.11
We propose a novel secure image semantic communication (SISC) framework over multiple- input multiple-output (MIMO) fading channels. PDF p.1
Zhang et al. [21] developed a joint source–channel autoencoder for image semantic extraction, employing a mean squared error (MSE) loss function to effectively balance transmission efficiency and privacy protection. PDF p.2
Due to the broadcast nature of wireless channels [23], eavesdroppers can intercept transmitted signals and exploit a leaked semantic decoder to infer the original information content. PDF p.2
In particular, we propose a novel semantic noise-aided secure image communication framework (SISC) over MIMO fading wiretap channels, where the transmitter aims to send image source data to the legitimate user in the presence of an eavesdropper. PDF p.2
中间语义表示是什么
A semantic noise-aware multi- head self-attention (SN-MSA) module is proposed to effec- tively protect semantic features by jointly considering the CSI and the semantic noise map. PDF p.2
In order to obtain the suitable semantic noise map, a learnable semantic noise generator is introduced as a preprocessing stage, which enables the elements in attention weights to adaptively interfere according to source semantic features and legitimate channel conditions. PDF p.2
As shown in Fig. 1(b), at the BS side, the semantic encoder Eθ(·) : RH×W ×3 7→RCL parameterized by θ, encodes the source image with the help of side information including estimated channel state information (CSI) of SU, ˆHs ∈CNr×Nt, and the semantic noise, m, into the semantic feature z ∈RCL, where CL refers to the sequence length of the extracted semantic information. PDF p.3
The transmitted signal x satisfies the average power constraint 1 Ntk ∥x∥2 ≤1. (1) The whole semantic information encoding process can be summarized as s Eθ(·) −−−→z Eϱ(·) −−−→x, (2) where the semantic symbol vector x is given by x = Eϱ(Eθ(s, Hs, m)). (3) 2) Wireless Wiretap Channel Transmission: After Deep- JSCC encoder mapping, the BS adopts the transmit beam- former P ∈CNt×k to transmit the signal data. PDF p.3
数字化方案与实际传输开销
数字化判断：数字调制或混合数字-模拟链路 语义表示被映射到有限星座、编码 bit 或数字/模拟并行分支，已经触及可部署物理层接口。
Then, the sequences are fed into a series of fully connected layers for linear projection and feature flattening to generate feature em- bedding tokens. PDF p.5
These blocks aim to downsample the feature embedding tokens into H 2j × W 2j ×Cj, where Cj denotes the embedding dimension. PDF p.5
Li, “Robust semantic communications with masked VQ-VAE enabled codebook,” IEEE Trans. PDF p.14
bit / token / channel-use / CBR 证据
The channel bandwidth ratio (CBR) is defined as ρ = k H×W ×3, which reflects the utilization of channel resources per symbol. PDF p.3
The digital baseline employs BPG image codec with low-density parity-check (LDPC) for channel cod- ing, quadrature phase shift keying (QPSK) for modu- lation, and the singular value decomposition (SVD) for 0.02 0.04 0.06 0.08 0.10 0.12 CBR 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 MS-SSIM SISC-SU SISC-Eve DeepJSCEC-SU DeepJSCEC-Eve ESCS-SU ESCS-Eve SecureMSE-SU SecureMSE-Eve SU Eve Fig. 8. PDF p.11
MS-SSIM performance versus different CBRs in the MIMO fading channel with Pmax = 35 dBm. precoding. PDF p.11
Experimental Results 1) Image Reconstruction Performance: Fig. 6(a) illustrates the PSNR difference between the legitimate SU and the eavesdropper Eve as a function of varying channel conditions with CBR set to ρ = 1/12. PDF p.11
To further investigate the impact of CBR constraints on the transmission efficiency and security, we evaluate the MS-SSIM performance under different CBR values. PDF p.12
信道处理机制：decoder 实际收到什么
分类：数字语义特征 + 传统信道编码 decoder 通常看到信道译码后的 bit/index；若论文只假设译码成功，则未真正学习处理残余 index 跳变。
In contrast to technical noise caused by physical impairments such as channel noise, interference, or fading, semantic noise originates from problems in understanding the context, mismatches in knowledge, or flaws in the deep learning models employed for semantic encoding and decoding [26]. PDF p.2
The received pilot can be expressed as ˆA = HsA + n, (6) where ˆA ∈RNr×Nt denotes the received pilots at SU, n is the additive channel noise experienced during the pilot transmission. PDF p.4
Given the maximum transmission delay Tmax, the maximum transmission delay constraint is equivalent to Rs ≥Rmin s ≜ k BTmax . (14) For the legitimate Rx, the received semantics pass through the channel decoder, Eϕ(·) : Ck 7→RCL parameterized by ϕ to obtain semantics ˆzs ∈RCL. PDF p.4
The image reconstruction process at the legitimate SU can be expressed by ¯ys Eϕ(·) −−−→ˆzs Eφ(·) −−−→ˆss, (15) where the recovered ˆss is given by ˆss = Eφ(Eϕ(¯ys)). (16) At the eavesdropper Rx, the semantic decoder and channel decoder at Eve employ the same network architecture and shared weight parameters as those of the legitimate SU to ensure a fair performance comparison. PDF p.4
SNR 条件：50 dB、35 dB、10dB、2.35 dB、35 dB、15 dB、35 dB、15 dB
主要实验结论（带全文页码）
Based on the modified attention scores in Eq. (22), the dual CSI-SN fusion module in the SN-MSA network improves attention distributions under varying channel conditions for SU. PDF p.6
Other Network Structure of the SISC Framework The SISC decoder is developed based on the Swin Trans- former architecture, where the patch merging blocks are replaced with patch reverse-merging blocks to achieve up- sampling. PDF p.7
LEARNABLE SEMANTIC NOISE DESIGN In this section, we introduce a learnable semantic noise generation mechanism to improve the security of the system to diverse source and channel distributions. PDF p.7
Furthermore, by adopting learnable semantic noise rather than a fixed one, the proposed system achieves an adaptive trade-off between semantic fidelity and transmission security, mitigating over-distortion for the SU and insufficient perturbation against Eve. PDF p.7
Next, as illustrated in Algorithm 2, the transceiver beamformer is well designed to improve the system throughput and transmission efficiency. PDF p.10
Semantic Noise-Aided Secure Image Transmission Over MIMO Fading Channels，原 PDF 第 3 页（架构/方法页）。Semantic Noise-Aided Secure Image Transmission Over MIMO Fading Channels，原 PDF 第 10 页（关键结果页）。
排除与边界记录
这些条目在检索中出现，但因预印本、综述/愿景、MDPI、主题边界或被期刊扩展版取代而未进入核心表。
年份
标题
原因
2021
Semantic Communication Systems for Speech Transmission
非正式期刊/会议技术论文类型：preprint
2021
Semantic Communications for Speech Recognition
非正式期刊/会议技术论文类型：preprint
2021
Task-Oriented Multi-User Semantic Communications
非正式期刊/会议技术论文类型：preprint
2021
Task-Oriented Multi-User Semantic Communications for Multimodal Data
非正式期刊/会议技术论文类型：preprint
2021
Task-Oriented Multi-User Semantic Communications for VQA Task
非正式期刊/会议技术论文类型：preprint
2021
Task-Oriented Semantic Communications for Multimodal Data.
非正式期刊/会议技术论文类型：preprint
2022
A Robust Deep Learning Enabled Semantic Communication System for Text
非正式期刊/会议技术论文类型：preprint
2022
A Unified Multi-Task Semantic Communication System for Multimodal Data
非正式期刊/会议技术论文类型：preprint
2022
A Unified Multi-Task Semantic Communication System with Domain Adaptation
非正式期刊/会议技术论文类型：preprint
2022
Deep Learning Enabled Semantic Communications with Speech Recognition and Synthesis
非正式期刊/会议技术论文类型：preprint
2022
Goal-Oriented Semantic Communications for 6G Networks
非正式期刊/会议技术论文类型：preprint
2022
Vector Quantized Semantic Communication System
非正式期刊/会议技术论文类型：preprint
2023
Contrastive Learning based Semantic Communication for Wireless Image Transmission
非正式期刊/会议技术论文类型：preprint
2023
Semantic Communication with Memory
非正式期刊/会议技术论文类型：preprint
2023
Vector Quantized Semantic Communication System
非正式期刊/会议技术论文类型：preprint
2024
A Robust Semantic Communication System for Image
非正式期刊/会议技术论文类型：preprint
2024
Federated Contrastive Learning for Personalized Semantic Communication
非正式期刊/会议技术论文类型：preprint
2024
Generative Artificial Intelligence (GAI) for Mobile Communications: A Diffusion Model Perspective
非正式期刊/会议技术论文类型：preprint
2024
Hybrid Digital-Analog Semantic Communications
非正式期刊/会议技术论文类型：preprint
2025
A Robust Image Semantic Communication System With Multi-Scale Vision Transformer
综述、教程、愿景或编者按，非正式技术研究论文
2025
Image Semantic Communication with Quadtree Partition-based Coding
非正式期刊/会议技术论文类型：preprint
2021
Semantic Communications for Speech Recognition
存在同团队、同方法家族且作者高度重合的正式期刊扩展版；按既定口径由期刊版取代：Deep Learning Enabled Semantic Communications With Speech Recognition and Synthesis (IEEE TWC, 2023)
2021
Semantic Communications for Speech Signals
存在同团队、同方法家族且作者高度重合的正式期刊扩展版；按既定口径由期刊版取代：Semantic Communication Systems for Speech Transmission (IEEE JSAC, 2021)
2022
A Robust Deep Learning Enabled Semantic Communication System for Text
存在同团队、同方法家族且作者高度重合的正式期刊扩展版；按既定口径由期刊版取代：A Robust Semantic Text Communication System (IEEE TWC, 2024)
2022
A Unified Multi-Task Semantic Communication System with Domain Adaptation
存在同团队、同方法家族且作者高度重合的正式期刊扩展版；按既定口径由期刊版取代：A Unified Multi-Task Semantic Communication System for Multimodal Data (IEEE TCOM, 2024)
2023
Contrastive Learning based Semantic Communication for Wireless Image Transmission
