Zhijin Qin 团队语义通信论文调研

Recently, deep learned enabled end-to-end commu- nication systems have been developed to merge all physical layer blocks in the traditional communication systems, which make joint transceiver optimization possible. Powered by deep learning, nat- ural language processing has achieved great success in analyzing and understanding a large amount of language texts. Inspired by research results in both areas, we aim to provide a new view on communication systems from the semantic level. Particularly, we propose a deep learning based semantic communication system, named DeepSC, for text transmission. Based on the Transformer, the DeepSC aims at maximizing the system capacity and minimiz- ing the semantic errors by recovering the meaning of sentences, rather than bit- or symbol-errors in traditional communications. Moreover, transfer learning is used to ensure the DeepSC appli- cable to different communication environments and to accelerate the model training process. To justify the performance of semantic communications accurately, we also initialize a new metric, named sentence similarity. Compared with the traditional communication system without considering semantic information exchange, the proposed DeepSC is more robust to channel variation and is able to achieve better performance, especially in the low signal-to-noise (SNR) regime, as demonstrated by the extensive simulation results. Index Terms—Deep learning, end-to-end communication, semantic communication, transfer learning, Transformer.

• Powered by deep learning, nat- ural language processing has achieved great success in analyzing and understanding a large amount of language texts.
• Particularly, we propose a deep learning based semantic communication system, named DeepSC, for text transmission.
• Compared with the traditional communication system without considering semantic information exchange, the proposed DeepSC is more robust to channel variation and is able to achieve better performance, especially in the low signal-to-noise (SNR) regime, as demonstrated by the extensive simulation results.

Semantic communications could improve the trans- mission efficiency significantly by exploring the semantic informa- tion. In this paper, we make an effort to recover the transmitted speech signals in the semantic communication systems, which minimizes the error at the semantic level rather than the bit or symbol level. Particularly, we design a deep learning (DL)- enabled semantic communication system for speech signals, named DeepSC-S. In order to improve the recovery accuracy of speech signals, especially for the essential information, DeepSC- S is developed based on an attention mechanism by utilizing a squeeze-and-excitation (SE) network. The motivation behind the attention mechanism is to identify the essential speech information by providing higher weights to them when training the neural network. Moreover, in order to facilitate the proposed DeepSC-S for dynamic channel environments, we find a general model to cope with various channel conditions without retraining. Furthermore, we investigate DeepSC-S in telephone systems as well as multimedia transmission systems to verify the model adaptation in practice. The simulation results demonstrate that our proposed DeepSC-S outperforms the traditional communi- cations in both cases in terms of the speech signals metrics, such as signal-to-distortion ration and perceptual evaluation of speech distortion. Besides, DeepSC-S is more robust to channel variations, especially in the low signal-to-noise (SNR) regime.

• The simulation results demonstrate that our proposed DeepSC-S outperforms the traditional communi- cations in both cases in terms of the speech signals metrics, such as signal-to-distortion ration and perceptual evaluation of speech distortion.
• INTRODUCTION I NSPIRED by the success in various areas, deep learning (DL) has been considered as a promising candidate for communications to achieve higher system performance with more intelligence [2], [3].
• Shannon and Weaver [9] categorized communications into three levels: • Level A: how accurately can the symbols of communica- tion be transmitted? (The technical problem) • Level B: how precisely do the transmitted symbols convey the desired meaning? (The semantic problem) • Level C: how effectively does the received meaning affect conduct in the desired way? (The effectiveness problem) This indicates the feasibility to transmit the semantic informa- tion, instead of the bits or symbols, to achieve higher system efficiency.

The traditional communications transmit all the source data represented by bits, regardless of the content of source and the semantic information required by the receiver. However, in some applications, the receiver only needs part of the source data that represents critical semantic information, which prompts to transmit the application-related information, especially when bandwidth resources are limited. In this pa- per, we consider a semantic communication system for speech recognition by designing the transceiver as an end-to-end (E2E) system. Particularly, a deep learning (DL)-enabled semantic communication system, named DeepSC-SR, is developed to learn and extract text-related semantic features at the transmitter, which motivates the system to transmit much less than the source speech data without performance degradation. Moreover, in order to facilitate the proposed DeepSC-SR for dynamic channel environments, we investigate a robust model to cope with various channel environments without requiring retraining. The simulation results demonstrate that our proposed DeepSC- SR outperforms the traditional communication systems in terms of the speech recognition metrics, such as character-error-rate and word-error-rate, and is more robust to channel variations, especially in the low signal-to-noise (SNR) regime.

• The simulation results demonstrate that our proposed DeepSC- SR outperforms the traditional communication systems in terms of the speech recognition metrics, such as character-error-rate and word-error-rate, and is more robust to channel variations, especially in the low signal-to-noise (SNR) regime.
• Particularly, the DL-enabled intelligent communications have achieved lots of successes in physical layer communications [2], [3] and wireless resource allocations [4], [5].
• The communication systems utilizing DL techniques are typically designed to transmit digital bit sequences and op- timized by minimizing the bit-error rate (BER) or symbol- error rate (SER), which achieves the first level communications according to the categorization by Shannon and Weaver [6].

We consider a semantic communication system for speech signals, named DeepSC-S. Motivated by the break- throughs in deep learning (DL), we make an effort to recover the transmitted speech signals in the semantic communication systems, which minimizes the error at the semantic level rather than the bit level or symbol level as in the traditional commu- nication systems. Particularly, based on an attention mechanism employing squeeze-and-excitation (SE) networks, we design the transceiver as an end-to-end (E2E) system, which learns and extracts the essential speech information. Furthermore, in order to facilitate the proposed DeepSC-S to work well on dynamic practical communication scenarios, we find a model yielding good performance when coping with various channel environments without retraining process. The simulation results demonstrate that our proposed DeepSC-S is more robust to channel varia- tions and outperforms the traditional communication systems, especially in the low signal-to-noise (SNR) regime. Index Terms—Deep learning, end-to-end communication, semantic communication, speech transmission, squeeze-and- excitation networks.

• The simulation results demonstrate that our proposed DeepSC-S is more robust to channel varia- tions and outperforms the traditional communication systems, especially in the low signal-to-noise (SNR) regime.
• This indicates the feasibility to transmit the semantic information, instead of the bits or symbols, to achieve higher system efficiency.
• Particularly, an initial research on semantic communication systems for text information has been developed [13], which mitigates the semantic error to achieve Nash equilibrium.

Semantic communication, regarded as the breakthrough beyond the Shannon paradigm, aims at the successful transmission of semantic information conveyed by the source rather than the accurate reception of each single symbol or bit regardless of its meaning. This article provides an overview on semantic communications. After a brief review of Shannon information theory, we discuss semantic communications with theory, framework, and system design enabled by deep learning. Different from the symbol/bit error rate used for measuring conventional communication systems, performance metrics for semantic communications are also discussed. The article concludes with several open questions in semantic communications.

• Next, we present the developments of DL-enabled semantic com- munications for multimodal data transmission, including text, image, and audio.
• The semantic compression can achieve higher transmission rate.
• To achieve a tradeoff between the transmission accuracy and the number of symbols used for each message, a metric [27] has been designed as γ = 1 N × (1 −ψ (s,ˆs)) , (17) where N is the number of symbols per message and ψ (s,ˆs) is the semantic error between s and ˆs.

Semantic communications focus on the transmission of semantic features. In this letter, we consider a task-oriented multi-user semantic communication system for multimodal data transmission. Particularly, partial users transmit images while the others transmit texts to inquiry the information about the images. To exploit the correlation among the multimodal data from mul- tiple users, we propose a deep neural network enabled semantic communication system, named MU-DeepSC, to execute the visual question answering (VQA) task as an example. Specifically, the transceiver for MU-DeepSC is designed and optimized jointly to capture the features from the correlated multimodal data for task-oriented transmission. Simulation results demonstrate that the proposed MU-DeepSC is more robust to channel variations than the traditional communication systems, especially in the low signal-to-noise (SNR) regime.

• To exploit the correlation among the multimodal data from mul- tiple users, we propose a deep neural network enabled semantic communication system, named MU-DeepSC, to execute the visual question answering (VQA) task as an example.
• Simulation results demonstrate that the proposed MU-DeepSC is more robust to channel variations than the traditional communication systems, especially in the low signal-to-noise (SNR) regime.
• In this letter, we present our initial results in multi-user semantic communication for multimodal data.

The sixth generation (6G) mobile networks will reshape the world by offering instant, efficient, and intelligent hyper-connectivity, as envisioned by the previously proposed Ubiquitous-X 6G networks. Such hyper-massive and global connectivity will introduce tremendous challenges into the operation and management of 6G networks, calling for revolutionary theories and technological innovations. To this end, we propose a new route to boost network capabilities toward a wisdom-evolutionary and primitive-concise network (WePCN) vision for the Ubiquitous-X 6G network. In particular, we aim to concretize the evolution path toward the WePCN by first conceiving a new semantic representation framework, namely semantic base, and then establishing an intelligent and efficient semantic communication (IE-SC) network architecture. In the IE-SC architecture, a semantic intelligence plane is employed to interconnect the semantic-empowered physical-bearing layer, network protocol layer, and application-intent layer via semantic information flows. The proposed architecture integrates artificial intelligence and network technologies to enable intelligent interactions among various communication objects in 6G. It features a lower bandwidth requirement, less redundancy, and more accurate intent identification. We also present a brief review of recent advances in semantic communications and highlight potential use cases, complemented by a range of open challenges for 6G.

• 未从全文自动抽取到足够稳定的结果句；请结合摘要与原文。

With the advent of the 6G era, the concept of seman- tic communication has attracted increasing attention. Compared with conventional communication systems, semantic communica- tion systems are not only affected by physical noise existing in the wireless communication environment, e.g., additional white Gaussian noise, but also by semantic noise due to the source and the nature of deep learning-based systems. In this paper, we elaborate on the mechanism of semantic noise. In particular, we categorize semantic noise into two categories: literal semantic noise and adversarial semantic noise. The former is caused by written errors or expression ambiguity, while the latter is caused by perturbations or attacks added to the embedding layer via the semantic channel. To prevent semantic noise from influencing semantic communication systems, we present a robust deep learning enabled semantic communication system (R-DeepSC) that leverages a calibrated self-attention mechanism and adver- sarial training to tackle semantic noise. Compared with baseline models that only consider physical noise for text transmission, the proposed R-DeepSC achieves remarkable performance in dealing with semantic noise under different signal-to-noise ratios.

• To prevent semantic noise from influencing semantic communication systems, we present a robust deep learning enabled semantic communication system (R-DeepSC) that leverages a calibrated self-attention mechanism and adver- sarial training to tackle semantic noise.
• Compared with baseline models that only consider physical noise for text transmission, the proposed R-DeepSC achieves remarkable performance in dealing with semantic noise under different signal-to-noise ratios.
• To combat the semantic noise, we propose a robust deep learning en- abled semantic communication system named R-DeepSC. • For the literal semantic noise, we tailor the transformer- based model and present a calibrated self-attention mech- anism for error correction to ensure the semantic fidelity. • For the adversarial semantic noise, we adopt an adversar- ial training method to train the system.

The task-oriented semantic communication sys- tems have achieved significant performance gain, however, the paradigm that employs a model for a specific task might be limited, since the system has to be updated once the task is changed or multiple models are stored for serving various tasks. To address this issue, we firstly propose a unified deep learning enabled semantic communication system (U-DeepSC), where a unified model is developed to serve various transmission tasks. To jointly serve these tasks in one model with fixed parameters, we employ domain adaptation in the training procedure to specify the task-specific features for each task. Thus, the system only needs to transmit the task-specific features, rather than all the features, to reduce the transmission overhead. Moreover, since each task is of different difficulty and requires different number of layers to achieve satisfactory performance, we develop the multi-exit architecture to provide early-exit results for relatively simple tasks. In the experiments, we employ a proposed U- DeepSC to serve five tasks with multi-modalities. Simulation re- sults demonstrate that our proposed U-DeepSC achieves compa- rable performance to the task-oriented semantic communication system designed for a specific task with significant transmission overhead reduction and much less number of model parameters.

• Moreover, since each task is of different difficulty and requires different number of layers to achieve satisfactory performance, we develop the multi-exit architecture to provide early-exit results for relatively simple tasks.
• Simulation re- sults demonstrate that our proposed U-DeepSC achieves compa- rable performance to the task-oriented semantic communication system designed for a specific task with significant transmission overhead reduction and much less number of model parameters.
• To address this issue, semantic communications have been considered as a promising technology to achieve better performance [2], [3].

Communication systems to date primarily aim at reliably communicating bit sequences. Such an approach provides efficient engi- neering designs that are agnostic to the meanings of the messages or to the goal that the message exchange aims to achieve. Next generation systems, however, can be potentially enriched by folding message semantics and goals of communication into their design. Further, these systems can be made cognizant of the context in which communication exchange takes place, providing avenues for novel design insights. This tutorial summarizes the efforts to date, starting from its early adaptations, semantic-aware and task-oriented communications, covering the foun- dations, algorithms and potential implementations. The focus is on approaches that utilize information theory to provide the foundations, as well as the significant role of learning in semantics and task-aware communications.

• Such an approach provides efficient engi- neering designs that are agnostic to the meanings of the messages or to the goal that the message exchange aims to achieve.
• Some of its properties follow from well-known experimental results in hearing, e.g., the ear is relatively insensitive to phase and the sensitivity to amplitude and frequency is roughly logarithmic.” Here, Shannon uses the term “evaluation” to refer to different fidelity measures.
• This measure in general will depend on the underlying goal of the communication, the ‘significance’ of the source signal to be communicated for this goal, and the ‘fidelity’ of the reconstruction in achieving this goal.

With the aim of accomplishing intelligence tasks, semantic communications transmit task-related information only, yielding significant performance gains over conventional com- munications. To guarantee user requirements for different types of tasks, we perform the semantic-aware resource allocation in a multi-cell multi-task network in this paper. Specifically, an approximate measure of semantic entropy is first developed to quantify the semantic information for different tasks, based on which a novel quality-of-experience (QoE) model is proposed. We formulate the QoE-aware semantic resource allocation in terms of the number of transmitted semantic symbols, channel assignment, and power allocation. To solve this problem, we first decouple it into two independent subproblems. The first one is to optimize the number of transmitted semantic symbols with given channel assignment and power allocation, which is solved by the exhaustive searching method. The second one is the channel assignment and power allocation subproblem, which is modeled as a many-to-one matching game and solved by the proposed low- complexity matching algorithm. Simulation results demonstrate the effectiveness and superiority of the proposed method on the overall QoE.

• Simulation results demonstrate the effectiveness and superiority of the proposed method on the overall QoE.
• Specifically, based on the developed approximate semantic entropy, a novel QoE model is used to formulate the optimization problem in terms of the number of transmitted semantic symbols, channel assignment, and power allocation. • The formulated problem is decoupled into two sub- problems, which are solved by the exhaustive search- ing method and a low-complexity matching algorithm, respectively. • Simulation results verify the superiority of the proposed QoE-aware semantic resource allocation method in terms of the overall QoE against the baselines.
• In particular, the proposed algorithm aims to keep the utility of users increasing by the swap operation to achieve the stable matching for the first case, while focuses on the utility of channels for the second case.

Semantic communications have shown its great potential to improve the transmission reliability, especially in the low signal-to-noise regime. However, resource allocation for semantic communications still remains unexplored, which is a critical issue in guaranteeing the semantic transmission reliability and the communication efficiency. To fill this gap, we investigate the spectral efficiency in the semantic domain and rethink the semantic-aware resource allocation issue. Specifically, taking text semantic communication as an example, the semantic spectral efficiency (S-SE) is defined for the first time, and is used to optimize resource allocation in terms of channel assignment and the number of transmitted semantic symbols. Additionally, for fair comparison of semantic and conventional communication systems, a transform method is developed to convert the con- ventional bit-based spectral efficiency to the S-SE. Simulation results demonstrate the validity and feasibility of the proposed resource allocation method, as well as the superiority of semantic communications in terms of the S-SE.

• Simulation results demonstrate the validity and feasibility of the proposed resource allocation method, as well as the superiority of semantic communications in terms of the S-SE.
• Then a new formulation is proposed and solved to max- imize the overall S-SE in terms of channel assignment and the number of transmitted semantic symbols. • To make a fair comparison between semantic and con- ventional communication systems, a transform method is developed to convert the bit-based SE to the S-SE. • Simulation results verify the effectiveness of the proposed resource allocation model, as well as the superiority of semantic communication systems in terms of the S-SE.
• Section IV introduces a transform method for fair comparison of semantic and conventional communication systems and presents the simulation results.

Although the semantic communications have exhib- ited satisfactory performance in a large number of tasks, the impact of semantic noise and the robustness of the systems have not been well investigated. Semantic noise is a particular kind of noise in semantic communication systems, which refers to the misleading between the intended semantic symbols and received ones. In this paper, we first propose a framework for the robust end-to-end semantic communication systems to combat the semantic noise. Particularly, we analyze the causes of semantic noise and propose a practical method to generate it. To remove the effect of semantic noise, adversarial training is proposed to incorporate the samples with semantic noise in the training dataset. Then, the masked autoencoder (MAE) is designed as the architecture of a robust semantic communication system, where a portion of the input is masked. To further improve the robustness of semantic communication systems, we firstly employ the vector quantization-variational autoen- coder (VQ-VAE) to design a discrete codebook shared by the transmitter and the receiver for encoded feature representation. Thus, the transmitter simply needs to transmit the indices of these features in the codebook. Simulation results show that our proposed method significantly improves the robustness of semantic communication systems against semantic noise with significant reduction on the transmission overhead.

• Simulation results show that our proposed method significantly improves the robustness of semantic communication systems against semantic noise with significant reduction on the transmission overhead.
• In this paper, we propose a DL-enabled end-to-end robust semantic communication system to combat the semantic noise and our main contributions are summarized as follows. • We firstly model the semantic noise by employing an iterative FGSM to instantly generate the semantic noise.
• The transmitter simply needs to send the indices of the features in the codebook to the receiver, which significantly reduces the transmission overhead. • Simulation results show that our proposed method sig- nificantly improves the robustness of semantic commu- nication systems against semantic noise with significant reduction on the transmission overhead.

As a key technology in metaversa, wireless ultimate extended reality (XR) has attracted extensive attentions from both industry and academia. However, the stringent latency and ultra-high data rates requirements have hindered the de- velopment of wireless ultimate XR. Instead of transmitting the original source data bit-by-bit, semantic communications focus on the successful delivery of semantic information contained in the source, which have shown great potentials in reduc- ing the data traffic of wireless systems. Inspired by semantic communications, this article develops a joint semantic sensing, rendering, and communication framework for wireless ultimate XR. In particular, semantic sensing is used to improve the sensing efficiency by exploring the spatial-temporal distributions of semantic information. Semantic rendering is designed to reduce the costs on semantically-redundant pixels. Next, semantic communications are adopted for high data transmission efficiency in wireless ultimate XR. Then, two case studies are provided to demonstrate the effectiveness of the proposed framework. Finally, potential research directions are identified to boost the development of semantic-aware wireless ultimate XR.

• However, latency in nearly all components in a wireless XR system, including sensing, ren- dering, and communication parts makes the ultra-low latency difficult to achieve when generating and transmitting virtual contents with high resolutions.
• Secondly, the data transmission rate is expected to exceed tens of Gbps in ultimate XR [2], which is far above the achievable capacity of the existing wireless networks with peak from 0.1 Gbps to 2.0 Gbps [1].
• In this article, we propose a framework, including semantic sensing, semantic rendering, and semantic communications, to support wireless ultimate XR.

The vigorous developments of Internet of Things make it possible to extend its computing and storage capabilities to computing tasks in the aerial system with collaboration of cloud and edge, especially for artificial intelligence (AI) tasks based on deep learning (DL). Collecting a large amount of image/video data, Unmanned aerial vehicles (UAVs) can only handover intelligent analysis tasks to the back-end mobile edge computing (MEC) server due to their limited storage and computing capabilities. How to efficiently transmit the most correlated information for the AI model is a challenging topic. Inspired by the task-oriented communication in recent years, we propose a new aerial image transmission paradigm for the scene classification task. A lightweight model is developed on the front- end UAV for semantic blocks transmission with perception of images and channel conditions. In order to achieve the tradeoff between transmission latency and classification accuracy, deep reinforcement learning (DRL) is used to explore the semantic blocks which have the best contribution to the back-end classifier under various channel conditions. Experimental results show that the proposed method can significantly improve classification accu- racy compared to the fixed transmission strategy and traditional content perception methods.

• Inspired by the task-oriented communication in recent years, we propose a new aerial image transmission paradigm for the scene classification task.
• In order to achieve the tradeoff between transmission latency and classification accuracy, deep reinforcement learning (DRL) is used to explore the semantic blocks which have the best contribution to the back-end classifier under various channel conditions.
• Experimental results show that the proposed method can significantly improve classification accu- racy compared to the fixed transmission strategy and traditional content perception methods.

While semantic communications have shown the potential in the case of single-modal single-users, its applica- tions to the multi-user scenario remain limited. In this paper, we investigate deep learning (DL) based multi-user semantic communication systems for transmitting single-modal data and multimodal data, respectively. We will adopt three intelligent tasks, including, image retrieval, machine translation, and visual question answering (VQA) as the transmission goal of semantic communication systems. We will then propose a Transformer based unique framework to unify the structure of transmitters for different tasks. For the single-modal multi-user system, we will propose two Transformer based models, named, DeepSC-IR and DeepSC-MT, to perform image retrieval and machine translation, respectively. In this case, DeepSC-IR is trained to optimize the distance in embedding space between images and DeepSC-MT is trained to minimize the semantic errors by recovering the seman- tic meaning of sentences. For the multimodal multi-user system, we develop a Transformer enabled model, named, DeepSC-VQA, for the VQA task by extracting text-image information at the transmitters and fusing it at the receiver. In particular, a novel layer-wise Transformer is designed to help fuse multimodal data by adding connection between each of the encoder and decoder layers. Numerical results will show that the proposed models are superior to traditional communications in terms of the robustness to channels, computational complexity, transmission delay, and the task-execution performance at various task-specific metrics.

• For the multimodal multi-user system, we develop a Transformer enabled model, named, DeepSC-VQA, for the VQA task by extracting text-image information at the transmitters and fusing it at the receiver.
• Such scenarios are achieved by collecting multimodal data from the various sensors so as to provide the information in a complementary manner and fuse them at the server/cloud.
• The main contributions of this paper are summarized as follows: • We propose a Transformer [24] based transmitter struc- ture, which is applicable for both text and image trans- mission by effectively extracting semantic information for different tasks.

Semantic communication is regarded as the break- through beyond the Shannon paradigm, which transmits the semantic information only to improve the communication effi- ciency significantly. This article first introduces a framework for the generalized semantic communication system, which exploits the semantic information in both the multimodal source and the wireless channel environment. Subsequently, the deep learning enabled end-to-end semantic communications and the environ- ment semantics aided wireless communications are demonstrated through two user cases. The article is concluded with several re- search challenges to boost the development of such a generalized semantic communication system.

• In this article, we propose a generalized semantic commu- nications to exploit semantics in both sources and channels, which takes a different view from existing works on semantic communications.
• Particularly, Shannon channel capacity provides the achievable upper bound for a point-to- point communication system.
• According to a semantic knowledge base shared by the transmitter and the receiver, the semantic representations can be achieved by a DL-enabled joint semantic-channel coding scheme before transmitting over physical channels, which could provide high robustness to both channel impairment and semantic noise.

In this paper, we develop a deep learning based semantic communication system for speech trans- mission, named DeepSC-ST. We take the speech recog- nition and speech synthesis as the transmission tasks of the communication system, respectively. First, the speech recognition-related semantic features are extracted for transmission by a joint semantic-channel encoder and the text is recovered at the receiver based on the received semantic features, which significantly reduces the required amount of data transmission without performance degra- dation. Then, we perform speech synthesis at the receiver, which dedicates to re-generate the speech signals by feeding the recognized text and the speaker information into a neural network module. To enable the DeepSC-ST adaptive to dynamic channel environments, we identify a robust model to cope with different channel conditions. According to the simulation results, the proposed DeepSC- ST significantly outperforms conventional communication systems and existing DL-enabled communication systems, especially in the low signal-to-noise ratio (SNR) regime. A software demonstration is further developed as a proof-of- concept of the DeepSC-ST.

• 1 Deep Learning Enabled Semantic Communications with Speech Recognition and Synthesis Zhenzi Weng, Zhijin Qin, Xiaoming Tao, Chengkang Pan, Guangyi Liu, and Geoffrey Ye Li Abstract—In this paper, we develop a deep learning based semantic communication system for speech trans- mission, named DeepSC-ST.
• According to the simulation results, the proposed DeepSC- ST significantly outperforms conventional communication systems and existing DL-enabled communication systems, especially in the low signal-to-noise ratio (SNR) regime.
• To achieve the data reconstruction, the global semantic information is extracted for transmission.

While semantic communications succeed in effec- tively transmitting due to the strong capability to extract the essential semantic information, it is still far from intelligent communications. In this paper, we introduce an essential com- ponent, memory, into semantic communications to mimic human communications. Particularly, we propose a deep learning (DL) based semantic communication system with memory, named Mem-DeepSC, by considering the scenario question answer as the task at the receiver. We exploit universal Transformer based transceiver to extract the semantic information and introduce the memory module to enhance the semantic decoding capability at the receiver. Moreover, we derive the semantic channel capacity and propose a consecutive dynamic transmission method to minimize the transmission latency. Numerical results show that Mem-DeepSC is superior to benchmarks in terms of answer accuracy and the number of transmitted symbols.

• Particularly, we propose a deep learning (DL) based semantic communication system with memory, named Mem-DeepSC, by considering the scenario question answer as the task at the receiver.
• Particu- larly, Xie et al. [2] proposed a DL enabled joint semantic- channel coding scheme, named DeepSC, to support the text semantic information transmission, which achieves significant performance gain.
• Particularly, we develop a DL enabled semantic communication system with memory (Mem-DeepSC) to address the aforementioned chal- lenges.

The sixth generation (6G) mobile communication systems could serve new multimodal services, such as virtual reality (VR), augmented reality (AR), holographic projection. These new services are accompanied by new demands for intelligence, personalization and collaboration. Semantic communications are increasingly attracting attention due to its potential to support the aforementioned multimodal services. Therefore, in this article, we discuss key technologies for semantic communications, which involve semantic representation, semantic encoding and decoding, and transmission of semantic information. First, we present existing work on semantic communications, such as sketch graph structure representation of images, semantic reconstruction, and end-to-end semantic communication systems. Second, we identify challenges in semantic communication systems and propose the unified semantic representations, semantic codecs, and multiple access techniques for support the semantic information transmission. Last, we present the results of semantic encoding and decoding methods and the coexistence of semantics and bits transmission supported by the non-orthogonal multiple access technique (NOMA).

• First, we present existing work on semantic com- munications, such as sketch graph structure repre- sentation of images, semantic reconstruction, and end-to-end semantic communication systems.
• Last, we present the results of semantic encoding and decoding methods and the coexistence of semantics and bits transmission supported by the non-orthogonal multiple access technique (NOMA).
• In this article, we propose a framework of semantic communications for multimedia data.

By transmitting task-related information only, se- mantic communications yield significant performance gains over conventional communications. However, the lack of mature se- mantic theory about semantic information quantification and performance evaluation makes it challenging to perform resource allocation for semantic communications, especially when multiple tasks coexist in the network. To cope with this challenge, we propose a quality-of-experience (QoE) based semantic-aware resource allocation method for multi-task networks in this paper. First, semantic entropy is defined to quantify the semantic information for different tasks, and the relationship between semantic entropy and Shannon entropy is analyzed. Then, we develop a novel QoE model to formulate the semantic-aware resource allocation in terms of semantic compression, channel assignment, and transmit power. The compatibility of the for- mulated problem with conventional communications is further demonstrated. To solve this problem, we decouple it into two subproblems and solved them by a developed deep Q-network (DQN) based method and a proposed low-complexity matching algorithm, respectively. Finally, simulation results validate the effectiveness and superiority of the proposed method, as well as its compatibility with conventional communications. Index Terms—Semantic entropy, quality-of-experience, semantic-aware resource allocation, multi-task networks.

• To cope with this challenge, we propose a quality-of-experience (QoE) based semantic-aware resource allocation method for multi-task networks in this paper.
• Then, we develop a novel QoE model to formulate the semantic-aware resource allocation in terms of semantic compression, channel assignment, and transmit power.
• Finally, simulation results validate the effectiveness and superiority of the proposed method, as well as its compatibility with conventional communications.

Although semantic communications have exhibited satisfactory performance on a large number of tasks, the impact of semantic noise and the robustness of the systems have not been well investigated. Semantic noise refers to the misleading between the intended semantic symbols and received ones, thus causes the failure of tasks. In this paper, we first propose a framework for the robust end-to-end semantic communication systems to combat the semantic noise. In particular, we analyze sample-dependent and sample-independent semantic noise. To combat the seman- tic noise, the adversarial training with weight perturbation is developed to incorporate the samples with semantic noise in the training dataset. Then, we propose to mask a portion of the input, where the semantic noise appears frequently, and design the masked vector quantized-variational autoencoder (VQ-VAE) with the noise-related masking strategy. We use a discrete codebook shared by the transmitter and the receiver for encoded feature representation. To further improve the system robustness, we develop a feature importance module (FIM) to suppress the noise-related and task-unrelated features. Thus, the transmit- ter simply needs to transmit the indices of these important task-related features in the codebook. Simulation results show that the proposed method can be applied in many downstream tasks and significantly improve the robustness against semantic noise with remarkable reduction on the transmission overhead.

• Then, we propose to mask a portion of the input, where the semantic noise appears frequently, and design the masked vector quantized-variational autoencoder (VQ-VAE) with the noise-related masking strategy.
• To further improve the system robustness, we develop a feature importance module (FIM) to suppress the noise-related and task-unrelated features.
• Simulation results show that the proposed method can be applied in many downstream tasks and significantly improve the robustness against semantic noise with remarkable reduction on the transmission overhead.

Internet of Vehicles (IoV) is expected to become the central infrastructure to provide advanced services to connected vehicles and users for higher transportation efficiency and secu- rity. A variety of emerging applications/services bring explosively growing demands for mobile data traffic between connected vehicles and roadside units (RSU), imposing the significant challenge of spectrum scarcity to IoV. In this paper, we propose a cooperative semantic-aware architecture to convey essential semantics from collaborated users to servers for lowering the data traffic. In contrast to current solutions that are mainly based on piling up highly complex signal processing techniques and multiple access capabilities in terms of syntactic communications, this paper puts forth the idea of semantic-aware content delivery in IoV. Specifically, the successful transmission of essential semantics of the source data is pursued, rather than the accurate reception of symbols regardless of its meaning as in conventional syntactic communications. To assess the benefits of the proposed architecture, we provide a case study of the image retrieval task for vehicles in intelligent transportation systems. Simulation results demonstrate that the proposed architecture outperforms the existing solutions with fewer radio resources, especially in a low signal-to-noise-ratio (SNR) regime, which can shed light on the potential of the proposed architecture in extending the applications in extreme environments.

• In this paper, we propose a cooperative semantic-aware architecture to convey essential semantics from collaborated users to servers for lowering the data traffic.
• Simulation results demonstrate that the proposed architecture outperforms the existing solutions with fewer radio resources, especially in a low signal-to-noise-ratio (SNR) regime, which can shed light on the potential of the proposed architecture in extending the applications in extreme environments.
• By jointly optimizing the semantic and channel coding, the point-to-point semantic transmission for text, image, and speech is achieved, outperforming the conven- tional syntactic-based system, especially in the low signal-to- noise (SNR) regime.

While semantic communication succeeds in effi- ciently transmitting due to the strong capability to extract the essential semantic information, it is still far from the intelligent or human-like communications. In this paper, we introduce an essential component, memory, into semantic communications to mimic human communications. Particularly, we investigate a deep learning (DL) based semantic communication system with memory, named Mem-DeepSC, by considering the scenario question answer task. We exploit the universal Transformer based transceiver to extract the semantic information and introduce the memory module to process the context information. Moreover, we derive the relationship between the length of semantic signal and the channel noise to validate the possibility of dynamic transmis- sion. Specially, we propose two dynamic transmission methods to enhance the transmission reliability as well as to reduce the communication overheads by masking some unessential elements, which are recognized through training the model with mutual information. Numerical results show that the proposed Mem- DeepSC is superior to benchmarks in terms of answer accuracy and transmission efficiency, i.e., number of transmitted symbols.

• Specially, we propose two dynamic transmission methods to enhance the transmission reliability as well as to reduce the communication overheads by masking some unessential elements, which are recognized through training the model with mutual information.
• Particularly, we develop a DL enabled semantic communication system with memory (Mem-DeepSC) to ad- dress the aforementioned challenges.
• Finally, the third step is to optimize the entire system jointly to achieve the global optimization.

Wireless extended reality (XR) has attracted wide attentions as a promising technology to improve users’ mobility and quality of experience. However, the ultra-high data rate requirement of wireless XR has hindered its development for many years. To overcome this challenge, we develop a semantic communication framework, where semantically-unimportant in- formation is highly-compressed or discarded in semantic coders, significantly improving the transmission efficiency. Besides, con- sidering the fact that some source content may have less amount of semantic information or have higher tolerance to channel noise, we propose a universal variable-length semantic-channel coding method. In particular, we first use a rate allocation network to estimate the best code length for semantic information and then adjust the coding process accordingly. By adopting some proxy functions, the whole framework is trained in an end-to- end manner. Numerical results show that our semantic system significantly outperforms traditional transmission methods and the proposed variable-length coding scheme is superior to the fixed-length coding methods.

• To overcome this challenge, we develop a semantic communication framework, where semantically-unimportant in- formation is highly-compressed or discarded in semantic coders, significantly improving the transmission efficiency.
• Besides, con- sidering the fact that some source content may have less amount of semantic information or have higher tolerance to channel noise, we propose a universal variable-length semantic-channel coding method.
• Numerical results show that our semantic system significantly outperforms traditional transmission methods and the proposed variable-length coding scheme is superior to the fixed-length coding methods.

Deep learning based image compressed sensing (CS) has achieved great success. However, existing CS systems mainly adopt a fixed measurement matrix to images, ignoring the fact the optimal mea- surement numbers and bases are different for different images. To further improve the sensing efficiency, we propose a novel semantic- aware image CS system. In our system, the encoder first uses a fixed number of base CS measurements to sense different images. Accord- ing to the base CS results, the encoder then employs a policy network to analyze the semantic information in images and determines the measurement matrix for different image areas. At the decoder side, a semantic-aware initial reconstruction network is developed to deal with the changes of measurement matrices used at the encoder. A rate-distortion training loss is further introduced to dynamically ad- just the average compression ratio for the semantic-aware CS system and the policy network is trained jointly with the encoder and the decoder in an end-to-end manner by using some proxy functions. Numerical results show that the proposed semantic-aware image CS system is superior to the traditional ones with fixed measurement matrices.

• 2023 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 17–20, 2023, ROME, ITALY SEMANTIC-AWARE IMAGE COMPRESSED SENSING Bowen Zhang⋆ Zhijin Qin† Geoffrey Ye Li⋆ ⋆Department of Electrical and Electronic Engineering, Imperial College London, London, UK † Department of Electronic Engineering, Tsinghua University, Beijing, China ABSTRACT Deep learning based image compressed sensing (CS) has achieved great success.
• To further improve the sensing efficiency, we propose a novel semantic- aware image CS system.

Semantic communications have been utilized to execute numerous intelligent tasks by transmitting task-related semantic information instead of bits. In this paper, we propose a semantic-aware speech-to-text transmission system over MIMO channels with single-user, named SAC-ST. Particularly, a se- mantic communication system to serve the speech-to-text task at the receiver is first designed, which compresses the semantic information and generates the text-related semantic features by leveraging the transformer module. Moreover, a novel neural network-enabled semantic-aware network is proposed to facilitate the transmission with high semantic fidelity, which identifies the critical semantic information and guarantees them to be recovered accurately. According to the simulation results, the proposed SAC-ST outperforms the communication framework without the semantic-aware network for speech-to-text transmis- sion over MIMO channels in terms of the speech-to-text metrics, especially in the low signal-to-noise regime.

• In this paper, we propose a semantic-aware speech-to-text transmission system over MIMO channels with single-user, named SAC-ST.
• According to the simulation results, the proposed SAC-ST outperforms the communication framework without the semantic-aware network for speech-to-text transmis- sion over MIMO channels in terms of the speech-to-text metrics, especially in the low signal-to-noise regime.
• The conventional communications, i.e., the first level of communications, convert the input message into the bit se- quence and aim to achieve lower bit-error rate (BER) using the proper coding and decoding algorithms.

Semantic communications have been regarded as a promising solution for the next generation communication systems to alleviate the spectral resource shortage and the network congestion. Existing image semantic communication systems extract and transmit global semantics, which can cope with the downstream data reconstruction or intelligent tasks. However, the semantics involved in these methods are still severely redundant and uninterpretable. In this work, we propose a novel task-oriented semantic communication framework based on scene graph, named DeepSC-SG. Specifically, we first devise a scene graph based semantic encoder, which extracts the explainable scene graph semantics from the input images and encodes the semantics into informative graph embeddings. Then we design a joint source-channel (JSC) codec to combat physical channel impairment. After receiving the semantics, a semantic decoder is devised to achieve the downstream image retrieval task by computing scene graph similarities. Simulation results demonstrate that the proposed DeepSC-SG is fairly robust to the channel variations compared to the traditional communication systems, which has great potential in realizing downstream intelligent tasks like image retrieval with significantly reduced size of transmitted data.

• In this work, we propose a novel task-oriented semantic communication framework based on scene graph, named DeepSC-SG.
• After receiving the semantics, a semantic decoder is devised to achieve the downstream image retrieval task by computing scene graph similarities.
• Simulation results demonstrate that the proposed DeepSC-SG is fairly robust to the channel variations compared to the traditional communication systems, which has great potential in realizing downstream intelligent tasks like image retrieval with significantly reduced size of transmitted data.

Semantic communications execute intelligent tasks at the receiver by only transmitting necessary information. In this paper, we introduce TOS-ST, a task-oriented semantic communication system for speech transmission, which efficiently serves the semantic tasks at the receiver, including speech-to- text translation and speech-to-speech translation. Particularly, TOS-ST condenses the input speech in the source language and extracts the task-related semantics features prior to transmis- sion. At the receiver, these features are recovered and utilized by the neural network-based semantic preserver and machine translation module to generate the uncorrupted text in the target language. To perform the speech-to-speech translation task, the translated text passes through a sophisticated neural network to obtain speech in the target language. According to the simulation results, the TOS-ST outperforms conventional speech transmission systems and exhibits higher robustness against channel impairment.

• According to the simulation results, the TOS-ST outperforms conventional speech transmission systems and exhibits higher robustness against channel impairment.
• Huang et al. [7] designed a generative adversarial network-enabled semantic coding architecture to achieve image reconstruction according to the rate-semantic- perceptual criterion.
• Weng et al. [9] investigated an efficient speech transmission system, named DeepSC-ST, by leveraging the recurrent neural network (RNN) to process the semantic infor- mation within lengthy speech sequences, which dramatically lowers the transmission data and achieves speech recognition and speech synthesis tasks at the receiver.

Although analog semantic communication systems have received considerable attention in the literature, there is less work on digital semantic communication systems. In this letter, we develop a deep learning (DL)-enabled vector quantized (VQ) semantic communication system for image transmission, named VQ-DeepSC. Specifically, we propose a convolutional neural network (CNN)-based transceiver to extract multi-scale semantic features of images and introduce multi-scale semantic embedding spaces to perform semantic feature quantization, ren- dering the data compatible with digital communication systems. Furthermore, we employ adversarial training to improve the quality of received images by introducing a PatchGAN dis- criminator. Experimental results demonstrate that the proposed VQ-DeepSC is more robustness than BPG in digital communica- tion systems and has comparable MS-SSIM performance to the DeepJSCC method.

• In this letter, we develop a deep learning (DL)-enabled vector quantized (VQ) semantic communication system for image transmission, named VQ-DeepSC.
• Specifically, we propose a convolutional neural network (CNN)-based transceiver to extract multi-scale semantic features of images and introduce multi-scale semantic embedding spaces to perform semantic feature quantization, ren- dering the data compatible with digital communication systems.
• Experimental results demonstrate that the proposed VQ-DeepSC is more robustness than BPG in digital communica- tion systems and has comparable MS-SSIM performance to the DeepJSCC method.

Recently, semantic communication (SC) has been re- garded as one of the most potential paradigms of 6G. Current SC frameworks require the physical layer channel state information (CSI) in order to handle the severe signal distortion induced by channel fading. Since practical CSI cannot be obtained accurately and the overhead of channel estimation cannot be neglected, we therefore propose a generative adversarial network (GAN) based SC framework (Ti-GSC) that doesn’t require CSI. In Ti-GSC, two main modules, i.e., an autoencoder-based encoder-decoder module (AEDM) and a GAN-based non-CSI signal distortion suppression (SDS) module (GSDSM) are included where AEDM first encodes the data in the semantic dimension at the source be- fore transmission, and then GSDSM suppresses the distortion of the received signals in both syntactic and semantic dimensions at the destination. At last, AEDM decodes the distortion-suppressed signal at the destination. As SDS only relies on learning the syntactic distribution and the semantics of the transmitted data, no prior information such as CSI is needed by GSDSM. In order to measure signal distortion, a novel loss function is proposed where two terms, i.e., a syntactic distortion loss term and a semantic distortion loss term are newly added, and a differentiable semantic measurement method is designed based on the intermediate layers of the AEDM decoder. To achieve better training results of Ti-GSC , two training schemes, i.e., the joint optimization based training (JOT) and the alternating optimization based training (AOT) are designed for the proposed Ti-GSC. Experimental results show that JOT is more efficient for Ti-GSC, and Ti-GSC outperforms conventional communication frameworks in terms of bilingual evaluation understudy (BLEU) score in both Rician and Rayleigh fading channels. Moreover, without CSI, BLEU score achieved by Ti-GSC is about 40% and 62% higher than that achieved by existing SC frameworks in Rician and Rayleigh fading, respectively. Besides, each term of the presented loss function has great impact on the BLEU performance of Ti-GSC, where in Rician fading syntactic learning This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. Jin Mao, Ke Xiong and, Ming Liu are with the Engineering Research Center of Network Management Technology for High Speed Railway of Ministry of Education, School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China, also with the Collaborative Innovation Center of Railway Traffic Safety, Beijing Jiaotong University, Beijing 100044, China, and also with the National Engineering Research Center of Advanced Network Technologies, Beijing Jiaotong University, Beijing 100044, China. E-mail: (jinmao@bjtu.edu.cn; kxiong@bjtu.edu.cn; mingliu@bjtu.edu.cn). Zhijin Qin is with the Department of Electronic Engineering, Tsinghua University, Beijing 100084, China (e-mail: qinzhijin@tsinghua.edu.cn). Wei Chen and Pingyi Fan are with the Beijing National Research Center for Information Science and Technology, and the Department of Elec- tronic Engineering, Tsinghua University, Beijing 100084, China (e-mail: wchen@tsinghua.edu.cn; fpy@tsinghua.edu.cn). Khaled Ben Letaief is with the Department of Electrical and Computer En- gineering, The Hong Kong University of Science and Technology (HKUST), Hong Kong, China (e-mail: eekhaled@ece.ust.hk). has the greatest impact, and in Rayleigh fading, the adversarial learning becomes important.

• To achieve better training results of Ti-GSC , two training schemes, i.e., the joint optimization based training (JOT) and the alternating optimization based training (AOT) are designed for the proposed Ti-GSC.
• Experimental results show that JOT is more efficient for Ti-GSC, and Ti-GSC outperforms conventional communication frameworks in terms of bilingual evaluation understudy (BLEU) score in both Rician and Rayleigh fading channels.
• Moreover, without CSI, BLEU score achieved by Ti-GSC is about 40% and 62% higher than that achieved by existing SC frameworks in Rician and Rayleigh fading, respectively.

Semantic communications have gained significant attention as a promising approach to address the transmission bottleneck, especially with the continuous development of 6G techniques. Distinct from the well investigated physical channel impairments, this paper focuses on semantic impairments in image, particularly those arising from adversarial perturba- tions. Specifically, we propose a novel metric for quantifying the intensity of semantic impairment and develop a semantic impairment dataset. Furthermore, we introduce a deep learning enabled semantic communication system, termed as DeepSC- RI, to enhance the robustness of image transmission, which incorporates a multi-scale semantic extractor with a dual-branch architecture for extracting semantics with varying granularity, thereby improving the robustness of the system. The fine-grained branch incorporates a semantic importance evaluation module to identify and prioritize crucial semantics, while the coarse- grained branch adopts a hierarchical approach for capturing the robust semantics. These two streams of semantics are seam- lessly integrated via an advanced cross-attention-based semantic fusion module. Experimental results demonstrate the superior performance of DeepSC-RI under various levels of semantic impairment intensity.

• Specifically, we propose a novel metric for quantifying the intensity of semantic impairment and develop a semantic impairment dataset.
• Experimental results demonstrate the superior performance of DeepSC-RI under various levels of semantic impairment intensity.
• This achieve- ment is made possible by utilizing computational resources arXiv:2403.09222v1 [eess.SP] 14 Mar 2024 Physical Channel Adversarial Image “Dog” Purified Image “horse” … … Semantic Encoder … … … … Channel Encoder Channel Decoder … … Coarse Grained Semantic Extractor Fine Grained Semantic Extractor … … … … Sematic Fusion Semantic Decoder Transmitter Receiver Semantic Channel … … … … … … Fig. 1.

Semantic communication is increasingly viewed as a promising solution to improve the transmission efficiency. However, semantic communications are susceptible not only to physical channel impairments, but also to semantic impairments, which degrade semantic understanding at the receiver and disrupt the associated downstream tasks. Hence, we focus our attention on the robustness of semantic communications against semantic impairments. Specifically, we first categorize textual semantic impairments into three categories based on their sources. Then, we propose a robust deep learning enabled semantic communication system (R-DeepSC) by introducing a semantic corrector for robust semantic encoding so as to facilitate semantic transmission. Moreover, we develop a non- autoregressive version of R-DeepSC, namely NA-RDeepSC, which offers improved inference speed by relying on a non- autoregressive architecture and an adaptive generator embedded into the semantic decoder. NA-RDeepSC performs semantic decoding in parallel, hence reducing the decoding complexity from O(n) to O(1) with a comparable performance to that of R-DeepSC. Our experimental results demonstrate the superior robustness of the proposed R-DeepSC and NA- RDeepSC architectures in eliminating semantic impairments, hence highlighting the significance of this work in advancing the development of robust semantic communications. Index Terms— Semantic communication, semantic impairments, calibrated self-attention, non-autoregressive, text transmission.

• Then, we propose a robust deep learning enabled semantic communication system (R-DeepSC) by introducing a semantic corrector for robust semantic encoding so as to facilitate semantic transmission.
• Moreover, we develop a non- autoregressive version of R-DeepSC, namely NA-RDeepSC, which offers improved inference speed by relying on a non- autoregressive architecture and an adaptive generator embedded into the semantic decoder.
• Our experimental results demonstrate the superior robustness of the proposed R-DeepSC and NA- RDeepSC architectures in eliminating semantic impairments, hence highlighting the significance of this work in advancing the development of robust semantic communications.

Semantic communications are expected to improve the transmission efficiency in Internet of Things (IoT) networks. However, the distributed nature of networks and heterogeneity of devices challenge the secure utilization of semantic com- munication systems. In this paper, we develop a distributed semantic communication system that achieves the security and efficiency during update and usage phases. A blockchain-based trust scheme for update is designed to continuously train and syn- chronize the system in dynamic IoT environments. To improve the updating efficiency, we propose a flexible semantic coding method base on compressive semantic knowledge bases. It greatly reduces the amount of data shared among devices for system update, and realizes the flexible adjustment of the size of knowledge bases and the number of transmitted signal symbols in model training and inference stages. In the usage phase, a signature mechanism for lossy semantics is introduced to guarantee the integrity and authenticity of the transmitted semantics in lossy semantic communications. We further design a noise-aware differential privacy mechanism, which introduces optimized noise based on the different channel information available to heterogeneous devices. Experiments on text transmission tasks show that the proposed system achieves the protection of the integrity and privacy for exchanged semantics, and reduces the data to be transmitted in the update phase by about 35% to 88%, and in the usage phase by 60% compared with related works.

• In this paper, we develop a distributed semantic communication system that achieves the security and efficiency during update and usage phases.
• To improve the updating efficiency, we propose a flexible semantic coding method base on compressive semantic knowledge bases.
• Experiments on text transmission tasks show that the proposed system achieves the protection of the integrity and privacy for exchanged semantics, and reduces the data to be transmitted in the update phase by about 35% to 88%, and in the usage phase by 60% compared with related works.

Task-oriented semantic communications have achieved significant performance gains. However, the employed deep neural networks in semantic communications have to be updated when the task is changed or multiple models need to be stored for performing different tasks. To address this issue, we develop a unified deep learning-enabled semantic communication system (U-DeepSC), where a unified end-to-end framework can serve many different tasks with multiple modalities of data. As the number of required features varies from task to task, we propose a vector-wise dynamic scheme that can adjust the number of transmitted symbols for different tasks. Moreover, our dynamic scheme can also adaptively adjust the number of transmitted features under different channel conditions to optimize the transmission efficiency. Particularly, we devise a lightweight feature selection module (FSM) to evaluate the importance of feature vectors, which can hierarchically drop redundant feature vectors and significantly accelerate the inference. To reduce the transmission overhead, we then design a unified codebook for feature representation to serve multiple tasks, where only the indices of these task-specific features in the codebook are transmitted. According to the simulation results, the proposed U-DeepSC achieves comparable performance to the task-oriented semantic communication system designed for a specific task but with significant reduction in both transmission overhead and model size.

• IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 72, NO. 7, JULY 2024 4101 A Unified Multi-Task Semantic Communication System for Multimodal Data Guangyi Zhang , Graduate Student Member, IEEE, Qiyu Hu , Student Member, IEEE, Zhijin Qin , Senior Member, IEEE, Yunlong Cai , Senior Member, IEEE, Guanding Yu , Senior Member, IEEE, and Xiaoming Tao , Senior Member, IEEE Abstract— Task-oriented semantic communications have achieved significant performance gains.
• To address this issue, we develop a unified deep learning-enabled semantic communication system (U-DeepSC), where a unified end-to-end framework can serve many different tasks with multiple modalities of data.
• As the number of required features varies from task to task, we propose a vector-wise dynamic scheme that can adjust the number of transmitted symbols for different tasks.

Semantic communication, recognized as a promis- ing technology for future intelligent applications, has received widespread research attention. Despite the potential of semantic communication to enhance transmission reliability, especially in low signal-to-noise (SNR) environments, the critical issue of resource allocation and compatibility in the dynamic wireless environment remains largely unexplored. In this paper, we propose an adaptive semantic resource allocation paradigm with semantic-bit quantization (SBQ) compatibly for existing wireless communications, where the inaccurate environment perception introduced by the additional mapping relationship between semantic metrics and transmission metrics is solved. In order to investigate the performance of semantic communication networks, the quality of service for semantic communication (SC- QoS), including the semantic quantization efficiency (SQE) and transmission latency, is proposed for the first time. A problem of maximizing the overall effective SC-QoS is formulated by jointly optimizing the transmit beamforming of the base station, the bits for semantic representation, the subchannel assignment, and the bandwidth resource allocation. To address the non-convex formulated problem, an intelligent resource allocation scheme is proposed based on a hybrid deep reinforcement learning (DRL) algorithm, where the intelligent agent can perceive both semantic tasks and dynamic wireless environments. Simulation results demonstrate that our design can effectively combat semantic noise and achieve superior performance in wireless communi- cations compared to several benchmark schemes. Furthermore, compared to mapping-guided paradigm based resource allocation schemes, our proposed adaptive scheme can achieve up to 13% performance improvement in terms of SC-QoS.

• In this paper, we propose an adaptive semantic resource allocation paradigm with semantic-bit quantization (SBQ) compatibly for existing wireless communications, where the inaccurate environment perception introduced by the additional mapping relationship between semantic metrics and transmission metrics is solved.
• Simulation results demonstrate that our design can effectively combat semantic noise and achieve superior performance in wireless communi- cations compared to several benchmark schemes.
• Furthermore, compared to mapping-guided paradigm based resource allocation schemes, our proposed adaptive scheme can achieve up to 13% performance improvement in terms of SC-QoS.

| Artificial intelligence (AI) and machine learning (ML) have shown tremendous potential in reshaping the landscape of wireless communications and are, therefore, widely expected to be an indispensable part of the next- generation wireless network. This article presents an overview of how AI/ML and wireless communications interact synergistically to improve system performance and provides useful tips and tricks on realizing such performance gains when training AI/ML models. In particular, we discuss in detail the use of AI/ML to revolutionize key physical layer Manuscript received 2 January 2024; revised 8 April 2024; accepted 22 July 2024. Date of publication 20 August 2024; date of current version 30 October 2024. The work of Zhijin Qin was supported in part by the National Key Research and Development Program of China under Grant 2023YFB2904300 and in part by the National Natural Science Foundation of China (NSFC) under Grant 62293484. The work of Le Liang was supported in part by the Natural Science Foundation of Jiangsu Province under Grant BK20220810 and in part by NSFC under Grant 62201145. The work of Shi Jin was supported in part by NSFC under Grant 62261160576, in part by the Key Technologies Research and Development Program of Jiangsu (Prospective and Key Technologies for Industry) under Grant BE2023022 and Grant BE2023022-1, and in part by the Fundamental Research Funds for the Central Universities under Grant 2242023K5003. The work of Xiaoming Tao was supported by NSFC under Grant 61925105. (Corresponding author: Le Liang.) Zhijin Qin, Zijing Wang, and Xiaoming Tao are with the Department of Electronic Engineering, Tsinghua University, Beijing 100084, China, with the State Key Laboratory of Space Network and Communications, Beijing 100084, China, and also with Beijing National Research Center for Information Science and Technology, Beijing 100084, China (e-mail: qinzhijin@tsinghua.edu.cn; wangzijing@tsinghua.edu.cn; taoxm@tsinghua.edu.cn). Le Liang is with the National Mobile Communications Research Laboratory, Southeast University, Nanjing 210096, China, and also with the Purple Mountain Laboratories, Nanjing 211111, China (e-mail: lliang@seu.edu.cn). Shi Jin is with the National Mobile Communications Research Laboratory, Southeast University, Nanjing 210096, China (e-mail: jinshi@seu.edu.cn). Wen Tong is with the Wireless Technology Labs, Huawei Technologies, Ottawa, ON K2K 3J1, Canada (e-mail: tongwen@huawei.com). Geoffrey Ye Li is with the School of Electrical and Electronic Engineering, Imperial College London, SW7 2AZ London, U.K. (e-mail: geoffrey.li@imperial.ac.uk). Digital Object Identifier 10.1109/JPROC.2024.3437730 and lower medium access control (MAC) layer functionalities in traditional wireless communication systems. In addition, we provide a comprehensive overview of the AI/ML-enabled semantic communication systems, including key techniques from data generation to transmission. We also investigate the role of AI/ML as an optimization tool to facilitate the design of efficient resource allocation algorithms in wireless communication networks at both bit and semantic levels. Finally, we analyze major challenges and roadblocks in applying AI/ML in practical wireless system design and share our thoughts and insights on potential solutions.

• Developments of Wireless Communications The success of wireless communications is one of the biggest achievements in the history of science and technol- ogy, changing the way people and machines interact with each other.
• For instance, AI models need substantial data for effective training to achieve near-optimal performance; thus, the scarcity of computing resources emerges as a prominent hurdle and hinders the network robustness.
• Therefore, the lack of explicit treatment for algorithmic selection and misuse of ML algorithms pose a challenge to achieving optimal performance tailored to the underlying transmission task.

Camera sensors have been widely used in intelligent robotic systems. Developing camera sensors with high sensing efficiency has always been important to reduce the power, memory, and other related resources. Inspired by recent success on programmable sensors and deep optic methods, we design a novel video compressed sensing system with spatially-variant compression ratios, which achieves higher imaging quality than the existing snapshot compressed imaging methods with the same sensing costs. In this article, we also investigate the data transmission methods for programmable sensors, where the performance of communication systems is evaluated by the reconstructed images or videos rather than the transmission of sensor data itself. Usually, different reconstruction algorithms are designed for applications in high dynamic range imaging, video compressive sensing, or motion debluring. This task-aware property inspires a semantic communication framework for programmable sensors. In this work, a policy-gradient based reinforcement learning method is introduced to achieve the explicit trade-off between the compression (or transmission) rate and the image distortion. Numerical results show the superiority of the proposed methods over existing baselines.

• Inspired by recent success on programmable sensors and deep optic methods, we design a novel video compressed sensing system with spatially-variant compression ratios, which achieves higher imaging quality than the existing snapshot compressed imaging methods with the same sensing costs.
• In this work, a policy-gradient based reinforcement learning method is introduced to achieve the explicit trade-off between the compression (or transmission) rate and the image distortion.
• If pixels within SCI systems can be generated under different compres- sion ratios, the video compressive sensing system can maintain high quality reconstruction and achieve efficient measurement.

The trend of massive connectivity pushes forward the explosive growth of end devices. The emergence of various applications has prompted a demand for pervasive connectivity and more efficient computing paradigms. On the other hand, the lack of computational capacity of the end devices restricts the implementation of the intelligent applications, and becomes a bottleneck of the multiple access for supporting massive connectivity. Mobile cloud computing (MCC) and mobile edge computing (MEC) techniques enable end devices to offload local computation-intensive tasks to servers by networks. In this paper, we consider the cloud-edge-end collaborative networks to utilize distributed computing resources. Furthermore, we apply task-oriented semantic communications to tackle the fast- varying channel between the end devices and MEC servers and reduce the communication cost. To minimize long-term energy consumption on constraints queue stability and computational delay, a Lyapunov-guided deep reinforcement learning hybrid (DRLH) framework is proposed to solve the mixed integer non- linear programming (MINLP) problem. The long-term energy consumption minimization problem is transformed into the deterministic problem in each time frame. The DRLH framework integrates a model-free deep reinforcement learning algorithm with a model-based mathematical optimization algorithm to mitigate computational complexity and leverage the scenario information, so that improving the convergence performance. Numerical results demonstrate that the proposed DRLH frame- work achieves near-optimal performance on energy consumption while stabilizing all queues.

• Numerical results demonstrate that the proposed DRLH frame- work achieves near-optimal performance on energy consumption while stabilizing all queues.
• In [13], Zhou et al. consider both computation and communication resources for ESs and local devices, jointly optimizing the transmit power and computational capability for local devices, and achieving the energy-efficient resource allocation.
• Furthermore, these conventional model-based methods usually require a large number of long-term iterations until achieve a good performance, which cannot satisfy the task execution latency requirement and is impractical to the fast- varying wireless channels. 3 On the other hand, data-driven solutions, e.g., machine learning approaches provide a promising solution to tackle the complexity issue.

Semantic communication has shown great potential in boosting the effectiveness and reliability of communications. However, its systems to date are mostly enabled by deep learning, which requires demanding computing resources. This article proposes a framework for the computing networks en- abled semantic communication system, aiming to offer sufficient computing resources for semantic processing and transmission. Key techniques including semantic sampling and reconstruction, semantic-channel coding, semantic-aware resource allocation and optimization are introduced based on the cloud-edge-end com- puting coordination. Two use cases are demonstrated to show advantages of the proposed framework. The article concludes with several future research directions.

• In this article, we propose a framework of computing networks enabled semantic communication, which differs from existing ones by introducing the connected computing.
• In [5], the edges actively monitor the available communication and computation resources, enabling them to estimate the end-to- end latency and achievable data rate accordingly.
• Some recent results implied that joint source-channel coding can outperform separate schemes, especially in the short block- length regime [8].

Semantic communication technology is regarded as a method surpassing the Shannon limit of bit transmission, capable of effectively enhancing transmission efficiency. However, current approaches that directly map content to transmission symbols are challenging to deploy in practice, imposing significant limitations on the development of semantic communication. To address this challenge, we propose a hybrid bit and semantic communication system, named HybridBSC, in which encoded semantic information is inserted into bit information for transmission via conventional digital communication systems utilizing same spectrum resources. The system can be easily deployed using existing communication architecture to achieve bit and semantic information transmission. Particularly, we design a semantic insertion and extraction scheme to implement this strategy. Furthermore, we conduct experimental validation based on the pluto-based software defined radio (SDR) platform in a real wireless channel, demonstrating that the proposed strategy can simultaneously transmit semantic and bit information.

• To address this challenge, we propose a hybrid bit and semantic communication system, named HybridBSC, in which encoded semantic information is inserted into bit information for transmission via conventional digital communication systems utilizing same spectrum resources.
• The system can be easily deployed using existing communication architecture to achieve bit and semantic information transmission.
• To achieve this goal, we insert semantic information into the traditional bit- 5 encoded information to generate hybrid data to be transmitted.

A key issue in Semantic Communications (SemCom) is how to transmit semantic information over the air, where digital and analog transmission schemes are the typical solutions. However, digital SemCom suffers from leveling-off and cliff- edge effects and analog SemCom faces data security issues. To address these challenges, we propose a novel hybrid digital- analog transmission for SemCom, called HDA-DeepSC. The proposed method integrates the advantages of both digital and analog transmission with digital-analog allocation and fusion modules. We have also developed new loss functions that take into account long-distance dependencies between pixels, essential information recovery, and optimal bitstream generation. The numerical experiments show that HDA-DeepSC achieves better image quality compared to digital and analog SemCom, high- lighting its superior performance in terms of data security.

• To address these challenges, we propose a novel hybrid digital- analog transmission for SemCom, called HDA-DeepSC.
• The numerical experiments show that HDA-DeepSC achieves better image quality compared to digital and analog SemCom, high- lighting its superior performance in terms of data security.
• Besides, analog semantic communications cannot achieve ac- curate transmission due to the approximately infinite candidate sets in continuous signals.

Information and communication technology (ICT) has been an essential part of modern society. However, the current communication systems are not sufficient to meet the demands of emerging applications. Intellicise (intelligent and concise) wireless networks, with their inherent characteristics of intelligence- endogenous and primitive-concise, have been proposed as a promising research direction. In this paper, we focus on intellicise wireless networks from semantic communication (SemCom). We present a comprehensive framework of intellicise wireless networks, including components such as brain for intellicise wireless networks (BIWN), intellicise signal processing, intellicise information transmission, intellicise network organization, and intellicise service bearing. We also investigate the enabling technologies and driving factors of intellicise wireless networks. Subsequently, we introduce the applications of intellicise wireless networks and envision new services. Finally, we outline the challenges of implementing intellicise wireless networks from a broad perspective and discuss possible solutions.

• We present a comprehensive framework of intellicise wireless networks, including components such as brain for intellicise wireless networks (BIWN), intellicise signal processing, intellicise information transmission, intellicise network organization, and intellicise service bearing.
• The ultimate goal of SemCom is to accurately deliver the semantic information to achieve various downstream tasks.
• To achieve this goal, the transmitter and receiver should be redesigned to guarantee high semantic fidelity under unpredictable communication environments. 1553-877X c⃝2024 IEEE.

Learning-task oriented semantic communication is pivotal in optimizing transmission efficiency by extracting and conveying essential semantics tailored to the specific tasks, such as image reconstruction and classification. Nevertheless, the challenge of eavesdropping poses a formidable threat to semantic privacy due to open nature of wireless communications. In this paper, intelligent reflective surface (IRS)-enhanced secure semantic communication (IRS-SSC) is proposed to guarantee the physical layer security from a task-oriented semantic perspective. Specifically, a multi-layer codebook is exploited to discretize continuous semantic features and describe semantics with different numbers of bits, thereby meeting the need for hierarchical semantic representation and further enhancing the transmission effi- ciency. Novel semantic security metrics, i.e., secure semantic rate (S-SR) and secure semantic spectrum efficiency (S-SSE), are defined to map the task-oriented security requirements at the application layer This work was supported by the National Key Research and Development Program of China under Grant 2020YFB1807602, the National Natural Science Foundation of China under Grant 62271267 and the open research fund of National Mobile Communications Research Laboratory, Southeast University under Grant 2024D16. Lingyi Wang is with the College of Science, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China (e-mail: lingyiwang@njupt.edu.cn). Wei Wu is with the College of Communication and Information Engineering, Nanjing University of Posts and Telecommuni- cations, Nanjing, 210003, China, and also with the National Mobile Communications Research Laboratory, Southeast University, Nanjing, 210096, China (e-mail: weiwu@njupt.edu.cn). Fuhui Zhou and Qihui Wu are with the College of Electronic and Information Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, 210000, China (e-mail: zhoufuhui@ieee.org, wuqihui2014@sina.com). Zhijin Qin is with Department of Electronic Engineering, Tsinghua University, Beijing, China. She is also with the Beijing National Research Center for Information Science and Technology, Beijing, China, and the State Key Laboratory of Space Network and Communications, Beijing, China. (email: qinzhijin@tsinghua.edu.cn). 2 into the physical layer. To achieve artificial intelligence (AI)-native secure communication, we propose a noise disturbance enhanced hybrid deep reinforcement learning (NdeHDRL)-based resource allocation scheme. This scheme dynamically maximizes the S-SSE by jointly optimizing the bits for semantic representations, reflective coefficients of the IRS, and the subchannel assignment. Moreover, we propose a novel semantic context awared state space (SCA-SS) to fusion the high-dimensional semantic space and the observable system state space, which enables the agent to perceive semantic context and solves the dimensional catastrophe problem. Simulation results demonstrate the efficiency of our proposed schemes in both enhancing the security performance and the S-SSE compared to several benchmark schemes.

• To achieve artificial intelligence (AI)-native secure communication, we propose a noise disturbance enhanced hybrid deep reinforcement learning (NdeHDRL)-based resource allocation scheme.
• Moreover, we propose a novel semantic context awared state space (SCA-SS) to fusion the high-dimensional semantic space and the observable system state space, which enables the agent to perceive semantic context and solves the dimensional catastrophe problem.
• Simulation results demonstrate the efficiency of our proposed schemes in both enhancing the security performance and the S-SSE compared to several benchmark schemes.

The limited capabilities of user equipment restrict the local implementation of computation-intensive applications. Edge computing, especially the edge intelligence system, en- ables local users to offload the computation tasks to the edge servers to reduce the computational energy consumption of user equipment and accelerate fast task execution. However, the limited bandwidth of upstream channels may increase the task transmission latency and affect the computation offloading performance. To overcome the challenge arising from scarce wireless communication resources, we propose a semantic-aware multi-modal task offloading system that facilitates the extraction and offloading of semantic task information to edge servers. To cope with the different tasks with multi-modal data, a unified quality of experience (QoE) criterion is designed. Furthermore, a proximal policy optimization-based multi-agent reinforcement learning algorithm (MAPPO) is proposed to coordinate the resource management for wireless communications and computa- tion in a distributed and low computational complexity manner. Simulation results verify that the proposed MAPPO algorithm outperforms other reinforcement learning algorithms and fixed schemes in terms of task execution speed and the overall system QoE.

• To overcome the challenge arising from scarce wireless communication resources, we propose a semantic-aware multi-modal task offloading system that facilitates the extraction and offloading of semantic task information to edge servers.
• Simulation results verify that the proposed MAPPO algorithm outperforms other reinforcement learning algorithms and fixed schemes in terms of task execution speed and the overall system QoE.
• Zhou et al. [7] consider both communication and computation resources for local devices and ESs, jointly optimizing computational ca- pability and transmit power of local users to achieve energy- efficient resource allocation.

The last seventy years have witnessed the transition of communication from Shannon's theoretical concept to current high-efficiency practical systems. With respect to Shannon information theory, to fulfill the high-bandwidth utilization, any further increase in the data rate requires a significant augmentation in the received signal power unless the bandwidth is extended in proportion to the incremental data rate. Although many such technologies have achieved tremendous success in today's communication systems, they also lead to severe high-frequency coverage costs, complicated signal processing, high energy consumption, etc. Together with extraordinary promises, naively increasing channel capacity cannot address all communication problems, especially in future intelligent eras. It is the very time for radical rethinking of classical communication mechanisms.

• Although many such technologies have achieved tremendous success in today’s com- munication systems, they also lead to severe high-frequency coverage costs, complicated signal processing, high energy consumption, etc.
• The call for paper achieved a meaningful success, having stimulated a large number of contributions for semantic communications focused on novel perspec- tives of mathematical theory analysis, novel technol- ogies for typical application and communication scenarios, and novel paradigms facing new commu- nication scenarios.
• This special issue focuses on exploring new direc- tions and achievements for the theory, technology, GUEST EDITORIAL GUEST EDITORIAL iv © China Communications Magazine Co., Ltd. · July 2024 and application of semantic communication.

Semantic communications have been utilized to execute numerous intelligent tasks by transmitting task-related semantic information instead of bits. In this article, we propose a semantic-aware speech-to-text transmission system for the single-user multiple-input multiple-output (MIMO) and multi- user MIMO communication scenarios, named SAC-ST. Partic- ularly, a semantic communication system to serve the speech- to-text task at the receiver is first designed, which compresses the semantic information and generates the low-dimensional semantic features by leveraging the transformer module. In addition, a novel semantic-aware network is proposed to facilitate transmission with high semantic fidelity by identifying the critical semantic information and guaranteeing its accurate recovery. Furthermore, we extend the SAC-ST with a neural network- enabled channel estimation network to mitigate the dependence on accurate channel state information and validate the feasibility of SAC-ST in practical communication environments. Simulation results will show that the proposed SAC-ST outperforms the communication framework without the semantic-aware network for speech-to-text transmission over the MIMO channels in terms of the speech-to-text metrics, especially in the low signal-to- noise regime. Moreover, the SAC-ST with the developed channel estimation network is comparable to the SAC-ST with perfect channel state information.

• In this article, we propose a semantic-aware speech-to-text transmission system for the single-user multiple-input multiple-output (MIMO) and multi- user MIMO communication scenarios, named SAC-ST.
• Simulation results will show that the proposed SAC-ST outperforms the communication framework without the semantic-aware network for speech-to-text transmission over the MIMO channels in terms of the speech-to-text metrics, especially in the low signal-to- noise regime.
• The conventional communications, i.e., the first level of communications, convert the input message into a bit se- quence and aim to achieve a low bit-error rate (BER) by leveraging proper coding and decoding algorithms.

Although semantic communication has shown great performance in various types of data transmission, the problem of semantic synchronization between multimodal data has not been well studied. Semantic synchronization is a challenging issue that requires the transmitted information to be synchronized in both semantic and time domains. In this article, we propose a synchronous semantic communication system for video and speech transmission, which the real-time facial transmission is adopted as the use case. Particularly, to achieve time domain synchronization, we design an efficient semantic transmitter to send multimodal data packets. 3D Morphable Mode (3DMM) coefficients and text are employed as semantic information, achieving semantic interactivity and lower bandwidth. To address synchronization in semantic domain, we firstly employ the visual voice clone at the receiver. Visual-guided speech synthesis module is designed to align text and facial semantics. Thus, the generated speech is synchronized with video frames in both semantic and time domains. The simulation results show that our proposed system achieves high- quality synchronous transmission of video and speech with reducing transmission overhead.

• In this article, we propose a synchronous semantic communication system for video and speech transmission, which the real-time facial transmission is adopted as the use case.
• Particularly, to achieve time domain synchronization, we design an efficient semantic transmitter to send multimodal data packets. 3D Morphable Mode (3DMM) coefficients and text are employed as semantic information, achieving semantic interactivity and lower bandwidth.
• The simulation results show that our proposed system achieves high- quality synchronous transmission of video and speech with reducing transmission overhead.

Semantic communications have shown great poten- tial in reducing the transmitted data amount through the powerful capability to extract and transmit essential semantic information. Although existing works have achieved certain transmission efficiency, challenges such as efficiently interpretable semantic extraction, and dynamically adaptive channel cod- ing have not been fully explored. In this paper, we tackle these challenges by proposing a task-oriented scene graph-based semantic communication system with adaptive channel coding, named GRACE, to perform image retrieval task. To enhance the interpretability of semantic communications and reduce semantic redundancy, we introduce a scene graph semantic encoder. This encoder fully exploits informative scene graph semantics, effec- tively extracting scene graphs and performing further semantic coding. Additionally, to handle variable channel conditions in real-world scenarios, we develop a semantic-aware adaptive channel coding to adapt to channel conditions and reduce the communication resources. At the receiver, the image retrieval task is accomplished based on the recovered scene graph semantics. The experimental results demonstrate the superiority of the proposed system compared to other communication systems in terms of task-execution performance, robustness against channel variations, transmission efficiency, and computational complexity.

• Although existing works have achieved certain transmission efficiency, challenges such as efficiently interpretable semantic extraction, and dynamically adaptive channel cod- ing have not been fully explored.
• Additionally, to handle variable channel conditions in real-world scenarios, we develop a semantic-aware adaptive channel coding to adapt to channel conditions and reduce the communication resources.
• The experimental results demonstrate the superiority of the proposed system compared to other communication systems in terms of task-execution performance, robustness against channel variations, transmission efficiency, and computational complexity.

Deep learning enabled semantic communications has shown great potential to significantly improve transmission efficiency and alleviate spectrum scarcity, by effectively exchang- ing the semantics behind the data. Recently, the emergence of large models, boasting billions of parameters, has unveiled remarkable human-like intelligence, offering a promising avenue for advancing semantic communication by enhancing seman- tic understanding and contextual understanding. This article systematically investigates the large model-empowered semantic communication systems from potential applications to system design. First, we propose a new semantic communication ar- chitecture that seamlessly integrates large models into semantic communication through the introduction of a memory module. Then, the typical applications are illustrated to show the benefits of the new architecture. Besides, we discuss the key designs in implementing the new semantic communication systems from module design to system training. Finally, the potential research directions are identified to boost the large model-empowered semantic communications.

• First, we propose a new semantic communication ar- chitecture that seamlessly integrates large models into semantic communication through the introduction of a memory module.
• Therefore, achieving powerful semantic representa- tions and semantic understanding is one of the key problems in improving the performance of semantic communication systems.
• Zhijin Qin and Xiaoming Tao are with the Department of Elec- tronic Engineering, Tsinghua University, Beijing, China (e-mail: qinzhi- jin@tsinghua.edu.cn, taoxm@tsinghua.edu.cn). (Zhijin Qin is the correspond- ing author.) Zhu Han is with the Department of Electrical and Computer Engineering, University of Houston, Houston, USA (e-mail: zhan2@uh.edu). vast data, large models appear to show a human-like ability in intelligent tasks, which has been proposed as a promising new technology for achieving general artificial intelligence.

With the advent of the 6G era, the efficient trans- mission of large-scale, high-resolution videos poses a signifi- cant challenge to existing communication systems. Compared to traditional video communication systems, which are becoming increasingly complex and difficult to optimize, semantic commu- nication offers new possibilities and flexibility for optimization, effectively resisting noise interference with low-design-complexity joint coding. In this paper, we propose an efficient semantic- channel joint coding model for wireless video transmission, which adjusts the rate output based on the contextual information of video sequences and channel conditions, ensuring the reliability of video transmission even under different channel conditions. Across standard video test sequences under different scenarios, experiments show that under reliable channel state conditions, our scheme can save 22% to 56% of bandwidth compared to traditional video wireless transmission systems composed of H.265, 5G LDPC, and digital modulation, while also overcoming the cliff effect and achieving efficient video transmission under harsh channel conditions. Our scheme provides strong technical support for the practical application of semantic communication.

• In this paper, we propose an efficient semantic- channel joint coding model for wireless video transmission, which adjusts the rate output based on the contextual information of video sequences and channel conditions, ensuring the reliability of video transmission even under different channel conditions.
• Across standard video test sequences under different scenarios, experiments show that under reliable channel state conditions, our scheme can save 22% to 56% of bandwidth compared to traditional video wireless transmission systems composed of H.265, 5G LDPC, and digital modulation, while also overcoming the cliff effect and achieving efficient video transmission under harsh channel conditions.
• Lossy video com- pression standards like H.264/Advanced Video Coding (AVC) [2], H.265/High Efficiency Video Coding (HEVC) [3], and H.266/Versatile Video Coding (VVC) [4] have achieved sub- stantial compression but at the cost of increased algorithmic complexity.

Semantic communications have demonstrated exceptional performance across various tasks, yet they are susceptible to semantic impairments due to the inherent vulnerability of deep neural networks. This paper focuses on semantic impairments in images, particularly those stemming from adversarial perturbations. We introduce a novel metric for quantifying the level of semantic impairment and create a semantic impairment dataset. Furthermore, we propose a deep learning enabled semantic communication system for robust image transmission, termed as DeepSC-RI. The proposed system harnesses a multi-scale semantic extractor with a dual-branch design tailored for extracting semantics with varying granularity, thereby boosting the robustness of the system. The fine-grained branch incorporates a semantic importance evaluation module to identify and prioritize crucial semantics through self-attention score manipulations, while the coarse-grained branch adopts a hierarchical approach for progressively capturing the robust semantics. These two streams of semantics are seamlessly integrated via an advanced cross-attention-based semantic fusion module. Experimental results highlight the superior performance of DeepSC-RI under diverse channel conditions, across various levels of semantic impairment intensity, and in multiple tasks.

• Furthermore, we propose a deep learning enabled semantic communication system for robust image transmission, termed as DeepSC-RI.
• Experimental results highlight the superior performance of DeepSC-RI under diverse channel conditions, across various levels of semantic impairment intensity, and in multiple tasks.
• Distinct from the methods previously discussed, our work eliminates the need to retrain downstream task models and avoids the redundant computations, which is achieved by utilizing computational resources allocated for semantic communications to effectively mitigate semantic impairments.

Deep learning enabled semantic communications are attracting extensive attention. However, most works normally ig- nore the data acquisition process and suffer from robustness issues under dynamic channel environment. In this paper, we propose an adaptive joint sampling-semantic-channel coding (Adaptive- JSSCC) framework. Specifically, we propose a semantic-aware sampling and reconstruction method to optimize the number of samples dynamically for each region of the images. According to semantic significance, we optimize sampling matrices for each region of the most individually and obtain a semantic sampling ratio distribution map shared with the receiver. Through the guidance of the map, high-quality reconstruction is achieved. Meanwhile, attention-based channel adaptive module (ACAM) is designed to overcome the neural network model mismatch between the training and testing channel environment during sampling- reconstruction and encoding-decoding. To this end, signal-to-noise ratio (SNR) is employed as an extra parameter input to integrate and reorganize intermediate characteristics. Simulation results show that the proposed Adaptive-JSSCC effectively reduces the amount of data acquisition without degrading the reconstruction performance in comparison to the state-of-the-art, and it is highly adaptable and adjustable to dynamic channel environments.

• In this paper, we propose an adaptive joint sampling-semantic-channel coding (Adaptive- JSSCC) framework.
• Specifically, we propose a semantic-aware sampling and reconstruction method to optimize the number of samples dynamically for each region of the images.
• Through the guidance of the map, high-quality reconstruction is achieved.

The convergence of artificial intelligence (AI) and wireless communications has driven the emergence of semantic communication (SC), a paradigm that prioritizes context-aware semantic exchange over traditional bit-level transmission. Although enhancing efficiency and task-specific reliability, this advancing capability is accompanied by significant security challenges that remain underexplored. In this paper, we provide an overview of security challenges in SC systems, with a particular focus on the confidentiality, integrity, and availability of the wireless transmission and generative AI (GAI) models. To defend against risks of model confidentiality compromise and semantic feature leakage, we propose a solution integrating trusted execution environments (TEEs) for secure model inference and adversarial cryptography for the protection of semantics over realistic wireless channels. Test results show it achieves close-to-black-box attack resistance in model stealing effectiveness, and the BLEU scores of eavesdropping attackers are effectively reduced to below 0.1 across various SNR levels. Finally, we discuss potential open issues and solutions for enhancing the SC security, paving the way for future research in this critical area. The proposed framework demonstrates promising results in enhancing both model and data confidentiality, contributing to the development of secure SC systems for 6G networks.

• To defend against risks of model confidentiality compromise and semantic feature leakage, we propose a solu- tion integrating trusted execution environments (TEEs) for secure model inference and adversar- ial cryptography for the protection of semantics over realistic wireless channels.
• Test results show it achieves close-to-black-box attack resistance in model stealing effectiveness, and the BLEU scores of eavesdropping attackers are effectively reduced to below 0.1 across various SNR levels.
• Different from traditional wireless systems focusing on bit-level accuracy, SC emphasizes semantic-level transmissions, aiming to achieve more intelligent and efficient tasks by extracting and conveying semantics.

Different from traditional secure communication that focuses on symbolic protection at the physical layer, semantic secure communication requires further attention to semantic- level task performance at the application layer. There is a research gap on how to comprehensively evaluate and optimize the security performance of semantic communication. In order to fill this gap, a unified semantic security metric, the cross- layer semantic secure rate (CLSSR), is defined to estimate cross- layer security requirements at both the physical layer and the application layer. Then, we formulate the maximization problem of the CLSSR with the mixed integer nonlinear programming (MINLP). We propose a hierarchical AI-native semantic secure communication network with a reinforcement learning (RL)- based semantic resource allocation scheme, aiming to ensure the cross-layer semantic security (CL-SS). Finally, we prove the convergence of our proposed intelligent resource allocation, and the simulation results demonstrate that our proposed CLSS method outperforms the traditional physical layer semantic security (PL-SS) method in terms of both task reliability and CLSSR.

• We propose a hierarchical AI-native semantic secure communication network with a reinforcement learning (RL)- based semantic resource allocation scheme, aiming to ensure the cross-layer semantic security (CL-SS).
• Finally, we prove the convergence of our proposed intelligent resource allocation, and the simulation results demonstrate that our proposed CLSS method outperforms the traditional physical layer semantic security (PL-SS) method in terms of both task reliability and CLSSR.
• Then, we propose the hierarchical AI-native semantic secure communication network with an RL-based semantic resource allocation scheme, which can overcome the formulated cross-layer semantic security rate (CLSSR) maximization problem with the mixed integer nonlinear pro- arXiv:2503.12818v1 [cs.IT] 17 Mar 2025 2 Fig. 1.

As semantic communication can efficiently extract semantic information and significantly reduce the amount of data transmission, an increasing number of studies are exploring deep learning-based semantic communication methods. This paper proposes a convolutional neural network (ConvNet)-based semantic communication system for wireless image transmission called ConvSC, which utilizes deep learning techniques for semantic information extraction and reconstruction. To address the adaptation of semantic communication systems to different channel states, a semantic adaptive module is proposed, which adjusts the weights of different parts of the semantic information based on channel state information. The model was evaluated under additive white Gaussian noise (AWGN) channel, Rayleigh fading channel, and underwater wireless optical communication (UWOC) channel. Simulation results demonstrate that, compared to the baseline model, our proposed ConvSC achieves superior performance across multiple metrics in various channel environ- ments.

• Simulation results demonstrate that, compared to the baseline model, our proposed ConvSC achieves superior performance across multiple metrics in various channel environ- ments.
• JSCC achieves comparable or even better end-to-end transmission performance than traditional separate source and channel coding schemes by jointly opti- mizing the source and channel coding processes.
• For wireless image transmission, Bourtsoulatze et al. [6] first proposed a deep learning-based joint source-channel coding (DeepJSCC) scheme, which outperforms separate transmission schemes at low signal-to-noise ratio (SNR).

This paper proposes a novel diffusion-enabled plug- gable encryption/decryption modules design against semantic eavesdropping, where the pluggable modules are optionally assembled into the semantic communication system for prevent- ing eavesdropping. Inspired by the artificial noise (AN)-based security schemes in traditional wireless communication systems, in this paper, AN is introduced into semantic communication systems for the first time to prevent semantic eavesdropping. However, the introduction of AN also poses challenges for the legitimate receiver in extracting semantic information. Re- cently, denoising diffusion probabilistic models (DDPM) have demonstrated their powerful capabilities in generating multime- dia content. Here, the paired pluggable modules are carefully designed using DDPM. Specifically, the pluggable encryption module generates AN and adds it to the output of the semantic transmitter, while the pluggable decryption module before se- mantic receiver uses DDPM to generate the detailed semantic information by removing both AN and the channel noise. In the scenario where the transmitter lacks eavesdropper’s knowledge, the artificial Gaussian noise (AGN) is used as AN. We first model a power allocation optimization problem to determine the power of AGN, in which the objective is to minimize the weighted sum of data reconstruction error of legal link, the mutual information of illegal link, and the channel input distortion. Then, a deep reinforcement learning framework using deep deterministic policy gradient is proposed to solve the optimization problem. In the scenario where the transmitter is aware of the eavesdropper’s knowledge, we propose an AN generation method based on adversarial residual networks (ARN). Unlike the previous scenario, the mutual information term in the objective function is replaced by the confidence of eavesdropper correctly retrieving private information. The adversarial residual network is then trained to minimize the modified objective function. The simulation results show that the diffusion-enabled pluggable encryption module prevents semantic eavesdropping while the pluggable decryption module achieves the high-quality semantic communication.

• In the scenario where the transmitter is aware of the eavesdropper’s knowledge, we propose an AN generation method based on adversarial residual networks (ARN).
• The simulation results show that the diffusion-enabled pluggable encryption module prevents semantic eavesdropping while the pluggable decryption module achieves the high-quality semantic communication.
• As a post-Shannon communication paradigm, semantic communication has a great potential to achieve the classic Shannon’s limit.

Effective resource allocation is critical to the effi- ciency of semantic communications, especially when faced with limited energy supplies and constrained communication resources. The unique requirements of semantic communications have brought new challenges to the joint optimization of energy and communication resources. This letter presents a novel deep reinforcement learning (DRL)-based algorithm to maximize the long-term average semantic energy efficiency (S-EE) for energy harvesting-powered semantic communication systems. Particularly, the Multi-discrete Proximal Policy Optimization (MPPO) is applied to determine discrete actions, including channel allocation and the number of semantic symbols, while the Deep Deterministic Policy Gradient (DDPG) is employed to learn continuous transmit powers. MPPO uses multiple heads to cope with multi-discrete actions, thereby reducing the output dimension of actions. Simulation results indicate that the proposed algorithm achieves faster convergence and enhances S-EE by 31.6%, compared to its benchmarks.

• Simulation results indicate that the proposed algorithm achieves faster convergence and enhances S-EE by 31.6%, compared to its benchmarks.
• Simulation results show that the proposed MPPO-DDPG algorithm can improve the S-EE of the studied semantic-aware systems by 31.6%, compared to the benchmarks. 2162-2345 c⃝2025 IEEE.
• This is achieved by optimizing the channel allocation x tn,m, the transmit power Ptn,m,s and transmitted semantic symbols per word ktn.

In this article, we propose a robust semantic com- munication system for speech transmission, named Ross-S2T, to execute the speech-to-text translation (S2TT) transmission efficiently. First, a deep semantic encoder is developed to directly convert speech in the source language to textual features associ- ated with the target language, facilitating the end-to-end (E2E) semantic exchange to perform the S2TT task and reducing the amount of transmission data without performance degradation. To mitigate semantic impairments inherent in the corrupted speech, a novel generative adversarial network (GAN)-enabled deep semantic compensator is established to estimate the hidden semantic information within the speech and extract deep semantic features simultaneously, which enables robust semantic transmis- sion for corrupted speech. Furthermore, a semantic probe-aided compensator is devised to enhance the semantic fidelity of recovered semantic features and improve the understandability of the target text. According to simulation results, the proposed Ross-S2T exhibits superior S2TT performance compared to conventional approaches and high robustness against semantic impairments.

• 1380 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 25, 2026 Generative Semantic Communications for Robust Speech-to-Text Translation Zhenzi Weng , Member, IEEE, Zijing Wang , Member, IEEE, Zhijin Qin , Senior Member, IEEE, and Xiaoming Tao , Senior Member, IEEE Abstract—In this article, we propose a robust semantic com- munication system for speech transmission, named Ross-S2T, to execute the speech-to-text translation (S2TT) transmission efficiently.
• According to simulation results, the proposed Ross-S2T exhibits superior S2TT performance compared to conventional approaches and high robustness against semantic impairments.
• The advance- ment of semantic communications derives from the ability to explore semantic information and achieve semantic exchange, which revolutionizes many aspects of wireless communica- tions [3].

Digital and analog semantic communications (Sem- Com) face inherent limitations such as data security concerns in analog SemCom, as well as leveling-off and cliff-edge effects in digital SemCom. In order to overcome these challenges, we propose a novel SemCom framework and a corresponding system called HDA-DeepSC, which leverages a hybrid digital- analog approach for multimedia transmission. This is achieved through the introduction of analog-digital allocation and fusion modules. To strike a balance between data rate and distortion, we design new loss functions that take into account long-distance dependencies in the semantic distortion constraint, essential information recovery in the channel distortion constraint, and optimal bit stream generation in the rate constraint. Addi- tionally, we propose denoising diffusion-based signal detection techniques, which involve carefully designed variance schedules and sampling algorithms to refine transmitted signals. Through extensive numerical experiments, we will demonstrate that HDA- DeepSC exhibits robustness to channel variations and is capable of supporting various communication scenarios. Our proposed framework outperforms existing benchmarks in terms of peak signal-to-noise ratio and multi-scale structural similarity, show- casing its superiority in semantic communication quality.

• In order to overcome these challenges, we propose a novel SemCom framework and a corresponding system called HDA-DeepSC, which leverages a hybrid digital- analog approach for multimedia transmission.
• This is achieved through the introduction of analog-digital allocation and fusion modules.
• Addi- tionally, we propose denoising diffusion-based signal detection techniques, which involve carefully designed variance schedules and sampling algorithms to refine transmitted signals.

Deep learning based semantic communication (DeepSC) system has emerged as a promising paradigm for efficient wireless transmission. However, existing image DeepSC methods, frequently encounter challenges in balancing rate- distortion performance and computational complexity, and often exhibit inferior performance compared to traditional schemes, especially on high-resolution datasets. To address these limita- tions, we propose a novel image DeepSC system, using quadtree partition-based joint semantic-channel coding, named Quad- DeepSC, which maintains low complexity while achieving state- of-the-art transmission performance. Based on maturing learned image compression technologies, we establish a unified DeepSC system design and training pipeline. The proposed Quad-DeepSC integrates quadtree partition-based entropy estimation and fea- ture coding modules with lightweight feature extraction and re- construction networks to form an end-to-end architecture. During training, all components except the feature coding modules are jointly optimized as a compact learned image codec, Quad-LIC, for source compression tasks. The pretrained Quad-LIC is then embedded into Quad-DeepSC and fine-tuned end-to-end over wireless channels. Extensive experimental results demonstrate that Quad-DeepSC is the first DeepSC system to surpass con- ventional communication systems, which employ VTM for source coding and adopt the optimal MCS index under 3GPP standards for channel coding and digital modulation, in performance across datasets of varying resolutions. Notably, both Quad-DeepSC and Quad-LIC exhibit minimal latency, rendering them well-suited for deployment in real-time wireless communication systems.

• To address these limita- tions, we propose a novel image DeepSC system, using quadtree partition-based joint semantic-channel coding, named Quad- DeepSC, which maintains low complexity while achieving state- of-the-art transmission performance.
• Extensive experimental results demonstrate that Quad-DeepSC is the first DeepSC system to surpass con- ventional communication systems, which employ VTM for source coding and adopt the optimal MCS index under 3GPP standards for channel coding and digital modulation, in performance across datasets of varying resolutions.
• Our project will be available at https://github.com/hyh-bingo/Quad-LIC Quad-DeepSC. (VVC), have achieved remarkable compression performance.

In recent years, the Transformer architecture has achieved outstanding performance across a wide range of tasks and modalities. Token is the unified input and output repre- sentation in Transformer-based models, which has become a fundamental information unit. In this work, we consider the problem of token communication, studying how to transmit tokens efficiently and reliably. Point cloud, a prevailing three- dimensional format which exhibits a more complex spatial structure compared to image or video, is chosen to be the information source. We utilize the set abstraction method to obtain point tokens. Subsequently, to get a more informative and transmission-friendly representation based on tokens, we propose a joint semantic-channel and modulation (JSCCM) scheme for the token encoder, mapping point tokens to standard digital constellation points (modulated tokens). Specifically, the JSCCM consists of two parallel Point Transformer-based encoders and a differential modulator which combines the Gumel-softmax and soft quantization methods. Besides, the rate allocator and channel adapter are developed, facilitating adaptive generation of high- quality modulated tokens conditioned on both semantic informa- tion and channel conditions. Extensive simulations demonstrate that the proposed method outperforms both joint semantic- channel coding and traditional separate coding, achieving over 1dB gain in reconstruction and more than 6× compression ratio in modulated symbols.

• 1 Joint Semantic-Channel Coding and Modulation for Token Communications Jingkai Ying, Graduate Student Member, IEEE, Zhijin Qin, Senior Member, IEEE, Yulong Feng, Liejun Wang, and Xiaoming Tao, Senior Member, IEEE Abstract—In recent years, the Transformer architecture has achieved outstanding performance across a wide range of tasks and modalities.
• Subsequently, to get a more informative and transmission-friendly representation based on tokens, we propose a joint semantic-channel and modulation (JSCCM) scheme for the token encoder, mapping point tokens to standard digital constellation points (modulated tokens).
• Extensive simulations demonstrate that the proposed method outperforms both joint semantic- channel coding and traditional separate coding, achieving over 1dB gain in reconstruction and more than 6× compression ratio in modulated symbols.

As a promising 6G enabler beyond conventional bit-level transmission, semantic communication can considerably reduce required bandwidth resources, while its combination with multiple access requires further exploration. This paper proposes a knowledge distillation-driven and diffusion-enhanced (KDD) semantic non-orthogonal multiple access (NOMA), named KDD- SemNOMA, for multi-user uplink wireless image transmission. Specifically, to ensure robust feature transmission across di- verse transmission conditions, we firstly develop a ConvNeXt- based deep joint source and channel coding architecture with enhanced adaptive feature module. This module incorporates signal-to-noise ratio and channel state information to dynamically adapt to additive white Gaussian noise and Rayleigh fading channels. Furthermore, to improve image restoration quality without inference overhead, we introduce a two-stage knowledge distillation strategy, i.e., a teacher model, trained on interference- free orthogonal transmission, guides a student model via fea- ture affinity distillation and cross-head prediction distillation. Moreover, a diffusion model-based refinement stage leverages generative priors to transform initial SemNOMA outputs into high-fidelity images with enhanced perceptual quality. Extensive experiments on CIFAR-10 and FFHQ-256 datasets demonstrate superior performance over state-of-the-art methods, delivering satisfactory reconstruction performance even at extremely poor channel conditions. These results highlight the advantages in both pixel-level accuracy and perceptual metrics, effectively mitigating interference and enabling high-quality image recovery.

• Semantic communication has achieved remarkable progress in single-user scenarios.
• The paper [16] arXiv:2509.07363v3 [cs.IT] 26 Jan 2026 2 combines beamforming algorithm and semantic coding in the downlink massive multiple-input multiple-output (MIMO) scenario to achieve multi-user image downlink semantic trans- mission.
• Extensive experiments on different datasets have shown that our method significantly outperforms previous multi- user DeepJSCC NOMA methods in terms of image pixel distortion and perceptual metrics, demonstrating its capability to preserve the original transmitted image semantics.

Semantic communication has shown exceptional per- formance in various tasks, such as image classification, owing to the advancements in deep learning technologies. However, due to the openness of wireless channels and the vulnerability of neural networks, semantic communication faces significant challenges from semantic impairment in the physical channel. In this paper, semantic impairment refers to the minor pertur- bations that cause discrepancies between the received features and the expected ones, which can lead to errors in image classification. We design four constraints from the perspectives of semantic level, concealment level and efficiency level to simu- late potential malicious semantic impairment. These constraints are employed to generate adversarial perturbations specifically targeting semantic communication systems, ensuring that the perturbations can more effectively disrupt the normal function of the systems. Moreover, we innovatively propose knowledge graph enhanced anti-impairment cognitive semantic communica- tion, which combines knowledge graph and adversarial training to boost robustness against semantic impairment. Specifically, we leverage the shared knowledge graph to transmit triplet information from the transmitter to the receiver in the form of indices and introduce the triplet information as additional information into the decoder to facilitate the decoding process. Received 18 May 2025; revised 1 September 2025 and 18 November 2025; accepted 18 December 2025. Date of publication 30 December 2025; date of current version 5 February 2026. This work was sup- ported in part by the National Natural Science Foundation of China under Grants 62271267 and 62471254; in part by the Open Research Fund of the National Mobile Communications Research Laboratory, South- east University under Grant 2024D16; in part by the Open Fund of the Anhui Province Key Laboratory of Cyberspace Security Situation Aware- ness and Evaluation under Grant CSSAE-2023-008 and in part by the Jiangsu Provincial Major Science and Technology Special Fund under Grant BG2024002. The associate editor coordinating the review of this article and approving it for publication was H. Du. (Corresponding author: Fuhui Zhou.) Wei Wu is with the College of Communication and Information Engineer- ing, Nanjing University of Posts and Telecommunications, Nanjing 210003, China, also with the National Mobile Communications Research Laboratory, Southeast University, Nanjing 210096, China, and also with Anhui Province Key Laboratory of Cyberspace Security Situation Awareness and Evaluation, Hefei 230037, China (e-mail: weiwu@njupt.edu.cn). Tianle Yao and Han Hu are with the College of Communication and Information Engineering, Nanjing University of Posts and Telecom- munications, Nanjing 210003, China (e-mail: 1023010436@njupt.edu.cn; hanhu@njupt.edu.cn). Fuhui Zhou and Qihui Wu are with the College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing 210000, China (e-mail: zhoufuhui@ieee.org; wuqihui2014@sina.com). Zhijin Qin is with the Department of Electronic Engineering, Tsinghua University, Beijing 100084, China, also with the State Key Laboratory of Space Network and Communications, Beijing 100084, China, and also with Beijing National Research Center for Information Science and Technology, Beijing 100084, China (e-mail: qinzhijin@tsinghua.edu.cn). Digital Object Identifier 10.1109/TCOMM.2025.3649647 Simulation results show that our proposed knowledge graph enhanced cognitive semantic communication system achieves higher classification accuracy and robustness in environments with low signal-to-noise ratio and semantic impairment, com- pared to existing Better Portable Graphics (BPG) and Joint Source-Channel Coding(JSCC) schemes.

• Digital Object Identifier 10.1109/TCOMM.2025.3649647 Simulation results show that our proposed knowledge graph enhanced cognitive semantic communication system achieves higher classification accuracy and robustness in environments with low signal-to-noise ratio and semantic impairment, com- pared to existing Better Portable Graphics (BPG) and Joint Source-Channel Coding(JSCC) schemes.
• Through pre-training and fine- tuning, this model can learn deep semantic information from context and achieve excellent performance in many NLP tasks.
• In this way, Xie achieved a balance between transmission quality and band- width efficiency.

The rapid development of generative artificial intel- ligence (AI) has introduced significant opportunities for enhanc- ing the efficiency and accuracy of image transmission within semantic communication systems. Despite these advancements, existing methodologies often neglect the difference in importance of different regions of the image, potentially compromising the reconstruction quality of visually critical content. To address this issue, we introduce an innovative generative semantic communication system that refines semantic granularity by segmenting images into key and non-key regions. Key regions, which contain essential visual information, are processed using an image oriented semantic encoder, while non-key regions are efficiently compressed through an image-to-text modeling approach. Additionally, to mitigate the substantial storage and computational demands posed by large AI models, the proposed system employs a lightweight deployment strategy incorporat- ing model quantization and low-rank adaptation fine-tuning techniques, significantly boosting resource utilization without sacrificing performance. Simulation results demonstrate that the proposed system outperforms traditional methods in terms of both semantic fidelity and visual quality, thereby affirming its effectiveness for image transmission tasks.

• Simulation results demonstrate that the proposed system outperforms traditional methods in terms of both semantic fidelity and visual quality, thereby affirming its effectiveness for image transmission tasks.
• Considering the computational and storage requirements of these large generative models, model lightweight is essential to achieve large-scale edge deployment of GAI-based se- mantic communication systems.
• The main contributions of this paper are summarized as follows: • We propose a generative semantic communication frame- work that identifies the minimal set of essential image features for transmission, while encoding the remaining residual information as structured text prompts.

In this paper, we introduce a large model- empowered streaming semantic communication system for speech transmission across various languages, named LSSC-ST. Specifi- cally, we devise an edge-device collaborative semantic communi- cation architecture by offloading the intricate semantic extraction and channel coding modules to edge servers, thereby reducing the computational burden on local devices. To support multilingual speech transmission, pre-trained large speech models are utilized to learn unified semantic features from speech in different languages, breaking the constraint of a single input language and enhancing the practicality of the LSSC-ST. Moreover, the input speech is sequentially streamed into the developed system as short speech segments, which enables low transmission latency without degrading the quality of the produced speech. A novel dynamic speech segmentation algorithm is proposed to further reduce the transmission latency by adaptively adjusting the duration of speech segments. According to simulation results, the LSSC- ST provides more accurate speech transmission and achieves a streaming manner with lower latency compared to the existing non-streaming semantic communication systems.

• According to simulation results, the LSSC- ST provides more accurate speech transmission and achieves a streaming manner with lower latency compared to the existing non-streaming semantic communication systems.
• The conventional communication paradigm falls under syntax communications, quantifies information at the bit level, and aims to achieve a low bit-error rate (BER) or symbol- error rate (SER).
• Section IV presents experimental results and Section V concludes this paper.

Existing speech semantic communication systems mainly based on Joint Source-Channel Coding (JSCC) archi- tectures have demonstrated impressive performance, but their effectiveness remains limited by model structures specifically de- signed for particular tasks and datasets. Recent advances indicate that generative large models pre-trained on massive datasets, can achieve outstanding performance arexhibit exceptional per- formance across diverse downstream tasks with minimal fine- tuning. To exploit the rich semantic knowledge embedded in large models and enable adaptive transmission over lossy channels, we propose a Large Speech Model enabled Semantic Commu- nication (LargeSC) system. Simultaneously achieving adaptive compression and robust transmission over lossy channels remains challenging, requiring trade-offs among compression efficiency, speech quality, and latency. In this work, we employ the Mimi as a speech codec, converting speech into discrete tokens compatible with existing network architectures. We propose an adaptive controller module that enables adaptive transmission and in- band Unequal Error Protection (UEP), dynamically adjusting to both speech content and packet loss probability under band- width constraints. Additionally, we employ Low-Rank Adaptation (LoRA) to finetune the Moshi foundation model for generative recovery of lost speech tokens. Simulation results show that the proposed system supports bandwidths ranging from 550 bps to 2.06 kbps, outperforms conventional baselines in speech quality under high packet loss rates and achieves an end-to-end latency of approximately 460 ms, thereby demonstrating its potential for real-time deployment.

• Recent advances indicate that generative large models pre-trained on massive datasets, can achieve outstanding performance arexhibit exceptional per- formance across diverse downstream tasks with minimal fine- tuning.
• To exploit the rich semantic knowledge embedded in large models and enable adaptive transmission over lossy channels, we propose a Large Speech Model enabled Semantic Commu- nication (LargeSC) system.
• Simultaneously achieving adaptive compression and robust transmission over lossy channels remains challenging, requiring trade-offs among compression efficiency, speech quality, and latency.

Artificial intelligence (AI) promises to revolutionize the design, optimization and management of next-generation communication systems. In this article, we explore the integra- tion of large AI models (LAMs) into semantic communications (SemCom) by leveraging their multi-modal data processing and generation capabilities. Although LAMs bring unprecedented abilities to extract semantics from raw data, this integration entails multifaceted challenges including high resource demands, model complexity, and the need for adaptability across diverse modalities and tasks. To overcome these challenges, we propose a LAM-based multi-task SemCom (MTSC) architecture, which includes an adaptive model compression strategy and a federated split fine-tuning approach to facilitate the efficient deployment of LAM-based semantic models in resource-limited networks. Furthermore, a retrieval-augmented generation scheme is imple- mented to synthesize the most recent local and global knowledge bases to enhance the accuracy of semantic extraction and content generation, thereby improving the inference performance. Fi- nally, simulation results demonstrate the efficacy of the proposed LAM-based MTSC architecture, highlighting the performance enhancements across various downstream tasks under varying channel conditions.

• To overcome these challenges, we propose a LAM-based multi-task SemCom (MTSC) architecture, which includes an adaptive model compression strategy and a federated split fine-tuning approach to facilitate the efficient deployment of LAM-based semantic models in resource-limited networks.
• Fi- nally, simulation results demonstrate the efficacy of the proposed LAM-based MTSC architecture, highlighting the performance enhancements across various downstream tasks under varying channel conditions.
• Recently, by extracting semantics from different data modalities, multi- modal SemCom is able to achieve a more comprehensive understanding of the multi-modal input [5].

The non-terrestrial network (NTN) is a key technology that will enable global coverage and seamless connectivity, while the limited available bandwidth and unstable channel conditions hinder the development of NTNs. Semantic communication focuses on the successful transmission of significant information rather than all bits of data, which offers a potential solution to break through the performance bottleneck in NTNs. In order to achieve efficient and reliable transmission, this article explores the role of semantic communication in NTNs and proposes a semantic communication- empowered NTN framework. The key components in the proposed framework are explained in detail, from data generation to transmission. Illustrative use cases and initial numerical results are demonstrated to show that semantic communication could be an enabler in enhancing efficiency and robustness in NTN-assisted Internet of Things and emergency communication scenarios. Challenges and future directions are also discussed to provide an insightful outlook on the implementation of semantic communication in NTNs.

• In order to achieve efficient and reliable transmission, this arti- cle explores the role of semantic communication in NTNs and proposes a semantic communica- tion-empowered NTN framework.
• Then, we propose an architecture of semantic communica- tion-empowered NTN and introduce in detail the key role of semantic communication in NTN from data generation to transmission.
• Although voice transmission requires less band- width compared to video transmission, high-fidelity voice transmission is still difficult to achieve in exist- ing NTNs.

Semantic communications have the potential to improve transmission efficiency and support intelligent tasks. However, the commonly used global sampling-based pattern ignores the fact that the data processing ability of edge trans- mitters is strictly limited and only a small part of information is available for a single sample in some scenarios. This paper proposes a novel partial sampling-based semantic communication (PSSC) framework where an edge transmitter is guided by feedback from the receiver to locate and collect only part of content relevant to the target task. Taking the vision-based task as an example, the transmitter selectively samples a small patch of a large-size image until the intelligent task is successfully executed at the receiver. The selection of sampling location is modeled as a partially observable Markov decision process problem and an intelligent approach based on reinforcement learning is proposed to solve the problem. In addition, a recurrent neural network-based receiver is designed to fuse information received over multiple transmission rounds. Besides, we prove that the feedback does not increase the semantic channel capacity. Simulation results demonstrate that the proposed framework can locate the informative areas accurately and achieve competitive performance compared to the existing global sampling-based methods.

• Simulation results demonstrate that the proposed framework can locate the informative areas accurately and achieve competitive performance compared to the existing global sampling-based methods.
• Existing studies have explored various transceiver structures to achieve effective semantic communications [10], [11], [12], [13], [14], [15], [16].
• Indeed, the best performance can be achieved by using a very complex NN.

We present a novel framework called progressive learned image transmission (PLIT) that utilizes a hierarchical variational autoencoder (VAE) to catalyze semantic communica- tion. PLIT employs autoregressive generation through bottom-up and top-down paths to create several feature representa- tions of the transmitted image, effectively capturing contextual information. In this paper, we investigate the progressive trans- mission of these representations, particularly in the context of successive refinement. In this scenario, the representations are sent to the receiver in phases, with representations in later phases serving to enhance the image quality. Diverging from previous works, our proposed PLIT offers improved flexibility as it is able to dynamically determine the transmission rate. Specifically, PLIT can transform each representation into different numbers of channel symbols, guided by the hierarchical VAE’s learned priors that indicate the entropy of each representation. In addition, we devise a rate attention mechanism to help adjust the encoding strategy to realize different transmission rates. Furthermore, we introduce a spatial grouping strategy to reduce communication overhead for rate matching without compromis- ing image fidelity. Extensive experiments show that our proposed approach outperforms existing baseline methods in terms of rate-distortion performance and maintains robust performance against channel noise.

• 3640 IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. 11, NO. 6, DECEMBER 2025 Progressive Learned Image Transmission for Semantic Communication Using Hierarchical VAE Guangyi Zhang , Graduate Student Member, IEEE, Hanlei Li, Yunlong Cai , Senior Member, IEEE, Qiyu Hu , Student Member, IEEE, Guanding Yu , Senior Member, IEEE, and Zhijin Qin , Senior Member, IEEE Abstract—We present a novel framework called progressive learned image transmission (PLIT) that utilizes a hierarchical variational autoencoder (VAE) to catalyze semantic communica- tion.
• Extensive experiments show that our proposed approach outperforms existing baseline methods in terms of rate-distortion performance and maintains robust performance against channel noise.
• ZHANG et al.: PLIT FOR SEMANTIC COMMUNICATION USING HIERARCHICAL VAE 3641 For instance, the receiver can terminate receiving once it achieves the desired vision quality, thereby saving time and energy.

In this paper, we propose a robust semantic com- munication system for speech transmission, named Ross-S2T, by delivering the essential semantic information. Specifically, we consider the speech-to-text translation (S2TT) as the transmission goal. First, a new deep semantic encoder is developed to convert speech in the source language to textual features associated with the target language, facilitating the end-to-end semantic exchange to perform the S2TT task and reducing the transmission data without performance degradation. To mitigate semantic impairments inherent in the corrupted speech, a novel generative adversarial network (GAN)-enabled deep semantic compensator is established to estimate the lost semantic information within the speech and extract deep semantic features simultaneously, which enables robust semantic transmission for corrupted speech. Furthermore, a semantic probe-aided compensator is devised to enhance the semantic fidelity of recovered semantic features and improve the understandability of the target text. According to simulation results, the proposed Ross-S2T exhibits superior S2TT performance compared to conventional approaches and high robustness against semantic impairments.

• arXiv:2403.05187v3 [eess.AS] 4 Jul 2025 Robust Semantic Communications for Speech Transmission Zhenzi Weng∗, Zhijin Qin†, and Geoffrey Ye Li∗ ∗Department of Electrical and Electronic Engineering, Imperial College London, London, UK † Department of Electronic Engineering, Tsinghua University, Beijing, China Email: z.weng@imperial.ac.uk, qinzhijin@tsinghua.edu.cn, geoffrey.li@imperial.ac.uk Abstract—In this paper, we propose a robust semantic com- munication system for speech transmission, named Ross-S2T, by delivering the essential semantic information.
• According to simulation results, the proposed Ross-S2T exhibits superior S2TT performance compared to conventional approaches and high robustness against semantic impairments.
• The advancement of semantic communications derives from the ability to explore semantic information and achieve semantic exchange, which revolution- izes many aspects of wireless communications [2].

The burgeoning technology of deep learning-based semantic communications has significantly enhanced the effi- ciency and reliability of wireless communication systems by facilitating the transmission of semantic features. However, secu- rity threats, notably the interception of sensitive data, remain a significant challenge for secure communications. To safeguard the confidentiality of transmitted semantics and effectively coun- teract eavesdropping threats, this letter proposes a secure deep learning-based semantic communication system, SecureDSC. It comprises semantic encoder/decoder, channel encoder/decoder, and encryption/decryption modules with a key processing net- work. By incorporating a symmetric encryption module and an attacker-oriented adversarial network, SecureDSC guarantees the secure transmission between legitimate users in the semantic communications. Besides, experiments are conducted to evaluate the effectiveness and feasibility of the proposed scheme.

• Compared to traditional communication systems, semantic communications achieve higher transmission efficiency and lower bandwidth with less susceptible to noise or other interference [2].
• To counteract eavesdropping attacks within wireless chan- nels in semantic communications, we propose an encrypted semantic communication framework that guarantees secure transmission while significantly maintaining the performance of semantics extraction and interpretation.
• The system is based on the DeepSC proposed in [2], which has demonstrated excellent performance in achieving high data rates and minimizing semantic errors.

Underwater communication is essential for envi- ronmental monitoring, marine biology research, and underwa- ter exploration. Traditional underwater communication faces limitations like low bandwidth, high latency, and susceptibility to noise, while semantic communication (SC) offers a promis- ing solution by focusing on the exchange of semantics rather than symbols or bits. However, SC encounters challenges in underwater environments, including semantic information mis- match and difficulties in accurately identifying and transmitting critical information that aligns with the diverse requirements of underwater applications. To address these challenges, we propose a novel Semantic Communication (SC) framework based on Large Language Models (LLMs). Our framework leverages visual LLMs to perform semantic compression and prioritization of underwater image data according to the query from users. By identifying and encoding key semantic elements within the images, the system selectively transmits high-priority information while applying higher compression rates to less critical regions. On the receiver side, an LLM-based recovery mechanism, along with Global Vision ControlNet and Key Region ControlNet networks, aids in reconstructing the images, thereby enhancing communication efficiency and robustness. Our framework re- duces the overall data size to 0.8% of the original. Experimental results demonstrate that our method significantly outperforms existing approaches, ensuring high-quality, semantically accurate image reconstruction.

• To address these challenges, we propose a novel Semantic Communication (SC) framework based on Large Language Models (LLMs).
• Experimental results demonstrate that our method significantly outperforms existing approaches, ensuring high-quality, semantically accurate image reconstruction.
• In contrast, SC can extract semantic information, achieving accurate semantic transmission [15] and reducing the amount of transmitted data, thereby optimizing the use of limited bandwidth.

With the rapid growth of intelligent services, com- munication targets are shifting from humans to artificial intelli- gent (AI) agents, which require new paradigms to enable real- time perception, decision-making, and collaboration. Semantic communication, which conveys task-relevant meaning rather than raw data, offers a promising solution. However, its practical deployment remains constrained by dynamic environments and limited resources. To address these issues, this article proposes a semantic-driven AI agent communication framework and devel- ops three enabling techniques. First, semantic adaptation trans- mission applies fine-tuning with real or generative samples to efficiently adapt models to varying environments. Second, seman- tic lightweight transmission incorporates pruning, quantization, and perception-aware sampling to reduce model complexity and alleviate computational burden on edge agents. Third, semantic self-evolution control employs distributed hierarchical decision- making to optimize multi-dimensional resources, enabling robust multi-agent collaboration in dynamic environments. Simulation results show that the proposed solutions achieve faster conver- gence and stronger robustness, while the proposed distributed hierarchical optimization method significantly outperforms con- ventional decision-making schemes, highlighting its potential for AI agent communication networks.

• Simulation results show that the proposed solutions achieve faster conver- gence and stronger robustness, while the proposed distributed hierarchical optimization method significantly outperforms con- ventional decision-making schemes, highlighting its potential for AI agent communication networks.
• Experimental results demonstrate that the proposed strategies significantly improve task execution efficiency.
• Through capability orchestration, multiple agents can form cooperative networks to achieve collective intelli- gence.

Packet loss is a critical issue in real-time video transmission systems. While video semantic communication with joint source-channel coding designs has demonstrated high per- formance in wireless channels at the physical layer, strategies for mitigating the effects of packet loss at higher layers have not been well studied. It is difficult to localize lost semantic features under packet loss scenarios, as semantic features need to be converted into bitstreams and then encapsulated into packets. In this paper, we propose a semantic-enabled video encoder that creates mappings between semantic information and data packets, leveraging the Packet Erasure Channel (PEC) to simulate packet loss conditions. Additionally, we design a semantic-enabled video decoder that improves information recovery in poor channel conditions. Furthermore, a semantic quality optimizer is placed at the receiver to enhance the adaptability of the real-time video transmission system under time-varying PEC. Experimental results show that the proposed scheme offers superior robustness against packet loss scenarios compared to conventional and AI- based benchmarks.

• In this paper, we propose a semantic-enabled video encoder that creates mappings between semantic information and data packets, leveraging the Packet Erasure Channel (PEC) to simulate packet loss conditions.
• Experimental results show that the proposed scheme offers superior robustness against packet loss scenarios compared to conventional and AI- based benchmarks.
• To achieve better compression of semantic features, video semantic compression [11], [12] typically predicts feature dis- tributions and uses entropy coding to transform them into bitstreams.

Although the semantic communication with joint semantic-channel coding design has shown promising perfor- mance in transmitting data of different modalities over physical layer channels, the synchronization and packet-level forward error correction of multimodal semantics have not been well studied. Due to the independent design of semantic encoders, synchronizing multimodal features in both the semantic and time domains is a challenging problem. In this paper, we take the facial video and speech transmission as an example and propose a Synchronous Multimodal Semantic Communication System (SyncSC) with Packet-Level Coding. To achieve semantic and time synchronization, 3D Morphable Mode (3DMM) coeffi- cients and text are transmitted as semantics, and we propose a semantic codec that achieves similar quality of reconstruction and synchronization with lower bandwidth, compared to traditional methods. To protect semantic packets under the erasure channel, we propose a packet-Level Forward Error Correction (FEC) method, called PacSC, that maintains a certain visual quality performance even at high packet loss rates. Particularly, for text packets, a text packet loss concealment module, called TextPC, based on Bidirectional Encoder Representations from Transformers (BERT) is proposed, which significantly improves the performance of traditional FEC methods. The simulation results show that our proposed SyncSC reduce transmission overhead and achieve high-quality synchronous transmission of video and speech over the packet loss network. Index Terms—Semantic communication, synchronization, packet-level forward correction, talking face transmission, speech transmission.

• To achieve semantic and time synchronization, 3D Morphable Mode (3DMM) coeffi- cients and text are transmitted as semantics, and we propose a semantic codec that achieves similar quality of reconstruction and synchronization with lower bandwidth, compared to traditional methods.
• To protect semantic packets under the erasure channel, we propose a packet-Level Forward Error Correction (FEC) method, called PacSC, that maintains a certain visual quality performance even at high packet loss rates.
• The simulation results show that our proposed SyncSC reduce transmission overhead and achieve high-quality synchronous transmission of video and speech over the packet loss network.

: In recent years, deep learning-based se- mantic communications have shown great potential to enhance the performance of communication sys- tems. This has led to the belief that semantic commu- nications represent a breakthrough beyond the Shan- non paradigm and will play an essential role in future communications. To narrow the gap between current research and future vision, after an overview of se- mantic communications, this article presents and dis- cusses ten fundamental and critical challenges in to- day’s semantic communication field. These challenges are divided into theory foundation, system design, and practical implementation. Challenges related to the theory foundation including semantic capacity, en- tropy, and rate-distortion are discussed first. Then, the system design challenges encompassing architecture, knowledge base, joint semantic-channel coding, tai- lored transmission scheme, and impairment are posed. The last two challenges associated with the practical implementation lie in cross-layer optimization for net- works and standardization. For each challenge, efforts to date and thoughtful insights are provided.

• It is important to recognize that even though current DL-based semantic communications have achieved significant success, this does not diminish the impor- tance of continued research in semantic theory.
• One of the most important and famous achievements of Shannon’s information theory is channel capacity.
• This suggests that by employing a semantic encoder with low semantic ambiguity and a semantic decoder with robust inference capabilities, combined with a substantial shared knowledge base, it is possible to achieve high-rate semantic communi- cation using a low-rate physical channel.

Accurate and timely image transmission is critical for emerging time-sensitive applications such as remote sensing in satellite-assisted Internet of Things. However, the bandwidth limitation poses a significant challenge in existing wireless systems, making it difficult to fulfill the requirements of both high-fidelity and low-latency image transmission. Semantic communication is expected to break through the performance bottleneck by focusing on the transmission of goal-oriented semantic information rather than raw data. In this paper, we employ a new timeliness metric named the value of information (VoI) and propose an adaptive joint source and channel coding (JSCC) method for image transmission that simultaneously considers both reconstruction quality and timeliness. Specifically, we first design a JSCC framework for image transmission with adaptive code length. Next, we formulate a VoI maximization problem by optimizing the transmission code length of the adaptive JSCC under the reconstruction quality constraint. Then, a deep reinforcement learning-based algorithm is proposed to solve the optimization problem efficiently. Experimental results show that the proposed method significantly outperforms baseline schemes in terms of reconstruction quality and timeliness, particularly in low signal- to-noise ratio conditions, offering a promising solution for efficient and robust image transmission in time-sensitive wireless networks.

• Experimental results show that the proposed method significantly outperforms baseline schemes in terms of reconstruction quality and timeliness, particularly in low signal- to-noise ratio conditions, offering a promising solution for efficient and robust image transmission in time-sensitive wireless networks.
• Motivated by the above observations, we propose a timeliness-aware semantic communication framework that arXiv:2509.19754v1 [eess.SP] 24 Sep 2025 jointly considers reconstruction quality and transmission time- liness, using an adaptive semantic codec and dynamic code length allocation.
• The main contributions of this paper are summarized as follows: • We propose a novel JSCC framework for image trans- mission with adaptive code length, which is able to simultaneously guarantee the reconstruction quality and timeliness. • We utilize a new performance metric and formulate an op- timization problem to enhance information timeliness by controlling the code length of JSCC under reconstruction quality constraints. • We implement an adaptive JSCC codec along with a deep reinforcement learning (DRL)-based optimizer, achieving an effective trade-off between image reconstruction qual- ity and information timeliness.

With the explosive growth of network scale and data volume, wireless communication is facing an increasingly severe limitation of spectrum resources. Semantic communi- cation has emerged as a promising paradigm to break the bandwidth bottleneck by transmitting significant task-oriented semantic information rather than raw data. In practical real- time wireless applications, semantics of information exhibit diverse spatial and temporal correlations depending on in- trinsic dynamics of source and extrinsic dynamics of environ- ment. Motivated by this observation, this paper develops a novel information-theoretic metric to quantify the semantic value of spatiotemporal information. Specifically, a semantic value of information (SVoI) framework is proposed based on the mutual information, which characterises the reduction in uncertainty when predicting an unknown system state using past semantic spatiotemporal correlated observations. Focusing on general Gaussian Markov models, closed-form expressions of the SVoI are derived. Effects of both separable and coupled spatiotemporal correlations on SVoI are further investigated analytically. Numerical simulations are conducted to validate the theoretical analysis of SVoI and its bounds. The proposed SVoI metric jointly captures the impact of semantic spatiotemporal correlation of source, timeliness of information, and channel conditions, which could serve as an effective optimisation objective for the design of next- generation semantic-aware communication systems.

• The performance of the SVoI is verified through simulation results.
• In contrast, upper bounds represent the ideal maximum achievable SVoI , serving as a theoretical benchmark for optimal system performance.

Semantic communication has demonstrated signifi- cant potential for image transmission, especially in bandwidth- limited and low signal-to-noise ratio scenarios. However, most existing methods are based on analog transmission, which poses challenges to the compatibility with existing digital communica- tion systems. Existing digital semantic communication methods commonly adopt conventional quadrature amplitude modulation constellations, which mismatch the empirical distribution of semantic features produced by the semantic encoder. This paper proposes a distribution-aware learnable modulation for seman- tic communication framework, which bridges semantic feature representations and discrete modulation through constellation learning. Specifically, a learnable constellation module, initialized with an amplitude phase shift keying geometric prior, is devel- oped to refine the constellation geometry as a trainable codebook, enabling modulation symbols to better align with the distribution of semantic features. To enable end-to-end optimization, a two- stage training strategy is introduced, combining differentiable soft assignment with straight-through estimator. Simulation re- sults show that the proposed framework consistently outperforms existing digital semantic communication schemes and achieves performance comparable to advanced analog methods. Index Terms—Semantic communication, joint semantic- channel coding, digital modulation, constellation learning.

• Simulation re- sults show that the proposed framework consistently outperforms existing digital semantic communication schemes and achieves performance comparable to advanced analog methods.
• The pioneering DeepJSCC [3] showed that a neural transceiver can directly map images to channel symbols and outperform conventional digital schemes in bandwidth- limited or low signal-to-noise ratio (SNR) regimes.
• To address the issue, we develop a distribution-aware learnable modulation for semantic communication (DLM-SC).

Real-time immersive video communications, partic- ularly high-fidelity 3D telepresence, necessitates a synergistic balance between instantaneous dynamic scene reconstruction and high-efficiency data transmission. While recent advancements in feed-forward 3D Gaussian Splatting (3DGS) have enabled real-time rendering, performing multi-view video coding and 3D reconstruction in a decoupled manner leads to suboptimal compression efficiency and high computational complexity. To address this, we propose GS-SCNet, the first unified end-to-end framework that seamlessly integrates Generalizable 3D Gaussian Splatting reconstruction with a dedicated deep Semantic Coding pipeline. Our architecture is underpinned by two core technical contributions: (i) we introduce a Disparity-Guided Parallel Se- mantic Codec that exploits epipolar geometric priors to facilitate cross-view contextual interaction via disparity compensation and semantic fusion, thereby enabling real-time parallel processing of stereo streams while significantly enhancing rate-distortion performance, and (ii) we develop a Lightweight Gaussian Param- eter Predictor which directly projects decoded semantic latents into 3DGS attributes, obviating the necessity for intermediate pixel-domain reconstruction. By coupling the codec with the task- specific predictor, our framework extracts geometric correlations only once, effectively eliminating the redundant computational bottleneck inherent in conventional decoupled paradigms. Ex- tensive evaluations on both synthetic and real-world human datasets demonstrate that GS-SCNet achieves a superior trade- off across compression efficiency, rendering quality, and real- time performance. Notably,our framework exhibits strong cross- domain generalization and robustness against compression arti- facts when applied to out-of-domain real-world data, significantly outperforming conventional decoupled transmission paradigms.

• To address this, we propose GS-SCNet, the first unified end-to-end framework that seamlessly integrates Generalizable 3D Gaussian Splatting reconstruction with a dedicated deep Semantic Coding pipeline.
• Our architecture is underpinned by two core technical contributions: (i) we introduce a Disparity-Guided Parallel Se- mantic Codec that exploits epipolar geometric priors to facilitate cross-view contextual interaction via disparity compensation and semantic fusion, thereby enabling real-time parallel processing of stereo streams while significantly enhancing rate-distortion performance, and (ii) we develop a Lightweight Gaussian Param- eter Predictor which directly projects decoded semantic latents into 3DGS attributes, obviating the necessity for intermediate pixel-domain reconstruction.
• Ex- tensive evaluations on both synthetic and real-world human datasets demonstrate that GS-SCNet achieves a superior trade- off across compression efficiency, rendering quality, and real- time performance.

摘要未获。

• 未从全文自动抽取到足够稳定的结果句；请结合摘要与原文。

Task-oriented semantic communication (TOSC) envisions efficient information exchange for the social Internet of Things (SIoT). However, deploying federated learning (FL) for TOSC faces critical challenges arising from non-independent and identically distributed (non-IID) data and diverse fading channels. Conventional static curriculum methods are often impractical in such dynamic and resource-limited environments. In this letter, we propose a novel framework that incorporates meta-curriculum learning (MCL) into the federated seman- tic communication to achieve adaptive and robust training. Specifically, we design a lightweight, plug-and-play reinforce- ment learning (RL) agent that dynamically optimizes local data scheduling via intrinsic feedback, eliminating dependencies on external public datasets. Simulation results demonstrate that MCL reduces communication rounds by 56% compared to static curriculum learning within the FedAvg and FedDF frameworks, confirming its scalability and generalizability in heterogeneous edge networks.

• In this letter, we propose a novel framework that incorporates meta-curriculum learning (MCL) into the federated seman- tic communication to achieve adaptive and robust training.
• Simulation results demonstrate that MCL reduces communication rounds by 56% compared to static curriculum learning within the FedAvg and FedDF frameworks, confirming its scalability and generalizability in heterogeneous edge networks.
• Therefore, fixed rules struggle to achieve optimal resource allocation.

In high-mobility scenarios with time-frequency doubly-selective channels, existing semantic communication sys- tems suffer significant performance degradation. To address this issue, we propose a semantic communication framework that synergistically integrates multiple-input multiple-output orthog- onal time frequency space (MIMO-OTFS) with semantic-aware sub-channel allocation. First, an entropy module is employed to evaluate importance of different semantic features, and the Kendall correlation coefficient is used to quantify the align- ment between semantic importance and sub-channel conditions. Subsequently, joint optimization of the encoder and decoder is achieved through a comprehensive loss function that balances image classification accuracy, reconstruction quality, and sub- channel matching degree. Experimental results confirm the superior reconstruction quality of our proposed framework com- pared to conventional semantic communication systems based on orthogonal frequency division multiplexing in high-mobility channel environment.

• To address this issue, we propose a semantic communication framework that synergistically integrates multiple-input multiple-output orthog- onal time frequency space (MIMO-OTFS) with semantic-aware sub-channel allocation.
• Subsequently, joint optimization of the encoder and decoder is achieved through a comprehensive loss function that balances image classification accuracy, reconstruction quality, and sub- channel matching degree.
• Experimental results confirm the superior reconstruction quality of our proposed framework com- pared to conventional semantic communication systems based on orthogonal frequency division multiplexing in high-mobility channel environment.

Ultra-high-resolution streaming and emerging im- mersive services are driving rapidly increasing wireless video traffic. However, perceptually pleasing video transmission over bandwidth-limited and latency-constrained wireless links remains challenging for conventional separated source-channel systems, which primarily target bit-level reliability and often suffer per- formance degradation under short-blocklength transmission. In addition, pixel-level distortion optimization does not necessarily align with human perception, while existing learned video codecs may incur high complexity and raise deployment issues. This paper proposes PVSC, a perception-aware video semantic com- munication framework for real-time wireless video transmission. PVSC eliminates explicit motion-vector transmission and exploits spatio-temporal feature coding to generate compact and channel- robust symbol streams. It also specifies side-information format- ting, reference-buffer management, and lightweight rate con- trol, enabling stable receiver-side reconstruction and bandwidth- adaptive inference with a single model. Extensive experiments demonstrate that PVSC achieves superior performance across diverse datasets, resolutions, GOP configurations, and channel conditions. Compared with the engineered “VTM + 5G LDPC” baseline, PVSC saves up to about 75% and 87% bandwidth at comparable LPIPS and DISTS, respectively, while enabling real- time inference on a single NVIDIA RTX 4090 GPU.

• Extensive experiments demonstrate that PVSC achieves superior performance across diverse datasets, resolutions, GOP configurations, and channel conditions.
• As shown by the “DCVC-RT/VTM-17.0 + 5G LDPC” scheme, LPIPS-oriented optimization achieves better perceptual quality at lower bandwidth, which is also supported by DISTS.
• Motivated by these gaps, we propose a perception-aware video semantic communication system (PVSC) tailored to bandwidth-limited and latency-constrained wireless links.

Holographic video communication is considered a paradigm shift in visual communications, becoming increasingly popular for its ability to offer immersive experiences. This article provides an overview of holographic video communication and outlines the requirements of a holographic video communication system. Particularly, following a brief review of semantic com- munication, an architecture for a semantic-enabled holographic video communication system is presented. Key technologies, including semantic sampling, joint semantic-channel coding, and semantic-aware transmission, are designed based on the proposed architecture. Two related use cases are presented to demonstrate the performance gain of the proposed methods. Finally, potential research topics are discussed to pave the way for the realization of semantic-enabled holographic video communications.

• Architecture of a Semantic-Aware Holographic Video Com- munication System We propose an architecture for semantic-aware HVC as demonstrated in Fig. 3.
• Therefore, we propose a DL-based sampling model that enables the utilization of semantic information.
• These sampled points achieve superior preservation of geometric structures in critical regions and exhibit enhanced task-specific representation capabilities.

Integrated learning and communication (ILAC) unifies learned transceivers with radio resource management, where semantic feature multiple access (SFMA) enables paired users to superpose their learned representations over shared time-frequency resources. Unlike conventional multiple access schemes, SFMA interference arises in the learned feature space and depends jointly on the user pair, the transmit power, and the compression ratio. This coupling ties binary pairing deci- sions to continuous resource variables, yielding a mixed-integer non-convex optimisation problem. To address this problem, we first propose similarity-conditioned SFMA (SC-SFMA), a Swin Transformer-based transceiver whose dual-conditioned similarity modulator (DC-SimM) gates cross-user feature fusion according to the inter-user semantic similarity. We then characterise the resulting pair-dependent interference by a bivariate logistic function parameterised by transmit power and compression ratio, thereby bridging the learned transceiver with network- level optimisation. On this basis, we formulate a sum-rate maximisation problem subject to per-user distortion, latency, energy, power, and bandwidth constraints. To solve this problem, we develop a three-block alternating optimisation algorithm that integrates dual-decomposition-assisted compression ratio allocation, trust-region successive convex approximation (SCA) for joint power–bandwidth optimisation, and dynamic feasible graph-based user pairing. Simulation results show that SC-SFMA achieves considerable peak signal-to-noise ratio (PSNR) and multi-scale structural similarity index measure (MS-SSIM) gains over deep joint source–channel coding (JSCC) and separation- based baselines. The proposed optimisation framework attains significant sum rate improvements than conventional multiple access baselines.

• To solve this problem, we develop a three-block alternating optimisation algorithm that integrates dual-decomposition-assisted compression ratio allocation, trust-region successive convex approximation (SCA) for joint power–bandwidth optimisation, and dynamic feasible graph-based user pairing.
• Simulation results show that SC-SFMA achieves considerable peak signal-to-noise ratio (PSNR) and multi-scale structural similarity index measure (MS-SSIM) gains over deep joint source–channel coding (JSCC) and separation- based baselines.
• Contributions The main contributions of this paper are summarised as follows. • We propose SC-SFMA, a Swin Transformer-based transceiver whose dual-conditioned similarity modulator (DC-SimM) adaptively gates cross-user feature fusion ac- cording to inter-user semantic similarity, enabling grace- ful interpolation between independent coding and full feature fusion within the ILAC framework. • We establish a pair-conditioned analytical framework that bridges the learned transceiver with radio resource management.

Semantic communications (SemCom) is a promising task-oriented paradigm in which semantic features exhibit non- uniform importance. Consequently, unequal error protection (UEP), which allocates resources based on semantic importance, plays a pivotal role in maximizing system utility. However, most existing schemes adopt passive importance evaluation, which neither proactively reshapes the importance distribution nor explores its impact on UEP performance. In this paper, we propose a novel importance-ordered semantic feature restruc- turing (ISFR) scheme that proactively enforces a descending importance hierarchy and jointly optimizes multi-dimensional resources to improve system utility. Specifically, modules with decreasing retention probabilities and increasing distortion levels are employed, which drive the model to concentrate key semantics into front-end features and thus strengthen importance differen- tiation. Moreover, a joint optimization problem that jointly opti- mizes channel matching, feature selection, modulation schemes, and power allocation is formulated to minimize the importance- weighted total semantic distortion. To solve this non-convex problem, a hierarchical decoupling strategy is proposed, which decomposes it into four tractable subproblems. This approach leverages the ordered prior to drastically prune the search space for feature selection and modulation, while integrating greedy- based channel matching and convex power allocation. Simulation results demonstrate that the proposed ISFR scheme outperforms traditional uniform importance-based schemes under harsh chan- nel conditions and limited resources, validating the significant robustness improvement enabled by the concentration of key semantic information.

• In this paper, we propose a novel importance-ordered semantic feature restruc- turing (ISFR) scheme that proactively enforces a descending importance hierarchy and jointly optimizes multi-dimensional resources to improve system utility.
• Simulation results demonstrate that the proposed ISFR scheme outperforms traditional uniform importance-based schemes under harsh chan- nel conditions and limited resources, validating the significant robustness improvement enabled by the concentration of key semantic information.
• SI-UEP characterizes the task contribution of semantic features as semantic importance, then leverages it to guide differentiated transmission and resource allocation, thereby achieving non-uniform reliability.

Semantic communication, enabled by deep joint source-channel coding (DeepJSCC), is widely expected to inherit the vulnerability of deep learning to adversarial perturbations. This paper challenges this prevailing belief and reveals a counter- intuitive finding: semantic communication systems exhibit unan- ticipated adversarial robustness that can exceed that of classical separate source-channel coding systems. On the theoretical front, we establish fundamental bounds on the minimum attack power required to induce a target distortion, overcoming the analytical intractability of highly nonlinear DeepJSCC models by leveraging Lipschitz smoothness. We prove that the implicit regularization from noisy training forces decoder smoothness, a property that inherently provides built-in protection against adversarial attacks. To enable rigorous and fair comparison, we develop two novel attack methodologies that address previously unexplored vulnerabilities: a structure-aware vulnerable set attack that, for the first time, exploits graph-theoretic vulnerabilities in LDPC codes to induce decoding failure with minimal energy, and a progressive gradient ascent attack that leverages the differentiability of DeepJSCC to efficiently find minimum-power perturbations. Designing such attacks is challenging, as classical systems lack gradient information while semantic systems require navigating high-dimensional, non-convex spaces; our methods fill these critical gaps in the literature. Extensive experiments demonstrate that semantic communication requires up to 14- 16× more attack power to achieve the same distortion as classical systems, empirically substantiating its superior robustness.

• To enable rigorous and fair comparison, we develop two novel attack methodologies that address previously unexplored vulnerabilities: a structure-aware vulnerable set attack that, for the first time, exploits graph-theoretic vulnerabilities in LDPC codes to induce decoding failure with minimal energy, and a progressive gradient ascent attack that leverages the differentiability of DeepJSCC to efficiently find minimum-power perturbations.
• Extensive experiments demonstrate that semantic communication requires up to 14- 16× more attack power to achieve the same distortion as classical systems, empirically substantiating its superior robustness.
• Enabled by advances in deep learning (DL), deep joint source-channel coding (DeepJSCC) [2], [10] has emerged as the prototypical realization of the semantic communication paradigm, achieving strong rate-distortion performance, ro- bustness to channel noise, and high compression efficiency across diverse communication and perception tasks [11].