Research News

A new approach to streaming technology may significantly improve how users experience virtual reality and augmented reality environments, according to a study from NYU Tandon School of Engineering.

The research — presented in a paper at the 16th ACM Multimedia Systems Conference on April 1, 2025 — describes a method for directly predicting visible content in immersive 3D environments, potentially reducing bandwidth requirements by up to 7-fold while maintaining visual quality.

The technology is being applied in an ongoing NYU Tandon National Science Foundation-funded project to bring point cloud video to dance education, making 3D dance instruction streamable on standard devices with lower bandwidth requirements.

"The fundamental challenge with streaming immersive content has always been the massive amount of data required," explained Yong Liu — professor in the Electrical and Computer Engineering Department (ECE) at NYU Tandon and faculty member at both NYU Tandon's Center for Advanced Technology in Telecommunications (CATT) and NYU WIRELESS — who led the research team. "Traditional video streaming sends everything within a frame. This new approach is more like having your eyes follow you around a room — it only processes what you're actually looking at."

The technology addresses the "Field-of-View (FoV)" challenge for immersive experiences. Current AR/VR applications demand high bandwidth — a point cloud video (which renders 3D scenes as collections of data points in space) consisting of 1 million points per frame requires more than 120 megabits per second, nearly 10 times the bandwidth of standard high-definition video.

Unlike traditional approaches that first predict where a user will look and then calculate what's visible, this new method directly predicts content visibility in the 3D scene. By avoiding this two-step process, the approach reduces error accumulation and improves prediction accuracy.

The system divides 3D space into "cells" and treats each cell as a node in a graph network. It uses transformer-based graph neural networks to capture spatial relationships between neighboring cells, and recurrent neural networks to analyze how visibility patterns evolve over time.

For pre-recorded virtual reality experiences, the system can predict what will be visible for a user 2-5 seconds ahead, a significant improvement over previous systems that could only accurately predict a user’s FoV a fraction of a second ahead.

"What makes this work particularly interesting is the time horizon," said Liu. "Previous systems could only accurately predict what a user would see a fraction of a second ahead. This team has extended that."

The research team's approach reduces prediction errors by up to 50% compared to existing methods for long-term predictions, while maintaining real-time performance of more than 30 frames per second even for point cloud videos with over 1 million points.

For consumers, this could mean more responsive AR/VR experiences with reduced data usage, while developers can create more complex environments without requiring ultra-fast internet connections.

"We're seeing a transition where AR/VR is moving from specialized applications to consumer entertainment and everyday productivity tools," Liu said. "Bandwidth has been a constraint. This research helps address that limitation."

The researchers released their code to support continued development. Their work was supported in part by the US National Science Foundation (NSF) grant 2312839.

In addition to Liu, the paper's authors are Chen Li and Tongyu Zong both NYU Tandon Ph.D candidates in Electrical Engineering; Yueyu Hu, NYU Tandon Ph.D. candidate in Electrical and Electronics Engineering; and Yao Wang, NYU Tandon professor who sits in ECE, the Biomedical Engineering Department, CATT and NYU WIRELESS.

C. Li, T. Zong, Y. Hu, Y. Wang, Y. Liu. 2025. Spatial Visibility and Temporal Dynamics: Rethinking Field of View Prediction in Adaptive Point Cloud Video Streaming. In Proceedings of the 16th ACM Multimedia Systems Conference (MMSys '25). Association for Computing Machinery, New York, NY, USA, 24–34.

Large Language Models (LLMs) have rapidly become an integral part of our digital landscape, powering everything from chatbots to code generators. However, as these AI systems increasingly rely on proprietary, cloud-hosted models, concerns over user privacy and data security have escalated. How can we harness the power of AI without exposing sensitive data?

A recent study, Entropy-Guided Attention for Private LLMs by Nandan Kumar Jha, a Ph.D. candidate at the NYU Center for Cybersecurity (CCS), and Brandon Reagen, Assistant Professor in the Department of Electrical and Computer Engineering and a member of CCS, introduces a novel approach to making AI more secure. The paper was presented at the Privacy-preserving Artificial Intelligence workshop at the AAAI Workshop on Privacy-Preserving Artificial Intelligence in early March.

The researchers delve into a fundamental, yet often overlooked, property of neural networks: entropy — the measure of information uncertainty within a system. Their work proposes that by understanding entropy’s role in AI architectures, we can improve the privacy, efficiency, and reliability of LLMs.

The Privacy Paradox in AI

When we interact with AI models — whether asking a virtual assistant for medical advice or using AI-powered legal research tools — our input data is typically processed in the cloud. This means user queries, even if encrypted in transit, are ultimately decrypted for processing by the model. This presents a fundamental privacy risk: sensitive data could be exposed, either unintentionally through leaks or maliciously via cyberattacks.

To design efficient private LLMs, researchers must rethink the architecture these models are built on. However, simply removing nonlinearities destabilizes training and disrupts the core functionality of components like the attention mechanism.

“Nonlinearities are the lifeblood of neural networks,” says Jha. “They enable models to learn rich representations and capture complex patterns.”

The field of Private Inference (PI) aims to solve this problem by allowing AI models to operate directly on encrypted data, ensuring that neither the user nor the model provider ever sees the raw input. However, PI comes with significant computational costs. Encryption methods that protect privacy also make computation more complex, leading to higher latency and energy consumption — two major roadblocks to practical deployment.

To tackle these challenges, Jha and Reagen’s research focuses on the nonlinear transformations within AI models. In deep learning, nonlinear functions like activation functions play a crucial role in shaping how models process information. The researchers explore how these nonlinearities affect entropy — specifically, the diversity of information being passed through different layers of a transformer model.

“Our work directly tackles this challenge and takes a fundamentally different approach to privacy,” says Jha. “It removes nonlinear operations while preserving as much of the model’s functionality as possible.”

Using Shannon’s entropy as a quantitative measure, they reveal two key failure modes that occur when nonlinearity is removed:

1. Entropy Collapse (Deep Layers): In the absence of nonlinearity, later layers in the network fail to retain useful information, leading to unstable training.
2. Entropic Overload (Early Layers): Without proper entropy control, earlier layers fail to efficiently utilize the Multi-Head Attention (MHA) mechanism, reducing the model’s ability to capture diverse representations.

This insight is new — it suggests that entropy isn’t just a mathematical abstraction but a key design principle that determines whether a model can function properly.

A New AI Blueprint

Armed with these findings, the researchers propose an entropy-guided attention mechanism that dynamically regulates information flow in transformer models. Their approach consists of Entropy Regularization — a new technique that prevents early layers from being overwhelmed by excessive information — and PI-Friendly Normalization — alternative methods to standard layer normalization that help stabilize training while preserving privacy.

By strategically regulating the entropy of attention distributions, they were able to maintain coherent, trainable behavior even in drastically simplified models, which ensures that attention weights remain meaningful, avoiding degenerate patterns that commonly arise once nonlinearity is removed, where a disproportionate number of heads exhibit extreme behavior — collapsing to near one-hot attention (low entropy) or diffusing attention uniformly (high entropy) — both of which impair the model’s ability to focus and generalize.

This work bridges the gap between information theory and architectural design, establishing entropy dynamics as a principled guide for developing efficient privacy-preserving LLMs. It represents a crucial step toward making privacy-preserving AI more practical and efficient in real-world applications. By bridging the gap between information theory and neural architecture design, their work offers a roadmap for developing AI models that are not only more private but also computationally efficient.

The team has also open-sourced their implementation, inviting researchers and developers to experiment with their entropy-guided approach.

arXiv:2501.03489v2 [cs.LG] 8 Jan 2025

Snap a photo of your meal, and artificial intelligence instantly tells you its calorie count, fat content, and nutritional value — no more food diaries or guesswork.

This futuristic scenario is now much closer to reality, thanks to an AI system developed by NYU Tandon School of Engineering researchers that promises a new tool for the millions of people who want to manage their weight, diabetes and other diet-related health conditions.

The technology, detailed in a paper presented at the 6th IEEE International Conference on Mobile Computing and Sustainable Informatics, uses advanced deep-learning algorithms to recognize food items in images and calculate their nutritional content, including calories, protein, carbohydrates and fat.

For over a decade, NYU's Fire Research Group, which includes the paper's lead author Prabodh Panindre and co-author Sunil Kumar, has studied critical firefighter health and operational challenges. Several research studies show that 73-88% of career and 76-87% of volunteer firefighters are overweight or obese, facing increased cardiovascular and other health risks that threaten operational readiness. These findings directly motivated the development of their AI-powered food-tracking system.

"Traditional methods of tracking food intake rely heavily on self-reporting, which is notoriously unreliable," said Panindre, Associate Research Professor of NYU Tandon School of Engineering’s Department of Mechanical Engineering. "Our system removes human error from the equation."

Despite the apparent simplicity of the concept, developing reliable food recognition AI has stumped researchers for years. Previous attempts struggled with three fundamental challenges that the NYU Tandon team appears to have overcome.

"The sheer visual diversity of food is staggering," said Kumar, Professor of Mechanical Engineering at NYU Abu Dhabi and Global Network Professor of Mechanical Engineering at NYU Tandon. "Unlike manufactured objects with standardized appearances, the same dish can look dramatically different based on who prepared it. A burger from one restaurant bears little resemblance to one from another place, and homemade versions add another layer of complexity."

Earlier systems also faltered when estimating portion sizes — a crucial factor in nutritional calculations. The NYU team's advance is their volumetric computation function, which uses advanced image processing to measure the exact area each food occupies on a plate.

The system correlates the area occupied by each food item with density and macronutrient data to convert 2D images into nutritional assessments. This integration of volumetric computations with the AI model enables precise analysis without manual input, solving a longstanding challenge in automated dietary tracking.

The third major hurdle has been computational efficiency. Previous models required too much processing power to be practical for real-time use, often necessitating cloud processing that introduced delays and privacy concerns.

The researchers used a powerful image-recognition technology called YOLOv8 with ONNX Runtime (a tool that helps AI programs run more efficiently) to build a food-identification program that runs on a website instead of as a downloadable app, allowing people to simply visit it using their phone's web browser to analyze meals and track their diet.

When tested on a pizza slice, the system calculated 317 calories, 10 grams of protein, 40 grams of carbohydrates, and 13 grams of fat — nutritional values that closely matched reference standards. It performed similarly well when analyzing more complex dishes such as idli sambhar, a South Indian specialty featuring steamed rice cakes with lentil stew, for which it calculated 221 calories, 7 grams of protein, 46 grams of carbohydrates and just 1 gram of fat.

"One of our goals was to ensure the system works across diverse cuisines and food presentations," said Panindre. "We wanted it to be as accurate with a hot dog — 280 calories according to our system — as it is with baklava, a Middle Eastern pastry that our system identifies as having 310 calories and 18 grams of fat."

The researchers solved data challenges by combining similar food categories, removing food types with too few examples, and giving extra emphasis to certain foods during training. These techniques helped refine their training dataset from countless initial images to a more balanced set of 95,000 instances across 214 food categories.

The technical performance metrics are impressive: the system achieved a mean Average Precision (mAP) score of 0.7941 at an Intersection over Union (IoU) threshold of 0.5. For non-specialists, this means the AI can accurately locate and identify food items approximately 80% of the time, even when they overlap or are partially obscured.

The system has been deployed as a web application that works on mobile devices, making it potentially accessible to anyone with a smartphone. The researchers describe their current system as a "proof-of-concept" that could be refined and expanded for broader healthcare applications very soon.

In addition to Panindre and Kumar, the paper's authors are Praneeth Kumar Thummalapalli and Tanmay Mandal, both master’s degree students in NYU Tandon’s Department of Computer Science and Engineering.

Researchers at New York University are developing a novel cell therapy that could offer a longer-lasting, potentially more effective treatment for Alzheimer’s disease by clearing toxic proteins from the brain.

Instead of requiring repeated antibody infusions, which can be costly and cause inflammation, this new approach aims to use engineered immune cells to target and remove amyloid plaques — one of the hallmarks of Alzheimer’s disease.

The project has been awarded a $4.2 million grant from the National Institutes of Health’s National Institute on Aging to fund research over the next five years.

The multiple-Principal Investigator (MPI) research team is led by contact MPI Martin Sadowski, Professor of Neurology, Psychiatry, and Biochemistry and Molecular Pharmacology at NYU Grossman School of Medicine. He is joined by MPIs Paul M. Mathews, Research Associate Professor in the Department of Psychiatry at NYU Grossman School of Medicine, and David M. Truong, Assistant Professor of Biomedical Engineering and Pathology at NYU Tandon School of Engineering.

Truong’s lab is playing a key role in the genetic engineering of immune cells for the therapy, building on his expertise in stem cell engineering and synthetic biology. His team is working on designing “off-the-shelf” immune cells—cells that do not need to be taken from the patient but can instead be manufactured and prepared in advance.

Truong described the motivation behind the project as deeply personal. “I have Alzheimer's in my family, and I wanted to use my expertise to help introduce an innovative therapy that could really change the way we treat the disease,” he said.

The team is developing a type of engineered macrophage, a kind of immune cell that can identify and remove harmful proteins in the brain. These cells will be created from human induced pluripotent stem cells, a renewable source of cells that can be genetically modified in the lab.

The engineered cells will be designed to target and bind to amyloid plaques for removal, optimize brain access by reducing competition from the brain’s own immune cells, and include built-in safety mechanisms to deactivate the therapy if necessary.

Unlike many other experimental Alzheimer’s treatments, this approach does not require injecting the cells directly into the brain. Instead, they will be delivered through the bloodstream, where they can cross the blood-brain barrier and begin clearing harmful proteins, avoiding invasive procedures while ensuring effective treatment.

To further enhance safety, the therapy includes a built-in “kill switch” that allows doctors to deactivate the cells if needed. If unintended side effects occur, a specific drug can be administered to eliminate them, ensuring the treatment remains both controlled and adaptable.

Once Truong’s lab finalizes the engineering of these human cells, they will be handed off to Sadowski’s team for testing in Alzheimer’s disease models. The Nathan S. Kline Institute for Psychiatric Research will assist in evaluating how well the cells remove amyloid plaques, while NYU Grossman School of Medicine will analyze how the cells behave in the brain.

The therapy is an adaptation of chimeric antigen receptor (CAR) technology, which has been revolutionary in cancer treatment. While CAR-T cell therapies have been used to fight blood cancers, this research aims to adapt similar technology to neurodegenerative diseases like Alzheimer’s.

Truong noted that cell therapy is a rapidly evolving field and that while CAR-T therapy has primarily been used against cancer, this project is pushing the boundaries to see if such an approach could work for Alzheimer’s, a disease that affects millions and has few effective treatment options.

The five-year grant follows an R61/R33 funding model, meaning that the first two years are dedicated to proving the feasibility of the therapy. If the team meets its key scientific milestones, funding will continue for three additional years to move the research toward clinical readiness.