Tencent Hunyuan Team Answers Developer Questions Late at Night: Next-Generation Model Set for Major Expansion, Smaller Models Already on Roadmap

Deep News
Nov 21

In the early hours of November 21, Yao Shunyu, Chief AI Scientist at Tencent (00700) and head of the Hunyuan large model team, along with researchers from the Hunyuan team, held an AMA (Ask Me Anything) session on Reddit. This marked the first time the Hunyuan team publicly answered questions in a developer community.

During the two-hour concentrated Q&A session, the Hunyuan team received over 300 comments covering model technical roadmaps, architecture trade-offs, future plans, safety evaluations, and more. Throughout the discussion, Yao Shunyu and the Hunyuan team disclosed considerable information, such as: "Hy4-mini (a smaller-sized model) is already on our roadmap, so stay tuned." They also stated, "We will consider making a smaller image model, and stay tuned for the video model as well."

Regarding the fact that companies like ByteDance and Kimi are pushing ultra-large-scale models with parameter counts reaching 3T or even 5T, while they remain at 700B, the team responded: "Hy3 and Hy4 have already shown that solidly built smaller models can often outperform larger ones. More importantly, they cost less, and you can learn more and gain more in the process."

For developers concerned about local deployment, tool calling, and the open-source ecosystem, they provided specific responses and expressed their emphasis on community feedback. On the safety front, the team emphasized advancing responsible open-sourcing through harmful data filtering, training safeguards, and red-team evaluations, while acknowledging the need to continue reducing over-refusals.

The following is a partial transcript of the Q&A organized by topic: (A represents replies from the Tencent Hunyuan team)

Next-Generation Model Set for Major Expansion, Smaller Models Already Under Research

Q: Is there a possibility of releasing Hy4-mini (a smaller-sized model)?

A: Of course! It is already on our roadmap, so stay tuned.

Q: Is it possible to release a 24–31B dense model?

A: It is possible, and we are considering it.

Q: Thank you for open-sourcing the model weights! Is Tencent considering releasing a "Flash Lite" model, such as 35B-A3B? Such models could be very useful, for example for customer-facing chatbots, document parsing, classification, simple automation, and other tasks that benefit from low overhead and high speed.

A: Yes, stay tuned for a frontier Flash-Lite model.

Q: Are there plans to release a new open-source image model? HunyuanImage-3.0 is great, but it is too large for most configurations. I hope you first make a smaller 5–12B model for local use, then a larger model, because we need more open-source image models. I would also like to see a new generation of HunyuanVideo.

A: We will consider making a smaller image model; stay tuned for the video model as well.

Q: Do you think these models have reached their capacity limits, meaning they have already "absorbed" all the intelligence they can hold? Will future smarter Hy models require more parameters, or can you deliver stronger models at the same scale, for example through further post-training of Hy4?

A: I think there is still great potential.

A: We believe Hy4 still has significant room for improvement, especially in post-training. We are investing heavily in large-scale reinforcement learning while continuously improving user experience and model safety. Stay tuned for future updates! We also plan to train larger models. We believe in the scaling law and will continue to invest!

Q: It seems that companies like ByteDance and Kimi are pushing ultra-large-scale models with parameter counts reaching 3T or even 5T, while you remain at 700B. Why is that? How do you view the actual impact of parameter count on intelligence? Do you have any plans in this regard?

A: Hy3 and Hy4 have already shown that solidly built smaller models can often outperform larger ones. More importantly, they cost less, and you can learn more and gain more in the process.

Q: By Hy 4.5, will you expand the parameter count to 1.5 trillion to 2 trillion, correct?

A: The scale of the next-generation Hy model will certainly expand significantly further.

Q: Is it possible to release a recurrent MoE model?

A: Of course it is possible. Please keep following.

Q: How is progress on continual learning?

A: This is a very promising direction, and we are actively researching it.

Q: If Hy4 were allowed to run autonomously, do you think it could currently develop the next-generation model through recursive self-improvement (RSI)? Also, do you have benchmarks to measure this capability that go beyond evaluating autonomy in terms of "autonomously completing a task for n hours and calling tools m times"?

A: Our RSI capability is still at an early stage. We have developed several internal benchmarks to evaluate key RSI capabilities. Our current priority is to build a solid foundation, improve model intelligence, and work closely with products to deliver a better user experience.

Q: With Hy4 and Hyra 2.0, you have already been able to improve training infrastructure and train better small models. What experiments will you do next to prove that the models trained in this round will be better at model research and development in the next round? How can models become better at self-learning?

A: We are developing Hyra 3.0, so stay tuned!

Q: Do you intend to make reinforcement learning a focus of scientific research in the future, especially for medium-scale models like Hy4 Flash at the 500B parameter scale? Also, do you think Chinese labs can challenge Claude's dominance in the field of biology?

A: We actively invest in reinforcement learning technologies for scientific research, and we also place great emphasis on safety issues.

"Overly Sparse MoE Architectures Are Not Friendly to Inference"

Q: Besides the design that Hy4 ultimately adopted, what was the most competitive architecture option the team seriously considered but ultimately abandoned? Why was it abandoned?

A: We weighed GQA and MLA for a long time. MLA has significantly higher computational intensity than GQA, which makes compute capability the bottleneck during speculative decoding on inference GPUs. However, under the same KV cache size, MLA offers better model quality than GQA. Fortunately, after introducing DSA, we reduced MLA's computational intensity during the decoding stage. This ultimately changed the trade-off and led us to adopt MLA.

Q: Will you consider other attention mechanisms in the future, such as QSA?

A: Of course! We are open to various architectural innovations.

Q: Excluding Engram parameters, Hy4 flash is 156B-A12B (156 billion total parameters, 12 billion activated parameters, 7.7% activation ratio). Qwen3.8-Flash is 125B-A6B (4.8% activation ratio); DSv4.1 is 552B-A16B (2.9% activation ratio), dropping to A8B (1.45%) during the prefill stage. How important is higher activation density to intelligence? Have you tested how much model quality would degrade with 156B-A8B or 156B-A5B?

A: We conducted scaling ladder experiments with different MoE sparsity levels, similar to the experiments in the MAI-Thinking-1 paper, and reached roughly the same conclusion: with a fixed number of activated parameters, increasing sparsity improves computational efficiency. So this direction does indeed work. But 156B-A8B or A5B is a fundamentally different trade-off: it keeps the total parameter count unchanged while reducing activated parameters. I expect this would significantly harm model quality. Activated parameter count essentially determines your compute budget. Increasing sparsity can partially offset the loss through efficiency gains, but not entirely; reducing activated parameters ultimately means less computation and a weaker model.

Q: Along this line, I have long hoped to see a lab truly push sparsity to a new level. For example, 200B-A3B would be very interesting, and now Engram can be added as well. Maybe it would perform terribly, maybe it would be great. It would certainly be fast, and I think it would undoubtedly be smarter than a 35B-A3B model. How far can MoE go?

A: My view is that overly sparse MoE architectures are not friendly to inference: they consume a large amount of GPU memory but cannot fully utilize compute capability (FLOPs). At the same inference throughput, we can certainly run a model with the same total parameter count but more activated parameters.

Q: What you said is indeed true for high-parallelism serving in data centers. However, given the current hardware market, the situation for local deployment is very different. Due to lower memory bandwidth, A12B runs slowly on Strix Halo and DGX Spark, while QFN performs reasonably well under the same bandwidth conditions.

A: I understand what you mean. For on-device models, we will adopt a different design approach, enabling tighter co-design between model architecture and GPU hardware on edge devices.

Q: Once a model fabricates command syntax or parameter flags, CLI easily makes mistakes. Could you elaborate on how you address these issues through reinforcement learning while avoiding loss of context?

A: In terms of tool-calling accuracy, we have invested heavily in reinforcement learning data, including adding difficult cases where models are prone to errors. We also use fine-grained reward mechanisms to impose heavier penalties on errors in key details. In terms of context retention, we are continuously expanding the context length used in reinforcement learning training and specifically improving the model's context compression capability.

Q: I really like Shunyu's earliest ReAct work. When training Hy4 and Hy4 flash to perform long-horizon agent/CLI loop tasks, what were the hardest reward hacking or state drift problems to solve in multi-turn reinforcement learning?

A: The hardest is reward hacking caused by vulnerabilities in environment design. For example, preventing the agent from directly searching online for answers instead of truly solving the task.

Q: Welcome! Like Hy4 preview, the official Hy4 version sometimes still takes too long to reason. Does this "overthinking" come from reinforcement learning reward design or the distribution of training data? Can the model learn when to stop?

A: Hy4's goal is to maximize intelligence, so an overly long chain of thought (CoT) is a known issue. However, this also allows us to distill it into a highly cost-effective Hy4 flash. For flash, we use reinforcement learning to ensure a shorter chain of thought.

Q: In the post-training stage, what is the hardest behavior to improve?

A: Overthinking.

Q: During the research and development process, what was the most difficult moment?

A: For us, it was scaling from Hy3 to Hy4. The detailed ablation experiments we did on the data were not as thorough as I had hoped, so there was a lot of uncertainty when we started training. Honestly, I was quite nervous when training first began.

Q: Besides the release results, what made you happiest during the research and development process itself?

A: Fortunately, the training ultimately went unexpectedly smoothly. It felt like pushing all the chips in and actually winning.

Still Committed to Open Source, Hoping Model Output Reads More Like Human Writing

Q: I am glad to see the red-team testing details for Hy4. Usually, stronger safety tuning makes open models refuse basic penetration testing, reverse engineering, or CTF tasks. How do you avoid over-refusing these dual-use programming prompts?

A: In cybersecurity training, we focus on defensive tasks, vulnerability analysis, and remediation, while excluding data primarily involving exploitation and offensive attacks. We also conduct red-team evaluations to assess potential risks. Avoiding over-refusal remains an important area for improvement.

Q: Vulnerability discovery and reverse engineering are not inherently offensive. Incident response teams use reverse engineering for malware analysis, deobfuscation, and forensic investigation to understand how attackers compromised systems. From a defensive perspective, enabling defense teams to discover vulnerabilities before attackers is one of the important ways to reduce the attack surface of critical infrastructure. Doesn't restricting a model's cybersecurity capabilities impose a cost on defenders? Especially since these capabilities are inherently dual-use, restricting one side also severely harms the other.

A: According to our evaluations, Hy4 has relatively strong vulnerability discovery capability, while its exploitation capability is relatively weak.

Q: What monitoring systems have you deployed during training and evaluation?

A: We monitor potential reward hacking and misalignment issues throughout the training and evaluation process, and fix the corresponding environments when problems arise.

Q: During internal operations such as training, evaluation, and deployment, especially for agent tasks, what network control measures do you use? For example, allowlists or blocklists for outbound network access? Also, thank you very much for your detailed answers, and I look forward to carefully reading the safety notes.

A: Yes, we use allowlists and blocklists. During evaluation, because we found that frontier models have many ways to exploit loopholes, we repeatedly revise the environments and these lists.

Q: How much attention do you pay to creative writing and readability? If there were a model that was clearly more readable and easier to understand when explaining things, that would be one of the important reasons for me to switch from Qwen 3.8 Flash Next.

A: We also believe creative writing and readability are very important, and we are making them key improvement areas.

A: A lot of attention, but we still have room for improvement. Sometimes a good example is more helpful than five paragraphs of text.

Q: I have noticed that the writing ability of large language models is degrading. I used to use Claude 3.5 to edit text, but now I no longer use 5.5. I find that the only model that can output barely adequate code documentation is Gemini Pro. Qwen 3.6 also writes better than 3.8. Is this because models are trained on code and have become overfitted, or for some other reason?

A: We have noticed this as well. We believe it may be related to the model's training objectives. We hope the content written by models reads more like human writing, with fewer formulaic expressions. We are improving it, so stay tuned!

Q: Are you considering paying more attention to unverifiable instruction-following capabilities? I have noticed that many Western closed-source models perform better in this regard, such as complying with style constraints.

A: (Pulls out a small notebook and writes furiously.) Noted!

Q: You did not publish results on common benchmarks such as GPQA Diamond, tau bench, IFBench, and SWE Bench. Is there any particular reason?

A: To be frank, we present scores on newer and more difficult benchmarks, such as Agents' Last Exam and ProgramBench. The benchmarks you mentioned were released earlier, and their scores have already saturated.

Q: I saw test results on a primarily PHP code repository, where Hy4 flash handled shell and DevOps tasks very well, such as edge cases in ZFS dataset migration. How do you handle special issues in PHP type inference?

A: Better training data, diverse internal evaluations, and "not deliberately gaming benchmarks," haha.

Q: Besides research papers, do you communicate with other open large language model labs? Or is competition in this field quite intense?

A: We actively learn from the open large language model community and also hope to contribute more of our own research results back to the community.

Q: Finally, I would like to ask the researchers answering questions about some personal feelings: How do you feel about participating in the development of open-weight models?

A: Being able to contribute to the open-source community is very rewarding. Especially seeing so many people paying attention to our work.

Q: If you could make one wish, what would you hope a large language model could do for you in the future? Realistic or wildly imaginative answers are both fine, for example generating an anime companion, or a perfect image of steamed buns. Fun answers like that are completely okay too. Thank you for taking the time to participate in this AMA, and I look forward to using more of your results in the future!

A: As a complete food lover, I hope a large language model can create entirely new recipes for me, and ensure that the taste exactly matches my preferences!

Disclaimer: Investing carries risk. This is not financial advice. The above content should not be regarded as an offer, recommendation, or solicitation on acquiring or disposing of any financial products, any associated discussions, comments, or posts by author or other users should not be considered as such either. It is solely for general information purpose only, which does not consider your own investment objectives, financial situations or needs. TTM assumes no responsibility or warranty for the accuracy and completeness of the information, investors should do their own research and may seek professional advice before investing.

Most Discussed

  1. 1
     
     
     
     
  2. 2
     
     
     
     
  3. 3
     
     
     
     
  4. 4
     
     
     
     
  5. 5
     
     
     
     
  6. 6
     
     
     
     
  7. 7
     
     
     
     
  8. 8
     
     
     
     
  9. 9
     
     
     
     
  10. 10