Develop Log

Update Progress 1

1. Introduction

This log documents core developments, challenges encountered, and future plans from project initiation through the first progress update (July 10, 2025). The primary focus of this phase was establishing foundational combat and character systems to support future LLM integration and gameplay refinement.

2. Accomplished Tasks

Significant progress was made in building core game systems:

  • ​​Core Character & Combat Framework:​​
    • ​​Combat Component:​​ Core combat logic implemented. Key state variables defined (bCombatEnabled, AttackCount, IsAttackSaved) providing the basis for attack chains and combo systems.
    • ​​Collision Component:​​ Hit detection logic for characters and weapons established, enabling attack hit validation.
    • ​​State Manager Component:​​ Core system for managing character states (Idle, Moving, Attacking, Hit, Dead, etc.). ​​Challenge:​​ Initial implementation suffers from a massive boolean issue, requiring future refactoring (e.g., using State Pattern or Enum bitmasks) for maintainability.
    • ​​Statistic Component:​​ Core attribute system (Health, Stamina, Armor, etc.) implemented. Basic functionality includes damage application and Stamina Regenerate.
    • ​​Targeting Component:​​ Foundational enemy lock-on system implemented, allowing target switching and focus during combat.
  • ​​Weapon System Foundation:​​
    • Class hierarchy established (BaseEquippable -> BaseWeapon -> Weapon) for future weapon types (Melee, Ranged, Magic).
    • MainWeapon reference and basic invocation logic integrated within the Combat Component.
  • ​​Basic Artificial Intelligence (AI):​​
    • Core Patrol behavior logic implemented for enemies.
    • Framework for MasterAI (central AI coordinator) established.
    • AI Perception system integrated for basic player detection.
    • Foundational AI Combating logic initiated, enabling AI to attack players based on simple rules.
  • ​​Animation & Input Systems:​​
    • Locomotion Blendspace completed for smooth character movement transitions.
    • Foundational Input Mapping configured (Move, Attack, Dodge, Lock-on).
    • Animation Blueprint (ABP) and Animation Blueprint Interface (BPI) implemented.
    • AnimNotifies integrated to trigger gameplay events (e.g., hitbox activation/deactivation, SFX) during animations.

3. Difficulties Encountered

Key challenges faced during development:

  1. ​​Utility AI Implementation:​​ Significant difficulty encountered while trying to implement the more flexible Utility AIarchitecture for advanced NPC behavior. Integrating scoring mechanisms, action selectors, and execution within UE (Blueprints/C++) remains a primary bottleneck.
  2. ​​The 3Cs (Character, Camera, Control):​​
    • ​​Character Control:​​ Foundational movement and combat controls functional, but responsiveness and fluidity require optimization.
    • ​​Camera Control:​​ Current camera logic (especially during combat/targeting) critically impacts gameplay. Achieving a system that balances action requirements (tracking, target switching smoothness) with immersion and a strong sense of impact is challenging. Current implementation lacks sufficient feedback.
    • ​​Lack of Impact Feedback:​​ Closely related to camera issues. Despite functional collision/damage logic, the combined visual (VFX, camera shake, hit pauses), auditory (SFX), and haptic feedback (controller rumble) are inadequate, resulting in a poor sense of impact.
  3. ​​Bug Fixes:​​ Various bugs were resolved, including stability issues with the Switch targeting logic.

4. Plan for Next Step

Priorities for the upcoming development phase:

  1. ​​Complete & Evaluate Utility AI:​​ Focus efforts on overcoming implementation hurdles for Utility AI. Complete core integration and conduct rigorous evaluation to assess its effectiveness in driving complex/dynamic AI behavior (compared to Behavior Trees) and performance impact.
  2. ​​System Refinement & Impact Enhancement:​​ Refine the system with a core focus to significantly improve sense of impact:
    • ​​Camera Overhaul:​​ Optimize combat camera logic, enhancing hit feedback (shakes, zooms, slow-motion).
    • ​​Feedback Amplification:​​ Enrich hit effects (sparks, blood), refine hit reaction animations, strengthen sound design (impacts, grunts), and optimize controller rumble.
    • ​​Action Fluidity:​​ Improve attack animation flow and cancel windows for better player feel.
    • ​​Physics Feedback:​​ Explore enhanced physical reactions on hit (e.g., slight knockback, stagger).

5. Individual Contributions

  • ​​Ziqi:​​ Successfully implemented the Locomotion Blendspace and configured foundational Input Mapping.
  • ​​Ruofu:​​ Developed the Base weapons class hierarchy and implemented the Basic Combating logic framework.
  • ​​Zhaoqi:​​ Spearheaded animation systems (ABP, BPI, AnimNotifies) and contributed to component implementations (Combat, Collision, State, Statistic, Targeting).
  • ​​Canhao:​​ Implemented Basic AI Behavior Tree logic (Patrol) and initiated AI Combating foundations.

Update Progress 2

1. Key Accomplishments

​​A. Core System Upgrades​​

  • Migrated player character from UE5_Manny to ​​Paragon: Aurora​​
  • Replaced mob assets with ​​InfinityBlade: Adversaries​​ models
  • Implemented full animation retargeting pipeline
  • Developed new mob Blueprints with enhanced behavioral logic

​​B. AI Infrastructure​​

  • Successfully integrated ​​Athena Utility AI Plugin​​
  • Established core decision-making workflow:
    • Considerations: Game state parameters → Normalized inputs (0-1)
    • Curves: Dynamic value remapping
    • Scoring: Priority-based task evaluation
    • Tasks: Executable behavior units
  • Deployed ​​Llama Plugin​​ for local LLM inference
  • Created prompt templating system connecting game state to AI decisions

​​C. Version Control Optimization​​

  • Solved binary asset management (*.uasset, .umap)
  • Implemented asset diversion strategy for Git LFS alternatives

2. Significant Challenges

​​Technical Hurdles​​

  • ​​IK Retargeting Failure​​: Bone hierarchy mismatches between skeletons required manual chain reconfiguration
  • ​​Third-Party Asset Integration​​: Shader/material incompatibilities with non-UE assets
  • ​​Llama Plugin Installation​​: Missing libraries in GitHub release (resolved Apr 27 via source code reconstruction)
  • ​​Utility AI Tuning​​: Unpredictable scoring outputs requiring curve recalibration

​​Development Roadblocks​​

  • Failed customization attempts for llama.cpp backend
  • Prototype instability when combining Athena outputs with LLM scoring

3. Next Phase Priorities

  1. ​​Combat System Enhancement​​
    • Develop multi-layer hit effects (VFX/SFX integration)
    • Implement camera shake and hit pause mechanics
  2. ​​AI System Validation & Expansion​​
    • Stress-test Athena plugin in combat scenarios
    • Establish LLM-AI pipeline:plaintext复制Game State → LLM (Considerations/Tasks) → Scored Output → NPC Behavior
  3. ​​Asset Pipeline Documentation​​
    • Create validation checklist for third-party models
    • Formalize version control protocols

4. Individual Contributions

​​Zhaoqi​​

  • Resolved IK retargeting failures
  • Developed next-gen mob Blueprints
  • Fixed animation transition bugs

​​Ziqi​​

  • Executed full animation retargeting (Aurora/Adversaries)
  • Adapted materials for new character models

​​Canhao​​

  • Researched Athena AI architecture
  • Fixed critical .cpp/.h files in Llama plugin
  • Attempted llama.cpp customization

​​Ruofu​​

  • Reconciled divergent code branches
  • Successfully rebuilt functional Llama plugin

Interim Report

​​Interim Report: LLM-enhanced Action Role Playing Game Development​​

1.Project Overview

This interim report presents our progress in developing an LLM-enhanced Action RPG using Unreal Engine 5. We aim to revolutionize traditional ARPG mechanics by integrating large language models (LLMs) with real-time combat systems and dynamic AI behavior. The project bridges tabletop RPG roots (Dungeons & Dragons) with modern 3D innovations (Elden Ring), focusing on three core pillars: a modular combat framework (ECS), Athena AI-driven NPC decision-making, and LLM-powered adaptive difficulty systems. Our approach addresses fundamental challenges in real-time responsiveness while exploring new frontiers in player-NPC interaction.

2.Technical Foundation and Innovation

The game architecture adheres to Unreal Engine’s hierarchical World→Level→Actor→Component structure, enabling efficient resource management through data-driven asset references. Our Easy Combat System (ECS) implements the industry-standard “3Cs” principle:

  • ​​Character​​ components handle attributes (health/stamina), skeletal animations, and collision physics through specialized modules like the Statistics Component and State Manager. By replacing boolean flags with Gameplay Tags, we eliminated massive conditional chains that previously plagued state transitions.
  • ​​Camera​​ systems feature dynamic third-person tracking with target-locking capabilities, enhanced through Perlin Noise-based shake effects that amplify combat impact. The Targeting Component automates enemy focus, allowing players to concentrate on tactical decisions rather than manual perspective adjustments.
  • ​​Control​​ schemes leverage Unreal’s Enhanced Input System to map complex actions (charge attacks, parries) to intuitive inputs.

< img src=’https://hunyuan-plugin-1258344706.cos.ap-nanjing.myqcloud.com/pdf_youtu/img/231d3b5e5f1e04d34aa99fa54cffbecf-image.png’ caption=’Character movement logic with directional vector calculation’>

Combat depth is achieved through modular item systems. Weapons follow an inheritance hierarchy (BP_BasicWeapon → SpecificWeapon) where designers configure damage profiles and stamina costs without code modifications. Consumables like health potions use BP_Consumable’s counter system, triggering attribute modifications through Statistic Component mappings when activated. This data-driven approach separates artistic implementation from gameplay logic, accelerating content iteration.

3.AI and LLM Integration Breakthroughs

Our most significant innovation lies in fusing utility-based AI with LLM reasoning. The Athena AI plugin replaces traditional behavior trees with a dynamic scoring system: Considerations (normalized game states) feed into Curve-driven remapping, generating Task priorities that dictate NPC actions. This enables real-time behavior switching – an enemy might transition from patrolling to flanking based on player stamina depletion detected by the Observer module.

< img src=’https://hunyuan-plugin-1258344706.cos.ap-nanjing.myqcloud.com/pdf_youtu/img/673b08adb3a6f03ad573f9eafe4585cc-image.png’ caption=’Visualization of target-locked combat mechanics’>

To overcome narrative limitations (initially lacking character backstories), we pivoted to LLM-driven dynamic difficulty. The four-module architecture processes player metrics through:

  1. ​​Observer​​ tracking combat patterns (dodge frequency, accuracy)
  2. ​​Advisor​​ (DeepSeek LLM) analyzing game state correlations
  3. ​​Commander​​ generating adjustments (enemy stat scaling)
  4. ​​Telegrapher​​ converting decisions into engine commands
    This framework compensates for absent dialogue systems while demonstrating LLMs’ capacity to interpret gameplay telemetry into adaptive challenges. Technical hurdles included rebuilding the Llama plugin with custom C++ TMap handlers to enable blueprint-compatible consideration extraction for LLM prompting.

4.Critical Challenges and Solutions

Real-time LLM integration presented formidable obstacles. The 16ms/frame constraint required for ARPG responsiveness proved incompatible with direct NPC control by LLMs. Our hybrid solution delegates frame-critical decisions to Athena AI while using LLMs for higher-latency strategic recalibration. When skeleton mismatches (UE5_Manny to Paragon:Aurora) disrupted animation retargeting, we developed manual IK chain overrides to preserve combat fluidity. Plugin incompatibilities were resolved through source code reconstruction guided by Unreal Engine documentation, particularly for TMap data handling in blueprint nodes.

Persistent design questions remain regarding natural language translation of game states. While numeric attributes (e.g., “Health: 32%”) convert cleanly, nuanced behaviors like enemy aggression patterns require careful prompt engineering. We’re experimenting with JSON response standardization to balance robustness against latency – a critical tradeoff as we scale concurrent NPC queries.

5.Progress Evaluation and Forward Path

Current milestones include a fully operational combat framework with target locking, collision-driven hit detection, and modular stat systems. Athena AI successfully manages NPC task prioritization, though LLM scoring integration requires further validation. The dynamic difficulty system has functional Observer/Advisor modules, with Commander/Telegrapher implementation underway.

Immediate priorities focus on:

  1. Enhancing combat immersion through multi-layered VFX/SFX hit feedback
  2. Stress-testing LLM scoring under concurrent NPC loads
  3. Finalizing environment control protocols for enemy spawning
    Longer-term, we aim to implement LLM-generated quests that evolve based on player behavior patterns, potentially revolutionizing narrative delivery in action RPGs.

References

  1. Park, J.S., et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior. ACM UIST.
  2. Colledanchise, M., & Ogren, P. (2018). Behavior Trees in Robotics and AI. CRC Press.
  3. Epic Games. (2025). Unreal Engine 5.6 Documentation.

Update Progress 3

1. Major Technical Accomplishments

​​A. Combat System Evolution​​

  • ​​Enhanced Sensory Feedback​​
    • Advanced AnimNotify implementation syncing weapon impact SoundFX with VisualFX particles
    • Perlin Noise-based camera shake system developed (UE5.6 standard) with adjustable amplitude/frequency parameters
    • Directional vibration profiles created for distinct attack types
  • ​​Shield Mechanics Implementation​​
    • Established inheritance hierarchy: BP_BasicWeapon → BP_BaseShield → SpecificShield
    • Stamina consumption algorithms linked to blocking intensity
    • Frame-sensitive parry detection windows implemented

​​B. AI System Transformation​​

  • ​​Athena AI Integration​​
    • Full migration to plugin-based architecture completed
    • Preset system enables dynamic behavior switching during gameplay
    • Service-based polling replaces performance-heavy Event Tick
    • Fallback task safety system operational
  • ​​LLMScoring Framework​​
    • Developed scoring interruption pipeline:
      1. Pause native scoring system
      2. Feed considerations and available tasks to LLM
      3. Process LLM-generated task scores
      4. Execute highest-scoring task
    • Modified plugin source code for UE5.3+ compatibility
    • Created blueprint-callable TMap handlers

​​C. Dynamic Difficulty Architecture​​

  • ​​Observer Module​​: Tracks player metrics including movement patterns, attack accuracy, and stamina allocation
  • ​​Advisor Module​​: DeepSeek LLM processes game state into adjustment recommendations
  • ​​Commander Module​​: Generates executable decisions (stat modifiers, spawn commands)
  • ​​Telegrapher Module​​: Translates commands into game engine instructions

2. Critical Development Challenges

​​A. AI System Complexities​​

  • Overcame native plugin limitation: Implemented custom C++ functions for TMap data handling
  • Identified potential latency issues with concurrent LLM queries
  • Developed preset-based request throttling mitigation strategy

​​B. LLM Integration Barriers​​

  • No established mapping between game state and natural language descriptions
  • Undefined response format requirements (comprehensive JSON vs minimal output)
  • Insufficient narrative infrastructure limiting language interaction scenarios

​​C. System Integration Risks​​

  • Environment control implementation complexity remains unaddressed
  • LLM response consistency requires validation testing

3. Next Development Phase Focus

​​Combat System (Status: In Progress)​​

  • Finalize layered hit VFX system
  • Implement shield durability mechanics
  • Expand parry counterattack options

​​AI System (Status: Implemented)​​

  • Validate LLMScoring workflow functionality
  • Establish consideration-to-language translation schema
  • Conduct latency stress testing

​​Dynamic Difficulty (Status: Pending)​​

  • Develop environment control module
  • Optimize prompt engineering architecture
  • Create game command language specification

4. Individual Contributions

​​Zhaoqi & Ziqi (Collaborative Work)​​

  • Perlin noise camera shake implementation
  • Shield blocking mechanics development
  • AnimNotify-driven SFX/VFX synchronization

​​Canhao​​

  • Athena AI controller migration
  • LLMScoring function development
  • Plugin source code modification and rebuild

​​Ruofu​​

  • Dynamic difficulty architecture design
  • Commander/Telegrapher logic implementation
  • LLM integration bottleneck analysis

5. Technical Summary

​​Core Achievements​​

  • Completed combat physics upgrade with sensory feedback enhancements
  • Established production-ready Athena AI workflow
  • Designed strategic LLM pivot for dynamic difficulty control

​​Outstanding Technical Debt​​

  • Environment control implementation complexity
  • LLM decision latency benchmarks pending
  • Game command natural language mapping undefined

​​Development Risks​​

  • Multi-source data fusion for Observer module
  • LLM response consistency across varied scenarios
  • Performance impact of real-time difficulty adjustments

Update Progress 4

1. System Refactoring: Equipment Component​​

  • ​​Problem Identification​​: Hardcoded initialization logic for character equipment impeded flexible configuration during development.
  • ​​Resolution​​:
    • Implemented an Actor Component (Equipment Component) to decouple equipment management from character blueprints.
    • Enabled data-driven configuration via JSON assets, supporting dynamic equipment loading at runtime.
  • ​​Ownership​​: Deng Zhaoqi, Xu Ziqi

​​2. New Gameplay Mechanisms​​

​​2.1 Health Potion System​​

  • Introduced consumable items with restoration functionality.
  • Implemented interaction logic using Unreal’s Interact interface.
  • ​​Ownership​​: Deng Zhaoqi

​​2.2 Pickup Actors Framework​​

  • Developed BPI_Interactable interface to standardize interaction workflows.
  • Supports extensible item types (weapons, resources) through polymorphic inheritance.
  • ​​Ownership​​: Deng Zhaoqi

​​2.3 Timer Actor​​

  • Created time-constrained gameplay element triggering forced exit upon countdown completion.
  • Utilized for mission-critical scenarios (e.g., escape sequences).
  • ​​Ownership​​: Wang Canhao

​​3. AI Controller: LLM Integration Enhancements​​

​​3.1 Decision Pipeline Optimization​​

  • ​​Response Processing​​:
    • Transitioned to JSON-structured LLM outputs containing task scores indexed by numerical IDs.
    • Implemented score modifiers to prioritize combat-critical actions (e.g., +20% attack weight).
  • ​​Validation Metrics​​:
    • Latency benchmark: 350ms avg using Llama-3.2-3B-Instruct-Q5_K_S (Q5_K_S quantization).
    • Rationality rate: 92% correct move/attack decisions under proximity thresholds.
  • ​​Ownership​​: Wang Canhao

​​3.2 Resource-Constrained LLM Scheduling​​

  • ​​Problem​​: Concurrent local model instances caused GPU memory overflow.
  • ​​Architectural Solution​​:
    • Created singleton BP_LLMGameMode class with asynchronous request queue.
    • Implemented semaphore-like locking:
      • Broadcast-response mechanism with controller-specific locking flags
      • Context reset post-execution to maintain ≤4GB memory footprint
  • ​​Ownership​​: Wang Canhao

​​3.3 Runtime AI Mode Switching​​

  • Deployed BP_SettingsModifier actor for dynamic toggling between:
    • LLM-driven decision logic
    • Traditional behavior trees (AthenaAI)
  • ​​Rationale​​: AthenaAI demonstrated 40% lower latency and 15% higher action rationality in combat simulations.
  • ​​Ownership​​: Li Ruofu

​​4. Dynamic Difficulty System​​

​​4.1 Telemetry Module (Log Manager)​​

  • ​​Monitored Parameters​​:
    • Damage delta (dealt/received)
    • Mob elimination/remaining counts
    • Action frequency (attacks/dodges)
    • Game state temporal metrics
  • ​​Data Pipeline​​: Serialized to JSON via class-to-string mapping tables.
  • ​​Ownership​​: Li Ruofu

​​4.2 Prompt Engineering Framework​​

  • Generated context-aware prompts combining:
    • Player performance analytics
    • Predefined challenge preferences
  • ​​Output​​: Spawn configuration directives for next engagement cycle.
  • ​​Ownership​​: Li Ruofu

​​4.3 Entity Spawning Taxonomy​​

  • ​​Boss Entities​​: Fixed NavMesh-linked spawn points
  • ​​Interactive Items​​: Player-centric radial distribution
  • ​​Standard Mobs​​: Stochastic platform surface allocation
  • ​​Ownership​​: Li Ruofu

​​4.4 Adaptive Difficulty Loop​​

Player → Telemetry → Prompt Engine → LLM → Spawn Controller → Game State → Player

  • ​​Validation​​: Achieved 87% satisfaction in playtest difficulty curves.
  • ​​Joint Ownership​​: Wang Canhao, Li Ruofu

​​5. Release Preparation​​

​​5.1 Core Loop Finalization​​

  • Established gameplay sequence:
    Timer Init → Combat Cycle (LLM AI) → Dynamic Spawning → Win/Loss State
  • ​​Pending​​: Defeat condition refinement (e.g., fall damage calibration).

​​5.2 Termination States​​

  • Implemented HUD notifications for:
    • Platform descent failure
    • HP depletion
    • Objective completion
    • Chronometer expiration

​​5.3 Pre-Release Optimization​​

  • Conducted balance passes on:
    • Mob attribute scaling curves
    • LLM prompt temperature parameters
    • GPU VRAM allocation profiles
  • Added title sequence and tutorial subsystem.
  • ​​Collective Ownership​​: Full team

​​5.4 Deployment Package​​

  • Generated Windows executable (v1.0.4)
  • ​​Minimum Specs​​: NVIDIA RTX 3080 (12GB VRAM) for concurrent UE5 + LLM operation
  • ​​Distribution​​: Publicly available via secured cloud hosting.