Hello everyone. Today I will present meta-adaptive context engineering, or Meta-AC for short, which is a new framework designed to optimize AI agents beyond single dimension approaches. We will explore how orchestrating multiple adaptation strategies can overcome the limitations of existing context engineering methods.

Now a little introduction about myself. So I'm Alberto Romero, I'm the co-founder and CEO at jointly. And for context at jointly we build the main specialized agents for regulated industries where policy adherance constraints are particularly strict. Most of our research work is in the area of self-optimizing agent architectures using systematic approaches.

Now about myself, I have spent 20 plus years at the intersection of AI and data. Some of my recent experience includes being the CTO and co-founder of human AI, think ML-based risk prediction for mobility, which was acquired by AON in 2023 and in my previous role I headed up city bank's genai engineering team.

Now here's our agenda for today. We'll begin with the motivation and problems that current systems face. Then we'll review the AC framework and its limitations.

After surveying recent research insights, we'll introduce the Meta-AC approach. We'll discuss its architecture and strategy toolbox, show some results and finish with future directions on challenges.

Now the agentic context engineering framework or AC for short, for which you've got the paper link on the slide there. So it's it's very popular framework and the paper came out a few months ago. Basically organizes a patient into three roles. First of all there's a generator that produces reasoning paths. Then there's a reflector that extracts lessons. And finally, there is a curator that synthesizes these lessons into incremental updates.

AC uses incremental delta updates and a grow and refine mechanism to prevent context collapse and maintain relevance.

Now, most importantly, it can improve without label data by learning directly from execution feedback.

Now so AC has been quite successful and has achieved substantial gains across some of the most popular HM benchmarks like Upworld or finer almost an 11% compared to previous state-of-the-art approaches such as Japa or DC.

Um and it's also achieved an 8.6% um gain on financial reasoning tasks.

Um there are four fundamental limitations um for AC that I'm going to reflect on and um just discuss on the next slide. Um and those form the basis for um for meta AC basically.

Now as I was saying um despite it strength AC has got four critical failure modes. First it is highly dependent on the reflector. Um so when reflection fails the context becomes noisy and even harmful.

Uh secondly there's feedback brittleleness which means that when ground truth signals are weak or absent AC may reinforce incorrect behaviors.

Third, the the task complexity blindness um which leads to treat simple and complex tasks the same which can be a waste of resource uh and also a miss of opportunities um for optimization

and then finally um AC optimizes only the context dimension so ignores compute memory and parameter updates.

Now the 24 and 25 research landscape offers um four key insights in my views. First of all uh verification me mechanisms uh like self evaluation, multimodel consensus and execution checks are really important for robustness of any solution.

Secondly, uh adaptive compute allocation shows that small models can outperform much larger ones by selectively increasing inference steps.

The third one is that structured memory architectures outperform linear context context accumulation by organizing facts as graphs or multi-randular memories.

Then finally, test time training bridges inference and learning uh and enables temporary parameter updates to yield large accuracy gains.

So these advances suggest that we need a hybrid multi-dimensional system.

Now, MetaC um addresses AC's limitation by adding a meta controller that learns to orchestrate multiple adaptation strategies based on a task's complexity, uncertainty, verifiability, and also resource constraints. So instead of applying the same procedure to every problem, Metaac profiles each task and allocates the right combination of strategies across context, compute, verification, memory and parameter dimensions.

Um so this adaptive uh learned coordination is what enables it to outperform single dimension methods.

Now the the meta framework consists of four layers. So getting into the architecture

um the first layer is the task profiling one which assesses complexity uncertainty verifiability and resource budgets.

Then there is a lightweight meta controller that selects and allocates adaptation strategies accordingly.

The next layer down is a strategy execution one and the carries out the reflection, adaptive compute, hierarchical verification, structure memory retrieval and selective uh test time training. And then finally uh there's a feedback aggregation layer that collects the outcomes and updates the meta controllers policy through metalarning.

So this layer design allows the system to learn from its experience and uh continuously refine its decision making.

Now in terms of the task profiling um there are four key dimensions that are being assessed. The first one is uh semantic complexity. So this is basically an embedding based similarity to uh known dash distributions that gets produced.

Uh second one is uncertainty quantification. Uh think of it as a relative softmax uh scoring that predicts model confidence.

The third one is verifiability assessment. So whether we can execute and validate the output.

And then the fourth one is resource availability. So we take into consideration the context window, the compute budget and even other constraints such as time.

So the output of this layer of the task profiling layer is a 32dimensional task embedding which is what fits as input into the meta controller.

Now in terms of the strategy toolbox um meta draws from six strategies. First one is minimal context which uses concise prompts for simple tasks.

Um then we use AC reflection uh which retains the generator reflector curator loop for incremental knowledge accumulation um as established by uh standard AC.

Then we also use adaptive compute which scales the number of reasoning steps or samples based on the task difficulty.

We also use hierarchical verification that combines self-evaluation multimodal consensus and execution checks.

uh adaptive memory uh that retrieves relevant information from structured multi granular memories and then finally we use selective test time training which applies temporary parameter updates such as lower adapters for high stakes tasks.

So the meta controller learns to combine these tools effectively over time.

Now the um reward formula um upon which the the learning strategy is selected accounts for the following components. Um the first one is the correctness of an action or prediction which is accuracy.

Then we also have the penalty associated um with resources used or negative outcomes. So one minus cost and then is the trustworthiness of the models which is self-expressed certainty.

So the confidence calibration basically uh with weighted importance determined by the hyperparameters alpha, beta and gamma.

In terms of the uh metalarning loop um we have four sources of feedback collection. Uh first of all is task outcomes. The success failure or correctness um of the task. Then we've got the strategy performance. So what is the individual contribution of each strategy to the overall performance of the task?

Then we also have efficiency metrics such as the compute, latency, memory. And then finally we've got confidence calibration. So where predictions are accurate.

Um so moving on to um how we go on about uh solving the uh the limitations from AC. The first one was the weak reflector problem. So AC's issue is that there is a a 50 to 60% performance drop when reflector quality degrades. Um with beta AC we introduce um uh three things basically. So first of all is quality gates. Um so it's a learned classifier that blocks harmful deltas and secondly

there's a multi- signal reflector uh or reflection which basically um is an ensemble of specialist models uh when there is a level of uncertainty.

Uh and then the third one is adaptive strategy allocation. So the meta controller learns when reflection fails and then it roots to verification or test time compute instead.

Um so we can expect to maintain an 80% plus performance even when the uh reflector degrades around 30%.

Now the the second um limitation we had was um the feedback quality brittleleness.

So what we observe with AC is that there can be significant degradation without reliable ground truth signals. Uh with beta AC we introduce a hierarchical verification cascade um where we can expect a 50 to 60% reduction in errors from poor feedback and that's through three tiers.

The first tier is self verification which is just fast filter. We just accept if the confidence level is over a certain value. Second tier is a multimodel consensus. So we leverage a diverse range of models such as GBT4, claude and dips and we do confidence weighted voting. And then the tier three is execution based verification uh where we leverage code sandbox APA API validation and schema compliance.

Um the the third um limitation we had was uh task complexity mismatch. Um so in a sense the fact that AC uses uniform processing um also for simple tasks which can be a waste of resource. So meta adapts uh strategy allocation dynamically rather than using the same heavy pipeline for everything. The alphas are allocation weights for the six optimization strategies and they represent how much computational budget is assigned to each strategy for a given task.

So simple tasks um require minimal processing can save n around a 90% uh compute compared to standard AC. moderate tasks um is more of a balanced approach um that include AC plus verification and then complex tasks um basically heavy test time compute multiple attempts and memory retrieval.

Um so just to conclude with some results um and and these are initial results uh we have observed um around an 8 to 11% uh improvement on agent benchmarks.

Um we have also observed a six to eight points improvement on on some domain specific tasks. um also a 30 to 40% reduction in compute costs um through the allocation of um adaptive strategies

um and overall there's um there's more robustness more consistency um and you know we can generalize better we can use the framework across a a diverse range of of domains so the conclusion is that um meta can can orchestrate ates a context compute and verification and memory and parameter adaptation and produce a robust uh self-improvement um framework for agents.

Um future work will implement uh and evaluate the full system across uh a a more diverse range of domains and we'll continue exploring metalarning and this will involve also incorporating um additional strategies as well.

Now I also wanted to touch on um additional applications of meta that I think are quite relevant. Um so first one is um for multimodel AI systems. So for example deciding when to use vision versus uh language processing again can be um a like a a meta adaptive uh strategy decisioning.

Um also when you have uh compound AI systems that um require different models for different stages um and the complexity is um you know is substantial uh we can actually um uh in a in a meta adaptive manner uh select the most effective uh strategies to to resolve a task and to end.

um also um for human collaboration um so in other words to determine when to have a human in the loop and also for continual learning systems um where we are balancing exploration versus exploitation.

Um so the the core takeaway is that optimization requires a meta layer of intelligence and and that has to be trained um and you know um it requires um a lot of trial and error before it can actually um perform at the right level.

In terms of the future direction and challenges um there are still several challenges that remain. So the meta controllers training u may be unstable um due to sparse rewards and that this can be mitigated through curriculum learning. Uh also robust advantage estimation and um regularization of entropy.

Also computational overhead from profiling and multiple uh strategies um needs to be reduced with efficient models. Um we can leverage things like lazy execution, batching and caching.

Um also uh the ver verification uh cascades can be brittle if all models um make the same mistake. So we need diverse models um with confidence waiting and human oversight um as well as active learning. uh metalarning loops require substantial data. Uh synthetic task uh task generation of policy learning uh transfer from related domains and sample efficient algorithms uh can also help as well. And finally uh addressing these ch these challenges um is going to be key to scaling meta and applying it across um a wide range of domains.

So that was all from me. Thank you very much for listening. Um, and yeah, uh, appreciate you being there. Thank you.