Native ALB forwarding bypasses service-request callbacks; supported pre-routing extension hook? (Pro dev230)

[AndreKurait]

Observed behavior

On licensed stock localstack/localstack-pro:dev 2026.11.0.dev230 (build 558bad4c, tested 2026-10-20), a request forwarded through the native ALB gateway route bypasses callbacks appended to localstack.gateway.handlers.serve_custom_service_request_handlers.

The same callback executes for ordinary SDK API requests. A callback appended to handlers.preprocess_request executes for the ALB-forwarded request and its injected header reaches the target. Native ALB forwarding itself succeeds with HTTP 200.

This is an extension compatibility report: integrations doing request-header adaptation in the service-request callback do not see native ALB traffic. Please confirm whether this ordering is intentional and what supported extension hook should be used for processing requests before native ALB routing. If these callbacks are deliberately service-API-only, a documented supported pre-routing hook/migration path would address the compatibility gap rather than changing route ordering.

Environment

• Linux/amd64; licensed stock Pro image; no vendor provider or routing monkeypatches.
• Manifest: sha256:fbe339f5b5d893e5bac43472af717964e09d00844d4aeecbce4b287b02ffb553.
• Image ID: sha256:b8d3cad6b383453d3d61a85bb0884cb903d90a409c672c9503a784d95ebf8097.
• An init hook registers two observers and starts an HTTP echo target. Both callbacks use the same marker-writing logic. Only dummy AWS credentials and isolated local endpoints are used.

Reproduction

The following is a reduced version of the component probe used for the observations below. The complete probe also inspected stores and tested name collisions; those unrelated checks are omitted here. Save the files as hook.py and repro.py in an empty directory. Export your existing LOCALSTACK_AUTH_TOKEN without placing its value in either file.

Run in a disposable Docker network (choose an unused subnet if this one conflicts with your environment):

docker network create --subnet 172.28.241.0/24 alb-callback-repro
docker run -d --name alb-callback-repro --network alb-callback-repro --ip 172.28.241.10   -e LOCALSTACK_AUTH_TOKEN -e EC2_VM_MANAGER=mock   -v "$PWD/hook.py:/etc/localstack/init/ready.d/hook.py:ro"   localstack/localstack-pro@sha256:fbe339f5b5d893e5bac43472af717964e09d00844d4aeecbce4b287b02ffb553
docker exec -i alb-callback-repro /opt/code/localstack/.venv/bin/python - < repro.py
# After inspecting the output:
docker rm -f -v alb-callback-repro
docker network rm alb-callback-repro

hook.py:

import json
import threading
import types
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlsplit
from pathlib import Path
from localstack.gateway import handlers

events=[]

def preprocess(chain, context, response):
    events.append({'stage':'preprocess','path':context.request.path,'host':context.request.headers.get('Host')})
    context.request.headers['X-Audit-Preprocess']='observed'

def service_callback(chain, context, response):
    events.append({'stage':'service','path':context.request.path,'host':context.request.headers.get('Host')})
    context.request.headers['X-Audit-Service-Callback']='observed'

handlers.preprocess_request.append(preprocess)
handlers.serve_custom_service_request_handlers.append(service_callback)

class Probe(BaseHTTPRequestHandler):
    def log_message(self,*args):pass
    def do_GET(self):
        result={'events':list(events)} if self.path=='/state' else {'path':self.path,'headers':dict(self.headers)}
        data=json.dumps(result).encode()
        self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)

server=ThreadingHTTPServer(('0.0.0.0',18080),Probe)
threading.Thread(target=server.serve_forever,daemon=True).start()
print('AUDIT_OBSERVERS_READY',flush=True)

repro.py:

import json
import time
import uuid
import boto3
import requests
from botocore.config import Config

endpoint='http://127.0.0.1:4566'
for attempt in range(60):
    try:
        if requests.get('http://127.0.0.1:18080/ready',timeout=1).status_code==200:break
    except requests.RequestException:pass
    time.sleep(1)
else:raise RuntimeError('In-process observer did not become ready')

name='native-audit-'+uuid.uuid4().hex[:8]

def client(service,account):
    return boto3.client(service,endpoint_url=endpoint,region_name='us-east-1',aws_access_key_id=account,aws_secret_access_key='test',config=Config(proxies={},retries={'max_attempts':0},connect_timeout=3,read_timeout=20))

def make_alb(account):
    ec2=client('ec2',account);elb=client('elbv2',account)
    vpc=ec2.create_vpc(CidrBlock='172.28.240.0/22')['Vpc']['VpcId']
    subnets=[ec2.create_subnet(VpcId=vpc,CidrBlock=cidr,AvailabilityZone=zone)['Subnet']['SubnetId'] for cidr,zone in [('172.28.241.0/25','us-east-1a'),('172.28.241.128/25','us-east-1b')]]
    lb=elb.create_load_balancer(Name=name,Subnets=subnets,Scheme='internal',Type='application')['LoadBalancers'][0]
    return ec2,elb,vpc,subnets,lb

first=make_alb('111111111111')
ec2,elb,vpc,subnets,lb=first
arn=lb['LoadBalancerArn']
tg=elb.create_target_group(Name=name,Protocol='HTTP',Port=18080,VpcId=vpc,TargetType='ip')['TargetGroups'][0]['TargetGroupArn']
elb.register_targets(TargetGroupArn=tg,Targets=[{'Id':'172.28.241.10','Port':18080}])
listener=elb.create_listener(LoadBalancerArn=arn,Protocol='HTTP',Port=80,DefaultActions=[{'Type':'forward','TargetGroupArn':tg}])['Listeners'][0]['ListenerArn']

response=requests.get(endpoint+'/audit-forward',headers={'Host':lb['DNSName']},timeout=15)
state=requests.get('http://127.0.0.1:18080/state',timeout=5).json()
result={'http_status':response.status_code,'forwarded':response.json(),'events':state['events']}
elb.delete_listener(ListenerArn=listener)
elb.delete_load_balancer(LoadBalancerArn=arn)
elb.delete_target_group(TargetGroupArn=tg)
for row in [first]:
    for subnet in row[3]:row[0].delete_subnet(SubnetId=subnet)
    row[0].delete_vpc(VpcId=row[2])
result['cleanup']='All probe AWS resources deleted'
print(json.dumps(result,indent=2,default=str),flush=True)

Actual observations from the complete component probe

{
  "native_alb_forwarding_succeeded": true,
  "preprocess_reached_target": true,
  "service_callback_reached_target": false,
  "service_callback_positive_control": true
}

The target received X-Audit-Preprocess: observed, but no X-Audit-Service-Callback header. The event log contains a preprocessing event for /audit-forward, no service callback event for that path, and service callback events for ordinary SDK requests to /. This rules out the service callback simply failing to register.

Requested behavior / clarification

Please provide or document a supported extension hook that runs before native ALB forwarding, allowing request-header adaptation. If update_request_handlers / the service-request callback is intended to cover this traffic, please investigate the skipped invocation. The preprocessing list works in this build, but it would help to know the supported integration contract before depending on that internal registration point.

Related store migration observation (not a separate AWS API defect)

The same runtime creates native ALBs in localstack.pro.core.services.elbv2.models.elbv2_stores, while the corresponding Moto elbv2_backends registry is empty. A Moto-only registry inspection therefore no longer discovers native ALBs. An integration updated to inspect the Pro store found the unique ALB and still rejected same-name collisions across accounts and against classic ELB.

We recognize that Moto registry internals need not be a supported extension interface. This is included as migration context and a request for a supported discovery interface/documentation, not a claim that the Pro store migration itself is incorrect. The ALB callback ordering can be reproduced independently using the files above.

[alexrashed]

Hi @AndreKurait!

Thanks for bringing this up! LocalStack Extensions are currently in Preview, where our intention is to provide a method to extend LocalStack with additional tools and integrations. The explicit API is localstack.extensions.api (and since extensions are still in preview, this API could change / break).
From your report, I understand that you want to intercept and mark each request received by an emulated service by importing direct internals of the LocalStack Gateway, which is outside of this defined extension API.
However, we would be super interested in your use-case. Could you please reach out to us in our Community Slack? Feel free to ping me or send a DM!