Skip to content

High memory usage when embedding large texts #222

Open

Description

@Barney241

Hi i am experiencing high memory usages which caused my pod to be killed because of exeeding its limits. after some experiments i found its related to text length that i am trying to embed.

when using models such as e5 large or paraphrase base v2 the program starts with about 1.5GB of ram usage which is expected for this model, but after some iterations this process uses 16GB of virtual memory and 6GB of ram. which is a lot

I tried other smaller models as well and its happening for them too but in smaller scale. So my intuiton was that i am just using texts that are too long so i cut all to max 100 chars. which slowed the ram usage increase, but still if not using constant batch of text it kept growing untill the program was eventually killed.

For more context i am building vector api that returns embeddings for texts. so model sessions is active from the start.

To reproduce you can use this array of texts with e5 large model. in default example with infinite loop

[
            "Spain",
            "Spain,Barcelona,Avignon,Maspalomas,Disfrutar,Bohemia Suites & Spa,Mandarin Oriental, Barcelona,Restaurant Milano",
            "",
            "Seattle's best vegan restaurants",
            "Seattle,The Wayward Vegan,Lin Handmade Dumplings & Noodles,The Sushi Samurai (100% Vegan),Harvest Beat,Cafe Flora,Araya's Place,Plum Bistro,Georgetown Liquor Company,Pi Vegan Pizzeria,Cycle Dogs",
            "Must try vegan restaurants when visiting Seattle!",
            "10 must visit breweries in Seattle!",
            "Seattle,Great Notion Brewing - Ballard,Bizarre Brewing,Stoup Brewing - Ballard,Reuben's Brews - The Ballard Taproom,Urban Family Brewing Co.,Fremont Brewing's Urban Beer Garden,Seapine Brewing Company,Obec Brewing,Cloudburst Brewing,Holy Mountain Brewing Company",
            "Seattle is a mecca for incredible beer, here are 10 must visit breweries!",
            "Newport",
            "Newport,Elizabeth Oceanfront Suites,Rogue Bayfront Public House,Local Ocean Seafoods",
            "",
            "Iceland",
            "Iceland,Seljalandsfoss,Skaftafell",
            "",
            "Paris",
            "Paris,Eiffel Tower,Place de la Concorde,Jardin des Plantes",
            "",
            "Weekend in Tokyo",
            "Tokyo,Tokyo,Tokyo Metropolitan Government Building,Circus Tokyo,Asakura Museum of Sculpture,Kanae,Hotel Sunroute Plaza Shinjuku,Inshotei,Nata de Cristiano,GYOPAO Gyoza Roppongi,d47 SHOKUDO,Shibuya Scramble Crossing,Little Nap Coffee Stand,Shinjuku Golden Gai,Sarutahiko Coffee,Mag's Park Rooftop Shibuya Crossing,Ueno Toshogu Shrine,Kayaba Coffee,Sakurai Japanese Tea Experience,Ōkuma Shōkai,Meiji Jingu",
            "",
            "The Best Cheap Eats in Tokyo",
            "Tokyo,Tokyo,Yamatarō,Karaage Hitosuji,Butagumi Shokudo,Torikatsu Chicken Shibuya,Niigata Katsudon Tarekatsu Shibuya,Matsuya,YAKINIKU LIKE Shibuya Udagawacho,Ginza Shabutsu,Hakata Furyu Akihabara flagship shop,Pork Vindaloo Taberu Fukudaitoryo",
            "",
            "Port Clyde",
            "Port Clyde,Saint George,Boothbay,Bristol,Rockland,Rockland Café,Monhegan Boat Line,Pemaquid Point Lighthouse,Ocean Point Walk,Marshall Point Lighthouse",
            "",
            "Las Vegas",
            "Las Vegas,Las Vegas,Shin Lim - Limitless,Tacos El Gordo,High Roller,Fountains of Bellagio,M&M’S Las Vegas,SAHARA Las Vegas,Bavette's Steakhouse & Bar,Delmonico Steakhouse,Skybar,Shark Reef Aquarium at Mandalay Bay,Fremont Street Experience,Las Vegas Hilton at Resorts World,Secret Pizza,Allegiant Stadium,Eataly",
            "",
            "Hershey",
            "Hershey,The Mill in Hershey,Hershey Gardens,The Hotel Hershey",
            "",
            "Paia - Best beaches ",
            "Paia,Paia Inn,Paia Fish Market Restaurant,Milagros Food Co.,Paia Surf Lessons,Paia Bay,Mana Foods,Baldwin Beach Park,Café Des Amis,Flatbread Company",
            "",
            "24 Wild Hours on the Las Vegas Strip",
            "Las Vegas Strip,Las Vegas,Garden of the Gods Pool Oasis,High Roller,The Shops at Crystals,MGM Grand,Marquee Nightclub,Shin Lim - Limitless,Ghost Donkey,Tacos El Gordo,Momofuku,Eggslut,WAKUDA,Sphere",
            "",
            "10 Artsy Attractions in Barcelona",
            "Barcelona,Fundació Antoni Tàpies,Museu Nacional d'Art de Catalunya,Picasso Museum,Casa Batlló,CaixaForum Barcelona,El Gat de Botero,Moco Museum Barcelona,La Pedrera-Casa Milà,Barcelona Museum of Contemporary Art,Joan Miró Foundation",
            "",
            "Las Vegas",
            "Las Vegas,Las Vegas,Red Rock Canyon National Conservation Area,Fremont Street Experience,Fountains of Bellagio",
            "",
            "5 Incredible Dubrovnik Eats with Epic Views",
            "Dubrovnik,Komolac,Bistro Izvor,Nautika,Restaurant 360,Komarda,Restaurant Panorama",
            "",
            "Midweek Mammoth Ski Trip",
            "Mammoth Lakes,Bishop,Mammoth Mountain,The Stove,Erick Schat's Bakery,Mammoth Mountain Inn,Convict Lake,Roberto's Cafe,Holy Smoke Texas Style BBQ",
            "",
            "Oregon",
            "Oregon,Eugene,Columbia River Gorge National Scenic Area,Multnomah Falls,Hult Center for the Performing Arts",
            "",
            "Maldives",
            "Maldives,Malé,Hulhumalé,Fihalholi,Hulhule Island,Velassaru,Naladhu Private Island Maldives,Ithaa Undersea Restaurant,Seagull Café,Fish Market,Hulhumale,Sea House Café,MU Beach Bar & Grill",
            "",
            "10 must try breweries in Seattle!",
            "Seattle,Fremont Brewing's Urban Beer Garden,Holy Mountain Brewing Company,Cloudburst Brewing,Standard Brewing,Great Notion Brewing - Ballard,Seapine Brewing Company,Bizarre Brewing,Obec Brewing,Stoup Brewing - Ballard,Reuben's Brews - The Ballard Taproom",
            "Seattle is a mecca for incredible beer! Here are the breweries that are a must visit while here!",
            "Japan",
            "Japan,Sapporo,Kyoto,Hikone,Lake Biwa,Funaoka Onsen,Hōheikyō Hot Spring",
            "",
            "The 10 Best Outdoor Adventures in Dubrovnik",
            "Dubrovnik,Studenci,Dubrovnik Game of Thrones Tour,Dubrovnik Local Guides - City Walls Walking Tour,Lokrum,Dubrovnik Local Guides - Old Town Walking Tour,Blue Cave Dubrovnik | Official tours,Plaža Sveti Jakov,Dubrovnik Elafiti Cruise,반예비치,Betina Cave beach,Kravica Waterfall",
            "",
            "The Perfect Barcelona Weekend",
            "Barcelona,Moog,SlowMov,Banksy museum - Barcelona,Can Paixano,Entrepans Díaz,Bo de B,Barceloneta Beach,Palau de la Música Catalana,Funky Eatery,Nomad Coffee - Lab & Shop,Majestic Hotel & Spa Barcelona,La Sagrada Familia,Mercado de La Boqueria,Park Güell,La Pepita,Three Marks Coffee",
            "",
            "Japan",
            "Japan,Tokyo,Osaka,Kyoto,Urayasu,Nisekohirafu 2 Jo,Arashiyama Bamboo Forest,Aman Tokyo,Tokyo Disneyland,Afuri,OKO - Fun Okonomiyaki Bar,Dotonbori River,Sukiyabashi Jiro,Imperial Palace",
            "",
            "Aruba",
            "Aruba,Noord,Oranjestad,West Deck,Buccaneer Restaurant,Eagle Beach,Hyatt Regency Aruba Resort Spa And Casino,Palm Beach,Malmok Beach",
            "",
            "Czechia drone map",
            "Czechia,Štěchovice,Pňovany,Brno,Most,Nalžovské Hory,Velké Karlovice,Kovářov,Velemín,Bartošovice v Orlických horách,Kdyně,Seč,Dvůr Králové nad Labem,Dolní Morava,Chrudim,Jílové,Nechanice,Skalná,Třemošnice,Černý Důl,Turnov,Chudenice,Oslov,Libošovice,Velká Jesenice,Jílové u Prahy,Svojanov,Rabyně,Místo,Přítluky,Žleby,Nové Dvory,Točník,Podolí I,Heřmanice,Klapý,Nedvědice,Bříství,Klenová,Ráby,Zřícenina hradu Ballymote,Kost,Čarták lookout (Súkenická viewtower),Most u Červené nad Vltavou,Milešovka,Royal Forest Dam,Vyhlídka na přehradu,Veveří Castle,castle Svojanov,Nové Mlýny,Hospital Kuks,Bridge Pňovany,Hrádek u Nechanic,Vyhlídka Bednář,Ruins of the Chapel Exaltation. Crisis,Vyhlídka Neratov,Točník Castle,Nákladní lanovka Černý důl,Podolský most,Žampach viaduct,lookout Rýzmberk,Hněvín,Lichnice Castle ruins,Viewpoint Máj,Obora Žleby,Klenová - castle, chateau, gallery,lookout Bolfánek,Pernštejn Castle,Rock lookout Hlavatice,Žďákov Bridge,Seč Dam,Vildštejn Castle,Kunětice Mountain,Hazmburk,Hasištejn Castle,Rozkoš Lake,Velhartice,Železniční viadukt,Stezka v oblacích,Děčínský Sněžník,Kačina,Rozhledna Bára",
            "flying from castle park is prohibited, but never had problem to get permission from the castle,tip fromhttps://www.facebook.com/groups/letamesdrony/permalink/3295302460761543,tip fromhttps://www.facebook.com/groups/letamesdrony/posts/3295168087441647/,tip fromhttps://www.facebook.com/groups/letamesdrony/posts/3294486957509760/,https://www.youtube.com/watch?v=uh5Jv12hn7c&ab_channel=siamifpv",
            "10 Iconic Portland Eats ",
            "Portland,Hat Yai,Departure Restaurant + Lounge,Stumptown Coffee Roasters,Le Pigeon,Lil' America,Pine State Biscuits | NW 23rd,Great Notion Brewing - NW28th,Nong's Khao Man Gai (Downtown),Matt's BBQ,Voodoo Doughnut - Old Town",
            "",
            "Hottest new restaurants in Seattle!",
            "Seattle,Kirkland,Burien,Sap Sap Lao Deli & Cafe,Sushi Suzuki,Layers Green Lake,El Encanto,Ballard Beer Box,Ananas Pizzeria,Watson's Counter,Donna's,Stevie's Famous Burien,Miss Pho,Ben's Bread Co.,Sammich Seattle,G.H PASTA Co.,The Boat,Gold Coast Ghal Kitchen,FamilyFriend",
            "Come check out the hottest new restaurants in Seattle.",
            "1 Week in Los Angeles",
            "Los Angeles,Avalon,Hermosa Beach,Pasadena,Long Beach,Beverly Hills,Manhattan Beach,Central LA,San Marino,ROW DTLA,Stahl House,Mt Ada,Griffith Park,Tail O' the Pup,The Westin Long Beach,In-N-Out Burger,Catalina Express,Beverly Hills Park,Book Soup,Descanso Beach Club,Pages: a bookstore,Rose Bowl Flea Market,curious...,Pizzana,Uncorked The Wine Shop,The Shay - Destination by Hyatt,Avalon Hotel Beverly Hills,Gyu-Kaku Japanese BBQ,Anderton Court Shops - Frank Lloyd Wright,Airplane Landing View Point,Gum Tree Shop & Café,Catalina Museum For Art & History,Los Angeles County Museum of Art,Martha’s Hermosa Beach,Griffith Observatory,Catalina Casino,The Langham Huntington, Pasadena,Veterans Parkway (Hermosa Valley Greenbelt),Hollywood Walk of Fame,Clark Street Diner,Sunset Boulevard,Vroman's Bookstore,Dialog Cafe,HomeState,The Huntington Library, Art Museum, and Botanical Gardens,Avalon Grille,Los Angeles International Airport,Vista",
            "With 7 days you can see LA the way many first-time visitors do, as well as dig a little deeper. Hope you enjoy!This is a shopping day! Start at Gum Tree and if you need coffee, get it here before browsing their shop. I recommend a Dirty Chai or one of their seasonal specialty drinks.,Reward yourself for the early exercise with a White Corn Omelette and Bloody Mary at Martha's. Since it is mere steps from the beach, walk down to it to put your feed in the sand afterward.,You'll find all the quirky and super fun gifts at this LGBTQIA-friendly shop that always has a wonderful display in their window for PRIDE.,Go here before you tuck into bed for the night. Urban Light in the evening has its own special vibe and since it's outside the museum entrance, it's accessible 24/7/365.,Leave time today or tomorrow for the pool at this boutique hotel with interior design by Kelly Wearstler. ,After taking the ferry back to the mainland, get a final taste of California. The In-N-Out by LAX brings the bonus of being right under a landing zone so you watch planes land while enjoying your cheeseburger animal style. ,While watching planes land, take off your shoes. Walking on grass with bare feet is supposed to help you before and after flying.,Safe travels and enjoy your flight home!,Time your visit to arrive on a Saturday before the Rose Bowl Flea Market the next day. It takes place every second Sunday of the month. Head straight to The Langham and check-in. Take advantage of their amazing Chuan Spa to unwind after your flight.,Wake up super early and go shopping at the Flea. The early bird truly gets the worm and the best finds. It's like an LA right of passage and a great way to pick up a souvenir you literally can't find anywhere else.,Reward yourself for waking up early by heading here after the Flea. Nosh on a chorizo taco, queso and a Ranchwater or Paloma for maximum satisfaction. They have vegetarian options too!,Spend the balance of your day in the stacks finding your next read before you start the [possibly long - LA traffic!] drive to your next hotel.,This will be a packed day! So pick up some caffeine, if that's your thing, at the Alfred Coffee on the corner of Beverly Drive and Santa Monica to fuel up. Then walk across the street and stroll the park, headed in the direction of the Beverly Hills Sign.,Look for the angled circular staircase and tower by Frank Lloyd Wright that's hiding in plain sight here. If it's open, take a walk up to the top for a nice view of Rodeo Drive.,A cold Asahi or Sapporo goes well with their BBQ.,Try to catch sunset here in the park or at the top by the Griffith Observatory. Top tip if you have a rental car: since there is never enough parking up by the Observatory, look for the shuttle that allows you to park near the Greek Theater and get a ride up the hill. If not, Uber straight to the top.,Make sure to head inside and see the Tesla Coil if science is your thing!,Enjoy Happy Hour here where they have wine flights and a great selection of bottles you could take home to your hotel to enjoy before leaving California.,Start the day early with a wonderful walk on this greenbelt that goes through the beach communities of the South Bay.,If you want to be so close to the beach you could touch it for Happy Hour, or dinner, go to Vista instead of Uncorked. Or start at Uncorked and finish your day with dinner at Vista!,Pick up a book to read at the beach club tomorrow!,Stay here for the amazing happy hour, views over Catalina, the included golf carts so you can explore the island at leisure and the wonderfully sumptuous guest rooms.,Take the casino tour where you will understand why there's no gambling in this island landmark!,If you don't yet have a book for the plane ride home, this is a great place to pick one up. Or a second one!,Grab lunch at this landmark that harkens back to Hollywood's golden eras. ,You have to get tickets to tour this home before you arrive in Los Angeles but it's worth planning ahead for!,You can't leave Los Angeles without trying the pizza. It gives New York a run for its money and this is one of the best spots for it in LA.,Because you have to visit the stars when you're in Los Angeles, right?,Stay at The Westin so you're close to the Long Beach ferry to Catalina Island which you will want to board early the next day! It has a Travel Sustainability Rating of Level 3 according to booking.com!,Take an early ferry to Catalina Island to spend the day and night at California's version of The Riviera.,Visit this museum so you understand the connection the Chicago Cubs and Marilyn Monroe have to the island.,Enjoy dinner here before heading up the hill to Mt. Ada for a wonderful night's rest.,Lay out on the beach or enjoy the delicious tacos at their restaurant. If you're feeling the lux vibe, reserve a cabana. You can also snorkel or kayak in the bay.,Sit down for brinner, aka: breakfast for dinner, at this locally-loved but Tik Tok-friendly diner that's on the way from the Walk of Fame to Griffith Park.,Grab breakfast or brunch at this West Hollywood classic.,If you're not too tired from walking the Flea, head to this garden which is one of the most beautiful in all of Los Angeles.,If you can't get tickets to Stahl House or you want to experience another part of LA, check out ROW which is shops and restaurants that represent the best of creative LA. You can also feast on excellent pizza here, instead of Pizzana, at Pizzeria Bianco.,Take in the The Sunset Strip and do some shopping before returning to Book Soup. ,This hotel is incredibly situated whether you're driving yourself, taking public transportation or grabbing ubers during your visit to Los Angeles. It's also across the street from The Platform which is a super hot outdoor shopping and events complex.",
            "Beach Trip on Paros Island",
            "Paros,Santa Maria,Parasporos,Naousa,Chrisi Akti,Pounta,Kolimpithres,Kolympethres Beach,Parasporos Beach Club,Kalogeros beach,Faragas,Santa Maria,Paros Park,Port of Pounda,Παραλία Παρασπόρος,Golden Beach Paros,Piperi Beach,Santa Maria Beach Bar",
            "Famous for being the best overall beach on Paros!,Cool rocks and views.,Be sure to check out the healing mud.,Considered the best beach in Naoussa.,Rocks, caves, hiking and more!,Parasporos Beach is the best beach near Parikia.,Great spot for windsurfing!,Ferry to Antiparos",
            ]

Activity

  1. NirantK commented on May 10, 2024

    @NirantK
    Contributor

    There might be a few things at play here:

    1. Size of the text — we load everything in memory, both the text and it's vectors. This means as vectors get computed, there is more RAM consumption which keeps increasing.
    2. Data Parallel — we use Python's multi-processing under the hood to split large lists of strings into smaller and process in parallel. This can lead to large RAM consumption as well sometimes.

    Can you give some estimates of 3 things:

    1. How many tokens are there per sentence?
    2. How many sentences are there?
    3. What are the device RAM and CPU specs?
  2. Barney241 commented on May 13, 2024

    @Barney241
    Author

    @NirantK

    1. tokens not shure at the moment but in latest testing i used max of 512 characters for e5 large model
    2. single batch had about 130 items
    3. 64GB RAM, AMD Ryzen 9 5950X 16-Core Processor

    But the main problem i have is that after processing of batch is done the memory is not released. it only keeps growing untill the process gets killed. i think that the problem is in onnx session when using continuosly without recreating. when i use the model dirrectly from Hugging face, there is no growth of memory and no problem

  3. NirantK commented on May 14, 2024

    @NirantK
    Contributor

    Thanks for sharing specifics @Barney241 — very helpful.

    i think that the problem is in onnx session when using continuosly without recreating.

    This'd be among my areas to investigate as well. Thanks for sharing. I'll look into this as energy permits! I'm leaving this issue open and marking it as a bug because of the impact it has on how usable FastEmbed is.

  4. johnreyev commented on Sep 6, 2024

    @johnreyev

    Hi, any updates on this? We've experienced the same issue.

  5. joein commented on Sep 11, 2024

    @joein
    Member

    Hi @johnreyev

    Could you tell us the version of fastembed you're using and maybe a reproducible code snippet?
    We've tried to reproduce the issue with the code above, but unsuccessfully, we haven't been able to observe the memory growth

  6. johnreyev commented on Sep 12, 2024

    @johnreyev

    Hi @joein, you can try this

    fastembed==0.3.6
    lorem==0.1.1
    
    from timeit import default_timer as timer
    import lorem
    from fastembed import TextEmbedding
    
    def run():
        model = TextEmbedding(model_name='BAAI/bge-small-en-v1.5')
    
        # start timer
        start = timer()
    
        # Generate a large amount of sample text
        docs = [lorem.text() for _ in range(10000)]
    
        print(f'Each document is {len(docs[0])} chars long.')
        print(f'Embedding {len(docs)} documents...')
    
        # embed the documents
        embeddings = model.embed(documents=docs, batch_size=512)
    
        proto_embeddings = [
            embedding.tolist()
            for embedding in embeddings
        ]
    
        print(f'len(embedding_list) {len(proto_embeddings)}')
        print('Done!')
    
        # end timer
        end = timer()
    
        print(f'Time taken: {end - start:.2f} seconds')
    
    if __name__ == '__main__':
        run()
    
    

    It gets stuck at this:
    Screenshot 2024-09-12 at 10 07 38 AM

    For the device spec I'm using 14-inch MacBook Pro with M3 Pro

    Screenshot 2024-09-12 at 10 09 51 AM
  7. hh-space-invader commented on Sep 12, 2024

    @hh-space-invader
    Contributor

    @johnreyev
    I would suggest to do this:

    for embedding in model.embed(documents=docs, batch_size=512):
        first_batch_result = embedding.tolist()
        # process batch here, e.g., save it to a file or database or whatever
    

    rather than this:

    proto_embeddings = [
        embedding.tolist()
        for embedding in embeddings
    ]
    

    because you are embedding the documents in batches but you save them all in one variable and it might explode your memory when you increase the document length

  8. vladiliescu commented on Mar 18, 2025

    @vladiliescu

    Reproduces for me as well. When running an A/B test with fastembed versus raw sentence-transformers, for the same amount of texts to embed (a medium PDF of about 10MB, 762 pages), the process using sentence-transformers doesn't go over 2GB of RAM, whereas the process using fastembed goes to about 9GB of RAM.

    It's not related to keeping the texts (or embeddings) in memory, that happens with both approaches, it's something specific to fastembed

  9. philwinder commented on May 23, 2025

    @philwinder

    Same happening for me too. This is preventing me from using fastembed as much as I'd like to.

    This may be of interest. When I set .embed(..., parallel=0) It spun up a load of new processes. Which is fine. But look at the memory usage of each process. It's a bit more each time. I would have expected them to be all the same... 🤔

    Image

  10. CByBB commented on May 23, 2025

    @CByBB

    Hello 👋 I hope to contribute on this project.

    I'm a full-stack Web & Blockchain Developer currently open to new opportunities.

    I specialize in building casino & betting platforms, as well as developing games—on-chain and off-chain. From smart contract logic to sleek frontend UI, I cover the whole stack.

    • 🕹️ Casino sites, betting engines, game mechanics, custom dashboards
    • 🛠️ Solidity, web3.js, React, Node, Next.js, Tailwind, PostgreSQL, Redis
    • 🌍 Available across all time zones
    • ⏱️ Can commit to 60+ hours per week

    If you're working on something in the iGaming, DeFi, or blockchain gaming space, and need a dev who can move fast and build solid—feel free to reach out.

    Please drop a message.

  11. jozefRudy commented on Oct 3, 2025

    @jozefRudy

    the problem is not only memory consumption, but that it never goes down (is freed).

    i have very simple server:

    from fastapi import FastAPI, HTTPException
    from pydantic import BaseModel
    from fastembed import TextEmbedding
    import numpy as np
    import logging
    import os
    import gc
    
    logging.basicConfig(level=logging.INFO)
    logger = logging.getLogger(__name__)
    
    parallel_threads = int(os.getenv("EMBED_THREADS", str(os.cpu_count())))
    logger.info(f"Using parallel={parallel_threads}")
    model = TextEmbedding(model_name="nomic-ai/nomic-embed-text-v1.5-Q")
    app = FastAPI()
    
    
    class EmbedRequest(BaseModel):
        input: list[str]
    
    def quantize_to_binary(embedding: np.ndarray) -> list[int]:
        """Convert float embedding to binary (0 or 1)"""
        return (embedding >= 0).astype(np.uint8).tolist()
    
    
    def get_batch_size(texts):
        MIN_BATCH_SIZE = 5
        MAX_BATCH_SIZE = 30
    
        MEM_LIMIT_GB = float(os.getenv("MEM_LIMIT_GB", "2"))
        MAX_CHARS_PER_BATCH = int(MEM_LIMIT_GB * 5_000)
    
        total_chars = sum(len(t) for t in texts)
        avg_chars = total_chars / len(texts) if texts else 0
        batch_size = max(
            MIN_BATCH_SIZE, min(MAX_BATCH_SIZE, int(MAX_CHARS_PER_BATCH / avg_chars))
        )
        return batch_size
    
    
    def _process_embeddings(texts: list[str], quantize: bool):
        if len(texts) > 100:
            raise HTTPException(400, "Max 100 texts per request")
        if not texts or any(not t.strip() for t in texts):
            raise HTTPException(400, "Empty text not allowed")
    
        try:
            batch_size = get_batch_size(texts)
            logger.info(f"batch size: {batch_size}")
            embeds = list(model.embed(texts, batch_size=batch_size))
    
            if quantize:
                result = [quantize_to_binary(e) for e in embeds]
            else:
                result = [e.tolist() for e in embeds]
    
            return result
        except Exception as e:
            logger.error(f"Embedding failed: {e}")
            raise HTTPException(500, f"Embedding failed: {str(e)}")
    
    
    @app.post("/v1/embeddings")
    def embeddings(req: EmbedRequest):
        embeds = _process_embeddings(req.input, False)
        return {"embeddings": embeds}
    
    
    @app.post("/v1/embeddings/binary")
    def embeddings_binary(req: EmbedRequest):
        embeds = _process_embeddings(req.input, True)
        return {"embeddings": embeds}
    
    
    @app.get("/health")
    def health():
        return {"status": "ok"}

    And memory never goes down, even if CPU does, when i send it a batch, it does its job, both CPU/RAM jump, but then CPU goes to around 0% when done (as it should), but RAM stays high forever, is never freed.

  12. jozefRudy commented on Oct 6, 2025

    @jozefRudy

    ok, for the others, save yourselves time and just use llama.cpp.

    Dockerized usage is similar to this, it's API is OpenAI compatible, and performance is better than this.

    Example:

    llama-nomic-q4:
      image: ghcr.io/ggerganov/llama.cpp:server
      volumes:
        - ./models:/models
      ports:
        - "8081:8081"
      command: >
        -m /models/nomic-embed-text-v1.5.Q4_0.gguf --host 0.0.0.0 --port 8081 --embedding
    
  13. cosmic-fire-eng commented on Jul 24, 2026

    @cosmic-fire-eng

    Adding a data point from a container deployment, since this bit us in the same "process killed at the pod/cgroup memory limit" way as the original report.

    Setup: embedding ~550 short texts (tool descriptions, a few hundred chars each) with BAAI/bge-small-en-v1.5, inside a Docker container with a 5 GiB cgroup memory limit (no swap headroom) on a 6-core host. fastembed 0.3.6 with onnxruntime 1.26 — old versions, but the relevant defaults (batch_size=256, threads unset) are the same on current main.

    With the defaults — TextEmbedding(model_name=...) then .embed(texts) — the cgroup total spiked to ~6.5 GiB during the embed and the kernel OOM-killed the container. One profiling gotcha: an isolated test of the same embed showed only ~425 MB parent-process RSS, which had us looking in the wrong place for a while. The growth shows up in the cgroup total (memory.current), not the parent RSS.

    With TextEmbedding(model_name=..., threads=1) and .embed(texts, batch_size=16), the same workload on the same machine peaks at ~868 MB, steady state ~700 MB–1 GB. (Setting parallel>=1 made things worse for us — each worker process reloads the model.)

    So at least in our case this wasn't a slow leak — it was onnxruntime's per-core threads/arenas plus the large default batch, which allocate far more than the model size suggests on many-core hosts. That's exactly the shape that dies under k8s/docker memory limits while looking fine on an unconstrained dev box. enable_cpu_mem_arena=False (exposed since #578) looks like another useful knob here, though we haven't measured that one.

    A short docs note along the lines of "running under container memory limits: set threads=1 and a small batch_size" would probably save people a lot of debugging. Happy to open a docs PR for that if it'd be welcome. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions