Files
black-sliverandFabian Dill 5442df64fd MultiServer, customserver: move data package handling, optimize RAM usage (#5981)
* MultiServer, customserver: move data package handling

Create a new class that handles conversion worlds+embedded -> context data.
Create a derived class that uses static_server_data+pony instead.
There is also a not very efficient feature to deduplicate strings (may need perf testing).

By moving code around, we can simplify a lot of the world loading.
Where code lines were touched, some typing and some reformatting was added.
The back compat for GetDataPackage without games was finally dropped.
This was done as a cleanup because the refactoring touched those lines anyway.

Also reworked the per-context dicts and the RoomInfo to hopefully be more efficient
by ignoring unused games. (Generating the list of used games was required for the new
code anyway.)

Side effect of the MultiServer cache: we now load worlds lazily (but still all at once)
and don't modify the games package in place. If needed we create copies.
This almost gets us to the point where MultiServer doesn't need worlds - it still needs
them for the forbidden items.
There is a bonus optimization that deduplicates strings in name_groups that may have bad
performance and may need some perf testing if we run into issues.

* MultiServer: fix string deduplication in data package cache

* MultiServer, customserver: minor formatting fixes

* MultiServer: fix data package cache for missing checksum case

* MultiServer: improve string deduplication in games package cache

* customserver: handle missing checksum in datapackage cache

* customserver: typing cleanup in games package cache

* Test, MultiServer, customser: add tests for games package cache

* Test, MultiServer: reorder imports

Hopefully this fixes the random test failures with pytest-xdist

* customserver: make WebHost import lazy in games package cache

and fix test

* customserver: games package cache: fix py3.11 compat

* MultiServer, customserver: cache: typing improvements

* customserver: cache: reformat

* Test: move customserver tests to not interfere with webhost

* MultiServer, customserver: cache: rename module

* Multiserver: better error message for missing Datapackage

* customserver: cache: invert if for readability

* customserver: optimize non_hintable_names

* Use an empty frozenset singleton
* Only include non-empty sets in static_server_data
* Only prefill dict with played games

This should save ~2-4KB per room and slightly speedup lookup.

---------

Co-authored-by: Fabian Dill <[email protected]>
2026-09-24 23:33:30 +02:00

97 lines
4.1 KiB
Python

import typing as t
from weakref import WeakValueDictionary
from NetUtils import GamesPackage
GameAndChecksum = tuple[str, str | None]
ItemNameGroups = dict[str, list[str]]
LocationNameGroups = dict[str, list[str]]
K = t.TypeVar("K")
V = t.TypeVar("V")
class DictLike(dict[K, V]):
__slots__ = ("__weakref__",)
class GamesPackageCache:
# NOTE: this uses 3 separate collections because unpacking the get() result would end the container lifetime
_reduced_games_packages: WeakValueDictionary[GameAndChecksum, GamesPackage]
"""Does not include item_name_groups nor location_name_groups"""
_item_name_groups: WeakValueDictionary[GameAndChecksum, dict[str, list[str]]]
_location_name_groups: WeakValueDictionary[GameAndChecksum, dict[str, list[str]]]
def __init__(self) -> None:
self._reduced_games_packages = WeakValueDictionary()
self._item_name_groups = WeakValueDictionary()
self._location_name_groups = WeakValueDictionary()
def _get(
self,
cache_key: GameAndChecksum,
) -> tuple[GamesPackage | None, ItemNameGroups | None, LocationNameGroups | None]:
if cache_key[1] is None:
return None, None, None
return (
self._reduced_games_packages.get(cache_key, None),
self._item_name_groups.get(cache_key, None),
self._location_name_groups.get(cache_key, None),
)
def get(
self,
game: str,
full_games_package: GamesPackage,
) -> tuple[GamesPackage, ItemNameGroups, LocationNameGroups]:
"""Loads and caches embedded data package provided by multidata"""
cache_key = (game, full_games_package.get("checksum", None))
cached_reduced_games_package, cached_item_name_groups, cached_location_name_groups = self._get(cache_key)
if cached_reduced_games_package is None:
cached_reduced_games_package = t.cast(
t.Any,
DictLike(
{
"item_name_to_id": full_games_package["item_name_to_id"],
"location_name_to_id": full_games_package["location_name_to_id"],
"checksum": full_games_package.get("checksum", None),
}
),
)
if cache_key[1] is not None: # only cache if checksum is available
self._reduced_games_packages[cache_key] = cached_reduced_games_package
if cached_item_name_groups is None:
# optimize strings to be references instead of copies
item_names = {name: name for name in cached_reduced_games_package["item_name_to_id"].keys()}
cached_item_name_groups = DictLike(
{
group_name: [item_names.get(item_name, item_name) for item_name in group_items]
for group_name, group_items in full_games_package["item_name_groups"].items()
}
)
if cache_key[1] is not None: # only cache if checksum is available
self._item_name_groups[cache_key] = cached_item_name_groups
if cached_location_name_groups is None:
# optimize strings to be references instead of copies
location_names = {name: name for name in cached_reduced_games_package["location_name_to_id"].keys()}
cached_location_name_groups = DictLike(
{
group_name: [location_names.get(location_name, location_name) for location_name in group_locations]
for group_name, group_locations in full_games_package.get("location_name_groups", {}).items()
}
)
if cache_key[1] is not None: # only cache if checksum is available
self._location_name_groups[cache_key] = cached_location_name_groups
return cached_reduced_games_package, cached_item_name_groups, cached_location_name_groups
def get_static(self, game: str) -> tuple[GamesPackage, ItemNameGroups, LocationNameGroups]:
"""Loads legacy data package from installed worlds"""
import worlds
return self.get(game, worlds.network_data_package["games"][game])