* MultiServer, customserver: move data package handling
Create a new class that handles conversion worlds+embedded -> context data.
Create a derived class that uses static_server_data+pony instead.
There is also a not very efficient feature to deduplicate strings (may need perf testing).
By moving code around, we can simplify a lot of the world loading.
Where code lines were touched, some typing and some reformatting was added.
The back compat for GetDataPackage without games was finally dropped.
This was done as a cleanup because the refactoring touched those lines anyway.
Also reworked the per-context dicts and the RoomInfo to hopefully be more efficient
by ignoring unused games. (Generating the list of used games was required for the new
code anyway.)
Side effect of the MultiServer cache: we now load worlds lazily (but still all at once)
and don't modify the games package in place. If needed we create copies.
This almost gets us to the point where MultiServer doesn't need worlds - it still needs
them for the forbidden items.
There is a bonus optimization that deduplicates strings in name_groups that may have bad
performance and may need some perf testing if we run into issues.
* MultiServer: fix string deduplication in data package cache
* MultiServer, customserver: minor formatting fixes
* MultiServer: fix data package cache for missing checksum case
* MultiServer: improve string deduplication in games package cache
* customserver: handle missing checksum in datapackage cache
* customserver: typing cleanup in games package cache
* Test, MultiServer, customser: add tests for games package cache
* Test, MultiServer: reorder imports
Hopefully this fixes the random test failures with pytest-xdist
* customserver: make WebHost import lazy in games package cache
and fix test
* customserver: games package cache: fix py3.11 compat
* MultiServer, customserver: cache: typing improvements
* customserver: cache: reformat
* Test: move customserver tests to not interfere with webhost
* MultiServer, customserver: cache: rename module
* Multiserver: better error message for missing Datapackage
* customserver: cache: invert if for readability
* customserver: optimize non_hintable_names
* Use an empty frozenset singleton
* Only include non-empty sets in static_server_data
* Only prefill dict with played games
This should save ~2-4KB per room and slightly speedup lookup.
---------
Co-authored-by: Fabian Dill <[email protected]>
* Added ability to define custom port ranges the WebHost will use for game servers, instead of pure random.
* - Added better fallback to default port range when a custom range fails
- Updated config to be clearer
* Added ability to define custom port ranges the WebHost will use for game servers, instead of pure random.
* - Added better fallback to default port range when a custom range fails
- Updated config to be clearer
* Updated soft-fail message
* Removed dead import from customserver.py
* Update requirements.txt
Settings requirements to main core branch
* fix what reviewers said and add some improvements
* remove unused argument
* try fixing test with try
* use yaml lists instead of string for config
* fix value type bug on ephemeral type
* reuse sockets with websockets api instead of opening and closing them
* add used ports cache and filter used ports when looking for ports
* fix port randomizer
* Apply suggestions from code review
Co-authored-by: Duck <[email protected]>
* fix some reviews
* use weights for random port and remove more-itertools
* fix net_connections not working on macOS
* rename variables and functions
* lazy init `get_used_ports`
* change `game_ports` to be `tuple`
* fix last_used_ports not being updated locally
* fix random choices and move game_port conversion into tuple
* Apply suggestions from code review
Co-authored-by: Duck <[email protected]>
* use a named tuple on parse_game_ports
* only use ranges
* do it the duck way
* this should check all usable ports before failing
* fix while loop
* add return type to weighted random
* Update WebHostLib/customserver.py
Co-authored-by: black-sliver <[email protected]>
* simplify tuple conversion check
* add tests
* reformat file and change `create_random_port_socket` test
* add more test cases for parse_game_ports
* try to prevent busy-looping on create random port socket when doing test
* simplify parse game port tests to one assertListEqual
* make the range lesser for port test
* reduce range on macOS
* Apply suggestions from code review
Co-authored-by: black-sliver <[email protected]>
* Update WebHostLib/customserver.py
Co-authored-by: black-sliver <[email protected]>
* Apply suggestions from code review
Co-authored-by: Doug Hoskisson <[email protected]>
* remove unused import
* Update WebHostLib/customserver.py
Co-authored-by: Doug Hoskisson <[email protected]>
* use generator expressions
* check for 0-tuple
* use some kind of shuffled queue
* update tests
* refactor new port handling into a class (#1)
* change time to monotonic
* Update docs/webhost configuration sample.yaml
Co-authored-by: black-sliver <[email protected]>
* add psutil 7.2.2 as requirement
* Update WebHostLib/requirements.txt
Co-authored-by: black-sliver <[email protected]>
---------
Co-authored-by: Lexipherous <[email protected]>
Co-authored-by: Duck <[email protected]>
Co-authored-by: black-sliver <[email protected]>
Co-authored-by: Doug Hoskisson <[email protected]>
* Test: add hosting simulation test
* WebHost: add weak typing to get_app()
* MultiServer: add typing to auto_saver_thread
* MultiServer: don't cancel task, properly end it
* customserver: stop auto-save thread from saving after shutdown
and make sure it stops, another potential memory leak
* MultiServer, customserver: make datapackage small again
* customserver: collect/finish room tasks
Hopefully fixes the memory leak we are seeing
* CI: test hosting
* Test: hosting: verify autohoster saves on Ctrl+C
* customserver: save when stopping via Ctrl+C
when switching to multiple rooms per process, we ended up modifying the static server data
because that's how _load works and the data is now shared between multiple rooms.
Currently when a room shuts down while clients are connected it instantly spins back up. This fixes that behaviour categorically.
I still don't know why or when this problem started, but it's certainly wreaking havok on prod.
* Core: lazy-load worlds in unpickler
this should hopefully fix customserver's memory consumption
* WebHost: move imports around to save memory in MP
* MultiServer: prefer loading _speedups without pyximport
This saves ~15MB per MP and speeds up module import if it was built in-place.
* Tests: fix tests for changed WebHost imports
* CustomServer: run GC after setup
* CustomServer: cleanup exception handling
* WebHost: Make custom server prefer ipv4 for display
* WebHost: Make server retry saving in case of connection issues
* WebHost: fix autolaunch guardians getting stuck waiting for the oldest two rooms.
Probably not related to the issues of the system itself getting stuck, but should be fixed anyway.
* WebHost: logfile is meant to be guarded by access cookie
* WebHost: set patch target to null if port is not valid, disabling auto-connect
