The IPython At Home

[viviridian]

recently i've been poking at script modding for the sims 4, eli's been having weird deadlock issues with some of his (over 100gb of) sims 4 custom content and script mods. i'm trying to build out some tooling to give me some visibility into what the process is doing when python locks up.

i would just run ipython or something to get a repl but a bunch of the standard library is just missing, notably including async & coroutine support. but I do have access to threads and http responders. so i am building my own ipython in a cave with a box of scraps

this is what it looks like atm

the bottom is using codemirror 5, and i have autocomplete hooked up (!)

the source is over here atm https://git.vvn.space/vivlim/ts4-http-dev-server/src/branch/main/Scripts/vivlib (very likely to move around, possibly to a repo that isn't specific to ts4)

[viviridian]

last night I worked on an aggregating tracer. python lets you hook up some code to be run whenever a function is called, the resulting stream of events is *very* noisy but if I group things by directory I can start to see patterns in where time is being spent in various installed script mods. i think this should be a huge help in deadlock-finding

https://git.vvn.space/vivlim/ts4-http-dev-server/src/branch/main/Scripts/vivlib/ts4_script_tracer.py

it is pretty slow though. so i might do less processing in python and just pump a stream of events out to be filtered and aggregated in something else

[lifning]

this is very neat!! i have to imagine debugging that many of other people's scripts at once is always going to be a struggle, but if you're gonna do it, better to make some well-suited tools for the job.

(also, i'm picturing someone on the phone with an EA support rep suspiciously named Claude® 5 years from now, responding to their complaints about Sims 4 deadlocks with "Have you tried reinventing IPython from first principles and building a function-tracing profiler? Some customers have found that helps identify the issue.")

[viviridian]

i wanted to get a real profiler running in ts4_x64.exe and that was an ordeal.

- i wanted to run pyinstrument, which has a native extension module and uses async. it is able to export a speedscope-compatible file
- async isn't available in ts4 python so I hacked it out
- when i try to load a native extension module in ts4 python, the interpreter dll uses the non-standard name 'python37_x64.dll' instead of 'python37.dll'. python has a check for whether the dll named in an extension module's PE header symbol import table matches it, and will reject an extension module if it isn't an exact match. that means I can't load existing extension modules
- i tried fiddling with the PE header in a hex editor but there isn't enough space to add '_x64' without needing to relocate stuff, and that was too hairy for me to reasonably do
- so instead i forked cpython and modified it to build python37_x64.dll. extension modules built against *that* are loadable in ts4_x64.exe !