|
8298
|
Sephyr Magic mod for Minecraft by @Szczurek | Star Sephyr Magic mod for Minecraft by @Szczurek | Stardance...
|
Szczurek
|
0
|
177.4
|
0
|
[]
|
[{"post_id": 84615, "ts": "202 [{"post_id": 84615, "ts": "2026-10-29T22:28:12Z", "hours": 10.4481, "text": "Faster, smaller, better and a mushroom shark A big success has happened! I managed to replace the IPA speech recognition model “Multipa” with “Zipa”. The new model is over 4x smaller (260MB vs 1.3GB), it is noticeably faster to compute, and the best part?! Depending on the case it has similar or better accuracy then Multipa, which means less missed or mistakenly detected words! This means now it take less time for the spell to be detected after you say it. You can see it in the video, where Zipa is much faster, and Multipa misses the last word each time (it didn’t detect ˈɔnːdˌɔː nor fɒksɑm ). Few devlogs ago, when I originally implemented the model I said it wasn’t useable, but two fixes changed that: The sliding window audio cutting had to be slightly adjusted, because the model always outputs 4 logits less than Multipa. I discovered the model works much better with more context, even if it’s not relevant. I recorded myself saying “mushroom shark” and saying 6 different words. In each case adding the shark audio to the audio of the word made the IPA output of the word better! I don’t know why it works like that. Maybe the model then has better understanding of the speaker’s voice? What matters is that it helps! I adjusted the sliding window chunker to give the model additional 1.5s of previous speech audio, output. It’s translation is later cut off, so the result stays the same. I was able to do that without making latency significantly larger thanks to the new model being really fast. These two changes made the models comparable, so I updated Sephyr to use Zipa. Leap spell tweaking I spent a bunch of time working on the leap spell, to make it feel nicer. I made it apply a continuous force over time on the player instead of a one big impulse. I also made it apply some vertical force to the player depending on the looking angle, so that the player launches forward instead of rubbing against the ground. To make it smooth I engineered three math functions in Desmos, that control the amount of force added over time. Finally I updated the VFX, attaching it to player and making particles simulate in world space. This way the trails now follows the player. Making microphone selectable So far the mod always used a default system input device for capturing the audio with no way to control it. This works most of the time, but sometimes something can be wrong or you might have multiple input devices. So I need to give the user control. Sounds simple, but it required a surprisingly big portion of work. I had to update Sepple (the keyword detection engine) to support selecting a device, and changing it at runtime (which requires restarting the capture). Then I had to update the Rust<->Java bindings to accomodate that, and finally I had to wire up in Minecraft code. I decided to expose this setting as a /sephyr microphone <mic> command. Here I got a little bit stuck on implementing the microphone name autocompletion, but luckily a friend of mine knew, and told me how to do that. Here I ran out of devlog time (it’s already 19 minutes over), but it’s almost working!"}, {"post_id": 83038, "ts": "2026-10-26T17:13:18Z", "hours": 8.6053, "text": "Wind beam spell The wind beam spell is now done! Check it out in the video! Make sure to see the failure compilation at the end! It took surprisingly loooong to make. I additionally track my time outside Hackatime, in my task manager (which includes all time spent, not just coding). It clocked 8 hours, 15 minutes of my work just to implement such simple spell. Much of it was spent on fighting with math and the particles vfx, which were not cooperating. The spell’s area of affect is a 3d rectangle, that rotates with the player. I had to set up math, that could find entities in it, first by calculating an axis-aligned bounding box, and the filtering entites that overlap with it with an another check of whether they are touching the shape. The worst part with math is that a single wrong minus sign can break everything, and it’s quite hard to track down where it went wrong. After I managed to make it work, I noticed the VFX is wrong, rendering in every possible direction, except the right one. After another battle the particles were in place, but now the collisions were wrong again! Finally I got both right. Other things I also did a bunch of other smaller things: I added tracking of whether the spell was started, to fix it running more then once on the client sometimes. I refactored the spell serialization code to use normal Codec s instead of StreamCodec s, so that spell data can be persisted. I added client side spell ticking, which I needed for debugging, and which might come useful later. I set up a bunch of mod’s metadata - github url, set version to 0.1.0, ensured it has depencencies declared properly. I set up bundling of some java dependencies of the mod in the jar, so that it can be ran outside of development environment. I made an icon for the mod. I submitted the mod for a review on modrinth, so that it can be published. As you can see, as the end of Stardance approaches I’m preparing to polish and publish the mod for you to try out!"}, {"post_id": 82241, "ts": "2026-10-24T11:30:52Z", "hours": 10.525, "text": "Leap VFX and new spells The leap spell now has the simple VFX made last devlog attached. This is now possible thanks to the changes described below. It’s ugly, it usually just clips into the ground, but it’s better then nothing. I also added a new spell - ʔalɪˈvɑn wˈanga - wind shot . It shoots a wind projectile, for now - a wind charge, in the future - a custom one. You can see both in attached video. I also started working on a third spell - wind beam, but I wasn’t able to finish it this devlog. Spell entites The casting code rework is now done. After struggling for quite a lot of time, I managed to implement a SpellEntity , and rework how spells are built and registered. Let me tell you why it was hard. Each spell is unique, with it’s own set of modifiers that translate to unique set of data that the spell needs to run. That data is needed both on the server, to run the logic, and on the client to display VFX. That meant I needed to find a way, to serialize the data for the client, and then have it figure out, what it just received. What came in clutch here is registry dispatch, descibed nicely on Fabric Docs. Mojang built an entire serialization system called Codecs, which let’s you describe how a class can be serialized. Registry dispatch is a feature, where each you register each class to registry, along with such codec, and Minecraft can the use that to figure out how to deal with it. For that it uses a separate type, that I called SpellDefinition . It holds the codec, the brigadier spell node, the spell’s element, and also two new options - ticking and lifetimeTicks . The full casting pipeline now works like that: Player says spell words Those are detected by Sepple, send to server and accumulated. When a new word is added the server uses Brigadier to check if the spell is finished, or timeout ran. If so, the spell is cast - the brigadier command is executed. Each spell implements the code executed here, and either runs it’s logic instantly, or creates a new instance of it’s own Spell subclass. Here any data the spell needs can be gathered, like the targets or parameters, and stored in the subclass. Then it is ran using it’s SpellCtx context class. SpellCtx holds details like the entity that casted the spell, a postion, a direction and some other details. When a Spell is ran, a new SpellEntity holding it is spawned. The entity is synced to the client by Minecraft, along with the spell it is holding. The entity calls the run function of the spell, and runClient on the client once. This is where a spell can implement it’s one-time logic, and pm client side - VFX. Additionally, if ticking was enabled in the definition, on each tick the tick function of the spell is called by the entity. This is where behavior that runs over time can be implemented. After the amount of ticks configured in the definition the spell entity despawns. As you can see I have much more flexibility now, compare to the old system, where I only had a simple serverside callback for the logic. /sephyr cast command During development you sometimes need to cast one spell a lot, quickly. This happened when I was tweaking the leap spell VFX. The leap spell has the ɛvi (self) word in it, the detection of which often fails, because it is so short. It tired me, so I implemented a /sephyr cast <spell words> command, which can be used to run the spell. It’s creative mode only, because it ruins the point of the mod, but it is really helpful for testing. I managed to implement it quite neatly. The sephyr spells, similary to minecraft commands use the Brigadier command framework, but are registered separately. So I wrote a function, which walks the spell command tree, and registers analogous minecraft commands under /sephyr cast ."}, {"post_id": 81656, "ts": "2026-10-21T22:33:18Z", "hours": 9.8467, "text": "Multiloader modding is pain Can’t NeoForge and Fabric just have one common api, for all the boring stuff like registration, networking and events? I’ve had enough, so I spent 2 hours porting my mod to Balm , which is a library providing exactly that. The result is beautiful - ~150 lines are gone, and the loader specific subprojects fit under 10 lines of code, as I was able to move everything to common. Now for the worst irony: I spent my time doing all of that and the next thing I’m gonna do is… disable fabric support 😭 Why, you may ask? I was forced into it by reality read on First VFX I made my first VFX, for the leap spell. You can see it in the attached video. It’s going to spawn under player’s legs. For now it’s a really simple one, as I’m rushing to add a bit more of content, than just a single spell, before Stardance ends. It is made using Photon Editor , which is a VFX editor for Minecraft. It has a lot of good stuff, like emitters, timeline and animation curves, which will be crucial in making more advanced effects. Only downside: It doesn’t support Fabric, which is why support for it is going to be disabled for now, until I manage to hopefully make a port it in the future. Reworking spell internals (again) After making the effect, I realized I have no way to run it from the spell code, because spell logic runs server side only, and VFX needs to be played on the client. So I wen’t back to the drawing board, and I’m going to change how spells run. They are going be entities now, but more on that next devlog, because it’s not done yet, as I ran into friction in the code, which triggered me to do the Balm refactor, and in the meantime the time counter hit 9,5h. Do NOT hardcode stuff please I had to do some work on my laptop today, which has NixOS. So I update my Sephyr clone, try to run it, and bada bing bada bum, it doesn’t work! What’s going on? After a hour of debugging, it turned out that NeoForge devs hardcoded their Gradle plugin to run one specific task using Java 21. The rest runs on Java 25 (the one used by Minecraft 26.1). So, be default, a NeoForge setup for Mc 26.1+ requires two java version. Normally the second one is autodownloaded, but that doesn’t work on NixOS, and that’s also bloat. I don’t want that! So I made a bug report . Fortunetely it isn’t really hardcoded, just dumb, and I was able to fix it with this line in build.gradle Update to Burn 0.22 Burn 0.22 was recently released, removing the Backend generic from most of it’s traits. I updated Sepple to work with that. I also had to regenerate the model code for the multipa model. I was hoping the update would provide some speedup to speech processing, but I wasn’t able to measure any difference. Replacing Multipa with Zipa I found a new speech-to-IPA model. It’s called Zipa , and it looks very promising. It’s around 4x smaller then multipa, which means faster processing and less to download for end users. Furthermore, according to it’s paper, and my few tests it has noticeable better accuracy! So I implemnted it. It was a little more trouble then multipa, because the model has a much richer pre-processing. The audio waveform is turned into a mel spectogram, Fourier Transform is done, and other stuff happens. Buuut there is a problem. It turns out the model is great at doing recognition on longer audio sequences, and doesn’t really work that well on short ones. For them it performed worse then multipa! That means it’s not suitable for doing recognition in realtime, at least when plugged in the same way multipa was. However I have a couple ideas to try, which might, hopefully, solve the issue. Until I do that, it is set aside on a separate branch."}, {"post_id": 79314, "ts": "2026-10-13T14:21:33Z", "hours": 10.1344, "text": "Debugging gui is (almost) done What is going on, why already 10 hours passed? This was menat to be a quick sidequest, where I built this gui tool for debugging how well detecting magic keywords is working BUUT given it is the third devlog about it That means it took over 30 hours! Well, happens. So, all of this devlog was spent building all the plumbing needed to capture all the data from the detection pipeline and sending it in realtime to the app. All of it is using done using channels. They are injected into the detection pipeline either as Tap<T> ‘s - a new pipeline element that clones data it sees, sends to channel, and passes it on, or using wrapper types around existing processors. The app then receives the data and displays in on the plot. I also encountered and fixed few bugs, that I didn’t expect. For example originally the zoom went to crazy values, because a timestamp used to determine where the data ends wasn’t converted to be relative to the timeline start. I also found an quite an issue introduced 3 months ago, where the timestamps attached to audio weren’t calculated properly, making entire audio chunks have identical stamps. It was minor enough that I never noticed till now, when the data landed on the plot and resulted in this funny thing you can see on the attached screenshot. Finally I added a simple scrolling animation, to amortize the jitter caused by bigger chunks of audio coming, so the plot doesn’t jump that much when going live, but it’s still not as smooth as I would like it to be. As mentioned before I was using AI assistance in strictly controlled manner (I carefully planned, revieved, cleaned up and adjusted to my liking all the code generated by it) to build this debugging tool faster, and by now I’m very tired of it. At the end of the work it started doing stupid things (possibly because some context was lost?), which made me question if it was making me faster or slower. Luckily the work is mostly over, and I can finally get to other, more interesting stuff, that will be fun to do, with no AI. Next I want to use the tool to improve the detection a bit, and spend more time working on the actual mod."}, {"post_id": 77831, "ts": "2026-10-08T13:13:43Z", "hours": 10.2597, "text": "Keyword detection UI is done Look as this! It’s so pretty! Seeing all the data visualised is quite mesmerizing. I know the image will probably attract more views, so lemme explain what you are looking at. I’m making a minecraft mod where you cast spells by saying custom made up words, and this is a tool for debugging the keyword detection system. It’s meant for me, to help me make the detection more accurate, and reliable. I’ve been building it for the last ~20 hours, and in the last two I built the rendering of all the 8 strips. Check previous devlog if you want to know what exactly they display. It was pretty smooth sailing. As said before, since this is just a debugging tool for me, that I want to have finished quickly for it I’m using AI assistence, by letting it do stuff in small, pre-planned steps, and unsloppifying all the code between them. It’s working well. What helps, is that entire app is built on the egui / eframe crates, which I used before, and are somewhat familiar with. I gotta give credit, where is due, their api is really intuitive for the most part, and simple to use. The entire chart is also zoomable, and scrollable, and making this work was a bit of pain. I wanted to limit the accesible range to only around the data (so you can’t scroll away to far from it), and egui_plot doesn’t really provide a way to do it, so some manual calculations had to be done. I wanted to have some breathing space by allowing 0.25x of the zoom after last audio sample to be visible. This was the worst part, as it’s kind of a recursive problem - the zoom value affects how much zoom is allowed. But after around two hours I manged to figure it out, but keeping the original zoom value independantly of the limit calculated from it. Worth noting - what you see is just a mockup data, and the gui isn’t actually hooked up to the detection system, as that’s what I’m going to do next. After that I can hopefully improve the detection, and the finally go back to minecraft coding."}, {"post_id": 76510, "ts": "2026-10-03T19:41:08Z", "hours": 11.5369, "text": "Visualizing the keyword detection pipeline I’m not happy with the accuracy of the magic words detection system. It work half of the time, sometimes better, sometimes worse depending on the word. This make casting longer spells quite troublesome. Debugging why a specific word wasn’t detected is hard, debug prints can only get you so far. So I decided to go on a crazy sidequest and build a gui tool, that will help me debug this. I spent over 2 hours designing the concept, and over 10 implementing it, and it’s around 40% ready. The tool will display 8 strips / plots, and each plot will display some data, all on a shared timeline on the bottom, that will update live as the audio is processed. Strips 1, 2, and 4 will display waveforms of the audio at different processing stages. Strip 3 will show Voice Activity Detection score (which is used to filter out non-speech). Strips 5, 6 and 7 will display detected IPA tokens, accumulated text buffer, and detected words respectively. Strip 8 will display latency stats, which will help in optimizing it. This way I will be able to easily see, “oh, this phoneme wasn’t detected” or “oh, VAD cut out this word” or “oh, some mess accumulated in the buffer” and tweak my algorithms. AI - it’s bad, or is it? Despite looking quite simple on the surface, the app is still quite an undertaking. And it’s just a debugging tool! A thing to help me get the main goal done! I don’t want to spend too much time on it. So I considered using AI as a tool ( NOT vibecoding ). I hate vibecod(ing, ers), and I’ve always steered off AI, because very often I wasn’t satisfied with the output quality. I take pride in writing good code. But recently I’ve been trying out OpenCode here and there (on free models), and was quite surprised with it’s capabilities. Sure, it’s still often slop, but a one you can quite quickly correct, and sometimes be more effective if used correctly. AI - is it helpful? So I decided to do an experiment - can carefully planned AI usage help me write the debugging tool quickly? I started off by designing the functionality of the tool - I described the idea in detail to MiMo-V2.6-Flash, and asked it to interrogate me about as many details as possible and needed. I had to answer 35 questions, some of which were quite observant, picking me on nooks and crannies. After that a markdown file describing the design was written, and then went through few corrections. Next up, in a new session I asked MiMo to create a bullet point outline of steps needed to be done to implement the tool specified in the design file. Then one by one I went over each bullet point, asking the just released LongCat 2.5 (which has a bigger ctx window) to ask me questions, and formulate an implementation plan of it, present it to me, I gave corrections, and the point in file was replaced. This way I ended up with a second markdown file - an implementation plan. This took way too long, and was really boring. So, finally coding time. I do it like this: Ask opencode to implement a step of the plan Review the diff in vscode Remove fluff and slop manually, make sure all new code is good quality. Give feedback if I something doesn’t work like expected, or also fix it manually. After it’s good - commit Repeat for the next step It’s still not super fun, but at least I’m touching the code now, so much better already. And it is efficient! I believe it definitely would take me longer, if I wasn’t using AI. So the morale is: If you want fun, or doing something important - do it manually. If you want to do something quickly and get it out of the way less important. Here is a screen of how it is looking now: the general shape is in place, but there is no plots yet."}, {"post_id": 72706, "ts": "2026-09-23T15:27:59Z", "hours": 6.3589, "text": "The Stack Overflow For some reason my mod broke and started crashing during startup, and it did so in the worst possible way - without any error message! All I got in the Intellij Idea, was a message that the process exited with the signal code 11. WHAT? That signal corresponds to SIGSEGV signal on Linux, which generally means you did something wrong with the memory. But this is Rust! There shouldn’t be any memory issues! I started searching, and I learned there is a tool called coredumpctl, which lets me access a special dump file generated in such cases, and analyze it in a debugger like GDB. So I did that, and what I found was even more bizzare! The debugger was pointing to a start of a function, before the first line. I added some debug prints, and to my disbelief, the print before calling that function ran, and the the second one, which was the first line inside the function didn’t! Fast forward a bit I managed to figure out the problem, with some help from a friend. It turned out I encountered an stack overflow! When the function was called, Rust tried to expand the stack, and crashed because the stack ran out! Why? Because the code was loaded in Java, and Java by default allocates less memory for the stack ! The solution was manually requesting a bigger stack size during thread creation in Java. I also discovered, why the problem appeared - in the Rust file download code I had allocated an array buffer of 1MB. It landed on the stack, and that was too much! The solution was using a Vec, which lives on the heap instead. Stupid problem, but took 3 hours of my time 😩 Mod init progress reporting Sephyr requires the IPA speech recognition model to work, which needs to be downloaded on the first run, and subsequently loaded from the disk. The logic was overhauled providing more corectness, and hooked up to a series of toasts. File download progress bar is shown, and the intialization state (or failure) is coummunicated to the user! See it in the video"}, {"post_id": 71471, "ts": "2026-09-20T21:50:14Z", "hours": 9.7653, "text": "Wind spells have sound now Using LMMS and ZynAddSubFx I made two wind sound effects, which now play when casting a wind spell. You can hear them in the attached video. Cross-Compiling is fun The speech recognition system is written in Rust, which compiles down to native code. This means that it needs to be compiled separately for each operating system, and processor architecture. Rust is more friendly to it, then C or C++ for example, but as always, compilation finds it own weird ways to break. I took the setup used by Sable as a base, with two Docker containers, one for Linux and Mac, and one for Windows. BUT: The Rust version installed in the docker image was not new enough for one of my dependencies - I had to update the rust toolchain in the container image. Compiling for linux required ALSA headers for audio access - I had to figure out, how to install them in a way, that works for both x86_64 and arm64, without breaking each other. The latest version of the base image has Mac cross compilation broken - I had to use a previous one. The image ships a MacOS SDK from 5 years ago - too old, doesn’t work, so it also had to be replaced. - the container is kinda becoming a Ship of Theseus xD My code was pulling openssl as a dependency, which was failing to compile - I had to spent over an hour, figuring out how to get rid of it from the dependency tree, discovering I can do so by updating a dependency to an unreleased version. Gradle obviously required some tweaks, who doesn’t love Gradle right? In the end I managed to get it working, which brings the mod closer to being usable by the public, outside dev environment. Downloading the model The mod works by running an IPA speech recognition model locally. The model weights around 1.2G, making it too large to be embedded in the mod jar. So I added automatic download support - the model can now be downloaded automatically on the first time an user runs the mod. There is even progress tracking, though it’s not hooked up to Minecraft yet. A Toast is also shown when the loading fails, making the mod more user friendly."}, {"post_id": 69097, "ts": "2026-09-16T11:23:50Z", "hours": 10.2308, "text": "SPELLCASTING WORKS! The main mechanic of the mod is working! YOU CAN CAST SPELLS WITH VOICE! Finally! It took soo long… After over two months of development, I finally have a proof, that my crazy idea actually works, and was worth trying out. As you can see (and hear) in the video I implemented a simple wind leap spell, as the first one in the mod. Note: There are no visuals or sounds yet, but they will be added, and this is just a proof of concept. The spell comes in three variants ʔalɪˈvɑn vɛɾtɛɾ ɛvi ˈonːdˌoː - wind launches you forward ʔalɪˈvɑn vɛɾtɛɾ ɛvi ãntikɔ - wind launches you backward ʔalɪˈvɑn vɛɾtɛɾ ɛvi fɒksɑm - wind launches you up (no matter where you are looking), not shown in the video Additionaly, it can be cast as just ʔalɪˈvɑn vɛɾtɛɾ ɛvi , which is a shortcut for the first variant. This can seen in the third attempt, where the system didin’t catch onːdˌoː , but the spell worked anyway. This flexibility is easily possible, thanks to Mojang’s brigadier library. Brigadier powers the / commands in vanilla, and now the spell syntax and execution in my mod. I settled on it after trying two alternative approaches and failing to make them work. After all, my spells are just sentences, with the only difference, is that you speak them instead of writing."}, {"post_id": 55219, "ts": "2026-08-26T20:03:24Z", "hours": 10.6233, "text": "Note: This one is kinda boring, mostly plumbing stuff to get things to work, feel free to skip it. Sephyr (the mod) I’m working on setting up the the logic for registering and casting spells. This is quite hard, as the design of the mod is far from complete, which means I don’t even know what exactly do I need. I created a registry of elements, which are a source of a color, and are going to have other effects in the future. The first word of a spell is its element. For example in the spell ʔalɪˈvɑn vɛɾtɛɾ ɛvi the word ʔalɪˈvɑn means wind. I also made a basic system for figuring out the spell the player is casting. For now its very simple and can’t handle more complex spells, for example, with optional words. The entirety is a big mess as I’m figuring things out, and the fact I’m making this mod for both Neo and Fabric doesn’t make it any easier. I also updated the mod to Minecraft 26.1.2, which means there is going to be no 1.21.1 release. Sepple (word detection system) I keep working on collecting data to finetine the speech to ipa ml model. After discovering a better way of getting the audio files while writing last devlog I scraped the download script, and instead replaced it with a new one to extract needed files from the 20GB Wiktinary audio dump tar. I also cleaned up the entire code, turning it back to human standards, after a bit of Opencode usage. (Since those are one-time scripts to extract data I decided using a bit of AI in this area won’t hurt. I don’t use AI for the rest of the project.). I also prepared a fourth script, that will use the existing multipa model, to generate IPA transcription of the files. Then files with generated IPA being too different from original IPA will be discarded, as a way to detect files, where the IPA doesn’t match the audio (because the audio contains more text for example)."}, {"post_id": 45897, "ts": "2026-08-12T17:21:05Z", "hours": 10.2356, "text": "Sephyr (the mod) A proper lifecycle of the detection system was added. The IPA model is loaded in background during game start, but the audio capture and processing start waits until you join a world, and stops once you leave it. The Rust code now has a logging hook, so that it can do civilized logging using the Java logger, instead of blatantly writing to stdout. Sepple (the voice casting detection system) I made the pipeline system cancellable, so that the mod can cleanly stop the detection, which was not possible before. Support for running the speech detection inference on GPU was added as a feature flag. For now it is not used by default, as it showed no measurable improvements over running on CPU on my PC, but it might be useful on other, weaker devices. Collecting data for training As I said too many times, the speech to IPA model used for magic speech detection is not quite the quality I would like it to be. I’m hoping to improve it by fine-tuning it on its weak areas, but for that I need some data. My idea is to use wiktionary as a data source. It is a free online dictionary run by the same people who run Wikipedia. Most words on it have the pronunciation attached both as an IPA and an audio file, which makes it ideal for the task. I downloaded a preprocessed jsonl dump file from kaikki and wrote a script to extract just words and ipa from it. Unfortunetely it was missing urls of the audio files. So I had to download a raw xml dump of the entire english wiktionary, and spend few hours writing a script to extract them. Then I used deepseek a generate a third script, to download all the audio files from a list. There is going to be almost 20k of them, and the script is running right now as I’m writing this, and it is going to take multiple hours as I can only download around 1 file per second to respect rate limits and not overburden wikimedia. And just now while writing I realized I’m an idiot, because kaikki provides a 20GB tar file with all the audio, downloading which would be more efficient."}, {"post_id": 41944, "ts": "2026-08-06T14:02:27Z", "hours": 9.6106, "text": "Integrating Rust IPA word detection with Minecraft With the IPA based magic keyword detection (which is written in Rust) working I spent all of this devlog integrating it into Minecraft. I created a new repo for the mod, using the Jared’s template. I decided to go with supporting both Fabric and Neoforge, and picked the 1.21.1 Minecraft version, though in the future I’m planning to move to 26.2 or later. I set up an additional gradle subproject and a rust package, that both together act as a Java to Rust bridge, using the j4rs crate. Using it I managed to initialize Sepple (the detection system), and get it running. The tricky part was going the other direction, to pass detected words from Rust to Java. For some reason Rust wasn’t detecting the JVM and was trying to spawn a new one! After a bit of experimentation I concluded, that I can only call Java from Rust only during the period when Java calls Rust, and only on the same thread. In the end I managed to get it working, by spawning a new thread on the Java side, and calling a blocking Rust function, that kept calling a Java callback passed to it. Originally I tried to use a static function as a callback, but for some reason (probably some kind of classloading isolation) the system was failing with some crazy null errors. I also had to adjust Sepple a bit, to not require a bunch of random json files at specific paths, and to provide an mpsc chanell directly outside its pipeline, for less boilerplate. With that I have a proof of concept working! Minecraft is receiving the detected words and printing them to chat. Note that those are all custom words, created by me specifically for this mod. This is why I had to create an entire new system for detecting them, and why they are written in the International Phonetic Language, which is also why they look a little weird in the minecraft’s font. The Java<->Rust setup is also not very good yet. For example the packaging of compiled Rust is very basic, and you can’t control the word detection from Java - no way to stop it or restart. There is also no good error handling, which means if a file is missing it will just crash everything. But it works, and I can continue working on the mod, and making it better!"}, {"post_id": 36043, "ts": "2026-07-27T21:21:56Z", "hours": 10.1719, "text": "About project Sephyr is a magic mod, with spell casting powered by speech live translated into International Phonetic Alphabet, to detecting custom-made spell words in it. Adding Voice Activity Detection - part 2 After the refactor I was able to finally add the VAD. But after few tests I noticed that the VAD was making things worse, not better! That was as a part of the previous devlog (which ran out of character limit). I added a pipeline part that writes every chunk of audio that passes through it to a file, so it can be inspected. With it I found the main problem: It took few hundred milliseconds before the VAD detected speech, so a small chunk was always cut off from the start. To fix this I added context keeping: A bunch of previous chunks are kept, and when speech is detected they are also taken into consideration. I also made the sliding window chunking aware of VAD - the VAD detector sends a signal when it detects an end of speech. The chunker then flushes its buffer into the pipeline, so that it doesn’t get stuck until next speech, and replaces it with precise amount of silenece. This way the start of next audio doesn’t get cut off later due to the sliding window mechanism. The author of the bunsen crate (providing the VAD model implementation) also released version 0.29.0, which fixed all the issues I encountered in the last devlog, and updated its API slighly, which meant that after adjusting my code I could stop using my own fork of it. Improving word detection from IPA My friend found a crate named phonetics-rs and suggested using it to improve the algorithm. This crate calculates the difference between two IPA strings, taking into account how similar they sound. This is much more accurate than just comparing strings on grapheme level. I modified my algorithm to use it, and it turned out to be a great idea, that significantly improved the detection rate of magic words. I also changed the dictionary to be sorted by longest words first, so that they take priority over short ones. So far, testing and improving the word detection algorithm has been the most fun out of the entire project. Looking at IPA strings, and figuring out why some speech has or hasn’t been classified as some word is quite fascinating. For example, I encountered a string pɾilifɛra which the model output after me saying prizim and fɛra , the two magic words that I made up for the spell system. Surprisingly, my algorithm said the string contained just prizim despite there being a literal fɛra in it. As I found out, pɾilif is closer then pɾili to prizim , and it fell under the threshold, so the algorithm took that part and left out ɛra in the buffer which was too far from fɛra . After seeing this, I made the algorithm scan the string for exact matches first, and then do fuzzy phonetic matching, also ensuring the order is preserved. I also tuned the thresholds, adding more and more tests (there is now 8 of them) to ensure the algorithm behaves well. Finally, I noticed the IPA output from the model changes based on audio’s offset in time. So I wrote a script that lets me test this, and see the effect of different offsets. What’s next There is a saying in data science: garbage in - garbage out. This is also true for my project. The detection works decently, but it also quite often outputs false positives during normal speech. I believe I did I all could to compensate for the unsatisfying quality of the speech to IPA translation model. Now I will be shifting to the Minecraft part of the project - creating a mod, integrating the detection system into it and creating spells. In paralell, I will be exploring the possibility of finetuning the model, to improve the accuracy of it and hopefully get better results. Watch the video with audio, that’s the part that matters right? :)"}, {"post_id": 27916, "ts": "2026-07-13T20:52:15Z", "hours": 30.4097, "text": "Finding words in long IPA strings For the magic system I need to find individual words in the ever accumulating string of IPA characters. To do that I hold a list of valid words in memory. When there is some text, I iterate over the dictionary, calculating a levenshtein distance between each word and start of the string. Then the word with the lowest score is taken and if the score is under a set threshold, the word is considered a match, and the matching part is remved from the buffer. To cover more cases different lengths of the string are checked. If no match is found, one character is removed from the string, and the process repeats, until all words are found. Fun fact: When comparing IPA and other non-ascii strings you need to make sure you are considering graphemes and not characters, because in UTF-8 what we perceive as one symbol, can be encoded as many characters in memory. So far this algorithm works quite well, though in the future it might need some performance optimization, or tuning, if it starts yelding too many false positives. Other misc changes: Migrated main fn to use clap for parsing arguments Added flake.nix with dev env so I can run this code on my NixOS laptop Added time information to the pipeline - Together with the audio from the mic, a timestamp is captured, and passed through entire pipeline up to the end, where it is used to discard text older then 200ms from the buffer, so that stale data doesn’t mess up new words. Adding VAD I noticed that the speech to IPA model likes to output random data when no one is talking, so I decided to add VAD - Voice Activity Detection - a system that detects if someone is talking, and only then passes the audio on. I estimated this to take me around 2 hours, and oh, how wrong I was? It took 10x that, which is why this devlog documents 3x the normal amount of time. And it still isn’t fully ready! So let me tell you what I did and why it took so long. In the Burn (ML/AI lib for Rust) discord I noticed a solution just dropped out of the sky: Merely few days before a Silero VAD model implementation was added to a library named bunsen . ML models take input as Tensor s - multidimensional arrays, which do not say in what form or shape the data is expected. So I had to figure that out myself. Fortunetely it was just 512 elements long chunks of 16khz audio with no additional pre-processing. Knowing that I fed the microphone input to the model and began to watch it output complete nonsense unrelated to whether I was speaking or not. So I started debugging - checked my code, wrote an equivalent in python using the official silero implementation to compare results - completely different. So I cloned the bunsen library, wrote an unit test there and ran off a Opencode agent. I also contacted the library author. He found one major issue, and the agent identified a second problem and wrote a solutuion - Silero VAD expects 64 samples of the previous chunk to be passed along the 512 to work correctly. I cleaned up the code, which was quite a pain because rust-analyzer was super slow on that repository, and made a PR, but it wasn’t merged because the author went with his own way of implementing it. The big refactor Coming back to my project I realized that my code strucutre makes it very hard to integrate VAD into it. So I spent the next 6 hours of so refactoring almost every file in the codebase into a new pipeline system. It lets you compose a pipeline out of a producer, many processors, and a consumer. It makes use of async, which saves on the amount of threads used. Static type checking ensures you can’t compose parts that wouldn’t work together, and all elements can also define contracts on the size of data chunks they produce and consume. I ran out of the character limit, so the rest of VAD story will be in next devlog."}, {"post_id": 11428, "ts": "2026-06-18T18:11:53Z", "hours": 4.8753, "text": "Live speech to IPA transcription in Rust using ML I finished my rough implementation of a live speech transcription pipeline. It captures my speech from the mic, and processes it with the Multipa model using the Burn framework. To make it work live I implemented a sliding window that chunks the audio into 2 second fragments, each of which is processed separately, and then assembled into a single string. The chunks overlap, and are then sliced, so that audio that was on the borders of a cut (which will reduce the quality) is thrown away, and the text for it is taken from the next chunk, for which it is at the center. The inference is running on cpu for simplicity and compatiblity, so by sacraficing a bit of pain I added threading (where each chunk is processed on a separate thread) to make sure the pipeline can keep up. I’m slightly worried about how the perfromance will be when this is going to run along Minecraft, but hopefully I will be able to make it as lightwieght and unintruisive as possible. Finally I reworked how the text is accumulated, switching from a vector of logits to a string. Since IPA has no concept of words, just phonemes, everything I say is appened to one long string. So I added a very simple silence detection, where if a captured chunk returns no text a space is added to the string. For easier teasting I also improved the main function, so that the program can: take a file and process it all at once take a file and process it using the pipeline run live mic capture on the pipeline What’s next - improving the detection In theory this is all I need, and I can start working on the Minecraft mod. However I’m not satisfied with the quality of the transcription, and at the level it currently is most spell detection will probably fail. The quality of the model itself is pretty bad. If the description on github of the Multipa model is correct, the model was trained on merely 9 hours of speech data, which probably explains it. The model is also sensitive to noise, and distractions like other people talking, and sometimes outputs garbage. So the next devlog I will be experimenting with various techinques of filtering and cleaning up the audio the model is receiving, and potentially even training a new model myself on a bigger dataset. Attached video Working demo of live capture and transcription of me talking."}, {"post_id": 9209, "ts": "2026-06-15T13:48:33Z", "hours": 3.7658, "text": "Sephyr - project overview Sephyr is an unique magic mod for Minecraft. It aims to differentiate itself by: Introducing a voice powered spell casting system , that’s both fun to use and intricate. You will be reading spells out loud into your mic in a magic language, and what will you say will affect the result of the spell. Having stunning, detailed visuals unique to each spell. This is going to be achieved by spending way too much effort on rendering. I want spell casting to be something satisfying, and the look is a key part of it. The project consits of two parts: Sephyr - The minecraft mod itself containing the spells, mechanics and visuals, written in kotlin. Sepple - Speech processing engine, responsible for transforming your babble into something Sephyr can work with and figure out what have you just said. Sepple overview Sepple is a foundation of the voice powered spell casting. It is an audio processing library written in rust, that Sephyr will depend on and embed. It captures your mic live and pre-processes the speech. Then it uses the Burn framework to locally and quickly run a speech recognition model called Multipa . The model transcribes speech into the International Phonetic Alphabet (IPA), a textual notation of various sounds that make up the spoken language. Finally it tries to match the result against a custom dictionary of magic words, which will form sentences - the spells you will be casting. This devlog Before Stardance launched I have written a rough skeleton for Sepple. With some trouble I managed to import the Multipa model to Burn via ONNX export from Pytorch, and implement the required post-processing to decode model’s output into IPA. I also started working on a pipeline for mic capture and sliding window processing, so that the audio can be transcribed live. After Stardance started I noticed that the same input audio file gives quite different result in Pytorch then in Rust. I spent few hours debugging why that is, and found a missing piece in my implemenation. For the Multipa model to work correctly the samples need to be normalized using a so-called z-score normalization. That means calculating the mean of the input and its standard deviation (a statistics measure), and applying a formula that rescales the data, so that it has a mean of 0 and standard deviation of 1. After implemenating that the model started to work correctly and give the same output for the same input on both sides. Picture: “Hello Stardance. Sepple is working now” translated to IPA using pytorch and the fixed Rust implementation. It is not veeery accurate, but it is good enough."}]...
|
Title: Sephyr Magic mod for Minecraft by @Szczurek Title: Sephyr Magic mod for Minecraft by @Szczurek | Stardance
Devlogs:
Sephyr - project overview Sephyr is an unique magic mod for Minecraft. It aims to differentiate itself by: Introducing a voice powered spell casting system , that’s both fun to use and intricate. You will be reading spells out loud into your mic in a magic language, and what will you say will affect the result of the spell. Having stunning, detailed visuals unique to each spell. This is going to be achieved by spending way too much effort on rendering. I want spell casting to be something satisfying, and the look is a key part of it. The project consits of two parts: Sephyr - The minecraft mod itself containing the spells, mechanics and visuals, written in kotlin. Sepple - Speech processing engine, responsible for transforming your babble into something Sephyr can work with and figure out what have you just said. Sepple overview Sepple is a foundation of the voice powered spell casting. It is an audio processing library written in rust, that Sephyr will depend on and embed. It captures your mic live and pre-processes the speech. Then it uses the Burn framework to locally and quickly run a speech recognition model called Multipa . The model transcribes speech into the International Phonetic Alphabet (IPA), a textual notation of various sounds that make up the spoken language. Finally it tries to match the result against a custom dictionary of magic words, which will form sentences - the spells you will be casting. This devlog Before Stardance launched I have written a rough skeleton for Sepple. With some trouble I managed to import the Multipa model to Burn via ONNX export from Pytorch, and implement the required post-processing to decode model’s output into IPA. I also started working on a pipeline for mic capture and sliding window processing, so that the audio can be transcribed live. After Stardance started I noticed that the same input audio file gives quite different result in Pytorch then in Rust. I spent few hours debugging why that is, and found a missing piece in my implemenation. For the Multipa model to work correctly the samples need to be normalized using a so-called z-score normalization. That means calculating the mean of the input and its standard deviation (a statistics measure), and applying a formula that rescales the data, so that it has a mean of 0 and standard deviation of 1. After implemenating that the model started to work correctly and give the same output for the same input on both sides. Picture: “Hello Stardance. Sepple is working now” translated to IPA using pytorch and the fixed Rust implementation. It is not veeery accurate, but it is good enough.
Live speech to IPA transcription in Rust using ML I finished my rough implementation of a live speech transcription pipeline. It captures my speech from the mic, and processes it with the Multipa model using the Burn framework. To make it work live I implemented a sliding window that chunks the audio into 2 second fragments, each of which is processed separately, and then assembled into a single string. The chunks overlap, and are then sliced, so that audio that was on the borders of a cut (which will reduce the quality) is thrown away, and the text for it is taken from the next chunk, for which it is at the center. The inference is running on cpu for simplicity and compatiblity, so by sacraficing a bit of pain I added threading (where each chunk is processed on a separate thread) to make sure the pipeline can keep up. I’m slightly worried about how the perfromance will be when this is going to run along Minecraft, but hopefully I will be able to make it as lightwieght and unintruisive as possible. Finally I reworked how the text is accumulated, switching from a vector of logits to a string. Since IPA has no concept of words, just phonemes, everything I say is appened to one long string. So I added a very simple silence detection, where if a captured chunk returns no text a space is added to the string. For easier teasting I also improved the main function, so that the program can: take a file and process it all at once take a file and process it using the pipeline run live mic capture on the pipeline What’s next - improving the detection In theory this is all I need, and I can start working on the Minecraft mod. However I’m not satisfied with the quality of the transcription, and at the level it currently is most spell detection will probably fail. The quality of the model itself is pretty bad. If the description on github of the Multipa model is correct, the model was trained on merely 9 hours of speech data, which probably explains it. The model is also sensitive to noise, and distractions like other people talking, and sometimes outputs garbage. So the next devlog I will be experimenting with various techinques of filtering and cleaning up the audio the model is receiving, and potentially even training a new model myself on a bigger dataset. Attached video Working demo of live capture and transcription of me talking.
Finding words in long IPA strings For the magic system I need to find individual words in the ever accumulating string of IPA characters. To do that I hold a list of valid words in memory. When there is some text, I iterate over the dictionary, calculating a levenshtein distance between each word and start of the string. Then the word with the lowest score is taken and if the score is under a set threshold, the word is considered a match, and the matching part is remved from the buffer. To cover more cases different lengths of the string are checked. If no match is found, one character is removed from the string, and the process repeats, until all words are found. Fun fact: When comparing IPA and other non-ascii strings you need to make sure you are considering graphemes and not characters, because in UTF-8 what we perceive as one symbol, can be encoded as many characters in memory. So far this algorithm works quite well, though in the future it might need some performance optimization, or tuning, if it starts yelding too many false positives. Other misc changes: Migrated main fn to use clap for parsing arguments Added flake.nix with dev env so I can run this code on my NixOS laptop Added time information to the pipeline - Together with the audio from the mic, a timestamp is captured, and passed through entire pipeline up to the end, where it is used to discard text older then 200ms from the buffer, so that stale data doesn’t mess up new words. Adding VAD I noticed that the speech to IPA model likes to output random data when no one is talking, so I decided to add VAD - Voice Activity Detection - a system that detects if someone is talking, and only then passes the audio on. I estimated this to take me around 2 hours, and oh, how wrong I was? It took 10x that, which is why this devlog documents 3x the normal amount of time. And it still isn’t fully ready! So let me tell you what I did and why it took so long. In the Burn (ML/AI lib for Rust) discord I noticed a solution just dropped out of the sky: Merely few days before a Silero VAD model implementation was added to a library named bunsen . ML models take input as Tensor s - multidimensional arrays, which do not say in what form or shape the data is expected. So I had to figure that out myself. Fortunetely it was just 512 elements long chunks of 16khz audio with no additional pre-processing. Knowing that I fed the microphone input to the model and began to watch it output complete nonsense unrelated to whether I was speaking or not. So I started debugging - checked my code, wrote an equivalent in python using the official silero implementation to compare results - completely different. So I cloned the bunsen library, wrote an unit test there and ran off a Opencode agent. I also contacted the library author. He found one major issue, and the agent identified a second problem and wrote a solutuion - Silero VAD expects 64 samples of the previous chunk to be passed along the 512 to work correctly. I cleaned up the code, which was quite a pain because rust-analyzer was super slow on that repository, and made a PR, but it wasn’t merged because the author went with his own way of implementing it. The big refactor Coming back to my project I realized that my code strucutre makes it very hard to integrate VAD into it. So I spent the next 6 hours of so refactoring almost every file in the codebase into a new pipeline system. It lets you compose a pipeline out of a producer, many processors, and a consumer. It makes use of async, which saves on the amount of threads used. Static type checking ensures you can’t compose parts that wouldn’t work together, and all elements can also define contracts on the size of data chunks they produce and consume. I ran out of the character limit, so the rest of VAD story will be in next devlog.
About project Sephyr is a magic mod, with spell casting powered by speech live translated into International Phonetic Alphabet, to detecting custom-made spell words in it. Adding Voice Activity Detection - part 2 After the refactor I was able to finally add the VAD. But after few tests I noticed that the VAD was making things worse, not better! That was as a part of the previous devlog (which ran out of character limit). I added a pipeline part that writes every chunk of audio that passes through it to a file, so it can be inspected. With it I found the main problem: It took few hundred milliseconds before the VAD detected speech, so a small chunk was always cut off from the start. To fix this I added context keeping: A bunch of previous chunks are kept, and when speech is detected they are also taken into consideration. I also made the sliding window chunking aware of VAD - the VAD detector sends a signal when it detects an end of speech. The chunker then flushes its buffer into the pipeline, so that it doesn’t get stuck until next speech, and replaces it with precise amount of silenece. This way the start of next audio doesn’t get cut off later due to the sliding window mechanism. The author of the bunsen crate (providing the VAD model implementation) also released version 0.29.0, which fixed all the issues I encountered in the last devlog, and updated its API slighly, which meant that after adjusting my code I could stop using my own fork of it. Improving word detection from IPA My friend found a crate named phonetics-rs and suggested using it to improve the algorithm. This crate calculates the difference between two IPA strings, taking into account how similar they sound. This is much more accurate than just comparing strings on grapheme level. I modified my algorithm to use it, and it turned out to be a great idea, that significantly improved the detection rate of magic words. I also changed the dictionary to be sorted by longest words first, so that they take priority over short ones. So far, testing and improving the word detection algorithm has been the most fun out of the entire project. Looking at IPA strings, and figuring out why some speech has or hasn’t been classified as some word is quite fascinating. For example, I encountered a string pɾilifɛra which the model output after me saying prizim and fɛra , the two magic words that I made up for the spell system. Surprisingly, my algorithm said the string contained just prizim despite there being a literal fɛra in it. As I found out, pɾilif is closer then pɾili to prizim , and it fell under the threshold, so the algorithm took that part and left out ɛra in the buffer which was too far from fɛra . After seeing this, I made the algorithm scan the string for exact matches first, and then do fuzzy phonetic matching, also ensuring the order is preserved. I also tuned the thresholds, adding more and more tests (there is now 8 of them) to ensure the algorithm behaves well. Finally, I noticed the IPA output from the model changes based on audio’s offset in time. So I wrote a script that lets me test this, and see the effect of different offsets. What’s next There is a saying in data science: garbage in - garbage out. This is also true for my project. The detection works decently, but it also quite often outputs false positives during normal speech. I believe I did I all could to compensate for the unsatisfying quality of the speech to IPA translation model. Now I will be shifting to the Minecraft part of the project - creating a mod, integrating the detection system into it and creating spells. In paralell, I will be exploring the possibility of finetuning the model, to improve the accuracy of it and hopefully get better results. Watch the video with audio, that’s the part that matters right? :)
Integrating Rust IPA word detection with Minecraft With the IPA based magic keyword detection (which is written in Rust) working I spent all of this devlog integrating it into Minecraft. I created a new repo for the mod, using the Jared’s template. I decided to go with supporting both Fabric and Neoforge, and picked the 1.21.1 Minecraft version, though in the future I’m planning to move to 26.2 or later. I set up an additional gradle subproject and a rust package, that both together act as a Java to Rust bridge, using the j4rs crate. Using it I managed to initialize Sepple (the detection system), and get it running. The tricky part was going the other direction, to pass detected words from Rust to Java. For some reason Rust wasn’t detecting the JVM and was trying to spawn a new one! After a bit of experimentation I concluded, that I can only call Java from Rust only during the period when Java calls Rust, and only on the same thread. In the end I managed to get it working, by spawning a new thread on the Java side, and calling a blocking Rust function, that kept calling a Java callback passed to it. Originally I tried to use a static function as a callback, but for some reason (probably some kind of classloading isolation) the system was failing with some crazy null errors. I also had to adjust Sepple a bit, to not require a bunch of random json files at specific paths, and to provide an mpsc chanell directly outside its pipeline, for less boilerplate. With that I have a proof of concept working! Minecraft is receiving the detected words and printing them to chat. Note that those are all custom words, created by me specifically for this mod. This is why I had to create an entire new system for detecting them, and why they are written in the International Phonetic Language, which is also why they look a little weird in the minecraft’s font. The Java<->Rust setup is also not very good yet. For example the packaging of compiled Rust is very basic, and you can’t control the word detection from Java - no way to stop it or restart. There is also no good error handling, which means if a file is missing it will just crash everything. But it works, and I can continue working on the mod, and making it better!
Sephyr (the mod) A proper lifecycle of the detection system was added. The IPA model is loaded in background during game start, but the audio capture and processing start waits until you join a world, and stops once you leave it. The Rust code now has a logging hook, so that it can do civilized logging using the Java logger, instead of blatantly writing to stdout. Sepple (the voice casting detection system) I made the pipeline system cancellable, so that the mod can cleanly stop the detection, which was not possible before. Support for running the speech detection inference on GPU was added as a feature flag. For now it is not used by default, as it showed no measurable improvements over running on CPU on my PC, but it might be useful on other, weaker devices. Collecting data for training As I said too many times, the speech to IPA model used for magic speech detection is not quite the quality I would like it to be. I’m hoping to improve it by fine-tuning it on its weak areas, but for that I need some data. My idea is to use wiktionary as a data source. It is a free online dictionary run by the same people who run Wikipedia. Most words on it have the pronunciation attached both as an IPA and an audio file, which makes it ideal for the task. I downloaded a preprocessed jsonl dump file from kaikki and wrote a script to extract just words and ipa from it. Unfortunetely it was missing urls of the audio files. So I had to download a raw xml dump of the entire english wiktionary, and spend few hours writing a script to extract them. Then I used deepseek a generate a third script, to download all the audio files from a list. There is going to be almost 20k of them, and the script is running right now as I’m writing this, and it is going to take multiple hours as I can only download around 1 file per second to respect rate limits and not overburden wikimedia. And just now while writing I realized I’m an idiot, because kaikki provides a 20GB tar file with all the audio, downloading which would be more efficient.
Note: This one is kinda boring, mostly plumbing stuff to get things to work, feel free to skip it. Sephyr (the mod) I’m working on setting up the the logic for registering and casting spells. This is quite hard, as the design of the mod is far from complete, which means I don’t even know what exactly do I need. I created a registry of elements, which are a source of a color, and are going to have other effects in the future. The first word of a spell is its element. For example in the spell ʔalɪˈvɑn vɛɾtɛɾ ɛvi the word ʔalɪˈvɑn means wind. I also made a basic system for figuring out the spell the player is casting. For now its very simple and can’t handle more complex spells, for example, with optional words. The entirety is a big mess as I’m figuring things out, and the fact I’m making this mod for both Neo and Fabric doesn’t make it any easier. I also updated the mod to Minecraft 26.1.2, which means there is going to be no 1.21.1 release. Sepple (word detection system) I keep working on collecting data to finetine the speech to ipa ml model. After discovering a better way of getting the audio files while writing last devlog I scraped the download script, and instead replaced it with a new one to extract needed files from the 20GB Wiktinary audio dump tar. I also cleaned up the entire code, turning it back to human standards, after a bit of Opencode usage. (Since those are one-time scripts to extract data I decided using a bit of AI in this area won’t hurt. I don’t use AI for the rest of the project.). I also prepared a fourth script, that will use the existing multipa model, to generate IPA transcription of the files. Then files with generated IPA being too different from original IPA will be discarded, as a way to detect files, where the IPA doesn’t match the audio (because the audio contains more text for example).
SPELLCASTING WORKS! The main mechanic of the mod is working! YOU CAN CAST SPELLS WITH VOICE! Finally! It took soo long… After over two months of development, I finally have a proof, that my crazy idea actually works, and was worth trying out. As you can see (and hear) in the video I implemented a simple wind leap spell, as the first one in the mod. Note: There are no visuals or sounds yet, but they will be added, and this is just a proof of concept. The spell comes in three variants ʔalɪˈvɑn vɛɾtɛɾ ɛvi ˈonːdˌoː - wind launches you forward ʔalɪˈvɑn vɛɾtɛɾ ɛvi ãntikɔ - wind launches you backward ʔalɪˈvɑn vɛɾtɛɾ ɛvi fɒksɑm - wind launches you up (no matter where you are looking), not shown in the video Additionaly, it can be cast as just ʔalɪˈvɑn vɛɾtɛɾ ɛvi , which is a shortcut for the first variant. This can seen in the third attempt, where the system didin’t catch onːdˌoː , but the spell worked anyway. This flexibility is easily possible, thanks to Mojang’s brigadier library. Brigadier powers the / commands in vanilla, and now the spell syntax and execution in my mod. I settled on it after trying two alternative approaches and failing to make them work. After all, my spells are just sentences, with the only difference, is that you speak them instead of writing.
Wind spells have sound now Using LMMS and ZynAddSubFx I made two wind sound effects, which now play when casting a wind spell. You can hear them in the attached video. Cross-Compiling is fun The speech recognition system is written in Rust, which compiles down to native code. This means that it needs to be compiled separately for each operating system, and processor architecture. Rust is more friendly to it, then C or C++ for example, but as always, compilation finds it own weird ways to break. I took the setup used by Sable as a base, with two Docker containers, one for Linux and Mac, and one for Windows. BUT: The Rust version installed in the docker image was not new enough for one of my dependencies - I had to update the rust toolchain in the container image. Compiling for linux required ALSA headers for audio access - I had to figure out, how to install them in a way, that works for both x86_64 and arm64, without breaking each other. The latest version of the base image has Mac cross compilation broken - I had to use a previous one. The image ships a MacOS SDK from 5 years ago - too old, doesn’t work, so it also had to be replaced. - the container is kinda becoming a Ship of Theseus xD My code was pulling openssl as a dependency, which was failing to compile - I had to spent over an hour, figuring out how to get rid of it from the dependency tree, discovering I can do so by updating a dependency to an unreleased version. Gradle obviously required some tweaks, who doesn’t love Gradle right? In the end I managed to get it working, which brings the mod closer to being usable by the public, outside dev environment. Downloading the model The mod works by running an IPA speech recognition model locally. The model weights around 1.2G, making it too large to be embedded in the mod jar. So I added automatic download support - the model can now be downloaded automatically on the first time an user runs the mod. There is even progress tracking, though it’s not hooked up to Minecraft yet. A Toast is also shown when the loading fails, making the mod more user friendly.
The Stack Overflow For some reason my mod broke and started crashing during startup, and it did so in the worst possible way - without any error message! All I got in the Intellij Idea, was a message that the process exited with the signal code 11. WHAT? That signal corresponds to SIGSEGV signal on Linux, which generally means you did something wrong with the memory. But this is Rust! There shouldn’t be any memory issues! I started searching, and I learned there is a tool called coredumpctl, which lets me access a special dump file generated in such cases, and analyze it in a debugger like GDB. So I did that, and what I found was even more bizzare! The debugger was pointing to a start of a function, before the first line. I added some debug prints, and to my disbelief, the print before calling that function ran, and the the second one, which was the first line inside the function didn’t! Fast forward a bit I managed to figure out the problem, with some help from a friend. It turned out I encountered an stack overflow! When the function was called, Rust tried to expand the stack, and crashed because the stack ran out! Why? Because the code was loaded in Java, and Java by default allocates less memory for the stack ! The solution was manually requesting a bigger stack size during thread creation in Java. I also discovered, why the problem appeared - in the Rust file download code I had allocated an array buffer of 1MB. It landed on the stack, and that was too much! The solution was using a Vec, which lives on the heap instead. Stupid problem, but took 3 hours of my time 😩 Mod init progress reporting Sephyr requires the IPA speech recognition model to work, which needs to be downloaded on the first run, and subsequently loaded from the disk. The logic was overhauled providing more corectness, and hooked up to a series of toasts. File download progress bar is shown, and the intialization state (or failure) is coummunicated to the user! See it in the video
Visualizing the keyword detection pipeline I’m not happy with the accuracy of the magic words detection system. It work half of the time, sometimes better, sometimes worse depending on the word. This make casting longer spells quite troublesome. Debugging why a specific word wasn’t detected is hard, debug prints can only get you so far. So I decided to go on a crazy sidequest and build a gui tool, that will help me debug this. I spent over 2 hours designing the concept, and over 10 implementing it, and it’s around 40% ready. The tool will display 8 strips / plots, and each plot will display some data, all on a shared timeline on the bottom, that will update live as the audio is processed. Strips 1, 2, and 4 will display waveforms of the audio at different processing stages. Strip 3 will show Voice Activity Detection score (which is used to filter out non-speech). Strips 5, 6 and 7 will display detected IPA tokens, accumulated text buffer, and detected words respectively. Strip 8 will display latency stats, which will help in optimizing it. This way I will be able to easily see, “oh, this phoneme wasn’t detected” or “oh, VAD cut out this word” or “oh, some mess accumulated in the buffer” and tweak my algorithms. AI - it’s bad, or is it? Despite looking quite simple on the surface, the app is still quite an undertaking. And it’s just a debugging tool! A thing to help me get the main goal done! I don’t want to spend too much time on it. So I considered using AI as a tool ( NOT vibecoding ). I hate vibecod(ing, ers), and I’ve always steered off AI, because very often I wasn’t satisfied with the output quality. I take pride in writing good code. But recently I’ve been trying out OpenCode here and there (on free models), and was quite surprised with it’s capabilities. Sure, it’s still often slop, but a one you can quite quickly correct, and sometimes be more effective if used correctly. AI - is it helpful? So I decided to do an experiment - can carefully planned AI usage help me write the debugging tool quickly? I started off by designing the functionality of the tool - I described the idea in detail to MiMo-V2.6-Flash, and asked it to interrogate me about as many details as possible and needed. I had to answer 35 questions, some of which were quite observant, picking me on nooks and crannies. After that a markdown file describing the design was written, and then went through few corrections. Next up, in a new session I asked MiMo to create a bullet point outline of steps needed to be done to implement the tool specified in the design file. Then one by one I went over each bullet point, asking the just released LongCat 2.5 (which has a bigger ctx window) to ask me questions, and formulate an implementation plan of it, present it to me, I gave corrections, and the point in file was replaced. This way I ended up with a second markdown file - an implementation plan. This took way too long, and was really boring. So, finally coding time. I do it like this: Ask opencode to implement a step of the plan Review the diff in vscode Remove fluff and slop manually, make sure all new code is good quality. Give feedback if I something doesn’t work like expected, or also fix it manually. After it’s good - commit Repeat for the next step It’s still not super fun, but at least I’m touching the code now, so much better already. And it is efficient! I believe it definitely would take me longer, if I wasn’t using AI. So the morale is: If you want fun, or doing something important - do it manually. If you want to do something quickly and get it out of the way less important. Here is a screen of how it is looking now: the general shape is in place, but there is no plots yet.
Keyword detection UI is done Look as this! It’s so pretty! Seeing all the data visualised is quite mesmerizing. I know the image will probably attract more views, so lemme explain what you are looking at. I’m making a minecraft mod where you cast spells by saying custom made up words, and this is a tool for debugging the keyword detection system. It’s meant for me, to help me make the detection more accurate, and reliable. I’ve been building it for the last ~20 hours, and in the last two I built the rendering of all the 8 strips. Check previous devlog if you want to know what exactly they display. It was pretty smooth sailing. As said before, since this is just a debugging tool for me, that I want to have finished quickly for it I’m using AI assistence, by letting it do stuff in small, pre-planned steps, and unsloppifying all the code between them. It’s working well. What helps, is that entire app is built on the egui / eframe crates, which I used before, and are somewhat familiar with. I gotta give credit, where is due, their api is really intuitive for the most part, and simple to use. The entire chart is also zoomable, and scrollable, and making this work was a bit of pain. I wanted to limit the accesible range to only around the data (so you can’t scroll away to far from it), and egui_plot doesn’t really provide a way to do it, so some manual calculations had to be done. I wanted to have some breathing space by allowing 0.25x of the zoom after last audio sample to be visible. This was the worst part, as it’s kind of a recursive problem - the zoom value affects how much zoom is allowed. But after around two hours I manged to figure it out, but keeping the original zoom value independantly of the limit calculated from it. Worth noting - what you see is just a mockup data, and the gui isn’t actually hooked up to the detection system, as that’s what I’m going to do next. After that I can hopefully improve the detection, and the finally go back to minecraft coding.
Debugging gui is (almost) done What is going on, why already 10 hours passed? This was menat to be a quick sidequest, where I built this gui tool for debugging how well detecting magic keywords is working BUUT given it is the third devlog about it That means it took over 30 hours! Well, happens. So, all of this devlog was spent building all the plumbing needed to capture all the data from the detection pipeline and sending it in realtime to the app. All of it is using done using channels. They are injected into the detection pipeline either as Tap<T> ‘s - a new pipeline element that clones data it sees, sends to channel, and passes it on, or using wrapper types around existing processors. The app then receives the data and displays in on the plot. I also encountered and fixed few bugs, that I didn’t expect. For example originally the zoom went to crazy values, because a timestamp used to determine where the data ends wasn’t converted to be relative to the timeline start. I also found an quite an issue introduced 3 months ago, where the timestamps attached to audio weren’t calculated properly, making entire audio chunks have identical stamps. It was minor enough that I never noticed till now, when the data landed on the plot and resulted in this funny thing you can see on the attached screenshot. Finally I added a simple scrolling animation, to amortize the jitter caused by bigger chunks of audio coming, so the plot doesn’t jump that much when going live, but it’s still not as smooth as I would like it to be. As mentioned before I was using AI assistance in strictly controlled manner (I carefully planned, revieved, cleaned up and adjusted to my liking all the code generated by it) to build this debugging tool faster, and by now I’m very tired of it. At the end of the work it started doing stupid things (possibly because some context was lost?), which made me question if it was making me faster or slower. Luckily the work is mostly over, and I can finally get to other, more interesting stuff, that will be fun to do, with no AI. Next I want to use the tool to improve the detection a bit, and spend more time working on the actual mod.
Multiloader modding is pain Can’t NeoForge and Fabric just have one common api, for all the boring stuff like registration, networking and events? I’ve had enough, so I spent 2 hours porting my mod to Balm , which is a library providing exactly that. The result is beautiful - ~150 lines are gone, and the loader specific subprojects fit under 10 lines of code, as I was able to move everything to common. Now for the worst irony: I spent my time doing all of that and the next thing I’m gonna do is… disable fabric support 😭 Why, you may ask? I was forced into it by reality read on First VFX I made my first VFX, for the leap spell. You can see it in the attached video. It’s going to spawn under player’s legs. For now it’s a really simple one, as I’m rushing to add a bit more of content, than just a single spell, before Stardance ends. It is made using Photon Editor , which is a VFX editor for Minecraft. It has a lot of good stuff, like emitters, timeline and animation curves, which will be crucial in making more advanced effects. Only downside: It doesn’t support Fabric, which is why support for it is going to be disabled for now, until I manage to hopefully make a port it in the future. Reworking spell internals (again) After making the effect, I realized I have no way to run it from the spell code, because spell logic runs server side only, and VFX needs to be played on the client. So I wen’t back to the drawing board, and I’m going to change how spells run. They are going be entities now, but more on that next devlog, because it’s not done yet, as I ran into friction in the code, which triggered me to do the Balm refactor, and in the meantime the time counter hit 9,5h. Do NOT hardcode stuff please I had to do some work on my laptop today, which has NixOS. So I update my Sephyr clone, try to run it, and bada bing bada bum, it doesn’t work! What’s going on? After a hour of debugging, it turned out that NeoForge devs hardcoded their Gradle plugin to run one specific task using Java 21. The rest runs on Java 25 (the one used by Minecraft 26.1). So, be default, a NeoForge setup for Mc 26.1+ requires two java version. Normally the second one is autodownloaded, but that doesn’t work on NixOS, and that’s also bloat. I don’t want that! So I made a bug report . Fortunetely it isn’t really hardcoded, just dumb, and I was able to fix it with this line in build.gradle Update to Burn 0.22 Burn 0.22 was recently released, removing the Backend generic from most of it’s traits. I updated Sepple to work with that. I also had to regenerate the model code for the multipa model. I was hoping the update would provide some speedup to speech processing, but I wasn’t able to measure any difference. Replacing Multipa with Zipa I found a new speech-to-IPA model. It’s called Zipa , and it looks very promising. It’s around 4x smaller then multipa, which means faster processing and less to download for end users. Furthermore, according to it’s paper, and my few tests it has noticeable better accuracy! So I implemnted it. It was a little more trouble then multipa, because the model has a much richer pre-processing. The audio waveform is turned into a mel spectogram, Fourier Transform is done, and other stuff happens. Buuut there is a problem. It turns out the model is great at doing recognition on longer audio sequences, and doesn’t really work that well on short ones. For them it performed worse then multipa! That means it’s not suitable for doing recognition in realtime, at least when plugged in the same way multipa was. However I have a couple ideas to try, which might, hopefully, solve the issue. Until I do that, it is set aside on a separate branch.
Leap VFX and new spells The leap spell now has the simple VFX made last devlog attached. This is now possible thanks to the changes described below. It’s ugly, it usually just clips into the ground, but it’s better then nothing. I also added a new spell - ʔalɪˈvɑn wˈanga - wind shot . It shoots a wind projectile, for now - a wind charge, in the future - a custom one. You can see both in attached video. I also started working on a third spell - wind beam, but I wasn’t able to finish it this devlog. Spell entites The casting code rework is now done. After struggling for quite a lot of time, I managed to implement a SpellEntity , and rework how spells are built and registered. Let me tell you why it was hard. Each spell is unique, with it’s own set of modifiers that translate to unique set of data that the spell needs to run. That data is needed both on the server, to run the logic, and on the client to display VFX. That meant I needed to find a way, to serialize the data for the client, and then have it figure out, what it just received. What came in clutch here is registry dispatch, descibed nicely on Fabric Docs. Mojang built an entire serialization system called Codecs, which let’s you describe how a class can be serialized. Registry dispatch is a feature, where each you register each class to registry, along with such codec, and Minecraft can the use that to figure out how to deal with it. For that it uses a separate type, that I called SpellDefinition . It holds the codec, the brigadier spell node, the spell’s element, and also two new options - ticking and lifetimeTicks . The full casting pipeline now works like that: Player says spell words Those are detected by Sepple, send to server and accumulated. When a new word is added the server uses Brigadier to check if the spell is finished, or timeout ran. If so, the spell is cast - the brigadier command is executed. Each spell implements the code executed here, and either runs it’s logic instantly, or creates a new instance of it’s own Spell subclass. Here any data the spell needs can be gathered, like the targets or parameters, and stored in the subclass. Then it is ran using it’s SpellCtx context class. SpellCtx holds details like the entity that casted the spell, a postion, a direction and some other details. When a Spell is ran, a new SpellEntity holding it is spawned. The entity is synced to the client by Minecraft, along with the spell it is holding. The entity calls the run function of the spell, and runClient on the client once. This is where a spell can implement it’s one-time logic, and pm client side - VFX. Additionally, if ticking was enabled in the definition, on each tick the tick function of the spell is called by the entity. This is where behavior that runs over time can be implemented. After the amount of ticks configured in the definition the spell entity despawns. As you can see I have much more flexibility now, compare to the old system, where I only had a simple serverside callback for the logic. /sephyr cast command During development you sometimes need to cast one spell a lot, quickly. This happened when I was tweaking the leap spell VFX. The leap spell has the ɛvi (self) word in it, the detection of which often fails, because it is so short. It tired me, so I implemented a /sephyr cast <spell words> command, which can be used to run the spell. It’s creative mode only, because it ruins the point of the mod, but it is really helpful for testing. I managed to implement it quite neatly. The sephyr spells, similary to minecraft commands use the Brigadier command framework, but are registered separately. So I wrote a function, which walks the spell command tree, and registers analogous minecraft commands under /sephyr cast .
Wind beam spell The wind beam spell is now done! Check it out in the video! Make sure to see the failure compilation at the end! It took surprisingly loooong to make. I additionally track my time outside Hackatime, in my task manager (which includes all time spent, not just coding). It clocked 8 hours, 15 minutes of my work just to implement such simple spell. Much of it was spent on fighting with math and the particles vfx, which were not cooperating. The spell’s area of affect is a 3d rectangle, that rotates with the player. I had to set up math, that could find entities in it, first by calculating an axis-aligned bounding box, and the filtering entites that overlap with it with an another check of whether they are touching the shape. The worst part with math is that a single wrong minus sign can break everything, and it’s quite hard to track down where it went wrong. After I managed to make it work, I noticed the VFX is wrong, rendering in every possible direction, except the right one. After another battle the particles were in place, but now the collisions were wrong again! Finally I got both right. Other things I also did a bunch of other smaller things: I added tracking of whether the spell was started, to fix it running more then once on the client sometimes. I refactored the spell serialization code to use normal Codec s instead of StreamCodec s, so that spell data can be persisted. I added client side spell ticking, which I needed for debugging, and which might come useful later. I set up a bunch of mod’s metadata - github url, set version to 0.1.0, ensured it has depencencies declared properly. I set up bundling of some java dependencies of the mod in the jar, so that it can be ran outside of development environment. I made an icon for the mod. I submitted the mod for a review on modrinth, so that it can be published. As you can see, as the end of Stardance approaches I’m preparing to polish and publish the mod for you to try out!
Faster, smaller, better and a mushroom shark A big success has happened! I managed to replace the IPA speech recognition model “Multipa” with “Zipa”. The new model is over 4x smaller (260MB vs 1.3GB), it is noticeably faster to compute, and the best part?! Depending on the case it has similar or better accuracy then Multipa, which means less missed or mistakenly detected words! This means now it take less time for the spell to be detected after you say it. You can see it in the video, where Zipa is much faster, and Multipa misses the last word each time (it didn’t detect ˈɔnːdˌɔː nor fɒksɑm ). Few devlogs ago, when I originally implemented the model I said it wasn’t useable, but two fixes changed that: The sliding window audio cutting had to be slightly adjusted, because the model always outputs 4 logits less than Multipa. I discovered the model works much better with more context, even if it’s not relevant. I recorded myself saying “mushroom shark” and saying 6 different words. In each case adding the shark audio to the audio of the word made the IPA output of the word better! I don’t know why it works like that. Maybe the model then has better understanding of the speaker’s voice? What matters is that it helps! I adjusted the sliding window chunker to give the model additional 1.5s of previous speech audio, output. It’s translation is later cut off, so the result stays the same. I was able to do that without making latency significantly larger thanks to the new model being really fast. These two changes made the models comparable, so I updated Sephyr to use Zipa. Leap spell tweaking I spent a bunch of time working on the leap spell, to make it feel nicer. I made it apply a continuous force over time on the player instead of a one big impulse. I also made it apply some vertical force to the player depending on the looking angle, so that the player launches forward instead of rubbing against the ground. To make it smooth I engineered three math functions in Desmos, that control the amount of force added over time. Finally I updated the VFX, attaching it to player and making particles simulate in world space. This way the trails now follows the player. Making microphone selectable So far the mod always used a default system input device for capturing the audio with no way to control it. This works most of the time, but sometimes something can be wrong or you might have multiple input devices. So I need to give the user control. Sounds simple, but it required a surprisingly big portion of work. I had to update Sepple (the keyword detection engine) to support selecting a device, and changing it at runtime (which requires restarting the capture). Then I had to update the Rust<->Java bindings to accomodate that, and finally I had to wire up in Minecraft code. I decided to expose this setting as a /sephyr microphone <mic> command. Here I got a little bit stuck on implementing the microphone name autocompletion, but luckily a friend of mine knew, and told me how to do that. Here I ran out of devlog time (it’s already 19 minutes over), but it’s almost working!...
|
1793361015.829123
|
1793965815.829123
|