|
49944
|
Trawl - CTF Toolkit by @yuv1 | Stardance
|
yuv1
|
1
|
55.65
|
0
|
[{"ship_num": 1, "ts": "2026-0 [{"ship_num": 1, "ts": "2026-09-04T18:24:13Z", "hours": 52.0, "multiplier": 18.88, "payout": 0, "blessing": "neutral", "text": "Trawl Trawl is a CTF toolkit for checking suspicious files, encoded text, binaries, and challenge websites from one place. In a CTF, you often get an image, audio file, string, or binary and have to figure out what is hidden inside it. Usually that means opening a bunch of different tools and already knowing what to try. Trawl tries to make that easier. You give it the challenge, and it runs checks that make sense for that input. Try it Live: trawlctf.vercel.app Source: github.com/Yuv1s/Trawl The easiest way to test it: Open the site. Click Tour . Trawl loads a sample file with a hidden flag. Watch the checks run and see what finds it. There are also more sample files under Demos , so you do not need your own CTF challenge. What it can do Trawl has 19 main checks in the web app. It can: inspect PNG, BMP, GIF, JPEG, and WAV files for hidden data check image and audio bits show image bit planes and audio spectrograms find metadata, readable strings, embedded files, and hidden file data inspect ZIP files and PDF structure inspect ELF and PE binaries unwrap encoded text such as base64 and hex solve several common classical ciphers collect useful findings and detected flags in one results panel There is also an optional local web scanner called Remora. It can crawl a CTF website, inspect pages, scripts, headers, and resources, then send useful findings back to Trawl. How it works Most of the analysis is written in Rust and compiled to WebAssembly. The frontend is built with SvelteKit. For normal file and text analysis, the input stays in your browser instead of being uploaded to a server. No account is needed, and the app can keep working offline after it loads. Remora is separate because website scanning needs network access. It runs locally on your computer and connects to the target you give it. I also wrote the analysis and file parsing code instead of relying on existing analysis libraries. The Rust core uses wasm-bindgen to connect Rust to JavaScript. Why I built it The annoying part of CTF forensics was having lots of separate tools and needing to know which one to use before I even understood the challenge. I wanted something where I could drop in the file first and get a useful starting point. I also added the tour and demo files so someone who has never used Trawl can still open the site and understand what it does. What took the most work A big part of the work was avoiding false positives. A detector that marks everything as suspicious is not useful, so I made test files where I knew what was hidden and clean files where nothing should be found. I also tested the ELF and PE parsers with real binaries and compared results with tools like readelf and objdump . One bug I found was that Trawl correctly detected a flag inside a GIF, but the result was stored in the wrong place, so it never appeared in the UI. That pushed me to test the full path from reading the file to actually showing the result. What I learned I learned a lot about image formats, audio samples, compression, steganography, classical cryptography, binary formats, WebAssembly, browser limits, and local networking. The biggest lesson was that finding the answer in code is only part of making a useful tool. The result also has to be tested, understandable, and easy to find.", "repo": "https://github.com/Yuv1s/Trawl", "demo": "https://trawlctf.vercel.app"}]...
|
[{"post_id": 65607, "ts": "202 [{"post_id": 65607, "ts": "2026-09-10T20:56:28Z", "hours": 3.6467, "text": "Trawl has just been shipped!"}, {"post_id": 61673, "ts": "2026-09-04T18:22:19Z", "hours": 0.6467, "text": "Now live at https://trawlctf.vercel.app"}, {"post_id": 61144, "ts": "2026-09-03T22:15:43Z", "hours": 6.5825, "text": "Reversing was the one category left. Trawl had nothing for it. Everything else was covered by the time the tour shipped: steganography, cryptography (Hill and Playfair joined a few weeks back, along with columnar transposition wide enough that brute force stops working), PDF documents read the same way a ZIP already was, object by object, checked against what the file’s own index claims. Drop a binary, though, and the best Trawl could do was name the format and stop there. I’m not building a decompiler. Ghidra and IDA earned that word over years of real compiler-engineering work, and half a decompiler is worse than none: wrong pseudocode still reads as confident, and confidently wrong is the last thing a forensics tool should ever hand someone. What Trawl does instead is the pass every real session starts with anyway, before a disassembler even opens: the header, the section table, what the binary calls, what it offers, and whether it was actually built with the defenses that matter, NX, ASLR, RELRO, a stack canary, Control Flow Guard. Two formats that don’t agree on anything ELF went first, since it’s most of what a CTF binary actually is. Then PE, the Windows half, which turned out to be a different kind of problem. An ELF section header says outright where its bytes sit in the file. A PE barely says anything that way. It addresses its import and export tables by where they’ll land in memory once the loader gets to them, so reading either one means walking the section table first, just to translate an address nobody gave you a file offset for. The two parsers ended up sharing almost nothing except a type underneath them, built so one panel could read both. Where a protection has no equivalent in the other format, the panel just leaves it out instead of guessing. A PE has nothing RELRO would describe, and showing RELRO: none there would report a defense as missing that the format never offered to begin with. The bug where the layout only worked on files I made up myself Testing this against binaries I built myself went fine, because of course it did. I wrote them to go fine. So I went and found real ones instead. gcc output straight out of WSL: a stripped build, a static build, a shared library. Real Windows system binaries too, notepad.exe and kernel32.dll . I checked every one byte for byte against readelf and objdump until the entry points and export counts lined up exactly. Then I compiled a tiny C++ file. Three lines, a map of vectors, nothing unusual. Its export name came out at 286 characters. That’s what name mangling does to the plainest code without even trying. My import list rendered each name as a small chip, capped with a percentage-based max-width so nothing would run off the page. It did nothing. A grid item won’t shrink below its own content unless you tell it to, and I hadn’t, so the chip just kept going: clipped mid-character, no ellipsis, dragging the whole panel sideways with it. Fine on every file I’d invented. Broken on the first real one I pointed it at. AI helped with the writing and most of the parsing code, same as always. The real time went into finding bytes I hadn’t written myself to test it against, a real compiler, a real linker, a symbol nobody designed to be readable. Testing against your own fixtures only ever proves the code agrees with itself. Next up: Windows registry hives, for tracing which USB stick was plugged in when."}, {"post_id": 58436, "ts": "2026-08-30T21:07:08Z", "hours": 14.1186, "text": "Trawl could find anything. Nobody knew where to start. Trawl has gotten good. It cracks AES by hand, forges JWTs, reads flags out of pixels and cookies and the low bits of a sound file. On the mock CTF site I keep building to break, it clears 24 of 26 now. None of that helps the person who opens it for the first time and sees a grid of two dozen tools and a box that says “drop a file.” What file? They don’t have one. They came to look, and there is nothing to look at. So I built the tour. The tour that drops the file for you Tick “take the tour” and Trawl hands itself a sample image with a flag already hidden in the low bits, runs every tool against it before you have done anything, and then walks you through what each one turned up. The flag it already found. The tool that read it out of the red channel. The one that does not read the message but estimates how much is hidden. Two minutes, no CTF knowledge required, and by the end you have watched the whole thing work on a real answer. Because one demo is not much of a demo, there is a library behind it now. Drop-in files for every tool: a flag in a PNG’s low bits, a flag stuck on after the image ends, a flag in the second frame of a GIF, a flag written into the low bit of every audio sample, a flag that only reads once you subtract one frame from the next. Each one hides its answer somewhere different. A button in the corner opens them any time, so the people who skipped the tour and the people who came back a week later can still poke at something real. The bug where the tool found it and never told anyone The GIF samples embarrassed me. Drop one, and the tool would decode the second frame, find the flag, and then say nothing. The cod-end, the panel where every find is supposed to surface, sat empty. It was not that the flag went unfound. It was found, written down, and filed one level too deep to ever be shown. The panel only reads findings from a file’s children, and a plain GIF has no children, so its own results fell through the floor. A few lines to lift them back up. The kind of bug that passes every test, because the test checks that the flag was found, not that a human would ever see it. AI helped with the writing and the mechanical parts, same as always. The real time went into the gap between “the tool works” and “a stranger can tell that it works,” which turns out to be most of the distance. The feature list still sounds small. A tour, some sample files, a button. But the whole point was to make the thing findable, and none of the finding matters if the first thing you see is a shrug. Next up: the last two flags on that mock site, and maybe a screenshot in the readme now that there is something worth showing."}, {"post_id": 55039, "ts": "2026-08-26T13:54:36Z", "hours": 11.1219, "text": "The scanner connected. That was the easy part. Last time, the whole win was that the scanner connected at all. Trawl spots the local process, the page swaps to a URL box, and it starts crawling. But a scanner that connects and finds nothing is just a slow way to load a webpage. So this time I made it actually find the flags. On the mock CTF site I built last time to be broken, the web scan went from 4 findings to 18. Counting the image tools, the whole thing now gets 23 of the 26. Teaching it to decode, not just fetch A flag is almost never sitting in plain text waiting to be read. It’s base64 hiding in a cookie. It’s hex in a JavaScript variable. It’s a colour written as CSS escape codes, or an array of numbers XORed against a single byte, or an ETag that’s really just base64 printed backwards. So the scanner now tries all of those on everything it pulls down: every page, every script, every response header. It keeps a result only when a decode actually produces a flag, and shuts up otherwise. That last part matters more than it sounds. A tool that reports something on every site is noise, not a finding. One challenge hid an AES key and its payload in an image’s metadata. The rule I keep breaking my own back over is no dependencies, nothing borrowed, so instead of pulling in a crypto library I wrote AES by hand and checked it against the official test vectors. It works. I would not recommend the experience. The bug that made a working scanner look broken For a while every scan felt like it hung. The scanner was fine. localhost was the problem. localhost resolves to the IPv6 address first. My test site only listened on IPv4. Windows takes about two seconds to give up on the dead address before falling back to the one that works, and two seconds across fifteen requests is a scan that looks frozen. The fix was one line: try IPv4 first. The debugging was not one line. Then, the morning I went to cut the release, a routine Rust update added a new lint and broke the build on code I hadn’t touched in weeks. Pinned the version, fixed the lint, shipped. AI helped with the writing and some of the mechanical refactors, but the real time went where it always goes: into the two seconds nobody sees, and the compiler that changed its mind overnight. The feature list sounds small again. Reads a site, decodes what it finds, cracks an image. The list is never the part that takes the week. Next up is JWTs, so the scanner can start forging its way into the challenges it currently just knocks on."}, {"post_id": 52676, "ts": "2026-08-22T22:59:35Z", "hours": 10.2175, "text": "I spent hours making a web scanner not be a web scanner The obvious way to scan a website is to send the URL to a server and let the server do the ugly work. Trawl does not get that luxury. One of the rules I have tried to keep since the first version is that everything happens on your machine . No files uploaded, no URLs sent to me, no remote API quietly doing the interesting part. That became significantly harder when I added web reconnaissance. The scanner that had to stay local Browsers are deliberately terrible at crawling arbitrary websites. CORS, networking restrictions, and a dozen other sensible security decisions get in the way. So I ended up building a small local scanner and shipping it as a GitHub release. Installation is a one-line winget command. Once it is running, Trawl detects it automatically. The interface changes live, asks for a URL, and the scanner starts digging through the site for things that probably should not be public: hidden files, exposed secrets, interesting endpoints, and other CTF-shaped mistakes. Nothing leaves the machine. That sentence took far longer to make true than it looks . AI helped with writing and refactoring parts of it, but most of the time disappeared into figuring out how the browser, local process, networking, detection, and UI should talk to each other without turning Trawl into the server-backed app I specifically did not want to build. So I built a website designed to be broken Testing a recon scanner against random real sites is a terrible test. So I made my own mock CTF site instead. It has challenges from easy through expert, with deliberately hidden files and secrets for Trawl to discover. I now have a controlled target where I know exactly what should be found, which makes false positives and missed findings much harder to hide. I also improved the Vigenère decoder while I was in there. The feature list sounds small: better decoder, local scanner, test site. The amount of plumbing underneath it absolutely was not."}, {"post_id": 49928, "ts": "2026-08-18T21:55:13Z", "hours": 1.0806, "text": "The hard part was knowing when to stop Here is what someone hands you in a competition. NWE2ZDc4Njg1YTMzNzQzMDYxNDg0YTZjNWE1NjM5 Eighty-eight characters of nothing. No hint about what it is or how many times it has been through the wringer. You paste it into Trawl and it comes back: base64: 5a6d78685a33743061484a6c5a56397359586c6c636e4e665a47566c hex: ZmxhZ3t0aHJlZV9sYXllcnNfZGVlcH0= base64: flag{three_layers_deep} Three layers, unwrapped one at a time, and the answer sitting at the bottom. Nobody told it there were three. Nobody told it what any of them were. The new half This is Mantis, the code breaking side of the project, and the first thing I have built that does not need a file at all. Paste a string, get an answer. It is named after the mantis shrimp, which punches hard enough to crack a crab shell and sees colours we do not have the eyes for. Force and perception, which turns out to be the whole job. Decoding was easy. Stopping was not. I expected the tricky part to be writing the decoders. It was not. There are thirteen of them now and most took twenty minutes. The problem is that decoders always succeed. Take the sentence “the quick brown fox jumps over the lazy dog”. Delete the spaces and what you have left is, by pure coincidence, a perfectly legal base64 string. Feed it to a base64 decoder and it will not complain. It will hand you back a paragraph of garbage with total confidence. So a tool that decodes whatever it is given will happily take your answer, turn it into noise, and report that as progress. Mine did exactly that, repeatedly, while I was building it. The fix was to stop asking “can this be decoded” and start asking “did decoding it make things better”. Every layer now has to earn its place. It gets kept because of what fell out of it, never because of how it looked going in. Then I got it wrong the other way Once it was cautious, it became too cautious. Look at that chain again. The middle step decodes to a wall of hex digits. That is not more readable than the base64 above it. It is arguably worse. So my careful new version took one look, decided the second layer was not an improvement, and stopped one step short of the answer every single time. Which is the lesson, really. Getting closer to something does not always look like getting closer. It had to learn to follow a chain to the bottom and judge the destination, not each footstep. Where the catch goes When it does find something, it lands in a panel at the top of the page called the Cod-end. That is the closed end of a trawl net, the bit that drags along behind and where everything you caught ends up. It felt like the right name for the box that holds the answers. What it still cannot do It does not break real encryption, and it never will. If something is properly encrypted, Trawl says so and stops, rather than producing a confident sentence made of nothing. It also cannot solve every classical cipher yet. Two of the harder ones need a much better model of what English looks like than the short word list it currently carries, and I would rather ship the gap than fake it. Next up: hashes."}, {"post_id": 49567, "ts": "2026-08-18T13:25:48Z", "hours": 1.9528, "text": "I cooked my own laptop, then hid a word inside a noise The fans came on around eleven and never stopped. By mid-afternoon my laptop was doing an impression of a hairdryer and the temperature had gone past 100°C, which is where water boils and laptops start making decisions about their own survival. I opened Task Manager expecting a runaway browser tab. Twenty-four copies of my own program. All running. All mine. Every time I wanted to check my work I started a little local server, and every time I finished I told it to stop. It said it stopped. It did not stop. It had a second process doing the real work, and killing the first left the second running happily, forever, burning a core each. They had been stacking up since ten in the morning. The lesson is not “check your processes.” It is that I trusted a success message instead of looking. The program told me what I wanted to hear and I took its word for it, which is a strange mistake to make on a project whose entire premise is not taking things at face value. I check now. Every time. It takes four seconds. Sound has a shape Here is the thing I built this month, and I still think it is the best trick in the whole project. Every sound is a stack of pitches, high and low, shifting over time. Plot pitch going up the page and time going across, and you get a picture of what a noise is actually made of. A flute is a clean line. A cymbal is a smear. Speech is a mess of stripes that a trained eye can almost read. Now run it backwards. Decide what you want that picture to look like. Write a word, say. Then generate the exact noise that would produce it. What comes out is a real audio file. You can play it. It sounds like static, like a radio between stations, and there is nothing in it your ears will ever pick up. Show it to anyone and they hear noise, because it is noise. Look at its shape and the word is just sitting there. I built the part of Trawl that draws that shape, and then I wrote a little tool that goes the other way so I could make my own. I fed it a picture of two words stacked on top of each other. Eight seconds of grinding static came out. I dropped it back into Trawl and the words were there, drawn in sound, perfectly legible. Getting it wrong twice first My first attempt was mush. Grey bands, no letters. The reason is a bit lovely. To read the shape of a sound you have to chop it into short slices and analyse each one, and my slices were wider than the letters. It was like trying to read a book through a window that shows three words at once, smeared together. Fixed by giving each column of the picture more time. The second attempt had bright vertical bars slashed through it. Those were clicks. Thirty separate tones were all changing volume at the exact same instant, and a sharp change in sound is a click, and a click is every pitch at once. It drew a line straight through my picture. Fixed by easing the volume changes instead of stepping them. Same reason a song fades out rather than stopping dead. Both of those are in the code with comments explaining them, mostly so future me does not do it again. The one I refused to build There is a hiding technique called F5 that I cannot reliably detect. The published method for spotting it needs a copy of the original, untouched file to compare against, and I do not have one. Nobody does. That is the point of the puzzle. I could have shipped something that guesses and looks confident about it. Instead the tool shows you the raw numbers and says, plainly, that it does not know. A tool that finds something every time is not a good tool. It is a broken one that happens to be reassuring, and I would rather ship an honest gap than a convincing lie. The image and sound half is done. Codes and ciphers next."}, {"post_id": 49293, "ts": "2026-08-18T00:51:00Z", "hours": 0.4928, "text": "Devlog 4: Trawl has a face Trawl has a logo now, and I like it more than I expected to. The obvious move was a net. I drew several. A mesh narrowing to a cod-end, the side profile of a trawl, a rotated diamond lattice that looked like real netting up close. All fine, all saying nothing. A net tells you where the name came from, which is the least interesting fact about this project. So it is a scan reticle instead. Corner brackets around a grid of dots, one of them lit amber. That is the whole tool in one image. Brackets mean examination and people read them that way without being taught. The grid is whatever Trawl is looking at. The amber dot is the finding. And the amber is load-bearing. In the interface that colour has exactly one job, marking something anomalous, and it turns up nowhere else. No amber buttons, no amber headings. The logo obeys the same rule, so the mark is not decoration. It is a hit. I did ship it invisible for about ten minutes, because currentColor resolves to black once the SVG leaves the app. One amber dot floating in the void. Fixed. Next is Cod-end, phase two, where Cuttlefish stops being PNG-only."}, {"post_id": 49263, "ts": "2026-08-17T23:15:13Z", "hours": 1.1847, "text": "Devlog 3: the PNG half is finished Here is my favourite way to hide a message in a picture, and Trawl can now catch it. A PNG can store its colours as a numbered list. Pixel one uses colour seven, pixel two uses colour twelve, and so on. Nothing in the format stops that list from holding the same colour twice. So if entry 10 and entry 200 are both the same shade of blue, every blue pixel can point at either one. The image on screen is pixel for pixel identical. The file underneath is not. That choice is a hidden bit, and even a small 64 by 64 image gives you about a thousand bytes of them. That was the last thing on my list. PNG support is done. What done means Trawl reads every valid PNG now, including the two kinds it used to refuse outright. It opens compressed comment blocks instead of shrugging at them. It finds text written the Windows way, where every character is followed by an invisible null byte that made my old scanner walk straight past it. And its brute-forcer tries reading hidden bits down the columns as well as along the rows, which doubled the number of combinations from 42 to 84. Sixteen tools, all working on one format properly. That felt worth stopping to notice. The bug where it found two flags and threw them both away I planted a message in a compressed comment and another as wide text, then ran them through. Neither showed up. Both had actually been found. The problem was one line where I merged results: I was taking either the results from the PNG-specific reader or the results from the general one, never both. So the general scanner would find the wide text, hand it over, and the merge step would quietly drop it on the floor. Two correct detectors, one bad if . A related version of the same mistake: I inflate compressed comments now, but the flag scan had already run before that text existed, so nobody ever looked at it. It re-scans after decompressing. The one I got wrong for twenty minutes One of my clean test images started tripping a detector. My first guess was the new interlaced decoder, since that was the newest thing. So I built the same picture again without interlacing and ran it. Same false alarm. Not the decoder. The real cause was my test image itself. The detector works by checking whether pairs of colour values appear equally often, which a hidden message forces and a real photograph has no reason to do. My test image was a smooth gradient, and smooth gradients spread their values so evenly that the pairs come out equal on their own. I had built a control that genuinely looks guilty. I gave it a tone curve, the way a real camera would, and it went quiet. That is the fifth time this project a test image has been the thing that was broken. Next Everything except PNG. GIF and BMP first, since they should light up all sixteen tools the moment they decode. Then extracting files that are hidden inside other files, which Trawl currently finds and then refuses to hand over."}, {"post_id": 49179, "ts": "2026-08-17T20:24:23Z", "hours": 1.7364, "text": "Devlog 2: it found the hidden message, then cut it off mid-sentence You can hide text inside a photo by nudging the colour of individual pixels by one step. The image looks identical to any eye and to most software. The message just sits there, spread across a few thousand pixels, waiting for someone who knows how to look. Trawl is the thing I am building that looks. The good part I hid a note inside a photo of the Eiffel Tower at night and dropped it in. Trawl found it: If you can read this than Trawl was a success! If not then well.. lets just say.. prepare to los Right message, right place, picked out of 42 different ways it could have been hidden. Then it stopped mid-word. That was not a decoding failure. I had written a line months earlier that trimmed the preview to 96 characters, and my note was 155. The tool read every single word and showed me two thirds of one. Somehow more annoying than it failing outright. Fixed, and there is now a button that pulls the whole thing out. The message that was never there An earlier test went the other way. Different image, and Trawl confidently announced it had found BM{GEBF} . Nothing was hidden in that file. Those eight characters turned up by chance inside the image’s compressed data, and my scanner was hunting for that shape anywhere in the file. Give it half a megabyte of near-random bytes and the shape appears on its own eventually. A tool that invents answers is worse than one that finds nothing. Matches now get judged by where they came from, and anything sitting in compressed data gets demoted with the reason shown rather than announced as a discovery. Both bugs had the same cause My test images were fake in a way that mattered. I was generating smooth colour gradients to test against. Real photos are not smooth. They have grain, texture, tiny variations between neighbouring pixels. My detectors passed every test I wrote and then fell over on the first real photograph, three separate times, because the thing I was testing against did not behave like the thing I was testing for. I now write a test that checks the test image before any test that uses it. That one took me embarrassingly long to arrive at. The feature I deleted I built automatic warnings into one of the new tools, then found it flagging a completely clean image. I could have nudged the threshold until the warning went away. But then the number would exist because it made one test pass, not because it meant anything, and it would start guessing again on the next photo. So I deleted the feature. The tool shows you the measurements and makes no accusations. The two tools that do make claims are based on published research from 1999 and 2001. Given an image with a quarter of its pixels carrying hidden data, they estimate 25.0% and 25.9%. Those I trust. Next Reading camera metadata, which is where people hide things when they are not hiding them in the pixels."}, {"post_id": 48664, "ts": "2026-08-17T04:53:48Z", "hours": 2.5947, "text": "Devlog 1: Trawl, and the bug I caught before it cost me a week Trawl is a CTF toolkit that runs in the browser. Drop a file in, it runs every check I’ve written at once, and tells you what looked wrong. Nothing uploads anywhere, because there is no server to upload to. It did not start out this big. The first version was only a steganography tool, because that is the category I keep landing in and the workflow is tedious: exiftool, then strings, then binwalk, then zsteg, then StegSolve, a Java applet from 2011. Five tools, five interfaces, all of them installed before you can look at a single PNG. Plenty of beginners skip forensics entirely for that reason. A couple of weeks in I realized the same complaint covers almost every CTF category, so the scope moved. Trawl is the hub. Cuttlefish is the steganography module inside it, and the only one that exists so far. A trawl is a net you drag through water to see what comes up. Cuttlefish hide by rewriting their own surface, which is what LSB embedding does to an image. What Cuttlefish runs now Eight checks, PNG only: pixel decode with verification, chunk walk, CRC validation, post-IEND data, tEXt/zTXt/iTXt metadata, ASCII strings, flag-shape scan, and an LSB sweep across channel order, bit plane and bit order. In the screenshot one of the eight fired. The sweep pulled testCTF{hello} out of RGB, bit 0, MSB first. One of 42 combinations carried anything readable. The bug I would not have found without testing for it Before writing any analysis I wrote a single test: build a PNG with a known pattern in the low bits, decode it through the browser, check the bits survive. They don’t, by default. Canvas alpha premultiplication and colour management both rewrite pixel values slightly. For ordinary image work nobody notices. For steganography it destroys exactly the data you are looking for, and every detector downstream would have been measuring noise while looking completely functional. The fix is passing colorSpaceConversion: 'none' and premultiplyAlpha: 'none' to createImageBitmap . Cuttlefish now reports the decode step as verified so I can confirm the guarantee held for the file in front of me. Had I built the sweep before checking, I would have spent days debugging math that was already correct. On the Rust The core is Rust compiled to WASM, running in a Web Worker, partly because the sweep is real compute and JavaScript stutters on it, partly because I wanted to learn Rust properly. I wrote the detector logic myself. AI helped me refactor it and add comments, which I am noting because it is true, and because the parts I struggled through are the parts I actually understand now. Next The bit-plane wall: all 8 planes across every channel rendered at once, so an anomalous plane becomes something you see rather than something you brute-force into. Cuttlefish is not finished. The sweep only handles PNG, there is no bit-plane view, and no statistical detection at all yet. What I have proves the pipeline runs end to end on a flag I planted myself, which is a real milestone and also not much more than that. Trawl has exactly one module, which makes “hub” a generous word for it right now."}]...
|
Title: Trawl - CTF Toolkit by @yuv1 | Stardance
D Title: Trawl - CTF Toolkit by @yuv1 | Stardance
Devlogs:
Devlog 1: Trawl, and the bug I caught before it cost me a week Trawl is a CTF toolkit that runs in the browser. Drop a file in, it runs every check I’ve written at once, and tells you what looked wrong. Nothing uploads anywhere, because there is no server to upload to. It did not start out this big. The first version was only a steganography tool, because that is the category I keep landing in and the workflow is tedious: exiftool, then strings, then binwalk, then zsteg, then StegSolve, a Java applet from 2011. Five tools, five interfaces, all of them installed before you can look at a single PNG. Plenty of beginners skip forensics entirely for that reason. A couple of weeks in I realized the same complaint covers almost every CTF category, so the scope moved. Trawl is the hub. Cuttlefish is the steganography module inside it, and the only one that exists so far. A trawl is a net you drag through water to see what comes up. Cuttlefish hide by rewriting their own surface, which is what LSB embedding does to an image. What Cuttlefish runs now Eight checks, PNG only: pixel decode with verification, chunk walk, CRC validation, post-IEND data, tEXt/zTXt/iTXt metadata, ASCII strings, flag-shape scan, and an LSB sweep across channel order, bit plane and bit order. In the screenshot one of the eight fired. The sweep pulled testCTF{hello} out of RGB, bit 0, MSB first. One of 42 combinations carried anything readable. The bug I would not have found without testing for it Before writing any analysis I wrote a single test: build a PNG with a known pattern in the low bits, decode it through the browser, check the bits survive. They don’t, by default. Canvas alpha premultiplication and colour management both rewrite pixel values slightly. For ordinary image work nobody notices. For steganography it destroys exactly the data you are looking for, and every detector downstream would have been measuring noise while looking completely functional. The fix is passing colorSpaceConversion: 'none' and premultiplyAlpha: 'none' to createImageBitmap . Cuttlefish now reports the decode step as verified so I can confirm the guarantee held for the file in front of me. Had I built the sweep before checking, I would have spent days debugging math that was already correct. On the Rust The core is Rust compiled to WASM, running in a Web Worker, partly because the sweep is real compute and JavaScript stutters on it, partly because I wanted to learn Rust properly. I wrote the detector logic myself. AI helped me refactor it and add comments, which I am noting because it is true, and because the parts I struggled through are the parts I actually understand now. Next The bit-plane wall: all 8 planes across every channel rendered at once, so an anomalous plane becomes something you see rather than something you brute-force into. Cuttlefish is not finished. The sweep only handles PNG, there is no bit-plane view, and no statistical detection at all yet. What I have proves the pipeline runs end to end on a flag I planted myself, which is a real milestone and also not much more than that. Trawl has exactly one module, which makes “hub” a generous word for it right now.
Devlog 2: it found the hidden message, then cut it off mid-sentence You can hide text inside a photo by nudging the colour of individual pixels by one step. The image looks identical to any eye and to most software. The message just sits there, spread across a few thousand pixels, waiting for someone who knows how to look. Trawl is the thing I am building that looks. The good part I hid a note inside a photo of the Eiffel Tower at night and dropped it in. Trawl found it: If you can read this than Trawl was a success! If not then well.. lets just say.. prepare to los Right message, right place, picked out of 42 different ways it could have been hidden. Then it stopped mid-word. That was not a decoding failure. I had written a line months earlier that trimmed the preview to 96 characters, and my note was 155. The tool read every single word and showed me two thirds of one. Somehow more annoying than it failing outright. Fixed, and there is now a button that pulls the whole thing out. The message that was never there An earlier test went the other way. Different image, and Trawl confidently announced it had found BM{GEBF} . Nothing was hidden in that file. Those eight characters turned up by chance inside the image’s compressed data, and my scanner was hunting for that shape anywhere in the file. Give it half a megabyte of near-random bytes and the shape appears on its own eventually. A tool that invents answers is worse than one that finds nothing. Matches now get judged by where they came from, and anything sitting in compressed data gets demoted with the reason shown rather than announced as a discovery. Both bugs had the same cause My test images were fake in a way that mattered. I was generating smooth colour gradients to test against. Real photos are not smooth. They have grain, texture, tiny variations between neighbouring pixels. My detectors passed every test I wrote and then fell over on the first real photograph, three separate times, because the thing I was testing against did not behave like the thing I was testing for. I now write a test that checks the test image before any test that uses it. That one took me embarrassingly long to arrive at. The feature I deleted I built automatic warnings into one of the new tools, then found it flagging a completely clean image. I could have nudged the threshold until the warning went away. But then the number would exist because it made one test pass, not because it meant anything, and it would start guessing again on the next photo. So I deleted the feature. The tool shows you the measurements and makes no accusations. The two tools that do make claims are based on published research from 1999 and 2001. Given an image with a quarter of its pixels carrying hidden data, they estimate 25.0% and 25.9%. Those I trust. Next Reading camera metadata, which is where people hide things when they are not hiding them in the pixels.
Devlog 3: the PNG half is finished Here is my favourite way to hide a message in a picture, and Trawl can now catch it. A PNG can store its colours as a numbered list. Pixel one uses colour seven, pixel two uses colour twelve, and so on. Nothing in the format stops that list from holding the same colour twice. So if entry 10 and entry 200 are both the same shade of blue, every blue pixel can point at either one. The image on screen is pixel for pixel identical. The file underneath is not. That choice is a hidden bit, and even a small 64 by 64 image gives you about a thousand bytes of them. That was the last thing on my list. PNG support is done. What done means Trawl reads every valid PNG now, including the two kinds it used to refuse outright. It opens compressed comment blocks instead of shrugging at them. It finds text written the Windows way, where every character is followed by an invisible null byte that made my old scanner walk straight past it. And its brute-forcer tries reading hidden bits down the columns as well as along the rows, which doubled the number of combinations from 42 to 84. Sixteen tools, all working on one format properly. That felt worth stopping to notice. The bug where it found two flags and threw them both away I planted a message in a compressed comment and another as wide text, then ran them through. Neither showed up. Both had actually been found. The problem was one line where I merged results: I was taking either the results from the PNG-specific reader or the results from the general one, never both. So the general scanner would find the wide text, hand it over, and the merge step would quietly drop it on the floor. Two correct detectors, one bad if . A related version of the same mistake: I inflate compressed comments now, but the flag scan had already run before that text existed, so nobody ever looked at it. It re-scans after decompressing. The one I got wrong for twenty minutes One of my clean test images started tripping a detector. My first guess was the new interlaced decoder, since that was the newest thing. So I built the same picture again without interlacing and ran it. Same false alarm. Not the decoder. The real cause was my test image itself. The detector works by checking whether pairs of colour values appear equally often, which a hidden message forces and a real photograph has no reason to do. My test image was a smooth gradient, and smooth gradients spread their values so evenly that the pairs come out equal on their own. I had built a control that genuinely looks guilty. I gave it a tone curve, the way a real camera would, and it went quiet. That is the fifth time this project a test image has been the thing that was broken. Next Everything except PNG. GIF and BMP first, since they should light up all sixteen tools the moment they decode. Then extracting files that are hidden inside other files, which Trawl currently finds and then refuses to hand over.
Devlog 4: Trawl has a face Trawl has a logo now, and I like it more than I expected to. The obvious move was a net. I drew several. A mesh narrowing to a cod-end, the side profile of a trawl, a rotated diamond lattice that looked like real netting up close. All fine, all saying nothing. A net tells you where the name came from, which is the least interesting fact about this project. So it is a scan reticle instead. Corner brackets around a grid of dots, one of them lit amber. That is the whole tool in one image. Brackets mean examination and people read them that way without being taught. The grid is whatever Trawl is looking at. The amber dot is the finding. And the amber is load-bearing. In the interface that colour has exactly one job, marking something anomalous, and it turns up nowhere else. No amber buttons, no amber headings. The logo obeys the same rule, so the mark is not decoration. It is a hit. I did ship it invisible for about ten minutes, because currentColor resolves to black once the SVG leaves the app. One amber dot floating in the void. Fixed. Next is Cod-end, phase two, where Cuttlefish stops being PNG-only.
I cooked my own laptop, then hid a word inside a noise The fans came on around eleven and never stopped. By mid-afternoon my laptop was doing an impression of a hairdryer and the temperature had gone past 100°C, which is where water boils and laptops start making decisions about their own survival. I opened Task Manager expecting a runaway browser tab. Twenty-four copies of my own program. All running. All mine. Every time I wanted to check my work I started a little local server, and every time I finished I told it to stop. It said it stopped. It did not stop. It had a second process doing the real work, and killing the first left the second running happily, forever, burning a core each. They had been stacking up since ten in the morning. The lesson is not “check your processes.” It is that I trusted a success message instead of looking. The program told me what I wanted to hear and I took its word for it, which is a strange mistake to make on a project whose entire premise is not taking things at face value. I check now. Every time. It takes four seconds. Sound has a shape Here is the thing I built this month, and I still think it is the best trick in the whole project. Every sound is a stack of pitches, high and low, shifting over time. Plot pitch going up the page and time going across, and you get a picture of what a noise is actually made of. A flute is a clean line. A cymbal is a smear. Speech is a mess of stripes that a trained eye can almost read. Now run it backwards. Decide what you want that picture to look like. Write a word, say. Then generate the exact noise that would produce it. What comes out is a real audio file. You can play it. It sounds like static, like a radio between stations, and there is nothing in it your ears will ever pick up. Show it to anyone and they hear noise, because it is noise. Look at its shape and the word is just sitting there. I built the part of Trawl that draws that shape, and then I wrote a little tool that goes the other way so I could make my own. I fed it a picture of two words stacked on top of each other. Eight seconds of grinding static came out. I dropped it back into Trawl and the words were there, drawn in sound, perfectly legible. Getting it wrong twice first My first attempt was mush. Grey bands, no letters. The reason is a bit lovely. To read the shape of a sound you have to chop it into short slices and analyse each one, and my slices were wider than the letters. It was like trying to read a book through a window that shows three words at once, smeared together. Fixed by giving each column of the picture more time. The second attempt had bright vertical bars slashed through it. Those were clicks. Thirty separate tones were all changing volume at the exact same instant, and a sharp change in sound is a click, and a click is every pitch at once. It drew a line straight through my picture. Fixed by easing the volume changes instead of stepping them. Same reason a song fades out rather than stopping dead. Both of those are in the code with comments explaining them, mostly so future me does not do it again. The one I refused to build There is a hiding technique called F5 that I cannot reliably detect. The published method for spotting it needs a copy of the original, untouched file to compare against, and I do not have one. Nobody does. That is the point of the puzzle. I could have shipped something that guesses and looks confident about it. Instead the tool shows you the raw numbers and says, plainly, that it does not know. A tool that finds something every time is not a good tool. It is a broken one that happens to be reassuring, and I would rather ship an honest gap than a convincing lie. The image and sound half is done. Codes and ciphers next.
The hard part was knowing when to stop Here is what someone hands you in a competition. NWE2ZDc4Njg1YTMzNzQzMDYxNDg0YTZjNWE1NjM5 Eighty-eight characters of nothing. No hint about what it is or how many times it has been through the wringer. You paste it into Trawl and it comes back: base64: 5a6d78685a33743061484a6c5a56397359586c6c636e4e665a47566c hex: ZmxhZ3t0aHJlZV9sYXllcnNfZGVlcH0= base64: flag{three_layers_deep} Three layers, unwrapped one at a time, and the answer sitting at the bottom. Nobody told it there were three. Nobody told it what any of them were. The new half This is Mantis, the code breaking side of the project, and the first thing I have built that does not need a file at all. Paste a string, get an answer. It is named after the mantis shrimp, which punches hard enough to crack a crab shell and sees colours we do not have the eyes for. Force and perception, which turns out to be the whole job. Decoding was easy. Stopping was not. I expected the tricky part to be writing the decoders. It was not. There are thirteen of them now and most took twenty minutes. The problem is that decoders always succeed. Take the sentence “the quick brown fox jumps over the lazy dog”. Delete the spaces and what you have left is, by pure coincidence, a perfectly legal base64 string. Feed it to a base64 decoder and it will not complain. It will hand you back a paragraph of garbage with total confidence. So a tool that decodes whatever it is given will happily take your answer, turn it into noise, and report that as progress. Mine did exactly that, repeatedly, while I was building it. The fix was to stop asking “can this be decoded” and start asking “did decoding it make things better”. Every layer now has to earn its place. It gets kept because of what fell out of it, never because of how it looked going in. Then I got it wrong the other way Once it was cautious, it became too cautious. Look at that chain again. The middle step decodes to a wall of hex digits. That is not more readable than the base64 above it. It is arguably worse. So my careful new version took one look, decided the second layer was not an improvement, and stopped one step short of the answer every single time. Which is the lesson, really. Getting closer to something does not always look like getting closer. It had to learn to follow a chain to the bottom and judge the destination, not each footstep. Where the catch goes When it does find something, it lands in a panel at the top of the page called the Cod-end. That is the closed end of a trawl net, the bit that drags along behind and where everything you caught ends up. It felt like the right name for the box that holds the answers. What it still cannot do It does not break real encryption, and it never will. If something is properly encrypted, Trawl says so and stops, rather than producing a confident sentence made of nothing. It also cannot solve every classical cipher yet. Two of the harder ones need a much better model of what English looks like than the short word list it currently carries, and I would rather ship the gap than fake it. Next up: hashes.
I spent hours making a web scanner not be a web scanner The obvious way to scan a website is to send the URL to a server and let the server do the ugly work. Trawl does not get that luxury. One of the rules I have tried to keep since the first version is that everything happens on your machine . No files uploaded, no URLs sent to me, no remote API quietly doing the interesting part. That became significantly harder when I added web reconnaissance. The scanner that had to stay local Browsers are deliberately terrible at crawling arbitrary websites. CORS, networking restrictions, and a dozen other sensible security decisions get in the way. So I ended up building a small local scanner and shipping it as a GitHub release. Installation is a one-line winget command. Once it is running, Trawl detects it automatically. The interface changes live, asks for a URL, and the scanner starts digging through the site for things that probably should not be public: hidden files, exposed secrets, interesting endpoints, and other CTF-shaped mistakes. Nothing leaves the machine. That sentence took far longer to make true than it looks . AI helped with writing and refactoring parts of it, but most of the time disappeared into figuring out how the browser, local process, networking, detection, and UI should talk to each other without turning Trawl into the server-backed app I specifically did not want to build. So I built a website designed to be broken Testing a recon scanner against random real sites is a terrible test. So I made my own mock CTF site instead. It has challenges from easy through expert, with deliberately hidden files and secrets for Trawl to discover. I now have a controlled target where I know exactly what should be found, which makes false positives and missed findings much harder to hide. I also improved the Vigenère decoder while I was in there. The feature list sounds small: better decoder, local scanner, test site. The amount of plumbing underneath it absolutely was not.
The scanner connected. That was the easy part. Last time, the whole win was that the scanner connected at all. Trawl spots the local process, the page swaps to a URL box, and it starts crawling. But a scanner that connects and finds nothing is just a slow way to load a webpage. So this time I made it actually find the flags. On the mock CTF site I built last time to be broken, the web scan went from 4 findings to 18. Counting the image tools, the whole thing now gets 23 of the 26. Teaching it to decode, not just fetch A flag is almost never sitting in plain text waiting to be read. It’s base64 hiding in a cookie. It’s hex in a JavaScript variable. It’s a colour written as CSS escape codes, or an array of numbers XORed against a single byte, or an ETag that’s really just base64 printed backwards. So the scanner now tries all of those on everything it pulls down: every page, every script, every response header. It keeps a result only when a decode actually produces a flag, and shuts up otherwise. That last part matters more than it sounds. A tool that reports something on every site is noise, not a finding. One challenge hid an AES key and its payload in an image’s metadata. The rule I keep breaking my own back over is no dependencies, nothing borrowed, so instead of pulling in a crypto library I wrote AES by hand and checked it against the official test vectors. It works. I would not recommend the experience. The bug that made a working scanner look broken For a while every scan felt like it hung. The scanner was fine. localhost was the problem. localhost resolves to the IPv6 address first. My test site only listened on IPv4. Windows takes about two seconds to give up on the dead address before falling back to the one that works, and two seconds across fifteen requests is a scan that looks frozen. The fix was one line: try IPv4 first. The debugging was not one line. Then, the morning I went to cut the release, a routine Rust update added a new lint and broke the build on code I hadn’t touched in weeks. Pinned the version, fixed the lint, shipped. AI helped with the writing and some of the mechanical refactors, but the real time went where it always goes: into the two seconds nobody sees, and the compiler that changed its mind overnight. The feature list sounds small again. Reads a site, decodes what it finds, cracks an image. The list is never the part that takes the week. Next up is JWTs, so the scanner can start forging its way into the challenges it currently just knocks on.
Trawl could find anything. Nobody knew where to start. Trawl has gotten good. It cracks AES by hand, forges JWTs, reads flags out of pixels and cookies and the low bits of a sound file. On the mock CTF site I keep building to break, it clears 24 of 26 now. None of that helps the person who opens it for the first time and sees a grid of two dozen tools and a box that says “drop a file.” What file? They don’t have one. They came to look, and there is nothing to look at. So I built the tour. The tour that drops the file for you Tick “take the tour” and Trawl hands itself a sample image with a flag already hidden in the low bits, runs every tool against it before you have done anything, and then walks you through what each one turned up. The flag it already found. The tool that read it out of the red channel. The one that does not read the message but estimates how much is hidden. Two minutes, no CTF knowledge required, and by the end you have watched the whole thing work on a real answer. Because one demo is not much of a demo, there is a library behind it now. Drop-in files for every tool: a flag in a PNG’s low bits, a flag stuck on after the image ends, a flag in the second frame of a GIF, a flag written into the low bit of every audio sample, a flag that only reads once you subtract one frame from the next. Each one hides its answer somewhere different. A button in the corner opens them any time, so the people who skipped the tour and the people who came back a week later can still poke at something real. The bug where the tool found it and never told anyone The GIF samples embarrassed me. Drop one, and the tool would decode the second frame, find the flag, and then say nothing. The cod-end, the panel where every find is supposed to surface, sat empty. It was not that the flag went unfound. It was found, written down, and filed one level too deep to ever be shown. The panel only reads findings from a file’s children, and a plain GIF has no children, so its own results fell through the floor. A few lines to lift them back up. The kind of bug that passes every test, because the test checks that the flag was found, not that a human would ever see it. AI helped with the writing and the mechanical parts, same as always. The real time went into the gap between “the tool works” and “a stranger can tell that it works,” which turns out to be most of the distance. The feature list still sounds small. A tour, some sample files, a button. But the whole point was to make the thing findable, and none of the finding matters if the first thing you see is a shrug. Next up: the last two flags on that mock site, and maybe a screenshot in the readme now that there is something worth showing.
Reversing was the one category left. Trawl had nothing for it. Everything else was covered by the time the tour shipped: steganography, cryptography (Hill and Playfair joined a few weeks back, along with columnar transposition wide enough that brute force stops working), PDF documents read the same way a ZIP already was, object by object, checked against what the file’s own index claims. Drop a binary, though, and the best Trawl could do was name the format and stop there. I’m not building a decompiler. Ghidra and IDA earned that word over years of real compiler-engineering work, and half a decompiler is worse than none: wrong pseudocode still reads as confident, and confidently wrong is the last thing a forensics tool should ever hand someone. What Trawl does instead is the pass every real session starts with anyway, before a disassembler even opens: the header, the section table, what the binary calls, what it offers, and whether it was actually built with the defenses that matter, NX, ASLR, RELRO, a stack canary, Control Flow Guard. Two formats that don’t agree on anything ELF went first, since it’s most of what a CTF binary actually is. Then PE, the Windows half, which turned out to be a different kind of problem. An ELF section header says outright where its bytes sit in the file. A PE barely says anything that way. It addresses its import and export tables by where they’ll land in memory once the loader gets to them, so reading either one means walking the section table first, just to translate an address nobody gave you a file offset for. The two parsers ended up sharing almost nothing except a type underneath them, built so one panel could read both. Where a protection has no equivalent in the other format, the panel just leaves it out instead of guessing. A PE has nothing RELRO would describe, and showing RELRO: none there would report a defense as missing that the format never offered to begin with. The bug where the layout only worked on files I made up myself Testing this against binaries I built myself went fine, because of course it did. I wrote them to go fine. So I went and found real ones instead. gcc output straight out of WSL: a stripped build, a static build, a shared library. Real Windows system binaries too, notepad.exe and kernel32.dll . I checked every one byte for byte against readelf and objdump until the entry points and export counts lined up exactly. Then I compiled a tiny C++ file. Three lines, a map of vectors, nothing unusual. Its export name came out at 286 characters. That’s what name mangling does to the plainest code without even trying. My import list rendered each name as a small chip, capped with a percentage-based max-width so nothing would run off the page. It did nothing. A grid item won’t shrink below its own content unless you tell it to, and I hadn’t, so the chip just kept going: clipped mid-character, no ellipsis, dragging the whole panel sideways with it. Fine on every file I’d invented. Broken on the first real one I pointed it at. AI helped with the writing and most of the parsing code, same as always. The real time went into finding bytes I hadn’t written myself to test it against, a real compiler, a real linker, a symbol nobody designed to be readable. Testing against your own fixtures only ever proves the code agrees with itself. Next up: Windows registry hives, for tracing which USB stick was plugged in when.
Now live at https://trawlctf.vercel.app
Trawl has just been shipped!
Ships:
Trawl Trawl is a CTF toolkit for checking suspicious files, encoded text, binaries, and challenge websites from one place. In a CTF, you often get an image, audio file, string, or binary and have to figure out what is hidden inside it. Usually that means opening a bunch of different tools and already knowing what to try. Trawl tries to make that easier. You give it the challenge, and it runs checks that make sense for that input. Try it Live: trawlctf.vercel.app Source: github.com/Yuv1s/Trawl The easiest way to test it: Open the site. Click Tour . Trawl loads a sample file with a hidden flag. Watch the checks run and see what finds it. There are also more sample files under Demos , so you do not need your own CTF challenge. What it can do Trawl has 19 main checks in the web app. It can: inspect PNG, BMP, GIF, JPEG, and WAV files for hidden data check image and audio bits show image bit planes and audio spectrograms find metadata, readable strings, embedded files, and hidden file data inspect ZIP files and PDF structure inspect ELF and PE binaries unwrap encoded text such as base64 and hex solve several common classical ciphers collect useful findings and detected flags in one results panel There is also an optional local web scanner called Remora. It can crawl a CTF website, inspect pages, scripts, headers, and resources, then send useful findings back to Trawl. How it works Most of the analysis is written in Rust and compiled to WebAssembly. The frontend is built with SvelteKit. For normal file and text analysis, the input stays in your browser instead of being uploaded to a server. No account is needed, and the app can keep working offline after it loads. Remora is separate because website scanning needs network access. It runs locally on your computer and connects to the target you give it. I also wrote the analysis and file parsing code instead of relying on existing analysis libraries. The Rust core uses wasm-bindgen to connect Rust to JavaScript. Why I built it The annoying part of CTF forensics was having lots of separate tools and needing to know which one to use before I even understood the challenge. I wanted something where I could drop in the file first and get a useful starting point. I also added the tour and demo files so someone who has never used Trawl can still open the site and understand what it does. What took the most work A big part of the work was avoiding false positives. A detector that marks everything as suspicious is not useful, so I made test files where I knew what was hidden and clean files where nothing should be found. I also tested the ELF and PE parsers with real binaries and compared results with tools like readelf and objdump . One bug I found was that Trawl correctly detected a flag inside a GIF, but the result was stored in the wrong place, so it never appeared in the UI. That pushed me to test the full path from reading the file to actually showing the result. What I learned I learned a lot about image formats, audio samples, compression, steganography, classical cryptography, binary formats, WebAssembly, browser limits, and local networking. The biggest lesson was that finding the answer in code is only part of making a useful tool. The result also has to be tested, understandable, and easy to find....
|
1792878662.1165133
|
1793742662.1165133
|