|
11289
|
fabrica by @acandoo | Stardance
|
acandoo
|
0
|
64.36
|
0
|
[]
|
[{"post_id": 16534, "ts": "202 [{"post_id": 16534, "ts": "2026-06-26T21:55:13Z", "hours": 2.6689, "text": "hello again! my main focus for this devlog has been on experimenting with loading/transition behavior for queries. before, the page would remove all of the card components and just show a “Results: loading…” screen and an unstyled <li> loading element, but i wanted to improve that and make it more animated and less “flashy”. the first part of the transition comes on the “leading edge”, when the user updates their query and information has not yet been fetched. if i let the previous cards and results stay up, the site will be perceived as laggy and not responsive. so, my instinct would be to place a “skeleton UI” but that in itself would be a DOM update/layout reflow, which i wanted to avoid for performance reasons. i tried blurring individual card elements, but i didn’t like how it looked. maybe i settle with a skeleton UI or come up with some other combination of CSS effects, i don’t know. either way, i didn’t want the effect to show up if the request hits the cache, so instead of using solid.js’s existing useTransition hook, i manually combined the underlying startTransition function with my own pending signal, which i only set to true if the query was not found in the cache. this does mean that on cache hits the code looks up the map two times (once on the initial check for the pending signal, another on the resource fetch), but i don’t think trying to optimize that away with a boolean or something would end up being faster than just letting v8 optimize the double Map lookup. for the second part–or, trailing edge–of the interaction, i had this idea of constructing a view transition so that cards that showed up previously would move nicely to their resulting spot. that unfortunately made things really laggy, especially since view-transition-name would be declared on both the cards themselves and many items within the card, as those would need to be labeled for the view transition to the product page. the visual effect was really nice though, so my plan to make it work is to finally introduce virtual scroll… yes i know i’m fixing more javascript with more javascript, but at least the site works without it! right?! additional minor thoughts: i’m starting to not like the star/basket buttons. i was looking online for product card inspiration, and most of the designs i found were a lot more minimal, many of them even omit the “Learn more” button for a cleaner visual aesthetic. for now i’ll probably change out the buttons to just the underlying icon outlines, but keep the bounding box. my qualm with removing “Learn more” is discoverability on mobile, since i couldn’t disclose product page functionality through a hover effect. maybe once the card reaches a certain height on the viewport then i’ll activate the hover progresive disclosure for mobile, but idk. maybe i should fetch in the background? with the transition effect, it would be nice for it to apply when you backspace, and at that point the site isn’t really doing any more network stuff aside from prefetching hovered cards, so i could fetch all of the sliced versions of the query (i.e. for “beam” it would fetch “b”, “be” and “bea” in the background). should i persist the results count on scroll? maybe integrated within the navigation bar, or in smaller text in a pill under, idk."}, {"post_id": 15325, "ts": "2026-06-24T21:56:53Z", "hours": 3.2861, "text": "hello! so far i’ve been working on responsive/adaptive design for the search page. i’m kinda putting off doing the individual pages in place of refining the search page. the logo is properly sized on safari. i don’t know what is it about safari and fit-content that just makes everything break horribly, but i had to choose manual widths for making the logo adaptive instead of relying on the image width. for some reason the logo tile gradually increases width despite me setting the width as static (gah css!). at some point in the future i should integrate the UI work with browserstack/tests. the hero page is now a proper <a> element instead of an <input> with focus intercepted. to keep compatibility with no javascript, it SSR’s as an <input> element before switching out to an <a> on component mount. this means that on mobile the keyboard won’t open and close when trying to search. unfortunately iOS safari doesn’t support autofocus… the nav bar is now the same width as the cards. the culprit was an empty div… i had adjusted for flexbox gap being janky by adding more padding, but now that i eliminated the element i could remove the padding composition and make everything the same width. this does mean that when scrolling down the curvature of the search bar/logo isn’t consistent with the title cards, but hey everything’s a compromise. bigger card buttons & button icons. they feel almost too big, but i know that click targets are important for accessibility. i also made the “Learn more” button a full pill instead of a rounded square. i guess the convention i’m intuitively going for is “rounder UI elements = more desired/friendly path”. i’m still upset that the icon library doesn’t work, it would be so nice. even more adaptive cards. i set up the breakpoints so that the image moves vertical first, then once there’s not enough space for the “Learn more” button the star and basket, the star and basket disappear since their functionality can be on the product page."}, {"post_id": 14180, "ts": "2026-06-23T05:26:35Z", "hours": 3.0925, "text": "yeah so i ended up working more… i finished up the transition to card view. i had a lot of trouble because i was putting the shrink/grow/basis tailwind classes in the <article> tag within the <li> which meant that all the flexbox stuff wasn’t applying properly. 😅 i figured out how to get that ellipses text wrapping. for some reason i couldn’t get overflow: hidden + whitespace: nowrap to work without completely breaking the flow of the page, but line-clamp did work! a bunch of miscellaneous work. added a no-JS warning to the /search page, changed the tracked event for the initial search page preload and filtered so it was only mouse hovers, stricter pre-commit hook, font preload, and color changes. i need to work on the product pages next, but also i have been caught by scope creep.. i’ve decided that in the future i want to add auth and user/organization work and an actual database since now i want the project to become an inventory system itself. it makes sense UI wise and i’ll get to play with databases finally. for first ship though i’ll probably just use localStorage. since the attached recording i’ve added placeholder buttons for adding to favorites and the shopping cart… the favorites button is easy enough, it’s just a toggle, maybe some cute css animations, though whether the auth solution i end up (probably betterauth) supports it is a whole other story… the shopping cart is a different level of hard JS-less though especially since there’s not enough UI space to add a quantity amount. my plan for the interaction was that hover inactive would give the “Add to shopping cart” dialog, click inactive -> active opens up a quantity dialog, setting the quantity to 0 resets the buttom, and hover active brings up the quantity dialog again."}, {"post_id": 13931, "ts": "2026-06-22T21:06:34Z", "hours": 4.9658, "text": "productive day of UI work today! i started off by taking care of some meta-work within the repo. i added a pre-commit hook for linting and formatting, added tailwind class sorting to my formatter config, pushed more vscode settings and extensions to the repository, updated pnpm, and also put a pre-install hook so that others that try setting up the repository without pnpm will get a warning (at least in theory). in the future i want to add a hook to enforce gitmoji but i’ll leave that for free time when i’m less motivated to code. i added a view transition for the logo going into search. i just put the logo within its own Solid component so that both the home page and search have the same view-transition-name . i also made it adaptable on the search screen so it collapses to its smaller form by just cropping the svg instead of making another network request. though if i wanted that as flexible as possible i’d probably modify the <picture> element to also media-query-match against device width. that way graceful degradation also works JS-less. product cards look a lot nicer! now unfortunately i have a bunch of duplicate js styles across a lot of components, so i’ll probably set down some base styles or classes. what i would love for CSS are tailwind utility classes where the groupings are build-time optimized. there are still a couple of things on my plate UI-wise: some of the UI cards aren’t occupying the full width. idk why since i set their width via w-full but idk i’m probably going to switch out the UI to more of a grid formation anyways. turn the cards into a grid formation. they’re way too long and taking up valuable space. figure out what to do with text. i tried getting only the first sentence, i tried overlay: hidden , i’m not sure but am currently leaning towards some sort of blur/fade effect. make the <Suspense> boundary narrower. my current desired functionality is for all the cards to “gray out” but with the original item list length, before the resource is fetched and the dom update happens. minimal DOM update on resource fetching. i need to swap out the signal-backed Resource for a store-backed one that reconciles, but i still have to figure out solid.js’s API shape. and also tanstack virtual but i need to do cards first. product pages."}, {"post_id": 13563, "ts": "2026-06-22T06:23:49Z", "hours": 8.1175, "text": "okay, take two on writing through this devlog since stardance just consumed my writing since it went over the word limit… i did a lot of work since my last devlog with the UI, from view transitions to SSR to SEO to wrestling with build tools and broken npm packges. i don’t have the energy to write through everything again but i do have this demo that shows off just how cool view transitions are, the speed of SSR (though i had to make it cold start, which again i can’t be bothered to write about why a second time), and graceful degradation for javascript-less environments. next up i’m working on some housekeeping stuff with sorting tailwind classes, then formatting the search page, wiring up view transitions between the search page and product pages, more DOM optimization with better solid.js state management, and maybe adding tanstack virtualizer (though it’s kinda overkill, but allows me to just set every image to eagerly load) and keeping support for no-JS. please stardance team could you fix entries getting sent into the void i wrote a really good devlog entry the first time around…"}, {"post_id": 12347, "ts": "2026-06-20T04:26:55Z", "hours": 2.6478, "text": "i don’t have much to share so far, been reading documentation & doing miscellaneous work, but right now i’m getting myself carried away with color palettes and fonts and branding design first. i designed a lil’ logo! i’ll update the banner for this project to include the logo and typography and whatnot, but i decided on fonts (dm sans for display, ibm plex sans/mono, i know hurr hurr very original). originally i was going to use space grotesk as the display/heading font but i didn’t want my website to look too much like axon robotics et al. and i didn’t want to use orbitron because i feel like the wordmark should be a distinct font/character. i emailed gobilda asking about CAD file scraping and their GraphQL endpoint that I discovered (which unfortunately needs credentials…), which gives me an excuse to not worry about the parsing issues with the scraper until they email me back or a week has passed. so in the meantime, i’ll work on web design. in terms of pure functionality though i did also add a static product page generator for each of the products, which was why i thought astro was a really great fit for this project since i’m focusing on moving more stuff to build-time and improving DX instead. i do want to set up an account system and auth and all that jazz because the next big feature would be automatic PRs from suggestions, that just adds/modifies a json file in a partdb overrides folder in the repo. that’ll also give me an excuse to validate my monorepo design by putting some of that logic in another package. finally, i know in my ai policy i’m using ai very minimally and i’ve disabled ai features in vscode… but could we let this one slide :) why those hex codes specifically iykyk"}, {"post_id": 11484, "ts": "2026-06-18T19:20:13Z", "hours": 4.1439, "text": "IT WORKS!!!!!!! okay so for some reason the type signature of the search library that i’m using, flexsearch, didn’t give the right type signatures for when i put in enrich: true on searching. i had to do some manual array processing, which probably slows down the results a bit but it’s still under 15 milliseconds max. it’s incredibly fast, it seems like the bundle would work for edge functions so it’d be similarly fast in real life, and i can’t wait to implement the service worker backend to make things even faster. from here i want to refine the querying so you can filter by manufacturer, categories, etc. and refine the database items. and also have a pretty UI :)"}, {"post_id": 10948, "ts": "2026-06-17T23:12:45Z", "hours": 0.9117, "text": "most of the parsing issues have been fixed! the final list is 1321 items, which is the same amount as previous scrapes, so i don’t think there are any issues there. a couple things still worry me a bit (but they’re not things that hinder progress elsewhere): the build command lingers after it says it’s finished building. i suspect it’s because some promise ends up never resolving, but i’ll have to look into that further the specifications field is messed up. i should also add a string sanitizer later on to the helpers file, but in the migration to cheerio something with the specification parsing logic got messed up. a cached build still takes two minutes. ideally i would like it to be more around the 30 second range, but i have to figure out if there are any bigger things worth caching. there’s just a type error i have to deal with when it comes to error handling, and everything should be good! i’ll move on to more site prototyping since i have a dummy database to test things against."}, {"post_id": 10911, "ts": "2026-06-17T21:42:48Z", "hours": 3.5503, "text": "update: i’ve moved all of the logic to use cheerio and worker threads! i tried running just build , and… of course, it didn’t work. a few of the issues were easy to diagnose: i forgot to create a directory on startup, or put .js instead of .ts as a dynamic import path, or potentially had multiple concurrent writes that could occur. but now, i’m encountering an issue where my requests don’t want to transfer, which i initially assumed was because the object had too many layers.. but trying structuredClone on a similar object works. for other work: i’ve finally removed JSDOM from dependencies, and also updated pnpm on the side. i enabled sorting imports, which helped clean up some of the mess that was accumulating. i added helper scripts for dealing with the cache. finally, i moved the rate limited fetch within the goBILDA adapter and passed as an argument to helper functions, which would allow for future adapters to have their own rate limits."}, {"post_id": 10747, "ts": "2026-06-17T17:13:36Z", "hours": 0.7389, "text": "I LOVE METAPROGRAMMING okay so for some background i worked on moving the getPages method of the goBILDA adapter to a worker thread method (which parses the goBILDA sitemap to get the main category pages), but i realized from the type error that occurred that the whole type inference debacle got messed up as there was no url parameter in the method input… so now i’ve unleashed a generic-type guard-inference inferno. typescript is a turing complete language combined with a turing complete type system… but at least there aren’t any generics explicitly declared within the code, nor any as keywords sprinkled through this section!"}, {"post_id": 10463, "ts": "2026-06-17T06:06:16Z", "hours": 4.1686, "text": "got busy this past week with a bunch of other duties, so slow progress. i set up a thread pool class with a promise API and type safety… which is kinda overkill but that’s a bit of the whole goal with this project :p but so far i offloaded initially parsing the web page for a <meta http-equiv=\"refresh\"> and determining the link. which is not a long task, so not a huge main thread blocker! but it would allow me to cement the code structure with the worker pool. the thing was, i really wanted to optimize this build, and the useCache hook i wrote only supported JSON stringifyable values, which means no array buffers. the fetch API allows you to get the response directly as an ArrayBuffer, which, once you mark as transferrable, means that a thread can work directly on the buffer without having to copy all of the data. and since memory contention is one of the two problems i’m trying to solve here… that’s incredibly useful. the thing was, i couldn’t just insert it directly into the JSON cache entry since it would be parsed by JSON.parse as a string, so i set up another content-addressed directory ( cache/blobs instead of cache/files , where most things go) and modified the types to ensure that it was still type safe. turns out… making that type work out in the function was a PAIN,, especially considering i had generics involved. i was able to work through it with an atrocious amount of as keywords sprinkled throughout, but it’s not very elegant. in terms of the site, i opened up figma and started working on some prototypes! i’ve been meaning to learn figma and figured this was the perfect opportunity to go through the process. i love how, for the most part, everything is really intuitive. but the “frame” and “group” and “component” concepts being separate things was really confusing to me. i’m really tired and about to lose my train of thought but for tomorrow i’ll work on porting the rest of the DOM processing over to the worker threads and to cheerio. i’ll also probably have to rethink how code is organized because now the “adapter” is split across two files. there’s definitely a way to make things more elegant, though."}, {"post_id": 8497, "ts": "2026-06-14T03:57:04Z", "hours": 1.4094, "text": "many OOMs and attempts to use chrome devtools later… i think tomorrow i’m going to switch out everything to use cheerio instead of JSDOM to reduce memory usage, then maybe add a thread pool while i’m at it. currently i’ve just bumped the “old memory” limit up to 16 WHOLE GIGABYTES…. gah javascript.. i’ll also have to debug a JSON parsing problem with a few of the pages. i have no idea what it might be aside from JSON.parse being finicky or JSON.stringify not properly escaping but i highly doubt that’s the reason."}, {"post_id": 8421, "ts": "2026-06-14T00:10:46Z", "hours": 1.8519, "text": "so i tried caching webpages and… um… whoops. there are so many footguns with concurrency… i had to modify a record to also store a promise chain so that writes didn’t happen concurrently… and i had run out of memory on a rebuild with cached elements which i assumed was because i was thrashing the filesystem by immediately closing file descriptors once things were read, so i set up an object to keep the file handles around but because of the async boundary they had gotten garbage collected… urgh"}, {"post_id": 8352, "ts": "2026-06-13T21:39:28Z", "hours": 8.8886, "text": "wow i haven’t logged in a while! the past few days were the last bits of school, so not a lot of work done per day. i’ve decided to change up how the partdb is built, since i discovered unplugin-macros which really helps streamline build-time logic and the final export of the package. combined with tsdown the whole build situation of the package is a lot cleaner. in addition, i added filesystem-level caching so that when this eventually gets built on github actions it will have a cache folder that it can use, and with proper expiration handled. i still have to fine tune what computations get cached, but soon hopefully cached builds won’t take the heat death of the universe to complete! since the caching is time stamped it also makes garbage collection of the cache trivial and runnable in a github action. that code isn’t written yet but isn’t too hard. for that reason i also bumped up the node version because i wanted to use Temporal to deal with timestamps and durations :) in terms of other minor things: a rudimentary site package was added, with a template astro site. i did research into different serverless platforms lint errors fixed more code comments/inline documentation made the just build recipe functional proper error message logging for product page parsing i’ll get out of my meta-work slump soon, soon!!"}, {"post_id": 5343, "ts": "2026-06-09T02:32:50Z", "hours": 1.1783, "text": "not much to update on, but i decided to clean up the commit history with gitmoji . shout out gleamlins for showing me gitmoji! i wanted to implement it for this project because i think it’s really neat and cute! there are probably still some stylistic discrepancies/wrong things within the commit history but for now my perfectionism is satisfied :)"}, {"post_id": 5030, "ts": "2026-06-08T18:03:31Z", "hours": 0.5753, "text": "VICTORY!!! from here it’s a matter of refining the scraper to make sure there are no duplicates / useless parts / specs are good. the final pieces of the puzzle were removing JSDOM caching to reduce memory usage (since i already check for duplicates reasonably well before the request is made), adding some more filtering for irrelevant products, manually garbage collecting with global.gc() (!!), and allowing for errors in product parsing. i could probably make it more robust by forcing all the categories to be evaluated before products."}, {"post_id": 4991, "ts": "2026-06-08T17:14:15Z", "hours": 1.7214, "text": "i crashed node 😭😭😭 the final pieces were adding lists of already-visited websites, and rate limiting myself so i don’t crash goBILDA. whoops…. i need to break up writes to the file in smaller pieces. i’ll probably write per product processed. and since things are bound to go wrong with a scraper there’s probably some exception to the rule with some random product that doesn’t make it in the scraper / isn’t processed correctly."}, {"post_id": 4591, "ts": "2026-06-07T22:55:54Z", "hours": 1.5803, "text": "alright, final bit of work for today. since the last log, i put some more logging messages and files to diagnose different issues. i caught one where sometimes goBILDA didn’t redirect through an HTTP code but rather through <meta http-equiv=\"refresh\"> which was annoying. and some category pages had a link to other category pages given the SAME SET OF ATTRIBUTES AS PRODUCTS, or took from an entirely different place in the DOM because the product list is CLIENT SIDE RENDERED. i’m sorry but i am NOT about to pull out puppeteer for a (presumably) 1,000+ SKU website. i also added a cache to fetched URLs so that i didn’t bombard goBILDA with duplicate requests as i worked. also on that note, i put in a custom user agent while i was at it so that they hopefully don’t block what i’m doing 😊 tomorrow i’ll also probably work on deduplicating DOM queries by keeping track of a list of scanned product pages and filtering out pages that are irrelevant for the catalog’s purposes."}, {"post_id": 4442, "ts": "2026-06-07T19:24:16Z", "hours": 1.1836, "text": "we are getting somewhere!!! now a couple of more rounds of bug fixing and i should have all the metadata for products for goBILDA! from there it’s manual overriding and filtering to remove bundles and anything of that sort."}, {"post_id": 4356, "ts": "2026-06-07T17:25:41Z", "hours": 7.6792, "text": "hey everyone! i’ve been thinking through this project in notes (and in my head) for a little while now, and just got started putting things down in code starting friday or so. i’m working on fabrica, an overengineered parts library/search engine for robotics teams. currently i’ve structured out how to code static database generation, decided on repository structure, and scaffolded tooling and helper functions for later use, and am now working on a data scraper for goBILDA parts. it’s not the fancy website stuff, but getting all of the data was something i wanted to get down first. this is a passion project, and am taking my time with everything so i’ve turned off AI features. i. introduction so over the 2025-26 season i’ve had to deal with the onshape for ftc plugin and left unsatisfied. its search performance was shoddy, it occasionally went out, the parts library was limited, and there was unintuitive duplication for parts like channel and shafting. as i explored the frc cad ecosystem, i came upon MkCAD, and now FRCDesignApp. i liked how it was open source, the search function worked a lot better, yet… i felt like it was inadequate of a solution. why not frcdesignapp? well, because i wanted to make one that generates the database at build-time, i wanted semantic type-as-you-search, and i had ambitions to—well, aside from having it under the ultro moniker—integrate it with a future inventory system. i also wanted it to be written in full stack typescript rather than a python backend, since it would allow for further front-end optimizations like SSR and asynchronous data streaming with type safety. design goals search should work with many different types of queries. starting with part numbers, a search for “1120-0014-0360” should return with that sku. “14 hole gobilda u channel” should return first a 14-hole regular u channel, then low side, then mini low side u channel, then 13 and 15 hole counterparts; “gobilda c channel” should return low side u channel, then mini low-side; “bearings for gobilda” should return bearings not only from gobilda, but also 14mm flanged bearings from mcmaster, etc etc. i want the search experience to far surpass other sites’ catalog websites so you’re incentivized to stay in one place and not have to go to mcmaster-carr or west coast products or gobilda for their search experience. search should be as fast as humanly possible. i’ll get into the architectural implications of that in a bit, but results should show up as you type, should still be fast on shoddy internet connections, and should render results quickly to the screen. in the case of URL-inputted queries, the page should partially render, with the background UI fully loaded and the search results asynchronously given; SSR should not be blocked by result fetching. search should be programmatic — you can access product information through a REST API, perhaps even GraphQL for getting a bulk list of relevant products. this allows the app to integrate into inventory systems (as mentioned before), external developments, and further incentivize contribution to the database. contributing new parts should be trivial — the repository should have clear instructions as to how to open a PR, cli utilities to fill out information, and a clear workflow overall even for those who don’t know how to code. accounts can select favorites , and their search results similarly adapt to put higher weight on semantically similar parts. SKUs and onshape parts should be parametric when possible, and first party when possible. for example, rev has official onshape models for most, if not all of their SKUs. but on that note, if you want things to be parametric and relational so that it’s clear that something like shafting has different variations in length, channel can come in different hole numbers, gears can come in different sizes, motors in different gear ratios and…"}]...
|
Title: fabrica by @acandoo | Stardance
Devlogs:
h Title: fabrica by @acandoo | Stardance
Devlogs:
hey everyone! i’ve been thinking through this project in notes (and in my head) for a little while now, and just got started putting things down in code starting friday or so. i’m working on fabrica, an overengineered parts library/search engine for robotics teams. currently i’ve structured out how to code static database generation, decided on repository structure, and scaffolded tooling and helper functions for later use, and am now working on a data scraper for goBILDA parts. it’s not the fancy website stuff, but getting all of the data was something i wanted to get down first. this is a passion project, and am taking my time with everything so i’ve turned off AI features. i. introduction so over the 2025-26 season i’ve had to deal with the onshape for ftc plugin and left unsatisfied. its search performance was shoddy, it occasionally went out, the parts library was limited, and there was unintuitive duplication for parts like channel and shafting. as i explored the frc cad ecosystem, i came upon MkCAD, and now FRCDesignApp. i liked how it was open source, the search function worked a lot better, yet… i felt like it was inadequate of a solution. why not frcdesignapp? well, because i wanted to make one that generates the database at build-time, i wanted semantic type-as-you-search, and i had ambitions to—well, aside from having it under the ultro moniker—integrate it with a future inventory system. i also wanted it to be written in full stack typescript rather than a python backend, since it would allow for further front-end optimizations like SSR and asynchronous data streaming with type safety. design goals search should work with many different types of queries. starting with part numbers, a search for “1120-0014-0360” should return with that sku. “14 hole gobilda u channel” should return first a 14-hole regular u channel, then low side, then mini low side u channel, then 13 and 15 hole counterparts; “gobilda c channel” should return low side u channel, then mini low-side; “bearings for gobilda” should return bearings not only from gobilda, but also 14mm flanged bearings from mcmaster, etc etc. i want the search experience to far surpass other sites’ catalog websites so you’re incentivized to stay in one place and not have to go to mcmaster-carr or west coast products or gobilda for their search experience. search should be as fast as humanly possible. i’ll get into the architectural implications of that in a bit, but results should show up as you type, should still be fast on shoddy internet connections, and should render results quickly to the screen. in the case of URL-inputted queries, the page should partially render, with the background UI fully loaded and the search results asynchronously given; SSR should not be blocked by result fetching. search should be programmatic — you can access product information through a REST API, perhaps even GraphQL for getting a bulk list of relevant products. this allows the app to integrate into inventory systems (as mentioned before), external developments, and further incentivize contribution to the database. contributing new parts should be trivial — the repository should have clear instructions as to how to open a PR, cli utilities to fill out information, and a clear workflow overall even for those who don’t know how to code. accounts can select favorites , and their search results similarly adapt to put higher weight on semantically similar parts. SKUs and onshape parts should be parametric when possible, and first party when possible. for example, rev has official onshape models for most, if not all of their SKUs. but on that note, if you want things to be parametric and relational so that it’s clear that something like shafting has different variations in length, channel can come in different hole numbers, gears can come in different sizes, motors in different gear ratios and…
we are getting somewhere!!! now a couple of more rounds of bug fixing and i should have all the metadata for products for goBILDA! from there it’s manual overriding and filtering to remove bundles and anything of that sort.
alright, final bit of work for today. since the last log, i put some more logging messages and files to diagnose different issues. i caught one where sometimes goBILDA didn’t redirect through an HTTP code but rather through <meta http-equiv="refresh"> which was annoying. and some category pages had a link to other category pages given the SAME SET OF ATTRIBUTES AS PRODUCTS, or took from an entirely different place in the DOM because the product list is CLIENT SIDE RENDERED. i’m sorry but i am NOT about to pull out puppeteer for a (presumably) 1,000+ SKU website. i also added a cache to fetched URLs so that i didn’t bombard goBILDA with duplicate requests as i worked. also on that note, i put in a custom user agent while i was at it so that they hopefully don’t block what i’m doing 😊 tomorrow i’ll also probably work on deduplicating DOM queries by keeping track of a list of scanned product pages and filtering out pages that are irrelevant for the catalog’s purposes.
i crashed node 😭😭😭 the final pieces were adding lists of already-visited websites, and rate limiting myself so i don’t crash goBILDA. whoops…. i need to break up writes to the file in smaller pieces. i’ll probably write per product processed. and since things are bound to go wrong with a scraper there’s probably some exception to the rule with some random product that doesn’t make it in the scraper / isn’t processed correctly.
VICTORY!!! from here it’s a matter of refining the scraper to make sure there are no duplicates / useless parts / specs are good. the final pieces of the puzzle were removing JSDOM caching to reduce memory usage (since i already check for duplicates reasonably well before the request is made), adding some more filtering for irrelevant products, manually garbage collecting with global.gc() (!!), and allowing for errors in product parsing. i could probably make it more robust by forcing all the categories to be evaluated before products.
not much to update on, but i decided to clean up the commit history with gitmoji . shout out gleamlins for showing me gitmoji! i wanted to implement it for this project because i think it’s really neat and cute! there are probably still some stylistic discrepancies/wrong things within the commit history but for now my perfectionism is satisfied :)
wow i haven’t logged in a while! the past few days were the last bits of school, so not a lot of work done per day. i’ve decided to change up how the partdb is built, since i discovered unplugin-macros which really helps streamline build-time logic and the final export of the package. combined with tsdown the whole build situation of the package is a lot cleaner. in addition, i added filesystem-level caching so that when this eventually gets built on github actions it will have a cache folder that it can use, and with proper expiration handled. i still have to fine tune what computations get cached, but soon hopefully cached builds won’t take the heat death of the universe to complete! since the caching is time stamped it also makes garbage collection of the cache trivial and runnable in a github action. that code isn’t written yet but isn’t too hard. for that reason i also bumped up the node version because i wanted to use Temporal to deal with timestamps and durations :) in terms of other minor things: a rudimentary site package was added, with a template astro site. i did research into different serverless platforms lint errors fixed more code comments/inline documentation made the just build recipe functional proper error message logging for product page parsing i’ll get out of my meta-work slump soon, soon!!
so i tried caching webpages and… um… whoops. there are so many footguns with concurrency… i had to modify a record to also store a promise chain so that writes didn’t happen concurrently… and i had run out of memory on a rebuild with cached elements which i assumed was because i was thrashing the filesystem by immediately closing file descriptors once things were read, so i set up an object to keep the file handles around but because of the async boundary they had gotten garbage collected… urgh
many OOMs and attempts to use chrome devtools later… i think tomorrow i’m going to switch out everything to use cheerio instead of JSDOM to reduce memory usage, then maybe add a thread pool while i’m at it. currently i’ve just bumped the “old memory” limit up to 16 WHOLE GIGABYTES…. gah javascript.. i’ll also have to debug a JSON parsing problem with a few of the pages. i have no idea what it might be aside from JSON.parse being finicky or JSON.stringify not properly escaping but i highly doubt that’s the reason.
got busy this past week with a bunch of other duties, so slow progress. i set up a thread pool class with a promise API and type safety… which is kinda overkill but that’s a bit of the whole goal with this project :p but so far i offloaded initially parsing the web page for a <meta http-equiv="refresh"> and determining the link. which is not a long task, so not a huge main thread blocker! but it would allow me to cement the code structure with the worker pool. the thing was, i really wanted to optimize this build, and the useCache hook i wrote only supported JSON stringifyable values, which means no array buffers. the fetch API allows you to get the response directly as an ArrayBuffer, which, once you mark as transferrable, means that a thread can work directly on the buffer without having to copy all of the data. and since memory contention is one of the two problems i’m trying to solve here… that’s incredibly useful. the thing was, i couldn’t just insert it directly into the JSON cache entry since it would be parsed by JSON.parse as a string, so i set up another content-addressed directory ( cache/blobs instead of cache/files , where most things go) and modified the types to ensure that it was still type safe. turns out… making that type work out in the function was a PAIN,, especially considering i had generics involved. i was able to work through it with an atrocious amount of as keywords sprinkled throughout, but it’s not very elegant. in terms of the site, i opened up figma and started working on some prototypes! i’ve been meaning to learn figma and figured this was the perfect opportunity to go through the process. i love how, for the most part, everything is really intuitive. but the “frame” and “group” and “component” concepts being separate things was really confusing to me. i’m really tired and about to lose my train of thought but for tomorrow i’ll work on porting the rest of the DOM processing over to the worker threads and to cheerio. i’ll also probably have to rethink how code is organized because now the “adapter” is split across two files. there’s definitely a way to make things more elegant, though.
I LOVE METAPROGRAMMING okay so for some background i worked on moving the getPages method of the goBILDA adapter to a worker thread method (which parses the goBILDA sitemap to get the main category pages), but i realized from the type error that occurred that the whole type inference debacle got messed up as there was no url parameter in the method input… so now i’ve unleashed a generic-type guard-inference inferno. typescript is a turing complete language combined with a turing complete type system… but at least there aren’t any generics explicitly declared within the code, nor any as keywords sprinkled through this section!
update: i’ve moved all of the logic to use cheerio and worker threads! i tried running just build , and… of course, it didn’t work. a few of the issues were easy to diagnose: i forgot to create a directory on startup, or put .js instead of .ts as a dynamic import path, or potentially had multiple concurrent writes that could occur. but now, i’m encountering an issue where my requests don’t want to transfer, which i initially assumed was because the object had too many layers.. but trying structuredClone on a similar object works. for other work: i’ve finally removed JSDOM from dependencies, and also updated pnpm on the side. i enabled sorting imports, which helped clean up some of the mess that was accumulating. i added helper scripts for dealing with the cache. finally, i moved the rate limited fetch within the goBILDA adapter and passed as an argument to helper functions, which would allow for future adapters to have their own rate limits.
most of the parsing issues have been fixed! the final list is 1321 items, which is the same amount as previous scrapes, so i don’t think there are any issues there. a couple things still worry me a bit (but they’re not things that hinder progress elsewhere): the build command lingers after it says it’s finished building. i suspect it’s because some promise ends up never resolving, but i’ll have to look into that further the specifications field is messed up. i should also add a string sanitizer later on to the helpers file, but in the migration to cheerio something with the specification parsing logic got messed up. a cached build still takes two minutes. ideally i would like it to be more around the 30 second range, but i have to figure out if there are any bigger things worth caching. there’s just a type error i have to deal with when it comes to error handling, and everything should be good! i’ll move on to more site prototyping since i have a dummy database to test things against.
IT WORKS!!!!!!! okay so for some reason the type signature of the search library that i’m using, flexsearch, didn’t give the right type signatures for when i put in enrich: true on searching. i had to do some manual array processing, which probably slows down the results a bit but it’s still under 15 milliseconds max. it’s incredibly fast, it seems like the bundle would work for edge functions so it’d be similarly fast in real life, and i can’t wait to implement the service worker backend to make things even faster. from here i want to refine the querying so you can filter by manufacturer, categories, etc. and refine the database items. and also have a pretty UI :)
i don’t have much to share so far, been reading documentation & doing miscellaneous work, but right now i’m getting myself carried away with color palettes and fonts and branding design first. i designed a lil’ logo! i’ll update the banner for this project to include the logo and typography and whatnot, but i decided on fonts (dm sans for display, ibm plex sans/mono, i know hurr hurr very original). originally i was going to use space grotesk as the display/heading font but i didn’t want my website to look too much like axon robotics et al. and i didn’t want to use orbitron because i feel like the wordmark should be a distinct font/character. i emailed gobilda asking about CAD file scraping and their GraphQL endpoint that I discovered (which unfortunately needs credentials…), which gives me an excuse to not worry about the parsing issues with the scraper until they email me back or a week has passed. so in the meantime, i’ll work on web design. in terms of pure functionality though i did also add a static product page generator for each of the products, which was why i thought astro was a really great fit for this project since i’m focusing on moving more stuff to build-time and improving DX instead. i do want to set up an account system and auth and all that jazz because the next big feature would be automatic PRs from suggestions, that just adds/modifies a json file in a partdb overrides folder in the repo. that’ll also give me an excuse to validate my monorepo design by putting some of that logic in another package. finally, i know in my ai policy i’m using ai very minimally and i’ve disabled ai features in vscode… but could we let this one slide :) why those hex codes specifically iykyk
okay, take two on writing through this devlog since stardance just consumed my writing since it went over the word limit… i did a lot of work since my last devlog with the UI, from view transitions to SSR to SEO to wrestling with build tools and broken npm packges. i don’t have the energy to write through everything again but i do have this demo that shows off just how cool view transitions are, the speed of SSR (though i had to make it cold start, which again i can’t be bothered to write about why a second time), and graceful degradation for javascript-less environments. next up i’m working on some housekeeping stuff with sorting tailwind classes, then formatting the search page, wiring up view transitions between the search page and product pages, more DOM optimization with better solid.js state management, and maybe adding tanstack virtualizer (though it’s kinda overkill, but allows me to just set every image to eagerly load) and keeping support for no-JS. please stardance team could you fix entries getting sent into the void i wrote a really good devlog entry the first time around…
productive day of UI work today! i started off by taking care of some meta-work within the repo. i added a pre-commit hook for linting and formatting, added tailwind class sorting to my formatter config, pushed more vscode settings and extensions to the repository, updated pnpm, and also put a pre-install hook so that others that try setting up the repository without pnpm will get a warning (at least in theory). in the future i want to add a hook to enforce gitmoji but i’ll leave that for free time when i’m less motivated to code. i added a view transition for the logo going into search. i just put the logo within its own Solid component so that both the home page and search have the same view-transition-name . i also made it adaptable on the search screen so it collapses to its smaller form by just cropping the svg instead of making another network request. though if i wanted that as flexible as possible i’d probably modify the <picture> element to also media-query-match against device width. that way graceful degradation also works JS-less. product cards look a lot nicer! now unfortunately i have a bunch of duplicate js styles across a lot of components, so i’ll probably set down some base styles or classes. what i would love for CSS are tailwind utility classes where the groupings are build-time optimized. there are still a couple of things on my plate UI-wise: some of the UI cards aren’t occupying the full width. idk why since i set their width via w-full but idk i’m probably going to switch out the UI to more of a grid formation anyways. turn the cards into a grid formation. they’re way too long and taking up valuable space. figure out what to do with text. i tried getting only the first sentence, i tried overlay: hidden , i’m not sure but am currently leaning towards some sort of blur/fade effect. make the <Suspense> boundary narrower. my current desired functionality is for all the cards to “gray out” but with the original item list length, before the resource is fetched and the dom update happens. minimal DOM update on resource fetching. i need to swap out the signal-backed Resource for a store-backed one that reconciles, but i still have to figure out solid.js’s API shape. and also tanstack virtual but i need to do cards first. product pages.
yeah so i ended up working more… i finished up the transition to card view. i had a lot of trouble because i was putting the shrink/grow/basis tailwind classes in the <article> tag within the <li> which meant that all the flexbox stuff wasn’t applying properly. 😅 i figured out how to get that ellipses text wrapping. for some reason i couldn’t get overflow: hidden + whitespace: nowrap to work without completely breaking the flow of the page, but line-clamp did work! a bunch of miscellaneous work. added a no-JS warning to the /search page, changed the tracked event for the initial search page preload and filtered so it was only mouse hovers, stricter pre-commit hook, font preload, and color changes. i need to work on the product pages next, but also i have been caught by scope creep.. i’ve decided that in the future i want to add auth and user/organization work and an actual database since now i want the project to become an inventory system itself. it makes sense UI wise and i’ll get to play with databases finally. for first ship though i’ll probably just use localStorage. since the attached recording i’ve added placeholder buttons for adding to favorites and the shopping cart… the favorites button is easy enough, it’s just a toggle, maybe some cute css animations, though whether the auth solution i end up (probably betterauth) supports it is a whole other story… the shopping cart is a different level of hard JS-less though especially since there’s not enough UI space to add a quantity amount. my plan for the interaction was that hover inactive would give the “Add to shopping cart” dialog, click inactive -> active opens up a quantity dialog, setting the quantity to 0 resets the buttom, and hover active brings up the quantity dialog again.
hello! so far i’ve been working on responsive/adaptive design for the search page. i’m kinda putting off doing the individual pages in place of refining the search page. the logo is properly sized on safari. i don’t know what is it about safari and fit-content that just makes everything break horribly, but i had to choose manual widths for making the logo adaptive instead of relying on the image width. for some reason the logo tile gradually increases width despite me setting the width as static (gah css!). at some point in the future i should integrate the UI work with browserstack/tests. the hero page is now a proper <a> element instead of an <input> with focus intercepted. to keep compatibility with no javascript, it SSR’s as an <input> element before switching out to an <a> on component mount. this means that on mobile the keyboard won’t open and close when trying to search. unfortunately iOS safari doesn’t support autofocus… the nav bar is now the same width as the cards. the culprit was an empty div… i had adjusted for flexbox gap being janky by adding more padding, but now that i eliminated the element i could remove the padding composition and make everything the same width. this does mean that when scrolling down the curvature of the search bar/logo isn’t consistent with the title cards, but hey everything’s a compromise. bigger card buttons & button icons. they feel almost too big, but i know that click targets are important for accessibility. i also made the “Learn more” button a full pill instead of a rounded square. i guess the convention i’m intuitively going for is “rounder UI elements = more desired/friendly path”. i’m still upset that the icon library doesn’t work, it would be so nice. even more adaptive cards. i set up the breakpoints so that the image moves vertical first, then once there’s not enough space for the “Learn more” button the star and basket, the star and basket disappear since their functionality can be on the product page.
hello again! my main focus for this devlog has been on experimenting with loading/transition behavior for queries. before, the page would remove all of the card components and just show a “Results: loading…” screen and an unstyled <li> loading element, but i wanted to improve that and make it more animated and less “flashy”. the first part of the transition comes on the “leading edge”, when the user updates their query and information has not yet been fetched. if i let the previous cards and results stay up, the site will be perceived as laggy and not responsive. so, my instinct would be to place a “skeleton UI” but that in itself would be a DOM update/layout reflow, which i wanted to avoid for performance reasons. i tried blurring individual card elements, but i didn’t like how it looked. maybe i settle with a skeleton UI or come up with some other combination of CSS effects, i don’t know. either way, i didn’t want the effect to show up if the request hits the cache, so instead of using solid.js’s existing useTransition hook, i manually combined the underlying startTransition function with my own pending signal, which i only set to true if the query was not found in the cache. this does mean that on cache hits the code looks up the map two times (once on the initial check for the pending signal, another on the resource fetch), but i don’t think trying to optimize that away with a boolean or something would end up being faster than just letting v8 optimize the double Map lookup. for the second part–or, trailing edge–of the interaction, i had this idea of constructing a view transition so that cards that showed up previously would move nicely to their resulting spot. that unfortunately made things really laggy, especially since view-transition-name would be declared on both the cards themselves and many items within the card, as those would need to be labeled for the view transition to the product page. the visual effect was really nice though, so my plan to make it work is to finally introduce virtual scroll… yes i know i’m fixing more javascript with more javascript, but at least the site works without it! right?! additional minor thoughts: i’m starting to not like the star/basket buttons. i was looking online for product card inspiration, and most of the designs i found were a lot more minimal, many of them even omit the “Learn more” button for a cleaner visual aesthetic. for now i’ll probably change out the buttons to just the underlying icon outlines, but keep the bounding box. my qualm with removing “Learn more” is discoverability on mobile, since i couldn’t disclose product page functionality through a hover effect. maybe once the card reaches a certain height on the viewport then i’ll activate the hover progresive disclosure for mobile, but idk. maybe i should fetch in the background? with the transition effect, it would be nice for it to apply when you backspace, and at that point the site isn’t really doing any more network stuff aside from prefetching hovered cards, so i could fetch all of the sliced versions of the query (i.e. for “beam” it would fetch “b”, “be” and “bea” in the background). should i persist the results count on scroll? maybe integrated within the navigation bar, or in smaller text in a pill under, idk....
|
1793293043.5498035
|
1793897843.5498035
|