GSA used to maintain a combined catalog that was refreshed a few times per year and searchable.
I’m also not sure if there’s still a requirement for agencies to keep their own code.json up to date. It’s hard to tell when each department refreshes, but it seems like HHS hasn’t updated theirs since March of 2022.
In this case, GSA used to scrape and combine and then stopped. Also GSA used to ask agencies to update their inventory and then stopped.
I think if GSA just asked, it would increase the recency and completeness of the code.jsons.
Also, what’s kind of funny is that since open source projects, by their nature, are publicly visible, GSA could probably just scrape and combine together and not rely on lots of different agencies to have their own processes.
> Every 20 minutes it grabs the latest copy of that JSON endpoint, pretty-prints it (for diff readability) using jq and commits it back to the repo if it has changed.
> This means I now have a commit log of changes to that information
A static site builder can rebuild just the pages of the site that need to be changed once in a Github Action that updates the site when a Pull Request is merged to main.
Though, if the data quality is insufficient because the data sources are not updated, then downstream apps and static sites that depend upon the data are also insufficient.
There are ways to do this, but GSA just doesn’t do it.
Years ago they used to have a system that would combine all the code.jsons into a single db and provide a query interface. They stopped funding that system and redesigned this static site. But could have used GitHub actions or something to fetch and combine the code.jsons and do everything client side. That still wouldn’t have needed maintenance costs.
datasette is a webapp and CLI built on SQLite and Python. datasette-lite is the pyodide + WebAssembly build of datasette which can be served as static HTML, JS, and WASM SQlite.
> Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API.
> Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world.
> You can use this tool to open any SQLite database file that is hosted online and served with a `access-control-allow-origin: ` CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using `datasette publish`.*
> [...] You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you
> To load a Parquet file, pass a URL to `?parquet=`
There are various *-to-sqlite utilities that load data into a SQLite database for use with e.g. datasette. E.g. Pandas with `dtype_backend='arrow'` saves to Parquet.
datasette-scraper scrapes sitemaps.xml and crawls though it could surely be repurposed to instead scrape a list of code.json URLs within the datasette process, which is powered by asyncio and the asynchronous uvicorn ASGI HTTP web server.
(TIL datasette-scraper parses HTML with selectolax; and Selectolax with Modest or Lexbor is ~25x faster at HTML parsing than BeautifulSoup in the selectolax benchmark:
https://github.com/rushter/selectolax#simple-benchmark )
A GitHub Action would run regularly, fetch each code.json, save each to a git repo, and then upsert each into a SQLite database to be published with e.g. datasette or datasette-lite.
