Friday, March 28, 2025

Lookup Speed of uint Sets in Go

[]uint vs map[uint]struct{}

A wee bit of premature optimization, I was coding a thing where I wanted a set of uint, and I wanted to filter several million records against them. I had a hunch that []uint with binary search would be faster than hashing the key and doing lookup in a map[uint]. I was right, but only if the set of uint is smaller than 1000. Below 1000 elements, slices.BinarySearch() is faster, somewhere between 100 and 1000 is the crossover point in this test and after that map[uint]struct{} is faster.

I'm not couning setup time to build the set, just lookup time to do 100_000_000 lookups against the set.

ns/op (lower is better)
set sizearraymap
103.9525.911
1005.8437.104
10007.8996.008
1000010.356.746
10000013.159.547


Friday, March 21, 2025

How to develop an atproto firehose source with relay and goat

Big changes are coming to the atproto firehose with sync 1.1 and major code changes to the relay. This blog post is about developing a custom firehose source (e.g. a PDS implementation) and ensuring it is compatible with Bluesky's relay and firehose consumer libraries.

First, install Go 1.23 or newer

Then, git clone the Bluesky indigo repo and build the relay

git clone https://github.com/bluesky-social/indigo.git
cd indigo/cmd/relay
go build

If you have your custom PDS out deployed somewhere with https enabled, start the relay, it will log to stdout/stderr:

./relay --admin-key hunter2 --api-listen :2470 --metrics-listen :2471 --time-seq

If your custom PDS is local or otherwise plain http, `--crawl-insecure-ws` allows non-https PDSes

./relay --admin-key hunter2 --api-listen :2470 --metrics-listen :2471 --time-seq --crawl-insecure-ws

 In a separate terminal, tell your relay to start crawling your PDS and adjust its rate limits

curl --silent --include -H 'Authorization: Bearer hunter2' -H 'Content-Type: application/json' --data '{"hostname":"mygreatpds.address.tld"}' http://127.0.0.1:2470/admin/pds/requestCrawl

curl --silent --include -H 'Authorization: Bearer hunter2' -H 'Content-Type: application/json' --data '{"host":"mygreatpds.address.tld","per_second":5000,"crawl_rate":50,"repo_limit":10000000,"per_hour":50000000,"per_day":500000000}' http://127.0.0.1:2470/admin/pds/changeLimits

Is it working? Let's browse the firehose with `goat`

cd indigo/cmd/goat
go build
./goat firehose --relay-host ws://127.0.0.1:2470

If you create a record in your PDS, you should see it on the relay firehose.
You can also `goat firehose` your PDS directly. Ideally if you have both running you should see an event in both within a few milliseconds of each other.

The other thing to check is /metrics on your relay:

curl --silent http://127.0.0.1:2471/metrics

One useful filter is to get events tied to your PDS name

curl --silent http://127.0.0.1:2471/metrics | grep mygreatpds

That's about it! If all the events you emit out of your com.atproto.sync.subscribeRepos endpoint gets through to `goat firehose` reading the relay, you're in good condition to be part of the atproto world!

Friday, August 23, 2024

Local Development With an AT Proto PDS

Using docker and curl to do the first steps of what could be creating an atproto app.

The Bluesky Personal Data Server (PDS) comes with instructions and automation for taking over a server and running a node connected to bsky.app, but I wanted to run a local disconnected PDS so I just muck around with creating fake users and posting data as if I was creating a new atproto app.

Inside their system is a docker container, and we can run that by itself:

docker image pull ghcr.io/bluesky-social/pds:latest

You'll probably want data somewhere other than /tmp

mkdir -p /tmp/pdsdata/blocks

PDS_ADMIN_PASSWORD=$(openssl rand --hex 16)
PDS_JWT_SECRET=$(openssl rand --hex 16)
PLC_ROTATION_KEY=$(openssl ecparam --name secp256k1 --genkey --noout --outform DER | tail --bytes=+8 | head --bytes=32 | xxd --plain --cols 32)
PDS_DATADIR=/tmp/pdsdata
PDS_HOSTNAME=foo.bar.com

cat<<EOF >/tmp/pdsdata/pds.env
PDS_HOSTNAME=${PDS_HOSTNAME}
PDS_JWT_SECRET=${PDS_JWT_SECRET}
PDS_ADMIN_PASSWORD=${PDS_ADMIN_PASSWORD}
PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX=${PLC_ROTATION_KEY}
PDS_DATA_DIRECTORY=/pds
PDS_BLOBSTORE_DISK_LOCATION=/pds/blocks
PDS_DID_PLC_URL=https://plc.directory
PDS_BSKY_APP_VIEW_URL=https://api.bsky.app
PDS_BSKY_APP_VIEW_DID=did:web:api.bsky.app
PDS_REPORT_SERVICE_URL=https://mod.bsky.app
PDS_REPORT_SERVICE_DID=did:plc:ar7c4by46qjdydhdevvrndac
PDS_CRAWLERS=https://bsky.network
LOG_ENABLED=true
NODE_ENV=development
EOF

docker run -d --name pds --restart=unless-stopped --network=host -v /b2/bolson/Bluesky/pdsdata:/pds --env-file=/b2/bolson/Bluesky/pdsdata/pds.env ghcr.io/bluesky-social/pds:latest

With that, it's probably running. If you ever need any of the secrets, there in pdsdata/pds.env

Run a couple quick queries that should return server state
curl -L 'http://localhost:3000/xrpc/com.atproto.sync.listRepos?limit=100'
curl -L 'http://localhost:3000/xrpc/com.atproto.server.describeServer'

Let's create a user. First, create an invite code:
curl -L -X POST 'http://localhost:3000/xrpc/com.atproto.server.createInviteCode' --user "admin:${PDS_ADMIN_PASSWORD}" --data-raw '{"useCount":1}' -H 'Content-Type: application/json'

This will return something like
{"code":"foo-bar-com-zx6uq-cjkle"}

Then actually create the user:
curl -L -X POST 'http://localhost:3000/xrpc/com.atproto.server.createAccount' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
--user "admin:${PDS_ADMIN_PASSWORD}" \
--data-raw '{"handle": "bob2.foo.bar.com","inviteCode":"foo-bar-com-zx6uq-cjkle","email":"bob2@foo.bar.com","password":"hunter2"}'

 This returns a bunch of json about the created user, the parts we need for next steps are "did" and "accesJwt", like: 

"did":"did:plc:ks35mpkznuqalfrr3drm7tfu""accessJwt":"eyJhbGciOiJIUzI1NiJ9.eyJzY29wZSI6ImNvbS5hdHByb3RvLmFjY2VzcyIsImF1ZCI6ImRpZDp3ZWI6ZG8uYm9sc29uLm9yZyIsInN1YiI6ImRpZDpwbGM6a3MzNW1wa3pudXFhbGZycjNkcm03dGZ1IiwiaWF0IjoxNzI0NDI3NTk0LCJleHAiOjE3MjQ0MzQ3OTR9._uG-PTAqNLQT7PqohWFv-0lvyQtz_ud5XOwCqnjDPqs"

Let's put that in shell to make the next commands nicer

ACCESS_JWT="eyJhbGciOiJIUzI1NiJ9.eyJzY29wZSI6ImNvbS5hdHByb3RvLmFjY2VzcyIsImF1ZCI6ImRpZDp3ZWI6ZG8uYm9sc29uLm9yZyIsInN1YiI6ImRpZDpwbGM6a3MzNW1wa3pudXFhbGZycjNkcm03dGZ1IiwiaWF0IjoxNzI0NDI3NTk0LCJleHAiOjE3MjQ0MzQ3OTR9._uG-PTAqNLQT7PqohWFv-0lvyQtz_ud5XOwCqnjDPqs"

The first couple 'admin' APIs used HTTP user:password with the user as 'admin' and the password set for the whole server in setup back in pds.env. Later actions as a user put the access JWT blob into the Authorization header.

Post a record! (note validate=false because I'm not posting proper lexicon data)

curl -L -X POST 'http://localhost:3000/xrpc/com.atproto.repo.createRecord' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Authorization: Bearer '${ACCESS_JWT} \
--data-raw '{"repo":"did:plc:ks35mpkznuqalfrr3drm7tfu","collection":"com.bar.wat","rkey":"aoeustnh1234098","validate":false,"record":{"a":"foo","b":1234}}'

That should return json with info about the new created record. Then I did that a few times with different "rkey" and tweaking the record contents each time.

List the records back out:

curl -L 'http://localhost:3000/xrpc/com.atproto.repo.listRecords?repo=did:plc:ks35mpkznuqalfrr3drm7tfu&collection=org.bolson.vote'

Should return json:

{records:[{...},{...},{...}], "cursor":"some rkey"}

Great! Now I feel like I'm ready to put and retrieve data in an atproto PDS and start developing an app. There are atproto SDKs, but I wanted to make sure I really got it by doing it all in curl first.

Sunday, August 4, 2024

Goodbye Blockchain

In 2018 I started working in blockchain/cryptocurrency stuff at Algorand. They had a clever new system that fixed the gigawatt-waste problem of Bitcoin and was 100x faster, and I figured "I guess blockchain is something people want, and this is good tech, I'll try it".

I spent 4 years there and was always proud of the engineering quality and the culture had a healthy skepticism and distance from the hype around blockchain things.

What I really wanted from the dream of cryptocurrency is to unseat the Visa/MC duopoly and their $.30+3% drain on every transaction. I wanted transactions of any size, censorship resistant, for $0.01 or less. But current climate and laws prevent that.

I'm pretty tired of crypto-bro culture. I don't care about NFTs. Nobody should care about NFTs. "shitcoins" are explicitly value-less and people in the biz call them shitcoins and yet still worry about optimizing their tech to support more shitcoins. This sucks.

Crypto-bros apparently don't care that Twitter is a cesspool and feeding that machine makes the world a worse place. And they don't care that Bitcoin wastes gigawatts per year and must be doomed to fail.

One of the biggest innovations of Bitcoin may be that it created a distributed leaderless Ponzi scheme. Hype it enough and create the next generation of suckers to buy in and buy out your share at a profit.

The crypto markets seem to be entirely hype driven. Worse tech with more hype can beat better tech. It's a big downer when I want to take pride in working on better tech. I no longer care to take a bet on which blockchains will survive to 5 or 10 years from now. I quit.

New job: try to make the world a better place through good social media (and make a buck in the process) at Bluesky Social. I'll mostly be on backend just trying to keep the place running reliably and efficiently, but I'll be doing my part on new features too.

Follow me there at:
https://bsky.app/profile/bolson.bsky.social

Saturday, June 29, 2024

How Much Complexity Can Your Tribe Manage?

 I've heard of a Sci-Fi thought experiment: how many people do you need to ship on the 100 year journey to another solar system? We'd need some number of people to specialize in all of the things required for modern civilization, and some number of people to do all the farmer/plumber/electrician jobs to keep the world running. Some Sci-Fi brain thought the number was 10_000 people.

Ok, now think about your small company of 100 people. How much complexity can you manage? How many systems can you maintain?

I think I've worked a few small companies where the answer should have been 'fewer'. And a few less crucial things got neglected, and it was messy. Maybe there could have been a better way to consciously wind down the complexity and do fewer things better.

Thursday, June 20, 2024

Bitcoin Must Die

Bitcoin is horribly wasteful. The University of Cambridge Bitcoin estimate is that it currently burns 150 terawatt-hours per year. (2023 US generation was 4178 TWh.)

Multiply that by $0.0773/kWh (March 2024 EIA cost to industrial sector) and we get the operational cost of Bitcoin (power alone! doesn't include cost of hardware!) at $11.5 billion dollars a year.

As of this morning (2024-06-20) Coinbase estimates that all Bitcoin is worth about $1.3 trillion. The electricity operating cost should be depreciating that value by 0.9% per year (hardware costs should take another chunk out).

According to blockchain.com, there were 153_415_993 bitcoin txns in 2023. Divide out the operating eletricity cost and that's $75.58 per transaction. Recent txn fees are more like $5-$6. The Bitcoin system loses $70 on every transaction. The $1.3 trillion valuation is based on hype and hope and a bubble and the inflow of new rubes on the broad base of the lowest run of the pyramid scheme.

What a sustainable blockchain would be

Transaction fees for moving value from here to there (I think we can do better than credit card ($0.30 + 3%) or FedNow $0.05).
Transaction fees for recording a fact (comparable to notarizing, or trademark/copyright filing fees).
If all those fees are enough to pay for operations, and an acceptible margin of profit, congratulations, you have a financial institution.

Those 153_415_993 transactions on Bitcoin in 2023 come to an average of 7 transactions per second. In 2019 I demonstrated the Algorand blockchain achieving 75 TPS running on three Raspberry Pi 3B+ toy computers. Algorand*, Aptos* and other newer blockchains continue to offer transactions for under a penny, and could be sustainable at this rate.

BUT, the regulatory environment around blockchains continues to be murky at best, crushing at worst. It remains to see how this will settle out in jurisdictions around the world and if those markets will continue to have enough business to support the several blockchains competing to grow their network effect and win in the market.

(* I have a stake in Algorand and Aptos, other blockchains are probably also technically competent ;-) )

Monday, January 22, 2024

Cloud compute with SSD is a great deal

Want fast data processing? Use attached SSD on your cloud compute nodes. It's a great deal.

Consider the AWS m6gd.8xlarge instance type. It has an attached 1900 GB NVMe SSD. One source benchmarks it at 444_552 Read IOPS, 186_467 Write IOPS. It costs $1055.87/Month.

Right next door is the m6d.8xlarge instance without SSD for $899.36/Month.

Using the AWS price calculator, replacing that storage with EBS and provisioned IOPS we have:

gp3 16_000 IOPS 1900 GB: $217.00/Month

io2 64_000 IOPS 1900 GB: $3,773.50/Month

Neither of those plans is cheaper than the faster attached SSD.

This is in terms of AWS, but I'm pretty sure it applies to GCP and other providers too.

The benefit of EBS is that it's more durable, survives instance shutdowns and crashes. But, I've worked at several jobs where a large-ish chunk of data needs fast processing, but will be extensively backed up or otherwise redundantly stored elsewhere. Losing a server would be a temporary inconvenience easily recovered.

Wednesday, October 25, 2023

Five Developer Anti-Patterns

on my mind today, after 20+ years of writing code

1. Premature Optimization. Well known. You made the code more complex and less clear to make something faster, but it didn't need to be faster, and you wasted your own work making something faster that didn't need to be faster. (faster/smaller/other optimization goal)

2. Premature Generalization. Only slightly less well known. Making the system that could be anything, could grow in any direction, has hooks for every eventuality, but those most of those things won't be used and were a waste of time to build and add complexity and may ultimately make it harder to develop in the direction the code actually needs to go.

3. Dependency Bloat. Everything you import is a possible bug and a possible supply chain attack. It slows down builds. When I code review a pull request that adds a library maybe I should code review the entire library added. (A few things get a pass for sufficient reputation and community usage.)

4. Excessive Cleverness. Yeah, you could do it in a perl one-liner, but maybe you should write it plainly, opened up into normal nested loops on several lines to show all the parts clearly.

5. Spooky Action/Magic. If I call thing.foo.bar() is that exactly what happens or does hidden code happen automatically? If thing.foo is type A but can be automatically converted to type B to run .bar() on, that might be bad magic. (Ruby on Rails is the worst at this, C++ does it a bunch, and Rust does it more than it should; Go pushes back hard to the point of being a little tedious about it)

Saturday, June 17, 2023

Off Grid Starlink Power Usage

I have a rectangular dish I got early 2022. I have an off grid solar power system running it, so I like to keep track of how much power various components are using.

The Starlink system sits by itself on a smart Victron Energy inverter (a small computer and other things work on a separate DC-DC converter from the battery). The inverter posts status to a serial port about once a second. Across 883129 data points (about 10 days) I gathered data and did some analysis.

The mean AC current supplied was 0.32 Amps. The peak was 0.9 Amps. (The mean AC voltage output was 120.01.)

Here's a histogram of the number of samples at each output current:

0.2 A	88114
0.3 A	556319
0.4 A	215524
0.5 A	19858
0.6 A	2627
0.7 A	560
0.8 A	114
0.9 A	13

I've seen other reports that the Starlink terminal uses about 40 watts and we're in the neighborhood of that. Also the good news is I'm well within the specs of the 250 VA rated inverter I got for this.

I plotted the results but didn't see any pattern to the spikes. Nothing jumped out like "once an hour" or "every morning at 2am". Maybe it has to do with where the satellites are, the weather, or birds flying over head.

I'll keep collecting data but I'm not expecting any updates to this. But yay, now I know.

Tuesday, March 7, 2023

Yoga for Nerds

I grew up a nerd, which is the opposite of a jock, so embodied athletic things were not part of my identity. I was more reaching for the ideal of brain-in-vat purity. (Did band and ran 5k anyway) At various times yoga has pushed me away (too much woo-woo usually) but now it works for me, here’s how

Monday, February 6, 2023

Local Election Drama Shows Need For Open Source Elections

tl; dr: Election; certification; software oops!; swearing in; lawsuit.

The Oakland, California School Board election of 2022 is still contested. At first it seemed straightforward; the election and certification went by apparently well. Then, December 28, the Alameda County Registrar of Voters posted an oops, "its RCV tally system was not configured properly". In January the initially certified winner, Nick Resnick, was sworn in. Mike Hutchinson should have won by the corrected count, and he's pursuing a court challenge to fix the outcome in his favor.[1]

The specifics of the problem to me sound like a bug in the implementation of the "Instant Runoff Vote" algorithm used in this RCV election. From the AC ROV press release:

It should have been configured to advance ballots to the next ranking immediately when no candidate was selected for a particular round.  This means that if no candidate was selected in the first round on the ballot, then the second-round ranking would count as the first-round ranking, the third-round ranking would count as the second round ranking, and so on.

To me that doesn't sound like a "configuration" issue, but a straight up bug in the implementation of IRV. I can't imagine the behavior described as being the desired 'correct' behavior in any context. The bigger problem is that we don't know what happened because we don't have the source code. I can post my count of the election, but so what? We don't know what's happening in the official count.

I previously posted about how San Francisco elections don't quite reproduce, not with a big enough margin to change any outcomes, but it's still at least a little worrying that the official cast vote record can't seem to reproduce the officially certified counts. Where did those counts come from? What data were they using? What software and settings did they use to count the votes? We don't know. San Francisco is doing good for the world by publishing their full cast vote data, and we should thank them for that, (and Minneapolis, Minnesota, and the State of Alaska, and a few other places intermittently posting RCV data).

We need our elections running Open Source software that can be publicly verified, otherwise we kinda can't have elections be publicly verified. In this case, a hand count of the 26,714 votes is still a manageable inconvenience, but a future state-wide election with millions of votes might not be feasible.

Thursday, October 20, 2022

Off-grid Base Camp Solar Power

 800 watts of panels, 5.4 kWH of battery*, satellite internet, all I could need for work-from-woods


This was a constant tinkering project, but it always basically worked all summer.

Hackish Off-grid Weather Station (2021)

In 2020 I noted that you can build a small gadget-charging off-grid solar power system for about $200. In 2021 I deployed it to a spot in the woods of Vermont.

This is largely a tale of what kinda-worked and what I should do better next time.

Friday, October 7, 2022

Victron Energy VE.Direct hackish

I have been increasingly pleased by Solar energy components from Victron Energy. Most of their products have a "VE.Direct" port which is just 5V UART serial at 19200 8N1 (a few older battery module units are 3.3V). If you ask nicely and register at their site they will give you pretty thorough technical documentation of the protocol over that serial port. I've been using this python module for parsing VE.Direct serial port broadcast (status update just sends once a second, this may be all the data you need for many uses).

They have an official $30 VE.Direct USB serial adapter that plugs in to their equipment and Just Works. But I work with embedded stuff to know that those are some pretty cheap commodity parts potentially. So I set up with this USB to UART TTL (5v or 3.3V) module that's $10 for a 5 pack and this JST 2.0 PH cable, and for $3 and some soldering I have my hackish VE.Direct adapter cable. Worked the first time!



Wednesday, September 28, 2022

IRV Fails Again, Alaska 2022-08-16

Instant Runoff Voting, the form of Ranked Choice Voting most enacted in the US, known for decades to be flawed, has failed again.

The August 16, 2022 special election in Alaska elected Mary Peltola to US House, but more people wanted Nick Begich.

This isn't an obvious conclusion from the top line reported results, so let's dig and see how this happened.

The IRV rounds played out like this, with Begich in third and eliminated in the first round.

Round 1Round 2
Peltola, Mary S.7580391375
Palin, Sarah5893786195
Begich, Nick54009
exhausted330414626
overvote23693
active188749177570
But let's look at all the second choice votes, from Peltola and Palin also:

FirstSecond
73948Peltola, Mary S.41424Begich, Nick
8447Write-in
3635Palin, Sarah
56669Palin, Sarah32551Begich, Nick
3197Peltola, Mary S.
1214Write-in
51762Begich, Nick25699Palin, Sarah
14093Peltola, Mary S.
1391Write-in
3051Write-in1075Begich, Nick
997Peltola, Mary S.
429Palin, Sarah

Begich was the clear consensus second choice of Peltola voters and Palin voters, and he had a sizeable first-choice vote as well.

When you add it all up, the pairwise preferences look like this:
123
(1) Begich, Nick88212101500
(2) Peltola, Mary S.7951591438
(3) Palin, Sarah6369386283
Peltola had good first choice support, but much weaker second choice support, so ultimately more people wanted Begich over Peltola than the other way around.

With no additional complexity and no additional cost, we could have better Ranked Choice Voting that recognizes the democratic principle that if more people want A than B, A should win.

Monday, March 14, 2022

Time Zones, DST, Act Locally

Lots of debate about DST and time zones and should we change them.

Around where I am in Boston, with DST, sunrise varies by over two hours from 05:06 to 07:22, and sunset varies by over four hours from 4:11pm to 8:25pm. (DST is really "sunrise stabilization time".)

What I think I hear underneath is that people want their schedules aligned to the sun. The want more sun in their day. They want their waking aligned to the sun and the end of their day aligned with the sun.

Then they run up against needing a bunch of states going in on change together, or even an act of Congress. The sad fact of astronomy is that summer days are over 15 hours but winter days are only 9 hours and maybe with that scarcity of daylight we're just not going to make everyone happy.

But, we can act locally starting any time. Businesses can change their hours. School districts can change their hours. There's nothing sacred about school starting at a time named "7:35 am" or whatever. DST or not, that time could even shift a couple times during the year, maybe in steps of half an hour. If I was running a coffee shop I'd be tempted to declare "opens at sunrise" (or an hour before sunrise in some places).

Act locally, start now.

Monday, February 21, 2022

2020 San Francisco RCV Election Not Reproducible

The good news is that San Francisco publishes full ballot data from their elections [1]. It's a quarter-gigabyte zip file full of JSON and I can go digging into it and look for deeper patterns in how people rank candidates on their ballot. Is there a hidden story about who 2nd-choice candidates were and what people really want? Did IRV screw up and we should switch to Condorcet?

The bad news is that I can't reproduce their results. They publish the full data, but the program they run isn't open source.

Saturday, February 19, 2022

RCV Nearly Null Result

The Board of Elections of New York City released full vote data for the Ranked Choice Votes (RCV) for the 2021-06-22 primary election. I downloaded this data and processed it with my software to do some analysis. Out of 63 elections: 3 had different outcomes than pick-one (Yay! This way is better!) and none had different outcomes IRV vs Condorcet. I still maintain that IRV is inferior because sometimes it can fail and for no additional cost or complexity we can have something better, but that 'sometimes' is looking more rare in practice, though it did happen.

Thursday, January 27, 2022

Open Source Voting Machines: 2022-01

A year ago I posted a teaser of some work I'm doing on Open Source Voting Machines.

The ballots are looking a lot better now. There's a proper header and 'how to vote' instructions. The big feature I've been working for the last month is ballot recognition. Given a scan of any known ballot page, identify which precinct it is and scan the appropriate bubbles. With that I think this is now much closer to being a practical system where a central scanning operation could receive absentee ballots and precinct ballots and scan them all. All this with no barcodes, it's 100% human readable. This can be important psychologically to voters because there are no mysterious markings for people to get suspicious of. Most places I've voted still have a plain serial number on each ballot to prevent duplication, that can be addded if desired.

The test ballot I'm generating is based on random words from the Linux dictionary, so just look at it visually and don't read too much into the nonsense ;-)



Thursday, January 20, 2022

Best Wordle Start Word: raise

 I'd seen a couple other posts and decided to do the analysis myself.

There are two word lists of 5 letter words; one with 2315 words and one with 10657 words. Others have noted that the shorter list is probably words that could ever be a solution to wordle, and the longer list is things that will also be accepted as guesses. Linux /usr/share/dict/words contains only 6112 words that match ^[a-zA-Z]{5}$ .

Doing letter frequency analysis on the union of these lists leads to the impression that the best word (with the most frequent letters) is "soare" (an obsolete word for a young hawk). Doing this just on the solution list yields "later" "alter" or "alert".

But we can do better. wordlehack.go does the following:

for each guess word:

  for each target word:

    for some {target} and {guess}, how many possible words remain?

  get the sum of remaning words for all targets

So, what word has on average the fewest remaining words after it has been guessed, across all possible targets? "raise"

This takes into account letter position, not just letter frequency. "arise" is not quite as good.

And if "raise" gets you nothing, the best next word is "clout".

So, there you go, go forth and be more reputable among your peers. raise your clout.

---

Source code (jupyter notebook python and Go)

https://github.com/brianolson/wordlehack

Go play the game

https://www.powerlanguage.co.uk/wordle/