caleb gross

Pelican, or pelican't? A hint at Claude evals

Sat, 13 Jun 2026 00:00:00 +0000

I’m a big fan and frequent user of Claude’s deep research feature, so it caught my eye when the task title from my research subagent was a bit longer than usual. I’d asked about WWI causes and got the research title “Causes of WWI research disabled warnings about…” with some unexpected trailing text at the end. Confused, I inspected the title with Chrome DevTools and found that the h2 header overflowed far past the truncated text shown in the UI. The full title discussed re-enabling system update warnings for Windows and macOS—clearly unrelated to my prompt!

You can just say it

Thu, 28 May 2026 00:00:00 +0000

There is a weird collection of arguments for appraising the value of humans and their creative artifacts. It usually goes something like this: In the age of AI, we should still prefer humans in certain roles because AI could never perform the tasks required for that role. Or, a human can at least do it better. Or, perhaps the output from a human and AI may look similar, but human output is preferable for subtle stylistic reasons that an AI cannot reproduce. Or, at least the AI cannot reproduce it consistently. Observe the scuff marks around the base of the goalpost from constant movement. Allow 28 days for the concrete to cure.

O(N) the Money: Scaling Vulnerability Research with LLMs

Mon, 20 Oct 2025 00:00:00 +0000

Let’s go back to the basics and consider a very simple goal: What if we could find and fix all the vulnerabilities in open source software? This is a pretty ambitious goal, admittedly, but…

Slice: SAST + LLM Interprocedural Context Extractor

Wed, 20 Aug 2025 00:00:00 +0000

Earlier this summer, Sean Heelan published a great blog post detailing his use of o3 to find a use-after-free vulnerability in the Linux kernel. The internet lit up in response, and for good…

I used o3 to profile myself from my saved Pocket links

Mon, 07 Jul 2025 00:00:00 +0000

Welp, Pocket shuts down tomorrow despite our pleas for it to stay. While migrating¹ all of my saved articles, I noticed that I’ve got almost 900 saved articles spanning nearly 7 years. That’s a goldmine of stuff-I-like data! Some quick analysis using xsv²:

𝄢 unzip pocket.zip && xsv headers part_000000.csv
1   title
2   url
3   time_added
4   tags
5   status

𝄢 xsv sample 1 part_000000.csv | xsv flatten
title       The Uncertain Future of American Libraries
url         https://mek.fyi/posts/the-uncertain-future-of-american-libraries
time_added  1678243022
tags
status      unread

𝄢 xsv count part_000000.csv
878

𝄢 xsv select time_added part_000000.csv |
    xsv stats | xsv select min,max | xsv flatten |
    while read header value; do
        echo -n "$header "
        date -I -d "@$value"
    done
min 2018-07-12
max 2025-06-26

Experiment

Recalling Simon Willison’s recent geoguessing challenge for o3, I considered, “What might o3 be able to tell me about myself, simply based on a list of URLs I’ve chosen to save?” I opened a temporary chat with o3 (doesn’t use ChatGPT’s memory feature) and prompted:

Using LLMs to solve security problems

Fri, 28 Feb 2025 00:00:00 +0000

TL;DR: Raink—a novel, general-purpose listwise document ranking algorithm using an LLM as the ranking model—can be used to solve non-trivial security problems.

A very simple explanation of how the Raink algorithm works:

Split big list of items into small groups (e.g., 10 items per group)
Ask the LLM to rank/order each small group according prompt relevance
Shuffle everything and repeat this process several times
Keep track of how each item performs across different groups
Focus more attention on the items that consistently rank highly

The approach is fast, cheap, and handles far more items than would fit in a typical LLM’s context window. Key features of the algorithm and its implementation:

Hard problems that reduce to document ranking

Mon, 24 Feb 2025 00:00:00 +0000

There are two claims I’d like to make:

LLMs can be used effectively¹ for listwise document ranking.
Some complex problems can (surprisingly) be solved by transforming them into document ranking problems.

I’ve primarily explored both of these claims in the context of using patch diffing to locate N-day vulnerabilities—a sufficiently domain-specific problem that can be solved using general purpose language models as comparators in document ranking algorithms. I demonstrated at RVAsec ‘24 that listwise document ranking can be used to locate the specific function in a patch diff that actually fixes a vulnerability described by a security advisory, and later wrote on the Bishop Fox blog in greater defense of listwise ranking by publishing a command-line tool implementation (raink) to prove the idea.

Why I left Air Force Cyber

Wed, 29 Nov 2023 00:00:00 +0000

I wanted to write this before I’ve been out longer than I’d been in. I agonized over my decision to leave the Air Force—largely because I experienced it as an issue of professional identity.

Why I joined

After wrestling at a high school math and science academy, I wanted to keep up physical fitness and develop leadership skills in a technical environment. I also needed financial assistance to fund college. I had Army, Marine, and Naval officers represented among my grandparents—but rather than inspiring any overwhelming patriotism in me, I think military lineage simply enabled me to consider the possibility of going military at all. I considered attending a military service academy but chose a more balanced experience with Air Force ROTC at the University of Virginia.

zip snip: Social engineering attack using a .zip domain

Sun, 14 May 2023 00:00:00 +0000

Google recently launched the .zip TLD, and Twitter has been up in arms. It’s hard to imagine that .zip domains would be used for anything legitimate, and would instead give way to more creative phishing attacks. Accordingly, I quickly snagged source-code.zip while these domains were rapidly being bought up, and showed an example of how it might be used to trick a software developer into downloading a malicious ZIP file—yet there are still some that quite reasonably ask, “Should we really be so concerned about .zip TLDs?”

How I keep up with new content

Mon, 06 Mar 2023 00:00:00 +0000

Or, using a content aggregator like it’s 1999.

Daniel Miessler writes a lot—and a lot of that lot is about RSS. Daniel inspired me to curate my media intake with RSS (and also more generally to start ~~writing~~ learning in public). Here are a few RSS-related tools and principles I’ve picked up over the past 6 months. It essentially boils down to discovering new content sources via Twitter, etc., and using a content aggregator (and supporting tools) to bring those many sources into a single view.

Zero-knowledge appointment scheduler

Wed, 25 Jan 2023 00:00:00 +0000

I recently (finally) started using online appointment scheduling software.

There are many players in this space, but Calendly is the most popular and featureful. It’s awesome—but in order to use it, you have to give it direct access to whichever calendars you want it to check for conflicts.

That means access to all sensitive details like online meeting URLs, attendee information, etc.—and it gives me the creeps. This isn’t unique to Calendly as most booking software works exactly this way. Additionally, Calendly’s free tier is limited in that:

Free as in (tier)

Wed, 23 Nov 2022 00:00:00 +0000

I tend to push a product’s free tier pretty hard.

I’ve been using the same free Dropbox account for 11 years (originally 2 GB, but permanently upgraded to 22 GB through various referrals and promotions long ago). I have several free Zoho email accounts grandfathered into supporting custom domains, SMTP, and email forwarding (they still offer a free tier with custom domain support, but without those latter features). I use free Inoreader while compensating for its restrictions by layering on free equivalent RSS services like Kill the Newsletter!, siftrss, Grepfeed, and Pocket.