Identity-per-Transaction
Most automated data pipelines authenticate with long-lived service accounts. They're over-privileged by default, they sit unrotated for 90 days at a stretch, and in a FedRAMP-High healthcare system a single compromised ingestion role can mean read access to an entire data lake. The standard mitigations (key rotation, least-privilege policies) reduce the odds of compromise. They don't reduce what compromise costs you once it happens.
Identity-Per-Transaction changes what's at stake. Instead of a static credential shared across a session or a service, the pipeline mints a unique, cryptographically scoped credential for each individual transaction, then discards it. The credential is scoped to a single object and lives for the duration of one processing event. When it leaks, it's already expired, and it never had access to anything but the one object it was minted for.
The result
In a production FedRAMP-High environment, this cut the blast radius of a compromised identity by over 99.9% relative to standard bucket-wide roles, while keeping throughput linear. The math is straightforward: if a traditional role exposes all N objects in a store and a per-transaction credential exposes one, the reduction is (N−1)/N. At a million objects, that's 99.9999%. The interesting engineering is in making per-transaction minting cheap enough that it doesn't wreck throughput, which is the part the paper works through in detail.
One honest limit
IPT moves the root of trust rather than eliminating it. The credential-minting broker is itself a bounded but non-zero point of trust, and the reference architecture treats hardening and monitoring it as part of the design rather than an afterthought. I'd rather state that plainly than pretend the trust surface goes to zero.
IPT is implemented and public as pymayfly, an open-source library on PyPI with AWS, Azure, and GCP backends, and documented in full in the IEEE BigDataSecurity reference-architecture paper.
Open Problems
What I’m working on now
IPT handles credential-mediated access at the batch level. It leaves real problems open, and those open problems are the current research. I'm keeping the specifics of the newer two close while they develop, but here's the shape of each.
Can per-transaction identity survive streaming?
IPT's transaction boundary is clean when you're processing discrete files. It gets much harder when data arrives as a continuous stream, where "a transaction" isn't obviously one thing and the credential lifecycle has to keep pace with the flow rather than a file-arrival event. The open question is whether the per-transaction guarantee holds when the transaction itself stops being discrete, and what has to change in the minting path to keep it cheap at streaming rates.
How do you hold an autonomous agent accountable?
As agentic systems start taking their own actions inside data pipelines, "who authorized this, and can you prove it after the fact" gets much harder to answer. A human operator leaves an intent trail. An agent making chained decisions at machine speed doesn't, at least not without being designed to. The question I'm working is what an accountability layer for agentic activity looks like when the actor isn't a person and the actions aren't known in advance.
How do you hold an autonomous agent accountable?
My Ghost Drivers work showed that anonymized spatiotemporal data can be re-identified with 96% precision from movement patterns alone. This direction comes at re-identification from the defensive side. Membership inference and re-identification attacks tend to leave a signature in the sequence of queries an adversary runs against a dataset, since probing for a specific record looks different from ordinary use. The question I'm working on is whether tracking that query trajectory, modeled as a graph, is enough to detect and interrupt these attacks before they succeed. It builds on the re-identification methods from Ghost Drivers, turned around to defend rather than attack.
If you're working in any of these areas, I'd like to compare notes.