Staff Software Engineer

About Prophecy

The leader in AI-native data preparation and analysis, Prophecy is revolutionizing how the world's top enterprises turn data chaos into reliable insights. We introduce the AI-native data lifecycle (generate, refine, deploy) where our industry leading AI agents and humans work hand-in-hand in visual and document interfaces to analyze, transform and prepare data, to ship trusted insights at enterprise scale. To learn more, visit us on LinkedIn.

Position Summary

This is an exceptional opportunity to be a technical leader on the team that powers the heart of Prophecy: the Runtime, our orchestration and execution service. Every data workflow , whether generated by our AI data agents or built by a human in the visual editor , is planned, scheduled, and run at enterprise scale by Runtime, directly against the customer's own platform (Databricks, Snowflake, BigQuery, and more). When a business user asks an agent a question and sees trusted results seconds later, Runtime is what made it happen.

As a Staff Engineer, you will set the technical direction for this service: the distributed execution engine, the task coordinator, the multi-engine connectors, and the fast interactive previews that make the AI-native experience feel instant. You will own hard, ambiguous, high-leverage problems end-to-end , from architecture to production reliability , and raise the bar for the engineers around you.

We're seeking A players to work with A players. We're a high-growth company on a once-in-a-lifetime journey to revolutionize the way data is prepared and analyzed. Be a part of it.

The impact you will have

Own the engine behind the product. Architect and evolve the runtime service that turns agent- and human-authored data workflows into high-performance jobs that run reliably in production and interactively during authoring. This is deep-tech at the center of the platform.

Make it fast and make it scale. Design the distributed execution and task-coordination layer , scheduling, concurrency, resource management, streaming of sample data (interims) , so that pipelines run efficiently from a single interactive preview to large production workloads.

Run everywhere the customer's data lives. Extend execution across multiple engines and warehouses , Databricks, Snowflake, BigQuery, Spark, dbt, Airflow , with first-class connectors, query pushdown, and consistent behavior across clouds.

Power the AI data agents. Build the execution path that lets our agents generate, run, validate, and refine pipelines in a tight loop , the difference between a plausible answer and a trusted one.

Own reliability and observability. Drive production robustness: fault tolerance, graceful degradation, recovery and restartability, monitoring, and the SLAs that enterprise customers depend on.

The rest of this description is on the employer’s own page.