Senior Software Engineer

About the Job
We are seeking a Senior Software Engineer to join our team in Bahrain, working within a cyber security environment that handles a vast volume of security telemetry from various customers. The platform enables automated analysis and equips our operations teams with the tools needed to make informed decisions. Due to the multi-tenant nature of the platform, it is essential that every stage of the pipeline effectively accommodates strict tenant isolation, fluctuating traffic, third-party rate limits, and potential partial failures. Consequently, the primary challenges faced are rooted in distributed systems.

This is a small, highly skilled team, where engineers actively deploy to production, take ownership of their contributions, and collaborate directly with operations teams that utilize our product on a daily basis. We prioritize high engineering standards, and you will be responsible for one or more subsystems crucial to the platform's performance under load. This ownership encompasses design, delivery, production operations, and maintaining reliability metrics. We are looking for individuals who have successfully handled similar responsibilities in previous roles, demonstrating the ability to devise systems relied upon by many, address failures at scale, and promptly implement effective solutions. Mentoring junior engineers towards taking ownership of their assignments will also be part of your role.

Role Requirements

• Six or more years of professional software engineering experience, with at least two years in a senior or lead capacity within a production environment.
• Experience as a designer and operator of a system at scale, ready to discuss key metrics such as throughput, data volume, tenant or user counts, latency percentiles, and availability. Be prepared to share instances of system failures, associated costs, and the resolutions that were applied.
• In-depth knowledge of distributed systems, including queues and streams, workflow engines, retries and idempotency, consistency trade-offs, handling partial failures, backpressure management, and observability.
• Proficient in TypeScript on Node.js and Python, with the discernment to utilize each programming language as appropriate, along with comfort in reading and resolving issues in either one.
• Familiarity with cloud services and Kubernetes; experience running services on Kubernetes in a production setting is essential.
• Proven production ownership, including defining service-level objectives, leading incident responses, conducting blameless post-incident reviews, and ensuring follow-up actions are completed.
• A strong track record of excellence, evident through promotions received ahead of the standard timeline, ownership of vital systems for previous employers, public contributions such as presentations, publications, patents, or open-source projects recognized by users, and verified favorable reputations from former colleagues. You should be able to demonstrate measurable outcomes that you influenced, such as significant improvements in cost, latency, or reliability.
• A security-focused mindset, having shipped systems where threat models were critically important. Treat tenant isolation as a security boundary, rather than a mere configuration, and design systems to withstand hostile inputs.

Preferred Qualifications

• Experience building and operating multi-tenant platforms where data and credential isolation was crucial.
• In-depth experience with durable-execution or workflow engines, or with large-scale streaming platforms.
• Proficiency in integrating with security tooling APIs, understanding their rate limits and nuances.
• Knowledge of graph databases and familiarity with open security data standards relevant to events and threat intelligence.
• Hands-on experience defending production systems, whether in an engineering or operations role.
• Experience in productionizing machine-learning or language-model components, applying proper evaluation disciplines. Note that this is primarily a software engineering role, so this experience is useful but not strictly required.

This is a hands-on senior engineering role that focuses on architectural ownership and includes mentoring responsibilities, rather than a management position.