Data Engineer

Data Engineer

W2 Contract

Salary Range: $124,800 - $145,600 per year

Location: Cupertino, CA - Remote Role

Duties and Responsibilities:

• Design, deploy, and manage Astronomer.io (Astro) environments, including organization/workspace setup, Deployments, queues, executors, secrets, and runtime upgrades.
• Utilize the Astronomer CLI and Astro Platform API for local development, automation, and CI/CD workflows (astro dev, astro deploy, service accounts, and API tokens).
• Implement and manage workflow orchestration using Apache Airflow on Astronomer, including DAG optimization, deferrable operators, retries, backfills, and task dependencies.
• Integrate OpenLineage on Astronomer to track end-to-end data lineage and enable visibility across pipelines, datasets, and ML workflows.
• Set up Astro Observe to define data products, manage freshness SLAs, and configure automated alerts (Slack, PagerDuty, email) without modifying DAG code.
• Deploy and manage Astronomer in hybrid mode (SaaS control plane + customer data plane) to meet enterprise security, compliance, and privacy requirements.
• Collaborate closely with cross-functional teams to maintain high data quality, reliable pipelines, and robust observability using Astro logs, metrics, and dashboards.
• Troubleshoot and optimize Airflow DAGs, containerized workloads, and Astronomer Deployments for scalability, cost-efficiency, and high availability.

Requirements and Qualifications:

• 5+ years of data engineering experience designing and managing large-scale, distributed data systems.
• Expert-level experience with Astronomer.io (Astro):
• Hands-on with Astro Deployments, runtime management, executors, and queues.
• Proficiency with Astro CLI and Astro Platform API for automation and CI/CD.
• Experience configuring Astro Observe, data product definitions, SLAs, and alerts.
• Strong knowledge of OpenLineage integration with Astronomer for data and ML lineage tracking.
• Deep expertise in Apache Airflow on Astronomer: DAG authoring, sensors/deferrables, retries, and provider integrations.
• Solid understanding of Kubernetes for Astronomer data-plane operations and scaling containerized data applications.
• Proven ability to design observability strategies for Airflow pipelines using metrics, tracing, and logging.

Preferred Qualifications:

• Experience integrating Astronomer with metadata platforms like Marquez, DataHub, or Atlan for lineage visualization.
• Familiarity with DBT on Airflow using Astronomer Cosmos.
• Prior experience deploying Astronomer in regulated environments with strict compliance and governance controls.
• Knowledge of cloud-native data ecosystems (AWS/GCP/Azure) and secure, multi-region Astronomer setups.

Bayside Solutions, Inc. is not able to sponsor any candidates at this time. Additionally, candidates for this position must qualify as a W2 candidate.

Bayside Solutions, Inc. may collect your personal information during the position application process. Please reference Bayside Solutions, Inc.'s CCPA Privacy Policy at www.baysidesolutions.com.