Data Platform Engineer
4+ Months W2 Contract
San Jose, CA (Hybrid)
Benefits You’ll Love:
NextDeavor offers health, vision and dental benefits for contract employees
Paid sick leave eligibility is contingent on state of residence
Optional 401k Plan (excludes employer match)
Opportunity to get your foot in the door at a well-established corporation, with potential for extended or permanent full-time employment (NextDeavor boasts an impressive conversion rate of approximately 70%)!
Here’s how you’ll make an impact on the team:
Setup and maintain production scale Databricks environment on public cloud such as Microsoft Azure and AWS (Amazon Web Services)
Setup and maintain production scale data storage such as ADLS (Azure Data Lake Storage) and AWS S3 for multiple tenant teams using our Data Platform
Setup and maintain production scale micro services to support the daily operation of our data platform. Services include job scheduling, security, financial, and administrative services, etc.
Provide triage and guidance to the team on various support issues raised by our tenants
Develop tools and automation solutions for configuration management, service deployments, monitoring, and alerting to assist with daily RTB (Running the Business) operations
Budget and monitor cloud spend, always think of ways to avoid cloud resource wastage, utilize 3rd party tools, or develop your own tools to help the team with cost optimization
Assure security and privacy compliance and implement Adobe Security & Compliance solutions to lock down data stored in our data lake
Explore GenAI technologies and find opportunity to integrate them with our data platform, providing platform enhancement or improving platform user experience in the end
Work with various 3rd party vendors for troubleshooting, proof of concept, and other collaborative projects to enhance our product.
Here’s what you’ll need to be successful in this role:
Cloud Infrastructure Administration and Automation:
AWS
Azure
Proficient with following storage technologies:
ADLS Gen2
AWS S3
Hive or MySQL
MongoDB
Vector Databases
Setup, troubleshoot and maintain following technologies:
Databricks Workspace, includes but not limited to:
Unity Catalog
Vector Search
SQL Warehouse
Serverless Compute
Spark workloads
Airflow and DAGs
Azure Kubernetes Service or Elastic Kubernetes Service