Cloud Platform Administrator
Job Description
Location: Bengaluru, KA
Position Type: Full-Time
Hours: 9:00am to 5:00pm Weekdays (Monday – Friday)
Key Responsibilities
- User & Access Management:Creating users, setting roles, managing access controls.
- Cluster & Resource Management:Provisioning clusters, monitoring resource usage, optimizing performance.
- Job Management: Scheduling, managing, and optimizing jobs and workflows.
- Monitoring & Troubleshooting: Monitoring system performance, troubleshooting issues.
- Platform Maintenance: Managing updates, backups, integrations, and disaster recovery.
- Collaboration: Working with data teams and cloud administrators to optimize and maintain the environment.
- Cost Optimization: Monitoring resource usage and controlling costs.
Technical Experience
Cloud platform expertise (AWS, Azure, GCP)
- Familiarity with the major cloud providers (AWS, Azure, Google Cloud).
Databricks-specific admin experience (clusters, jobs, workspaces)
- Managing Databricks workspaces, clusters, jobs, and libraries.
- Configuring and scaling Databricks clusters, including creating cluster policies.
- Understanding Databricks security settings, including access control and user management.
- Knowledge of data security and governance, especially using Databricks’ security features such as secret management and data encryption.
Security and compliance management
- Understanding of role-based access control (RBAC) and fine-grained access control (FGAC) in Databricks.
- Knowledge of security auditing and the ability to monitor logs for suspicious activities.
Experience with Apache Spark and data engineering.
- Experience with Apache Spark and distributed computing (Databricks is built on top of Spark).
- Familiarity with Delta Lake (Databricks’ optimized data lake).
- Understanding data pipelines and how to schedule, monitor, and manage them.
Automation and scripting skills (Python, Scala, SQL)
- Automation: Experience with automating workflows in Databricks, such as creating and managing clusters, jobs, and notebooks.
- Scripting languages: Strong skills in Python, Scala, or SQL for managing tasks or interacting with the Databricks API.
- Familiarity with REST APIs to integrate Databricks with other systems or automate tasks.
Monitoring, troubleshooting, and performance optimization
- Familiarity with Databricks monitoring tools (e.g., cluster metrics, job logs).
- Ability to monitor resource usage and performance, troubleshoot issues, and optimize resource allocation.
- Knowledge of debugging Spark jobs and resolving common issues related to cluster failures, performance bottlenecks, and resource contention.
Collaboration and communication
- Ability to collaborate with data scientists, data engineers, and other teams to ensure the Databricks environment meets their needs.
- Communication skills to effectively explain technical concepts to non-technical stakeholders.
- Problem-Solving: Ability to troubleshoot issues and resolve them efficiently.
- Adaptability: Keeping up with evolving technologies and best practices.
