Python for engineers - course 65,000 rub. from Slurm, training 3 months, Date January 15, 2024.

Techlead, Data Engineer at Skyeng / ex Slurm, ISPsystem, Grid Dynamics

Sr. Software engineer in Test, Auriga, ex ISPsystem

#1: Basic Python Syntax and Structures

Why: learning to understand the logic of Python programs, as well as writing simple programs, is the most important task. “Great” if you are not familiar with Python syntax.

• Data types and variables, mutable/immutable and simple/composite data types, debugging techniques.
• Conditional statement - logical operators, simple conditions, nested conditions and switch statement replacement.
• Basic loops - while and for loops, iterators, loop interrupts.
• Functions, methods of strings, lists and dictionaries.
• Generating and handling exceptions.

Practice: a set of small microtasks for each lesson.

#2: Improved Python Proficiency: Optimizations and OOP

Why: Let's master the features of Python - it is due to them that it is so simple and practical. We will also reveal the meaning of those three letters (we are talking about OOP) without academic theory and with a clear purpose

• Advanced work with loops - yield statement and generators, insights about loops in Python, else statement in a loop, optimization of comprehensions loops, etc.
• Special types of structures: frozendict, defaultdict, etc.
• Basic OOP concepts: classes, class instances, encapsulation, inheritance and polymorphism.
• Creating context managers for your types: the with construct.

Practice: a set of small microtasks for each lesson.
Advanced Practice: Service Usage Audit.
The CTO began to suspect that some services were no longer being used by the teams. The problem is that the module for monitoring used services has not been updated for the last ten years: it does not can upload aggregated data, and the format of the returned values ​​does not correspond to generally accepted ones standards. You have been chosen to extract the captured metrics, aggregate them by type and team, and provide this information to the CTO for an initial assessment of the scope of the problem.

#3: Network connections

Why: Almost every service has an interface for connecting via some network protocol. And without connection, interaction is impossible. Fortunately, the Python ecosystem has client modules for almost any protocol.

• Pip package and installing third party modules.
• Paramiko module for executing commands via ssh.
• The requests module for making HTTP requests.
• Overview of modules for working with databases and message brokers.

Practice. The development team is introducing a new methodology: If you allow an alert to occur, you will receive a task. The analytical subsystem looks for only selected errors and sends messages about them to the Kafka message broker. Your task is to complete the cycle of returning bugs to developers: your consumer should automatically create tasks with the required description and priority in Trello.

No. 4: Working with text in various formats

Why: Connecting to the service is only half the battle. The other half is information sharing. And information is often a set of text characters in a certain format. Modules will help you decode and encode. Don't need all the information? Regular expressions will help you extract what’s important and discard the rest.

• The re module and regular expressions.
• Modules for working with data in different formats: separated values, json, yaml, xml.
• Using command line arguments: argparse module.

Practice: Creating a source of service usage data.
During the audit of the use of services, you identified important information for the business, even the CEO became interested. It was decided to analyze the lost money and not allow such situations to happen again. To do this, you need to give analysts a data acquisition tool so that they can prepare reports. The problem is that the monitoring module returns service limits for a separate request in yaml formats, and prices for services are returned by the billing system in xml format. It is necessary to combine data on the current load with limits and prices. The analytics department requested aggregated information in JSON format with the ability to specify a time interval and aggregation step.

No. 5: Interaction with the operating system

Why: The importance of interacting with the operating system does not need to be explained further. How to connect it with Python is in this lesson.

• Reading and writing files.
• os module - reading environment variables, working with directories and rights, working with processes.
• The subprocess module for interactive interaction with processes.

Practice: automatic provision of access to servers.
During the campaign to abandon unused services, a curious situation arose: a grocery server teams shut down, but the infrastructure team periodically used it as a host for staging. It turned out that the product team did not use it because periodically someone would overwrite their settings with their own. It was decided that now the resource will be assigned only to members of one team, and automation will help to avoid mistakes. You, as the one who started this mess, need to write an agent that will periodically poll the rights management system and make changes to the configuration of rights within installed services and, if necessary, give the services a command to re-read configurations.

No. 6: K8S operator in Python (live-coding stream)

October 13 at 19:00
Why: let's expand the capabilities of K8S to suit your tasks.

No. 7: Writing our own module for Ansible

Why: Ansible is a powerful configuration management system, and even expandable with its own modules. What a coincidence that it itself and most of the modules are written in Python.

• Writing your own modules for Ansible.

Practice: writing a rights management module.
A long time ago, in a galaxy far, far away, you already wrote an agent to issue rights to certain services. It's time to change pull to push and Ansible will help with this. All you need is just a module.

No. 8: Creation and use cases of your API

Why: Preparing code for deployment is one of the most important tasks. Our own scripts for stage pipelines help make this process more flexible and convenient.

• Creating a REST API in Flask.
• Creating your own prometheus exporter with Prometheus Python Client and Flask.

Practice: uploading data to a third-party monitoring system.
The costs of unused equipment exceeded even pessimistic forecasts. Now the engineering team has another area of ​​responsibility - monitoring unused services. To do this, you need to periodically poll the billing system through your script and pass the data to Prometheus. The format of the received data is still not suitable. You need to implement a connector. And at the same time write an endpoint so that the analytics department always has up-to-date information in JSON format at hand.

AMA session + INFORMAL meeting with course speakers

October 28 at 19:00
We meet to discuss concerns of course participants.

#9: Testing your application's API

Why: Sometimes it's better to have no code than code that is unstable. In order not to be afraid of breaking your code, you need to write tests.

• Types of tests: unit, integration and end-to-end.
• Overview of the pyhamcrest module and its matchers.
• Architecture and capabilities of pytest.
• Using pytest and pyhamcrest to write unit tests.

Practice: writing tests using pytest and pyhamcrest for your API.

No. 10: Interaction with CVS and DevOps systems

• Using third-party modules using the example of integration into Gitlab pipelines.
• Using pygit to get information about code changes.

Practice: generating a change log from commits.
The engineering team liked your solutions so much that they were inspired by them and started writing their own. But people always forget to write descriptions for releases. To achieve this, the team decided to implement commit conventions and generate changelogs directly from commits when merging the dev branch with the release one, and if the name of the commit does not comply with commit conventions, do not allow merge-request until merge.

#11: Chatops with Errbot in Python

Why: The main business problems are not application performance or even errors occurring in the code. The most serious problems arise when employee communication is ineffective. Chatops is one way to solve this problem.

• Chatops concept: what problems does Chatops solve?
• Errbot framework: installation, creation of a basic plugin template, configuration and launch.
• Errbot framework: creating your own plugin for Chatops with various options for processing messages.

Final project

Deadline review and protection for those interested

After the course, you will have a project on Git: you can automate your work task or do one of the proposed options.
The finished project can be used as a portfolio case and shown when applying for a job.