HackerRank stats

My current HackerRank stats:

	Contest (World)	Contest (Russia)	Practice (World)	Practice (Russia)
Top	10%	30%	10%	-
Percentile	91.47	71.08	92	-
Rank	10 247 (out of 120 108)	504 (out of 1 743)	3 049 (out of 757 851)	82 (out of 7 748)

Definition of Done

This is a sample definition of done that I use in mature development process environments.

Task is considered done if all of the below conditions are met:

1. Source code that corresponds to the task description has been developed
2. Unit tests for the task that test the source code are have been developed
3. All unit tests (including developed for the task) pass successfully
4. New features are covered by integration test
5. Integration test passes successfully
6. Changes that describe installation and migration process related to the task are provided in corresponding documentation or deployment scripts
7. Change log is updated
8. Pull request has passed code review (all comments are either covered by fixes or somehow resolved with responsible reviewer) and merged to upstream repository
9. Notes for QA people are added to the task
10. Task deployment provided for test environment
11. The task has passed manual testing successfully: discovered bugs are either fixed or brought out to separate issues for later fixing
12. The task has been deployed to live, it does not expose bugs and shows correct behavior in live

Strict dependencies

UPDATE 2019-02-21: It seems that pipenv solves the problem completely.

Use strict version dependencies to prevent unexpected upgrades. Apply the same rule to the dependencies of your dependencies recursively. Example: dependency-package-name==x.y.z

What happens when you do not follow the above rule? Let us see a development cycle on a time line.

Bob is a developer who does not follow the rule above. When he needs to add a new dependecy he just adds it to setup.py or requirements.txt without specifying a version. Then he uses pip install -e . or pip install -r requirements.txt to install the dependecy. Since an exact version of a dependecy is not specified pip installs the latest available version of the dependecy at the moment. Everything work fine and Bob happily continues a development and uses the dependecy in his code. The dependency will not be upgraded when new version is release, because pip does not upgrade by default. It checks for the presence of the dependency only, not its version if a version is not specified.

Time passes, say a month or two or a year, and at some point Alice joins the team to help Bob with the development. Alice sets up her development enviroment and runs pip install -e . or pip install -r requirements.txt to install all dependecies. Since exact versions of dependecies are not specified pip installs the latest available versions of the dependecies at the moment of installation. Modern development cycles are short and releases are frequent, so it is very luckily that Alice gets newer versions that Bob has.

First consequence of the events is that Bob and Alice start developing in different environments which is bad, because they may experience different behavior of the same dependecies but of different versions. Something that works for Alice would not work for Bob and vice verse. It leads to ineffective loss of time investigating this "magical" behavior.

Another consequence is that a newer version of a depency may become backward incompatible intentionally, by error or of improper usage. I had several such cases in my experience. In this case the program will fail with an exception or run with a logical error which is worse. Alice will still need to contact Bob and ask him to pip freeze to learn what is a working version of a dependcy to install it exactly.

The worst case is then a working combination of dependencies is lost. For example Bob has left the company before Alice joined the company. In this case Alice will need to downgrade a dependecy or a combination of dependecies version by version to find a working version.

The same happens when a production environment should be deployed. What versions of dependencies should installed if developers set up their environments a half year ago? Which developer's environment represents a master set of dependency versions?

One more thing that happens when dependecies are not strict. It prevents managed dependecies upgrades. Upgrades happen randomly along with setting up new environments which may lead to unplanned work because of backward incompatibilities or need for upgrading own code along with dependecies.

My other Python development practices

My Python software development practices

UPDATE ON 2022-11-12

This post is continuously updated list of software development practices that I use every day developing software in Python. They include coding conventions, software design principles and software project management principles.

Well-known conventions and practices

• PEP8: Style Guide for Python Code (use code autoformatters and linters to enforce it)
• Google Python Style Guide
• PEP20: The Zen of Python
• DRY principle
• KISS principle
• YAGNI principle
• Principle of least astonishment
• Clean Code: A Handbook of Agile Software Craftsmanship

Extensions, exceptions and customization to well-known conventions and practices from the previous section

Python-specific

• Python code maximum line length is 100 or 120 characters
• Preferred order of attributes within a class:
• NAMED_CONSTANTS
• class data attributes
• __init__() if present
• __magic_methods__()
• @properties
• @staticmethods
• @classmethods
• _private methods
• public methods (related private and public methods should be placed together for better readability)
• When overriding a method always call an overridden parent class method (with super) and return its result (if you are not altering it) even if the overridden method is not supposed to return anything then None at the moment (it is a forward compatibility measure)
• Usage of introspection (e.g. getattr(), hasattr()) and __magic_methods__ override should be well-considered. If a feature can be implemented without introspection or explicit __magic_methods__ it should implemented with out them (with very rare exceptions). Usage of introspection or explicit __magic_methods__ usually is a sign of bad code design or "reinvention of the wheel"
• Use strict version dependencies to prevent unexpected upgrades. Apply the same rule to the dependencies of your dependencies recursively. Example: dependency-package-name==x.y.z (explanation) → Use poetry
• Use Python packaging instead of requirements.txt and git-based deploys → Use poetry

Development Process

• Collective code ownership
• Code review is a good investment of your time
• Favor code development performance (productivity) over code run-time performance (with reasonable trade-offs)
• Favor good enough code and code development performance over perfect code when you have tight deadlines. You can always improve later
• It is fine to write an optimal code in the first place (in terms of size, performance or resource usage) if it comes at zero or near zero cost (we should not deoptimize code on purpose to avoid being accused in premature optimization).
• Dedicate time on code run-time performance optimization only if it is really needed. Premature optimization may result in waste of time
• Every time you put a dirty hack or just something that can be done better into your code put a TODO near it with an explanation and/or description of actions for improvement. It will help to track technical debt and help other developers during refactoring or regular development to understand if they are right to judge this code as strange and to be improved
• If it is hard to choose between alternative technical solutions, then choose any of them instead of wasting time trying to make the right choice. Later when you have more information or circumstances change you can refactor if the original choice was wrong
• Using a language feature just to show others that you know the feature is unprofessional
• Manage to see more lines at the screen at the same time for lesser defect count (vertical orientation of display and meaningless lines elimination may help)
• Keep development environment close to production as much as possible
• Keep testing environment identical to production environment (including the deployment procedure)
• Never submit changes that are known to break something that already works
• Never remove something that you do not understand or because you do not understand it

Code Design

• Write code to be read by humans
• Favor code readability over code run-time performance (with reasonable trade-offs)
• Maintain code reuse of own code and reuse code from publicly available libraries. Reinventing the wheel is a waste of time
• Avoid code copy & paste unless you have very strong reasons for it
• Know the difference between agile and universal: instead of writing a code for all imaginary future use cases write the code that can be easily adapted to many of the future use cases (also known as maintainable code)
• Maintain the least possible cyclomatic complexity of the code for better readability and smaller defect count
• Favor a readable code over a well commented code. A necessity for comment is good sign of a poor readability of the code
• Consider writing a logging message instead of a comment. It will serve two purposes
• Code run-time performance optimization should start from identifying bottlenecks and their elimination instead of something that is easy to optimize
• If a literal used twice in the code then prefer putting it to a named constant. Even literals that occur only once deserve to be put in a named constant, because named constant will describe the nature of the literal
• Do not put .gitignore into git repository. Every developer should be free to ignore whatever extra file he/she has locally (.gitignore should not aggregate every developer's local "mess") → Use .git/info/exclude for developer specific ignores

Code style

• Maintain a shorter code size in number of lines and characters for better readability and lesser defect count unless it impacts performance or readability
• TODO format: TODO(author) <LOW | MEDIUM | HIGH | CRITICAL>: textual description of todo
• A longer variable, function, class or method name is better than an unclear or ambiguous name
• Code style should be consistent across the entire code base
• Code base should not contain commented out source code unless it is a part of a comment or TODO.
• Non-ASCII characters should not be a part of the source code. Presence of such characters is a sign of poor localization, internationalization or parameterization of the code 

Organizational (may be boring for some developers)

• Every decision should be reasonable (based on reasons)
• Actual responsibility for a decision always lays on the person(s) who made the decision independent of how it cooked or formalized
• Every decision should be considered in every longevity term (short, medium and long)
• Every technical decision should be made considering its effect on business as the most important criteria
• Technical solutions of a highest efficiency (= effect / expenses) should be chosen during the decision making process
• Honor established processes, rules, methodologies and patterns, but do not hesitate to step out if it is beneficial in terms of profit and loss

Backup linux virtual machine from cloud provider

Not every cloud provider provides downloadable disk backup for your virtual machines. For those who do not provide such a feature the following instruction may be useful.

In order to backup running linux you will need another running linux with ssh access to store image.

Learn about device you are about to backup:

$ sudo fdisk -l

Disk /dev/vda: 21.5 GB, 21476933632 bytes

255 heads, 63 sectors/track, 2611 cylinders, total 41947136 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

  Device Boot      Start         End      Blocks   Id  System

/dev/vda1               1    41947135    20973567+  ee  GPT

Dump device to another server

# The operation may take long time. It is recommended to run within “screen” (https://www.gnu.org/software/screen/manual/screen.html).

# (!!!) Replace /dev/vda1 and username@hostname with your values

sudo dd if=/dev/vda1 bs=4k conv=noerror,sync | bzip2 | ssh -c blowfish username@hostname 'dd of=/root/image.dd.bz2'

Uncompress the image

bunzip2 image.dd.bz2

Shirk image

sudo umount /mnt/image

resize2fs -f -M image.dd

Mount image and check its integrity

sudo mkdir -p /mnt/image

sudo mount -o loop,rw image.dd /mnt/image

df -h

ls /mnt/image

tail /mnt/image/var/log/syslog

Compress the image

mv image.dd image.small.dd

bzip2 image.small.dd

Use backup later

bunzip2 image.small.dd.bz2

sudo mkdir -p /mnt/image

sudo mount -o loop,rw image.small.dd /mnt/image

cd /mnt/image

101 Hack May 2016

Yesterday night I took part in my first rated 101 Hack May 2016 contest at HackerRank:

• Ranked 134 of 976 participants
• Bronze medal badge
• 4472nd (165th in Russia) place on Contest Algorithms Leaderboard
• Rated: "O(logN)" 
• 2136 points score

By the way, I have 86th percentile in Practice Algorithms Domain, placed 3496th (84th in Russia) Practtice Algorithms Leaderboard

https://www.hackerrank.com/dmugtasimov

80th percentile is reached at HackerRank

I have reached 80th percentile at HackerRank in Algorithms domain.

Score	1614
Rank in Russia	120th (out of 4641)
Worldwide rank	5208th (out of 468048)
Worldwide percentile	80th

Please, review my profile for details: https://www.hackerrank.com/dmugtasimov