Digital Moleskine

Cursor Way of Pagination

2022-11-27T00:00:00+00:00

Many articles and notes have been written about the disadvantages of “regular” pages when browsing lists of content. The major disadvantages are the speed of SQL OFFSET and duplicates of “edge” elements.

The next generation of dynamically changing content (blog feeds, tweets, Instagram posts) requires advanced content navigation approaches. One of them is cursor-based pagination.

The idea of cursor-based pagination is moving between “next” and “previous” pages quickly, without duplicates or missing content. Usually users don’t have “pages” navigation—it looks like infinite scroll on the feed page. Examples of this type of navigation are the infinite feeds of Twitter, Instagram, Facebook, etc.

Developers who integrate with this type of pagination also don’t have “pages” in the API but have a “cursor” to the next or previous page. It looks like:

Implementation

The minimal model of any content is:

PRIMARY KEY – unique item identifier, could be integer, uuid, or any other type;
DATA – a field, or number of fields of “human” content;
ORDER – a field, or number of fields to determine the order content should be displayed in;

Modern platforms use algorithms to determine the order of the content for each user; this approach is out of scope for this article.

Since the ORDER field often is not unique, we can’t use it exclusively. We need something more reliable than that.

So let’s build the pagination of news feed. We’ll use item created date as an ORDER.

The table news will look as follows.

 uid | title           | content                       | datetime
--------------------------------------------------------------------
 ... | World War Z     | Lorem ipsum dolor sit ...     | 1638004125
 ... | Etiam Molestie  | Maecenas vulputate mi ...     | 1624774080
 ... | Nam eu Lobortis | Mauris facilisis sagittis ... | 1614406080

Since datetime is not unique, but ID is, we need to create a single directional list of items using it.

The database index will help to chain items by those fields.

Now query to get the next page will be:

Links to previous and next pages could be provided by the API as part of the response, but the application using the API could also build them on its own according to the latest response.

Getting the previous page is a little bit different:

So we change the order but then reverse the items before sending them as a response.

Cursor Way of Pagination was originally published by Sergey Kuzmich at Digital Moleskine on November 27, 2022.

WordPress Configuration Cheatsheet

2021-04-26T00:00:00+00:00

Original post was located at https://blog.ripstech.com/2018/wordpress-configuration-cheat-sheet/.

WordPress is the most frequently installed web application in the world. The system is operated not only by experienced developers but also by beginners. This post summarizes what to look out for when configuring your WordPress installation’s security.

1. Disable Debugging

The debug functionality should not be active in a production environment, as it might provide useful information to potential attackers.

The debug information is very helpful during the development of a WordPress application. However, it is important to make sure that the parameter is set back to false before loading the system onto a remote server. Errors should be logged, but they must not be visible to unauthorized users.

2. Use Strong Database Credentials

Each WordPress installation should have its own database with a dedicated user, and a secure and unique password.

When installing WordPress, a table prefix can be specified. This can allow you to run several installations using one database. However, sharing one database between different installations is dangerous because if one instance gets hijacked, the other installations will also be at risk. In addition, you should avoid using the database root account, as it has full access to all databases on the server.

3. Use Unique Keys and Salts

The salts and keys must be unique for each WordPress installation, as this is the only way to ensure secure user management.

The salts and keys are important for different functionalities in a WordPress application. Among other things for a secure login and a session management. The values are automatically chosen randomly during the installation. However, a WordPress installation might be duplicated. On deployment, it is advised to generate new values for these constants. These values can be retrieved via the following API. https://api.wordpress.org/secret-key/1.1/salt/

4. Use SSL Encryption

No one should be able to read the traffic between your server and your users. Use SSL encryption and force WordPress to use only this type of connection.

It is now standard practice to encrypt your traffic. Many browser manufacturers mark un-encrypted connections as dangerous. Often web hosters offer their customers simple free certificates to enable SSL encryption. Also providers like Let’s Encrypt offer free certificates.

5. Prohibit Database Repair

WordPress supports automatic database repair. This should not be possible without a backup of the database.

The path wp-admin/maint/repair.php, which anyone with sufficient permissions can call, triggers this automatic process. This might be helpful for a broken or fragmented database schema. This should never be done without a proper backup of the database.

6. Disallow Unfiltered Content

Admins and Editors are able to publish unfiltered HTML or files. If that is not absolutely necessary, activate these filters.

By default, Admins and Editors are able to write unfiltered HTML in post title, post content, and comments. The filetype filter can also be deactivated. This filter should remain activated.

7. Enable Auto Security Update

Your WordPress should always be up to date so that it can resist the latest attacks. Activate the automatic security updates.

Since WordPress is a wide-spread application, many attackers test for known vulnerabilities in outdated installations. Don’t forget to perform a backup before each core upgrade, so only the minor update with current security patches should be installed automatically.

8. Block External Request

Access from your installation to external resources should be restricted. It is advisable to only accept trusted sources.

Not only the external access to WordPress should be secured, but also the access from the system to external resources. Here, the principle of white-list instead of blacklist applies.

9. Disallow File Modifications

Prohibit the editing of code lines in plugins and themes by your admins and editors. Deactivate the file editor for all extensions.

By manipulating code, a trusted extension can become a gateway for attackers.

10. Disallow Install Extensions

To harden your installation completely you can prevent any changes to your system by prohibiting the installation of plugins and themes.

Themes and plugins can not only bring advantages for your WordPress. You must trust the creators of these extensions. Activate this barrier to prevent your admins and editors from loading untrusted extensions.

The cheat sheet lists the 10 most important configurations. Many of these settings are set correctly by default when installing WordPress while others can be changed to harden your security. Note that not all of the suggested settings for a secure configuration might be suitable for your application, depending on its environment and development stage. With the help of static analysis tools, these and other settings can also be checked automatically.

WordPress Configuration Cheatsheet was originally published by Sergey Kuzmich at Digital Moleskine on April 26, 2021.

Git Ignore-On-Commit

2019-04-07T00:00:00+00:00

I used SVN & Tortoise SVN years ago. There was one cool feature: ignore-on-commit.

Now I use Git to track projects and file changes. I’m pretty sure that Tortoise GIT has an ignore-on-commit feature, but I prefer to use Git within the terminal. Anyway, there is a straightforward way to achieve the same behavior.

git update-index --assume-unchanged <filename> makes file changes ‘invisible’ for git tree status;
git update-index --no-assume-unchanged <filename> unhides file from being ‘invisible’ for git;
git ls-files -v | grep '^h' | cut -c3-
shows all ‘invisible’ files;

There is one more tip with it - use aliases. Put lines below to .gitconfig file.

Now the commands at the beginning are available the following way:

git hide <filename>
makes file changes ‘invisible’ for git tree status;
git unhide <filename> unhides file from being ‘invisible’ for git;
git show-hidden
shows all ‘invisible’ files;

That’s it!

Git Ignore-On-Commit was originally published by Sergey Kuzmich at Digital Moleskine on April 07, 2019.

Terraform Module Example

2019-01-29T00:00:00+00:00

In addition to the previous article I would also like to write about Terraform Modules as a way to boilerplate common deployment tasks.

Let’s make a module to quickly deploy WordPress website with DigitalOcean provider.

Getting Started

Commonly Terraform module contains the next items: input variables, output variables and main service definition. There are three files from previous article variables.tf, credentials.tf and service.tf.

Create a directory for your future Terraform module wordpress/ and let’s start with files of service definition. Rename service.tf to main.tf.

* since droplet name is required argument, use "${coalesce(var.droplet_name, var.domain)}". It returns the first non-empty value from provided arguments, so you are able to give custom droplet name, otherwise it will be the same as a domain name.

Next create provider.tf with provider definition:

* we skip token argument and will provide it via environment variable

At this moment there are complete service and provider definitions. But no module outputs and no input variables are defined, so do it.

DigitalOcean WordPress specific setup

DigitalOcean’s WordPress One-Click application requires few manual steps to make website available for the public, such as providing ServerName for apache configuration.

Usually you need to login with SSH to droplet and then it runs /opt/digitalocean/wp_setup.sh script to make required setup with interactive prompt, but we’ll perform the setup with Terraform.

Let’s take only things we need and put it into user_data.sh file to apply it as user_data on droplet creation.

But there is no domain we want to set for the droplet. Also we can’t easily pass Terraform variable into this script. Terraform has a solution for it called template_file.

Rename user_data.sh to user_data.tpl and replace $dom bash variable with Terraform’s ${domain} variable on line 7.

Now, create one more file to describe this template user_data.tf.

The latest thing is to use created template_file as a droplet’s user_data.

Distribution

I prefer to use GitHub to store and distribute modules.

You need to create a GitHub repository and just put all files into it.

Check out the repo with this configuration.

There are few more ways to distribute modules. You can check it out on HashiCorp Docs.

Usage

Create directory for your project and create service.tf file inside.

That’s it! You just need to provide valid credentials to perform this action.

Credentials

I use the following way to provide credentials for Terraform:

a file with the actual credentials as environment variables (somewhere in the user’s home directory): export DIGITALOCEAN_TOKEN=8d50ac...059828b
a .env file in the Terraform configuration directory to wrap environment variables in a Terraform-friendly way: export TF_VAR_token=$DIGITALOCEAN_TOKEN

The DigitalOcean provider requires the token argument as an access_key, so I wrap $DIGITALOCEAN_TOKEN into $TF_VAR_token=$DIGITALOCEAN_TOKEN.

Now, to apply any changes I just source ~/do_credentials file (the first one) and run Terraform commands.

References

Terraform Module Example was originally published by Sergey Kuzmich at Digital Moleskine on January 29, 2019.

Facebook Enforces HTTPS - Developer's Thoughts

2018-05-21T00:00:00+00:00

This blog post is a cry from the heart. I don’t know why, but there are more and more people who don’t even try to think before doing anything.

Some time ago Facebook introduced new requirements for API applications. They want you to use only HTTPS as a transfer protocol to protect sensitive data from insecure transfer. I’m happy about it, but as usual they half-assed it.

Most of the applications I have developed are deployed to AWS or similar providers and use the following infrastructure scheme:

So, I handle the secure connection with a load balancer and the application doesn’t even know about the existence of the HTTPS connection. It just works. But locally developers don’t have a load balancer and it’s okay to communicate with the application over HTTP.

Facebook enforces HTTPS for applications and doesn’t allow it to be turned off even if the application isn’t published for users. Okay, there is another option: Facebook gives the ability to make a kind of ‘test’ version of an existing application with different API keys and secrets. But even in a test application you are still forced to use HTTPS. So you can’t check Facebook login without HTTPS enabled. What the heck are you doing, Facebook?

I know that I can make a self-signed certificate or use a free certificate from Let’s Encrypt, but why should I mess with it if I can resolve it on the infrastructure layer? With this enforcement developers need to “implement” a local “secure” environment. Seriously?

🔥 Facebook burn in Hell! 🔥

UPD January 24, 2019: It is possible to use HTTP protocol for applications in development mode, but only with localhost domain.

Facebook Enforces HTTPS - Developer's Thoughts was originally published by Sergey Kuzmich at Digital Moleskine on May 21, 2018.

How to Deploy Applications to DigitalOcean with Terraform

2018-03-14T00:00:00+00:00

I like DigitalOcean as a cloud servers provider because it is pretty simple and doesn’t cost too much. It already has complete enough infrastructure things for small websites and large services.

DigitalOcean has a clean user-friendly Control Panel UI to manage droplets, networks and so on. I used to configure applications manually, but there is a way to do it much faster and more reliably.

Terraform is a tool which allows to define any kind of infrastructure as a simple text document and deploy or update it with a single command.

Getting Started

Every single web application has at least two things: domain and server. It means if you need to run the simplest website you need to create a server (DigitalOcean calls it droplets) and map a domain to server’s IP address.

Droplet Definition

You can create droplet with DigitalOcean Control Panel with ‘Create Droplet’ button, filling droplet’s name, choosing droplet type, region and size.

Droplet definition with Terraform looks the next way:

* The list of all available droplet arguments.

Domain Definition

The same things with adding domains. You can do it via DigitalOcean Control Panel or provide Terraform configuration:

Deployment

Begin with creating directory to store Terraform configuration files. Then create service.tf (the filename can be anything you want) file in this directory and put domain and droplet definitions into.

Domain ip_address uses a reference to the defined droplet as a value to automatically map the droplet to the domain.

Terraform is smart enough to understand that domain can’t be created without droplet’s IP address, so first it will create a droplet, and then it will give correct ip_address for the domain value.

Almost done. The last thing to do is to set provider credentials, so create credentials.tf file in the same directory and put next lines into:

You need to generate a DigitalOcean API token in the API section of the DigitalOcean Control Panel

Here we go…

Initialize Terraform provider with terraform init. It downloads required provider files and initializes Terraform configuration directory (almost like git init command initializes repository in working directory).

Run terraform plan, it gives you the list of actions that Terraform will performe on DigitalOcean. With current infrastructure definition you should see output like this:

It’s time to terraform apply. Confirm actions and you’ll get defined resources on your DigitalOcean account. Check it out on Control Panel.

Roll-back

To destroy this infrastructure just run terraform destroy.

Don’t be afraid to lose other droplets and services on your account by terraform destroy. Terraform stores defined and deployed resources in terraform.tfstate file, so it will only remove resources defined and deployed from this working directory.

Re-usable Configuration

At this moment dynamic values (such as domain name, droplet type and size) are directly used in resource definitions. It is ok, but if you want to make re-usable configuration you need to export values to variables.

Variable Definition

Terraform variable looks the next way:

* Type, description and default variables are optional arguments.

Export Variables

Create a variables.tf file and think about which arguments may be changed and which arguments should be changed in similar deployments (applications).

For example, droplet size and image can be the same for different projects, so you can provide default value for this arguments, but domain name can’t be the same for different projects, so you just need to define that you’re requiring this argument but it won’t have default value.

Now replace credentials.tf and service.tf files values with new created variables:

Next you should provide values for variables when making terraform apply. Values can be passed by:

Environment Variables;
Interactive Prompt;
Variable Files.

Environment Variables

You can provide values via environment. The name of the environment variable must start with TF_VAR_ followed by the variable_name, and the value is the value of the variable.

For example, variables above can be set by:

Interactive Prompt

Alternatively you can provide values just by terraform apply and Terraform will ask for it (which does not have defaults) interactively:

Variable Files

And the last way to provide values is files which match terraform.tfvars or *.auto.tfvars in the configuration directory, Terraform automatically loads them to populate variables. For example:

Conclusion

With this setup you can deploy a big number of services in a few minutes. Here is only example of the easiest and the smallest infrastructure, but with Terraform you can describe infrastructure of any complexity and size.

References

How to Deploy Applications to DigitalOcean with Terraform was originally published by Sergey Kuzmich at Digital Moleskine on March 14, 2018.

WordPress Plugin Deployment Using GitHub and Travis CI

2018-01-07T00:00:00+00:00

I love WordPress and I’m sure it is the best solution for corporate websites and personal blogs. I’m happy to contribute code for WordPress, create plugins and themes. I’ve made a few public plugins: Inline Spoilers and Sanitize Cyrillic.
But there is one unpleasant thing that everytime stopped me. WordPress wants you to use SVN to store and deploy public plugins and themes but development with SVN is too annoying for me. There are topics how to use git for WordPress but it is all about git-svn.

I’m expecting that you already have a submitted WordPress plugin and just want to avoid SVN things, otherwise, please, take a look at this guide and welcome back.

Step 1: Travis CI Configuration

At this point, you have plugin’s code hosted on WordPress SVN and GitHub. Next you need to create configuration and describe all things you want to be executed by Travis.

Start with creating ‘dummy’ .travis.yml file in your repository to tell Travis what to do:

Travis will proceed all future commits and pull requests for the repository with .travis.yml file.

Step 2: Configure WordPress Plugin Assets

WordPress has custom assets/ directory in plugin SVN repository to store plugin page assets such as screenshots, icons, and etc.

Create assets/ directory and put empty .gitkeep file into to make sure it exists even without any file inside.

Step 3: Write Deployment Script

🤹‍♀️ ~ that’s where the magic begins…

To avoid SVN stuff just put it to shell script, create deploy/deploy.sh script file with next contents:

* Used global variables will be covered a little bit later.

It will do all deployment stuff, you just need to execute this script.

Wait…

Actually you don’t need to execute script to deploy a new release. Travis will do that for you.

Step 4: Enable Travis CI Deployment

I think you don’t need to deploy each new commit, you need to tell Travis to execute deployment script only on “some” specific events.

I’d prefer using tags to specify plugin releases. I want Travis to submit a new plugin version to WordPress each time I push a new git tag.

To enable deployment on git tags provide next configuration to .travis.yml:

* Travis sees git tags the same way as branches.

I like to use semver for projects, I’ve set semver regular expression as a branch filter to allow deployments only for this kind of tags.

Environment variables

$SVN_REPOSITORY - WordPress plugin SVN repository URL.
$TRAVIS_TAG - Tag label. (This variable is fetched and provided by Travis from GitHub)
$SVN_USERNAME - Encrypted WordPress account username.
$SVN_PASSWORD - Encrypted WordPress account password.

How to define encrypted variables in .travis.yml.

Conclusion

Now you have a complete setup for an automated plugin deployment process.

With this way you don’t need to do any SVN specific actions. You may just develop plugins and when you’re ready to give users a new version just push tag for commit you want to deploy. Travis will do all required stuff and after a few minutes you’ll see new deployed version on WordPress website.

References

WordPress Plugin Deployment Using GitHub and Travis CI was originally published by Sergey Kuzmich at Digital Moleskine on January 07, 2018.