Overview

In this post we cover a strategy for managing configurations for programs using a Singleton pattern to create a static Config class.

This allows the user to create an instance of the Config class, pointing it to a specific config file, which it loads into memory.

The Config class provides several static methods for accessing configuration options from the config file. Here's an example of its usage:

Config('/path/to/config.json')

if Config.get_foo() == "bar":
    do …

Summary

In a prior post, we covered how to write a mock API server that stored a thread as a class attribute and used it to run the server in the background by starting a thread.

However, we neglected to cover how to actually use the mock API server. So here we include some examples of how you can use the mock API server to write better tests for components that require interacting with APIs.

The MockAPIServer Class

Let's start with a recap of the mock API …

Overview

In this post we discuss a way of mocking an API server during tests. This technique will let you create a fake API server that can respond to API calls however you want.

The technique is twofold:

• First, we create a mock API handler that extends BaseHTTPRequestHandler, which is the built-in HTTP server class in Python. We can extend the server class to control how it responds to requests - to …

Overview

In this short blog post, we talk about how and when you can take a method signature that defines input positional arguments by name, like this:

def foo(arg1, arg2, arg3):
    pass

and write code that will return a dictionary containing a keyword arguments-like structure:

>>> foo('red', 'blue', 'green')
{
    'arg1': 'red',
    'arg2': 'blue',
    'arg3': 'green'
}

We will cover an example of writing a decorator that utilizes input arguments from both the decorator and from the …

Confuse-A-Constructor

Today, we are going to confuse a constructor.

What is the constructor?

One of the first concepts encountered in object-oriented programming is that of the constructor, the method that …

Overview

This post is a summary of a useful Python programming pattern called the Registry pattern.

The Registry pattern is a way of keeping track of all subclasses of a given class.

More details about this pattern are available at https://github.com/faif/python-patterns.

What is the Registry Pattern …

Summary

This blog post covers a useful pattern for incorporating terraform commands into a Makefile.

This is useful for cases where terraform is being used to manage infrastructure. In the end you will be able to run a command like

make plan-infra
make deploy-infra

and and have this call the …

Summary

In this post, we cover a pattern for automatically generating a requirements.txt file that has the latest compatible versions of required software, and that specifies the full and exact version of each package to make the Python environment reproducible.

This will turn a requirements input file (called requirements.txt.in for example) that looks like

numpy

into a requirements file that specifies the exact version of numpy and all dependencies, like

numpy==1.18 …

Overview

This post covers a technique for mocking AWS in unit tests so that you can test functionality that normally requires API calls and handling responses, by mocking those responses instead of making actual API calls.

A Simple Example: Mocking API Responses

The Genuine AWS Call

Let's start with an example of an AWS API call. Here's how our program will be structured: start with a driver lister.py that creates an AWS secrets manager client and defines a function to list …

Introduction: A Simple CNN Example

As part of our weekly Deep Learning for Genomics reading group here in the Lab for Data Intensive Biology (DIB Lab), we are applying convolutional neural networks (deep learning) to various problems in genomics and biology …

Favorite Pi Approximations

What's your favorite \(\pi\) approximation?

Some of my favorite approximations of \(\pi\) come from Ramanujan-Sato series. These are mathematical series that generalize from a remarkable formula for \(\pi\) given by Srinivasa Ramanujan, an Indian mathematician:

About the Five-Letter Words

In Volume 4 Fascicle 0 of Donald Knuth's Art of Computer Programming, Knuth introduces a tool for exploring concepts in graph theory: the five-letter words. This is a collection …

About the Five-Letter Words

In Volume 4, Facsimile 0 of Donald Knuth's Art of Computer Programming, in which Knuth covers graph theory, he introduces a list of five-letter words as part of a data set useful in exploring graph theory and graph algorithms.

The list of words is part of the Stanford Graph Base, a set of data sets that are useful for studying graph theory and networks.

See Five Letter Words on the charlesreid1.com wiki for details.

Different by 1, Revisited

This post …

We're excited to announce the public release of centillion, a document search engine.

centillion is a search tool that can be used by any individual or organization to index Github repositories (including the content of markdown files), Google Drive folders (including the content of .docx files), and Disqus comment threads.

centillion is tested using Travis CI.

centillion was originally written for the NIH Data Commons effort (which recently concluded). centillion was built to facilitate information-finding in a project with hundreds of people at dozens of institutions generating a sea of email threads, Google Drive folders, markdown files, websites, and Github …

A short post with some thoughts on how writing Go code has helped me learn to stop worrying and love auto-formatting tools.

Go code is terse. Not Python-terse, but terse. And unlike Java, you don't find yourself constantly resorting to the security blanket of objects - something that Python (mercifully) can go either way on.

I used Java when I taught computer science at South Seattle College, and remember telling students once that one day, students taking CSC 142/143 would be using Go instead of Java. These days, I'm not as certain of that, but given that Go's strengths are …

Table of Contents

• What's your favorite PEP?
• PEP 0
• PEP 8
• PEP 20
• PEP 3099
  • Addendum: 2 to 3 Changes
• PEP 202
• All the PEPs on Github

What's your favorite PEP?

PEPs, or Python Enhancement Proposals, are documents in which features, additions, or general ideas are proposed as additions to the core Python language.

As a Python user, we believe it's important to ask questions like this.

Picking a "favorite PEP" is not just about having a ready and clever answer to a question you might expect in a technical interview; the PEP documents really are important, and really do …

A Predicament

Recently we spent some time contributing to dib-lab/eelpond (renamed to elvers), an executable Snakemake workflow for running the eelpond mRNAseq workflow.

In the process of tracking down a confusing bug in the Snakemake workflow, we used Snakemake's ability to print a directed acyclic graph (hereafter referred to as a dag) representing its task graph. Snakemake prints the dot …

NOTE: These ideas are implemented in the repository charlesreid1/2019-snakemake-byok8s.

Recap: Workflows as Executables

In our previous blog post, Building Snakemake Command Line Wrappers we covered some approaches to making Snakemake workflows into …

NOTE: These ideas are implemented in the repository charlesreid1/2019-snakemake-cli.

Basic Idea: Wrapping Snakemake API Calls

2018-snakemake-cli

This blog post covers the implementation of an idea that was originally explored in a blog post from Titus Brown, Pydoit, snakemake, and workflows-as-applications.

That blog post implemented a basic command line wrapper around the Snakemake API to demonstrate how a Snakemake workflow could be …

Flask + ILLO = Flaskadillo

On October 15, 2018, I had the opportunity to offer an in-lab learning opportunity (ILLO) at the Lab for Data Intensive Biology. The ILLO focused on Flask, a useful Python library for creating and running web servers. This library is useful because it has a very low learning curve, but also has the complexity to handle complicated, real-world projects.

As a part of this in-lab learning opportunity, I created repository with five simple Flask examples to highlight five useful capabilities of Flask.

The repository is called flaskadillo and it is available on git.charlesreid1.com or on …

A list of various ongoing projects:

• pandocs and panflute - how do i pandoc
• captain hook

The Git College of Surgery:

• git college of surgery
• The first successful git-commit-ectomy took place on Friday, June 1, 2018. See https://pages.charlesreid1.com/git-commit-ectomy
• Finishing this requires a better way to visualize git commits
• To do that, we have developed git-subway-maps

Python + APIs:

• building an API that calls APIs so you can API while you API (a webhook that calls a hook - see captain hook)
• testing APIs with Python + requests (currently top secret, coming soon.)

Python + Command line:

• command line utilities with python …

Table of Contents

• Starting example
• What needs to be changed
• Formatting the data
  • MediaWiki edits
  • Git commits
• Creating the color map
• How the calendar code works
• Final result

Starting example

Let's begin with a D3 example. Mike Bostock provided a Calendar View block illustrating how to draw a very interesting visualization of large amounts of data over time:

You might recognize this type of graph from Github, whose activity graph shows the same visualization.

The data shown in this example consists of several years of stock market data. It is a simple but very large data set, with each data …

This is Part 4 of a 4-part blog post on the mathematics of the 4x4 Rubik's Cube, its relation to algorithms, and some curious properties of Rubik's Cubes.

See Part 1 of this blog post here: Part 1: Representations

See Part 2 of this blog post here: Part 2: Permutations

See Part 3 of this blog post here: Part 3: Factoring Permutations

You are currently reading Part 4 of this blog post: Part 4: Sequence Order

Table of Contents

• Introduction

  • Order of a Sequence
  • What We Have Covered So Far

• Factoring Rubik's Cube Permutations

  • Factoring Permutations: A Review
  • Factoring Rubik's …

This is Part 3 of a 4-part blog post on the mathematics of the 4x4 Rubik's Cube, its relation to algorithms, and some curious properties of Rubik's Cubes.

See Part 1 of this blog post here: Part 1: Representations

See Part 2 of this blog post here: Part 2: Permutations

You are currently reading Part 3 of this blog post: Part 3: Factoring Permutations

See Part 4 of this blog post here: Part 4: Sequence Order

Table of Contents

• Introduction: Cycles, Sequences, and Order

  • Cycles
  • Sequences
  • Order

• Intercalation Product

  • Definition
  • Properties

• Factoring Permutations Using Knuth's Theorem A

  • Significance of Factors …

This is Part 2 of a 4-part blog post on the mathematics of the 4x4 Rubik's Cube, its relation to algorithms, and some curious properties of Rubik's Cubes.

See Part 1 of this blog post here: Part 1: Representations

You are currently reading Part 2 of this blog post: Part 2: Permutations

See Part 3 of this blog post here: Part 3: Factoring Permutations

See Part 4 of this blog post here: Part 4: Sequence Order

Table of Contents

• Introduction: Sequences and Permutations

• Representing Permutations: Two-Row Notation

  • Two-Row Notation
  • Two-Row Notation for Rubik's Cube

• Sequences

  • Review of Move/Sequence Notation …

This is Part 1 of a 4-part blog post on the mathematics of the 4x4 Rubik's Cube, its relation to algorithms, and some curious properties of Rubik's Cubes.

You are currently reading Part 1 of this blog post: Part 1: Representations

See Part 2 of this blog post here: Part 2: Permutations

See Part 3 of this blog post here: Part 3: Factoring Permutations

See Part 4 of this blog post here: Part 4: Sequence Order

Table of Contents

• Introduction: Why the Rubik's Cube

  • Why the 4x4 Rubik's Cube

• How the Rubik's Cube Works

  • The Pieces
  • Face Notation
  • Color Notation …

Generating Permutations

In today's post we're going to discuss the generation of permutations.

Often, in combinatorics problems, we are interested in how many different instances or configurations of a particular thing we can have (what we'll call "enumeration" or "counting"). However, that is different from wanting to actually see all of those configurations. Indeed, if we are counting something with an astronomical number of configurations, we don't want to try to list all of them.

However, as usual, Donald Knuth, who covers the topic of permutation generation in Volume 4A of his classic work, The Art of Computer Programming, uncovers …

NOTE: The code covered in this post uses Python 3. The scripts can be converted to Python 2 with minimal effort, but the author would encourage any user of Python 2 to "put on your big kid pants" and make the switch to Python 3. Let's all make this painful, drawn-out switch from Python 2 to Python 3 a thing of the past, shall we?

Table of Contents

• Introduction
• A Simple Manual Exmaple
• Pseudocode
• Python Code
• Output and Timing

Introduction

The letter/word coverage problem, as presented by Donald Knuth in Volume 4, Facicle 0 of his masterpiece Art of …

NOTE: The code covered in this post uses Python 3. The scripts can be converted to Python 2 with minimal effort, but the author would encourage any user of Python 2 to "put on your big kid pants" and make the switch to Python 3. Let's all make this painful, drawn-out switch from Python 2 to Python 3 a thing of the past, shall we?

Table of Contents

• Introduction
• Five-Letter Words with k Distinct Letters
  • Examining a Variation
• Lexicographic Ordering of Letters
  • Five-Letter Words with Lexicographically Ordered Letters
  • Five-Letter Words with Lexicographically Reversed Letters
• Finding Palindromes
  • Palindromes
  • Palindrome Pairs
  • Near …

NOTE: The code covered in this post uses Python 3. The scripts can be converted to Python 2 with minimal effort, but the author would encourage any user of Python 2 to "put on your big kid pants" and make the switch to Python 3. Let's all make this painful, drawn-out switch from Python 2 to Python 3 a thing of the past, shall we?

Table of Contents

• About the Five Letter Words
• The Usefulness of Five Letter Words
• Warm-Up Exercises
• Get Words Function
• Euclidean Distance
  • Euclidean Distance Code
  • Examples
  • Different-by-N Code
• Mo Words, Mo Problems
• References

About the Five-Letter …

• Undergraduate Research (UGR) Project: The Technologies
  • Overview of the UGR Project
  • Student-Led Components
    • Raspberry Pi
    • Question Skills
    • Aircrack/Airodump
    • Pi CSV Files
    • NoSQL Database
  • Backend Components
    • Stunnel
    • Virtual Private Server
    • Docker
  • Technologies Flowchart

Undergraduate Research Project (UGR): The Technologies

In this post we'll cover some of the technologies that were used in our South Seatte College undergraduate research project. The project involved an ensemble of different technologies to complete each component of the data analysis pipeline. Some components were planned for, but other components were implemented due to "surprise" challenges that cropped up during the course of the project, while …

Table of Contents:

• Overview of the Undergraduate Research (UGR) Project
  • South Seattle UGR Project
  • Engineering Design Project
  • Project Components: Extract, Store, and Analyze
  • The Internet of Things: Not Just a Buzzword
  • UW Research Symposium

Overview of the Undergraduate Research (UGR) Project

South Seattle UGR Project

For the past year, in addition to my duties as a computer science and math instructor at South Seattle College, I have served as a research mentor for an NSF-funded undergraduate research project involving (off-and-on) five different South Seattle students - all of whom have expressed interest in transferring to the University of Washington's computer science …

Table of Contents

• Background
• N Queens Revisited
• Perl Solution
• Python Solution
• Python vs. Perl: Walltime vs. Number of Queens
• Perl Profiling and Results
• Python Profiling
• Python Profiling Results
• Comparing Python to Perl
• Python vs. Perl: Walltime vs. Number of Solutions Tested
• The Winner: Perl for Small Problems, Python for Big Ones
• Sources

Background

Revisiting the N queens problem, this time implementing the solution in Python.

Verb-oriented solution, functional, and based on Perl solution

More fair comparison - both are interpreted languages, not compiled languages

Compare Python and Perl, ease of implementation, speed, flexibility

N Queens Problem

As a recap from the …