Howdy all! I hope everyone is riding out this delta covid surge reasonably well.

This will likely be my last Pywikibot report. My code reviews are stuck and working on Pywikibot is remarkably lonely. Pywikibot is neat, but it’s difficult to stay interested when my contributions dawdle on a shelf. C’est la vie.

Roman Colosseum

In lieu of code I’m binging Death Throes of the Republic. Rome’s collapse began with the senate’s murder of the reformer Tiberius Gracchus. Breaking the norm against political violence just once spiraled out of control into a tit-for-tat revenge cycle that must have horrified its original perpetrators.

Rome offers a troubling warning of what the January 6th lynch mob could have begun. Folks, lets not play with fire.


Type Hints

Prior to my Rome binge I doubled down on Pywikibot’s type hints. Poor Xqt. I kinda buried him in code reviews…

Happy summer everyone! This month my sister and I camped at Fort Townsend, which aside from adorable douglas squirrels features an explosive dismantling tower. Its plaque reads…

“This brick building was a US Navy Explosives Laboratory built during World War II. Enemy submarine torpedoes and sea mines were brought here to be dissembled. A large two million volt x-ray machine was utilized to detect the exact position of the detonation fuse. Ten inches of steel could be penetrated by the high powered machine.”

I love history. According to Dan Carlin’s Supernova in the East Japan had significantly superior torpedoes at the outbreak of war. Maybe this tower helped close that gap?

Torpedo tower


Type Hints

Much of my work from last month is stuck in code review purgatory, so this month I swapped to type hinting which is far quicker to review. Pywikibot has 94 files so like our scripts this will be an ongoing project.

Naturally this is accompanied by small bug fixes as I go…

Happy summer everyone! Washington sizzled this month amid a historic heat dome. Willis Carrier, you are a national treasure.

I love AC


Pywikibot Scripts

To parallelize rampup with development I sunk this month into our scripts. I plan to refactor every script and expand their test coverage so this will be an ongoing project. This month included…

add_text.py

movepages.py

listpages.py


Deprecation Policy

In consultation with Xqt and JJMC89 I wrote a deprecation policy for pywikibot. From version 6.4.0 onward pywikibot will use symantic versioning, and assure backward compatibility for minor version bumps.

I also dropped our use of DeprecationWarnings since invisible notifications… aren’t particularly helpful.


Verbosity Reduction

Last, but probably my favorite work this month, I greatly reduced the verbosity of our test and sphinx build output.

My test reduction cropped our unit tests from 536 lines to ~20 (96% reduction). Documentation compilation dropped from 120 lines to 48 (60% reduction).

Hi all! This month I moved on from Tor to begin volunteering with Wikipedia. Covid taught me the importance of face to face contact, and Wikipedia has local Seattle meetups that could scratch an itch Tor didn’t.

Something I desperately look forward to now that I have…

Vaccine


To get my feet wet I invested this month toward standardizing and making minor adjustments to pywikibot

Happy election season, everyone! Isolation drove me to spend a week just reading the Constitution and surrounding case law. My favorite is the 27th amendment, ratified 202 years after it was proposed because a TA gave 19 year old Gregory Watson a bad grade. Civics is delightfully odd.

Vote


Has it really been three months since my last post? Pandemic malaise has weighed me down, but also my work hasn’t been terribly sexy…

Honestly since Shari’s departure I haven’t felt excited by anything at Tor. I should poke around some other communities to see what’s around.

Dear diary, pandemic day 7,114. Placed a fancy hat on the fish. Questioned sanity, but voices reassure me I’m still sane. Treasonous cucumber sentenced to food processor, but asparagus now conspires too? Pez dispenser is all I can trust.

Fish with hat


Asyncio

Vegetable coups aside, Illia and I migrated Stem to asyncio! Txtorcon was previously the only asynchronous game in town, but now Stem supports both synchronous and asynchronous usage.

import asyncio

from stem.control import Controller


async def print_version_async():
  async with Controller.from_port() as controller:
    await controller.authenticate()
    print('[with asyncio] tor is version %s' % await controller.get_version())


def print_version_sync():
  with Controller.from_port() as controller:
    controller.authenticate()
    print('[without asyncio] tor is version %s' % controller.get_version())

print_version_sync()
asyncio.run(print_version_async())
% python demo.py 
[without asyncio] tor is version 0.4.5.0-alpha-dev (git-9d922b8eaae54242)
[with asyncio] tor is version 0.4.5.0-alpha-dev (git-9d922b8eaae54242)

Internally Stem is now asynchronous from the bottom up. This provides deeper control over our execution, for example asyncio.wait_for() can apply a timeout to any of our methods.

Usually asyncio doesn’t play well with conventional usage, but Stem transparently applies a compatibility layer so synchronous users likely won’t even notice this change.

Hi all. After vanishing the last few months I’m delighted to announce a major new Stem feature: type hints!

Python 3’s type information provide benefits familiar to statically typed languages like Java:

IDE autocompletion

pycharm demo

Static type checks

mypy demo

Fresh off the press, this has many rough edges. In particular our library’s internal metaprogramming is incompatible with type data. However, as we refactor our APIs for Stem’s 2.0 release this will only improve.

Aloha! COVID-19 reached Seattle so I’m hunkering down, but prior to that I visited my family in Kona, Hawaii.

Lazy turtles and cute geckos aside, I particularly loved Kona Joe’s roastery tour and Pu`uhonua O Hōnaunau. If you travel to the big island check ’em out!

Family in Hawaii


Deprecation Removal

This month we dropped Python 2 support, and with it 1,200 lines of compatibility hacks. Stem’s next release will require Python 3.6 or greater.

This month I also culled another 2,900 lines of deprecated code. Like any sophisticated library Stem grew organically over the last eight years. Some features are a hit, others clutter our API. Stem 2.x is our opportunity to streamline so expect more rearchitecture going forward.

Winter is such a snoozy season. Eggnog and Stem’s holiday release fading to memory, January I got back to work.

What Big Dreams You Have


Port BridgeDB to Python 3

Python 2’s discontinuation makes it porting season! Stem and Nyx are ahead of the curve, so this month I migrated BridgeDB for Philipp.

This was a lot of work, but only possible thanks to Isis’ great test coverage. My compliments to our former maintainer!


GitHub Migration

Rather than move to GitLab I joined Ooni and part of the Network Team on GitHub. For tickets and pull requests please visit Stem and Nyx on their new home!

Georg asked to subsume responsibility for DocTor so he can move it according to his platform preference instead.

Closing the decade I’m pleased to announce Stem 1.8, the final release in Stem’s 1.x series and with it Python 2.x support.

What is Stem, you ask? For those who aren’t familiar with it Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications like Nyx.

https://stem.torproject.org/

So what’s new in this release?


CollecTor Downloader

Through our descriptor archive, CollecTor, Stem can now read Tor’s network topology at any prior point in time.

For example, listing today’s exits is as simple as…

import datetime
import stem.descriptor.collector

yesterday = datetime.datetime.utcnow() - datetime.timedelta(days = 1)
exits = {}

for desc in stem.descriptor.collector.get_server_descriptors(start = yesterday):
  if desc.exit_policy.is_exiting_allowed():
    exits[desc.fingerprint] = desc

print('%i relays published an exiting policy today...\n' % len(exits))

for fingerprint, desc in exits.items():
  print('  %s (%s)' % (desc.nickname, fingerprint))
% python demo.py 
1229 relays published an exiting policy today...

  MrExit (D628F6BB2330B3F78DBB4BED466B0A586D74782E)
  pangea03 (F21DFB7CCD5EEF3E021086EC96EF7CFCAA72F4F3)
  MacarenaValdes (5E3FD31B9DC279C06AD051D68BE08914F6CD3B46)
  TEMPORA (05EAA0696DCB694D6811042348DACD5059FE64AD)
  Quintex43 (1E5136DDC52FAE1219208F0A6BADB0BA62587EE6)

Bandwidth Metrics

Bandwidth Authorities generate the latency heuristics that govern Tor’s path selection. Guiding circuits to be fast, without overburdening individual relays.

With Stem you can peruse this information…

import stem.descriptor.remote
import stem.util.str_tools

bandwidth_file = stem.descriptor.remote.get_bandwidth_file().run()[0]

print('Bandwidth measurements are...\n')

for fingerprint, measurement in bandwidth_file.measurements.items():
  bandwidth = '%s/s' % stem.util.str_tools.size_label(1024 * int(measurement.get('bw', '0')))
  print('  * %s (%s) averaged %s' % (measurement.get('nick', ''), fingerprint, bandwidth))
Bandwidth measurements are...

  * DigiGesTor1e1 (0111BA9B604669E636FFD5B503F382A4B7AD6E80) averaged 23 MB/s
  * WonderWoman42 (E5AA85FA69CDC31900C86E6427C7E5DE11DE9E2D) averaged 37 MB/s
  * alterspalter (B6F0BC2B93CB3EFFFFF724CB4F5E025FB15EFB70) averaged 2 MB/s
  * blueberry (FE80E192AD48A1BEB02D88EBC7663061176E1A79) averaged 1 KB/s
...

Hidden Service v3 Descriptors

With George’s help Stem now reads, decrypts, and even creates HSv3 descriptors. For example…

from stem.descriptor.hidden_service import (
  HiddenServiceDescriptorV3,
  InnerLayer,
  IntroductionPointV3,
)

print(HiddenServiceDescriptorV3.content(
  inner_layer = InnerLayer.create(
    introduction_points = [
      IntroductionPointV3.create('1.1.1.1', 9001),
      IntroductionPointV3.create('2.2.2.2', 9001),
      IntroductionPointV3.create('3.3.3.3', 9001),
    ],
  ),
))
% python demo.py
hs-descriptor 3
descriptor-lifetime 180
descriptor-signing-key-cert
-----BEGIN ED25519 CERT-----
AQgABqvHAX8wXzJY+FqoJQPXNZ8u+SQGPZ1WN/r3hUna0R2AXQnEAQAgBAAuqibl
ALcKa/4nHtLZn2zKV8L4XIpkRyRm7btWPLpYN5Gseb03H5exL+I3SqfG3uNDw5QK
CmPlCQUy3usouSwhO/qWgdy0//bP5kRDma5GDXXWoi3+xTKM6Jez7TGxPAU=
-----END ED25519 CERT-----
revision-counter 1573695064
superencrypted
-----BEGIN MESSAGE-----
aDJodcMjhCvz1K7JCJEAH1H24hvoZ7gZw53AhPdvpHu+5d1Ogwio4qcIXEK1pEgy
QFF1fE6tnCzsk++eMa2WaKwIJYGLPoCnta78H5Ve6VoMj+Pyb5rE6wPTMTPSVm6M
UjllArr7DS8YcofloDxu3iwC3JZYFt/LB6ahq6lBKeot2BD/11pNggkZrZOCLgNQ
pUVyQau7K8ynagVlNNESnI3FccOBaBB4Xa5mObK2ylyiLQ08MqaImW7X2gxeZltT
/C/xtiJXGm2CzkjPpBpMWm09p7/a97GEWca5e8+fhpmGrN7zjAwjYInTvQHS5AyU
7eUFg8ItrRxAiRq4fbe/zepiq2vgfj1Pt7uxC0KCTcLWpd9O/FIvcFSk27Yrtniw
... etc...
-----END MESSAGE-----
signature VDDXXLvgU6qjRI4zfJR3GbQuVjz98qO0LI5gsI60LtGXK2POZ4E+3YVVWuVaEkvMsZaku5qCutIcu74/WQMxCQ