portrait picture


balancing software engineering & infosec

My Twitter Account is old enough to sneak out and drink

posted on Saturday 10th of October 2020 in ,

It has been thirteen years. In the tech world that is basically two to three life times of a regular startup. If I recall my time on Twitter correctly I opened an account when tweeting by SMS was still a thing, decided I do not like it and came back when the web interface became the favored method of tweeting.

Since then I have seen Twitter clients come and go. I have seen them lock down their API nearly killing third party clients and tanking the UI and UX of their own client. I have tweeted when the MacBook Air was pulled out of an envelop. I have plurked – remember Plurk? 🙂 – when the fail whale was the only thing you would see on the page. It was the platform I used most to talk to a sweet girl that later became my wife. As most others I complained about all the abusive and hostile behavior certain groups show on the platform – as on any other free platform gaining enough traction.

Twitter had a certain impact on my personal and professional life. Enough that I am willing to pay for it to make it better and keep it around. Looking back Twitter „was dying“ at least once every other year. And obviously other social networks were ready to fill the gap. (How did that work out App.net?)

Looking back at 13 years there are lots of memories. And putting it in the context of a teen, my account would likely sneak out to have a beer with friends. Feels crazy how long ago I tweeted a few random lines trying to figure out what the platform is, certainly not able to anticipate what it will become.

Twitter is not perfect. It messed up in the past. A lot. Sometimes they tried to reverse course, sometimes they stuck to decisions anyone who is not holding Twitter stock likely thinks were bad. There are cetainly enough problems that pain the platform to add work for months (if not years) to their backlogs. But I am looking forward to hopefully many more years of this crazy place where I can have a short conversation with professional esport players on the other side of the planet, check in what friends are up to, what my profession is angry about this week and get funny cat memes delivered directly to my device on a regular basis.

Amazon Prime Day

posted on Monday 5th of October 2020 in

I like the idea of Amazon Prime Day. Order the things you want throughout the week and get all of them delivered at the same time. Many of the things I order via Amazon are not time critical. I do not care when exactly they arrive, as long as they do. So batching them seems like a good idea, if only to not have someone drive to my house three times a week. Sadly Prime Day does not reliably work in our region – Amazon failed to deliver items for the third time in a row.

I wanted to do some work on my office desk over the weekend. Add new power strips, add an RaspberryPi for an always on dashboard and finally replace a cable that is not in the best shape anymore. I ordered everything on Monday and could have gotten it by Monday evening via same day express or on Tuesday via regular Prime.

You see, for me Friday is an excellent day for Prime Day. I have the things ready to go on Saturday when I do most work around the house and I do not have new toys sitting on the table tempting me to play with them during the week, when I should be doing other work. Sadly the power strips were not delivered, they arrived today. Why? No idea. Most likely some availability or allocation issues on Friday. My scotch was not delivered as part of the Prime Day delivery either, but by a separate carrier. (That might’ve been an age-restriction and/or transporting alcoholic beverages thing, although they did not make my wife sign anything.)

For five products I had three people drive to my house to deliver them. This is one more than if I would have ordered them via same day delivery – also my weekend plans would not have been messed with. Prime Day is a nice idea, in theory. Sadly… Execution sucks.

Deep dive: Django Q and SQS

posted on Wednesday 23rd of September 2020 in

When working on a Django application the de facto recommendation for a task queue is Celery. I believe this is a good recommendation. It is kind of like buying IBM – “no one was ever fired for buying IBM”. I started using Django Q more recently and it is doing a great job. One system I built using it processes roughly 400k tasks per day. Surely not the largest system and surely not the most impressive number, but decent enough to say that Django Q is a solid choice.

But as with many smaller projects there are sometimes a few gotchas you are running into. This becomes painfully obvious when setting up an app using SQS. Let me walk you through the steps I took to make Django Q play nicely with our AWS setup at Grove Collaborative.

Redis is great, but…

First of all you have to configure Django Q to use SQS. You do this by adding the Q_CLUSTER dictionary to your settings.py with the sqs key.

If you are familiar with AWS and boto3 you might know that you can either provide the AWS region when initialising a new connection or you can have a standard config stored outside your project which is automatically used. Or you avoid access keys and the hassle of managing them and start using IAM roles.

Django Qs documentation actually explains this option and shows all three keys for the SQS configuration as optional. So a valid configuration would be

    'name': 'SQSExample',
    'workers': 4,
    'timeout': 60,
    'retry': 90,
    'queue_limit': 100,
    'bulk': 5,
    'sqs': {}

But when you try to deploy your app this way it might start raising an exception explaining that it cannot connect to Redis. If we look at the get_broker function we see that Redis is the default. Conf.SQS is simply the value of Q_CLUSTER["sqs"]. Know what is going on here?

settings = {
    "sqs": {}

x = settings.get("sqs", None)

if x:
  print("let's use SQS!")
  print("okay, Redis it is")

This code snippet will also tell you that we will be using Redis. Empty dictionaries evaluate to False. According to the documentation the configuration is valid, but it will not work.

Computer says no

Okay, we got this. We are finally loading the SQS broker. But we might run into an exception telling us that we cannot create a queue.

Usually when you provision a new AWS environment you setup the instances, database, queues,… all the services you want AWS to host for you. While setting them up you most likely create an IAM role or access key with the right amount of permissions to use the services the way your service will use them.

So why do we get permission errors when starting our service?

Django Q calls boto3 create_queue method when initialising the broker. If a queue already exists, assuming you do not pass around conflicting attributes, it returns the URL for the queue. All good, right? Not if your IAM role or access key does not have “create permissions” for SQS, which it should not need.

An easy fix is creating your own broker and overwriting get_queue.

# coding: utf-8
from django_q.brokers.aws_sqs import Sqs

class GetSqsBroker(Sqs):
	self.sqs = self.connection.resource("sqs")
    return self.sqs.get_queue_by_name(QueueName=self.list_key)

The only thing left to do is specify the broker in your settings. Assuming your class lives in broker.py you replace sqs with broker_class and a path to your custom broker so it can be imported.

    'name': 'SQSExample',
    'workers': 4,
    'timeout': 60,
    'retry': 90,
    'queue_limit': 100,
    'bulk': 5,
    "broker_class": "broker.GetSqsBroker",

This surely is not an as generic implementation as the one Django Q ships with and one could argue that it is less comfortable to use. I would counter that with the argument that libraries should not randomly create resources or expect permissions not absolutely necessary to operate.

What was my queue name again?

While working through the code you will often see Conf.PREFIX and list_key being mentioned.

Looking at the configuration class we know Conf.PREFIX is the name we specify in the configuration dictionary. __init__ is setting list_key to Conf.PREFIX if the keyword argument is not passed in when initialising a new instance of the broker. For completeness the list_key on the instance is set here. When we finally get the queue the list key is used as queue name.

I do not think this is a surprise, but when you implement a custom broker or try to debug why you see unexpected error messages you should know the way data and configuration flows through the system. When you design a library you often will have to mutate variable names going from a human readable config file to a generic implementation. This is just a nice, simple to follow example I will surely use one day during a training.

Django Q is ready for production

As I mentioned earlier I have successfully used Django Q at a decent scale and I do not expect it to fall over at larger scales. You might want to set your save_limit to -1. Overall it is easy to use, straight forward to debug and small enough to not be a big liability if you would ever have to take over maintenance.

While you can surely create your own broker and  ensure your configuration dictionary has a key in it, you should not have to. I opened two pull requests to fix those issues.

Thoughts on Apples Time Flies event

posted on Thursday 17th of September 2020 in , ,

Let me start with saying that I did not expect much from the event and only followed it via MacRumors.com live feed. And as it turns out I was right. Not because it was a bad event or that there were no announcements, just nothing I expected to be particularly excited about. Yet a few things actually caught my eye.

I will not spend a lot of time on the Apple Watch. More health. Watch faces. The interesting part was that Apple discontinues the ceramic version. When the first version was released there was a stupidly dragged out discussion about the gold version. Which was a nice and well-working marketing stunt back then. But all other premium models did not see the same hype and from what I have heard did not sell that well. Apple and other companies trained their consumers well on thinking they need to update every year (or every other year). And the more frequent you know you will be updating, the less likely most people are to spend a significant premium. It also shows a big difference I always pointed out since the first Watch was released. This is not a watch. It is a fitness tracker. A small phone. A music player. It displays the time, but essentially it is an extension of your phone. An actual watch actually worth spending the amount of money for that Apple wants to charge for their premium version is hand made and can be passed down to your kids.

The new iPad Air is nice and I appreciate Apple putting a TouchID sensor back in. Not because I dislike FaceID, but considering we all will be wearing masks for a long time and that the form factor of the sensor will allow them to ship it in an iPhone will make day to day life easier.

What really excites me is the iPad. An A12 chip, keyboard and pencil support? And with an educational discount $299? This is what I would buy for my kids. Nothing on the market comes close to the value for the price. We can obviously have the usual discussion around closed ecosystems and kids not learning to compile their own kernel, but the outcome will be the same as for the last ten years.

One nice update I feel like pointing out is AppleCare+ now covering 2 accidental damage incidents every 12 month. This just makes it more of a „do not ask, just click the button“ type of purchase for basically any device.

Apple One and Apple Fitness+ are a nice addition to the subscription offering. One makes a lot of sense if you go all in on Apples ecosystem and if you are already on the storage plan and maybe one or two services you will likely be able to safe some money or get other services „for free“. The Premier plan will not be available in Germany as we do not have News or Fitness+, so I am not sure if it will be worth going for the mid tier and paying for the 2TB storage separately. Fitness+ seems like the natural evolution of Apples focus on health and it if the content is well curated it will enable people to train from home with proper directions and without bro-science. (Looking at you YouTube…)

What I did not expect was actually seeing something announced that I will be excited enough about to place an order. Well, I was wrong.

The Solo Loop looks like a great idea for two reasons. When working out I usually wear my watch around the mid section of my arm, as I often wear weight lifting gloves. Regular sport bands are not wide enough for fit comfortably, so this is always a hassle. The other reason is wearing the Watch during the day. I type a lot. And I cannot stand a regular watch band, something about the feeling of it between my wrist and the desk is just off-putting. I do not have this problem with a regular wristband. Curious to see if this means I will be wearing my Watch at home as well.

Apple delays Anti-Ad-Tracking feature

posted on Friday 11th of September 2020 in ,

One of the features I was looking forward to with iOS 14 was Apple taking a hard stance on ad targeting and enabling consumers to get some more control over what they will be exposed to and how their data will be handled. Sadly Apple will be delaying the feature. Ignoring the delay, this feature seems to work as advertised and I am really excited when it becomes available.

Facebook and other advertisers expect that customers will not want to share their IDFA’s for ad targeting purposes and will therefore decline consent for the ad blocking popups that Apple has implemented in ‌iOS 14‌.

It nearly is like people do not want to see targeted ads, have their data collected or be constantly be confronted with advertisement.

Mobile developers that spoke to The Information said that they’ve had little time to prepare for Apple’s change, which was announced in June alongside ‌iOS 14‌. Apple has also not provided a way for them to target ads without using the IDFA.

So the feature is indeed working as intended. This makes me even more sad that it is delayed. It looks like we finally have technical means to do something about the mess we are in. And the only parties profiting off of it are getting really nervous.

What is actually a bit stunning is the fact that advertisers act as they have a right to track people however they want and without consent. It speaks to the whole industry and how they perceive their target audience.

One thing I have often not seen being discussed are the long term implications of this. With ad revenue potentially dropping for all parties new monetization models are needed. Someone has to pay the bills and keep the lights on. Hopefully we go back to an old model, namely developers charging an appropriate amount of money. I am a bit worried that the same people celebrating the anti-ad-tracking feature are the same that believe apps can be developed sustainably for a one time $5 payment.

I could see a hybrid model work well. You either see targeted ads and are tracked across the app or a pay for a subscription. An appropriate one time payment might also work, but considering that the cost of software is not well understood right now this might turn out to not work that well. Find a model which gives people the choice how they want to pay. Including proper education what the implications of both payment options are.